* Stop 19 test files racing on one PowerShell startup cache
Backend CI run 32341628757 on `1c3dde199` finished `284 failed, 8498 passed`. Every
one of the 284 was a pwsh subprocess ending `died with <Signals.SIGABRT: 6>`, across
19 files that all read as Windows-installer regressions. None of them were. 222 of
the aborts land inside a two-second window, 88 at 07:09:30 and 133 at 07:09:31,
which is a mass kill of every live pwsh rather than independent per-test flakiness.
The cause
------------------------------------------------------------------------
Every `-NonInteractive` startup reads and rewrites an ~83 KB
`$XDG_CACHE_HOME/powershell/StartupProfileData-NonInteractive`, and XDG_CACHE_HOME
defaults to `$HOME/.cache`. Under `-n 4` all four xdist workers share one HOME, so
the whole job's pwsh processes race on one file and a startup that deserialises a
half-written one dies before it reaches our script. `Stack overflow.` is .NET's
failfast, which cannot unwind a blown stack, so it prints one line and calls
abort(); that is the SIGABRT (PowerShell/PowerShell#24461).
Measured twice, independently, 4000 startups per arm:
run 1 shared cache dir 7/4000 died {-11: 3, -6: 4}
private cache dirs 0/4000
run 2 shared cache dir 11/4000 died {-11: 10, -6: 1}
private cache dirs 0/4000
Three distinct crash shapes appeared, and each names the torn file rather than our
scripts: `Stack overflow.`, `System.IO.FileLoadException: The given assembly name`,
and `System.ArgumentException: String cannot have zero length.` 18 deaths in 8000
shared startups, 0 in 8000 private.
CI agrees from the other direction. Of the pwsh-heavy files in that run, exactly one
had zero failures, tests/test_windows_amd_gpu_scan_fallback.py, and it is the only
one that hands its child a private HOME, across roughly 80 startups where the run's
own rate predicts about 16 failures.
What is NOT established
------------------------------------------------------------------------
Neither experiment reproduces CI's rate. Roughly 20% of pwsh startups died there
against 0.2 to 0.3% here, and at CI's actual `-n 4` on this box I measured 0/1200 in
both arms: the race needed 48-way concurrency before it appeared at all. The likely
reason is that four workers on a 4-core runner are in real contention while four
threads on a 192-core box almost never overlap in the critical section, but that is
reasoning and not a measurement, so treat the mechanism as proven and the magnitude
as unexplained. That is also why this does not stop at removing the shared file.
Three layers, in order
------------------------------------------------------------------------
1. Remove the contended resource. One cache directory per xdist worker, fresh per
session. Workers run their tests one at a time, so within a worker the startups
stay sequential and the cache still does its job warm; across workers the
directories are disjoint and there is nothing left to race on. Fresh rather than a
stable path, because a cache torn by an earlier run would otherwise poison every
later session on the same box.
2. Retry a run that produced no verdict. Three attempts, unslept, because the trigger
is process startup rather than a resource that frees up.
3. Attribute what is left. A crash raises PwshInterpreterCrash naming the interpreter.
Layer 1 is the fix; 2 and 3 exist because of the unexplained magnitude above.
Deliberately NOT done: bounding pwsh concurrency with a lock, or giving up `-n 4`.
The workflow records 806.1s to 219.7s from that flag, and the contended resource can
be removed rather than rationed.
The rule that keeps this honest
------------------------------------------------------------------------
A signal is not a verdict, so retrying it papers over nothing: the script never ran
to its end. A normal exit is returned untouched on the first attempt whatever its
code, so a pwsh that runs and gives the WRONG answer still fails with its own
message. Getting that second half wrong would turn this into a way to retry real
regressions into green, which is worse than the bug it fixes, so both directions are
executed in tests/studio/test_pwsh_interpreter_crash_attribution.py against a real
SIGABRT rather than reviewed.
Mutation-tested: relaxing the crash test from `returncode < 0` to `returncode != 0`
fails test_a_clean_run_with_the_wrong_answer_still_fails_with_its_own_message and
test_a_clean_run_is_not_retried, which are exactly the two that guard that direction.
This also generalises `_run_pwsh` from tests/studio/test_install_phase_timing.py,
added earlier today for a second, signal-free shape: pwsh printing its "The
PowerShell process will exit" banner and exiting normally with empty stdout. That one
cannot be seen in the exit status, so it stays a text match.
Verified
------------------------------------------------------------------------
tests/python/test_windows_xformers_installer.py, tests/studio/test_install_phase_timing.py,
tests/studio/install/ and the new guard: 2635 passed, 3 skipped.
Guard alone: 5 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop the subprocess import the pwsh conversion left behind
Source lint's import-hoist check is right: every subprocess.run in
test_windows_xformers_installer.py became run_pwsh, so `import subprocess` has no
references left except the one inside a comment explaining why run_pwsh is used
instead. Its wording names the shape exactly -- "was used before, now unused
(references re-pointed)" -- which is what a mechanical call-site rewrite leaves
behind.
Swept the other 18 converted files the same way with an AST pass rather than by
eye. This was the only real one: the remaining hits are `from __future__ import
annotations`, which every such scan reports, and a PropertyMock in
test_rocm_support.py that is present on main unchanged.
42 passed.
* Suppress the core dump on the forged SIGABRT
tests/test_deliberate_crashes_suppress_cores.py caught this: the abort child had no
PR_SET_DUMPABLE=0, so each of these aborts piped a multi-MB core to apport before the
child could be reaped. The guard is right and its message names the fix.
The child still exits -6 and PR_GET_DUMPABLE reads 0, so all five verdicts are
unchanged. Linux-only and non-fatal elsewhere: Windows has no CDLL(None) and pipes no
core, so arming it there would trade a no-op for a lost test.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
* Ask uv for a Python that can import torch, and skip the one that cannot
Fixes#7803.
CPython 3.13.8 carries python/cpython#139783: inspect.getsourcelines() drops a
function body when a decorator is followed by a comment. That is the shape of
the @_overload_method blocks in torch 2.11's nn/modules/rnn.py, which are parsed
at import time, so `import torch` dies with IndentationError. 3.13.9 was an
expedited release carrying only that fix.
install.sh asked uv for a bare "3.13" and let it choose the patch. Measured with
uv 0.9.2 and only 3.13.8 present:
--python 3.13 -> 3.13.8 then import torch: IndentationError
--python >=3.13.9,<3.14 -> 3.13.12 then import torch: 2.11.0+cpu OK
So the request is the fix. PYTHON_SKIP names the releases that cannot run the
stack and _python_request turns a bare 3.13 into the range; a venv left on a
skipped interpreter by an earlier run is recreated, on any platform, which the
previous check could not do because it was gated on macOS arm64.
UV_MIN_VERSION also moves to 0.9.3, the first uv whose bundled manifest carries
3.13.9. That is belt-and-braces rather than the fix, since the range resolves on
0.9.2 too. Raising it pulls every 0.8.16-0.9.2 host into the refresh block, so
an existing uv in that range is no longer fatal when the network is unreachable.
Windows reaches such an interpreter differently: uv is handed a resolved path,
never a version, so it cannot pick the patch, but Find-CompatiblePython matches
on the minor version and would return an already-installed 3.13.8.
Remove-SkippedPython turns that into "not found" so the caller installs
$PythonFallbackFullVersion (3.13.13). The uv floor is left alone there, since
the uv-managed Python path is not taken on Windows.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Screen the skipped patch inside the resolver, and never delete the only venv
Windows: Find-CompatiblePython matched on the minor, so the screen sat on
its result and every other caller of the resolver -- the two install
helpers among them -- could still be handed the interpreter the first
call had just rejected. Nulling the result also ended the search, so a
host with 3.13.8 and a healthy 3.12 failed instead of using the 3.12. The
patch is already in the string the minor comes from, so screening during
enumeration costs no extra subprocess and keeps the fallback ladder.
install.sh: the legacy-layout migration moves the old environment into
$VENV_DIR without arming the rollback, so a plain rm -rf before a
recreate that then fails leaves the machine with nothing. Move it aside
through the existing rollback machinery instead.
Ask uv for the series minus the skipped patches rather than for a floor
above them: an offline host, or a uv whose manifest predates 3.13.9, can
still have a good cached 3.13.7 that a floor would refuse. Measured with
uv 0.10.7, only 3.13.7 and 3.13.8 installed, --offline: "3.13" gives
3.13.8, ">=3.13.9,<3.14" errors, ">=3.13,<3.14,!=3.13.8" gives 3.13.7.
Hoist the install.ps1 extractions out of the f-strings: a backslash in an
f-string expression is a syntax error before 3.12 and the repo is 3.9+.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Run the resolver driver only where its fake launcher can execute
The launcher is a /bin/sh script, and Windows has neither a shebang nor a
PATHEXT entry for an extensionless file, so Get-Command never finds it and
Find-CompatiblePython reports "none" whatever versions the tree holds.
Two cases failed on a real windows-latest runner and the third passed for
that reason rather than on merit. Skip the three on Windows, where the
rest of the file still covers the screen, and pair the negative case with
a positive control so a harness that cannot run the launcher fails
instead of quietly agreeing.
* Do not screen the interpreter for an install that never imports torch
Every entry in the skip list is there for one reason: it cannot import
torch. A --no-torch/-NoTorch install never does, so refusing the machine's
only 3.13 would send a locked-down GGUF-only host into a download it may
not be able to complete, over a package it will not install.
Also turn away anything that is not a plain X.Y before the arithmetic:
a relative --python path like 3.13/bin/python survived the globs, and
dash aborts the whole install with "Illegal number" rather than reaching
uv. And keep the uv version probe alive on an image with no awk, which is
precisely the host the offline exception around it exists to protect --
the pipeline exits 127 there and set -e was killing the install before
the code could treat the version as unreadable.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Daniel Han