unsloth/tests/python/test_windows_git_gate.py
Daniel Han 827d25931b
Stop 19 test files racing on one PowerShell startup cache (#9371)
* Stop 19 test files racing on one PowerShell startup cache

Backend CI run 32341628757 on `1c3dde199` finished `284 failed, 8498 passed`. Every
one of the 284 was a pwsh subprocess ending `died with <Signals.SIGABRT: 6>`, across
19 files that all read as Windows-installer regressions. None of them were. 222 of
the aborts land inside a two-second window, 88 at 07:09:30 and 133 at 07:09:31,
which is a mass kill of every live pwsh rather than independent per-test flakiness.

The cause
------------------------------------------------------------------------
Every `-NonInteractive` startup reads and rewrites an ~83 KB
`$XDG_CACHE_HOME/powershell/StartupProfileData-NonInteractive`, and XDG_CACHE_HOME
defaults to `$HOME/.cache`. Under `-n 4` all four xdist workers share one HOME, so
the whole job's pwsh processes race on one file and a startup that deserialises a
half-written one dies before it reaches our script. `Stack overflow.` is .NET's
failfast, which cannot unwind a blown stack, so it prints one line and calls
abort(); that is the SIGABRT (PowerShell/PowerShell#24461).

Measured twice, independently, 4000 startups per arm:

  run 1  shared cache dir     7/4000 died  {-11: 3, -6: 4}
         private cache dirs   0/4000
  run 2  shared cache dir    11/4000 died  {-11: 10, -6: 1}
         private cache dirs   0/4000

Three distinct crash shapes appeared, and each names the torn file rather than our
scripts: `Stack overflow.`, `System.IO.FileLoadException: The given assembly name`,
and `System.ArgumentException: String cannot have zero length.` 18 deaths in 8000
shared startups, 0 in 8000 private.

CI agrees from the other direction. Of the pwsh-heavy files in that run, exactly one
had zero failures, tests/test_windows_amd_gpu_scan_fallback.py, and it is the only
one that hands its child a private HOME, across roughly 80 startups where the run's
own rate predicts about 16 failures.

What is NOT established
------------------------------------------------------------------------
Neither experiment reproduces CI's rate. Roughly 20% of pwsh startups died there
against 0.2 to 0.3% here, and at CI's actual `-n 4` on this box I measured 0/1200 in
both arms: the race needed 48-way concurrency before it appeared at all. The likely
reason is that four workers on a 4-core runner are in real contention while four
threads on a 192-core box almost never overlap in the critical section, but that is
reasoning and not a measurement, so treat the mechanism as proven and the magnitude
as unexplained. That is also why this does not stop at removing the shared file.

Three layers, in order
------------------------------------------------------------------------
1. Remove the contended resource. One cache directory per xdist worker, fresh per
   session. Workers run their tests one at a time, so within a worker the startups
   stay sequential and the cache still does its job warm; across workers the
   directories are disjoint and there is nothing left to race on. Fresh rather than a
   stable path, because a cache torn by an earlier run would otherwise poison every
   later session on the same box.
2. Retry a run that produced no verdict. Three attempts, unslept, because the trigger
   is process startup rather than a resource that frees up.
3. Attribute what is left. A crash raises PwshInterpreterCrash naming the interpreter.

Layer 1 is the fix; 2 and 3 exist because of the unexplained magnitude above.

Deliberately NOT done: bounding pwsh concurrency with a lock, or giving up `-n 4`.
The workflow records 806.1s to 219.7s from that flag, and the contended resource can
be removed rather than rationed.

The rule that keeps this honest
------------------------------------------------------------------------
A signal is not a verdict, so retrying it papers over nothing: the script never ran
to its end. A normal exit is returned untouched on the first attempt whatever its
code, so a pwsh that runs and gives the WRONG answer still fails with its own
message. Getting that second half wrong would turn this into a way to retry real
regressions into green, which is worse than the bug it fixes, so both directions are
executed in tests/studio/test_pwsh_interpreter_crash_attribution.py against a real
SIGABRT rather than reviewed.

Mutation-tested: relaxing the crash test from `returncode < 0` to `returncode != 0`
fails test_a_clean_run_with_the_wrong_answer_still_fails_with_its_own_message and
test_a_clean_run_is_not_retried, which are exactly the two that guard that direction.

This also generalises `_run_pwsh` from tests/studio/test_install_phase_timing.py,
added earlier today for a second, signal-free shape: pwsh printing its "The
PowerShell process will exit" banner and exiting normally with empty stdout. That one
cannot be seen in the exit status, so it stays a text match.

Verified
------------------------------------------------------------------------
tests/python/test_windows_xformers_installer.py, tests/studio/test_install_phase_timing.py,
tests/studio/install/ and the new guard: 2635 passed, 3 skipped.
Guard alone: 5 passed.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Drop the subprocess import the pwsh conversion left behind

Source lint's import-hoist check is right: every subprocess.run in
test_windows_xformers_installer.py became run_pwsh, so `import subprocess` has no
references left except the one inside a comment explaining why run_pwsh is used
instead. Its wording names the shape exactly -- "was used before, now unused
(references re-pointed)" -- which is what a mechanical call-site rewrite leaves
behind.

Swept the other 18 converted files the same way with an AST pass rather than by
eye. This was the only real one: the remaining hits are `from __future__ import
annotations`, which every such scan reports, and a PropertyMock in
test_rocm_support.py that is present on main unchanged.

42 passed.

* Suppress the core dump on the forged SIGABRT

tests/test_deliberate_crashes_suppress_cores.py caught this: the abort child had no
PR_SET_DUMPABLE=0, so each of these aborts piped a multi-MB core to apport before the
child could be reaped. The guard is right and its message names the fix.

The child still exits -6 and PR_GET_DUMPABLE reads 0, so all five verdicts are
unchanged. Linux-only and non-fatal elsewhere: Windows has no CDLL(None) and pipes no
core, so arming it there would trade a no-op for a lost test.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
2026-08-20 04:22:41 -07:00

140 lines
4.9 KiB
Python

# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
"""Git is optional on the consumer Windows path, but still required for source builds."""
from __future__ import annotations
import os
import shutil
from pathlib import Path
import pytest
from unsloth_pwsh_runner import run_pwsh
REPO_ROOT = Path(__file__).resolve().parents[2]
SETUP_PS1 = REPO_ROOT / "studio" / "setup.ps1"
_START = "$gitNeeded = ($env:STUDIO_LOCAL_INSTALL -eq '1')"
_TAIL = "if (-not $_localLlamaBuilt) {"
def _git_gate_block() -> str:
"""Slice the real $gitNeeded computation out of setup.ps1 so the test cannot drift."""
source = SETUP_PS1.read_text(encoding = "utf-8")
start = source.index(_START)
brace = source.index("{", source.index(_TAIL, start))
depth = 0
for index in range(brace, len(source)):
if source[index] == "{":
depth += 1
elif source[index] == "}":
depth -= 1
if depth == 0:
return source[start : index + 1]
raise AssertionError("Unclosed git gate block in setup.ps1")
def _function(name: str) -> str:
"""Inject the real helper the block calls; an undefined one is a silent no-op
under Continue, which would let the layout scan always report 'nothing built'."""
source = SETUP_PS1.read_text(encoding = "utf-8")
start = source.index(f"function {name} {{")
depth = 0
for index in range(source.index("{", start), len(source)):
if source[index] == "{":
depth += 1
elif source[index] == "}":
depth -= 1
if depth == 0:
return source[start : index + 1]
raise AssertionError(f"Unclosed function {name} in setup.ps1")
def _script() -> str:
return f"""
$DefaultLlamaPrForce = "0"
$DefaultLlamaSource = "https://github.com/ggml-org/llama.cpp"
$DefaultLlamaTag = "latest"
{_function("Test-AccessDeniedError")}
{_function("Get-PathState")}
function Exit-PathAccessDenied {{ param($Path, $Label, [switch]$UserSupplied) throw "denied: $Path" }}
{_git_gate_block()}
Write-Output $gitNeeded
"""
def _needs_git(env: dict[str, str]) -> bool:
merged = {k: v for k, v in os.environ.items() if not k.startswith(("UNSLOTH_", "STUDIO_"))}
merged.update(env)
# run_pwsh, not subprocess.run: every case in this file goes through here, so a pwsh
# that died at startup would come back as $gitNeeded computing the wrong answer for
# one environment. See tests/_shared/unsloth_pwsh_runner.py.
result = run_pwsh(
["pwsh", "-NoProfile", "-NonInteractive", "-Command", _script()],
check = True,
capture_output = True,
text = True,
env = merged,
)
return result.stdout.strip() == "True"
pwsh_only = pytest.mark.skipif(shutil.which("pwsh") is None, reason = "PowerShell is unavailable")
@pwsh_only
@pytest.mark.parametrize(
("env", "expected"),
[
# The consumer install: prebuilt wheels and a prebuilt llama.cpp, so no git.
({}, False),
# --local clones unsloth-zoo.
({"STUDIO_LOCAL_INSTALL": "1"}, True),
# Opt-in source builds clone llama.cpp.
({"UNSLOTH_LLAMA_FORCE_COMPILE": "1"}, True),
({"UNSLOTH_LLAMA_PR": "1234"}, True),
# PR_FORCE only forces a build for a positive integer.
({"UNSLOTH_LLAMA_PR_FORCE": "0"}, False),
({"UNSLOTH_LLAMA_PR_FORCE": "not-a-number"}, False),
({"UNSLOTH_LLAMA_PR_FORCE": "1234"}, True),
# "master" is a branch with no release, so Phase 4 always builds it from source.
({"UNSLOTH_LLAMA_TAG": "master"}, True),
# A release tag resolves to a prebuilt bundle.
({"UNSLOTH_LLAMA_TAG": "latest"}, False),
({"UNSLOTH_LLAMA_TAG": "b8635"}, False),
],
)
def test_git_is_required_only_for_local_and_source_builds(env, expected):
assert _needs_git(env) is expected
@pwsh_only
def test_a_built_local_llama_dir_drops_the_source_build_git_requirement(tmp_path):
(tmp_path / "llama-server.exe").write_text("", encoding = "utf-8")
env = {
"UNSLOTH_LOCAL_LLAMA_CPP_DIR": str(tmp_path),
"UNSLOTH_LLAMA_FORCE_COMPILE": "1",
}
# Reusing an existing binary skips both the prebuilt download and the source build.
assert _needs_git(env) is False
@pwsh_only
@pytest.mark.parametrize("trigger", ["UNSLOTH_LLAMA_FORCE_COMPILE", "UNSLOTH_LLAMA_PR"])
def test_an_unbuilt_local_llama_dir_still_requires_git(tmp_path, trigger):
# Nothing built at the canonical install location falls through to the normal install,
# so the source build still runs and still needs git. Suppressing the requirement here
# let a no-git host silently degrade to a prebuilt instead.
env = {
"UNSLOTH_LOCAL_LLAMA_CPP_DIR": str(tmp_path),
trigger: "1",
}
assert _needs_git(env) is True
@pwsh_only
def test_an_unbuilt_local_llama_dir_alone_does_not_require_git(tmp_path):
assert _needs_git({"UNSLOTH_LOCAL_LLAMA_CPP_DIR": str(tmp_path)}) is False