mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-25 08:42:25 +00:00
827d25931b
* Stop 19 test files racing on one PowerShell startup cache
Backend CI run 32341628757 on `1c3dde199` finished `284 failed, 8498 passed`. Every
one of the 284 was a pwsh subprocess ending `died with <Signals.SIGABRT: 6>`, across
19 files that all read as Windows-installer regressions. None of them were. 222 of
the aborts land inside a two-second window, 88 at 07:09:30 and 133 at 07:09:31,
which is a mass kill of every live pwsh rather than independent per-test flakiness.
The cause
------------------------------------------------------------------------
Every `-NonInteractive` startup reads and rewrites an ~83 KB
`$XDG_CACHE_HOME/powershell/StartupProfileData-NonInteractive`, and XDG_CACHE_HOME
defaults to `$HOME/.cache`. Under `-n 4` all four xdist workers share one HOME, so
the whole job's pwsh processes race on one file and a startup that deserialises a
half-written one dies before it reaches our script. `Stack overflow.` is .NET's
failfast, which cannot unwind a blown stack, so it prints one line and calls
abort(); that is the SIGABRT (PowerShell/PowerShell#24461).
Measured twice, independently, 4000 startups per arm:
run 1 shared cache dir 7/4000 died {-11: 3, -6: 4}
private cache dirs 0/4000
run 2 shared cache dir 11/4000 died {-11: 10, -6: 1}
private cache dirs 0/4000
Three distinct crash shapes appeared, and each names the torn file rather than our
scripts: `Stack overflow.`, `System.IO.FileLoadException: The given assembly name`,
and `System.ArgumentException: String cannot have zero length.` 18 deaths in 8000
shared startups, 0 in 8000 private.
CI agrees from the other direction. Of the pwsh-heavy files in that run, exactly one
had zero failures, tests/test_windows_amd_gpu_scan_fallback.py, and it is the only
one that hands its child a private HOME, across roughly 80 startups where the run's
own rate predicts about 16 failures.
What is NOT established
------------------------------------------------------------------------
Neither experiment reproduces CI's rate. Roughly 20% of pwsh startups died there
against 0.2 to 0.3% here, and at CI's actual `-n 4` on this box I measured 0/1200 in
both arms: the race needed 48-way concurrency before it appeared at all. The likely
reason is that four workers on a 4-core runner are in real contention while four
threads on a 192-core box almost never overlap in the critical section, but that is
reasoning and not a measurement, so treat the mechanism as proven and the magnitude
as unexplained. That is also why this does not stop at removing the shared file.
Three layers, in order
------------------------------------------------------------------------
1. Remove the contended resource. One cache directory per xdist worker, fresh per
session. Workers run their tests one at a time, so within a worker the startups
stay sequential and the cache still does its job warm; across workers the
directories are disjoint and there is nothing left to race on. Fresh rather than a
stable path, because a cache torn by an earlier run would otherwise poison every
later session on the same box.
2. Retry a run that produced no verdict. Three attempts, unslept, because the trigger
is process startup rather than a resource that frees up.
3. Attribute what is left. A crash raises PwshInterpreterCrash naming the interpreter.
Layer 1 is the fix; 2 and 3 exist because of the unexplained magnitude above.
Deliberately NOT done: bounding pwsh concurrency with a lock, or giving up `-n 4`.
The workflow records 806.1s to 219.7s from that flag, and the contended resource can
be removed rather than rationed.
The rule that keeps this honest
------------------------------------------------------------------------
A signal is not a verdict, so retrying it papers over nothing: the script never ran
to its end. A normal exit is returned untouched on the first attempt whatever its
code, so a pwsh that runs and gives the WRONG answer still fails with its own
message. Getting that second half wrong would turn this into a way to retry real
regressions into green, which is worse than the bug it fixes, so both directions are
executed in tests/studio/test_pwsh_interpreter_crash_attribution.py against a real
SIGABRT rather than reviewed.
Mutation-tested: relaxing the crash test from `returncode < 0` to `returncode != 0`
fails test_a_clean_run_with_the_wrong_answer_still_fails_with_its_own_message and
test_a_clean_run_is_not_retried, which are exactly the two that guard that direction.
This also generalises `_run_pwsh` from tests/studio/test_install_phase_timing.py,
added earlier today for a second, signal-free shape: pwsh printing its "The
PowerShell process will exit" banner and exiting normally with empty stdout. That one
cannot be seen in the exit status, so it stays a text match.
Verified
------------------------------------------------------------------------
tests/python/test_windows_xformers_installer.py, tests/studio/test_install_phase_timing.py,
tests/studio/install/ and the new guard: 2635 passed, 3 skipped.
Guard alone: 5 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop the subprocess import the pwsh conversion left behind
Source lint's import-hoist check is right: every subprocess.run in
test_windows_xformers_installer.py became run_pwsh, so `import subprocess` has no
references left except the one inside a comment explaining why run_pwsh is used
instead. Its wording names the shape exactly -- "was used before, now unused
(references re-pointed)" -- which is what a mechanical call-site rewrite leaves
behind.
Swept the other 18 converted files the same way with an AST pass rather than by
eye. This was the only real one: the remaining hits are `from __future__ import
annotations`, which every such scan reports, and a PropertyMock in
test_rocm_support.py that is present on main unchanged.
42 passed.
* Suppress the core dump on the forged SIGABRT
tests/test_deliberate_crashes_suppress_cores.py caught this: the abort child had no
PR_SET_DUMPABLE=0, so each of these aborts piped a multi-MB core to apport before the
child could be reaped. The guard is right and its message names the fix.
The child still exits -6 and PR_GET_DUMPABLE reads 0, so all five verdicts are
unchanged. Linux-only and non-fatal elsewhere: Windows has no CDLL(None) and pipes no
core, so arming it there would trade a no-op for a lost test.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
98 lines
3.8 KiB
Python
98 lines
3.8 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
|
|
|
|
"""The direct VC++ runtime download must negotiate TLS 1.2 and run only Microsoft's binary."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import shutil
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from unsloth_pwsh_runner import run_pwsh
|
|
|
|
|
|
REPO_ROOT = Path(__file__).resolve().parents[2]
|
|
SETUP_PS1 = REPO_ROOT / "studio" / "setup.ps1"
|
|
|
|
_START = '$url = "https://aka.ms/vs/17/release/vc_redist.x64.exe"'
|
|
_END = "Remove-Item -LiteralPath $dst -Force -ErrorAction SilentlyContinue\n }"
|
|
|
|
|
|
def _download_block() -> str:
|
|
"""Slice the real download block out of setup.ps1 so the test cannot drift."""
|
|
source = SETUP_PS1.read_text(encoding = "utf-8")
|
|
start = source.index(_START)
|
|
end = source.index(_END, start) + len(_END)
|
|
return source[start:end]
|
|
|
|
|
|
def test_the_download_is_verified_as_microsoft_signed_before_it_runs():
|
|
# No pwsh needed: Get-AuthenticodeSignature is Windows-only, so the ordering of the three
|
|
# steps in the real block is the thing to hold still. A verification placed after
|
|
# Start-Process, or one that only checks Status, would still "pass" on a swapped binary.
|
|
block = _download_block()
|
|
download = block.index("Invoke-WebRequest")
|
|
verify = block.index("Get-AuthenticodeSignature", download)
|
|
execute = block.index("Start-Process", verify)
|
|
assert download < verify < execute
|
|
assert "SignatureStatus]::Valid" in block
|
|
# Loose on the quoting, since an RDN value may arrive quoted, strict on the publisher.
|
|
assert "Microsoft Corporation" in block
|
|
|
|
|
|
def _script(starting_protocol: str) -> str:
|
|
# Start from a non-zero set that lacks Tls12. Tls13 is the only such value modern .NET
|
|
# accepts, and it stands in for the legacy Ssl3/Tls default of Windows PowerShell 5.1.
|
|
return f"""
|
|
function substep {{ param($a, $b) }}
|
|
function Refresh-Environment {{ }}
|
|
function Invoke-WebRequest {{
|
|
param($Uri, $OutFile, [switch]$UseBasicParsing, $TimeoutSec)
|
|
Write-Output "DURING=$([System.Net.ServicePointManager]::SecurityProtocol)"
|
|
throw "stop before Start-Process"
|
|
}}
|
|
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::{starting_protocol}
|
|
{_download_block()}
|
|
Write-Output "AFTER=$([System.Net.ServicePointManager]::SecurityProtocol)"
|
|
"""
|
|
|
|
|
|
def _run(starting_protocol: str) -> dict[str, str]:
|
|
# The TLS assertions read the BEFORE/DURING/AFTER lines this script prints, so an
|
|
# interpreter that never got as far as running the download block would look like
|
|
# setup.ps1 failing to negotiate TLS 1.2 at all.
|
|
result = run_pwsh(
|
|
["pwsh", "-NoProfile", "-NonInteractive", "-Command", _script(starting_protocol)],
|
|
check = True,
|
|
capture_output = True,
|
|
text = True,
|
|
)
|
|
out = {}
|
|
for line in result.stdout.splitlines():
|
|
if "=" in line:
|
|
key, _, value = line.partition("=")
|
|
out[key.strip()] = value.strip()
|
|
return out
|
|
|
|
|
|
pwsh_only = pytest.mark.skipif(shutil.which("pwsh") is None, reason = "PowerShell is unavailable")
|
|
|
|
|
|
@pwsh_only
|
|
def test_tls12_is_added_for_the_download_and_restored_after():
|
|
seen = _run("Tls13")
|
|
during = {part.strip() for part in seen["DURING"].split(",")}
|
|
assert "Tls12" in during, "the download must negotiate TLS 1.2 or aka.ms refuses it"
|
|
assert "Tls13" in during, "adding TLS 1.2 must not drop protocols the host already allowed"
|
|
assert seen["AFTER"] == "Tls13", "the process-wide protocol must be restored"
|
|
|
|
|
|
@pwsh_only
|
|
def test_system_default_is_left_alone():
|
|
# SystemDefault means "let the OS choose" and already covers TLS 1.2+; pinning it to
|
|
# Tls12 would strip TLS 1.3 from every later request in the process.
|
|
seen = _run("SystemDefault")
|
|
assert seen["DURING"] == "SystemDefault"
|
|
assert seen["AFTER"] == "SystemDefault"
|