mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-16 04:13:54 +00:00
* CI: prove an interrupted install can never masquerade as a healthy one Nothing in CI had ever interrupted an install, which is how the reported failure shipped: quit the desktop app mid-install, the app SIGTERMs the installer process group, and if that lands during 'studio deps' the venv loses structlog. Preflight then probes 'unsloth -h' and 'studio desktop-capabilities', both of which succeed because the CLI's own deps are core, so the app reported ManagedReady with can_auto_repair=false while the backend died on import. A permanent dead end. This kills the installer at each interesting phase and asserts the result is either genuinely healthy or explicitly repairable, never silently ready. 13 legs across macos-14, ubuntu-latest and windows: each of the dependency-pass steps plus the coarse phases (venv, torch, unsloth, setup). The kill targets the process GROUP, matching install.rs. Killing only the leader leaves uv and python children to finish the dependency pass, and the test would quietly prove nothing. Windows has no process groups, so that leg walks the CIM parent links instead, which is the same reason the app carries windows_job.rs. One shared probe for all platforms. The Windows check used to be bespoke inline PowerShell that only ran -h and desktop-capabilities, so it could not observe studio_install_ok, verify-install or desktop-runtime-check: it would have reported FALSE_READY for the very PRs that add them, no matter how well they worked. The probe boots the backend as ground truth and owns the whole process tree, since terminating only the parent leaves children holding the port. install.sh runs with --local, which is load-bearing rather than a convenience: without it the installer resolves unsloth from PyPI and the venv gets the PUBLISHED CLI, so no branch-side change is present and every deeper probe reports 'absent' regardless of what the branch does. Verified: against a tree without the detection, windows kill@studio-deps reports FALSE_READY, reproducing the user report exactly. With #7492 merged the same leg reports REPAIRABLE, and all 13 legs pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the POSIX legs actually run the installer, and fail if they do not install.sh --tauri rejects a custom UNSLOTH_STUDIO_HOME outright (the desktop app still uses the legacy ~/.unsloth/studio root), and this workflow set one at workflow level for every job. So all 11 macOS and Linux legs exited about a second in with ERROR: UNSLOTH_STUDIO_HOME is not supported with --tauri. produced no CLI, took the probe's NO_CLI 'safe' branch and reported success. They were vacuously green. Only the two Windows legs were real, because install.ps1 has no equivalent guard. The override now applies to the Windows job only, and the POSIX legs read the legacy root, which is where --tauri installs. The runner is ephemeral so the real home is as disposable as the override. Also adds the check that makes this class of mistake loud: a leg asserts its kill actually landed on the marker it was aimed at, using the interrupt_reason the driver already records. A leg that never reached its kill point proves nothing, and NO_CLI made that indistinguishable from a pass. * Make the interrupted-install legs able to fail The probe treated a present .desktop-install-in-progress marker as proof of a repairable state, but the drivers seed it unconditionally and never clear it, so REPAIRABLE was unconditional and FALSE_READY unreachable. The Windows leg had no kill-landed guard, blanket continue-on-error, and no repair re-run; -SkipTorch was silently dropped, since install.ps1 parses only --no-torch. Judge the re-run by whether the backend boots, on both platforms. The log grep matched the frontend build printing "up to date" and failed a leg whose venv was fine. * Drop the interrupt cell that could never be interrupted install.sh --local sets skip_base, so install_python_stack returns before any "base packages" label is printed. The kill had nothing to land on and the installer ran to completion, reaching [TAURI:DONE] in 62s. * Fail the leg when the installer finished instead of being killed The driver set reason=marker-hit before the post-marker sleep and never rechecked, so a step whose work was already cached could run to completion inside that beat and still be recorded as an interruption. The landing assertion tests reason != marker-hit, so a fully completed install passed green having interrupted nothing. Reproduced with a stub that exits during the delay: reported marker-hit / killed=true / exit=0 next to "install finished fully". Set the reason after the sleep, on both drivers. Also trigger on _studio_deps.py and install_manifest.py, where the two decisions the probe asserts on are actually implemented. * Kill the group, and stop the probe blocking on a full pipe The escalation was gated on the leader still being alive, so a leader that exits promptly on SIGTERM while a uv or python descendant ignores it skipped the SIGKILL entirely, and wait reaped only the leader. Proven with a descendant that traps TERM: pre-fix its heartbeat keeps ticking while the probe would be running, post-fix it stops. Signal the group unconditionally and drain it after the reap, since an unreaped leader is still a member of its own group. The probe started the backend on stdout=PIPE and read nothing until after the poll loop, so a backend logging more than the pipe buffer during import blocked before binding. Measured 65536 bytes here; a child emitting 200 KB never reaches its bind line, which would make backend_ok false for a healthy install. Write straight to the artefact file. Also trigger on studio/backend/requirements/**, where structlog is declared. * Make the NO_CLI legs assert repair, and fix the Windows straggler sweep Two of the interrupted-install legs were passing without testing anything. The re-run assertion skipped verdict=NO_CLI, but a kill at "venv" or "torch" lands before install.sh ever prints "Installing Unsloth" (:2125, :3667, :3961), so those legs can only ever produce NO_CLI. Three non-gating-exempt cells (macos-14 kill@venv, macos-14 kill@torch, ubuntu-latest kill@torch) therefore asserted nothing beyond a marker appearing in a log. NO_CLI is now included: a re-run must produce a booting backend regardless of how little the first run managed to install. Each re-run step grows an existence check first, because the probe exits without writing verdict.json when the binary is absent and the json.load would crash rather than report. The Windows straggler sweep matched nothing at all. UNSLOTH_STUDIO_HOME arrives as D:\a\r\r/.studio-home, since the workflow joins ${{ github.workspace }} with a forward slash, while Process.Path is all backslashes, so the literal -like missed even the venv's own python.exe. uv is never under the studio home in any case: install.ps1 takes it from winget or astral.sh. Normalise the separators, match uv by name (the runner is ephemeral and runs no other uv), and skip the home comparison entirely when the variable is empty, which would otherwise turn the pattern into "**" and kill every python on the runner. * Run the Windows legs as the desktop does, and judge repair by what preflight reads The Windows matrix set a workspace-scoped UNSLOTH_STUDIO_HOME, which forces install.ps1 down the shell-install path: install.ps1:189-215 rejects a custom root under --tauri, so those legs ran with UNSLOTH_TAURI_MODE=0, the frontend build on and no bundled-file overlay, while the desktop always spawns the installer as --tauri with the variable scrubbed (install.rs:202 and :356). The torch leg could not even reach its marker: "Installing PyTorch" is printed only by Write-TauriLog (install.ps1:2440), so it was killed at the deadline. Both legs now run --tauri --local at the default root, and the probe and re-run resolve the CLI under %USERPROFILE%\.unsloth\studio. The probe counted `studio verify-install` and `studio desktop-runtime-check` failures as proof the app can repair, but preflight/managed.rs runs only `-h` and `studio desktop-capabilities --json` (:357) and reads studio_install_ok from that payload (:445); neither deeper command is invoked anywhere under studio/src-tauri. A leg where capabilities regressed to ready while only those standalone commands saw the damage would have passed green with the app stuck on ManagedReady, which is the exact false negative this workflow exists to catch. They are still run and recorded in verdict.json, just no longer repair evidence. An interrupted install can leave the console script in place while its venv interpreter is gone. The probes go through run(), which catches OSError, but the backend spawn did not, so the probe aborted before writing verdict.json and both workflows died on the json.load instead of reporting. That state is now recorded as backend_spawn_error and lands on REPAIRABLE, which is what `-h` failing already implies. On win32 the CLI re-spawns the server as a child and waits on it (unsloth_cli/commands/studio.py:1543), and CREATE_NEW_PROCESS_GROUP does not make terminate() reach descendants, so the reap left a server holding the venv open while the repair step reinstalled into files Windows had locked. Use taskkill /F /T for the tree. The straggler sweep now falls back to the default studio root, since under --tauri there is no UNSLOTH_STUDIO_HOME to match on. * Judge the install the way preflight does, and reap the whole probe group Read desktop-capabilities the way the desktop reads it. preflight/managed.rs pipes stdout and sends stderr to /dev/null (managed.rs:358), then hands the whole stdout buffer to serde_json (managed.rs:414). The probe concatenated both streams and scanned to the first brace, so a single diagnostic line on stderr made json.loads raise on the trailing text, studio_install_ok stayed "absent", and a broken backend was reported FALSE_READY over an install the real app parses, sees as incomplete, and offers to repair. That fails a valid recovery change for a reason that exists only in the probe. stdout and stderr are now captured separately and stdout is parsed strictly; a payload that does not parse counts as repair evidence, matching the Stale the desktop reports when the capability probe returns nothing (managed.rs:521). A booting backend alone is not a finished install. The manifest is written last (install_python_stack.py:3255), so a kill after "studio deps" but before it, the data-designer leg, leaves a venv whose backend boots while desktop-capabilities still reports studio_install_ok=false and preflight reports Stale (managed.rs:445). Calling that HEALTHY skipped the re-run step, so the leg asserted nothing beyond a marker appearing and never exercised the version fast path that is supposed to clear an incomplete install, which is the half of the bug that strands the user. HEALTHY now requires both. Escalate to the process group after reaping the probe's backend. reap() returned as soon as proc.wait() succeeded, and the leader exits promptly on SIGTERM while a uvicorn worker does not, so the SIGKILL iteration was skipped and that worker kept the port and the venv open while the repair step reinstalled underneath it. It also read os.getpgid(proc.pid) after the reap, which raises. The pgid is now captured up front and SIGKILL always goes to the group, the same escalation interrupt-install.sh:94 makes. A heartbeat experiment left the group alive with the old sequence and empty with the new one. Trigger the workflow on pyproject.toml. Every leg installs the checkout with --local, so that file decides the unsloth console script and the core dependencies the probe leans on: -h and desktop-capabilities only survive a torn install because typer/click/rich are declared there. No other install workflow interrupts the installer, so such a change would otherwise merge without a single leg running. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Judge an absent capability field and a dead -h the way preflight does The probe left studio_install_ok=absent undecided and judged those installs on whether the backend booted. preflight/managed.rs:445 tests studio_install_ok != Some(true), so an absent field is Stale exactly like a false one; a CLI too old to carry it is already rejected one check earlier on desktop_manageability_version. The gap mattered in both directions: a payload that stopped carrying the field reported HEALTHY on every booting leg and skipped the re-run assertion this workflow exists to make, and a torn venv with a working -h was failed as FALSE_READY even though the app would have offered repair. unsloth_cli/commands/studio.py is in this workflow's path filter precisely to catch that class of change, so it must not be the thing that silences it. The verdict also consulted cli_h_ok only in the repairable arm, so a CLI that cannot print help was called HEALTHY whenever the backend happened to boot. probe_managed_bin runs -h first and returns Stale cli_unusable before it ever reaches the capability probe (managed.rs:465-478), so that install goes to repair in the real app and the leg must assert it here. * Judge the probes on the desktop's deadline, and interrupt the host it uses Preflight gives each managed probe ten seconds and nothing more: managed.rs:337 wraps `unsloth -h` and managed.rs:390 wraps `studio desktop-capabilities --json` in a tokio timeout, kills the child on expiry, and returns Stale as "cli_unusable" or "desktop_capability_probe_failed". The probe allowed three minutes, so a venv torn badly enough that its CLI only answers after half a minute of retries was recorded HEALTHY here while the real app shows it as repairable. That skips the re-run assertion the leg exists to make, which is the same false-HEALTHY hole the studio_install_ok and -h gating already closed. Both calls now use the desktop's ten seconds, and the elapsed time is recorded so a leg that flips for timing reasons says so in the artefact. On Windows the installer child now runs where the desktop runs it. install.rs 325-339 spawns the bundled install.ps1 as powershell.exe with -NoLogo -NoProfile -NonInteractive -WindowStyle Hidden -ExecutionPolicy Bypass -File, so Windows PowerShell 5.1 is the only host a real desktop install ever uses. The interrupted run and the repair re-run both used pwsh 7, and every other Windows job in .github runs install.ps1 under pwsh too, so the installer's behaviour on 5.1 was covered by nothing: .NET Framework instead of .NET, OEM console encoding instead of UTF-8, and different native-command and OSArchitecture reporting are all real sources of divergence. A workflow whose point is to reproduce what the app does cannot run a different interpreter than the app does. The driver itself stays under pwsh; only the installer child and the repair invocation change. * Tighten the interrupted-install comments * Tighten the probe docstrings * Fail the leg when the installer completed inside the kill window * Land the kill in the marked step, and reject non-boolean capabilities Two holes found from the staging run's own logs. The venv leg never interrupted the venv step. Creating the venv takes ~0.1s, so by the time the 1s poll noticed its line the installer was already in "Installing PyTorch", and the flat 3s beat sent the signal there: staging run 30419729244 shows both step lines in the tail and a kill 4s in. That made the leg a duplicate of the torch leg while its label claimed otherwise. Both drivers now poll in half-second slices, cut the beat short the moment a later [TAURI:STEP] line appears, and print the step the signal actually landed in, warning when it is not the marked one. Sub-step markers such as "studio deps" print no step line, so they keep the whole beat and never warn. studio_install_ok is Option<bool> (managed.rs:43), so serde rejects a non-boolean and the whole payload fails to deserialize, which the desktop reports as Stale. bool() read a JSON string "false" as True, so the probe called a torn install ready. Only a literal JSON true counts now. * Tighten the interrupt driver comments * Fail the leg when the signal landed after the marked step The cut-short added last round only helps when the marked step is still the last [TAURI:STEP] line at the moment the poll notices it. Creating the venv takes ~0.1s (staging run 30419729244: 03:31:07.371 -> 07.478), less than the 0.5s poll, so the next step's line is usually already in the log when the marker matches, the step count never changes during the beat, and the full 3s elapses inside "Installing PyTorch". Reproduced with a stub installer against the driver at head: kill at 4.11s, step at kill "Installing PyTorch". The leg then duplicates the torch leg while its matrix label claims the venv step, and passed green on nothing but a :⚠️:. Both drivers now skip the beat entirely when the marked step is already over, so the kill goes out at once instead of deeper into the next step, and both record interrupt_step_mismatch in interrupt.env. The landing assertion fails on it: a warning that cannot fail the leg proves nothing. Sub-step markers ("studio deps", "pip bootstrap") print no step line of their own and stay exempt, as before. The venv leg becomes experimental. Its step is shorter than any log poll can resolve, so it must not block the PR on a race it cannot win, and it still probes the earliest torn state whenever it does land. * Tighten the interrupted-install workflow and driver comments * Land the kill in the phase each leg is named for Splitting the log on \r shows that 5 of the 12 legs of staging run 30419729244 interrupted a later phase than their label claims, and the run was fully green. The venv leg's install.log is byte-identical to the torch leg's. So is pip-bootstrap's to unsloth-extras'. Worse, both "studio deps" legs, the cells that reproduce the reported bug, were killed at "7/10 data designer deps" and "12/14 local plugin": their own probe artefacts report backend_ok=true, so structlog was installed and the flagship cell was passing on the manifest gate alone. Two causes. The dependency pass rewrites ONE physical line with \r (install_python_stack.py:2499), so its sub-steps are CR-separated segments and a line-based check could not see one end; the drivers exempted them and warned about nothing. And the flat 3s beat between the marker and the signal is longer than several phases, while every phase label prints BEFORE its work starts, so the beat pushed the signal past the phase instead of into it. Both drivers now split on \r, track the running phase at both levels, and judge a sub-step marker against the running sub-step and a step marker against the running step, so a step is not "over" because the sub-steps beneath it advanced. The beat defaults to 0 and is set per leg, 3s only for torch, unsloth and setup, which run for minutes. The mismatch is recorded in interrupt.env and the landing assertion fails on it, in both languages. Both detectors were replayed against the 12 real logs from 30419729244 and agree with the artefacts on every leg. venv and pip-bootstrap become experimental: their phases are shorter than any log poll can resolve. * Fail the Windows leg when the installer never exited The driver writes installer_exit=running when the installer outlived Stop-Tree and WaitForExit, and 'running' is not '0', so the landing assertion accepted it. A live installer writing into the venv while the probe reads it is not an interrupted install. Only a real integer exit code counts now, checked against 0, 143, 137, -1, running and the empty string. * Drop the legs that cannot land, and prove the kill was delivered Two cells never interrupted the phase they were named for. "Creating virtual environment" runs 0.107s (staging 30419729244, 03:31:07.371 -> 07.478) and "1/10 pip bootstrap" is over just as fast, both shorter than any poll that watches the log, so in 30423181897 and 30424366953 the signal landed in "Installing PyTorch" and "2/10 unsloth extras" every time. Each was another leg wearing a false label, so they are gone rather than allowed to fail, and continue-on-error goes with them: a leg permitted to fail asserts nothing. The only coverage lost is a venv caught half-written, which interruption cannot reach at this resolution; the torch leg's signal lands ~3s into a multi-minute download, so it already leaves a complete venv with nothing installed into it. The landing check also accepted an installer that failed on its own. A dependency error between the driver's last liveness check and the signal exits non-zero, which the exit != 0 guard let through as a kill. POSIX now requires 143 or 137, the only statuses a signal produces here and what all ten POSIX legs of 30424366953 reported. Recording whether kill(2) returned 0 would not separate them, since the unreaped leader keeps its group alive. Windows has no such status, so the driver records whether Stop-Process actually terminated the installer: it throws on a process already gone, so the flag is false exactly when there was nothing left to interrupt. * Signal at the marker, with no beat to overshoot the phase Staging run 30426111484 failed the macOS torch leg on the landing check: the 3s beat carried the signal from "Installing PyTorch" into "Installing Unsloth", because the PyTorch step, which this workflow called minutes long, finished in under three seconds. The beat only ever existed to land mid-work, and it cannot do that safely: every label prints before its work starts, so detection is already inside the phase, and any wait is a bet on how long that phase runs. It lost in 30419729244 and again here. So the beat is gone rather than retuned, and with it the matrix knob and the driver parameter on both platforms. The landing check stays and can still fail, since a phase shorter than one poll is seen only after it ends. The Windows driver also polled every 500ms while its own comment claimed a fifth of a second. That is 2.5 slices of overshoot the POSIX side does not carry, and it is now 200ms like the POSIX loop. * Kill the installer before its children, not after The depth-first walk killed the child install.ps1 was waiting on and only then the root, which races the leader's own reaction to that death. It is not a theoretical race: in staging run 30424366953 install.ps1 had already printed "unsloth studio setup failed (exit code -1)" by the time Stop-Process reached it. A leader that wins the race makes Stop-Process throw, and the new root-kill assertion would then fail a leg whose interruption the driver really did deliver. The tree is now snapshotted first, since a dead parent leaves nothing to walk, then the root goes down ahead of its descendants. A dead leader cannot react to a child and cannot respawn one either, which is what the depth-first order was for. * CI: give the probe the desktop's startup grace and fail a nonzero repair The probe allowed the backend 120s to answer /api/health while the desktop waits 5 minutes (BACKEND_STARTUP_GRACE_PERIOD, commands.rs:9), so a slow but healthy install could be reported FALSE_READY. A broken backend exits at once and the poll breaks on it, so the longer deadline only bounds a live backend. The re-run step also accepted a HEALTHY probe over an installer that exited nonzero. setup.sh does fallible sidecar and GPU setup after the manifest is written, and the desktop returns the repair error without starting the backend (commands.rs:615-630). The Windows leg ignored powershell.exe's status entirely. * Tighten the interrupted-install comments Comments only: shorter wording for the same rationale, no code touched. * CI: raise the interrupted-install job timeout above its own deadlines A leg configures up to 25 minutes to the marker plus two probe passes of up to 17 minutes each around a repair install, so the 60 minute limit could cancel a slow runner mid-assertion. Legs land in 6 to 10 minutes in practice. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothai@gmail.com>
209 lines
10 KiB
PowerShell
209 lines
10 KiB
PowerShell
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
|
|
#
|
|
# Windows counterpart of interrupt-install.sh: run install.ps1 and kill it partway
|
|
# through, reproducing a user quitting the desktop app mid-install.
|
|
#
|
|
# Windows has no process groups (hence the app's windows_job.rs), so this kills the whole
|
|
# process TREE: killing the leader alone leaves uv/python children to finish the dep pass.
|
|
#
|
|
# Usage:
|
|
# pwsh -File .github/scripts/interrupt-install.ps1 -Marker 'studio deps' `
|
|
# -LogPath logs/install.log -InstallArgs '--tauri --no-torch --local'
|
|
[CmdletBinding()]
|
|
param(
|
|
[string]$Marker = '',
|
|
[string]$LogPath = 'logs/install.log',
|
|
[string]$InstallArgs = '',
|
|
[int]$KillAtSeconds = 900
|
|
)
|
|
|
|
$ErrorActionPreference = 'Continue'
|
|
New-Item -ItemType Directory -Force -Path (Split-Path -Parent $LogPath) | Out-Null
|
|
Set-Content -Path $LogPath -Value '' -Encoding utf8
|
|
|
|
# Stand in for the desktop app, which writes this before spawning the installer
|
|
# (install.rs). We kill install.ps1 directly, so without it #7490's marker is absent for an
|
|
# unrelated reason, exactly what the Windows legs reported. Both roots: Rust hardcodes
|
|
# ~/.unsloth/studio, CI overrides UNSLOTH_STUDIO_HOME. Never cleared, by design.
|
|
foreach ($dir in @($env:UNSLOTH_STUDIO_HOME, (Join-Path $HOME '.unsloth\studio'))) {
|
|
if ([string]::IsNullOrWhiteSpace($dir)) { continue }
|
|
try {
|
|
New-Item -ItemType Directory -Force -Path $dir -ErrorAction Stop | Out-Null
|
|
Set-Content -Path (Join-Path $dir '.desktop-install-in-progress') -Value '' -ErrorAction Stop
|
|
} catch { Write-Host "[interrupt] could not seed install marker in ${dir}: $_" }
|
|
}
|
|
|
|
# Its own host, so stdout can be redirected to the log while we poll. That host is WINDOWS
|
|
# PowerShell 5.1, not pwsh, with install.rs:325-339's exact flags: the only host a real
|
|
# desktop install uses, while every other Windows job in .github runs install.ps1 under
|
|
# pwsh 7, leaving 5.1 behaviour (.NET Framework, OEM/ANSI console encoding, different
|
|
# native-command and OSArchitecture reporting) covered by nothing. Only the installer child
|
|
# and the repair re-run change host; the driver stays under pwsh.
|
|
$argList = @(
|
|
'-NoLogo', '-NoProfile', '-NonInteractive',
|
|
'-WindowStyle', 'Hidden',
|
|
'-ExecutionPolicy', 'Bypass',
|
|
'-File', 'install.ps1'
|
|
)
|
|
if ($InstallArgs) { $argList += $InstallArgs.Split(' ') }
|
|
$proc = Start-Process -FilePath 'powershell.exe' -ArgumentList $argList `
|
|
-RedirectStandardOutput $LogPath -RedirectStandardError "$LogPath.err" `
|
|
-PassThru -NoNewWindow
|
|
Write-Host "[interrupt] installer pid=$($proc.Id) marker='$Marker' deadline=${KillAtSeconds}s"
|
|
|
|
# Proof that the signal was DELIVERED, not merely attempted. The installer can fail on its
|
|
# own between the last HasExited check and Stop-Tree, and a natural failure carries a
|
|
# non-zero exit code just like a kill does, so the exit status alone cannot separate the two
|
|
# on Windows. Stop-Process throws on a process already gone, so this flag is false exactly
|
|
# when there was nothing left to interrupt.
|
|
$script:rootKilled = $false
|
|
|
|
function Get-Descendants([int]$RootId) {
|
|
# Depth-first, deepest first. CIM gives the parent link Windows has no process groups for.
|
|
$ids = @()
|
|
foreach ($k in @(Get-CimInstance Win32_Process -Filter "ParentProcessId=$RootId" -ErrorAction SilentlyContinue)) {
|
|
$kid = [int]$k.ProcessId
|
|
$ids += Get-Descendants $kid
|
|
$ids += $kid
|
|
}
|
|
return $ids
|
|
}
|
|
|
|
function Stop-Tree([int]$RootId) {
|
|
# Snapshot the whole tree BEFORE killing anything: once a parent is gone its children are
|
|
# orphaned with no ParentProcessId left to walk, so the walk has to happen first.
|
|
$descendants = @(Get-Descendants $RootId)
|
|
# Then the ROOT, ahead of its children. install.ps1 watches the child it waits on: in
|
|
# staging run 30424366953 it had already printed "unsloth studio setup failed (exit code
|
|
# -1)" by the time Stop-Process reached it. Killing children first races the leader's own
|
|
# exit, and a leader that wins makes Stop-Process throw over an interruption the driver
|
|
# did deliver, failing the leg for nothing. Dead first, it can neither react nor respawn
|
|
# what we are about to kill.
|
|
try {
|
|
Stop-Process -Id $RootId -Force -ErrorAction Stop
|
|
Write-Host "[interrupt] killed installer pid=$RootId"
|
|
if ($RootId -eq $proc.Id) { $script:rootKilled = $true }
|
|
}
|
|
catch { if ($RootId -eq $proc.Id) { Write-Host "[interrupt] installer pid=$RootId was already gone: $_" } }
|
|
foreach ($id in $descendants) {
|
|
try { Stop-Process -Id $id -Force -ErrorAction Stop; Write-Host "[interrupt] killed pid=$id" }
|
|
catch { }
|
|
}
|
|
}
|
|
|
|
# A leg can be aimed at either kind of phase, and only one of them is a line. install.ps1
|
|
# prints "[TAURI:STEP] <name>" lines, while the dependency pass rewrites ONE physical line
|
|
# with \r (install_python_stack.py:2499), so its sub-steps are CR-separated SEGMENTS.
|
|
# Splitting on \r is what makes a sub-step's END observable at all.
|
|
$SubRe = '\[[=-]+\]\s*\d+/\d+\s'
|
|
|
|
function Get-PhaseLines([string]$Path) {
|
|
$raw = Get-Content -Path $Path -Raw -ErrorAction SilentlyContinue
|
|
if (-not $raw) { return @() }
|
|
return @(($raw -replace "`r", "`n") -split "`n")
|
|
}
|
|
|
|
function Get-LastPhase([string]$Path) {
|
|
$p = @(Get-PhaseLines $Path | Where-Object { $_ -match '^\[TAURI:STEP\]' -or $_ -match $SubRe })
|
|
if ($p.Count) { return $p[-1] }
|
|
return ''
|
|
}
|
|
|
|
# True when the phase the marker named is no longer the running one. A sub-step marker is
|
|
# judged against the running sub-step, a step marker against the running step -- a step is
|
|
# not "over" because the sub-steps beneath it advanced.
|
|
function Test-MarkedPhaseOver {
|
|
if (-not $Marker) { return $false }
|
|
$lines = @(Get-PhaseLines $LogPath)
|
|
$subs = @($lines | Where-Object { $_ -match $SubRe })
|
|
if ($subs | Where-Object { $_ -match $Marker }) {
|
|
$last = Get-LastPhase $LogPath
|
|
return -not ($last -match $SubRe -and $last -match $Marker)
|
|
}
|
|
$steps = @($lines | Where-Object { $_ -match '^\[TAURI:STEP\]' })
|
|
if ($steps | Where-Object { $_ -match $Marker }) {
|
|
return ($steps[-1] -notmatch $Marker)
|
|
}
|
|
return $false
|
|
}
|
|
|
|
$killed = $false
|
|
$reason = ''
|
|
# Fifth-of-a-second slices, matching the POSIX driver: every phase label prints BEFORE its
|
|
# work, so this delay IS the whole distance between the label and the signal and the only
|
|
# thing that can push the kill past the end of a short phase. It slept 500ms while claiming
|
|
# a fifth, carrying 2.5 slices of overshoot the POSIX side does not.
|
|
for ($i = 0; $i -lt ($KillAtSeconds * 5); $i++) {
|
|
if ($proc.HasExited) { $reason = 'exited-before-marker'; break }
|
|
if ($Marker) {
|
|
$hit = Select-String -Path $LogPath -Pattern $Marker -SimpleMatch:$false -ErrorAction SilentlyContinue
|
|
if ($hit) {
|
|
# Same as the POSIX driver: signal at detection, never after a delay. The label
|
|
# prints before the work, so the kill is inside the phase the moment the line appears,
|
|
# and any wait is a bet on the phase outlasting it that staging runs 30419729244 and
|
|
# 30426111484 both lost.
|
|
#
|
|
# The installer can still exit on its own between the match and the signal, which
|
|
# would record marker-hit over an install that interrupted nothing.
|
|
if ($proc.HasExited) { $reason = 'exited-before-signal'; break }
|
|
$reason = 'marker-hit'
|
|
$killed = $true
|
|
break
|
|
}
|
|
}
|
|
Start-Sleep -Milliseconds 200
|
|
}
|
|
if (-not $killed -and -not $proc.HasExited) { if (-not $reason) { $reason = 'deadline' }; $killed = $true }
|
|
|
|
if ($killed) {
|
|
Write-Host "[interrupt] killing process tree of $($proc.Id) ($reason)"
|
|
Stop-Tree $proc.Id
|
|
# Any straggler uv/python that reparented away from the installer. The old sweep matched
|
|
# nothing: UNSLOTH_STUDIO_HOME arrives as `D:\a\r\r/.studio-home` (github.workspace joined
|
|
# with a forward slash) while Process.Path is all backslashes, so the literal -like missed
|
|
# even the venv's own python -- hence the separator normalisation, and uv by name (it
|
|
# lives outside the studio home and the ephemeral runner has no other uv). Under --tauri
|
|
# there is no UNSLOTH_STUDIO_HOME, so fall back to install.ps1's root.
|
|
$studioRoot = if ([string]::IsNullOrWhiteSpace($env:UNSLOTH_STUDIO_HOME)) { Join-Path $HOME '.unsloth\studio' }
|
|
else { $env:UNSLOTH_STUDIO_HOME }
|
|
$homeNorm = if ([string]::IsNullOrWhiteSpace($studioRoot)) { $null }
|
|
else { ($studioRoot -replace '/', '\').TrimEnd('\') }
|
|
foreach ($p in @(Get-Process -Name 'uv', 'python', 'pythonw' -ErrorAction SilentlyContinue)) {
|
|
$path = $null
|
|
try { $path = $p.Path } catch { }
|
|
$inHome = $homeNorm -and $path -and ($path -like "$homeNorm\*")
|
|
if ($p.ProcessName -eq 'uv' -or $inHome) {
|
|
try { Stop-Process -Id $p.Id -Force; Write-Host "[interrupt] swept $($p.ProcessName) pid=$($p.Id)" } catch { }
|
|
}
|
|
}
|
|
}
|
|
|
|
try { $proc.WaitForExit(30000) | Out-Null } catch { }
|
|
$rc = if ($proc.HasExited) { $proc.ExitCode } else { 'running' }
|
|
Write-Host "[interrupt] installer exit=$rc reason=$reason killed=$killed root_killed=$($script:rootKilled)"
|
|
Write-Host '[interrupt] last log lines:'
|
|
Get-Content $LogPath -Tail 15 -ErrorAction SilentlyContinue
|
|
|
|
if ($Marker -and -not (Select-String -Path $LogPath -Pattern $Marker -ErrorAction SilentlyContinue)) {
|
|
Write-Host "::warning::marker '$Marker' never appeared -- killed at the deadline, not the intended step"
|
|
}
|
|
# Where the signal actually landed. A phase that ended before the poll saw the marker sends
|
|
# the kill into a LATER phase, so the leg duplicates whichever leg owns that phase while its
|
|
# own label claims otherwise.
|
|
$lastPhase = Get-LastPhase $LogPath
|
|
Write-Host "[interrupt] phase at kill: $lastPhase"
|
|
$mismatch = Test-MarkedPhaseOver
|
|
if ($mismatch) {
|
|
Write-Host "::warning::killed in '$lastPhase', not the marked phase -- that phase was already over"
|
|
}
|
|
# Lower-cased so the workflow compares it the same way on every platform, and only simple
|
|
# values: the POSIX side sources this file.
|
|
@(
|
|
"interrupt_reason=$reason"
|
|
"interrupt_killed=$killed"
|
|
"interrupt_root_killed=$(if ($script:rootKilled) { 'true' } else { 'false' })"
|
|
"installer_exit=$rc"
|
|
"interrupt_phase_mismatch=$(if ($mismatch) { 'true' } else { 'false' })"
|
|
) | Set-Content -Path (Join-Path (Split-Path -Parent $LogPath) 'interrupt.env') -Encoding utf8
|
|
exit 0
|