37 commits

Author	SHA1	Message	Date
Pascal André	489fab4a71	Support OpenCode V2 in unsloth start (#9275) ... * Support OpenCode V2 in unsloth start * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep stable OpenCode guide coverage * Follow the OpenCode V2 stable release * Honor OpenCode V2 policy and server semantics * Fix OpenCode V2 launch command recipes --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-20 04:42:41 +02:00
Daniel Han	6e57736ad9	Stop blaming the recipe for a turn that never came back (#9266) ... * Stop blaming the recipe for a turn that never came back `connection (codex)` fails on main and on unrelated branches, and the two lines it prints contradict each other: ##[warning] ... judging the turn on its assertions instead of calling it guide drift. ##[error] [guide drift] agent=codex: the documented launch command exited non-zero (rc=124) ... so the documented flow in start.py drifted. Neither is right. run_timed printed the warning unconditionally, but three callers (connection, resume, attribution-ab) have no assertion that can rescue a partial turn and treat a cap as fatal on purpose. So it promised the opposite of what was about to happen, for exactly the callers most likely to hit it. It now states the fact and leaves the verdict to the caller. The error is worse, because it sends the reader to the one file that is not implicated. The transcript shows codex launching perfectly -- correct provider, correct model, correct profile -- and then sitting on ERROR: Reconnecting... 1/5 for the full 600s. Nothing about the documented flow had drifted; the model server never answered. A cap means the launch command was fine and the turn never came back, which is a different failure with a different owner. Still fatal. Waiving a cap here would report "connection OK" for a recipe that printed a banner and then blocked on a headless prompt, which is the failure this job exists to catch, and assert_reply cannot tell a finished reply from a startup banner. Only the attribution changes. A non-zero exit from the launch command is still reported as drift, which is the case guide_fail is right about. Four static guards, since the script needs five agent CLIs and a live model server to run. All four mutants (run_timed speaking for its callers again, the timeout routed back through guide_fail, the timeout waived instead of fatal, a non-zero exit no longer reporting drift) were reintroduced and confirmed red. Wired into workflow-trigger-lint, the only job with no paths filter. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the waiver guard enforce its own sentence, not a magic number The shell suite went red on this branch: 'exactly the expected TIMED_OUT sites (got 4, want 3)'. Worth deciding which of the two was wrong rather than moving the number. The heading is 'only file-edit turn 1 rescues a soft timeout', and that is the invariant worth having. The assertion under it counted every consultation of TIMED_OUT and pinned the total at 3 -- so it conflated the single waiver with the fatal checks beside it, and its own comment admitted as much ('turn 1's guard, PLUS the resume and attribution-ab fatal checks'). The fourth site this branch adds makes a cap MORE explicitly fatal at connection. Failing a guard named for waivers because a fatal check was added is the assertion being over-specified, not the change being wrong. The neighbouring semantic assertions agree: 'connection guard has no TIMED_OUT escape' passes. So the two shapes are counted separately. The waiver is the || form and is the only one that lets execution continue past a cap; everything else consults TIMED_OUT to stop. That enforces the sentence rather than a number, and keeps all the power: a waiver added anywhere still fails. Mutation-tested against the repaired guard -- a waiver added to the connection guard, resume's fatal check turned into a waiver, and the connection check deleted outright are all caught. 44 pass, 0 fail. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-19 03:21:20 -07:00
Daniel Han	fdf83cbd6a	Make apt fail fast, so the retry has something to retry (#9260) ... * Make apt fail fast, so the retry has something to retry #9256 bounded the stalls. This says why they happened and stops them. Reading the four logs, every one ends the same way: 04:47:02 Get:5 https://archive.ubuntu.com/ubuntu noble-security InRelease [126 kB] 05:16:29 ##[error]The operation was canceled. Twenty-nine minutes of silence mid-fetch of a 126 kB index file, and two of the four hung on that same file. What came before it is the other half: azure.archive.ubuntu.com was Ign'd four times over thirty seconds, so apt had already failed over through /etc/apt/apt-mirrors.txt to the public archive, which is not provisioned for this fleet. apt did not treat any of that as an error. Acquire::http::Timeout defaults to 120s and is an idle timeout, so a socket that is open and trickling never trips it. Cutting it to 20s with internal retries turns a 29 minute hang into a ~1 minute failure, and that matters beyond speed: the wall-clock kill is what orphans the dpkg lock, so an apt that gives up on its own is one we never have to kill. Not pinning a mirror. The evidence does not support it: in one stall Azure was dead and the public archive hung, in another Azure was serving fine and the transfer stalled at a 13.6 MB package. Neither is reliably better, and the mirrorlist failover is already the right mechanism. It just needed to be allowed to give up. The written config is validated against apt itself (apt-config dump round-trips all four options), and the helper's retry, timeout, giving up, recovery and no-command paths are exercised end to end. * Wait on all four apt locks, and stop racing a slow mirror Two bugs, both mine, both caught by CI on this branch. 1. apt-get update takes /var/lib/apt/lists/lock and nothing else. The helper waited on /var/lib/dpkg/lock-frontend only, so after an attempt was killed mid-update the wait saw a free lock, retried immediately, and produced E: Could not get lock /var/lib/apt/lists/lock. It is held by process 2420 (apt-get) twice in under two seconds. Three attempts, one real one -- exactly the 'a retry that cannot succeed is worse than none' failure the header warns about, in the one case it was written for. It now waits on all four: dpkg's two, lists, and archives. 2. 150s per attempt was racing the mirror, not bounding it. The mirror was degraded rather than dead, so every attempt got killed at the same point and none finished. Two attempts of 360s instead of three of 150: the bound exists to stop an infinite hang, not to beat a slow server. studio-update-smoke's job budget goes 15 to 25 to fit the new worst case; the guard checks that arithmetic. * Ask the image's package lists first, and refresh only if they miss The mirror outage is still live. From this branch's own GGUF smoke: 06:55:33 Ign:4 http://azure.archive.ubuntu.com/ubuntu noble-backports InRelease 06:55:35 Get:5 https://archive.ubuntu.com/ubuntu noble-security InRelease [126 kB] 07:01:11 (nothing, for 5m36s, then the attempt's cap) Azure is unreachable and the public archive it fails over to is not provisioned for this fleet. Acquire::http::Timeout does not save us there: it is an idle timeout, and a server dribbling a byte every few seconds never trips it, so the wall-clock bound is what ends the attempt. The bound is working as intended -- the step now fails in 14 minutes with the reason printed, where before it spent 30 and was reported as "cancelled" with nothing said. But a bounded failure is still a failure, and the operation that fails is one we mostly do not need. `apt-get update` refreshes every index for every suite. The runner image already ships populated lists, and these steps install a handful of ordinary packages. So try the install first, and refresh only when it misses: apt-get install -y X || { apt-get update && apt-get install -y X; } In the common case the slowest and most failure-prone apt operation is never performed at all. When the image's lists really are too stale -- the "Unable to locate package" case the runner-images maintainers warn about -- the update still runs and the install is retried, so nothing is traded away. Not applied to clean-machine-install-ci: a fresh WSL image genuinely has no lists, so install-first would always miss, and a bare machine is that leg's whole premise. The three outcomes were checked as shell rather than reasoned about: resolvable (update never reached, rc=0), not resolvable (update runs, rc=0), and both failing (rc=1, so the failure still propagates under the `bash -e` GitHub runs steps with).	2026-08-19 00:19:50 -07:00
Daniel Han	cabed07f95	Bound every apt step in CI, and fix the retry that never ran (#9256) ... * Give apt one bounded, retrying entry point, and fix the retry that never ran Two problems, one of them mine. The retry I added to the Playwright install last round was dead on arrival. GitHub runs `run:` blocks as `bash -e`, and I wrote the loop as timeout --signal=TERM --kill-after=30 480 python -m playwright "$@" rc=$? A bare failing command under -e aborts the step then and there, so the second attempt never ran. The step exited 124 having printed none of its own warnings -- indistinguishable from the unbounded stall it was meant to replace, which is exactly how it read on #9251's indicator and banner shards, both dying at 8m05s. The previous version used `if timeout ...; then` and was exempt from -e; changing it to read the exit code is what broke it. Reproduced both forms under `bash -e`. The second is broader. `Linux deps` is a bare `apt-get update && apt-get install` with no bound at all, and on #9251 it sat on the Azure mirror for 28 minutes. Of 24 apt-invoking steps across 13 workflows, 2 were bounded. An unbounded apt step is not slow, it is silent: it spends the job's whole budget, GitHub scores the result as "cancelled" rather than a failure, prints no reason, and skips every step after it. .github/scripts/retry-with-apt-lock.sh is now the single definition: per-attempt timeout, retry, dpkg-lock wait between attempts, and exit codes reported honestly (124/137 as a stall, anything else as the status it actually returned). It carries no `set -e` and uses `|| rc=$?`, so it works whatever the caller set. Six paths exercised locally, including the regression above: called from a `bash -e` step, both attempts still run. Sizing respects what this file already learned. The note on ui-smoke's timeout-minutes records that cutting the JOB budget to 20 turned this same stall from a slow pass into a red build, because one attempt had nowhere to go. The retry is what changes that, not a shorter wait: 180s is ~14x the 13 seconds `Linux deps` takes healthy, so only a mirror that is genuinely gone burns all three attempts. Every step's worst case fits inside its own bound and every step bound sits under the job's 30, which stays the final backstop and is deliberately not tightened. Listed in the workflow's paths filter, since both apt steps now execute it. The other 20 unbounded apt steps are left alone: every stall observed so far has been in this workflow, and converting release workflows unreviewed is a worse trade than leaving them until one of them actually stalls. * Bound every apt step in CI, not just the two that were noticed Three jobs were lost to an unbounded apt step in one day, each in a different workflow and each reported as "cancelled" with no reason: Chat UI Tests (chat) at 30m18s, Frontend build at 16m38s inside playwright install --with-deps, and Source lint at 5m02s on a job whose whole budget was 5 minutes. None of the three said anything about its actual subject, and the last was noticed only because someone happened to open a job that said "cancelled". Routes the remaining twelve on-runner apt steps through the shared retry helper, gives each an explicit step timeout, and adds a guard so the next one cannot be added without one. update and install now go as a single unit: retrying the install after a stalled update just re-reads the same broken package list. playwright install --with-deps counts as an apt step, because that is what it is. Leaving it out by name is what cost the Frontend build job. Job budgets were raised where the step's bounded worst case no longer fit: source-lint 5 to 20, api-smoke 12 to 20, frontend build 20 to 40. A job timeout is a backstop for the unforeseen; using it as the bound on a known-flaky step is what turns a diagnosable failure into a silent one. Not converted, with reasons recorded in the guard: clean-machine-install-ci runs apt inside bare containers and WSL with no checkout and deliberately no sudo, and the two steps where apt is the assertion rather than the setup. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-18 23:18:35 -07:00
Daniel Han	06220da4d2	Run two short jobs as background lanes of Lint CI, not on their own runners (#9176) ... * Run two short jobs as background lanes of Lint CI, not on their own runners Two workflows each held a runner slot on every commit for a few seconds of read-only checking: Unsloth load-orchestrator CI :: test ~33s, its own slot Lockfile supply-chain audit :: audit ~6s, its own slot Both now run as background lanes inside Lint CI, which has no path filter and was already going to occupy a runner on every commit. Three slots become one. Backgrounded rather than appended as steps, so the lanes overlap the ~65s of lint instead of adding to it: ubuntu-latest has four cores and the lint steps are single-threaded. Measured locally on the real step bodies, a 5s lane alongside 5s of foreground work took 7s total, not 10s. Absorbing a narrower-triggered job into an unfiltered one can only reduce the slots a commit takes. The absorbed work now runs on commits that would not have triggered it, on a runner that was going to exist anyway. Three things about this were not obvious and are worth recording: * Each lane needs its own venv. The load-orchestrator lane installs fastapi and friends while the foreground steps use the interpreter Lint CI's own pip install populated, and two concurrent installs into one site-packages is a real race. * The launch has to detach from the step's stdout and stderr. A background child inherits those pipes and the step is not considered finished while a writer holds them: launching a 4s lane took 4.0s before the redirect and 0.0s after. Nothing would have gone red, the overlap would just have silently disappeared. * A missing exit-status file has to be a failure. A lane that never started is otherwise indistinguishable from one that passed. The payloads live in .github/scripts/lane-*.sh, called by both the lane and the standalone workflow, so there is one definition. studio-load-orchestrator-ci.yml keeps workflow_dispatch as an escape hatch; its push trigger had no paths filter at all, so it was running on every commit to main regardless of the four paths its PR trigger listed. lockfile-audit.yml keeps its nightly schedule, which is a different check: it re-reads the lockfiles against advisories published since the last commit, which no commit-triggered run can do. tests/studio/test_absorbed_lanes_still_run.py covers the silent failure modes. Eight mutations verified red, including an uncollected lane, a missing sentinel read as success, and a launch that stops detaching. * Run the lane guard on workflow-only PRs The guard covers work moved off its own runner into a background lane, and every way that regresses -- a lane launched but never collected, a missing exit status read as success, a launch that stops detaching -- is an edit to lint-ci.yml or to one of the two absorbed workflows. No job's paths filter matches a workflow-only change: workflow-trigger-lint enumerates specific test modules, and Backend CI matches tests/** and its own YAML but not the workflow files guarded here. So the suite would first be collected by Backend CI's unfiltered push on main, after the change had merged. That matters more than usual because every failure mode here is silent. Nothing goes red when a lane stops being collected; the absorbed suites just quietly stop being able to fail anything. * Run the workflow guards in one pytest, and run the ten that ran nowhere workflow-trigger-lint is the only job in the repo with no paths filter, which makes it the only job a workflow-only PR is guaranteed to start. Every other job filters on source paths: Backend CI matches studio/**, tests/**, scripts/** and its own YAML, but not arbitrary workflow files. So a guard that reads a workflow file and is not run by this job has a specific silent hole. The edit it exists to reject is by definition a workflow-only edit, and on such a PR it is never collected. It is collected later by Backend CI's unfiltered push on main, after the change has merged. The guard still works, it just stops being able to block anything. Ten modules were in that state at once, including the indicator-parallelism guard, the GGUF phase-independence guard and the Windows small-checks guard, each of which reads a workflow and can only be broken by editing one. Three separate review rounds reported one instance each before the pattern was visible. They now run in ONE pytest invocation rather than a step per module. Measured over these 18 modules: 53.9s as a single call against 300.8s as one each, because this repo's conftest is expensive to import and a step per module pays it every time. That also removes the reason the list stopped growing: adding a guard no longer means adding a step. tests/studio/test_workflow_guards_run_unfiltered.py keeps the list honest, with two exemptions carrying their reason (PIL and a local utils helper, neither of which that job installs). It caught itself on its first run, being a workflow-reading module that was not yet in the list. Three mutations verified red: a guard dropped from the list, the invocation split back into two steps, and the job gaining a paths filter. * Give the consolidated guard job the cores and the budget it now needs Absorbing 12 more guard modules into one step made that step the job, and the job still carried a 5 minute budget sized for the old set. Serial the 21 modules are 209s on a 192-core box, which would have failed on timeout on a slower runner rather than on anything real. So: pytest-xdist, `-n 4`, and 15 minutes. Pinned to 4 rather than `auto` on purpose. ubuntu-latest is a 4-core runner so on CI the two are identical, but `auto` scales to the host and every xdist worker re-imports this repo's expensive conftest. Measured here: `auto` spawned 192 workers and took 327s, worse than running serially, while `-n 4` is 42s. A fixed width behaves the same everywhere and cannot be made pathological by the machine it lands on. 304 tests, identical results serial and parallel.	2026-08-18 05:27:55 -07:00
Daniel Han	68815aa888	Do not let the signal handler's own logging stop it dying (#9083) ... * Do not let the signal handler's own logging stop it dying A cancelled launcher exited 0 instead of dying of its signal, intermittently, and only on loaded CI runners. The cause was in the handler's first line. A signal handler runs on the main thread wherever that thread happened to be. If it was inside a write to stdout, the interpreter refuses the second one: RuntimeError: reentrant call inside <_io.BufferedWriter name='<stdout>'> That is what the handler's opening _log call did. The exception escaped the handler before it could re-raise the signal, main()'s except BaseException caught it, and the process exited on whatever code main computed. A cancelled job then reads as a completed one. It needs the signal to land inside a print, which is why it never reproduced locally and why it took a contended runner to show it. The line that identified it is the diagnostic added in 9079, which printed the launcher's own log on failure. Two changes, either of which is sufficient, kept because they fail independently: logging from the handler is now best effort, falling back to os.write, which goes straight to the file descriptor and takes no lock the interrupted frame could hold; and the death moves into a finally, so nothing above it can prevent the process dying of its signal. The regression test makes the reentrancy deterministic by raising from the handler's log call rather than racing a print. Against the current main it fails with returncode 0, which is the symptom as observed. Worth recording: 9072 attributed this to a transient OSError from release(). That mechanism is real and its fix stands, but it was not what CI was hitting. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop the handler's diagnostic rather than blocking on it Raising is not the only way a log line can stop the handler. stdout in CI is a pipe, and if the collector stops draining it a write parks in the kernel instead of failing, so no except and no finally runs. The announcement is the FIRST thing the handler does, before release(), so the kernels would keep billing until something killed the launcher from outside: the one outcome this file exists to prevent. That backpressure is also what leaves the main thread interrupted mid-write, so it arrives with the reentrancy the previous commit fixed rather than instead of it. _log_from_signal now asks whether the descriptor is writable before touching it and drops the line if not. POSIX reports a pipe writable only when PIPE_BUF bytes fit, and these lines are far shorter, so a positive answer means the write completes. Any error from select answers no. The new test reproduces it rather than simulating it: the launcher writes until the pipe is full and the test never reads a byte, so the main thread is asleep inside a write, holding the buffer lock, when SIGTERM lands. Without the guard it hangs the full 120s budget and the kernel is never deleted. 536 kaggle tests pass. * Move the nonblocking check into _log, so release()'s own logging is covered The previous commit guarded only the lines the handler prints itself. release() logs through the ordinary path -- delete_kernel() reports a refused delete that way -- so on a full stdout pipe the handler still stalled, just one frame deeper, before its retry and before the finally that re-raises the signal. Same outcome: a kernel billing until something kills the launcher from outside. The check now sits in _log, gated on a flag the handler sets on entry, so every line printed from that point on drops instead of stalling however it was reached. Nothing changes on the ordinary path, which is every line this script prints before something kills it. The regression test missed this because its fake deletion succeeded silently. It now logs on the way through, which is what a real refused delete does. Remove the guard and the test hangs the full 120s budget with the kernel never deleted; the previous commit's fallback guard does not save it. 536 kaggle tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Put every stdout line behind the handler's protections, not just the handler's own Two paths still bypassed them. release() warns about a kernel it could not delete with a raw print(), which is emitted on exactly the branch where a kernel is still billing, so a full pipe stalls the handler there before the finally can re-raise the signal. And _log dropped a line when the descriptor was unwritable but still raised when the buffer lock was held, and that reentrancy propagates out of delete_kernel() and out of release(), so a kernel Kaggle would have accepted on its third retry is never asked a third time. So the split is now between what a line IS and how it is written. _write_line carries the protections, _log adds the [launch] prefix on top, and the leaked-kernel warning goes through _write_line directly: through _log it would arrive as '[launch] ::warning ...', and GitHub matches an annotation from the start of the line, which would silently demote the one message that says a kernel is still billing. Three tests, each failing on the unfixed shape: a delete that is refused twice and accepted on the third still makes all three attempts while every write raises; a launcher whose deletes never succeed still dies of its signal with a full pipe; and the warning is read back out of a real main() run rather than from the source. The first version of that last one called the writer directly and passed on the mutation it was meant to catch. 536 kaggle tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-17 05:06:51 -07:00
Daniel Han	fa6be357dd	One multi-turn smoke script for all three operating systems (#9086) ... * One multi-turn smoke script for all three operating systems The check ran inline in the Linux, macOS and Windows smoke workflows as three copies of the same script, and being three copies is how one of them stopped checking. On 2026-05-22 an unrelated event-loop fix (#5669) turned the Linux copy's determinism assertion into a printed warning. macOS and Windows kept it, and are otherwise identical in logic: diffed, the only differences are that warning and some comments Windows had dropped. Nothing compared the three, so for three months the leg that runs on every pull request was the one not asserting, while the two that still asserted run rarely. The script moves to .github/scripts/studio_smoke/multi_turn_chat.py and all three call it. Nothing in it was platform specific; the only thing that differed per caller is the port, which it already read from BASE_URL. That removes 264 lines of YAML heredoc. It reads no environment and imports no SDK at module level, so the checking half is importable, and the new tests exercise it rather than grepping it: divergence between the two runs raises, trailing whitespace alone does not, an empty reply raises, and a transcript with no history grounding raises. Reverting the script to the warning form fails one of them; re-inlining a copy into a leg fails two. Worth stating plainly: this re-enables on Linux an assertion that has not run there since May. If greedy decoding is genuinely not reproducible on that leg, it will now fail rather than print. That is the right way round, and it will be one failure in one place rather than three copies to reconcile. The guard runs from workflow-trigger-lint.yml, which carries no paths filter, because a pull request that re-inlines one leg edits a workflow and nothing else. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Assert both runs are non-empty, not just the first The stripped comparison cannot tell the two apart: a second run returning "" against a first returning "\n" compares EQUAL, so a server that stopped answering halfway through printed OK. The Linux copy asserted both before this was consolidated; the macOS copy it was taken from asserted only the first, and I kept that half. The test now covers an empty SECOND run and the exact whitespace-against-nothing pair the comparison is blind to. Dropping the new assertion fails it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-17 04:59:17 -07:00
Daniel Han	5307992416	Keep the killed-by-signal exit status when release() fails (#9072) ... * Keep the killed-by-signal exit status when release() fails A cancelled launcher reported success. The signal handler deletes the kernels and then re-raises the signal so the status still reads killed, but the delete was outside any try: a raise there propagated out of the handler into whatever the main thread was doing, main()'s except BaseException caught it, and finish() called release() a second time. When that second call worked, which is what a transient OSError out of a subprocess spawn on a loaded runner looks like, main RETURNED 0 and the job read as completed. Caught on a contended runner rather than by inspection, and reproduced by failing the first delete only: returncode 0 instead of -SIGTERM. The delete is now best effort, retried once, and the death by signal happens either way. If both attempts fail the slugs stay in the registry and the next launcher's orphan sweep reclaims them, which is the same path a kill -9 already takes. The subprocess waits in the suite move from 30s to a named budget of 120s. They guard against a launcher that never dies, not against latency, and the repo suite now runs four of these at once on a four-core runner. * Make the stall outlast the death budget, and assert the signal did the killing Raising the wait to 120s put it past the 60s stall the launcher sits in, which defangs the tests it was meant to make robust: a handler that swallows its signal leaves the process asleep and then resuming, so it wakes, runs finish(), deletes the kernels through the ordinary path and exits INSIDE the wait. The deletion assertions are satisfied by that, so those tests would have passed on a launcher that ignored the signal entirely. The stall is now a named 900s against the named 120s budget, and a test asserts the relationship rather than leaving it to whoever next tunes one of the two numbers. The deletion test also asserts the exit status now. Without it, the deletion is satisfied by finish() doing its usual work, so nothing there said the signal was what caused it. Verified by making the handler return instead of re-raising: with the stall at 900s that fails all four signal tests, and with it at 60s the relationship test fails as well. * Tighten the comments added here	2026-08-17 01:00:58 -07:00
Daniel Han	482abfabfa	Give Unsloth Studio its first CUDA coverage, on a Kaggle T4 (#8489) ... * Run deterministic notebook smoke tests on real Kaggle T4s The notebooks are written for a T4: sm_75, fp16 with a gradient scaler, xformers rather than flash-attention, 16GB. Nothing in CI runs on that card, so a regression that only appears there reaches users first. Adds a gated job that launches one Kaggle GPU session and runs the smoke payload once per T4. The account has a 60h weekly budget shared with other consumers, so the job is a deliberate minority consumer of it: paths filters (only 9.9% of a week's 444 main commits touch unsloth/), a ~10% sampling draw keyed on the run id so a re-run cannot reroll, a quota floor read live from Kaggle's own API, and a stand-down when the account already has a kernel in flight. Expected cost is 5-7 GPU-hours a week. Only a payload that ran on a T4 and disagreed with its assertions turns the check red. Throttling, the 2-kernel concurrency cap, a dead session or a download that would not complete are all warnings, because a free external service blocking merges would be ignored within a week. The payload asserts three things, worth different amounts. Two fresh processes must agree bitwise on per-step loss and grad_norm; that one is exact, and it is fresh processes because two in-process cycles disagreed from the first step while two processes agreed on all ten. Greedy decoding must emit the canary string exactly, which catches trained weights never reaching the generate call. Metrics must sit inside a band around a committed reference, which is a tolerance and never an equality, since fp16 reduction order moves with the driver and the libraries. Ten steps rather than three: under fp16 the scaler skips the first two overflowing steps, so a three-step run lands one real update and the canary never forms. The dataset is supplied as prompt/completion columns so the loss falls only on the answer. * Make the payload cells the T4 rejected actually run there Two launches on real Kaggle T4s died in the scaffolding before a training step ran, both of them in generated code that nothing had ever parsed. The dependency probe imported unsloth_zoo before unsloth. Zoo's __init__ ends with a find_spec("unsloth") guard and raised "Please install Unsloth via pip install unsloth" on a session where unsloth was installed and imported cleanly one entry later, so the payload aborted over a dependency that was not missing. Probe unsloth first, and invalidate the import caches after the pip installs that this interpreter did not see at startup. The reference argument was generated as a shell fragment and spliced into the middle of a Python list literal, with doubled braces that left ROOT unsubstituted. The cell was a SyntaxError, and the workflow passes --reference on every path, so no payload could ever have reported. Verified locally by executing the regenerated run cell end to end: two fresh processes, bitwise identical metrics, canary exact. * Bound the in-flight kernel survey by session length, not kernel count The concurrency check status-checked the twelve most recently run kernels, which is a sample rather than a search: a kernel that started three hours ago and is still running is missed the moment twelve newer ones have since run, and the push then fails at the account capacity cap and is reported as infra. Kaggle kills a notebook session at twelve hours, and the listing is sorted by last run time, which for an unfinished kernel is when it started (measured: a kernel pushed at 10:05:19Z lists as 10:05:19.297). So walking until the first entry older than that ceiling covers every kernel that can still be in flight, and stopping there is exhaustive rather than arbitrary. An hour of slack is added for clock skew. Two silent ways to read an unanswerable question as a green light are closed with it. Hitting the page cap now marks the survey incomplete, and a survey where no status could be read at all is unknown rather than idle. Both stand down, which is the cheap direction: the next commit draws again in minutes. On the live account this now costs three status calls instead of twelve. * Stop the reference band check from passing NaN unconditionally The band comparison did its NaN handling by accident, and it got it wrong. Under fp16 the gradient scaler logs a NaN grad_norm on every step it skips, so a committed T4 reference genuinely contains NaN. Left to the arithmetic, abs(x - NaN) is NaN and NaN > tolerance is False, so those steps passed whatever they held, including the case the check exists for: a step that used to overflow and no longer does. Compare NaN to NaN as equal and NaN against a number as a deviation, and treat a field logged on one side only as a deviation too, since no tolerance covers a change in shape. The mapping from an out-of-band verdict to the failure list moves into reference_failures() so the path that turns the job red can be tested without a GPU. The references README now says plainly that no reference is committed, that one must not be borrowed from other hardware, and how to lift one out of a green run's evidence rather than spending a session on it. It also records a measurement: the absolute floor never engages on this trajectory. The smallest value on the ten-step curve is a loss of 0.1428 against a floor of 0.05, so the floor changes no verdict today and is kept for a configuration that does go lower. * Name the cause when a kernel finishes and reports nothing A kernel that ends COMPLETE with no payload report is the hardest outcome to read: the summary has no metrics to show and the reason sits in an artifact nobody downloads. It is also the outcome both real T4 failures took. Kaggle returns the kernel log as a JSON array of stream records rather than as text, so the interesting line arrives split across dozens of them and reading the file directly shows a wall of JSON. Flatten it, filter to the driver and payload sentinels plus tracebacks, and fold the tail into the job summary. Checked against the evidence of a real failed run: the SyntaxError that killed it is now visible without leaving the summary. * Cover the four T4 CI fixes with CPU-only tests The generated cells are now compiled on both the with-reference and without-reference paths, which is what would have caught a SyntaxError that instead cost a Kaggle session, and the dependency probe's module order is asserted. The in-flight survey gets a fake Kaggle client and the case that motivated it: one kernel still running behind forty newer finished ones, which a count-bounded scan misses and a time-bounded scan cannot. Also the window boundary, timezone-aware and missing timestamps, the page cap, and the difference between some statuses unreadable and all of them. The band check is now proved to fail rather than assumed to work: a perturbation half a band-width past the limit, a moved scaler skip pattern in both directions, a field that stopped being logged, and a length mismatch, each asserted through to the failure list. Three tests perturb the committed reference itself and skip, saying so, until one exists. 41 passed, 3 skipped. * Parse every generated cell on every build path before spending a session Two of the three Kaggle sessions spent so far died on generated code that nothing had ever parsed. The compile check now builds all three paths the generator has, using the workflow's own argument list verbatim, including the one where a committed reference exists and is carried inline as a fourth file: that path becomes live the moment a green run supplies the file, and would otherwise first be exercised on Kaggle. Parsing alone is not enough. A template hole that substitutes to a bare identifier parses and then NameErrors, which costs the same session, so every cell is also checked in execution order for names nothing before it defines. Both checks were confirmed to go red: the first on the exact shell-fragment-in-a-list-literal defect that cost session three, the second on a run cell reading a name no cell binds. The carried sources are also decoded the way the kernel will decode them and compared byte for byte with the repo, so the T4 cannot quietly run something other than what is committed. * Commit the T4 reference the first green Kaggle run produced Kernel danielhanchen/unsloth-t4-ci-e3c6661f, terminal state COMPLETE, both payloads passing on their own Tesla T4 of one session. Two fresh processes agreed bitwise on all ten steps on both cards, max_abs_diff exactly 0.0 for loss and for grad_norm, and the two cards independently produced the same ten values as each other, so four processes agree rather than two. All four cycles emitted the canary exactly. The scaler skipped steps 1 to 3 every time, which is the behaviour the ten-step length exists to accommodate. The file is reports[0] of that run, copied whole by the recipe the README already documented, and nothing in it is from other hardware. One assumption the run contradicted, now recorded rather than repeated: the session had no xformers. unsloth installs --no-deps, unsloth_zoo does not carry xformers and the Kaggle image does not either, so the banner read Xformers = None and this trace is the fallback attention path. Adding xformers to the install would move these numbers and require a recapture. The floor's smallest observed value moves from 0.1428 to 0.0871, still above the 0.05 denominator floor, so the floor stays inert and the test that re-derives that keeps passing. The three tests that perturb the committed reference now run instead of skipping. 45 passed, 0 skipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the in-flight tolerance the Kaggle gate stands down on The gate declines to launch whenever any kernel of the account is in flight, which is stricter than Kaggle's 2-kernel cap requires. That is the intended policy, so it is now a named constant with the tradeoff written next to it rather than an implicit property of the code, and a --allow-in-flight argument that cannot be raised to the cap itself. Default behaviour is unchanged. Also files the workflow under 'never' in ci-preempt.json: cancelling a run does not stop the Kaggle kernel it already pushed, and the orphan then bills quota to its own ceiling with nobody watching. * Run three steps, and refuse a reference captured at another count max_steps drops from 10 to 3. Measured consequence: under fp16 the gradient scaler starts at 65536 and skips every step it overflows on, and the committed reference shows steps 1, 2 and 3 all skipped, so a 3-step run of the old configuration applied zero optimizer updates. --init-loss-scale pins the scaler below the overflow point so those three steps are real updates, and optimisation_failures() fails a run whose every step was skipped rather than letting it report as training. The committed reference was captured at 10 steps and no longer describes the run. check_reference now reads the max_steps the reference records and refuses to compare across counts, as a failure with both counts named -- including when the reference does not say what it was captured at. A quiet pass there would be a green check comparing a run against a curve it has nothing to do with. The reference is therefore stale until one recapture run. The workflow gains skip_reference_band for exactly that dispatch, and the recipe is in references/README.md. Also brings the workflow onto the repo's conventions: pinned action SHAs matching the rest of CI, a workflow-level concurrency group that deliberately does not cancel in progress, a fork guard on the gate job, typed dispatch inputs and an artifact name without the run id. * Keep the run at ten steps, and say why three does not work The committed reference is the evidence: the fp16 scaler reports grad_norm NaN at steps 1, 2 and 3 and a finite value from step 4. It starts at 65536, halves on each overflow, and skips the step it overflowed on, so a 3-step run applies zero optimizer updates. Loss stays around 10, the canary never forms, and the band would compare three points of a curve that never moved. Shortening also saves nothing worth having: a launch costs about 0.08h and that is almost all pip install, not training. The step-count guard, the all-steps-skipped guard and --init-loss-scale all stay, since they are what makes a shorter run possible later and what stops a vacuous one reporting green. The pin is off by default so the committed reference keeps applying. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Turn one T4 payload into four, and pair a pinned control with a canary The workflow ran a single tiny SFT payload on both cards of one Kaggle session and spent about 1.4 GPU-h a week doing it. The budget is now 40, and the question asked of it is wider: catch regressions in trl, transformers, accelerate, peft, bitsandbytes, torch and vllm, and catch torch.compile falling over on sm_75. Four legs now, two kernels of two T4s each, described once in .github/scripts/kaggle_t4_ci/legs.py rather than restated in YAML: control the existing tiny Qwen2.5-0.5B determinism run, pinned canary the same run on the newest library set Unsloth's own constraints allow gptoss gpt-oss-20b LoRA, for torch.compile and the forced float32 path grpo Qwen3-4B GRPO through a vLLM engine on the same card control and canary are one instrument rather than two tests. Same payload, same seed, same data, same step count, on the two cards of the same session; the only difference between them is the installed versions. A red canary beside a green control is a library release, and the job summary already puts the two version sets side by side so the bisect is a diff rather than an investigation. The reference band applies to the control alone: two library sets do not produce one fp16 trajectory, so the canary asserts what does not depend on versions instead. The pins are evidence rather than preference. transformers and trl are the pair the committed reference was captured with; peft, accelerate and bitsandbytes come from probe kernels 8161ceb9 and 7ab727f1, two independent T4 sessions that ran the identical base install and agreed. torch stays unpinned and recorded, because it comes from the Kaggle image and replacing it is the failure mode the grpo leg spent two probes on. The gate now separates this workflow's own kernels from everybody else's. It takes both of Kaggle's concurrency slots, and only when the account is otherwise idle: one kernel belonging to anyone else still stands it down entirely, which is the same policy as before stated at the right grain. * Leave the vLLM leg unwired, and stop trusting Kaggle's own kernel timeout Three probe sessions tried to run the Qwen3-4B GRPO leg on a T4 and none of them reached a training step. None of them failed for a reason to do with sm_75, memory or GRPO either: vllm==0.11.2 pins torch==2.9.0, so the leg has to replace the Kaggle image's torch, and that is what killed all three. With the image's site-packages visible, pip treats torch's pinned NVIDIA runtime packages as already satisfied by the copies belonging to 2.10, and the result is a torch that installs cleanly and cannot be imported -- libcusparseLt.so.0 first, then ncclCommWindowRegister once that one was named explicitly, identically on vllm 0.11.2 and 0.15.1. An isolated venv resolving the whole stack got further and then the session wedged. So the leg keeps its payload, its assertions and its install, and moves to legs.UNWIRED with the three kernel slugs and what each one measured. Wiring it now would make the check permanently red and spend the budget doing it. The second kernel keeps its spare T4, which costs nothing, and that is where the leg goes when the install works. The third probe also cost about an hour of quota teaching us that the push-time kernel timeout is not a budget control. Pushed with -t 5400, its own nbconvert crashed at t=406s and the session sat in RUNNING for over two hours -- past that ceiling and past the launcher's own deadline, because one status call had blocked with no socket timeout to bound it. Deleting the kernel by hand took the account's used-hours figure back down, which is the measurement that settles which control is real. The launcher now sets a socket timeout and deletes every kernel it pushed on every path out, and the workflow header says which of the three bounds is load-bearing. Sampling recomputed on the payload that actually ships: 40%, from measured per-kernel cost rather than the old estimate, with the note that wiring a fourth leg means recomputing it rather than editing one line. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-solve the GRPO leg's install so it stops replacing the image's torch Three probe sessions never reached a training step, and all three died the same way: vllm==0.11.2 pins torch==2.9.0, so the leg had to replace Kaggle's torch 2.10.0+cu128, and pip treats the image's NVIDIA runtime packages -- which belong to 2.10 -- as satisfying the new torch's pins. That imports as libcusparseLt.so.0 missing, then as libtorch_cuda.so undefined symbol ncclCommWindowRegister one package along, then as an isolated venv that spent an hour of quota resolving a CUDA stack and never produced payload output. None of that is a question about sm_75, memory or GRPO. It is one line: no vLLM release in the 0.11-0.16 range pins the torch the image ships. Releases 0.17.0 through 0.19.1 pin torch==2.10.0 exactly, so pin 0.19.1 and the leg replaces nothing, keeps system_site_packages, and installs like any other. Drop xformers with it. Its vLLM attention backend was deleted in 0.12.0, so it would be a package nothing selects. sm_75 has no FlashAttention and no FlashInfer, and the ladder in vllm/platforms/cuda.py falls through both to TRITON_ATTN; the leg names that in VLLM_ATTENTION_BACKEND so a release that reorders it goes red here rather than quietly selecting something else. 7.5 is still in CUDA_SUPPORTED_ARCHS at v0.19.1 and fp16 is a supported dtype below capability 8.0. Still unwired. Two things are static evidence rather than a run: whether 0.19.1 starts on sm_75 at all, and whether 8GB of 16-bit weights plus an engine plus a LoRA trainer fit in 14.56GB. UNWIRED says so in those words, and a test asserts the note still names the pin and still says what is unknown, so nobody wires this off prose that outlived its measurement. Also ports the vacuous-test fix from the stacked Studio branch: pre-commit.ci reformatting default=10 to default = 10 stopped the payload regex matching, so the step-count agreement check was comparing nothing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop the GRPO leg JIT-compiling flashinfer on a machine that cannot link it First probe on the re-solved install, kernel unsloth-t4-ci-e2d9ce9b. The install works: torch stayed at the image's 2.10.0+cu128, vllm 0.19.1 installed and imported, xformers was absent as intended, and the payload ran on a real Tesla T4 reporting capability 7.5 with TRITON_ATTN accepted. It reached engine construction, which is further than any earlier probe got by the whole width of the install. It died in flashinfer 0.6.6's JIT, and not for an sm_75 reason. All three .cu files compiled cleanly for -gencode=arch=compute_75,code=sm_75. The LINK failed: /usr/bin/ld: cannot find -lcuda -L/usr/local/cuda/lib64/stubs is already on the command line, so the image has no driver stub libcuda.so, only the runtime libcuda.so.1. Nothing about that is fixable from a payload. Set VLLM_USE_FLASHINFER_SAMPLER=0 so nothing JITs. The sampler has a native path, and skipping the build also saves a four-file nvcc compile inside a session billed by wall clock. UNWIRED records what the probe measured and narrows what is left to one question: whether the engine builds now. * Make -lcuda resolvable so flashinfer can link what it compiled Second probe, kernel unsloth-t4-ci-916d5986, failed identically to the first with VLLM_USE_FLASHINFER_SAMPLER=0 set. That is the informative part: the JIT is not reached only through the sampler, so switching off one consumer is whack-a-mole. The failure was never sm_75. Both sessions compiled all three .cu files cleanly for -gencode=arch=compute_75,code=sm_75 and died at the link: /usr/bin/ld: cannot find -lcuda -L/usr/local/cuda/lib64/stubs is already on the command line; the image ships no libcuda.so at all, only the versioned libcuda.so.1, and the linker will not resolve -lcuda against a soname. Normally the toolkit's driver stub fills that gap; this image has the directory and not the file. So symlink libcuda.so at the real driver in a scratch dir and put it on LIBRARY_PATH, which gcc and ld search for -l. No root, nothing written into /usr/local, and it fixes every flashinfer op rather than one. Linking against the real driver instead of a stub is correct here: the driver is present, which is the whole reason a stub would have substituted for it. Also widen the captured traceback to head AND tail. The last probe's 6000-char tail was entirely ninja's own output, so the Python frames naming the caller were exactly what got dropped. Probe 2 also measured the fit question the notebook raised: peak reserved 7.96GB of 14.56 with load_in_4bit false, so 16-bit weights are not what is short of room here. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only count a libcuda the linker will actually search for Kernel unsloth-t4-ci-d0d480b6: the shim reported already_linkable and did nothing, and the link failed anyway. It had found libcuda.so in /usr/local/cuda/compat -- which is real, and which nothing passes with -L. The failing ninja line names /usr/local/cuda/lib64 and /usr/local/cuda/lib64/stubs, and those are the only two directories that can answer -lcuda. So the precondition now checks exactly those two, and compat becomes what it is actually good for: a symlink TARGET when ldconfig and ctypes both come up empty. A library the linker will not search for is not a library the linker can find. Two tests: one reproduces the compat-only machine and asserts the shim is built anyway, one pins the searched directory list, since widening it is precisely how the check went wrong. * Give the base model a chat template, and record what the link shim did The link shim works. Kernel unsloth-t4-ci-27b0dc2e reported libcuda_shim: needed true, applied true, searched [/usr/local/cuda/lib64, /usr/local/cuda/lib64/stubs], real /usr/local/cuda/compat/libcuda.so and flashinfer is gone from the failure entirely. Memory went from 7.96GB to 11.36GB of 14.56, which is the vLLM engine actually loading, and the run reached trainer.train() -> _run_epoch -> training_step. Both questions legs.UNWIRED was holding this leg for -- does vLLM start on sm_75, does it fit in 14.56GB -- are answered by that. It then failed on the payload's own gap: ValueError: Cannot use chat template functions because tokenizer.chat_template is not set unsloth/Qwen3-4B-Base is a base model and ships none. The notebook this leg comes from solves it with an SFT priming stage that installs a template before GRPO starts; this leg has no priming stage, so it sets a minimal ChatML template directly and records in the report which of the two worlds the run was in. The base model stays. GRPO on an instruct model would measure the instruct tuning as much as the run, and the payload's rewards are format-and-digit rewards a base model can move inside three steps. A test renders the template rather than matching it as a string: a template that does not render would trade a failure at step 1 for a failure at step 1 with a longer traceback. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Wire the GRPO leg: it runs on a T4 and the rewards move Kernel unsloth-t4-ci-53efcc4e passed on a real Tesla T4. Step 2 logged reward_std 0.707 and grad_norm 0.772, peak 13.60GB of 14.56GB, three steps in 192s, and fast_generate returned coherent text. reward_std is the criterion rather than loss, because TRL's GRPO loss is ~0 by construction at num_iterations=1 and beta=0 and would pass on a run that learned nothing. The config in args is the one that fit, not the notebook's. Two probes with seq 2048, 4 generations, rank 32 and utilization 0.9 died in the backward at unsloth_zoo/gradient_checkpointing.py:1013, peaking at 15.97GB in 16-bit and 19.25GB in 4-bit. 4-bit alone is not the lever: it peaked HIGHER, because quantizing weights does nothing for activations while utilization 0.9 still hands vLLM ~13GB up front, and UNSLOTH_VLLM_STANDBY returns the weights during training but not the KV cache reservation. An earlier probe reported this as 'CUDA error: an illegal memory access was encountered' from empty_cache() inside vLLM standby. That was the async shadow of the same OOM; CUDA_LAUNCH_BLOCKING=1 produced the honest error. UNWIRED is now empty. The test that read its grpo entry is replaced by the invariants that outlive any one leg: every leg is wired or explained, nothing is both, and an unwired note says what is open. Two new tests pin the config and the vLLM pin, so restoring either to a value that OOMs fails here rather than on Kaggle. * Add a CUDA_LAUNCH_BLOCKING switch to the GRPO payload An illegal memory access is reported at whatever CUDA call synchronises next, which on the first T4 GRPO run was empty_cache() inside vLLM standby, nowhere near the kernel that faulted. Re-running with this flag produced the honest error instead: torch.OutOfMemoryError in the backward. That is what turned an unexplained CUDA fault into a memory budget, and the budget is now the leg's config. The environment variables are set before the first import torch in the process rather than merely before train(), because they have to precede CUDA context creation. It serialises every kernel launch, so it stays a switch and never a default, and it is recorded in the report because it changes what a timing means. * Give Studio its first CUDA coverage, on a Kaggle T4 Every Studio workflow in this repo runs on ubuntu-latest, macos-15 or windows-latest. macOS gives Metal and MLX some hardware; the CUDA path is exercised by nothing at all, and studio-inference-smoke.yml uses a 270M GGUF precisely because it has to decode on a CPU. tests/kaggle/studio_gpu installs Studio the supported way on a Kaggle T4, starts it headless, and asserts three things a CPU runner cannot: that a GGUF is actually offloaded to the GPU rather than falling back to CPU and returning text anyway, that a LoRA training run completes and leaves an adapter on disk, and that a GGUF export against a CUDA llama.cpp produces a file that loads. The repo's existing playwright_chat_ui.py then drives the browser path, last, because it stops the server on its way out. The gate and the launcher are the notebook leg's, reused unchanged: the payload prints the same report line, so the transport needed no edit. The two legs share a concurrency group because Kaggle's 2-kernel cap is per account and is not workflow-aware. Sampled at 4 percent of a filter measured at ~760 eligible invocations a week, which is ~30 launches and ~23 GPU-h against the notebook leg's 15-20 and 1.4. The rate is lower and the spend is fifteen times higher because the eligible stream is five times larger and a launch costs nine times as much; the arithmetic is in the workflow header. * Budget the two Kaggle GPU legs against one account, not two The Studio leg's budget block was solved against the notebook leg's numbers from before that leg was re-tuned: 0.08 h per launch, 10 percent, ~1.4 GPU-h/week. The notebook leg now costs 0.25 h at 40 percent for up to 23.1 GPU-h in a busy week, and Studio was sized at ~23 of its own. Two legs at 23 each is 46 of the 50 hours CI is allowed on a 60 h account, with the entire margin spent on the cheaper one, and the two reserve floors were picked independently. Re-solve both against one shared allowance: Studio 35 h, notebooks 15 h. Studio goes to 5 percent (~38 launches, ~28 GPU-h) and notebooks to 15 percent (13-35 launches, 3.3-8.7 GPU-h). The reserves now encode priority rather than each leg's own ceiling: notebooks stand down once the account is 35 h spent, Studio runs on to 50. The percentage is the one axis where Studio reads lower, and it cannot be made to agree. Studio's eligible stream is ~3.3x larger and each launch costs 3x more, so a point of rate is worth about ten times as much there; matching the notebook leg's 15 percent would be ~86 GPU-h a week. Both budget blocks now say so, and a test re-derives the joint spend from the two headers instead of trusting either total. Also fix a test that had gone vacuous: pre-commit.ci reformatting default=10 to default = 10 stopped the payload regex matching, so the step count agreement check was comparing nothing. Match count is asserted now. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Write down where the Kaggle sampling percentages come from The percentages in the two workflows were arithmetic nobody could check. This records the measurements they came from, taken 2026-08-11 over a 28-day window: 985 commits to main, 1157 PRs, and 8 of 60 sampled PRs touching the paths a T4 training run could regress, so a 13% path-filter rate and roughly 75 candidate events per week. Against 30 GPU-h/week for one account at ~0.75 GPU-h per launch, the ceiling is about 33 launches per week if everything went to one workflow. The 15/5 split and the 25/10 reserve-hour floors follow from that, and the file says what would change them: a second account, a wider path filter, or a session cost above ~1 GPU-h. * Give the GRPO leg its own kernel: pairing it with gpt-oss fails Kernel unsloth-t4-ci-70a2f4eb ran the wired pair on the two cards of one session. gptoss passed: 32 unique graphs, 779 calls captured, 2 graph breaks, peak 12.78GB. grpo failed with the illegal memory access again, at exactly the 13.60GB peak at which it PASSED alone on kernel unsloth-t4-ci-53efcc4e, same config to the flag. Same peak, different outcome, so the constraint is not GPU memory: the driver pins one card per payload. gpt-oss offloads to host RAM and vLLM wants host RAM, and a Kaggle session has one host. A third kernel costs a queue slot rather than quota, since Kaggle runs two at a time and bills wall clock per session. That is cheaper than a leg which fails half the time for a reason the report cannot show. Two tests pin the rule in both directions: grpo must not share a kernel with gptoss, and control must still share one with canary, because those two are a matched pair whose comparison is the whole instrument. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Unwire the GRPO leg: running it alone reproduced the failure I blamed on pairing I gave grpo a kernel of its own earlier today on the reasoning that sharing a session with gpt-oss was what broke it. It had failed paired (unsloth-t4-ci-70a2f4eb) and passed alone (unsloth-t4-ci-53efcc4e), so pairing looked like the variable, and I wrote two tests pinning the separation. That reasoning was wrong. Running the leg ALONE again (unsloth-t4-ci-c98f14be, built from this branch with --legs grpo) reproduced the paired failure exactly: unsloth_zoo/vllm_utils.py:601 sleep() -> torch.cuda.empty_cache() torch.AcceleratorError: CUDA error: an illegal memory access was encountered engine_built false, peak 13.8GB/13.6GB of 14.56GB. The passing session and both failing ones agree to the digit on every recorded version -- torch 2.10.0+cu128, transformers 5.5.0, trl 0.24.0, peft 0.19.1, vllm 0.19.1, unsloth 2026.8.15, zoo 2026.8.10 -- on the same peak, with UNSLOTH_VLLM_STANDBY=1 throughout. One contrasting observation was never enough to blame a shared host. So what three sessions show is an intermittent illegal memory access on Turing, one pass in three, not a pairing effect. A leg that passes one session in three cannot tell CI anything: it would go red for a reason no reader could act on. grpo moves to UNWIRED with the three session ids and the open question written down, rather than back into a kernel. The note keeps what the leg already earned -- the vLLM/torch pin, the attention backend, the libcuda shim, the chat template, the T4-sized config -- so re-wiring means answering the IMA rather than redoing the install. test_grpo_does_not_share_a_session_with_gptoss is replaced rather than deleted: it asserted the opposite thing for the disproven reason, and would now pass vacuously since grpo is in no kernel at all. The replacement pins that grpo stays unwired and that the note still names all three sessions, so the evidence cannot be dropped on the way back in. 145 passed. * Retire the bootstrap password at login, so the session can reach the routes First hardware run of this payload, Kaggle kernel unsloth-t4-ci-412345d2 on two Tesla T4s: preflight, studio_ready, authenticate and the Playwright chat_ui driver all passed, and the four assertions the payload actually exists for did not run. gpu_inference: POST /api/inference/load -> HTTP 403: {"detail": "Password change required"} lora_training: POST /api/train/start -> HTTP 403: {"detail": "Password change required"} A bootstrap account is created with must_change_password set, and get_current_subject turns that into a 403 on every route except the password-change one. So login yielded a token that authenticated and could do nothing, and the login step reported success while inference, tool calling, training and export were all unmeasured behind it. login now completes the change when Studio's own Token says must_change_password, and carries the token that /api/auth/change-password mints. The replacement is secrets.token_urlsafe(24): never whitespace, which the route rejects, and never equal to the bootstrap value, which it also rejects. It is not returned or stored -- nothing needs it again. Four tests, all off-box against a scripted HTTP layer: the change happens, and the session carries the post-change token an account already past the gate does NOT get its password rotated a refused change raises instead of carrying on with a token that cannot act a change that answers without a token raises The third is the one that matters: continuing past a failed change is how the first run produced four unmeasured assertions and no explanation. Two things this does NOT fix, recorded so the next run is read correctly. gguf_export reported install_kind=None, i.e. no llama.cpp install under STUDIO_HOME at all. That may well be downstream of the 403 rather than independent: POST /api/inference/load is what pulls a llama.cpp for a GGUF model, and it never ran. The collected kernel.log is 223 lines and holds none of the install output, so this is not settled either way -- the re-run answers it. gpu_inference also failed its own evidence test: text came back, but no probe could show the GPU was used. The payload calls that a failure rather than a pass, which is the behaviour I want kept. 89 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hand the UI driver the password the session actually holds Second hardware run, kernel unsloth-t4-ci-9ddd8ae4. Retiring the bootstrap password fixed three assertions and broke a fourth: gpu_inference FAIL -> PASS tool_calling FAIL -> PASS lora_training FAIL -> PASS chat_ui_driver PASS -> FAIL: "the bootstrap password is gone, so the driver cannot log in" My own doing. assert_chat_ui read $STUDIO_HOME/auth/.bootstrap_password off disk, and authenticate() has to retire that value to get past Studio's forced password change, so the file is stale by the time the driver runs. The driver's first phase rotates the password itself and asserts the old one stops working, so it needs the value the session is CURRENTLY authenticated by -- not the seeded one. login now leaves that value on Studio.password and assert_chat_ui passes it, falling back to the seeded file only when no login has happened. The commit that introduced the retirement said the replacement was "never returned or stored: nothing needs it again". That was wrong, and the docstring now says so and says why. Studio.password is a credential held for the run, unlike anything else in that module, so the module docstring records it and assert_chat_ui adds it to the scrub set before it can reach a log. test_the_session_remembers_which_password_is_current pins both directions: the retired session reports the replacement it minted, and a session that never had to change reports the password it logged in with. Still failing, and NOT downstream of the 403 after all: gguf_export reports install_kind=None on a run where inference, tool calling and training all passed, so no llama.cpp is installed under STUDIO_HOME regardless of the auth gate. I said last time that this might resolve itself once the 403 was fixed. It did not. It is a separate gap and it stays open. 90 passed. * Record what launch blocking did to the GRPO leg: it removed the fault The UNWIRED note asked for one experiment: run the leg with --cuda-launch-blocking so the illegal memory access reports at its real call site instead of surfacing at torch.cuda.empty_cache(). Done, kernel unsloth-t4-ci-b1f23e34, and the answer is not the one the note expected. With blocking on there was no illegal memory access at all. engine_built true, three steps, same 13.8GB peak, same versions to the digit. A fault that disappears when the launches are serialised is a race, which is what one pass in four already suggested and which no amount of re-running the leg will pin down. The run also exposed a SECOND problem, and it is separate from the first. It failed on reward_std = [0.0, 0.0, 0.0] with grad_norm 0.0 at every step. The completions in the report are coherent prose, so this is not the model collapsing into degenerate output. It is two completions scoring identically. The leg runs num_generations = 2 because that is what fits on a 14.56GB card, and at two samples a tie on a coarse reward is ordinary rather than a bug. So the leg's pass criterion is fragile at the size it has to be to fit, which is worth knowing before anyone reads a red grpo leg as a regression. The note now records both, says which question each answered, and states the two that remain open: where the race is, and what pass criterion is honest at num_generations = 2. The test gains the fourth session id and a check that the note no longer reads as though launch blocking were still the next thing to try -- a stale "try this next" is how an experiment gets run twice. 145 passed. * Re-seed the account before the UI driver, because the two want opposite states Third hardware run. The API assertions all pass now: preflight, studio_ready, authenticate, gpu_inference, tool_calling, lora_training -- all PASS and the Playwright driver still fails, for a third and final reason in this chain: [ui] STEP change-password through UI (Setup your account) [ui] change-password form attempt 1 failed: Timeout 60000ms exceeded waiting for locator("#new-password"); page.url=.../login The driver's first UI step drives the forced-change form. authenticate() has to retire that same bootstrap password over the API to reach /api/inference/load and /api/train/start at all. So by the time the driver runs, Studio shows an ordinary login and #new-password does not exist. That is not a value problem, it is a state problem, and the previous commit only fixed the value. The API path wants the change DONE; the driver wants it PENDING. One account cannot be in both states, and three runs walked the whole chain to get here: 412345d2 failed the API assertions on the gate, 9ddd8ae4 fixed those and failed the driver on a stale password, this one fixed the password and failed the driver on the form being gone. assert_chat_ui now restarts Studio before handing over. start_server() removes $STUDIO_HOME/auth, which is what re-seeds the bootstrap password, so the driver gets exactly the state its first step expects. It is cheap here specifically: this assertion runs last and stops the server anyway, so nothing after it needs the API session, and the driver loads its own model through the UI. test_the_ui_driver_gets_a_freshly_seeded_account pins the restart, that the password is read AFTER it, and that the retired session's password is not what gets handed over -- the restart is the whole fix, and a refactor that drops it puts the payload straight back to a driver that cannot find the form. Unchanged and still open: gguf_export reports install_kind=None. Three runs now, one of them with inference, tool calling and training all green, so no llama.cpp is installed under STUDIO_HOME and this is not downstream of the auth gate. 91 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Test the newest transformers and trl, not the newest zoo permits The canary leg is documented as "the newest permitted library set" and that is exactly what it installs: it resolves WITH unsloth_zoo in the resolution, so zoo's own metadata is a ceiling. Measured against the two runs that carried it: canary transformers 5.5.0 trl 0.24.0 peft 0.20.0 accelerate 1.14.0 control transformers 5.5.0 trl 0.24.0 peft 0.19.1 accelerate 1.13.0 PyPI the same day: transformers 5.15.0, trl 1.9.2, peft 0.20.0, accelerate 1.14.0. The canary moved peft and accelerate to genuine latest and moved transformers and trl not at all, which is what made it look like it was working. zoo pins transformers <=5.5.0 and trl <=0.24.0, ten minors and a whole major behind, so this CI could not detect a transformers 5.6+ or trl 1.x regression: it never installed one. The new frontier leg upgrades transformers and trl with their dependencies and without zoo in the resolution, which clears the cap because pip enforces only the requirements of packages in the resolution. It runs the same SFT payload as control and canary, so all three are directly comparable. I expected it red. It is green. On a real T4: transformers 5.15.0, trl 1.9.2, datasets 5.0.1, ten steps, canary emitted, and two fresh processes agreeing bitwise. Unsloth trains and generates correctly a whole trl major above what zoo's metadata permits. Getting there took two kernels. The first used --no-deps plus a blanket tokenizers upgrade and died before running anything, because an unbounded upgrade overshoots the ceiling transformers declares: tokenizers<=0.23.0,>=0.22.0 is required, but found tokenizers==0.23.1 safetensors>=0.8.0 is required, but found safetensors==0.7.0 A test pins that shape out, along with the zoo-in-the-resolution mistake that would silently turn this into a second canary. It goes on the second kernel's idle T4, so it costs no quota: a Kaggle session bills wall clock once, not per card. What it does NOT catch is recorded in the leg: the loss trajectory is not the control's, and step 1 is computed before any update on identical weights, data and seed, so 10.3222 against 6.4367 is a different loss function rather than drift. Both converge. Which objective is intended is separate work. * Install a CUDA llama.cpp before asserting the export used one Four hardware runs reported llama_cpp_install_kind=None and failed the GGUF export assertion for it. The reason was never subtle: nothing had ever installed a llama.cpp under STUDIO_HOME, so the assertion was measuring an absence rather than a selection. install_llama_prebuilt.py resolves a real "linux-cuda" kind on an x64 CUDA host, so the bundle was available the whole time and simply never fetched. The install runs before the server, so the export route sees a llama.cpp that was already there rather than one appearing underneath it, and it is its own assertion because "the CUDA bundle would not install on this box" and "it installed and the export against it failed" are different findings. Its result is deliberately not checked by execute(). A box where the bundle cannot be installed still runs inference, tool calling, training and the UI, and still reports the export failing for the reason it actually failed. Three tests. One of them CALLS install_llama_cpp rather than reading its source, because run() in this payload already applies capture_output and text and passing either again is a TypeError that no source-reading test would catch -- which is how the first draft of this was written wrong. * Record what install.sh left behind before replacing it build_kernel.py runs install.sh --local, whose own comment says it puts a llama.cpp on disk. Four runs still reported install_kind=None, so either it never installed one or it installed something else. Fetching a CUDA bundle fixes both cases and would hide which one it was, so the prior state is recorded first and the original question stays answerable. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Look for llama.cpp where it is, not where the payload assumed Five hardware runs reported llama_cpp_install_kind=None and failed the GGUF export for it. There was no missing llama.cpp. install.sh --local installs one into ~/.unsloth/llama.cpp, which is what install_llama_prebuilt.py's own default resolves to, and this payload read STUDIO_HOME/llama.cpp, a path nothing ever wrote to. Adding an explicit install last commit is what proved it, because it recorded what was there first. The installer selected the right bundle for a T4 accept ...linux-x64-cuda13-older.tar.gz coverage=75-89 supported=75,80,86,89 reject ...linux-x64-cuda13-newer.tar.gz reasons=missing_sms=75 out_of_range_sms=75 and then said existing llama.cpp install already matches selected release b10360-mix-87da1a2; skipping download and install while install_kind_before and install_kind_after at the requested directory were both None and the return code was 0. So a CUDA llama.cpp was present the whole time, the sm_75 selection works, and only the assertion was wrong. llama_cpp_marker checks STUDIO_HOME first, then the canonical location, and returns None when neither has a marker so a real absence stays a real absence. Against the exact situation the last run recorded, the old read answers None and the new one answers linux-cuda. Separately, and not fixed here: install_llama_prebuilt.py --install-dir X exits 0 having written nothing to X whenever the canonical directory already holds a matching release. A caller who names a directory and gets a success with an empty directory has been told something untrue. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Read the field the marker has, not the one it never had Six hardware runs reported llama_cpp_install_kind=None and failed the GGUF export assertion for it. There was no missing llama.cpp and no wrong directory. install_llama_prebuilt.py writes install_kind ONLY into the JSON its resolver prints to stdout. The marker it writes to disk records requested_tag tag release_tag published_repo asset force_cpu llama_backend asset_sha256 source_* ggml_tree bundle_profile runtime_line coverage_class install_fingerprint prebuilt_fallback_used installed_at_utc and no install_kind at any point. So reading install_kind from the marker answered None for every bundle on every box, including a working CUDA one, and the assertion turned that into a failure. The last run had the installer saying "existing llama.cpp install already matches selected release b10360-mix-87da1a2; skipping download and install" while the payload reported nothing installed. runtime_line is the field that answers the question: cuda12 / cuda13 for the CUDA bundles, and the backend name otherwise. It is written into the same dict as the asset, so the two cannot disagree, and asset is kept as a fallback for a marker too old to carry it. is_cuda_install now matches the runtime line's shape rather than a fixed set of names. A fixed set needs a new entry per CUDA major and fails CLOSED when it lacks one, reporting a working cuda14 install as not-CUDA, which is the same failure this commit is undoing. The existing test wrote {"install_kind": "linux-cuda"} into a marker and asserted it came back. That is how this survived: the test agreed with the payload about a field neither read from reality, and stayed green through all six runs. It is rewritten against a marker the installer actually produces. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Measure the load, not the unload it follows The GGUF export probe reported device_vram_delta_mib: -1866.0 and was scored as a load that never reached the GPU. It had not: a 3004 MiB chat model was evicted inside the same window the 531 MB GGUF loaded in, because POST /load with force unloads first and the baseline was sampled before the request. Unload explicitly, wait for the reading to stop falling, then take the baseline. Best-effort throughout, since the delta is evidence and not the assertion: a refused unload or a missing nvidia-smi leaves the probe running. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Read the status fields the response actually declares Run 8 shipped the unload-before-baseline fix and came back with device_vram_delta_mib: -1866.0, byte-identical to run 7. The fix ran and did nothing: it looked for model_path, model and active_model_name, and InferenceStatusResponse carries none of those, so no model was ever identified and no unload was ever sent. Read model_identifier (documented as the loadable identifier, which is what /unload's model_path wants), falling back to active_model and then to loaded[0]. A new test parses the field names out of InferenceStatusResponse and asserts the payload only reads names it declares, so the next guess fails in the suite rather than forty minutes into a kernel. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Delete the Kaggle kernel when the launcher is killed, not only when it returns release() is the budget control: a kernel left behind bills to its own ceiling with nobody reading the result, and that ceiling has been observed not to stop a wedged one. But it was reachable only from finish(), which runs only on paths that RETURN. KeyboardInterrupt was explicitly re-raised past it, and the default SIGTERM disposition exits without running atexit or finally. So Ctrl-C, `kill`, and a cancelled GitHub Actions workflow all left the kernel running. The file's own comment claimed the kernels were released "on this path and on every other", which was true for returns and false for signals. Four paths, four covers: normal return / handled error finish() unhandled exception atexit Ctrl-C, kill, Actions cancel SIGINT/SIGTERM/SIGHUP handlers kill -9 the orphan sweep at the next launch The signal handler re-raises through SIG_DFL so the exit status still reads "killed by signal N"; exiting 0 would make a cancelled job look completed. kill -9 cannot be caught, so a pushed slug is now recorded to logs/kaggle_inflight.json the instant it exists, and every launch sweeps entries whose owning pid is gone. Keyed on pid deliberately: an entry whose owner is still alive belongs to a run in progress, and deleting it would destroy a concurrent run and report the absence as a failure of the code under test. A delete that fails keeps its entry rather than forgetting it. The sweep runs before pushing, so reclaimed session slots are available to the run doing the reclaiming. Tests drive real subprocesses and real signals, since the property is about process death. Verified failing against the previous behaviour: SIGINT, SIGTERM and the unhandled-exception case all leave the kernel behind without these handlers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only forget a Kaggle kernel once it is actually gone kaggle kernels delete answers a transient rejection with a nonzero exit and subprocess.run does not raise on that, so both cleanup paths recorded a refused delete as done: sweep_orphans dropped the registry entry and release_kernels set released=True, leaving a running kernel billing with nothing left to reclaim it. Check the return code in both. Two more holes in the same registry. result["kernels"] was published after the whole push loop, so a SIGTERM once the first kernel was up found no list and deleted nothing; it is now the same list object, assigned before the loop. And --keep-kernel left an entry naming a pid that dies with the launcher, so the next sweep called the retained kernel an orphan and deleted exactly what the flag asked to keep. * Stop the Studio leg passing on evidence it never had Six ways this job could go green without measuring anything. The evidence bundle was never unpacked: the launcher writes each kernel into kaggle_evidence/<slug>/ and the workflow hands collect_evidence.py the parent, so a top-level glob found nothing and every run reported that the payload emitted no bundle. Recurse, the way extract_reports already does. An install.sh regression read as infrastructure. sh() raised SystemExit, so papermill stopped before any T4_SMOKE_REPORT existed, the launcher filed the run as infra and the reporter exited 0 -- silently passing the installer failures this workflow's path filter selects for. Failures of the checkout under test now emit a failing report first; a missing GPU or a clone that would not download keep the no-report infra path. A payload killed by the driver deadline does the same, but only once the payload itself had started. The process-level VRAM probe could never fire: InferenceStatusResponse declares neither llama_server_pid nor pid, and FastAPI drops what the response model does not declare, so the lookup was always None. Find the llama-server processes in /proc instead. Offload evidence outlived its load. offloaded_layers() takes the last match in the whole log, so a reload that logged nothing inherited the chat model's line and passed on evidence from a different model. Read each load forward from a mark taken just before it. A diverged run counted as a trained one. A T4 has no bf16, and an fp16 run that goes NaN still reaches completed and still saves an adapter, while trained_steps counted anything that was not None. Only finite losses count, and non-finite ones are named as a failure. An export slower than its HTTP timeout crashed the payload. The route blocks for the whole export and the transport timeout is 900s against a 1200s deadline, so the TimeoutError escaped the StudioError handler; it is now treated as in-flight and the status poll decides. Alongside: the restart that re-seeds the account no longer truncates the backend log of every earlier assertion, the over-cap evidence fallback ships the logs it says it ships rather than the report alone, the result line is printed after the evidence is packaged so a packaging crash cannot publish a pass, the training and export polls stop when Studio dies, Studio is launched from the interpreter running the payload rather than whatever is on PATH, the gate is told this leg pushes one kernel rather than the default two, and the advertised opt-in label can start the workflow. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the encoding on every checked-in read, and survive a box with no nvidia-smi tests/test_source_read_encoding.py fails on 33 read_text() calls in the two Kaggle harnesses: unpinned text I/O on a repo file is cp1252 on a stock Windows install and dies the moment that file gains a non-ASCII byte. The second one is the payload's own. environment() runs inside finish(), on every path out of the run, so a session Kaggle handed no driver raised FileNotFoundError from gpu_inventory() and ended with a traceback and no report at all. run() now answers a missing binary the way a failing one does, and the preflight says there is no GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Spend a Kaggle session on the opt-in label, not on every label after it Two ways this leg spends or loses a GPU session it should not. GitHub has no per-label filter on the pull_request event, so adding `labeled` to the activity types starts the workflow for EVERY label added to an affected pull request. The gate is handed the whole label list, so once kaggle-studio-gpu-ci sits on the pull request it reads every one of those events as forced, and a routine label change launches another 70-minute Kaggle run. Guard the gate job on the label that fired the event; push and the other pull_request activities carry no `github.event.label` and fall through unchanged. The other way is the reverse: a run that should be red exits 0. The dependency probe raises out of the cell when the Studio venv reports no CUDA, and with no T4_SMOKE_REPORT the shared launcher files the run as `infra`. install.sh --local resolving a CPU-only torch is exactly the CUDA install regression this workflow's path filter selects for, and it took that path. Ask nvidia-smi which case it is: a GPU the host can see and the venv cannot use is a payload failure, and no GPU at all stays infra. * Treat a stalled kaggle push as transport, not as a red verdict Every other Kaggle transport failure -- a 503, a refused push, a capacity rejection -- returns a reason from push() and exits 0 as infra. A push that stalls past the 600s ceiling did not: TimeoutExpired escaped launch.py, so the process ended before finish() could write launch_result.json, the launch step exited nonzero and GitHub marked the job red. The reporter that reads that file and calls the run NOT RUN never got a say, and a red there says the code under test failed when nothing ever reached a T4. Catch it in push() and report push_timeout. * Do not let a truncated copy of a chunk overwrite the complete one * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the encoding on every checked-in file the T4 harness reads Repo tests (CPU) is red on this branch: test_source_read_encoding.py::test_checked_in_file_reads_name_an_encoding AssertionError: 17 file reads in the test trees touch a checked-in file with the platform default encoding, so they break on Windows as soon as that file gains a non-ASCII byte. All 17 are in tests/kaggle/test_t4_smoke_harness.py, which this branch adds, so this is ours and not inherited. The reads pull in the workflow YAML, the smoke scripts, the committed reference JSON and ci-preempt.json, and none of them said what encoding to read them as. On Linux that resolves to UTF-8 and nobody notices; on a Windows runner it resolves to the ANSI code page and the first non-ASCII byte in any of those files takes the test out with a UnicodeDecodeError that has nothing to do with what the test is checking. Every one now passes encoding = "utf-8", matching how the rest of the suite spells it. Reads of files the test itself just wrote under tmp_path are left alone: the gate only covers checked-in files, and those are ours end to end. test_source_read_encoding.py 1 passed tests/kaggle/test_t4_smoke_harness.py 148 passed * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Flatten Kaggle's kernel log before extracting the reports from it Kaggle's kernels/output returns the log as a JSON array of {stream_name, time, data} records, one record per line, so no line in the file starts with T4_SMOKE_REPORT. extract_reports read it verbatim, so the log fallback -- the one that covers a run whose executed notebook never came back -- recovered nothing and a failed assertion was scored as infra with the workflow exiting 0. report.py::kernel_log_text and collect_evidence.py::iter_text already flatten the records; extract_reports was the reader that did not. The records are also cut by write rather than by line, so the join has to happen before any line splitting. * Keep the slug a timed out kaggle push may have created A push whose CLI stalls past the 600s ceiling says nothing about whether Kaggle took it. Kaggle can accept the push and start the kernel billing before the response is lost, and the branch returned infra while keeping neither the slug nor an inflight entry, so finish() had nothing to delete and the next launcher's orphan sweep had nothing to see. The kernel ran to its ceiling with nobody reading the result. The slug is ours and already decided at that point, so record it in the inflight registry and hand it back as orphan_slug: release_kernels deletes it on the way out, and a launcher killed first leaves the entry for the next sweep. It is deliberately not returned as slug, because the caller must not wait on a kernel that may never have been created. * Align the Kaggle budget guide with the workflow it documents The guide named --percent 15 and --reserve-hours 25 against a 30h account; the workflow runs 40 and 20 against the 60h figure measured on 2026-08-11, and it is the only consumer of that quota today. Following the guide would have tuned the reserve from a baseline that no longer exists. Re-derived from the same live numbers as the workflow header, and says which of the two is the source of truth when they disagree. * Stand the gate down on any unreadable kernel status The concurrency verdict refused only when EVERY in-window status came back an error. One unreadable status among readable ones was treated as an idle account, and driving concurrency_verdict with {busy: [], own: [], foreign: [], surveyed: 2, unreadable: 1, complete: True} returned (True, ""). That kernel can be the human session the zero-foreign policy exists to yield to, and proceeding takes the account's last slot: the first push wins it, the second comes back at capacity, and the run reports a subset of its legs. A 404 is kept separate and stays benign. A deleted kernel is not an unknown state and the launcher deletes every kernel it pushes, so blocking on that would wedge the gate shut rather than make it careful. * Close six gaps in the Kaggle T4 notebook CI workflow - The pull_request trigger used GitHub's default activity types, which are opened, synchronize and reopened only. The kaggle-t4-ci label the gate advertises as a human override could therefore never start a run on its own. Subscribe to labeled as well. - pyproject.toml was in neither paths filter, though every payload installs the commit under test as a distribution built from it. BUDGET.md counted it as a watched path already. - The job deadline of 120 minutes was shorter than the launcher's worst case. Two sequential pushes can take 2 x (4 x 600s + 315s of backoff) and the 5400s polling window only starts after them, so about 187 minutes with the deletions; a runner killed at 120 takes finish() -> release() with it and leaves pushed kernels billing quota. 200, with the arithmetic written down beside it. - The gate surveys quota and in-flight kernels before t4-smoke queues on the account-wide concurrency group, so a second sampled run could launch on an approval a full run old. Re-ask with the slot in hand, immediately before the push; --force true skips the sampling draw and nothing else. - --zoo-ref main let each payload's own pip resolve a different unsloth-zoo commit, including between the control and the canary within one session, and zoo is not in pins/control.txt either. Resolve it once with git ls-remote and pass that SHA to every leg. - A dispatch with max_steps != 10 still shipped the 10-step reference, and check_reference reports step_count_mismatch as a failure, so the run could only ever go red on arithmetic. Drop the band automatically when the counts differ, and say so. Also: check out the same head SHA the launcher installs, so the harness and the package under test are one snapshot rather than the merge tree and the head; and run tests/kaggle/test_t4_smoke_harness.py on the runner before anything is built or pushed, since testpaths limits pytest to tests/security and nothing else collected it. * Read back what the Kaggle kernel actually reported Two ways a real payload verdict was being lost in transport, both ending in verdict=infra and a green workflow. The kernel log fallback could never work. Kaggle's kernels/output hands the log over as a JSON array of {stream_name, time, data} records, which is why report.kernel_log_text flattens it before reading; extract_reports scanned the file as text, so no line began with the report prefix and the fallback that exists for the run whose notebook never came back found nothing. Record boundaries are not line boundaries either, so the records are joined before the scan rather than read one at a time. push() documented a fresh slug per attempt and used one for all four. Pushing to an id that already exists does not replace it: Kaggle files a new version and starts a second batch session, and kernels status and kernels/output send no version label, so they answer for the newest session only. A retry after a lost response therefore collected the wrong execution's evidence while the first kept a session slot and billed unseen. Each attempt now takes its own slug and discards the previous one first, and every slug filed is returned and recorded, since a push that reported an error may still have landed. * Give each payload its own everything, and report a probe failure Four ways the two payloads of one kernel were not actually isolated, plus the import probe exiting without a verdict. The probe was the P1. When the commit under test breaks import unsloth, the verify cell raises before the run cell can write a report, the launcher extracts none, and no reports is classified as infra -- so a deterministic import regression exits green. A probe failure is a verdict, not missing evidence, so it now emits its own failing report. nvidia-smi failing or a one-GPU allocation fell back to N_GPU = 1 and pinned both payloads to device 0. Each child still saw exactly one card and passed its own visibility assertion, so the contention came back looking like a code failure. A shortfall is infrastructure and stands the kernel down before a thread starts, which is what the comment above it already claimed. A payload whose virtualenv could not be built kept the base python3 kernel and installed into the shared system site-packages. That is the one thing the per-child venv exists to prevent: the legs install deliberately different library sets, so the last writer wins and the resulting import error reads as a regression. The payload is skipped instead. Both payloads materialised into one source directory and compiled into one unsloth_compiled_cache. The sources are byte-identical copies but write_bytes truncates first, and the compile cache is a relative path resolved against a working directory both papermill children inherit while compiling the same modules against different transformers versions. Each payload now owns both, and the prune globs still reach them. Finally, --smoke-args is appended after every leg's own arguments, so the workflow's --max-steps 10 landed after the gpt-oss leg's --max-steps 3 and argparse took the last one. The 20B leg was training for ten steps rather than the three that were measured to fit. A shared argument no longer reaches a leg that sets that option itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 06:30:18 -07:00
Daniel Han	bb11b371e7	Run deterministic notebook smoke tests on real Kaggle T4s (#8440) ... * Run deterministic notebook smoke tests on real Kaggle T4s The notebooks are written for a T4: sm_75, fp16 with a gradient scaler, xformers rather than flash-attention, 16GB. Nothing in CI runs on that card, so a regression that only appears there reaches users first. Adds a gated job that launches one Kaggle GPU session and runs the smoke payload once per T4. The account has a 60h weekly budget shared with other consumers, so the job is a deliberate minority consumer of it: paths filters (only 9.9% of a week's 444 main commits touch unsloth/), a ~10% sampling draw keyed on the run id so a re-run cannot reroll, a quota floor read live from Kaggle's own API, and a stand-down when the account already has a kernel in flight. Expected cost is 5-7 GPU-hours a week. Only a payload that ran on a T4 and disagreed with its assertions turns the check red. Throttling, the 2-kernel concurrency cap, a dead session or a download that would not complete are all warnings, because a free external service blocking merges would be ignored within a week. The payload asserts three things, worth different amounts. Two fresh processes must agree bitwise on per-step loss and grad_norm; that one is exact, and it is fresh processes because two in-process cycles disagreed from the first step while two processes agreed on all ten. Greedy decoding must emit the canary string exactly, which catches trained weights never reaching the generate call. Metrics must sit inside a band around a committed reference, which is a tolerance and never an equality, since fp16 reduction order moves with the driver and the libraries. Ten steps rather than three: under fp16 the scaler skips the first two overflowing steps, so a three-step run lands one real update and the canary never forms. The dataset is supplied as prompt/completion columns so the loss falls only on the answer. * Make the payload cells the T4 rejected actually run there Two launches on real Kaggle T4s died in the scaffolding before a training step ran, both of them in generated code that nothing had ever parsed. The dependency probe imported unsloth_zoo before unsloth. Zoo's __init__ ends with a find_spec("unsloth") guard and raised "Please install Unsloth via pip install unsloth" on a session where unsloth was installed and imported cleanly one entry later, so the payload aborted over a dependency that was not missing. Probe unsloth first, and invalidate the import caches after the pip installs that this interpreter did not see at startup. The reference argument was generated as a shell fragment and spliced into the middle of a Python list literal, with doubled braces that left ROOT unsubstituted. The cell was a SyntaxError, and the workflow passes --reference on every path, so no payload could ever have reported. Verified locally by executing the regenerated run cell end to end: two fresh processes, bitwise identical metrics, canary exact. * Bound the in-flight kernel survey by session length, not kernel count The concurrency check status-checked the twelve most recently run kernels, which is a sample rather than a search: a kernel that started three hours ago and is still running is missed the moment twelve newer ones have since run, and the push then fails at the account capacity cap and is reported as infra. Kaggle kills a notebook session at twelve hours, and the listing is sorted by last run time, which for an unfinished kernel is when it started (measured: a kernel pushed at 10:05:19Z lists as 10:05:19.297). So walking until the first entry older than that ceiling covers every kernel that can still be in flight, and stopping there is exhaustive rather than arbitrary. An hour of slack is added for clock skew. Two silent ways to read an unanswerable question as a green light are closed with it. Hitting the page cap now marks the survey incomplete, and a survey where no status could be read at all is unknown rather than idle. Both stand down, which is the cheap direction: the next commit draws again in minutes. On the live account this now costs three status calls instead of twelve. * Stop the reference band check from passing NaN unconditionally The band comparison did its NaN handling by accident, and it got it wrong. Under fp16 the gradient scaler logs a NaN grad_norm on every step it skips, so a committed T4 reference genuinely contains NaN. Left to the arithmetic, abs(x - NaN) is NaN and NaN > tolerance is False, so those steps passed whatever they held, including the case the check exists for: a step that used to overflow and no longer does. Compare NaN to NaN as equal and NaN against a number as a deviation, and treat a field logged on one side only as a deviation too, since no tolerance covers a change in shape. The mapping from an out-of-band verdict to the failure list moves into reference_failures() so the path that turns the job red can be tested without a GPU. The references README now says plainly that no reference is committed, that one must not be borrowed from other hardware, and how to lift one out of a green run's evidence rather than spending a session on it. It also records a measurement: the absolute floor never engages on this trajectory. The smallest value on the ten-step curve is a loss of 0.1428 against a floor of 0.05, so the floor changes no verdict today and is kept for a configuration that does go lower. * Name the cause when a kernel finishes and reports nothing A kernel that ends COMPLETE with no payload report is the hardest outcome to read: the summary has no metrics to show and the reason sits in an artifact nobody downloads. It is also the outcome both real T4 failures took. Kaggle returns the kernel log as a JSON array of stream records rather than as text, so the interesting line arrives split across dozens of them and reading the file directly shows a wall of JSON. Flatten it, filter to the driver and payload sentinels plus tracebacks, and fold the tail into the job summary. Checked against the evidence of a real failed run: the SyntaxError that killed it is now visible without leaving the summary. * Cover the four T4 CI fixes with CPU-only tests The generated cells are now compiled on both the with-reference and without-reference paths, which is what would have caught a SyntaxError that instead cost a Kaggle session, and the dependency probe's module order is asserted. The in-flight survey gets a fake Kaggle client and the case that motivated it: one kernel still running behind forty newer finished ones, which a count-bounded scan misses and a time-bounded scan cannot. Also the window boundary, timezone-aware and missing timestamps, the page cap, and the difference between some statuses unreadable and all of them. The band check is now proved to fail rather than assumed to work: a perturbation half a band-width past the limit, a moved scaler skip pattern in both directions, a field that stopped being logged, and a length mismatch, each asserted through to the failure list. Three tests perturb the committed reference itself and skip, saying so, until one exists. 41 passed, 3 skipped. * Parse every generated cell on every build path before spending a session Two of the three Kaggle sessions spent so far died on generated code that nothing had ever parsed. The compile check now builds all three paths the generator has, using the workflow's own argument list verbatim, including the one where a committed reference exists and is carried inline as a fourth file: that path becomes live the moment a green run supplies the file, and would otherwise first be exercised on Kaggle. Parsing alone is not enough. A template hole that substitutes to a bare identifier parses and then NameErrors, which costs the same session, so every cell is also checked in execution order for names nothing before it defines. Both checks were confirmed to go red: the first on the exact shell-fragment-in-a-list-literal defect that cost session three, the second on a run cell reading a name no cell binds. The carried sources are also decoded the way the kernel will decode them and compared byte for byte with the repo, so the T4 cannot quietly run something other than what is committed. * Commit the T4 reference the first green Kaggle run produced Kernel danielhanchen/unsloth-t4-ci-e3c6661f, terminal state COMPLETE, both payloads passing on their own Tesla T4 of one session. Two fresh processes agreed bitwise on all ten steps on both cards, max_abs_diff exactly 0.0 for loss and for grad_norm, and the two cards independently produced the same ten values as each other, so four processes agree rather than two. All four cycles emitted the canary exactly. The scaler skipped steps 1 to 3 every time, which is the behaviour the ten-step length exists to accommodate. The file is reports[0] of that run, copied whole by the recipe the README already documented, and nothing in it is from other hardware. One assumption the run contradicted, now recorded rather than repeated: the session had no xformers. unsloth installs --no-deps, unsloth_zoo does not carry xformers and the Kaggle image does not either, so the banner read Xformers = None and this trace is the fallback attention path. Adding xformers to the install would move these numbers and require a recapture. The floor's smallest observed value moves from 0.1428 to 0.0871, still above the 0.05 denominator floor, so the floor stays inert and the test that re-derives that keeps passing. The three tests that perturb the committed reference now run instead of skipping. 45 passed, 0 skipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the in-flight tolerance the Kaggle gate stands down on The gate declines to launch whenever any kernel of the account is in flight, which is stricter than Kaggle's 2-kernel cap requires. That is the intended policy, so it is now a named constant with the tradeoff written next to it rather than an implicit property of the code, and a --allow-in-flight argument that cannot be raised to the cap itself. Default behaviour is unchanged. Also files the workflow under 'never' in ci-preempt.json: cancelling a run does not stop the Kaggle kernel it already pushed, and the orphan then bills quota to its own ceiling with nobody watching. * Run three steps, and refuse a reference captured at another count max_steps drops from 10 to 3. Measured consequence: under fp16 the gradient scaler starts at 65536 and skips every step it overflows on, and the committed reference shows steps 1, 2 and 3 all skipped, so a 3-step run of the old configuration applied zero optimizer updates. --init-loss-scale pins the scaler below the overflow point so those three steps are real updates, and optimisation_failures() fails a run whose every step was skipped rather than letting it report as training. The committed reference was captured at 10 steps and no longer describes the run. check_reference now reads the max_steps the reference records and refuses to compare across counts, as a failure with both counts named -- including when the reference does not say what it was captured at. A quiet pass there would be a green check comparing a run against a curve it has nothing to do with. The reference is therefore stale until one recapture run. The workflow gains skip_reference_band for exactly that dispatch, and the recipe is in references/README.md. Also brings the workflow onto the repo's conventions: pinned action SHAs matching the rest of CI, a workflow-level concurrency group that deliberately does not cancel in progress, a fork guard on the gate job, typed dispatch inputs and an artifact name without the run id. * Keep the run at ten steps, and say why three does not work The committed reference is the evidence: the fp16 scaler reports grad_norm NaN at steps 1, 2 and 3 and a finite value from step 4. It starts at 65536, halves on each overflow, and skips the step it overflowed on, so a 3-step run applies zero optimizer updates. Loss stays around 10, the canary never forms, and the band would compare three points of a curve that never moved. Shortening also saves nothing worth having: a launch costs about 0.08h and that is almost all pip install, not training. The step-count guard, the all-steps-skipped guard and --init-loss-scale all stay, since they are what makes a shorter run possible later and what stops a vacuous one reporting green. The pin is off by default so the committed reference keeps applying. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Turn one T4 payload into four, and pair a pinned control with a canary The workflow ran a single tiny SFT payload on both cards of one Kaggle session and spent about 1.4 GPU-h a week doing it. The budget is now 40, and the question asked of it is wider: catch regressions in trl, transformers, accelerate, peft, bitsandbytes, torch and vllm, and catch torch.compile falling over on sm_75. Four legs now, two kernels of two T4s each, described once in .github/scripts/kaggle_t4_ci/legs.py rather than restated in YAML: control the existing tiny Qwen2.5-0.5B determinism run, pinned canary the same run on the newest library set Unsloth's own constraints allow gptoss gpt-oss-20b LoRA, for torch.compile and the forced float32 path grpo Qwen3-4B GRPO through a vLLM engine on the same card control and canary are one instrument rather than two tests. Same payload, same seed, same data, same step count, on the two cards of the same session; the only difference between them is the installed versions. A red canary beside a green control is a library release, and the job summary already puts the two version sets side by side so the bisect is a diff rather than an investigation. The reference band applies to the control alone: two library sets do not produce one fp16 trajectory, so the canary asserts what does not depend on versions instead. The pins are evidence rather than preference. transformers and trl are the pair the committed reference was captured with; peft, accelerate and bitsandbytes come from probe kernels 8161ceb9 and 7ab727f1, two independent T4 sessions that ran the identical base install and agreed. torch stays unpinned and recorded, because it comes from the Kaggle image and replacing it is the failure mode the grpo leg spent two probes on. The gate now separates this workflow's own kernels from everybody else's. It takes both of Kaggle's concurrency slots, and only when the account is otherwise idle: one kernel belonging to anyone else still stands it down entirely, which is the same policy as before stated at the right grain. * Leave the vLLM leg unwired, and stop trusting Kaggle's own kernel timeout Three probe sessions tried to run the Qwen3-4B GRPO leg on a T4 and none of them reached a training step. None of them failed for a reason to do with sm_75, memory or GRPO either: vllm==0.11.2 pins torch==2.9.0, so the leg has to replace the Kaggle image's torch, and that is what killed all three. With the image's site-packages visible, pip treats torch's pinned NVIDIA runtime packages as already satisfied by the copies belonging to 2.10, and the result is a torch that installs cleanly and cannot be imported -- libcusparseLt.so.0 first, then ncclCommWindowRegister once that one was named explicitly, identically on vllm 0.11.2 and 0.15.1. An isolated venv resolving the whole stack got further and then the session wedged. So the leg keeps its payload, its assertions and its install, and moves to legs.UNWIRED with the three kernel slugs and what each one measured. Wiring it now would make the check permanently red and spend the budget doing it. The second kernel keeps its spare T4, which costs nothing, and that is where the leg goes when the install works. The third probe also cost about an hour of quota teaching us that the push-time kernel timeout is not a budget control. Pushed with -t 5400, its own nbconvert crashed at t=406s and the session sat in RUNNING for over two hours -- past that ceiling and past the launcher's own deadline, because one status call had blocked with no socket timeout to bound it. Deleting the kernel by hand took the account's used-hours figure back down, which is the measurement that settles which control is real. The launcher now sets a socket timeout and deletes every kernel it pushed on every path out, and the workflow header says which of the three bounds is load-bearing. Sampling recomputed on the payload that actually ships: 40%, from measured per-kernel cost rather than the old estimate, with the note that wiring a fourth leg means recomputing it rather than editing one line. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-solve the GRPO leg's install so it stops replacing the image's torch Three probe sessions never reached a training step, and all three died the same way: vllm==0.11.2 pins torch==2.9.0, so the leg had to replace Kaggle's torch 2.10.0+cu128, and pip treats the image's NVIDIA runtime packages -- which belong to 2.10 -- as satisfying the new torch's pins. That imports as libcusparseLt.so.0 missing, then as libtorch_cuda.so undefined symbol ncclCommWindowRegister one package along, then as an isolated venv that spent an hour of quota resolving a CUDA stack and never produced payload output. None of that is a question about sm_75, memory or GRPO. It is one line: no vLLM release in the 0.11-0.16 range pins the torch the image ships. Releases 0.17.0 through 0.19.1 pin torch==2.10.0 exactly, so pin 0.19.1 and the leg replaces nothing, keeps system_site_packages, and installs like any other. Drop xformers with it. Its vLLM attention backend was deleted in 0.12.0, so it would be a package nothing selects. sm_75 has no FlashAttention and no FlashInfer, and the ladder in vllm/platforms/cuda.py falls through both to TRITON_ATTN; the leg names that in VLLM_ATTENTION_BACKEND so a release that reorders it goes red here rather than quietly selecting something else. 7.5 is still in CUDA_SUPPORTED_ARCHS at v0.19.1 and fp16 is a supported dtype below capability 8.0. Still unwired. Two things are static evidence rather than a run: whether 0.19.1 starts on sm_75 at all, and whether 8GB of 16-bit weights plus an engine plus a LoRA trainer fit in 14.56GB. UNWIRED says so in those words, and a test asserts the note still names the pin and still says what is unknown, so nobody wires this off prose that outlived its measurement. Also ports the vacuous-test fix from the stacked Studio branch: pre-commit.ci reformatting default=10 to default = 10 stopped the payload regex matching, so the step-count agreement check was comparing nothing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop the GRPO leg JIT-compiling flashinfer on a machine that cannot link it First probe on the re-solved install, kernel unsloth-t4-ci-e2d9ce9b. The install works: torch stayed at the image's 2.10.0+cu128, vllm 0.19.1 installed and imported, xformers was absent as intended, and the payload ran on a real Tesla T4 reporting capability 7.5 with TRITON_ATTN accepted. It reached engine construction, which is further than any earlier probe got by the whole width of the install. It died in flashinfer 0.6.6's JIT, and not for an sm_75 reason. All three .cu files compiled cleanly for -gencode=arch=compute_75,code=sm_75. The LINK failed: /usr/bin/ld: cannot find -lcuda -L/usr/local/cuda/lib64/stubs is already on the command line, so the image has no driver stub libcuda.so, only the runtime libcuda.so.1. Nothing about that is fixable from a payload. Set VLLM_USE_FLASHINFER_SAMPLER=0 so nothing JITs. The sampler has a native path, and skipping the build also saves a four-file nvcc compile inside a session billed by wall clock. UNWIRED records what the probe measured and narrows what is left to one question: whether the engine builds now. * Make -lcuda resolvable so flashinfer can link what it compiled Second probe, kernel unsloth-t4-ci-916d5986, failed identically to the first with VLLM_USE_FLASHINFER_SAMPLER=0 set. That is the informative part: the JIT is not reached only through the sampler, so switching off one consumer is whack-a-mole. The failure was never sm_75. Both sessions compiled all three .cu files cleanly for -gencode=arch=compute_75,code=sm_75 and died at the link: /usr/bin/ld: cannot find -lcuda -L/usr/local/cuda/lib64/stubs is already on the command line; the image ships no libcuda.so at all, only the versioned libcuda.so.1, and the linker will not resolve -lcuda against a soname. Normally the toolkit's driver stub fills that gap; this image has the directory and not the file. So symlink libcuda.so at the real driver in a scratch dir and put it on LIBRARY_PATH, which gcc and ld search for -l. No root, nothing written into /usr/local, and it fixes every flashinfer op rather than one. Linking against the real driver instead of a stub is correct here: the driver is present, which is the whole reason a stub would have substituted for it. Also widen the captured traceback to head AND tail. The last probe's 6000-char tail was entirely ninja's own output, so the Python frames naming the caller were exactly what got dropped. Probe 2 also measured the fit question the notebook raised: peak reserved 7.96GB of 14.56 with load_in_4bit false, so 16-bit weights are not what is short of room here. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only count a libcuda the linker will actually search for Kernel unsloth-t4-ci-d0d480b6: the shim reported already_linkable and did nothing, and the link failed anyway. It had found libcuda.so in /usr/local/cuda/compat -- which is real, and which nothing passes with -L. The failing ninja line names /usr/local/cuda/lib64 and /usr/local/cuda/lib64/stubs, and those are the only two directories that can answer -lcuda. So the precondition now checks exactly those two, and compat becomes what it is actually good for: a symlink TARGET when ldconfig and ctypes both come up empty. A library the linker will not search for is not a library the linker can find. Two tests: one reproduces the compat-only machine and asserts the shim is built anyway, one pins the searched directory list, since widening it is precisely how the check went wrong. * Give the base model a chat template, and record what the link shim did The link shim works. Kernel unsloth-t4-ci-27b0dc2e reported libcuda_shim: needed true, applied true, searched [/usr/local/cuda/lib64, /usr/local/cuda/lib64/stubs], real /usr/local/cuda/compat/libcuda.so and flashinfer is gone from the failure entirely. Memory went from 7.96GB to 11.36GB of 14.56, which is the vLLM engine actually loading, and the run reached trainer.train() -> _run_epoch -> training_step. Both questions legs.UNWIRED was holding this leg for -- does vLLM start on sm_75, does it fit in 14.56GB -- are answered by that. It then failed on the payload's own gap: ValueError: Cannot use chat template functions because tokenizer.chat_template is not set unsloth/Qwen3-4B-Base is a base model and ships none. The notebook this leg comes from solves it with an SFT priming stage that installs a template before GRPO starts; this leg has no priming stage, so it sets a minimal ChatML template directly and records in the report which of the two worlds the run was in. The base model stays. GRPO on an instruct model would measure the instruct tuning as much as the run, and the payload's rewards are format-and-digit rewards a base model can move inside three steps. A test renders the template rather than matching it as a string: a template that does not render would trade a failure at step 1 for a failure at step 1 with a longer traceback. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Wire the GRPO leg: it runs on a T4 and the rewards move Kernel unsloth-t4-ci-53efcc4e passed on a real Tesla T4. Step 2 logged reward_std 0.707 and grad_norm 0.772, peak 13.60GB of 14.56GB, three steps in 192s, and fast_generate returned coherent text. reward_std is the criterion rather than loss, because TRL's GRPO loss is ~0 by construction at num_iterations=1 and beta=0 and would pass on a run that learned nothing. The config in args is the one that fit, not the notebook's. Two probes with seq 2048, 4 generations, rank 32 and utilization 0.9 died in the backward at unsloth_zoo/gradient_checkpointing.py:1013, peaking at 15.97GB in 16-bit and 19.25GB in 4-bit. 4-bit alone is not the lever: it peaked HIGHER, because quantizing weights does nothing for activations while utilization 0.9 still hands vLLM ~13GB up front, and UNSLOTH_VLLM_STANDBY returns the weights during training but not the KV cache reservation. An earlier probe reported this as 'CUDA error: an illegal memory access was encountered' from empty_cache() inside vLLM standby. That was the async shadow of the same OOM; CUDA_LAUNCH_BLOCKING=1 produced the honest error. UNWIRED is now empty. The test that read its grpo entry is replaced by the invariants that outlive any one leg: every leg is wired or explained, nothing is both, and an unwired note says what is open. Two new tests pin the config and the vLLM pin, so restoring either to a value that OOMs fails here rather than on Kaggle. * Add a CUDA_LAUNCH_BLOCKING switch to the GRPO payload An illegal memory access is reported at whatever CUDA call synchronises next, which on the first T4 GRPO run was empty_cache() inside vLLM standby, nowhere near the kernel that faulted. Re-running with this flag produced the honest error instead: torch.OutOfMemoryError in the backward. That is what turned an unexplained CUDA fault into a memory budget, and the budget is now the leg's config. The environment variables are set before the first import torch in the process rather than merely before train(), because they have to precede CUDA context creation. It serialises every kernel launch, so it stays a switch and never a default, and it is recorded in the report because it changes what a timing means. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Write down where the Kaggle sampling percentages come from The percentages in the two workflows were arithmetic nobody could check. This records the measurements they came from, taken 2026-08-11 over a 28-day window: 985 commits to main, 1157 PRs, and 8 of 60 sampled PRs touching the paths a T4 training run could regress, so a 13% path-filter rate and roughly 75 candidate events per week. Against 30 GPU-h/week for one account at ~0.75 GPU-h per launch, the ceiling is about 33 launches per week if everything went to one workflow. The 15/5 split and the 25/10 reserve-hour floors follow from that, and the file says what would change them: a second account, a wider path filter, or a session cost above ~1 GPU-h. * Give the GRPO leg its own kernel: pairing it with gpt-oss fails Kernel unsloth-t4-ci-70a2f4eb ran the wired pair on the two cards of one session. gptoss passed: 32 unique graphs, 779 calls captured, 2 graph breaks, peak 12.78GB. grpo failed with the illegal memory access again, at exactly the 13.60GB peak at which it PASSED alone on kernel unsloth-t4-ci-53efcc4e, same config to the flag. Same peak, different outcome, so the constraint is not GPU memory: the driver pins one card per payload. gpt-oss offloads to host RAM and vLLM wants host RAM, and a Kaggle session has one host. A third kernel costs a queue slot rather than quota, since Kaggle runs two at a time and bills wall clock per session. That is cheaper than a leg which fails half the time for a reason the report cannot show. Two tests pin the rule in both directions: grpo must not share a kernel with gptoss, and control must still share one with canary, because those two are a matched pair whose comparison is the whole instrument. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Unwire the GRPO leg: running it alone reproduced the failure I blamed on pairing I gave grpo a kernel of its own earlier today on the reasoning that sharing a session with gpt-oss was what broke it. It had failed paired (unsloth-t4-ci-70a2f4eb) and passed alone (unsloth-t4-ci-53efcc4e), so pairing looked like the variable, and I wrote two tests pinning the separation. That reasoning was wrong. Running the leg ALONE again (unsloth-t4-ci-c98f14be, built from this branch with --legs grpo) reproduced the paired failure exactly: unsloth_zoo/vllm_utils.py:601 sleep() -> torch.cuda.empty_cache() torch.AcceleratorError: CUDA error: an illegal memory access was encountered engine_built false, peak 13.8GB/13.6GB of 14.56GB. The passing session and both failing ones agree to the digit on every recorded version -- torch 2.10.0+cu128, transformers 5.5.0, trl 0.24.0, peft 0.19.1, vllm 0.19.1, unsloth 2026.8.15, zoo 2026.8.10 -- on the same peak, with UNSLOTH_VLLM_STANDBY=1 throughout. One contrasting observation was never enough to blame a shared host. So what three sessions show is an intermittent illegal memory access on Turing, one pass in three, not a pairing effect. A leg that passes one session in three cannot tell CI anything: it would go red for a reason no reader could act on. grpo moves to UNWIRED with the three session ids and the open question written down, rather than back into a kernel. The note keeps what the leg already earned -- the vLLM/torch pin, the attention backend, the libcuda shim, the chat template, the T4-sized config -- so re-wiring means answering the IMA rather than redoing the install. test_grpo_does_not_share_a_session_with_gptoss is replaced rather than deleted: it asserted the opposite thing for the disproven reason, and would now pass vacuously since grpo is in no kernel at all. The replacement pins that grpo stays unwired and that the note still names all three sessions, so the evidence cannot be dropped on the way back in. 145 passed. * Record what launch blocking did to the GRPO leg: it removed the fault The UNWIRED note asked for one experiment: run the leg with --cuda-launch-blocking so the illegal memory access reports at its real call site instead of surfacing at torch.cuda.empty_cache(). Done, kernel unsloth-t4-ci-b1f23e34, and the answer is not the one the note expected. With blocking on there was no illegal memory access at all. engine_built true, three steps, same 13.8GB peak, same versions to the digit. A fault that disappears when the launches are serialised is a race, which is what one pass in four already suggested and which no amount of re-running the leg will pin down. The run also exposed a SECOND problem, and it is separate from the first. It failed on reward_std = [0.0, 0.0, 0.0] with grad_norm 0.0 at every step. The completions in the report are coherent prose, so this is not the model collapsing into degenerate output. It is two completions scoring identically. The leg runs num_generations = 2 because that is what fits on a 14.56GB card, and at two samples a tie on a coarse reward is ordinary rather than a bug. So the leg's pass criterion is fragile at the size it has to be to fit, which is worth knowing before anyone reads a red grpo leg as a regression. The note now records both, says which question each answered, and states the two that remain open: where the race is, and what pass criterion is honest at num_generations = 2. The test gains the fourth session id and a check that the note no longer reads as though launch blocking were still the next thing to try -- a stale "try this next" is how an experiment gets run twice. 145 passed. * Test the newest transformers and trl, not the newest zoo permits The canary leg is documented as "the newest permitted library set" and that is exactly what it installs: it resolves WITH unsloth_zoo in the resolution, so zoo's own metadata is a ceiling. Measured against the two runs that carried it: canary transformers 5.5.0 trl 0.24.0 peft 0.20.0 accelerate 1.14.0 control transformers 5.5.0 trl 0.24.0 peft 0.19.1 accelerate 1.13.0 PyPI the same day: transformers 5.15.0, trl 1.9.2, peft 0.20.0, accelerate 1.14.0. The canary moved peft and accelerate to genuine latest and moved transformers and trl not at all, which is what made it look like it was working. zoo pins transformers <=5.5.0 and trl <=0.24.0, ten minors and a whole major behind, so this CI could not detect a transformers 5.6+ or trl 1.x regression: it never installed one. The new frontier leg upgrades transformers and trl with their dependencies and without zoo in the resolution, which clears the cap because pip enforces only the requirements of packages in the resolution. It runs the same SFT payload as control and canary, so all three are directly comparable. I expected it red. It is green. On a real T4: transformers 5.15.0, trl 1.9.2, datasets 5.0.1, ten steps, canary emitted, and two fresh processes agreeing bitwise. Unsloth trains and generates correctly a whole trl major above what zoo's metadata permits. Getting there took two kernels. The first used --no-deps plus a blanket tokenizers upgrade and died before running anything, because an unbounded upgrade overshoots the ceiling transformers declares: tokenizers<=0.23.0,>=0.22.0 is required, but found tokenizers==0.23.1 safetensors>=0.8.0 is required, but found safetensors==0.7.0 A test pins that shape out, along with the zoo-in-the-resolution mistake that would silently turn this into a second canary. It goes on the second kernel's idle T4, so it costs no quota: a Kaggle session bills wall clock once, not per card. What it does NOT catch is recorded in the leg: the loss trajectory is not the control's, and step 1 is computed before any update on identical weights, data and seed, so 10.3222 against 6.4367 is a different loss function rather than drift. Both converge. Which objective is intended is separate work. * Name the encoding on every checked-in file the T4 harness reads Repo tests (CPU) is red on this branch: test_source_read_encoding.py::test_checked_in_file_reads_name_an_encoding AssertionError: 17 file reads in the test trees touch a checked-in file with the platform default encoding, so they break on Windows as soon as that file gains a non-ASCII byte. All 17 are in tests/kaggle/test_t4_smoke_harness.py, which this branch adds, so this is ours and not inherited. The reads pull in the workflow YAML, the smoke scripts, the committed reference JSON and ci-preempt.json, and none of them said what encoding to read them as. On Linux that resolves to UTF-8 and nobody notices; on a Windows runner it resolves to the ANSI code page and the first non-ASCII byte in any of those files takes the test out with a UnicodeDecodeError that has nothing to do with what the test is checking. Every one now passes encoding = "utf-8", matching how the rest of the suite spells it. Reads of files the test itself just wrote under tmp_path are left alone: the gate only covers checked-in files, and those are ours end to end. test_source_read_encoding.py 1 passed tests/kaggle/test_t4_smoke_harness.py 148 passed * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align the Kaggle budget guide with the workflow it documents The guide named --percent 15 and --reserve-hours 25 against a 30h account; the workflow runs 40 and 20 against the 60h figure measured on 2026-08-11, and it is the only consumer of that quota today. Following the guide would have tuned the reserve from a baseline that no longer exists. Re-derived from the same live numbers as the workflow header, and says which of the two is the source of truth when they disagree. * Stand the gate down on any unreadable kernel status The concurrency verdict refused only when EVERY in-window status came back an error. One unreadable status among readable ones was treated as an idle account, and driving concurrency_verdict with {busy: [], own: [], foreign: [], surveyed: 2, unreadable: 1, complete: True} returned (True, ""). That kernel can be the human session the zero-foreign policy exists to yield to, and proceeding takes the account's last slot: the first push wins it, the second comes back at capacity, and the run reports a subset of its legs. A 404 is kept separate and stays benign. A deleted kernel is not an unknown state and the launcher deletes every kernel it pushes, so blocking on that would wedge the gate shut rather than make it careful. * Close six gaps in the Kaggle T4 notebook CI workflow - The pull_request trigger used GitHub's default activity types, which are opened, synchronize and reopened only. The kaggle-t4-ci label the gate advertises as a human override could therefore never start a run on its own. Subscribe to labeled as well. - pyproject.toml was in neither paths filter, though every payload installs the commit under test as a distribution built from it. BUDGET.md counted it as a watched path already. - The job deadline of 120 minutes was shorter than the launcher's worst case. Two sequential pushes can take 2 x (4 x 600s + 315s of backoff) and the 5400s polling window only starts after them, so about 187 minutes with the deletions; a runner killed at 120 takes finish() -> release() with it and leaves pushed kernels billing quota. 200, with the arithmetic written down beside it. - The gate surveys quota and in-flight kernels before t4-smoke queues on the account-wide concurrency group, so a second sampled run could launch on an approval a full run old. Re-ask with the slot in hand, immediately before the push; --force true skips the sampling draw and nothing else. - --zoo-ref main let each payload's own pip resolve a different unsloth-zoo commit, including between the control and the canary within one session, and zoo is not in pins/control.txt either. Resolve it once with git ls-remote and pass that SHA to every leg. - A dispatch with max_steps != 10 still shipped the 10-step reference, and check_reference reports step_count_mismatch as a failure, so the run could only ever go red on arithmetic. Drop the band automatically when the counts differ, and say so. Also: check out the same head SHA the launcher installs, so the harness and the package under test are one snapshot rather than the merge tree and the head; and run tests/kaggle/test_t4_smoke_harness.py on the runner before anything is built or pushed, since testpaths limits pytest to tests/security and nothing else collected it. * Read back what the Kaggle kernel actually reported Two ways a real payload verdict was being lost in transport, both ending in verdict=infra and a green workflow. The kernel log fallback could never work. Kaggle's kernels/output hands the log over as a JSON array of {stream_name, time, data} records, which is why report.kernel_log_text flattens it before reading; extract_reports scanned the file as text, so no line began with the report prefix and the fallback that exists for the run whose notebook never came back found nothing. Record boundaries are not line boundaries either, so the records are joined before the scan rather than read one at a time. push() documented a fresh slug per attempt and used one for all four. Pushing to an id that already exists does not replace it: Kaggle files a new version and starts a second batch session, and kernels status and kernels/output send no version label, so they answer for the newest session only. A retry after a lost response therefore collected the wrong execution's evidence while the first kept a session slot and billed unseen. Each attempt now takes its own slug and discards the previous one first, and every slug filed is returned and recorded, since a push that reported an error may still have landed. * Give each payload its own everything, and report a probe failure Four ways the two payloads of one kernel were not actually isolated, plus the import probe exiting without a verdict. The probe was the P1. When the commit under test breaks import unsloth, the verify cell raises before the run cell can write a report, the launcher extracts none, and no reports is classified as infra -- so a deterministic import regression exits green. A probe failure is a verdict, not missing evidence, so it now emits its own failing report. nvidia-smi failing or a one-GPU allocation fell back to N_GPU = 1 and pinned both payloads to device 0. Each child still saw exactly one card and passed its own visibility assertion, so the contention came back looking like a code failure. A shortfall is infrastructure and stands the kernel down before a thread starts, which is what the comment above it already claimed. A payload whose virtualenv could not be built kept the base python3 kernel and installed into the shared system site-packages. That is the one thing the per-child venv exists to prevent: the legs install deliberately different library sets, so the last writer wins and the resulting import error reads as a regression. The payload is skipped instead. Both payloads materialised into one source directory and compiled into one unsloth_compiled_cache. The sources are byte-identical copies but write_bytes truncates first, and the compile cache is a relative path resolved against a working directory both papermill children inherit while compiling the same modules against different transformers versions. Each payload now owns both, and the prune globs still reach them. Finally, --smoke-args is appended after every leg's own arguments, so the workflow's --max-steps 10 landed after the gpt-oss leg's --max-steps 3 and argparse took the last one. The 20B leg was training for ten steps rather than the three that were measured to fit. A shared argument no longer reaches a leg that sets that option itself. * Count a moved step and a one-sided field as run-to-run differences compare_metrics zipped the two traces positionally and skipped any field that was not on both sides, so a grad_norm logged by one fresh process and not by the other left identical true, and so did a step coordinate that shifted, duplicated or reordered while the values stayed in place. Both are nondeterminism, which is the one thing this comparison exists to find, and check_reference already treats one-sided presence as a change in the shape of what the trainer logged rather than as a skip. * Assert the SFT leg's canary, adapter, repeats and reference properly Six gaps, all of them a green run that established less than it claimed. The canary was a substring test, so the exact, tolerance-free assertion the docstring describes twice accepted '__UNSLOTH__!!!<anything>' -- which is what a stopping or EOS regression produces. It is now an exact match after stripping surrounding whitespace, with both readings recorded so a red says which of the two it was. The saved adapter was only ever checked for a filename while inference ran on the in-memory model, and the caller-side reload the comment promised does not exist anywhere. The serialized weights are now read back and checked for tensors that are present, finite and not all zero; lora_B starts at zero, so an all-zero file is an untrained adapter that would reload without complaint. --repeat 3 compared cycles 0 and 1 and threw the rest away. Every extra cycle is now compared against the baseline, and the summary keys report.py renders off the reproducibility block keep their shape. optimisation_failures counted an infinite grad_norm as an applied update because inf equals itself, so a run whose every step overflowed to inf rather than NaN reported as a training run that trained. It requires a finite norm now, which is what the loss check three lines above already did. check_reference guarded max_steps out of a config block it was already holding whole. The learning rate, the optimizer, the LoRA shape, the model and the commit of the model repository that was actually read are compared on the same terms, and so are the per-step coordinates, all of them refusing before a single number is compared. Older references keep working: a key the file does not carry is listed unchecked rather than called a mismatch. A cycle that never wrote its report produced a failure report with no environment block, which is the case where knowing which library set died matters most. The fingerprint and the config are now read before the cycles and carried into that report. * Assert the float32 path, the training compile and the optimizer update The gpt-oss leg exists for Unsloth's FORCE_FLOAT32 behaviour on a card with no bf16, recorded the fp16/bf16/force-dtype state it ended up in on every run, and asserted none of it. A run that quietly went through fp16 instead logs finite losses, compiles and generates, and reports green while the one thing this leg uniquely covers was never exercised. It is now a failure, conditioned on the card rather than hardcoded to T4: where bf16 IS supported the patch not firing is correct. The compile assertion read Dynamo's process-global counters after training and asked for at least one unique graph. Loading a 20B checkpoint through Unsloth fills those counters long before trainer.train() is called, so a training path that fell back to eager entirely still satisfied it. The counters are now sampled before training too and the assertion is on the delta. grad_norm was captured in every metric entry and read by nothing. A run whose gradients are all zero keeps a finite loss, compiles, and generates text from the untrained base model, so the leg reported LoRA training it had not done. At least one logged norm must now be finite and non-zero, and as in the SFT leg the check declines to decide where grad_norm was never logged at all. * Generate through the trained adapter, and keep the engine verdict The final fast_generate passed lora_request=None. fast_generate is the vLLM engine's own generate and the trained LoRA reaches it only through save_lora plus load_lora, so the check generated from the base weights and passed whether or not the adapter could be transferred into the engine at all -- which is the second of the two questions this probe exists to answer. The request is built now, what happened is recorded, and a transfer that failed is a failure. engine_built was set on a local dict that train() only returned at the end, so any exception after the engine was built threw the fact away and the report recorded engine_built false. That is the opposite of what happened and it erases the distinction the probe needs between vLLM being unable to start on sm_75 and GRPO failing later. Facts are published into the report as soon as they are known. A reward on some steps satisfied the check that the docstring describes as a reward on every step: a step whose reward functions never ran was filtered out and the others covered for it. Every step that logged a loss must now carry a reward; the summary row train() appends carries no loss and is not a step. Every logged grad_norm being NaN also passed. Under fp16 on this card every step can overflow and be skipped while loss, reward and reward_std are all still logged and base-model generation still returns text, so the length check alone was satisfied by a run that updated nothing. * Take the reference from the control report, not from reports[0] extract_reports walks sorted(rglob(...)) over per-kernel directories named after a random Kaggle slug, and inside the control/canary kernel t4_canary_output.ipynb sorts before t4_control_output.ipynb. Driving the real extractor over a four-leg run puts frontier first. Following the recapture recipe as written therefore commits the canary, frontier or gpt-oss trace as the control reference, and every later run then band-checks against a curve it has nothing to do with. The recipe now selects the report whose label is control and asserts there is exactly one, and carries the model and the resolved checkpoint commit through with the config. Also 45 CPU-only tests for what the payloads assert, which is the half of the harness that test_t4_smoke_harness.py does not cover: it checks the launcher, the gate and the generated notebook, and this checks whether a given result dict is called a pass. All 45 fail on the parent of this series apart from the ten that pin behaviour that was already right. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Let two runs that both overflowed on the same step agree An fp16 overflow does not only log a NaN gradient norm. clip_grad_norm_ over a gradient holding an infinity returns infinity, so the same skipped step can come back as inf on both cycles of the same configuration. compare_metrics handled the NaN case explicitly and then subtracted everything else, and abs(inf - inf) is NaN, which is not equal to 0.0. Two bitwise identical traces were therefore reported as nondeterministic, and the summary printed DIFFERED with a max_abs_diff of 0.0 beside it. Equal values now short circuit before the subtraction. A one sided infinity, and two of opposite sign, still reach it and still come out as differences. * Start the deletion deadline before the first push, not after the last A Kaggle kernel bills from the moment the push is accepted, and the only control that has been observed to stop one is the launcher deleting it. That deletion was scheduled from a deadline started after every push had returned, so the first kernel got --max-wait on top of however long the second push spent in its retries: four attempts at the 600s subprocess ceiling plus the backoffs is about 45 minutes, which turns a 90 minute ceiling into 135 minutes of billing. The deadline is now one invocation-wide clock started before the first push, so time spent pushing comes out of the wait rather than being added to it. The reservation the gate makes was understated for the same reason. It was derived from the ceiling passed to Kaggle at push time, which this workflow already records as unreliable; it is now derived from the launcher's own bound, one --max-wait per kernel, and the harness suite keeps the two in step. * Delete every slug a push filed, not only the accepted one push() files a fresh slug per attempt and returns all of them, because a push that reported an error may still have landed: Kaggle answers an accepted push with a 5xx or a reset connection often enough to be a known issue. Nothing then reconciled that list. Cleanup read entry["slug"], which a failed push does not set at all, so two cases left a session running with nobody watching it: - the last attempt of a push that never succeeded, which is exactly the ambiguous one that made the retry necessary; - an earlier attempt of a push that later did succeed, since the discard before each retry is best effort and subprocess.run does not raise on a refused delete. Both keep one of the account's two concurrency slots and bill GPU quota to the kernel's own ceiling. Release now walks every slug the entry filed and records which of them are gone. A delete for a slug Kaggle never created is refused and costs one call, which is why this can be unconditional. The job deadline arithmetic is recomputed for the extra deletions and for the polling deadline no longer stacking on the pushes. * Report an install that cannot be resolved as a failed payload The install cell retried three times and then raised, emitting no payload report. The launcher extracts reports and, finding none for that leg, calls the run partial or infra and exits green -- so a commit whose distribution cannot be resolved, or a control pin set that stopped resolving, passed the one job that exists to test packaging metadata. Only the later verify cell reported a failure, and the install cell never reaches it. The exhausted branch now writes the same kind of structured report the dependency probe writes, naming the group and the last pip error, before it raises. The retries are also backed off. They were immediate, so all three landed inside the same upstream blip, and the third failure has to mean the resolution is impossible rather than that one minute was bad. * Keep the resolved versions in an import failure report report.version_table builds the per leg comparison out of the report dictionaries, not by scraping the kernel log, so a leg whose report carries no versions is missing from the one table that answers the question the control and canary pairing exists to answer: which release did the red leg have that the green one did not. The versions were computed and printed one line above the failure and then discarded. They are now kept in a name, carried into the report, and the computation is wrapped: a half installed distribution can make importlib.metadata raise, and losing the report to a diagnostic would put the leg back to reporting nothing at all. * Stand the run down when the zoo commit cannot be resolved The fallback to the branch name restored exactly what pinning the zoo revision removed. unsloth-zoo is not in the control pin file, and every payload pips independently on the kernel, so with main the control leg and the version canary can resolve two different zoo commits inside one session. That invalidates the control's reference band and the attribution the pairing exists for, and the report records a distribution version rather than a commit, so the drift is not visible after the fact either. The ls-remote is retried three times, since one blip is not a verdict, and a resolution that still cannot be made stands the run down: a warning, no kernels pushed, and a green job, which is how every other infrastructure stand-down in this workflow behaves. Nothing that spends a Kaggle session runs after it. * Run every CPU suite in tests/kaggle before spending a session The step named one file, and pyproject.toml restricts default discovery to tests/security, so the two suites added after it were collected by no invocation anywhere. Measured at this commit: the step collected 159 tests of the 227 in the directory. The 68 that ran nowhere are the transport and payload assertion suites, which cover the report extraction, the payload pass and fail rules, the reference checks and the per leg isolation. It runs the directory now, so a suite added next to these is picked up without this file being touched, and there is a test that collects whatever the step names and asserts every suite in the directory is in the result. Measured in a bare venv with pytest, transformers and CPU torch: 226 passed. * Record the kernel a recaptured reference came from, not the leg label The recipe wrote report["label"] into source_kernel, which is always the string control. That field is the only thing in the file pointing outwards, at the hardware execution its band was measured on, and it is what makes a recapture auditable while the evidence artifact is still around. The committed file carries a real slug, so following the recipe would have replaced it with a word that names nothing. The recipe now derives the slug from launch_result.json: each kernel collects into a directory named after the last segment of its slug and each leg's executed notebook is named after the leg, so the control leg's kernel is identified rather than guessed at, and the recipe refuses if that does not come out to exactly one. The recipe is now executed by the test rather than pattern matched, against a two kernel evidence tree. * Pin the reference model, and say what the band did not compare The committed reference carried no identity keys at all, so the checkpoint pin added with the reference config never ran on any workflow invocation and the band could return ok after the loaded revision moved. Two of those keys are unknowable without another T4 session, but one is not: the control leg passes no --model, so the reference belongs to DEFAULT_MODEL. It is recorded now and that comparison is live. resolved_checkpoint and resolved_revision stay unrecorded until the next recapture, because inventing a commit hash for a run that happened is worse than admitting the file does not carry one, and refusing on their absence would turn every run red over a reference that is otherwise sound. What changes is that the gap is no longer silent. A key present on one side only is a pin that did not run: it is recorded under config_unchecked in both directions now, including the one where the reference names a commit and the run could not read one, and report.py puts that list on the job summary next to the band verdict. A key neither side claims stays unreported, since nothing was asserted about it. * Record the count a fresh runner sees for the harness step The comment beside the step names what a bare venv with pytest, transformers and CPU torch collects, and it moved with the tests added alongside these fixes. Measured, not estimated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Record a push that times out instead of letting it kill the cleanup subprocess.run(timeout=600) raises TimeoutExpired rather than returning a failed push, and nothing caught it. Pushing the second notebook after the first was accepted therefore left main() without finish()/release() and without launch_result.json, so the accepted kernel billed to its own ceiling with nobody reading it, and the slugs the timed-out attempt had filed -- the most ambiguous ones there are -- were lost with the exception. The timeout is now a failed attempt like any other: the slugs stay in `attempts`, the retry discards the previous one, and release() reconciles what is left. The whole of main() past authentication also runs under a guard that still deletes and still writes the result on any unforeseen abort, since nothing in the workflow cleans up after this script. * Read the adapter itself, so a silent grad_norm cannot pass a leg `if norms and not applied` passes on an EMPTY list, and the gptoss and grpo legs had nothing else to fall back on: neither saves an adapter or reloads one, so a Trainer or TRL change that stops logging grad_norm takes the only evidence of an optimizer update with it and both legs go on reporting green. Finite losses, captured graphs, rewards and non-empty generation are all produced by the base model and the loader on their own. Both legs now fingerprint their LoRA parameters before training and after, and that reading decides it, with grad_norm as the fallback rather than the source. Silence is still not evidence of nothing applied, which is the position the earlier round took; it is now answered from the weights instead of guessed at. Only when BOTH instruments are gone is the leg red, and then for what it is: the training path it claims to cover can no longer be shown to have run. The tiny SFT leg is unchanged. It reads its saved adapter back off disk and fails on an all-zero one, which is the same question already answered. * Stand down on an unrelated label, and pin a dispatched ref to one commit Two ways this workflow spends more than it budgeted for. The `labeled` trigger fires for EVERY label, not just the opt-in one, so any label applied to an eligible pull request started a run and a fresh sampling draw. Once kaggle-t4-ci was present it was worse: it stays in the label list, so every later label of any kind arrived as an override and forced a session. The budget arithmetic counts pull request opens and pushes and no label activity at all. The gate now takes the event action and the label that arrived, and stands a `labeled` run down unless that label is the opt-in one. Every other action is unaffected. A dispatched unsloth_ref was forwarded to the payloads unchanged, and each payload pip-installs on the kernel by itself, so a branch name could resolve to a different commit in each of the four legs -- the control and the canary compared across two different Unsloths, invisibly, since the report records a distribution version rather than a commit. It is resolved once now, here, exactly as the zoo revision below it is, and a ref that cannot be pinned stands the run down rather than installing a moving branch. What a dispatch varies is still only the package under test: the harness stays on the checked-out tree, because most of it does not exist on older refs. Label names and the dispatched ref travel through the environment rather than being interpolated into the shell. * Say in the budget file what a label event costs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Close five ways a leg could pass on a run that trained nothing usable Every one of these is a report that stays healthy while the thing the leg uniquely covers did not happen. - A LoRA weight that went NaN or infinite was the strongest possible pass: the fingerprint returned a number, `NaN != finite` read as "the adapter changed", and update_verdict said `applied` on exactly the no-telemetry path it was written to decide. A non-finite reading is now its own verdict, it beats a healthy grad_norm, and both call sites fail on anything that is not `applied` rather than on the two verdicts they happen to know about. - The saved adapter was accepted on a nonzero tensor count, which lora_A satisfies at initialisation. An adapter whose B matrices were all zero or were dropped contributes exactly nothing and reloads as the base model, so the B matrices are counted specifically and their absence is unusable rather than fine. - UNSLOTH_FORCE_FLOAT32 was tested for being nonempty. The loader writes "0" on its ordinary branch before deciding whether to force, and every production consumer reads `== "1"`, so forcing could stop firing with fp16 and bf16 both still false and the leg green. - The compile assertion fell back to the absolute graph count when no pre-training baseline was read. That count is the loader's, so a training path that ran entirely eager passed in the one case where training could not be isolated. A missing baseline is now unverifiable. - Reference drift divided through infinities to NaN, and NaN > tol is False while max(worst, NaN) is worst, so a step that used to be finite and now overflows was accepted without even moving worst_rel. Equal signed infinities are the unchanged case; every other pairing is a deviation, decided before the division. * Report a payload that crashed, and re-emit its report on one line The run cell wraps the payload in a child process and is the only thing that sees the child's exit status. Two ways it threw that away: A nonzero exit with no report printed "NO REPORT WRITTEN" and nothing else. No report at all is `infra` at the launcher and one report of two is `partial`, and both leave the workflow green, so a CUDA segfault, a native abort or an OOM kill was accepted silently while the definitive exit status was in hand. The crash is now emitted as a `passed: false` report carrying the return code and a stderr tail. The recovery path for a report whose own compact line fell out of the retained stdout tail echoed the file verbatim. Every payload writes that file indented and the launcher parses whole lines, so it was handed a lone brace to decode and the definitive result was downgraded anyway. The file is parsed and re-serialized compactly, and a file that will not parse takes the crash path rather than disappearing. * Delete the kernels for real, and prove the ref before paying for it Four things this harness reported as done without ever establishing them. Release: subprocess.run does not raise on a nonzero exit, so the cleanup loop recorded every slug as released whatever Kaggle answered. The live case is worse than a transient refusal. kaggle==1.7.4.5 has no "kernels delete" subcommand at all (it landed in 1.7.5.0, Kaggle/kaggle-cli#762, first released in 1.8.0), so argparse answered every delete with "invalid choice: 'delete'" and exit 2, and the run still called the kernel released while it billed on to its own ceiling. delete_kernel now reads the exit code, retries a refusal, and a slug it cannot confirm gone is named in the log, in launch_result.json and in a workflow annotation. The pin goes to 2.2.4, because the version is load bearing and 1.7.4.5 could not do the job. Verified against the live API: 1.7.4.5's authenticate() refuses KAGGLE_API_TOKEN, the only credential this workflow has, and demands a kaggle.json nothing here writes; quota_view(), which gate.py reads the remaining accelerator hours from, exists only on 2.x, so below it the call raised AttributeError into a handler that recorded "quota unreadable" and let the run proceed. Dispatched ref: a 40-character SHA was accepted on shape alone and nothing asked whether unslothai/unsloth has that commit, so a mistyped one pushed the paid kernels, failed every payload's git install, and reported the pull request red for a commit that never existed. git ls-remote cannot answer that (it matches refs and exits 0 with empty output for any SHA); a git fetch of the object can, and it is the same reachability pip needs. Applied to the ls-remote result too, not only to the SHA the form supplied. GPU probe: the verify cell's bare device_count assert aborted before the run cell could emit its fallback report, so a CPU-only or CUDA-incompatible torch wheel left the launcher with no report, which is infra, which exits 0. It now emits passed: false the way the import probe above it does, for any failure in the cell rather than the count alone. gpt-oss: the forced-float32 assertion was conditioned on bf16_supported is False, and main() records environment = {"error": ...} for the whole probe when it raises. An unreadable card reading therefore skipped the one check this leg uniquely carries while training, losses, adapter and generation all still passed. Anything that is not a literal True or False is now a failure, and the healthy floor fixture carries the reading so every case below it exercises the check. tests/kaggle: 292 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Say the same things about this CI in fewer lines Comment-only pass over the Kaggle T4 notebook CI: the prose blocks that explain the leg pairing, the budget arithmetic, the cleanup contract and the payload assertions were several paragraphs where a few lines carry the same meaning. Every measured fact stays: the kaggle client pin and the three things 1.7.4.5 cannot do, the ls-remote vs fetch behaviour on a bogus SHA, the vLLM torch pin windows, the recorded losses, peaks, kernel ids and version boundaries. No code changes, verified by an AST comparison over every touched file. * Count both deletion paths in the deadline that protects them The job deadline exists so the runner is never killed during release(), which is what deletes the kernels and stops the billing. Its arithmetic counted one delete per filed slug and none of the discards, so it was about half the launcher's real bound: one delete DELETE_ATTEMPTS 3 x 180s plus backoffs 5 + 10 = 555s push, per notebook 4 x 600s + backoffs + 3 discards = 4380s two notebooks = 8760s release(), 4 filed slugs per kernel = 4440s evidence ~600s total 13800s That is 230 minutes against a 200 minute deadline, so a wedged run could be killed mid-cleanup and orphan the kernels the deadline is there to get deleted. The deadline moves to 260, the delete ceiling gets a name, and the harness test now derives every term from launch.py's constants instead of hardcoding one call at 180s, so it can fail when they move. The gate had the matching hole: no socket timeout, and the Kaggle client has none of its own, so a stalled call blocks forever, --soft-fail never sees an exception and the 10 minute job timeout reports red on a workflow whose contract is that only a failed assertion on a T4 is red. It now sets the deadline before the first call, as launch.py does, and the in-flight survey gets a wall-clock budget: running out of it stops the walk as incomplete, which is already a skip, rather than being killed by the job. tests/kaggle: 296 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep three non-payload outcomes from costing quota or a red check Three ways this workflow could spend a Kaggle session, or colour a pull request red, for something that is not the code under test. push() filed a slug and only published it by RETURNING. The retry loop handles the subprocess timeout, so every other raise (subprocess.run with text=True decodes strictly and raises UnicodeDecodeError on a malformed response; the runner can answer OSError or MemoryError) unwound past the line that recorded the entry, and release() then iterated a list with no entry for that notebook at all. A kernel Kaggle may have just accepted was left billing to its own ceiling. The list is now the caller's: main() publishes the entry before the push and push() fills it as it files each slug, so reconciliation no longer depends on the call returning. max_steps is free text on a dispatch and was forwarded unchecked. A value that is not an integer reaches the payload's argparse, which exits 2 with no report, and the generated cell reports that crash as a failing leg on purpose, so three legs come back red after the kernels were pushed and the model downloaded. A value below the fp16 scaler's leading skipped steps applies no optimizer update at all, and the payload's own checks fail it. check_steps.py answers both before the recheck, and MEASURES the floor as the shortest prefix of the committed reference that optimisation_failures accepts (5, on the committed trace) rather than restating a number that would go stale as soon as the reference moved. The build step also takes the value through the environment now rather than interpolating it. The evidence upload had no continue-on-error, so an artifact-service outage failed the job, and the job is what the pull request shows. The verdict is Report's, which runs under always() and can still conclude pass. It is best effort now, and says so out loud: continue-on-error leaves outcome at failure while conclusion becomes success, and the step after it reads that outcome and warns. tests/kaggle: 302 passed. * Bound the four checks that were stated rather than measured Evidence collection had no deadline. The job timeout counted 300s per kernel for it, a number no code enforced: one listing walks OUTPUT_PAGE_LIMIT pages at the 120s socket ceiling, 2400s for a single kernel, and each executed notebook is another 300s with no cap on how many Kaggle lists. Measured against the launcher at the previous head, two kernels can spend 14520s there, against 600s budgeted, and the phase runs BEFORE release(), so the runner is killed with billable kernels up. launch.py now starts one EVIDENCE_BUDGET_SEC budget before the first collection, shared by every kernel, clamps each listing page and each download to what is left of it, and marks a collection truncated. The job deadline reads that constant instead of restating a per-kernel term, and deadline=None now means a fresh budget rather than no ceiling. The reference band never checked the card. The reference records gpu_name "Tesla T4" and gpu_capability "sm_75" because a loss trace belongs to its hardware -- no bf16, xformers attention -- and every other reference-defining property already refuses before comparing. Only the GPU COUNT was checked, on the kernel, which cannot see the file. A P100 run against the committed T4 trace returned status "ok" with no failures, so a hardware difference read as a code regression. check_reference now gates on the reference's own environment block, so a recaptured reference moves the gate with it, and "it does not say" stays unchecked rather than a mismatch. adapter_config.json was checked for being JSON. "{}" parses, so a save that wrote no LoRA fields at all scored config_readable and the leg passed on a directory PEFT cannot load. The question is whether PEFT can rebuild the adapter, so it is asked of PEFT on the path a reload takes, PEFT_TYPE_TO_CONFIG_MAPPING[peft_type].from_pretrained; PeftConfig.from_pretrained alone returns peft_type None for "{}" and reports nothing wrong. The rebuilt config is then compared against the arguments the payload handed get_peft_model, so a well-formed config for a different adapter is a difference rather than a field list this file guessed at. The version table omitted the packages the legs move. legs.py records the frontier resolution installing huggingface_hub 1.27.0 and datasets 5.0.1, and the tokenizers and safetensors ceilings that forced deps back on, none of which were in GOAL_PACKAGES, so a failure caused by one produced a comparison table in which nothing differed. They are recorded now. The related hole is that pin_failures answered from a table keyed on that fixed list: a pin outside it had no entry and came back "it is not installed" for a package installed and correct. The probe list is derived from the pin file, and an unprobed pin is its own third outcome rather than an invented failure. pytest tests/kaggle: 315 passed. The workflow's recorded count is updated and its harness step installs peft, without which the two new adapter checks importorskip and cannot fail. * Enforce three bounds that stopped at the edge of the thing they bound Evidence deadline: urlopen's timeout is per blocking socket operation, not a ceiling on the transfer, so an endpoint that keeps returning bytes renews it forever and one resp.read() outlasts the whole 600s budget while every deadline check sits before the call. Measured against a body arriving in 20 chunks a minute apart: 1211s listing, 1260s download. Read in chunks with read1, the absolute deadline re-checked before each and the live socket re-clamped to what is left, streaming downloads to disk instead of into memory. A listing or a notebook abandoned that way now says the evidence is truncated. Reference hardware: check_reference compared gpu_name and gpu_capability only when the run supplied them, and main() records environment = {"error": ...} when the probe raises, while the fingerprint omits every gpu_* key outright when torch.cuda.is_available() is False. Either way the live values were absent, both keys were filed as unchecked, and the control leg reported an ok reference check without establishing the T4 the trace belongs to. What the reference does not say stays a skip; what the run cannot say about a key the reference does name is now hardware_unverified and fatal. Harness step: peft depends on torch, so installing it first satisfied that from the default index and the CPU-index line that followed found the requirement already satisfied and installed nothing, leaving the CUDA set on the runner in the window reserved before the push. CPU torch goes first, as version-compat-ci does, and the result is checked with torch.version.cuda rather than assumed. tests/kaggle: 323 passed. * Report an exhausted weekly Kaggle quota as a failure, not a skip Every gate stand-down was a green skip, exhaustion included, so a week with no accelerator hours left looked exactly like a workflow nobody had wired up: t4-smoke skipped, no annotation, nothing to read. The quota floor now exits nonzero carrying GPU capacity exhausted - please wait until next week - you can ignore this CI failure followed by the numbers the quota API already returned (hours left, the weekly total, and the refresh time), in the log line and in the job summary, so the reader can see when it clears and that there is nothing for them to fix. Only genuine exhaustion. It is decided where it already sat, before the concurrency survey and before any kernel is pushed, so it costs one API call rather than a Kaggle session. An UNREADABLE quota stays a green skip: unknown is not exhausted. So do the sampling draw, the wrong opt-in label, the missing token, a failure to authenticate, a busy account and the launcher-side stand-downs, all with their wording unchanged. --soft-fail becomes a request rather than a default. It defaulted to True, which would have made the new red unreachable, so it is now three states: unset, --soft-fail, --no-soft-fail. An error in the gate itself is still a skip unless --no-soft-fail is passed, exactly as before. The gate job asks for the red and nothing swallows it. The recheck inside t4-smoke passes --soft-fail and stands down green instead: it runs after approval with the account slot in hand, so reaching it means the hours went while the run queued, and red there would be the same message after a runner, the harness suite and the kernel build. Ten tests pin it: the exact sentence, the exit codes on both sides of --soft-fail, that the survey never runs once the quota is short, that an unreadable quota and a busy account stay green, and that nothing downstream turns the gate job's failure back into a pass. 333 pass. * Reserve what a run can bill, and assert two results that were only recorded Three checks that stopped at the edge of the thing they bound. The gate reserved 4 GPU-h against a launcher whose billable window is the whole invocation, not the polling. A kernel bills from the moment Kaggle accepts it until a delete is confirmed, so the push retries (each one discarding the previous attempt's slug), the evidence phase and release() are inside that window as surely as the wait is: launch.py's own constants put one invocation at ~13800s, and Kaggle runs at most --kernels sessions for this account at once, each billing its wall clock once. That is 2 x 13800s = 7.7 GPU-h. The reservation is now 8, and the harness suite recomputes the bound from launch.py rather than checking the number, in the same helper the job deadline is derived from. The old assertion read budget >= kernels x --max-wait, which 4h satisfied while a wedged run could bill 7.7h and take the 20h reserve down with it. The reference identity pinned the configuration and not the rows. canary_dataset.jsonl is inside this workflow's paths filter, so editing it TRIGGERS the run that would then be band-checked against a trace of the old rows: a small edit passes the tolerance and reports green on a comparison that means nothing, a larger one is reported as a code regression. The payload now records a sha256 of the parsed rows in config.dataset_digest and check_reference refuses on it exactly as it refuses on max_steps, the digest of the committed dataset is in the committed reference, and a harness test compares the two on the runner before any session is paid for. Digesting the parsed rows rather than the file's bytes means reformatting does not force a session-costing recapture; changing a row, the order or the count does. The gpt-oss leg recorded placement_after_load on every run and asserted on none. Its documented result is that the 20B checkpoint fits and trains wholly on one T4 with about 1.8GB to spare, and that is the claim that degrades quietly: a loader or memory regression that spills to CPU, disk or meta still logs finite losses, still moves the adapter, still compiles and still generates. failures_for now fails on parameters off the visible CUDA device, on hf_device_map offload, and on a placement record it cannot read, the last on the same three-way rule the bf16 reading already gets. accelerate's offload does not support training at all, so a run that reaches it is not a slower version of this leg. tests/kaggle: 345 passed. * Report a dependency import that exits the process The verify cell's import probe caught Exception, so an import that ends in sys.exit() went straight past it: SystemExit derives from BaseException expressly so that code catching Exception does not swallow it. The cell then aborted before writing the report the probe exists to write, the run cell below it never ran, and a leg that reports nothing is partial or infra at the launcher, both of which exit 0. transformers is a live carrier rather than a hypothetical one: it defines OptionalDependencyNotAvailable as a BaseException subclass and raises it at module scope, and its own lazy loader re-raises only Exception. So catch BaseException here, as the GPU probe a few lines below already does for the same reason, and re-raise KeyboardInterrupt: that is the runner cancelling the job, not a broken dependency, and recording it as one would make a cancelled run read like a regression. The cell is executed in the tests rather than pattern matched, so the verdict is observed the way the launcher will read it. * Resolve the dependencies of the distribution under test pyproject.toml is in this workflow's trigger paths because the payloads install the commit under test as a distribution, and every leg then installed it with --no-deps. pip enforces the requirements of packages IN a resolution and merely warns about the rest, so with the tested distribution outside every resolution the file could change its dependencies and nothing here could tell: a requirement it adds is never installed, one it tightens is never checked against what is already there, and the import probe passes anyway whenever the dependency is reached by a delayed code path. A user meets all three at pip install unsloth. The --no-deps rationale was that the overlay must not walk the set unsloth_zoo had just resolved. What unsloth actually declares is typer, rich, pydantic, pyyaml, nest-asyncio, structlog and click: none of them is resolved by zoo and none is pinned by a leg, so there is nothing to walk. A pyproject that does name one of zoo's packages would move it, which is the regression this exists to show rather than a side effect to suppress. Resolving is only half of it, since pip reports a conflict with something ALREADY installed as a warning and exits 0. So the verify cell also asks pip check, and reads only the lines pip attributes to the distribution under test: the Kaggle image carries pre-existing conflicts of its own, and the frontier leg installs a transformers that zoo's metadata forbids on purpose, so the exit code belongs to the environment while only those lines belong to this leg. An unsatisfied requirement is written out as a verdict, like the two checks above it, because the run cell is the only other thing that reports and is never reached from there. The name the check keys on is read off the requirement the legs install, so the two cannot drift into checking nothing. * Count a kernel Kaggle says is not there as released Cleanup reconciles every slug a push filed, not only the accepted one, and most of those were never accepted: push() files a fresh slug per attempt, and a retry's _discard() deletes the previous one without recording that it worked. So release() asks Kaggle to delete slugs that were never created, and slugs that are already gone. A not-found answer settles the only question cleanup asks, which is whether the slot is still billing. Reading it as a failed cleanup spent DELETE_ATTEMPTS on an absent kernel, ahead of the accepted one that is the only kernel still running, and then named it in a "may still be running, delete them by hand" warning pointing a human at a slug that does not exist. The warning is the one signal this emits, so a false one costs more than the calls. The gate already reads a 404 as a kernel definitively not running, for the same reason and against the same account, so this reads it through the gate's GONE_MARKERS rather than a second list that could drift. The pinned client surfaces one as "404 Client Error: Not Found for url: ..." on stderr with exit 1: kagglesdk calls raise_for_status and cli.py prints the HTTPError. Every other nonzero exit keeps its retries. A 5xx, a reset connection or an argparse refusal says nothing about whether the kernel is up, and trusting the exit code alone would abandon a live session silently. * Refuse to push kernels the runner may not live to delete The job deadline is set to the launcher's worst case plus about 30 minutes for everything that runs before it, and nothing enforces the second term. A checkout, a pip install off a slow index, the harness suite and three network resolutions have no cumulative limit of their own, so the allowance is an assumption about their duration rather than a property of the run. Spend it and the launcher still pushes, and the runner is then killed with kernels up: GitHub sends SIGINT to the step's entry process and kills the process tree about ten seconds later, which is not a window in which release()'s DELETE_ATTEMPTS retries can finish. A kernel nobody deletes bills accelerator quota to its own ceiling with nobody reading the result, which is the exact outcome the deadline exists to prevent. So the launcher is handed the moment the job dies and answers the question itself, before authentication and long before the first push. Less than its own worst case left and it pushes nothing and stands down green, which is what this workflow's failure semantics give every infrastructure outcome: no quota spent, and nothing learned about the code. That worst case now lives in launch.py, computed from the constants that produce it, because three consumers derive from it and all three are wrong by the same amount if a phase is dropped: the job deadline, the quota the gate reserves, and the new guard. The harness suite keeps deriving it independently from the source text and asserts the two agree, so a term lost on either side is a red test rather than a run that pushes with no room to clean up, or one that stands down on every invocation. The start of the job is recorded by the job's FIRST step. Taken any later, the deadline computed from it sits after the real one, and the guard reads optimistic in the one direction that costs quota. The minutes are restated in the launch step's environment because a job cannot read its own timeout-minutes; the suite asserts the two agree. * Ask PEFT what the saved adapter should be called The saved adapter is read back with safetensors and counted, which answers whether the bytes deserialize and not whether PEFT would consume them. PEFT matches its state dict by NAME: set_peft_model_state_dict ends in load_state_dict(..., strict=False) and nothing reads the returned unexpected_keys, so tensors under names it does not recognise are ignored without a word. Measured on peft 0.20.0 against a real saved adapter: strip base_model.model. from every key, or leave the adapter name in (lora_B.default.weight, which is what filtering model.state_dict() by hand produces instead of calling get_peft_model_state_dict), and PeftModel.from_pretrained raises nothing while every lora_B comes back at zero. The file still deserializes, still holds the same nonzero lora_B matrices, still carries a loadable adapter_config.json, so every reading this payload took off the bytes is identical to a healthy save and the leg passes on an adapter that reloads to the base model. The oracle is PEFT's own answer for this run: save_pretrained writes exactly get_peft_model_state_dict(self, ...), so calling it on the model that was just saved reproduces the key set the file is supposed to hold. Nothing is restated here, no prefix and no target-module list, so a legitimate peft renaming moves both sides at once and only a save that disagrees with the running peft is a difference. Three answers, and they are not one failure. A key PEFT names that the file does not carry is a dropped weight. A LoRA tensor under a name PEFT does not use is a weight its loader ignores. A non-LoRA tensor beside the adapter is recorded and not failed, since save_pretrained may legitimately write an embedding or a modules_to_save copy that PEFT never has to match. A run that could not derive the key set at all fails rather than recording an unchecked field, or the strongest thing this asserts would switch itself off silently. * Compare the step count the validator parsed, not the string it was given check_steps.parse_steps accepts "+10", "010" and surrounding whitespace as the ten they are, and the payload's argparse agrees, so those spellings run exactly the reference's run. The build step compared the raw dispatch string against the reference's count instead, read an identical run as a different one, and dropped the committed reference band from it: a green run with the only cross-run check in this workflow never applied, announced in a warning naming a step count that was not different. Both sides of that comparison now come from the step that validated them. check_steps emits the parsed count, and the reference's own count through the payload's reference_step_count rather than a second reading of the same JSON in shell, which also takes the reference filename back to the leg registry. The regression test executes the build step, on the outputs a real check_steps run wrote, with the env resolved from the step's own env block, so a build step that goes back to the raw dispatch string fails it. * Recheck the cleanup window after auth, and keep undecodable payload output from losing the verdict The launcher measured the job's remaining wall clock once, before authenticating, and pushed on that answer. With KAGGLE_API_TOKEN, the only credential the workflow passes, kaggle 2.2.4 authenticates by introspecting the token over HTTP, bounded only by SOCKET_TIMEOUT_SEC, so a window that fitted by less than 120s was already gone at the first push and the runner could be killed during release() with kernels still billing. The check is now a helper asked twice, before and after authentication. The generated run cell decoded the payload's stdout and stderr with subprocess.run(text=True), which is strict: one malformed byte from a native crash handler raised UnicodeDecodeError before the synthetic report below it, papermill aborted the cell, and the launcher called a leg that died partial or infra, both green. Decoding is now errors="replace". --------- Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 04:47:08 -07:00
Daniel Han	1b48147d8e	Windows: stop depending on the generated unsloth.exe console script (#8592) ... * Windows setup: install uv from a pinned release instead of running remote script text studio/setup.ps1 piped astral's install.ps1 straight into Invoke-Expression. That download-and-execute shape is the single construct AMSI providers and cloud ML scanners score hardest, and install.ps1 already replaced it with a pinned-SHA-256 archive download. Port the same implementation across. Progress goes to the pipeline rather than the console, so the quiet path swallows it exactly as it swallowed astral's installer output and the printed lines around the call site are unchanged. * Windows: stop pairing a hidden window with a bypassed execution policy The Studio shortcut launched launch-studio.ps1 with -WindowStyle Hidden and -ExecutionPolicy Bypass on the same command line. That pair is what Microsoft's own detections key on, and studio/src-tauri/src/install.rs already refuses it for the app's own launch of install.ps1. The installer writes launch-studio.ps1 itself, so the file carries no mark-of-the-web and RemoteSigned loads it. The hidden window is unchanged, so the shortcut behaves exactly as before. The generated launcher's own child launch moves to RemoteSigned for the same reason: it runs an inline -Command against an executable, where no script file is loaded and the two policies are equivalent. Also refresh a stale comment in studio/setup.ps1 that attributed the PSModulePath fix to astral's uv installer, which no longer runs in-process. * Installers: keep download-and-run command lines out of the shipped script text AMSI scans install.ps1 in full before a single line of it runs, and generic script classifiers read install.sh the same way inside the Linux bundle. Both headers rehearsed the piped web one-liner five times over, plus a scriptblock form and an execution-policy bypass, none of which anything in the scripts reads and all of which the README already documents. Point at the README instead and reword the in-body comments that quoted the one-liner as shorthand. Every printed line is untouched: the remediation text the installers show users still spells out the command in full. Same treatment for scripts/uninstall.ps1's header. * Windows: resolve process image paths with one Win32_Process query install.ps1's venv-holder probe opened a handle to every running PID through inline C# compiled at runtime. Opening a handle per process is a shape AV heuristics score hard, and it bought nothing: Win32_Process reports ExecutablePath for exactly the processes those handles could be opened against, and answers for all of them in a single query instead of once per PID. The remaining file-canonicalisation imports stay -- handle-based resolution of linked ancestors has no faithful Windows PowerShell 5.1 equivalent, and it runs on security-relevant paths. Falls back to the per-process .Path when the query is unavailable, so a degraded WMI repository degrades exactly as the old code did on a process it could not open. * Desktop: say who blocked the install when AMSI stops the script PowerShell hands the whole top-level script block to AMSI while compiling it, so a security product's verdict arrives as a parse error over the entire file before install.ps1 runs a statement: no [TAURI:ERROR] marker, no phase log, and a stderr tail the user cannot act on. unsloth#8523 shows what that looks like in the UI -- "Installation failed: + FullyQualifiedErrorId : ScriptContainedMaliciousContent". Recognise the two stable error ids on either stream and append what the user actually needs: nothing was installed, nothing was changed, it is a false positive, update definitions and retry, do not turn off endpoint protection. The raw id stays in the message, because the diagnostics report and any vendor submission both need it. Matches the id, never the message text, which is localized, and tolerates the cmdlet suffix the Invoke-Expression form carries. * Desktop: ship each bundle only the installer it can run resolve_install_script picks install.sh on unix and install.ps1 everywhere else, but the shared Tauri config bundled both into every target. The Linux AppImage therefore carried 280 KB of Windows PowerShell it can never execute -- and it is the largest script body a generic classifier walking the squashfs reads, which is where Microsoft's Trojan:Script/Wacatac.B!ml verdict on 0.1.701-beta landed. Move the resource map into the per-platform configs. The clean-machine job already fails when a Linux bundle ships no install.sh; it now also fails when one ships install.ps1, so the split cannot silently regress in either direction. The .deb scanned clean with the same payload, so this is surface reduction rather than a proven fix for that verdict. * POSIX installers: install uv from a pinned release before falling back install.sh downloaded astral's install.sh to a temp file, ran it and deleted the file; studio/setup.sh piped it straight into a shell. Both are, shape for shape, what a dropper does, and generic ML script classifiers score them accordingly -- the 0.1.701-beta Linux AppImage came back Trojan:Script/Wacatac.B!ml while the .deb carrying the same scripts came back clean. Fetch the pinned release archive and verify a hardcoded SHA-256 instead, matching what install.ps1 already does on Windows. Only the four mainstream targets are pinned: musl, armv7 and any host without a digest tool keep the path they have today, because guessing a target triple wrong would break the install outright and that costs far more than the heuristic score of the fallback. Destination, PATH handling and every printed line are unchanged, so a host that takes either path ends up in the same state it did before. * tests: pin the installer shapes antivirus heuristics score One file collecting what was removed, so it cannot drift back: no remote script run in-process, no encoded or base64 payload, no hidden window paired with a bypassed execution policy, no handle opened against another process, and no new runtime-compiled native import outside an allowlist that carries a reason for each entry that stays. The last test is the other half of the contract. Hardening must not change what a user sees, so the remediation lines the installers print -- which still spell out the web one-liner in full -- are asserted verbatim. Removing the one-liner from comments is the point; removing it from what the user is told to run would be a regression. Runs on the existing discovery-based pytest step, no workflow list to update. * release: emit a false-positive submission packet for whatever gets flagged The build job assembles a Microsoft submission packet, but only for the Windows -setup.exe. The detection that actually arrived on 0.1.701-beta was Trojan:Script/Wacatac.B!ml on the Linux AppImage, so nothing was produced for the one asset that needed it. The VirusTotal job already knows which assets were flagged and by which engines, so put the packet there: hash, size and both portals, for every flagged asset whatever platform it came from, with a note that clearance is per hash and per vendor. Engine names are not repeated -- they are third-party text and already appear escaped under Flagging engines. The gate stays advisory; this only makes acting on it take seconds. * Revert "Windows: resolve process image paths with one Win32_Process query" This reverts commit 7897865c9. tests/python/test_windows_installer_concurrency_guard.py bans Get-CimInstance and $process.Path from Get-RunningStudioVenvProcesses outright, and requires the native image-path lookup. That contract came out of #7764, which closed a set of races where the installer inferred "in use" from something other than a confirmed executable identity and blocked installs that should have proceeded. Win32_Process.ExecutablePath does answer the same question, but a wrongly blocked install costs far more than the heuristic weight of three native imports. Record the imports in the AV-shapes allowlist with that reasoning instead, and keep the ban on the process-memory APIs, which the installer has no use for. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments added by this branch Opening comment-reduction pass over the PR diff: same intent, fewer lines. Cut hardest on the prose that restated the PR description rather than explaining the code next to it. Comments and docstrings only, verified with comment_tools.py check --strip-docstrings across every Python file in the diff. * Drop an unused helper from the uv pinned-release test * Fix three review findings on the installer hardening Stray-resource check aborted the step it was meant to assert. grep exits 1 when it selects nothing, and under this step's set -o pipefail plus the runner's bash -e that kills the assignment outright, so every correctly split .deb failed clean-machine CI before reaching the check. Both lookups take || true now: no match is the passing case for the stray one, and for install.sh it was swallowing the explicit annotation in favour of a bare exit 1. studio/setup.sh skipped astral's XDG_DATA_HOME/../bin destination tier, which install.sh, install.ps1 and studio/setup.ps1 all honour. A host that configured an XDG location got uv under ~/.local/bin instead, where no later shell looks for it. The session PATH prepend hid it at install time. The AMSI guidance claimed nothing was changed even when the block landed on the nested studio/setup.ps1, which install.ps1 launches through the same inherited pipes after the venv, PyTorch and the packages are already on disk. Split the wording on whether a [TAURI:STEP] marker has been seen: a pre-start block produces none, so the reassurance is only given where it is true. * Key the submission packet on the flagged count, not the engine list stats and results are separate fields of the same VirusTotal response, so an asset can carry a flagged count with no readable results map. The summary table reports that asset and the packet skipped it, which is exactly the one that needs a packet. Select on stats.flagged and keep the engine list for the Flagging engines section, which is correctly keyed on having engines to name. * Drop the bundle stray-resource assertion from clean-machine CI That job downloads a published release, never a bundle built from the branch, so asserting the new resource split there turns every run red until a release ships with it. The split is a property of the Tauri config, and tests/studio/test_tauri_installer_resource_contract.py already enforces it at the right layer. The || true on the install.sh lookup stays: it is what lets the explicit annotation print instead of the step dying on grep's exit 1 under pipefail. * Windows: stop depending on the generated unsloth.exe console script Fixes #8490. On Windows the `unsloth` entry point is materialised as a generated, unsigned launcher .exe. AppLocker, WDAC and Smart App Control deny it, while the venv's python.exe, a copy of the signed CPython binary, still runs. The installer died at "running unsloth studio setup" with `Program 'unsloth.exe' failed to run: An Application Control policy has blocked this file`, and because the launch throws rather than returning an exit code, it escaped Install-UnslothStudio and printed a raw NativeCommandFailed dump instead of a diagnostic. The desktop updater already solved this in update.rs by reaching the CLI through the interpreter. This applies the same idea everywhere else: the setup handoff, autostart, the shortcut launcher, the Tauri backend, auth provisioning, the install health probe, the preflight probes and the `studio run` respawn. unsloth.exe is still generated, still hardlinked to the shim, and still works. Nothing depends on it any more. Also adds `python -m unsloth_cli` as a supported entry point, and a bin\unsloth.cmd companion to the shim so `unsloth.cmd` is available where the .exe is denied. The trampoline is one string shared by install.ps1, process.rs and studio.py: import sys, os; sys.path[:1] = [x for x in sys.path[:1] if x not in ('', os.getcwd())]; sys.argv[0] = 'unsloth'; from unsloth_cli import app; app() Both halves are load bearing. argv[0] is assigned before the import because unsloth_cli decides at import time whether it is the console script, which gates the UTF-8 stream setup and the -np<N> rewrite, and it keeps typer's prog_name at `unsloth`. The sys.path[:1] filter drops the working directory entry that `python -c` adds and a console script does not, which is what lets the invocation stay off -I: -I would drop it too, but also PYTHONPATH, PYTHONWARNINGS and user site-packages, which the console script honours. Behaviour on a machine with no policy is unchanged, and that is enforced rather than asserted. tests/python/test_module_entry_point.py compares stdout, stderr and exit code between the console script, `-m unsloth_cli` and the trampoline over --version, --help, `studio --help` and two error paths. The writes are idempotent: bin\unsloth.cmd, launch-studio.ps1 and the .lnk files are content compared, so a second install changes no bytes and no timestamps. tests/studio/test_application_control_cli_fallback.ps1 pins the pieces that are easy to get wrong: the failure is classified off the exception (Win32 1260), never off $LASTEXITCODE, which no process was created to set; Start-Process gets one pre-quoted command line, since -ArgumentList joins an array with spaces and quotes nothing; and bin\unsloth.cmd only counts as an ownership marker when its contents match the shim we write, so an unrelated file of that name in a custom root cannot qualify it for removal. The new windows-application-control-ci.yml leg reproduces the report: AppLocker denies only Scripts\unsloth.exe for a standard user, a negative control proves the rule is actually enforced (the job fails loudly if the stub runs), and the full installer then has to succeed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments added for the Application Control fix * Add the AGPL header to the module entry point test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drain the shim launch probe's pipes before waiting on it * Harden the cmd shim ownership marker, updater env and launcher hints * Run the Application Control CI leg without --tauri so the pinned root applies * Stub the runtime gate so the Windows launcher tests run on Windows * Isolate the advertised module route from the working directory * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the contradictory updater env assertion and the user-site fallback * Treat a quarantined stub as a managed install on Windows * Tighten the duplicated trampoline rationale to one authoritative copy * Windows: keep a quarantined launcher and a partial migration recoverable Two follow-ups on the Application Control work. An antivirus quarantine deletes the unsigned unsloth.exe rather than denying it. The updater then found no launcher, no copy to restore, and reported a broken update, rolling back a package that was in fact fine. Absence is now excused the same way a policy denial is, but only after every recovery copy has been tried, so a launcher that could be put back still is. find_unsloth_binary_in_studio_dir accepted a bare python.exe in layout order, so an interrupted migration leaving a partial new environment beside a working legacy .venv targeted the broken one. A launcher anywhere now outranks an interpreter on its own. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows: let studio run start a venv whose console script was quarantined The Windows respawn goes through the interpreter and never launches Scripts\unsloth.exe, but the gate before it still required that file, so an install whose stub antivirus had taken aborted with "Unsloth venv missing 'unsloth' entry point" despite being able to run. The installed package now answers for the deleted stub, one layer down and just as cheap. POSIX still proves a CLI with the console script it execs. * CI: apply the AppLocker policy before AppIDSvc reads it The negative control watched the denied user start the stub. The job started AppIDSvc and set the policy afterwards, and the service loads the effective policy when it starts, so it was enforcing nothing; gpupdate does not make it re-read a local policy. Restart the service once the policy is in place, and retry the control while enforcement goes live, which is asynchronous and unsignalled. * Cover the uv host matrix and repeat application in the pinned-release test The pinned path picks an archive per host triple, and a wrong pick installs a binary that cannot execute, which is worse than not installing at all. Drive _uv_pinned_asset over 20 host combinations and require each one to return its own triple or decline to the fallback. Also run the installer three times over one HOME and require an identical tree, and require a stale uv at the destination to be replaced rather than joined by a second copy: the installer is re-run on every upgrade and every repair. * Windows: close the parity and old-install gaps found by the idempotency audit Five independent audits of the before/after parity bar, plus local simulations, turned up six things worth fixing. Parity, on machines with no policy at all: - Under PYTHONSAFEPATH or -P there is no implicit -c working-directory entry to strip, so sys.path[0] is whatever PYTHONPATH put there and the console script honours it. The filter removed it anyway; a PYTHONPATH starting at the working directory was measured selecting a different package through the trampoline than through the console script. - The backend start log went from a joined argument string to Rust's debug list on every platform. It is what users paste into issues. Idempotency: - The .cmd shim and launch-studio.ps1 compared decoded text, which drops a BOM and ignores case, so a BOM-prefixed shim was called unchanged and left with cmd.exe reading the BOM as part of @echo off. Both compare bytes now, launcher preamble included. - A run killed between the temp write and the rename left a temp file no later run would collect, since each names its own after its PID. Swept, skipping any whose owner is still alive. - The Application Control probe cached its verdict in :, which under irm | iex is the caller's session, so a second run in one console answered from the first run's machine state. Old installs: - An installer older than the shim directory never created one, and unsloth studio update is the only route those installs take back into install.ps1, so they never gained the .cmd. Created there now. - A migration interrupted by an open handle can split either layout. The finder now prefers a launcher with its interpreter beside it in either base, then an interpreter alone, then a launcher alone, so neither half of a split tree wins by layout order. * Pick the pinned uv archive off a positive libc check, not the absence of musl An independent audit pass found the Linux selector accepts any host whose ldd output does not say musl. That is not the same question astral's installer asks: it checks a minimum glibc and drops to its musl-static archive below it, so three hosts that worked before this branch now get a GNU binary that cannot exec, and the helper reports success so the fallback never runs. aarch64 with glibc below 2.28 (Ubuntu 18.04) x86_64 with glibc below 2.17 (RHEL 6) a musl image with no ldd at all, where the probe simply finds nothing Read the version instead, from ldd or getconf, and require it to clear astral's floor for the triple. Anything unreadable declines to the fallback. Also ask the userland for its bitness rather than trusting uname on a 64-bit kernel running a 32-bit userland, and follow astral in reading hw.optional.arm64 so a translated shell under Rosetta 2 still gets the native macOS build. Three more from the same pass: Report success only when the destination uv is executable. A copy onto a busy or read-only destination could leave a file that is not, and reporting success there skipped the fallback. Nothing is unwound on the failure path on purpose: the fallback installs over whatever is at the destination, and deleting there would take out a working uv the host already had. Clear the mark of the web on the launcher we author. WriteAllText replaces the unnamed data stream and leaves other NTFS streams alone, so a launch-studio.ps1 that somehow carried one would keep it across the rewrite, and RemoteSigned refuses a marked unsigned script. Store the security-block kind and resolve its wording in message(). stdout and stderr are read by independent threads, so a [TAURI:STEP] written before a block can be observed after it, and freezing the wording at observation time could tell a user nothing was changed on a run that had already installed PyTorch. Also require the error id to appear as the value of a FullyQualifiedErrorId field, so a scanner log that merely names it cannot attach antivirus guidance to whatever fails next. The host matrix in the shell test grows to 28 rows covering every case above, and removing the new gate fails 8 of them. install.rs gains two tests: 37 pass. * Replace a symlinked uv destination instead of writing through it Three from the review on the previous head. cp onto a destination that is a symlink follows the link, so installing over `~/.local/bin/uv -> /opt/homebrew/bin/uv` rewrote the Homebrew binary in place and left the link pointing at a file another package manager owns. Stage next to the destination and rename over it: rename replaces the link itself, and it is atomic, so a concurrent reader never sees a half-written uv either. The staging file is removed when the rename fails, so a failed run leaves no debris. Verify the Windows copy the same way the shell scripts now do. Copy-Item is non-terminating under the caller's ErrorActionPreference, so a locked or ACL-denied destination let execution reach `$haveUv = $true` and the function reported success over whatever was already there. Compare the destination against the archive we just verified, so a stale uv.exe cannot pass for the one we meant to install. install.ps1 carried the same shape and gets the same treatment. Point the header links at the heading that exists. The README has no "Install Unsloth Studio"; it is "Unsloth Studio (web UI)", whose anchor is #unsloth-studio-web-ui. Three test cases cover the symlink: the file behind the link is untouched, the link itself is replaced, and no staging file survives. Reverting the fix fails two of them. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * CI: keep one Application Control negative-control log per attempt Enforcement went live on the fourth try on the hosted runner, and a single overwritten log left the evidence artifact showing a pre-enforcement attempt's output beside a passing step. * Fail the build when the uv pin drifts from a version floor Before the pin, astral's endpoint always delivered the newest uv, so raising UV_MIN_VERSION was safe on its own. It is not any more: a floor above the pin means a host with no uv gets 0.12.1 installed and then judged too old by the same script that installed it, on the one path where the pin is what runs. Two checks. All four installers must name the same uv, or which version a machine ends up with depends on which script reached it first. And the pin must clear every floor in the tree (UV_MIN_VERSION, UV_OFFLINE_MIN_VERSION, $UvMinVersion). Raising a floor past the pin fails the first, bumping one installer's pin alone fails the second. * Fix two Windows-only test failures that predate this branch test_path_identity_failure_is_reported_as_unknown failed on both shells, on main as much as here. Test-StudioPathEqual reports an unresolvable path identity through Write-StudioLine, the harness extracts the mutex helpers but not that, and these scripts run under -ErrorActionPreference Stop, so the catch path died with CommandNotFound before the test could measure anything. Extracted rather than stubbed: it is self-contained, and a stub would keep passing if the real call ever went wrong. A new check asserts every installer function the extracted helpers call is in the harness, and it runs on every platform, so the next drift cannot hide where only a Windows runner would see it. Measured on a Windows runner: main fails 24 of these, this branch fails 2, and both of those 2 are in main's set. With this, 0. * Write the shell profile entry the pinned uv path no longer gets for free The P1 here is a real regression and it took a second look to see why. install.sh decides whether to add ~/.local/bin to the user's shell profile with `case ":$PATH:"`, near the end of the run. By then this process has prepended that directory twice, once for the uv bootstrap and once for the venv, so the guard answers yes for a login shell that would answer no and the profile line is never written. That was survivable while astral's installer ran, because it wrote its own profile line and its env file. The pinned path writes neither, so on a fresh account whose login PATH lacks ~/.local/bin the install succeeds, the current shell works, and the next terminal cannot find `unsloth` or `uv`. Snapshot the inherited PATH before anything prepends to it and test the guard against that. Two more from the same review. Honour a configured uv mirror exclusively. UV_INSTALLER_GHE_BASE_URL and UV_INSTALLER_GITHUB_BASE_URL already win outright in both PowerShell installers and in astral's own; the shell path ignored them and tried the public hosts first. A restricted network sets one precisely because those hosts are unreachable, and download() has no timeout, so it would hang rather than reach the fallback. Do not let the twin of an earlier clear erase a later AMSI verdict. Clear-TauriInstallError writes one logical clear to BOTH streams (install.ps1:198) and independent threads read them, so a block observed between a clear and its own twin was discarded by the twin. Ignore a clear identical to the one just processed; a genuine later recovery carries different text and still clears. Two tests cover both directions, 39 install tests pass. * Stage the uv copy under a per-process name An audit pass reproduced a race I introduced with the symlink fix. Both POSIX helpers staged through a fixed destination-side name, so two installers targeting one directory shared it: A finishes copying the staging file B opens the same path with truncation A renames that inode into place as uv B keeps writing through its open descriptor, which is now the published uv The published uv was observable at zero bytes until B resumed, which makes the claim in the comment about a concurrent reader flatly wrong. install.ps1 is covered by its named mutex, but nothing serialises the POSIX helpers, and studio/setup.sh runs standalone on every studio update. mktemp in the destination directory instead. Each rename then publishes a file no other process can still be writing, which is what the atomicity argument needed all along. The loser cleans up its own staging file and declines, so the caller falls back rather than reporting a success it did not achieve. * Keep a default install as quiet as it was when a uv mirror misbehaves Two console regressions from the audit pass, both on paths the install still recovers from. download() runs curl -LsSf, and -S deliberately prints its own errors. The fallback ran under run_maybe_quiet, so a failed download printed nothing before; the pinned attempts run outside that wrapper, so an unreachable mirror now put two curl: (N) lines on the console of a default install that then succeeded. Redirect stderr on the speculative attempts only, leaving download() untouched for every other caller. [TAURI:WARN] is a marker level install.sh has never emitted, and the app forwards unknown markers to its progress UI verbatim (install.rs:639), so a digest mismatch would have surfaced as raw text in the desktop window. Make it a verbose only stderr line: the next mirror or the fallback still runs, so a default install has nothing to say here. Printed-string diff against the merge base is back to additions inside $(...) capture plus that one verbose-gated line, with nothing removed or changed. * Ask the installed uv whether it runs before skipping the fallback The libc gate reads a glibc version from ldd or getconf and treats that as proof a GNU binary will execute. It is not. A stripped NixOS-derived image without nix-ld reports a glibc version through getconf while its loader lives in the Nix store, so the pinned x86_64 uv asks for /lib64/ld-linux-x86-64.so.2 and gets nothing. Every static check passed, so the helper reported success, the astral fallback was skipped, and the first real uv call failed with No such file or directory. astral's installer fails its own glibc probe on that host and ships the fully static musl archive, which runs. The user went from a working uv to none. The archive is digest-verified astral uv by the time it is placed, so ask it: run --version and require it to succeed. One exec closes the whole class rather than this one host, covering a wrong triple, a loader that is not where the binary looks, and a destination we could not really write. A test drives an archive whose uv cannot execute and requires the helper to decline; removing the exec check fails it. * Pair every clear with its twin, not just the previous one install.ps1 clears after each recovered step, so a lagging reader can be several clears behind when a block lands. With clears A then B on one stream and A's twin arriving on the other after the verdict, asking only whether this is the message just seen answers no, and the delayed twin discarded the verdict the guidance exists to explain. Each logical clear emits exactly two markers, so count unpaired ones by message: the first sighting is the clear, the next pairs with it. A test drives the A, B, verdict, A', B' ordering; 40 install tests pass. * Close the exactness gaps found by ten adversarial audits Ten independent audits, each asked to falsify the claim that this is pure hardening. Six things were worth changing. - The desktop updater is isolated again. It shipped with -I, it is the one managed invocation nobody types by hand, and it decides which install gets rewritten, so a user-site unsloth_cli must not answer `from unsloth_cli import app` there. Every other call site inherits, because the console script does. - The trampoline ends in sys.exit(app()), like the generated console script, so a returned value becomes the exit status. Typer raises SystemExit itself today, but the two routes have to agree. - A launcher that could not be restored keeps its recovery copies. Judged healthy through the interpreter is not the same as repaired, and deleting the copies threw away what a later run needed. - `unsloth studio update` puts the shim directory on PATH. An installer older than that directory put the venv Scripts dir there instead, so the .cmd was written where nothing would look for it. - The console script reconfigured its streams twice off Windows, once through the import gate and once through the module-entry path. - Replacing a bin\unsloth.cmd that carries neither our marker nor our trampoline now says so. Also states the scope plainly: this answers EXE-and-DLL enforcement of the unsigned console script. A machine that also enforces AppLocker's Script collection denies .cmd and .ps1 alike, and install.ps1 would not have run there either. The two test harnesses that extract functions out of install.ps1 now assert they define everything those functions call; both had already shipped a gap that made a check pass for the wrong reason. * Ask the interpreter, not site-packages, whether the managed CLI is there The quarantine fallback accepted an unsloth-*.dist-info or an unsloth_cli/ directory as proof of a runnable CLI. Neither is: an interrupted install, or an editable install whose checkout has moved, leaves metadata with nothing to import. This gate sits in front of the headless-public strip of .bootstrap_password, so a false yes lands the exact lockout its placement exists to prevent -- a public Studio with no login page and no plaintext recovery credential. find_spec through the managed interpreter answers the question the trampoline will actually ask, with the same sys.path[0] scrub so a checkout in the caller's cwd cannot stand in for the venv. A probe that produces no verdict at all falls back to the old on-disk layout, so a half-quarantined install still starts. * Hide the import probe's console window, as every other managed probe does * Validate the staged uv before it replaces a working one My own exec check was on the wrong side of the rename. The sequence that bites: a host has a uv good enough for UV_OFFLINE_MIN_VERSION but below UV_MIN_VERSION, so the block runs with _uv_present_before true; the pinned path renames over that working binary; the --version check then fails because the loader is missing or the destination is mounted noexec; the fallback download also fails. The installer neither restores the old uv nor reports that none is available, and every later command runs the broken one. Test the staging file instead, before the rename. It sits on the destination filesystem, so it answers the noexec question too, and a binary that cannot run here never gets to replace one that could. Two tests: a working incumbent uv survives an archive whose uv cannot execute, and the rejected staging file is cleaned up. Moving the check back after the rename fails the first. * Make each managed CLI probe ask the question its launch will answer Three findings from the latest review round, one theme: a probe that stands in for a launch has to run under the same conditions as that launch, or it can pass where the launch then fails. * The quarantine gate in `studio run` asked find_spec whether unsloth_cli resolves. It resolves for an emptied unsloth_cli/ directory (find_spec calls that a namespace package), for a package whose __init__ raises, and for one whose dependencies an interrupted install never fetched, and the trampoline's `from unsloth_cli import app` fails on all three. Verified: an empty package directory in a bare venv gives find_spec True and ImportError on the import. This gate stands in front of the headless-public strip of .bootstrap_password, so a false pass there is a public Studio with no login page and no plaintext recovery credential. The probe now performs that exact import. * The updater's interpreter health check ran without isolation while the launch it predicts, build_update_command in studio/src-tauri/src/update.rs, runs under Isolation::Isolated with PYTHONHOME/PYTHONPATH cleared. A foreign checkout on PYTHONPATH could answer --version for a managed package the update had broken, and validate_launcher would keep an update the next desktop launch cannot start. _managed_cli_argv now takes the same isolated flag the Rust Isolation enum carries; the health probe is the only caller that sets it, and a test pins that it stays the only one. Every other invocation keeps PYTHON* parity with the console script. * Binary resolution, second pass. With an interrupted migration leaving an interpreter in both layouts and a launcher in neither, layout order handed back the new base even when its site-packages was empty and the legacy base still held the package. A directory test rather than an import probe: this runs on the launch path and from the capability checks, so it stays a stat. Tests: the four unimportable package shapes, the isolated/inherited argv split and its single caller, and both directions of the two-interpreter tie-break. The old whole-file "no -I anywhere" assertion is now read off the ternary, since one deliberate -I exists. * Stop the AMSI guidance claiming more than it knows Two of these are honesty defects in text a blocked user reads. "nothing was changed on this machine" is false. Rust starts a diagnostics attempt and its phase log before PowerShell is ever spawned, and spawn_script can create ~/.unsloth first, so a pre-start block has already written to disk. The honest claim is that no installation step ran. "This is a false positive" is not something the classifier can know. It proves the output carries a PowerShell error id and nothing about the script's integrity, and install.ps1 can sit in a user-writable directory, so a locally modified copy can earn a genuine verdict. Telling someone to report a correct detection to their vendor is worse than telling them to reinstall from an official package first and only escalate if an unmodified copy is still blocked. Two smaller ones from the same pass. The matcher tested for the field name and the id independently, so a line naming both in prose qualified; it now requires the id to follow the colon and end at a comma or whitespace, which is what the comment always claimed. And the clear-pairing map is bounded: legitimate producers use a small fixed label set, and child output must not be able to grow it without limit. 42 install tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Honour astral's download override, and stop Unblock-File asking Three from the second audit round. Unblock-File declares SupportsShouldProcess at the default Medium impact, so a profile that sets $ConfirmPreference to Medium or Low gets a prompt from the line I added, even for a launcher that never carried the stream. -ErrorAction does not suppress a ShouldProcess prompt, and a noninteractive host turns it into an error that skips shortcut setup entirely. -Confirm:$false. UV_DOWNLOAD_URL and its older alias INSTALLER_DOWNLOAD_URL outrank the mirror variables in astral's installer, and the merge-base path inherited that because it ran astral's script. All four implementations now honour them first and exclusively. My earlier comment argued they point at a version the pin would reject, but that reasoning had it backwards: a host sets one because it cannot reach the public endpoints, so ignoring it meant public egress first and, with no timeout on the download, a hang instead of a fallback. The pin still applies, so a source serving a different build fails the digest and the caller falls back to astral's installer, which honours the same variable. chmod 0755 on the staging file rather than +x. cp gives it the umask default and +x then adds execute only where the umask allowed read, so a umask of 077 left uv unusable for every other account on a shared machine. astral ships them 0755. Four checks pin the override precedence across all four installers and the mode across both shell ones, with the behaviour verified against a stubbed downloader. * Validate uv before it replaces an incumbent on Windows, and bound the probe install.ps1 and studio/setup.ps1 copied the extracted uv.exe straight over the destination and only asked whether it ran afterwards. A host with a working older uv and a policy (AppLocker, WDAC, endpoint protection) that refuses the new one was left with neither. Run the extracted binary where it landed first, then keep a copy of the incumbent across the publish and restore it if the published copy will not run, since Windows has no atomic replace for a file that may be open. The probe itself is bounded: Start-Process with a 20s WaitForExit and redirected streams, and on POSIX no stdin plus a 20s ceiling where timeout exists. A binary this installer just downloaded must not be able to hang an unattended install by prompting or by never exiting. install.sh and studio/setup.sh also published the pinned uvx after rejecting the pinned uv, leaving a pairing that is never built or tested. A uv that fails to stage, copy or run now abandons the whole placement. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Verify each uv mirror, and persist PATH when the account has no rc file Both PowerShell installers checked the archive digest once, after the download loop had already broken out. A captive portal or a proxy answering 200 with its own body is a successful download by every measure Invoke-WebRequest has, so the first mirror consumed the only attempt and the second, healthy one was never tried. The digest now decides whether a mirror counts as served. install.sh picked a shell profile from .zshrc, .bashrc or .profile and did nothing when none existed. A fresh account has none: astral's installer used to create its own PATH setup there, the pinned path does not, so the next terminal resolved neither unsloth nor uv. Fall back to creating ~/.profile, which every POSIX login shell reads. The existing content guard keeps it written once. * Remove the install.sh a Windows upgrade would otherwise keep forever Windows bundles now carry only install.ps1, but NSIS writes the current resource manifest and deletes nothing, and the uninstaller deletes only what is in that manifest. An in-place upgrade from a release that bundled both installers left install.sh in $INSTDIR permanently, which also made the non-recursive RMDir "$INSTDIR" fail at uninstall. The pre-install and pre-uninstall hooks now delete it, so the population most likely to upgrade actually gets the split. Also silence the speculative mktemp -d in the pinned uv path: its failure falls back to astral's installer, so an unusable TMPDIR printed a line the user could not act on and that the merge base did not print. * Remove the pinned uv temporaries when an install is interrupted The pinned path unpacks a 40 MB archive into a work directory and stages the binary next to the destination, but only cleaned both up when the helper returned normally. A Ctrl-C in between left the archive behind and left a staging file inside a directory that is on PATH. Both paths are now published to the exit and signal traps as they are created and cleared when the helper releases them. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Persist the PATH the way each shell actually reads it, and fail a half-published pair Four follow-ups from review: A uvx that the archive carried but that could not be staged or renamed left uv published next to a stale or missing uvx and still reported success, skipping the fallback that would have installed both. Either half failing now fails the placement, in install.sh and studio/setup.sh. studio/setup.sh had none of the interrupt cleanup install.sh gained: a Ctrl-C left the unpacked archive behind and a staging file inside a directory on PATH. It now owns HUP, INT, TERM and EXIT for the duration of the pinned install and hands them back on the way out. fish sources none of the POSIX rc files, so the ~/.profile fallback was a no-op for a fish user. The persistence helper writes a conf.d drop-in with fish_add_path there, and honours ZDOTDIR for zsh. UV_INSTALL_DIR, UV_UNMANAGED_INSTALL, XDG_BIN_HOME and XDG_DATA_HOME can put uv somewhere other than ~/.local/bin, and astral's installer wrote a PATH line for whichever it picked. The pinned path now persists its own destination too, with UV_NO_MODIFY_PATH honoured as astral honours it. * Make the Windows uv publish a real transaction, and quote persisted paths The companion copies ran bare: under install.ps1's Stop preference a locked or ACL-denied destination threw past the rollback and left a mismatched set with the backups still on disk, and under setup.ps1's Continue preference it kept a stale companion and reported success. Both now copy under -ErrorAction Stop inside the transaction, so any failure unwinds like the others. A failed restore also used to delete the backup anyway, which is the one path in this block that could leave the host with less than it started with: the two things that make a restore fail, an open incumbent and a denied ACL, are the same two that made the replace risky. The backup is now kept and named. fish takes an unquoted path with a space as two directories, neither of which exists, so the drop-in single-quotes it; and the rc line is written inside double quotes, so a uv directory holding a dollar or a backtick is escaped. The second test caught a doubled backslash in the escaper itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments added by this branch Comments only, no code touched: 161 comment lines become 106 across install.sh, install.ps1, studio/setup.sh, studio/setup.ps1 and the NSIS hooks. Each one keeps the reason it was written for, said once. Verified with the PowerShell AST parser, sh -n and bash -n, the 50-check uv pinned release suite and 114 installer tests, and by confirming the diff contains no non-comment line. * Tighten the install.rs comments too Comments only: 35 lines become 27, each keeping the reason it was written for. 42 install tests pass and the diff contains no non-comment line. * Abort on a companion that cannot be backed up, and pair clears by stream A uvx.exe that could not be copied aside, because it is locked or its ACL denies reads, was skipped and the new uv.exe published anyway, so the function reported success with a mismatched pair and the fallback never ran. Any backup failure now fails the placement and runs the rollback, in install.ps1 and studio/setup.ps1. The ERROR_CLEAR pairing keyed only on the message, so two real clears of one label on one stream were taken for a clear and its twin. That happens: _install_torch_default_index emits its recovery during the install and again during the ROCm repair. A verdict landing between them was then erased by the genuinely later clear arriving on the other stream. The map is keyed by stream as well, so only the opposite stream's copy can consume a pending marker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not fail an install because the uv probe could not get an answer Three clean-machine CI legs that pass on main failed on this branch: arm64 and two Windows containers, all three with winget unavailable, which is the only condition under which the pinned fallback runs. Each downloaded the right asset, passed the digest, and then failed the probe. Start-Process -NoNewWindow with redirected streams does not behave in a container or on the arm64 image the way it does in a desktop session, and a boolean probe reported that as a broken binary and aborted the install. The probe is now tri-state. Only the binary answering non-zero is a failure. A launch that throws or a wait that times out is inconclusive, and since the digest already proved the bytes are astral's pinned release, an inconclusive probe publishes as the pre-pin code did. Every path prints why, with the captured stderr and the exit code, so the next occurrence is not opaque. Also from review: the POSIX path now stages both binaries and publishes them together with the incumbents saved aside, so a failed uvx rename restores the uv it replaced instead of leaving a new uv beside a stale uvx; the Windows rollback records the destination before the copy that can truncate it; and UV_UNMANAGED_INSTALL suppresses the profile write, as it does for astral. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Give setup.sh the same pair publish and PATH persistence as install.sh studio/setup.sh published uv and then uvx one after the other, so a failed uvx rename left a new uv beside the host's stale one, and the remote fallback can be unavailable. It now stages both, validates uv, and publishes the two renames back to back with the incumbents saved aside, restoring them if the second fails. setup.sh is also run directly for local and Colab setup, where astral's installer used to write the profile line for whichever destination it chose. Without one the PATH export died with that shell and every later run reinstalled uv. It now persists its own destination, with fish handled on its own terms and both of astral's opt-outs honoured. * Treat an empty uv exit code as no verdict, not as a failure The arm64 clean-machine leg still failed on the tri-state probe, and the diagnostic that came with it said why: "uv --version exited ." with no number. WaitForExit(ms) can return before the exit code is cached, so ExitCode was empty and an empty value is not 0, which read a working uv as broken. The parameterless WaitForExit settles it and returns at once because the process has already exited, and a code that is still missing is inconclusive rather than a failure, which is the same rule the launch and timeout paths already follow. Verified against pwsh that a real non-zero exit and a real launch failure still classify as failed and unknown respectively. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Eight review fixes across the uv publish and PATH persistence The fish escaper in studio/setup.sh reached sed as an invalid expression, so a fish user running setup directly would have had setup killed under set -e right after uv was published. It now matches the one in install.sh, and the test runs both escapers rather than reading them. An incumbent that cannot be hard-linked or copied cannot be restored either, so publishing over it would be a one-way move. Both shells now decline. Writing that test turned up that my own rollback deleted both incumbents when nothing had been published, since the no-predecessor branch cannot tell the two cases apart; the rollback is now reached only after a publish was attempted. A rollback with no predecessor removes the binary it published, rather than leaving half a pair the host never had. A signal between the two renames left the undo copy as the only reference to the incumbent, and the handler deleted it. It restores it now, in both shells. setup.sh prepended ~/.local/bin unconditionally after a successful pinned install, so a stale uv there could shadow a custom UV_INSTALL_DIR destination and the rest of setup would run the wrong one. That prepend is now only for astral's installer, which is what writes there. PATH entries are compared literally rather than as case patterns, so a destination holding *, ? or [ is not mistaken for an unrelated entry. On Windows, a .unsloth-old left behind by a failed restore is the only copy of a working uv, and the next run reused that exact name. It takes a distinct one. * Keep the pinned uv first on PATH, and only count an active profile entry install.sh prepends ~/.local/bin after the uv bootstrap, and astral's env file does too, so a custom UV_INSTALL_DIR destination was pushed behind a stale uv sitting in the home directory and every bare uv below picked the wrong one. The pinned destination goes back in front. setup.sh had the same shape and was fixed in defd2292a. The profile check treated any occurrence of the destination text as proof the PATH entry was already there, so a commented-out old export, or /opt/uv-old when the destination is /opt/uv, suppressed the write and left the next shell without uv. Comments are stripped and the directory has to appear as a whole entry. * Close five review findings on the quarantine and stubless paths * The installer still required Scripts\unsloth.exe to exist, and aborted the whole install when it did not. That reasoning held for a policy, which denies the file and leaves it on disk, but not for antivirus, which quarantines it out of a venv that still runs, and nothing past that point executes it: the setup handoff, the shortcuts and bin\unsloth.cmd all go through the interpreter. It refused to install or repair Studio for exactly the machines this change is for. Absence now asks the interpreter for --version through the trampoline, and only a venv that cannot answer fails, with the same older-unsloth guidance. * The import probe's no-verdict fallback is now split by cause. A timeout keeps the on-disk layout, because slow is not broken: a cold venv under an antivirus scan is exactly that, and the re-exec has no timeout of its own. A failure to START the interpreter fails closed, because the re-exec runs that same interpreter and will fail the same way, and the caller strips .bootstrap_password before re-execing on a headless public launch. * The updater's interpreter fallback used the launcher's 10s timeout for a call that has to import the entire CLI package. That is the work the import probe's 60s ceiling is deliberately generous for, and under the antivirus scan this path exists to survive the short one would call a healthy update broken and roll it back, once per recovery candidate. * Binary resolution now accepts an unsloth-*.dist-info alongside the package directory when ranking stubless venvs, matching _managed_cli_site_packages_ layout. A PEP 660 editable install leaves a .pth and a dist-info and no unsloth_cli/ at all, so the directory test alone ranked a working legacy venv below an empty new one. * managed_bin_fingerprint required fs::metadata on the launcher, which the stubless layout deliberately reports as a path that does not exist, so the capability cache could be neither read nor written and every preflight paid both probe subprocesses again. It falls back to python.exe, which is what starts the CLI there, while the cache key stays the launcher path. Tests: the fail-closed/fallback split in both directions, the timeout contract and that the two constants differ, the editable-install ranking with an unrelated dist-info as the negative control, the stubless fingerprint and its invalidation, and the installer gate through the extracted AST harness. * Gate the NSIS tidy-up, and remove an orphan uv on signal The pre-install hook runs before the user can still cancel, and $INSTDIR can be a directory they picked in the GUI, so deleting install.sh there could take a file that was never ours. Both hooks now only act where our own executable already is. A signal between the two renames restored a predecessor but did nothing when there was none, leaving a 0.12.1 uv beside whatever uvx the machine had. It now removes what it published, which is what the ordinary rollback already does. * Write the uv PATH entry to every startup file astral's installer wired astral's uv installer wires ~/.profile, each of .bashrc, .bash_profile and .bash_login that exists, .zshrc or .zshenv under ZDOTDIR, and a fish drop-in under ~/.config. Replacing that installer with a pinned archive meant the PATH entry only reached the one file for whichever shell happened to be running, so a bash user whose .bash_profile does not source .bashrc, a /bin/sh login, or anyone who later switched shells would have no uv on PATH where they used to. Both POSIX installers now write the same set, once each, with the existing whole-entry check keeping a re-run idempotent. Files that do not exist are not created, apart from ~/.profile, which astral creates too. * Cut the uv publish back to what the common case needs The rollback machinery that grew over the review rounds covered cases a user is very unlikely to meet: an incumbent that cannot be hard-linked, a signal landing between two renames, a restore that itself fails, a second installer racing the first. It was 281 net lines, and every finding in the last two rounds was in it rather than in the hardening. What stays is what the common case needs. POSIX stages both binaries, runs the staged uv, and publishes the pair with two renames; a failure anywhere before them leaves the destination untouched, and the caller falls back to astral's installer exactly as before. Windows probes the extracted uv.exe before touching the destination, then copies the three under -ErrorAction Stop and re-checks the digest at the destination. The staging files are still removed on a signal, since they live in a directory that is on PATH. 64 shell checks and 114 installer tests cover the rest. * Match the exact fish entry, and let a UNC launcher load The fish drop-in is the only thing that puts uv on a fish user's PATH, since fish reads none of the POSIX files, and its check treated any occurrence of the directory as proof: /opt/uv-old suppressed /opt/uv. It now matches the exact fish_add_path line it would write. A launcher on a UNC share is a remote script to PowerShell, and RemoteSigned refuses an unsigned one, so a roaming profile got a shortcut that exits without starting Studio. That case, and only that case, uses Bypass, and drops -WindowStyle Hidden with it so the pair the detections key on never appears. * Wire every startup file on a DEFAULT install too, and give setup.ps1 a fallback The all-profile PATH write was gated on the uv destination differing from ~/.local/bin, which is exactly where a normal install puts it, so every ordinary machine still got the single-file write the shim path has always done. Three independent audits found this. The gate is gone, and the idempotency check now also matches the $HOME-relative spelling the shim block writes, so the default case does not end up with two lines for one directory. studio/setup.ps1 replaced astral's installer with the pinned archive and had nothing to fall back to. A failed pinned install therefore left UseUv false and silently ran torch, bitsandbytes, Triton and the rest through pip: a different resolver, not just a different download. winget is the fallback, as install.ps1 already does, rather than the remote script this branch exists to remove. * Make the quarantine case survive root inference, PATH and the reset hint Three more from review, all the same shape: a Windows path that still treats the generated unsloth.exe as the only evidence of an install. * Root inference. _looks_like_installer_managed_studio_home accepted share/studio.conf or bin\unsloth.exe, and only install.sh writes studio.conf, so on a custom-root Windows install the quarantinable launcher was the only sentinel there was. Once antivirus took it, STUDIO_HOME fell back to ~/.unsloth/studio and every studio subcommand read and wrote the wrong tree while reporting success. bin\unsloth.cmd now counts, validated against the same marker pair and 8 KB ceiling Test-UnslothCmdShimFile and the uninstaller's recursive-delete guard use, because this decides which installation the CLI manages and the directory is on PATH. * The PATH gate took any leaf named unsloth.cmd as a usable launcher. Write-UnslothCmdShim warns and leaves an unwritable file alone, so a foreign shim in a custom root survives the run, and counting it put its directory on PATH and advertised someone else's command as the policy-safe way in. It goes through Test-UnslothCmdShimFile now. * The reset-password hint always advertised `-I -m unsloth_cli` on Windows. -I implies -s, so a pip install --user install was handed a command that cannot find its own package, and the person reading it is by definition already locked out. It now checks whether the package is inside the interpreter's prefix and otherwise prints the bootstrap unsloth_cli/__main__.py documents for exactly this case, which carries no double quote and so wraps identically for cmd and PowerShell. Tests: root inference through a validated .cmd with four rejected impostors, an oversized shim, POSIX unchanged; the PATH gate as a source contract; and a new studio/backend/tests/test_reset_password_command.py covering both interpreter shapes, the spaced-path fallback, the prefix check, and drift between the bootstrap here and _WINDOWS_CLI_ENTRYPOINT. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-08-13 07:54:51 -07:00
Daniel Han	ce3f5c90ed	Studio: set DYLD_LIBRARY_PATH for llama-server on macOS, and classify macOS startup failures (#8574) ... * Studio: give llama-server a dyld search path on macOS, and stop losing macOS startup diagnostics _llama_server_env_for_binary branched win32 vs "everything else", and that else branch is Linux: WSL/ROCm probing, pip nvidia wheel globs, CUDA toolkit paths, and LD_LIBRARY_PATH. dyld ignores LD_LIBRARY_PATH, so llama-server was launched on macOS with no library search path at all, while the installer's own staged-binary validation does set DYLD_LIBRARY_PATH and therefore passed on a path the real launch never took. sd_cpp_engine and stt_ggml_sidecar already map the variable per platform; llama_cpp.py was the outlier. Add the darwin branch plus a shared _loader_path_var(), and use it at the two other sites that equated "not Windows" with LD_LIBRARY_PATH (the Vulkan probe and the CPU fallback replay). Classification was Linux-only too. dyld shares no wording with glibc, so "Library not loaded", "Reason: tried: ... (no such file)", an invalid code signature, an incompatible architecture and "built for macOS X which is newer than running OS" all fell through to "check that the GGUF file is valid and you have enough memory" - advice about a file and a resource that were never the problem. Classify them, reusing the existing provenance-aware missing/unloadable library messages, and keep them ahead of the returncode heuristics: a dyld diagnostic is a fact, signal 9 is a guess. macOS also SIGKILLs a binary whose code signature is invalid before it can print anything, so signal 9 there now offers both readings instead of memory alone. Finally, make the unknown-failure fallback self-diagnosing. Studio already writes every byte of llama-server's stdout and stderr to a per-attempt log, and already keeps the last 50 lines in memory, but the 400 response threw both away precisely when nothing recognised the output. It now carries a bounded, control-character-stripped output tail and the log path. With neither, the old message is returned byte for byte. Reported in #8566, where a Mac could load no GGUF at all and the error named the two things that were fine. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the platform in the SIGKILL test macOS SIGKILLs a binary with an invalid code signature the same way the OOM killer does, so the -9 message names both readings there. The existing test asserted the .wslconfig wording against whatever host it ran on, which made it fail on the macOS runners. Pin it to the non-Darwin platform and assert the two messages stay distinct. * Update the capability-probe env test for the macOS loader path The test simulated darwin and asserted the binary's dir landed on LD_LIBRARY_PATH, which is the bug: probe_server_capabilities goes through _llama_server_env_for_binary, so on macOS it ran llama-server --help with a variable dyld ignores. Assert DYLD_LIBRARY_PATH instead, and that an inherited LD_LIBRARY_PATH is left as the user set it. * Address review: bound and scrub the diagnostics, judge the right dyld candidate Six fixes from the review of the first two commits. A stray "Reason:" anywhere in the output used to be read as dyld's, because the capture was DOTALL and matched the first occurrence anywhere, running to the end of the output. llama.cpp prints its own "Reason:" lines, and Studio appends its own health-timeout marker, so the classifier could quote either back as dyld's diagnosis, or pick a verdict from a line belonging to a different failure. The reason is now bounded to its own line plus the indented continuations and read only after the "Library not loaded:" it explains. dyld4 lists every path it searched with a per-path verdict, so a mixed list is normal: our own dylib missing, and a leftover Intel Homebrew copy under /usr/local reporting an incompatible architecture or a policy-blocked signature. Scanning the flattened list let the scariest entry decide, which told users their install had been tampered with when a file was merely absent. Parse the "tried:" list and judge the candidate in the runtime's own lib dir. A stray "Reason:" also disabled the symbol-mismatch branch entirely, since the lib/reason block returned instead of falling through. "@rpath/libfoo.dylib" was passed to the shared message helpers as if it were a path, producing "missing from that exact location" for a search directive that is not a location. Strip the directive down to the soname. The macOS SIGKILL message hardcoded `unsloth studio update`, which cannot touch a pinned LLAMA_SERVER_PATH, and promised the log would say which cause it was when a signature kill leaves the log empty. Route it through _runtime_remedy and let it carry whatever diagnostics exist. The output tail is llama-server's own stdout, and llama-server inherits nearly all of Studio's environment, so a wrapper or diagnostic build echoing its env would have put an API key in the load error. Redact credential-shaped env values, plus bare hf_ and Bearer tokens, and slice the tail before filtering it character by character so a runaway unterminated line is not walked in full. Also fixes the same missing dyld search path in the RAG embedding server, which Apple Silicon reaches through its use_gpu branch. It is the same bug as the chat server's, in a second launcher. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Assert Studio's llama-server launch environment in the macOS CI The Mac job already installs a real llama.cpp and launches llama-server to prove it loads on that host, but it launches it from the shell's own environment, which says nothing about the one Studio builds for its child. That environment was Linux-shaped on macOS (LD_LIBRARY_PATH, which dyld ignores), and the installer's own staged-binary validation does set DYLD_LIBRARY_PATH, so nothing at install time could see the defect. Assert it here, where a real runtime is installed. A unit test with a monkeypatched sys.platform can only prove the branch is taken; this proves the launch environment points dyld at the dylibs that are actually on disk. Runs under the interpreter the `unsloth` shim points at, so the backend imports resolve without installing anything extra, and skips cleanly if no shim is on PATH. Verified on macos-14: fails on the pre-fix revision of llama_cpp.py, passes on the current one. * Cover the CPU, wrapper and probe paths the first macOS fix missed Five follow-ups from the second review round, all consequences of the first fix not reaching far enough. The RAG embedding server only got the dyld search path on its GPU branch, because it was bolted onto the CUDA helper. A CPU start (EMBED_DEVICE=cpu, or the retry after a failed GPU start) loads the same sibling dylibs, so the path now applies whichever device it lands on. That server also passed Path(binary).parent as the library directory, while the chat backend resolves the entrypoint with _llama_lib_dir. The managed install puts an entrypoint in front of the real server, so on a normal install the wrapper's own directory was named instead of the one holding the dylibs. Its capability probe ran `llama-server --help` with no loader environment at all, ahead of any launch. A bundle that needs the search path dies in the loader there, and the error text reads as help output with no --embedding in it, so the user was told their build lacks embedding support instead of that it failed to load. The probe now runs under the same environment as the launch. On macOS the launch resolves a shell entrypoint before spawning it. SIP purges DYLD_* while starting the protected /bin/sh, so the loader path did not survive a wrapper's exec of the real binary. The wrapper does nothing but exec the target, so this is the same launch without the shell hop. Secret redaction in the diagnostics tail now uses the same predicate that decides what to strip from a managed server's own environment, plus URL userinfo and common token shapes. The previous name-marker list missed DATABASE_URL, REDIS_URL and GITHUB_PAT, which is exactly the class of variable whose name says nothing about its contents. * Read the symbol's provenance, and stop the CI check from skipping itself Two follow-ups from the third review round. A "Symbol not found" with no missing file was always reported as llama.cpp libraries from different builds. That is right when dyld expected the symbol in one of our dylibs, and wrong when it expected it in a system framework: there the build wants a newer macOS than this one, and reinstalling the same build cannot make the OS export the symbol. Read the "Expected in:" path and pick the remedy from it. The MTLResidency special case is the same failure, now covered by the general rule as well. The macOS launch-environment assertion looked the interpreter up with `command -v unsloth`. The clean-machine lane scrubs PATH to system directories and puts the shim under its own UNSLOTH_STUDIO_HOME, so the lookup came back empty exactly there and the check skipped, while the job still reported success. It resolves from STUDIO_HOME first now, and a missing interpreter fails rather than skips: an install that produced a llama-server but no reachable interpreter is a broken install, and a skip is indistinguishable from a pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the tauri venv, the embedding entrypoint, and the launch api-key Three follow-ups from the fourth review round. The interpreter lookup added last round misses the tauri delivery, which nests its venv one level deeper at $STUDIO_HOME/studio/unsloth_studio. Since the same commit made a missing interpreter fail rather than skip, that turned a good tauri install into a red job. Both layouts are candidates now. The embedding server built its loader environment from the resolved directory but still probed and launched the entrypoint itself, so on macOS SIP would take DYLD_* away again on the way through the wrapper. It resolves the executable once, before the capability probe, so both the probe and the spawn run the real binary. The startup diagnostics could echo our own --api-key. It is minted per launch with secrets.token_urlsafe(32), so it is in no environment variable to look up and matches no token shape; the log path already treated it as sensitive but the API error did not. It is now redacted by its flag, and the live value is handed to the scrubber so it goes wherever it appears. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope the macOS resolution and the redaction more tightly Five follow-ups from the fifth review round, all narrowing something the previous rounds made too broad. Resolving a shell entrypoint past its exec line is right for the one the installer generates and wrong for a user's own: a custom LLAMA_SERVER_PATH wrapper may export backend variables before it execs, and jumping to the target would drop that silently. Resolution is now gated on the binary being ours, through a single _exec_path_for_launch used by the chat launch, the capability probe and the embedding backend. That probe needed it too. Only load_model resolved, so the startup, status and preflight routes still probed the wrapper, lost DYLD_* to SIP, and read back inconclusive capabilities, which clamps parallel slots and disables DSpark and DFlash sizing on a server that supports both. Stripping only the leading directive left "@loader_path/../Frameworks/libomp.dylib" as "../Frameworks/libomp.dylib", still path-like enough to earn the same "missing from that exact location" advice one level in. Placeholder install names reduce to the library name now. The classifier is pure text and runs on every platform, so keying a macOS diagnosis on the words "Symbol not found" claimed a Linux wrapper's plugin error as a mixed macOS runtime, and suppressed the output tail that would have shown what really happened. It now requires dyld's own framing. Redaction skipped any value under eight characters, which is the right rule for a port number and the wrong one for DATABASE_PASSWORD=hunter2. Short values from secret-named variables are redacted where they appear beside their name, rather than globally. * Tighten the comments this change adds Comment-only pass over the new code: same intent, 27 fewer lines. The explanations of why macOS needed each of these were written while working the problem out and read like it. * Compare the binary revision in the path space the launch recorded _binary_revision keys on the path string, and the macOS launch now resolves a managed entrypoint to its target, so the changed-since-launch check compared a resolved path against an unresolved discovery path. On a managed install, where the entrypoint is a symlink or wrapper, those never match, so an unchanged llama.cpp read as a fresh update and every Apply reloaded the model. Resolve on the comparison side too. The regression test fails without the fix, and its sibling pins that a genuine update is still detected. * Treat a --with-llama-cpp-dir tree as the user's own _is_unsloth_managed_binary asked only whether the binary sits under the managed root, which a --with-llama-cpp-dir install does: setup.sh makes the canonical llama.cpp directory a symlink to the user's checkout. The update flow already detects that case and refuses to write through the link, so the same install was being called managed here and unmanaged there. Two consequences, both now fixed by consulting the same predicate the update flow uses: the macOS launch resolved past the user's own entrypoint, dropping whatever setup it does, and a failure told them to run an updater that declines to touch their tree. * Pin the inherited value instead of asserting the key is absent The macOS embedding-env test asserted LD_LIBRARY_PATH was not in the child env at all. That env is a copy of os.environ, so the key is there whenever the ambient environment has it, whatever this branch does: the test passed alone and failed in a full run. Set a sentinel and assert it survives untouched, matching how the chat-backend test states the same invariant. * Scope the non-macOS behaviour changes out, and bound the dyld parsing Five independent audits of this branch, all asked the same question: is anything outside macOS different afterwards. Everything they found that was, is fixed here, each one reproduced against origin/main first. Cross-platform behaviour restored: - The macOS loader classifier ran on any output that merely contained 'Library not loaded:' or 'Symbol not found'. llama.cpp echoes GGUF metadata to stderr while loading, so a model whose general.name holds either string got library advice, and on two paths that outranked a correct answer: status 127 and signal 15. Every branch now requires dyld's own line-anchored framing. - _is_unsloth_managed_binary learning about --with-llama-cpp-dir also moved the gate on the Vulkan CPU fallback, which reads and copies the tree and needs no updater. Split into a second predicate so those installs keep the fallback on every platform. - The embedding capability probe was given a rebuilt environment, and its GPU library dir was resolved through the entrypoint, on all three platforms. Both are macOS-only now; Linux and Windows are byte for byte as they were. Correctness: - An exact LLAMA_SERVER_PATH pin now outranks inferred ownership. A wrapper pinned inside a managed tree read as ours and was resolved past, dropping whatever it exported before its exec line. - A quoted 'Expected in:' path failed the /System/ test, so a too-new build was blamed on a mismatched install. - Older dyld says 'mach-o, but wrong architecture'; 'Bad CPU type in executable' never reaches dyld at all. Both were falling through. - The signal 9 wording no longer reads as an exhaustive pair of causes. Bounds and idempotency: - The tried-candidate scan went quadratic on adversarial output: 100KB of "'a' (" took 6.3s against 0.0s on main. The reason is windowed before matching, capped after it, and candidates are counted. - _with_startup_diagnostics recognises its own output, so a second call cannot double the tail. One call site exists, so this cannot fire yet. - One shared loader-path prepend, deduplicating on the normalised spelling, and the embedding one returns a new dict instead of editing the caller's. - Bearer matching covers the full token68 alphabet and Basic. Verified by differential probes against origin/main: 1080 child-env scenarios over platform x GPU vendor x install layout x inherited environment differ on darwin only, and 1392 classifier scenarios differ only where dyld is genuinely the cause or the generic fallback gains its new output tail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make three loader tests independent of the host they run on They simulate a POSIX platform by monkeypatching sys.platform, but os.path and pathlib stay the host's, so on the Windows runner Path('/opt/llama/bin/llama-server').resolve() came back drive-anchored and splitting an LD_LIBRARY_PATH on ':' cut 'C:\...' after the drive letter. Product behaviour is not involved: the darwin branch never runs on Windows, and the Linux branch's own separator is correct there. Paths now come from tmp_path, and the ordering assertion uses startswith rather than splitting on a character that appears inside Windows paths. * Stub the install-tree gate where the CPU replay tests stub provenance _cpu_isolated_binary now asks _is_llama_install_tree rather than _is_unsloth_managed_binary, and three fixtures stub only the latter, so the gate fell through to the real _llama_install_root on a tmp_path with no install marker and every staged-runtime test got None. Product behaviour is not involved. Checked directly against origin/main on four trees: a marked install, a --with-llama-cpp-dir link, an unmarked directory and a binary pinned outside any install. The new gate returns what main's gate returned in all four, including restoring True for the link, which is the case this PR was fixing. * Use the Sequence import instead of hiding it in a string annotation verify_import_hoist flagged Sequence as added-but-unused: the three annotations that need it are quoted, so nothing references the name at module scope. Both Sequence and Optional are unconditional imports here, so the quotes bought nothing. Unquoting them makes the import real. scripts/verify_import_hoist.py --before origin/main --after now reports PASS rather than OVERALL: FAIL. * Follow a wrapper chain, and make re-classification a fixed point Two findings from a second round of ten independent audits, both reproduced against origin/main first. _resolve_llama_binary followed one hop. A wrapper whose target is another wrapper resolved to the intermediate script, which on macOS is the very defect this fix exists to avoid (SIP drops DYLD_* through the shell) and also handed _llama_lib_dir the wrapper's directory instead of the one holding the dylibs. It now follows the chain, bounded, and stops on a repeat so a pair pointing at each other cannot spin. _with_startup_diagnostics guarded its "message" argument, but the composition that actually grows is feeding a classified result back in as the child "output": 222 -> 413 -> 604 characters over three passes, and a specific dyld diagnosis downgraded to the generic fallback. The detector is now shared and applied to both sides. A classified message carries no marker of its own, so one wrap still happens before it settles; tagging the return type to close that too is a bigger change to a widely used error path than an unreachable property justifies, and the test says so. No caller feeds either composition today. These keep stated properties true rather than fixing live bugs. * Skip only the entrypoint the installer wrote, and redact quoted secrets Two P2 items from this round, both reproduced against head first. _llama_install_root treats the directory named by UNSLOTH_LLAMA_CPP_PATH as the active install with no marker file needed, so provenance alone answered "ours" for a wrapper at the root of a user's own checkout, and launching its target dropped whatever it exported. The LLAMA_SERVER_PATH pin added earlier covers only the case where they name the file. install_llama_prebuilt writes a fixed three-line wrapper, so that exact shape, a symlink, or a real executable is now the condition for resolving past an entrypoint; anything else is theirs and is launched as written. _resolve_llama_binary keeps resolving any wrapper, because _llama_lib_dir needs the directory holding the dylibs and that is the target's directory whoever wrote the script. The name-adjacent rule for environment values under eight characters matched only a bare NAME=value, so DB_PASSWORD='hunter2' and the JSON form went into the startup-output tail intact. It now allows quotes around the name and the value, requiring the same quote on both sides so an unrelated neighbouring quote cannot be swallowed. Verified before and after on a markerless custom directory and on five env-dump spellings; 680 tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Widen the tail window by the longest secret before redacting _scrub_secret_values matches a whole value, so a credential longer than the 8000-character prefilter lost its head to the slice that runs before it, and the surviving suffix matched neither the literal nor any token shape. A PEM key or a service-account blob dumped by a wrapper therefore reached the API error. Reproduced with a 12010-character MY_PRIVATE_KEY: the marker at its end came through before, and does not now. The window is widened by the longest value the scrubber could be asked to match, capped at 256KB so a pathological environment value cannot turn a bounded slice into a scan of everything the child printed. The final 2000 character bound on the emitted tail is unchanged. Worth recording: a secret SHORTER than the window cannot straddle its boundary, since a value whose end is inside a window wider than itself has its start inside too. Only values longer than the window were ever at risk, and a test pins that reasoning. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Decide on the wrapper's shape alone, not on how it was reached The exact LLAMA_SERVER_PATH pin added two rounds ago short-circuited resolution outright, to protect a custom wrapper's setup. That was too broad. Pointing LLAMA_SERVER_PATH at the installer's own entrypoint is a supported configuration, and refusing to resolve it just because it was named put the launch back through /bin/sh, where SIP drops DYLD_* before the real binary runs, reproducing #8566 for that setup and for the capability and embedding probes with it. _is_installer_entrypoint already answers the real question, so the pin check is gone. An installer-template wrapper does nothing but exec its target, so resolving past one is never a loss however it was reached; anything else is somebody's own script and is launched as written however it was reached. Reproduced inside a marked install tree before and after: a pinned template wrapper now resolves to build/bin/llama-server, a pinned custom wrapper is still launched as written. 684 tests pass, and the 1080-scenario child environment differential still differs from main on darwin only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Never trust the child's output for being our own framing _classify_llama_start_failure returned the child's stdout unchanged when that stdout carried the framing _with_startup_diagnostics writes. I added that early return two rounds ago to keep re-classification a fixed point. It was a bad trade: the guard reads untrusted text, so a wrapper or a diagnostic build whose first line happens to be "llama-server output:" or "Full log: " had its whole stdout returned as the API error, past the redaction and past the 2000-character cap. Measured on the branch before this commit, with an environment dump behind the heading: 50067 characters returned verbatim with the credential intact and no log path, against 2136 characters, scrubbed, for the same bytes without the heading. Removed the early return. The only guard on our own framing is the one in _with_startup_diagnostics, which reads the message this module built itself, and that one is safe. The property this drops was for a caller that does not exist. What replaces it is the property that actually matters and is now pinned by a test: repeated passes stay bounded. Each pass wraps the previous message in a fresh tail, roughly 136 characters, and the tail cap stops it at 2136 no matter how many times it runs. Tests: the fixed-point test is replaced by a bounded-growth test plus a parametrised regression test covering all four ways the heading can appear. 141 tests in the classification suite, 688 across the changed areas. * Do not let the child's output rewrite the diagnosis Attaching a tail of llama-server's own stdout to the failure message had a consequence I missed. The inference route scans the whole error string for unsupported-model phrases: _NOT_SUPPORTED_HINTS = ("No config file found", "not yet supported", "is not supported", "does not support") Those now match inside the quoted child output. So a llama-server that prints a line like "ggml_vulkan: device Intel(R) UHD does not support 16-bit storage" gets its failure rewritten to "This model is not supported yet. Try a different model." The model was fine; the message sends the user to replace it. llama.cpp prints lines of that shape for ordinary reasons, so this was reachable on Linux and Windows as much as on macOS. Reproduced on the branch before this commit with three different phrasings. _diagnosis_text() cuts the message at the diagnostics block, and both the unsupported-model matcher and the NVFP4 matcher now decide on the part this backend wrote. The evidence still reaches the user, and is still quoted in full when a rewrite does happen; it just no longer votes on the diagnosis. A message with no diagnostics block is returned unchanged, so every other error source is untouched. Tests: 11 new, covering four real llama.cpp phrasings, the same phrase in our own text (still rewritten), our text carrying a tail (still rewritten, evidence preserved), the NVFP4 matcher, and four shapes that must pass through unchanged. 710 tests pass across the changed areas. Whole backend suite against origin/main: 244 failures on each side, the same 244, no regressions. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Redact by name as well as by value, and bound the library name Two findings from a fresh adversarial pass over the diagnostics tail, both reproduced on the branch before this commit. Redaction missed an encoded secret. _scrub_secret_values matched the credential's literal text, so it only worked when the child printed the value verbatim. A wrapper dumping its environment as JSON prints DB_PASSWORD = pa"ss\word-12345 as {"DB_PASSWORD": "pa\"ss\\word-12345"} which is a different string. The escaped form went into the API error fully reconstructible. URL-encoding it did the same. _redact_secret_assignments replaces whatever sits beside a secret-looking NAME, in the three shapes an environment dump takes: shell-ish, JSON and bare. Positional, so the encoding does not matter, and it also covers a value we never set and therefore could not have matched on. The old value pass stays, because it catches a credential under a name that does not look like one. The remaining gap is an unrecognised name AND an escaped value. Checked against over-redaction, which would defeat the point of the tail: PATH, port, n_ctx, model_path, timings and llama_model_loader lines are all untouched. Both regex arms are unambiguous single-character alternations, so the pass is linear; 200000-character pathological inputs run in 7 to 52 ms. The library name was unbounded. A dyld install name is a path and macOS PATH_MAX is 1024, but nothing enforced that on untrusted output: a 200093-character input produced a 100444-character HTTP error. Capped at 2048 with an ellipsis, which cannot truncate anything the loader could have produced. A real name is unchanged. Tests: 22 new across encoded secrets, the never-set value, six over-redaction controls, four linearity checks and the name cap. 722 pass across the changed areas. The 1392-scenario classifier differential against origin/main is unchanged: 540 non-dyld differences, all still strictly additive, 0 violations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Redact every classified branch, fix the quoted arm, stop at custom wrappers Three findings, all reproduced on the branch before this commit. Two of the three are defects in code I added earlier in this PR. Redaction only covered the tail. _scrub_secret_values ran inside _with_startup_diagnostics, so it protected the fallback and nothing else. Every other branch also quotes untrusted text: a dyld message names the library it could not load, and that name comes from the child. A credential appearing there went out in the API error while the same credential in the tail was starred out. Verified with a token as the library name: leaked before, redacted now. The fix is one boundary rather than one guard per interpolation, which is the arrangement that let this through in the first place. The classifier body moves to _classify_start_failure_text and the public function scrubs whatever comes back. Scrubbing twice is a no-op, since a redacted value no longer matches. The quoted arm ended on either quote character. So a JSON value holding an apostrophe, or a shell value holding a quote, failed to match, fell through to the bare arm, and that arm stops at whitespace: {"DB_PASSWORD": "prefix' supersecret"} left ` supersecret"` standing in the API error. The arm now closes on the delimiter it opened with. Still linear, still an unambiguous alternation. Launch resolution stepped over somebody else's wrapper. Approving the outer entrypoint says nothing about what it points at, so an installer-shaped symlink whose target was a hand-written wrapper had that wrapper skipped on macOS, along with any exports it set, even though running the symlink directly would have executed it. _resolve_llama_binary grows a template_only flag: launching stops at the first link that is not the installer's own template, while the library-directory lookup keeps following the whole chain, because that is where the dylibs sit whoever wrote the links. Pinned by a test on each of the three shapes. Tests: 9 new. 731 pass across the changed areas, 1482 across every llama.cpp and secret-handling suite. Both differentials against origin/main are unchanged: 1392 classifier scenarios with 540 non-dyld differences, all still strictly additive and 0 violations, and 1080 env scenarios differing on darwin only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Close three more holes in the name-anchored redaction All three reproduced on the previous head. Each is reachable by a wrapper or crash handler that prints its own configuration rather than the environment we can match values against, which is the case the name pass exists to cover and where it was weakest. An unterminated quoted value leaked its tail. A truncated line ends the output mid-value, so DB_PASSWORD="prefix supersecret has no closing delimiter, cannot match the terminated arm, and fell to the bare arm, which stops at whitespace and left ` supersecret` behind. A third arm now runs an opened-but-unclosed quote to the end of the line, ordered after the terminated arm so a properly closed value still prefers that one. The opening delimiter is kept and no closing one is invented, since that would misreport what the child printed. Only the value's line is consumed; a following line survives. A config-style key was never tested. The name accepted shell identifiers only, so `db-password` and `api-key` never reached is_secret_env_name at all. The name now accepts hyphens and dots, and the separator is normalised to an underscore before the predicate is asked, because the predicate's markers are underscored (API_KEY, PRIVATE_KEY). The predicate stays the one place that decides what counts as a secret. Checked hard against over-redaction, since a wider name is the obvious way to make this worse: model-path, ggml.backend, n-gpu-layers, cache.type-k, llama_model_loader and timing lines are all untouched. Overlapping literals masked each other. With TOKEN_A=abcdefgh and TOKEN_B="abcdefgh VERYSECRET", replacing the short value first rewrote the long one's only occurrence to "*** VERYSECRET", after which the long value no longer matched itself and its tail went out. The minted secrets and the environment's own values are now collected into one set and replaced longest first, so a credential cannot partially mask another before it is scrubbed. Order-independent, verified both ways round. Tests: 20 new. 185 in the classification suite, 1715 across every llama.cpp and secret-handling suite. Both differentials against origin/main unchanged: 540 non-dyld classifier differences, all strictly additive, 0 violations; 1080 env scenarios differing on darwin only. Linearity re-checked on the widened pattern: 200000-character inputs in 15 to 61 ms. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-13 06:32:35 -07:00
Wasim Yousef Said	72ab966221	Installer: suppress macOS uv developer tools dialog (#8479) ... * Desktop: stop the loopback client following redirects `loopback_http::client` is the client that posts `.desktop_secret` to /api/auth/desktop-login, and it was built without a redirect policy. reqwest follows up to 10 redirects by default, and its cross-host protection strips headers rather than bodies, so a responder answering 307 (which preserves the method and the body) would carry the secret to whatever the Location header names, after the loopback URL had already been checked. Its sibling `streaming_client` already refuses redirects for exactly this reason: "Redirects are refused so a loopback URL cannot be bounced off-host after the check." Give `client` the same policy. No behaviour change for any real backend, which never redirects these routes. * Suppress macOS uv developer tools dialog * Update workspace guard for uv wrapper * Stop the installer raising the macOS command line developer tools dialog On a Mac without the Command Line Tools, /usr/bin/git, lipo, install_name_tool and friends are libxcselect shims. Executing one resolves no developer dir and posts to com.apple.dt.CommandLineTools.installondemand, which draws the 'requires the command line developer tools' dialog naming the tool. Resolving the path does not; only execution does. Two call sites execute a shim on the consumer path: _has_working_git ran 'git --version' to decide whether git works, so on a clean Mac the probe raised the dialog it exists to detect. It now answers from the resolved path when that path is exactly /usr/bin/git and no toolchain is selected. Deliberately narrow: a Homebrew, MacPorts or Xcode.app git earlier on PATH is a real binary and is still probed by executing it, so a Mac with a working git but no CLT selected behaves exactly as before. An earlier version of this gated on 'no CLT implies no working git' and broke that case, which the existing test caught. xcode-select -p only asks which toolchain is selected and never prompts. The venv arch probe called lipo first and fell back to file -L. lipo is a shim; 2>/dev/null hides its stderr but not a GUI dialog. file is base system and always answers, so the order is swapped. Both spellings feed the same case below, against 'Mach-O 64-bit executable arm64' or 'universal binary ... [x86_64] [arm64]' rather than lipo's 'arm64' / 'x86_64 arm64', so the branch taken is unchanged. clean-machine-assert.sh already made this same swap for its own use. The cctools binaries were missing from the clean machine CI tool list, so none of this was visible: trace mode generated no wrapper and the absent list never checked them. install_name_tool, lipo, otool, objdump, vtool, strip and nm are added, which is what makes these fixes regression testable. test_macos_clt_gate.sh gains two cases pinning the contract: with a shim git and no toolchain selected the probe answers no WITHOUT executing it, proven by a stub that records execution into a marker file, and with a real git elsewhere on PATH the stub IS executed. The first assertion passed vacuously when written (wrong temp path meant the marker could never be created) and was fixed by making its pair fail first. 18 to 23 passing. * Preserve working git on Intel macOS * Keep the git shim guard on under Rosetta --------- Co-authored-by: danielhanchen <unslothai@gmail.com> Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-08-11 12:19:30 -07:00
Wasim Yousef Said	67af9aa825	Desktop: unify normal release updater flow (#8298) ... * CI: point desktop updater test at fork * Studio: simplify desktop setup progress UI * CI: limit updater A/B build to macOS and Windows * CI: publish macOS and Windows updater test assets * CI: build Linux and Windows updater test assets * CI: pin updater test to Windows 2022 * Fix latest-main updater test workflow merge * Resolve latest-main startup message merge * Desktop: unify normal release updater flow * Desktop: validate updater signatures * Desktop: preserve generated updater signature * Restore macOS and target the existing v release for PR #8298 Restores the macOS leg that was dropped from the release pipeline: the macos-latest matrix entry, the .dmg and .app.tar.gz assets, the darwin-aarch64 platform entries in latest.json, and darwin-aarch64 in the required families of both release-desktop.yml and publish-desktop-updater.yml. Without them no macOS bundle is published and macOS clients find no matching platform in the manifest, so they stop updating entirely. Targets the v{version} release that already exists instead of creating it. The tag is cut when main is tagged, before this workflow is dispatched, so the old "tag already exists" guard failed every run on this repository; it only passed on a fork where the tags were absent. The guard now requires the release to exist and refuses only when it already carries desktop assets, naming the delete-asset commands to recover a failed publish. Provenance is appended to the release body rather than replacing it, since that body is the changelog. Windows step conditions follow the restored windows-latest matrix entry. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address the review on PR #8298 Gate the asset and manifest uploads on the draft input. The target release is already public, so a validation-only run was publishing unapproved binaries and latest.json to it. Move the provenance edit after the uploads and replace any earlier section instead of skipping it, so a retry that follows a partial upload records the digests that actually shipped rather than the previous build's. Reject a prerelease target in both guards. GitHub cannot mark a prerelease latest, so catching it only at promotion left the bundles already public. Re-read GitHub latest immediately before promotion. The downgrade check runs before a build that can take an hour, and promoting past a newer release would hand every client an older manifest. Build from the release tag rather than the dispatch ref. The release is published before the workflow runs and main keeps moving, so the bundles could come from unrelated source and provenance could record a SHA that is not the tag's. Skip the updater validation when the release carries no latest.json. The v release is published before the bundles land, so the release event fired first and failed on every release; it now fails closed only when the release cannot be read. Scope the signature sweep to the desktop bundles, since the release is shared. Add the AGPL-3.0 header to the two new files, in the style the repository uses. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep updater discovery on desktop metadata and record the built commit for PR #8298 * Send make_latest as the documented string for PR #8298 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair the release-creation tests and the publish-side guard for PR #8298 * Fail closed on promotion, order numbered prereleases and keep the pointer forwardable for PR #8298 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve the newest desktop release lazily and record the updater pointer gap for PR #8298 --------- Co-authored-by: danielhanchen <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-10 01:01:39 -07:00
oobabooga	495ba21ebd	Studio: add MiniMax H3 video generation (#7989) ... * Add MiniMax H3 video generation * Improve MiniMax H3 memory routing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address MiniMax H3 review feedback * Keep H3's VAE off the CPU path, so low_vram stops aborting low_vram maps to the `model` policy, and offload_flags emits --vae-on-cpu for it unconditionally. On H3 that kills the process: ggml/src/ggml-cpu/ops.cpp:6321: GGML_ASSERT(src0->type == GGML_TYPE_F16) failed deterministically, SIGABRT, exit 134. Bisected on the flags: --vae-on-cpu with --audio-vae aborts, the same command without --audio-vae renders in 95.87s, and an fp16-converted audio VAE aborts too. So the trigger is the audio VAE, not the video one, and the F32 type is imposed inside stable-diffusion.cpp rather than by the file: ggml_conv_1d hardcodes an F16 im2col destination (ggml/src/ggml.c), ggml_compute_forward_im2col_f16 then asserts the KERNEL is F16, and audio_conv_weight_type (src/model/vae/ltx_audio_vae.hpp) maps only BF16 to F16 and lets F32 through. It cannot be fixed by shipping a different checkpoint. low_vram is the one mode a small-card user reaches for, so this drops the flag rather than the mode. offload_flags takes vae_on_cpu, defaulting True so no other family changes, and the H3 native path passes False. --offload-to-cpu and --clip-on-cpu still apply, which is where the saving actually is: the denoiser dominates, and with --offload-to-cpu the whole model peaks at 13.14 GiB. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Point the H3 GGUF pick at the unsloth mirror Main added test_curated_gguf_repos_are_unsloth_mirrors, which requires every curated video gguf_repo to live under unsloth/ so a one-click pick cannot 404 when a community repack is renamed or taken down. H3 was the one family still on a community repo. This was meant to be part of the merge commit but was left in the worktree, so CI on that commit still saw the old value. unsloth/MiniMax-H3-GGUF is still private and has to be made public before this merges, or the pick will 401. No CI check reads it. * Pin H3's native cfg-scale under test H3 is distilled and CFG-free: its empty unconditional prompt encodes to zero tokens, and the transposed tensor that produces trips GGML_ASSERT(!ggml_is_transposed(a)) in ggml.c. SIGABRT, exit 134. Measured: cfg 1.0 renders, cfg 1.5 and cfg 4.0 both abort. sd.cpp defaults cfg-scale to 7.0, so this is a crash a plausible refactor reintroduces by forwarding guidance the way every other family does. The native path already hardcodes 1.0 and the family sets supports_cfg = False, but nothing held either in place. supports_cfg only gates the diffusers path; the native path builds its own params. Checked the test fails when cfg_scale is changed to forward guidance, so it is not passing vacuously. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin diffusers by source archive instead of git, so macOS installs work The macos-15-intel leg fails deterministically, not flakily: process didn't exit successfully: `/usr/bin/git init` (exit status: 1) --- stderr xcode-select: note: No developer tools were found, requesting install. That runner has no Xcode, so /usr/bin/git is the developer-tools shim and exits 1 for everything. uv needs a working git to resolve a git+https dependency, so the diffusers pin this branch added cannot install there at all. Main is unaffected because it depends on plain diffusers. GitHub serves the same commit as a source archive, which uv installs with no git involved. Verified by stubbing git to fail exactly the way the macOS shim does: the git+ form reproduces the CI error, the archive form installs diffusers 0.40.0.dev0 from the same SHA with MiniMaxH3Transformer3DModel present and exported. Also drops a full clone of diffusers from every install. * Allow the diffusers source build on the clean-machine legs With the archive pin the macOS leg gets past `git init` and installs, but then trips the nobuild guard: built from source: diffusers -- these must resolve to wheels on a clean machine There is no wheel to resolve to. MiniMax-H3 support is not in any diffusers release, so this branch has to pin a commit, and neither a git URL nor a source archive can produce a wheel from an index. Added to the same allowlist that already carries triton-kernels for the same reason. Checked against the bar the comment there sets: diffusers builds with plain setuptools, declares no ext_modules, and its tree has zero .c/.cpp/.pyx/.rs/.cu files and no shipped binaries, so the PEP 517 build is a pure-Python copy step and needs no toolchain. The separate compiler-invocation check in both scripts is untouched and still fires if one is ever needed. Verified the allowlist logic still rejects a non-allowlisted source build (a log building both diffusers and numpy reports only numpy). Remove this entry once a diffusers release carries H3 and the requirement goes back to a version specifier. Noted in both scripts. * Baseline the hf-hub retry loop reopened by the 1.x upgrade `pip scan-packages :: hf-stack` fails on this branch with 1 unbaselined CRITICAL: C2 polling/beaconing loop detected Package: huggingface-hub File: huggingface_hub/utils/_http.py L461: while True: This is on us, not upstream drift. Main pins huggingface-hub==0.36.2; this branch needs >=1.23.0,<2.0 because diffusers at the pinned commit requires it, so the resolved version moves 0.36.2 -> 1.27.0. The baseline already carries this exact file and check at L462 and L298, from earlier versions. It did not carry over because the key hashes the matched code, not the line number, and the surrounding code changed across the major version. That is the guard behaving correctly: changed code in a baselined file reopens for review rather than staying suppressed. Reviewed it rather than just re-suppressing. L461 is the retry loop in `http_backoff`: bounded by `nb_tries > max_retries`, the URL comes from the caller, there is no hardcoded endpoint and nothing is exfiltrated. Same benign construct as the entries it replaces. Entry generated with the scanner's own _evidence_hash rather than hand-written, and inserted beside its siblings so the diff stays 8 lines. Verified: with the baseline the scan is exit 0 with 4 suppressed, without it exit 1, so the guard still bites. * Lift the macOS-arm huggingface-hub cap that this branch made unsatisfiable `mac macos-15 / trace / file` fails on this branch and passes on main. uv reports: No solution found when resolving dependencies: Because you require huggingface-hub>=0.34.0,<1.0 and huggingface-hub>=1.23.0,<2.0, your requirements are unsatisfiable. This branch moved base.txt, no-torch-runtime.txt, studio.txt and constraints.txt to hub >=1.23.0,<2.0, because diffusers at the pinned commit requires it, but left the flat <1.0 cap in overrides-darwin-arm64.txt. That file is macOS-arm only, which is why only the mac legs see it and Linux and Windows stayed green. The failure then presents as something else entirely: uv gives up, the installer falls back to pip, and the clean-machine trace fails on "installer invoked toolchain: rustc" rather than on the resolution. The cap's own comment explains it exists so the resolver can never pair hub 1.x with a pinned transformers 4.57.6 / hub 0.36.2 stack. That is still true below python 3.10 and still capped there. At 3.10 and above this branch is on transformers 5.5.0 and hub 1.x, so the premise is gone, and mlx-audio's own >=1.0 floor is satisfied by the 1.x window anyway. Checked by collecting every hub specifier that applies per Python version across all five files: py3.9 resolves to 0.36.2 as intended, py3.13 to 1.23.0/1.27.0. Before this change py3.13 resolved to nothing. * Make H3's native download use the repo the family advertises The curated-mirror test main added only inspects VideoFamily.gguf_repo. H3's native path does not read that field: video_minimax_h3.py has its own H3_GGUF_REPO constant, used for both the transformer and the Qwen3-VL encoder. So pointing the family at the unsloth mirror in the previous commit satisfied that test while the actual one-click download still came from a community repack, which is the exact failure the test exists to prevent. Pointed the constant at the same mirror and added test_the_h3_native_repo_matches_the_family_gguf_repo to pin the pair, so the two cannot drift apart again. Verified it fails when the constant is put back to leejet, so it is not passing vacuously. The mirror now carries the Qwen3-VL encoder quants alongside the denoisers, byte-identical in size to the community ones, so this repo alone satisfies both of h3_native_hub_files' hub entries. The encoder is part of MiniMaxAI/MiniMax-H3 itself (FL2VA/text_encoder) which we already mirror publicly at unsloth/MiniMax-H3 under the same licence, so shipping a quantization of it beside the denoisers is the same act. Also checked the encoder-tier routing survives the dynamic rung names: -UD-Q2_K_XL selects the Q2_K_M encoder and -UD-Q3_K_XL the Q4_K_M one, asserted in the new test. Updated the download-plan test, which hardcoded the old repo id. NOTE: unsloth/MiniMax-H3-GGUF is private. Unlike before, that now really does gate this: the native path downloads from it. It has to be public before this merges. * Pin H3's companion-checkpoint guard under test validate_h3_transformer_filename had no test. That mattered less when the denoisers lived alone; the mirror now ships the Qwen3-VL encoder quants in the same repo, so the picker lists both and a user can name either. Loading a 12-17 GB encoder as the transformer would fail deep inside sd-cli instead of at the boundary. The accept cases include the dynamic rung names on purpose. The guard is a prefix/suffix check and `-UD-Q2_K_XL` is a shape it had never seen when it was written; it happens to pass, and now that is asserted rather than assumed. Checked the test fails when the prefix check is dropped, so it is not vacuous. * Record why H3 drops --vae-on-cpu, now that the abort is fixed The comment justified the drop entirely by an sd.cpp abort. That abort is fixed in the Unsloth fork, which would have made this look like a stale workaround to revert once the fix reaches the pinned prebuilt. Measured on a build carrying the fix, 640x384, 25 frames, 4 steps, q4_K, with --offload-to-cpu --clip-on-cpu already applied: adding --vae-on-cpu moved peak VRAM 12.42 -> 12.42 GiB and wall time 20.9s -> 100.4s. Under --offload-to-cpu the peak is set by the streamed denoiser, so the flag saves nothing and costs 4.8x. It stays off on its own merits. * Pin the sd.cpp prebuilt that actually renders MiniMax-H3 The pin was master-812-ea7f0c8, a stock upstream build, and on a stock build H3 does not work: it aborts on the default --cfg-scale, aborts again on --vae-on-cpu, and its 1-D norms are quantized into an output uncorrelated with its own bf16 reference (LPIPS 0.981). The Studio side worked around the first by pinning cfg to 1.0 and the second by dropping the flag; the third had no workaround on the consumer side at all. All three are fixed in unslothai/stable-diffusion.cpp and open upstream as leejet/stable-diffusion.cpp#1861, #1862 and #1863. The mirror's prebuilt pipeline now applies them on top of the aged upstream tag it already builds, and marks such a build with a -u<id> suffix naming the patch set, so master-813-bfbef5b-u0665242 is upstream master-813-bfbef5b plus those three patches and nothing else. The patches are deleted once upstream releases them, at which point this pin goes back to a plain tag. Verified on the published Linux x86_64 asset, not on a local build: both new error strings are in the shipped sd-cli, and running it on a q4_K H3 denoiser without --mode vid_gen now exits 1 with the instruction instead of core dumping on a ggml assert. test_video_backend's fake engine returned the old tag as its version string, which read like a second pin; it only needs a non-None value, so it now says so. * Close two gaps the H3 mirror switch opened Both are consequences of the two preceding commits, found in review. The prebuilt pin is now mirror-only (master-813-bfbef5b-u0665242), and _resolve_with_fallback still asked leejet for it. That request is a guaranteed 404 by construction, since the -u<id> suffix marks a build only the mirror makes, so it was a wasted round trip on every install. Worse, when the mirror genuinely cannot serve a host the fallback lands on leejet's latest, which has none of the H3 fixes. For every other model that is the right trade, better a stock native engine than none. For H3 it is not visible: it aborts on the default cfg-scale, aborts on --vae-on-cpu, and a blanket --type renders a broken video rather than failing. A user who saw only the generic 'falling back to leejet' line had nothing connecting that to the output. It now says so. Second, hub/utils/gguf.py filtered H3 companion GGUFs by the old community repo id only. The mirror the family and catalog now advertise carries the Qwen3-VL encoder quants beside the denoisers, so a 12 GB text encoder was being aggregated as if it were a selectable transformer quant. Both bundle repos are now recognised, case-insensitively, and the cache-dir match follows. Tests are mutation-verified rather than assumed: restoring the upstream 404 attempt fails the ordering test, removing the H3 warning fails the fallback test, and dropping the mirror from the bundle set fails three. The ordering test deliberately makes the mirror serve nothing, because with the mirror serving the first attempt succeeds and the upstream attempts are never reached, which made an earlier version of it pass under its own mutation. A third test pins the native loader's H3_GGUF_REPO to the bundle set, since those live in different files and a future repo move that updated only the loader would silently reintroduce the same aggregation bug. * Exclude MiniMax-H3's small-M projections from int8 _INT8_FAMILY_EXCLUDE_NAME_TOKENS has entries for qwen-image and hunyuanvideo-1.5 but none for minimax-h3, and H3 needs one for the same reason they do. H3's adaLN projection is named adaln_proj, which no token in the generic list matches: 'norm' is the closest and does not appear in the name. On the dense checkpoint that projection is Linear(2688 -> 96768), so it clears min_features = 512, gets quantized, and then runs at M = 1. Inductor lowers int8 matmul to _int_mm, which requires M > 16, so it raises 'self.size(0) needs to be greater than 16, but got 1' at the first denoise. The offline prequant builder bakes it in happily, which is exactly the drift the exclude list exists to prevent for Flux and Qwen. The pruned-modulation form hides this rather than fixing it: there adaln_proj is Linear(8 -> 96768) and falls under min_features anyway. So this exclusion is what makes the DENSE path correct and is a no-op on the pruned one. context_embedder and token_refiner are added for the same reason hunyuanvideo-1.5 excludes its text stream. Measured at M = 10 text tokens against the video stream's thousands, they are 3.47% of GEMM time even in the slow eager int8 path, so leaving them bf16 costs nothing measurable. This is what made an earlier measurement conclude int8 does not work on H3. It does: on the pruned form int8 compiles and is 4.24% +-0.54% faster than fp8 at identical memory, paired over 12 renders. * Say which H3 component could not be downloaded, and why H3 pulls four files from two repos, and the Hub returns the same 'Repository Not Found ... make sure you are authenticated' for a repo that does not exist, one that is private, and one your token does not cover. A user reading that has no way to tell which of the four failed, and the wording points away from the real cause whenever the repo exists but is not public. That is the state the GGUF mirror is in today: it is unpublished, so picking H3 fails with a message suggesting the user fix their token, which will not help. This replaces it with the repo, the component, and the actual remedy, and says the other components are unaffected so the failure is not read as total. Gated repos get different wording, since accepting a licence is a different action from waiting for a repo to be published. Anything that is not a recognised access error is passed back unchanged rather than reworded, so a timeout or a full disk still reads as itself. The helper returns the exception instead of raising, so the caller keeps raise-from and the original traceback survives. Mutation-verified three ways: rewording every error (timeouts included), fixing the component name to 'denoiser', and giving gated repos the private wording each fail the test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop MiniMax-H3 holding two copies of its video VAE Two thirds of an H3 render's peak is not activations. Measured at 640x384 across 124 frames, a 20.25 GB int8 denoiser peaks at 36.96 GB, and the gap is almost all weights: the video VAE alone is 10.42 GB because diffusers pins it to float32, and a further 4.91 GB is autocast's own float16 copy of those same weights. A memory snapshot puts 92.9% of the transient in 437 blocks allocated from nn.Linear, largest 67.1 MB, which is the decoder's [2048, 16384] SwiGLU projection in float16. MiniMaxH3VideoDecodeStep wraps vae.decode in torch.autocast(float16), and autocast caches every weight it casts for the lifetime of the region, so the float32 original and its float16 twin are both resident through the whole decode. Storing those weights as float16 up front makes the cast a no-op and removes both. This is not an approximation: x.to(float16).to(float16) is x.to(float16), and the four regression tests check that on a real matmul rather than on the reasoning. The audio VAE decode is not under autocast, so it keeps float32. t2va starts from noise and never encodes, so vae.encoder and vae.quant_conv go too. That part is gated on the workflow name rather than dropped unconditionally, because an image-conditioned workflow needs them. Measured 36.96 -> 28.37 GB peak with the encoder drop and the pre-cast, 28.27 with expandable_segments as well, over 5 prompts x 2 seeds. Speed is unchanged (-0.05% +-0.86% eager, -0.34% +-2.23% compiled), and every arm hashes identically to its control on latents, audio_latents, frames and audio. The estimator's base still reads 68.5 GB. That figure was measured on the bfloat16 modular components, not the int8 arm above, so it stays put until it is re-measured in the same configuration rather than adjusted by arithmetic. * Pad MiniMax-H3's small-M int8 linears instead of leaving them dense torch._int_mm asserts self.size(0) > 16. torchao's eager path never trips it (safe_int_mm falls back to a widened matmul), but inductor lowers the same quantized linear straight to _int_mm, so any quantized Linear invoked at a small activation row count crashes under torch.compile. Until now the fix was to leave those linears dense bf16, which on H3 meant excluding context_embedder and both token_refiner blocks: 13 linears, 798M parameters, 0.80 GB of weights the int8 checkpoint was not allowed to touch. Pad instead. diffusion_quant_pad.PadToMinM pads the flattened row count up to 32, runs the GEMM and slices the result back, so the module compiles with no change to the quantization config and the caller's rows come back bitwise unchanged. Verified bitwise on all 65 (module, M) cases across H3's 13 linears at M = 10, 13, 14, 17, 19, on real torchao-quantized weights; compiling those same modules unpadded raises the _int_mm assert at M = 10, 13 and 14. Two properties carry that exactness and both are asserted rather than assumed. The pad rows replicate row 0, not zeros: an all-zero row has amax 0, so the activation quantizer divides by zero. And the activation scale must be per row, so each kept row's scale comes from that row alone; a quantized Linear whose granularity cannot be proven per row raises instead of being quietly skipped, because a half-padded transformer compiles on the wrapped modules and crashes on the rest. Everything below pad_to normalises to pad_to rather than only what is below the floor, so one inductor graph covers every prompt length in the range. H3's seven eval prompts run at M = 10..19, which straddles the floor, and padding only to 17 would leave three shapes behind. The wrapper reparents the Linear, so it runs after quantize_ on the runtime path and after load_state_dict on the prequant one. The offline builder drives quantize_ directly and saves the state dict, so it never sees a wrapper; PadToMinM is also state-dict transparent as a second line of defence, saving and loading under its own prefix so a wrapped transformer still writes context_embedder.weight. Scoped to minimax-h3. qwen-image, qwen-image-edit and hunyuanvideo-1.5 have the same small-M shape but published int8 prequant checkpoints whose metadata bakes the current exclusion set, and _validate_checkpoint compares that set against exclude_tokens_for_scheme, so they move only together with a rebuild. adaln_proj stays excluded for a different reason: on the dense checkpoint it is Linear(2688 -> 96768) and runs at M = 1, while on the pruned form it is Linear(8 -> 96768) and falls under min_features anyway (verified: the filter rejects all 51 for min_features). Measured on B200, 640x384 x 124 frames, 4 steps, 7 prompts x 2 seeds, the two arms alternated within each cell so drift on a shared box cancels: checkpoint 21.052 -> 20.254 GB (-0.798, -3.8%) transformer 21.051 -> 20.253 GB (-0.798) load peak 21.137 -> 20.336 GB (-0.801) render peak 37.766 -> 36.966 GB (-0.800, -2.1%) step time +0.0597 s +-0.0057 eager, +0.0533 s +-0.0086 compiled The time is torchao's un-fused eager activation quantization on 13 modules that run ONCE per render, so it is a fixed cost rather than a per-step one, and it does not scale with steps or resolution. Compiling those modules alongside the blocks removes it: +0.0034 s +-0.0040, no detectable difference. Quality is unchanged as far as n = 14 can resolve: against the same bf16 twin the padded arm sits +0.0070 +-0.0151 LPIPS from the excluded one, which rules out a degradation larger than 0.022. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Classify the MiniMax-H3 GGUF bundle as video in the cached inventory The H3 GGUFs are stable-diffusion.cpp conversions and carry no metadata keys at all (kv_count 0), so general.architecture is absent where the LTX-2 and Wan video GGUFs declare ltxv/wan. _arch_to_task therefore left the downloaded repo with no task, which drops it from the Video picker's On Device list and hands it to chat as a plain GGUF. Key the two bundle repo ids before the arch is consulted. * Release the VIDEO GPU claim when H3 native falls back to the CPU build On a CUDA/ROCm host /video/load acquires the VIDEO arbiter owner because the resolved device target is not CPU. _run_load_h3_native then asks for an accelerator-matched sd-cli, and the pinned prebuilt release publishes no Linux CUDA/ROCm archive, so ensure_sd_cpp_binary returns None and the load commits the CPU build with native_device = cpu. Nothing dropped the VIDEO claim, so the next chat or image acquire evicted and unloaded an H3 runtime that holds no VRAM. Release the claim once the CPU fallback is committed, through release_if so the token check is atomic against a newer load that already took ownership. Mirrors the CPU-only native release /images/load already does. * Video: protect the native H3 companion repos, cancel the modular denoise, forward the hub token - VideoBackend.loaded_repo_ids() publishes the repos the committed native H3 runtime re-reads every generation (Qwen encoder + both VAEs), and the delete-cached guard consults it, so deleting a companion under a loaded model is refused. - The H3 modular workflow no longer falls back to a null progress context: the denoise loop drives pipe.scheduler.step once per step, so the existing wrapper gives it the same per-step progress and cancellation the other callback-less pipeline gets. - load_components() gets the Settings token, so gated/private component loads are not issued anonymously. * Reject stable-diffusion.cpp builds that predate MiniMax-H3 support ensure_sd_cpp_binary hands back whatever find_sd_cpp_binary locates and only probes that it runs, so an install upgraded from an older Studio kept serving its pre-H3 managed sd-cli. The H3 load's only gate is SdCppEngine.version(), which that binary passes, so the load reported ready and the failure surfaced on the first generation, after the whole bundle had downloaded. Gate the H3 path alone on the capability instead of a version string: upstream added --ref-video and the other H3-only options in the same commit that added MiniMax-H3 (leejet/stable-diffusion.cpp#1854, master-812-ea7f0c8), and the release prebuilts report 'version unknown, commit unknown' because they are built without a .git directory, so --help is the only usable signal. Image generation keeps accepting any user-supplied build. A stale copy under the installer-owned root is removed so the pinned prebuilt reinstalls; a build the user supplied is left in place and the load fails naming it, the same ownership split _usable_or_discard_managed makes. A --help that cannot be read means 'cannot tell', never 'no H3'. * Distinguish a reused CPU sd-cli from an accelerator build on an H3 load On a Linux CUDA host the first H3 load installs the CPU prebuilt through the fallback and correctly commits native_device = cpu, because the pinned tag publishes no Linux CUDA, ROCm or Vulkan asset. Every later load then calls ensure_sd_cpp_binary(accelerator = cuda), which finds that same CPU binary and returns it without looking at what it was built for, so the fallback was skipped, native_device stayed cuda, and Studio applied GPU offload policy and retained the VIDEO gpu_arbiter claim while sd-cli ran wholly on the CPU. A later chat or image acquire then evicted an unrelated GPU model. Fall back on what the binary offers rather than on whether one was returned: sd-cli --list-devices prints one name/description line per available ggml backend device, so a CPU-only build answers with CPU alone. The second load now reaches the same cpu conclusion as the first, which is what lets the existing release_if drop the stale claim. Unreadable output, or an older build that rejects the flag, keeps the GPU: neither says the accelerator is missing. * Guard the H3 companion repos while a native video load is downloading _run_load_h3_native downloads from repo_id, the H3 GGUF companion and the H3 component repo, but the in-flight state only carried repo_id and base_repo. The cached-model delete guard reads loading_repo_ids(), so it allowed deleting Comfy-Org/MiniMax-H3, and the GGUF companion when the load comes from another mirror or a local file, while those files were still downloading, which fails the load. Carry the companions on the loading state, the way the image backend's _SdLoading already does, and publish them from loading_repo_ids(). This is the in-flight twin of loaded_repo_ids() and covers the same repos. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep curated Recommended rows searchable and give them their metadata Two things went wrong with the curated video models the Recommended list paints from the catalog rather than from a live Hub listing. Search dropped them. The Recommended search matched the query against `recommendedIds`, which filters out every id already on disk because a downloaded model gets its own On Device row. The unfiltered Recommended list does not filter that way: it renders the curated seeds and badges the downloaded ones. So a curated model was visible in the list and unfindable by typing its name the moment it was downloaded, and only a live listing row could bring it back, which a repo the listing does not return never gets. `searchableRecommendedIds` unions the seed ids with the listing ids, seeds first, deduped case insensitively (the HF cache lowercases repo ids), so both lists agree on what exists. Rows rendered bare. Everything past the id came from the listing alone, so a curated row the listing never returns showed no parameter chip and no capability glyph while its neighbours showed both. The catalog now carries the two facts nothing else can supply: `totalParams` per artifact and `capabilities` per group, read through `curatedTotalParamsFor` and `curatedCapabilitiesFor`. Both are fallbacks only. A listing row wins wherever there is one, because real tags and a Hub-reported total outrank anything hand written here. The parameter counts are measured, not guessed. The MiniMax-H3 figure is the sum of the tensor shapes in the BF16 GGUF of the FL2VA denoiser that repo publishes; the LTX-2.3 figure is what the Hub reports for its repo, carried so the row looks the same offline or rate limited. One more inconsistency fell out of the same place: `searchRowFits` hides anything it cannot size (`requireKnown`), and it could not size a curated repo with no listing row and no "<n>B" token in its id, so turning on "Fits on device" hid from search a row the unfiltered list still showed. It now falls back to the curated total the same way it already fell back to the curated size. Covered by studio/frontend/tests/recommended-curated-row-metadata.test.ts: 15 assertions over the search pool, the two catalog lookups, the fit check, and the four picker call sites that read them. * Load MiniMax-H3 from a hosted pre-quantized denoiser Video families had no way to reach a hosted pre-quantized DENOISER. VideoFamily carried gguf_repo and te_prequant_repos (the text encoder) only, and nothing in video.py consulted a denoiser table, so a MiniMax-H3 load either pulled the full 66.3 GB bfloat16 DiT or nothing at all. The image side already solves this with DiffusionFamily.prequant_repos plus family_prequant_repo(), so this follows that shape rather than inventing a second one. Four parts. VideoFamily gains prequant_repos, prequant_variant_repos and prequant_subfolder, resolved by video_family_prequant_repo() / video_family_prequant_schemes(), mirroring the image resolver. The registries stay separate, as the module header requires, so the base-id normaliser is local rather than imported. The shared resolver learns an optional subfolder. The hosted video checkpoints nest theirs one level down instead of keeping it at the repo root, and the prefix has to reach BOTH candidate names or the primary 404 is followed by a second one and the load silently falls back to the dense download. Always a literal forward slash: these are Hub repo paths, and a Windows join would miss the cache. The cache and download plumbing already handled a nested name, so only the filename builder changed, and every existing call site is byte-identical. The modular workflow builds its denoiser through its own component loader, so there is no dense module to quantise in place. A hosted checkpoint is therefore the only way to run that transformer quantized, and pre-seeding it with update_components() before load_components() is also what stops the dense download: load_components(names=None) skips a component whose attribute is already set. Passing names= instead would have forfeited the workflow's own block pruning, which is what already avoids the 61.7 GB Ref2VA transformer. Those checkpoints carry the pruned adaLN, where the modulation is a rank-8 affine factorization of the time-embedding curve plus a shared table, and roughly 40% of the released model's parameters go. Against the base repo's dense config the model is four keys short, one over and fifty-one shapes wrong, so the strict load fails and the checkpoint is unloadable by every route. video_minimax_h3_adaln.py reshapes the model between from_config and load_state_dict: table lookup with interpolation instead of the timestep MLP, no SiLU (the table already holds the activation's own output projected onto the basis), and the modulation cast to the block stack's dtype, without which the first quantized matmul dies on mismatched dtypes. Bound per instance, so a dense load in the same process is untouched. Finally the refusal becomes honest. A single-file pick on a modular family used to reach the loader only after ~98.7 GB had downloaded AND after the resident pipeline had been evicted to make room for it, because download-plan returned 200 and validation passed. Both refusals now run in validate_load_request, ahead of the diffusers availability probe so they still fire where diffusers cannot be imported, and each names what to pick instead. download-plan forwards transformer_quant to validation and to the plan, without which the quant-keyed refusal never fires on the route that stages the download and the plan stages shards the load never opens. * Point both MiniMax-H3 schemes at one hosted pre-quantized repo The two hosted pre-quantized denoisers were split across two repos with the checkpoint nested one level down, so reaching them needed a mechanism the image side has never had: a VideoFamily.prequant_subfolder field, a prequant_subfolder_prefix() normaliser, and a subfolder keyword threaded through both resolve_prequant_source() and usable_prequant_source(). Both schemes now live in ONE hosted repo, at the root, named <Model>-<SCHEME>.pt. That is the layout every image-side prequant repo already uses, and it is exactly what prequant_repo_filename() builds unaided, so the whole mechanism goes. Match an existing convention and the code should shrink, not grow: -22 lines in diffusion_prequant.py, -6 in video_families.py, -6 in video.py, with no new concept to carry. Landing on the primary name also fixes a memory-planning under-credit. cached_checkpoint_path() deliberately credits only the PRIMARY filename, so that a cached legacy artifact cannot pin a stale name once a repo ships the real one. While these checkpoints were published as transformer_<scheme>.pt the primary never existed: every hit came through fallback_filename, and planning therefore read an already-cached checkpoint as "this would have to download" and handed the pick to GGUF. The primary is now the published name, so the probe hits it. fallback_filename stays. It still covers repos that have not been renamed, and dropping it is a separate decision from this one. Tests: the subfolder-prefix cases are replaced by the naming they now guard -- both schemes resolving to one repo, the primary resolving to a root-level <Model>-<SCHEME>.pt with no directory component on any platform, the cache probe being asked for that primary name, and the repo's own scheme suffix being stripped and replaced rather than carried through. Five mutations run, each caught by the named test and reverted: two repos again (M1), the primary nested under prequant/ again (M2), the suffix strip removed (M3), the cache probe keyed on the fallback (M4), the fallback name dropped (M5). * Add MiniMax-H3 image and reference video conditioning * Improve H3 finalization progress * Report real sd.cpp progress on the Video page instead of a frozen 0 of 30 A native (GGUF) video generation reported phase "denoise", step 0/30 for its entire run and then flipped straight to "completed". Two separate things were wrong, and the progress endpoint could not move until both were fixed. The parser looked for r"(?:step|sampling)\D+(\d+)/(\d+)". sd-cli's sampling bar contains neither word. It prints |=========> | 7/30 - 21.50s/it so nothing ever matched. Anchor the pattern on the bar and on the trailing speed unit instead. A bare "n/m" is deliberately not enough: an unrelated ratio in some other log line must not drive the progress bar. The reader also delivered every redraw one step late. sd-cli redraws in place, and its carriage return LEADS the record while the newline only arrives on the final step, so a reader keyed on CR/LF cannot produce step 1 until step 2 has been flushed. Treat the erase-to-end-of-line that closes each redraw as a terminator too, and read the pipe with buffer.read1 so a record that carries no newline is not stuck behind a blocking readline. Escapes are stripped before a record reaches on_log or the error tail. Streams without a raw .buffer keep the old line iteration. The same bar shape is printed by three different things, so the parser now tells them apart rather than reporting whichever came last. Weight load prints it with a byte rate, and tiled VAE decode prints an identical s/it bar counting TILES: without a guard a run finished sampling at 30/30 and then jumped backwards to "step 1/16". Load and decode are real work with no sampling step, so they report their own phase and a null step rather than a fake 0 of 30. ETA is measured from the first step, not from job start, so the one-off weight load is not charged to every remaining step. Verified end to end against a real CUDA sd-cli MiniMax-H3 generation: the step advances 1..6 over 6.2s..9.7s of wall clock, load and decode are reported as themselves, and the ETA tracks. * Read sd-cli's in-place progress redraws so the Video bar moves during sampling The native H3 progress bar had two independent causes and fixing either alone changed nothing observable. The bar pattern is now correct, but the reader still was not. sd-cli redraws its sampling bar in place: one printf per step shaped "\r<bar> <n>/<total> - <speed>\033[K", with a newline only on the final step of a phase. The drain loop did `for raw in proc.stdout`, which terminates on LF only, so every redraw sat in the buffer until the next one's carriage return arrived and the last one until sampling was already over. The Video page saw nothing. Split the raw pipe into records on CR, LF, or the trailing erase-to-end-of-line, reading through buffer.read1 with an incremental UTF-8 decoder so a multi-byte character straddling two reads survives, and strip the CSI escapes before the record reaches on_log or the error tail. Streams without a raw .buffer (test doubles, non-pipes) fall back to line iteration. The new backend test drives the real byte stream through both halves, one flush per read, and asserts each step is visible on the read that carried it rather than one redraw later. * Give MiniMax-H3 first and last frame conditioning in the video backend MiniMax-H3's released transformer is the FL2VA one: text-to-video is the same checkpoint run with no keyframes. Studio only ever ran it text-only, so the Video generate request had nowhere to attach a reference frame. The load used to prune the block graph to t2va. That argument prunes STATICALLY, so an fl2va-pruned pipeline runs the keyframe blocks on every request and cannot serve a text-only one at all: it raises packing an empty conditioning list. The load now keeps the whole auto graph, which selects per request, and bounds load_components to the keyframe workflow's component set instead, so the 61.7 GB Ref2VA partition is no more loaded than before. Measured against the released checkpoint: a text-only request through this pipeline is bit-identical, video and audio, to the same request through a t2va-pruned one. A keyframe is a geometry anchor, not just conditioning, so the canvas comes from its aspect ratio through the released arithmetic (768 short edge, area capped at 768x1344, both axes rounded to 32) rather than from whatever resolution preset was selected. An arbitrary size produces a garbled clip rather than an error. sd.cpp already implements the same conditioning, so that side is the existing --init-img / --end-img flags with the frames staged as PNGs. Only MiniMax-H3 declares the capability, and status reports it, so Wan and LTX do not grow a control that does nothing. * Cover the MiniMax-H3 keyframe path with tests Registry (which families declare it, and the canvas a keyframe resolves to), the load wiring (whole block graph, component set still bounded, VAE encoder kept), request handling (decode, refusal, canvas override, what reaches the pipeline call) and the sd-cli argv. * Add the reference-frame controls to the Video page First and last frame pickers, shown only for a family whose status declares keyframe conditioning, so Wan and LTX are unchanged. The Images page's source picker moves to a shared component rather than the Video page growing a second upload path; both send the same data URL to the same backend decoder. While a frame is attached the Resolution preset is disabled and says why: the frame's aspect ratio resolves the clip's size, the way the model itself does. The gallery recipe records which ends were pinned. * Check the keyframe canvas against the pipeline's own resolver The canvas rule is a checkpoint contract, so pin it to the released implementation rather than only to hand-written expectations. Skipped where diffusers does not ship MiniMax-H3, which is most runners. * Revert the standalone H3 keyframe implementation oobabooga/unsloth#121 covers first frame, last frame, first-and-last, Ref2VA and the canvas rule, and it reached the same load construction independently. Two implementations of the same feature on one branch is worse than either, so this takes mine back out and leaves the branch ready for that work to land whole. The one finding worth keeping from it is already reflected there: passing workflow= to ModularPipeline.from_pretrained prunes the block graph statically, so the pipeline must be built unpruned and only load_components bounded. * Keep the pre-quantized MiniMax-H3 denoiser resident so a generation can run Loading H3 with a hosted pre-quantized denoiser worked, but every generation died on its first denoise step: Attempted to set the storage of a tensor on device "cuda:0" to a storage on different device "cpu". This is no longer allowed; the devices must match. ComponentsManager.enable_auto_cpu_offload parks every component on the CPU and moves each one onto the accelerator inside its own pre_forward, that is from within the block that is already executing. The text encoder and the VAEs survive that; a torchao-quantized denoiser does not, because the device change reaches return_and_correct_aliasing, which tries to alias a CPU storage to an accelerator tensor. Moving the same module at load time, outside any executing block, works. So place it once at load and take it out of the offload rotation: drop its hook, unlist it from the other components' eviction candidates, and move it. Everything else is unchanged, and the encoder and VAEs still offload around it. Keeping it resident is what asking for a quantized denoiser buys in the first place: the hosted checkpoint is about 20 GB against 66.3 GB dense. Verified end to end on a B200: MiniMaxAI/MiniMax-H3 loaded with the hosted fp8 denoiser, then a 1280x768, 124-frame clip generated from a start frame in 167s. The clip's first frame matches the supplied image and the motion is coherent. * Apply the pinned Diffusers revision on a fresh install, not just an update MiniMax-H3 needs a Diffusers revision newer than any published release, and Studio refuses to load it otherwise. The pin was in studio/backend/requirements/base.txt, and a clean install.sh run still ended up on diffusers 0.39.0 from PyPI, every time, with nothing in the log to say so. base.txt is never installed by install.sh. install.sh installs unsloth itself, whose own metadata pulls a diffusers release in transitively, and then runs install_python_stack.py with SKIP_STUDIO_BASE=1, where the base-packages step is a bare `pass`. So the pin applied on `unsloth studio update` and on the no-torch path (install.sh installs no-torch-runtime.txt directly) and was dead on exactly the path a new user takes. Reproduced on a clean install into a throwaway prefix before and after: 0.39.0, then 0.40.0.dev0 with MiniMaxH3Transformer3DModel present. The revision now lives in its own diffusers-pin.txt, installed by a step that sits outside every skip_base / NO_TORCH branch and after every other requirements file, so nothing left in the run can re-resolve diffusers back to a release behind it. No forced reinstall is needed: a direct URL requirement is not satisfied by a resident registry install, so the step is a no-op once the environment is already on the pin. tests/studio/install/test_diffusers_pin.py holds the shape in place: exactly one requirements file may name diffusers, the pin must be a full commit sha rather than a moving ref, the install step must sit at function top level rather than under a conditional, and it must come after every other requirements install. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: let the Video and Images pickers see a local diffusers pipeline A model already on disk never reached the pickers' On Device list unless it happened to keep a weight file beside a root config.json. Every image and video model downloaded as a pipeline keeps its weights in component subdirs under a root model_index.json instead, so the hub inventory scan behind /api/hub/local rejected it. Pointing Custom Folders at one was worse than empty: the LM Studio publisher walk descended into the pipeline and offered vae, transformer, text_encoder and audio_vae as four separate models, none of them loadable. Teach the hub scanner the same pipeline-root test routes/models.py already applies, in the three places that only accepted a root config plus loose weights, and keep a pipeline row through the custom-folder format filter: the layout has no loose weight to classify, so the row is "unknown" by construction rather than by fault. * Cover the local diffusers pipeline scan with tests * Pin what the pipeline exemption must not let through Three gaps in the cover added with the scan change, each found by mutating the fix and watching the suite stay green. The custom-folder format filter now waves a row through on its shape, and nothing said what it still has to reject: replacing the whole predicate with True passed. A folder holding a config.json and no weights, which an aborted download leaves behind, reports the same "unknown" format and no loader can start it, so it pins the boundary. The predicate is applied to every row the filter did not already accept, and a row's path can be a GGUF file rather than a directory. A missing path, a file, and a directory whose model_index.json is itself a directory must answer False rather than raise, because an exception there fails the scan and empties the picker. The publisher walk was only covered one level up. Adding the model folder itself as a scan folder is the obvious thing to do and used to publish vae, transformer and text_encoder as three models. * Drop the unused H3_TASK_KEYFRAMES import from the video backend video.py only branches on H3_TASK_REFERENCES; the keyframe constant is read from video_minimax_h3 directly by the tests that need it. The hoisted-import safety net in Source lint flags the unused name as a blocker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage the hosted pre-quantized H3 denoiser in the video download plan The plan already drops the dense transformer shards whenever a hosted pre-quantized checkpoint covers them, but nothing put that checkpoint back: an int8 or fp8 H3 stage skipped 66.3 GB of base shards and added none of the 20.25 GB artifact the load actually opens. The byte total under-reported the stage by the size of the checkpoint, the disk preflight cleared a volume that could not hold it, and an offline stage completed without the one file the load needs. _denoiser_prequant_hub_files mirrors the pre-cast encoder helper: it resolves the family's hosted checkpoint, confirms the file really exists on the Hub, and prefers the repo-root name over the legacy scheme name in the same order the load tries them. An unreachable repo is logged and yields no files, so a gated or renamed artifact keeps the dense shards instead of sinking the plan. The progress-bar estimate is deliberately left alone: it counts cached bytes for the checkpoint and base repos only, so adding a third repo there would leave the bar permanently short of 100 percent. * List both H3 denoiser partitions in the picker, not only FL2VA The bundle-repo filter accepted only minimax_h3_fl2va*, so every published minimax_h3_ref2va* quant was hidden from the variant listing. The loader disagrees: validate_h3_transformer_filename accepts either partition, on the grounds that which one is picked IS the task, and h3_transformer_task routes Ref2VA to the reference-video workflow this PR adds. The community bundle repo publishes three Ref2VA quants today, so the reference path was unreachable from the remote catalog. Accept both prefixes from one shared tuple and keep excluding the Qwen3-VL encoder and VAE companions, which are never picks for either partition. The filter test asserted the old behaviour and is updated with it. * Bound H3 reference-audio decoding to the trained window The reference-video decoder already selects, resizes and refuses incrementally because the encoded size says nothing about the decoded size. The audio decoder did not: it appended every resampled block to a list and then allocated a second full buffer in np.concatenate, with no duration or sample cap. The route accepts 32 MiB of encoded audio, which is over half an hour of compressed stereo. That lands as roughly 1.9 GB of float32 and doubles again in the concatenate, and up to three references are accepted per request, so an ordinary long music or podcast file picked by mistake could exhaust the host before the background job even started. H3's reference window is 15 seconds, so anything past it is unusable rather than merely large. Refuse it while decoding, with the same shape of message the video guard uses, instead of decoding it first. * Refuse a quantized H3 reference load instead of seeding the keyframe denoiser The hosted pre-quantized checkpoints are FL2VA (keyframe) denoisers. Ref2VA shares their module shapes and the same base model, so resolve_prequant_source handed one back for a reference load, it passed every metadata check, and seeding it made load_components skip the real Ref2VA transformer. The request then generated from the wrong partition rather than failing, which is the worst of the three outcomes. The route accepts h3_task, so this was reachable from the public API even though the picker does not expose the choice yet. validate_load_request now refuses the pairing with a message naming the workable alternatives, in the same place it already refuses a scheme with no hosted checkpoint, and the modular loader drops to the released components if a direct call reaches it. Nothing changes for keyframe loads, which are what the checkpoints are. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Skip modular-workflow families in the dense text-encoder plan assertion The plan-unchanged sweep from main walks every family and asserts the dense budget plan_diffusion_memory received. MiniMax-H3 is the first modular-workflow family to reach that list, and load_pipeline dispatches to the workflow's own loader before the planner runs: each component is built by its own from_pretrained, so there is no single dense pipeline to budget and no plan call to assert on. Skip it the way the sweep already skips wan2.2-t2v-a14b. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope the paired-axes canvas rule to keyframe requests A half-specified width/height is long-standing, documented API behaviour: the backend resolves the missing axis from the family's default preset, in both validate_video_request_shape and _resolve_keyframes. Applying the new paired-axes rule as an unconditional request validator rejected those calls with a 422 before family validation ever ran, breaking existing LTX, Wan, Hunyuan and prompt-only H3 clients. The rule still holds where it means something: with a keyframe present the canvas is matched to the source aspect whenever either axis is missing, so the axis the caller sent would be silently discarded. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the CUDA sd-cli pin and translate a mirror-only tag upstream The merge with main brings in the accelerator-aware installer, whose fallback translates a mirror-only -u<id> pin back to the upstream release it was built from instead of skipping the upstream attempt. That is strictly better: skipping kept the round trip cheap but dropped the pin entirely on every host the mirror does not build, leaving them on upstream latest. test_a_mirror_only_pin_is_never_requested_upstream asserted the old shape, that the fallback settled for upstream latest. It now pins the new one: never the literal -u<id> string, the translated release instead, and no latest attempt at all because the translated pin succeeds. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-08-09 10:05:58 -07:00
Michael Han	b3705f44b9	Studio: add a loaded models indicator with per-model eject (#8082) ... * Studio: add a loaded models indicator with per-model eject A loaded model is only visible on the page that loaded it. Leave Chat for the Hub or Train and nothing on screen says a model is still holding memory, so the only way to check or release it is to navigate back. Adds a compact card in the bottom-right listing everything resident, with an eject button per row. Chat, images, video and dictation each own a runtime and their own /status, so all four are read and merged. A TTS load is split out from a chat load by audio_type: same picker, but it does not answer prompts. It joins the existing bottom-right stack in provider.tsx alongside the update banners and the download panel, so nothing overlaps. Collapses to a pill, and that choice persists. On by default, with a toggle in Settings, General, Notifications. Reads fail soft per runtime, so a chat-only host missing /video keeps its other rows. Ejecting the active chat model reuses the API monitor's read, unload, re-read sequence, since an API auto-switch can replace the model mid-eject. * Studio: address review on the loaded models indicator Eject the row's own model, not whatever is active. An API auto-switch can land between the poll and the click, and re-reading unconditionally would free a model nobody asked to free while reporting the clicked one as ejected. The sequence moves to eject-chat-model.ts, taking its I/O as deps like api-monitor/unload-resident.ts, so the scoping is under test. Fall back to the top-level STT fields. Servers predating the per-engine split report the resident Transformers model only there, so those installs showed no dictation row and had no eject for it. Bound each status read. authFetch has no timeout, so a runtime that accepts the connection and never answers left the Promise.all pending forever, and with it the in-flight guard that gates every later refresh. Re-sync Images and Video after an eject. Both hold their own status and re-read it on tab activation, not on a timer, so ejecting from the indicator left their controls offering to generate on a freed runtime. Add both preference keys to "Reset all local preferences", which otherwise left the indicator hidden or collapsed after a reset. * Studio: verify the row before the identity-less unloads /images/unload, /video/unload and the STT unload carry no model id and release whatever their runtime holds, so acting on a row up to one poll old could free a model nobody clicked while reporting the clicked one as ejected. The image and video races need a load landing in that window; dictation loads on demand and releases when idle, so its engine can change with no user action at all. Each now re-reads its runtime and only unloads when the resident model still matches the row. This narrows the window to the round trip rather than closing it, which would take a backend that accepts the model id. Ejecting reports an outcome instead of a bare string, so a stale row says the model is already gone and names what took its place, rather than claiming a success it did not perform. An unreadable dictation status is now an error rather than a false success. * Studio: stop dictation rows printing their engine twice The llama.cpp and whisper.cpp STT sidecars report their engine name as the device (stt_mtmd_sidecar.device is "llama.cpp", stt_ggml_sidecar.device is "whisper.cpp"), so a row read "Dictation - llama.cpp - llama.cpp". Only the Transformers sidecar reports a real device. joinDetail now drops repeated parts, so the engine is named once and a genuine device still shows beside it. * Studio: make the loaded models card movable and keep it off the Live monitor The card and the Live monitor both anchor to the bottom-right, so opening the monitor buried the card underneath it. The monitor is draggable and resizable, so a fixed offset would only hold until it moved. The monitor now publishes its box. The corner stack lifts above it, but only while it is in the stack's column and low enough to be in the way, so a monitor dragged elsewhere leaves the corner alone. This clears the update banners and the download panel too, which had the same overlap. The card also gets a drag handle: anchored to its corner until moved, then kept where it was left, clamped back into view if the window shrinks. Its position persists and joins the reset list. * Studio: use the AI brain icon and lead the Notifications section The loaded models card and the Train wizard's Model step both take AiBrain01Icon, and the indicator toggle moves above the llama.cpp one so the Notifications section leads with it. * Studio: let the collapsed indicator be dragged too The drag handle only existed in the expanded header, so the pill could not be moved. The pill is now its own handle: a press becomes a drag past a few pixels, below that it stays the click that expands the card, and reading the flag clears it so a keyboard activation is not swallowed. Re-clamping now also runs on a ResizeObserver, so expanding a pill dragged to the bottom edge pulls the taller card back into view instead of growing off screen. * Studio: address review on the loaded models indicator Keep the Chat picker selection when an eject leaves the model resident, so a reload during the run no longer empties the picker while the model still runs. Republish the Live monitor's frame from a layout effect. ResizeObserver does not fire for a position-only change, so dragging the monitor left the overlay stack dodging its old corner. Cap the overlay stack's height to the space above its own bottom inset. Lifting it over the monitor without shortening it pushed the top of a long download list off screen. Measure the stack column from the update banners (448px), not the download panel (400px), so a monitor beside a banner is still dodged. Name the precision an image or video pipeline loaded at, so a quantised build is distinguishable from bf16 in the row. * Studio: make the loaded models card resizable The card is anchored bottom-right, where a native CSS resize grip has nowhere to grow, so it resizes from a grip at the leading corner instead: the anchored corner is held still and the box opens up and to the left. The grip shares the title icon's slot rather than adding another control to a small header, and double-clicking it returns the card to its default size and corner. The size persists next to the position, is clamped to a floor and to the room available, and is cleared by Reset all local preferences. * Studio: open a loaded model's page from its row Clicking a row now goes to where that model is used: chat models to Chat, image and video pipelines to their tabs, dictation to the Voice settings tab that drives the sidecars. The target follows the runtime holding the weights, not the kind, so a Whisper checkpoint in the chat slot goes to Chat rather than to dictation. Navigation carries no search params, so it only changes page: it does not start a new thread or reload anything. Dictation moves to the Audio page once that lands. * Studio: revert the resizable loaded models card Back to the fixed-size card. Reverts 96749d1f. Also read /api/inference/status directly instead of through the chat barrel, matching the dictation read alongside it. The server logs show the indicator's chat read never reaching the backend: across every session the status count sits at roughly half the images/video/dictation count, matching the chat page's own 10s poll rather than the indicator's 5s one, which is why a loaded text model showed no row while an image model did. * Studio: keep the loaded models card reachable, and stop the ejects that report work they did not do Two of these break the feature outright, the rest report success for an eject that never happened. The card can become permanently unreachable. use-drag-position captured panelRef.current once, in an effect keyed [position, reclamp]. The indicator returns null until the first poll produces a row, so that effect always ran while the ref was still null: no ResizeObserver was ever constructed, and the resize listener closed over null. Neither dependency changes when the node finally mounts, so it never re-ran, and nothing in production ever called clampToViewport except an active drag. Dragging the card on a 2560x1440 screen and reopening at 1440x900 left it at x=2502, off screen, taking its own drag handle and collapse button with it; resizing did not help. The only way back was devtools or Reset all local preferences. The node now arrives through state so the effect re-subscribes, and the stored position is clamped as it is read so it cannot paint off screen even for one frame. Locale parity fails. CI's first gate is i18n:check:strict and the two new keys were added to en.ts only, so all 11 overlays were short. Added everywhere. The chat eject reported ejects it never performed. When nothing matching was resident it POSTed /unload anyway and returned stillResident: null without re-reading. The backend answers 200 "unloaded" for a model it never held, so the user got a success toast for a no-op and their picker selection was cleared off the back of it. Reachable by an ordinary sequence: loading B evicts A, and the card shows A for up to one poll. It now reports the same replaced / already-gone outcomes the runtime rows use, and only names a model directly for a cached row, which is the one case a status read cannot see. The dictation eject could release another engine's model. /audio/stt/unload silently serves gguf from the transformers engine when whisper-server is absent, and the unload response body is a fixed {loaded_model: null}, so a 200 was not evidence this engine let go. It now re-reads and reports what the engine actually holds. No timeout on the eject path. READ_TIMEOUT_MS guarded only readLoadedModels; a runtime that accepted the connection and never answered left the row disabled with a spinner until reload. Same bound, raised rather than swallowed. An audio-input VLM was filed under Speech. The split was audio_type !== "whisper", but the backend's own is_audio_input_type names two input types, whisper and audio_vlm. A model that listens was labelled as one that speaks, which is the confusion this card exists to remove. A stale poll could restore an ejected row, and the trailing refresh could not repair it because the in-flight guard was still held by that same read. A generation counter retires reads issued before an eject, and a pending flag keeps the trailing refresh. The drag now captures the pointer, as the Live monitor's does. Without it a release over another window is never delivered and the card follows the cursor. Backend: the card polls four runtimes every 5s for as long as the app is open, and only /api/inference/status was in _QUIET_POLL_PATHS. Its three siblings are now too, or every desktop tauri.log carries three lines every five seconds. Tests tests/loaded-models-platform-matrix.test.ts walks the payloads [Windows, Linux, WSL, macOS] x [NVIDIA, AMD ROCm, Intel XPU, CPU-only, Apple] actually produce. None of it is visible from a CUDA box: ROCm reports itself as cuda, Apple reports mps, XPU reports xpu for images but cpu for dictation because the STT device probe never checks torch.xpu, and the sd.cpp engine omits model_kind and puts "gguf" in dtype. tests/loaded-models-backcompat.test.ts covers the desktop app running against an older wheel, which is a supported configuration: it ships its own frontend bundle and adopts a separately installed backend. /video/status and the STT mtmd block are both days old, so a 404 route, a missing engine block, the pre-split top-level fields and absent optional fields all have to degrade rather than blank the rows a working runtime returned. Also pins that fields a future backend adds are ignored. tests/loaded-models-drag-restore.test.ts guards the restore geometry and, by reading the source, the wiring it depends on. Four of its cases fail against the code before this commit. tests/studio/playwright_loaded_models_indicator.py runs 28 checks in Chromium, Firefox and WebKit, covering Chrome, Edge and the WebView2, WKWebView and WebKitGTK the desktop app embeds. It stubs the four status endpoints with page.route, so it needs no model, no GPU and no llama.cpp build, and can therefore replay the AMD, Apple and CPU-only payloads on any runner. The position restore is here rather than in the node suite because it needs a real layout and a real ResizeObserver. Wired into the Linux, macOS and Windows UI workflows alongside the existing permission runs. Against the code before this commit it reports 25/28, failing exactly on the off-screen card and the mislabelled audio-input model. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: read the indicator's rows in one call, so the eject checks stop racing them Staging CI on macos-14 caught this: every check up to the eject passed, then the run died on a 60s locator timeout. rows() read count() and then indexed nth(i). The eject checks poll it while the row is disappearing, which is the whole point of them, so the row could go between the two calls and nth(1) would block for the full locator timeout rather than reporting a shorter list. evaluate_all takes the snapshot in a single evaluation, so there is no window. card_text had the same count()-then-read shape; it is now bounded and returns empty rather than blocking when the card is mid-change. Test-only. Confirmed 28/28 in Chromium, Firefox and WebKit after the change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: finish the second review round on the indicator Run the media pages' full unload cleanup on an indicator eject, not just a status refresh. Images and Video kept lastLoad and canReapply, so Reapply still offered to reload the model that was just ejected, and the stale quantisation stayed on screen. Both now share that cleanup with their own Unload rather than duplicating half of it. Cap the overlay stack under a monitor parked high in its column. Avoidance was skipped entirely whenever the monitor's bottom sat above the midpoint, on the grounds that the corner was free, but the stack grows upwards from the bottom and a full download list plus the card reaches well past halfway. It is capped to the gap below such a monitor instead, since lifting over it would be wrong. stackBottomInset and stackMaxHeight now come from one stackGeometry call, so the two cannot disagree about where the monitor is. * Studio: stop the chat header claiming an evicted model is loaded Loading an image or video model evicts the chat model: the GPU arbiter allows one owner and logs 'evicting chat for diffusion'. Nothing on the chat page said so. modelLoaded was '!!checkpoint && !modelLoading', which reads the picker selection rather than what the backend holds, and the status poll's null-active_model branch reset three capability flags but left the checkpoint alone. So the header kept its green tick for a model that was gone and the next prompt came back as a bare 'Request failed (400)'. The store now tracks residentCheckpoint from /api/inference/status, published in applyActiveModelStatusToStore so the poll and the readopt path share it, and the header derives loaded from that. undefined until the first read, so startup does not flash 'not loaded'; an external model is unaffected, having nothing resident. An eviction also raises a toast naming the model, since it is otherwise silent and the card in the corner was the only thing reporting it. * Studio: fix the header tick for a model that was evicted The previous commit added residency tracking but changed the wrong modelLoaded: that one gates image attachments. The tick in the header comes from the model selector's own isLoaded, which was 'selected !== ""', so it still lit up for a chat model an image load had already evicted. The selector now takes an optional loaded prop, defaulting to the old rule so the Images and Video pickers are untouched, and the chat header passes residency through it. A test reads both the selector rule and the header call site out of the source, since the failure mode here was wiring, not logic. * Studio: show a loading model in the indicator, with the toast The load toast appeared at once and the card up to five seconds later, so a slow load looked like nothing was happening anywhere but the toast. The four load calls now announce themselves. model-eject-events becomes model-lifecycle-events, and withModelLoadNotice wraps loadModel, loadDiffusionModel, loadVideoModel and loadSttModel, so every caller is covered without each page remembering to and the announcement cannot drift from the request. The indicator adds a row on the start event and re-reads on the settle one, rather than waiting for its next tick. Loads are also read from the polls that already carry them: /status reports loading[] and the dictation status reports loading per engine. That covers a load started in another tab or before the page opened, which no event can. A loading row shows a spinner where the eject button goes, since there is nothing resident to release yet, and a status row for the same runtime always replaces the announced one. * Studio: let an audio VLM stay a chat row, and keep an exiting monitor from clearing the live frame Two review items from the latest round. An audio VLM is a chat model. is_audio means TTS in this codebase -- both ModelConfig constructors build it as `audio_type is not None and audio_type != "audio_vlm"`, llama.cpp keeps _is_audio False for csm/whisper/audio_vlm, and mlx_inference says so outright: "audio_vlm (omni audio input; is_audio stays False -- it means TTS and redirects in the chat route)". The frontend already agrees elsewhere: isMultimodalResponse counts audio_vlm as multimodal chat capability, which is what lets a user attach audio to a chat message. So the audio types split three ways, not two: whisper is the ASR sidecar, audio_vlm answers prompts and happens to listen, and the remaining four speak. The previous rule sorted every audio type as speaks-or-listens and filed Gemma 3n under Dictation. Nothing was user-visibly wrong, because is_audio is never true for audio_vlm and the row already fell through to Chat -- but the rule encoded a false invariant and the test pinned it as the contract, which is the part worth fixing. Reopening the Live monitor during its exit animation blanked the frame. AnimatePresence defaults to sync mode, so the replacement panel mounts while the old one is still animating out, and the old one unmounts last. Its cleanup called setFrame(null) unconditionally, wiping the frame the replacement had just published. Nothing republishes afterwards: the reconcile effect is ResizeObserver-driven and the republish effect keys on layout, so a monitor left sitting still never writes again. The overlay stack then dropped back to the bottom-right corner and covered the open monitor, including its close button. The store now records which panel published the frame, and clearFrame is a no-op unless that panel still owns it. tests/monitor-frame-ownership.test.ts covers the reopen-during-exit case, a lone close still clearing, the replacement clearing its own frame afterwards, and that no-op writes still do not notify (the overlay stack re-renders on every notification). Six of its seven cases fail without the change. * Studio: keep the images and video loading rows up for the whole background load Both loads only start the work: begin_load hands off to a daemon thread and the POST returns at once, while /images/status and /video/status keep reporting loaded: false until it finishes. withModelLoadNotice settled on the POST, so the indicator row appeared and vanished within one round trip while the page toast went on saying "loading" for minutes, which is the exact gap the notice exists to close. withBackgroundLoadNotice settles from the same load-progress endpoint the toast watches instead. The poll belongs to the load call rather than to a page, so the row still settles when the user navigates away mid-load; a load that never started settles from the POST, so exactly one of the two paths ends the notice. An unreadable progress read is not treated as terminal, since a restarting backend would otherwise drop a live row, and an hour-long deadline covers a backend that never answers again. * Studio: say what the eject read bound does and does not cover * Studio: bound each background progress read, and drop pending loads on disable Two follow-ups on the loading rows. The hour-long deadline only bounded the settle loop if every turn of it ended. A load-progress request that accepted the connection and never answered parked the await forever, so the deadline was never tested again and the row stayed loading for the life of the tab. Each read now runs under its own AbortController timer, and both progress endpoints take a signal so the request is really dropped, not just ignored. A plain controller rather than AbortSignal.timeout, which the older WebKitGTK builds Tauri embeds do not have. Nothing listens for the terminal event while the indicator is disabled, so a load in flight when the pref went off left a pending entry that came back on re-enable as a row no poll could retire: withPendingLoads only yields to a status row for the same runtime, and a failed or since-unloaded load has none. The pending map is now emptied on the disable transition, adjusted during render rather than from an effect so the stale rows never reach the DOM. * Studio: re-read the chat status when another runtime loads The header kept its tick on an evicted model because nothing ever told the chat page the model had gone. syncInferenceStatusToStore, the only thing that updates residency, runs on mount and from refreshModelLists when the model lists change. It is not on a timer. So an image load evicting the chat model was never observed here: residentCheckpoint stayed undefined, which reads as loaded, and the previous two commits could not take effect. Confirmed against the running server rather than by reading: /api/inference/status reports active_model null while the header still showed the model ticked, and /api/models/list, which only that sync calls, was absent from the log for the whole period after the eviction. The chat runtime now re-reads whenever another runtime finishes a load, using the lifecycle event added for the indicator. Loras are skipped: only residency is in question. * Studio: settle a cancelled load, and show a replacement next to what it replaces Two faults in the loading rows, both in the window where a load is in flight. A cancelled or evicted load leaves load-progress reporting null: all three engines return that only for "nothing loading and nothing loaded", and unload bumps the load token and drops the loading marker, so the phase never changes again. Treating null as non-terminal was wrong -- begin_load records its loading state before the POST answers, so a read before the first byte moves already says downloading, and null after that means the load ended with nothing to show. Ejecting a model whose replacement was still downloading therefore replaced the row with a spinner that had no eject on it and stood for the full hour deadline. The Images page's own poll has always treated null as terminal, "else this loop spins forever". An unreadable read stays distinct from it, as undefined. The other half is why that eject was offered at all. Images and video keep the old pipeline resident while the replacement downloads, freeing it only at the commit, so status reports the old model for the whole pull. Suppressing the announcement on the runtime alone hid the incoming model for minutes, showing only the one being replaced. A status row now wins only when it describes the same load; a row that is itself loading still wins, which is what keeps chat and dictation from announcing a load twice when the backend spells the name differently. * Studio: retire a settled load row only once a read replaces it The row vanished at random. On the settle event the optimistic row was dropped immediately and a read was asked for, but that read is a round trip away and can be queued behind one already in flight. For that gap the card had one row fewer, and nothing at all when it was the only one, so it blinked out and came back a moment later. The row is now retired by the read that supersedes it rather than ahead of it, so the card hands over from the announced row to the status row with no gap either way. A read that fails still retires it, from the same finally, so a row cannot be left spinning. * Studio: initialise the loaded models preference before the indicator There is a real cycle here: the barrel evaluates the indicator, the indicator imports the settings barrel, that eagerly evaluates SettingsDialog and so general-tab, and general-tab's reset list dereferences LOADED_MODELS_PREFERENCE_KEYS back through the barrel it is still inside. It happens to work today only because app.tsx reaches the settings barrel first, which finishes the preference module before general-tab's body runs. Entering the loaded models barrel first is enough to break it: under Vite dev, which serves native ESM and so has no bundler to reorder the graph, that throws "Cannot access 'LOADED_MODELS_PREFERENCE_KEYS' before initialization" and the app never starts. Exporting the preference module first evaluates the constant before the indicator is touched, so either entry order is safe. Pinned by a test, since a bundler can hide this. * Studio: gate the picker and hub "Loaded" marks on residency too The header tick was one of three places claiming a model was loaded, and all three read the picker selection. An image or video load evicts the chat model, which leaves the selection alone, so the dropdown's green "Loaded" badge and the Model hub cards went on marking a model the backend had already released. Both now use the same rule as the header: resident, not merely picked. An unread status stays as it was, so nothing flashes on startup, and an external selection is untouched since it holds no local weights. * Studio: drag both floating panels without a render per frame Dragging the loaded models card or the Live monitor was heavy. Three things, all per pointermove: each move set React state, so a trackpad reporting faster than the display refreshed rendered frames nobody saw; the move was written to left/top, which is the layout path, and both panels are expensive to lay out again (a wide blurred shadow on one, a backdrop filter on the other); and the card's reclamp effect was keyed on the position it was updating, so every frame disconnected its ResizeObserver and built a new one, each rebuild forcing a synchronous layout. A drag now records the offset, asks for one animation frame, and moves the node with a translate3d, so it costs no render at all. Release hands the offset back to left/top, written to the node as well as to state so it cannot flash at the spot it started from, and folds in a frame still owed rather than landing behind a flick. The monitor also published its box to the shared frame store on every frame, re-rendering each overlay subscribed to it. That now settles on release, so the stack moves aside once instead of chasing the drag. * Studio: drop the chat pick when the server evicts the model Loading an image model evicts the chat one, and the picker went on naming it. The last few rounds only dimmed the marks that said "loaded", which was the wrong end of it: the name on its own reads as "this is my model", and sending to it comes back a bare 400. An eviction now clears the pick, so the picker drops to "Select model", which is already what a server-side unload does. Guarded on the model having been known resident and on no load being in flight, since a load also reports no active model while it runs and must not wipe the pick the user just made. * Studio: time the load row's giving-up from the last healthy read, not the start An absolute hour deadline retired the row on a download that was still visibly progressing: a large video checkpoint on a slow link legitimately runs longer than that, and every progress read still said downloading. The window now measures a stall, restarting on each healthy read, so only a sustained run of unreadable polls gives up and a load that keeps reporting progress is never abandoned. Also drop the drag sentinel when the handle that started the drag has no click to consume it. The expanded card's grip is not a button, so after dragging by it the flag stayed set, and the collapsed pill's next click read a drag that was not its own and refused to expand. Cleared on both end paths: pointerup and the buttons === 0 bail for a release the window never saw. * Studio: let the loaded models card be closed, and reopen it on the next load Adds an X beside the collapse chevron, and swaps the card's brain badge for the sparkle. Closing is its own flag, not the Settings toggle. They read as the same thing but are not: closing means "not now", so the next model load brings the card back, while the toggle means "stop showing me this" and no load may override it. Reopening happens on the start of a load, so the card is up for the same stretch as the toast, and setting the flag to what it already holds is inert, since every load start writes it. Kept in storage rather than in memory, like the collapsed state, so a reload does not undo a close. * Studio: eject with the eject glyph, and badge the card with the single sparkle A row's X released the model's weights while the header's X only closed the card, so one glyph stood for two unrelated acts. The row now uses Remove circle, which is already the model picker's own eject shortcut, and the header keeps the X the Live monitor closes with. The badge was Sparkles, which is the two-star glyph. Sparkle is a separate icon and the free set shipped in 4.1.1 predates it, so it is defined in lib/ beside the chevrons rather than moving every other glyph in the app to a newer icon set for one shape. The path is the published stroke-rounded outline with its fixed stroke swapped for currentColor. * Studio: drop the drag sentinel option, which guarded a state that cannot happen I added this on a review note and the note was wrong. Every drag handle wires onPointerDown to startDrag, and startDrag zeroes movedRef; pointerdown fires before click, so the pill always clears the sentinel before its own click reads it. A drag by the expanded grip cannot leave a stale flag for the pill to trip over, and the clickFollows option was guarding a state the code cannot reach. Checked rather than reasoned: with the option removed, dragging by the grip, collapsing, and clicking the pill once reopens the card, on all three engines. That sequence is now a browser check, and it asserts the collapse really happened first so it cannot pass vacuously. The unit tests assert the invariant that actually holds the line, which is that every handle goes through startDrag. * Studio: keep recording loads while the card is closed, so one can reopen it Closing the card is meant to mean "not now", with the next load bringing it back. The dismissal was cleared correctly, from the indicator's own subscription, but the rows come from useLoadedModels and that subscription early-returned while the card was hidden. So the loading: true event was never recorded, and the card reopened onto nothing. Chat and dictation limped: the 5s poll synthesises their loading rows from status.loading and the per-engine flag. Images and video have no such fallback, so with nothing else resident the card did not come back at all until the load finished, which on a large download is minutes. Reproduced in a real browser before fixing, and the reopened card now reads "Image . Loading" for the model that reopened it. Recording is now separate from showing: only the Settings toggle stops it, since that is the one that means "stop telling me", while a closed card or a route that hides the card keeps listening. The pending map is still emptied when recording stops, so a load missed while the preference was off cannot come back as a row no poll can retire. * Studio: do not read an unreadable dictation status as an empty one The dictation unload answers {loaded_model: null} whatever it did, and the backend serves gguf from the transformers engine when whisper-server is absent, so the eject re-reads the status to find out what actually happened. But readSttStatus returns null on any non-2xx as well as for an empty runtime, and the re-read treated the two the same: a transient 404 or 500 on the confirming read toasted "Ejected" and dropped a row whose model was still holding memory. That is the exact case the re-read exists to catch. An unconfirmed unload is now its own outcome rather than being collapsed into success or failure, since neither is true, and it warns instead of claiming the memory was freed. Images and video already raise here, since their unload responses go through parseJson; only dictation swallowed it. * Studio: re-read chat residency when a media load takes the GPU, not when it ends The arbiter evicts chat inside the image or video load POST, before the background download starts, but the chat runtime only re-read its status when that load settled. Since the settle waits for the whole download, the picker and the header went on naming the evicted model for its entire duration. Measured against a live backend rather than inferred: with Qwen3-0.6B-GGUF resident, /api/inference/status reported active_model null 1.8s after the image load POST returned, and a send to the model the picker still offered answered 400 "No model loaded. Call POST /inference/load first." The load is now announced a second time once the POST returns, which is the instant the eviction has committed; announcing only before it would have had listeners re-read a status that was still correct. Dictation is excluded, since it is a sidecar and takes no GPU ownership. Also settle the media page's own busy state when an eject cancels its load. The indicator shows the resident row next to the incoming one during a replacement, so that eject is reachable; the page's listener then stops the load-progress poll, which is the only thing that clears busy. Left set it locked the page -- the picker ignores every choice, and Unload is not rendered once the status read comes back empty -- and both pages stay mounted for the session, so only a reload recovered. * Studio: stop the indicator reporting ejects it did not perform Three of the four eject paths could report a success with no unload behind it, and the card could overwrite a saved position it was only adapting to the current window. Found by driving the branch against a real backend in Chromium, Firefox and WebKit with the four /status endpoints stubbed. A stale row over an idle runtime said "Ejected X". ejectRuntimeRow folded verifyResident's "gone" verdict into "ejected", so a row up to one poll old whose runtime had already released everything reported an unload that never ran. Images, video and dictation all reach this without anyone doing anything: the dictation sidecars release themselves when idle. It is now a distinct alreadyFree outcome, and the toast says the model was no longer loaded. The row still goes, since its memory is free either way. That also retires the `&& resident` guard, which could not be false: verifyResident only answers "replaced" when resident is set. A cached chat row was unloaded without confirmation. It is the one path with no scoped re-read, because a cached row is never the active model and so can never match one, and /unload answers 200 for a name the backend no longer holds. So the success was the call itself rather than evidence of a release, and the picker was cleared on it. ejectChatModel now takes an optional readCached and checks the row against what the runtime still holds, the way every other path checks its own. The card overwrote a position saved on a larger screen. The persist ran from an effect on `position`, and a reclamp changes `position` too, so opening the app on a laptop wrote the laptop's clamp back to localStorage and going back to the large monitor left the card where the laptop had put it. Only a landed drag is stored now; the read path clamps, so nothing needs the clamp kept. Arabic said the card is in the bottom-left. Nothing sets document.dir and the stack is anchored with a physical `right-4`, so it is bottom-right there too. Covered by three cases in the node suite for the cached row, one for the persist rule, and one in the Playwright suite for the stale row, which runs on all three engines. Full suite 896 pass, typecheck, i18n parity and build clean, and the same on a tree merged with current main (1126 pass). * Studio: keep a row whose runtime could not answer, rather than reading it as gone A read that fails or times out comes back as null and the describe helpers map null to no rows, so one bad poll took a loaded model off the card. Against a remote Studio all four reads can fail on the same blip, and the whole card went while every runtime stayed resident. An unreadable source now keeps what it last showed. A readable one is still replaced outright, including by an empty answer, so an unload clears the row as before -- that is the distinction that matters, and both halves are checked in the browser suite: a sustained 503 on the chat status keeps the row, and a readable empty status still retires it. Reverting the change fails the first and leaves the second passing. * Studio: sequence the chat refreshes, and reopen a closed card for a silent load Announcing a media load twice put two chat refreshes in flight inside the load POST's own duration, measured at 1.8s against a live backend. They read the status either side of the eviction and answer in whatever order the network gives, so the older one could land last and re-pin the model the newer had just seen released, leaving chat claiming a model that 400s on send until the load settled. Every refresh now takes a generation and the newest wins; a superseded one writes nothing back and reports no failure either, since its answer describes a moment that has passed. Separately, the close tooltip promises "Back on the next model load", and that did not hold for a load started outside the frontend: the OpenAI-compatible API and auto-switch raise no lifecycle event, and dismissing the card had also stopped the poll, so nothing was left to notice. The poll now runs while the card is closed, and a row appearing that was not there on the previous poll reopens it. The first poll after closing is the baseline rather than a reopen, or -- since a dismissal survives a reload -- whatever was already resident would make the card impossible to close at all. * Studio: keep a settled load's row when the read that should replace it failed An optimistic row is retired once a status read has replaced it with a real one. But readLoadedModels is handed the polled rows only, and an optimistic row lives in the pending map, so a read that could not see that source has nothing to preserve: retiring anyway took the row for a model that had just finished loading off the card, on the strength of a request that failed. The read now reports which sources it could say nothing about. Those stay settled, so the next readable poll retires them instead, and a read that failed outright is treated as evidence about none of them. A source that did answer is retired exactly as before, including when it answers empty, or the row would never go at all. * Studio: stop writing off an eject target that was only demoted to the cache Eject an active row that was replaced inside the card's five-second poll window and nothing was released. The standard backend does not evict on a switch: load_model only moves active_model_name and leaves the previous model in backend.models (core/inference/inference.py:620), which /status still reports under `loaded` (routes/inference.py:8112). So a switch from another tab, the OpenAI-compatible API or the CLI demotes A to a cached entry that is still holding its weights, the scoped read cannot match it, and the click reported "A is no longer loaded. B took its place and was left alone" having unloaded nothing. With the replacement since gone it read as alreadyFree and dropped the row as well. The branch now asks what the runtime is really holding before writing the row off, reusing the readCached the confirming read already had, so the demoted model is unloaded by name and confirmed like any other. A target that is genuinely gone still reports replaced and unloads nothing, and with readCached omitted the behaviour is unchanged. GGUF is unaffected: that branch reports loaded as the single active id. Three cases in the node suite: a row replaced while still cached, a row replaced and really gone, and an idle runtime still holding the row. The first and third fail with the source reverted. * Studio: keep the indicator off the login page, and surface what a GGUF load left behind Two from the latest round, both real. Recording polled four protected endpoints on /login. Widening the recording flag past `enabled` was right for dismissal, so a closed card still hears the load that reopens it, but it went one step too far: `canShowIndicator` also carries the auth gate, and gating recording on the Settings toggle alone left the poll running before there was a session. Each tick then sent four requests that 401 and drove authFetch's refresh-then-redirect ladder, every five seconds, for as long as the login page was open. Reachability is now hoisted and shared, so tracking survives a dismissal but not a route with no session. A model cached behind a GGUF was invisible and could not be released. The GGUF load unloads only the ACTIVE Unsloth model (routes/inference.py:6603-6610), so a Transformers model cached behind it keeps its weights, but the GGUF status branch reported `loaded` as the single active id. So after loading A, then B, then a GGUF, A held VRAM that nothing surfaced and no client could free. That branch now also reports what the registry still holds, deduped against the GGUF's own id and peeking rather than constructing the orchestrator. The unload route already reaches a cached model by name through _resident_standard_model_name, so the rows this adds are ejectable as they stand. Three backend cases and one frontend case, each failing with its source reverted. 919 node tests and 31 backend tests pass; typecheck, i18n parity and build clean. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: lift the corner stack above the chat composer, which the card was covering The loaded models card is the first overlay in the bottom-right stack that is persistent rather than transient, and the chat composer docks to the bottom of that same column once a thread has turns. The card landed on top of the Send button and swallowed the click: Playwright resolved the button, found it visible, enabled and stable, then retried for 60s while "unsloth/gemma-3-270m-it-GGUF ... subtree intercepts pointer events". Caught by the chat UI suite on both ubuntu-latest and macos-14. The stack already knows how to keep clear of something: the Live monitor publishes its box and stackGeometry lifts the stack above it. That store held one box, from one publisher, so the composer had nowhere to say where it is. It now keeps a box per publisher and dodges their union, which is also the right answer while the monitor is mid-exit and two panels are briefly published at once. The composer publishes through a small usePublishedFrame hook, re-measuring on resize and through a ResizeObserver because it grows with its input and moves from centred to docked without a resize event. A hidden composer measures 0x0 and is withdrawn rather than published, which would otherwise pull the union to the top-left and pin the stack there. Verified against a live backend at 1280x900, 1280x720 and 1440x900 with the composer docked: the element under the Send button's centre is the composer, and reverting the publish alone puts the overlay stack back there at all three. 1289 node tests pass, typecheck, i18n parity and build clean. * Studio: fold the stack's obstacles one at a time, not as their bounding box Follow-up to the composer fix. Publishing a second box made the store hold two, and reading their union as one obstacle was wrong: a tall monitor and the wide docked composer share almost no area, so the rectangle around the pair covers most of the viewport. stackBottomInset then lifted the stack to its clamp near the top of the screen and put it back over the monitor it was dodging. The chat UI suite caught it on Windows and macOS as the card swallowing the monitor's Close button, at the step that drags the monitor up the column first, which is exactly when the union is most unlike its parts. stackGeometry now folds per box: each asks for the lift it needs, the stack takes the largest, and the height is the shortest any of them allows. The single-box maths is untouched, so its existing cases still pin it. The store keeps the union as a summary only, and says so. Three cases added: the fold against a monitor parked high, which is the case that diverges, plus an empty list and a single box matching the old calls. 1295 node tests pass, typecheck and build clean, and the docked-composer probe still finds the composer under the Send button at three viewports. * Studio: let only the newest status read write on the media pages The Images and Video pages hold their own status and re-read it on tab activation and on their own actions, never on a timer. Two reads are therefore in flight across an eject: the activation read that saw the pipeline loaded, and the post-eject read that saw it gone. Responses have no order, so the older one could land last and put the loaded status back, leaving the page offering to generate against a runtime that was already free, with no poll coming to correct it. Both pages had it, identically. Every read now takes a ticket before it awaits, and only the newest may write. All three writers on each page go through the same gate, not just the refresh: the load-progress read and the unload response could overwrite a newer answer just as easily. Also drops the merged rectangle the frame store still exposed. Nothing read it after the fold landed, but it was the shape the review picked up on, and it merged obstacles that do not touch: the rectangle around a left-hand monitor and a bottom-right composer spans the empty space between them. The store keeps the boxes apart and says why; stackGeometry already folds them one at a time. Four cases pin the sequencing, all four failing with the pages reverted. 1299 node tests pass, typecheck, i18n parity and build clean. --------- Co-authored-by: shimmyshimmer <danielhanchen@gmail.com> Co-authored-by: danielhanchen <moonshotaisubstack@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-08 08:27:45 -07:00
Daniel Han	7103733d7b	Studio: hold the Mac capability verdict while an MLX repair can still overturn it (#8152) ... * Studio: keep the Mac capability verdict provisional while MLX repair runs * Studio: bound the MLX repair hold so a scheduler that never runs cannot spin the tabs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate the Mac capability verdict on a real Apple Silicon host The hold that keeps /api/health from publishing a chat-only verdict an MLX self-heal is about to overturn was covered only by unit tests, which drive it with a fake clock and a stand-in repair worker. Nothing ran it where `import mlx.core` succeeds or fails for real. tests/studio/mac_capability_verdict_smoke.py boots the real `unsloth studio` through .github/scripts/boot-studio-api-only.sh and polls /api/health from the first reply the socket gives, judging the whole sequence rather than the last answer, since the reported bug was a verdict that was wrong for a while and right afterwards. Three boots, one per scenario, because the self-heal is once per process: - real-mlx: the verdict must settle chat_only=false with /api/system reporting device_backend "mlx", and no reply on the way there may publish chat_only true. - no-mlx-settles: MLX made unimportable and UNSLOTH_DISABLE_MLX_AUTOREPAIR=1. The verdict must still settle promptly to mlx_unavailable, or a Mac with no repair coming spins Train and Video for the session. - no-mlx-repair: MLX made unimportable, self-heal on. No reply may publish the verdict while the installer runs, including well past the pre-start grace so the hold is provably the live worker, and it must settle once the install fails. MLX is made unimportable the way tests/studio/test_hardware_dispatch_matrix.py does it, through the import system, but from a sitecustomize on PYTHONPATH since there is a server to boot. It hides mlx from PathFinder rather than raising, so the find_spec in unsloth/__init__.py answers None instead of crashing. The installer in the third scenario is a stub uv first on PATH that records its argv and takes its time; utils.mlx_repair is untouched and runs its real thread, real command and real subprocess against it. The recorded argv is asserted, so a run where the self-heal never reached the installer fails rather than passing quietly. mlx-ci.yml gains the three steps plus a venv at $STUDIO_HOME/unsloth_studio, which is what `unsloth studio` re-execs into and refuses to start without. Building it over the job's own site-packages gets the real launch path without the ten minutes install.sh costs. Path filters pick up main.py, mlx_repair.py and the two boot scripts; timeout goes to 40 minutes for the in-flight boot, which waits out the torch warm before the self-heal is even scheduled. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * CI: let boot-studio-api-only.sh actually boot API-only The script exports UNSLOTH_API_ONLY=1 and its banner says --api-only, but nothing in unsloth_cli reads that variable. Whether the web UI is served is decided by the CLI flag alone; the backend only reads the variable back out in main.py to pick a CORS profile, after run.py has set it from the flag. Every caller has therefore been booting a server that serves the frontend, which is what the Playwright UI smokes need, so the flag is opt-in and no existing caller changes. The capability-verdict boots in mlx-ci.yml are the first caller without a built studio/frontend/dist (it is gitignored, and that job installs with pip rather than install.sh). Without the flag the server printed "Unsloth frontend build not found" and exited before binding, which is how this surfaced. Also in the smoke driver: give up a wait as soon as the server process is gone rather than spending the full budget, and flush the boot banner when it is written so it does not surface minutes later next to the timeout it preceded. * Studio: drop the verdict cache and bound both halves of the MLX repair hold Remove the localStorage hardware-verdict cache. It did not buy what it claimed: the store deliberately kept `fetched` false, so `capabilitiesUnknown()` is still true on a returning user's first paint and the rows spin exactly as they do without it. Its only demonstrated effect was unbreaking a `/settings` deep link, and it introduced a real regression in exchange. model-selector.tsx, pickers.tsx, model-inspector.tsx and app-sidebar.tsx read `chatOnly` with no `capabilitiesUnknown()` gate, so a cached `chatOnly=false` during a repair window offers MLX-only models that cannot load - worse than the conservative `navigator.platform` guess it replaced. env.ts and both frontend test files are back to what main has, and nothing under studio/frontend changes in this PR. Start the pre-start handoff grace on the first reading that sees the warm stopped rather than the last reading that saw it running. The warm's final stages are C-extension imports that hold the GIL for seconds at a time, so health requests queue behind them and the next one served can be the first in a minute. Measured from the last observed poll the grace could already be spent by the time anyone could ask, publishing the mlx_unavailable verdict during the very handoff the hold exists to cover. A stopped reading between two running ones is a lull, not the end, so it reopens. Bound a live repair worker with _WORKER_BUDGET_S, the subprocess timeout plus 300s. attempt_mlx_repair times the uv call but not the mlx_stack_available() imports that verify the install nor the detect_hardware() pass after it, and those import mlx.core, mlx_lm and mlx_vlm on a stack this module already assumes can park indefinitely. An alive thread was an unbounded answer, so a parked worker held the verdict provisional for the whole session. Fix _superseded_by_mlx_repair and mlx_repair_in_flight docstrings, which claimed a cap that did not exist. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-08 06:47:11 -07:00
Daniel Han	95d9f23a64	Normalize the /api/health wait and extract it into one script (#8060)	2026-08-06 22:32:49 -07:00
Daniel Han	7e47cc03c1	Extract the API-only Studio boot into one script (#8029) ... Twenty-one steps across eight workflows ran the same five-line body, varying only in port, log filename and the name of the environment variable the pid lands in. .github/scripts/boot-studio-api-only.sh takes those three as flags. Three parts of that body are easy to get subtly wrong, which is the argument for one definition rather than twenty-one: rm -rf, not `reset-password`. The boot only re-seeds a fresh .bootstrap_password when the auth directory is absent, so a caller that resets keeps the old password and the test authenticates against stale state. The pid has to reach $GITHUB_ENV, because the step that stops the server is a different step with a different shell. Both streams go to the log. The server writes startup diagnostics to stderr, so a redirect without 2>&1 leaves an empty log on exactly the failure someone needs it for. Not folded into serve-unsloth-run.sh, despite the plan saying so. That script boots `unsloth run --disable-tools` and its body is banner-key parsing and /v1/models resolution; these boot `unsloth studio --api-only` with auth wiped and no banner. The two share the word boot and nothing else, and each header now says which is which. Deliberately does not wait for health: the callers' waits differ, some polling /api/health and stopping while others go on to rotate the password and load a model in the same step. Left alone: three boots inside Playwright retry loops, which re-boot mid-loop and reassign the pid in the running shell, and three boot-briefly-and-kill steps that keep the pid in a local variable rather than $GITHUB_ENV. Each caller's paths filter gains the script, so an edit to it re-runs the workflows that call it -- the trap #8015 hit when the install body moved out from under those filters.	2026-08-06 07:40:28 -07:00
Daniel Han	607b928adf	CI: run the three mac inference smokes as phases of one job (#7978) ... * CI: run the three mac inference smokes as phases of one job The three jobs in this workflow differed only in model, port and test body. Their checkout, setup-node, setup-python, install.sh --local --no-torch and assert-llama-loads.sh preamble was byte-identical, so every triggering PR ran the same install on three macos-14 runners at once. Concurrent macOS jobs are capped at 5 per account and that pool is shared with unslothai/unsloth-zoo, so this one workflow took 3 of the 5 slots to derive the same install three times. They are now three sequential phases sharing a single install. Each phase keeps its own model cache, port, server log and uploaded artifact, so a failure still names one phase and the earlier phases' logs survive it. Collisions the merge had to resolve, none of which were visible from the job list alone: - cache-gguf and download-gguf were defined in both the tool-calling and json-images jobs. Duplicate step ids inside one job are a hard error, so they are suffixed -tools and -vision. - both jobs cached to the same gguf-cache directory with different models. Merged, the vision phase could have restored tool-phase bytes. Split into gguf-cache-tools and gguf-cache-vision. - all three boots wrote logs/studio.log, so merged, each phase would have truncated the previous phase's diagnostics. Split per phase. Phase-specific values are set only through , never at job level. Whether a later write beats a job-level env: key of the same name is not something the Actions docs pin down, and a phase quietly talking to the previous phase's port is a bad way to discover the answer. Everything inside ${{ }} is a literal for the same reason: a cache key that silently resolved to the wrong model would still pass, just against the wrong bytes. The literals resolve to the same strings as before, so existing caches still hit. Verified: one job, 37 steps, no duplicate ids or artifact names, no dangling steps.* references, exactly one install, three distinct ports and logs, and the extracted test-command set is identical to before once the cache-directory rename is normalised. macOS jobs in this repo: 29 -> 27 on top of #7975 (31 -> 29 on main alone). * CI: bump the renamed GGUF cache keys so the merged phases can cache at all Caught by an A/B on staging CI, not by inspection. Splitting gguf-cache into gguf-cache-tools and gguf-cache-vision kept the cache keys byte-identical, which looked safe. It is not. actions/cache identifies an entry by key AND a version hash derived from the path input, while key uniqueness is global per ref. So the pre-existing gguf-cache entries still hold the old keys: a restore into the new directory misses on version, and the matching save is then refused with "Unable to reserve cache with key ..., another job may be creating this cache". The result is not a one-time repopulation. It is permanent: two consecutive staging runs of the merged workflow both logged "Cache not found for input keys" and both failed to save, so both re-downloaded the tool model (~890MB) and the vision model (~2.4GB + 986MB). That is roughly 3.3GB per run, forever, on a macOS runner from a pool capped at 5 concurrent jobs. Bumping the key suffixes (tools v1 -> v2, vision v2 -> v3) gives the new paths a free key so the entries actually populate. The rule this encodes: changing an actions/cache path requires changing the key too, even though the key alone looks like it fully identifies the entry. * CI: keep per-phase timeouts and phase independence in the merged mac smoke job Merging the three mac inference jobs into one job lost two properties the separate jobs had. A step with no explicit `if:` gets an implicit `success()`, and that check is job-wide, not per-step. So one failed step in phase 1 skipped every remaining step of phases 2 and 3, and a tool-calling or vision regression went unreported until phase 1 was fixed and the job re-run. Phases 2 and 3 now run under `!cancelled() && steps.assert-llama.outcome == 'success'`: a failure inside a phase no longer skips the phases after it, while a failure in the SHARED preamble (checkout, install, llama.cpp load check) still skips them, since assert-llama is itself skipped in that case. The job still ends red, because the failed step already set the job status. The three jobs were also capped at 25 / 25 / 30 minutes each; the merged job had only the aggregate 90. hf-download-with-retry.sh retries forever and names the enclosing timeout as its bound, so one stalled download could hold a macOS slot for the full 90 minutes and leave nothing for the phases behind it. Every step that can block now carries its own timeout-minutes, with 90 kept as the outer guard. Also drops the two leftover job divider comments that still announced jobs 2 and 3 in the middle of the step list. * CI: keep the shared preamble out of phase 1's failure chain The previous round gated phases 2 and 3 on steps.assert-llama.outcome, so a failure inside one phase no longer suppresses the others. It left the two shared preamble steps on the implicit success() check, which reopens the same hole one level up. Install Unsloth and Assert llama.cpp both had no if:, so GitHub applied success(), which is true only when every previous step succeeded. Phase 1's Prime HF_HOME sits above them and carries a 15-minute timeout, and a step timeout marks the step failed rather than cancelled. So a stalled or failing phase-1 model download skips Install, skips Assert, leaves steps.assert-llama.outcome == 'skipped', and silently takes phases 2 and 3 with it. Before the consolidation those were independent jobs and still ran. Install now runs under !cancelled(), and Assert gates on the install step explicitly instead of on the job-wide implicit check. A phase-1 GGUF problem is a phase-1 resource problem: the install probe that install.sh itself needs is fetched live by hf_hub_download when the cache is cold, so the bootstrap can still succeed and tool-calling and vision still get exercised. A genuinely broken bootstrap still fails Install, which still skips Assert and correctly holds both later phases out. Ordering is unchanged, so no cache key or path moves. Behaviour, verified against the step table: all green -> phases run; prime-hf fails or times out -> phases run; checkout, setup-python or install fails -> phases skipped; assert fails -> phases skipped; run cancelled -> all skipped. * Bound the phase-1 cache actions in the mac inference smoke job Phases 2 and 3 got timeout-minutes on their cache restore/save pairs in the previous follow-up, but phase 1's pair was left unbounded. Before consolidation that phase was a separate job with its own 25-minute timeout, so a stalled cache action was bounded by it; after the merge the only bound is the job's 90 minutes, and a stall there would hold a macOS slot and starve phases 2 and 3. 15 minutes matches the sibling Prime HF_HOME step and phase 2's pair.	2026-08-06 01:52:18 -07:00
Daniel Han	1301e16004	Keep the file-edit turn 2 prompt to one instruction, and stop waiving its cap (#7846) ... * Keep the file-edit turn 2 prompt to one instruction, and stop waiving its cap * Tighten the turn 2 comments	2026-08-04 23:47:42 -07:00
Daniel Han	cf4acbcce5	Make the Colab oracle tripwire able to fire, and stop blaming start.py for a hung agent CLI (#7838) ... * Make the Colab oracle tripwire able to fire, and stop blaming start.py for a hung agent CLI * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep a connection-mode cap fatal, and do not read 137 as a timeout * Note that the opencode non-exit is intermittent * Close three gaps the waiver opened: turn-2 side effect, attribution-ab, and the guard tests * Bound a TERM-resistant CLI again, and judge a capped turn 2 on an artifact * Gate both timeout statuses on the clock, keep the cron lint reachable, and make refresh --all atomic * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-04 07:42:42 -07:00
Daniel Han	3c6343b455	Sign the NSIS plugin DLLs, and fail the release on any unsigned file (#7819) ... * Sign the NSIS plugin DLLs, and fail the release on any unsigned file Both 0.1.51-beta and 0.1.512-beta ship four unsigned DLLs: NSISdl.dll, System.dll, StartMenu.dll and nsDialogs.dll. Only nsis_tauri_utils.dll carries the Unsloth AI Inc. signature. tauri-bundler is doing the work already. When a signing identity is configured it copies Plugins/x86-unicode aside, signs all five files listed in NSIS_PLUGIN_FILES, and exports the copy as the NSISPLUGINS environment variable. Nothing consumes it: no NSIS template references $%NSISPLUGINS%, and the only plugin directory added is ADDITIONALPLUGINSPATH, which points at the copy's additional subdirectory and holds exactly one file. So one DLL resolves from the signed copy and the other four fall back to the default unsigned one. Add the directory in our template, guarded so unsigned local builds still compile, and add a check that unpacks each bundle and fails listing every unsigned executable rather than stopping at the first. * Fail the signing gate when a bundle cannot be unpacked 7-Zip leaves a partial tree behind on error and can fall back to its PE handler, which yields sections rather than the payload. Both cases passed the gate having verified nothing. Move !addplugindir above the includes so the signed copy is registered before any plugin is packed. * Tighten comments on the bundle signing gate * Sign the bundled install script and gate on it install.ps1 ships as a bundle resource and is the first thing the app runs after install, but the bundler never signs it and the gate did not look at scripts. Sign it before the build packs it in, and add .ps1/.psm1 to the checked set. Verified on a Windows runner that signtool attaches a valid signature to a .ps1; trusted-signing-cli lists ps1 as supported and does not filter by extension. * Tighten comments on the bundle signing gate * Reduce the Windows false positive surface Three changes, none of which alter behaviour: install.rs no longer passes -WindowStyle Hidden -ExecutionPolicy Bypass to powershell.exe. CREATE_NO_WINDOW already suppresses the console and the NSIS-extracted script carries no mark-of-the-web, so RemoteSigned loads it. That flag pair is the command line Microsoft ships as a detection test. install.ps1 installs uv from a pinned, SHA-256 verified release archive instead of evaluating a downloaded script in-process. installer.nsi fills in CompanyName and InternalName, which upstream's template leaves empty. * Match astral's installer on unmanaged installs, mirrors, and the PATH probe UV_UNMANAGED_INSTALL forces no-modify-path in astral's installer, so it must here too, and honour UV_INSTALLER_GHE_BASE_URL / UV_INSTALLER_GITHUB_BASE_URL so a mirrored environment still works. Every mirror serves the same asset, so the pinned hash is unchanged; UV_DOWNLOAD_URL stays unhonoured because it points at an arbitrary version the pin would then reject. Record where uv actually landed. Refresh-SessionPath rebuilds PATH machine first and drops the in-process prepend, and the recovery probe never checked the XDG_DATA_HOME or unmanaged destinations, so a good install could still report failure. Required by the change above, which leaves that prepend as the only thing putting uv on PATH. * Tighten comments on the false positive reduction changes --------- Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>	2026-08-04 04:12:15 -07:00
Daniel Han	b35bdcbea3	CI: prove an interrupted install can never masquerade as a healthy one (#7552) ... * CI: prove an interrupted install can never masquerade as a healthy one Nothing in CI had ever interrupted an install, which is how the reported failure shipped: quit the desktop app mid-install, the app SIGTERMs the installer process group, and if that lands during 'studio deps' the venv loses structlog. Preflight then probes 'unsloth -h' and 'studio desktop-capabilities', both of which succeed because the CLI's own deps are core, so the app reported ManagedReady with can_auto_repair=false while the backend died on import. A permanent dead end. This kills the installer at each interesting phase and asserts the result is either genuinely healthy or explicitly repairable, never silently ready. 13 legs across macos-14, ubuntu-latest and windows: each of the dependency-pass steps plus the coarse phases (venv, torch, unsloth, setup). The kill targets the process GROUP, matching install.rs. Killing only the leader leaves uv and python children to finish the dependency pass, and the test would quietly prove nothing. Windows has no process groups, so that leg walks the CIM parent links instead, which is the same reason the app carries windows_job.rs. One shared probe for all platforms. The Windows check used to be bespoke inline PowerShell that only ran -h and desktop-capabilities, so it could not observe studio_install_ok, verify-install or desktop-runtime-check: it would have reported FALSE_READY for the very PRs that add them, no matter how well they worked. The probe boots the backend as ground truth and owns the whole process tree, since terminating only the parent leaves children holding the port. install.sh runs with --local, which is load-bearing rather than a convenience: without it the installer resolves unsloth from PyPI and the venv gets the PUBLISHED CLI, so no branch-side change is present and every deeper probe reports 'absent' regardless of what the branch does. Verified: against a tree without the detection, windows kill@studio-deps reports FALSE_READY, reproducing the user report exactly. With #7492 merged the same leg reports REPAIRABLE, and all 13 legs pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the POSIX legs actually run the installer, and fail if they do not install.sh --tauri rejects a custom UNSLOTH_STUDIO_HOME outright (the desktop app still uses the legacy ~/.unsloth/studio root), and this workflow set one at workflow level for every job. So all 11 macOS and Linux legs exited about a second in with ERROR: UNSLOTH_STUDIO_HOME is not supported with --tauri. produced no CLI, took the probe's NO_CLI 'safe' branch and reported success. They were vacuously green. Only the two Windows legs were real, because install.ps1 has no equivalent guard. The override now applies to the Windows job only, and the POSIX legs read the legacy root, which is where --tauri installs. The runner is ephemeral so the real home is as disposable as the override. Also adds the check that makes this class of mistake loud: a leg asserts its kill actually landed on the marker it was aimed at, using the interrupt_reason the driver already records. A leg that never reached its kill point proves nothing, and NO_CLI made that indistinguishable from a pass. * Make the interrupted-install legs able to fail The probe treated a present .desktop-install-in-progress marker as proof of a repairable state, but the drivers seed it unconditionally and never clear it, so REPAIRABLE was unconditional and FALSE_READY unreachable. The Windows leg had no kill-landed guard, blanket continue-on-error, and no repair re-run; -SkipTorch was silently dropped, since install.ps1 parses only --no-torch. Judge the re-run by whether the backend boots, on both platforms. The log grep matched the frontend build printing "up to date" and failed a leg whose venv was fine. * Drop the interrupt cell that could never be interrupted install.sh --local sets skip_base, so install_python_stack returns before any "base packages" label is printed. The kill had nothing to land on and the installer ran to completion, reaching [TAURI:DONE] in 62s. * Fail the leg when the installer finished instead of being killed The driver set reason=marker-hit before the post-marker sleep and never rechecked, so a step whose work was already cached could run to completion inside that beat and still be recorded as an interruption. The landing assertion tests reason != marker-hit, so a fully completed install passed green having interrupted nothing. Reproduced with a stub that exits during the delay: reported marker-hit / killed=true / exit=0 next to "install finished fully". Set the reason after the sleep, on both drivers. Also trigger on _studio_deps.py and install_manifest.py, where the two decisions the probe asserts on are actually implemented. * Kill the group, and stop the probe blocking on a full pipe The escalation was gated on the leader still being alive, so a leader that exits promptly on SIGTERM while a uv or python descendant ignores it skipped the SIGKILL entirely, and wait reaped only the leader. Proven with a descendant that traps TERM: pre-fix its heartbeat keeps ticking while the probe would be running, post-fix it stops. Signal the group unconditionally and drain it after the reap, since an unreaped leader is still a member of its own group. The probe started the backend on stdout=PIPE and read nothing until after the poll loop, so a backend logging more than the pipe buffer during import blocked before binding. Measured 65536 bytes here; a child emitting 200 KB never reaches its bind line, which would make backend_ok false for a healthy install. Write straight to the artefact file. Also trigger on studio/backend/requirements/**, where structlog is declared. * Make the NO_CLI legs assert repair, and fix the Windows straggler sweep Two of the interrupted-install legs were passing without testing anything. The re-run assertion skipped verdict=NO_CLI, but a kill at "venv" or "torch" lands before install.sh ever prints "Installing Unsloth" (:2125, :3667, :3961), so those legs can only ever produce NO_CLI. Three non-gating-exempt cells (macos-14 kill@venv, macos-14 kill@torch, ubuntu-latest kill@torch) therefore asserted nothing beyond a marker appearing in a log. NO_CLI is now included: a re-run must produce a booting backend regardless of how little the first run managed to install. Each re-run step grows an existence check first, because the probe exits without writing verdict.json when the binary is absent and the json.load would crash rather than report. The Windows straggler sweep matched nothing at all. UNSLOTH_STUDIO_HOME arrives as D:\a\r\r/.studio-home, since the workflow joins ${{ github.workspace }} with a forward slash, while Process.Path is all backslashes, so the literal -like missed even the venv's own python.exe. uv is never under the studio home in any case: install.ps1 takes it from winget or astral.sh. Normalise the separators, match uv by name (the runner is ephemeral and runs no other uv), and skip the home comparison entirely when the variable is empty, which would otherwise turn the pattern into "**" and kill every python on the runner. * Run the Windows legs as the desktop does, and judge repair by what preflight reads The Windows matrix set a workspace-scoped UNSLOTH_STUDIO_HOME, which forces install.ps1 down the shell-install path: install.ps1:189-215 rejects a custom root under --tauri, so those legs ran with UNSLOTH_TAURI_MODE=0, the frontend build on and no bundled-file overlay, while the desktop always spawns the installer as --tauri with the variable scrubbed (install.rs:202 and :356). The torch leg could not even reach its marker: "Installing PyTorch" is printed only by Write-TauriLog (install.ps1:2440), so it was killed at the deadline. Both legs now run --tauri --local at the default root, and the probe and re-run resolve the CLI under %USERPROFILE%\.unsloth\studio. The probe counted `studio verify-install` and `studio desktop-runtime-check` failures as proof the app can repair, but preflight/managed.rs runs only `-h` and `studio desktop-capabilities --json` (:357) and reads studio_install_ok from that payload (:445); neither deeper command is invoked anywhere under studio/src-tauri. A leg where capabilities regressed to ready while only those standalone commands saw the damage would have passed green with the app stuck on ManagedReady, which is the exact false negative this workflow exists to catch. They are still run and recorded in verdict.json, just no longer repair evidence. An interrupted install can leave the console script in place while its venv interpreter is gone. The probes go through run(), which catches OSError, but the backend spawn did not, so the probe aborted before writing verdict.json and both workflows died on the json.load instead of reporting. That state is now recorded as backend_spawn_error and lands on REPAIRABLE, which is what `-h` failing already implies. On win32 the CLI re-spawns the server as a child and waits on it (unsloth_cli/commands/studio.py:1543), and CREATE_NEW_PROCESS_GROUP does not make terminate() reach descendants, so the reap left a server holding the venv open while the repair step reinstalled into files Windows had locked. Use taskkill /F /T for the tree. The straggler sweep now falls back to the default studio root, since under --tauri there is no UNSLOTH_STUDIO_HOME to match on. * Judge the install the way preflight does, and reap the whole probe group Read desktop-capabilities the way the desktop reads it. preflight/managed.rs pipes stdout and sends stderr to /dev/null (managed.rs:358), then hands the whole stdout buffer to serde_json (managed.rs:414). The probe concatenated both streams and scanned to the first brace, so a single diagnostic line on stderr made json.loads raise on the trailing text, studio_install_ok stayed "absent", and a broken backend was reported FALSE_READY over an install the real app parses, sees as incomplete, and offers to repair. That fails a valid recovery change for a reason that exists only in the probe. stdout and stderr are now captured separately and stdout is parsed strictly; a payload that does not parse counts as repair evidence, matching the Stale the desktop reports when the capability probe returns nothing (managed.rs:521). A booting backend alone is not a finished install. The manifest is written last (install_python_stack.py:3255), so a kill after "studio deps" but before it, the data-designer leg, leaves a venv whose backend boots while desktop-capabilities still reports studio_install_ok=false and preflight reports Stale (managed.rs:445). Calling that HEALTHY skipped the re-run step, so the leg asserted nothing beyond a marker appearing and never exercised the version fast path that is supposed to clear an incomplete install, which is the half of the bug that strands the user. HEALTHY now requires both. Escalate to the process group after reaping the probe's backend. reap() returned as soon as proc.wait() succeeded, and the leader exits promptly on SIGTERM while a uvicorn worker does not, so the SIGKILL iteration was skipped and that worker kept the port and the venv open while the repair step reinstalled underneath it. It also read os.getpgid(proc.pid) after the reap, which raises. The pgid is now captured up front and SIGKILL always goes to the group, the same escalation interrupt-install.sh:94 makes. A heartbeat experiment left the group alive with the old sequence and empty with the new one. Trigger the workflow on pyproject.toml. Every leg installs the checkout with --local, so that file decides the unsloth console script and the core dependencies the probe leans on: -h and desktop-capabilities only survive a torn install because typer/click/rich are declared there. No other install workflow interrupts the installer, so such a change would otherwise merge without a single leg running. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Judge an absent capability field and a dead -h the way preflight does The probe left studio_install_ok=absent undecided and judged those installs on whether the backend booted. preflight/managed.rs:445 tests studio_install_ok != Some(true), so an absent field is Stale exactly like a false one; a CLI too old to carry it is already rejected one check earlier on desktop_manageability_version. The gap mattered in both directions: a payload that stopped carrying the field reported HEALTHY on every booting leg and skipped the re-run assertion this workflow exists to make, and a torn venv with a working -h was failed as FALSE_READY even though the app would have offered repair. unsloth_cli/commands/studio.py is in this workflow's path filter precisely to catch that class of change, so it must not be the thing that silences it. The verdict also consulted cli_h_ok only in the repairable arm, so a CLI that cannot print help was called HEALTHY whenever the backend happened to boot. probe_managed_bin runs -h first and returns Stale cli_unusable before it ever reaches the capability probe (managed.rs:465-478), so that install goes to repair in the real app and the leg must assert it here. * Judge the probes on the desktop's deadline, and interrupt the host it uses Preflight gives each managed probe ten seconds and nothing more: managed.rs:337 wraps `unsloth -h` and managed.rs:390 wraps `studio desktop-capabilities --json` in a tokio timeout, kills the child on expiry, and returns Stale as "cli_unusable" or "desktop_capability_probe_failed". The probe allowed three minutes, so a venv torn badly enough that its CLI only answers after half a minute of retries was recorded HEALTHY here while the real app shows it as repairable. That skips the re-run assertion the leg exists to make, which is the same false-HEALTHY hole the studio_install_ok and -h gating already closed. Both calls now use the desktop's ten seconds, and the elapsed time is recorded so a leg that flips for timing reasons says so in the artefact. On Windows the installer child now runs where the desktop runs it. install.rs 325-339 spawns the bundled install.ps1 as powershell.exe with -NoLogo -NoProfile -NonInteractive -WindowStyle Hidden -ExecutionPolicy Bypass -File, so Windows PowerShell 5.1 is the only host a real desktop install ever uses. The interrupted run and the repair re-run both used pwsh 7, and every other Windows job in .github runs install.ps1 under pwsh too, so the installer's behaviour on 5.1 was covered by nothing: .NET Framework instead of .NET, OEM console encoding instead of UTF-8, and different native-command and OSArchitecture reporting are all real sources of divergence. A workflow whose point is to reproduce what the app does cannot run a different interpreter than the app does. The driver itself stays under pwsh; only the installer child and the repair invocation change. * Tighten the interrupted-install comments * Tighten the probe docstrings * Fail the leg when the installer completed inside the kill window * Land the kill in the marked step, and reject non-boolean capabilities Two holes found from the staging run's own logs. The venv leg never interrupted the venv step. Creating the venv takes ~0.1s, so by the time the 1s poll noticed its line the installer was already in "Installing PyTorch", and the flat 3s beat sent the signal there: staging run 30419729244 shows both step lines in the tail and a kill 4s in. That made the leg a duplicate of the torch leg while its label claimed otherwise. Both drivers now poll in half-second slices, cut the beat short the moment a later [TAURI:STEP] line appears, and print the step the signal actually landed in, warning when it is not the marked one. Sub-step markers such as "studio deps" print no step line, so they keep the whole beat and never warn. studio_install_ok is Option<bool> (managed.rs:43), so serde rejects a non-boolean and the whole payload fails to deserialize, which the desktop reports as Stale. bool() read a JSON string "false" as True, so the probe called a torn install ready. Only a literal JSON true counts now. * Tighten the interrupt driver comments * Fail the leg when the signal landed after the marked step The cut-short added last round only helps when the marked step is still the last [TAURI:STEP] line at the moment the poll notices it. Creating the venv takes ~0.1s (staging run 30419729244: 03:31:07.371 -> 07.478), less than the 0.5s poll, so the next step's line is usually already in the log when the marker matches, the step count never changes during the beat, and the full 3s elapses inside "Installing PyTorch". Reproduced with a stub installer against the driver at head: kill at 4.11s, step at kill "Installing PyTorch". The leg then duplicates the torch leg while its matrix label claims the venv step, and passed green on nothing but a :⚠️:. Both drivers now skip the beat entirely when the marked step is already over, so the kill goes out at once instead of deeper into the next step, and both record interrupt_step_mismatch in interrupt.env. The landing assertion fails on it: a warning that cannot fail the leg proves nothing. Sub-step markers ("studio deps", "pip bootstrap") print no step line of their own and stay exempt, as before. The venv leg becomes experimental. Its step is shorter than any log poll can resolve, so it must not block the PR on a race it cannot win, and it still probes the earliest torn state whenever it does land. * Tighten the interrupted-install workflow and driver comments * Land the kill in the phase each leg is named for Splitting the log on \r shows that 5 of the 12 legs of staging run 30419729244 interrupted a later phase than their label claims, and the run was fully green. The venv leg's install.log is byte-identical to the torch leg's. So is pip-bootstrap's to unsloth-extras'. Worse, both "studio deps" legs, the cells that reproduce the reported bug, were killed at "7/10 data designer deps" and "12/14 local plugin": their own probe artefacts report backend_ok=true, so structlog was installed and the flagship cell was passing on the manifest gate alone. Two causes. The dependency pass rewrites ONE physical line with \r (install_python_stack.py:2499), so its sub-steps are CR-separated segments and a line-based check could not see one end; the drivers exempted them and warned about nothing. And the flat 3s beat between the marker and the signal is longer than several phases, while every phase label prints BEFORE its work starts, so the beat pushed the signal past the phase instead of into it. Both drivers now split on \r, track the running phase at both levels, and judge a sub-step marker against the running sub-step and a step marker against the running step, so a step is not "over" because the sub-steps beneath it advanced. The beat defaults to 0 and is set per leg, 3s only for torch, unsloth and setup, which run for minutes. The mismatch is recorded in interrupt.env and the landing assertion fails on it, in both languages. Both detectors were replayed against the 12 real logs from 30419729244 and agree with the artefacts on every leg. venv and pip-bootstrap become experimental: their phases are shorter than any log poll can resolve. * Fail the Windows leg when the installer never exited The driver writes installer_exit=running when the installer outlived Stop-Tree and WaitForExit, and 'running' is not '0', so the landing assertion accepted it. A live installer writing into the venv while the probe reads it is not an interrupted install. Only a real integer exit code counts now, checked against 0, 143, 137, -1, running and the empty string. * Drop the legs that cannot land, and prove the kill was delivered Two cells never interrupted the phase they were named for. "Creating virtual environment" runs 0.107s (staging 30419729244, 03:31:07.371 -> 07.478) and "1/10 pip bootstrap" is over just as fast, both shorter than any poll that watches the log, so in 30423181897 and 30424366953 the signal landed in "Installing PyTorch" and "2/10 unsloth extras" every time. Each was another leg wearing a false label, so they are gone rather than allowed to fail, and continue-on-error goes with them: a leg permitted to fail asserts nothing. The only coverage lost is a venv caught half-written, which interruption cannot reach at this resolution; the torch leg's signal lands ~3s into a multi-minute download, so it already leaves a complete venv with nothing installed into it. The landing check also accepted an installer that failed on its own. A dependency error between the driver's last liveness check and the signal exits non-zero, which the exit != 0 guard let through as a kill. POSIX now requires 143 or 137, the only statuses a signal produces here and what all ten POSIX legs of 30424366953 reported. Recording whether kill(2) returned 0 would not separate them, since the unreaped leader keeps its group alive. Windows has no such status, so the driver records whether Stop-Process actually terminated the installer: it throws on a process already gone, so the flag is false exactly when there was nothing left to interrupt. * Signal at the marker, with no beat to overshoot the phase Staging run 30426111484 failed the macOS torch leg on the landing check: the 3s beat carried the signal from "Installing PyTorch" into "Installing Unsloth", because the PyTorch step, which this workflow called minutes long, finished in under three seconds. The beat only ever existed to land mid-work, and it cannot do that safely: every label prints before its work starts, so detection is already inside the phase, and any wait is a bet on how long that phase runs. It lost in 30419729244 and again here. So the beat is gone rather than retuned, and with it the matrix knob and the driver parameter on both platforms. The landing check stays and can still fail, since a phase shorter than one poll is seen only after it ends. The Windows driver also polled every 500ms while its own comment claimed a fifth of a second. That is 2.5 slices of overshoot the POSIX side does not carry, and it is now 200ms like the POSIX loop. * Kill the installer before its children, not after The depth-first walk killed the child install.ps1 was waiting on and only then the root, which races the leader's own reaction to that death. It is not a theoretical race: in staging run 30424366953 install.ps1 had already printed "unsloth studio setup failed (exit code -1)" by the time Stop-Process reached it. A leader that wins the race makes Stop-Process throw, and the new root-kill assertion would then fail a leg whose interruption the driver really did deliver. The tree is now snapshotted first, since a dead parent leaves nothing to walk, then the root goes down ahead of its descendants. A dead leader cannot react to a child and cannot respawn one either, which is what the depth-first order was for. * CI: give the probe the desktop's startup grace and fail a nonzero repair The probe allowed the backend 120s to answer /api/health while the desktop waits 5 minutes (BACKEND_STARTUP_GRACE_PERIOD, commands.rs:9), so a slow but healthy install could be reported FALSE_READY. A broken backend exits at once and the poll breaks on it, so the longer deadline only bounds a live backend. The re-run step also accepted a HEALTHY probe over an installer that exited nonzero. setup.sh does fallible sidecar and GPU setup after the manifest is written, and the desktop returns the repair error without starting the backend (commands.rs:615-630). The Windows leg ignored powershell.exe's status entirely. * Tighten the interrupted-install comments Comments only: shorter wording for the same rationale, no code touched. * CI: raise the interrupted-install job timeout above its own deadlines A leg configures up to 25 minutes to the marker plus two probe passes of up to 17 minutes each around a repair install, so the 60 minute limit could cancel a slow runner mid-assertion. Legs land in 6 to 10 minutes in practice. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothai@gmail.com>	2026-07-29 06:27:33 -07:00
Daniel Han	b6781a8bfe	CI: prove the installer works on a machine with no developer toolchain (#7551) ... * CI: prove the installer works on a machine with no developer toolchain No job has ever run the installer on a machine without one. studio-mac-install-matrix.yml is the only macOS installer job and it runs 'bash install.sh --local --no-torch' on runners that already have the Xcode CLT selected and setup-python preinstalled, so the CLT gate never fires there, and --local is precisely the mode that legitimately needs git. Repo-wide there was zero coverage of xcode-select or CommandLineTools outside install.sh itself. clean-machine-install-ci.yml runs the installer on a genuinely stripped machine. macOS legs move /var/db/xcode_select_link, /Library/Developer/CommandLineTools, /Applications/Xcode*.app and Homebrew aside, so xcode-select -p, git, cc and clang really do fail, and restore unconditionally afterwards. Removing the select-link alone is not enough: xcode-select falls through to a full Xcode.app and re-arms /usr/bin/git. Linux legs use containers, which are genuinely clean. Windows legs cover winget visible and masked, plus windows-11-arm. A WSL leg covers the 126 lines of WSL-specific install.sh logic that had no runtime test. Each macOS leg runs four deliveries: pipe (the advertised command, and the shape that turns an early exit into curl (56)), file (separates installer logic from pipe delivery), no-torch, and tauri (stdin closed, no tty, as the desktop app invokes it). One leg records every toolchain invocation and asserts the trace, which is the real deliverable: proof the installer never reached for a compiler rather than proof it happened to succeed. The asserts test that tools do NOT WORK rather than that they are absent from PATH. On a real virgin Mac /usr/bin/git and /usr/bin/cc exist as CLT stubs, so 'command -v git' succeeds and only running it tells the truth. desktop-app-clean-machine-ci.yml installs and launches the SHIPPED desktop app release on a stripped machine, covering Gatekeeper and quarantine on macOS, NSIS silent install on Windows, and Xvfb with WebKit2GTK on Linux. Known limit, stated plainly: hosted macOS runners are developer machines. Masking reproduces this bug and proves the installer does not invoke a toolchain, but it cannot prove no hidden dependency exists on a truly virgin Mac. An ephemeral-VM lane is the follow-up. * Point the llama assert at the right root, and name the Intel limitation The tauri leg installs to the legacy root because --tauri refuses a custom UNSLOTH_STUDIO_HOME. Its install succeeds end to end, but llama.cpp lives at <root>/llama.cpp while the venv is at <root>/studio, so the assert was pointed one level too deep. On macos-15-intel /usr/bin/git keeps working once the CLT are gone, so it is not CLT-provided there and no masking can remove it, while cc and clang do become stubs. Calling that 'masking failed' was wrong. That leg allowlists git explicitly and says why, so the assert stays strict everywhere else. * Make the clean-machine legs able to fail The toolchain strip never ran on the automatic triggers: inputs exists only for workflow_dispatch, and GitHub coerces '' and false alike to 0, so `inputs.strip_toolchain != false` was false. Confirmed on a pull_request run where the strip step reports skipped. Gate on the event instead. Also: scrub the Machine and User registry PATH, since install.ps1 rebuilds $env:Path from them mid-install and the toolchain came back; stop dropping WindowsApps unconditionally, which removed winget on the winget=visible leg too; fail rather than annotate when a bundle ships no installer or no CLI; run the bundled installer, which a headless launch never reaches; resolve the newest desktop-v* release instead of a pinned immutable tag; and give the two macOS matrix rows distinct artifact names. * Make the Windows and Linux clean-machine legs honest The Windows scrub only touched PATH, so the legs were green while not clean: run 30365014702 logged "python ABSENT" and then "Python 3.13 already installed" with uv resolving C:\hostedtoolcache\windows\Python\3.13.14\arm64\python.exe. py.exe lives in C:\Windows and uv discovers interpreters itself, so take the toolcache off disk and fail when tooling survives, instead of only printing it. The Linux desktop legs never stripped anything, and the tauri.log step was all || true so it could not fail. Run the bundled installer the way install.rs does, with --tauri alone, and assert torch: passing --no-torch skipped the slowest half of first launch and let the venv check pass over it. Pin the WSL rootfs to a dated build; current/ is a rolling alias and the digest next to it is fixed. * Give the Linux and WSL legs an assertion that can fail The Linux rows' only post-install gate was nobuild, a log grep, so an installer exiting 0 having produced nothing kept a required leg green. The WSL job and the Windows job both already check the install runs; the Linux job now does too. The WSL detection half only printed its Select-String, and the alternation also matches "platform linux", so a regression that skipped every WSL-specific branch would still pass as a plain-Linux install. Assert the exact marker, stripping ANSI first since step writes the label in reverse video. Probed against three fixtures: real wsl log passes, platform linux fails, missing log fails. * Tighten the clean-machine comments Compress the comment blocks across the clean-machine workflows and scripts. The explanations of why each check is written the way it is stay; the padding, restatement and duplication go. No code or workflow logic changes. * Point the nightly at the repo that publishes, and let its checks fail REL_REPO defaulted to unsloth-test/unsloth-test, which holds one release frozen at 2026-07-27, while release-desktop.yml publishes into github.repository. The schedule was re-testing the same fixture forever and could never see a broken production bundle. The windows job carried a blanket continue-on-error, so its NSIS assertions could not gate. lipo -archs prints and exits 0 for a thin binary and `|| true` swallowed even that, so the architecture was never checked; fall back to file, which survives the CLT mask. And require the preflight disposition line rather than the mere existence of tauri.log, which setup_logging creates at process start regardless. * Stop four clean-machine checks from passing over a real failure Re-run `absent` after the install on the masked macOS legs. It only ran before, so an installer that quietly selected the Xcode CLT or installed a compiler left the leg green while every later source build could succeed, which is the one thing clean-machine-assert.sh says `absent` guards the whole run against. Fail the Windows simulation when py.exe can still start an interpreter. The launcher binary itself may stay, but Find-CompatiblePython probes `py` first (install.ps1:1130-1153), so an interpreter registered outside the two renamed toolcache directories gets reused and Python bootstrap is never exercised. Exempting `py` without ever running it left that unchecked. Propagate the WSL installer exit code. It was printed and discarded, and the CLI check does not compensate: install.sh links the `unsloth` shim (4174-4182) before it reports a failing studio/setup.sh (4219-4230), so a late setup failure leaves a shim whose --version succeeds. Run the bundled installer in the Linux desktop jobs. The launch step only proves the process stayed alive, and on a fresh home preflight reports not_installed and the app waits on the install screen, so both required rows passed after 90 seconds without ever touching the shipped install.sh. Locate the resource in the deb payload or the extracted AppImage, run it the way install.rs does, and require a managed venv that can import torch. * Prove the trace wrapper records before trusting an empty trace The `notools` check reads an absence: it passes when the trace file contains no compiler, git or brew invocation. A shim directory that never reached PATH produces exactly the same empty file as an installer that touched nothing, so the single leg carrying that assertion would stay green no matter what the installer did. "Verify the simulation actually took effect" only ran for mask mode, which left the trace leg with nothing checking its own instrumentation. Call git explicitly after sourcing the environment and require it to appear in the trace, then truncate the file so the self-test entry does not count against the install. The call has to be explicit because macOS reaches _has_working_git only under STUDIO_LOCAL_INSTALL (install.sh:2026), so no consumer leg on that platform probes git on its own. * Stop the Windows clean-machine check failing on its own probe exit code All three Windows legs failed "Verify the simulation took effect" with no ::error:: printed at all. The check itself was right: the mask step logged "masked toolcache python: C:\hostedtoolcache\windows\Python", python/git/cmake/cl were ABSENT, no `py -3.x` probe started an interpreter, and the winget assertions were satisfied. The step still exited 1. The cause is $LASTEXITCODE leaking out of the step. The last external command is the `py -3.13` probe, which is SUPPOSED to fail; Get-Command and Write-Host are cmdlets and never reset $LASTEXITCODE, and the runner appends `if ((Test-Path -LiteralPath variable:\LASTEXITCODE)) { exit $LASTEXITCODE }` to every pwsh step (actions/runner#351). So a clean machine reported failure, and because this step runs before Install, no Windows leg has ever reached the installer. Clear $LASTEXITCODE after the probe loop and end with an explicit exit 0. The leak detection is untouched: a surviving python/git/cmake/cl, or a `py -3.x` that actually starts, still exits 1. Also print each probe's exit code and output, so the next failure here explains itself instead of being silent, and label `py -0p` as what it is. The launcher reads the registry, which the on-disk toolcache rename cannot rewrite, so -0p keeps naming paths that no longer exist. Unlabelled it reads like a leak. Accept the Fedora leg's real outcome instead of a message that can be absent The fedora assertion only accepted the unsupported-package-manager hard exit. That is still what this ref's install.sh does, but the pending installer change replaces it with a warning that lets the install continue, at which point the old grep matches nothing and the step fails for the wrong reason. Handle both, strictly. If the log shows the newer "using prebuilt llama.cpp (missing:" warning, the Linux gate demonstrably did not hard-stop, and the only tolerated failure past that point is release lag: install.sh comes from this ref while unsloth comes from PyPI, and the released studio/install_python_stack.py has no "skip triton kernels when git is missing" guard, so it still fetches the git+https triton_kernels requirement on a machine with no git. Anything else after that warning fails the step. Otherwise the old hard-exit message is still required. A missing log, a bootstrap outage or any unrecognised failure all remain errors, and the step retires to a plain success assertion once a release ships the no-git skip. * Make the AppImage Linux row actually extract, and hold Linux to the macOS preflight bar The appimage row invoked the extractor by bare filename, and a command word with no slash is resolved through PATH rather than the working directory, so the extraction exited 127 and the bundled-installer assertion below it never ran. Prefix it with ./ so the row exercises what it claims to. The Linux log step also asserted nothing: it skipped a missing log with continue and discarded the grep with || true. The launch step only proves the process stayed alive for 90 seconds, and the bundled-installer checks do not exercise the Rust preflight path, so an app that hung before preflight completed passed both required Linux rows. Require the same desktop_preflight completed disposition= record the macOS rows already do. * Put the branch's own Python under test on the clean-machine legs install.sh and install.ps1 come from the ref under test, but they install unsloth from PyPI, which is the consumer path and has to stay that way. That left everything Python-side coming out of the released wheel: studio/setup.sh, studio/setup.ps1, studio/install_python_stack.py, and every requirements and constraints file those resolve through Path(__file__). A branch that changes constraints.txt or setup.ps1 therefore got a green run that proved nothing about the change, and some legs proved less than they looked. The Fedora assertion was already carrying a hand-written workaround for exactly this, tolerating a triton/git failure on the grounds that the released package lags the ref. Legs marked overlay: true now re-point the venv at the ref just before studio setup runs, through UNSLOTH_CI_SOURCE_OVERLAY: a --no-deps editable install of the checkout. That makes import studio resolve to the working tree, so the existing setup-script lookup finds the ref's setup.sh / setup.ps1 and install_python_stack reads the ref's constraints, with no other change to either installer. Not --local: --local additionally installs unsloth-zoo from a git+https URL, which genuinely needs git, and git absence is the whole point of the masked legs. The overlay resolves no dependencies and clones nothing, so it holds up with git, cmake and the compilers all gone. It is not a consumer knob either: no flag, no usage entry, ignored unless the variable names a directory with a pyproject.toml in it. Four legs stay on the released package deliberately, each for its own reason, recorded in the header: the mac pipe legs keep an end-to-end signal on what a user actually runs; the trace leg would otherwise answer its own question, since the editable build calls git through setuptools-scm's file finder; the non-root Linux leg dies before a venv exists; and WSL only ever receives install.sh, not a source tree. Two supporting fixes the overlay depends on or exposes: install_python_stack.py discarded uv's output whenever a step succeeded, so the nobuild assertion, which reads the install log, could not see a source build in the dependency phase at all. That is the phase that installs studio.txt, where an sdist-only dependency actually turns up, and it reported "built: none" regardless. It now echoes successful output under UNSLOTH_VERBOSE, matching what install.sh's run_install_cmd already does. nobuild now ignores "Building <name> @ file://" lines. A local-path build is something the caller pointed at, never a dependency resolution chose, and index dependencies always print <name>==<version>, so a real sdist from PyPI is still caught, including one named unsloth. Each overlaid leg also asserts it really was overlaid, so an unset variable cannot quietly put the whole matrix back on the released wheel. * Allowlist the triton-kernels pure-Python sdist, and record why Windows on ARM is red The two ubuntu2404 root legs went red at "Assert no source build" reporting triton-kernels. That is not a regression in what the installer does. Those builds have always happened; they only became visible now that pip_install stopped discarding uv's output on success, which is what finally let the nobuild check read the dependency phase at all. So the question was whether each build actually needs a compiler. Checked against the real artifacts rather than assumed: openai-whisper 20250625, randomname 0.2.1, argbind 0.3.9 -- no version of any of the three has ever published a wheel; antlr4-python3-runtime is pinned at 4.9.3, below the first release that ships one. All four sdists use setuptools.build_meta, declare no ext_modules, and contain no .c/.cpp/.pyx/.rs file. Already allowlisted, correctly. triton-kernels is the same category and was the only name failing. It is pinned to the triton repo's python/triton_kernels subdirectory; that tree is 75 files of Python, a four-line pyproject.toml, no setup.py and no native source at all. The kernels are Triton DSL compiled at runtime, not at install time. It is also a direct URL the installer names itself rather than something resolution picked, and only Linux reaches it. It belongs in the allowlist, so add it with that reasoning written down. The allowlist match is now lowercased and underscore-folded on both sides. The requirement spells the package triton_kernels while uv prints triton-kernels, and an allowlist that matched only one spelling would pass by luck rather than by intent. A plain pyarrow sdist is still caught. The two data-designer @ file:// plugin builds needed nothing: they are in-tree local paths, already dropped by the same rule that exempts the source overlay's own build. Separately, the windows-11-arm leg fails for a real reason and should keep failing. The ARM handling itself works, the log shows torchaudio being skipped and torch plus torchvision installing from wheels. What stops it is that pyarrow and hf-transfer publish no win_arm64 wheel at all, so uv falls back to their sdists and they fail on CMake configure and on openssl-sys wanting perl. That is a product gap on the platform, not a gap in the simulation, so the leg stays experimental and keeps reporting it. Record that above the matrix entry so the next reader does not re-diagnose it. * Exercise the bundled Windows installer, and stop mislabelling installer sources Four things that let a leg go green while proving nothing. The desktop Windows job installed the bundle and launched it, and that was all. On a fresh profile preflight reports not_installed and the app sits on the install screen waiting for a click, so the process happily stays alive for 90 seconds without the bundled install.ps1 ever running. A bundle that shipped no install.ps1 resource, or a broken one, passed this job -- which is the packaged app failure the workflow exists to catch. macOS and Linux already invoke their bundled script directly; Windows now does the same, via the resource NSIS laid down next to the exe, invoked the way install.rs invokes it, then asserts the managed venv exists and can import torch. Its timeout goes to 60 minutes because a full torch install on a Windows runner is the slowest of the three. A manual run that selects installer_source: published only redirected the macOS and Linux jobs. WSL kept copying the checked-out install.sh and Windows kept running the checked-out install.ps1, so a run asking whether the script on unsloth.ai works reported on this ref under the published label. Both now honor the selection; install.ps1 advertises its own unsloth.ai URL, so published has a meaning on Windows too. Both branches stay empty on pull_request and push, so automatic runs are unchanged. The push-to-main filter listed only install.sh, install.ps1 and this workflow, while the PR filter also covers setup.sh, setup.ps1, install_python_stack.py and the clean-machine helpers. A direct push touching those skipped the workflow entirely, so the post-merge backstop never ran for the files the source overlay was added to cover. The two lists now match. Neither filter covered studio/backend/requirements, even though the overlay exists precisely so a constraints change is resolved on a machine with no compiler and no cached wheels. The update-smoke workflows cannot stand in: they start from a preinstalled Python and full developer tooling. * Make the Linux and Windows desktop legs clean, and honour published on every macOS delivery The desktop workflow claims all three platforms are stripped, but only macOS and Windows had a strip step and the Windows one scrubbed the process PATH only. Both gaps let a bundle that needs a developer toolchain pass the one workflow whose premise is that it must not. Linux: the job ignored strip_toolchain entirely and ran the bundled install.sh with the runner's git, gcc, cmake and make in /usr/bin. clean-machine-env.sh now has a Linux --remove branch that moves the resolved tool binaries aside, recorded in restore.sh, and the job calls it plus `assert absent` after the apt step (the .deb install needs dpkg) and before the bundled installer, with a restore step to match macOS. The loop repeats per tool so a name present in both /usr/bin and /usr/local/bin is fully masked rather than half masked. Windows: rewriting $env:PATH does not survive the bundled install.ps1, which calls Refresh-SessionPath (318-337) and rebuilds $env:Path from the Machine and User registry values, and py.exe in C:\Windows reaches the toolcache whatever PATH says. Ported the on-disk toolcache rename, the Machine/User registry scrub and the py -3.11/-3.12/-3.13 start probe from clean-machine-install-ci.yml, so the strip is proven rather than assumed. Windows preflight: the log step was Test-Path, Get-Content and Select-String, none of which can fail, so an app that hangs before preflight passed on the 90 second liveness check alone. It now asserts a tauri.log exists and carries a `desktop_preflight completed disposition=` line, the same unconstrained check macOS and Linux already make. The disposition VALUE is deliberately not constrained: ManagedReady over an unbootable venv is the reported bug. installer_source on macOS: only the pipe delivery branched on it, so a `published` dispatch ran the checked-out script on six of the eight macOS rows while the run was labelled published. The script is now resolved once at the top of the Install step and used by the file and tauri deliveries; pipe still re-fetches through the live transport, because that is half of what it tests. Linux, WSL and Windows already honoured the input. Also shortened the comments across the changed files, keeping the reasoning that says why each check exists. * Run the Windows installer under PowerShell 5.1, the only shell a clean machine has The Windows Install step ran `& $script` inside a `shell: pwsh` step, so install.ps1 was executing under PowerShell 7. A genuinely clean Windows box does not have PowerShell 7: Windows ships powershell.exe (Windows PowerShell 5.1) and pwsh is a separate install that the hosted runner image happens to preinstall. So the one workflow whose premise is a machine that has never seen a developer toolchain was testing the installer under a shell that machine would not have, and no other Windows job anywhere in .github exercises install.ps1 under 5.1. Invoke it the way the desktop does (install.rs:325-339, and the bundled installer step in desktop-app-clean-machine-ci.yml): powershell.exe with -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Bypass -File. The pwsh step wrapper stays, since it is only the installer that has to be under 5.1. Calling powershell.exe with `&` keeps the output in the pipeline, so Tee-Object still fills logs/install.log, and $LASTEXITCODE after the pipeline is the child's real exit code, so $rc and `exit $rc` are unchanged. install.ps1 and studio/setup.ps1 hold no PowerShell 7-only constructs: no `#Requires` above 5.1, no `&&`/`||` chain operators, no ternary, no null-coalescing, no ForEach-Object -Parallel, no $IsWindows/$PSStyle, and no 6+ cmdlets or parameters. setup.ps1 declares `#Requires -Version 5.1`, and its three $PSVersionTable branches gate a 7-only preference on the 7 side with a 5.1 fallback. Every Invoke-WebRequest already passes -UseBasicParsing, which 5.1 needs because it otherwise reaches for the IE engine. * Assert the Windows desktop strip actually took effect The desktop job's Windows masking renamed the toolcache Python, scrubbed the Machine and User registry PATH, and probed `py`, but nothing checked that `python`, `git`, `cmake` or `cl` were gone. The drop list is heuristic path fragment matching, so a runner image that moves any of those outside those fragments leaves the bundled install.ps1 reusing hosted developer tooling while the job still reports a clean machine. PATH written to $GITHUB_ENV only applies to later steps, so the check has to live in a step of its own; it carries the same event gate as the strip, exempts `py` (it lives in C:\Windows and stays, which is why the start probe is the real evidence), and resets $LASTEXITCODE before exiting 0 so an intentionally failing probe cannot fail a clean machine. Also correct the no-winget matrix note: that leg is not failing for an unfixed product reason. It stops at the unconditional git gate in setup.ps1 only on this ref, and with that gate relaxed it passes along with every other leg, so the row is a merge order dependency and stays required. * Resolve the desktop release including drafts, the convention this repo ships All three desktop legs died at the download step with an empty REL_TAG. The resolver passed --exclude-drafts while REL_REPO now defaults to github.repository, and every desktop-v* release in unslothai/unsloth is a draft: desktop-v0.1.50-beta and desktop-v0.1.471-beta are both drafts carrying the .dmg, .deb, .AppImage and setup.exe, while only the non-desktop tags like v0.1.501-beta are published. Excluding drafts therefore matched nothing and no leg could ever run against a production bundle. Drop --exclude-drafts so the newest desktop-v* release is found. A draft has no tag ref, so releases/tags/<tag> 404s for one, but gh resolves drafts over GraphQL and gh release download <tag> fetches their assets normally, so the download call is unchanged. Listing drafts requires push access, which for GITHUB_TOKEN means contents: write, so the workflow permission is raised from read and annotated. When nothing resolves the leg still fails hard rather than skipping: with no bundle to install there is nothing to prove, so a green run would be a lie. The error now names both causes, no release cut yet or a token that cannot see drafts. Also stop the restore step swallowing its own failure. `bash .clean-machine/restore.sh || true` printed "No such file or directory" whenever an earlier step failed before the toolchain was stripped, and hid a genuinely broken restore just the same. Skip explicitly when the file is absent and let a real restore failure surface. Same fix in clean-machine-install-ci.yml, which had the identical line. * Skip the desktop jobs on fork PRs instead of failing them Every desktop-v* release in this repo is a draft, and GitHub lists drafts only to a token with push access, which is why resolving one needs contents: write. A pull request from a fork receives a read-only token no matter what the workflow declares, so on those runs the resolver cannot see any release and the job died on "no desktop-v* release visible", accusing the repo of having no bundle when the real cause is the trigger. This workflow runs on pull_request for changes to itself and the stripping scripts, so an outside contributor editing either would have hit that. Guard the three jobs on the head repo not being a fork. A skipped job is honest here: it does not claim to have tested a bundle it was never able to download, and it is not reported as a pass. * Close the free headroom in the clean-machine simulation Assert arch and signature on every downloaded Mach-O. This is the one genuine gap the simulation had: Rosetta 2 is preinstalled on hosted runners and absent from a factory-fresh Mac, so an x86_64-only llama.cpp, whisper.cpp, Node or uv payload runs green here and dies with "bad CPU type in executable" for the user. llama-server launching under `assert-llama-loads.sh` does not rule that out, because Rosetta makes it launch. The new `macho` check reads `file -b` (`lipo` is an xcrun shim and is gone after masking, as the desktop lane already notes) and keys the expected arch off `uname -m`, so macos-15-intel expects x86_64. It also requires at least an ad-hoc signature on arm64, which closes the AMFI "Killed: 9" class that uv has already been bitten by; the check is skipped on x86_64, where unsigned code loads fine and so is not the same defect. It fails when the scan finds nothing, since an empty scan reads exactly like a clean one. Make absence real rather than PATH-hidden. uv probes well-known interpreter locations and the framework loader ignores PATH entirely, so hiding the toolcache only hid it from `command -v`. Empty /usr/local (it EXISTS on a factory-fresh Mac as a SIP-exempt firmlink, and is empty; it is /usr/local/bin that is absent, so the directory itself stays), move the hosted toolcache and /Library/Frameworks/Python.framework aside, and clear the developer dotdirs and caches. A populated uv or pip cache can also satisfy a resolution that would fail on a user's machine. Every removal goes through --remove and is recorded in the generated restore.sh, guarded so a path the install recreated is not buried inside its own restore. Unset CI, GITHUB_* and RUNNER_* for the installer process only. An installer branching on CI=true is a hidden dependency no consumer exercises. Scoped to the child so the step's own $GITHUB_OUTPUT still resolves. Record spctl --status and csrutil status. Neither is documented for these images and both change what a binary is allowed to do. * Pin the two failures no change here can fix, and add the virgin Windows container lane Three red checks, two of which test something this branch does not own. desktop linux deb / appimage run the SHIPPED bundle's own install.sh, and desktop-v0.1.50-beta was cut on 2026-07-21, before #7547 merged on 07-29. That bundle still carries the old optional-dependency gate, so on a stripped runner it exits 2 at [TAURI:NEED_SUDO] cmake git build-essential libcurl4-openssl-dev and never creates a venv. Current main's _check_linux_deps runs the same set through _SMART_APT_OPTIONAL, which suppresses every escalation path, so only a new release can change this. The step now pins that exact outcome: the exit code must be 2 and the log must carry exactly that package list, anything else still fails, and finding _SMART_APT_OPTIONAL in the extracted install.sh (the guard #7547 added) turns into a hard error saying to delete the pin. The venv and torch assertions stay and still run whenever the installer succeeds. win windows-11-arm gets a native ARM64 CPython, and torchaudio publishes no win_arm64 wheel at any version, so the PyTorch step cannot resolve. The fix is in install.ps1 on #7549, still open. Same treatment: the Install step is continue-on-error and a new step requires all three of the PyTorch step, the torchaudio resolution error and the missing win_arm64 platform tag, so any other failure is red. The row leaves experimental so the job is required, and the pin errors out as soon as the venv interpreter reports anything but win-arm64, which is what #7549 landing looks like. Adds the virgin Windows container lane as two jobs here rather than a sibling workflow: same premise as the win legs, same path filters, and masked-versus-real reads better side by side. The hosted Windows legs cannot test the VC++ 2015-2022 runtime (it ships in the runner image's System32) or a Windows with no Microsoft Store, and a servercore:ltsc2022 container on windows-2022 answers both. The probe asserts no python, py, git, cmake, cl, winget or uv on PATH, on disk or in the registry, and now also asserts vcruntime140.dll, vcruntime140_1.dll and msvcp140.dll are absent, which is the one thing the hosted runner cannot un-ship. Both container install rows stop at studio/setup.ps1's winget-only git gate on this branch, since #7549 is what relaxes it, so both are pinned the same way. The overlay row additionally requires the UNSLOTH_CI_SOURCE_OVERLAY hook to have fired, unconditionally: without that it would be indistinguishable from the released-wheel row, and the hook is this branch's own feature. Container notes carried over from the spike: never docker pull when the image is cached, since MCR has shipped an image ahead of the runner host before; wait for the Docker daemon, because one leg died in 21s on npipe:////./pipe/docker_engine and that flake misreads as "Windows containers unavailable"; drive docker from a run: step, because the job-level container: key is Linux-only. The root CA store is seeded after the virginity assertion, restoring what a real Windows already has, because studio/install_node_prebuilt.py downloads Node with bare urllib.request.urlopen and hits CERTIFICATE_VERIFY_FAILED against the empty container ROOT store. That product bug is left alone here. * Check signatures on Mach-O main executables only The macho check asserted a valid signature for every Mach-O under the studio home, and failed the macos-15 mask/pipe leg on 29 files: lxml, charset_normalizer, cygrpc, _upb, fontTools, caio, brotli and a bundled libportaudio.dylib. Those are MH_BUNDLE and MH_DYLIB images dlopen'd into a process without library validation, they ship unsigned in the wheels, and the same run had already installed and imported them with the installer exiting 0. Key the signature half off the Mach-O filetype and run it only on main executables. Report an absent seal separately from one that fails to verify, and capture codesign output instead of piping it into grep, which returned the unsigned exit status through pipefail and called every unsigned binary broken. The architecture half is unchanged and still a hard failure: it is what closes the Rosetta 2 gap. The zero-Mach-O guard is unchanged. The .venv_t5_* sidecars stay in scope; setup.sh creates them during a normal install and transformers_version.py puts them on sys.path, so they are payload. * Make the WSL job gate, assert Windows installed no toolchain, strip before the .deb * Assert the root Linux legs did not compile llama.cpp with the apt-installed toolchain * Pin the macOS desktop legs on the same pre-7547 release lag The Linux rows already pin the shipped bundle's own install.sh exiting 2 at the NEED_SUDO handshake. macos-15 and macos-26 fail the same way for the same reason: desktop-v0.1.50-beta predates #7547, so the bundled installer still hard-exits on the Xcode CLT gate that #7547 turned into a warning. Accept exit 1 plus that exact gate line, and nothing else. _check_macos_deps is the function #7547 added, so its presence in the bundle means the release caught up and the block errors out asking for the pin to be deleted. * Pin the WSL pipe truncation and the masked-winget git gate The WSL leg dies at install.sh:2082 with an unterminated quoted string. Nothing is wrong with that line: piping the script into sh is not atomic. dash reads it from the pipe in 8192-byte blocks and runs each command as it parses, and install.sh:2007 calls _maybe_reroute_strixhalo_to_2404, which on WSL alone shells out to Windows interop; interop relays the stdin it inherited and drains the pipe. dash has 11 blocks buffered at that point, ending at byte 90112, which falls inside "$STUDIO_LOCAL_INSTALL" on line 2082. Truncating install.sh at 90112 and parsing it reproduces the message verbatim, and running the whole file under a stdin-draining interop stub reproduces the exit code too. #7548 wraps the body in _unsloth_main so sh parses everything before running anything, and the same reproduction against its head is clean. The eight green staging runs cited when this job's continue-on-error came off were all on trees that already carried #7548, so that evidence never covered this branch. Pin the exact signature instead: exit 2 plus the shell's own unterminated-quoted-string error, with the _unsloth_main marker read back out of the distro as the flip condition. Pin winget=masked the same way. studio/setup.ps1:1655-1669 gates on git unconditionally and can only fetch it through winget, so masking winget leaves no way to satisfy it. #7549 relaxes the gate, and its wording appearing in the tree retires the pin. * Retire the WSL pipe pin now that #7548 is in main The pin flipped exactly as designed: it looks for _unsloth_main in the installer it actually ran, and #7548 put it there. Delete the pin and the CLI waiver, and assert the opposite instead. WSL is the only platform whose install shells out to Windows interop mid-script, and interop relays the stdin it inherited, so this job is the one that can catch the pipe being drained again. A truncation here is now a hard failure. * Gate the no-elevation Linux install and split off the no-transport case * Assert no source build on the hosted Windows legs and keep winget for the desktop lane * Retry the container root CA seeding instead of failing on one Windows Update timeout * Run the clean-machine workflow for the prebuilt installer helpers it overlays * Narrow the container pin to its own gates and scan uv and the venv interpreter for arch * Tighten the clean-machine comments * Re-assert toolchain absence after the desktop .deb pulls its dependencies * Retire the #7549 pins and add a wget-only Linux leg #7549 is in main, so the three known-outcome pins that were waiting on it are stale and would now hard-error by design. Each is replaced by the assertion it was standing in for rather than deleted: win windows-11-arm now gates. The x64-on-ARM64 resolver is asserted as an outcome: the venv interpreter reports win-amd64 from its own sysconfig, and torchaudio (no win_arm64 wheel at any version) is installed. Measured on the integration branch before #7549 merged: "only a native ARM64 Python 3.13 was found" -> "installing x64 Python" -> torchaudio 2.10.0+cpu, install green. win windows-latest / winget=masked now gates. The relaxed git gate is asserted from both sides: the old unconditional message must be absent, the no-git branch must have been reached (so the row cannot pass because git leaked back onto PATH), and setup.ps1 must report git as absent-but-not-required. Both Windows rows, and the visible one, gained the usability check the Linux legs have had and Windows never did: a managed interpreter, an unsloth CLI on disk, and that CLI actually running. nobuild and the toolchain check only read the log, so an installer that exited 0 having produced nothing satisfied them. The torch assert also loses its fallback to whatever `python` resolves to. The virgin container overlay row gates, and asserts what only that lane can: it is the one environment whose System32 does not already ship the VC++ 2015-2022 runtime, so it is the only place Ensure-VCRedist's direct aka.ms download can be proved to run rather than be short-circuited. The overlay=false row keeps a pin, with a new reason: it installs unsloth from PyPI on purpose, and setup.ps1 inside 2026.7.5 (uploaded the 23rd) predates #7549, so it still stops at the old gate. That is release lag, it flips on the next release, and the pinned signature is now the old wording rather than "#7549 has not landed". Also adds linux ubuntu2404-nonroot-wget. install.sh's download() takes curl or wget and _transport_missing is true only when both are gone, so a wget-only box is supported on paper, but the gating nonroot leg provisions ca-certificates AND curl, so curl won every probe and the wget branch had never run. Same image, same no-sudo user, same asserts, wget instead of curl, and curl proved absent on disk for root and for tester before AND after the install, so the claim is that every download went through wget rather than that curl happened to be unused. * Tighten the clean-machine CI comments Comments only, no assertion logic, pins or leg definitions touched. Reflowed every rationale block to denser wording and removed the duplication that had built up across repeated steps: the desktop workflow repeated the fork-PR skip, the desktop-v* tag resolution and the restore-runner note once per platform, and the installer workflow repeated its path-filter rationale in both the pull_request and push blocks. Those now point at the first copy. Every WHY is kept: why the masked legs avoid install.sh --local, what UNSLOTH_CI_SOURCE_OVERLAY is for, why `absent` tests "must not work" rather than command -v, why the .venv_t5_* sidecars are in the macho scan scope, why the signature check is main-executables-only, why each nobuild allowlist entry is a pure-Python sdist, why the WSL job gates and what the pipe truncation was, and why the virgin container's overlay=false row is still pinned. Proved comments-only three ways: both workflow revisions parsed with yaml.safe_load_all and every leaf walked (only `run:` scalars differ); every changed bash body and .sh compared byte-for-byte after `bash --pretty-print -n`; every changed pwsh body and .ps1 compared as a token stream with Comment and NewLine tokens dropped. A negative control injecting one non-comment line into each layer makes all of them fail. * Clean machine CI: strip Strawberry, make the Fedora pin gating, run the Linux CLI desktop windows failed the strip verification because windows-latest ships a MinGW toolchain under C:\Strawberry\c\bin, which matches none of the drop fragments; the installer workflow already scrubs it. Fedora sat behind job-level continue-on-error, so its outcome pin could not fail the run. Tolerate the install step instead, as the no-transport row does. The Linux usable-install check only tested the executable bit; Windows and WSL already execute the CLI. The macho scan now fails when no venv interpreter was scanned, rather than letting uv alone satisfy the outside-root guard. * Clean machine CI: tighten the comments Round 12 comment reduction: compress wording, keep every reason. Comments only, verified with a YAML leaf walk (differences only inside run: scalars, only on # lines), bash --pretty-print -n byte comparison, a PowerShell token-stream diff and a Python AST comparison. * Clean machine CI: dereference the venv interpreter, pin the deb deps and the Windows disposition file did not follow the <venv>/bin/python symlink find -L printed, so it answered 'symbolic link to ...' and the Mach-O test dropped the one interpreter the Rosetta scan exists to check. Read with file -Lb and count what was classified, not what was found. apt treats a toolchain package the strip only renamed as already installed, so a .deb that started declaring git or cmake would never restore it and the absent re-check would still pass. Assert the declared Depends instead. The Windows lane accepted any preflight disposition although the bundled installer was already required to build a working venv; NotInstalled or ManagedStale there means the app cannot boot what it just installed. * Clean machine CI: assert every masked tool, and re-select the developer dir last clean-machine-env.sh moves ten tools aside and only warns when a move fails, but absent checked four of them, so a surviving gcc -- which install.sh probes for build-essential -- went unnoticed. restore.sh ran xcode-select --switch before the line that moved CommandLineTools back, so it named a still-masked directory, failed into || true and left the selection link unrestored. Capture the original selection and re-apply it after both directory restores. --------- Co-authored-by: danielhanchen <unslothai@gmail.com>	2026-07-29 06:21:40 -07:00
Nilay	52609fb890	Studio: reset-password rotates the credential in place instead of deleting auth.db (#7573) ... * reset-password: rotate the admin credential in place instead of deleting auth.db * reset-password: fix the CI callers and error handling for the in-place rotation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * reset-password: narrow the CI change to the jobs that read .bootstrap_password * reset-password: stop over-claiming what the reset revokes and when it takes effect * auth: bind token issuance to the credential version that was verified * auth: bind credential-creating writes to the version the request authenticated with * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * auth: bind the change-password and workflow-key writes to their own credential version * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * auth: read the credential version inside the transaction that validated it * data-recipe: answer 401 when a reset revokes the credential mid job start * Fix lint blocker and Windows path assertion for PR #7573 Drop the now-unused validate_api_key import from studio/backend/auth/authentication.py. Every call site moved to validate_api_key_with_credential, so the Source lint job's import-hoist gate flagged it as a blocker. The wrapper itself stays in storage.py; test_api_key_expiry.py still exercises it. Make test_run_reexec_forwards_resolved_frontend_on_public_launch compare against str(Path(...)) instead of a POSIX literal. _find_frontend_dist returns a Path, so on Windows the forwarded value is \fake\studio\frontend\dist and the assertion could never pass there. Pre-existing, surfaced by running unsloth_cli/tests on Windows. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-07-29 01:40:12 -07:00
Michael Han	2916e84499	Studio: clarify tool permission controls (#7181)	2026-07-20 09:55:39 -03:00
Daniel Han	03590f696e	Give opencode real timeout headroom in Local Agent Guides CI (#7235) ... * Raise the opencode invoke timeout in Local Agent Guides CI The connection (opencode) cell flakes with a 600s timeout reported as guide drift, but it is not a hang: in a passing run the same opencode run finishes in ~482s (08:12:31 to 08:20:33), right against the shared AGENT_INVOKE_TIMEOUT of 600s, so about one run in six drifts past the cap. opencode is the slow outlier. The print-mode agents (claude -p, codex exec) run one turn against a minimal injected system prompt, while opencode run runs its own full turn with opencode's large system prompt plus a separate small_model call to name the session (start.py pins small_model to the same 4B the server hosts). On a CPU-served gemma-4-E4B that is about 8 minutes, leaving no margin under 600s. Double opencode's per-invoke timeout in agent-guides-drive.sh and keep the tight 600s cap for the fast agents, so a genuine headless-TTY hang still fails quickly. 1200s stays well under the 40-minute job budget. * Normalize the agent invoke timeout before doubling it for opencode Strip an optional trailing 's' from AGENT_INVOKE_TIMEOUT so the opencode arithmetic, and the "${TIMEOUT}s" timeout message, stay valid if a timeout(1)-style suffix is ever configured. * Only double the opencode timeout for a bare-integer seconds value Guard the arithmetic so a GNU timeout(1) duration suffix (s/m/h/d, including floats like 0.5s) is passed through unchanged instead of breaking the expansion; timeout(1) parses those directly. Bare seconds still double. --------- Co-authored-by: danielhanchen <unslothai@gmail.com>	2026-07-19 06:08:54 -07:00
Michael Han	6d8c18cd1a	Replace standalone Studio wording with Unsloth (#7221) ... * Replace standalone Studio wording with Unsloth Replace the single word Studio with Unsloth wherever it is used as shorthand for Unsloth Studio in docs, CLI output, UI strings, i18n locales, workflow display names, comments and docstrings. Kept unchanged: the full name Unsloth Studio, third party product names (LM Studio, Visual Studio, Mac Studio), feature names (Recipe Studio, Fine-tuning Studio and its translations), and all identifiers such as env vars, commands, paths and filenames. * Address review feedback on the Studio wording rename Use "an" before Unsloth where the rename left the article as "a". Restore the split brand where Unsloth and Studio render as two halves of the full product name: the onboarding sidebar subtitle and the IPv6 localhost warning. Scope two messages to the full name Unsloth Studio where plain Unsloth was misleading: the AMD README bullet and the CLI studio setup error.	2026-07-19 00:47:04 -07:00
Daniel Han	c1e06e9ddf	unsloth start: add --persist to keep and reopen agent sessions (#7014) ... * unsloth start: add --resume to persist and reopen agent sessions `unsloth start <agent>` launches a coding agent whose home is a throwaway temp dir wiped on exit, so codex/openclaw/hermes/pi (which relocate their whole home there) cannot resume a conversation after you quit. opencode and claude keep their session data in a fixed user dir, so they already resume. Add an opt-in --resume/--no-resume flag: it routes the launch to the stable Unsloth agents dir (the same one --no-launch already uses) so the session survives the exit, never touching the user's own ~/.<agent>. A bare --resume also reopens the last conversation via the agent's native flag (codex `resume --last`, opencode/claude/pi `--continue`). The default is unchanged: a plain launch still uses a temp dir and persists nothing. Add a dispatch-only `resume` job to the Local Agent Guides CI that drives the real launch path and asserts the split: codex/pi are wiped without --resume and persist with it, while opencode/claude persist either way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * unsloth start: rename --resume to --persist The session flag collided with agents' own resume flags. `unsloth start claude --resume <id>` used to forward `--resume <id>` straight to Claude (which keeps its history in ~/.claude regardless), so a boolean --resume on unsloth start would have swallowed the session id and turned it into a stray prompt. Name the persistence flag --persist instead, so every agent's native resume flag (claude --resume <id>, codex resume, opencode --continue, ...) still passes through untouched. Behavior is otherwise identical: --persist keeps a launched agent's session under the Unsloth agents dir, and a bare --persist reopens the last conversation. Add a regression test that `--resume <id>` passes through verbatim, and in the CI resume experiment skip the redundant second pass for opencode/claude (they persist either way, and a second CPU turn only risks a timeout). * unsloth start: correct --persist help and drop the buggy auto-resume Reword the --persist help to be accurate: claude and opencode keep sessions in the user's own stores and resume regardless, so --persist only stabilizes the otherwise-ephemeral relocated home of codex/openclaw/hermes/pi. Drop the bare-launch auto-append of native resume tokens: it errored on a first launch with no prior session, and was inconsistent between launch and no-launch. --persist now only keeps the session dir; resume via the agent's own command (e.g. `unsloth start codex --persist resume`), which now finds it. In the CI resume experiment, fail the pass when the launched turn exits non-zero, so a write-then-error is not misread as PERSISTED. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-09 11:47:59 +02:00
Lee Jackson	6ef0936180	Fix OpenClaw start default to local TUI (#6937) ... * fix: launch OpenClaw local TUI by default * Fix/adjust OpenClaw launch paths for PR #6937 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Default OpenClaw to the local TUI only on a bare invocation The first-arg startswith('-') branch rewrote passthrough globals into a broken command: OpenClaw's grammar is openclaw [--dev] [--profile <name>] <command>, so 'unsloth start openclaw --profile test' became 'openclaw tui --local --profile test', but tui does not accept --profile (or --dev), so the invocation failed. A leading '--flag value' is ambiguous between a global (--profile test) and a tui option (--message hi), so it cannot be reinterpreted safely. Default to the local TUI only when no passthrough args are given, and forward everything else verbatim so OpenClaw parses it under its own grammar. The bare-launch default (the point of this change) is preserved; explicit subcommands and global flags pass through. --------- Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com> Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>	2026-07-08 04:25:42 -07:00
Daniel Han	69f8e0b228	Clear stale yolo approval state on no-launch reruns (#6868) ... * Clear stale yolo approval state on no-launch reruns The no-launch session config dir is deliberately reused across runs, but the config writers only ever added the --yolo auto-approval settings and never removed them. After one --yolo --no-launch run, every later run without --yolo kept OpenClaw's tools.exec security=full/ask=off policy plus exec-approvals.json, and OpenCode's permission allow block, so tool execution stayed silently pre-approved. Non-yolo runs now reset that state: OpenClaw drops the exec policy keys and the yolo defaults in exec-approvals.json (approvals OpenClaw itself recorded are kept; the file is removed when only the yolo payload is left), and OpenCode drops the permission block. Launch mode is untouched since it already uses an ephemeral temp dir. * Strip only yolo-written values on non-yolo cleanup Match each field against the exact value the yolo path writes before removing it, so a stricter exec policy, approvals defaults set by the user or the OpenClaw UI, and deny/ask OpenCode permission entries all survive a plain no-launch rerun. An unparseable exec-approvals.json is left in place, matching how an unparseable config is handled. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Write a prompting policy on non-yolo instead of deleting to a permissive default OpenClaw and OpenCode both treat an omitted policy as permissive: OpenClaw's effective exec policy for an unset tools.exec is security=full/ask=off on the gateway host, and OpenCode defaults an unset permission to allow. So clearing the yolo values on a non-yolo run did not restore prompting, it fell back to those permissive defaults and left tool execution auto-approved. A non-yolo run now writes an explicit prompting policy: OpenClaw gets security=allowlist/ask=on-miss (verified to prompt even with the approvals file removed, since the stricter of config and approvals wins), and OpenCode gets edit/bash/webfetch=ask. Only a permissive/yolo value is tightened; a stricter deny (or an ask the user set) is preserved, and the yolo approvals defaults are still stripped. The file-edit CI path opts opencode/openclaw into --yolo, since those agents now prompt by default and the headless test needs auto-approval. * Respect existing exec mode, sandbox/node host, and global permission rules on non-yolo reset The non-yolo reset for openclaw/opencode assumed an omitted policy was the permissive yolo default and rewrote it, which corrupted or weakened stricter setups it should have preserved: - OpenClaw tools.exec.mode is the normalized policy knob and cannot be combined with explicit security/ask (OpenClaw rejects the whole config), so writing security+ask alongside a mode:deny/ask policy both broke the config and relaxed it. Leave a mode-based policy untouched. - host=sandbox defaults to security=deny and host=node routes to a paired node; neither is written by --yolo (which only writes host=gateway). Treating the missing security as full and popping host broadened those into gateway/auto exec. Only rewrite a gateway-routed permissive policy, and never pop a non-gateway host. - OpenCode permission can be a string ("deny") or a {"*": ...} catch-all. The old code dropped a string form and overrode a catch-all by writing per-tool ask, weakening a stricter user rule. Now a string is left in place, a catch-all governs absent tools, and only an effective allow is tightened. - The non-yolo ask policy only lived in OPENCODE_CONFIG, which loads below project opencode.json, so a project config allowing edit/bash/webfetch still auto-approved. Carry the ask policy in OPENCODE_CONFIG_CONTENT (above project config) too, symmetric to how yolo carries its allow. Also harden the openclaw path against a malformed non-dict tools value. Adds tests for mode/sandbox/node hosts, string and catch-all permissions, and the inline ask policy over a project config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope non-yolo resets to the exact yolo fingerprint and preserve granular denies OpenClaw: reset only the exact host=gateway/security=full/ask=off policy --yolo writes, so an omitted or host=auto/sandbox/node policy (which can resolve to a sandbox security=deny default) is no longer broadened to allowlist/on-miss, and a deliberate tools.exec.mode is left alone (OpenClaw never migrates our security/ask write into a mode). OpenCode: carry a granular object or a deny inline verbatim so a per-tool user rule is not collapsed to a blanket ask, but floor any object that grants allow anywhere to the string ask (which fully replaces a project object) so no inline allow pattern can leak through into a silent auto-approve on a non-yolo session. * Stop overriding project config on non-yolo; require full approvals fingerprint The non-yolo OpenCode reset carried a session permission in OPENCODE_CONFIG_CONTENT, which outranks the project opencode.json we cannot read. That inline override could not correctly reflect the project: it weakened a project deny to a prompt, mishandled global string rules, leaked through a granular object's permissive default when no catch-all was present, collapsed an object with an allow (losing its deny), and missed per-agent permissions. All of these stem from forcing a value over an unknown project config. A non-yolo run now only undoes what --yolo wrote: it flips our own explicit per-tool allow back to ask in our config file and carries no permission inline, so the project's own permissions are honored as written. Clearing our persisted yolo state is the actual fix; --yolo still carries its allow inline so it works over a project config. OpenClaw approvals cleanup now strips the yolo defaults only when the full fingerprint (security=full, ask=off, askFallback=full) is present, so a mixed user policy that merely shares askFallback=full (whose omitted default is deny) is kept intact. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-07 00:06:48 -07:00
Nilay	b8400f40df	CLI: Rename unsloth connect to unsloth start (#6613) ... * replaced connect with start * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * Studio: build the coding-agent command from the selected server The API keys panel showed a hardcoded `unsloth start claude`. `unsloth start` defaults to 127.0.0.1:8888 and only mints a key for a loopback server, so a non-default port or a tunnel/remote base would target the wrong server or fail to mint. Build the command from the panel base/key (and emit a key for non-loopback), matching the other snippets in the panel. * CLI: keep `unsloth connect` as a hidden alias for `unsloth start` Avoids breaking existing scripts and docs that still call `unsloth connect`. * Tests: stub _unstarted_cleanup in same-task disconnect test The test builds _SameTaskStreamingResponse via __new__, so set the attribute that __call__ now reads. * Match coding-agent command loopback check to the CLI 127.0.0.0/8 rule (#6613) * Keep unsloth_cli.commands.connect importable as a deprecated shim (#6613) * Format the new coding-agents panel strings and import per biome (#6613) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop the unsloth connect alias and shim; unsloth start is the only command (#6613) * Route unsloth connect to unsloth start as a hidden backward-compatible alias (#6613) * Forward unsloth run model-load flags to unsloth start (gguf-variant, context-length, load-in-4bit, tensor-parallel) (#6613) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Session-scope coding agent config in unsloth start Configure each agent for the current session instead of writing the Studio endpoint, key, and default model into the user's own config. Codex, OpenCode, OpenClaw, and Hermes get a private config relocated through their config-path env vars (CODEX_HOME, OPENCODE_CONFIG overlay, OPENCLAW_CONFIG_PATH plus OPENCLAW_STATE_DIR, HERMES_HOME). Claude Code suppresses the attribution header for the session via the CLAUDE_CODE_ATTRIBUTION_HEADER env var plus a --settings overlay, with no ~/.claude write. --launch uses an ephemeral temp dir removed after the agent exits; --no-launch uses a stable Unsloth-owned dir and prints the matching export lines. * Read relocated agent session config in Local Agent Guides CI The contract crosscheck and the openclaw/hermes patch helpers now read each agent's config from the relocated path printed by unsloth start --no-launch (CODEX_HOME, OPENCODE_CONFIG, OPENCLAW_CONFIG_PATH, HERMES_HOME) instead of fixed home paths. The Claude attribution A/B toggles the header for the session only (shipped-config HIT vs vanilla MISS) instead of editing ~/.claude/settings.json. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Skip the POSIX-only --no-launch parser test on Windows test_no_launch_output_is_parseable mirrors the #6547 bash CI parser, which greps export/unset lines and only runs on Linux/macOS runners. On Windows --no-launch prints PowerShell ($env: / Remove-Item), so the export-line assertion does not apply there. Cross-OS staging CI surfaced this. * Size Claude Code's auto-compact window to the loaded model's context Claude Code auto-compacts against its native (~600k token) window, so against a smaller local model it overflows the server's context (silent truncation) long before it compacts. Set CLAUDE_CODE_AUTO_COMPACT_WINDOW to the loaded model's real context length (the value codex/openclaw already get via model_context_window / contextWindow). Omitted when the model reports no context length. * Pin OpenCode/Hermes context window and set 90% compaction across agents Feed every agent the server-determined sequence length (the value /v1/models reports from runtime_context_length) and a ~90% compaction threshold. OpenCode: a custom-provider model with no limit defaults to context 0, which silently disables auto-compaction, so set limit.context/output and scale the compaction buffer to 10% of the window. Hermes: pin model.context_length (it otherwise falls back to a 256k default when the server's /v1/models omits the field) and set compression.threshold 0.9. Claude: add CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=90 alongside the window. Codex (model_context_window) and OpenClaw (contextWindow) already carried the window and auto-manage off it. * Add `unsloth start pi` recipe Pi was the only agent without a built-in recipe, so the agent-guides CI hand-wrote ~/.pi/agent/models.json. Add a first-class `pi` command mirroring the others: - write_pi_config writes the session-scoped OpenAI-compatible provider config (key in the config, like openclaw/opencode). - pi() launches `pi --provider unsloth --model <id>` (Pi defaults to the google provider, so the provider/model are pinned on the command line) with HOME relocated for the session. Pi has no config-dir env var and resolves ~/.pi off $HOME, so HOME-scoping keeps the user's ~/.pi untouched. Migrate the agent-guides CI off the hand-written config onto the `unsloth start pi --no-launch` path (connection + file-edit), with a crosscheck for the provider api, so the documented recipe is exercised. * Harden unsloth start for Windows and WSL agent launches Address the Codex review on PR 6613: - write_pi_config now pins the loaded contextWindow and a sane maxTokens so Pi compacts instead of overflowing a small Studio context (it otherwise assumes its 128000 default), matching the other agents. - pi() sets USERPROFILE (and HOMEDRIVE/HOMEPATH when present) alongside HOME on native Windows, where Node resolves ~/.pi via USERPROFILE rather than HOME, so the session no longer reads or writes the user's real ~/.pi. - The WSLENV bridge flags path-valued vars with /p so a Windows npm shim under /mnt receives translated paths, while scalar vars (the numeric context window) pass through untranslated. WSLENV is deduped on the bare name. - _print_env prints the launch command with PowerShell-safe quoting so the inline --settings JSON survives copy-paste on native Windows --no-launch. Add tests for the WSLENV path flagging, PowerShell quoting, the Pi context window, and the Pi USERPROFILE relocation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Set CLAUDE_CODE_NO_FLICKER for the Claude session A local server streams in bursts, so Claude Code's full-screen TUI redraw flickers between tokens. Disable it for the session via CLAUDE_CODE_NO_FLICKER, alongside the other CLAUDE_CODE_* session env knobs. * Add a normalized --yolo flag routed to each agent's auto-approve mode It is easy to forget which agent spells "run tools without prompting" which way, so `unsloth start` now accepts all three spellings as one option (--yolo, --dangerously-skip-permissions, --dangerously-bypass-approvals-and-sandbox) and routes to the agent's own mechanism: - claude: --dangerously-skip-permissions - codex: --dangerously-bypass-approvals-and-sandbox - hermes: --yolo - pi: --approve (Pi's only approval gate is project trust) - opencode: a permission allow block in opencode.json (no CLI flag exists) - openclaw: tools.exec security=full / ask=off / host=gateway (no CLI flag exists) Because the option is parsed by `unsloth start`, the "wrong" spelling for an agent still routes correctly instead of leaking through to the agent and erroring. IS_SANDBOX is deliberately left unset for Claude so its root/sandbox safety gate still applies. Adds routing, cross-routing, and per-config tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix review findings: IPv6 loopback command, pi USERPROFILE under WSL, yolo guard From a 10-reviewer pass over the PR: - studio/frontend agent-command.ts: normalize bracketed IPv6 hosts. URL.hostname returns "[::1]" for http://[::1]:8888, which never matched the "::1" loopback checks, so the copied command embedded the placeholder API key for a local IPv6 server instead of the bare auto-minting command. Now [::1] is treated as loopback like the CLI's is_loopback_url, so the command matches the CLI contract. - pi(): also relocate USERPROFILE (and HOMEDRIVE/HOMEPATH) when running under WSL against a /mnt Windows shim, not just on native Windows. Windows Node resolves ~/.pi via USERPROFILE, and the WSLENV bridge translates the path, so pi no longer falls back to the user's real ~/.pi in that case. - _yolo_command_flags: use .get so a config-based agent (or a typo) yields no flag instead of a latent KeyError. Adds tests for the WSL pi USERPROFILE relocation, the yolo unmapped-agent guard, and that opencode/openclaw --yolo stays config-only (no argv flag). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix round-2 review findings: WSLENV /p upgrade, agent help text - _merge_wslenv now upgrades a user's pre-existing unflagged WSLENV entry (e.g. a bare HOME or USERPROFILE) to the path-translated form (HOME/p) instead of leaving it as-is, so a Windows agent shim under WSL receives the translated session path rather than the raw Linux path. - Generalize the `unsloth start` registration help to list all six agents (was only "Claude Code, Codex"). Adds a test for the WSLENV unflagged-entry upgrade. * Fix round-3 review findings: complete openclaw --yolo, refresh stale copy - openclaw --yolo now also writes the host approvals file (exec-approvals.json with defaults security=full / ask=off / askFallback=full) alongside the tools.exec config. OpenClaw gates tool execution on both layers (the stricter wins), so the config alone could still leave it prompting or denying. Mirrors `openclaw exec-policy preset yolo`. ask=off means nothing is ever prompted, so the runtime socket block is unnecessary. - Studio API panel copy: clarify that a local server auto-mints the key while a remote one embeds it in the command, and add pi to the swap hint. - Local Agent Guides CI: drop the stale "pi has no start.py recipe" note now that all six agents are driven via `unsloth start <agent> --no-launch`. Adds the openclaw approvals-file assertions and a no-yolo openclaw test. * start: parse claude --version with a regex so a format change does not drop optimization flags * start: offer to install a missing agent (prompt then run its install command) * start: auto-start a Studio server for --model when none is running, and stop it on exit * inference: surface an actionable message when llama-server cannot compile a tool grammar * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix review findings: kill the auto-started server tree on Windows; apply the tool-grammar message to the OpenAI passthrough too * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * start: split --model org/repo:variant so a running session is not evicted `unsloth start <agent> --model org/repo:QUANT` failed against an already-running Studio server and, worse, killed whatever model another session had loaded. /v1/models lists a loaded GGUF under its bare repo id (e.g. unsloth/Qwen3-1.7B-GGUF), so _resolve_model never matched the `:QUANT`-suffixed request. It then POSTed /api/inference/load with model_path=org/repo:QUANT, which (a) Hugging Face rejects ("Repo id must use alphanumeric chars, '-', '_' or '.'") and (b) evicts the model the other session was using, so a second 'unsloth start' in a new tmux/terminal tore down the first. Re-running the command then attached to the now-empty server, which is why it 'worked the second time'. Mirror the org/repo:QUANT -> org/repo + --gguf-variant QUANT shorthand that 'unsloth run' and llama.cpp already accept, splitting it in _connect before we match or serve. Matching now resolves against the loaded bare repo id (no spurious reload, no eviction), and any real load uses a valid repo id plus gguf_variant. An explicit --gguf-variant still wins; local paths and Windows drive letters pass through untouched. The auto-serve path likewise spawns 'unsloth run --model org/repo --gguf-variant QUANT'. * start: harden auth-key handling, codex teardown, and CI transcript redaction Three review findings: 1. CI could leak a live key. agent-guides-drive.sh printed the raw 'unsloth start --no-launch' transcript (which carries export UNSLOTH_API_KEY / ANTHROPIC_AUTH_TOKEN lines) to the Actions log on both the failure path and the success path before redact() ran. Add cat_redacted() and use it for those two prints, so the key is scrubbed on the way to the log while the on-disk file stays intact for the env parsing that follows. 2. Outages masqueraded as bad keys. _key_accepted caught a broad Exception and returned False, so a 5xx or timeout while checking a cached key looked like a rejection: it discarded a good key and minted extra ones (local) or reported 'no saved key' (remote). Only treat HTTP 401/403 as a rejection; let other errors propagate so a real outage surfaces. 3. Codex preflight could leave the auto-started server up. _require_gguf_for_codex runs after _connect may have auto-started Studio but before _run installs its teardown finally, so a preflight rejection (e.g. a transformers-backend model) left the server holding the port/GPU until the atexit backstop. Tear it down explicitly at the point of failure. Tests: a 5xx on a saved key surfaces without minting; a non-GGUF codex preflight tears down the auto-served server. * start: fix IPv6/portless studio URLs, Pi config-dir isolation, and Pi install recipe Four review findings: 1. Pi ignored the session config when PI_CODING_AGENT_DIR was already set. Pi's getAgentDir() reads process.env.PI_CODING_AGENT_DIR before falling back to $HOME/.pi/agent, so a value inherited from the user's shell sent Pi to their real config and skipped our provider/key (the HOME relocation alone was not enough). Pin PI_CODING_AGENT_DIR at the session's .pi/agent dir; it is path-valued so the WSL bridge translates it automatically. 2. Pi install hint dropped Pi's documented --ignore-scripts. Pi's README installs with 'npm install -g --ignore-scripts @earendil-works/pi-coding-agent' and notes it needs no install scripts, so accepting the prompt now follows that safe recipe. 3. Auto-start ignored a portless UNSLOTH_STUDIO_URL. unsloth run binds to 'parsed.port or 8888', so http://127.0.0.1 launched the child on 8888 but the health poll (and the returned base) still used port 80, stalling until the startup timeout. Normalize the base to host:8888 (IPv6-safe) before starting and polling. 4. API-panel command mistook IPv6 loopback for the bare default. The bare 'unsloth start' only probes 127.0.0.1:8888 on the IPv4 stack, so http://[::1]:8888 must carry an explicit UNSLOTH_STUDIO_URL. Drop ::1 from the bare-default host set while keeping it a loopback host (URL emitted, no key needed). Tests: PI_CODING_AGENT_DIR is set to the session dir; _effective_base normalizes portless/IPv6 bases; a portless UNSLOTH_STUDIO_URL auto-serves on :8888. * start: apply fresh-review findings across CLI, CI, and the API-panel command From a fresh multi-reviewer pass over the merged head plus the latest Codex bot review: 1. Load knobs now always consult the server. _resolve_model matched on model id alone, so --gguf-variant / --context-length / --no-load-in-4bit / --tensor-parallel were silently ignored whenever the id was already loaded (asking for UD-Q4_K_XL kept a Q8_0 serving). With any explicit knob the CLI defers to /api/inference/load, whose already-loaded dedup answers without reloading when variant and settings match, so a second session running the same command still attaches without evicting the first. 2. OpenCode --yolo and the session model pin now ride in OPENCODE_CONFIG_CONTENT. A project's own opencode.json outranks OPENCODE_CONFIG, so a repo config could silently override the session model and the --yolo permission block; OPENCODE_CONFIG_CONTENT outranks project config. The API key stays in the private file, never in printed env. 3. The --no-launch recipe's last line is a self-contained one-liner (inline VAR=value assignments before the command, conflicting vars blanked). People copy just the last line, and a bare codex/claude there ran against the user's real ~/.codex or Anthropic credentials with zero isolation, e.g. inheriting a pre-existing damaged ~/.codex state DB and blaming the recipe. The CI drive script scrubs the key from the one 'invoking:' echo this adds. 4. The auto-serve log is 0600 and the parent handle is closed. It sat world-readable in the shared tempdir under a predictable name while carrying the minted sk-unsloth- key from the unsloth run banner. 5. _key_accepted fails with a clean message on outages. Non-auth errors (5xx, network, timeout) surfaced as a raw traceback; 401/403 still mean a rejected key. 6. _effective_base strips URL paths, and https loopback targets never auto-serve. http://127.0.0.1:8888/studio polled /studio/api/health (404) and https://127.0.0.1 polled the wrong scheme, both spinning until the 15-minute startup timeout. 7. API-panel command: only literal 127.0.0.1:8888 earns the bare command. localhost can resolve to ::1, which the bare CLI never probes, so it keeps UNSLOTH_STUDIO_URL. 8. CI artifact sweep covers redacted-configs/ and agent-workdir/, not just logs/. Tests: 125 CLI tests pass (new coverage for each fix), 156 backend tests pass, ruff clean. Adds an unsloth connect alias regression test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * start: hand Pi a clean screen at launch Pi paints inline from wherever the cursor sits: its first render assumes a clean screen instead of clearing or entering the alternate screen itself (current Pi never emits a clear at startup). Launched under unsloth start, that left the session starting mid-scroll beneath the connection output. Clear the screen (click.clear, cross-platform, no-op without a TTY) right before the Studio banner so Pi opens exactly one line down on a clean viewport. Launch path only: --no-launch recipes and piped output are never wiped, and alternate-screen agents are left alone. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * start: auto-override hermes' 64K context floor for small model windows Hermes refuses to initialize when the served model's context window is under 64,000 tokens, and a second copy of the same check rejects the compression model mid-session. write_hermes_config previously pinned the real window, so any small local model (e.g. 40,960) failed at startup with manual config.yaml instructions. For windows below the floor the recipe now claims 65,536 in model.context_length, scales compression.threshold so compaction still fires at 90% of the real window, and sets auxiliary.compression.context_length to cover the mid-session check. Windows at or above the floor keep the exact previous behavior. * ci: install pi with --ignore-scripts, matching the start.py hint The pi cell predates the pi recipe in start.py and still installed the package with lifecycle scripts enabled, so CI stopped exercising the exact command users are prompted to run. npm_retry now passes extra flags through, the pi branch mirrors the install hint verbatim, and the stale no-recipe comment is refreshed. * ci: fail loudly when a relocation var is missing from connect output The empty-string guards ran after appending /config.toml or /config.yaml, so they could never fire: crosscheck_contract silently skipped its contract checks and patch_hermes_tools died on the root path with a bare traceback. Check the raw variable first and guide_fail with the real cause. * staging: 6613 round 6 (https elision, no-launch home reuse, auto-start key fallback) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: shimmyshimmer <107991372+shimmyshimmer@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>	2026-07-03 08:17:27 -07:00
Daniel Han	264f1a04f8	Add Local Agent Guides CI (#6547) ... Boot `unsloth run --disable-tools` against a small GGUF and drive each supported coding agent (claude, codex, hermes, openclaw, opencode, pi) through its documented `unsloth connect <agent> --no-launch` recipe, so the connect flow in unsloth_cli/commands/connect.py stays exercised end to end and regressions surface as a failing check. Per-agent matrix, three jobs: - connection: assert a non-empty, error-free reply to a trivial prompt - file-edit: a two-turn create-and-run hello.py test (dispatch/schedule only, skipped on pull_request) - prompt-cache: verify llama.cpp prefix-cache reuse across requests The GitHub-hosted runners are CPU-only, so each request is trimmed to the smallest prompt that still drives the recipe: claude with --tools to drop unused tool schemas (--allowedTools only gates permission, it does not shrink the prompt), hermes with an empty platform_toolsets.cli, and openclaw with a minimal agent definition. hermes and openclaw run a multi-turn tool loop in file-edit that a CPU runner cannot finish in time, so those two cells are best-effort; their endpoint wiring is still hard-gated by the connection job. A preflight step HTTP-checks each agent's API dialect before install so a server-side contract regression is reported separately from agent or guide drift.	2026-06-22 04:21:48 -07:00
Daniel Han	e83d4ae072	Windows installer: fix DiskPart UAC mid-install, drive-root cache, and spurious unsloth.exe rename warning (#6296) ... * Windows installer: fix DiskPart UAC, drive-root cache, spurious rename warning, CPU-base messaging amd-smi gate (DiskPart UAC mid-install): the AMD torch wheel ships hipInfo.exe inside the venv, and the bitsandbytes fix prepends that venv Scripts dir to PATH. shutil.which("hipinfo") then found it and flipped _amd_smi_allowed() to True, so the post-install AMD probe fell through to `amd-smi list` (the venv hipInfo failed to report gcnArchName, which is why the arch came from the GPU-name table) and amd-smi elevated, popping the DiskPart UAC. Fix: a hipinfo resolved inside the active venv (sys.prefix) is the torch-wheel binary, not a HIP SDK, and must not open the gate. Mirrored in install_python_stack.py, install_llama_prebuilt.py, and backend utils/hardware/amd.py (the runtime VRAM poller had the same latent prompt). TORCHINDUCTOR_CACHE_DIR: move from C:\tc to <StudioHome>\TORCHINDUCTOR_CACHE_DIR so the inductor/Triton cache lives under the user's Studio home, not the system drive root. Long paths are already enabled above so deep inductor paths still fit. unsloth.exe rename: skip the rename (and its "pip may fail with WinError 32" warning) when SKIP_STUDIO_BASE=1. In the install.ps1 flow base packages are not reinstalled, so unsloth.exe is never rewritten; the self-rename only failed because setup runs via unsloth.exe (the running launcher holds its own file). The 'studio update' flow still attempts it. CPU PyTorch messaging: clarify that the CPU base is temporary and setup replaces it with GPU ROCm wheels, and print an explicit "GPU ROCm PyTorch installed" line after the AMD wheels land, so the log makes clear the final install is GPU-accelerated. Adds two regression tests covering the venv-internal vs external hipInfo gate. Verified end-to-end on a Strix Halo box (Radeon 8060S / gfx1151): install.ps1 --local from this branch completed exit 0 with no DiskPart prompt, no rename warning, the cache under the Studio home, and "GPU ROCm PyTorch installed (gfx1151)"; Studio then booted and detected "ROCm (HIP 7.13.99004) -- AMD Radeon 8060S Graphics". * Windows installer: drop the unreliable unsloth.exe rename and its WinError 32 warning setup.ps1 used to rename the running unsloth.exe out of the way before the base-package upgrade so pip could replace it. That rename never actually worked: setup runs *via* unsloth.exe, so renaming our own running uv-trampoline launcher failed with a sharing violation (WinError 32) and only printed a scary 'could not rename unsloth.exe; pip may fail with WinError 32' warning on every Windows install and update. It also was not needed. pip tolerates a running/locked console-script .exe: it moves the old one aside and writes the new one. The base upgrade routes through pip on Windows, so the upgrade succeeds (or, in the install.ps1 flow with SKIP_STUDIO_BASE=1, the base is not touched at all) and unsloth.exe is left intact either way. Removing the rename block and its failed-install restore block removes the false warning for all Windows devices in both the install and update flows. * Windows installer: gate venv-internal hipInfo.exe in PowerShell amd-smi probe; harden venv path checks Follow-up to PR #6296. - install.ps1 and setup.ps1: ignore the AMD torch wheel hipInfo.exe that lives inside the Studio venv when probing for a HIP SDK, so amd-smi no longer reopens the DiskPart UAC during install/update. Mirrors _path_inside_venv in the Python installers, which already do this. - amd.py, install_llama_prebuilt.py, install_python_stack.py: normcase the venv containment check (Windows paths are case-insensitive) and run the HIP_PATH/ROCM_PATH candidate through it too. - setup.ps1: fall back to a short TORCHINDUCTOR cache dir when long paths are unavailable, and create the dir wildcard-safely. - tests: isolate sys.prefix in the gate helper, add HIP_PATH/ROCM_PATH cases, and assert the PowerShell venv exclusion. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows installer: install ROCm PyTorch directly for a known AMD arch When the GPU arch is known (name-inferred from the GPU-name table) but ROCm could not be probe-verified (no HIP SDK, no amd-smi), the bootstrap installed a CPU PyTorch base that setup.ps1 then force-reinstalled as ROCm. The repo.amd.com wheels bundle their own runtime (no HIP SDK required), which setup.ps1 already relies on, so the CPU base was a pure wasted download/install. - Gate the ROCm index on a known arch, not only on probe-verified ROCm, so a mapped arch installs ROCm torch directly. Unmapped arches and no-GPU hosts still get CPU (unchanged). - Fall back to a CPU base if the ROCm-index install fails, so a transient repo.amd.com outage does not abort the install (setup.ps1 retries ROCm). - Correct the stale comment that claimed ROCm wheels need a confirmed HIP SDK. - Add a regression test for the arch-based gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows installer: correct the unsloth.exe rename-removal comment The comment claimed the base upgrade 'routes through pip on Windows' and that pip 'moves the old unsloth.exe aside, then writes the new one'. That is not what the code does. install_python_stack tries uv first; on a locked launcher uv aborts and falls back to pip, but the pip fallback strips --upgrade-package and base.txt lists only bare unsloth/unsloth-zoo, so pip finds them already satisfied and no-ops. The running unsloth.exe is left intact at its current version either way. Reword the comment to describe the real uv-first / pip-fallback-no-op behavior. No functional change. * Windows installer: close two gaps in the venv-internal hipinfo exclusion Review follow-up. The amd-smi/DiskPart gate could still reopen in two cases: - setup.ps1 ran the HIP probe long before $VenvDir is assigned, so without VIRTUAL_ENV (the `unsloth studio update` path) $venvRoots was empty and the venv-internal hipInfo.exe was not recognized. Seed the venv root from UNSLOTH_SETUP_PYTHON and the default Studio home too (both installers). - The HIP_PATH/ROCM_PATH candidate was accepted without the venv filter, so an env var pointing into the venv (AMD wheel) still set $HipSdkInstalled. Run Test-HipinfoIsVenvInternal on the candidate as well (both installers). Extend the PS gate test to assert both. Both .ps1 parse clean; install tests pass (the venv-internal / HIP probe coverage at 359 passed). * Windows installer: correct the CPU-base message for arches with no ROCm wheels After gating the ROCm index on a known arch, a mapped arch sets $ROCmIndexUrl and installs ROCm directly, so it no longer reaches the "temporary CPU base" branch. That branch is now reached only by a name-inferred arch with no ROCm wheels (e.g. RDNA2 gfx103X), where setup.ps1 does NOT install ROCm. The old text ("setup replaces it with GPU ROCm wheels ... the final install IS GPU-accelerated") was therefore always wrong there. Say plainly that PyTorch stays on CPU for this GPU. * Windows installer: seed the venv-internal hipInfo check from a custom Studio home Test-HipinfoIsVenvInternal seeded the venv root from VIRTUAL_ENV, VenvDir, the setup python, and the default %USERPROFILE% path only. A standalone `unsloth studio update` with a custom UNSLOTH_STUDIO_HOME (or STUDIO_HOME alias) and none of those set would not recognize the venv hipInfo on PATH, reopening the amd-smi/DiskPart gate. Seed the custom home too, in both installers, and assert it in the gate test. * Studio installer: resolve venv aliases and expand ~ in the hipInfo venv filter Two review points on the amd-smi/DiskPart UAC gate: 1. _path_inside_venv compared os.path.abspath of sys.prefix and the hipInfo path, which does not resolve symlinks, junctions, or 8.3 short names. A venv reached through an aliased path then fails the check, so its bundled hipInfo.exe is mistaken for an external HIP SDK and amd-smi runs (the DiskPart prompt this fix exists to suppress). Switch to os.path.realpath in all three copies (amd.py, install_llama_prebuilt.py, install_python_stack.py). 2. setup.ps1's early venv-internal hipInfo probe seeded the venv root from a custom Studio home (UNSLOTH_STUDIO_HOME / STUDIO_HOME) without expanding a leading ~, while the canonical resolver does. With a tilde form, [IO.Path]::GetFullPath kept the literal ~ relative to cwd, so the custom-home hipInfo escaped the filter and reopened the gate. Expand ~ in the probe the same way as the resolver. tests/studio/install/test_pr5940_followups.py: 30 passed (adds a symlink realpath case and a setup.ps1 tilde-expansion guard). * Studio installer: mirror the hipInfo venv filter and ROCm wheel pins into install.ps1 Follow-up review on the same install.ps1 paths: 1. install.ps1's venv-internal hipInfo probe (Test-HipinfoIsVenvInternal) seeded the venv root from a custom Studio home without expanding a leading ~, unlike the canonical resolver and setup.ps1. A tilde form left [IO.Path]::GetFullPath with the literal ~ (relative to cwd), so the custom-home hipInfo escaped the filter and reopened the amd-smi/DiskPart gate. Expand ~ in the probe, matching the setup.ps1 fix. 2. The AMD ROCm path installed torchvision/torchaudio bare while pinning torch to below 2.12. AMD's per-arch index publishes the companions independently and may ship torchvision 0.27 (for torch 2.12) before removing 0.26, so a bare resolve can pick an ABI-incompatible set and fall back to CPU. Add torchvision/torchaudio floor maps and pass the pinned specs, mirroring setup.ps1 and install_python_stack.py. 3. The ROCm-to-CPU fallback torch install used Invoke-InstallCommand (no retry), the only torch step in the file without it. Switch to Invoke-InstallCommandRetry so the recovery path survives a transient index failure. tests/studio/install/test_pr5940_followups.py: 33 passed (parametrized tilde check over both installers, a torch/companion floor-map parity test, and a CPU-fallback retry guard). * Studio installer: scan all PATH hipinfo so the venv copy can't shadow a real HIP SDK The amd-smi HIP-SDK probe used shutil.which("hipinfo") / Get-Command hipinfo, which return only the first hit on PATH. The AMD torch wheel ships hipInfo.exe inside the venv and the bnb fix (plus the Studio backend) prepend the venv Scripts dir to PATH, so that venv-internal copy lands first. When a real HIP SDK hipinfo sits later on PATH with HIP_PATH/ROCM_PATH unset, the first-hit probe stopped at the venv copy, treated it as "not a HIP SDK", and closed the amd-smi gate -- AMD users in that PATH-only SDK setup lost amd-smi telemetry and could fall back to CPU. Scan every PATH entry and keep the first hipinfo that is not venv-internal; only the venv copy is ignored, so the UAC/DiskPart suppression is unchanged. Applied to all three Python copies (install_llama_prebuilt.py, install_python_stack.py, backend/utils/hardware/amd.py) via a new _external_hipinfo_on_path helper, and both PowerShell callers (install.ps1, setup.ps1) now use Get-Command hipinfo -All filtered by Test-HipinfoIsVenvInternal. tests/studio/install/test_pr5940_followups.py: 36 passed (real-PATH scan tests, a shadow-regression test for the exact venv-first ordering, and a parity check that every Python copy uses the scanning helper). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio uninstallers: fix leftovers (false "removed", shared icon, llama lock) Auditing a dual native+WSL uninstall on a real device surfaced three leftovers: 1. uninstall.ps1 removed the data dir (which holds unsloth.ico) before the shortcuts that reference that icon, so Explorer's icon cache briefly held it open. Remove-Item -Recurse reported success yet left the locked file, and the dir was never re-attempted, so it orphaned with a false "removed" log. _RemovePath now verifies the path is actually gone (retrying transient locks) and reports honestly, and the data dir is re-swept after the shortcuts go. 2. install.sh writes a shared unsloth.ico to %LOCALAPPDATA%\Unsloth Studio for the WSL shortcut, but uninstall.sh never removed it, orphaning the icon (and dir) after a WSL uninstall. uninstall.sh now drops that icon and the dir when empty, in both the powershell.exe and drvfs-fallback paths. 3. ~/.unsloth/.llama.cpp.install.lock was never removed, so the rmdir of ~/.unsloth failed and the dir lingered. Both uninstallers now remove the lock. Verified by running both uninstallers on a real dual install: device fully clean (no install dirs, shortcuts, PATH/registry entries, shared icon, or lock left). * install.sh: auto-route Strix Halo WSL to an existing Ubuntu 24.04 ROCm-on-WSL is the GPU runtime for Strix Halo and only targets Ubuntu 24.04. When the installer runs in a newer default distro (e.g. 26.04) it cannot enable the GPU and silently falls back to CPU. If a 24.04 distro already exists, re-run the install there and stop in the current one so the GPU path is taken without the user having to know about the distro requirement. Runs before venv creation so the wrong distro is left untouched, guards against re-route loops via UNSLOTH_WSL_REROUTED, leaves a working ROCm distro alone (librocdxg present), and skips the GGUF-only / opt-out / non-Strix cases. When no 24.04 distro exists we keep today's behaviour: continue to CPU and print the `wsl --install Ubuntu-24.04` guidance, never auto-downloading a distro. Adds tests/sh/test_strixhalo_wsl_reroute.sh (hermetic: extracts the function, rewrites its paths to fixtures, mocks wsl.exe) covering the full decision matrix, wired into tests/run_all.sh. * uninstall.ps1: keep shared unsloth.ico for a surviving WSL shortcut A dual native+WSL install shares %LOCALAPPDATA%\Unsloth Studio\unsloth.ico: install.sh points the WSL shortcut's icon there while the native install owns the dir. The native uninstaller removed the whole dir unconditionally, so uninstalling native while keeping WSL left the WSL shortcut with a blank icon. The old code only avoided this when Explorer happened to hold the icon open, which is unreliable; on a real dual install the dir was deleted and the WSL shortcut went blank. _RemoveDataDirKeepingWslIcon now scans the Start Menu + Desktop for a surviving "Unsloth Studio (WSL ...).lnk" and, if found, removes everything in the data dir except unsloth.ico (keeping the dir) instead of deleting it; with no WSL shortcut it removes the dir as before. uninstall.sh still drops the icon and the empty dir when WSL itself is uninstalled, so every uninstall order ends clean. Adds tests/studio/test_uninstall_dual_install_icon.ps1 (AST-extracts the helper and runs it against a temp dir with controlled shortcut dirs) covering the dual, native-only, empty, and missing-dir cases, wired into the windows-inference smoke workflow. Verified on a real dual install: native uninstall now keeps unsloth.ico and the WSL shortcut's icon stays intact. * installer: condense AMD/ROCm code comments (no behavior change) Tighten the comments added for the Strix Halo native+WSL installer work so they are shorter and clearer without losing intent: the venv-internal hipInfo amd-smi gate, the ROCm torch/companion floor maps, the WSL 24.04 reroute, and the dual-install uninstall icon handling. Comment-only; code paths unchanged. 107 insertions, 166 deletions across 11 files. * install.sh: run the Strix Halo WSL reroute before any STUDIO_HOME write The reroute fired after mkdir -p "$STUDIO_HOME" and the legacy-venv migration, so rerouting 26.04 -> 24.04 left an empty ~/.unsloth/studio stub in the origin distro (and ran venv migration in the distro about to be abandoned). Move the reroute ahead of the venv section so the origin distro is left untouched, matching the function's own comment. Behavior is identical on every non-reroute path. * installer: fix ROCm CPU-fallback, hipinfo gate edge cases, uninstall icon, WSL 22.04 - install.ps1: clear $ROCmIndexUrl/$ROCmTorchFloor after the CPU fallback so the flavor-repair block does not retry the failed ROCm index and abort the install; pin the ROCm companion specs ($visionSpec/$audioSpec) in the repair path too. - install.ps1 + setup.ps1: skip a bare drive root in Test-HipinfoIsVenvInternal so a non-venv UNSLOTH_SETUP_PYTHON does not match the whole drive; iterate HIP_PATH/HIP_PATH_57/ROCM_PATH and take the first non-venv hipinfo. - amd.py, install_llama_prebuilt.py, install_python_stack.py: strip surrounding quotes from PATH entries before probing for hipinfo. - install.sh: pipefail the WSL reroute curl|sh; do not reroute supported Ubuntu 22.04. - uninstall.sh: keep the shared unsloth.ico while any Unsloth shortcut (native or another WSL distro) still references it, in both the powershell and drvfs paths. - tests: regression coverage for all of the above. * installer: forward reroute options, guard ROCm bootstrap, harden hipinfo gate - install.sh: forward the caller's --package/--python/--verbose/--tauri and a custom UNSLOTH_STUDIO_HOME into the WSL reroute (was a bare default install); bail on --local; run the reroute BEFORE dependency/uv install so the origin distro is left untouched; set UNSLOTH_SKIP_ROCM_WSL_SETUP after a failed reroute so the later ROCm-on-WSL bootstrap does not install into the unsupported origin distro. - install.ps1 + setup.ps1: Get-Command hipinfo -CommandType Application so only real executables match (not an alias/function named hipinfo). - uninstall.ps1: guard $env:APPDATA when building the default shortcut search dirs. - tests: cover option forwarding, --local bail, the bootstrap guard, and the gate change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * installer: guard origin ROCm bootstrap on every CPU-only fallback; harden ~ expansion WSL reroute: the no-wsl.exe, no-24.04-target and --local fallbacks all tell the user the install continues CPU-only, but only the failed-reroute branch set UNSLOTH_SKIP_ROCM_WSL_SETUP=1. The later _maybe_bootstrap_rocm_wsl gate keys off that flag, so the other three branches could still install ROCm into the unsupported origin distro (e.g. 26.04). Set the skip guard on all of them. Forward UNSLOTH_ROCM_WSL_AUTO into the reroute so a Tauri/consented GPU bootstrap carries through to the rerouted 24.04 child instead of dropping to the prompt path. install.ps1/setup.ps1: guard the venv-probe ~ expansion on a non-empty $env:USERPROFILE so Join-Path does not throw on a profile-less service account. Tests: add no-wsl.exe and UNSLOTH_ROCM_WSL_AUTO reroute cases, the USERPROFILE guard assertion, and route shell-test fixtures through a single trap-cleaned root. * installer: pin + soften Windows ROCm Python repair, reroute to 22.04, harden gates install_python_stack.py: the Windows AMD ROCm repair in _ensure_rocm_torch() installed bare torch/torchvision/torchaudio via the fatal pip_install -- the same asymmetry already fixed on the PowerShell side. A transient repo.amd.com failure could abort the whole install even after install.ps1/setup.ps1 fell back to CPU. Pin companions per-arch (gfx120X/Strix -> the rocm7.2 trio, mirroring the PS floor maps) and make the retry nonfatal: keep the existing build and let the user re-run update to retry ROCm, so the chain install.ps1 -> setup.ps1 -> stack stays CPU-safe. install.sh: reroute now targets an installed Ubuntu 24.04 OR 22.04 (24.04 preferred); both are AMD-supported for ROCm-on-WSL, matching the leave-alone set, so a box with only 22.04 reaches the GPU instead of staying CPU-only. install.ps1/setup.ps1: a bare ~ for UNSLOTH_STUDIO_HOME left an empty Join-Path child (PS 5.1 throws); fall back to USERPROFILE directly and only join a real remainder. _path_inside_venv (amd.py + both installers): guard a root-dir sys.prefix so commonpath can't classify every path on the drive as venv-internal (defensive; venv never at root). uninstall.sh: guard an empty LOCALAPPDATA in the PS-interop icon cleanup (mirror APPDATA). Tests: add 22.04-target reroute cases, Windows ROCm pin+nonfatal coverage (text + behavioral), root-dir guard coverage, and bare-~/LOCALAPPDATA guard assertions. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install.sh: match WSL reroute target by exact distro name, not substring The 24.04/22.04 reroute target was chosen with grep -F (substring), so a custom distro such as 'Ubuntu-24.04-test' (with no exact Ubuntu-24.04) was picked as the target; the later 'wsl -d Ubuntu-24.04' then fails and the Strix Halo install stays CPU-only. Match whole lines (grep -ixF) and reuse the matched name so only a real Ubuntu-24.04/22.04 is targeted. Adds substring-rejection + exact-vs-custom tests. * install.sh: keep the WSL reroute target to Ubuntu 24.04 (helper-supported only) The ROCm-on-WSL bootstrap (scripts/install_rocm_wsl_strixhalo.sh) dies on any VERSION_ID other than 24.04 and pins the noble repo, so treating 22.04 as GPU-supported let the parent report a successful reroute while the child fell back to CPU. Drop 22.04 from the supported set and the reroute target list; 24.04 stays the sole target (keeping the exact whole-line distro match). An already-working ROCm on any other version is still left alone by the librocdxg check above. tests: reroute 22.04 cases updated to the 24.04-only behavior; make the "no wsl.exe" case hermetic so a real host wsl.exe can't leak in on dev boxes; stop the tauri exit-order check from mis-flagging the reroute helper's [ "$TAURI_MODE" = true ] && ... --tauri one-liner. * installer: tighten comment wording across the Strix Halo install/uninstall paths Condense the verbose multi-line comment blocks (amd-smi hipinfo gate, ROCm torch install + CPU fallback, WSL reroute, uninstall icon-keep) into fewer, clearer lines. Comments and a few docstrings only; no code, logic, or behavior change. Verified with bash -n, the PowerShell parser, and ast.parse, and the installer test suite still passes. * add AGPL-3.0 SPDX headers to the .sh/.ps1 scripts missing them Every shell and PowerShell script under the Studio/installer surface now carries the standard SPDX-License-Identifier: AGPL-3.0-only + copyright header (after the shebang where present): the installer (install.sh, install.ps1), build.sh, the .github and src-tauri scripts, the installer test suite, and the moe kernel test. Header-only, line endings preserved; bash -n, the PowerShell parser, and the installer tests all pass. * installer: drop the duplicate AGPL header from install.sh and install.ps1 Both already carry an SPDX-License-Identifier: AGPL-3.0-only header below their usage comment block; the prior header pass added a second one at the top because it only scanned the first few lines. Remove the duplicate so each file keeps a single original header. * installer: force-reinstall CPU fallback torch; propagate Tauri NEED_SUDO from reroute install.ps1/setup.ps1: when the AMD ROCm wheel install fails and we fall back to a CPU base, force-reinstall the torch/vision/audio triplet. A failed ROCm install can leave an unpinned ROCm torch (e.g. 2.10.0+rocm on gfx110X/gfx90a) that still satisfies the CPU torch>=2.4,<2.11.0 range, so without --force-reinstall uv keeps the ROCm build and only swaps the companions -- a mismatched venv the flavor-repair block won't fix. setup.ps1 scopes the forced reinstall to the ROCm-fallback path () so the genuine CPU-only install stays fast. install.sh: the Strix Halo WSL reroute treated every nonzero child exit as a reroute failure and fell back to CPU. In --tauri mode the child uses exit 2 ([TAURI:NEED_SUDO]) to ask the desktop app to elevate for the target distro; capture the child's exit code and propagate exit 2 in Tauri mode (the child already printed the NEED_SUDO line) instead of masking it. CLI mode still falls back to CPU on a generic failure. Tests: reroute Tauri exit-2 propagation (and non-Tauri CPU-fallback) cases; run_func now preserves the child exit code; force-reinstall assertions for both PowerShell installers. Note: codex's _rr_q apostrophe finding is a false positive -- the helper already emits POSIX-correct 'O'\''Brien' and round-trips under both sh and bash. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * setup.ps1: fix $cpuForce array collapse in the ROCm->CPU torch fallback An if-expression assignment ($cpuForce = if ($ROCmCpuFallback) { @("--force-reinstall") }) collapses the single-element array to a scalar string, so @cpuForce splatting enumerated it character-by-character into broken single-letter args (- - f o r c e ...), which made uv/pip reject the install and aborted the whole Studio setup on the AMD ROCm->CPU fallback path. Build $cpuForce as a real array assigned outside the if-expression so the splat passes a single --force-reinstall arg. Genuine CPU-only installs stay fast (empty array, no flag). Test now asserts the array-build form and rejects the if-expression form. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * uninstall: remove the isolated Node.js runtime (~/.unsloth/node) The isolated Node.js runtime (install_node_prebuilt.py, added with the managed-Node change) installs to ~/.unsloth/node in default mode -- a sibling of studio, so deleting <studio> leaves it behind (~200MB orphaned after uninstall). Both uninstallers already remove the other default-mode siblings (llama.cpp/.cache/.staging); add node alongside them. uninstall.ps1 also adds it to the handle-lock sweep so a held node.exe can't block the delete. Env/custom mode nests node under the custom root, removed with that root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-22 03:09:08 -07:00
Daniel Han	366937de44	studio: pick a macOS llama.cpp prebuilt that loads on the host OS (#5883) ... Make macOS llama.cpp prebuilt selection host-OS-version aware: skip a prebuilt whose minimum-OS exceeds the host and walk back to the newest release that loads (macOS 26 keeps latest; 14/15 land on a compatible older release). Source-build fallback pins CMAKE_OSX_DEPLOYMENT_TARGET=13.3. CI: binary-load assertion plus a macos-14/15/26 install matrix. No change to Linux/Windows or CUDA selection.	2026-05-31 00:59:25 -07:00
Daniel Han	54a86c3514	ci: route every hf download through xet-tuned stall-retry wrapper (#5476) ... Root cause of the Mac json-images 30 min timeout (run 25950714888 / PR #5430): huggingface_hub>=1.15 deprecated `hf_transfer` and routes every transfer through `hf-xet`. The CI step's unpinned `pip install --upgrade huggingface_hub hf_transfer` jumped to 1.15.0 + hf-xet 1.5.0, the 940 MB mmproj finished in ~21s, then the 3 GB gemma-4 GGUF made it to ~46% and went completely silent for the remaining 29 minutes -- no progress bytes, no error, no exit -- until the job timeout fired. This wraps every CI `hf download` in a new `.github/scripts/hf-download-with-retry.sh`: * Drops the no-op `HF_HUB_ENABLE_HF_TRANSFER=1` prefix and the `hf_transfer` install (both are deprecated on 1.15+ and only emit a FutureWarning now). * Exports the hf-xet high-performance knobs Daniel asked for: HF_XET_HIGH_PERFORMANCE=1 HF_XET_CHUNK_CACHE_SIZE_BYTES=0 HF_XET_NUM_CONCURRENT_RANGE_GETS=64 HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=0 HF_XET_CLIENT_READ_TIMEOUT=500 * Watchdogs each attempt: if `hf download` has not exited after HF_DOWNLOAD_STALL_SECONDS (default 180s = 3 min), SIGTERM, sleep 2, SIGKILL, then loop. Retries are unbounded; the enclosing job's `timeout-minutes` is the real cap. * Optional 3rd positional `LOCAL_DIR` -- omitted lets `hf` use the default HF_HUB_CACHE, which is what the HF_HOME-priming jobs need. 19 call sites migrated across mlx-ci.yml + 9 studio-*-smoke.yml workflows. The inline `python -c "from huggingface_hub import hf_hub_download; ..."` block in mlx-ci.yml is also routed through the wrapper so every hf transfer in CI gets the same treatment. Also reverts the json-images timeout 45 -> 30 from #5475: the bump was masking this hang, not fixing it.	2026-05-15 21:11:56 -07:00