3 commits

Author	SHA1	Message	Date
Daniel Han	482abfabfa	Give Unsloth Studio its first CUDA coverage, on a Kaggle T4 (#8489) ... * Run deterministic notebook smoke tests on real Kaggle T4s The notebooks are written for a T4: sm_75, fp16 with a gradient scaler, xformers rather than flash-attention, 16GB. Nothing in CI runs on that card, so a regression that only appears there reaches users first. Adds a gated job that launches one Kaggle GPU session and runs the smoke payload once per T4. The account has a 60h weekly budget shared with other consumers, so the job is a deliberate minority consumer of it: paths filters (only 9.9% of a week's 444 main commits touch unsloth/), a ~10% sampling draw keyed on the run id so a re-run cannot reroll, a quota floor read live from Kaggle's own API, and a stand-down when the account already has a kernel in flight. Expected cost is 5-7 GPU-hours a week. Only a payload that ran on a T4 and disagreed with its assertions turns the check red. Throttling, the 2-kernel concurrency cap, a dead session or a download that would not complete are all warnings, because a free external service blocking merges would be ignored within a week. The payload asserts three things, worth different amounts. Two fresh processes must agree bitwise on per-step loss and grad_norm; that one is exact, and it is fresh processes because two in-process cycles disagreed from the first step while two processes agreed on all ten. Greedy decoding must emit the canary string exactly, which catches trained weights never reaching the generate call. Metrics must sit inside a band around a committed reference, which is a tolerance and never an equality, since fp16 reduction order moves with the driver and the libraries. Ten steps rather than three: under fp16 the scaler skips the first two overflowing steps, so a three-step run lands one real update and the canary never forms. The dataset is supplied as prompt/completion columns so the loss falls only on the answer. * Make the payload cells the T4 rejected actually run there Two launches on real Kaggle T4s died in the scaffolding before a training step ran, both of them in generated code that nothing had ever parsed. The dependency probe imported unsloth_zoo before unsloth. Zoo's __init__ ends with a find_spec("unsloth") guard and raised "Please install Unsloth via pip install unsloth" on a session where unsloth was installed and imported cleanly one entry later, so the payload aborted over a dependency that was not missing. Probe unsloth first, and invalidate the import caches after the pip installs that this interpreter did not see at startup. The reference argument was generated as a shell fragment and spliced into the middle of a Python list literal, with doubled braces that left ROOT unsubstituted. The cell was a SyntaxError, and the workflow passes --reference on every path, so no payload could ever have reported. Verified locally by executing the regenerated run cell end to end: two fresh processes, bitwise identical metrics, canary exact. * Bound the in-flight kernel survey by session length, not kernel count The concurrency check status-checked the twelve most recently run kernels, which is a sample rather than a search: a kernel that started three hours ago and is still running is missed the moment twelve newer ones have since run, and the push then fails at the account capacity cap and is reported as infra. Kaggle kills a notebook session at twelve hours, and the listing is sorted by last run time, which for an unfinished kernel is when it started (measured: a kernel pushed at 10:05:19Z lists as 10:05:19.297). So walking until the first entry older than that ceiling covers every kernel that can still be in flight, and stopping there is exhaustive rather than arbitrary. An hour of slack is added for clock skew. Two silent ways to read an unanswerable question as a green light are closed with it. Hitting the page cap now marks the survey incomplete, and a survey where no status could be read at all is unknown rather than idle. Both stand down, which is the cheap direction: the next commit draws again in minutes. On the live account this now costs three status calls instead of twelve. * Stop the reference band check from passing NaN unconditionally The band comparison did its NaN handling by accident, and it got it wrong. Under fp16 the gradient scaler logs a NaN grad_norm on every step it skips, so a committed T4 reference genuinely contains NaN. Left to the arithmetic, abs(x - NaN) is NaN and NaN > tolerance is False, so those steps passed whatever they held, including the case the check exists for: a step that used to overflow and no longer does. Compare NaN to NaN as equal and NaN against a number as a deviation, and treat a field logged on one side only as a deviation too, since no tolerance covers a change in shape. The mapping from an out-of-band verdict to the failure list moves into reference_failures() so the path that turns the job red can be tested without a GPU. The references README now says plainly that no reference is committed, that one must not be borrowed from other hardware, and how to lift one out of a green run's evidence rather than spending a session on it. It also records a measurement: the absolute floor never engages on this trajectory. The smallest value on the ten-step curve is a loss of 0.1428 against a floor of 0.05, so the floor changes no verdict today and is kept for a configuration that does go lower. * Name the cause when a kernel finishes and reports nothing A kernel that ends COMPLETE with no payload report is the hardest outcome to read: the summary has no metrics to show and the reason sits in an artifact nobody downloads. It is also the outcome both real T4 failures took. Kaggle returns the kernel log as a JSON array of stream records rather than as text, so the interesting line arrives split across dozens of them and reading the file directly shows a wall of JSON. Flatten it, filter to the driver and payload sentinels plus tracebacks, and fold the tail into the job summary. Checked against the evidence of a real failed run: the SyntaxError that killed it is now visible without leaving the summary. * Cover the four T4 CI fixes with CPU-only tests The generated cells are now compiled on both the with-reference and without-reference paths, which is what would have caught a SyntaxError that instead cost a Kaggle session, and the dependency probe's module order is asserted. The in-flight survey gets a fake Kaggle client and the case that motivated it: one kernel still running behind forty newer finished ones, which a count-bounded scan misses and a time-bounded scan cannot. Also the window boundary, timezone-aware and missing timestamps, the page cap, and the difference between some statuses unreadable and all of them. The band check is now proved to fail rather than assumed to work: a perturbation half a band-width past the limit, a moved scaler skip pattern in both directions, a field that stopped being logged, and a length mismatch, each asserted through to the failure list. Three tests perturb the committed reference itself and skip, saying so, until one exists. 41 passed, 3 skipped. * Parse every generated cell on every build path before spending a session Two of the three Kaggle sessions spent so far died on generated code that nothing had ever parsed. The compile check now builds all three paths the generator has, using the workflow's own argument list verbatim, including the one where a committed reference exists and is carried inline as a fourth file: that path becomes live the moment a green run supplies the file, and would otherwise first be exercised on Kaggle. Parsing alone is not enough. A template hole that substitutes to a bare identifier parses and then NameErrors, which costs the same session, so every cell is also checked in execution order for names nothing before it defines. Both checks were confirmed to go red: the first on the exact shell-fragment-in-a-list-literal defect that cost session three, the second on a run cell reading a name no cell binds. The carried sources are also decoded the way the kernel will decode them and compared byte for byte with the repo, so the T4 cannot quietly run something other than what is committed. * Commit the T4 reference the first green Kaggle run produced Kernel danielhanchen/unsloth-t4-ci-e3c6661f, terminal state COMPLETE, both payloads passing on their own Tesla T4 of one session. Two fresh processes agreed bitwise on all ten steps on both cards, max_abs_diff exactly 0.0 for loss and for grad_norm, and the two cards independently produced the same ten values as each other, so four processes agree rather than two. All four cycles emitted the canary exactly. The scaler skipped steps 1 to 3 every time, which is the behaviour the ten-step length exists to accommodate. The file is reports[0] of that run, copied whole by the recipe the README already documented, and nothing in it is from other hardware. One assumption the run contradicted, now recorded rather than repeated: the session had no xformers. unsloth installs --no-deps, unsloth_zoo does not carry xformers and the Kaggle image does not either, so the banner read Xformers = None and this trace is the fallback attention path. Adding xformers to the install would move these numbers and require a recapture. The floor's smallest observed value moves from 0.1428 to 0.0871, still above the 0.05 denominator floor, so the floor stays inert and the test that re-derives that keeps passing. The three tests that perturb the committed reference now run instead of skipping. 45 passed, 0 skipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the in-flight tolerance the Kaggle gate stands down on The gate declines to launch whenever any kernel of the account is in flight, which is stricter than Kaggle's 2-kernel cap requires. That is the intended policy, so it is now a named constant with the tradeoff written next to it rather than an implicit property of the code, and a --allow-in-flight argument that cannot be raised to the cap itself. Default behaviour is unchanged. Also files the workflow under 'never' in ci-preempt.json: cancelling a run does not stop the Kaggle kernel it already pushed, and the orphan then bills quota to its own ceiling with nobody watching. * Run three steps, and refuse a reference captured at another count max_steps drops from 10 to 3. Measured consequence: under fp16 the gradient scaler starts at 65536 and skips every step it overflows on, and the committed reference shows steps 1, 2 and 3 all skipped, so a 3-step run of the old configuration applied zero optimizer updates. --init-loss-scale pins the scaler below the overflow point so those three steps are real updates, and optimisation_failures() fails a run whose every step was skipped rather than letting it report as training. The committed reference was captured at 10 steps and no longer describes the run. check_reference now reads the max_steps the reference records and refuses to compare across counts, as a failure with both counts named -- including when the reference does not say what it was captured at. A quiet pass there would be a green check comparing a run against a curve it has nothing to do with. The reference is therefore stale until one recapture run. The workflow gains skip_reference_band for exactly that dispatch, and the recipe is in references/README.md. Also brings the workflow onto the repo's conventions: pinned action SHAs matching the rest of CI, a workflow-level concurrency group that deliberately does not cancel in progress, a fork guard on the gate job, typed dispatch inputs and an artifact name without the run id. * Keep the run at ten steps, and say why three does not work The committed reference is the evidence: the fp16 scaler reports grad_norm NaN at steps 1, 2 and 3 and a finite value from step 4. It starts at 65536, halves on each overflow, and skips the step it overflowed on, so a 3-step run applies zero optimizer updates. Loss stays around 10, the canary never forms, and the band would compare three points of a curve that never moved. Shortening also saves nothing worth having: a launch costs about 0.08h and that is almost all pip install, not training. The step-count guard, the all-steps-skipped guard and --init-loss-scale all stay, since they are what makes a shorter run possible later and what stops a vacuous one reporting green. The pin is off by default so the committed reference keeps applying. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Turn one T4 payload into four, and pair a pinned control with a canary The workflow ran a single tiny SFT payload on both cards of one Kaggle session and spent about 1.4 GPU-h a week doing it. The budget is now 40, and the question asked of it is wider: catch regressions in trl, transformers, accelerate, peft, bitsandbytes, torch and vllm, and catch torch.compile falling over on sm_75. Four legs now, two kernels of two T4s each, described once in .github/scripts/kaggle_t4_ci/legs.py rather than restated in YAML: control the existing tiny Qwen2.5-0.5B determinism run, pinned canary the same run on the newest library set Unsloth's own constraints allow gptoss gpt-oss-20b LoRA, for torch.compile and the forced float32 path grpo Qwen3-4B GRPO through a vLLM engine on the same card control and canary are one instrument rather than two tests. Same payload, same seed, same data, same step count, on the two cards of the same session; the only difference between them is the installed versions. A red canary beside a green control is a library release, and the job summary already puts the two version sets side by side so the bisect is a diff rather than an investigation. The reference band applies to the control alone: two library sets do not produce one fp16 trajectory, so the canary asserts what does not depend on versions instead. The pins are evidence rather than preference. transformers and trl are the pair the committed reference was captured with; peft, accelerate and bitsandbytes come from probe kernels 8161ceb9 and 7ab727f1, two independent T4 sessions that ran the identical base install and agreed. torch stays unpinned and recorded, because it comes from the Kaggle image and replacing it is the failure mode the grpo leg spent two probes on. The gate now separates this workflow's own kernels from everybody else's. It takes both of Kaggle's concurrency slots, and only when the account is otherwise idle: one kernel belonging to anyone else still stands it down entirely, which is the same policy as before stated at the right grain. * Leave the vLLM leg unwired, and stop trusting Kaggle's own kernel timeout Three probe sessions tried to run the Qwen3-4B GRPO leg on a T4 and none of them reached a training step. None of them failed for a reason to do with sm_75, memory or GRPO either: vllm==0.11.2 pins torch==2.9.0, so the leg has to replace the Kaggle image's torch, and that is what killed all three. With the image's site-packages visible, pip treats torch's pinned NVIDIA runtime packages as already satisfied by the copies belonging to 2.10, and the result is a torch that installs cleanly and cannot be imported -- libcusparseLt.so.0 first, then ncclCommWindowRegister once that one was named explicitly, identically on vllm 0.11.2 and 0.15.1. An isolated venv resolving the whole stack got further and then the session wedged. So the leg keeps its payload, its assertions and its install, and moves to legs.UNWIRED with the three kernel slugs and what each one measured. Wiring it now would make the check permanently red and spend the budget doing it. The second kernel keeps its spare T4, which costs nothing, and that is where the leg goes when the install works. The third probe also cost about an hour of quota teaching us that the push-time kernel timeout is not a budget control. Pushed with -t 5400, its own nbconvert crashed at t=406s and the session sat in RUNNING for over two hours -- past that ceiling and past the launcher's own deadline, because one status call had blocked with no socket timeout to bound it. Deleting the kernel by hand took the account's used-hours figure back down, which is the measurement that settles which control is real. The launcher now sets a socket timeout and deletes every kernel it pushed on every path out, and the workflow header says which of the three bounds is load-bearing. Sampling recomputed on the payload that actually ships: 40%, from measured per-kernel cost rather than the old estimate, with the note that wiring a fourth leg means recomputing it rather than editing one line. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-solve the GRPO leg's install so it stops replacing the image's torch Three probe sessions never reached a training step, and all three died the same way: vllm==0.11.2 pins torch==2.9.0, so the leg had to replace Kaggle's torch 2.10.0+cu128, and pip treats the image's NVIDIA runtime packages -- which belong to 2.10 -- as satisfying the new torch's pins. That imports as libcusparseLt.so.0 missing, then as libtorch_cuda.so undefined symbol ncclCommWindowRegister one package along, then as an isolated venv that spent an hour of quota resolving a CUDA stack and never produced payload output. None of that is a question about sm_75, memory or GRPO. It is one line: no vLLM release in the 0.11-0.16 range pins the torch the image ships. Releases 0.17.0 through 0.19.1 pin torch==2.10.0 exactly, so pin 0.19.1 and the leg replaces nothing, keeps system_site_packages, and installs like any other. Drop xformers with it. Its vLLM attention backend was deleted in 0.12.0, so it would be a package nothing selects. sm_75 has no FlashAttention and no FlashInfer, and the ladder in vllm/platforms/cuda.py falls through both to TRITON_ATTN; the leg names that in VLLM_ATTENTION_BACKEND so a release that reorders it goes red here rather than quietly selecting something else. 7.5 is still in CUDA_SUPPORTED_ARCHS at v0.19.1 and fp16 is a supported dtype below capability 8.0. Still unwired. Two things are static evidence rather than a run: whether 0.19.1 starts on sm_75 at all, and whether 8GB of 16-bit weights plus an engine plus a LoRA trainer fit in 14.56GB. UNWIRED says so in those words, and a test asserts the note still names the pin and still says what is unknown, so nobody wires this off prose that outlived its measurement. Also ports the vacuous-test fix from the stacked Studio branch: pre-commit.ci reformatting default=10 to default = 10 stopped the payload regex matching, so the step-count agreement check was comparing nothing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop the GRPO leg JIT-compiling flashinfer on a machine that cannot link it First probe on the re-solved install, kernel unsloth-t4-ci-e2d9ce9b. The install works: torch stayed at the image's 2.10.0+cu128, vllm 0.19.1 installed and imported, xformers was absent as intended, and the payload ran on a real Tesla T4 reporting capability 7.5 with TRITON_ATTN accepted. It reached engine construction, which is further than any earlier probe got by the whole width of the install. It died in flashinfer 0.6.6's JIT, and not for an sm_75 reason. All three .cu files compiled cleanly for -gencode=arch=compute_75,code=sm_75. The LINK failed: /usr/bin/ld: cannot find -lcuda -L/usr/local/cuda/lib64/stubs is already on the command line, so the image has no driver stub libcuda.so, only the runtime libcuda.so.1. Nothing about that is fixable from a payload. Set VLLM_USE_FLASHINFER_SAMPLER=0 so nothing JITs. The sampler has a native path, and skipping the build also saves a four-file nvcc compile inside a session billed by wall clock. UNWIRED records what the probe measured and narrows what is left to one question: whether the engine builds now. * Make -lcuda resolvable so flashinfer can link what it compiled Second probe, kernel unsloth-t4-ci-916d5986, failed identically to the first with VLLM_USE_FLASHINFER_SAMPLER=0 set. That is the informative part: the JIT is not reached only through the sampler, so switching off one consumer is whack-a-mole. The failure was never sm_75. Both sessions compiled all three .cu files cleanly for -gencode=arch=compute_75,code=sm_75 and died at the link: /usr/bin/ld: cannot find -lcuda -L/usr/local/cuda/lib64/stubs is already on the command line; the image ships no libcuda.so at all, only the versioned libcuda.so.1, and the linker will not resolve -lcuda against a soname. Normally the toolkit's driver stub fills that gap; this image has the directory and not the file. So symlink libcuda.so at the real driver in a scratch dir and put it on LIBRARY_PATH, which gcc and ld search for -l. No root, nothing written into /usr/local, and it fixes every flashinfer op rather than one. Linking against the real driver instead of a stub is correct here: the driver is present, which is the whole reason a stub would have substituted for it. Also widen the captured traceback to head AND tail. The last probe's 6000-char tail was entirely ninja's own output, so the Python frames naming the caller were exactly what got dropped. Probe 2 also measured the fit question the notebook raised: peak reserved 7.96GB of 14.56 with load_in_4bit false, so 16-bit weights are not what is short of room here. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only count a libcuda the linker will actually search for Kernel unsloth-t4-ci-d0d480b6: the shim reported already_linkable and did nothing, and the link failed anyway. It had found libcuda.so in /usr/local/cuda/compat -- which is real, and which nothing passes with -L. The failing ninja line names /usr/local/cuda/lib64 and /usr/local/cuda/lib64/stubs, and those are the only two directories that can answer -lcuda. So the precondition now checks exactly those two, and compat becomes what it is actually good for: a symlink TARGET when ldconfig and ctypes both come up empty. A library the linker will not search for is not a library the linker can find. Two tests: one reproduces the compat-only machine and asserts the shim is built anyway, one pins the searched directory list, since widening it is precisely how the check went wrong. * Give the base model a chat template, and record what the link shim did The link shim works. Kernel unsloth-t4-ci-27b0dc2e reported libcuda_shim: needed true, applied true, searched [/usr/local/cuda/lib64, /usr/local/cuda/lib64/stubs], real /usr/local/cuda/compat/libcuda.so and flashinfer is gone from the failure entirely. Memory went from 7.96GB to 11.36GB of 14.56, which is the vLLM engine actually loading, and the run reached trainer.train() -> _run_epoch -> training_step. Both questions legs.UNWIRED was holding this leg for -- does vLLM start on sm_75, does it fit in 14.56GB -- are answered by that. It then failed on the payload's own gap: ValueError: Cannot use chat template functions because tokenizer.chat_template is not set unsloth/Qwen3-4B-Base is a base model and ships none. The notebook this leg comes from solves it with an SFT priming stage that installs a template before GRPO starts; this leg has no priming stage, so it sets a minimal ChatML template directly and records in the report which of the two worlds the run was in. The base model stays. GRPO on an instruct model would measure the instruct tuning as much as the run, and the payload's rewards are format-and-digit rewards a base model can move inside three steps. A test renders the template rather than matching it as a string: a template that does not render would trade a failure at step 1 for a failure at step 1 with a longer traceback. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Wire the GRPO leg: it runs on a T4 and the rewards move Kernel unsloth-t4-ci-53efcc4e passed on a real Tesla T4. Step 2 logged reward_std 0.707 and grad_norm 0.772, peak 13.60GB of 14.56GB, three steps in 192s, and fast_generate returned coherent text. reward_std is the criterion rather than loss, because TRL's GRPO loss is ~0 by construction at num_iterations=1 and beta=0 and would pass on a run that learned nothing. The config in args is the one that fit, not the notebook's. Two probes with seq 2048, 4 generations, rank 32 and utilization 0.9 died in the backward at unsloth_zoo/gradient_checkpointing.py:1013, peaking at 15.97GB in 16-bit and 19.25GB in 4-bit. 4-bit alone is not the lever: it peaked HIGHER, because quantizing weights does nothing for activations while utilization 0.9 still hands vLLM ~13GB up front, and UNSLOTH_VLLM_STANDBY returns the weights during training but not the KV cache reservation. An earlier probe reported this as 'CUDA error: an illegal memory access was encountered' from empty_cache() inside vLLM standby. That was the async shadow of the same OOM; CUDA_LAUNCH_BLOCKING=1 produced the honest error. UNWIRED is now empty. The test that read its grpo entry is replaced by the invariants that outlive any one leg: every leg is wired or explained, nothing is both, and an unwired note says what is open. Two new tests pin the config and the vLLM pin, so restoring either to a value that OOMs fails here rather than on Kaggle. * Add a CUDA_LAUNCH_BLOCKING switch to the GRPO payload An illegal memory access is reported at whatever CUDA call synchronises next, which on the first T4 GRPO run was empty_cache() inside vLLM standby, nowhere near the kernel that faulted. Re-running with this flag produced the honest error instead: torch.OutOfMemoryError in the backward. That is what turned an unexplained CUDA fault into a memory budget, and the budget is now the leg's config. The environment variables are set before the first import torch in the process rather than merely before train(), because they have to precede CUDA context creation. It serialises every kernel launch, so it stays a switch and never a default, and it is recorded in the report because it changes what a timing means. * Give Studio its first CUDA coverage, on a Kaggle T4 Every Studio workflow in this repo runs on ubuntu-latest, macos-15 or windows-latest. macOS gives Metal and MLX some hardware; the CUDA path is exercised by nothing at all, and studio-inference-smoke.yml uses a 270M GGUF precisely because it has to decode on a CPU. tests/kaggle/studio_gpu installs Studio the supported way on a Kaggle T4, starts it headless, and asserts three things a CPU runner cannot: that a GGUF is actually offloaded to the GPU rather than falling back to CPU and returning text anyway, that a LoRA training run completes and leaves an adapter on disk, and that a GGUF export against a CUDA llama.cpp produces a file that loads. The repo's existing playwright_chat_ui.py then drives the browser path, last, because it stops the server on its way out. The gate and the launcher are the notebook leg's, reused unchanged: the payload prints the same report line, so the transport needed no edit. The two legs share a concurrency group because Kaggle's 2-kernel cap is per account and is not workflow-aware. Sampled at 4 percent of a filter measured at ~760 eligible invocations a week, which is ~30 launches and ~23 GPU-h against the notebook leg's 15-20 and 1.4. The rate is lower and the spend is fifteen times higher because the eligible stream is five times larger and a launch costs nine times as much; the arithmetic is in the workflow header. * Budget the two Kaggle GPU legs against one account, not two The Studio leg's budget block was solved against the notebook leg's numbers from before that leg was re-tuned: 0.08 h per launch, 10 percent, ~1.4 GPU-h/week. The notebook leg now costs 0.25 h at 40 percent for up to 23.1 GPU-h in a busy week, and Studio was sized at ~23 of its own. Two legs at 23 each is 46 of the 50 hours CI is allowed on a 60 h account, with the entire margin spent on the cheaper one, and the two reserve floors were picked independently. Re-solve both against one shared allowance: Studio 35 h, notebooks 15 h. Studio goes to 5 percent (~38 launches, ~28 GPU-h) and notebooks to 15 percent (13-35 launches, 3.3-8.7 GPU-h). The reserves now encode priority rather than each leg's own ceiling: notebooks stand down once the account is 35 h spent, Studio runs on to 50. The percentage is the one axis where Studio reads lower, and it cannot be made to agree. Studio's eligible stream is ~3.3x larger and each launch costs 3x more, so a point of rate is worth about ten times as much there; matching the notebook leg's 15 percent would be ~86 GPU-h a week. Both budget blocks now say so, and a test re-derives the joint spend from the two headers instead of trusting either total. Also fix a test that had gone vacuous: pre-commit.ci reformatting default=10 to default = 10 stopped the payload regex matching, so the step count agreement check was comparing nothing. Match count is asserted now. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Write down where the Kaggle sampling percentages come from The percentages in the two workflows were arithmetic nobody could check. This records the measurements they came from, taken 2026-08-11 over a 28-day window: 985 commits to main, 1157 PRs, and 8 of 60 sampled PRs touching the paths a T4 training run could regress, so a 13% path-filter rate and roughly 75 candidate events per week. Against 30 GPU-h/week for one account at ~0.75 GPU-h per launch, the ceiling is about 33 launches per week if everything went to one workflow. The 15/5 split and the 25/10 reserve-hour floors follow from that, and the file says what would change them: a second account, a wider path filter, or a session cost above ~1 GPU-h. * Give the GRPO leg its own kernel: pairing it with gpt-oss fails Kernel unsloth-t4-ci-70a2f4eb ran the wired pair on the two cards of one session. gptoss passed: 32 unique graphs, 779 calls captured, 2 graph breaks, peak 12.78GB. grpo failed with the illegal memory access again, at exactly the 13.60GB peak at which it PASSED alone on kernel unsloth-t4-ci-53efcc4e, same config to the flag. Same peak, different outcome, so the constraint is not GPU memory: the driver pins one card per payload. gpt-oss offloads to host RAM and vLLM wants host RAM, and a Kaggle session has one host. A third kernel costs a queue slot rather than quota, since Kaggle runs two at a time and bills wall clock per session. That is cheaper than a leg which fails half the time for a reason the report cannot show. Two tests pin the rule in both directions: grpo must not share a kernel with gptoss, and control must still share one with canary, because those two are a matched pair whose comparison is the whole instrument. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Unwire the GRPO leg: running it alone reproduced the failure I blamed on pairing I gave grpo a kernel of its own earlier today on the reasoning that sharing a session with gpt-oss was what broke it. It had failed paired (unsloth-t4-ci-70a2f4eb) and passed alone (unsloth-t4-ci-53efcc4e), so pairing looked like the variable, and I wrote two tests pinning the separation. That reasoning was wrong. Running the leg ALONE again (unsloth-t4-ci-c98f14be, built from this branch with --legs grpo) reproduced the paired failure exactly: unsloth_zoo/vllm_utils.py:601 sleep() -> torch.cuda.empty_cache() torch.AcceleratorError: CUDA error: an illegal memory access was encountered engine_built false, peak 13.8GB/13.6GB of 14.56GB. The passing session and both failing ones agree to the digit on every recorded version -- torch 2.10.0+cu128, transformers 5.5.0, trl 0.24.0, peft 0.19.1, vllm 0.19.1, unsloth 2026.8.15, zoo 2026.8.10 -- on the same peak, with UNSLOTH_VLLM_STANDBY=1 throughout. One contrasting observation was never enough to blame a shared host. So what three sessions show is an intermittent illegal memory access on Turing, one pass in three, not a pairing effect. A leg that passes one session in three cannot tell CI anything: it would go red for a reason no reader could act on. grpo moves to UNWIRED with the three session ids and the open question written down, rather than back into a kernel. The note keeps what the leg already earned -- the vLLM/torch pin, the attention backend, the libcuda shim, the chat template, the T4-sized config -- so re-wiring means answering the IMA rather than redoing the install. test_grpo_does_not_share_a_session_with_gptoss is replaced rather than deleted: it asserted the opposite thing for the disproven reason, and would now pass vacuously since grpo is in no kernel at all. The replacement pins that grpo stays unwired and that the note still names all three sessions, so the evidence cannot be dropped on the way back in. 145 passed. * Retire the bootstrap password at login, so the session can reach the routes First hardware run of this payload, Kaggle kernel unsloth-t4-ci-412345d2 on two Tesla T4s: preflight, studio_ready, authenticate and the Playwright chat_ui driver all passed, and the four assertions the payload actually exists for did not run. gpu_inference: POST /api/inference/load -> HTTP 403: {"detail": "Password change required"} lora_training: POST /api/train/start -> HTTP 403: {"detail": "Password change required"} A bootstrap account is created with must_change_password set, and get_current_subject turns that into a 403 on every route except the password-change one. So login yielded a token that authenticated and could do nothing, and the login step reported success while inference, tool calling, training and export were all unmeasured behind it. login now completes the change when Studio's own Token says must_change_password, and carries the token that /api/auth/change-password mints. The replacement is secrets.token_urlsafe(24): never whitespace, which the route rejects, and never equal to the bootstrap value, which it also rejects. It is not returned or stored -- nothing needs it again. Four tests, all off-box against a scripted HTTP layer: the change happens, and the session carries the post-change token an account already past the gate does NOT get its password rotated a refused change raises instead of carrying on with a token that cannot act a change that answers without a token raises The third is the one that matters: continuing past a failed change is how the first run produced four unmeasured assertions and no explanation. Two things this does NOT fix, recorded so the next run is read correctly. gguf_export reported install_kind=None, i.e. no llama.cpp install under STUDIO_HOME at all. That may well be downstream of the 403 rather than independent: POST /api/inference/load is what pulls a llama.cpp for a GGUF model, and it never ran. The collected kernel.log is 223 lines and holds none of the install output, so this is not settled either way -- the re-run answers it. gpu_inference also failed its own evidence test: text came back, but no probe could show the GPU was used. The payload calls that a failure rather than a pass, which is the behaviour I want kept. 89 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hand the UI driver the password the session actually holds Second hardware run, kernel unsloth-t4-ci-9ddd8ae4. Retiring the bootstrap password fixed three assertions and broke a fourth: gpu_inference FAIL -> PASS tool_calling FAIL -> PASS lora_training FAIL -> PASS chat_ui_driver PASS -> FAIL: "the bootstrap password is gone, so the driver cannot log in" My own doing. assert_chat_ui read $STUDIO_HOME/auth/.bootstrap_password off disk, and authenticate() has to retire that value to get past Studio's forced password change, so the file is stale by the time the driver runs. The driver's first phase rotates the password itself and asserts the old one stops working, so it needs the value the session is CURRENTLY authenticated by -- not the seeded one. login now leaves that value on Studio.password and assert_chat_ui passes it, falling back to the seeded file only when no login has happened. The commit that introduced the retirement said the replacement was "never returned or stored: nothing needs it again". That was wrong, and the docstring now says so and says why. Studio.password is a credential held for the run, unlike anything else in that module, so the module docstring records it and assert_chat_ui adds it to the scrub set before it can reach a log. test_the_session_remembers_which_password_is_current pins both directions: the retired session reports the replacement it minted, and a session that never had to change reports the password it logged in with. Still failing, and NOT downstream of the 403 after all: gguf_export reports install_kind=None on a run where inference, tool calling and training all passed, so no llama.cpp is installed under STUDIO_HOME regardless of the auth gate. I said last time that this might resolve itself once the 403 was fixed. It did not. It is a separate gap and it stays open. 90 passed. * Record what launch blocking did to the GRPO leg: it removed the fault The UNWIRED note asked for one experiment: run the leg with --cuda-launch-blocking so the illegal memory access reports at its real call site instead of surfacing at torch.cuda.empty_cache(). Done, kernel unsloth-t4-ci-b1f23e34, and the answer is not the one the note expected. With blocking on there was no illegal memory access at all. engine_built true, three steps, same 13.8GB peak, same versions to the digit. A fault that disappears when the launches are serialised is a race, which is what one pass in four already suggested and which no amount of re-running the leg will pin down. The run also exposed a SECOND problem, and it is separate from the first. It failed on reward_std = [0.0, 0.0, 0.0] with grad_norm 0.0 at every step. The completions in the report are coherent prose, so this is not the model collapsing into degenerate output. It is two completions scoring identically. The leg runs num_generations = 2 because that is what fits on a 14.56GB card, and at two samples a tie on a coarse reward is ordinary rather than a bug. So the leg's pass criterion is fragile at the size it has to be to fit, which is worth knowing before anyone reads a red grpo leg as a regression. The note now records both, says which question each answered, and states the two that remain open: where the race is, and what pass criterion is honest at num_generations = 2. The test gains the fourth session id and a check that the note no longer reads as though launch blocking were still the next thing to try -- a stale "try this next" is how an experiment gets run twice. 145 passed. * Re-seed the account before the UI driver, because the two want opposite states Third hardware run. The API assertions all pass now: preflight, studio_ready, authenticate, gpu_inference, tool_calling, lora_training -- all PASS and the Playwright driver still fails, for a third and final reason in this chain: [ui] STEP change-password through UI (Setup your account) [ui] change-password form attempt 1 failed: Timeout 60000ms exceeded waiting for locator("#new-password"); page.url=.../login The driver's first UI step drives the forced-change form. authenticate() has to retire that same bootstrap password over the API to reach /api/inference/load and /api/train/start at all. So by the time the driver runs, Studio shows an ordinary login and #new-password does not exist. That is not a value problem, it is a state problem, and the previous commit only fixed the value. The API path wants the change DONE; the driver wants it PENDING. One account cannot be in both states, and three runs walked the whole chain to get here: 412345d2 failed the API assertions on the gate, 9ddd8ae4 fixed those and failed the driver on a stale password, this one fixed the password and failed the driver on the form being gone. assert_chat_ui now restarts Studio before handing over. start_server() removes $STUDIO_HOME/auth, which is what re-seeds the bootstrap password, so the driver gets exactly the state its first step expects. It is cheap here specifically: this assertion runs last and stops the server anyway, so nothing after it needs the API session, and the driver loads its own model through the UI. test_the_ui_driver_gets_a_freshly_seeded_account pins the restart, that the password is read AFTER it, and that the retired session's password is not what gets handed over -- the restart is the whole fix, and a refactor that drops it puts the payload straight back to a driver that cannot find the form. Unchanged and still open: gguf_export reports install_kind=None. Three runs now, one of them with inference, tool calling and training all green, so no llama.cpp is installed under STUDIO_HOME and this is not downstream of the auth gate. 91 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Test the newest transformers and trl, not the newest zoo permits The canary leg is documented as "the newest permitted library set" and that is exactly what it installs: it resolves WITH unsloth_zoo in the resolution, so zoo's own metadata is a ceiling. Measured against the two runs that carried it: canary transformers 5.5.0 trl 0.24.0 peft 0.20.0 accelerate 1.14.0 control transformers 5.5.0 trl 0.24.0 peft 0.19.1 accelerate 1.13.0 PyPI the same day: transformers 5.15.0, trl 1.9.2, peft 0.20.0, accelerate 1.14.0. The canary moved peft and accelerate to genuine latest and moved transformers and trl not at all, which is what made it look like it was working. zoo pins transformers <=5.5.0 and trl <=0.24.0, ten minors and a whole major behind, so this CI could not detect a transformers 5.6+ or trl 1.x regression: it never installed one. The new frontier leg upgrades transformers and trl with their dependencies and without zoo in the resolution, which clears the cap because pip enforces only the requirements of packages in the resolution. It runs the same SFT payload as control and canary, so all three are directly comparable. I expected it red. It is green. On a real T4: transformers 5.15.0, trl 1.9.2, datasets 5.0.1, ten steps, canary emitted, and two fresh processes agreeing bitwise. Unsloth trains and generates correctly a whole trl major above what zoo's metadata permits. Getting there took two kernels. The first used --no-deps plus a blanket tokenizers upgrade and died before running anything, because an unbounded upgrade overshoots the ceiling transformers declares: tokenizers<=0.23.0,>=0.22.0 is required, but found tokenizers==0.23.1 safetensors>=0.8.0 is required, but found safetensors==0.7.0 A test pins that shape out, along with the zoo-in-the-resolution mistake that would silently turn this into a second canary. It goes on the second kernel's idle T4, so it costs no quota: a Kaggle session bills wall clock once, not per card. What it does NOT catch is recorded in the leg: the loss trajectory is not the control's, and step 1 is computed before any update on identical weights, data and seed, so 10.3222 against 6.4367 is a different loss function rather than drift. Both converge. Which objective is intended is separate work. * Install a CUDA llama.cpp before asserting the export used one Four hardware runs reported llama_cpp_install_kind=None and failed the GGUF export assertion for it. The reason was never subtle: nothing had ever installed a llama.cpp under STUDIO_HOME, so the assertion was measuring an absence rather than a selection. install_llama_prebuilt.py resolves a real "linux-cuda" kind on an x64 CUDA host, so the bundle was available the whole time and simply never fetched. The install runs before the server, so the export route sees a llama.cpp that was already there rather than one appearing underneath it, and it is its own assertion because "the CUDA bundle would not install on this box" and "it installed and the export against it failed" are different findings. Its result is deliberately not checked by execute(). A box where the bundle cannot be installed still runs inference, tool calling, training and the UI, and still reports the export failing for the reason it actually failed. Three tests. One of them CALLS install_llama_cpp rather than reading its source, because run() in this payload already applies capture_output and text and passing either again is a TypeError that no source-reading test would catch -- which is how the first draft of this was written wrong. * Record what install.sh left behind before replacing it build_kernel.py runs install.sh --local, whose own comment says it puts a llama.cpp on disk. Four runs still reported install_kind=None, so either it never installed one or it installed something else. Fetching a CUDA bundle fixes both cases and would hide which one it was, so the prior state is recorded first and the original question stays answerable. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Look for llama.cpp where it is, not where the payload assumed Five hardware runs reported llama_cpp_install_kind=None and failed the GGUF export for it. There was no missing llama.cpp. install.sh --local installs one into ~/.unsloth/llama.cpp, which is what install_llama_prebuilt.py's own default resolves to, and this payload read STUDIO_HOME/llama.cpp, a path nothing ever wrote to. Adding an explicit install last commit is what proved it, because it recorded what was there first. The installer selected the right bundle for a T4 accept ...linux-x64-cuda13-older.tar.gz coverage=75-89 supported=75,80,86,89 reject ...linux-x64-cuda13-newer.tar.gz reasons=missing_sms=75 out_of_range_sms=75 and then said existing llama.cpp install already matches selected release b10360-mix-87da1a2; skipping download and install while install_kind_before and install_kind_after at the requested directory were both None and the return code was 0. So a CUDA llama.cpp was present the whole time, the sm_75 selection works, and only the assertion was wrong. llama_cpp_marker checks STUDIO_HOME first, then the canonical location, and returns None when neither has a marker so a real absence stays a real absence. Against the exact situation the last run recorded, the old read answers None and the new one answers linux-cuda. Separately, and not fixed here: install_llama_prebuilt.py --install-dir X exits 0 having written nothing to X whenever the canonical directory already holds a matching release. A caller who names a directory and gets a success with an empty directory has been told something untrue. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Read the field the marker has, not the one it never had Six hardware runs reported llama_cpp_install_kind=None and failed the GGUF export assertion for it. There was no missing llama.cpp and no wrong directory. install_llama_prebuilt.py writes install_kind ONLY into the JSON its resolver prints to stdout. The marker it writes to disk records requested_tag tag release_tag published_repo asset force_cpu llama_backend asset_sha256 source_* ggml_tree bundle_profile runtime_line coverage_class install_fingerprint prebuilt_fallback_used installed_at_utc and no install_kind at any point. So reading install_kind from the marker answered None for every bundle on every box, including a working CUDA one, and the assertion turned that into a failure. The last run had the installer saying "existing llama.cpp install already matches selected release b10360-mix-87da1a2; skipping download and install" while the payload reported nothing installed. runtime_line is the field that answers the question: cuda12 / cuda13 for the CUDA bundles, and the backend name otherwise. It is written into the same dict as the asset, so the two cannot disagree, and asset is kept as a fallback for a marker too old to carry it. is_cuda_install now matches the runtime line's shape rather than a fixed set of names. A fixed set needs a new entry per CUDA major and fails CLOSED when it lacks one, reporting a working cuda14 install as not-CUDA, which is the same failure this commit is undoing. The existing test wrote {"install_kind": "linux-cuda"} into a marker and asserted it came back. That is how this survived: the test agreed with the payload about a field neither read from reality, and stayed green through all six runs. It is rewritten against a marker the installer actually produces. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Measure the load, not the unload it follows The GGUF export probe reported device_vram_delta_mib: -1866.0 and was scored as a load that never reached the GPU. It had not: a 3004 MiB chat model was evicted inside the same window the 531 MB GGUF loaded in, because POST /load with force unloads first and the baseline was sampled before the request. Unload explicitly, wait for the reading to stop falling, then take the baseline. Best-effort throughout, since the delta is evidence and not the assertion: a refused unload or a missing nvidia-smi leaves the probe running. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Read the status fields the response actually declares Run 8 shipped the unload-before-baseline fix and came back with device_vram_delta_mib: -1866.0, byte-identical to run 7. The fix ran and did nothing: it looked for model_path, model and active_model_name, and InferenceStatusResponse carries none of those, so no model was ever identified and no unload was ever sent. Read model_identifier (documented as the loadable identifier, which is what /unload's model_path wants), falling back to active_model and then to loaded[0]. A new test parses the field names out of InferenceStatusResponse and asserts the payload only reads names it declares, so the next guess fails in the suite rather than forty minutes into a kernel. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Delete the Kaggle kernel when the launcher is killed, not only when it returns release() is the budget control: a kernel left behind bills to its own ceiling with nobody reading the result, and that ceiling has been observed not to stop a wedged one. But it was reachable only from finish(), which runs only on paths that RETURN. KeyboardInterrupt was explicitly re-raised past it, and the default SIGTERM disposition exits without running atexit or finally. So Ctrl-C, `kill`, and a cancelled GitHub Actions workflow all left the kernel running. The file's own comment claimed the kernels were released "on this path and on every other", which was true for returns and false for signals. Four paths, four covers: normal return / handled error finish() unhandled exception atexit Ctrl-C, kill, Actions cancel SIGINT/SIGTERM/SIGHUP handlers kill -9 the orphan sweep at the next launch The signal handler re-raises through SIG_DFL so the exit status still reads "killed by signal N"; exiting 0 would make a cancelled job look completed. kill -9 cannot be caught, so a pushed slug is now recorded to logs/kaggle_inflight.json the instant it exists, and every launch sweeps entries whose owning pid is gone. Keyed on pid deliberately: an entry whose owner is still alive belongs to a run in progress, and deleting it would destroy a concurrent run and report the absence as a failure of the code under test. A delete that fails keeps its entry rather than forgetting it. The sweep runs before pushing, so reclaimed session slots are available to the run doing the reclaiming. Tests drive real subprocesses and real signals, since the property is about process death. Verified failing against the previous behaviour: SIGINT, SIGTERM and the unhandled-exception case all leave the kernel behind without these handlers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only forget a Kaggle kernel once it is actually gone kaggle kernels delete answers a transient rejection with a nonzero exit and subprocess.run does not raise on that, so both cleanup paths recorded a refused delete as done: sweep_orphans dropped the registry entry and release_kernels set released=True, leaving a running kernel billing with nothing left to reclaim it. Check the return code in both. Two more holes in the same registry. result["kernels"] was published after the whole push loop, so a SIGTERM once the first kernel was up found no list and deleted nothing; it is now the same list object, assigned before the loop. And --keep-kernel left an entry naming a pid that dies with the launcher, so the next sweep called the retained kernel an orphan and deleted exactly what the flag asked to keep. * Stop the Studio leg passing on evidence it never had Six ways this job could go green without measuring anything. The evidence bundle was never unpacked: the launcher writes each kernel into kaggle_evidence/<slug>/ and the workflow hands collect_evidence.py the parent, so a top-level glob found nothing and every run reported that the payload emitted no bundle. Recurse, the way extract_reports already does. An install.sh regression read as infrastructure. sh() raised SystemExit, so papermill stopped before any T4_SMOKE_REPORT existed, the launcher filed the run as infra and the reporter exited 0 -- silently passing the installer failures this workflow's path filter selects for. Failures of the checkout under test now emit a failing report first; a missing GPU or a clone that would not download keep the no-report infra path. A payload killed by the driver deadline does the same, but only once the payload itself had started. The process-level VRAM probe could never fire: InferenceStatusResponse declares neither llama_server_pid nor pid, and FastAPI drops what the response model does not declare, so the lookup was always None. Find the llama-server processes in /proc instead. Offload evidence outlived its load. offloaded_layers() takes the last match in the whole log, so a reload that logged nothing inherited the chat model's line and passed on evidence from a different model. Read each load forward from a mark taken just before it. A diverged run counted as a trained one. A T4 has no bf16, and an fp16 run that goes NaN still reaches completed and still saves an adapter, while trained_steps counted anything that was not None. Only finite losses count, and non-finite ones are named as a failure. An export slower than its HTTP timeout crashed the payload. The route blocks for the whole export and the transport timeout is 900s against a 1200s deadline, so the TimeoutError escaped the StudioError handler; it is now treated as in-flight and the status poll decides. Alongside: the restart that re-seeds the account no longer truncates the backend log of every earlier assertion, the over-cap evidence fallback ships the logs it says it ships rather than the report alone, the result line is printed after the evidence is packaged so a packaging crash cannot publish a pass, the training and export polls stop when Studio dies, Studio is launched from the interpreter running the payload rather than whatever is on PATH, the gate is told this leg pushes one kernel rather than the default two, and the advertised opt-in label can start the workflow. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the encoding on every checked-in read, and survive a box with no nvidia-smi tests/test_source_read_encoding.py fails on 33 read_text() calls in the two Kaggle harnesses: unpinned text I/O on a repo file is cp1252 on a stock Windows install and dies the moment that file gains a non-ASCII byte. The second one is the payload's own. environment() runs inside finish(), on every path out of the run, so a session Kaggle handed no driver raised FileNotFoundError from gpu_inventory() and ended with a traceback and no report at all. run() now answers a missing binary the way a failing one does, and the preflight says there is no GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Spend a Kaggle session on the opt-in label, not on every label after it Two ways this leg spends or loses a GPU session it should not. GitHub has no per-label filter on the pull_request event, so adding `labeled` to the activity types starts the workflow for EVERY label added to an affected pull request. The gate is handed the whole label list, so once kaggle-studio-gpu-ci sits on the pull request it reads every one of those events as forced, and a routine label change launches another 70-minute Kaggle run. Guard the gate job on the label that fired the event; push and the other pull_request activities carry no `github.event.label` and fall through unchanged. The other way is the reverse: a run that should be red exits 0. The dependency probe raises out of the cell when the Studio venv reports no CUDA, and with no T4_SMOKE_REPORT the shared launcher files the run as `infra`. install.sh --local resolving a CPU-only torch is exactly the CUDA install regression this workflow's path filter selects for, and it took that path. Ask nvidia-smi which case it is: a GPU the host can see and the venv cannot use is a payload failure, and no GPU at all stays infra. * Treat a stalled kaggle push as transport, not as a red verdict Every other Kaggle transport failure -- a 503, a refused push, a capacity rejection -- returns a reason from push() and exits 0 as infra. A push that stalls past the 600s ceiling did not: TimeoutExpired escaped launch.py, so the process ended before finish() could write launch_result.json, the launch step exited nonzero and GitHub marked the job red. The reporter that reads that file and calls the run NOT RUN never got a say, and a red there says the code under test failed when nothing ever reached a T4. Catch it in push() and report push_timeout. * Do not let a truncated copy of a chunk overwrite the complete one * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the encoding on every checked-in file the T4 harness reads Repo tests (CPU) is red on this branch: test_source_read_encoding.py::test_checked_in_file_reads_name_an_encoding AssertionError: 17 file reads in the test trees touch a checked-in file with the platform default encoding, so they break on Windows as soon as that file gains a non-ASCII byte. All 17 are in tests/kaggle/test_t4_smoke_harness.py, which this branch adds, so this is ours and not inherited. The reads pull in the workflow YAML, the smoke scripts, the committed reference JSON and ci-preempt.json, and none of them said what encoding to read them as. On Linux that resolves to UTF-8 and nobody notices; on a Windows runner it resolves to the ANSI code page and the first non-ASCII byte in any of those files takes the test out with a UnicodeDecodeError that has nothing to do with what the test is checking. Every one now passes encoding = "utf-8", matching how the rest of the suite spells it. Reads of files the test itself just wrote under tmp_path are left alone: the gate only covers checked-in files, and those are ours end to end. test_source_read_encoding.py 1 passed tests/kaggle/test_t4_smoke_harness.py 148 passed * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Flatten Kaggle's kernel log before extracting the reports from it Kaggle's kernels/output returns the log as a JSON array of {stream_name, time, data} records, one record per line, so no line in the file starts with T4_SMOKE_REPORT. extract_reports read it verbatim, so the log fallback -- the one that covers a run whose executed notebook never came back -- recovered nothing and a failed assertion was scored as infra with the workflow exiting 0. report.py::kernel_log_text and collect_evidence.py::iter_text already flatten the records; extract_reports was the reader that did not. The records are also cut by write rather than by line, so the join has to happen before any line splitting. * Keep the slug a timed out kaggle push may have created A push whose CLI stalls past the 600s ceiling says nothing about whether Kaggle took it. Kaggle can accept the push and start the kernel billing before the response is lost, and the branch returned infra while keeping neither the slug nor an inflight entry, so finish() had nothing to delete and the next launcher's orphan sweep had nothing to see. The kernel ran to its ceiling with nobody reading the result. The slug is ours and already decided at that point, so record it in the inflight registry and hand it back as orphan_slug: release_kernels deletes it on the way out, and a launcher killed first leaves the entry for the next sweep. It is deliberately not returned as slug, because the caller must not wait on a kernel that may never have been created. * Align the Kaggle budget guide with the workflow it documents The guide named --percent 15 and --reserve-hours 25 against a 30h account; the workflow runs 40 and 20 against the 60h figure measured on 2026-08-11, and it is the only consumer of that quota today. Following the guide would have tuned the reserve from a baseline that no longer exists. Re-derived from the same live numbers as the workflow header, and says which of the two is the source of truth when they disagree. * Stand the gate down on any unreadable kernel status The concurrency verdict refused only when EVERY in-window status came back an error. One unreadable status among readable ones was treated as an idle account, and driving concurrency_verdict with {busy: [], own: [], foreign: [], surveyed: 2, unreadable: 1, complete: True} returned (True, ""). That kernel can be the human session the zero-foreign policy exists to yield to, and proceeding takes the account's last slot: the first push wins it, the second comes back at capacity, and the run reports a subset of its legs. A 404 is kept separate and stays benign. A deleted kernel is not an unknown state and the launcher deletes every kernel it pushes, so blocking on that would wedge the gate shut rather than make it careful. * Close six gaps in the Kaggle T4 notebook CI workflow - The pull_request trigger used GitHub's default activity types, which are opened, synchronize and reopened only. The kaggle-t4-ci label the gate advertises as a human override could therefore never start a run on its own. Subscribe to labeled as well. - pyproject.toml was in neither paths filter, though every payload installs the commit under test as a distribution built from it. BUDGET.md counted it as a watched path already. - The job deadline of 120 minutes was shorter than the launcher's worst case. Two sequential pushes can take 2 x (4 x 600s + 315s of backoff) and the 5400s polling window only starts after them, so about 187 minutes with the deletions; a runner killed at 120 takes finish() -> release() with it and leaves pushed kernels billing quota. 200, with the arithmetic written down beside it. - The gate surveys quota and in-flight kernels before t4-smoke queues on the account-wide concurrency group, so a second sampled run could launch on an approval a full run old. Re-ask with the slot in hand, immediately before the push; --force true skips the sampling draw and nothing else. - --zoo-ref main let each payload's own pip resolve a different unsloth-zoo commit, including between the control and the canary within one session, and zoo is not in pins/control.txt either. Resolve it once with git ls-remote and pass that SHA to every leg. - A dispatch with max_steps != 10 still shipped the 10-step reference, and check_reference reports step_count_mismatch as a failure, so the run could only ever go red on arithmetic. Drop the band automatically when the counts differ, and say so. Also: check out the same head SHA the launcher installs, so the harness and the package under test are one snapshot rather than the merge tree and the head; and run tests/kaggle/test_t4_smoke_harness.py on the runner before anything is built or pushed, since testpaths limits pytest to tests/security and nothing else collected it. * Read back what the Kaggle kernel actually reported Two ways a real payload verdict was being lost in transport, both ending in verdict=infra and a green workflow. The kernel log fallback could never work. Kaggle's kernels/output hands the log over as a JSON array of {stream_name, time, data} records, which is why report.kernel_log_text flattens it before reading; extract_reports scanned the file as text, so no line began with the report prefix and the fallback that exists for the run whose notebook never came back found nothing. Record boundaries are not line boundaries either, so the records are joined before the scan rather than read one at a time. push() documented a fresh slug per attempt and used one for all four. Pushing to an id that already exists does not replace it: Kaggle files a new version and starts a second batch session, and kernels status and kernels/output send no version label, so they answer for the newest session only. A retry after a lost response therefore collected the wrong execution's evidence while the first kept a session slot and billed unseen. Each attempt now takes its own slug and discards the previous one first, and every slug filed is returned and recorded, since a push that reported an error may still have landed. * Give each payload its own everything, and report a probe failure Four ways the two payloads of one kernel were not actually isolated, plus the import probe exiting without a verdict. The probe was the P1. When the commit under test breaks import unsloth, the verify cell raises before the run cell can write a report, the launcher extracts none, and no reports is classified as infra -- so a deterministic import regression exits green. A probe failure is a verdict, not missing evidence, so it now emits its own failing report. nvidia-smi failing or a one-GPU allocation fell back to N_GPU = 1 and pinned both payloads to device 0. Each child still saw exactly one card and passed its own visibility assertion, so the contention came back looking like a code failure. A shortfall is infrastructure and stands the kernel down before a thread starts, which is what the comment above it already claimed. A payload whose virtualenv could not be built kept the base python3 kernel and installed into the shared system site-packages. That is the one thing the per-child venv exists to prevent: the legs install deliberately different library sets, so the last writer wins and the resulting import error reads as a regression. The payload is skipped instead. Both payloads materialised into one source directory and compiled into one unsloth_compiled_cache. The sources are byte-identical copies but write_bytes truncates first, and the compile cache is a relative path resolved against a working directory both papermill children inherit while compiling the same modules against different transformers versions. Each payload now owns both, and the prune globs still reach them. Finally, --smoke-args is appended after every leg's own arguments, so the workflow's --max-steps 10 landed after the gpt-oss leg's --max-steps 3 and argparse took the last one. The 20B leg was training for ten steps rather than the three that were measured to fit. A shared argument no longer reaches a leg that sets that option itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 06:30:18 -07:00
Daniel Han	bb11b371e7	Run deterministic notebook smoke tests on real Kaggle T4s (#8440) ... * Run deterministic notebook smoke tests on real Kaggle T4s The notebooks are written for a T4: sm_75, fp16 with a gradient scaler, xformers rather than flash-attention, 16GB. Nothing in CI runs on that card, so a regression that only appears there reaches users first. Adds a gated job that launches one Kaggle GPU session and runs the smoke payload once per T4. The account has a 60h weekly budget shared with other consumers, so the job is a deliberate minority consumer of it: paths filters (only 9.9% of a week's 444 main commits touch unsloth/), a ~10% sampling draw keyed on the run id so a re-run cannot reroll, a quota floor read live from Kaggle's own API, and a stand-down when the account already has a kernel in flight. Expected cost is 5-7 GPU-hours a week. Only a payload that ran on a T4 and disagreed with its assertions turns the check red. Throttling, the 2-kernel concurrency cap, a dead session or a download that would not complete are all warnings, because a free external service blocking merges would be ignored within a week. The payload asserts three things, worth different amounts. Two fresh processes must agree bitwise on per-step loss and grad_norm; that one is exact, and it is fresh processes because two in-process cycles disagreed from the first step while two processes agreed on all ten. Greedy decoding must emit the canary string exactly, which catches trained weights never reaching the generate call. Metrics must sit inside a band around a committed reference, which is a tolerance and never an equality, since fp16 reduction order moves with the driver and the libraries. Ten steps rather than three: under fp16 the scaler skips the first two overflowing steps, so a three-step run lands one real update and the canary never forms. The dataset is supplied as prompt/completion columns so the loss falls only on the answer. * Make the payload cells the T4 rejected actually run there Two launches on real Kaggle T4s died in the scaffolding before a training step ran, both of them in generated code that nothing had ever parsed. The dependency probe imported unsloth_zoo before unsloth. Zoo's __init__ ends with a find_spec("unsloth") guard and raised "Please install Unsloth via pip install unsloth" on a session where unsloth was installed and imported cleanly one entry later, so the payload aborted over a dependency that was not missing. Probe unsloth first, and invalidate the import caches after the pip installs that this interpreter did not see at startup. The reference argument was generated as a shell fragment and spliced into the middle of a Python list literal, with doubled braces that left ROOT unsubstituted. The cell was a SyntaxError, and the workflow passes --reference on every path, so no payload could ever have reported. Verified locally by executing the regenerated run cell end to end: two fresh processes, bitwise identical metrics, canary exact. * Bound the in-flight kernel survey by session length, not kernel count The concurrency check status-checked the twelve most recently run kernels, which is a sample rather than a search: a kernel that started three hours ago and is still running is missed the moment twelve newer ones have since run, and the push then fails at the account capacity cap and is reported as infra. Kaggle kills a notebook session at twelve hours, and the listing is sorted by last run time, which for an unfinished kernel is when it started (measured: a kernel pushed at 10:05:19Z lists as 10:05:19.297). So walking until the first entry older than that ceiling covers every kernel that can still be in flight, and stopping there is exhaustive rather than arbitrary. An hour of slack is added for clock skew. Two silent ways to read an unanswerable question as a green light are closed with it. Hitting the page cap now marks the survey incomplete, and a survey where no status could be read at all is unknown rather than idle. Both stand down, which is the cheap direction: the next commit draws again in minutes. On the live account this now costs three status calls instead of twelve. * Stop the reference band check from passing NaN unconditionally The band comparison did its NaN handling by accident, and it got it wrong. Under fp16 the gradient scaler logs a NaN grad_norm on every step it skips, so a committed T4 reference genuinely contains NaN. Left to the arithmetic, abs(x - NaN) is NaN and NaN > tolerance is False, so those steps passed whatever they held, including the case the check exists for: a step that used to overflow and no longer does. Compare NaN to NaN as equal and NaN against a number as a deviation, and treat a field logged on one side only as a deviation too, since no tolerance covers a change in shape. The mapping from an out-of-band verdict to the failure list moves into reference_failures() so the path that turns the job red can be tested without a GPU. The references README now says plainly that no reference is committed, that one must not be borrowed from other hardware, and how to lift one out of a green run's evidence rather than spending a session on it. It also records a measurement: the absolute floor never engages on this trajectory. The smallest value on the ten-step curve is a loss of 0.1428 against a floor of 0.05, so the floor changes no verdict today and is kept for a configuration that does go lower. * Name the cause when a kernel finishes and reports nothing A kernel that ends COMPLETE with no payload report is the hardest outcome to read: the summary has no metrics to show and the reason sits in an artifact nobody downloads. It is also the outcome both real T4 failures took. Kaggle returns the kernel log as a JSON array of stream records rather than as text, so the interesting line arrives split across dozens of them and reading the file directly shows a wall of JSON. Flatten it, filter to the driver and payload sentinels plus tracebacks, and fold the tail into the job summary. Checked against the evidence of a real failed run: the SyntaxError that killed it is now visible without leaving the summary. * Cover the four T4 CI fixes with CPU-only tests The generated cells are now compiled on both the with-reference and without-reference paths, which is what would have caught a SyntaxError that instead cost a Kaggle session, and the dependency probe's module order is asserted. The in-flight survey gets a fake Kaggle client and the case that motivated it: one kernel still running behind forty newer finished ones, which a count-bounded scan misses and a time-bounded scan cannot. Also the window boundary, timezone-aware and missing timestamps, the page cap, and the difference between some statuses unreadable and all of them. The band check is now proved to fail rather than assumed to work: a perturbation half a band-width past the limit, a moved scaler skip pattern in both directions, a field that stopped being logged, and a length mismatch, each asserted through to the failure list. Three tests perturb the committed reference itself and skip, saying so, until one exists. 41 passed, 3 skipped. * Parse every generated cell on every build path before spending a session Two of the three Kaggle sessions spent so far died on generated code that nothing had ever parsed. The compile check now builds all three paths the generator has, using the workflow's own argument list verbatim, including the one where a committed reference exists and is carried inline as a fourth file: that path becomes live the moment a green run supplies the file, and would otherwise first be exercised on Kaggle. Parsing alone is not enough. A template hole that substitutes to a bare identifier parses and then NameErrors, which costs the same session, so every cell is also checked in execution order for names nothing before it defines. Both checks were confirmed to go red: the first on the exact shell-fragment-in-a-list-literal defect that cost session three, the second on a run cell reading a name no cell binds. The carried sources are also decoded the way the kernel will decode them and compared byte for byte with the repo, so the T4 cannot quietly run something other than what is committed. * Commit the T4 reference the first green Kaggle run produced Kernel danielhanchen/unsloth-t4-ci-e3c6661f, terminal state COMPLETE, both payloads passing on their own Tesla T4 of one session. Two fresh processes agreed bitwise on all ten steps on both cards, max_abs_diff exactly 0.0 for loss and for grad_norm, and the two cards independently produced the same ten values as each other, so four processes agree rather than two. All four cycles emitted the canary exactly. The scaler skipped steps 1 to 3 every time, which is the behaviour the ten-step length exists to accommodate. The file is reports[0] of that run, copied whole by the recipe the README already documented, and nothing in it is from other hardware. One assumption the run contradicted, now recorded rather than repeated: the session had no xformers. unsloth installs --no-deps, unsloth_zoo does not carry xformers and the Kaggle image does not either, so the banner read Xformers = None and this trace is the fallback attention path. Adding xformers to the install would move these numbers and require a recapture. The floor's smallest observed value moves from 0.1428 to 0.0871, still above the 0.05 denominator floor, so the floor stays inert and the test that re-derives that keeps passing. The three tests that perturb the committed reference now run instead of skipping. 45 passed, 0 skipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the in-flight tolerance the Kaggle gate stands down on The gate declines to launch whenever any kernel of the account is in flight, which is stricter than Kaggle's 2-kernel cap requires. That is the intended policy, so it is now a named constant with the tradeoff written next to it rather than an implicit property of the code, and a --allow-in-flight argument that cannot be raised to the cap itself. Default behaviour is unchanged. Also files the workflow under 'never' in ci-preempt.json: cancelling a run does not stop the Kaggle kernel it already pushed, and the orphan then bills quota to its own ceiling with nobody watching. * Run three steps, and refuse a reference captured at another count max_steps drops from 10 to 3. Measured consequence: under fp16 the gradient scaler starts at 65536 and skips every step it overflows on, and the committed reference shows steps 1, 2 and 3 all skipped, so a 3-step run of the old configuration applied zero optimizer updates. --init-loss-scale pins the scaler below the overflow point so those three steps are real updates, and optimisation_failures() fails a run whose every step was skipped rather than letting it report as training. The committed reference was captured at 10 steps and no longer describes the run. check_reference now reads the max_steps the reference records and refuses to compare across counts, as a failure with both counts named -- including when the reference does not say what it was captured at. A quiet pass there would be a green check comparing a run against a curve it has nothing to do with. The reference is therefore stale until one recapture run. The workflow gains skip_reference_band for exactly that dispatch, and the recipe is in references/README.md. Also brings the workflow onto the repo's conventions: pinned action SHAs matching the rest of CI, a workflow-level concurrency group that deliberately does not cancel in progress, a fork guard on the gate job, typed dispatch inputs and an artifact name without the run id. * Keep the run at ten steps, and say why three does not work The committed reference is the evidence: the fp16 scaler reports grad_norm NaN at steps 1, 2 and 3 and a finite value from step 4. It starts at 65536, halves on each overflow, and skips the step it overflowed on, so a 3-step run applies zero optimizer updates. Loss stays around 10, the canary never forms, and the band would compare three points of a curve that never moved. Shortening also saves nothing worth having: a launch costs about 0.08h and that is almost all pip install, not training. The step-count guard, the all-steps-skipped guard and --init-loss-scale all stay, since they are what makes a shorter run possible later and what stops a vacuous one reporting green. The pin is off by default so the committed reference keeps applying. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Turn one T4 payload into four, and pair a pinned control with a canary The workflow ran a single tiny SFT payload on both cards of one Kaggle session and spent about 1.4 GPU-h a week doing it. The budget is now 40, and the question asked of it is wider: catch regressions in trl, transformers, accelerate, peft, bitsandbytes, torch and vllm, and catch torch.compile falling over on sm_75. Four legs now, two kernels of two T4s each, described once in .github/scripts/kaggle_t4_ci/legs.py rather than restated in YAML: control the existing tiny Qwen2.5-0.5B determinism run, pinned canary the same run on the newest library set Unsloth's own constraints allow gptoss gpt-oss-20b LoRA, for torch.compile and the forced float32 path grpo Qwen3-4B GRPO through a vLLM engine on the same card control and canary are one instrument rather than two tests. Same payload, same seed, same data, same step count, on the two cards of the same session; the only difference between them is the installed versions. A red canary beside a green control is a library release, and the job summary already puts the two version sets side by side so the bisect is a diff rather than an investigation. The reference band applies to the control alone: two library sets do not produce one fp16 trajectory, so the canary asserts what does not depend on versions instead. The pins are evidence rather than preference. transformers and trl are the pair the committed reference was captured with; peft, accelerate and bitsandbytes come from probe kernels 8161ceb9 and 7ab727f1, two independent T4 sessions that ran the identical base install and agreed. torch stays unpinned and recorded, because it comes from the Kaggle image and replacing it is the failure mode the grpo leg spent two probes on. The gate now separates this workflow's own kernels from everybody else's. It takes both of Kaggle's concurrency slots, and only when the account is otherwise idle: one kernel belonging to anyone else still stands it down entirely, which is the same policy as before stated at the right grain. * Leave the vLLM leg unwired, and stop trusting Kaggle's own kernel timeout Three probe sessions tried to run the Qwen3-4B GRPO leg on a T4 and none of them reached a training step. None of them failed for a reason to do with sm_75, memory or GRPO either: vllm==0.11.2 pins torch==2.9.0, so the leg has to replace the Kaggle image's torch, and that is what killed all three. With the image's site-packages visible, pip treats torch's pinned NVIDIA runtime packages as already satisfied by the copies belonging to 2.10, and the result is a torch that installs cleanly and cannot be imported -- libcusparseLt.so.0 first, then ncclCommWindowRegister once that one was named explicitly, identically on vllm 0.11.2 and 0.15.1. An isolated venv resolving the whole stack got further and then the session wedged. So the leg keeps its payload, its assertions and its install, and moves to legs.UNWIRED with the three kernel slugs and what each one measured. Wiring it now would make the check permanently red and spend the budget doing it. The second kernel keeps its spare T4, which costs nothing, and that is where the leg goes when the install works. The third probe also cost about an hour of quota teaching us that the push-time kernel timeout is not a budget control. Pushed with -t 5400, its own nbconvert crashed at t=406s and the session sat in RUNNING for over two hours -- past that ceiling and past the launcher's own deadline, because one status call had blocked with no socket timeout to bound it. Deleting the kernel by hand took the account's used-hours figure back down, which is the measurement that settles which control is real. The launcher now sets a socket timeout and deletes every kernel it pushed on every path out, and the workflow header says which of the three bounds is load-bearing. Sampling recomputed on the payload that actually ships: 40%, from measured per-kernel cost rather than the old estimate, with the note that wiring a fourth leg means recomputing it rather than editing one line. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-solve the GRPO leg's install so it stops replacing the image's torch Three probe sessions never reached a training step, and all three died the same way: vllm==0.11.2 pins torch==2.9.0, so the leg had to replace Kaggle's torch 2.10.0+cu128, and pip treats the image's NVIDIA runtime packages -- which belong to 2.10 -- as satisfying the new torch's pins. That imports as libcusparseLt.so.0 missing, then as libtorch_cuda.so undefined symbol ncclCommWindowRegister one package along, then as an isolated venv that spent an hour of quota resolving a CUDA stack and never produced payload output. None of that is a question about sm_75, memory or GRPO. It is one line: no vLLM release in the 0.11-0.16 range pins the torch the image ships. Releases 0.17.0 through 0.19.1 pin torch==2.10.0 exactly, so pin 0.19.1 and the leg replaces nothing, keeps system_site_packages, and installs like any other. Drop xformers with it. Its vLLM attention backend was deleted in 0.12.0, so it would be a package nothing selects. sm_75 has no FlashAttention and no FlashInfer, and the ladder in vllm/platforms/cuda.py falls through both to TRITON_ATTN; the leg names that in VLLM_ATTENTION_BACKEND so a release that reorders it goes red here rather than quietly selecting something else. 7.5 is still in CUDA_SUPPORTED_ARCHS at v0.19.1 and fp16 is a supported dtype below capability 8.0. Still unwired. Two things are static evidence rather than a run: whether 0.19.1 starts on sm_75 at all, and whether 8GB of 16-bit weights plus an engine plus a LoRA trainer fit in 14.56GB. UNWIRED says so in those words, and a test asserts the note still names the pin and still says what is unknown, so nobody wires this off prose that outlived its measurement. Also ports the vacuous-test fix from the stacked Studio branch: pre-commit.ci reformatting default=10 to default = 10 stopped the payload regex matching, so the step-count agreement check was comparing nothing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop the GRPO leg JIT-compiling flashinfer on a machine that cannot link it First probe on the re-solved install, kernel unsloth-t4-ci-e2d9ce9b. The install works: torch stayed at the image's 2.10.0+cu128, vllm 0.19.1 installed and imported, xformers was absent as intended, and the payload ran on a real Tesla T4 reporting capability 7.5 with TRITON_ATTN accepted. It reached engine construction, which is further than any earlier probe got by the whole width of the install. It died in flashinfer 0.6.6's JIT, and not for an sm_75 reason. All three .cu files compiled cleanly for -gencode=arch=compute_75,code=sm_75. The LINK failed: /usr/bin/ld: cannot find -lcuda -L/usr/local/cuda/lib64/stubs is already on the command line, so the image has no driver stub libcuda.so, only the runtime libcuda.so.1. Nothing about that is fixable from a payload. Set VLLM_USE_FLASHINFER_SAMPLER=0 so nothing JITs. The sampler has a native path, and skipping the build also saves a four-file nvcc compile inside a session billed by wall clock. UNWIRED records what the probe measured and narrows what is left to one question: whether the engine builds now. * Make -lcuda resolvable so flashinfer can link what it compiled Second probe, kernel unsloth-t4-ci-916d5986, failed identically to the first with VLLM_USE_FLASHINFER_SAMPLER=0 set. That is the informative part: the JIT is not reached only through the sampler, so switching off one consumer is whack-a-mole. The failure was never sm_75. Both sessions compiled all three .cu files cleanly for -gencode=arch=compute_75,code=sm_75 and died at the link: /usr/bin/ld: cannot find -lcuda -L/usr/local/cuda/lib64/stubs is already on the command line; the image ships no libcuda.so at all, only the versioned libcuda.so.1, and the linker will not resolve -lcuda against a soname. Normally the toolkit's driver stub fills that gap; this image has the directory and not the file. So symlink libcuda.so at the real driver in a scratch dir and put it on LIBRARY_PATH, which gcc and ld search for -l. No root, nothing written into /usr/local, and it fixes every flashinfer op rather than one. Linking against the real driver instead of a stub is correct here: the driver is present, which is the whole reason a stub would have substituted for it. Also widen the captured traceback to head AND tail. The last probe's 6000-char tail was entirely ninja's own output, so the Python frames naming the caller were exactly what got dropped. Probe 2 also measured the fit question the notebook raised: peak reserved 7.96GB of 14.56 with load_in_4bit false, so 16-bit weights are not what is short of room here. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only count a libcuda the linker will actually search for Kernel unsloth-t4-ci-d0d480b6: the shim reported already_linkable and did nothing, and the link failed anyway. It had found libcuda.so in /usr/local/cuda/compat -- which is real, and which nothing passes with -L. The failing ninja line names /usr/local/cuda/lib64 and /usr/local/cuda/lib64/stubs, and those are the only two directories that can answer -lcuda. So the precondition now checks exactly those two, and compat becomes what it is actually good for: a symlink TARGET when ldconfig and ctypes both come up empty. A library the linker will not search for is not a library the linker can find. Two tests: one reproduces the compat-only machine and asserts the shim is built anyway, one pins the searched directory list, since widening it is precisely how the check went wrong. * Give the base model a chat template, and record what the link shim did The link shim works. Kernel unsloth-t4-ci-27b0dc2e reported libcuda_shim: needed true, applied true, searched [/usr/local/cuda/lib64, /usr/local/cuda/lib64/stubs], real /usr/local/cuda/compat/libcuda.so and flashinfer is gone from the failure entirely. Memory went from 7.96GB to 11.36GB of 14.56, which is the vLLM engine actually loading, and the run reached trainer.train() -> _run_epoch -> training_step. Both questions legs.UNWIRED was holding this leg for -- does vLLM start on sm_75, does it fit in 14.56GB -- are answered by that. It then failed on the payload's own gap: ValueError: Cannot use chat template functions because tokenizer.chat_template is not set unsloth/Qwen3-4B-Base is a base model and ships none. The notebook this leg comes from solves it with an SFT priming stage that installs a template before GRPO starts; this leg has no priming stage, so it sets a minimal ChatML template directly and records in the report which of the two worlds the run was in. The base model stays. GRPO on an instruct model would measure the instruct tuning as much as the run, and the payload's rewards are format-and-digit rewards a base model can move inside three steps. A test renders the template rather than matching it as a string: a template that does not render would trade a failure at step 1 for a failure at step 1 with a longer traceback. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Wire the GRPO leg: it runs on a T4 and the rewards move Kernel unsloth-t4-ci-53efcc4e passed on a real Tesla T4. Step 2 logged reward_std 0.707 and grad_norm 0.772, peak 13.60GB of 14.56GB, three steps in 192s, and fast_generate returned coherent text. reward_std is the criterion rather than loss, because TRL's GRPO loss is ~0 by construction at num_iterations=1 and beta=0 and would pass on a run that learned nothing. The config in args is the one that fit, not the notebook's. Two probes with seq 2048, 4 generations, rank 32 and utilization 0.9 died in the backward at unsloth_zoo/gradient_checkpointing.py:1013, peaking at 15.97GB in 16-bit and 19.25GB in 4-bit. 4-bit alone is not the lever: it peaked HIGHER, because quantizing weights does nothing for activations while utilization 0.9 still hands vLLM ~13GB up front, and UNSLOTH_VLLM_STANDBY returns the weights during training but not the KV cache reservation. An earlier probe reported this as 'CUDA error: an illegal memory access was encountered' from empty_cache() inside vLLM standby. That was the async shadow of the same OOM; CUDA_LAUNCH_BLOCKING=1 produced the honest error. UNWIRED is now empty. The test that read its grpo entry is replaced by the invariants that outlive any one leg: every leg is wired or explained, nothing is both, and an unwired note says what is open. Two new tests pin the config and the vLLM pin, so restoring either to a value that OOMs fails here rather than on Kaggle. * Add a CUDA_LAUNCH_BLOCKING switch to the GRPO payload An illegal memory access is reported at whatever CUDA call synchronises next, which on the first T4 GRPO run was empty_cache() inside vLLM standby, nowhere near the kernel that faulted. Re-running with this flag produced the honest error instead: torch.OutOfMemoryError in the backward. That is what turned an unexplained CUDA fault into a memory budget, and the budget is now the leg's config. The environment variables are set before the first import torch in the process rather than merely before train(), because they have to precede CUDA context creation. It serialises every kernel launch, so it stays a switch and never a default, and it is recorded in the report because it changes what a timing means. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Write down where the Kaggle sampling percentages come from The percentages in the two workflows were arithmetic nobody could check. This records the measurements they came from, taken 2026-08-11 over a 28-day window: 985 commits to main, 1157 PRs, and 8 of 60 sampled PRs touching the paths a T4 training run could regress, so a 13% path-filter rate and roughly 75 candidate events per week. Against 30 GPU-h/week for one account at ~0.75 GPU-h per launch, the ceiling is about 33 launches per week if everything went to one workflow. The 15/5 split and the 25/10 reserve-hour floors follow from that, and the file says what would change them: a second account, a wider path filter, or a session cost above ~1 GPU-h. * Give the GRPO leg its own kernel: pairing it with gpt-oss fails Kernel unsloth-t4-ci-70a2f4eb ran the wired pair on the two cards of one session. gptoss passed: 32 unique graphs, 779 calls captured, 2 graph breaks, peak 12.78GB. grpo failed with the illegal memory access again, at exactly the 13.60GB peak at which it PASSED alone on kernel unsloth-t4-ci-53efcc4e, same config to the flag. Same peak, different outcome, so the constraint is not GPU memory: the driver pins one card per payload. gpt-oss offloads to host RAM and vLLM wants host RAM, and a Kaggle session has one host. A third kernel costs a queue slot rather than quota, since Kaggle runs two at a time and bills wall clock per session. That is cheaper than a leg which fails half the time for a reason the report cannot show. Two tests pin the rule in both directions: grpo must not share a kernel with gptoss, and control must still share one with canary, because those two are a matched pair whose comparison is the whole instrument. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Unwire the GRPO leg: running it alone reproduced the failure I blamed on pairing I gave grpo a kernel of its own earlier today on the reasoning that sharing a session with gpt-oss was what broke it. It had failed paired (unsloth-t4-ci-70a2f4eb) and passed alone (unsloth-t4-ci-53efcc4e), so pairing looked like the variable, and I wrote two tests pinning the separation. That reasoning was wrong. Running the leg ALONE again (unsloth-t4-ci-c98f14be, built from this branch with --legs grpo) reproduced the paired failure exactly: unsloth_zoo/vllm_utils.py:601 sleep() -> torch.cuda.empty_cache() torch.AcceleratorError: CUDA error: an illegal memory access was encountered engine_built false, peak 13.8GB/13.6GB of 14.56GB. The passing session and both failing ones agree to the digit on every recorded version -- torch 2.10.0+cu128, transformers 5.5.0, trl 0.24.0, peft 0.19.1, vllm 0.19.1, unsloth 2026.8.15, zoo 2026.8.10 -- on the same peak, with UNSLOTH_VLLM_STANDBY=1 throughout. One contrasting observation was never enough to blame a shared host. So what three sessions show is an intermittent illegal memory access on Turing, one pass in three, not a pairing effect. A leg that passes one session in three cannot tell CI anything: it would go red for a reason no reader could act on. grpo moves to UNWIRED with the three session ids and the open question written down, rather than back into a kernel. The note keeps what the leg already earned -- the vLLM/torch pin, the attention backend, the libcuda shim, the chat template, the T4-sized config -- so re-wiring means answering the IMA rather than redoing the install. test_grpo_does_not_share_a_session_with_gptoss is replaced rather than deleted: it asserted the opposite thing for the disproven reason, and would now pass vacuously since grpo is in no kernel at all. The replacement pins that grpo stays unwired and that the note still names all three sessions, so the evidence cannot be dropped on the way back in. 145 passed. * Record what launch blocking did to the GRPO leg: it removed the fault The UNWIRED note asked for one experiment: run the leg with --cuda-launch-blocking so the illegal memory access reports at its real call site instead of surfacing at torch.cuda.empty_cache(). Done, kernel unsloth-t4-ci-b1f23e34, and the answer is not the one the note expected. With blocking on there was no illegal memory access at all. engine_built true, three steps, same 13.8GB peak, same versions to the digit. A fault that disappears when the launches are serialised is a race, which is what one pass in four already suggested and which no amount of re-running the leg will pin down. The run also exposed a SECOND problem, and it is separate from the first. It failed on reward_std = [0.0, 0.0, 0.0] with grad_norm 0.0 at every step. The completions in the report are coherent prose, so this is not the model collapsing into degenerate output. It is two completions scoring identically. The leg runs num_generations = 2 because that is what fits on a 14.56GB card, and at two samples a tie on a coarse reward is ordinary rather than a bug. So the leg's pass criterion is fragile at the size it has to be to fit, which is worth knowing before anyone reads a red grpo leg as a regression. The note now records both, says which question each answered, and states the two that remain open: where the race is, and what pass criterion is honest at num_generations = 2. The test gains the fourth session id and a check that the note no longer reads as though launch blocking were still the next thing to try -- a stale "try this next" is how an experiment gets run twice. 145 passed. * Test the newest transformers and trl, not the newest zoo permits The canary leg is documented as "the newest permitted library set" and that is exactly what it installs: it resolves WITH unsloth_zoo in the resolution, so zoo's own metadata is a ceiling. Measured against the two runs that carried it: canary transformers 5.5.0 trl 0.24.0 peft 0.20.0 accelerate 1.14.0 control transformers 5.5.0 trl 0.24.0 peft 0.19.1 accelerate 1.13.0 PyPI the same day: transformers 5.15.0, trl 1.9.2, peft 0.20.0, accelerate 1.14.0. The canary moved peft and accelerate to genuine latest and moved transformers and trl not at all, which is what made it look like it was working. zoo pins transformers <=5.5.0 and trl <=0.24.0, ten minors and a whole major behind, so this CI could not detect a transformers 5.6+ or trl 1.x regression: it never installed one. The new frontier leg upgrades transformers and trl with their dependencies and without zoo in the resolution, which clears the cap because pip enforces only the requirements of packages in the resolution. It runs the same SFT payload as control and canary, so all three are directly comparable. I expected it red. It is green. On a real T4: transformers 5.15.0, trl 1.9.2, datasets 5.0.1, ten steps, canary emitted, and two fresh processes agreeing bitwise. Unsloth trains and generates correctly a whole trl major above what zoo's metadata permits. Getting there took two kernels. The first used --no-deps plus a blanket tokenizers upgrade and died before running anything, because an unbounded upgrade overshoots the ceiling transformers declares: tokenizers<=0.23.0,>=0.22.0 is required, but found tokenizers==0.23.1 safetensors>=0.8.0 is required, but found safetensors==0.7.0 A test pins that shape out, along with the zoo-in-the-resolution mistake that would silently turn this into a second canary. It goes on the second kernel's idle T4, so it costs no quota: a Kaggle session bills wall clock once, not per card. What it does NOT catch is recorded in the leg: the loss trajectory is not the control's, and step 1 is computed before any update on identical weights, data and seed, so 10.3222 against 6.4367 is a different loss function rather than drift. Both converge. Which objective is intended is separate work. * Name the encoding on every checked-in file the T4 harness reads Repo tests (CPU) is red on this branch: test_source_read_encoding.py::test_checked_in_file_reads_name_an_encoding AssertionError: 17 file reads in the test trees touch a checked-in file with the platform default encoding, so they break on Windows as soon as that file gains a non-ASCII byte. All 17 are in tests/kaggle/test_t4_smoke_harness.py, which this branch adds, so this is ours and not inherited. The reads pull in the workflow YAML, the smoke scripts, the committed reference JSON and ci-preempt.json, and none of them said what encoding to read them as. On Linux that resolves to UTF-8 and nobody notices; on a Windows runner it resolves to the ANSI code page and the first non-ASCII byte in any of those files takes the test out with a UnicodeDecodeError that has nothing to do with what the test is checking. Every one now passes encoding = "utf-8", matching how the rest of the suite spells it. Reads of files the test itself just wrote under tmp_path are left alone: the gate only covers checked-in files, and those are ours end to end. test_source_read_encoding.py 1 passed tests/kaggle/test_t4_smoke_harness.py 148 passed * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align the Kaggle budget guide with the workflow it documents The guide named --percent 15 and --reserve-hours 25 against a 30h account; the workflow runs 40 and 20 against the 60h figure measured on 2026-08-11, and it is the only consumer of that quota today. Following the guide would have tuned the reserve from a baseline that no longer exists. Re-derived from the same live numbers as the workflow header, and says which of the two is the source of truth when they disagree. * Stand the gate down on any unreadable kernel status The concurrency verdict refused only when EVERY in-window status came back an error. One unreadable status among readable ones was treated as an idle account, and driving concurrency_verdict with {busy: [], own: [], foreign: [], surveyed: 2, unreadable: 1, complete: True} returned (True, ""). That kernel can be the human session the zero-foreign policy exists to yield to, and proceeding takes the account's last slot: the first push wins it, the second comes back at capacity, and the run reports a subset of its legs. A 404 is kept separate and stays benign. A deleted kernel is not an unknown state and the launcher deletes every kernel it pushes, so blocking on that would wedge the gate shut rather than make it careful. * Close six gaps in the Kaggle T4 notebook CI workflow - The pull_request trigger used GitHub's default activity types, which are opened, synchronize and reopened only. The kaggle-t4-ci label the gate advertises as a human override could therefore never start a run on its own. Subscribe to labeled as well. - pyproject.toml was in neither paths filter, though every payload installs the commit under test as a distribution built from it. BUDGET.md counted it as a watched path already. - The job deadline of 120 minutes was shorter than the launcher's worst case. Two sequential pushes can take 2 x (4 x 600s + 315s of backoff) and the 5400s polling window only starts after them, so about 187 minutes with the deletions; a runner killed at 120 takes finish() -> release() with it and leaves pushed kernels billing quota. 200, with the arithmetic written down beside it. - The gate surveys quota and in-flight kernels before t4-smoke queues on the account-wide concurrency group, so a second sampled run could launch on an approval a full run old. Re-ask with the slot in hand, immediately before the push; --force true skips the sampling draw and nothing else. - --zoo-ref main let each payload's own pip resolve a different unsloth-zoo commit, including between the control and the canary within one session, and zoo is not in pins/control.txt either. Resolve it once with git ls-remote and pass that SHA to every leg. - A dispatch with max_steps != 10 still shipped the 10-step reference, and check_reference reports step_count_mismatch as a failure, so the run could only ever go red on arithmetic. Drop the band automatically when the counts differ, and say so. Also: check out the same head SHA the launcher installs, so the harness and the package under test are one snapshot rather than the merge tree and the head; and run tests/kaggle/test_t4_smoke_harness.py on the runner before anything is built or pushed, since testpaths limits pytest to tests/security and nothing else collected it. * Read back what the Kaggle kernel actually reported Two ways a real payload verdict was being lost in transport, both ending in verdict=infra and a green workflow. The kernel log fallback could never work. Kaggle's kernels/output hands the log over as a JSON array of {stream_name, time, data} records, which is why report.kernel_log_text flattens it before reading; extract_reports scanned the file as text, so no line began with the report prefix and the fallback that exists for the run whose notebook never came back found nothing. Record boundaries are not line boundaries either, so the records are joined before the scan rather than read one at a time. push() documented a fresh slug per attempt and used one for all four. Pushing to an id that already exists does not replace it: Kaggle files a new version and starts a second batch session, and kernels status and kernels/output send no version label, so they answer for the newest session only. A retry after a lost response therefore collected the wrong execution's evidence while the first kept a session slot and billed unseen. Each attempt now takes its own slug and discards the previous one first, and every slug filed is returned and recorded, since a push that reported an error may still have landed. * Give each payload its own everything, and report a probe failure Four ways the two payloads of one kernel were not actually isolated, plus the import probe exiting without a verdict. The probe was the P1. When the commit under test breaks import unsloth, the verify cell raises before the run cell can write a report, the launcher extracts none, and no reports is classified as infra -- so a deterministic import regression exits green. A probe failure is a verdict, not missing evidence, so it now emits its own failing report. nvidia-smi failing or a one-GPU allocation fell back to N_GPU = 1 and pinned both payloads to device 0. Each child still saw exactly one card and passed its own visibility assertion, so the contention came back looking like a code failure. A shortfall is infrastructure and stands the kernel down before a thread starts, which is what the comment above it already claimed. A payload whose virtualenv could not be built kept the base python3 kernel and installed into the shared system site-packages. That is the one thing the per-child venv exists to prevent: the legs install deliberately different library sets, so the last writer wins and the resulting import error reads as a regression. The payload is skipped instead. Both payloads materialised into one source directory and compiled into one unsloth_compiled_cache. The sources are byte-identical copies but write_bytes truncates first, and the compile cache is a relative path resolved against a working directory both papermill children inherit while compiling the same modules against different transformers versions. Each payload now owns both, and the prune globs still reach them. Finally, --smoke-args is appended after every leg's own arguments, so the workflow's --max-steps 10 landed after the gpt-oss leg's --max-steps 3 and argparse took the last one. The 20B leg was training for ten steps rather than the three that were measured to fit. A shared argument no longer reaches a leg that sets that option itself. * Count a moved step and a one-sided field as run-to-run differences compare_metrics zipped the two traces positionally and skipped any field that was not on both sides, so a grad_norm logged by one fresh process and not by the other left identical true, and so did a step coordinate that shifted, duplicated or reordered while the values stayed in place. Both are nondeterminism, which is the one thing this comparison exists to find, and check_reference already treats one-sided presence as a change in the shape of what the trainer logged rather than as a skip. * Assert the SFT leg's canary, adapter, repeats and reference properly Six gaps, all of them a green run that established less than it claimed. The canary was a substring test, so the exact, tolerance-free assertion the docstring describes twice accepted '__UNSLOTH__!!!<anything>' -- which is what a stopping or EOS regression produces. It is now an exact match after stripping surrounding whitespace, with both readings recorded so a red says which of the two it was. The saved adapter was only ever checked for a filename while inference ran on the in-memory model, and the caller-side reload the comment promised does not exist anywhere. The serialized weights are now read back and checked for tensors that are present, finite and not all zero; lora_B starts at zero, so an all-zero file is an untrained adapter that would reload without complaint. --repeat 3 compared cycles 0 and 1 and threw the rest away. Every extra cycle is now compared against the baseline, and the summary keys report.py renders off the reproducibility block keep their shape. optimisation_failures counted an infinite grad_norm as an applied update because inf equals itself, so a run whose every step overflowed to inf rather than NaN reported as a training run that trained. It requires a finite norm now, which is what the loss check three lines above already did. check_reference guarded max_steps out of a config block it was already holding whole. The learning rate, the optimizer, the LoRA shape, the model and the commit of the model repository that was actually read are compared on the same terms, and so are the per-step coordinates, all of them refusing before a single number is compared. Older references keep working: a key the file does not carry is listed unchecked rather than called a mismatch. A cycle that never wrote its report produced a failure report with no environment block, which is the case where knowing which library set died matters most. The fingerprint and the config are now read before the cycles and carried into that report. * Assert the float32 path, the training compile and the optimizer update The gpt-oss leg exists for Unsloth's FORCE_FLOAT32 behaviour on a card with no bf16, recorded the fp16/bf16/force-dtype state it ended up in on every run, and asserted none of it. A run that quietly went through fp16 instead logs finite losses, compiles and generates, and reports green while the one thing this leg uniquely covers was never exercised. It is now a failure, conditioned on the card rather than hardcoded to T4: where bf16 IS supported the patch not firing is correct. The compile assertion read Dynamo's process-global counters after training and asked for at least one unique graph. Loading a 20B checkpoint through Unsloth fills those counters long before trainer.train() is called, so a training path that fell back to eager entirely still satisfied it. The counters are now sampled before training too and the assertion is on the delta. grad_norm was captured in every metric entry and read by nothing. A run whose gradients are all zero keeps a finite loss, compiles, and generates text from the untrained base model, so the leg reported LoRA training it had not done. At least one logged norm must now be finite and non-zero, and as in the SFT leg the check declines to decide where grad_norm was never logged at all. * Generate through the trained adapter, and keep the engine verdict The final fast_generate passed lora_request=None. fast_generate is the vLLM engine's own generate and the trained LoRA reaches it only through save_lora plus load_lora, so the check generated from the base weights and passed whether or not the adapter could be transferred into the engine at all -- which is the second of the two questions this probe exists to answer. The request is built now, what happened is recorded, and a transfer that failed is a failure. engine_built was set on a local dict that train() only returned at the end, so any exception after the engine was built threw the fact away and the report recorded engine_built false. That is the opposite of what happened and it erases the distinction the probe needs between vLLM being unable to start on sm_75 and GRPO failing later. Facts are published into the report as soon as they are known. A reward on some steps satisfied the check that the docstring describes as a reward on every step: a step whose reward functions never ran was filtered out and the others covered for it. Every step that logged a loss must now carry a reward; the summary row train() appends carries no loss and is not a step. Every logged grad_norm being NaN also passed. Under fp16 on this card every step can overflow and be skipped while loss, reward and reward_std are all still logged and base-model generation still returns text, so the length check alone was satisfied by a run that updated nothing. * Take the reference from the control report, not from reports[0] extract_reports walks sorted(rglob(...)) over per-kernel directories named after a random Kaggle slug, and inside the control/canary kernel t4_canary_output.ipynb sorts before t4_control_output.ipynb. Driving the real extractor over a four-leg run puts frontier first. Following the recapture recipe as written therefore commits the canary, frontier or gpt-oss trace as the control reference, and every later run then band-checks against a curve it has nothing to do with. The recipe now selects the report whose label is control and asserts there is exactly one, and carries the model and the resolved checkpoint commit through with the config. Also 45 CPU-only tests for what the payloads assert, which is the half of the harness that test_t4_smoke_harness.py does not cover: it checks the launcher, the gate and the generated notebook, and this checks whether a given result dict is called a pass. All 45 fail on the parent of this series apart from the ten that pin behaviour that was already right. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Let two runs that both overflowed on the same step agree An fp16 overflow does not only log a NaN gradient norm. clip_grad_norm_ over a gradient holding an infinity returns infinity, so the same skipped step can come back as inf on both cycles of the same configuration. compare_metrics handled the NaN case explicitly and then subtracted everything else, and abs(inf - inf) is NaN, which is not equal to 0.0. Two bitwise identical traces were therefore reported as nondeterministic, and the summary printed DIFFERED with a max_abs_diff of 0.0 beside it. Equal values now short circuit before the subtraction. A one sided infinity, and two of opposite sign, still reach it and still come out as differences. * Start the deletion deadline before the first push, not after the last A Kaggle kernel bills from the moment the push is accepted, and the only control that has been observed to stop one is the launcher deleting it. That deletion was scheduled from a deadline started after every push had returned, so the first kernel got --max-wait on top of however long the second push spent in its retries: four attempts at the 600s subprocess ceiling plus the backoffs is about 45 minutes, which turns a 90 minute ceiling into 135 minutes of billing. The deadline is now one invocation-wide clock started before the first push, so time spent pushing comes out of the wait rather than being added to it. The reservation the gate makes was understated for the same reason. It was derived from the ceiling passed to Kaggle at push time, which this workflow already records as unreliable; it is now derived from the launcher's own bound, one --max-wait per kernel, and the harness suite keeps the two in step. * Delete every slug a push filed, not only the accepted one push() files a fresh slug per attempt and returns all of them, because a push that reported an error may still have landed: Kaggle answers an accepted push with a 5xx or a reset connection often enough to be a known issue. Nothing then reconciled that list. Cleanup read entry["slug"], which a failed push does not set at all, so two cases left a session running with nobody watching it: - the last attempt of a push that never succeeded, which is exactly the ambiguous one that made the retry necessary; - an earlier attempt of a push that later did succeed, since the discard before each retry is best effort and subprocess.run does not raise on a refused delete. Both keep one of the account's two concurrency slots and bill GPU quota to the kernel's own ceiling. Release now walks every slug the entry filed and records which of them are gone. A delete for a slug Kaggle never created is refused and costs one call, which is why this can be unconditional. The job deadline arithmetic is recomputed for the extra deletions and for the polling deadline no longer stacking on the pushes. * Report an install that cannot be resolved as a failed payload The install cell retried three times and then raised, emitting no payload report. The launcher extracts reports and, finding none for that leg, calls the run partial or infra and exits green -- so a commit whose distribution cannot be resolved, or a control pin set that stopped resolving, passed the one job that exists to test packaging metadata. Only the later verify cell reported a failure, and the install cell never reaches it. The exhausted branch now writes the same kind of structured report the dependency probe writes, naming the group and the last pip error, before it raises. The retries are also backed off. They were immediate, so all three landed inside the same upstream blip, and the third failure has to mean the resolution is impossible rather than that one minute was bad. * Keep the resolved versions in an import failure report report.version_table builds the per leg comparison out of the report dictionaries, not by scraping the kernel log, so a leg whose report carries no versions is missing from the one table that answers the question the control and canary pairing exists to answer: which release did the red leg have that the green one did not. The versions were computed and printed one line above the failure and then discarded. They are now kept in a name, carried into the report, and the computation is wrapped: a half installed distribution can make importlib.metadata raise, and losing the report to a diagnostic would put the leg back to reporting nothing at all. * Stand the run down when the zoo commit cannot be resolved The fallback to the branch name restored exactly what pinning the zoo revision removed. unsloth-zoo is not in the control pin file, and every payload pips independently on the kernel, so with main the control leg and the version canary can resolve two different zoo commits inside one session. That invalidates the control's reference band and the attribution the pairing exists for, and the report records a distribution version rather than a commit, so the drift is not visible after the fact either. The ls-remote is retried three times, since one blip is not a verdict, and a resolution that still cannot be made stands the run down: a warning, no kernels pushed, and a green job, which is how every other infrastructure stand-down in this workflow behaves. Nothing that spends a Kaggle session runs after it. * Run every CPU suite in tests/kaggle before spending a session The step named one file, and pyproject.toml restricts default discovery to tests/security, so the two suites added after it were collected by no invocation anywhere. Measured at this commit: the step collected 159 tests of the 227 in the directory. The 68 that ran nowhere are the transport and payload assertion suites, which cover the report extraction, the payload pass and fail rules, the reference checks and the per leg isolation. It runs the directory now, so a suite added next to these is picked up without this file being touched, and there is a test that collects whatever the step names and asserts every suite in the directory is in the result. Measured in a bare venv with pytest, transformers and CPU torch: 226 passed. * Record the kernel a recaptured reference came from, not the leg label The recipe wrote report["label"] into source_kernel, which is always the string control. That field is the only thing in the file pointing outwards, at the hardware execution its band was measured on, and it is what makes a recapture auditable while the evidence artifact is still around. The committed file carries a real slug, so following the recipe would have replaced it with a word that names nothing. The recipe now derives the slug from launch_result.json: each kernel collects into a directory named after the last segment of its slug and each leg's executed notebook is named after the leg, so the control leg's kernel is identified rather than guessed at, and the recipe refuses if that does not come out to exactly one. The recipe is now executed by the test rather than pattern matched, against a two kernel evidence tree. * Pin the reference model, and say what the band did not compare The committed reference carried no identity keys at all, so the checkpoint pin added with the reference config never ran on any workflow invocation and the band could return ok after the loaded revision moved. Two of those keys are unknowable without another T4 session, but one is not: the control leg passes no --model, so the reference belongs to DEFAULT_MODEL. It is recorded now and that comparison is live. resolved_checkpoint and resolved_revision stay unrecorded until the next recapture, because inventing a commit hash for a run that happened is worse than admitting the file does not carry one, and refusing on their absence would turn every run red over a reference that is otherwise sound. What changes is that the gap is no longer silent. A key present on one side only is a pin that did not run: it is recorded under config_unchecked in both directions now, including the one where the reference names a commit and the run could not read one, and report.py puts that list on the job summary next to the band verdict. A key neither side claims stays unreported, since nothing was asserted about it. * Record the count a fresh runner sees for the harness step The comment beside the step names what a bare venv with pytest, transformers and CPU torch collects, and it moved with the tests added alongside these fixes. Measured, not estimated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Record a push that times out instead of letting it kill the cleanup subprocess.run(timeout=600) raises TimeoutExpired rather than returning a failed push, and nothing caught it. Pushing the second notebook after the first was accepted therefore left main() without finish()/release() and without launch_result.json, so the accepted kernel billed to its own ceiling with nobody reading it, and the slugs the timed-out attempt had filed -- the most ambiguous ones there are -- were lost with the exception. The timeout is now a failed attempt like any other: the slugs stay in `attempts`, the retry discards the previous one, and release() reconciles what is left. The whole of main() past authentication also runs under a guard that still deletes and still writes the result on any unforeseen abort, since nothing in the workflow cleans up after this script. * Read the adapter itself, so a silent grad_norm cannot pass a leg `if norms and not applied` passes on an EMPTY list, and the gptoss and grpo legs had nothing else to fall back on: neither saves an adapter or reloads one, so a Trainer or TRL change that stops logging grad_norm takes the only evidence of an optimizer update with it and both legs go on reporting green. Finite losses, captured graphs, rewards and non-empty generation are all produced by the base model and the loader on their own. Both legs now fingerprint their LoRA parameters before training and after, and that reading decides it, with grad_norm as the fallback rather than the source. Silence is still not evidence of nothing applied, which is the position the earlier round took; it is now answered from the weights instead of guessed at. Only when BOTH instruments are gone is the leg red, and then for what it is: the training path it claims to cover can no longer be shown to have run. The tiny SFT leg is unchanged. It reads its saved adapter back off disk and fails on an all-zero one, which is the same question already answered. * Stand down on an unrelated label, and pin a dispatched ref to one commit Two ways this workflow spends more than it budgeted for. The `labeled` trigger fires for EVERY label, not just the opt-in one, so any label applied to an eligible pull request started a run and a fresh sampling draw. Once kaggle-t4-ci was present it was worse: it stays in the label list, so every later label of any kind arrived as an override and forced a session. The budget arithmetic counts pull request opens and pushes and no label activity at all. The gate now takes the event action and the label that arrived, and stands a `labeled` run down unless that label is the opt-in one. Every other action is unaffected. A dispatched unsloth_ref was forwarded to the payloads unchanged, and each payload pip-installs on the kernel by itself, so a branch name could resolve to a different commit in each of the four legs -- the control and the canary compared across two different Unsloths, invisibly, since the report records a distribution version rather than a commit. It is resolved once now, here, exactly as the zoo revision below it is, and a ref that cannot be pinned stands the run down rather than installing a moving branch. What a dispatch varies is still only the package under test: the harness stays on the checked-out tree, because most of it does not exist on older refs. Label names and the dispatched ref travel through the environment rather than being interpolated into the shell. * Say in the budget file what a label event costs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Close five ways a leg could pass on a run that trained nothing usable Every one of these is a report that stays healthy while the thing the leg uniquely covers did not happen. - A LoRA weight that went NaN or infinite was the strongest possible pass: the fingerprint returned a number, `NaN != finite` read as "the adapter changed", and update_verdict said `applied` on exactly the no-telemetry path it was written to decide. A non-finite reading is now its own verdict, it beats a healthy grad_norm, and both call sites fail on anything that is not `applied` rather than on the two verdicts they happen to know about. - The saved adapter was accepted on a nonzero tensor count, which lora_A satisfies at initialisation. An adapter whose B matrices were all zero or were dropped contributes exactly nothing and reloads as the base model, so the B matrices are counted specifically and their absence is unusable rather than fine. - UNSLOTH_FORCE_FLOAT32 was tested for being nonempty. The loader writes "0" on its ordinary branch before deciding whether to force, and every production consumer reads `== "1"`, so forcing could stop firing with fp16 and bf16 both still false and the leg green. - The compile assertion fell back to the absolute graph count when no pre-training baseline was read. That count is the loader's, so a training path that ran entirely eager passed in the one case where training could not be isolated. A missing baseline is now unverifiable. - Reference drift divided through infinities to NaN, and NaN > tol is False while max(worst, NaN) is worst, so a step that used to be finite and now overflows was accepted without even moving worst_rel. Equal signed infinities are the unchanged case; every other pairing is a deviation, decided before the division. * Report a payload that crashed, and re-emit its report on one line The run cell wraps the payload in a child process and is the only thing that sees the child's exit status. Two ways it threw that away: A nonzero exit with no report printed "NO REPORT WRITTEN" and nothing else. No report at all is `infra` at the launcher and one report of two is `partial`, and both leave the workflow green, so a CUDA segfault, a native abort or an OOM kill was accepted silently while the definitive exit status was in hand. The crash is now emitted as a `passed: false` report carrying the return code and a stderr tail. The recovery path for a report whose own compact line fell out of the retained stdout tail echoed the file verbatim. Every payload writes that file indented and the launcher parses whole lines, so it was handed a lone brace to decode and the definitive result was downgraded anyway. The file is parsed and re-serialized compactly, and a file that will not parse takes the crash path rather than disappearing. * Delete the kernels for real, and prove the ref before paying for it Four things this harness reported as done without ever establishing them. Release: subprocess.run does not raise on a nonzero exit, so the cleanup loop recorded every slug as released whatever Kaggle answered. The live case is worse than a transient refusal. kaggle==1.7.4.5 has no "kernels delete" subcommand at all (it landed in 1.7.5.0, Kaggle/kaggle-cli#762, first released in 1.8.0), so argparse answered every delete with "invalid choice: 'delete'" and exit 2, and the run still called the kernel released while it billed on to its own ceiling. delete_kernel now reads the exit code, retries a refusal, and a slug it cannot confirm gone is named in the log, in launch_result.json and in a workflow annotation. The pin goes to 2.2.4, because the version is load bearing and 1.7.4.5 could not do the job. Verified against the live API: 1.7.4.5's authenticate() refuses KAGGLE_API_TOKEN, the only credential this workflow has, and demands a kaggle.json nothing here writes; quota_view(), which gate.py reads the remaining accelerator hours from, exists only on 2.x, so below it the call raised AttributeError into a handler that recorded "quota unreadable" and let the run proceed. Dispatched ref: a 40-character SHA was accepted on shape alone and nothing asked whether unslothai/unsloth has that commit, so a mistyped one pushed the paid kernels, failed every payload's git install, and reported the pull request red for a commit that never existed. git ls-remote cannot answer that (it matches refs and exits 0 with empty output for any SHA); a git fetch of the object can, and it is the same reachability pip needs. Applied to the ls-remote result too, not only to the SHA the form supplied. GPU probe: the verify cell's bare device_count assert aborted before the run cell could emit its fallback report, so a CPU-only or CUDA-incompatible torch wheel left the launcher with no report, which is infra, which exits 0. It now emits passed: false the way the import probe above it does, for any failure in the cell rather than the count alone. gpt-oss: the forced-float32 assertion was conditioned on bf16_supported is False, and main() records environment = {"error": ...} for the whole probe when it raises. An unreadable card reading therefore skipped the one check this leg uniquely carries while training, losses, adapter and generation all still passed. Anything that is not a literal True or False is now a failure, and the healthy floor fixture carries the reading so every case below it exercises the check. tests/kaggle: 292 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Say the same things about this CI in fewer lines Comment-only pass over the Kaggle T4 notebook CI: the prose blocks that explain the leg pairing, the budget arithmetic, the cleanup contract and the payload assertions were several paragraphs where a few lines carry the same meaning. Every measured fact stays: the kaggle client pin and the three things 1.7.4.5 cannot do, the ls-remote vs fetch behaviour on a bogus SHA, the vLLM torch pin windows, the recorded losses, peaks, kernel ids and version boundaries. No code changes, verified by an AST comparison over every touched file. * Count both deletion paths in the deadline that protects them The job deadline exists so the runner is never killed during release(), which is what deletes the kernels and stops the billing. Its arithmetic counted one delete per filed slug and none of the discards, so it was about half the launcher's real bound: one delete DELETE_ATTEMPTS 3 x 180s plus backoffs 5 + 10 = 555s push, per notebook 4 x 600s + backoffs + 3 discards = 4380s two notebooks = 8760s release(), 4 filed slugs per kernel = 4440s evidence ~600s total 13800s That is 230 minutes against a 200 minute deadline, so a wedged run could be killed mid-cleanup and orphan the kernels the deadline is there to get deleted. The deadline moves to 260, the delete ceiling gets a name, and the harness test now derives every term from launch.py's constants instead of hardcoding one call at 180s, so it can fail when they move. The gate had the matching hole: no socket timeout, and the Kaggle client has none of its own, so a stalled call blocks forever, --soft-fail never sees an exception and the 10 minute job timeout reports red on a workflow whose contract is that only a failed assertion on a T4 is red. It now sets the deadline before the first call, as launch.py does, and the in-flight survey gets a wall-clock budget: running out of it stops the walk as incomplete, which is already a skip, rather than being killed by the job. tests/kaggle: 296 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep three non-payload outcomes from costing quota or a red check Three ways this workflow could spend a Kaggle session, or colour a pull request red, for something that is not the code under test. push() filed a slug and only published it by RETURNING. The retry loop handles the subprocess timeout, so every other raise (subprocess.run with text=True decodes strictly and raises UnicodeDecodeError on a malformed response; the runner can answer OSError or MemoryError) unwound past the line that recorded the entry, and release() then iterated a list with no entry for that notebook at all. A kernel Kaggle may have just accepted was left billing to its own ceiling. The list is now the caller's: main() publishes the entry before the push and push() fills it as it files each slug, so reconciliation no longer depends on the call returning. max_steps is free text on a dispatch and was forwarded unchecked. A value that is not an integer reaches the payload's argparse, which exits 2 with no report, and the generated cell reports that crash as a failing leg on purpose, so three legs come back red after the kernels were pushed and the model downloaded. A value below the fp16 scaler's leading skipped steps applies no optimizer update at all, and the payload's own checks fail it. check_steps.py answers both before the recheck, and MEASURES the floor as the shortest prefix of the committed reference that optimisation_failures accepts (5, on the committed trace) rather than restating a number that would go stale as soon as the reference moved. The build step also takes the value through the environment now rather than interpolating it. The evidence upload had no continue-on-error, so an artifact-service outage failed the job, and the job is what the pull request shows. The verdict is Report's, which runs under always() and can still conclude pass. It is best effort now, and says so out loud: continue-on-error leaves outcome at failure while conclusion becomes success, and the step after it reads that outcome and warns. tests/kaggle: 302 passed. * Bound the four checks that were stated rather than measured Evidence collection had no deadline. The job timeout counted 300s per kernel for it, a number no code enforced: one listing walks OUTPUT_PAGE_LIMIT pages at the 120s socket ceiling, 2400s for a single kernel, and each executed notebook is another 300s with no cap on how many Kaggle lists. Measured against the launcher at the previous head, two kernels can spend 14520s there, against 600s budgeted, and the phase runs BEFORE release(), so the runner is killed with billable kernels up. launch.py now starts one EVIDENCE_BUDGET_SEC budget before the first collection, shared by every kernel, clamps each listing page and each download to what is left of it, and marks a collection truncated. The job deadline reads that constant instead of restating a per-kernel term, and deadline=None now means a fresh budget rather than no ceiling. The reference band never checked the card. The reference records gpu_name "Tesla T4" and gpu_capability "sm_75" because a loss trace belongs to its hardware -- no bf16, xformers attention -- and every other reference-defining property already refuses before comparing. Only the GPU COUNT was checked, on the kernel, which cannot see the file. A P100 run against the committed T4 trace returned status "ok" with no failures, so a hardware difference read as a code regression. check_reference now gates on the reference's own environment block, so a recaptured reference moves the gate with it, and "it does not say" stays unchecked rather than a mismatch. adapter_config.json was checked for being JSON. "{}" parses, so a save that wrote no LoRA fields at all scored config_readable and the leg passed on a directory PEFT cannot load. The question is whether PEFT can rebuild the adapter, so it is asked of PEFT on the path a reload takes, PEFT_TYPE_TO_CONFIG_MAPPING[peft_type].from_pretrained; PeftConfig.from_pretrained alone returns peft_type None for "{}" and reports nothing wrong. The rebuilt config is then compared against the arguments the payload handed get_peft_model, so a well-formed config for a different adapter is a difference rather than a field list this file guessed at. The version table omitted the packages the legs move. legs.py records the frontier resolution installing huggingface_hub 1.27.0 and datasets 5.0.1, and the tokenizers and safetensors ceilings that forced deps back on, none of which were in GOAL_PACKAGES, so a failure caused by one produced a comparison table in which nothing differed. They are recorded now. The related hole is that pin_failures answered from a table keyed on that fixed list: a pin outside it had no entry and came back "it is not installed" for a package installed and correct. The probe list is derived from the pin file, and an unprobed pin is its own third outcome rather than an invented failure. pytest tests/kaggle: 315 passed. The workflow's recorded count is updated and its harness step installs peft, without which the two new adapter checks importorskip and cannot fail. * Enforce three bounds that stopped at the edge of the thing they bound Evidence deadline: urlopen's timeout is per blocking socket operation, not a ceiling on the transfer, so an endpoint that keeps returning bytes renews it forever and one resp.read() outlasts the whole 600s budget while every deadline check sits before the call. Measured against a body arriving in 20 chunks a minute apart: 1211s listing, 1260s download. Read in chunks with read1, the absolute deadline re-checked before each and the live socket re-clamped to what is left, streaming downloads to disk instead of into memory. A listing or a notebook abandoned that way now says the evidence is truncated. Reference hardware: check_reference compared gpu_name and gpu_capability only when the run supplied them, and main() records environment = {"error": ...} when the probe raises, while the fingerprint omits every gpu_* key outright when torch.cuda.is_available() is False. Either way the live values were absent, both keys were filed as unchecked, and the control leg reported an ok reference check without establishing the T4 the trace belongs to. What the reference does not say stays a skip; what the run cannot say about a key the reference does name is now hardware_unverified and fatal. Harness step: peft depends on torch, so installing it first satisfied that from the default index and the CPU-index line that followed found the requirement already satisfied and installed nothing, leaving the CUDA set on the runner in the window reserved before the push. CPU torch goes first, as version-compat-ci does, and the result is checked with torch.version.cuda rather than assumed. tests/kaggle: 323 passed. * Report an exhausted weekly Kaggle quota as a failure, not a skip Every gate stand-down was a green skip, exhaustion included, so a week with no accelerator hours left looked exactly like a workflow nobody had wired up: t4-smoke skipped, no annotation, nothing to read. The quota floor now exits nonzero carrying GPU capacity exhausted - please wait until next week - you can ignore this CI failure followed by the numbers the quota API already returned (hours left, the weekly total, and the refresh time), in the log line and in the job summary, so the reader can see when it clears and that there is nothing for them to fix. Only genuine exhaustion. It is decided where it already sat, before the concurrency survey and before any kernel is pushed, so it costs one API call rather than a Kaggle session. An UNREADABLE quota stays a green skip: unknown is not exhausted. So do the sampling draw, the wrong opt-in label, the missing token, a failure to authenticate, a busy account and the launcher-side stand-downs, all with their wording unchanged. --soft-fail becomes a request rather than a default. It defaulted to True, which would have made the new red unreachable, so it is now three states: unset, --soft-fail, --no-soft-fail. An error in the gate itself is still a skip unless --no-soft-fail is passed, exactly as before. The gate job asks for the red and nothing swallows it. The recheck inside t4-smoke passes --soft-fail and stands down green instead: it runs after approval with the account slot in hand, so reaching it means the hours went while the run queued, and red there would be the same message after a runner, the harness suite and the kernel build. Ten tests pin it: the exact sentence, the exit codes on both sides of --soft-fail, that the survey never runs once the quota is short, that an unreadable quota and a busy account stay green, and that nothing downstream turns the gate job's failure back into a pass. 333 pass. * Reserve what a run can bill, and assert two results that were only recorded Three checks that stopped at the edge of the thing they bound. The gate reserved 4 GPU-h against a launcher whose billable window is the whole invocation, not the polling. A kernel bills from the moment Kaggle accepts it until a delete is confirmed, so the push retries (each one discarding the previous attempt's slug), the evidence phase and release() are inside that window as surely as the wait is: launch.py's own constants put one invocation at ~13800s, and Kaggle runs at most --kernels sessions for this account at once, each billing its wall clock once. That is 2 x 13800s = 7.7 GPU-h. The reservation is now 8, and the harness suite recomputes the bound from launch.py rather than checking the number, in the same helper the job deadline is derived from. The old assertion read budget >= kernels x --max-wait, which 4h satisfied while a wedged run could bill 7.7h and take the 20h reserve down with it. The reference identity pinned the configuration and not the rows. canary_dataset.jsonl is inside this workflow's paths filter, so editing it TRIGGERS the run that would then be band-checked against a trace of the old rows: a small edit passes the tolerance and reports green on a comparison that means nothing, a larger one is reported as a code regression. The payload now records a sha256 of the parsed rows in config.dataset_digest and check_reference refuses on it exactly as it refuses on max_steps, the digest of the committed dataset is in the committed reference, and a harness test compares the two on the runner before any session is paid for. Digesting the parsed rows rather than the file's bytes means reformatting does not force a session-costing recapture; changing a row, the order or the count does. The gpt-oss leg recorded placement_after_load on every run and asserted on none. Its documented result is that the 20B checkpoint fits and trains wholly on one T4 with about 1.8GB to spare, and that is the claim that degrades quietly: a loader or memory regression that spills to CPU, disk or meta still logs finite losses, still moves the adapter, still compiles and still generates. failures_for now fails on parameters off the visible CUDA device, on hf_device_map offload, and on a placement record it cannot read, the last on the same three-way rule the bf16 reading already gets. accelerate's offload does not support training at all, so a run that reaches it is not a slower version of this leg. tests/kaggle: 345 passed. * Report a dependency import that exits the process The verify cell's import probe caught Exception, so an import that ends in sys.exit() went straight past it: SystemExit derives from BaseException expressly so that code catching Exception does not swallow it. The cell then aborted before writing the report the probe exists to write, the run cell below it never ran, and a leg that reports nothing is partial or infra at the launcher, both of which exit 0. transformers is a live carrier rather than a hypothetical one: it defines OptionalDependencyNotAvailable as a BaseException subclass and raises it at module scope, and its own lazy loader re-raises only Exception. So catch BaseException here, as the GPU probe a few lines below already does for the same reason, and re-raise KeyboardInterrupt: that is the runner cancelling the job, not a broken dependency, and recording it as one would make a cancelled run read like a regression. The cell is executed in the tests rather than pattern matched, so the verdict is observed the way the launcher will read it. * Resolve the dependencies of the distribution under test pyproject.toml is in this workflow's trigger paths because the payloads install the commit under test as a distribution, and every leg then installed it with --no-deps. pip enforces the requirements of packages IN a resolution and merely warns about the rest, so with the tested distribution outside every resolution the file could change its dependencies and nothing here could tell: a requirement it adds is never installed, one it tightens is never checked against what is already there, and the import probe passes anyway whenever the dependency is reached by a delayed code path. A user meets all three at pip install unsloth. The --no-deps rationale was that the overlay must not walk the set unsloth_zoo had just resolved. What unsloth actually declares is typer, rich, pydantic, pyyaml, nest-asyncio, structlog and click: none of them is resolved by zoo and none is pinned by a leg, so there is nothing to walk. A pyproject that does name one of zoo's packages would move it, which is the regression this exists to show rather than a side effect to suppress. Resolving is only half of it, since pip reports a conflict with something ALREADY installed as a warning and exits 0. So the verify cell also asks pip check, and reads only the lines pip attributes to the distribution under test: the Kaggle image carries pre-existing conflicts of its own, and the frontier leg installs a transformers that zoo's metadata forbids on purpose, so the exit code belongs to the environment while only those lines belong to this leg. An unsatisfied requirement is written out as a verdict, like the two checks above it, because the run cell is the only other thing that reports and is never reached from there. The name the check keys on is read off the requirement the legs install, so the two cannot drift into checking nothing. * Count a kernel Kaggle says is not there as released Cleanup reconciles every slug a push filed, not only the accepted one, and most of those were never accepted: push() files a fresh slug per attempt, and a retry's _discard() deletes the previous one without recording that it worked. So release() asks Kaggle to delete slugs that were never created, and slugs that are already gone. A not-found answer settles the only question cleanup asks, which is whether the slot is still billing. Reading it as a failed cleanup spent DELETE_ATTEMPTS on an absent kernel, ahead of the accepted one that is the only kernel still running, and then named it in a "may still be running, delete them by hand" warning pointing a human at a slug that does not exist. The warning is the one signal this emits, so a false one costs more than the calls. The gate already reads a 404 as a kernel definitively not running, for the same reason and against the same account, so this reads it through the gate's GONE_MARKERS rather than a second list that could drift. The pinned client surfaces one as "404 Client Error: Not Found for url: ..." on stderr with exit 1: kagglesdk calls raise_for_status and cli.py prints the HTTPError. Every other nonzero exit keeps its retries. A 5xx, a reset connection or an argparse refusal says nothing about whether the kernel is up, and trusting the exit code alone would abandon a live session silently. * Refuse to push kernels the runner may not live to delete The job deadline is set to the launcher's worst case plus about 30 minutes for everything that runs before it, and nothing enforces the second term. A checkout, a pip install off a slow index, the harness suite and three network resolutions have no cumulative limit of their own, so the allowance is an assumption about their duration rather than a property of the run. Spend it and the launcher still pushes, and the runner is then killed with kernels up: GitHub sends SIGINT to the step's entry process and kills the process tree about ten seconds later, which is not a window in which release()'s DELETE_ATTEMPTS retries can finish. A kernel nobody deletes bills accelerator quota to its own ceiling with nobody reading the result, which is the exact outcome the deadline exists to prevent. So the launcher is handed the moment the job dies and answers the question itself, before authentication and long before the first push. Less than its own worst case left and it pushes nothing and stands down green, which is what this workflow's failure semantics give every infrastructure outcome: no quota spent, and nothing learned about the code. That worst case now lives in launch.py, computed from the constants that produce it, because three consumers derive from it and all three are wrong by the same amount if a phase is dropped: the job deadline, the quota the gate reserves, and the new guard. The harness suite keeps deriving it independently from the source text and asserts the two agree, so a term lost on either side is a red test rather than a run that pushes with no room to clean up, or one that stands down on every invocation. The start of the job is recorded by the job's FIRST step. Taken any later, the deadline computed from it sits after the real one, and the guard reads optimistic in the one direction that costs quota. The minutes are restated in the launch step's environment because a job cannot read its own timeout-minutes; the suite asserts the two agree. * Ask PEFT what the saved adapter should be called The saved adapter is read back with safetensors and counted, which answers whether the bytes deserialize and not whether PEFT would consume them. PEFT matches its state dict by NAME: set_peft_model_state_dict ends in load_state_dict(..., strict=False) and nothing reads the returned unexpected_keys, so tensors under names it does not recognise are ignored without a word. Measured on peft 0.20.0 against a real saved adapter: strip base_model.model. from every key, or leave the adapter name in (lora_B.default.weight, which is what filtering model.state_dict() by hand produces instead of calling get_peft_model_state_dict), and PeftModel.from_pretrained raises nothing while every lora_B comes back at zero. The file still deserializes, still holds the same nonzero lora_B matrices, still carries a loadable adapter_config.json, so every reading this payload took off the bytes is identical to a healthy save and the leg passes on an adapter that reloads to the base model. The oracle is PEFT's own answer for this run: save_pretrained writes exactly get_peft_model_state_dict(self, ...), so calling it on the model that was just saved reproduces the key set the file is supposed to hold. Nothing is restated here, no prefix and no target-module list, so a legitimate peft renaming moves both sides at once and only a save that disagrees with the running peft is a difference. Three answers, and they are not one failure. A key PEFT names that the file does not carry is a dropped weight. A LoRA tensor under a name PEFT does not use is a weight its loader ignores. A non-LoRA tensor beside the adapter is recorded and not failed, since save_pretrained may legitimately write an embedding or a modules_to_save copy that PEFT never has to match. A run that could not derive the key set at all fails rather than recording an unchecked field, or the strongest thing this asserts would switch itself off silently. * Compare the step count the validator parsed, not the string it was given check_steps.parse_steps accepts "+10", "010" and surrounding whitespace as the ten they are, and the payload's argparse agrees, so those spellings run exactly the reference's run. The build step compared the raw dispatch string against the reference's count instead, read an identical run as a different one, and dropped the committed reference band from it: a green run with the only cross-run check in this workflow never applied, announced in a warning naming a step count that was not different. Both sides of that comparison now come from the step that validated them. check_steps emits the parsed count, and the reference's own count through the payload's reference_step_count rather than a second reading of the same JSON in shell, which also takes the reference filename back to the leg registry. The regression test executes the build step, on the outputs a real check_steps run wrote, with the env resolved from the step's own env block, so a build step that goes back to the raw dispatch string fails it. * Recheck the cleanup window after auth, and keep undecodable payload output from losing the verdict The launcher measured the job's remaining wall clock once, before authenticating, and pushed on that answer. With KAGGLE_API_TOKEN, the only credential the workflow passes, kaggle 2.2.4 authenticates by introspecting the token over HTTP, bounded only by SOCKET_TIMEOUT_SEC, so a window that fitted by less than 120s was already gone at the first push and the runner could be killed during release() with kernels still billing. The check is now a helper asked twice, before and after authentication. The generated run cell decoded the payload's stdout and stderr with subprocess.run(text=True), which is strict: one malformed byte from a native crash handler raised UnicodeDecodeError before the synthetic report below it, papermill aborted the cell, and the launcher called a leg that died partial or infra, both green. Decoding is now errors="replace". --------- Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 04:47:08 -07:00
Daniel Han	33880a192a	CI: add an inert cross-repo capacity sweeper (do not merge yet) (#8071) ... * CI: add an inert cross-repo capacity sweeper Answers 'when llama.cpp or whisper.cpp is cutting a release, can we free capacity by cancelling CI here'. Yes, mechanically. Whether it is worth it is a separate question and this ships switched off so that question can be answered with real numbers rather than in the abstract. Native concurrency: cannot do this. Groups are scoped to one repository, so there is no expression that means 'llama.cpp is releasing'. Cross-repo needs an explicit API call. Shape. The obvious version puts the kill switch in the release repo and needs a credential with Actions write on every target. This is the inverse: each CI repo carries its own sweeper and acts with its own GITHUB_TOKEN, so no cross-repo Actions credential exists anywhere. A release only gets to ask, via repository_dispatch, and asking needs Contents write, not Actions write. The blast radius is .github/ci-preempt.json plus one file. Defaults chosen so the first thing anyone tries is nearly harmless: - Nothing happens at all unless the repository variable CI_PREEMPT_ENABLED is the string 'true'. Unset, which is how this merges, the sweeper still resolves exactly what it would have cancelled and prints it. That dry run is the useful half before anyone commits to enabling it. - pull_request runs are never cancelled, on any path. Taking a contributor's feedback away is visible, is attributed to us, and only they can undo it. - Default events is schedule alone. Nobody is waiting on a nightly and it runs again tomorrow. Nightlies are also where the load is: version-compat 9 legs, local-agent-guides 17, notebooks 11, security-audit 8. - Only workflows named in .github/ci-preempt.json can be touched, and the never list is subtracted after, so an edit to the heavy list cannot reach a release. - /cancel, never /force-cancel. force-cancel is what bypasses always(), which is where people put artifact uploads and cleanup. Un-pausing, which is the part that has to be right. GitHub re-evaluates job if: conditions on cancellation and does not cancel always() jobs, so a release can carry its own resume job and have it survive. What is undocumented is whether such a job starts when it never got a runner, which is the case a paused CI repo creates. So the release-side resume is the fast path and the guard job here is the floor under it: it runs on its own cron, is not gated on the enable flag, depends on nothing, and re-enables anything sitting in disabled_manually longer than MAX_DISABLED_MINUTES. There is no state to lose. The disabled workflow is the state and updated_at is the timestamp, so the worst case is bounded by a number in this file rather than by whether another repository behaved. pause is the escalation, not the default: cancelling loses the race against the next push, but disabling means a push during the window gets no run and no Re-run button, and GitHub does not document whether disabling drains runs already queued. Against a 70-deep backlog that distinction decides whether it buys anything. Not wired to any release yet, and no credential exists for it. If it is ever turned on, mint a GitHub App with Contents write scoped to unsloth and unsloth-zoo rather than a second account-wide PAT. * CI: default the sweeper to linux,windows, not macos Measured the three release pipelines per job rather than assuming, and the macOS-starvation premise does not hold for any of them. llama.cpp run 30854854994 (08-03): the two macOS jobs waited 6 seconds while Linux accumulated 5h47m of queue across 22 jobs and Windows 5h48m across 17. llama.cpp run 31136280015 (08-07) was run-time bound outright, critical path 1h12m37s of build behind 3 seconds of queue. release-desktop attempt-1 runs: macOS queued 1m54s / 0m03s / 19m10s against Windows 2h54m / 45m / 53m. macOS is the fastest leg every time, by one to two orders of magnitude. whisper.cpp v1.9.2-unsloth.3 took 6h39m wall for ~11 minutes of compute, but the run sits wholly inside the 08-06 GitHub Actions incident: the entire org held 0-6 running jobs against a ~1,100 job queue for four hours. A concurrency cap produces running near 60 with a deep queue, not running near zero. So freeing the 5-macOS pool buys these pipelines close to nothing, and the class with real leverage is the one whose runs people are actually waiting on. Default classes moves to linux,windows and the macos list keeps the numbers next to it so nobody re-derives the wrong conclusion. events still defaults to schedule alone. * CI: fix the repository_dispatch class default and use RUNNER_TEMP The dispatch input default moved to linux,windows but the shell fallback that covers a repository_dispatch arriving without a classes payload still said macos, so the automated path would have swept the class the measurements say is not the bottleneck. Scratch files move from /tmp to RUNNER_TEMP, matching the rest of the fleet, and the backlog figure in the pause comment is now the measured peak of 1,175 rather than a spot reading. * CI: scope the capacity sweeper's run query and harden the guard The sweeper read queued runs from the repository-wide /actions/runs with a workflow_id query parameter. That endpoint has no such filter and ignores the parameter rather than rejecting it, so the query returned every run in the repository and the allowlist stopped meaning anything: with the flag on, a default sweep would have cancelled release-desktop runs and this workflow's own guard. Read runs from /actions/workflows/{id}/runs instead, and re-check each returned run's workflow_id before cancelling it. Also in the sweeper: - Paginate the run query. per_page=100 alone stops at the first page, which is short of a nightly's backlog during exactly the conditions this is for. Take the full list before cancelling any of it, because cancelling moves a run out of the status filter that defines the result set. A status search still stops at 1000 results, so a pass is best effort; note that. - Validate hold_minutes before the arithmetic. Bash evaluates the contents of a variable named in an arithmetic expression, so a dispatch payload of BASH_VERSINFO[$(cmd)] ran cmd with this job's actions:write token, exit 0 and no diagnostic. That is reachable today with the feature switched off, and it would have turned the Contents-only dispatch credential into arbitrary code execution. Validate at the point of use, since a newline in a later payload field can append a second hold= line to GITHUB_OUTPUT and the runner keeps the last one. - Handle a failed run listing explicitly. The default shell is bash -e {0} and set -uo pipefail does not clear that -e. And in the guard: - Gate it on CI_PREEMPT_ENABLED like every other destructive path. Without it the file was not inert on merge: twice an hour it would re-enable any allowlisted workflow a human had disabled for more than two hours, and with the feature off nothing here ever disabled anything. Off, it now reports what it would restore and changes nothing. Resume before clearing the variable. - Fail loudly when workflow discovery fails. Reading from a process substitution discards the API call's exit status, and pipefail does not cover it, so a rate-limited listing left restored=0 and a summary that looked like a healthy pass while workflows stayed disabled past the advertised bound. - Subtract the poll interval from the cutoff. Twice-hourly polling against a flat 120m cutoff restored up to a full interval late, ~150m against an advertised 120m. - Subtract never from the guard's allowlist, as the sweep path already does. Correct three claims the code did not support: the worst case is bounded by MAX_DISABLED_MINUTES plus however long the guard waits for a runner from the pool it exists to relieve; updated_at is undocumented and also moves when a workflow is renamed, which restarts a paused workflow's clock; and the guard cannot tell a pause from a hand disable, so an allowlisted workflow that must stay down has to leave ci-preempt.json first. Recording ownership instead would reintroduce the state loss this design removes, since the pause job can die between the disable call and the write. Still inert: no default changed, no gate removed, nothing enabled. * CI: fail the capacity sweeper when a workflow cannot be restored `::error::` writes an annotation and nothing else; a step's status comes from its exit code alone. Both restore paths ended green regardless. The guard emitted an annotation for a failed enable and let the scheduled run pass, so the one workflow left disabled past MAX_DISABLED_MINUTES was invisible to anything watching for a red scheduled run. It now counts the failures, writes the summary, and exits nonzero once every id has been tried. `enable_all` was written as `gh api ... && echo ok || echo error`, whose status is the last echo, so it returned 0 even when every call 429'd and the step went on to write "resume complete" and exit 0. A release reading that conclusion as "CI is back" would have been wrong for up to the guard's window plus its own queue wait. It now returns nonzero after attempting all ids, and the resume branch reports INCOMPLETE and fails. Both are no-ops while CI_PREEMPT_ENABLED is unset: DRY skips the enable call before either counter can move. Verified by extracting both step scripts and running them under `bash -e` against a stub gh: failure gives exit 1 with every id attempted and the summary still written, success and the dry state both give exit 0 with zero write calls. * CI: report the capacity sweeper's dry run as the no-op it is The job summary is the whole product of this file while it ships switched off, and in pause mode it asserted a state change that never happened: "allowlist disabled; the guard re-enables it 120m from now" was printed unconditionally after a dry run had only printed "would disable" and called nothing. Doubly wrong, since the guard is gated on the same variable and has no clock running either. The disable loop now counts, and the line states what happened: the hypothetical count when dry, the real count otherwise, plus a partial-pause line when some ids were lost to a rate limit. Three more things found while tracing that path, all reachable with the feature off: - `classes` of ',' or ' ' killed the step outright. `grep -v '^$'` exits 1 when it filters everything, pipefail promotes it, and the -e in the default `bash -e {0}` takes the step down with no annotation and no summary, which is the exact outcome the hold_minutes note above it says a malformed field must not produce. The `- [""]` in the jq already drops empty entries, so the grep only ever added the failure mode. Empty now lands on the same "no workflows selected" warning an unknown class does. - hold_minutes in a dry run multiplied the headline count. Nothing is cancelled, so nothing leaves status=queued and every pass re-counted the same runs: three queued runs report as 720 at the 240m ceiling, while pinning an ubuntu slot for four hours to reprint one list. Dry runs now sweep once and say so. - `events` did not strip spaces where `classes` does, so `schedule, push` silently swept only schedule. Also records why resume does not take a pause lease, which is a real gap rather than an oversight: overlapping releases lose one pause. Under actions:write plus contents:read the only stores in reach are artifacts and caches, both written after the disable call, which is the same write-after-disable hole the guard note already rejects. A lease that leaks blocks every later resume and holds CI off for the guard's whole window, where resuming early only costs the surviving release some queue time and one more dispatch to repair. * CI: list mixed-platform workflows under every runner class they consume The allowlist filed each workflow under one headline class, but disable and cancel are both whole-workflow: there is no way to reach one leg of a matrix. So a workflow filed under macos alone was invisible to a pause naming linux or windows while its linux and windows jobs kept refilling the pool that pause was paying for. Resolved every `runs-on` in the allowlist, matrix and include rows expanded, against what each file is listed under. Four mismatches, not one: clean-machine-install-ci.yml is 6 linux and 7 windows legs (the wsl job, the windows matrix, and the two windows_container jobs at 1391 and 1469), interrupted-install-ci.yml is 2 and 2, startup-profile-ci.yml is 1 and 1, all three filed under macos only; cross-platform-parity-ci.yml has a linux leg and was filed under windows only. On the default linux,windows pause that left 9 linux and 10 windows legs free against the 46 and 9 it reached. Each is now listed under every class it consumes. The cost is the converse, that a linux pause takes clean-machine's macOS legs down too, and that is the right way round: lost coverage is bounded by the guard and a cancelled run keeps its re-run button, while a pause that does not hold fails at the only thing it is for. The file already made the same trade in the worse direction, since a macos pause disables 9 linux and 10 windows legs. Both jq expressions already carry `unique`, verified: linux,windows resolves 13 raw entries to 9 ids, so nothing is disabled twice. * CI: keep the guard to its own trigger and a pause to its own window Three things, all in the same direction: the workflow should not claim or consume more than it actually holds. The guard had no event condition, so it ran on every repository_dispatch and manual dispatch as well as its cron. That took a second ubuntu-22.04 runner out of the pool the sweep job in the same run was trying to free, and ran an enable loop on ci-pause, the one event asking for the opposite. It now carries the complement of the condition already on sweep. Nothing is lost: the floor under a pause is the cron, and a manual restore is mode=resume, which enables the whole allowlist rather than only what is past the cutoff. hold_minutes accepted 240 for a pause, but the guard restores the allowlist at 90m and the hold loop only repeats the cancel pass, never disables again. Every minute past that was cancel-only work reported as a pause, over a pool refilling through the triggers cancellation deliberately does not touch, pull_request runs above all. Pause holds are clamped to the guard's restore threshold with a warning. Not the other way round: a job re-disabling to see its hold out would be fighting its own safety floor for hours. A long cancel-only hold is still available as mode=sweep, which has no disable to outlive, so its 240m ceiling is unchanged. The resume summary had the same defect the pause summary was fixed for in 4bbbdcfae, and I missed the sibling. With the flag unset enable_all prints "would enable" and returns success, so the dry run recorded "resume complete" over zero API calls. It now reports the count as hypothetical when DRY=1 and states the real count otherwise. Checked against a stub gh: pause clamps at 240 and is untouched at 60, sweep clamps at neither, the dry no-hold rule still applies after the clamp, and the resume summary reads "would have re-enabled 13" dry, "complete: 13 re-enabled" live, "INCOMPLETE" on a failed enable. * Wire the desktop release to the sweeper, with no credential The three release pipelines this file exists to help all queue behind ordinary CI. release-desktop is the one that needs nothing new to ask for capacity: it runs in this repository, so its own GITHUB_TOKEN with actions: write can dispatch the sweeper directly. No secret, no PAT, nothing to rotate or expire. Measured on the three clean attempt-1 runs, this pipeline is Windows bound, not macOS bound: 2h54m of Windows queue against 1m54s for macOS on 30948399091. The default classes stay linux,windows for that reason. The job has no needs: and nothing needs it, so it cannot delay or fail a release, and it is continue-on-error on top of that. release-desktop.yml is already in the never list, so a sweep it asks for cannot cancel it. Still inert. Until CI_PREEMPT_ENABLED is set here the sweeper resolves what it would have cancelled and prints it to the job summary without touching anything, which is what makes this safe to land before deciding whether to enable it: the next desktop release reports how much it would have freed.	2026-08-07 02:08:37 -07:00