mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-19 22:04:03 +00:00
482abfabfa
* Run deterministic notebook smoke tests on real Kaggle T4s
The notebooks are written for a T4: sm_75, fp16 with a gradient scaler,
xformers rather than flash-attention, 16GB. Nothing in CI runs on that
card, so a regression that only appears there reaches users first.
Adds a gated job that launches one Kaggle GPU session and runs the smoke
payload once per T4. The account has a 60h weekly budget shared with
other consumers, so the job is a deliberate minority consumer of it:
paths filters (only 9.9% of a week's 444 main commits touch unsloth/),
a ~10% sampling draw keyed on the run id so a re-run cannot reroll, a
quota floor read live from Kaggle's own API, and a stand-down when the
account already has a kernel in flight. Expected cost is 5-7 GPU-hours
a week.
Only a payload that ran on a T4 and disagreed with its assertions turns
the check red. Throttling, the 2-kernel concurrency cap, a dead session
or a download that would not complete are all warnings, because a free
external service blocking merges would be ignored within a week.
The payload asserts three things, worth different amounts. Two fresh
processes must agree bitwise on per-step loss and grad_norm; that one is
exact, and it is fresh processes because two in-process cycles disagreed
from the first step while two processes agreed on all ten. Greedy
decoding must emit the canary string exactly, which catches trained
weights never reaching the generate call. Metrics must sit inside a band
around a committed reference, which is a tolerance and never an equality,
since fp16 reduction order moves with the driver and the libraries.
Ten steps rather than three: under fp16 the scaler skips the first two
overflowing steps, so a three-step run lands one real update and the
canary never forms. The dataset is supplied as prompt/completion columns
so the loss falls only on the answer.
* Make the payload cells the T4 rejected actually run there
Two launches on real Kaggle T4s died in the scaffolding before a training
step ran, both of them in generated code that nothing had ever parsed.
The dependency probe imported unsloth_zoo before unsloth. Zoo's __init__
ends with a find_spec("unsloth") guard and raised "Please install Unsloth
via pip install unsloth" on a session where unsloth was installed and
imported cleanly one entry later, so the payload aborted over a dependency
that was not missing. Probe unsloth first, and invalidate the import caches
after the pip installs that this interpreter did not see at startup.
The reference argument was generated as a shell fragment and spliced into
the middle of a Python list literal, with doubled braces that left ROOT
unsubstituted. The cell was a SyntaxError, and the workflow passes
--reference on every path, so no payload could ever have reported.
Verified locally by executing the regenerated run cell end to end: two
fresh processes, bitwise identical metrics, canary exact.
* Bound the in-flight kernel survey by session length, not kernel count
The concurrency check status-checked the twelve most recently run kernels,
which is a sample rather than a search: a kernel that started three hours
ago and is still running is missed the moment twelve newer ones have since
run, and the push then fails at the account capacity cap and is reported as
infra.
Kaggle kills a notebook session at twelve hours, and the listing is sorted
by last run time, which for an unfinished kernel is when it started
(measured: a kernel pushed at 10:05:19Z lists as 10:05:19.297). So walking
until the first entry older than that ceiling covers every kernel that can
still be in flight, and stopping there is exhaustive rather than arbitrary.
An hour of slack is added for clock skew.
Two silent ways to read an unanswerable question as a green light are
closed with it. Hitting the page cap now marks the survey incomplete, and
a survey where no status could be read at all is unknown rather than idle.
Both stand down, which is the cheap direction: the next commit draws again
in minutes.
On the live account this now costs three status calls instead of twelve.
* Stop the reference band check from passing NaN unconditionally
The band comparison did its NaN handling by accident, and it got it wrong.
Under fp16 the gradient scaler logs a NaN grad_norm on every step it skips,
so a committed T4 reference genuinely contains NaN. Left to the arithmetic,
abs(x - NaN) is NaN and NaN > tolerance is False, so those steps passed
whatever they held, including the case the check exists for: a step that
used to overflow and no longer does. Compare NaN to NaN as equal and NaN
against a number as a deviation, and treat a field logged on one side only
as a deviation too, since no tolerance covers a change in shape.
The mapping from an out-of-band verdict to the failure list moves into
reference_failures() so the path that turns the job red can be tested
without a GPU.
The references README now says plainly that no reference is committed, that
one must not be borrowed from other hardware, and how to lift one out of a
green run's evidence rather than spending a session on it. It also records
a measurement: the absolute floor never engages on this trajectory. The
smallest value on the ten-step curve is a loss of 0.1428 against a floor of
0.05, so the floor changes no verdict today and is kept for a configuration
that does go lower.
* Name the cause when a kernel finishes and reports nothing
A kernel that ends COMPLETE with no payload report is the hardest outcome
to read: the summary has no metrics to show and the reason sits in an
artifact nobody downloads. It is also the outcome both real T4 failures
took.
Kaggle returns the kernel log as a JSON array of stream records rather than
as text, so the interesting line arrives split across dozens of them and
reading the file directly shows a wall of JSON. Flatten it, filter to the
driver and payload sentinels plus tracebacks, and fold the tail into the
job summary. Checked against the evidence of a real failed run: the
SyntaxError that killed it is now visible without leaving the summary.
* Cover the four T4 CI fixes with CPU-only tests
The generated cells are now compiled on both the with-reference and
without-reference paths, which is what would have caught a SyntaxError that
instead cost a Kaggle session, and the dependency probe's module order is
asserted.
The in-flight survey gets a fake Kaggle client and the case that motivated
it: one kernel still running behind forty newer finished ones, which a
count-bounded scan misses and a time-bounded scan cannot. Also the window
boundary, timezone-aware and missing timestamps, the page cap, and the
difference between some statuses unreadable and all of them.
The band check is now proved to fail rather than assumed to work: a
perturbation half a band-width past the limit, a moved scaler skip pattern
in both directions, a field that stopped being logged, and a length
mismatch, each asserted through to the failure list. Three tests perturb
the committed reference itself and skip, saying so, until one exists.
41 passed, 3 skipped.
* Parse every generated cell on every build path before spending a session
Two of the three Kaggle sessions spent so far died on generated code that
nothing had ever parsed. The compile check now builds all three paths the
generator has, using the workflow's own argument list verbatim, including
the one where a committed reference exists and is carried inline as a
fourth file: that path becomes live the moment a green run supplies the
file, and would otherwise first be exercised on Kaggle.
Parsing alone is not enough. A template hole that substitutes to a bare
identifier parses and then NameErrors, which costs the same session, so
every cell is also checked in execution order for names nothing before it
defines. Both checks were confirmed to go red: the first on the exact
shell-fragment-in-a-list-literal defect that cost session three, the
second on a run cell reading a name no cell binds.
The carried sources are also decoded the way the kernel will decode them
and compared byte for byte with the repo, so the T4 cannot quietly run
something other than what is committed.
* Commit the T4 reference the first green Kaggle run produced
Kernel danielhanchen/unsloth-t4-ci-e3c6661f, terminal state COMPLETE, both
payloads passing on their own Tesla T4 of one session. Two fresh processes
agreed bitwise on all ten steps on both cards, max_abs_diff exactly 0.0 for
loss and for grad_norm, and the two cards independently produced the same
ten values as each other, so four processes agree rather than two. All four
cycles emitted the canary exactly. The scaler skipped steps 1 to 3 every
time, which is the behaviour the ten-step length exists to accommodate.
The file is reports[0] of that run, copied whole by the recipe the README
already documented, and nothing in it is from other hardware.
One assumption the run contradicted, now recorded rather than repeated: the
session had no xformers. unsloth installs --no-deps, unsloth_zoo does not
carry xformers and the Kaggle image does not either, so the banner read
Xformers = None and this trace is the fallback attention path. Adding
xformers to the install would move these numbers and require a recapture.
The floor's smallest observed value moves from 0.1428 to 0.0871, still
above the 0.05 denominator floor, so the floor stays inert and the test
that re-derives that keeps passing.
The three tests that perturb the committed reference now run instead of
skipping. 45 passed, 0 skipped.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Name the in-flight tolerance the Kaggle gate stands down on
The gate declines to launch whenever any kernel of the account is in
flight, which is stricter than Kaggle's 2-kernel cap requires. That is
the intended policy, so it is now a named constant with the tradeoff
written next to it rather than an implicit property of the code, and a
--allow-in-flight argument that cannot be raised to the cap itself.
Default behaviour is unchanged.
Also files the workflow under 'never' in ci-preempt.json: cancelling a
run does not stop the Kaggle kernel it already pushed, and the orphan
then bills quota to its own ceiling with nobody watching.
* Run three steps, and refuse a reference captured at another count
max_steps drops from 10 to 3. Measured consequence: under fp16 the
gradient scaler starts at 65536 and skips every step it overflows on,
and the committed reference shows steps 1, 2 and 3 all skipped, so a
3-step run of the old configuration applied zero optimizer updates.
--init-loss-scale pins the scaler below the overflow point so those
three steps are real updates, and optimisation_failures() fails a run
whose every step was skipped rather than letting it report as training.
The committed reference was captured at 10 steps and no longer
describes the run. check_reference now reads the max_steps the
reference records and refuses to compare across counts, as a failure
with both counts named -- including when the reference does not say
what it was captured at. A quiet pass there would be a green check
comparing a run against a curve it has nothing to do with.
The reference is therefore stale until one recapture run. The workflow
gains skip_reference_band for exactly that dispatch, and the recipe is
in references/README.md.
Also brings the workflow onto the repo's conventions: pinned action
SHAs matching the rest of CI, a workflow-level concurrency group that
deliberately does not cancel in progress, a fork guard on the gate
job, typed dispatch inputs and an artifact name without the run id.
* Keep the run at ten steps, and say why three does not work
The committed reference is the evidence: the fp16 scaler reports grad_norm
NaN at steps 1, 2 and 3 and a finite value from step 4. It starts at 65536,
halves on each overflow, and skips the step it overflowed on, so a 3-step
run applies zero optimizer updates. Loss stays around 10, the canary never
forms, and the band would compare three points of a curve that never moved.
Shortening also saves nothing worth having: a launch costs about 0.08h and
that is almost all pip install, not training.
The step-count guard, the all-steps-skipped guard and --init-loss-scale all
stay, since they are what makes a shorter run possible later and what stops
a vacuous one reporting green. The pin is off by default so the committed
reference keeps applying.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Turn one T4 payload into four, and pair a pinned control with a canary
The workflow ran a single tiny SFT payload on both cards of one Kaggle
session and spent about 1.4 GPU-h a week doing it. The budget is now 40,
and the question asked of it is wider: catch regressions in trl,
transformers, accelerate, peft, bitsandbytes, torch and vllm, and catch
torch.compile falling over on sm_75.
Four legs now, two kernels of two T4s each, described once in
.github/scripts/kaggle_t4_ci/legs.py rather than restated in YAML:
control the existing tiny Qwen2.5-0.5B determinism run, pinned
canary the same run on the newest library set Unsloth's own
constraints allow
gptoss gpt-oss-20b LoRA, for torch.compile and the forced float32 path
grpo Qwen3-4B GRPO through a vLLM engine on the same card
control and canary are one instrument rather than two tests. Same payload,
same seed, same data, same step count, on the two cards of the same
session; the only difference between them is the installed versions. A red
canary beside a green control is a library release, and the job summary
already puts the two version sets side by side so the bisect is a diff
rather than an investigation. The reference band applies to the control
alone: two library sets do not produce one fp16 trajectory, so the canary
asserts what does not depend on versions instead.
The pins are evidence rather than preference. transformers and trl are the
pair the committed reference was captured with; peft, accelerate and
bitsandbytes come from probe kernels 8161ceb9 and 7ab727f1, two independent
T4 sessions that ran the identical base install and agreed. torch stays
unpinned and recorded, because it comes from the Kaggle image and replacing
it is the failure mode the grpo leg spent two probes on.
The gate now separates this workflow's own kernels from everybody else's.
It takes both of Kaggle's concurrency slots, and only when the account is
otherwise idle: one kernel belonging to anyone else still stands it down
entirely, which is the same policy as before stated at the right grain.
* Leave the vLLM leg unwired, and stop trusting Kaggle's own kernel timeout
Three probe sessions tried to run the Qwen3-4B GRPO leg on a T4 and none of
them reached a training step. None of them failed for a reason to do with
sm_75, memory or GRPO either: vllm==0.11.2 pins torch==2.9.0, so the leg has
to replace the Kaggle image's torch, and that is what killed all three.
With the image's site-packages visible, pip treats torch's pinned NVIDIA
runtime packages as already satisfied by the copies belonging to 2.10, and
the result is a torch that installs cleanly and cannot be imported --
libcusparseLt.so.0 first, then ncclCommWindowRegister once that one was
named explicitly, identically on vllm 0.11.2 and 0.15.1. An isolated venv
resolving the whole stack got further and then the session wedged.
So the leg keeps its payload, its assertions and its install, and moves to
legs.UNWIRED with the three kernel slugs and what each one measured. Wiring
it now would make the check permanently red and spend the budget doing it.
The second kernel keeps its spare T4, which costs nothing, and that is where
the leg goes when the install works.
The third probe also cost about an hour of quota teaching us that the
push-time kernel timeout is not a budget control. Pushed with -t 5400, its
own nbconvert crashed at t=406s and the session sat in RUNNING for over two
hours -- past that ceiling and past the launcher's own deadline, because one
status call had blocked with no socket timeout to bound it. Deleting the
kernel by hand took the account's used-hours figure back down, which is the
measurement that settles which control is real. The launcher now sets a
socket timeout and deletes every kernel it pushed on every path out, and
the workflow header says which of the three bounds is load-bearing.
Sampling recomputed on the payload that actually ships: 40%, from measured
per-kernel cost rather than the old estimate, with the note that wiring a
fourth leg means recomputing it rather than editing one line.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Re-solve the GRPO leg's install so it stops replacing the image's torch
Three probe sessions never reached a training step, and all three died the
same way: vllm==0.11.2 pins torch==2.9.0, so the leg had to replace Kaggle's
torch 2.10.0+cu128, and pip treats the image's NVIDIA runtime packages --
which belong to 2.10 -- as satisfying the new torch's pins. That imports as
libcusparseLt.so.0 missing, then as libtorch_cuda.so undefined symbol
ncclCommWindowRegister one package along, then as an isolated venv that spent
an hour of quota resolving a CUDA stack and never produced payload output.
None of that is a question about sm_75, memory or GRPO. It is one line: no
vLLM release in the 0.11-0.16 range pins the torch the image ships. Releases
0.17.0 through 0.19.1 pin torch==2.10.0 exactly, so pin 0.19.1 and the leg
replaces nothing, keeps system_site_packages, and installs like any other.
Drop xformers with it. Its vLLM attention backend was deleted in 0.12.0, so
it would be a package nothing selects. sm_75 has no FlashAttention and no
FlashInfer, and the ladder in vllm/platforms/cuda.py falls through both to
TRITON_ATTN; the leg names that in VLLM_ATTENTION_BACKEND so a release that
reorders it goes red here rather than quietly selecting something else. 7.5
is still in CUDA_SUPPORTED_ARCHS at v0.19.1 and fp16 is a supported dtype
below capability 8.0.
Still unwired. Two things are static evidence rather than a run: whether
0.19.1 starts on sm_75 at all, and whether 8GB of 16-bit weights plus an
engine plus a LoRA trainer fit in 14.56GB. UNWIRED says so in those words,
and a test asserts the note still names the pin and still says what is
unknown, so nobody wires this off prose that outlived its measurement.
Also ports the vacuous-test fix from the stacked Studio branch: pre-commit.ci
reformatting default=10 to default = 10 stopped the payload regex matching,
so the step-count agreement check was comparing nothing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Stop the GRPO leg JIT-compiling flashinfer on a machine that cannot link it
First probe on the re-solved install, kernel unsloth-t4-ci-e2d9ce9b. The
install works: torch stayed at the image's 2.10.0+cu128, vllm 0.19.1
installed and imported, xformers was absent as intended, and the payload ran
on a real Tesla T4 reporting capability 7.5 with TRITON_ATTN accepted. It
reached engine construction, which is further than any earlier probe got by
the whole width of the install.
It died in flashinfer 0.6.6's JIT, and not for an sm_75 reason. All three .cu
files compiled cleanly for -gencode=arch=compute_75,code=sm_75. The LINK
failed:
/usr/bin/ld: cannot find -lcuda
-L/usr/local/cuda/lib64/stubs is already on the command line, so the image
has no driver stub libcuda.so, only the runtime libcuda.so.1. Nothing about
that is fixable from a payload.
Set VLLM_USE_FLASHINFER_SAMPLER=0 so nothing JITs. The sampler has a native
path, and skipping the build also saves a four-file nvcc compile inside a
session billed by wall clock. UNWIRED records what the probe measured and
narrows what is left to one question: whether the engine builds now.
* Make -lcuda resolvable so flashinfer can link what it compiled
Second probe, kernel unsloth-t4-ci-916d5986, failed identically to the first
with VLLM_USE_FLASHINFER_SAMPLER=0 set. That is the informative part: the
JIT is not reached only through the sampler, so switching off one consumer is
whack-a-mole.
The failure was never sm_75. Both sessions compiled all three .cu files
cleanly for -gencode=arch=compute_75,code=sm_75 and died at the link:
/usr/bin/ld: cannot find -lcuda
-L/usr/local/cuda/lib64/stubs is already on the command line; the image ships
no libcuda.so at all, only the versioned libcuda.so.1, and the linker will
not resolve -lcuda against a soname. Normally the toolkit's driver stub fills
that gap; this image has the directory and not the file.
So symlink libcuda.so at the real driver in a scratch dir and put it on
LIBRARY_PATH, which gcc and ld search for -l. No root, nothing written into
/usr/local, and it fixes every flashinfer op rather than one. Linking against
the real driver instead of a stub is correct here: the driver is present,
which is the whole reason a stub would have substituted for it.
Also widen the captured traceback to head AND tail. The last probe's
6000-char tail was entirely ninja's own output, so the Python frames naming
the caller were exactly what got dropped.
Probe 2 also measured the fit question the notebook raised: peak reserved
7.96GB of 14.56 with load_in_4bit false, so 16-bit weights are not what is
short of room here.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Only count a libcuda the linker will actually search for
Kernel unsloth-t4-ci-d0d480b6: the shim reported already_linkable and did
nothing, and the link failed anyway. It had found libcuda.so in
/usr/local/cuda/compat -- which is real, and which nothing passes with -L.
The failing ninja line names /usr/local/cuda/lib64 and
/usr/local/cuda/lib64/stubs, and those are the only two directories that can
answer -lcuda.
So the precondition now checks exactly those two, and compat becomes what it
is actually good for: a symlink TARGET when ldconfig and ctypes both come up
empty. A library the linker will not search for is not a library the linker
can find.
Two tests: one reproduces the compat-only machine and asserts the shim is
built anyway, one pins the searched directory list, since widening it is
precisely how the check went wrong.
* Give the base model a chat template, and record what the link shim did
The link shim works. Kernel unsloth-t4-ci-27b0dc2e reported
libcuda_shim: needed true, applied true,
searched [/usr/local/cuda/lib64, /usr/local/cuda/lib64/stubs],
real /usr/local/cuda/compat/libcuda.so
and flashinfer is gone from the failure entirely. Memory went from 7.96GB to
11.36GB of 14.56, which is the vLLM engine actually loading, and the run
reached trainer.train() -> _run_epoch -> training_step. Both questions
legs.UNWIRED was holding this leg for -- does vLLM start on sm_75, does it
fit in 14.56GB -- are answered by that.
It then failed on the payload's own gap:
ValueError: Cannot use chat template functions because
tokenizer.chat_template is not set
unsloth/Qwen3-4B-Base is a base model and ships none. The notebook this leg
comes from solves it with an SFT priming stage that installs a template
before GRPO starts; this leg has no priming stage, so it sets a minimal
ChatML template directly and records in the report which of the two worlds
the run was in.
The base model stays. GRPO on an instruct model would measure the instruct
tuning as much as the run, and the payload's rewards are format-and-digit
rewards a base model can move inside three steps.
A test renders the template rather than matching it as a string: a template
that does not render would trade a failure at step 1 for a failure at step 1
with a longer traceback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Wire the GRPO leg: it runs on a T4 and the rewards move
Kernel unsloth-t4-ci-53efcc4e passed on a real Tesla T4. Step 2 logged
reward_std 0.707 and grad_norm 0.772, peak 13.60GB of 14.56GB, three steps
in 192s, and fast_generate returned coherent text. reward_std is the
criterion rather than loss, because TRL's GRPO loss is ~0 by construction at
num_iterations=1 and beta=0 and would pass on a run that learned nothing.
The config in args is the one that fit, not the notebook's. Two probes with
seq 2048, 4 generations, rank 32 and utilization 0.9 died in the backward at
unsloth_zoo/gradient_checkpointing.py:1013, peaking at 15.97GB in 16-bit and
19.25GB in 4-bit. 4-bit alone is not the lever: it peaked HIGHER, because
quantizing weights does nothing for activations while utilization 0.9 still
hands vLLM ~13GB up front, and UNSLOTH_VLLM_STANDBY returns the weights
during training but not the KV cache reservation.
An earlier probe reported this as 'CUDA error: an illegal memory access
was encountered' from empty_cache() inside vLLM standby. That was the async
shadow of the same OOM; CUDA_LAUNCH_BLOCKING=1 produced the honest error.
UNWIRED is now empty. The test that read its grpo entry is replaced by the
invariants that outlive any one leg: every leg is wired or explained, nothing
is both, and an unwired note says what is open. Two new tests pin the config
and the vLLM pin, so restoring either to a value that OOMs fails here rather
than on Kaggle.
* Add a CUDA_LAUNCH_BLOCKING switch to the GRPO payload
An illegal memory access is reported at whatever CUDA call synchronises
next, which on the first T4 GRPO run was empty_cache() inside vLLM standby,
nowhere near the kernel that faulted. Re-running with this flag produced the
honest error instead: torch.OutOfMemoryError in the backward. That is what
turned an unexplained CUDA fault into a memory budget, and the budget is now
the leg's config.
The environment variables are set before the first import torch in the
process rather than merely before train(), because they have to precede CUDA
context creation. It serialises every kernel launch, so it stays a switch and
never a default, and it is recorded in the report because it changes what a
timing means.
* Give Studio its first CUDA coverage, on a Kaggle T4
Every Studio workflow in this repo runs on ubuntu-latest, macos-15 or
windows-latest. macOS gives Metal and MLX some hardware; the CUDA path is
exercised by nothing at all, and studio-inference-smoke.yml uses a 270M
GGUF precisely because it has to decode on a CPU.
tests/kaggle/studio_gpu installs Studio the supported way on a Kaggle T4,
starts it headless, and asserts three things a CPU runner cannot: that a
GGUF is actually offloaded to the GPU rather than falling back to CPU and
returning text anyway, that a LoRA training run completes and leaves an
adapter on disk, and that a GGUF export against a CUDA llama.cpp produces a
file that loads. The repo's existing playwright_chat_ui.py then drives the
browser path, last, because it stops the server on its way out.
The gate and the launcher are the notebook leg's, reused unchanged: the
payload prints the same report line, so the transport needed no edit. The
two legs share a concurrency group because Kaggle's 2-kernel cap is per
account and is not workflow-aware.
Sampled at 4 percent of a filter measured at ~760 eligible invocations a
week, which is ~30 launches and ~23 GPU-h against the notebook leg's 15-20
and 1.4. The rate is lower and the spend is fifteen times higher because
the eligible stream is five times larger and a launch costs nine times as
much; the arithmetic is in the workflow header.
* Budget the two Kaggle GPU legs against one account, not two
The Studio leg's budget block was solved against the notebook leg's numbers
from before that leg was re-tuned: 0.08 h per launch, 10 percent, ~1.4
GPU-h/week. The notebook leg now costs 0.25 h at 40 percent for up to 23.1
GPU-h in a busy week, and Studio was sized at ~23 of its own. Two legs at 23
each is 46 of the 50 hours CI is allowed on a 60 h account, with the entire
margin spent on the cheaper one, and the two reserve floors were picked
independently.
Re-solve both against one shared allowance: Studio 35 h, notebooks 15 h.
Studio goes to 5 percent (~38 launches, ~28 GPU-h) and notebooks to 15
percent (13-35 launches, 3.3-8.7 GPU-h). The reserves now encode priority
rather than each leg's own ceiling: notebooks stand down once the account is
35 h spent, Studio runs on to 50.
The percentage is the one axis where Studio reads lower, and it cannot be
made to agree. Studio's eligible stream is ~3.3x larger and each launch
costs 3x more, so a point of rate is worth about ten times as much there;
matching the notebook leg's 15 percent would be ~86 GPU-h a week. Both
budget blocks now say so, and a test re-derives the joint spend from the two
headers instead of trusting either total.
Also fix a test that had gone vacuous: pre-commit.ci reformatting
default=10 to default = 10 stopped the payload regex matching, so the step
count agreement check was comparing nothing. Match count is asserted now.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Write down where the Kaggle sampling percentages come from
The percentages in the two workflows were arithmetic nobody could check. This
records the measurements they came from, taken 2026-08-11 over a 28-day
window: 985 commits to main, 1157 PRs, and 8 of 60 sampled PRs touching the
paths a T4 training run could regress, so a 13% path-filter rate and roughly
75 candidate events per week.
Against 30 GPU-h/week for one account at ~0.75 GPU-h per launch, the ceiling
is about 33 launches per week if everything went to one workflow. The 15/5
split and the 25/10 reserve-hour floors follow from that, and the file says
what would change them: a second account, a wider path filter, or a session
cost above ~1 GPU-h.
* Give the GRPO leg its own kernel: pairing it with gpt-oss fails
Kernel unsloth-t4-ci-70a2f4eb ran the wired pair on the two cards of one
session. gptoss passed: 32 unique graphs, 779 calls captured, 2 graph breaks,
peak 12.78GB. grpo failed with the illegal memory access again, at exactly
the 13.60GB peak at which it PASSED alone on kernel unsloth-t4-ci-53efcc4e,
same config to the flag.
Same peak, different outcome, so the constraint is not GPU memory: the driver
pins one card per payload. gpt-oss offloads to host RAM and vLLM wants host
RAM, and a Kaggle session has one host.
A third kernel costs a queue slot rather than quota, since Kaggle runs two at
a time and bills wall clock per session. That is cheaper than a leg which
fails half the time for a reason the report cannot show.
Two tests pin the rule in both directions: grpo must not share a kernel with
gptoss, and control must still share one with canary, because those two are a
matched pair whose comparison is the whole instrument.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Unwire the GRPO leg: running it alone reproduced the failure I blamed on pairing
I gave grpo a kernel of its own earlier today on the reasoning that sharing a
session with gpt-oss was what broke it. It had failed paired
(unsloth-t4-ci-70a2f4eb) and passed alone (unsloth-t4-ci-53efcc4e), so pairing
looked like the variable, and I wrote two tests pinning the separation.
That reasoning was wrong. Running the leg ALONE again
(unsloth-t4-ci-c98f14be, built from this branch with --legs grpo) reproduced
the paired failure exactly:
unsloth_zoo/vllm_utils.py:601 sleep() -> torch.cuda.empty_cache()
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
engine_built false, peak 13.8GB/13.6GB of 14.56GB. The passing session and both
failing ones agree to the digit on every recorded version -- torch 2.10.0+cu128,
transformers 5.5.0, trl 0.24.0, peft 0.19.1, vllm 0.19.1, unsloth 2026.8.15,
zoo 2026.8.10 -- on the same peak, with UNSLOTH_VLLM_STANDBY=1 throughout. One
contrasting observation was never enough to blame a shared host.
So what three sessions show is an intermittent illegal memory access on Turing,
one pass in three, not a pairing effect. A leg that passes one session in three
cannot tell CI anything: it would go red for a reason no reader could act on.
grpo moves to UNWIRED with the three session ids and the open question written
down, rather than back into a kernel.
The note keeps what the leg already earned -- the vLLM/torch pin, the attention
backend, the libcuda shim, the chat template, the T4-sized config -- so
re-wiring means answering the IMA rather than redoing the install.
test_grpo_does_not_share_a_session_with_gptoss is replaced rather than deleted:
it asserted the opposite thing for the disproven reason, and would now pass
vacuously since grpo is in no kernel at all. The replacement pins that grpo
stays unwired and that the note still names all three sessions, so the evidence
cannot be dropped on the way back in.
145 passed.
* Retire the bootstrap password at login, so the session can reach the routes
First hardware run of this payload, Kaggle kernel unsloth-t4-ci-412345d2 on two
Tesla T4s: preflight, studio_ready, authenticate and the Playwright chat_ui
driver all passed, and the four assertions the payload actually exists for did
not run.
gpu_inference: POST /api/inference/load -> HTTP 403: {"detail": "Password change required"}
lora_training: POST /api/train/start -> HTTP 403: {"detail": "Password change required"}
A bootstrap account is created with must_change_password set, and
get_current_subject turns that into a 403 on every route except the
password-change one. So login yielded a token that authenticated and could do
nothing, and the login step reported success while inference, tool calling,
training and export were all unmeasured behind it.
login now completes the change when Studio's own Token says
must_change_password, and carries the token that /api/auth/change-password
mints. The replacement is secrets.token_urlsafe(24): never whitespace, which
the route rejects, and never equal to the bootstrap value, which it also
rejects. It is not returned or stored -- nothing needs it again.
Four tests, all off-box against a scripted HTTP layer:
the change happens, and the session carries the post-change token
an account already past the gate does NOT get its password rotated
a refused change raises instead of carrying on with a token that cannot act
a change that answers without a token raises
The third is the one that matters: continuing past a failed change is how the
first run produced four unmeasured assertions and no explanation.
Two things this does NOT fix, recorded so the next run is read correctly.
gguf_export reported install_kind=None, i.e. no llama.cpp install under
STUDIO_HOME at all. That may well be downstream of the 403 rather than
independent: POST /api/inference/load is what pulls a llama.cpp for a GGUF
model, and it never ran. The collected kernel.log is 223 lines and holds none
of the install output, so this is not settled either way -- the re-run answers
it.
gpu_inference also failed its own evidence test: text came back, but no probe
could show the GPU was used. The payload calls that a failure rather than a
pass, which is the behaviour I want kept.
89 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Hand the UI driver the password the session actually holds
Second hardware run, kernel unsloth-t4-ci-9ddd8ae4. Retiring the bootstrap
password fixed three assertions and broke a fourth:
gpu_inference FAIL -> PASS
tool_calling FAIL -> PASS
lora_training FAIL -> PASS
chat_ui_driver PASS -> FAIL: "the bootstrap password is gone, so the driver
cannot log in"
My own doing. assert_chat_ui read $STUDIO_HOME/auth/.bootstrap_password off
disk, and authenticate() has to retire that value to get past Studio's forced
password change, so the file is stale by the time the driver runs. The driver's
first phase rotates the password itself and asserts the old one stops working,
so it needs the value the session is CURRENTLY authenticated by -- not the
seeded one.
login now leaves that value on Studio.password and assert_chat_ui passes it,
falling back to the seeded file only when no login has happened. The commit
that introduced the retirement said the replacement was "never returned or
stored: nothing needs it again". That was wrong, and the docstring now says so
and says why.
Studio.password is a credential held for the run, unlike anything else in that
module, so the module docstring records it and assert_chat_ui adds it to the
scrub set before it can reach a log.
test_the_session_remembers_which_password_is_current pins both directions: the
retired session reports the replacement it minted, and a session that never had
to change reports the password it logged in with.
Still failing, and NOT downstream of the 403 after all: gguf_export reports
install_kind=None on a run where inference, tool calling and training all
passed, so no llama.cpp is installed under STUDIO_HOME regardless of the
auth gate. I said last time that this might resolve itself once the 403 was
fixed. It did not. It is a separate gap and it stays open.
90 passed.
* Record what launch blocking did to the GRPO leg: it removed the fault
The UNWIRED note asked for one experiment: run the leg with
--cuda-launch-blocking so the illegal memory access reports at its real call
site instead of surfacing at torch.cuda.empty_cache(). Done, kernel
unsloth-t4-ci-b1f23e34, and the answer is not the one the note expected.
With blocking on there was no illegal memory access at all. engine_built true,
three steps, same 13.8GB peak, same versions to the digit. A fault that
disappears when the launches are serialised is a race, which is what one pass
in four already suggested and which no amount of re-running the leg will pin
down.
The run also exposed a SECOND problem, and it is separate from the first. It
failed on reward_std = [0.0, 0.0, 0.0] with grad_norm 0.0 at every step. The
completions in the report are coherent prose, so this is not the model
collapsing into degenerate output. It is two completions scoring identically.
The leg runs num_generations = 2 because that is what fits on a 14.56GB card,
and at two samples a tie on a coarse reward is ordinary rather than a bug. So
the leg's pass criterion is fragile at the size it has to be to fit, which is
worth knowing before anyone reads a red grpo leg as a regression.
The note now records both, says which question each answered, and states the
two that remain open: where the race is, and what pass criterion is honest at
num_generations = 2. The test gains the fourth session id and a check that the
note no longer reads as though launch blocking were still the next thing to
try -- a stale "try this next" is how an experiment gets run twice.
145 passed.
* Re-seed the account before the UI driver, because the two want opposite states
Third hardware run. The API assertions all pass now:
preflight, studio_ready, authenticate, gpu_inference, tool_calling,
lora_training -- all PASS
and the Playwright driver still fails, for a third and final reason in this
chain:
[ui] STEP change-password through UI (Setup your account)
[ui] change-password form attempt 1 failed: Timeout 60000ms exceeded
waiting for locator("#new-password"); page.url=.../login
The driver's first UI step drives the forced-change form. authenticate() has
to retire that same bootstrap password over the API to reach /api/inference/load
and /api/train/start at all. So by the time the driver runs, Studio shows an
ordinary login and #new-password does not exist.
That is not a value problem, it is a state problem, and the previous commit
only fixed the value. The API path wants the change DONE; the driver wants it
PENDING. One account cannot be in both states, and three runs walked the whole
chain to get here: 412345d2 failed the API assertions on the gate, 9ddd8ae4
fixed those and failed the driver on a stale password, this one fixed the
password and failed the driver on the form being gone.
assert_chat_ui now restarts Studio before handing over. start_server() removes
$STUDIO_HOME/auth, which is what re-seeds the bootstrap password, so the driver
gets exactly the state its first step expects. It is cheap here specifically:
this assertion runs last and stops the server anyway, so nothing after it needs
the API session, and the driver loads its own model through the UI.
test_the_ui_driver_gets_a_freshly_seeded_account pins the restart, that the
password is read AFTER it, and that the retired session's password is not what
gets handed over -- the restart is the whole fix, and a refactor that drops it
puts the payload straight back to a driver that cannot find the form.
Unchanged and still open: gguf_export reports install_kind=None. Three runs
now, one of them with inference, tool calling and training all green, so no
llama.cpp is installed under STUDIO_HOME and this is not downstream of the auth
gate.
91 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Test the newest transformers and trl, not the newest zoo permits
The canary leg is documented as "the newest permitted library set" and that
is exactly what it installs: it resolves WITH unsloth_zoo in the resolution,
so zoo's own metadata is a ceiling. Measured against the two runs that
carried it:
canary transformers 5.5.0 trl 0.24.0 peft 0.20.0 accelerate 1.14.0
control transformers 5.5.0 trl 0.24.0 peft 0.19.1 accelerate 1.13.0
PyPI the same day: transformers 5.15.0, trl 1.9.2, peft 0.20.0, accelerate
1.14.0. The canary moved peft and accelerate to genuine latest and moved
transformers and trl not at all, which is what made it look like it was
working. zoo pins transformers <=5.5.0 and trl <=0.24.0, ten minors and a
whole major behind, so this CI could not detect a transformers 5.6+ or trl
1.x regression: it never installed one.
The new frontier leg upgrades transformers and trl with their dependencies
and without zoo in the resolution, which clears the cap because pip enforces
only the requirements of packages in the resolution. It runs the same SFT
payload as control and canary, so all three are directly comparable.
I expected it red. It is green. On a real T4: transformers 5.15.0, trl 1.9.2,
datasets 5.0.1, ten steps, canary emitted, and two fresh processes agreeing
bitwise. Unsloth trains and generates correctly a whole trl major above what
zoo's metadata permits.
Getting there took two kernels. The first used --no-deps plus a blanket
tokenizers upgrade and died before running anything, because an unbounded
upgrade overshoots the ceiling transformers declares:
tokenizers<=0.23.0,>=0.22.0 is required, but found tokenizers==0.23.1
safetensors>=0.8.0 is required, but found safetensors==0.7.0
A test pins that shape out, along with the zoo-in-the-resolution mistake that
would silently turn this into a second canary.
It goes on the second kernel's idle T4, so it costs no quota: a Kaggle
session bills wall clock once, not per card.
What it does NOT catch is recorded in the leg: the loss trajectory is not the
control's, and step 1 is computed before any update on identical weights,
data and seed, so 10.3222 against 6.4367 is a different loss function rather
than drift. Both converge. Which objective is intended is separate work.
* Install a CUDA llama.cpp before asserting the export used one
Four hardware runs reported llama_cpp_install_kind=None and failed the GGUF
export assertion for it. The reason was never subtle: nothing had ever
installed a llama.cpp under STUDIO_HOME, so the assertion was measuring an
absence rather than a selection. install_llama_prebuilt.py resolves a real
"linux-cuda" kind on an x64 CUDA host, so the bundle was available the whole
time and simply never fetched.
The install runs before the server, so the export route sees a llama.cpp that
was already there rather than one appearing underneath it, and it is its own
assertion because "the CUDA bundle would not install on this box" and "it
installed and the export against it failed" are different findings.
Its result is deliberately not checked by execute(). A box where the bundle
cannot be installed still runs inference, tool calling, training and the UI,
and still reports the export failing for the reason it actually failed.
Three tests. One of them CALLS install_llama_cpp rather than reading its
source, because run() in this payload already applies capture_output and text
and passing either again is a TypeError that no source-reading test would
catch -- which is how the first draft of this was written wrong.
* Record what install.sh left behind before replacing it
build_kernel.py runs install.sh --local, whose own comment says it puts a
llama.cpp on disk. Four runs still reported install_kind=None, so either it
never installed one or it installed something else. Fetching a CUDA bundle
fixes both cases and would hide which one it was, so the prior state is
recorded first and the original question stays answerable.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Look for llama.cpp where it is, not where the payload assumed
Five hardware runs reported llama_cpp_install_kind=None and failed the GGUF
export for it. There was no missing llama.cpp. install.sh --local installs one
into ~/.unsloth/llama.cpp, which is what install_llama_prebuilt.py's own
default resolves to, and this payload read STUDIO_HOME/llama.cpp, a path
nothing ever wrote to.
Adding an explicit install last commit is what proved it, because it recorded
what was there first. The installer selected the right bundle for a T4
accept ...linux-x64-cuda13-older.tar.gz coverage=75-89 supported=75,80,86,89
reject ...linux-x64-cuda13-newer.tar.gz reasons=missing_sms=75 out_of_range_sms=75
and then said
existing llama.cpp install already matches selected release
b10360-mix-87da1a2; skipping download and install
while install_kind_before and install_kind_after at the requested directory
were both None and the return code was 0. So a CUDA llama.cpp was present the
whole time, the sm_75 selection works, and only the assertion was wrong.
llama_cpp_marker checks STUDIO_HOME first, then the canonical location, and
returns None when neither has a marker so a real absence stays a real absence.
Against the exact situation the last run recorded, the old read answers None
and the new one answers linux-cuda.
Separately, and not fixed here: install_llama_prebuilt.py --install-dir X
exits 0 having written nothing to X whenever the canonical directory already
holds a matching release. A caller who names a directory and gets a success
with an empty directory has been told something untrue.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Read the field the marker has, not the one it never had
Six hardware runs reported llama_cpp_install_kind=None and failed the GGUF
export assertion for it. There was no missing llama.cpp and no wrong
directory. install_llama_prebuilt.py writes install_kind ONLY into the JSON
its resolver prints to stdout. The marker it writes to disk records
requested_tag tag release_tag published_repo asset force_cpu llama_backend
asset_sha256 source_* ggml_tree bundle_profile runtime_line coverage_class
install_fingerprint prebuilt_fallback_used installed_at_utc
and no install_kind at any point. So reading install_kind from the marker
answered None for every bundle on every box, including a working CUDA one,
and the assertion turned that into a failure. The last run had the installer
saying "existing llama.cpp install already matches selected release
b10360-mix-87da1a2; skipping download and install" while the payload reported
nothing installed.
runtime_line is the field that answers the question: cuda12 / cuda13 for the
CUDA bundles, and the backend name otherwise. It is written into the same
dict as the asset, so the two cannot disagree, and asset is kept as a fallback
for a marker too old to carry it.
is_cuda_install now matches the runtime line's shape rather than a fixed set
of names. A fixed set needs a new entry per CUDA major and fails CLOSED when
it lacks one, reporting a working cuda14 install as not-CUDA, which is the
same failure this commit is undoing.
The existing test wrote {"install_kind": "linux-cuda"} into a marker and
asserted it came back. That is how this survived: the test agreed with the
payload about a field neither read from reality, and stayed green through all
six runs. It is rewritten against a marker the installer actually produces.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Measure the load, not the unload it follows
The GGUF export probe reported device_vram_delta_mib: -1866.0 and was scored
as a load that never reached the GPU. It had not: a 3004 MiB chat model was
evicted inside the same window the 531 MB GGUF loaded in, because POST /load
with force unloads first and the baseline was sampled before the request.
Unload explicitly, wait for the reading to stop falling, then take the
baseline. Best-effort throughout, since the delta is evidence and not the
assertion: a refused unload or a missing nvidia-smi leaves the probe running.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Read the status fields the response actually declares
Run 8 shipped the unload-before-baseline fix and came back with
device_vram_delta_mib: -1866.0, byte-identical to run 7. The fix ran and did
nothing: it looked for model_path, model and active_model_name, and
InferenceStatusResponse carries none of those, so no model was ever
identified and no unload was ever sent.
Read model_identifier (documented as the loadable identifier, which is what
/unload's model_path wants), falling back to active_model and then to
loaded[0]. A new test parses the field names out of InferenceStatusResponse
and asserts the payload only reads names it declares, so the next guess fails
in the suite rather than forty minutes into a kernel.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Delete the Kaggle kernel when the launcher is killed, not only when it returns
release() is the budget control: a kernel left behind bills to its own
ceiling with nobody reading the result, and that ceiling has been observed
not to stop a wedged one. But it was reachable only from finish(), which runs
only on paths that RETURN. KeyboardInterrupt was explicitly re-raised past
it, and the default SIGTERM disposition exits without running atexit or
finally. So Ctrl-C, `kill`, and a cancelled GitHub Actions workflow all left
the kernel running. The file's own comment claimed the kernels were released
"on this path and on every other", which was true for returns and false for
signals.
Four paths, four covers:
normal return / handled error finish()
unhandled exception atexit
Ctrl-C, kill, Actions cancel SIGINT/SIGTERM/SIGHUP handlers
kill -9 the orphan sweep at the next launch
The signal handler re-raises through SIG_DFL so the exit status still reads
"killed by signal N"; exiting 0 would make a cancelled job look completed.
kill -9 cannot be caught, so a pushed slug is now recorded to
logs/kaggle_inflight.json the instant it exists, and every launch sweeps
entries whose owning pid is gone. Keyed on pid deliberately: an entry whose
owner is still alive belongs to a run in progress, and deleting it would
destroy a concurrent run and report the absence as a failure of the code
under test. A delete that fails keeps its entry rather than forgetting it.
The sweep runs before pushing, so reclaimed session slots are available to
the run doing the reclaiming.
Tests drive real subprocesses and real signals, since the property is about
process death. Verified failing against the previous behaviour: SIGINT,
SIGTERM and the unhandled-exception case all leave the kernel behind without
these handlers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Only forget a Kaggle kernel once it is actually gone
kaggle kernels delete answers a transient rejection with a nonzero exit and
subprocess.run does not raise on that, so both cleanup paths recorded a
refused delete as done: sweep_orphans dropped the registry entry and
release_kernels set released=True, leaving a running kernel billing with
nothing left to reclaim it. Check the return code in both.
Two more holes in the same registry. result["kernels"] was published after
the whole push loop, so a SIGTERM once the first kernel was up found no list
and deleted nothing; it is now the same list object, assigned before the
loop. And --keep-kernel left an entry naming a pid that dies with the
launcher, so the next sweep called the retained kernel an orphan and deleted
exactly what the flag asked to keep.
* Stop the Studio leg passing on evidence it never had
Six ways this job could go green without measuring anything.
The evidence bundle was never unpacked: the launcher writes each kernel into
kaggle_evidence/<slug>/ and the workflow hands collect_evidence.py the
parent, so a top-level glob found nothing and every run reported that the
payload emitted no bundle. Recurse, the way extract_reports already does.
An install.sh regression read as infrastructure. sh() raised SystemExit, so
papermill stopped before any T4_SMOKE_REPORT existed, the launcher filed the
run as infra and the reporter exited 0 -- silently passing the installer
failures this workflow's path filter selects for. Failures of the checkout
under test now emit a failing report first; a missing GPU or a clone that
would not download keep the no-report infra path. A payload killed by the
driver deadline does the same, but only once the payload itself had started.
The process-level VRAM probe could never fire: InferenceStatusResponse
declares neither llama_server_pid nor pid, and FastAPI drops what the
response model does not declare, so the lookup was always None. Find the
llama-server processes in /proc instead.
Offload evidence outlived its load. offloaded_layers() takes the last match
in the whole log, so a reload that logged nothing inherited the chat model's
line and passed on evidence from a different model. Read each load forward
from a mark taken just before it.
A diverged run counted as a trained one. A T4 has no bf16, and an fp16 run
that goes NaN still reaches completed and still saves an adapter, while
trained_steps counted anything that was not None. Only finite losses count,
and non-finite ones are named as a failure.
An export slower than its HTTP timeout crashed the payload. The route blocks
for the whole export and the transport timeout is 900s against a 1200s
deadline, so the TimeoutError escaped the StudioError handler; it is now
treated as in-flight and the status poll decides.
Alongside: the restart that re-seeds the account no longer truncates the
backend log of every earlier assertion, the over-cap evidence fallback ships
the logs it says it ships rather than the report alone, the result line is
printed after the evidence is packaged so a packaging crash cannot publish a
pass, the training and export polls stop when Studio dies, Studio is launched
from the interpreter running the payload rather than whatever is on PATH,
the gate is told this leg pushes one kernel rather than the default two, and
the advertised opt-in label can start the workflow.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Name the encoding on every checked-in read, and survive a box with no nvidia-smi
tests/test_source_read_encoding.py fails on 33 read_text() calls in the two
Kaggle harnesses: unpinned text I/O on a repo file is cp1252 on a stock
Windows install and dies the moment that file gains a non-ASCII byte.
The second one is the payload's own. environment() runs inside finish(), on
every path out of the run, so a session Kaggle handed no driver raised
FileNotFoundError from gpu_inventory() and ended with a traceback and no
report at all. run() now answers a missing binary the way a failing one does,
and the preflight says there is no GPU.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Spend a Kaggle session on the opt-in label, not on every label after it
Two ways this leg spends or loses a GPU session it should not.
GitHub has no per-label filter on the pull_request event, so adding
`labeled` to the activity types starts the workflow for EVERY label added
to an affected pull request. The gate is handed the whole label list, so
once kaggle-studio-gpu-ci sits on the pull request it reads every one of
those events as forced, and a routine label change launches another
70-minute Kaggle run. Guard the gate job on the label that fired the
event; push and the other pull_request activities carry no
`github.event.label` and fall through unchanged.
The other way is the reverse: a run that should be red exits 0. The
dependency probe raises out of the cell when the Studio venv reports no
CUDA, and with no T4_SMOKE_REPORT the shared launcher files the run as
`infra`. install.sh --local resolving a CPU-only torch is exactly the
CUDA install regression this workflow's path filter selects for, and it
took that path. Ask nvidia-smi which case it is: a GPU the host can see
and the venv cannot use is a payload failure, and no GPU at all stays
infra.
* Treat a stalled kaggle push as transport, not as a red verdict
Every other Kaggle transport failure -- a 503, a refused push, a capacity
rejection -- returns a reason from push() and exits 0 as infra. A push that
stalls past the 600s ceiling did not: TimeoutExpired escaped launch.py, so the
process ended before finish() could write launch_result.json, the launch step
exited nonzero and GitHub marked the job red. The reporter that reads that file
and calls the run NOT RUN never got a say, and a red there says the code under
test failed when nothing ever reached a T4.
Catch it in push() and report push_timeout.
* Do not let a truncated copy of a chunk overwrite the complete one
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Name the encoding on every checked-in file the T4 harness reads
Repo tests (CPU) is red on this branch:
test_source_read_encoding.py::test_checked_in_file_reads_name_an_encoding
AssertionError: 17 file reads in the test trees touch a checked-in file
with the platform default encoding, so they break on Windows as soon as
that file gains a non-ASCII byte.
All 17 are in tests/kaggle/test_t4_smoke_harness.py, which this branch adds,
so this is ours and not inherited. The reads pull in the workflow YAML, the
smoke scripts, the committed reference JSON and ci-preempt.json, and none of
them said what encoding to read them as. On Linux that resolves to UTF-8 and
nobody notices; on a Windows runner it resolves to the ANSI code page and the
first non-ASCII byte in any of those files takes the test out with a
UnicodeDecodeError that has nothing to do with what the test is checking.
Every one now passes encoding = "utf-8", matching how the rest of the suite
spells it. Reads of files the test itself just wrote under tmp_path are left
alone: the gate only covers checked-in files, and those are ours end to end.
test_source_read_encoding.py 1 passed
tests/kaggle/test_t4_smoke_harness.py 148 passed
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Flatten Kaggle's kernel log before extracting the reports from it
Kaggle's kernels/output returns the log as a JSON array of {stream_name,
time, data} records, one record per line, so no line in the file starts
with T4_SMOKE_REPORT. extract_reports read it verbatim, so the log
fallback -- the one that covers a run whose executed notebook never came
back -- recovered nothing and a failed assertion was scored as infra with
the workflow exiting 0.
report.py::kernel_log_text and collect_evidence.py::iter_text already
flatten the records; extract_reports was the reader that did not. The
records are also cut by write rather than by line, so the join has to
happen before any line splitting.
* Keep the slug a timed out kaggle push may have created
A push whose CLI stalls past the 600s ceiling says nothing about whether
Kaggle took it. Kaggle can accept the push and start the kernel billing
before the response is lost, and the branch returned infra while keeping
neither the slug nor an inflight entry, so finish() had nothing to delete
and the next launcher's orphan sweep had nothing to see. The kernel ran
to its ceiling with nobody reading the result.
The slug is ours and already decided at that point, so record it in the
inflight registry and hand it back as orphan_slug: release_kernels
deletes it on the way out, and a launcher killed first leaves the entry
for the next sweep. It is deliberately not returned as slug, because the
caller must not wait on a kernel that may never have been created.
* Align the Kaggle budget guide with the workflow it documents
The guide named --percent 15 and --reserve-hours 25 against a 30h account;
the workflow runs 40 and 20 against the 60h figure measured on 2026-08-11,
and it is the only consumer of that quota today. Following the guide would
have tuned the reserve from a baseline that no longer exists.
Re-derived from the same live numbers as the workflow header, and says which
of the two is the source of truth when they disagree.
* Stand the gate down on any unreadable kernel status
The concurrency verdict refused only when EVERY in-window status came back
an error. One unreadable status among readable ones was treated as an idle
account, and driving concurrency_verdict with
{busy: [], own: [], foreign: [], surveyed: 2, unreadable: 1, complete: True}
returned (True, ""). That kernel can be the human session the zero-foreign
policy exists to yield to, and proceeding takes the account's last slot: the
first push wins it, the second comes back at capacity, and the run reports a
subset of its legs.
A 404 is kept separate and stays benign. A deleted kernel is not an unknown
state and the launcher deletes every kernel it pushes, so blocking on that
would wedge the gate shut rather than make it careful.
* Close six gaps in the Kaggle T4 notebook CI workflow
- The pull_request trigger used GitHub's default activity types, which are
opened, synchronize and reopened only. The kaggle-t4-ci label the gate
advertises as a human override could therefore never start a run on its
own. Subscribe to labeled as well.
- pyproject.toml was in neither paths filter, though every payload installs
the commit under test as a distribution built from it. BUDGET.md counted
it as a watched path already.
- The job deadline of 120 minutes was shorter than the launcher's worst
case. Two sequential pushes can take 2 x (4 x 600s + 315s of backoff) and
the 5400s polling window only starts after them, so about 187 minutes with
the deletions; a runner killed at 120 takes finish() -> release() with it
and leaves pushed kernels billing quota. 200, with the arithmetic written
down beside it.
- The gate surveys quota and in-flight kernels before t4-smoke queues on the
account-wide concurrency group, so a second sampled run could launch on an
approval a full run old. Re-ask with the slot in hand, immediately before
the push; --force true skips the sampling draw and nothing else.
- --zoo-ref main let each payload's own pip resolve a different unsloth-zoo
commit, including between the control and the canary within one session,
and zoo is not in pins/control.txt either. Resolve it once with git
ls-remote and pass that SHA to every leg.
- A dispatch with max_steps != 10 still shipped the 10-step reference, and
check_reference reports step_count_mismatch as a failure, so the run could
only ever go red on arithmetic. Drop the band automatically when the counts
differ, and say so.
Also: check out the same head SHA the launcher installs, so the harness and
the package under test are one snapshot rather than the merge tree and the
head; and run tests/kaggle/test_t4_smoke_harness.py on the runner before
anything is built or pushed, since testpaths limits pytest to tests/security
and nothing else collected it.
* Read back what the Kaggle kernel actually reported
Two ways a real payload verdict was being lost in transport, both ending in
verdict=infra and a green workflow.
The kernel log fallback could never work. Kaggle's kernels/output hands the
log over as a JSON array of {stream_name, time, data} records, which is why
report.kernel_log_text flattens it before reading; extract_reports scanned
the file as text, so no line began with the report prefix and the fallback
that exists for the run whose notebook never came back found nothing. Record
boundaries are not line boundaries either, so the records are joined before
the scan rather than read one at a time.
push() documented a fresh slug per attempt and used one for all four. Pushing
to an id that already exists does not replace it: Kaggle files a new version
and starts a second batch session, and kernels status and kernels/output send
no version label, so they answer for the newest session only. A retry after a
lost response therefore collected the wrong execution's evidence while the
first kept a session slot and billed unseen. Each attempt now takes its own
slug and discards the previous one first, and every slug filed is returned
and recorded, since a push that reported an error may still have landed.
* Give each payload its own everything, and report a probe failure
Four ways the two payloads of one kernel were not actually isolated, plus
the import probe exiting without a verdict.
The probe was the P1. When the commit under test breaks import unsloth, the
verify cell raises before the run cell can write a report, the launcher
extracts none, and no reports is classified as infra -- so a deterministic
import regression exits green. A probe failure is a verdict, not missing
evidence, so it now emits its own failing report.
nvidia-smi failing or a one-GPU allocation fell back to N_GPU = 1 and pinned
both payloads to device 0. Each child still saw exactly one card and passed
its own visibility assertion, so the contention came back looking like a code
failure. A shortfall is infrastructure and stands the kernel down before a
thread starts, which is what the comment above it already claimed.
A payload whose virtualenv could not be built kept the base python3 kernel
and installed into the shared system site-packages. That is the one thing the
per-child venv exists to prevent: the legs install deliberately different
library sets, so the last writer wins and the resulting import error reads as
a regression. The payload is skipped instead.
Both payloads materialised into one source directory and compiled into one
unsloth_compiled_cache. The sources are byte-identical copies but write_bytes
truncates first, and the compile cache is a relative path resolved against a
working directory both papermill children inherit while compiling the same
modules against different transformers versions. Each payload now owns both,
and the prune globs still reach them.
Finally, --smoke-args is appended after every leg's own arguments, so the
workflow's --max-steps 10 landed after the gpt-oss leg's --max-steps 3 and
argparse took the last one. The 20B leg was training for ten steps rather
than the three that were measured to fit. A shared argument no longer reaches
a leg that sets that option itself.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <unslothshared@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
|
||
|---|---|---|
| .. | ||
| gpu_assert.py | ||
| run_studio_gpu.py | ||
| studio_client.py | ||
| train_canary.jsonl | ||