7340 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Vineeth Sai Varikuntla	203007d190	Give the llama-flag fixtures the parallelSlotsClamped they now require (#8837)	2026-08-14 16:07:38 -03:00
Daniel Han	ba466ca095	Update README.md	2026-08-14 08:39:36 -07:00
Daniel Han	8059be6bca	Bump install.sh / install.ps1 pin to unsloth>=2026.8.18 (#8836) ... * Bump install.sh / install.ps1 pin to unsloth>=2026.8.18 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 08:06:18 -07:00
Daniel Han	456cbef667	Update _version.py	2026-08-14 07:55:01 -07:00
Daniel Han	a8be2a87b4	Bump install.sh / install.ps1 pins to unsloth>=2026.8.17, unsloth-zoo>=2026.8.12 (#8831)	2026-08-14 07:38:08 -07:00
Daniel Han	afa9d8109d	Update _version.py	2026-08-14 07:21:10 -07:00
Daniel Han	cb7c71c26e	Update pyproject.toml	2026-08-14 07:19:57 -07:00
Daniel Han	23d670aa76	Offer the media pickers only what the host can run, and name the H3 speed gap (#8822) ... * Offer the media pickers only what the host can run, and name the H3 speed gap The Video picker showed MiniMax H3's bf16 pipeline row on Apple Silicon, where loading it always fails: video.py refuses every modular Diffusers workflow on Metal because the auto CPU offload needs mem_get_info, which torch.mps does not expose. The row was only ever deprioritised by the size filter, so on any sort other than Recommended it stayed clickable and always 400d. Classify the host as accelerated, gguf-only or unknown and offer curated artifacts accordingly, so a host that can only run the native engine sees only the GGUF rows. Capability is kept apart from size: curatedArtifactFitsDevice still answers whether there is room, and a 128 GB Mac has room for a model it cannot place. The unknown state matters because the GPU hook opens at budgetKnown false, and reading that as CPU-only would blink every non-GGUF row out and back on a real GPU host. On an accelerated host the two H3 rows now read Fast FP8 and Slow. The pair differ by roughly 10x and the old names said nothing about it. Keyed on the two H3 ids rather than on format, since most non-GGUF rows are plain bf16 and Auto precision resolves to int8 or bf16 on cards without fp8. Downloaded weights keep their row either way. * Hide only the artifact the backend refuses, not every non-GGUF row The filter keyed on format, so a gguf-only host lost every non-GGUF artifact. The backend refuses far less than that: modular_workflow is set on one family, MiniMax H3, and video.py raises for it on MPS alone. Everything else places fine there. video_capability() certifies Apple Silicon, the diffusion target has a first-class MPS bfloat16 path, and the STT rows run through the whisper.cpp sidecar whatever format the catalog labels them. Measured, the format rule dropped 20 of 32 image rows and 7 of 9 video rows, taking six image families and four video families out of the picker entirely. It also reached the Audio page, which passes AUDIO_CATALOG into the same picker: all five Whisper rows went, on the two host classes where the sidecar makes them the primary path. pickers.tsx already carries a comment about this exact trap, from the last time a GGUF-only rule hid every non-GGUF model. Key it on the repo id instead, the way the speed suffix already keys its own H3 case. Video now drops one row on a gguf-only host and image and audio drop none. * Filter curated search results by host too Search reaches the live Hub, so a query re-landed the row the seed and Recommended filters had just dropped. The Mac format check in that chain admits safetensors, so MiniMaxAI/MiniMax-H3 came back clickable and was still refused at load. Same predicate as the other two lists, downloaded exception included.	2026-08-14 07:16:42 -07:00
Daniel Han	0fb21e8a89	Revert "Studio: verify the installed version after the update pass (#8505)" (#8824) ... This reverts commit fe79a5bbc2.	2026-08-14 07:01:19 -07:00
Nilay	fe79a5bbc2	Studio: verify the installed version after the update pass (#8505) ... * Studio: verify installed version after the update pass * Studio update verify: PEP 440 compare, Requires-Python escape, LATEST_VER init * Studio update verify: accept only the newest interpreter-compatible release * Studio update verify: reuse the fetched PyPI response in the ps1 probe * Studio update verify: base64 the probe temp path (apostrophe-safe) * Studio update verify: filter compatible releases by wheel tags / sdist * Studio update verify: LiteralPath for the temp release table * Studio update verify: skip the strict check when a custom package index is active * Studio update verify: warn on older-but-successful, fail only when missing * Studio update verify: keep missing check on custom indexes, PEP 503 probe names * Studio update verify: run the missing-package probe even when PyPI is unreachable * Studio update verify: isolate probes from PYTHONPATH, generic kept message * Studio update verify: drop cwd from ps1 probe paths, non-fatal temp-table failures * Studio update verify: require loadable payload, keep venv site-packages in scrub * Studio update verify: drop the success manifest when the package is missing * Studio update verify: retry, invalidate, and surface a stuck success manifest * Studio update verify: keep the payload probe clear of the installer-helper guard The post-update probe matched a package initializer by its joined filename. The installer-helper guard in tests/test_installer_interactive_prompts.py scans every installer for filename-shaped tokens and resolves each one against the script's own directory, so that literal resolved to studio/__init__.py and the guard reported it as a helper the installers invoke but nobody scans. Parity and the CPU repo tests have been red on that since the probe landed. Match on stem and suffix instead. Same predicate, no filename token to resolve, and studio/__init__.py stays out of SCANNED_SCRIPTS where it does not belong. Verified on 3.10, 3.11 and 3.13 against fabricated venv trees: a dist-info whose payload was deleted still reports __MISSING__ through this branch on 3.10 and 3.11, and a RECORD-only install whose payload is intact still confirms. From 3.12 on, Distribution.files applies skip_missing_files, so this branch never sees a deleted path there and the top_level.txt branch above carries the check. --------- Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-08-14 07:00:53 -07:00
Wasim Yousef Said	83e163f13c	Revert "Desktop: ship a complete Linux AppImage (#8695)" (#8823) ... This reverts commit 93a77d4104.	2026-08-14 06:55:54 -07:00
Daniel Han	0282789cf1	Studio: apply the provider metadata block to DNS aliases as well (#8782) ... * Studio: apply the provider metadata block to DNS aliases as well The always-on cloud metadata refusal only read the hostname text, so a caller-supplied name resolving to 169.254.169.254 was accepted by default and became the outbound target with the provider API key attached. DNS resolution only happened behind UNSLOTH_STUDIO_BLOCK_PRIVATE_PROVIDER_URLS. Names are now resolved far enough to apply the metadata block to them. Private-address rejection stays opt-in. Shipped registry hosts and IP literals skip the lookup, an unresolvable name stays allowed, and the lookup is bounded at 2s on a daemon thread with its verdict cached for 300s per host. * Studio tests: assert the resolver-free path by recording, not raising An AssertionError raised inside the lookup's worker thread never reaches the test, so the shipped-provider and IP-literal case passed whether or not it resolved. Record the calls and assert the list is empty. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: share one bounded, cached lookup between both provider URL checks _reject_non_public did its own unbounded, uncached getaddrinfo, so with UNSLOTH_STUDIO_BLOCK_PRIVATE_PROVIDER_URLS=1 a validation resolved the same host twice and the second lookup could stall the event loop with no bound at all. Both checks now go through _resolve_host, which returns None when the resolver did not answer. The two callers read that in opposite directions: the metadata check has nothing to refuse, the opt-in check refuses, which is the behaviour each had before. * Studio: tighten the provider URL lookup comments * Studio: resolve the host httpx will dial, and stop caching a DNS timeout getaddrinfo encodes a Unicode host through the stdlib idna codec (IDNA 2003) and httpx through the idna package (IDNA 2008). They disagree on the deviation characters, so straße.de was checked as strasse.de and dialled as xn--strae-oqa.de, two hosts with different owners. The lookup now uses the same ASCII form httpx does. A timeout is also no longer cached as an answer. The transport waits longer than this validator does, so an authoritative server that replies just after the deadline would otherwise get its address treated as unseen, and remembered that way for 300s. * Studio: match resolved addresses against the metadata services, not 169.254/16 The link-local network clause exists so that typing a link-local address at the metadata service is refused, and for a literal it stays. Applied to a RESOLVED address it means something else entirely: 169.254/16 is the general IPv4 link-local range, so a self-assigned host, an mDNS .local name on a network without DHCP, or a captive portal answering every query would each have been read as the metadata service and refused. Nothing refuses those today and this change should not be what starts. * Studio: cap how many provider DNS lookups can be in flight join() bounds how long a lookup blocks the caller, not the lookup: an abandoned resolver thread lives until the platform gives up, and rotating hostnames defeat the cache, so one thread per request could pile up. Past 8 in flight the check now reports no answer rather than starting another, which is what this file did before it resolved anything at all, and the slot frees when the resolver actually returns. * Studio: keep the private-address opt-in exactly as strict, and as patient, as before Two ways the shared lookup could refuse a host the opt-in path used to accept. A resolver slower than the bounded lookup's deadline is ordinary (the Linux default is 5s per server, twice) and that path blocked on an unbounded getaddrinfo before this PR, so it now falls back to the same unbounded call rather than reading the deadline as a refusal. And a transient failure is no longer written to the cache, so one SERVFAIL costs a retry instead of five minutes of refusal. Only answers are cached now, which is also what makes a timeout harmless. * Studio: release the in-flight slot a resolver worker actually took The worker read the semaphore off the module by name, so anything that rebinds it while an abandoned lookup is still asleep, which the test fixture does between tests, made that worker release an instance it never acquired and BoundedSemaphore raise inside the thread. It holds a local reference now. * Studio: list Tencent's metadata addresses, and stop re-resolving after a timeout Narrowing the resolved-address check to the metadata services themselves dropped the ones that live on link-local and were only ever caught by the 169.254/16 clause. metadata.tencentyun.com is reachable at 169.254.0.23 on VPC and 169.254.10.10 on the classic network, so both are listed exactly. The opt-in path also started a second bounded lookup of a host the metadata check had just given up on, then fell back to a third, unbounded one. It reads the cache the first lookup fills and otherwise goes straight to the fallback, so one validation spends one deadline. * Studio: make a busy resolver pool wait its turn rather than skip the check Returning no answer the instant the pool was busy meant a caller who stalled the slots got the next hostname waved through unchecked. A caller now waits for a slot up to the same deadline, and the pool is larger, so saturating it has to be sustained rather than momentary. It still ends in no answer, which the default path allows, and that is the deliberate half: refusing instead would let any resolver trouble stop the operator configuring a provider at all. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: mirror httpx's host escaping for ASCII names too The IDNA fix covered Unicode hosts and left the ASCII fast path diverging: httpx percent-encodes what RFC 3986 does not allow in a reg-name, so safe^alias.example is dialled as safe%5Ealias.example, a name whose parent zone can answer differently from the one being checked. The lookup applies the same quoting now, and a test pins _transport_host against httpx's own raw_host rather than against a copy of its rules. * Studio: shorten the DNS deadline the event loop actually waits on The wait is synchronous inside async handlers, so it is shared by every concurrent request. A resolver that can answer a provider hostname does so well inside half a second, and past the deadline the answer is treated as unknown either way, so the extra second and a half bought accuracy for nobody. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 06:55:20 -07:00
Daniel Han	2098b7cd43	Add an extra llama-server arguments box to the model settings (#8702) ... * Add an extra llama-server arguments box to the model settings llama-server documents 283 flags and Studio already emits or manages about 115 of them, so the long tail has no way to be set from the app. The pass-through itself already existed end to end (LoadRequest.llama_extra_args, validate_extra_args, per-model persistence, the CLI), it just had no control, which is why the overrides route deliberately preserves the field when the panel saves. Adds the control, and closes the denylist gap that opens once a flag can be pasted in rather than typed on a command line. --tools was already denied because it enables exec_shell_command; --agent is documented as enabling all built-in tools, so it reached the same capability one alias away. Also denied: --tools-runtime (docker:, podman:, ssh:<target>), the MCP server flags, the CORS flags, --media-path, the log redirection flags Studio parses its own startup output through, --slot-save-path, and the flags that exit instead of serving. --slots and --props stay a user choice: Studio reads GET /props and never /slots. The editor checks what is typed against the INSTALLED binary's --help rather than a list shipped with Unsloth, since a custom or newer llama.cpp is exactly the case a bundled list would get wrong. probe_server_capabilities already parsed that map and threw it away; it is now returned by GET /api/inference/llama-flags. A flag the build does not document warns but still loads, an unprobeable build says nothing, and a flag Studio owns is refused with the control that sets it named. A flag that shadows a control is allowed and noted, because the backend appends extras last and reconciles its own sizing, and the CLI has always allowed it. With the box empty the emitted command is byte-identical to before. * Read the stored flags under the repo key too, and reword the unknown-argument error The panel keys per-model settings by repo:QUANT, but the overrides route carries flags stored under the bare repo id into the first per-quant save, so reading only the exact key showed an empty box for an entry that was about to be inherited. Caught by driving the real panel: flags set through the API for unsloth/gemma-3-270m-it-GGUF did not appear. The unknown-argument message pointed every reader at the extra arguments box. Nothing reaching the classifier says whose flag it was, and Unsloth emits its own conditionally on the capability probe, so a binary swapped under a cached probe lands there too and would be sent to a box it never came from. * Block the load when an extra argument would be refused The row painted a managed flag red and left Load model enabled, so the panel started a load validate_extra_args then refused. Seen on a real Studio: --parallel 8 showed its error and the button stayed live. The row now reports loadability up, and the objection leaves with the row so a model without the box (diffusion) still loads. * Send the extra arguments on the panel's own load The Load button in the model settings goes through use-chat-model-runtime, which builds its payload field by field, so the flags stayed in the config and never reached llama-server. Caught by loading gemma-3-270m from the panel with --top-k 20 in the box and reading the emitted command, which did not have it. Omitted rather than nulled when the config has not read them, so the route still preserves flags set outside this panel. * Refresh a comment whose premise this change retires The forget path said flags have no UI control. They do now; what the sentence was actually about is that an omitted field is preserved, which is what a save that never opened the box relies on. * Bound the argument-error scan main's test_a_huge_unterminated_line_is_cheap caught this on the rebase: the two new regexes ran over the whole capture, and _drain_stdout keeps an unterminated line whole, so a 10 MB single line was scanned twice and the classifier went past its 200 ms budget. llama.cpp prints the argument error and exits, so the tail is where it always is. * Drop, do not refuse, flags an existing install stored before the denylist grew Widening the denylist is the one part of this change that acts on data already on disk. An override written when --log-file or --slot-save-path was still allowed still holds it, and the two paths that CARRY such a value over rather than receive it were refusing: - the settings save re-validated the carried-over list, so changing Context Length on that model returned 400 naming a flag the user was not editing and cannot reach from that payload - the load path caught the same refusal but degraded to no extras at all, so one name added to the denylist silently took every other flag with it Both now use drop_managed_flags, which removes only the denied names (with their values, or a bare /var/log/x would be read as a positional model path) and re-validates what is left. An argument the caller just sent is still refused loudly; an argument merely carried over from storage is dropped quietly. Also off the event loop: the flag catalogue route awaited probe_server_capabilities inline, and on a cold cache that runs llama-server --help with a 10s timeout. The startup probes were moved to a thread for exactly this reason (test_startup_llama_probe_non_blocking), so the first open of the panel after an update would have stalled every other request. Two new suites: the compatibility one covers every carry-over path with flags an older build allowed, and the platform one runs the Cartesian product of Linux/WSL/Windows/macOS with NVIDIA/multi-GPU/Vulkan/CPU-only, asserting the command is unchanged with the box empty and that an extra arg never moves a placement flag. Twenty thousand random inputs through the box and through Python's shlex also agree on every one that does not contain a backslash against a newline or end in a lone backslash. Those two are deliberate: POSIX 2.2.1 makes an unquoted backslash before a newline a line continuation, which shlex (a lexer, not a shell) does not implement, and a text field that refuses a half-typed trailing escape is worse than one that carries the character. Both are pinned as named tests now. The same run showed an unquoted Windows path loses its separators, which is what every POSIX shell does and not something to change in the splitter, so the hint asks for quotes around backslashes rather than only around spaces. * Name the oversized parametrize cases so they can run on Windows Windows CI errored 16 tests in setup with ValueError: the environment variable is longer than 32767 characters. pytest puts the whole parameter in the node id and the node id into PYTEST_CURRENT_TEST, and these cases are 100 KB blobs, so the limit is hit before the test body runs. Linux and macOS have no such cap, which is why it went unnoticed. The blobs still exercise the linear-time claim; only their ids are short now. * Make the box and the launch agree, and keep the objection while the row is hidden Ten fixes from the review round, the two largest of which were the box telling the truth about what will run: - Hydration filled the textarea but left the config undefined, so clicking Load without editing sent no llama_extra_args at all. The route's omission path inherits from a resident same-model process, not from the stored override, so a cold start launched without the flags the box was showing. Confirmed against a live Studio: the override held --numa distribute and the emitted command did not. Hydration now writes the config too. - Collapsing Advanced settings unmounts the row, and its cleanup withdrew the objection while the tokens stayed in the config, so Load was re-enabled for arguments the backend refuses. The panel now holds the flag and retires it on a model change instead. The rest: - StrictMode replays an effect (setup, cleanup, setup), and marking the key before the request left the first fetch cancelled and the second setup returning early, so hydration never ran in development. Marked once a response is in hand. - The lookup checked two literal keys while the overrides route folds identities and falls back from repo:QUANT to the bare repo, so a model whose entry was written under another casing showed an empty box. resolveStoredExtraArgs mirrors that order. - The editor checked the token cap but not the byte cap, so a grammar or JSON schema over 32 KiB was loadable in the UI and refused by validate_extra_args. Measured in UTF-8, which is what the backend counts. - isDefaultConfig did not look at llamaExtraArgs, so a config whose only change was the box read as default and savePerModelConfig deleted the entry it was asked to remember. - Records carrying the new field are stamped v3. This scheme exists so a client from before a field refuses to rewrite records containing it; without the stamp an older cached tab would normalize the arguments back out on its next save. - The catalogue published llama.cpp's removal stubs (--draft-max and friends, listed only to say the argument has been removed), so the editor called a removed flag supported and stayed quiet about a load that then failed. _is_real already existed for this. - probe_ok equated "parsed something" with "read the whole thing". A --help that exits nonzero after partial output leaves a non-empty catalogue, and every flag past the failure point would have been reported as unsupported. The probe's own result is published and used. * Name the scan-tail constant for what it bounds It only ever bounded the argument-error branches. The dyld branches above it have to keep reading the whole capture, because the pathological part of the input they guard against is the library name inside the line itself (TestTheLibraryNameIsBounded), so a tail would cut off the framing they match on. * Close the third carry-over path, and say what the GPU picker takes away The auto-switch path was the one place a stored override still became an explicit request: model_override_load_kwargs copies llama_extra_args straight into LoadRequest, and an explicit list is refused rather than trimmed, so an override written before a flag was denylisted broke every OpenAI auto-switch and idle reload of that model until someone rewrote the entry by hand. It is sanitized at that seam now, like the inheritance and settings-save paths. The editor also told two small lies and enabled two loads that could not succeed: - --device, -dev, --main-gpu and -mg are stripped from the command whenever the GPU picker owns placement (_strip_device_extra_args), so the "passed after the settings above, so yours wins" reading was wrong for them. They are now called removed, and only while a GPU is actually selected. - --ctx-size nope and -ngl many reach parse_ctx_override and parse_gpu_layers_override, which refuse the load. The row mirrors those checks. - A control character (the usual source being a command copied out of coloured terminal output) is refused by the backend and was loadable here. - A GGUF that turns out to be DiffusionGemma after classification kept its tokens in the loadable config, and the diffusion shim records them without ever passing them to anything. Cleared with the other flags that path already drops. And the trim that keeps a legacy list inside the new bounds no longer leaves a flag without its value: ['--grammar', <33 KiB>] became ['--grammar'], which this validator accepts (it knows the arity of only a few flags) and llama-server then refuses at startup. /validate now carries the arguments too. It already resolves and passes them to the estimator; only the request field was missing, so a --ctx-size or cache override sized the preflight differently from the load it was approving. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hydrate where the panel always mounts, and carry the flags into auto-load Four things a closer read of the review turned up. Hydration ran inside the extra-arguments row, which lives in Advanced settings and is not rendered while that section is collapsed, so a panel opened closed never fetched the stored override and a cold load launched without it. Moved to the panel, which always mounts. The startup auto-load built its request from the local config and left the flags out. There is nothing resident at that point for the omission path to inherit them from, so a remembered list was dropped on exactly the load that most needs it. The preflight check sends them too, or it would approve a command the load then refuses. The flag catalogue was cached for the whole session. A llama.cpp update or a backend switch replaces the binary in place, so a newly added flag kept being reported as unknown. One minute now, plus an explicit drop on both paths that change the binary. Short aliases were parsed out of --help and thrown away, so typing -fa warned that a valid flag was not in this build. * Keep a wrapped paste to its own tokens, check every copy, log names only A backslash-newline contributes nothing, so it must not mark the start of a token: a command wrapped and indented under its first line, which is how they are pasted, gained an empty token at that indentation and sent llama-server a bare positional it reads as a model path. The numeric-value check ran once per flag name, but llama.cpp reads the last occurrence and so does parse_gpu_layers_override, so `-ngl 20 -ngl many` left Load enabled for a request that comes back 400. Checked at every occurrence now, reported once. drop_managed_flags put a rejected VALUE into the list its callers join into a warning log. A stored value carrying ANSI escapes belongs in no log line, so it is a placeholder now, and the flag it belonged to is named rather than disappearing silently. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the box, the button and the preflight honest about what will launch The textarea seeded itself once. Reset and the panel's own hydration both replace the arguments while the row is mounted, so the box could show one thing while Load sent another. It now follows a change it did not make, and ignores the echo of the user's own typing. Load was not gated on the stored-arguments read, so a fast click on a cold model launched without them. Gated now, and released on failure or after four seconds, because the gate is there to win a race rather than to make the button depend on a service being up. /validate took the field but then passed None to the resolver, which returns that argument unchanged for an explicit list, so a --ctx-size override was missing from the estimate that approves the load. The overrides fold split on the last colon, which is legal in a POSIX filename: /models/foo:Bar.gguf folded onto the real, different file /models/foo:bar.gguf. It uses splitQuantSuffix now, as the backend does. An unpaired surrogate passed validation and then made Popen raise while it encoded argv, after the load had begun switching models. Refused at the boundary, and dropped rather than kept on the carry-over paths. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Read the same override row the server would, and clear what Reset clears Five of these are about the panel and the auto-switch loader disagreeing over one stored map, which matters because the panel hydrates from a row and then sends what it found as an explicit list. The lookup fell through to the bare repo id whenever the variant row carried no arguments, where the server breaks at the first non-empty entry and reads its fields from there. Two keys that fold together now resolve to nothing, as resolve_model_override_key does, rather than whichever was enumerated first. A WSL drive mount folds case insensitively, because it is a Windows volume seen through Linux and the server folds it that way. Hydration also turned a stored list into an explicit request, so a flag added to the denylist since it was saved came back as a 400 instead of being dropped quietly; the stored list is sanitised against this build's denylist before it becomes one. The numeric check accepted a missing or out-of-range value, so --ctx-size with nothing after it, --ctx-size -1 and -ngl -2 all left Load enabled for a request parse_ctx_override and parse_gpu_layers_override refuse. Reset left the field absent, which the load path reads as inherit, so a reload after Reset kept the very flags the empty box said were gone. It is an explicit clear now. * Sanitize a stored list the way the server does, and fold paths its way The hydration sanitizer only removed denied flags, but this branch also made the bounds, control characters and unpaired surrogates refusals, and a list saved by the previous release can carry any of them. It now mirrors drop_managed_flags whole, including shedding from the tail without leaving a flag without its value. Path folding only lowercased, where the server also swaps backslashes for slashes, trims trailing separators and folds a UNC share, so an override stored as C:\Models\Foo.gguf was missed for a picker key spelled c:/models/foo.gguf and a forward-slash UNC path was read as a case-sensitive POSIX one. The cache-type and split-mode flags take a value that _last_flag_value insists on, so a bare -sm or a trailing --cache-type-k now reads as the error it is rather than only a note about shadowing. The four-second release was on the Load gate, which let a click through while the overrides response was in hand and only the cold --help probe was outstanding. The deadline is on the catalogue now: past it the stored list is applied with whatever the catalogue gave, and the gate itself has only a last-resort release. A stored flag whose own token carried a control character was dropped without its value, leaving a bare positional llama-server reads as a model path, and its name went into the log line escapes and all. The backend switch invalidated the flag catalogue when the job STARTED, before the binary was replaced. It happens on successful completion now, for a tab that only watched the switch as well. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep an emoji, read the denylist without the probe, and leave other models alone The surrogate check matched both halves of every well-formed pair, so a stored chat template or grammar carrying an emoji was dropped on hydration along with its flag, though Python encodes it happily. Only an unpaired half is refused now, which is the one Popen raises on. Sanitizing waited on the flag catalogue, and its first read runs llama-server --help with a ten second timeout. The denylist needs no binary at all, so the route answers it on its own now and hydration waits for that instead: no window in which a legacy flag is left in an explicit request, and no deadline to release early. The scrub list covered the twins of every denied flag except the logging ones. Studio classifies a failed start by reading llama-server's output and emits nothing later that would override an inherited redirect, so LLAMA_ARG_LOG_FILE turned every failure into the same opaque diagnosis. A GGUF reclassified as diffusion after the box was typed into kept the row's objection while the arguments were stripped from what loads, so Load stayed disabled over arguments the request no longer carried. And a Transformers or MLX model waited on both requests before Load became live, though nothing outside GGUF renders the row or sends the field. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Let a grammar wrap, read the load-path keys, and restore what was running The control-character check I tightened last round swallowed the newline, so a quoted multi-line grammar, schema or chat template disabled Load even though validate_extra_args accepts tabs and newlines on purpose. The editor now uses the backend's rule exactly. Hydration asked under the repo id only. A cached GGUF outside the active HF cache loads by its snapshot path, and the auto-switch loader reads the path-qualified key first, so an override left there stayed the one API loads apply while the panel showed none of it. The candidates are the loader's own, in its order, including the filename-label key an early build wrote for a loose .gguf. Windows passes CreateProcess one string for the whole command line, capped at 32767 characters, so 32 KiB of extras alone left nothing for the model path and Unsloth's own flags. The cap is 24 KiB there, refused at the boundary rather than inside Popen once the switch has started. And a rollback after a failed switch omitted the arguments, which by then inherits from the newly resident target rather than from the model being restored, so the previous model came back without the flags it had been running. The client keeps what it last knew that model was running and sends it explicitly. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard the cleared list in the rollback snapshot tsc -b is stricter than the standalone check I ran: llamaExtraArgs is string[] | null, so reading .length on the cleared case failed the build. * Refuse an ownerless value, size the preflight, and stop trusting a stale probe A bare token in the box belonged to no flag and was accepted. Today's llama-server answers "invalid argument" and refuses to start, so it is a failed load rather than a 400, and a build that did take a positional would read it as the model path, which is the one thing denying -m and --model exists to prevent. Every value must now follow its flag, with the two-value flag in llama-server's help allowed for. The Windows cap counted bytes, but list2cmdline doubles backslashes before a quote, so an escape-heavy grammar passed the cap and still blew CreateProcess's limit once serialized. The check measures what Popen would actually write. The runtime preflight left the arguments out while the load sent them, so /validate approved a differently sized command. During training that means approving the switch, unloading the resident model, and having /load refuse the target. With Advanced collapsed the row never mounts, so nothing judged a stored list this build refuses and Load stayed live for a request that 400s. The hydration judges it now, which the managed set is enough for. A diffusion GGUF is GGUF-shaped but runs through the shim that appends no llama-server flags, so it no longer waits on the hydration either. And an invalidated catalogue kept its in-flight request, which wrote the previous binary's flags back and was handed to every caller for the TTL. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Say so in the box when a value has no flag The backend refuses it, so without this the editor looked content and the load came back 400. Proven against a live Studio: /load answers 'extra llama-server args cannot contain a bare value' for a bare path, and the box now says the same before the button is pressed. * Retire the bare-positional pass-through the new check contradicts Staging CI caught two pinned expectations I had not run: a suite asserted that a bare token passes through, and the tab and newline cases passed their string with no flag in front of it. The first is the contract this round deliberately changes, so it now pins the refusal and says why: llama-server answers "invalid argument: foo" and refuses to start either way, and a build that did take a positional would read it as the model path. The second keeps its intent, with the string where such a string actually arrives, as its flag's value. * Prove the scan bound by behaviour instead of by a stopwatch This test is the one job still red on Windows, and it is red on main there too: a 0.2s budget on a shared runner measures the runner, not the code. The bound is observable without timing at all, so it is asserted directly now: an argument error 10 MB back is not reported, one in the tail is, and only an unbounded scan can tell the difference. Checked both ways, by widening the bound and watching the buried error reappear. The stopwatch survives as a catastrophic guard at five seconds, which no runner can trip and no sane implementation can exceed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Let the backend answer what a load will apply, and publish what it knows The panel kept re-deriving the backend's own answers in the browser, and each round found another way the two drift. Both now come from the source. Which stored row a load reads is resolved server-side: the candidate order the auto-switch loader uses lives in one function that the loader itself calls, and the overrides route can be asked for a named model's entry. That closes the casefold gap for good (Straße and STRASSE are one path to Python and two to a browser), along with the whole class it belongs to. The local resolver stays as the fallback for a backend without the parameter. The flag catalogue now says which flags take no value, read from the same --help it already parsed, so the editor can tell "--verbose foo" (a typo llama-server refuses) from "--numa distribute". It also reports this host's size limits, so a 25 KiB grammar is refused in the box on Windows rather than at the load, and the quoted-length rule is mirrored exactly: a port of list2cmdline, checked against CPython's own output over 2010 generated cases with no mismatches. Two more from the same review: a model whose arguments live only on the server no longer opens looking like every default, and a diffusion GGUF neither sends them (its runner appends none) nor has them recorded, with the backend clearing them when its authoritative classification picks that runner. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the user's verdict, bound what the child says, and close two env twins An edit made while the overrides request was in flight had its verdict replaced by the answer about the stored list, so Load went live for invalid text; the hydration now leaves both the config and the judgement alone once anything has been typed. The same request also sanitizes the list already in local storage, which a build that allowed those flags may have written and nothing else would catch while Advanced stays collapsed. A failed start quoted an argument straight out of the child's output, and a wrapper on LLAMA_SERVER_PATH can print 64 KiB of non-whitespace, so the quoted fragment is bounded now. Two more environment twins go with the flags they mirror: --api-prefix moves the /health endpoint every load waits on, and an inherited API key makes the healthy child refuse the requests Studio sends without one. llama.cpp spells one of them LLAMA_API_KEY rather than LLAMA_ARG_, so the invariant tying twins to denied flags now accepts both prefixes. And a compare pane resolves the stored arguments before it loads, since the omission path inherits only from a resident instance of the same model, which is exactly what a pane switching away from the other one does not have. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep a two-value flag whole, and judge what each end can actually know Five items from the last review. Two-value flags: validate_extra_args now tracks the values --control-vector-layer-range is owed and refuses a list that stops between them, and drop_managed_flags sheds the whole option when the size bound would take only its second value. Half of it is not a smaller version of it: llama-server refuses the option on the command line, so a trimmed list used to fail at spawn instead of at the boundary. /validate now drops pass-through arguments for a confirmed diffusion GGUF, as /load already does. The estimate that approves the load was reading a --ctx-size out of a list the diffusion runner never receives. The classification is only authoritative after placement, so the drop belongs there rather than in the caller, whose staged metadata can be inconclusive. The stored-argument sanitizer takes the host's own bounds instead of the constants: a Windows server takes 24 KiB and holds a quoted-command budget besides, and trimming to 32 KiB left a list /load answers 400 on. Compare panes now sanitize the list they fetch from the overrides map for the same reason the panel does, so an install upgraded across a denylist change keeps comparing. The editor also refuses a catalogued value-taking flag left without its value at the end of the text. That arity is only knowable from the probed help, which is why the check lives here and not in the backend validator; an unprobed build and an unknown flag both keep the benefit of the doubt. Also fixes a regression in the help parser: the flag-arity map and the removal-stub test were initialised inside the probe's try block, so a binary that timed out raised UnboundLocalError instead of falling back to "nothing known about this build". * Publish what the server is running, and stop guessing at the rest Five items from the last review. An attached value counts as one of the two: "--control-vector-layer-range=1" still owes its END, on both ends. llama.cpp b10360 refuses the "--flag=value" form outright ("error: invalid argument"), so this shape was never going to launch, but treating it as a complete option left Load enabled and the failure arrived at startup instead. /api/inference/status now publishes requested_llama_extra_args, and the store seeds its loaded baseline from it. A tab opened while a model was already running never saw the load, so that baseline stayed null; a switch that launched the target and then failed during refresh rolls back with the field omitted, and because the failed target is left resident, omission cannot inherit across models. The previous model came back without the arguments it was running. The panel no longer sanitizes a list the user typed while hydration was in flight. It snapshots what the config held when the request went out, so a legacy stored list is still cleaned while live input is left alone: typing --agent during that window used to clear the box instead of showing the error. A collapsed Advanced section is re-judged once the flag catalogue arrives. The row keeps its verdict when it unmounts, by design, but a verdict reached before the probe answered did not know which flags this build documents, and collapsing froze it, so a bare --threads kept Load enabled. Manual GPU memory now reports the offload flags it removes rather than claiming they win. The route translates a layer count into the first-class field before stripping, so -ngl really does win; nothing does that for --n-cpu-moe or --fit, which are dropped and replaced by the controls' own values. * Say which arguments the launch will drop, and stop one that aborts it Three items from the last review, plus a CI fix. Model Memory removes flags too. apply_model_memory_policy emits its own load mode when "Keep model in GPU memory" is on and strips every other load-mode bearing flag, because a trailing one resets the mode and would drop the lock; "Don't reserve system RAM" drops the flags that hold a full host copy, leaving mmap and DirectIO alone. The editor said those arguments win, so one could be typed, saved and never passed. It now reports them as removed and names the setting that owns them, the way it already does for GPU-owned placement. A pass-through batch size below max(slots, 2) is refused. The loader raises the value it emits itself, with measured evidence in the launcher comment that b1 aborts at any slot count and b4 with 8 slots aborts, but an extra -b is appended after it and wins, so the load started and llama-server aborted on the assertion. The floor here is the same one the batch control shows, including its limit: with Slots blank only the hard 2 is asserted, since the server-wide default is not visible to the page. A background auto-load now resolves a server-only override. resolveInitialConfig reads local storage, so an override written through the API or from another browser left the field undefined, and at startup there is no resident process for /load to inherit from: the model came up without the arguments saved for it. Fetched and sanitized like the panel and the compare panes do, and sent to both the preflight and the load. The diagnostics context is now one object rather than a run of booleans. Every setting that owns a flag group adds one, and five positional flags would be read wrong at a call site long before they stopped compiling. CI: the two HF credential-persistence tests waited on a fixed budget of setImmediate turns for a real promise chain to settle. That is a bet on how many turns the runtime needs, and a two-core runner running the suite in parallel loses it; they now wait on a deadline. * Carry the arguments through every path that loads a model Four items from the last review, and the backend-test failure the last one caused. Applying from the Hub's full-page settings now passes the submitted config into the selection. The runtime store does not carry llamaExtraArgs, so a selection made without the config left the field undefined: the load omitted it and the route kept the resident server's list, which meant an edit made there did nothing and a clear did nothing either. The background auto-load resolves the override under the advertised repository id as well as the path it loads from. Cached inventory can hand back a different loadId, and the row was written under whichever of the two the user was looking at, so passing one identity twice missed it. It also sanitizes a list that came from local storage, not only one fetched from the server. That copy was written by whatever build was running then, so a flag added to the managed set since would be sent explicitly and answered with a 400, and the remembered model would not come up at all. A collapsed Advanced section stops objecting once the list is empty. The row leaves its verdict standing when it unmounts, by design, because the tokens it objected to still go out with the load; after Reset there are none, and Load stayed disabled over arguments the request no longer carried. Backend: the runtime projection reads requested_extra_args through getattr. The drift check below it is what turns a backend missing a runtime field into one clear error naming all of them, and reading the attribute directly pre-empted it with a bare AttributeError, which is what the two projection tests caught. * Read a flag the way llama.cpp reads it, and keep the standing verdict Six items from the last review. The underscore spelling is a spelling, not an attached value. _flag_name folds --ctx_size onto --ctx-size and llama.cpp takes both, so deciding attachment by comparing the folded name against the raw token read "--ctx_size 4096" as a flag carrying its own value and then refused the 4096 as a bare token. That is a list the CLI has passed through since the field existed. Attachment is now decided from the token itself: an "=" form, or an attached short like -np8. A flag that interrupts another's value is reported. "--numa --verbose" leaves --numa without the value this build's help says it takes, and the obligation was overwritten by the next flag before the end-of-input check could see it. The batch floor follows the server-wide slot count. /llama-flags now publishes default_parallel_slots, read the way the loader reads it, because the browser cannot see that number: with Slots blank the launch serves four, so -b 2 cleared the editor's hard floor of 2 and aborted llama-server anyway. Compare panes sanitize a local list as well as a fetched one, for the reason the auto-load path already does: either becomes an explicit argument, and a config saved by an older build can still name a flag that is managed now. The hidden revalidation can only tighten its verdict. It judges the tokens, and formatting them back into a string supplies the closing quote the user has not typed, so an unclosed quote read as clean and re-enabled Load. A mounted row re-reads the catalogue when the binary changes. Updating llama.cpp from the banner invalidated the module cache, but a row that had already read it kept judging arity against the old build's help until the section was closed and reopened. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair a stored list the way the server does, and refuse the batch that aborts Five items from the last review. The stored-argument sanitizer now sheds the shapes validate_extra_args refuses, not only the flags it denies. A token belonging to no flag goes (llama-server reads a positional as the model path, which is what denying -m exists to prevent), a two-value option left half-written goes whole in either spelling, and a flag whose value the backend's own parser refuses goes with its value. drop_managed_flags gets all of this for free by re-validating after each cut it makes; this mirror trims by size alone, so it has to know the rules. It removes the option rather than the tail, which keeps the rest of a legacy list working. A 3000-case differential run against the real validator now comes back clean; before it left 457 lists the backend would still have refused. /load enforces the batch floor for callers that never see the editor. The launcher raises the --batch-size it emits itself, with the measurements recorded beside it, but an extra -b is appended after that and wins, so a CLI or API caller sending -b 1 got a server that aborts during startup, after the previous model had been unloaded. Now a 400, checked once the effective slot count is known. A value this side cannot read is still left to llama-server, which names it better than a guess would. The loaded-arguments baseline follows a same-model reload from anywhere. Pinned at the first read, a tab kept the old list after another tab or an API client reloaded the same model with different arguments or with none, and a later failed switch resent it from the rollback path. The in-flight guard stays: while a load is in flight performLoad owns these values. The hidden validation subscribes to catalogue invalidation as well. The row holds its own subscription, but it is unmounted precisely when that check is the one running, so an in-app llama.cpp update with Advanced collapsed left a verdict reached against the previous binary standing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tell an empty list from an unknown one, and blame the right owner Two items from the last review. An explicit empty list is no longer collapsed to null, in the status the backend publishes and in the baseline the browser records. The rollback sends llama_extra_args only when it has one, and omitting it is what makes /load inherit, so a same-model reload that added arguments and then failed its refresh restored the previous runtime carrying the arguments it was rolling back. null still means nothing was ever recorded, which is the only case where inheriting is the right answer. A rejected argument VALUE no longer sends every reader to the extra-arguments box. Studio emits its own options conditionally on the capability probe, so a build whose --flash-attn enum does not take the managed value rejects something the box never held, and editing it cannot fix that. The classifier now takes the load's extras and names the box only when they really carry the flag; otherwise it gives the same neutral advice the unknown-argument branch already gives. An alias spelling falls back to neutral rather than guessing, since a wrong "you set this" is worse than no attribution. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Take the scale as a second token again, and read the load's own answer Four items from the last review. --lora-scaled and --control-vector-scaled may take their scale as a separate token. Today's llama.cpp writes it into the value (FNAME:SCALE) and older builds took it on its own, which _sidecar_weight_files already reads both ways, so the positional check refused a list those installs had been loading. Allowed rather than owed: requiring the second token would refuse the current syntax just as badly. Both ends carry the same rule, and a third bare token still has no owner. The committed baseline is what the load says it applied. A same-model reload that omits llama_extra_args but sets max_seq_length has its inherited --ctx-size stripped before launch, and the status refresh that would notice runs while the load is still in flight, so the next rollback resent a flag the reload had removed. The response's own echo is read first now. The legacy overrides fallback searches the caller's identities. A backend that predates the resolved field answers with the whole map and needs to be told which keys to read; the auto-load and compare callers passed none, so against an older server they searched nothing and launched without the stored arguments. The hidden hydration check applies the slot floor it already knows. Without it a stored --batch-size 2 released Load against a four-slot server, and a click before the full catalogue check landed reached the deterministic 400. * Prove the tail bound by behaviour in the second scan test too The Windows staging runner failed this one at 204ms against a 200ms budget, which is the runner and not the classifier: allocating and slicing a 10 MB capture is most of that. The sibling test was already converted to assert the bound by behaviour; this one kept the stopwatch. It now asserts what the bound produces: an argument error inside the scanned tail is reported, and one buried two tails back is not, which is only possible if the scan stopped short of the head. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Size the batch floor by the slots the load will really serve The floor was computed from the requested parallel slots, but load_model clamps that number twice before launch: a llama.cpp without --kv-unified serves one slot, and the diffusion runner takes no --parallel at all. A four-slot ask against such a build refused --batch-size 2 for a command that would have run it. _effective_parallel_slots mirrors both clamps and now backs the /load floor and the default_parallel_slots the editor sizes itself from, so the browser and the server agree on the same number. /api/inference/validate runs the same validation as the load it approves, so a switch is not approved on arguments the load will refuse. A successful load records the arguments the server was actually invoked with as the rollback baseline, on the auto-load path and in a compare pane; a non-GGUF load clears it instead of leaving the previous GGUF's list standing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scrub the inherited TLS twins, and let a cleared box clear its fallback Studio starts llama-server with the parent environment, and llama.cpp reads LLAMA_ARG_* before argv. LLAMA_ARG_SSL_KEY_FILE and LLAMA_ARG_SSL_CERT_FILE were not among the twins scrubbed, so a deployment that sets them for its own llama.cpp had the child listen on https while Studio probed /health and proxied over http: the server came up healthy and every load timed out. Measured on b10360, where the pair turns 'listening on http://127.0.0.1:PORT' into 'listening on https://...'. Both flags are already denied, so the twins belong in the same list. The first per-quant save copies a legacy bare entry's flags onto the quant key and leaves the bare entry in place. Clearing the box then writes an empty list, an entry with nothing usable left is not stored, and the next load falls back to the bare key and applies the flags again, from a row no page can show. A clear now reaches those fallbacks too, taking only the flags and only when no other quant could still be reading them, which is the rule the explicit forget already follows. A save that omits the field, and the fill pass, are unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse the attached --flag=value spelling llama.cpp has never had common/arg.cpp looks the whole token up in its option map and folds only the underscore spelling, so --top-k=20 is not --top-k carrying a value: it is an argument llama-server has never heard of. Measured on b10342 and b10360, where --top-k=20, --ctx-size=4096 and --flash-attn=on each exit with 'error: invalid argument'. Both the validator and the editor treated any = as an attached value, so the load was approved, the resident model was torn down, and the child then refused to start. It is refused at the boundary instead, while the message still has somewhere to go, naming the two arguments to write. Splitting it here would be a guess: for a switch the text after = is not a value, and this module cannot know an ordinary flag's arity. A managed name keeps the message that says which control owns it, and an = inside a VALUE (--override-kv a=int:2) is untouched. The carry-over drops such a token beside the denied names rather than leaving it to the tail trim, which would have cost every flag written after it. The editor's own trim had the same underscore blind spot as the validator did: it compared the normalized flag name against the raw token, so a value shed for the size bounds left --grammar_file standing. Any flag whose value has gone now goes with it. Backend 1665 tests pass; frontend 2364; the differential fuzz over 3000 sanitizer outputs still has the validator refusing none of them. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scrub every documented env twin, and keep the slot probe off the loop The scrub list had grown one name at a time. It is now the whole set: every '(env: NAME)' the bundled b10342 help documents for a flag this module refuses, which adds the web UI and its MCP proxy (upstream marks that one as not for untrusted environments), --path's LLAMA_ARG_STATIC_PATH, the multi-model server mode, the model and repo selectors, and the host, port and slot count. llama.cpp applies the environment before argv, so the ones Studio always emits were overridden anyway; the rest were reachable through the environment while the same flag was refused in the box. A mapping records which flag each twin belongs to, since the name does not always say, and the drift test reads that rather than deriving it from the string. HF_TOKEN is deliberately left alone: it is the standard Hugging Face credential Studio's own downloads use, not a llama-server behaviour switch, and the child is always given a local model path rather than a repo to fetch. /api/inference/llama-flags computed the effective slot default inline, which on a cold cache is llama-server --help with a ten second timeout on the event loop, including on the managed-only answer that exists to avoid waiting for a probe. It runs in a worker thread now, and a single-slot default still needs no probe. Invalidating the flag catalogue now drops the managed answer as well. It carries defaultParallelSlots beside the denylist, and that is the effective count, so a tab that had already fetched it went on sizing the hidden hydration check's batch floor from the binary that was just replaced. * Record a cleared box as a row of its own, and refuse padded flags Clearing the box for one quant wrote an empty list, which stores no row, so the next load fell through to the legacy bare repository row and applied the flags again. The first fix stripped that row instead, which was wrong twice over: the bare row is the fallback for every quant that has no row, so forgetting Q4's flags took Q6's with them, and the guard against that made the clear do nothing whenever a sibling quant had a row of its own. An explicit clear now keeps a row on the quant's own key saying it has no flags, and only when a fallback would otherwise answer for that model. Nothing else is touched: the legacy row keeps serving the quants still reading it, and a clear with nothing behind it stores nothing, as before. An explicit forget still removes the row. A flag quoted with stray spaces was accepted too. _flag_name strips before it looks anything up, so "--top-k " passed the denylist and the arity walk as --top-k and went to the child with the space on it, and llama.cpp looks the whole token up: measured on b10342, it answers 'error: invalid argument: --top-k', naming a flag that reads as correct in the log. Refused now in the validator and in the row, with the value dropped alongside it on the carry-over path. A VALUE may still end in whitespace, which a grammar or a chat template routinely does. Backend 1106 tests pass; frontend 2366; the differential fuzz over 3000 sanitizer outputs, now including padded and attached spellings, still has the validator refusing none of them. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the projector twins out of the scrub, and stub the auto-load hydration The env-twin sweep took LLAMA_ARG_MMPROJ and LLAMA_ARG_MMPROJ_URL with it. --mmproj is refused in the box because Unsloth resolves the projector itself, but the environment twin is an INPUT: _launch_has_mmproj reads both to know the launch has a projector at all, and that is what keeps the vision and audio state of a model loaded through an inherited one. Scrubbing them cleared that state, which test_gpu_init_crash_message caught on three cases. Only the paravirtual CPU recovery drops them now, as before, where an unpinned projector is the corrupt path it is undoing. The pooling twins stay listed but noted: load_model already pops LLAMA_ARG_POOLING / _RERANKING / _EMBEDDINGS itself. The auto-load failure-gate harness slices chat-adapter.ts and evaluates it, so the stored-arguments hydration added there left three names undefined and every scenario failed on the harness's own missing-symbol assertion. Stubbed neutrally: these scenarios are about the failure gate, not about which flags a model launches with, so fetchLoadExtraArgs answers 'nothing stored' and the sanitizer is identity, leaving the /load payload each scenario asserts on unchanged. Merged origin/main first, so this branch carries #8783 (a deliberate segfault in the probe tests no longer dumps core). * Validate the command the load will run, and hold the quotes the user typed /load translates an explicit -ngl out of the extra arguments into the first-class field when Manual GPU Memory owns the offload flags, then strips them. /validate did not, so the call that APPROVES the switch judged a different command than the one that runs: a diffusion GGUF asked with gpu_layers 0 and '-ngl 20' was approved as a load that places nothing on any device and cannot compete with training for VRAM, and then launched twenty layers on it; the opposite pairing refused a load that only ever runs on the CPU. Same translation and same strip now, under the same manual-only condition, so both paths size the same launch. The managed flag answer is guarded by the catalogue generation, as the full catalogue already was. Its denylist is Unsloth's own, but defaultParallelSlots travels with it and that is the effective count for the probed binary, so a request already on the wire when llama.cpp is replaced could repopulate the cache the invalidation had just cleared and keep the old build's slot floor for the rest of the session. Its finally is conditional too, or it would clear a newer request's in-flight promise. A quoted value beginning with a hyphen is a value again. parseExtraArgs drops the quotes, so '--chat-template "- hello"' left a flag-shaped token in value position and the row called the option's value missing, disabling Load over a list the backend accepts and llama.cpp reads correctly. The tokeniser records which tokens were quoted and the diagnostics read that only in value POSITION: a flag quoted out of habit is still a flag, and a quoted token with no option in front of it is judged as written. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the remembered arguments into every launch, and clamp the floor too A launch that only applies the remembered config to the runtime lost the flags. applyPerModelConfigToRuntime has no field for them, and /load inherits them only from the SAME resident model, so the Hub launch and the training handoff ran a cold start, or a switch from another model, without the arguments the user had remembered for it. Both selections now carry the resolved config itself, and neither invents one when nothing is remembered: the field stays absent, which is what lets a resident model keep its own flags. The batch floor had the same shape of gap as the published default did. That default is effective, but an EXPLICIT Slots value chosen in the panel is not: a build without --kv-unified serves one slot however many are asked for, so Slots 4 with '--batch-size 2' was refused in the editor while the backend, which clamps to one, accepts exactly that command. The catalogue now publishes whether this build clamps, computed by asking the same helper the load uses, and the three floor sites share one function so the row and the two hidden checks cannot drift. A backend that publishes neither field reads as 'not clamped', which is the behaviour those builds had before. * Pass the slot clamp into the collapsed-panel hydration check * Judge the same command the launcher runs Three things the editor and the routes were judging differently from the server. Validate the pass-through list before the manual offload translation rewrites it. In manual GPU memory mode /validate translated "-ngl" out of the list and stripped the token before validate_extra_args ever saw it, so "--gpu-layers=20" passed the preflight and /load answered 400 on the same list, and a malformed "-ngl bad" raised an uncaught ValueError instead of the 400 it was meant to be. Size the batch floor from the slots an embedding GGUF actually serves. --embedding caps the batch at the micro-batch and llama-server aborts when that is below the slot count, so load_model reduces the slots to it before launching: four slots with "-ub 2" and "--batch-size 2" launches at two slots, where two is the floor, and both routes were refusing it. Read only after the floor has already refused, so the header read is paid on the way to a 400 rather than on every load, and only a positively classified embedding GGUF relaxes anything. Keep an explicit empty argument list apart from a missing one when the panel hydrates. The settings page writes an empty list on purpose when the box is cleared for a quant whose bare-repository row still carries arguments; read as "nothing stored" the panel left the field undefined, its next Load omitted it, and /load carried the resident model's arguments over, which is exactly what had been cleared. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the cleared-box tombstone, and keep its header read off the loop The panel's explicit-clear hydration rests on a backend rule with no test of its own: a save that clears the box for one quant keeps a row holding an empty list, so a load stops there instead of falling through to a legacy bare-repository row that still carries flags. Four cases now pin it: the clear that suppresses the fallback while leaving the bare row for the other quants, an empty save with nothing to suppress storing nothing at all, a fill never writing one, and the flag deciding only what an empty list means. The embedding classification behind the batch floor reads the GGUF header, so run it in a thread: both routes reach it from an async handler that is also serving download progress polls. --------- Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 06:51:21 -07:00
Daniel Han	c3230a749a	Hand the LoRA to vLLM on TRL 1.x GRPO rollouts (#8701) ... * Hand the LoRA to vLLM on TRL 1.x GRPO rollouts TRL 1.10.0 dropped the collective_rpc("reload_weights") call from VLLMGeneration.generate, so the regex anchored on it matched nothing and raised. The lora_request injection ran after that anchor in the same function, so it never happened, and _init_vllm/sync_weights had already been installed by then. fast_inference=True GRPO therefore sampled every rollout from the base model with the adapter ignored, and reported finite losses while doing it. Wrap VLLMGeneration.generate instead of rewriting its source, and intercept on the vLLM engine (generate/chat/collective_rpc), which is the part Unsloth owns. Restore all three methods if any patch fails, so the half-patched state is unreachable, and wake a sleeping engine before sync_weights returns early. * Raise the TRL cap to 1.10.0, and open the datasets cap to 4.x Two metadata caps, and both have to move together or neither does anything. trl<=0.24.0 is metadata only. Nothing in unsloth or unsloth_zoo reads the TRL version and refuses 1.x at runtime; the only comparisons are floors at trainer.py:725 and :993. Measured against real TRL 1.10.0 with the trees on PYTHONPATH so no resolver ever sees the cap: import ok, tests/ -k grpo 991 passed, tests/version_compat 1702 passed, and four real 3-step GRPO runs whose grad-norms and reward stats match TRL 0.22.2 to every digit. The 5-test gap against 0.22.2 is this repo's own TRL >= 1.7.0 gated assertions, which 0.22.2 skips and 1.10.0 runs and passes. datasets<4.4.0 is the second cap, and raising only the TRL one ships a pin no user exercises: with the datasets cap in place, a resolver asked for trl<=1.10.0 settles on trl==0.29.1, because TRL 1.10.0 requires datasets>=4.7.0. So the datasets cap has to open at the same time. It opens as a hole rather than a floor bump. A datasets>=4.7.0 floor would drop Python 3.9 and force 4.x on TRL 0.22.2 users who do not need it. The hole !=4.4.*,!=4.5.0 is exactly what patch_datasets in import_fixes.py already refuses at runtime over the RLock recursion bug, so metadata and the runtime guard now agree instead of the metadata being the stricter of the two. Verified on datasets 4.8.5 with TRL 1.10.0: 991 / 1702 passed, exit 0 both, and the GRPO smoke numbers are identical to the datasets 3.6.0 run to every digit. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Test that the GRPO rollout adapter reaches vLLM on every TRL generate shape The failure this locks down is silent: with TRL 1.10.0's generate, the old source patch anchored on collective_rpc("reload_weights") matched nothing, raised, and took the lora_request injection down with it, so fast_inference GRPO sampled every rollout from the base model at exit 0. The tests are shape-driven rather than version-driven. A synthetic trl.generation.vllm_generation goes into sys.modules with a VLLMGeneration whose _init_vllm and sync_weights carry the anchors the source patches need, and whose generate reaches a fake engine the way a given TRL era reaches it: sync_weights on 1.10.0, collective_rpc on 0.22.2, chat for conversational rollouts, and no engine at all in server mode. Then the real vllm_generation_init_patch runs over it and the fake engine records what arrived. No vLLM, no GPU, no network, and importlib_version is monkeypatched so the installed TRL is irrelevant - these pass against the 0.25.1 in this env, which the patch would otherwise skip outright. Also covered: the engine methods are restored after the call so a later model.fast_generate is clean, an engine that already carried its own generate gets it back rather than deleted, a TRL-created engine is left alone, a sleeping engine is woken before sync_weights returns early, a failed sibling patch rolls all three methods back, and patching twice does not double wrap. Seven of the ten fail on c16021504. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop the TRL and datasets cap raise, and keep only the rollout fix The cap raise is what moves anyone onto TRL 1.x, and that is a separate decision from fixing the silent rollout bug. Split out so this PR changes no resolved version for anyone: the pins go back to trl<=0.24.0 and datasets<4.4.0 exactly as main has them. What is left is inert on the TRL this repo installs. vllm_generation_init_patch returns at rl_replacements.py before touching anything unless the installed TRL is 0.28.0 or newer, and TRL 0.22.2 has no trl.generation package at all. It only does something for someone who has installed a newer TRL themselves, which is exactly who hits the bug. The tests build a synthetic trl module rather than importing the real one, so they run on any installed version: 10 passed here against TRL 0.25.1. The cap raise, with the datasets hole it needs to be reachable at all, moves to its own PR so it can be judged on its own evidence. * Verify the rollout patch against every vLLM release, and stop injecting over a positional adapter Checked LLM.generate, LLM.chat, LLM.collective_rpc, LLM.wake_up and LLM.sleep on every vLLM release from 0.11.0 to 0.27.1. All are present at each one, LLM has no __slots__ and no __setattr__/__getattr__ hook, and none of the methods are decorated, so the instance-level override always takes. One real gap turned up. lora_request is keyword-only on generate at every version, but on chat it is an ordinary positional parameter whose index has already moved once (tokenization_kwargs landed in 0.18.0). A TRL that fills it positionally plus an injector that adds it as a keyword is "got multiple values for argument", not a missing adapter. Bind the callable's real signature rather than counting arguments, so a future reshuffle cannot reintroduce it. A keyword lora_request = None is still overridden: on a shared-weights engine that means base-model rollouts, which is the bug this patch exists to fix. Three tests: a signature-accurate engine carrying vLLM 0.27.1's parameter lists on both entry points, the positional-chat case, and the explicit-None case. The positional one fails with TypeError without this change. 13 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 06:46:48 -07:00
alkinun	7bf8ade2db	Studio: enforce the research browser regression harnesses (#8736) ... * Studio: make research browser harnesses enforce their claims * Have the browser harnesses own their dev server for PR #8736 The new CI step backgrounds `npm run dev`, so $! is the npm wrapper and the EXIT trap leaves the vite node child alive holding the port and the step's stdout pipe. Reproduced: after the trap fires the port still answers 200. The ANSI smoke already solved this, so its start_vite / stop_process / drain_process_output move into _playwright_robust.py and all three harnesses use them. The CI step is two plain python3 calls now, and each file runs standalone the way its own docstring says. Readiness checks content, not status. Vite's SPA fallback answers 200 with index.html for anything missing, so a status check passes on a deleted smoke page. Confirmed locally: /smoke-DOESNOTEXIST.html returns 200. The report phase keeps a real, hit-tested click. A synthetic element.click() lands even with `body { pointer-events: none }` stranded, which is the freeze under test, so clicks_registered had stopped covering it. Measured both ways against a stranded layer: synthetic +1, real +0 and not actionable. The stall probe stays alongside it as main_thread_stall_ms, since nothing here reads an input timestamp, and its budget goes to 1000ms: 500 left only 1.2x against 342-416ms measured on a loaded host, and 1000 still fails ten times the report size (1518ms). Also: chat wall time and rAF count come from one page evaluation so they bracket the same interval; smoke-ansi-main.tsx joins the typechecked entries; the ANSI default port moves off the contended 8000; the job timeout goes to 20 minutes now that two browser smokes sit inside it; and the contract test pins the new verdicts plus the self-hosting rule. Verified: all three harnesses exit 0 standalone with no leftover vite and every port closed; typecheck 0; 2389 frontend tests; contract tests pass and fail when the new guards are removed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden the shared dev-server lifecycle for PR #8736 Simulated the failure modes of the self-hosting change in an isolated venv. Five were real; all five are fixed here, each with a test that fails when the fix is reverted. 1. A busy port was measured instead of refused. Under --strictPort our vite exits, and the readiness poll then talks to whoever else holds the port. Worst case is a squatter serving a page that happens to contain the entry string, which content matching alone accepts. start_vite now refuses an occupied port up front. 2. A dead server cost the full 120s timeout, three times per CI run. Readiness now takes the process and gives up the moment it exits, surfacing vite's own last lines. Measured: 120s to 0.0s. 3. SIGTERM orphaned the server. `finally` covers exceptions and SIGINT but not SIGTERM, which is what a CI cancel sends, so the exact leak this work is about survived a cancelled job. SIGTERM and SIGHUP now tear down registered servers, chaining to any previous handler, and atexit covers the rest. 4. Teardown could raise over the failure that called it. stop_process runs from a `finally`, and a child outliving SIGKILL turned a clean harness failure into a TimeoutExpired traceback with the real error lost. 5. An exported-but-empty SMOKE_BASE_URL counted as external, so no server started and the harness drove "" as its base URL. Empty now means unset. tests/studio/test_playwright_server_lifecycle.py drives both platform branches by injecting os.name, so the Windows path (CREATE_NEW_PROCESS_GROUP, taskkill /T then /T /F) is checked on every run rather than on someone's machine. It needs no browser and no npm, so CI runs it before the Chromium install. Verified: all three harnesses pass self-hosting and in the pre-existing SMOKE_BASE_URL mode, where an external server is correctly left running; ANSI smoke passes on chromium, firefox and webkit; no leftover processes and every port closed; 22 simulations and 23 in-repo tests pass; all four mutants of the fixes above are caught. * Run the lifecycle tests after the playwright install They import the harnesses, which import playwright, so the browserless step I put before the install could never have worked on a clean runner. Caught by staging CI, reproduced locally in a pytest-only venv (same 5 failures), and confirmed fixed in that same venv once playwright is present: 23 passed. pytest moves into the existing install line rather than getting its own pip call. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments added for PR #8736 Comment-only pass: collapses the duplicated SMOKE_BASE_URL notes, the teardown and readiness asides, and the click rationale in the contract test. Intent unchanged, 12 fewer lines. Pre-existing comments from #8633 are left alone. AST-verified comments only (comment_tools.py check: 4/4 OK), 56 tests pass. * Skip the harness-import tests when playwright is absent Cross-platform staging CI on macos-14 failed 5 of 23: the two tests that reload a harness module pull in playwright.sync_api, and that runner does not ship it. Several workflows run tests/studio without installing playwright, so the file broke collection there rather than only in Frontend CI. pytest.importorskip is the convention already used in this directory (test_cached_model_path_selection.py, test_pdf_qa_recipe_contract.py). The other 18 touch stdlib-only helpers and are unaffected. Verified in a pytest-only venv: 18 passed, 5 skipped, where it was 5 failed. With playwright present all 17 still run rather than skipping. * Honour SMOKE_BASE_URL in the ANSI smoke It started its own server unconditionally, so pointing it at an external one still spawned a second vite, and with the new occupied-port check it now raises before the external page is ever tested. Reproduced against a server on the default 5203: RuntimeError: 127.0.0.1:5203 is already serving. The other two harnesses already derive OWNS_SERVER from SMOKE_BASE_URL; this brings the third into line. Self-hosting is unchanged, since OWNS_SERVER is true there. Verified both ways: external mode exits 0 and leaves that server running; self-hosting still starts and stops its own. * Let the POSIX teardown tests run on Windows Cross-platform staging on windows-latest failed 3 of 23: AttributeError: <module 'os' (frozen)> has no attribute 'killpg' os.killpg is POSIX-only, so monkeypatch.setattr had no attribute to replace and raised during setup. The tests that exist to prove the Windows branch is covered were themselves the ones Windows could not run. raising = False lets them install the stub on either platform; the assertions are unchanged, and os.name is already injected so the POSIX branch is what they exercise. Verified with os.killpg deleted from the interpreter to mimic the runner: 17 passed, same as on POSIX. * Make the Windows paths actually work in the harness and its tests Two Windows-only breaks, both confirmed against the Python docs before fixing. signal.SIGKILL is Unix-only, so the tests that force os.name to posix to exercise the POSIX teardown could not name it on a Windows interpreter, and raised after the fake process timed out. raising=False on os.killpg was not enough. A posix_branch fixture now stubs both, and the assertion compares against the same portable constant. start_vite ran a bare npm. On Windows npm is the batch file npm.cmd, and CreateProcess cannot execute a .cmd with shell=False, so the self-hosting default this PR documents would have failed with FileNotFoundError before vite started. shutil.which honours PATHEXT and resolves npm.cmd there, and returns /usr/bin/npm here. Verified with os.killpg and signal.SIGKILL both deleted from the interpreter: 17 passed, same as POSIX. The ANSI smoke still runs end to end on Linux through shutil.which. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 06:44:58 -07:00
Long Yixing	ca7c83fafd	Studio: stop an unreachable Hugging Face endpoint from stalling the backend (#8799) ... * fix(studio): stop an unreachable hub from stalling the RAG embedder Naming the embedder's GGUF went straight to `list_repo_files`, which has no timeout parameter and whose pagination layer passes an explicit `timeout=None` that overrides any client-level default. `socket.create_connection` applies a connect timeout per address and races no address families, so on a host whose route to the hub blackholes that call blocks until the kernel exhausts its SYN retries — while `_lifecycle_lock` is held, so every reader queues behind it. Resolve from the local Hugging Face cache first. `_model_path` is per-process, so every restart previously paid a hub call to name a file already on disk; the cache answers in tens of milliseconds and never touches the network. Only the revision `refs/main` names is used, since that is what a download would serve, and a hit pins the embedder to it: `embedding_identity` does not record a revision, so silently adopting republished weights would leave a persisted index answering queries from one model with documents embedded by another. On a genuine miss the hub is still used, now under a wall-clock deadline and inside the same forced-offline-when-unreachable guard the chat GGUF path uses. If the hub cannot name a file, a cached GGUF of another variant is adopted with a warning rather than leaving the install with no embedder; a transfer that fails for its own reasons still surfaces. Fixes #8778 * fix(studio): bound the prebuilt release-freshness fetch `urlopen(timeout = 5.0)` looks capped, but `socket.create_connection` applies that timeout once per address as it walks the `getaddrinfo` results, and races no address families. A host whose leading addresses blackhole therefore pays five seconds for each one before reaching a working address, so the effective cost is the timeout multiplied by the address count rather than the timeout itself. `/api/inference/status` reads this fetch, so that multiplication becomes the route's response time. Failed lookups are only memoised for 60s, which is shorter than the stall itself, so back-to-back status reads each pay it again. Run the fetch under a wall-clock deadline. A missed deadline is a failure like any other: it feeds the existing failure cache and the last-good disk value, so the freshness banner fails open exactly as it already does offline. * fix(studio): keep cache-first embedder resolution faithful to the listing Review of the cache-first path surfaced four ways a local snapshot could answer differently from the full repo listing it stands in for. Companion-repository precedence is restored. A custom model resolves through its derived `-GGUF` companion first and reaches the model repo only when the companion has no GGUF; consulting the cache for both candidates let a file cached under the fallback pre-empt a companion the hub could still resolve, and then tagged it as current. The cache is now consulted for the preferred repo only, while the offline degrade still reaches both. Selection no longer depends on arrival order. A listing arrives ordered and a directory scan does not, so among equal-length names the tiebreak now falls back to the name itself; without it a complete cached shard set could yield shard 2, which lacks the metadata llama-server needs. Split sets are verified before use. A snapshot holding part of a set cannot serve it, so the winner is checked and, if unservable, dropped and the pick retried — failing the lookup outright would let an incomplete family shadow a complete one that merely sorts later. Verifying per winner rather than per file keeps this to a single sibling walk. MTP drafters join mmproj in the exclusion. A drafter is a companion rather than a model, and a cache subset holding only the companion was read as holding the embedder. Repo ids now resolve through `resolve_cached_repo_id_case`, so a repo id typed with different casing than its cache folder still finds its GGUF instead of re-downloading, or failing outright when the hub is unreachable.	2026-08-14 06:39:53 -07:00
Daniel Han	8bfc61c1f5	Studio: never start llama-server at native context on Metal (#8709) ... * Studio: never start llama-server at native context on Metal llama.cpp reads "-c 0" as fit_params_min_ctx = UINT32_MAX, which pins the model's full native context and disables the reduction --fit would otherwise do. load_model already knows this and says so in a comment. On Apple Silicon no GPU is enumerated, so the Apple unified-memory cap is the only thing holding the context down, and two paths reach the command builder with a zero context after that cap has been skipped or discarded. The cap is guarded on effective_ctx > 0, so a GGUF whose metadata carries no context length is never capped at all. The `except Exception` around GPU selection restores the original request, which is 0 when context is on Auto. That throws away a context the cap had already computed, and the handler logs "using --fit on" while emitting the one argument that turns --fit off. Either way llama-server starts at the model's native length on unified memory and dies in KV or compute allocation, which is the over-commit #5118 and #6529 describe. _metal_zero_ctx_floor returns the context to start at instead, the same 4096 the cap itself falls back to when it cannot estimate KV, or the model's own length when that is shorter. It is a separate helper rather than an inline condition because the existing coverage in this area is source-string inspection, and a decision worth making is worth testing directly. Left alone: Auto-layers, which omits -c entirely and lets --fit size the context, and manual offload, where the user owns memory management including the cap. Inert off Apple Silicon, since the budget is 0 there. The floor sits above the base cmd list rather than between it and the -c emission. test_llama_cpp_no_context_shift pins those two as adjacent, and it was right to complain when the first version of this separated them. Tests: 21 new. 125 in the three suites that own this behaviour, 2257 across the llama.cpp, memory-mode, context-fit, placement and vram suites. The five video failures are present on main unchanged. * Apply the repo's kwarg-spacing format to the new tests * Stop a pass-through "-c 0" from undoing the Metal context floor User extras are appended after Studio's own -c, and llama.cpp is last-wins ("only last value will be used"), so a pass-through "-c 0" outlived both the new floor and the existing Apple cap and re-pinned the model's native length. Verified against llama.cpp common/arg.cpp: the -c handler sets fit_params_min_ctx = UINT32_MAX on value 0, which disables the context reduction --fit would otherwise do, so "-c 0" is strictly worse than passing no -c at all when memory is tight. Drop a zero context override from the extras on Metal. Only zero: a positive -c stays honored exactly as before. This matches a decision load_model already makes, since explicit_ctx is requested_ctx > 0, so "-c 0" never counted as an honored override and the cap already ran for it; the trailing copy in the extras was silently undoing the cap's answer. Whatever context Studio computed is what survives, so a model whose native length is 2048 still launches at 2048 rather than a flat floor. Inert off Apple Silicon and on manual memory management, same scope as the floor. Also corrects the floor docstring: the cited reports surface as llama-server's "Compute error." at decode, not a start-time allocation death. Tests: argv-level coverage driving the real load_model for the floor path, the capped path, both -c spellings, a positive override, and the off-Metal no-op, plus unit tests for the new strip helper. * Keep the Metal floor under a context ceiling the cap already computed On the exception path max_available_ctx outlives the reset that discards effective_ctx, so the cap's KV-based answer can already be below 4096. The floor ignored it and floored back up, which re-created a smaller version of the same over-commit and then advertised 4096 as the ceiling through max_context_length, /v1/models and the frontend context slider. Take the ceiling into account when it exists. The no-metadata path never ran the cap and has none, so it still floors to 4096 as before. * Keep the requested extras whole, and advertise only what we launch Two problems with the zero-context strip, both flagged in review and both reproduced before touching anything. The strip rewrote what the user is recorded as having asked for. extra_args was overwritten in place, and the commit block stores both _extra_args and _requested_extra_args from it. _requested_extra_args is the comparator side that a later Apply is matched against, so an unchanged request compared unequal to itself: user asked for : ['--threads', '8', '-c', '0', '--mlock'] stored as requested : ['--threads', '8', '--mlock'] next identical Apply: ['--threads', '8', '-c', '0', '--mlock'] comparator equal? : False which reloads the model on every Apply. A pre-strip copy is now kept for the requested side; the stripped list is still what launches and what is stored as effective. The advertised ceiling outlived the cap that never ran. On the exception path max_available_ctx still holds the native length its initialiser put there, and nothing established that the length fits, but max() kept it: exception path: cap never ran floor=4096 published=262144 cap ran, KV ceiling 3000 floor=3000 published=3000 so the server ran at 4096 while max_context_length told the UI 262144 was the largest that fits, and its warning threshold came from the same number. The floor is already bounded by any real ceiling the cap did compute, so the ceiling is now replaced by the floor rather than maxed with it. Worth recording: the first probe for the second problem was misleading. _apple_metal_memory_budget_bytes returns 0 off Apple Silicon, so the floor came back 0 on this Linux box and the mismatch looked different from what it is. Re-run with the budget stubbed, the numbers above are the real ones. Tests: 8 new. 35 in the Metal context suite, 1816 across the llama.cpp, memory-mode, context-fit and placement suites. The four video failures are in the origin/main baseline. * Drop the pass-through zero context in Auto-layers too Auto-layers (manual memory mode with a negative layer count) is the one mode where the context is decided entirely by --fit: the managed command omits -c on purpose so llama-server can size it. User extras are appended after that and llama.cpp is last-wins, so a pass-through -c 0 survives and sets fit_params_min_ctx = UINT32_MAX, which is exactly the flag that stops --fit reducing the context. On Metal that pins the native length and re-creates the allocation failure this branch exists to prevent. Drop the zero override there as well. A fixed manual layer count still keeps it, since there the user owns the budget, and everything off Apple Silicon is unchanged. Also replaces the source-string assertion on the pre-strip extras copy with a behavioural one: it read the exact formatting of an assignment and broke when the formatter joined it onto one line. * Judge the Metal context guards on the request, not the paravirtual pin On a virtualised Apple GPU the CPU pin rewrites every placement, a plain Auto one included, to manual with 0 layers, in the locals and in the intent, and it does so above the two Metal context guards. Both guards read gpu_memory_mode there, so the default Auto request on a virtualised Mac looked like a caller-owned manual configuration: the floor stayed inert and a pass-through -c 0 was kept, which is the native-context over-commit this branch exists to prevent. Off that hardware the same request emits -c 4096. Read the ownership question once, before the pin, and pass it to both guards. A manual mode with a fixed layer count is still the caller's, on a virtualised device as anywhere else. Also fixes the comparator on that path. The drafter drop snapshots the extras for the duplicate-load comparison, and that snapshot is taken after the zero-context strip and was preferred over the pre-strip copy, so an identical repeat Apply compared unequal and reloaded the model every time. Prefer the pre-strip copy; the snapshot is only ever set alongside it, and the spec strip it guards runs later still. * Drop an inherited zero context where no -c is emitted llama.cpp resolves LLAMA_ARG_CTX_SIZE through the same handler as -c, in a loop that runs before argv is parsed, so a 0 there sets fit_params_min_ctx = UINT32_MAX exactly as the flag does. Everywhere Studio emits a -c the command line wins and the variable is harmless, but Auto-layers deliberately emits none so --fit can size the context, and there an inherited zero cancels the fit the mode is entirely sized by. Drop it in that case, in the same shape as the LLAMA_ARG_THREADS scrub alongside it. Only a zero, and only under the conditions that already drop a pass-through zero, so a positive inherited context stays the legitimate way to size an Auto-layers launch and nothing changes off Apple Silicon or on a caller-owned budget. * Tighten the comments on the Metal zero-context guards --------- Co-authored-by: danielhanchen <unslothai@gmail.com>	2026-08-14 06:38:25 -07:00
Long Yixing	b69bfe0216	fix(studio): keep a slow install alive and name what it is downloading (#8805) ... A desktop install was killed after two hours of wall clock regardless of progress. On a slow link the cu126 torch wheel takes longer than that on its own, so the install could never finish, and each kill wasted every byte already fetched because uv restarts an interrupted download from zero. The installer also keeps uv's output in a log it only prints on failure, so those two hours looked identical to a hang, both to the user watching and to anyone reading the logs afterwards. install.sh and install.ps1 now turn uv's announcements for downloads of at least 50 MiB into `[TAURI:DL]` / `[TAURI:DL_DONE]` protocol lines, while the per-package chatter stays in the log where it was: a full dependency set is dozens of announcements plus a line per installed package, which would bury the installer's own output. The app consumes the markers without displaying them, so a healthy install looks exactly as it did before. Once five minutes pass with nothing printed it reports the step, the package being fetched, its size and elapsed time, and it now gives up only at a twelve-hour backstop. Markers travel on stderr from install.sh, alongside the other protocol lines that function already writes there. That also keeps them clear of the verbose path's redactor, whose sed block-buffers its output: a marker queued behind it would reach the app only once the download it announces had finished. install.ps1 writes them on stdout, so both reader threads filter them out of the UI. Where awk is absent, which this script already supports elsewhere, the filter degrades to plain capture rather than closing the pipeline under the child. There is deliberately no rule that stops an install for being quiet. That needs evidence that work is happening, and the markers cannot supply it everywhere: the uv calls under `studio setup` capture their output rather than streaming it, so a silence rule would stop healthy installs in exactly the phase this change exists to protect. Extending the markers to those wrappers is follow-up work. Validated by driving both installers' real command wrappers against recorded `uv pip install` output -- `run_install_cmd` under sh, and `Invoke-InstallCommand` under PowerShell across both its quiet and verbose arms -- plus a timing check that a marker is readable while its download is still running, and the existing installer and watchdog suites. Both installers are pinned to the same 50 MiB threshold, so they cannot drift apart. Refs #8698	2026-08-14 06:36:44 -07:00
Nilay	731ea4cd73	Studio: save a chat or reply to project sources (#8588) ... * Studio: save a chat or reply to project sources * Studio: invalidate the sources probe before and after chat saves * Studio: resolve the reply's project from its thread before saving * Studio: refresh an open Sources panel when a source is saved elsewhere * Studio: refresh the sources panel without resurrecting a deleted row invalidateProjectSources dispatched the refresh event, and the panel calls it before each of its own mutations as well as after. handleRemove drops the row optimistically and then invalidates, so the listener refetched while the DELETE had not been sent yet and the row came straight back; useRagDocuments has no request sequencing, so the staler of the two answers can also land last. Split the two jobs. invalidateProjectSources stays a cache drop. announceProjectSourcesUpdated drops the cache and tells a mounted list to refetch, and only the save path calls it, after the upload. The panel's subscription moves into subscribeProjectSourcesUpdated so the projectId filter and the teardown are testable without a DOM. * Studio: name and report a chat saved to project sources Five things a save got wrong once it left the upload call. The filename. The backend stores the bytes under a uuid and re-sanitises the name for its own metadata, so this was never a path safety problem, but it is what the sources panel lists. Break the Windows device names, which stay reserved through an extension, drop control characters and trailing dots and spaces, clamp to a byte budget on a code point boundary so a surrogate pair is never split, and fall back to chat.md where the backend's own pass would leave just "_.md". Success. The toast fired on upload acceptance; indexing is a job, and the panel hides a document whose job failed, so a failed ingest was silent behind "Saved to project sources." Poll the job and raise the same "Couldn't index X" the panel's own upload path raises. Errors. The upload failure was caught and toasted but never returned, so the callers could not tell a save that worked from one that did not, and a pair save could not stop or report a partial failure. Pair saves. Both halves carried the row's single title, so one click produced two toasts and two sources with the same name. Name each half after the model that answered it, and report the count once. Wording. A reply saved under the chat's own title collided with the whole chat saved under it; mark the reply apart. And an empty conversation reported "No messages in this conversation to export." from something that is not an export. * Studio: cover saving a chat or reply to project sources The reply's destination is the one that matters: it comes from the thread being read, not from the project the sidebar has selected, or a save uploads one project's chat into another project's sources. thread.tsx is TSX node cannot load, so that test reads the source the way rag-availability-marker.test.ts does for the same file; swapping the argument back to activeProjectId fails it. The rest covers the filename rules, the pair naming, and the save itself: one toast per click, a failure that says why, a failed ingest that is not left silent, and a mounted sources list that actually refetches rather than merely being sent an event. The toast stub records instead of swallowing, and the authFetch stub can be given a handler for the one test that needs a response, restored per test so nothing leaks into the suites that expect no network at all. * Studio: tell a compare pair's halves apart when they share a model label The LoRA compare runs one checkpoint with the adapter off and on, so both threads record the same modelId and the two saved sources came out with byte-identical titles. The sources panel renders only the filename, and the backend keys uploads by uuid, so the pair arrived as two indistinguishable rows. Name a colliding half by its modelType, or by position when that is absent; two different models keep the plain name. * Studio: save a reply whole, and name compare halves by pane not arrival getCopyText joins text parts alone, so saving a reply dropped its reasoning, tool calls and citations, and a reply that was only a tool call reported as having no content. Route it through the same conversion the whole-chat save uses. listStoredChatThreads sorts by updatedAt, so the positional fallback numbered the halves by whichever answered last: saving one pair twice could swap the two names. modelType already carries the pane, so name from that and keep position as a last resort only. --------- Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-08-14 06:35:04 -07:00
Maheswar Kumar	6c5385a11f	studio: attach youtube transcripts from pasted chat links (#8648) ... * studio: attach youtube transcripts from pasted chat links Pasting a YouTube link in the chat composer now offers to attach the video's transcript as a text file, so the model reads what was said instead of a bare URL. Part of #8526. The offer renders above the composer surface with the same treatment as the prompt-queue stack, and retracts on its own once the link leaves the draft. Accepting builds a `<title> transcript.txt` File and hands it to `aui.composer().addAttachment`, the same entry point as paste, drag-drop and the file picker, so `TextAttachmentAdapter` handles it exactly like any other text attachment. The file carries the title, channel, duration, URL and caption language ahead of the transcript. Backend, in `core/youtube_transcript.py` behind `POST /api/youtube/transcript`: `extract_video_id` accepts /watch, youtu.be, /shorts, /embed, /live and /v on the youtube.com, youtu.be and youtube-nocookie.com hosts, and rejects everything else so a path segment cannot stand in for the host. `fetch_transcript` calls `youtubei/v1/player` with the ANDROID client, picks a track, and downloads it as fmt=json3. The ANDROID client is the reason this works without a third-party package. Caption URLs taken from the watch page's `ytInitialPlayerResponse` belong to the WEB client, and YouTube answers those with 200 and an empty body unless the request carries a proof-of-origin token that only its BotGuard JS can mint. The ANDROID client's URLs still resolve unsigned. Track selection prefers the requested languages, human-written captions over auto-generated ones, and falls back to the track paired with the video's default audio track rather than index 0, which on a multi-language video is usually an unrelated translation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: fix youtube transcript locale pick and stale fetch _select_track scanned tracks once per requested language and accepted a base-language match on the way, so a pt-BR request took the pt-PT track when that one was listed first. Each language now tries an exact locale match before falling back to the base language, still preferring human-written captions over auto-generated ones. The composer prompt is now keyed by URL. Pasting a second link while the first transcript was still loading reused the component instance, so the unmount cleanup never aborted the first request; it completed, attached the wrong video's transcript, and dismissed the offer for the link now in the draft. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden the YouTube transcript fetch for PR #8648 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-08-14 06:34:36 -07:00
Long Yixing	11f59ebb30	Studio: fix GGUF image-capability detection for API auto-switch and audio-only projectors (#8807) ... * fix(studio): read the named quant in the /v1 image capability check An image or audio request that auto-switches models is refused unless the target GGUF has a companion mmproj. The check received the resolver's load path but not the quant resolved with it, so for a directory it fell back to the root-level detector. A repo that files every quant under a per-quant subdirectory — which is what a snapshot holds after downloading one such quant — has no weight file at its root, so the check found none, classified the snapshot through the Transformers config reader instead, and answered "not vision". The request then failed with "The requested model does not support the image or audio input in this request", while loading the same model from Studio served images fine, because a resident model skips the check. Resolve the weight file the way the loader does: a named quant on a directory goes through the variant lookup, everything else through the root-level detector. A caller that names no quant keeps the previous path. Fixes #8772 * fix(studio): do not report an audio-only GGUF as a vision model An mmproj is attached for audio input too (ultravox, Voxtral, Qwen3-ASR), so taking a projector's presence as proof of image input advertises an image button those models cannot honour: the picker and cached-model rows show a vision badge, the pre-swap capability probe admits an image request, and the image reaches llama-server as an opaque failure instead of a typed refusal. Read the projector's own `clip.has_vision_encoder` / `clip.has_audio_encoder` instead. A projector accepts images unless it claims audio and not vision, so a file declaring neither -- an older convert, or one this reader cannot open -- keeps today's answer, and a dual-modality projector (Qwen2.5-Omni declares both) stays image-capable. The capability probe, the loaded backend's `is_vision`, and the local GGUF listing that feeds the badges all resolve through the same predicate, so they cannot drift apart. What the loader attaches is unchanged: `--mmproj` is still passed for an audio-only model, since that is how its audio input works, and the launch decides attachment from local variables rather than from the `is_vision` property. `utils/models/model_config.list_local_gguf_variants` keeps flagging any projector, because its flag is spent on the VRAM guard's size estimate rather than on capability, and an audio projector's bytes are resident like any other. Validated on real projector files: ultravox-v0.5, Voxtral-Mini and Qwen3-ASR flip to not-vision, while gemma-3-4b, Qwen2.5-VL-7B, Qwen3-VL-8B, MiniCPM-o-2.6 and Qwen2.5-Omni keep vision.	2026-08-14 06:32:48 -07:00
Daniel Han	482abfabfa	Give Unsloth Studio its first CUDA coverage, on a Kaggle T4 (#8489) ... * Run deterministic notebook smoke tests on real Kaggle T4s The notebooks are written for a T4: sm_75, fp16 with a gradient scaler, xformers rather than flash-attention, 16GB. Nothing in CI runs on that card, so a regression that only appears there reaches users first. Adds a gated job that launches one Kaggle GPU session and runs the smoke payload once per T4. The account has a 60h weekly budget shared with other consumers, so the job is a deliberate minority consumer of it: paths filters (only 9.9% of a week's 444 main commits touch unsloth/), a ~10% sampling draw keyed on the run id so a re-run cannot reroll, a quota floor read live from Kaggle's own API, and a stand-down when the account already has a kernel in flight. Expected cost is 5-7 GPU-hours a week. Only a payload that ran on a T4 and disagreed with its assertions turns the check red. Throttling, the 2-kernel concurrency cap, a dead session or a download that would not complete are all warnings, because a free external service blocking merges would be ignored within a week. The payload asserts three things, worth different amounts. Two fresh processes must agree bitwise on per-step loss and grad_norm; that one is exact, and it is fresh processes because two in-process cycles disagreed from the first step while two processes agreed on all ten. Greedy decoding must emit the canary string exactly, which catches trained weights never reaching the generate call. Metrics must sit inside a band around a committed reference, which is a tolerance and never an equality, since fp16 reduction order moves with the driver and the libraries. Ten steps rather than three: under fp16 the scaler skips the first two overflowing steps, so a three-step run lands one real update and the canary never forms. The dataset is supplied as prompt/completion columns so the loss falls only on the answer. * Make the payload cells the T4 rejected actually run there Two launches on real Kaggle T4s died in the scaffolding before a training step ran, both of them in generated code that nothing had ever parsed. The dependency probe imported unsloth_zoo before unsloth. Zoo's __init__ ends with a find_spec("unsloth") guard and raised "Please install Unsloth via pip install unsloth" on a session where unsloth was installed and imported cleanly one entry later, so the payload aborted over a dependency that was not missing. Probe unsloth first, and invalidate the import caches after the pip installs that this interpreter did not see at startup. The reference argument was generated as a shell fragment and spliced into the middle of a Python list literal, with doubled braces that left ROOT unsubstituted. The cell was a SyntaxError, and the workflow passes --reference on every path, so no payload could ever have reported. Verified locally by executing the regenerated run cell end to end: two fresh processes, bitwise identical metrics, canary exact. * Bound the in-flight kernel survey by session length, not kernel count The concurrency check status-checked the twelve most recently run kernels, which is a sample rather than a search: a kernel that started three hours ago and is still running is missed the moment twelve newer ones have since run, and the push then fails at the account capacity cap and is reported as infra. Kaggle kills a notebook session at twelve hours, and the listing is sorted by last run time, which for an unfinished kernel is when it started (measured: a kernel pushed at 10:05:19Z lists as 10:05:19.297). So walking until the first entry older than that ceiling covers every kernel that can still be in flight, and stopping there is exhaustive rather than arbitrary. An hour of slack is added for clock skew. Two silent ways to read an unanswerable question as a green light are closed with it. Hitting the page cap now marks the survey incomplete, and a survey where no status could be read at all is unknown rather than idle. Both stand down, which is the cheap direction: the next commit draws again in minutes. On the live account this now costs three status calls instead of twelve. * Stop the reference band check from passing NaN unconditionally The band comparison did its NaN handling by accident, and it got it wrong. Under fp16 the gradient scaler logs a NaN grad_norm on every step it skips, so a committed T4 reference genuinely contains NaN. Left to the arithmetic, abs(x - NaN) is NaN and NaN > tolerance is False, so those steps passed whatever they held, including the case the check exists for: a step that used to overflow and no longer does. Compare NaN to NaN as equal and NaN against a number as a deviation, and treat a field logged on one side only as a deviation too, since no tolerance covers a change in shape. The mapping from an out-of-band verdict to the failure list moves into reference_failures() so the path that turns the job red can be tested without a GPU. The references README now says plainly that no reference is committed, that one must not be borrowed from other hardware, and how to lift one out of a green run's evidence rather than spending a session on it. It also records a measurement: the absolute floor never engages on this trajectory. The smallest value on the ten-step curve is a loss of 0.1428 against a floor of 0.05, so the floor changes no verdict today and is kept for a configuration that does go lower. * Name the cause when a kernel finishes and reports nothing A kernel that ends COMPLETE with no payload report is the hardest outcome to read: the summary has no metrics to show and the reason sits in an artifact nobody downloads. It is also the outcome both real T4 failures took. Kaggle returns the kernel log as a JSON array of stream records rather than as text, so the interesting line arrives split across dozens of them and reading the file directly shows a wall of JSON. Flatten it, filter to the driver and payload sentinels plus tracebacks, and fold the tail into the job summary. Checked against the evidence of a real failed run: the SyntaxError that killed it is now visible without leaving the summary. * Cover the four T4 CI fixes with CPU-only tests The generated cells are now compiled on both the with-reference and without-reference paths, which is what would have caught a SyntaxError that instead cost a Kaggle session, and the dependency probe's module order is asserted. The in-flight survey gets a fake Kaggle client and the case that motivated it: one kernel still running behind forty newer finished ones, which a count-bounded scan misses and a time-bounded scan cannot. Also the window boundary, timezone-aware and missing timestamps, the page cap, and the difference between some statuses unreadable and all of them. The band check is now proved to fail rather than assumed to work: a perturbation half a band-width past the limit, a moved scaler skip pattern in both directions, a field that stopped being logged, and a length mismatch, each asserted through to the failure list. Three tests perturb the committed reference itself and skip, saying so, until one exists. 41 passed, 3 skipped. * Parse every generated cell on every build path before spending a session Two of the three Kaggle sessions spent so far died on generated code that nothing had ever parsed. The compile check now builds all three paths the generator has, using the workflow's own argument list verbatim, including the one where a committed reference exists and is carried inline as a fourth file: that path becomes live the moment a green run supplies the file, and would otherwise first be exercised on Kaggle. Parsing alone is not enough. A template hole that substitutes to a bare identifier parses and then NameErrors, which costs the same session, so every cell is also checked in execution order for names nothing before it defines. Both checks were confirmed to go red: the first on the exact shell-fragment-in-a-list-literal defect that cost session three, the second on a run cell reading a name no cell binds. The carried sources are also decoded the way the kernel will decode them and compared byte for byte with the repo, so the T4 cannot quietly run something other than what is committed. * Commit the T4 reference the first green Kaggle run produced Kernel danielhanchen/unsloth-t4-ci-e3c6661f, terminal state COMPLETE, both payloads passing on their own Tesla T4 of one session. Two fresh processes agreed bitwise on all ten steps on both cards, max_abs_diff exactly 0.0 for loss and for grad_norm, and the two cards independently produced the same ten values as each other, so four processes agree rather than two. All four cycles emitted the canary exactly. The scaler skipped steps 1 to 3 every time, which is the behaviour the ten-step length exists to accommodate. The file is reports[0] of that run, copied whole by the recipe the README already documented, and nothing in it is from other hardware. One assumption the run contradicted, now recorded rather than repeated: the session had no xformers. unsloth installs --no-deps, unsloth_zoo does not carry xformers and the Kaggle image does not either, so the banner read Xformers = None and this trace is the fallback attention path. Adding xformers to the install would move these numbers and require a recapture. The floor's smallest observed value moves from 0.1428 to 0.0871, still above the 0.05 denominator floor, so the floor stays inert and the test that re-derives that keeps passing. The three tests that perturb the committed reference now run instead of skipping. 45 passed, 0 skipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the in-flight tolerance the Kaggle gate stands down on The gate declines to launch whenever any kernel of the account is in flight, which is stricter than Kaggle's 2-kernel cap requires. That is the intended policy, so it is now a named constant with the tradeoff written next to it rather than an implicit property of the code, and a --allow-in-flight argument that cannot be raised to the cap itself. Default behaviour is unchanged. Also files the workflow under 'never' in ci-preempt.json: cancelling a run does not stop the Kaggle kernel it already pushed, and the orphan then bills quota to its own ceiling with nobody watching. * Run three steps, and refuse a reference captured at another count max_steps drops from 10 to 3. Measured consequence: under fp16 the gradient scaler starts at 65536 and skips every step it overflows on, and the committed reference shows steps 1, 2 and 3 all skipped, so a 3-step run of the old configuration applied zero optimizer updates. --init-loss-scale pins the scaler below the overflow point so those three steps are real updates, and optimisation_failures() fails a run whose every step was skipped rather than letting it report as training. The committed reference was captured at 10 steps and no longer describes the run. check_reference now reads the max_steps the reference records and refuses to compare across counts, as a failure with both counts named -- including when the reference does not say what it was captured at. A quiet pass there would be a green check comparing a run against a curve it has nothing to do with. The reference is therefore stale until one recapture run. The workflow gains skip_reference_band for exactly that dispatch, and the recipe is in references/README.md. Also brings the workflow onto the repo's conventions: pinned action SHAs matching the rest of CI, a workflow-level concurrency group that deliberately does not cancel in progress, a fork guard on the gate job, typed dispatch inputs and an artifact name without the run id. * Keep the run at ten steps, and say why three does not work The committed reference is the evidence: the fp16 scaler reports grad_norm NaN at steps 1, 2 and 3 and a finite value from step 4. It starts at 65536, halves on each overflow, and skips the step it overflowed on, so a 3-step run applies zero optimizer updates. Loss stays around 10, the canary never forms, and the band would compare three points of a curve that never moved. Shortening also saves nothing worth having: a launch costs about 0.08h and that is almost all pip install, not training. The step-count guard, the all-steps-skipped guard and --init-loss-scale all stay, since they are what makes a shorter run possible later and what stops a vacuous one reporting green. The pin is off by default so the committed reference keeps applying. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Turn one T4 payload into four, and pair a pinned control with a canary The workflow ran a single tiny SFT payload on both cards of one Kaggle session and spent about 1.4 GPU-h a week doing it. The budget is now 40, and the question asked of it is wider: catch regressions in trl, transformers, accelerate, peft, bitsandbytes, torch and vllm, and catch torch.compile falling over on sm_75. Four legs now, two kernels of two T4s each, described once in .github/scripts/kaggle_t4_ci/legs.py rather than restated in YAML: control the existing tiny Qwen2.5-0.5B determinism run, pinned canary the same run on the newest library set Unsloth's own constraints allow gptoss gpt-oss-20b LoRA, for torch.compile and the forced float32 path grpo Qwen3-4B GRPO through a vLLM engine on the same card control and canary are one instrument rather than two tests. Same payload, same seed, same data, same step count, on the two cards of the same session; the only difference between them is the installed versions. A red canary beside a green control is a library release, and the job summary already puts the two version sets side by side so the bisect is a diff rather than an investigation. The reference band applies to the control alone: two library sets do not produce one fp16 trajectory, so the canary asserts what does not depend on versions instead. The pins are evidence rather than preference. transformers and trl are the pair the committed reference was captured with; peft, accelerate and bitsandbytes come from probe kernels 8161ceb9 and 7ab727f1, two independent T4 sessions that ran the identical base install and agreed. torch stays unpinned and recorded, because it comes from the Kaggle image and replacing it is the failure mode the grpo leg spent two probes on. The gate now separates this workflow's own kernels from everybody else's. It takes both of Kaggle's concurrency slots, and only when the account is otherwise idle: one kernel belonging to anyone else still stands it down entirely, which is the same policy as before stated at the right grain. * Leave the vLLM leg unwired, and stop trusting Kaggle's own kernel timeout Three probe sessions tried to run the Qwen3-4B GRPO leg on a T4 and none of them reached a training step. None of them failed for a reason to do with sm_75, memory or GRPO either: vllm==0.11.2 pins torch==2.9.0, so the leg has to replace the Kaggle image's torch, and that is what killed all three. With the image's site-packages visible, pip treats torch's pinned NVIDIA runtime packages as already satisfied by the copies belonging to 2.10, and the result is a torch that installs cleanly and cannot be imported -- libcusparseLt.so.0 first, then ncclCommWindowRegister once that one was named explicitly, identically on vllm 0.11.2 and 0.15.1. An isolated venv resolving the whole stack got further and then the session wedged. So the leg keeps its payload, its assertions and its install, and moves to legs.UNWIRED with the three kernel slugs and what each one measured. Wiring it now would make the check permanently red and spend the budget doing it. The second kernel keeps its spare T4, which costs nothing, and that is where the leg goes when the install works. The third probe also cost about an hour of quota teaching us that the push-time kernel timeout is not a budget control. Pushed with -t 5400, its own nbconvert crashed at t=406s and the session sat in RUNNING for over two hours -- past that ceiling and past the launcher's own deadline, because one status call had blocked with no socket timeout to bound it. Deleting the kernel by hand took the account's used-hours figure back down, which is the measurement that settles which control is real. The launcher now sets a socket timeout and deletes every kernel it pushed on every path out, and the workflow header says which of the three bounds is load-bearing. Sampling recomputed on the payload that actually ships: 40%, from measured per-kernel cost rather than the old estimate, with the note that wiring a fourth leg means recomputing it rather than editing one line. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-solve the GRPO leg's install so it stops replacing the image's torch Three probe sessions never reached a training step, and all three died the same way: vllm==0.11.2 pins torch==2.9.0, so the leg had to replace Kaggle's torch 2.10.0+cu128, and pip treats the image's NVIDIA runtime packages -- which belong to 2.10 -- as satisfying the new torch's pins. That imports as libcusparseLt.so.0 missing, then as libtorch_cuda.so undefined symbol ncclCommWindowRegister one package along, then as an isolated venv that spent an hour of quota resolving a CUDA stack and never produced payload output. None of that is a question about sm_75, memory or GRPO. It is one line: no vLLM release in the 0.11-0.16 range pins the torch the image ships. Releases 0.17.0 through 0.19.1 pin torch==2.10.0 exactly, so pin 0.19.1 and the leg replaces nothing, keeps system_site_packages, and installs like any other. Drop xformers with it. Its vLLM attention backend was deleted in 0.12.0, so it would be a package nothing selects. sm_75 has no FlashAttention and no FlashInfer, and the ladder in vllm/platforms/cuda.py falls through both to TRITON_ATTN; the leg names that in VLLM_ATTENTION_BACKEND so a release that reorders it goes red here rather than quietly selecting something else. 7.5 is still in CUDA_SUPPORTED_ARCHS at v0.19.1 and fp16 is a supported dtype below capability 8.0. Still unwired. Two things are static evidence rather than a run: whether 0.19.1 starts on sm_75 at all, and whether 8GB of 16-bit weights plus an engine plus a LoRA trainer fit in 14.56GB. UNWIRED says so in those words, and a test asserts the note still names the pin and still says what is unknown, so nobody wires this off prose that outlived its measurement. Also ports the vacuous-test fix from the stacked Studio branch: pre-commit.ci reformatting default=10 to default = 10 stopped the payload regex matching, so the step-count agreement check was comparing nothing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop the GRPO leg JIT-compiling flashinfer on a machine that cannot link it First probe on the re-solved install, kernel unsloth-t4-ci-e2d9ce9b. The install works: torch stayed at the image's 2.10.0+cu128, vllm 0.19.1 installed and imported, xformers was absent as intended, and the payload ran on a real Tesla T4 reporting capability 7.5 with TRITON_ATTN accepted. It reached engine construction, which is further than any earlier probe got by the whole width of the install. It died in flashinfer 0.6.6's JIT, and not for an sm_75 reason. All three .cu files compiled cleanly for -gencode=arch=compute_75,code=sm_75. The LINK failed: /usr/bin/ld: cannot find -lcuda -L/usr/local/cuda/lib64/stubs is already on the command line, so the image has no driver stub libcuda.so, only the runtime libcuda.so.1. Nothing about that is fixable from a payload. Set VLLM_USE_FLASHINFER_SAMPLER=0 so nothing JITs. The sampler has a native path, and skipping the build also saves a four-file nvcc compile inside a session billed by wall clock. UNWIRED records what the probe measured and narrows what is left to one question: whether the engine builds now. * Make -lcuda resolvable so flashinfer can link what it compiled Second probe, kernel unsloth-t4-ci-916d5986, failed identically to the first with VLLM_USE_FLASHINFER_SAMPLER=0 set. That is the informative part: the JIT is not reached only through the sampler, so switching off one consumer is whack-a-mole. The failure was never sm_75. Both sessions compiled all three .cu files cleanly for -gencode=arch=compute_75,code=sm_75 and died at the link: /usr/bin/ld: cannot find -lcuda -L/usr/local/cuda/lib64/stubs is already on the command line; the image ships no libcuda.so at all, only the versioned libcuda.so.1, and the linker will not resolve -lcuda against a soname. Normally the toolkit's driver stub fills that gap; this image has the directory and not the file. So symlink libcuda.so at the real driver in a scratch dir and put it on LIBRARY_PATH, which gcc and ld search for -l. No root, nothing written into /usr/local, and it fixes every flashinfer op rather than one. Linking against the real driver instead of a stub is correct here: the driver is present, which is the whole reason a stub would have substituted for it. Also widen the captured traceback to head AND tail. The last probe's 6000-char tail was entirely ninja's own output, so the Python frames naming the caller were exactly what got dropped. Probe 2 also measured the fit question the notebook raised: peak reserved 7.96GB of 14.56 with load_in_4bit false, so 16-bit weights are not what is short of room here. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only count a libcuda the linker will actually search for Kernel unsloth-t4-ci-d0d480b6: the shim reported already_linkable and did nothing, and the link failed anyway. It had found libcuda.so in /usr/local/cuda/compat -- which is real, and which nothing passes with -L. The failing ninja line names /usr/local/cuda/lib64 and /usr/local/cuda/lib64/stubs, and those are the only two directories that can answer -lcuda. So the precondition now checks exactly those two, and compat becomes what it is actually good for: a symlink TARGET when ldconfig and ctypes both come up empty. A library the linker will not search for is not a library the linker can find. Two tests: one reproduces the compat-only machine and asserts the shim is built anyway, one pins the searched directory list, since widening it is precisely how the check went wrong. * Give the base model a chat template, and record what the link shim did The link shim works. Kernel unsloth-t4-ci-27b0dc2e reported libcuda_shim: needed true, applied true, searched [/usr/local/cuda/lib64, /usr/local/cuda/lib64/stubs], real /usr/local/cuda/compat/libcuda.so and flashinfer is gone from the failure entirely. Memory went from 7.96GB to 11.36GB of 14.56, which is the vLLM engine actually loading, and the run reached trainer.train() -> _run_epoch -> training_step. Both questions legs.UNWIRED was holding this leg for -- does vLLM start on sm_75, does it fit in 14.56GB -- are answered by that. It then failed on the payload's own gap: ValueError: Cannot use chat template functions because tokenizer.chat_template is not set unsloth/Qwen3-4B-Base is a base model and ships none. The notebook this leg comes from solves it with an SFT priming stage that installs a template before GRPO starts; this leg has no priming stage, so it sets a minimal ChatML template directly and records in the report which of the two worlds the run was in. The base model stays. GRPO on an instruct model would measure the instruct tuning as much as the run, and the payload's rewards are format-and-digit rewards a base model can move inside three steps. A test renders the template rather than matching it as a string: a template that does not render would trade a failure at step 1 for a failure at step 1 with a longer traceback. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Wire the GRPO leg: it runs on a T4 and the rewards move Kernel unsloth-t4-ci-53efcc4e passed on a real Tesla T4. Step 2 logged reward_std 0.707 and grad_norm 0.772, peak 13.60GB of 14.56GB, three steps in 192s, and fast_generate returned coherent text. reward_std is the criterion rather than loss, because TRL's GRPO loss is ~0 by construction at num_iterations=1 and beta=0 and would pass on a run that learned nothing. The config in args is the one that fit, not the notebook's. Two probes with seq 2048, 4 generations, rank 32 and utilization 0.9 died in the backward at unsloth_zoo/gradient_checkpointing.py:1013, peaking at 15.97GB in 16-bit and 19.25GB in 4-bit. 4-bit alone is not the lever: it peaked HIGHER, because quantizing weights does nothing for activations while utilization 0.9 still hands vLLM ~13GB up front, and UNSLOTH_VLLM_STANDBY returns the weights during training but not the KV cache reservation. An earlier probe reported this as 'CUDA error: an illegal memory access was encountered' from empty_cache() inside vLLM standby. That was the async shadow of the same OOM; CUDA_LAUNCH_BLOCKING=1 produced the honest error. UNWIRED is now empty. The test that read its grpo entry is replaced by the invariants that outlive any one leg: every leg is wired or explained, nothing is both, and an unwired note says what is open. Two new tests pin the config and the vLLM pin, so restoring either to a value that OOMs fails here rather than on Kaggle. * Add a CUDA_LAUNCH_BLOCKING switch to the GRPO payload An illegal memory access is reported at whatever CUDA call synchronises next, which on the first T4 GRPO run was empty_cache() inside vLLM standby, nowhere near the kernel that faulted. Re-running with this flag produced the honest error instead: torch.OutOfMemoryError in the backward. That is what turned an unexplained CUDA fault into a memory budget, and the budget is now the leg's config. The environment variables are set before the first import torch in the process rather than merely before train(), because they have to precede CUDA context creation. It serialises every kernel launch, so it stays a switch and never a default, and it is recorded in the report because it changes what a timing means. * Give Studio its first CUDA coverage, on a Kaggle T4 Every Studio workflow in this repo runs on ubuntu-latest, macos-15 or windows-latest. macOS gives Metal and MLX some hardware; the CUDA path is exercised by nothing at all, and studio-inference-smoke.yml uses a 270M GGUF precisely because it has to decode on a CPU. tests/kaggle/studio_gpu installs Studio the supported way on a Kaggle T4, starts it headless, and asserts three things a CPU runner cannot: that a GGUF is actually offloaded to the GPU rather than falling back to CPU and returning text anyway, that a LoRA training run completes and leaves an adapter on disk, and that a GGUF export against a CUDA llama.cpp produces a file that loads. The repo's existing playwright_chat_ui.py then drives the browser path, last, because it stops the server on its way out. The gate and the launcher are the notebook leg's, reused unchanged: the payload prints the same report line, so the transport needed no edit. The two legs share a concurrency group because Kaggle's 2-kernel cap is per account and is not workflow-aware. Sampled at 4 percent of a filter measured at ~760 eligible invocations a week, which is ~30 launches and ~23 GPU-h against the notebook leg's 15-20 and 1.4. The rate is lower and the spend is fifteen times higher because the eligible stream is five times larger and a launch costs nine times as much; the arithmetic is in the workflow header. * Budget the two Kaggle GPU legs against one account, not two The Studio leg's budget block was solved against the notebook leg's numbers from before that leg was re-tuned: 0.08 h per launch, 10 percent, ~1.4 GPU-h/week. The notebook leg now costs 0.25 h at 40 percent for up to 23.1 GPU-h in a busy week, and Studio was sized at ~23 of its own. Two legs at 23 each is 46 of the 50 hours CI is allowed on a 60 h account, with the entire margin spent on the cheaper one, and the two reserve floors were picked independently. Re-solve both against one shared allowance: Studio 35 h, notebooks 15 h. Studio goes to 5 percent (~38 launches, ~28 GPU-h) and notebooks to 15 percent (13-35 launches, 3.3-8.7 GPU-h). The reserves now encode priority rather than each leg's own ceiling: notebooks stand down once the account is 35 h spent, Studio runs on to 50. The percentage is the one axis where Studio reads lower, and it cannot be made to agree. Studio's eligible stream is ~3.3x larger and each launch costs 3x more, so a point of rate is worth about ten times as much there; matching the notebook leg's 15 percent would be ~86 GPU-h a week. Both budget blocks now say so, and a test re-derives the joint spend from the two headers instead of trusting either total. Also fix a test that had gone vacuous: pre-commit.ci reformatting default=10 to default = 10 stopped the payload regex matching, so the step count agreement check was comparing nothing. Match count is asserted now. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Write down where the Kaggle sampling percentages come from The percentages in the two workflows were arithmetic nobody could check. This records the measurements they came from, taken 2026-08-11 over a 28-day window: 985 commits to main, 1157 PRs, and 8 of 60 sampled PRs touching the paths a T4 training run could regress, so a 13% path-filter rate and roughly 75 candidate events per week. Against 30 GPU-h/week for one account at ~0.75 GPU-h per launch, the ceiling is about 33 launches per week if everything went to one workflow. The 15/5 split and the 25/10 reserve-hour floors follow from that, and the file says what would change them: a second account, a wider path filter, or a session cost above ~1 GPU-h. * Give the GRPO leg its own kernel: pairing it with gpt-oss fails Kernel unsloth-t4-ci-70a2f4eb ran the wired pair on the two cards of one session. gptoss passed: 32 unique graphs, 779 calls captured, 2 graph breaks, peak 12.78GB. grpo failed with the illegal memory access again, at exactly the 13.60GB peak at which it PASSED alone on kernel unsloth-t4-ci-53efcc4e, same config to the flag. Same peak, different outcome, so the constraint is not GPU memory: the driver pins one card per payload. gpt-oss offloads to host RAM and vLLM wants host RAM, and a Kaggle session has one host. A third kernel costs a queue slot rather than quota, since Kaggle runs two at a time and bills wall clock per session. That is cheaper than a leg which fails half the time for a reason the report cannot show. Two tests pin the rule in both directions: grpo must not share a kernel with gptoss, and control must still share one with canary, because those two are a matched pair whose comparison is the whole instrument. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Unwire the GRPO leg: running it alone reproduced the failure I blamed on pairing I gave grpo a kernel of its own earlier today on the reasoning that sharing a session with gpt-oss was what broke it. It had failed paired (unsloth-t4-ci-70a2f4eb) and passed alone (unsloth-t4-ci-53efcc4e), so pairing looked like the variable, and I wrote two tests pinning the separation. That reasoning was wrong. Running the leg ALONE again (unsloth-t4-ci-c98f14be, built from this branch with --legs grpo) reproduced the paired failure exactly: unsloth_zoo/vllm_utils.py:601 sleep() -> torch.cuda.empty_cache() torch.AcceleratorError: CUDA error: an illegal memory access was encountered engine_built false, peak 13.8GB/13.6GB of 14.56GB. The passing session and both failing ones agree to the digit on every recorded version -- torch 2.10.0+cu128, transformers 5.5.0, trl 0.24.0, peft 0.19.1, vllm 0.19.1, unsloth 2026.8.15, zoo 2026.8.10 -- on the same peak, with UNSLOTH_VLLM_STANDBY=1 throughout. One contrasting observation was never enough to blame a shared host. So what three sessions show is an intermittent illegal memory access on Turing, one pass in three, not a pairing effect. A leg that passes one session in three cannot tell CI anything: it would go red for a reason no reader could act on. grpo moves to UNWIRED with the three session ids and the open question written down, rather than back into a kernel. The note keeps what the leg already earned -- the vLLM/torch pin, the attention backend, the libcuda shim, the chat template, the T4-sized config -- so re-wiring means answering the IMA rather than redoing the install. test_grpo_does_not_share_a_session_with_gptoss is replaced rather than deleted: it asserted the opposite thing for the disproven reason, and would now pass vacuously since grpo is in no kernel at all. The replacement pins that grpo stays unwired and that the note still names all three sessions, so the evidence cannot be dropped on the way back in. 145 passed. * Retire the bootstrap password at login, so the session can reach the routes First hardware run of this payload, Kaggle kernel unsloth-t4-ci-412345d2 on two Tesla T4s: preflight, studio_ready, authenticate and the Playwright chat_ui driver all passed, and the four assertions the payload actually exists for did not run. gpu_inference: POST /api/inference/load -> HTTP 403: {"detail": "Password change required"} lora_training: POST /api/train/start -> HTTP 403: {"detail": "Password change required"} A bootstrap account is created with must_change_password set, and get_current_subject turns that into a 403 on every route except the password-change one. So login yielded a token that authenticated and could do nothing, and the login step reported success while inference, tool calling, training and export were all unmeasured behind it. login now completes the change when Studio's own Token says must_change_password, and carries the token that /api/auth/change-password mints. The replacement is secrets.token_urlsafe(24): never whitespace, which the route rejects, and never equal to the bootstrap value, which it also rejects. It is not returned or stored -- nothing needs it again. Four tests, all off-box against a scripted HTTP layer: the change happens, and the session carries the post-change token an account already past the gate does NOT get its password rotated a refused change raises instead of carrying on with a token that cannot act a change that answers without a token raises The third is the one that matters: continuing past a failed change is how the first run produced four unmeasured assertions and no explanation. Two things this does NOT fix, recorded so the next run is read correctly. gguf_export reported install_kind=None, i.e. no llama.cpp install under STUDIO_HOME at all. That may well be downstream of the 403 rather than independent: POST /api/inference/load is what pulls a llama.cpp for a GGUF model, and it never ran. The collected kernel.log is 223 lines and holds none of the install output, so this is not settled either way -- the re-run answers it. gpu_inference also failed its own evidence test: text came back, but no probe could show the GPU was used. The payload calls that a failure rather than a pass, which is the behaviour I want kept. 89 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hand the UI driver the password the session actually holds Second hardware run, kernel unsloth-t4-ci-9ddd8ae4. Retiring the bootstrap password fixed three assertions and broke a fourth: gpu_inference FAIL -> PASS tool_calling FAIL -> PASS lora_training FAIL -> PASS chat_ui_driver PASS -> FAIL: "the bootstrap password is gone, so the driver cannot log in" My own doing. assert_chat_ui read $STUDIO_HOME/auth/.bootstrap_password off disk, and authenticate() has to retire that value to get past Studio's forced password change, so the file is stale by the time the driver runs. The driver's first phase rotates the password itself and asserts the old one stops working, so it needs the value the session is CURRENTLY authenticated by -- not the seeded one. login now leaves that value on Studio.password and assert_chat_ui passes it, falling back to the seeded file only when no login has happened. The commit that introduced the retirement said the replacement was "never returned or stored: nothing needs it again". That was wrong, and the docstring now says so and says why. Studio.password is a credential held for the run, unlike anything else in that module, so the module docstring records it and assert_chat_ui adds it to the scrub set before it can reach a log. test_the_session_remembers_which_password_is_current pins both directions: the retired session reports the replacement it minted, and a session that never had to change reports the password it logged in with. Still failing, and NOT downstream of the 403 after all: gguf_export reports install_kind=None on a run where inference, tool calling and training all passed, so no llama.cpp is installed under STUDIO_HOME regardless of the auth gate. I said last time that this might resolve itself once the 403 was fixed. It did not. It is a separate gap and it stays open. 90 passed. * Record what launch blocking did to the GRPO leg: it removed the fault The UNWIRED note asked for one experiment: run the leg with --cuda-launch-blocking so the illegal memory access reports at its real call site instead of surfacing at torch.cuda.empty_cache(). Done, kernel unsloth-t4-ci-b1f23e34, and the answer is not the one the note expected. With blocking on there was no illegal memory access at all. engine_built true, three steps, same 13.8GB peak, same versions to the digit. A fault that disappears when the launches are serialised is a race, which is what one pass in four already suggested and which no amount of re-running the leg will pin down. The run also exposed a SECOND problem, and it is separate from the first. It failed on reward_std = [0.0, 0.0, 0.0] with grad_norm 0.0 at every step. The completions in the report are coherent prose, so this is not the model collapsing into degenerate output. It is two completions scoring identically. The leg runs num_generations = 2 because that is what fits on a 14.56GB card, and at two samples a tie on a coarse reward is ordinary rather than a bug. So the leg's pass criterion is fragile at the size it has to be to fit, which is worth knowing before anyone reads a red grpo leg as a regression. The note now records both, says which question each answered, and states the two that remain open: where the race is, and what pass criterion is honest at num_generations = 2. The test gains the fourth session id and a check that the note no longer reads as though launch blocking were still the next thing to try -- a stale "try this next" is how an experiment gets run twice. 145 passed. * Re-seed the account before the UI driver, because the two want opposite states Third hardware run. The API assertions all pass now: preflight, studio_ready, authenticate, gpu_inference, tool_calling, lora_training -- all PASS and the Playwright driver still fails, for a third and final reason in this chain: [ui] STEP change-password through UI (Setup your account) [ui] change-password form attempt 1 failed: Timeout 60000ms exceeded waiting for locator("#new-password"); page.url=.../login The driver's first UI step drives the forced-change form. authenticate() has to retire that same bootstrap password over the API to reach /api/inference/load and /api/train/start at all. So by the time the driver runs, Studio shows an ordinary login and #new-password does not exist. That is not a value problem, it is a state problem, and the previous commit only fixed the value. The API path wants the change DONE; the driver wants it PENDING. One account cannot be in both states, and three runs walked the whole chain to get here: 412345d2 failed the API assertions on the gate, 9ddd8ae4 fixed those and failed the driver on a stale password, this one fixed the password and failed the driver on the form being gone. assert_chat_ui now restarts Studio before handing over. start_server() removes $STUDIO_HOME/auth, which is what re-seeds the bootstrap password, so the driver gets exactly the state its first step expects. It is cheap here specifically: this assertion runs last and stops the server anyway, so nothing after it needs the API session, and the driver loads its own model through the UI. test_the_ui_driver_gets_a_freshly_seeded_account pins the restart, that the password is read AFTER it, and that the retired session's password is not what gets handed over -- the restart is the whole fix, and a refactor that drops it puts the payload straight back to a driver that cannot find the form. Unchanged and still open: gguf_export reports install_kind=None. Three runs now, one of them with inference, tool calling and training all green, so no llama.cpp is installed under STUDIO_HOME and this is not downstream of the auth gate. 91 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Test the newest transformers and trl, not the newest zoo permits The canary leg is documented as "the newest permitted library set" and that is exactly what it installs: it resolves WITH unsloth_zoo in the resolution, so zoo's own metadata is a ceiling. Measured against the two runs that carried it: canary transformers 5.5.0 trl 0.24.0 peft 0.20.0 accelerate 1.14.0 control transformers 5.5.0 trl 0.24.0 peft 0.19.1 accelerate 1.13.0 PyPI the same day: transformers 5.15.0, trl 1.9.2, peft 0.20.0, accelerate 1.14.0. The canary moved peft and accelerate to genuine latest and moved transformers and trl not at all, which is what made it look like it was working. zoo pins transformers <=5.5.0 and trl <=0.24.0, ten minors and a whole major behind, so this CI could not detect a transformers 5.6+ or trl 1.x regression: it never installed one. The new frontier leg upgrades transformers and trl with their dependencies and without zoo in the resolution, which clears the cap because pip enforces only the requirements of packages in the resolution. It runs the same SFT payload as control and canary, so all three are directly comparable. I expected it red. It is green. On a real T4: transformers 5.15.0, trl 1.9.2, datasets 5.0.1, ten steps, canary emitted, and two fresh processes agreeing bitwise. Unsloth trains and generates correctly a whole trl major above what zoo's metadata permits. Getting there took two kernels. The first used --no-deps plus a blanket tokenizers upgrade and died before running anything, because an unbounded upgrade overshoots the ceiling transformers declares: tokenizers<=0.23.0,>=0.22.0 is required, but found tokenizers==0.23.1 safetensors>=0.8.0 is required, but found safetensors==0.7.0 A test pins that shape out, along with the zoo-in-the-resolution mistake that would silently turn this into a second canary. It goes on the second kernel's idle T4, so it costs no quota: a Kaggle session bills wall clock once, not per card. What it does NOT catch is recorded in the leg: the loss trajectory is not the control's, and step 1 is computed before any update on identical weights, data and seed, so 10.3222 against 6.4367 is a different loss function rather than drift. Both converge. Which objective is intended is separate work. * Install a CUDA llama.cpp before asserting the export used one Four hardware runs reported llama_cpp_install_kind=None and failed the GGUF export assertion for it. The reason was never subtle: nothing had ever installed a llama.cpp under STUDIO_HOME, so the assertion was measuring an absence rather than a selection. install_llama_prebuilt.py resolves a real "linux-cuda" kind on an x64 CUDA host, so the bundle was available the whole time and simply never fetched. The install runs before the server, so the export route sees a llama.cpp that was already there rather than one appearing underneath it, and it is its own assertion because "the CUDA bundle would not install on this box" and "it installed and the export against it failed" are different findings. Its result is deliberately not checked by execute(). A box where the bundle cannot be installed still runs inference, tool calling, training and the UI, and still reports the export failing for the reason it actually failed. Three tests. One of them CALLS install_llama_cpp rather than reading its source, because run() in this payload already applies capture_output and text and passing either again is a TypeError that no source-reading test would catch -- which is how the first draft of this was written wrong. * Record what install.sh left behind before replacing it build_kernel.py runs install.sh --local, whose own comment says it puts a llama.cpp on disk. Four runs still reported install_kind=None, so either it never installed one or it installed something else. Fetching a CUDA bundle fixes both cases and would hide which one it was, so the prior state is recorded first and the original question stays answerable. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Look for llama.cpp where it is, not where the payload assumed Five hardware runs reported llama_cpp_install_kind=None and failed the GGUF export for it. There was no missing llama.cpp. install.sh --local installs one into ~/.unsloth/llama.cpp, which is what install_llama_prebuilt.py's own default resolves to, and this payload read STUDIO_HOME/llama.cpp, a path nothing ever wrote to. Adding an explicit install last commit is what proved it, because it recorded what was there first. The installer selected the right bundle for a T4 accept ...linux-x64-cuda13-older.tar.gz coverage=75-89 supported=75,80,86,89 reject ...linux-x64-cuda13-newer.tar.gz reasons=missing_sms=75 out_of_range_sms=75 and then said existing llama.cpp install already matches selected release b10360-mix-87da1a2; skipping download and install while install_kind_before and install_kind_after at the requested directory were both None and the return code was 0. So a CUDA llama.cpp was present the whole time, the sm_75 selection works, and only the assertion was wrong. llama_cpp_marker checks STUDIO_HOME first, then the canonical location, and returns None when neither has a marker so a real absence stays a real absence. Against the exact situation the last run recorded, the old read answers None and the new one answers linux-cuda. Separately, and not fixed here: install_llama_prebuilt.py --install-dir X exits 0 having written nothing to X whenever the canonical directory already holds a matching release. A caller who names a directory and gets a success with an empty directory has been told something untrue. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Read the field the marker has, not the one it never had Six hardware runs reported llama_cpp_install_kind=None and failed the GGUF export assertion for it. There was no missing llama.cpp and no wrong directory. install_llama_prebuilt.py writes install_kind ONLY into the JSON its resolver prints to stdout. The marker it writes to disk records requested_tag tag release_tag published_repo asset force_cpu llama_backend asset_sha256 source_* ggml_tree bundle_profile runtime_line coverage_class install_fingerprint prebuilt_fallback_used installed_at_utc and no install_kind at any point. So reading install_kind from the marker answered None for every bundle on every box, including a working CUDA one, and the assertion turned that into a failure. The last run had the installer saying "existing llama.cpp install already matches selected release b10360-mix-87da1a2; skipping download and install" while the payload reported nothing installed. runtime_line is the field that answers the question: cuda12 / cuda13 for the CUDA bundles, and the backend name otherwise. It is written into the same dict as the asset, so the two cannot disagree, and asset is kept as a fallback for a marker too old to carry it. is_cuda_install now matches the runtime line's shape rather than a fixed set of names. A fixed set needs a new entry per CUDA major and fails CLOSED when it lacks one, reporting a working cuda14 install as not-CUDA, which is the same failure this commit is undoing. The existing test wrote {"install_kind": "linux-cuda"} into a marker and asserted it came back. That is how this survived: the test agreed with the payload about a field neither read from reality, and stayed green through all six runs. It is rewritten against a marker the installer actually produces. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Measure the load, not the unload it follows The GGUF export probe reported device_vram_delta_mib: -1866.0 and was scored as a load that never reached the GPU. It had not: a 3004 MiB chat model was evicted inside the same window the 531 MB GGUF loaded in, because POST /load with force unloads first and the baseline was sampled before the request. Unload explicitly, wait for the reading to stop falling, then take the baseline. Best-effort throughout, since the delta is evidence and not the assertion: a refused unload or a missing nvidia-smi leaves the probe running. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Read the status fields the response actually declares Run 8 shipped the unload-before-baseline fix and came back with device_vram_delta_mib: -1866.0, byte-identical to run 7. The fix ran and did nothing: it looked for model_path, model and active_model_name, and InferenceStatusResponse carries none of those, so no model was ever identified and no unload was ever sent. Read model_identifier (documented as the loadable identifier, which is what /unload's model_path wants), falling back to active_model and then to loaded[0]. A new test parses the field names out of InferenceStatusResponse and asserts the payload only reads names it declares, so the next guess fails in the suite rather than forty minutes into a kernel. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Delete the Kaggle kernel when the launcher is killed, not only when it returns release() is the budget control: a kernel left behind bills to its own ceiling with nobody reading the result, and that ceiling has been observed not to stop a wedged one. But it was reachable only from finish(), which runs only on paths that RETURN. KeyboardInterrupt was explicitly re-raised past it, and the default SIGTERM disposition exits without running atexit or finally. So Ctrl-C, `kill`, and a cancelled GitHub Actions workflow all left the kernel running. The file's own comment claimed the kernels were released "on this path and on every other", which was true for returns and false for signals. Four paths, four covers: normal return / handled error finish() unhandled exception atexit Ctrl-C, kill, Actions cancel SIGINT/SIGTERM/SIGHUP handlers kill -9 the orphan sweep at the next launch The signal handler re-raises through SIG_DFL so the exit status still reads "killed by signal N"; exiting 0 would make a cancelled job look completed. kill -9 cannot be caught, so a pushed slug is now recorded to logs/kaggle_inflight.json the instant it exists, and every launch sweeps entries whose owning pid is gone. Keyed on pid deliberately: an entry whose owner is still alive belongs to a run in progress, and deleting it would destroy a concurrent run and report the absence as a failure of the code under test. A delete that fails keeps its entry rather than forgetting it. The sweep runs before pushing, so reclaimed session slots are available to the run doing the reclaiming. Tests drive real subprocesses and real signals, since the property is about process death. Verified failing against the previous behaviour: SIGINT, SIGTERM and the unhandled-exception case all leave the kernel behind without these handlers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only forget a Kaggle kernel once it is actually gone kaggle kernels delete answers a transient rejection with a nonzero exit and subprocess.run does not raise on that, so both cleanup paths recorded a refused delete as done: sweep_orphans dropped the registry entry and release_kernels set released=True, leaving a running kernel billing with nothing left to reclaim it. Check the return code in both. Two more holes in the same registry. result["kernels"] was published after the whole push loop, so a SIGTERM once the first kernel was up found no list and deleted nothing; it is now the same list object, assigned before the loop. And --keep-kernel left an entry naming a pid that dies with the launcher, so the next sweep called the retained kernel an orphan and deleted exactly what the flag asked to keep. * Stop the Studio leg passing on evidence it never had Six ways this job could go green without measuring anything. The evidence bundle was never unpacked: the launcher writes each kernel into kaggle_evidence/<slug>/ and the workflow hands collect_evidence.py the parent, so a top-level glob found nothing and every run reported that the payload emitted no bundle. Recurse, the way extract_reports already does. An install.sh regression read as infrastructure. sh() raised SystemExit, so papermill stopped before any T4_SMOKE_REPORT existed, the launcher filed the run as infra and the reporter exited 0 -- silently passing the installer failures this workflow's path filter selects for. Failures of the checkout under test now emit a failing report first; a missing GPU or a clone that would not download keep the no-report infra path. A payload killed by the driver deadline does the same, but only once the payload itself had started. The process-level VRAM probe could never fire: InferenceStatusResponse declares neither llama_server_pid nor pid, and FastAPI drops what the response model does not declare, so the lookup was always None. Find the llama-server processes in /proc instead. Offload evidence outlived its load. offloaded_layers() takes the last match in the whole log, so a reload that logged nothing inherited the chat model's line and passed on evidence from a different model. Read each load forward from a mark taken just before it. A diverged run counted as a trained one. A T4 has no bf16, and an fp16 run that goes NaN still reaches completed and still saves an adapter, while trained_steps counted anything that was not None. Only finite losses count, and non-finite ones are named as a failure. An export slower than its HTTP timeout crashed the payload. The route blocks for the whole export and the transport timeout is 900s against a 1200s deadline, so the TimeoutError escaped the StudioError handler; it is now treated as in-flight and the status poll decides. Alongside: the restart that re-seeds the account no longer truncates the backend log of every earlier assertion, the over-cap evidence fallback ships the logs it says it ships rather than the report alone, the result line is printed after the evidence is packaged so a packaging crash cannot publish a pass, the training and export polls stop when Studio dies, Studio is launched from the interpreter running the payload rather than whatever is on PATH, the gate is told this leg pushes one kernel rather than the default two, and the advertised opt-in label can start the workflow. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the encoding on every checked-in read, and survive a box with no nvidia-smi tests/test_source_read_encoding.py fails on 33 read_text() calls in the two Kaggle harnesses: unpinned text I/O on a repo file is cp1252 on a stock Windows install and dies the moment that file gains a non-ASCII byte. The second one is the payload's own. environment() runs inside finish(), on every path out of the run, so a session Kaggle handed no driver raised FileNotFoundError from gpu_inventory() and ended with a traceback and no report at all. run() now answers a missing binary the way a failing one does, and the preflight says there is no GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Spend a Kaggle session on the opt-in label, not on every label after it Two ways this leg spends or loses a GPU session it should not. GitHub has no per-label filter on the pull_request event, so adding `labeled` to the activity types starts the workflow for EVERY label added to an affected pull request. The gate is handed the whole label list, so once kaggle-studio-gpu-ci sits on the pull request it reads every one of those events as forced, and a routine label change launches another 70-minute Kaggle run. Guard the gate job on the label that fired the event; push and the other pull_request activities carry no `github.event.label` and fall through unchanged. The other way is the reverse: a run that should be red exits 0. The dependency probe raises out of the cell when the Studio venv reports no CUDA, and with no T4_SMOKE_REPORT the shared launcher files the run as `infra`. install.sh --local resolving a CPU-only torch is exactly the CUDA install regression this workflow's path filter selects for, and it took that path. Ask nvidia-smi which case it is: a GPU the host can see and the venv cannot use is a payload failure, and no GPU at all stays infra. * Treat a stalled kaggle push as transport, not as a red verdict Every other Kaggle transport failure -- a 503, a refused push, a capacity rejection -- returns a reason from push() and exits 0 as infra. A push that stalls past the 600s ceiling did not: TimeoutExpired escaped launch.py, so the process ended before finish() could write launch_result.json, the launch step exited nonzero and GitHub marked the job red. The reporter that reads that file and calls the run NOT RUN never got a say, and a red there says the code under test failed when nothing ever reached a T4. Catch it in push() and report push_timeout. * Do not let a truncated copy of a chunk overwrite the complete one * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the encoding on every checked-in file the T4 harness reads Repo tests (CPU) is red on this branch: test_source_read_encoding.py::test_checked_in_file_reads_name_an_encoding AssertionError: 17 file reads in the test trees touch a checked-in file with the platform default encoding, so they break on Windows as soon as that file gains a non-ASCII byte. All 17 are in tests/kaggle/test_t4_smoke_harness.py, which this branch adds, so this is ours and not inherited. The reads pull in the workflow YAML, the smoke scripts, the committed reference JSON and ci-preempt.json, and none of them said what encoding to read them as. On Linux that resolves to UTF-8 and nobody notices; on a Windows runner it resolves to the ANSI code page and the first non-ASCII byte in any of those files takes the test out with a UnicodeDecodeError that has nothing to do with what the test is checking. Every one now passes encoding = "utf-8", matching how the rest of the suite spells it. Reads of files the test itself just wrote under tmp_path are left alone: the gate only covers checked-in files, and those are ours end to end. test_source_read_encoding.py 1 passed tests/kaggle/test_t4_smoke_harness.py 148 passed * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Flatten Kaggle's kernel log before extracting the reports from it Kaggle's kernels/output returns the log as a JSON array of {stream_name, time, data} records, one record per line, so no line in the file starts with T4_SMOKE_REPORT. extract_reports read it verbatim, so the log fallback -- the one that covers a run whose executed notebook never came back -- recovered nothing and a failed assertion was scored as infra with the workflow exiting 0. report.py::kernel_log_text and collect_evidence.py::iter_text already flatten the records; extract_reports was the reader that did not. The records are also cut by write rather than by line, so the join has to happen before any line splitting. * Keep the slug a timed out kaggle push may have created A push whose CLI stalls past the 600s ceiling says nothing about whether Kaggle took it. Kaggle can accept the push and start the kernel billing before the response is lost, and the branch returned infra while keeping neither the slug nor an inflight entry, so finish() had nothing to delete and the next launcher's orphan sweep had nothing to see. The kernel ran to its ceiling with nobody reading the result. The slug is ours and already decided at that point, so record it in the inflight registry and hand it back as orphan_slug: release_kernels deletes it on the way out, and a launcher killed first leaves the entry for the next sweep. It is deliberately not returned as slug, because the caller must not wait on a kernel that may never have been created. * Align the Kaggle budget guide with the workflow it documents The guide named --percent 15 and --reserve-hours 25 against a 30h account; the workflow runs 40 and 20 against the 60h figure measured on 2026-08-11, and it is the only consumer of that quota today. Following the guide would have tuned the reserve from a baseline that no longer exists. Re-derived from the same live numbers as the workflow header, and says which of the two is the source of truth when they disagree. * Stand the gate down on any unreadable kernel status The concurrency verdict refused only when EVERY in-window status came back an error. One unreadable status among readable ones was treated as an idle account, and driving concurrency_verdict with {busy: [], own: [], foreign: [], surveyed: 2, unreadable: 1, complete: True} returned (True, ""). That kernel can be the human session the zero-foreign policy exists to yield to, and proceeding takes the account's last slot: the first push wins it, the second comes back at capacity, and the run reports a subset of its legs. A 404 is kept separate and stays benign. A deleted kernel is not an unknown state and the launcher deletes every kernel it pushes, so blocking on that would wedge the gate shut rather than make it careful. * Close six gaps in the Kaggle T4 notebook CI workflow - The pull_request trigger used GitHub's default activity types, which are opened, synchronize and reopened only. The kaggle-t4-ci label the gate advertises as a human override could therefore never start a run on its own. Subscribe to labeled as well. - pyproject.toml was in neither paths filter, though every payload installs the commit under test as a distribution built from it. BUDGET.md counted it as a watched path already. - The job deadline of 120 minutes was shorter than the launcher's worst case. Two sequential pushes can take 2 x (4 x 600s + 315s of backoff) and the 5400s polling window only starts after them, so about 187 minutes with the deletions; a runner killed at 120 takes finish() -> release() with it and leaves pushed kernels billing quota. 200, with the arithmetic written down beside it. - The gate surveys quota and in-flight kernels before t4-smoke queues on the account-wide concurrency group, so a second sampled run could launch on an approval a full run old. Re-ask with the slot in hand, immediately before the push; --force true skips the sampling draw and nothing else. - --zoo-ref main let each payload's own pip resolve a different unsloth-zoo commit, including between the control and the canary within one session, and zoo is not in pins/control.txt either. Resolve it once with git ls-remote and pass that SHA to every leg. - A dispatch with max_steps != 10 still shipped the 10-step reference, and check_reference reports step_count_mismatch as a failure, so the run could only ever go red on arithmetic. Drop the band automatically when the counts differ, and say so. Also: check out the same head SHA the launcher installs, so the harness and the package under test are one snapshot rather than the merge tree and the head; and run tests/kaggle/test_t4_smoke_harness.py on the runner before anything is built or pushed, since testpaths limits pytest to tests/security and nothing else collected it. * Read back what the Kaggle kernel actually reported Two ways a real payload verdict was being lost in transport, both ending in verdict=infra and a green workflow. The kernel log fallback could never work. Kaggle's kernels/output hands the log over as a JSON array of {stream_name, time, data} records, which is why report.kernel_log_text flattens it before reading; extract_reports scanned the file as text, so no line began with the report prefix and the fallback that exists for the run whose notebook never came back found nothing. Record boundaries are not line boundaries either, so the records are joined before the scan rather than read one at a time. push() documented a fresh slug per attempt and used one for all four. Pushing to an id that already exists does not replace it: Kaggle files a new version and starts a second batch session, and kernels status and kernels/output send no version label, so they answer for the newest session only. A retry after a lost response therefore collected the wrong execution's evidence while the first kept a session slot and billed unseen. Each attempt now takes its own slug and discards the previous one first, and every slug filed is returned and recorded, since a push that reported an error may still have landed. * Give each payload its own everything, and report a probe failure Four ways the two payloads of one kernel were not actually isolated, plus the import probe exiting without a verdict. The probe was the P1. When the commit under test breaks import unsloth, the verify cell raises before the run cell can write a report, the launcher extracts none, and no reports is classified as infra -- so a deterministic import regression exits green. A probe failure is a verdict, not missing evidence, so it now emits its own failing report. nvidia-smi failing or a one-GPU allocation fell back to N_GPU = 1 and pinned both payloads to device 0. Each child still saw exactly one card and passed its own visibility assertion, so the contention came back looking like a code failure. A shortfall is infrastructure and stands the kernel down before a thread starts, which is what the comment above it already claimed. A payload whose virtualenv could not be built kept the base python3 kernel and installed into the shared system site-packages. That is the one thing the per-child venv exists to prevent: the legs install deliberately different library sets, so the last writer wins and the resulting import error reads as a regression. The payload is skipped instead. Both payloads materialised into one source directory and compiled into one unsloth_compiled_cache. The sources are byte-identical copies but write_bytes truncates first, and the compile cache is a relative path resolved against a working directory both papermill children inherit while compiling the same modules against different transformers versions. Each payload now owns both, and the prune globs still reach them. Finally, --smoke-args is appended after every leg's own arguments, so the workflow's --max-steps 10 landed after the gpt-oss leg's --max-steps 3 and argparse took the last one. The 20B leg was training for ten steps rather than the three that were measured to fit. A shared argument no longer reaches a leg that sets that option itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 06:30:18 -07:00
Daniel Han	df64e5d795	Round the aggregate GPU VRAM total so it stops printing float residue (#8820) ... * Round the aggregate GPU VRAM total so it stops printing float residue The backend rounds each device to 2dp, so summing them back up reintroduces binary floating point error: three B200s at 179.06 GiB each add up to 537.1800000000001. The training run preview card interpolates the total without rounding it again, so it showed that in full. Round the sum back to the precision the devices arrived with, and move the helper next to the other VRAM rules in gpu-vram.ts so it can be unit tested without pulling the auth and React graph in behind it. use-system.ts re-exports it for its existing callers. * Tighten the aggregate VRAM comment	2026-08-14 06:25:56 -07:00
Nilay	7ae2069ea9	Cover the Audio header in the macOS drag-region contract test (#8718) ... The audio page fix itself landed in #8445, which rebuilt the header as the same 408px grid rail Images uses and gave the band pointer-events-none relative z-40 with pointer-events-auto on the control groups. What that PR did not do was extend test_media_page_headers_out_stack_the_mac_drag_region to the Audio page, so nothing pinned the opt-in: stripping pointer-events-auto off the audio model selector left the whole suite green. Parametrise the loop over the three media pages and match on the pointer-events-auto opt-in alone rather than on one layout's flex utilities. Audio and Images seat their mode pills in a grid cell, so the old "pointer-events-auto flex ... items-center gap-" pattern skipped them silently. Removing the opt-in from any of the five control groups across Images, Video and Audio now fails the test. The inner per-group assertion this replaces was tautological: the regex already required the literal pointer-events-auto that it went on to assert.	2026-08-14 06:05:27 -07:00
Nilay	1fb184deb4	Studio: self-repair sidecars whose extensions were built for another Python (#8705) ... * Studio: detect ABI-mismatched sidecar extensions so self-repair fires after a Python upgrade * Studio: scan extension basenames and flag stable-ABI binaries on free-threaded builds Match the version tag on the file basename rather than the whole RECORD path, so a directory component carrying a wheel-style tag does not wipe the sidecar. Report .abi3 binaries only under a free-threaded interpreter, which cannot load them. Fix the stale-tag pick in the detection test so it stays stale under cp313t. --------- Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-08-14 06:05:14 -07:00
Nilay	2899072197	Studio: drag to reorder pinned models in the hub (#8591) ... * Studio: drag to reorder pinned models in the hub * Studio: do not let a cancelled pin drag reorder the list The pinned grid reorders live on dragenter, which is what makes the drag readable, but movePinned also persisted on every one of those. A drag the user cancelled with Escape or released over the grid gaps still ended with the new order applied and already written to localStorage, and dragend could only clear the drag marker. Measured across chromium, firefox and webkit: a cancelled drag left the reordered list behind and it survived a reload. A drag is now a session. beginPinnedDrag snapshots the order, movePinned keeps the intermediate moves in memory, and the session either commits once on drop or restores the snapshot when the drag ends without one. The live preview is unchanged. A three-cell drag drops from three localStorage writes to one, so other windows no longer get a storage event per hover. If another window rewrote the pin set mid-drag the newer list wins, since rolling back a stale snapshot would resurrect a pin removed elsewhere. Two smaller fixes in the same grid. The 0.4 drag opacity keyed off the pin key, so a repo cached in two formats, which renders two rows sharing one pin key, dimmed both cells while only one was being dragged; it now keys off the row. And itemPinKey is gated on the key actually being pinned, so a cell can never advertise a drag that movePinned has nothing to move. Tests go from 4 to 24: the splice convention asserted by name in both directions, degenerate movePinned inputs, and the drag session including the cancel path. The case named "keeps keys not shown in the current view in place" asserted something else, since a key moving past a hidden entry necessarily shifts it by one slot; it is renamed to "in relative order", which is what it checks and the right guarantee for a list the hub and the model selector share. * Studio: let another window's pin order survive a cancelled drag A drag session snapshots the pin order at dragstart and rolls back to it when the drag ends without a drop. If another Studio window rewrote the list while the drag was in flight, that snapshot is stale: the storage listener has already replaced state.pinned with the newer order, so the rollback puts this window back on an order that is no longer in localStorage. The previous guard only declined the rollback when the KEY SET differed, so a pin added or removed elsewhere was respected but a pure cross-window reorder was not. Nothing is lost at the moment of the rollback, since a cancel writes nothing. The damage lands on the next write from this window, which persists the whole list: with ["a","b","c"] locally, another window reordering to ["c","b","a"] mid-drag, and a cancel, the following pin of "d" wrote ["d","a","b","c"] and the other window's order was gone. That case is now a test and it failed before this change. The storage listener records the order it installed when a drag is live, and the session falls back to that instead of the snapshot. Tracking the order rather than a bare "a storage event happened" flag is what makes the two paths fall out on their own: the fallback is the order that is actually in localStorage, so a cancel lands on it whatever the other window did to the keys, and the sameKeys multiset compare is no longer needed at all. It also drops the moves this drag previewed on top of the newer list, which a bare flag that just declined the rollback would have left applied. Declining is not enough and merging is not defensible: the user abandoned the drag, so this window has no reordering intent left to merge in, and the other window's order is the only order anyone wrote. On the commit path the drop still wins, but it now wins on top of the other window's list rather than the snapshot, and "nothing moved, so write nothing" is measured against that same base so a drop that only re-applies what the other window already stored writes nothing. The recorded order is cleared at beginPinnedDrag, so it only ever describes the session it landed in. The test named "a cancel rolls back even when a same-set write landed mid-drag" never simulated a mid-drag write, so it was a duplicate of the plain cancel case while advertising cover for exactly this bug. The storage listener could not be reached from a test at all, because the localStorage fake installed a no-op addEventListener; it now keeps its listeners so a test can deliver a real storage event, and every cross-window case goes through the real listener. Cases added for a reorder, an add and a remove landing mid-drag, the clobber on the next pin toggle, a drop rather than a cancel after one lands, a redundant commit, an event for an unrelated key, an event with no drag in flight, and the reset between sessions. Frontend suite 1632 pass / 0 fail, typecheck clean. The chromium 151 / firefox 153 / webkit 26.5 drag matrix is unchanged: a cancelled drag still restores its order with zero writes in all three, a three-cell drag still writes once instead of three times, and a drop still persists. * Studio: do not let a dataset row drag the model pin order Pin keys carry no repo type, so a downloaded dataset whose repoId also names a pinned model lands in the hub's Pinned slice on the same predicate a model row uses. Two such rows made the dataset list reorder the user's model pins, and the order survived a reload, while the row menu offers datasets no pin action at all. Gate the drag key on the list mode so the affordance matches. --------- Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-08-14 06:04:58 -07:00
Nilay	824caca05b	Desktop: Keep server stopped across webview reloads (#8567) ... * Keep server stopped across webview reloads * Persist stop intent before shutdown await and for external disconnects * Honor stop intent in owned_ready preflight path * Check stop intent before preflight to avoid its adoption side effects * Guard every sessionStorage access behind a stop-intent helper An opaque webview origin, private browsing or blocked cookies make sessionStorage throw on access. The read in checkInstallAndStart ran outside its try, so a SecurityError there rejected the mount effect's floating promise and left the startup screen on "checking" with no way out, and the write in stopServer would have aborted the stop before it reached stop_server. Move the key and its three accesses into server-stop-intent.ts, where each one is wrapped and unreadable storage reads as no intent, which is the pre-marker behaviour. Matches the localStorage shim in api-monitor/overlay-store.ts. Also clear the marker ahead of the re-entry guard in startManagedServer, so a start the user asked for retires an earlier stop even when that call no-ops. * Add tests for the desktop stop-intent marker Covers the marker helpers against working, throwing and absent storage, asserts the key has a single owner in src/, and pins the call-site ordering the feature depends on: the stop check ahead of desktop_preflight, the marker written before the stop_server await it has to outlive, the rollback on a failed stop, and the clear on every deliberate start. * Desktop: serialize stop attempts so a concurrent stop cannot clear the intent The tray toggle branches on statusRef, which stays running until the invoke resolves, so a second Stop click ran a second shutdown against the backend the first was already taking down. With an adopted backend the loser can fail while the winner succeeds, and its rollback then cleared the shared stop marker, so the next reload started the server again. Guard stopServer with an in-flight ref, mirroring the double-start guard on the start path. --------- Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-08-14 06:04:24 -07:00
Daniel Han	300292b2f0	fix(amd): read ROCm VRAM through amd-smi so the backend keeps no HIP context (#8669) ... * Read ROCm VRAM through amd-smi so the backend keeps no HIP context torch.cuda.mem_get_info creates a CUDA/HIP primary context that the process never releases: 612 MiB measured on a B200 with CUDA 13, 692-712 MiB reported elsewhere. get_device_properties and get_device_capability do not, and agree with mem_get_info on total memory exactly. LlamaCppBackend._get_gpu_memory asks nvidia-smi first, so NVIDIA hosts were already clean. ROCm hosts, and NVIDIA hosts without nvidia-smi on PATH, fell straight through to torch, so a backend serving GGUF models held ~700 MiB for a number it only reads, on a card whose models run in a llama-server child that then cannot use it. Two docstrings promised the probe was torch-free. amd-smi answers the same question from a subprocess. The new branch sits between nvidia-smi and torch, reuses utils/hardware/amd.py's runner, unit parsing and circuit breaker, honours the visibility mask, and gives a unified-memory APU the same host reserve and total 0 as the torch branch. The torch fallback stays for hosts with no smi tool. Callers read an empty result as no GPU (_resolve_auto picks a backend from it, _gpu_available gates the embedder, the loader fit drops to CPU), so an unknown there would silently move inference off the GPU; the context is the lesser cost. Parity on GPU 5, amd-smi output synthesised from the live nvidia-smi reading of the same card: free 179165 MiB and total 183359 MiB both branches, 0 MiB context on the new path against 612 MiB on the torch path. NVIDIA numbers are untouched, and the branch is inert off ROCm. Rebased over #7670, which added for_llama_server and the ROCm arch gate to the same function. The new branch returns before the gate ran, so it would have put #7624 back for llama-server placement. _rocm_arch_gate_keep now holds the gate, and both branches call it, so the two cannot drift. It costs no context of its own: get_device_properties measured 0MiB against mem_get_info's 620MiB on GPU 5. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Defer to torch on unified-memory APUs, and read a single-GPU amd-smi dict for PR #8669 amd-smi reports only the dedicated VRAM carve-out on a unified-memory APU, while HIP reports the far larger GTT pool the models actually run in. On a 128GiB Strix Halo with an 8GiB BIOS carve-out that is 8GiB against ~100GiB, so the new branch would have handed the GGUF fitter the slice and refused models that fit today. The branch now declines the whole host when any visible device is a unified-memory APU: discrete ROCm cards keep the context saving, APU hosts keep byte-identical pre-PR numbers, and the two branches cannot disagree because only one ever answers. Reconstructing the pool from total_gtt was the other option and was rejected: HIP's mem_get_info on an APU is not documented as equal to GTT alone, and the older vram / fb_memory_usage shapes carry no GTT fields, so the two branches would drift, which is what _rocm_arch_gate_keep was unified to prevent. Separately, _gpu_entries' documented bare single-GPU dict fallback was dead: data.get("gpu", [data]) returns the numeric id for that shape and enumerate(int) raises. An envelope key now only counts when its value is really a list, which also closes enumerate(None) on {"gpus": null}. * Translate amd-smi gpu ids into HIP ids, and answer for every visible device amd-smi numbers its devices in discovery order over its own KFD view; HIP numbers them by KFD node id (amdsmi computes hip_id as node_id - smallest_node_id). The two agree on most hosts and not on all of them, and the branch was keying VRAM rows by amd-smi's number, comparing it against HIP/ROCR/CUDA visibility tokens and then pinning the child with it. On a host where they differ, the larger card's memory lands on the smaller card's HIP id and a load that fit before this PR can OOM. amd-smi list -e (ROCm 6.4.0+, amdsmi_get_gpu_enumeration_info) is the mapping AMD publishes for exactly this. Read it and re-key, and decline when it is not readable: an older CLI rejects -e, hip_id reads N/A when the library cannot reach the device's KFD node, and identity is then an assumption rather than a fact. A single-GPU host skips the call, since one card on the host forces the mapping, so the ROCm desktops this branch mostly runs on keep the context saving on any amd-smi version. That -e failure is kept out of the circuit breaker: an old CLI is not an amd-smi outage, and three of them would disable the utilization poller too. _resolve_visible_physical_ids returns None both for "no mask" and for a mask it cannot parse, and the branch read the second as the first. ROCr accepts UUID tokens (ROCR_VISIBLE_DEVICES=GPU-<uuid>), and amd-smi ignores the mask entirely, so a deliberately hidden card was being offered for automatic ranking. Use the existing _visibility_mask_is_unmappable and defer to torch, which enumerates the masked set. Separately, get_gpu_vram_mib drops a row whose total reads 0, and what was left was a non-empty list the caller takes as the whole host. On an APU beside a dGPU that turned two devices into one, so the "fewer than 2 usable GPUs" arm switched off tensor parallelism on a host main reports as two. get_gpu_vram_report now also returns every id amd-smi enumerated, and the branch answers for every visible device or declines entirely. The existing rocm fixture only patched _resolve_visible_physical_ids and left the shell's own CUDA_VISIBLE_DEVICES in the environment, which now reads as a mask set but unresolvable; clear it there too, the same trap as #8662. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Decline the amd-smi VRAM branch when HIP cannot open a device amd-smi reads the driver over sysfs and libdrm, HIP needs /dev/kfd and a matching HSA runtime. They come apart on a container started with --device=/dev/dri but no --device=/dev/kfd, and on a torch wheel built against a ROCm the installed runtime does not match: amd-smi lists the card and hipGetDeviceCount returns 0, so torch.cuda.is_available() is False while _torch_is_rocm() stays True. The llama-server child is a HIP process too, so it dies exactly where torch did. Before this branch existed the torch fallback returned [] for such a host and the load went to CPU. The amd-smi branch returns first and hands placement a device nothing can open. Neither existing guard catches it, since both are torch readers that fail open: _rocm_unified_memory_gpu_ids answers set() and _rocm_arch_by_physical_id answers {} the moment is_available() is False, so a unified-memory APU is not even recognised as one and its dedicated carve-out is taken for the whole pool. The gate costs nothing this path was not already paying: _rocm_unified_memory_gpu_ids calls is_available() and get_device_properties() further down the same function. Neither creates a primary context (measured: no compute-apps entry, against 612 MiB for mem_get_info), because is_available() is hipGetDeviceCount rather than _lazy_init. Both rocm fixtures now declare HIP reachability instead of inheriting the host's, the same trap as #8662: unpatched they read the test machine's own GPU and 16 of them fail on a CPU-only runner. * Answer the amd-smi VRAM branch only for HIP's own inventory for PR #8669 GPU_DEVICE_ORDINAL is a ROCm visibility variable that _resolve_visible_physical_ids does not read, so the branch took no mask from a process that had one. Defer to torch while it is set, the way hardware.py::_rocm_visibility_mask_active gates the system-wide overlay. Compare the visible device count against hipGetDeviceCount as well: a device-cgroup container given one /dev/dri/renderD* node and no env var leaves amd-smi enumerating the whole host, and a row amd-smi omitted leaves a single-GPU shape the id-mapping shortcut accepts as a one-card host. Compare the totals too, with the same 10% margin the System tab's overlay uses. An APU _rocm_classify_unified_memory cannot name (a gfx1103 Phoenix wheel with no is_integrated flag) has amd-smi reporting the BIOS carve-out where HIP reports the GTT pool, and a partition reads the other way round. Devices torch cannot describe fail open. * Decline the amd-smi VRAM branch when ROCm visibility masks are stacked ROCR_VISIBLE_DEVICES filters and renumbers agents inside the ROCr runtime, and clr's Device::init then indexes HIP_VISIBLE_DEVICES (or its CUDA_VISIBLE_DEVICES twin) into the agent vector ROCr left behind, so the two compose: ROCR_VISIBLE_DEVICES=1,2 with HIP_VISIBLE_DEVICES=0 opens physical GPU 1. _active_gpu_visibility_mask returns the single highest-precedence value, which reads [0] there, and on a host whose cards share a model every later check on the amd-smi branch agrees with it: HIP opens one device, the mask names one, and the totals match. The branch then reported physical GPU 0's free VRAM for a process that can only reach GPU 1. Defer to torch instead, which enumerates in HIP's own space; composing the masks would need the physical to ROCr mapping ROCr just renumbered away. HIP and CUDA together are one layer, not two, and Windows has no ROCr layer, so neither defers. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 05:55:18 -07:00
Maheswar Kumar	e7c1a13a52	studio: report prompt speed in tokens per second for safetensors chat (#8647) ... * studio: report prompt speed in tokens per second for safetensors chat The chat timing popover showed a prompt speed and a generation speed for GGUF and MLX, but nothing for safetensors models: `_record_generation_stats` latched usage only, so the frontend fell back to client-side total time and chunk count. Add `core/inference/generation_timing.py`. `GenerationTimer` splits one `model.generate()` call into a prefill window and a decode window, with the boundary stamped by a logits processor. Transformers calls the processor once per step, the first time with the prompt still unextended, so that first call lands right after the prefill forward pass. `build_generation_timings` maps the split onto the same `timings` shape llama-server emits, so the existing worker -> route -> usage-chunk plumbing carries it with no changes. Wired into every safetensors path that streams text to chat: `generate_stream`, `_generate_vision_response`, and `generate_audio_input_response`. The timer starts inside `_generation_lock` and after any adapter swap, so a queued request's wait is never billed as prefill. A rate is omitted rather than reported as zero when its window or token count is empty, so the UI keeps its client-side fallback instead of showing an invented speed. Frontend: the response details sheet had "Prompt eval" in milliseconds and a "Speed" row but no prompt rate, so add "Prompt speed" beside them. The message timing popover already rendered the row and needed no change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: wait for the device before stamping the prefill boundary Reaching the logits processor only means the prefill kernels were queued, so the boundary landed on dispatch time and the queued compute was charged to decode. Measured on an RTX 3080 with unsloth/Qwen3-0.6B and a 2048-token prompt: 27.8ms against 72.3ms once synchronized, a 2.65x overstated prompt speed. GenerationTimer now drains the device before each stamp. The prefill boundary reads the device off the scores tensor it is handed and latches it for finish(), which has no tensor of its own. A CPU run and a backend without a synchronize entry point stay no-ops. * Gate the prompt speed row and harden the device wait for PR #8647 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-08-14 05:55:06 -07:00
Maheswar Kumar	bb67053381	studio: open linked-folder sources in binary mode on windows (#8621) ... * open linked-folder sources in binary mode on windows folder_sync._snapshot opened each linked source with os.O_RDONLY and no O_BINARY, so the Windows CRT opened it in text mode. Reads then collapse CRLF to LF and stop at the first Ctrl-Z, while _copy_exact requires exactly st_size bytes. The short read raised "Linked source changed while it was copied", every file landed in the failure list, and the folder reported "N file(s) could not be indexed". That hit every format: CRLF text and markdown, and any Flate-compressed PDF or docx, where a 0x1A byte appears within the first few KB. A 225 KB 20-page PDF read back as 2 KB. Bulk upload was unaffected because it never goes through _snapshot. Python documents this: "on Windows adding O_BINARY is needed to open files in binary mode". backend/auth/storage.py already carries the same guard. Fixes #8617 * Fix linked-folder snapshots rejecting every file on Windows os.scandir leaves st_dev and st_ino at 0 on Windows, so the identity _scan records never matches the one os.fstat fills in during _snapshot and every file is rejected as changed during reconciliation. _scan already guards for a zero inode, so apply the same guard here and compare the post-copy stat against the pre-copy stat instead of the scan metadata. Also correct the O_BINARY comment: os.fdopen forces the descriptor binary before any read, so the flag guards a future raw os.read rather than fixing a live short-copy, and swap the flag-spy test for behaviour tests. * Recover a Windows file identity in the scan for PR #8621 --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-08-14 05:54:56 -07:00
Daniel Han	19bad46ed5	Studio: split the Full access rewrite claims per API, and drop the local-desktop promise (#8605) ... * Split the API coverage per rewrite, and drop the local-desktop promise _makedirs calls _remap and never the generic fallback, so the two rewrites do not cover the same APIs. Measured: os.makedirs under a missing absolute parent creates the real host path, while open() with the same shape lands in the working directory. Saying both reach the mkdir calls would have the model report a directory as sandboxed after it modified the host, and a successful mkdir prints nothing to correct it. The Git Bash branch of the shell note promises a detached program's window appears on the user's desktop. That was fine next to the old wording and contradicts the new sentence about a remote host or container, so the Full access variant points at that machine's desktop instead. The shell-selection guidance the note exists for is untouched. * Say the unrewritten mkdir attempts the host path, not that it creates it The clause promised that os.makedirs under a missing absolute parent really does create the directory. It is the rewriting that is absent, not the filesystem: measured under the same shim, makedirs into a mode-500 parent raised PermissionError and created nothing, on the host or in the workdir. Same standard the prefix branch already keeps, where /mnt/data is named only inside a conditional and never categorically. The new test carries that measurement rather than asserting os.makedirs semantics from memory. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Collect this module on Windows, and scope the mkdir clause to the prefixes it means A skipif argument is evaluated when the decorator is applied, so os.geteuid() ran at import on a platform that has no geteuid, before the Windows marker above it could skip anything. That took the whole module down at collection, all 44 tests, not just the POSIX-only one. Reproduced by deleting os.geteuid the way Windows lacks it: 1 error during collection, no tests collected. Guarded with hasattr, the same run collects 44. The mkdir clause was also flat where the shim is not. _makedirs calls _remap, and _remap's convention branch keeps the suffix under the working directory, so /mnt/data/reports has a missing absolute parent and still lands in the workdir. Measured under the real shim with no /mnt/data present: created ./reports, raised nothing, host path untouched, and the shim printed the redirect itself. A model told makedirs under a missing absolute parent is never rewritten would report the host path for a directory in its own working directory, which is what the closing sentence asks it not to do. The clause now names the prefixes, and a new test measures that branch rather than asserting the wording alone. Restoring the categorical wording fails four tests. studio/backend/tests/test_full_access_tool_prompt.py: 44 passed. --------- Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-08-14 05:49:08 -07:00
Daniel Han	ad96b66659	Wheel the voice model picker once its rows are on screen (#8743) ... * Wheel the voice model picker once its rows are on screen The macOS UI job fails on "Voice model picker did not wheel-scroll". The step fills the STT search with "whisper", waits for the results container to overflow, hovers it and sends one wheel event, then gives scrollTop 5s to move. The container that satisfies the overflow wait is not the one the wheel lands on. The query is debounced 300ms and the list is then swapped for a single spinner row for as long as the Hugging Face search takes, so the first paint that overflows is the pre-search default list. On the 3-core macOS runner the hover plus wheel arrives after that swap, into a container one row tall with nothing to scroll, and no further wheel is ever sent. Instrumenting the step on a macos-15 runner shows exactly that, and shows the widget itself is fine: A-after-overflow-wait: scrollTop=0 scrollHeight=285 clientHeight=256 rows=7 wheels=0 C-after-wheel-1: scrollTop=0 scrollHeight=1169 clientHeight=256 rows=43 wheels=1 E-after-wheel-2: scrollTop=700 scrollHeight=1169 clientHeight=256 rows=43 wheels=2 The wheel is delivered (wheels goes 0 -> 1) and useWheelScrollRef returns early because scrollHeight was not greater than clientHeight at that instant; a second wheel on the settled list moves scrollTop by exactly the 700px delta. Nothing to fix in the frontend. So gate the wheel on the state a user actually scrolls: rendered model rows (every row is a button, the loading and empty states are plain divs) plus overflow, and re-wheel until a 30s deadline so a swap landing mid-wheel costs a retry instead of the job. The assertion still requires a real mouse wheel to move scrollTop, and it is now stricter than before, since the pre-search default list no longer satisfies it. The failure message carries the container geometry so the next failure is attributable from one run. If the search returns no rows at all, which only happens on a runner that cannot reach Hugging Face, the query is cleared once and the wheel is asserted against the built-in list that ships in the bundle, so the scroll assertion does not become a network test. * Take the built-in list only when the Hub is provably unreachable The wheel step had a fallback that fired on the absence of searched rows alone: after 15s without them it cleared the query and asserted the wheel against the built-in curated list, which is the pre-search list the step was rewritten to stop accepting. A genuine break in Hugging Face search rendering therefore came out as "WARN no 'whisper' search rows" and a PASS, and the claim that the built-in list no longer satisfies the assertion did not hold in that branch. Two loopholes, closed together. The fallback now needs evidence that the Hub is what is missing: a failed huggingface.co request, a 429/5xx from it, or STUDIO_UI_HF_OFFLINE=1 on a runner whose egress is blackholed rather than refused. Requests are watched on the context, not the page, since the frontend backs off 30s after a failed Hub request and may not retry inside this step, and since a replacement page must stay covered. ERR_ABORTED counts only while the step holds the picker open on one query, because it is equally how a superseded query or an unmounting picker ends. With no such evidence the step fails with the container geometry and the reachability verdict. Rows plus overflow was also satisfiable by the built-in list itself, which is rows and overflows from the moment the popover opens, so a fast runner could clear the wait inside the 300ms debounce and never wheel a searched row. The built-in rows are snapshotted before the query is typed and the wait now requires the list to have become something else. Driving a local replica of SttModelPicker with headless Chromium, with the Hub answering 200 and the search rendering stubbed to stay empty: the step passed 5 of 5 before (0.3s through the debounce window, 17.8s through the cleared-search fallback) and fails 5 of 5 after, at 15.5s, with rows=0 scrollHeight=46 clientHeight=46 hub_unreachable=False. With huggingface.co refused (ERR_CONNECTION_FAILED), aborted at the frontend's own 15s search timeout, answering 429, or blackholed under STUDIO_UI_HF_OFFLINE=1, it still passes on the built-in list. With the wheel hook stubbed to a no-op it fails in both worlds, at 43 searched rows and at 7 built-in rows, so the fallback keeps its teeth. The pre-PR step still reproduces the CI failure verbatim on the same replica, TimeoutError('Page.wait_for_function: Timeout 5000ms exceeded.'), where this one passes. The 30s ceiling stays and is now derived rather than round: the searched rows are up to 15.3s away (300ms debounce plus the frontend's 15s HF_SEARCH_TIMEOUT_MS), the row wait is 15.5s so it clears that and also outlives the abort that ends a blackholed search, and the rest covers a 2s wheel retry and the fallback's own pass over the built-in list. Only a failing run pays it; a passing run still leaves on the first wheel. * Keep the picker wheel step honest when its row wait runs out Three holes, all in what the step concludes from a row wait that expired. A renderer crash inside that wait was swallowed by the broad handler around it. The geometry read then failed too, so the step recorded an ordinary "did not wheel-scroll" and the macOS crash recovery below it was never reached. The wheel wait already re-raises on page_crashed; the row wait now does the same. Hub-unreachable evidence was kept for the whole session and never dropped, so one transient failure earlier in the run let a later search that renders no rows clear the query and pass on the built-in list. A served huggingface.co response now clears it, which is what the frontend does with its own offline state (markRemoteNetworkOnline in studio/frontend/src/features/hub/lib/network.ts). The 15.5s row budget starts at fill, but the frontend starts its 15s search timeout at the debounced request, which leaves 200ms for the debounce callback, its render and the search effect. A starved runner with blackholed egress overruns that, so the wait gave up before the abort that permits the fallback and the step hard-failed. It now re-bases the budget onto the request while one is still open, counting only requests issued after the query was typed. Measured on a local replica of the picker, 8 runs per cell. Renderer crash during the row wait: 8 hard failures before, 8 crash recoveries after. Late debounce with a blackholed Hub: 8 failures before, 8 passes after. Stale failure with the Hub answering and rows that never render: 8 passes before, 8 failures after. Unchanged either way: on-time debounce with a blackholed Hub 8 passes, rows that never render with no stale failure 8 failures, healthy 8 passes, and useWheelScrollRef stubbed to a no-op behind the dialog scroll lock 8 failures. * Judge Hub connectivity by origin, and re-base the row budget per request Two things the wheel step got wrong about the Hugging Face requests it watches. The connectivity evidence was a substring test, so datasets-server.huggingface.co counted, and the training split lookup calls it. The frontend keys its backoff by exact origin, and the picker queries huggingface.co, so a failure at a sibling host says nothing about the search. Measured over seven urls the substring form is wrong on four of them: it accepts datasets-server, cdn-lfs and huggingface.co.attacker, and rejects a case-varied host. It now parses the url and compares the netloc. The budget re-basing was one-shot per step, but the picker searches twice in sequence, unsloth-owned then general. A slow but healthy first search spent the extension, and the second then opened with its own full 15s that the step could no longer wait for, so a healthy run could go red with most of that budget unused. The extension is now granted per request. The same request is still re-based once, not twice, so a single hanging search cannot keep buying time. * Let the re-based row budget outlast the one-search step deadline The model picker searches twice in sequence, and the wheel step's deadline was sized for one. Re-basing the row wait onto the second request therefore bought nothing: rows_ms is min()ed against what is left of that deadline, so the second search was cut off mid-flight and reported as a scroll failure while it was still inside its own frontend timeout. Push the deadline past the request just re-based onto, bounded by a ceiling measured from the step's start so a page that keeps opening requests cannot hold the step open. --------- Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-08-14 05:44:23 -07:00
Daniel Han	93225592c0	Tighten the comments on the versionless torch probe tests (#8811)	2026-08-14 05:40:53 -07:00
Nilay	083d613b95	Exclude studio/backend/tests from the wheel (#8501) ... * exclude studio/backend/tests from the wheel * Exclude nested studio backend test packages from the wheel * Actually keep the backend test suites out of the wheel Dropping studio.backend.tests from packages.find only stops setuptools treating it as an importable package. include-package-data then hands every tracked file to the nearest parent package that survived, so the 505 files under studio/backend/tests plus the 11 under studio/backend/hub/tests came straight back as package data of studio.backend and the wheel was byte for byte the size it was before (43.95 MB, 2708 entries, both refs identical). exclude-package-data has the highest precedence of the file selection options, so repeat the veto there. Measured: 43.95 MB / 2708 entries -> 40.71 MB / 2192 entries, and the 516 entries removed are exactly the two test trees, nothing else. The patterns are source layout relative, so the sdist is unchanged (3666 entries either way) and editable installs still see the suites. Extend test_package_discovery.py so this cannot regress: it now replays setuptools' nearest-parent-package rule over the tracked file list and asserts no backend test file would ship, and checks a built wheel directly when dist/ has one. --------- Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-08-14 05:36:50 -07:00
Daniel Han	70a0f1a1b3	Studio: gate --flash-attn, --no-context-shift and --jinja on the binary (#8710) ... * Studio: gate --flash-attn, --no-context-shift and --jinja on the binary probe_server_capabilities checks 17 optional flags with _is_real before emitting them. These three were emitted unconditionally. That is fine for the pinned prebuilt, which has all three. It is not fine for a stale or user-supplied LLAMA_SERVER_PATH, which Studio supports on purpose: an unknown argument makes llama-server exit immediately rather than start degraded, and the user gets a generic startup failure with no indication which flag was at fault. --flash-attn is the sharp one. Newer llama.cpp declares it with an enum, "-fa, --flash-attn [on|off|auto]"; older builds take a bare boolean and read the following "on" as a stray positional. So the value, not the flag, is what an older binary rejects, and only the value is conditional here. Dropping -fa entirely on a build that does have it would be a silent performance regression, which is the worse trade of the two. All three gates FAIL OPEN, unlike the other 17. They are part of today's command on every launch, so an unreadable or unparseable --help has to keep emitting them; only a build whose help positively lacks one drops it. That is enforced in three places: the unprobeable-binary dict, the pre-probe initialisers, and the .get() defaults at the emission sites. An empty `blocks` is treated as "the probe told us nothing", not "the flag is missing". One existing test changed. test_flag_sits_inside_the_base_cmd_list pinned --no-context-shift as a literal inside the base cmd list, which is exactly what this moves. It is replaced by a test of the new contract: the flag is emitted, the gate defaults to True, and both probe defaults are True. The sibling assertion that -c stays grouped with the base list is kept. Worth flagging for review: the value-form detection is the one part I could not verify against a real older binary, only against both help spellings. It fails open in every uncertain case, so the pinned prebuilt is unaffected, but a second opinion from anyone with an old llama-server to hand would be welcome. Tests: 20 new across two files. 1938 pass in the llama.cpp, capability and launch-flag suites. * Apply the repo's kwarg-spacing format to the new tests * Require a successful probe before dropping a flag, and drop a bare --flash-attn on the retry Two follow-ups flagged in review on the capability gate. A partial help listing from a binary that exits nonzero is still a failed probe. The gate keyed on a non-empty `blocks`, so a wrapper that printed a few options and then exited nonzero made the fail-open defaults authoritative for everything absent from the fragment, silently dropping --no-context-shift (context rotates again) or --jinja (no template rendering). The gate now also requires probe_ok; a real llama-server prints usage and exits 0, so no supported build is affected. The flash-attn crash recovery rewrote a bare --flash-attn to --flash-attn=off. llama.cpp looks argv tokens up verbatim in arg_to_options after only an underscore-to-dash rewrite, so it never splits on '=' on any build, and the bare form only exists on builds whose flag takes no value at all. The retry now drops the token instead, and drops an inherited LLAMA_ARG_FLASH_ATTN with it, since the env is applied before argv and argv can no longer say off. The launched flash-attn state is recorded from that decision rather than inferred from an argv that no longer mentions the flag. Tests are now behavioural: the probe runs against a stubbed --help rendered the way common_arg::to_string does, over exit codes, stderr, CRLF and the pre-enum declaration. * Gate the flash-attention flag itself, not just its value Flagged in review. A build that predates flash attention returns parseable help with no --flash-attn in it, and the gate only decided whether to append a value, so the flag itself still went out and the binary exited on an unknown argument. llama.cpp gained -fa in b2775; anything older has none. supports_flash_attn now answers whether the flag exists, separately from flash_attn_takes_value answering whether its declaration takes one. Both fail open, so an unreadable --help still emits --flash-attn on exactly as before, and only an authoritative listing may drop it. There is no speed to protect in that case: the build the probe just read has no flash attention to lose. The launched flash-attention state follows the same decision, so the slot-save KV estimate does not assume a kernel the build never had. * Drop an inherited LLAMA_ARG_FLASH_ATTN when the build has no --flash-attn llama.cpp resolves each env var through the argument that declares it, so a build whose help never mentions --flash-attn never reads LLAMA_ARG_FLASH_ATTN either. An inherited one is therefore dead to the server but not to us: the accounting reads env plus argv, and would record flash attention as on for a server running without it, which under-sizes the padded V cache. * Reset a quantized V cache when the build has no --flash-attn Dropping the flag on a build whose help lacks it left the KV cache type emitted from the user's setting untouched, and llama.cpp aborts init with "V cache quantization requires flash_attn". The crash-recovery rung already resets V for that abort, but _with_flash_attn_off returns None when the argv has no flash-attn flag to turn off, so on such a build nothing caught it. Reuse that same reset at the launch site, and drop an inherited quantized V-cache env var alongside the flash-attn one. K is left alone: it has no flash-attention dependency and resetting it would enlarge the K cache. * Reset the V cache before the launch is logged The reset sat after the "Starting llama-server" line, so the log named a cache type the child never ran. Move the argv reset above it and leave only the env-only drop in the environment block, which needs env to exist first. The rewrite preserves the list length, so _spec_start still indexes the same tokens.	2026-08-14 05:35:53 -07:00
Maheswar Kumar	753ed31209	studio: give minimax-h3 the sd.cpp graph-cut flags so it renders on a smaller card (#8644) ... * studio: give minimax-h3 the sd.cpp graph-cut flags so it renders on a smaller card H3 allocates each module whole on the device when its graph runs, so --offload-to-cpu is not enough on its own: the 20.5 GB pruned denoiser and the 17 GB Qwen3-VL encoder ask for the full module at once and cudaMalloc fails on a 12 GB card. The abort lands in the conditioner before any tensor is resident, so VRAM never rises and it presents as a model that never loaded rather than an out-of-memory error. sd.cpp's graph-cut executor is what splits a module into segments that fit, and Studio never asked for it. The H3 native path now adds --max-vram -1 --stream-layers. A negative budget auto-detects free VRAM per device, sparing that many GiB, and a module that already fits merges back into a single segment, so a setup with room pays nothing for this. It is deliberately not gated on the memory mode: auto and fast are what users reach first and are exactly the modes that fail. sd_cpp_supports_graph_cut reads both options out of --help and fails closed, the opposite default to the H3 gate next to it. sd-cli exits non-zero on an option it does not know, so an unreadable probe has to emit nothing rather than break a build that renders today. The CPU runtime never asks, since it allocates from system RAM. Part of #8636. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * carry the sd.cpp project banner in the h3 help fixtures The identity gate added in #8560 reads --help for a stable-diffusion.cpp banner before the H3 capability marker, so these fixtures need one to reach the load path they exercise. * Gate --stream-layers on --offload-to-cpu for PR #8644 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-08-14 05:27:39 -07:00
Maheswar Kumar	57e4bdf2fa	fix grouped_gemm crash on the documented gather_indices=None default (#8629) ... * fix grouped_gemm crash on the documented gather_indices=None default grouped_gemm() defaults gather_indices to None and only asserts it is present when permute_x or permute_y is set, but it then normalised it with an unconditional gather_indices.view(-1). Passing the documented default with both permutes off raised AttributeError: 'NoneType' object has no attribute 'view'. grouped_gemm_dX had the same unconditional dereference, gather_indices.shape[0], used to size dX, so fixing only the wrapper moved the identical crash into the backward pass. grouped_gemm_forward and grouped_gemm_dW were already guarded. None of the three Triton kernels read gather_indices_ptr outside their PERMUTE_X or PERMUTE_Y branches, so callers passing activations already in expert-contiguous order no longer need a dummy torch.arange(total_tokens) the kernel never reads. That is a documented use of permute_x=False. Verified on an RTX 3080 (sm_86) with torch 2.13.0+cu130 and triton 3.7.1: forward and backward with gather_indices=None are bit-identical to the torch.arange workaround and exact against a reference matmul. Fixes #8627 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Parametrise the gather_indices=None backward test on topk for PR #8629 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-08-14 05:27:25 -07:00
Daniel Han	02f63cd419	Carry a provider-run tool's result into the next turn (#8713) ... * Studio: carry a provider-run tool's result into the next turn Hosted and local tools coexist in one turn: Gemini can return a code-execution result while asking for a local web_search, and OpenAI can generate an image before requesting one. The hosted output reached the client as its own _toolEvent frame, but the loop rebuilds the assistant message from the text and tool calls it saw, so that output was absent from the conversation replayed on the follow-up request. The model answered from the local results alone, having lost what it had just produced. Recorded per call id, so a repeated end event cannot duplicate it, and replayed as text rather than as native items: the native shape differs per provider (Gemini codeExecutionResult, an OpenAI image call) while every provider can read its own prior turn's prose. A result that is missing, blank, not a string, or has no call id is ignored rather than trusted. Studio's own tool events are written with a top-level type and never appear as _toolEvent, so local results cannot be replayed twice by this. * Replay a hosted call's operation, strip its frontend sentinels, note its image Three gaps in the first cut, all found in review. The tool_end producers generally omit tool_name, and for Gemini code execution the code that ran is only ever in the tool_start arguments, so a result recorded on its own replayed as an unlabelled value the model could not interpret. Both halves of the call are recorded now and the operation labels its result. A hosted result can carry a full base64 data URI after the __IMAGES__ sentinel, which is there for the card rather than the model. Replaying it verbatim would have put megabytes into the next request for something the model cannot read, so hosted text now goes through the same strip_result_for_model the local results already use. image_generation reports an empty result and carries the picture in image_b64, so requiring non-empty text dropped every generated image, which is exactly the mixed hosted-and-local case this is meant to fix. That it happened is recorded without the bytes, for the same reason the sentinel is stripped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cap hosted output, keep it across a stall, and notice a stdout-less plot Three more from review, all in the replay added on this branch. Gemini code execution that produces a plot and nothing else returns a result that is only the __IMAGES__ sentinel, and does not set image_b64. Stripping it left an empty string with no image marker, so the entry looked empty and the follow-up was told nothing had been produced. The sentinel is noticed before the strip now. Hosted results went into the next prompt uncapped, while local execution has always held what the model sees to 16000 characters. One verbose hosted call could fill the follow-up context on its own, so the replayed copy is held to the same limit. A turn where a hosted tool completed but the model only said what it was about to do takes the stall reprompt, which returns to the provider from above the replay. The reprompted request was therefore told to continue from a search whose output it could no longer see. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the stalled replay in one turn, label the operation, honour the configured cap The stall reprompt appended its hosted block as a new assistant message, so a continued partial was split across two assistant turns. It now merges into the partial the same way the main replay does. The header rendered the whole arguments dict, which carries Gemini's native executableCode part with its thoughtSignature and OpenAI's paired reasoning item with its encrypted content. Those are replay plumbing for the provider, not text a model can read, and they filled the header's budget with truncated base64. The image generation prompt is also only on the end event, so the two halves are merged rather than read from the start alone. The hosted result now goes through the stripper with its own tool name, as local results do, so a fetched page ending in a well formed __FILES__ line keeps it, and the cap is read from the configured local one instead of a copy of its default. * Report a hosted call that ran and printed nothing, and say when its label was cut Gemini reports code that produced no stdout as an empty result, and the code it ran is only ever on the tool_start, so an entry skipped for having no result took the whole execution with it: the next turn saw no trace that anything had run. It now records that the call ended and replays it as (no output), the same answer the local tools and the other hosted paths give. A start with no end is still left out, and a non-string result is still a malformed frame rather than an empty outcome. The label was also cut at 2000 characters with no notice. Anthropic passes the model's whole tool input through as the arguments, so a created file lives there and answers with only Created, and a silent cut reads as a line that simply stops. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the turn's thought signature into the stalled replay Gemini 3 stows a text part's thoughtSignature on the delta, the outbound translator pins it back onto the last text part from assistant.extra_content and nowhere else, and a turn replayed without it is rejected rather than answered. The main replay already carries it; the stall reprompt returns to the provider from above that and did not. * Tighten comments in hosted result replay * Keep an argument the model meant, and read the image sentinel as an envelope Dropping empty values took Anthropic's str_replace deletions with it: new_str of the empty string is the deletion, the schema allows it, and without the key the next turn cannot tell it from a value that was never captured. The guard is not needed anyway, since the provisional empty prompt OpenAI opens an image generation with is overwritten when the two halves merge. The image sentinel is now validated the way the local strippers validate theirs, so a fetched page that merely writes the marker is prose rather than a picture the turn never produced. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Assert on the snippet rather than the URL in the replay tests Checking that a bare URL appears in a string reads to the scanner as URL sanitization by substring, which is the one thing that check exists to flag. The fixture now carries a distinctive snippet and the assertions look for that, so they prove the same thing without the pattern. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-08-14 05:22:45 -07:00
Daniel Han	62b12f0065	Studio: run the Transformers dictation engine in a spawn child (#8668) ... * Studio: run the Transformers dictation engine in a spawn child The Transformers STT engine loaded Whisper in the FastAPI backend process, so the first dictation left a CUDA context there that no unload can return. Measured on a B200 with whisper-small: the backend went 0 -> 1110 -> 1412 MiB and settled at 850 MiB after unload, with memory_allocated at 9.9 MiB, so nearly all of the residue is the context and the allocator floor. whisper-tiny leaves the same 814 MiB, which confirms it is per process, not per model. The whisper.cpp and mtmd engines already run out of process; this brings the Transformers engine into line, following the chat worker's spawn pattern. The backend now stays at 0 MiB across load, transcribe and unload, and the child takes its context with it when the idle timer ends it. Device selection, dtype, the CPU retry, the Hub cache and download worker, PyAV decoding, 30s windowing, the idle timer, cancellation and unload semantics all stay where they were. Only from_pretrained and generate moved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Close a worker rejected by a late cancel, and do not restart the shutdown wait for PR #8668 Two leaks in the spawn-child path, both found in review. A cancel that lands after start() returned dropped the only handle to a live child (candidate = None), and _release_engine_locked closed self._engine, which was already None. The child kept the model and its CUDA context. Closing the rejected candidate is now shared with _release_engine_locked via _close_engine. After the 10s cancel grace expired, close() began a fresh 10s join on a child that can no longer read a shutdown message, so wait_for_load_to_settle blocked about 20s rather than the documented 10s. close() takes graceful_timeout, and the cancel path passes 0 to go straight to terminate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep dictation on a host that cannot spawn, and do not forget a live worker Two fixes for PR #8668. A host that forbids spawn (a sandbox, or a frozen POSIX build) had working dictation before the engine moved out of process, and every failure to start the child came back as HTTP 500 after it. WhisperWorker.start now reports a refusal to spawn as SttWorkerSpawnError, and _build_model answers that by loading in the backend on the CPU, which takes no accelerator context. The fallback only engages on the CPU attempt, so a spawn failure on an accelerator still goes through the existing GPU-to-CPU retry first. WhisperWorker.close forgot the child's pid and dropped its handle without checking the child had died, so a worker wedged in a driver call kept its VRAM while unload reported success and no sweep could find it again. close now returns True only once the process is confirmed dead, and keeps the handle and the adoption record otherwise, matching the chat worker's _shutdown_subprocess. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep a worker that outlived the kill resident, and fall back when a child never starts close() already answers False for a child that survived terminate and kill, but the sidecar threw that answer away: _release_engine_locked cleared the engine either way, so loaded_model reported nothing resident while the child still held its accelerator memory, and the next load started a second one over it. Propagate the failed close, keep the survivor resident with the idle timer rearmed so the release is retried, and refuse a replacement load until its child is confirmed dead. Process.start() returning also says only that the exec worked. A frozen POSIX build re-runs its own binary instead of an interpreter, so the child is gone before it can read the load command and the parent raised a plain SttWorkerError; the accelerator attempt then retried another identical child on the CPU and the in-process fallback was never reached. A child that answered nothing and exited with a positive code never got its loop running, so report that as a host that cannot spawn. A signal death and any child that did answer keep their own error. * Keep a late-cancelled worker that outlived the kill accounted for PR #8668 The cancel that lands after start() came back closes the candidate, but a child wedged in a driver call outlives terminate and kill, and close() answers False for exactly that. The result was dropped and the candidate cleared, so the sidecar reported nothing resident while ~850 MiB was still held, and training admission, which reads loaded_model, was free to launch into it. That is the case _release_engine_locked already refuses to create for the resident engine. Keep the survivor instead, on the device the child itself reports, and rearm the idle timer so the release is retried rather than the handle stranded. One close: a second terminate-and-kill round on the way out would only spend another wait on a child that just proved it answers neither. * Keep a worker that may still hold its memory accounted for PR #8668 An unanswerable liveness probe now reads as resident, a close() that raises counts as a failed release unless the child is confirmed dead, and the resident fields are cleared only once the worker has exited rather than before the close, so summarize_resident_stt() cannot report nothing while the child still holds its accelerator context. * Preserve cancellation when the command timeout expires for PR #8668 A cancel observed shortly before the load or transcribe deadline used to reach the older absolute timeout first, raising a generic worker error and taking a second full shutdown wait. Once cancellation has been seen, terminate and raise the phase's own cancellation instead, so a cancelled load still maps to 409 and a client disconnect to 499. * Have the dictation child say it is ready before it loads for PR #8668 A child that dies without answering is read as a host that cannot spawn, which the caller answers by loading in the backend instead. On Windows there are no signals, so a native fault inside the model load ends the child with a positive exit code exactly as a child that never bootstrapped does, and that crash was answered by repeating the same native load inside the backend process the user is talking to. The child now answers ready before it touches a command, so the classification rests on whether a fresh interpreter ever came up rather than on an exit code that cannot tell the two apart. * Keep a child that outlived the kill inside start() accounted for PR #8668 start() ends its own child on every failure, so a handle that still reports a live process outlived terminate and kill and holds its memory yet. That handle was dropped: _build_model only ever returns a worker on success, so the load that failed left the process known to nobody, the sidecar reported nothing resident, and training was admitted against memory that is not free. The cancel that lands while the child is inside from_pretrained is the ordinary way to reach it. The handle is now handed back to the load, which keeps it exactly as a release that could not kill its worker does, and an accelerator attempt that leaves one is refused rather than retried onto a second child that would sit beside the first and then be installed over it. A worker held that way is held for its memory, not for its answers: it was already asked to shut down, terminated and killed. Hand it to the next dictation and the caller waits out the whole command timeout under the model lock, so it is flagged and a later load starts one of its own, or is turned away with the retryable refusal a busy switch already gets. * Signal dictation child readiness with an event, not a queued word, for PR #8668 * Retire a dictation worker that outlived a cancelled command for PR #8668 * Name the kill signal per platform in the dictation worker tests for PR #8668 * Tighten the dictation worker comments for PR #8668 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 05:12:54 -07:00
oobabooga	eb869b856c	Studio: optimize startup with on-demand language catalogs (#8623) ... * Studio: optimize startup with on-demand language catalogs * Studio: preserve pending locale intent during lazy loads * Studio: make locale hydration cancellable * Studio: a failed locale catalog must not disable the rest of personalization sync Two problems in the same six lines of the hydration effect. The early return on a failed catalog exits before setHydratedGeneration(generation) at the end of the same block, and the outbound save effect gates on exactly that value while the hydration effect only re-runs on [enabled]. So the gate never opened again for the session. The comment frames this as pausing the language, but payload() also carries profile, avatar, theme, palette and the whole sidebar customization, none of which was pushed afterwards, with nothing surfacing it to the user. It is reachable: /assets is served immutable-cached while index.html is no-store, so a tab left open across an in-place update asks for a hashed catalog chunk the new build no longer has, gets a 404, and the fetch rejects. Second, 'superseded' was treated as success. If the remote language load is superseded by a user picking another language, and that newer request then fails, the hook recorded the remote language as the synchronized baseline and the debounced save pushed the stale local preference back over the server's. setLocale takes an adoptOnFailure option, used only by hydration, that routes a failed load through commitFallbackLocale: the preference is adopted and the messages fall back to English. The preference hydration applies is already the server's stored truth, so adopting it keeps the local value equal to the remote one and the baseline honest, while a user's own failed pick is still not adopted and still not persisted. Neither path writes storage on failure, so nothing records a choice that did not work. Also fixes a smaller one found alongside: applyPreference's synchronous-throw path called failPreference before this request had become the pending one, clearing the marker an unrelated in-flight request was relying on and blanking its spinner. * Finish hydration when a locale load is superseded, and keep a failed locale retryable for PR #8623 * Keep auto detection retryable when its catalog fails for PR #8623 Showing the in-effect locale only when it differs from the preference left auto as the one preference a user could not retry: a failed auto still reads as auto, and its fallback locale is indistinguishable from a successful detection that landed on the same language. Radix holds auto, so picking Auto-detect again fires nothing and the app looks like it ignored the click. The store now records why the preference and the locale diverge. commitFallbackLocale sets the flag, commitPreference clears it, and failPreference sets it only when the standing preference was refreshed onto a locale the app is not showing: a rejected pick leaves the standing preference serving its own catalog and must stay named. The failed chunk is not negatively cached (messages.ts deletes the in-flight entry in .finally), so a re-pick genuinely re-issues the import. * Bound locale hydration and make a failed catalog retry re-request for PR #8623 Two things a stalled or failed catalog could still do. A catalog request that is accepted and then never completes rejects nothing, so personalization hydration awaited setLocale forever: the save gate never opened again and every later edit to display name, nickname, avatar, theme, palette, sidebar or greeting sloth stayed local for the session. setLocale now takes the same bounded fallback initializeLocale uses at startup, and hydration asks for it; a late catalog still commits over the fallback. And the retry itself did not re-request. Chrome, Edge and Firefox before 155 keep a failed module in the module map keyed by URL, so dropping our own promise only made the store willing to ask again. A retry now re-imports the chunk with a one-off query, taken from the URL the browser reports in the failure. Only a retry carries it, so the first load of a hashed catalog keeps its normal caching. * Bound a language pick that names no timeout of its own for PR #8623 A catalog request that is accepted and then never completes rejects nothing, so the language menu, a storage event and a languagechange refresh all had nothing to settle them: the pending marker stayed set for the rest of the session, the menu spun on a language it would never reach, and because a never-settling load also never runs the finally that clears the in-flight map, every later pick of that language was handed the same dead promise. Reloading the app was the only way out. Startup and hydration already pass their own bound, so give every other path the store's. It is longer than startup's, because this one is a deliberate download rather than the render everything else is waiting on, and the load is still left alone, so a slow catalog commits over the fallback rather than being lost. Before the lazy catalogs every language was in the bundle and a pick could not wait on the network at all, so this is the one shape that was strictly worse than the old behaviour. * Evict a timed out catalog load before retrying it for PR #8623 A load that never settles never clears its own entry in the in-flight map, so the pick that timed out kept being handed back to every later attempt and the language could not be retried without reloading the app. The selection timeout now forgets the load it gave up on, identified by promise so a newer attempt is never evicted, while leaving the load itself running so a late catalog still commits. * Evict a timed out startup catalog so the next pick re-requests it for PR #8623 * Adopt a cross-tab language whose catalog fails and keep the fallback selectable for PR #8623 A storage event carries a preference another tab has already written to the shared store, so treat it like hydration: adopt it even when its catalog will not load here. Refusing it left this tab holding the preference the user had just replaced, disagreeing with storage until a reload, and the next personalization save uploaded that stale language over the cross-tab choice. A controlled Select never fires onValueChange for the value it already holds, so whichever entry the language menu names is the one entry that cannot be picked. After a catalog failure both are needed: the failed preference is the retry, and the fallback in effect is how a user accepts English as a real preference instead of retrying a chunk that will not load. Name neither, which is what an empty value means to Radix, and show the language in effect as the placeholder. --------- Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-08-14 05:12:33 -07:00
Daniel Han	a0c1723bd1	Studio tests: pin the empty-version torch probe distinction at the repair paths (#8803) ... * Studio tests: pin the empty-version torch probe distinction at the repair paths _probe_torch_runtime draws a deliberate line between "" and None: an empty __version__ is a torch the pins repair, while None is a probe that told us nothing and must leave the venv alone. TestProbeParsing covers that at the probe. Nothing covered it where it decides something. That gap matters because the per-path probes classified inside the subprocess, from torch.version.cuda and .hip as well as the version string, so a consumer that gates on the version alone silently stops repairing a wheel whose runtime fields say plainly what it is. Collapsing the distinction to `if not _version` makes three of these four fail: a CPU pin stops replacing a versionless CUDA or ROCm build, and an XPU pin stops repairing an unidentifiable one. The fourth records the other side, where the two readings agree. Test-only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 05:09:30 -07:00
Daniel Han	d50f49e105	Ask for the transformers a training model needs, before the run starts (#8721) ... * Ask for the transformers a training model needs, before the run starts Muse Glimmer needs a transformers release no installed version ships. Chat already handles this: /validate reports requires_transformers_upgrade, and the composer offers to install the latest sidecar before loading. Training never asked. Starting a Muse Glimmer run went straight to the loader and died there with an architecture the installed transformers does not know. Training now runs the same consent step chat does, before the custom code gate, because installing a newer transformers changes what the load would run. Two things the Configure preview got wrong and now discloses: - A model no installed transformers ships reads as ready to train. It is not; the run stops on a consent dialog first. - The latest sidecar loads 16-bit only, so a run that installs it cannot honour a 4-bit request. A "QLoRA, 4-bit" preview understated VRAM by roughly threefold. The preview now says 4-bit is unavailable for that model. /validate grows forces_16bit so the frontend does not have to re-derive the sidecar's rule, and resume runs go through the same gate as fresh ones. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Ask the upgrade gate about the model the run actually loads Three things the gate got wrong. A resume can be attested against a 4-bit model load that the latest sidecar permanently refuses: effective_training_load_in_4bit raises ExactResumeResourcesUnavailable for it the moment latest_tier_active_for turns true, and that sidecar is a persistent overlay. Accepting Install from the new resume gate therefore stranded a checkpoint that would otherwise have resumed. The stored config cannot say so on its own (_sanitize_db_config strips require_exact_resume_resources and require_exact_model_resource before the row is written), so the resume names its run on /transformers-upgrade-check and the backend recomputes the requirement from the provenance marker. With a custom-code fallback the resume loads on the current runtime in the 4-bit mode it needs, so no install is offered; without one the install is the only way the run starts at all, and the dialog is raised as before. The check was handed the Hub identifier for a cached model while the remote-code gate and the worker both open the pinned snapshot, so a repo whose current config.json names a different architecture than the snapshot on disk was inspected instead of the model being loaded. The check now takes the same four cache-pin fields as /models/remote-code-scan and resolves them by that route's precedence, and each start path resolves the pin once for both gates. A merely offered upgrade no longer claims 16-bit when the model can load through its own repo code: the dialog offers that way out, taking it installs nothing, and the run loads bnb 4-bit. /validate already exempts this case. * Answer the upgrade gate for the load the run really performs Three follow-ups on the training transformers-upgrade preflight. Resolve a LoRA's base from the load target, not the Hub identifier. The worker reads the base out of the adapter config it opens, and so does the scan route; this route still read the repo's current one, so a pin taken before the repo repointed base_model_name_or_path had the upgrade, custom-code and precision answers describing a base the run never loads. Stop offering an install that cannot rescue an exact 4-bit resume. Once the sidecar is active, effective_training_load_in_4bit raises for exactly the config the backend answered install_breaks_exact_resume with, so the resume fails with or without the install and consent buys only a persistent overlay that also retires 4-bit for later runs on that model. The start is now refused with the reason instead. A model with custom code keeps the way out it had. Re-ask the Configure preview after an install. The notice cache outlived the sidecar it described, so returning to Configure in the same session kept offering a release that was already installed and kept previewing 4-bit for a run the new overlay loads in 16-bit. The dialog store now counts completed installs and the cache keys on that count. Declining a dev-only upgrade no longer says to start the run again to install it: that dialog has no install action, so the instruction could never be carried out. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Let the test project see the ambient globals it now reaches The training upgrade gate hands the chat feature a dynamic import, so the tests that drive the gate pull src/features/chat into the test project for the first time, and with it the Web Speech dictation adapter. That adapter relies on the ambient declarations in src/speech-recognition.d.ts, which nothing imports, so "include": ["tests"] never reached them and the Typecheck step failed with five Cannot find name errors in a file this branch does not touch. The app project only avoids it by including all of src. Add the ambient declarations to the test project's include. * Pin an implicit cache selection, and carry the custom-code verdict Two more follow-ups on the training upgrade preflight. Take the cache pin whenever prefer_local_cache is set, not only when a path came with it. A cached inventory row can carry a null cachePath, and the Train tab still sends the flag for it, so _resolve_model_snapshot has to search the cache roots exactly as routes/models.py and /train/start already do with it. Requiring a path left those selections judged on the repo's current architecture while the worker loads the snapshot. The Configure preview now resolves the same way, so the flag travels on its own rather than being read back off the path, and the preview cache keys on it. Carry the upgrade check's custom-code verdict into the custom-code gate. confirmRemoteCodeIfNeeded falls back to the caller's requiresTrustRemoteCode when the scan request itself fails, and the training callers were handing it the stored config flag, which a fresh run leaves false. For a model that needs its own modeling code the fallback then skipped consent and started a worker with trust_remote_code off, which cannot load it. The check has already answered that question, so the outcome now reports it and both start paths pass it on, the way chat passes its validate result. * Wait for the in-flight support snapshot instead of answering no upgrade The Configure preview starts its own transformers-upgrade check the moment the tab renders. On a cold snapshot that check fetches PyPI and the transformers config mapping, and a user who presses Start before it lands sends a second, concurrent check. _get_snapshot deduplicated those by handing the loser None, which reads as "no upgrade needed" through latest_transformers_supports and check_upgrade_for_model, so the start gate proceeded and the run launched on a model no installed transformers can load. The loser now waits on the running fetch and returns its snapshot, bounded at 20s so a request can never sit for a refresh's worst case; past that it falls through to the same graceful answer a failed refresh gives. Still exactly one fetch in flight. * Answer the upgrade gate from the fetch, the release and the live generation Four ways the gate still answered something other than what it knows. The wait a concurrent caller makes for an in-flight support snapshot was a flat 20 seconds, but a refresh is five sequential URLs, each allowed one retry at the five-second fetch timeout, so it can legitimately run for fifty. A wait that expires mid-fetch answers None, and None reads as "no upgrade needed" all the way up to Start, which launches the run on the architecture the gate exists to stop. The bound is now derived from the refresh's own budget, so tuning a timeout or a retry cannot silently reopen it. The consent dialog offered "Continue with custom code" only once an install had failed. Training raises that dialog before a run starts, so for a model that ships its own modeling code the first dialog was Install or Cancel, and Install activates the latest sidecar, which trains 16-bit. The 4-bit run the user asked for had no way to start. The fallback is now offered next to Install whenever it exists, out of one shared decision the dialog and its tests both read. install_breaks_exact_resume is answered from the run's own provenance, without regard to whether any release exists to install, so a resume on a dev-only architecture was told that installing would strand it and to start a new run instead: nothing can be installed, and a new run cannot load either. That branch now applies only to an installable release, and the dev-only path explains the real situation. The Configure preview's notice cache keyed on the sidecar generation but let any mismatch redefine it. A check fired before an install can still be in flight when it lands and resolve after the post-install check has answered; its write rewound the generation, cleared the fresh entry and stored the stale one, and since the effect behind it was already cleaned up nothing re-rendered and nothing re-asked. Superseded reads and writes are now ignored. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bound the refresh transfer, and disclose the 16-bit Install urlopen's timeout is a socket timeout: CPython documents it as "a timeout in seconds for blocking operations like the connection attempt", so it bounds each individual read, never the whole transfer. A response that dribbles bytes just inside it therefore never expires -- measured here at 12.0s under timeout=5.0 -- which made every bound derived from that timeout an underestimate, the in-flight wait included. A loser whose wait expired mid-refresh answered None, and None reads as "no upgrade needed" all the way up to Start, so the run launched on the architecture this gate exists to stop. So the transfer gets its own wall-clock budget, checked between read1 calls, and the per-attempt worst case is derived from that budget plus the one socket read still blocking when it runs out. The in-flight wait is derived from the attempt, and is now only a backstop: a waiter re-waits while that same refresh is genuinely still running, since the winner clears the busy flag and sets the event in one locked finally. Separately, the Configure preview reported one precision for a run whose precision depends on which action the user takes. A model that both ships its own modeling code and is shipped by the offered release gets both dialog actions: keeping the custom code loads it on the current transformers in 4-bit, which is what forces_16bit answers for, while Install activates the latest sidecar, which trains 16-bit. The card offered that install next to an unqualified "QLoRA - 4-bit", understating the VRAM of the run by roughly threefold if the user picked it. An offered install the backend does not already call 16-bit is exactly that case and nothing else, so the card now names the choice. * Tighten the comments across the upgrade gate for PR #8721 Same explanations, fewer lines. The comments had grown to the point of restating the code around them and repeating the same VRAM argument in five places; each one now makes its point once. Comments and docstrings only, verified per file with an AST comparison of the code either side of the edit (24/24 unchanged). typecheck clean, ruff clean, 37 frontend tests and 197 backend tests pass. * Pin the compatibility guarantees for PR #8721 with regression tests The existing tests prove the gate fires. These pin the opposite property, which is what decides whether merging is safe: everything that worked before it has to behave exactly as it did. Backend, test_training_transformers_upgrade_gate.py: - an old client sends model_name alone, and a minimal response reads as the pre-gate behaviour, so a bundle newer than its backend still starts - forces_16bit over all eight combinations of sidecar, installable release and custom code, since the preview draws its VRAM claim from it - four failure types out of the preflights, none of which may escape a route that now runs in front of every start - the route stays behind authentication Backend, test_transformers_latest.py, which chat reaches through /validate: - ordinary responses, plain and chunked, come back whole - a multi-chunk body is not truncated by the transfer budget - a 404 stays distinguishable from a failure, since auto_mappings.py does not exist on pre-5.10 tags - a truncated source fails the lookup rather than caching a short map that would offer an upgrade to every model missing from it Frontend, training-transformers-upgrade.test.ts: - 404, 405, 500 and a network failure each leave the start untouched and raise no dialog - a partially populated upgrade is never offered as an install - a field from a newer backend is ignored rather than fatal Measured while writing these: against a hostile mirror the snapshot refresh returns in 20s where the derived bound is 150s, a second start after a failure costs 0.000s behind the 300s backoff, and 50 offline checks take 0.1ms. The Configure notice and the consent dialog render identically in Chromium, Firefox and WebKit with no console errors. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the sidecar guard on every target, and drop a redundant bool() Two failures CI caught that the local suites did not, both introduced by this branch and both passing on main. The offline guard on latest_tier_active_for was keyed on the bare name targets. The guard argument is what the call READS, and that function resolves a remote adapter's base itself, so the whole tuple is the honest key; the repo's other two call sites pass a tuple literal for the same reason. tests/test_offline_gguf_cache_fallback.py asserts it. requires_trust_remote_code was re-wrapped in bool() when constructing the response. The value is already a bool, and the literal text reads as the raw-YAML pattern the GGUF security-gate consistency test forbids. Dropped there and on the two beside it. 218 gate/provenance/latest tests pass, ruff clean. * Tighten the comments added with the regression tests Final pass over what landed after the last one. Same points, fewer lines. Comments only, AST-verified (3/3 code unchanged). 125 backend and 20 frontend tests pass, ruff clean. * Drop an incomplete regex escape in the training upgrade test for PR #8721 --------- Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 04:59:56 -07:00
Daniel Han	8fef5ed221	Read the descender guard's class list wherever it is written (#8765) ... * Read the descender guard's class list wherever it is written Repo tests (CPU) is red on main: test_model_selector_trigger_label_uses_leading_tight fails with "could not find ModelSelectorTrigger model-name span". The component is fine. #8445 gave the trigger label a caller override, so the span went from className="..." to className={cn("...", triggerLabelClassName)}. The class list itself did not change: it still carries truncate with leading-tight and no leading-none, which is exactly what the guard exists to require. What broke is the guard. It matched on the JSX attribute syntax, so wrapping the same string in cn() made it find nothing, and its only reaction to finding nothing was to say so. A guard that goes quiet the moment a label grows an override is not guarding much, since taking an override is a normal thing for a label to do. It now reads the class literals wherever they are written and matches on the token set, so both the attribute form and the cn() argument form are covered. The pre-#8445 literal is still matched, so this is not a narrowing. * Read the sidebar block's class list the same way The account-block check next to it keyed on `<div className="...">` for the same reason and would go quiet the same way the moment that div takes an override, so it now shares the one matcher. Selecting on its four specific tokens is what makes reading every literal safe. Behaviour is unchanged: the same single div matches. --------- Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-08-14 04:53:28 -07:00
Daniel Han	9cc6e650c3	Studio: budget the Metal context fit from free memory, not total (#8706) ... * Studio: budget the Metal context fit from free memory, not total _apple_metal_memory_budget_bytes fed the Apple context fit from MLX's max_recommended_working_set_size, falling back to psutil.virtual_memory().total. Both describe the machine, not the moment: they are static properties, so the budget came out the same on an idle Mac and on one already holding several gigabytes. On a 16 GB Mac that is roughly 10 GB of working set times the 0.85 fraction, so about 9 GB budgeted, while Studio's own idle footprint is already over a gigabyte once the warm thread has imported torch and, on Apple Silicon, MLX, before anything the user is running. The fit then sized a context against headroom that was not there and llama-server died in KV or compute allocation, which is the failure #5118 and #6529 describe and a plausible reading of #8566 on 16 GB. Take whichever is smaller, the device ceiling or what is actually free. available rather than free, since on macOS that counts the inactive and purgeable pages the kernel hands back under pressure, so it is the number a new allocation can expect. It can only lower the budget: on a machine with plenty free the working set still binds, which is the case that number was chosen for. No behaviour change off Apple Silicon, where the function returns 0 before any of this. One existing test had to change, and it is worth being explicit about why: test_uses_metal_working_set stubbed only `total`, so `available` was read from whichever host ran the suite. It was passing by accident of this machine having enough free. Both stubs are now pinned, via a _install_fake_psutil helper, so the test measures the code rather than the runner. Tests: 65 in the context-fit suite, 1608 across the llama.cpp, Apple and hardware suites. Five new cases: a busy 16 GB machine budgets from free, a free machine is still bound by the device ceiling, a psutil without `available` keeps the old answer rather than collapsing to zero, and the working set still applies with no psutil at all. * Apply the repo's kwarg-spacing format to the new tests * Say what available actually measures on macOS, and why the fraction still applies Comment only, no behaviour change. available is inactive plus free in psutil's macOS implementation, so it adds back the reclaimable cache that free leaves out; it is still an estimate, since it counts dirty inactive pages that cost a compression to reclaim and ignores the compressor. Also records the second job _APPLE_UNIFIED_MEMORY_FRACTION picks up here: over a static ceiling it was pure headroom, but available is a snapshot and llama-server spends seconds loading weights, so another process can take memory inside that window.	2026-08-14 04:51:56 -07:00
Daniel Han	8c6048ad2a	Make two backend test guards able to fail (#8760) ... * Check the stub-before-import guard structurally, not by substring _stubs_before joined the lines above the first heavy import and asked whether the text contained "unsloth" and either "stub" or "sys.modules". Nothing in that matches an actual stub. A module docstring that merely discusses stubbing unsloth satisfies it, as does a _stub_if_missing helper that is defined and never called, so a file could lose the call site that installs its stubs and the guard would still report no offender while the job died at collection. The check now reads the code that runs before the import: the module-scope statements above it, walked minus the def/class bodies that only run when something calls them. It counts when those name unsloth as a string constant AND either call a stub-named helper or write sys.modules (assignment or setdefault / update, not a get or an "in sys.modules" probe). The two halves are looked for across the whole prefix rather than inside one statement, because the names usually sit in a module-level table that a later loop feeds to the helper, as in test_training_progress_callback.py. It can fail. Renaming _stub_if_missing to _noop_if_missing across the four stubbing test modules, which is the whole of what installs their stubs as far as this guard can see: before 4 passed after AssertionError: 3 test module(s) import a backend module that needs unsloth at module scope without stubbing its heavy deps first ... ['test_audio_type_inconclusive.py', 'test_trainer_stdout_quiet.py', 'test_training_progress_callback.py'] 1 failed, 3 passed, exit 1 Under the old substring form that same rename left the guard green at 3 passed, which is the hole. Reverting the rename returns it to 4 passed, exit 0. test_only_an_installed_stub_counts_as_stubbing pins the four shapes the substring form accepted, and the positive self-test grows the loop form and the bare sys.modules assignment so neither reads as unstubbed. Also drops one stale sentence in the split-form comment. It credits the pinning to a textual prefilter having dropped that import, but no merged revision of this file behaved that way: the version before #8740 prefiltered on both "core.training.trainer" and "core.training import trainer", so it caught the split form, and the current one has no prefilter at all. The reason the self-test asserts through _is_offender stands and stays. * Make the orphaned-task test actually reach the tool-loop drain test_no_asyncio_task_is_orphaned_when_the_loop_is_closed_mid_tool broke out of the async for and closed the generator, and the step worker is never pending at a yield: stream_with_studio_tools sets step_task back to None the moment asyncio.wait returns and before it yields anything, so the finally handed _drain_step_task None every time. Replacing the whole body of _drain_step_task with a bare return left the test green, which means it asserted nothing about the drain. The pending worker only exists across a suspension when the consumer is cancelled inside __anext__, which is what happens to the request task when a client disconnects mid tool call: CancelledError lands in the loop while it is awaiting the step, so the finally gets a live to_thread task. The test now drives the generator from its own pump task, cancels that task while the tool is blocked, then closes. The tool double returns on the cancel flag the drain sets, and the census is taken before the harness releases it, so a worker only the harness could free still counts as pending. It can fail. With _drain_step_task gutted to a bare return, on this branch: await asyncio.wait({step_task}) asyncio.exceptions.CancelledError ... tool_stream.close() ValueError: generator already executing 1 failed, 51 deselected, exit 1 which is the exact failure mode the drain's docstring describes: close() while next() is still running skips the generator's own cleanup. Restoring the drain: 52 passed, exit 0, and stable over five consecutive runs. * Pair the stub name with the operation that installs it The two halves of the stub check were looked for anywhere above the heavy import, so a file that stubs something else while the word unsloth merely appears above it read as stubbed. Turning the loop in test_training_progress_callback.py into one that stubs only trl leaves the file unstubbed for unsloth, which is what kills collection, and the guard still reported no offender. The name and the operation now have to meet in one statement. The name may still arrive through a module-level table the statement reads, because that is how the real files are written, so the table+loop form stays clean while a loose name next to an unrelated sys.modules write does not. --------- Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-08-14 04:49:54 -07:00
Daniel Han	71ad2b315c	Studio: run the LoRA scan off the event loop (#8392) ... * Studio: run the LoRA scan off the event loop GET /api/models/loras walks the outputs and exports directories and reads a config per checkpoint, and it did that directly in the coroutine. For the whole duration the server could not serve anything else, streamed chat tokens included, and the model picker polls this route so the stall repeated. Measured on this box, purely the blocking part: | checkpoints | event loop blocked | |---|---| | 50 | 4.0 ms | | 200 | 15.5 ms | | 500 | 37.8 ms | Moved the filesystem half into _scan_loras_sync and awaited it with asyncio.to_thread, which is the pattern already used in routes/llama.py and routes/whisper.py. No behaviour change: same rows, same order, same errors. The test drives the route with a deliberately slow scan while a heartbeat coroutine ticks, and asserts the longest gap between ticks stays well under the scan. It fails on the unfixed route, where the heartbeat gets no turn at all until the scan returns. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: danielhanchen <elliegouldingstuff@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-08-14 04:49:26 -07:00
Daniel Han	bb11b371e7	Run deterministic notebook smoke tests on real Kaggle T4s (#8440) ... * Run deterministic notebook smoke tests on real Kaggle T4s The notebooks are written for a T4: sm_75, fp16 with a gradient scaler, xformers rather than flash-attention, 16GB. Nothing in CI runs on that card, so a regression that only appears there reaches users first. Adds a gated job that launches one Kaggle GPU session and runs the smoke payload once per T4. The account has a 60h weekly budget shared with other consumers, so the job is a deliberate minority consumer of it: paths filters (only 9.9% of a week's 444 main commits touch unsloth/), a ~10% sampling draw keyed on the run id so a re-run cannot reroll, a quota floor read live from Kaggle's own API, and a stand-down when the account already has a kernel in flight. Expected cost is 5-7 GPU-hours a week. Only a payload that ran on a T4 and disagreed with its assertions turns the check red. Throttling, the 2-kernel concurrency cap, a dead session or a download that would not complete are all warnings, because a free external service blocking merges would be ignored within a week. The payload asserts three things, worth different amounts. Two fresh processes must agree bitwise on per-step loss and grad_norm; that one is exact, and it is fresh processes because two in-process cycles disagreed from the first step while two processes agreed on all ten. Greedy decoding must emit the canary string exactly, which catches trained weights never reaching the generate call. Metrics must sit inside a band around a committed reference, which is a tolerance and never an equality, since fp16 reduction order moves with the driver and the libraries. Ten steps rather than three: under fp16 the scaler skips the first two overflowing steps, so a three-step run lands one real update and the canary never forms. The dataset is supplied as prompt/completion columns so the loss falls only on the answer. * Make the payload cells the T4 rejected actually run there Two launches on real Kaggle T4s died in the scaffolding before a training step ran, both of them in generated code that nothing had ever parsed. The dependency probe imported unsloth_zoo before unsloth. Zoo's __init__ ends with a find_spec("unsloth") guard and raised "Please install Unsloth via pip install unsloth" on a session where unsloth was installed and imported cleanly one entry later, so the payload aborted over a dependency that was not missing. Probe unsloth first, and invalidate the import caches after the pip installs that this interpreter did not see at startup. The reference argument was generated as a shell fragment and spliced into the middle of a Python list literal, with doubled braces that left ROOT unsubstituted. The cell was a SyntaxError, and the workflow passes --reference on every path, so no payload could ever have reported. Verified locally by executing the regenerated run cell end to end: two fresh processes, bitwise identical metrics, canary exact. * Bound the in-flight kernel survey by session length, not kernel count The concurrency check status-checked the twelve most recently run kernels, which is a sample rather than a search: a kernel that started three hours ago and is still running is missed the moment twelve newer ones have since run, and the push then fails at the account capacity cap and is reported as infra. Kaggle kills a notebook session at twelve hours, and the listing is sorted by last run time, which for an unfinished kernel is when it started (measured: a kernel pushed at 10:05:19Z lists as 10:05:19.297). So walking until the first entry older than that ceiling covers every kernel that can still be in flight, and stopping there is exhaustive rather than arbitrary. An hour of slack is added for clock skew. Two silent ways to read an unanswerable question as a green light are closed with it. Hitting the page cap now marks the survey incomplete, and a survey where no status could be read at all is unknown rather than idle. Both stand down, which is the cheap direction: the next commit draws again in minutes. On the live account this now costs three status calls instead of twelve. * Stop the reference band check from passing NaN unconditionally The band comparison did its NaN handling by accident, and it got it wrong. Under fp16 the gradient scaler logs a NaN grad_norm on every step it skips, so a committed T4 reference genuinely contains NaN. Left to the arithmetic, abs(x - NaN) is NaN and NaN > tolerance is False, so those steps passed whatever they held, including the case the check exists for: a step that used to overflow and no longer does. Compare NaN to NaN as equal and NaN against a number as a deviation, and treat a field logged on one side only as a deviation too, since no tolerance covers a change in shape. The mapping from an out-of-band verdict to the failure list moves into reference_failures() so the path that turns the job red can be tested without a GPU. The references README now says plainly that no reference is committed, that one must not be borrowed from other hardware, and how to lift one out of a green run's evidence rather than spending a session on it. It also records a measurement: the absolute floor never engages on this trajectory. The smallest value on the ten-step curve is a loss of 0.1428 against a floor of 0.05, so the floor changes no verdict today and is kept for a configuration that does go lower. * Name the cause when a kernel finishes and reports nothing A kernel that ends COMPLETE with no payload report is the hardest outcome to read: the summary has no metrics to show and the reason sits in an artifact nobody downloads. It is also the outcome both real T4 failures took. Kaggle returns the kernel log as a JSON array of stream records rather than as text, so the interesting line arrives split across dozens of them and reading the file directly shows a wall of JSON. Flatten it, filter to the driver and payload sentinels plus tracebacks, and fold the tail into the job summary. Checked against the evidence of a real failed run: the SyntaxError that killed it is now visible without leaving the summary. * Cover the four T4 CI fixes with CPU-only tests The generated cells are now compiled on both the with-reference and without-reference paths, which is what would have caught a SyntaxError that instead cost a Kaggle session, and the dependency probe's module order is asserted. The in-flight survey gets a fake Kaggle client and the case that motivated it: one kernel still running behind forty newer finished ones, which a count-bounded scan misses and a time-bounded scan cannot. Also the window boundary, timezone-aware and missing timestamps, the page cap, and the difference between some statuses unreadable and all of them. The band check is now proved to fail rather than assumed to work: a perturbation half a band-width past the limit, a moved scaler skip pattern in both directions, a field that stopped being logged, and a length mismatch, each asserted through to the failure list. Three tests perturb the committed reference itself and skip, saying so, until one exists. 41 passed, 3 skipped. * Parse every generated cell on every build path before spending a session Two of the three Kaggle sessions spent so far died on generated code that nothing had ever parsed. The compile check now builds all three paths the generator has, using the workflow's own argument list verbatim, including the one where a committed reference exists and is carried inline as a fourth file: that path becomes live the moment a green run supplies the file, and would otherwise first be exercised on Kaggle. Parsing alone is not enough. A template hole that substitutes to a bare identifier parses and then NameErrors, which costs the same session, so every cell is also checked in execution order for names nothing before it defines. Both checks were confirmed to go red: the first on the exact shell-fragment-in-a-list-literal defect that cost session three, the second on a run cell reading a name no cell binds. The carried sources are also decoded the way the kernel will decode them and compared byte for byte with the repo, so the T4 cannot quietly run something other than what is committed. * Commit the T4 reference the first green Kaggle run produced Kernel danielhanchen/unsloth-t4-ci-e3c6661f, terminal state COMPLETE, both payloads passing on their own Tesla T4 of one session. Two fresh processes agreed bitwise on all ten steps on both cards, max_abs_diff exactly 0.0 for loss and for grad_norm, and the two cards independently produced the same ten values as each other, so four processes agree rather than two. All four cycles emitted the canary exactly. The scaler skipped steps 1 to 3 every time, which is the behaviour the ten-step length exists to accommodate. The file is reports[0] of that run, copied whole by the recipe the README already documented, and nothing in it is from other hardware. One assumption the run contradicted, now recorded rather than repeated: the session had no xformers. unsloth installs --no-deps, unsloth_zoo does not carry xformers and the Kaggle image does not either, so the banner read Xformers = None and this trace is the fallback attention path. Adding xformers to the install would move these numbers and require a recapture. The floor's smallest observed value moves from 0.1428 to 0.0871, still above the 0.05 denominator floor, so the floor stays inert and the test that re-derives that keeps passing. The three tests that perturb the committed reference now run instead of skipping. 45 passed, 0 skipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the in-flight tolerance the Kaggle gate stands down on The gate declines to launch whenever any kernel of the account is in flight, which is stricter than Kaggle's 2-kernel cap requires. That is the intended policy, so it is now a named constant with the tradeoff written next to it rather than an implicit property of the code, and a --allow-in-flight argument that cannot be raised to the cap itself. Default behaviour is unchanged. Also files the workflow under 'never' in ci-preempt.json: cancelling a run does not stop the Kaggle kernel it already pushed, and the orphan then bills quota to its own ceiling with nobody watching. * Run three steps, and refuse a reference captured at another count max_steps drops from 10 to 3. Measured consequence: under fp16 the gradient scaler starts at 65536 and skips every step it overflows on, and the committed reference shows steps 1, 2 and 3 all skipped, so a 3-step run of the old configuration applied zero optimizer updates. --init-loss-scale pins the scaler below the overflow point so those three steps are real updates, and optimisation_failures() fails a run whose every step was skipped rather than letting it report as training. The committed reference was captured at 10 steps and no longer describes the run. check_reference now reads the max_steps the reference records and refuses to compare across counts, as a failure with both counts named -- including when the reference does not say what it was captured at. A quiet pass there would be a green check comparing a run against a curve it has nothing to do with. The reference is therefore stale until one recapture run. The workflow gains skip_reference_band for exactly that dispatch, and the recipe is in references/README.md. Also brings the workflow onto the repo's conventions: pinned action SHAs matching the rest of CI, a workflow-level concurrency group that deliberately does not cancel in progress, a fork guard on the gate job, typed dispatch inputs and an artifact name without the run id. * Keep the run at ten steps, and say why three does not work The committed reference is the evidence: the fp16 scaler reports grad_norm NaN at steps 1, 2 and 3 and a finite value from step 4. It starts at 65536, halves on each overflow, and skips the step it overflowed on, so a 3-step run applies zero optimizer updates. Loss stays around 10, the canary never forms, and the band would compare three points of a curve that never moved. Shortening also saves nothing worth having: a launch costs about 0.08h and that is almost all pip install, not training. The step-count guard, the all-steps-skipped guard and --init-loss-scale all stay, since they are what makes a shorter run possible later and what stops a vacuous one reporting green. The pin is off by default so the committed reference keeps applying. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Turn one T4 payload into four, and pair a pinned control with a canary The workflow ran a single tiny SFT payload on both cards of one Kaggle session and spent about 1.4 GPU-h a week doing it. The budget is now 40, and the question asked of it is wider: catch regressions in trl, transformers, accelerate, peft, bitsandbytes, torch and vllm, and catch torch.compile falling over on sm_75. Four legs now, two kernels of two T4s each, described once in .github/scripts/kaggle_t4_ci/legs.py rather than restated in YAML: control the existing tiny Qwen2.5-0.5B determinism run, pinned canary the same run on the newest library set Unsloth's own constraints allow gptoss gpt-oss-20b LoRA, for torch.compile and the forced float32 path grpo Qwen3-4B GRPO through a vLLM engine on the same card control and canary are one instrument rather than two tests. Same payload, same seed, same data, same step count, on the two cards of the same session; the only difference between them is the installed versions. A red canary beside a green control is a library release, and the job summary already puts the two version sets side by side so the bisect is a diff rather than an investigation. The reference band applies to the control alone: two library sets do not produce one fp16 trajectory, so the canary asserts what does not depend on versions instead. The pins are evidence rather than preference. transformers and trl are the pair the committed reference was captured with; peft, accelerate and bitsandbytes come from probe kernels 8161ceb9 and 7ab727f1, two independent T4 sessions that ran the identical base install and agreed. torch stays unpinned and recorded, because it comes from the Kaggle image and replacing it is the failure mode the grpo leg spent two probes on. The gate now separates this workflow's own kernels from everybody else's. It takes both of Kaggle's concurrency slots, and only when the account is otherwise idle: one kernel belonging to anyone else still stands it down entirely, which is the same policy as before stated at the right grain. * Leave the vLLM leg unwired, and stop trusting Kaggle's own kernel timeout Three probe sessions tried to run the Qwen3-4B GRPO leg on a T4 and none of them reached a training step. None of them failed for a reason to do with sm_75, memory or GRPO either: vllm==0.11.2 pins torch==2.9.0, so the leg has to replace the Kaggle image's torch, and that is what killed all three. With the image's site-packages visible, pip treats torch's pinned NVIDIA runtime packages as already satisfied by the copies belonging to 2.10, and the result is a torch that installs cleanly and cannot be imported -- libcusparseLt.so.0 first, then ncclCommWindowRegister once that one was named explicitly, identically on vllm 0.11.2 and 0.15.1. An isolated venv resolving the whole stack got further and then the session wedged. So the leg keeps its payload, its assertions and its install, and moves to legs.UNWIRED with the three kernel slugs and what each one measured. Wiring it now would make the check permanently red and spend the budget doing it. The second kernel keeps its spare T4, which costs nothing, and that is where the leg goes when the install works. The third probe also cost about an hour of quota teaching us that the push-time kernel timeout is not a budget control. Pushed with -t 5400, its own nbconvert crashed at t=406s and the session sat in RUNNING for over two hours -- past that ceiling and past the launcher's own deadline, because one status call had blocked with no socket timeout to bound it. Deleting the kernel by hand took the account's used-hours figure back down, which is the measurement that settles which control is real. The launcher now sets a socket timeout and deletes every kernel it pushed on every path out, and the workflow header says which of the three bounds is load-bearing. Sampling recomputed on the payload that actually ships: 40%, from measured per-kernel cost rather than the old estimate, with the note that wiring a fourth leg means recomputing it rather than editing one line. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-solve the GRPO leg's install so it stops replacing the image's torch Three probe sessions never reached a training step, and all three died the same way: vllm==0.11.2 pins torch==2.9.0, so the leg had to replace Kaggle's torch 2.10.0+cu128, and pip treats the image's NVIDIA runtime packages -- which belong to 2.10 -- as satisfying the new torch's pins. That imports as libcusparseLt.so.0 missing, then as libtorch_cuda.so undefined symbol ncclCommWindowRegister one package along, then as an isolated venv that spent an hour of quota resolving a CUDA stack and never produced payload output. None of that is a question about sm_75, memory or GRPO. It is one line: no vLLM release in the 0.11-0.16 range pins the torch the image ships. Releases 0.17.0 through 0.19.1 pin torch==2.10.0 exactly, so pin 0.19.1 and the leg replaces nothing, keeps system_site_packages, and installs like any other. Drop xformers with it. Its vLLM attention backend was deleted in 0.12.0, so it would be a package nothing selects. sm_75 has no FlashAttention and no FlashInfer, and the ladder in vllm/platforms/cuda.py falls through both to TRITON_ATTN; the leg names that in VLLM_ATTENTION_BACKEND so a release that reorders it goes red here rather than quietly selecting something else. 7.5 is still in CUDA_SUPPORTED_ARCHS at v0.19.1 and fp16 is a supported dtype below capability 8.0. Still unwired. Two things are static evidence rather than a run: whether 0.19.1 starts on sm_75 at all, and whether 8GB of 16-bit weights plus an engine plus a LoRA trainer fit in 14.56GB. UNWIRED says so in those words, and a test asserts the note still names the pin and still says what is unknown, so nobody wires this off prose that outlived its measurement. Also ports the vacuous-test fix from the stacked Studio branch: pre-commit.ci reformatting default=10 to default = 10 stopped the payload regex matching, so the step-count agreement check was comparing nothing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop the GRPO leg JIT-compiling flashinfer on a machine that cannot link it First probe on the re-solved install, kernel unsloth-t4-ci-e2d9ce9b. The install works: torch stayed at the image's 2.10.0+cu128, vllm 0.19.1 installed and imported, xformers was absent as intended, and the payload ran on a real Tesla T4 reporting capability 7.5 with TRITON_ATTN accepted. It reached engine construction, which is further than any earlier probe got by the whole width of the install. It died in flashinfer 0.6.6's JIT, and not for an sm_75 reason. All three .cu files compiled cleanly for -gencode=arch=compute_75,code=sm_75. The LINK failed: /usr/bin/ld: cannot find -lcuda -L/usr/local/cuda/lib64/stubs is already on the command line, so the image has no driver stub libcuda.so, only the runtime libcuda.so.1. Nothing about that is fixable from a payload. Set VLLM_USE_FLASHINFER_SAMPLER=0 so nothing JITs. The sampler has a native path, and skipping the build also saves a four-file nvcc compile inside a session billed by wall clock. UNWIRED records what the probe measured and narrows what is left to one question: whether the engine builds now. * Make -lcuda resolvable so flashinfer can link what it compiled Second probe, kernel unsloth-t4-ci-916d5986, failed identically to the first with VLLM_USE_FLASHINFER_SAMPLER=0 set. That is the informative part: the JIT is not reached only through the sampler, so switching off one consumer is whack-a-mole. The failure was never sm_75. Both sessions compiled all three .cu files cleanly for -gencode=arch=compute_75,code=sm_75 and died at the link: /usr/bin/ld: cannot find -lcuda -L/usr/local/cuda/lib64/stubs is already on the command line; the image ships no libcuda.so at all, only the versioned libcuda.so.1, and the linker will not resolve -lcuda against a soname. Normally the toolkit's driver stub fills that gap; this image has the directory and not the file. So symlink libcuda.so at the real driver in a scratch dir and put it on LIBRARY_PATH, which gcc and ld search for -l. No root, nothing written into /usr/local, and it fixes every flashinfer op rather than one. Linking against the real driver instead of a stub is correct here: the driver is present, which is the whole reason a stub would have substituted for it. Also widen the captured traceback to head AND tail. The last probe's 6000-char tail was entirely ninja's own output, so the Python frames naming the caller were exactly what got dropped. Probe 2 also measured the fit question the notebook raised: peak reserved 7.96GB of 14.56 with load_in_4bit false, so 16-bit weights are not what is short of room here. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only count a libcuda the linker will actually search for Kernel unsloth-t4-ci-d0d480b6: the shim reported already_linkable and did nothing, and the link failed anyway. It had found libcuda.so in /usr/local/cuda/compat -- which is real, and which nothing passes with -L. The failing ninja line names /usr/local/cuda/lib64 and /usr/local/cuda/lib64/stubs, and those are the only two directories that can answer -lcuda. So the precondition now checks exactly those two, and compat becomes what it is actually good for: a symlink TARGET when ldconfig and ctypes both come up empty. A library the linker will not search for is not a library the linker can find. Two tests: one reproduces the compat-only machine and asserts the shim is built anyway, one pins the searched directory list, since widening it is precisely how the check went wrong. * Give the base model a chat template, and record what the link shim did The link shim works. Kernel unsloth-t4-ci-27b0dc2e reported libcuda_shim: needed true, applied true, searched [/usr/local/cuda/lib64, /usr/local/cuda/lib64/stubs], real /usr/local/cuda/compat/libcuda.so and flashinfer is gone from the failure entirely. Memory went from 7.96GB to 11.36GB of 14.56, which is the vLLM engine actually loading, and the run reached trainer.train() -> _run_epoch -> training_step. Both questions legs.UNWIRED was holding this leg for -- does vLLM start on sm_75, does it fit in 14.56GB -- are answered by that. It then failed on the payload's own gap: ValueError: Cannot use chat template functions because tokenizer.chat_template is not set unsloth/Qwen3-4B-Base is a base model and ships none. The notebook this leg comes from solves it with an SFT priming stage that installs a template before GRPO starts; this leg has no priming stage, so it sets a minimal ChatML template directly and records in the report which of the two worlds the run was in. The base model stays. GRPO on an instruct model would measure the instruct tuning as much as the run, and the payload's rewards are format-and-digit rewards a base model can move inside three steps. A test renders the template rather than matching it as a string: a template that does not render would trade a failure at step 1 for a failure at step 1 with a longer traceback. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Wire the GRPO leg: it runs on a T4 and the rewards move Kernel unsloth-t4-ci-53efcc4e passed on a real Tesla T4. Step 2 logged reward_std 0.707 and grad_norm 0.772, peak 13.60GB of 14.56GB, three steps in 192s, and fast_generate returned coherent text. reward_std is the criterion rather than loss, because TRL's GRPO loss is ~0 by construction at num_iterations=1 and beta=0 and would pass on a run that learned nothing. The config in args is the one that fit, not the notebook's. Two probes with seq 2048, 4 generations, rank 32 and utilization 0.9 died in the backward at unsloth_zoo/gradient_checkpointing.py:1013, peaking at 15.97GB in 16-bit and 19.25GB in 4-bit. 4-bit alone is not the lever: it peaked HIGHER, because quantizing weights does nothing for activations while utilization 0.9 still hands vLLM ~13GB up front, and UNSLOTH_VLLM_STANDBY returns the weights during training but not the KV cache reservation. An earlier probe reported this as 'CUDA error: an illegal memory access was encountered' from empty_cache() inside vLLM standby. That was the async shadow of the same OOM; CUDA_LAUNCH_BLOCKING=1 produced the honest error. UNWIRED is now empty. The test that read its grpo entry is replaced by the invariants that outlive any one leg: every leg is wired or explained, nothing is both, and an unwired note says what is open. Two new tests pin the config and the vLLM pin, so restoring either to a value that OOMs fails here rather than on Kaggle. * Add a CUDA_LAUNCH_BLOCKING switch to the GRPO payload An illegal memory access is reported at whatever CUDA call synchronises next, which on the first T4 GRPO run was empty_cache() inside vLLM standby, nowhere near the kernel that faulted. Re-running with this flag produced the honest error instead: torch.OutOfMemoryError in the backward. That is what turned an unexplained CUDA fault into a memory budget, and the budget is now the leg's config. The environment variables are set before the first import torch in the process rather than merely before train(), because they have to precede CUDA context creation. It serialises every kernel launch, so it stays a switch and never a default, and it is recorded in the report because it changes what a timing means. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Write down where the Kaggle sampling percentages come from The percentages in the two workflows were arithmetic nobody could check. This records the measurements they came from, taken 2026-08-11 over a 28-day window: 985 commits to main, 1157 PRs, and 8 of 60 sampled PRs touching the paths a T4 training run could regress, so a 13% path-filter rate and roughly 75 candidate events per week. Against 30 GPU-h/week for one account at ~0.75 GPU-h per launch, the ceiling is about 33 launches per week if everything went to one workflow. The 15/5 split and the 25/10 reserve-hour floors follow from that, and the file says what would change them: a second account, a wider path filter, or a session cost above ~1 GPU-h. * Give the GRPO leg its own kernel: pairing it with gpt-oss fails Kernel unsloth-t4-ci-70a2f4eb ran the wired pair on the two cards of one session. gptoss passed: 32 unique graphs, 779 calls captured, 2 graph breaks, peak 12.78GB. grpo failed with the illegal memory access again, at exactly the 13.60GB peak at which it PASSED alone on kernel unsloth-t4-ci-53efcc4e, same config to the flag. Same peak, different outcome, so the constraint is not GPU memory: the driver pins one card per payload. gpt-oss offloads to host RAM and vLLM wants host RAM, and a Kaggle session has one host. A third kernel costs a queue slot rather than quota, since Kaggle runs two at a time and bills wall clock per session. That is cheaper than a leg which fails half the time for a reason the report cannot show. Two tests pin the rule in both directions: grpo must not share a kernel with gptoss, and control must still share one with canary, because those two are a matched pair whose comparison is the whole instrument. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Unwire the GRPO leg: running it alone reproduced the failure I blamed on pairing I gave grpo a kernel of its own earlier today on the reasoning that sharing a session with gpt-oss was what broke it. It had failed paired (unsloth-t4-ci-70a2f4eb) and passed alone (unsloth-t4-ci-53efcc4e), so pairing looked like the variable, and I wrote two tests pinning the separation. That reasoning was wrong. Running the leg ALONE again (unsloth-t4-ci-c98f14be, built from this branch with --legs grpo) reproduced the paired failure exactly: unsloth_zoo/vllm_utils.py:601 sleep() -> torch.cuda.empty_cache() torch.AcceleratorError: CUDA error: an illegal memory access was encountered engine_built false, peak 13.8GB/13.6GB of 14.56GB. The passing session and both failing ones agree to the digit on every recorded version -- torch 2.10.0+cu128, transformers 5.5.0, trl 0.24.0, peft 0.19.1, vllm 0.19.1, unsloth 2026.8.15, zoo 2026.8.10 -- on the same peak, with UNSLOTH_VLLM_STANDBY=1 throughout. One contrasting observation was never enough to blame a shared host. So what three sessions show is an intermittent illegal memory access on Turing, one pass in three, not a pairing effect. A leg that passes one session in three cannot tell CI anything: it would go red for a reason no reader could act on. grpo moves to UNWIRED with the three session ids and the open question written down, rather than back into a kernel. The note keeps what the leg already earned -- the vLLM/torch pin, the attention backend, the libcuda shim, the chat template, the T4-sized config -- so re-wiring means answering the IMA rather than redoing the install. test_grpo_does_not_share_a_session_with_gptoss is replaced rather than deleted: it asserted the opposite thing for the disproven reason, and would now pass vacuously since grpo is in no kernel at all. The replacement pins that grpo stays unwired and that the note still names all three sessions, so the evidence cannot be dropped on the way back in. 145 passed. * Record what launch blocking did to the GRPO leg: it removed the fault The UNWIRED note asked for one experiment: run the leg with --cuda-launch-blocking so the illegal memory access reports at its real call site instead of surfacing at torch.cuda.empty_cache(). Done, kernel unsloth-t4-ci-b1f23e34, and the answer is not the one the note expected. With blocking on there was no illegal memory access at all. engine_built true, three steps, same 13.8GB peak, same versions to the digit. A fault that disappears when the launches are serialised is a race, which is what one pass in four already suggested and which no amount of re-running the leg will pin down. The run also exposed a SECOND problem, and it is separate from the first. It failed on reward_std = [0.0, 0.0, 0.0] with grad_norm 0.0 at every step. The completions in the report are coherent prose, so this is not the model collapsing into degenerate output. It is two completions scoring identically. The leg runs num_generations = 2 because that is what fits on a 14.56GB card, and at two samples a tie on a coarse reward is ordinary rather than a bug. So the leg's pass criterion is fragile at the size it has to be to fit, which is worth knowing before anyone reads a red grpo leg as a regression. The note now records both, says which question each answered, and states the two that remain open: where the race is, and what pass criterion is honest at num_generations = 2. The test gains the fourth session id and a check that the note no longer reads as though launch blocking were still the next thing to try -- a stale "try this next" is how an experiment gets run twice. 145 passed. * Test the newest transformers and trl, not the newest zoo permits The canary leg is documented as "the newest permitted library set" and that is exactly what it installs: it resolves WITH unsloth_zoo in the resolution, so zoo's own metadata is a ceiling. Measured against the two runs that carried it: canary transformers 5.5.0 trl 0.24.0 peft 0.20.0 accelerate 1.14.0 control transformers 5.5.0 trl 0.24.0 peft 0.19.1 accelerate 1.13.0 PyPI the same day: transformers 5.15.0, trl 1.9.2, peft 0.20.0, accelerate 1.14.0. The canary moved peft and accelerate to genuine latest and moved transformers and trl not at all, which is what made it look like it was working. zoo pins transformers <=5.5.0 and trl <=0.24.0, ten minors and a whole major behind, so this CI could not detect a transformers 5.6+ or trl 1.x regression: it never installed one. The new frontier leg upgrades transformers and trl with their dependencies and without zoo in the resolution, which clears the cap because pip enforces only the requirements of packages in the resolution. It runs the same SFT payload as control and canary, so all three are directly comparable. I expected it red. It is green. On a real T4: transformers 5.15.0, trl 1.9.2, datasets 5.0.1, ten steps, canary emitted, and two fresh processes agreeing bitwise. Unsloth trains and generates correctly a whole trl major above what zoo's metadata permits. Getting there took two kernels. The first used --no-deps plus a blanket tokenizers upgrade and died before running anything, because an unbounded upgrade overshoots the ceiling transformers declares: tokenizers<=0.23.0,>=0.22.0 is required, but found tokenizers==0.23.1 safetensors>=0.8.0 is required, but found safetensors==0.7.0 A test pins that shape out, along with the zoo-in-the-resolution mistake that would silently turn this into a second canary. It goes on the second kernel's idle T4, so it costs no quota: a Kaggle session bills wall clock once, not per card. What it does NOT catch is recorded in the leg: the loss trajectory is not the control's, and step 1 is computed before any update on identical weights, data and seed, so 10.3222 against 6.4367 is a different loss function rather than drift. Both converge. Which objective is intended is separate work. * Name the encoding on every checked-in file the T4 harness reads Repo tests (CPU) is red on this branch: test_source_read_encoding.py::test_checked_in_file_reads_name_an_encoding AssertionError: 17 file reads in the test trees touch a checked-in file with the platform default encoding, so they break on Windows as soon as that file gains a non-ASCII byte. All 17 are in tests/kaggle/test_t4_smoke_harness.py, which this branch adds, so this is ours and not inherited. The reads pull in the workflow YAML, the smoke scripts, the committed reference JSON and ci-preempt.json, and none of them said what encoding to read them as. On Linux that resolves to UTF-8 and nobody notices; on a Windows runner it resolves to the ANSI code page and the first non-ASCII byte in any of those files takes the test out with a UnicodeDecodeError that has nothing to do with what the test is checking. Every one now passes encoding = "utf-8", matching how the rest of the suite spells it. Reads of files the test itself just wrote under tmp_path are left alone: the gate only covers checked-in files, and those are ours end to end. test_source_read_encoding.py 1 passed tests/kaggle/test_t4_smoke_harness.py 148 passed * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align the Kaggle budget guide with the workflow it documents The guide named --percent 15 and --reserve-hours 25 against a 30h account; the workflow runs 40 and 20 against the 60h figure measured on 2026-08-11, and it is the only consumer of that quota today. Following the guide would have tuned the reserve from a baseline that no longer exists. Re-derived from the same live numbers as the workflow header, and says which of the two is the source of truth when they disagree. * Stand the gate down on any unreadable kernel status The concurrency verdict refused only when EVERY in-window status came back an error. One unreadable status among readable ones was treated as an idle account, and driving concurrency_verdict with {busy: [], own: [], foreign: [], surveyed: 2, unreadable: 1, complete: True} returned (True, ""). That kernel can be the human session the zero-foreign policy exists to yield to, and proceeding takes the account's last slot: the first push wins it, the second comes back at capacity, and the run reports a subset of its legs. A 404 is kept separate and stays benign. A deleted kernel is not an unknown state and the launcher deletes every kernel it pushes, so blocking on that would wedge the gate shut rather than make it careful. * Close six gaps in the Kaggle T4 notebook CI workflow - The pull_request trigger used GitHub's default activity types, which are opened, synchronize and reopened only. The kaggle-t4-ci label the gate advertises as a human override could therefore never start a run on its own. Subscribe to labeled as well. - pyproject.toml was in neither paths filter, though every payload installs the commit under test as a distribution built from it. BUDGET.md counted it as a watched path already. - The job deadline of 120 minutes was shorter than the launcher's worst case. Two sequential pushes can take 2 x (4 x 600s + 315s of backoff) and the 5400s polling window only starts after them, so about 187 minutes with the deletions; a runner killed at 120 takes finish() -> release() with it and leaves pushed kernels billing quota. 200, with the arithmetic written down beside it. - The gate surveys quota and in-flight kernels before t4-smoke queues on the account-wide concurrency group, so a second sampled run could launch on an approval a full run old. Re-ask with the slot in hand, immediately before the push; --force true skips the sampling draw and nothing else. - --zoo-ref main let each payload's own pip resolve a different unsloth-zoo commit, including between the control and the canary within one session, and zoo is not in pins/control.txt either. Resolve it once with git ls-remote and pass that SHA to every leg. - A dispatch with max_steps != 10 still shipped the 10-step reference, and check_reference reports step_count_mismatch as a failure, so the run could only ever go red on arithmetic. Drop the band automatically when the counts differ, and say so. Also: check out the same head SHA the launcher installs, so the harness and the package under test are one snapshot rather than the merge tree and the head; and run tests/kaggle/test_t4_smoke_harness.py on the runner before anything is built or pushed, since testpaths limits pytest to tests/security and nothing else collected it. * Read back what the Kaggle kernel actually reported Two ways a real payload verdict was being lost in transport, both ending in verdict=infra and a green workflow. The kernel log fallback could never work. Kaggle's kernels/output hands the log over as a JSON array of {stream_name, time, data} records, which is why report.kernel_log_text flattens it before reading; extract_reports scanned the file as text, so no line began with the report prefix and the fallback that exists for the run whose notebook never came back found nothing. Record boundaries are not line boundaries either, so the records are joined before the scan rather than read one at a time. push() documented a fresh slug per attempt and used one for all four. Pushing to an id that already exists does not replace it: Kaggle files a new version and starts a second batch session, and kernels status and kernels/output send no version label, so they answer for the newest session only. A retry after a lost response therefore collected the wrong execution's evidence while the first kept a session slot and billed unseen. Each attempt now takes its own slug and discards the previous one first, and every slug filed is returned and recorded, since a push that reported an error may still have landed. * Give each payload its own everything, and report a probe failure Four ways the two payloads of one kernel were not actually isolated, plus the import probe exiting without a verdict. The probe was the P1. When the commit under test breaks import unsloth, the verify cell raises before the run cell can write a report, the launcher extracts none, and no reports is classified as infra -- so a deterministic import regression exits green. A probe failure is a verdict, not missing evidence, so it now emits its own failing report. nvidia-smi failing or a one-GPU allocation fell back to N_GPU = 1 and pinned both payloads to device 0. Each child still saw exactly one card and passed its own visibility assertion, so the contention came back looking like a code failure. A shortfall is infrastructure and stands the kernel down before a thread starts, which is what the comment above it already claimed. A payload whose virtualenv could not be built kept the base python3 kernel and installed into the shared system site-packages. That is the one thing the per-child venv exists to prevent: the legs install deliberately different library sets, so the last writer wins and the resulting import error reads as a regression. The payload is skipped instead. Both payloads materialised into one source directory and compiled into one unsloth_compiled_cache. The sources are byte-identical copies but write_bytes truncates first, and the compile cache is a relative path resolved against a working directory both papermill children inherit while compiling the same modules against different transformers versions. Each payload now owns both, and the prune globs still reach them. Finally, --smoke-args is appended after every leg's own arguments, so the workflow's --max-steps 10 landed after the gpt-oss leg's --max-steps 3 and argparse took the last one. The 20B leg was training for ten steps rather than the three that were measured to fit. A shared argument no longer reaches a leg that sets that option itself. * Count a moved step and a one-sided field as run-to-run differences compare_metrics zipped the two traces positionally and skipped any field that was not on both sides, so a grad_norm logged by one fresh process and not by the other left identical true, and so did a step coordinate that shifted, duplicated or reordered while the values stayed in place. Both are nondeterminism, which is the one thing this comparison exists to find, and check_reference already treats one-sided presence as a change in the shape of what the trainer logged rather than as a skip. * Assert the SFT leg's canary, adapter, repeats and reference properly Six gaps, all of them a green run that established less than it claimed. The canary was a substring test, so the exact, tolerance-free assertion the docstring describes twice accepted '__UNSLOTH__!!!<anything>' -- which is what a stopping or EOS regression produces. It is now an exact match after stripping surrounding whitespace, with both readings recorded so a red says which of the two it was. The saved adapter was only ever checked for a filename while inference ran on the in-memory model, and the caller-side reload the comment promised does not exist anywhere. The serialized weights are now read back and checked for tensors that are present, finite and not all zero; lora_B starts at zero, so an all-zero file is an untrained adapter that would reload without complaint. --repeat 3 compared cycles 0 and 1 and threw the rest away. Every extra cycle is now compared against the baseline, and the summary keys report.py renders off the reproducibility block keep their shape. optimisation_failures counted an infinite grad_norm as an applied update because inf equals itself, so a run whose every step overflowed to inf rather than NaN reported as a training run that trained. It requires a finite norm now, which is what the loss check three lines above already did. check_reference guarded max_steps out of a config block it was already holding whole. The learning rate, the optimizer, the LoRA shape, the model and the commit of the model repository that was actually read are compared on the same terms, and so are the per-step coordinates, all of them refusing before a single number is compared. Older references keep working: a key the file does not carry is listed unchecked rather than called a mismatch. A cycle that never wrote its report produced a failure report with no environment block, which is the case where knowing which library set died matters most. The fingerprint and the config are now read before the cycles and carried into that report. * Assert the float32 path, the training compile and the optimizer update The gpt-oss leg exists for Unsloth's FORCE_FLOAT32 behaviour on a card with no bf16, recorded the fp16/bf16/force-dtype state it ended up in on every run, and asserted none of it. A run that quietly went through fp16 instead logs finite losses, compiles and generates, and reports green while the one thing this leg uniquely covers was never exercised. It is now a failure, conditioned on the card rather than hardcoded to T4: where bf16 IS supported the patch not firing is correct. The compile assertion read Dynamo's process-global counters after training and asked for at least one unique graph. Loading a 20B checkpoint through Unsloth fills those counters long before trainer.train() is called, so a training path that fell back to eager entirely still satisfied it. The counters are now sampled before training too and the assertion is on the delta. grad_norm was captured in every metric entry and read by nothing. A run whose gradients are all zero keeps a finite loss, compiles, and generates text from the untrained base model, so the leg reported LoRA training it had not done. At least one logged norm must now be finite and non-zero, and as in the SFT leg the check declines to decide where grad_norm was never logged at all. * Generate through the trained adapter, and keep the engine verdict The final fast_generate passed lora_request=None. fast_generate is the vLLM engine's own generate and the trained LoRA reaches it only through save_lora plus load_lora, so the check generated from the base weights and passed whether or not the adapter could be transferred into the engine at all -- which is the second of the two questions this probe exists to answer. The request is built now, what happened is recorded, and a transfer that failed is a failure. engine_built was set on a local dict that train() only returned at the end, so any exception after the engine was built threw the fact away and the report recorded engine_built false. That is the opposite of what happened and it erases the distinction the probe needs between vLLM being unable to start on sm_75 and GRPO failing later. Facts are published into the report as soon as they are known. A reward on some steps satisfied the check that the docstring describes as a reward on every step: a step whose reward functions never ran was filtered out and the others covered for it. Every step that logged a loss must now carry a reward; the summary row train() appends carries no loss and is not a step. Every logged grad_norm being NaN also passed. Under fp16 on this card every step can overflow and be skipped while loss, reward and reward_std are all still logged and base-model generation still returns text, so the length check alone was satisfied by a run that updated nothing. * Take the reference from the control report, not from reports[0] extract_reports walks sorted(rglob(...)) over per-kernel directories named after a random Kaggle slug, and inside the control/canary kernel t4_canary_output.ipynb sorts before t4_control_output.ipynb. Driving the real extractor over a four-leg run puts frontier first. Following the recapture recipe as written therefore commits the canary, frontier or gpt-oss trace as the control reference, and every later run then band-checks against a curve it has nothing to do with. The recipe now selects the report whose label is control and asserts there is exactly one, and carries the model and the resolved checkpoint commit through with the config. Also 45 CPU-only tests for what the payloads assert, which is the half of the harness that test_t4_smoke_harness.py does not cover: it checks the launcher, the gate and the generated notebook, and this checks whether a given result dict is called a pass. All 45 fail on the parent of this series apart from the ten that pin behaviour that was already right. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Let two runs that both overflowed on the same step agree An fp16 overflow does not only log a NaN gradient norm. clip_grad_norm_ over a gradient holding an infinity returns infinity, so the same skipped step can come back as inf on both cycles of the same configuration. compare_metrics handled the NaN case explicitly and then subtracted everything else, and abs(inf - inf) is NaN, which is not equal to 0.0. Two bitwise identical traces were therefore reported as nondeterministic, and the summary printed DIFFERED with a max_abs_diff of 0.0 beside it. Equal values now short circuit before the subtraction. A one sided infinity, and two of opposite sign, still reach it and still come out as differences. * Start the deletion deadline before the first push, not after the last A Kaggle kernel bills from the moment the push is accepted, and the only control that has been observed to stop one is the launcher deleting it. That deletion was scheduled from a deadline started after every push had returned, so the first kernel got --max-wait on top of however long the second push spent in its retries: four attempts at the 600s subprocess ceiling plus the backoffs is about 45 minutes, which turns a 90 minute ceiling into 135 minutes of billing. The deadline is now one invocation-wide clock started before the first push, so time spent pushing comes out of the wait rather than being added to it. The reservation the gate makes was understated for the same reason. It was derived from the ceiling passed to Kaggle at push time, which this workflow already records as unreliable; it is now derived from the launcher's own bound, one --max-wait per kernel, and the harness suite keeps the two in step. * Delete every slug a push filed, not only the accepted one push() files a fresh slug per attempt and returns all of them, because a push that reported an error may still have landed: Kaggle answers an accepted push with a 5xx or a reset connection often enough to be a known issue. Nothing then reconciled that list. Cleanup read entry["slug"], which a failed push does not set at all, so two cases left a session running with nobody watching it: - the last attempt of a push that never succeeded, which is exactly the ambiguous one that made the retry necessary; - an earlier attempt of a push that later did succeed, since the discard before each retry is best effort and subprocess.run does not raise on a refused delete. Both keep one of the account's two concurrency slots and bill GPU quota to the kernel's own ceiling. Release now walks every slug the entry filed and records which of them are gone. A delete for a slug Kaggle never created is refused and costs one call, which is why this can be unconditional. The job deadline arithmetic is recomputed for the extra deletions and for the polling deadline no longer stacking on the pushes. * Report an install that cannot be resolved as a failed payload The install cell retried three times and then raised, emitting no payload report. The launcher extracts reports and, finding none for that leg, calls the run partial or infra and exits green -- so a commit whose distribution cannot be resolved, or a control pin set that stopped resolving, passed the one job that exists to test packaging metadata. Only the later verify cell reported a failure, and the install cell never reaches it. The exhausted branch now writes the same kind of structured report the dependency probe writes, naming the group and the last pip error, before it raises. The retries are also backed off. They were immediate, so all three landed inside the same upstream blip, and the third failure has to mean the resolution is impossible rather than that one minute was bad. * Keep the resolved versions in an import failure report report.version_table builds the per leg comparison out of the report dictionaries, not by scraping the kernel log, so a leg whose report carries no versions is missing from the one table that answers the question the control and canary pairing exists to answer: which release did the red leg have that the green one did not. The versions were computed and printed one line above the failure and then discarded. They are now kept in a name, carried into the report, and the computation is wrapped: a half installed distribution can make importlib.metadata raise, and losing the report to a diagnostic would put the leg back to reporting nothing at all. * Stand the run down when the zoo commit cannot be resolved The fallback to the branch name restored exactly what pinning the zoo revision removed. unsloth-zoo is not in the control pin file, and every payload pips independently on the kernel, so with main the control leg and the version canary can resolve two different zoo commits inside one session. That invalidates the control's reference band and the attribution the pairing exists for, and the report records a distribution version rather than a commit, so the drift is not visible after the fact either. The ls-remote is retried three times, since one blip is not a verdict, and a resolution that still cannot be made stands the run down: a warning, no kernels pushed, and a green job, which is how every other infrastructure stand-down in this workflow behaves. Nothing that spends a Kaggle session runs after it. * Run every CPU suite in tests/kaggle before spending a session The step named one file, and pyproject.toml restricts default discovery to tests/security, so the two suites added after it were collected by no invocation anywhere. Measured at this commit: the step collected 159 tests of the 227 in the directory. The 68 that ran nowhere are the transport and payload assertion suites, which cover the report extraction, the payload pass and fail rules, the reference checks and the per leg isolation. It runs the directory now, so a suite added next to these is picked up without this file being touched, and there is a test that collects whatever the step names and asserts every suite in the directory is in the result. Measured in a bare venv with pytest, transformers and CPU torch: 226 passed. * Record the kernel a recaptured reference came from, not the leg label The recipe wrote report["label"] into source_kernel, which is always the string control. That field is the only thing in the file pointing outwards, at the hardware execution its band was measured on, and it is what makes a recapture auditable while the evidence artifact is still around. The committed file carries a real slug, so following the recipe would have replaced it with a word that names nothing. The recipe now derives the slug from launch_result.json: each kernel collects into a directory named after the last segment of its slug and each leg's executed notebook is named after the leg, so the control leg's kernel is identified rather than guessed at, and the recipe refuses if that does not come out to exactly one. The recipe is now executed by the test rather than pattern matched, against a two kernel evidence tree. * Pin the reference model, and say what the band did not compare The committed reference carried no identity keys at all, so the checkpoint pin added with the reference config never ran on any workflow invocation and the band could return ok after the loaded revision moved. Two of those keys are unknowable without another T4 session, but one is not: the control leg passes no --model, so the reference belongs to DEFAULT_MODEL. It is recorded now and that comparison is live. resolved_checkpoint and resolved_revision stay unrecorded until the next recapture, because inventing a commit hash for a run that happened is worse than admitting the file does not carry one, and refusing on their absence would turn every run red over a reference that is otherwise sound. What changes is that the gap is no longer silent. A key present on one side only is a pin that did not run: it is recorded under config_unchecked in both directions now, including the one where the reference names a commit and the run could not read one, and report.py puts that list on the job summary next to the band verdict. A key neither side claims stays unreported, since nothing was asserted about it. * Record the count a fresh runner sees for the harness step The comment beside the step names what a bare venv with pytest, transformers and CPU torch collects, and it moved with the tests added alongside these fixes. Measured, not estimated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Record a push that times out instead of letting it kill the cleanup subprocess.run(timeout=600) raises TimeoutExpired rather than returning a failed push, and nothing caught it. Pushing the second notebook after the first was accepted therefore left main() without finish()/release() and without launch_result.json, so the accepted kernel billed to its own ceiling with nobody reading it, and the slugs the timed-out attempt had filed -- the most ambiguous ones there are -- were lost with the exception. The timeout is now a failed attempt like any other: the slugs stay in `attempts`, the retry discards the previous one, and release() reconciles what is left. The whole of main() past authentication also runs under a guard that still deletes and still writes the result on any unforeseen abort, since nothing in the workflow cleans up after this script. * Read the adapter itself, so a silent grad_norm cannot pass a leg `if norms and not applied` passes on an EMPTY list, and the gptoss and grpo legs had nothing else to fall back on: neither saves an adapter or reloads one, so a Trainer or TRL change that stops logging grad_norm takes the only evidence of an optimizer update with it and both legs go on reporting green. Finite losses, captured graphs, rewards and non-empty generation are all produced by the base model and the loader on their own. Both legs now fingerprint their LoRA parameters before training and after, and that reading decides it, with grad_norm as the fallback rather than the source. Silence is still not evidence of nothing applied, which is the position the earlier round took; it is now answered from the weights instead of guessed at. Only when BOTH instruments are gone is the leg red, and then for what it is: the training path it claims to cover can no longer be shown to have run. The tiny SFT leg is unchanged. It reads its saved adapter back off disk and fails on an all-zero one, which is the same question already answered. * Stand down on an unrelated label, and pin a dispatched ref to one commit Two ways this workflow spends more than it budgeted for. The `labeled` trigger fires for EVERY label, not just the opt-in one, so any label applied to an eligible pull request started a run and a fresh sampling draw. Once kaggle-t4-ci was present it was worse: it stays in the label list, so every later label of any kind arrived as an override and forced a session. The budget arithmetic counts pull request opens and pushes and no label activity at all. The gate now takes the event action and the label that arrived, and stands a `labeled` run down unless that label is the opt-in one. Every other action is unaffected. A dispatched unsloth_ref was forwarded to the payloads unchanged, and each payload pip-installs on the kernel by itself, so a branch name could resolve to a different commit in each of the four legs -- the control and the canary compared across two different Unsloths, invisibly, since the report records a distribution version rather than a commit. It is resolved once now, here, exactly as the zoo revision below it is, and a ref that cannot be pinned stands the run down rather than installing a moving branch. What a dispatch varies is still only the package under test: the harness stays on the checked-out tree, because most of it does not exist on older refs. Label names and the dispatched ref travel through the environment rather than being interpolated into the shell. * Say in the budget file what a label event costs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Close five ways a leg could pass on a run that trained nothing usable Every one of these is a report that stays healthy while the thing the leg uniquely covers did not happen. - A LoRA weight that went NaN or infinite was the strongest possible pass: the fingerprint returned a number, `NaN != finite` read as "the adapter changed", and update_verdict said `applied` on exactly the no-telemetry path it was written to decide. A non-finite reading is now its own verdict, it beats a healthy grad_norm, and both call sites fail on anything that is not `applied` rather than on the two verdicts they happen to know about. - The saved adapter was accepted on a nonzero tensor count, which lora_A satisfies at initialisation. An adapter whose B matrices were all zero or were dropped contributes exactly nothing and reloads as the base model, so the B matrices are counted specifically and their absence is unusable rather than fine. - UNSLOTH_FORCE_FLOAT32 was tested for being nonempty. The loader writes "0" on its ordinary branch before deciding whether to force, and every production consumer reads `== "1"`, so forcing could stop firing with fp16 and bf16 both still false and the leg green. - The compile assertion fell back to the absolute graph count when no pre-training baseline was read. That count is the loader's, so a training path that ran entirely eager passed in the one case where training could not be isolated. A missing baseline is now unverifiable. - Reference drift divided through infinities to NaN, and NaN > tol is False while max(worst, NaN) is worst, so a step that used to be finite and now overflows was accepted without even moving worst_rel. Equal signed infinities are the unchanged case; every other pairing is a deviation, decided before the division. * Report a payload that crashed, and re-emit its report on one line The run cell wraps the payload in a child process and is the only thing that sees the child's exit status. Two ways it threw that away: A nonzero exit with no report printed "NO REPORT WRITTEN" and nothing else. No report at all is `infra` at the launcher and one report of two is `partial`, and both leave the workflow green, so a CUDA segfault, a native abort or an OOM kill was accepted silently while the definitive exit status was in hand. The crash is now emitted as a `passed: false` report carrying the return code and a stderr tail. The recovery path for a report whose own compact line fell out of the retained stdout tail echoed the file verbatim. Every payload writes that file indented and the launcher parses whole lines, so it was handed a lone brace to decode and the definitive result was downgraded anyway. The file is parsed and re-serialized compactly, and a file that will not parse takes the crash path rather than disappearing. * Delete the kernels for real, and prove the ref before paying for it Four things this harness reported as done without ever establishing them. Release: subprocess.run does not raise on a nonzero exit, so the cleanup loop recorded every slug as released whatever Kaggle answered. The live case is worse than a transient refusal. kaggle==1.7.4.5 has no "kernels delete" subcommand at all (it landed in 1.7.5.0, Kaggle/kaggle-cli#762, first released in 1.8.0), so argparse answered every delete with "invalid choice: 'delete'" and exit 2, and the run still called the kernel released while it billed on to its own ceiling. delete_kernel now reads the exit code, retries a refusal, and a slug it cannot confirm gone is named in the log, in launch_result.json and in a workflow annotation. The pin goes to 2.2.4, because the version is load bearing and 1.7.4.5 could not do the job. Verified against the live API: 1.7.4.5's authenticate() refuses KAGGLE_API_TOKEN, the only credential this workflow has, and demands a kaggle.json nothing here writes; quota_view(), which gate.py reads the remaining accelerator hours from, exists only on 2.x, so below it the call raised AttributeError into a handler that recorded "quota unreadable" and let the run proceed. Dispatched ref: a 40-character SHA was accepted on shape alone and nothing asked whether unslothai/unsloth has that commit, so a mistyped one pushed the paid kernels, failed every payload's git install, and reported the pull request red for a commit that never existed. git ls-remote cannot answer that (it matches refs and exits 0 with empty output for any SHA); a git fetch of the object can, and it is the same reachability pip needs. Applied to the ls-remote result too, not only to the SHA the form supplied. GPU probe: the verify cell's bare device_count assert aborted before the run cell could emit its fallback report, so a CPU-only or CUDA-incompatible torch wheel left the launcher with no report, which is infra, which exits 0. It now emits passed: false the way the import probe above it does, for any failure in the cell rather than the count alone. gpt-oss: the forced-float32 assertion was conditioned on bf16_supported is False, and main() records environment = {"error": ...} for the whole probe when it raises. An unreadable card reading therefore skipped the one check this leg uniquely carries while training, losses, adapter and generation all still passed. Anything that is not a literal True or False is now a failure, and the healthy floor fixture carries the reading so every case below it exercises the check. tests/kaggle: 292 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Say the same things about this CI in fewer lines Comment-only pass over the Kaggle T4 notebook CI: the prose blocks that explain the leg pairing, the budget arithmetic, the cleanup contract and the payload assertions were several paragraphs where a few lines carry the same meaning. Every measured fact stays: the kaggle client pin and the three things 1.7.4.5 cannot do, the ls-remote vs fetch behaviour on a bogus SHA, the vLLM torch pin windows, the recorded losses, peaks, kernel ids and version boundaries. No code changes, verified by an AST comparison over every touched file. * Count both deletion paths in the deadline that protects them The job deadline exists so the runner is never killed during release(), which is what deletes the kernels and stops the billing. Its arithmetic counted one delete per filed slug and none of the discards, so it was about half the launcher's real bound: one delete DELETE_ATTEMPTS 3 x 180s plus backoffs 5 + 10 = 555s push, per notebook 4 x 600s + backoffs + 3 discards = 4380s two notebooks = 8760s release(), 4 filed slugs per kernel = 4440s evidence ~600s total 13800s That is 230 minutes against a 200 minute deadline, so a wedged run could be killed mid-cleanup and orphan the kernels the deadline is there to get deleted. The deadline moves to 260, the delete ceiling gets a name, and the harness test now derives every term from launch.py's constants instead of hardcoding one call at 180s, so it can fail when they move. The gate had the matching hole: no socket timeout, and the Kaggle client has none of its own, so a stalled call blocks forever, --soft-fail never sees an exception and the 10 minute job timeout reports red on a workflow whose contract is that only a failed assertion on a T4 is red. It now sets the deadline before the first call, as launch.py does, and the in-flight survey gets a wall-clock budget: running out of it stops the walk as incomplete, which is already a skip, rather than being killed by the job. tests/kaggle: 296 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep three non-payload outcomes from costing quota or a red check Three ways this workflow could spend a Kaggle session, or colour a pull request red, for something that is not the code under test. push() filed a slug and only published it by RETURNING. The retry loop handles the subprocess timeout, so every other raise (subprocess.run with text=True decodes strictly and raises UnicodeDecodeError on a malformed response; the runner can answer OSError or MemoryError) unwound past the line that recorded the entry, and release() then iterated a list with no entry for that notebook at all. A kernel Kaggle may have just accepted was left billing to its own ceiling. The list is now the caller's: main() publishes the entry before the push and push() fills it as it files each slug, so reconciliation no longer depends on the call returning. max_steps is free text on a dispatch and was forwarded unchecked. A value that is not an integer reaches the payload's argparse, which exits 2 with no report, and the generated cell reports that crash as a failing leg on purpose, so three legs come back red after the kernels were pushed and the model downloaded. A value below the fp16 scaler's leading skipped steps applies no optimizer update at all, and the payload's own checks fail it. check_steps.py answers both before the recheck, and MEASURES the floor as the shortest prefix of the committed reference that optimisation_failures accepts (5, on the committed trace) rather than restating a number that would go stale as soon as the reference moved. The build step also takes the value through the environment now rather than interpolating it. The evidence upload had no continue-on-error, so an artifact-service outage failed the job, and the job is what the pull request shows. The verdict is Report's, which runs under always() and can still conclude pass. It is best effort now, and says so out loud: continue-on-error leaves outcome at failure while conclusion becomes success, and the step after it reads that outcome and warns. tests/kaggle: 302 passed. * Bound the four checks that were stated rather than measured Evidence collection had no deadline. The job timeout counted 300s per kernel for it, a number no code enforced: one listing walks OUTPUT_PAGE_LIMIT pages at the 120s socket ceiling, 2400s for a single kernel, and each executed notebook is another 300s with no cap on how many Kaggle lists. Measured against the launcher at the previous head, two kernels can spend 14520s there, against 600s budgeted, and the phase runs BEFORE release(), so the runner is killed with billable kernels up. launch.py now starts one EVIDENCE_BUDGET_SEC budget before the first collection, shared by every kernel, clamps each listing page and each download to what is left of it, and marks a collection truncated. The job deadline reads that constant instead of restating a per-kernel term, and deadline=None now means a fresh budget rather than no ceiling. The reference band never checked the card. The reference records gpu_name "Tesla T4" and gpu_capability "sm_75" because a loss trace belongs to its hardware -- no bf16, xformers attention -- and every other reference-defining property already refuses before comparing. Only the GPU COUNT was checked, on the kernel, which cannot see the file. A P100 run against the committed T4 trace returned status "ok" with no failures, so a hardware difference read as a code regression. check_reference now gates on the reference's own environment block, so a recaptured reference moves the gate with it, and "it does not say" stays unchecked rather than a mismatch. adapter_config.json was checked for being JSON. "{}" parses, so a save that wrote no LoRA fields at all scored config_readable and the leg passed on a directory PEFT cannot load. The question is whether PEFT can rebuild the adapter, so it is asked of PEFT on the path a reload takes, PEFT_TYPE_TO_CONFIG_MAPPING[peft_type].from_pretrained; PeftConfig.from_pretrained alone returns peft_type None for "{}" and reports nothing wrong. The rebuilt config is then compared against the arguments the payload handed get_peft_model, so a well-formed config for a different adapter is a difference rather than a field list this file guessed at. The version table omitted the packages the legs move. legs.py records the frontier resolution installing huggingface_hub 1.27.0 and datasets 5.0.1, and the tokenizers and safetensors ceilings that forced deps back on, none of which were in GOAL_PACKAGES, so a failure caused by one produced a comparison table in which nothing differed. They are recorded now. The related hole is that pin_failures answered from a table keyed on that fixed list: a pin outside it had no entry and came back "it is not installed" for a package installed and correct. The probe list is derived from the pin file, and an unprobed pin is its own third outcome rather than an invented failure. pytest tests/kaggle: 315 passed. The workflow's recorded count is updated and its harness step installs peft, without which the two new adapter checks importorskip and cannot fail. * Enforce three bounds that stopped at the edge of the thing they bound Evidence deadline: urlopen's timeout is per blocking socket operation, not a ceiling on the transfer, so an endpoint that keeps returning bytes renews it forever and one resp.read() outlasts the whole 600s budget while every deadline check sits before the call. Measured against a body arriving in 20 chunks a minute apart: 1211s listing, 1260s download. Read in chunks with read1, the absolute deadline re-checked before each and the live socket re-clamped to what is left, streaming downloads to disk instead of into memory. A listing or a notebook abandoned that way now says the evidence is truncated. Reference hardware: check_reference compared gpu_name and gpu_capability only when the run supplied them, and main() records environment = {"error": ...} when the probe raises, while the fingerprint omits every gpu_* key outright when torch.cuda.is_available() is False. Either way the live values were absent, both keys were filed as unchecked, and the control leg reported an ok reference check without establishing the T4 the trace belongs to. What the reference does not say stays a skip; what the run cannot say about a key the reference does name is now hardware_unverified and fatal. Harness step: peft depends on torch, so installing it first satisfied that from the default index and the CPU-index line that followed found the requirement already satisfied and installed nothing, leaving the CUDA set on the runner in the window reserved before the push. CPU torch goes first, as version-compat-ci does, and the result is checked with torch.version.cuda rather than assumed. tests/kaggle: 323 passed. * Report an exhausted weekly Kaggle quota as a failure, not a skip Every gate stand-down was a green skip, exhaustion included, so a week with no accelerator hours left looked exactly like a workflow nobody had wired up: t4-smoke skipped, no annotation, nothing to read. The quota floor now exits nonzero carrying GPU capacity exhausted - please wait until next week - you can ignore this CI failure followed by the numbers the quota API already returned (hours left, the weekly total, and the refresh time), in the log line and in the job summary, so the reader can see when it clears and that there is nothing for them to fix. Only genuine exhaustion. It is decided where it already sat, before the concurrency survey and before any kernel is pushed, so it costs one API call rather than a Kaggle session. An UNREADABLE quota stays a green skip: unknown is not exhausted. So do the sampling draw, the wrong opt-in label, the missing token, a failure to authenticate, a busy account and the launcher-side stand-downs, all with their wording unchanged. --soft-fail becomes a request rather than a default. It defaulted to True, which would have made the new red unreachable, so it is now three states: unset, --soft-fail, --no-soft-fail. An error in the gate itself is still a skip unless --no-soft-fail is passed, exactly as before. The gate job asks for the red and nothing swallows it. The recheck inside t4-smoke passes --soft-fail and stands down green instead: it runs after approval with the account slot in hand, so reaching it means the hours went while the run queued, and red there would be the same message after a runner, the harness suite and the kernel build. Ten tests pin it: the exact sentence, the exit codes on both sides of --soft-fail, that the survey never runs once the quota is short, that an unreadable quota and a busy account stay green, and that nothing downstream turns the gate job's failure back into a pass. 333 pass. * Reserve what a run can bill, and assert two results that were only recorded Three checks that stopped at the edge of the thing they bound. The gate reserved 4 GPU-h against a launcher whose billable window is the whole invocation, not the polling. A kernel bills from the moment Kaggle accepts it until a delete is confirmed, so the push retries (each one discarding the previous attempt's slug), the evidence phase and release() are inside that window as surely as the wait is: launch.py's own constants put one invocation at ~13800s, and Kaggle runs at most --kernels sessions for this account at once, each billing its wall clock once. That is 2 x 13800s = 7.7 GPU-h. The reservation is now 8, and the harness suite recomputes the bound from launch.py rather than checking the number, in the same helper the job deadline is derived from. The old assertion read budget >= kernels x --max-wait, which 4h satisfied while a wedged run could bill 7.7h and take the 20h reserve down with it. The reference identity pinned the configuration and not the rows. canary_dataset.jsonl is inside this workflow's paths filter, so editing it TRIGGERS the run that would then be band-checked against a trace of the old rows: a small edit passes the tolerance and reports green on a comparison that means nothing, a larger one is reported as a code regression. The payload now records a sha256 of the parsed rows in config.dataset_digest and check_reference refuses on it exactly as it refuses on max_steps, the digest of the committed dataset is in the committed reference, and a harness test compares the two on the runner before any session is paid for. Digesting the parsed rows rather than the file's bytes means reformatting does not force a session-costing recapture; changing a row, the order or the count does. The gpt-oss leg recorded placement_after_load on every run and asserted on none. Its documented result is that the 20B checkpoint fits and trains wholly on one T4 with about 1.8GB to spare, and that is the claim that degrades quietly: a loader or memory regression that spills to CPU, disk or meta still logs finite losses, still moves the adapter, still compiles and still generates. failures_for now fails on parameters off the visible CUDA device, on hf_device_map offload, and on a placement record it cannot read, the last on the same three-way rule the bf16 reading already gets. accelerate's offload does not support training at all, so a run that reaches it is not a slower version of this leg. tests/kaggle: 345 passed. * Report a dependency import that exits the process The verify cell's import probe caught Exception, so an import that ends in sys.exit() went straight past it: SystemExit derives from BaseException expressly so that code catching Exception does not swallow it. The cell then aborted before writing the report the probe exists to write, the run cell below it never ran, and a leg that reports nothing is partial or infra at the launcher, both of which exit 0. transformers is a live carrier rather than a hypothetical one: it defines OptionalDependencyNotAvailable as a BaseException subclass and raises it at module scope, and its own lazy loader re-raises only Exception. So catch BaseException here, as the GPU probe a few lines below already does for the same reason, and re-raise KeyboardInterrupt: that is the runner cancelling the job, not a broken dependency, and recording it as one would make a cancelled run read like a regression. The cell is executed in the tests rather than pattern matched, so the verdict is observed the way the launcher will read it. * Resolve the dependencies of the distribution under test pyproject.toml is in this workflow's trigger paths because the payloads install the commit under test as a distribution, and every leg then installed it with --no-deps. pip enforces the requirements of packages IN a resolution and merely warns about the rest, so with the tested distribution outside every resolution the file could change its dependencies and nothing here could tell: a requirement it adds is never installed, one it tightens is never checked against what is already there, and the import probe passes anyway whenever the dependency is reached by a delayed code path. A user meets all three at pip install unsloth. The --no-deps rationale was that the overlay must not walk the set unsloth_zoo had just resolved. What unsloth actually declares is typer, rich, pydantic, pyyaml, nest-asyncio, structlog and click: none of them is resolved by zoo and none is pinned by a leg, so there is nothing to walk. A pyproject that does name one of zoo's packages would move it, which is the regression this exists to show rather than a side effect to suppress. Resolving is only half of it, since pip reports a conflict with something ALREADY installed as a warning and exits 0. So the verify cell also asks pip check, and reads only the lines pip attributes to the distribution under test: the Kaggle image carries pre-existing conflicts of its own, and the frontier leg installs a transformers that zoo's metadata forbids on purpose, so the exit code belongs to the environment while only those lines belong to this leg. An unsatisfied requirement is written out as a verdict, like the two checks above it, because the run cell is the only other thing that reports and is never reached from there. The name the check keys on is read off the requirement the legs install, so the two cannot drift into checking nothing. * Count a kernel Kaggle says is not there as released Cleanup reconciles every slug a push filed, not only the accepted one, and most of those were never accepted: push() files a fresh slug per attempt, and a retry's _discard() deletes the previous one without recording that it worked. So release() asks Kaggle to delete slugs that were never created, and slugs that are already gone. A not-found answer settles the only question cleanup asks, which is whether the slot is still billing. Reading it as a failed cleanup spent DELETE_ATTEMPTS on an absent kernel, ahead of the accepted one that is the only kernel still running, and then named it in a "may still be running, delete them by hand" warning pointing a human at a slug that does not exist. The warning is the one signal this emits, so a false one costs more than the calls. The gate already reads a 404 as a kernel definitively not running, for the same reason and against the same account, so this reads it through the gate's GONE_MARKERS rather than a second list that could drift. The pinned client surfaces one as "404 Client Error: Not Found for url: ..." on stderr with exit 1: kagglesdk calls raise_for_status and cli.py prints the HTTPError. Every other nonzero exit keeps its retries. A 5xx, a reset connection or an argparse refusal says nothing about whether the kernel is up, and trusting the exit code alone would abandon a live session silently. * Refuse to push kernels the runner may not live to delete The job deadline is set to the launcher's worst case plus about 30 minutes for everything that runs before it, and nothing enforces the second term. A checkout, a pip install off a slow index, the harness suite and three network resolutions have no cumulative limit of their own, so the allowance is an assumption about their duration rather than a property of the run. Spend it and the launcher still pushes, and the runner is then killed with kernels up: GitHub sends SIGINT to the step's entry process and kills the process tree about ten seconds later, which is not a window in which release()'s DELETE_ATTEMPTS retries can finish. A kernel nobody deletes bills accelerator quota to its own ceiling with nobody reading the result, which is the exact outcome the deadline exists to prevent. So the launcher is handed the moment the job dies and answers the question itself, before authentication and long before the first push. Less than its own worst case left and it pushes nothing and stands down green, which is what this workflow's failure semantics give every infrastructure outcome: no quota spent, and nothing learned about the code. That worst case now lives in launch.py, computed from the constants that produce it, because three consumers derive from it and all three are wrong by the same amount if a phase is dropped: the job deadline, the quota the gate reserves, and the new guard. The harness suite keeps deriving it independently from the source text and asserts the two agree, so a term lost on either side is a red test rather than a run that pushes with no room to clean up, or one that stands down on every invocation. The start of the job is recorded by the job's FIRST step. Taken any later, the deadline computed from it sits after the real one, and the guard reads optimistic in the one direction that costs quota. The minutes are restated in the launch step's environment because a job cannot read its own timeout-minutes; the suite asserts the two agree. * Ask PEFT what the saved adapter should be called The saved adapter is read back with safetensors and counted, which answers whether the bytes deserialize and not whether PEFT would consume them. PEFT matches its state dict by NAME: set_peft_model_state_dict ends in load_state_dict(..., strict=False) and nothing reads the returned unexpected_keys, so tensors under names it does not recognise are ignored without a word. Measured on peft 0.20.0 against a real saved adapter: strip base_model.model. from every key, or leave the adapter name in (lora_B.default.weight, which is what filtering model.state_dict() by hand produces instead of calling get_peft_model_state_dict), and PeftModel.from_pretrained raises nothing while every lora_B comes back at zero. The file still deserializes, still holds the same nonzero lora_B matrices, still carries a loadable adapter_config.json, so every reading this payload took off the bytes is identical to a healthy save and the leg passes on an adapter that reloads to the base model. The oracle is PEFT's own answer for this run: save_pretrained writes exactly get_peft_model_state_dict(self, ...), so calling it on the model that was just saved reproduces the key set the file is supposed to hold. Nothing is restated here, no prefix and no target-module list, so a legitimate peft renaming moves both sides at once and only a save that disagrees with the running peft is a difference. Three answers, and they are not one failure. A key PEFT names that the file does not carry is a dropped weight. A LoRA tensor under a name PEFT does not use is a weight its loader ignores. A non-LoRA tensor beside the adapter is recorded and not failed, since save_pretrained may legitimately write an embedding or a modules_to_save copy that PEFT never has to match. A run that could not derive the key set at all fails rather than recording an unchecked field, or the strongest thing this asserts would switch itself off silently. * Compare the step count the validator parsed, not the string it was given check_steps.parse_steps accepts "+10", "010" and surrounding whitespace as the ten they are, and the payload's argparse agrees, so those spellings run exactly the reference's run. The build step compared the raw dispatch string against the reference's count instead, read an identical run as a different one, and dropped the committed reference band from it: a green run with the only cross-run check in this workflow never applied, announced in a warning naming a step count that was not different. Both sides of that comparison now come from the step that validated them. check_steps emits the parsed count, and the reference's own count through the payload's reference_step_count rather than a second reading of the same JSON in shell, which also takes the reference filename back to the leg registry. The regression test executes the build step, on the outputs a real check_steps run wrote, with the env resolved from the step's own env block, so a build step that goes back to the raw dispatch string fails it. * Recheck the cleanup window after auth, and keep undecodable payload output from losing the verdict The launcher measured the job's remaining wall clock once, before authenticating, and pushed on that answer. With KAGGLE_API_TOKEN, the only credential the workflow passes, kaggle 2.2.4 authenticates by introspecting the token over HTTP, bounded only by SOCKET_TIMEOUT_SEC, so a window that fitted by less than 120s was already gone at the first push and the runner could be killed during release() with kernels still billing. The check is now a helper asked twice, before and after authentication. The generated run cell decoded the payload's stdout and stderr with subprocess.run(text=True), which is strict: one malformed byte from a native crash handler raised UnicodeDecodeError before the synthetic report below it, papermill aborted the cell, and the launcher called a leg that died partial or infra, both green. Decoding is now errors="replace". --------- Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 04:47:08 -07:00
Daniel Han	cb07e02d7b	Studio tests: pin the torch probe invalidation contract to both installers (#8795) ... * Studio tests: pin the torch probe invalidation contract to both installers #8779 made pip_install_try() drop the memoized torch classification, because it installs the Windows AMD ROCm trio and a memo it does not clear can answer for the build it just replaced. That fix shipped with no test: only pip_install() was covered, so reverting the pip_install_try() line kept the suite green. Three tests, each of which fails with that line removed: - pip_install_try() drops the memo, mirroring the pip_install() case - a real reinstall is really reclassified: two real torch packages on disk and a real probe subprocess either side of a real call into each installer, so the answer that replaces the memo is shown to come from the venv as it is now rather than from the mock that dropped it - every function that builds a pip or uv install command invalidates. Read out of the module with ast rather than listed here, so a third installer helper cannot be added without either invalidating or failing this test No production change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the torchao consumer to the reinstalled torch as well Folds in the one test #8796 had that this branch did not. The other two there duplicate what is already here, so #8796 closes in favour of this. _select_torchao_spec reads _probe_installed_torch_version() between the two repair points, which makes it the consumer that would actually read a memo left over from before a reinstall, and pin torchao against the torch that was just replaced. The existing cases assert on _probe_torch_runtime directly; this one goes through the accessor that the torchao path really calls. Fails with the _invalidate_torch_runtime_probe() line removed from pip_install_try, along with the three siblings, and passes with it. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>	2026-08-14 04:46:23 -07:00
Daniel Han	9c69529705	Windows: stop compiling C# for colour on hosts that already render it (#8767) ... * Windows: stop compiling C# for colour on hosts that already render it Enable-StudioVirtualTerminal is called unconditionally by install.ps1 and studio/setup.ps1, and it reaches Add-Type, which runs the C# compiler and drops a source file in %TEMP% on every install. An ANY.RUN submission of the shipped 0.1.701-beta Windows build captured that as two csc.exe processes and a "Suspicious source code drop". Under Windows Terminal there is nothing to enable: it always renders VT. Ask for that case first and skip the compile. All three conjuncts are load-bearing. WT_SESSION is inherited, so the desktop app's console-less spawn carries it into a pipe, and without the redirect check the Studio log panel would fill with escape sequences. $Host.UI.SupportsVirtualTerminal reports what the host CAN render, not whether this output buffer has ENABLE_VIRTUAL_TERMINAL_PROCESSING set, so it cannot carry the decision alone either. Nothing else moves. Outside this one function both scripts are identical to main line for line, and $script:StudioVtOk is the only value the function feeds, so the same verdict means the same bytes. The other compile stays. UnslothStudioFinalPathV2 feeds Get-StudioRuntimePathHash, which Python derives the same mutex name from byte for byte, so a managed fast path differing on case or an 8.3 name would let two installers each believe they hold the install lock. Guards: test_installer_av_shapes.py fails if the compile moves back ahead of the host check or loses a conjunct, and test_windows_setup_output_encoding.py runs this function beside the one it replaces on a real Windows host, with WT_SESSION forced set and forced empty, asserting the same verdict and the same banner bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Decide the redirected case without the compiler, not the Windows Terminal one Review caught that the WT_SESSION test was unsound. WT_SESSION is inherited, so a run launched from Windows Terminal into a NEW legacy console, which is what an elevated install gets, carries it with stdout not redirected and a buffer that has no ENABLE_VIRTUAL_TERMINAL_PROCESSING. SupportsVirtualTerminal reports host capability rather than the state of that buffer, so the branch would have claimed VT and printed literal escape sequences. There is no sound way to learn the current buffer's mode without GetConsoleMode, which is the compile. So decide the other direction instead: a redirected stdout is not a console, GetConsoleMode fails on a non-console handle, and the compiled path could then only return $false. Return it directly. This is provably identical rather than probably identical, and it covers the case that was actually measured: install.rs spawns install.ps1 with a pipe, so the desktop install is exactly where the compile was happening. Also drops the env plumbing from _run_console_less. It is lru_cached, so a dict argument would have raised TypeError before PowerShell was ever spawned, and the Windows parity job would have failed rather than proving anything. The parity case no longer needs it: the console-less probe IS the redirected case, so the early return is the branch under test rather than a bystander. * Reconstruct the exact merge-base function in the VT parity test The regex stripped only the guard and left the four comments above it behind, so the reconstructed predecessor was merge-base code plus comments rather than the merge-base function. Comments do not execute, so the comparison was still measuring the right thing, but a test that says it compares against the real predecessor should do that. Verified both files now reconstruct byte for byte. Also drops a stale WT_SESSION reference from an assertion message, left over from the design this PR replaced. * Tighten the comments this PR adds --------- Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 04:44:37 -07:00
Daniel Han	eb367015cb	Studio: close seven detection gaps in the deliberate-crash guard (#8788) ... * Studio: close seven detection gaps in the deliberate-crash guard Follow-up to #8783. Seven holes in the AST scan, all reproduced against synthetic files before changing anything. Three of them fail CI on correct code, which is the worse direction for a guard to be wrong in. False positives, code that is already safe but was reported: * A script assembled through an annotated assignment (SCRIPT: str = SUP + ...) was not folded, so the crash half was judged alone with no suppression beside it. The fold now walks every Assign and AnnAssign rather than the module body. * Any string literal counted as a candidate child script, so an ordinary expectation such as assert "ctypes.string_at(0)" in script was reported as a deliberate crash. A string now only counts if it reaches a call argument, directly or through a name, a list, or a concatenation, which is what handing it to a child interpreter looks like. * A crash function calling a suppress_core() helper was reported, because the helper body sits outside the crash function's scope. One level of local helper is now followed. Factoring the suppression out is good practice and must not fail. Misses, code that dumps a core but passed: * The prefilter reused the exact crash markers, so ctypes.string_at( 0) was skipped before it was ever parsed. It now matches the call name and lets the AST decide precision, which is the right split of responsibility between the two. * Suppression counted anywhere in the scope, so a prctl placed after the fault passed. It must now precede the crash line. * Only symbolic signal names were recognised, so raise_signal(11) and os.kill(pid, 6) slipped through. Numeric forms count too. * Script snippets were matched textually, so from os import abort; abort() was missed. Snippets are Python, so they are parsed and run through the same call detector, which also picks up aliased and differently spaced forms for free. Ten new fixtures, one per hole plus the negatives that must stay quiet: Playwright's route.abort(), a variable re-raise, and os.kill with SIGKILL, which never dumps. 19 passed in 1.32s. 32 of 1049 files parsed, 833ms scan. Reverting the two guarded files to their pre-#8783 state still fails the guard and names both. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Close seven more detector gaps in the deliberate-crash guard for PR #8788 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Crash guard: close four more detection gaps, decline four Four of the eight review findings were reproducible misses, each verified against the guard before and after: * A command vector bound to a name. `CMD = [sys.executable, "-c", SCRIPT]` then `subprocess.run(CMD)` resolved nothing, because the name environment folds only strings and a list does not fold. That is ordinary subprocess usage, not a contrivance, so the child script went unread entirely. * A crash imported under an alias. `from os import abort as die` binds `die`, the rules are keyed on `abort`, and the alias map was a set of bound names with no way back to the rule. The file was also skipped before parsing, since the prefilter looks for call shapes and an aliased call contains none, so the alias handling could never have run. Both ends fixed: the map is now bound name to rule name, and the prefilter matches import spellings. * `obj.suppress_core()` credited to an unrelated local `def suppress_core`. Helper following now requires a bare local call. * An `async def` suppressor that is called but not awaited builds a coroutine and runs no prctl, so the fault after it still dumps. Only awaited calls count now. Declined, with reasons rather than silence. Three findings (a name reassigned after the call that uses it, a nested scope rebinding a name used at module level, two same-named helpers in different scopes) are one underlying issue: resolution that ignores position and scope nesting. Fixing it properly needs a flow-sensitive analyser, and fixing it partially is how a guard starts failing CI for innocent code. The triggering shapes are contrived, and this is a heuristic backstop rather than a soundness proof. The fourth, an exec snippet inside an already-suppressed snippet, is exotic and errs toward a loud false positive rather than a silent miss. Seven cases added to the table, two of them negative, so the four fixes cannot regress and the two behaviours that must stay allowed (awaited async helpers, rebound aliases) are pinned. Prefilter still admits 32 of 1049 files, so the widening costs nothing. Stale timing in the comment corrected: the scope and alias analysis added over this PR moved the full pass from 496ms to ~1.5s. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-14 04:12:45 -07:00