2 commits

Author	SHA1	Message	Date
oobabooga	495ba21ebd	Studio: add MiniMax H3 video generation (#7989) ... * Add MiniMax H3 video generation * Improve MiniMax H3 memory routing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address MiniMax H3 review feedback * Keep H3's VAE off the CPU path, so low_vram stops aborting low_vram maps to the `model` policy, and offload_flags emits --vae-on-cpu for it unconditionally. On H3 that kills the process: ggml/src/ggml-cpu/ops.cpp:6321: GGML_ASSERT(src0->type == GGML_TYPE_F16) failed deterministically, SIGABRT, exit 134. Bisected on the flags: --vae-on-cpu with --audio-vae aborts, the same command without --audio-vae renders in 95.87s, and an fp16-converted audio VAE aborts too. So the trigger is the audio VAE, not the video one, and the F32 type is imposed inside stable-diffusion.cpp rather than by the file: ggml_conv_1d hardcodes an F16 im2col destination (ggml/src/ggml.c), ggml_compute_forward_im2col_f16 then asserts the KERNEL is F16, and audio_conv_weight_type (src/model/vae/ltx_audio_vae.hpp) maps only BF16 to F16 and lets F32 through. It cannot be fixed by shipping a different checkpoint. low_vram is the one mode a small-card user reaches for, so this drops the flag rather than the mode. offload_flags takes vae_on_cpu, defaulting True so no other family changes, and the H3 native path passes False. --offload-to-cpu and --clip-on-cpu still apply, which is where the saving actually is: the denoiser dominates, and with --offload-to-cpu the whole model peaks at 13.14 GiB. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Point the H3 GGUF pick at the unsloth mirror Main added test_curated_gguf_repos_are_unsloth_mirrors, which requires every curated video gguf_repo to live under unsloth/ so a one-click pick cannot 404 when a community repack is renamed or taken down. H3 was the one family still on a community repo. This was meant to be part of the merge commit but was left in the worktree, so CI on that commit still saw the old value. unsloth/MiniMax-H3-GGUF is still private and has to be made public before this merges, or the pick will 401. No CI check reads it. * Pin H3's native cfg-scale under test H3 is distilled and CFG-free: its empty unconditional prompt encodes to zero tokens, and the transposed tensor that produces trips GGML_ASSERT(!ggml_is_transposed(a)) in ggml.c. SIGABRT, exit 134. Measured: cfg 1.0 renders, cfg 1.5 and cfg 4.0 both abort. sd.cpp defaults cfg-scale to 7.0, so this is a crash a plausible refactor reintroduces by forwarding guidance the way every other family does. The native path already hardcodes 1.0 and the family sets supports_cfg = False, but nothing held either in place. supports_cfg only gates the diffusers path; the native path builds its own params. Checked the test fails when cfg_scale is changed to forward guidance, so it is not passing vacuously. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin diffusers by source archive instead of git, so macOS installs work The macos-15-intel leg fails deterministically, not flakily: process didn't exit successfully: `/usr/bin/git init` (exit status: 1) --- stderr xcode-select: note: No developer tools were found, requesting install. That runner has no Xcode, so /usr/bin/git is the developer-tools shim and exits 1 for everything. uv needs a working git to resolve a git+https dependency, so the diffusers pin this branch added cannot install there at all. Main is unaffected because it depends on plain diffusers. GitHub serves the same commit as a source archive, which uv installs with no git involved. Verified by stubbing git to fail exactly the way the macOS shim does: the git+ form reproduces the CI error, the archive form installs diffusers 0.40.0.dev0 from the same SHA with MiniMaxH3Transformer3DModel present and exported. Also drops a full clone of diffusers from every install. * Allow the diffusers source build on the clean-machine legs With the archive pin the macOS leg gets past `git init` and installs, but then trips the nobuild guard: built from source: diffusers -- these must resolve to wheels on a clean machine There is no wheel to resolve to. MiniMax-H3 support is not in any diffusers release, so this branch has to pin a commit, and neither a git URL nor a source archive can produce a wheel from an index. Added to the same allowlist that already carries triton-kernels for the same reason. Checked against the bar the comment there sets: diffusers builds with plain setuptools, declares no ext_modules, and its tree has zero .c/.cpp/.pyx/.rs/.cu files and no shipped binaries, so the PEP 517 build is a pure-Python copy step and needs no toolchain. The separate compiler-invocation check in both scripts is untouched and still fires if one is ever needed. Verified the allowlist logic still rejects a non-allowlisted source build (a log building both diffusers and numpy reports only numpy). Remove this entry once a diffusers release carries H3 and the requirement goes back to a version specifier. Noted in both scripts. * Baseline the hf-hub retry loop reopened by the 1.x upgrade `pip scan-packages :: hf-stack` fails on this branch with 1 unbaselined CRITICAL: C2 polling/beaconing loop detected Package: huggingface-hub File: huggingface_hub/utils/_http.py L461: while True: This is on us, not upstream drift. Main pins huggingface-hub==0.36.2; this branch needs >=1.23.0,<2.0 because diffusers at the pinned commit requires it, so the resolved version moves 0.36.2 -> 1.27.0. The baseline already carries this exact file and check at L462 and L298, from earlier versions. It did not carry over because the key hashes the matched code, not the line number, and the surrounding code changed across the major version. That is the guard behaving correctly: changed code in a baselined file reopens for review rather than staying suppressed. Reviewed it rather than just re-suppressing. L461 is the retry loop in `http_backoff`: bounded by `nb_tries > max_retries`, the URL comes from the caller, there is no hardcoded endpoint and nothing is exfiltrated. Same benign construct as the entries it replaces. Entry generated with the scanner's own _evidence_hash rather than hand-written, and inserted beside its siblings so the diff stays 8 lines. Verified: with the baseline the scan is exit 0 with 4 suppressed, without it exit 1, so the guard still bites. * Lift the macOS-arm huggingface-hub cap that this branch made unsatisfiable `mac macos-15 / trace / file` fails on this branch and passes on main. uv reports: No solution found when resolving dependencies: Because you require huggingface-hub>=0.34.0,<1.0 and huggingface-hub>=1.23.0,<2.0, your requirements are unsatisfiable. This branch moved base.txt, no-torch-runtime.txt, studio.txt and constraints.txt to hub >=1.23.0,<2.0, because diffusers at the pinned commit requires it, but left the flat <1.0 cap in overrides-darwin-arm64.txt. That file is macOS-arm only, which is why only the mac legs see it and Linux and Windows stayed green. The failure then presents as something else entirely: uv gives up, the installer falls back to pip, and the clean-machine trace fails on "installer invoked toolchain: rustc" rather than on the resolution. The cap's own comment explains it exists so the resolver can never pair hub 1.x with a pinned transformers 4.57.6 / hub 0.36.2 stack. That is still true below python 3.10 and still capped there. At 3.10 and above this branch is on transformers 5.5.0 and hub 1.x, so the premise is gone, and mlx-audio's own >=1.0 floor is satisfied by the 1.x window anyway. Checked by collecting every hub specifier that applies per Python version across all five files: py3.9 resolves to 0.36.2 as intended, py3.13 to 1.23.0/1.27.0. Before this change py3.13 resolved to nothing. * Make H3's native download use the repo the family advertises The curated-mirror test main added only inspects VideoFamily.gguf_repo. H3's native path does not read that field: video_minimax_h3.py has its own H3_GGUF_REPO constant, used for both the transformer and the Qwen3-VL encoder. So pointing the family at the unsloth mirror in the previous commit satisfied that test while the actual one-click download still came from a community repack, which is the exact failure the test exists to prevent. Pointed the constant at the same mirror and added test_the_h3_native_repo_matches_the_family_gguf_repo to pin the pair, so the two cannot drift apart again. Verified it fails when the constant is put back to leejet, so it is not passing vacuously. The mirror now carries the Qwen3-VL encoder quants alongside the denoisers, byte-identical in size to the community ones, so this repo alone satisfies both of h3_native_hub_files' hub entries. The encoder is part of MiniMaxAI/MiniMax-H3 itself (FL2VA/text_encoder) which we already mirror publicly at unsloth/MiniMax-H3 under the same licence, so shipping a quantization of it beside the denoisers is the same act. Also checked the encoder-tier routing survives the dynamic rung names: -UD-Q2_K_XL selects the Q2_K_M encoder and -UD-Q3_K_XL the Q4_K_M one, asserted in the new test. Updated the download-plan test, which hardcoded the old repo id. NOTE: unsloth/MiniMax-H3-GGUF is private. Unlike before, that now really does gate this: the native path downloads from it. It has to be public before this merges. * Pin H3's companion-checkpoint guard under test validate_h3_transformer_filename had no test. That mattered less when the denoisers lived alone; the mirror now ships the Qwen3-VL encoder quants in the same repo, so the picker lists both and a user can name either. Loading a 12-17 GB encoder as the transformer would fail deep inside sd-cli instead of at the boundary. The accept cases include the dynamic rung names on purpose. The guard is a prefix/suffix check and `-UD-Q2_K_XL` is a shape it had never seen when it was written; it happens to pass, and now that is asserted rather than assumed. Checked the test fails when the prefix check is dropped, so it is not vacuous. * Record why H3 drops --vae-on-cpu, now that the abort is fixed The comment justified the drop entirely by an sd.cpp abort. That abort is fixed in the Unsloth fork, which would have made this look like a stale workaround to revert once the fix reaches the pinned prebuilt. Measured on a build carrying the fix, 640x384, 25 frames, 4 steps, q4_K, with --offload-to-cpu --clip-on-cpu already applied: adding --vae-on-cpu moved peak VRAM 12.42 -> 12.42 GiB and wall time 20.9s -> 100.4s. Under --offload-to-cpu the peak is set by the streamed denoiser, so the flag saves nothing and costs 4.8x. It stays off on its own merits. * Pin the sd.cpp prebuilt that actually renders MiniMax-H3 The pin was master-812-ea7f0c8, a stock upstream build, and on a stock build H3 does not work: it aborts on the default --cfg-scale, aborts again on --vae-on-cpu, and its 1-D norms are quantized into an output uncorrelated with its own bf16 reference (LPIPS 0.981). The Studio side worked around the first by pinning cfg to 1.0 and the second by dropping the flag; the third had no workaround on the consumer side at all. All three are fixed in unslothai/stable-diffusion.cpp and open upstream as leejet/stable-diffusion.cpp#1861, #1862 and #1863. The mirror's prebuilt pipeline now applies them on top of the aged upstream tag it already builds, and marks such a build with a -u<id> suffix naming the patch set, so master-813-bfbef5b-u0665242 is upstream master-813-bfbef5b plus those three patches and nothing else. The patches are deleted once upstream releases them, at which point this pin goes back to a plain tag. Verified on the published Linux x86_64 asset, not on a local build: both new error strings are in the shipped sd-cli, and running it on a q4_K H3 denoiser without --mode vid_gen now exits 1 with the instruction instead of core dumping on a ggml assert. test_video_backend's fake engine returned the old tag as its version string, which read like a second pin; it only needs a non-None value, so it now says so. * Close two gaps the H3 mirror switch opened Both are consequences of the two preceding commits, found in review. The prebuilt pin is now mirror-only (master-813-bfbef5b-u0665242), and _resolve_with_fallback still asked leejet for it. That request is a guaranteed 404 by construction, since the -u<id> suffix marks a build only the mirror makes, so it was a wasted round trip on every install. Worse, when the mirror genuinely cannot serve a host the fallback lands on leejet's latest, which has none of the H3 fixes. For every other model that is the right trade, better a stock native engine than none. For H3 it is not visible: it aborts on the default cfg-scale, aborts on --vae-on-cpu, and a blanket --type renders a broken video rather than failing. A user who saw only the generic 'falling back to leejet' line had nothing connecting that to the output. It now says so. Second, hub/utils/gguf.py filtered H3 companion GGUFs by the old community repo id only. The mirror the family and catalog now advertise carries the Qwen3-VL encoder quants beside the denoisers, so a 12 GB text encoder was being aggregated as if it were a selectable transformer quant. Both bundle repos are now recognised, case-insensitively, and the cache-dir match follows. Tests are mutation-verified rather than assumed: restoring the upstream 404 attempt fails the ordering test, removing the H3 warning fails the fallback test, and dropping the mirror from the bundle set fails three. The ordering test deliberately makes the mirror serve nothing, because with the mirror serving the first attempt succeeds and the upstream attempts are never reached, which made an earlier version of it pass under its own mutation. A third test pins the native loader's H3_GGUF_REPO to the bundle set, since those live in different files and a future repo move that updated only the loader would silently reintroduce the same aggregation bug. * Exclude MiniMax-H3's small-M projections from int8 _INT8_FAMILY_EXCLUDE_NAME_TOKENS has entries for qwen-image and hunyuanvideo-1.5 but none for minimax-h3, and H3 needs one for the same reason they do. H3's adaLN projection is named adaln_proj, which no token in the generic list matches: 'norm' is the closest and does not appear in the name. On the dense checkpoint that projection is Linear(2688 -> 96768), so it clears min_features = 512, gets quantized, and then runs at M = 1. Inductor lowers int8 matmul to _int_mm, which requires M > 16, so it raises 'self.size(0) needs to be greater than 16, but got 1' at the first denoise. The offline prequant builder bakes it in happily, which is exactly the drift the exclude list exists to prevent for Flux and Qwen. The pruned-modulation form hides this rather than fixing it: there adaln_proj is Linear(8 -> 96768) and falls under min_features anyway. So this exclusion is what makes the DENSE path correct and is a no-op on the pruned one. context_embedder and token_refiner are added for the same reason hunyuanvideo-1.5 excludes its text stream. Measured at M = 10 text tokens against the video stream's thousands, they are 3.47% of GEMM time even in the slow eager int8 path, so leaving them bf16 costs nothing measurable. This is what made an earlier measurement conclude int8 does not work on H3. It does: on the pruned form int8 compiles and is 4.24% +-0.54% faster than fp8 at identical memory, paired over 12 renders. * Say which H3 component could not be downloaded, and why H3 pulls four files from two repos, and the Hub returns the same 'Repository Not Found ... make sure you are authenticated' for a repo that does not exist, one that is private, and one your token does not cover. A user reading that has no way to tell which of the four failed, and the wording points away from the real cause whenever the repo exists but is not public. That is the state the GGUF mirror is in today: it is unpublished, so picking H3 fails with a message suggesting the user fix their token, which will not help. This replaces it with the repo, the component, and the actual remedy, and says the other components are unaffected so the failure is not read as total. Gated repos get different wording, since accepting a licence is a different action from waiting for a repo to be published. Anything that is not a recognised access error is passed back unchanged rather than reworded, so a timeout or a full disk still reads as itself. The helper returns the exception instead of raising, so the caller keeps raise-from and the original traceback survives. Mutation-verified three ways: rewording every error (timeouts included), fixing the component name to 'denoiser', and giving gated repos the private wording each fail the test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop MiniMax-H3 holding two copies of its video VAE Two thirds of an H3 render's peak is not activations. Measured at 640x384 across 124 frames, a 20.25 GB int8 denoiser peaks at 36.96 GB, and the gap is almost all weights: the video VAE alone is 10.42 GB because diffusers pins it to float32, and a further 4.91 GB is autocast's own float16 copy of those same weights. A memory snapshot puts 92.9% of the transient in 437 blocks allocated from nn.Linear, largest 67.1 MB, which is the decoder's [2048, 16384] SwiGLU projection in float16. MiniMaxH3VideoDecodeStep wraps vae.decode in torch.autocast(float16), and autocast caches every weight it casts for the lifetime of the region, so the float32 original and its float16 twin are both resident through the whole decode. Storing those weights as float16 up front makes the cast a no-op and removes both. This is not an approximation: x.to(float16).to(float16) is x.to(float16), and the four regression tests check that on a real matmul rather than on the reasoning. The audio VAE decode is not under autocast, so it keeps float32. t2va starts from noise and never encodes, so vae.encoder and vae.quant_conv go too. That part is gated on the workflow name rather than dropped unconditionally, because an image-conditioned workflow needs them. Measured 36.96 -> 28.37 GB peak with the encoder drop and the pre-cast, 28.27 with expandable_segments as well, over 5 prompts x 2 seeds. Speed is unchanged (-0.05% +-0.86% eager, -0.34% +-2.23% compiled), and every arm hashes identically to its control on latents, audio_latents, frames and audio. The estimator's base still reads 68.5 GB. That figure was measured on the bfloat16 modular components, not the int8 arm above, so it stays put until it is re-measured in the same configuration rather than adjusted by arithmetic. * Pad MiniMax-H3's small-M int8 linears instead of leaving them dense torch._int_mm asserts self.size(0) > 16. torchao's eager path never trips it (safe_int_mm falls back to a widened matmul), but inductor lowers the same quantized linear straight to _int_mm, so any quantized Linear invoked at a small activation row count crashes under torch.compile. Until now the fix was to leave those linears dense bf16, which on H3 meant excluding context_embedder and both token_refiner blocks: 13 linears, 798M parameters, 0.80 GB of weights the int8 checkpoint was not allowed to touch. Pad instead. diffusion_quant_pad.PadToMinM pads the flattened row count up to 32, runs the GEMM and slices the result back, so the module compiles with no change to the quantization config and the caller's rows come back bitwise unchanged. Verified bitwise on all 65 (module, M) cases across H3's 13 linears at M = 10, 13, 14, 17, 19, on real torchao-quantized weights; compiling those same modules unpadded raises the _int_mm assert at M = 10, 13 and 14. Two properties carry that exactness and both are asserted rather than assumed. The pad rows replicate row 0, not zeros: an all-zero row has amax 0, so the activation quantizer divides by zero. And the activation scale must be per row, so each kept row's scale comes from that row alone; a quantized Linear whose granularity cannot be proven per row raises instead of being quietly skipped, because a half-padded transformer compiles on the wrapped modules and crashes on the rest. Everything below pad_to normalises to pad_to rather than only what is below the floor, so one inductor graph covers every prompt length in the range. H3's seven eval prompts run at M = 10..19, which straddles the floor, and padding only to 17 would leave three shapes behind. The wrapper reparents the Linear, so it runs after quantize_ on the runtime path and after load_state_dict on the prequant one. The offline builder drives quantize_ directly and saves the state dict, so it never sees a wrapper; PadToMinM is also state-dict transparent as a second line of defence, saving and loading under its own prefix so a wrapped transformer still writes context_embedder.weight. Scoped to minimax-h3. qwen-image, qwen-image-edit and hunyuanvideo-1.5 have the same small-M shape but published int8 prequant checkpoints whose metadata bakes the current exclusion set, and _validate_checkpoint compares that set against exclude_tokens_for_scheme, so they move only together with a rebuild. adaln_proj stays excluded for a different reason: on the dense checkpoint it is Linear(2688 -> 96768) and runs at M = 1, while on the pruned form it is Linear(8 -> 96768) and falls under min_features anyway (verified: the filter rejects all 51 for min_features). Measured on B200, 640x384 x 124 frames, 4 steps, 7 prompts x 2 seeds, the two arms alternated within each cell so drift on a shared box cancels: checkpoint 21.052 -> 20.254 GB (-0.798, -3.8%) transformer 21.051 -> 20.253 GB (-0.798) load peak 21.137 -> 20.336 GB (-0.801) render peak 37.766 -> 36.966 GB (-0.800, -2.1%) step time +0.0597 s +-0.0057 eager, +0.0533 s +-0.0086 compiled The time is torchao's un-fused eager activation quantization on 13 modules that run ONCE per render, so it is a fixed cost rather than a per-step one, and it does not scale with steps or resolution. Compiling those modules alongside the blocks removes it: +0.0034 s +-0.0040, no detectable difference. Quality is unchanged as far as n = 14 can resolve: against the same bf16 twin the padded arm sits +0.0070 +-0.0151 LPIPS from the excluded one, which rules out a degradation larger than 0.022. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Classify the MiniMax-H3 GGUF bundle as video in the cached inventory The H3 GGUFs are stable-diffusion.cpp conversions and carry no metadata keys at all (kv_count 0), so general.architecture is absent where the LTX-2 and Wan video GGUFs declare ltxv/wan. _arch_to_task therefore left the downloaded repo with no task, which drops it from the Video picker's On Device list and hands it to chat as a plain GGUF. Key the two bundle repo ids before the arch is consulted. * Release the VIDEO GPU claim when H3 native falls back to the CPU build On a CUDA/ROCm host /video/load acquires the VIDEO arbiter owner because the resolved device target is not CPU. _run_load_h3_native then asks for an accelerator-matched sd-cli, and the pinned prebuilt release publishes no Linux CUDA/ROCm archive, so ensure_sd_cpp_binary returns None and the load commits the CPU build with native_device = cpu. Nothing dropped the VIDEO claim, so the next chat or image acquire evicted and unloaded an H3 runtime that holds no VRAM. Release the claim once the CPU fallback is committed, through release_if so the token check is atomic against a newer load that already took ownership. Mirrors the CPU-only native release /images/load already does. * Video: protect the native H3 companion repos, cancel the modular denoise, forward the hub token - VideoBackend.loaded_repo_ids() publishes the repos the committed native H3 runtime re-reads every generation (Qwen encoder + both VAEs), and the delete-cached guard consults it, so deleting a companion under a loaded model is refused. - The H3 modular workflow no longer falls back to a null progress context: the denoise loop drives pipe.scheduler.step once per step, so the existing wrapper gives it the same per-step progress and cancellation the other callback-less pipeline gets. - load_components() gets the Settings token, so gated/private component loads are not issued anonymously. * Reject stable-diffusion.cpp builds that predate MiniMax-H3 support ensure_sd_cpp_binary hands back whatever find_sd_cpp_binary locates and only probes that it runs, so an install upgraded from an older Studio kept serving its pre-H3 managed sd-cli. The H3 load's only gate is SdCppEngine.version(), which that binary passes, so the load reported ready and the failure surfaced on the first generation, after the whole bundle had downloaded. Gate the H3 path alone on the capability instead of a version string: upstream added --ref-video and the other H3-only options in the same commit that added MiniMax-H3 (leejet/stable-diffusion.cpp#1854, master-812-ea7f0c8), and the release prebuilts report 'version unknown, commit unknown' because they are built without a .git directory, so --help is the only usable signal. Image generation keeps accepting any user-supplied build. A stale copy under the installer-owned root is removed so the pinned prebuilt reinstalls; a build the user supplied is left in place and the load fails naming it, the same ownership split _usable_or_discard_managed makes. A --help that cannot be read means 'cannot tell', never 'no H3'. * Distinguish a reused CPU sd-cli from an accelerator build on an H3 load On a Linux CUDA host the first H3 load installs the CPU prebuilt through the fallback and correctly commits native_device = cpu, because the pinned tag publishes no Linux CUDA, ROCm or Vulkan asset. Every later load then calls ensure_sd_cpp_binary(accelerator = cuda), which finds that same CPU binary and returns it without looking at what it was built for, so the fallback was skipped, native_device stayed cuda, and Studio applied GPU offload policy and retained the VIDEO gpu_arbiter claim while sd-cli ran wholly on the CPU. A later chat or image acquire then evicted an unrelated GPU model. Fall back on what the binary offers rather than on whether one was returned: sd-cli --list-devices prints one name/description line per available ggml backend device, so a CPU-only build answers with CPU alone. The second load now reaches the same cpu conclusion as the first, which is what lets the existing release_if drop the stale claim. Unreadable output, or an older build that rejects the flag, keeps the GPU: neither says the accelerator is missing. * Guard the H3 companion repos while a native video load is downloading _run_load_h3_native downloads from repo_id, the H3 GGUF companion and the H3 component repo, but the in-flight state only carried repo_id and base_repo. The cached-model delete guard reads loading_repo_ids(), so it allowed deleting Comfy-Org/MiniMax-H3, and the GGUF companion when the load comes from another mirror or a local file, while those files were still downloading, which fails the load. Carry the companions on the loading state, the way the image backend's _SdLoading already does, and publish them from loading_repo_ids(). This is the in-flight twin of loaded_repo_ids() and covers the same repos. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep curated Recommended rows searchable and give them their metadata Two things went wrong with the curated video models the Recommended list paints from the catalog rather than from a live Hub listing. Search dropped them. The Recommended search matched the query against `recommendedIds`, which filters out every id already on disk because a downloaded model gets its own On Device row. The unfiltered Recommended list does not filter that way: it renders the curated seeds and badges the downloaded ones. So a curated model was visible in the list and unfindable by typing its name the moment it was downloaded, and only a live listing row could bring it back, which a repo the listing does not return never gets. `searchableRecommendedIds` unions the seed ids with the listing ids, seeds first, deduped case insensitively (the HF cache lowercases repo ids), so both lists agree on what exists. Rows rendered bare. Everything past the id came from the listing alone, so a curated row the listing never returns showed no parameter chip and no capability glyph while its neighbours showed both. The catalog now carries the two facts nothing else can supply: `totalParams` per artifact and `capabilities` per group, read through `curatedTotalParamsFor` and `curatedCapabilitiesFor`. Both are fallbacks only. A listing row wins wherever there is one, because real tags and a Hub-reported total outrank anything hand written here. The parameter counts are measured, not guessed. The MiniMax-H3 figure is the sum of the tensor shapes in the BF16 GGUF of the FL2VA denoiser that repo publishes; the LTX-2.3 figure is what the Hub reports for its repo, carried so the row looks the same offline or rate limited. One more inconsistency fell out of the same place: `searchRowFits` hides anything it cannot size (`requireKnown`), and it could not size a curated repo with no listing row and no "<n>B" token in its id, so turning on "Fits on device" hid from search a row the unfiltered list still showed. It now falls back to the curated total the same way it already fell back to the curated size. Covered by studio/frontend/tests/recommended-curated-row-metadata.test.ts: 15 assertions over the search pool, the two catalog lookups, the fit check, and the four picker call sites that read them. * Load MiniMax-H3 from a hosted pre-quantized denoiser Video families had no way to reach a hosted pre-quantized DENOISER. VideoFamily carried gguf_repo and te_prequant_repos (the text encoder) only, and nothing in video.py consulted a denoiser table, so a MiniMax-H3 load either pulled the full 66.3 GB bfloat16 DiT or nothing at all. The image side already solves this with DiffusionFamily.prequant_repos plus family_prequant_repo(), so this follows that shape rather than inventing a second one. Four parts. VideoFamily gains prequant_repos, prequant_variant_repos and prequant_subfolder, resolved by video_family_prequant_repo() / video_family_prequant_schemes(), mirroring the image resolver. The registries stay separate, as the module header requires, so the base-id normaliser is local rather than imported. The shared resolver learns an optional subfolder. The hosted video checkpoints nest theirs one level down instead of keeping it at the repo root, and the prefix has to reach BOTH candidate names or the primary 404 is followed by a second one and the load silently falls back to the dense download. Always a literal forward slash: these are Hub repo paths, and a Windows join would miss the cache. The cache and download plumbing already handled a nested name, so only the filename builder changed, and every existing call site is byte-identical. The modular workflow builds its denoiser through its own component loader, so there is no dense module to quantise in place. A hosted checkpoint is therefore the only way to run that transformer quantized, and pre-seeding it with update_components() before load_components() is also what stops the dense download: load_components(names=None) skips a component whose attribute is already set. Passing names= instead would have forfeited the workflow's own block pruning, which is what already avoids the 61.7 GB Ref2VA transformer. Those checkpoints carry the pruned adaLN, where the modulation is a rank-8 affine factorization of the time-embedding curve plus a shared table, and roughly 40% of the released model's parameters go. Against the base repo's dense config the model is four keys short, one over and fifty-one shapes wrong, so the strict load fails and the checkpoint is unloadable by every route. video_minimax_h3_adaln.py reshapes the model between from_config and load_state_dict: table lookup with interpolation instead of the timestep MLP, no SiLU (the table already holds the activation's own output projected onto the basis), and the modulation cast to the block stack's dtype, without which the first quantized matmul dies on mismatched dtypes. Bound per instance, so a dense load in the same process is untouched. Finally the refusal becomes honest. A single-file pick on a modular family used to reach the loader only after ~98.7 GB had downloaded AND after the resident pipeline had been evicted to make room for it, because download-plan returned 200 and validation passed. Both refusals now run in validate_load_request, ahead of the diffusers availability probe so they still fire where diffusers cannot be imported, and each names what to pick instead. download-plan forwards transformer_quant to validation and to the plan, without which the quant-keyed refusal never fires on the route that stages the download and the plan stages shards the load never opens. * Point both MiniMax-H3 schemes at one hosted pre-quantized repo The two hosted pre-quantized denoisers were split across two repos with the checkpoint nested one level down, so reaching them needed a mechanism the image side has never had: a VideoFamily.prequant_subfolder field, a prequant_subfolder_prefix() normaliser, and a subfolder keyword threaded through both resolve_prequant_source() and usable_prequant_source(). Both schemes now live in ONE hosted repo, at the root, named <Model>-<SCHEME>.pt. That is the layout every image-side prequant repo already uses, and it is exactly what prequant_repo_filename() builds unaided, so the whole mechanism goes. Match an existing convention and the code should shrink, not grow: -22 lines in diffusion_prequant.py, -6 in video_families.py, -6 in video.py, with no new concept to carry. Landing on the primary name also fixes a memory-planning under-credit. cached_checkpoint_path() deliberately credits only the PRIMARY filename, so that a cached legacy artifact cannot pin a stale name once a repo ships the real one. While these checkpoints were published as transformer_<scheme>.pt the primary never existed: every hit came through fallback_filename, and planning therefore read an already-cached checkpoint as "this would have to download" and handed the pick to GGUF. The primary is now the published name, so the probe hits it. fallback_filename stays. It still covers repos that have not been renamed, and dropping it is a separate decision from this one. Tests: the subfolder-prefix cases are replaced by the naming they now guard -- both schemes resolving to one repo, the primary resolving to a root-level <Model>-<SCHEME>.pt with no directory component on any platform, the cache probe being asked for that primary name, and the repo's own scheme suffix being stripped and replaced rather than carried through. Five mutations run, each caught by the named test and reverted: two repos again (M1), the primary nested under prequant/ again (M2), the suffix strip removed (M3), the cache probe keyed on the fallback (M4), the fallback name dropped (M5). * Add MiniMax-H3 image and reference video conditioning * Improve H3 finalization progress * Report real sd.cpp progress on the Video page instead of a frozen 0 of 30 A native (GGUF) video generation reported phase "denoise", step 0/30 for its entire run and then flipped straight to "completed". Two separate things were wrong, and the progress endpoint could not move until both were fixed. The parser looked for r"(?:step|sampling)\D+(\d+)/(\d+)". sd-cli's sampling bar contains neither word. It prints |=========> | 7/30 - 21.50s/it so nothing ever matched. Anchor the pattern on the bar and on the trailing speed unit instead. A bare "n/m" is deliberately not enough: an unrelated ratio in some other log line must not drive the progress bar. The reader also delivered every redraw one step late. sd-cli redraws in place, and its carriage return LEADS the record while the newline only arrives on the final step, so a reader keyed on CR/LF cannot produce step 1 until step 2 has been flushed. Treat the erase-to-end-of-line that closes each redraw as a terminator too, and read the pipe with buffer.read1 so a record that carries no newline is not stuck behind a blocking readline. Escapes are stripped before a record reaches on_log or the error tail. Streams without a raw .buffer keep the old line iteration. The same bar shape is printed by three different things, so the parser now tells them apart rather than reporting whichever came last. Weight load prints it with a byte rate, and tiled VAE decode prints an identical s/it bar counting TILES: without a guard a run finished sampling at 30/30 and then jumped backwards to "step 1/16". Load and decode are real work with no sampling step, so they report their own phase and a null step rather than a fake 0 of 30. ETA is measured from the first step, not from job start, so the one-off weight load is not charged to every remaining step. Verified end to end against a real CUDA sd-cli MiniMax-H3 generation: the step advances 1..6 over 6.2s..9.7s of wall clock, load and decode are reported as themselves, and the ETA tracks. * Read sd-cli's in-place progress redraws so the Video bar moves during sampling The native H3 progress bar had two independent causes and fixing either alone changed nothing observable. The bar pattern is now correct, but the reader still was not. sd-cli redraws its sampling bar in place: one printf per step shaped "\r<bar> <n>/<total> - <speed>\033[K", with a newline only on the final step of a phase. The drain loop did `for raw in proc.stdout`, which terminates on LF only, so every redraw sat in the buffer until the next one's carriage return arrived and the last one until sampling was already over. The Video page saw nothing. Split the raw pipe into records on CR, LF, or the trailing erase-to-end-of-line, reading through buffer.read1 with an incremental UTF-8 decoder so a multi-byte character straddling two reads survives, and strip the CSI escapes before the record reaches on_log or the error tail. Streams without a raw .buffer (test doubles, non-pipes) fall back to line iteration. The new backend test drives the real byte stream through both halves, one flush per read, and asserts each step is visible on the read that carried it rather than one redraw later. * Give MiniMax-H3 first and last frame conditioning in the video backend MiniMax-H3's released transformer is the FL2VA one: text-to-video is the same checkpoint run with no keyframes. Studio only ever ran it text-only, so the Video generate request had nowhere to attach a reference frame. The load used to prune the block graph to t2va. That argument prunes STATICALLY, so an fl2va-pruned pipeline runs the keyframe blocks on every request and cannot serve a text-only one at all: it raises packing an empty conditioning list. The load now keeps the whole auto graph, which selects per request, and bounds load_components to the keyframe workflow's component set instead, so the 61.7 GB Ref2VA partition is no more loaded than before. Measured against the released checkpoint: a text-only request through this pipeline is bit-identical, video and audio, to the same request through a t2va-pruned one. A keyframe is a geometry anchor, not just conditioning, so the canvas comes from its aspect ratio through the released arithmetic (768 short edge, area capped at 768x1344, both axes rounded to 32) rather than from whatever resolution preset was selected. An arbitrary size produces a garbled clip rather than an error. sd.cpp already implements the same conditioning, so that side is the existing --init-img / --end-img flags with the frames staged as PNGs. Only MiniMax-H3 declares the capability, and status reports it, so Wan and LTX do not grow a control that does nothing. * Cover the MiniMax-H3 keyframe path with tests Registry (which families declare it, and the canvas a keyframe resolves to), the load wiring (whole block graph, component set still bounded, VAE encoder kept), request handling (decode, refusal, canvas override, what reaches the pipeline call) and the sd-cli argv. * Add the reference-frame controls to the Video page First and last frame pickers, shown only for a family whose status declares keyframe conditioning, so Wan and LTX are unchanged. The Images page's source picker moves to a shared component rather than the Video page growing a second upload path; both send the same data URL to the same backend decoder. While a frame is attached the Resolution preset is disabled and says why: the frame's aspect ratio resolves the clip's size, the way the model itself does. The gallery recipe records which ends were pinned. * Check the keyframe canvas against the pipeline's own resolver The canvas rule is a checkpoint contract, so pin it to the released implementation rather than only to hand-written expectations. Skipped where diffusers does not ship MiniMax-H3, which is most runners. * Revert the standalone H3 keyframe implementation oobabooga/unsloth#121 covers first frame, last frame, first-and-last, Ref2VA and the canvas rule, and it reached the same load construction independently. Two implementations of the same feature on one branch is worse than either, so this takes mine back out and leaves the branch ready for that work to land whole. The one finding worth keeping from it is already reflected there: passing workflow= to ModularPipeline.from_pretrained prunes the block graph statically, so the pipeline must be built unpruned and only load_components bounded. * Keep the pre-quantized MiniMax-H3 denoiser resident so a generation can run Loading H3 with a hosted pre-quantized denoiser worked, but every generation died on its first denoise step: Attempted to set the storage of a tensor on device "cuda:0" to a storage on different device "cpu". This is no longer allowed; the devices must match. ComponentsManager.enable_auto_cpu_offload parks every component on the CPU and moves each one onto the accelerator inside its own pre_forward, that is from within the block that is already executing. The text encoder and the VAEs survive that; a torchao-quantized denoiser does not, because the device change reaches return_and_correct_aliasing, which tries to alias a CPU storage to an accelerator tensor. Moving the same module at load time, outside any executing block, works. So place it once at load and take it out of the offload rotation: drop its hook, unlist it from the other components' eviction candidates, and move it. Everything else is unchanged, and the encoder and VAEs still offload around it. Keeping it resident is what asking for a quantized denoiser buys in the first place: the hosted checkpoint is about 20 GB against 66.3 GB dense. Verified end to end on a B200: MiniMaxAI/MiniMax-H3 loaded with the hosted fp8 denoiser, then a 1280x768, 124-frame clip generated from a start frame in 167s. The clip's first frame matches the supplied image and the motion is coherent. * Apply the pinned Diffusers revision on a fresh install, not just an update MiniMax-H3 needs a Diffusers revision newer than any published release, and Studio refuses to load it otherwise. The pin was in studio/backend/requirements/base.txt, and a clean install.sh run still ended up on diffusers 0.39.0 from PyPI, every time, with nothing in the log to say so. base.txt is never installed by install.sh. install.sh installs unsloth itself, whose own metadata pulls a diffusers release in transitively, and then runs install_python_stack.py with SKIP_STUDIO_BASE=1, where the base-packages step is a bare `pass`. So the pin applied on `unsloth studio update` and on the no-torch path (install.sh installs no-torch-runtime.txt directly) and was dead on exactly the path a new user takes. Reproduced on a clean install into a throwaway prefix before and after: 0.39.0, then 0.40.0.dev0 with MiniMaxH3Transformer3DModel present. The revision now lives in its own diffusers-pin.txt, installed by a step that sits outside every skip_base / NO_TORCH branch and after every other requirements file, so nothing left in the run can re-resolve diffusers back to a release behind it. No forced reinstall is needed: a direct URL requirement is not satisfied by a resident registry install, so the step is a no-op once the environment is already on the pin. tests/studio/install/test_diffusers_pin.py holds the shape in place: exactly one requirements file may name diffusers, the pin must be a full commit sha rather than a moving ref, the install step must sit at function top level rather than under a conditional, and it must come after every other requirements install. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: let the Video and Images pickers see a local diffusers pipeline A model already on disk never reached the pickers' On Device list unless it happened to keep a weight file beside a root config.json. Every image and video model downloaded as a pipeline keeps its weights in component subdirs under a root model_index.json instead, so the hub inventory scan behind /api/hub/local rejected it. Pointing Custom Folders at one was worse than empty: the LM Studio publisher walk descended into the pipeline and offered vae, transformer, text_encoder and audio_vae as four separate models, none of them loadable. Teach the hub scanner the same pipeline-root test routes/models.py already applies, in the three places that only accepted a root config plus loose weights, and keep a pipeline row through the custom-folder format filter: the layout has no loose weight to classify, so the row is "unknown" by construction rather than by fault. * Cover the local diffusers pipeline scan with tests * Pin what the pipeline exemption must not let through Three gaps in the cover added with the scan change, each found by mutating the fix and watching the suite stay green. The custom-folder format filter now waves a row through on its shape, and nothing said what it still has to reject: replacing the whole predicate with True passed. A folder holding a config.json and no weights, which an aborted download leaves behind, reports the same "unknown" format and no loader can start it, so it pins the boundary. The predicate is applied to every row the filter did not already accept, and a row's path can be a GGUF file rather than a directory. A missing path, a file, and a directory whose model_index.json is itself a directory must answer False rather than raise, because an exception there fails the scan and empties the picker. The publisher walk was only covered one level up. Adding the model folder itself as a scan folder is the obvious thing to do and used to publish vae, transformer and text_encoder as three models. * Drop the unused H3_TASK_KEYFRAMES import from the video backend video.py only branches on H3_TASK_REFERENCES; the keyframe constant is read from video_minimax_h3 directly by the tests that need it. The hoisted-import safety net in Source lint flags the unused name as a blocker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage the hosted pre-quantized H3 denoiser in the video download plan The plan already drops the dense transformer shards whenever a hosted pre-quantized checkpoint covers them, but nothing put that checkpoint back: an int8 or fp8 H3 stage skipped 66.3 GB of base shards and added none of the 20.25 GB artifact the load actually opens. The byte total under-reported the stage by the size of the checkpoint, the disk preflight cleared a volume that could not hold it, and an offline stage completed without the one file the load needs. _denoiser_prequant_hub_files mirrors the pre-cast encoder helper: it resolves the family's hosted checkpoint, confirms the file really exists on the Hub, and prefers the repo-root name over the legacy scheme name in the same order the load tries them. An unreachable repo is logged and yields no files, so a gated or renamed artifact keeps the dense shards instead of sinking the plan. The progress-bar estimate is deliberately left alone: it counts cached bytes for the checkpoint and base repos only, so adding a third repo there would leave the bar permanently short of 100 percent. * List both H3 denoiser partitions in the picker, not only FL2VA The bundle-repo filter accepted only minimax_h3_fl2va*, so every published minimax_h3_ref2va* quant was hidden from the variant listing. The loader disagrees: validate_h3_transformer_filename accepts either partition, on the grounds that which one is picked IS the task, and h3_transformer_task routes Ref2VA to the reference-video workflow this PR adds. The community bundle repo publishes three Ref2VA quants today, so the reference path was unreachable from the remote catalog. Accept both prefixes from one shared tuple and keep excluding the Qwen3-VL encoder and VAE companions, which are never picks for either partition. The filter test asserted the old behaviour and is updated with it. * Bound H3 reference-audio decoding to the trained window The reference-video decoder already selects, resizes and refuses incrementally because the encoded size says nothing about the decoded size. The audio decoder did not: it appended every resampled block to a list and then allocated a second full buffer in np.concatenate, with no duration or sample cap. The route accepts 32 MiB of encoded audio, which is over half an hour of compressed stereo. That lands as roughly 1.9 GB of float32 and doubles again in the concatenate, and up to three references are accepted per request, so an ordinary long music or podcast file picked by mistake could exhaust the host before the background job even started. H3's reference window is 15 seconds, so anything past it is unusable rather than merely large. Refuse it while decoding, with the same shape of message the video guard uses, instead of decoding it first. * Refuse a quantized H3 reference load instead of seeding the keyframe denoiser The hosted pre-quantized checkpoints are FL2VA (keyframe) denoisers. Ref2VA shares their module shapes and the same base model, so resolve_prequant_source handed one back for a reference load, it passed every metadata check, and seeding it made load_components skip the real Ref2VA transformer. The request then generated from the wrong partition rather than failing, which is the worst of the three outcomes. The route accepts h3_task, so this was reachable from the public API even though the picker does not expose the choice yet. validate_load_request now refuses the pairing with a message naming the workable alternatives, in the same place it already refuses a scheme with no hosted checkpoint, and the modular loader drops to the released components if a direct call reaches it. Nothing changes for keyframe loads, which are what the checkpoints are. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Skip modular-workflow families in the dense text-encoder plan assertion The plan-unchanged sweep from main walks every family and asserts the dense budget plan_diffusion_memory received. MiniMax-H3 is the first modular-workflow family to reach that list, and load_pipeline dispatches to the workflow's own loader before the planner runs: each component is built by its own from_pretrained, so there is no single dense pipeline to budget and no plan call to assert on. Skip it the way the sweep already skips wan2.2-t2v-a14b. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope the paired-axes canvas rule to keyframe requests A half-specified width/height is long-standing, documented API behaviour: the backend resolves the missing axis from the family's default preset, in both validate_video_request_shape and _resolve_keyframes. Applying the new paired-axes rule as an unconditional request validator rejected those calls with a 422 before family validation ever ran, breaking existing LTX, Wan, Hunyuan and prompt-only H3 clients. The rule still holds where it means something: with a keyframe present the canvas is matched to the source aspect whenever either axis is missing, so the axis the caller sent would be silently discarded. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the CUDA sd-cli pin and translate a mirror-only tag upstream The merge with main brings in the accelerator-aware installer, whose fallback translates a mirror-only -u<id> pin back to the upstream release it was built from instead of skipping the upstream attempt. That is strictly better: skipping kept the round trip cheap but dropped the pin entirely on every host the mirror does not build, leaving them on upstream latest. test_a_mirror_only_pin_is_never_requested_upstream asserted the old shape, that the fallback settled for upstream latest. It now pins the new one: never the literal -u<id> string, the translated release instead, and no latest attempt at all because the translated pin succeeds. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-08-09 10:05:58 -07:00
Daniel Han	b6781a8bfe	CI: prove the installer works on a machine with no developer toolchain (#7551) ... * CI: prove the installer works on a machine with no developer toolchain No job has ever run the installer on a machine without one. studio-mac-install-matrix.yml is the only macOS installer job and it runs 'bash install.sh --local --no-torch' on runners that already have the Xcode CLT selected and setup-python preinstalled, so the CLT gate never fires there, and --local is precisely the mode that legitimately needs git. Repo-wide there was zero coverage of xcode-select or CommandLineTools outside install.sh itself. clean-machine-install-ci.yml runs the installer on a genuinely stripped machine. macOS legs move /var/db/xcode_select_link, /Library/Developer/CommandLineTools, /Applications/Xcode*.app and Homebrew aside, so xcode-select -p, git, cc and clang really do fail, and restore unconditionally afterwards. Removing the select-link alone is not enough: xcode-select falls through to a full Xcode.app and re-arms /usr/bin/git. Linux legs use containers, which are genuinely clean. Windows legs cover winget visible and masked, plus windows-11-arm. A WSL leg covers the 126 lines of WSL-specific install.sh logic that had no runtime test. Each macOS leg runs four deliveries: pipe (the advertised command, and the shape that turns an early exit into curl (56)), file (separates installer logic from pipe delivery), no-torch, and tauri (stdin closed, no tty, as the desktop app invokes it). One leg records every toolchain invocation and asserts the trace, which is the real deliverable: proof the installer never reached for a compiler rather than proof it happened to succeed. The asserts test that tools do NOT WORK rather than that they are absent from PATH. On a real virgin Mac /usr/bin/git and /usr/bin/cc exist as CLT stubs, so 'command -v git' succeeds and only running it tells the truth. desktop-app-clean-machine-ci.yml installs and launches the SHIPPED desktop app release on a stripped machine, covering Gatekeeper and quarantine on macOS, NSIS silent install on Windows, and Xvfb with WebKit2GTK on Linux. Known limit, stated plainly: hosted macOS runners are developer machines. Masking reproduces this bug and proves the installer does not invoke a toolchain, but it cannot prove no hidden dependency exists on a truly virgin Mac. An ephemeral-VM lane is the follow-up. * Point the llama assert at the right root, and name the Intel limitation The tauri leg installs to the legacy root because --tauri refuses a custom UNSLOTH_STUDIO_HOME. Its install succeeds end to end, but llama.cpp lives at <root>/llama.cpp while the venv is at <root>/studio, so the assert was pointed one level too deep. On macos-15-intel /usr/bin/git keeps working once the CLT are gone, so it is not CLT-provided there and no masking can remove it, while cc and clang do become stubs. Calling that 'masking failed' was wrong. That leg allowlists git explicitly and says why, so the assert stays strict everywhere else. * Make the clean-machine legs able to fail The toolchain strip never ran on the automatic triggers: inputs exists only for workflow_dispatch, and GitHub coerces '' and false alike to 0, so `inputs.strip_toolchain != false` was false. Confirmed on a pull_request run where the strip step reports skipped. Gate on the event instead. Also: scrub the Machine and User registry PATH, since install.ps1 rebuilds $env:Path from them mid-install and the toolchain came back; stop dropping WindowsApps unconditionally, which removed winget on the winget=visible leg too; fail rather than annotate when a bundle ships no installer or no CLI; run the bundled installer, which a headless launch never reaches; resolve the newest desktop-v* release instead of a pinned immutable tag; and give the two macOS matrix rows distinct artifact names. * Make the Windows and Linux clean-machine legs honest The Windows scrub only touched PATH, so the legs were green while not clean: run 30365014702 logged "python ABSENT" and then "Python 3.13 already installed" with uv resolving C:\hostedtoolcache\windows\Python\3.13.14\arm64\python.exe. py.exe lives in C:\Windows and uv discovers interpreters itself, so take the toolcache off disk and fail when tooling survives, instead of only printing it. The Linux desktop legs never stripped anything, and the tauri.log step was all || true so it could not fail. Run the bundled installer the way install.rs does, with --tauri alone, and assert torch: passing --no-torch skipped the slowest half of first launch and let the venv check pass over it. Pin the WSL rootfs to a dated build; current/ is a rolling alias and the digest next to it is fixed. * Give the Linux and WSL legs an assertion that can fail The Linux rows' only post-install gate was nobuild, a log grep, so an installer exiting 0 having produced nothing kept a required leg green. The WSL job and the Windows job both already check the install runs; the Linux job now does too. The WSL detection half only printed its Select-String, and the alternation also matches "platform linux", so a regression that skipped every WSL-specific branch would still pass as a plain-Linux install. Assert the exact marker, stripping ANSI first since step writes the label in reverse video. Probed against three fixtures: real wsl log passes, platform linux fails, missing log fails. * Tighten the clean-machine comments Compress the comment blocks across the clean-machine workflows and scripts. The explanations of why each check is written the way it is stay; the padding, restatement and duplication go. No code or workflow logic changes. * Point the nightly at the repo that publishes, and let its checks fail REL_REPO defaulted to unsloth-test/unsloth-test, which holds one release frozen at 2026-07-27, while release-desktop.yml publishes into github.repository. The schedule was re-testing the same fixture forever and could never see a broken production bundle. The windows job carried a blanket continue-on-error, so its NSIS assertions could not gate. lipo -archs prints and exits 0 for a thin binary and `|| true` swallowed even that, so the architecture was never checked; fall back to file, which survives the CLT mask. And require the preflight disposition line rather than the mere existence of tauri.log, which setup_logging creates at process start regardless. * Stop four clean-machine checks from passing over a real failure Re-run `absent` after the install on the masked macOS legs. It only ran before, so an installer that quietly selected the Xcode CLT or installed a compiler left the leg green while every later source build could succeed, which is the one thing clean-machine-assert.sh says `absent` guards the whole run against. Fail the Windows simulation when py.exe can still start an interpreter. The launcher binary itself may stay, but Find-CompatiblePython probes `py` first (install.ps1:1130-1153), so an interpreter registered outside the two renamed toolcache directories gets reused and Python bootstrap is never exercised. Exempting `py` without ever running it left that unchecked. Propagate the WSL installer exit code. It was printed and discarded, and the CLI check does not compensate: install.sh links the `unsloth` shim (4174-4182) before it reports a failing studio/setup.sh (4219-4230), so a late setup failure leaves a shim whose --version succeeds. Run the bundled installer in the Linux desktop jobs. The launch step only proves the process stayed alive, and on a fresh home preflight reports not_installed and the app waits on the install screen, so both required rows passed after 90 seconds without ever touching the shipped install.sh. Locate the resource in the deb payload or the extracted AppImage, run it the way install.rs does, and require a managed venv that can import torch. * Prove the trace wrapper records before trusting an empty trace The `notools` check reads an absence: it passes when the trace file contains no compiler, git or brew invocation. A shim directory that never reached PATH produces exactly the same empty file as an installer that touched nothing, so the single leg carrying that assertion would stay green no matter what the installer did. "Verify the simulation actually took effect" only ran for mask mode, which left the trace leg with nothing checking its own instrumentation. Call git explicitly after sourcing the environment and require it to appear in the trace, then truncate the file so the self-test entry does not count against the install. The call has to be explicit because macOS reaches _has_working_git only under STUDIO_LOCAL_INSTALL (install.sh:2026), so no consumer leg on that platform probes git on its own. * Stop the Windows clean-machine check failing on its own probe exit code All three Windows legs failed "Verify the simulation took effect" with no ::error:: printed at all. The check itself was right: the mask step logged "masked toolcache python: C:\hostedtoolcache\windows\Python", python/git/cmake/cl were ABSENT, no `py -3.x` probe started an interpreter, and the winget assertions were satisfied. The step still exited 1. The cause is $LASTEXITCODE leaking out of the step. The last external command is the `py -3.13` probe, which is SUPPOSED to fail; Get-Command and Write-Host are cmdlets and never reset $LASTEXITCODE, and the runner appends `if ((Test-Path -LiteralPath variable:\LASTEXITCODE)) { exit $LASTEXITCODE }` to every pwsh step (actions/runner#351). So a clean machine reported failure, and because this step runs before Install, no Windows leg has ever reached the installer. Clear $LASTEXITCODE after the probe loop and end with an explicit exit 0. The leak detection is untouched: a surviving python/git/cmake/cl, or a `py -3.x` that actually starts, still exits 1. Also print each probe's exit code and output, so the next failure here explains itself instead of being silent, and label `py -0p` as what it is. The launcher reads the registry, which the on-disk toolcache rename cannot rewrite, so -0p keeps naming paths that no longer exist. Unlabelled it reads like a leak. Accept the Fedora leg's real outcome instead of a message that can be absent The fedora assertion only accepted the unsupported-package-manager hard exit. That is still what this ref's install.sh does, but the pending installer change replaces it with a warning that lets the install continue, at which point the old grep matches nothing and the step fails for the wrong reason. Handle both, strictly. If the log shows the newer "using prebuilt llama.cpp (missing:" warning, the Linux gate demonstrably did not hard-stop, and the only tolerated failure past that point is release lag: install.sh comes from this ref while unsloth comes from PyPI, and the released studio/install_python_stack.py has no "skip triton kernels when git is missing" guard, so it still fetches the git+https triton_kernels requirement on a machine with no git. Anything else after that warning fails the step. Otherwise the old hard-exit message is still required. A missing log, a bootstrap outage or any unrecognised failure all remain errors, and the step retires to a plain success assertion once a release ships the no-git skip. * Make the AppImage Linux row actually extract, and hold Linux to the macOS preflight bar The appimage row invoked the extractor by bare filename, and a command word with no slash is resolved through PATH rather than the working directory, so the extraction exited 127 and the bundled-installer assertion below it never ran. Prefix it with ./ so the row exercises what it claims to. The Linux log step also asserted nothing: it skipped a missing log with continue and discarded the grep with || true. The launch step only proves the process stayed alive for 90 seconds, and the bundled-installer checks do not exercise the Rust preflight path, so an app that hung before preflight completed passed both required Linux rows. Require the same desktop_preflight completed disposition= record the macOS rows already do. * Put the branch's own Python under test on the clean-machine legs install.sh and install.ps1 come from the ref under test, but they install unsloth from PyPI, which is the consumer path and has to stay that way. That left everything Python-side coming out of the released wheel: studio/setup.sh, studio/setup.ps1, studio/install_python_stack.py, and every requirements and constraints file those resolve through Path(__file__). A branch that changes constraints.txt or setup.ps1 therefore got a green run that proved nothing about the change, and some legs proved less than they looked. The Fedora assertion was already carrying a hand-written workaround for exactly this, tolerating a triton/git failure on the grounds that the released package lags the ref. Legs marked overlay: true now re-point the venv at the ref just before studio setup runs, through UNSLOTH_CI_SOURCE_OVERLAY: a --no-deps editable install of the checkout. That makes import studio resolve to the working tree, so the existing setup-script lookup finds the ref's setup.sh / setup.ps1 and install_python_stack reads the ref's constraints, with no other change to either installer. Not --local: --local additionally installs unsloth-zoo from a git+https URL, which genuinely needs git, and git absence is the whole point of the masked legs. The overlay resolves no dependencies and clones nothing, so it holds up with git, cmake and the compilers all gone. It is not a consumer knob either: no flag, no usage entry, ignored unless the variable names a directory with a pyproject.toml in it. Four legs stay on the released package deliberately, each for its own reason, recorded in the header: the mac pipe legs keep an end-to-end signal on what a user actually runs; the trace leg would otherwise answer its own question, since the editable build calls git through setuptools-scm's file finder; the non-root Linux leg dies before a venv exists; and WSL only ever receives install.sh, not a source tree. Two supporting fixes the overlay depends on or exposes: install_python_stack.py discarded uv's output whenever a step succeeded, so the nobuild assertion, which reads the install log, could not see a source build in the dependency phase at all. That is the phase that installs studio.txt, where an sdist-only dependency actually turns up, and it reported "built: none" regardless. It now echoes successful output under UNSLOTH_VERBOSE, matching what install.sh's run_install_cmd already does. nobuild now ignores "Building <name> @ file://" lines. A local-path build is something the caller pointed at, never a dependency resolution chose, and index dependencies always print <name>==<version>, so a real sdist from PyPI is still caught, including one named unsloth. Each overlaid leg also asserts it really was overlaid, so an unset variable cannot quietly put the whole matrix back on the released wheel. * Allowlist the triton-kernels pure-Python sdist, and record why Windows on ARM is red The two ubuntu2404 root legs went red at "Assert no source build" reporting triton-kernels. That is not a regression in what the installer does. Those builds have always happened; they only became visible now that pip_install stopped discarding uv's output on success, which is what finally let the nobuild check read the dependency phase at all. So the question was whether each build actually needs a compiler. Checked against the real artifacts rather than assumed: openai-whisper 20250625, randomname 0.2.1, argbind 0.3.9 -- no version of any of the three has ever published a wheel; antlr4-python3-runtime is pinned at 4.9.3, below the first release that ships one. All four sdists use setuptools.build_meta, declare no ext_modules, and contain no .c/.cpp/.pyx/.rs file. Already allowlisted, correctly. triton-kernels is the same category and was the only name failing. It is pinned to the triton repo's python/triton_kernels subdirectory; that tree is 75 files of Python, a four-line pyproject.toml, no setup.py and no native source at all. The kernels are Triton DSL compiled at runtime, not at install time. It is also a direct URL the installer names itself rather than something resolution picked, and only Linux reaches it. It belongs in the allowlist, so add it with that reasoning written down. The allowlist match is now lowercased and underscore-folded on both sides. The requirement spells the package triton_kernels while uv prints triton-kernels, and an allowlist that matched only one spelling would pass by luck rather than by intent. A plain pyarrow sdist is still caught. The two data-designer @ file:// plugin builds needed nothing: they are in-tree local paths, already dropped by the same rule that exempts the source overlay's own build. Separately, the windows-11-arm leg fails for a real reason and should keep failing. The ARM handling itself works, the log shows torchaudio being skipped and torch plus torchvision installing from wheels. What stops it is that pyarrow and hf-transfer publish no win_arm64 wheel at all, so uv falls back to their sdists and they fail on CMake configure and on openssl-sys wanting perl. That is a product gap on the platform, not a gap in the simulation, so the leg stays experimental and keeps reporting it. Record that above the matrix entry so the next reader does not re-diagnose it. * Exercise the bundled Windows installer, and stop mislabelling installer sources Four things that let a leg go green while proving nothing. The desktop Windows job installed the bundle and launched it, and that was all. On a fresh profile preflight reports not_installed and the app sits on the install screen waiting for a click, so the process happily stays alive for 90 seconds without the bundled install.ps1 ever running. A bundle that shipped no install.ps1 resource, or a broken one, passed this job -- which is the packaged app failure the workflow exists to catch. macOS and Linux already invoke their bundled script directly; Windows now does the same, via the resource NSIS laid down next to the exe, invoked the way install.rs invokes it, then asserts the managed venv exists and can import torch. Its timeout goes to 60 minutes because a full torch install on a Windows runner is the slowest of the three. A manual run that selects installer_source: published only redirected the macOS and Linux jobs. WSL kept copying the checked-out install.sh and Windows kept running the checked-out install.ps1, so a run asking whether the script on unsloth.ai works reported on this ref under the published label. Both now honor the selection; install.ps1 advertises its own unsloth.ai URL, so published has a meaning on Windows too. Both branches stay empty on pull_request and push, so automatic runs are unchanged. The push-to-main filter listed only install.sh, install.ps1 and this workflow, while the PR filter also covers setup.sh, setup.ps1, install_python_stack.py and the clean-machine helpers. A direct push touching those skipped the workflow entirely, so the post-merge backstop never ran for the files the source overlay was added to cover. The two lists now match. Neither filter covered studio/backend/requirements, even though the overlay exists precisely so a constraints change is resolved on a machine with no compiler and no cached wheels. The update-smoke workflows cannot stand in: they start from a preinstalled Python and full developer tooling. * Make the Linux and Windows desktop legs clean, and honour published on every macOS delivery The desktop workflow claims all three platforms are stripped, but only macOS and Windows had a strip step and the Windows one scrubbed the process PATH only. Both gaps let a bundle that needs a developer toolchain pass the one workflow whose premise is that it must not. Linux: the job ignored strip_toolchain entirely and ran the bundled install.sh with the runner's git, gcc, cmake and make in /usr/bin. clean-machine-env.sh now has a Linux --remove branch that moves the resolved tool binaries aside, recorded in restore.sh, and the job calls it plus `assert absent` after the apt step (the .deb install needs dpkg) and before the bundled installer, with a restore step to match macOS. The loop repeats per tool so a name present in both /usr/bin and /usr/local/bin is fully masked rather than half masked. Windows: rewriting $env:PATH does not survive the bundled install.ps1, which calls Refresh-SessionPath (318-337) and rebuilds $env:Path from the Machine and User registry values, and py.exe in C:\Windows reaches the toolcache whatever PATH says. Ported the on-disk toolcache rename, the Machine/User registry scrub and the py -3.11/-3.12/-3.13 start probe from clean-machine-install-ci.yml, so the strip is proven rather than assumed. Windows preflight: the log step was Test-Path, Get-Content and Select-String, none of which can fail, so an app that hangs before preflight passed on the 90 second liveness check alone. It now asserts a tauri.log exists and carries a `desktop_preflight completed disposition=` line, the same unconstrained check macOS and Linux already make. The disposition VALUE is deliberately not constrained: ManagedReady over an unbootable venv is the reported bug. installer_source on macOS: only the pipe delivery branched on it, so a `published` dispatch ran the checked-out script on six of the eight macOS rows while the run was labelled published. The script is now resolved once at the top of the Install step and used by the file and tauri deliveries; pipe still re-fetches through the live transport, because that is half of what it tests. Linux, WSL and Windows already honoured the input. Also shortened the comments across the changed files, keeping the reasoning that says why each check exists. * Run the Windows installer under PowerShell 5.1, the only shell a clean machine has The Windows Install step ran `& $script` inside a `shell: pwsh` step, so install.ps1 was executing under PowerShell 7. A genuinely clean Windows box does not have PowerShell 7: Windows ships powershell.exe (Windows PowerShell 5.1) and pwsh is a separate install that the hosted runner image happens to preinstall. So the one workflow whose premise is a machine that has never seen a developer toolchain was testing the installer under a shell that machine would not have, and no other Windows job anywhere in .github exercises install.ps1 under 5.1. Invoke it the way the desktop does (install.rs:325-339, and the bundled installer step in desktop-app-clean-machine-ci.yml): powershell.exe with -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Bypass -File. The pwsh step wrapper stays, since it is only the installer that has to be under 5.1. Calling powershell.exe with `&` keeps the output in the pipeline, so Tee-Object still fills logs/install.log, and $LASTEXITCODE after the pipeline is the child's real exit code, so $rc and `exit $rc` are unchanged. install.ps1 and studio/setup.ps1 hold no PowerShell 7-only constructs: no `#Requires` above 5.1, no `&&`/`||` chain operators, no ternary, no null-coalescing, no ForEach-Object -Parallel, no $IsWindows/$PSStyle, and no 6+ cmdlets or parameters. setup.ps1 declares `#Requires -Version 5.1`, and its three $PSVersionTable branches gate a 7-only preference on the 7 side with a 5.1 fallback. Every Invoke-WebRequest already passes -UseBasicParsing, which 5.1 needs because it otherwise reaches for the IE engine. * Assert the Windows desktop strip actually took effect The desktop job's Windows masking renamed the toolcache Python, scrubbed the Machine and User registry PATH, and probed `py`, but nothing checked that `python`, `git`, `cmake` or `cl` were gone. The drop list is heuristic path fragment matching, so a runner image that moves any of those outside those fragments leaves the bundled install.ps1 reusing hosted developer tooling while the job still reports a clean machine. PATH written to $GITHUB_ENV only applies to later steps, so the check has to live in a step of its own; it carries the same event gate as the strip, exempts `py` (it lives in C:\Windows and stays, which is why the start probe is the real evidence), and resets $LASTEXITCODE before exiting 0 so an intentionally failing probe cannot fail a clean machine. Also correct the no-winget matrix note: that leg is not failing for an unfixed product reason. It stops at the unconditional git gate in setup.ps1 only on this ref, and with that gate relaxed it passes along with every other leg, so the row is a merge order dependency and stays required. * Resolve the desktop release including drafts, the convention this repo ships All three desktop legs died at the download step with an empty REL_TAG. The resolver passed --exclude-drafts while REL_REPO now defaults to github.repository, and every desktop-v* release in unslothai/unsloth is a draft: desktop-v0.1.50-beta and desktop-v0.1.471-beta are both drafts carrying the .dmg, .deb, .AppImage and setup.exe, while only the non-desktop tags like v0.1.501-beta are published. Excluding drafts therefore matched nothing and no leg could ever run against a production bundle. Drop --exclude-drafts so the newest desktop-v* release is found. A draft has no tag ref, so releases/tags/<tag> 404s for one, but gh resolves drafts over GraphQL and gh release download <tag> fetches their assets normally, so the download call is unchanged. Listing drafts requires push access, which for GITHUB_TOKEN means contents: write, so the workflow permission is raised from read and annotated. When nothing resolves the leg still fails hard rather than skipping: with no bundle to install there is nothing to prove, so a green run would be a lie. The error now names both causes, no release cut yet or a token that cannot see drafts. Also stop the restore step swallowing its own failure. `bash .clean-machine/restore.sh || true` printed "No such file or directory" whenever an earlier step failed before the toolchain was stripped, and hid a genuinely broken restore just the same. Skip explicitly when the file is absent and let a real restore failure surface. Same fix in clean-machine-install-ci.yml, which had the identical line. * Skip the desktop jobs on fork PRs instead of failing them Every desktop-v* release in this repo is a draft, and GitHub lists drafts only to a token with push access, which is why resolving one needs contents: write. A pull request from a fork receives a read-only token no matter what the workflow declares, so on those runs the resolver cannot see any release and the job died on "no desktop-v* release visible", accusing the repo of having no bundle when the real cause is the trigger. This workflow runs on pull_request for changes to itself and the stripping scripts, so an outside contributor editing either would have hit that. Guard the three jobs on the head repo not being a fork. A skipped job is honest here: it does not claim to have tested a bundle it was never able to download, and it is not reported as a pass. * Close the free headroom in the clean-machine simulation Assert arch and signature on every downloaded Mach-O. This is the one genuine gap the simulation had: Rosetta 2 is preinstalled on hosted runners and absent from a factory-fresh Mac, so an x86_64-only llama.cpp, whisper.cpp, Node or uv payload runs green here and dies with "bad CPU type in executable" for the user. llama-server launching under `assert-llama-loads.sh` does not rule that out, because Rosetta makes it launch. The new `macho` check reads `file -b` (`lipo` is an xcrun shim and is gone after masking, as the desktop lane already notes) and keys the expected arch off `uname -m`, so macos-15-intel expects x86_64. It also requires at least an ad-hoc signature on arm64, which closes the AMFI "Killed: 9" class that uv has already been bitten by; the check is skipped on x86_64, where unsigned code loads fine and so is not the same defect. It fails when the scan finds nothing, since an empty scan reads exactly like a clean one. Make absence real rather than PATH-hidden. uv probes well-known interpreter locations and the framework loader ignores PATH entirely, so hiding the toolcache only hid it from `command -v`. Empty /usr/local (it EXISTS on a factory-fresh Mac as a SIP-exempt firmlink, and is empty; it is /usr/local/bin that is absent, so the directory itself stays), move the hosted toolcache and /Library/Frameworks/Python.framework aside, and clear the developer dotdirs and caches. A populated uv or pip cache can also satisfy a resolution that would fail on a user's machine. Every removal goes through --remove and is recorded in the generated restore.sh, guarded so a path the install recreated is not buried inside its own restore. Unset CI, GITHUB_* and RUNNER_* for the installer process only. An installer branching on CI=true is a hidden dependency no consumer exercises. Scoped to the child so the step's own $GITHUB_OUTPUT still resolves. Record spctl --status and csrutil status. Neither is documented for these images and both change what a binary is allowed to do. * Pin the two failures no change here can fix, and add the virgin Windows container lane Three red checks, two of which test something this branch does not own. desktop linux deb / appimage run the SHIPPED bundle's own install.sh, and desktop-v0.1.50-beta was cut on 2026-07-21, before #7547 merged on 07-29. That bundle still carries the old optional-dependency gate, so on a stripped runner it exits 2 at [TAURI:NEED_SUDO] cmake git build-essential libcurl4-openssl-dev and never creates a venv. Current main's _check_linux_deps runs the same set through _SMART_APT_OPTIONAL, which suppresses every escalation path, so only a new release can change this. The step now pins that exact outcome: the exit code must be 2 and the log must carry exactly that package list, anything else still fails, and finding _SMART_APT_OPTIONAL in the extracted install.sh (the guard #7547 added) turns into a hard error saying to delete the pin. The venv and torch assertions stay and still run whenever the installer succeeds. win windows-11-arm gets a native ARM64 CPython, and torchaudio publishes no win_arm64 wheel at any version, so the PyTorch step cannot resolve. The fix is in install.ps1 on #7549, still open. Same treatment: the Install step is continue-on-error and a new step requires all three of the PyTorch step, the torchaudio resolution error and the missing win_arm64 platform tag, so any other failure is red. The row leaves experimental so the job is required, and the pin errors out as soon as the venv interpreter reports anything but win-arm64, which is what #7549 landing looks like. Adds the virgin Windows container lane as two jobs here rather than a sibling workflow: same premise as the win legs, same path filters, and masked-versus-real reads better side by side. The hosted Windows legs cannot test the VC++ 2015-2022 runtime (it ships in the runner image's System32) or a Windows with no Microsoft Store, and a servercore:ltsc2022 container on windows-2022 answers both. The probe asserts no python, py, git, cmake, cl, winget or uv on PATH, on disk or in the registry, and now also asserts vcruntime140.dll, vcruntime140_1.dll and msvcp140.dll are absent, which is the one thing the hosted runner cannot un-ship. Both container install rows stop at studio/setup.ps1's winget-only git gate on this branch, since #7549 is what relaxes it, so both are pinned the same way. The overlay row additionally requires the UNSLOTH_CI_SOURCE_OVERLAY hook to have fired, unconditionally: without that it would be indistinguishable from the released-wheel row, and the hook is this branch's own feature. Container notes carried over from the spike: never docker pull when the image is cached, since MCR has shipped an image ahead of the runner host before; wait for the Docker daemon, because one leg died in 21s on npipe:////./pipe/docker_engine and that flake misreads as "Windows containers unavailable"; drive docker from a run: step, because the job-level container: key is Linux-only. The root CA store is seeded after the virginity assertion, restoring what a real Windows already has, because studio/install_node_prebuilt.py downloads Node with bare urllib.request.urlopen and hits CERTIFICATE_VERIFY_FAILED against the empty container ROOT store. That product bug is left alone here. * Check signatures on Mach-O main executables only The macho check asserted a valid signature for every Mach-O under the studio home, and failed the macos-15 mask/pipe leg on 29 files: lxml, charset_normalizer, cygrpc, _upb, fontTools, caio, brotli and a bundled libportaudio.dylib. Those are MH_BUNDLE and MH_DYLIB images dlopen'd into a process without library validation, they ship unsigned in the wheels, and the same run had already installed and imported them with the installer exiting 0. Key the signature half off the Mach-O filetype and run it only on main executables. Report an absent seal separately from one that fails to verify, and capture codesign output instead of piping it into grep, which returned the unsigned exit status through pipefail and called every unsigned binary broken. The architecture half is unchanged and still a hard failure: it is what closes the Rosetta 2 gap. The zero-Mach-O guard is unchanged. The .venv_t5_* sidecars stay in scope; setup.sh creates them during a normal install and transformers_version.py puts them on sys.path, so they are payload. * Make the WSL job gate, assert Windows installed no toolchain, strip before the .deb * Assert the root Linux legs did not compile llama.cpp with the apt-installed toolchain * Pin the macOS desktop legs on the same pre-7547 release lag The Linux rows already pin the shipped bundle's own install.sh exiting 2 at the NEED_SUDO handshake. macos-15 and macos-26 fail the same way for the same reason: desktop-v0.1.50-beta predates #7547, so the bundled installer still hard-exits on the Xcode CLT gate that #7547 turned into a warning. Accept exit 1 plus that exact gate line, and nothing else. _check_macos_deps is the function #7547 added, so its presence in the bundle means the release caught up and the block errors out asking for the pin to be deleted. * Pin the WSL pipe truncation and the masked-winget git gate The WSL leg dies at install.sh:2082 with an unterminated quoted string. Nothing is wrong with that line: piping the script into sh is not atomic. dash reads it from the pipe in 8192-byte blocks and runs each command as it parses, and install.sh:2007 calls _maybe_reroute_strixhalo_to_2404, which on WSL alone shells out to Windows interop; interop relays the stdin it inherited and drains the pipe. dash has 11 blocks buffered at that point, ending at byte 90112, which falls inside "$STUDIO_LOCAL_INSTALL" on line 2082. Truncating install.sh at 90112 and parsing it reproduces the message verbatim, and running the whole file under a stdin-draining interop stub reproduces the exit code too. #7548 wraps the body in _unsloth_main so sh parses everything before running anything, and the same reproduction against its head is clean. The eight green staging runs cited when this job's continue-on-error came off were all on trees that already carried #7548, so that evidence never covered this branch. Pin the exact signature instead: exit 2 plus the shell's own unterminated-quoted-string error, with the _unsloth_main marker read back out of the distro as the flip condition. Pin winget=masked the same way. studio/setup.ps1:1655-1669 gates on git unconditionally and can only fetch it through winget, so masking winget leaves no way to satisfy it. #7549 relaxes the gate, and its wording appearing in the tree retires the pin. * Retire the WSL pipe pin now that #7548 is in main The pin flipped exactly as designed: it looks for _unsloth_main in the installer it actually ran, and #7548 put it there. Delete the pin and the CLI waiver, and assert the opposite instead. WSL is the only platform whose install shells out to Windows interop mid-script, and interop relays the stdin it inherited, so this job is the one that can catch the pipe being drained again. A truncation here is now a hard failure. * Gate the no-elevation Linux install and split off the no-transport case * Assert no source build on the hosted Windows legs and keep winget for the desktop lane * Retry the container root CA seeding instead of failing on one Windows Update timeout * Run the clean-machine workflow for the prebuilt installer helpers it overlays * Narrow the container pin to its own gates and scan uv and the venv interpreter for arch * Tighten the clean-machine comments * Re-assert toolchain absence after the desktop .deb pulls its dependencies * Retire the #7549 pins and add a wget-only Linux leg #7549 is in main, so the three known-outcome pins that were waiting on it are stale and would now hard-error by design. Each is replaced by the assertion it was standing in for rather than deleted: win windows-11-arm now gates. The x64-on-ARM64 resolver is asserted as an outcome: the venv interpreter reports win-amd64 from its own sysconfig, and torchaudio (no win_arm64 wheel at any version) is installed. Measured on the integration branch before #7549 merged: "only a native ARM64 Python 3.13 was found" -> "installing x64 Python" -> torchaudio 2.10.0+cpu, install green. win windows-latest / winget=masked now gates. The relaxed git gate is asserted from both sides: the old unconditional message must be absent, the no-git branch must have been reached (so the row cannot pass because git leaked back onto PATH), and setup.ps1 must report git as absent-but-not-required. Both Windows rows, and the visible one, gained the usability check the Linux legs have had and Windows never did: a managed interpreter, an unsloth CLI on disk, and that CLI actually running. nobuild and the toolchain check only read the log, so an installer that exited 0 having produced nothing satisfied them. The torch assert also loses its fallback to whatever `python` resolves to. The virgin container overlay row gates, and asserts what only that lane can: it is the one environment whose System32 does not already ship the VC++ 2015-2022 runtime, so it is the only place Ensure-VCRedist's direct aka.ms download can be proved to run rather than be short-circuited. The overlay=false row keeps a pin, with a new reason: it installs unsloth from PyPI on purpose, and setup.ps1 inside 2026.7.5 (uploaded the 23rd) predates #7549, so it still stops at the old gate. That is release lag, it flips on the next release, and the pinned signature is now the old wording rather than "#7549 has not landed". Also adds linux ubuntu2404-nonroot-wget. install.sh's download() takes curl or wget and _transport_missing is true only when both are gone, so a wget-only box is supported on paper, but the gating nonroot leg provisions ca-certificates AND curl, so curl won every probe and the wget branch had never run. Same image, same no-sudo user, same asserts, wget instead of curl, and curl proved absent on disk for root and for tester before AND after the install, so the claim is that every download went through wget rather than that curl happened to be unused. * Tighten the clean-machine CI comments Comments only, no assertion logic, pins or leg definitions touched. Reflowed every rationale block to denser wording and removed the duplication that had built up across repeated steps: the desktop workflow repeated the fork-PR skip, the desktop-v* tag resolution and the restore-runner note once per platform, and the installer workflow repeated its path-filter rationale in both the pull_request and push blocks. Those now point at the first copy. Every WHY is kept: why the masked legs avoid install.sh --local, what UNSLOTH_CI_SOURCE_OVERLAY is for, why `absent` tests "must not work" rather than command -v, why the .venv_t5_* sidecars are in the macho scan scope, why the signature check is main-executables-only, why each nobuild allowlist entry is a pure-Python sdist, why the WSL job gates and what the pipe truncation was, and why the virgin container's overlay=false row is still pinned. Proved comments-only three ways: both workflow revisions parsed with yaml.safe_load_all and every leaf walked (only `run:` scalars differ); every changed bash body and .sh compared byte-for-byte after `bash --pretty-print -n`; every changed pwsh body and .ps1 compared as a token stream with Comment and NewLine tokens dropped. A negative control injecting one non-comment line into each layer makes all of them fail. * Clean machine CI: strip Strawberry, make the Fedora pin gating, run the Linux CLI desktop windows failed the strip verification because windows-latest ships a MinGW toolchain under C:\Strawberry\c\bin, which matches none of the drop fragments; the installer workflow already scrubs it. Fedora sat behind job-level continue-on-error, so its outcome pin could not fail the run. Tolerate the install step instead, as the no-transport row does. The Linux usable-install check only tested the executable bit; Windows and WSL already execute the CLI. The macho scan now fails when no venv interpreter was scanned, rather than letting uv alone satisfy the outside-root guard. * Clean machine CI: tighten the comments Round 12 comment reduction: compress wording, keep every reason. Comments only, verified with a YAML leaf walk (differences only inside run: scalars, only on # lines), bash --pretty-print -n byte comparison, a PowerShell token-stream diff and a Python AST comparison. * Clean machine CI: dereference the venv interpreter, pin the deb deps and the Windows disposition file did not follow the <venv>/bin/python symlink find -L printed, so it answered 'symbolic link to ...' and the Mach-O test dropped the one interpreter the Rosetta scan exists to check. Read with file -Lb and count what was classified, not what was found. apt treats a toolchain package the strip only renamed as already installed, so a .deb that started declaring git or cmake would never restore it and the absent re-check would still pass. Assert the declared Depends instead. The Windows lane accepted any preflight disposition although the bundled installer was already required to build a working venv; NotInstalled or ManagedStale there means the app cannot boot what it just installed. * Clean machine CI: assert every masked tool, and re-select the developer dir last clean-machine-env.sh moves ten tools aside and only warns when a move fails, but absent checked four of them, so a surviving gcc -- which install.sh probes for build-essential -- went unnoticed. restore.sh ran xcode-select --switch before the line that moved CommandLineTools back, so it named a still-masked directory, failed into || true and left the selection link unrestored. Capture the original selection and re-apply it after both directory restores. --------- Co-authored-by: danielhanchen <unslothai@gmail.com>	2026-07-29 06:21:40 -07:00