3 commits

Author	SHA1	Message	Date
oobabooga	495ba21ebd	Studio: add MiniMax H3 video generation (#7989) ... * Add MiniMax H3 video generation * Improve MiniMax H3 memory routing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address MiniMax H3 review feedback * Keep H3's VAE off the CPU path, so low_vram stops aborting low_vram maps to the `model` policy, and offload_flags emits --vae-on-cpu for it unconditionally. On H3 that kills the process: ggml/src/ggml-cpu/ops.cpp:6321: GGML_ASSERT(src0->type == GGML_TYPE_F16) failed deterministically, SIGABRT, exit 134. Bisected on the flags: --vae-on-cpu with --audio-vae aborts, the same command without --audio-vae renders in 95.87s, and an fp16-converted audio VAE aborts too. So the trigger is the audio VAE, not the video one, and the F32 type is imposed inside stable-diffusion.cpp rather than by the file: ggml_conv_1d hardcodes an F16 im2col destination (ggml/src/ggml.c), ggml_compute_forward_im2col_f16 then asserts the KERNEL is F16, and audio_conv_weight_type (src/model/vae/ltx_audio_vae.hpp) maps only BF16 to F16 and lets F32 through. It cannot be fixed by shipping a different checkpoint. low_vram is the one mode a small-card user reaches for, so this drops the flag rather than the mode. offload_flags takes vae_on_cpu, defaulting True so no other family changes, and the H3 native path passes False. --offload-to-cpu and --clip-on-cpu still apply, which is where the saving actually is: the denoiser dominates, and with --offload-to-cpu the whole model peaks at 13.14 GiB. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Point the H3 GGUF pick at the unsloth mirror Main added test_curated_gguf_repos_are_unsloth_mirrors, which requires every curated video gguf_repo to live under unsloth/ so a one-click pick cannot 404 when a community repack is renamed or taken down. H3 was the one family still on a community repo. This was meant to be part of the merge commit but was left in the worktree, so CI on that commit still saw the old value. unsloth/MiniMax-H3-GGUF is still private and has to be made public before this merges, or the pick will 401. No CI check reads it. * Pin H3's native cfg-scale under test H3 is distilled and CFG-free: its empty unconditional prompt encodes to zero tokens, and the transposed tensor that produces trips GGML_ASSERT(!ggml_is_transposed(a)) in ggml.c. SIGABRT, exit 134. Measured: cfg 1.0 renders, cfg 1.5 and cfg 4.0 both abort. sd.cpp defaults cfg-scale to 7.0, so this is a crash a plausible refactor reintroduces by forwarding guidance the way every other family does. The native path already hardcodes 1.0 and the family sets supports_cfg = False, but nothing held either in place. supports_cfg only gates the diffusers path; the native path builds its own params. Checked the test fails when cfg_scale is changed to forward guidance, so it is not passing vacuously. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin diffusers by source archive instead of git, so macOS installs work The macos-15-intel leg fails deterministically, not flakily: process didn't exit successfully: `/usr/bin/git init` (exit status: 1) --- stderr xcode-select: note: No developer tools were found, requesting install. That runner has no Xcode, so /usr/bin/git is the developer-tools shim and exits 1 for everything. uv needs a working git to resolve a git+https dependency, so the diffusers pin this branch added cannot install there at all. Main is unaffected because it depends on plain diffusers. GitHub serves the same commit as a source archive, which uv installs with no git involved. Verified by stubbing git to fail exactly the way the macOS shim does: the git+ form reproduces the CI error, the archive form installs diffusers 0.40.0.dev0 from the same SHA with MiniMaxH3Transformer3DModel present and exported. Also drops a full clone of diffusers from every install. * Allow the diffusers source build on the clean-machine legs With the archive pin the macOS leg gets past `git init` and installs, but then trips the nobuild guard: built from source: diffusers -- these must resolve to wheels on a clean machine There is no wheel to resolve to. MiniMax-H3 support is not in any diffusers release, so this branch has to pin a commit, and neither a git URL nor a source archive can produce a wheel from an index. Added to the same allowlist that already carries triton-kernels for the same reason. Checked against the bar the comment there sets: diffusers builds with plain setuptools, declares no ext_modules, and its tree has zero .c/.cpp/.pyx/.rs/.cu files and no shipped binaries, so the PEP 517 build is a pure-Python copy step and needs no toolchain. The separate compiler-invocation check in both scripts is untouched and still fires if one is ever needed. Verified the allowlist logic still rejects a non-allowlisted source build (a log building both diffusers and numpy reports only numpy). Remove this entry once a diffusers release carries H3 and the requirement goes back to a version specifier. Noted in both scripts. * Baseline the hf-hub retry loop reopened by the 1.x upgrade `pip scan-packages :: hf-stack` fails on this branch with 1 unbaselined CRITICAL: C2 polling/beaconing loop detected Package: huggingface-hub File: huggingface_hub/utils/_http.py L461: while True: This is on us, not upstream drift. Main pins huggingface-hub==0.36.2; this branch needs >=1.23.0,<2.0 because diffusers at the pinned commit requires it, so the resolved version moves 0.36.2 -> 1.27.0. The baseline already carries this exact file and check at L462 and L298, from earlier versions. It did not carry over because the key hashes the matched code, not the line number, and the surrounding code changed across the major version. That is the guard behaving correctly: changed code in a baselined file reopens for review rather than staying suppressed. Reviewed it rather than just re-suppressing. L461 is the retry loop in `http_backoff`: bounded by `nb_tries > max_retries`, the URL comes from the caller, there is no hardcoded endpoint and nothing is exfiltrated. Same benign construct as the entries it replaces. Entry generated with the scanner's own _evidence_hash rather than hand-written, and inserted beside its siblings so the diff stays 8 lines. Verified: with the baseline the scan is exit 0 with 4 suppressed, without it exit 1, so the guard still bites. * Lift the macOS-arm huggingface-hub cap that this branch made unsatisfiable `mac macos-15 / trace / file` fails on this branch and passes on main. uv reports: No solution found when resolving dependencies: Because you require huggingface-hub>=0.34.0,<1.0 and huggingface-hub>=1.23.0,<2.0, your requirements are unsatisfiable. This branch moved base.txt, no-torch-runtime.txt, studio.txt and constraints.txt to hub >=1.23.0,<2.0, because diffusers at the pinned commit requires it, but left the flat <1.0 cap in overrides-darwin-arm64.txt. That file is macOS-arm only, which is why only the mac legs see it and Linux and Windows stayed green. The failure then presents as something else entirely: uv gives up, the installer falls back to pip, and the clean-machine trace fails on "installer invoked toolchain: rustc" rather than on the resolution. The cap's own comment explains it exists so the resolver can never pair hub 1.x with a pinned transformers 4.57.6 / hub 0.36.2 stack. That is still true below python 3.10 and still capped there. At 3.10 and above this branch is on transformers 5.5.0 and hub 1.x, so the premise is gone, and mlx-audio's own >=1.0 floor is satisfied by the 1.x window anyway. Checked by collecting every hub specifier that applies per Python version across all five files: py3.9 resolves to 0.36.2 as intended, py3.13 to 1.23.0/1.27.0. Before this change py3.13 resolved to nothing. * Make H3's native download use the repo the family advertises The curated-mirror test main added only inspects VideoFamily.gguf_repo. H3's native path does not read that field: video_minimax_h3.py has its own H3_GGUF_REPO constant, used for both the transformer and the Qwen3-VL encoder. So pointing the family at the unsloth mirror in the previous commit satisfied that test while the actual one-click download still came from a community repack, which is the exact failure the test exists to prevent. Pointed the constant at the same mirror and added test_the_h3_native_repo_matches_the_family_gguf_repo to pin the pair, so the two cannot drift apart again. Verified it fails when the constant is put back to leejet, so it is not passing vacuously. The mirror now carries the Qwen3-VL encoder quants alongside the denoisers, byte-identical in size to the community ones, so this repo alone satisfies both of h3_native_hub_files' hub entries. The encoder is part of MiniMaxAI/MiniMax-H3 itself (FL2VA/text_encoder) which we already mirror publicly at unsloth/MiniMax-H3 under the same licence, so shipping a quantization of it beside the denoisers is the same act. Also checked the encoder-tier routing survives the dynamic rung names: -UD-Q2_K_XL selects the Q2_K_M encoder and -UD-Q3_K_XL the Q4_K_M one, asserted in the new test. Updated the download-plan test, which hardcoded the old repo id. NOTE: unsloth/MiniMax-H3-GGUF is private. Unlike before, that now really does gate this: the native path downloads from it. It has to be public before this merges. * Pin H3's companion-checkpoint guard under test validate_h3_transformer_filename had no test. That mattered less when the denoisers lived alone; the mirror now ships the Qwen3-VL encoder quants in the same repo, so the picker lists both and a user can name either. Loading a 12-17 GB encoder as the transformer would fail deep inside sd-cli instead of at the boundary. The accept cases include the dynamic rung names on purpose. The guard is a prefix/suffix check and `-UD-Q2_K_XL` is a shape it had never seen when it was written; it happens to pass, and now that is asserted rather than assumed. Checked the test fails when the prefix check is dropped, so it is not vacuous. * Record why H3 drops --vae-on-cpu, now that the abort is fixed The comment justified the drop entirely by an sd.cpp abort. That abort is fixed in the Unsloth fork, which would have made this look like a stale workaround to revert once the fix reaches the pinned prebuilt. Measured on a build carrying the fix, 640x384, 25 frames, 4 steps, q4_K, with --offload-to-cpu --clip-on-cpu already applied: adding --vae-on-cpu moved peak VRAM 12.42 -> 12.42 GiB and wall time 20.9s -> 100.4s. Under --offload-to-cpu the peak is set by the streamed denoiser, so the flag saves nothing and costs 4.8x. It stays off on its own merits. * Pin the sd.cpp prebuilt that actually renders MiniMax-H3 The pin was master-812-ea7f0c8, a stock upstream build, and on a stock build H3 does not work: it aborts on the default --cfg-scale, aborts again on --vae-on-cpu, and its 1-D norms are quantized into an output uncorrelated with its own bf16 reference (LPIPS 0.981). The Studio side worked around the first by pinning cfg to 1.0 and the second by dropping the flag; the third had no workaround on the consumer side at all. All three are fixed in unslothai/stable-diffusion.cpp and open upstream as leejet/stable-diffusion.cpp#1861, #1862 and #1863. The mirror's prebuilt pipeline now applies them on top of the aged upstream tag it already builds, and marks such a build with a -u<id> suffix naming the patch set, so master-813-bfbef5b-u0665242 is upstream master-813-bfbef5b plus those three patches and nothing else. The patches are deleted once upstream releases them, at which point this pin goes back to a plain tag. Verified on the published Linux x86_64 asset, not on a local build: both new error strings are in the shipped sd-cli, and running it on a q4_K H3 denoiser without --mode vid_gen now exits 1 with the instruction instead of core dumping on a ggml assert. test_video_backend's fake engine returned the old tag as its version string, which read like a second pin; it only needs a non-None value, so it now says so. * Close two gaps the H3 mirror switch opened Both are consequences of the two preceding commits, found in review. The prebuilt pin is now mirror-only (master-813-bfbef5b-u0665242), and _resolve_with_fallback still asked leejet for it. That request is a guaranteed 404 by construction, since the -u<id> suffix marks a build only the mirror makes, so it was a wasted round trip on every install. Worse, when the mirror genuinely cannot serve a host the fallback lands on leejet's latest, which has none of the H3 fixes. For every other model that is the right trade, better a stock native engine than none. For H3 it is not visible: it aborts on the default cfg-scale, aborts on --vae-on-cpu, and a blanket --type renders a broken video rather than failing. A user who saw only the generic 'falling back to leejet' line had nothing connecting that to the output. It now says so. Second, hub/utils/gguf.py filtered H3 companion GGUFs by the old community repo id only. The mirror the family and catalog now advertise carries the Qwen3-VL encoder quants beside the denoisers, so a 12 GB text encoder was being aggregated as if it were a selectable transformer quant. Both bundle repos are now recognised, case-insensitively, and the cache-dir match follows. Tests are mutation-verified rather than assumed: restoring the upstream 404 attempt fails the ordering test, removing the H3 warning fails the fallback test, and dropping the mirror from the bundle set fails three. The ordering test deliberately makes the mirror serve nothing, because with the mirror serving the first attempt succeeds and the upstream attempts are never reached, which made an earlier version of it pass under its own mutation. A third test pins the native loader's H3_GGUF_REPO to the bundle set, since those live in different files and a future repo move that updated only the loader would silently reintroduce the same aggregation bug. * Exclude MiniMax-H3's small-M projections from int8 _INT8_FAMILY_EXCLUDE_NAME_TOKENS has entries for qwen-image and hunyuanvideo-1.5 but none for minimax-h3, and H3 needs one for the same reason they do. H3's adaLN projection is named adaln_proj, which no token in the generic list matches: 'norm' is the closest and does not appear in the name. On the dense checkpoint that projection is Linear(2688 -> 96768), so it clears min_features = 512, gets quantized, and then runs at M = 1. Inductor lowers int8 matmul to _int_mm, which requires M > 16, so it raises 'self.size(0) needs to be greater than 16, but got 1' at the first denoise. The offline prequant builder bakes it in happily, which is exactly the drift the exclude list exists to prevent for Flux and Qwen. The pruned-modulation form hides this rather than fixing it: there adaln_proj is Linear(8 -> 96768) and falls under min_features anyway. So this exclusion is what makes the DENSE path correct and is a no-op on the pruned one. context_embedder and token_refiner are added for the same reason hunyuanvideo-1.5 excludes its text stream. Measured at M = 10 text tokens against the video stream's thousands, they are 3.47% of GEMM time even in the slow eager int8 path, so leaving them bf16 costs nothing measurable. This is what made an earlier measurement conclude int8 does not work on H3. It does: on the pruned form int8 compiles and is 4.24% +-0.54% faster than fp8 at identical memory, paired over 12 renders. * Say which H3 component could not be downloaded, and why H3 pulls four files from two repos, and the Hub returns the same 'Repository Not Found ... make sure you are authenticated' for a repo that does not exist, one that is private, and one your token does not cover. A user reading that has no way to tell which of the four failed, and the wording points away from the real cause whenever the repo exists but is not public. That is the state the GGUF mirror is in today: it is unpublished, so picking H3 fails with a message suggesting the user fix their token, which will not help. This replaces it with the repo, the component, and the actual remedy, and says the other components are unaffected so the failure is not read as total. Gated repos get different wording, since accepting a licence is a different action from waiting for a repo to be published. Anything that is not a recognised access error is passed back unchanged rather than reworded, so a timeout or a full disk still reads as itself. The helper returns the exception instead of raising, so the caller keeps raise-from and the original traceback survives. Mutation-verified three ways: rewording every error (timeouts included), fixing the component name to 'denoiser', and giving gated repos the private wording each fail the test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop MiniMax-H3 holding two copies of its video VAE Two thirds of an H3 render's peak is not activations. Measured at 640x384 across 124 frames, a 20.25 GB int8 denoiser peaks at 36.96 GB, and the gap is almost all weights: the video VAE alone is 10.42 GB because diffusers pins it to float32, and a further 4.91 GB is autocast's own float16 copy of those same weights. A memory snapshot puts 92.9% of the transient in 437 blocks allocated from nn.Linear, largest 67.1 MB, which is the decoder's [2048, 16384] SwiGLU projection in float16. MiniMaxH3VideoDecodeStep wraps vae.decode in torch.autocast(float16), and autocast caches every weight it casts for the lifetime of the region, so the float32 original and its float16 twin are both resident through the whole decode. Storing those weights as float16 up front makes the cast a no-op and removes both. This is not an approximation: x.to(float16).to(float16) is x.to(float16), and the four regression tests check that on a real matmul rather than on the reasoning. The audio VAE decode is not under autocast, so it keeps float32. t2va starts from noise and never encodes, so vae.encoder and vae.quant_conv go too. That part is gated on the workflow name rather than dropped unconditionally, because an image-conditioned workflow needs them. Measured 36.96 -> 28.37 GB peak with the encoder drop and the pre-cast, 28.27 with expandable_segments as well, over 5 prompts x 2 seeds. Speed is unchanged (-0.05% +-0.86% eager, -0.34% +-2.23% compiled), and every arm hashes identically to its control on latents, audio_latents, frames and audio. The estimator's base still reads 68.5 GB. That figure was measured on the bfloat16 modular components, not the int8 arm above, so it stays put until it is re-measured in the same configuration rather than adjusted by arithmetic. * Pad MiniMax-H3's small-M int8 linears instead of leaving them dense torch._int_mm asserts self.size(0) > 16. torchao's eager path never trips it (safe_int_mm falls back to a widened matmul), but inductor lowers the same quantized linear straight to _int_mm, so any quantized Linear invoked at a small activation row count crashes under torch.compile. Until now the fix was to leave those linears dense bf16, which on H3 meant excluding context_embedder and both token_refiner blocks: 13 linears, 798M parameters, 0.80 GB of weights the int8 checkpoint was not allowed to touch. Pad instead. diffusion_quant_pad.PadToMinM pads the flattened row count up to 32, runs the GEMM and slices the result back, so the module compiles with no change to the quantization config and the caller's rows come back bitwise unchanged. Verified bitwise on all 65 (module, M) cases across H3's 13 linears at M = 10, 13, 14, 17, 19, on real torchao-quantized weights; compiling those same modules unpadded raises the _int_mm assert at M = 10, 13 and 14. Two properties carry that exactness and both are asserted rather than assumed. The pad rows replicate row 0, not zeros: an all-zero row has amax 0, so the activation quantizer divides by zero. And the activation scale must be per row, so each kept row's scale comes from that row alone; a quantized Linear whose granularity cannot be proven per row raises instead of being quietly skipped, because a half-padded transformer compiles on the wrapped modules and crashes on the rest. Everything below pad_to normalises to pad_to rather than only what is below the floor, so one inductor graph covers every prompt length in the range. H3's seven eval prompts run at M = 10..19, which straddles the floor, and padding only to 17 would leave three shapes behind. The wrapper reparents the Linear, so it runs after quantize_ on the runtime path and after load_state_dict on the prequant one. The offline builder drives quantize_ directly and saves the state dict, so it never sees a wrapper; PadToMinM is also state-dict transparent as a second line of defence, saving and loading under its own prefix so a wrapped transformer still writes context_embedder.weight. Scoped to minimax-h3. qwen-image, qwen-image-edit and hunyuanvideo-1.5 have the same small-M shape but published int8 prequant checkpoints whose metadata bakes the current exclusion set, and _validate_checkpoint compares that set against exclude_tokens_for_scheme, so they move only together with a rebuild. adaln_proj stays excluded for a different reason: on the dense checkpoint it is Linear(2688 -> 96768) and runs at M = 1, while on the pruned form it is Linear(8 -> 96768) and falls under min_features anyway (verified: the filter rejects all 51 for min_features). Measured on B200, 640x384 x 124 frames, 4 steps, 7 prompts x 2 seeds, the two arms alternated within each cell so drift on a shared box cancels: checkpoint 21.052 -> 20.254 GB (-0.798, -3.8%) transformer 21.051 -> 20.253 GB (-0.798) load peak 21.137 -> 20.336 GB (-0.801) render peak 37.766 -> 36.966 GB (-0.800, -2.1%) step time +0.0597 s +-0.0057 eager, +0.0533 s +-0.0086 compiled The time is torchao's un-fused eager activation quantization on 13 modules that run ONCE per render, so it is a fixed cost rather than a per-step one, and it does not scale with steps or resolution. Compiling those modules alongside the blocks removes it: +0.0034 s +-0.0040, no detectable difference. Quality is unchanged as far as n = 14 can resolve: against the same bf16 twin the padded arm sits +0.0070 +-0.0151 LPIPS from the excluded one, which rules out a degradation larger than 0.022. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Classify the MiniMax-H3 GGUF bundle as video in the cached inventory The H3 GGUFs are stable-diffusion.cpp conversions and carry no metadata keys at all (kv_count 0), so general.architecture is absent where the LTX-2 and Wan video GGUFs declare ltxv/wan. _arch_to_task therefore left the downloaded repo with no task, which drops it from the Video picker's On Device list and hands it to chat as a plain GGUF. Key the two bundle repo ids before the arch is consulted. * Release the VIDEO GPU claim when H3 native falls back to the CPU build On a CUDA/ROCm host /video/load acquires the VIDEO arbiter owner because the resolved device target is not CPU. _run_load_h3_native then asks for an accelerator-matched sd-cli, and the pinned prebuilt release publishes no Linux CUDA/ROCm archive, so ensure_sd_cpp_binary returns None and the load commits the CPU build with native_device = cpu. Nothing dropped the VIDEO claim, so the next chat or image acquire evicted and unloaded an H3 runtime that holds no VRAM. Release the claim once the CPU fallback is committed, through release_if so the token check is atomic against a newer load that already took ownership. Mirrors the CPU-only native release /images/load already does. * Video: protect the native H3 companion repos, cancel the modular denoise, forward the hub token - VideoBackend.loaded_repo_ids() publishes the repos the committed native H3 runtime re-reads every generation (Qwen encoder + both VAEs), and the delete-cached guard consults it, so deleting a companion under a loaded model is refused. - The H3 modular workflow no longer falls back to a null progress context: the denoise loop drives pipe.scheduler.step once per step, so the existing wrapper gives it the same per-step progress and cancellation the other callback-less pipeline gets. - load_components() gets the Settings token, so gated/private component loads are not issued anonymously. * Reject stable-diffusion.cpp builds that predate MiniMax-H3 support ensure_sd_cpp_binary hands back whatever find_sd_cpp_binary locates and only probes that it runs, so an install upgraded from an older Studio kept serving its pre-H3 managed sd-cli. The H3 load's only gate is SdCppEngine.version(), which that binary passes, so the load reported ready and the failure surfaced on the first generation, after the whole bundle had downloaded. Gate the H3 path alone on the capability instead of a version string: upstream added --ref-video and the other H3-only options in the same commit that added MiniMax-H3 (leejet/stable-diffusion.cpp#1854, master-812-ea7f0c8), and the release prebuilts report 'version unknown, commit unknown' because they are built without a .git directory, so --help is the only usable signal. Image generation keeps accepting any user-supplied build. A stale copy under the installer-owned root is removed so the pinned prebuilt reinstalls; a build the user supplied is left in place and the load fails naming it, the same ownership split _usable_or_discard_managed makes. A --help that cannot be read means 'cannot tell', never 'no H3'. * Distinguish a reused CPU sd-cli from an accelerator build on an H3 load On a Linux CUDA host the first H3 load installs the CPU prebuilt through the fallback and correctly commits native_device = cpu, because the pinned tag publishes no Linux CUDA, ROCm or Vulkan asset. Every later load then calls ensure_sd_cpp_binary(accelerator = cuda), which finds that same CPU binary and returns it without looking at what it was built for, so the fallback was skipped, native_device stayed cuda, and Studio applied GPU offload policy and retained the VIDEO gpu_arbiter claim while sd-cli ran wholly on the CPU. A later chat or image acquire then evicted an unrelated GPU model. Fall back on what the binary offers rather than on whether one was returned: sd-cli --list-devices prints one name/description line per available ggml backend device, so a CPU-only build answers with CPU alone. The second load now reaches the same cpu conclusion as the first, which is what lets the existing release_if drop the stale claim. Unreadable output, or an older build that rejects the flag, keeps the GPU: neither says the accelerator is missing. * Guard the H3 companion repos while a native video load is downloading _run_load_h3_native downloads from repo_id, the H3 GGUF companion and the H3 component repo, but the in-flight state only carried repo_id and base_repo. The cached-model delete guard reads loading_repo_ids(), so it allowed deleting Comfy-Org/MiniMax-H3, and the GGUF companion when the load comes from another mirror or a local file, while those files were still downloading, which fails the load. Carry the companions on the loading state, the way the image backend's _SdLoading already does, and publish them from loading_repo_ids(). This is the in-flight twin of loaded_repo_ids() and covers the same repos. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep curated Recommended rows searchable and give them their metadata Two things went wrong with the curated video models the Recommended list paints from the catalog rather than from a live Hub listing. Search dropped them. The Recommended search matched the query against `recommendedIds`, which filters out every id already on disk because a downloaded model gets its own On Device row. The unfiltered Recommended list does not filter that way: it renders the curated seeds and badges the downloaded ones. So a curated model was visible in the list and unfindable by typing its name the moment it was downloaded, and only a live listing row could bring it back, which a repo the listing does not return never gets. `searchableRecommendedIds` unions the seed ids with the listing ids, seeds first, deduped case insensitively (the HF cache lowercases repo ids), so both lists agree on what exists. Rows rendered bare. Everything past the id came from the listing alone, so a curated row the listing never returns showed no parameter chip and no capability glyph while its neighbours showed both. The catalog now carries the two facts nothing else can supply: `totalParams` per artifact and `capabilities` per group, read through `curatedTotalParamsFor` and `curatedCapabilitiesFor`. Both are fallbacks only. A listing row wins wherever there is one, because real tags and a Hub-reported total outrank anything hand written here. The parameter counts are measured, not guessed. The MiniMax-H3 figure is the sum of the tensor shapes in the BF16 GGUF of the FL2VA denoiser that repo publishes; the LTX-2.3 figure is what the Hub reports for its repo, carried so the row looks the same offline or rate limited. One more inconsistency fell out of the same place: `searchRowFits` hides anything it cannot size (`requireKnown`), and it could not size a curated repo with no listing row and no "<n>B" token in its id, so turning on "Fits on device" hid from search a row the unfiltered list still showed. It now falls back to the curated total the same way it already fell back to the curated size. Covered by studio/frontend/tests/recommended-curated-row-metadata.test.ts: 15 assertions over the search pool, the two catalog lookups, the fit check, and the four picker call sites that read them. * Load MiniMax-H3 from a hosted pre-quantized denoiser Video families had no way to reach a hosted pre-quantized DENOISER. VideoFamily carried gguf_repo and te_prequant_repos (the text encoder) only, and nothing in video.py consulted a denoiser table, so a MiniMax-H3 load either pulled the full 66.3 GB bfloat16 DiT or nothing at all. The image side already solves this with DiffusionFamily.prequant_repos plus family_prequant_repo(), so this follows that shape rather than inventing a second one. Four parts. VideoFamily gains prequant_repos, prequant_variant_repos and prequant_subfolder, resolved by video_family_prequant_repo() / video_family_prequant_schemes(), mirroring the image resolver. The registries stay separate, as the module header requires, so the base-id normaliser is local rather than imported. The shared resolver learns an optional subfolder. The hosted video checkpoints nest theirs one level down instead of keeping it at the repo root, and the prefix has to reach BOTH candidate names or the primary 404 is followed by a second one and the load silently falls back to the dense download. Always a literal forward slash: these are Hub repo paths, and a Windows join would miss the cache. The cache and download plumbing already handled a nested name, so only the filename builder changed, and every existing call site is byte-identical. The modular workflow builds its denoiser through its own component loader, so there is no dense module to quantise in place. A hosted checkpoint is therefore the only way to run that transformer quantized, and pre-seeding it with update_components() before load_components() is also what stops the dense download: load_components(names=None) skips a component whose attribute is already set. Passing names= instead would have forfeited the workflow's own block pruning, which is what already avoids the 61.7 GB Ref2VA transformer. Those checkpoints carry the pruned adaLN, where the modulation is a rank-8 affine factorization of the time-embedding curve plus a shared table, and roughly 40% of the released model's parameters go. Against the base repo's dense config the model is four keys short, one over and fifty-one shapes wrong, so the strict load fails and the checkpoint is unloadable by every route. video_minimax_h3_adaln.py reshapes the model between from_config and load_state_dict: table lookup with interpolation instead of the timestep MLP, no SiLU (the table already holds the activation's own output projected onto the basis), and the modulation cast to the block stack's dtype, without which the first quantized matmul dies on mismatched dtypes. Bound per instance, so a dense load in the same process is untouched. Finally the refusal becomes honest. A single-file pick on a modular family used to reach the loader only after ~98.7 GB had downloaded AND after the resident pipeline had been evicted to make room for it, because download-plan returned 200 and validation passed. Both refusals now run in validate_load_request, ahead of the diffusers availability probe so they still fire where diffusers cannot be imported, and each names what to pick instead. download-plan forwards transformer_quant to validation and to the plan, without which the quant-keyed refusal never fires on the route that stages the download and the plan stages shards the load never opens. * Point both MiniMax-H3 schemes at one hosted pre-quantized repo The two hosted pre-quantized denoisers were split across two repos with the checkpoint nested one level down, so reaching them needed a mechanism the image side has never had: a VideoFamily.prequant_subfolder field, a prequant_subfolder_prefix() normaliser, and a subfolder keyword threaded through both resolve_prequant_source() and usable_prequant_source(). Both schemes now live in ONE hosted repo, at the root, named <Model>-<SCHEME>.pt. That is the layout every image-side prequant repo already uses, and it is exactly what prequant_repo_filename() builds unaided, so the whole mechanism goes. Match an existing convention and the code should shrink, not grow: -22 lines in diffusion_prequant.py, -6 in video_families.py, -6 in video.py, with no new concept to carry. Landing on the primary name also fixes a memory-planning under-credit. cached_checkpoint_path() deliberately credits only the PRIMARY filename, so that a cached legacy artifact cannot pin a stale name once a repo ships the real one. While these checkpoints were published as transformer_<scheme>.pt the primary never existed: every hit came through fallback_filename, and planning therefore read an already-cached checkpoint as "this would have to download" and handed the pick to GGUF. The primary is now the published name, so the probe hits it. fallback_filename stays. It still covers repos that have not been renamed, and dropping it is a separate decision from this one. Tests: the subfolder-prefix cases are replaced by the naming they now guard -- both schemes resolving to one repo, the primary resolving to a root-level <Model>-<SCHEME>.pt with no directory component on any platform, the cache probe being asked for that primary name, and the repo's own scheme suffix being stripped and replaced rather than carried through. Five mutations run, each caught by the named test and reverted: two repos again (M1), the primary nested under prequant/ again (M2), the suffix strip removed (M3), the cache probe keyed on the fallback (M4), the fallback name dropped (M5). * Add MiniMax-H3 image and reference video conditioning * Improve H3 finalization progress * Report real sd.cpp progress on the Video page instead of a frozen 0 of 30 A native (GGUF) video generation reported phase "denoise", step 0/30 for its entire run and then flipped straight to "completed". Two separate things were wrong, and the progress endpoint could not move until both were fixed. The parser looked for r"(?:step|sampling)\D+(\d+)/(\d+)". sd-cli's sampling bar contains neither word. It prints |=========> | 7/30 - 21.50s/it so nothing ever matched. Anchor the pattern on the bar and on the trailing speed unit instead. A bare "n/m" is deliberately not enough: an unrelated ratio in some other log line must not drive the progress bar. The reader also delivered every redraw one step late. sd-cli redraws in place, and its carriage return LEADS the record while the newline only arrives on the final step, so a reader keyed on CR/LF cannot produce step 1 until step 2 has been flushed. Treat the erase-to-end-of-line that closes each redraw as a terminator too, and read the pipe with buffer.read1 so a record that carries no newline is not stuck behind a blocking readline. Escapes are stripped before a record reaches on_log or the error tail. Streams without a raw .buffer keep the old line iteration. The same bar shape is printed by three different things, so the parser now tells them apart rather than reporting whichever came last. Weight load prints it with a byte rate, and tiled VAE decode prints an identical s/it bar counting TILES: without a guard a run finished sampling at 30/30 and then jumped backwards to "step 1/16". Load and decode are real work with no sampling step, so they report their own phase and a null step rather than a fake 0 of 30. ETA is measured from the first step, not from job start, so the one-off weight load is not charged to every remaining step. Verified end to end against a real CUDA sd-cli MiniMax-H3 generation: the step advances 1..6 over 6.2s..9.7s of wall clock, load and decode are reported as themselves, and the ETA tracks. * Read sd-cli's in-place progress redraws so the Video bar moves during sampling The native H3 progress bar had two independent causes and fixing either alone changed nothing observable. The bar pattern is now correct, but the reader still was not. sd-cli redraws its sampling bar in place: one printf per step shaped "\r<bar> <n>/<total> - <speed>\033[K", with a newline only on the final step of a phase. The drain loop did `for raw in proc.stdout`, which terminates on LF only, so every redraw sat in the buffer until the next one's carriage return arrived and the last one until sampling was already over. The Video page saw nothing. Split the raw pipe into records on CR, LF, or the trailing erase-to-end-of-line, reading through buffer.read1 with an incremental UTF-8 decoder so a multi-byte character straddling two reads survives, and strip the CSI escapes before the record reaches on_log or the error tail. Streams without a raw .buffer (test doubles, non-pipes) fall back to line iteration. The new backend test drives the real byte stream through both halves, one flush per read, and asserts each step is visible on the read that carried it rather than one redraw later. * Give MiniMax-H3 first and last frame conditioning in the video backend MiniMax-H3's released transformer is the FL2VA one: text-to-video is the same checkpoint run with no keyframes. Studio only ever ran it text-only, so the Video generate request had nowhere to attach a reference frame. The load used to prune the block graph to t2va. That argument prunes STATICALLY, so an fl2va-pruned pipeline runs the keyframe blocks on every request and cannot serve a text-only one at all: it raises packing an empty conditioning list. The load now keeps the whole auto graph, which selects per request, and bounds load_components to the keyframe workflow's component set instead, so the 61.7 GB Ref2VA partition is no more loaded than before. Measured against the released checkpoint: a text-only request through this pipeline is bit-identical, video and audio, to the same request through a t2va-pruned one. A keyframe is a geometry anchor, not just conditioning, so the canvas comes from its aspect ratio through the released arithmetic (768 short edge, area capped at 768x1344, both axes rounded to 32) rather than from whatever resolution preset was selected. An arbitrary size produces a garbled clip rather than an error. sd.cpp already implements the same conditioning, so that side is the existing --init-img / --end-img flags with the frames staged as PNGs. Only MiniMax-H3 declares the capability, and status reports it, so Wan and LTX do not grow a control that does nothing. * Cover the MiniMax-H3 keyframe path with tests Registry (which families declare it, and the canvas a keyframe resolves to), the load wiring (whole block graph, component set still bounded, VAE encoder kept), request handling (decode, refusal, canvas override, what reaches the pipeline call) and the sd-cli argv. * Add the reference-frame controls to the Video page First and last frame pickers, shown only for a family whose status declares keyframe conditioning, so Wan and LTX are unchanged. The Images page's source picker moves to a shared component rather than the Video page growing a second upload path; both send the same data URL to the same backend decoder. While a frame is attached the Resolution preset is disabled and says why: the frame's aspect ratio resolves the clip's size, the way the model itself does. The gallery recipe records which ends were pinned. * Check the keyframe canvas against the pipeline's own resolver The canvas rule is a checkpoint contract, so pin it to the released implementation rather than only to hand-written expectations. Skipped where diffusers does not ship MiniMax-H3, which is most runners. * Revert the standalone H3 keyframe implementation oobabooga/unsloth#121 covers first frame, last frame, first-and-last, Ref2VA and the canvas rule, and it reached the same load construction independently. Two implementations of the same feature on one branch is worse than either, so this takes mine back out and leaves the branch ready for that work to land whole. The one finding worth keeping from it is already reflected there: passing workflow= to ModularPipeline.from_pretrained prunes the block graph statically, so the pipeline must be built unpruned and only load_components bounded. * Keep the pre-quantized MiniMax-H3 denoiser resident so a generation can run Loading H3 with a hosted pre-quantized denoiser worked, but every generation died on its first denoise step: Attempted to set the storage of a tensor on device "cuda:0" to a storage on different device "cpu". This is no longer allowed; the devices must match. ComponentsManager.enable_auto_cpu_offload parks every component on the CPU and moves each one onto the accelerator inside its own pre_forward, that is from within the block that is already executing. The text encoder and the VAEs survive that; a torchao-quantized denoiser does not, because the device change reaches return_and_correct_aliasing, which tries to alias a CPU storage to an accelerator tensor. Moving the same module at load time, outside any executing block, works. So place it once at load and take it out of the offload rotation: drop its hook, unlist it from the other components' eviction candidates, and move it. Everything else is unchanged, and the encoder and VAEs still offload around it. Keeping it resident is what asking for a quantized denoiser buys in the first place: the hosted checkpoint is about 20 GB against 66.3 GB dense. Verified end to end on a B200: MiniMaxAI/MiniMax-H3 loaded with the hosted fp8 denoiser, then a 1280x768, 124-frame clip generated from a start frame in 167s. The clip's first frame matches the supplied image and the motion is coherent. * Apply the pinned Diffusers revision on a fresh install, not just an update MiniMax-H3 needs a Diffusers revision newer than any published release, and Studio refuses to load it otherwise. The pin was in studio/backend/requirements/base.txt, and a clean install.sh run still ended up on diffusers 0.39.0 from PyPI, every time, with nothing in the log to say so. base.txt is never installed by install.sh. install.sh installs unsloth itself, whose own metadata pulls a diffusers release in transitively, and then runs install_python_stack.py with SKIP_STUDIO_BASE=1, where the base-packages step is a bare `pass`. So the pin applied on `unsloth studio update` and on the no-torch path (install.sh installs no-torch-runtime.txt directly) and was dead on exactly the path a new user takes. Reproduced on a clean install into a throwaway prefix before and after: 0.39.0, then 0.40.0.dev0 with MiniMaxH3Transformer3DModel present. The revision now lives in its own diffusers-pin.txt, installed by a step that sits outside every skip_base / NO_TORCH branch and after every other requirements file, so nothing left in the run can re-resolve diffusers back to a release behind it. No forced reinstall is needed: a direct URL requirement is not satisfied by a resident registry install, so the step is a no-op once the environment is already on the pin. tests/studio/install/test_diffusers_pin.py holds the shape in place: exactly one requirements file may name diffusers, the pin must be a full commit sha rather than a moving ref, the install step must sit at function top level rather than under a conditional, and it must come after every other requirements install. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: let the Video and Images pickers see a local diffusers pipeline A model already on disk never reached the pickers' On Device list unless it happened to keep a weight file beside a root config.json. Every image and video model downloaded as a pipeline keeps its weights in component subdirs under a root model_index.json instead, so the hub inventory scan behind /api/hub/local rejected it. Pointing Custom Folders at one was worse than empty: the LM Studio publisher walk descended into the pipeline and offered vae, transformer, text_encoder and audio_vae as four separate models, none of them loadable. Teach the hub scanner the same pipeline-root test routes/models.py already applies, in the three places that only accepted a root config plus loose weights, and keep a pipeline row through the custom-folder format filter: the layout has no loose weight to classify, so the row is "unknown" by construction rather than by fault. * Cover the local diffusers pipeline scan with tests * Pin what the pipeline exemption must not let through Three gaps in the cover added with the scan change, each found by mutating the fix and watching the suite stay green. The custom-folder format filter now waves a row through on its shape, and nothing said what it still has to reject: replacing the whole predicate with True passed. A folder holding a config.json and no weights, which an aborted download leaves behind, reports the same "unknown" format and no loader can start it, so it pins the boundary. The predicate is applied to every row the filter did not already accept, and a row's path can be a GGUF file rather than a directory. A missing path, a file, and a directory whose model_index.json is itself a directory must answer False rather than raise, because an exception there fails the scan and empties the picker. The publisher walk was only covered one level up. Adding the model folder itself as a scan folder is the obvious thing to do and used to publish vae, transformer and text_encoder as three models. * Drop the unused H3_TASK_KEYFRAMES import from the video backend video.py only branches on H3_TASK_REFERENCES; the keyframe constant is read from video_minimax_h3 directly by the tests that need it. The hoisted-import safety net in Source lint flags the unused name as a blocker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage the hosted pre-quantized H3 denoiser in the video download plan The plan already drops the dense transformer shards whenever a hosted pre-quantized checkpoint covers them, but nothing put that checkpoint back: an int8 or fp8 H3 stage skipped 66.3 GB of base shards and added none of the 20.25 GB artifact the load actually opens. The byte total under-reported the stage by the size of the checkpoint, the disk preflight cleared a volume that could not hold it, and an offline stage completed without the one file the load needs. _denoiser_prequant_hub_files mirrors the pre-cast encoder helper: it resolves the family's hosted checkpoint, confirms the file really exists on the Hub, and prefers the repo-root name over the legacy scheme name in the same order the load tries them. An unreachable repo is logged and yields no files, so a gated or renamed artifact keeps the dense shards instead of sinking the plan. The progress-bar estimate is deliberately left alone: it counts cached bytes for the checkpoint and base repos only, so adding a third repo there would leave the bar permanently short of 100 percent. * List both H3 denoiser partitions in the picker, not only FL2VA The bundle-repo filter accepted only minimax_h3_fl2va*, so every published minimax_h3_ref2va* quant was hidden from the variant listing. The loader disagrees: validate_h3_transformer_filename accepts either partition, on the grounds that which one is picked IS the task, and h3_transformer_task routes Ref2VA to the reference-video workflow this PR adds. The community bundle repo publishes three Ref2VA quants today, so the reference path was unreachable from the remote catalog. Accept both prefixes from one shared tuple and keep excluding the Qwen3-VL encoder and VAE companions, which are never picks for either partition. The filter test asserted the old behaviour and is updated with it. * Bound H3 reference-audio decoding to the trained window The reference-video decoder already selects, resizes and refuses incrementally because the encoded size says nothing about the decoded size. The audio decoder did not: it appended every resampled block to a list and then allocated a second full buffer in np.concatenate, with no duration or sample cap. The route accepts 32 MiB of encoded audio, which is over half an hour of compressed stereo. That lands as roughly 1.9 GB of float32 and doubles again in the concatenate, and up to three references are accepted per request, so an ordinary long music or podcast file picked by mistake could exhaust the host before the background job even started. H3's reference window is 15 seconds, so anything past it is unusable rather than merely large. Refuse it while decoding, with the same shape of message the video guard uses, instead of decoding it first. * Refuse a quantized H3 reference load instead of seeding the keyframe denoiser The hosted pre-quantized checkpoints are FL2VA (keyframe) denoisers. Ref2VA shares their module shapes and the same base model, so resolve_prequant_source handed one back for a reference load, it passed every metadata check, and seeding it made load_components skip the real Ref2VA transformer. The request then generated from the wrong partition rather than failing, which is the worst of the three outcomes. The route accepts h3_task, so this was reachable from the public API even though the picker does not expose the choice yet. validate_load_request now refuses the pairing with a message naming the workable alternatives, in the same place it already refuses a scheme with no hosted checkpoint, and the modular loader drops to the released components if a direct call reaches it. Nothing changes for keyframe loads, which are what the checkpoints are. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Skip modular-workflow families in the dense text-encoder plan assertion The plan-unchanged sweep from main walks every family and asserts the dense budget plan_diffusion_memory received. MiniMax-H3 is the first modular-workflow family to reach that list, and load_pipeline dispatches to the workflow's own loader before the planner runs: each component is built by its own from_pretrained, so there is no single dense pipeline to budget and no plan call to assert on. Skip it the way the sweep already skips wan2.2-t2v-a14b. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope the paired-axes canvas rule to keyframe requests A half-specified width/height is long-standing, documented API behaviour: the backend resolves the missing axis from the family's default preset, in both validate_video_request_shape and _resolve_keyframes. Applying the new paired-axes rule as an unconditional request validator rejected those calls with a 422 before family validation ever ran, breaking existing LTX, Wan, Hunyuan and prompt-only H3 clients. The rule still holds where it means something: with a keyframe present the canvas is matched to the source aspect whenever either axis is missing, so the axis the caller sent would be silently discarded. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the CUDA sd-cli pin and translate a mirror-only tag upstream The merge with main brings in the accelerator-aware installer, whose fallback translates a mirror-only -u<id> pin back to the upstream release it was built from instead of skipping the upstream attempt. That is strictly better: skipping kept the round trip cheap but dropped the pin entirely on every host the mirror does not build, leaving them on upstream latest. test_a_mirror_only_pin_is_never_requested_upstream asserted the old shape, that the fallback settled for upstream latest. It now pins the new one: never the literal -u<id> string, the translated release instead, and no latest attempt at all because the translated pin succeeds. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-08-09 10:05:58 -07:00
Daniel Han	c6ad59a7d1	Studio: stop leaking child processes on an abnormal exit (#8170) ... * Studio: stop leaking child processes on an abnormal exit A Windows user could not update Studio until they killed a stray python by hand: the tool sandbox runs its payload under a shell wrapper, the kill path reaped only the wrapper, and `unsloth studio update` then refused to run because a process still held the managed environment. - taskkill /T on Windows, so a tool payload cannot outlive its wrapper - a console-close handler, since CTRL_CLOSE_EVENT never becomes a Python signal and the graceful shutdown was skipped entirely when the window was closed - the desktop updater drains the app job before standing down crash cleanup, and re-arms it when the install never happens - children are recorded on disk and swept at the next startup, which is the only reaper macOS has after a crash or a force quit - the job status is logged instead of failing silently Also fixes a liveness probe that used os.kill(pid, 0); on Windows that is TerminateProcess, so it killed the process it was asking about. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Review fixes: console handler must not touch signals, one child record per owner, drop the job drain - the console handler ran _signal_handler on the thread Windows creates for the event, where signal.signal raises, so closing the window did no cleanup at all - bound that work to the ~5s Windows allows before it kills the process - one record file per owner pid: two Studios can share a home, and a single file let the second erase the first's children - add a Windows process identity (creation time) and refuse to signal a pid that cannot be verified - drop the whole-job drain: it would also terminate the WebView2 hosts, and cleanup_child_processes already taskkills the backend tree * Harden the child record against a malformed or older file A record that is not an object, or whose children are not dicts, raised out of the startup sweep and would have stopped Studio from starting. Pair the Linux start time with the command name as well, since start time alone has 10ms granularity. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Close the gaps the first pass left Ctrl+C on Windows raised UnboundLocalError inside the console callback, where the BOOL result is then undefined, so the event could be reported as handled and Studio would not stop. The updater no longer re-arms kill-on-close before relaunching, which would have made the old process kill the replacement it just started, and it resets the exit-cleanup guard so a retry after a failed installer still reaps the backend. The RAG embedder and cloudflared are recorded like the other sidecars, a llama-server that survived a failed kill stays recorded, the Windows kill path checks the captured creation time before taskkill, and record writes are serialised. * Give the Popen doubles the pid a real one always has * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fail closed where the answer is not certain The delayed Windows kill skips a captured pid it cannot verify, since the job object still takes the tree at exit. An owner whose identity cannot be read counts as live rather than gone, so a momentary ps failure no longer costs a running Studio its sidecars, and that lookup pins TZ so a timezone change does not read as a different process. A child that outlived terminate_all keeps its record instead of losing the only handle on it, with zombies told apart from survivors. The whole relaunch handoff is inside the recovery scope, so any path that leaves this process running re-arms cleanup. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Never signal a pid the graceful sweep cannot verify terminate_all now applies the same test the startup sweep does: a pid whose identity cannot be read is left alone and kept in the record for the next launch to retry, rather than signalled on the chance it is still ours. The startup reaper re-checks liveness before dropping a record, so a kill that did not take stays reapable, and the breadcrumb unlink happens under the record lock so a concurrent adopt cannot have its record deleted from under it. * Only claim the guarantee when it is actually there Linux startup probes prctl with the read-only PR_GET_PDEATHSIG before reporting the parent-death signal as in force, so a seccomp or container policy that blocks it is reported as such rather than as a guarantee nothing keeps. The backstop sweep takes its snapshot under the lock the writes already hold. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Record every lifetime-bound child, and reach a Windows tree without its leader The DiffusionGemma runner and sd-cli were spawned with lifetime kwargs that are empty on macOS and never recorded, so nothing could reap them. Both now adopt at spawn. The Windows tool capture also revalidated a pid that no longer exists once the wrapper exits, which is the case it was added for; each tool tree gets its own job object instead, with the pid path as the fallback. The startup breadcrumb sweep takes the tree too, not just the leader. Probe PR_SET_PDEATHSIG itself, since seccomp can filter prctl per operation, and stop the post-update backend restart when kill-on-close cannot be re-armed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Give the diffusion runner its own group, and gate every restart path The runner shares Studio's process group, so the startup sweep could only signal the runner itself and its visual server kept the GPU. start_new_session makes it a group leader, which is what _posix_terminate needs to killpg. Skip and Restart is offered on every error, so gating only the recovery path still let a user start a backend while kill-on-close was disarmed. The flag moved to a ref both paths check. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reap a group whose leader is gone, and drop comm from pid identity A recorded leader can exit first and leave its group running, and the sweep then skipped the entry and deleted the record. The child's own process group is recorded at adopt time (only when it leads one, never Studio's) and signalled when the leader is gone. The group id is the dead leader's pid, which the kernel holds while any task still references it as a group, so it cannot belong to anyone else. comm is mutable, so a worker calling prctl(PR_SET_NAME) or setproctitle read as a recycled pid and was dropped unsignalled. Identity is the start time alone; records written with starttime:comm still compare equal. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep a group's record until the group is gone forget_pid dropped the recorded group as soon as its leader was reaped, so a shim exiting before its visual server took the only handle on that group with it. The record is kept while the group still has members, and the backstop then reaps it. The Windows backstop now takes the tree, matching the startup sweep, and the identity probe prototypes CloseHandle like every other handle-width call in this module. * Put a retained child's group back with it terminate_all pops the recorded group before deciding what to do, and both paths that put the pid back dropped it. A leader exiting later then left nothing able to reach its descendants. * Believe only confirmed kills in the sweep taskkill returning nonzero was treated as success, so the documented single-pid fallback never ran. A group that survived SIGKILL reported itself resolved, which deleted the last handle on it. And one sweep pass could skip a worker record whose owner was terminated later in the same pass, so it repeats while it keeps finding things. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep records the sweep could not resolve A group that survived termination left unresolved false, so the breadcrumb was deleted with the group still running; the backstop had the same gap. A zombie owner also read as a live Studio and shielded all of its sidecars. The retry path re-enters installUpdate and spawns the backend updater, so every path that starts a child goes through the same re-arm check. The revisit pass now reconsiders only records deferred for a live owner, so nothing is signalled twice. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Record tool subprocesses, and do not mistake zombies for a live group The tool subprocesses lead their own session on POSIX but were never recorded, so a force quit mid-call left them with nothing able to find them. They are adopted at spawn and forgotten on confirmed exit. A leader terminated while alive can leave its group behind, which is the same loss of the only handle as the dead-leader case. Checked in both the sweep and the backstop. killpg(pgid, 0) succeeds for a zombie, so a finished group read as alive and would have kept its record forever. Membership now ignores zombies. cloudflared is adopted under the lifecycle lock, before a concurrent stop can reap it and leave the adoption to record a recycled pid. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Record the component installer, and never wait on a zombie stream_installer passed the lifetime kwargs but never adopted, and those kwargs are empty on macOS, so an installer outliving its owner kept rewriting files under the next launch. A zombie answers every liveness probe, so terminating one burned the full grace period per record and reaped nothing. Zombies take the dead-leader path, which still reaps a group they left behind. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reset the record lock after a fork, and ask the job whether it is armed A fork while another thread was inside adopt_pid leaves the child with the record lock held and no thread to release it, so the next adoption there blocks forever. The child-side reset already rebuilt the spawner lock and now rebuilds this one too. A webview reload rebuilds the update hook with its re-arm flag back at its initial value while the Windows job can still be disarmed, so the first gate after a mount reads the job's own limit flags instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Retry a child's identity, and treat an unanswerable query as disarmed A ps that timed out once was recorded as no identity at all, and an entry without one is never signalled, so that child survived every later launch. The capture is retried while the process is still there. On the desktop, a failed desktop_update_cleanup_armed is not evidence that kill-on-close is in force, so the gate now fails closed and re-arms. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Start the component installer in its own session, and pin the owner identity The installer spawns a validation llama-server, and PDEATHSIG reaches only the direct child while the startup sweep can signal only what the record names. Leading its own group puts the whole installer tree within reach of both. The owner identity is captured once through the same retry a child's goes through: recorded as None, any process that later reuses the pid reads as the owner still running and the children in that record are never reaped. A fork child drops it, since its pid is not the parent's. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the installer in the group the desktop kills, and heal a missing identity Reverting the new session from the last round: the desktop stop path force-kills this backend's process group, and a session of its own took the installer out of it, so a wedged or SIGKILLed backend left the installer still rewriting files. Group membership is the stronger guarantee of the two; the record still names the installer for the macOS sweep. A child whose identity could not be read is retried on every breadcrumb write while it is alive, so a probe that failed once no longer leaves an entry nothing will ever signal. A fork child also drops the inherited pid registries: adopting anything would have written a record claiming its parent's children. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Install the fork reset where the first record is written It was registered only from the Linux spawn path, which returns before that everywhere else, so on macOS a fork child kept this process's children and a record it wrote later claimed them. * Take the diffusion group down on stop, and do not wait on a zombie The desktop shutdown only waits for the ordinary stop path, so that is where the shim's own group has to go: the group is captured before the wait reaps the leader and killed once the leader is gone. The session of its own stays, since it is what lets the startup sweep reach a visual server after a crash. A group holding nothing but a zombie answers killpg(pgid, 0), so without a members check every stale record cost the full grace period where pid 1 does not reap. A ps that exits nonzero is not an empty group: reporting it as one let forget_pid drop the only record of a live descendant. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep the re-arm failure message off the Studio branding * Studio: do not spend the shutdown budget waiting on a child that has exited An exited child nobody has waited on answers signal 0 exactly like a live one, so the terminate wait spent its full timeout on a process that was already gone, once per tracked child and in series. Measured 15.1s for three of them, 0.16s after this. The group path keeps waiting while a member is still running, since that is what holds the GPU. The state read costs a fork off Linux, so it happens twice a second rather than at the poll rate. * Studio: answer the group check from the leader instead of scanning every process Enumerating a process group reads the state of every process on the machine, which on a busy box is 60ms+, and forget_pid did it on each stop. A leader that is running already settles the question, and a pid that was never recorded has neither a group to check nor a record to rewrite. Measured 62ms -> 0.01ms per stop; the leader-has-gone case still falls through to the scan, which is what finds a child holding the GPU behind it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: reach the installer's validation server, and never start a backend under a disarmed job The installer starts a llama-server to validate a build. It is a grandchild of the backend, so a parent-death signal reaches the installer alone and the sweep had nothing recording where the server was: an abnormal exit left it holding the GPU and the staged files. The installer now announces it on stdout, in its own process group, and the update flow adopts it for as long as it runs. On Windows the armed check moves to the path that spawns: the UI gate runs per update action, but a webview remount can start a backend on its own, which is exactly the orphan the job object exists to prevent. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: clear the cleanup guard with the re-arm, and survive a malformed record Re-enabling the Windows job on the spawn path left TERMINATION_CLEANUP set, so the next update attempt read cleanup as armed, skipped the resume, and its pre-exit hook suspended kill-on-close without stopping the backend first. An identity read back from a record can be any JSON value; reaching split() with a number raised through the whole startup sweep, so one bad file left every other orphan running. Non-strings now read as unverifiable, which keeps the pid unsignalled rather than trusted. A group whose members exited on the SIGTERM keeps answering killpg(pgid, 0) where pid 1 does not reap, so the reap now rechecks membership instead of waiting out the grace period. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep the record when taskkill leaves the tree standing The leader-only fallback runs when the job object was unavailable, which is exactly when the record is the only handle on those workers. Both callers read the dead leader as the tree being gone and dropped it, so anything that survived became unreachable. The tree kill now reports whether it took, and a failure keeps the pid tracked and its record on disk for the next launch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: wait for the group, not the leader, before dropping a record The installer announced its validation server as stopped as soon as the group leader exited, which drops the record while a child that ignored the SIGTERM is still holding the GPU. It now waits for the group to empty, escalates to SIGKILL, and only announces the stop once nothing is left. An installer timeout killed the installer alone and left the announced server for a sweep that never runs while this process lives; those children are now terminated with it. The diffusion group id is kept from the spawn, so a shim that exited before the kill path (a failed health check, a crash before a reload) no longer leaves its visual server with nothing able to reach it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make the diffusion group assertion formatting-agnostic * Studio: stop announced validation servers on any installer exit The timeout path took them; a nonzero exit or a stream that ended mid-line left them running. This process stays up after an update failure, and its own live record shields those pids from a sweep that would not run anyway, so the cleanup now happens in the finally that covers every way out. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: reach a validation server whose leader or tree kill is gone terminate_pid falls back to the recorded process group when the leader has already exited, keeps the record when a Windows tree kill could not be confirmed, drains the announced children under a lock so the watchdog and the reader thread cannot race, and the installer arms the parent-death signal on the validation server it puts in a session of its own. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the diffusion runner in the group the desktop stops, and check identity before a single-pid kill The desktop stops this backend by signalling its process group and force-kills it five seconds later, so a runner in a session of its own survives a backend that is slow to shut down and keeps the GPU until the next launch sweeps it. Put it back in that group, as the component installer already is, and reach the visual server by walking the runner's children instead of killpg. The cached group id goes with it: a pid is reusable once nothing holds the number as a process group any more, so an id kept past its group eventually names a stranger. terminate_pid signalled on the pid alone. An announced validation server can exit without the line that clears it, so run the same identity test terminate_all does before either termination branch. * Keep a macOS validation server in the installer's process group The server was started in a session of its own everywhere, but only Linux can pair that with a parent-death signal. On macOS it left the group Studio force-kills while the only record of it is the announcement the backend has yet to read, so a kill in that window orphaned it with nothing able to find it. It stays in the inherited group there, and the kill path only reaches for killpg when the server actually leads a group, so a shared group is never signalled. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-09 08:06:29 -07:00
oobabooga	df5f139bac	Studio: add image generation, editing workflows and LoRA training with Unsloth GGUFs (#6763) ... * Tighten comments in the image stack tests and scripts * Close video single-file, training reservation, and image mount-resume gaps Route on-device single-checkpoint video folders through the single_file loader: a bare local .safetensors directory (no model_index.json) is advertised as a pipeline with no filename, so validation rejected it before it could load. Reinterpret the pick as a single_file load of the sole checkpoint, mirroring the image load route. Treat a reserved-but-not-yet-spawned LLM training start as active in is_training_active() so /images/load, /video/load, and /diffusion/start cannot race the reserved run for VRAM during the pre-spawn free window. Mirrors the diffusion training service reservation. Resume an in-flight image generation on the Images page mount: probe generate-progress, re-enter the poll loop, and refresh the gallery on completion so a run started elsewhere is reflected and its saved image appears without a manual refresh. Seed resident image defaults from the resolved base_repo rather than a possibly path-shaped repo_id so the first resident generation uses the right recipe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Publish image generation active state before pre-denoise setup generate() assigned self._gen only at the pipe() call, after deferred compile, LoRA resolution/application, and ControlNet download/build had run. Across that setup window generate_progress() reported inactive even though _generate_lock was held, so a reloaded page's mount probe showed idle and let a second generate queue behind the first. Publish an active step-0 _GenState the moment the generation lock is acquired, before the setup work, and clear it in the outer finally so a setup-time error cannot leave the UI stuck active. Mirrors the video backend's queued phase and the training start guard. * Studio: fix diffusion install ownership, dataset upload atomicity, gallery pagination, and teardown races install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the install created the target directory or it was empty. Adopting a pre-existing, unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made it eligible for the uninstaller's recursive delete. routes/training upload: make the multi-file promotion transactional. Back up each displaced original and roll every destination back on any failure, so a mid-loop rename error can no longer partially overwrite the live dataset. routes/training _resolve_dataset_folder: reject a symlinked dataset directory and prove the resolved folder stays under the datasets root, so image read/caption/delete cannot escape the root through a link. routes/training delete: escape glob metacharacters in the thumbnail filename so deleting an image named like [ab].png removes only its own thumbnails. image_gallery / video_gallery listing: filter records against the response schema inside the pager via a valid callback, so offset/limit/has_more all count over accepted records. A leading schema-invalid record no longer returns an empty page with has_more=true and stalls infinite scroll at offset 0. image_gallery / video_gallery save: publish via a temp file plus atomic rename (the sidecar is the video pair's commit marker) and clean up on failure, so a partial write never surfaces a truncated PNG or strands an orphan MP4. diffusion_train_common discovery: treat an empty caption sidecar as a metadata tombstone that still falls through to the dreambooth instance prompt, so clearing every metadata caption no longer fails with no captioned images found. diffusion backend unload: wait for an in-flight denoise to exit before tearing down process-wide patches and state, mirroring the load path. diffusion_engine_router: serialize the whole check/unload/publish transition so a concurrent selection cannot return the engine being unloaded. uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a user's own sd-server is not terminated for a directory we then keep. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject native batch seeds outside the JSON-safe range * Refuse sd.cpp install into unowned non-empty target dir When the install target already exists, is non-empty and lacks the .unsloth-studio-owned marker (a user's own stable-diffusion.cpp checkout, or unrelated files beside a custom Studio root), install() previously still extracted the release into it. Skipping the ownership marker only stopped the uninstaller from deleting the directory; extraction still merged binaries into the user's working tree and could overwrite same-named files. Fail up front with a clear message pointing the user at a fresh/empty location before any download or extraction, leaving their directory untouched. Update the ownership test suite to assert the refusal. * Studio: gate dataset uploads on the symlink check and surface local video single-file checkpoints * Tighten comments and docstrings added by the image-generation fixes * Studio: close arbiter load-registration race and surface native progress + local pipeline folders Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path. Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once. Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: include marker-owned custom sd.cpp roots in the uninstall stop scan; harden Pester install against the nuget.exe PSGallery bootstrap * Studio: surface local pipeline scan roots, tag single-file checkpoints by filename, mark companion-only pipelines partial - _scan_models_dir: admit a scan folder that is itself a diffusers pipeline (root model_index.json, weights in transformer/ vae/ subdirs). _is_model_directory rejects such a root, so the child scan would list the component subdirs as bogus models and hide the real pipeline; treat the root as one model via _local_pipeline_index. - _local_is_diffusers / _local_model_task: include the sole checkpoint filename in the family-detection needles (_local_family_needles, resolved via resolve_local_single_file). A generically named folder holding one loadable qwen-image-*.safetensors / ltx-*.safetensors identifies its family only from the filename; the load route already resolves that file, so tag it or the task-scoped picker (which rejects task=null) hides the on-device model. - list_cached_models: mark a companion-only base snapshot partial. A GGUF image load prefetches the base repo's VAE / text-encoder / model_index.json but skips the transformer (the GGUF supplies it); the snapshot has a pipeline manifest yet is not a loadable BF16 pipeline, and _cached_repo_partial misses it. _repo_pipeline_missing_denoiser flags a pipeline snapshot whose transformer/ or unet/ component carries no weight, so the picker drops it instead of advertising it as fully on-device. * Studio: preserve foreign gallery files, force safetensors on remote ControlNets, and close dataset/seed/GPU gaps Gallery clear/delete now scope to Studio-owned files: image_gallery and video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or orphan file the listing already hides), so clear() and a guessed-id delete no longer destroy files the gallery never surfaced. Remote ControlNets now force use_safetensors: a bare owner/name reaches from_pretrained without the base trust gate, and the Hub scan fails open when unavailable, so requiring safetensors closes the pickle deserialization vector. POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root before removing the tree (marker-gated), mirroring the Windows stop-before-delete scan; a live native server no longer survives unlinking its binary. Diffusion dataset containment: the training-start read path and the discovery picker route bare names through the protected resolver, so a symlinked dataset is rejected / not advertised like the caption/delete routes already do. Uploads gain the inference decode guard (oversized real images 400 before OOMing the trainer) and dataset upload/caption/delete/import are blocked with 409 while a diffusion run is active. JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8 instead of raising AttributeError / 500. LoRA family compatibility is enforced in the shared resolver, not only the picker, so a direct API client cannot apply a mismatched-family adapter. GPU arbiter gains release_if so the image/video unload idle-check and release are atomic against a concurrent same-owner load's registration. Native batch recipes persist the base batch_seed and restore replays from it, so a native batch_index>0 image no longer advances its seed twice. FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B, 9B -> Qwen3-8B) instead of the single family default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: track the loaded GGUF filename so native companion resolution reproduces the load identity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate local-pipeline image tagging on a real family; validate video sidecars before delete/clear * Studio: tighten image-generation fix comments and docstrings * Studio: gate gallery serve/export on ownership; keep image progress active until persisted; reserve diffusion training before the dataset scan * Studio: restore Reapply target on async image/video load errors; recheck training state before dataset commit; reclaim partial sd.cpp installs on retry Images/Video: a background model load that fails AFTER starting (error/eviction during download) leaves the previous pipeline resident, but handleLoad had already overwritten lastLoad.current with the failed pick, so "Reapply to loaded model" reloaded the failed model. Carry the prior Reapply target into the poller and restore it on the async error/null paths, mirroring the quant rollback. Training: an in-flight diffusion dataset upload passed _require_diffusion_dataset_mutable() at entry but could still commit files after a concurrent /diffusion/start reserved the training slot, mutating the dataset underneath the trainer. Re-check the interlock immediately before the commit phase; a 409 there leaves the staged temps for the finally to clean. sd.cpp install: an interrupted extraction (disk full, killed process, a raising post-extract cudart fetch) left the target non-empty with no owner marker, so the next lazy install tripped the "not a Studio-managed directory" refusal and wedged native install. Write the ownership marker before the partial writes when the target is reclaimable, so a retry recognises the debris as ours and re-extracts. * fp8 DiT quant: floor the dynamic activation scale with activation_value_lb An all-zero activation token row makes the dynamic per-row fp8 scale 0, which turns the quantized data to NaN and the render to black frames on torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels clamp zero rows internally, so the bug only reproduces on machines without them, which is most user environments. Zero rows are real inputs, not a corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and Qwen-Image regenerate zero rows inside their transformer blocks every step. Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig whenever the installed torchao supports the kwarg (Float8Tensor rework, 0.13+), checked via inspect.signature so older torchao keeps exactly the current behaviour; the existing Float8MMConfig fallback chain is unchanged. Verified on GPU: with the forced plain-torch kernel path a zero-row input NaNs without the floor and stays finite with it, and end to end on HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal render (LPIPS 0.225); on Wan the floor matches the condition_embedder exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels present are unaffected, and pre-quantized fp8 checkpoints stay valid since weight scales are untouched. * Wire hosted pre-quantized DiT checkpoints into the image families Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image (int8 only there; fp8 is family-denied), z-image and krea-2 at the unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8 transformer checkpoints, so the fast quant path loads the small pre-quantized file instead of materialising the dense bf16 transformer and quantising on device. Measured on FLUX.2-dev int8: build peak drops from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical 30.7 GB resident after either path since loading a checkpoint is bit-identical to on-the-fly quantisation. The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source now derives that model-name filename from the repo id (scheme suffix stripped case-insensitively) and carries the legacy transformer_<scheme>.pt as a fallback the resolver tries when the primary 404s, keeping older repos loadable. Wiring a repo also exposed a fallback hazard: with a prequant source present, the dense-fit preflight used to be skipped entirely, so a failed prequant download would fall through to the dense bf16 load the memory plan never budgeted, OOMing after eviction. The preflight now always runs and gates an allow_dense_fallback flag through _load_dense_quant_pipeline: a dense misfit still skips the fast path when no prequant exists, but with one it proceeds and a prequant failure raises to the GGUF build instead of loading dense. The same flag is set when the auto-policy replans an offloaded GGUF against a prequant-sized transient. Tests updated to the new filename convention plus new coverage for the derivation and the legacy-name fallback; the prequant-skips-refit test now asserts the re-check runs and forbids the dense fallback. Verified end to end on GPU: z-image int8 resolves the hosted repo, downloads the model-name file and renders (6.8s load, 5.9 GB peak). * Route krea-2 through its per-component loader on the transformer-quant fast path _assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take (Krea2Transformer2DModel has no from_single_file). Assemble per-component via load_krea2_pipeline like the pipeline-kind and single-file paths already do. Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and render through the Studio images tab. * Keep Qwen-Image's text-stream linears bf16 on int8 (short prompts break torch._int_mm) Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than 16, but got 13' (measured on B200 through the Studio images tab). Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme, family) and the prequant checkpoint validation, so a checkpoint baked under the old token list is rejected and re-quantised instead of loaded crashing. The text stream runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069). * Harden the diffusion memory plan against transient free-VRAM undercounts A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model (companions exceed budget) and silently served the GGUF as-is; the identical retry went resident and engaged the hosted prequant. The plan arithmetic was byte-identical across both loads (required 90,228 MiB, resident needs free of about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is device-wide and instantaneous: a transient foreign CUDA context briefly held about 100 GB at the first snapshot, and the planner trusted that single read. Three changes: - settled_snapshot_device_memory: on cuda, synchronize + empty_cache (best-effort) and take the MAX free over up to 3 spaced reads. A transient can only shrink free, so the max rejects transient undercounts while a persistent tenant still caps every read. _plan_memory now uses it. - plan_fits_total_capacity + one replan retry: when the dense/prequant candidate fits TOTAL device capacity under the standard reserve and the 0.85 resident margin, an offload verdict can only stem from the free reading, so the loader re-snapshots and replans once before declining the fast path. Explicit balanced/low_vram modes skip the retry (they offload by mode). - diffusion.transformer_quant_declined log line with required/budget/free and the plan reasons, so the next decline is diagnosable from the server log (previously silent). Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the hosted prequant resident (offload=none). * Add FLUX.2 Klein and FLUX.2-dev DiT LoRA training Register flux.2-klein and flux.2-dev in the DiT trainer following the upstream DreamBooth references: latents train patchified and batch-norm normalized from the VAE posterior mode, the packed forward reuses step-invariant position ids, and the guidance vector (3.5) is gated on the variant's guidance_embeds config. Conditioning stacks load per variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline for Klein) and are encoded and freed before the transformer lands on the device. The fused single-stream to_qkv_mlp_proj joins the attention projections in the LoRA targets; the single-stream out projection stays dense because its to_out suffix would also match the double-stream ModuleList container. Wire both families through the training registry (family set, labels, VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them trainable with train base repos in the family registry, add FLUX.2-dev to the gated-repo token check, and trust both official bases for training downloads. Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev int8 (52.0s) runs train with finite decreasing loss and the saved adapters apply on the bf16 base pipeline (weight 0 reproduces the base image exactly, weight 1 visibly restyles it). * Support LoRA adapters on torchao int8/fp8 quantized image pipelines Adapters are baked at load time: they attach to the dense transformer, then quantize_ converts only the frozen base linears (the lora_ side path is excluded by name), then the loader compiles. Post-quant PEFT injection is not possible on a manually quantized module, so the prequant shortcut is skipped for a baked load and the memory plan is sized for the dense build (force_dense on the quant candidate). At generation time the baked topology is frozen: weight tweaks and disabling (scale 0 reproduces the quantized base exactly) go through set_adapters, while adding or removing adapters returns a clean 400 telling the client to reload with the new selection. supports_lora now returns True for int8/fp8 diffusers loads (checked before the gguf-kind early return, since the quant fast path keeps the picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The load request model takes an optional loras list, threaded through begin_load on both engines (native ignores it and keeps applying LoRA at generation). Verified end to end on GPU: Z-Image GGUF picker + int8 + trained adapter loads through the API, bake marker logged, weight 1.0 vs 0 renders differ visibly, weight 0.5 accepted live, unknown adapter rejected as 400. Affected suites: 296 passed. * Add FLUX.1 Krea dev to the image model catalog Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it runs under the existing flux.1 family unchanged. Wire it up end to end: - Catalog group with the gated official bf16 pipeline and the open QuantStack GGUF quants; the gated artifact is skipped on auto-routing when undownloaded. - Trust the official repo for non-GGUF from_pretrained loads, next to the other black-forest-labs bases. - Generation defaults: 28 steps at guidance 4.5 per the model card. The generic "krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the id, which would have produced garbage output; the new flux.1-krea key precedes it on both the backend table and the images page table. - The flux.1 prequant checkpoints are schnell-based; the loader's baked base_model_id validation refuses them for the Krea-dev base, so int8/fp8 requests dense-quantize instead (covered by existing prequant tests). * Resolve pre-quantized checkpoints per base variant One family entry covers several published variants whose weights differ (flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on (family, scheme) alone, so only the default base could ever be served: the loader's baked base_model_id validation correctly refused the schnell checkpoint for dev and Krea-dev bases and every such load paid the dense download plus on-the-fly quantise. Add an optional prequant_variant_repos table on DiffusionFamily as (base_repo, scheme, repo_id) triples and thread the resolved base repo through resolve_prequant_source / usable_prequant_source and their three call sites (load fast path, memory-plan probe, auto-policy candidate). A base without its own entry keeps returning the family default, preserving the existing refuse-then-dense behavior exactly. Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8 checkpoints (built in the earlier campaign but never reachable) and the new unsloth/FLUX.1-Krea-dev-FP8. * Add the Lumina Image 2.0 family to the image catalog Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the generic from_pretrained pipeline path loads it as a new lumina-2 family: - Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next checkpoints are a different arch and must stay unknown rather than crash mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image. - Trust the official repo for non-GGUF loads; bf16 component table entry (ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident). - Generation defaults 50 steps / guidance 4.0 per the model card, and the generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated, signature-gated): the pipeline default (1.0) runs the CFG double-forward on every step and oversaturates output. - Catalog group with the single ungated bf16 pipeline artifact (11 GB resident) plus routing assertions; images page defaults row. - No GGUF artifact: none exists upstream (only finetune/LLM quants), so the dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now. Offline probes of the future prequant campaign: int8 and fp8 both engage and render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory drift with intact quality), so neither scheme is family-denied. * Wire the hosted Lumina Image 2.0 int8/fp8 checkpoints Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 / SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing checkpoint repo conventions. * Add the HunyuanImage 2.1 family to the image backend The hunyuanvideo-community diffusers mirror carries the full stack in standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers 0.39 loads natively, so the generic from_pretrained pipeline path covers everything with no per-component assembly. Family notes: - The call's guidance knob is distilled_guidance_scale (there is no guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG runs inside the repo's guider at its baked scale. Defaults follow the card recipe: 50 steps, 3.25. - 2K-native: verified live at both 1024 and 2048. - Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no diffusers pipeline and stays excluded with its stated reason). - int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed bf16); a short prompt does not trip the int8 torch._int_mm minimum on this arch, so no family exclude entry is needed. - bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB. * Surface HunyuanImage 2.1 in the image model catalog Catalog group with the open bf16 mirror pipeline (~50 GB resident, so a bare click on a consumer card routes to the QuantStack GGUF quants, which load and render through the generic GGUF path, verified live) plus the images page defaults (50 steps, guidance 3.25 feeding distilled_guidance_scale). * Add the HiDream-I1 family to the image backend A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open Full / Dev / Fast repos (same arch); per-variant generation defaults follow the upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28 and Fast 16 guidance-free). The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not ship its weights; the official example passes the gated meta-llama repo in by hand. The loader instead assembles the component from the open unsloth mirror (byte-identical weights, already inside the non-GGUF trust gate), injected at the three pipeline from_pretrained sites, with output_hidden_states matching the official example. Memory planning counts the assembled TE4: 34.2 GB DiT + 28.8 GB encoders, ~63 GB bf16-resident. * Surface HiDream I1 in the image model catalog One catalog group with the three official bf16 pipelines (Full, plus the Dev and Fast distillations as labeled artifacts) at their ~63 GB resident size, so auto-routing keeps this a datacenter-GPU pick. city96's GGUF is deliberately not wired: the GGUF path would need the same Llama TE4 assembly for very small demand. Images-page defaults mirror the backend table with the variant keys ahead of the generic hidream key. * Pin the measured HiDream quant verdict in tests int8 and fp8 both engage and render cleanly on this family, including short prompts on int8: the routed MoE expert Linears only ever see the concatenated image+text stream (M far above the torch._int_mm minimum), so no deny entry and no family exclude tokens are warranted. Assert that so a future table edit cannot silently regress the measured behavior. * Wire the hosted HunyuanImage 2.1 int8/fp8 checkpoints Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors (456 quantized) dequantize equal between the loaded checkpoint and a fresh quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this family's own run-to-run nondeterminism (identical weights and seed reproduce a 17/255 mean pixel delta through the 50-step guider pipeline); per-case hard checks pass and the drift is compositional, reviewed visually. Uploaded to unsloth/HunyuanImage-2.1-FP8. * Fix silent LoRA drop and wasted transformer prefetch on GGUF quant loads Two live-test findings on the images load path: - transformer_quant with baked LoRAs, when the dense quantized build is declined for memory or fails: the load completed as a plain GGUF with the adapters silently dropped (HTTP success, supports_lora=false after the fact) -- wrong output with no signal. The load now fails with the recovery options (drop the adapters, free VRAM, or pick a smaller model). Weight-0 adapters still count as no bake request, and the plain no-LoRA decline keeps its silent GGUF fallback. - A fresh GGUF load on a small GPU prefetched the base repo's full bf16 transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch widening only checked scheme viability, not whether the device could ever hold the candidate resident. Gate the widening on total device capacity (reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is certain to decline the dense build never pays the download; capable devices keep the prefetch. * Fix video progress under-reporting during load and generate Two live-test findings on the video progress endpoints: - load-progress downloaded_bytes froze mid-download: the counter used scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the last completed blob for the whole multi-GB shard pull while the disk kept filling. Count the repo's cache directory directly (completed plus incomplete blobs, snapshot symlinks skipped so nothing is double-counted). - generate-progress reported total_steps=null / fraction=0 while step advanced: the video API only carried the native total field while the image API exposes total_steps and fraction, so one poller could not work against both. Derive the image-compatible aliases in generate_progress and declare them on the response model; the native total stays for back-compat. * Wire the hosted HiDream I1 int8/fp8 checkpoints Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291 int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means 0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0). Uploaded to unsloth/HiDream-I1-Full-FP8. * Add a pre-cast text-encoder loader for the layerwise fp8 scheme The runtime text_encoder_quant=fp8 path downloads the full bf16 text encoder and layerwise-casts it in place on every fresh load. For the heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load time on a fresh machine. diffusion_te_prequant.py loads a pre-cast fp8-storage state dict instead: meta-init the encoder skeleton from the checkpoint's te_class, load_state_dict(assign=True), rebuild on CPU if non-persistent buffers stay on meta, then re-apply the same layerwise cast to install the upcast hooks. The cast is a deterministic storage transform, so the loaded encoder is bit-identical to dense-load-then-cast by construction. v1 hosts the layerwise fp8 storage scheme only: its state dict is plain tensors (torch.load(weights_only=True), no pickle execution). The dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao subclass wrappers at runtime and are deliberately not hosted. Checkpoints validate format, scheme, component and base_model_id before use and any problem falls back to the dense download and cast. Local path overrides reuse the DiT prequant allowlist env var. Families opt in via a new te_prequant_repos (scheme, component, repo_id) field on both DiffusionFamily and VideoFamily; the field defaults empty so nothing changes until a gate-validated artifact is wired. * Inject hosted pre-cast text encoders during pipeline assembly Wire te_prequant_pipe_kwargs into the three pipeline assembly sites: the diffusion full-pipeline branch, the diffusion transformer-only and GGUF branch (where the companion TE is the big remaining download), and the shared video assembly path before the pipeline/component split. Injection is gated exactly like the runtime cast (mode normalized to fp8, device supported, family not denied), so it can never engage where quantize_text_encoders would not; the later quantize_text_encoders call re-applies the cast idempotently and keeps status reporting truthful. With no hosted checkpoint configured the call returns {} and assembly loads the dense encoder as before. * Add the pre-cast text-encoder checkpoint builder Applies the runtime layerwise fp8 storage cast to a model's dense text encoder once and saves the cast state dict with baked metadata (format tag, base_model_id, family, scheme, component, te_class, versions) in the layout diffusion_te_prequant.py validates. Resolves the encoder class from the checkpoint's config.architectures so the recorded te_class matches what the pipeline instantiates. CPU-runnable: the cast touches storage dtypes only. * Test the pre-cast text-encoder load path Hermetic CPU coverage for diffusion_te_prequant: the checkpoint filename convention, family-table resolution by scheme and component with malformed entries skipped, resolution priority (path override, hosted repo, none) and the fp8-only scheme gate, the checkpoint validation matrix (wrong format, missing state_dict, wrong scheme, wrong component, wrong or missing base_model_id) with base case folding, the local-path allowlist refusal and missing-file fallback, and the assembly injection gating (mode, hosted entry, device support, family deny, load failure, successful injection). Also pins the te_prequant_repos field on both family dataclasses and that no family ships a hosted TE checkpoint until the campaign wires one. * Fix pre-cast TE checkpoint loading and engagement reporting Two bugs found while building the hosted checkpoints: - The builder recorded torch.__version__ (a TorchVersion object) in the checkpoint metadata, so torch.load(weights_only=True) rejected every artifact and the loader silently fell back to the dense download. Record plain strings. - Re-applying the layerwise fp8 cast to an injected pre-cast encoder raised on the duplicate hook registration, making quantize_text_encoders report the engaged cast as failed (status showed no TE quant while the encoder ran fp8). _cast_fp8 now returns early when the hooks are already installed. Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B. * Wire the hosted pre-cast fp8 text encoders qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a hosted pre-cast fp8 text encoder from their unsloth -FP8 repos: - unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB - unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB - unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB Every checkpoint verified bit-identical to dense-load-then-cast (729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the real backends with the repo engagement marker. Tests cover the wired entries, the resolver filenames, builder metadata weights_only survival, and the idempotent re-cast. * Report the compute dtype on fp8-cast encoders and inject the pre-cast TE on the dense fast path Two more findings from the hosted-TE GPU smokes: - Module.dtype reports the first floating parameter, which after the layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt embed and latent dtypes from encoder.dtype and feeds them to randn_tensor, which has no fp8 kernel, so ANY flux.2 load with text_encoder_quant=fp8 crashed at generation (pre-existing, runtime cast included). The cast now swaps in a subclass whose dtype property reports the compute dtype; forward behaviour is unchanged. - The dense transformer_quant fast path assembles companions through _assemble_pipe, which never received the pre-cast TE injection, so the hosted encoder engaged on full-pipeline and GGUF builds but not on the fast path. Threaded through like the other two branches. Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE, render non-black, and report text_encoder_quant=fp8 truthfully. * Key the fp8 cast idempotency on an explicit completion marker Hook presence alone cannot distinguish a legitimately pre-cast text encoder from leftover hooks after a cast that failed mid-pass, so the early return now requires the completion marker _cast_fp8 sets once the hooks are fully installed. Leftover partial state keeps failing closed. Also tolerates non-Module encoder doubles in the hook probe and the dtype override. * Extend the fp8 TE quant to HiDream's Llama text_encoder_4 The generic quantize_text_encoders pass only covers text_encoder.._3, so HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path now lives there: when the requested TE quant is layerwise fp8 and the device/family qualify, TE4 prefers the hosted pre-cast checkpoint (unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then- cast; a mid-pass cast failure reloads a fresh dense encoder instead of shipping partial state. The pre-cast loader and builder gain config_subfolder/config_overrides for standalone encoder repos whose config sits at the root and whose pipeline needs forward flags (output_hidden_states/attentions). Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches), hosted checkpoint engages through the real backend (marker + status fp8), load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs the dense-TE render (gate 0.25), non-black frames. * Correct the ltx-2 resident TE estimate to the bf16 cast size The memory plan's bf16_components_gb held 50.4 GB for the LTX text encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download), not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4 GB resident. The 26 GB over-estimate pushed the auto plan toward offload on cards that fit the real footprint. Comments and the size-table test now pin the resident semantics. * Host pre-cast fp8 text encoders for four more families Round 2 of the hosted TE set, each bit-identical to dense-load-then-cast and gated through the real backend (marker + status fp8 + same-seed LPIPS vs dense TEs): - FLUX.1 T5-XXL (text_encoder_2): 9.52 -> 5.90 GB, one artifact for schnell/dev/Krea-dev (T5 shards byte-identical across all three, verified sha256). 220 tensors, 144 fp8, LPIPS 0.109. - Lumina Gemma2-2B: fp32 hub store 10.46 -> 3.20 GB (3.3x download cut). 288 tensors, 182 fp8, LPIPS 0.041. - Z-Image Qwen3-4B: 8.04 -> 4.41 GB. 399 tensors, 252 fp8, LPIPS 0.112. NOT shared with flux.2-klein-4B: klein retrained layer 35's MLP (verified tensor diff, maxdiff 0.86), so klein hosts no entry. - Krea-2 Qwen3-VL-4B: 8.88 -> 4.83 GB. 713 tensors, 460 fp8, LPIPS 0.082. The constructor-assembled krea pipeline takes the encoder directly (load_krea2_pipeline text_encoder kwarg); the loader remaps 5.x rope_parameters and re-ties weights after assign so the rebuilt encoder matches the builder's structure. HunyuanImage 2.1 reuses the Qwen-Image artifact outright: its Qwen2.5-VL text encoder is byte-identical (every shard sha256, 16,584,414,544 bytes), recorded in the new component-level base-equivalence table the checkpoint validator consults. The injection loop now covers text_encoder.._3 so a family can host several components. Live check: LPIPS 0.123 vs dense. * Report the fp8-cast compute dtype without swapping the encoder class The dtype override swapped encoder.__class__ to a dynamic subclass, which breaks transformers' kwargs-based output recording: a fp8-cast Qwen3VLModel stopped returning hidden_states and every krea-2 generation with text_encoder_quant=fp8 crashed at encode_prompt (regression from the HiDream TE4 change; caught by the krea hosted-TE live smoke). The override is now a property shadowed on the ORIGINAL class that prefers a per-instance compute-dtype attribute, so class identity is preserved and uncast instances keep the stock behaviour. The idempotency test now pins exact class identity and the uncast-sibling fallback. * Pass the calibrated distilled sigma curve to LTX-2.3 8-step runs The 22B distilled DiT was trained against ltx_core's fixed DISTILLED_SIGMA_VALUES, but the diffusers scheduler derives 8-step spacing from resolution-shifted flow matching and lands far off at every reachable mu (second sigma 0.945-0.981 vs 0.99375, tail 0.37-0.61 -> 0.1 vs 0.725 -> 0.42 -> 0). At the distilled default step count the backend now passes the list verbatim, neutralising the scheduler's dynamic shift and terminal stretch for the call (they distort even explicit sigmas) and restoring them afterwards. Other step counts and the dev/base DiT keep the scheduler's own spacing. Live-verified on B200 through the video branch backend: the scheduler holds the exact curve after an 8-step distilled GGUF generation, config restored, healthy clip. Also reword the transformer_quant resolved reason to the measured reality: quant halves resident weights and hosted checkpoints cut load time, while per-step speed is roughly bf16 parity. * Pin the fp8 weight-quantize kernel against silent MSLK switching torchao's Float8Tensor KernelPreference defaults to AUTO, which switches the weight-quantize kernel to MSLK whenever an mslk package is importable on sm90+. Measured on B200: that changes fp8 scale rounding bitwise (8/8 FLUX matrices differ, scales ~55 percent of bytes), so a box that merely gains mslk would break the hosted-prequant bit-identity invariant; the mslk path is also slower under torch.compile (opaque extern call blocks inductor's quantize fusion, FLUX.1 fp8 e2e 1.149 to 1.624 s). Pin KernelPreference.TORCH explicitly, matching current no-mslk behaviour bit for bit; signature-gated for older torchao. GPU-smoked (finite, rel err 0.037) and pinned by test. * Shift Qwen-Image training sigmas to the inference distribution Qwen-Image's scheduler skips its static shift under use_dynamic_shifting, so the DiT trainer was drawing UNSHIFTED uniform-schedule sigmas for it (mean sigma 0.50) while inference always runs the exponential mu = log 3 shift plus the shift_terminal 0.02 stretch. Add a flow_shift config lever: "auto" (the new qwen-image default) rebuilds the training sigma table through the scheduler's own time_shift and stretch_shift_to_terminal so the draw matches the inference distribution exactly (mean sigma 0.72); a numeric value applies the standard linear shift s*u/(1+(s-1)*u); 1.0 keeps the historical identity behavior and stays the default for FLUX, Z-Image and Krea 2. The model timestep conditioning follows the shifted sigma, gathered in fp32 so bf16 rounding never skews it. Also wire two opt-in levers with off defaults: cfg_dropout (per-sample empty-prompt conditioning dropout, encoded alongside the captions before the text encoders are freed) and weighting_scheme="bell" (bsmntw-style mid-schedule Gaussian loss weighting normalized to mean 1). Verified with two 80-step rank-8 bf16 LoRA runs on Qwen/Qwen-Image (identity vs auto, same seed): both converge with finite decreasing loss and produce coherent same-seed previews. Unit tests cover the exact transform, the shifted sampling distribution, per-family defaults and config plumbing. * Add LoRA EMA, a persistent conditioning cache, and aspect bucketing helpers diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only the trainable adapter params (warmup-ramped decay, default 0.99, exported as a second adapter under output_dir/ema), PersistentConditioningCache stores latent posterior stats and caption embeddings as safetensors keyed by content hash + family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect datasets into same-area divisor-snapped shapes. The DiT trainer wires the first two behind config flags that default to the current behavior: ema_decay (0 disables) and cond_cache_dir (None disables). A fully warm cache skips loading the VAE and text encoders entirely; a cache hit is bit-identical to a fresh encode, including the per-channel qwen latent normalization. Also fixes the stale _gather_sigmas call in the perf test that still passed the scheduler instead of the sigma table. * Tighten torchao configs and note the FSDP2 design for the DiT trainer nf4 loads now enable double quantization (~0.4 bits/param off the frozen base scales at no fidelity cost), fp8 training uses the rowwise recipe when the torchao build ships it (per-row scaling confines the DiT activation outliers that a tensor-wide scale collapses), and the inference quant filter gains a per-scheme GEMM-tiling divisibility floor (16 for scaled_mm, 32 for MX blocks) so one ragged Linear cannot crash the first denoise after a clean quantize pass. plans/fsdp2_diffusion_design.md records the multi-GPU design: bf16/fp8 over FSDP2 with per-block units, LoRA attached before sharding, int8 out of scope (DTensor over the quantized subclass is undefined), per-family notes. * Batch diffusion inference with per-image seeds, an inference conditioning cache, and GGUF loader fixes Batched generation: /images/generate takes a prompts list (one image per prompt, txt2img only) or a seeds list (one prompt, one image per seed); the legacy batch_size path derives per-image seeds base..base+n-1 like the native engine. Every image gets its own torch.Generator so any batch member replays alone from its gallery recipe; the whole list runs as one forward by default with OOM backoff that halves a failed chunk, and an explicit batch_size caps images per forward. Validated 10-22x over serial engines on 32-image suites with LPIPS deltas within 0.002. Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR (the inference sibling of the trainers' cond_cache_dir, same persistent store) wraps encode_prompt so repeated prompts skip the text-encoder forward entirely; verified bit-identical outputs. Bypassed while LoRA adapters are attached; tensor-argument calls pass through uncached. Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a different compiled graph than the dense family) and batched calls register every distinct (w, h, batch) chunk shape they ran, so the heavy GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B CFG-batched) are paid once ever. GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in the single-file converter; diffusers' FLUX.2 converter KeyErrors on it and the Qwen-Image identity mapping strands the model on meta. * Correct batched seed-replay docs to match measured behavior Same-seed images at the same batch shape are bit-identical; a solo regeneration with the recorded seed matches its batched rendition up to batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255, LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed bit-identity across batch shapes. * Note that batched bit-identity assumes a settled compiled graph The first generation issued while the deferred compile is still in flight can deviate transiently (observed once on a cold fp8 build: mean abs pixel delta 0.063/255); once the graph is settled, same-seed same-batch-shape images are bit-identical across runs. * Studio sidebar: Image03/FlimSlate icons, More flyout, Train row, New pills - Images uses Image03Icon and Video uses FlimSlateIcon. - New "More" row (MoreHorizontalIcon) opens a right-side flyout on click or hover holding Video, Recipes and Export; the close is delayed 180ms so the pointer can cross the gap. Its SidebarMenuButton deliberately takes `title` rather than `tooltip`: with `tooltip` the button returns a Tooltip root and DropdownMenuTrigger asChild would hand its ref to a non-DOM node. - Dropped the "Train" section heading; Train is now a top-level row between Images and More. data-tour="navbar" moves to the surviving nav group so the product tour keeps its anchor. - "New" pill beside Images and (inside the flyout) Video, via NavBadge. * Studio sidebar: match flyout rows and New pills to the existing scales - More flyout rows dropped their sidebar-row typography and size-icon override, which fought DropdownMenuItem's own scale (text-sm, gap-2.5, px-3 py-2 and size-4 icons) and rendered oversized glyphs and text next to the nav. - New pill reuses the brand "beta" badge recipe (nav-badge font, --ui-font-scale sizing, nav token colours) rather than hardcoded 9px values. - The More row's native title tooltip (an OS box on hover) is replaced by the app's Tooltip, wrapped around DropdownMenuTrigger so both triggers compose onto the same button, and shown only on the collapsed rail like other nav rows. * Settings: pin and reorder the sidebar navigation Adds a "Sidebar navigation" section to Settings -> Appearance, above the existing profile-menu customizer, with the same drag-to-reorder + switch UI. - New sidebarNav preference: one { id, pinned } entry per navigable row (projects, hub, images, train, video, recipes, export), array order = render order. Defaults match the shipped layout, so an untouched install is unchanged. - Unpinning moves a row into the More flyout rather than hiding it, so no page becomes unreachable. New chat and Search stay fixed as actions. - app-sidebar now renders from one navRows descriptor map, so a pinned row and its flyout counterpart cannot drift; the More row appears only when something is unpinned and highlights off whatever it actually holds. - Mirrored in the backend PersonalizationCustomization: without it the model's extra="ignore" would drop the field, and because sync replaces local state with the server's copy once customization is saved, the user's pin order would reset on the next sync. The validator dedupes and back-fills like sidebarMenu but preserves the client's order, since here order is meaningful. Frontend typecheck, i18n parity and catalog checks pass; 32 personalization tests pass, including a round-trip asserting a reordered list survives a save. * Sidebar customizer: drop the Search row, skip More for a lone item - Search is reached from the top bar, so it is no longer previewed as a fixed sidebar nav row; New chat stays. - More now appears only when it would hold two or more rows. A single unpinned row renders inline in its saved order position instead: a flyout wrapping one item costs a click and earns nothing. The customizer's More preview follows the same threshold. * Sidebar settings: hide a lone unpinned tab, match New chat icon, rename Profile menu - With exactly one tab unpinned, both More and that tab are dropped, so nothing is drawn for it (previously it rendered inline). The page stays reachable by URL. - The customizer's New chat preview uses PencilEdit02Icon, the icon the real row renders; Edit03Icon was a different glyph. - "Sidebar menu" is now "Profile menu", described as the shortcuts behind your name at the bottom of the sidebar, so it no longer reads as a second name for the navigation section above it. * Tighten comments in the new sidebar and delete-guard code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio sidebar: keep the More row highlighted while its panel is open Moving the pointer into the flyout left the row unhighlighted while the panel stayed open. The row now carries data-menu-open, added to the nav hover selector list. Not data-state: the tooltip and menu triggers both write that attribute, so whichever lands last wins. * Images: use the shared pill toggle for Create/Train and pad the panels - Create/Train was the only segmented control on its own Tabs styling. It now uses PillTabs, the same control as the model picker and Hub toggles, pinned to the header row's 34px. PillTabs takes an icon per tab, so the inline-span workaround for TabsTrigger goes away. - pt-3 on both the Create and Train panels, which sat flush against the model selector row. * Images: make the workflow picker a dropdown instead of a 7-up strip Seven workflows in a 340px rail left ~48px each, so the labels crowded and the hints were only reachable as title tooltips. The strip is now a dropdown: the trigger shows the current workflow and its hint, and each row carries its own description. A row the loaded model can't run is disabled and shows the reason in place of the hint, so the gating explains itself. Adding a workflow no longer shrinks the others. * Images: workflow icons, hint under the trigger, more top room, unclipped Train cards - Each workflow carries an icon, shown on the closed trigger and on every row. - The trigger is one line (icon plus name). The selected workflow's description moved below it, where it reads like the Field hints further down the rail. - pt-6 instead of pt-3 on both Create and Train, so the cards clear the model selector row. - The Train right column scrolls while its cards use ring-1, which draws outside the box and was clipped at the scroll edges. p-px gives the ring room. * Images: one-line workflow rows, stronger trigger fill, roomier mode tabs - Dropdown rows are icon plus name only. The selected row's description already shows under the trigger, and a disabled row keeps its reason as a title. - Trigger fill moves to the bg-foreground/[0.07] dark:bg-foreground/[0.12] pair the hub cards use, so it reads against the card in both modes. - Description under the trigger goes from text-ui-10 to text-ui-11p5. - More horizontal padding on the Create / Train tabs. * Images: drop card borders for the composer shadow, keep scrollbars inside, use app controls in Train - Cards lose ring-1 for .panel-soft-surface: the composer's shadow in light, flat in dark, matching .chat-composer-surface and the menus. - Both rails now clip (overflow-hidden) with the scroller inside, so the scrollbar can't ride over the rounded corner. Same shape video-page already uses. - Train's 9 native selects become the app Select, so they no longer open an OS-native menu, and the native file input is hidden behind a Choose images button that reports the count. - Image previews use explicit 8-10px radii: this theme sets --radius to 1.1rem, so rounded-md was 15.6px and the thumbnails read as circles. * Images: one card for controls and preview, chat sliders, wider softer shadow - Controls and preview were two floating cards; they now share one card split by a divider. The Advanced dock stays separate since it toggles. - SliderField wraps Chat's ParamSlider, so the sliders match Chat (label row with the value, full-width neutral track) instead of a green track with a spin box. All 14 call sites keep their props. - panel-soft-surface goes from 0 2px 8px -2px /0.16 to 0 4px 22px -6px /0.10: lighter, spread wider. * Images: flat Create and Train panes, hover-only scrollbars, tidier Train dataset step Both Images tabs now sit on the page background like the Hub: no card, no shadow, no bounding box. A single rule divides the controls rail from the preview canvas (Create) and from the run area (Train), and the settings and previous-runs sections read as panes rather than nested cards. Also: - Scrollbars in these panes use the existing hover-scrollbar recipe, so the thumb only shows while the pane is hovered. - Workflow rows explain themselves with a tooltip after a short hover, which also works on disabled rows, and the descriptions are much shorter. - Training images rows are name plus image count; the license stays on the example card. - The upload step loses its dashed box, the buttons match the sizes around them, and Upload only appears once files are picked. - The empty preview uses the same icon as the Images nav item. * Images: full-height panes, wider settings rail, Create/Train offset from the selector The rule between the panes now runs the whole page height (the row drops its bottom padding and each pane pads its own content), the settings rail is wider on both Create and Train, and the Create/Train switch sits further right of the model selector. * Images: put both tabs on the Hub's centered measure Top bar and content now share mx-auto max-w-1100 with px-5 / sm:px-8, so Create and Train sit at the same width and position as the Hub instead of running edge to edge. * Images: restore the top bar position, drop the panes lower under it * Images: center the mode switch, flip the arrow with the orientation, app tooltips everywhere The Create/Train switch is centered on the page instead of trailing the model selector, with wider buttons. The flip control's arrows now rotate with the orientation and its label says which way the flip goes. Every native title tooltip on the page is now the app's tooltip, so they all get the rounded surface instead of the OS box. * Images Train: plainer field text, no green buttons, columns that stop colliding - The dataset name, trigger prompt, adapter name and custom base fields now say what they are in plain words instead of leaning on example values. - Import, Upload, Back, Back to settings and Train another are outline buttons, not green ones. - Example thumbnails are landscape tiles, so photos are not cropped to chunky squares. - Settings cells get min-w-0 and the select value truncates, so a long option like the nf4 label no longer widens its column into the next one. - The number stepper sits a little further in from the field edge. - Create and Train are wider. * Images Train: roomier example cards with Import on the thumbnail row * Video: same treatment as the Images tabs - No cards: the rail and the canvas sit on the page background, divided by a rule that runs the full page height, on the Hub's centered measure. - Wider rail, chat's sliders, hover-only scrollbars. - Every native title tooltip is now the app's tooltip, including the clip cards. - Reapply and Cancel are outline buttons, the empty state uses the Video nav icon, and the clip tiles are less rounded. * Images and Video: narrower generation rail, matching Train headings Create and Video rails go from 392px to 368px. Train a LoRA and Training settings are now the same size and both in the heading font: the h2 already picks it up from the base rule, so the settings header opts in with font-heading and the weight that rule pins. * Images Train: shorter copy throughout Family notes, example descriptions, precision labels and every helper line are trimmed so they stop wrapping to three lines and colliding with the next column. The nf4 label now fits its select without truncating. * Images Train: a little more spacing between field groups * Images and Video: tighten code comments * Fix training start NameError, the load-order guard test and CPU-only diffusion tests - start_training forwards resume_source_run_id to _start_training_impl, which reads it. Without it every start raised NameError. - Restore main's anchor in the load-marker order test: the file now has an earlier `if config.is_gguf:`, so indexing the first one compared the wrong branch. - The two diffusion tests that reach diffusers now skip when it is absent, matching the CPU repo-test env. - The UI smoke finds nav rows that live in the sidebar's More flyout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a null metadata caption as no caption str(None) stored the literal "None" as the caption, so a null row counted as captioned and would have trained on that text. Also drops an unused import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix invalid-UTF-8 500s, the flat Canny map and the dropped DiT knobs read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption sidecar or video sidecar 500d the info, upload and gallery routes. A flat image now yields the all-black edge map instead of its own luminance, and the four DiT loss knobs the trainer implements are declared so model_dump keeps them. * Use the ui font-size tokens instead of raw px text utilities text-[11px] and friends ignore the UI font size preference, which the repo's font-scale contract test enforces. Same rendered size at the default scale. * Fix diffusion dataset 500s, the dropout-1.0 no-op run and the reset base pick Four correctness fixes on the training side: - The labeling grid read caption sidecars under except OSError, but a non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad file 500d /diffusion/dataset/{name}/images and the grid could not be opened to repair it. Read it as no caption, matching the info summary. - An image past Pillow's own hard limit raises DecompressionBombError, which derives straight from Exception and so escaped the upload guard's (OSError, UnidentifiedImageError, ValueError) and returned 500 instead of the intended 400. - lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0): lora_A and lora_B receive no gradient and the run saves an untrained adapter while reporting normal progress. Bound it below 1.0, matching the LLM request schema. - The train panel re-seeded the base repo on every dataset refresh because the family object identity changes on each info fetch, so an upload or caption save silently replaced the user's chosen base and the run started on a different model. Track the pick and only re-seed on a real family change. * Show the retained failure when a video page mounts after a failed job Mount-time recovery handled only phase=completed, so reloading the page after a multi-minute generation failed left an idle view with no diagnosis: the backend keeps the terminal failed record only until the next job, and nothing else survives the reload. Surface it the same way the poll does, filtering the cancelled sentinel. * Fix batched generation crashes, cache keying and unreplayable recipes Four bugs in the batched inference path, all found by review: - A mixed-prompt batch sent a scalar negative prompt against a prompt list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX true-CFG encode a batch-1 negative against batch-N latents and fail in the transformer's text/image concat. Broadcast it to match the batch. - The FBCache step-cache reset sat above the chunk loop. diffusers only resets that state at the end of a successful call, so a forward that raised (the OOM the backoff is meant to recover) left its own residual behind and the halved retry died on a shape mismatch. Reset before every forward instead. - The conditioning cache keyed on the checkpoint alone, but a GGUF or single-file load takes its text encoders from the companion base, so the same checkpoint against a different base reused the previous base's embeddings. Key the base too. - Gallery records stored the base seed and the requested batch size even when a prompts/seeds list drove the run, so restoring the second image of seeds=[5, 99] replayed seed 5. List-driven outputs now record as single-image recipes on their own seed. Also bound strength above 0: every img2img pipeline derives its step count from it, so 0 leaves zero denoising steps and either raises or, on SDXL, crashes on empty latents. * Fix quantized-load LoRA bake, prequant family exclusions and outpaint canvas Six review findings across the Images page and model scanning: - The quantized (int8/fp8) load path can only attach LoRA adapters before quantization, but the frontend load request had no loras field, so every generation after such a load was rejected and each reload repeated it. Send the selection with the load. - build_prequant_checkpoint passed no family to the scheme exclusions while recording the family in metadata, so a Qwen int8 artifact baked the short-M text-stream linears and was then rejected wholesale by the loader's family-keyed check. - Registering a bare single-file checkpoint directory produced no On Device row even though the images loader can load it; only its parent worked. Admit that shape when nothing else matched. - Unload left the Reapply target set, so the repair path was skipped and Reapply reloaded the ejected model. Clear it, as the video page does. - Both FLUX.2 bases were trusted for training but not inference, so Deploy to Create rejected every FLUX.2 adapter. - Outpaint allocated the grown canvas before downscaling, exceeding the browser canvas area cap on a large photo; an over-cap canvas is unusable, so Extend silently posted a fully transparent image and mask. Scale the source first. * Send the picked GGUF filename with the quant so diffusion loads fire The variant expander emitted only the quant label, and nothing else in the frontend set ggufFilename, so the Images and Video pages could never take their GGUF branch: both gate it on meta.ggufVariant and meta.ggufFilename, then fall through to the single-file path, which returns because the id is a repo id and not a .gguf name. Every quant pick was a silent dead click, with no load request reaching the backend. The filename was already on the variant row (the picker keys its list on it, and the variant validator requires a non-empty string), so thread it through the click handler. The chat path is unaffected: it reads ggufVariant and never needed the filename. * Version the conditioning cache key and reject non-finite flow_shift Two correctness fixes: - The cache keyed the checkpoint and its companion base by name only, so a Hub repo advancing to a new commit, or a local directory updated in place, kept returning embeddings from the previous text encoder. Pair both with a revision marker: the locally resolved commit sha for a Hub repo, config plus text-encoder file stats for a directory. Neither loads the encoders, so a warm run still keeps them off the GPU. - flow_shift only checked positivity, but JSON accepts 1e309, which floats to inf, and inf <= 0 is False while NaN fails every comparison. The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which poisons every sampled sigma and saves a corrupted adapter while progress looks normal. Require a finite value. * Keep curated models listed, guard the video companion repo, pin diffusers Three review findings: - The picker filtered every catalog member out of Recommended and Hub search on the way to canonical group rows, but nothing renders those rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and unused). A task-scoped picker's models list is catalogToModelOptions(), i.e. group members exclusively, so both lists came back empty and no curated model could be discovered or downloaded. Keep the artifacts listed until the grouped UI exists. - The video delete guard compared only repo_id, so deleting the companion base of a loaded GGUF video model was allowed even though it supplies the VAE and text encoders. Compare base_repo too, matching what the images guard already does for its companions. - diffusers was declared unversioned while the diffusion stack requires 0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and Z-Image pipelines), so an upgrade could keep an older release and selecting an advertised model failed until the user upgraded by hand. * Namespace the trainer conditioning cache per checkpoint, bound the learning rate - The trainer keyed its persistent conditioning cache on family and resolution only, while the keys themselves carry just the caption or image content and crop variant. One cache directory reused for two checkpoints, or for the same repo at a new revision, let a warm run skip loading its encoders and train on the other model's embeddings and latent statistics. Namespace on the base checkpoint and its resolved revision as well. The revision helper now lives beside the cache in diffusion_train_extras and the inference wrapper delegates to it, so the two cannot disagree about what counts as the same source. - The diffusion learning rate only checked positivity, but 1e309 floats to inf and satisfies gt, so the route evicted the resident models and started AdamW with an infinite rate: the first step destroys the adapter while progress looks normal and the result is saved. Bound it below 1.0, matching the LLM schema, which rejects inf for the same reason. * Fix GGUF image model picks doing nothing, and pick the train base in the top bar The quant rows never forwarded the .gguf filename, so every hub GGUF pick on Images/Video fell through to a silent return. On Train the top bar now picks the training base instead of a generation model, which is GGUF-only and untrainable. * Pin diffusion and video loads to the live HF cache root Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache folder does not update: progress counted the old root while the download wrote to the new one, and from_pretrained could split one model across both. * Add the diffusion download plan endpoint Reports the repos and exact files a pick needs so the download manager can stage them with the loader's own file scope. A plain snapshot would add the packaged root single, transformer shards and fp16 twins the loader never opens. * Add a file-scoped flavour to the Hub download job Lets a consumer that reads a deliberate subset of a repo stage it through the normal download manager. Keyed as "@scope" so it never collides with a quant or with the repo's full snapshot, and the file list rides the registry so an XET to HTTP retry respawns the same scoped job. * Stage image and video downloads through the Hub download manager They downloaded inline inside the load, so they had none of the manager's disk preflight, manifest verification, resume or panel progress. Picks now stage as scoped jobs carrying the loader's own file list, then load from a warm cache. * Fetch staged GGUF checkpoints as scoped jobs, and stop calling diffusion models unsupported A GGUF entry went out as a full snapshot, whose ignore list drops *.gguf: the job finished at once having fetched only docs, and the repo landed on device unloadable. Every entry is scoped now. The Hub also no longer tags image/video models as unsupported (they run on their own pages), and those pickers name what they select. * Apply the picker task filter to local model sections LM Studio, ./models and custom-folder rows ignored it, so the Images picker listed chat GGUFs that 400 on a diffusion load. The backend already tags every local model with a task for this purpose. * Route a chat pick of a diffusion model to the Images or Video page Chat cannot load one, so it was either hidden or failed on load. The unfiltered picker now lists on-device diffusion models and navigates to the page that runs them, passing the repo and quant so that page loads it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route the real GGUF filename, keep non-GGUF curated models, key scoped downloads by file set Five review findings, four of them ways a click did nothing or fetched the wrong thing: - A chat pick of a diffusion model routed ggufVariant (a label like Q4_K_M) in the search param the target page uses verbatim as the GGUF filename, so the load asked for a file that does not exist. Route ggufFilename; no filename means a curated non-GGUF pick, loaded as a pipeline. - The task-scoped pickers kept only GGUF repos, so the catalog's bf16, bnb-4bit and single-file fp8 artifacts could not be discovered or downloaded on the Images and Video pages even though loadSpecFor knows how to load them. Keep curated artifacts whatever their format, in Recommended and in Hub search. - Both pages deduplicated routed selections on the model alone, and they now stay mounted, so picking the same repo again -- another quant, or the same one after chat evicted it -- returned early without loading or clearing the query string. Key on model and quant. - Every scoped image download shared one @diffusion job key regardless of the requested files, so switching quant mid-download adopted the running job: the UI waited on the first file set, then loaded a file that was never fetched. Include a digest of the file set in the key. - A scoped plan silently dropped requested files missing from Hub metadata, and snapshot_download succeeds when an allow pattern matches nothing, so the job reported completion and triggered a load with required files absent. Fail the job instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the scoped download key derivable, and stop the hidden page hijacking a route Four review findings, the first a regression from my own last commit: - Keying scoped download jobs by a digest of the file set broke the download manager: it builds that key client-side (it polls and cancels before any response tells it a key), so it watched and cancelled a key no worker owned and never fired its ready callback. Keep the derivable "@scope" key and refuse the second request instead when a live job on the slot is fetching a different file set -- decided inside the registry claim, under the lock, so a concurrent claim cannot slip past it. The manager records the file set on the job as well, so a sibling quant's transfer is not adopted locally either. - Both diffusion pages read the route query through a loose useSearch and both stay mounted once visited, so the hidden one consumed the other's ?model=: it navigated back to its own route and tried to load, say, an image checkpoint as a video model. Only the visible page consumes it. - The staged download plan was built without the configured HF token or the Advanced values the load itself sends. The token matters most: the backend's Hub metadata lookup is best-effort, so a gated base silently planned no companion entry and the load pulled those multi-GB files inline, outside the manager. The memory/quant controls decide whether the base transformer/ shards are needed at all, and the route dropped memory_mode, cpu_offload, the prequant path and the LoRA selection before asking for the plan. - The video preview kept playing after leaving the page: the keep-alive layout only hides it, and display:none does not pause a media element, so a clip the user unmuted kept its audio going over the next page. Pause on the active transition and do not auto-replay while hidden. Also completes the hand-built request bodies in the hub download tests: the scoped-files field this branch added to the route read as an AttributeError against them, failing five tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the GGUF variant contract test against the merged handler signature The assertion pinned the exact single-line call handleVariantClick(v.quant, v.downloaded, expectedBytes, v.filename), but the handler takes (quant, filename, downloaded, sizeBytes) and prettier wraps the call across lines, so the mandatory repository test job failed on every push. Match the call structurally and assert the filename really is forwarded in the handler's argument order. * Stop a background page and a stale record taking the GPU or a download with them Five fixes from a review pass over the diffusion work. delete-finetuned rmtree'd a model the Images or Video engine was holding: every guard on that route is chat-only, and Images loads any local path, so deleting a local diffusion model under the storage root pulled the weights (and the companion VAE / text encoders sd.cpp re-reads each generation) out from under a live pipeline. The cached-model route already refuses this; the trained/exported one now does too, matching by path rather than repo id, and failing open on a chat-only install so it cannot block ordinary deletes. A staged download finishing while its page was hidden loaded the model and evicted whatever the user was actually using: both diffusion pages stay mounted behind the router and a load takes the GPU unconditionally. The pick is now held until its page is on screen again, which is also what chat does. A scoped download could report success having fetched nothing. With Hugging Face metadata unavailable no manifest is written, so verification is a no-op, and snapshot_download returns an existing snapshot folder without downloading when its own repo_info call fails. A repo already on disk from a full snapshot job (which ignores *.gguf) therefore completed with no weights and auto-loaded against them. The requested file list needs no network, so it is checked against the disk directly. The XET to HTTP retry reclaimed the job slot without the scoped file list, and that claim overwrites the stored record, so a later identical scoped start compared an empty list against the real one and 409'd instead of adopting the running download. The DiT accelerator gate probed torch.mps.is_available(), which only exists from torch 2.5 while the supported floor is 2.4. All three probes shared one try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only host still evicted the resident pipeline, downloaded the encoders and died in the child. Each accelerator is probed on its own now, through torch.backends.mps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage what an LTX-2.3 load reads, keep a routed file's load kind, drop an unbakeable LoRA Three from the latest review. The video download plan always asked for the wide base file list, so an LTX-2.3 pick staged the 2.0 base's VAEs, vocoder and connectors that the checkpoint supplies itself, while the companion files the 2.3 assembly does read were left out of the plan and pulled inline at load, outside the panel's progress, cancel and disk preflight. The plan now recognises a 2.3 pick by name (the load keeps the authoritative header probe, and under-guessing only falls back to the load-time pull), narrows the base list, and stages the extras in the same entry as the checkpoint so one repo stays one scoped job. A pick routed from the chat picker arrives as ?model= and ?quant= with no picker metadata, so a bare local .gguf or .safetensors was loaded as a pipeline: an explicit model_kind wins over the backend's filename sniffing, so it evicted the resident model and then failed on the missing model_index.json. Both pages now derive the load kind from the path, the same way their own picker handlers do. A torchao int8/fp8 build takes adapters only at load time. Switching artifact inside one family keeps the LoRA selection, since the family did not change, but the load did not bake it, so the next generation was rejected with 'reload the model with the adapter selection' while the picker still showed the adapter as active. The selection is now dropped once per resident build, with a message saying to pick and load again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cancel an evicted safetensors load, spare the arbiter for CPU-only chat, fetch clips lazily Four fixes from the latest review round: - The GPU arbiter's chat evictor only cancelled the llama.cpp side. The orchestrator publishes active_model_name once its worker reports success, so an in-flight safetensors load was visible only as an entry in loading_models and finished onto the GPU after ownership had transferred. Cancel every pending load, and give the safetensors branch the post-load ownership recheck the GGUF branch already had. - A manual gpu_layers=0 GGUF load runs on the CPU with the GPUs hidden from the child, yet it took the arbiter unconditionally: it cancelled a running image or video generation for a model needing no VRAM, then held CHAT ownership so the next GPU workload unloaded it for nothing. Gate the acquire on the same predicate the launch-time CPU-only mask uses, as the image and video loaders gate on their resolved device. - The staged-download hook subscribes per repo, not per job, so another job on the same repo advanced the staged queue (starting a load whose scoped files were still downloading) or wiped a queue that was still running. Compare the variant each callback carries, like the chat page's auto-load does. - The video gallery fetched every record of a page into an object URL that lives until the page closes: 50 clips at tens to hundreds of MB each, for cards the user may never scroll to. Fetch a clip as its card nears the strip's edge, plus the selected one the player needs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments across the diffusion backend Comment-only pass over the Python this PR touches: drop what the code already says, collapse multi-line explanations that still read on one line, and keep the reasoning that is not recoverable from the code. No code, docstring semantics or behaviour changes; verified with an AST comparison against the previous revision, and the backend suite is unchanged (same 37 environment failures as before: the API integration tests that need a live keyed server, the flash-attn install hooks, and the GPU memory fields). * Invalidate latents on a VAE swap, keep a cut-off generation, surface the EMA adapter - source_revision() scanned the checkpoint root plus text_encoder/tokenizer but not vae, so swapping or fine-tuning the VAE in place left the conditioning cache namespace unchanged and a warm run trained against latents from the old checkpoint. Include the vae directory, like any other component the cached tensors come from. - /images/generate answers only when the images are saved, and secure mode's tunnel caps an origin response near 100 seconds, which a native CPU or a high-step run passes routinely. The page reported failure while the work kept running, and a retry would duplicate it. A lost response (fetch rejection or a gateway status the origin never answered) is now told apart from a refusal: the page waits out generate-progress and reloads the gallery, so the run it started still lands. - The trainer emits the EMA adapter's path with the terminal event, but the state update dropped it, so neither the run history nor either response schema carried it and an enabled EMA left nothing discoverable. Keep it, and show it next to the primary adapter. - weighting_scheme advertised a choice of timestep sampling; sampling is always logit-normal and the flag only selects the bell loss weights. Describe what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide unloadable cached rows, hold the dataset interlock, bound a GIF export - The cached-model listing tagged any repo with a model_index.json as text-to-image, so a community pipeline the image loader's trust rule refuses still got a row in the Images picker, and a detected-but-untrusted video repo fell through to that same tag. Gate the image tag on the load path's rule and hide an untrusted video repo outright. - A routed diffusion pick only carries a GGUF filename, which is all the chat picker has, so a curated single-file artifact arrived with no quant and was loaded as a pipeline: from_pretrained on a repo with no model_index.json. Pass the page's own catalog spec into the route pick, so a routed pick resolves to exactly what a direct pick on that page resolves to. - The dataset mutation endpoints checked is_active() and only then handed their filesystem work to a thread, so a start reserving in that gap changed captions or removed images underneath the preflight or the running trainer. The interlock is now registered for the whole request under the lock reserve() uses, and a start refuses while a mutation is open rather than waiting on it. - GIF export held every kept frame as a paletted image before encoding; a clip may be 2048x2048 for 1024 frames, and at the 12 fps target the step is 1, so one export click could allocate over 4 GB and take the backend down. Downscale past 720 px and widen the step to keep at most 300 frames. - seed accepted any Python int, so an out-of-range one passed every preflight, evicted the resident models, spawned the trainer and only then died in torch.manual_seed. Bound it to torch's 64-bit range in the request and config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the accelerator probes in the DiT family-metadata tests Six tests read family_train_infos() (or a start preflight) without pinning the host probes, so they only held on a machine with a bf16 accelerator: on a GPU-less runner the DiT gate empties precision_modes, turns supports_compile off, and replaces any other preflight message with the no-accelerator note, and all six failed there. A conftest fixture pins both probes for exactly those tests, so they assert the family metadata they are about on every host. The gate's own CPU-only behaviour keeps its dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the pipeline task into the hub inventory the pickers read - The task-scoped pickers filter On Device rows on a task, and the chat picker routes a diffusion pick by the same field, but those rows come from the /api/hub inventory, which never carried one: the Images and Video pickers listed nothing on device and the routing never fired. Both cached scans and the local listing now tag rows with the classifiers the models API already uses, the schemas and the frontend adapter carry it through, and a row the backend classified as a generation task is exempt from the chat-only guard that was also dropping it. - The local routing map was keyed by model_id while the row click passes id (a filesystem load id for a models_dir or LM Studio entry), so the lookup missed and the pick fell through to the chat loader. Key both. - A staged download whose start answered "error" left its head in place, where the effect never re-runs and onReady never fires, so the pick was stranded until the user reselected. Clear the queue and say so. - Every scoped pick in a repo shares the @diffusion variant, so the variant alone cannot tell two file sets apart: restaging while the first job finished let its completion pass for the new pick and load a checkpoint that had not downloaded. Bind the callbacks to the repo + file set they started, and to the staging generation. - A rejected generate POST does not say whether it reached the backend, so an immediately idle progress read was ambiguous and a submission that never landed looked like a finished image. Require evidence: progress seen active, or a gallery record that was not there before the POST. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the OpenAI image URL fetchable, keep WebM audio, stream example imports Four review items on the diffusion Studio work: - response_format=url returned the bearer-gated gallery route, which a standard image client downloads with no Authorization header, so the default response format was unusable. Mint a short-lived HMAC link instead (the shape RAG already uses for pdf.js) served by a signed route, and leave the gallery route itself bearer-only. - A manual gpu_layers=0 load carrying speculative_type="off" -- a value the UI persists and sends -- read as GPU-bearing, so it took the GPU arbiter and evicted a resident image/video pipeline even though the launcher hides the GPUs for it. Canonicalize the mode and exempt "off". - The curated example import prepared the whole split before the loop stopped at the 10-100 image cap; m1guelpf/nouns is 49,859 rows / 328 MB. Stream instead, with the prepared load kept as a fallback for a repo that cannot stream. - WebM export dropped the audio track an LTX-2 clip carries, silently, on the format offered for web embeds. Mux it as Opus through a resampler + FIFO, and keep exporting the video alone on a build without libopus. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not stub out triton on a GPU host when the Xet backend fails to import The lazy loader retries `import unsloth_zoo.hf_xet_fallback` under UNSLOTH_ZOO_DISABLE_GPU_INIT=1 whenever the first attempt raises. That flag makes unsloth_zoo take its MLX/CPU path, which injects triton and bitsandbytes STUBS into sys.modules for the rest of the process. On a working GPU box whose first import failed for an unrelated reason (a bitsandbytes/CUDA mismatch, say) the retry succeeds, so Studio boots looking healthy and then dies at the first CUDA-only kernel: a GGUF or compiled diffusion generation hits the stub and returns NotImplementedError: Unsloth: 'triton.tools.experimental_descriptor.enable_in_pytorch' was called on Apple Silicon / MLX, where triton is stubbed out. so every image generation 500s with an Apple-Silicon message on a Linux CUDA host, while the load reports success. Found by loading Z-Image-Turbo GGUF through the API on a box where bitsandbytes could not initialise. Gate the retry on the host genuinely having no accelerator. The Xet stall watchdog is optional and already degrades with a warning; a process whose triton is stubbed out is not recoverable. The warning now says why it did not retry. * Fix the lost-generation proof set, the settle timeout and the hub inventory's diffusion gates Seven fixes from the latest review round on the Images page and the hub cache inventory. Images page: - The lost-POST settle path built its "already seen" gallery id set inside the catch, after the request failed. By then the earlier runs of the same batch had already prepended their records, so run 2 could accept run 1's image as proof that its own request reached the backend. The set is now captured once before the first POST and grows with every record the batch produces. - settleLostGeneration fell out of its SETTLE_MAX_MS loop and returned normally, so a wedged generation was counted as done and the next run started against a busy backend. It now throws on timeout. - Restoring a recipe cleared the ControlNet selection but left the workflow tab and the init / mask / reference images pointing at whatever was loaded, so the next Generate conditioned on an unrelated image. It now clears all of them and returns to Create. - The download plan omitted the adapter selection the load itself bakes in. A baked LoRA forces the dense build path, so the plan described a different file set than the load that followed and the rest was pulled inline, outside the download manager. Both now derive the list from one helper. Hub cache inventory: - A download for a repo an Images or Video load is staging was allowed to start: only the llama.cpp loader was consulted. Both diffusion backends already expose loading_repo_ids for the delete guard, and the download guard now reads them too. - A companion-only prefetch (pipeline manifest plus VAE and text encoder, no transformer) passed the snapshot-partial check, since every file its manifest expected did arrive, and was advertised as on-device although from_pretrained cannot load it. - The single-file flag never reached the picker through the hub inventory path, so a checkpoint-only diffusion repo read as a full pipeline and failed after the handoff. The two pipeline-shape helpers now live in hub/utils/inventory_scan.py so /api/models/cached and the hub inventory classify the same repos the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop adopting an unknown scoped download, leaking raced blobs and resurrecting deleted clips Three items from the latest review round. A scoped download job carries a deliberate file subset, and every file set of one repo rides the same "@scope" slot. A client that adopts a live job from the backend had no file list to compare against: the active-downloads response never carried one, so an adopted job's set was unknown and any later scoped request for the same repo read as "already started". Selecting a different checkpoint then waited on the wrong transfer and tried to load a file nobody fetched. The response now publishes the scoped file list, adoption records it, and an unknown set no longer satisfies a scoped request. A gallery record can be deleted while its blob is still downloading. The delete revokes the URL present at that moment, so the fetch that lands afterwards inserted a fresh object URL for a record no card renders and nothing can revoke: a full MP4, tens to hundreds of MB, pinned for the rest of the session, and once per raced fetch. Both galleries now discard a blob whose record went away, with an epoch covering the video page's Clear all. The video backend keeps the last completed job until the next one starts, and the Video page merges that record on mount to cover a job that finished after the gallery fetch. Deleting the clip left the record in place, so every reload prepended a ghost card whose file request 404s until another generation replaced it. Deleting the clip, or clearing the gallery, now clears the matching terminal record, and the page skips a record it deleted itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve revisions from the live cache, serialize dataset imports, and stop pinning every gallery blob Five more items from the review round. The conditioning-cache revision marker read huggingface_hub's import-time HF_HUB_CACHE constant. Studio can move its cache during a session and loading follows the live setting, so after a move the marker went unresolved (or pointed into the previous root) and pulling a new revision of the same checkpoint no longer invalidated the cache: a warm run could reuse the old encoder's embeddings and the old VAE's latents. It now looks in the active Studio cache first and keeps the environment and the library constant as fallbacks, which the trainer subprocess still needs. The dataset interlock counts mutations rather than excluding them, so two imports of different examples into the same empty name both got past the emptiness check. The winner promoted its staging directory atomically; the loser found the folder non-empty, fell back to a per-file move, and merged its images and captions into the winner's dataset. Imports now take a per-folder lock, a second one is refused with 409, and the emptiness check is repeated under the lock. On Windows the sd.cpp asset resolver filtered only by accelerator token, so a Windows arm64 host matched an x64 zip, downloaded and installed it, and failed later when the binary would not run. It now filters by architecture the way the Darwin and Linux branches do. Every gallery page fetched every PNG up front and kept the object URL for the session, so scrolling a large gallery grew memory without bound for tiles the user may never look at. The Images strip now fetches a tile as it nears view, like the Video strip, and keeps the eager path only where IntersectionObserver is unavailable. A 503 carrying a JSON body comes from the application, not a proxy, so it is surfaced as the error it is instead of entering lost-response settlement and being reported as a request that never reached the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the class of a failed generation instead of a bare "Image generation failed." Found on a macOS runner: the native renderer aborts inside its own text encoder there, and the page showed only "Image generation failed." with the sd-server backtrace left in the server log, so nothing about the failure reached the user. The failure is now classified into fixed text, out of memory and native-process death, so the message says what happened and what to try. None of the engine's own output is echoed, since a native tail carries local paths and argv; that stays in the log, and an unrecognised failure keeps the original literal. * Treat an undecodable caption sidecar as the tombstone the trainer sees Uploads store .txt and .caption sidecars as raw bytes, so one can hold invalid UTF-8. The trainer treats any existing sidecar, decodable or not, as an empty tombstone and never falls back to the metadata row for that image. The labeling grid and the dataset summary read an undecodable sidecar as absent instead, so both showed a metadata caption that the run would silently replace with the instance prompt, and counted the image as captioned. Both now track sidecar presence separately, so what the user reviews is what the run trains on. * Keep the reason a native server died, not just its backtrace A ggml abort prints its cause first and then a stack trace, so reporting the last twenty captured lines gave twenty addresses and nothing about the failure: on the macOS runner the native server died on an unimplemented Metal op and the message carried only frame pointers. The captured tail now leads with the lines that name a cause and keeps recent context after them, for both the startup failure and the mid-request death. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop staging the dense text encoder for an fp8 video load Two halves of the same gap, found while measuring the LTX-2.3 download plan: - The video download plan and the scoped pre-download never saw text_encoder_quant. An fp8 request loads a hosted pre-cast encoder, so asking for one still staged and downloaded the base repo's dense Gemma3 (48.79 GB of Lightricks/LTX-2 on the 2.3 distilled pick) that the pipeline then never opened. The plan now drops those shards and stages the pre-cast checkpoint instead; their configs stay, since the pre-cast loader still meta-inits the encoder from the base repo's component config. - The LTX-2.3 assembly builds every component itself, so pipe_kwargs (which carries the pre-cast encoder for from_pretrained) never reached it and an fp8 request silently loaded the dense encoder anyway. It is passed across explicitly now. The dense skip is earned, not assumed: only a pre-cast checkpoint that resolves on the Hub lets the plan drop the dense shards, and only one already fetched to disk lets the pull drop them, so an unpublished or gated artifact leaves both exactly as they were. If injection still fails after that, the load tops the dense weights back up rather than handing from_pretrained a snapshot with no encoder in it. Measured against the real Hub on the 2.3 distilled Q4_K_M pick: 67.24 GB before, 18.92 GB with a 0.43 GB stand-in for the pre-cast artifact (the base entry drops from 24 files / 48.79 GB to 13 files / 0.04 GB). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the file's typing idiom for flow_shift models/training.py annotates with typing constructs throughout (105 Optional[...], no Union), and flow_shift was the one place using PEP 604. Union[] reads like the rest of the file, and it also drops the runtime evaluation that would raise on Python 3.9. * Bound the gallery blob cache, and three interlock fixes Four review findings, all reproduced first: - The gallery object-URL caches were unbounded. A clip runs from a few MB to a few hundred, both pages stay mounted after their first visit, and entries were only dropped on delete, so scrolling pinned everything for the session. Both pages now share a byte-budgeted LRU (512 MB video / 192 MB images) keyed off the visibility signal the near-viewport fetching already provides. On-screen media, the selected clip or image, and the item just fetched are never evicted, so eviction is invisible and a single item larger than the whole budget cannot evict itself into a refetch loop. - The image, video and chat load guards ran two independent training probes but returned early when the FIRST one raised, so an unreadable LLM backend disabled the diffusion interlock and a load could proceed straight into an active diffusion trainer on the same GPU. The probes are independent now. - An engine switch swallowed a failed teardown and published the new engine anyway, which is exactly the leak the unload exists to prevent: the arbiter's evictor, /images/unload and the next load all resolve through get_active_diffusion_engine(), so the still-resident pipeline (or a live sd-server) became unreachable and the next load allocated on top of it. The switch now fails and leaves the old engine published, so it stays reclaimable. - The native generation timeout was 30 minutes while the Images page waits up to 6 hours (SETTLE_MAX_MS), so slow-but-progressing CPU jobs died deterministically at the deadline. Measured on GPU-less runners, a 512x512 4-step Q2_K generation took 900 s on Linux and 1465 s on Windows, so larger images or step counts clear half an hour easily. The ceiling now matches the page's window and applies to the whole request: chunks of a split batch share one deadline instead of each getting a full budget. Cancellation is unchanged. Declined: gating the huggingfacenotorch extra off Python 3.9 over the conditional diffusers marker. The marker is deliberate and its comment says why: diffusers dropped 3.9 in 0.38, so pinning >=0.39 outright leaves pip no candidate and the whole extra unresolvable there. The pipelines it names live in studio/backend, which cannot install on 3.9 anyway (studio.txt pins matplotlib==3.10.9 and fastmcp>=3.0.2, both requires_python >=3.10), and the extra is the general core one, so the alternative drops 3.9 for library users who never touch Studio. * Close the load-versus-training-start race, and two picker fixes - The image and video load guards read is_active() and only then selected an engine, acquired the arbiter and registered the load. A /train/diffusion/start reserving inside that window freed residents the load had not registered yet, so the trainer came up beside a brand-new pipeline. The service already had exactly the right pattern for this in dataset_mutation, so gpu_load_admission mirrors it: reserve() refuses while an admission is open, an admission refuses once a start is reserved, both decided under the one lock. The span is only the registration, since begin_load returns as soon as the load is registered and _free_gpu_for_diffusion_training preempts an in-flight load from that point. Chat is deliberately not covered: its load spans an eviction plus a multi-minute GGUF load, and it admits models that fit beside training by design, which is a different contract from the diffusion pipeline's all-or-nothing one. - Hugging Face gives the LTX-2 family the image-to-video pipeline_tag (both Lightricks/LTX-2 and unsloth/LTX-2.3-GGUF report it), so a text-to-video-only filter dropped the flagship audio family out of Video Hub search while the rest of the app routed it to Video. - Task-scoped quant fit sized picks against the LARGEST visible device while resolve_diffusion_device_target returns a bare "cuda" and torch places on the current one. On a heterogeneous host that recommended a checkpoint sized for the bigger card and then loaded it onto the smaller one. Fit now uses the device the load actually lands on; identical on a homogeneous host. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expose the persistent conditioning cache in the start schema DiffusionLoraConfig has carried cond_cache_dir for a while and the DiT trainer acts on it, but DiffusionTrainingStartRequest omitted the field, so Pydantic dropped it silently and every API-driven run fell back to the in-memory cache that is rebuilt from scratch each time. The warm path skips loading the VAE and the multi-GB text encoders on a rerun whose images, captions and resolution are unchanged, so this was a real capability that could not be reached. Contained like output_dir rather than left to the trainer subprocess's cwd, since it is another directory the trainer writes to. Blank or omitted still means the in-memory cache, so it must not resolve to the outputs root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix diffusion policy and classification issues from review fp8 auto precision defaulted to precise accumulate on any non-consumer GPU, which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates for both accumulate modes there, so the hardware premise held, but the cost is in the cuBLAS path rather than the published rate. Default to fast accumulate: measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8 TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a large win where it bites and free where it does not. Precise accumulate stays available via transformer_quant_fast_accum. Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the whole line was tagged image-diffusion-unsupported and hidden from the Images "On Device" list, though validate_load_request loads them. Resolve shared archs from the repo/file name like bare "wan" already does, with a test asserting the picker and the loader agree for every family. The sage attention on-demand install ran an unpinned `pip install sageattention`, but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the install always "succeeded", wrote an unusable version into the running venv, and was rejected on the next line. Carry the dispatcher's floor so pip resolves nothing instead. The dense-quant disk gate sized the download from the bf16-RESIDENT table. The fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB gate), leaving a window where the check passed and the download filled the disk; Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by published bytes, verified against HF sibling metadata for all 12 families. Patch installs went through unsloth_zoo, which refuses to import unless UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports unsloth at boot so it never showed there, but any other process ran silently unpatched with every install returning False, which is 13 test failures on a clean environment. Import unsloth and retry once, memoised per process. Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was unreachable; the gallery recipe recorded loras from the generate request alone, losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully cached load; and pickers.tsx imported three catalog-group helpers it never used. Reported by oobabooga. * Keep the sd.cpp text encoder on CPU under Metal macos-14 loads FLUX.2-klein-4B Q2_K natively on mps and then dies on the first generation with exit code -6: ggml_metal_op_encode_impl: error: unsupported op 'RMS_NORM' -> ggml_abort LLMEmbedder::encode_prompt -> LLMRunner::compute -> GGMLRunner::compute ggml's Metal backend gates RMS_NORM on contiguous rows and aborts the process when that does not hold, with no per-op CPU fallback, so any LLM text encoder (Qwen3 for FLUX.2 and Z-Image, T5 for FLUX.1) takes sd-server down. The encoder runs once per prompt while the DiT runs every step, so pinning only the encoder keeps Metal for the part that matters. UNSLOTH_DIFFUSION_SD_CPP_METAL_TE_GPU=1 opts back in once ggml grows the kernel. * Gate the unsloth retry in the diffusion patch backend The retry added for the clean-environment patch failures is not free: importing unsloth pulls torch in behind it, which costs ~940 MB of RSS measured in a process that had neither, and on a host with no accelerator it fails anyway. A cross-platform CI job that had generated fine at ~900 s later died 19 s in with SIGTERM and every 'if: always()' step skipped, which is the runner being torn down rather than a step failing. Retry only when torch is already imported (true of the server and of anything patching a real module, and the condition that stops the retry from being what loads torch), unsloth is installed but not yet imported, and the first failure was the ImportError the sentinel guard raises. The clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only retry the unsloth import where it can succeed The gate still let the retry run on hosts unsloth does not support, which is where it is most harmful: a 7 GB macOS runner lost the Studio server 26 s into a load, and the Linux runner was torn down mid-generation. Neither MPS nor plain CPU can complete the import, so the retry there pays the cost and fails anyway. Require an accelerator unsloth actually supports (CUDA/ROCm via torch.cuda, or XPU), with UNSLOTH_ALLOW_CPU as the documented override, and hoist the predicate to module level so it is tested directly rather than through the import system. On a CPU-only host the retry no longer fires at all; on CUDA the clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard old diffusers, stream video exports, record conditioned recipes Three fixes from review. The 0.39-only pipeline classes (Flux2Klein, Z-Image, Krea 2, LTX-2, HunyuanImage) were resolved by getattr deep in the load, so on the older diffusers that packaging still allows on Python 3.9 -- diffusers dropped 3.9 in 0.38 and this project still supports it, so the 0.39 floor has to be conditional or the extra becomes unresolvable -- an advertised model failed with a bare AttributeError after its checkpoint had already been downloaded. Krea 2 already guarded itself this way; assert_pipeline_class_available now runs the same check for every image and video family from validation, before any fetch, and names the version and the fix. WebM export accumulated the whole VP9 output in a BytesIO and returned it as one bytes object that the response held again. The request caps allow 2048x2048 for 1024 frames, so an export runs to hundreds of MB and concurrent clicks could exhaust the process, while the MP4 route beside it already streamed from disk. transcode_to_file encodes to a temp file and the route returns a FileResponse with a background unlink, so nothing large is resident. A conditioned generation's recipe carried only the txt2img fields, so the gallery presented an inpaint or upscale result as a complete Create recipe and restoring it replayed an unrelated text-to-image request. The images themselves are still not persisted (user uploads with their own lifetime), but the workflow and its scalars are, restore reapplies them, and the toast now names the inputs that have to be supplied again instead of silently landing on Create. Reported by Codex. * Per-load video cancel event, family-gated image picker, cond cache refusal A cancelled video load could resume: begin_load cleared the shared cancel event, and unload() drops _loading without waiting for the worker, so the next load cleared the very object the cancelled worker was watching and its multi-gigabyte pull ran on alongside the replacement until the token check at the end. Each load now gets its own threading.Event, passed down through _fetch_te_prequant and _predownload_base, so a cancelled worker stays cancelled. A cached repo with a model_index.json was advertised as text-to-image on the trust rule alone, but validate_load_request also requires a detected image family, so a trusted pipeline of an unsupported class produced a picker row that deterministically 400s. The picker now applies both gates, mirroring the video branch. cond_cache_dir was accepted for sdxl and then ignored: only the DiT trainer reads it, while the SDXL trainer builds a per-run in-memory latent cache, so the promised cross-run reuse never happened. The route now refuses it with a 400 that names the families which do support it, checked against the resolved family so an omitted model_family with an SDXL base is caught too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the frontend build broken by the gallery blob cache tsc -b failed on the branch head, so npm run build produced no dist and every platform job fell back to --api-only: blob-url-cache.ts(29,15): TS1294: This syntax is not allowed when 'erasableSyntaxOnly' is enabled dataset-labeling-grid.tsx / dataset-showcase.tsx: Argument of type '{ url: string; bytes: number; }' is not assignable to parameter of type 'string' The cache took its budget as a constructor parameter property, which the project's tsconfig forbids, and fetchGalleryObjectUrl now returns the blob size alongside the URL for that budget, which the two dataset thumbnail components still consumed as a bare string. Declare the field explicitly and destructure the URL at both call sites. tsc -b is clean and vite build emits dist again. * Recover from a ggml unsupported-op abort by restarting on the CPU backend ggml checks every node against the device's supports_op and calls GGML_ABORT when one is not implemented, because a single-backend graph has nowhere else to put it: there is no per-op CPU fallback. The whole sd-server dies with SIGABRT mid-generation and the user gets "the native image renderer stopped unexpectedly" with no way forward. Seen on macos-14 arm64 with FLUX.2-klein-4B Q2_K through the cross-platform CI: the text encoder is already pinned to CPU, and the abort moved into the denoise loop instead. ggml_metal_op_encode_impl: error: unsupported op 'MUL_MAT' -> ggml_abort StableDiffusionGGML::sample -> sample_k_diffusion A retry on the same backend would abort identically, so the load is restarted once with --backend cpu (the only flag that changes which backend executes the graph; --offload-to-cpu moves parameters, not compute) and the generation is re-submitted. The same checkpoint then renders slower rather than not at all. Strictly bounded: the signature must carry both the unsupported-op line and ggml_abort, the device must not already be CPU, and it happens once per load, so an OOM kill or a genuine crash still surfaces as itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix two tests that only fail in a full-suite run The 3.10 CI leg resolves PyAV 17, where av.container.OutputContainer is an immutable C type, so the no-libopus export test died on "cannot set 'add_stream' attribute of immutable type" before it asserted anything. Inject the refusal by wrapping the container av.open() returns instead; modules stay patchable on every build. Removing the injection makes the test fail again, so it still covers the branch it is named for. The Xet shim's degraded-path tests drop utils.hf_xet_fallback from sys.modules and import a throwaway copy. Restoring only the sys.modules entry left the utils package attribute bound to the throwaway, and the two disagreed for the rest of the process: a later monkeypatch of the dotted target patched one copy while the code under test imported the other, so the patch did nothing and test_fetch_te_prequant_only_reports_what_it_downloaded reached the real Hub and got a 401. Restore both bindings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop an ignored-cancel sd-server, guard deletes during diffusion training, repair unusable managed binaries sd-server does not interrupt an in-flight job, so when it ignores a cancel the grace branch abandoned the poll and reported cancellation while the native job kept a core (or the GPU) busy to completion and held the server's job slot. The comment said the caller stops the server, but only unload does that immediately: a superseding load stops it after its multi-gigabyte download, and a load that then fails never gets there. Stop it here, as the deadline branch already does. DELETE /api/models/delete-finetuned checked only the LLM trainer, so it could rmtree the output directory a live diffusion LoRA run was about to write its adapter into. Consult the diffusion training service too, like the dataset mutation and model-load routes. find_sd_*_binary only checks is_file(), so an interrupted extraction (or a prebuilt for the wrong CPU) left a present-but-unrunnable binary the installer never retried: every load probed it, fell back to diffusers, and native inference stayed off until the directory was deleted by hand. Probe it and reinstall, but only for a copy under the installer-owned root -- SD_CLI_PATH, UNSLOTH_SD_CPP_PATH, an in-tree build and anything on PATH are the user's. * Plan the pre-cast text encoder, and make the cross-trainer GPU admission atomic An fp8 text-encoder request loads a hosted PRE-CAST checkpoint, but the image download plan never received text_encoder_quant, so the manager staged the base repo's dense encoder (FLUX.2-dev's Mistral-24B is ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) and the load then pulled the pre-cast file inline, outside the manager's progress and disk preflight. The plan now takes the field, resolves the hosted artifact with the same resolver the injection uses, stages that file, and drops only those components' dense weight shards. The load's own prefetch takes the same treatment, since it paid the same cost. Only a checkpoint that really resolves on the Hub earns the drop, so a gated or renamed artifact still stages the dense encoder the load will fall back to. The two trainers admitted each other with independent check-then-act guards: the diffusion route checks the LLM backend several network-bound preflights before it reserves, and the LLM route checks the diffusion service well before it spawns, so two near-simultaneous starts could both pass and train on one GPU. reserve() now re-tests the LLM backend under its own lock, and the LLM route holds the diffusion service's gpu_load_admission across its spawn, so exactly one of the two wins. Both halves fail open, so a chat-only install still trains. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the Advanced options a staged download planned against Staging does not set busy, so while a multi-gigabyte download runs the user can still change precision, memory mode, speed or the baked LoRA selection. The pending record held only the repo and artifact, and the completed download fired a load that read the CURRENT state: the staged file set could then be missing files that load needs (fetched inline, with no progress and no disk preflight) or hold gigabytes it no longer uses. One snapshot of every Advanced control is now taken when the plan is built, and it travels with the pending record into the load, so the load that runs is the one the download was planned for. * Do not advertise a family the installed diffusers cannot build The newer families (Z-Image, Krea 2, FLUX.2, LTX-2, HunyuanImage) exist only from diffusers 0.39, and 0.39 cannot be installed on Python 3.9 at all -- diffusers dropped 3.9 in 0.38, so the requirement is conditional or the whole extra becomes unresolvable. On such an environment the picker still offered those rows, every pick failed deterministically, and the error's advice to run pip install -U diffusers could not fix it without also upgrading Python. The cached-repo picker now applies the same availability check validate_load_request does, which is keyed on the pipeline class actually present rather than on the Python version, so it is also right for an intentionally pinned older diffusers on 3.10+. Fails open when diffusers cannot be imported at all: that is a different problem and the load path reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the diffusion engine selection after each router test The active engine is module state, and several tests set it by plain assignment because what _activate does to it is the thing under test, so monkeypatch could not undo it. A leaked ENGINE_SD_CPP left get_active_diffusion_engine() handing back the sd.cpp backend for the rest of the process, and every later route that reads the active engine then saw an unloaded model: eight tests in test_openai_images_generations_route.py returned 503 in a full-suite run while passing on their own. The autouse fixture now snapshots and restores it. * Stream gallery clips, and close three races around them Four fixes from the latest review pass. The video gallery downloaded each clip into a blob before it could play, so playback waited on the whole file (tens to hundreds of MB), seeking was limited to what had arrived, and every viewed clip stayed pinned in the webview. The file route already streams and serves ranges; it just could not be a <video src> because it is bearer-gated. Mint a short-lived signed link instead (its own HMAC secret, 12 hour TTL, separate from the image links) and hand it to the element, which then fetches only the ranges it plays. That removes the blob budget, its LRU and every revoke on this page. The sd.cpp readiness probe accepted any process answering on the port, so a foreign server that grabbed the port between the bind check and the spawn was adopted as ours. Confirm the listener is our child before reporting ready, and stay best-effort (psutil missing, an unknown owner, or any probe error still passes) so the check can only reject a definitely foreign process. Dataset import held its lock for the extract but not for the upload path, so two concurrent uploads into the same folder interleaved; take the same lock and return 409. And reject Windows device names (CON, NUL, COM1..9, LPT1..9, with or without an extension) plus trailing periods in dataset names, which are unopenable on Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build the image download plan for the engine that will load /images/download-plan always asked the diffusers backend, while /images/load picks the engine per host: a GGUF pick on a machine with no usable GPU routes to native sd.cpp, which reads a single-file VAE plus text encoders and never opens the base repo's sharded components. Measured on unsloth/FLUX.2-klein-4B-GGUF (Q2_K): the plan staged 7.66 GB of FLUX.2-klein-4B components the native load discards, and the 7.80 GB sd-cli actually needs was then fetched inline by the loader, outside the download manager's progress and its disk preflight. Z-Image-Turbo is the same shape. The plan now asks whichever engine the load will select. predict_engine() applies the selection policy without any side effect: it activates nothing (staging a download must not unload the resident model) and only locates the binary rather than installing it, but still counts an installable binary as available, since that is what the load does on a fresh host. The native backend gains a download_plan built from the same _asset_specs the loader fetches, returning the same envelope, so the manager stages exactly the files sd-cli opens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let a queued generation outlive the model, and three scan fixes Five items from the latest review; four were real. An unload or arbiter eviction only cancels the generation holding _generate_lock. A second request queued behind it holds no cancel event yet, and Python locks are not FIFO, so it could take the lock the instant the active denoise released it, still see a loaded pipeline, and run a whole new denoise after the model was told to go away: the eviction then waits minutes for it and an image lands after the eject. Unload and a superseding load now raise a fence under _lock before they queue, and a generation that wins the lock while one is pending refuses instead. The cached-model scan judged pipeline completeness across every revision, so a repo holding an older complete snapshot plus a newer companion-only one read as complete while the snapshot from_pretrained actually opens has no transformer. Both scans now look at the revision the loader will open. Deleting a dataset image deleted its caption sidecar unconditionally, which for cat.jpg alongside cat.png removed the caption the survivor still resolves to. The sidecar now goes only with the last image of that stem, matching what the thumbnail cleanup beside it already did. Importing an example into a folder that holds no images but does hold files fell back to promoting the staging dir one file at a time, so an interruption left a partial dataset that the image_count check accepts as complete on retry. Those files are folded into the staging dir instead and the promotion stays a single atomic rename. The MPS generator report does not apply: torch.Generator(device="mps") has worked since PyTorch 2.0 (pytorch/pytorch#91348) and the studio installer pins torch>=2.4. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten diffusion comments Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes. * Tighten diffusion comments (second pass) Collapse the remaining multi-line comment blocks in the video page, training routes and service, sd.cpp server and installer, memory and speed planners, and the shared request models. Comments only, no code or behaviour changes. * Tighten diffusion comments (third pass) Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes. * Restore the dataset when an example import cannot be promoted Promotion folds the folder's pre-existing entries into the staging dir so the swap is one atomic rename. Every failure after that fold left the user with nothing: the 409 path and an os.replace error both fell through to 'finally: shutil.rmtree(staging)', which deleted the entries that had just been moved in there, while the response said 'Nothing was written'. A same-named entry was also unlinked outright before the promotion was known to succeed. Park superseded same-name entries in a rescue dir instead of deleting them, record every move, and restore all of them if any step of the promotion fails. The fold loop itself is covered too: renaming a non-writable directory raises EACCES on POSIX, which previously escaped as a 500 after the earlier entries had already been moved out. A failed rename now maps to the same retryable 409 as the rmdir conflict. Verified with the reported trigger (a non-empty mode-500 directory whose name collides with an imported file): the folder listing is now identical before and after the failed import. * Drain the teardown fence on a failing unload, give each load its own cancel event Two independent leaks on the image path, both already solved elsewhere in the same file. unload() incremented _teardown_waiters, ran _unload_locked() and decremented, with no try/finally, while the superseding-load path used a finally for the same pair. _unload_locked ends in clear_gpu_cache(), whose CUDA branch calls synchronize/empty_cache/ipc_collect unguarded, and a sticky CUDA fault makes those raise. The count then never drained, so every later generation was refused as cancelled for the life of the process, a fresh load included, since begin_load's own increment and decrement are symmetric. unload() is reached from the chat/video GPU handoff, the engine router and two training routes, so one fault during an ordinary handoff wedged image generation until restart. Release it in a finally. The image and native backends each cleared one shared cancel Event on a new load. unload() sets that event to cancel an in-flight multi-GB download and drops _loading in the same breath, so a replacement load is admitted while the cancelled worker is still inside the fetch, and its clear() re-enabled the very object that worker was watching: the cancelled download resumed and ran alongside the replacement. Take a fresh Event per load and thread it to the worker, as the video backend already does, and set it under the lock since begin_load now rebinds the attribute. * Name utf-8 on the diffusion text I/O and the sd.cpp subprocess pipes tests/test_text_io_encoding.py failed on five files this branch adds. Text I/O without an explicit encoding falls back to the Windows ANSI codepage, so a non-ASCII path or manifest value round-trips corrupted, and the three sd.cpp pipes decode the child's UTF-8 output as ANSI on Windows despite already passing errors = 'replace'. Eleven read_text() / write_text() sites across diffusion_compile_cache, diffusion_ideogram4 and diffusion_krea2, plus text = True on the sd-cli version probe, the sd-cli run and the sd-server pipe. * Record the load-time build on a gallery image's recipe A gallery record documents itself as the image's full generation recipe and is embedded in the PNG, but the only load-related field it carried was the repo id. A GGUF repo holds many quants, so that does not say which one made the pixels, and it says nothing about an adapter baked in at load time. The fallback meant to cover the baked case could never fire: with no loras on the request _adjust_baked_loras zeroes every baked adapter and _active_lora_pairs drops zero-weight entries, so active_loras was always empty. A baked-and-disabled build is not the same pipeline as a never-baked one, so the recipe could not reconstruct the image once the model was rebuilt. Persist model_kind, gguf_filename, transformer_quant and the baked adapter names, read off the load state rather than the request, and show them in the recipe popover. The new fields are optional with defaults, which matters because list_gallery_images drops any record that fails validation, so a required field would have emptied every existing gallery; a regression test pins that. * Drop eleven duplicated comment tails, restore the mxfp8 denial note The comment passes collapsed several wrapped blocks onto one line without deleting the last physical line of the original wrap, leaving the tail of each sentence repeated as its own comment underneath. Two of the eleven were re-worded rather than byte-identical, so a strict suffix match missed them. 6e16ad16f also dropped the line justifying the qwen-image mxfp8 denial while that rule stayed live in _FAMILY_SCHEME_DENY, under a header that then documented only fp8 and nvfp4. Restored. Comments only; verified with the AST gate. * Only let the dense-quant fallback use shards the prefetch actually staged The prefetch skips the base repo's transformer/ shards whenever a prequant checkpoint is expected, since that checkpoint replaces them. But a prequant fetch can fail for reasons the planner cannot see: an unpublished, gated or renamed artifact, a hub 5xx, a proxy, a checkpoint the validator rejects. The loader then fell through to from_pretrained(subfolder = 'transformer') and pulled those shards inside the load lock during 'finalizing', after the previous pipeline was already evicted, where the cancel event has no reach, load_progress has already reported bytes_downloaded == bytes_total, and the cache-disk gate had only reserved the small prequant checkpoint. That is verbatim the situation _dense_quant_prefetch_needed's own docstring exists to prevent. Gate the in-loader dense fallback on the shards being staged, read off the returned file list rather than the request so a failed size estimate closes it too, and let the GGUF build take over otherwise, which is what the prequant-sized replan already does one branch over. It is also the invariant the text-encoder path already enforces: only a component that really resolves may have its dense weights dropped from a plan or a prefetch. Adds the missing coverage for the gate's prequant arm, which the existing disk-gate tests never reached. * Refuse a training output_dir that resolves to the outputs root resolve_output_dir strips a leading 'outputs' and drops '' / '.' segments, so '.', './', './.', 'outputs', 'outputs/outputs' and ' . ' all clean away to nothing and land on the outputs root itself rather than a run directory under it. The DiT trainer then writes pytorch_lora_weights.safetensors flat into the root, where scan_trained_models and scan_checkpoints cannot see it (both filter is_dir()), and a second such run overwrites the first. The UI only checks the field is non-empty, so 'outputs' is one plausible run name away. Refuse it with a 400 that says what to do instead. cond_cache_dir collapses the same way but has an honest 'off' to fall back to, so a root-resolving value now means the in-memory cache, which is what the comment above it already promised: otherwise a run drops one flat safetensors per cached latent and caption into the directory trained models live in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair three defects the main merge left behind Git merged all three files without a conflict, but the result was wrong in each case. The tests only started failing once the merged tree was run. routes/inference.py duplicated the GGUF load block: this branch had moved the gguf_load_in_flight marker and the _hub_download_blocks_gguf_load guard under "if config.is_gguf and config.gguf_hf_repo", and the conflict resolution re-added main's copy at the old position, so both ran. Dropped main's copy; the earlier placement is the deliberate one, so a 409 from the hub guard cannot tear down a resident Images or Video pipeline. test_gpu_selection.py still called _hf_offline_if_dns_dead, which main renamed to _hf_offline_if_unreachable_for (#7591). Disjoint edits, so no conflict, but four route-error tests referenced a function that no longer exists. test_gguf_load_cache_reuse.py anchored its ordering assertion with rindex over "if config.is_gguf:", taking the last one before the load marker. That only held while _resolve_inherited_extra_args sat above every such line; main has since hoisted it above the gpu_ids preflight, so the anchor landed between the call and the marker and the assertion compared against an unrelated later call site. The ordering it checks is a property of _load_model_impl as a whole, so it now anchors on the function. The ordering itself is intact: _resolve_inherited_extra_args, then the gguf_load_in_flight marker, then the hub guard, then the chat handoff, then unload_model. * Stop a real sd.cpp install from breaking its own discovery tests The five "nothing is installed" assertions in test_sd_cpp_engine.py cleared SD_CLI_PATH and UNSLOTH_SD_CPP_PATH and patched Path.home, which covers hops 1, 2 and the fallback half of hop 3. It leaves two hops live. Hop 3 goes through managed_install_root(), which honors UNSLOTH_STUDIO_HOME and STUDIO_HOME and resolves to the stable-diffusion.cpp directory beside the studio home. That is the documented way to run side-by-side Studios, so anyone who has one set gets a real sd-cli back from a finder the test expects to return None. Hop 4 is the in-tree developer build, which does the same for anyone who built sd.cpp inside the checkout. Isolated with an autouse fixture rather than another helper call, because reaching the failure needs no fixture: SdCppEngine(binary = None) runs the finder from its constructor, which is why test_generate_raises_when_binary_missing failed too. The fixture points the studio home and the in-tree root at an empty tmp tree, so every hop is answered by the test rather than by the box. The in-tree root moves into a named in_tree_install_root() so it can be pointed somewhere empty; behaviour is unchanged, including the OSError and IndexError guard on an unexpected layout. * Make the sd.cpp uninstall test actually extract the code it tests The two sed ranges anchored the production fragments at column 0, but both blocks sit inside the main removal function and are indented, so each range matched nothing and LOOP_FILE / DEFAULT_FILE came out empty. Sourcing an empty file is a no-op, so the suite reported 4 passed / 7 failed: the seven removal assertions failed because nothing ran, and the four that passed were 'kept' assertions that pass trivially when nothing runs. The shell job auto-discovers tests/sh/test_*.sh and runs each under set -e, and this file is not in its skip list, so it was a deterministic red. Anchor on optional leading whitespace and fail loudly on an empty or _remove_path-less fragment, so a future reshuffle of uninstall.sh cannot make the suite vacuous again. Now 11 passed, 0 failed, against the real removal loop and the real default-mode block. * Agree on what an engine can build, and on what the installer owns Two gates, each half-applied. The unbuildable-family gate had one caller, the image branch of the cached repo picker. The GGUF classifier, the local-model classifier and the video branch had none, so on a diffusers too old for a family the picker still offered its GGUF and the load then failed. Meanwhile the loader asserted the diffusers pipeline class before engine selection, so a GGUF this host routes to native sd.cpp, which instantiates no pipeline class at all, was refused with an upgrade instruction that could not help. Both are wrong one-sidedly: hide a family only when NEITHER engine can build it, and demand the diffusers class only when diffusers is what will load it. One predicate, family_buildable_here, now answers both, so the picker and the loader cannot disagree. The population is real on Python 3.9, whose diffusers ceiling is 0.36: Flux2KleinPipeline, Krea2Pipeline and LTX2Pipeline are all absent there, and FLUX.2-klein GGUF is a repo in this PR's title. That assertion also raised RuntimeError, which /images/load maps to 409, the status that otherwise means a load is already in progress, and which escaped /images/download-plan (it catches ValueError and FileNotFoundError) as a bare 500 with the message lost. It is an unloadable pick like every other, so it raises ValueError and both routes answer 400 with the text intact. The managed-binary repair used a path test for ownership while the installer uses a marker. On a managed root without the marker, which is any install predating it, the repair deleted sd-server and the reinstall was then refused, permanently, because the surviving sd-cli keeps the directory non-empty so the marker can never be claimed: the user went from an unrunnable binary to no binary and no way back. Require the marker before discarding, which is the same definition of ours that uninstall.sh already uses to keep a user's own stable-diffusion.cpp checkout. A genuinely interrupted extraction still self-heals, since install() writes the marker before it extracts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gguf and av to the studio extra so the wheel install matches studio.txt This branch added both to studio/backend/requirements/studio.txt, which is what install.sh uses, but never to the studio extra in pyproject.toml, which is what 'pip install unsloth[studio]' uses. Nothing else keeps the two in sync, so tests/studio/install/test_studio_extra_matches_requirements.py failed deterministically, and the CI job runs the whole tests/ tree. The drift is not only a red test: gguf backs diffusers' GGUFQuantizationConfig and av does the MP4 encode and audio mux, so a wheel install got a Studio that cannot read a GGUF or export a video, which is most of what this PR adds. * Fix two test-suite defects: the CPU-only patch gate and a popped module The patch backend reaches unsloth_zoo's helpers only through an ~940 MB 'import unsloth', which a CPU-only host cannot complete, so every patch install returned False and nine arch/eager/compile tests failed there -- exactly the runners the retry was narrowed to protect. unsloth_zoo only wants UNSLOTH_IS_PRESENT in the environment, which costs nothing and needs no accelerator, and the conftest already sets its sibling UNSLOTH_ALLOW_CPU. Set it there too, as run.py and main.py already do at module scope, so no real server ever took the expensive route. CPU-only goes from 9 failed to 24 passed, 1 skipped; the GPU run is unchanged at 29 passed. test_setup_cache_env_hf_home popped utils.hf_cache_settings to model a fresh process and never restored it, so a later import built a second module object and rebound it on the utils package. test_hf_cache_settings then wrote its setting into one object while core.inference.diffusion read the other, the same split-module failure the xet shim already had. Restore both bindings in teardown: 21 pass together, and each file still passes alone. * Restore the setup_fail assertions the main merge reverted A fourth instance of the class 9541cc535 fixed: the merge took main's studio/setup.sh, which #7644 changed to abort through the setup_fail helper so desktop mode still emits [TAURI:ERROR], but kept this branch's older copy of the test, which still asserted the literal 'exit 1'. setup.sh is byte-identical to main here and the only diff in this file was the reverted assertions, so take main's version. 60 passed. * Fence the video teardown, reject non-finite clipping knobs, stop a test installing sd.cpp The video backend never had the teardown fence the image backend documents. Its unload and its superseding-load path both signalled the active generation, then did 'with self._generate_lock: pass' and tore the state down with the lock free. A generation queued behind that barrier holds no cancel event yet, so the signal cannot reach it, and Python locks are not FIFO: it won the lock the instant the barrier released it, read a still-loaded state and denoised a whole clip against the pipeline being freed. Reproduced on both paths, where the queued generation returned a finished MP4. Mirror the image backend: count waiters, refuse a generation while one is pending, and tear down inside the barrier with the counter released in a finally. _teardown_state becomes _teardown_state_locked since the caller now holds both locks. max_grad_norm and snr_gamma were bounded on one side only, so 1e309 floated to inf and passed. clip_grad_norm_ then computes an infinite clip coefficient, clamps it to 1.0 and scales nothing, and min(snr, inf) / snr makes every min-SNR weight 1.0. The run starts, reports normal progress and trains with the requested knob silently disabled. Measured both. NaN already failed the bounds; allow_inf_nan makes that explicit. learning_rate and flow_shift already guard this exact vector. test_load_routes_to_sd_cpp_on_cpu stubbed ensure_sd_cpp_binary but not ensure_sd_server_binary, which select_and_activate_engine probes first with installs enabled, so the unit test downloaded and unpacked 108 MB into the developer's real ~/.unsloth when UNSLOTH_STUDIO_HOME was unset. Now 0 bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the video schedule reset on the resolved defaults, not the repo id A GGUF repo holds several variants, so another client swapping a distilled build for the base one from the same repo changes the steps and guidance the backend reports while repo_id stays put. modelChanged stayed false, the page kept the previous schedule, and later generations ran an 8-step distilled setting on a model that expects roughly 40 steps and CFG 4. Include the reported defaults in the key. They only move when the resident artifact does, so a manual steps change still survives a status poll. * Retire the release-lag pin now that 2026.7.6 carries the relaxed gates The virgin Windows container row that installs from PyPI on purpose was tolerated with continue-on-error, because a Server Core container has no Microsoft Store and so no winget, and the released studio/setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime. #7549 relaxed both, but it landed on 2026-07-28 and the newest wheel was 2026.7.5 from the 23rd, so the row could not get past it. The pin carried its own tripwire for exactly this moment, and it has fired: unsloth 2026.7.6 shipped 2026-07-29 and the released wheel now installs end to end, so the step reported success and the assertion errored with "delete this pin and drop continue-on-error from the Install step so this row gates". Doing that. Both rows now gate unconditionally, and the assertion that enumerated the accepted failure signatures goes with the pin it protected, since there is no longer an accepted failure for it to describe. The other three release-lag mentions in this file are descriptive rather than tolerated-failure pins: they explain why the fedora and ubuntu-nonroot legs run with overlay: true, and both already refuse a triton failure as a regression rather than accepting it as lag. Left alone. * Pin the hosted pre-quant onto the plain-torch fp8 kernel A Z-Image GGUF pick at the DEFAULT speed mode, once the hosted fp8 pre-quant is actually reachable, dies at generate with an HTTP 500: torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors. Developer debug context: unsupported operator: mslk.f8f8bf16_rowwise.default _fp8_config already pins KernelPreference.TORCH when it BUILDS a config, precisely because the default AUTO switches to the MSLK kernel wherever an mslk package is importable (sm90+). A hosted checkpoint escapes that pin completely: the preference is serialized per Float8Tensor, and all 239 weights in the published Z-Image-Turbo-FP8 checkpoint carry AUTO. Loading it re-arms the very kernel the pin exists to avoid, mslk.f8f8bf16_rowwise has no fake impl, and the first compiled generate therefore cannot be traced. Isolated away from the product to be sure of the mechanism: quantise one Linear three ways on this box and compile each. KernelPreference.TORCH eager ok, compiled ok KernelPreference.AUTO eager ok, compiled FAILS on mslk.f8f8bf16_rowwise library default eager ok, compiled FAILS the same way So the pin is correct and necessary, and the only gap is that the hosted path never got it. _validate_checkpoint checks scheme, granularity, base model, min_features, exclude tokens and fast_accum, but not this. Rewriting the preference on load is safe: it selects a matmul kernel, it is not weight data, so the tensors stay bit-identical and the checkpoint's own state_dict_sha256 still describes them. It is also the faster path compiled, since the opaque extern call blocks inductor quantize fusion. Why this went unnoticed: the pre-quant repos are private, so nothing that could not read them ever took this path. It becomes the default the moment they are readable. Verified end to end at 1024x1024, 8 steps, speed_mode default, on the published checkpoint: before HTTP 500 at generate, diffusion.generate_failed after pinned 239 weights to the plain-torch fp8 kernel, load 10.0 s, transformer_quant fp8 with compiled engaged, cold 7.9 s, warm 0.87 0.87 0.87 0.86 0.87 s That warm number also beats the 1.4 s recorded for this shape on 2026-07-26. 159 prequant and transformer-quant tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fp8 kernel pin so the 500 cannot come back quietly The hosted pre-quant re-arming MSLK was only visible as an HTTP 500 on a compiled generate, which needs a GPU, a readable private repo and an importable mslk to reproduce. None of that is available in CI, so the pin would rot unnoticed. Three hermetic cases on _pin_kernel_preference instead: AUTO weights are rewritten and already-TORCH ones are left alone (counting only what changed), a weight that refuses the assignment does not sink the whole load, and with no torchao enum available the checkpoint is left exactly as saved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments across the image generation, video and training code Rewrite the comments added by this branch to be shorter and clearer: collapse multi-line explanations into one or two lines, drop comments that only restate the code, and keep the rationale that explains why a choice was made. No code changes. * Studio: let the pipeline-class guard survive a host with no diffusers Backend CI installs the CPU-only dependency set, which has no diffusers, and runs without a GPU. 13 tests failed there for three separate reasons, one of which is a real product bug rather than a test-environment gap. assert_pipeline_class_available did a bare "import diffusers", so on any host without it the call raised ModuleNotFoundError instead of the ValueError its own contract promises. /images/download-plan catches only ValueError and FileNotFoundError, so that escaped as a bare 500 with the message lost, which is the exact failure the guard exists to prevent. It is reachable in production: the native sd.cpp engine serves GGUF picks on a CPU or Apple host where diffusers is never installed. Absent diffusers now returns, since the check answers "is the installed diffusers new enough for this family" and there is no version to judge; a pick that genuinely needs diffusers still fails in the loader with its own message. The rest were tests asserting through gates unrelated to what they cover: - Three cond_cache_dir route tests drive a DiT family, which the start route refuses without an accelerator. They now take the existing dit_train_host fixture, written for this case, so they keep testing the schema on every host. - test_download_plan_forwards_the_load_time_controls stubs the diffusers planner but sends a GGUF pick, which routes to the native planner on a GPU-less host. It now pins the engine, so it tests kwarg forwarding rather than the hardware. Engine selection keeps its own tests. - The pipeline-class guard test needed a real diffusers only for its sweep over every shipped family. That half is split out and skips on its own; the stub-driven refusal needs no diffusers and still runs. Added a test for the absent-diffusers contract above. 409 pass on a GPU host with diffusers; 301 pass and 1 skips with diffusers blocked and no GPU. * Studio: cut the image generation comments down further Second pass over the comments this branch adds. Collapse the multi-line blocks that still read as paragraphs, rewrite the longest one-liners so they say the same thing in fewer words, and drop a stale comment that had drifted away from the constant it described. No code changes: only comments and whitespace. * Studio: workflow rail, sidebar Video row, and Train settings polish Sidebar - Pin Video under Images by default; persist store bumped to v3 so an untouched install adopts it while a custom arrangement is left alone. - Add "Customize sidebar" to the end of the More flyout, opening Settings > Appearance scrolled to the sidebar nav section. Images - Replace the workflow dropdown with an icon rail down the left edge: all seven workflows visible, keyboard nav, tooltips carry the labels. - Workflows stay selectable with no model loaded, so one can be set up before picking a model. Generate is already gated on a loaded model. - Video moves out of the Create/Train strip into a link at the far right, with a matching Images link on the Video page. - Wider gutters around the settings column, headings matched to Train. Train - Field guidance moves into "i" tooltips; only state that limits a control stays on the page. Steps, LoRA rank and the rest gain hints. - /info reports params, qlora_vram_gb, gated and note as fields, so the family note renders as chips. vram_note is rebuilt from them unchanged for older clients. Shared - Number steppers appear on hover or focus. - Lighter control border (#e9e9e9) on Images and Video, light mode only. * Studio: move Advanced inline on Images and Video - Replace the top-bar toggle and right-docked Advanced panel with a disclosure under Seed, so load-time tuning sits with the settings it affects and opening it no longer shrinks the preview canvas. - Video uses the bordered variant, since offload and memory decide whether a model fits at all. - Open state persists per page in localStorage. - Tighten the Steps unit trigger: 14px chevron, less right padding. * Studio: full width media pages and layout polish - Drop the 1100px cap on Images Create, Images Train and Video, so the preview canvas grows with the window instead of sitting in a band. - Align the workflow rail to the model selector label, widen the gap before its divider, and restore the divider itself. - Advanced: more room above and below, icon and label sized to the slider rows, and the same quiet row on Video as on Images. - Give Video a pane heading and description, matching Images. - More space above Seed on both pages. - More no longer takes the active style when the current page is one of its own rows. * Studio: fix the Extend side toggles The resting outline used a ring, and index.css blanks the ring on the button holding mouse focus, so the side you just clicked lost its border until focus moved. Use a border for the outline and leave the ring to focus-visible. Also separates the two states properly (the off state had no surface of its own), drops the border in dark mode as the inputs do, and adds the aria-pressed and focus-visible styling the buttons were missing. * Studio: drop the border on the Extend side toggles Light mode should carry no border either, so the fill alone marks the state: a muted surface when off, a primary tint when on. Matches the borderless treatment dark mode already had. * Studio: move the Images workflows into the sidebar The workflow switcher was a vertical icon rail on the page. It now lives in the sidebar under Images, so the page keeps that width for the canvas. - Workflows list under the Images row, with a chevron to fold them away. Hovering the row peeks them in a flyout on the standard menu surface. - Clicking Images while already there toggles the list instead of navigating. - The listed workflow carries the highlight, so the Images row drops it. - Create takes its own icon: it was sharing the New chat pencil. - Model hub moves above Projects. Persisted layouts bump to v4 and only adopt the new order where the stored one is still a shipped default. - Images and Video content both start at 32px, clear of the sidebar. * Studio: simplify the Images workflow submenu Dropping the hover flyout: two ways to reach the same seven workflows, one of them overlapping the list right below it, read as clutter. - The flyout is gone. The workflows are rows under Images and nothing else. - On the Images page they are always open, since they are that page's switcher. Elsewhere they stay folded, and the row's chevron opens them. - Create takes the sparkles icon. - Create and Train both pad their settings column to 40px a side, so the two tabs line up with each other and with the model selector. * Studio: dock the generate action and tidy the Create controls The primary action sat at the foot of a long scroll, so it was off screen until you scrolled for it. - Generate, Video's Generate and Train's Start training float at the bottom of their settings column. No bar behind them: hover lightens the fill rather than thinning it, and the disabled state is opaque, so the controls underneath never show through. - Aspect ratios read as names: Square (1:1), Widescreen (16:9) and so on. - Width and height replace their sliders with two compact boxes. Type a size or pick one from the menu; the value still snaps into range and still drives the locked ratio. - Negative prompt gets a quiet disclosure under the prompt. It was there before but only above guidance 0, which is not the default, so it never showed. - The info "i" is smaller across the settings UI. * Studio: reveal Images and Video field hints on hover The "i" next to every field label sat there permanently, which made a column of settings read as busier than it is. Scoped to the diffusion pages by CSS rather than per component: the hints come from the page's own Field and SelectRow, the train panel's FieldLabel, and the chat ParamSlider, which is shared and should not change for chat. Reveals when the pointer is inside the field, on focus-within for keyboard, and stays inert while hidden. InfoHint grows a data-slot so the rule has a stable hook, rather than keying off its aria-label. * Studio: one-line sliders, edge fades, shared negative prompt - Steps, Guidance and the rest put label, track and value on one row. ParamSlider is shared with chat, so this is an opt-in prop and chat keeps the stacked layout. - The settings columns fade at whichever edge they run past, as the sidebar and model picker do, instead of cutting off. New useScrollFades hook drives all three. - Negative prompt moves to a shared component and Video picks it up, so both pages collapse it the same way. - Right gutter sits closer to the rule, Steps takes a bigger break above it, and the Advanced rule sits between its neighbours rather than up against the field above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add the API monitor to the sidebar nav Sits under More, below Export. Points at the monitor page, not the API keys dialog the profile menu opens. No store version bump: an id missing from a stored layout is appended with its default pinned state, so existing sidebars gain the row at the end and keep their own order. * Studio: take main's copy of the sidebar spinner test The merge kept this branch's older version, which pins the Recents row to h-[33px]. Main already relaxed that to any height, since row density moves independently of the trailing column the test is about. * Studio: fix FLUX.2 klein size resolution, HunyuanVideo tier routing, and the personalization save Five fixes found while auditing the model registries against the live Hub. FLUX.2 klein 9B loaded against the 4B config. One family covers both klein sizes and defaults to 4B, relying on the base_model card tag for the real base, but the trust allowlist only had klein-4B, so the correct tag was discarded and a 9B GGUF (inner_dim 4096) was loaded against a 4B config (3072). That surfaced as a bare shape mismatch from inside the GGUF quantizer naming neither the file nor the repo. Adds the missing allowlist entries and a header-level size check that fails early with a legible message; the check is fail-open, so an unreadable file, a non-FLUX.2 family, or an unmapped base leaves the load exactly as it was. The sd.cpp text encoder rule matched the literal "klein-9b", so klein-base-9B was handed the 4B encoder. HunyuanVideo-1.5 720p checkpoints routed to the 480p family. Only the literal 720p_t2v path was aliased, so 720p_i2v and every GGUF repack fell through to the generic token and inherited the 480p base repo, which also supplies the VAE and text encoder. The tier is baked into the weights (transformer target_size 640 vs 960, scheduler shift 5.0 vs 9.0) and the two bucket lists are disjoint, so this ran the whole pipeline off-tier. Resolution presets that the checkpoints were never trained for. Wan2.2 TI2V-5B is 720P-only upstream (SUPPORTED_SIZES is exactly 704x1280 and 1280x704, asserted in generate.py), and the HunyuanVideo 480p square preset 624x624 is not a bucket of its tier; 640x640 is. Personalization could never be saved. The frontend ships eight sidebar nav ids and refills every missing one on each save, but the backend Literal had seven and no "api", so every PUT to /api/settings/personalization returned 422. Aligns the server defaults and the Literal with the shipped layout. A catalog row pointing at a file that does not exist. unsloth/Qwen-Image-2512-FP8 holds torch prequant .pt checkpoints, not qwen-image-2512-fp8.safetensors, and fp8 is denied for this family anyway because it renders black, yet the row was the auto-route target on any GPU above 40 GB. Also marks FLUX.1-schnell, Krea-2-Turbo and the two Ideogram repos gated: all four are gated on the Hub today, and schnell being Apache-2.0 does not make it ungated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a case-only duplicate upload destroying an image, and size the load against the right GPU Two fixes from the review round. A dataset upload containing two names that differ only by case silently loses one on Windows and macOS. The in-batch duplicate check is an exact string compare, so "Cat.png" and "cat.png" both pass, but on a case-folding filesystem they are one destination: the commit step moves the first staged part aside, writes the second over it, then deletes the backup, while the response still reports both as uploaded. Captions collapse the same way and never even reach the image branch. No single folder can hold such a pair, but the Windows and macOS open dialogs flatten search and Recents results across folders into one multi-selectable list, which is a normal way a LoRA dataset gets assembled. Rejecting the pair everywhere would regress Linux, where the two really are different files with their own sidecars, so the check probes the filesystem once per process instead of keying off sys.platform: macOS also ships case-sensitive APFS volumes, and a Linux host can keep its Studio home on an exFAT or NTFS mount. A failed probe answers "case-sensitive", which leaves today's behaviour untouched. The quantization fit budget was sized against the wrong card under a reordering CUDA_VISIBLE_DEVICES. A bare "cuda" load lands on visible ordinal 0, but the reducer took the lowest PHYSICAL index, which the nvidia-smi path reports as index_kind "physical". Under "3,1" that sizes against GPU 1 while the pipeline loads onto GPU 3. The hook's own interface doc already promised the lowest visible ordinal. Ranks by visible_ordinal, falling back to index only for an older backend that omits it. * Studio: rework the diffusion dataset panel and align the pane gutters Upload: - one Upload button beside the dataset name, with an upload icon - picking files uploads them, no second confirm click - Add button beside the Training images dropdown, so images can go into a set that already exists - fall back to the upload form when a selected dataset no longer resolves Labeling grid: - two columns at any width; sm:grid-cols-3 crushed the tiles in a fixed width column - pin the caption size so the Textarea's md:text-sm does not outsize a tile - name the tile hover group; a bare one revealed every tile's Remove at once - Remove is an icon button on the image, not a word over the artwork - drop the amber tile fill, keep the No caption label - match the header, toggle and status text to the section's type scale Gutters: - the run area and the preview canvas now sit 40px off the rule, the gutter the settings column has off the page edge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give the dataset preview per-image hover and remove, and wrap it onto rows - each thumbnail is its own button, so it can carry an X that removes just that image; the strip was one button and could not nest a second control - hover lightens the image instead of drawing a border - wraps onto more rows rather than scrolling sideways, which also keeps the bottom corners rounded - samples 12 thumbnails, up from 8, now that a second row is available - a delete refreshes the panel's dataset counts * Studio: show the workflow icon beside the Images and Video headings Images reads it off WORKFLOW_TABS, so it is the same icon the sidebar submenu shows and it follows the active workflow. Video's heading is also Create, so it takes the Create icon rather than the nav row's film slate. * Studio: reject a duplicate LoRA id on the diffusion load path too DiffusionGenerateRequest already refuses a repeated adapter id, with a comment naming the hazard: _resolve_lora_set suffixes colliding adapter names, so the same id resolves the SAME adapter twice and set_adapters stacks both copies past the per-adapter weight bound. DiffusionLoadRequest bounds only the list length, and it matters more there. Generation-time stacking spoils one image; on the load path the adapters are baked into the quantized build before compilation, so the unintended combination rides every image until a reload. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: teach the GGUF reuse test double about holds_no_vram The resident-GGUF fast path consults llama_backend.holds_no_vram before asserting CHAT ownership, but two tests hand it a SimpleNamespace built before that attribute existed, so they raised AttributeError and returned a 500. A real LlamaCppBackend exposes it as a property; the doubles now carry it too. Product code is unchanged: these pass on main and were the branch breaking its own test, not a defect in the guard. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelhan <michaelhan2050@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com>	2026-08-04 08:11:01 -07:00