mirror of
https://github.com/unslothai/unsloth.git
synced 2026-07-10 00:08:58 +00:00
6148 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Daniel Han
|
55043acf50
|
Diffusion training base precision modes: bf16 speed mode (2.3-2.6x), int8, fp8, auto (#6839)
* [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add SDXL diffusion family (U-Net pipeline support) SDXL is the first U-Net family in the diffusion backend: its denoiser is pipe.unet (UNet2DConditionModel), not a DiT pipe.transformer, and a single-file .safetensors is the whole pipeline rather than a transformer-only file. The backend previously assumed a DiT transformer everywhere, so add the two hooks a U-Net family needs and register SDXL. DiffusionFamily gains denoiser_attr ("transformer" for DiT, "unet" for SDXL) and single_file_is_pipeline (SDXL loads a single file via pipeline_class.from_single_file with the base repo as config, instead of transformer_class.from_single_file plus a companion assembly). _align_vae_dtype now reads the denoiser generically so img2img and inpaint keep the VAE and U-Net dtypes aligned. The non-GGUF trust gate is extended with a short, exact-match, safetensors-only allowlist of official base repos (the SDXL base/refiner and sdxl-turbo), because SDXL ships only as a full pipeline and has no unsloth-hosted GGUF. Local paths stay trusted as before; a random repo, even one that detects as SDXL, is still rejected. The image-conditioned and ControlNet workflows are the standard SDXL pipelines, built around the resident modules via from_pipe like every other family, so SDXL gets txt2img, img2img, inpaint, outpaint, upscale, LoRA and ControlNet. There is no native sd.cpp mapping yet, so the no-GPU route falls back to diffusers. Frontend catalog gains SDXL Base 1.0 and SDXL Turbo entries with SDXL step/guidance defaults (Turbo: few steps, no CFG; base: ~30 steps, real CFG). Tests: new test_diffusion_sdxl.py (family shape, detection, trust allowlist, model kind, U-Net VAE-dtype alignment, LoRA gate) plus loader-branch tests in test_diffusion_backend.py (pipeline-kind from_pretrained, single-file whole-pipeline from_single_file, allowlist accept/reject). Verified live on GPU: sdxl-turbo loads both as a pipeline and as a single file and generates coherent txt2img + img2img. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images: LoRA free-text Hugging Face entry + recipe round-trip The backend has always accepted a bare Hugging Face repo id (owner/name, or owner/name:weight-file.safetensors) as a LoRA, downloading and applying it. But the picker only rendered when the curated catalog had entries, and the catalog is empty, so there was no UI path to apply any LoRA. Show the LoRA section whenever the loaded model supports LoRA, and replace the curated-only dropdown with a text input: type a Hub repo id, or pick a discovered adapter from a datalist of suggestions when the catalog is populated. Also restore LoRAs when loading a recipe. restoreSettings now parses the recipe's "id:weight" strings (splitting on the last colon, since the id itself may contain one for a specific weight file) back into the selection, so replaying a saved image reproduces its adapters. The generate payload trims hand-typed ids and drops empty / zero-weight rows, and a model swap clears the selection (a LoRA is family-specific) without discarding a free-text pick that is not in the curated list. * Add diffusion LoRA training (SDXL text-to-image) First diffusion training path in Studio: train a LoRA on the SDXL U-Net from an image + caption dataset and export it as a diffusers .safetensors that the existing diffusion LoRA loader (and any diffusers pipeline) can load. core/training/diffusion_lora_trainer.py: - DiffusionLoraConfig with validation/defaults (rank, alpha, targets, lr, steps, grad accumulation, resolution, min-SNR gamma, gradient checkpointing, lr scheduler, seed, mixed precision). - discover_image_caption_pairs: captions from metadata.jsonl / captions.jsonl, per-image .txt/.caption sidecars, or a dreambooth instance_prompt fallback (pure, unit-tested). - run_diffusion_lora_training: the loop -- freeze base, PEFT-wrap the U-Net attention projections, VAE-encode (fp32 VAE to avoid the SDXL fp16 overflow), sample noise + timesteps, predict, MSE loss with optional min-SNR weighting (epsilon / v-prediction), AdamW + get_scheduler + grad accumulation + grad clipping, then export via save_lora_weights. Emits worker-protocol events (model_load_*, progress, complete) and polls should_stop for a clean stop with a partial save. - run_diffusion_training_process: mp.Queue subprocess adapter (event_queue / stop_queue), so the training worker can spawn it; plus a CLI entry point. Only SDXL (U-Net) is trained here; DiT families and the Studio UI form + route wiring are follow-ups. The trainer is decoupled and worker-ready. Tests: test_diffusion_lora_trainer.py covers caption discovery (metadata / sidecar / instance prompt / skip-uncaptioned / errors), config normalisation + validation, the SDXL add-time-ids, and the dict->config adapter. Verified live on GPU: a 60-step SDXL LoRA run lowers the loss, exports a ~45 MB adapter, and loading it back shifts generation from baseline (mean abs pixel diff ~55/255). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * diffusion trainer: emit learning_rate in progress events (Studio pump compatibility) The Studio training pump reads 'learning_rate' from progress events; the diffusion trainer emitted 'lr'. Rename the field (and the CLI reader) so the trainer's events are directly consumable by the existing training status/SSE machinery when it is wired into the worker, without a translation shim. * Wire diffusion LoRA training into the Studio API Make the SDXL LoRA trainer reachable from the app with a small, self-contained job service and JSON routes, deliberately separate from the LLM TrainingBackend (whose lifecycle -- LLM config build, per-run SQLite rows, matplotlib plots, transfer-to-chat- inference -- is text-training specific and would mis-handle a diffusion run). core/training/diffusion_training_service.py: DiffusionTrainingService runs one job at a time -- validate the config cheaply (before any spawn), spawn the trainer subprocess (spawn context, parent-lifetime bound), pump its events (model_load_* / progress / complete / error) into an in-memory status snapshot, and support a clean stop. The subprocess context and target are injectable so the full start -> pump -> status -> complete path is unit-tested without real multiprocessing or torch. routes/training.py: POST /api/train/diffusion/start (400 on a bad config, 409 when a job is already running), POST /api/train/diffusion/stop, GET /api/train/diffusion/status (JSON poll). models/training.py: DiffusionTrainingStartRequest + response schemas mirroring DiffusionLoraConfig, so model_dump() passes straight through. Tests: test_diffusion_training.py -- service happy path, bad-config-before-spawn, concurrent-job rejection, clean stop, crash-without-terminal-event, event transitions; plus route wiring via the FastAPI TestClient (start / 422 / 400 / 409 / status / stop) with a mocked service. The diffusion trainer's progress events already use the field names this path expects. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images: add a Train LoRA (SDXL) dialog Surface the diffusion training API in the Images page. A "Train LoRA" button in the top bar opens a self-contained dialog to fine-tune an SDXL LoRA on a folder of images: pick the base model, dataset folder, output folder, an optional instance prompt, and the core hyperparameters (steps, rank, resolution, batch, learning rate), then Start. The dialog polls the training status while open and shows a progress bar, step count, live loss, and the saved adapter path, with a Stop button for a clean stop. The dialog is independent of the loaded generation model (training runs in its own subprocess), and prefills the base model with the loaded checkpoint when it is SDXL, else the SDXL base. api.ts gains startDiffusionTraining / stopDiffusionTraining / getDiffusionTrainingStatus plus their types, matching the /api/train/diffusion routes. * Import diffusion training schemas from models.training directly The import-hoist lint flags newly re-exported names in the models/__init__.py hub as unused (it does not treat __all__ membership as a use). Import the three diffusion training schemas straight from models.training in routes/training.py, where they are used in the route annotations and calls, and drop the __init__ re-export. * Remove stray async task scratch outputs committed by mistake * ControlNet: reject filesystem-like ids and do not cache a model past an unload race Two review findings on the ControlNet path: - resolve_controlnet's bare-repo fallback accepted any id with a slash, so a path-shaped id (/tmp/x, ../x) reached from_pretrained as a local directory. Restrict the fallback to a strict owner/name HF repo id shape. - _controlnet_pipe now re-checks the cancel event after the blocking from_pretrained: an unload that raced the download had already cleared the caches, so caching the late module would pin it past the unload. * Pipeline prefetch: fetch only the default torch weights A full-pipeline prefetch kept every repo file outside assets/, so an official repo that ships multiple formats (SDXL Base: fp16 variants, ONNX, OpenVINO, Flax, a top-level single-file twin) downloaded tens of GB from_pretrained never loads. Skip non-torch exports and dtype-variant twins in _pipeline_file_downloaded, and drop a component .bin when the same directory carries a picked safetensors weight (diffusers' own preference). * Diffusion LoRA training: fall back to fp16 when CUDA lacks bf16 The default mixed_precision=bf16 hard-fails on pre-Ampere GPUs (T4 / V100 / RTX 20xx) which have no bf16 compute; check torch.cuda.is_bf16_supported() and drop to fp16 there. * Diffusion training service: join the old pump outside the lock start() joined a finished job's pump thread while holding the service lock, but the pump's final state writes need that same lock, so the join always burned its full timeout and a stale pump could then overwrite the new job's state. Join outside the lock (with a re-check after), and fence _apply_event and the exit handler by process identity so a superseded pump can never touch the current job's state. Adds regression tests for both. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA training: harden config handling, cancellation, SDXL conditioning, and safety Addresses review findings on the SDXL LoRA trainer: - Gate the base model with the same trust check as inference (unsloth/*, allowlisted official bases, or a local path) before from_pretrained, so an untrusted remote repo is never fetched or deserialised. - Check the stop signal before the (slow) model load, not only between steps, so a cancel during download is honoured; a stop may carry save=False to cancel without leaving a partial adapter. - Per-sample SDXL add_time_ids from the actual crop (original size + crop offset, with the offset mirrored on horizontal flip) instead of a fixed uncropped-square tensor. - Apply EXIF orientation before resize/crop so rotated photos train upright. - Skip gradient clipping when max_grad_norm <= 0 (the Studio 'disable' value) instead of scaling every gradient to zero. - Coerce Studio config strings/blanks: learning_rate string to float, blank hf_token to anonymous, gradient_checkpointing 'none'/'true'/'unsloth' to bool; reject a zero/negative lora_alpha or learning_rate. - Alias the generic Studio training payload keys (model_name/max_steps/batch_size/lora_r/ lr_scheduler_type/random_seed) onto the diffusion field names. - Mirror the trained adapter into loras/diffusion so the Images LoRA picker discovers it. - Report worker exceptions in both message and error keys so the failure is not lost. Adds regression tests for the config coercion/validation and aliasing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ControlNet: address review findings on the diffusers path - resolve_controlnet enforces catalog family compatibility so a direct API call cannot load a ControlNet built for another family through the wrong pipeline. - Unknown ControlNet ids now surface as a 400 (call site maps FileNotFoundError to ValueError) instead of a generic 500. - strength 0 disables ControlNet entirely, so a no-op selection never pays the download / VRAM cost; the control image is decoded and validated BEFORE the ControlNet is resolved or built, so a malformed image fails fast for the same reason. - ControlNet loads use the base compute dtype (state.dtype is a display string, not a torch.dtype, so it silently fell back to float32) and honor the base offload policy via group offloading instead of forcing the module resident. - Empty/malformed HF token coerced to anonymous access. - Flux Union ControlNet control_mode mapped from the selected control type. - resolve_controlnet drops the unused hf_token/cancel_event params. - ControlNetSpec validates guidance_start <= guidance_end (clean 422). - Images UI ControlNet Select shows its placeholder when nothing is selected. Adds regression tests for family enforcement and the union control-mode map. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion training API: LLM interlock, pre-spawn VRAM free, path containment, no dropped knobs Four review findings on the diffusion training start path: - It spawned the SDXL trainer without checking the LLM TrainingBackend, so a start while an LLM run was active put two trainers on the same GPU. Add a symmetric interlock: diffusion start returns 409 when LLM training is active, and LLM start refuses while a diffusion job is active. - It went straight to service.start() without freeing GPU residents. Add a pre-spawn free of the export subprocess, the resident Images pipeline (with an arbiter release), and chat models, mirroring the LLM start path. - data_dir / output_dir were passed through unresolved, so Studio-relative names failed and absolute paths bypassed containment. Resolve them with resolve_dataset_path / resolve_output_dir before spawn (400 on an uncontained path). - The request model dropped max_grad_norm and lora_target_modules, so runs that set them trained with defaults. Add both fields. The gemini pump-join deadlock was already fixed earlier (join outside the lock + proc-identity fence). Note: honoring a stop DURING model load is a trainer-loop change owned by the diffusion training engine PR (should_stop polled before the first optimizer step). Adds route + model regression tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA: harden resolution, native tag precedence, and diffusers teardown Address review findings on the LoRA path: - resolve_one: normalise a blank/whitespace hf_token to None (anonymous access) and reject a client-supplied weight file with traversal / absolute path. - resolve_specs: convert FileNotFoundError from an unknown/stale id to ValueError so the route returns 400 instead of a generic 500. - _scan_local: disambiguate local adapters that share a stem (foo.safetensors vs foo.gguf) so each is uniquely addressable. - inject_prompt_tags: the backend-validated weight now wins over a user-typed <lora:ALIAS:...> for a selected adapter; unselected user tags are left alone. - diffusers _apply_loras: reject a .gguf adapter with a clear error before touching the pipe (diffusers loads safetensors only). - _unload_locked: drop the explicit unload_lora_weights() on teardown; the pipe is dropped wholesale (freeing adapters), so the previous call could race an in-flight denoise on the same pipe. - Images page: use a stable LoRA key and clear the selection (not just the options) when the catalog refresh fails. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion: guard trust check against OSError and validate conditioning inputs - _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid characters (or a bare owner/name id) can't raise OSError; treat any failure as not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request still raises the clear FileNotFoundError for a genuinely missing local pick. - generate(): reject mask_image / upscale / reference_images supplied without an input image, and reject reference_images on a family that does not support reference conditioning, instead of silently degrading to txt2img / img2img. * SDXL: reject GGUF up front, skip unused base weights, drop refiner, and harden helpers Addresses review findings on the SDXL family: - Reject a GGUF load for single_file_is_pipeline families (SDXL) in validate_load_request, before the route evicts the current model; SDXL has no transformer-only GGUF variant. - Skip base-repo weight files when a whole-pipeline single file is loaded: from_single_file (config=base) needs only the base config/tokenizer/scheduler, so a local .safetensors no longer triggers a multi-GB base download. - Remove the SDXL refiner from the non-GGUF trust allowlist: it is an img2img-only pipeline but this backend loads every sdxl repo as the base txt2img pipeline. - Normalize a blank/whitespace hf_token to None once in load_pipeline so every load branch degrades to anonymous instead of erroring on a malformed token. - Read the denoiser dtype from a parameter (compile-wrapped modules may lack .dtype) and access state.family.denoiser_attr directly. Adds/updates regression tests for the trust allowlist, GGUF rejection, and base-config filter. * Images: preserve restored LoRAs through model load and never send hidden LoRAs - The LoRA effect cleared the selection on every load->capable transition, which wiped adapters restored from a gallery recipe before the model finished loading. Track the previously-loaded family in a ref and clear only on a real family swap; keep the selection on the initial load and on unload. - Gate the generate payload's loras on loraCapable so a restored selection that is hidden (loaded model does not support LoRA) is never sent to the backend. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images Train LoRA dialog: token, validation, precision, base-repo prefill, gating, refresh Nine review findings on the SDXL training dialog: - Forward the saved Hub token so a gated/private SDXL base can be trained (the image load flow already sends it). - Re-seed the base-model field from the current default each time the dialog opens; the keep-alive dialog otherwise kept its mount-time default after a model loaded. - Prefill from base_repo (the diffusers pipeline) rather than repo_id, which for a GGUF/single-file SDXL load is the checkpoint path from_pretrained can't open. - Add client-side validation of steps/rank/resolution/batch/learning-rate before the request. - Expose a precision selector (bf16/fp16/fp32) so non-bf16 GPUs can train from the UI, not only the API. - Gate the dialog on the active Images route (active && trainOpen) so switching tabs closes it and stops its polling. - Rescan the LoRA picker when a run completes, so a freshly-trained adapter appears without a model reload. - Cap the dialog height and scroll the body so the Start/Stop footer stays reachable on short viewports. - Correct the copy to not over-promise picker auto-discovery. Freeing the resident Images pipeline before training is handled backend-side in the diffusion training start route. * Merge diffusion-sdxl into diffusion-lora-ux; keep options-only LoRA catch The catalog-refresh .catch from the lower branch clears the selected adapters too, which is right for its catalog-only picker but wrong here: this picker holds free-text HF repo ids that are valid without being in the catalog, so a transient refresh failure must not wipe them. Family swaps still clear the selection and hidden LoRAs are never sent. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Train LoRA dialog: stop suggesting absolute paths the backend rejects The dataset and output placeholders showed /path/to/... examples, but the training routes resolve those fields inside the Studio home and reject absolute paths outside the approved roots, so following the placeholder produced a 400. Use folder-name placeholders and say in the labels and the dialog description where each folder resolves. * Align the VAE to the denoiser's first FLOATING dtype, not its first parameter A GGUF-quantized transformer's leading parameters are packed uint8 storage, so reading next(parameters()).dtype handed nn.Module.to() an integer dtype and every image-conditioned generation on a GGUF model (Qwen-Image-Edit) failed with a 500. Probe the parameters for the first floating dtype, treat an all-integer module as a no-op, and also catch TypeError so an unexpected dtype can never break generation. Regression test included. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Count LR scheduler warmup/decay in optimizer steps, not micro-steps lr_sched.step() runs once per outer optimizer step (after the gradient accumulation inner loop), for train_steps total. The scheduler was configured with num_warmup_steps and num_training_steps multiplied by gradient_accumulation_steps, so with accumulation > 1 a warmup or non-constant schedule stretched past the run and never reached the intended decay. Count both in optimizer steps. * Address Codex review findings on the image-workflows PR Keep diffusion.py importable without torch: the compile/arch patch modules import torch at module level, so import them lazily at their load/unload call sites instead of at module load. This restores the torchless contract so get_diffusion_backend() works on a CPU/native sd.cpp install. Match family reject keywords and aliases as whole path/name segments, not raw substrings, so an unrelated word like edited, edition, or kontextual no longer misroutes or hides a valid base image model, while supported edit families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same segment matching in the picker task filter. Route FLUX.2-dev native guidance through --guidance like the other FLUX families rather than --cfg-scale. Reject native upscale requests that have no input image. Read image header dimensions and reject over-limit inputs before decoding pixels, so a crafted small-payload image cannot spike memory. Reject an upscale that would shrink the source below its input size. Validate the model_kind against the filename extension before the GPU handoff. Estimate a local diffusers pipeline's size from its on-disk weights so auto memory planning does not skip offload and OOM. Report workflows: [txt2img] from the native backend status so the Create tab stays enabled for a loaded native model. Clamp the outpaint canvas to the backend's 4096px decode limit. Adds regression tests for segment matching and kind/extension validation. * Guard inference loads and worker lifetime against diffusion training Teach the chat and image load guards about an active diffusion (SDXL) LoRA job: a chat load is refused (its footprint cannot be fit-checked against the trainer) and an image load is refused outright, mirroring the existing LLM training guards, so a load can no longer allocate GPU memory alongside the trainer and undo the pre-start cleanup. Bind the diffusion trainer subprocess to the parent's lifetime and scrub the native path lease secret from it by running the child through run_without_native_path_secret, matching the inference/export/LLM workers, so a Studio crash or kill no longer leaves the trainer holding the GPU. Reset in_model_load on the complete and error terminal events: a stop or failure during model loading otherwise leaves the status reporting a stale loading indicator after the job has ended. * Harden diffusion LoRA handling on the diffusers and native paths Reject LoRA on a torch.compile'd diffusers transformer (Speed=default/max): diffusers requires the adapter loaded before compilation, so applying one to the already-compiled module fails with adapter-key mismatches. The status gate now hides the picker and generate raises a clear message instead. Convert a cancelled Hub LoRA download (RuntimeError Cancelled) to the diffusion cancellation sentinel in resolve_specs, so an unload/superseding load during resolution maps to a 409 instead of a generic server error. Drop weight-0 LoRA rows before the native support gate so a request carrying only disabled adapters stays a no-op on families where native LoRA is unsupported, matching the diffusers path. Reject duplicate LoRA ids in the request model: both apply paths suffix colliding names, so a repeated id would stack the same adapter past its per-adapter weight bound. Strip all user-typed <lora:...> prompt tags on the native path (only the selected adapters are materialized in the managed lora-model-dir, so an unselected tag can never resolve), and restore saved LoRA selections from a gallery recipe so restore reproduces a LoRA image. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden ControlNet resolve, gallery metadata, and the control-type picker Check cancellation immediately after a ControlNet from_pretrained and before any device placement, so an unload/eviction that raced the download does not allocate several GB onto the GPU after the load was already cleared. Require a loadable weight or shard index (not just config.json) before a local ControlNet folder is advertised, so an interrupted copy is hidden instead of failing deep in from_pretrained as a generic 500. Do not record a strength-0 ControlNet in the gallery recipe: it is treated as disabled and skipped, so the image is unconditioned and the metadata must not claim a ControlNet was applied. Build the control-type picker from the selected ControlNet's advertised control_types instead of a hardcoded passthrough/canny pair, so a union model with a precomputed depth or pose map sends the correct control_mode. * Address further Codex findings on the image-workflows PR - Persist the actual output image size in the gallery recipe instead of the request sliders: Transform/Inpaint/Edit derive the size from the uploaded image, Extend grows the canvas, and Upscale resizes it, so the sliders recorded (and later restored) the wrong dimensions for those workflows. - Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file name) in validate_load_request, so the unloadable pick fails before chat is evicted rather than deep in from_pretrained. - Only publish an image-conditioned from_pipe wrapper to the shared aux cache when the load is still current: from_pipe runs under the generate lock but not the state lock, so an unload racing its construction could otherwise cache a wrapper over torn-down modules that a later load would reuse. - Verify the Windows CUDA runtime archive checksum before extracting it, like the main sd-cli archive, so a corrupt or tampered runtime is rejected rather than extracted next to the binary. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse non-SDXL base models at diffusion training start The trainer only supports the SDXL U-Net, but a FLUX / Qwen-Image / Z-Image repo or a GGUF filename passed as base_model was accepted and then failed minutes later inside StableDiffusionXLPipeline.from_pretrained with an unrelated-looking error. Add a name-based guard in normalized() so known DiT-family names and .gguf checkpoints are rejected up front, which the API start route surfaces as an immediate 400 with a message that says exactly which bases are trainable. Unrecognisable names still pass through so custom local SDXL checkpoints keep working. * Add diffusion dataset upload and training info endpoints Training an image LoRA required knowing the Studio home layout and copying files onto the server by hand, which is the most confusing step of the whole flow. Two small endpoints fix that: - GET /api/train/diffusion/info reports the datasets and outputs roots plus every dataset folder that contains images (with image/caption counts), so the UI can offer a picker instead of a blind free-text path. - POST /api/train/diffusion/dataset uploads images and optional caption .txt / metadata.jsonl files into a named folder under the datasets root, creating it on first use and accumulating on repeat uploads so large sets can arrive in batches. Names are validated to a single path component and files stream to disk under the same per-upload size cap as LLM dataset uploads. The returned name is a valid data_dir for /diffusion/start. * Rework the Train LoRA dialog into a guided SDXL flow The dialog assumed users knew the Studio home layout and that only SDXL is trainable, and hid both facts behind free-text fields. Restructure it around the three real decisions: - Base model is a dropdown of the trainable SDXL picks (Base 1.0, Turbo, the loaded SDXL pipeline when there is one) with a custom repo/path escape hatch, instead of a bare text field defaulting to a repo id. - Training images come from an in-browser upload (new dataset endpoints) or a picker over existing dataset folders with image/caption counts. No shell access or knowledge of the datasets root is needed any more, and the captioning rules are explained inline. - The output field is now Adapter name and the instance prompt is labelled as the trigger prompt, with a no-captions warning wired to the selected dataset's actual caption count. Hyperparameters collapse behind a training settings toggle since the defaults suit a first run. A completed run says where the adapter went and offers Done / Train another, and the top-bar button gets an icon and a plainer description. The dialog title states the SDXL-only scope and that other families load LoRAs but cannot train them yet. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Validate diffusion training config before freeing the GPU The start route freed resident GPU workloads (export, Images pipeline, chat) before the service validated the config, so a start that was then refused, now including a non-SDXL base model, tore down the user's loaded model for nothing. Run the same cheap normalise pass first; the LLM path already follows this rule via its before_spawn hook. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor diffusion LoRA training into a family-aware platform Split the SDXL trainer into a shared, architecture-agnostic layer so more model families can be trained without duplicating the plumbing: - New core/training/diffusion_train_common.py holds the config + validation, dataset discovery, event emission, stop protocol, adapter publishing, and a lazy trainer registry (get_trainer). diffusion_lora_trainer.py keeps the SDXL-specific loop and re-exports the moved names so existing imports are unchanged. - The SDXL-only base-model blocklist becomes a positive check: the family is resolved from the base model (or an explicit model_family) via the diffusion family registry, and a known-but-not-yet-trainable family is refused with a clear message. Unknown custom names still default to the SDXL trainer. - DiffusionFamily gains a trainable flag and train_base_repos; SDXL is marked trainable. DiT families flip on when their trainers land. - Trained adapters now write a <name>.json metadata sidecar (family, base model, rank, trigger prompt, ...) that the LoRA scanner reads to family-gate the adapter in the picker instead of showing it as unknown for every model. - The training base-model trust allowlist adds the official FLUX.1-dev, Z-Image-Turbo, and Qwen-Image repos (safetensors-only, no remote code). * Retain diffusion training loss history and expose it in status The training service kept only the latest loss, so a live loss chart could show a single point. Fold each progress event into bounded (step, loss, lr) history arrays (capped at 4000 points, decimated when full) plus the latest throughput and peak VRAM, and record the family / base model / catalog path on completion. The status endpoint returns these as a nested metric_history object the UI can chart directly, and the start request accepts an optional model_family override. * Tests for the diffusion training platform Cover the trainer registry (get_trainer resolves SDXL, unknown family raises), family resolution (explicit model_family validation, resolved_family on the config), the metadata sidecar write + scan read with family gating, and the service loss-history folding (append, bad-point skipping, decimation at cap, family/perf fields) plus the status route nesting metric_history. * Add diffusion dataset labeling and example-import endpoints The Train tab needs to let users caption small datasets in the browser and pull in a ready-made set to see training work end to end, neither of which the upload-only endpoint supported. Add, under /api/train/diffusion/dataset: - GET {name}/images lists every image with its resolved caption (metadata beats a per-image sidecar, matching the trainer's discovery order) so uncaptioned images are visible and flaggable. - GET {name}/image/{filename} serves an image, with ?thumb=<px> returning a cached downscaled JPEG kept in a hidden .thumbs subdir (regenerated when the source is newer) so the labeling grid stays light. - PUT {name}/caption/{filename} writes, or when blank clears, the .txt sidecar; DELETE {name}/image/{filename} removes the image plus its sidecars and thumbnails. - GET dataset-examples lists a curated, license-labelled registry, and POST dataset/import-example materializes one into a dataset folder as numbered images + .txt captions. Two loaders cover the shapes seen in the wild: streaming rows from datasets.load_dataset (dog-example, Tuxemon) and a snapshot + jsonl walk for imagefolder repos whose captions live in a non-standard *.jsonl (the public-domain tarot set). Imports are idempotent and cap the image count. Filenames and dataset names are validated against path traversal and pinned inside the datasets root. * Test diffusion dataset labeling and example-import endpoints Cover caption precedence, thumbnail generation and .thumbs exclusion, caption write/clear, image delete cleanup, path-traversal rejection on names and filenames, and example import with a mocked datasets.load_dataset (files plus sidecars written, idempotent second call, cap respected, load failure mapped to 502). * Add flow-matching DiT LoRA trainers (FLUX.1-dev, Qwen-Image, Z-Image) Extends diffusion LoRA training beyond SDXL to the three popular DiT families via a single shared flow-matching loop parameterised by small per-family specs (loading, prompt/latent encoding, transformer forward, save). Verified against diffusers 0.38.0: - FLUX.1-dev: 2x2 latent packing + image ids, guidance-embed forward, on-the-fly nf4 QLoRA of the 12B transformer (the dev repo is gated, so training needs the user's HF token). - Qwen-Image: 5D VAE latents normalised by the per-channel latents_mean/std, img_shapes forward, prequant nf4 base by default (on-the-fly nf4 for the bf16 base). - Z-Image: list I/O with the reversed timestep convention and a negated prediction, bf16 only. The registry (get_trainer) and DiffusionFamily.trainable / train_base_repos now route these families to the DiT trainer; the SDXL blocklist guard is replaced by a positive family resolution that also rejects GGUF repos (inference-only) and still-unsupported families. Per-family defaults + labels + VRAM notes are exposed via family_train_infos for the Train UI. Memory: caption embeddings are precomputed once and the text encoders freed before the loop; gradient checkpointing (non-reentrant, required for bnb 4-bit) and 8-bit AdamW are on by default. * Speed up + shrink SDXL LoRA training (precompute text embeds, 8-bit AdamW) SDXL re-encoded every caption with both CLIP text encoders on every step (pure waste, since captions are constant) and kept the encoders resident. Precompute each unique caption's embeddings once, then free the text encoders before the loop: numerically identical (embeddings are deterministic and this consumes no torch RNG, so the noise/timestep stream is unchanged) but faster and ~1.5 GB lighter. Default the optimizer to 8-bit AdamW (bitsandbytes) with an fp32 fallback, halving optimizer state with no meaningful LoRA quality cost. Env toggles (UNSLOTH_DIFFUSION_NO_PRECOMPUTE / _FP32_OPTIM) let the accuracy guard A/B the paths. * Expose trainable families in /diffusion/info and preflight gated bases The training info endpoint now returns the trainable model families (name, label, default + allowed base repos, recommended defaults, and a VRAM/access note) so the Train UI can offer a base picker with realistic guidance. The start route preflights a gated base repo (HEAD model_index.json with the user's token) BEFORE freeing resident GPU workloads, so a missing FLUX.1-dev license/token fails fast with an actionable 400 instead of evicting the loaded model and then hitting a confusing mid-load 401. * Tests for DiT trainers, family resolution, info families, gated preflight Cover the DiT spec table, the QLoRA prequant heuristic, the Z-Image bf16-only guard, the gated-repo name check, family resolution now that FLUX/Qwen/Z-Image are trainable (and GGUF repos are rejected as inference-only), the families list in /diffusion/info, and the gated-base 400 preflight that leaves the GPU untouched. * Add diffusion training API client: metrics, families, dataset labeling, examples Extends the Images training client for the Train tab: the status type now carries metric_history (step/loss/lr) plus catalog_path/family/base_model/samples_per_second/ peak_memory_gb; the start request gains model_family; and info gains an optional families list (per-family bases + defaults). Adds typed calls for the dataset labeling and one-click example endpoints: list images with captions, thumbnail URL, write/clear a caption, delete an image, list example datasets, and import an example. * Add diffusion Train panel: config, dataset labeling, live charts, deploy New full-page training workspace for the Images tab. Left column configures the run: model family (FLUX.1-dev, Qwen-Image, Z-Image, SDXL in popularity order, with per-family VRAM/license notes and defaults, backfilled from the backend families list when present), base repo, dataset (existing folder, browser upload, or one-click example import), an in-browser caption labeling grid (per-image thumbnail + caption saved on blur, delete, uncaptioned highlight), adapter name, trigger prompt, and collapsed training settings. Right column shows the live run: progress + loss/avg/speed/peak-VRAM readouts, the reused training loss/LR charts fed from metric_history, and a completion card that deploys the adapter into Create or starts another run. * Wire Create/Train tab switch into the Images page and deploy flow Replaces the Train LoRA dialog with a top-bar Create | Train segmented control next to the model selector. Create renders the existing generation workspace unchanged; Train renders the full-page training panel (unmounted in Create so its polling stops while the backend run and its retained metric history survive a tab switch). Adds a deploy handler: loading the trained adapter's base as a pipeline, queueing the adapter so the LoRA discovery effect applies it once the base is loaded and LoRA-capable for the matching family (with a mismatch warning), seeding the prompt with the trigger, and switching back to Create. Removes the now-unused dialog. * Wrap the DiT training forward in bf16 autocast The fp32 LoRA parameters and the bnb 4-bit base matmuls need a single compute dtype during the forward, exactly like the diffusers dreambooth scripts run under accelerator.autocast. Without it the 4-bit backward on FLUX.1-dev fails with an illegal-address CUBLAS error partway into the first step. Z-Image and Qwen-Image smokes are unaffected and the SDXL path (its own trainer) is untouched. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix Train tab example cards and Create/Train tab layout The example-dataset cards used a two-column grid in the ~340px config column, which wrapped titles one word per line and let the long license text overrun into the neighbouring card. Switch to one card per row with a horizontal layout: title with a compact truncated license badge (full text in the tooltip), a two-line clamped description, and the Import button on the right. The Create/Train switch had an icon inside the Train trigger that overhung the pill corner. Drop the icon, make both triggers a fixed equal width so the active pill sits flush in the top bar. * Show only loss and learning-rate charts for diffusion training The Train tab reused the LLM charts section, which also rendered an empty Grad Norm card and an Eval Loss card showing an Evaluation not configured placeholder with a red smear. Neither applies to diffusion LoRA training. Add a diffusion-only two-card view that reuses the loss and learning-rate cards directly with fixed presentation defaults, and note under the loss chart that per-step loss is noisy by design so users read the smoothed line for the trend rather than the raw jitter. * Add a dataset preview strip to the Train tab When a dataset with images is selected, show a strip of up to 8 sampled thumbnails with a +N more tile, so users can see what is in the folder before training. Clicking the strip opens the existing caption review grid. Samples are drawn evenly across the folder and refresh on dataset change or after an upload/import. * Stop example cards from overflowing the Train config column The example-dataset cards still overran the ~340px config column: the license used the Badge component whose baked-in w-fit and whitespace-nowrap ignored the max-width and truncate, and the grid children had the default min-width auto so wide content pushed past the column edge and clipped the Import buttons. Replace the badge with a plain truncating pill span, and give the config column min-w-0 with overflow-x-hidden so nothing escapes its width. * Add Smithsonian Butterflies and Nouns example datasets Two permissive ~100-image sets for the Train tab: huggan/smithsonian_butterflies_subset (CC0, the classic diffusers-docs training set, imported as a subject set with a trigger prompt since its metadata columns are species names not captions) and m1guelpf/nouns (CC0, captioned pixel-art avatars via the text column). Both cap at 100 images. * Paginate the Train tab caption grid with prev/next controls Large example datasets (100+ images) rendered every tile at once, so the caption review grid grew unbounded. Show 24 images per page with < > chevrons and an x-y of N indicator; a new dataset or refresh resets to the first page. * Offer example datasets in the Train dropdown with previews Add an Examples group to the training-images dropdown that imports a curated dataset in one pick, alongside the existing cards. Cards now show up to three preview thumbnails pulled from the public HF datasets-server so the set is visible before download. Hide the trigger prompt when every image already has a caption (a captioned style set needs no trigger), and turn the training-settings toggle into a ghost button with a rotating chevron. * Clamp the training base repo to the selected family The base-model select's state could briefly hold the previous family's repo after a family switch (the reseed effect runs a beat later, and a value with no matching option makes the browser display the first option anyway). The request then carried the stale repo: picking Qwen or Z-Image still sent black-forest-labs/FLUX.1-dev and surfaced FLUX's gated-repo error under the wrong family. Derive an effectiveBase clamped to the current family's repos and use it for the select value, the start request, and the deploy fallback. Also move the Trigger prompt above Adapter name: the trigger describes the dataset, the name only labels the output. * Speed up diffusion LoRA training and cut DiT peak VRAM by a third Perf core for the diffusion trainers, defaults preserving the training math: - Phased model loading: the pipeline now loads without its transformer (conditioning only), captions are encoded and the text encoders freed, the VAE latent cache is built and the VAE freed, and only then does the transformer load. The multi-GB denoiser never shares VRAM with the encoders, cutting measured peak VRAM on B200: FLUX 17.1 -> 10.4 GB, Qwen-Image 19.1 -> 12.8 GB, Z-Image 7.3 -> 4.7 GB. - Latent cache (cache_latents, default on): per-image crop/flip variants (cache_variants, default 4 vs the single frozen variant of the diffusers --cache_latents) store the VAE posterior's affine parameters, so every step still draws a fresh VAE sample; a cached center-crop Z-Image run matches the uncached one at the bf16 nondeterminism floor. - True batching: train_batch_size now actually batches the transformer forward (it was silently 1). nf4 dequant dominates the step cost, so batch 4 lands near batch-1 step time: 4.0x samples/s on Qwen-Image, 3.1x on FLUX, 2.1x on Z-Image, with multi-seed loss envelopes overlapping batch-1. - LR scheduler support in the DiT loop (lr_scheduler / lr_warmup_steps were accepted but ignored); progress events now report the real per-step LR. - TF32 + high fp32 matmul precision under enable_tf32 (default on), snapshot/restored around the run. cudnn.benchmark is scoped to a caller opt-in only: autotuning the fp32 VAE convs doubled peak VRAM on the DiT families for zero steady-state gain. - Vectorized sigma gathering (drops a per-step Python search loop), cached FLUX img_ids/guidance, fused torch AdamW fallback, steady-state samples_per_second (excludes the first-step warmup). - Regional torch.compile plumbing (compile_transformer off/on/auto with eager fallback): auto stays off over a bitsandbytes base where compile is a net loss (27 s warmup, slightly slower steady on Z-Image); it arms automatically for the dense/quantized speed modes that follow. - Stop parity with the LLM trainer: /api/train/diffusion/stop accepts an optional {save} body and the service forwards save=False as a no-save cancel; a new preparing event surfaces cache-build progress. - SDXL trainer gets the same latent cache, perf flags, and fused fallback; its batching, LR schedule, and min-SNR stay as they were. Verified: 83 backend tests green; per-family 30-40 step runs with adapter round-trip generation through the normal LoRA path (FLUX, Qwen-Image, Z-Image all pass). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add base_precision speed modes to DiT training: bf16 2.3-2.6x, int8, fp8 New base_precision config for the DiT trainers: nf4 (unchanged default) | bf16 | int8 | fp8 | auto, advertised per family + per machine through /api/train/diffusion/info (precision_modes, recommended_precision, supports_compile) so the UI can gate the selector. - bf16: dense transformer + regional torch.compile (auto-armed). The measured speed mode: 2.3x nf4 on FLUX (1.81 -> 4.12 steps/s), 2.6x on Z-Image (2.5 -> 6.38 steps/s) on B200, at dense-weight VRAM (FLUX 24.7 GB / Z-Image 13.6 GB peak vs 10.4 / 4.7 for nf4). - int8: torchao weight-only int8 on the frozen base, quantized AFTER add_adapter (quantizing first trips peft 0.18's TorchaoLoraLinear, which is incompatible with the torchao 0.16 config API). Runs eager: inductor rejects the int8 subclass training graph (aliased subclass outputs), so compile is force-disabled for it. - fp8: torchao convert_to_float8_training on the frozen linears (filter skips lora_ modules, proj_out, non-divisible-by-16 dims, pad_inner_dim), applied after add_adapter, compile auto-armed. Works and round-trips, but measured SLOWER than compiled bf16 at LoRA-training shapes (FLUX 3.15 vs 4.12 steps/s; Z-Image similar), so it is an explicit opt-in and auto never picks it. - auto: free VRAM (measured before load) + dense-size table -> bf16 when it fits with headroom, int8 in the middle band, else nf4. Prequant bnb repos always resolve to nf4; dense modes on them are rejected at validation with a pointer to the family's dense base. Two crashes found and fixed along the way: - The cuDNN SDPA backend's training graph fails on the FLUX attention shapes (torch 2.10 + cu130, B200): mha_graph.execute errors, then the context degrades into illegal memory accesses. The perf-flag guard now pins flash/mem-efficient SDPA for the run (mathematically equivalent, snapshot/restored). nf4 escaped it by routing attention differently. - Regional compile now uses dynamic=True (the inference layer's proven default): dynamic=False specialisation fused a gemm_and_bias epilogue that failed with CUBLAS_STATUS_EXECUTION_FAILED on the FLUX training graph; dynamic=True is also faster (Z-Image 3.84 -> 6.38 steps/s). Verified: 98 backend tests green (new test_diffusion_base_precision.py: validation, auto policy table, fp8 filter, compile gating, /info fields); per-mode 40-step runs on FLUX + Z-Image with loss means inside the nf4 envelope and adapter round-trip generation through the normal LoRA path for bf16-, fp8-, and int8-trained adapters. * Clear TF32 flags when enable_tf32 is off so the opt-out is strict fp32 * Address review: auto int8 requires the dense-load transient to fit, dense modes are CUDA-only, auto respects bf16 compute, exact cudnn SDPA restore * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: fp32 latent cache stats + strict-JSON-safe progress floats - Latent caches (DiT + SDXL) now hold the posterior mean/std in fp32 and draw the per-step sample in fp32, casting only the result to the training dtype. This matches the in-loop path (encode fp32 -> sample fp32 -> cast) exactly instead of sampling in bf16; the cache is tiny so the doubled RAM is negligible. - The training service nulls non-finite floats (NaN/Inf loss, avg_loss, learning_rate) at its single ingestion point so status snapshots and persisted run records stay strict-JSON serializable; the metric history skips non-finite loss points. Test covers NaN/Inf progress followed by a finite point. * Address review: gate auto int8 on torchao, scope dense validation to DiT - base_precision="auto" only picks int8 when torchao is importable (the int8 quantize has no runtime fallback, unlike fp8); otherwise the middle band falls back to nf4. Threaded as a parameter so the policy stays pure. - The dense-mode validation (prequant base / bf16 compute) now applies only to DiT families: sdxl ignores base_precision entirely, so a leftover value can no longer fail an SDXL run. The mode-name validity check still runs everywhere. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gate int8 and fp8 on a functional torchao import, not find_spec The Windows ROCm torchao import stub satisfies find_spec and even lets from torchao.quantization import quantize_ succeed, but its quantize_ is a no-op: auto would pick int8, leave the transformer dense, and disable compile as if it were quantized. has_functional_torchao imports the exact symbols the int8 path uses and rejects the stub via its sentinel; both the auto picker and the /info advertised modes now use it * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Skip the sigma-gather test when diffusers is not installed CI runs the backend suite without diffusers; the test checks our index math against the scheduler's own gather, so it skips rather than fails there. * Coerce cache_latents and enable_tf32 string flags in the config dict path The generic Studio config dict path can deliver these flags as strings, and a non-empty string like "false" is truthy, so an opt-out silently no-ops (the latent cache still builds, TF32 stays on). Coerce them the same way gradient_checkpointing already is. * Remove committed runtime scratch artifacts and ignore their dirs logs/ (a 1.3 MB ComfyUI object_info dump plus stale PID files), temp/ (PR body and commit message scratch), and async_task_outputs/ (agent task transcripts) are environment specific runtime artifacts that were committed by accident and carry stale local state into every checkout. Remove them and gitignore the directories so they cannot be re-added. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore pre-Ampere bf16 fail-fast in the DiT trainer The perf rewrite dropped the bf16 capability guard, so a pre-Ampere CUDA device (T4/V100/RTX 20xx) would die deep in model load with an opaque dtype error instead of a clear message. Restores parity with the SDXL trainer. * Size-gate the automatic diffusion latent cache The latent cache holds two fp32 posterior tensors per crop/flip variant per image, pinned on CUDA hosts, so datasets with thousands of images can exhaust host or pinned memory with no fallback. Estimate the cache size from the first real encoded latent and fall back to per-step VAE encoding when it exceeds a 4 GiB budget. UNSLOTH_DIFFUSION_FORCE_LATENT_CACHE bypasses the gate; the existing UNSLOTH_DIFFUSION_NO_LATENT_CACHE opt-out is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gate DiT training precision: deny fp8 for Qwen, gate explicit int8 on torchao, gate advertised dense modes + route on bf16 - normalized() + family_train_infos() mirror the inference fp8 deny for Qwen-Image (activation outliers exceed fp8's range and corrupt the trained result); int8 stays allowed and the UI no longer advertises fp8 for it. - _resolve_base_precision() gates an explicit int8 on a FUNCTIONAL torchao, the same gate auto and /info already apply, so a missing/stub torchao fails fast instead of silently loading dense with compile disabled. - train_precision_modes() gates the dense modes (bf16/int8/fp8/auto) on torch.cuda.is_bf16_supported(), so a non-bf16 CUDA GPU (T4/V100/RTX 20xx) is offered only nf4 instead of a start that evicts resident models and then fails. - start_diffusion_training preflights bf16 support for the DiT families BEFORE _free_gpu_for_diffusion_training(), so any DiT start (nf4 included, since the trainer requires bf16 unconditionally on CUDA) fails fast without eviction. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gate DiT training on functional torchao for explicit int8; hide always-400 DiT modes on non-bf16 GPUs The start route preflight only rejected non-bf16 GPUs; an explicit int8 request on a host with a missing or stub torchao passed the preflight, evicted resident GPU workloads, then died in the trainer child (its int8 base quantizer has no fallback). Fold both gates into training_precision_preflight_error so int8-without-torchao fails fast before eviction. Also empty the advertised DiT precision_modes (and surface the reason in vram_note, drop compile) whenever the bf16 preflight would reject the family, so /info never offers an nf4 DiT option the route always 400s. * Reject dense DiT precisions on a CUDA-absent host before eviction; stabilize family-info tests The start-route preflight caught the bf16-GPU and int8-torchao requirements but not the dense precisions' CUDA requirement: on a GPU-less host bf16_unsupported_reason exempts CPU-only, so a bf16/fp8 (or int8-with-torchao) DiT request passed the preflight, evicted resident workloads, then raised only in the trainer child. Add the dense-mode CUDA gate mirroring _resolve_base_precision so the doomed run is rejected up front. Also pin bf16_unsupported_reason in the two positive-path family-info tests so they are deterministic across GPU types (a non-bf16 CUDA box would otherwise empty every DiT family's advertised modes). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
pre-commit-ci[bot]
|
e800675128 |
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci |
||
|
Daniel Han
|
fc1e099124 |
Harden ControlNet loads, thumbnail cache keys, API training guard, and picker roving keys
Review follow-ups on the image-generation PR: - ControlNet: resolve_controlnet accepts a bare owner/name repo without the non-GGUF base trust gate, and _controlnet_pipe hands it straight to from_pretrained. A malicious pickle .bin would deserialize on load, so run the same Hugging Face malware preflight (evaluate_file_security) the chat and export loaders use before any remote ControlNet load; local dirs are exempt. - Dataset thumbnails: key the cache on the full filename instead of the stem so sample.png and sample.jpg no longer collide on one .thumbs file (which could serve or delete the wrong image); the delete cleanup globs the same key. - Diffusion training start: mirror start_training's API-key guard so an API client cannot start training (which frees VRAM by unloading chat) while an inference request is streaming; it now returns 409 before any GPU is freed. - Model picker: include the curated safetensors row keys in the recommended roving key list so arrow-key navigation reaches those rows instead of hitting the duplicate option-missing id. Tests: ControlNet malware gate (remote blocked before from_pretrained, local skipped), thumbnail same-stem cache separation, API-key diffusion-start 409 before GPU free. Full diffusion suites green. |
||
|
Daniel Han
|
f313cfd7e5 |
Security audit: baseline the new huggingface-hub Sandboxes findings
The latest huggingface-hub release added the Sandboxes feature. Its bootstrap (_sandbox.py) fetches the static sbx-server binary into /tmp with an Authorization header and marks it executable, which is exactly the staged-dropper pattern the scanner hunts, and three while True polling loops in _sandbox.py / hf_api.py / utils/_http.py match the beaconing heuristic. All four verified against the official huggingface/huggingface_hub repository: the snippet is the documented sandbox server injection and the loops are deadline-style job and sandbox polling. Entries generated with --write-baseline and reviewed line by line; scan_packages.py huggingface-hub now exits 0 with the four findings suppressed. |
||
|
Daniel Han
|
7a05d8655b |
Stream every DiT through group offload, not just the primary transformer
A dual-DiT pipeline (Ideogram 4's unconditional tower) placed its second denoiser resident under the group tier, which defeats the tier since the pair rarely fits where one alone did not. Stream transformer_2 and unconditional_transformer alongside the transformer and keep only the smaller companions resident. |
||
|
pre-commit-ci[bot]
|
bcf25ca569 |
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci |
||
|
Daniel Han
|
e3ad2263fb | Merge branch 'image-generation' of https://github.com/unslothai/unsloth into image-generation | ||
|
Daniel Han
|
c53a6cb65e |
Address review findings: dataset preflight, sd.cpp unload barrier, caption tombstone, ControlNet cache race
- Run the trainer's caption discovery in the start route BEFORE freeing GPU residents, so a missing or uncaptionable dataset 400s without evicting the loaded chat/Images model. - sd.cpp unload now waits out a cancelled one-shot generation on the generate lock before reporting the device free, matching the diffusers backend. - Clearing a caption that came from metadata.jsonl writes an empty sidecar tombstone instead of unlinking (both readers treat an existing sidecar as authoritative), so the cleared label cannot resurface. - The ControlNet wrapper pipe is only cached while its load is still current, closing the unload race the model cache already handled. |
||
|
pre-commit-ci[bot]
|
919661ddf0 |
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci |
||
|
Daniel Han
|
62cae5fe71 |
Load image pipelines from the prefetched snapshot instead of re-sweeping the hub
The prefetch already scopes the file list (no packaged root singles, no dtype-variant twins, no ONNX/Flax exports), but from_pretrained was then called with the hub id, and its own snapshot sweep re-downloaded the skipped files anyway: 24 GB per FLUX.1 repo and 65 GB on FLUX.2-dev, as found in the blob cache. Return the snapshot dir from the prefetch (keyed on the pipeline manifest) and hand it to every pipeline-assembly from_pretrained site; any prefetch failure keeps the hub id and the old behavior. |
||
|
pre-commit-ci[bot]
|
f6f198fd5f |
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci |
||
|
Daniel Han
|
25d9cf9604 | Stub diffusers.hooks too in the no-diffusers cache test | ||
|
Daniel Han
|
e605075508 | Fix diffusion training validation and honor lr_scheduler and batch size in the DiT trainer | ||
|
Daniel Han
|
098809d2fe | Reject sd-cli batch runs and clear stale output targets before a run | ||
|
Daniel Han
|
76eee534ea | Gate explicit attention kernels on NVIDIA CUDA and roll back partial FBCache hooks | ||
|
Daniel Han
|
f1d9c88606 | Validate load modes before eviction and wait out a cancelled denoise on unload | ||
|
Daniel Han
|
6a8b0b47e7 |
Fix review findings on image generation: failed-load VRAM, API defaults, preflights
- Free reserved VRAM in the diffusion load worker's failure path: a load-time OOM never commits _state and the next load's _unload_locked early-returns, so nothing else reclaimed the half-built pipeline's memory - Use a monotonic clock for the denoise ETA rate - Sync _GENERATION_DEFAULTS with the UI table: kontext, flux.2-dev, sdxl-turbo and SDXL base rows so /v1/images/generations stops falling back to 9 steps / CFG 0 - 400 (not sanitized 500) when /v1/images/generations hits an edit-only model - Fail fast on pre-Ampere CUDA in the DiT trainer instead of dying in model load - Run the trainer trust gate in the diffusion training route before freeing GPU residents so an untrusted base cannot tear down loaded chat/Images models - Protect native sd.cpp companion VAE/text-encoder repos from cache deletion while a load is downloading them - Exempt the task-scoped Images picker from the chat-only GGUF/MLX format gate so local diffusers pipelines stay selectable on no-GPU hosts |
||
|
Daniel Han
|
c241886c67 | Merge branch 'image-generation' of https://github.com/unslothai/unsloth into image-generation | ||
|
Daniel Han
|
24de50062c |
Enable conv-direct in the default native speed profile
Measured on the fresh linux x64 prebuilt (z-image Q8_0, sd-cli, 192 CPU threads, 512x512, 9 steps, steady state): sampling 56.1s vs 51.3s (about 9 percent faster), VAE decode unchanged, peak RSS identical. The sd.cpp engine only serves the no-GPU tier, so the default profile now matches max: --diffusion-fa plus --diffusion-conv-direct. |
||
|
pre-commit-ci[bot]
|
ab56d81935 |
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci |
||
|
Daniel Han
|
7bf80f6a4e |
Deny fp8/mxfp8/nvfp4 dense quant for the Qwen DiT (black frames, measured)
A 28-pair accuracy gate on a B200 (same-seed vs the dense bf16 reference) found per-row fp8 dynamic quant renders EVERY qwen-image frame black (mean luma 0.0000, SSIM 0.016), reproduced identically with on-the-fly quantize_ on the dense transformer, so it is the model's activation range, not a checkpoint artifact. mxfp8 shows real semantic damage at 1024px (CLIP delta mean 0.0146, worst cases 0.064/0.102) and nvfp4 measures LPIPS mean 0.51. int8 dynamic (per-token scales) is excellent on Qwen: LPIPS mean 0.069, SSIM 0.958. The per-scheme smoke probe only proves the GEMM kernel runs, so it cannot catch model-level breakage. Add _FAMILY_SCHEME_DENY consulted by select_transformer_quant_scheme: auto skips denied schemes (Qwen lands on int8) and an explicit denied request returns None, the same GGUF-fallback contract as an unsupported scheme. Family is threaded from the three diffusion.py call sites; existing behavior is unchanged for every other family. 4 new tests; 529 diffusion tests green; CI-sim green. |
||
|
Daniel Han
|
d146209f88 |
Rename the GGUF compute control to Dtype and simplify the empty-state copy
The always-visible description under the select is gone (the hint tooltip keeps the full detail) and the no-model gallery placeholder now reads 'Select a diffusion model to load'. |
||
|
pre-commit-ci[bot]
|
e6d775d0fc |
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci |
||
|
Daniel Han
|
1d3aa53d1f |
Validate the training config before importing diffusers and pin the arbiter test's device
The fp16-on-bf16-family refusal in run_dit_lora_training now fires before the heavy imports, so a host without diffusers gets the real validation error instead of ModuleNotFoundError. test_in_progress_returns_409_after_validation_passes pins the resolved device to cuda because the load route only takes the GPU arbiter for non-CPU loads, which made the ownership assert host-dependent. |
||
|
Daniel Han
|
630689032e | Merge branch 'image-generation' of https://github.com/unslothai/unsloth into image-generation | ||
|
Daniel Han
|
8ad8a58742 |
Add grad norm chart, clearer completion state, Windows caption keys, GGUF compute copy
- Trainers emit the pre-clip gradient norm; the service keeps a bounded grad_norm history and the Train tab renders a Grad Norm chart next to Loss and LR - Completed runs show 'Training complete' with a celebratory marker in the success color instead of a plain status word - metadata.jsonl caption keys now match on Windows (as_posix relative paths) in both the trainer discovery and the dataset image records - RMSNorm eager patch skips installation on torch builds without F.rms_norm instead of failing at forward time - GGUF compute description no longer says the GGUF is dequantised: the INT8/FP8/FP4 modes load the base model's bf16 transformer and quantise that directly; label no longer wraps in the Advanced panel |
||
|
pre-commit-ci[bot]
|
07f27b23e8 |
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci |
||
|
Daniel Han
|
6f9d8b356c | Merge branch 'image-generation' of https://github.com/unslothai/unsloth into image-generation | ||
|
Daniel Han
|
bfbb902610 |
Fix review findings: engine unload before training, caption precedence, dataset caption counts, local diffusers tagging, family LoRA targets
- Unload the ACTIVE image engine (sd_cpp or diffusers) before diffusion training starts, not just the diffusers singleton - Count metadata.jsonl captions in dataset summaries so metadata-captioned datasets are not reported as uncaptioned - Sidecar captions now override metadata rows everywhere (grid edits win); trainer and dataset API agree - Tag local diffusers image checkpoints with text-to-image so they appear in the Images picker - Family LoRA targets (_FLUX_TARGETS etc) apply when the config carries the generic defaults; explicit overrides still win |
||
|
pre-commit-ci[bot]
|
bbca561d38 |
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci |
||
|
Daniel Han
|
71ab68fd55 |
Merge remote-tracking branch 'origin/main' into image-generation
# Conflicts: # studio/frontend/src/app/router.tsx |
||
oobabooga
|
87ea2ca231 | Drop accidentally committed test files | ||
|
Daniel Han
|
17e55db92b
|
Images page Train tab: multi-family training UI, loss charts, labeling, deploy (#6822)
* Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler Address review feedback on #6724: - engine router: unload the engine being deactivated on a switch, so the old model is not left resident-but-unreachable (the evictor only targets the active engine). - generate route: sd.cpp execution errors (nonzero exit / timeout / missing output) now map to 500, not 409 (which only means not-loaded / cancelled). - native batch: return per-image seeds and persist the actual seed for each image so every batch image is reproducible. - Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the stable-diffusion.cpp docs; other families keep sd-cli defaults. - honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload offload flags on the native load instead of hardcoding them off. - fail the load when the sd-cli binary is present but not runnable (version() now returns None on exec error / nonzero exit). - size estimate: only treat the transformer asset as a possible local path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in load_prequantized_transformer ends in torch.load(weights_only=False), which executes arbitrary code from the pickle. The transformer_prequant_path load-request field reached that unpickle for any local file an authenticated caller named, so a request could trigger remote code execution. Refuse the source.kind=='path' branch unless the operator sets UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted and unaffected. Document the requirement on the API field and add gate tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa - apply_attention_backend now restores the native default when no backend is requested or a kernel fails. diffusers keeps a process-wide active attention backend that set_attention_backend updates, and a fresh transformer's processors follow it, so a load that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile load pinned, breaking the bit-identical/off guarantee. - select_attention_backend drops flash3/flash4 up front when the CUDA capability is below Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit request on the wrong card set fine then crashed mid-generation; it now falls back to native. - Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a valid alias of native) is accepted instead of 422-rejected by Pydantic. - Drop the dead replace('-','_') normalization (no alias uses dashes/underscores). - perf_levers_probe.py output dir is now relative to the script, not a hardcoded path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF - apply_step_cache now engages only via the transformer's native enable_cache (the diffusers CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model now runs uncached per the best-effort contract. - GGUF transformers are quantized (the default Studio load path), so they now use the higher quantized FBCache threshold when the caller leaves it unset, instead of the dense default that could keep the cache from triggering. - fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so fullgraph=True failed warmup and silently measured an eager cached run); output dir is now relative to the script, not a hardcoded path. * Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder _is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate. Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0). * Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just TypeError on older torchao), which would break the mxfp8 config helper instead of falling back to the default. Catch both so the fallback is robust. quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds CUDA memory during the per-row VRAM probe; output dir relative to the script. * Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load - snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the whole snapshot. restore_backend_flags restores each flag independently so one failure can't leave the others leaked process-wide. - load_pipeline restores the flags (and clears the GPU cache) when the build fails after apply_speed_optims mutated the process-wide flags but before _state captured them for unload to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and contaminated later off generations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs without producing output (or without closing stdout) would never reach proc.wait and the wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the PROCESS, so the main thread always enforces the timeout and kills a hung process (which closes the pipe and ends the reader). Add a test that times out even when stdout blocks, and make the no-binary test hermetic so a host-installed sd-cli can't leak in. * Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn The filter callback can be invoked without a module name, so fqn.lower() would raise AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens, so the linear is kept) instead of crashing the quantise pass. * Studio diffusion (Phase 16) review fixes: native engine robustness - sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64); a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds aliased to the same image. Random seeds stay 53-bit (JS-safe). - sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub fall back to anonymous instead of failing auth on a blank token. - sd_cpp_backend: a superseding load now cancels the in-flight generation, so the old sd-cli can no longer return/persist an image from the previous model. - diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a slow 10+ GB free / CUDA sync does not block engine selection. - diffusion_engine_router: probe sd-cli runnability (version()) before committing to native, so a present-but-unrunnable binary falls back to diffusers at selection. - diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a CPU-only install can still reach the native sd.cpp engine instead of failing load. - tests updated for the runnability probe + a not-runnable fallback case. * Studio diffusion (Phase 9) review fixes: prequant safety + validation - SECURITY: a request-supplied local pre-quant path is now unpickled only when it resolves inside an operator-configured ALLOWLIST of directories (UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in, once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on any path a load request named (arbitrary code execution). realpath() blocks symlink escapes; a bare on/off toggle is no longer a wildcard. - Validate the checkpoint's min_features against the runtime Linear filter, so a checkpoint that quantised a different layer set is rejected instead of silently loading a model that mismatches the dense path while reporting the same scheme. - Tolerant base_model_id compare (exact or same final path/repo segment), so a local path or fork of the canonical base is accepted instead of falling back to dense. - _has_meta_tensors uses any(chain(...)) (no intermediate lists). - prequant verify/probe scripts use repo-relative paths (+ env overrides), not the author's absolute /mnt paths. - tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts - diffusion_memory: when group offload is unavailable and the plan falls back to whole-module offload, enable VAE tiling (the group plan left it off, but the fallback is the low-VRAM path where the decode spike can OOM). Covers both the group and sequential fallback branches. - perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a balanced bit-identity regression actually fails the check instead of exiting 0. - compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and the load-progress poll has a 30 min deadline instead of looping forever on a hang. - test for the group->model fallback enabling VAE tiling. * Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path - diffusion: a torchao-quantized transformer is committed only compiled. A dense model resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF it replaced), so when transformer_quant engaged and speed resolved to off, promote to default (regional compile); warn loudly if compile still does not engage. - diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK); otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF. - nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path). - test asserts the eager-quant -> default-compile promotion. * Studio diffusion (Phase 10) review fixes: attention gating + probe isolation - diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation, so auto now stays on native SDPA there. - diffusion_attention: _active_attention_backend handles get_active_backend() returning an enum/None (not a tuple); the old unpack always raised and was swallowed, so the native-restore short-circuit never fired. - perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul so the inductor_flags variant doesn't leak into later compiled rows. - tests for the SM80 cuDNN gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening - install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket); extract through a per-member containment check (Zip-Slip guard); expanduser the --install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch the separately-published cudart runtime DLL archive so sd-cli.exe can start. - sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie. - tests: Zip-Slip rejection, normal extraction, studio-home discovery. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of the literal --output path. SdCppEngine.generate checked only the literal path, so a batch generation would exit 0 and then raise 'no image' (or return a stale file). generate now returns the literal path when present and otherwise falls back to the numbered siblings; single-image behavior is unchanged. Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected without error. * Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats Codex review on the native engine arg builder: - build_sd_cpp_command emitted --width/--height unconditionally, so an img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of the input. width/height are now Optional (None = unset): an image-conditioned run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives the size from the input image (set_width_and_height_if_unset); a plain txt2img run with unset dims keeps the prior 1024x1024 default; explicit dims are always honored. width/height are read only by the builder, so the type change is local. - build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...) that silently swallowed repeats=0 into sd-cli's default of one pass, turning an explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError and emits the flag for any explicit value != 1. Tests: img2img unset dims omit width/height (init_img and ref_images), explicit dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at the default. (Two pre-existing binary-discovery tests fail only because a real sd-cli is installed in this dev environment; unrelated to this change.) * Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc Codex review: the transformer_prequant_path field description still told operators to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the prior security fix made that variable a directory allowlist -- _allowed_prequant_roots deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator following the documented =1 would have every transformer_prequant_path request silently refused. The description now states it must name one or more allowlisted directories and that a bare on/off value is not accepted. Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does not say =1, and describes an allowlist/directory (guards against doc drift). * Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset Codex review on attention-backend selection: - Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first generation with no fallback. select_attention_backend now applies _cudnn_attention_supported() to an explicit cuDNN request too. - flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3 is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a (min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+. - apply_attention_backend's success path left diffusers' process-wide active backend pinned to the kernel it set; a later component whose processors are unconfigured (backend None) would inherit it. It now resets the global registry to native after a successful per-transformer set (the transformer keeps its own backend), best-effort. Also fixed _active_attention_backend: get_active_backend() returns a (name, fn) tuple, so the prior code stringified the tuple and never matched a name, defeating the native-restore short-circuit. Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on SM90; global registry reset after a successful set; _active_attention_backend reads the tuple return. * Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300 (Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So _is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they are treated as data-center class and keep the intended fp8-first behavior. Test: extends the datacenter parametrize with 'NVIDIA B300' and 'NVIDIA GH200 480GB' (now _is_consumer_gpu False). * Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder Codex review: the M=1 modulation/embedder exclusion was wired only into the dense runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint quantised the AdaLN modulation and conditioning-embedder linears, and loading it via transformer_prequant_path (the load path only loads already-quantised tensors, it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes for the runtime path. Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 -> the M=1 exclusion, every other scheme -> none) and use it in both the runtime quantiser and the builder, so a prequant artifact's quantised-layer set always matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion). Test: int8_exclude_name_tokens returns the exclusion for int8 and () for fp8/nvfp4/mxfp8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race Codex review on the native-engine routing: - The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat) unconditionally after engine selection. A native sd.cpp load on a pure-CPU host never touches the GPU, so that needlessly tore down the resident chat model. The handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it. - sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when _run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so the setting was unverifiable. status now derives them from state.offload_flags (still 'none' on CPU, where the flags are empty). - _run_load committed the new state without cancelling/waiting on a generation that started during the (slow) asset download, so a stale sd-cli run against the OLD model could finish afterward and persist an image from the previous model once the new load reported ready. The commit now signals the in-flight cancel and waits on _generate_lock before swapping _state (taken only at commit, so the download never serialises against generation), mirroring the diffusers load path. Tests: CPU native load skips the arbiter while a GPU native load takes it; status reports offload active when flags are set; _run_load cancels and waits for an in-flight generation before committing. * Studio diffusion (Phase 14) review round 2: align helper name with the stack Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the identical helper already present higher in the diffusion stack (Phase 16). The helper definition, the runtime quantiser call, and the offline builder are now byte-identical to that version, so the two branches no longer introduce a divergent name for the same single-source-of-truth and the stack merges without a conflict on this fix. No behavior change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: eager patches + torch.compile cache speed phase Adds the opt-in speed path for the GGUF diffusion transformer behind a selectable speed mode (default off, so output is unchanged until a profile is chosen): - diffusion_eager_patches.py: shared eager fast-paths (channels_last, attention/backend selection, fused norms and QKV) installed at load and rolled back on unload or failed load. - diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent torch.compile cache and the GGUF-transformer compile wiring. - diffusion_arch_patches.py: architecture-specific patches. - diffusion_patch_backend.py: shared install/restore plumbing. - diffusion_speed.py: speed-profile planning. Tests for each module plus the benchmarking and probe scripts used to measure speed, memory, and accuracy of the path. * Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI Backend: - Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file fp8 transformers, gated to the unsloth org plus a curated allowlist. - Image-conditioned workflows built with Pipeline.from_pipe so they reuse the loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint, outpaint, and a hires-fix upscale pass. - Instruction editing as its own family kind (Qwen-Image-Edit-2511, FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and multi-reference) plus klein inpaint. - Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the decoded image size and cap upscale output to avoid OOM on large inputs. - Fixes: from_pipe defaulting to a float32 recast that crashed torchao quantized transformers; image-conditioned calls forcing the slider size onto the input image. Native sd.cpp engine rejects image-conditioned and reference requests it cannot serve. Frontend: - Redesigned Images page with capability-gated workflow tabs (Create, Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor, client-side outpaint, and a multi-reference picker. - Advanced options moved to a right-docked panel mirroring Chat: closed by default, toggled by a single fixed top-bar button that stays in place. sd.cpp installer: pin the release, verify each download's sha256, add a download timeout, and make the source repo configurable for a future mirror. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: correct the Advanced panel comment (closed by default, fixed toggle) * Studio diffusion: LoRA adapters for the Images workflow Add community LoRA support across both diffusion backends, the single biggest step toward broad image-workflow coverage. Backend - New shared module core/inference/diffusion_lora.py: adapter discovery (local scan + curated catalog + owner/name[:file] Hub refs), download via hf_hub_download_with_xet_fallback, alias sanitization, native managed-dir materialization with collision-broken aliases, prompt-tag injection (deduped against user-typed tags), and a supports_lora gate. - Native sd-cli: resolve + materialize selected LoRAs into a per-run managed dir, inject <lora:ALIAS:w> tags, pass --lora-model-dir with --lora-apply-mode auto. The arg builder already emitted these flags. - Diffusers: non-fused load_lora_weights + set_adapters manager, tracked on the pipe so an unchanged selection is a no-op and a model swap resets; cleared on unload. Never fuses (breaks quantized transformers and blocks live weight tweaks). - Gated off where unsupported: torchao fp8/int8 dense, GGUF-via-diffusers, and native Qwen-Image (no LoRA name-conversion branch upstream). - Request contract: optional loras on DiffusionGenerateRequest; empty or omitted is identical to today. supports_lora surfaced in status; chosen LoRAs persisted in gallery recipe metadata. - New GET /api/models/diffusion-loras for the picker (family-filtered). Frontend - Repeatable multi-LoRA picker (adapter select + weight slider 0..2 + remove), gated by the loaded model's supports_lora and family, max 8. Tests - New test_diffusion_lora.py (14): helpers, request validation, native tag/dir wiring, diffusers set_adapters manager, supports_lora matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: ControlNet for the Images workflow (diffusers) Add ControlNet conditioning, the #2 most-used diffusion workflow after LoRA, on the diffusers backend for the families with ControlNet pipelines (FLUX.1 and Qwen-Image), with Union models as the default picks. Backend - New core/inference/diffusion_controlnet.py: family-gated discovery (curated Union models + local dirs + bare owner/name repos), resolution to a loadable repo/dir, control-image preprocessing (passthrough + a dependency-free canny edge map), and a supports_controlnet gate. - diffusion.py: a ControlNet manager parallel to the LoRA one. Loads the (small) ControlNet model once via from_pretrained (cached by id) and builds the family's ControlNet pipeline via Pipeline.from_pipe(base, controlnet=model), reusing the resident base modules at their loaded dtype (no reload, no recast). Passes the control image + conditioning scale + guidance start/end at generate time; cleared on unload. - Families: FLUX.1 -> FluxControlNetPipeline/Model, Qwen-Image -> QwenImageControlNetPipeline/Model. Others declare none (gated off). - Gated off for the native engine, GGUF-via-diffusers, and torchao fp8/int8 dense (same rule as LoRA). v1 conditions txt2img only. - Request contract: optional controlnet on DiffusionGenerateRequest; supports_controlnet in status; the choice persisted in gallery meta. - New GET /api/models/diffusion-controlnets for the picker. Frontend - A ControlNet control in the Images rail (model select + control-image upload + control-type select + strength slider), gated by the loaded model's supports_controlnet + family, shown for text-to-image. Tests - New test_diffusion_controlnet.py (10): discovery/resolve/preprocess/gate helpers, request validation, family wiring, and the diffusers pipe manager (loads once, caches, from_pipe with controlnet, rejects unsupported families). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio ControlNet: show the picker on the Create tab (workflow id is 'create', not 'txt2img') The ControlNet control gated on workflow === 'txt2img', but the Images workflow tab ids are create/transform/inpaint/extend/upscale/reference/edit -- there is no 'txt2img'. So the picker never rendered even with a ControlNet-capable model loaded. Gate on 'create' (the text-to-image tab) for both the picker and the request wiring. Found via a live Playwright capture of the running Studio. * Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a stray "gguf" tag on the Hub but ship no .gguf files. The model search classified them as GGUF from the bare tag, so the picker rendered the GGUF variant expander, which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata (populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected. * Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline load when the id is unsloth-hosted or on-device (the backend infers the family + base repo and gates loads to unsloth/* or local paths), and show a clear message otherwise instead of silently ignoring the click. Curated and GGUF paths are unchanged. * Studio Images: keep curated safetensors models in Recommended after download The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet), so they vanished from the picker after the first load and could only be found by typing an exact search. The row already renders a downloaded badge, matching how GGUF Recommended rows stay visible when cached. Drop the exclusion so the curated safetensors always list. * Studio diffusion LoRA: sanitize dots out of adapter aliases The LoRA alias is used as the diffusers PEFT adapter name, and PEFT rejects names containing "." (module name can't contain "."). sanitize_alias kept dots, so a LoRA whose filename carries a version tag (e.g. Qwen-Image-2512-Lightning-8steps-V1.0-bf16) failed to apply with a 400. Replace dots too; the alias stays a valid native <lora:NAME:w> filename stem. Adds regression coverage for internal dots. * Studio Images: clarify the GGUF transformer-quant Advanced control Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode" with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/ FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF, falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only. * Studio Images: list on-device unsloth diffusion models in the picker The Images picker's On Device tab hid every non-GGUF cached repo whenever a task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit and FP8 safetensors) never showed up there. List cached repos that pass the task gate, limited under a filter to unsloth-hosted ones so base repos (which fail the diffusion load trust gate) don't appear only to dead-end on click. Chat behavior is unchanged: the task gate still drops image repos there. * Studio: hide single-file image checkpoints from the chat model picker The chat picker treats a cached repo as an image model, and hides it, only when it ships a diffusers model_index.json. Single-file, ComfyUI, and ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models. Fall back to resolving the repo id against the known diffusion families, the same resolver the Images backend loads from, so these checkpoints are tagged text-to-image and stay in the Images picker only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: add the FLUX.2-dev model family Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline) resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline + Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated, reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror. text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from klein's turbo defaults. Verified live: GGUF load resolves the family + gated base repo and generates a real 1024x1024 image on GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: clearer error for an unsupported diffusion model When a repo id resolves to no diffusion family the load raised 'Could not infer a diffusion family... Pass family_override (z-image)', which points at an unrelated family and doesn't say what is supported. Replace it with a message that lists the supported families (from a new supported_family_names helper) and notes that video models and image models whose diffusers transformer has no single-file loader are not supported. Applies to both the diffusers and native sd.cpp load paths. Also refreshes two stale family-registry comments that still called FLUX.2-dev omitted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add SDXL diffusion family (U-Net pipeline support) SDXL is the first U-Net family in the diffusion backend: its denoiser is pipe.unet (UNet2DConditionModel), not a DiT pipe.transformer, and a single-file .safetensors is the whole pipeline rather than a transformer-only file. The backend previously assumed a DiT transformer everywhere, so add the two hooks a U-Net family needs and register SDXL. DiffusionFamily gains denoiser_attr ("transformer" for DiT, "unet" for SDXL) and single_file_is_pipeline (SDXL loads a single file via pipeline_class.from_single_file with the base repo as config, instead of transformer_class.from_single_file plus a companion assembly). _align_vae_dtype now reads the denoiser generically so img2img and inpaint keep the VAE and U-Net dtypes aligned. The non-GGUF trust gate is extended with a short, exact-match, safetensors-only allowlist of official base repos (the SDXL base/refiner and sdxl-turbo), because SDXL ships only as a full pipeline and has no unsloth-hosted GGUF. Local paths stay trusted as before; a random repo, even one that detects as SDXL, is still rejected. The image-conditioned and ControlNet workflows are the standard SDXL pipelines, built around the resident modules via from_pipe like every other family, so SDXL gets txt2img, img2img, inpaint, outpaint, upscale, LoRA and ControlNet. There is no native sd.cpp mapping yet, so the no-GPU route falls back to diffusers. Frontend catalog gains SDXL Base 1.0 and SDXL Turbo entries with SDXL step/guidance defaults (Turbo: few steps, no CFG; base: ~30 steps, real CFG). Tests: new test_diffusion_sdxl.py (family shape, detection, trust allowlist, model kind, U-Net VAE-dtype alignment, LoRA gate) plus loader-branch tests in test_diffusion_backend.py (pipeline-kind from_pretrained, single-file whole-pipeline from_single_file, allowlist accept/reject). Verified live on GPU: sdxl-turbo loads both as a pipeline and as a single file and generates coherent txt2img + img2img. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images: LoRA free-text Hugging Face entry + recipe round-trip The backend has always accepted a bare Hugging Face repo id (owner/name, or owner/name:weight-file.safetensors) as a LoRA, downloading and applying it. But the picker only rendered when the curated catalog had entries, and the catalog is empty, so there was no UI path to apply any LoRA. Show the LoRA section whenever the loaded model supports LoRA, and replace the curated-only dropdown with a text input: type a Hub repo id, or pick a discovered adapter from a datalist of suggestions when the catalog is populated. Also restore LoRAs when loading a recipe. restoreSettings now parses the recipe's "id:weight" strings (splitting on the last colon, since the id itself may contain one for a specific weight file) back into the selection, so replaying a saved image reproduces its adapters. The generate payload trims hand-typed ids and drops empty / zero-weight rows, and a model swap clears the selection (a LoRA is family-specific) without discarding a free-text pick that is not in the curated list. * Add diffusion LoRA training (SDXL text-to-image) First diffusion training path in Studio: train a LoRA on the SDXL U-Net from an image + caption dataset and export it as a diffusers .safetensors that the existing diffusion LoRA loader (and any diffusers pipeline) can load. core/training/diffusion_lora_trainer.py: - DiffusionLoraConfig with validation/defaults (rank, alpha, targets, lr, steps, grad accumulation, resolution, min-SNR gamma, gradient checkpointing, lr scheduler, seed, mixed precision). - discover_image_caption_pairs: captions from metadata.jsonl / captions.jsonl, per-image .txt/.caption sidecars, or a dreambooth instance_prompt fallback (pure, unit-tested). - run_diffusion_lora_training: the loop -- freeze base, PEFT-wrap the U-Net attention projections, VAE-encode (fp32 VAE to avoid the SDXL fp16 overflow), sample noise + timesteps, predict, MSE loss with optional min-SNR weighting (epsilon / v-prediction), AdamW + get_scheduler + grad accumulation + grad clipping, then export via save_lora_weights. Emits worker-protocol events (model_load_*, progress, complete) and polls should_stop for a clean stop with a partial save. - run_diffusion_training_process: mp.Queue subprocess adapter (event_queue / stop_queue), so the training worker can spawn it; plus a CLI entry point. Only SDXL (U-Net) is trained here; DiT families and the Studio UI form + route wiring are follow-ups. The trainer is decoupled and worker-ready. Tests: test_diffusion_lora_trainer.py covers caption discovery (metadata / sidecar / instance prompt / skip-uncaptioned / errors), config normalisation + validation, the SDXL add-time-ids, and the dict->config adapter. Verified live on GPU: a 60-step SDXL LoRA run lowers the loss, exports a ~45 MB adapter, and loading it back shifts generation from baseline (mean abs pixel diff ~55/255). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * diffusion trainer: emit learning_rate in progress events (Studio pump compatibility) The Studio training pump reads 'learning_rate' from progress events; the diffusion trainer emitted 'lr'. Rename the field (and the CLI reader) so the trainer's events are directly consumable by the existing training status/SSE machinery when it is wired into the worker, without a translation shim. * Wire diffusion LoRA training into the Studio API Make the SDXL LoRA trainer reachable from the app with a small, self-contained job service and JSON routes, deliberately separate from the LLM TrainingBackend (whose lifecycle -- LLM config build, per-run SQLite rows, matplotlib plots, transfer-to-chat- inference -- is text-training specific and would mis-handle a diffusion run). core/training/diffusion_training_service.py: DiffusionTrainingService runs one job at a time -- validate the config cheaply (before any spawn), spawn the trainer subprocess (spawn context, parent-lifetime bound), pump its events (model_load_* / progress / complete / error) into an in-memory status snapshot, and support a clean stop. The subprocess context and target are injectable so the full start -> pump -> status -> complete path is unit-tested without real multiprocessing or torch. routes/training.py: POST /api/train/diffusion/start (400 on a bad config, 409 when a job is already running), POST /api/train/diffusion/stop, GET /api/train/diffusion/status (JSON poll). models/training.py: DiffusionTrainingStartRequest + response schemas mirroring DiffusionLoraConfig, so model_dump() passes straight through. Tests: test_diffusion_training.py -- service happy path, bad-config-before-spawn, concurrent-job rejection, clean stop, crash-without-terminal-event, event transitions; plus route wiring via the FastAPI TestClient (start / 422 / 400 / 409 / status / stop) with a mocked service. The diffusion trainer's progress events already use the field names this path expects. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images: add a Train LoRA (SDXL) dialog Surface the diffusion training API in the Images page. A "Train LoRA" button in the top bar opens a self-contained dialog to fine-tune an SDXL LoRA on a folder of images: pick the base model, dataset folder, output folder, an optional instance prompt, and the core hyperparameters (steps, rank, resolution, batch, learning rate), then Start. The dialog polls the training status while open and shows a progress bar, step count, live loss, and the saved adapter path, with a Stop button for a clean stop. The dialog is independent of the loaded generation model (training runs in its own subprocess), and prefills the base model with the loaded checkpoint when it is SDXL, else the SDXL base. api.ts gains startDiffusionTraining / stopDiffusionTraining / getDiffusionTrainingStatus plus their types, matching the /api/train/diffusion routes. * Import diffusion training schemas from models.training directly The import-hoist lint flags newly re-exported names in the models/__init__.py hub as unused (it does not treat __all__ membership as a use). Import the three diffusion training schemas straight from models.training in routes/training.py, where they are used in the route annotations and calls, and drop the __init__ re-export. * Remove stray async task scratch outputs committed by mistake * ControlNet: reject filesystem-like ids and do not cache a model past an unload race Two review findings on the ControlNet path: - resolve_controlnet's bare-repo fallback accepted any id with a slash, so a path-shaped id (/tmp/x, ../x) reached from_pretrained as a local directory. Restrict the fallback to a strict owner/name HF repo id shape. - _controlnet_pipe now re-checks the cancel event after the blocking from_pretrained: an unload that raced the download had already cleared the caches, so caching the late module would pin it past the unload. * Pipeline prefetch: fetch only the default torch weights A full-pipeline prefetch kept every repo file outside assets/, so an official repo that ships multiple formats (SDXL Base: fp16 variants, ONNX, OpenVINO, Flax, a top-level single-file twin) downloaded tens of GB from_pretrained never loads. Skip non-torch exports and dtype-variant twins in _pipeline_file_downloaded, and drop a component .bin when the same directory carries a picked safetensors weight (diffusers' own preference). * Diffusion LoRA training: fall back to fp16 when CUDA lacks bf16 The default mixed_precision=bf16 hard-fails on pre-Ampere GPUs (T4 / V100 / RTX 20xx) which have no bf16 compute; check torch.cuda.is_bf16_supported() and drop to fp16 there. * Diffusion training service: join the old pump outside the lock start() joined a finished job's pump thread while holding the service lock, but the pump's final state writes need that same lock, so the join always burned its full timeout and a stale pump could then overwrite the new job's state. Join outside the lock (with a re-check after), and fence _apply_event and the exit handler by process identity so a superseded pump can never touch the current job's state. Adds regression tests for both. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA training: harden config handling, cancellation, SDXL conditioning, and safety Addresses review findings on the SDXL LoRA trainer: - Gate the base model with the same trust check as inference (unsloth/*, allowlisted official bases, or a local path) before from_pretrained, so an untrusted remote repo is never fetched or deserialised. - Check the stop signal before the (slow) model load, not only between steps, so a cancel during download is honoured; a stop may carry save=False to cancel without leaving a partial adapter. - Per-sample SDXL add_time_ids from the actual crop (original size + crop offset, with the offset mirrored on horizontal flip) instead of a fixed uncropped-square tensor. - Apply EXIF orientation before resize/crop so rotated photos train upright. - Skip gradient clipping when max_grad_norm <= 0 (the Studio 'disable' value) instead of scaling every gradient to zero. - Coerce Studio config strings/blanks: learning_rate string to float, blank hf_token to anonymous, gradient_checkpointing 'none'/'true'/'unsloth' to bool; reject a zero/negative lora_alpha or learning_rate. - Alias the generic Studio training payload keys (model_name/max_steps/batch_size/lora_r/ lr_scheduler_type/random_seed) onto the diffusion field names. - Mirror the trained adapter into loras/diffusion so the Images LoRA picker discovers it. - Report worker exceptions in both message and error keys so the failure is not lost. Adds regression tests for the config coercion/validation and aliasing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ControlNet: address review findings on the diffusers path - resolve_controlnet enforces catalog family compatibility so a direct API call cannot load a ControlNet built for another family through the wrong pipeline. - Unknown ControlNet ids now surface as a 400 (call site maps FileNotFoundError to ValueError) instead of a generic 500. - strength 0 disables ControlNet entirely, so a no-op selection never pays the download / VRAM cost; the control image is decoded and validated BEFORE the ControlNet is resolved or built, so a malformed image fails fast for the same reason. - ControlNet loads use the base compute dtype (state.dtype is a display string, not a torch.dtype, so it silently fell back to float32) and honor the base offload policy via group offloading instead of forcing the module resident. - Empty/malformed HF token coerced to anonymous access. - Flux Union ControlNet control_mode mapped from the selected control type. - resolve_controlnet drops the unused hf_token/cancel_event params. - ControlNetSpec validates guidance_start <= guidance_end (clean 422). - Images UI ControlNet Select shows its placeholder when nothing is selected. Adds regression tests for family enforcement and the union control-mode map. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion training API: LLM interlock, pre-spawn VRAM free, path containment, no dropped knobs Four review findings on the diffusion training start path: - It spawned the SDXL trainer without checking the LLM TrainingBackend, so a start while an LLM run was active put two trainers on the same GPU. Add a symmetric interlock: diffusion start returns 409 when LLM training is active, and LLM start refuses while a diffusion job is active. - It went straight to service.start() without freeing GPU residents. Add a pre-spawn free of the export subprocess, the resident Images pipeline (with an arbiter release), and chat models, mirroring the LLM start path. - data_dir / output_dir were passed through unresolved, so Studio-relative names failed and absolute paths bypassed containment. Resolve them with resolve_dataset_path / resolve_output_dir before spawn (400 on an uncontained path). - The request model dropped max_grad_norm and lora_target_modules, so runs that set them trained with defaults. Add both fields. The gemini pump-join deadlock was already fixed earlier (join outside the lock + proc-identity fence). Note: honoring a stop DURING model load is a trainer-loop change owned by the diffusion training engine PR (should_stop polled before the first optimizer step). Adds route + model regression tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA: harden resolution, native tag precedence, and diffusers teardown Address review findings on the LoRA path: - resolve_one: normalise a blank/whitespace hf_token to None (anonymous access) and reject a client-supplied weight file with traversal / absolute path. - resolve_specs: convert FileNotFoundError from an unknown/stale id to ValueError so the route returns 400 instead of a generic 500. - _scan_local: disambiguate local adapters that share a stem (foo.safetensors vs foo.gguf) so each is uniquely addressable. - inject_prompt_tags: the backend-validated weight now wins over a user-typed <lora:ALIAS:...> for a selected adapter; unselected user tags are left alone. - diffusers _apply_loras: reject a .gguf adapter with a clear error before touching the pipe (diffusers loads safetensors only). - _unload_locked: drop the explicit unload_lora_weights() on teardown; the pipe is dropped wholesale (freeing adapters), so the previous call could race an in-flight denoise on the same pipe. - Images page: use a stable LoRA key and clear the selection (not just the options) when the catalog refresh fails. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion: guard trust check against OSError and validate conditioning inputs - _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid characters (or a bare owner/name id) can't raise OSError; treat any failure as not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request still raises the clear FileNotFoundError for a genuinely missing local pick. - generate(): reject mask_image / upscale / reference_images supplied without an input image, and reject reference_images on a family that does not support reference conditioning, instead of silently degrading to txt2img / img2img. * SDXL: reject GGUF up front, skip unused base weights, drop refiner, and harden helpers Addresses review findings on the SDXL family: - Reject a GGUF load for single_file_is_pipeline families (SDXL) in validate_load_request, before the route evicts the current model; SDXL has no transformer-only GGUF variant. - Skip base-repo weight files when a whole-pipeline single file is loaded: from_single_file (config=base) needs only the base config/tokenizer/scheduler, so a local .safetensors no longer triggers a multi-GB base download. - Remove the SDXL refiner from the non-GGUF trust allowlist: it is an img2img-only pipeline but this backend loads every sdxl repo as the base txt2img pipeline. - Normalize a blank/whitespace hf_token to None once in load_pipeline so every load branch degrades to anonymous instead of erroring on a malformed token. - Read the denoiser dtype from a parameter (compile-wrapped modules may lack .dtype) and access state.family.denoiser_attr directly. Adds/updates regression tests for the trust allowlist, GGUF rejection, and base-config filter. * Images: preserve restored LoRAs through model load and never send hidden LoRAs - The LoRA effect cleared the selection on every load->capable transition, which wiped adapters restored from a gallery recipe before the model finished loading. Track the previously-loaded family in a ref and clear only on a real family swap; keep the selection on the initial load and on unload. - Gate the generate payload's loras on loraCapable so a restored selection that is hidden (loaded model does not support LoRA) is never sent to the backend. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images Train LoRA dialog: token, validation, precision, base-repo prefill, gating, refresh Nine review findings on the SDXL training dialog: - Forward the saved Hub token so a gated/private SDXL base can be trained (the image load flow already sends it). - Re-seed the base-model field from the current default each time the dialog opens; the keep-alive dialog otherwise kept its mount-time default after a model loaded. - Prefill from base_repo (the diffusers pipeline) rather than repo_id, which for a GGUF/single-file SDXL load is the checkpoint path from_pretrained can't open. - Add client-side validation of steps/rank/resolution/batch/learning-rate before the request. - Expose a precision selector (bf16/fp16/fp32) so non-bf16 GPUs can train from the UI, not only the API. - Gate the dialog on the active Images route (active && trainOpen) so switching tabs closes it and stops its polling. - Rescan the LoRA picker when a run completes, so a freshly-trained adapter appears without a model reload. - Cap the dialog height and scroll the body so the Start/Stop footer stays reachable on short viewports. - Correct the copy to not over-promise picker auto-discovery. Freeing the resident Images pipeline before training is handled backend-side in the diffusion training start route. * Merge diffusion-sdxl into diffusion-lora-ux; keep options-only LoRA catch The catalog-refresh .catch from the lower branch clears the selected adapters too, which is right for its catalog-only picker but wrong here: this picker holds free-text HF repo ids that are valid without being in the catalog, so a transient refresh failure must not wipe them. Family swaps still clear the selection and hidden LoRAs are never sent. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Train LoRA dialog: stop suggesting absolute paths the backend rejects The dataset and output placeholders showed /path/to/... examples, but the training routes resolve those fields inside the Studio home and reject absolute paths outside the approved roots, so following the placeholder produced a 400. Use folder-name placeholders and say in the labels and the dialog description where each folder resolves. * Align the VAE to the denoiser's first FLOATING dtype, not its first parameter A GGUF-quantized transformer's leading parameters are packed uint8 storage, so reading next(parameters()).dtype handed nn.Module.to() an integer dtype and every image-conditioned generation on a GGUF model (Qwen-Image-Edit) failed with a 500. Probe the parameters for the first floating dtype, treat an all-integer module as a no-op, and also catch TypeError so an unexpected dtype can never break generation. Regression test included. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Count LR scheduler warmup/decay in optimizer steps, not micro-steps lr_sched.step() runs once per outer optimizer step (after the gradient accumulation inner loop), for train_steps total. The scheduler was configured with num_warmup_steps and num_training_steps multiplied by gradient_accumulation_steps, so with accumulation > 1 a warmup or non-constant schedule stretched past the run and never reached the intended decay. Count both in optimizer steps. * Address Codex review findings on the image-workflows PR Keep diffusion.py importable without torch: the compile/arch patch modules import torch at module level, so import them lazily at their load/unload call sites instead of at module load. This restores the torchless contract so get_diffusion_backend() works on a CPU/native sd.cpp install. Match family reject keywords and aliases as whole path/name segments, not raw substrings, so an unrelated word like edited, edition, or kontextual no longer misroutes or hides a valid base image model, while supported edit families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same segment matching in the picker task filter. Route FLUX.2-dev native guidance through --guidance like the other FLUX families rather than --cfg-scale. Reject native upscale requests that have no input image. Read image header dimensions and reject over-limit inputs before decoding pixels, so a crafted small-payload image cannot spike memory. Reject an upscale that would shrink the source below its input size. Validate the model_kind against the filename extension before the GPU handoff. Estimate a local diffusers pipeline's size from its on-disk weights so auto memory planning does not skip offload and OOM. Report workflows: [txt2img] from the native backend status so the Create tab stays enabled for a loaded native model. Clamp the outpaint canvas to the backend's 4096px decode limit. Adds regression tests for segment matching and kind/extension validation. * Guard inference loads and worker lifetime against diffusion training Teach the chat and image load guards about an active diffusion (SDXL) LoRA job: a chat load is refused (its footprint cannot be fit-checked against the trainer) and an image load is refused outright, mirroring the existing LLM training guards, so a load can no longer allocate GPU memory alongside the trainer and undo the pre-start cleanup. Bind the diffusion trainer subprocess to the parent's lifetime and scrub the native path lease secret from it by running the child through run_without_native_path_secret, matching the inference/export/LLM workers, so a Studio crash or kill no longer leaves the trainer holding the GPU. Reset in_model_load on the complete and error terminal events: a stop or failure during model loading otherwise leaves the status reporting a stale loading indicator after the job has ended. * Harden diffusion LoRA handling on the diffusers and native paths Reject LoRA on a torch.compile'd diffusers transformer (Speed=default/max): diffusers requires the adapter loaded before compilation, so applying one to the already-compiled module fails with adapter-key mismatches. The status gate now hides the picker and generate raises a clear message instead. Convert a cancelled Hub LoRA download (RuntimeError Cancelled) to the diffusion cancellation sentinel in resolve_specs, so an unload/superseding load during resolution maps to a 409 instead of a generic server error. Drop weight-0 LoRA rows before the native support gate so a request carrying only disabled adapters stays a no-op on families where native LoRA is unsupported, matching the diffusers path. Reject duplicate LoRA ids in the request model: both apply paths suffix colliding names, so a repeated id would stack the same adapter past its per-adapter weight bound. Strip all user-typed <lora:...> prompt tags on the native path (only the selected adapters are materialized in the managed lora-model-dir, so an unselected tag can never resolve), and restore saved LoRA selections from a gallery recipe so restore reproduces a LoRA image. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden ControlNet resolve, gallery metadata, and the control-type picker Check cancellation immediately after a ControlNet from_pretrained and before any device placement, so an unload/eviction that raced the download does not allocate several GB onto the GPU after the load was already cleared. Require a loadable weight or shard index (not just config.json) before a local ControlNet folder is advertised, so an interrupted copy is hidden instead of failing deep in from_pretrained as a generic 500. Do not record a strength-0 ControlNet in the gallery recipe: it is treated as disabled and skipped, so the image is unconditioned and the metadata must not claim a ControlNet was applied. Build the control-type picker from the selected ControlNet's advertised control_types instead of a hardcoded passthrough/canny pair, so a union model with a precomputed depth or pose map sends the correct control_mode. * Address further Codex findings on the image-workflows PR - Persist the actual output image size in the gallery recipe instead of the request sliders: Transform/Inpaint/Edit derive the size from the uploaded image, Extend grows the canvas, and Upscale resizes it, so the sliders recorded (and later restored) the wrong dimensions for those workflows. - Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file name) in validate_load_request, so the unloadable pick fails before chat is evicted rather than deep in from_pretrained. - Only publish an image-conditioned from_pipe wrapper to the shared aux cache when the load is still current: from_pipe runs under the generate lock but not the state lock, so an unload racing its construction could otherwise cache a wrapper over torn-down modules that a later load would reuse. - Verify the Windows CUDA runtime archive checksum before extracting it, like the main sd-cli archive, so a corrupt or tampered runtime is rejected rather than extracted next to the binary. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse non-SDXL base models at diffusion training start The trainer only supports the SDXL U-Net, but a FLUX / Qwen-Image / Z-Image repo or a GGUF filename passed as base_model was accepted and then failed minutes later inside StableDiffusionXLPipeline.from_pretrained with an unrelated-looking error. Add a name-based guard in normalized() so known DiT-family names and .gguf checkpoints are rejected up front, which the API start route surfaces as an immediate 400 with a message that says exactly which bases are trainable. Unrecognisable names still pass through so custom local SDXL checkpoints keep working. * Add diffusion dataset upload and training info endpoints Training an image LoRA required knowing the Studio home layout and copying files onto the server by hand, which is the most confusing step of the whole flow. Two small endpoints fix that: - GET /api/train/diffusion/info reports the datasets and outputs roots plus every dataset folder that contains images (with image/caption counts), so the UI can offer a picker instead of a blind free-text path. - POST /api/train/diffusion/dataset uploads images and optional caption .txt / metadata.jsonl files into a named folder under the datasets root, creating it on first use and accumulating on repeat uploads so large sets can arrive in batches. Names are validated to a single path component and files stream to disk under the same per-upload size cap as LLM dataset uploads. The returned name is a valid data_dir for /diffusion/start. * Rework the Train LoRA dialog into a guided SDXL flow The dialog assumed users knew the Studio home layout and that only SDXL is trainable, and hid both facts behind free-text fields. Restructure it around the three real decisions: - Base model is a dropdown of the trainable SDXL picks (Base 1.0, Turbo, the loaded SDXL pipeline when there is one) with a custom repo/path escape hatch, instead of a bare text field defaulting to a repo id. - Training images come from an in-browser upload (new dataset endpoints) or a picker over existing dataset folders with image/caption counts. No shell access or knowledge of the datasets root is needed any more, and the captioning rules are explained inline. - The output field is now Adapter name and the instance prompt is labelled as the trigger prompt, with a no-captions warning wired to the selected dataset's actual caption count. Hyperparameters collapse behind a training settings toggle since the defaults suit a first run. A completed run says where the adapter went and offers Done / Train another, and the top-bar button gets an icon and a plainer description. The dialog title states the SDXL-only scope and that other families load LoRAs but cannot train them yet. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Validate diffusion training config before freeing the GPU The start route freed resident GPU workloads (export, Images pipeline, chat) before the service validated the config, so a start that was then refused, now including a non-SDXL base model, tore down the user's loaded model for nothing. Run the same cheap normalise pass first; the LLM path already follows this rule via its before_spawn hook. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor diffusion LoRA training into a family-aware platform Split the SDXL trainer into a shared, architecture-agnostic layer so more model families can be trained without duplicating the plumbing: - New core/training/diffusion_train_common.py holds the config + validation, dataset discovery, event emission, stop protocol, adapter publishing, and a lazy trainer registry (get_trainer). diffusion_lora_trainer.py keeps the SDXL-specific loop and re-exports the moved names so existing imports are unchanged. - The SDXL-only base-model blocklist becomes a positive check: the family is resolved from the base model (or an explicit model_family) via the diffusion family registry, and a known-but-not-yet-trainable family is refused with a clear message. Unknown custom names still default to the SDXL trainer. - DiffusionFamily gains a trainable flag and train_base_repos; SDXL is marked trainable. DiT families flip on when their trainers land. - Trained adapters now write a <name>.json metadata sidecar (family, base model, rank, trigger prompt, ...) that the LoRA scanner reads to family-gate the adapter in the picker instead of showing it as unknown for every model. - The training base-model trust allowlist adds the official FLUX.1-dev, Z-Image-Turbo, and Qwen-Image repos (safetensors-only, no remote code). * Retain diffusion training loss history and expose it in status The training service kept only the latest loss, so a live loss chart could show a single point. Fold each progress event into bounded (step, loss, lr) history arrays (capped at 4000 points, decimated when full) plus the latest throughput and peak VRAM, and record the family / base model / catalog path on completion. The status endpoint returns these as a nested metric_history object the UI can chart directly, and the start request accepts an optional model_family override. * Tests for the diffusion training platform Cover the trainer registry (get_trainer resolves SDXL, unknown family raises), family resolution (explicit model_family validation, resolved_family on the config), the metadata sidecar write + scan read with family gating, and the service loss-history folding (append, bad-point skipping, decimation at cap, family/perf fields) plus the status route nesting metric_history. * Add diffusion dataset labeling and example-import endpoints The Train tab needs to let users caption small datasets in the browser and pull in a ready-made set to see training work end to end, neither of which the upload-only endpoint supported. Add, under /api/train/diffusion/dataset: - GET {name}/images lists every image with its resolved caption (metadata beats a per-image sidecar, matching the trainer's discovery order) so uncaptioned images are visible and flaggable. - GET {name}/image/{filename} serves an image, with ?thumb=<px> returning a cached downscaled JPEG kept in a hidden .thumbs subdir (regenerated when the source is newer) so the labeling grid stays light. - PUT {name}/caption/{filename} writes, or when blank clears, the .txt sidecar; DELETE {name}/image/{filename} removes the image plus its sidecars and thumbnails. - GET dataset-examples lists a curated, license-labelled registry, and POST dataset/import-example materializes one into a dataset folder as numbered images + .txt captions. Two loaders cover the shapes seen in the wild: streaming rows from datasets.load_dataset (dog-example, Tuxemon) and a snapshot + jsonl walk for imagefolder repos whose captions live in a non-standard *.jsonl (the public-domain tarot set). Imports are idempotent and cap the image count. Filenames and dataset names are validated against path traversal and pinned inside the datasets root. * Test diffusion dataset labeling and example-import endpoints Cover caption precedence, thumbnail generation and .thumbs exclusion, caption write/clear, image delete cleanup, path-traversal rejection on names and filenames, and example import with a mocked datasets.load_dataset (files plus sidecars written, idempotent second call, cap respected, load failure mapped to 502). * Add flow-matching DiT LoRA trainers (FLUX.1-dev, Qwen-Image, Z-Image) Extends diffusion LoRA training beyond SDXL to the three popular DiT families via a single shared flow-matching loop parameterised by small per-family specs (loading, prompt/latent encoding, transformer forward, save). Verified against diffusers 0.38.0: - FLUX.1-dev: 2x2 latent packing + image ids, guidance-embed forward, on-the-fly nf4 QLoRA of the 12B transformer (the dev repo is gated, so training needs the user's HF token). - Qwen-Image: 5D VAE latents normalised by the per-channel latents_mean/std, img_shapes forward, prequant nf4 base by default (on-the-fly nf4 for the bf16 base). - Z-Image: list I/O with the reversed timestep convention and a negated prediction, bf16 only. The registry (get_trainer) and DiffusionFamily.trainable / train_base_repos now route these families to the DiT trainer; the SDXL blocklist guard is replaced by a positive family resolution that also rejects GGUF repos (inference-only) and still-unsupported families. Per-family defaults + labels + VRAM notes are exposed via family_train_infos for the Train UI. Memory: caption embeddings are precomputed once and the text encoders freed before the loop; gradient checkpointing (non-reentrant, required for bnb 4-bit) and 8-bit AdamW are on by default. * Speed up + shrink SDXL LoRA training (precompute text embeds, 8-bit AdamW) SDXL re-encoded every caption with both CLIP text encoders on every step (pure waste, since captions are constant) and kept the encoders resident. Precompute each unique caption's embeddings once, then free the text encoders before the loop: numerically identical (embeddings are deterministic and this consumes no torch RNG, so the noise/timestep stream is unchanged) but faster and ~1.5 GB lighter. Default the optimizer to 8-bit AdamW (bitsandbytes) with an fp32 fallback, halving optimizer state with no meaningful LoRA quality cost. Env toggles (UNSLOTH_DIFFUSION_NO_PRECOMPUTE / _FP32_OPTIM) let the accuracy guard A/B the paths. * Expose trainable families in /diffusion/info and preflight gated bases The training info endpoint now returns the trainable model families (name, label, default + allowed base repos, recommended defaults, and a VRAM/access note) so the Train UI can offer a base picker with realistic guidance. The start route preflights a gated base repo (HEAD model_index.json with the user's token) BEFORE freeing resident GPU workloads, so a missing FLUX.1-dev license/token fails fast with an actionable 400 instead of evicting the loaded model and then hitting a confusing mid-load 401. * Tests for DiT trainers, family resolution, info families, gated preflight Cover the DiT spec table, the QLoRA prequant heuristic, the Z-Image bf16-only guard, the gated-repo name check, family resolution now that FLUX/Qwen/Z-Image are trainable (and GGUF repos are rejected as inference-only), the families list in /diffusion/info, and the gated-base 400 preflight that leaves the GPU untouched. * Add diffusion training API client: metrics, families, dataset labeling, examples Extends the Images training client for the Train tab: the status type now carries metric_history (step/loss/lr) plus catalog_path/family/base_model/samples_per_second/ peak_memory_gb; the start request gains model_family; and info gains an optional families list (per-family bases + defaults). Adds typed calls for the dataset labeling and one-click example endpoints: list images with captions, thumbnail URL, write/clear a caption, delete an image, list example datasets, and import an example. * Add diffusion Train panel: config, dataset labeling, live charts, deploy New full-page training workspace for the Images tab. Left column configures the run: model family (FLUX.1-dev, Qwen-Image, Z-Image, SDXL in popularity order, with per-family VRAM/license notes and defaults, backfilled from the backend families list when present), base repo, dataset (existing folder, browser upload, or one-click example import), an in-browser caption labeling grid (per-image thumbnail + caption saved on blur, delete, uncaptioned highlight), adapter name, trigger prompt, and collapsed training settings. Right column shows the live run: progress + loss/avg/speed/peak-VRAM readouts, the reused training loss/LR charts fed from metric_history, and a completion card that deploys the adapter into Create or starts another run. * Wire Create/Train tab switch into the Images page and deploy flow Replaces the Train LoRA dialog with a top-bar Create | Train segmented control next to the model selector. Create renders the existing generation workspace unchanged; Train renders the full-page training panel (unmounted in Create so its polling stops while the backend run and its retained metric history survive a tab switch). Adds a deploy handler: loading the trained adapter's base as a pipeline, queueing the adapter so the LoRA discovery effect applies it once the base is loaded and LoRA-capable for the matching family (with a mismatch warning), seeding the prompt with the trigger, and switching back to Create. Removes the now-unused dialog. * Wrap the DiT training forward in bf16 autocast The fp32 LoRA parameters and the bnb 4-bit base matmuls need a single compute dtype during the forward, exactly like the diffusers dreambooth scripts run under accelerator.autocast. Without it the 4-bit backward on FLUX.1-dev fails with an illegal-address CUBLAS error partway into the first step. Z-Image and Qwen-Image smokes are unaffected and the SDXL path (its own trainer) is untouched. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix Train tab example cards and Create/Train tab layout The example-dataset cards used a two-column grid in the ~340px config column, which wrapped titles one word per line and let the long license text overrun into the neighbouring card. Switch to one card per row with a horizontal layout: title with a compact truncated license badge (full text in the tooltip), a two-line clamped description, and the Import button on the right. The Create/Train switch had an icon inside the Train trigger that overhung the pill corner. Drop the icon, make both triggers a fixed equal width so the active pill sits flush in the top bar. * Show only loss and learning-rate charts for diffusion training The Train tab reused the LLM charts section, which also rendered an empty Grad Norm card and an Eval Loss card showing an Evaluation not configured placeholder with a red smear. Neither applies to diffusion LoRA training. Add a diffusion-only two-card view that reuses the loss and learning-rate cards directly with fixed presentation defaults, and note under the loss chart that per-step loss is noisy by design so users read the smoothed line for the trend rather than the raw jitter. * Add a dataset preview strip to the Train tab When a dataset with images is selected, show a strip of up to 8 sampled thumbnails with a +N more tile, so users can see what is in the folder before training. Clicking the strip opens the existing caption review grid. Samples are drawn evenly across the folder and refresh on dataset change or after an upload/import. * Stop example cards from overflowing the Train config column The example-dataset cards still overran the ~340px config column: the license used the Badge component whose baked-in w-fit and whitespace-nowrap ignored the max-width and truncate, and the grid children had the default min-width auto so wide content pushed past the column edge and clipped the Import buttons. Replace the badge with a plain truncating pill span, and give the config column min-w-0 with overflow-x-hidden so nothing escapes its width. * Add Smithsonian Butterflies and Nouns example datasets Two permissive ~100-image sets for the Train tab: huggan/smithsonian_butterflies_subset (CC0, the classic diffusers-docs training set, imported as a subject set with a trigger prompt since its metadata columns are species names not captions) and m1guelpf/nouns (CC0, captioned pixel-art avatars via the text column). Both cap at 100 images. * Paginate the Train tab caption grid with prev/next controls Large example datasets (100+ images) rendered every tile at once, so the caption review grid grew unbounded. Show 24 images per page with < > chevrons and an x-y of N indicator; a new dataset or refresh resets to the first page. * Offer example datasets in the Train dropdown with previews Add an Examples group to the training-images dropdown that imports a curated dataset in one pick, alongside the existing cards. Cards now show up to three preview thumbnails pulled from the public HF datasets-server so the set is visible before download. Hide the trigger prompt when every image already has a caption (a captioned style set needs no trigger), and turn the training-settings toggle into a ghost button with a rotating chevron. * Clamp the training base repo to the selected family The base-model select's state could briefly hold the previous family's repo after a family switch (the reseed effect runs a beat later, and a value with no matching option makes the browser display the first option anyway). The request then carried the stale repo: picking Qwen or Z-Image still sent black-forest-labs/FLUX.1-dev and surfaced FLUX's gated-repo error under the wrong family. Derive an effectiveBase clamped to the current family's repos and use it for the select value, the start request, and the deploy fallback. Also move the Trigger prompt above Adapter name: the trigger describes the dataset, the name only labels the output. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
|
Daniel Han
|
78e3cb237f
|
Diffusion dataset labeling and example-import APIs (#6821)
* [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): pre-quantized transformer loading The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs 13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline (scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True) the quantized weights, so the dense bf16 never touches the GPU. Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same torchao config + min_features filter the runtime path uses, applied ahead of time. New core/inference/diffusion_prequant.py (resolve_prequant_source + load_prequantized_transformer, best-effort, lazy imports). diffusion.py _load_dense_quant_pipeline tries the pre-quant source first and falls back to the dense materialise+quantise path, then to GGUF, so the default is unchanged. DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for the resolver, the meta-init+assign loader, and the backend branch selection + fallbacks; GPU verification via scripts/verify_prequant_backend.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): attention-backend selection Add a selectable attention kernel via the diffusers set_attention_backend dispatcher. Attention is memory-bandwidth bound, so a better kernel is an end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV matmuls torchao never touches) and composes with torch.compile. auto picks the best exact backend for the device: cuDNN fused attention (_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default (below the compile/quant noise floor); native SDPA elsewhere and when speed=off (so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/ xformers/aiter are honored, and an unavailable kernel falls back to the default rather than failing the load. New core/inference/diffusion_attention.py (normalize + per-device select + apply, best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline; attention_backend threads through begin_load / load_pipeline / status like the other load knobs. New request field attention_backend + status field. Hermetic CPU tests for normalize / select policy / apply fallback, plus route threading + 422. Measured via scripts/perf_levers_probe.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the arch tier to put int8 first on a consumer/workstation GPU (detected by the existing _is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first. Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation (-> int8) and data-center Ada/Hopper/Blackwell (-> fp8). * Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across denoise steps a DiT's output settles, so once the first block's residual barely changes the remaining blocks are skipped and their cached output reused. diffusers ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the standalone apply_first_block_cache hook as a fallback). Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 -> 2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar. OFF by default and a per-load opt-in: the win scales with step count, so it is for many-step models (Flux / Qwen-Image) and pointless for few-step distilled models (e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction of the trajectory. It composes with regional compile only with fullgraph=False (the cache's per-step decision is a torch.compiler.disable graph break), which the speed layer now switches to automatically when a cache is engaged. Best-effort: a model whose block signature the hook does not recognise is caught and the load proceeds uncached. - new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache (enable_cache / apply_first_block_cache fallback; threshold auto-raised for a quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import). - diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph when a cache is engaged. - diffusion.py: apply_step_cache before compile; thread transformer_cache / transformer_cache_threshold through begin_load -> load_pipeline and report the engaged mode in status(). - models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and transformer_cache_threshold request fields, engaged mode in the status response. - hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold selection, and best-effort failure handling, plus route threading + validation. - scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM / LPIPS vs the compiled no-cache baseline). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears) The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with 'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2 *_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing min_features filter did not exclude them. Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed, guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers (M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and quantises these layers fine, so the exclusion is int8-only. Sequence embedders (context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note 'context_embedder' contains the substring 'text_embed', which is why the token is the specific 'time_text_embed', not 'text_embed'. Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s (5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already working) are unchanged. - diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes exclude_name_tokens; quantize_transformer passes it for int8 only. - hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps attention / FFN / sequence-embedder linears), while fp8 keeps them. - scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's Linear layers and derive the exclusion list. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears) The prequant-checkpoint builder applied the dense quant filter without the int8-only M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8 checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now produces a working checkpoint on every supported model, giving int8 (the consumer-preferred scheme) the same ~2x load-VRAM and download reduction fp8 already had. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the native stable-diffusion.cpp engine instead of diffusers, with diffusers as the guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM. - diffusion_engine_router: centralised engine selection (built on the existing select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason. - sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface backed by sd-cli, with lazy binary install, registry-driven asset fetch, step-progress parsing, and cancellation. - diffusion_families: per-family single-file VAE + text-encoder asset mapping. - sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled). - routes/inference + gpu_arbiter: drive the active engine via the router; the API now reports the active engine and any fallback reason. - tests for the backend, router, route selection, and cancellation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler Address review feedback on #6724: - engine router: unload the engine being deactivated on a switch, so the old model is not left resident-but-unreachable (the evictor only targets the active engine). - generate route: sd.cpp execution errors (nonzero exit / timeout / missing output) now map to 500, not 409 (which only means not-loaded / cancelled). - native batch: return per-image seeds and persist the actual seed for each image so every batch image is reproducible. - Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the stable-diffusion.cpp docs; other families keep sd-cli defaults. - honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload offload flags on the native load instead of hardcoding them off. - fail the load when the sd-cli binary is present but not runnable (version() now returns None on exec error / nonzero exit). - size estimate: only treat the transformer asset as a possible local path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in load_prequantized_transformer ends in torch.load(weights_only=False), which executes arbitrary code from the pickle. The transformer_prequant_path load-request field reached that unpickle for any local file an authenticated caller named, so a request could trigger remote code execution. Refuse the source.kind=='path' branch unless the operator sets UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted and unaffected. Document the requirement on the API field and add gate tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa - apply_attention_backend now restores the native default when no backend is requested or a kernel fails. diffusers keeps a process-wide active attention backend that set_attention_backend updates, and a fresh transformer's processors follow it, so a load that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile load pinned, breaking the bit-identical/off guarantee. - select_attention_backend drops flash3/flash4 up front when the CUDA capability is below Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit request on the wrong card set fine then crashed mid-generation; it now falls back to native. - Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a valid alias of native) is accepted instead of 422-rejected by Pydantic. - Drop the dead replace('-','_') normalization (no alias uses dashes/underscores). - perf_levers_probe.py output dir is now relative to the script, not a hardcoded path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF - apply_step_cache now engages only via the transformer's native enable_cache (the diffusers CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model now runs uncached per the best-effort contract. - GGUF transformers are quantized (the default Studio load path), so they now use the higher quantized FBCache threshold when the caller leaves it unset, instead of the dense default that could keep the cache from triggering. - fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so fullgraph=True failed warmup and silently measured an eager cached run); output dir is now relative to the script, not a hardcoded path. * Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder _is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate. Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0). * Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just TypeError on older torchao), which would break the mxfp8 config helper instead of falling back to the default. Catch both so the fallback is robust. quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds CUDA memory during the per-row VRAM probe; output dir relative to the script. * Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load - snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the whole snapshot. restore_backend_flags restores each flag independently so one failure can't leave the others leaked process-wide. - load_pipeline restores the flags (and clears the GPU cache) when the build fails after apply_speed_optims mutated the process-wide flags but before _state captured them for unload to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and contaminated later off generations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs without producing output (or without closing stdout) would never reach proc.wait and the wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the PROCESS, so the main thread always enforces the timeout and kills a hung process (which closes the pipe and ends the reader). Add a test that times out even when stdout blocks, and make the no-binary test hermetic so a host-installed sd-cli can't leak in. * Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn The filter callback can be invoked without a module name, so fqn.lower() would raise AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens, so the linear is kept) instead of crashing the quantise pass. * Studio diffusion (Phase 16) review fixes: native engine robustness - sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64); a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds aliased to the same image. Random seeds stay 53-bit (JS-safe). - sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub fall back to anonymous instead of failing auth on a blank token. - sd_cpp_backend: a superseding load now cancels the in-flight generation, so the old sd-cli can no longer return/persist an image from the previous model. - diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a slow 10+ GB free / CUDA sync does not block engine selection. - diffusion_engine_router: probe sd-cli runnability (version()) before committing to native, so a present-but-unrunnable binary falls back to diffusers at selection. - diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a CPU-only install can still reach the native sd.cpp engine instead of failing load. - tests updated for the runnability probe + a not-runnable fallback case. * Studio diffusion (Phase 9) review fixes: prequant safety + validation - SECURITY: a request-supplied local pre-quant path is now unpickled only when it resolves inside an operator-configured ALLOWLIST of directories (UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in, once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on any path a load request named (arbitrary code execution). realpath() blocks symlink escapes; a bare on/off toggle is no longer a wildcard. - Validate the checkpoint's min_features against the runtime Linear filter, so a checkpoint that quantised a different layer set is rejected instead of silently loading a model that mismatches the dense path while reporting the same scheme. - Tolerant base_model_id compare (exact or same final path/repo segment), so a local path or fork of the canonical base is accepted instead of falling back to dense. - _has_meta_tensors uses any(chain(...)) (no intermediate lists). - prequant verify/probe scripts use repo-relative paths (+ env overrides), not the author's absolute /mnt paths. - tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts - diffusion_memory: when group offload is unavailable and the plan falls back to whole-module offload, enable VAE tiling (the group plan left it off, but the fallback is the low-VRAM path where the decode spike can OOM). Covers both the group and sequential fallback branches. - perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a balanced bit-identity regression actually fails the check instead of exiting 0. - compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and the load-progress poll has a 30 min deadline instead of looping forever on a hang. - test for the group->model fallback enabling VAE tiling. * Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path - diffusion: a torchao-quantized transformer is committed only compiled. A dense model resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF it replaced), so when transformer_quant engaged and speed resolved to off, promote to default (regional compile); warn loudly if compile still does not engage. - diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK); otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF. - nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path). - test asserts the eager-quant -> default-compile promotion. * Studio diffusion (Phase 10) review fixes: attention gating + probe isolation - diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation, so auto now stays on native SDPA there. - diffusion_attention: _active_attention_backend handles get_active_backend() returning an enum/None (not a tuple); the old unpack always raised and was swallowed, so the native-restore short-circuit never fired. - perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul so the inductor_flags variant doesn't leak into later compiled rows. - tests for the SM80 cuDNN gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening - install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket); extract through a per-member containment check (Zip-Slip guard); expanduser the --install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch the separately-published cudart runtime DLL archive so sd-cli.exe can start. - sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie. - tests: Zip-Slip rejection, normal extraction, studio-home discovery. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of the literal --output path. SdCppEngine.generate checked only the literal path, so a batch generation would exit 0 and then raise 'no image' (or return a stale file). generate now returns the literal path when present and otherwise falls back to the numbered siblings; single-image behavior is unchanged. Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected without error. * Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats Codex review on the native engine arg builder: - build_sd_cpp_command emitted --width/--height unconditionally, so an img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of the input. width/height are now Optional (None = unset): an image-conditioned run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives the size from the input image (set_width_and_height_if_unset); a plain txt2img run with unset dims keeps the prior 1024x1024 default; explicit dims are always honored. width/height are read only by the builder, so the type change is local. - build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...) that silently swallowed repeats=0 into sd-cli's default of one pass, turning an explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError and emits the flag for any explicit value != 1. Tests: img2img unset dims omit width/height (init_img and ref_images), explicit dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at the default. (Two pre-existing binary-discovery tests fail only because a real sd-cli is installed in this dev environment; unrelated to this change.) * Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc Codex review: the transformer_prequant_path field description still told operators to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the prior security fix made that variable a directory allowlist -- _allowed_prequant_roots deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator following the documented =1 would have every transformer_prequant_path request silently refused. The description now states it must name one or more allowlisted directories and that a bare on/off value is not accepted. Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does not say =1, and describes an allowlist/directory (guards against doc drift). * Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset Codex review on attention-backend selection: - Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first generation with no fallback. select_attention_backend now applies _cudnn_attention_supported() to an explicit cuDNN request too. - flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3 is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a (min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+. - apply_attention_backend's success path left diffusers' process-wide active backend pinned to the kernel it set; a later component whose processors are unconfigured (backend None) would inherit it. It now resets the global registry to native after a successful per-transformer set (the transformer keeps its own backend), best-effort. Also fixed _active_attention_backend: get_active_backend() returns a (name, fn) tuple, so the prior code stringified the tuple and never matched a name, defeating the native-restore short-circuit. Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on SM90; global registry reset after a successful set; _active_attention_backend reads the tuple return. * Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300 (Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So _is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they are treated as data-center class and keep the intended fp8-first behavior. Test: extends the datacenter parametrize with 'NVIDIA B300' and 'NVIDIA GH200 480GB' (now _is_consumer_gpu False). * Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder Codex review: the M=1 modulation/embedder exclusion was wired only into the dense runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint quantised the AdaLN modulation and conditioning-embedder linears, and loading it via transformer_prequant_path (the load path only loads already-quantised tensors, it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes for the runtime path. Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 -> the M=1 exclusion, every other scheme -> none) and use it in both the runtime quantiser and the builder, so a prequant artifact's quantised-layer set always matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion). Test: int8_exclude_name_tokens returns the exclusion for int8 and () for fp8/nvfp4/mxfp8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race Codex review on the native-engine routing: - The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat) unconditionally after engine selection. A native sd.cpp load on a pure-CPU host never touches the GPU, so that needlessly tore down the resident chat model. The handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it. - sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when _run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so the setting was unverifiable. status now derives them from state.offload_flags (still 'none' on CPU, where the flags are empty). - _run_load committed the new state without cancelling/waiting on a generation that started during the (slow) asset download, so a stale sd-cli run against the OLD model could finish afterward and persist an image from the previous model once the new load reported ready. The commit now signals the in-flight cancel and waits on _generate_lock before swapping _state (taken only at commit, so the download never serialises against generation), mirroring the diffusers load path. Tests: CPU native load skips the arbiter while a GPU native load takes it; status reports offload active when flags are set; _run_load cancels and waits for an in-flight generation before committing. * Studio diffusion (Phase 14) review round 2: align helper name with the stack Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the identical helper already present higher in the diffusion stack (Phase 16). The helper definition, the runtime quantiser call, and the offline builder are now byte-identical to that version, so the two branches no longer introduce a divergent name for the same single-source-of-truth and the stack merges without a conflict on this fix. No behavior change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: eager patches + torch.compile cache speed phase Adds the opt-in speed path for the GGUF diffusion transformer behind a selectable speed mode (default off, so output is unchanged until a profile is chosen): - diffusion_eager_patches.py: shared eager fast-paths (channels_last, attention/backend selection, fused norms and QKV) installed at load and rolled back on unload or failed load. - diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent torch.compile cache and the GGUF-transformer compile wiring. - diffusion_arch_patches.py: architecture-specific patches. - diffusion_patch_backend.py: shared install/restore plumbing. - diffusion_speed.py: speed-profile planning. Tests for each module plus the benchmarking and probe scripts used to measure speed, memory, and accuracy of the path. * Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI Backend: - Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file fp8 transformers, gated to the unsloth org plus a curated allowlist. - Image-conditioned workflows built with Pipeline.from_pipe so they reuse the loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint, outpaint, and a hires-fix upscale pass. - Instruction editing as its own family kind (Qwen-Image-Edit-2511, FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and multi-reference) plus klein inpaint. - Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the decoded image size and cap upscale output to avoid OOM on large inputs. - Fixes: from_pipe defaulting to a float32 recast that crashed torchao quantized transformers; image-conditioned calls forcing the slider size onto the input image. Native sd.cpp engine rejects image-conditioned and reference requests it cannot serve. Frontend: - Redesigned Images page with capability-gated workflow tabs (Create, Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor, client-side outpaint, and a multi-reference picker. - Advanced options moved to a right-docked panel mirroring Chat: closed by default, toggled by a single fixed top-bar button that stays in place. sd.cpp installer: pin the release, verify each download's sha256, add a download timeout, and make the source repo configurable for a future mirror. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: correct the Advanced panel comment (closed by default, fixed toggle) * Studio diffusion: LoRA adapters for the Images workflow Add community LoRA support across both diffusion backends, the single biggest step toward broad image-workflow coverage. Backend - New shared module core/inference/diffusion_lora.py: adapter discovery (local scan + curated catalog + owner/name[:file] Hub refs), download via hf_hub_download_with_xet_fallback, alias sanitization, native managed-dir materialization with collision-broken aliases, prompt-tag injection (deduped against user-typed tags), and a supports_lora gate. - Native sd-cli: resolve + materialize selected LoRAs into a per-run managed dir, inject <lora:ALIAS:w> tags, pass --lora-model-dir with --lora-apply-mode auto. The arg builder already emitted these flags. - Diffusers: non-fused load_lora_weights + set_adapters manager, tracked on the pipe so an unchanged selection is a no-op and a model swap resets; cleared on unload. Never fuses (breaks quantized transformers and blocks live weight tweaks). - Gated off where unsupported: torchao fp8/int8 dense, GGUF-via-diffusers, and native Qwen-Image (no LoRA name-conversion branch upstream). - Request contract: optional loras on DiffusionGenerateRequest; empty or omitted is identical to today. supports_lora surfaced in status; chosen LoRAs persisted in gallery recipe metadata. - New GET /api/models/diffusion-loras for the picker (family-filtered). Frontend - Repeatable multi-LoRA picker (adapter select + weight slider 0..2 + remove), gated by the loaded model's supports_lora and family, max 8. Tests - New test_diffusion_lora.py (14): helpers, request validation, native tag/dir wiring, diffusers set_adapters manager, supports_lora matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: ControlNet for the Images workflow (diffusers) Add ControlNet conditioning, the #2 most-used diffusion workflow after LoRA, on the diffusers backend for the families with ControlNet pipelines (FLUX.1 and Qwen-Image), with Union models as the default picks. Backend - New core/inference/diffusion_controlnet.py: family-gated discovery (curated Union models + local dirs + bare owner/name repos), resolution to a loadable repo/dir, control-image preprocessing (passthrough + a dependency-free canny edge map), and a supports_controlnet gate. - diffusion.py: a ControlNet manager parallel to the LoRA one. Loads the (small) ControlNet model once via from_pretrained (cached by id) and builds the family's ControlNet pipeline via Pipeline.from_pipe(base, controlnet=model), reusing the resident base modules at their loaded dtype (no reload, no recast). Passes the control image + conditioning scale + guidance start/end at generate time; cleared on unload. - Families: FLUX.1 -> FluxControlNetPipeline/Model, Qwen-Image -> QwenImageControlNetPipeline/Model. Others declare none (gated off). - Gated off for the native engine, GGUF-via-diffusers, and torchao fp8/int8 dense (same rule as LoRA). v1 conditions txt2img only. - Request contract: optional controlnet on DiffusionGenerateRequest; supports_controlnet in status; the choice persisted in gallery meta. - New GET /api/models/diffusion-controlnets for the picker. Frontend - A ControlNet control in the Images rail (model select + control-image upload + control-type select + strength slider), gated by the loaded model's supports_controlnet + family, shown for text-to-image. Tests - New test_diffusion_controlnet.py (10): discovery/resolve/preprocess/gate helpers, request validation, family wiring, and the diffusers pipe manager (loads once, caches, from_pipe with controlnet, rejects unsupported families). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio ControlNet: show the picker on the Create tab (workflow id is 'create', not 'txt2img') The ControlNet control gated on workflow === 'txt2img', but the Images workflow tab ids are create/transform/inpaint/extend/upscale/reference/edit -- there is no 'txt2img'. So the picker never rendered even with a ControlNet-capable model loaded. Gate on 'create' (the text-to-image tab) for both the picker and the request wiring. Found via a live Playwright capture of the running Studio. * Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a stray "gguf" tag on the Hub but ship no .gguf files. The model search classified them as GGUF from the bare tag, so the picker rendered the GGUF variant expander, which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata (populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected. * Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline load when the id is unsloth-hosted or on-device (the backend infers the family + base repo and gates loads to unsloth/* or local paths), and show a clear message otherwise instead of silently ignoring the click. Curated and GGUF paths are unchanged. * Studio Images: keep curated safetensors models in Recommended after download The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet), so they vanished from the picker after the first load and could only be found by typing an exact search. The row already renders a downloaded badge, matching how GGUF Recommended rows stay visible when cached. Drop the exclusion so the curated safetensors always list. * Studio diffusion LoRA: sanitize dots out of adapter aliases The LoRA alias is used as the diffusers PEFT adapter name, and PEFT rejects names containing "." (module name can't contain "."). sanitize_alias kept dots, so a LoRA whose filename carries a version tag (e.g. Qwen-Image-2512-Lightning-8steps-V1.0-bf16) failed to apply with a 400. Replace dots too; the alias stays a valid native <lora:NAME:w> filename stem. Adds regression coverage for internal dots. * Studio Images: clarify the GGUF transformer-quant Advanced control Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode" with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/ FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF, falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only. * Studio Images: list on-device unsloth diffusion models in the picker The Images picker's On Device tab hid every non-GGUF cached repo whenever a task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit and FP8 safetensors) never showed up there. List cached repos that pass the task gate, limited under a filter to unsloth-hosted ones so base repos (which fail the diffusion load trust gate) don't appear only to dead-end on click. Chat behavior is unchanged: the task gate still drops image repos there. * Studio: hide single-file image checkpoints from the chat model picker The chat picker treats a cached repo as an image model, and hides it, only when it ships a diffusers model_index.json. Single-file, ComfyUI, and ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models. Fall back to resolving the repo id against the known diffusion families, the same resolver the Images backend loads from, so these checkpoints are tagged text-to-image and stay in the Images picker only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: add the FLUX.2-dev model family Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline) resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline + Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated, reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror. text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from klein's turbo defaults. Verified live: GGUF load resolves the family + gated base repo and generates a real 1024x1024 image on GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: clearer error for an unsupported diffusion model When a repo id resolves to no diffusion family the load raised 'Could not infer a diffusion family... Pass family_override (z-image)', which points at an unrelated family and doesn't say what is supported. Replace it with a message that lists the supported families (from a new supported_family_names helper) and notes that video models and image models whose diffusers transformer has no single-file loader are not supported. Applies to both the diffusers and native sd.cpp load paths. Also refreshes two stale family-registry comments that still called FLUX.2-dev omitted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add SDXL diffusion family (U-Net pipeline support) SDXL is the first U-Net family in the diffusion backend: its denoiser is pipe.unet (UNet2DConditionModel), not a DiT pipe.transformer, and a single-file .safetensors is the whole pipeline rather than a transformer-only file. The backend previously assumed a DiT transformer everywhere, so add the two hooks a U-Net family needs and register SDXL. DiffusionFamily gains denoiser_attr ("transformer" for DiT, "unet" for SDXL) and single_file_is_pipeline (SDXL loads a single file via pipeline_class.from_single_file with the base repo as config, instead of transformer_class.from_single_file plus a companion assembly). _align_vae_dtype now reads the denoiser generically so img2img and inpaint keep the VAE and U-Net dtypes aligned. The non-GGUF trust gate is extended with a short, exact-match, safetensors-only allowlist of official base repos (the SDXL base/refiner and sdxl-turbo), because SDXL ships only as a full pipeline and has no unsloth-hosted GGUF. Local paths stay trusted as before; a random repo, even one that detects as SDXL, is still rejected. The image-conditioned and ControlNet workflows are the standard SDXL pipelines, built around the resident modules via from_pipe like every other family, so SDXL gets txt2img, img2img, inpaint, outpaint, upscale, LoRA and ControlNet. There is no native sd.cpp mapping yet, so the no-GPU route falls back to diffusers. Frontend catalog gains SDXL Base 1.0 and SDXL Turbo entries with SDXL step/guidance defaults (Turbo: few steps, no CFG; base: ~30 steps, real CFG). Tests: new test_diffusion_sdxl.py (family shape, detection, trust allowlist, model kind, U-Net VAE-dtype alignment, LoRA gate) plus loader-branch tests in test_diffusion_backend.py (pipeline-kind from_pretrained, single-file whole-pipeline from_single_file, allowlist accept/reject). Verified live on GPU: sdxl-turbo loads both as a pipeline and as a single file and generates coherent txt2img + img2img. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images: LoRA free-text Hugging Face entry + recipe round-trip The backend has always accepted a bare Hugging Face repo id (owner/name, or owner/name:weight-file.safetensors) as a LoRA, downloading and applying it. But the picker only rendered when the curated catalog had entries, and the catalog is empty, so there was no UI path to apply any LoRA. Show the LoRA section whenever the loaded model supports LoRA, and replace the curated-only dropdown with a text input: type a Hub repo id, or pick a discovered adapter from a datalist of suggestions when the catalog is populated. Also restore LoRAs when loading a recipe. restoreSettings now parses the recipe's "id:weight" strings (splitting on the last colon, since the id itself may contain one for a specific weight file) back into the selection, so replaying a saved image reproduces its adapters. The generate payload trims hand-typed ids and drops empty / zero-weight rows, and a model swap clears the selection (a LoRA is family-specific) without discarding a free-text pick that is not in the curated list. * Add diffusion LoRA training (SDXL text-to-image) First diffusion training path in Studio: train a LoRA on the SDXL U-Net from an image + caption dataset and export it as a diffusers .safetensors that the existing diffusion LoRA loader (and any diffusers pipeline) can load. core/training/diffusion_lora_trainer.py: - DiffusionLoraConfig with validation/defaults (rank, alpha, targets, lr, steps, grad accumulation, resolution, min-SNR gamma, gradient checkpointing, lr scheduler, seed, mixed precision). - discover_image_caption_pairs: captions from metadata.jsonl / captions.jsonl, per-image .txt/.caption sidecars, or a dreambooth instance_prompt fallback (pure, unit-tested). - run_diffusion_lora_training: the loop -- freeze base, PEFT-wrap the U-Net attention projections, VAE-encode (fp32 VAE to avoid the SDXL fp16 overflow), sample noise + timesteps, predict, MSE loss with optional min-SNR weighting (epsilon / v-prediction), AdamW + get_scheduler + grad accumulation + grad clipping, then export via save_lora_weights. Emits worker-protocol events (model_load_*, progress, complete) and polls should_stop for a clean stop with a partial save. - run_diffusion_training_process: mp.Queue subprocess adapter (event_queue / stop_queue), so the training worker can spawn it; plus a CLI entry point. Only SDXL (U-Net) is trained here; DiT families and the Studio UI form + route wiring are follow-ups. The trainer is decoupled and worker-ready. Tests: test_diffusion_lora_trainer.py covers caption discovery (metadata / sidecar / instance prompt / skip-uncaptioned / errors), config normalisation + validation, the SDXL add-time-ids, and the dict->config adapter. Verified live on GPU: a 60-step SDXL LoRA run lowers the loss, exports a ~45 MB adapter, and loading it back shifts generation from baseline (mean abs pixel diff ~55/255). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * diffusion trainer: emit learning_rate in progress events (Studio pump compatibility) The Studio training pump reads 'learning_rate' from progress events; the diffusion trainer emitted 'lr'. Rename the field (and the CLI reader) so the trainer's events are directly consumable by the existing training status/SSE machinery when it is wired into the worker, without a translation shim. * Wire diffusion LoRA training into the Studio API Make the SDXL LoRA trainer reachable from the app with a small, self-contained job service and JSON routes, deliberately separate from the LLM TrainingBackend (whose lifecycle -- LLM config build, per-run SQLite rows, matplotlib plots, transfer-to-chat- inference -- is text-training specific and would mis-handle a diffusion run). core/training/diffusion_training_service.py: DiffusionTrainingService runs one job at a time -- validate the config cheaply (before any spawn), spawn the trainer subprocess (spawn context, parent-lifetime bound), pump its events (model_load_* / progress / complete / error) into an in-memory status snapshot, and support a clean stop. The subprocess context and target are injectable so the full start -> pump -> status -> complete path is unit-tested without real multiprocessing or torch. routes/training.py: POST /api/train/diffusion/start (400 on a bad config, 409 when a job is already running), POST /api/train/diffusion/stop, GET /api/train/diffusion/status (JSON poll). models/training.py: DiffusionTrainingStartRequest + response schemas mirroring DiffusionLoraConfig, so model_dump() passes straight through. Tests: test_diffusion_training.py -- service happy path, bad-config-before-spawn, concurrent-job rejection, clean stop, crash-without-terminal-event, event transitions; plus route wiring via the FastAPI TestClient (start / 422 / 400 / 409 / status / stop) with a mocked service. The diffusion trainer's progress events already use the field names this path expects. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images: add a Train LoRA (SDXL) dialog Surface the diffusion training API in the Images page. A "Train LoRA" button in the top bar opens a self-contained dialog to fine-tune an SDXL LoRA on a folder of images: pick the base model, dataset folder, output folder, an optional instance prompt, and the core hyperparameters (steps, rank, resolution, batch, learning rate), then Start. The dialog polls the training status while open and shows a progress bar, step count, live loss, and the saved adapter path, with a Stop button for a clean stop. The dialog is independent of the loaded generation model (training runs in its own subprocess), and prefills the base model with the loaded checkpoint when it is SDXL, else the SDXL base. api.ts gains startDiffusionTraining / stopDiffusionTraining / getDiffusionTrainingStatus plus their types, matching the /api/train/diffusion routes. * Import diffusion training schemas from models.training directly The import-hoist lint flags newly re-exported names in the models/__init__.py hub as unused (it does not treat __all__ membership as a use). Import the three diffusion training schemas straight from models.training in routes/training.py, where they are used in the route annotations and calls, and drop the __init__ re-export. * Remove stray async task scratch outputs committed by mistake * ControlNet: reject filesystem-like ids and do not cache a model past an unload race Two review findings on the ControlNet path: - resolve_controlnet's bare-repo fallback accepted any id with a slash, so a path-shaped id (/tmp/x, ../x) reached from_pretrained as a local directory. Restrict the fallback to a strict owner/name HF repo id shape. - _controlnet_pipe now re-checks the cancel event after the blocking from_pretrained: an unload that raced the download had already cleared the caches, so caching the late module would pin it past the unload. * Pipeline prefetch: fetch only the default torch weights A full-pipeline prefetch kept every repo file outside assets/, so an official repo that ships multiple formats (SDXL Base: fp16 variants, ONNX, OpenVINO, Flax, a top-level single-file twin) downloaded tens of GB from_pretrained never loads. Skip non-torch exports and dtype-variant twins in _pipeline_file_downloaded, and drop a component .bin when the same directory carries a picked safetensors weight (diffusers' own preference). * Diffusion LoRA training: fall back to fp16 when CUDA lacks bf16 The default mixed_precision=bf16 hard-fails on pre-Ampere GPUs (T4 / V100 / RTX 20xx) which have no bf16 compute; check torch.cuda.is_bf16_supported() and drop to fp16 there. * Diffusion training service: join the old pump outside the lock start() joined a finished job's pump thread while holding the service lock, but the pump's final state writes need that same lock, so the join always burned its full timeout and a stale pump could then overwrite the new job's state. Join outside the lock (with a re-check after), and fence _apply_event and the exit handler by process identity so a superseded pump can never touch the current job's state. Adds regression tests for both. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA training: harden config handling, cancellation, SDXL conditioning, and safety Addresses review findings on the SDXL LoRA trainer: - Gate the base model with the same trust check as inference (unsloth/*, allowlisted official bases, or a local path) before from_pretrained, so an untrusted remote repo is never fetched or deserialised. - Check the stop signal before the (slow) model load, not only between steps, so a cancel during download is honoured; a stop may carry save=False to cancel without leaving a partial adapter. - Per-sample SDXL add_time_ids from the actual crop (original size + crop offset, with the offset mirrored on horizontal flip) instead of a fixed uncropped-square tensor. - Apply EXIF orientation before resize/crop so rotated photos train upright. - Skip gradient clipping when max_grad_norm <= 0 (the Studio 'disable' value) instead of scaling every gradient to zero. - Coerce Studio config strings/blanks: learning_rate string to float, blank hf_token to anonymous, gradient_checkpointing 'none'/'true'/'unsloth' to bool; reject a zero/negative lora_alpha or learning_rate. - Alias the generic Studio training payload keys (model_name/max_steps/batch_size/lora_r/ lr_scheduler_type/random_seed) onto the diffusion field names. - Mirror the trained adapter into loras/diffusion so the Images LoRA picker discovers it. - Report worker exceptions in both message and error keys so the failure is not lost. Adds regression tests for the config coercion/validation and aliasing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ControlNet: address review findings on the diffusers path - resolve_controlnet enforces catalog family compatibility so a direct API call cannot load a ControlNet built for another family through the wrong pipeline. - Unknown ControlNet ids now surface as a 400 (call site maps FileNotFoundError to ValueError) instead of a generic 500. - strength 0 disables ControlNet entirely, so a no-op selection never pays the download / VRAM cost; the control image is decoded and validated BEFORE the ControlNet is resolved or built, so a malformed image fails fast for the same reason. - ControlNet loads use the base compute dtype (state.dtype is a display string, not a torch.dtype, so it silently fell back to float32) and honor the base offload policy via group offloading instead of forcing the module resident. - Empty/malformed HF token coerced to anonymous access. - Flux Union ControlNet control_mode mapped from the selected control type. - resolve_controlnet drops the unused hf_token/cancel_event params. - ControlNetSpec validates guidance_start <= guidance_end (clean 422). - Images UI ControlNet Select shows its placeholder when nothing is selected. Adds regression tests for family enforcement and the union control-mode map. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion training API: LLM interlock, pre-spawn VRAM free, path containment, no dropped knobs Four review findings on the diffusion training start path: - It spawned the SDXL trainer without checking the LLM TrainingBackend, so a start while an LLM run was active put two trainers on the same GPU. Add a symmetric interlock: diffusion start returns 409 when LLM training is active, and LLM start refuses while a diffusion job is active. - It went straight to service.start() without freeing GPU residents. Add a pre-spawn free of the export subprocess, the resident Images pipeline (with an arbiter release), and chat models, mirroring the LLM start path. - data_dir / output_dir were passed through unresolved, so Studio-relative names failed and absolute paths bypassed containment. Resolve them with resolve_dataset_path / resolve_output_dir before spawn (400 on an uncontained path). - The request model dropped max_grad_norm and lora_target_modules, so runs that set them trained with defaults. Add both fields. The gemini pump-join deadlock was already fixed earlier (join outside the lock + proc-identity fence). Note: honoring a stop DURING model load is a trainer-loop change owned by the diffusion training engine PR (should_stop polled before the first optimizer step). Adds route + model regression tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA: harden resolution, native tag precedence, and diffusers teardown Address review findings on the LoRA path: - resolve_one: normalise a blank/whitespace hf_token to None (anonymous access) and reject a client-supplied weight file with traversal / absolute path. - resolve_specs: convert FileNotFoundError from an unknown/stale id to ValueError so the route returns 400 instead of a generic 500. - _scan_local: disambiguate local adapters that share a stem (foo.safetensors vs foo.gguf) so each is uniquely addressable. - inject_prompt_tags: the backend-validated weight now wins over a user-typed <lora:ALIAS:...> for a selected adapter; unselected user tags are left alone. - diffusers _apply_loras: reject a .gguf adapter with a clear error before touching the pipe (diffusers loads safetensors only). - _unload_locked: drop the explicit unload_lora_weights() on teardown; the pipe is dropped wholesale (freeing adapters), so the previous call could race an in-flight denoise on the same pipe. - Images page: use a stable LoRA key and clear the selection (not just the options) when the catalog refresh fails. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion: guard trust check against OSError and validate conditioning inputs - _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid characters (or a bare owner/name id) can't raise OSError; treat any failure as not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request still raises the clear FileNotFoundError for a genuinely missing local pick. - generate(): reject mask_image / upscale / reference_images supplied without an input image, and reject reference_images on a family that does not support reference conditioning, instead of silently degrading to txt2img / img2img. * SDXL: reject GGUF up front, skip unused base weights, drop refiner, and harden helpers Addresses review findings on the SDXL family: - Reject a GGUF load for single_file_is_pipeline families (SDXL) in validate_load_request, before the route evicts the current model; SDXL has no transformer-only GGUF variant. - Skip base-repo weight files when a whole-pipeline single file is loaded: from_single_file (config=base) needs only the base config/tokenizer/scheduler, so a local .safetensors no longer triggers a multi-GB base download. - Remove the SDXL refiner from the non-GGUF trust allowlist: it is an img2img-only pipeline but this backend loads every sdxl repo as the base txt2img pipeline. - Normalize a blank/whitespace hf_token to None once in load_pipeline so every load branch degrades to anonymous instead of erroring on a malformed token. - Read the denoiser dtype from a parameter (compile-wrapped modules may lack .dtype) and access state.family.denoiser_attr directly. Adds/updates regression tests for the trust allowlist, GGUF rejection, and base-config filter. * Images: preserve restored LoRAs through model load and never send hidden LoRAs - The LoRA effect cleared the selection on every load->capable transition, which wiped adapters restored from a gallery recipe before the model finished loading. Track the previously-loaded family in a ref and clear only on a real family swap; keep the selection on the initial load and on unload. - Gate the generate payload's loras on loraCapable so a restored selection that is hidden (loaded model does not support LoRA) is never sent to the backend. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images Train LoRA dialog: token, validation, precision, base-repo prefill, gating, refresh Nine review findings on the SDXL training dialog: - Forward the saved Hub token so a gated/private SDXL base can be trained (the image load flow already sends it). - Re-seed the base-model field from the current default each time the dialog opens; the keep-alive dialog otherwise kept its mount-time default after a model loaded. - Prefill from base_repo (the diffusers pipeline) rather than repo_id, which for a GGUF/single-file SDXL load is the checkpoint path from_pretrained can't open. - Add client-side validation of steps/rank/resolution/batch/learning-rate before the request. - Expose a precision selector (bf16/fp16/fp32) so non-bf16 GPUs can train from the UI, not only the API. - Gate the dialog on the active Images route (active && trainOpen) so switching tabs closes it and stops its polling. - Rescan the LoRA picker when a run completes, so a freshly-trained adapter appears without a model reload. - Cap the dialog height and scroll the body so the Start/Stop footer stays reachable on short viewports. - Correct the copy to not over-promise picker auto-discovery. Freeing the resident Images pipeline before training is handled backend-side in the diffusion training start route. * Merge diffusion-sdxl into diffusion-lora-ux; keep options-only LoRA catch The catalog-refresh .catch from the lower branch clears the selected adapters too, which is right for its catalog-only picker but wrong here: this picker holds free-text HF repo ids that are valid without being in the catalog, so a transient refresh failure must not wipe them. Family swaps still clear the selection and hidden LoRAs are never sent. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Train LoRA dialog: stop suggesting absolute paths the backend rejects The dataset and output placeholders showed /path/to/... examples, but the training routes resolve those fields inside the Studio home and reject absolute paths outside the approved roots, so following the placeholder produced a 400. Use folder-name placeholders and say in the labels and the dialog description where each folder resolves. * Align the VAE to the denoiser's first FLOATING dtype, not its first parameter A GGUF-quantized transformer's leading parameters are packed uint8 storage, so reading next(parameters()).dtype handed nn.Module.to() an integer dtype and every image-conditioned generation on a GGUF model (Qwen-Image-Edit) failed with a 500. Probe the parameters for the first floating dtype, treat an all-integer module as a no-op, and also catch TypeError so an unexpected dtype can never break generation. Regression test included. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Count LR scheduler warmup/decay in optimizer steps, not micro-steps lr_sched.step() runs once per outer optimizer step (after the gradient accumulation inner loop), for train_steps total. The scheduler was configured with num_warmup_steps and num_training_steps multiplied by gradient_accumulation_steps, so with accumulation > 1 a warmup or non-constant schedule stretched past the run and never reached the intended decay. Count both in optimizer steps. * Address Codex review findings on the image-workflows PR Keep diffusion.py importable without torch: the compile/arch patch modules import torch at module level, so import them lazily at their load/unload call sites instead of at module load. This restores the torchless contract so get_diffusion_backend() works on a CPU/native sd.cpp install. Match family reject keywords and aliases as whole path/name segments, not raw substrings, so an unrelated word like edited, edition, or kontextual no longer misroutes or hides a valid base image model, while supported edit families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same segment matching in the picker task filter. Route FLUX.2-dev native guidance through --guidance like the other FLUX families rather than --cfg-scale. Reject native upscale requests that have no input image. Read image header dimensions and reject over-limit inputs before decoding pixels, so a crafted small-payload image cannot spike memory. Reject an upscale that would shrink the source below its input size. Validate the model_kind against the filename extension before the GPU handoff. Estimate a local diffusers pipeline's size from its on-disk weights so auto memory planning does not skip offload and OOM. Report workflows: [txt2img] from the native backend status so the Create tab stays enabled for a loaded native model. Clamp the outpaint canvas to the backend's 4096px decode limit. Adds regression tests for segment matching and kind/extension validation. * Guard inference loads and worker lifetime against diffusion training Teach the chat and image load guards about an active diffusion (SDXL) LoRA job: a chat load is refused (its footprint cannot be fit-checked against the trainer) and an image load is refused outright, mirroring the existing LLM training guards, so a load can no longer allocate GPU memory alongside the trainer and undo the pre-start cleanup. Bind the diffusion trainer subprocess to the parent's lifetime and scrub the native path lease secret from it by running the child through run_without_native_path_secret, matching the inference/export/LLM workers, so a Studio crash or kill no longer leaves the trainer holding the GPU. Reset in_model_load on the complete and error terminal events: a stop or failure during model loading otherwise leaves the status reporting a stale loading indicator after the job has ended. * Harden diffusion LoRA handling on the diffusers and native paths Reject LoRA on a torch.compile'd diffusers transformer (Speed=default/max): diffusers requires the adapter loaded before compilation, so applying one to the already-compiled module fails with adapter-key mismatches. The status gate now hides the picker and generate raises a clear message instead. Convert a cancelled Hub LoRA download (RuntimeError Cancelled) to the diffusion cancellation sentinel in resolve_specs, so an unload/superseding load during resolution maps to a 409 instead of a generic server error. Drop weight-0 LoRA rows before the native support gate so a request carrying only disabled adapters stays a no-op on families where native LoRA is unsupported, matching the diffusers path. Reject duplicate LoRA ids in the request model: both apply paths suffix colliding names, so a repeated id would stack the same adapter past its per-adapter weight bound. Strip all user-typed <lora:...> prompt tags on the native path (only the selected adapters are materialized in the managed lora-model-dir, so an unselected tag can never resolve), and restore saved LoRA selections from a gallery recipe so restore reproduces a LoRA image. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden ControlNet resolve, gallery metadata, and the control-type picker Check cancellation immediately after a ControlNet from_pretrained and before any device placement, so an unload/eviction that raced the download does not allocate several GB onto the GPU after the load was already cleared. Require a loadable weight or shard index (not just config.json) before a local ControlNet folder is advertised, so an interrupted copy is hidden instead of failing deep in from_pretrained as a generic 500. Do not record a strength-0 ControlNet in the gallery recipe: it is treated as disabled and skipped, so the image is unconditioned and the metadata must not claim a ControlNet was applied. Build the control-type picker from the selected ControlNet's advertised control_types instead of a hardcoded passthrough/canny pair, so a union model with a precomputed depth or pose map sends the correct control_mode. * Address further Codex findings on the image-workflows PR - Persist the actual output image size in the gallery recipe instead of the request sliders: Transform/Inpaint/Edit derive the size from the uploaded image, Extend grows the canvas, and Upscale resizes it, so the sliders recorded (and later restored) the wrong dimensions for those workflows. - Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file name) in validate_load_request, so the unloadable pick fails before chat is evicted rather than deep in from_pretrained. - Only publish an image-conditioned from_pipe wrapper to the shared aux cache when the load is still current: from_pipe runs under the generate lock but not the state lock, so an unload racing its construction could otherwise cache a wrapper over torn-down modules that a later load would reuse. - Verify the Windows CUDA runtime archive checksum before extracting it, like the main sd-cli archive, so a corrupt or tampered runtime is rejected rather than extracted next to the binary. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse non-SDXL base models at diffusion training start The trainer only supports the SDXL U-Net, but a FLUX / Qwen-Image / Z-Image repo or a GGUF filename passed as base_model was accepted and then failed minutes later inside StableDiffusionXLPipeline.from_pretrained with an unrelated-looking error. Add a name-based guard in normalized() so known DiT-family names and .gguf checkpoints are rejected up front, which the API start route surfaces as an immediate 400 with a message that says exactly which bases are trainable. Unrecognisable names still pass through so custom local SDXL checkpoints keep working. * Add diffusion dataset upload and training info endpoints Training an image LoRA required knowing the Studio home layout and copying files onto the server by hand, which is the most confusing step of the whole flow. Two small endpoints fix that: - GET /api/train/diffusion/info reports the datasets and outputs roots plus every dataset folder that contains images (with image/caption counts), so the UI can offer a picker instead of a blind free-text path. - POST /api/train/diffusion/dataset uploads images and optional caption .txt / metadata.jsonl files into a named folder under the datasets root, creating it on first use and accumulating on repeat uploads so large sets can arrive in batches. Names are validated to a single path component and files stream to disk under the same per-upload size cap as LLM dataset uploads. The returned name is a valid data_dir for /diffusion/start. * Rework the Train LoRA dialog into a guided SDXL flow The dialog assumed users knew the Studio home layout and that only SDXL is trainable, and hid both facts behind free-text fields. Restructure it around the three real decisions: - Base model is a dropdown of the trainable SDXL picks (Base 1.0, Turbo, the loaded SDXL pipeline when there is one) with a custom repo/path escape hatch, instead of a bare text field defaulting to a repo id. - Training images come from an in-browser upload (new dataset endpoints) or a picker over existing dataset folders with image/caption counts. No shell access or knowledge of the datasets root is needed any more, and the captioning rules are explained inline. - The output field is now Adapter name and the instance prompt is labelled as the trigger prompt, with a no-captions warning wired to the selected dataset's actual caption count. Hyperparameters collapse behind a training settings toggle since the defaults suit a first run. A completed run says where the adapter went and offers Done / Train another, and the top-bar button gets an icon and a plainer description. The dialog title states the SDXL-only scope and that other families load LoRAs but cannot train them yet. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Validate diffusion training config before freeing the GPU The start route freed resident GPU workloads (export, Images pipeline, chat) before the service validated the config, so a start that was then refused, now including a non-SDXL base model, tore down the user's loaded model for nothing. Run the same cheap normalise pass first; the LLM path already follows this rule via its before_spawn hook. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor diffusion LoRA training into a family-aware platform Split the SDXL trainer into a shared, architecture-agnostic layer so more model families can be trained without duplicating the plumbing: - New core/training/diffusion_train_common.py holds the config + validation, dataset discovery, event emission, stop protocol, adapter publishing, and a lazy trainer registry (get_trainer). diffusion_lora_trainer.py keeps the SDXL-specific loop and re-exports the moved names so existing imports are unchanged. - The SDXL-only base-model blocklist becomes a positive check: the family is resolved from the base model (or an explicit model_family) via the diffusion family registry, and a known-but-not-yet-trainable family is refused with a clear message. Unknown custom names still default to the SDXL trainer. - DiffusionFamily gains a trainable flag and train_base_repos; SDXL is marked trainable. DiT families flip on when their trainers land. - Trained adapters now write a <name>.json metadata sidecar (family, base model, rank, trigger prompt, ...) that the LoRA scanner reads to family-gate the adapter in the picker instead of showing it as unknown for every model. - The training base-model trust allowlist adds the official FLUX.1-dev, Z-Image-Turbo, and Qwen-Image repos (safetensors-only, no remote code). * Retain diffusion training loss history and expose it in status The training service kept only the latest loss, so a live loss chart could show a single point. Fold each progress event into bounded (step, loss, lr) history arrays (capped at 4000 points, decimated when full) plus the latest throughput and peak VRAM, and record the family / base model / catalog path on completion. The status endpoint returns these as a nested metric_history object the UI can chart directly, and the start request accepts an optional model_family override. * Tests for the diffusion training platform Cover the trainer registry (get_trainer resolves SDXL, unknown family raises), family resolution (explicit model_family validation, resolved_family on the config), the metadata sidecar write + scan read with family gating, and the service loss-history folding (append, bad-point skipping, decimation at cap, family/perf fields) plus the status route nesting metric_history. * Add diffusion dataset labeling and example-import endpoints The Train tab needs to let users caption small datasets in the browser and pull in a ready-made set to see training work end to end, neither of which the upload-only endpoint supported. Add, under /api/train/diffusion/dataset: - GET {name}/images lists every image with its resolved caption (metadata beats a per-image sidecar, matching the trainer's discovery order) so uncaptioned images are visible and flaggable. - GET {name}/image/{filename} serves an image, with ?thumb=<px> returning a cached downscaled JPEG kept in a hidden .thumbs subdir (regenerated when the source is newer) so the labeling grid stays light. - PUT {name}/caption/{filename} writes, or when blank clears, the .txt sidecar; DELETE {name}/image/{filename} removes the image plus its sidecars and thumbnails. - GET dataset-examples lists a curated, license-labelled registry, and POST dataset/import-example materializes one into a dataset folder as numbered images + .txt captions. Two loaders cover the shapes seen in the wild: streaming rows from datasets.load_dataset (dog-example, Tuxemon) and a snapshot + jsonl walk for imagefolder repos whose captions live in a non-standard *.jsonl (the public-domain tarot set). Imports are idempotent and cap the image count. Filenames and dataset names are validated against path traversal and pinned inside the datasets root. * Test diffusion dataset labeling and example-import endpoints Cover caption precedence, thumbnail generation and .thumbs exclusion, caption write/clear, image delete cleanup, path-traversal rejection on names and filenames, and example import with a mocked datasets.load_dataset (files plus sidecars written, idempotent second call, cap respected, load failure mapped to 502). * Add flow-matching DiT LoRA trainers (FLUX.1-dev, Qwen-Image, Z-Image) Extends diffusion LoRA training beyond SDXL to the three popular DiT families via a single shared flow-matching loop parameterised by small per-family specs (loading, prompt/latent encoding, transformer forward, save). Verified against diffusers 0.38.0: - FLUX.1-dev: 2x2 latent packing + image ids, guidance-embed forward, on-the-fly nf4 QLoRA of the 12B transformer (the dev repo is gated, so training needs the user's HF token). - Qwen-Image: 5D VAE latents normalised by the per-channel latents_mean/std, img_shapes forward, prequant nf4 base by default (on-the-fly nf4 for the bf16 base). - Z-Image: list I/O with the reversed timestep convention and a negated prediction, bf16 only. The registry (get_trainer) and DiffusionFamily.trainable / train_base_repos now route these families to the DiT trainer; the SDXL blocklist guard is replaced by a positive family resolution that also rejects GGUF repos (inference-only) and still-unsupported families. Per-family defaults + labels + VRAM notes are exposed via family_train_infos for the Train UI. Memory: caption embeddings are precomputed once and the text encoders freed before the loop; gradient checkpointing (non-reentrant, required for bnb 4-bit) and 8-bit AdamW are on by default. * Speed up + shrink SDXL LoRA training (precompute text embeds, 8-bit AdamW) SDXL re-encoded every caption with both CLIP text encoders on every step (pure waste, since captions are constant) and kept the encoders resident. Precompute each unique caption's embeddings once, then free the text encoders before the loop: numerically identical (embeddings are deterministic and this consumes no torch RNG, so the noise/timestep stream is unchanged) but faster and ~1.5 GB lighter. Default the optimizer to 8-bit AdamW (bitsandbytes) with an fp32 fallback, halving optimizer state with no meaningful LoRA quality cost. Env toggles (UNSLOTH_DIFFUSION_NO_PRECOMPUTE / _FP32_OPTIM) let the accuracy guard A/B the paths. * Expose trainable families in /diffusion/info and preflight gated bases The training info endpoint now returns the trainable model families (name, label, default + allowed base repos, recommended defaults, and a VRAM/access note) so the Train UI can offer a base picker with realistic guidance. The start route preflights a gated base repo (HEAD model_index.json with the user's token) BEFORE freeing resident GPU workloads, so a missing FLUX.1-dev license/token fails fast with an actionable 400 instead of evicting the loaded model and then hitting a confusing mid-load 401. * Tests for DiT trainers, family resolution, info families, gated preflight Cover the DiT spec table, the QLoRA prequant heuristic, the Z-Image bf16-only guard, the gated-repo name check, family resolution now that FLUX/Qwen/Z-Image are trainable (and GGUF repos are rejected as inference-only), the families list in /diffusion/info, and the gated-base 400 preflight that leaves the GPU untouched. * Wrap the DiT training forward in bf16 autocast The fp32 LoRA parameters and the bnb 4-bit base matmuls need a single compute dtype during the forward, exactly like the diffusers dreambooth scripts run under accelerator.autocast. Without it the 4-bit backward on FLUX.1-dev fails with an illegal-address CUBLAS error partway into the first step. Z-Image and Qwen-Image smokes are unaffected and the SDXL path (its own trainer) is untouched. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add Smithsonian Butterflies and Nouns example datasets Two permissive ~100-image sets for the Train tab: huggan/smithsonian_butterflies_subset (CC0, the classic diffusers-docs training set, imported as a subject set with a trigger prompt since its metadata columns are species names not captions) and m1guelpf/nouns (CC0, captioned pixel-art avatars via the text column). Both cap at 100 images. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
|
Daniel Han
|
565b98be93
|
FLUX.1-dev, Qwen-Image, and Z-Image LoRA training (#6820)
* [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40; GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it. Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s), no overflow, quality below the quant noise floor. So the default leans to accuracy on data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force) lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off). 187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override threading). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add NVFP4 probe documenting it is not yet a win on torch 2.9 scripts/nvfp4_probe.py measures NVFP4 via torchao on the real Z-Image transformer. Finding (B200, 1024px/8 steps): NVFP4 is a torchao feature and DOES run with use_triton_kernel=False (the default triton path needs the missing MSLK library), but only at bf16-compile rate (0.667s vs fp8 0.592s) -- it dequantises FP4->bf16 rather than using the FP4 tensor cores. The real FP4 speedup needs MSLK or torch>=2.11 + torchao's CUTLASS FP4 GEMM. The smoke probe (default triton=True) already keeps NVFP4 out of auto on this env, so auto correctly stays on fp8; NVFP4 activates automatically once fast. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): prefer fp8 over nvfp4 in Blackwell auto ladder Validated NVFP4 on torch 2.11 + torchao CUTLASS FP4 in an isolated env. The FP4 tensor-core GEMM is genuinely active there (a 16384^3 GEMM hits ~3826 TFLOPS, 2.52x bf16 and 1.37x fp8), but it only beats fp8 on very large GEMMs. At the diffusion transformer's shapes (hidden ~3072, MLP ~12288, M~4096) NVFP4 is both slower (0.81x fp8 end to end on Z-Image 1024px) and less accurate (LPIPS 0.166 vs fp8's 0.044). Reorder the Blackwell auto ladder to fp8 before nvfp4 so auto is correct even on a future MSLK-equipped box; nvfp4 stays an explicit opt-in. Add scripts/nvfp4_t211_probe.py (extension diagnostics + GEMM micro + end-to-end). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): pre-quantized transformer loading The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs 13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline (scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True) the quantized weights, so the dense bf16 never touches the GPU. Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same torchao config + min_features filter the runtime path uses, applied ahead of time. New core/inference/diffusion_prequant.py (resolve_prequant_source + load_prequantized_transformer, best-effort, lazy imports). diffusion.py _load_dense_quant_pipeline tries the pre-quant source first and falls back to the dense materialise+quantise path, then to GGUF, so the default is unchanged. DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for the resolver, the meta-init+assign loader, and the backend branch selection + fallbacks; GPU verification via scripts/verify_prequant_backend.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): attention-backend selection Add a selectable attention kernel via the diffusers set_attention_backend dispatcher. Attention is memory-bandwidth bound, so a better kernel is an end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV matmuls torchao never touches) and composes with torch.compile. auto picks the best exact backend for the device: cuDNN fused attention (_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default (below the compile/quant noise floor); native SDPA elsewhere and when speed=off (so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/ xformers/aiter are honored, and an unavailable kernel falls back to the default rather than failing the load. New core/inference/diffusion_attention.py (normalize + per-device select + apply, best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline; attention_backend threads through begin_load / load_pipeline / status like the other load knobs. New request field attention_backend + status field. Hermetic CPU tests for normalize / select policy / apply fallback, plus route threading + 422. Measured via scripts/perf_levers_probe.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the arch tier to put int8 first on a consumer/workstation GPU (detected by the existing _is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first. Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation (-> int8) and data-center Ada/Hopper/Blackwell (-> fp8). * Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across denoise steps a DiT's output settles, so once the first block's residual barely changes the remaining blocks are skipped and their cached output reused. diffusers ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the standalone apply_first_block_cache hook as a fallback). Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 -> 2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar. OFF by default and a per-load opt-in: the win scales with step count, so it is for many-step models (Flux / Qwen-Image) and pointless for few-step distilled models (e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction of the trajectory. It composes with regional compile only with fullgraph=False (the cache's per-step decision is a torch.compiler.disable graph break), which the speed layer now switches to automatically when a cache is engaged. Best-effort: a model whose block signature the hook does not recognise is caught and the load proceeds uncached. - new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache (enable_cache / apply_first_block_cache fallback; threshold auto-raised for a quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import). - diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph when a cache is engaged. - diffusion.py: apply_step_cache before compile; thread transformer_cache / transformer_cache_threshold through begin_load -> load_pipeline and report the engaged mode in status(). - models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and transformer_cache_threshold request fields, engaged mode in the status response. - hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold selection, and best-effort failure handling, plus route threading + validation. - scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM / LPIPS vs the compiled no-cache baseline). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears) The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with 'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2 *_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing min_features filter did not exclude them. Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed, guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers (M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and quantises these layers fine, so the exclusion is int8-only. Sequence embedders (context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note 'context_embedder' contains the substring 'text_embed', which is why the token is the specific 'time_text_embed', not 'text_embed'. Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s (5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already working) are unchanged. - diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes exclude_name_tokens; quantize_transformer passes it for int8 only. - hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps attention / FFN / sequence-embedder linears), while fp8 keeps them. - scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's Linear layers and derive the exclusion list. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears) The prequant-checkpoint builder applied the dense quant filter without the int8-only M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8 checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now produces a working checkpoint on every supported model, giving int8 (the consumer-preferred scheme) the same ~2x load-VRAM and download reduction fp8 already had. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the native stable-diffusion.cpp engine instead of diffusers, with diffusers as the guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM. - diffusion_engine_router: centralised engine selection (built on the existing select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason. - sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface backed by sd-cli, with lazy binary install, registry-driven asset fetch, step-progress parsing, and cancellation. - diffusion_families: per-family single-file VAE + text-encoder asset mapping. - sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled). - routes/inference + gpu_arbiter: drive the active engine via the router; the API now reports the active engine and any fallback reason. - tests for the backend, router, route selection, and cancellation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler Address review feedback on #6724: - engine router: unload the engine being deactivated on a switch, so the old model is not left resident-but-unreachable (the evictor only targets the active engine). - generate route: sd.cpp execution errors (nonzero exit / timeout / missing output) now map to 500, not 409 (which only means not-loaded / cancelled). - native batch: return per-image seeds and persist the actual seed for each image so every batch image is reproducible. - Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the stable-diffusion.cpp docs; other families keep sd-cli defaults. - honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload offload flags on the native load instead of hardcoding them off. - fail the load when the sd-cli binary is present but not runnable (version() now returns None on exec error / nonzero exit). - size estimate: only treat the transformer asset as a possible local path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in load_prequantized_transformer ends in torch.load(weights_only=False), which executes arbitrary code from the pickle. The transformer_prequant_path load-request field reached that unpickle for any local file an authenticated caller named, so a request could trigger remote code execution. Refuse the source.kind=='path' branch unless the operator sets UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted and unaffected. Document the requirement on the API field and add gate tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa - apply_attention_backend now restores the native default when no backend is requested or a kernel fails. diffusers keeps a process-wide active attention backend that set_attention_backend updates, and a fresh transformer's processors follow it, so a load that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile load pinned, breaking the bit-identical/off guarantee. - select_attention_backend drops flash3/flash4 up front when the CUDA capability is below Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit request on the wrong card set fine then crashed mid-generation; it now falls back to native. - Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a valid alias of native) is accepted instead of 422-rejected by Pydantic. - Drop the dead replace('-','_') normalization (no alias uses dashes/underscores). - perf_levers_probe.py output dir is now relative to the script, not a hardcoded path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF - apply_step_cache now engages only via the transformer's native enable_cache (the diffusers CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model now runs uncached per the best-effort contract. - GGUF transformers are quantized (the default Studio load path), so they now use the higher quantized FBCache threshold when the caller leaves it unset, instead of the dense default that could keep the cache from triggering. - fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so fullgraph=True failed warmup and silently measured an eager cached run); output dir is now relative to the script, not a hardcoded path. * Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder _is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate. Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0). * Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just TypeError on older torchao), which would break the mxfp8 config helper instead of falling back to the default. Catch both so the fallback is robust. quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds CUDA memory during the per-row VRAM probe; output dir relative to the script. * Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load - snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the whole snapshot. restore_backend_flags restores each flag independently so one failure can't leave the others leaked process-wide. - load_pipeline restores the flags (and clears the GPU cache) when the build fails after apply_speed_optims mutated the process-wide flags but before _state captured them for unload to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and contaminated later off generations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs without producing output (or without closing stdout) would never reach proc.wait and the wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the PROCESS, so the main thread always enforces the timeout and kills a hung process (which closes the pipe and ends the reader). Add a test that times out even when stdout blocks, and make the no-binary test hermetic so a host-installed sd-cli can't leak in. * Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn The filter callback can be invoked without a module name, so fqn.lower() would raise AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens, so the linear is kept) instead of crashing the quantise pass. * Studio diffusion (Phase 16) review fixes: native engine robustness - sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64); a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds aliased to the same image. Random seeds stay 53-bit (JS-safe). - sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub fall back to anonymous instead of failing auth on a blank token. - sd_cpp_backend: a superseding load now cancels the in-flight generation, so the old sd-cli can no longer return/persist an image from the previous model. - diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a slow 10+ GB free / CUDA sync does not block engine selection. - diffusion_engine_router: probe sd-cli runnability (version()) before committing to native, so a present-but-unrunnable binary falls back to diffusers at selection. - diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a CPU-only install can still reach the native sd.cpp engine instead of failing load. - tests updated for the runnability probe + a not-runnable fallback case. * Studio diffusion (Phase 9) review fixes: prequant safety + validation - SECURITY: a request-supplied local pre-quant path is now unpickled only when it resolves inside an operator-configured ALLOWLIST of directories (UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in, once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on any path a load request named (arbitrary code execution). realpath() blocks symlink escapes; a bare on/off toggle is no longer a wildcard. - Validate the checkpoint's min_features against the runtime Linear filter, so a checkpoint that quantised a different layer set is rejected instead of silently loading a model that mismatches the dense path while reporting the same scheme. - Tolerant base_model_id compare (exact or same final path/repo segment), so a local path or fork of the canonical base is accepted instead of falling back to dense. - _has_meta_tensors uses any(chain(...)) (no intermediate lists). - prequant verify/probe scripts use repo-relative paths (+ env overrides), not the author's absolute /mnt paths. - tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts - diffusion_memory: when group offload is unavailable and the plan falls back to whole-module offload, enable VAE tiling (the group plan left it off, but the fallback is the low-VRAM path where the decode spike can OOM). Covers both the group and sequential fallback branches. - perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a balanced bit-identity regression actually fails the check instead of exiting 0. - compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and the load-progress poll has a 30 min deadline instead of looping forever on a hang. - test for the group->model fallback enabling VAE tiling. * Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path - diffusion: a torchao-quantized transformer is committed only compiled. A dense model resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF it replaced), so when transformer_quant engaged and speed resolved to off, promote to default (regional compile); warn loudly if compile still does not engage. - diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK); otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF. - nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path). - test asserts the eager-quant -> default-compile promotion. * Studio diffusion (Phase 10) review fixes: attention gating + probe isolation - diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation, so auto now stays on native SDPA there. - diffusion_attention: _active_attention_backend handles get_active_backend() returning an enum/None (not a tuple); the old unpack always raised and was swallowed, so the native-restore short-circuit never fired. - perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul so the inductor_flags variant doesn't leak into later compiled rows. - tests for the SM80 cuDNN gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening - install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket); extract through a per-member containment check (Zip-Slip guard); expanduser the --install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch the separately-published cudart runtime DLL archive so sd-cli.exe can start. - sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie. - tests: Zip-Slip rejection, normal extraction, studio-home discovery. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of the literal --output path. SdCppEngine.generate checked only the literal path, so a batch generation would exit 0 and then raise 'no image' (or return a stale file). generate now returns the literal path when present and otherwise falls back to the numbered siblings; single-image behavior is unchanged. Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected without error. * Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats Codex review on the native engine arg builder: - build_sd_cpp_command emitted --width/--height unconditionally, so an img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of the input. width/height are now Optional (None = unset): an image-conditioned run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives the size from the input image (set_width_and_height_if_unset); a plain txt2img run with unset dims keeps the prior 1024x1024 default; explicit dims are always honored. width/height are read only by the builder, so the type change is local. - build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...) that silently swallowed repeats=0 into sd-cli's default of one pass, turning an explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError and emits the flag for any explicit value != 1. Tests: img2img unset dims omit width/height (init_img and ref_images), explicit dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at the default. (Two pre-existing binary-discovery tests fail only because a real sd-cli is installed in this dev environment; unrelated to this change.) * Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc Codex review: the transformer_prequant_path field description still told operators to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the prior security fix made that variable a directory allowlist -- _allowed_prequant_roots deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator following the documented =1 would have every transformer_prequant_path request silently refused. The description now states it must name one or more allowlisted directories and that a bare on/off value is not accepted. Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does not say =1, and describes an allowlist/directory (guards against doc drift). * Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset Codex review on attention-backend selection: - Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first generation with no fallback. select_attention_backend now applies _cudnn_attention_supported() to an explicit cuDNN request too. - flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3 is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a (min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+. - apply_attention_backend's success path left diffusers' process-wide active backend pinned to the kernel it set; a later component whose processors are unconfigured (backend None) would inherit it. It now resets the global registry to native after a successful per-transformer set (the transformer keeps its own backend), best-effort. Also fixed _active_attention_backend: get_active_backend() returns a (name, fn) tuple, so the prior code stringified the tuple and never matched a name, defeating the native-restore short-circuit. Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on SM90; global registry reset after a successful set; _active_attention_backend reads the tuple return. * Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300 (Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So _is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they are treated as data-center class and keep the intended fp8-first behavior. Test: extends the datacenter parametrize with 'NVIDIA B300' and 'NVIDIA GH200 480GB' (now _is_consumer_gpu False). * Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder Codex review: the M=1 modulation/embedder exclusion was wired only into the dense runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint quantised the AdaLN modulation and conditioning-embedder linears, and loading it via transformer_prequant_path (the load path only loads already-quantised tensors, it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes for the runtime path. Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 -> the M=1 exclusion, every other scheme -> none) and use it in both the runtime quantiser and the builder, so a prequant artifact's quantised-layer set always matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion). Test: int8_exclude_name_tokens returns the exclusion for int8 and () for fp8/nvfp4/mxfp8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race Codex review on the native-engine routing: - The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat) unconditionally after engine selection. A native sd.cpp load on a pure-CPU host never touches the GPU, so that needlessly tore down the resident chat model. The handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it. - sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when _run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so the setting was unverifiable. status now derives them from state.offload_flags (still 'none' on CPU, where the flags are empty). - _run_load committed the new state without cancelling/waiting on a generation that started during the (slow) asset download, so a stale sd-cli run against the OLD model could finish afterward and persist an image from the previous model once the new load reported ready. The commit now signals the in-flight cancel and waits on _generate_lock before swapping _state (taken only at commit, so the download never serialises against generation), mirroring the diffusers load path. Tests: CPU native load skips the arbiter while a GPU native load takes it; status reports offload active when flags are set; _run_load cancels and waits for an in-flight generation before committing. * Studio diffusion (Phase 14) review round 2: align helper name with the stack Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the identical helper already present higher in the diffusion stack (Phase 16). The helper definition, the runtime quantiser call, and the offline builder are now byte-identical to that version, so the two branches no longer introduce a divergent name for the same single-source-of-truth and the stack merges without a conflict on this fix. No behavior change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: eager patches + torch.compile cache speed phase Adds the opt-in speed path for the GGUF diffusion transformer behind a selectable speed mode (default off, so output is unchanged until a profile is chosen): - diffusion_eager_patches.py: shared eager fast-paths (channels_last, attention/backend selection, fused norms and QKV) installed at load and rolled back on unload or failed load. - diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent torch.compile cache and the GGUF-transformer compile wiring. - diffusion_arch_patches.py: architecture-specific patches. - diffusion_patch_backend.py: shared install/restore plumbing. - diffusion_speed.py: speed-profile planning. Tests for each module plus the benchmarking and probe scripts used to measure speed, memory, and accuracy of the path. * Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI Backend: - Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file fp8 transformers, gated to the unsloth org plus a curated allowlist. - Image-conditioned workflows built with Pipeline.from_pipe so they reuse the loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint, outpaint, and a hires-fix upscale pass. - Instruction editing as its own family kind (Qwen-Image-Edit-2511, FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and multi-reference) plus klein inpaint. - Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the decoded image size and cap upscale output to avoid OOM on large inputs. - Fixes: from_pipe defaulting to a float32 recast that crashed torchao quantized transformers; image-conditioned calls forcing the slider size onto the input image. Native sd.cpp engine rejects image-conditioned and reference requests it cannot serve. Frontend: - Redesigned Images page with capability-gated workflow tabs (Create, Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor, client-side outpaint, and a multi-reference picker. - Advanced options moved to a right-docked panel mirroring Chat: closed by default, toggled by a single fixed top-bar button that stays in place. sd.cpp installer: pin the release, verify each download's sha256, add a download timeout, and make the source repo configurable for a future mirror. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: correct the Advanced panel comment (closed by default, fixed toggle) * Studio diffusion: LoRA adapters for the Images workflow Add community LoRA support across both diffusion backends, the single biggest step toward broad image-workflow coverage. Backend - New shared module core/inference/diffusion_lora.py: adapter discovery (local scan + curated catalog + owner/name[:file] Hub refs), download via hf_hub_download_with_xet_fallback, alias sanitization, native managed-dir materialization with collision-broken aliases, prompt-tag injection (deduped against user-typed tags), and a supports_lora gate. - Native sd-cli: resolve + materialize selected LoRAs into a per-run managed dir, inject <lora:ALIAS:w> tags, pass --lora-model-dir with --lora-apply-mode auto. The arg builder already emitted these flags. - Diffusers: non-fused load_lora_weights + set_adapters manager, tracked on the pipe so an unchanged selection is a no-op and a model swap resets; cleared on unload. Never fuses (breaks quantized transformers and blocks live weight tweaks). - Gated off where unsupported: torchao fp8/int8 dense, GGUF-via-diffusers, and native Qwen-Image (no LoRA name-conversion branch upstream). - Request contract: optional loras on DiffusionGenerateRequest; empty or omitted is identical to today. supports_lora surfaced in status; chosen LoRAs persisted in gallery recipe metadata. - New GET /api/models/diffusion-loras for the picker (family-filtered). Frontend - Repeatable multi-LoRA picker (adapter select + weight slider 0..2 + remove), gated by the loaded model's supports_lora and family, max 8. Tests - New test_diffusion_lora.py (14): helpers, request validation, native tag/dir wiring, diffusers set_adapters manager, supports_lora matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: ControlNet for the Images workflow (diffusers) Add ControlNet conditioning, the #2 most-used diffusion workflow after LoRA, on the diffusers backend for the families with ControlNet pipelines (FLUX.1 and Qwen-Image), with Union models as the default picks. Backend - New core/inference/diffusion_controlnet.py: family-gated discovery (curated Union models + local dirs + bare owner/name repos), resolution to a loadable repo/dir, control-image preprocessing (passthrough + a dependency-free canny edge map), and a supports_controlnet gate. - diffusion.py: a ControlNet manager parallel to the LoRA one. Loads the (small) ControlNet model once via from_pretrained (cached by id) and builds the family's ControlNet pipeline via Pipeline.from_pipe(base, controlnet=model), reusing the resident base modules at their loaded dtype (no reload, no recast). Passes the control image + conditioning scale + guidance start/end at generate time; cleared on unload. - Families: FLUX.1 -> FluxControlNetPipeline/Model, Qwen-Image -> QwenImageControlNetPipeline/Model. Others declare none (gated off). - Gated off for the native engine, GGUF-via-diffusers, and torchao fp8/int8 dense (same rule as LoRA). v1 conditions txt2img only. - Request contract: optional controlnet on DiffusionGenerateRequest; supports_controlnet in status; the choice persisted in gallery meta. - New GET /api/models/diffusion-controlnets for the picker. Frontend - A ControlNet control in the Images rail (model select + control-image upload + control-type select + strength slider), gated by the loaded model's supports_controlnet + family, shown for text-to-image. Tests - New test_diffusion_controlnet.py (10): discovery/resolve/preprocess/gate helpers, request validation, family wiring, and the diffusers pipe manager (loads once, caches, from_pipe with controlnet, rejects unsupported families). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio ControlNet: show the picker on the Create tab (workflow id is 'create', not 'txt2img') The ControlNet control gated on workflow === 'txt2img', but the Images workflow tab ids are create/transform/inpaint/extend/upscale/reference/edit -- there is no 'txt2img'. So the picker never rendered even with a ControlNet-capable model loaded. Gate on 'create' (the text-to-image tab) for both the picker and the request wiring. Found via a live Playwright capture of the running Studio. * Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a stray "gguf" tag on the Hub but ship no .gguf files. The model search classified them as GGUF from the bare tag, so the picker rendered the GGUF variant expander, which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata (populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected. * Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline load when the id is unsloth-hosted or on-device (the backend infers the family + base repo and gates loads to unsloth/* or local paths), and show a clear message otherwise instead of silently ignoring the click. Curated and GGUF paths are unchanged. * Studio Images: keep curated safetensors models in Recommended after download The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet), so they vanished from the picker after the first load and could only be found by typing an exact search. The row already renders a downloaded badge, matching how GGUF Recommended rows stay visible when cached. Drop the exclusion so the curated safetensors always list. * Studio diffusion LoRA: sanitize dots out of adapter aliases The LoRA alias is used as the diffusers PEFT adapter name, and PEFT rejects names containing "." (module name can't contain "."). sanitize_alias kept dots, so a LoRA whose filename carries a version tag (e.g. Qwen-Image-2512-Lightning-8steps-V1.0-bf16) failed to apply with a 400. Replace dots too; the alias stays a valid native <lora:NAME:w> filename stem. Adds regression coverage for internal dots. * Studio Images: clarify the GGUF transformer-quant Advanced control Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode" with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/ FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF, falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only. * Studio Images: list on-device unsloth diffusion models in the picker The Images picker's On Device tab hid every non-GGUF cached repo whenever a task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit and FP8 safetensors) never showed up there. List cached repos that pass the task gate, limited under a filter to unsloth-hosted ones so base repos (which fail the diffusion load trust gate) don't appear only to dead-end on click. Chat behavior is unchanged: the task gate still drops image repos there. * Studio: hide single-file image checkpoints from the chat model picker The chat picker treats a cached repo as an image model, and hides it, only when it ships a diffusers model_index.json. Single-file, ComfyUI, and ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models. Fall back to resolving the repo id against the known diffusion families, the same resolver the Images backend loads from, so these checkpoints are tagged text-to-image and stay in the Images picker only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: add the FLUX.2-dev model family Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline) resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline + Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated, reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror. text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from klein's turbo defaults. Verified live: GGUF load resolves the family + gated base repo and generates a real 1024x1024 image on GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: clearer error for an unsupported diffusion model When a repo id resolves to no diffusion family the load raised 'Could not infer a diffusion family... Pass family_override (z-image)', which points at an unrelated family and doesn't say what is supported. Replace it with a message that lists the supported families (from a new supported_family_names helper) and notes that video models and image models whose diffusers transformer has no single-file loader are not supported. Applies to both the diffusers and native sd.cpp load paths. Also refreshes two stale family-registry comments that still called FLUX.2-dev omitted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add SDXL diffusion family (U-Net pipeline support) SDXL is the first U-Net family in the diffusion backend: its denoiser is pipe.unet (UNet2DConditionModel), not a DiT pipe.transformer, and a single-file .safetensors is the whole pipeline rather than a transformer-only file. The backend previously assumed a DiT transformer everywhere, so add the two hooks a U-Net family needs and register SDXL. DiffusionFamily gains denoiser_attr ("transformer" for DiT, "unet" for SDXL) and single_file_is_pipeline (SDXL loads a single file via pipeline_class.from_single_file with the base repo as config, instead of transformer_class.from_single_file plus a companion assembly). _align_vae_dtype now reads the denoiser generically so img2img and inpaint keep the VAE and U-Net dtypes aligned. The non-GGUF trust gate is extended with a short, exact-match, safetensors-only allowlist of official base repos (the SDXL base/refiner and sdxl-turbo), because SDXL ships only as a full pipeline and has no unsloth-hosted GGUF. Local paths stay trusted as before; a random repo, even one that detects as SDXL, is still rejected. The image-conditioned and ControlNet workflows are the standard SDXL pipelines, built around the resident modules via from_pipe like every other family, so SDXL gets txt2img, img2img, inpaint, outpaint, upscale, LoRA and ControlNet. There is no native sd.cpp mapping yet, so the no-GPU route falls back to diffusers. Frontend catalog gains SDXL Base 1.0 and SDXL Turbo entries with SDXL step/guidance defaults (Turbo: few steps, no CFG; base: ~30 steps, real CFG). Tests: new test_diffusion_sdxl.py (family shape, detection, trust allowlist, model kind, U-Net VAE-dtype alignment, LoRA gate) plus loader-branch tests in test_diffusion_backend.py (pipeline-kind from_pretrained, single-file whole-pipeline from_single_file, allowlist accept/reject). Verified live on GPU: sdxl-turbo loads both as a pipeline and as a single file and generates coherent txt2img + img2img. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images: LoRA free-text Hugging Face entry + recipe round-trip The backend has always accepted a bare Hugging Face repo id (owner/name, or owner/name:weight-file.safetensors) as a LoRA, downloading and applying it. But the picker only rendered when the curated catalog had entries, and the catalog is empty, so there was no UI path to apply any LoRA. Show the LoRA section whenever the loaded model supports LoRA, and replace the curated-only dropdown with a text input: type a Hub repo id, or pick a discovered adapter from a datalist of suggestions when the catalog is populated. Also restore LoRAs when loading a recipe. restoreSettings now parses the recipe's "id:weight" strings (splitting on the last colon, since the id itself may contain one for a specific weight file) back into the selection, so replaying a saved image reproduces its adapters. The generate payload trims hand-typed ids and drops empty / zero-weight rows, and a model swap clears the selection (a LoRA is family-specific) without discarding a free-text pick that is not in the curated list. * Add diffusion LoRA training (SDXL text-to-image) First diffusion training path in Studio: train a LoRA on the SDXL U-Net from an image + caption dataset and export it as a diffusers .safetensors that the existing diffusion LoRA loader (and any diffusers pipeline) can load. core/training/diffusion_lora_trainer.py: - DiffusionLoraConfig with validation/defaults (rank, alpha, targets, lr, steps, grad accumulation, resolution, min-SNR gamma, gradient checkpointing, lr scheduler, seed, mixed precision). - discover_image_caption_pairs: captions from metadata.jsonl / captions.jsonl, per-image .txt/.caption sidecars, or a dreambooth instance_prompt fallback (pure, unit-tested). - run_diffusion_lora_training: the loop -- freeze base, PEFT-wrap the U-Net attention projections, VAE-encode (fp32 VAE to avoid the SDXL fp16 overflow), sample noise + timesteps, predict, MSE loss with optional min-SNR weighting (epsilon / v-prediction), AdamW + get_scheduler + grad accumulation + grad clipping, then export via save_lora_weights. Emits worker-protocol events (model_load_*, progress, complete) and polls should_stop for a clean stop with a partial save. - run_diffusion_training_process: mp.Queue subprocess adapter (event_queue / stop_queue), so the training worker can spawn it; plus a CLI entry point. Only SDXL (U-Net) is trained here; DiT families and the Studio UI form + route wiring are follow-ups. The trainer is decoupled and worker-ready. Tests: test_diffusion_lora_trainer.py covers caption discovery (metadata / sidecar / instance prompt / skip-uncaptioned / errors), config normalisation + validation, the SDXL add-time-ids, and the dict->config adapter. Verified live on GPU: a 60-step SDXL LoRA run lowers the loss, exports a ~45 MB adapter, and loading it back shifts generation from baseline (mean abs pixel diff ~55/255). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * diffusion trainer: emit learning_rate in progress events (Studio pump compatibility) The Studio training pump reads 'learning_rate' from progress events; the diffusion trainer emitted 'lr'. Rename the field (and the CLI reader) so the trainer's events are directly consumable by the existing training status/SSE machinery when it is wired into the worker, without a translation shim. * Wire diffusion LoRA training into the Studio API Make the SDXL LoRA trainer reachable from the app with a small, self-contained job service and JSON routes, deliberately separate from the LLM TrainingBackend (whose lifecycle -- LLM config build, per-run SQLite rows, matplotlib plots, transfer-to-chat- inference -- is text-training specific and would mis-handle a diffusion run). core/training/diffusion_training_service.py: DiffusionTrainingService runs one job at a time -- validate the config cheaply (before any spawn), spawn the trainer subprocess (spawn context, parent-lifetime bound), pump its events (model_load_* / progress / complete / error) into an in-memory status snapshot, and support a clean stop. The subprocess context and target are injectable so the full start -> pump -> status -> complete path is unit-tested without real multiprocessing or torch. routes/training.py: POST /api/train/diffusion/start (400 on a bad config, 409 when a job is already running), POST /api/train/diffusion/stop, GET /api/train/diffusion/status (JSON poll). models/training.py: DiffusionTrainingStartRequest + response schemas mirroring DiffusionLoraConfig, so model_dump() passes straight through. Tests: test_diffusion_training.py -- service happy path, bad-config-before-spawn, concurrent-job rejection, clean stop, crash-without-terminal-event, event transitions; plus route wiring via the FastAPI TestClient (start / 422 / 400 / 409 / status / stop) with a mocked service. The diffusion trainer's progress events already use the field names this path expects. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images: add a Train LoRA (SDXL) dialog Surface the diffusion training API in the Images page. A "Train LoRA" button in the top bar opens a self-contained dialog to fine-tune an SDXL LoRA on a folder of images: pick the base model, dataset folder, output folder, an optional instance prompt, and the core hyperparameters (steps, rank, resolution, batch, learning rate), then Start. The dialog polls the training status while open and shows a progress bar, step count, live loss, and the saved adapter path, with a Stop button for a clean stop. The dialog is independent of the loaded generation model (training runs in its own subprocess), and prefills the base model with the loaded checkpoint when it is SDXL, else the SDXL base. api.ts gains startDiffusionTraining / stopDiffusionTraining / getDiffusionTrainingStatus plus their types, matching the /api/train/diffusion routes. * Import diffusion training schemas from models.training directly The import-hoist lint flags newly re-exported names in the models/__init__.py hub as unused (it does not treat __all__ membership as a use). Import the three diffusion training schemas straight from models.training in routes/training.py, where they are used in the route annotations and calls, and drop the __init__ re-export. * Remove stray async task scratch outputs committed by mistake * ControlNet: reject filesystem-like ids and do not cache a model past an unload race Two review findings on the ControlNet path: - resolve_controlnet's bare-repo fallback accepted any id with a slash, so a path-shaped id (/tmp/x, ../x) reached from_pretrained as a local directory. Restrict the fallback to a strict owner/name HF repo id shape. - _controlnet_pipe now re-checks the cancel event after the blocking from_pretrained: an unload that raced the download had already cleared the caches, so caching the late module would pin it past the unload. * Pipeline prefetch: fetch only the default torch weights A full-pipeline prefetch kept every repo file outside assets/, so an official repo that ships multiple formats (SDXL Base: fp16 variants, ONNX, OpenVINO, Flax, a top-level single-file twin) downloaded tens of GB from_pretrained never loads. Skip non-torch exports and dtype-variant twins in _pipeline_file_downloaded, and drop a component .bin when the same directory carries a picked safetensors weight (diffusers' own preference). * Diffusion LoRA training: fall back to fp16 when CUDA lacks bf16 The default mixed_precision=bf16 hard-fails on pre-Ampere GPUs (T4 / V100 / RTX 20xx) which have no bf16 compute; check torch.cuda.is_bf16_supported() and drop to fp16 there. * Diffusion training service: join the old pump outside the lock start() joined a finished job's pump thread while holding the service lock, but the pump's final state writes need that same lock, so the join always burned its full timeout and a stale pump could then overwrite the new job's state. Join outside the lock (with a re-check after), and fence _apply_event and the exit handler by process identity so a superseded pump can never touch the current job's state. Adds regression tests for both. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA training: harden config handling, cancellation, SDXL conditioning, and safety Addresses review findings on the SDXL LoRA trainer: - Gate the base model with the same trust check as inference (unsloth/*, allowlisted official bases, or a local path) before from_pretrained, so an untrusted remote repo is never fetched or deserialised. - Check the stop signal before the (slow) model load, not only between steps, so a cancel during download is honoured; a stop may carry save=False to cancel without leaving a partial adapter. - Per-sample SDXL add_time_ids from the actual crop (original size + crop offset, with the offset mirrored on horizontal flip) instead of a fixed uncropped-square tensor. - Apply EXIF orientation before resize/crop so rotated photos train upright. - Skip gradient clipping when max_grad_norm <= 0 (the Studio 'disable' value) instead of scaling every gradient to zero. - Coerce Studio config strings/blanks: learning_rate string to float, blank hf_token to anonymous, gradient_checkpointing 'none'/'true'/'unsloth' to bool; reject a zero/negative lora_alpha or learning_rate. - Alias the generic Studio training payload keys (model_name/max_steps/batch_size/lora_r/ lr_scheduler_type/random_seed) onto the diffusion field names. - Mirror the trained adapter into loras/diffusion so the Images LoRA picker discovers it. - Report worker exceptions in both message and error keys so the failure is not lost. Adds regression tests for the config coercion/validation and aliasing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ControlNet: address review findings on the diffusers path - resolve_controlnet enforces catalog family compatibility so a direct API call cannot load a ControlNet built for another family through the wrong pipeline. - Unknown ControlNet ids now surface as a 400 (call site maps FileNotFoundError to ValueError) instead of a generic 500. - strength 0 disables ControlNet entirely, so a no-op selection never pays the download / VRAM cost; the control image is decoded and validated BEFORE the ControlNet is resolved or built, so a malformed image fails fast for the same reason. - ControlNet loads use the base compute dtype (state.dtype is a display string, not a torch.dtype, so it silently fell back to float32) and honor the base offload policy via group offloading instead of forcing the module resident. - Empty/malformed HF token coerced to anonymous access. - Flux Union ControlNet control_mode mapped from the selected control type. - resolve_controlnet drops the unused hf_token/cancel_event params. - ControlNetSpec validates guidance_start <= guidance_end (clean 422). - Images UI ControlNet Select shows its placeholder when nothing is selected. Adds regression tests for family enforcement and the union control-mode map. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion training API: LLM interlock, pre-spawn VRAM free, path containment, no dropped knobs Four review findings on the diffusion training start path: - It spawned the SDXL trainer without checking the LLM TrainingBackend, so a start while an LLM run was active put two trainers on the same GPU. Add a symmetric interlock: diffusion start returns 409 when LLM training is active, and LLM start refuses while a diffusion job is active. - It went straight to service.start() without freeing GPU residents. Add a pre-spawn free of the export subprocess, the resident Images pipeline (with an arbiter release), and chat models, mirroring the LLM start path. - data_dir / output_dir were passed through unresolved, so Studio-relative names failed and absolute paths bypassed containment. Resolve them with resolve_dataset_path / resolve_output_dir before spawn (400 on an uncontained path). - The request model dropped max_grad_norm and lora_target_modules, so runs that set them trained with defaults. Add both fields. The gemini pump-join deadlock was already fixed earlier (join outside the lock + proc-identity fence). Note: honoring a stop DURING model load is a trainer-loop change owned by the diffusion training engine PR (should_stop polled before the first optimizer step). Adds route + model regression tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA: harden resolution, native tag precedence, and diffusers teardown Address review findings on the LoRA path: - resolve_one: normalise a blank/whitespace hf_token to None (anonymous access) and reject a client-supplied weight file with traversal / absolute path. - resolve_specs: convert FileNotFoundError from an unknown/stale id to ValueError so the route returns 400 instead of a generic 500. - _scan_local: disambiguate local adapters that share a stem (foo.safetensors vs foo.gguf) so each is uniquely addressable. - inject_prompt_tags: the backend-validated weight now wins over a user-typed <lora:ALIAS:...> for a selected adapter; unselected user tags are left alone. - diffusers _apply_loras: reject a .gguf adapter with a clear error before touching the pipe (diffusers loads safetensors only). - _unload_locked: drop the explicit unload_lora_weights() on teardown; the pipe is dropped wholesale (freeing adapters), so the previous call could race an in-flight denoise on the same pipe. - Images page: use a stable LoRA key and clear the selection (not just the options) when the catalog refresh fails. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion: guard trust check against OSError and validate conditioning inputs - _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid characters (or a bare owner/name id) can't raise OSError; treat any failure as not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request still raises the clear FileNotFoundError for a genuinely missing local pick. - generate(): reject mask_image / upscale / reference_images supplied without an input image, and reject reference_images on a family that does not support reference conditioning, instead of silently degrading to txt2img / img2img. * SDXL: reject GGUF up front, skip unused base weights, drop refiner, and harden helpers Addresses review findings on the SDXL family: - Reject a GGUF load for single_file_is_pipeline families (SDXL) in validate_load_request, before the route evicts the current model; SDXL has no transformer-only GGUF variant. - Skip base-repo weight files when a whole-pipeline single file is loaded: from_single_file (config=base) needs only the base config/tokenizer/scheduler, so a local .safetensors no longer triggers a multi-GB base download. - Remove the SDXL refiner from the non-GGUF trust allowlist: it is an img2img-only pipeline but this backend loads every sdxl repo as the base txt2img pipeline. - Normalize a blank/whitespace hf_token to None once in load_pipeline so every load branch degrades to anonymous instead of erroring on a malformed token. - Read the denoiser dtype from a parameter (compile-wrapped modules may lack .dtype) and access state.family.denoiser_attr directly. Adds/updates regression tests for the trust allowlist, GGUF rejection, and base-config filter. * Images: preserve restored LoRAs through model load and never send hidden LoRAs - The LoRA effect cleared the selection on every load->capable transition, which wiped adapters restored from a gallery recipe before the model finished loading. Track the previously-loaded family in a ref and clear only on a real family swap; keep the selection on the initial load and on unload. - Gate the generate payload's loras on loraCapable so a restored selection that is hidden (loaded model does not support LoRA) is never sent to the backend. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images Train LoRA dialog: token, validation, precision, base-repo prefill, gating, refresh Nine review findings on the SDXL training dialog: - Forward the saved Hub token so a gated/private SDXL base can be trained (the image load flow already sends it). - Re-seed the base-model field from the current default each time the dialog opens; the keep-alive dialog otherwise kept its mount-time default after a model loaded. - Prefill from base_repo (the diffusers pipeline) rather than repo_id, which for a GGUF/single-file SDXL load is the checkpoint path from_pretrained can't open. - Add client-side validation of steps/rank/resolution/batch/learning-rate before the request. - Expose a precision selector (bf16/fp16/fp32) so non-bf16 GPUs can train from the UI, not only the API. - Gate the dialog on the active Images route (active && trainOpen) so switching tabs closes it and stops its polling. - Rescan the LoRA picker when a run completes, so a freshly-trained adapter appears without a model reload. - Cap the dialog height and scroll the body so the Start/Stop footer stays reachable on short viewports. - Correct the copy to not over-promise picker auto-discovery. Freeing the resident Images pipeline before training is handled backend-side in the diffusion training start route. * Merge diffusion-sdxl into diffusion-lora-ux; keep options-only LoRA catch The catalog-refresh .catch from the lower branch clears the selected adapters too, which is right for its catalog-only picker but wrong here: this picker holds free-text HF repo ids that are valid without being in the catalog, so a transient refresh failure must not wipe them. Family swaps still clear the selection and hidden LoRAs are never sent. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Train LoRA dialog: stop suggesting absolute paths the backend rejects The dataset and output placeholders showed /path/to/... examples, but the training routes resolve those fields inside the Studio home and reject absolute paths outside the approved roots, so following the placeholder produced a 400. Use folder-name placeholders and say in the labels and the dialog description where each folder resolves. * Align the VAE to the denoiser's first FLOATING dtype, not its first parameter A GGUF-quantized transformer's leading parameters are packed uint8 storage, so reading next(parameters()).dtype handed nn.Module.to() an integer dtype and every image-conditioned generation on a GGUF model (Qwen-Image-Edit) failed with a 500. Probe the parameters for the first floating dtype, treat an all-integer module as a no-op, and also catch TypeError so an unexpected dtype can never break generation. Regression test included. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Count LR scheduler warmup/decay in optimizer steps, not micro-steps lr_sched.step() runs once per outer optimizer step (after the gradient accumulation inner loop), for train_steps total. The scheduler was configured with num_warmup_steps and num_training_steps multiplied by gradient_accumulation_steps, so with accumulation > 1 a warmup or non-constant schedule stretched past the run and never reached the intended decay. Count both in optimizer steps. * Address Codex review findings on the image-workflows PR Keep diffusion.py importable without torch: the compile/arch patch modules import torch at module level, so import them lazily at their load/unload call sites instead of at module load. This restores the torchless contract so get_diffusion_backend() works on a CPU/native sd.cpp install. Match family reject keywords and aliases as whole path/name segments, not raw substrings, so an unrelated word like edited, edition, or kontextual no longer misroutes or hides a valid base image model, while supported edit families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same segment matching in the picker task filter. Route FLUX.2-dev native guidance through --guidance like the other FLUX families rather than --cfg-scale. Reject native upscale requests that have no input image. Read image header dimensions and reject over-limit inputs before decoding pixels, so a crafted small-payload image cannot spike memory. Reject an upscale that would shrink the source below its input size. Validate the model_kind against the filename extension before the GPU handoff. Estimate a local diffusers pipeline's size from its on-disk weights so auto memory planning does not skip offload and OOM. Report workflows: [txt2img] from the native backend status so the Create tab stays enabled for a loaded native model. Clamp the outpaint canvas to the backend's 4096px decode limit. Adds regression tests for segment matching and kind/extension validation. * Guard inference loads and worker lifetime against diffusion training Teach the chat and image load guards about an active diffusion (SDXL) LoRA job: a chat load is refused (its footprint cannot be fit-checked against the trainer) and an image load is refused outright, mirroring the existing LLM training guards, so a load can no longer allocate GPU memory alongside the trainer and undo the pre-start cleanup. Bind the diffusion trainer subprocess to the parent's lifetime and scrub the native path lease secret from it by running the child through run_without_native_path_secret, matching the inference/export/LLM workers, so a Studio crash or kill no longer leaves the trainer holding the GPU. Reset in_model_load on the complete and error terminal events: a stop or failure during model loading otherwise leaves the status reporting a stale loading indicator after the job has ended. * Harden diffusion LoRA handling on the diffusers and native paths Reject LoRA on a torch.compile'd diffusers transformer (Speed=default/max): diffusers requires the adapter loaded before compilation, so applying one to the already-compiled module fails with adapter-key mismatches. The status gate now hides the picker and generate raises a clear message instead. Convert a cancelled Hub LoRA download (RuntimeError Cancelled) to the diffusion cancellation sentinel in resolve_specs, so an unload/superseding load during resolution maps to a 409 instead of a generic server error. Drop weight-0 LoRA rows before the native support gate so a request carrying only disabled adapters stays a no-op on families where native LoRA is unsupported, matching the diffusers path. Reject duplicate LoRA ids in the request model: both apply paths suffix colliding names, so a repeated id would stack the same adapter past its per-adapter weight bound. Strip all user-typed <lora:...> prompt tags on the native path (only the selected adapters are materialized in the managed lora-model-dir, so an unselected tag can never resolve), and restore saved LoRA selections from a gallery recipe so restore reproduces a LoRA image. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden ControlNet resolve, gallery metadata, and the control-type picker Check cancellation immediately after a ControlNet from_pretrained and before any device placement, so an unload/eviction that raced the download does not allocate several GB onto the GPU after the load was already cleared. Require a loadable weight or shard index (not just config.json) before a local ControlNet folder is advertised, so an interrupted copy is hidden instead of failing deep in from_pretrained as a generic 500. Do not record a strength-0 ControlNet in the gallery recipe: it is treated as disabled and skipped, so the image is unconditioned and the metadata must not claim a ControlNet was applied. Build the control-type picker from the selected ControlNet's advertised control_types instead of a hardcoded passthrough/canny pair, so a union model with a precomputed depth or pose map sends the correct control_mode. * Address further Codex findings on the image-workflows PR - Persist the actual output image size in the gallery recipe instead of the request sliders: Transform/Inpaint/Edit derive the size from the uploaded image, Extend grows the canvas, and Upscale resizes it, so the sliders recorded (and later restored) the wrong dimensions for those workflows. - Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file name) in validate_load_request, so the unloadable pick fails before chat is evicted rather than deep in from_pretrained. - Only publish an image-conditioned from_pipe wrapper to the shared aux cache when the load is still current: from_pipe runs under the generate lock but not the state lock, so an unload racing its construction could otherwise cache a wrapper over torn-down modules that a later load would reuse. - Verify the Windows CUDA runtime archive checksum before extracting it, like the main sd-cli archive, so a corrupt or tampered runtime is rejected rather than extracted next to the binary. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse non-SDXL base models at diffusion training start The trainer only supports the SDXL U-Net, but a FLUX / Qwen-Image / Z-Image repo or a GGUF filename passed as base_model was accepted and then failed minutes later inside StableDiffusionXLPipeline.from_pretrained with an unrelated-looking error. Add a name-based guard in normalized() so known DiT-family names and .gguf checkpoints are rejected up front, which the API start route surfaces as an immediate 400 with a message that says exactly which bases are trainable. Unrecognisable names still pass through so custom local SDXL checkpoints keep working. * Add diffusion dataset upload and training info endpoints Training an image LoRA required knowing the Studio home layout and copying files onto the server by hand, which is the most confusing step of the whole flow. Two small endpoints fix that: - GET /api/train/diffusion/info reports the datasets and outputs roots plus every dataset folder that contains images (with image/caption counts), so the UI can offer a picker instead of a blind free-text path. - POST /api/train/diffusion/dataset uploads images and optional caption .txt / metadata.jsonl files into a named folder under the datasets root, creating it on first use and accumulating on repeat uploads so large sets can arrive in batches. Names are validated to a single path component and files stream to disk under the same per-upload size cap as LLM dataset uploads. The returned name is a valid data_dir for /diffusion/start. * Rework the Train LoRA dialog into a guided SDXL flow The dialog assumed users knew the Studio home layout and that only SDXL is trainable, and hid both facts behind free-text fields. Restructure it around the three real decisions: - Base model is a dropdown of the trainable SDXL picks (Base 1.0, Turbo, the loaded SDXL pipeline when there is one) with a custom repo/path escape hatch, instead of a bare text field defaulting to a repo id. - Training images come from an in-browser upload (new dataset endpoints) or a picker over existing dataset folders with image/caption counts. No shell access or knowledge of the datasets root is needed any more, and the captioning rules are explained inline. - The output field is now Adapter name and the instance prompt is labelled as the trigger prompt, with a no-captions warning wired to the selected dataset's actual caption count. Hyperparameters collapse behind a training settings toggle since the defaults suit a first run. A completed run says where the adapter went and offers Done / Train another, and the top-bar button gets an icon and a plainer description. The dialog title states the SDXL-only scope and that other families load LoRAs but cannot train them yet. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Validate diffusion training config before freeing the GPU The start route freed resident GPU workloads (export, Images pipeline, chat) before the service validated the config, so a start that was then refused, now including a non-SDXL base model, tore down the user's loaded model for nothing. Run the same cheap normalise pass first; the LLM path already follows this rule via its before_spawn hook. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor diffusion LoRA training into a family-aware platform Split the SDXL trainer into a shared, architecture-agnostic layer so more model families can be trained without duplicating the plumbing: - New core/training/diffusion_train_common.py holds the config + validation, dataset discovery, event emission, stop protocol, adapter publishing, and a lazy trainer registry (get_trainer). diffusion_lora_trainer.py keeps the SDXL-specific loop and re-exports the moved names so existing imports are unchanged. - The SDXL-only base-model blocklist becomes a positive check: the family is resolved from the base model (or an explicit model_family) via the diffusion family registry, and a known-but-not-yet-trainable family is refused with a clear message. Unknown custom names still default to the SDXL trainer. - DiffusionFamily gains a trainable flag and train_base_repos; SDXL is marked trainable. DiT families flip on when their trainers land. - Trained adapters now write a <name>.json metadata sidecar (family, base model, rank, trigger prompt, ...) that the LoRA scanner reads to family-gate the adapter in the picker instead of showing it as unknown for every model. - The training base-model trust allowlist adds the official FLUX.1-dev, Z-Image-Turbo, and Qwen-Image repos (safetensors-only, no remote code). * Retain diffusion training loss history and expose it in status The training service kept only the latest loss, so a live loss chart could show a single point. Fold each progress event into bounded (step, loss, lr) history arrays (capped at 4000 points, decimated when full) plus the latest throughput and peak VRAM, and record the family / base model / catalog path on completion. The status endpoint returns these as a nested metric_history object the UI can chart directly, and the start request accepts an optional model_family override. * Tests for the diffusion training platform Cover the trainer registry (get_trainer resolves SDXL, unknown family raises), family resolution (explicit model_family validation, resolved_family on the config), the metadata sidecar write + scan read with family gating, and the service loss-history folding (append, bad-point skipping, decimation at cap, family/perf fields) plus the status route nesting metric_history. * Add flow-matching DiT LoRA trainers (FLUX.1-dev, Qwen-Image, Z-Image) Extends diffusion LoRA training beyond SDXL to the three popular DiT families via a single shared flow-matching loop parameterised by small per-family specs (loading, prompt/latent encoding, transformer forward, save). Verified against diffusers 0.38.0: - FLUX.1-dev: 2x2 latent packing + image ids, guidance-embed forward, on-the-fly nf4 QLoRA of the 12B transformer (the dev repo is gated, so training needs the user's HF token). - Qwen-Image: 5D VAE latents normalised by the per-channel latents_mean/std, img_shapes forward, prequant nf4 base by default (on-the-fly nf4 for the bf16 base). - Z-Image: list I/O with the reversed timestep convention and a negated prediction, bf16 only. The registry (get_trainer) and DiffusionFamily.trainable / train_base_repos now route these families to the DiT trainer; the SDXL blocklist guard is replaced by a positive family resolution that also rejects GGUF repos (inference-only) and still-unsupported families. Per-family defaults + labels + VRAM notes are exposed via family_train_infos for the Train UI. Memory: caption embeddings are precomputed once and the text encoders freed before the loop; gradient checkpointing (non-reentrant, required for bnb 4-bit) and 8-bit AdamW are on by default. * Speed up + shrink SDXL LoRA training (precompute text embeds, 8-bit AdamW) SDXL re-encoded every caption with both CLIP text encoders on every step (pure waste, since captions are constant) and kept the encoders resident. Precompute each unique caption's embeddings once, then free the text encoders before the loop: numerically identical (embeddings are deterministic and this consumes no torch RNG, so the noise/timestep stream is unchanged) but faster and ~1.5 GB lighter. Default the optimizer to 8-bit AdamW (bitsandbytes) with an fp32 fallback, halving optimizer state with no meaningful LoRA quality cost. Env toggles (UNSLOTH_DIFFUSION_NO_PRECOMPUTE / _FP32_OPTIM) let the accuracy guard A/B the paths. * Expose trainable families in /diffusion/info and preflight gated bases The training info endpoint now returns the trainable model families (name, label, default + allowed base repos, recommended defaults, and a VRAM/access note) so the Train UI can offer a base picker with realistic guidance. The start route preflights a gated base repo (HEAD model_index.json with the user's token) BEFORE freeing resident GPU workloads, so a missing FLUX.1-dev license/token fails fast with an actionable 400 instead of evicting the loaded model and then hitting a confusing mid-load 401. * Tests for DiT trainers, family resolution, info families, gated preflight Cover the DiT spec table, the QLoRA prequant heuristic, the Z-Image bf16-only guard, the gated-repo name check, family resolution now that FLUX/Qwen/Z-Image are trainable (and GGUF repos are rejected as inference-only), the families list in /diffusion/info, and the gated-base 400 preflight that leaves the GPU untouched. * Wrap the DiT training forward in bf16 autocast The fp32 LoRA parameters and the bnb 4-bit base matmuls need a single compute dtype during the forward, exactly like the diffusers dreambooth scripts run under accelerator.autocast. Without it the 4-bit backward on FLUX.1-dev fails with an illegal-address CUBLAS error partway into the first step. Z-Image and Qwen-Image smokes are unaffected and the SDXL path (its own trainer) is untouched. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
|
Daniel Han
|
0adf7101ed
|
Diffusion training platform: family registry, metric history, adapter metadata (#6819)
* Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source) Add an opt-in transformer_quant mode that loads the dense bf16 transformer and torchao-quantises it onto the low-precision tensor cores, instead of the GGUF transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's 0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor. GGUF+compile stays the low-memory default and the fallback. The mode is gated on CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB); any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe (Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips the tiny projections that crash int8's torch._int_mm. New module mirrors diffusion_precision.py; quant runs before compile before placement. 184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py is the standalone torchao lever probe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): consumer-GPU tuning - lock fp8 fast accumulate, prefer fp8 over mxfp8, reject 2:4 sparsity Consumer Blackwell halves tensor-core throughput on FP32 accumulate (fp8 419 vs 838 TFLOPS with FP16 accumulate; bf16 209), so: - fp8 config locks use_fast_accum=True (Float8MMConfig). torchao already defaults it on; pinning it guards consumer cards against a default change. On B200 it is identical speed and slightly better quality (LPIPS 0.050 vs 0.091). - the Blackwell auto ladder prefers fp8 over mxfp8 (measured faster + more accurate). 2:4 semi-structured sparsity evaluated and rejected (scripts/sparse_accum_probe.py): 2:4 magnitude-prune + fp8 gives LPIPS 0.858 (broken image) with no fine-tune, the cuSPARSELt kernel errors on torch 2.9, and it does not compose with torch.compile (our main ~2x). Documented as a dead end, not shipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add fp8 fast-accum overflow verification probe scripts/fp8_overflow_check.py hooks every quantised linear during a real Z-Image generation and reports max-abs + non-finite counts for use_fast_accum True vs False. Confirms fast accumulation is an accumulation-precision knob, not an overflow one: across 276 linears, including Z-Image's ~1.0e6 activation peaks (which overflow FP16), 0 non-finite elements and identical max-abs for both modes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40; GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it. Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s), no overflow, quality below the quant noise floor. So the default leans to accuracy on data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force) lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off). 187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override threading). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add NVFP4 probe documenting it is not yet a win on torch 2.9 scripts/nvfp4_probe.py measures NVFP4 via torchao on the real Z-Image transformer. Finding (B200, 1024px/8 steps): NVFP4 is a torchao feature and DOES run with use_triton_kernel=False (the default triton path needs the missing MSLK library), but only at bf16-compile rate (0.667s vs fp8 0.592s) -- it dequantises FP4->bf16 rather than using the FP4 tensor cores. The real FP4 speedup needs MSLK or torch>=2.11 + torchao's CUTLASS FP4 GEMM. The smoke probe (default triton=True) already keeps NVFP4 out of auto on this env, so auto correctly stays on fp8; NVFP4 activates automatically once fast. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): prefer fp8 over nvfp4 in Blackwell auto ladder Validated NVFP4 on torch 2.11 + torchao CUTLASS FP4 in an isolated env. The FP4 tensor-core GEMM is genuinely active there (a 16384^3 GEMM hits ~3826 TFLOPS, 2.52x bf16 and 1.37x fp8), but it only beats fp8 on very large GEMMs. At the diffusion transformer's shapes (hidden ~3072, MLP ~12288, M~4096) NVFP4 is both slower (0.81x fp8 end to end on Z-Image 1024px) and less accurate (LPIPS 0.166 vs fp8's 0.044). Reorder the Blackwell auto ladder to fp8 before nvfp4 so auto is correct even on a future MSLK-equipped box; nvfp4 stays an explicit opt-in. Add scripts/nvfp4_t211_probe.py (extension diagnostics + GEMM micro + end-to-end). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): pre-quantized transformer loading The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs 13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline (scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True) the quantized weights, so the dense bf16 never touches the GPU. Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same torchao config + min_features filter the runtime path uses, applied ahead of time. New core/inference/diffusion_prequant.py (resolve_prequant_source + load_prequantized_transformer, best-effort, lazy imports). diffusion.py _load_dense_quant_pipeline tries the pre-quant source first and falls back to the dense materialise+quantise path, then to GGUF, so the default is unchanged. DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for the resolver, the meta-init+assign loader, and the backend branch selection + fallbacks; GPU verification via scripts/verify_prequant_backend.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): attention-backend selection Add a selectable attention kernel via the diffusers set_attention_backend dispatcher. Attention is memory-bandwidth bound, so a better kernel is an end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV matmuls torchao never touches) and composes with torch.compile. auto picks the best exact backend for the device: cuDNN fused attention (_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default (below the compile/quant noise floor); native SDPA elsewhere and when speed=off (so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/ xformers/aiter are honored, and an unavailable kernel falls back to the default rather than failing the load. New core/inference/diffusion_attention.py (normalize + per-device select + apply, best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline; attention_backend threads through begin_load / load_pipeline / status like the other load knobs. New request field attention_backend + status field. Hermetic CPU tests for normalize / select policy / apply fallback, plus route threading + 422. Measured via scripts/perf_levers_probe.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the arch tier to put int8 first on a consumer/workstation GPU (detected by the existing _is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first. Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation (-> int8) and data-center Ada/Hopper/Blackwell (-> fp8). * Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across denoise steps a DiT's output settles, so once the first block's residual barely changes the remaining blocks are skipped and their cached output reused. diffusers ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the standalone apply_first_block_cache hook as a fallback). Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 -> 2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar. OFF by default and a per-load opt-in: the win scales with step count, so it is for many-step models (Flux / Qwen-Image) and pointless for few-step distilled models (e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction of the trajectory. It composes with regional compile only with fullgraph=False (the cache's per-step decision is a torch.compiler.disable graph break), which the speed layer now switches to automatically when a cache is engaged. Best-effort: a model whose block signature the hook does not recognise is caught and the load proceeds uncached. - new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache (enable_cache / apply_first_block_cache fallback; threshold auto-raised for a quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import). - diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph when a cache is engaged. - diffusion.py: apply_step_cache before compile; thread transformer_cache / transformer_cache_threshold through begin_load -> load_pipeline and report the engaged mode in status(). - models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and transformer_cache_threshold request fields, engaged mode in the status response. - hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold selection, and best-effort failure handling, plus route threading + validation. - scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM / LPIPS vs the compiled no-cache baseline). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears) The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with 'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2 *_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing min_features filter did not exclude them. Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed, guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers (M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and quantises these layers fine, so the exclusion is int8-only. Sequence embedders (context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note 'context_embedder' contains the substring 'text_embed', which is why the token is the specific 'time_text_embed', not 'text_embed'. Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s (5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already working) are unchanged. - diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes exclude_name_tokens; quantize_transformer passes it for int8 only. - hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps attention / FFN / sequence-embedder linears), while fp8 keeps them. - scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's Linear layers and derive the exclusion list. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears) The prequant-checkpoint builder applied the dense quant filter without the int8-only M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8 checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now produces a working checkpoint on every supported model, giving int8 (the consumer-preferred scheme) the same ~2x load-VRAM and download reduction fp8 already had. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the native stable-diffusion.cpp engine instead of diffusers, with diffusers as the guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM. - diffusion_engine_router: centralised engine selection (built on the existing select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason. - sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface backed by sd-cli, with lazy binary install, registry-driven asset fetch, step-progress parsing, and cancellation. - diffusion_families: per-family single-file VAE + text-encoder asset mapping. - sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled). - routes/inference + gpu_arbiter: drive the active engine via the router; the API now reports the active engine and any fallback reason. - tests for the backend, router, route selection, and cancellation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler Address review feedback on #6724: - engine router: unload the engine being deactivated on a switch, so the old model is not left resident-but-unreachable (the evictor only targets the active engine). - generate route: sd.cpp execution errors (nonzero exit / timeout / missing output) now map to 500, not 409 (which only means not-loaded / cancelled). - native batch: return per-image seeds and persist the actual seed for each image so every batch image is reproducible. - Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the stable-diffusion.cpp docs; other families keep sd-cli defaults. - honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload offload flags on the native load instead of hardcoding them off. - fail the load when the sd-cli binary is present but not runnable (version() now returns None on exec error / nonzero exit). - size estimate: only treat the transformer asset as a possible local path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in load_prequantized_transformer ends in torch.load(weights_only=False), which executes arbitrary code from the pickle. The transformer_prequant_path load-request field reached that unpickle for any local file an authenticated caller named, so a request could trigger remote code execution. Refuse the source.kind=='path' branch unless the operator sets UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted and unaffected. Document the requirement on the API field and add gate tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa - apply_attention_backend now restores the native default when no backend is requested or a kernel fails. diffusers keeps a process-wide active attention backend that set_attention_backend updates, and a fresh transformer's processors follow it, so a load that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile load pinned, breaking the bit-identical/off guarantee. - select_attention_backend drops flash3/flash4 up front when the CUDA capability is below Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit request on the wrong card set fine then crashed mid-generation; it now falls back to native. - Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a valid alias of native) is accepted instead of 422-rejected by Pydantic. - Drop the dead replace('-','_') normalization (no alias uses dashes/underscores). - perf_levers_probe.py output dir is now relative to the script, not a hardcoded path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF - apply_step_cache now engages only via the transformer's native enable_cache (the diffusers CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model now runs uncached per the best-effort contract. - GGUF transformers are quantized (the default Studio load path), so they now use the higher quantized FBCache threshold when the caller leaves it unset, instead of the dense default that could keep the cache from triggering. - fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so fullgraph=True failed warmup and silently measured an eager cached run); output dir is now relative to the script, not a hardcoded path. * Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder _is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate. Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0). * Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just TypeError on older torchao), which would break the mxfp8 config helper instead of falling back to the default. Catch both so the fallback is robust. quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds CUDA memory during the per-row VRAM probe; output dir relative to the script. * Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load - snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the whole snapshot. restore_backend_flags restores each flag independently so one failure can't leave the others leaked process-wide. - load_pipeline restores the flags (and clears the GPU cache) when the build fails after apply_speed_optims mutated the process-wide flags but before _state captured them for unload to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and contaminated later off generations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs without producing output (or without closing stdout) would never reach proc.wait and the wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the PROCESS, so the main thread always enforces the timeout and kills a hung process (which closes the pipe and ends the reader). Add a test that times out even when stdout blocks, and make the no-binary test hermetic so a host-installed sd-cli can't leak in. * Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn The filter callback can be invoked without a module name, so fqn.lower() would raise AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens, so the linear is kept) instead of crashing the quantise pass. * Studio diffusion (Phase 16) review fixes: native engine robustness - sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64); a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds aliased to the same image. Random seeds stay 53-bit (JS-safe). - sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub fall back to anonymous instead of failing auth on a blank token. - sd_cpp_backend: a superseding load now cancels the in-flight generation, so the old sd-cli can no longer return/persist an image from the previous model. - diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a slow 10+ GB free / CUDA sync does not block engine selection. - diffusion_engine_router: probe sd-cli runnability (version()) before committing to native, so a present-but-unrunnable binary falls back to diffusers at selection. - diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a CPU-only install can still reach the native sd.cpp engine instead of failing load. - tests updated for the runnability probe + a not-runnable fallback case. * Studio diffusion (Phase 9) review fixes: prequant safety + validation - SECURITY: a request-supplied local pre-quant path is now unpickled only when it resolves inside an operator-configured ALLOWLIST of directories (UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in, once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on any path a load request named (arbitrary code execution). realpath() blocks symlink escapes; a bare on/off toggle is no longer a wildcard. - Validate the checkpoint's min_features against the runtime Linear filter, so a checkpoint that quantised a different layer set is rejected instead of silently loading a model that mismatches the dense path while reporting the same scheme. - Tolerant base_model_id compare (exact or same final path/repo segment), so a local path or fork of the canonical base is accepted instead of falling back to dense. - _has_meta_tensors uses any(chain(...)) (no intermediate lists). - prequant verify/probe scripts use repo-relative paths (+ env overrides), not the author's absolute /mnt paths. - tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts - diffusion_memory: when group offload is unavailable and the plan falls back to whole-module offload, enable VAE tiling (the group plan left it off, but the fallback is the low-VRAM path where the decode spike can OOM). Covers both the group and sequential fallback branches. - perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a balanced bit-identity regression actually fails the check instead of exiting 0. - compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and the load-progress poll has a 30 min deadline instead of looping forever on a hang. - test for the group->model fallback enabling VAE tiling. * Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path - diffusion: a torchao-quantized transformer is committed only compiled. A dense model resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF it replaced), so when transformer_quant engaged and speed resolved to off, promote to default (regional compile); warn loudly if compile still does not engage. - diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK); otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF. - nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path). - test asserts the eager-quant -> default-compile promotion. * Studio diffusion (Phase 10) review fixes: attention gating + probe isolation - diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation, so auto now stays on native SDPA there. - diffusion_attention: _active_attention_backend handles get_active_backend() returning an enum/None (not a tuple); the old unpack always raised and was swallowed, so the native-restore short-circuit never fired. - perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul so the inductor_flags variant doesn't leak into later compiled rows. - tests for the SM80 cuDNN gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening - install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket); extract through a per-member containment check (Zip-Slip guard); expanduser the --install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch the separately-published cudart runtime DLL archive so sd-cli.exe can start. - sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie. - tests: Zip-Slip rejection, normal extraction, studio-home discovery. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of the literal --output path. SdCppEngine.generate checked only the literal path, so a batch generation would exit 0 and then raise 'no image' (or return a stale file). generate now returns the literal path when present and otherwise falls back to the numbered siblings; single-image behavior is unchanged. Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected without error. * Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats Codex review on the native engine arg builder: - build_sd_cpp_command emitted --width/--height unconditionally, so an img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of the input. width/height are now Optional (None = unset): an image-conditioned run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives the size from the input image (set_width_and_height_if_unset); a plain txt2img run with unset dims keeps the prior 1024x1024 default; explicit dims are always honored. width/height are read only by the builder, so the type change is local. - build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...) that silently swallowed repeats=0 into sd-cli's default of one pass, turning an explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError and emits the flag for any explicit value != 1. Tests: img2img unset dims omit width/height (init_img and ref_images), explicit dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at the default. (Two pre-existing binary-discovery tests fail only because a real sd-cli is installed in this dev environment; unrelated to this change.) * Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc Codex review: the transformer_prequant_path field description still told operators to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the prior security fix made that variable a directory allowlist -- _allowed_prequant_roots deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator following the documented =1 would have every transformer_prequant_path request silently refused. The description now states it must name one or more allowlisted directories and that a bare on/off value is not accepted. Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does not say =1, and describes an allowlist/directory (guards against doc drift). * Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset Codex review on attention-backend selection: - Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first generation with no fallback. select_attention_backend now applies _cudnn_attention_supported() to an explicit cuDNN request too. - flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3 is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a (min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+. - apply_attention_backend's success path left diffusers' process-wide active backend pinned to the kernel it set; a later component whose processors are unconfigured (backend None) would inherit it. It now resets the global registry to native after a successful per-transformer set (the transformer keeps its own backend), best-effort. Also fixed _active_attention_backend: get_active_backend() returns a (name, fn) tuple, so the prior code stringified the tuple and never matched a name, defeating the native-restore short-circuit. Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on SM90; global registry reset after a successful set; _active_attention_backend reads the tuple return. * Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300 (Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So _is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they are treated as data-center class and keep the intended fp8-first behavior. Test: extends the datacenter parametrize with 'NVIDIA B300' and 'NVIDIA GH200 480GB' (now _is_consumer_gpu False). * Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder Codex review: the M=1 modulation/embedder exclusion was wired only into the dense runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint quantised the AdaLN modulation and conditioning-embedder linears, and loading it via transformer_prequant_path (the load path only loads already-quantised tensors, it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes for the runtime path. Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 -> the M=1 exclusion, every other scheme -> none) and use it in both the runtime quantiser and the builder, so a prequant artifact's quantised-layer set always matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion). Test: int8_exclude_name_tokens returns the exclusion for int8 and () for fp8/nvfp4/mxfp8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race Codex review on the native-engine routing: - The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat) unconditionally after engine selection. A native sd.cpp load on a pure-CPU host never touches the GPU, so that needlessly tore down the resident chat model. The handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it. - sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when _run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so the setting was unverifiable. status now derives them from state.offload_flags (still 'none' on CPU, where the flags are empty). - _run_load committed the new state without cancelling/waiting on a generation that started during the (slow) asset download, so a stale sd-cli run against the OLD model could finish afterward and persist an image from the previous model once the new load reported ready. The commit now signals the in-flight cancel and waits on _generate_lock before swapping _state (taken only at commit, so the download never serialises against generation), mirroring the diffusers load path. Tests: CPU native load skips the arbiter while a GPU native load takes it; status reports offload active when flags are set; _run_load cancels and waits for an in-flight generation before committing. * Studio diffusion (Phase 14) review round 2: align helper name with the stack Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the identical helper already present higher in the diffusion stack (Phase 16). The helper definition, the runtime quantiser call, and the offline builder are now byte-identical to that version, so the two branches no longer introduce a divergent name for the same single-source-of-truth and the stack merges without a conflict on this fix. No behavior change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: eager patches + torch.compile cache speed phase Adds the opt-in speed path for the GGUF diffusion transformer behind a selectable speed mode (default off, so output is unchanged until a profile is chosen): - diffusion_eager_patches.py: shared eager fast-paths (channels_last, attention/backend selection, fused norms and QKV) installed at load and rolled back on unload or failed load. - diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent torch.compile cache and the GGUF-transformer compile wiring. - diffusion_arch_patches.py: architecture-specific patches. - diffusion_patch_backend.py: shared install/restore plumbing. - diffusion_speed.py: speed-profile planning. Tests for each module plus the benchmarking and probe scripts used to measure speed, memory, and accuracy of the path. * Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI Backend: - Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file fp8 transformers, gated to the unsloth org plus a curated allowlist. - Image-conditioned workflows built with Pipeline.from_pipe so they reuse the loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint, outpaint, and a hires-fix upscale pass. - Instruction editing as its own family kind (Qwen-Image-Edit-2511, FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and multi-reference) plus klein inpaint. - Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the decoded image size and cap upscale output to avoid OOM on large inputs. - Fixes: from_pipe defaulting to a float32 recast that crashed torchao quantized transformers; image-conditioned calls forcing the slider size onto the input image. Native sd.cpp engine rejects image-conditioned and reference requests it cannot serve. Frontend: - Redesigned Images page with capability-gated workflow tabs (Create, Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor, client-side outpaint, and a multi-reference picker. - Advanced options moved to a right-docked panel mirroring Chat: closed by default, toggled by a single fixed top-bar button that stays in place. sd.cpp installer: pin the release, verify each download's sha256, add a download timeout, and make the source repo configurable for a future mirror. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: correct the Advanced panel comment (closed by default, fixed toggle) * Studio diffusion: LoRA adapters for the Images workflow Add community LoRA support across both diffusion backends, the single biggest step toward broad image-workflow coverage. Backend - New shared module core/inference/diffusion_lora.py: adapter discovery (local scan + curated catalog + owner/name[:file] Hub refs), download via hf_hub_download_with_xet_fallback, alias sanitization, native managed-dir materialization with collision-broken aliases, prompt-tag injection (deduped against user-typed tags), and a supports_lora gate. - Native sd-cli: resolve + materialize selected LoRAs into a per-run managed dir, inject <lora:ALIAS:w> tags, pass --lora-model-dir with --lora-apply-mode auto. The arg builder already emitted these flags. - Diffusers: non-fused load_lora_weights + set_adapters manager, tracked on the pipe so an unchanged selection is a no-op and a model swap resets; cleared on unload. Never fuses (breaks quantized transformers and blocks live weight tweaks). - Gated off where unsupported: torchao fp8/int8 dense, GGUF-via-diffusers, and native Qwen-Image (no LoRA name-conversion branch upstream). - Request contract: optional loras on DiffusionGenerateRequest; empty or omitted is identical to today. supports_lora surfaced in status; chosen LoRAs persisted in gallery recipe metadata. - New GET /api/models/diffusion-loras for the picker (family-filtered). Frontend - Repeatable multi-LoRA picker (adapter select + weight slider 0..2 + remove), gated by the loaded model's supports_lora and family, max 8. Tests - New test_diffusion_lora.py (14): helpers, request validation, native tag/dir wiring, diffusers set_adapters manager, supports_lora matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: ControlNet for the Images workflow (diffusers) Add ControlNet conditioning, the #2 most-used diffusion workflow after LoRA, on the diffusers backend for the families with ControlNet pipelines (FLUX.1 and Qwen-Image), with Union models as the default picks. Backend - New core/inference/diffusion_controlnet.py: family-gated discovery (curated Union models + local dirs + bare owner/name repos), resolution to a loadable repo/dir, control-image preprocessing (passthrough + a dependency-free canny edge map), and a supports_controlnet gate. - diffusion.py: a ControlNet manager parallel to the LoRA one. Loads the (small) ControlNet model once via from_pretrained (cached by id) and builds the family's ControlNet pipeline via Pipeline.from_pipe(base, controlnet=model), reusing the resident base modules at their loaded dtype (no reload, no recast). Passes the control image + conditioning scale + guidance start/end at generate time; cleared on unload. - Families: FLUX.1 -> FluxControlNetPipeline/Model, Qwen-Image -> QwenImageControlNetPipeline/Model. Others declare none (gated off). - Gated off for the native engine, GGUF-via-diffusers, and torchao fp8/int8 dense (same rule as LoRA). v1 conditions txt2img only. - Request contract: optional controlnet on DiffusionGenerateRequest; supports_controlnet in status; the choice persisted in gallery meta. - New GET /api/models/diffusion-controlnets for the picker. Frontend - A ControlNet control in the Images rail (model select + control-image upload + control-type select + strength slider), gated by the loaded model's supports_controlnet + family, shown for text-to-image. Tests - New test_diffusion_controlnet.py (10): discovery/resolve/preprocess/gate helpers, request validation, family wiring, and the diffusers pipe manager (loads once, caches, from_pipe with controlnet, rejects unsupported families). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio ControlNet: show the picker on the Create tab (workflow id is 'create', not 'txt2img') The ControlNet control gated on workflow === 'txt2img', but the Images workflow tab ids are create/transform/inpaint/extend/upscale/reference/edit -- there is no 'txt2img'. So the picker never rendered even with a ControlNet-capable model loaded. Gate on 'create' (the text-to-image tab) for both the picker and the request wiring. Found via a live Playwright capture of the running Studio. * Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a stray "gguf" tag on the Hub but ship no .gguf files. The model search classified them as GGUF from the bare tag, so the picker rendered the GGUF variant expander, which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata (populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected. * Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline load when the id is unsloth-hosted or on-device (the backend infers the family + base repo and gates loads to unsloth/* or local paths), and show a clear message otherwise instead of silently ignoring the click. Curated and GGUF paths are unchanged. * Studio Images: keep curated safetensors models in Recommended after download The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet), so they vanished from the picker after the first load and could only be found by typing an exact search. The row already renders a downloaded badge, matching how GGUF Recommended rows stay visible when cached. Drop the exclusion so the curated safetensors always list. * Studio diffusion LoRA: sanitize dots out of adapter aliases The LoRA alias is used as the diffusers PEFT adapter name, and PEFT rejects names containing "." (module name can't contain "."). sanitize_alias kept dots, so a LoRA whose filename carries a version tag (e.g. Qwen-Image-2512-Lightning-8steps-V1.0-bf16) failed to apply with a 400. Replace dots too; the alias stays a valid native <lora:NAME:w> filename stem. Adds regression coverage for internal dots. * Studio Images: clarify the GGUF transformer-quant Advanced control Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode" with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/ FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF, falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only. * Studio Images: list on-device unsloth diffusion models in the picker The Images picker's On Device tab hid every non-GGUF cached repo whenever a task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit and FP8 safetensors) never showed up there. List cached repos that pass the task gate, limited under a filter to unsloth-hosted ones so base repos (which fail the diffusion load trust gate) don't appear only to dead-end on click. Chat behavior is unchanged: the task gate still drops image repos there. * Studio: hide single-file image checkpoints from the chat model picker The chat picker treats a cached repo as an image model, and hides it, only when it ships a diffusers model_index.json. Single-file, ComfyUI, and ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models. Fall back to resolving the repo id against the known diffusion families, the same resolver the Images backend loads from, so these checkpoints are tagged text-to-image and stay in the Images picker only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: add the FLUX.2-dev model family Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline) resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline + Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated, reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror. text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from klein's turbo defaults. Verified live: GGUF load resolves the family + gated base repo and generates a real 1024x1024 image on GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: clearer error for an unsupported diffusion model When a repo id resolves to no diffusion family the load raised 'Could not infer a diffusion family... Pass family_override (z-image)', which points at an unrelated family and doesn't say what is supported. Replace it with a message that lists the supported families (from a new supported_family_names helper) and notes that video models and image models whose diffusers transformer has no single-file loader are not supported. Applies to both the diffusers and native sd.cpp load paths. Also refreshes two stale family-registry comments that still called FLUX.2-dev omitted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add SDXL diffusion family (U-Net pipeline support) SDXL is the first U-Net family in the diffusion backend: its denoiser is pipe.unet (UNet2DConditionModel), not a DiT pipe.transformer, and a single-file .safetensors is the whole pipeline rather than a transformer-only file. The backend previously assumed a DiT transformer everywhere, so add the two hooks a U-Net family needs and register SDXL. DiffusionFamily gains denoiser_attr ("transformer" for DiT, "unet" for SDXL) and single_file_is_pipeline (SDXL loads a single file via pipeline_class.from_single_file with the base repo as config, instead of transformer_class.from_single_file plus a companion assembly). _align_vae_dtype now reads the denoiser generically so img2img and inpaint keep the VAE and U-Net dtypes aligned. The non-GGUF trust gate is extended with a short, exact-match, safetensors-only allowlist of official base repos (the SDXL base/refiner and sdxl-turbo), because SDXL ships only as a full pipeline and has no unsloth-hosted GGUF. Local paths stay trusted as before; a random repo, even one that detects as SDXL, is still rejected. The image-conditioned and ControlNet workflows are the standard SDXL pipelines, built around the resident modules via from_pipe like every other family, so SDXL gets txt2img, img2img, inpaint, outpaint, upscale, LoRA and ControlNet. There is no native sd.cpp mapping yet, so the no-GPU route falls back to diffusers. Frontend catalog gains SDXL Base 1.0 and SDXL Turbo entries with SDXL step/guidance defaults (Turbo: few steps, no CFG; base: ~30 steps, real CFG). Tests: new test_diffusion_sdxl.py (family shape, detection, trust allowlist, model kind, U-Net VAE-dtype alignment, LoRA gate) plus loader-branch tests in test_diffusion_backend.py (pipeline-kind from_pretrained, single-file whole-pipeline from_single_file, allowlist accept/reject). Verified live on GPU: sdxl-turbo loads both as a pipeline and as a single file and generates coherent txt2img + img2img. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images: LoRA free-text Hugging Face entry + recipe round-trip The backend has always accepted a bare Hugging Face repo id (owner/name, or owner/name:weight-file.safetensors) as a LoRA, downloading and applying it. But the picker only rendered when the curated catalog had entries, and the catalog is empty, so there was no UI path to apply any LoRA. Show the LoRA section whenever the loaded model supports LoRA, and replace the curated-only dropdown with a text input: type a Hub repo id, or pick a discovered adapter from a datalist of suggestions when the catalog is populated. Also restore LoRAs when loading a recipe. restoreSettings now parses the recipe's "id:weight" strings (splitting on the last colon, since the id itself may contain one for a specific weight file) back into the selection, so replaying a saved image reproduces its adapters. The generate payload trims hand-typed ids and drops empty / zero-weight rows, and a model swap clears the selection (a LoRA is family-specific) without discarding a free-text pick that is not in the curated list. * Add diffusion LoRA training (SDXL text-to-image) First diffusion training path in Studio: train a LoRA on the SDXL U-Net from an image + caption dataset and export it as a diffusers .safetensors that the existing diffusion LoRA loader (and any diffusers pipeline) can load. core/training/diffusion_lora_trainer.py: - DiffusionLoraConfig with validation/defaults (rank, alpha, targets, lr, steps, grad accumulation, resolution, min-SNR gamma, gradient checkpointing, lr scheduler, seed, mixed precision). - discover_image_caption_pairs: captions from metadata.jsonl / captions.jsonl, per-image .txt/.caption sidecars, or a dreambooth instance_prompt fallback (pure, unit-tested). - run_diffusion_lora_training: the loop -- freeze base, PEFT-wrap the U-Net attention projections, VAE-encode (fp32 VAE to avoid the SDXL fp16 overflow), sample noise + timesteps, predict, MSE loss with optional min-SNR weighting (epsilon / v-prediction), AdamW + get_scheduler + grad accumulation + grad clipping, then export via save_lora_weights. Emits worker-protocol events (model_load_*, progress, complete) and polls should_stop for a clean stop with a partial save. - run_diffusion_training_process: mp.Queue subprocess adapter (event_queue / stop_queue), so the training worker can spawn it; plus a CLI entry point. Only SDXL (U-Net) is trained here; DiT families and the Studio UI form + route wiring are follow-ups. The trainer is decoupled and worker-ready. Tests: test_diffusion_lora_trainer.py covers caption discovery (metadata / sidecar / instance prompt / skip-uncaptioned / errors), config normalisation + validation, the SDXL add-time-ids, and the dict->config adapter. Verified live on GPU: a 60-step SDXL LoRA run lowers the loss, exports a ~45 MB adapter, and loading it back shifts generation from baseline (mean abs pixel diff ~55/255). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * diffusion trainer: emit learning_rate in progress events (Studio pump compatibility) The Studio training pump reads 'learning_rate' from progress events; the diffusion trainer emitted 'lr'. Rename the field (and the CLI reader) so the trainer's events are directly consumable by the existing training status/SSE machinery when it is wired into the worker, without a translation shim. * Wire diffusion LoRA training into the Studio API Make the SDXL LoRA trainer reachable from the app with a small, self-contained job service and JSON routes, deliberately separate from the LLM TrainingBackend (whose lifecycle -- LLM config build, per-run SQLite rows, matplotlib plots, transfer-to-chat- inference -- is text-training specific and would mis-handle a diffusion run). core/training/diffusion_training_service.py: DiffusionTrainingService runs one job at a time -- validate the config cheaply (before any spawn), spawn the trainer subprocess (spawn context, parent-lifetime bound), pump its events (model_load_* / progress / complete / error) into an in-memory status snapshot, and support a clean stop. The subprocess context and target are injectable so the full start -> pump -> status -> complete path is unit-tested without real multiprocessing or torch. routes/training.py: POST /api/train/diffusion/start (400 on a bad config, 409 when a job is already running), POST /api/train/diffusion/stop, GET /api/train/diffusion/status (JSON poll). models/training.py: DiffusionTrainingStartRequest + response schemas mirroring DiffusionLoraConfig, so model_dump() passes straight through. Tests: test_diffusion_training.py -- service happy path, bad-config-before-spawn, concurrent-job rejection, clean stop, crash-without-terminal-event, event transitions; plus route wiring via the FastAPI TestClient (start / 422 / 400 / 409 / status / stop) with a mocked service. The diffusion trainer's progress events already use the field names this path expects. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images: add a Train LoRA (SDXL) dialog Surface the diffusion training API in the Images page. A "Train LoRA" button in the top bar opens a self-contained dialog to fine-tune an SDXL LoRA on a folder of images: pick the base model, dataset folder, output folder, an optional instance prompt, and the core hyperparameters (steps, rank, resolution, batch, learning rate), then Start. The dialog polls the training status while open and shows a progress bar, step count, live loss, and the saved adapter path, with a Stop button for a clean stop. The dialog is independent of the loaded generation model (training runs in its own subprocess), and prefills the base model with the loaded checkpoint when it is SDXL, else the SDXL base. api.ts gains startDiffusionTraining / stopDiffusionTraining / getDiffusionTrainingStatus plus their types, matching the /api/train/diffusion routes. * Import diffusion training schemas from models.training directly The import-hoist lint flags newly re-exported names in the models/__init__.py hub as unused (it does not treat __all__ membership as a use). Import the three diffusion training schemas straight from models.training in routes/training.py, where they are used in the route annotations and calls, and drop the __init__ re-export. * Remove stray async task scratch outputs committed by mistake * ControlNet: reject filesystem-like ids and do not cache a model past an unload race Two review findings on the ControlNet path: - resolve_controlnet's bare-repo fallback accepted any id with a slash, so a path-shaped id (/tmp/x, ../x) reached from_pretrained as a local directory. Restrict the fallback to a strict owner/name HF repo id shape. - _controlnet_pipe now re-checks the cancel event after the blocking from_pretrained: an unload that raced the download had already cleared the caches, so caching the late module would pin it past the unload. * Pipeline prefetch: fetch only the default torch weights A full-pipeline prefetch kept every repo file outside assets/, so an official repo that ships multiple formats (SDXL Base: fp16 variants, ONNX, OpenVINO, Flax, a top-level single-file twin) downloaded tens of GB from_pretrained never loads. Skip non-torch exports and dtype-variant twins in _pipeline_file_downloaded, and drop a component .bin when the same directory carries a picked safetensors weight (diffusers' own preference). * Diffusion LoRA training: fall back to fp16 when CUDA lacks bf16 The default mixed_precision=bf16 hard-fails on pre-Ampere GPUs (T4 / V100 / RTX 20xx) which have no bf16 compute; check torch.cuda.is_bf16_supported() and drop to fp16 there. * Diffusion training service: join the old pump outside the lock start() joined a finished job's pump thread while holding the service lock, but the pump's final state writes need that same lock, so the join always burned its full timeout and a stale pump could then overwrite the new job's state. Join outside the lock (with a re-check after), and fence _apply_event and the exit handler by process identity so a superseded pump can never touch the current job's state. Adds regression tests for both. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA training: harden config handling, cancellation, SDXL conditioning, and safety Addresses review findings on the SDXL LoRA trainer: - Gate the base model with the same trust check as inference (unsloth/*, allowlisted official bases, or a local path) before from_pretrained, so an untrusted remote repo is never fetched or deserialised. - Check the stop signal before the (slow) model load, not only between steps, so a cancel during download is honoured; a stop may carry save=False to cancel without leaving a partial adapter. - Per-sample SDXL add_time_ids from the actual crop (original size + crop offset, with the offset mirrored on horizontal flip) instead of a fixed uncropped-square tensor. - Apply EXIF orientation before resize/crop so rotated photos train upright. - Skip gradient clipping when max_grad_norm <= 0 (the Studio 'disable' value) instead of scaling every gradient to zero. - Coerce Studio config strings/blanks: learning_rate string to float, blank hf_token to anonymous, gradient_checkpointing 'none'/'true'/'unsloth' to bool; reject a zero/negative lora_alpha or learning_rate. - Alias the generic Studio training payload keys (model_name/max_steps/batch_size/lora_r/ lr_scheduler_type/random_seed) onto the diffusion field names. - Mirror the trained adapter into loras/diffusion so the Images LoRA picker discovers it. - Report worker exceptions in both message and error keys so the failure is not lost. Adds regression tests for the config coercion/validation and aliasing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ControlNet: address review findings on the diffusers path - resolve_controlnet enforces catalog family compatibility so a direct API call cannot load a ControlNet built for another family through the wrong pipeline. - Unknown ControlNet ids now surface as a 400 (call site maps FileNotFoundError to ValueError) instead of a generic 500. - strength 0 disables ControlNet entirely, so a no-op selection never pays the download / VRAM cost; the control image is decoded and validated BEFORE the ControlNet is resolved or built, so a malformed image fails fast for the same reason. - ControlNet loads use the base compute dtype (state.dtype is a display string, not a torch.dtype, so it silently fell back to float32) and honor the base offload policy via group offloading instead of forcing the module resident. - Empty/malformed HF token coerced to anonymous access. - Flux Union ControlNet control_mode mapped from the selected control type. - resolve_controlnet drops the unused hf_token/cancel_event params. - ControlNetSpec validates guidance_start <= guidance_end (clean 422). - Images UI ControlNet Select shows its placeholder when nothing is selected. Adds regression tests for family enforcement and the union control-mode map. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion training API: LLM interlock, pre-spawn VRAM free, path containment, no dropped knobs Four review findings on the diffusion training start path: - It spawned the SDXL trainer without checking the LLM TrainingBackend, so a start while an LLM run was active put two trainers on the same GPU. Add a symmetric interlock: diffusion start returns 409 when LLM training is active, and LLM start refuses while a diffusion job is active. - It went straight to service.start() without freeing GPU residents. Add a pre-spawn free of the export subprocess, the resident Images pipeline (with an arbiter release), and chat models, mirroring the LLM start path. - data_dir / output_dir were passed through unresolved, so Studio-relative names failed and absolute paths bypassed containment. Resolve them with resolve_dataset_path / resolve_output_dir before spawn (400 on an uncontained path). - The request model dropped max_grad_norm and lora_target_modules, so runs that set them trained with defaults. Add both fields. The gemini pump-join deadlock was already fixed earlier (join outside the lock + proc-identity fence). Note: honoring a stop DURING model load is a trainer-loop change owned by the diffusion training engine PR (should_stop polled before the first optimizer step). Adds route + model regression tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA: harden resolution, native tag precedence, and diffusers teardown Address review findings on the LoRA path: - resolve_one: normalise a blank/whitespace hf_token to None (anonymous access) and reject a client-supplied weight file with traversal / absolute path. - resolve_specs: convert FileNotFoundError from an unknown/stale id to ValueError so the route returns 400 instead of a generic 500. - _scan_local: disambiguate local adapters that share a stem (foo.safetensors vs foo.gguf) so each is uniquely addressable. - inject_prompt_tags: the backend-validated weight now wins over a user-typed <lora:ALIAS:...> for a selected adapter; unselected user tags are left alone. - diffusers _apply_loras: reject a .gguf adapter with a clear error before touching the pipe (diffusers loads safetensors only). - _unload_locked: drop the explicit unload_lora_weights() on teardown; the pipe is dropped wholesale (freeing adapters), so the previous call could race an in-flight denoise on the same pipe. - Images page: use a stable LoRA key and clear the selection (not just the options) when the catalog refresh fails. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion: guard trust check against OSError and validate conditioning inputs - _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid characters (or a bare owner/name id) can't raise OSError; treat any failure as not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request still raises the clear FileNotFoundError for a genuinely missing local pick. - generate(): reject mask_image / upscale / reference_images supplied without an input image, and reject reference_images on a family that does not support reference conditioning, instead of silently degrading to txt2img / img2img. * SDXL: reject GGUF up front, skip unused base weights, drop refiner, and harden helpers Addresses review findings on the SDXL family: - Reject a GGUF load for single_file_is_pipeline families (SDXL) in validate_load_request, before the route evicts the current model; SDXL has no transformer-only GGUF variant. - Skip base-repo weight files when a whole-pipeline single file is loaded: from_single_file (config=base) needs only the base config/tokenizer/scheduler, so a local .safetensors no longer triggers a multi-GB base download. - Remove the SDXL refiner from the non-GGUF trust allowlist: it is an img2img-only pipeline but this backend loads every sdxl repo as the base txt2img pipeline. - Normalize a blank/whitespace hf_token to None once in load_pipeline so every load branch degrades to anonymous instead of erroring on a malformed token. - Read the denoiser dtype from a parameter (compile-wrapped modules may lack .dtype) and access state.family.denoiser_attr directly. Adds/updates regression tests for the trust allowlist, GGUF rejection, and base-config filter. * Images: preserve restored LoRAs through model load and never send hidden LoRAs - The LoRA effect cleared the selection on every load->capable transition, which wiped adapters restored from a gallery recipe before the model finished loading. Track the previously-loaded family in a ref and clear only on a real family swap; keep the selection on the initial load and on unload. - Gate the generate payload's loras on loraCapable so a restored selection that is hidden (loaded model does not support LoRA) is never sent to the backend. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images Train LoRA dialog: token, validation, precision, base-repo prefill, gating, refresh Nine review findings on the SDXL training dialog: - Forward the saved Hub token so a gated/private SDXL base can be trained (the image load flow already sends it). - Re-seed the base-model field from the current default each time the dialog opens; the keep-alive dialog otherwise kept its mount-time default after a model loaded. - Prefill from base_repo (the diffusers pipeline) rather than repo_id, which for a GGUF/single-file SDXL load is the checkpoint path from_pretrained can't open. - Add client-side validation of steps/rank/resolution/batch/learning-rate before the request. - Expose a precision selector (bf16/fp16/fp32) so non-bf16 GPUs can train from the UI, not only the API. - Gate the dialog on the active Images route (active && trainOpen) so switching tabs closes it and stops its polling. - Rescan the LoRA picker when a run completes, so a freshly-trained adapter appears without a model reload. - Cap the dialog height and scroll the body so the Start/Stop footer stays reachable on short viewports. - Correct the copy to not over-promise picker auto-discovery. Freeing the resident Images pipeline before training is handled backend-side in the diffusion training start route. * Merge diffusion-sdxl into diffusion-lora-ux; keep options-only LoRA catch The catalog-refresh .catch from the lower branch clears the selected adapters too, which is right for its catalog-only picker but wrong here: this picker holds free-text HF repo ids that are valid without being in the catalog, so a transient refresh failure must not wipe them. Family swaps still clear the selection and hidden LoRAs are never sent. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Train LoRA dialog: stop suggesting absolute paths the backend rejects The dataset and output placeholders showed /path/to/... examples, but the training routes resolve those fields inside the Studio home and reject absolute paths outside the approved roots, so following the placeholder produced a 400. Use folder-name placeholders and say in the labels and the dialog description where each folder resolves. * Align the VAE to the denoiser's first FLOATING dtype, not its first parameter A GGUF-quantized transformer's leading parameters are packed uint8 storage, so reading next(parameters()).dtype handed nn.Module.to() an integer dtype and every image-conditioned generation on a GGUF model (Qwen-Image-Edit) failed with a 500. Probe the parameters for the first floating dtype, treat an all-integer module as a no-op, and also catch TypeError so an unexpected dtype can never break generation. Regression test included. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Count LR scheduler warmup/decay in optimizer steps, not micro-steps lr_sched.step() runs once per outer optimizer step (after the gradient accumulation inner loop), for train_steps total. The scheduler was configured with num_warmup_steps and num_training_steps multiplied by gradient_accumulation_steps, so with accumulation > 1 a warmup or non-constant schedule stretched past the run and never reached the intended decay. Count both in optimizer steps. * Address Codex review findings on the image-workflows PR Keep diffusion.py importable without torch: the compile/arch patch modules import torch at module level, so import them lazily at their load/unload call sites instead of at module load. This restores the torchless contract so get_diffusion_backend() works on a CPU/native sd.cpp install. Match family reject keywords and aliases as whole path/name segments, not raw substrings, so an unrelated word like edited, edition, or kontextual no longer misroutes or hides a valid base image model, while supported edit families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same segment matching in the picker task filter. Route FLUX.2-dev native guidance through --guidance like the other FLUX families rather than --cfg-scale. Reject native upscale requests that have no input image. Read image header dimensions and reject over-limit inputs before decoding pixels, so a crafted small-payload image cannot spike memory. Reject an upscale that would shrink the source below its input size. Validate the model_kind against the filename extension before the GPU handoff. Estimate a local diffusers pipeline's size from its on-disk weights so auto memory planning does not skip offload and OOM. Report workflows: [txt2img] from the native backend status so the Create tab stays enabled for a loaded native model. Clamp the outpaint canvas to the backend's 4096px decode limit. Adds regression tests for segment matching and kind/extension validation. * Guard inference loads and worker lifetime against diffusion training Teach the chat and image load guards about an active diffusion (SDXL) LoRA job: a chat load is refused (its footprint cannot be fit-checked against the trainer) and an image load is refused outright, mirroring the existing LLM training guards, so a load can no longer allocate GPU memory alongside the trainer and undo the pre-start cleanup. Bind the diffusion trainer subprocess to the parent's lifetime and scrub the native path lease secret from it by running the child through run_without_native_path_secret, matching the inference/export/LLM workers, so a Studio crash or kill no longer leaves the trainer holding the GPU. Reset in_model_load on the complete and error terminal events: a stop or failure during model loading otherwise leaves the status reporting a stale loading indicator after the job has ended. * Harden diffusion LoRA handling on the diffusers and native paths Reject LoRA on a torch.compile'd diffusers transformer (Speed=default/max): diffusers requires the adapter loaded before compilation, so applying one to the already-compiled module fails with adapter-key mismatches. The status gate now hides the picker and generate raises a clear message instead. Convert a cancelled Hub LoRA download (RuntimeError Cancelled) to the diffusion cancellation sentinel in resolve_specs, so an unload/superseding load during resolution maps to a 409 instead of a generic server error. Drop weight-0 LoRA rows before the native support gate so a request carrying only disabled adapters stays a no-op on families where native LoRA is unsupported, matching the diffusers path. Reject duplicate LoRA ids in the request model: both apply paths suffix colliding names, so a repeated id would stack the same adapter past its per-adapter weight bound. Strip all user-typed <lora:...> prompt tags on the native path (only the selected adapters are materialized in the managed lora-model-dir, so an unselected tag can never resolve), and restore saved LoRA selections from a gallery recipe so restore reproduces a LoRA image. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden ControlNet resolve, gallery metadata, and the control-type picker Check cancellation immediately after a ControlNet from_pretrained and before any device placement, so an unload/eviction that raced the download does not allocate several GB onto the GPU after the load was already cleared. Require a loadable weight or shard index (not just config.json) before a local ControlNet folder is advertised, so an interrupted copy is hidden instead of failing deep in from_pretrained as a generic 500. Do not record a strength-0 ControlNet in the gallery recipe: it is treated as disabled and skipped, so the image is unconditioned and the metadata must not claim a ControlNet was applied. Build the control-type picker from the selected ControlNet's advertised control_types instead of a hardcoded passthrough/canny pair, so a union model with a precomputed depth or pose map sends the correct control_mode. * Address further Codex findings on the image-workflows PR - Persist the actual output image size in the gallery recipe instead of the request sliders: Transform/Inpaint/Edit derive the size from the uploaded image, Extend grows the canvas, and Upscale resizes it, so the sliders recorded (and later restored) the wrong dimensions for those workflows. - Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file name) in validate_load_request, so the unloadable pick fails before chat is evicted rather than deep in from_pretrained. - Only publish an image-conditioned from_pipe wrapper to the shared aux cache when the load is still current: from_pipe runs under the generate lock but not the state lock, so an unload racing its construction could otherwise cache a wrapper over torn-down modules that a later load would reuse. - Verify the Windows CUDA runtime archive checksum before extracting it, like the main sd-cli archive, so a corrupt or tampered runtime is rejected rather than extracted next to the binary. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse non-SDXL base models at diffusion training start The trainer only supports the SDXL U-Net, but a FLUX / Qwen-Image / Z-Image repo or a GGUF filename passed as base_model was accepted and then failed minutes later inside StableDiffusionXLPipeline.from_pretrained with an unrelated-looking error. Add a name-based guard in normalized() so known DiT-family names and .gguf checkpoints are rejected up front, which the API start route surfaces as an immediate 400 with a message that says exactly which bases are trainable. Unrecognisable names still pass through so custom local SDXL checkpoints keep working. * Add diffusion dataset upload and training info endpoints Training an image LoRA required knowing the Studio home layout and copying files onto the server by hand, which is the most confusing step of the whole flow. Two small endpoints fix that: - GET /api/train/diffusion/info reports the datasets and outputs roots plus every dataset folder that contains images (with image/caption counts), so the UI can offer a picker instead of a blind free-text path. - POST /api/train/diffusion/dataset uploads images and optional caption .txt / metadata.jsonl files into a named folder under the datasets root, creating it on first use and accumulating on repeat uploads so large sets can arrive in batches. Names are validated to a single path component and files stream to disk under the same per-upload size cap as LLM dataset uploads. The returned name is a valid data_dir for /diffusion/start. * Rework the Train LoRA dialog into a guided SDXL flow The dialog assumed users knew the Studio home layout and that only SDXL is trainable, and hid both facts behind free-text fields. Restructure it around the three real decisions: - Base model is a dropdown of the trainable SDXL picks (Base 1.0, Turbo, the loaded SDXL pipeline when there is one) with a custom repo/path escape hatch, instead of a bare text field defaulting to a repo id. - Training images come from an in-browser upload (new dataset endpoints) or a picker over existing dataset folders with image/caption counts. No shell access or knowledge of the datasets root is needed any more, and the captioning rules are explained inline. - The output field is now Adapter name and the instance prompt is labelled as the trigger prompt, with a no-captions warning wired to the selected dataset's actual caption count. Hyperparameters collapse behind a training settings toggle since the defaults suit a first run. A completed run says where the adapter went and offers Done / Train another, and the top-bar button gets an icon and a plainer description. The dialog title states the SDXL-only scope and that other families load LoRAs but cannot train them yet. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Validate diffusion training config before freeing the GPU The start route freed resident GPU workloads (export, Images pipeline, chat) before the service validated the config, so a start that was then refused, now including a non-SDXL base model, tore down the user's loaded model for nothing. Run the same cheap normalise pass first; the LLM path already follows this rule via its before_spawn hook. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor diffusion LoRA training into a family-aware platform Split the SDXL trainer into a shared, architecture-agnostic layer so more model families can be trained without duplicating the plumbing: - New core/training/diffusion_train_common.py holds the config + validation, dataset discovery, event emission, stop protocol, adapter publishing, and a lazy trainer registry (get_trainer). diffusion_lora_trainer.py keeps the SDXL-specific loop and re-exports the moved names so existing imports are unchanged. - The SDXL-only base-model blocklist becomes a positive check: the family is resolved from the base model (or an explicit model_family) via the diffusion family registry, and a known-but-not-yet-trainable family is refused with a clear message. Unknown custom names still default to the SDXL trainer. - DiffusionFamily gains a trainable flag and train_base_repos; SDXL is marked trainable. DiT families flip on when their trainers land. - Trained adapters now write a <name>.json metadata sidecar (family, base model, rank, trigger prompt, ...) that the LoRA scanner reads to family-gate the adapter in the picker instead of showing it as unknown for every model. - The training base-model trust allowlist adds the official FLUX.1-dev, Z-Image-Turbo, and Qwen-Image repos (safetensors-only, no remote code). * Retain diffusion training loss history and expose it in status The training service kept only the latest loss, so a live loss chart could show a single point. Fold each progress event into bounded (step, loss, lr) history arrays (capped at 4000 points, decimated when full) plus the latest throughput and peak VRAM, and record the family / base model / catalog path on completion. The status endpoint returns these as a nested metric_history object the UI can chart directly, and the start request accepts an optional model_family override. * Tests for the diffusion training platform Cover the trainer registry (get_trainer resolves SDXL, unknown family raises), family resolution (explicit model_family validation, resolved_family on the config), the metadata sidecar write + scan read with family gating, and the service loss-history folding (append, bad-point skipping, decimation at cap, family/perf fields) plus the status route nesting metric_history. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
|
Daniel Han
|
f1ab727b5c
|
Images: add a Train LoRA (SDXL) dialog (#6794)
* Studio diffusion (Phase 7): accuracy-preserving speed pass Re-review of the diffusion stack (#6675/#6679/#6680) surfaced one real accuracy bug and a dead-on-arrival speed path; this fixes both and adds the lossless / near-lossless wins, all measured on a B200. Correctness: - TF32 global-state leak (fix). speed_mode=max flipped torch.backends.*.allow_tf32 process-wide and never restored them, so a later `off` load silently inherited TF32 and was no longer bit-identical. Added snapshot_backend_flags / restore_backend_flags (TF32 + cudnn.benchmark), captured before the speed layer runs and restored on unload. Verified: load max -> unload -> load off is now byte-identical (PSNR inf) to a fresh off. - sd-cli timeout could hang forever. _run() blocked in `for line in stdout` and only checked the timeout after EOF, so a child stuck in model load / GPU init with no output ignored the timeout. Drained stdout on a reader thread with a wall-clock deadline. Added a silent-hang regression test. Speed (diffusers path), near-lossless, opt-in tiers: - Regional torch.compile now runs on the GGUF transformer. The is_gguf gate (and Z-Image's supports_torch_compile=False) were stale: compile_repeated_blocks compiles and runs ~2.2x faster on the GGUF Z-Image transformer on torch 2.9.1 / diffusers 0.38 (the per-op dequant stays eager, the rest of the block compiles). Measured: off 1.80s -> default 0.82s/gen (+54.7%), PSNR 37.7 dB vs eager -- far above the Q4 quant noise floor (~21 dB), so it does not move output quality. Gate relaxed; default tier delivers it. - cudnn.benchmark added to the default tier (autotunes the fixed-shape VAE convs). - torch.inference_mode() around the pipeline call (lossless, strictly faster than the no_grad diffusers uses internally). Memory path: - VAE tiling (not bit-identical >1MP) restricted to the model/sequential/CPU tiers; the balanced (group) tier keeps exact slicing only, so it is now bit-identical to the resident image (verified PSNR inf) and slightly faster. - Group offload adds non_blocking + record_stream on the CUDA stream path to overlap each block's H2D copy with compute (lossless; gated on the installed diffusers signature so older versions still work). Native (sd.cpp) path: - native_speed_flags: a first-class speed knob (default -> --diffusion-fa, a near-lossless CUDA win that was previously only added on offload tiers; max also -> --diffusion-conv-direct). conv-direct stays opt-in: measured +45% on CUDA, so it is never auto-on. Engine generate() merges it, de-duped against offload flags. Default profile: a GGUF model with no explicit speed_mode now resolves to the `default` profile (resolve_speed_mode), since compile's perturbation sits below the quantisation noise floor and so does not reduce quality versus the dense reference; out of the box a GGUF Z-Image generation drops from 1.80s to 0.81s. Dense models stay `off` / bit-identical, and an explicit speed_mode -- including "off" -- is always honored, so the byte-identical path remains one flag away and is the regression reference. Tooling: scripts/compile_probe.py (eager vs compiled GGUF probe), scripts/ perf_verify.py (the B200 verification above), and diffusion_bench.py gains --speed-mode so the speed tiers are benchmarkable. Tests: 183 passing (was 166); new coverage for the backend-flag snapshot/restore, GGUF compile eligibility, the balanced tiling/slicing split, native_speed_flags + the engine de-dup, and the sd-cli silent-hang timeout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): max tier uses max-autotune-no-cudagraphs + engine/lever benchmarks The opt-in `max` speed tier now compiles the repeated block with mode=max-autotune-no-cudagraphs (dynamic=False) instead of the default mode: Triton autotuning for GEMM/conv-heavier models, gated to the tier where a longer cold compile is acceptable. CUDA-graph modes (reduce-overhead / max-autotune) are deliberately avoided -- both crash on the regionally-compiled block (its static output buffer is overwritten across denoise steps), measured. Adds two reproducible benchmarks used to validate the optimization research: - scripts/compare_engines.py: PyTorch (diffusers GGUF) vs native sd.cpp head-to-head. - scripts/leverage_probe.py: coordinate_descent_tuning + FirstBlockCache probes. Measured on B200 (Z-Image Q4_K_M, 1024px, 8 steps): default compile 0.80s/gen; coordinate_descent_tuning 0.79s (within noise, already covered by max-autotune); FirstBlockCache does not run on Z-Image (diffusers 0.38 block-detection / Dynamo). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source) Add an opt-in transformer_quant mode that loads the dense bf16 transformer and torchao-quantises it onto the low-precision tensor cores, instead of the GGUF transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's 0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor. GGUF+compile stays the low-memory default and the fallback. The mode is gated on CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB); any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe (Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips the tiny projections that crash int8's torch._int_mm. New module mirrors diffusion_precision.py; quant runs before compile before placement. 184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py is the standalone torchao lever probe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): consumer-GPU tuning - lock fp8 fast accumulate, prefer fp8 over mxfp8, reject 2:4 sparsity Consumer Blackwell halves tensor-core throughput on FP32 accumulate (fp8 419 vs 838 TFLOPS with FP16 accumulate; bf16 209), so: - fp8 config locks use_fast_accum=True (Float8MMConfig). torchao already defaults it on; pinning it guards consumer cards against a default change. On B200 it is identical speed and slightly better quality (LPIPS 0.050 vs 0.091). - the Blackwell auto ladder prefers fp8 over mxfp8 (measured faster + more accurate). 2:4 semi-structured sparsity evaluated and rejected (scripts/sparse_accum_probe.py): 2:4 magnitude-prune + fp8 gives LPIPS 0.858 (broken image) with no fine-tune, the cuSPARSELt kernel errors on torch 2.9, and it does not compose with torch.compile (our main ~2x). Documented as a dead end, not shipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add fp8 fast-accum overflow verification probe scripts/fp8_overflow_check.py hooks every quantised linear during a real Z-Image generation and reports max-abs + non-finite counts for use_fast_accum True vs False. Confirms fast accumulation is an accumulation-precision knob, not an overflow one: across 276 linears, including Z-Image's ~1.0e6 activation peaks (which overflow FP16), 0 non-finite elements and identical max-abs for both modes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40; GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it. Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s), no overflow, quality below the quant noise floor. So the default leans to accuracy on data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force) lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off). 187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override threading). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add NVFP4 probe documenting it is not yet a win on torch 2.9 scripts/nvfp4_probe.py measures NVFP4 via torchao on the real Z-Image transformer. Finding (B200, 1024px/8 steps): NVFP4 is a torchao feature and DOES run with use_triton_kernel=False (the default triton path needs the missing MSLK library), but only at bf16-compile rate (0.667s vs fp8 0.592s) -- it dequantises FP4->bf16 rather than using the FP4 tensor cores. The real FP4 speedup needs MSLK or torch>=2.11 + torchao's CUTLASS FP4 GEMM. The smoke probe (default triton=True) already keeps NVFP4 out of auto on this env, so auto correctly stays on fp8; NVFP4 activates automatically once fast. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): prefer fp8 over nvfp4 in Blackwell auto ladder Validated NVFP4 on torch 2.11 + torchao CUTLASS FP4 in an isolated env. The FP4 tensor-core GEMM is genuinely active there (a 16384^3 GEMM hits ~3826 TFLOPS, 2.52x bf16 and 1.37x fp8), but it only beats fp8 on very large GEMMs. At the diffusion transformer's shapes (hidden ~3072, MLP ~12288, M~4096) NVFP4 is both slower (0.81x fp8 end to end on Z-Image 1024px) and less accurate (LPIPS 0.166 vs fp8's 0.044). Reorder the Blackwell auto ladder to fp8 before nvfp4 so auto is correct even on a future MSLK-equipped box; nvfp4 stays an explicit opt-in. Add scripts/nvfp4_t211_probe.py (extension diagnostics + GEMM micro + end-to-end). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): pre-quantized transformer loading The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs 13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline (scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True) the quantized weights, so the dense bf16 never touches the GPU. Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same torchao config + min_features filter the runtime path uses, applied ahead of time. New core/inference/diffusion_prequant.py (resolve_prequant_source + load_prequantized_transformer, best-effort, lazy imports). diffusion.py _load_dense_quant_pipeline tries the pre-quant source first and falls back to the dense materialise+quantise path, then to GGUF, so the default is unchanged. DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for the resolver, the meta-init+assign loader, and the backend branch selection + fallbacks; GPU verification via scripts/verify_prequant_backend.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): attention-backend selection Add a selectable attention kernel via the diffusers set_attention_backend dispatcher. Attention is memory-bandwidth bound, so a better kernel is an end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV matmuls torchao never touches) and composes with torch.compile. auto picks the best exact backend for the device: cuDNN fused attention (_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default (below the compile/quant noise floor); native SDPA elsewhere and when speed=off (so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/ xformers/aiter are honored, and an unavailable kernel falls back to the default rather than failing the load. New core/inference/diffusion_attention.py (normalize + per-device select + apply, best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline; attention_backend threads through begin_load / load_pipeline / status like the other load knobs. New request field attention_backend + status field. Hermetic CPU tests for normalize / select policy / apply fallback, plus route threading + 422. Measured via scripts/perf_levers_probe.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the arch tier to put int8 first on a consumer/workstation GPU (detected by the existing _is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first. Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation (-> int8) and data-center Ada/Hopper/Blackwell (-> fp8). * Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across denoise steps a DiT's output settles, so once the first block's residual barely changes the remaining blocks are skipped and their cached output reused. diffusers ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the standalone apply_first_block_cache hook as a fallback). Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 -> 2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar. OFF by default and a per-load opt-in: the win scales with step count, so it is for many-step models (Flux / Qwen-Image) and pointless for few-step distilled models (e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction of the trajectory. It composes with regional compile only with fullgraph=False (the cache's per-step decision is a torch.compiler.disable graph break), which the speed layer now switches to automatically when a cache is engaged. Best-effort: a model whose block signature the hook does not recognise is caught and the load proceeds uncached. - new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache (enable_cache / apply_first_block_cache fallback; threshold auto-raised for a quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import). - diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph when a cache is engaged. - diffusion.py: apply_step_cache before compile; thread transformer_cache / transformer_cache_threshold through begin_load -> load_pipeline and report the engaged mode in status(). - models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and transformer_cache_threshold request fields, engaged mode in the status response. - hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold selection, and best-effort failure handling, plus route threading + validation. - scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM / LPIPS vs the compiled no-cache baseline). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears) The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with 'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2 *_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing min_features filter did not exclude them. Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed, guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers (M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and quantises these layers fine, so the exclusion is int8-only. Sequence embedders (context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note 'context_embedder' contains the substring 'text_embed', which is why the token is the specific 'time_text_embed', not 'text_embed'. Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s (5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already working) are unchanged. - diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes exclude_name_tokens; quantize_transformer passes it for int8 only. - hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps attention / FFN / sequence-embedder linears), while fp8 keeps them. - scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's Linear layers and derive the exclusion list. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears) The prequant-checkpoint builder applied the dense quant filter without the int8-only M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8 checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now produces a working checkpoint on every supported model, giving int8 (the consumer-preferred scheme) the same ~2x load-VRAM and download reduction fp8 already had. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the native stable-diffusion.cpp engine instead of diffusers, with diffusers as the guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM. - diffusion_engine_router: centralised engine selection (built on the existing select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason. - sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface backed by sd-cli, with lazy binary install, registry-driven asset fetch, step-progress parsing, and cancellation. - diffusion_families: per-family single-file VAE + text-encoder asset mapping. - sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled). - routes/inference + gpu_arbiter: drive the active engine via the router; the API now reports the active engine and any fallback reason. - tests for the backend, router, route selection, and cancellation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler Address review feedback on #6724: - engine router: unload the engine being deactivated on a switch, so the old model is not left resident-but-unreachable (the evictor only targets the active engine). - generate route: sd.cpp execution errors (nonzero exit / timeout / missing output) now map to 500, not 409 (which only means not-loaded / cancelled). - native batch: return per-image seeds and persist the actual seed for each image so every batch image is reproducible. - Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the stable-diffusion.cpp docs; other families keep sd-cli defaults. - honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload offload flags on the native load instead of hardcoding them off. - fail the load when the sd-cli binary is present but not runnable (version() now returns None on exec error / nonzero exit). - size estimate: only treat the transformer asset as a possible local path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in load_prequantized_transformer ends in torch.load(weights_only=False), which executes arbitrary code from the pickle. The transformer_prequant_path load-request field reached that unpickle for any local file an authenticated caller named, so a request could trigger remote code execution. Refuse the source.kind=='path' branch unless the operator sets UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted and unaffected. Document the requirement on the API field and add gate tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa - apply_attention_backend now restores the native default when no backend is requested or a kernel fails. diffusers keeps a process-wide active attention backend that set_attention_backend updates, and a fresh transformer's processors follow it, so a load that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile load pinned, breaking the bit-identical/off guarantee. - select_attention_backend drops flash3/flash4 up front when the CUDA capability is below Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit request on the wrong card set fine then crashed mid-generation; it now falls back to native. - Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a valid alias of native) is accepted instead of 422-rejected by Pydantic. - Drop the dead replace('-','_') normalization (no alias uses dashes/underscores). - perf_levers_probe.py output dir is now relative to the script, not a hardcoded path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF - apply_step_cache now engages only via the transformer's native enable_cache (the diffusers CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model now runs uncached per the best-effort contract. - GGUF transformers are quantized (the default Studio load path), so they now use the higher quantized FBCache threshold when the caller leaves it unset, instead of the dense default that could keep the cache from triggering. - fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so fullgraph=True failed warmup and silently measured an eager cached run); output dir is now relative to the script, not a hardcoded path. * Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder _is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate. Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0). * Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just TypeError on older torchao), which would break the mxfp8 config helper instead of falling back to the default. Catch both so the fallback is robust. quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds CUDA memory during the per-row VRAM probe; output dir relative to the script. * Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load - snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the whole snapshot. restore_backend_flags restores each flag independently so one failure can't leave the others leaked process-wide. - load_pipeline restores the flags (and clears the GPU cache) when the build fails after apply_speed_optims mutated the process-wide flags but before _state captured them for unload to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and contaminated later off generations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs without producing output (or without closing stdout) would never reach proc.wait and the wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the PROCESS, so the main thread always enforces the timeout and kills a hung process (which closes the pipe and ends the reader). Add a test that times out even when stdout blocks, and make the no-binary test hermetic so a host-installed sd-cli can't leak in. * Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn The filter callback can be invoked without a module name, so fqn.lower() would raise AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens, so the linear is kept) instead of crashing the quantise pass. * Studio diffusion (Phase 16) review fixes: native engine robustness - sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64); a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds aliased to the same image. Random seeds stay 53-bit (JS-safe). - sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub fall back to anonymous instead of failing auth on a blank token. - sd_cpp_backend: a superseding load now cancels the in-flight generation, so the old sd-cli can no longer return/persist an image from the previous model. - diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a slow 10+ GB free / CUDA sync does not block engine selection. - diffusion_engine_router: probe sd-cli runnability (version()) before committing to native, so a present-but-unrunnable binary falls back to diffusers at selection. - diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a CPU-only install can still reach the native sd.cpp engine instead of failing load. - tests updated for the runnability probe + a not-runnable fallback case. * Studio diffusion (Phase 9) review fixes: prequant safety + validation - SECURITY: a request-supplied local pre-quant path is now unpickled only when it resolves inside an operator-configured ALLOWLIST of directories (UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in, once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on any path a load request named (arbitrary code execution). realpath() blocks symlink escapes; a bare on/off toggle is no longer a wildcard. - Validate the checkpoint's min_features against the runtime Linear filter, so a checkpoint that quantised a different layer set is rejected instead of silently loading a model that mismatches the dense path while reporting the same scheme. - Tolerant base_model_id compare (exact or same final path/repo segment), so a local path or fork of the canonical base is accepted instead of falling back to dense. - _has_meta_tensors uses any(chain(...)) (no intermediate lists). - prequant verify/probe scripts use repo-relative paths (+ env overrides), not the author's absolute /mnt paths. - tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts - diffusion_memory: when group offload is unavailable and the plan falls back to whole-module offload, enable VAE tiling (the group plan left it off, but the fallback is the low-VRAM path where the decode spike can OOM). Covers both the group and sequential fallback branches. - perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a balanced bit-identity regression actually fails the check instead of exiting 0. - compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and the load-progress poll has a 30 min deadline instead of looping forever on a hang. - test for the group->model fallback enabling VAE tiling. * Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path - diffusion: a torchao-quantized transformer is committed only compiled. A dense model resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF it replaced), so when transformer_quant engaged and speed resolved to off, promote to default (regional compile); warn loudly if compile still does not engage. - diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK); otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF. - nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path). - test asserts the eager-quant -> default-compile promotion. * Studio diffusion (Phase 10) review fixes: attention gating + probe isolation - diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation, so auto now stays on native SDPA there. - diffusion_attention: _active_attention_backend handles get_active_backend() returning an enum/None (not a tuple); the old unpack always raised and was swallowed, so the native-restore short-circuit never fired. - perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul so the inductor_flags variant doesn't leak into later compiled rows. - tests for the SM80 cuDNN gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening - install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket); extract through a per-member containment check (Zip-Slip guard); expanduser the --install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch the separately-published cudart runtime DLL archive so sd-cli.exe can start. - sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie. - tests: Zip-Slip rejection, normal extraction, studio-home discovery. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of the literal --output path. SdCppEngine.generate checked only the literal path, so a batch generation would exit 0 and then raise 'no image' (or return a stale file). generate now returns the literal path when present and otherwise falls back to the numbered siblings; single-image behavior is unchanged. Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected without error. * Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats Codex review on the native engine arg builder: - build_sd_cpp_command emitted --width/--height unconditionally, so an img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of the input. width/height are now Optional (None = unset): an image-conditioned run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives the size from the input image (set_width_and_height_if_unset); a plain txt2img run with unset dims keeps the prior 1024x1024 default; explicit dims are always honored. width/height are read only by the builder, so the type change is local. - build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...) that silently swallowed repeats=0 into sd-cli's default of one pass, turning an explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError and emits the flag for any explicit value != 1. Tests: img2img unset dims omit width/height (init_img and ref_images), explicit dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at the default. (Two pre-existing binary-discovery tests fail only because a real sd-cli is installed in this dev environment; unrelated to this change.) * Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc Codex review: the transformer_prequant_path field description still told operators to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the prior security fix made that variable a directory allowlist -- _allowed_prequant_roots deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator following the documented =1 would have every transformer_prequant_path request silently refused. The description now states it must name one or more allowlisted directories and that a bare on/off value is not accepted. Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does not say =1, and describes an allowlist/directory (guards against doc drift). * Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset Codex review on attention-backend selection: - Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first generation with no fallback. select_attention_backend now applies _cudnn_attention_supported() to an explicit cuDNN request too. - flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3 is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a (min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+. - apply_attention_backend's success path left diffusers' process-wide active backend pinned to the kernel it set; a later component whose processors are unconfigured (backend None) would inherit it. It now resets the global registry to native after a successful per-transformer set (the transformer keeps its own backend), best-effort. Also fixed _active_attention_backend: get_active_backend() returns a (name, fn) tuple, so the prior code stringified the tuple and never matched a name, defeating the native-restore short-circuit. Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on SM90; global registry reset after a successful set; _active_attention_backend reads the tuple return. * Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300 (Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So _is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they are treated as data-center class and keep the intended fp8-first behavior. Test: extends the datacenter parametrize with 'NVIDIA B300' and 'NVIDIA GH200 480GB' (now _is_consumer_gpu False). * Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder Codex review: the M=1 modulation/embedder exclusion was wired only into the dense runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint quantised the AdaLN modulation and conditioning-embedder linears, and loading it via transformer_prequant_path (the load path only loads already-quantised tensors, it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes for the runtime path. Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 -> the M=1 exclusion, every other scheme -> none) and use it in both the runtime quantiser and the builder, so a prequant artifact's quantised-layer set always matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion). Test: int8_exclude_name_tokens returns the exclusion for int8 and () for fp8/nvfp4/mxfp8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race Codex review on the native-engine routing: - The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat) unconditionally after engine selection. A native sd.cpp load on a pure-CPU host never touches the GPU, so that needlessly tore down the resident chat model. The handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it. - sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when _run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so the setting was unverifiable. status now derives them from state.offload_flags (still 'none' on CPU, where the flags are empty). - _run_load committed the new state without cancelling/waiting on a generation that started during the (slow) asset download, so a stale sd-cli run against the OLD model could finish afterward and persist an image from the previous model once the new load reported ready. The commit now signals the in-flight cancel and waits on _generate_lock before swapping _state (taken only at commit, so the download never serialises against generation), mirroring the diffusers load path. Tests: CPU native load skips the arbiter while a GPU native load takes it; status reports offload active when flags are set; _run_load cancels and waits for an in-flight generation before committing. * Studio diffusion (Phase 14) review round 2: align helper name with the stack Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the identical helper already present higher in the diffusion stack (Phase 16). The helper definition, the runtime quantiser call, and the offline builder are now byte-identical to that version, so the two branches no longer introduce a divergent name for the same single-source-of-truth and the stack merges without a conflict on this fix. No behavior change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: eager patches + torch.compile cache speed phase Adds the opt-in speed path for the GGUF diffusion transformer behind a selectable speed mode (default off, so output is unchanged until a profile is chosen): - diffusion_eager_patches.py: shared eager fast-paths (channels_last, attention/backend selection, fused norms and QKV) installed at load and rolled back on unload or failed load. - diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent torch.compile cache and the GGUF-transformer compile wiring. - diffusion_arch_patches.py: architecture-specific patches. - diffusion_patch_backend.py: shared install/restore plumbing. - diffusion_speed.py: speed-profile planning. Tests for each module plus the benchmarking and probe scripts used to measure speed, memory, and accuracy of the path. * Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI Backend: - Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file fp8 transformers, gated to the unsloth org plus a curated allowlist. - Image-conditioned workflows built with Pipeline.from_pipe so they reuse the loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint, outpaint, and a hires-fix upscale pass. - Instruction editing as its own family kind (Qwen-Image-Edit-2511, FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and multi-reference) plus klein inpaint. - Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the decoded image size and cap upscale output to avoid OOM on large inputs. - Fixes: from_pipe defaulting to a float32 recast that crashed torchao quantized transformers; image-conditioned calls forcing the slider size onto the input image. Native sd.cpp engine rejects image-conditioned and reference requests it cannot serve. Frontend: - Redesigned Images page with capability-gated workflow tabs (Create, Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor, client-side outpaint, and a multi-reference picker. - Advanced options moved to a right-docked panel mirroring Chat: closed by default, toggled by a single fixed top-bar button that stays in place. sd.cpp installer: pin the release, verify each download's sha256, add a download timeout, and make the source repo configurable for a future mirror. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: correct the Advanced panel comment (closed by default, fixed toggle) * Studio diffusion: LoRA adapters for the Images workflow Add community LoRA support across both diffusion backends, the single biggest step toward broad image-workflow coverage. Backend - New shared module core/inference/diffusion_lora.py: adapter discovery (local scan + curated catalog + owner/name[:file] Hub refs), download via hf_hub_download_with_xet_fallback, alias sanitization, native managed-dir materialization with collision-broken aliases, prompt-tag injection (deduped against user-typed tags), and a supports_lora gate. - Native sd-cli: resolve + materialize selected LoRAs into a per-run managed dir, inject <lora:ALIAS:w> tags, pass --lora-model-dir with --lora-apply-mode auto. The arg builder already emitted these flags. - Diffusers: non-fused load_lora_weights + set_adapters manager, tracked on the pipe so an unchanged selection is a no-op and a model swap resets; cleared on unload. Never fuses (breaks quantized transformers and blocks live weight tweaks). - Gated off where unsupported: torchao fp8/int8 dense, GGUF-via-diffusers, and native Qwen-Image (no LoRA name-conversion branch upstream). - Request contract: optional loras on DiffusionGenerateRequest; empty or omitted is identical to today. supports_lora surfaced in status; chosen LoRAs persisted in gallery recipe metadata. - New GET /api/models/diffusion-loras for the picker (family-filtered). Frontend - Repeatable multi-LoRA picker (adapter select + weight slider 0..2 + remove), gated by the loaded model's supports_lora and family, max 8. Tests - New test_diffusion_lora.py (14): helpers, request validation, native tag/dir wiring, diffusers set_adapters manager, supports_lora matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: ControlNet for the Images workflow (diffusers) Add ControlNet conditioning, the #2 most-used diffusion workflow after LoRA, on the diffusers backend for the families with ControlNet pipelines (FLUX.1 and Qwen-Image), with Union models as the default picks. Backend - New core/inference/diffusion_controlnet.py: family-gated discovery (curated Union models + local dirs + bare owner/name repos), resolution to a loadable repo/dir, control-image preprocessing (passthrough + a dependency-free canny edge map), and a supports_controlnet gate. - diffusion.py: a ControlNet manager parallel to the LoRA one. Loads the (small) ControlNet model once via from_pretrained (cached by id) and builds the family's ControlNet pipeline via Pipeline.from_pipe(base, controlnet=model), reusing the resident base modules at their loaded dtype (no reload, no recast). Passes the control image + conditioning scale + guidance start/end at generate time; cleared on unload. - Families: FLUX.1 -> FluxControlNetPipeline/Model, Qwen-Image -> QwenImageControlNetPipeline/Model. Others declare none (gated off). - Gated off for the native engine, GGUF-via-diffusers, and torchao fp8/int8 dense (same rule as LoRA). v1 conditions txt2img only. - Request contract: optional controlnet on DiffusionGenerateRequest; supports_controlnet in status; the choice persisted in gallery meta. - New GET /api/models/diffusion-controlnets for the picker. Frontend - A ControlNet control in the Images rail (model select + control-image upload + control-type select + strength slider), gated by the loaded model's supports_controlnet + family, shown for text-to-image. Tests - New test_diffusion_controlnet.py (10): discovery/resolve/preprocess/gate helpers, request validation, family wiring, and the diffusers pipe manager (loads once, caches, from_pipe with controlnet, rejects unsupported families). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio ControlNet: show the picker on the Create tab (workflow id is 'create', not 'txt2img') The ControlNet control gated on workflow === 'txt2img', but the Images workflow tab ids are create/transform/inpaint/extend/upscale/reference/edit -- there is no 'txt2img'. So the picker never rendered even with a ControlNet-capable model loaded. Gate on 'create' (the text-to-image tab) for both the picker and the request wiring. Found via a live Playwright capture of the running Studio. * Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a stray "gguf" tag on the Hub but ship no .gguf files. The model search classified them as GGUF from the bare tag, so the picker rendered the GGUF variant expander, which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata (populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected. * Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline load when the id is unsloth-hosted or on-device (the backend infers the family + base repo and gates loads to unsloth/* or local paths), and show a clear message otherwise instead of silently ignoring the click. Curated and GGUF paths are unchanged. * Studio Images: keep curated safetensors models in Recommended after download The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet), so they vanished from the picker after the first load and could only be found by typing an exact search. The row already renders a downloaded badge, matching how GGUF Recommended rows stay visible when cached. Drop the exclusion so the curated safetensors always list. * Studio diffusion LoRA: sanitize dots out of adapter aliases The LoRA alias is used as the diffusers PEFT adapter name, and PEFT rejects names containing "." (module name can't contain "."). sanitize_alias kept dots, so a LoRA whose filename carries a version tag (e.g. Qwen-Image-2512-Lightning-8steps-V1.0-bf16) failed to apply with a 400. Replace dots too; the alias stays a valid native <lora:NAME:w> filename stem. Adds regression coverage for internal dots. * Studio Images: clarify the GGUF transformer-quant Advanced control Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode" with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/ FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF, falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only. * Studio Images: list on-device unsloth diffusion models in the picker The Images picker's On Device tab hid every non-GGUF cached repo whenever a task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit and FP8 safetensors) never showed up there. List cached repos that pass the task gate, limited under a filter to unsloth-hosted ones so base repos (which fail the diffusion load trust gate) don't appear only to dead-end on click. Chat behavior is unchanged: the task gate still drops image repos there. * Studio: hide single-file image checkpoints from the chat model picker The chat picker treats a cached repo as an image model, and hides it, only when it ships a diffusers model_index.json. Single-file, ComfyUI, and ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models. Fall back to resolving the repo id against the known diffusion families, the same resolver the Images backend loads from, so these checkpoints are tagged text-to-image and stay in the Images picker only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: add the FLUX.2-dev model family Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline) resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline + Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated, reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror. text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from klein's turbo defaults. Verified live: GGUF load resolves the family + gated base repo and generates a real 1024x1024 image on GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: clearer error for an unsupported diffusion model When a repo id resolves to no diffusion family the load raised 'Could not infer a diffusion family... Pass family_override (z-image)', which points at an unrelated family and doesn't say what is supported. Replace it with a message that lists the supported families (from a new supported_family_names helper) and notes that video models and image models whose diffusers transformer has no single-file loader are not supported. Applies to both the diffusers and native sd.cpp load paths. Also refreshes two stale family-registry comments that still called FLUX.2-dev omitted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add SDXL diffusion family (U-Net pipeline support) SDXL is the first U-Net family in the diffusion backend: its denoiser is pipe.unet (UNet2DConditionModel), not a DiT pipe.transformer, and a single-file .safetensors is the whole pipeline rather than a transformer-only file. The backend previously assumed a DiT transformer everywhere, so add the two hooks a U-Net family needs and register SDXL. DiffusionFamily gains denoiser_attr ("transformer" for DiT, "unet" for SDXL) and single_file_is_pipeline (SDXL loads a single file via pipeline_class.from_single_file with the base repo as config, instead of transformer_class.from_single_file plus a companion assembly). _align_vae_dtype now reads the denoiser generically so img2img and inpaint keep the VAE and U-Net dtypes aligned. The non-GGUF trust gate is extended with a short, exact-match, safetensors-only allowlist of official base repos (the SDXL base/refiner and sdxl-turbo), because SDXL ships only as a full pipeline and has no unsloth-hosted GGUF. Local paths stay trusted as before; a random repo, even one that detects as SDXL, is still rejected. The image-conditioned and ControlNet workflows are the standard SDXL pipelines, built around the resident modules via from_pipe like every other family, so SDXL gets txt2img, img2img, inpaint, outpaint, upscale, LoRA and ControlNet. There is no native sd.cpp mapping yet, so the no-GPU route falls back to diffusers. Frontend catalog gains SDXL Base 1.0 and SDXL Turbo entries with SDXL step/guidance defaults (Turbo: few steps, no CFG; base: ~30 steps, real CFG). Tests: new test_diffusion_sdxl.py (family shape, detection, trust allowlist, model kind, U-Net VAE-dtype alignment, LoRA gate) plus loader-branch tests in test_diffusion_backend.py (pipeline-kind from_pretrained, single-file whole-pipeline from_single_file, allowlist accept/reject). Verified live on GPU: sdxl-turbo loads both as a pipeline and as a single file and generates coherent txt2img + img2img. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images: LoRA free-text Hugging Face entry + recipe round-trip The backend has always accepted a bare Hugging Face repo id (owner/name, or owner/name:weight-file.safetensors) as a LoRA, downloading and applying it. But the picker only rendered when the curated catalog had entries, and the catalog is empty, so there was no UI path to apply any LoRA. Show the LoRA section whenever the loaded model supports LoRA, and replace the curated-only dropdown with a text input: type a Hub repo id, or pick a discovered adapter from a datalist of suggestions when the catalog is populated. Also restore LoRAs when loading a recipe. restoreSettings now parses the recipe's "id:weight" strings (splitting on the last colon, since the id itself may contain one for a specific weight file) back into the selection, so replaying a saved image reproduces its adapters. The generate payload trims hand-typed ids and drops empty / zero-weight rows, and a model swap clears the selection (a LoRA is family-specific) without discarding a free-text pick that is not in the curated list. * Add diffusion LoRA training (SDXL text-to-image) First diffusion training path in Studio: train a LoRA on the SDXL U-Net from an image + caption dataset and export it as a diffusers .safetensors that the existing diffusion LoRA loader (and any diffusers pipeline) can load. core/training/diffusion_lora_trainer.py: - DiffusionLoraConfig with validation/defaults (rank, alpha, targets, lr, steps, grad accumulation, resolution, min-SNR gamma, gradient checkpointing, lr scheduler, seed, mixed precision). - discover_image_caption_pairs: captions from metadata.jsonl / captions.jsonl, per-image .txt/.caption sidecars, or a dreambooth instance_prompt fallback (pure, unit-tested). - run_diffusion_lora_training: the loop -- freeze base, PEFT-wrap the U-Net attention projections, VAE-encode (fp32 VAE to avoid the SDXL fp16 overflow), sample noise + timesteps, predict, MSE loss with optional min-SNR weighting (epsilon / v-prediction), AdamW + get_scheduler + grad accumulation + grad clipping, then export via save_lora_weights. Emits worker-protocol events (model_load_*, progress, complete) and polls should_stop for a clean stop with a partial save. - run_diffusion_training_process: mp.Queue subprocess adapter (event_queue / stop_queue), so the training worker can spawn it; plus a CLI entry point. Only SDXL (U-Net) is trained here; DiT families and the Studio UI form + route wiring are follow-ups. The trainer is decoupled and worker-ready. Tests: test_diffusion_lora_trainer.py covers caption discovery (metadata / sidecar / instance prompt / skip-uncaptioned / errors), config normalisation + validation, the SDXL add-time-ids, and the dict->config adapter. Verified live on GPU: a 60-step SDXL LoRA run lowers the loss, exports a ~45 MB adapter, and loading it back shifts generation from baseline (mean abs pixel diff ~55/255). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * diffusion trainer: emit learning_rate in progress events (Studio pump compatibility) The Studio training pump reads 'learning_rate' from progress events; the diffusion trainer emitted 'lr'. Rename the field (and the CLI reader) so the trainer's events are directly consumable by the existing training status/SSE machinery when it is wired into the worker, without a translation shim. * Wire diffusion LoRA training into the Studio API Make the SDXL LoRA trainer reachable from the app with a small, self-contained job service and JSON routes, deliberately separate from the LLM TrainingBackend (whose lifecycle -- LLM config build, per-run SQLite rows, matplotlib plots, transfer-to-chat- inference -- is text-training specific and would mis-handle a diffusion run). core/training/diffusion_training_service.py: DiffusionTrainingService runs one job at a time -- validate the config cheaply (before any spawn), spawn the trainer subprocess (spawn context, parent-lifetime bound), pump its events (model_load_* / progress / complete / error) into an in-memory status snapshot, and support a clean stop. The subprocess context and target are injectable so the full start -> pump -> status -> complete path is unit-tested without real multiprocessing or torch. routes/training.py: POST /api/train/diffusion/start (400 on a bad config, 409 when a job is already running), POST /api/train/diffusion/stop, GET /api/train/diffusion/status (JSON poll). models/training.py: DiffusionTrainingStartRequest + response schemas mirroring DiffusionLoraConfig, so model_dump() passes straight through. Tests: test_diffusion_training.py -- service happy path, bad-config-before-spawn, concurrent-job rejection, clean stop, crash-without-terminal-event, event transitions; plus route wiring via the FastAPI TestClient (start / 422 / 400 / 409 / status / stop) with a mocked service. The diffusion trainer's progress events already use the field names this path expects. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images: add a Train LoRA (SDXL) dialog Surface the diffusion training API in the Images page. A "Train LoRA" button in the top bar opens a self-contained dialog to fine-tune an SDXL LoRA on a folder of images: pick the base model, dataset folder, output folder, an optional instance prompt, and the core hyperparameters (steps, rank, resolution, batch, learning rate), then Start. The dialog polls the training status while open and shows a progress bar, step count, live loss, and the saved adapter path, with a Stop button for a clean stop. The dialog is independent of the loaded generation model (training runs in its own subprocess), and prefills the base model with the loaded checkpoint when it is SDXL, else the SDXL base. api.ts gains startDiffusionTraining / stopDiffusionTraining / getDiffusionTrainingStatus plus their types, matching the /api/train/diffusion routes. * Import diffusion training schemas from models.training directly The import-hoist lint flags newly re-exported names in the models/__init__.py hub as unused (it does not treat __all__ membership as a use). Import the three diffusion training schemas straight from models.training in routes/training.py, where they are used in the route annotations and calls, and drop the __init__ re-export. * Remove stray async task scratch outputs committed by mistake * ControlNet: reject filesystem-like ids and do not cache a model past an unload race Two review findings on the ControlNet path: - resolve_controlnet's bare-repo fallback accepted any id with a slash, so a path-shaped id (/tmp/x, ../x) reached from_pretrained as a local directory. Restrict the fallback to a strict owner/name HF repo id shape. - _controlnet_pipe now re-checks the cancel event after the blocking from_pretrained: an unload that raced the download had already cleared the caches, so caching the late module would pin it past the unload. * Pipeline prefetch: fetch only the default torch weights A full-pipeline prefetch kept every repo file outside assets/, so an official repo that ships multiple formats (SDXL Base: fp16 variants, ONNX, OpenVINO, Flax, a top-level single-file twin) downloaded tens of GB from_pretrained never loads. Skip non-torch exports and dtype-variant twins in _pipeline_file_downloaded, and drop a component .bin when the same directory carries a picked safetensors weight (diffusers' own preference). * Diffusion LoRA training: fall back to fp16 when CUDA lacks bf16 The default mixed_precision=bf16 hard-fails on pre-Ampere GPUs (T4 / V100 / RTX 20xx) which have no bf16 compute; check torch.cuda.is_bf16_supported() and drop to fp16 there. * Diffusion training service: join the old pump outside the lock start() joined a finished job's pump thread while holding the service lock, but the pump's final state writes need that same lock, so the join always burned its full timeout and a stale pump could then overwrite the new job's state. Join outside the lock (with a re-check after), and fence _apply_event and the exit handler by process identity so a superseded pump can never touch the current job's state. Adds regression tests for both. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA training: harden config handling, cancellation, SDXL conditioning, and safety Addresses review findings on the SDXL LoRA trainer: - Gate the base model with the same trust check as inference (unsloth/*, allowlisted official bases, or a local path) before from_pretrained, so an untrusted remote repo is never fetched or deserialised. - Check the stop signal before the (slow) model load, not only between steps, so a cancel during download is honoured; a stop may carry save=False to cancel without leaving a partial adapter. - Per-sample SDXL add_time_ids from the actual crop (original size + crop offset, with the offset mirrored on horizontal flip) instead of a fixed uncropped-square tensor. - Apply EXIF orientation before resize/crop so rotated photos train upright. - Skip gradient clipping when max_grad_norm <= 0 (the Studio 'disable' value) instead of scaling every gradient to zero. - Coerce Studio config strings/blanks: learning_rate string to float, blank hf_token to anonymous, gradient_checkpointing 'none'/'true'/'unsloth' to bool; reject a zero/negative lora_alpha or learning_rate. - Alias the generic Studio training payload keys (model_name/max_steps/batch_size/lora_r/ lr_scheduler_type/random_seed) onto the diffusion field names. - Mirror the trained adapter into loras/diffusion so the Images LoRA picker discovers it. - Report worker exceptions in both message and error keys so the failure is not lost. Adds regression tests for the config coercion/validation and aliasing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ControlNet: address review findings on the diffusers path - resolve_controlnet enforces catalog family compatibility so a direct API call cannot load a ControlNet built for another family through the wrong pipeline. - Unknown ControlNet ids now surface as a 400 (call site maps FileNotFoundError to ValueError) instead of a generic 500. - strength 0 disables ControlNet entirely, so a no-op selection never pays the download / VRAM cost; the control image is decoded and validated BEFORE the ControlNet is resolved or built, so a malformed image fails fast for the same reason. - ControlNet loads use the base compute dtype (state.dtype is a display string, not a torch.dtype, so it silently fell back to float32) and honor the base offload policy via group offloading instead of forcing the module resident. - Empty/malformed HF token coerced to anonymous access. - Flux Union ControlNet control_mode mapped from the selected control type. - resolve_controlnet drops the unused hf_token/cancel_event params. - ControlNetSpec validates guidance_start <= guidance_end (clean 422). - Images UI ControlNet Select shows its placeholder when nothing is selected. Adds regression tests for family enforcement and the union control-mode map. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion training API: LLM interlock, pre-spawn VRAM free, path containment, no dropped knobs Four review findings on the diffusion training start path: - It spawned the SDXL trainer without checking the LLM TrainingBackend, so a start while an LLM run was active put two trainers on the same GPU. Add a symmetric interlock: diffusion start returns 409 when LLM training is active, and LLM start refuses while a diffusion job is active. - It went straight to service.start() without freeing GPU residents. Add a pre-spawn free of the export subprocess, the resident Images pipeline (with an arbiter release), and chat models, mirroring the LLM start path. - data_dir / output_dir were passed through unresolved, so Studio-relative names failed and absolute paths bypassed containment. Resolve them with resolve_dataset_path / resolve_output_dir before spawn (400 on an uncontained path). - The request model dropped max_grad_norm and lora_target_modules, so runs that set them trained with defaults. Add both fields. The gemini pump-join deadlock was already fixed earlier (join outside the lock + proc-identity fence). Note: honoring a stop DURING model load is a trainer-loop change owned by the diffusion training engine PR (should_stop polled before the first optimizer step). Adds route + model regression tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA: harden resolution, native tag precedence, and diffusers teardown Address review findings on the LoRA path: - resolve_one: normalise a blank/whitespace hf_token to None (anonymous access) and reject a client-supplied weight file with traversal / absolute path. - resolve_specs: convert FileNotFoundError from an unknown/stale id to ValueError so the route returns 400 instead of a generic 500. - _scan_local: disambiguate local adapters that share a stem (foo.safetensors vs foo.gguf) so each is uniquely addressable. - inject_prompt_tags: the backend-validated weight now wins over a user-typed <lora:ALIAS:...> for a selected adapter; unselected user tags are left alone. - diffusers _apply_loras: reject a .gguf adapter with a clear error before touching the pipe (diffusers loads safetensors only). - _unload_locked: drop the explicit unload_lora_weights() on teardown; the pipe is dropped wholesale (freeing adapters), so the previous call could race an in-flight denoise on the same pipe. - Images page: use a stable LoRA key and clear the selection (not just the options) when the catalog refresh fails. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion: guard trust check against OSError and validate conditioning inputs - _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid characters (or a bare owner/name id) can't raise OSError; treat any failure as not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request still raises the clear FileNotFoundError for a genuinely missing local pick. - generate(): reject mask_image / upscale / reference_images supplied without an input image, and reject reference_images on a family that does not support reference conditioning, instead of silently degrading to txt2img / img2img. * SDXL: reject GGUF up front, skip unused base weights, drop refiner, and harden helpers Addresses review findings on the SDXL family: - Reject a GGUF load for single_file_is_pipeline families (SDXL) in validate_load_request, before the route evicts the current model; SDXL has no transformer-only GGUF variant. - Skip base-repo weight files when a whole-pipeline single file is loaded: from_single_file (config=base) needs only the base config/tokenizer/scheduler, so a local .safetensors no longer triggers a multi-GB base download. - Remove the SDXL refiner from the non-GGUF trust allowlist: it is an img2img-only pipeline but this backend loads every sdxl repo as the base txt2img pipeline. - Normalize a blank/whitespace hf_token to None once in load_pipeline so every load branch degrades to anonymous instead of erroring on a malformed token. - Read the denoiser dtype from a parameter (compile-wrapped modules may lack .dtype) and access state.family.denoiser_attr directly. Adds/updates regression tests for the trust allowlist, GGUF rejection, and base-config filter. * Images: preserve restored LoRAs through model load and never send hidden LoRAs - The LoRA effect cleared the selection on every load->capable transition, which wiped adapters restored from a gallery recipe before the model finished loading. Track the previously-loaded family in a ref and clear only on a real family swap; keep the selection on the initial load and on unload. - Gate the generate payload's loras on loraCapable so a restored selection that is hidden (loaded model does not support LoRA) is never sent to the backend. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images Train LoRA dialog: token, validation, precision, base-repo prefill, gating, refresh Nine review findings on the SDXL training dialog: - Forward the saved Hub token so a gated/private SDXL base can be trained (the image load flow already sends it). - Re-seed the base-model field from the current default each time the dialog opens; the keep-alive dialog otherwise kept its mount-time default after a model loaded. - Prefill from base_repo (the diffusers pipeline) rather than repo_id, which for a GGUF/single-file SDXL load is the checkpoint path from_pretrained can't open. - Add client-side validation of steps/rank/resolution/batch/learning-rate before the request. - Expose a precision selector (bf16/fp16/fp32) so non-bf16 GPUs can train from the UI, not only the API. - Gate the dialog on the active Images route (active && trainOpen) so switching tabs closes it and stops its polling. - Rescan the LoRA picker when a run completes, so a freshly-trained adapter appears without a model reload. - Cap the dialog height and scroll the body so the Start/Stop footer stays reachable on short viewports. - Correct the copy to not over-promise picker auto-discovery. Freeing the resident Images pipeline before training is handled backend-side in the diffusion training start route. * Merge diffusion-sdxl into diffusion-lora-ux; keep options-only LoRA catch The catalog-refresh .catch from the lower branch clears the selected adapters too, which is right for its catalog-only picker but wrong here: this picker holds free-text HF repo ids that are valid without being in the catalog, so a transient refresh failure must not wipe them. Family swaps still clear the selection and hidden LoRAs are never sent. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Train LoRA dialog: stop suggesting absolute paths the backend rejects The dataset and output placeholders showed /path/to/... examples, but the training routes resolve those fields inside the Studio home and reject absolute paths outside the approved roots, so following the placeholder produced a 400. Use folder-name placeholders and say in the labels and the dialog description where each folder resolves. * Align the VAE to the denoiser's first FLOATING dtype, not its first parameter A GGUF-quantized transformer's leading parameters are packed uint8 storage, so reading next(parameters()).dtype handed nn.Module.to() an integer dtype and every image-conditioned generation on a GGUF model (Qwen-Image-Edit) failed with a 500. Probe the parameters for the first floating dtype, treat an all-integer module as a no-op, and also catch TypeError so an unexpected dtype can never break generation. Regression test included. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Count LR scheduler warmup/decay in optimizer steps, not micro-steps lr_sched.step() runs once per outer optimizer step (after the gradient accumulation inner loop), for train_steps total. The scheduler was configured with num_warmup_steps and num_training_steps multiplied by gradient_accumulation_steps, so with accumulation > 1 a warmup or non-constant schedule stretched past the run and never reached the intended decay. Count both in optimizer steps. * Address Codex review findings on the image-workflows PR Keep diffusion.py importable without torch: the compile/arch patch modules import torch at module level, so import them lazily at their load/unload call sites instead of at module load. This restores the torchless contract so get_diffusion_backend() works on a CPU/native sd.cpp install. Match family reject keywords and aliases as whole path/name segments, not raw substrings, so an unrelated word like edited, edition, or kontextual no longer misroutes or hides a valid base image model, while supported edit families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same segment matching in the picker task filter. Route FLUX.2-dev native guidance through --guidance like the other FLUX families rather than --cfg-scale. Reject native upscale requests that have no input image. Read image header dimensions and reject over-limit inputs before decoding pixels, so a crafted small-payload image cannot spike memory. Reject an upscale that would shrink the source below its input size. Validate the model_kind against the filename extension before the GPU handoff. Estimate a local diffusers pipeline's size from its on-disk weights so auto memory planning does not skip offload and OOM. Report workflows: [txt2img] from the native backend status so the Create tab stays enabled for a loaded native model. Clamp the outpaint canvas to the backend's 4096px decode limit. Adds regression tests for segment matching and kind/extension validation. * Guard inference loads and worker lifetime against diffusion training Teach the chat and image load guards about an active diffusion (SDXL) LoRA job: a chat load is refused (its footprint cannot be fit-checked against the trainer) and an image load is refused outright, mirroring the existing LLM training guards, so a load can no longer allocate GPU memory alongside the trainer and undo the pre-start cleanup. Bind the diffusion trainer subprocess to the parent's lifetime and scrub the native path lease secret from it by running the child through run_without_native_path_secret, matching the inference/export/LLM workers, so a Studio crash or kill no longer leaves the trainer holding the GPU. Reset in_model_load on the complete and error terminal events: a stop or failure during model loading otherwise leaves the status reporting a stale loading indicator after the job has ended. * Harden diffusion LoRA handling on the diffusers and native paths Reject LoRA on a torch.compile'd diffusers transformer (Speed=default/max): diffusers requires the adapter loaded before compilation, so applying one to the already-compiled module fails with adapter-key mismatches. The status gate now hides the picker and generate raises a clear message instead. Convert a cancelled Hub LoRA download (RuntimeError Cancelled) to the diffusion cancellation sentinel in resolve_specs, so an unload/superseding load during resolution maps to a 409 instead of a generic server error. Drop weight-0 LoRA rows before the native support gate so a request carrying only disabled adapters stays a no-op on families where native LoRA is unsupported, matching the diffusers path. Reject duplicate LoRA ids in the request model: both apply paths suffix colliding names, so a repeated id would stack the same adapter past its per-adapter weight bound. Strip all user-typed <lora:...> prompt tags on the native path (only the selected adapters are materialized in the managed lora-model-dir, so an unselected tag can never resolve), and restore saved LoRA selections from a gallery recipe so restore reproduces a LoRA image. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden ControlNet resolve, gallery metadata, and the control-type picker Check cancellation immediately after a ControlNet from_pretrained and before any device placement, so an unload/eviction that raced the download does not allocate several GB onto the GPU after the load was already cleared. Require a loadable weight or shard index (not just config.json) before a local ControlNet folder is advertised, so an interrupted copy is hidden instead of failing deep in from_pretrained as a generic 500. Do not record a strength-0 ControlNet in the gallery recipe: it is treated as disabled and skipped, so the image is unconditioned and the metadata must not claim a ControlNet was applied. Build the control-type picker from the selected ControlNet's advertised control_types instead of a hardcoded passthrough/canny pair, so a union model with a precomputed depth or pose map sends the correct control_mode. * Address further Codex findings on the image-workflows PR - Persist the actual output image size in the gallery recipe instead of the request sliders: Transform/Inpaint/Edit derive the size from the uploaded image, Extend grows the canvas, and Upscale resizes it, so the sliders recorded (and later restored) the wrong dimensions for those workflows. - Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file name) in validate_load_request, so the unloadable pick fails before chat is evicted rather than deep in from_pretrained. - Only publish an image-conditioned from_pipe wrapper to the shared aux cache when the load is still current: from_pipe runs under the generate lock but not the state lock, so an unload racing its construction could otherwise cache a wrapper over torn-down modules that a later load would reuse. - Verify the Windows CUDA runtime archive checksum before extracting it, like the main sd-cli archive, so a corrupt or tampered runtime is rejected rather than extracted next to the binary. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse non-SDXL base models at diffusion training start The trainer only supports the SDXL U-Net, but a FLUX / Qwen-Image / Z-Image repo or a GGUF filename passed as base_model was accepted and then failed minutes later inside StableDiffusionXLPipeline.from_pretrained with an unrelated-looking error. Add a name-based guard in normalized() so known DiT-family names and .gguf checkpoints are rejected up front, which the API start route surfaces as an immediate 400 with a message that says exactly which bases are trainable. Unrecognisable names still pass through so custom local SDXL checkpoints keep working. * Add diffusion dataset upload and training info endpoints Training an image LoRA required knowing the Studio home layout and copying files onto the server by hand, which is the most confusing step of the whole flow. Two small endpoints fix that: - GET /api/train/diffusion/info reports the datasets and outputs roots plus every dataset folder that contains images (with image/caption counts), so the UI can offer a picker instead of a blind free-text path. - POST /api/train/diffusion/dataset uploads images and optional caption .txt / metadata.jsonl files into a named folder under the datasets root, creating it on first use and accumulating on repeat uploads so large sets can arrive in batches. Names are validated to a single path component and files stream to disk under the same per-upload size cap as LLM dataset uploads. The returned name is a valid data_dir for /diffusion/start. * Rework the Train LoRA dialog into a guided SDXL flow The dialog assumed users knew the Studio home layout and that only SDXL is trainable, and hid both facts behind free-text fields. Restructure it around the three real decisions: - Base model is a dropdown of the trainable SDXL picks (Base 1.0, Turbo, the loaded SDXL pipeline when there is one) with a custom repo/path escape hatch, instead of a bare text field defaulting to a repo id. - Training images come from an in-browser upload (new dataset endpoints) or a picker over existing dataset folders with image/caption counts. No shell access or knowledge of the datasets root is needed any more, and the captioning rules are explained inline. - The output field is now Adapter name and the instance prompt is labelled as the trigger prompt, with a no-captions warning wired to the selected dataset's actual caption count. Hyperparameters collapse behind a training settings toggle since the defaults suit a first run. A completed run says where the adapter went and offers Done / Train another, and the top-bar button gets an icon and a plainer description. The dialog title states the SDXL-only scope and that other families load LoRAs but cannot train them yet. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Validate diffusion training config before freeing the GPU The start route freed resident GPU workloads (export, Images pipeline, chat) before the service validated the config, so a start that was then refused, now including a non-SDXL base model, tore down the user's loaded model for nothing. Run the same cheap normalise pass first; the LLM path already follows this rule via its before_spawn hook. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
|
Daniel Han
|
c9cf7a55ca
|
Wire diffusion LoRA training into the Studio API (#6792)
* Studio diffusion: cross-platform device policy, fp16 guard, lock split, validate-before-evict Phase 1 of porting the richer diffusion stack onto the image-generation backend. - Add a compartmentalized device/dtype policy module (diffusion_device.py) resolving CUDA/ROCm/XPU/MPS/CPU with capability flags. Keeps the NVIDIA capability-based bf16 choice; ROCm and XPU are isolated; MPS uses bf16 or fp32, never a silent fp16 that renders a black image. - Add a per-family fp16_incompatible flag (Z-Image) and promote a resolved float16 to float32 for those families so they do not produce black images. - Split the backend locks: a generation holds only _generate_lock, so status, unload, and a new load are never blocked by a long denoise. Add per-generation cancellation via callback_on_step_end so an eviction or a superseding load preempts a running generation; a replacement load waits for it to stop before allocating, so two pipelines never sit in VRAM at once. - Validate a load request before the GPU handoff so an unloadable pick never evicts a working chat model, and reject missing local paths up front. - Add CPU-only tests for the device policy, dtype guard, lock split and cancellation, and validate-before-evict, plus a GPU benchmark/regression script (scripts/diffusion_bench.py) measuring latency, peak VRAM, and PSNR against a saved reference. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2A): measured-budget memory planner + offload/VAE policy Add a lean, backend-agnostic memory policy that picks a CPU-offload policy and VAE tiling/slicing from measured free device memory vs the model's estimated resident footprint, then applies it to the built pipeline. auto stays resident when the model fits (byte-identical to the prior resident path), and falls to whole-module offload when tight; fast/balanced/low_vram are explicit overrides. Sequential submodule offload is unreliable for GGUF transformers on diffusers 0.38, so it falls back to whole-module offload and status reports the policy actually engaged. Verified on Z-Image-Turbo Q4_K_M (B200): auto reproduces the resident image with no VRAM/latency regression (PSNR inf); balanced/low_vram cut generation peak VRAM 47.9% (15951 -> 8318 MB) with byte-identical output, at the expected latency cost. 73 prior + 35 new CPU tests pass. * Studio diffusion (Phase 2D): streamed block-level offload + functional VAE tiling Add a streamed 'group' offload tier (diffusers apply_group_offloading, block_level, use_stream) that keeps the transformer flowing through the GPU a few blocks at a time while the text encoder / VAE stay resident, and fix VAE tiling to drive the VAE submodule (pipelines like Z-Image expose enable_tiling on pipe.vae, not the pipeline). apply_memory_plan now returns the (policy, tiling) actually engaged so status never overstates either, and group falls back to whole-module offload when the transformer can't be streamed. Measured on Z-Image (B200), all lossless (PSNR inf vs resident): balanced/group cuts generation peak VRAM 32% (15951 -> 10840 MB) at near-resident speed (2.07 -> 2.99s); low_vram/model cuts it 48% (-> 8318 MB) but is slower (7.99s). Mode names now match that tradeoff: balanced = stream the transformer, low_vram = offload every component. auto picks group when the companions fit resident, else model. 112 CPU tests pass. * Studio diffusion (Phase 5): image quality-vs-quant accuracy harness Add scripts/diffusion_quality.py, the accuracy analogue of the KLD workflow: hold prompt + seed fixed, render a grid with a reference quant (default BF16), then render each candidate quant and measure drift from the reference. Records mean PSNR + SSIM (pure-numpy, no skimage/scipy) and optional CLIP text-alignment + image-similarity (transformers, --clip), plus file size, latency, and peak VRAM, then prints a quality-vs-cost table and recommends the smallest quant within a quality budget. --selftest validates the metrics on synthetic images with no GPU or model. Verified on Z-Image (B200): the table degrades monotonically with quant size (Q8 -> Q4 -> Q2: PSNR 21.7 -> 15.5, SSIM 0.82 -> 0.61), while CLIP-text stays flat (~0.34) -- quantization erodes fine detail far more than prompt adherence. * Studio diffusion (Phase 3): opt-in speed layer (channels_last / compile / TF32) Add a speed_mode knob (off by default, so the render path stays bit-identical): default applies channels_last VAE + regional torch.compile of the denoiser's repeated block where eligible; max also enables TF32 matmul and fused QKV. Regional compile is gated off for the GGUF transformer (dequantises per-op) and for families flagged not compile-friendly (a new supports_torch_compile flag, False for Z-Image), so it activates automatically only once a non-GGUF bf16 transformer is loaded. Speed optims run before placement/offload, per the diffusers composition order. status now reports speed_mode + the optims actually engaged. Verified on Z-Image (B200): default -> ['channels_last'], max -> ['channels_last', 'tf32'], compile correctly skipped for GGUF; generation works in every mode. 121 CPU tests pass. * Studio diffusion (Phase 2B): opt-in fp8 text-encoder layerwise casting Add a text_encoder_fp8 knob that casts the companion text encoder(s) to fp8 (e4m3) storage via diffusers apply_layerwise_casting, upcasting per layer to the bf16 compute dtype while normalisations and embeddings stay full precision. Applied before placement, gated to CUDA + bf16, best-effort (a failure leaves the encoder dense). status reports which encoders were cast. Verified on Z-Image (B200, balanced/group mode where the encoder stays resident): generation peak VRAM dropped 37% (10840 -> 6791 MB, below the lowest-VRAM offload) at near-resident speed. It is a memory-vs-quality tradeoff, not free -- ~20 dB PSNR vs the bf16 encoder, a larger shift than one transformer quant step -- so it is off by default and documented as such, with the Phase 5 harness to size the cost. 127 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2C): NVFP4 text-encoder quant (+ generalise fp8 knob) Generalise the text-encoder precision knob from a fp8 bool to text_encoder_quant (fp8 | nvfp4). nvfp4 quantises the companion text encoder to 4-bit via torchao NVFP4 weight-only (two-level microscaling) on Blackwell's FP4 tensor cores; fp8 stays the broader-hardware path (cc>=8.9). Both are gated, best-effort, and run before placement; status reports the mode actually engaged. This is the lean realisation of GGUF-native text-encoder quant: 4-bit on the encoder without the 3045-line port. Verified on Z-Image (B200, balanced/group where the encoder stays resident), vs the bf16 encoder: nvfp4 cut generation peak VRAM 48% (10840 -> 5593 MB, the lowest TE option, below whole-model offload) at near-fp8 quality (16.4 vs 17.1 dB PSNR), and both quants ran faster than bf16. A memory-vs-quality tradeoff (off by default); size it per model with the Phase 5 quality harness. diffusion_bench gains --text-encoder-quant. 129 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): native stable-diffusion.cpp engine for CPU/Mac Adds the CPU / Apple-Silicon tier of the two-engine strategy, mirroring the chat backend's llama.cpp shell-out. Diffusers stays the default on CUDA / ROCm / XPU; this covers the hardware diffusers serves poorly, consuming the same split GGUF assets Studio already curates. - sd_cpp_args.py: pure sd-cli command builder. Maps the family to its text-encoder flag (Z-Image Qwen3 to --llm, Qwen-Image to --qwen2vl, FLUX.1 CLIP-L + T5), and the diffusers memory policy (none/group/model/sequential) to sd.cpp's offload flags (--offload-to-cpu / --clip-on-cpu / --vae-on-cpu / --vae-tiling / --diffusion-fa), so one user knob drives both engines. - sd_cpp_engine.py: SdCppEngine over a located sd-cli. find_sd_cpp_binary() with the same precedence as the llama finder (env override, then the Studio install root, then in-tree, then PATH), an is_available/version probe, and a one-shot subprocess generate that streams progress and returns the PNG. runtime_env() prepends the binary's directory to the platform library path so a prebuilt's bundled libstable-diffusion.so resolves. select_diffusion_engine() is the pure routing decision (GPU backends to diffusers, CPU/MPS to native when present). - install_sd_cpp_prebuilt.py: resolve + download the per-host prebuilt (macOS-arm64/Metal, Linux x86_64 CPU, Vulkan/ROCm/Windows variants) into the Studio install root. resolve_release_asset() is a pure, unit-tested host-to-asset matrix. - scripts/sd_cpp_smoke.py: end-to-end native generation harness. Tests (CPU-only, subprocess/filesystem stubbed): 49 new across args, engine, routing, runtime env, and the installer resolver. Full diffusion suite 166 passing. Verified on a B200 box: built sd-cli (CUDA) and the prebuilt (CPU) both generate Z-Image-Turbo Q4_K end to end through SdCppEngine: balanced (group offload, 5.0s gen), low_vram (full CPU offload + VAE tiling, 13.4s), and the dynamically-linked CPU prebuilt (50.4s on CPU), all producing coherent images. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 6): img2img / inpaint / edit / LoRA / upscale on the native engine Builds on Phase 4's native stable-diffusion.cpp engine, extending it from text-to-image to the wider feature surface, since sd.cpp supports all of these through the binary already. Pure command-builder additions plus one engine method, so the txt2img path is unchanged. - sd_cpp_args.py: SdCppGenParams gains image-conditioning fields. init_img + strength make a run img2img, adding mask makes it inpaint, ref_images drives FLUX-Kontext / Qwen-Image-Edit style editing (repeated --ref-image), and lora_dir + the <lora:name:weight> prompt syntax select LoRAs. New SdCppUpscaleParams + build_sd_cpp_upscale_command for the ESRGAN upscale run mode (input image + esrgan model, no prompt / text encoders). - sd_cpp_engine.py: the subprocess runner is factored into a shared _run() so generate() (now carrying the conditioning flags) and a new upscale() reuse the same streaming / error / output-check path. - scripts/sd_cpp_smoke.py: --task {txt2img,img2img,upscale} with --init-img / --strength / --upscale-model / --upscale-repeats. Tests: 10 new across the img2img / inpaint / edit / LoRA flag construction, the upscale builder and its validation, and the engine's img2img + upscale paths. Full diffusion suite 176 passing. Verified on a B200 box through SdCppEngine: img2img (Z-Image-Turbo Q4_K, the init image conditioned at strength 0.6, 4.8s) and ESRGAN upscale (512x512 -> 2048x2048 via RealESRGAN_x4plus_anime_6B, 2.7s), both producing coherent images. Video and the diffusers-path feature wiring are deferred. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): accuracy-preserving speed pass Re-review of the diffusion stack (#6675/#6679/#6680) surfaced one real accuracy bug and a dead-on-arrival speed path; this fixes both and adds the lossless / near-lossless wins, all measured on a B200. Correctness: - TF32 global-state leak (fix). speed_mode=max flipped torch.backends.*.allow_tf32 process-wide and never restored them, so a later `off` load silently inherited TF32 and was no longer bit-identical. Added snapshot_backend_flags / restore_backend_flags (TF32 + cudnn.benchmark), captured before the speed layer runs and restored on unload. Verified: load max -> unload -> load off is now byte-identical (PSNR inf) to a fresh off. - sd-cli timeout could hang forever. _run() blocked in `for line in stdout` and only checked the timeout after EOF, so a child stuck in model load / GPU init with no output ignored the timeout. Drained stdout on a reader thread with a wall-clock deadline. Added a silent-hang regression test. Speed (diffusers path), near-lossless, opt-in tiers: - Regional torch.compile now runs on the GGUF transformer. The is_gguf gate (and Z-Image's supports_torch_compile=False) were stale: compile_repeated_blocks compiles and runs ~2.2x faster on the GGUF Z-Image transformer on torch 2.9.1 / diffusers 0.38 (the per-op dequant stays eager, the rest of the block compiles). Measured: off 1.80s -> default 0.82s/gen (+54.7%), PSNR 37.7 dB vs eager -- far above the Q4 quant noise floor (~21 dB), so it does not move output quality. Gate relaxed; default tier delivers it. - cudnn.benchmark added to the default tier (autotunes the fixed-shape VAE convs). - torch.inference_mode() around the pipeline call (lossless, strictly faster than the no_grad diffusers uses internally). Memory path: - VAE tiling (not bit-identical >1MP) restricted to the model/sequential/CPU tiers; the balanced (group) tier keeps exact slicing only, so it is now bit-identical to the resident image (verified PSNR inf) and slightly faster. - Group offload adds non_blocking + record_stream on the CUDA stream path to overlap each block's H2D copy with compute (lossless; gated on the installed diffusers signature so older versions still work). Native (sd.cpp) path: - native_speed_flags: a first-class speed knob (default -> --diffusion-fa, a near-lossless CUDA win that was previously only added on offload tiers; max also -> --diffusion-conv-direct). conv-direct stays opt-in: measured +45% on CUDA, so it is never auto-on. Engine generate() merges it, de-duped against offload flags. Default profile: a GGUF model with no explicit speed_mode now resolves to the `default` profile (resolve_speed_mode), since compile's perturbation sits below the quantisation noise floor and so does not reduce quality versus the dense reference; out of the box a GGUF Z-Image generation drops from 1.80s to 0.81s. Dense models stay `off` / bit-identical, and an explicit speed_mode -- including "off" -- is always honored, so the byte-identical path remains one flag away and is the regression reference. Tooling: scripts/compile_probe.py (eager vs compiled GGUF probe), scripts/ perf_verify.py (the B200 verification above), and diffusion_bench.py gains --speed-mode so the speed tiers are benchmarkable. Tests: 183 passing (was 166); new coverage for the backend-flag snapshot/restore, GGUF compile eligibility, the balanced tiling/slicing split, native_speed_flags + the engine de-dup, and the sd-cli silent-hang timeout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): max tier uses max-autotune-no-cudagraphs + engine/lever benchmarks The opt-in `max` speed tier now compiles the repeated block with mode=max-autotune-no-cudagraphs (dynamic=False) instead of the default mode: Triton autotuning for GEMM/conv-heavier models, gated to the tier where a longer cold compile is acceptable. CUDA-graph modes (reduce-overhead / max-autotune) are deliberately avoided -- both crash on the regionally-compiled block (its static output buffer is overwritten across denoise steps), measured. Adds two reproducible benchmarks used to validate the optimization research: - scripts/compare_engines.py: PyTorch (diffusers GGUF) vs native sd.cpp head-to-head. - scripts/leverage_probe.py: coordinate_descent_tuning + FirstBlockCache probes. Measured on B200 (Z-Image Q4_K_M, 1024px, 8 steps): default compile 0.80s/gen; coordinate_descent_tuning 0.79s (within noise, already covered by max-autotune); FirstBlockCache does not run on Z-Image (diffusers 0.38 block-detection / Dynamo). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source) Add an opt-in transformer_quant mode that loads the dense bf16 transformer and torchao-quantises it onto the low-precision tensor cores, instead of the GGUF transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's 0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor. GGUF+compile stays the low-memory default and the fallback. The mode is gated on CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB); any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe (Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips the tiny projections that crash int8's torch._int_mm. New module mirrors diffusion_precision.py; quant runs before compile before placement. 184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py is the standalone torchao lever probe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): consumer-GPU tuning - lock fp8 fast accumulate, prefer fp8 over mxfp8, reject 2:4 sparsity Consumer Blackwell halves tensor-core throughput on FP32 accumulate (fp8 419 vs 838 TFLOPS with FP16 accumulate; bf16 209), so: - fp8 config locks use_fast_accum=True (Float8MMConfig). torchao already defaults it on; pinning it guards consumer cards against a default change. On B200 it is identical speed and slightly better quality (LPIPS 0.050 vs 0.091). - the Blackwell auto ladder prefers fp8 over mxfp8 (measured faster + more accurate). 2:4 semi-structured sparsity evaluated and rejected (scripts/sparse_accum_probe.py): 2:4 magnitude-prune + fp8 gives LPIPS 0.858 (broken image) with no fine-tune, the cuSPARSELt kernel errors on torch 2.9, and it does not compose with torch.compile (our main ~2x). Documented as a dead end, not shipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add fp8 fast-accum overflow verification probe scripts/fp8_overflow_check.py hooks every quantised linear during a real Z-Image generation and reports max-abs + non-finite counts for use_fast_accum True vs False. Confirms fast accumulation is an accumulation-precision knob, not an overflow one: across 276 linears, including Z-Image's ~1.0e6 activation peaks (which overflow FP16), 0 non-finite elements and identical max-abs for both modes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40; GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it. Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s), no overflow, quality below the quant noise floor. So the default leans to accuracy on data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force) lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off). 187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override threading). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add NVFP4 probe documenting it is not yet a win on torch 2.9 scripts/nvfp4_probe.py measures NVFP4 via torchao on the real Z-Image transformer. Finding (B200, 1024px/8 steps): NVFP4 is a torchao feature and DOES run with use_triton_kernel=False (the default triton path needs the missing MSLK library), but only at bf16-compile rate (0.667s vs fp8 0.592s) -- it dequantises FP4->bf16 rather than using the FP4 tensor cores. The real FP4 speedup needs MSLK or torch>=2.11 + torchao's CUTLASS FP4 GEMM. The smoke probe (default triton=True) already keeps NVFP4 out of auto on this env, so auto correctly stays on fp8; NVFP4 activates automatically once fast. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): prefer fp8 over nvfp4 in Blackwell auto ladder Validated NVFP4 on torch 2.11 + torchao CUTLASS FP4 in an isolated env. The FP4 tensor-core GEMM is genuinely active there (a 16384^3 GEMM hits ~3826 TFLOPS, 2.52x bf16 and 1.37x fp8), but it only beats fp8 on very large GEMMs. At the diffusion transformer's shapes (hidden ~3072, MLP ~12288, M~4096) NVFP4 is both slower (0.81x fp8 end to end on Z-Image 1024px) and less accurate (LPIPS 0.166 vs fp8's 0.044). Reorder the Blackwell auto ladder to fp8 before nvfp4 so auto is correct even on a future MSLK-equipped box; nvfp4 stays an explicit opt-in. Add scripts/nvfp4_t211_probe.py (extension diagnostics + GEMM micro + end-to-end). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): pre-quantized transformer loading The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs 13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline (scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True) the quantized weights, so the dense bf16 never touches the GPU. Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same torchao config + min_features filter the runtime path uses, applied ahead of time. New core/inference/diffusion_prequant.py (resolve_prequant_source + load_prequantized_transformer, best-effort, lazy imports). diffusion.py _load_dense_quant_pipeline tries the pre-quant source first and falls back to the dense materialise+quantise path, then to GGUF, so the default is unchanged. DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for the resolver, the meta-init+assign loader, and the backend branch selection + fallbacks; GPU verification via scripts/verify_prequant_backend.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): attention-backend selection Add a selectable attention kernel via the diffusers set_attention_backend dispatcher. Attention is memory-bandwidth bound, so a better kernel is an end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV matmuls torchao never touches) and composes with torch.compile. auto picks the best exact backend for the device: cuDNN fused attention (_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default (below the compile/quant noise floor); native SDPA elsewhere and when speed=off (so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/ xformers/aiter are honored, and an unavailable kernel falls back to the default rather than failing the load. New core/inference/diffusion_attention.py (normalize + per-device select + apply, best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline; attention_backend threads through begin_load / load_pipeline / status like the other load knobs. New request field attention_backend + status field. Hermetic CPU tests for normalize / select policy / apply fallback, plus route threading + 422. Measured via scripts/perf_levers_probe.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the arch tier to put int8 first on a consumer/workstation GPU (detected by the existing _is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first. Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation (-> int8) and data-center Ada/Hopper/Blackwell (-> fp8). * Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across denoise steps a DiT's output settles, so once the first block's residual barely changes the remaining blocks are skipped and their cached output reused. diffusers ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the standalone apply_first_block_cache hook as a fallback). Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 -> 2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar. OFF by default and a per-load opt-in: the win scales with step count, so it is for many-step models (Flux / Qwen-Image) and pointless for few-step distilled models (e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction of the trajectory. It composes with regional compile only with fullgraph=False (the cache's per-step decision is a torch.compiler.disable graph break), which the speed layer now switches to automatically when a cache is engaged. Best-effort: a model whose block signature the hook does not recognise is caught and the load proceeds uncached. - new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache (enable_cache / apply_first_block_cache fallback; threshold auto-raised for a quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import). - diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph when a cache is engaged. - diffusion.py: apply_step_cache before compile; thread transformer_cache / transformer_cache_threshold through begin_load -> load_pipeline and report the engaged mode in status(). - models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and transformer_cache_threshold request fields, engaged mode in the status response. - hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold selection, and best-effort failure handling, plus route threading + validation. - scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM / LPIPS vs the compiled no-cache baseline). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears) The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with 'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2 *_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing min_features filter did not exclude them. Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed, guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers (M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and quantises these layers fine, so the exclusion is int8-only. Sequence embedders (context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note 'context_embedder' contains the substring 'text_embed', which is why the token is the specific 'time_text_embed', not 'text_embed'. Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s (5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already working) are unchanged. - diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes exclude_name_tokens; quantize_transformer passes it for int8 only. - hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps attention / FFN / sequence-embedder linears), while fp8 keeps them. - scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's Linear layers and derive the exclusion list. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears) The prequant-checkpoint builder applied the dense quant filter without the int8-only M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8 checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now produces a working checkpoint on every supported model, giving int8 (the consumer-preferred scheme) the same ~2x load-VRAM and download reduction fp8 already had. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the native stable-diffusion.cpp engine instead of diffusers, with diffusers as the guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM. - diffusion_engine_router: centralised engine selection (built on the existing select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason. - sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface backed by sd-cli, with lazy binary install, registry-driven asset fetch, step-progress parsing, and cancellation. - diffusion_families: per-family single-file VAE + text-encoder asset mapping. - sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled). - routes/inference + gpu_arbiter: drive the active engine via the router; the API now reports the active engine and any fallback reason. - tests for the backend, router, route selection, and cancellation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler Address review feedback on #6724: - engine router: unload the engine being deactivated on a switch, so the old model is not left resident-but-unreachable (the evictor only targets the active engine). - generate route: sd.cpp execution errors (nonzero exit / timeout / missing output) now map to 500, not 409 (which only means not-loaded / cancelled). - native batch: return per-image seeds and persist the actual seed for each image so every batch image is reproducible. - Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the stable-diffusion.cpp docs; other families keep sd-cli defaults. - honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload offload flags on the native load instead of hardcoding them off. - fail the load when the sd-cli binary is present but not runnable (version() now returns None on exec error / nonzero exit). - size estimate: only treat the transformer asset as a possible local path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in load_prequantized_transformer ends in torch.load(weights_only=False), which executes arbitrary code from the pickle. The transformer_prequant_path load-request field reached that unpickle for any local file an authenticated caller named, so a request could trigger remote code execution. Refuse the source.kind=='path' branch unless the operator sets UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted and unaffected. Document the requirement on the API field and add gate tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa - apply_attention_backend now restores the native default when no backend is requested or a kernel fails. diffusers keeps a process-wide active attention backend that set_attention_backend updates, and a fresh transformer's processors follow it, so a load that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile load pinned, breaking the bit-identical/off guarantee. - select_attention_backend drops flash3/flash4 up front when the CUDA capability is below Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit request on the wrong card set fine then crashed mid-generation; it now falls back to native. - Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a valid alias of native) is accepted instead of 422-rejected by Pydantic. - Drop the dead replace('-','_') normalization (no alias uses dashes/underscores). - perf_levers_probe.py output dir is now relative to the script, not a hardcoded path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF - apply_step_cache now engages only via the transformer's native enable_cache (the diffusers CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model now runs uncached per the best-effort contract. - GGUF transformers are quantized (the default Studio load path), so they now use the higher quantized FBCache threshold when the caller leaves it unset, instead of the dense default that could keep the cache from triggering. - fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so fullgraph=True failed warmup and silently measured an eager cached run); output dir is now relative to the script, not a hardcoded path. * Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder _is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate. Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0). * Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just TypeError on older torchao), which would break the mxfp8 config helper instead of falling back to the default. Catch both so the fallback is robust. quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds CUDA memory during the per-row VRAM probe; output dir relative to the script. * Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load - snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the whole snapshot. restore_backend_flags restores each flag independently so one failure can't leave the others leaked process-wide. - load_pipeline restores the flags (and clears the GPU cache) when the build fails after apply_speed_optims mutated the process-wide flags but before _state captured them for unload to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and contaminated later off generations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs without producing output (or without closing stdout) would never reach proc.wait and the wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the PROCESS, so the main thread always enforces the timeout and kills a hung process (which closes the pipe and ends the reader). Add a test that times out even when stdout blocks, and make the no-binary test hermetic so a host-installed sd-cli can't leak in. * Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn The filter callback can be invoked without a module name, so fqn.lower() would raise AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens, so the linear is kept) instead of crashing the quantise pass. * Studio diffusion (Phase 16) review fixes: native engine robustness - sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64); a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds aliased to the same image. Random seeds stay 53-bit (JS-safe). - sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub fall back to anonymous instead of failing auth on a blank token. - sd_cpp_backend: a superseding load now cancels the in-flight generation, so the old sd-cli can no longer return/persist an image from the previous model. - diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a slow 10+ GB free / CUDA sync does not block engine selection. - diffusion_engine_router: probe sd-cli runnability (version()) before committing to native, so a present-but-unrunnable binary falls back to diffusers at selection. - diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a CPU-only install can still reach the native sd.cpp engine instead of failing load. - tests updated for the runnability probe + a not-runnable fallback case. * Studio diffusion (Phase 9) review fixes: prequant safety + validation - SECURITY: a request-supplied local pre-quant path is now unpickled only when it resolves inside an operator-configured ALLOWLIST of directories (UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in, once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on any path a load request named (arbitrary code execution). realpath() blocks symlink escapes; a bare on/off toggle is no longer a wildcard. - Validate the checkpoint's min_features against the runtime Linear filter, so a checkpoint that quantised a different layer set is rejected instead of silently loading a model that mismatches the dense path while reporting the same scheme. - Tolerant base_model_id compare (exact or same final path/repo segment), so a local path or fork of the canonical base is accepted instead of falling back to dense. - _has_meta_tensors uses any(chain(...)) (no intermediate lists). - prequant verify/probe scripts use repo-relative paths (+ env overrides), not the author's absolute /mnt paths. - tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts - diffusion_memory: when group offload is unavailable and the plan falls back to whole-module offload, enable VAE tiling (the group plan left it off, but the fallback is the low-VRAM path where the decode spike can OOM). Covers both the group and sequential fallback branches. - perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a balanced bit-identity regression actually fails the check instead of exiting 0. - compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and the load-progress poll has a 30 min deadline instead of looping forever on a hang. - test for the group->model fallback enabling VAE tiling. * Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path - diffusion: a torchao-quantized transformer is committed only compiled. A dense model resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF it replaced), so when transformer_quant engaged and speed resolved to off, promote to default (regional compile); warn loudly if compile still does not engage. - diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK); otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF. - nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path). - test asserts the eager-quant -> default-compile promotion. * Studio diffusion (Phase 10) review fixes: attention gating + probe isolation - diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation, so auto now stays on native SDPA there. - diffusion_attention: _active_attention_backend handles get_active_backend() returning an enum/None (not a tuple); the old unpack always raised and was swallowed, so the native-restore short-circuit never fired. - perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul so the inductor_flags variant doesn't leak into later compiled rows. - tests for the SM80 cuDNN gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening - install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket); extract through a per-member containment check (Zip-Slip guard); expanduser the --install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch the separately-published cudart runtime DLL archive so sd-cli.exe can start. - sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie. - tests: Zip-Slip rejection, normal extraction, studio-home discovery. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of the literal --output path. SdCppEngine.generate checked only the literal path, so a batch generation would exit 0 and then raise 'no image' (or return a stale file). generate now returns the literal path when present and otherwise falls back to the numbered siblings; single-image behavior is unchanged. Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected without error. * Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats Codex review on the native engine arg builder: - build_sd_cpp_command emitted --width/--height unconditionally, so an img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of the input. width/height are now Optional (None = unset): an image-conditioned run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives the size from the input image (set_width_and_height_if_unset); a plain txt2img run with unset dims keeps the prior 1024x1024 default; explicit dims are always honored. width/height are read only by the builder, so the type change is local. - build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...) that silently swallowed repeats=0 into sd-cli's default of one pass, turning an explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError and emits the flag for any explicit value != 1. Tests: img2img unset dims omit width/height (init_img and ref_images), explicit dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at the default. (Two pre-existing binary-discovery tests fail only because a real sd-cli is installed in this dev environment; unrelated to this change.) * Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc Codex review: the transformer_prequant_path field description still told operators to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the prior security fix made that variable a directory allowlist -- _allowed_prequant_roots deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator following the documented =1 would have every transformer_prequant_path request silently refused. The description now states it must name one or more allowlisted directories and that a bare on/off value is not accepted. Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does not say =1, and describes an allowlist/directory (guards against doc drift). * Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset Codex review on attention-backend selection: - Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first generation with no fallback. select_attention_backend now applies _cudnn_attention_supported() to an explicit cuDNN request too. - flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3 is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a (min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+. - apply_attention_backend's success path left diffusers' process-wide active backend pinned to the kernel it set; a later component whose processors are unconfigured (backend None) would inherit it. It now resets the global registry to native after a successful per-transformer set (the transformer keeps its own backend), best-effort. Also fixed _active_attention_backend: get_active_backend() returns a (name, fn) tuple, so the prior code stringified the tuple and never matched a name, defeating the native-restore short-circuit. Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on SM90; global registry reset after a successful set; _active_attention_backend reads the tuple return. * Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300 (Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So _is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they are treated as data-center class and keep the intended fp8-first behavior. Test: extends the datacenter parametrize with 'NVIDIA B300' and 'NVIDIA GH200 480GB' (now _is_consumer_gpu False). * Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder Codex review: the M=1 modulation/embedder exclusion was wired only into the dense runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint quantised the AdaLN modulation and conditioning-embedder linears, and loading it via transformer_prequant_path (the load path only loads already-quantised tensors, it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes for the runtime path. Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 -> the M=1 exclusion, every other scheme -> none) and use it in both the runtime quantiser and the builder, so a prequant artifact's quantised-layer set always matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion). Test: int8_exclude_name_tokens returns the exclusion for int8 and () for fp8/nvfp4/mxfp8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race Codex review on the native-engine routing: - The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat) unconditionally after engine selection. A native sd.cpp load on a pure-CPU host never touches the GPU, so that needlessly tore down the resident chat model. The handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it. - sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when _run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so the setting was unverifiable. status now derives them from state.offload_flags (still 'none' on CPU, where the flags are empty). - _run_load committed the new state without cancelling/waiting on a generation that started during the (slow) asset download, so a stale sd-cli run against the OLD model could finish afterward and persist an image from the previous model once the new load reported ready. The commit now signals the in-flight cancel and waits on _generate_lock before swapping _state (taken only at commit, so the download never serialises against generation), mirroring the diffusers load path. Tests: CPU native load skips the arbiter while a GPU native load takes it; status reports offload active when flags are set; _run_load cancels and waits for an in-flight generation before committing. * Studio diffusion (Phase 14) review round 2: align helper name with the stack Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the identical helper already present higher in the diffusion stack (Phase 16). The helper definition, the runtime quantiser call, and the offline builder are now byte-identical to that version, so the two branches no longer introduce a divergent name for the same single-source-of-truth and the stack merges without a conflict on this fix. No behavior change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: eager patches + torch.compile cache speed phase Adds the opt-in speed path for the GGUF diffusion transformer behind a selectable speed mode (default off, so output is unchanged until a profile is chosen): - diffusion_eager_patches.py: shared eager fast-paths (channels_last, attention/backend selection, fused norms and QKV) installed at load and rolled back on unload or failed load. - diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent torch.compile cache and the GGUF-transformer compile wiring. - diffusion_arch_patches.py: architecture-specific patches. - diffusion_patch_backend.py: shared install/restore plumbing. - diffusion_speed.py: speed-profile planning. Tests for each module plus the benchmarking and probe scripts used to measure speed, memory, and accuracy of the path. * Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI Backend: - Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file fp8 transformers, gated to the unsloth org plus a curated allowlist. - Image-conditioned workflows built with Pipeline.from_pipe so they reuse the loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint, outpaint, and a hires-fix upscale pass. - Instruction editing as its own family kind (Qwen-Image-Edit-2511, FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and multi-reference) plus klein inpaint. - Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the decoded image size and cap upscale output to avoid OOM on large inputs. - Fixes: from_pipe defaulting to a float32 recast that crashed torchao quantized transformers; image-conditioned calls forcing the slider size onto the input image. Native sd.cpp engine rejects image-conditioned and reference requests it cannot serve. Frontend: - Redesigned Images page with capability-gated workflow tabs (Create, Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor, client-side outpaint, and a multi-reference picker. - Advanced options moved to a right-docked panel mirroring Chat: closed by default, toggled by a single fixed top-bar button that stays in place. sd.cpp installer: pin the release, verify each download's sha256, add a download timeout, and make the source repo configurable for a future mirror. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: correct the Advanced panel comment (closed by default, fixed toggle) * Studio diffusion: LoRA adapters for the Images workflow Add community LoRA support across both diffusion backends, the single biggest step toward broad image-workflow coverage. Backend - New shared module core/inference/diffusion_lora.py: adapter discovery (local scan + curated catalog + owner/name[:file] Hub refs), download via hf_hub_download_with_xet_fallback, alias sanitization, native managed-dir materialization with collision-broken aliases, prompt-tag injection (deduped against user-typed tags), and a supports_lora gate. - Native sd-cli: resolve + materialize selected LoRAs into a per-run managed dir, inject <lora:ALIAS:w> tags, pass --lora-model-dir with --lora-apply-mode auto. The arg builder already emitted these flags. - Diffusers: non-fused load_lora_weights + set_adapters manager, tracked on the pipe so an unchanged selection is a no-op and a model swap resets; cleared on unload. Never fuses (breaks quantized transformers and blocks live weight tweaks). - Gated off where unsupported: torchao fp8/int8 dense, GGUF-via-diffusers, and native Qwen-Image (no LoRA name-conversion branch upstream). - Request contract: optional loras on DiffusionGenerateRequest; empty or omitted is identical to today. supports_lora surfaced in status; chosen LoRAs persisted in gallery recipe metadata. - New GET /api/models/diffusion-loras for the picker (family-filtered). Frontend - Repeatable multi-LoRA picker (adapter select + weight slider 0..2 + remove), gated by the loaded model's supports_lora and family, max 8. Tests - New test_diffusion_lora.py (14): helpers, request validation, native tag/dir wiring, diffusers set_adapters manager, supports_lora matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: ControlNet for the Images workflow (diffusers) Add ControlNet conditioning, the #2 most-used diffusion workflow after LoRA, on the diffusers backend for the families with ControlNet pipelines (FLUX.1 and Qwen-Image), with Union models as the default picks. Backend - New core/inference/diffusion_controlnet.py: family-gated discovery (curated Union models + local dirs + bare owner/name repos), resolution to a loadable repo/dir, control-image preprocessing (passthrough + a dependency-free canny edge map), and a supports_controlnet gate. - diffusion.py: a ControlNet manager parallel to the LoRA one. Loads the (small) ControlNet model once via from_pretrained (cached by id) and builds the family's ControlNet pipeline via Pipeline.from_pipe(base, controlnet=model), reusing the resident base modules at their loaded dtype (no reload, no recast). Passes the control image + conditioning scale + guidance start/end at generate time; cleared on unload. - Families: FLUX.1 -> FluxControlNetPipeline/Model, Qwen-Image -> QwenImageControlNetPipeline/Model. Others declare none (gated off). - Gated off for the native engine, GGUF-via-diffusers, and torchao fp8/int8 dense (same rule as LoRA). v1 conditions txt2img only. - Request contract: optional controlnet on DiffusionGenerateRequest; supports_controlnet in status; the choice persisted in gallery meta. - New GET /api/models/diffusion-controlnets for the picker. Frontend - A ControlNet control in the Images rail (model select + control-image upload + control-type select + strength slider), gated by the loaded model's supports_controlnet + family, shown for text-to-image. Tests - New test_diffusion_controlnet.py (10): discovery/resolve/preprocess/gate helpers, request validation, family wiring, and the diffusers pipe manager (loads once, caches, from_pipe with controlnet, rejects unsupported families). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio ControlNet: show the picker on the Create tab (workflow id is 'create', not 'txt2img') The ControlNet control gated on workflow === 'txt2img', but the Images workflow tab ids are create/transform/inpaint/extend/upscale/reference/edit -- there is no 'txt2img'. So the picker never rendered even with a ControlNet-capable model loaded. Gate on 'create' (the text-to-image tab) for both the picker and the request wiring. Found via a live Playwright capture of the running Studio. * Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a stray "gguf" tag on the Hub but ship no .gguf files. The model search classified them as GGUF from the bare tag, so the picker rendered the GGUF variant expander, which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata (populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected. * Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline load when the id is unsloth-hosted or on-device (the backend infers the family + base repo and gates loads to unsloth/* or local paths), and show a clear message otherwise instead of silently ignoring the click. Curated and GGUF paths are unchanged. * Studio Images: keep curated safetensors models in Recommended after download The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet), so they vanished from the picker after the first load and could only be found by typing an exact search. The row already renders a downloaded badge, matching how GGUF Recommended rows stay visible when cached. Drop the exclusion so the curated safetensors always list. * Studio diffusion LoRA: sanitize dots out of adapter aliases The LoRA alias is used as the diffusers PEFT adapter name, and PEFT rejects names containing "." (module name can't contain "."). sanitize_alias kept dots, so a LoRA whose filename carries a version tag (e.g. Qwen-Image-2512-Lightning-8steps-V1.0-bf16) failed to apply with a 400. Replace dots too; the alias stays a valid native <lora:NAME:w> filename stem. Adds regression coverage for internal dots. * Studio Images: clarify the GGUF transformer-quant Advanced control Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode" with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/ FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF, falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only. * Studio Images: list on-device unsloth diffusion models in the picker The Images picker's On Device tab hid every non-GGUF cached repo whenever a task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit and FP8 safetensors) never showed up there. List cached repos that pass the task gate, limited under a filter to unsloth-hosted ones so base repos (which fail the diffusion load trust gate) don't appear only to dead-end on click. Chat behavior is unchanged: the task gate still drops image repos there. * Studio: hide single-file image checkpoints from the chat model picker The chat picker treats a cached repo as an image model, and hides it, only when it ships a diffusers model_index.json. Single-file, ComfyUI, and ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models. Fall back to resolving the repo id against the known diffusion families, the same resolver the Images backend loads from, so these checkpoints are tagged text-to-image and stay in the Images picker only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: add the FLUX.2-dev model family Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline) resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline + Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated, reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror. text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from klein's turbo defaults. Verified live: GGUF load resolves the family + gated base repo and generates a real 1024x1024 image on GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: clearer error for an unsupported diffusion model When a repo id resolves to no diffusion family the load raised 'Could not infer a diffusion family... Pass family_override (z-image)', which points at an unrelated family and doesn't say what is supported. Replace it with a message that lists the supported families (from a new supported_family_names helper) and notes that video models and image models whose diffusers transformer has no single-file loader are not supported. Applies to both the diffusers and native sd.cpp load paths. Also refreshes two stale family-registry comments that still called FLUX.2-dev omitted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add SDXL diffusion family (U-Net pipeline support) SDXL is the first U-Net family in the diffusion backend: its denoiser is pipe.unet (UNet2DConditionModel), not a DiT pipe.transformer, and a single-file .safetensors is the whole pipeline rather than a transformer-only file. The backend previously assumed a DiT transformer everywhere, so add the two hooks a U-Net family needs and register SDXL. DiffusionFamily gains denoiser_attr ("transformer" for DiT, "unet" for SDXL) and single_file_is_pipeline (SDXL loads a single file via pipeline_class.from_single_file with the base repo as config, instead of transformer_class.from_single_file plus a companion assembly). _align_vae_dtype now reads the denoiser generically so img2img and inpaint keep the VAE and U-Net dtypes aligned. The non-GGUF trust gate is extended with a short, exact-match, safetensors-only allowlist of official base repos (the SDXL base/refiner and sdxl-turbo), because SDXL ships only as a full pipeline and has no unsloth-hosted GGUF. Local paths stay trusted as before; a random repo, even one that detects as SDXL, is still rejected. The image-conditioned and ControlNet workflows are the standard SDXL pipelines, built around the resident modules via from_pipe like every other family, so SDXL gets txt2img, img2img, inpaint, outpaint, upscale, LoRA and ControlNet. There is no native sd.cpp mapping yet, so the no-GPU route falls back to diffusers. Frontend catalog gains SDXL Base 1.0 and SDXL Turbo entries with SDXL step/guidance defaults (Turbo: few steps, no CFG; base: ~30 steps, real CFG). Tests: new test_diffusion_sdxl.py (family shape, detection, trust allowlist, model kind, U-Net VAE-dtype alignment, LoRA gate) plus loader-branch tests in test_diffusion_backend.py (pipeline-kind from_pretrained, single-file whole-pipeline from_single_file, allowlist accept/reject). Verified live on GPU: sdxl-turbo loads both as a pipeline and as a single file and generates coherent txt2img + img2img. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images: LoRA free-text Hugging Face entry + recipe round-trip The backend has always accepted a bare Hugging Face repo id (owner/name, or owner/name:weight-file.safetensors) as a LoRA, downloading and applying it. But the picker only rendered when the curated catalog had entries, and the catalog is empty, so there was no UI path to apply any LoRA. Show the LoRA section whenever the loaded model supports LoRA, and replace the curated-only dropdown with a text input: type a Hub repo id, or pick a discovered adapter from a datalist of suggestions when the catalog is populated. Also restore LoRAs when loading a recipe. restoreSettings now parses the recipe's "id:weight" strings (splitting on the last colon, since the id itself may contain one for a specific weight file) back into the selection, so replaying a saved image reproduces its adapters. The generate payload trims hand-typed ids and drops empty / zero-weight rows, and a model swap clears the selection (a LoRA is family-specific) without discarding a free-text pick that is not in the curated list. * Add diffusion LoRA training (SDXL text-to-image) First diffusion training path in Studio: train a LoRA on the SDXL U-Net from an image + caption dataset and export it as a diffusers .safetensors that the existing diffusion LoRA loader (and any diffusers pipeline) can load. core/training/diffusion_lora_trainer.py: - DiffusionLoraConfig with validation/defaults (rank, alpha, targets, lr, steps, grad accumulation, resolution, min-SNR gamma, gradient checkpointing, lr scheduler, seed, mixed precision). - discover_image_caption_pairs: captions from metadata.jsonl / captions.jsonl, per-image .txt/.caption sidecars, or a dreambooth instance_prompt fallback (pure, unit-tested). - run_diffusion_lora_training: the loop -- freeze base, PEFT-wrap the U-Net attention projections, VAE-encode (fp32 VAE to avoid the SDXL fp16 overflow), sample noise + timesteps, predict, MSE loss with optional min-SNR weighting (epsilon / v-prediction), AdamW + get_scheduler + grad accumulation + grad clipping, then export via save_lora_weights. Emits worker-protocol events (model_load_*, progress, complete) and polls should_stop for a clean stop with a partial save. - run_diffusion_training_process: mp.Queue subprocess adapter (event_queue / stop_queue), so the training worker can spawn it; plus a CLI entry point. Only SDXL (U-Net) is trained here; DiT families and the Studio UI form + route wiring are follow-ups. The trainer is decoupled and worker-ready. Tests: test_diffusion_lora_trainer.py covers caption discovery (metadata / sidecar / instance prompt / skip-uncaptioned / errors), config normalisation + validation, the SDXL add-time-ids, and the dict->config adapter. Verified live on GPU: a 60-step SDXL LoRA run lowers the loss, exports a ~45 MB adapter, and loading it back shifts generation from baseline (mean abs pixel diff ~55/255). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * diffusion trainer: emit learning_rate in progress events (Studio pump compatibility) The Studio training pump reads 'learning_rate' from progress events; the diffusion trainer emitted 'lr'. Rename the field (and the CLI reader) so the trainer's events are directly consumable by the existing training status/SSE machinery when it is wired into the worker, without a translation shim. * Wire diffusion LoRA training into the Studio API Make the SDXL LoRA trainer reachable from the app with a small, self-contained job service and JSON routes, deliberately separate from the LLM TrainingBackend (whose lifecycle -- LLM config build, per-run SQLite rows, matplotlib plots, transfer-to-chat- inference -- is text-training specific and would mis-handle a diffusion run). core/training/diffusion_training_service.py: DiffusionTrainingService runs one job at a time -- validate the config cheaply (before any spawn), spawn the trainer subprocess (spawn context, parent-lifetime bound), pump its events (model_load_* / progress / complete / error) into an in-memory status snapshot, and support a clean stop. The subprocess context and target are injectable so the full start -> pump -> status -> complete path is unit-tested without real multiprocessing or torch. routes/training.py: POST /api/train/diffusion/start (400 on a bad config, 409 when a job is already running), POST /api/train/diffusion/stop, GET /api/train/diffusion/status (JSON poll). models/training.py: DiffusionTrainingStartRequest + response schemas mirroring DiffusionLoraConfig, so model_dump() passes straight through. Tests: test_diffusion_training.py -- service happy path, bad-config-before-spawn, concurrent-job rejection, clean stop, crash-without-terminal-event, event transitions; plus route wiring via the FastAPI TestClient (start / 422 / 400 / 409 / status / stop) with a mocked service. The diffusion trainer's progress events already use the field names this path expects. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Import diffusion training schemas from models.training directly The import-hoist lint flags newly re-exported names in the models/__init__.py hub as unused (it does not treat __all__ membership as a use). Import the three diffusion training schemas straight from models.training in routes/training.py, where they are used in the route annotations and calls, and drop the __init__ re-export. * Remove stray async task scratch outputs committed by mistake * ControlNet: reject filesystem-like ids and do not cache a model past an unload race Two review findings on the ControlNet path: - resolve_controlnet's bare-repo fallback accepted any id with a slash, so a path-shaped id (/tmp/x, ../x) reached from_pretrained as a local directory. Restrict the fallback to a strict owner/name HF repo id shape. - _controlnet_pipe now re-checks the cancel event after the blocking from_pretrained: an unload that raced the download had already cleared the caches, so caching the late module would pin it past the unload. * Pipeline prefetch: fetch only the default torch weights A full-pipeline prefetch kept every repo file outside assets/, so an official repo that ships multiple formats (SDXL Base: fp16 variants, ONNX, OpenVINO, Flax, a top-level single-file twin) downloaded tens of GB from_pretrained never loads. Skip non-torch exports and dtype-variant twins in _pipeline_file_downloaded, and drop a component .bin when the same directory carries a picked safetensors weight (diffusers' own preference). * Diffusion LoRA training: fall back to fp16 when CUDA lacks bf16 The default mixed_precision=bf16 hard-fails on pre-Ampere GPUs (T4 / V100 / RTX 20xx) which have no bf16 compute; check torch.cuda.is_bf16_supported() and drop to fp16 there. * Diffusion training service: join the old pump outside the lock start() joined a finished job's pump thread while holding the service lock, but the pump's final state writes need that same lock, so the join always burned its full timeout and a stale pump could then overwrite the new job's state. Join outside the lock (with a re-check after), and fence _apply_event and the exit handler by process identity so a superseded pump can never touch the current job's state. Adds regression tests for both. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA training: harden config handling, cancellation, SDXL conditioning, and safety Addresses review findings on the SDXL LoRA trainer: - Gate the base model with the same trust check as inference (unsloth/*, allowlisted official bases, or a local path) before from_pretrained, so an untrusted remote repo is never fetched or deserialised. - Check the stop signal before the (slow) model load, not only between steps, so a cancel during download is honoured; a stop may carry save=False to cancel without leaving a partial adapter. - Per-sample SDXL add_time_ids from the actual crop (original size + crop offset, with the offset mirrored on horizontal flip) instead of a fixed uncropped-square tensor. - Apply EXIF orientation before resize/crop so rotated photos train upright. - Skip gradient clipping when max_grad_norm <= 0 (the Studio 'disable' value) instead of scaling every gradient to zero. - Coerce Studio config strings/blanks: learning_rate string to float, blank hf_token to anonymous, gradient_checkpointing 'none'/'true'/'unsloth' to bool; reject a zero/negative lora_alpha or learning_rate. - Alias the generic Studio training payload keys (model_name/max_steps/batch_size/lora_r/ lr_scheduler_type/random_seed) onto the diffusion field names. - Mirror the trained adapter into loras/diffusion so the Images LoRA picker discovers it. - Report worker exceptions in both message and error keys so the failure is not lost. Adds regression tests for the config coercion/validation and aliasing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ControlNet: address review findings on the diffusers path - resolve_controlnet enforces catalog family compatibility so a direct API call cannot load a ControlNet built for another family through the wrong pipeline. - Unknown ControlNet ids now surface as a 400 (call site maps FileNotFoundError to ValueError) instead of a generic 500. - strength 0 disables ControlNet entirely, so a no-op selection never pays the download / VRAM cost; the control image is decoded and validated BEFORE the ControlNet is resolved or built, so a malformed image fails fast for the same reason. - ControlNet loads use the base compute dtype (state.dtype is a display string, not a torch.dtype, so it silently fell back to float32) and honor the base offload policy via group offloading instead of forcing the module resident. - Empty/malformed HF token coerced to anonymous access. - Flux Union ControlNet control_mode mapped from the selected control type. - resolve_controlnet drops the unused hf_token/cancel_event params. - ControlNetSpec validates guidance_start <= guidance_end (clean 422). - Images UI ControlNet Select shows its placeholder when nothing is selected. Adds regression tests for family enforcement and the union control-mode map. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion training API: LLM interlock, pre-spawn VRAM free, path containment, no dropped knobs Four review findings on the diffusion training start path: - It spawned the SDXL trainer without checking the LLM TrainingBackend, so a start while an LLM run was active put two trainers on the same GPU. Add a symmetric interlock: diffusion start returns 409 when LLM training is active, and LLM start refuses while a diffusion job is active. - It went straight to service.start() without freeing GPU residents. Add a pre-spawn free of the export subprocess, the resident Images pipeline (with an arbiter release), and chat models, mirroring the LLM start path. - data_dir / output_dir were passed through unresolved, so Studio-relative names failed and absolute paths bypassed containment. Resolve them with resolve_dataset_path / resolve_output_dir before spawn (400 on an uncontained path). - The request model dropped max_grad_norm and lora_target_modules, so runs that set them trained with defaults. Add both fields. The gemini pump-join deadlock was already fixed earlier (join outside the lock + proc-identity fence). Note: honoring a stop DURING model load is a trainer-loop change owned by the diffusion training engine PR (should_stop polled before the first optimizer step). Adds route + model regression tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA: harden resolution, native tag precedence, and diffusers teardown Address review findings on the LoRA path: - resolve_one: normalise a blank/whitespace hf_token to None (anonymous access) and reject a client-supplied weight file with traversal / absolute path. - resolve_specs: convert FileNotFoundError from an unknown/stale id to ValueError so the route returns 400 instead of a generic 500. - _scan_local: disambiguate local adapters that share a stem (foo.safetensors vs foo.gguf) so each is uniquely addressable. - inject_prompt_tags: the backend-validated weight now wins over a user-typed <lora:ALIAS:...> for a selected adapter; unselected user tags are left alone. - diffusers _apply_loras: reject a .gguf adapter with a clear error before touching the pipe (diffusers loads safetensors only). - _unload_locked: drop the explicit unload_lora_weights() on teardown; the pipe is dropped wholesale (freeing adapters), so the previous call could race an in-flight denoise on the same pipe. - Images page: use a stable LoRA key and clear the selection (not just the options) when the catalog refresh fails. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion: guard trust check against OSError and validate conditioning inputs - _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid characters (or a bare owner/name id) can't raise OSError; treat any failure as not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request still raises the clear FileNotFoundError for a genuinely missing local pick. - generate(): reject mask_image / upscale / reference_images supplied without an input image, and reject reference_images on a family that does not support reference conditioning, instead of silently degrading to txt2img / img2img. * SDXL: reject GGUF up front, skip unused base weights, drop refiner, and harden helpers Addresses review findings on the SDXL family: - Reject a GGUF load for single_file_is_pipeline families (SDXL) in validate_load_request, before the route evicts the current model; SDXL has no transformer-only GGUF variant. - Skip base-repo weight files when a whole-pipeline single file is loaded: from_single_file (config=base) needs only the base config/tokenizer/scheduler, so a local .safetensors no longer triggers a multi-GB base download. - Remove the SDXL refiner from the non-GGUF trust allowlist: it is an img2img-only pipeline but this backend loads every sdxl repo as the base txt2img pipeline. - Normalize a blank/whitespace hf_token to None once in load_pipeline so every load branch degrades to anonymous instead of erroring on a malformed token. - Read the denoiser dtype from a parameter (compile-wrapped modules may lack .dtype) and access state.family.denoiser_attr directly. Adds/updates regression tests for the trust allowlist, GGUF rejection, and base-config filter. * Images: preserve restored LoRAs through model load and never send hidden LoRAs - The LoRA effect cleared the selection on every load->capable transition, which wiped adapters restored from a gallery recipe before the model finished loading. Track the previously-loaded family in a ref and clear only on a real family swap; keep the selection on the initial load and on unload. - Gate the generate payload's loras on loraCapable so a restored selection that is hidden (loaded model does not support LoRA) is never sent to the backend. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Merge diffusion-sdxl into diffusion-lora-ux; keep options-only LoRA catch The catalog-refresh .catch from the lower branch clears the selected adapters too, which is right for its catalog-only picker but wrong here: this picker holds free-text HF repo ids that are valid without being in the catalog, so a transient refresh failure must not wipe them. Family swaps still clear the selection and hidden LoRAs are never sent. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align the VAE to the denoiser's first FLOATING dtype, not its first parameter A GGUF-quantized transformer's leading parameters are packed uint8 storage, so reading next(parameters()).dtype handed nn.Module.to() an integer dtype and every image-conditioned generation on a GGUF model (Qwen-Image-Edit) failed with a 500. Probe the parameters for the first floating dtype, treat an all-integer module as a no-op, and also catch TypeError so an unexpected dtype can never break generation. Regression test included. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Count LR scheduler warmup/decay in optimizer steps, not micro-steps lr_sched.step() runs once per outer optimizer step (after the gradient accumulation inner loop), for train_steps total. The scheduler was configured with num_warmup_steps and num_training_steps multiplied by gradient_accumulation_steps, so with accumulation > 1 a warmup or non-constant schedule stretched past the run and never reached the intended decay. Count both in optimizer steps. * Address Codex review findings on the image-workflows PR Keep diffusion.py importable without torch: the compile/arch patch modules import torch at module level, so import them lazily at their load/unload call sites instead of at module load. This restores the torchless contract so get_diffusion_backend() works on a CPU/native sd.cpp install. Match family reject keywords and aliases as whole path/name segments, not raw substrings, so an unrelated word like edited, edition, or kontextual no longer misroutes or hides a valid base image model, while supported edit families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same segment matching in the picker task filter. Route FLUX.2-dev native guidance through --guidance like the other FLUX families rather than --cfg-scale. Reject native upscale requests that have no input image. Read image header dimensions and reject over-limit inputs before decoding pixels, so a crafted small-payload image cannot spike memory. Reject an upscale that would shrink the source below its input size. Validate the model_kind against the filename extension before the GPU handoff. Estimate a local diffusers pipeline's size from its on-disk weights so auto memory planning does not skip offload and OOM. Report workflows: [txt2img] from the native backend status so the Create tab stays enabled for a loaded native model. Clamp the outpaint canvas to the backend's 4096px decode limit. Adds regression tests for segment matching and kind/extension validation. * Guard inference loads and worker lifetime against diffusion training Teach the chat and image load guards about an active diffusion (SDXL) LoRA job: a chat load is refused (its footprint cannot be fit-checked against the trainer) and an image load is refused outright, mirroring the existing LLM training guards, so a load can no longer allocate GPU memory alongside the trainer and undo the pre-start cleanup. Bind the diffusion trainer subprocess to the parent's lifetime and scrub the native path lease secret from it by running the child through run_without_native_path_secret, matching the inference/export/LLM workers, so a Studio crash or kill no longer leaves the trainer holding the GPU. Reset in_model_load on the complete and error terminal events: a stop or failure during model loading otherwise leaves the status reporting a stale loading indicator after the job has ended. * Harden diffusion LoRA handling on the diffusers and native paths Reject LoRA on a torch.compile'd diffusers transformer (Speed=default/max): diffusers requires the adapter loaded before compilation, so applying one to the already-compiled module fails with adapter-key mismatches. The status gate now hides the picker and generate raises a clear message instead. Convert a cancelled Hub LoRA download (RuntimeError Cancelled) to the diffusion cancellation sentinel in resolve_specs, so an unload/superseding load during resolution maps to a 409 instead of a generic server error. Drop weight-0 LoRA rows before the native support gate so a request carrying only disabled adapters stays a no-op on families where native LoRA is unsupported, matching the diffusers path. Reject duplicate LoRA ids in the request model: both apply paths suffix colliding names, so a repeated id would stack the same adapter past its per-adapter weight bound. Strip all user-typed <lora:...> prompt tags on the native path (only the selected adapters are materialized in the managed lora-model-dir, so an unselected tag can never resolve), and restore saved LoRA selections from a gallery recipe so restore reproduces a LoRA image. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden ControlNet resolve, gallery metadata, and the control-type picker Check cancellation immediately after a ControlNet from_pretrained and before any device placement, so an unload/eviction that raced the download does not allocate several GB onto the GPU after the load was already cleared. Require a loadable weight or shard index (not just config.json) before a local ControlNet folder is advertised, so an interrupted copy is hidden instead of failing deep in from_pretrained as a generic 500. Do not record a strength-0 ControlNet in the gallery recipe: it is treated as disabled and skipped, so the image is unconditioned and the metadata must not claim a ControlNet was applied. Build the control-type picker from the selected ControlNet's advertised control_types instead of a hardcoded passthrough/canny pair, so a union model with a precomputed depth or pose map sends the correct control_mode. * Address further Codex findings on the image-workflows PR - Persist the actual output image size in the gallery recipe instead of the request sliders: Transform/Inpaint/Edit derive the size from the uploaded image, Extend grows the canvas, and Upscale resizes it, so the sliders recorded (and later restored) the wrong dimensions for those workflows. - Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file name) in validate_load_request, so the unloadable pick fails before chat is evicted rather than deep in from_pretrained. - Only publish an image-conditioned from_pipe wrapper to the shared aux cache when the load is still current: from_pipe runs under the generate lock but not the state lock, so an unload racing its construction could otherwise cache a wrapper over torn-down modules that a later load would reuse. - Verify the Windows CUDA runtime archive checksum before extracting it, like the main sd-cli archive, so a corrupt or tampered runtime is rejected rather than extracted next to the binary. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse non-SDXL base models at diffusion training start The trainer only supports the SDXL U-Net, but a FLUX / Qwen-Image / Z-Image repo or a GGUF filename passed as base_model was accepted and then failed minutes later inside StableDiffusionXLPipeline.from_pretrained with an unrelated-looking error. Add a name-based guard in normalized() so known DiT-family names and .gguf checkpoints are rejected up front, which the API start route surfaces as an immediate 400 with a message that says exactly which bases are trainable. Unrecognisable names still pass through so custom local SDXL checkpoints keep working. * Add diffusion dataset upload and training info endpoints Training an image LoRA required knowing the Studio home layout and copying files onto the server by hand, which is the most confusing step of the whole flow. Two small endpoints fix that: - GET /api/train/diffusion/info reports the datasets and outputs roots plus every dataset folder that contains images (with image/caption counts), so the UI can offer a picker instead of a blind free-text path. - POST /api/train/diffusion/dataset uploads images and optional caption .txt / metadata.jsonl files into a named folder under the datasets root, creating it on first use and accumulating on repeat uploads so large sets can arrive in batches. Names are validated to a single path component and files stream to disk under the same per-upload size cap as LLM dataset uploads. The returned name is a valid data_dir for /diffusion/start. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Validate diffusion training config before freeing the GPU The start route freed resident GPU workloads (export, Images pipeline, chat) before the service validated the config, so a start that was then refused, now including a non-SDXL base model, tore down the user's loaded model for nothing. Run the same cheap normalise pass first; the LLM path already follows this rule via its before_spawn hook. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
|
Daniel Han
|
1601562738
|
Add diffusion LoRA training (SDXL text-to-image) (#6791)
* Studio diffusion: cross-platform device policy, fp16 guard, lock split, validate-before-evict Phase 1 of porting the richer diffusion stack onto the image-generation backend. - Add a compartmentalized device/dtype policy module (diffusion_device.py) resolving CUDA/ROCm/XPU/MPS/CPU with capability flags. Keeps the NVIDIA capability-based bf16 choice; ROCm and XPU are isolated; MPS uses bf16 or fp32, never a silent fp16 that renders a black image. - Add a per-family fp16_incompatible flag (Z-Image) and promote a resolved float16 to float32 for those families so they do not produce black images. - Split the backend locks: a generation holds only _generate_lock, so status, unload, and a new load are never blocked by a long denoise. Add per-generation cancellation via callback_on_step_end so an eviction or a superseding load preempts a running generation; a replacement load waits for it to stop before allocating, so two pipelines never sit in VRAM at once. - Validate a load request before the GPU handoff so an unloadable pick never evicts a working chat model, and reject missing local paths up front. - Add CPU-only tests for the device policy, dtype guard, lock split and cancellation, and validate-before-evict, plus a GPU benchmark/regression script (scripts/diffusion_bench.py) measuring latency, peak VRAM, and PSNR against a saved reference. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2A): measured-budget memory planner + offload/VAE policy Add a lean, backend-agnostic memory policy that picks a CPU-offload policy and VAE tiling/slicing from measured free device memory vs the model's estimated resident footprint, then applies it to the built pipeline. auto stays resident when the model fits (byte-identical to the prior resident path), and falls to whole-module offload when tight; fast/balanced/low_vram are explicit overrides. Sequential submodule offload is unreliable for GGUF transformers on diffusers 0.38, so it falls back to whole-module offload and status reports the policy actually engaged. Verified on Z-Image-Turbo Q4_K_M (B200): auto reproduces the resident image with no VRAM/latency regression (PSNR inf); balanced/low_vram cut generation peak VRAM 47.9% (15951 -> 8318 MB) with byte-identical output, at the expected latency cost. 73 prior + 35 new CPU tests pass. * Studio diffusion (Phase 2D): streamed block-level offload + functional VAE tiling Add a streamed 'group' offload tier (diffusers apply_group_offloading, block_level, use_stream) that keeps the transformer flowing through the GPU a few blocks at a time while the text encoder / VAE stay resident, and fix VAE tiling to drive the VAE submodule (pipelines like Z-Image expose enable_tiling on pipe.vae, not the pipeline). apply_memory_plan now returns the (policy, tiling) actually engaged so status never overstates either, and group falls back to whole-module offload when the transformer can't be streamed. Measured on Z-Image (B200), all lossless (PSNR inf vs resident): balanced/group cuts generation peak VRAM 32% (15951 -> 10840 MB) at near-resident speed (2.07 -> 2.99s); low_vram/model cuts it 48% (-> 8318 MB) but is slower (7.99s). Mode names now match that tradeoff: balanced = stream the transformer, low_vram = offload every component. auto picks group when the companions fit resident, else model. 112 CPU tests pass. * Studio diffusion (Phase 5): image quality-vs-quant accuracy harness Add scripts/diffusion_quality.py, the accuracy analogue of the KLD workflow: hold prompt + seed fixed, render a grid with a reference quant (default BF16), then render each candidate quant and measure drift from the reference. Records mean PSNR + SSIM (pure-numpy, no skimage/scipy) and optional CLIP text-alignment + image-similarity (transformers, --clip), plus file size, latency, and peak VRAM, then prints a quality-vs-cost table and recommends the smallest quant within a quality budget. --selftest validates the metrics on synthetic images with no GPU or model. Verified on Z-Image (B200): the table degrades monotonically with quant size (Q8 -> Q4 -> Q2: PSNR 21.7 -> 15.5, SSIM 0.82 -> 0.61), while CLIP-text stays flat (~0.34) -- quantization erodes fine detail far more than prompt adherence. * Studio diffusion (Phase 3): opt-in speed layer (channels_last / compile / TF32) Add a speed_mode knob (off by default, so the render path stays bit-identical): default applies channels_last VAE + regional torch.compile of the denoiser's repeated block where eligible; max also enables TF32 matmul and fused QKV. Regional compile is gated off for the GGUF transformer (dequantises per-op) and for families flagged not compile-friendly (a new supports_torch_compile flag, False for Z-Image), so it activates automatically only once a non-GGUF bf16 transformer is loaded. Speed optims run before placement/offload, per the diffusers composition order. status now reports speed_mode + the optims actually engaged. Verified on Z-Image (B200): default -> ['channels_last'], max -> ['channels_last', 'tf32'], compile correctly skipped for GGUF; generation works in every mode. 121 CPU tests pass. * Studio diffusion (Phase 2B): opt-in fp8 text-encoder layerwise casting Add a text_encoder_fp8 knob that casts the companion text encoder(s) to fp8 (e4m3) storage via diffusers apply_layerwise_casting, upcasting per layer to the bf16 compute dtype while normalisations and embeddings stay full precision. Applied before placement, gated to CUDA + bf16, best-effort (a failure leaves the encoder dense). status reports which encoders were cast. Verified on Z-Image (B200, balanced/group mode where the encoder stays resident): generation peak VRAM dropped 37% (10840 -> 6791 MB, below the lowest-VRAM offload) at near-resident speed. It is a memory-vs-quality tradeoff, not free -- ~20 dB PSNR vs the bf16 encoder, a larger shift than one transformer quant step -- so it is off by default and documented as such, with the Phase 5 harness to size the cost. 127 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2C): NVFP4 text-encoder quant (+ generalise fp8 knob) Generalise the text-encoder precision knob from a fp8 bool to text_encoder_quant (fp8 | nvfp4). nvfp4 quantises the companion text encoder to 4-bit via torchao NVFP4 weight-only (two-level microscaling) on Blackwell's FP4 tensor cores; fp8 stays the broader-hardware path (cc>=8.9). Both are gated, best-effort, and run before placement; status reports the mode actually engaged. This is the lean realisation of GGUF-native text-encoder quant: 4-bit on the encoder without the 3045-line port. Verified on Z-Image (B200, balanced/group where the encoder stays resident), vs the bf16 encoder: nvfp4 cut generation peak VRAM 48% (10840 -> 5593 MB, the lowest TE option, below whole-model offload) at near-fp8 quality (16.4 vs 17.1 dB PSNR), and both quants ran faster than bf16. A memory-vs-quality tradeoff (off by default); size it per model with the Phase 5 quality harness. diffusion_bench gains --text-encoder-quant. 129 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): native stable-diffusion.cpp engine for CPU/Mac Adds the CPU / Apple-Silicon tier of the two-engine strategy, mirroring the chat backend's llama.cpp shell-out. Diffusers stays the default on CUDA / ROCm / XPU; this covers the hardware diffusers serves poorly, consuming the same split GGUF assets Studio already curates. - sd_cpp_args.py: pure sd-cli command builder. Maps the family to its text-encoder flag (Z-Image Qwen3 to --llm, Qwen-Image to --qwen2vl, FLUX.1 CLIP-L + T5), and the diffusers memory policy (none/group/model/sequential) to sd.cpp's offload flags (--offload-to-cpu / --clip-on-cpu / --vae-on-cpu / --vae-tiling / --diffusion-fa), so one user knob drives both engines. - sd_cpp_engine.py: SdCppEngine over a located sd-cli. find_sd_cpp_binary() with the same precedence as the llama finder (env override, then the Studio install root, then in-tree, then PATH), an is_available/version probe, and a one-shot subprocess generate that streams progress and returns the PNG. runtime_env() prepends the binary's directory to the platform library path so a prebuilt's bundled libstable-diffusion.so resolves. select_diffusion_engine() is the pure routing decision (GPU backends to diffusers, CPU/MPS to native when present). - install_sd_cpp_prebuilt.py: resolve + download the per-host prebuilt (macOS-arm64/Metal, Linux x86_64 CPU, Vulkan/ROCm/Windows variants) into the Studio install root. resolve_release_asset() is a pure, unit-tested host-to-asset matrix. - scripts/sd_cpp_smoke.py: end-to-end native generation harness. Tests (CPU-only, subprocess/filesystem stubbed): 49 new across args, engine, routing, runtime env, and the installer resolver. Full diffusion suite 166 passing. Verified on a B200 box: built sd-cli (CUDA) and the prebuilt (CPU) both generate Z-Image-Turbo Q4_K end to end through SdCppEngine: balanced (group offload, 5.0s gen), low_vram (full CPU offload + VAE tiling, 13.4s), and the dynamically-linked CPU prebuilt (50.4s on CPU), all producing coherent images. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 6): img2img / inpaint / edit / LoRA / upscale on the native engine Builds on Phase 4's native stable-diffusion.cpp engine, extending it from text-to-image to the wider feature surface, since sd.cpp supports all of these through the binary already. Pure command-builder additions plus one engine method, so the txt2img path is unchanged. - sd_cpp_args.py: SdCppGenParams gains image-conditioning fields. init_img + strength make a run img2img, adding mask makes it inpaint, ref_images drives FLUX-Kontext / Qwen-Image-Edit style editing (repeated --ref-image), and lora_dir + the <lora:name:weight> prompt syntax select LoRAs. New SdCppUpscaleParams + build_sd_cpp_upscale_command for the ESRGAN upscale run mode (input image + esrgan model, no prompt / text encoders). - sd_cpp_engine.py: the subprocess runner is factored into a shared _run() so generate() (now carrying the conditioning flags) and a new upscale() reuse the same streaming / error / output-check path. - scripts/sd_cpp_smoke.py: --task {txt2img,img2img,upscale} with --init-img / --strength / --upscale-model / --upscale-repeats. Tests: 10 new across the img2img / inpaint / edit / LoRA flag construction, the upscale builder and its validation, and the engine's img2img + upscale paths. Full diffusion suite 176 passing. Verified on a B200 box through SdCppEngine: img2img (Z-Image-Turbo Q4_K, the init image conditioned at strength 0.6, 4.8s) and ESRGAN upscale (512x512 -> 2048x2048 via RealESRGAN_x4plus_anime_6B, 2.7s), both producing coherent images. Video and the diffusers-path feature wiring are deferred. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): accuracy-preserving speed pass Re-review of the diffusion stack (#6675/#6679/#6680) surfaced one real accuracy bug and a dead-on-arrival speed path; this fixes both and adds the lossless / near-lossless wins, all measured on a B200. Correctness: - TF32 global-state leak (fix). speed_mode=max flipped torch.backends.*.allow_tf32 process-wide and never restored them, so a later `off` load silently inherited TF32 and was no longer bit-identical. Added snapshot_backend_flags / restore_backend_flags (TF32 + cudnn.benchmark), captured before the speed layer runs and restored on unload. Verified: load max -> unload -> load off is now byte-identical (PSNR inf) to a fresh off. - sd-cli timeout could hang forever. _run() blocked in `for line in stdout` and only checked the timeout after EOF, so a child stuck in model load / GPU init with no output ignored the timeout. Drained stdout on a reader thread with a wall-clock deadline. Added a silent-hang regression test. Speed (diffusers path), near-lossless, opt-in tiers: - Regional torch.compile now runs on the GGUF transformer. The is_gguf gate (and Z-Image's supports_torch_compile=False) were stale: compile_repeated_blocks compiles and runs ~2.2x faster on the GGUF Z-Image transformer on torch 2.9.1 / diffusers 0.38 (the per-op dequant stays eager, the rest of the block compiles). Measured: off 1.80s -> default 0.82s/gen (+54.7%), PSNR 37.7 dB vs eager -- far above the Q4 quant noise floor (~21 dB), so it does not move output quality. Gate relaxed; default tier delivers it. - cudnn.benchmark added to the default tier (autotunes the fixed-shape VAE convs). - torch.inference_mode() around the pipeline call (lossless, strictly faster than the no_grad diffusers uses internally). Memory path: - VAE tiling (not bit-identical >1MP) restricted to the model/sequential/CPU tiers; the balanced (group) tier keeps exact slicing only, so it is now bit-identical to the resident image (verified PSNR inf) and slightly faster. - Group offload adds non_blocking + record_stream on the CUDA stream path to overlap each block's H2D copy with compute (lossless; gated on the installed diffusers signature so older versions still work). Native (sd.cpp) path: - native_speed_flags: a first-class speed knob (default -> --diffusion-fa, a near-lossless CUDA win that was previously only added on offload tiers; max also -> --diffusion-conv-direct). conv-direct stays opt-in: measured +45% on CUDA, so it is never auto-on. Engine generate() merges it, de-duped against offload flags. Default profile: a GGUF model with no explicit speed_mode now resolves to the `default` profile (resolve_speed_mode), since compile's perturbation sits below the quantisation noise floor and so does not reduce quality versus the dense reference; out of the box a GGUF Z-Image generation drops from 1.80s to 0.81s. Dense models stay `off` / bit-identical, and an explicit speed_mode -- including "off" -- is always honored, so the byte-identical path remains one flag away and is the regression reference. Tooling: scripts/compile_probe.py (eager vs compiled GGUF probe), scripts/ perf_verify.py (the B200 verification above), and diffusion_bench.py gains --speed-mode so the speed tiers are benchmarkable. Tests: 183 passing (was 166); new coverage for the backend-flag snapshot/restore, GGUF compile eligibility, the balanced tiling/slicing split, native_speed_flags + the engine de-dup, and the sd-cli silent-hang timeout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): max tier uses max-autotune-no-cudagraphs + engine/lever benchmarks The opt-in `max` speed tier now compiles the repeated block with mode=max-autotune-no-cudagraphs (dynamic=False) instead of the default mode: Triton autotuning for GEMM/conv-heavier models, gated to the tier where a longer cold compile is acceptable. CUDA-graph modes (reduce-overhead / max-autotune) are deliberately avoided -- both crash on the regionally-compiled block (its static output buffer is overwritten across denoise steps), measured. Adds two reproducible benchmarks used to validate the optimization research: - scripts/compare_engines.py: PyTorch (diffusers GGUF) vs native sd.cpp head-to-head. - scripts/leverage_probe.py: coordinate_descent_tuning + FirstBlockCache probes. Measured on B200 (Z-Image Q4_K_M, 1024px, 8 steps): default compile 0.80s/gen; coordinate_descent_tuning 0.79s (within noise, already covered by max-autotune); FirstBlockCache does not run on Z-Image (diffusers 0.38 block-detection / Dynamo). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source) Add an opt-in transformer_quant mode that loads the dense bf16 transformer and torchao-quantises it onto the low-precision tensor cores, instead of the GGUF transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's 0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor. GGUF+compile stays the low-memory default and the fallback. The mode is gated on CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB); any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe (Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips the tiny projections that crash int8's torch._int_mm. New module mirrors diffusion_precision.py; quant runs before compile before placement. 184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py is the standalone torchao lever probe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): consumer-GPU tuning - lock fp8 fast accumulate, prefer fp8 over mxfp8, reject 2:4 sparsity Consumer Blackwell halves tensor-core throughput on FP32 accumulate (fp8 419 vs 838 TFLOPS with FP16 accumulate; bf16 209), so: - fp8 config locks use_fast_accum=True (Float8MMConfig). torchao already defaults it on; pinning it guards consumer cards against a default change. On B200 it is identical speed and slightly better quality (LPIPS 0.050 vs 0.091). - the Blackwell auto ladder prefers fp8 over mxfp8 (measured faster + more accurate). 2:4 semi-structured sparsity evaluated and rejected (scripts/sparse_accum_probe.py): 2:4 magnitude-prune + fp8 gives LPIPS 0.858 (broken image) with no fine-tune, the cuSPARSELt kernel errors on torch 2.9, and it does not compose with torch.compile (our main ~2x). Documented as a dead end, not shipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add fp8 fast-accum overflow verification probe scripts/fp8_overflow_check.py hooks every quantised linear during a real Z-Image generation and reports max-abs + non-finite counts for use_fast_accum True vs False. Confirms fast accumulation is an accumulation-precision knob, not an overflow one: across 276 linears, including Z-Image's ~1.0e6 activation peaks (which overflow FP16), 0 non-finite elements and identical max-abs for both modes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40; GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it. Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s), no overflow, quality below the quant noise floor. So the default leans to accuracy on data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force) lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off). 187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override threading). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add NVFP4 probe documenting it is not yet a win on torch 2.9 scripts/nvfp4_probe.py measures NVFP4 via torchao on the real Z-Image transformer. Finding (B200, 1024px/8 steps): NVFP4 is a torchao feature and DOES run with use_triton_kernel=False (the default triton path needs the missing MSLK library), but only at bf16-compile rate (0.667s vs fp8 0.592s) -- it dequantises FP4->bf16 rather than using the FP4 tensor cores. The real FP4 speedup needs MSLK or torch>=2.11 + torchao's CUTLASS FP4 GEMM. The smoke probe (default triton=True) already keeps NVFP4 out of auto on this env, so auto correctly stays on fp8; NVFP4 activates automatically once fast. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): prefer fp8 over nvfp4 in Blackwell auto ladder Validated NVFP4 on torch 2.11 + torchao CUTLASS FP4 in an isolated env. The FP4 tensor-core GEMM is genuinely active there (a 16384^3 GEMM hits ~3826 TFLOPS, 2.52x bf16 and 1.37x fp8), but it only beats fp8 on very large GEMMs. At the diffusion transformer's shapes (hidden ~3072, MLP ~12288, M~4096) NVFP4 is both slower (0.81x fp8 end to end on Z-Image 1024px) and less accurate (LPIPS 0.166 vs fp8's 0.044). Reorder the Blackwell auto ladder to fp8 before nvfp4 so auto is correct even on a future MSLK-equipped box; nvfp4 stays an explicit opt-in. Add scripts/nvfp4_t211_probe.py (extension diagnostics + GEMM micro + end-to-end). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): pre-quantized transformer loading The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs 13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline (scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True) the quantized weights, so the dense bf16 never touches the GPU. Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same torchao config + min_features filter the runtime path uses, applied ahead of time. New core/inference/diffusion_prequant.py (resolve_prequant_source + load_prequantized_transformer, best-effort, lazy imports). diffusion.py _load_dense_quant_pipeline tries the pre-quant source first and falls back to the dense materialise+quantise path, then to GGUF, so the default is unchanged. DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for the resolver, the meta-init+assign loader, and the backend branch selection + fallbacks; GPU verification via scripts/verify_prequant_backend.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): attention-backend selection Add a selectable attention kernel via the diffusers set_attention_backend dispatcher. Attention is memory-bandwidth bound, so a better kernel is an end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV matmuls torchao never touches) and composes with torch.compile. auto picks the best exact backend for the device: cuDNN fused attention (_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default (below the compile/quant noise floor); native SDPA elsewhere and when speed=off (so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/ xformers/aiter are honored, and an unavailable kernel falls back to the default rather than failing the load. New core/inference/diffusion_attention.py (normalize + per-device select + apply, best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline; attention_backend threads through begin_load / load_pipeline / status like the other load knobs. New request field attention_backend + status field. Hermetic CPU tests for normalize / select policy / apply fallback, plus route threading + 422. Measured via scripts/perf_levers_probe.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the arch tier to put int8 first on a consumer/workstation GPU (detected by the existing _is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first. Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation (-> int8) and data-center Ada/Hopper/Blackwell (-> fp8). * Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across denoise steps a DiT's output settles, so once the first block's residual barely changes the remaining blocks are skipped and their cached output reused. diffusers ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the standalone apply_first_block_cache hook as a fallback). Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 -> 2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar. OFF by default and a per-load opt-in: the win scales with step count, so it is for many-step models (Flux / Qwen-Image) and pointless for few-step distilled models (e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction of the trajectory. It composes with regional compile only with fullgraph=False (the cache's per-step decision is a torch.compiler.disable graph break), which the speed layer now switches to automatically when a cache is engaged. Best-effort: a model whose block signature the hook does not recognise is caught and the load proceeds uncached. - new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache (enable_cache / apply_first_block_cache fallback; threshold auto-raised for a quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import). - diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph when a cache is engaged. - diffusion.py: apply_step_cache before compile; thread transformer_cache / transformer_cache_threshold through begin_load -> load_pipeline and report the engaged mode in status(). - models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and transformer_cache_threshold request fields, engaged mode in the status response. - hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold selection, and best-effort failure handling, plus route threading + validation. - scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM / LPIPS vs the compiled no-cache baseline). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears) The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with 'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2 *_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing min_features filter did not exclude them. Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed, guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers (M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and quantises these layers fine, so the exclusion is int8-only. Sequence embedders (context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note 'context_embedder' contains the substring 'text_embed', which is why the token is the specific 'time_text_embed', not 'text_embed'. Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s (5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already working) are unchanged. - diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes exclude_name_tokens; quantize_transformer passes it for int8 only. - hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps attention / FFN / sequence-embedder linears), while fp8 keeps them. - scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's Linear layers and derive the exclusion list. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears) The prequant-checkpoint builder applied the dense quant filter without the int8-only M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8 checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now produces a working checkpoint on every supported model, giving int8 (the consumer-preferred scheme) the same ~2x load-VRAM and download reduction fp8 already had. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the native stable-diffusion.cpp engine instead of diffusers, with diffusers as the guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM. - diffusion_engine_router: centralised engine selection (built on the existing select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason. - sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface backed by sd-cli, with lazy binary install, registry-driven asset fetch, step-progress parsing, and cancellation. - diffusion_families: per-family single-file VAE + text-encoder asset mapping. - sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled). - routes/inference + gpu_arbiter: drive the active engine via the router; the API now reports the active engine and any fallback reason. - tests for the backend, router, route selection, and cancellation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler Address review feedback on #6724: - engine router: unload the engine being deactivated on a switch, so the old model is not left resident-but-unreachable (the evictor only targets the active engine). - generate route: sd.cpp execution errors (nonzero exit / timeout / missing output) now map to 500, not 409 (which only means not-loaded / cancelled). - native batch: return per-image seeds and persist the actual seed for each image so every batch image is reproducible. - Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the stable-diffusion.cpp docs; other families keep sd-cli defaults. - honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload offload flags on the native load instead of hardcoding them off. - fail the load when the sd-cli binary is present but not runnable (version() now returns None on exec error / nonzero exit). - size estimate: only treat the transformer asset as a possible local path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in load_prequantized_transformer ends in torch.load(weights_only=False), which executes arbitrary code from the pickle. The transformer_prequant_path load-request field reached that unpickle for any local file an authenticated caller named, so a request could trigger remote code execution. Refuse the source.kind=='path' branch unless the operator sets UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted and unaffected. Document the requirement on the API field and add gate tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa - apply_attention_backend now restores the native default when no backend is requested or a kernel fails. diffusers keeps a process-wide active attention backend that set_attention_backend updates, and a fresh transformer's processors follow it, so a load that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile load pinned, breaking the bit-identical/off guarantee. - select_attention_backend drops flash3/flash4 up front when the CUDA capability is below Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit request on the wrong card set fine then crashed mid-generation; it now falls back to native. - Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a valid alias of native) is accepted instead of 422-rejected by Pydantic. - Drop the dead replace('-','_') normalization (no alias uses dashes/underscores). - perf_levers_probe.py output dir is now relative to the script, not a hardcoded path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF - apply_step_cache now engages only via the transformer's native enable_cache (the diffusers CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model now runs uncached per the best-effort contract. - GGUF transformers are quantized (the default Studio load path), so they now use the higher quantized FBCache threshold when the caller leaves it unset, instead of the dense default that could keep the cache from triggering. - fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so fullgraph=True failed warmup and silently measured an eager cached run); output dir is now relative to the script, not a hardcoded path. * Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder _is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate. Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0). * Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just TypeError on older torchao), which would break the mxfp8 config helper instead of falling back to the default. Catch both so the fallback is robust. quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds CUDA memory during the per-row VRAM probe; output dir relative to the script. * Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load - snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the whole snapshot. restore_backend_flags restores each flag independently so one failure can't leave the others leaked process-wide. - load_pipeline restores the flags (and clears the GPU cache) when the build fails after apply_speed_optims mutated the process-wide flags but before _state captured them for unload to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and contaminated later off generations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs without producing output (or without closing stdout) would never reach proc.wait and the wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the PROCESS, so the main thread always enforces the timeout and kills a hung process (which closes the pipe and ends the reader). Add a test that times out even when stdout blocks, and make the no-binary test hermetic so a host-installed sd-cli can't leak in. * Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn The filter callback can be invoked without a module name, so fqn.lower() would raise AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens, so the linear is kept) instead of crashing the quantise pass. * Studio diffusion (Phase 16) review fixes: native engine robustness - sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64); a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds aliased to the same image. Random seeds stay 53-bit (JS-safe). - sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub fall back to anonymous instead of failing auth on a blank token. - sd_cpp_backend: a superseding load now cancels the in-flight generation, so the old sd-cli can no longer return/persist an image from the previous model. - diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a slow 10+ GB free / CUDA sync does not block engine selection. - diffusion_engine_router: probe sd-cli runnability (version()) before committing to native, so a present-but-unrunnable binary falls back to diffusers at selection. - diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a CPU-only install can still reach the native sd.cpp engine instead of failing load. - tests updated for the runnability probe + a not-runnable fallback case. * Studio diffusion (Phase 9) review fixes: prequant safety + validation - SECURITY: a request-supplied local pre-quant path is now unpickled only when it resolves inside an operator-configured ALLOWLIST of directories (UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in, once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on any path a load request named (arbitrary code execution). realpath() blocks symlink escapes; a bare on/off toggle is no longer a wildcard. - Validate the checkpoint's min_features against the runtime Linear filter, so a checkpoint that quantised a different layer set is rejected instead of silently loading a model that mismatches the dense path while reporting the same scheme. - Tolerant base_model_id compare (exact or same final path/repo segment), so a local path or fork of the canonical base is accepted instead of falling back to dense. - _has_meta_tensors uses any(chain(...)) (no intermediate lists). - prequant verify/probe scripts use repo-relative paths (+ env overrides), not the author's absolute /mnt paths. - tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts - diffusion_memory: when group offload is unavailable and the plan falls back to whole-module offload, enable VAE tiling (the group plan left it off, but the fallback is the low-VRAM path where the decode spike can OOM). Covers both the group and sequential fallback branches. - perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a balanced bit-identity regression actually fails the check instead of exiting 0. - compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and the load-progress poll has a 30 min deadline instead of looping forever on a hang. - test for the group->model fallback enabling VAE tiling. * Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path - diffusion: a torchao-quantized transformer is committed only compiled. A dense model resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF it replaced), so when transformer_quant engaged and speed resolved to off, promote to default (regional compile); warn loudly if compile still does not engage. - diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK); otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF. - nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path). - test asserts the eager-quant -> default-compile promotion. * Studio diffusion (Phase 10) review fixes: attention gating + probe isolation - diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation, so auto now stays on native SDPA there. - diffusion_attention: _active_attention_backend handles get_active_backend() returning an enum/None (not a tuple); the old unpack always raised and was swallowed, so the native-restore short-circuit never fired. - perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul so the inductor_flags variant doesn't leak into later compiled rows. - tests for the SM80 cuDNN gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening - install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket); extract through a per-member containment check (Zip-Slip guard); expanduser the --install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch the separately-published cudart runtime DLL archive so sd-cli.exe can start. - sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie. - tests: Zip-Slip rejection, normal extraction, studio-home discovery. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of the literal --output path. SdCppEngine.generate checked only the literal path, so a batch generation would exit 0 and then raise 'no image' (or return a stale file). generate now returns the literal path when present and otherwise falls back to the numbered siblings; single-image behavior is unchanged. Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected without error. * Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats Codex review on the native engine arg builder: - build_sd_cpp_command emitted --width/--height unconditionally, so an img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of the input. width/height are now Optional (None = unset): an image-conditioned run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives the size from the input image (set_width_and_height_if_unset); a plain txt2img run with unset dims keeps the prior 1024x1024 default; explicit dims are always honored. width/height are read only by the builder, so the type change is local. - build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...) that silently swallowed repeats=0 into sd-cli's default of one pass, turning an explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError and emits the flag for any explicit value != 1. Tests: img2img unset dims omit width/height (init_img and ref_images), explicit dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at the default. (Two pre-existing binary-discovery tests fail only because a real sd-cli is installed in this dev environment; unrelated to this change.) * Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc Codex review: the transformer_prequant_path field description still told operators to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the prior security fix made that variable a directory allowlist -- _allowed_prequant_roots deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator following the documented =1 would have every transformer_prequant_path request silently refused. The description now states it must name one or more allowlisted directories and that a bare on/off value is not accepted. Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does not say =1, and describes an allowlist/directory (guards against doc drift). * Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset Codex review on attention-backend selection: - Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first generation with no fallback. select_attention_backend now applies _cudnn_attention_supported() to an explicit cuDNN request too. - flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3 is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a (min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+. - apply_attention_backend's success path left diffusers' process-wide active backend pinned to the kernel it set; a later component whose processors are unconfigured (backend None) would inherit it. It now resets the global registry to native after a successful per-transformer set (the transformer keeps its own backend), best-effort. Also fixed _active_attention_backend: get_active_backend() returns a (name, fn) tuple, so the prior code stringified the tuple and never matched a name, defeating the native-restore short-circuit. Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on SM90; global registry reset after a successful set; _active_attention_backend reads the tuple return. * Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300 (Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So _is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they are treated as data-center class and keep the intended fp8-first behavior. Test: extends the datacenter parametrize with 'NVIDIA B300' and 'NVIDIA GH200 480GB' (now _is_consumer_gpu False). * Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder Codex review: the M=1 modulation/embedder exclusion was wired only into the dense runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint quantised the AdaLN modulation and conditioning-embedder linears, and loading it via transformer_prequant_path (the load path only loads already-quantised tensors, it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes for the runtime path. Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 -> the M=1 exclusion, every other scheme -> none) and use it in both the runtime quantiser and the builder, so a prequant artifact's quantised-layer set always matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion). Test: int8_exclude_name_tokens returns the exclusion for int8 and () for fp8/nvfp4/mxfp8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race Codex review on the native-engine routing: - The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat) unconditionally after engine selection. A native sd.cpp load on a pure-CPU host never touches the GPU, so that needlessly tore down the resident chat model. The handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it. - sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when _run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so the setting was unverifiable. status now derives them from state.offload_flags (still 'none' on CPU, where the flags are empty). - _run_load committed the new state without cancelling/waiting on a generation that started during the (slow) asset download, so a stale sd-cli run against the OLD model could finish afterward and persist an image from the previous model once the new load reported ready. The commit now signals the in-flight cancel and waits on _generate_lock before swapping _state (taken only at commit, so the download never serialises against generation), mirroring the diffusers load path. Tests: CPU native load skips the arbiter while a GPU native load takes it; status reports offload active when flags are set; _run_load cancels and waits for an in-flight generation before committing. * Studio diffusion (Phase 14) review round 2: align helper name with the stack Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the identical helper already present higher in the diffusion stack (Phase 16). The helper definition, the runtime quantiser call, and the offline builder are now byte-identical to that version, so the two branches no longer introduce a divergent name for the same single-source-of-truth and the stack merges without a conflict on this fix. No behavior change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: eager patches + torch.compile cache speed phase Adds the opt-in speed path for the GGUF diffusion transformer behind a selectable speed mode (default off, so output is unchanged until a profile is chosen): - diffusion_eager_patches.py: shared eager fast-paths (channels_last, attention/backend selection, fused norms and QKV) installed at load and rolled back on unload or failed load. - diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent torch.compile cache and the GGUF-transformer compile wiring. - diffusion_arch_patches.py: architecture-specific patches. - diffusion_patch_backend.py: shared install/restore plumbing. - diffusion_speed.py: speed-profile planning. Tests for each module plus the benchmarking and probe scripts used to measure speed, memory, and accuracy of the path. * Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI Backend: - Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file fp8 transformers, gated to the unsloth org plus a curated allowlist. - Image-conditioned workflows built with Pipeline.from_pipe so they reuse the loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint, outpaint, and a hires-fix upscale pass. - Instruction editing as its own family kind (Qwen-Image-Edit-2511, FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and multi-reference) plus klein inpaint. - Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the decoded image size and cap upscale output to avoid OOM on large inputs. - Fixes: from_pipe defaulting to a float32 recast that crashed torchao quantized transformers; image-conditioned calls forcing the slider size onto the input image. Native sd.cpp engine rejects image-conditioned and reference requests it cannot serve. Frontend: - Redesigned Images page with capability-gated workflow tabs (Create, Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor, client-side outpaint, and a multi-reference picker. - Advanced options moved to a right-docked panel mirroring Chat: closed by default, toggled by a single fixed top-bar button that stays in place. sd.cpp installer: pin the release, verify each download's sha256, add a download timeout, and make the source repo configurable for a future mirror. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: correct the Advanced panel comment (closed by default, fixed toggle) * Studio diffusion: LoRA adapters for the Images workflow Add community LoRA support across both diffusion backends, the single biggest step toward broad image-workflow coverage. Backend - New shared module core/inference/diffusion_lora.py: adapter discovery (local scan + curated catalog + owner/name[:file] Hub refs), download via hf_hub_download_with_xet_fallback, alias sanitization, native managed-dir materialization with collision-broken aliases, prompt-tag injection (deduped against user-typed tags), and a supports_lora gate. - Native sd-cli: resolve + materialize selected LoRAs into a per-run managed dir, inject <lora:ALIAS:w> tags, pass --lora-model-dir with --lora-apply-mode auto. The arg builder already emitted these flags. - Diffusers: non-fused load_lora_weights + set_adapters manager, tracked on the pipe so an unchanged selection is a no-op and a model swap resets; cleared on unload. Never fuses (breaks quantized transformers and blocks live weight tweaks). - Gated off where unsupported: torchao fp8/int8 dense, GGUF-via-diffusers, and native Qwen-Image (no LoRA name-conversion branch upstream). - Request contract: optional loras on DiffusionGenerateRequest; empty or omitted is identical to today. supports_lora surfaced in status; chosen LoRAs persisted in gallery recipe metadata. - New GET /api/models/diffusion-loras for the picker (family-filtered). Frontend - Repeatable multi-LoRA picker (adapter select + weight slider 0..2 + remove), gated by the loaded model's supports_lora and family, max 8. Tests - New test_diffusion_lora.py (14): helpers, request validation, native tag/dir wiring, diffusers set_adapters manager, supports_lora matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: ControlNet for the Images workflow (diffusers) Add ControlNet conditioning, the #2 most-used diffusion workflow after LoRA, on the diffusers backend for the families with ControlNet pipelines (FLUX.1 and Qwen-Image), with Union models as the default picks. Backend - New core/inference/diffusion_controlnet.py: family-gated discovery (curated Union models + local dirs + bare owner/name repos), resolution to a loadable repo/dir, control-image preprocessing (passthrough + a dependency-free canny edge map), and a supports_controlnet gate. - diffusion.py: a ControlNet manager parallel to the LoRA one. Loads the (small) ControlNet model once via from_pretrained (cached by id) and builds the family's ControlNet pipeline via Pipeline.from_pipe(base, controlnet=model), reusing the resident base modules at their loaded dtype (no reload, no recast). Passes the control image + conditioning scale + guidance start/end at generate time; cleared on unload. - Families: FLUX.1 -> FluxControlNetPipeline/Model, Qwen-Image -> QwenImageControlNetPipeline/Model. Others declare none (gated off). - Gated off for the native engine, GGUF-via-diffusers, and torchao fp8/int8 dense (same rule as LoRA). v1 conditions txt2img only. - Request contract: optional controlnet on DiffusionGenerateRequest; supports_controlnet in status; the choice persisted in gallery meta. - New GET /api/models/diffusion-controlnets for the picker. Frontend - A ControlNet control in the Images rail (model select + control-image upload + control-type select + strength slider), gated by the loaded model's supports_controlnet + family, shown for text-to-image. Tests - New test_diffusion_controlnet.py (10): discovery/resolve/preprocess/gate helpers, request validation, family wiring, and the diffusers pipe manager (loads once, caches, from_pipe with controlnet, rejects unsupported families). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio ControlNet: show the picker on the Create tab (workflow id is 'create', not 'txt2img') The ControlNet control gated on workflow === 'txt2img', but the Images workflow tab ids are create/transform/inpaint/extend/upscale/reference/edit -- there is no 'txt2img'. So the picker never rendered even with a ControlNet-capable model loaded. Gate on 'create' (the text-to-image tab) for both the picker and the request wiring. Found via a live Playwright capture of the running Studio. * Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a stray "gguf" tag on the Hub but ship no .gguf files. The model search classified them as GGUF from the bare tag, so the picker rendered the GGUF variant expander, which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata (populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected. * Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline load when the id is unsloth-hosted or on-device (the backend infers the family + base repo and gates loads to unsloth/* or local paths), and show a clear message otherwise instead of silently ignoring the click. Curated and GGUF paths are unchanged. * Studio Images: keep curated safetensors models in Recommended after download The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet), so they vanished from the picker after the first load and could only be found by typing an exact search. The row already renders a downloaded badge, matching how GGUF Recommended rows stay visible when cached. Drop the exclusion so the curated safetensors always list. * Studio diffusion LoRA: sanitize dots out of adapter aliases The LoRA alias is used as the diffusers PEFT adapter name, and PEFT rejects names containing "." (module name can't contain "."). sanitize_alias kept dots, so a LoRA whose filename carries a version tag (e.g. Qwen-Image-2512-Lightning-8steps-V1.0-bf16) failed to apply with a 400. Replace dots too; the alias stays a valid native <lora:NAME:w> filename stem. Adds regression coverage for internal dots. * Studio Images: clarify the GGUF transformer-quant Advanced control Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode" with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/ FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF, falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only. * Studio Images: list on-device unsloth diffusion models in the picker The Images picker's On Device tab hid every non-GGUF cached repo whenever a task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit and FP8 safetensors) never showed up there. List cached repos that pass the task gate, limited under a filter to unsloth-hosted ones so base repos (which fail the diffusion load trust gate) don't appear only to dead-end on click. Chat behavior is unchanged: the task gate still drops image repos there. * Studio: hide single-file image checkpoints from the chat model picker The chat picker treats a cached repo as an image model, and hides it, only when it ships a diffusers model_index.json. Single-file, ComfyUI, and ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models. Fall back to resolving the repo id against the known diffusion families, the same resolver the Images backend loads from, so these checkpoints are tagged text-to-image and stay in the Images picker only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: add the FLUX.2-dev model family Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline) resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline + Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated, reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror. text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from klein's turbo defaults. Verified live: GGUF load resolves the family + gated base repo and generates a real 1024x1024 image on GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: clearer error for an unsupported diffusion model When a repo id resolves to no diffusion family the load raised 'Could not infer a diffusion family... Pass family_override (z-image)', which points at an unrelated family and doesn't say what is supported. Replace it with a message that lists the supported families (from a new supported_family_names helper) and notes that video models and image models whose diffusers transformer has no single-file loader are not supported. Applies to both the diffusers and native sd.cpp load paths. Also refreshes two stale family-registry comments that still called FLUX.2-dev omitted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add SDXL diffusion family (U-Net pipeline support) SDXL is the first U-Net family in the diffusion backend: its denoiser is pipe.unet (UNet2DConditionModel), not a DiT pipe.transformer, and a single-file .safetensors is the whole pipeline rather than a transformer-only file. The backend previously assumed a DiT transformer everywhere, so add the two hooks a U-Net family needs and register SDXL. DiffusionFamily gains denoiser_attr ("transformer" for DiT, "unet" for SDXL) and single_file_is_pipeline (SDXL loads a single file via pipeline_class.from_single_file with the base repo as config, instead of transformer_class.from_single_file plus a companion assembly). _align_vae_dtype now reads the denoiser generically so img2img and inpaint keep the VAE and U-Net dtypes aligned. The non-GGUF trust gate is extended with a short, exact-match, safetensors-only allowlist of official base repos (the SDXL base/refiner and sdxl-turbo), because SDXL ships only as a full pipeline and has no unsloth-hosted GGUF. Local paths stay trusted as before; a random repo, even one that detects as SDXL, is still rejected. The image-conditioned and ControlNet workflows are the standard SDXL pipelines, built around the resident modules via from_pipe like every other family, so SDXL gets txt2img, img2img, inpaint, outpaint, upscale, LoRA and ControlNet. There is no native sd.cpp mapping yet, so the no-GPU route falls back to diffusers. Frontend catalog gains SDXL Base 1.0 and SDXL Turbo entries with SDXL step/guidance defaults (Turbo: few steps, no CFG; base: ~30 steps, real CFG). Tests: new test_diffusion_sdxl.py (family shape, detection, trust allowlist, model kind, U-Net VAE-dtype alignment, LoRA gate) plus loader-branch tests in test_diffusion_backend.py (pipeline-kind from_pretrained, single-file whole-pipeline from_single_file, allowlist accept/reject). Verified live on GPU: sdxl-turbo loads both as a pipeline and as a single file and generates coherent txt2img + img2img. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images: LoRA free-text Hugging Face entry + recipe round-trip The backend has always accepted a bare Hugging Face repo id (owner/name, or owner/name:weight-file.safetensors) as a LoRA, downloading and applying it. But the picker only rendered when the curated catalog had entries, and the catalog is empty, so there was no UI path to apply any LoRA. Show the LoRA section whenever the loaded model supports LoRA, and replace the curated-only dropdown with a text input: type a Hub repo id, or pick a discovered adapter from a datalist of suggestions when the catalog is populated. Also restore LoRAs when loading a recipe. restoreSettings now parses the recipe's "id:weight" strings (splitting on the last colon, since the id itself may contain one for a specific weight file) back into the selection, so replaying a saved image reproduces its adapters. The generate payload trims hand-typed ids and drops empty / zero-weight rows, and a model swap clears the selection (a LoRA is family-specific) without discarding a free-text pick that is not in the curated list. * Add diffusion LoRA training (SDXL text-to-image) First diffusion training path in Studio: train a LoRA on the SDXL U-Net from an image + caption dataset and export it as a diffusers .safetensors that the existing diffusion LoRA loader (and any diffusers pipeline) can load. core/training/diffusion_lora_trainer.py: - DiffusionLoraConfig with validation/defaults (rank, alpha, targets, lr, steps, grad accumulation, resolution, min-SNR gamma, gradient checkpointing, lr scheduler, seed, mixed precision). - discover_image_caption_pairs: captions from metadata.jsonl / captions.jsonl, per-image .txt/.caption sidecars, or a dreambooth instance_prompt fallback (pure, unit-tested). - run_diffusion_lora_training: the loop -- freeze base, PEFT-wrap the U-Net attention projections, VAE-encode (fp32 VAE to avoid the SDXL fp16 overflow), sample noise + timesteps, predict, MSE loss with optional min-SNR weighting (epsilon / v-prediction), AdamW + get_scheduler + grad accumulation + grad clipping, then export via save_lora_weights. Emits worker-protocol events (model_load_*, progress, complete) and polls should_stop for a clean stop with a partial save. - run_diffusion_training_process: mp.Queue subprocess adapter (event_queue / stop_queue), so the training worker can spawn it; plus a CLI entry point. Only SDXL (U-Net) is trained here; DiT families and the Studio UI form + route wiring are follow-ups. The trainer is decoupled and worker-ready. Tests: test_diffusion_lora_trainer.py covers caption discovery (metadata / sidecar / instance prompt / skip-uncaptioned / errors), config normalisation + validation, the SDXL add-time-ids, and the dict->config adapter. Verified live on GPU: a 60-step SDXL LoRA run lowers the loss, exports a ~45 MB adapter, and loading it back shifts generation from baseline (mean abs pixel diff ~55/255). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * diffusion trainer: emit learning_rate in progress events (Studio pump compatibility) The Studio training pump reads 'learning_rate' from progress events; the diffusion trainer emitted 'lr'. Rename the field (and the CLI reader) so the trainer's events are directly consumable by the existing training status/SSE machinery when it is wired into the worker, without a translation shim. * Remove stray async task scratch outputs committed by mistake * ControlNet: reject filesystem-like ids and do not cache a model past an unload race Two review findings on the ControlNet path: - resolve_controlnet's bare-repo fallback accepted any id with a slash, so a path-shaped id (/tmp/x, ../x) reached from_pretrained as a local directory. Restrict the fallback to a strict owner/name HF repo id shape. - _controlnet_pipe now re-checks the cancel event after the blocking from_pretrained: an unload that raced the download had already cleared the caches, so caching the late module would pin it past the unload. * Pipeline prefetch: fetch only the default torch weights A full-pipeline prefetch kept every repo file outside assets/, so an official repo that ships multiple formats (SDXL Base: fp16 variants, ONNX, OpenVINO, Flax, a top-level single-file twin) downloaded tens of GB from_pretrained never loads. Skip non-torch exports and dtype-variant twins in _pipeline_file_downloaded, and drop a component .bin when the same directory carries a picked safetensors weight (diffusers' own preference). * Diffusion LoRA training: fall back to fp16 when CUDA lacks bf16 The default mixed_precision=bf16 hard-fails on pre-Ampere GPUs (T4 / V100 / RTX 20xx) which have no bf16 compute; check torch.cuda.is_bf16_supported() and drop to fp16 there. * Diffusion LoRA training: harden config handling, cancellation, SDXL conditioning, and safety Addresses review findings on the SDXL LoRA trainer: - Gate the base model with the same trust check as inference (unsloth/*, allowlisted official bases, or a local path) before from_pretrained, so an untrusted remote repo is never fetched or deserialised. - Check the stop signal before the (slow) model load, not only between steps, so a cancel during download is honoured; a stop may carry save=False to cancel without leaving a partial adapter. - Per-sample SDXL add_time_ids from the actual crop (original size + crop offset, with the offset mirrored on horizontal flip) instead of a fixed uncropped-square tensor. - Apply EXIF orientation before resize/crop so rotated photos train upright. - Skip gradient clipping when max_grad_norm <= 0 (the Studio 'disable' value) instead of scaling every gradient to zero. - Coerce Studio config strings/blanks: learning_rate string to float, blank hf_token to anonymous, gradient_checkpointing 'none'/'true'/'unsloth' to bool; reject a zero/negative lora_alpha or learning_rate. - Alias the generic Studio training payload keys (model_name/max_steps/batch_size/lora_r/ lr_scheduler_type/random_seed) onto the diffusion field names. - Mirror the trained adapter into loras/diffusion so the Images LoRA picker discovers it. - Report worker exceptions in both message and error keys so the failure is not lost. Adds regression tests for the config coercion/validation and aliasing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ControlNet: address review findings on the diffusers path - resolve_controlnet enforces catalog family compatibility so a direct API call cannot load a ControlNet built for another family through the wrong pipeline. - Unknown ControlNet ids now surface as a 400 (call site maps FileNotFoundError to ValueError) instead of a generic 500. - strength 0 disables ControlNet entirely, so a no-op selection never pays the download / VRAM cost; the control image is decoded and validated BEFORE the ControlNet is resolved or built, so a malformed image fails fast for the same reason. - ControlNet loads use the base compute dtype (state.dtype is a display string, not a torch.dtype, so it silently fell back to float32) and honor the base offload policy via group offloading instead of forcing the module resident. - Empty/malformed HF token coerced to anonymous access. - Flux Union ControlNet control_mode mapped from the selected control type. - resolve_controlnet drops the unused hf_token/cancel_event params. - ControlNetSpec validates guidance_start <= guidance_end (clean 422). - Images UI ControlNet Select shows its placeholder when nothing is selected. Adds regression tests for family enforcement and the union control-mode map. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA: harden resolution, native tag precedence, and diffusers teardown Address review findings on the LoRA path: - resolve_one: normalise a blank/whitespace hf_token to None (anonymous access) and reject a client-supplied weight file with traversal / absolute path. - resolve_specs: convert FileNotFoundError from an unknown/stale id to ValueError so the route returns 400 instead of a generic 500. - _scan_local: disambiguate local adapters that share a stem (foo.safetensors vs foo.gguf) so each is uniquely addressable. - inject_prompt_tags: the backend-validated weight now wins over a user-typed <lora:ALIAS:...> for a selected adapter; unselected user tags are left alone. - diffusers _apply_loras: reject a .gguf adapter with a clear error before touching the pipe (diffusers loads safetensors only). - _unload_locked: drop the explicit unload_lora_weights() on teardown; the pipe is dropped wholesale (freeing adapters), so the previous call could race an in-flight denoise on the same pipe. - Images page: use a stable LoRA key and clear the selection (not just the options) when the catalog refresh fails. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion: guard trust check against OSError and validate conditioning inputs - _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid characters (or a bare owner/name id) can't raise OSError; treat any failure as not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request still raises the clear FileNotFoundError for a genuinely missing local pick. - generate(): reject mask_image / upscale / reference_images supplied without an input image, and reject reference_images on a family that does not support reference conditioning, instead of silently degrading to txt2img / img2img. * SDXL: reject GGUF up front, skip unused base weights, drop refiner, and harden helpers Addresses review findings on the SDXL family: - Reject a GGUF load for single_file_is_pipeline families (SDXL) in validate_load_request, before the route evicts the current model; SDXL has no transformer-only GGUF variant. - Skip base-repo weight files when a whole-pipeline single file is loaded: from_single_file (config=base) needs only the base config/tokenizer/scheduler, so a local .safetensors no longer triggers a multi-GB base download. - Remove the SDXL refiner from the non-GGUF trust allowlist: it is an img2img-only pipeline but this backend loads every sdxl repo as the base txt2img pipeline. - Normalize a blank/whitespace hf_token to None once in load_pipeline so every load branch degrades to anonymous instead of erroring on a malformed token. - Read the denoiser dtype from a parameter (compile-wrapped modules may lack .dtype) and access state.family.denoiser_attr directly. Adds/updates regression tests for the trust allowlist, GGUF rejection, and base-config filter. * Images: preserve restored LoRAs through model load and never send hidden LoRAs - The LoRA effect cleared the selection on every load->capable transition, which wiped adapters restored from a gallery recipe before the model finished loading. Track the previously-loaded family in a ref and clear only on a real family swap; keep the selection on the initial load and on unload. - Gate the generate payload's loras on loraCapable so a restored selection that is hidden (loaded model does not support LoRA) is never sent to the backend. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Merge diffusion-sdxl into diffusion-lora-ux; keep options-only LoRA catch The catalog-refresh .catch from the lower branch clears the selected adapters too, which is right for its catalog-only picker but wrong here: this picker holds free-text HF repo ids that are valid without being in the catalog, so a transient refresh failure must not wipe them. Family swaps still clear the selection and hidden LoRAs are never sent. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align the VAE to the denoiser's first FLOATING dtype, not its first parameter A GGUF-quantized transformer's leading parameters are packed uint8 storage, so reading next(parameters()).dtype handed nn.Module.to() an integer dtype and every image-conditioned generation on a GGUF model (Qwen-Image-Edit) failed with a 500. Probe the parameters for the first floating dtype, treat an all-integer module as a no-op, and also catch TypeError so an unexpected dtype can never break generation. Regression test included. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Count LR scheduler warmup/decay in optimizer steps, not micro-steps lr_sched.step() runs once per outer optimizer step (after the gradient accumulation inner loop), for train_steps total. The scheduler was configured with num_warmup_steps and num_training_steps multiplied by gradient_accumulation_steps, so with accumulation > 1 a warmup or non-constant schedule stretched past the run and never reached the intended decay. Count both in optimizer steps. * Address Codex review findings on the image-workflows PR Keep diffusion.py importable without torch: the compile/arch patch modules import torch at module level, so import them lazily at their load/unload call sites instead of at module load. This restores the torchless contract so get_diffusion_backend() works on a CPU/native sd.cpp install. Match family reject keywords and aliases as whole path/name segments, not raw substrings, so an unrelated word like edited, edition, or kontextual no longer misroutes or hides a valid base image model, while supported edit families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same segment matching in the picker task filter. Route FLUX.2-dev native guidance through --guidance like the other FLUX families rather than --cfg-scale. Reject native upscale requests that have no input image. Read image header dimensions and reject over-limit inputs before decoding pixels, so a crafted small-payload image cannot spike memory. Reject an upscale that would shrink the source below its input size. Validate the model_kind against the filename extension before the GPU handoff. Estimate a local diffusers pipeline's size from its on-disk weights so auto memory planning does not skip offload and OOM. Report workflows: [txt2img] from the native backend status so the Create tab stays enabled for a loaded native model. Clamp the outpaint canvas to the backend's 4096px decode limit. Adds regression tests for segment matching and kind/extension validation. * Harden diffusion LoRA handling on the diffusers and native paths Reject LoRA on a torch.compile'd diffusers transformer (Speed=default/max): diffusers requires the adapter loaded before compilation, so applying one to the already-compiled module fails with adapter-key mismatches. The status gate now hides the picker and generate raises a clear message instead. Convert a cancelled Hub LoRA download (RuntimeError Cancelled) to the diffusion cancellation sentinel in resolve_specs, so an unload/superseding load during resolution maps to a 409 instead of a generic server error. Drop weight-0 LoRA rows before the native support gate so a request carrying only disabled adapters stays a no-op on families where native LoRA is unsupported, matching the diffusers path. Reject duplicate LoRA ids in the request model: both apply paths suffix colliding names, so a repeated id would stack the same adapter past its per-adapter weight bound. Strip all user-typed <lora:...> prompt tags on the native path (only the selected adapters are materialized in the managed lora-model-dir, so an unselected tag can never resolve), and restore saved LoRA selections from a gallery recipe so restore reproduces a LoRA image. * Harden ControlNet resolve, gallery metadata, and the control-type picker Check cancellation immediately after a ControlNet from_pretrained and before any device placement, so an unload/eviction that raced the download does not allocate several GB onto the GPU after the load was already cleared. Require a loadable weight or shard index (not just config.json) before a local ControlNet folder is advertised, so an interrupted copy is hidden instead of failing deep in from_pretrained as a generic 500. Do not record a strength-0 ControlNet in the gallery recipe: it is treated as disabled and skipped, so the image is unconditioned and the metadata must not claim a ControlNet was applied. Build the control-type picker from the selected ControlNet's advertised control_types instead of a hardcoded passthrough/canny pair, so a union model with a precomputed depth or pose map sends the correct control_mode. * Address further Codex findings on the image-workflows PR - Persist the actual output image size in the gallery recipe instead of the request sliders: Transform/Inpaint/Edit derive the size from the uploaded image, Extend grows the canvas, and Upscale resizes it, so the sliders recorded (and later restored) the wrong dimensions for those workflows. - Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file name) in validate_load_request, so the unloadable pick fails before chat is evicted rather than deep in from_pretrained. - Only publish an image-conditioned from_pipe wrapper to the shared aux cache when the load is still current: from_pipe runs under the generate lock but not the state lock, so an unload racing its construction could otherwise cache a wrapper over torn-down modules that a later load would reuse. - Verify the Windows CUDA runtime archive checksum before extracting it, like the main sd-cli archive, so a corrupt or tampered runtime is rejected rather than extracted next to the binary. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse non-SDXL base models at diffusion training start The trainer only supports the SDXL U-Net, but a FLUX / Qwen-Image / Z-Image repo or a GGUF filename passed as base_model was accepted and then failed minutes later inside StableDiffusionXLPipeline.from_pretrained with an unrelated-looking error. Add a name-based guard in normalized() so known DiT-family names and .gguf checkpoints are rejected up front, which the API start route surfaces as an immediate 400 with a message that says exactly which bases are trainable. Unrecognisable names still pass through so custom local SDXL checkpoints keep working. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
|
Daniel Han
|
e60c4a5110
|
Images: LoRA free-text Hugging Face entry + recipe round-trip (#6789)
* Studio diffusion: cross-platform device policy, fp16 guard, lock split, validate-before-evict Phase 1 of porting the richer diffusion stack onto the image-generation backend. - Add a compartmentalized device/dtype policy module (diffusion_device.py) resolving CUDA/ROCm/XPU/MPS/CPU with capability flags. Keeps the NVIDIA capability-based bf16 choice; ROCm and XPU are isolated; MPS uses bf16 or fp32, never a silent fp16 that renders a black image. - Add a per-family fp16_incompatible flag (Z-Image) and promote a resolved float16 to float32 for those families so they do not produce black images. - Split the backend locks: a generation holds only _generate_lock, so status, unload, and a new load are never blocked by a long denoise. Add per-generation cancellation via callback_on_step_end so an eviction or a superseding load preempts a running generation; a replacement load waits for it to stop before allocating, so two pipelines never sit in VRAM at once. - Validate a load request before the GPU handoff so an unloadable pick never evicts a working chat model, and reject missing local paths up front. - Add CPU-only tests for the device policy, dtype guard, lock split and cancellation, and validate-before-evict, plus a GPU benchmark/regression script (scripts/diffusion_bench.py) measuring latency, peak VRAM, and PSNR against a saved reference. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2A): measured-budget memory planner + offload/VAE policy Add a lean, backend-agnostic memory policy that picks a CPU-offload policy and VAE tiling/slicing from measured free device memory vs the model's estimated resident footprint, then applies it to the built pipeline. auto stays resident when the model fits (byte-identical to the prior resident path), and falls to whole-module offload when tight; fast/balanced/low_vram are explicit overrides. Sequential submodule offload is unreliable for GGUF transformers on diffusers 0.38, so it falls back to whole-module offload and status reports the policy actually engaged. Verified on Z-Image-Turbo Q4_K_M (B200): auto reproduces the resident image with no VRAM/latency regression (PSNR inf); balanced/low_vram cut generation peak VRAM 47.9% (15951 -> 8318 MB) with byte-identical output, at the expected latency cost. 73 prior + 35 new CPU tests pass. * Studio diffusion (Phase 2D): streamed block-level offload + functional VAE tiling Add a streamed 'group' offload tier (diffusers apply_group_offloading, block_level, use_stream) that keeps the transformer flowing through the GPU a few blocks at a time while the text encoder / VAE stay resident, and fix VAE tiling to drive the VAE submodule (pipelines like Z-Image expose enable_tiling on pipe.vae, not the pipeline). apply_memory_plan now returns the (policy, tiling) actually engaged so status never overstates either, and group falls back to whole-module offload when the transformer can't be streamed. Measured on Z-Image (B200), all lossless (PSNR inf vs resident): balanced/group cuts generation peak VRAM 32% (15951 -> 10840 MB) at near-resident speed (2.07 -> 2.99s); low_vram/model cuts it 48% (-> 8318 MB) but is slower (7.99s). Mode names now match that tradeoff: balanced = stream the transformer, low_vram = offload every component. auto picks group when the companions fit resident, else model. 112 CPU tests pass. * Studio diffusion (Phase 5): image quality-vs-quant accuracy harness Add scripts/diffusion_quality.py, the accuracy analogue of the KLD workflow: hold prompt + seed fixed, render a grid with a reference quant (default BF16), then render each candidate quant and measure drift from the reference. Records mean PSNR + SSIM (pure-numpy, no skimage/scipy) and optional CLIP text-alignment + image-similarity (transformers, --clip), plus file size, latency, and peak VRAM, then prints a quality-vs-cost table and recommends the smallest quant within a quality budget. --selftest validates the metrics on synthetic images with no GPU or model. Verified on Z-Image (B200): the table degrades monotonically with quant size (Q8 -> Q4 -> Q2: PSNR 21.7 -> 15.5, SSIM 0.82 -> 0.61), while CLIP-text stays flat (~0.34) -- quantization erodes fine detail far more than prompt adherence. * Studio diffusion (Phase 3): opt-in speed layer (channels_last / compile / TF32) Add a speed_mode knob (off by default, so the render path stays bit-identical): default applies channels_last VAE + regional torch.compile of the denoiser's repeated block where eligible; max also enables TF32 matmul and fused QKV. Regional compile is gated off for the GGUF transformer (dequantises per-op) and for families flagged not compile-friendly (a new supports_torch_compile flag, False for Z-Image), so it activates automatically only once a non-GGUF bf16 transformer is loaded. Speed optims run before placement/offload, per the diffusers composition order. status now reports speed_mode + the optims actually engaged. Verified on Z-Image (B200): default -> ['channels_last'], max -> ['channels_last', 'tf32'], compile correctly skipped for GGUF; generation works in every mode. 121 CPU tests pass. * Studio diffusion (Phase 2B): opt-in fp8 text-encoder layerwise casting Add a text_encoder_fp8 knob that casts the companion text encoder(s) to fp8 (e4m3) storage via diffusers apply_layerwise_casting, upcasting per layer to the bf16 compute dtype while normalisations and embeddings stay full precision. Applied before placement, gated to CUDA + bf16, best-effort (a failure leaves the encoder dense). status reports which encoders were cast. Verified on Z-Image (B200, balanced/group mode where the encoder stays resident): generation peak VRAM dropped 37% (10840 -> 6791 MB, below the lowest-VRAM offload) at near-resident speed. It is a memory-vs-quality tradeoff, not free -- ~20 dB PSNR vs the bf16 encoder, a larger shift than one transformer quant step -- so it is off by default and documented as such, with the Phase 5 harness to size the cost. 127 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2C): NVFP4 text-encoder quant (+ generalise fp8 knob) Generalise the text-encoder precision knob from a fp8 bool to text_encoder_quant (fp8 | nvfp4). nvfp4 quantises the companion text encoder to 4-bit via torchao NVFP4 weight-only (two-level microscaling) on Blackwell's FP4 tensor cores; fp8 stays the broader-hardware path (cc>=8.9). Both are gated, best-effort, and run before placement; status reports the mode actually engaged. This is the lean realisation of GGUF-native text-encoder quant: 4-bit on the encoder without the 3045-line port. Verified on Z-Image (B200, balanced/group where the encoder stays resident), vs the bf16 encoder: nvfp4 cut generation peak VRAM 48% (10840 -> 5593 MB, the lowest TE option, below whole-model offload) at near-fp8 quality (16.4 vs 17.1 dB PSNR), and both quants ran faster than bf16. A memory-vs-quality tradeoff (off by default); size it per model with the Phase 5 quality harness. diffusion_bench gains --text-encoder-quant. 129 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): native stable-diffusion.cpp engine for CPU/Mac Adds the CPU / Apple-Silicon tier of the two-engine strategy, mirroring the chat backend's llama.cpp shell-out. Diffusers stays the default on CUDA / ROCm / XPU; this covers the hardware diffusers serves poorly, consuming the same split GGUF assets Studio already curates. - sd_cpp_args.py: pure sd-cli command builder. Maps the family to its text-encoder flag (Z-Image Qwen3 to --llm, Qwen-Image to --qwen2vl, FLUX.1 CLIP-L + T5), and the diffusers memory policy (none/group/model/sequential) to sd.cpp's offload flags (--offload-to-cpu / --clip-on-cpu / --vae-on-cpu / --vae-tiling / --diffusion-fa), so one user knob drives both engines. - sd_cpp_engine.py: SdCppEngine over a located sd-cli. find_sd_cpp_binary() with the same precedence as the llama finder (env override, then the Studio install root, then in-tree, then PATH), an is_available/version probe, and a one-shot subprocess generate that streams progress and returns the PNG. runtime_env() prepends the binary's directory to the platform library path so a prebuilt's bundled libstable-diffusion.so resolves. select_diffusion_engine() is the pure routing decision (GPU backends to diffusers, CPU/MPS to native when present). - install_sd_cpp_prebuilt.py: resolve + download the per-host prebuilt (macOS-arm64/Metal, Linux x86_64 CPU, Vulkan/ROCm/Windows variants) into the Studio install root. resolve_release_asset() is a pure, unit-tested host-to-asset matrix. - scripts/sd_cpp_smoke.py: end-to-end native generation harness. Tests (CPU-only, subprocess/filesystem stubbed): 49 new across args, engine, routing, runtime env, and the installer resolver. Full diffusion suite 166 passing. Verified on a B200 box: built sd-cli (CUDA) and the prebuilt (CPU) both generate Z-Image-Turbo Q4_K end to end through SdCppEngine: balanced (group offload, 5.0s gen), low_vram (full CPU offload + VAE tiling, 13.4s), and the dynamically-linked CPU prebuilt (50.4s on CPU), all producing coherent images. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 6): img2img / inpaint / edit / LoRA / upscale on the native engine Builds on Phase 4's native stable-diffusion.cpp engine, extending it from text-to-image to the wider feature surface, since sd.cpp supports all of these through the binary already. Pure command-builder additions plus one engine method, so the txt2img path is unchanged. - sd_cpp_args.py: SdCppGenParams gains image-conditioning fields. init_img + strength make a run img2img, adding mask makes it inpaint, ref_images drives FLUX-Kontext / Qwen-Image-Edit style editing (repeated --ref-image), and lora_dir + the <lora:name:weight> prompt syntax select LoRAs. New SdCppUpscaleParams + build_sd_cpp_upscale_command for the ESRGAN upscale run mode (input image + esrgan model, no prompt / text encoders). - sd_cpp_engine.py: the subprocess runner is factored into a shared _run() so generate() (now carrying the conditioning flags) and a new upscale() reuse the same streaming / error / output-check path. - scripts/sd_cpp_smoke.py: --task {txt2img,img2img,upscale} with --init-img / --strength / --upscale-model / --upscale-repeats. Tests: 10 new across the img2img / inpaint / edit / LoRA flag construction, the upscale builder and its validation, and the engine's img2img + upscale paths. Full diffusion suite 176 passing. Verified on a B200 box through SdCppEngine: img2img (Z-Image-Turbo Q4_K, the init image conditioned at strength 0.6, 4.8s) and ESRGAN upscale (512x512 -> 2048x2048 via RealESRGAN_x4plus_anime_6B, 2.7s), both producing coherent images. Video and the diffusers-path feature wiring are deferred. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): accuracy-preserving speed pass Re-review of the diffusion stack (#6675/#6679/#6680) surfaced one real accuracy bug and a dead-on-arrival speed path; this fixes both and adds the lossless / near-lossless wins, all measured on a B200. Correctness: - TF32 global-state leak (fix). speed_mode=max flipped torch.backends.*.allow_tf32 process-wide and never restored them, so a later `off` load silently inherited TF32 and was no longer bit-identical. Added snapshot_backend_flags / restore_backend_flags (TF32 + cudnn.benchmark), captured before the speed layer runs and restored on unload. Verified: load max -> unload -> load off is now byte-identical (PSNR inf) to a fresh off. - sd-cli timeout could hang forever. _run() blocked in `for line in stdout` and only checked the timeout after EOF, so a child stuck in model load / GPU init with no output ignored the timeout. Drained stdout on a reader thread with a wall-clock deadline. Added a silent-hang regression test. Speed (diffusers path), near-lossless, opt-in tiers: - Regional torch.compile now runs on the GGUF transformer. The is_gguf gate (and Z-Image's supports_torch_compile=False) were stale: compile_repeated_blocks compiles and runs ~2.2x faster on the GGUF Z-Image transformer on torch 2.9.1 / diffusers 0.38 (the per-op dequant stays eager, the rest of the block compiles). Measured: off 1.80s -> default 0.82s/gen (+54.7%), PSNR 37.7 dB vs eager -- far above the Q4 quant noise floor (~21 dB), so it does not move output quality. Gate relaxed; default tier delivers it. - cudnn.benchmark added to the default tier (autotunes the fixed-shape VAE convs). - torch.inference_mode() around the pipeline call (lossless, strictly faster than the no_grad diffusers uses internally). Memory path: - VAE tiling (not bit-identical >1MP) restricted to the model/sequential/CPU tiers; the balanced (group) tier keeps exact slicing only, so it is now bit-identical to the resident image (verified PSNR inf) and slightly faster. - Group offload adds non_blocking + record_stream on the CUDA stream path to overlap each block's H2D copy with compute (lossless; gated on the installed diffusers signature so older versions still work). Native (sd.cpp) path: - native_speed_flags: a first-class speed knob (default -> --diffusion-fa, a near-lossless CUDA win that was previously only added on offload tiers; max also -> --diffusion-conv-direct). conv-direct stays opt-in: measured +45% on CUDA, so it is never auto-on. Engine generate() merges it, de-duped against offload flags. Default profile: a GGUF model with no explicit speed_mode now resolves to the `default` profile (resolve_speed_mode), since compile's perturbation sits below the quantisation noise floor and so does not reduce quality versus the dense reference; out of the box a GGUF Z-Image generation drops from 1.80s to 0.81s. Dense models stay `off` / bit-identical, and an explicit speed_mode -- including "off" -- is always honored, so the byte-identical path remains one flag away and is the regression reference. Tooling: scripts/compile_probe.py (eager vs compiled GGUF probe), scripts/ perf_verify.py (the B200 verification above), and diffusion_bench.py gains --speed-mode so the speed tiers are benchmarkable. Tests: 183 passing (was 166); new coverage for the backend-flag snapshot/restore, GGUF compile eligibility, the balanced tiling/slicing split, native_speed_flags + the engine de-dup, and the sd-cli silent-hang timeout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): max tier uses max-autotune-no-cudagraphs + engine/lever benchmarks The opt-in `max` speed tier now compiles the repeated block with mode=max-autotune-no-cudagraphs (dynamic=False) instead of the default mode: Triton autotuning for GEMM/conv-heavier models, gated to the tier where a longer cold compile is acceptable. CUDA-graph modes (reduce-overhead / max-autotune) are deliberately avoided -- both crash on the regionally-compiled block (its static output buffer is overwritten across denoise steps), measured. Adds two reproducible benchmarks used to validate the optimization research: - scripts/compare_engines.py: PyTorch (diffusers GGUF) vs native sd.cpp head-to-head. - scripts/leverage_probe.py: coordinate_descent_tuning + FirstBlockCache probes. Measured on B200 (Z-Image Q4_K_M, 1024px, 8 steps): default compile 0.80s/gen; coordinate_descent_tuning 0.79s (within noise, already covered by max-autotune); FirstBlockCache does not run on Z-Image (diffusers 0.38 block-detection / Dynamo). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source) Add an opt-in transformer_quant mode that loads the dense bf16 transformer and torchao-quantises it onto the low-precision tensor cores, instead of the GGUF transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's 0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor. GGUF+compile stays the low-memory default and the fallback. The mode is gated on CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB); any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe (Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips the tiny projections that crash int8's torch._int_mm. New module mirrors diffusion_precision.py; quant runs before compile before placement. 184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py is the standalone torchao lever probe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): consumer-GPU tuning - lock fp8 fast accumulate, prefer fp8 over mxfp8, reject 2:4 sparsity Consumer Blackwell halves tensor-core throughput on FP32 accumulate (fp8 419 vs 838 TFLOPS with FP16 accumulate; bf16 209), so: - fp8 config locks use_fast_accum=True (Float8MMConfig). torchao already defaults it on; pinning it guards consumer cards against a default change. On B200 it is identical speed and slightly better quality (LPIPS 0.050 vs 0.091). - the Blackwell auto ladder prefers fp8 over mxfp8 (measured faster + more accurate). 2:4 semi-structured sparsity evaluated and rejected (scripts/sparse_accum_probe.py): 2:4 magnitude-prune + fp8 gives LPIPS 0.858 (broken image) with no fine-tune, the cuSPARSELt kernel errors on torch 2.9, and it does not compose with torch.compile (our main ~2x). Documented as a dead end, not shipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add fp8 fast-accum overflow verification probe scripts/fp8_overflow_check.py hooks every quantised linear during a real Z-Image generation and reports max-abs + non-finite counts for use_fast_accum True vs False. Confirms fast accumulation is an accumulation-precision knob, not an overflow one: across 276 linears, including Z-Image's ~1.0e6 activation peaks (which overflow FP16), 0 non-finite elements and identical max-abs for both modes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40; GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it. Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s), no overflow, quality below the quant noise floor. So the default leans to accuracy on data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force) lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off). 187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override threading). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add NVFP4 probe documenting it is not yet a win on torch 2.9 scripts/nvfp4_probe.py measures NVFP4 via torchao on the real Z-Image transformer. Finding (B200, 1024px/8 steps): NVFP4 is a torchao feature and DOES run with use_triton_kernel=False (the default triton path needs the missing MSLK library), but only at bf16-compile rate (0.667s vs fp8 0.592s) -- it dequantises FP4->bf16 rather than using the FP4 tensor cores. The real FP4 speedup needs MSLK or torch>=2.11 + torchao's CUTLASS FP4 GEMM. The smoke probe (default triton=True) already keeps NVFP4 out of auto on this env, so auto correctly stays on fp8; NVFP4 activates automatically once fast. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): prefer fp8 over nvfp4 in Blackwell auto ladder Validated NVFP4 on torch 2.11 + torchao CUTLASS FP4 in an isolated env. The FP4 tensor-core GEMM is genuinely active there (a 16384^3 GEMM hits ~3826 TFLOPS, 2.52x bf16 and 1.37x fp8), but it only beats fp8 on very large GEMMs. At the diffusion transformer's shapes (hidden ~3072, MLP ~12288, M~4096) NVFP4 is both slower (0.81x fp8 end to end on Z-Image 1024px) and less accurate (LPIPS 0.166 vs fp8's 0.044). Reorder the Blackwell auto ladder to fp8 before nvfp4 so auto is correct even on a future MSLK-equipped box; nvfp4 stays an explicit opt-in. Add scripts/nvfp4_t211_probe.py (extension diagnostics + GEMM micro + end-to-end). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): pre-quantized transformer loading The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs 13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline (scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True) the quantized weights, so the dense bf16 never touches the GPU. Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same torchao config + min_features filter the runtime path uses, applied ahead of time. New core/inference/diffusion_prequant.py (resolve_prequant_source + load_prequantized_transformer, best-effort, lazy imports). diffusion.py _load_dense_quant_pipeline tries the pre-quant source first and falls back to the dense materialise+quantise path, then to GGUF, so the default is unchanged. DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for the resolver, the meta-init+assign loader, and the backend branch selection + fallbacks; GPU verification via scripts/verify_prequant_backend.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): attention-backend selection Add a selectable attention kernel via the diffusers set_attention_backend dispatcher. Attention is memory-bandwidth bound, so a better kernel is an end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV matmuls torchao never touches) and composes with torch.compile. auto picks the best exact backend for the device: cuDNN fused attention (_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default (below the compile/quant noise floor); native SDPA elsewhere and when speed=off (so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/ xformers/aiter are honored, and an unavailable kernel falls back to the default rather than failing the load. New core/inference/diffusion_attention.py (normalize + per-device select + apply, best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline; attention_backend threads through begin_load / load_pipeline / status like the other load knobs. New request field attention_backend + status field. Hermetic CPU tests for normalize / select policy / apply fallback, plus route threading + 422. Measured via scripts/perf_levers_probe.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the arch tier to put int8 first on a consumer/workstation GPU (detected by the existing _is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first. Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation (-> int8) and data-center Ada/Hopper/Blackwell (-> fp8). * Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across denoise steps a DiT's output settles, so once the first block's residual barely changes the remaining blocks are skipped and their cached output reused. diffusers ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the standalone apply_first_block_cache hook as a fallback). Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 -> 2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar. OFF by default and a per-load opt-in: the win scales with step count, so it is for many-step models (Flux / Qwen-Image) and pointless for few-step distilled models (e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction of the trajectory. It composes with regional compile only with fullgraph=False (the cache's per-step decision is a torch.compiler.disable graph break), which the speed layer now switches to automatically when a cache is engaged. Best-effort: a model whose block signature the hook does not recognise is caught and the load proceeds uncached. - new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache (enable_cache / apply_first_block_cache fallback; threshold auto-raised for a quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import). - diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph when a cache is engaged. - diffusion.py: apply_step_cache before compile; thread transformer_cache / transformer_cache_threshold through begin_load -> load_pipeline and report the engaged mode in status(). - models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and transformer_cache_threshold request fields, engaged mode in the status response. - hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold selection, and best-effort failure handling, plus route threading + validation. - scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM / LPIPS vs the compiled no-cache baseline). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears) The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with 'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2 *_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing min_features filter did not exclude them. Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed, guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers (M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and quantises these layers fine, so the exclusion is int8-only. Sequence embedders (context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note 'context_embedder' contains the substring 'text_embed', which is why the token is the specific 'time_text_embed', not 'text_embed'. Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s (5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already working) are unchanged. - diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes exclude_name_tokens; quantize_transformer passes it for int8 only. - hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps attention / FFN / sequence-embedder linears), while fp8 keeps them. - scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's Linear layers and derive the exclusion list. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears) The prequant-checkpoint builder applied the dense quant filter without the int8-only M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8 checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now produces a working checkpoint on every supported model, giving int8 (the consumer-preferred scheme) the same ~2x load-VRAM and download reduction fp8 already had. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the native stable-diffusion.cpp engine instead of diffusers, with diffusers as the guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM. - diffusion_engine_router: centralised engine selection (built on the existing select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason. - sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface backed by sd-cli, with lazy binary install, registry-driven asset fetch, step-progress parsing, and cancellation. - diffusion_families: per-family single-file VAE + text-encoder asset mapping. - sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled). - routes/inference + gpu_arbiter: drive the active engine via the router; the API now reports the active engine and any fallback reason. - tests for the backend, router, route selection, and cancellation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler Address review feedback on #6724: - engine router: unload the engine being deactivated on a switch, so the old model is not left resident-but-unreachable (the evictor only targets the active engine). - generate route: sd.cpp execution errors (nonzero exit / timeout / missing output) now map to 500, not 409 (which only means not-loaded / cancelled). - native batch: return per-image seeds and persist the actual seed for each image so every batch image is reproducible. - Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the stable-diffusion.cpp docs; other families keep sd-cli defaults. - honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload offload flags on the native load instead of hardcoding them off. - fail the load when the sd-cli binary is present but not runnable (version() now returns None on exec error / nonzero exit). - size estimate: only treat the transformer asset as a possible local path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in load_prequantized_transformer ends in torch.load(weights_only=False), which executes arbitrary code from the pickle. The transformer_prequant_path load-request field reached that unpickle for any local file an authenticated caller named, so a request could trigger remote code execution. Refuse the source.kind=='path' branch unless the operator sets UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted and unaffected. Document the requirement on the API field and add gate tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa - apply_attention_backend now restores the native default when no backend is requested or a kernel fails. diffusers keeps a process-wide active attention backend that set_attention_backend updates, and a fresh transformer's processors follow it, so a load that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile load pinned, breaking the bit-identical/off guarantee. - select_attention_backend drops flash3/flash4 up front when the CUDA capability is below Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit request on the wrong card set fine then crashed mid-generation; it now falls back to native. - Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a valid alias of native) is accepted instead of 422-rejected by Pydantic. - Drop the dead replace('-','_') normalization (no alias uses dashes/underscores). - perf_levers_probe.py output dir is now relative to the script, not a hardcoded path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF - apply_step_cache now engages only via the transformer's native enable_cache (the diffusers CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model now runs uncached per the best-effort contract. - GGUF transformers are quantized (the default Studio load path), so they now use the higher quantized FBCache threshold when the caller leaves it unset, instead of the dense default that could keep the cache from triggering. - fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so fullgraph=True failed warmup and silently measured an eager cached run); output dir is now relative to the script, not a hardcoded path. * Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder _is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate. Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0). * Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just TypeError on older torchao), which would break the mxfp8 config helper instead of falling back to the default. Catch both so the fallback is robust. quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds CUDA memory during the per-row VRAM probe; output dir relative to the script. * Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load - snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the whole snapshot. restore_backend_flags restores each flag independently so one failure can't leave the others leaked process-wide. - load_pipeline restores the flags (and clears the GPU cache) when the build fails after apply_speed_optims mutated the process-wide flags but before _state captured them for unload to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and contaminated later off generations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs without producing output (or without closing stdout) would never reach proc.wait and the wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the PROCESS, so the main thread always enforces the timeout and kills a hung process (which closes the pipe and ends the reader). Add a test that times out even when stdout blocks, and make the no-binary test hermetic so a host-installed sd-cli can't leak in. * Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn The filter callback can be invoked without a module name, so fqn.lower() would raise AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens, so the linear is kept) instead of crashing the quantise pass. * Studio diffusion (Phase 16) review fixes: native engine robustness - sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64); a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds aliased to the same image. Random seeds stay 53-bit (JS-safe). - sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub fall back to anonymous instead of failing auth on a blank token. - sd_cpp_backend: a superseding load now cancels the in-flight generation, so the old sd-cli can no longer return/persist an image from the previous model. - diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a slow 10+ GB free / CUDA sync does not block engine selection. - diffusion_engine_router: probe sd-cli runnability (version()) before committing to native, so a present-but-unrunnable binary falls back to diffusers at selection. - diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a CPU-only install can still reach the native sd.cpp engine instead of failing load. - tests updated for the runnability probe + a not-runnable fallback case. * Studio diffusion (Phase 9) review fixes: prequant safety + validation - SECURITY: a request-supplied local pre-quant path is now unpickled only when it resolves inside an operator-configured ALLOWLIST of directories (UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in, once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on any path a load request named (arbitrary code execution). realpath() blocks symlink escapes; a bare on/off toggle is no longer a wildcard. - Validate the checkpoint's min_features against the runtime Linear filter, so a checkpoint that quantised a different layer set is rejected instead of silently loading a model that mismatches the dense path while reporting the same scheme. - Tolerant base_model_id compare (exact or same final path/repo segment), so a local path or fork of the canonical base is accepted instead of falling back to dense. - _has_meta_tensors uses any(chain(...)) (no intermediate lists). - prequant verify/probe scripts use repo-relative paths (+ env overrides), not the author's absolute /mnt paths. - tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts - diffusion_memory: when group offload is unavailable and the plan falls back to whole-module offload, enable VAE tiling (the group plan left it off, but the fallback is the low-VRAM path where the decode spike can OOM). Covers both the group and sequential fallback branches. - perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a balanced bit-identity regression actually fails the check instead of exiting 0. - compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and the load-progress poll has a 30 min deadline instead of looping forever on a hang. - test for the group->model fallback enabling VAE tiling. * Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path - diffusion: a torchao-quantized transformer is committed only compiled. A dense model resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF it replaced), so when transformer_quant engaged and speed resolved to off, promote to default (regional compile); warn loudly if compile still does not engage. - diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK); otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF. - nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path). - test asserts the eager-quant -> default-compile promotion. * Studio diffusion (Phase 10) review fixes: attention gating + probe isolation - diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation, so auto now stays on native SDPA there. - diffusion_attention: _active_attention_backend handles get_active_backend() returning an enum/None (not a tuple); the old unpack always raised and was swallowed, so the native-restore short-circuit never fired. - perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul so the inductor_flags variant doesn't leak into later compiled rows. - tests for the SM80 cuDNN gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening - install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket); extract through a per-member containment check (Zip-Slip guard); expanduser the --install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch the separately-published cudart runtime DLL archive so sd-cli.exe can start. - sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie. - tests: Zip-Slip rejection, normal extraction, studio-home discovery. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of the literal --output path. SdCppEngine.generate checked only the literal path, so a batch generation would exit 0 and then raise 'no image' (or return a stale file). generate now returns the literal path when present and otherwise falls back to the numbered siblings; single-image behavior is unchanged. Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected without error. * Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats Codex review on the native engine arg builder: - build_sd_cpp_command emitted --width/--height unconditionally, so an img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of the input. width/height are now Optional (None = unset): an image-conditioned run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives the size from the input image (set_width_and_height_if_unset); a plain txt2img run with unset dims keeps the prior 1024x1024 default; explicit dims are always honored. width/height are read only by the builder, so the type change is local. - build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...) that silently swallowed repeats=0 into sd-cli's default of one pass, turning an explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError and emits the flag for any explicit value != 1. Tests: img2img unset dims omit width/height (init_img and ref_images), explicit dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at the default. (Two pre-existing binary-discovery tests fail only because a real sd-cli is installed in this dev environment; unrelated to this change.) * Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc Codex review: the transformer_prequant_path field description still told operators to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the prior security fix made that variable a directory allowlist -- _allowed_prequant_roots deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator following the documented =1 would have every transformer_prequant_path request silently refused. The description now states it must name one or more allowlisted directories and that a bare on/off value is not accepted. Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does not say =1, and describes an allowlist/directory (guards against doc drift). * Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset Codex review on attention-backend selection: - Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first generation with no fallback. select_attention_backend now applies _cudnn_attention_supported() to an explicit cuDNN request too. - flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3 is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a (min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+. - apply_attention_backend's success path left diffusers' process-wide active backend pinned to the kernel it set; a later component whose processors are unconfigured (backend None) would inherit it. It now resets the global registry to native after a successful per-transformer set (the transformer keeps its own backend), best-effort. Also fixed _active_attention_backend: get_active_backend() returns a (name, fn) tuple, so the prior code stringified the tuple and never matched a name, defeating the native-restore short-circuit. Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on SM90; global registry reset after a successful set; _active_attention_backend reads the tuple return. * Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300 (Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So _is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they are treated as data-center class and keep the intended fp8-first behavior. Test: extends the datacenter parametrize with 'NVIDIA B300' and 'NVIDIA GH200 480GB' (now _is_consumer_gpu False). * Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder Codex review: the M=1 modulation/embedder exclusion was wired only into the dense runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint quantised the AdaLN modulation and conditioning-embedder linears, and loading it via transformer_prequant_path (the load path only loads already-quantised tensors, it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes for the runtime path. Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 -> the M=1 exclusion, every other scheme -> none) and use it in both the runtime quantiser and the builder, so a prequant artifact's quantised-layer set always matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion). Test: int8_exclude_name_tokens returns the exclusion for int8 and () for fp8/nvfp4/mxfp8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race Codex review on the native-engine routing: - The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat) unconditionally after engine selection. A native sd.cpp load on a pure-CPU host never touches the GPU, so that needlessly tore down the resident chat model. The handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it. - sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when _run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so the setting was unverifiable. status now derives them from state.offload_flags (still 'none' on CPU, where the flags are empty). - _run_load committed the new state without cancelling/waiting on a generation that started during the (slow) asset download, so a stale sd-cli run against the OLD model could finish afterward and persist an image from the previous model once the new load reported ready. The commit now signals the in-flight cancel and waits on _generate_lock before swapping _state (taken only at commit, so the download never serialises against generation), mirroring the diffusers load path. Tests: CPU native load skips the arbiter while a GPU native load takes it; status reports offload active when flags are set; _run_load cancels and waits for an in-flight generation before committing. * Studio diffusion (Phase 14) review round 2: align helper name with the stack Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the identical helper already present higher in the diffusion stack (Phase 16). The helper definition, the runtime quantiser call, and the offline builder are now byte-identical to that version, so the two branches no longer introduce a divergent name for the same single-source-of-truth and the stack merges without a conflict on this fix. No behavior change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: eager patches + torch.compile cache speed phase Adds the opt-in speed path for the GGUF diffusion transformer behind a selectable speed mode (default off, so output is unchanged until a profile is chosen): - diffusion_eager_patches.py: shared eager fast-paths (channels_last, attention/backend selection, fused norms and QKV) installed at load and rolled back on unload or failed load. - diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent torch.compile cache and the GGUF-transformer compile wiring. - diffusion_arch_patches.py: architecture-specific patches. - diffusion_patch_backend.py: shared install/restore plumbing. - diffusion_speed.py: speed-profile planning. Tests for each module plus the benchmarking and probe scripts used to measure speed, memory, and accuracy of the path. * Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI Backend: - Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file fp8 transformers, gated to the unsloth org plus a curated allowlist. - Image-conditioned workflows built with Pipeline.from_pipe so they reuse the loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint, outpaint, and a hires-fix upscale pass. - Instruction editing as its own family kind (Qwen-Image-Edit-2511, FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and multi-reference) plus klein inpaint. - Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the decoded image size and cap upscale output to avoid OOM on large inputs. - Fixes: from_pipe defaulting to a float32 recast that crashed torchao quantized transformers; image-conditioned calls forcing the slider size onto the input image. Native sd.cpp engine rejects image-conditioned and reference requests it cannot serve. Frontend: - Redesigned Images page with capability-gated workflow tabs (Create, Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor, client-side outpaint, and a multi-reference picker. - Advanced options moved to a right-docked panel mirroring Chat: closed by default, toggled by a single fixed top-bar button that stays in place. sd.cpp installer: pin the release, verify each download's sha256, add a download timeout, and make the source repo configurable for a future mirror. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: correct the Advanced panel comment (closed by default, fixed toggle) * Studio diffusion: LoRA adapters for the Images workflow Add community LoRA support across both diffusion backends, the single biggest step toward broad image-workflow coverage. Backend - New shared module core/inference/diffusion_lora.py: adapter discovery (local scan + curated catalog + owner/name[:file] Hub refs), download via hf_hub_download_with_xet_fallback, alias sanitization, native managed-dir materialization with collision-broken aliases, prompt-tag injection (deduped against user-typed tags), and a supports_lora gate. - Native sd-cli: resolve + materialize selected LoRAs into a per-run managed dir, inject <lora:ALIAS:w> tags, pass --lora-model-dir with --lora-apply-mode auto. The arg builder already emitted these flags. - Diffusers: non-fused load_lora_weights + set_adapters manager, tracked on the pipe so an unchanged selection is a no-op and a model swap resets; cleared on unload. Never fuses (breaks quantized transformers and blocks live weight tweaks). - Gated off where unsupported: torchao fp8/int8 dense, GGUF-via-diffusers, and native Qwen-Image (no LoRA name-conversion branch upstream). - Request contract: optional loras on DiffusionGenerateRequest; empty or omitted is identical to today. supports_lora surfaced in status; chosen LoRAs persisted in gallery recipe metadata. - New GET /api/models/diffusion-loras for the picker (family-filtered). Frontend - Repeatable multi-LoRA picker (adapter select + weight slider 0..2 + remove), gated by the loaded model's supports_lora and family, max 8. Tests - New test_diffusion_lora.py (14): helpers, request validation, native tag/dir wiring, diffusers set_adapters manager, supports_lora matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: ControlNet for the Images workflow (diffusers) Add ControlNet conditioning, the #2 most-used diffusion workflow after LoRA, on the diffusers backend for the families with ControlNet pipelines (FLUX.1 and Qwen-Image), with Union models as the default picks. Backend - New core/inference/diffusion_controlnet.py: family-gated discovery (curated Union models + local dirs + bare owner/name repos), resolution to a loadable repo/dir, control-image preprocessing (passthrough + a dependency-free canny edge map), and a supports_controlnet gate. - diffusion.py: a ControlNet manager parallel to the LoRA one. Loads the (small) ControlNet model once via from_pretrained (cached by id) and builds the family's ControlNet pipeline via Pipeline.from_pipe(base, controlnet=model), reusing the resident base modules at their loaded dtype (no reload, no recast). Passes the control image + conditioning scale + guidance start/end at generate time; cleared on unload. - Families: FLUX.1 -> FluxControlNetPipeline/Model, Qwen-Image -> QwenImageControlNetPipeline/Model. Others declare none (gated off). - Gated off for the native engine, GGUF-via-diffusers, and torchao fp8/int8 dense (same rule as LoRA). v1 conditions txt2img only. - Request contract: optional controlnet on DiffusionGenerateRequest; supports_controlnet in status; the choice persisted in gallery meta. - New GET /api/models/diffusion-controlnets for the picker. Frontend - A ControlNet control in the Images rail (model select + control-image upload + control-type select + strength slider), gated by the loaded model's supports_controlnet + family, shown for text-to-image. Tests - New test_diffusion_controlnet.py (10): discovery/resolve/preprocess/gate helpers, request validation, family wiring, and the diffusers pipe manager (loads once, caches, from_pipe with controlnet, rejects unsupported families). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio ControlNet: show the picker on the Create tab (workflow id is 'create', not 'txt2img') The ControlNet control gated on workflow === 'txt2img', but the Images workflow tab ids are create/transform/inpaint/extend/upscale/reference/edit -- there is no 'txt2img'. So the picker never rendered even with a ControlNet-capable model loaded. Gate on 'create' (the text-to-image tab) for both the picker and the request wiring. Found via a live Playwright capture of the running Studio. * Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a stray "gguf" tag on the Hub but ship no .gguf files. The model search classified them as GGUF from the bare tag, so the picker rendered the GGUF variant expander, which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata (populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected. * Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline load when the id is unsloth-hosted or on-device (the backend infers the family + base repo and gates loads to unsloth/* or local paths), and show a clear message otherwise instead of silently ignoring the click. Curated and GGUF paths are unchanged. * Studio Images: keep curated safetensors models in Recommended after download The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet), so they vanished from the picker after the first load and could only be found by typing an exact search. The row already renders a downloaded badge, matching how GGUF Recommended rows stay visible when cached. Drop the exclusion so the curated safetensors always list. * Studio diffusion LoRA: sanitize dots out of adapter aliases The LoRA alias is used as the diffusers PEFT adapter name, and PEFT rejects names containing "." (module name can't contain "."). sanitize_alias kept dots, so a LoRA whose filename carries a version tag (e.g. Qwen-Image-2512-Lightning-8steps-V1.0-bf16) failed to apply with a 400. Replace dots too; the alias stays a valid native <lora:NAME:w> filename stem. Adds regression coverage for internal dots. * Studio Images: clarify the GGUF transformer-quant Advanced control Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode" with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/ FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF, falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only. * Studio Images: list on-device unsloth diffusion models in the picker The Images picker's On Device tab hid every non-GGUF cached repo whenever a task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit and FP8 safetensors) never showed up there. List cached repos that pass the task gate, limited under a filter to unsloth-hosted ones so base repos (which fail the diffusion load trust gate) don't appear only to dead-end on click. Chat behavior is unchanged: the task gate still drops image repos there. * Studio: hide single-file image checkpoints from the chat model picker The chat picker treats a cached repo as an image model, and hides it, only when it ships a diffusers model_index.json. Single-file, ComfyUI, and ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models. Fall back to resolving the repo id against the known diffusion families, the same resolver the Images backend loads from, so these checkpoints are tagged text-to-image and stay in the Images picker only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: add the FLUX.2-dev model family Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline) resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline + Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated, reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror. text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from klein's turbo defaults. Verified live: GGUF load resolves the family + gated base repo and generates a real 1024x1024 image on GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: clearer error for an unsupported diffusion model When a repo id resolves to no diffusion family the load raised 'Could not infer a diffusion family... Pass family_override (z-image)', which points at an unrelated family and doesn't say what is supported. Replace it with a message that lists the supported families (from a new supported_family_names helper) and notes that video models and image models whose diffusers transformer has no single-file loader are not supported. Applies to both the diffusers and native sd.cpp load paths. Also refreshes two stale family-registry comments that still called FLUX.2-dev omitted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add SDXL diffusion family (U-Net pipeline support) SDXL is the first U-Net family in the diffusion backend: its denoiser is pipe.unet (UNet2DConditionModel), not a DiT pipe.transformer, and a single-file .safetensors is the whole pipeline rather than a transformer-only file. The backend previously assumed a DiT transformer everywhere, so add the two hooks a U-Net family needs and register SDXL. DiffusionFamily gains denoiser_attr ("transformer" for DiT, "unet" for SDXL) and single_file_is_pipeline (SDXL loads a single file via pipeline_class.from_single_file with the base repo as config, instead of transformer_class.from_single_file plus a companion assembly). _align_vae_dtype now reads the denoiser generically so img2img and inpaint keep the VAE and U-Net dtypes aligned. The non-GGUF trust gate is extended with a short, exact-match, safetensors-only allowlist of official base repos (the SDXL base/refiner and sdxl-turbo), because SDXL ships only as a full pipeline and has no unsloth-hosted GGUF. Local paths stay trusted as before; a random repo, even one that detects as SDXL, is still rejected. The image-conditioned and ControlNet workflows are the standard SDXL pipelines, built around the resident modules via from_pipe like every other family, so SDXL gets txt2img, img2img, inpaint, outpaint, upscale, LoRA and ControlNet. There is no native sd.cpp mapping yet, so the no-GPU route falls back to diffusers. Frontend catalog gains SDXL Base 1.0 and SDXL Turbo entries with SDXL step/guidance defaults (Turbo: few steps, no CFG; base: ~30 steps, real CFG). Tests: new test_diffusion_sdxl.py (family shape, detection, trust allowlist, model kind, U-Net VAE-dtype alignment, LoRA gate) plus loader-branch tests in test_diffusion_backend.py (pipeline-kind from_pretrained, single-file whole-pipeline from_single_file, allowlist accept/reject). Verified live on GPU: sdxl-turbo loads both as a pipeline and as a single file and generates coherent txt2img + img2img. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Images: LoRA free-text Hugging Face entry + recipe round-trip The backend has always accepted a bare Hugging Face repo id (owner/name, or owner/name:weight-file.safetensors) as a LoRA, downloading and applying it. But the picker only rendered when the curated catalog had entries, and the catalog is empty, so there was no UI path to apply any LoRA. Show the LoRA section whenever the loaded model supports LoRA, and replace the curated-only dropdown with a text input: type a Hub repo id, or pick a discovered adapter from a datalist of suggestions when the catalog is populated. Also restore LoRAs when loading a recipe. restoreSettings now parses the recipe's "id:weight" strings (splitting on the last colon, since the id itself may contain one for a specific weight file) back into the selection, so replaying a saved image reproduces its adapters. The generate payload trims hand-typed ids and drops empty / zero-weight rows, and a model swap clears the selection (a LoRA is family-specific) without discarding a free-text pick that is not in the curated list. * Remove stray async task scratch outputs committed by mistake * ControlNet: reject filesystem-like ids and do not cache a model past an unload race Two review findings on the ControlNet path: - resolve_controlnet's bare-repo fallback accepted any id with a slash, so a path-shaped id (/tmp/x, ../x) reached from_pretrained as a local directory. Restrict the fallback to a strict owner/name HF repo id shape. - _controlnet_pipe now re-checks the cancel event after the blocking from_pretrained: an unload that raced the download had already cleared the caches, so caching the late module would pin it past the unload. * Pipeline prefetch: fetch only the default torch weights A full-pipeline prefetch kept every repo file outside assets/, so an official repo that ships multiple formats (SDXL Base: fp16 variants, ONNX, OpenVINO, Flax, a top-level single-file twin) downloaded tens of GB from_pretrained never loads. Skip non-torch exports and dtype-variant twins in _pipeline_file_downloaded, and drop a component .bin when the same directory carries a picked safetensors weight (diffusers' own preference). * ControlNet: address review findings on the diffusers path - resolve_controlnet enforces catalog family compatibility so a direct API call cannot load a ControlNet built for another family through the wrong pipeline. - Unknown ControlNet ids now surface as a 400 (call site maps FileNotFoundError to ValueError) instead of a generic 500. - strength 0 disables ControlNet entirely, so a no-op selection never pays the download / VRAM cost; the control image is decoded and validated BEFORE the ControlNet is resolved or built, so a malformed image fails fast for the same reason. - ControlNet loads use the base compute dtype (state.dtype is a display string, not a torch.dtype, so it silently fell back to float32) and honor the base offload policy via group offloading instead of forcing the module resident. - Empty/malformed HF token coerced to anonymous access. - Flux Union ControlNet control_mode mapped from the selected control type. - resolve_controlnet drops the unused hf_token/cancel_event params. - ControlNetSpec validates guidance_start <= guidance_end (clean 422). - Images UI ControlNet Select shows its placeholder when nothing is selected. Adds regression tests for family enforcement and the union control-mode map. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA: harden resolution, native tag precedence, and diffusers teardown Address review findings on the LoRA path: - resolve_one: normalise a blank/whitespace hf_token to None (anonymous access) and reject a client-supplied weight file with traversal / absolute path. - resolve_specs: convert FileNotFoundError from an unknown/stale id to ValueError so the route returns 400 instead of a generic 500. - _scan_local: disambiguate local adapters that share a stem (foo.safetensors vs foo.gguf) so each is uniquely addressable. - inject_prompt_tags: the backend-validated weight now wins over a user-typed <lora:ALIAS:...> for a selected adapter; unselected user tags are left alone. - diffusers _apply_loras: reject a .gguf adapter with a clear error before touching the pipe (diffusers loads safetensors only). - _unload_locked: drop the explicit unload_lora_weights() on teardown; the pipe is dropped wholesale (freeing adapters), so the previous call could race an in-flight denoise on the same pipe. - Images page: use a stable LoRA key and clear the selection (not just the options) when the catalog refresh fails. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion: guard trust check against OSError and validate conditioning inputs - _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid characters (or a bare owner/name id) can't raise OSError; treat any failure as not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request still raises the clear FileNotFoundError for a genuinely missing local pick. - generate(): reject mask_image / upscale / reference_images supplied without an input image, and reject reference_images on a family that does not support reference conditioning, instead of silently degrading to txt2img / img2img. * SDXL: reject GGUF up front, skip unused base weights, drop refiner, and harden helpers Addresses review findings on the SDXL family: - Reject a GGUF load for single_file_is_pipeline families (SDXL) in validate_load_request, before the route evicts the current model; SDXL has no transformer-only GGUF variant. - Skip base-repo weight files when a whole-pipeline single file is loaded: from_single_file (config=base) needs only the base config/tokenizer/scheduler, so a local .safetensors no longer triggers a multi-GB base download. - Remove the SDXL refiner from the non-GGUF trust allowlist: it is an img2img-only pipeline but this backend loads every sdxl repo as the base txt2img pipeline. - Normalize a blank/whitespace hf_token to None once in load_pipeline so every load branch degrades to anonymous instead of erroring on a malformed token. - Read the denoiser dtype from a parameter (compile-wrapped modules may lack .dtype) and access state.family.denoiser_attr directly. Adds/updates regression tests for the trust allowlist, GGUF rejection, and base-config filter. * Images: preserve restored LoRAs through model load and never send hidden LoRAs - The LoRA effect cleared the selection on every load->capable transition, which wiped adapters restored from a gallery recipe before the model finished loading. Track the previously-loaded family in a ref and clear only on a real family swap; keep the selection on the initial load and on unload. - Gate the generate payload's loras on loraCapable so a restored selection that is hidden (loaded model does not support LoRA) is never sent to the backend. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Merge diffusion-sdxl into diffusion-lora-ux; keep options-only LoRA catch The catalog-refresh .catch from the lower branch clears the selected adapters too, which is right for its catalog-only picker but wrong here: this picker holds free-text HF repo ids that are valid without being in the catalog, so a transient refresh failure must not wipe them. Family swaps still clear the selection and hidden LoRAs are never sent. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align the VAE to the denoiser's first FLOATING dtype, not its first parameter A GGUF-quantized transformer's leading parameters are packed uint8 storage, so reading next(parameters()).dtype handed nn.Module.to() an integer dtype and every image-conditioned generation on a GGUF model (Qwen-Image-Edit) failed with a 500. Probe the parameters for the first floating dtype, treat an all-integer module as a no-op, and also catch TypeError so an unexpected dtype can never break generation. Regression test included. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address Codex review findings on the image-workflows PR Keep diffusion.py importable without torch: the compile/arch patch modules import torch at module level, so import them lazily at their load/unload call sites instead of at module load. This restores the torchless contract so get_diffusion_backend() works on a CPU/native sd.cpp install. Match family reject keywords and aliases as whole path/name segments, not raw substrings, so an unrelated word like edited, edition, or kontextual no longer misroutes or hides a valid base image model, while supported edit families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same segment matching in the picker task filter. Route FLUX.2-dev native guidance through --guidance like the other FLUX families rather than --cfg-scale. Reject native upscale requests that have no input image. Read image header dimensions and reject over-limit inputs before decoding pixels, so a crafted small-payload image cannot spike memory. Reject an upscale that would shrink the source below its input size. Validate the model_kind against the filename extension before the GPU handoff. Estimate a local diffusers pipeline's size from its on-disk weights so auto memory planning does not skip offload and OOM. Report workflows: [txt2img] from the native backend status so the Create tab stays enabled for a loaded native model. Clamp the outpaint canvas to the backend's 4096px decode limit. Adds regression tests for segment matching and kind/extension validation. * Harden diffusion LoRA handling on the diffusers and native paths Reject LoRA on a torch.compile'd diffusers transformer (Speed=default/max): diffusers requires the adapter loaded before compilation, so applying one to the already-compiled module fails with adapter-key mismatches. The status gate now hides the picker and generate raises a clear message instead. Convert a cancelled Hub LoRA download (RuntimeError Cancelled) to the diffusion cancellation sentinel in resolve_specs, so an unload/superseding load during resolution maps to a 409 instead of a generic server error. Drop weight-0 LoRA rows before the native support gate so a request carrying only disabled adapters stays a no-op on families where native LoRA is unsupported, matching the diffusers path. Reject duplicate LoRA ids in the request model: both apply paths suffix colliding names, so a repeated id would stack the same adapter past its per-adapter weight bound. Strip all user-typed <lora:...> prompt tags on the native path (only the selected adapters are materialized in the managed lora-model-dir, so an unselected tag can never resolve), and restore saved LoRA selections from a gallery recipe so restore reproduces a LoRA image. * Harden ControlNet resolve, gallery metadata, and the control-type picker Check cancellation immediately after a ControlNet from_pretrained and before any device placement, so an unload/eviction that raced the download does not allocate several GB onto the GPU after the load was already cleared. Require a loadable weight or shard index (not just config.json) before a local ControlNet folder is advertised, so an interrupted copy is hidden instead of failing deep in from_pretrained as a generic 500. Do not record a strength-0 ControlNet in the gallery recipe: it is treated as disabled and skipped, so the image is unconditioned and the metadata must not claim a ControlNet was applied. Build the control-type picker from the selected ControlNet's advertised control_types instead of a hardcoded passthrough/canny pair, so a union model with a precomputed depth or pose map sends the correct control_mode. * Address further Codex findings on the image-workflows PR - Persist the actual output image size in the gallery recipe instead of the request sliders: Transform/Inpaint/Edit derive the size from the uploaded image, Extend grows the canvas, and Upscale resizes it, so the sliders recorded (and later restored) the wrong dimensions for those workflows. - Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file name) in validate_load_request, so the unloadable pick fails before chat is evicted rather than deep in from_pretrained. - Only publish an image-conditioned from_pipe wrapper to the shared aux cache when the load is still current: from_pipe runs under the generate lock but not the state lock, so an unload racing its construction could otherwise cache a wrapper over torn-down modules that a later load would reuse. - Verify the Windows CUDA runtime archive checksum before extracting it, like the main sd-cli archive, so a corrupt or tampered runtime is rejected rather than extracted next to the binary. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
|
Daniel Han
|
64f92bccb9
|
Add SDXL diffusion family (U-Net pipeline support) (#6788)
* Studio diffusion: cross-platform device policy, fp16 guard, lock split, validate-before-evict Phase 1 of porting the richer diffusion stack onto the image-generation backend. - Add a compartmentalized device/dtype policy module (diffusion_device.py) resolving CUDA/ROCm/XPU/MPS/CPU with capability flags. Keeps the NVIDIA capability-based bf16 choice; ROCm and XPU are isolated; MPS uses bf16 or fp32, never a silent fp16 that renders a black image. - Add a per-family fp16_incompatible flag (Z-Image) and promote a resolved float16 to float32 for those families so they do not produce black images. - Split the backend locks: a generation holds only _generate_lock, so status, unload, and a new load are never blocked by a long denoise. Add per-generation cancellation via callback_on_step_end so an eviction or a superseding load preempts a running generation; a replacement load waits for it to stop before allocating, so two pipelines never sit in VRAM at once. - Validate a load request before the GPU handoff so an unloadable pick never evicts a working chat model, and reject missing local paths up front. - Add CPU-only tests for the device policy, dtype guard, lock split and cancellation, and validate-before-evict, plus a GPU benchmark/regression script (scripts/diffusion_bench.py) measuring latency, peak VRAM, and PSNR against a saved reference. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2A): measured-budget memory planner + offload/VAE policy Add a lean, backend-agnostic memory policy that picks a CPU-offload policy and VAE tiling/slicing from measured free device memory vs the model's estimated resident footprint, then applies it to the built pipeline. auto stays resident when the model fits (byte-identical to the prior resident path), and falls to whole-module offload when tight; fast/balanced/low_vram are explicit overrides. Sequential submodule offload is unreliable for GGUF transformers on diffusers 0.38, so it falls back to whole-module offload and status reports the policy actually engaged. Verified on Z-Image-Turbo Q4_K_M (B200): auto reproduces the resident image with no VRAM/latency regression (PSNR inf); balanced/low_vram cut generation peak VRAM 47.9% (15951 -> 8318 MB) with byte-identical output, at the expected latency cost. 73 prior + 35 new CPU tests pass. * Studio diffusion (Phase 2D): streamed block-level offload + functional VAE tiling Add a streamed 'group' offload tier (diffusers apply_group_offloading, block_level, use_stream) that keeps the transformer flowing through the GPU a few blocks at a time while the text encoder / VAE stay resident, and fix VAE tiling to drive the VAE submodule (pipelines like Z-Image expose enable_tiling on pipe.vae, not the pipeline). apply_memory_plan now returns the (policy, tiling) actually engaged so status never overstates either, and group falls back to whole-module offload when the transformer can't be streamed. Measured on Z-Image (B200), all lossless (PSNR inf vs resident): balanced/group cuts generation peak VRAM 32% (15951 -> 10840 MB) at near-resident speed (2.07 -> 2.99s); low_vram/model cuts it 48% (-> 8318 MB) but is slower (7.99s). Mode names now match that tradeoff: balanced = stream the transformer, low_vram = offload every component. auto picks group when the companions fit resident, else model. 112 CPU tests pass. * Studio diffusion (Phase 5): image quality-vs-quant accuracy harness Add scripts/diffusion_quality.py, the accuracy analogue of the KLD workflow: hold prompt + seed fixed, render a grid with a reference quant (default BF16), then render each candidate quant and measure drift from the reference. Records mean PSNR + SSIM (pure-numpy, no skimage/scipy) and optional CLIP text-alignment + image-similarity (transformers, --clip), plus file size, latency, and peak VRAM, then prints a quality-vs-cost table and recommends the smallest quant within a quality budget. --selftest validates the metrics on synthetic images with no GPU or model. Verified on Z-Image (B200): the table degrades monotonically with quant size (Q8 -> Q4 -> Q2: PSNR 21.7 -> 15.5, SSIM 0.82 -> 0.61), while CLIP-text stays flat (~0.34) -- quantization erodes fine detail far more than prompt adherence. * Studio diffusion (Phase 3): opt-in speed layer (channels_last / compile / TF32) Add a speed_mode knob (off by default, so the render path stays bit-identical): default applies channels_last VAE + regional torch.compile of the denoiser's repeated block where eligible; max also enables TF32 matmul and fused QKV. Regional compile is gated off for the GGUF transformer (dequantises per-op) and for families flagged not compile-friendly (a new supports_torch_compile flag, False for Z-Image), so it activates automatically only once a non-GGUF bf16 transformer is loaded. Speed optims run before placement/offload, per the diffusers composition order. status now reports speed_mode + the optims actually engaged. Verified on Z-Image (B200): default -> ['channels_last'], max -> ['channels_last', 'tf32'], compile correctly skipped for GGUF; generation works in every mode. 121 CPU tests pass. * Studio diffusion (Phase 2B): opt-in fp8 text-encoder layerwise casting Add a text_encoder_fp8 knob that casts the companion text encoder(s) to fp8 (e4m3) storage via diffusers apply_layerwise_casting, upcasting per layer to the bf16 compute dtype while normalisations and embeddings stay full precision. Applied before placement, gated to CUDA + bf16, best-effort (a failure leaves the encoder dense). status reports which encoders were cast. Verified on Z-Image (B200, balanced/group mode where the encoder stays resident): generation peak VRAM dropped 37% (10840 -> 6791 MB, below the lowest-VRAM offload) at near-resident speed. It is a memory-vs-quality tradeoff, not free -- ~20 dB PSNR vs the bf16 encoder, a larger shift than one transformer quant step -- so it is off by default and documented as such, with the Phase 5 harness to size the cost. 127 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2C): NVFP4 text-encoder quant (+ generalise fp8 knob) Generalise the text-encoder precision knob from a fp8 bool to text_encoder_quant (fp8 | nvfp4). nvfp4 quantises the companion text encoder to 4-bit via torchao NVFP4 weight-only (two-level microscaling) on Blackwell's FP4 tensor cores; fp8 stays the broader-hardware path (cc>=8.9). Both are gated, best-effort, and run before placement; status reports the mode actually engaged. This is the lean realisation of GGUF-native text-encoder quant: 4-bit on the encoder without the 3045-line port. Verified on Z-Image (B200, balanced/group where the encoder stays resident), vs the bf16 encoder: nvfp4 cut generation peak VRAM 48% (10840 -> 5593 MB, the lowest TE option, below whole-model offload) at near-fp8 quality (16.4 vs 17.1 dB PSNR), and both quants ran faster than bf16. A memory-vs-quality tradeoff (off by default); size it per model with the Phase 5 quality harness. diffusion_bench gains --text-encoder-quant. 129 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): native stable-diffusion.cpp engine for CPU/Mac Adds the CPU / Apple-Silicon tier of the two-engine strategy, mirroring the chat backend's llama.cpp shell-out. Diffusers stays the default on CUDA / ROCm / XPU; this covers the hardware diffusers serves poorly, consuming the same split GGUF assets Studio already curates. - sd_cpp_args.py: pure sd-cli command builder. Maps the family to its text-encoder flag (Z-Image Qwen3 to --llm, Qwen-Image to --qwen2vl, FLUX.1 CLIP-L + T5), and the diffusers memory policy (none/group/model/sequential) to sd.cpp's offload flags (--offload-to-cpu / --clip-on-cpu / --vae-on-cpu / --vae-tiling / --diffusion-fa), so one user knob drives both engines. - sd_cpp_engine.py: SdCppEngine over a located sd-cli. find_sd_cpp_binary() with the same precedence as the llama finder (env override, then the Studio install root, then in-tree, then PATH), an is_available/version probe, and a one-shot subprocess generate that streams progress and returns the PNG. runtime_env() prepends the binary's directory to the platform library path so a prebuilt's bundled libstable-diffusion.so resolves. select_diffusion_engine() is the pure routing decision (GPU backends to diffusers, CPU/MPS to native when present). - install_sd_cpp_prebuilt.py: resolve + download the per-host prebuilt (macOS-arm64/Metal, Linux x86_64 CPU, Vulkan/ROCm/Windows variants) into the Studio install root. resolve_release_asset() is a pure, unit-tested host-to-asset matrix. - scripts/sd_cpp_smoke.py: end-to-end native generation harness. Tests (CPU-only, subprocess/filesystem stubbed): 49 new across args, engine, routing, runtime env, and the installer resolver. Full diffusion suite 166 passing. Verified on a B200 box: built sd-cli (CUDA) and the prebuilt (CPU) both generate Z-Image-Turbo Q4_K end to end through SdCppEngine: balanced (group offload, 5.0s gen), low_vram (full CPU offload + VAE tiling, 13.4s), and the dynamically-linked CPU prebuilt (50.4s on CPU), all producing coherent images. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 6): img2img / inpaint / edit / LoRA / upscale on the native engine Builds on Phase 4's native stable-diffusion.cpp engine, extending it from text-to-image to the wider feature surface, since sd.cpp supports all of these through the binary already. Pure command-builder additions plus one engine method, so the txt2img path is unchanged. - sd_cpp_args.py: SdCppGenParams gains image-conditioning fields. init_img + strength make a run img2img, adding mask makes it inpaint, ref_images drives FLUX-Kontext / Qwen-Image-Edit style editing (repeated --ref-image), and lora_dir + the <lora:name:weight> prompt syntax select LoRAs. New SdCppUpscaleParams + build_sd_cpp_upscale_command for the ESRGAN upscale run mode (input image + esrgan model, no prompt / text encoders). - sd_cpp_engine.py: the subprocess runner is factored into a shared _run() so generate() (now carrying the conditioning flags) and a new upscale() reuse the same streaming / error / output-check path. - scripts/sd_cpp_smoke.py: --task {txt2img,img2img,upscale} with --init-img / --strength / --upscale-model / --upscale-repeats. Tests: 10 new across the img2img / inpaint / edit / LoRA flag construction, the upscale builder and its validation, and the engine's img2img + upscale paths. Full diffusion suite 176 passing. Verified on a B200 box through SdCppEngine: img2img (Z-Image-Turbo Q4_K, the init image conditioned at strength 0.6, 4.8s) and ESRGAN upscale (512x512 -> 2048x2048 via RealESRGAN_x4plus_anime_6B, 2.7s), both producing coherent images. Video and the diffusers-path feature wiring are deferred. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): accuracy-preserving speed pass Re-review of the diffusion stack (#6675/#6679/#6680) surfaced one real accuracy bug and a dead-on-arrival speed path; this fixes both and adds the lossless / near-lossless wins, all measured on a B200. Correctness: - TF32 global-state leak (fix). speed_mode=max flipped torch.backends.*.allow_tf32 process-wide and never restored them, so a later `off` load silently inherited TF32 and was no longer bit-identical. Added snapshot_backend_flags / restore_backend_flags (TF32 + cudnn.benchmark), captured before the speed layer runs and restored on unload. Verified: load max -> unload -> load off is now byte-identical (PSNR inf) to a fresh off. - sd-cli timeout could hang forever. _run() blocked in `for line in stdout` and only checked the timeout after EOF, so a child stuck in model load / GPU init with no output ignored the timeout. Drained stdout on a reader thread with a wall-clock deadline. Added a silent-hang regression test. Speed (diffusers path), near-lossless, opt-in tiers: - Regional torch.compile now runs on the GGUF transformer. The is_gguf gate (and Z-Image's supports_torch_compile=False) were stale: compile_repeated_blocks compiles and runs ~2.2x faster on the GGUF Z-Image transformer on torch 2.9.1 / diffusers 0.38 (the per-op dequant stays eager, the rest of the block compiles). Measured: off 1.80s -> default 0.82s/gen (+54.7%), PSNR 37.7 dB vs eager -- far above the Q4 quant noise floor (~21 dB), so it does not move output quality. Gate relaxed; default tier delivers it. - cudnn.benchmark added to the default tier (autotunes the fixed-shape VAE convs). - torch.inference_mode() around the pipeline call (lossless, strictly faster than the no_grad diffusers uses internally). Memory path: - VAE tiling (not bit-identical >1MP) restricted to the model/sequential/CPU tiers; the balanced (group) tier keeps exact slicing only, so it is now bit-identical to the resident image (verified PSNR inf) and slightly faster. - Group offload adds non_blocking + record_stream on the CUDA stream path to overlap each block's H2D copy with compute (lossless; gated on the installed diffusers signature so older versions still work). Native (sd.cpp) path: - native_speed_flags: a first-class speed knob (default -> --diffusion-fa, a near-lossless CUDA win that was previously only added on offload tiers; max also -> --diffusion-conv-direct). conv-direct stays opt-in: measured +45% on CUDA, so it is never auto-on. Engine generate() merges it, de-duped against offload flags. Default profile: a GGUF model with no explicit speed_mode now resolves to the `default` profile (resolve_speed_mode), since compile's perturbation sits below the quantisation noise floor and so does not reduce quality versus the dense reference; out of the box a GGUF Z-Image generation drops from 1.80s to 0.81s. Dense models stay `off` / bit-identical, and an explicit speed_mode -- including "off" -- is always honored, so the byte-identical path remains one flag away and is the regression reference. Tooling: scripts/compile_probe.py (eager vs compiled GGUF probe), scripts/ perf_verify.py (the B200 verification above), and diffusion_bench.py gains --speed-mode so the speed tiers are benchmarkable. Tests: 183 passing (was 166); new coverage for the backend-flag snapshot/restore, GGUF compile eligibility, the balanced tiling/slicing split, native_speed_flags + the engine de-dup, and the sd-cli silent-hang timeout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): max tier uses max-autotune-no-cudagraphs + engine/lever benchmarks The opt-in `max` speed tier now compiles the repeated block with mode=max-autotune-no-cudagraphs (dynamic=False) instead of the default mode: Triton autotuning for GEMM/conv-heavier models, gated to the tier where a longer cold compile is acceptable. CUDA-graph modes (reduce-overhead / max-autotune) are deliberately avoided -- both crash on the regionally-compiled block (its static output buffer is overwritten across denoise steps), measured. Adds two reproducible benchmarks used to validate the optimization research: - scripts/compare_engines.py: PyTorch (diffusers GGUF) vs native sd.cpp head-to-head. - scripts/leverage_probe.py: coordinate_descent_tuning + FirstBlockCache probes. Measured on B200 (Z-Image Q4_K_M, 1024px, 8 steps): default compile 0.80s/gen; coordinate_descent_tuning 0.79s (within noise, already covered by max-autotune); FirstBlockCache does not run on Z-Image (diffusers 0.38 block-detection / Dynamo). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source) Add an opt-in transformer_quant mode that loads the dense bf16 transformer and torchao-quantises it onto the low-precision tensor cores, instead of the GGUF transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's 0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor. GGUF+compile stays the low-memory default and the fallback. The mode is gated on CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB); any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe (Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips the tiny projections that crash int8's torch._int_mm. New module mirrors diffusion_precision.py; quant runs before compile before placement. 184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py is the standalone torchao lever probe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): consumer-GPU tuning - lock fp8 fast accumulate, prefer fp8 over mxfp8, reject 2:4 sparsity Consumer Blackwell halves tensor-core throughput on FP32 accumulate (fp8 419 vs 838 TFLOPS with FP16 accumulate; bf16 209), so: - fp8 config locks use_fast_accum=True (Float8MMConfig). torchao already defaults it on; pinning it guards consumer cards against a default change. On B200 it is identical speed and slightly better quality (LPIPS 0.050 vs 0.091). - the Blackwell auto ladder prefers fp8 over mxfp8 (measured faster + more accurate). 2:4 semi-structured sparsity evaluated and rejected (scripts/sparse_accum_probe.py): 2:4 magnitude-prune + fp8 gives LPIPS 0.858 (broken image) with no fine-tune, the cuSPARSELt kernel errors on torch 2.9, and it does not compose with torch.compile (our main ~2x). Documented as a dead end, not shipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add fp8 fast-accum overflow verification probe scripts/fp8_overflow_check.py hooks every quantised linear during a real Z-Image generation and reports max-abs + non-finite counts for use_fast_accum True vs False. Confirms fast accumulation is an accumulation-precision knob, not an overflow one: across 276 linears, including Z-Image's ~1.0e6 activation peaks (which overflow FP16), 0 non-finite elements and identical max-abs for both modes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40; GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it. Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s), no overflow, quality below the quant noise floor. So the default leans to accuracy on data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force) lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off). 187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override threading). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add NVFP4 probe documenting it is not yet a win on torch 2.9 scripts/nvfp4_probe.py measures NVFP4 via torchao on the real Z-Image transformer. Finding (B200, 1024px/8 steps): NVFP4 is a torchao feature and DOES run with use_triton_kernel=False (the default triton path needs the missing MSLK library), but only at bf16-compile rate (0.667s vs fp8 0.592s) -- it dequantises FP4->bf16 rather than using the FP4 tensor cores. The real FP4 speedup needs MSLK or torch>=2.11 + torchao's CUTLASS FP4 GEMM. The smoke probe (default triton=True) already keeps NVFP4 out of auto on this env, so auto correctly stays on fp8; NVFP4 activates automatically once fast. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): prefer fp8 over nvfp4 in Blackwell auto ladder Validated NVFP4 on torch 2.11 + torchao CUTLASS FP4 in an isolated env. The FP4 tensor-core GEMM is genuinely active there (a 16384^3 GEMM hits ~3826 TFLOPS, 2.52x bf16 and 1.37x fp8), but it only beats fp8 on very large GEMMs. At the diffusion transformer's shapes (hidden ~3072, MLP ~12288, M~4096) NVFP4 is both slower (0.81x fp8 end to end on Z-Image 1024px) and less accurate (LPIPS 0.166 vs fp8's 0.044). Reorder the Blackwell auto ladder to fp8 before nvfp4 so auto is correct even on a future MSLK-equipped box; nvfp4 stays an explicit opt-in. Add scripts/nvfp4_t211_probe.py (extension diagnostics + GEMM micro + end-to-end). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): pre-quantized transformer loading The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs 13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline (scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True) the quantized weights, so the dense bf16 never touches the GPU. Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same torchao config + min_features filter the runtime path uses, applied ahead of time. New core/inference/diffusion_prequant.py (resolve_prequant_source + load_prequantized_transformer, best-effort, lazy imports). diffusion.py _load_dense_quant_pipeline tries the pre-quant source first and falls back to the dense materialise+quantise path, then to GGUF, so the default is unchanged. DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for the resolver, the meta-init+assign loader, and the backend branch selection + fallbacks; GPU verification via scripts/verify_prequant_backend.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): attention-backend selection Add a selectable attention kernel via the diffusers set_attention_backend dispatcher. Attention is memory-bandwidth bound, so a better kernel is an end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV matmuls torchao never touches) and composes with torch.compile. auto picks the best exact backend for the device: cuDNN fused attention (_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default (below the compile/quant noise floor); native SDPA elsewhere and when speed=off (so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/ xformers/aiter are honored, and an unavailable kernel falls back to the default rather than failing the load. New core/inference/diffusion_attention.py (normalize + per-device select + apply, best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline; attention_backend threads through begin_load / load_pipeline / status like the other load knobs. New request field attention_backend + status field. Hermetic CPU tests for normalize / select policy / apply fallback, plus route threading + 422. Measured via scripts/perf_levers_probe.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the arch tier to put int8 first on a consumer/workstation GPU (detected by the existing _is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first. Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation (-> int8) and data-center Ada/Hopper/Blackwell (-> fp8). * Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across denoise steps a DiT's output settles, so once the first block's residual barely changes the remaining blocks are skipped and their cached output reused. diffusers ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the standalone apply_first_block_cache hook as a fallback). Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 -> 2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar. OFF by default and a per-load opt-in: the win scales with step count, so it is for many-step models (Flux / Qwen-Image) and pointless for few-step distilled models (e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction of the trajectory. It composes with regional compile only with fullgraph=False (the cache's per-step decision is a torch.compiler.disable graph break), which the speed layer now switches to automatically when a cache is engaged. Best-effort: a model whose block signature the hook does not recognise is caught and the load proceeds uncached. - new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache (enable_cache / apply_first_block_cache fallback; threshold auto-raised for a quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import). - diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph when a cache is engaged. - diffusion.py: apply_step_cache before compile; thread transformer_cache / transformer_cache_threshold through begin_load -> load_pipeline and report the engaged mode in status(). - models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and transformer_cache_threshold request fields, engaged mode in the status response. - hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold selection, and best-effort failure handling, plus route threading + validation. - scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM / LPIPS vs the compiled no-cache baseline). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears) The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with 'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2 *_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing min_features filter did not exclude them. Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed, guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers (M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and quantises these layers fine, so the exclusion is int8-only. Sequence embedders (context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note 'context_embedder' contains the substring 'text_embed', which is why the token is the specific 'time_text_embed', not 'text_embed'. Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s (5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already working) are unchanged. - diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes exclude_name_tokens; quantize_transformer passes it for int8 only. - hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps attention / FFN / sequence-embedder linears), while fp8 keeps them. - scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's Linear layers and derive the exclusion list. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears) The prequant-checkpoint builder applied the dense quant filter without the int8-only M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8 checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now produces a working checkpoint on every supported model, giving int8 (the consumer-preferred scheme) the same ~2x load-VRAM and download reduction fp8 already had. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the native stable-diffusion.cpp engine instead of diffusers, with diffusers as the guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM. - diffusion_engine_router: centralised engine selection (built on the existing select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason. - sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface backed by sd-cli, with lazy binary install, registry-driven asset fetch, step-progress parsing, and cancellation. - diffusion_families: per-family single-file VAE + text-encoder asset mapping. - sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled). - routes/inference + gpu_arbiter: drive the active engine via the router; the API now reports the active engine and any fallback reason. - tests for the backend, router, route selection, and cancellation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler Address review feedback on #6724: - engine router: unload the engine being deactivated on a switch, so the old model is not left resident-but-unreachable (the evictor only targets the active engine). - generate route: sd.cpp execution errors (nonzero exit / timeout / missing output) now map to 500, not 409 (which only means not-loaded / cancelled). - native batch: return per-image seeds and persist the actual seed for each image so every batch image is reproducible. - Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the stable-diffusion.cpp docs; other families keep sd-cli defaults. - honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload offload flags on the native load instead of hardcoding them off. - fail the load when the sd-cli binary is present but not runnable (version() now returns None on exec error / nonzero exit). - size estimate: only treat the transformer asset as a possible local path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in load_prequantized_transformer ends in torch.load(weights_only=False), which executes arbitrary code from the pickle. The transformer_prequant_path load-request field reached that unpickle for any local file an authenticated caller named, so a request could trigger remote code execution. Refuse the source.kind=='path' branch unless the operator sets UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted and unaffected. Document the requirement on the API field and add gate tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa - apply_attention_backend now restores the native default when no backend is requested or a kernel fails. diffusers keeps a process-wide active attention backend that set_attention_backend updates, and a fresh transformer's processors follow it, so a load that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile load pinned, breaking the bit-identical/off guarantee. - select_attention_backend drops flash3/flash4 up front when the CUDA capability is below Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit request on the wrong card set fine then crashed mid-generation; it now falls back to native. - Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a valid alias of native) is accepted instead of 422-rejected by Pydantic. - Drop the dead replace('-','_') normalization (no alias uses dashes/underscores). - perf_levers_probe.py output dir is now relative to the script, not a hardcoded path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF - apply_step_cache now engages only via the transformer's native enable_cache (the diffusers CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model now runs uncached per the best-effort contract. - GGUF transformers are quantized (the default Studio load path), so they now use the higher quantized FBCache threshold when the caller leaves it unset, instead of the dense default that could keep the cache from triggering. - fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so fullgraph=True failed warmup and silently measured an eager cached run); output dir is now relative to the script, not a hardcoded path. * Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder _is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate. Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0). * Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just TypeError on older torchao), which would break the mxfp8 config helper instead of falling back to the default. Catch both so the fallback is robust. quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds CUDA memory during the per-row VRAM probe; output dir relative to the script. * Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load - snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the whole snapshot. restore_backend_flags restores each flag independently so one failure can't leave the others leaked process-wide. - load_pipeline restores the flags (and clears the GPU cache) when the build fails after apply_speed_optims mutated the process-wide flags but before _state captured them for unload to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and contaminated later off generations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs without producing output (or without closing stdout) would never reach proc.wait and the wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the PROCESS, so the main thread always enforces the timeout and kills a hung process (which closes the pipe and ends the reader). Add a test that times out even when stdout blocks, and make the no-binary test hermetic so a host-installed sd-cli can't leak in. * Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn The filter callback can be invoked without a module name, so fqn.lower() would raise AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens, so the linear is kept) instead of crashing the quantise pass. * Studio diffusion (Phase 16) review fixes: native engine robustness - sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64); a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds aliased to the same image. Random seeds stay 53-bit (JS-safe). - sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub fall back to anonymous instead of failing auth on a blank token. - sd_cpp_backend: a superseding load now cancels the in-flight generation, so the old sd-cli can no longer return/persist an image from the previous model. - diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a slow 10+ GB free / CUDA sync does not block engine selection. - diffusion_engine_router: probe sd-cli runnability (version()) before committing to native, so a present-but-unrunnable binary falls back to diffusers at selection. - diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a CPU-only install can still reach the native sd.cpp engine instead of failing load. - tests updated for the runnability probe + a not-runnable fallback case. * Studio diffusion (Phase 9) review fixes: prequant safety + validation - SECURITY: a request-supplied local pre-quant path is now unpickled only when it resolves inside an operator-configured ALLOWLIST of directories (UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in, once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on any path a load request named (arbitrary code execution). realpath() blocks symlink escapes; a bare on/off toggle is no longer a wildcard. - Validate the checkpoint's min_features against the runtime Linear filter, so a checkpoint that quantised a different layer set is rejected instead of silently loading a model that mismatches the dense path while reporting the same scheme. - Tolerant base_model_id compare (exact or same final path/repo segment), so a local path or fork of the canonical base is accepted instead of falling back to dense. - _has_meta_tensors uses any(chain(...)) (no intermediate lists). - prequant verify/probe scripts use repo-relative paths (+ env overrides), not the author's absolute /mnt paths. - tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts - diffusion_memory: when group offload is unavailable and the plan falls back to whole-module offload, enable VAE tiling (the group plan left it off, but the fallback is the low-VRAM path where the decode spike can OOM). Covers both the group and sequential fallback branches. - perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a balanced bit-identity regression actually fails the check instead of exiting 0. - compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and the load-progress poll has a 30 min deadline instead of looping forever on a hang. - test for the group->model fallback enabling VAE tiling. * Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path - diffusion: a torchao-quantized transformer is committed only compiled. A dense model resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF it replaced), so when transformer_quant engaged and speed resolved to off, promote to default (regional compile); warn loudly if compile still does not engage. - diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK); otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF. - nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path). - test asserts the eager-quant -> default-compile promotion. * Studio diffusion (Phase 10) review fixes: attention gating + probe isolation - diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation, so auto now stays on native SDPA there. - diffusion_attention: _active_attention_backend handles get_active_backend() returning an enum/None (not a tuple); the old unpack always raised and was swallowed, so the native-restore short-circuit never fired. - perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul so the inductor_flags variant doesn't leak into later compiled rows. - tests for the SM80 cuDNN gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening - install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket); extract through a per-member containment check (Zip-Slip guard); expanduser the --install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch the separately-published cudart runtime DLL archive so sd-cli.exe can start. - sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie. - tests: Zip-Slip rejection, normal extraction, studio-home discovery. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of the literal --output path. SdCppEngine.generate checked only the literal path, so a batch generation would exit 0 and then raise 'no image' (or return a stale file). generate now returns the literal path when present and otherwise falls back to the numbered siblings; single-image behavior is unchanged. Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected without error. * Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats Codex review on the native engine arg builder: - build_sd_cpp_command emitted --width/--height unconditionally, so an img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of the input. width/height are now Optional (None = unset): an image-conditioned run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives the size from the input image (set_width_and_height_if_unset); a plain txt2img run with unset dims keeps the prior 1024x1024 default; explicit dims are always honored. width/height are read only by the builder, so the type change is local. - build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...) that silently swallowed repeats=0 into sd-cli's default of one pass, turning an explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError and emits the flag for any explicit value != 1. Tests: img2img unset dims omit width/height (init_img and ref_images), explicit dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at the default. (Two pre-existing binary-discovery tests fail only because a real sd-cli is installed in this dev environment; unrelated to this change.) * Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc Codex review: the transformer_prequant_path field description still told operators to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the prior security fix made that variable a directory allowlist -- _allowed_prequant_roots deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator following the documented =1 would have every transformer_prequant_path request silently refused. The description now states it must name one or more allowlisted directories and that a bare on/off value is not accepted. Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does not say =1, and describes an allowlist/directory (guards against doc drift). * Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset Codex review on attention-backend selection: - Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first generation with no fallback. select_attention_backend now applies _cudnn_attention_supported() to an explicit cuDNN request too. - flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3 is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a (min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+. - apply_attention_backend's success path left diffusers' process-wide active backend pinned to the kernel it set; a later component whose processors are unconfigured (backend None) would inherit it. It now resets the global registry to native after a successful per-transformer set (the transformer keeps its own backend), best-effort. Also fixed _active_attention_backend: get_active_backend() returns a (name, fn) tuple, so the prior code stringified the tuple and never matched a name, defeating the native-restore short-circuit. Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on SM90; global registry reset after a successful set; _active_attention_backend reads the tuple return. * Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300 (Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So _is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they are treated as data-center class and keep the intended fp8-first behavior. Test: extends the datacenter parametrize with 'NVIDIA B300' and 'NVIDIA GH200 480GB' (now _is_consumer_gpu False). * Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder Codex review: the M=1 modulation/embedder exclusion was wired only into the dense runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint quantised the AdaLN modulation and conditioning-embedder linears, and loading it via transformer_prequant_path (the load path only loads already-quantised tensors, it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes for the runtime path. Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 -> the M=1 exclusion, every other scheme -> none) and use it in both the runtime quantiser and the builder, so a prequant artifact's quantised-layer set always matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion). Test: int8_exclude_name_tokens returns the exclusion for int8 and () for fp8/nvfp4/mxfp8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race Codex review on the native-engine routing: - The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat) unconditionally after engine selection. A native sd.cpp load on a pure-CPU host never touches the GPU, so that needlessly tore down the resident chat model. The handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it. - sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when _run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so the setting was unverifiable. status now derives them from state.offload_flags (still 'none' on CPU, where the flags are empty). - _run_load committed the new state without cancelling/waiting on a generation that started during the (slow) asset download, so a stale sd-cli run against the OLD model could finish afterward and persist an image from the previous model once the new load reported ready. The commit now signals the in-flight cancel and waits on _generate_lock before swapping _state (taken only at commit, so the download never serialises against generation), mirroring the diffusers load path. Tests: CPU native load skips the arbiter while a GPU native load takes it; status reports offload active when flags are set; _run_load cancels and waits for an in-flight generation before committing. * Studio diffusion (Phase 14) review round 2: align helper name with the stack Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the identical helper already present higher in the diffusion stack (Phase 16). The helper definition, the runtime quantiser call, and the offline builder are now byte-identical to that version, so the two branches no longer introduce a divergent name for the same single-source-of-truth and the stack merges without a conflict on this fix. No behavior change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: eager patches + torch.compile cache speed phase Adds the opt-in speed path for the GGUF diffusion transformer behind a selectable speed mode (default off, so output is unchanged until a profile is chosen): - diffusion_eager_patches.py: shared eager fast-paths (channels_last, attention/backend selection, fused norms and QKV) installed at load and rolled back on unload or failed load. - diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent torch.compile cache and the GGUF-transformer compile wiring. - diffusion_arch_patches.py: architecture-specific patches. - diffusion_patch_backend.py: shared install/restore plumbing. - diffusion_speed.py: speed-profile planning. Tests for each module plus the benchmarking and probe scripts used to measure speed, memory, and accuracy of the path. * Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI Backend: - Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file fp8 transformers, gated to the unsloth org plus a curated allowlist. - Image-conditioned workflows built with Pipeline.from_pipe so they reuse the loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint, outpaint, and a hires-fix upscale pass. - Instruction editing as its own family kind (Qwen-Image-Edit-2511, FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and multi-reference) plus klein inpaint. - Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the decoded image size and cap upscale output to avoid OOM on large inputs. - Fixes: from_pipe defaulting to a float32 recast that crashed torchao quantized transformers; image-conditioned calls forcing the slider size onto the input image. Native sd.cpp engine rejects image-conditioned and reference requests it cannot serve. Frontend: - Redesigned Images page with capability-gated workflow tabs (Create, Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor, client-side outpaint, and a multi-reference picker. - Advanced options moved to a right-docked panel mirroring Chat: closed by default, toggled by a single fixed top-bar button that stays in place. sd.cpp installer: pin the release, verify each download's sha256, add a download timeout, and make the source repo configurable for a future mirror. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: correct the Advanced panel comment (closed by default, fixed toggle) * Studio diffusion: LoRA adapters for the Images workflow Add community LoRA support across both diffusion backends, the single biggest step toward broad image-workflow coverage. Backend - New shared module core/inference/diffusion_lora.py: adapter discovery (local scan + curated catalog + owner/name[:file] Hub refs), download via hf_hub_download_with_xet_fallback, alias sanitization, native managed-dir materialization with collision-broken aliases, prompt-tag injection (deduped against user-typed tags), and a supports_lora gate. - Native sd-cli: resolve + materialize selected LoRAs into a per-run managed dir, inject <lora:ALIAS:w> tags, pass --lora-model-dir with --lora-apply-mode auto. The arg builder already emitted these flags. - Diffusers: non-fused load_lora_weights + set_adapters manager, tracked on the pipe so an unchanged selection is a no-op and a model swap resets; cleared on unload. Never fuses (breaks quantized transformers and blocks live weight tweaks). - Gated off where unsupported: torchao fp8/int8 dense, GGUF-via-diffusers, and native Qwen-Image (no LoRA name-conversion branch upstream). - Request contract: optional loras on DiffusionGenerateRequest; empty or omitted is identical to today. supports_lora surfaced in status; chosen LoRAs persisted in gallery recipe metadata. - New GET /api/models/diffusion-loras for the picker (family-filtered). Frontend - Repeatable multi-LoRA picker (adapter select + weight slider 0..2 + remove), gated by the loaded model's supports_lora and family, max 8. Tests - New test_diffusion_lora.py (14): helpers, request validation, native tag/dir wiring, diffusers set_adapters manager, supports_lora matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: ControlNet for the Images workflow (diffusers) Add ControlNet conditioning, the #2 most-used diffusion workflow after LoRA, on the diffusers backend for the families with ControlNet pipelines (FLUX.1 and Qwen-Image), with Union models as the default picks. Backend - New core/inference/diffusion_controlnet.py: family-gated discovery (curated Union models + local dirs + bare owner/name repos), resolution to a loadable repo/dir, control-image preprocessing (passthrough + a dependency-free canny edge map), and a supports_controlnet gate. - diffusion.py: a ControlNet manager parallel to the LoRA one. Loads the (small) ControlNet model once via from_pretrained (cached by id) and builds the family's ControlNet pipeline via Pipeline.from_pipe(base, controlnet=model), reusing the resident base modules at their loaded dtype (no reload, no recast). Passes the control image + conditioning scale + guidance start/end at generate time; cleared on unload. - Families: FLUX.1 -> FluxControlNetPipeline/Model, Qwen-Image -> QwenImageControlNetPipeline/Model. Others declare none (gated off). - Gated off for the native engine, GGUF-via-diffusers, and torchao fp8/int8 dense (same rule as LoRA). v1 conditions txt2img only. - Request contract: optional controlnet on DiffusionGenerateRequest; supports_controlnet in status; the choice persisted in gallery meta. - New GET /api/models/diffusion-controlnets for the picker. Frontend - A ControlNet control in the Images rail (model select + control-image upload + control-type select + strength slider), gated by the loaded model's supports_controlnet + family, shown for text-to-image. Tests - New test_diffusion_controlnet.py (10): discovery/resolve/preprocess/gate helpers, request validation, family wiring, and the diffusers pipe manager (loads once, caches, from_pipe with controlnet, rejects unsupported families). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio ControlNet: show the picker on the Create tab (workflow id is 'create', not 'txt2img') The ControlNet control gated on workflow === 'txt2img', but the Images workflow tab ids are create/transform/inpaint/extend/upscale/reference/edit -- there is no 'txt2img'. So the picker never rendered even with a ControlNet-capable model loaded. Gate on 'create' (the text-to-image tab) for both the picker and the request wiring. Found via a live Playwright capture of the running Studio. * Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a stray "gguf" tag on the Hub but ship no .gguf files. The model search classified them as GGUF from the bare tag, so the picker rendered the GGUF variant expander, which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata (populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected. * Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline load when the id is unsloth-hosted or on-device (the backend infers the family + base repo and gates loads to unsloth/* or local paths), and show a clear message otherwise instead of silently ignoring the click. Curated and GGUF paths are unchanged. * Studio Images: keep curated safetensors models in Recommended after download The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet), so they vanished from the picker after the first load and could only be found by typing an exact search. The row already renders a downloaded badge, matching how GGUF Recommended rows stay visible when cached. Drop the exclusion so the curated safetensors always list. * Studio diffusion LoRA: sanitize dots out of adapter aliases The LoRA alias is used as the diffusers PEFT adapter name, and PEFT rejects names containing "." (module name can't contain "."). sanitize_alias kept dots, so a LoRA whose filename carries a version tag (e.g. Qwen-Image-2512-Lightning-8steps-V1.0-bf16) failed to apply with a 400. Replace dots too; the alias stays a valid native <lora:NAME:w> filename stem. Adds regression coverage for internal dots. * Studio Images: clarify the GGUF transformer-quant Advanced control Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode" with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/ FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF, falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only. * Studio Images: list on-device unsloth diffusion models in the picker The Images picker's On Device tab hid every non-GGUF cached repo whenever a task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit and FP8 safetensors) never showed up there. List cached repos that pass the task gate, limited under a filter to unsloth-hosted ones so base repos (which fail the diffusion load trust gate) don't appear only to dead-end on click. Chat behavior is unchanged: the task gate still drops image repos there. * Studio: hide single-file image checkpoints from the chat model picker The chat picker treats a cached repo as an image model, and hides it, only when it ships a diffusers model_index.json. Single-file, ComfyUI, and ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models. Fall back to resolving the repo id against the known diffusion families, the same resolver the Images backend loads from, so these checkpoints are tagged text-to-image and stay in the Images picker only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: add the FLUX.2-dev model family Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline) resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline + Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated, reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror. text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from klein's turbo defaults. Verified live: GGUF load resolves the family + gated base repo and generates a real 1024x1024 image on GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: clearer error for an unsupported diffusion model When a repo id resolves to no diffusion family the load raised 'Could not infer a diffusion family... Pass family_override (z-image)', which points at an unrelated family and doesn't say what is supported. Replace it with a message that lists the supported families (from a new supported_family_names helper) and notes that video models and image models whose diffusers transformer has no single-file loader are not supported. Applies to both the diffusers and native sd.cpp load paths. Also refreshes two stale family-registry comments that still called FLUX.2-dev omitted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add SDXL diffusion family (U-Net pipeline support) SDXL is the first U-Net family in the diffusion backend: its denoiser is pipe.unet (UNet2DConditionModel), not a DiT pipe.transformer, and a single-file .safetensors is the whole pipeline rather than a transformer-only file. The backend previously assumed a DiT transformer everywhere, so add the two hooks a U-Net family needs and register SDXL. DiffusionFamily gains denoiser_attr ("transformer" for DiT, "unet" for SDXL) and single_file_is_pipeline (SDXL loads a single file via pipeline_class.from_single_file with the base repo as config, instead of transformer_class.from_single_file plus a companion assembly). _align_vae_dtype now reads the denoiser generically so img2img and inpaint keep the VAE and U-Net dtypes aligned. The non-GGUF trust gate is extended with a short, exact-match, safetensors-only allowlist of official base repos (the SDXL base/refiner and sdxl-turbo), because SDXL ships only as a full pipeline and has no unsloth-hosted GGUF. Local paths stay trusted as before; a random repo, even one that detects as SDXL, is still rejected. The image-conditioned and ControlNet workflows are the standard SDXL pipelines, built around the resident modules via from_pipe like every other family, so SDXL gets txt2img, img2img, inpaint, outpaint, upscale, LoRA and ControlNet. There is no native sd.cpp mapping yet, so the no-GPU route falls back to diffusers. Frontend catalog gains SDXL Base 1.0 and SDXL Turbo entries with SDXL step/guidance defaults (Turbo: few steps, no CFG; base: ~30 steps, real CFG). Tests: new test_diffusion_sdxl.py (family shape, detection, trust allowlist, model kind, U-Net VAE-dtype alignment, LoRA gate) plus loader-branch tests in test_diffusion_backend.py (pipeline-kind from_pretrained, single-file whole-pipeline from_single_file, allowlist accept/reject). Verified live on GPU: sdxl-turbo loads both as a pipeline and as a single file and generates coherent txt2img + img2img. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove stray async task scratch outputs committed by mistake * ControlNet: reject filesystem-like ids and do not cache a model past an unload race Two review findings on the ControlNet path: - resolve_controlnet's bare-repo fallback accepted any id with a slash, so a path-shaped id (/tmp/x, ../x) reached from_pretrained as a local directory. Restrict the fallback to a strict owner/name HF repo id shape. - _controlnet_pipe now re-checks the cancel event after the blocking from_pretrained: an unload that raced the download had already cleared the caches, so caching the late module would pin it past the unload. * Pipeline prefetch: fetch only the default torch weights A full-pipeline prefetch kept every repo file outside assets/, so an official repo that ships multiple formats (SDXL Base: fp16 variants, ONNX, OpenVINO, Flax, a top-level single-file twin) downloaded tens of GB from_pretrained never loads. Skip non-torch exports and dtype-variant twins in _pipeline_file_downloaded, and drop a component .bin when the same directory carries a picked safetensors weight (diffusers' own preference). * ControlNet: address review findings on the diffusers path - resolve_controlnet enforces catalog family compatibility so a direct API call cannot load a ControlNet built for another family through the wrong pipeline. - Unknown ControlNet ids now surface as a 400 (call site maps FileNotFoundError to ValueError) instead of a generic 500. - strength 0 disables ControlNet entirely, so a no-op selection never pays the download / VRAM cost; the control image is decoded and validated BEFORE the ControlNet is resolved or built, so a malformed image fails fast for the same reason. - ControlNet loads use the base compute dtype (state.dtype is a display string, not a torch.dtype, so it silently fell back to float32) and honor the base offload policy via group offloading instead of forcing the module resident. - Empty/malformed HF token coerced to anonymous access. - Flux Union ControlNet control_mode mapped from the selected control type. - resolve_controlnet drops the unused hf_token/cancel_event params. - ControlNetSpec validates guidance_start <= guidance_end (clean 422). - Images UI ControlNet Select shows its placeholder when nothing is selected. Adds regression tests for family enforcement and the union control-mode map. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA: harden resolution, native tag precedence, and diffusers teardown Address review findings on the LoRA path: - resolve_one: normalise a blank/whitespace hf_token to None (anonymous access) and reject a client-supplied weight file with traversal / absolute path. - resolve_specs: convert FileNotFoundError from an unknown/stale id to ValueError so the route returns 400 instead of a generic 500. - _scan_local: disambiguate local adapters that share a stem (foo.safetensors vs foo.gguf) so each is uniquely addressable. - inject_prompt_tags: the backend-validated weight now wins over a user-typed <lora:ALIAS:...> for a selected adapter; unselected user tags are left alone. - diffusers _apply_loras: reject a .gguf adapter with a clear error before touching the pipe (diffusers loads safetensors only). - _unload_locked: drop the explicit unload_lora_weights() on teardown; the pipe is dropped wholesale (freeing adapters), so the previous call could race an in-flight denoise on the same pipe. - Images page: use a stable LoRA key and clear the selection (not just the options) when the catalog refresh fails. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion: guard trust check against OSError and validate conditioning inputs - _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid characters (or a bare owner/name id) can't raise OSError; treat any failure as not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request still raises the clear FileNotFoundError for a genuinely missing local pick. - generate(): reject mask_image / upscale / reference_images supplied without an input image, and reject reference_images on a family that does not support reference conditioning, instead of silently degrading to txt2img / img2img. * SDXL: reject GGUF up front, skip unused base weights, drop refiner, and harden helpers Addresses review findings on the SDXL family: - Reject a GGUF load for single_file_is_pipeline families (SDXL) in validate_load_request, before the route evicts the current model; SDXL has no transformer-only GGUF variant. - Skip base-repo weight files when a whole-pipeline single file is loaded: from_single_file (config=base) needs only the base config/tokenizer/scheduler, so a local .safetensors no longer triggers a multi-GB base download. - Remove the SDXL refiner from the non-GGUF trust allowlist: it is an img2img-only pipeline but this backend loads every sdxl repo as the base txt2img pipeline. - Normalize a blank/whitespace hf_token to None once in load_pipeline so every load branch degrades to anonymous instead of erroring on a malformed token. - Read the denoiser dtype from a parameter (compile-wrapped modules may lack .dtype) and access state.family.denoiser_attr directly. Adds/updates regression tests for the trust allowlist, GGUF rejection, and base-config filter. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align the VAE to the denoiser's first FLOATING dtype, not its first parameter A GGUF-quantized transformer's leading parameters are packed uint8 storage, so reading next(parameters()).dtype handed nn.Module.to() an integer dtype and every image-conditioned generation on a GGUF model (Qwen-Image-Edit) failed with a 500. Probe the parameters for the first floating dtype, treat an all-integer module as a no-op, and also catch TypeError so an unexpected dtype can never break generation. Regression test included. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address Codex review findings on the image-workflows PR Keep diffusion.py importable without torch: the compile/arch patch modules import torch at module level, so import them lazily at their load/unload call sites instead of at module load. This restores the torchless contract so get_diffusion_backend() works on a CPU/native sd.cpp install. Match family reject keywords and aliases as whole path/name segments, not raw substrings, so an unrelated word like edited, edition, or kontextual no longer misroutes or hides a valid base image model, while supported edit families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same segment matching in the picker task filter. Route FLUX.2-dev native guidance through --guidance like the other FLUX families rather than --cfg-scale. Reject native upscale requests that have no input image. Read image header dimensions and reject over-limit inputs before decoding pixels, so a crafted small-payload image cannot spike memory. Reject an upscale that would shrink the source below its input size. Validate the model_kind against the filename extension before the GPU handoff. Estimate a local diffusers pipeline's size from its on-disk weights so auto memory planning does not skip offload and OOM. Report workflows: [txt2img] from the native backend status so the Create tab stays enabled for a loaded native model. Clamp the outpaint canvas to the backend's 4096px decode limit. Adds regression tests for segment matching and kind/extension validation. * Harden diffusion LoRA handling on the diffusers and native paths Reject LoRA on a torch.compile'd diffusers transformer (Speed=default/max): diffusers requires the adapter loaded before compilation, so applying one to the already-compiled module fails with adapter-key mismatches. The status gate now hides the picker and generate raises a clear message instead. Convert a cancelled Hub LoRA download (RuntimeError Cancelled) to the diffusion cancellation sentinel in resolve_specs, so an unload/superseding load during resolution maps to a 409 instead of a generic server error. Drop weight-0 LoRA rows before the native support gate so a request carrying only disabled adapters stays a no-op on families where native LoRA is unsupported, matching the diffusers path. Reject duplicate LoRA ids in the request model: both apply paths suffix colliding names, so a repeated id would stack the same adapter past its per-adapter weight bound. Strip all user-typed <lora:...> prompt tags on the native path (only the selected adapters are materialized in the managed lora-model-dir, so an unselected tag can never resolve), and restore saved LoRA selections from a gallery recipe so restore reproduces a LoRA image. * Harden ControlNet resolve, gallery metadata, and the control-type picker Check cancellation immediately after a ControlNet from_pretrained and before any device placement, so an unload/eviction that raced the download does not allocate several GB onto the GPU after the load was already cleared. Require a loadable weight or shard index (not just config.json) before a local ControlNet folder is advertised, so an interrupted copy is hidden instead of failing deep in from_pretrained as a generic 500. Do not record a strength-0 ControlNet in the gallery recipe: it is treated as disabled and skipped, so the image is unconditioned and the metadata must not claim a ControlNet was applied. Build the control-type picker from the selected ControlNet's advertised control_types instead of a hardcoded passthrough/canny pair, so a union model with a precomputed depth or pose map sends the correct control_mode. * Address further Codex findings on the image-workflows PR - Persist the actual output image size in the gallery recipe instead of the request sliders: Transform/Inpaint/Edit derive the size from the uploaded image, Extend grows the canvas, and Upscale resizes it, so the sliders recorded (and later restored) the wrong dimensions for those workflows. - Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file name) in validate_load_request, so the unloadable pick fails before chat is evicted rather than deep in from_pretrained. - Only publish an image-conditioned from_pipe wrapper to the shared aux cache when the load is still current: from_pipe runs under the generate lock but not the state lock, so an unload racing its construction could otherwise cache a wrapper over torn-down modules that a later load would reuse. - Verify the Windows CUDA runtime archive checksum before extracting it, like the main sd-cli archive, so a corrupt or tampered runtime is rejected rather than extracted next to the binary. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
|
Daniel Han
|
a0607e41a4
|
Studio: use the Unsloth stable-diffusion.cpp prebuilt mirror for native diffusion (#6787)
* Studio diffusion: cross-platform device policy, fp16 guard, lock split, validate-before-evict Phase 1 of porting the richer diffusion stack onto the image-generation backend. - Add a compartmentalized device/dtype policy module (diffusion_device.py) resolving CUDA/ROCm/XPU/MPS/CPU with capability flags. Keeps the NVIDIA capability-based bf16 choice; ROCm and XPU are isolated; MPS uses bf16 or fp32, never a silent fp16 that renders a black image. - Add a per-family fp16_incompatible flag (Z-Image) and promote a resolved float16 to float32 for those families so they do not produce black images. - Split the backend locks: a generation holds only _generate_lock, so status, unload, and a new load are never blocked by a long denoise. Add per-generation cancellation via callback_on_step_end so an eviction or a superseding load preempts a running generation; a replacement load waits for it to stop before allocating, so two pipelines never sit in VRAM at once. - Validate a load request before the GPU handoff so an unloadable pick never evicts a working chat model, and reject missing local paths up front. - Add CPU-only tests for the device policy, dtype guard, lock split and cancellation, and validate-before-evict, plus a GPU benchmark/regression script (scripts/diffusion_bench.py) measuring latency, peak VRAM, and PSNR against a saved reference. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2A): measured-budget memory planner + offload/VAE policy Add a lean, backend-agnostic memory policy that picks a CPU-offload policy and VAE tiling/slicing from measured free device memory vs the model's estimated resident footprint, then applies it to the built pipeline. auto stays resident when the model fits (byte-identical to the prior resident path), and falls to whole-module offload when tight; fast/balanced/low_vram are explicit overrides. Sequential submodule offload is unreliable for GGUF transformers on diffusers 0.38, so it falls back to whole-module offload and status reports the policy actually engaged. Verified on Z-Image-Turbo Q4_K_M (B200): auto reproduces the resident image with no VRAM/latency regression (PSNR inf); balanced/low_vram cut generation peak VRAM 47.9% (15951 -> 8318 MB) with byte-identical output, at the expected latency cost. 73 prior + 35 new CPU tests pass. * Studio diffusion (Phase 2D): streamed block-level offload + functional VAE tiling Add a streamed 'group' offload tier (diffusers apply_group_offloading, block_level, use_stream) that keeps the transformer flowing through the GPU a few blocks at a time while the text encoder / VAE stay resident, and fix VAE tiling to drive the VAE submodule (pipelines like Z-Image expose enable_tiling on pipe.vae, not the pipeline). apply_memory_plan now returns the (policy, tiling) actually engaged so status never overstates either, and group falls back to whole-module offload when the transformer can't be streamed. Measured on Z-Image (B200), all lossless (PSNR inf vs resident): balanced/group cuts generation peak VRAM 32% (15951 -> 10840 MB) at near-resident speed (2.07 -> 2.99s); low_vram/model cuts it 48% (-> 8318 MB) but is slower (7.99s). Mode names now match that tradeoff: balanced = stream the transformer, low_vram = offload every component. auto picks group when the companions fit resident, else model. 112 CPU tests pass. * Studio diffusion (Phase 5): image quality-vs-quant accuracy harness Add scripts/diffusion_quality.py, the accuracy analogue of the KLD workflow: hold prompt + seed fixed, render a grid with a reference quant (default BF16), then render each candidate quant and measure drift from the reference. Records mean PSNR + SSIM (pure-numpy, no skimage/scipy) and optional CLIP text-alignment + image-similarity (transformers, --clip), plus file size, latency, and peak VRAM, then prints a quality-vs-cost table and recommends the smallest quant within a quality budget. --selftest validates the metrics on synthetic images with no GPU or model. Verified on Z-Image (B200): the table degrades monotonically with quant size (Q8 -> Q4 -> Q2: PSNR 21.7 -> 15.5, SSIM 0.82 -> 0.61), while CLIP-text stays flat (~0.34) -- quantization erodes fine detail far more than prompt adherence. * Studio diffusion (Phase 3): opt-in speed layer (channels_last / compile / TF32) Add a speed_mode knob (off by default, so the render path stays bit-identical): default applies channels_last VAE + regional torch.compile of the denoiser's repeated block where eligible; max also enables TF32 matmul and fused QKV. Regional compile is gated off for the GGUF transformer (dequantises per-op) and for families flagged not compile-friendly (a new supports_torch_compile flag, False for Z-Image), so it activates automatically only once a non-GGUF bf16 transformer is loaded. Speed optims run before placement/offload, per the diffusers composition order. status now reports speed_mode + the optims actually engaged. Verified on Z-Image (B200): default -> ['channels_last'], max -> ['channels_last', 'tf32'], compile correctly skipped for GGUF; generation works in every mode. 121 CPU tests pass. * Studio diffusion (Phase 2B): opt-in fp8 text-encoder layerwise casting Add a text_encoder_fp8 knob that casts the companion text encoder(s) to fp8 (e4m3) storage via diffusers apply_layerwise_casting, upcasting per layer to the bf16 compute dtype while normalisations and embeddings stay full precision. Applied before placement, gated to CUDA + bf16, best-effort (a failure leaves the encoder dense). status reports which encoders were cast. Verified on Z-Image (B200, balanced/group mode where the encoder stays resident): generation peak VRAM dropped 37% (10840 -> 6791 MB, below the lowest-VRAM offload) at near-resident speed. It is a memory-vs-quality tradeoff, not free -- ~20 dB PSNR vs the bf16 encoder, a larger shift than one transformer quant step -- so it is off by default and documented as such, with the Phase 5 harness to size the cost. 127 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2C): NVFP4 text-encoder quant (+ generalise fp8 knob) Generalise the text-encoder precision knob from a fp8 bool to text_encoder_quant (fp8 | nvfp4). nvfp4 quantises the companion text encoder to 4-bit via torchao NVFP4 weight-only (two-level microscaling) on Blackwell's FP4 tensor cores; fp8 stays the broader-hardware path (cc>=8.9). Both are gated, best-effort, and run before placement; status reports the mode actually engaged. This is the lean realisation of GGUF-native text-encoder quant: 4-bit on the encoder without the 3045-line port. Verified on Z-Image (B200, balanced/group where the encoder stays resident), vs the bf16 encoder: nvfp4 cut generation peak VRAM 48% (10840 -> 5593 MB, the lowest TE option, below whole-model offload) at near-fp8 quality (16.4 vs 17.1 dB PSNR), and both quants ran faster than bf16. A memory-vs-quality tradeoff (off by default); size it per model with the Phase 5 quality harness. diffusion_bench gains --text-encoder-quant. 129 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): native stable-diffusion.cpp engine for CPU/Mac Adds the CPU / Apple-Silicon tier of the two-engine strategy, mirroring the chat backend's llama.cpp shell-out. Diffusers stays the default on CUDA / ROCm / XPU; this covers the hardware diffusers serves poorly, consuming the same split GGUF assets Studio already curates. - sd_cpp_args.py: pure sd-cli command builder. Maps the family to its text-encoder flag (Z-Image Qwen3 to --llm, Qwen-Image to --qwen2vl, FLUX.1 CLIP-L + T5), and the diffusers memory policy (none/group/model/sequential) to sd.cpp's offload flags (--offload-to-cpu / --clip-on-cpu / --vae-on-cpu / --vae-tiling / --diffusion-fa), so one user knob drives both engines. - sd_cpp_engine.py: SdCppEngine over a located sd-cli. find_sd_cpp_binary() with the same precedence as the llama finder (env override, then the Studio install root, then in-tree, then PATH), an is_available/version probe, and a one-shot subprocess generate that streams progress and returns the PNG. runtime_env() prepends the binary's directory to the platform library path so a prebuilt's bundled libstable-diffusion.so resolves. select_diffusion_engine() is the pure routing decision (GPU backends to diffusers, CPU/MPS to native when present). - install_sd_cpp_prebuilt.py: resolve + download the per-host prebuilt (macOS-arm64/Metal, Linux x86_64 CPU, Vulkan/ROCm/Windows variants) into the Studio install root. resolve_release_asset() is a pure, unit-tested host-to-asset matrix. - scripts/sd_cpp_smoke.py: end-to-end native generation harness. Tests (CPU-only, subprocess/filesystem stubbed): 49 new across args, engine, routing, runtime env, and the installer resolver. Full diffusion suite 166 passing. Verified on a B200 box: built sd-cli (CUDA) and the prebuilt (CPU) both generate Z-Image-Turbo Q4_K end to end through SdCppEngine: balanced (group offload, 5.0s gen), low_vram (full CPU offload + VAE tiling, 13.4s), and the dynamically-linked CPU prebuilt (50.4s on CPU), all producing coherent images. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 6): img2img / inpaint / edit / LoRA / upscale on the native engine Builds on Phase 4's native stable-diffusion.cpp engine, extending it from text-to-image to the wider feature surface, since sd.cpp supports all of these through the binary already. Pure command-builder additions plus one engine method, so the txt2img path is unchanged. - sd_cpp_args.py: SdCppGenParams gains image-conditioning fields. init_img + strength make a run img2img, adding mask makes it inpaint, ref_images drives FLUX-Kontext / Qwen-Image-Edit style editing (repeated --ref-image), and lora_dir + the <lora:name:weight> prompt syntax select LoRAs. New SdCppUpscaleParams + build_sd_cpp_upscale_command for the ESRGAN upscale run mode (input image + esrgan model, no prompt / text encoders). - sd_cpp_engine.py: the subprocess runner is factored into a shared _run() so generate() (now carrying the conditioning flags) and a new upscale() reuse the same streaming / error / output-check path. - scripts/sd_cpp_smoke.py: --task {txt2img,img2img,upscale} with --init-img / --strength / --upscale-model / --upscale-repeats. Tests: 10 new across the img2img / inpaint / edit / LoRA flag construction, the upscale builder and its validation, and the engine's img2img + upscale paths. Full diffusion suite 176 passing. Verified on a B200 box through SdCppEngine: img2img (Z-Image-Turbo Q4_K, the init image conditioned at strength 0.6, 4.8s) and ESRGAN upscale (512x512 -> 2048x2048 via RealESRGAN_x4plus_anime_6B, 2.7s), both producing coherent images. Video and the diffusers-path feature wiring are deferred. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): accuracy-preserving speed pass Re-review of the diffusion stack (#6675/#6679/#6680) surfaced one real accuracy bug and a dead-on-arrival speed path; this fixes both and adds the lossless / near-lossless wins, all measured on a B200. Correctness: - TF32 global-state leak (fix). speed_mode=max flipped torch.backends.*.allow_tf32 process-wide and never restored them, so a later `off` load silently inherited TF32 and was no longer bit-identical. Added snapshot_backend_flags / restore_backend_flags (TF32 + cudnn.benchmark), captured before the speed layer runs and restored on unload. Verified: load max -> unload -> load off is now byte-identical (PSNR inf) to a fresh off. - sd-cli timeout could hang forever. _run() blocked in `for line in stdout` and only checked the timeout after EOF, so a child stuck in model load / GPU init with no output ignored the timeout. Drained stdout on a reader thread with a wall-clock deadline. Added a silent-hang regression test. Speed (diffusers path), near-lossless, opt-in tiers: - Regional torch.compile now runs on the GGUF transformer. The is_gguf gate (and Z-Image's supports_torch_compile=False) were stale: compile_repeated_blocks compiles and runs ~2.2x faster on the GGUF Z-Image transformer on torch 2.9.1 / diffusers 0.38 (the per-op dequant stays eager, the rest of the block compiles). Measured: off 1.80s -> default 0.82s/gen (+54.7%), PSNR 37.7 dB vs eager -- far above the Q4 quant noise floor (~21 dB), so it does not move output quality. Gate relaxed; default tier delivers it. - cudnn.benchmark added to the default tier (autotunes the fixed-shape VAE convs). - torch.inference_mode() around the pipeline call (lossless, strictly faster than the no_grad diffusers uses internally). Memory path: - VAE tiling (not bit-identical >1MP) restricted to the model/sequential/CPU tiers; the balanced (group) tier keeps exact slicing only, so it is now bit-identical to the resident image (verified PSNR inf) and slightly faster. - Group offload adds non_blocking + record_stream on the CUDA stream path to overlap each block's H2D copy with compute (lossless; gated on the installed diffusers signature so older versions still work). Native (sd.cpp) path: - native_speed_flags: a first-class speed knob (default -> --diffusion-fa, a near-lossless CUDA win that was previously only added on offload tiers; max also -> --diffusion-conv-direct). conv-direct stays opt-in: measured +45% on CUDA, so it is never auto-on. Engine generate() merges it, de-duped against offload flags. Default profile: a GGUF model with no explicit speed_mode now resolves to the `default` profile (resolve_speed_mode), since compile's perturbation sits below the quantisation noise floor and so does not reduce quality versus the dense reference; out of the box a GGUF Z-Image generation drops from 1.80s to 0.81s. Dense models stay `off` / bit-identical, and an explicit speed_mode -- including "off" -- is always honored, so the byte-identical path remains one flag away and is the regression reference. Tooling: scripts/compile_probe.py (eager vs compiled GGUF probe), scripts/ perf_verify.py (the B200 verification above), and diffusion_bench.py gains --speed-mode so the speed tiers are benchmarkable. Tests: 183 passing (was 166); new coverage for the backend-flag snapshot/restore, GGUF compile eligibility, the balanced tiling/slicing split, native_speed_flags + the engine de-dup, and the sd-cli silent-hang timeout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): max tier uses max-autotune-no-cudagraphs + engine/lever benchmarks The opt-in `max` speed tier now compiles the repeated block with mode=max-autotune-no-cudagraphs (dynamic=False) instead of the default mode: Triton autotuning for GEMM/conv-heavier models, gated to the tier where a longer cold compile is acceptable. CUDA-graph modes (reduce-overhead / max-autotune) are deliberately avoided -- both crash on the regionally-compiled block (its static output buffer is overwritten across denoise steps), measured. Adds two reproducible benchmarks used to validate the optimization research: - scripts/compare_engines.py: PyTorch (diffusers GGUF) vs native sd.cpp head-to-head. - scripts/leverage_probe.py: coordinate_descent_tuning + FirstBlockCache probes. Measured on B200 (Z-Image Q4_K_M, 1024px, 8 steps): default compile 0.80s/gen; coordinate_descent_tuning 0.79s (within noise, already covered by max-autotune); FirstBlockCache does not run on Z-Image (diffusers 0.38 block-detection / Dynamo). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source) Add an opt-in transformer_quant mode that loads the dense bf16 transformer and torchao-quantises it onto the low-precision tensor cores, instead of the GGUF transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's 0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor. GGUF+compile stays the low-memory default and the fallback. The mode is gated on CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB); any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe (Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips the tiny projections that crash int8's torch._int_mm. New module mirrors diffusion_precision.py; quant runs before compile before placement. 184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py is the standalone torchao lever probe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): consumer-GPU tuning - lock fp8 fast accumulate, prefer fp8 over mxfp8, reject 2:4 sparsity Consumer Blackwell halves tensor-core throughput on FP32 accumulate (fp8 419 vs 838 TFLOPS with FP16 accumulate; bf16 209), so: - fp8 config locks use_fast_accum=True (Float8MMConfig). torchao already defaults it on; pinning it guards consumer cards against a default change. On B200 it is identical speed and slightly better quality (LPIPS 0.050 vs 0.091). - the Blackwell auto ladder prefers fp8 over mxfp8 (measured faster + more accurate). 2:4 semi-structured sparsity evaluated and rejected (scripts/sparse_accum_probe.py): 2:4 magnitude-prune + fp8 gives LPIPS 0.858 (broken image) with no fine-tune, the cuSPARSELt kernel errors on torch 2.9, and it does not compose with torch.compile (our main ~2x). Documented as a dead end, not shipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add fp8 fast-accum overflow verification probe scripts/fp8_overflow_check.py hooks every quantised linear during a real Z-Image generation and reports max-abs + non-finite counts for use_fast_accum True vs False. Confirms fast accumulation is an accumulation-precision knob, not an overflow one: across 276 linears, including Z-Image's ~1.0e6 activation peaks (which overflow FP16), 0 non-finite elements and identical max-abs for both modes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40; GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it. Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s), no overflow, quality below the quant noise floor. So the default leans to accuracy on data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force) lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off). 187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override threading). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add NVFP4 probe documenting it is not yet a win on torch 2.9 scripts/nvfp4_probe.py measures NVFP4 via torchao on the real Z-Image transformer. Finding (B200, 1024px/8 steps): NVFP4 is a torchao feature and DOES run with use_triton_kernel=False (the default triton path needs the missing MSLK library), but only at bf16-compile rate (0.667s vs fp8 0.592s) -- it dequantises FP4->bf16 rather than using the FP4 tensor cores. The real FP4 speedup needs MSLK or torch>=2.11 + torchao's CUTLASS FP4 GEMM. The smoke probe (default triton=True) already keeps NVFP4 out of auto on this env, so auto correctly stays on fp8; NVFP4 activates automatically once fast. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): prefer fp8 over nvfp4 in Blackwell auto ladder Validated NVFP4 on torch 2.11 + torchao CUTLASS FP4 in an isolated env. The FP4 tensor-core GEMM is genuinely active there (a 16384^3 GEMM hits ~3826 TFLOPS, 2.52x bf16 and 1.37x fp8), but it only beats fp8 on very large GEMMs. At the diffusion transformer's shapes (hidden ~3072, MLP ~12288, M~4096) NVFP4 is both slower (0.81x fp8 end to end on Z-Image 1024px) and less accurate (LPIPS 0.166 vs fp8's 0.044). Reorder the Blackwell auto ladder to fp8 before nvfp4 so auto is correct even on a future MSLK-equipped box; nvfp4 stays an explicit opt-in. Add scripts/nvfp4_t211_probe.py (extension diagnostics + GEMM micro + end-to-end). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): pre-quantized transformer loading The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs 13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline (scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True) the quantized weights, so the dense bf16 never touches the GPU. Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same torchao config + min_features filter the runtime path uses, applied ahead of time. New core/inference/diffusion_prequant.py (resolve_prequant_source + load_prequantized_transformer, best-effort, lazy imports). diffusion.py _load_dense_quant_pipeline tries the pre-quant source first and falls back to the dense materialise+quantise path, then to GGUF, so the default is unchanged. DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for the resolver, the meta-init+assign loader, and the backend branch selection + fallbacks; GPU verification via scripts/verify_prequant_backend.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): attention-backend selection Add a selectable attention kernel via the diffusers set_attention_backend dispatcher. Attention is memory-bandwidth bound, so a better kernel is an end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV matmuls torchao never touches) and composes with torch.compile. auto picks the best exact backend for the device: cuDNN fused attention (_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default (below the compile/quant noise floor); native SDPA elsewhere and when speed=off (so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/ xformers/aiter are honored, and an unavailable kernel falls back to the default rather than failing the load. New core/inference/diffusion_attention.py (normalize + per-device select + apply, best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline; attention_backend threads through begin_load / load_pipeline / status like the other load knobs. New request field attention_backend + status field. Hermetic CPU tests for normalize / select policy / apply fallback, plus route threading + 422. Measured via scripts/perf_levers_probe.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the arch tier to put int8 first on a consumer/workstation GPU (detected by the existing _is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first. Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation (-> int8) and data-center Ada/Hopper/Blackwell (-> fp8). * Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across denoise steps a DiT's output settles, so once the first block's residual barely changes the remaining blocks are skipped and their cached output reused. diffusers ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the standalone apply_first_block_cache hook as a fallback). Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 -> 2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar. OFF by default and a per-load opt-in: the win scales with step count, so it is for many-step models (Flux / Qwen-Image) and pointless for few-step distilled models (e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction of the trajectory. It composes with regional compile only with fullgraph=False (the cache's per-step decision is a torch.compiler.disable graph break), which the speed layer now switches to automatically when a cache is engaged. Best-effort: a model whose block signature the hook does not recognise is caught and the load proceeds uncached. - new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache (enable_cache / apply_first_block_cache fallback; threshold auto-raised for a quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import). - diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph when a cache is engaged. - diffusion.py: apply_step_cache before compile; thread transformer_cache / transformer_cache_threshold through begin_load -> load_pipeline and report the engaged mode in status(). - models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and transformer_cache_threshold request fields, engaged mode in the status response. - hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold selection, and best-effort failure handling, plus route threading + validation. - scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM / LPIPS vs the compiled no-cache baseline). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears) The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with 'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2 *_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing min_features filter did not exclude them. Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed, guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers (M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and quantises these layers fine, so the exclusion is int8-only. Sequence embedders (context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note 'context_embedder' contains the substring 'text_embed', which is why the token is the specific 'time_text_embed', not 'text_embed'. Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s (5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already working) are unchanged. - diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes exclude_name_tokens; quantize_transformer passes it for int8 only. - hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps attention / FFN / sequence-embedder linears), while fp8 keeps them. - scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's Linear layers and derive the exclusion list. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears) The prequant-checkpoint builder applied the dense quant filter without the int8-only M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8 checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now produces a working checkpoint on every supported model, giving int8 (the consumer-preferred scheme) the same ~2x load-VRAM and download reduction fp8 already had. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the native stable-diffusion.cpp engine instead of diffusers, with diffusers as the guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM. - diffusion_engine_router: centralised engine selection (built on the existing select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason. - sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface backed by sd-cli, with lazy binary install, registry-driven asset fetch, step-progress parsing, and cancellation. - diffusion_families: per-family single-file VAE + text-encoder asset mapping. - sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled). - routes/inference + gpu_arbiter: drive the active engine via the router; the API now reports the active engine and any fallback reason. - tests for the backend, router, route selection, and cancellation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler Address review feedback on #6724: - engine router: unload the engine being deactivated on a switch, so the old model is not left resident-but-unreachable (the evictor only targets the active engine). - generate route: sd.cpp execution errors (nonzero exit / timeout / missing output) now map to 500, not 409 (which only means not-loaded / cancelled). - native batch: return per-image seeds and persist the actual seed for each image so every batch image is reproducible. - Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the stable-diffusion.cpp docs; other families keep sd-cli defaults. - honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload offload flags on the native load instead of hardcoding them off. - fail the load when the sd-cli binary is present but not runnable (version() now returns None on exec error / nonzero exit). - size estimate: only treat the transformer asset as a possible local path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in load_prequantized_transformer ends in torch.load(weights_only=False), which executes arbitrary code from the pickle. The transformer_prequant_path load-request field reached that unpickle for any local file an authenticated caller named, so a request could trigger remote code execution. Refuse the source.kind=='path' branch unless the operator sets UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted and unaffected. Document the requirement on the API field and add gate tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa - apply_attention_backend now restores the native default when no backend is requested or a kernel fails. diffusers keeps a process-wide active attention backend that set_attention_backend updates, and a fresh transformer's processors follow it, so a load that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile load pinned, breaking the bit-identical/off guarantee. - select_attention_backend drops flash3/flash4 up front when the CUDA capability is below Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit request on the wrong card set fine then crashed mid-generation; it now falls back to native. - Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a valid alias of native) is accepted instead of 422-rejected by Pydantic. - Drop the dead replace('-','_') normalization (no alias uses dashes/underscores). - perf_levers_probe.py output dir is now relative to the script, not a hardcoded path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF - apply_step_cache now engages only via the transformer's native enable_cache (the diffusers CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model now runs uncached per the best-effort contract. - GGUF transformers are quantized (the default Studio load path), so they now use the higher quantized FBCache threshold when the caller leaves it unset, instead of the dense default that could keep the cache from triggering. - fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so fullgraph=True failed warmup and silently measured an eager cached run); output dir is now relative to the script, not a hardcoded path. * Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder _is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate. Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0). * Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just TypeError on older torchao), which would break the mxfp8 config helper instead of falling back to the default. Catch both so the fallback is robust. quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds CUDA memory during the per-row VRAM probe; output dir relative to the script. * Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load - snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the whole snapshot. restore_backend_flags restores each flag independently so one failure can't leave the others leaked process-wide. - load_pipeline restores the flags (and clears the GPU cache) when the build fails after apply_speed_optims mutated the process-wide flags but before _state captured them for unload to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and contaminated later off generations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs without producing output (or without closing stdout) would never reach proc.wait and the wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the PROCESS, so the main thread always enforces the timeout and kills a hung process (which closes the pipe and ends the reader). Add a test that times out even when stdout blocks, and make the no-binary test hermetic so a host-installed sd-cli can't leak in. * Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn The filter callback can be invoked without a module name, so fqn.lower() would raise AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens, so the linear is kept) instead of crashing the quantise pass. * Studio diffusion (Phase 16) review fixes: native engine robustness - sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64); a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds aliased to the same image. Random seeds stay 53-bit (JS-safe). - sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub fall back to anonymous instead of failing auth on a blank token. - sd_cpp_backend: a superseding load now cancels the in-flight generation, so the old sd-cli can no longer return/persist an image from the previous model. - diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a slow 10+ GB free / CUDA sync does not block engine selection. - diffusion_engine_router: probe sd-cli runnability (version()) before committing to native, so a present-but-unrunnable binary falls back to diffusers at selection. - diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a CPU-only install can still reach the native sd.cpp engine instead of failing load. - tests updated for the runnability probe + a not-runnable fallback case. * Studio diffusion (Phase 9) review fixes: prequant safety + validation - SECURITY: a request-supplied local pre-quant path is now unpickled only when it resolves inside an operator-configured ALLOWLIST of directories (UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in, once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on any path a load request named (arbitrary code execution). realpath() blocks symlink escapes; a bare on/off toggle is no longer a wildcard. - Validate the checkpoint's min_features against the runtime Linear filter, so a checkpoint that quantised a different layer set is rejected instead of silently loading a model that mismatches the dense path while reporting the same scheme. - Tolerant base_model_id compare (exact or same final path/repo segment), so a local path or fork of the canonical base is accepted instead of falling back to dense. - _has_meta_tensors uses any(chain(...)) (no intermediate lists). - prequant verify/probe scripts use repo-relative paths (+ env overrides), not the author's absolute /mnt paths. - tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts - diffusion_memory: when group offload is unavailable and the plan falls back to whole-module offload, enable VAE tiling (the group plan left it off, but the fallback is the low-VRAM path where the decode spike can OOM). Covers both the group and sequential fallback branches. - perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a balanced bit-identity regression actually fails the check instead of exiting 0. - compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and the load-progress poll has a 30 min deadline instead of looping forever on a hang. - test for the group->model fallback enabling VAE tiling. * Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path - diffusion: a torchao-quantized transformer is committed only compiled. A dense model resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF it replaced), so when transformer_quant engaged and speed resolved to off, promote to default (regional compile); warn loudly if compile still does not engage. - diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK); otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF. - nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path). - test asserts the eager-quant -> default-compile promotion. * Studio diffusion (Phase 10) review fixes: attention gating + probe isolation - diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation, so auto now stays on native SDPA there. - diffusion_attention: _active_attention_backend handles get_active_backend() returning an enum/None (not a tuple); the old unpack always raised and was swallowed, so the native-restore short-circuit never fired. - perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul so the inductor_flags variant doesn't leak into later compiled rows. - tests for the SM80 cuDNN gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening - install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket); extract through a per-member containment check (Zip-Slip guard); expanduser the --install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch the separately-published cudart runtime DLL archive so sd-cli.exe can start. - sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie. - tests: Zip-Slip rejection, normal extraction, studio-home discovery. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of the literal --output path. SdCppEngine.generate checked only the literal path, so a batch generation would exit 0 and then raise 'no image' (or return a stale file). generate now returns the literal path when present and otherwise falls back to the numbered siblings; single-image behavior is unchanged. Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected without error. * Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats Codex review on the native engine arg builder: - build_sd_cpp_command emitted --width/--height unconditionally, so an img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of the input. width/height are now Optional (None = unset): an image-conditioned run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives the size from the input image (set_width_and_height_if_unset); a plain txt2img run with unset dims keeps the prior 1024x1024 default; explicit dims are always honored. width/height are read only by the builder, so the type change is local. - build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...) that silently swallowed repeats=0 into sd-cli's default of one pass, turning an explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError and emits the flag for any explicit value != 1. Tests: img2img unset dims omit width/height (init_img and ref_images), explicit dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at the default. (Two pre-existing binary-discovery tests fail only because a real sd-cli is installed in this dev environment; unrelated to this change.) * Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc Codex review: the transformer_prequant_path field description still told operators to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the prior security fix made that variable a directory allowlist -- _allowed_prequant_roots deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator following the documented =1 would have every transformer_prequant_path request silently refused. The description now states it must name one or more allowlisted directories and that a bare on/off value is not accepted. Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does not say =1, and describes an allowlist/directory (guards against doc drift). * Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset Codex review on attention-backend selection: - Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first generation with no fallback. select_attention_backend now applies _cudnn_attention_supported() to an explicit cuDNN request too. - flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3 is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a (min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+. - apply_attention_backend's success path left diffusers' process-wide active backend pinned to the kernel it set; a later component whose processors are unconfigured (backend None) would inherit it. It now resets the global registry to native after a successful per-transformer set (the transformer keeps its own backend), best-effort. Also fixed _active_attention_backend: get_active_backend() returns a (name, fn) tuple, so the prior code stringified the tuple and never matched a name, defeating the native-restore short-circuit. Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on SM90; global registry reset after a successful set; _active_attention_backend reads the tuple return. * Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300 (Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So _is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they are treated as data-center class and keep the intended fp8-first behavior. Test: extends the datacenter parametrize with 'NVIDIA B300' and 'NVIDIA GH200 480GB' (now _is_consumer_gpu False). * Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder Codex review: the M=1 modulation/embedder exclusion was wired only into the dense runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint quantised the AdaLN modulation and conditioning-embedder linears, and loading it via transformer_prequant_path (the load path only loads already-quantised tensors, it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes for the runtime path. Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 -> the M=1 exclusion, every other scheme -> none) and use it in both the runtime quantiser and the builder, so a prequant artifact's quantised-layer set always matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion). Test: int8_exclude_name_tokens returns the exclusion for int8 and () for fp8/nvfp4/mxfp8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race Codex review on the native-engine routing: - The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat) unconditionally after engine selection. A native sd.cpp load on a pure-CPU host never touches the GPU, so that needlessly tore down the resident chat model. The handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it. - sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when _run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so the setting was unverifiable. status now derives them from state.offload_flags (still 'none' on CPU, where the flags are empty). - _run_load committed the new state without cancelling/waiting on a generation that started during the (slow) asset download, so a stale sd-cli run against the OLD model could finish afterward and persist an image from the previous model once the new load reported ready. The commit now signals the in-flight cancel and waits on _generate_lock before swapping _state (taken only at commit, so the download never serialises against generation), mirroring the diffusers load path. Tests: CPU native load skips the arbiter while a GPU native load takes it; status reports offload active when flags are set; _run_load cancels and waits for an in-flight generation before committing. * Studio diffusion (Phase 14) review round 2: align helper name with the stack Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the identical helper already present higher in the diffusion stack (Phase 16). The helper definition, the runtime quantiser call, and the offline builder are now byte-identical to that version, so the two branches no longer introduce a divergent name for the same single-source-of-truth and the stack merges without a conflict on this fix. No behavior change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: eager patches + torch.compile cache speed phase Adds the opt-in speed path for the GGUF diffusion transformer behind a selectable speed mode (default off, so output is unchanged until a profile is chosen): - diffusion_eager_patches.py: shared eager fast-paths (channels_last, attention/backend selection, fused norms and QKV) installed at load and rolled back on unload or failed load. - diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent torch.compile cache and the GGUF-transformer compile wiring. - diffusion_arch_patches.py: architecture-specific patches. - diffusion_patch_backend.py: shared install/restore plumbing. - diffusion_speed.py: speed-profile planning. Tests for each module plus the benchmarking and probe scripts used to measure speed, memory, and accuracy of the path. * Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI Backend: - Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file fp8 transformers, gated to the unsloth org plus a curated allowlist. - Image-conditioned workflows built with Pipeline.from_pipe so they reuse the loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint, outpaint, and a hires-fix upscale pass. - Instruction editing as its own family kind (Qwen-Image-Edit-2511, FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and multi-reference) plus klein inpaint. - Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the decoded image size and cap upscale output to avoid OOM on large inputs. - Fixes: from_pipe defaulting to a float32 recast that crashed torchao quantized transformers; image-conditioned calls forcing the slider size onto the input image. Native sd.cpp engine rejects image-conditioned and reference requests it cannot serve. Frontend: - Redesigned Images page with capability-gated workflow tabs (Create, Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor, client-side outpaint, and a multi-reference picker. - Advanced options moved to a right-docked panel mirroring Chat: closed by default, toggled by a single fixed top-bar button that stays in place. sd.cpp installer: pin the release, verify each download's sha256, add a download timeout, and make the source repo configurable for a future mirror. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: correct the Advanced panel comment (closed by default, fixed toggle) * Studio diffusion: LoRA adapters for the Images workflow Add community LoRA support across both diffusion backends, the single biggest step toward broad image-workflow coverage. Backend - New shared module core/inference/diffusion_lora.py: adapter discovery (local scan + curated catalog + owner/name[:file] Hub refs), download via hf_hub_download_with_xet_fallback, alias sanitization, native managed-dir materialization with collision-broken aliases, prompt-tag injection (deduped against user-typed tags), and a supports_lora gate. - Native sd-cli: resolve + materialize selected LoRAs into a per-run managed dir, inject <lora:ALIAS:w> tags, pass --lora-model-dir with --lora-apply-mode auto. The arg builder already emitted these flags. - Diffusers: non-fused load_lora_weights + set_adapters manager, tracked on the pipe so an unchanged selection is a no-op and a model swap resets; cleared on unload. Never fuses (breaks quantized transformers and blocks live weight tweaks). - Gated off where unsupported: torchao fp8/int8 dense, GGUF-via-diffusers, and native Qwen-Image (no LoRA name-conversion branch upstream). - Request contract: optional loras on DiffusionGenerateRequest; empty or omitted is identical to today. supports_lora surfaced in status; chosen LoRAs persisted in gallery recipe metadata. - New GET /api/models/diffusion-loras for the picker (family-filtered). Frontend - Repeatable multi-LoRA picker (adapter select + weight slider 0..2 + remove), gated by the loaded model's supports_lora and family, max 8. Tests - New test_diffusion_lora.py (14): helpers, request validation, native tag/dir wiring, diffusers set_adapters manager, supports_lora matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: ControlNet for the Images workflow (diffusers) Add ControlNet conditioning, the #2 most-used diffusion workflow after LoRA, on the diffusers backend for the families with ControlNet pipelines (FLUX.1 and Qwen-Image), with Union models as the default picks. Backend - New core/inference/diffusion_controlnet.py: family-gated discovery (curated Union models + local dirs + bare owner/name repos), resolution to a loadable repo/dir, control-image preprocessing (passthrough + a dependency-free canny edge map), and a supports_controlnet gate. - diffusion.py: a ControlNet manager parallel to the LoRA one. Loads the (small) ControlNet model once via from_pretrained (cached by id) and builds the family's ControlNet pipeline via Pipeline.from_pipe(base, controlnet=model), reusing the resident base modules at their loaded dtype (no reload, no recast). Passes the control image + conditioning scale + guidance start/end at generate time; cleared on unload. - Families: FLUX.1 -> FluxControlNetPipeline/Model, Qwen-Image -> QwenImageControlNetPipeline/Model. Others declare none (gated off). - Gated off for the native engine, GGUF-via-diffusers, and torchao fp8/int8 dense (same rule as LoRA). v1 conditions txt2img only. - Request contract: optional controlnet on DiffusionGenerateRequest; supports_controlnet in status; the choice persisted in gallery meta. - New GET /api/models/diffusion-controlnets for the picker. Frontend - A ControlNet control in the Images rail (model select + control-image upload + control-type select + strength slider), gated by the loaded model's supports_controlnet + family, shown for text-to-image. Tests - New test_diffusion_controlnet.py (10): discovery/resolve/preprocess/gate helpers, request validation, family wiring, and the diffusers pipe manager (loads once, caches, from_pipe with controlnet, rejects unsupported families). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio ControlNet: show the picker on the Create tab (workflow id is 'create', not 'txt2img') The ControlNet control gated on workflow === 'txt2img', but the Images workflow tab ids are create/transform/inpaint/extend/upscale/reference/edit -- there is no 'txt2img'. So the picker never rendered even with a ControlNet-capable model loaded. Gate on 'create' (the text-to-image tab) for both the picker and the request wiring. Found via a live Playwright capture of the running Studio. * Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a stray "gguf" tag on the Hub but ship no .gguf files. The model search classified them as GGUF from the bare tag, so the picker rendered the GGUF variant expander, which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata (populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected. * Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline load when the id is unsloth-hosted or on-device (the backend infers the family + base repo and gates loads to unsloth/* or local paths), and show a clear message otherwise instead of silently ignoring the click. Curated and GGUF paths are unchanged. * Studio Images: keep curated safetensors models in Recommended after download The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet), so they vanished from the picker after the first load and could only be found by typing an exact search. The row already renders a downloaded badge, matching how GGUF Recommended rows stay visible when cached. Drop the exclusion so the curated safetensors always list. * Studio diffusion LoRA: sanitize dots out of adapter aliases The LoRA alias is used as the diffusers PEFT adapter name, and PEFT rejects names containing "." (module name can't contain "."). sanitize_alias kept dots, so a LoRA whose filename carries a version tag (e.g. Qwen-Image-2512-Lightning-8steps-V1.0-bf16) failed to apply with a 400. Replace dots too; the alias stays a valid native <lora:NAME:w> filename stem. Adds regression coverage for internal dots. * Studio Images: clarify the GGUF transformer-quant Advanced control Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode" with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/ FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF, falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only. * Studio Images: list on-device unsloth diffusion models in the picker The Images picker's On Device tab hid every non-GGUF cached repo whenever a task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit and FP8 safetensors) never showed up there. List cached repos that pass the task gate, limited under a filter to unsloth-hosted ones so base repos (which fail the diffusion load trust gate) don't appear only to dead-end on click. Chat behavior is unchanged: the task gate still drops image repos there. * Studio: hide single-file image checkpoints from the chat model picker The chat picker treats a cached repo as an image model, and hides it, only when it ships a diffusers model_index.json. Single-file, ComfyUI, and ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models. Fall back to resolving the repo id against the known diffusion families, the same resolver the Images backend loads from, so these checkpoints are tagged text-to-image and stay in the Images picker only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: add the FLUX.2-dev model family Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline) resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline + Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated, reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror. text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from klein's turbo defaults. Verified live: GGUF load resolves the family + gated base repo and generates a real 1024x1024 image on GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: clearer error for an unsupported diffusion model When a repo id resolves to no diffusion family the load raised 'Could not infer a diffusion family... Pass family_override (z-image)', which points at an unrelated family and doesn't say what is supported. Replace it with a message that lists the supported families (from a new supported_family_names helper) and notes that video models and image models whose diffusers transformer has no single-file loader are not supported. Applies to both the diffusers and native sd.cpp load paths. Also refreshes two stale family-registry comments that still called FLUX.2-dev omitted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: use the Unsloth stable-diffusion.cpp prebuilt mirror for native diffusion Point the native sd-cli/sd-server installer at unslothai/stable-diffusion.cpp, which builds and publishes our own CPU and Apple prebuilts (the same way unslothai/llama.cpp ships prebuilts). GPU hosts run diffusers and never reach the native installer, so the mirror only needs CPU (Linux/WSL/Windows) and Apple (Metal) builds. DEFAULT_REPO now points at unslothai/stable-diffusion.cpp and DEFAULT_TAG at master-741-484baa4. A leejet upstream fallback is added: if the mirror cannot serve a host (release missing, or a host we do not build) and the default repo is in use, install resolves against leejet so native install still works. An explicit UNSLOTH_SD_CPP_REPO override is respected as-is (no substitution). Tests cover the mirror asset matrix for every CPU/Apple host, the default-repo assertion, and the mirror-to-upstream fallback path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove stray async task scratch outputs committed by mistake * ControlNet: reject filesystem-like ids and do not cache a model past an unload race Two review findings on the ControlNet path: - resolve_controlnet's bare-repo fallback accepted any id with a slash, so a path-shaped id (/tmp/x, ../x) reached from_pretrained as a local directory. Restrict the fallback to a strict owner/name HF repo id shape. - _controlnet_pipe now re-checks the cancel event after the blocking from_pretrained: an unload that raced the download had already cleared the caches, so caching the late module would pin it past the unload. * sd-cli install: honor explicit accelerator/repo pins and route --print-asset through fallback Five review findings on the mirror install path: - resolve_release_asset returned a CPU build for an explicit --accelerator cuda/vulkan/rocm request when only a CPU asset existed (Windows and Linux), so the mirror CPU build was installed instead of the upstream accelerated build. An explicit GPU accelerator with no matching asset now returns None so the upstream fallback runs. - The upstream fallback keyed on the repo string, so setting UNSLOTH_SD_CPP_REPO to the default value still fell back to leejet. Gate the fallback on the env var being unset, so an explicit pin gets exactly that repo. - A pinned tag missing on the mirror but present upstream fell through to the mirror's unpinned latest. Try the exact pin on every repo before any repo's latest, so a pinned upstream build wins over an unpinned mirror-latest. - --print-asset used a mirror-only fetch and reported no match for hosts the mirror does not build. Route it through the same primary/upstream resolution. - Tidy the fetch-failed log line (own line, no trailing separator) and guard release.get(assets) against None. Adds regression tests for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ControlNet: address review findings on the diffusers path - resolve_controlnet enforces catalog family compatibility so a direct API call cannot load a ControlNet built for another family through the wrong pipeline. - Unknown ControlNet ids now surface as a 400 (call site maps FileNotFoundError to ValueError) instead of a generic 500. - strength 0 disables ControlNet entirely, so a no-op selection never pays the download / VRAM cost; the control image is decoded and validated BEFORE the ControlNet is resolved or built, so a malformed image fails fast for the same reason. - ControlNet loads use the base compute dtype (state.dtype is a display string, not a torch.dtype, so it silently fell back to float32) and honor the base offload policy via group offloading instead of forcing the module resident. - Empty/malformed HF token coerced to anonymous access. - Flux Union ControlNet control_mode mapped from the selected control type. - resolve_controlnet drops the unused hf_token/cancel_event params. - ControlNetSpec validates guidance_start <= guidance_end (clean 422). - Images UI ControlNet Select shows its placeholder when nothing is selected. Adds regression tests for family enforcement and the union control-mode map. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA: harden resolution, native tag precedence, and diffusers teardown Address review findings on the LoRA path: - resolve_one: normalise a blank/whitespace hf_token to None (anonymous access) and reject a client-supplied weight file with traversal / absolute path. - resolve_specs: convert FileNotFoundError from an unknown/stale id to ValueError so the route returns 400 instead of a generic 500. - _scan_local: disambiguate local adapters that share a stem (foo.safetensors vs foo.gguf) so each is uniquely addressable. - inject_prompt_tags: the backend-validated weight now wins over a user-typed <lora:ALIAS:...> for a selected adapter; unselected user tags are left alone. - diffusers _apply_loras: reject a .gguf adapter with a clear error before touching the pipe (diffusers loads safetensors only). - _unload_locked: drop the explicit unload_lora_weights() on teardown; the pipe is dropped wholesale (freeing adapters), so the previous call could race an in-flight denoise on the same pipe. - Images page: use a stable LoRA key and clear the selection (not just the options) when the catalog refresh fails. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion: guard trust check against OSError and validate conditioning inputs - _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid characters (or a bare owner/name id) can't raise OSError; treat any failure as not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request still raises the clear FileNotFoundError for a genuinely missing local pick. - generate(): reject mask_image / upscale / reference_images supplied without an input image, and reject reference_images on a family that does not support reference conditioning, instead of silently degrading to txt2img / img2img. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address Codex review findings on the image-workflows PR Keep diffusion.py importable without torch: the compile/arch patch modules import torch at module level, so import them lazily at their load/unload call sites instead of at module load. This restores the torchless contract so get_diffusion_backend() works on a CPU/native sd.cpp install. Match family reject keywords and aliases as whole path/name segments, not raw substrings, so an unrelated word like edited, edition, or kontextual no longer misroutes or hides a valid base image model, while supported edit families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same segment matching in the picker task filter. Route FLUX.2-dev native guidance through --guidance like the other FLUX families rather than --cfg-scale. Reject native upscale requests that have no input image. Read image header dimensions and reject over-limit inputs before decoding pixels, so a crafted small-payload image cannot spike memory. Reject an upscale that would shrink the source below its input size. Validate the model_kind against the filename extension before the GPU handoff. Estimate a local diffusers pipeline's size from its on-disk weights so auto memory planning does not skip offload and OOM. Report workflows: [txt2img] from the native backend status so the Create tab stays enabled for a loaded native model. Clamp the outpaint canvas to the backend's 4096px decode limit. Adds regression tests for segment matching and kind/extension validation. * Harden diffusion LoRA handling on the diffusers and native paths Reject LoRA on a torch.compile'd diffusers transformer (Speed=default/max): diffusers requires the adapter loaded before compilation, so applying one to the already-compiled module fails with adapter-key mismatches. The status gate now hides the picker and generate raises a clear message instead. Convert a cancelled Hub LoRA download (RuntimeError Cancelled) to the diffusion cancellation sentinel in resolve_specs, so an unload/superseding load during resolution maps to a 409 instead of a generic server error. Drop weight-0 LoRA rows before the native support gate so a request carrying only disabled adapters stays a no-op on families where native LoRA is unsupported, matching the diffusers path. Reject duplicate LoRA ids in the request model: both apply paths suffix colliding names, so a repeated id would stack the same adapter past its per-adapter weight bound. Strip all user-typed <lora:...> prompt tags on the native path (only the selected adapters are materialized in the managed lora-model-dir, so an unselected tag can never resolve), and restore saved LoRA selections from a gallery recipe so restore reproduces a LoRA image. * Harden ControlNet resolve, gallery metadata, and the control-type picker Check cancellation immediately after a ControlNet from_pretrained and before any device placement, so an unload/eviction that raced the download does not allocate several GB onto the GPU after the load was already cleared. Require a loadable weight or shard index (not just config.json) before a local ControlNet folder is advertised, so an interrupted copy is hidden instead of failing deep in from_pretrained as a generic 500. Do not record a strength-0 ControlNet in the gallery recipe: it is treated as disabled and skipped, so the image is unconditioned and the metadata must not claim a ControlNet was applied. Build the control-type picker from the selected ControlNet's advertised control_types instead of a hardcoded passthrough/canny pair, so a union model with a precomputed depth or pose map sends the correct control_mode. * Route the sd.cpp repo-fallback diagnostic to stderr --print-asset documents its stdout as the resolved asset name only, but the upstream repo-fallback path printed 'falling back to ...' to stdout before the asset line, so a script consuming the output as a single asset name parsed the log instead. Send the diagnostic to stderr; the install note stays on stdout. * Address further Codex findings on the image-workflows PR - Persist the actual output image size in the gallery recipe instead of the request sliders: Transform/Inpaint/Edit derive the size from the uploaded image, Extend grows the canvas, and Upscale resizes it, so the sliders recorded (and later restored) the wrong dimensions for those workflows. - Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file name) in validate_load_request, so the unloadable pick fails before chat is evicted rather than deep in from_pretrained. - Only publish an image-conditioned from_pipe wrapper to the shared aux cache when the load is still current: from_pipe runs under the generate lock but not the state lock, so an unload racing its construction could otherwise cache a wrapper over torn-down modules that a later load would reuse. - Verify the Windows CUDA runtime archive checksum before extracting it, like the main sd-cli archive, so a corrupt or tampered runtime is rejected rather than extracted next to the binary. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
|
Daniel Han
|
66281dd114
|
Studio diffusion: ControlNet for the Images workflow (diffusers) (#6773)
* Studio diffusion: cross-platform device policy, fp16 guard, lock split, validate-before-evict Phase 1 of porting the richer diffusion stack onto the image-generation backend. - Add a compartmentalized device/dtype policy module (diffusion_device.py) resolving CUDA/ROCm/XPU/MPS/CPU with capability flags. Keeps the NVIDIA capability-based bf16 choice; ROCm and XPU are isolated; MPS uses bf16 or fp32, never a silent fp16 that renders a black image. - Add a per-family fp16_incompatible flag (Z-Image) and promote a resolved float16 to float32 for those families so they do not produce black images. - Split the backend locks: a generation holds only _generate_lock, so status, unload, and a new load are never blocked by a long denoise. Add per-generation cancellation via callback_on_step_end so an eviction or a superseding load preempts a running generation; a replacement load waits for it to stop before allocating, so two pipelines never sit in VRAM at once. - Validate a load request before the GPU handoff so an unloadable pick never evicts a working chat model, and reject missing local paths up front. - Add CPU-only tests for the device policy, dtype guard, lock split and cancellation, and validate-before-evict, plus a GPU benchmark/regression script (scripts/diffusion_bench.py) measuring latency, peak VRAM, and PSNR against a saved reference. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2A): measured-budget memory planner + offload/VAE policy Add a lean, backend-agnostic memory policy that picks a CPU-offload policy and VAE tiling/slicing from measured free device memory vs the model's estimated resident footprint, then applies it to the built pipeline. auto stays resident when the model fits (byte-identical to the prior resident path), and falls to whole-module offload when tight; fast/balanced/low_vram are explicit overrides. Sequential submodule offload is unreliable for GGUF transformers on diffusers 0.38, so it falls back to whole-module offload and status reports the policy actually engaged. Verified on Z-Image-Turbo Q4_K_M (B200): auto reproduces the resident image with no VRAM/latency regression (PSNR inf); balanced/low_vram cut generation peak VRAM 47.9% (15951 -> 8318 MB) with byte-identical output, at the expected latency cost. 73 prior + 35 new CPU tests pass. * Studio diffusion (Phase 2D): streamed block-level offload + functional VAE tiling Add a streamed 'group' offload tier (diffusers apply_group_offloading, block_level, use_stream) that keeps the transformer flowing through the GPU a few blocks at a time while the text encoder / VAE stay resident, and fix VAE tiling to drive the VAE submodule (pipelines like Z-Image expose enable_tiling on pipe.vae, not the pipeline). apply_memory_plan now returns the (policy, tiling) actually engaged so status never overstates either, and group falls back to whole-module offload when the transformer can't be streamed. Measured on Z-Image (B200), all lossless (PSNR inf vs resident): balanced/group cuts generation peak VRAM 32% (15951 -> 10840 MB) at near-resident speed (2.07 -> 2.99s); low_vram/model cuts it 48% (-> 8318 MB) but is slower (7.99s). Mode names now match that tradeoff: balanced = stream the transformer, low_vram = offload every component. auto picks group when the companions fit resident, else model. 112 CPU tests pass. * Studio diffusion (Phase 5): image quality-vs-quant accuracy harness Add scripts/diffusion_quality.py, the accuracy analogue of the KLD workflow: hold prompt + seed fixed, render a grid with a reference quant (default BF16), then render each candidate quant and measure drift from the reference. Records mean PSNR + SSIM (pure-numpy, no skimage/scipy) and optional CLIP text-alignment + image-similarity (transformers, --clip), plus file size, latency, and peak VRAM, then prints a quality-vs-cost table and recommends the smallest quant within a quality budget. --selftest validates the metrics on synthetic images with no GPU or model. Verified on Z-Image (B200): the table degrades monotonically with quant size (Q8 -> Q4 -> Q2: PSNR 21.7 -> 15.5, SSIM 0.82 -> 0.61), while CLIP-text stays flat (~0.34) -- quantization erodes fine detail far more than prompt adherence. * Studio diffusion (Phase 3): opt-in speed layer (channels_last / compile / TF32) Add a speed_mode knob (off by default, so the render path stays bit-identical): default applies channels_last VAE + regional torch.compile of the denoiser's repeated block where eligible; max also enables TF32 matmul and fused QKV. Regional compile is gated off for the GGUF transformer (dequantises per-op) and for families flagged not compile-friendly (a new supports_torch_compile flag, False for Z-Image), so it activates automatically only once a non-GGUF bf16 transformer is loaded. Speed optims run before placement/offload, per the diffusers composition order. status now reports speed_mode + the optims actually engaged. Verified on Z-Image (B200): default -> ['channels_last'], max -> ['channels_last', 'tf32'], compile correctly skipped for GGUF; generation works in every mode. 121 CPU tests pass. * Studio diffusion (Phase 2B): opt-in fp8 text-encoder layerwise casting Add a text_encoder_fp8 knob that casts the companion text encoder(s) to fp8 (e4m3) storage via diffusers apply_layerwise_casting, upcasting per layer to the bf16 compute dtype while normalisations and embeddings stay full precision. Applied before placement, gated to CUDA + bf16, best-effort (a failure leaves the encoder dense). status reports which encoders were cast. Verified on Z-Image (B200, balanced/group mode where the encoder stays resident): generation peak VRAM dropped 37% (10840 -> 6791 MB, below the lowest-VRAM offload) at near-resident speed. It is a memory-vs-quality tradeoff, not free -- ~20 dB PSNR vs the bf16 encoder, a larger shift than one transformer quant step -- so it is off by default and documented as such, with the Phase 5 harness to size the cost. 127 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2C): NVFP4 text-encoder quant (+ generalise fp8 knob) Generalise the text-encoder precision knob from a fp8 bool to text_encoder_quant (fp8 | nvfp4). nvfp4 quantises the companion text encoder to 4-bit via torchao NVFP4 weight-only (two-level microscaling) on Blackwell's FP4 tensor cores; fp8 stays the broader-hardware path (cc>=8.9). Both are gated, best-effort, and run before placement; status reports the mode actually engaged. This is the lean realisation of GGUF-native text-encoder quant: 4-bit on the encoder without the 3045-line port. Verified on Z-Image (B200, balanced/group where the encoder stays resident), vs the bf16 encoder: nvfp4 cut generation peak VRAM 48% (10840 -> 5593 MB, the lowest TE option, below whole-model offload) at near-fp8 quality (16.4 vs 17.1 dB PSNR), and both quants ran faster than bf16. A memory-vs-quality tradeoff (off by default); size it per model with the Phase 5 quality harness. diffusion_bench gains --text-encoder-quant. 129 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): native stable-diffusion.cpp engine for CPU/Mac Adds the CPU / Apple-Silicon tier of the two-engine strategy, mirroring the chat backend's llama.cpp shell-out. Diffusers stays the default on CUDA / ROCm / XPU; this covers the hardware diffusers serves poorly, consuming the same split GGUF assets Studio already curates. - sd_cpp_args.py: pure sd-cli command builder. Maps the family to its text-encoder flag (Z-Image Qwen3 to --llm, Qwen-Image to --qwen2vl, FLUX.1 CLIP-L + T5), and the diffusers memory policy (none/group/model/sequential) to sd.cpp's offload flags (--offload-to-cpu / --clip-on-cpu / --vae-on-cpu / --vae-tiling / --diffusion-fa), so one user knob drives both engines. - sd_cpp_engine.py: SdCppEngine over a located sd-cli. find_sd_cpp_binary() with the same precedence as the llama finder (env override, then the Studio install root, then in-tree, then PATH), an is_available/version probe, and a one-shot subprocess generate that streams progress and returns the PNG. runtime_env() prepends the binary's directory to the platform library path so a prebuilt's bundled libstable-diffusion.so resolves. select_diffusion_engine() is the pure routing decision (GPU backends to diffusers, CPU/MPS to native when present). - install_sd_cpp_prebuilt.py: resolve + download the per-host prebuilt (macOS-arm64/Metal, Linux x86_64 CPU, Vulkan/ROCm/Windows variants) into the Studio install root. resolve_release_asset() is a pure, unit-tested host-to-asset matrix. - scripts/sd_cpp_smoke.py: end-to-end native generation harness. Tests (CPU-only, subprocess/filesystem stubbed): 49 new across args, engine, routing, runtime env, and the installer resolver. Full diffusion suite 166 passing. Verified on a B200 box: built sd-cli (CUDA) and the prebuilt (CPU) both generate Z-Image-Turbo Q4_K end to end through SdCppEngine: balanced (group offload, 5.0s gen), low_vram (full CPU offload + VAE tiling, 13.4s), and the dynamically-linked CPU prebuilt (50.4s on CPU), all producing coherent images. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 6): img2img / inpaint / edit / LoRA / upscale on the native engine Builds on Phase 4's native stable-diffusion.cpp engine, extending it from text-to-image to the wider feature surface, since sd.cpp supports all of these through the binary already. Pure command-builder additions plus one engine method, so the txt2img path is unchanged. - sd_cpp_args.py: SdCppGenParams gains image-conditioning fields. init_img + strength make a run img2img, adding mask makes it inpaint, ref_images drives FLUX-Kontext / Qwen-Image-Edit style editing (repeated --ref-image), and lora_dir + the <lora:name:weight> prompt syntax select LoRAs. New SdCppUpscaleParams + build_sd_cpp_upscale_command for the ESRGAN upscale run mode (input image + esrgan model, no prompt / text encoders). - sd_cpp_engine.py: the subprocess runner is factored into a shared _run() so generate() (now carrying the conditioning flags) and a new upscale() reuse the same streaming / error / output-check path. - scripts/sd_cpp_smoke.py: --task {txt2img,img2img,upscale} with --init-img / --strength / --upscale-model / --upscale-repeats. Tests: 10 new across the img2img / inpaint / edit / LoRA flag construction, the upscale builder and its validation, and the engine's img2img + upscale paths. Full diffusion suite 176 passing. Verified on a B200 box through SdCppEngine: img2img (Z-Image-Turbo Q4_K, the init image conditioned at strength 0.6, 4.8s) and ESRGAN upscale (512x512 -> 2048x2048 via RealESRGAN_x4plus_anime_6B, 2.7s), both producing coherent images. Video and the diffusers-path feature wiring are deferred. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): accuracy-preserving speed pass Re-review of the diffusion stack (#6675/#6679/#6680) surfaced one real accuracy bug and a dead-on-arrival speed path; this fixes both and adds the lossless / near-lossless wins, all measured on a B200. Correctness: - TF32 global-state leak (fix). speed_mode=max flipped torch.backends.*.allow_tf32 process-wide and never restored them, so a later `off` load silently inherited TF32 and was no longer bit-identical. Added snapshot_backend_flags / restore_backend_flags (TF32 + cudnn.benchmark), captured before the speed layer runs and restored on unload. Verified: load max -> unload -> load off is now byte-identical (PSNR inf) to a fresh off. - sd-cli timeout could hang forever. _run() blocked in `for line in stdout` and only checked the timeout after EOF, so a child stuck in model load / GPU init with no output ignored the timeout. Drained stdout on a reader thread with a wall-clock deadline. Added a silent-hang regression test. Speed (diffusers path), near-lossless, opt-in tiers: - Regional torch.compile now runs on the GGUF transformer. The is_gguf gate (and Z-Image's supports_torch_compile=False) were stale: compile_repeated_blocks compiles and runs ~2.2x faster on the GGUF Z-Image transformer on torch 2.9.1 / diffusers 0.38 (the per-op dequant stays eager, the rest of the block compiles). Measured: off 1.80s -> default 0.82s/gen (+54.7%), PSNR 37.7 dB vs eager -- far above the Q4 quant noise floor (~21 dB), so it does not move output quality. Gate relaxed; default tier delivers it. - cudnn.benchmark added to the default tier (autotunes the fixed-shape VAE convs). - torch.inference_mode() around the pipeline call (lossless, strictly faster than the no_grad diffusers uses internally). Memory path: - VAE tiling (not bit-identical >1MP) restricted to the model/sequential/CPU tiers; the balanced (group) tier keeps exact slicing only, so it is now bit-identical to the resident image (verified PSNR inf) and slightly faster. - Group offload adds non_blocking + record_stream on the CUDA stream path to overlap each block's H2D copy with compute (lossless; gated on the installed diffusers signature so older versions still work). Native (sd.cpp) path: - native_speed_flags: a first-class speed knob (default -> --diffusion-fa, a near-lossless CUDA win that was previously only added on offload tiers; max also -> --diffusion-conv-direct). conv-direct stays opt-in: measured +45% on CUDA, so it is never auto-on. Engine generate() merges it, de-duped against offload flags. Default profile: a GGUF model with no explicit speed_mode now resolves to the `default` profile (resolve_speed_mode), since compile's perturbation sits below the quantisation noise floor and so does not reduce quality versus the dense reference; out of the box a GGUF Z-Image generation drops from 1.80s to 0.81s. Dense models stay `off` / bit-identical, and an explicit speed_mode -- including "off" -- is always honored, so the byte-identical path remains one flag away and is the regression reference. Tooling: scripts/compile_probe.py (eager vs compiled GGUF probe), scripts/ perf_verify.py (the B200 verification above), and diffusion_bench.py gains --speed-mode so the speed tiers are benchmarkable. Tests: 183 passing (was 166); new coverage for the backend-flag snapshot/restore, GGUF compile eligibility, the balanced tiling/slicing split, native_speed_flags + the engine de-dup, and the sd-cli silent-hang timeout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): max tier uses max-autotune-no-cudagraphs + engine/lever benchmarks The opt-in `max` speed tier now compiles the repeated block with mode=max-autotune-no-cudagraphs (dynamic=False) instead of the default mode: Triton autotuning for GEMM/conv-heavier models, gated to the tier where a longer cold compile is acceptable. CUDA-graph modes (reduce-overhead / max-autotune) are deliberately avoided -- both crash on the regionally-compiled block (its static output buffer is overwritten across denoise steps), measured. Adds two reproducible benchmarks used to validate the optimization research: - scripts/compare_engines.py: PyTorch (diffusers GGUF) vs native sd.cpp head-to-head. - scripts/leverage_probe.py: coordinate_descent_tuning + FirstBlockCache probes. Measured on B200 (Z-Image Q4_K_M, 1024px, 8 steps): default compile 0.80s/gen; coordinate_descent_tuning 0.79s (within noise, already covered by max-autotune); FirstBlockCache does not run on Z-Image (diffusers 0.38 block-detection / Dynamo). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source) Add an opt-in transformer_quant mode that loads the dense bf16 transformer and torchao-quantises it onto the low-precision tensor cores, instead of the GGUF transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's 0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor. GGUF+compile stays the low-memory default and the fallback. The mode is gated on CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB); any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe (Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips the tiny projections that crash int8's torch._int_mm. New module mirrors diffusion_precision.py; quant runs before compile before placement. 184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py is the standalone torchao lever probe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): consumer-GPU tuning - lock fp8 fast accumulate, prefer fp8 over mxfp8, reject 2:4 sparsity Consumer Blackwell halves tensor-core throughput on FP32 accumulate (fp8 419 vs 838 TFLOPS with FP16 accumulate; bf16 209), so: - fp8 config locks use_fast_accum=True (Float8MMConfig). torchao already defaults it on; pinning it guards consumer cards against a default change. On B200 it is identical speed and slightly better quality (LPIPS 0.050 vs 0.091). - the Blackwell auto ladder prefers fp8 over mxfp8 (measured faster + more accurate). 2:4 semi-structured sparsity evaluated and rejected (scripts/sparse_accum_probe.py): 2:4 magnitude-prune + fp8 gives LPIPS 0.858 (broken image) with no fine-tune, the cuSPARSELt kernel errors on torch 2.9, and it does not compose with torch.compile (our main ~2x). Documented as a dead end, not shipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add fp8 fast-accum overflow verification probe scripts/fp8_overflow_check.py hooks every quantised linear during a real Z-Image generation and reports max-abs + non-finite counts for use_fast_accum True vs False. Confirms fast accumulation is an accumulation-precision knob, not an overflow one: across 276 linears, including Z-Image's ~1.0e6 activation peaks (which overflow FP16), 0 non-finite elements and identical max-abs for both modes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40; GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it. Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s), no overflow, quality below the quant noise floor. So the default leans to accuracy on data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force) lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off). 187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override threading). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add NVFP4 probe documenting it is not yet a win on torch 2.9 scripts/nvfp4_probe.py measures NVFP4 via torchao on the real Z-Image transformer. Finding (B200, 1024px/8 steps): NVFP4 is a torchao feature and DOES run with use_triton_kernel=False (the default triton path needs the missing MSLK library), but only at bf16-compile rate (0.667s vs fp8 0.592s) -- it dequantises FP4->bf16 rather than using the FP4 tensor cores. The real FP4 speedup needs MSLK or torch>=2.11 + torchao's CUTLASS FP4 GEMM. The smoke probe (default triton=True) already keeps NVFP4 out of auto on this env, so auto correctly stays on fp8; NVFP4 activates automatically once fast. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): prefer fp8 over nvfp4 in Blackwell auto ladder Validated NVFP4 on torch 2.11 + torchao CUTLASS FP4 in an isolated env. The FP4 tensor-core GEMM is genuinely active there (a 16384^3 GEMM hits ~3826 TFLOPS, 2.52x bf16 and 1.37x fp8), but it only beats fp8 on very large GEMMs. At the diffusion transformer's shapes (hidden ~3072, MLP ~12288, M~4096) NVFP4 is both slower (0.81x fp8 end to end on Z-Image 1024px) and less accurate (LPIPS 0.166 vs fp8's 0.044). Reorder the Blackwell auto ladder to fp8 before nvfp4 so auto is correct even on a future MSLK-equipped box; nvfp4 stays an explicit opt-in. Add scripts/nvfp4_t211_probe.py (extension diagnostics + GEMM micro + end-to-end). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): pre-quantized transformer loading The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs 13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline (scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True) the quantized weights, so the dense bf16 never touches the GPU. Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same torchao config + min_features filter the runtime path uses, applied ahead of time. New core/inference/diffusion_prequant.py (resolve_prequant_source + load_prequantized_transformer, best-effort, lazy imports). diffusion.py _load_dense_quant_pipeline tries the pre-quant source first and falls back to the dense materialise+quantise path, then to GGUF, so the default is unchanged. DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for the resolver, the meta-init+assign loader, and the backend branch selection + fallbacks; GPU verification via scripts/verify_prequant_backend.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): attention-backend selection Add a selectable attention kernel via the diffusers set_attention_backend dispatcher. Attention is memory-bandwidth bound, so a better kernel is an end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV matmuls torchao never touches) and composes with torch.compile. auto picks the best exact backend for the device: cuDNN fused attention (_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default (below the compile/quant noise floor); native SDPA elsewhere and when speed=off (so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/ xformers/aiter are honored, and an unavailable kernel falls back to the default rather than failing the load. New core/inference/diffusion_attention.py (normalize + per-device select + apply, best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline; attention_backend threads through begin_load / load_pipeline / status like the other load knobs. New request field attention_backend + status field. Hermetic CPU tests for normalize / select policy / apply fallback, plus route threading + 422. Measured via scripts/perf_levers_probe.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the arch tier to put int8 first on a consumer/workstation GPU (detected by the existing _is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first. Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation (-> int8) and data-center Ada/Hopper/Blackwell (-> fp8). * Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across denoise steps a DiT's output settles, so once the first block's residual barely changes the remaining blocks are skipped and their cached output reused. diffusers ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the standalone apply_first_block_cache hook as a fallback). Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 -> 2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar. OFF by default and a per-load opt-in: the win scales with step count, so it is for many-step models (Flux / Qwen-Image) and pointless for few-step distilled models (e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction of the trajectory. It composes with regional compile only with fullgraph=False (the cache's per-step decision is a torch.compiler.disable graph break), which the speed layer now switches to automatically when a cache is engaged. Best-effort: a model whose block signature the hook does not recognise is caught and the load proceeds uncached. - new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache (enable_cache / apply_first_block_cache fallback; threshold auto-raised for a quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import). - diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph when a cache is engaged. - diffusion.py: apply_step_cache before compile; thread transformer_cache / transformer_cache_threshold through begin_load -> load_pipeline and report the engaged mode in status(). - models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and transformer_cache_threshold request fields, engaged mode in the status response. - hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold selection, and best-effort failure handling, plus route threading + validation. - scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM / LPIPS vs the compiled no-cache baseline). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears) The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with 'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2 *_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing min_features filter did not exclude them. Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed, guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers (M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and quantises these layers fine, so the exclusion is int8-only. Sequence embedders (context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note 'context_embedder' contains the substring 'text_embed', which is why the token is the specific 'time_text_embed', not 'text_embed'. Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s (5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already working) are unchanged. - diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes exclude_name_tokens; quantize_transformer passes it for int8 only. - hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps attention / FFN / sequence-embedder linears), while fp8 keeps them. - scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's Linear layers and derive the exclusion list. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears) The prequant-checkpoint builder applied the dense quant filter without the int8-only M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8 checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now produces a working checkpoint on every supported model, giving int8 (the consumer-preferred scheme) the same ~2x load-VRAM and download reduction fp8 already had. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the native stable-diffusion.cpp engine instead of diffusers, with diffusers as the guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM. - diffusion_engine_router: centralised engine selection (built on the existing select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason. - sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface backed by sd-cli, with lazy binary install, registry-driven asset fetch, step-progress parsing, and cancellation. - diffusion_families: per-family single-file VAE + text-encoder asset mapping. - sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled). - routes/inference + gpu_arbiter: drive the active engine via the router; the API now reports the active engine and any fallback reason. - tests for the backend, router, route selection, and cancellation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler Address review feedback on #6724: - engine router: unload the engine being deactivated on a switch, so the old model is not left resident-but-unreachable (the evictor only targets the active engine). - generate route: sd.cpp execution errors (nonzero exit / timeout / missing output) now map to 500, not 409 (which only means not-loaded / cancelled). - native batch: return per-image seeds and persist the actual seed for each image so every batch image is reproducible. - Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the stable-diffusion.cpp docs; other families keep sd-cli defaults. - honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload offload flags on the native load instead of hardcoding them off. - fail the load when the sd-cli binary is present but not runnable (version() now returns None on exec error / nonzero exit). - size estimate: only treat the transformer asset as a possible local path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in load_prequantized_transformer ends in torch.load(weights_only=False), which executes arbitrary code from the pickle. The transformer_prequant_path load-request field reached that unpickle for any local file an authenticated caller named, so a request could trigger remote code execution. Refuse the source.kind=='path' branch unless the operator sets UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted and unaffected. Document the requirement on the API field and add gate tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa - apply_attention_backend now restores the native default when no backend is requested or a kernel fails. diffusers keeps a process-wide active attention backend that set_attention_backend updates, and a fresh transformer's processors follow it, so a load that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile load pinned, breaking the bit-identical/off guarantee. - select_attention_backend drops flash3/flash4 up front when the CUDA capability is below Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit request on the wrong card set fine then crashed mid-generation; it now falls back to native. - Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a valid alias of native) is accepted instead of 422-rejected by Pydantic. - Drop the dead replace('-','_') normalization (no alias uses dashes/underscores). - perf_levers_probe.py output dir is now relative to the script, not a hardcoded path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF - apply_step_cache now engages only via the transformer's native enable_cache (the diffusers CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model now runs uncached per the best-effort contract. - GGUF transformers are quantized (the default Studio load path), so they now use the higher quantized FBCache threshold when the caller leaves it unset, instead of the dense default that could keep the cache from triggering. - fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so fullgraph=True failed warmup and silently measured an eager cached run); output dir is now relative to the script, not a hardcoded path. * Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder _is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate. Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0). * Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just TypeError on older torchao), which would break the mxfp8 config helper instead of falling back to the default. Catch both so the fallback is robust. quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds CUDA memory during the per-row VRAM probe; output dir relative to the script. * Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load - snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the whole snapshot. restore_backend_flags restores each flag independently so one failure can't leave the others leaked process-wide. - load_pipeline restores the flags (and clears the GPU cache) when the build fails after apply_speed_optims mutated the process-wide flags but before _state captured them for unload to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and contaminated later off generations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs without producing output (or without closing stdout) would never reach proc.wait and the wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the PROCESS, so the main thread always enforces the timeout and kills a hung process (which closes the pipe and ends the reader). Add a test that times out even when stdout blocks, and make the no-binary test hermetic so a host-installed sd-cli can't leak in. * Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn The filter callback can be invoked without a module name, so fqn.lower() would raise AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens, so the linear is kept) instead of crashing the quantise pass. * Studio diffusion (Phase 16) review fixes: native engine robustness - sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64); a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds aliased to the same image. Random seeds stay 53-bit (JS-safe). - sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub fall back to anonymous instead of failing auth on a blank token. - sd_cpp_backend: a superseding load now cancels the in-flight generation, so the old sd-cli can no longer return/persist an image from the previous model. - diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a slow 10+ GB free / CUDA sync does not block engine selection. - diffusion_engine_router: probe sd-cli runnability (version()) before committing to native, so a present-but-unrunnable binary falls back to diffusers at selection. - diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a CPU-only install can still reach the native sd.cpp engine instead of failing load. - tests updated for the runnability probe + a not-runnable fallback case. * Studio diffusion (Phase 9) review fixes: prequant safety + validation - SECURITY: a request-supplied local pre-quant path is now unpickled only when it resolves inside an operator-configured ALLOWLIST of directories (UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in, once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on any path a load request named (arbitrary code execution). realpath() blocks symlink escapes; a bare on/off toggle is no longer a wildcard. - Validate the checkpoint's min_features against the runtime Linear filter, so a checkpoint that quantised a different layer set is rejected instead of silently loading a model that mismatches the dense path while reporting the same scheme. - Tolerant base_model_id compare (exact or same final path/repo segment), so a local path or fork of the canonical base is accepted instead of falling back to dense. - _has_meta_tensors uses any(chain(...)) (no intermediate lists). - prequant verify/probe scripts use repo-relative paths (+ env overrides), not the author's absolute /mnt paths. - tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts - diffusion_memory: when group offload is unavailable and the plan falls back to whole-module offload, enable VAE tiling (the group plan left it off, but the fallback is the low-VRAM path where the decode spike can OOM). Covers both the group and sequential fallback branches. - perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a balanced bit-identity regression actually fails the check instead of exiting 0. - compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and the load-progress poll has a 30 min deadline instead of looping forever on a hang. - test for the group->model fallback enabling VAE tiling. * Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path - diffusion: a torchao-quantized transformer is committed only compiled. A dense model resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF it replaced), so when transformer_quant engaged and speed resolved to off, promote to default (regional compile); warn loudly if compile still does not engage. - diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK); otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF. - nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path). - test asserts the eager-quant -> default-compile promotion. * Studio diffusion (Phase 10) review fixes: attention gating + probe isolation - diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation, so auto now stays on native SDPA there. - diffusion_attention: _active_attention_backend handles get_active_backend() returning an enum/None (not a tuple); the old unpack always raised and was swallowed, so the native-restore short-circuit never fired. - perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul so the inductor_flags variant doesn't leak into later compiled rows. - tests for the SM80 cuDNN gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening - install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket); extract through a per-member containment check (Zip-Slip guard); expanduser the --install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch the separately-published cudart runtime DLL archive so sd-cli.exe can start. - sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie. - tests: Zip-Slip rejection, normal extraction, studio-home discovery. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of the literal --output path. SdCppEngine.generate checked only the literal path, so a batch generation would exit 0 and then raise 'no image' (or return a stale file). generate now returns the literal path when present and otherwise falls back to the numbered siblings; single-image behavior is unchanged. Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected without error. * Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats Codex review on the native engine arg builder: - build_sd_cpp_command emitted --width/--height unconditionally, so an img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of the input. width/height are now Optional (None = unset): an image-conditioned run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives the size from the input image (set_width_and_height_if_unset); a plain txt2img run with unset dims keeps the prior 1024x1024 default; explicit dims are always honored. width/height are read only by the builder, so the type change is local. - build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...) that silently swallowed repeats=0 into sd-cli's default of one pass, turning an explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError and emits the flag for any explicit value != 1. Tests: img2img unset dims omit width/height (init_img and ref_images), explicit dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at the default. (Two pre-existing binary-discovery tests fail only because a real sd-cli is installed in this dev environment; unrelated to this change.) * Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc Codex review: the transformer_prequant_path field description still told operators to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the prior security fix made that variable a directory allowlist -- _allowed_prequant_roots deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator following the documented =1 would have every transformer_prequant_path request silently refused. The description now states it must name one or more allowlisted directories and that a bare on/off value is not accepted. Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does not say =1, and describes an allowlist/directory (guards against doc drift). * Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset Codex review on attention-backend selection: - Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first generation with no fallback. select_attention_backend now applies _cudnn_attention_supported() to an explicit cuDNN request too. - flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3 is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a (min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+. - apply_attention_backend's success path left diffusers' process-wide active backend pinned to the kernel it set; a later component whose processors are unconfigured (backend None) would inherit it. It now resets the global registry to native after a successful per-transformer set (the transformer keeps its own backend), best-effort. Also fixed _active_attention_backend: get_active_backend() returns a (name, fn) tuple, so the prior code stringified the tuple and never matched a name, defeating the native-restore short-circuit. Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on SM90; global registry reset after a successful set; _active_attention_backend reads the tuple return. * Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300 (Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So _is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they are treated as data-center class and keep the intended fp8-first behavior. Test: extends the datacenter parametrize with 'NVIDIA B300' and 'NVIDIA GH200 480GB' (now _is_consumer_gpu False). * Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder Codex review: the M=1 modulation/embedder exclusion was wired only into the dense runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint quantised the AdaLN modulation and conditioning-embedder linears, and loading it via transformer_prequant_path (the load path only loads already-quantised tensors, it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes for the runtime path. Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 -> the M=1 exclusion, every other scheme -> none) and use it in both the runtime quantiser and the builder, so a prequant artifact's quantised-layer set always matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion). Test: int8_exclude_name_tokens returns the exclusion for int8 and () for fp8/nvfp4/mxfp8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race Codex review on the native-engine routing: - The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat) unconditionally after engine selection. A native sd.cpp load on a pure-CPU host never touches the GPU, so that needlessly tore down the resident chat model. The handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it. - sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when _run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so the setting was unverifiable. status now derives them from state.offload_flags (still 'none' on CPU, where the flags are empty). - _run_load committed the new state without cancelling/waiting on a generation that started during the (slow) asset download, so a stale sd-cli run against the OLD model could finish afterward and persist an image from the previous model once the new load reported ready. The commit now signals the in-flight cancel and waits on _generate_lock before swapping _state (taken only at commit, so the download never serialises against generation), mirroring the diffusers load path. Tests: CPU native load skips the arbiter while a GPU native load takes it; status reports offload active when flags are set; _run_load cancels and waits for an in-flight generation before committing. * Studio diffusion (Phase 14) review round 2: align helper name with the stack Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the identical helper already present higher in the diffusion stack (Phase 16). The helper definition, the runtime quantiser call, and the offline builder are now byte-identical to that version, so the two branches no longer introduce a divergent name for the same single-source-of-truth and the stack merges without a conflict on this fix. No behavior change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: eager patches + torch.compile cache speed phase Adds the opt-in speed path for the GGUF diffusion transformer behind a selectable speed mode (default off, so output is unchanged until a profile is chosen): - diffusion_eager_patches.py: shared eager fast-paths (channels_last, attention/backend selection, fused norms and QKV) installed at load and rolled back on unload or failed load. - diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent torch.compile cache and the GGUF-transformer compile wiring. - diffusion_arch_patches.py: architecture-specific patches. - diffusion_patch_backend.py: shared install/restore plumbing. - diffusion_speed.py: speed-profile planning. Tests for each module plus the benchmarking and probe scripts used to measure speed, memory, and accuracy of the path. * Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI Backend: - Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file fp8 transformers, gated to the unsloth org plus a curated allowlist. - Image-conditioned workflows built with Pipeline.from_pipe so they reuse the loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint, outpaint, and a hires-fix upscale pass. - Instruction editing as its own family kind (Qwen-Image-Edit-2511, FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and multi-reference) plus klein inpaint. - Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the decoded image size and cap upscale output to avoid OOM on large inputs. - Fixes: from_pipe defaulting to a float32 recast that crashed torchao quantized transformers; image-conditioned calls forcing the slider size onto the input image. Native sd.cpp engine rejects image-conditioned and reference requests it cannot serve. Frontend: - Redesigned Images page with capability-gated workflow tabs (Create, Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor, client-side outpaint, and a multi-reference picker. - Advanced options moved to a right-docked panel mirroring Chat: closed by default, toggled by a single fixed top-bar button that stays in place. sd.cpp installer: pin the release, verify each download's sha256, add a download timeout, and make the source repo configurable for a future mirror. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: correct the Advanced panel comment (closed by default, fixed toggle) * Studio diffusion: LoRA adapters for the Images workflow Add community LoRA support across both diffusion backends, the single biggest step toward broad image-workflow coverage. Backend - New shared module core/inference/diffusion_lora.py: adapter discovery (local scan + curated catalog + owner/name[:file] Hub refs), download via hf_hub_download_with_xet_fallback, alias sanitization, native managed-dir materialization with collision-broken aliases, prompt-tag injection (deduped against user-typed tags), and a supports_lora gate. - Native sd-cli: resolve + materialize selected LoRAs into a per-run managed dir, inject <lora:ALIAS:w> tags, pass --lora-model-dir with --lora-apply-mode auto. The arg builder already emitted these flags. - Diffusers: non-fused load_lora_weights + set_adapters manager, tracked on the pipe so an unchanged selection is a no-op and a model swap resets; cleared on unload. Never fuses (breaks quantized transformers and blocks live weight tweaks). - Gated off where unsupported: torchao fp8/int8 dense, GGUF-via-diffusers, and native Qwen-Image (no LoRA name-conversion branch upstream). - Request contract: optional loras on DiffusionGenerateRequest; empty or omitted is identical to today. supports_lora surfaced in status; chosen LoRAs persisted in gallery recipe metadata. - New GET /api/models/diffusion-loras for the picker (family-filtered). Frontend - Repeatable multi-LoRA picker (adapter select + weight slider 0..2 + remove), gated by the loaded model's supports_lora and family, max 8. Tests - New test_diffusion_lora.py (14): helpers, request validation, native tag/dir wiring, diffusers set_adapters manager, supports_lora matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: ControlNet for the Images workflow (diffusers) Add ControlNet conditioning, the #2 most-used diffusion workflow after LoRA, on the diffusers backend for the families with ControlNet pipelines (FLUX.1 and Qwen-Image), with Union models as the default picks. Backend - New core/inference/diffusion_controlnet.py: family-gated discovery (curated Union models + local dirs + bare owner/name repos), resolution to a loadable repo/dir, control-image preprocessing (passthrough + a dependency-free canny edge map), and a supports_controlnet gate. - diffusion.py: a ControlNet manager parallel to the LoRA one. Loads the (small) ControlNet model once via from_pretrained (cached by id) and builds the family's ControlNet pipeline via Pipeline.from_pipe(base, controlnet=model), reusing the resident base modules at their loaded dtype (no reload, no recast). Passes the control image + conditioning scale + guidance start/end at generate time; cleared on unload. - Families: FLUX.1 -> FluxControlNetPipeline/Model, Qwen-Image -> QwenImageControlNetPipeline/Model. Others declare none (gated off). - Gated off for the native engine, GGUF-via-diffusers, and torchao fp8/int8 dense (same rule as LoRA). v1 conditions txt2img only. - Request contract: optional controlnet on DiffusionGenerateRequest; supports_controlnet in status; the choice persisted in gallery meta. - New GET /api/models/diffusion-controlnets for the picker. Frontend - A ControlNet control in the Images rail (model select + control-image upload + control-type select + strength slider), gated by the loaded model's supports_controlnet + family, shown for text-to-image. Tests - New test_diffusion_controlnet.py (10): discovery/resolve/preprocess/gate helpers, request validation, family wiring, and the diffusers pipe manager (loads once, caches, from_pipe with controlnet, rejects unsupported families). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio ControlNet: show the picker on the Create tab (workflow id is 'create', not 'txt2img') The ControlNet control gated on workflow === 'txt2img', but the Images workflow tab ids are create/transform/inpaint/extend/upscale/reference/edit -- there is no 'txt2img'. So the picker never rendered even with a ControlNet-capable model loaded. Gate on 'create' (the text-to-image tab) for both the picker and the request wiring. Found via a live Playwright capture of the running Studio. * Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a stray "gguf" tag on the Hub but ship no .gguf files. The model search classified them as GGUF from the bare tag, so the picker rendered the GGUF variant expander, which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata (populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected. * Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline load when the id is unsloth-hosted or on-device (the backend infers the family + base repo and gates loads to unsloth/* or local paths), and show a clear message otherwise instead of silently ignoring the click. Curated and GGUF paths are unchanged. * Studio Images: keep curated safetensors models in Recommended after download The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet), so they vanished from the picker after the first load and could only be found by typing an exact search. The row already renders a downloaded badge, matching how GGUF Recommended rows stay visible when cached. Drop the exclusion so the curated safetensors always list. * Studio diffusion LoRA: sanitize dots out of adapter aliases The LoRA alias is used as the diffusers PEFT adapter name, and PEFT rejects names containing "." (module name can't contain "."). sanitize_alias kept dots, so a LoRA whose filename carries a version tag (e.g. Qwen-Image-2512-Lightning-8steps-V1.0-bf16) failed to apply with a 400. Replace dots too; the alias stays a valid native <lora:NAME:w> filename stem. Adds regression coverage for internal dots. * Studio Images: clarify the GGUF transformer-quant Advanced control Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode" with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/ FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF, falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only. * Studio Images: list on-device unsloth diffusion models in the picker The Images picker's On Device tab hid every non-GGUF cached repo whenever a task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit and FP8 safetensors) never showed up there. List cached repos that pass the task gate, limited under a filter to unsloth-hosted ones so base repos (which fail the diffusion load trust gate) don't appear only to dead-end on click. Chat behavior is unchanged: the task gate still drops image repos there. * Studio: hide single-file image checkpoints from the chat model picker The chat picker treats a cached repo as an image model, and hides it, only when it ships a diffusers model_index.json. Single-file, ComfyUI, and ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models. Fall back to resolving the repo id against the known diffusion families, the same resolver the Images backend loads from, so these checkpoints are tagged text-to-image and stay in the Images picker only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: add the FLUX.2-dev model family Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline) resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline + Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated, reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror. text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from klein's turbo defaults. Verified live: GGUF load resolves the family + gated base repo and generates a real 1024x1024 image on GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: clearer error for an unsupported diffusion model When a repo id resolves to no diffusion family the load raised 'Could not infer a diffusion family... Pass family_override (z-image)', which points at an unrelated family and doesn't say what is supported. Replace it with a message that lists the supported families (from a new supported_family_names helper) and notes that video models and image models whose diffusers transformer has no single-file loader are not supported. Applies to both the diffusers and native sd.cpp load paths. Also refreshes two stale family-registry comments that still called FLUX.2-dev omitted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove stray async task scratch outputs committed by mistake * ControlNet: reject filesystem-like ids and do not cache a model past an unload race Two review findings on the ControlNet path: - resolve_controlnet's bare-repo fallback accepted any id with a slash, so a path-shaped id (/tmp/x, ../x) reached from_pretrained as a local directory. Restrict the fallback to a strict owner/name HF repo id shape. - _controlnet_pipe now re-checks the cancel event after the blocking from_pretrained: an unload that raced the download had already cleared the caches, so caching the late module would pin it past the unload. * ControlNet: address review findings on the diffusers path - resolve_controlnet enforces catalog family compatibility so a direct API call cannot load a ControlNet built for another family through the wrong pipeline. - Unknown ControlNet ids now surface as a 400 (call site maps FileNotFoundError to ValueError) instead of a generic 500. - strength 0 disables ControlNet entirely, so a no-op selection never pays the download / VRAM cost; the control image is decoded and validated BEFORE the ControlNet is resolved or built, so a malformed image fails fast for the same reason. - ControlNet loads use the base compute dtype (state.dtype is a display string, not a torch.dtype, so it silently fell back to float32) and honor the base offload policy via group offloading instead of forcing the module resident. - Empty/malformed HF token coerced to anonymous access. - Flux Union ControlNet control_mode mapped from the selected control type. - resolve_controlnet drops the unused hf_token/cancel_event params. - ControlNetSpec validates guidance_start <= guidance_end (clean 422). - Images UI ControlNet Select shows its placeholder when nothing is selected. Adds regression tests for family enforcement and the union control-mode map. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion LoRA: harden resolution, native tag precedence, and diffusers teardown Address review findings on the LoRA path: - resolve_one: normalise a blank/whitespace hf_token to None (anonymous access) and reject a client-supplied weight file with traversal / absolute path. - resolve_specs: convert FileNotFoundError from an unknown/stale id to ValueError so the route returns 400 instead of a generic 500. - _scan_local: disambiguate local adapters that share a stem (foo.safetensors vs foo.gguf) so each is uniquely addressable. - inject_prompt_tags: the backend-validated weight now wins over a user-typed <lora:ALIAS:...> for a selected adapter; unselected user tags are left alone. - diffusers _apply_loras: reject a .gguf adapter with a clear error before touching the pipe (diffusers loads safetensors only). - _unload_locked: drop the explicit unload_lora_weights() on teardown; the pipe is dropped wholesale (freeing adapters), so the previous call could race an in-flight denoise on the same pipe. - Images page: use a stable LoRA key and clear the selection (not just the options) when the catalog refresh fails. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion: guard trust check against OSError and validate conditioning inputs - _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid characters (or a bare owner/name id) can't raise OSError; treat any failure as not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request still raises the clear FileNotFoundError for a genuinely missing local pick. - generate(): reject mask_image / upscale / reference_images supplied without an input image, and reject reference_images on a family that does not support reference conditioning, instead of silently degrading to txt2img / img2img. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address Codex review findings on the image-workflows PR Keep diffusion.py importable without torch: the compile/arch patch modules import torch at module level, so import them lazily at their load/unload call sites instead of at module load. This restores the torchless contract so get_diffusion_backend() works on a CPU/native sd.cpp install. Match family reject keywords and aliases as whole path/name segments, not raw substrings, so an unrelated word like edited, edition, or kontextual no longer misroutes or hides a valid base image model, while supported edit families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same segment matching in the picker task filter. Route FLUX.2-dev native guidance through --guidance like the other FLUX families rather than --cfg-scale. Reject native upscale requests that have no input image. Read image header dimensions and reject over-limit inputs before decoding pixels, so a crafted small-payload image cannot spike memory. Reject an upscale that would shrink the source below its input size. Validate the model_kind against the filename extension before the GPU handoff. Estimate a local diffusers pipeline's size from its on-disk weights so auto memory planning does not skip offload and OOM. Report workflows: [txt2img] from the native backend status so the Create tab stays enabled for a loaded native model. Clamp the outpaint canvas to the backend's 4096px decode limit. Adds regression tests for segment matching and kind/extension validation. * Harden diffusion LoRA handling on the diffusers and native paths Reject LoRA on a torch.compile'd diffusers transformer (Speed=default/max): diffusers requires the adapter loaded before compilation, so applying one to the already-compiled module fails with adapter-key mismatches. The status gate now hides the picker and generate raises a clear message instead. Convert a cancelled Hub LoRA download (RuntimeError Cancelled) to the diffusion cancellation sentinel in resolve_specs, so an unload/superseding load during resolution maps to a 409 instead of a generic server error. Drop weight-0 LoRA rows before the native support gate so a request carrying only disabled adapters stays a no-op on families where native LoRA is unsupported, matching the diffusers path. Reject duplicate LoRA ids in the request model: both apply paths suffix colliding names, so a repeated id would stack the same adapter past its per-adapter weight bound. Strip all user-typed <lora:...> prompt tags on the native path (only the selected adapters are materialized in the managed lora-model-dir, so an unselected tag can never resolve), and restore saved LoRA selections from a gallery recipe so restore reproduces a LoRA image. * Harden ControlNet resolve, gallery metadata, and the control-type picker Check cancellation immediately after a ControlNet from_pretrained and before any device placement, so an unload/eviction that raced the download does not allocate several GB onto the GPU after the load was already cleared. Require a loadable weight or shard index (not just config.json) before a local ControlNet folder is advertised, so an interrupted copy is hidden instead of failing deep in from_pretrained as a generic 500. Do not record a strength-0 ControlNet in the gallery recipe: it is treated as disabled and skipped, so the image is unconditioned and the metadata must not claim a ControlNet was applied. Build the control-type picker from the selected ControlNet's advertised control_types instead of a hardcoded passthrough/canny pair, so a union model with a precomputed depth or pose map sends the correct control_mode. * Address further Codex findings on the image-workflows PR - Persist the actual output image size in the gallery recipe instead of the request sliders: Transform/Inpaint/Edit derive the size from the uploaded image, Extend grows the canvas, and Upscale resizes it, so the sliders recorded (and later restored) the wrong dimensions for those workflows. - Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file name) in validate_load_request, so the unloadable pick fails before chat is evicted rather than deep in from_pretrained. - Only publish an image-conditioned from_pipe wrapper to the shared aux cache when the load is still current: from_pipe runs under the generate lock but not the state lock, so an unload racing its construction could otherwise cache a wrapper over torn-down modules that a later load would reuse. - Verify the Windows CUDA runtime archive checksum before extracting it, like the main sd-cli archive, so a corrupt or tampered runtime is rejected rather than extracted next to the binary. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
|
Daniel Han
|
bfabbbe7b6
|
Studio diffusion: LoRA adapters for the Images workflow (#6771)
* Studio diffusion: cross-platform device policy, fp16 guard, lock split, validate-before-evict Phase 1 of porting the richer diffusion stack onto the image-generation backend. - Add a compartmentalized device/dtype policy module (diffusion_device.py) resolving CUDA/ROCm/XPU/MPS/CPU with capability flags. Keeps the NVIDIA capability-based bf16 choice; ROCm and XPU are isolated; MPS uses bf16 or fp32, never a silent fp16 that renders a black image. - Add a per-family fp16_incompatible flag (Z-Image) and promote a resolved float16 to float32 for those families so they do not produce black images. - Split the backend locks: a generation holds only _generate_lock, so status, unload, and a new load are never blocked by a long denoise. Add per-generation cancellation via callback_on_step_end so an eviction or a superseding load preempts a running generation; a replacement load waits for it to stop before allocating, so two pipelines never sit in VRAM at once. - Validate a load request before the GPU handoff so an unloadable pick never evicts a working chat model, and reject missing local paths up front. - Add CPU-only tests for the device policy, dtype guard, lock split and cancellation, and validate-before-evict, plus a GPU benchmark/regression script (scripts/diffusion_bench.py) measuring latency, peak VRAM, and PSNR against a saved reference. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2A): measured-budget memory planner + offload/VAE policy Add a lean, backend-agnostic memory policy that picks a CPU-offload policy and VAE tiling/slicing from measured free device memory vs the model's estimated resident footprint, then applies it to the built pipeline. auto stays resident when the model fits (byte-identical to the prior resident path), and falls to whole-module offload when tight; fast/balanced/low_vram are explicit overrides. Sequential submodule offload is unreliable for GGUF transformers on diffusers 0.38, so it falls back to whole-module offload and status reports the policy actually engaged. Verified on Z-Image-Turbo Q4_K_M (B200): auto reproduces the resident image with no VRAM/latency regression (PSNR inf); balanced/low_vram cut generation peak VRAM 47.9% (15951 -> 8318 MB) with byte-identical output, at the expected latency cost. 73 prior + 35 new CPU tests pass. * Studio diffusion (Phase 2D): streamed block-level offload + functional VAE tiling Add a streamed 'group' offload tier (diffusers apply_group_offloading, block_level, use_stream) that keeps the transformer flowing through the GPU a few blocks at a time while the text encoder / VAE stay resident, and fix VAE tiling to drive the VAE submodule (pipelines like Z-Image expose enable_tiling on pipe.vae, not the pipeline). apply_memory_plan now returns the (policy, tiling) actually engaged so status never overstates either, and group falls back to whole-module offload when the transformer can't be streamed. Measured on Z-Image (B200), all lossless (PSNR inf vs resident): balanced/group cuts generation peak VRAM 32% (15951 -> 10840 MB) at near-resident speed (2.07 -> 2.99s); low_vram/model cuts it 48% (-> 8318 MB) but is slower (7.99s). Mode names now match that tradeoff: balanced = stream the transformer, low_vram = offload every component. auto picks group when the companions fit resident, else model. 112 CPU tests pass. * Studio diffusion (Phase 5): image quality-vs-quant accuracy harness Add scripts/diffusion_quality.py, the accuracy analogue of the KLD workflow: hold prompt + seed fixed, render a grid with a reference quant (default BF16), then render each candidate quant and measure drift from the reference. Records mean PSNR + SSIM (pure-numpy, no skimage/scipy) and optional CLIP text-alignment + image-similarity (transformers, --clip), plus file size, latency, and peak VRAM, then prints a quality-vs-cost table and recommends the smallest quant within a quality budget. --selftest validates the metrics on synthetic images with no GPU or model. Verified on Z-Image (B200): the table degrades monotonically with quant size (Q8 -> Q4 -> Q2: PSNR 21.7 -> 15.5, SSIM 0.82 -> 0.61), while CLIP-text stays flat (~0.34) -- quantization erodes fine detail far more than prompt adherence. * Studio diffusion (Phase 3): opt-in speed layer (channels_last / compile / TF32) Add a speed_mode knob (off by default, so the render path stays bit-identical): default applies channels_last VAE + regional torch.compile of the denoiser's repeated block where eligible; max also enables TF32 matmul and fused QKV. Regional compile is gated off for the GGUF transformer (dequantises per-op) and for families flagged not compile-friendly (a new supports_torch_compile flag, False for Z-Image), so it activates automatically only once a non-GGUF bf16 transformer is loaded. Speed optims run before placement/offload, per the diffusers composition order. status now reports speed_mode + the optims actually engaged. Verified on Z-Image (B200): default -> ['channels_last'], max -> ['channels_last', 'tf32'], compile correctly skipped for GGUF; generation works in every mode. 121 CPU tests pass. * Studio diffusion (Phase 2B): opt-in fp8 text-encoder layerwise casting Add a text_encoder_fp8 knob that casts the companion text encoder(s) to fp8 (e4m3) storage via diffusers apply_layerwise_casting, upcasting per layer to the bf16 compute dtype while normalisations and embeddings stay full precision. Applied before placement, gated to CUDA + bf16, best-effort (a failure leaves the encoder dense). status reports which encoders were cast. Verified on Z-Image (B200, balanced/group mode where the encoder stays resident): generation peak VRAM dropped 37% (10840 -> 6791 MB, below the lowest-VRAM offload) at near-resident speed. It is a memory-vs-quality tradeoff, not free -- ~20 dB PSNR vs the bf16 encoder, a larger shift than one transformer quant step -- so it is off by default and documented as such, with the Phase 5 harness to size the cost. 127 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2C): NVFP4 text-encoder quant (+ generalise fp8 knob) Generalise the text-encoder precision knob from a fp8 bool to text_encoder_quant (fp8 | nvfp4). nvfp4 quantises the companion text encoder to 4-bit via torchao NVFP4 weight-only (two-level microscaling) on Blackwell's FP4 tensor cores; fp8 stays the broader-hardware path (cc>=8.9). Both are gated, best-effort, and run before placement; status reports the mode actually engaged. This is the lean realisation of GGUF-native text-encoder quant: 4-bit on the encoder without the 3045-line port. Verified on Z-Image (B200, balanced/group where the encoder stays resident), vs the bf16 encoder: nvfp4 cut generation peak VRAM 48% (10840 -> 5593 MB, the lowest TE option, below whole-model offload) at near-fp8 quality (16.4 vs 17.1 dB PSNR), and both quants ran faster than bf16. A memory-vs-quality tradeoff (off by default); size it per model with the Phase 5 quality harness. diffusion_bench gains --text-encoder-quant. 129 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): native stable-diffusion.cpp engine for CPU/Mac Adds the CPU / Apple-Silicon tier of the two-engine strategy, mirroring the chat backend's llama.cpp shell-out. Diffusers stays the default on CUDA / ROCm / XPU; this covers the hardware diffusers serves poorly, consuming the same split GGUF assets Studio already curates. - sd_cpp_args.py: pure sd-cli command builder. Maps the family to its text-encoder flag (Z-Image Qwen3 to --llm, Qwen-Image to --qwen2vl, FLUX.1 CLIP-L + T5), and the diffusers memory policy (none/group/model/sequential) to sd.cpp's offload flags (--offload-to-cpu / --clip-on-cpu / --vae-on-cpu / --vae-tiling / --diffusion-fa), so one user knob drives both engines. - sd_cpp_engine.py: SdCppEngine over a located sd-cli. find_sd_cpp_binary() with the same precedence as the llama finder (env override, then the Studio install root, then in-tree, then PATH), an is_available/version probe, and a one-shot subprocess generate that streams progress and returns the PNG. runtime_env() prepends the binary's directory to the platform library path so a prebuilt's bundled libstable-diffusion.so resolves. select_diffusion_engine() is the pure routing decision (GPU backends to diffusers, CPU/MPS to native when present). - install_sd_cpp_prebuilt.py: resolve + download the per-host prebuilt (macOS-arm64/Metal, Linux x86_64 CPU, Vulkan/ROCm/Windows variants) into the Studio install root. resolve_release_asset() is a pure, unit-tested host-to-asset matrix. - scripts/sd_cpp_smoke.py: end-to-end native generation harness. Tests (CPU-only, subprocess/filesystem stubbed): 49 new across args, engine, routing, runtime env, and the installer resolver. Full diffusion suite 166 passing. Verified on a B200 box: built sd-cli (CUDA) and the prebuilt (CPU) both generate Z-Image-Turbo Q4_K end to end through SdCppEngine: balanced (group offload, 5.0s gen), low_vram (full CPU offload + VAE tiling, 13.4s), and the dynamically-linked CPU prebuilt (50.4s on CPU), all producing coherent images. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 6): img2img / inpaint / edit / LoRA / upscale on the native engine Builds on Phase 4's native stable-diffusion.cpp engine, extending it from text-to-image to the wider feature surface, since sd.cpp supports all of these through the binary already. Pure command-builder additions plus one engine method, so the txt2img path is unchanged. - sd_cpp_args.py: SdCppGenParams gains image-conditioning fields. init_img + strength make a run img2img, adding mask makes it inpaint, ref_images drives FLUX-Kontext / Qwen-Image-Edit style editing (repeated --ref-image), and lora_dir + the <lora:name:weight> prompt syntax select LoRAs. New SdCppUpscaleParams + build_sd_cpp_upscale_command for the ESRGAN upscale run mode (input image + esrgan model, no prompt / text encoders). - sd_cpp_engine.py: the subprocess runner is factored into a shared _run() so generate() (now carrying the conditioning flags) and a new upscale() reuse the same streaming / error / output-check path. - scripts/sd_cpp_smoke.py: --task {txt2img,img2img,upscale} with --init-img / --strength / --upscale-model / --upscale-repeats. Tests: 10 new across the img2img / inpaint / edit / LoRA flag construction, the upscale builder and its validation, and the engine's img2img + upscale paths. Full diffusion suite 176 passing. Verified on a B200 box through SdCppEngine: img2img (Z-Image-Turbo Q4_K, the init image conditioned at strength 0.6, 4.8s) and ESRGAN upscale (512x512 -> 2048x2048 via RealESRGAN_x4plus_anime_6B, 2.7s), both producing coherent images. Video and the diffusers-path feature wiring are deferred. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): accuracy-preserving speed pass Re-review of the diffusion stack (#6675/#6679/#6680) surfaced one real accuracy bug and a dead-on-arrival speed path; this fixes both and adds the lossless / near-lossless wins, all measured on a B200. Correctness: - TF32 global-state leak (fix). speed_mode=max flipped torch.backends.*.allow_tf32 process-wide and never restored them, so a later `off` load silently inherited TF32 and was no longer bit-identical. Added snapshot_backend_flags / restore_backend_flags (TF32 + cudnn.benchmark), captured before the speed layer runs and restored on unload. Verified: load max -> unload -> load off is now byte-identical (PSNR inf) to a fresh off. - sd-cli timeout could hang forever. _run() blocked in `for line in stdout` and only checked the timeout after EOF, so a child stuck in model load / GPU init with no output ignored the timeout. Drained stdout on a reader thread with a wall-clock deadline. Added a silent-hang regression test. Speed (diffusers path), near-lossless, opt-in tiers: - Regional torch.compile now runs on the GGUF transformer. The is_gguf gate (and Z-Image's supports_torch_compile=False) were stale: compile_repeated_blocks compiles and runs ~2.2x faster on the GGUF Z-Image transformer on torch 2.9.1 / diffusers 0.38 (the per-op dequant stays eager, the rest of the block compiles). Measured: off 1.80s -> default 0.82s/gen (+54.7%), PSNR 37.7 dB vs eager -- far above the Q4 quant noise floor (~21 dB), so it does not move output quality. Gate relaxed; default tier delivers it. - cudnn.benchmark added to the default tier (autotunes the fixed-shape VAE convs). - torch.inference_mode() around the pipeline call (lossless, strictly faster than the no_grad diffusers uses internally). Memory path: - VAE tiling (not bit-identical >1MP) restricted to the model/sequential/CPU tiers; the balanced (group) tier keeps exact slicing only, so it is now bit-identical to the resident image (verified PSNR inf) and slightly faster. - Group offload adds non_blocking + record_stream on the CUDA stream path to overlap each block's H2D copy with compute (lossless; gated on the installed diffusers signature so older versions still work). Native (sd.cpp) path: - native_speed_flags: a first-class speed knob (default -> --diffusion-fa, a near-lossless CUDA win that was previously only added on offload tiers; max also -> --diffusion-conv-direct). conv-direct stays opt-in: measured +45% on CUDA, so it is never auto-on. Engine generate() merges it, de-duped against offload flags. Default profile: a GGUF model with no explicit speed_mode now resolves to the `default` profile (resolve_speed_mode), since compile's perturbation sits below the quantisation noise floor and so does not reduce quality versus the dense reference; out of the box a GGUF Z-Image generation drops from 1.80s to 0.81s. Dense models stay `off` / bit-identical, and an explicit speed_mode -- including "off" -- is always honored, so the byte-identical path remains one flag away and is the regression reference. Tooling: scripts/compile_probe.py (eager vs compiled GGUF probe), scripts/ perf_verify.py (the B200 verification above), and diffusion_bench.py gains --speed-mode so the speed tiers are benchmarkable. Tests: 183 passing (was 166); new coverage for the backend-flag snapshot/restore, GGUF compile eligibility, the balanced tiling/slicing split, native_speed_flags + the engine de-dup, and the sd-cli silent-hang timeout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): max tier uses max-autotune-no-cudagraphs + engine/lever benchmarks The opt-in `max` speed tier now compiles the repeated block with mode=max-autotune-no-cudagraphs (dynamic=False) instead of the default mode: Triton autotuning for GEMM/conv-heavier models, gated to the tier where a longer cold compile is acceptable. CUDA-graph modes (reduce-overhead / max-autotune) are deliberately avoided -- both crash on the regionally-compiled block (its static output buffer is overwritten across denoise steps), measured. Adds two reproducible benchmarks used to validate the optimization research: - scripts/compare_engines.py: PyTorch (diffusers GGUF) vs native sd.cpp head-to-head. - scripts/leverage_probe.py: coordinate_descent_tuning + FirstBlockCache probes. Measured on B200 (Z-Image Q4_K_M, 1024px, 8 steps): default compile 0.80s/gen; coordinate_descent_tuning 0.79s (within noise, already covered by max-autotune); FirstBlockCache does not run on Z-Image (diffusers 0.38 block-detection / Dynamo). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source) Add an opt-in transformer_quant mode that loads the dense bf16 transformer and torchao-quantises it onto the low-precision tensor cores, instead of the GGUF transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's 0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor. GGUF+compile stays the low-memory default and the fallback. The mode is gated on CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB); any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe (Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips the tiny projections that crash int8's torch._int_mm. New module mirrors diffusion_precision.py; quant runs before compile before placement. 184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py is the standalone torchao lever probe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): consumer-GPU tuning - lock fp8 fast accumulate, prefer fp8 over mxfp8, reject 2:4 sparsity Consumer Blackwell halves tensor-core throughput on FP32 accumulate (fp8 419 vs 838 TFLOPS with FP16 accumulate; bf16 209), so: - fp8 config locks use_fast_accum=True (Float8MMConfig). torchao already defaults it on; pinning it guards consumer cards against a default change. On B200 it is identical speed and slightly better quality (LPIPS 0.050 vs 0.091). - the Blackwell auto ladder prefers fp8 over mxfp8 (measured faster + more accurate). 2:4 semi-structured sparsity evaluated and rejected (scripts/sparse_accum_probe.py): 2:4 magnitude-prune + fp8 gives LPIPS 0.858 (broken image) with no fine-tune, the cuSPARSELt kernel errors on torch 2.9, and it does not compose with torch.compile (our main ~2x). Documented as a dead end, not shipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add fp8 fast-accum overflow verification probe scripts/fp8_overflow_check.py hooks every quantised linear during a real Z-Image generation and reports max-abs + non-finite counts for use_fast_accum True vs False. Confirms fast accumulation is an accumulation-precision knob, not an overflow one: across 276 linears, including Z-Image's ~1.0e6 activation peaks (which overflow FP16), 0 non-finite elements and identical max-abs for both modes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40; GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it. Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s), no overflow, quality below the quant noise floor. So the default leans to accuracy on data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force) lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off). 187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override threading). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add NVFP4 probe documenting it is not yet a win on torch 2.9 scripts/nvfp4_probe.py measures NVFP4 via torchao on the real Z-Image transformer. Finding (B200, 1024px/8 steps): NVFP4 is a torchao feature and DOES run with use_triton_kernel=False (the default triton path needs the missing MSLK library), but only at bf16-compile rate (0.667s vs fp8 0.592s) -- it dequantises FP4->bf16 rather than using the FP4 tensor cores. The real FP4 speedup needs MSLK or torch>=2.11 + torchao's CUTLASS FP4 GEMM. The smoke probe (default triton=True) already keeps NVFP4 out of auto on this env, so auto correctly stays on fp8; NVFP4 activates automatically once fast. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): prefer fp8 over nvfp4 in Blackwell auto ladder Validated NVFP4 on torch 2.11 + torchao CUTLASS FP4 in an isolated env. The FP4 tensor-core GEMM is genuinely active there (a 16384^3 GEMM hits ~3826 TFLOPS, 2.52x bf16 and 1.37x fp8), but it only beats fp8 on very large GEMMs. At the diffusion transformer's shapes (hidden ~3072, MLP ~12288, M~4096) NVFP4 is both slower (0.81x fp8 end to end on Z-Image 1024px) and less accurate (LPIPS 0.166 vs fp8's 0.044). Reorder the Blackwell auto ladder to fp8 before nvfp4 so auto is correct even on a future MSLK-equipped box; nvfp4 stays an explicit opt-in. Add scripts/nvfp4_t211_probe.py (extension diagnostics + GEMM micro + end-to-end). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): pre-quantized transformer loading The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs 13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline (scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True) the quantized weights, so the dense bf16 never touches the GPU. Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same torchao config + min_features filter the runtime path uses, applied ahead of time. New core/inference/diffusion_prequant.py (resolve_prequant_source + load_prequantized_transformer, best-effort, lazy imports). diffusion.py _load_dense_quant_pipeline tries the pre-quant source first and falls back to the dense materialise+quantise path, then to GGUF, so the default is unchanged. DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for the resolver, the meta-init+assign loader, and the backend branch selection + fallbacks; GPU verification via scripts/verify_prequant_backend.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): attention-backend selection Add a selectable attention kernel via the diffusers set_attention_backend dispatcher. Attention is memory-bandwidth bound, so a better kernel is an end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV matmuls torchao never touches) and composes with torch.compile. auto picks the best exact backend for the device: cuDNN fused attention (_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default (below the compile/quant noise floor); native SDPA elsewhere and when speed=off (so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/ xformers/aiter are honored, and an unavailable kernel falls back to the default rather than failing the load. New core/inference/diffusion_attention.py (normalize + per-device select + apply, best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline; attention_backend threads through begin_load / load_pipeline / status like the other load knobs. New request field attention_backend + status field. Hermetic CPU tests for normalize / select policy / apply fallback, plus route threading + 422. Measured via scripts/perf_levers_probe.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the arch tier to put int8 first on a consumer/workstation GPU (detected by the existing _is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first. Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation (-> int8) and data-center Ada/Hopper/Blackwell (-> fp8). * Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across denoise steps a DiT's output settles, so once the first block's residual barely changes the remaining blocks are skipped and their cached output reused. diffusers ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the standalone apply_first_block_cache hook as a fallback). Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 -> 2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar. OFF by default and a per-load opt-in: the win scales with step count, so it is for many-step models (Flux / Qwen-Image) and pointless for few-step distilled models (e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction of the trajectory. It composes with regional compile only with fullgraph=False (the cache's per-step decision is a torch.compiler.disable graph break), which the speed layer now switches to automatically when a cache is engaged. Best-effort: a model whose block signature the hook does not recognise is caught and the load proceeds uncached. - new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache (enable_cache / apply_first_block_cache fallback; threshold auto-raised for a quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import). - diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph when a cache is engaged. - diffusion.py: apply_step_cache before compile; thread transformer_cache / transformer_cache_threshold through begin_load -> load_pipeline and report the engaged mode in status(). - models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and transformer_cache_threshold request fields, engaged mode in the status response. - hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold selection, and best-effort failure handling, plus route threading + validation. - scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM / LPIPS vs the compiled no-cache baseline). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears) The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with 'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2 *_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing min_features filter did not exclude them. Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed, guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers (M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and quantises these layers fine, so the exclusion is int8-only. Sequence embedders (context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note 'context_embedder' contains the substring 'text_embed', which is why the token is the specific 'time_text_embed', not 'text_embed'. Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s (5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already working) are unchanged. - diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes exclude_name_tokens; quantize_transformer passes it for int8 only. - hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps attention / FFN / sequence-embedder linears), while fp8 keeps them. - scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's Linear layers and derive the exclusion list. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears) The prequant-checkpoint builder applied the dense quant filter without the int8-only M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8 checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now produces a working checkpoint on every supported model, giving int8 (the consumer-preferred scheme) the same ~2x load-VRAM and download reduction fp8 already had. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the native stable-diffusion.cpp engine instead of diffusers, with diffusers as the guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM. - diffusion_engine_router: centralised engine selection (built on the existing select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason. - sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface backed by sd-cli, with lazy binary install, registry-driven asset fetch, step-progress parsing, and cancellation. - diffusion_families: per-family single-file VAE + text-encoder asset mapping. - sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled). - routes/inference + gpu_arbiter: drive the active engine via the router; the API now reports the active engine and any fallback reason. - tests for the backend, router, route selection, and cancellation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler Address review feedback on #6724: - engine router: unload the engine being deactivated on a switch, so the old model is not left resident-but-unreachable (the evictor only targets the active engine). - generate route: sd.cpp execution errors (nonzero exit / timeout / missing output) now map to 500, not 409 (which only means not-loaded / cancelled). - native batch: return per-image seeds and persist the actual seed for each image so every batch image is reproducible. - Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the stable-diffusion.cpp docs; other families keep sd-cli defaults. - honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload offload flags on the native load instead of hardcoding them off. - fail the load when the sd-cli binary is present but not runnable (version() now returns None on exec error / nonzero exit). - size estimate: only treat the transformer asset as a possible local path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in load_prequantized_transformer ends in torch.load(weights_only=False), which executes arbitrary code from the pickle. The transformer_prequant_path load-request field reached that unpickle for any local file an authenticated caller named, so a request could trigger remote code execution. Refuse the source.kind=='path' branch unless the operator sets UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted and unaffected. Document the requirement on the API field and add gate tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa - apply_attention_backend now restores the native default when no backend is requested or a kernel fails. diffusers keeps a process-wide active attention backend that set_attention_backend updates, and a fresh transformer's processors follow it, so a load that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile load pinned, breaking the bit-identical/off guarantee. - select_attention_backend drops flash3/flash4 up front when the CUDA capability is below Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit request on the wrong card set fine then crashed mid-generation; it now falls back to native. - Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a valid alias of native) is accepted instead of 422-rejected by Pydantic. - Drop the dead replace('-','_') normalization (no alias uses dashes/underscores). - perf_levers_probe.py output dir is now relative to the script, not a hardcoded path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF - apply_step_cache now engages only via the transformer's native enable_cache (the diffusers CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model now runs uncached per the best-effort contract. - GGUF transformers are quantized (the default Studio load path), so they now use the higher quantized FBCache threshold when the caller leaves it unset, instead of the dense default that could keep the cache from triggering. - fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so fullgraph=True failed warmup and silently measured an eager cached run); output dir is now relative to the script, not a hardcoded path. * Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder _is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate. Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0). * Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just TypeError on older torchao), which would break the mxfp8 config helper instead of falling back to the default. Catch both so the fallback is robust. quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds CUDA memory during the per-row VRAM probe; output dir relative to the script. * Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load - snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the whole snapshot. restore_backend_flags restores each flag independently so one failure can't leave the others leaked process-wide. - load_pipeline restores the flags (and clears the GPU cache) when the build fails after apply_speed_optims mutated the process-wide flags but before _state captured them for unload to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and contaminated later off generations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs without producing output (or without closing stdout) would never reach proc.wait and the wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the PROCESS, so the main thread always enforces the timeout and kills a hung process (which closes the pipe and ends the reader). Add a test that times out even when stdout blocks, and make the no-binary test hermetic so a host-installed sd-cli can't leak in. * Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn The filter callback can be invoked without a module name, so fqn.lower() would raise AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens, so the linear is kept) instead of crashing the quantise pass. * Studio diffusion (Phase 16) review fixes: native engine robustness - sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64); a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds aliased to the same image. Random seeds stay 53-bit (JS-safe). - sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub fall back to anonymous instead of failing auth on a blank token. - sd_cpp_backend: a superseding load now cancels the in-flight generation, so the old sd-cli can no longer return/persist an image from the previous model. - diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a slow 10+ GB free / CUDA sync does not block engine selection. - diffusion_engine_router: probe sd-cli runnability (version()) before committing to native, so a present-but-unrunnable binary falls back to diffusers at selection. - diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a CPU-only install can still reach the native sd.cpp engine instead of failing load. - tests updated for the runnability probe + a not-runnable fallback case. * Studio diffusion (Phase 9) review fixes: prequant safety + validation - SECURITY: a request-supplied local pre-quant path is now unpickled only when it resolves inside an operator-configured ALLOWLIST of directories (UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in, once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on any path a load request named (arbitrary code execution). realpath() blocks symlink escapes; a bare on/off toggle is no longer a wildcard. - Validate the checkpoint's min_features against the runtime Linear filter, so a checkpoint that quantised a different layer set is rejected instead of silently loading a model that mismatches the dense path while reporting the same scheme. - Tolerant base_model_id compare (exact or same final path/repo segment), so a local path or fork of the canonical base is accepted instead of falling back to dense. - _has_meta_tensors uses any(chain(...)) (no intermediate lists). - prequant verify/probe scripts use repo-relative paths (+ env overrides), not the author's absolute /mnt paths. - tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts - diffusion_memory: when group offload is unavailable and the plan falls back to whole-module offload, enable VAE tiling (the group plan left it off, but the fallback is the low-VRAM path where the decode spike can OOM). Covers both the group and sequential fallback branches. - perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a balanced bit-identity regression actually fails the check instead of exiting 0. - compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and the load-progress poll has a 30 min deadline instead of looping forever on a hang. - test for the group->model fallback enabling VAE tiling. * Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path - diffusion: a torchao-quantized transformer is committed only compiled. A dense model resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF it replaced), so when transformer_quant engaged and speed resolved to off, promote to default (regional compile); warn loudly if compile still does not engage. - diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK); otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF. - nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path). - test asserts the eager-quant -> default-compile promotion. * Studio diffusion (Phase 10) review fixes: attention gating + probe isolation - diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation, so auto now stays on native SDPA there. - diffusion_attention: _active_attention_backend handles get_active_backend() returning an enum/None (not a tuple); the old unpack always raised and was swallowed, so the native-restore short-circuit never fired. - perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul so the inductor_flags variant doesn't leak into later compiled rows. - tests for the SM80 cuDNN gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening - install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket); extract through a per-member containment check (Zip-Slip guard); expanduser the --install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch the separately-published cudart runtime DLL archive so sd-cli.exe can start. - sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie. - tests: Zip-Slip rejection, normal extraction, studio-home discovery. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of the literal --output path. SdCppEngine.generate checked only the literal path, so a batch generation would exit 0 and then raise 'no image' (or return a stale file). generate now returns the literal path when present and otherwise falls back to the numbered siblings; single-image behavior is unchanged. Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected without error. * Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats Codex review on the native engine arg builder: - build_sd_cpp_command emitted --width/--height unconditionally, so an img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of the input. width/height are now Optional (None = unset): an image-conditioned run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives the size from the input image (set_width_and_height_if_unset); a plain txt2img run with unset dims keeps the prior 1024x1024 default; explicit dims are always honored. width/height are read only by the builder, so the type change is local. - build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...) that silently swallowed repeats=0 into sd-cli's default of one pass, turning an explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError and emits the flag for any explicit value != 1. Tests: img2img unset dims omit width/height (init_img and ref_images), explicit dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at the default. (Two pre-existing binary-discovery tests fail only because a real sd-cli is installed in this dev environment; unrelated to this change.) * Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc Codex review: the transformer_prequant_path field description still told operators to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the prior security fix made that variable a directory allowlist -- _allowed_prequant_roots deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator following the documented =1 would have every transformer_prequant_path request silently refused. The description now states it must name one or more allowlisted directories and that a bare on/off value is not accepted. Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does not say =1, and describes an allowlist/directory (guards against doc drift). * Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset Codex review on attention-backend selection: - Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first generation with no fallback. select_attention_backend now applies _cudnn_attention_supported() to an explicit cuDNN request too. - flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3 is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a (min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+. - apply_attention_backend's success path left diffusers' process-wide active backend pinned to the kernel it set; a later component whose processors are unconfigured (backend None) would inherit it. It now resets the global registry to native after a successful per-transformer set (the transformer keeps its own backend), best-effort. Also fixed _active_attention_backend: get_active_backend() returns a (name, fn) tuple, so the prior code stringified the tuple and never matched a name, defeating the native-restore short-circuit. Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on SM90; global registry reset after a successful set; _active_attention_backend reads the tuple return. * Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300 (Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So _is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they are treated as data-center class and keep the intended fp8-first behavior. Test: extends the datacenter parametrize with 'NVIDIA B300' and 'NVIDIA GH200 480GB' (now _is_consumer_gpu False). * Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder Codex review: the M=1 modulation/embedder exclusion was wired only into the dense runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint quantised the AdaLN modulation and conditioning-embedder linears, and loading it via transformer_prequant_path (the load path only loads already-quantised tensors, it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes for the runtime path. Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 -> the M=1 exclusion, every other scheme -> none) and use it in both the runtime quantiser and the builder, so a prequant artifact's quantised-layer set always matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion). Test: int8_exclude_name_tokens returns the exclusion for int8 and () for fp8/nvfp4/mxfp8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race Codex review on the native-engine routing: - The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat) unconditionally after engine selection. A native sd.cpp load on a pure-CPU host never touches the GPU, so that needlessly tore down the resident chat model. The handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it. - sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when _run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so the setting was unverifiable. status now derives them from state.offload_flags (still 'none' on CPU, where the flags are empty). - _run_load committed the new state without cancelling/waiting on a generation that started during the (slow) asset download, so a stale sd-cli run against the OLD model could finish afterward and persist an image from the previous model once the new load reported ready. The commit now signals the in-flight cancel and waits on _generate_lock before swapping _state (taken only at commit, so the download never serialises against generation), mirroring the diffusers load path. Tests: CPU native load skips the arbiter while a GPU native load takes it; status reports offload active when flags are set; _run_load cancels and waits for an in-flight generation before committing. * Studio diffusion (Phase 14) review round 2: align helper name with the stack Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the identical helper already present higher in the diffusion stack (Phase 16). The helper definition, the runtime quantiser call, and the offline builder are now byte-identical to that version, so the two branches no longer introduce a divergent name for the same single-source-of-truth and the stack merges without a conflict on this fix. No behavior change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: eager patches + torch.compile cache speed phase Adds the opt-in speed path for the GGUF diffusion transformer behind a selectable speed mode (default off, so output is unchanged until a profile is chosen): - diffusion_eager_patches.py: shared eager fast-paths (channels_last, attention/backend selection, fused norms and QKV) installed at load and rolled back on unload or failed load. - diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent torch.compile cache and the GGUF-transformer compile wiring. - diffusion_arch_patches.py: architecture-specific patches. - diffusion_patch_backend.py: shared install/restore plumbing. - diffusion_speed.py: speed-profile planning. Tests for each module plus the benchmarking and probe scripts used to measure speed, memory, and accuracy of the path. * Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI Backend: - Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file fp8 transformers, gated to the unsloth org plus a curated allowlist. - Image-conditioned workflows built with Pipeline.from_pipe so they reuse the loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint, outpaint, and a hires-fix upscale pass. - Instruction editing as its own family kind (Qwen-Image-Edit-2511, FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and multi-reference) plus klein inpaint. - Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the decoded image size and cap upscale output to avoid OOM on large inputs. - Fixes: from_pipe defaulting to a float32 recast that crashed torchao quantized transformers; image-conditioned calls forcing the slider size onto the input image. Native sd.cpp engine rejects image-conditioned and reference requests it cannot serve. Frontend: - Redesigned Images page with capability-gated workflow tabs (Create, Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor, client-side outpaint, and a multi-reference picker. - Advanced options moved to a right-docked panel mirroring Chat: closed by default, toggled by a single fixed top-bar button that stays in place. sd.cpp installer: pin the release, verify each download's sha256, add a download timeout, and make the source repo configurable for a future mirror. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: correct the Advanced panel comment (closed by default, fixed toggle) * Studio diffusion: LoRA adapters for the Images workflow Add community LoRA support across both diffusion backends, the single biggest step toward broad image-workflow coverage. Backend - New shared module core/inference/diffusion_lora.py: adapter discovery (local scan + curated catalog + owner/name[:file] Hub refs), download via hf_hub_download_with_xet_fallback, alias sanitization, native managed-dir materialization with collision-broken aliases, prompt-tag injection (deduped against user-typed tags), and a supports_lora gate. - Native sd-cli: resolve + materialize selected LoRAs into a per-run managed dir, inject <lora:ALIAS:w> tags, pass --lora-model-dir with --lora-apply-mode auto. The arg builder already emitted these flags. - Diffusers: non-fused load_lora_weights + set_adapters manager, tracked on the pipe so an unchanged selection is a no-op and a model swap resets; cleared on unload. Never fuses (breaks quantized transformers and blocks live weight tweaks). - Gated off where unsupported: torchao fp8/int8 dense, GGUF-via-diffusers, and native Qwen-Image (no LoRA name-conversion branch upstream). - Request contract: optional loras on DiffusionGenerateRequest; empty or omitted is identical to today. supports_lora surfaced in status; chosen LoRAs persisted in gallery recipe metadata. - New GET /api/models/diffusion-loras for the picker (family-filtered). Frontend - Repeatable multi-LoRA picker (adapter select + weight slider 0..2 + remove), gated by the loaded model's supports_lora and family, max 8. Tests - New test_diffusion_lora.py (14): helpers, request validation, native tag/dir wiring, diffusers set_adapters manager, supports_lora matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a stray "gguf" tag on the Hub but ship no .gguf files. The model search classified them as GGUF from the bare tag, so the picker rendered the GGUF variant expander, which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata (populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected. * Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline load when the id is unsloth-hosted or on-device (the backend infers the family + base repo and gates loads to unsloth/* or local paths), and show a clear message otherwise instead of silently ignoring the click. Curated and GGUF paths are unchanged. * Studio Images: keep curated safetensors models in Recommended after download The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet), so they vanished from the picker after the first load and could only be found by typing an exact search. The row already renders a downloaded badge, matching how GGUF Recommended rows stay visible when cached. Drop the exclusion so the curated safetensors always list. * Studio diffusion LoRA: sanitize dots out of adapter aliases The LoRA alias is used as the diffusers PEFT adapter name, and PEFT rejects names containing "." (module name can't contain "."). sanitize_alias kept dots, so a LoRA whose filename carries a version tag (e.g. Qwen-Image-2512-Lightning-8steps-V1.0-bf16) failed to apply with a 400. Replace dots too; the alias stays a valid native <lora:NAME:w> filename stem. Adds regression coverage for internal dots. * Studio Images: clarify the GGUF transformer-quant Advanced control Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode" with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/ FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF, falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only. * Studio Images: list on-device unsloth diffusion models in the picker The Images picker's On Device tab hid every non-GGUF cached repo whenever a task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit and FP8 safetensors) never showed up there. List cached repos that pass the task gate, limited under a filter to unsloth-hosted ones so base repos (which fail the diffusion load trust gate) don't appear only to dead-end on click. Chat behavior is unchanged: the task gate still drops image repos there. * Studio: hide single-file image checkpoints from the chat model picker The chat picker treats a cached repo as an image model, and hides it, only when it ships a diffusers model_index.json. Single-file, ComfyUI, and ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models. Fall back to resolving the repo id against the known diffusion families, the same resolver the Images backend loads from, so these checkpoints are tagged text-to-image and stay in the Images picker only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: add the FLUX.2-dev model family Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline) resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline + Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated, reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror. text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from klein's turbo defaults. Verified live: GGUF load resolves the family + gated base repo and generates a real 1024x1024 image on GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: clearer error for an unsupported diffusion model When a repo id resolves to no diffusion family the load raised 'Could not infer a diffusion family... Pass family_override (z-image)', which points at an unrelated family and doesn't say what is supported. Replace it with a message that lists the supported families (from a new supported_family_names helper) and notes that video models and image models whose diffusers transformer has no single-file loader are not supported. Applies to both the diffusers and native sd.cpp load paths. Also refreshes two stale family-registry comments that still called FLUX.2-dev omitted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove stray async task scratch outputs committed by mistake * Diffusion LoRA: harden resolution, native tag precedence, and diffusers teardown Address review findings on the LoRA path: - resolve_one: normalise a blank/whitespace hf_token to None (anonymous access) and reject a client-supplied weight file with traversal / absolute path. - resolve_specs: convert FileNotFoundError from an unknown/stale id to ValueError so the route returns 400 instead of a generic 500. - _scan_local: disambiguate local adapters that share a stem (foo.safetensors vs foo.gguf) so each is uniquely addressable. - inject_prompt_tags: the backend-validated weight now wins over a user-typed <lora:ALIAS:...> for a selected adapter; unselected user tags are left alone. - diffusers _apply_loras: reject a .gguf adapter with a clear error before touching the pipe (diffusers loads safetensors only). - _unload_locked: drop the explicit unload_lora_weights() on teardown; the pipe is dropped wholesale (freeing adapters), so the previous call could race an in-flight denoise on the same pipe. - Images page: use a stable LoRA key and clear the selection (not just the options) when the catalog refresh fails. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Diffusion: guard trust check against OSError and validate conditioning inputs - _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid characters (or a bare owner/name id) can't raise OSError; treat any failure as not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request still raises the clear FileNotFoundError for a genuinely missing local pick. - generate(): reject mask_image / upscale / reference_images supplied without an input image, and reject reference_images on a family that does not support reference conditioning, instead of silently degrading to txt2img / img2img. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address Codex review findings on the image-workflows PR Keep diffusion.py importable without torch: the compile/arch patch modules import torch at module level, so import them lazily at their load/unload call sites instead of at module load. This restores the torchless contract so get_diffusion_backend() works on a CPU/native sd.cpp install. Match family reject keywords and aliases as whole path/name segments, not raw substrings, so an unrelated word like edited, edition, or kontextual no longer misroutes or hides a valid base image model, while supported edit families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same segment matching in the picker task filter. Route FLUX.2-dev native guidance through --guidance like the other FLUX families rather than --cfg-scale. Reject native upscale requests that have no input image. Read image header dimensions and reject over-limit inputs before decoding pixels, so a crafted small-payload image cannot spike memory. Reject an upscale that would shrink the source below its input size. Validate the model_kind against the filename extension before the GPU handoff. Estimate a local diffusers pipeline's size from its on-disk weights so auto memory planning does not skip offload and OOM. Report workflows: [txt2img] from the native backend status so the Create tab stays enabled for a loaded native model. Clamp the outpaint canvas to the backend's 4096px decode limit. Adds regression tests for segment matching and kind/extension validation. * Harden diffusion LoRA handling on the diffusers and native paths Reject LoRA on a torch.compile'd diffusers transformer (Speed=default/max): diffusers requires the adapter loaded before compilation, so applying one to the already-compiled module fails with adapter-key mismatches. The status gate now hides the picker and generate raises a clear message instead. Convert a cancelled Hub LoRA download (RuntimeError Cancelled) to the diffusion cancellation sentinel in resolve_specs, so an unload/superseding load during resolution maps to a 409 instead of a generic server error. Drop weight-0 LoRA rows before the native support gate so a request carrying only disabled adapters stays a no-op on families where native LoRA is unsupported, matching the diffusers path. Reject duplicate LoRA ids in the request model: both apply paths suffix colliding names, so a repeated id would stack the same adapter past its per-adapter weight bound. Strip all user-typed <lora:...> prompt tags on the native path (only the selected adapters are materialized in the managed lora-model-dir, so an unselected tag can never resolve), and restore saved LoRA selections from a gallery recipe so restore reproduces a LoRA image. * Address further Codex findings on the image-workflows PR - Persist the actual output image size in the gallery recipe instead of the request sliders: Transform/Inpaint/Edit derive the size from the uploaded image, Extend grows the canvas, and Upscale resizes it, so the sliders recorded (and later restored) the wrong dimensions for those workflows. - Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file name) in validate_load_request, so the unloadable pick fails before chat is evicted rather than deep in from_pretrained. - Only publish an image-conditioned from_pipe wrapper to the shared aux cache when the load is still current: from_pipe runs under the generate lock but not the state lock, so an unload racing its construction could otherwise cache a wrapper over torn-down modules that a later load would reuse. - Verify the Windows CUDA runtime archive checksum before extracting it, like the main sd-cli archive, so a corrupt or tampered runtime is rejected rather than extracted next to the binary. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
Ayushman
|
c356427f30
|
Guard Windows ROCm torchao override skip (#6837)
Some checks failed
Studio GGUF CI / JSON, images (push) Has been cancelled
Studio load-orchestrator CI / test (push) Has been cancelled
Mac Studio API CI / Studio API & Auth Tests (push) Has been cancelled
Mac Studio GGUF CI / OpenAI, Anthropic API tests (push) Has been cancelled
Mac Studio GGUF CI / Tool calling Tests (push) Has been cancelled
Mac Studio GGUF CI / JSON, images (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-14) (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-15) (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-26) (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-15-intel) (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-26-intel) (push) Has been cancelled
Mac Studio UI CI / Chat UI Tests (push) Has been cancelled
Mac Studio Update CI / Studio Updating Tests (push) Has been cancelled
Studio Tauri CI / Tauri Linux debug build (no codesign) (push) Has been cancelled
Studio UI CI / Chat UI Tests (push) Has been cancelled
Studio Update CI / Studio Updating Tests (push) Has been cancelled
Windows Studio API CI / Studio API & Auth Tests (push) Has been cancelled
Windows Studio GGUF CI / OpenAI, Anthropic API tests (push) Has been cancelled
Windows Studio GGUF CI / Tool calling Tests (push) Has been cancelled
Windows Studio GGUF CI / JSON, images (push) Has been cancelled
Windows Studio GGUF CI / Studio install + inference without Visual Studio (push) Has been cancelled
Windows Studio GGUF CI / GPU prebuilt resolves without Visual Studio (push) Has been cancelled
Windows Studio GGUF CI / setup.ps1 unit tests (VS 2026 / CMake guard) (push) Has been cancelled
Windows Studio GGUF CI / real-VS detection (VS 2022) (push) Has been cancelled
Windows Studio GGUF CI / real-VS detection (VS 2026) (push) Has been cancelled
Windows Studio GGUF CI / VC++ runtime detect + install round-trip (windows-2025-vs2026) (push) Has been cancelled
Windows Studio GGUF CI / VC++ runtime detect + install round-trip (windows-latest) (push) Has been cancelled
Windows Studio UI CI / Chat UI Tests (push) Has been cancelled
Windows Studio Update CI / Studio Updating Tests (push) Has been cancelled
Wheel CI / Wheel build + content sanity + import smoke (push) Has been cancelled
* Fix: skip fp16/bf16 validation for full finetuning in RL trainers When doing full finetuning (FFT) of a bfloat16 model, the fp16/bf16 mismatch validation fires before the corrective logic runs, causing a misleading error even though the code would properly handle it downstream. Skip the validation when full_finetuning is active. Fixes #6731 * Fix: auto-correct fp16/bf16 mismatches for full finetuning before validation Instead of entirely skipping validation (which could let mismatches through when mixed_precision_dtype is float32), auto-correct explicit fp16/bf16 settings that conflict with the model's dtype for FFT. This way the existing validation still catches real mismatches for non-FFT cases, and the corrective logic below handles the normalized settings. Fixes the issue raised in Codex review of PR #6813. * Guard Windows ROCm torchao override skip Detect installed ROCm torch directly before applying the torchao override so Windows ROCm environments never install the crashing torchao package even if the earlier ROCm-installed flag is missing. * Update unsloth/models/rl.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update studio/install_python_stack.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Harden ROCm probe and sync RL precision flags Tolerate stray stdout noise when probing Windows ROCm torch installs by checking the last non-empty output line, matching the existing torch version probe behavior. Also keep args.fp16 and args.bf16 synchronized with the full-finetuning precision auto-corrections in the RL trainer patch so downstream eval settings see a consistent TrainingArguments state. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add MLX trainer compatibility shims Patch imported MLXTrainer and MLXTrainingConfig objects to preserve the expected dataclass field ordering and to provide a _train_dataset_for_batches fallback when older trainers or test doubles only expose train_dataset. Also add focused worker tests covering both compatibility paths. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope PR to Windows ROCm torchao guard * Restore PR scope to Windows ROCm guard * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * test: cover Windows ROCm torchao skip behavior * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Ayushman Paul <ayushman@HP> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com> |
||
ramisworld
|
01f7e14988
|
Fix Studio custom folders on Linux external drives (#6799)
* Fix external drive custom folder selection * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update studio/backend/tests/test_linux_external_media_paths.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep legacy media scan validation strict * Apply sensitive-dir denylist to legacy folder browser for PR #6799 The legacy /api/models browse endpoint gained the new /run/media mount roots in its allowlist but not the credential/config guard that scan-folder registration and the Hub browser already enforce. Filter sensitive names during enumeration and reject them in _resolve_browse_target so .ssh, .aws, .config, etc. under allowlisted roots stay unbrowseable, matching the Hub browser. Add a public contains_sensitive_path_component helper and cover the legacy resolver with a regression test. * Trim redundant comments in PR #6799 changes * Skip sensitive Linux media roots * Reject sensitive dirs at exact browse roots for PR #6799 Both _resolve_browse_target functions only checked contains_sensitive_path_component while walking descendant parts, so requesting an allowlisted root itself (empty relative path) returned it unchecked. A pre-existing scan-folder row under ~/.ssh, ~/.aws, ~/.config, etc. (registerable before the denylist was added) is re-added to the allowlist on upgrade and could then be browsed. Check the resolved target once before returning in both the legacy and Hub browsers, and cover the root case in both test suites. * fix: avoid unused path helper reexports * fix: import sensitive path helpers directly * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> |
||
|
Daniel Han
|
4562b537f8
|
Studio diffusion: fix FP8 transformer quant producing noise (per-row scaling) (#6772)
* Studio diffusion: cross-platform device policy, fp16 guard, lock split, validate-before-evict Phase 1 of porting the richer diffusion stack onto the image-generation backend. - Add a compartmentalized device/dtype policy module (diffusion_device.py) resolving CUDA/ROCm/XPU/MPS/CPU with capability flags. Keeps the NVIDIA capability-based bf16 choice; ROCm and XPU are isolated; MPS uses bf16 or fp32, never a silent fp16 that renders a black image. - Add a per-family fp16_incompatible flag (Z-Image) and promote a resolved float16 to float32 for those families so they do not produce black images. - Split the backend locks: a generation holds only _generate_lock, so status, unload, and a new load are never blocked by a long denoise. Add per-generation cancellation via callback_on_step_end so an eviction or a superseding load preempts a running generation; a replacement load waits for it to stop before allocating, so two pipelines never sit in VRAM at once. - Validate a load request before the GPU handoff so an unloadable pick never evicts a working chat model, and reject missing local paths up front. - Add CPU-only tests for the device policy, dtype guard, lock split and cancellation, and validate-before-evict, plus a GPU benchmark/regression script (scripts/diffusion_bench.py) measuring latency, peak VRAM, and PSNR against a saved reference. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2A): measured-budget memory planner + offload/VAE policy Add a lean, backend-agnostic memory policy that picks a CPU-offload policy and VAE tiling/slicing from measured free device memory vs the model's estimated resident footprint, then applies it to the built pipeline. auto stays resident when the model fits (byte-identical to the prior resident path), and falls to whole-module offload when tight; fast/balanced/low_vram are explicit overrides. Sequential submodule offload is unreliable for GGUF transformers on diffusers 0.38, so it falls back to whole-module offload and status reports the policy actually engaged. Verified on Z-Image-Turbo Q4_K_M (B200): auto reproduces the resident image with no VRAM/latency regression (PSNR inf); balanced/low_vram cut generation peak VRAM 47.9% (15951 -> 8318 MB) with byte-identical output, at the expected latency cost. 73 prior + 35 new CPU tests pass. * Studio diffusion (Phase 2D): streamed block-level offload + functional VAE tiling Add a streamed 'group' offload tier (diffusers apply_group_offloading, block_level, use_stream) that keeps the transformer flowing through the GPU a few blocks at a time while the text encoder / VAE stay resident, and fix VAE tiling to drive the VAE submodule (pipelines like Z-Image expose enable_tiling on pipe.vae, not the pipeline). apply_memory_plan now returns the (policy, tiling) actually engaged so status never overstates either, and group falls back to whole-module offload when the transformer can't be streamed. Measured on Z-Image (B200), all lossless (PSNR inf vs resident): balanced/group cuts generation peak VRAM 32% (15951 -> 10840 MB) at near-resident speed (2.07 -> 2.99s); low_vram/model cuts it 48% (-> 8318 MB) but is slower (7.99s). Mode names now match that tradeoff: balanced = stream the transformer, low_vram = offload every component. auto picks group when the companions fit resident, else model. 112 CPU tests pass. * Studio diffusion (Phase 5): image quality-vs-quant accuracy harness Add scripts/diffusion_quality.py, the accuracy analogue of the KLD workflow: hold prompt + seed fixed, render a grid with a reference quant (default BF16), then render each candidate quant and measure drift from the reference. Records mean PSNR + SSIM (pure-numpy, no skimage/scipy) and optional CLIP text-alignment + image-similarity (transformers, --clip), plus file size, latency, and peak VRAM, then prints a quality-vs-cost table and recommends the smallest quant within a quality budget. --selftest validates the metrics on synthetic images with no GPU or model. Verified on Z-Image (B200): the table degrades monotonically with quant size (Q8 -> Q4 -> Q2: PSNR 21.7 -> 15.5, SSIM 0.82 -> 0.61), while CLIP-text stays flat (~0.34) -- quantization erodes fine detail far more than prompt adherence. * Studio diffusion (Phase 3): opt-in speed layer (channels_last / compile / TF32) Add a speed_mode knob (off by default, so the render path stays bit-identical): default applies channels_last VAE + regional torch.compile of the denoiser's repeated block where eligible; max also enables TF32 matmul and fused QKV. Regional compile is gated off for the GGUF transformer (dequantises per-op) and for families flagged not compile-friendly (a new supports_torch_compile flag, False for Z-Image), so it activates automatically only once a non-GGUF bf16 transformer is loaded. Speed optims run before placement/offload, per the diffusers composition order. status now reports speed_mode + the optims actually engaged. Verified on Z-Image (B200): default -> ['channels_last'], max -> ['channels_last', 'tf32'], compile correctly skipped for GGUF; generation works in every mode. 121 CPU tests pass. * Studio diffusion (Phase 2B): opt-in fp8 text-encoder layerwise casting Add a text_encoder_fp8 knob that casts the companion text encoder(s) to fp8 (e4m3) storage via diffusers apply_layerwise_casting, upcasting per layer to the bf16 compute dtype while normalisations and embeddings stay full precision. Applied before placement, gated to CUDA + bf16, best-effort (a failure leaves the encoder dense). status reports which encoders were cast. Verified on Z-Image (B200, balanced/group mode where the encoder stays resident): generation peak VRAM dropped 37% (10840 -> 6791 MB, below the lowest-VRAM offload) at near-resident speed. It is a memory-vs-quality tradeoff, not free -- ~20 dB PSNR vs the bf16 encoder, a larger shift than one transformer quant step -- so it is off by default and documented as such, with the Phase 5 harness to size the cost. 127 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2C): NVFP4 text-encoder quant (+ generalise fp8 knob) Generalise the text-encoder precision knob from a fp8 bool to text_encoder_quant (fp8 | nvfp4). nvfp4 quantises the companion text encoder to 4-bit via torchao NVFP4 weight-only (two-level microscaling) on Blackwell's FP4 tensor cores; fp8 stays the broader-hardware path (cc>=8.9). Both are gated, best-effort, and run before placement; status reports the mode actually engaged. This is the lean realisation of GGUF-native text-encoder quant: 4-bit on the encoder without the 3045-line port. Verified on Z-Image (B200, balanced/group where the encoder stays resident), vs the bf16 encoder: nvfp4 cut generation peak VRAM 48% (10840 -> 5593 MB, the lowest TE option, below whole-model offload) at near-fp8 quality (16.4 vs 17.1 dB PSNR), and both quants ran faster than bf16. A memory-vs-quality tradeoff (off by default); size it per model with the Phase 5 quality harness. diffusion_bench gains --text-encoder-quant. 129 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): native stable-diffusion.cpp engine for CPU/Mac Adds the CPU / Apple-Silicon tier of the two-engine strategy, mirroring the chat backend's llama.cpp shell-out. Diffusers stays the default on CUDA / ROCm / XPU; this covers the hardware diffusers serves poorly, consuming the same split GGUF assets Studio already curates. - sd_cpp_args.py: pure sd-cli command builder. Maps the family to its text-encoder flag (Z-Image Qwen3 to --llm, Qwen-Image to --qwen2vl, FLUX.1 CLIP-L + T5), and the diffusers memory policy (none/group/model/sequential) to sd.cpp's offload flags (--offload-to-cpu / --clip-on-cpu / --vae-on-cpu / --vae-tiling / --diffusion-fa), so one user knob drives both engines. - sd_cpp_engine.py: SdCppEngine over a located sd-cli. find_sd_cpp_binary() with the same precedence as the llama finder (env override, then the Studio install root, then in-tree, then PATH), an is_available/version probe, and a one-shot subprocess generate that streams progress and returns the PNG. runtime_env() prepends the binary's directory to the platform library path so a prebuilt's bundled libstable-diffusion.so resolves. select_diffusion_engine() is the pure routing decision (GPU backends to diffusers, CPU/MPS to native when present). - install_sd_cpp_prebuilt.py: resolve + download the per-host prebuilt (macOS-arm64/Metal, Linux x86_64 CPU, Vulkan/ROCm/Windows variants) into the Studio install root. resolve_release_asset() is a pure, unit-tested host-to-asset matrix. - scripts/sd_cpp_smoke.py: end-to-end native generation harness. Tests (CPU-only, subprocess/filesystem stubbed): 49 new across args, engine, routing, runtime env, and the installer resolver. Full diffusion suite 166 passing. Verified on a B200 box: built sd-cli (CUDA) and the prebuilt (CPU) both generate Z-Image-Turbo Q4_K end to end through SdCppEngine: balanced (group offload, 5.0s gen), low_vram (full CPU offload + VAE tiling, 13.4s), and the dynamically-linked CPU prebuilt (50.4s on CPU), all producing coherent images. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 6): img2img / inpaint / edit / LoRA / upscale on the native engine Builds on Phase 4's native stable-diffusion.cpp engine, extending it from text-to-image to the wider feature surface, since sd.cpp supports all of these through the binary already. Pure command-builder additions plus one engine method, so the txt2img path is unchanged. - sd_cpp_args.py: SdCppGenParams gains image-conditioning fields. init_img + strength make a run img2img, adding mask makes it inpaint, ref_images drives FLUX-Kontext / Qwen-Image-Edit style editing (repeated --ref-image), and lora_dir + the <lora:name:weight> prompt syntax select LoRAs. New SdCppUpscaleParams + build_sd_cpp_upscale_command for the ESRGAN upscale run mode (input image + esrgan model, no prompt / text encoders). - sd_cpp_engine.py: the subprocess runner is factored into a shared _run() so generate() (now carrying the conditioning flags) and a new upscale() reuse the same streaming / error / output-check path. - scripts/sd_cpp_smoke.py: --task {txt2img,img2img,upscale} with --init-img / --strength / --upscale-model / --upscale-repeats. Tests: 10 new across the img2img / inpaint / edit / LoRA flag construction, the upscale builder and its validation, and the engine's img2img + upscale paths. Full diffusion suite 176 passing. Verified on a B200 box through SdCppEngine: img2img (Z-Image-Turbo Q4_K, the init image conditioned at strength 0.6, 4.8s) and ESRGAN upscale (512x512 -> 2048x2048 via RealESRGAN_x4plus_anime_6B, 2.7s), both producing coherent images. Video and the diffusers-path feature wiring are deferred. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): accuracy-preserving speed pass Re-review of the diffusion stack (#6675/#6679/#6680) surfaced one real accuracy bug and a dead-on-arrival speed path; this fixes both and adds the lossless / near-lossless wins, all measured on a B200. Correctness: - TF32 global-state leak (fix). speed_mode=max flipped torch.backends.*.allow_tf32 process-wide and never restored them, so a later `off` load silently inherited TF32 and was no longer bit-identical. Added snapshot_backend_flags / restore_backend_flags (TF32 + cudnn.benchmark), captured before the speed layer runs and restored on unload. Verified: load max -> unload -> load off is now byte-identical (PSNR inf) to a fresh off. - sd-cli timeout could hang forever. _run() blocked in `for line in stdout` and only checked the timeout after EOF, so a child stuck in model load / GPU init with no output ignored the timeout. Drained stdout on a reader thread with a wall-clock deadline. Added a silent-hang regression test. Speed (diffusers path), near-lossless, opt-in tiers: - Regional torch.compile now runs on the GGUF transformer. The is_gguf gate (and Z-Image's supports_torch_compile=False) were stale: compile_repeated_blocks compiles and runs ~2.2x faster on the GGUF Z-Image transformer on torch 2.9.1 / diffusers 0.38 (the per-op dequant stays eager, the rest of the block compiles). Measured: off 1.80s -> default 0.82s/gen (+54.7%), PSNR 37.7 dB vs eager -- far above the Q4 quant noise floor (~21 dB), so it does not move output quality. Gate relaxed; default tier delivers it. - cudnn.benchmark added to the default tier (autotunes the fixed-shape VAE convs). - torch.inference_mode() around the pipeline call (lossless, strictly faster than the no_grad diffusers uses internally). Memory path: - VAE tiling (not bit-identical >1MP) restricted to the model/sequential/CPU tiers; the balanced (group) tier keeps exact slicing only, so it is now bit-identical to the resident image (verified PSNR inf) and slightly faster. - Group offload adds non_blocking + record_stream on the CUDA stream path to overlap each block's H2D copy with compute (lossless; gated on the installed diffusers signature so older versions still work). Native (sd.cpp) path: - native_speed_flags: a first-class speed knob (default -> --diffusion-fa, a near-lossless CUDA win that was previously only added on offload tiers; max also -> --diffusion-conv-direct). conv-direct stays opt-in: measured +45% on CUDA, so it is never auto-on. Engine generate() merges it, de-duped against offload flags. Default profile: a GGUF model with no explicit speed_mode now resolves to the `default` profile (resolve_speed_mode), since compile's perturbation sits below the quantisation noise floor and so does not reduce quality versus the dense reference; out of the box a GGUF Z-Image generation drops from 1.80s to 0.81s. Dense models stay `off` / bit-identical, and an explicit speed_mode -- including "off" -- is always honored, so the byte-identical path remains one flag away and is the regression reference. Tooling: scripts/compile_probe.py (eager vs compiled GGUF probe), scripts/ perf_verify.py (the B200 verification above), and diffusion_bench.py gains --speed-mode so the speed tiers are benchmarkable. Tests: 183 passing (was 166); new coverage for the backend-flag snapshot/restore, GGUF compile eligibility, the balanced tiling/slicing split, native_speed_flags + the engine de-dup, and the sd-cli silent-hang timeout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): max tier uses max-autotune-no-cudagraphs + engine/lever benchmarks The opt-in `max` speed tier now compiles the repeated block with mode=max-autotune-no-cudagraphs (dynamic=False) instead of the default mode: Triton autotuning for GEMM/conv-heavier models, gated to the tier where a longer cold compile is acceptable. CUDA-graph modes (reduce-overhead / max-autotune) are deliberately avoided -- both crash on the regionally-compiled block (its static output buffer is overwritten across denoise steps), measured. Adds two reproducible benchmarks used to validate the optimization research: - scripts/compare_engines.py: PyTorch (diffusers GGUF) vs native sd.cpp head-to-head. - scripts/leverage_probe.py: coordinate_descent_tuning + FirstBlockCache probes. Measured on B200 (Z-Image Q4_K_M, 1024px, 8 steps): default compile 0.80s/gen; coordinate_descent_tuning 0.79s (within noise, already covered by max-autotune); FirstBlockCache does not run on Z-Image (diffusers 0.38 block-detection / Dynamo). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source) Add an opt-in transformer_quant mode that loads the dense bf16 transformer and torchao-quantises it onto the low-precision tensor cores, instead of the GGUF transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's 0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor. GGUF+compile stays the low-memory default and the fallback. The mode is gated on CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB); any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe (Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips the tiny projections that crash int8's torch._int_mm. New module mirrors diffusion_precision.py; quant runs before compile before placement. 184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py is the standalone torchao lever probe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): consumer-GPU tuning - lock fp8 fast accumulate, prefer fp8 over mxfp8, reject 2:4 sparsity Consumer Blackwell halves tensor-core throughput on FP32 accumulate (fp8 419 vs 838 TFLOPS with FP16 accumulate; bf16 209), so: - fp8 config locks use_fast_accum=True (Float8MMConfig). torchao already defaults it on; pinning it guards consumer cards against a default change. On B200 it is identical speed and slightly better quality (LPIPS 0.050 vs 0.091). - the Blackwell auto ladder prefers fp8 over mxfp8 (measured faster + more accurate). 2:4 semi-structured sparsity evaluated and rejected (scripts/sparse_accum_probe.py): 2:4 magnitude-prune + fp8 gives LPIPS 0.858 (broken image) with no fine-tune, the cuSPARSELt kernel errors on torch 2.9, and it does not compose with torch.compile (our main ~2x). Documented as a dead end, not shipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add fp8 fast-accum overflow verification probe scripts/fp8_overflow_check.py hooks every quantised linear during a real Z-Image generation and reports max-abs + non-finite counts for use_fast_accum True vs False. Confirms fast accumulation is an accumulation-precision knob, not an overflow one: across 276 linears, including Z-Image's ~1.0e6 activation peaks (which overflow FP16), 0 non-finite elements and identical max-abs for both modes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40; GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it. Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s), no overflow, quality below the quant noise floor. So the default leans to accuracy on data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force) lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off). 187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override threading). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add NVFP4 probe documenting it is not yet a win on torch 2.9 scripts/nvfp4_probe.py measures NVFP4 via torchao on the real Z-Image transformer. Finding (B200, 1024px/8 steps): NVFP4 is a torchao feature and DOES run with use_triton_kernel=False (the default triton path needs the missing MSLK library), but only at bf16-compile rate (0.667s vs fp8 0.592s) -- it dequantises FP4->bf16 rather than using the FP4 tensor cores. The real FP4 speedup needs MSLK or torch>=2.11 + torchao's CUTLASS FP4 GEMM. The smoke probe (default triton=True) already keeps NVFP4 out of auto on this env, so auto correctly stays on fp8; NVFP4 activates automatically once fast. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): prefer fp8 over nvfp4 in Blackwell auto ladder Validated NVFP4 on torch 2.11 + torchao CUTLASS FP4 in an isolated env. The FP4 tensor-core GEMM is genuinely active there (a 16384^3 GEMM hits ~3826 TFLOPS, 2.52x bf16 and 1.37x fp8), but it only beats fp8 on very large GEMMs. At the diffusion transformer's shapes (hidden ~3072, MLP ~12288, M~4096) NVFP4 is both slower (0.81x fp8 end to end on Z-Image 1024px) and less accurate (LPIPS 0.166 vs fp8's 0.044). Reorder the Blackwell auto ladder to fp8 before nvfp4 so auto is correct even on a future MSLK-equipped box; nvfp4 stays an explicit opt-in. Add scripts/nvfp4_t211_probe.py (extension diagnostics + GEMM micro + end-to-end). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): pre-quantized transformer loading The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs 13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline (scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True) the quantized weights, so the dense bf16 never touches the GPU. Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same torchao config + min_features filter the runtime path uses, applied ahead of time. New core/inference/diffusion_prequant.py (resolve_prequant_source + load_prequantized_transformer, best-effort, lazy imports). diffusion.py _load_dense_quant_pipeline tries the pre-quant source first and falls back to the dense materialise+quantise path, then to GGUF, so the default is unchanged. DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for the resolver, the meta-init+assign loader, and the backend branch selection + fallbacks; GPU verification via scripts/verify_prequant_backend.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): attention-backend selection Add a selectable attention kernel via the diffusers set_attention_backend dispatcher. Attention is memory-bandwidth bound, so a better kernel is an end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV matmuls torchao never touches) and composes with torch.compile. auto picks the best exact backend for the device: cuDNN fused attention (_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default (below the compile/quant noise floor); native SDPA elsewhere and when speed=off (so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/ xformers/aiter are honored, and an unavailable kernel falls back to the default rather than failing the load. New core/inference/diffusion_attention.py (normalize + per-device select + apply, best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline; attention_backend threads through begin_load / load_pipeline / status like the other load knobs. New request field attention_backend + status field. Hermetic CPU tests for normalize / select policy / apply fallback, plus route threading + 422. Measured via scripts/perf_levers_probe.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the arch tier to put int8 first on a consumer/workstation GPU (detected by the existing _is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first. Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation (-> int8) and data-center Ada/Hopper/Blackwell (-> fp8). * Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across denoise steps a DiT's output settles, so once the first block's residual barely changes the remaining blocks are skipped and their cached output reused. diffusers ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the standalone apply_first_block_cache hook as a fallback). Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 -> 2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar. OFF by default and a per-load opt-in: the win scales with step count, so it is for many-step models (Flux / Qwen-Image) and pointless for few-step distilled models (e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction of the trajectory. It composes with regional compile only with fullgraph=False (the cache's per-step decision is a torch.compiler.disable graph break), which the speed layer now switches to automatically when a cache is engaged. Best-effort: a model whose block signature the hook does not recognise is caught and the load proceeds uncached. - new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache (enable_cache / apply_first_block_cache fallback; threshold auto-raised for a quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import). - diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph when a cache is engaged. - diffusion.py: apply_step_cache before compile; thread transformer_cache / transformer_cache_threshold through begin_load -> load_pipeline and report the engaged mode in status(). - models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and transformer_cache_threshold request fields, engaged mode in the status response. - hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold selection, and best-effort failure handling, plus route threading + validation. - scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM / LPIPS vs the compiled no-cache baseline). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears) The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with 'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2 *_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing min_features filter did not exclude them. Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed, guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers (M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and quantises these layers fine, so the exclusion is int8-only. Sequence embedders (context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note 'context_embedder' contains the substring 'text_embed', which is why the token is the specific 'time_text_embed', not 'text_embed'. Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s (5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already working) are unchanged. - diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes exclude_name_tokens; quantize_transformer passes it for int8 only. - hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps attention / FFN / sequence-embedder linears), while fp8 keeps them. - scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's Linear layers and derive the exclusion list. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears) The prequant-checkpoint builder applied the dense quant filter without the int8-only M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8 checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now produces a working checkpoint on every supported model, giving int8 (the consumer-preferred scheme) the same ~2x load-VRAM and download reduction fp8 already had. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the native stable-diffusion.cpp engine instead of diffusers, with diffusers as the guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM. - diffusion_engine_router: centralised engine selection (built on the existing select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason. - sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface backed by sd-cli, with lazy binary install, registry-driven asset fetch, step-progress parsing, and cancellation. - diffusion_families: per-family single-file VAE + text-encoder asset mapping. - sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled). - routes/inference + gpu_arbiter: drive the active engine via the router; the API now reports the active engine and any fallback reason. - tests for the backend, router, route selection, and cancellation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler Address review feedback on #6724: - engine router: unload the engine being deactivated on a switch, so the old model is not left resident-but-unreachable (the evictor only targets the active engine). - generate route: sd.cpp execution errors (nonzero exit / timeout / missing output) now map to 500, not 409 (which only means not-loaded / cancelled). - native batch: return per-image seeds and persist the actual seed for each image so every batch image is reproducible. - Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the stable-diffusion.cpp docs; other families keep sd-cli defaults. - honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload offload flags on the native load instead of hardcoding them off. - fail the load when the sd-cli binary is present but not runnable (version() now returns None on exec error / nonzero exit). - size estimate: only treat the transformer asset as a possible local path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in load_prequantized_transformer ends in torch.load(weights_only=False), which executes arbitrary code from the pickle. The transformer_prequant_path load-request field reached that unpickle for any local file an authenticated caller named, so a request could trigger remote code execution. Refuse the source.kind=='path' branch unless the operator sets UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted and unaffected. Document the requirement on the API field and add gate tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa - apply_attention_backend now restores the native default when no backend is requested or a kernel fails. diffusers keeps a process-wide active attention backend that set_attention_backend updates, and a fresh transformer's processors follow it, so a load that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile load pinned, breaking the bit-identical/off guarantee. - select_attention_backend drops flash3/flash4 up front when the CUDA capability is below Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit request on the wrong card set fine then crashed mid-generation; it now falls back to native. - Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a valid alias of native) is accepted instead of 422-rejected by Pydantic. - Drop the dead replace('-','_') normalization (no alias uses dashes/underscores). - perf_levers_probe.py output dir is now relative to the script, not a hardcoded path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF - apply_step_cache now engages only via the transformer's native enable_cache (the diffusers CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model now runs uncached per the best-effort contract. - GGUF transformers are quantized (the default Studio load path), so they now use the higher quantized FBCache threshold when the caller leaves it unset, instead of the dense default that could keep the cache from triggering. - fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so fullgraph=True failed warmup and silently measured an eager cached run); output dir is now relative to the script, not a hardcoded path. * Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder _is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate. Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0). * Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just TypeError on older torchao), which would break the mxfp8 config helper instead of falling back to the default. Catch both so the fallback is robust. quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds CUDA memory during the per-row VRAM probe; output dir relative to the script. * Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load - snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the whole snapshot. restore_backend_flags restores each flag independently so one failure can't leave the others leaked process-wide. - load_pipeline restores the flags (and clears the GPU cache) when the build fails after apply_speed_optims mutated the process-wide flags but before _state captured them for unload to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and contaminated later off generations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs without producing output (or without closing stdout) would never reach proc.wait and the wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the PROCESS, so the main thread always enforces the timeout and kills a hung process (which closes the pipe and ends the reader). Add a test that times out even when stdout blocks, and make the no-binary test hermetic so a host-installed sd-cli can't leak in. * Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn The filter callback can be invoked without a module name, so fqn.lower() would raise AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens, so the linear is kept) instead of crashing the quantise pass. * Studio diffusion (Phase 16) review fixes: native engine robustness - sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64); a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds aliased to the same image. Random seeds stay 53-bit (JS-safe). - sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub fall back to anonymous instead of failing auth on a blank token. - sd_cpp_backend: a superseding load now cancels the in-flight generation, so the old sd-cli can no longer return/persist an image from the previous model. - diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a slow 10+ GB free / CUDA sync does not block engine selection. - diffusion_engine_router: probe sd-cli runnability (version()) before committing to native, so a present-but-unrunnable binary falls back to diffusers at selection. - diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a CPU-only install can still reach the native sd.cpp engine instead of failing load. - tests updated for the runnability probe + a not-runnable fallback case. * Studio diffusion (Phase 9) review fixes: prequant safety + validation - SECURITY: a request-supplied local pre-quant path is now unpickled only when it resolves inside an operator-configured ALLOWLIST of directories (UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in, once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on any path a load request named (arbitrary code execution). realpath() blocks symlink escapes; a bare on/off toggle is no longer a wildcard. - Validate the checkpoint's min_features against the runtime Linear filter, so a checkpoint that quantised a different layer set is rejected instead of silently loading a model that mismatches the dense path while reporting the same scheme. - Tolerant base_model_id compare (exact or same final path/repo segment), so a local path or fork of the canonical base is accepted instead of falling back to dense. - _has_meta_tensors uses any(chain(...)) (no intermediate lists). - prequant verify/probe scripts use repo-relative paths (+ env overrides), not the author's absolute /mnt paths. - tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts - diffusion_memory: when group offload is unavailable and the plan falls back to whole-module offload, enable VAE tiling (the group plan left it off, but the fallback is the low-VRAM path where the decode spike can OOM). Covers both the group and sequential fallback branches. - perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a balanced bit-identity regression actually fails the check instead of exiting 0. - compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and the load-progress poll has a 30 min deadline instead of looping forever on a hang. - test for the group->model fallback enabling VAE tiling. * Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path - diffusion: a torchao-quantized transformer is committed only compiled. A dense model resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF it replaced), so when transformer_quant engaged and speed resolved to off, promote to default (regional compile); warn loudly if compile still does not engage. - diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK); otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF. - nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path). - test asserts the eager-quant -> default-compile promotion. * Studio diffusion (Phase 10) review fixes: attention gating + probe isolation - diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation, so auto now stays on native SDPA there. - diffusion_attention: _active_attention_backend handles get_active_backend() returning an enum/None (not a tuple); the old unpack always raised and was swallowed, so the native-restore short-circuit never fired. - perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul so the inductor_flags variant doesn't leak into later compiled rows. - tests for the SM80 cuDNN gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening - install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket); extract through a per-member containment check (Zip-Slip guard); expanduser the --install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch the separately-published cudart runtime DLL archive so sd-cli.exe can start. - sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie. - tests: Zip-Slip rejection, normal extraction, studio-home discovery. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of the literal --output path. SdCppEngine.generate checked only the literal path, so a batch generation would exit 0 and then raise 'no image' (or return a stale file). generate now returns the literal path when present and otherwise falls back to the numbered siblings; single-image behavior is unchanged. Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected without error. * Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats Codex review on the native engine arg builder: - build_sd_cpp_command emitted --width/--height unconditionally, so an img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of the input. width/height are now Optional (None = unset): an image-conditioned run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives the size from the input image (set_width_and_height_if_unset); a plain txt2img run with unset dims keeps the prior 1024x1024 default; explicit dims are always honored. width/height are read only by the builder, so the type change is local. - build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...) that silently swallowed repeats=0 into sd-cli's default of one pass, turning an explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError and emits the flag for any explicit value != 1. Tests: img2img unset dims omit width/height (init_img and ref_images), explicit dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at the default. (Two pre-existing binary-discovery tests fail only because a real sd-cli is installed in this dev environment; unrelated to this change.) * Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc Codex review: the transformer_prequant_path field description still told operators to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the prior security fix made that variable a directory allowlist -- _allowed_prequant_roots deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator following the documented =1 would have every transformer_prequant_path request silently refused. The description now states it must name one or more allowlisted directories and that a bare on/off value is not accepted. Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does not say =1, and describes an allowlist/directory (guards against doc drift). * Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset Codex review on attention-backend selection: - Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first generation with no fallback. select_attention_backend now applies _cudnn_attention_supported() to an explicit cuDNN request too. - flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3 is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a (min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+. - apply_attention_backend's success path left diffusers' process-wide active backend pinned to the kernel it set; a later component whose processors are unconfigured (backend None) would inherit it. It now resets the global registry to native after a successful per-transformer set (the transformer keeps its own backend), best-effort. Also fixed _active_attention_backend: get_active_backend() returns a (name, fn) tuple, so the prior code stringified the tuple and never matched a name, defeating the native-restore short-circuit. Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on SM90; global registry reset after a successful set; _active_attention_backend reads the tuple return. * Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300 (Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So _is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they are treated as data-center class and keep the intended fp8-first behavior. Test: extends the datacenter parametrize with 'NVIDIA B300' and 'NVIDIA GH200 480GB' (now _is_consumer_gpu False). * Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder Codex review: the M=1 modulation/embedder exclusion was wired only into the dense runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint quantised the AdaLN modulation and conditioning-embedder linears, and loading it via transformer_prequant_path (the load path only loads already-quantised tensors, it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes for the runtime path. Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 -> the M=1 exclusion, every other scheme -> none) and use it in both the runtime quantiser and the builder, so a prequant artifact's quantised-layer set always matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion). Test: int8_exclude_name_tokens returns the exclusion for int8 and () for fp8/nvfp4/mxfp8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race Codex review on the native-engine routing: - The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat) unconditionally after engine selection. A native sd.cpp load on a pure-CPU host never touches the GPU, so that needlessly tore down the resident chat model. The handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it. - sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when _run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so the setting was unverifiable. status now derives them from state.offload_flags (still 'none' on CPU, where the flags are empty). - _run_load committed the new state without cancelling/waiting on a generation that started during the (slow) asset download, so a stale sd-cli run against the OLD model could finish afterward and persist an image from the previous model once the new load reported ready. The commit now signals the in-flight cancel and waits on _generate_lock before swapping _state (taken only at commit, so the download never serialises against generation), mirroring the diffusers load path. Tests: CPU native load skips the arbiter while a GPU native load takes it; status reports offload active when flags are set; _run_load cancels and waits for an in-flight generation before committing. * Studio diffusion (Phase 14) review round 2: align helper name with the stack Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the identical helper already present higher in the diffusion stack (Phase 16). The helper definition, the runtime quantiser call, and the offline builder are now byte-identical to that version, so the two branches no longer introduce a divergent name for the same single-source-of-truth and the stack merges without a conflict on this fix. No behavior change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: eager patches + torch.compile cache speed phase Adds the opt-in speed path for the GGUF diffusion transformer behind a selectable speed mode (default off, so output is unchanged until a profile is chosen): - diffusion_eager_patches.py: shared eager fast-paths (channels_last, attention/backend selection, fused norms and QKV) installed at load and rolled back on unload or failed load. - diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent torch.compile cache and the GGUF-transformer compile wiring. - diffusion_arch_patches.py: architecture-specific patches. - diffusion_patch_backend.py: shared install/restore plumbing. - diffusion_speed.py: speed-profile planning. Tests for each module plus the benchmarking and probe scripts used to measure speed, memory, and accuracy of the path. * Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI Backend: - Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file fp8 transformers, gated to the unsloth org plus a curated allowlist. - Image-conditioned workflows built with Pipeline.from_pipe so they reuse the loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint, outpaint, and a hires-fix upscale pass. - Instruction editing as its own family kind (Qwen-Image-Edit-2511, FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and multi-reference) plus klein inpaint. - Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the decoded image size and cap upscale output to avoid OOM on large inputs. - Fixes: from_pipe defaulting to a float32 recast that crashed torchao quantized transformers; image-conditioned calls forcing the slider size onto the input image. Native sd.cpp engine rejects image-conditioned and reference requests it cannot serve. Frontend: - Redesigned Images page with capability-gated workflow tabs (Create, Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor, client-side outpaint, and a multi-reference picker. - Advanced options moved to a right-docked panel mirroring Chat: closed by default, toggled by a single fixed top-bar button that stays in place. sd.cpp installer: pin the release, verify each download's sha256, add a download timeout, and make the source repo configurable for a future mirror. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: correct the Advanced panel comment (closed by default, fixed toggle) * Studio diffusion: fix FP8 transformer quant producing noise (per-row scaling) FP8 dense-quant produced pure noise on models with extreme activation outliers (z-image), while INT8 was fine. Root cause: torchao's default fp8 granularity is per-TENSOR, so z-image's MLP activation outliers (~6.6e4) force a tensor-wide scale that pushes every normal value below fp8 resolution and the denoise collapses to noise. INT8 dynamic is per-token by default, which is why it was unaffected. Fix: request PerRow granularity (per-token activation + per-output-channel weight) for the fp8 config, confining each outlier to its own row. The per-row scaled_mm is probed by _smoke_probe, so an arch or build without it falls through the ladder to int8. Validated on B200 (z-image, 1024px, 8 steps): per-tensor fp8 = noise, per-row fp8 = matches bf16; int8 unaffected. Adds a regression test asserting the fp8 config carries PerRow granularity. * Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a stray "gguf" tag on the Hub but ship no .gguf files. The model search classified them as GGUF from the bare tag, so the picker rendered the GGUF variant expander, which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata (populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected. * Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline load when the id is unsloth-hosted or on-device (the backend infers the family + base repo and gates loads to unsloth/* or local paths), and show a clear message otherwise instead of silently ignoring the click. Curated and GGUF paths are unchanged. * Studio Images: keep curated safetensors models in Recommended after download The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet), so they vanished from the picker after the first load and could only be found by typing an exact search. The row already renders a downloaded badge, matching how GGUF Recommended rows stay visible when cached. Drop the exclusion so the curated safetensors always list. * Studio Images: clarify the GGUF transformer-quant Advanced control Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode" with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/ FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF, falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only. * Studio Images: list on-device unsloth diffusion models in the picker The Images picker's On Device tab hid every non-GGUF cached repo whenever a task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit and FP8 safetensors) never showed up there. List cached repos that pass the task gate, limited under a filter to unsloth-hosted ones so base repos (which fail the diffusion load trust gate) don't appear only to dead-end on click. Chat behavior is unchanged: the task gate still drops image repos there. * Studio: hide single-file image checkpoints from the chat model picker The chat picker treats a cached repo as an image model, and hides it, only when it ships a diffusers model_index.json. Single-file, ComfyUI, and ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models. Fall back to resolving the repo id against the known diffusion families, the same resolver the Images backend loads from, so these checkpoints are tagged text-to-image and stay in the Images picker only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: add the FLUX.2-dev model family Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline) resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline + Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated, reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror. text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from klein's turbo defaults. Verified live: GGUF load resolves the family + gated base repo and generates a real 1024x1024 image on GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: clearer error for an unsupported diffusion model When a repo id resolves to no diffusion family the load raised 'Could not infer a diffusion family... Pass family_override (z-image)', which points at an unrelated family and doesn't say what is supported. Replace it with a message that lists the supported families (from a new supported_family_names helper) and notes that video models and image models whose diffusers transformer has no single-file loader are not supported. Applies to both the diffusers and native sd.cpp load paths. Also refreshes two stale family-registry comments that still called FLUX.2-dev omitted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove stray async task scratch outputs committed by mistake * fp8 prequant: reject stale per-tensor checkpoints The per-row fp8 fix only applies when quantising dense on the fly; a supplied transformer_prequant_path (or a future hosted fp8 repo) bypassed it, so an fp8 checkpoint baked with the old per-tensor layout still loaded and reproduced the z-image noise failure. Stamp the fp8 granularity into the checkpoint metadata at build time and require per-row at load time, so a stale artifact is rejected and the loader falls back to rebuilding/re-quantising. The gate is fp8-only (int8 and the others are unaffected). Adds regression tests. * Diffusion: guard trust check against OSError and validate conditioning inputs - _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid characters (or a bare owner/name id) can't raise OSError; treat any failure as not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request still raises the clear FileNotFoundError for a genuinely missing local pick. - generate(): reject mask_image / upscale / reference_images supplied without an input image, and reject reference_images on a family that does not support reference conditioning, instead of silently degrading to txt2img / img2img. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address Codex review findings on the image-workflows PR Keep diffusion.py importable without torch: the compile/arch patch modules import torch at module level, so import them lazily at their load/unload call sites instead of at module load. This restores the torchless contract so get_diffusion_backend() works on a CPU/native sd.cpp install. Match family reject keywords and aliases as whole path/name segments, not raw substrings, so an unrelated word like edited, edition, or kontextual no longer misroutes or hides a valid base image model, while supported edit families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same segment matching in the picker task filter. Route FLUX.2-dev native guidance through --guidance like the other FLUX families rather than --cfg-scale. Reject native upscale requests that have no input image. Read image header dimensions and reject over-limit inputs before decoding pixels, so a crafted small-payload image cannot spike memory. Reject an upscale that would shrink the source below its input size. Validate the model_kind against the filename extension before the GPU handoff. Estimate a local diffusers pipeline's size from its on-disk weights so auto memory planning does not skip offload and OOM. Report workflows: [txt2img] from the native backend status so the Create tab stays enabled for a loaded native model. Clamp the outpaint canvas to the backend's 4096px decode limit. Adds regression tests for segment matching and kind/extension validation. * Address further Codex findings on the image-workflows PR - Persist the actual output image size in the gallery recipe instead of the request sliders: Transform/Inpaint/Edit derive the size from the uploaded image, Extend grows the canvas, and Upscale resizes it, so the sliders recorded (and later restored) the wrong dimensions for those workflows. - Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file name) in validate_load_request, so the unloadable pick fails before chat is evicted rather than deep in from_pretrained. - Only publish an image-conditioned from_pipe wrapper to the shared aux cache when the load is still current: from_pipe runs under the generate lock but not the state lock, so an unload racing its construction could otherwise cache a wrapper over torn-down modules that a later load would reuse. - Verify the Windows CUDA runtime archive checksum before extracting it, like the main sd-cli archive, so a corrupt or tampered runtime is rejected rather than extracted next to the binary. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
|
Daniel Han
|
467e74baee
|
Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI redesign (#6769)
* Studio diffusion: cross-platform device policy, fp16 guard, lock split, validate-before-evict Phase 1 of porting the richer diffusion stack onto the image-generation backend. - Add a compartmentalized device/dtype policy module (diffusion_device.py) resolving CUDA/ROCm/XPU/MPS/CPU with capability flags. Keeps the NVIDIA capability-based bf16 choice; ROCm and XPU are isolated; MPS uses bf16 or fp32, never a silent fp16 that renders a black image. - Add a per-family fp16_incompatible flag (Z-Image) and promote a resolved float16 to float32 for those families so they do not produce black images. - Split the backend locks: a generation holds only _generate_lock, so status, unload, and a new load are never blocked by a long denoise. Add per-generation cancellation via callback_on_step_end so an eviction or a superseding load preempts a running generation; a replacement load waits for it to stop before allocating, so two pipelines never sit in VRAM at once. - Validate a load request before the GPU handoff so an unloadable pick never evicts a working chat model, and reject missing local paths up front. - Add CPU-only tests for the device policy, dtype guard, lock split and cancellation, and validate-before-evict, plus a GPU benchmark/regression script (scripts/diffusion_bench.py) measuring latency, peak VRAM, and PSNR against a saved reference. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2A): measured-budget memory planner + offload/VAE policy Add a lean, backend-agnostic memory policy that picks a CPU-offload policy and VAE tiling/slicing from measured free device memory vs the model's estimated resident footprint, then applies it to the built pipeline. auto stays resident when the model fits (byte-identical to the prior resident path), and falls to whole-module offload when tight; fast/balanced/low_vram are explicit overrides. Sequential submodule offload is unreliable for GGUF transformers on diffusers 0.38, so it falls back to whole-module offload and status reports the policy actually engaged. Verified on Z-Image-Turbo Q4_K_M (B200): auto reproduces the resident image with no VRAM/latency regression (PSNR inf); balanced/low_vram cut generation peak VRAM 47.9% (15951 -> 8318 MB) with byte-identical output, at the expected latency cost. 73 prior + 35 new CPU tests pass. * Studio diffusion (Phase 2D): streamed block-level offload + functional VAE tiling Add a streamed 'group' offload tier (diffusers apply_group_offloading, block_level, use_stream) that keeps the transformer flowing through the GPU a few blocks at a time while the text encoder / VAE stay resident, and fix VAE tiling to drive the VAE submodule (pipelines like Z-Image expose enable_tiling on pipe.vae, not the pipeline). apply_memory_plan now returns the (policy, tiling) actually engaged so status never overstates either, and group falls back to whole-module offload when the transformer can't be streamed. Measured on Z-Image (B200), all lossless (PSNR inf vs resident): balanced/group cuts generation peak VRAM 32% (15951 -> 10840 MB) at near-resident speed (2.07 -> 2.99s); low_vram/model cuts it 48% (-> 8318 MB) but is slower (7.99s). Mode names now match that tradeoff: balanced = stream the transformer, low_vram = offload every component. auto picks group when the companions fit resident, else model. 112 CPU tests pass. * Studio diffusion (Phase 5): image quality-vs-quant accuracy harness Add scripts/diffusion_quality.py, the accuracy analogue of the KLD workflow: hold prompt + seed fixed, render a grid with a reference quant (default BF16), then render each candidate quant and measure drift from the reference. Records mean PSNR + SSIM (pure-numpy, no skimage/scipy) and optional CLIP text-alignment + image-similarity (transformers, --clip), plus file size, latency, and peak VRAM, then prints a quality-vs-cost table and recommends the smallest quant within a quality budget. --selftest validates the metrics on synthetic images with no GPU or model. Verified on Z-Image (B200): the table degrades monotonically with quant size (Q8 -> Q4 -> Q2: PSNR 21.7 -> 15.5, SSIM 0.82 -> 0.61), while CLIP-text stays flat (~0.34) -- quantization erodes fine detail far more than prompt adherence. * Studio diffusion (Phase 3): opt-in speed layer (channels_last / compile / TF32) Add a speed_mode knob (off by default, so the render path stays bit-identical): default applies channels_last VAE + regional torch.compile of the denoiser's repeated block where eligible; max also enables TF32 matmul and fused QKV. Regional compile is gated off for the GGUF transformer (dequantises per-op) and for families flagged not compile-friendly (a new supports_torch_compile flag, False for Z-Image), so it activates automatically only once a non-GGUF bf16 transformer is loaded. Speed optims run before placement/offload, per the diffusers composition order. status now reports speed_mode + the optims actually engaged. Verified on Z-Image (B200): default -> ['channels_last'], max -> ['channels_last', 'tf32'], compile correctly skipped for GGUF; generation works in every mode. 121 CPU tests pass. * Studio diffusion (Phase 2B): opt-in fp8 text-encoder layerwise casting Add a text_encoder_fp8 knob that casts the companion text encoder(s) to fp8 (e4m3) storage via diffusers apply_layerwise_casting, upcasting per layer to the bf16 compute dtype while normalisations and embeddings stay full precision. Applied before placement, gated to CUDA + bf16, best-effort (a failure leaves the encoder dense). status reports which encoders were cast. Verified on Z-Image (B200, balanced/group mode where the encoder stays resident): generation peak VRAM dropped 37% (10840 -> 6791 MB, below the lowest-VRAM offload) at near-resident speed. It is a memory-vs-quality tradeoff, not free -- ~20 dB PSNR vs the bf16 encoder, a larger shift than one transformer quant step -- so it is off by default and documented as such, with the Phase 5 harness to size the cost. 127 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2C): NVFP4 text-encoder quant (+ generalise fp8 knob) Generalise the text-encoder precision knob from a fp8 bool to text_encoder_quant (fp8 | nvfp4). nvfp4 quantises the companion text encoder to 4-bit via torchao NVFP4 weight-only (two-level microscaling) on Blackwell's FP4 tensor cores; fp8 stays the broader-hardware path (cc>=8.9). Both are gated, best-effort, and run before placement; status reports the mode actually engaged. This is the lean realisation of GGUF-native text-encoder quant: 4-bit on the encoder without the 3045-line port. Verified on Z-Image (B200, balanced/group where the encoder stays resident), vs the bf16 encoder: nvfp4 cut generation peak VRAM 48% (10840 -> 5593 MB, the lowest TE option, below whole-model offload) at near-fp8 quality (16.4 vs 17.1 dB PSNR), and both quants ran faster than bf16. A memory-vs-quality tradeoff (off by default); size it per model with the Phase 5 quality harness. diffusion_bench gains --text-encoder-quant. 129 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): native stable-diffusion.cpp engine for CPU/Mac Adds the CPU / Apple-Silicon tier of the two-engine strategy, mirroring the chat backend's llama.cpp shell-out. Diffusers stays the default on CUDA / ROCm / XPU; this covers the hardware diffusers serves poorly, consuming the same split GGUF assets Studio already curates. - sd_cpp_args.py: pure sd-cli command builder. Maps the family to its text-encoder flag (Z-Image Qwen3 to --llm, Qwen-Image to --qwen2vl, FLUX.1 CLIP-L + T5), and the diffusers memory policy (none/group/model/sequential) to sd.cpp's offload flags (--offload-to-cpu / --clip-on-cpu / --vae-on-cpu / --vae-tiling / --diffusion-fa), so one user knob drives both engines. - sd_cpp_engine.py: SdCppEngine over a located sd-cli. find_sd_cpp_binary() with the same precedence as the llama finder (env override, then the Studio install root, then in-tree, then PATH), an is_available/version probe, and a one-shot subprocess generate that streams progress and returns the PNG. runtime_env() prepends the binary's directory to the platform library path so a prebuilt's bundled libstable-diffusion.so resolves. select_diffusion_engine() is the pure routing decision (GPU backends to diffusers, CPU/MPS to native when present). - install_sd_cpp_prebuilt.py: resolve + download the per-host prebuilt (macOS-arm64/Metal, Linux x86_64 CPU, Vulkan/ROCm/Windows variants) into the Studio install root. resolve_release_asset() is a pure, unit-tested host-to-asset matrix. - scripts/sd_cpp_smoke.py: end-to-end native generation harness. Tests (CPU-only, subprocess/filesystem stubbed): 49 new across args, engine, routing, runtime env, and the installer resolver. Full diffusion suite 166 passing. Verified on a B200 box: built sd-cli (CUDA) and the prebuilt (CPU) both generate Z-Image-Turbo Q4_K end to end through SdCppEngine: balanced (group offload, 5.0s gen), low_vram (full CPU offload + VAE tiling, 13.4s), and the dynamically-linked CPU prebuilt (50.4s on CPU), all producing coherent images. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 6): img2img / inpaint / edit / LoRA / upscale on the native engine Builds on Phase 4's native stable-diffusion.cpp engine, extending it from text-to-image to the wider feature surface, since sd.cpp supports all of these through the binary already. Pure command-builder additions plus one engine method, so the txt2img path is unchanged. - sd_cpp_args.py: SdCppGenParams gains image-conditioning fields. init_img + strength make a run img2img, adding mask makes it inpaint, ref_images drives FLUX-Kontext / Qwen-Image-Edit style editing (repeated --ref-image), and lora_dir + the <lora:name:weight> prompt syntax select LoRAs. New SdCppUpscaleParams + build_sd_cpp_upscale_command for the ESRGAN upscale run mode (input image + esrgan model, no prompt / text encoders). - sd_cpp_engine.py: the subprocess runner is factored into a shared _run() so generate() (now carrying the conditioning flags) and a new upscale() reuse the same streaming / error / output-check path. - scripts/sd_cpp_smoke.py: --task {txt2img,img2img,upscale} with --init-img / --strength / --upscale-model / --upscale-repeats. Tests: 10 new across the img2img / inpaint / edit / LoRA flag construction, the upscale builder and its validation, and the engine's img2img + upscale paths. Full diffusion suite 176 passing. Verified on a B200 box through SdCppEngine: img2img (Z-Image-Turbo Q4_K, the init image conditioned at strength 0.6, 4.8s) and ESRGAN upscale (512x512 -> 2048x2048 via RealESRGAN_x4plus_anime_6B, 2.7s), both producing coherent images. Video and the diffusers-path feature wiring are deferred. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): accuracy-preserving speed pass Re-review of the diffusion stack (#6675/#6679/#6680) surfaced one real accuracy bug and a dead-on-arrival speed path; this fixes both and adds the lossless / near-lossless wins, all measured on a B200. Correctness: - TF32 global-state leak (fix). speed_mode=max flipped torch.backends.*.allow_tf32 process-wide and never restored them, so a later `off` load silently inherited TF32 and was no longer bit-identical. Added snapshot_backend_flags / restore_backend_flags (TF32 + cudnn.benchmark), captured before the speed layer runs and restored on unload. Verified: load max -> unload -> load off is now byte-identical (PSNR inf) to a fresh off. - sd-cli timeout could hang forever. _run() blocked in `for line in stdout` and only checked the timeout after EOF, so a child stuck in model load / GPU init with no output ignored the timeout. Drained stdout on a reader thread with a wall-clock deadline. Added a silent-hang regression test. Speed (diffusers path), near-lossless, opt-in tiers: - Regional torch.compile now runs on the GGUF transformer. The is_gguf gate (and Z-Image's supports_torch_compile=False) were stale: compile_repeated_blocks compiles and runs ~2.2x faster on the GGUF Z-Image transformer on torch 2.9.1 / diffusers 0.38 (the per-op dequant stays eager, the rest of the block compiles). Measured: off 1.80s -> default 0.82s/gen (+54.7%), PSNR 37.7 dB vs eager -- far above the Q4 quant noise floor (~21 dB), so it does not move output quality. Gate relaxed; default tier delivers it. - cudnn.benchmark added to the default tier (autotunes the fixed-shape VAE convs). - torch.inference_mode() around the pipeline call (lossless, strictly faster than the no_grad diffusers uses internally). Memory path: - VAE tiling (not bit-identical >1MP) restricted to the model/sequential/CPU tiers; the balanced (group) tier keeps exact slicing only, so it is now bit-identical to the resident image (verified PSNR inf) and slightly faster. - Group offload adds non_blocking + record_stream on the CUDA stream path to overlap each block's H2D copy with compute (lossless; gated on the installed diffusers signature so older versions still work). Native (sd.cpp) path: - native_speed_flags: a first-class speed knob (default -> --diffusion-fa, a near-lossless CUDA win that was previously only added on offload tiers; max also -> --diffusion-conv-direct). conv-direct stays opt-in: measured +45% on CUDA, so it is never auto-on. Engine generate() merges it, de-duped against offload flags. Default profile: a GGUF model with no explicit speed_mode now resolves to the `default` profile (resolve_speed_mode), since compile's perturbation sits below the quantisation noise floor and so does not reduce quality versus the dense reference; out of the box a GGUF Z-Image generation drops from 1.80s to 0.81s. Dense models stay `off` / bit-identical, and an explicit speed_mode -- including "off" -- is always honored, so the byte-identical path remains one flag away and is the regression reference. Tooling: scripts/compile_probe.py (eager vs compiled GGUF probe), scripts/ perf_verify.py (the B200 verification above), and diffusion_bench.py gains --speed-mode so the speed tiers are benchmarkable. Tests: 183 passing (was 166); new coverage for the backend-flag snapshot/restore, GGUF compile eligibility, the balanced tiling/slicing split, native_speed_flags + the engine de-dup, and the sd-cli silent-hang timeout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): max tier uses max-autotune-no-cudagraphs + engine/lever benchmarks The opt-in `max` speed tier now compiles the repeated block with mode=max-autotune-no-cudagraphs (dynamic=False) instead of the default mode: Triton autotuning for GEMM/conv-heavier models, gated to the tier where a longer cold compile is acceptable. CUDA-graph modes (reduce-overhead / max-autotune) are deliberately avoided -- both crash on the regionally-compiled block (its static output buffer is overwritten across denoise steps), measured. Adds two reproducible benchmarks used to validate the optimization research: - scripts/compare_engines.py: PyTorch (diffusers GGUF) vs native sd.cpp head-to-head. - scripts/leverage_probe.py: coordinate_descent_tuning + FirstBlockCache probes. Measured on B200 (Z-Image Q4_K_M, 1024px, 8 steps): default compile 0.80s/gen; coordinate_descent_tuning 0.79s (within noise, already covered by max-autotune); FirstBlockCache does not run on Z-Image (diffusers 0.38 block-detection / Dynamo). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source) Add an opt-in transformer_quant mode that loads the dense bf16 transformer and torchao-quantises it onto the low-precision tensor cores, instead of the GGUF transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's 0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor. GGUF+compile stays the low-memory default and the fallback. The mode is gated on CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB); any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe (Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips the tiny projections that crash int8's torch._int_mm. New module mirrors diffusion_precision.py; quant runs before compile before placement. 184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py is the standalone torchao lever probe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): consumer-GPU tuning - lock fp8 fast accumulate, prefer fp8 over mxfp8, reject 2:4 sparsity Consumer Blackwell halves tensor-core throughput on FP32 accumulate (fp8 419 vs 838 TFLOPS with FP16 accumulate; bf16 209), so: - fp8 config locks use_fast_accum=True (Float8MMConfig). torchao already defaults it on; pinning it guards consumer cards against a default change. On B200 it is identical speed and slightly better quality (LPIPS 0.050 vs 0.091). - the Blackwell auto ladder prefers fp8 over mxfp8 (measured faster + more accurate). 2:4 semi-structured sparsity evaluated and rejected (scripts/sparse_accum_probe.py): 2:4 magnitude-prune + fp8 gives LPIPS 0.858 (broken image) with no fine-tune, the cuSPARSELt kernel errors on torch 2.9, and it does not compose with torch.compile (our main ~2x). Documented as a dead end, not shipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add fp8 fast-accum overflow verification probe scripts/fp8_overflow_check.py hooks every quantised linear during a real Z-Image generation and reports max-abs + non-finite counts for use_fast_accum True vs False. Confirms fast accumulation is an accumulation-precision knob, not an overflow one: across 276 linears, including Z-Image's ~1.0e6 activation peaks (which overflow FP16), 0 non-finite elements and identical max-abs for both modes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40; GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it. Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s), no overflow, quality below the quant noise floor. So the default leans to accuracy on data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force) lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off). 187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override threading). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add NVFP4 probe documenting it is not yet a win on torch 2.9 scripts/nvfp4_probe.py measures NVFP4 via torchao on the real Z-Image transformer. Finding (B200, 1024px/8 steps): NVFP4 is a torchao feature and DOES run with use_triton_kernel=False (the default triton path needs the missing MSLK library), but only at bf16-compile rate (0.667s vs fp8 0.592s) -- it dequantises FP4->bf16 rather than using the FP4 tensor cores. The real FP4 speedup needs MSLK or torch>=2.11 + torchao's CUTLASS FP4 GEMM. The smoke probe (default triton=True) already keeps NVFP4 out of auto on this env, so auto correctly stays on fp8; NVFP4 activates automatically once fast. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): prefer fp8 over nvfp4 in Blackwell auto ladder Validated NVFP4 on torch 2.11 + torchao CUTLASS FP4 in an isolated env. The FP4 tensor-core GEMM is genuinely active there (a 16384^3 GEMM hits ~3826 TFLOPS, 2.52x bf16 and 1.37x fp8), but it only beats fp8 on very large GEMMs. At the diffusion transformer's shapes (hidden ~3072, MLP ~12288, M~4096) NVFP4 is both slower (0.81x fp8 end to end on Z-Image 1024px) and less accurate (LPIPS 0.166 vs fp8's 0.044). Reorder the Blackwell auto ladder to fp8 before nvfp4 so auto is correct even on a future MSLK-equipped box; nvfp4 stays an explicit opt-in. Add scripts/nvfp4_t211_probe.py (extension diagnostics + GEMM micro + end-to-end). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): pre-quantized transformer loading The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs 13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline (scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True) the quantized weights, so the dense bf16 never touches the GPU. Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same torchao config + min_features filter the runtime path uses, applied ahead of time. New core/inference/diffusion_prequant.py (resolve_prequant_source + load_prequantized_transformer, best-effort, lazy imports). diffusion.py _load_dense_quant_pipeline tries the pre-quant source first and falls back to the dense materialise+quantise path, then to GGUF, so the default is unchanged. DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for the resolver, the meta-init+assign loader, and the backend branch selection + fallbacks; GPU verification via scripts/verify_prequant_backend.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): attention-backend selection Add a selectable attention kernel via the diffusers set_attention_backend dispatcher. Attention is memory-bandwidth bound, so a better kernel is an end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV matmuls torchao never touches) and composes with torch.compile. auto picks the best exact backend for the device: cuDNN fused attention (_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default (below the compile/quant noise floor); native SDPA elsewhere and when speed=off (so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/ xformers/aiter are honored, and an unavailable kernel falls back to the default rather than failing the load. New core/inference/diffusion_attention.py (normalize + per-device select + apply, best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline; attention_backend threads through begin_load / load_pipeline / status like the other load knobs. New request field attention_backend + status field. Hermetic CPU tests for normalize / select policy / apply fallback, plus route threading + 422. Measured via scripts/perf_levers_probe.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the arch tier to put int8 first on a consumer/workstation GPU (detected by the existing _is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first. Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation (-> int8) and data-center Ada/Hopper/Blackwell (-> fp8). * Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across denoise steps a DiT's output settles, so once the first block's residual barely changes the remaining blocks are skipped and their cached output reused. diffusers ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the standalone apply_first_block_cache hook as a fallback). Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 -> 2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar. OFF by default and a per-load opt-in: the win scales with step count, so it is for many-step models (Flux / Qwen-Image) and pointless for few-step distilled models (e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction of the trajectory. It composes with regional compile only with fullgraph=False (the cache's per-step decision is a torch.compiler.disable graph break), which the speed layer now switches to automatically when a cache is engaged. Best-effort: a model whose block signature the hook does not recognise is caught and the load proceeds uncached. - new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache (enable_cache / apply_first_block_cache fallback; threshold auto-raised for a quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import). - diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph when a cache is engaged. - diffusion.py: apply_step_cache before compile; thread transformer_cache / transformer_cache_threshold through begin_load -> load_pipeline and report the engaged mode in status(). - models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and transformer_cache_threshold request fields, engaged mode in the status response. - hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold selection, and best-effort failure handling, plus route threading + validation. - scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM / LPIPS vs the compiled no-cache baseline). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears) The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with 'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2 *_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing min_features filter did not exclude them. Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed, guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers (M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and quantises these layers fine, so the exclusion is int8-only. Sequence embedders (context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note 'context_embedder' contains the substring 'text_embed', which is why the token is the specific 'time_text_embed', not 'text_embed'. Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s (5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already working) are unchanged. - diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes exclude_name_tokens; quantize_transformer passes it for int8 only. - hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps attention / FFN / sequence-embedder linears), while fp8 keeps them. - scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's Linear layers and derive the exclusion list. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears) The prequant-checkpoint builder applied the dense quant filter without the int8-only M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8 checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now produces a working checkpoint on every supported model, giving int8 (the consumer-preferred scheme) the same ~2x load-VRAM and download reduction fp8 already had. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the native stable-diffusion.cpp engine instead of diffusers, with diffusers as the guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM. - diffusion_engine_router: centralised engine selection (built on the existing select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason. - sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface backed by sd-cli, with lazy binary install, registry-driven asset fetch, step-progress parsing, and cancellation. - diffusion_families: per-family single-file VAE + text-encoder asset mapping. - sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled). - routes/inference + gpu_arbiter: drive the active engine via the router; the API now reports the active engine and any fallback reason. - tests for the backend, router, route selection, and cancellation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler Address review feedback on #6724: - engine router: unload the engine being deactivated on a switch, so the old model is not left resident-but-unreachable (the evictor only targets the active engine). - generate route: sd.cpp execution errors (nonzero exit / timeout / missing output) now map to 500, not 409 (which only means not-loaded / cancelled). - native batch: return per-image seeds and persist the actual seed for each image so every batch image is reproducible. - Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the stable-diffusion.cpp docs; other families keep sd-cli defaults. - honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload offload flags on the native load instead of hardcoding them off. - fail the load when the sd-cli binary is present but not runnable (version() now returns None on exec error / nonzero exit). - size estimate: only treat the transformer asset as a possible local path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in load_prequantized_transformer ends in torch.load(weights_only=False), which executes arbitrary code from the pickle. The transformer_prequant_path load-request field reached that unpickle for any local file an authenticated caller named, so a request could trigger remote code execution. Refuse the source.kind=='path' branch unless the operator sets UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted and unaffected. Document the requirement on the API field and add gate tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa - apply_attention_backend now restores the native default when no backend is requested or a kernel fails. diffusers keeps a process-wide active attention backend that set_attention_backend updates, and a fresh transformer's processors follow it, so a load that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile load pinned, breaking the bit-identical/off guarantee. - select_attention_backend drops flash3/flash4 up front when the CUDA capability is below Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit request on the wrong card set fine then crashed mid-generation; it now falls back to native. - Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a valid alias of native) is accepted instead of 422-rejected by Pydantic. - Drop the dead replace('-','_') normalization (no alias uses dashes/underscores). - perf_levers_probe.py output dir is now relative to the script, not a hardcoded path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF - apply_step_cache now engages only via the transformer's native enable_cache (the diffusers CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model now runs uncached per the best-effort contract. - GGUF transformers are quantized (the default Studio load path), so they now use the higher quantized FBCache threshold when the caller leaves it unset, instead of the dense default that could keep the cache from triggering. - fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so fullgraph=True failed warmup and silently measured an eager cached run); output dir is now relative to the script, not a hardcoded path. * Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder _is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate. Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0). * Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just TypeError on older torchao), which would break the mxfp8 config helper instead of falling back to the default. Catch both so the fallback is robust. quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds CUDA memory during the per-row VRAM probe; output dir relative to the script. * Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load - snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the whole snapshot. restore_backend_flags restores each flag independently so one failure can't leave the others leaked process-wide. - load_pipeline restores the flags (and clears the GPU cache) when the build fails after apply_speed_optims mutated the process-wide flags but before _state captured them for unload to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and contaminated later off generations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs without producing output (or without closing stdout) would never reach proc.wait and the wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the PROCESS, so the main thread always enforces the timeout and kills a hung process (which closes the pipe and ends the reader). Add a test that times out even when stdout blocks, and make the no-binary test hermetic so a host-installed sd-cli can't leak in. * Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn The filter callback can be invoked without a module name, so fqn.lower() would raise AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens, so the linear is kept) instead of crashing the quantise pass. * Studio diffusion (Phase 16) review fixes: native engine robustness - sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64); a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds aliased to the same image. Random seeds stay 53-bit (JS-safe). - sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub fall back to anonymous instead of failing auth on a blank token. - sd_cpp_backend: a superseding load now cancels the in-flight generation, so the old sd-cli can no longer return/persist an image from the previous model. - diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a slow 10+ GB free / CUDA sync does not block engine selection. - diffusion_engine_router: probe sd-cli runnability (version()) before committing to native, so a present-but-unrunnable binary falls back to diffusers at selection. - diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a CPU-only install can still reach the native sd.cpp engine instead of failing load. - tests updated for the runnability probe + a not-runnable fallback case. * Studio diffusion (Phase 9) review fixes: prequant safety + validation - SECURITY: a request-supplied local pre-quant path is now unpickled only when it resolves inside an operator-configured ALLOWLIST of directories (UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in, once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on any path a load request named (arbitrary code execution). realpath() blocks symlink escapes; a bare on/off toggle is no longer a wildcard. - Validate the checkpoint's min_features against the runtime Linear filter, so a checkpoint that quantised a different layer set is rejected instead of silently loading a model that mismatches the dense path while reporting the same scheme. - Tolerant base_model_id compare (exact or same final path/repo segment), so a local path or fork of the canonical base is accepted instead of falling back to dense. - _has_meta_tensors uses any(chain(...)) (no intermediate lists). - prequant verify/probe scripts use repo-relative paths (+ env overrides), not the author's absolute /mnt paths. - tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts - diffusion_memory: when group offload is unavailable and the plan falls back to whole-module offload, enable VAE tiling (the group plan left it off, but the fallback is the low-VRAM path where the decode spike can OOM). Covers both the group and sequential fallback branches. - perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a balanced bit-identity regression actually fails the check instead of exiting 0. - compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and the load-progress poll has a 30 min deadline instead of looping forever on a hang. - test for the group->model fallback enabling VAE tiling. * Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path - diffusion: a torchao-quantized transformer is committed only compiled. A dense model resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF it replaced), so when transformer_quant engaged and speed resolved to off, promote to default (regional compile); warn loudly if compile still does not engage. - diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK); otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF. - nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path). - test asserts the eager-quant -> default-compile promotion. * Studio diffusion (Phase 10) review fixes: attention gating + probe isolation - diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation, so auto now stays on native SDPA there. - diffusion_attention: _active_attention_backend handles get_active_backend() returning an enum/None (not a tuple); the old unpack always raised and was swallowed, so the native-restore short-circuit never fired. - perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul so the inductor_flags variant doesn't leak into later compiled rows. - tests for the SM80 cuDNN gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening - install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket); extract through a per-member containment check (Zip-Slip guard); expanduser the --install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch the separately-published cudart runtime DLL archive so sd-cli.exe can start. - sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie. - tests: Zip-Slip rejection, normal extraction, studio-home discovery. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of the literal --output path. SdCppEngine.generate checked only the literal path, so a batch generation would exit 0 and then raise 'no image' (or return a stale file). generate now returns the literal path when present and otherwise falls back to the numbered siblings; single-image behavior is unchanged. Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected without error. * Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats Codex review on the native engine arg builder: - build_sd_cpp_command emitted --width/--height unconditionally, so an img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of the input. width/height are now Optional (None = unset): an image-conditioned run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives the size from the input image (set_width_and_height_if_unset); a plain txt2img run with unset dims keeps the prior 1024x1024 default; explicit dims are always honored. width/height are read only by the builder, so the type change is local. - build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...) that silently swallowed repeats=0 into sd-cli's default of one pass, turning an explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError and emits the flag for any explicit value != 1. Tests: img2img unset dims omit width/height (init_img and ref_images), explicit dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at the default. (Two pre-existing binary-discovery tests fail only because a real sd-cli is installed in this dev environment; unrelated to this change.) * Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc Codex review: the transformer_prequant_path field description still told operators to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the prior security fix made that variable a directory allowlist -- _allowed_prequant_roots deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator following the documented =1 would have every transformer_prequant_path request silently refused. The description now states it must name one or more allowlisted directories and that a bare on/off value is not accepted. Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does not say =1, and describes an allowlist/directory (guards against doc drift). * Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset Codex review on attention-backend selection: - Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first generation with no fallback. select_attention_backend now applies _cudnn_attention_supported() to an explicit cuDNN request too. - flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3 is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a (min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+. - apply_attention_backend's success path left diffusers' process-wide active backend pinned to the kernel it set; a later component whose processors are unconfigured (backend None) would inherit it. It now resets the global registry to native after a successful per-transformer set (the transformer keeps its own backend), best-effort. Also fixed _active_attention_backend: get_active_backend() returns a (name, fn) tuple, so the prior code stringified the tuple and never matched a name, defeating the native-restore short-circuit. Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on SM90; global registry reset after a successful set; _active_attention_backend reads the tuple return. * Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300 (Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So _is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they are treated as data-center class and keep the intended fp8-first behavior. Test: extends the datacenter parametrize with 'NVIDIA B300' and 'NVIDIA GH200 480GB' (now _is_consumer_gpu False). * Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder Codex review: the M=1 modulation/embedder exclusion was wired only into the dense runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint quantised the AdaLN modulation and conditioning-embedder linears, and loading it via transformer_prequant_path (the load path only loads already-quantised tensors, it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes for the runtime path. Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 -> the M=1 exclusion, every other scheme -> none) and use it in both the runtime quantiser and the builder, so a prequant artifact's quantised-layer set always matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion). Test: int8_exclude_name_tokens returns the exclusion for int8 and () for fp8/nvfp4/mxfp8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race Codex review on the native-engine routing: - The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat) unconditionally after engine selection. A native sd.cpp load on a pure-CPU host never touches the GPU, so that needlessly tore down the resident chat model. The handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it. - sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when _run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so the setting was unverifiable. status now derives them from state.offload_flags (still 'none' on CPU, where the flags are empty). - _run_load committed the new state without cancelling/waiting on a generation that started during the (slow) asset download, so a stale sd-cli run against the OLD model could finish afterward and persist an image from the previous model once the new load reported ready. The commit now signals the in-flight cancel and waits on _generate_lock before swapping _state (taken only at commit, so the download never serialises against generation), mirroring the diffusers load path. Tests: CPU native load skips the arbiter while a GPU native load takes it; status reports offload active when flags are set; _run_load cancels and waits for an in-flight generation before committing. * Studio diffusion (Phase 14) review round 2: align helper name with the stack Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the identical helper already present higher in the diffusion stack (Phase 16). The helper definition, the runtime quantiser call, and the offline builder are now byte-identical to that version, so the two branches no longer introduce a divergent name for the same single-source-of-truth and the stack merges without a conflict on this fix. No behavior change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: eager patches + torch.compile cache speed phase Adds the opt-in speed path for the GGUF diffusion transformer behind a selectable speed mode (default off, so output is unchanged until a profile is chosen): - diffusion_eager_patches.py: shared eager fast-paths (channels_last, attention/backend selection, fused norms and QKV) installed at load and rolled back on unload or failed load. - diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent torch.compile cache and the GGUF-transformer compile wiring. - diffusion_arch_patches.py: architecture-specific patches. - diffusion_patch_backend.py: shared install/restore plumbing. - diffusion_speed.py: speed-profile planning. Tests for each module plus the benchmarking and probe scripts used to measure speed, memory, and accuracy of the path. * Studio diffusion: image workflows (safetensors, image-conditioned, editing) + Images UI Backend: - Load non-GGUF safetensors models: full bnb-4bit pipelines and single-file fp8 transformers, gated to the unsloth org plus a curated allowlist. - Image-conditioned workflows built with Pipeline.from_pipe so they reuse the loaded transformer/VAE/text-encoder with no extra VRAM: img2img, inpaint, outpaint, and a hires-fix upscale pass. - Instruction editing as its own family kind (Qwen-Image-Edit-2511, FLUX.1-Kontext-dev) and FLUX.2-klein reference conditioning (single and multi-reference) plus klein inpaint. - Auto-resize odd-sized inputs to a multiple of 16 (and resize the matched mask) so img2img/inpaint/edit no longer reject non-/16 uploads. Bound the decoded image size and cap upscale output to avoid OOM on large inputs. - Fixes: from_pipe defaulting to a float32 recast that crashed torchao quantized transformers; image-conditioned calls forcing the slider size onto the input image. Native sd.cpp engine rejects image-conditioned and reference requests it cannot serve. Frontend: - Redesigned Images page with capability-gated workflow tabs (Create, Transform, Inpaint, Extend, Upscale, Reference, Edit), a brush mask editor, client-side outpaint, and a multi-reference picker. - Advanced options moved to a right-docked panel mirroring Chat: closed by default, toggled by a single fixed top-bar button that stays in place. sd.cpp installer: pin the release, verify each download's sha256, add a download timeout, and make the source repo configurable for a future mirror. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: correct the Advanced panel comment (closed by default, fixed toggle) * Studio: do not force diffusers pipelines cross-tagged gguf into the GGUF variant expander Some diffusers image repos (e.g. unsloth/Qwen-Image-2512-unsloth-bnb-4bit) carry a stray "gguf" tag on the Hub but ship no .gguf files. The model search classified them as GGUF from the bare tag, so the picker rendered the GGUF variant expander, which then dead-ended at "No GGUF variants found." Trust the bare gguf tag only when the repo is not a diffusers pipeline; the -GGUF name suffix and real gguf metadata (populated via expand=gguf) remain authoritative, so genuine GGUF repos are unaffected. * Studio Images: load non-curated unsloth/on-device diffusers repos instead of no-op handleModelSelect only loaded curated safetensors ids and GGUF variant picks; any other non-GGUF pick (an on-device diffusers folder, or a future unsloth diffusers image repo surfaced by search) silently did nothing. Treat such a pick as a full diffusers pipeline load when the id is unsloth-hosted or on-device (the backend infers the family + base repo and gates loads to unsloth/* or local paths), and show a clear message otherwise instead of silently ignoring the click. Curated and GGUF paths are unchanged. * Studio Images: keep curated safetensors models in Recommended after download The curated bnb-4bit / fp8 diffusion rows were filtered out of the Images picker's Recommended list once cached (curatedSafetensorsRows dropped anything in downloadedSet), so they vanished from the picker after the first load and could only be found by typing an exact search. The row already renders a downloaded badge, matching how GGUF Recommended rows stay visible when cached. Drop the exclusion so the curated safetensors always list. * Studio Images: clarify the GGUF transformer-quant Advanced control Renamed the confusing "Transformer quant / GGUF default" control to "GGUF speed mode" with an "Off (run the GGUF)" default, and reworded the hint to state plainly that FP8/INT8/ FP4 load the FULL base model (larger download + more VRAM) rather than re-packing the GGUF, falling back to the GGUF if it can't fit. Behavior unchanged; labels/hint only. * Studio Images: list on-device unsloth diffusion models in the picker The Images picker's On Device tab hid every non-GGUF cached repo whenever a task filter was active, so downloaded unsloth diffusion pipelines (bnb-4bit and FP8 safetensors) never showed up there. List cached repos that pass the task gate, limited under a filter to unsloth-hosted ones so base repos (which fail the diffusion load trust gate) don't appear only to dead-end on click. Chat behavior is unchanged: the task gate still drops image repos there. * Studio: hide single-file image checkpoints from the chat model picker The chat picker treats a cached repo as an image model, and hides it, only when it ships a diffusers model_index.json. Single-file, ComfyUI, and ControlNet image checkpoints (an FP8 Qwen-Image, a z-image safetensors, a Qwen-Image ControlNet) carry none, so they surfaced as loadable chat models. Fall back to resolving the repo id against the known diffusion families, the same resolver the Images backend loads from, so these checkpoints are tagged text-to-image and stay in the Images picker only. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: add the FLUX.2-dev model family Loading unsloth/FLUX.2-dev-GGUF failed because detect_family knew only the Qwen3-based FLUX.2-klein, so FLUX.2-dev (the full, Mistral-based Flux2Pipeline) resolved to nothing and the load errored. Add a flux.2-dev family: Flux2Pipeline + Flux2Transformer2DModel over the black-forest-labs/FLUX.2-dev base repo (gated, reachable with an HF token), with its FLUX.2 32-channel VAE and Mistral text encoder wired for the sd-cli path from the open Comfy-Org/flux2-dev mirror. text-to-image only: diffusers 0.38 ships no Flux2 img2img / inpaint pipeline for dev. Frontend gets sensible dev defaults (28 steps, guidance 4), distinct from klein's turbo defaults. Verified live: GGUF load resolves the family + gated base repo and generates a real 1024x1024 image on GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio Images: clearer error for an unsupported diffusion model When a repo id resolves to no diffusion family the load raised 'Could not infer a diffusion family... Pass family_override (z-image)', which points at an unrelated family and doesn't say what is supported. Replace it with a message that lists the supported families (from a new supported_family_names helper) and notes that video models and image models whose diffusers transformer has no single-file loader are not supported. Applies to both the diffusers and native sd.cpp load paths. Also refreshes two stale family-registry comments that still called FLUX.2-dev omitted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove stray async task scratch outputs committed by mistake * Diffusion: guard trust check against OSError and validate conditioning inputs - _is_trusted_diffusion_repo: wrap Path.exists() so a repo id with invalid characters (or a bare owner/name id) can't raise OSError; treat any failure as not-a-local-path and fall through to the unsloth/ allowlist. validate_load_request still raises the clear FileNotFoundError for a genuinely missing local pick. - generate(): reject mask_image / upscale / reference_images supplied without an input image, and reject reference_images on a family that does not support reference conditioning, instead of silently degrading to txt2img / img2img. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address Codex review findings on the image-workflows PR Keep diffusion.py importable without torch: the compile/arch patch modules import torch at module level, so import them lazily at their load/unload call sites instead of at module load. This restores the torchless contract so get_diffusion_backend() works on a CPU/native sd.cpp install. Match family reject keywords and aliases as whole path/name segments, not raw substrings, so an unrelated word like edited, edition, or kontextual no longer misroutes or hides a valid base image model, while supported edit families (Qwen-Image-Edit, FLUX Kontext) still resolve. Mirror the same segment matching in the picker task filter. Route FLUX.2-dev native guidance through --guidance like the other FLUX families rather than --cfg-scale. Reject native upscale requests that have no input image. Read image header dimensions and reject over-limit inputs before decoding pixels, so a crafted small-payload image cannot spike memory. Reject an upscale that would shrink the source below its input size. Validate the model_kind against the filename extension before the GPU handoff. Estimate a local diffusers pipeline's size from its on-disk weights so auto memory planning does not skip offload and OOM. Report workflows: [txt2img] from the native backend status so the Create tab stays enabled for a loaded native model. Clamp the outpaint canvas to the backend's 4096px decode limit. Adds regression tests for segment matching and kind/extension validation. * Address further Codex findings on the image-workflows PR - Persist the actual output image size in the gallery recipe instead of the request sliders: Transform/Inpaint/Edit derive the size from the uploaded image, Extend grows the canvas, and Upscale resizes it, so the sliders recorded (and later restored) the wrong dimensions for those workflows. - Reject a remote '*-GGUF' repo loaded as a full pipeline (no single-file name) in validate_load_request, so the unloadable pick fails before chat is evicted rather than deep in from_pretrained. - Only publish an image-conditioned from_pipe wrapper to the shared aux cache when the load is still current: from_pipe runs under the generate lock but not the state lock, so an unload racing its construction could otherwise cache a wrapper over torn-down modules that a later load would reuse. - Verify the Windows CUDA runtime archive checksum before extracting it, like the main sd-cli archive, so a corrupt or tampered runtime is rejected rather than extracted next to the binary. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
|
Daniel Han
|
9c2eacc35e
|
Studio: reserve CUDA context and mmproj/MTP soft overhead in the GGUF fit budget (#6718)
--------- Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
|
Daniel Han
|
7d8b2db236
|
Studio diffusion: persistent sd-server for the native engine (load once, serve many) (#6768)
* Studio diffusion: cross-platform device policy, fp16 guard, lock split, validate-before-evict Phase 1 of porting the richer diffusion stack onto the image-generation backend. - Add a compartmentalized device/dtype policy module (diffusion_device.py) resolving CUDA/ROCm/XPU/MPS/CPU with capability flags. Keeps the NVIDIA capability-based bf16 choice; ROCm and XPU are isolated; MPS uses bf16 or fp32, never a silent fp16 that renders a black image. - Add a per-family fp16_incompatible flag (Z-Image) and promote a resolved float16 to float32 for those families so they do not produce black images. - Split the backend locks: a generation holds only _generate_lock, so status, unload, and a new load are never blocked by a long denoise. Add per-generation cancellation via callback_on_step_end so an eviction or a superseding load preempts a running generation; a replacement load waits for it to stop before allocating, so two pipelines never sit in VRAM at once. - Validate a load request before the GPU handoff so an unloadable pick never evicts a working chat model, and reject missing local paths up front. - Add CPU-only tests for the device policy, dtype guard, lock split and cancellation, and validate-before-evict, plus a GPU benchmark/regression script (scripts/diffusion_bench.py) measuring latency, peak VRAM, and PSNR against a saved reference. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2A): measured-budget memory planner + offload/VAE policy Add a lean, backend-agnostic memory policy that picks a CPU-offload policy and VAE tiling/slicing from measured free device memory vs the model's estimated resident footprint, then applies it to the built pipeline. auto stays resident when the model fits (byte-identical to the prior resident path), and falls to whole-module offload when tight; fast/balanced/low_vram are explicit overrides. Sequential submodule offload is unreliable for GGUF transformers on diffusers 0.38, so it falls back to whole-module offload and status reports the policy actually engaged. Verified on Z-Image-Turbo Q4_K_M (B200): auto reproduces the resident image with no VRAM/latency regression (PSNR inf); balanced/low_vram cut generation peak VRAM 47.9% (15951 -> 8318 MB) with byte-identical output, at the expected latency cost. 73 prior + 35 new CPU tests pass. * Studio diffusion (Phase 2D): streamed block-level offload + functional VAE tiling Add a streamed 'group' offload tier (diffusers apply_group_offloading, block_level, use_stream) that keeps the transformer flowing through the GPU a few blocks at a time while the text encoder / VAE stay resident, and fix VAE tiling to drive the VAE submodule (pipelines like Z-Image expose enable_tiling on pipe.vae, not the pipeline). apply_memory_plan now returns the (policy, tiling) actually engaged so status never overstates either, and group falls back to whole-module offload when the transformer can't be streamed. Measured on Z-Image (B200), all lossless (PSNR inf vs resident): balanced/group cuts generation peak VRAM 32% (15951 -> 10840 MB) at near-resident speed (2.07 -> 2.99s); low_vram/model cuts it 48% (-> 8318 MB) but is slower (7.99s). Mode names now match that tradeoff: balanced = stream the transformer, low_vram = offload every component. auto picks group when the companions fit resident, else model. 112 CPU tests pass. * Studio diffusion (Phase 5): image quality-vs-quant accuracy harness Add scripts/diffusion_quality.py, the accuracy analogue of the KLD workflow: hold prompt + seed fixed, render a grid with a reference quant (default BF16), then render each candidate quant and measure drift from the reference. Records mean PSNR + SSIM (pure-numpy, no skimage/scipy) and optional CLIP text-alignment + image-similarity (transformers, --clip), plus file size, latency, and peak VRAM, then prints a quality-vs-cost table and recommends the smallest quant within a quality budget. --selftest validates the metrics on synthetic images with no GPU or model. Verified on Z-Image (B200): the table degrades monotonically with quant size (Q8 -> Q4 -> Q2: PSNR 21.7 -> 15.5, SSIM 0.82 -> 0.61), while CLIP-text stays flat (~0.34) -- quantization erodes fine detail far more than prompt adherence. * Studio diffusion (Phase 3): opt-in speed layer (channels_last / compile / TF32) Add a speed_mode knob (off by default, so the render path stays bit-identical): default applies channels_last VAE + regional torch.compile of the denoiser's repeated block where eligible; max also enables TF32 matmul and fused QKV. Regional compile is gated off for the GGUF transformer (dequantises per-op) and for families flagged not compile-friendly (a new supports_torch_compile flag, False for Z-Image), so it activates automatically only once a non-GGUF bf16 transformer is loaded. Speed optims run before placement/offload, per the diffusers composition order. status now reports speed_mode + the optims actually engaged. Verified on Z-Image (B200): default -> ['channels_last'], max -> ['channels_last', 'tf32'], compile correctly skipped for GGUF; generation works in every mode. 121 CPU tests pass. * Studio diffusion (Phase 2B): opt-in fp8 text-encoder layerwise casting Add a text_encoder_fp8 knob that casts the companion text encoder(s) to fp8 (e4m3) storage via diffusers apply_layerwise_casting, upcasting per layer to the bf16 compute dtype while normalisations and embeddings stay full precision. Applied before placement, gated to CUDA + bf16, best-effort (a failure leaves the encoder dense). status reports which encoders were cast. Verified on Z-Image (B200, balanced/group mode where the encoder stays resident): generation peak VRAM dropped 37% (10840 -> 6791 MB, below the lowest-VRAM offload) at near-resident speed. It is a memory-vs-quality tradeoff, not free -- ~20 dB PSNR vs the bf16 encoder, a larger shift than one transformer quant step -- so it is off by default and documented as such, with the Phase 5 harness to size the cost. 127 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 2C): NVFP4 text-encoder quant (+ generalise fp8 knob) Generalise the text-encoder precision knob from a fp8 bool to text_encoder_quant (fp8 | nvfp4). nvfp4 quantises the companion text encoder to 4-bit via torchao NVFP4 weight-only (two-level microscaling) on Blackwell's FP4 tensor cores; fp8 stays the broader-hardware path (cc>=8.9). Both are gated, best-effort, and run before placement; status reports the mode actually engaged. This is the lean realisation of GGUF-native text-encoder quant: 4-bit on the encoder without the 3045-line port. Verified on Z-Image (B200, balanced/group where the encoder stays resident), vs the bf16 encoder: nvfp4 cut generation peak VRAM 48% (10840 -> 5593 MB, the lowest TE option, below whole-model offload) at near-fp8 quality (16.4 vs 17.1 dB PSNR), and both quants ran faster than bf16. A memory-vs-quality tradeoff (off by default); size it per model with the Phase 5 quality harness. diffusion_bench gains --text-encoder-quant. 129 CPU tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): native stable-diffusion.cpp engine for CPU/Mac Adds the CPU / Apple-Silicon tier of the two-engine strategy, mirroring the chat backend's llama.cpp shell-out. Diffusers stays the default on CUDA / ROCm / XPU; this covers the hardware diffusers serves poorly, consuming the same split GGUF assets Studio already curates. - sd_cpp_args.py: pure sd-cli command builder. Maps the family to its text-encoder flag (Z-Image Qwen3 to --llm, Qwen-Image to --qwen2vl, FLUX.1 CLIP-L + T5), and the diffusers memory policy (none/group/model/sequential) to sd.cpp's offload flags (--offload-to-cpu / --clip-on-cpu / --vae-on-cpu / --vae-tiling / --diffusion-fa), so one user knob drives both engines. - sd_cpp_engine.py: SdCppEngine over a located sd-cli. find_sd_cpp_binary() with the same precedence as the llama finder (env override, then the Studio install root, then in-tree, then PATH), an is_available/version probe, and a one-shot subprocess generate that streams progress and returns the PNG. runtime_env() prepends the binary's directory to the platform library path so a prebuilt's bundled libstable-diffusion.so resolves. select_diffusion_engine() is the pure routing decision (GPU backends to diffusers, CPU/MPS to native when present). - install_sd_cpp_prebuilt.py: resolve + download the per-host prebuilt (macOS-arm64/Metal, Linux x86_64 CPU, Vulkan/ROCm/Windows variants) into the Studio install root. resolve_release_asset() is a pure, unit-tested host-to-asset matrix. - scripts/sd_cpp_smoke.py: end-to-end native generation harness. Tests (CPU-only, subprocess/filesystem stubbed): 49 new across args, engine, routing, runtime env, and the installer resolver. Full diffusion suite 166 passing. Verified on a B200 box: built sd-cli (CUDA) and the prebuilt (CPU) both generate Z-Image-Turbo Q4_K end to end through SdCppEngine: balanced (group offload, 5.0s gen), low_vram (full CPU offload + VAE tiling, 13.4s), and the dynamically-linked CPU prebuilt (50.4s on CPU), all producing coherent images. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 6): img2img / inpaint / edit / LoRA / upscale on the native engine Builds on Phase 4's native stable-diffusion.cpp engine, extending it from text-to-image to the wider feature surface, since sd.cpp supports all of these through the binary already. Pure command-builder additions plus one engine method, so the txt2img path is unchanged. - sd_cpp_args.py: SdCppGenParams gains image-conditioning fields. init_img + strength make a run img2img, adding mask makes it inpaint, ref_images drives FLUX-Kontext / Qwen-Image-Edit style editing (repeated --ref-image), and lora_dir + the <lora:name:weight> prompt syntax select LoRAs. New SdCppUpscaleParams + build_sd_cpp_upscale_command for the ESRGAN upscale run mode (input image + esrgan model, no prompt / text encoders). - sd_cpp_engine.py: the subprocess runner is factored into a shared _run() so generate() (now carrying the conditioning flags) and a new upscale() reuse the same streaming / error / output-check path. - scripts/sd_cpp_smoke.py: --task {txt2img,img2img,upscale} with --init-img / --strength / --upscale-model / --upscale-repeats. Tests: 10 new across the img2img / inpaint / edit / LoRA flag construction, the upscale builder and its validation, and the engine's img2img + upscale paths. Full diffusion suite 176 passing. Verified on a B200 box through SdCppEngine: img2img (Z-Image-Turbo Q4_K, the init image conditioned at strength 0.6, 4.8s) and ESRGAN upscale (512x512 -> 2048x2048 via RealESRGAN_x4plus_anime_6B, 2.7s), both producing coherent images. Video and the diffusers-path feature wiring are deferred. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): accuracy-preserving speed pass Re-review of the diffusion stack (#6675/#6679/#6680) surfaced one real accuracy bug and a dead-on-arrival speed path; this fixes both and adds the lossless / near-lossless wins, all measured on a B200. Correctness: - TF32 global-state leak (fix). speed_mode=max flipped torch.backends.*.allow_tf32 process-wide and never restored them, so a later `off` load silently inherited TF32 and was no longer bit-identical. Added snapshot_backend_flags / restore_backend_flags (TF32 + cudnn.benchmark), captured before the speed layer runs and restored on unload. Verified: load max -> unload -> load off is now byte-identical (PSNR inf) to a fresh off. - sd-cli timeout could hang forever. _run() blocked in `for line in stdout` and only checked the timeout after EOF, so a child stuck in model load / GPU init with no output ignored the timeout. Drained stdout on a reader thread with a wall-clock deadline. Added a silent-hang regression test. Speed (diffusers path), near-lossless, opt-in tiers: - Regional torch.compile now runs on the GGUF transformer. The is_gguf gate (and Z-Image's supports_torch_compile=False) were stale: compile_repeated_blocks compiles and runs ~2.2x faster on the GGUF Z-Image transformer on torch 2.9.1 / diffusers 0.38 (the per-op dequant stays eager, the rest of the block compiles). Measured: off 1.80s -> default 0.82s/gen (+54.7%), PSNR 37.7 dB vs eager -- far above the Q4 quant noise floor (~21 dB), so it does not move output quality. Gate relaxed; default tier delivers it. - cudnn.benchmark added to the default tier (autotunes the fixed-shape VAE convs). - torch.inference_mode() around the pipeline call (lossless, strictly faster than the no_grad diffusers uses internally). Memory path: - VAE tiling (not bit-identical >1MP) restricted to the model/sequential/CPU tiers; the balanced (group) tier keeps exact slicing only, so it is now bit-identical to the resident image (verified PSNR inf) and slightly faster. - Group offload adds non_blocking + record_stream on the CUDA stream path to overlap each block's H2D copy with compute (lossless; gated on the installed diffusers signature so older versions still work). Native (sd.cpp) path: - native_speed_flags: a first-class speed knob (default -> --diffusion-fa, a near-lossless CUDA win that was previously only added on offload tiers; max also -> --diffusion-conv-direct). conv-direct stays opt-in: measured +45% on CUDA, so it is never auto-on. Engine generate() merges it, de-duped against offload flags. Default profile: a GGUF model with no explicit speed_mode now resolves to the `default` profile (resolve_speed_mode), since compile's perturbation sits below the quantisation noise floor and so does not reduce quality versus the dense reference; out of the box a GGUF Z-Image generation drops from 1.80s to 0.81s. Dense models stay `off` / bit-identical, and an explicit speed_mode -- including "off" -- is always honored, so the byte-identical path remains one flag away and is the regression reference. Tooling: scripts/compile_probe.py (eager vs compiled GGUF probe), scripts/ perf_verify.py (the B200 verification above), and diffusion_bench.py gains --speed-mode so the speed tiers are benchmarkable. Tests: 183 passing (was 166); new coverage for the backend-flag snapshot/restore, GGUF compile eligibility, the balanced tiling/slicing split, native_speed_flags + the engine de-dup, and the sd-cli silent-hang timeout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7): max tier uses max-autotune-no-cudagraphs + engine/lever benchmarks The opt-in `max` speed tier now compiles the repeated block with mode=max-autotune-no-cudagraphs (dynamic=False) instead of the default mode: Triton autotuning for GEMM/conv-heavier models, gated to the tier where a longer cold compile is acceptable. CUDA-graph modes (reduce-overhead / max-autotune) are deliberately avoided -- both crash on the regionally-compiled block (its static output buffer is overwritten across denoise steps), measured. Adds two reproducible benchmarks used to validate the optimization research: - scripts/compare_engines.py: PyTorch (diffusers GGUF) vs native sd.cpp head-to-head. - scripts/leverage_probe.py: coordinate_descent_tuning + FirstBlockCache probes. Measured on B200 (Z-Image Q4_K_M, 1024px, 8 steps): default compile 0.80s/gen; coordinate_descent_tuning 0.79s (within noise, already covered by max-autotune); FirstBlockCache does not run on Z-Image (diffusers 0.38 block-detection / Dynamo). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source) Add an opt-in transformer_quant mode that loads the dense bf16 transformer and torchao-quantises it onto the low-precision tensor cores, instead of the GGUF transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's 0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor. GGUF+compile stays the low-memory default and the fallback. The mode is gated on CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB); any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe (Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips the tiny projections that crash int8's torch._int_mm. New module mirrors diffusion_precision.py; quant runs before compile before placement. 184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py is the standalone torchao lever probe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): consumer-GPU tuning - lock fp8 fast accumulate, prefer fp8 over mxfp8, reject 2:4 sparsity Consumer Blackwell halves tensor-core throughput on FP32 accumulate (fp8 419 vs 838 TFLOPS with FP16 accumulate; bf16 209), so: - fp8 config locks use_fast_accum=True (Float8MMConfig). torchao already defaults it on; pinning it guards consumer cards against a default change. On B200 it is identical speed and slightly better quality (LPIPS 0.050 vs 0.091). - the Blackwell auto ladder prefers fp8 over mxfp8 (measured faster + more accurate). 2:4 semi-structured sparsity evaluated and rejected (scripts/sparse_accum_probe.py): 2:4 magnitude-prune + fp8 gives LPIPS 0.858 (broken image) with no fine-tune, the cuSPARSELt kernel errors on torch 2.9, and it does not compose with torch.compile (our main ~2x). Documented as a dead end, not shipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add fp8 fast-accum overflow verification probe scripts/fp8_overflow_check.py hooks every quantised linear during a real Z-Image generation and reports max-abs + non-finite counts for use_fast_accum True vs False. Confirms fast accumulation is an accumulation-precision knob, not an overflow one: across 276 linears, including Z-Image's ~1.0e6 activation peaks (which overflow FP16), 0 non-finite elements and identical max-abs for both modes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40; GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it. Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s), no overflow, quality below the quant noise floor. So the default leans to accuracy on data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force) lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off). 187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override threading). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): add NVFP4 probe documenting it is not yet a win on torch 2.9 scripts/nvfp4_probe.py measures NVFP4 via torchao on the real Z-Image transformer. Finding (B200, 1024px/8 steps): NVFP4 is a torchao feature and DOES run with use_triton_kernel=False (the default triton path needs the missing MSLK library), but only at bf16-compile rate (0.667s vs fp8 0.592s) -- it dequantises FP4->bf16 rather than using the FP4 tensor cores. The real FP4 speedup needs MSLK or torch>=2.11 + torchao's CUTLASS FP4 GEMM. The smoke probe (default triton=True) already keeps NVFP4 out of auto on this env, so auto correctly stays on fp8; NVFP4 activates automatically once fast. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 8): prefer fp8 over nvfp4 in Blackwell auto ladder Validated NVFP4 on torch 2.11 + torchao CUTLASS FP4 in an isolated env. The FP4 tensor-core GEMM is genuinely active there (a 16384^3 GEMM hits ~3826 TFLOPS, 2.52x bf16 and 1.37x fp8), but it only beats fp8 on very large GEMMs. At the diffusion transformer's shapes (hidden ~3072, MLP ~12288, M~4096) NVFP4 is both slower (0.81x fp8 end to end on Z-Image 1024px) and less accurate (LPIPS 0.166 vs fp8's 0.044). Reorder the Blackwell auto ladder to fp8 before nvfp4 so auto is correct even on a future MSLK-equipped box; nvfp4 stays an explicit opt-in. Add scripts/nvfp4_t211_probe.py (extension diagnostics + GEMM micro + end-to-end). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): pre-quantized transformer loading The Phase 8 fast transformer_quant path materialises the dense bf16 transformer on the GPU and torchao-quantises it in place, so its load peak is ~2x GGUF's (~21 vs 13.4 GB) plus a ~12 GB download. Add a pre-quantized branch: quantise once offline (scripts/build_prequant_checkpoint.py) and at runtime build the transformer skeleton on the meta device (accelerate.init_empty_weights) and load_state_dict(assign=True) the quantized weights, so the dense bf16 never touches the GPU. Measured (B200, Z-Image fp8): full-pipeline GPU load peak 21.2 -> 14.6 GB (matching GGUF's 13.4), on-disk 12 -> 6.28 GB, output bit-identical (LPIPS 0.0). It is the same torchao config + min_features filter the runtime path uses, applied ahead of time. New core/inference/diffusion_prequant.py (resolve_prequant_source + load_prequantized_transformer, best-effort, lazy imports). diffusion.py _load_dense_quant_pipeline tries the pre-quant source first and falls back to the dense materialise+quantise path, then to GGUF, so the default is unchanged. DiffusionLoadRequest gains transformer_prequant_path; DiffusionFamily gains an empty prequant_repos map for hosted checkpoints (hosting deferred). Hermetic CPU tests for the resolver, the meta-init+assign loader, and the backend branch selection + fallbacks; GPU verification via scripts/verify_prequant_backend.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): attention-backend selection Add a selectable attention kernel via the diffusers set_attention_backend dispatcher. Attention is memory-bandwidth bound, so a better kernel is an end-to-end win orthogonal to the linear-weight quantisation (it speeds the QK/PV matmuls torchao never touches) and composes with torch.compile. auto picks the best exact backend for the device: cuDNN fused attention (_native_cudnn) on NVIDIA when a speed profile is active, measured ~1.18x end-to-end on a B200 (Z-Image 1024px/8 steps) with LPIPS ~0.004 vs the default (below the compile/quant noise floor); native SDPA elsewhere and when speed=off (so off stays bit-identical). Explicit native/cudnn/flash/flash3/flash4/sage/ xformers/aiter are honored, and an unavailable kernel falls back to the default rather than failing the load. New core/inference/diffusion_attention.py (normalize + per-device select + apply, best-effort, lazy imports). Set on pipe.transformer BEFORE compile in load_pipeline; attention_backend threads through begin_load / load_pipeline / status like the other load knobs. New request field attention_backend + status field. Hermetic CPU tests for normalize / select policy / apply fallback, plus route threading + 422. Measured via scripts/perf_levers_probe.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 11): prefer int8 on consumer GPUs in the auto ladder Consumer / workstation GPUs halve fp8 (and fp16/bf16) FP32-accumulate tensor-core throughput, while int8 runs at full rate (int32 accumulate is not nerfed). Public benchmarks (SDNQ across RTX 3090/4090/5090, AMD, Intel) confirm int8 via torch._int_mm is as fast or faster than fp8 on every consumer part, and the only path on pre-Ada consumer cards without fp8 tensor cores. So when transformer_quant=auto, reorder the arch tier to put int8 first on a consumer/workstation GPU (detected by the existing _is_consumer_gpu name heuristic), while data-center HBM parts keep fp8 first. Pure ladder reorder via _prefer_consumer_scheme; no new flags. Verified non-regression on a B200 (still picks fp8). Hermetic tests for consumer Blackwell/Ada/workstation (-> int8) and data-center Ada/Hopper/Blackwell (-> fp8). * Studio diffusion (Phase 12): First-Block-Cache step caching for many-step DiT Add opt-in step caching (First-Block-Cache) for the diffusion transformer. Across denoise steps a DiT's output settles, so once the first block's residual barely changes the remaining blocks are skipped and their cached output reused. diffusers ships it natively (FirstBlockCacheConfig + transformer.enable_cache, with the standalone apply_first_block_cache hook as a fallback). Measured on Flux.1-dev (28 steps, 1024px): ~1.4x on top of torch.compile (2.83 -> 2.03s) at LPIPS ~0.08 vs the no-cache output, well inside the quality bar. OFF by default and a per-load opt-in: the win scales with step count, so it is for many-step models (Flux / Qwen-Image) and pointless for few-step distilled models (e.g. Z-Image-Turbo at ~8 steps), where a single skipped step is a large fraction of the trajectory. It composes with regional compile only with fullgraph=False (the cache's per-step decision is a torch.compiler.disable graph break), which the speed layer now switches to automatically when a cache is engaged. Best-effort: a model whose block signature the hook does not recognise is caught and the load proceeds uncached. - new core/inference/diffusion_cache.py: normalize_transformer_cache + apply_step_cache (enable_cache / apply_first_block_cache fallback; threshold auto-raised for a quantised transformer per ParaAttention's fp8 guidance; lazy diffusers import). - diffusion_speed.py: apply_speed_optims takes cache_active; compile drops fullgraph when a cache is engaged. - diffusion.py: apply_step_cache before compile; thread transformer_cache / transformer_cache_threshold through begin_load -> load_pipeline and report the engaged mode in status(). - models/inference.py + routes/inference.py: transformer_cache (off | fbcache) and transformer_cache_threshold request fields, engaged mode in the status response. - hermetic tests for normalisation, the enable_cache / hook-fallback paths, threshold selection, and best-effort failure handling, plus route threading + validation. - scripts/fbcache_flux_probe.py: the Flux validation probe (latency / speedup / VRAM / LPIPS vs the compiled no-cache baseline). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 14): fix int8 dense quant on Flux / Qwen (skip M=1 modulation linears) The opt-in dense int8 transformer path crashed on Flux.1 and Qwen-Image with 'torch._int_mm: self.size(0) needs to be greater than 16, but got 1'. int8 dynamic quant goes through torch._int_mm, which requires the activation row count M > 16. A DiT's AdaLN modulation projections (Flux norm1.linear 3072->18432, Qwen img_mod.1 / txt_mod.1, Flux.2 *_modulation.linear) and its timestep / guidance / pooled-text conditioning embedders are computed once from the [batch, dim] conditioning vector (M = batch = 1), not per token, so they hit _int_mm at M=1 and crash. Their feature dims are large, so the existing min_features filter did not exclude them. Fix: the int8 filter now also skips any Linear whose fully-qualified name matches a modulation / conditioning-embedder token (norm, _mod, modulation, timestep_embed, guidance_embed, time_text_embed, pooled). These layers run at M=1 once per block and are a negligible share of the FLOPs, so int8 keeps the full speedup on the attention / FFN layers (M = sequence length). fp8 / nvfp4 / mxfp8 use scaled_mm, which has no M>16 limit and quantises these layers fine, so the exclusion is int8-only. Sequence embedders (context_embedder / x_embedder / txt_in, M = seq) are deliberately not excluded -- note 'context_embedder' contains the substring 'text_embed', which is why the token is the specific 'time_text_embed', not 'text_embed'. Measured on a B200 (1024px, transformer_quant=int8 + speed=default), int8 now runs on every supported model and is the fastest dense path on Flux/Qwen (int8 runs full-rate vs fp8's FP32-accumulate): FLUX.1-dev 9.62s eager -> 1.98s (4.86x, vs fp8 2.15s), Qwen-Image -> 1.87s (5.57x, vs fp8 2.09s), FLUX.1-schnell -> 0.41s (3.59x). Z-Image and Flux.2-klein (already working) are unchanged. - diffusion_transformer_quant.py: add _INT8_EXCLUDE_NAME_TOKENS; make_filter_fn takes exclude_name_tokens; quantize_transformer passes it for int8 only. - hermetic test that the int8 filter excludes the modulation / embedder linears (and keeps attention / FFN / sequence-embedder linears), while fp8 keeps them. - scripts/int8_linear_probe.py: the meta-device probe used to enumerate each transformer's Linear layers and derive the exclusion list. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 15): build int8 pre-quantized checkpoints (skip M=1 modulation linears) The prequant-checkpoint builder applied the dense quant filter without the int8-only M=1 modulation / conditioning-embedder exclusion the runtime path uses, so a built int8 checkpoint baked those projections as int8 and crashed (torch._int_mm needs M>16) at the first denoise step on Flux / Qwen. Factor the scheme->exclusion decision into a shared exclude_tokens_for_scheme() used by both the runtime quantise path and the offline builder so they can never drift, and apply it in build_prequant_checkpoint.py. int8 prequant now produces a working checkpoint on every supported model, giving int8 (the consumer-preferred scheme) the same ~2x load-VRAM and download reduction fp8 already had. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16): route no-GPU loads to the native sd.cpp engine When no CUDA/ROCm/XPU GPU is available, route diffusion load/generate to the native stable-diffusion.cpp engine instead of diffusers, with diffusers as the guaranteed fallback. On CPU sd.cpp is 1.4-2.8x faster and uses 1.5-2.2x less RAM. - diffusion_engine_router: centralised engine selection (built on the existing select_diffusion_engine), env opt-outs, MPS gating, recorded fallback reason. - sd_cpp_backend (SdCppDiffusionBackend): the diffusers backend method surface backed by sd-cli, with lazy binary install, registry-driven asset fetch, step-progress parsing, and cancellation. - diffusion_families: per-family single-file VAE + text-encoder asset mapping. - sd_cpp_engine: cancellation support (process-group kill + SdCppCancelled). - routes/inference + gpu_arbiter: drive the active engine via the router; the API now reports the active engine and any fallback reason. - tests for the backend, router, route selection, and cancellation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Phase 16 review fixes: engine-switch unload, sd.cpp error mapping, per-image seeds, Qwen sampler Address review feedback on #6724: - engine router: unload the engine being deactivated on a switch, so the old model is not left resident-but-unreachable (the evictor only targets the active engine). - generate route: sd.cpp execution errors (nonzero exit / timeout / missing output) now map to 500, not 409 (which only means not-loaded / cancelled). - native batch: return per-image seeds and persist the actual seed for each image so every batch image is reproducible. - Qwen-Image native path: apply --sampling-method euler --flow-shift 3 per the stable-diffusion.cpp docs; other families keep sd-cli defaults. - honor speed_mode (native --diffusion-fa) and, off-CPU, memory_mode/cpu_offload offload flags on the native load instead of hardcoding them off. - fail the load when the sd-cli binary is present but not runnable (version() now returns None on exec error / nonzero exit). - size estimate: only treat the transformer asset as a possible local path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 9): gate request-supplied local prequant paths behind operator opt-in load_prequantized_transformer ends in torch.load(weights_only=False), which executes arbitrary code from the pickle. The transformer_prequant_path load-request field reached that unpickle for any local file an authenticated caller named, so a request could trigger remote code execution. Refuse the source.kind=='path' branch unless the operator sets UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1; the first-party hosted-repo checkpoint stays trusted and unaffected. Document the requirement on the API field and add gate tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 10): reset the global attention backend on native, gate arch-specific kernels, accept sdpa - apply_attention_backend now restores the native default when no backend is requested or a kernel fails. diffusers keeps a process-wide active attention backend that set_attention_backend updates, and a fresh transformer's processors follow it, so a load that wanted native could silently inherit a backend (e.g. cuDNN) an earlier speed-profile load pinned, breaking the bit-identical/off guarantee. - select_attention_backend drops flash3/flash4 up front when the CUDA capability is below Hopper/Blackwell. diffusers only checks the kernels package at set time, so an explicit request on the wrong card set fine then crashed mid-generation; it now falls back to native. - Add the sdpa alias to the attention_backend Literal so an API request with sdpa (already a valid alias of native) is accepted instead of 422-rejected by Pydantic. - Drop the dead replace('-','_') normalization (no alias uses dashes/underscores). - perf_levers_probe.py output dir is now relative to the script, not a hardcoded path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 12): only engage FBCache on context-aware transformers; quantized threshold for GGUF - apply_step_cache now engages only via the transformer's native enable_cache (the diffusers CacheMixin path), which exists exactly when the pipeline wraps the transformer call in a cache_context. The standalone apply_first_block_cache fallback installed on non-CacheMixin transformers too (e.g. Z-Image), whose pipeline opens no cache_context, so the load reported transformer_cache=fbcache and then the first generation crashed inside the hook. Such a model now runs uncached per the best-effort contract. - GGUF transformers are quantized (the default Studio load path), so they now use the higher quantized FBCache threshold when the caller leaves it unset, instead of the dense default that could keep the cache from triggering. - fbcache_flux_probe.py: compile cached runs with fullgraph=False (FBCache is a graph break, so fullgraph=True failed warmup and silently measured an eager cached run); output dir is now relative to the script, not a hardcoded path. * Studio diffusion (Phase 11): keep professional RTX cards on the fp8 ladder _is_consumer_gpu treated professional parts (RTX PRO 6000 Blackwell, RTX 6000 Ada) as consumer because their names carry no datacenter token, so the auto ladder moved int8 ahead of fp8 and the fp8 path chose fast accumulate for them. The rest of the backend already classifies these as datacenter/professional (llama_cpp.py _DATACENTER_GPU_RE), so detect the same RTX PRO 6000 / RTX 6000 Ada markers here and keep fp8 first with precise accumulate. Also fix the consumer-Blackwell test to use compute capability (10, 0) instead of (12, 0). * Studio diffusion (Phase 8): tolerate missing torch.float8_e4m3fn in the mxfp8 config Accessing torch.float8_e4m3fn raises AttributeError on a torch build without it (not just TypeError on older torchao), which would break the mxfp8 config helper instead of falling back to the default. Catch both so the fallback is robust. quant_probe.py: same AttributeError fallback; run LPIPS on CPU so the scorer never holds CUDA memory during the per-row VRAM probe; output dir relative to the script. * Studio diffusion (Phase 7): robust backend-flag snapshot/restore and restore on failed speeded load - snapshot_backend_flags reads each flag defensively (getattr + hasattr), so a build/platform missing one (no cuda.matmul on CPU/MPS) still captures the rest instead of skipping the whole snapshot. restore_backend_flags restores each flag independently so one failure can't leave the others leaked process-wide. - load_pipeline restores the flags (and clears the GPU cache) when the build fails after apply_speed_optims mutated the process-wide flags but before _state captured them for unload to restore -- otherwise a failed default/max load left cudnn.benchmark/TF32 on and contaminated later off generations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4): enforce the sd-cli timeout while reading output Iterating proc.stdout directly blocks until the stream closes, so a sd-cli that hangs without producing output (or without closing stdout) would never reach proc.wait and the wall-clock timeout was silently bypassed. Drain stdout on a daemon thread and wait on the PROCESS, so the main thread always enforces the timeout and kills a hung process (which closes the pipe and ends the reader). Add a test that times out even when stdout blocks, and make the no-binary test hermetic so a host-installed sd-cli can't leak in. * Studio diffusion (Phase 14): guard the int8 exclusion filter against a None fqn The filter callback can be invoked without a module name, so fqn.lower() would raise AttributeError on None. Fall back to an empty name (nothing matches the exclusion tokens, so the linear is kept) instead of crashing the quantise pass. * Studio diffusion (Phase 16) review fixes: native engine robustness - sd_cpp_backend: stop truncating explicit seeds to 53 bits (mask to int64); a large requested seed was silently collapsed (2**53 -> 0) and distinct seeds aliased to the same image. Random seeds stay 53-bit (JS-safe). - sd_cpp_backend: sanitize empty/whitespace hf_token to None so HfApi/hf_hub fall back to anonymous instead of failing auth on a blank token. - sd_cpp_backend: a superseding load now cancels the in-flight generation, so the old sd-cli can no longer return/persist an image from the previous model. - diffusion_engine_router: run the previous engine's unload() OUTSIDE the lock so a slow 10+ GB free / CUDA sync does not block engine selection. - diffusion_engine_router: probe sd-cli runnability (version()) before committing to native, so a present-but-unrunnable binary falls back to diffusers at selection. - diffusion_device: resolve a torch-free CPU target when torch is unavailable, so a CPU-only install can still reach the native sd.cpp engine instead of failing load. - tests updated for the runnability probe + a not-runnable fallback case. * Studio diffusion (Phase 9) review fixes: prequant safety + validation - SECURITY: a request-supplied local pre-quant path is now unpickled only when it resolves inside an operator-configured ALLOWLIST of directories (UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH = dir[:dir...]). The previous boolean opt-in, once enabled for one trusted checkpoint, allowed torch.load(weights_only=False) on any path a load request named (arbitrary code execution). realpath() blocks symlink escapes; a bare on/off toggle is no longer a wildcard. - Validate the checkpoint's min_features against the runtime Linear filter, so a checkpoint that quantised a different layer set is rejected instead of silently loading a model that mismatches the dense path while reporting the same scheme. - Tolerant base_model_id compare (exact or same final path/repo segment), so a local path or fork of the canonical base is accepted instead of falling back to dense. - _has_meta_tensors uses any(chain(...)) (no intermediate lists). - prequant verify/probe scripts use repo-relative paths (+ env overrides), not the author's absolute /mnt paths. - tests: allowlist-dir opt-in, outside-allowlist refusal, min_features mismatch, fork tail. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 7) review fixes: offload fallback + bench scripts - diffusion_memory: when group offload is unavailable and the plan falls back to whole-module offload, enable VAE tiling (the group plan left it off, but the fallback is the low-VRAM path where the decode spike can OOM). Covers both the group and sequential fallback branches. - perf_verify: include the balanced-vs-off PSNR in the pass/fail condition, so a balanced bit-identity regression actually fails the check instead of exiting 0. - compare_engines: --vae/--llm default to None (were author-absolute /mnt paths), and the load-progress poll has a 30 min deadline instead of looping forever on a hang. - test for the group->model fallback enabling VAE tiling. * Studio diffusion (Phase 8) review fixes: quant compile + nvfp4 path - diffusion: a torchao-quantized transformer is committed only compiled. A dense model resolves to speed_mode=off, which would run the quant eager (~30x slower than the GGUF it replaced), so when transformer_quant engaged and speed resolved to off, promote to default (regional compile); warn loudly if compile still does not engage. - diffusion_transformer_quant: build the nvfp4 config with use_triton_kernel=False so the CUTLASS FP4 path is used (torchao defaults to the Triton kernel, which needs MSLK); otherwise the smoke probe fails on CUTLASS-only Blackwell and silently drops to GGUF. - nvfp4_probe: repo-relative output dir + --out-dir (was an author-absolute /mnt path). - test asserts the eager-quant -> default-compile promotion. * Studio diffusion (Phase 10) review fixes: attention gating + probe isolation - diffusion_attention: gate the auto cuDNN-attention upgrade on SM80+; on pre-Ampere NVIDIA (T4/V100) cuDNN fused SDPA is accepted at set time but fails at first generation, so auto now stays on native SDPA there. - diffusion_attention: _active_attention_backend handles get_active_backend() returning an enum/None (not a tuple); the old unpack always raised and was swallowed, so the native-restore short-circuit never fired. - perf_levers_probe: free the resident pipe on a skipped (attn/fbcache) variant; run LPIPS on CPU so it isn't charged to every variant's peak VRAM; reset force_fuse_int_mm_with_mul so the inductor_flags variant doesn't leak into later compiled rows. - tests for the SM80 cuDNN gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review fixes: sd.cpp installer + engine hardening - install_sd_cpp_prebuilt: download the release archive with urlopen + an explicit timeout + copyfileobj (urlretrieve has no timeout and hangs on a stalled socket); extract through a per-member containment check (Zip-Slip guard); expanduser the --install-dir so a tilde path is not taken literally; and on Windows CUDA also fetch the separately-published cudart runtime DLL archive so sd-cli.exe can start. - sd_cpp_engine: find_sd_cpp_binary honors UNSLOTH_STUDIO_HOME / STUDIO_HOME like the installer, so a custom-root install is discovered without UNSLOTH_SD_CPP_PATH; start sd-cli with the parent-death child_popen_kwargs so it is not orphaned on a backend crash; reap the SIGKILLed child (proc.wait) so a cancel/timeout does not leave a zombie. - tests: Zip-Slip rejection, normal extraction, studio-home discovery. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 4) review round 2: collect sd-cli batch outputs Codex review: when batch_count > 1, stable-diffusion.cpp's save_results() writes the numbered files <stem>_<idx><suffix> (base_0.png, base_1.png, ...) instead of the literal --output path. SdCppEngine.generate checked only the literal path, so a batch generation would exit 0 and then raise 'no image' (or return a stale file). generate now returns the literal path when present and otherwise falls back to the numbered siblings; single-image behavior is unchanged. Test: a fake sd-cli that writes img_0.png/img_1.png (not img.png) is collected without error. * Studio diffusion (Phase 6) review round 2: img2img source dims + upscale repeats Codex review on the native engine arg builder: - build_sd_cpp_command emitted --width/--height unconditionally, so an img2img/inpaint/edit run that left dims unset forced a 1024x1024 resize/crop of the input. width/height are now Optional (None = unset): an image-conditioned run (init_img or ref_images) with unset dims omits the flags so sd.cpp derives the size from the input image (set_width_and_height_if_unset); a plain txt2img run with unset dims keeps the prior 1024x1024 default; explicit dims are always honored. width/height are read only by the builder, so the type change is local. - build_sd_cpp_upscale_command used a truthiness guard (params.repeats and ...) that silently swallowed repeats=0 into sd-cli's default of one pass, turning an explicit no-op into a real upscale. It now rejects repeats < 1 with ValueError and emits the flag for any explicit value != 1. Tests: img2img unset dims omit width/height (init_img and ref_images), explicit dims emitted, txt2img keeps 1024; upscale rejects repeats=0 and omits the flag at the default. (Two pre-existing binary-discovery tests fail only because a real sd-cli is installed in this dev environment; unrelated to this change.) * Studio diffusion (Phase 9) review round 2: correct prequant allowlist doc Codex review: the transformer_prequant_path field description still told operators to enable local checkpoints with UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH=1, but the prior security fix made that variable a directory allowlist -- _allowed_prequant_roots deliberately drops bare on/off toggle tokens (1/true/yes/...). An operator following the documented =1 would have every transformer_prequant_path request silently refused. The description now states it must name one or more allowlisted directories and that a bare on/off value is not accepted. Test: asserts the field help references UNSLOTH_ALLOW_LOCAL_PREQUANT_PATH, does not say =1, and describes an allowlist/directory (guards against doc drift). * Studio diffusion (Phase 10) review round 2: cudnn/flash3 gating + registry reset Codex review on attention-backend selection: - Explicit attention_backend=cudnn skipped the SM80 gate that auto applies, so on pre-Ampere NVIDIA (T4 SM75 / V100 SM70) it set fine then crashed at the first generation with no fallback. select_attention_backend now applies _cudnn_attention_supported() to an explicit cuDNN request too. - flash3 used a minimum-only capability gate (>= SM90), so an explicit flash3 on a Blackwell B200 (SM100) passed and then failed at generation -- FlashAttention 3 is a Hopper-SM90 rewrite with no Blackwell kernel. The arch gate is now a (min, max-exclusive) range: flash3 is SM9x-only, flash4 stays SM100+. - apply_attention_backend's success path left diffusers' process-wide active backend pinned to the kernel it set; a later component whose processors are unconfigured (backend None) would inherit it. It now resets the global registry to native after a successful per-transformer set (the transformer keeps its own backend), best-effort. Also fixed _active_attention_backend: get_active_backend() returns a (name, fn) tuple, so the prior code stringified the tuple and never matched a name, defeating the native-restore short-circuit. Tests: explicit cudnn dropped below SM80; flash3 dropped on SM100 and allowed on SM90; global registry reset after a successful set; _active_attention_backend reads the tuple return. * Studio diffusion (Phase 11) review round 2: keep GH200/B300 on the fp8 ladder Codex review: _DATACENTER_GPU_TOKENS omitted GH200 (Grace-Hopper) and B300 (Blackwell Ultra), though it has the distinct GB200/GB300 superchip tokens. So _is_consumer_gpu returned True for 'NVIDIA GH200 480GB' / 'NVIDIA B300', and the auto ladder moved int8 ahead of fp8 on those data-center parts -- contradicting llama_cpp.py's datacenter regex, which lists both. Added GH200 and B300 so they are treated as data-center class and keep the intended fp8-first behavior. Test: extends the datacenter parametrize with 'NVIDIA B300' and 'NVIDIA GH200 480GB' (now _is_consumer_gpu False). * Studio diffusion (Phase 14) review round 2: apply int8 M=1 exclusion in the builder Codex review: the M=1 modulation/embedder exclusion was wired only into the dense runtime quantiser; the offline builder scripts/build_prequant_checkpoint.py called make_filter_fn(min_features) with no exclusion. So an int8 prequant checkpoint quantised the AdaLN modulation and conditioning-embedder linears, and loading it via transformer_prequant_path (the load path only loads already-quantised tensors, it can't re-skip them) reintroduced the torch._int_mm M=1 crash this phase fixes for the runtime path. Extracted int8_exclude_name_tokens(scheme) as the single source of truth (int8 -> the M=1 exclusion, every other scheme -> none) and use it in both the runtime quantiser and the builder, so a prequant artifact's quantised-layer set always matches the runtime. fp8/fp4/mx artifacts are byte-identical (empty exclusion). Test: int8_exclude_name_tokens returns the exclusion for int8 and () for fp8/nvfp4/mxfp8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion (Phase 16) review round 2: native CPU arbiter, status offload, load race Codex review on the native-engine routing: - The /images/load route took the GPU arbiter (acquire_for(DIFFUSION) -> evict chat) unconditionally after engine selection. A native sd.cpp load on a pure-CPU host never touches the GPU, so that needlessly tore down the resident chat model. The handoff is now gated: diffusers always takes it, a force-native sd.cpp load on a CUDA/XPU/MPS box still takes it, but a native sd.cpp load on a CPU host skips it. - sd_cpp status() hardcoded offload_policy 'none' / cpu_offload False even when _run_load computed real offload flags (balanced/low_vram/cpu_offload off-CPU), so the setting was unverifiable. status now derives them from state.offload_flags (still 'none' on CPU, where the flags are empty). - _run_load committed the new state without cancelling/waiting on a generation that started during the (slow) asset download, so a stale sd-cli run against the OLD model could finish afterward and persist an image from the previous model once the new load reported ready. The commit now signals the in-flight cancel and waits on _generate_lock before swapping _state (taken only at commit, so the download never serialises against generation), mirroring the diffusers load path. Tests: CPU native load skips the arbiter while a GPU native load takes it; status reports offload active when flags are set; _run_load cancels and waits for an in-flight generation before committing. * Studio diffusion (Phase 14) review round 2: align helper name with the stack Rename the int8 exclusion helper to exclude_tokens_for_scheme, matching the identical helper already present higher in the diffusion stack (Phase 16). The helper definition, the runtime quantiser call, and the offline builder are now byte-identical to that version, so the two branches no longer introduce a divergent name for the same single-source-of-truth and the stack merges without a conflict on this fix. No behavior change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio diffusion: persistent sd-server for the native engine (load once, serve many) The native sd.cpp tier ran sd-cli one-shot per image, so begin_load only resolved asset paths and every generation re-spawned sd-cli and reloaded the multi-GB GGUF from disk (a batch of N = N full reloads). This makes it a resident backend backed by stable-diffusion.cpp's persistent sd-server, mirroring the chat backend's llama-server lifecycle: - begin_load spawns sd-server once (the model loads there) and polls /v1/models until ready; unload kills it. - generate submits ONE async /sdcpp/v1/img_gen job for the whole batch (no reload), polls it to completion, and decodes the returned images. Step progress and ETA come from the server's stdout (the job JSON has no per-step field). - The one-shot sd-cli path is kept as an automatic fallback: it is used when sd-server is absent, and also when a present sd-server fails to start, so behavior is never worse than before. The public backend surface is unchanged, so routes/router need no change. New: sd_cpp_server.py (SdCppServer manager: spawn/readiness/job-submit-poll/cancel/stop, process spawned inside the drain thread so PR_SET_PDEATHSIG binds to the interpreter, not a transient thread; empty scratch dir for the server's per-request LoRA/upscaler/embd scans). Extended: sd_cpp_engine.py (find_sd_server_binary), sd_cpp_args.py (build_sd_cpp_server_command + build_img_gen_request), sd_cpp_backend.py (server/one-shot modes, ensure_sd_server_binary upgrades existing sd-cli-only installs), and the prebuilt installer (locate + chmod sd-server, which ships in the same archive as sd-cli). Verified on a B200 (Z-Image-Turbo-GGUF, CUDA sd-server): one model load across multiple generations (server pid stable, a single 'listening on:'), a batch served from one job with distinct per-image seeds, the second generation faster than the first, and unload/reload spawning a fresh process. 105 sd.cpp + 81 diffusion tests pass. Addresses the review of the Phase 16 native-engine PR. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio native diffusion: harden the persistent sd-server path Addresses review findings on the sd-server backend: - Router: treat a runnable sd-server as native availability, so an sd-server-only install (no sd-cli) still routes to the native engine instead of silently falling back to diffusers. - Backend: probe the sd-server binary before the multi-GB asset download, falling back to one-shot sd-cli up front when it cannot run. - Backend: a lazily cached one-shot fallback engine no longer pins the backend to one-shot; only an explicitly injected engine does, so a now-available server can be used on the next load. - Backend: mask explicit seeds to sd.cpp's signed int64 range before submitting a server job (large seeds were rejected/wrapped in server mode only), and split batches above the server's per-job limit into chunks, each with a timeout proportional to its image count. - Backend/server: make server startup cancellable. stop() signals an abort event before taking the lifecycle lock so a blocking readiness wait bails promptly; unload() stops a not-yet-committed pending server. - Backend: status() clears stale loaded state when the resident server has exited, so clients reload instead of hammering a dead process with 500s. - Server: abandon a poll whose best-effort cancel is not honored within a grace window (releasing the generate lock), report a pre-submit stop/cancel as cancellation (409, not 500), and verify JSON responses are the expected type before indexing. - Server: use a bounded deque for the stdout tail buffer. - Add native_mode to DiffusionStatusResponse so the field is not dropped by the response model. Adds regression tests for each behavioral change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * sd-server: harden native lifecycle and GPU install path - Treat a crashed sd-server probe (signal death / non-127 nonzero) as unavailable so a broken prebuilt falls back to diffusers instead of routing to a server that dies on startup. - Drop stale loaded state when a resident server has exited before a generate, returning the recoverable not-loaded path rather than a 500. - Reject incomplete server batches (fewer blobs than requested) like the one-shot path instead of silently dropping images. - Bound the server log tail in place (keep the deque(maxlen)) and bypass HTTP(S) proxies for the loopback client (trust_env=False). - Honor a stop() that arrives after the server is published but before start() takes the lock, so a cancelled load cannot leak a spawned model process. - Map a closed-client RuntimeError during poll to a cancellation when the generation is being cancelled, so unload races surface as 409 not 500. - Stop a timed-out server job (best-effort cancel then teardown) so an abandoned generation cannot keep denoising and block later loads. - Install the accelerator-matched sd-server build (ROCm/Vulkan/CUDA) and probe the resident server before auto-installing sd-cli, so a server-only or GPU host does not fetch the wrong or an unused binary. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |