301 commits

Author	SHA1	Message	Date
Daniel Han	d4fbc81d3a	Restore dropped FP8 weight_scale_inv tensors on load (#6978) ... * Restore dropped FP8 weight_scale_inv tensors on load Some block-scale FP8 checkpoints (for example Qwen3.6-27B-FP8, issue #6200) load with transformers leaving an mlp.gate_proj as a plain bf16 Linear instead of an fp8 module. Its raw quantized values are read into the bf16 weight and the weight_scale_inv is dropped as an unexpected key, so the weight is used un-scaled and the base model is garbage (perplexity around 2 million). After load, for every checkpoint weight_scale_inv whose live weight is not fp8, dequantize the orphaned weight in place using the block scale from the checkpoint index. Modules that were converted correctly keep an fp8 weight and are skipped, so healthy checkpoints and single-file checkpoints are a no-op. Verified on Qwen3.6-27B-FP8: 64 gate_proj scales restored, perplexity 2028902 to 8.9. No-op on Qwen3-8B-FP8 (all scales already live). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden FP8 weight_scale_inv restore from review - Skip restore when the model has no fp8 weights, so an intentionally dequantized load (load_in_16bit) is never re-scaled and corrupted. - Thread revision, subfolder and cache_dir through the index and shard downloads so scales come from the same snapshot as the weights. - Cover unsharded single-file model.safetensors checkpoints (no index). - Handle transposed block-scale layouts and skip on a true grid mismatch instead of applying a wrong scale. - Match text-only VLM loads where the language_model prefix was stripped. - Restore on the FastLanguageModel text path too, not only vision. - Handle a scalar weight_block_size; per-tensor error handling so one bad tensor cannot abort the rest or hide a partial mutation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address second review round on FP8 scale restore - Bound peak memory: dequantize block views in place with the fp32 scale broadcast instead of materializing a full expanded scale and fp32 copy, so a near-VRAM-limit load is not pushed into OOM by the repair. - Restore on the sequence-classification load path too. - Cover more VLM key remappings (language_model.model.* to model.language_model.*) when matching modules. - Skip the restore for variant loads (variant=...) rather than risk applying default-checkpoint scales to variant weights. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align FP8 scale restore revision with the loaded weights and warn on disk-offloaded layers In llama.py the CausalLM/SequenceClassification weight loads resolve model_name on its default branch (revision is not forwarded there), so read the dropped weight_scale_inv tensors from the same default branch instead of the requested revision, avoiding rescaling default-branch weights with scales from another revision. In loader_utils.py a disk-offloaded layer keeps its weight on the meta device until the offload hook materializes it, so the scale cannot be applied in place. Skip such layers explicitly and print a warning rather than silently leaving them unscaled. * Tighten comments in the FP8 scale restore path --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-09 06:44:44 -07:00
Daniel Han	fb5dc91bb4	Studio: remove dead direct_linux_release_plan path (#7030) ... parse_direct_linux_release_bundle and direct_linux_release_plan are no longer reached by any live code path. Fork Linux installs resolve through _fork_manifest_release_plans -> _linux_published_attempts, and the upstream (ggml-org) path uses direct_upstream_release_plan. The dead parser also called _resolve_linux_bundle_profile, which no longer exists, so its CUDA branch would raise NameError if ever executed. Drop both functions and the obsolete TestDirectLinuxNvidiaCpuGate; its live equivalent TestLinuxPublishedAttemptsNvidiaCpuGate already covers the NVIDIA no-silent-CPU behaviour.	2026-07-09 05:09:16 -07:00
Daniel Han	534c877d21	Keep native RoPE scaling when extending context; carry rope_theta for linear (#7028) ... * Keep native RoPE scaling when extending context; carry rope_theta for linear When max_seq_length exceeds a model's native window, the loader overwrote the model's rope_scaling with linear scaling. For models that already ship a scaled RoPE (llama3/yarn/longrope) that is far worse for long context, and on transformers v5 the linear dict omitted rope_theta (v5 keeps it under rope_parameters), so the rotary base fell back to 10000 and broke past ~8K tokens. Keep the native scaling and just widen the window; only synthesize linear for plain-RoPE models, and carry rope_theta so v5 keeps the real base. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only preserve native llama3 when extending context; keep linear fallback otherwise The patched attention constructor (patch_llama_rope_scaling) rebuilds only linear, llama3 and longrope and its longrope branch reads a top-level original_max_position_embeddings, so preserving yarn or a nested-only longrope config would raise during construction on transformers <= 4.47.1. Keep only llama3 native; yarn/longrope/other types fall back to the linear override, still carrying rope_theta. * Correct long-context extension comment to match llama3-only preservation --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-09 04:20:41 -07:00
Daniel Han	cd9d251f15	Fix fast inference crash on compressed-tensors FP8 models (#7025) ... * Fix fast_gemv crash on compressed-tensors FP8 models Loading a compressed-tensors FP8 checkpoint (for example unsloth/Llama-3.2-1B-Instruct-FP8-Block) with fast_inference=False and running a forward crashed with 'Parameter object has no attribute absmax' inside fast_gemv. A compressed-tensors CompressedLinear exposes an already dequantized bf16 weight at forward time while keeping a weight_scale Parameter. The quant state resolution in get_lora_parameters/get_lora_parameters_bias fell back to that weight_scale, so a bf16 weight was routed into the bitsandbytes fast_gemv/fast_dequantize path, which expects a bitsandbytes QuantState with an absmax attribute. Only fall back to weight_scale_inv/weight_scale when the weight is still fp8. A decompressed bf16 weight then resolves to no quant state and flows through the normal bf16 path, which already handles bias and the LoRA backward. Real fp8 and bitsandbytes 4bit weights are unchanged. * Skip the fast_gemv dispatch test before importing unsloth when bitsandbytes is absent	2026-07-09 04:10:59 -07:00
alkinun	216a1fad33	Fix Windows installer torch index override (#6972) ... * Fix Windows installer torch index override * Clear inherited uv index env vars for pinned installs in studio/setup.ps1 (#6898) * Harden setup.ps1 index-var clearing to truly remove vars (#6898) * Apply UV_DEFAULT_INDEX torch index fix to Linux/Mac install.sh (#6898) * Neutralize all uv index env vars for pinned torch installs (#6898) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-09 03:46:47 -07:00
Etherl	5e43c623b9	Fix FastSentenceTransformer Qwen embedding preprocessing (#6939) ... * Fix FastSentenceTransformer Qwen embedding preprocessing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Document Transformer.load embedding modality fix for #6881 * Harden #6881 fix and add forwards/backwards-compatible regression tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fall back to Transformer constructor on legacy sentence-transformers without Hub-capable load * Mirror legacy sentence-transformers fallback in embedding-parity tripwire test * Tighten #6881 comments and docstrings * Skip embedding-parity test on CPU-only runners since FastSentenceTransformer requires CUDA * Honor the transformer module's saved subfolder when loading modules.json records a path for the Transformer module (root for decoder embedders like Qwen3-Embedding, 0_Transformer for the classic layout). Pooling/Normalize already load from their saved path; thread the same path into Transformer.load as subfolder so config and tokenizer resolve like stock ST. stays a no-op, so single-module models are unchanged. * Make embedding-parity test bf16-aware fp16 overflows to NaN on bf16-native embedders such as EmbeddingGemma (Gemma3), producing a false parity failure. Prefer bf16 when the GPU supports it so the tripwire can guard the full documented embedding matrix (Qwen3-Embedding, EmbeddingGemma, BGE-M3, all-MiniLM, GTE-ModernBERT), not just fp16-safe models. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-07-09 01:46:22 -07:00
Daniel Han	8205d4c081	Retry the Studio UI shutdown re-login on transient goto timeout (#7027) ... * Retry the Studio UI shutdown re-login on transient goto timeout The Chat UI Playwright smoke intermittently failed at the pre-shutdown re-login: page.goto('/login') can hit a 60s TimeoutError on a slow runner even while the server is healthy, and the surrounding except only tolerated ERR_ABORTED / interrupted-navigation, so a plain timeout hard-failed the job. Wrap the re-login goto/wait/fill/submit in the same 3-attempt retry the change-password step already uses (recover_or_replace_page between tries, per-attempt fail screenshots, wait_for_health pre-gate). The composer wait stays outside the loop so a retry never re-navigates after login has set tokens (which would redirect to /chat via the guest guard); it remains the authoritative confirmation, so a genuinely broken login still fails. * Catch transient login-request failures and preserve error listeners on recovery Wait on the /api/auth/login POST inside the retry (via click_and_wait_for_response) so a transient 4xx/5xx is retried in-loop instead of surfacing only at the out-of-loop composer wait, matching the change-password step. When recover_or_replace_page swaps in a fresh page, re-attach the pageerror/console listeners so error tracking survives the replacement.	2026-07-09 01:46:14 -07:00
Michael Han	1b825213ea	Stabilize floating monitor drag (#6984) ... * Stabilize floating monitor drag * Restore floating monitor exit animation * Harden Windows Studio smoke checks * Keep API menu badge removed * Apply no-build-tools env overrides in-script The runner does not apply step-level env keys containing parentheses, so ProgramFiles(x86) kept its real value and Find-VsBuildTools still detected VS through vswhere. Set the overrides inside each pwsh step instead; child processes inherit them. The resolver step moves to pwsh because bash cannot export a variable named ProgramFiles(x86). * Reset chat UI session without a second browser context macOS runs Chromium with --single-process, where closing the last context tears down the whole browser, so the shutdown re-login died with TargetClosedError on new_page. Clear cookies and swap pages inside the same context instead, opening the replacement page before closing the old one. * Keep the no-build-tools Path filtered across session refreshes install.ps1's Refresh-SessionPath and setup.ps1's Refresh-Environment rebuild the session Path from the Machine and User registry scopes, so the process-level filter could be undone mid-install and re-expose CMake. Filter those scopes in the Prepare step with normalized dir matching and restore them in cleanup. * Drop stale localStorage auth tokens before re-login Auth tokens live in localStorage, not cookies, and the login guest guard redirects on their mere presence. Remove them during the session reset so the /login navigation is deterministic instead of relying on the tolerated redirect bounce.	2026-07-09 00:16:05 -07:00
ramisworld	81f789ba85	Guard FP8 Triton launches with tensor device context (#6888) ... --------- Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2026-07-08 18:32:51 -03:00
Vineeth Sai	85a068cfe1	Fix to_sharegpt optional block rendering "None" for missing extra columns (#6827) ... --------- Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2026-07-08 17:57:41 -03:00
Vineeth Sai	dc4618ce47	Fix duplicate unsloth/gemma-2b-bnb-4bit mapper key routing the base 4bit repo to the instruct model (#6891)	2026-07-08 17:40:39 -03:00
Vineeth Sai	92c3e48529	Fix BAD_MAPPINGS not redirecting the -unsloth-bnb-4bit dynamic quants (#6949) ... --------- Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2026-07-08 17:37:44 -03:00
Thomas Eric 🇧🇷	03cbe211a3	Studio: fix flash-attn and torchao install on Blackwell (sm_100+) GPUs (Closes #6961) (#6970) ... * fix: Remove moot has_blackwell_gpu() function Fixes unslothai/unsloth#6961. This function skipped flash-attn on Blackwell GPUs because no prebuilt wheel existed; Dao-AILab now ships one and url_exists() already gates resolution. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: use torchao 0.17.0 for Blackwell Fixes #6961. Torchao 0.16.0's cpp extensions are built against CUDA 12, so on a CUDA-13 torch (cu130 / Blackwell) they fail to load with "libcudart.so.12: cannot open shared object file". Select 0.17.0 there instead: its cpp targets torch 2.11, so it is skipped cleanly rather than crashing. CUDA-12 / ROCm / CPU torch 2.10 keeps 0.16.0 and its working kernels. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Condense torchao version-selection comments (no behavior change) * Support torch 2.11 in the Studio installer via the torch2.10 prebuilt wheels Map torch 2.11 to the torch2.10 prebuilt wheels for flash-attn, causal-conv1d, and mamba through wheel_utils.prebuilt_wheel_torch_mm, applied in direct_wheel_url (filename) and flash_attn_wheel_url (version). Those torch2.10 CUDA wheels load and pass each project's own test suite on torch 2.11 (verified on B200), so a torch 2.11 environment gets the prebuilt accelerators instead of skipping or building from source. Raise _CUDA_TORCH_PKG_SPEC to <2.12.0 (torchvision <0.27.0, torchaudio <2.12.0) so the CUDA torch repair path can install torch 2.11, where torchao 0.17's cpp kernels load cleanly. Add tests for the mapping. * Keep has_blackwell_gpu as a False stub for future arch gating * Restore has_blackwell_gpu as a return-False probe kept for future arch gating Keep the nvidia-smi compute_cap detection and its two call sites, but short-circuit with return False at the top so flash-attn is no longer skipped on Blackwell (sm_100+ now has prebuilt wheels and url_exists gates resolution). Drop the early return to re-enable arch-based detection later. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-07-08 06:38:10 -07:00
oobabooga	fcb1152c76	Studio: source CPU llama.cpp prebuilts from unslothai/llama.cpp (#6311) ... * Studio: source CPU llama.cpp prebuilts from the unslothai fork * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: reject unknown Linux CPU arches and keep ROCm-tooling hosts off the CPU prebuilt * Studio: extend the resolve-prebuilt ROCm-tooling guard to Windows * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: let ROCm-SDK-only CPU hosts take the fork CPU prebuilt * Studio: accept windows-arm64 prebuilt kind and refresh stale fork-routing comments * Studio: correct stale fork-routing comments and --resolve-prebuilt help * Refresh stale ggml-org routing comments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-07-08 05:34:59 -07:00
ErenAta16	e86b7874d4	feat: detect installed coding agent CLIs in Studio settings (#6909) ... * feat: detect installed coding agent CLIs in Studio settings The API-keys panel only ever showed the "claude" flavor of the `unsloth start` command, so anyone using Codex, OpenCode, OpenClaw, Hermes, or Pi had to manually rewrite the copied command by hand. Add a backend check that looks for each agent's CLI binary on PATH (shutil.which, mirroring the pattern already used elsewhere in studio/backend/utils) and expose it as GET /api/settings/coding-agents. The API-keys panel now renders a picker for all six supported agents, marks the ones it finds installed, and defaults to one of those instead of always falling back to claude. Includes unit tests for the detection helper. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * address review feedback on coding-agent detection Three fixes from PR review: - detect_installed_coding_agents now treats a PATH lookup failure as "not installed" instead of letting it bubble up and break the settings endpoint; added a regression test for it. - CodingAgentsResponse.agents is now typed as an immutable tuple instead of a list built from one, matching CODING_AGENTS itself. - Fixed a race in the API-keys panel: picking an agent while the installed-CLI check is still in flight could get silently overwritten once that check resolved. A ref now tracks whether the user has made a manual choice, so the auto-detected default only applies before that happens. * Address Codex feedback: GGUF gating and remote-detection scope - codex refuses to launch against a non-GGUF (transformers-backed) model (unsloth_cli's _require_gguf_for_codex), so auto-defaulting to it produced a copy-pasteable command that fails immediately whenever the loaded model isn't GGUF. Add useActiveModelIsGguf() (looks up the active checkpoint in the chat runtime store) and a correction effect that steers the auto-pick away from codex unless the loaded model qualifies, without ever touching a choice the user made by hand. - Detection runs via shutil.which on the Studio backend host, which isn't the same machine as the browser in a tunnel/remote session. Reword the 'installed'/'detected' copy to say so explicitly when the tunnel URL is in use, instead of implying the check ran on the viewer's own device. * Rework auto-default per review: loopback gating + inline GGUF check Replaces the previous approach with the exact shape discussed on the PR: - Export isLoopbackHost/normalizeHost from agent-command.ts. The detection endpoint runs shutil.which on the Studio backend, which only describes the browser's own machine when the base this panel targets resolves to loopback. For a LAN or tunnel/remote base, gate the whole thing off -- don't mark anything as "detected" and don't let it drive the default -- instead of just relabeling the copy. - Drop the separate GGUF-correction effect and useActiveModelIsGguf hook. Read useChatRuntimeStore.getState().activeGgufVariant inline inside the existing detection effect's .then() (so it doesn't need to sit in the effect's deps), and pick the first detected agent that isn't codex unless the loaded model is GGUF, leaving the existing default untouched when no compatible agent is detected. Verified both branches (loopback vs LAN/tunnel base, gguf vs non-gguf, manual pick preserved, no-compatible-agent fallback) with a standalone port of the .then() logic. * Address latest Codex findings: stale detection, model swap, cache - Clear detectedAgents (and skip the network call entirely) when the panel leaves a loopback base, instead of leaving a previous loopback detection result marked 'installed' for a command that now targets a LAN/tunnel/ remote host. - Add a separate, network-free correction effect keyed on the live activeGgufVariant: if codex was auto-picked while a GGUF model was loaded and the user then switches to a transformers-backed model while this panel stays mounted, steer away from codex instead of leaving a command that unsloth_cli's _require_gguf_for_codex will now reject. Never touches a manual pick. - Drop coding-agents.ts's module-lifetime cache. Installed-CLI detection is environment state, not a persisted setting, so a stale positive/negative from before the user installed something (or reopened the tab) is worse than one extra cheap local API call per mount; keep only the in-flight de-dupe for concurrent callers. Verified the correction-effect logic (gguf->non-gguf swap with/without a fallback, still-gguf no-op, manual pick never overridden) with a standalone port of the effect. * Make the codex/GGUF auto-pick symmetric in both directions The correction effect only steered away from codex when the model stopped being GGUF; it never steered back toward codex if the model became GGUF *after* a non-GGUF-gated fallback had already picked something else (e.g. codex is the only detected CLI, a transformers model is loaded so the selection correctly falls back to the claude default, then the user loads a GGUF model while the panel stays mounted -- codex never gets reconsidered). Consolidate into one effect that re-derives the preferred detected agent from scratch whenever detectedAgents or activeGgufVariant changes, in either direction, instead of only reacting to the codex-specific downgrade case. The fetch effect now only populates detectedAgents/availableAgents; this effect is the single source of truth for what gets auto-picked from that list. Never overrides a manual choice. Verified both transition directions plus the manual-pick-survives and initial-detection cases with a standalone port of the derivation logic. * Reset the auto-pick to the default when it stops being trustworthy Two more real gaps from the latest Codex pass on d988f52: - The unified derivation effect only handled the case where a *different* detected agent could take over. If codex was the only detected agent and auto-picked while a GGUF model was loaded, then the model stopped being GGUF, 'preferred' came back undefined and the effect silently left the selection on codex -- exactly the command unsloth_cli's _require_gguf_for_codex now rejects. Fall back to DEFAULT_AGENT in that case instead of leaving it untouched. - Leaving a loopback base cleared detectedAgents (so the 'installed' badges correctly disappear) but left whatever agent had been auto-picked from that now-stale, server-side-only detection still selected. Reset to DEFAULT_AGENT there too, unless the user picked by hand. Introduces a shared DEFAULT_AGENT constant instead of repeating the "claude" literal at each reset site. Verified all five cases (both new resets, both manual-pick-survives variants, and the existing multi-detected-agent fallback still preferring another compatible agent over resetting) with a standalone port of the effects. * Derive GGUF-ness from the actual loaded state, not just the variant string activeGgufVariant only covers an HF-repo GGUF pick (a specific quant variant string). A direct local .gguf file -- custom folder, LM Studio, or drag-drop -- is just as much a GGUF the codex preflight (unsloth_cli's _require_gguf_for_codex) would accept, but it never has a "variant" to report, so it read as non-GGUF here even though /api/inference/status correctly reports is_gguf: true for it. That mismatch could leave a Codex-only install not auto-selected, or reset an auto-picked Codex, for a model that actually supports it. Combined activeGgufVariant with activeNativePathToken (covers the drag-drop/picked-file case) and ggufContextLength (only ever populated when the backend last reported is_gguf: true for the active model, see applyActiveModelStatusToStore) so all three paths a model can be GGUF through are covered, matching the same is_gguf-or-equivalent check hasGgufSource already applies to a staged pick elsewhere in this codebase. * Clear stale native-path token on a non-GGUF status refresh When a native (drag-dropped or picked) GGUF was loaded and the backend later switches to a transformers model outside the UI load path, refresh() adopts the new /api/inference/status via setCheckpoint and applyActiveModelStatusToStore. Those reset activeGgufVariant and ggufContextLength but never clear activeNativePathToken, so the isGguf OR stays true after the switch and a Codex-only detection auto-selects unsloth start codex for a non-GGUF model its preflight rejects. Drop activeNativePathToken in applyActiveModelStatusToStore whenever the status is non-GGUF. A real GGUF load reports is_gguf: true, so its token is preserved (the load path owns it); only a non-GGUF status clears it. * Add the AGPL-3.0 header to the new studio contract test * Fix/adjust agent detection for PR #6909 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com> Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com>	2026-07-08 05:26:50 -07:00
Daniel Han	0e1ed88bb8	version-compat CI: fake CPU training runs for SFT/GRPO/DPO (#6965) ... * version-compat CI: fake CPU training runs for SFT/GRPO/DPO Adds a runtime layer on top of the patch-run canary: actually runs trainer.train() for a couple of steps on a CPU-only runner under the CUDA spoof, wrapping a plain tiny HF model in the Unsloth-patched trainer. Exercises the real train() loop (collation, generation, the injected _get_per_token_logps_and_entropies, loss, backward, optimizer) so a TRL or transformers change that breaks the loop at runtime -- not just the source structure -- surfaces here. No GPU, no meaningful numerics. Needs a chain of small CPU shims (eager torch.compile, dynamo suppress, cuda tensor-alloc redirect to CPU, model.for_training/for_inference equivalents) documented inline. Does not exercise Unsloth's Triton/GPU kernels (CPU can't). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * cpu fake-train: force adamw_torch + disable dynamo for CPU runner On a real CPU-build torch runner (GitHub CI) two things bit that a CUDA-build torch with GPUs hidden masked locally: - The default optimizer is adamw_8bit (bitsandbytes), whose is_on_gpu() check dies on CPU tensors. Force optim=adamw_torch in all three configs. - import unsloth reinstalls the real torch.compile over the eager passthrough, so the GRPO hot path (chunked_selective_log_softmax) actually compiles and inductor picks the spoofed CUDA device, crashing on device props (gcnArchName). Re-apply the eager passthrough after import and flip torch._dynamo.config.disable so every @torch.compile runs eager at call time. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * cpu fake-train: write checkpoints under pytest tmp_path Use pytest's tmp_path for each trainer's output_dir instead of a hardcoded relative temp/ci_* path, so a local pytest run does not leave untracked dirs in the repo tree and the tests are CWD-independent. * version-compat CI: disable dynamo at process level for the fake-run job Set TORCHDYNAMO_DISABLE / TORCH_COMPILE_DISABLE in the fake-run step env so dynamo/inductor is off before conftest.py's early import unsloth, not only via the per-test runtime shim. Defense in depth on the GPU-less runner: the GRPO hot path never compiles regardless of when its functions were decorated. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-08 04:06:28 -07:00
marcandrelarochelle	07c8bbbf5a	(GRPO) Fix PEFT replacement for TRL >= 1.7.0, add missing compute_aux_loss for TRL >= 1.7.0 (#6904) ... * Fix PEFT replacement for TRL >= 1.7.0, add missing compute_aux_loss for TRL 1.7.0 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix GRPO for TRL >= 1.7.0: PEFT ref-adapter removal and return arity rl.py: for trl >= 1.7.0, scope the PEFT removal regex to the ref-adapter block only by anchoring the end on ref_param.data.copy_(param.data), so it no longer also deletes the following gradient-checkpointing enable_input_require_grads() block. Neutralize TRL 1.7.0's `if _is_quantized_model:` bf16 cast the same way the existing is_loaded_in_4bit cast is handled. rl_replacements.py: initialize _extra_moe_kwargs before use (it was referenced before assignment whenever compute_aux_loss was passed) and only request output_router_logits when the aux loss is actually wanted. rl_replacements.py: _get_per_token_logps_and_entropies now returns a 3-tuple (logps, entropies, aux_loss) for trl >= 1.7.0 and a 2-tuple for older TRL, matching how every TRL call site unpacks the result. Without this, TRL 1.7.x _generate_and_score_completions unpacks 3 values from a 2-tuple and raises "not enough values to unpack (expected 3, got 2)". * Return zero aux_loss placeholder and drop inference-mode aux collection * GRPO TRL >= 1.7.0: reject router aux-loss opt-in at init; drop zero aux placeholder Unsloth's optimized GRPO forward cannot compute the MoE router auxiliary loss. Previously an explicit opt-in (router_aux_loss_coef > 0) returned a fabricated zero, silently training without the requested load-balancing penalty. Now reject it at trainer init with a clear NotImplementedError, and return None (not zero) for the aux slot of TRL's 3-tuple. Default stays off (coef 0), so the common path is unaffected. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GRPO hidden-states fallback: free ModelOutput before chunked log-softmax The old/ref logprob fallback binds the full ModelOutput (which holds every layer's hidden_states when output_hidden_states=True) and kept it alive across chunked_hidden_states_selective_log_softmax, an avoidable OOM on large models. Extract logits then del outputs in both the text and VLM branches. * Version-compat CI: proactively catch TRL GRPO breakage The existing TRL canary is a static symbol/source grep: it verifies symbols exist but is blind to structural changes (TRL 1.7.0's 2->3-tuple per-token-logps return arity and restructured PEFT ref-adapter block, which the fix in this PR addresses, both slipped past it because the methods still existed). Two additions: - test_trl_grpo_pinned_symbols.py: extend TRL_TAGS to 1.5/1.6/1.7 and pin the exact source-string contracts the rl.py / rl_replacements.py transforms depend on for TRL >= 1.7.0 (PEFT elif ref-adapter block + enable_input_require_grads survival, if _is_quantized_model, aux_loss_enabled anchor, compute_aux_loss arity). A future TRL change fails on main a few days before the PyPI release. - test_trl_grpo_fake_run.py + a version-compat-ci job: fake-CUDA run that drives the real GRPO/SFT/DPO source-transform patchers against latest + main TRL on a CPU-only runner (no training) and asserts the generated Unsloth trainer still satisfies the transform contracts. Catches behavioral regressions the grep cannot see. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fake-run test: use a normal Version import for the aux gate * version-compat CI: fix fake-run job gate + torch-absent collection - Drop the invalid job-level matrix if (matrix is not available in jobs.<id>.if -> 'Unrecognized named-value: matrix' fails the whole workflow). Use a single job that runs vs TRL latest always and re-runs vs TRL main only on schedule/dispatch via a step-level github.event_name guard. Validated with actionlint. - Module-level skip the fake-run test when torch is absent so daily-fresh-fetch (pytest-only, collects tests/version_compat/) does not crash on the top-level spoof import. * fake-run test: do not skip on import failure unsloth/trl are installed in the grpo-fake-run job, so a failing import is the import-time drift this canary must catch. Keep only the not-installed find_spec skips; let a real import error fail the test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GRPO arity gate: regex downgrade + fail loud + CI coverage The TRL < 1.7.0 per-token-logps return downgrade was an exact-string replace anchored on the full return line incl. its comment, so a reformat (e.g. pre-commit) could silently no-op it and ship a 3-tuple to older TRL. Switch to a regex tolerant of comment/whitespace drift, and raise if the anchor stops matching (re.subn count != 1) instead of failing silently. Add a monkeypatched trl_version unit test asserting both arities, since CI only installs TRL >= 1.7.0 and never exercised the downgrade otherwise. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fake-run: give SFT/DPO a real contract, not just ast-parse The SFT/DPO fake patch runs only checked the generated trainer parses. Also assert the shared QLoRA _is_quantized_model bf16 cast is neutralized (TRL 1.7's spelling, present in both sft_trainer and dpo_trainer), so a structural TRL change to that block is caught for SFT/DPO too, not just GRPO. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GRPO PEFT ref-adapter removal: lower gate to the TRL 1.4.0 floor The elif is_peft_model(model) and args.beta != 0.0: ref-adapter block was introduced in TRL 1.4.0 and is unchanged through 1.7.x, but the removal was gated at >= 1.7.0, so for 1.4 <= TRL < 1.7 the transform fell through to the 0.27 branch (which matches the older if is_peft_available()... form) and silently no-oped: a PEFT + beta != 0 GRPO run then computed the KL reference from the copied ref adapter instead of the base model. Lower the gate to 1.4.0 and keep the 1.7.0-only router aux-loss fail-fast nested. Widen the pinned-symbol contract test to run from 1.4.0 so the covered versions are actually exercised. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-07-08 04:05:03 -07:00
Long Yixing	934f879043	feat(mlx): route trainer callbacks (#6929)	2026-07-08 03:25:50 -07:00
Michael Han	7f9964f21e	Move New badge to System settings tab (#6963) ... * Move New badge to System settings tab Show the "New" badge on the System tab and drop it from Connections. * Stabilize refresh revocation UI test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-08 01:51:34 -07:00
Etherll	ba450b437e	Studio: add assistant response details panel (#6842) ... * Studio: add assistant response details panel * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide model badge by default, show on hover/focus Wrap MessageResponseModelBadge in a span with hidden/group-hover visibility classes to reduce visual clutter. The badge now only displays when hovering or focusing on the assistant message, improving the UI presentation. Updated corresponding tests to verify the new CSS classes. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-07 06:26:00 -07:00
Daniel Han	d79495dc96	Add RDNA 2/3/4 ROCm routing tests via a CPU-only torch spoof (#6935) ... * Add RDNA 2/3/4 ROCm routing tests via a CPU-only torch spoof Introduces tests/_zoo_rocm_spoof.py, the ROCm sibling of _zoo_aggressive_cuda_spoof.py: it reuses the CUDA spoof's torch.cuda no-op machinery and overlays an AMD Radeon identity (torch.version.hip, gcnArchName, capability) for any RDNA 2/3/4 gfx target, so hip code paths run on CPU-only CI with no AMD hardware. tests/studio/install/test_rocm_rdna_routing.py then asserts unsloth_zoo routes every RDNA arch (gfx1030/1031/1032/1034, gfx1100/1101/1102, gfx1150/1151, gfx1200/1201) correctly: device_type resolves to hip, llama.cpp target resolves to (rocm, gfx), and the per-family ROCm bundle suffix (gfx103X/gfx110X/gfx120X, or self for gfx1150/1151) is picked. The torch-facing checks run in a subprocess so the spoof never leaks into sibling tests and DEVICE_TYPE (cached at import) resolves from a clean process; the pure gfx-family mapping runs in-process. Guarded by importorskip so it runs where torch and unsloth_zoo are installed (the Repo tests CPU job) and skips elsewhere. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-07 04:41:37 -07:00
Daniel Han	bdb958e052	Guard RoPE scaling against the transformers v5 buffer blank; honor extended RoPE factor (#6925) ... * Guard RoPE scaling against the transformers v5 buffer blank; honor extended factor Add a family-agnostic guard that builds each rotary from a scaled config, blanks its non-persistent buffers (what transformers v5 does on load), runs loader._fix_rope_inv_freq, and asserts every buffer is restored to its scaled value (llama3 and longrope). This catches the whole bug class, not just the one call site, and is validated to fail on the pre-fix repair. Also make LlamaExtendedRotaryEmbedding read the llama3 factor from the config instead of hardcoding 8 (wrong for Llama-3.2, factor 32), falling back to the Llama-3.1 defaults when built without a config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pass config into extended rotary codegen; skip v5 round-trip on transformers 4.x - patch_llama_rope_scaling now builds the llama3 extended rotary with config=self.config so it reads the real factor (32 for Llama-3.2) instead of falling back to 8; the template already references self.config. - test_v5_blank_repair_roundtrip now skips when loader._NEEDS_ROPE_FIX is False, since _fix_rope_inv_freq is a no-op on transformers 4.x and cannot restore the blanked buffers there. * Raise stream deadlock-guard timeouts from 0.2s to 5.0s in passthrough tests These asyncio.wait_for guards bound test setup and cross-task event signaling that complete near-instantly on success; the 0.2s budget is a latency assertion in disguise and times out under CI scheduling load (seen on the 3.11 matrix leg while 3.10/3.12/3.13 pass the same commit). 5.0s matches the timeout used elsewhere in the suite and still fails fast on a real hang. No test relies on the guard expiring. * Extended rotary reads rope_parameters as well as rope_scaling transformers v5 stores llama3 scaling under config.rope_parameters and exposes rope_scaling only as a back-compat property. Reading that property works on 5.0-5.13 (verified: factor resolves to 32 for Llama-3.2), but a future release may drop the shim, after which the subclass path would fall back to factor 8. Read either field so the factor survives the rename. Adds test_extended_rotary_reads_rope_parameters_v5 (fails on the old single-field read: rope_parameters-only config resolves to 8, not 32). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-07 04:16:57 -07:00
Leo Borcherding	296cacb5a1	ROCm-on-WSL: support discrete Radeon (RDNA 3/4) in WSL, not just Strix Halo (#6915) ... * WSL ROCm: generalize ROCm-on-WSL bootstrap from Strix-only to any RDNA arch install_rocm_wsl_strixhalo.sh hardcoded gfx1151, so its verify step died on discrete Radeon cards even though the ROCm + librocdxg setup is arch-agnostic. Auto-detect the GPU arch from rocminfo (override via UNSLOTH_WSL_GFX), verify any GPU agent enumerates over DXG, and map the arch to AMD's per-arch wheel family for the optional smoke test (injecting librocdxg into torch/lib so torch's bundled ROCr finds the DXG bridge). Verified on gfx1200 (Radeon RX 9060 XT) in WSL2 + Ubuntu 24.04 -- torch.cuda now enumerates the GPU. * WSL ROCm: trigger the ROCm-on-WSL bootstrap for discrete Radeon GPUs too _maybe_bootstrap_rocm_wsl only fired for Strix APUs (matched via /proc/cpuinfo, which discrete cards don't appear in). Add _wsl_amd_gpu_name() -- queries the Windows host via WMI -- and broaden the trigger gate plus the 'already-usable ROCm' rocminfo check from gfx1151-only to any real GPU agent (gfxNNNN, excluding the gfx11-generic fallback ISA). The generalized bootstrap then auto-detects the arch. Enables 'curl install.sh | sh' to set up ROCm-on-WSL on discrete Radeon RX 7000/9000 in WSL2 + Ubuntu 24.04, not just Strix Halo/Point. * WSL ROCm: address review -- filter generic ISA in bootstrap, bound the host GPU query - install_rocm_wsl_strixhalo.sh: exclude the gfx11-generic fallback ISA in arch detection (grep -v generic), matching install.sh's rocminfo check, so a generic agent listed before the real one can't be picked as the arch. - install.sh: wrap the powershell.exe Win32_VideoController query in _run_bounded (10s timeout) so an unstable WSL-interop / busy host can't hang the installer. * WSL ROCm: harden arch-detect + librocdxg copy under set -eo pipefail (review) - _detected_gfx: append '|| true' so a no-GPU rocminfo (empty pipeline, non-zero under pipefail) doesn't abort the assignment before the '[ -z ]' branch prints the diagnostic + die message. - smoke-test librocdxg copy: gate on '[ -d "$_tlib" ]' instead of '[ -n ]' so a non-directory value can't make cp rename librocdxg to 'lib'. * WSL ROCm: address Codex review (gfx000, 24.04 reroute for discrete, test locator) - Exclude gfx000 (the CPU agent) from the WSL 'usable ROCm' check and the bootstrap arch-detect: match gfx[1-9] (nonzero arch), so a partial ROCm install that only reports the CPU ISA no longer short-circuits the librocdxg setup. (P2) - Reuse the Ubuntu-24.04 reroute for discrete Radeon: broaden _maybe_reroute_strixhalo_to_2404's gate with the same _wsl_amd_gpu_name (WMI) fallback, so a discrete card on 26.04 reroutes to a 24.04 distro like Strix does instead of falling to CPU. Moved _wsl_amd_gpu_name above the reroute and made it self-contained + 10s-bounded (it runs before _run_bounded is defined). (P2) - Update TestInstallShDropinPersistence to locate the gate by its unique '!/generic/' clause now that the gfx1151 literal is gone. (P1) * Condense ROCm-on-WSL comments in install.sh and bootstrap helper * Guard WSL reroute from NVIDIA hybrid hosts and fix GFX-override pipefail check * Honor CUDA_VISIBLE_DEVICES-hidden NVIDIA in the WSL reroute guard * Reuse _has_usable_nvidia_gpu in the WSL reroute guard --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-07-07 02:29:37 -07:00
Michael Han	af93868760	Fix repeated base model downloads across checkpoint exports (#6896) ... * Fix repeated base model downloads across checkpoint exports (#6890) Pre-warm the HF hub cache with the 16bit base weights before merge_and_overwrite_lora runs. The merge fetches shards with hf_hub_download(local_dir=...), which never populates the hub cache, so temporary merge directories (GGUF checkpoint exports) forced a full re-download of the base model for every checkpoint. The first export now downloads once into the cache and later exports copy from it. Skips itself when already cached, offline, on Kaggle/Colab, for local or nf4/fp4 bases, non-downloading save methods, or low disk. Opt out with UNSLOTH_PREWARM_HUB_CACHE=0. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Show MB for small base models in the pre-warm download message * Harden pre-warm: getattr for model config, abspath for relative HF_HUB_CACHE - Read config._name_or_path via getattr so a model without a config skips cleanly instead of taking the outer error path. - abspath the cache probe so a relative HF_HUB_CACHE walks up to a real root rather than "", which would zero the free-space check and skip pre-warm. Both from PR review; each covered by a test that fails without the fix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pre-warm the live-env HF cache so runtime redirects still hit (#6890) Resolve the hub cache the same way the merge does (unsloth_zoo _active_caches, live env) instead of huggingface_hub's import-time-frozen constants.HF_HUB_CACHE, and pass it as cache_dir to the cached probe, disk check and snapshot_download. Without this, a runtime HF_HOME/HF_HUB_CACHE redirect (unsloth_zoo redirect_hf_cache_if_readonly on a read-only default cache, or Studio) makes the pre-warm populate a different directory than the one the merge reads, so the cache-copy fast path misses and the base re-downloads on every export anyway. Adds 3 regression tests covering the cache_dir threading and the redirect case. * Apply ruff-format kwarg spacing to the pre-warm cache-dir changes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pre-warm the 16bit sibling for FP8 bases so their merged_16bit exports reuse the cache too For a merged_16bit export of an FP8 base with an existing 16bit sibling, the merge swaps to the sibling and downloads that (unsloth_zoo _resolve_fp8_16bit_sibling), so pre-warming the FP8 repo missed the cache and re-downloaded the sibling every export. Mirror the swap and pre-warm the sibling. No sibling still caches the FP8 repo for the in-place dequant path. Adds 2 regression tests. * Filter pre-warm shards through the safetensors index like the merge does Repos that ship a leftover shard set the index does not reference (e.g. granite-3.2) made the disk gate over-count and snapshot_download fetch shards the merge never reads. Mirror the merge: on the download path, keep only index-referenced shards. Runs after the already-cached check so the cached fast path stays network-free. Adds 2 tests. * Tighten pre-warm comments --------- Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>	2026-07-06 23:38:29 -07:00
Daniel Han	2fada48ef5	Fix llama3 RoPE scaling dropped on transformers v5 (#6907) ... * Fix llama3 RoPE scaling dropped on transformers v5 transformers v5 loads on meta then blanks non-persistent buffers, so _fix_rope_inv_freq rebuilds inv_freq after load. It recomputed a vanilla inv_freq and applied _apply_inv_freq_scaling, a no-op on the base LlamaRotaryEmbedding used by the config/llama3 path, so inv_freq ended up divided by 1 instead of the config factor (8 for Llama 3.1, 32 for Llama 3.2). This corrupts long-range positions and inflates long-context loss about 3-5x. transformers 4.x was unaffected. Route __init__ and the v5 repair through one _unsloth_recompute_inv_freq so they cannot diverge, and stash the config on the rotary module so the repair can rebuild the same scaled value. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add test for llama3 RoPE scaling under the transformers v5 repair * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update RoPE drift guard for the recompute refactor and guard the v5 repair The drift guard's AST tripwire asserted the config-scaling call lived in the if config is not None branch of LlamaRotaryEmbedding.__init__. The fix moved that into _unsloth_recompute_inv_freq, so follow it there (with a fallback to the old inline branch) and add a guard that loader._fix_rope_inv_freq rebuilds inv_freq through the same helper. Also add a CPU functional check of the helper and drop the redundant standalone test. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 09:13:14 -07:00
Anas Khan	cc99aab607	fix: correct class name in SyntheticDataKit.chunk_data guard message (#6901)	2026-07-06 07:11:57 -07:00
Anas Khan	c44d94f1ae	fix: map None quant method to q8_0 before lowercasing in GGUF export (#6889)	2026-07-06 07:11:49 -07:00
Daniel Han	c520662c12	Honor an explicit sdpa or flex_attention request when flash is disabled (#6847) ... * Honor an explicit sdpa or flex_attention request when flash is disabled When flash attention is disabled for a model, the fallback selection could downgrade a caller who explicitly passed attn_implementation='sdpa' or 'flex_attention' to a different backend, because the disable reason is flash-specific. Keep an explicit non-flash request as-is; flash requests still fall back as before. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten comments * Gate honor-explicit attention on provenance and flex support Only honor an explicit non-flash attention request when it comes from the caller argument, not from a config value the loaders synthesize (the language path seeds attn_implementation=sdpa). Honor explicit flex_attention only when supports_flex_attention is True so excluded/broken configs (e.g. gpt_oss) fall back instead of selecting a known-broken backend. Explicit sdpa stays honored. * Honor explicit sdpa through the resolver guard * Keep SDPA exclusions when honoring an explicit sdpa request An explicit attn_implementation="sdpa" was re-enabling sdpa for models in _SDPA_EXCLUDED_MODELS (e.g. gpt_oss) where sdpa is known-broken: the helper honored the request and the resolver's final not-supports_sdpa guard skipped the eager downgrade for any explicit request. Honor an explicit sdpa only when the model is not sdpa-excluded, mirroring the flex guard that already falls back for _FLEX_EXCLUDED_MODELS via supports_flex_attention. Conservative supports_sdpa=False (large head dim / attention-sink models) still honors an explicit sdpa; a synthesized/default sdpa still downgrades to eager. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Honor DISABLE_SDPA_MODEL_NAMES when honoring explicit sdpa The honor-explicit-sdpa guard only skipped the sdpa->eager downgrade for models in _SDPA_EXCLUDED_MODELS (gpt_oss). Gemma3/Gemma3Text disable SDPA through the loader's DISABLE_SDPA_MODEL_NAMES (their bundled SDPA modules are wrong), so an explicit sdpa request bypassed the downgrade and re-enabled a known-wrong path. Extend _is_sdpa_excluded to also treat DISABLE_SDPA_MODEL_NAMES membership as excluded, replicating the loader's trailing-comma substring match so gemma3 and gemma3_text match but gemma3n does not. Move the constant into _utils.py (single source of truth, re-exported from loader.py) to avoid a loader -> _utils cycle. Conservative supports_sdpa=False models not in either list still honor explicit sdpa. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 05:45:45 -07:00
Daniel Han	7cc1752a64	Scope MoE expert LoRA detection to actual MLP projection targets (#6849) ... * Scope MoE expert LoRA detection to actual MLP projection targets _moe_target_set_from_string treated any regex containing the substring mlp or ffn as targeting the expert MLP projections. Unsloth's auto-generated attention-only regex lists mlp, ffn and feed_forward as allowed intermediate path segments while its final group matches only q_proj/k_proj/v_proj/o_proj, so attention-only finetuning on MoE models silently enabled expert LoRA as well: the experts were trained and every MoE layer paid the extra expert LoRA grouped matmuls. Detect expert intent from the projection names themselves (gate_proj/up_proj/down_proj/gate_up_proj) instead of the mlp substring. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten comments * Detect MoE expert LoRA via mlp path segment, not proj names The auto-generated target regex always lists every projection leaf (q/k/v/o and gate/up/down), so keying detection on a proj name mis-fired: it enabled expert LoRA for attention-only regexes and dropped the mlp/ffn path regexes. Key on the mlp/ffn/feed_forward/experts path segment instead, which is present only when the MLP/experts are actually targeted. Add a regression test for the attention-only case. * Scope expert LoRA targets to the leaves a regex names An mlp path alternative with attention-only leaves, for example (mlp|self_attn).(q_proj|o_proj), no longer enables expert LoRA, and a regex naming a single expert leaf such as .*experts.*down_proj now targets only that projection instead of the whole broad set. Generic mlp projections (.*mlp.*proj) and the auto regex mlp tag block keep the broad set for fused-expert models whose leaves are plain Parameters. * Route explicit leaf list into MoE expert detection An attention-only explicit target_modules list routed through get_peft_regex for family scoping (e.g. FastVisionModel with vision layers off) yields a regex carrying the full mlp|feed_forward|ffn|dense component block even though its leaf group only names q/k/v/o_proj. Keying expert detection on that regex trained the experts for a language-only/attention-only request. Use the caller's original leaf list for detection; only the auto path uses the regex, where the mlp block is the sole MLP-intent signal on fused-expert models. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Respect finetune_mlp_modules and finetune_language_layers scope for MoE expert detection When an explicit leaf list that names MLP projections (gate_proj/up_proj/down_proj) is routed through get_peft_regex under finetune_mlp_modules=False, the scoped regex correctly drops the MLP leaves, but MoE expert detection was still keyed on the original list and re-added mlp.experts.* via target_parameters, training the experts the caller had frozen. Same gap for finetune_language_layers=False on vision-only runs. Prefer the original list only when MLP and language families are both in scope (preserving the attention-only fix); otherwise honor the scoped result so the frozen family is respected. Factored the choice into _select_moe_detection_targets with unit tests over the full selection matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 05:45:06 -07:00
Daniel Han	22bd86ecb7	Handle odd shapes and non-float scales in FP8BlockQuantLinear (#6848) ... * Handle odd shapes and non-float scales in FP8BlockQuantLinear Small fp8 checkpoints (e.g. tiny test models) break the block-quantized linear in three ways: weight scales stored in a float8 dtype such as float8_e8m0fnu have no triton dtype mapping; activations whose hidden dim is not a multiple of the activation quant block fail act_quant's divisibility assert; and weights whose dims are not multiples of the weight block cannot be tiled by the triton dequant kernel. Cast non-float scales to float32 on entry, and when the hidden dim does not divide into the activation block, dequantize the weight and run a plain matmul instead of the fp8 block matmul. The dequant goes through a new shape-safe helper that falls back to a torch-native scale expansion when the weight does not tile evenly; backward uses the same helper so the gradient path works for every shape the forward accepts. Full-size checkpoints are unaffected. * Add tiny / e8m0 fp8 block-quant regression test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix FP8 block-quant fallback: real block size in dequant and scalar-scale fast path * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route rectangular fp8 blocks through torch dequant and keep block_size across e8m0 upcast The triton weight_dequant kernel uses one BLOCK_SIZE for both axes, so rectangular blocks (block_size[0] != block_size[1]) mis-index the column scale and corrupt grad_X. Route those through the torch scale expansion, which handles each dimension independently, and keep the triton path for square blocks only. Also preserve a block_size attribute carried on the scale tensor across the e8m0 -> float32 upcast so the later lookup no longer falls back to [128, 128]. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 05:44:55 -07:00
Daniel Han	cb6737cbb8	Auto Xet to HTTP download fallback in from_pretrained; share Studio's fallback via unsloth_zoo (#6638)	2026-07-06 05:13:25 -07:00
Daniel Han	64f6526160	Fix export-time trust_remote_code bypass in FP8/INT8/GGUF-LoRA export (#6869) ... * Fix export-time trust_remote_code bypass in FP8/INT8/GGUF-LoRA export The torchao, compressed-tensors, and LoRA GGUF export paths re-read the merged checkpoint and used to set trust_remote_code from the checkpoint config's static auto_map (the torchao path also scanned the staged tokenizer/processor configs). A model that loads with built-in Transformers classes can carry an auto_map entry, which skips the load-time remote-code consent scan (that only runs when the load already requested trust_remote_code) yet flips trust_remote_code on at export, running unvetted custom code. Derive the reload trust_remote_code from the approved load decision instead: a new _loaded_via_remote_code() checks whether the in-memory model / tokenizer was itself loaded from custom code (its class lives in the transformers_modules package), walking PEFT / wrapper layers. Built-in-loaded models no longer gain trust from config metadata; genuine custom-code models (loaded with consent) still reload correctly. Add CPU-only regression tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden _loaded_via_remote_code against a None/missing __module__ Read type(node).__module__ via getattr and require a string before startswith, so a dynamically created or C-extension class with a None module does not raise during export. Add a regression test. * Split model and tokenizer trust for the compressed subprocess, walk processor components The compressed-tensors export collapsed model and tokenizer trust into one --trust-remote-code flag, so an approved custom tokenizer would have let an unapproved model's custom code run inside the quantization subprocess. The subprocess now takes --trust-remote-code-tokenizer for the processor load and keeps --trust-remote-code for the model loads, matching the torchao path's separate model_trust / tok_trust. _loaded_via_remote_code now also walks processor components (tokenizer, image_processor, feature_extractor, video_processor), so an approved custom tokenizer held inside a built-in ProcessorMixin keeps its trust on the export reload instead of failing with trust_remote_code=False. The walk is a bounded BFS with a seen set so wrapper cycles terminate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-05 05:16:39 -07:00
Ayushman	c356427f30	Guard Windows ROCm torchao override skip (#6837) ... * Fix: skip fp16/bf16 validation for full finetuning in RL trainers When doing full finetuning (FFT) of a bfloat16 model, the fp16/bf16 mismatch validation fires before the corrective logic runs, causing a misleading error even though the code would properly handle it downstream. Skip the validation when full_finetuning is active. Fixes #6731 * Fix: auto-correct fp16/bf16 mismatches for full finetuning before validation Instead of entirely skipping validation (which could let mismatches through when mixed_precision_dtype is float32), auto-correct explicit fp16/bf16 settings that conflict with the model's dtype for FFT. This way the existing validation still catches real mismatches for non-FFT cases, and the corrective logic below handles the normalized settings. Fixes the issue raised in Codex review of PR #6813. * Guard Windows ROCm torchao override skip Detect installed ROCm torch directly before applying the torchao override so Windows ROCm environments never install the crashing torchao package even if the earlier ROCm-installed flag is missing. * Update unsloth/models/rl.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update studio/install_python_stack.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Harden ROCm probe and sync RL precision flags Tolerate stray stdout noise when probing Windows ROCm torch installs by checking the last non-empty output line, matching the existing torch version probe behavior. Also keep args.fp16 and args.bf16 synchronized with the full-finetuning precision auto-corrections in the RL trainer patch so downstream eval settings see a consistent TrainingArguments state. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add MLX trainer compatibility shims Patch imported MLXTrainer and MLXTrainingConfig objects to preserve the expected dataclass field ordering and to provide a _train_dataset_for_batches fallback when older trainers or test doubles only expose train_dataset. Also add focused worker tests covering both compatibility paths. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope PR to Windows ROCm torchao guard * Restore PR scope to Windows ROCm guard * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * test: cover Windows ROCm torchao skip behavior * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Ayushman Paul <ayushman@HP> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com>	2026-07-03 19:24:29 +01:00
oobabooga	2b06616a7e	Fix TrainingArguments silently disabling unsloth gradient checkpointing (#6829) ... * Fix TrainingArguments silently disabling unsloth gradient checkpointing * Cover loaded adapters and preserve explicit None in GC restore * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-03 16:35:02 +01:00
Daniel Han	026141a4a4	Studio: multi-select export formats, portable FP8/INT8, GGUF LoRA, and source parity (#6767) ... * Studio: expose full compressed-tensors scheme set in an export formats dropdown * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: multi-select export formats, portable torchao FP8/INT8, GGUF LoRA, source parity Export page overhaul on top of the formats dropdown: - Unify merged precision into one sorted multi-select list (16-bit first, then 8-bit, then 4-bit). Drop "vLLM" from labels, add INT8 (W8A8), INT8 (W8A16), INT4 (W4A16), MXFP4, MXFP8. Quick formats render as toggle pills; the rest live in a multi-select "More formats" dropdown, so several formats export in one run. - Add a portable torchao FP8/INT8 save path (Float8WeightOnlyConfig / Int8WeightOnlyConfig) that needs no NVIDIA GPU to produce and loads in vLLM. FP8 serializes to safetensors, INT8 to .bin. Wired into save_pretrained_merged and push_to_hub_merged via a TORCHAO_EXPORT_SCHEMES registry and _unsloth_save_torchao, parallel to the compressed-tensors path. - Hide NVIDIA-only compressed-tensors formats when no NVIDIA GPU is present; keep 16-bit and portable FP8/INT8. The backend also rejects a compressed request on non-NVIDIA hardware so it stays authoritative. - Relax merged export to non-PEFT models so Local Model and Hugging Face sources get the same 16-bit / compressed / portable options. - GGUF: send the whole quant list in one call (merge once, quantize many). - LoRA: add a GGUF adapter option (convert_lora_to_gguf.py) with an outtype select (f16/bf16/f32/q8_0/auto), alongside the safetensors adapter. - Thread the new fields through models, routes, orchestrator, and worker; extend the export tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate export by accelerator with a torch-aware reason; fix export save dir naming Export runs through Unsloth, which requires a compute accelerator (NVIDIA/AMD/Intel GPU or Apple MLX) and has no CPU code path, so a bare-CPU host cannot export even with PyTorch installed. Add export_capability() in utils/hardware that reports export_supported plus a precise reason so the UI stops showing a generic "no GPU": - pytorch_not_installed: a --no-torch install (even a physical GPU is unusable) - no_accelerator: PyTorch present but no supported accelerator (bare CPU) - mlx_unavailable: Apple Silicon where the MLX stack is missing or too old Expose the fields on /api/system/hardware and /api/system, and guard the mutating export routes (load-checkpoint, export/merged|base|gguf|lora) with HTTP 400 and the reason, leaving read-only endpoints usable so the Export page still renders. Make core/export/export.py import without PyTorch and without a usable accelerator (the Unsloth import is caught) so the export worker degrades to a clear message instead of crashing at import. Frontend: keep /export reachable on chat-only hosts and gray out the method and format options with the backend reason (Alert plus disabled MethodPicker) instead of silently redirecting to /chat, so users see why export is unavailable. Also fix the export save directory producing "model/null" for Local Model and Hugging Face sources that have no run/checkpoint, naming the folder from the model id. * CI: validate Studio export capability gating on Linux, Windows and macOS Add a small pytest matrix that runs studio/backend/tests/test_export_capability.py on ubuntu-latest, windows-latest and macos-latest. It confirms, on each real OS, that hardware.export_capability() reports the right decision and reason (pytorch_not_installed, no_accelerator, or mlx_unavailable) and that the export backend imports without PyTorch and degrades to a clear message instead of crashing. Hosted runners have no GPU/MLX, so this covers the "export unavailable, here is why" path a Mac/Windows user without an accelerator sees; a real accelerator export is validated separately. The job installs only a CPU PyTorch plus the backend import deps (no unsloth, triton, or llama.cpp), so it runs in seconds with no GPU. * Studio export: address Codex review (source-aware gating, GGUF LoRA token/MLX/guard) Frontend (export-page): - Gate LoRA and quantized-model restrictions on the active source. isAdapter / isQuantized come from the selected checkpoint; in Local Model / Hugging Face ("model") source mode they were stale, so LoRA stayed wrongly enabled for a direct base model (backend then rejects "No adapter to export") and a stale "quantized" flag disabled every method for an unrelated, exportable model. Add effectiveIsAdapter / effectiveIsQuantized (false outside checkpoint mode) and use them in the method-reset effect and the MethodPicker disabled state. - Hide the GGUF LoRA option on a macOS/MLX host (the backend rejects GGUF LoRA on MLX), so users no longer pick it, wait through the load, and always fail. Disable the "GGUF adapter" button on a Mac host and never send loraGguf there. Backend (core/export/export.py): - Pass the HF token into the GGUF LoRA conversion (save_pretrained_gguf), so a gated/private base model's config fetch in convert_lora_to_gguf.py is authenticated; without it the load can succeed but the conversion fails. - Guard the save_pretrained_gguf capability check with getattr so an older Unsloth model that lacks the method returns the clean "not supported" message instead of an AttributeError that surfaces as a generic 500. * Studio export: address 2nd Codex review (CI index, empty merged, test import) - studio-export-capability-ci.yml: add --extra-index-url https://pypi.org/simple to the torch install so torch's transitive deps still resolve; --index-url alone replaces PyPI with only the CPU wheel index, which does not serve all of them. - export-page handleStart: reject an empty merged selection (mirrors canExport), so clicking the panel's Start button with every precision pill deselected no longer submits mergedSelections: [] and launches an unintended default 16-bit export. - test_export_imatrix_compressed: the torchao-registry test now reads unsloth/save.py as text (like the other ast/string checks) instead of `import unsloth.save`, which raised ModuleNotFoundError in the CPU studio-backend suite that has no unsloth installed. * Studio export: make comments succinct across the export changes * Studio export: use load token for local GGUF LoRA export of gated bases * Studio export: harden portable torchao path and gate multi-format Hub push torchao (_unsloth_save_torchao): - merge to an isolated temp staging dir so a co-selected 16-bit output at save_directory is not deleted - narrow VLM detection to vision_config / ForVisionText2Text so T5/BART/Whisper are not misrouted - forward trust_remote_code (from auto_map) to the reload so custom-code models export Export UI: - hide portable torchao formats on macOS/MLX (backend rejects quantized export there) - restrict a Hub merged export to a single format (each writes to the repo root) * Studio export: torchao tokenizer remote-code + XPU offload, scale GGUF timeout torchao (_unsloth_save_torchao): - honor auto_map in the staged tokenizer/processor configs (not just model.config) when deriving trust_remote_code, so custom-code tokenizers reload after the merge - offload single-device XPU models to CPU (and empty the XPU cache) before the reload, matching the CUDA path, so an Intel GPU that fits the model once does not OOM on the second copy Export orchestrator: - scale the GGUF wait timeout by the number of requested quants so a multi-quant list export of a large model does not time out at a flat 3600s * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio export: show portable torchao formats only on non-NVIDIA (CPU) hosts Portable torchao FP8/INT8 is the fallback for hosts without the NVIDIA compressed-tensors path. On an NVIDIA GPU the compressed-tensors FP8/FP4/INT formats are the intended path (llm-compressor auto-installs), so hide the portable duplicates there; keep them on CPU / non-NVIDIA hosts and continue hiding them on macOS/MLX. * Studio export: report all output folders and the exported formats - Multi-format merged export now collects every sibling output directory (one per selected precision) instead of only the last; the success banner lists them all. - Show the selected precision formats in the run summary (a Formats row, like GGUF Quantizations), so the panel says what is being exported rather than just 'Merged Model'. - Persist the selected formats in the run summary and seed them on mount, so navigating away and back (or toggling the export method) restores the selection instead of resetting to 16-bit. * Studio export: list all output formats, add GGUF LoRA target, default Q8_0, auto-select newest checkpoint - Progress/summary panel now shows a Formats row with the selected merged formats, and the success banner lists every output folder a multi-format merged run creates (one line per format) instead of only the last one. - Merged format selection is seeded from the active run, so navigating away and back (or switching method cards) no longer resets it to 16-bit. - GGUF / Llama.cpp now offers an Export target toggle (Full model or LoRA adapter) for adapter checkpoints, reusing the LoRA GGUF export path. - Removed the Auto GGUF LoRA output type and defaulted to Q8_0 in the UI, the request model, and the backend defaults; the outtype list is now Q8_0/F16/BF16/F32. Core save.py still accepts auto for external callers. - When a finetune has no checkpoint selected, auto-select the newest one. * Studio torchao export: robust reload class + optional VLM import Two fixes to the portable torchao FP8/INT8 export reload, from review of the narrowed VLM detection: - Encoder-decoder seq2seq checkpoints (T5/BART/Whisper) are not causal LMs. With the narrowed is_vlm test they now correctly skip the image-text class, but fell through to AutoModelForCausalLM and failed to reload after the merge. Reload them with their own architecture class from the config instead. - AutoModelForImageTextToText was imported unconditionally at the top of the torchao path, so on Transformers builds without that class the import aborted every torchao export (even text-only). Import it lazily only for a VLM, with the AutoModelForVision2Seq fallback used elsewhere in Unsloth. * Studio: enable FP8/FP4 compressed export for newer-transformers models The shipped llm-compressor 0.10.x pins transformers<=4.57.6, so FP8/FP4 export failed for models needing a transformers 5.x sidecar (Qwen3.5, Gemma-4, Qwen3-Next): the quantization subprocess crashed importing the removed TORCH_INIT_FUNCTIONS. Run the quantization against a dedicated llm-compressor-main "shadow": a --target package dir (transformers 5.10.2 + llm-compressor main + compressed-tensors) layered over the existing torch. It installs --no-deps so torch is never touched (works on any Studio torch build), is provisioned lazily and fingerprint-cached, and can be turned off with UNSLOTH_DISABLE_LLMCOMPRESSOR_MAIN. - transformers_version.py: provision + validate .venv_llmcompressor. - export.py: route all compressed exports through the shadow when available; else keep the workspace 0.10.x path and fail fast past its transformers ceiling. - save.py: launch _compressed_quantize.py with a clean PYTHONPATH = shadow. - _compressed_quantize.py: skip linear_attn / vision tower / MTP modules (matches the RedHatAI and NVIDIA reference quants, and is required by the grouped schemes). Verified all four schemes (fp8, w8a8, w4a16, mxfp4) on Qwen3.5-9B and Llama-3.2-1B, and fp8 on Gemma-4, end to end through Studio. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix GGUF LoRA export tests * Fix export CI expectations * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com> Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>	2026-07-03 08:25:10 -07:00
Daniel Han	9fd4a503e8	fast_generate: clear error for vLLM-style inputs when fast_inference=False (#6786) ... * fast_generate: clear error for vLLM-style inputs when fast_inference=False When fast_inference=False, fast_generate falls back to HuggingFace generate, and the wrapper already rejects vLLM-only usage (a sampling_params or lora_request kwarg, or a string prompt). A vLLM prompt dict ({'prompt':..., 'multi_modal_data':...}) or a SamplingParams passed positionally slipped through and hit transformers.generate, raising a cryptic 'SamplingParams object has no attribute update'. Detect both and raise the same clear 'only supported with fast_inference=True' error. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fast_generate: also reject positional list of SamplingParams and list of vLLM prompt dicts Address review feedback: the slow-mode guard missed SamplingParams passed inside a positional list and a list of {"prompt": ...} dicts, both valid vLLM batched shapes that leaked into transformers.generate. Fold the checks into small predicates and extend the GPU-free test (now 7 reject + 3 pass). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * test_fast_generate_slow_guard: expose assertions via a test_ function so pytest collects them The assertions lived in run(), only called from __main__, so pytest reported no tests collected and CI skipped the coverage. Rename to test_fast_generate_slow_guard; the standalone script entrypoint still works. * fast_generate: reject vLLM tokenized/embeds prompt dicts in the slow-mode guard vLLM also accepts prompt dicts keyed by prompt_token_ids or prompt_embeds, not just prompt/multi_modal_data. Those slipped past the slow-mode guard and fell through to HuggingFace generate with a cryptic error. Recognize all vLLM prompt-dict keys and add a TokensPrompt test case (now 8 reject + 3 pass). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fast_generate slow-mode guard: catch vLLM prompts= keyword form vLLM's generate names its first argument `prompts`, so a slow-mode call like fast_generate(prompts="hi") or prompts=[{"prompt": ...}] bypassed the guard and leaked into HuggingFace generate as an unexpected kwarg. Check kwargs["prompts"] with the same _is_vllm_prompt predicate and add two test cases. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fast_generate slow-mode guard: reject vLLM tokenized prompt kwargs vLLM's legacy call shape passes tokens as prompt_token_ids= (and prompt_embeds=), which are not HuggingFace generate arguments. In slow mode these bypassed the guard and leaked into HF generate as unexpected kwargs. Reject their presence with the same tokenize-first message and add a test case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fast_generate slow-mode guard: treat prompts= as vLLM-only prompts is a vLLM keyword, not a HuggingFace generate argument, so any value passed as prompts= (including a bare token-id list, which _is_vllm_prompt deliberately ignores for positional HF token ids) is a vLLM-style call. Reject prompts= / prompt_token_ids= / prompt_embeds= on presence, and keep the conservative _is_vllm_prompt check only for the positional arg. * fast_generate slow-mode guard: reject vLLM prompt kwargs on presence prompts / prompt_token_ids / prompt_embeds are vLLM-only keyword names that HuggingFace generate does not accept, so a defaulted call like prompts=None should raise the actionable slow-mode error instead of leaking a None kwarg into HF generate. Check membership in kwargs rather than a non-None value. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>	2026-07-03 08:16:32 -07:00
Long Yixing	d918245834	Add MLX-aware public Unsloth trainer API (#6462) ... * feat: add mlx public trainer api * test: cover mlx public trainer api * fix: preserve mlx epoch trainer configs * fix: pass mlx warmup ratio through config * fix: align mlx trainer dataset order * fix: keep mlx chat templates import-light * fix: infer mlx trainer context length * fix: mirror cuda mlx context defaults * fix: align mlx notebook trainer defaults * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: keep mlx public helpers import-light * refactor: reuse mlx optimizer normalization * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: address mlx review feedback * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: tighten mlx training argument parity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: align mlx trainer eos default * Fix MLX trainer to accept DataCollatorForSeq2Seq and handle TokenizerWrapper in get_chat_template * Trim redundant docstrings on internal MLX helpers * MLX review fixes: Studio optimizer import-safe on non-MLX hosts, preserve explicit max_length, skip MLX tests before import * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * MLX review round 2: defer max_length to model context, optimizer alias fallback for older zoo, skip non-MLX test on missing GPU deps * MLX review round 3: keep chat_templates importable without torch on MLX * fix: preserve MLX trainer notebook shims * fix: ignore CUDA tokenizer moves on MLX * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: harden MLX trainer shims * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: unwrap MLX scheduler enum args * fix: coerce integral MLX epoch counts * fix: spoof CUDA compatibility APIs on MLX * fix: harden MLX notebook compatibility shims * MLX: add torch.cuda.mem_get_info to the compatibility shim Notebook memory cells call torch.cuda.mem_get_info()[0] directly (not gated by is_available), so on MLX it raises without a shim. Return (free, total) bytes from the MLX device stats, consistent with the other torch.cuda compat helpers, and add a matching assertion to the compat-API test. * MLX: use active memory for mem_get_info; fix BatchEncoding.to keyword device Address review on the MLX compatibility shim: - torch.cuda.mem_get_info() now derives free bytes from current active MLX memory instead of the peak high-water mark, so a capacity check stays accurate after a transient spike or a prior run. - BatchEncoding.to(device=...) passed by keyword no longer forwards a positional None alongside the keyword (which raised "multiple values for 'device'"), so non-CUDA keyword moves like .to(device="cpu") delegate correctly. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * MLX: accept preserve_dataset_order; stub RL trainers with a clear error Two fixes so unmigrated notebooks behave predictably on MLX (torch present): - preserve_dataset_order is a real MLXTrainingConfig field but was missing from the extra-argument allowlist, so passing it (as a config or trainer kwarg) could be rejected as unknown on a zoo without the field. Add it to _MLX_IMPLEMENTED_EXTRA_ARGUMENTS so the documented no-shuffle path is reachable. - GRPO/DPO/ORPO (and KTO/PPO/Reward) have no MLX trainer yet. Retarget the ones the installed trl exposes to a stub that raises a clear 'not supported on MLX' error instead of importing the real torch/CUDA trainer and crashing deep inside it. Only existing trainers are retargeted (no invented attributes), idempotent across re-imports. * MLX: make RL-trainer stubbing import-safe; back current-memory APIs with active memory Address review on the MLX shims: - The RL-trainer stub loop probed trl with getattr(_trl, name), which triggers trl's lazy trainer import and pulls torch -- that can crash import unsloth on a torch-free MLX install just to check existence. Decide what to stub from trl.__all__ + already-materialized attrs (vars) instead; never resolve the real trainer. All trl trainer names are in __all__, so they are still stubbed (even torch-free), and the probe no longer imports torch. - torch.cuda.memory_reserved / memory_allocated (the current, non-max APIs) were aliased to peak max_memory_reserved. Back them with current active MLX memory so cleanup / capacity checks see live usage; max_* keep the peak high-water mark. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * MLX: keep TRL's SFTConfig epoch default under the trl.SFTConfig alias Unmigrated notebooks import SFTConfig from trl, which the MLX build aliases to the public training-args class. TRL/HF SFTConfig defaults to num_train_epochs=3 (max_steps=-1); the native MLX config defaults to max_steps=60. So an SFTConfig built without an explicit length silently ran 60 MLX steps instead of TRL's 3 epochs under the alias. Alias trl.SFTConfig to a thin subclass that seeds the TRL epoch default only when neither max_steps nor num_train_epochs is given; explicit lengths pass through untouched, and the native public args class keeps its MLX default. Epoch mode is supported by the MLX trainer. * MLX CI: keep the GGUF reload smoke under the job timeout The RELOAD-GGUF-via-llama-cli step timed out at 300s. BF16 GGUF decode is CPU-bound on the macOS runner (~10s+/token), so generating 24 tokens landed right on the 300s cliff and killed the process. This step is a save/reload integrity smoke (it only needs a few chars of output), so the token count is incidental: generate 8 tokens with explicit threads and a small headroom on the subprocess timeout, all env-tunable (UNSLOTH_GGUF_RELOAD_N / _THREADS / _TIMEOUT). Cuts the reload well under the 25 minute job budget. * MLX: broaden trainer stubs, real peak-memory reset, fix shim tests Address review on the MLX public API: - The SFTConfig identity tests asserted trl.SFTConfig is UnslothTrainingArguments, but the alias now points at the _MLXSFTConfig subclass that preserves TRL's epoch default, so the MLX suite failed before testing the shim. Assert issubclass instead. - torch.cuda.reset_peak_memory_stats was a no-op, so max_memory_reserved kept earlier model-load peaks across a scoped run. Wire it to mx.reset_peak_memory with the same core/metal fallback used for the reads. - The unsupported-trainer stubs were a fixed list, so trainers outside it (a newer RLOOTrainer) still routed to the real torch trainer. Derive the set from trl.__all__ (every non-SFT *Trainer) so all non-SFT surfaces fail with a clear MLX message; names come from __all__ so trl is never resolved. - The non-MLX export smoke skipped only on missing bitsandbytes/triton; other absent GPU deps (numpy/torch/unsloth-zoo, or _gpu_init re-raising ImportError) made it fail on CPU hosts. Skip on any ImportError. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: keep MLX notebook compatibility minimal * MLX CI: force CPU + small context for the GGUF reload smoke The RELOAD-GGUF-via-llama-cli step timed out even at 8 tokens (>420s), so it is a fixed hang, not per-token cost: on the paravirtual macOS runner GPU llama.cpp's Metal backend stalls, and the gemma3 GGUF advertises a 32768 context that llama-cli would otherwise fully allocate. Run llama-cli CPU-only (-ngl 0) with a small context (-c 256); keep generation short. All env-tunable (UNSLOTH_GGUF_RELOAD_NGL / _CTX / _N / _THREADS / _TIMEOUT). Also print llama.cpp's partial stdout/stderr on timeout so a future hang is diagnosable instead of an opaque TimeoutExpired. * MLX CI: export the reload-smoke GGUF as q8_0, not bf16 The GGUF reload via llama-cli timed out on the runner even CPU-only with a tiny context and 8 tokens. Root cause is the format, not the flags: the smoke exported quantization_method='not_quantized', which maps to a bf16 GGUF, and llama.cpp's bf16 CPU decode is unusably slow on the paravirtual macOS runner. Export q8_0 (fast_quantized, the exporter default and what users deploy) instead -- llama.cpp has optimized q8_0 CPU kernels, so the fresh-process reload loads and generates in seconds. The reload stays CPU-only (-ngl 0) with a small context. * test: clear TRL shim before availability check --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com>	2026-07-02 23:02:26 +01:00
Leo Borcherding	73e8245ee8	[Studio] Add --with-llama-cpp-dir installer flag to reuse a local llama.cpp (#6472) ... * Add --with-llama-cpp-dir flag to install.ps1 and install.sh Users can now pass --with-llama-cpp-dir /path/to/llama.cpp to the installer to skip downloading or building llama.cpp and use a local directory instead. A junction (Windows) or symlink (Linux/macOS) is created at the canonical install location, bypassing both the prebuilt download (Phase 3) and source build (Phase 4) steps in setup.ps1/setup.sh. The path is passed via UNSLOTH_LOCAL_LLAMA_CPP_DIR env var which setup.ps1 and setup.sh read directly. Ported from the idea in unslothai/unsloth#4384, reimplemented against current Studio architecture. * test: add static wiring test for --with-llama-cpp-dir flag Cross-checks install.sh, install.ps1, studio/setup.sh and studio/setup.ps1 so the flag's contract (parse -> UNSLOTH_LOCAL_LLAMA_CPP_DIR env var -> link local dir, skip prebuilt download and source build) can't silently regress. Wired into studio-backend-ci.yml alongside the other tests/sh installer tests. * Address review feedback on --with-llama-cpp-dir flag - setup.ps1: delete an existing junction/symlink via DirectoryInfo.Delete() instead of a recursive remove, which can traverse the link and wipe the user's real llama.cpp directory on PowerShell 5.1. - setup.ps1: short-circuit the build chain when a local dir is linked so CMake never runs inside the user's checkout when it lacks a Windows-layout binary. - install.sh / setup.sh: resolve paths with CDPATH= cd -P so a set CDPATH cannot corrupt the resolved path. - install.sh: seed _WITH_LLAMA_CPP_DIR from UNSLOTH_LOCAL_LLAMA_CPP_DIR so an exported env var (piped-install style) is honored instead of being clobbered. - setup.sh: create the root llama-quantize shim when linking a local source build so GGUF export's check_llama_cpp() still finds it. - setup.sh / setup.ps1: drop a stale link before the custom-home ownership assert so re-runs with the flag stay idempotent. - test: pin the new linked-dir build short-circuit. * Harden --with-llama-cpp-dir against Codex/Gemini review findings - install.sh: error when --with-llama-cpp-dir is the final arg with no path, matching the existing --package/--python post-loop guards (was a silent fallback to the normal prebuilt/source install). - studio/setup.sh: canonicalize LLAMA_CPP_DIR before the self-link no-op compare. _RESOLVED_LOCAL is fully resolved while LLAMA_CPP_DIR was textual, so a symlinked $HOME made the guard miss and the rm -rf could wipe the user's real llama.cpp tree. - studio/setup.sh: make the llama-quantize shim non-fatal; it writes through the link into the user's tree, which may be read-only (shared/CI cache), and under set -e a failed ln aborted an otherwise-good reuse. - studio/setup.ps1: detect a broken junction via Get-Item -Force instead of Test-Path so a dangling link from a prior run is removed and mklink can relink to a new valid directory. - studio/setup.ps1: use Copy-Item -LiteralPath so a source path containing [ ] isn't treated as a wildcard in the junction copy fallback. - tests: update the wiring assertions for the LiteralPath copy and the canonicalized compare. * Validate/reuse local llama.cpp tree and guard the in-use case Addresses the second Codex pass on the --with-llama-cpp-dir flag: - Validate the linked tree before disabling installs (setup.sh + setup.ps1): reusing a local dir skips BOTH the prebuilt download and the source build, so the dir must already contain a runnable llama-server (build/bin on Linux/macOS, build\bin\Release\llama-server.exe on Windows). Bail out with a clear message instead of linking an unbuilt/wrong-platform checkout and leaving Studio with no usable binary. - Treat a canonical-path target as already linked when it holds a build (setup.sh + setup.ps1): point the flag at ~/.unsloth/llama.cpp itself and an existing build is reused (skip prebuilt + source) rather than clobbered by the staged prebuilt installer (which uses os.replace()/replace). An empty canonical dir still falls through to the normal in-place install. - Abort when an in-use llama.cpp can't be removed on Windows (setup.ps1): Remove-Item -ErrorAction SilentlyContinue can silently leave a locked tree in place; detect that and stop with the same active-process message + exit 3 the prebuilt path uses, instead of junctioning over a half-present dir. Left as follow-up (already tracked by the PR author as a non-blocker): the in-app "Update llama.cpp" updater does not yet recognize a local-link install as externally managed; that fix belongs in studio/backend/utils/llama_cpp_update.py. * Accept all backend llama-server layouts in --with-llama-cpp-dir validation The linked-tree validation only accepted build/bin[/Release]/llama-server, but LlamaCppBackend._layout_candidates() resolves a root-level llama-server first, then build/bin, then build/bin/Release on Windows. A `make` build or a flat release extract (binary at the dir root) was therefore rejected with a hard installer failure even though Studio would have run it. Validate the same candidate set the backend uses in both setup scripts, and add wiring-test assertions so the check can't silently narrow again. * Treat --with-llama-cpp-dir local links as externally managed A --with-llama-cpp-dir install junctions/symlinks the canonical llama.cpp dir to the user's own checkout, but two backend paths still treated it as a Studio-owned tree: - The in-app updater (llama_cpp_update) offered and could apply an official prebuilt over the link, writing through it into the user's checkout (or failing) and silently dropping the link the flag created. - Orphan cleanup (LlamaCppBackend._kill_orphaned_servers) resolved the linked root into its kill allowlist, so a llama-server the user launched from the same checkout was classified as ours and killed on startup. Detect the canonical dir being a symlink/junction (reparse point) and treat the install as unmanaged: get_update_status reports unsupported, start_update refuses with reason "local_link", and the linked root is left out of the orphan allowlist. Adds behavioral tests (link vs plain dir, updater refusal, and the spared-vs-killed orphan control). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add behavioral shell test for --with-llama-cpp-dir linking The existing tests/sh/test_with_llama_cpp_dir_flag.sh is a static grep of the scripts. This adds a behavioral test that extracts the real link block from studio/setup.sh (by content anchors, with a self-validating extraction) and runs it against hermetic fake dirs, asserting the outcomes that matter: - an external CMake build links and arms neither the prebuilt download nor the source build - a flat / make tree (root-level llama-server, no build/bin) is accepted too - an unbuilt tree is rejected with a non-zero exit and no link left behind - relinking over a stale link preserves the target's contents (no data loss) - pointing at the canonical path is a no-op reuse, not a self-referential link Symlink-identity checks run only where real symlinks exist (skipped on Windows git-bash copy-mode); the link/skip/no-data-loss checks run everywhere. Wired into studio-backend-ci.yml next to the static test. * Install psutil in backend CI so orphan-cleanup tests run The new orphan-cleanup tests import psutil for the process scan, but the Backend CI deps step installed studio.txt plus a fixed extras list that omits it, so the two tests failed with ModuleNotFoundError. Add psutil to both backend pytest dep steps (kept in shared shape), and guard the import with pytest.importorskip so a minimal env without psutil skips these tests instead of erroring. --------- Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-02 22:11:20 +01:00
Gabriel Pereira Góes	22cd26f75d	feat: Implementation of the Portuguese (Brazil) language and VRAM/RAM monitor (#6509) ... * feat: Implementation of the Portuguese (Brazil) language and VRAM/RAM monitor. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update studio/frontend/src/hooks/use-gpu-utilization.ts Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update studio/backend/main.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update studio/backend/main.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update studio/backend/utils/hardware/hardware.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update studio/backend/utils/hardware/hardware.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update studio/frontend/src/features/settings/components/usage-examples.tsx Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update studio/backend/main.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update studio/backend/utils/hardware/hardware.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update studio/frontend/src/features/studio/sections/progress-section.tsx Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * fix: resolve automated review feedback on API shape * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix review issues for PR #6509: Cpu icon, VRAM percent, system polling - model-inspector: use the exported CpuIcon (Cpu is not a Hugeicons export) - app-sidebar: guard the VRAM percent on totalVram to avoid Infinity, and reset the system poll cache only after each request settles so a slow probe is reused instead of stacking overlapping requests - use-gpu-info: populate CPU/RAM on hosts without a GPU - progress-section: label GPUs by visible_ordinal instead of array index - hub-page: base the RAM label on systemRamTotalGb - usage-examples: emit JS sampling and tool options at the top level instead of nesting them under extra_body (the JS SDK does not unwrap extra_body) - main: read torch and transformers versions from package metadata instead of importing the libraries on every system poll, and guard the VRAM math against null values - hardware: translate a leftover comment to English * Harden /api/system: guard psutil.boot_time for PR #6509 Simulating restricted containers and some VMs (where psutil.boot_time can raise) showed the /api/system endpoint would 500 on the unguarded boot_time call, the same failure class already handled for cpu_freq, disk_usage, and Process. Wrap boot_time and return uptime_seconds as null when it is unavailable so the sidebar monitor degrades gracefully instead of breaking. Widen the uptime_seconds type to number | null to match. * Studio: make the sidebar hardware monitor a toggle (default on) for PR #6509 Adds a "Show hardware monitor" switch under Settings > Appearance > Layout, backed by a localStorage preference (default on), mirroring the existing useSidebarPin pattern. When turned off, the sidebar hides the VRAM/RAM meters and useSystemInfo stops the 3s /api/system poll entirely, so no nvidia-smi / SMI probes run while the monitor is disabled. Adds the en and pt-BR strings. * Studio: default the sidebar hardware monitor to off (opt-in) for PR #6509 * Studio pt-BR: fix three small translation defects for PR #6509 - learningRateDescription: "5e-5 for CPT" -> "5e-5 para CPT" (leftover English) - exportScopeRecents: "Recents" -> "Recentes" (untranslated) - relativeMonthsAgo/relativeYearsAgo: add the missing space ("há {count} meses"/ "há {count} anos") so they no longer render as "há 3meses" * Studio pt-BR: translate the last 10 fallback keys for PR #6509 Adds the settings.general.storage block (Armazenamento) and the settings.chat.modelDisclaimer pair, so pt-BR now covers all en keys (679/679) with no English fallbacks. * Studio: hide sidebar VRAM row on CPU-only hosts for PR #6509 * Studio: tighten and trim code comments for PR #6509 * fix: UI issue in the stop button dialog box (fine-tuning) * Studio pt-BR: translate 18 new keys from main merge (password dialog, GGUF export, dataset streaming) for PR #6509 * Rounding to GB * Fix/adjust System resources tab for PR #6509 * Fix/adjust GPU monitor review items for PR #6509 * Fix/adjust remaining GPU monitor review items for PR #6509 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix/adjust MLX resource fallback for PR #6509 * floating window implementation * resize for floating window * Fix resource monitor review items * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore frontend optional dependency lock entries * Make GPU selection tests hermetic * Fix GPU monitor CI test failures * Bound MLX GGUF reload smoke * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix MLX GGUF reload smoke exit --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: wasimysaid <wasimysdev@gmail.com>	2026-07-02 18:06:39 +02:00
Daniel Han	d91183d03f	Fix gpt-oss offload_embedding and generate() kwargs, and guard offload_embedding on tied/vLLM models (#6774) ... * Fix gpt-oss offload_embedding and generate() logits_to_keep on fused models offload_embedding=True moved embed_tokens to CPU but left the input/output device-shuffling forward hooks commented out ('[TODO] Doesn't seem to work!'), so an eager forward/generate with CUDA input_ids hit the CPU embedding and raised a device-mismatch RuntimeError. Re-implement them in a testable helper _install_offload_embedding_hooks that saves the origin device on the module (the pre-hook returns a new tensor, so a device stashed on the original input is lost) and runs the lookup on the embedding weight's CURRENT device. Reading the weight device at call time (not a hard-coded cpu) also handles a non-quantized (bf16) embedding that a later model.to(...) pulls back onto the GPU, which the hard-coded version broke in the opposite direction. unsloth_base_fast_generate injected logits_to_keep/num_logits_to_keep whenever an inner submodule forward accepted it, but transformers validates generate kwargs against the top-level prepare_inputs_for_generation (plus forward when it takes kwargs). On fused/PEFT-wrapped gpt-oss this raised 'model_kwargs are not used by the model: [logits_to_keep]'. Only inject when the top level would accept it, mirroring transformers _validate_model_kwargs. Behavior is unchanged for every model that works today. Adds tests/test_offload_embedding_hooks.py and tests/test_generate_kwarg_gate.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * gpt-oss offload hooks: store origin device on the tensor, not the shared module The pre-hook stashed the input device on embed_tokens itself, which races when concurrent forwards share the module (serving). Ride it on the moved tensor and read it from the post-hook args instead: stateless and thread-safe. * Also strip mm_token_type_ids that generate() rejects (Qwen3-VL vision GRPO) The vision processor (Transformers 5.x path) emits mm_token_type_ids, which Qwen3-VL's generate() then rejects in _validate_model_kwargs on transformers 4.x, so vision GRPO fails at the first rollout: ValueError: The following `model_kwargs` are not used by the model: ['mm_token_type_ids'] Unlike logits_to_keep this is an incoming kwarg rather than one we inject, so drop it in unsloth_base_fast_generate when the top level generate does not accept it, reusing the same _unsloth_generate_accepts_kwarg gate. Extends the GPU-free gate test with the accept/reject mm_token_type_ids cases (7/7 pass). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim mm_token_type_ids comment * Trim comments in gpt-oss offload/logits fix (comment-only) * gpt-oss offload: return embedding output to the decoder device, not the input's When offload_embedding moves the embedding to CPU, model.device can become CPU and inputs then arrive on CPU, so returning the output to the input device left it on CPU and the CUDA decoder hit a device mismatch. Capture the decoder device before offload and always return there. This also drops the per-request tensor state (stateless, so concurrent forwards stay correct). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * gpt-oss offload: refuse offload_embedding for tied word embeddings Tied models share embed_tokens.weight with lm_head, so offloading the weight to CPU strands the output projection there (device mismatch at generate) and saves no VRAM since lm_head still needs it on GPU. Detect the shared weight via get_output_embeddings and raise NotImplementedError instead of loading into a crash. Untied models (gpt-oss, Llama-3.1-8B) offload unchanged. Adds tests/test_offload_tied_guard.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * gpt-oss offload: skip embedding offload on fast_inference (vLLM) vLLM manages its own weights, so offload_embedding cannot apply on the fast_inference path (previously it was silently ignored). Disable it with a notice, mirroring the WSL and Windows skips. * Trim offload embedding comments (comment-only) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * gpt-oss offload: track decoder device live so it survives model.to() The post-hook returned the embedding output to a device captured at load time. If a model is loaded on CPU then moved with model.to(cuda), that device is stale and the output lands on the wrong device. Read the decoder device live from the (untied) output embeddings, keeping the captured device as a fallback. Adds a stale-fallback regression test. * Make generate-kwarg-gate cases pytest-collectable Cases lived in run(), which pytest does not collect, so CI never exercised the gate. Expose them as test_generate_kwarg_gate; still runnable via __main__. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * gpt-oss offload: skip a meta (disk-offloaded) lm_head as the return device A device_map that disk-offloads an untied lm_head leaves its weight on the meta device until that module's own hook runs, so reading it as the decoder device would move real hidden states to meta. Skip meta (and a missing weight) and fall back to the captured device. Adds a regression test. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-01 22:39:00 -07:00
Filip Trajkovic	c5adb69a10	Fix GRPO logit scaling when model is wrapped by DDP (#5955) ... --------- Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2026-07-02 00:01:15 -03:00
Daniel Han	ec4c044e70	Pin llm-compressor auto-install to a vetted version range (#6778) ... * Pin llm-compressor auto-install to a vetted version range install_llm_compressor() auto-installs llm-compressor on first use of an FP8/FP4 compressed export when it is not already present. The install command used the bare package name, so pip resolved to whatever the configured index served; a compromised, dependency-confused, or inflated-version ("999.0.0") release could then run under the Unsloth process at install and import time. Bound the automatic install to a vetted range (_LLM_COMPRESSOR_SPEC = "llmcompressor>=0.8.0,<0.13"), which the oneshot / QuantizationModifier API this uses supports, so pip can no longer jump to an arbitrary future or inflated version. An already-installed newer llm-compressor is still used as-is (the import short-circuits), so this only constrains the auto-install, never a user's own install. Add UNSLOTH_DISABLE_LLM_COMPRESSOR_AUTOINSTALL=1 to forbid the automatic install entirely and require a manual, vetted install, for locked-down or air-gapped environments. Update the manual-install hints to the pinned spec. Add tests/saving/test_llm_compressor_install_pin.py: static (ast) guards that the spec stays a bounded pin, that the install command never passes an unpinned llmcompressor literal, and that the opt-out env gate is evaluated before any install runs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Loosen llm-compressor auto-install ceiling to <1.0 so new models still export The earlier <0.13 ceiling was too tight: brand-new architectures (for example Qwen3_5ForConditionalGeneration / qwen3_5, gemma-4 MoE) can require a newer llm-compressor, and _unsloth_save_compressed_tensors already fails with "requires a newer llm-compressor" when a scheme is unavailable. Capping the auto-install at 0.12 would block getting that newer release and break compressed export for new models. Widen to llmcompressor>=0.8.0,<1.0. pip still auto-installs the latest 0.x (where new-architecture support lands), while the <1.0 ceiling continues to block a jump to an inflated-version ("999.0.0") or 1.0+ dependency-confusion release. An already-installed newer llm-compressor is still used as-is (the import short-circuits), and UNSLOTH_DISABLE_LLM_COMPRESSOR_AUTOINSTALL still forbids the automatic install entirely for locked-down environments. * Lower llm-compressor floor to 0.6.0 so supported old torch still resolves The >=0.8.0 floor conflicts with the torch this install pins in its constraints file. Unsloth supports torch>=2.4, but llm-compressor 0.7.0+ require torch>=2.7 (0.10+ need >=2.9, 0.12+ need >=2.10). On a supported torch 2.4-2.6 box pip then has no candidate in [0.8.0, 1.0) and FP8/FP4 export fails before quantization. Lower the floor to 0.6.0 (its metadata only needs torch>=1.7), which never conflicts with any supported torch. pip still prefers the newest compatible release, so modern torch continues to get the latest 0.x (0.12.0). The <1.0 ceiling that blocks an inflated-version supply-chain jump is unchanged. Add a regression test asserting the floor stays <= 0.6.0. * Cap llm-compressor auto-install ceiling to a vetted minor (<0.13) A bare <1.0 ceiling still admits any 0.x, so an inflated "0.999.0" served by a compromised or misconfigured index would win pip's highest-version selection -- the same dependency-confusion this pin is meant to block. Cap the ceiling to the current vetted minor (<0.13) so that jump is blocked; bump it deliberately, after vetting, when a newer llm-compressor is needed (e.g. for a brand-new architecture scheme). Current new models are unaffected: 0.12.0 is < 0.13 and supports them. The 0.6.0 floor (torch>=1.7 compatible) is unchanged, so resolution still works across Unsloth's whole supported torch range (2.4 -> 0.6.0 ... 2.12 -> 0.12.0). Add a regression test asserting the ceiling admits the current vetted release but blocks an inflated 0.x and the next major. * Trim comments in the llm-compressor pin (comment-only, no code change) * Cap llm-compressor auto-install to the exact vetted patch (<=0.12.0) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-01 04:48:38 -07:00
Daniel Han	73d9653d5b	scan_packages: key baseline on matched-code hash so payloads in baselined files are not auto-suppressed (#6552) ... * scan_packages: key baseline on matched-code hash The baseline matched on (package, package-relative file, check), which excluded the matched code, so a future finding of the same check in the same file was suppressed regardless of what the code did. A malicious future version of an already-baselined package could place a payload in the same file under the same check and pass the enforcing gate. Key the baseline on a hash of the matched code too. The hash is over the deduped, sorted set of matched spans with L<NN>: line markers stripped, so version bumps, line shifts and match reordering stay stable while new or changed flagged code reopens the finding. Version is left out of the key so routine dependency bumps do not reopen every entry. The hash is capped and recomputable from the stored evidence. Regenerate scan_packages_baseline.json against the current dependency set; the hf-stack, studio and extras scan shards pass enforcing (no active CRITICAL or HIGH). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_packages: refresh baseline for newer unsloth-zoo release A newer unsloth-zoo published after the first regenerate added tests/test_mlx_save_export_regressions.py, a benign test fixture (temporary_location="/tmp/ignored") that trips the /tmp dropper check. Regenerate the hf-stack shard against the current set so the entry is allowlisted; studio and extras are unchanged. * scan_packages: harden baseline loading against malformed JSON Guard against a non-dict top-level baseline and non-dict entries so a corrupt or hand-edited allowlist warns and fails closed instead of crashing with AttributeError, and treat an explicit evidence: null as empty. * scan_packages: hash the full match set, keep indentation, strip only the marker Address the evidence-hash review feedback: - Capture every matching line, not the first three, so a payload appended after existing matches in a baselined file and check reopens the finding instead of riding the sample. - Preserve leading indentation so a flagged line moved out of a guarded block reads as changed. - Strip only each span's prefix up to the first L<NN>: marker, so an L<NN>: inside the matched code is kept and a change to it reopens the finding. Evidence and its hash are stored in full and stay recomputable from the stored field. Regenerate the baseline; hf-stack, studio and extras pass enforcing with no active CRITICAL or HIGH. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_packages: bind baseline evidence to full matched code Address review feedback on the evidence-hash baseline key: - Split evidence only on real span delimiters (" | " before an L<NN>: marker, or a newline), so a bitwise-or or union type in matched code is no longer split apart into separate spans. - Record matched lines in full (drop the 160-char per-line cap) and record every distinct multiline match, so code appended past the cap or a second cross-line match reopens the finding instead of riding the first one. - Give the large-JS-bundle and .pth base64-blob findings a content digest instead of empty or prefix-only evidence, and record all .pth import lines, so a changed bundle, blob or import no longer inherits a baselined empty or truncated key. - Warn when a loaded baseline has entries without evidence_hash so a legacy baseline is regenerated rather than silently degraded. Regenerate scripts/scan_packages_baseline.json against the current dep set and add regression tests for each case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_packages: harden multiline and duplicate evidence handling Follow-up hardening so the evidence hash tracks the full matched code: - For DOTALL patterns that match across lines, record every line the match spans (not just the start line), so a change on a continuation line (the URL inside a baselined C2 loop, a swapped credential path) reopens the finding. A pathological greedy span is bounded to its head line plus a digest of the rest. - Keep duplicate spans in the canonical evidence so a second identical matched line in a new code path changes the key instead of deduping away. - Anchor the evidence prefix to strip only a genuine leading label or line-number marker, leaving a marker-like "L<NN>:" inside raw .pth code intact. - Make the legacy-baseline warning explicit that entries without an evidence_hash reopen rather than suppress under a coarse key. Regenerate scripts/scan_packages_baseline.json (same finding set; entries for same-file repeated checks are now tracked separately) and add tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_packages: bind every combo and large finding to its full content Close the remaining asymmetric-evidence gaps so a changed payload cannot ride a reviewed baseline entry: - Digest a capped multiline span from the code without line markers, so a pure line shift stays stable while a continuation-line change reopens. - Give the "Unusually large executable .pth" finding a content digest instead of keying on byte size and import-line count alone. - Record both contributing signals for the JS credential+network stealer, the shell credential+network and persistence-hook combos, and the hidden network+exec docstring payload, so changing the network/exec side reopens. - Allow punctuation in an evidence label prefix so a "network+exec:" label is stripped and line shifts do not change the key. Regenerate scripts/scan_packages_baseline.json and add tests for each case. * scan_packages: bind remaining Python combos; key npm baseline on evidence Python scanner: the openssl+key, anti-analysis, DNS-exfil and base64+exec+blob combos recorded only one contributing signal, so a changed payload on the other side could ride a reviewed baseline entry. Each now binds every co-occurring signal (and the blob is digested, since it can sit on a separate line from the decode call). npm scanner: scan_npm_packages.py keyed its allowlist on (package, path, pattern) only, the same coarse-key bypass the Python scanner just closed. Add an evidence hash to the key (schema v3, fail-closed on older baselines) and store full evidence. The committed baseline stays empty by design. Regenerate scripts/scan_packages_baseline.json and add tests for each case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_npm_packages: bind full blob evidence and harden baseline loader Follow-up on the npm evidence-hash key: - _evidence now records every match and, when a snippet is truncated for display, appends a digest of the full match. The obfuscated-blob key was hashing only the truncated first-match snippet, so a changed payload tail or an appended blob in the same package/file/pattern could ride a reviewed entry. - _load_baseline guards that the root is an object, entries is a list, and each entry is a dict before reading it, so a malformed baseline warns and fails closed instead of raising AttributeError. Add tests for a changed blob tail reopening the key and for malformed entries. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_packages: symmetric baseline-loader guards; bind npm outbound host context - Python _load_baseline now rejects a non-list "entries" with a warning instead of raising TypeError, matching the npm loader. - npm cred-surface-host (outbound) records the host with its URL path / fetch call / host config, so a changed outbound path, headers or body reopens the key rather than riding the bare host literal. Add tests for both. * scan_npm_packages: migrate v2 baselines and bind host-config outbound context - _load_baseline now migrates schema v2 entries by recomputing the evidence hash from stored evidence (with a legacy warning), matching the Python loader, instead of discarding them; only pre-v2 basename schemas are rejected. - The cred-surface-host (outbound) host-config branch now captures the whole line (path, headers, body), so a changed outbound payload on the same hostname line reopens the key instead of riding the bare host snippet. Add tests for v2 migration and the host-config context binding. * scan packages: bind PEM key bodies and npm windowed evidence to baseline keys scan_packages: embedded-key findings now pin the full PEM block (BEGIN..END) via a content digest, so a key body swapped under the same marker reopens the finding instead of riding the unchanged BEGIN line. Single-line and DER keys were already bound by their full matched line; marker-only references with no END block (validation header lists) are unaffected, so the committed baseline is unchanged. scan_npm_packages: _evidence now digests the full containing line whenever the shown snippet is only a window into it (short match on a long line, or a truncated payload), so a changed payload tail outside the display window reopens the key. The npm baseline is empty, so this changes no suppressions. Adds regression tests for both cases. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan packages: bind multi-line evidence and every blob to baseline keys _extract_evidence now extends each single-line match over its bracket continuations, so a multi-line call binds its argument lines and a changed URL or body on a continuation line reopens the key. After the per-line pass it also records cross-line matches the scan cannot otherwise see (a DOTALL regex, or a multi-line construct appended under a check that already had a one-line match), so an appended multiline payload reopens instead of riding the key. _blob_digest hashes every large base64 blob (not just the first) for the base64+exec finding and the .pth large-blob finding, so an appended or swapped second encoded payload reopens; single-blob files keep the same digest. scan_npm_packages _evidence digests the full logical line (the matched line plus its bracket-continuation lines), so a multi-line fetch's option and header lines bind and a changed payload on a following line reopens the outbound key. Regenerated the Python baseline: same package/file/check set, 24 entries pick up the wider multi-line evidence. Adds regression tests for each case. * scan packages: stop giant greedy spans from binding a whole-file digest When a greedy DOTALL pattern (reverse shell socket...subprocess, C2 loop) has its anchor tokens far apart, the match span covers the whole file. Digesting that span bound thousands of unrelated lines, so the evidence hash drifted on any edit between the anchors (a dependency bump reshuffling the file), which made a baselined finding reopen on an upstream release. The multiline pass now skips an oversized span when the per-line pass already bound the signal lines, so the evidence is the stable matched lines; a genuinely appended multi-line construct stays under the cap and is still recorded. Regenerated the Python baseline against Python 3.12 (the version the scan CI shards run) so the resolved dependency set matches CI. Same package/file/check set. Adds a regression test. * scan packages: tighten evidence binding (order, string brackets, span size) Address review follow-ups on the evidence extraction: - _canon_evidence keeps discovery (line) order instead of sorting. Line-shift stability already comes from stripping the L<NN>: markers, so order stays significant and reordering matched lines (a multi-line call's arguments) reopens the finding. - _logical_line_end (Python) and _logical_line_text (npm) blank string literals before counting brackets, so a ) inside a string argument does not close the logical line early and drop later argument lines. - The oversized-span skip now only drops a giant whole-file bridge (over 60 lines); a genuinely appended multi-line construct is recorded so its payload reopens, rather than riding an existing one-line match. - npm _logical_line_text binds the enclosing bracket group, so a host-config object whose { is on a prior line binds its path/headers/body lines. Regenerated the Python baseline (Python 3.12, matching the scan CI shards): same package/file/check set. Adds regression tests for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan npm packages: normalize and bound the logical-line digest - _evidence whitespace-normalizes the logical line before digesting (matching _evidence_hash), so a formatter-only reindent of the bound continuation lines does not change the sha256 suffix and reopen an unchanged finding. - _logical_line_text follows a bracket group to its close up to a hard 200-line cap (digest input only), so a config object longer than the backward window still binds its whole tail instead of silently truncating. Adds regression tests. npm baseline is empty, so no regeneration is needed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan: cap single-line evidence and widen npm opener window Cap each rendered evidence line at 200 chars in scan_packages.py: a long or minified one-line file is shown as a bounded prefix plus a sha256 of the full line, so a packed payload cannot dump unbounded content into the CI logs or baseline while a change past the cutoff still changes the digest and reopens the finding. Mirrors how the npm scanner bounds its snippets. Widen the npm backward opener window (_MAX_CONT_LINES 12 to 200, symmetric with the forward cap) so a host deep inside a large options object binds the whole object, not just its own line; a changed path, header, or body on any property reopens. Regenerate the Python baseline with Python 3.12: only the protobuf nspkg.pth and unsloth-zoo compiler.py evidence change, both from the new line cap; the package/file/check key set is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan: bind all host contexts, deep call continuations, far-back npm openers Three fail-closed evidence gaps surfaced by review of the previous round. scan_npm_packages.py: measure the forward bracket-group cap from the matched line (idx + _MAX_GROUP_LINES) instead of the opener, so an opener found near the widened backward limit no longer consumes the forward budget and drops the path, headers, or body that follow the host. scan_npm_packages.py: _outbound_host_evidence now records every outbound context form for a host (URL, fetch-context, host-config), claiming each non-overlapping match in form order, so a separate host-config request added beside an already-baselined URL changes the evidence and reopens the key. The common single-context case keeps its existing snippet. scan_packages.py: follow a matched Python call over its continuations up to a separate _MAX_CALL_LINES (40), decoupled from the 12-line display threshold, so a multi-line requests.post( binds its whole argument list in the digest and a changed body deep in the call reopens; bounded so a miscounted bracket cannot swallow unrelated code. No baseline change: the current dependency set has no matched call that closes between 13 and 40 lines, confirmed by a Python 3.12 regenerate that produced a byte-identical baseline. * scan: clamp npm depth, pin large bundles, follow backslash and bound .pth dump Four fail-closed evidence gaps surfaced by review of the previous round. scan_npm_packages.py: clamp the backward opener scan at depth 0 so a leading unmatched closer (a preceding block whose opener is outside the backward window) no longer drives depth negative and masks the real enclosing opener that follows; a host-config object after such a block now binds and a changed path reopens. scan_packages.py: a large JS bundle now pins its whole content even when another JS heuristic already fired. The bundle digest was only added when no other finding existed; it is now appended to every finding's evidence on a large bundle, so an unchanged obfuscation signature no longer lets changed payload elsewhere ride the matched-line key. scan_packages.py: _logical_line_end follows explicit backslash line continuations, so a call split with a backslash before its parenthesis binds the continuation line (URL/body) instead of returning at the zero-depth API line. scan_packages.py: the catch-all .pth import evidence is bounded through _cap_line (prefix plus a digest of every line) so a large .pth of benign imports cannot dump the whole member into the logs or baseline while an appended or swapped import still reopens. Baseline regenerated with Python 3.12: key set unchanged; one entry (unsloth-zoo compiler.py) gains the backslash-continued banner lines now bound by the continuation fix. * scan: handle multi-line strings, lifecycle bodies, and de-quadratic evidence Addresses a review round plus a performance audit of the evidence extractor. Correctness (fail-closed): - Bind the UNION of the single-line-blanked and multi-line-blanked bracket spans in both scanners. The multi-line view blanks a triple-quoted Python string or a backtick template literal that spans lines, so a `)` inside such a string no longer closes the enclosing call early and drop later arguments. The single-line view still counts a payload embedded INSIDE a string, so a dropper that hides a call in a string keeps its argument lines bound. Taking the larger span never shrinks the binding below either view, avoiding a fail-open regression. - cred-env-in-lifecycle now pins the whole lifecycle script body via a digest, so a changed non-token line (e.g. adding a curl exfil beside the token reference) reopens, not just a change on the token line. Performance / DoS (the scanner runs on attacker-controlled package files up to the 64 MiB / 16 MiB member caps, with no per-file time budget): - _extract_evidence precomputes newline offsets once and maps match offsets with bisect, removing the O(matches) whole-file content.count per match that made the finditer fallback quadratic (a crafted minified file went from ~13 s/MiB and hours at the cap to linear). - npm _index_text splits and string-blanks the file once per evidence call instead of per match (was O(matches x file) time and allocation). - Bound evidence output: _MAX_EVIDENCE_SPANS (Python) and _MAX_EVIDENCE_MATCHES (npm) fold the remainder into a digest so a file with thousands of matches cannot build a multi-megabyte evidence/baseline blob while an added/removed match past the cap still changes the key. - _outbound_host_evidence caps matches per form and bounds the overlap claim so a host repeated many times cannot make it quadratic. No baseline change: a Python 3.12 regenerate is byte-identical (the union equals the legacy single-line span for every current dependency file; the cap thresholds sit above the largest real entry), so these are forward-looking hardening with no drift. * scan: count all overflow matches, bind their context, blank JS regex literals Follow-ups on the evidence output caps from the previous commit. - _outbound_host_evidence no longer truncates each pattern's match iterator with islice; it iterates every match and runs the overlap dedup only while the display list is below the cap (so claimed stays bounded and the check is O(cap) per match, not quadratic), folding every match past the cap into the overflow digest. A host context beyond the 64th is counted again, so it reopens. - The overflow digest (both scanners, via a shared _overflow_digest) binds each overflow match's logical-line context, not just the regex match text, so a changed payload on an over-cap line reopens even with the matched token unchanged. - The multi-line JS blanked view now blanks regex-literal bodies (tracking the previous significant char for regex-vs-division and char classes for a literal `/` inside `[...]`), so a `)` inside `/)/` no longer closes an outbound call early. The bound span is the union of the single-line and multi-line views, so an imperfect regex decision only ever grows the span, never shrinks it. - The Python overflow digest canonicalizes spans (strips L<NN>: markers via _canon_evidence) before hashing, restoring line-shift stability for the over-cap region. No baseline change: the overflow branches only trigger above the per-finding caps (above the largest real entry), and the npm baseline is empty, so a Python 3.12 regenerate is byte-identical. * scan: refresh baseline for ipython interactiveshell.py span drift A newer ipython release changed the filesystem-enumeration span in IPython/core/interactiveshell.py, so its content digest no longer matched the baselined evidence and the studio scan shard flagged it as a non-baselined CRITICAL. Regenerated with Python 3.12: only the ipython entry's evidence_hash changes; the package/file/check key set is unchanged, and a studio enforcing spot-check exits 0. * Bound scanner evidence memory: stream overflow spans and cap lifecycle baseline size scan_packages.py: _extract_evidence no longer materializes a rendered span per match before slicing at the display cap. Once out holds _MAX_EVIDENCE_SPANS spans, further spans fold straight into a running digest, so a minified or padded file with hundreds of thousands of matching lines keeps memory bounded to the display cap instead of the match count. The fold reproduces _canon_evidence(" | ".join(overflow)) byte for byte, so the overflow digest and every baseline key are unchanged. scan_npm_packages.py: lifecycle-fetch-exec and cred-path-in-lifecycle stored the entire install script body as evidence, so --write-baseline on a package with a multi-MiB lifecycle script bloated the baseline JSON. Both now store a bounded matched snippet plus a body-sha256 digest, matching cred-env-in-lifecycle. The digest still binds the whole body, so a change to any line reopens the finding. Adds tests for the streamed overflow bound and the bounded-but-reopens lifecycle evidence. Baseline unchanged (byte-identical Python evidence; npm baseline empty). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make npm bracket-group scan order-aware so a same-line close-then-open binds _scan_group counted brackets with a per-line net (opens minus closes), which collapses intra-line order: a line that closes a prior block and then opens the host-config object on the same line, e.g. `}); const opts = {`, nets to <= 0, so the trailing `{` was dropped and the group started at the hostname line. A changed path/headers on the following lines then hashed to the same evidence and could ride an existing baseline key. Replace the net count with an order-aware (L, R) reduction per line (L closers needing an opener to the left, R openers needing a closer to the right) and apply it in order in both the backward and forward scans, clamping stray closers at 0. The trailing opener now stays visible so the whole object binds and a changed payload reopens. Per-line cost is unchanged (one C-level bracket findall), so the existing outbound-host evidence is byte-identical on all prior shapes; only the previously-dropped same-line case changes. Adds a regression test for it. * Harden scanner evidence: bound memory and bind Python call tails fail-closed Five fixes across both scanners, none of which change the committed baseline (a full regen of all three pip shards produced a byte-identical 185-key set). scan_npm_packages.py: _evidence and _outbound_host_evidence collected every regex match into a list before applying the 64-match display cap, so a text file under the size cap that repeats a cheap signal (such as NPM_TOKEN) millions of times could allocate a huge list of re.Match objects and stall or OOM before the overflow digest ran. They now stream from finditer and fold overflow as matches arrive via a shared _fold_overflow_match helper, byte-identical to the prior digest. scan_packages.py: - _extract_evidence kept inserting every unique over-cap span into the seen set even after it stopped appending to the display list, so a generated file with millions of one-line matches still grew that set unbounded. It now tracks spans only while filling the display list (per-line spans are unique by line number, so dropping them past the cap cannot miss a dedup). - _scan_line_end counted brackets with a per-line net, so a continued statement that closes on the same line it opens a flagged call (a leading "]" before "requests.post(") had the call's open paren cancelled and bound only the opener line. It now applies brackets in order via _bracket_lr (leading closers clamp at 0), matching the npm bracket fix. - a single-quoted string continued by a trailing backslash was not tracked across lines, so a close paren inside the continued string on the next line closed the call early; _blank_code_strings now carries the continuation. - a call with more argument lines than the soft cap was hashed only through the cap, so a changed data=/headers tail past it stayed suppressed; a closing call is now followed to its real close under a 200-line hard limit (a never-closing opener still stops at the 40-line soft cap so it cannot swallow the file). Adds regression tests for each. npm baseline is empty; the Python baseline is unchanged (verified byte-identical by regenerating all three shards). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bind giant DOTALL span anchors and add context to constant IOC evidence Two fail-closed gaps where a changed payload could keep the same evidence hash and stay suppressed by the baseline. scan_packages.py: a giant greedy DOTALL span (a cross-line IOC match bridging more than 60 lines, e.g. RE_TEMP_EXEC matching a /tmp line and a much-later subprocess line) was dropped entirely once the per-line pass had any match, so an appended cross-line payload -- a new /tmp line plus a later subprocess line that share no single line, so the per-line pass never binds them -- produced the same evidence and rode the key. The span is no longer dropped: it is bound by its head and tail anchor lines plus a digest over just those (no line numbers, so a pure line shift is stable). An added or moved anchor reopens the finding, while churn in the bridged interior stays stable, so this does not reintroduce whole-file drift. Two baseline entries (multiprocess test, unsloth-zoo scanner file) carry such a span and are refreshed; a full three-shard regen confirmed only those two keys change. scan_npm_packages.py: known-ioc-string and cred-surface-host (always-bad) recorded only the bare needle/host as evidence, so a reviewed tarball that kept the IOC string while altering the adjacent fetch/exfil body produced an identical key. They now bind matched-line context: known-ioc-string via the matched line and its bracket-group continuation, cred-surface-host (always-bad) via the outbound call context (path/headers/body, falling back to the bare host when not in an outbound call). A changed payload on the same call now reopens. Adds regression tests for each. npm baseline is empty; the Python baseline updates only the two giant-span entries. * Hash giant-span interiors, bind exec/eval trigger, JS content, intra-literal whitespace Four fail-closed gaps where a changed payload could keep the same evidence hash. scan_packages.py: - A giant bridged DOTALL span was bound only by its head and tail anchors, so a cross-line payload inserted into the bridged interior between unchanged outer anchors kept the same key. The whole span content is now digested (via _render), so any interior change reopens; a pure line shift stays stable because the digest is over the markerless code. Two baseline entries (multiprocess test, unsloth-zoo scanner file) carry such a span; with full-interior binding, multiprocess resolved at two versions across shards now yields two distinct entries where the anchor digest had collapsed them into one. - The exec/eval-with-hidden-payload findings omitted the visible exec/eval line that makes the hidden string executable, so flipping a harmless eval("1+1") to exec(__doc__) kept the same key while arming the payload. The trigger line from the real-code view is now bound into the evidence. - check_js_file extracted evidence with the Python-string-aware extractor, which does not blank JS backtick template literals, so a template containing a close paren closed a call's bracket span early and omitted later option/body lines. The full file content digest is now pinned to every JS finding (not just large bundles), binding the whole call. scan_npm_packages.py: the evidence canon collapsed all whitespace via split(), erasing whitespace inside JS string literals along with harmless indentation, so a changed request body 'a b' -> 'a b' kept the same key. A new _canon_preserve_strings collapses whitespace only OUTSIDE string literals (reindent-stable) while preserving it INSIDE single/double/backtick literals (intra-payload edits reopen). Used for the evidence hash and the logical-line digests. Adds regression tests for each. npm baseline is empty; the Python baseline updates the two giant-span entries and adds the second multiprocess version's entry. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-01 04:03:59 -07:00
Daniel Han	bc69dfad08	MLX CI: find llama-cli where save_pretrained_gguf actually installs it (#6777) ... * MLX CI: find llama-cli where save_pretrained_gguf actually installs it The GGUF reload step hardcoded the CWD-relative paths llama.cpp/llama-cli and llama.cpp/build/bin/llama-cli, but save_pretrained_gguf builds and installs llama.cpp under unsloth_zoo's LLAMA_CPP_DEFAULT_DIR ($UNSLOTH_LLAMA_CPP_PATH, else ~/.unsloth/llama.cpp), so the reload could not find the binary and failed the Mac M1 job with "llama-cli not found". _find_llama_cli now searches that install directory (and honors the env override) before falling back to the old CWD layout, with a recursive glob as a last resort. The search is a strict superset of the previous paths, so it cannot regress a layout that already worked. * MLX CI: return an absolute llama-cli path from the locator Resolve the located binary to an absolute path. If UNSLOTH_LLAMA_CPP_PATH is a relative directory (e.g. "."), Path(".") / "llama-cli" normalizes to the bare name "llama-cli", and subprocess.run treats a separator-less argument as a PATH lookup rather than a file to execute, raising FileNotFoundError. resolve() makes the returned path absolute so it always runs the intended binary. * MLX CI: give llama-cli EOF on stdin so GGUF reload cannot hang With the binary now found, the GGUF reload actually invokes llama-cli and it timed out after 300s generating 24 tokens on a 270m model, which is a stdin block rather than slow generation: subprocess.run captured stdout/stderr but left stdin inherited, so -no-cnv still left llama-cli waiting for interactive input. Pass stdin=subprocess.DEVNULL so it receives an immediate EOF and runs the single prompt to completion. --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com>	2026-07-01 00:49:23 -07:00
Daniel Han	8cc05ac89c	Reduce comments across recent fixes (#6776) ... Condense the verbose comments and docstrings added by the recent chat template, GPT-OSS detection, PEFT tensor-parallel, and Studio inference proxy fixes. Comments and whitespace only; no code changes.	2026-06-30 23:13:36 -07:00
Rod Boev	2246a6c9ae	fix: keep LoRA reloads working with PEFT 0.19 (#6748) ... * fix: keep LoRA reloads working with PEFT 0.19 * test: exercise the PEFT tensor-parallel symbol extractor * test: prove the full PEFT tensor-parallel seam * fix: harden PEFT tensor-parallel shims * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: fall back when PEFT tensor-parallel source inspection fails --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com>	2026-06-30 20:26:57 +01:00
Daniel Han	b72a8c4263	studio: explicit Cloudflare tunnel notice and public-exposure warning at startup (#6515) ... * studio: announce Cloudflare tunnel state and warn about public exposure on startup The startup banner only printed a line when a tunnel URL was up, so a plain `unsloth studio -H 0.0.0.0` launch silently created a public trycloudflare.com URL with no indication that Studio had become reachable from the internet. The only hint at the tunnel was the CLI help, shown when an invalid command was typed. Make the banner always state the tunnel state for wildcard binds: - ON: the public URL plus a warning that anyone with it can reach Studio from outside the network, and that --no-cloudflare keeps it local-only. - FAILED: requested but did not start (local network only). - OFF: --no-cloudflare was passed (local network only). Secure mode keeps its existing wording (the authenticated tunnel is intended and --no-cloudflare is not valid there). Clarify the --cloudflare help text in both the argparse and typer definitions. Default behavior is unchanged. Also surface the state on the `unsloth studio run` banner, which runs the server with silent=True and prints its own banner: it now calls _print_cloudflare_line too, so the ON/OFF/FAILED notice and public-exposure warning are no longer skipped on that path (previously it only echoed the URL when a tunnel was up). For the OFF and FAILED notices, do not claim "local network only" when the reachability probe just confirmed the raw port is reachable from the public internet: --no-cloudflare and a failed tunnel disable only the Cloudflare link, not the wildcard bind, so the message is reworded to flag the public raw port. * Fix/adjust Cloudflare banner warnings for PR #6515 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix/adjust Cloudflare banner comments for PR #6515 * Fix/adjust IPv6 Cloudflare tunnel gate for PR #6515 * Fix/adjust Cloudflare review comments for PR #6515 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix silent run Cloudflare notice * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: wasimysaid <wasimysdev@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-30 17:47:48 +02:00
Daniel Han	9369dd47e6	Add FP8/FP4 compressed export to save_pretrained_merged (#6706) ... * Add FP8/FP4 compressed export to save_pretrained_merged Adds compressed-tensors export (for vLLM) to save_pretrained_merged / push_to_hub_merged via llm-compressor, alongside the existing lora / merged_16bit / merged_4bit / gguf / torchao paths: model.save_pretrained_merged("model", tokenizer, save_method="fp8") Supported save_method values: fp8 (FP8_DYNAMIC), mxfp4, nvfp4 (W4A4) and mxfp8. The LoRA is merged to 16bit at save_directory, then a quantized checkpoint is written to save_directory + "-<fmt>". nvfp4 needs a small calibration set (defaults to ultrachat, overridable via calibration_dataset). Notes: - llm-compressor is installed lazily on first use, pinning the current torch and transformers via a constraints file so they are not upgraded (a plain install pulls transformers>=5 and breaks Unsloth). - Quantization runs in a separate process (unsloth/_compressed_quantize.py, launched by file path) so Unsloth's transformers attention patches do not interfere with the forward llm-compressor runs during calibration, mirroring how GGUF export shells out to llama.cpp. - mxfp8 needs a newer llm-compressor (transformers>=5); it is recognised and raises a clear error until that stack is available. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: main-process guard, calibration subsampling, tokenizer + dtype handling - Route the 16bit merge through unsloth_generic_save for both LoRA and full finetuned models, so non-PEFT models are written in 16bit consistently instead of saving the original (possibly quantized) weights directly. - Honor is_main_process: only the main process quantizes and writes the compressed output, so distributed ranks do not race on the same dirs. - Subsample an in-memory calibration Dataset before save_to_disk so large training sets are not fully copied to a temp dir. - Tolerate a missing tokenizer in the converter (data-free exports); still require one for calibration based schemes. - Open config.json via a context manager in both files. - Drop the redundant nvfp4 entry from the unsupported-name check (fp4 covers it). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add direct LoRA to GGUF export and harden FP8/FP4 compressed export - Run llm-compressor install and scheme check before the 16bit merge so unsupported schemes (e.g. mxfp8) fail fast without writing a checkpoint - Only the main process installs, merges, quantizes and uploads; isolate hub pushes to a temp dir and clean all temp dirs in a finally - Forward standard save kwargs (state_dict, max_shard_size, ...) to the merge - Fall back to the first dataset split for Hub calibration ids - Export LoRA adapters to GGUF via convert_lora_to_gguf.py: modernize save_pretrained_ggml/push_to_hub_ggml and add save_method="lora" to save_pretrained_gguf/push_to_hub_gguf; resolve base from the adapter config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix LoRA GGUF shell-injection test and compressed export trailing-slash path - Update tests/saving/test_save_shell_injection.py for the new delegation: the LoRA to GGUF conversion now lives in _unsloth_save_lora_gguf, so assert it passes argv as a list with no shell=True and that the legacy ggml wrappers delegate to it instead of calling subprocess.Popen directly - Normalize the local save_directory before building the "<dir>-<fmt>" sibling so a trailing slash no longer nests the compressed output inside the 16bit dir * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Polish FP8/FP4 and LoRA GGUF export after review - Warn (not silently downgrade) when an explicit quantization_method is not a valid LoRA GGUF outtype; default stays f16 - Correct the inference hardware note: MXFP8 is 8-bit (cc >= 8.9), only FP4 needs Blackwell for full activation quantization - Document that a local fp8/fp4 save keeps the 16bit merge at save_directory and writes the quantized checkpoint to save_directory + "-<fmt>" * Use sequential calibration pipeline and validate Hub access early - nvfp4 calibration no longer forces the memory-hungry "basic" pipeline. The quantization runs in a clean subprocess, so llm-compressor's default sequential pipeline (layer-by-layer onloading) works and lets large models that do not fit at once still calibrate; fall back to "basic" only if tracing fails - For push_to_hub compressed exports, create/validate the repo up front so a bad token or denied repo fails before the merge and quantization instead of after * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden compressed export: explicit sequential pipeline, base-tokenizer calibration, GPU memory - nvfp4 calibration now passes pipeline="sequential" explicitly (layer-by-layer onloading) instead of relying on the inferred default, with a "basic" fallback - Calibration datasets with a messages column no longer require a chat template: base / non-chat tokenizers fall back to concatenating message contents - Free the in-memory model's CUDA memory before the quantize subprocess loads its own copy from disk (best-effort, single-device non-quantized only; restored afterward), so a single GPU need not hold two copies at once - Create the calibration temp dir in the system temp location instead of next to the save directory, avoiding stray dirs in the workspace * Free the failed calibration model before the basic-pipeline retry In the sequential -> basic NVFP4 fallback, release the partially-processed model and clear the CUDA cache before loading a fresh copy, so the retry does not transiently hold two model copies on the GPU. * Harden calibration data handling and compressed-export edge cases - Calibration messages without a chat template now handle multimodal (list) content, None content, and null message rows instead of crashing on join - Raise a clear error when the calibration dataset is empty after subsampling - Reset llm-compressor's global session before freeing the model in the sequential -> basic NVFP4 fallback, so the old model is actually released - LoRA GGUF export accepts a single-element list quantization_method - Attach datasets metadata to the pushed repo on compressed hub exports - Warn (instead of silently) if the model cannot be restored to its device - Raise a clear error if the LoRA base model id cannot be determined * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Handle DatasetDict calibration, MoE routers, and MTP models in compressed export - Reduce an in-memory DatasetDict calibration set to a single split before row subsampling, so save_to_disk does not copy every split to the temp dir - For MoE models, keep the router/gate unquantized and pass moe_calibrate_all_experts so every expert is calibrated - Warn when a model carries MTP / speculative-decoding tensors that the compressed export does not include * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Support many more compressed-tensors schemes and address review - Expand save_method to cover the full set of compressed-tensors preset schemes: FP8 (dynamic/static/block), INT8, W8A8, W8A16, W4A16(+asym), W4A8, W4AFP8, MXFP4(+A16), NVFP4(+A16), plus the gated MXFP8; calibration is used only for the static-activation schemes (FP8 static, NVFP4) - Broaden the near-miss save_method error to cover int/w-prefixed names - MoE: also keep the Qwen shared-expert gate unquantized - Strip non-model-input columns from already-tokenized calibration data so the collator does not choke on a leftover messages column - Forward the Hub token to the LoRA converter and the quantize subprocess so gated/private base models and calibration datasets work without a global login * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Collapse compressed-tensors export help line so ruff-format converges The print line in print_quantization_methods needed two ruff-format passes to reach a fixpoint (merge implicit string concat, then collapse the single-arg print). pre-commit.ci applies one pass per run, so it kept reformatting. Land the converged single-line form directly. * Add CPU-only regression tests for the export API Cover all export paths without a GPU, for slow CPU-only CI: - pure-function checks of the compressed-tensors scheme registry and save_method normalization (aliases, calibration flags, near-miss errors) - AST checks that every merged saver dispatches compressed export, the GGUF savers expose the lora branch, torchao routes PTQ/QAT, the public methods stay attached, and the export subprocesses remain shell-safe (argv list, sys.executable, no shell) - monkeypatched dispatch checks that fp8/nvfp4/merged_16bit, the LoRA-GGUF outtype resolution, and torchao PTQ/QAT reach the right helper with the right arguments * Run the CPU-only export tests in consolidated CI tests/saving is --ignored by the Repo tests (CPU) job, so the new GPU-free export tests are added by path to consolidated-tests-ci.yml (collection sanity + Bucket-A run), alongside the existing CPU saving tests, so they actually execute on CPU CI. * Add GPU GGUF export + llama-cli inference smoke test tests/saving/test_gguf_export_and_inference.py: skipif no CUDA. Trains a tiny phrase-imprinting LoRA, exports a full-model q8_0 GGUF (merge -> convert_hf_to_gguf -> llama-quantize), asserts a valid GGUF (magic + size), and - when a llama-cli binary is available - runs one bounded generation (byte cap + watchdog kill) and asserts the trained phrase round-trips through HF -> GGUF -> quantize -> inference. The llama-cli step skips gracefully since the export only builds llama-quantize. * Fix variant mismatch in compressed (FP8/FP4) export save_pretrained_merged(..., save_method=fp8/nvfp4, variant=...) forwarded the variant into the intermediate 16bit merge, so Transformers wrote variant-named shards (model.<variant>.safetensors). The converter subprocess then reloaded that directory with the default weight filenames, so the compressed export failed after doing the merge. Pop the variant out of the intermediate merge (internal staging that the subprocess reloads with default names) and forward it via --variant so it is applied to the final compressed checkpoint instead. Add a CPU AST guard for the contract. * Harden export paths from review - install_llm_compressor: fall back to uv pip when this interpreter has no pip seeded (uv-created/relocatable venvs), instead of failing with No module named pip. - LoRA GGUF export: if convert_lora_to_gguf.py is missing (a prebuilt or reused CWD llama.cpp install carries binaries but not the converter script), force a dedicated source checkout that ships it. - push_to_hub_gguf(save_method=lora): return on non-main ranks, matching the local save_pretrained_gguf lora branch, so only rank 0 converts/uploads. - compressed export VLM detection: require a vision_config or a ForVisionText2Text architecture; a bare *ForConditionalGeneration also matches text seq2seq models (T5/BART/Whisper) and is no longer treated as a VLM on its own. - GGUF GPU smoke test: drop SFTConfig(max_length=1024), which raises under newer TRL padding-free training; length enforcement is not needed here. * Add imatrix option to GGUF export, enabling IQ low-bit quants save_pretrained_gguf / push_to_hub_gguf gain imatrix_file: None -> no imatrix (unchanged) '/path' -> pass to llama-quantize --imatrix (a *.gguf_file is renamed to *.gguf) True -> download the upstream unsloth/<base>-GGUF imatrix (imatrix_unsloth.dat or .gguf_file), raising a clear error if none exists An importance matrix unlocks the IQ low-bit quants (iq2_xxs, iq4_xs, ...), which were hard disabled before. They are gated: requesting one without an imatrix raises a clear error. - _resolve_imatrix_file resolves path/True (PEFT base first, normalized via get_model_name, derives unsloth/<base>-GGUF, copies out of the HF cache before renaming *.gguf_file). - IMATRIX_QUANTS registry replaces the old commented-out IQ entries; save_to_gguf accepts a resolved imatrix and threads it into the quantize calls. - The --imatrix flag is emitted by unsloth_zoo's quantize_gguf (companion change). save.py fails fast with an upgrade hint if the installed unsloth_zoo lacks the imatrix kwarg. Tests: tests/saving/test_imatrix_export.py (CPU: resolution, repo derivation, IQ gate, --imatrix wiring) wired into CI; tests/saving/test_gguf_export_and_inference.py extended with GPU iq2_xxs/iq4_xs export + inference. Verified end to end on Llama-3.2-1B: imatrix auto-downloaded, iq2_xxs/iq4_xs exported and run via llama.cpp. Note: requires the companion unsloth_zoo quantize_gguf imatrix change. * Address imatrix/compressed review feedback: unsloth org GGUF repo, fail-fast, calibration split - imatrix auto-resolve (imatrix_file=True): derive the upstream repo as unsloth/<base>-GGUF instead of <org>/<base>-GGUF, so official bases (e.g. meta-llama/Llama-3.1-8B-Instruct) find the matching Unsloth GGUF imatrix repo rather than failing on a nonexistent meta-llama/...-GGUF. - Resolve/validate the imatrix before the 16-bit merge in save_pretrained_gguf, so a bad path or an unavailable upstream imatrix fails fast instead of after a long, multi-GB merge. - Compressed calibration: when a Hub dataset has no "train" split, resolve the first split name and slice it, instead of materializing the whole dataset just to take num_samples rows. Keeps the original materialize-then-subselect path as a last resort. Tests: add unsloth/<base>-GGUF mapping for an official base id, and create the imatrix file in the quantize_gguf flag test (quantize_gguf now validates the imatrix exists). --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-30 03:40:16 -07:00
Daniel Han	de3c745fab	Fix full finetuning precision on V100 / no-bf16 GPUs (#5880) ... --------- Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2026-06-29 18:35:23 -03:00
Vineeth Sai	677ec0cc20	Fix gpt-oss detection in save: config.architectures is a list, not a string (#6711)	2026-06-29 02:44:00 -03:00