258 commits

Author	SHA1	Message	Date
Daniel Han	bc69dfad08	MLX CI: find llama-cli where save_pretrained_gguf actually installs it (#6777) ... * MLX CI: find llama-cli where save_pretrained_gguf actually installs it The GGUF reload step hardcoded the CWD-relative paths llama.cpp/llama-cli and llama.cpp/build/bin/llama-cli, but save_pretrained_gguf builds and installs llama.cpp under unsloth_zoo's LLAMA_CPP_DEFAULT_DIR ($UNSLOTH_LLAMA_CPP_PATH, else ~/.unsloth/llama.cpp), so the reload could not find the binary and failed the Mac M1 job with "llama-cli not found". _find_llama_cli now searches that install directory (and honors the env override) before falling back to the old CWD layout, with a recursive glob as a last resort. The search is a strict superset of the previous paths, so it cannot regress a layout that already worked. * MLX CI: return an absolute llama-cli path from the locator Resolve the located binary to an absolute path. If UNSLOTH_LLAMA_CPP_PATH is a relative directory (e.g. "."), Path(".") / "llama-cli" normalizes to the bare name "llama-cli", and subprocess.run treats a separator-less argument as a PATH lookup rather than a file to execute, raising FileNotFoundError. resolve() makes the returned path absolute so it always runs the intended binary. * MLX CI: give llama-cli EOF on stdin so GGUF reload cannot hang With the binary now found, the GGUF reload actually invokes llama-cli and it timed out after 300s generating 24 tokens on a 270m model, which is a stdin block rather than slow generation: subprocess.run captured stdout/stderr but left stdin inherited, so -no-cnv still left llama-cli waiting for interactive input. Pass stdin=subprocess.DEVNULL so it receives an immediate EOF and runs the single prompt to completion. --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com>	2026-07-01 00:49:23 -07:00
Daniel Han	8cc05ac89c	Reduce comments across recent fixes (#6776) ... Condense the verbose comments and docstrings added by the recent chat template, GPT-OSS detection, PEFT tensor-parallel, and Studio inference proxy fixes. Comments and whitespace only; no code changes.	2026-06-30 23:13:36 -07:00
Rod Boev	2246a6c9ae	fix: keep LoRA reloads working with PEFT 0.19 (#6748) ... * fix: keep LoRA reloads working with PEFT 0.19 * test: exercise the PEFT tensor-parallel symbol extractor * test: prove the full PEFT tensor-parallel seam * fix: harden PEFT tensor-parallel shims * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: fall back when PEFT tensor-parallel source inspection fails --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com>	2026-06-30 20:26:57 +01:00
Daniel Han	b72a8c4263	studio: explicit Cloudflare tunnel notice and public-exposure warning at startup (#6515) ... * studio: announce Cloudflare tunnel state and warn about public exposure on startup The startup banner only printed a line when a tunnel URL was up, so a plain `unsloth studio -H 0.0.0.0` launch silently created a public trycloudflare.com URL with no indication that Studio had become reachable from the internet. The only hint at the tunnel was the CLI help, shown when an invalid command was typed. Make the banner always state the tunnel state for wildcard binds: - ON: the public URL plus a warning that anyone with it can reach Studio from outside the network, and that --no-cloudflare keeps it local-only. - FAILED: requested but did not start (local network only). - OFF: --no-cloudflare was passed (local network only). Secure mode keeps its existing wording (the authenticated tunnel is intended and --no-cloudflare is not valid there). Clarify the --cloudflare help text in both the argparse and typer definitions. Default behavior is unchanged. Also surface the state on the `unsloth studio run` banner, which runs the server with silent=True and prints its own banner: it now calls _print_cloudflare_line too, so the ON/OFF/FAILED notice and public-exposure warning are no longer skipped on that path (previously it only echoed the URL when a tunnel was up). For the OFF and FAILED notices, do not claim "local network only" when the reachability probe just confirmed the raw port is reachable from the public internet: --no-cloudflare and a failed tunnel disable only the Cloudflare link, not the wildcard bind, so the message is reworded to flag the public raw port. * Fix/adjust Cloudflare banner warnings for PR #6515 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix/adjust Cloudflare banner comments for PR #6515 * Fix/adjust IPv6 Cloudflare tunnel gate for PR #6515 * Fix/adjust Cloudflare review comments for PR #6515 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix silent run Cloudflare notice * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: wasimysaid <wasimysdev@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-30 17:47:48 +02:00
Daniel Han	9369dd47e6	Add FP8/FP4 compressed export to save_pretrained_merged (#6706) ... * Add FP8/FP4 compressed export to save_pretrained_merged Adds compressed-tensors export (for vLLM) to save_pretrained_merged / push_to_hub_merged via llm-compressor, alongside the existing lora / merged_16bit / merged_4bit / gguf / torchao paths: model.save_pretrained_merged("model", tokenizer, save_method="fp8") Supported save_method values: fp8 (FP8_DYNAMIC), mxfp4, nvfp4 (W4A4) and mxfp8. The LoRA is merged to 16bit at save_directory, then a quantized checkpoint is written to save_directory + "-<fmt>". nvfp4 needs a small calibration set (defaults to ultrachat, overridable via calibration_dataset). Notes: - llm-compressor is installed lazily on first use, pinning the current torch and transformers via a constraints file so they are not upgraded (a plain install pulls transformers>=5 and breaks Unsloth). - Quantization runs in a separate process (unsloth/_compressed_quantize.py, launched by file path) so Unsloth's transformers attention patches do not interfere with the forward llm-compressor runs during calibration, mirroring how GGUF export shells out to llama.cpp. - mxfp8 needs a newer llm-compressor (transformers>=5); it is recognised and raises a clear error until that stack is available. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: main-process guard, calibration subsampling, tokenizer + dtype handling - Route the 16bit merge through unsloth_generic_save for both LoRA and full finetuned models, so non-PEFT models are written in 16bit consistently instead of saving the original (possibly quantized) weights directly. - Honor is_main_process: only the main process quantizes and writes the compressed output, so distributed ranks do not race on the same dirs. - Subsample an in-memory calibration Dataset before save_to_disk so large training sets are not fully copied to a temp dir. - Tolerate a missing tokenizer in the converter (data-free exports); still require one for calibration based schemes. - Open config.json via a context manager in both files. - Drop the redundant nvfp4 entry from the unsupported-name check (fp4 covers it). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add direct LoRA to GGUF export and harden FP8/FP4 compressed export - Run llm-compressor install and scheme check before the 16bit merge so unsupported schemes (e.g. mxfp8) fail fast without writing a checkpoint - Only the main process installs, merges, quantizes and uploads; isolate hub pushes to a temp dir and clean all temp dirs in a finally - Forward standard save kwargs (state_dict, max_shard_size, ...) to the merge - Fall back to the first dataset split for Hub calibration ids - Export LoRA adapters to GGUF via convert_lora_to_gguf.py: modernize save_pretrained_ggml/push_to_hub_ggml and add save_method="lora" to save_pretrained_gguf/push_to_hub_gguf; resolve base from the adapter config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix LoRA GGUF shell-injection test and compressed export trailing-slash path - Update tests/saving/test_save_shell_injection.py for the new delegation: the LoRA to GGUF conversion now lives in _unsloth_save_lora_gguf, so assert it passes argv as a list with no shell=True and that the legacy ggml wrappers delegate to it instead of calling subprocess.Popen directly - Normalize the local save_directory before building the "<dir>-<fmt>" sibling so a trailing slash no longer nests the compressed output inside the 16bit dir * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Polish FP8/FP4 and LoRA GGUF export after review - Warn (not silently downgrade) when an explicit quantization_method is not a valid LoRA GGUF outtype; default stays f16 - Correct the inference hardware note: MXFP8 is 8-bit (cc >= 8.9), only FP4 needs Blackwell for full activation quantization - Document that a local fp8/fp4 save keeps the 16bit merge at save_directory and writes the quantized checkpoint to save_directory + "-<fmt>" * Use sequential calibration pipeline and validate Hub access early - nvfp4 calibration no longer forces the memory-hungry "basic" pipeline. The quantization runs in a clean subprocess, so llm-compressor's default sequential pipeline (layer-by-layer onloading) works and lets large models that do not fit at once still calibrate; fall back to "basic" only if tracing fails - For push_to_hub compressed exports, create/validate the repo up front so a bad token or denied repo fails before the merge and quantization instead of after * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden compressed export: explicit sequential pipeline, base-tokenizer calibration, GPU memory - nvfp4 calibration now passes pipeline="sequential" explicitly (layer-by-layer onloading) instead of relying on the inferred default, with a "basic" fallback - Calibration datasets with a messages column no longer require a chat template: base / non-chat tokenizers fall back to concatenating message contents - Free the in-memory model's CUDA memory before the quantize subprocess loads its own copy from disk (best-effort, single-device non-quantized only; restored afterward), so a single GPU need not hold two copies at once - Create the calibration temp dir in the system temp location instead of next to the save directory, avoiding stray dirs in the workspace * Free the failed calibration model before the basic-pipeline retry In the sequential -> basic NVFP4 fallback, release the partially-processed model and clear the CUDA cache before loading a fresh copy, so the retry does not transiently hold two model copies on the GPU. * Harden calibration data handling and compressed-export edge cases - Calibration messages without a chat template now handle multimodal (list) content, None content, and null message rows instead of crashing on join - Raise a clear error when the calibration dataset is empty after subsampling - Reset llm-compressor's global session before freeing the model in the sequential -> basic NVFP4 fallback, so the old model is actually released - LoRA GGUF export accepts a single-element list quantization_method - Attach datasets metadata to the pushed repo on compressed hub exports - Warn (instead of silently) if the model cannot be restored to its device - Raise a clear error if the LoRA base model id cannot be determined * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Handle DatasetDict calibration, MoE routers, and MTP models in compressed export - Reduce an in-memory DatasetDict calibration set to a single split before row subsampling, so save_to_disk does not copy every split to the temp dir - For MoE models, keep the router/gate unquantized and pass moe_calibrate_all_experts so every expert is calibrated - Warn when a model carries MTP / speculative-decoding tensors that the compressed export does not include * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Support many more compressed-tensors schemes and address review - Expand save_method to cover the full set of compressed-tensors preset schemes: FP8 (dynamic/static/block), INT8, W8A8, W8A16, W4A16(+asym), W4A8, W4AFP8, MXFP4(+A16), NVFP4(+A16), plus the gated MXFP8; calibration is used only for the static-activation schemes (FP8 static, NVFP4) - Broaden the near-miss save_method error to cover int/w-prefixed names - MoE: also keep the Qwen shared-expert gate unquantized - Strip non-model-input columns from already-tokenized calibration data so the collator does not choke on a leftover messages column - Forward the Hub token to the LoRA converter and the quantize subprocess so gated/private base models and calibration datasets work without a global login * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Collapse compressed-tensors export help line so ruff-format converges The print line in print_quantization_methods needed two ruff-format passes to reach a fixpoint (merge implicit string concat, then collapse the single-arg print). pre-commit.ci applies one pass per run, so it kept reformatting. Land the converged single-line form directly. * Add CPU-only regression tests for the export API Cover all export paths without a GPU, for slow CPU-only CI: - pure-function checks of the compressed-tensors scheme registry and save_method normalization (aliases, calibration flags, near-miss errors) - AST checks that every merged saver dispatches compressed export, the GGUF savers expose the lora branch, torchao routes PTQ/QAT, the public methods stay attached, and the export subprocesses remain shell-safe (argv list, sys.executable, no shell) - monkeypatched dispatch checks that fp8/nvfp4/merged_16bit, the LoRA-GGUF outtype resolution, and torchao PTQ/QAT reach the right helper with the right arguments * Run the CPU-only export tests in consolidated CI tests/saving is --ignored by the Repo tests (CPU) job, so the new GPU-free export tests are added by path to consolidated-tests-ci.yml (collection sanity + Bucket-A run), alongside the existing CPU saving tests, so they actually execute on CPU CI. * Add GPU GGUF export + llama-cli inference smoke test tests/saving/test_gguf_export_and_inference.py: skipif no CUDA. Trains a tiny phrase-imprinting LoRA, exports a full-model q8_0 GGUF (merge -> convert_hf_to_gguf -> llama-quantize), asserts a valid GGUF (magic + size), and - when a llama-cli binary is available - runs one bounded generation (byte cap + watchdog kill) and asserts the trained phrase round-trips through HF -> GGUF -> quantize -> inference. The llama-cli step skips gracefully since the export only builds llama-quantize. * Fix variant mismatch in compressed (FP8/FP4) export save_pretrained_merged(..., save_method=fp8/nvfp4, variant=...) forwarded the variant into the intermediate 16bit merge, so Transformers wrote variant-named shards (model.<variant>.safetensors). The converter subprocess then reloaded that directory with the default weight filenames, so the compressed export failed after doing the merge. Pop the variant out of the intermediate merge (internal staging that the subprocess reloads with default names) and forward it via --variant so it is applied to the final compressed checkpoint instead. Add a CPU AST guard for the contract. * Harden export paths from review - install_llm_compressor: fall back to uv pip when this interpreter has no pip seeded (uv-created/relocatable venvs), instead of failing with No module named pip. - LoRA GGUF export: if convert_lora_to_gguf.py is missing (a prebuilt or reused CWD llama.cpp install carries binaries but not the converter script), force a dedicated source checkout that ships it. - push_to_hub_gguf(save_method=lora): return on non-main ranks, matching the local save_pretrained_gguf lora branch, so only rank 0 converts/uploads. - compressed export VLM detection: require a vision_config or a ForVisionText2Text architecture; a bare *ForConditionalGeneration also matches text seq2seq models (T5/BART/Whisper) and is no longer treated as a VLM on its own. - GGUF GPU smoke test: drop SFTConfig(max_length=1024), which raises under newer TRL padding-free training; length enforcement is not needed here. * Add imatrix option to GGUF export, enabling IQ low-bit quants save_pretrained_gguf / push_to_hub_gguf gain imatrix_file: None -> no imatrix (unchanged) '/path' -> pass to llama-quantize --imatrix (a *.gguf_file is renamed to *.gguf) True -> download the upstream unsloth/<base>-GGUF imatrix (imatrix_unsloth.dat or .gguf_file), raising a clear error if none exists An importance matrix unlocks the IQ low-bit quants (iq2_xxs, iq4_xs, ...), which were hard disabled before. They are gated: requesting one without an imatrix raises a clear error. - _resolve_imatrix_file resolves path/True (PEFT base first, normalized via get_model_name, derives unsloth/<base>-GGUF, copies out of the HF cache before renaming *.gguf_file). - IMATRIX_QUANTS registry replaces the old commented-out IQ entries; save_to_gguf accepts a resolved imatrix and threads it into the quantize calls. - The --imatrix flag is emitted by unsloth_zoo's quantize_gguf (companion change). save.py fails fast with an upgrade hint if the installed unsloth_zoo lacks the imatrix kwarg. Tests: tests/saving/test_imatrix_export.py (CPU: resolution, repo derivation, IQ gate, --imatrix wiring) wired into CI; tests/saving/test_gguf_export_and_inference.py extended with GPU iq2_xxs/iq4_xs export + inference. Verified end to end on Llama-3.2-1B: imatrix auto-downloaded, iq2_xxs/iq4_xs exported and run via llama.cpp. Note: requires the companion unsloth_zoo quantize_gguf imatrix change. * Address imatrix/compressed review feedback: unsloth org GGUF repo, fail-fast, calibration split - imatrix auto-resolve (imatrix_file=True): derive the upstream repo as unsloth/<base>-GGUF instead of <org>/<base>-GGUF, so official bases (e.g. meta-llama/Llama-3.1-8B-Instruct) find the matching Unsloth GGUF imatrix repo rather than failing on a nonexistent meta-llama/...-GGUF. - Resolve/validate the imatrix before the 16-bit merge in save_pretrained_gguf, so a bad path or an unavailable upstream imatrix fails fast instead of after a long, multi-GB merge. - Compressed calibration: when a Hub dataset has no "train" split, resolve the first split name and slice it, instead of materializing the whole dataset just to take num_samples rows. Keeps the original materialize-then-subselect path as a last resort. Tests: add unsloth/<base>-GGUF mapping for an official base id, and create the imatrix file in the quantize_gguf flag test (quantize_gguf now validates the imatrix exists). --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-30 03:40:16 -07:00
Daniel Han	de3c745fab	Fix full finetuning precision on V100 / no-bf16 GPUs (#5880) ... --------- Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2026-06-29 18:35:23 -03:00
Vineeth Sai	677ec0cc20	Fix gpt-oss detection in save: config.architectures is a list, not a string (#6711)	2026-06-29 02:44:00 -03:00
Vineeth Sai	20266a59eb	Fix custom chat templates with a {system_message} placeholder (dead code in _change_system_message) (#6735) ... --------- Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2026-06-29 00:36:39 -03:00
Daniel Han	4c72e09480	Studio: stop handing CI/user secrets to downloaded llama.cpp binaries (#6696) ... * Studio: stop handing CI/user secrets to downloaded llama.cpp binaries The macOS prebuilt path installs llama.cpp from the unslothai/llama.cpp fork's latest (unpinned, mutable) release and then executes the downloaded llama-server / llama-quantize binaries during install-time validation. binary_env() built that child environment from a full os.environ.copy(), so a compromised or tampered prebuilt would inherit every secret in the process: HF_TOKEN and the workflow GitHub tokens in CI, and HF / cloud credentials for end users running install.sh / setup.sh. We publish prebuilts daily, so pinning a release tag is not workable. Instead, neutralise the impact: these binaries have no reason to read any token, so strip secret-bearing variables (exact names plus TOKEN/SECRET/PASSWORD/CREDENTIAL/PRIVATE_KEY/API_KEY markers) before handing the env to a downloaded binary. The installer's own GitHub and Hugging Face API calls read os.environ directly, so authentication and release-API rate limiting are unaffected; PATH, LD_LIBRARY_PATH, DYLD_LIBRARY_PATH and CUDA/ROCm vars are preserved. One change covers the install-time validation path for all six macOS workflows and end users. Follow-up (separate, sequenced): publish build-provenance attestations from the fork's prebuilt workflows and verify them in CI, so a forged release is rejected rather than merely starved of secrets. * Strip KUBECONFIG, SSH_AUTH_SOCK, and PASSPHRASE-marked vars from binary env Extend the deny-list per PR review: KUBECONFIG and SSH_AUTH_SOCK are credential pointers/capabilities a downloaded binary never needs, and a PASSPHRASE marker catches SSH_PASSPHRASE / GPG_PASSPHRASE. Tests updated. * Studio: also scrub proxy/index env vars and URL-embedded credentials before running prebuilt binaries * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope mlx-ci secrets to the install + download commands for PR #6696 Drop the ambient step-level env block and pass GH/GITHUB/HF tokens only on the installer and GGUF-download commands, so the directly invoked llama-quantize / llama-server smoke runs see no secrets. The installer still reads tokens from os.environ for the releases API and probe fetch. * Trim verbose comments around the secret-env scrubber for PR #6696 Comment-only: condense the block comments added across this PR. Logic unchanged (comment_tools.py check confirms code-only signature equal). * Redirect HOME / cache pointers to an empty dir for prebuilt binaries (PR #6696) Address Codex P2: stripping token env vars still let a tampered binary read on-disk token stores (~/.cache/huggingface/token, ~/.aws/credentials, ~/.config/gh) through $HOME and the cache/config pointers. Point HOME plus the HF / XDG / Windows home pointers at a single empty throwaway dir for the downloaded-binary env. Defense in depth: a binary resolving the real home via getpwuid is out of scope and needs OS sandboxing. * Close residual credential-probe gaps for PR #6696 Address the latest Codex review: - Strip token-only URL userinfo too (scheme://ghp_token@host), not just the user:pass form. - Redirect HOMEDRIVE/HOMEPATH alongside USERPROFILE so a Windows binary cannot reconstruct the real profile from %HOMEDRIVE%%HOMEPATH%. - Drop explicit credential-file pointers (NETRC, PIP_CONFIG_FILE, DOCKER_CONFIG, GIT_CONFIG_GLOBAL) that live outside HOME. - Probe ldd with a secret-free env: linux_runtime_dirs ran ldd on the untrusted prebuilt with the inherited os.environ, and ldd may execute the binary, so it could observe HF_TOKEN/GITHUB_TOKEN during the probe. Factored the shared scrub into secret_free_environ(). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Separate token-bearing install from binary smoke; drop CI command files (PR #6696) Address the two P1s in the latest review: - mlx-ci: GitHub bakes secrets into the run-script text, so inline token assignments in a step that later runs the prebuilt let a tampered binary read them from the script. Split into a token-bearing install + download step that never launches a binary, and a secret-free smoke step that runs llama-quantize / llama-server. - secret_free_environ now drops the GitHub Actions command files (GITHUB_ENV, GITHUB_PATH, GITHUB_OUTPUT, GITHUB_STEP_SUMMARY, BASH_ENV) and the smoke step unsets them, so a tampered prebuilt cannot inject PATH/env into the later token-bearing MLX steps. * Run the prebuilt smoke last, after all token-bearing steps (PR #6696) Address the P1 workspace-poisoning vector: even with no secrets in its env, a tampered prebuilt could edit the checkout or installed modules, and the later HF_TOKEN MLX steps would then execute that poisoned code on push builds. Move the prebuilt install + smoke to the end of the job so the untrusted binary runs after every token-bearing step, leaving nothing for it to corrupt. The MLX GGUF reload uses a source-built llama-cli, not this prebuilt, so nothing depends on the earlier position. * Trim comments around the secret-env scrubber and prebuilt CI steps (PR #6696) Comment-only: condense the security-rationale block comments and merge the duplicated prebuilt-step description in mlx-ci. Logic unchanged (comment_tools.py check confirms the code-only signature is equal; install suite still passes). * Authenticate the GGUF export release-API lookup with the read-only GITHUB_TOKEN (PR #6696) * Rename env scrubber off the secret-named identifier CodeQL flags as a clear-text sink (PR #6696) --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-27 05:21:05 -07:00
oobabooga	c8bcacc3fe	Fix fast_inference crash on ABI-broken vLLM: probe compiled extensions, not just import vllm (#6621) ... * Fix fast_inference crash on ABI-broken vLLM: force-load compiled extensions in the broken-vLLM probe * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Broaden broken-vLLM probe: catch non-libcudart .so failures and _moe_C_stable_libtorch * Revert stray reformat of the PDL fix log line * Trim verbose comments in the broken-vLLM probe * Drop non-existent vllm._moe_C_stable_libtorch from the broken-vLLM probe * Shorten comments in broken vLLM extension detection Condense the docstrings and inline comments for the lazy-loaded vLLM probe and the new regression test while keeping the rationale. Comments only, no code changes (verified with an AST signature check and the existing tests). --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-06-26 22:43:36 -07:00
Daniel Han	1fcd69e662	Harden flaky Studio CI: retry VS-hide rename and tolerate same-URL nav interrupt (#6713) ... Two intermittent Studio CI failures, both runner-environment flakes unrelated to test logic: Windows 'Studio install + inference without Visual Studio': the 'Hide Visual Studio + CMake' step renames C:\Program Files\Microsoft Visual Studio to simulate a host with no build tools. A background handle on a Program Files directory (Defender scan or an MSBuild node) makes Rename-Item intermittently fail with 'Access is denied', and $ErrorActionPreference = Stop turns that into a hard job failure. Wrap the VS and cmake renames in both Hide steps in a short Rename-WithRetry (6 tries, 3s apart) to ride out the transient lock. macOS 'Chat UI Tests': the re-login goto to /login can be interrupted by the SPA auth guard redirecting to the same /login URL, which Playwright reports as 'Navigation to .../login is interrupted by another navigation to .../login'. The goto already tolerated ERR_ABORTED; broaden it to also tolerate the same-URL interrupt (the password-field wait right after confirms we landed on /login), and add the same signature to the two Playwright flake-retry harnesses as a safety net for any other navigation. Validated: playwright_chat_ui.py parses + byte-compiles, both workflow YAMLs parse, bash -n on the retry harnesses, PowerShell AST parse on all pwsh steps, and a functional check of Rename-WithRetry (succeeds, and rethrows after exhausting retries).	2026-06-26 19:45:46 -07:00
James Dawdy	b693ed7c91	fix: wrap unprotected evaluate() calls with robust_evaluate() to handle navigation context loss (#6677) ... * fix: wrap unprotected evaluate() calls with robust_evaluate() to handle navigation context loss Fixes PR #5911 - Playwright UI test error: 'Execution context was destroyed' The test had several direct page.evaluate() and locator.evaluate() calls that weren't wrapped with robust_evaluate(), which retries when navigation destroys the execution context mid-operation. Changes: - Wrap picker_visible_text() evaluate in robust_evaluate() - Wrap _bubble_count() evaluate in robust_evaluate() - Wrap assistant text query in robust_evaluate() - Wrap theme_item click evaluation in robust_evaluate() - Wrap background color/theme query in robust_evaluate() This ensures all execution context losses from concurrent navigation are properly caught and retried with exponential backoff, preventing transient failures in the UI test suite. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: revert robust_evaluate on theme_item.evaluate per Codex review The theme_item.evaluate('el => el.click()') is side-effecting — retrying after a context loss could double-toggle the theme. It's already inside a 3-attempt try/except loop that handles click failures gracefully. The other 4 changes (all read-only queries) remain wrapped in robust_evaluate() since retrying them is safe. * fix: wrap remaining chat UI evaluate --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com>	2026-06-26 16:56:52 +01:00
Matt Van Horn	e9c6364e1e	feat: improve Unsloth Studio chat title generation quality (#6697) ... * feat: improve Unsloth Studio chat title generation quality * fix: address self-review (guard echoed role labels before punctuation stripping) * Address title generation review feedback Consolidate the echo guard into a single leading-label check (now also covering base and lora) and drop the post-punctuation duplicate that could never match a colon once punctuation is stripped. Swap the slice-based first-assistant lookup for an indexed find to avoid copying the messages array, and note the brace counter's assumptions in the test helper. --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: Daniel Han <unslothai@gmail.com>	2026-06-26 01:55:59 -07:00
Daniel Han	1396c01253	Fix offline checkpoint load/export: "tokenizer is weirdly not loaded" (#6554) ... * Fix offline checkpoint load/export failing with "tokenizer is weirdly not loaded" Loading a fine-tuned checkpoint with no internet (e.g. a Studio export) crashed with "Unsloth: The tokenizer is weirdly not loaded? Please check if there is one." For a LoRA adapter the loader reassigns model_name to the base model repo id and only keeps the local checkpoint dir as tokenizer_name when it contains tokenizer_config.json, tokenizer.json AND special_tokens_map.json. Modern tokenizers (e.g. Gemma) store special tokens inside tokenizer_config.json and omit special_tokens_map.json, so tokenizer_name fell back to the base repo id. The tokenizer/processor loads in vision.py then hit the Hub with no local_files_only, so with no network they failed (AutoProcessor) or hung for minutes (AutoTokenizer) even though every file was already cached. loader.py: keep the local checkpoint dir as tokenizer_name when it has a tokenizer config plus the actual tokenizer files (tokenizer.json / tokenizer.model / vocab files); special_tokens_map.json is no longer required. vision.py: compute an effective local_files_only (explicit kwarg plus the HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE env vars, mirroring loader.py and diffusion.py) and thread it through every AutoConfig, AutoProcessor, AutoTokenizer and the manual VLM processor fallback, including the hf_hub_download in that fallback (which now prefers a local file). When a load fails and no offline env var is set, retry against the local cache. The retry forces HF offline mode because local_files_only alone does not stop AutoProcessor / AutoTokenizer from issuing a /api/models request during class resolution. The final error now explains the offline/cache cause instead of the misleading "weirdly not loaded" message. studio export: probe Hub reachability once per checkpoint load and pass local_files_only when offline so exports use the local checkpoint dir / cache instead of hanging or crashing with no internet. Online behavior is unchanged: the new flags default to off and the retry only runs after a network related failure. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: safer offline forcing, cached fallback config, proxy-aware probe Follow-up to the offline checkpoint load fix, addressing review feedback: - vision.py: only flip the process-wide HF offline flag when offline is actually requested (local_files_only / env) or after a real network failure, never pre-emptively while we might be online. The flip is now guarded by a lock + depth counter so nested or concurrent windows restore the flag correctly (no stale value). - vision.py: guard the get_auto_processor fallback so a network error there returns None and the local-cache retry still runs instead of escaping. - vision.py: in the manual VLM processor fallback, read tokenizer_config.json via hf_hub_download(..., local_files_only=...) so a cached repo-id config is still resolved offline and the model-specific image/video tokens are restored. - studio export: make the reachability probe proxy aware (probe the configured HTTP(S) proxy egress, honour NO_PROXY, use the endpoint port) so a proxy-only setup is not wrongly marked offline; allow UNSLOTH_OFFLINE_PROBE=0 to disable. - studio export: run the audio/vision type-detection probes inside the forced-offline window when offline, so their config/tokenizer reads hit the local cache instead of waiting out connection timeouts. Online behavior remains unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: gate offline retry, safer tokenizer_name pop, skip audio net probe offline - vision.py: only force the process-wide HF offline flag on the tokenizer retry when offline was requested or the captured primary error is actually network related, so a permanent tokenizer error no longer toggles global offline mode for other concurrent loads. - loader.py: always pop tokenizer_name out of kwargs and let a caller-supplied value win, avoiding a "multiple values for keyword argument 'tokenizer_name'" TypeError when it is also passed explicitly downstream. - model_config.py / export.py: add local_files_only to detect_audio_type so the raw requests.get tokenizer_config fetch is skipped offline (it ignores the HF offline flag), and pass it from the export probe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: classify LocalEntryNotFoundError as offline-related huggingface_hub's LocalEntryNotFoundError subclasses FileNotFoundError, so the "not isinstance(cur, FileNotFoundError)" guard in _is_offline_related_error was swallowing it and it could never be recognised as offline, despite being listed in the network error types. It means "not in cache and the Hub is unreachable", which is genuinely offline. Capture the class into an isinstance-checkable tuple (empty, hence a no-op, if the import is unavailable) and exclude it from the FileNotFoundError guard, so a real offline failure now triggers the local-cache retry while a plain missing-file error still propagates. * Address review: require merges.txt for BPE, status-gate HTTP errors, isolate local-only audio cache - loader.py: a local dir with vocab.json but no merges.txt (and no tokenizer.json) is not a loadable BPE tokenizer, so do not treat it as self-sufficient; require merges.txt alongside vocab.json in both gate blocks, otherwise fall back to the base model tokenizer as before. - vision.py: _is_offline_related_error no longer buckets every HfHubHTTPError / requests HTTPError as offline. HTTP errors are judged by status code: only a transient 5xx triggers the forced local-cache retry, while 401/403 (auth/gated) and 404 (missing) propagate as the real error instead of being masked. Hard signals (connection/timeout/OfflineModeIsEnabled/LocalEntryNotFoundError) still classify as offline. - model_config.py: include local_files_only in the audio-detection cache key so a local-only (offline) negative result cannot be reused by a later online probe, which would otherwise route an audio model through the text loader until restart. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address re-review: fix studio test stubs, force offline env in probe window, drop redundant retry - studio/backend/tests/test_vision_cache.py: the three _detect_audio_from_tokenizer stubs were called with the new local_files_only kwarg and raised TypeError, failing Backend CI. Add local_files_only to the stub signatures and add a test that a local-only negative does not poison a later online audio probe. - export.py: the type-detection probe window now also sets HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE env vars (saved/restored), not just the in-process flag. transformers_version._load_config_json / _check_tokenizer_config_needs_v5 gate their urllib fetches on the env vars, and is_vision_model may spawn a subprocess that inherits os.environ but not the in-process flag; without the env vars a probe-detected offline export could still block on a network timeout. - vision.py: only retry the processor load when the first attempt was online and failed with a network error. When local_files_only was already requested the first attempt was forced offline, so the previous retry just repeated identical failing work before the last-resort path. - model_config.py: correct the _audio_detection_cache type annotation to the 3-tuple key (name, token_fingerprint, local_files_only). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: thread-safe probe-offline env window, clear error for local dir without config - export.py: guard the HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE mutation in _force_offline_probe_window with a lock + depth counter (mirrors _force_hf_offline), so concurrent / nested export probes only flip on first entry and restore on last exit. This prevents overlapping export requests from permanently poisoning those env vars or restoring a stale value. - vision.py: in the VLM processor fallback, when tokenizer_name is a local directory, read its tokenizer_config.json directly and raise a clear FileNotFoundError if it is absent, instead of handing the local path to hf_hub_download (which would treat it as a repo id and raise a confusing HFValidationError / RepositoryNotFoundError). hf_hub_download is now only used for actual repo ids. * Address review: classify raw socket.gaierror DNS failures as offline Add the platform-specific getaddrinfo / DNS-resolution wording to the offline detection list in _is_offline_related_error so a bare socket.gaierror (an OSError subclass) is recovered from the local cache: "Name or service not known" and "Temporary failure in name resolution" (Linux) and "nodename nor servname provided" (macOS). Genuine non-network OSErrors (disk full, permission denied) and plain FileNotFoundError still propagate. * Address review: retry degraded VLM offline, force offline for text export + patch-tokenizer fallback - vision.py: a degraded VLM processor (text-only, no image_processor) whose manual fallback fails offline used to be kept, so image inputs broke even with cached files. _construct_vlm_processor_fallback now returns its failure error; _acquire_processor surfaces it, and the caller retries forced-offline when the result is None OR a degraded VLM and the failure was network related, keeping the original result if the retry is not strictly better (never regress). The retry is still gated on an online first attempt + offline-related error so a permanent error never flips the global offline flag. - vision.py: wrap the patch_tokenizer except-branch AutoTokenizer.from_pretrained in the same forced-offline-on-network-error pattern as the primary / last-resort loads, so an offline export where patch_tokenizer raises does not hang or fail. - export.py: force HF offline around the two FastLanguageModel loads (text and SNAC) when the probe detected offline. Their text tokenizer path (load_correct_tokenizer -> AutoTokenizer) does not forward local_files_only, so without this a text export could still contact the Hub. Added a small _offline_window_if helper reused by the probe and load windows. * Consolidate offline loading into one entry-point decision Decide offline once per entry point instead of at every HF call site. The prior approach threaded local_files_only into ~15 scattered config / tokenizer / processor / weight loads, each wrapped in its own try-online, classify-error, retry-forced-offline dance, which is what kept surfacing "another call site you missed", "another error shape misclassified", and global-flag thread-safety in review. FastLanguageModel / FastModel / FastBaseModel.from_pretrained now share an @_offline_aware_load decorator: when offline (explicit local_files_only kwarg or HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE env) it sets local_files_only and runs the whole load inside one _force_hf_offline() window so every nested HF call inherits it; when online it runs normally and, only if the load fails with a genuinely network-related error, retries once forced-offline. The online path is unchanged (no probe added) and 401 / 403 / 404 / permanent errors still propagate. Centralise the offline helpers in loader_utils.py as the single source of truth (shared by loader.py, re-exported from vision.py, and reused by the Studio exporter): - _force_hf_offline now sets the HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE env vars AND the in-process huggingface_hub / transformers flags, refcounted under one lock so nested / concurrent windows restore correctly. Setting the env vars covers env-gated urllib probes and spawned subprocesses too. - _get_effective_local_files_only, _is_offline_related_error (unchanged classifier, retains the 5xx-vs-4xx, LocalEntryNotFound and gaierror handling), _offline_aware_load, and _resolve_checkpoint_tokenizer_name. loader.py: wrap both entry points; drop the two duplicated env-var fallback blocks and the two byte-identical local-tokenizer-gate blocks (now _resolve_checkpoint_tokenizer_name). vision.py: drop the per-site force_offline params and the three retry gates (processor, patch_tokenizer fallback, last-resort). They now just surface the underlying error so the single entry-point safety net retries forced-offline. A network fallback error now takes precedence over a permanent primary error so the offline retry still fires when the manual VLM fallback needs cached repo files. studio/backend export.py: reuse the unified core _force_hf_offline (env + flags) and drop the duplicate probe-window primitive; the snac / text branches no longer need their own window. model_config.py: also gate the raw requests.get audio fallback on the HF offline env vars so it is covered even without the kwarg. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address 10-reviewer P1 findings: vision cache split, PEFT offline, retry OOM Split the Studio vision-detection cache by local_files_only, mirroring the audio cache fix. is_vision_model / _is_vision_model_uncached / _raw_config_has_vision_config / load_model_config now thread local_files_only, the cache key includes it, and the exporter passes it. Offline detection also skips the transformers-5 network subprocess and stays on the local cache, so an offline negative can no longer be keyed under the online entry and poison a later online probe. Adds a regression test mirroring the audio poison test. Forward local_files_only to both PeftModel.from_pretrained adapter-attach sites in loader.py so a cached remote LoRA adapter resolves from the local cache under explicit local-only / offline loads (defence-in-depth alongside the forced-offline window). _offline_aware_load: run the forced-offline retry OUTSIDE the except block and collect + empty the device cache first. An except-scoped exception keeps its __traceback__, which pins the failed attempt's frame locals (a partially loaded model) until the block exits; loading the model again while that copy is still alive could OOM a large VLM. Letting the except block close drops the traceback so the partial load is freed before the retry reallocates. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address Codex review: env-offline cache key + rebuild HF sessions in offline window Key the Studio audio and vision detection caches on the EFFECTIVE offline state (local_files_only OR the HF offline env vars), not just the kwarg. detect_audio_type and is_vision_model both skip the remote fetch / network subprocess when HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE is set even with the default local_files_only=False, so the result reflects offline; storing it under the online (False) key let an env-offline negative poison a later online lookup once the env var was cleared. Both now compute effective_offline once and use it for the cache key and the downstream call. Adds a regression test for the env-offline dimension. _force_hf_offline now rebuilds huggingface_hub's cached sessions on enter and exit (best-effort _reset_hf_sessions). On hub 0.x the offline adapter is baked into the per-thread requests.Session at creation, so flipping the constant alone leaves an already-cached online session able to hit the network inside the window (and an offline one stuck offline after restore); resetting forces the next get_session() to match the current flag. On hub 1.x offline is checked dynamically per request, so reset_sessions does not exist and the helper is a safe no-op. The third review point (release the failed load before retrying) was already fixed in af0f58a: the forced-offline retry now runs outside the except block and frees the device cache first, so the failed attempt's traceback-pinned partial model is released before the retry reallocates. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align Studio _env_offline parsing with the canonical offline helper model_config._env_offline gates the raw requests.get tokenizer-config fallback in detect_audio_type and the audio/vision detection cache keys, but it only accepted unstripped "1"/"true"/"yes". unsloth's offline helpers (loader_utils._env_says_offline and the from_pretrained env fallback) accept the canonical set {1,true,yes,on} after strip + lowercase, so HF_HUB_OFFLINE=on or HF_HUB_OFFLINE=" 1 " was treated as offline by the loaders but online here, leaving the raw network fetch reachable while "offline". Use the same strip + lowercase {1,true,yes,on} set. Adds parsing tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix lint: drop dead offline-helper re-exports from vision.py The import-hoist verifier (scripts/verify_import_hoist.py) flagged vision.py's re-export block as HOISTED-IMPORT-UNUSED blockers: it imported eight offline helpers from loader_utils but only used three internally (_get_effective_local_files_only, _is_offline_related_error, _offline_aware_load). The other five were imported purely to preserve `from unsloth.models.vision import X`, but nothing imports four of them from vision, and loader.py already imports _resolve_checkpoint_tokenizer_name straight from loader_utils. Import only the three names vision.py actually uses, and point the Studio exporter at the canonical source (from unsloth.models.loader_utils import _force_hf_offline) instead of re-exporting it through vision. loader_utils stays the single source of truth; no behaviour change. * Address Opus review: chain probe errors, unify env-offline, status-less HTTP Chain the original AutoConfig/PeftConfig probe exception into the combined RuntimeError in both FastLanguageModel.from_pretrained and FastModel.from_pretrained (`raise RuntimeError(combined_error) from (autoconfig_exc or peft_exc)`). The probes caught every Exception and stringified it, so the re-raised RuntimeError had no __cause__/__context__ and _is_offline_related_error could not classify it -- the network-down-but-cached auto-retry never fired for these entry points. With the cause chained, the decorator sees a ConnectionError/LocalEntryNotFoundError/5xx and retries forced-offline from cache; a permanent cause (404 / bad config) is still not offline-classified and propagates without a wasted retry. Unify the third offline-env parser: studio/backend/utils/transformers_version._env_offline now uses the canonical {1,true,yes,on} + strip + lowercase set (matching loader_utils._env_says_offline and model_config._env_offline), so HF_HUB_OFFLINE=on or " 1 " no longer leaks the direct urllib metadata fetches to the network. _is_offline_related_error: a status-less HTTP error (no response / unparseable code) now falls back to the network-wording check instead of being dropped, so a transient HTTP failure with clear "couldn't connect" wording is treated as offline. HTTP errors with a real status code still decide by code (4xx propagates, 5xx is offline). * Condense offline-loading code comments, drop dead helper, dedupe import for PR #6554 * Add unit tests for offline-loading helpers for PR #6554 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard load cleanup with try/finally and add retry-contract tests for PR #6554 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gc.collect retry-step test for PR #6554 * Tighten offline-loading comments and docstrings for PR #6554 * Raise the both-config-failed error before model-type lookup so offline retry fires for PR #6554 * Prefer offline cause for retry and bound export reachability probe for PR #6554 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Skip remote mapper while offline, harden text-load cleanup, and stop stacked offline retries for PR #6554 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Surface VLM fallback offline errors, probe offline before export version activation, and restore progress bars across retries for PR #6554 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore offline env after export version activation so the persistent worker re-decides per load for PR #6554 * Classify socket.gaierror and urllib URLError as offline by type for PR #6554 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Probe offline around export load preflights and never offline-retry TLS failures for PR #6554 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Force in-process offline for export preflights, verify proxy egress in probe, and skip caching offline version negatives for PR #6554 * Snapshot offline constants before forcing env and require local processor files for VLM checkpoints for PR #6554 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-25 23:16:53 -07:00
Lee Jackson	c873ef052d	Studio: prompt variables into prompt editor (#6434) ... * add custom and system variable feature in system prompt * missing function use * feat: prompt variables editor ux Co-authored-by: CodeMan62 <175127021+CodeMan62@users.noreply.github.com> * fix: guard prompt variable defaults * refine prompt variables editor layout * fix: harden prompt variable substitution * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refine prompt variables editor copy and built-in token labels --------- Co-authored-by: CodeMan62 <sharmahimanshu15082007@gmail.com> Co-authored-by: CodeMan62 <175127021+CodeMan62@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Leo Borcherding <borchborchmail@gmail.com>	2026-06-25 18:09:16 +01:00
Daniel Han	4929c5f769	Keep pad-named pad_tokens (e.g. <|vision_pad|>); fix Qwen3-Base load crash (#6652) ... * Keep pad-named pad_tokens; defer pad repair to shared unsloth_zoo.pad_token A pad-named token (e.g. <|vision_pad|>) is a valid pad. The narrow fallback that stripped vision pad tokens on text-only models is now a no-op; the active path delegates to the shared fix_pad_token in unsloth_zoo, which keeps pad-named tokens and only heals missing / eos-collision / out-of-range pads. This fixes the Qwen3-4B-Base load crash (its config ships pad_token=<|vision_pad|>): the old swap could not find a safe text pad (eos is <|endoftext|>, no unk_token) and left the tokenizer broken. Removes the unused _VISION_PAD_TOKENS / _SAFE_TEXT_PAD_TOKENS sets. Tests updated. Pairs with unslothai/unsloth-zoo#831. * Remove _fix_vision_pad_token; inline the no-op fallback A pad-named token (e.g. <|vision_pad|>) is a valid pad, so the old vision-pad swap helper has no purpose. _fix_pad_token now returns the tokenizer unchanged when the shared unsloth_zoo.pad_token module is unavailable, instead of routing through a no-op helper. Test WANTED set updated.	2026-06-25 04:41:09 -07:00
Daniel Han	f436d204f6	Installer: make UV_OVERRIDE space-safe on Apple Silicon (#6503) (#6639) ... * Installer: make UV_OVERRIDE space-safe on Apple Silicon (#6503) On Apple Silicon, install.sh exports UV_OVERRIDE pointing at the bundled overrides-darwin-arm64.txt. uv splits UV_OVERRIDE on whitespace, so a repo cloned under a path containing a space (e.g. /Users/me/Open Source/unsloth) truncates the value and every later uv call aborts with 'error: File not found: <truncated>' (the PyTorch install step in #6503). Copy the overrides file into a space-free temp dir and point uv at the copy when the path contains a space, mirroring the macOS/Linux handling already merged for the Python installer in #6534. The temp dir is removed in the exit trap, and the code falls back to the original path when no space-free temp dir is available, so the no-space and non-macOS paths are unchanged. Adds tests/sh/test_install_uv_override_space.sh, which extracts and runs the install.sh hardening block and checks the spaced, no-space, and spaced-TMPDIR fallback cases. * Installer: match all whitespace (not just spaces) in UV_OVERRIDE handling uv splits UV_OVERRIDE on any whitespace, so use the POSIX class *[[:space:]]* rather than a literal space in install.sh (catches tabs and newlines in the path too) and the matching test assertions. Use the portable awk bracket expression [$] instead of \$ in the extraction so the test runs the same under BSD awk (macOS) and GNU awk (Linux). Adds a tab-in-path case. * Installer: clear _UV_OVERRIDE_TMPDIR before the exit trap The exit trap rm -rf's _UV_OVERRIDE_TMPDIR. Initialize it to empty before registering the trap so an inherited environment value can never be removed; only a temp dir this script creates (Apple Silicon, spaced path) is cleaned. Adds a structural test asserting the init precedes the trap. * Run the install.sh UV_OVERRIDE space test in CI via a pytest wrapper The Shell installer tests job uses a fixed script list (not tests/run_all.sh), so the new shell test would not run on PRs. Add a pytest wrapper under tests/python/ that invokes it; the auto-discovered repo CPU test job collects tests/python/ and so executes the Apple Silicon spaced-path regression. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-24 17:34:18 -07:00
Daniel Han	bd2438ea65	Verify DiffusionGemma visual-server binary against approved checksums (#6635) ... ensure_diffusion_visual_server() downloaded the visual-server release asset with the unverified download_file() and marked it executable, bypassing the approved-checksum manifest that gates every other prebuilt llama.cpp artifact. The backend later auto-discovers that binary and launches it through DG_VISUAL_BIN, so a compromised or substituted release asset could place attacker-controlled native code in the install tree and have it executed under the Studio user. Require the matched asset to be present in the approved checksum manifest and download it through download_file_verified() with the published sha256. A name-matching asset that is absent from the manifest is refused rather than executed. Add regression tests covering the verified-download path and the refusal of an unapproved asset.	2026-06-24 06:37:41 -07:00
Daniel Han	c7c353d740	Pin isolated Node.js installer to committed sha256 digests (#6625) ... * Pin isolated Node.js installer to committed sha256 digests The isolated Node installer verified each downloaded archive only against SHASUMS256.txt fetched from the same nodejs.org origin as the archive, so a compromised CDN or TLS path could serve a malicious archive plus a matching checksum and gain code execution when the extracted node is run during the npm floor check and version probe. Anchor trust in studio/node_prebuilt_pins.json, a committed manifest of per-arch sha256 digests, and verify archives against it. The default channel installs the pinned version and never fetches the remote SHASUMS. Unpinned lts, latest, or explicit versions fail closed via UnpinnedNodeRefused unless UNSLOTH_NODE_ALLOW_UNVERIFIED=1, and the refusal is not swallowed by the keep-existing-on-transient-failure path. Ship the manifest in package-data. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review nits on the pinned Node installer - Drop the unused npm_min_major field from node_prebuilt_pins.json; the floor is the NPM_MIN_MAJOR module constant and the dead field could silently drift. - Reword the unpinned-refusal message so it does not tell a user already on the default to install it, and point the "add a pin" hint at the exact asset. - Decode the opt-in SHASUMS body with errors="replace" so a non-UTF8 response yields a clean PrebuiltFallback instead of an uncaught UnicodeDecodeError. - Tests: assert the refusal message (guards the main() catch order, not just the exit code), cover malformed-manifest parsing, and drive the opt-in remote-SHASUMS path end to end through install_prebuilt. * Tighten comments in the pinned Node installer Collapse multi-line rationale comments to single lines, drop docstrings on the obvious internal helpers (load_pins, pinned_sha256), and shorten the manifest note. Comments/docstrings only; verified code-unchanged via AST comparison. * Address Codex review: verify pins on existing installs; tomllib fallback - existing_install_matches now takes an expected_sha and the short-circuit passes the committed pin, so a version-matching but non-pinned or tampered install (e.g. from the old remote-SHASUMS path) is re-verified instead of kept. An unpinned target without opt-in no longer short-circuits on an existing install; it falls through to the UnpinnedNodeRefused fail-closed path. - The package-data test uses pytest.importorskip(tomllib/tomli) so it does not ModuleNotFoundError on the supported 3.9/3.10 interpreters. * Make the transient-failure keep-existing path pin-aware The previous commit added the pinned-digest check to the existing-install short-circuit but not to the post-download-failure fallback, which still kept any runnable same-version install via existing_install_usable(). A same-version install whose recorded sha256 is not the pin could therefore be kept on a transient download failure, the exact artifact the short-circuit rejects. Refuse to keep a same-version pin-mismatched install there too; a different usable version is still kept for offline resilience. * Bump pinned default Node to the current 24 LTS (24.18.0) Node 24 LTS moved to 24.18.0; since the default channel now resolves straight to the manifest, a frozen 24.17.0 would downgrade fresh installs and make UNSLOTH_NODE_VERSION=lts refuse the current LTS as unpinned. Update default_version and all six per-arch digests (verified against the official SHASUMS256.txt), and point the test INDEX/short-circuit fixtures at the new LTS. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-24 05:47:58 -07:00
Saicharan Ramineni	9d53656614	Make _uv_safe_path space-safe on macOS/Linux (#6503) (#6534) ... * Copy uv `-c`/`-r` paths to a space-free temp dir on macOS/Linux uv 0.11.x truncates a constraints/requirements path passed via `-c`/`-r` at the first space, so `unsloth studio` setup from a repo cloned under a path containing a space (e.g. `/Users/me/Open Source/unsloth`) fails with: error: File not found: `/Users/me/Open` _uv_safe_path() already worked around this on Windows via the 8.3 short path but returned the space-containing path unchanged on macOS/Linux, which have no 8.3 equivalent. Extend it to copy the (small, flat) constraints/requirements file into a space-free temp dir and hand uv the copy; the temp dirs are removed at process exit. Falls back to the original path on any error, so it is never worse than before. Refs unslothai/unsloth#6503 * Route UV_OVERRIDE through _uv_safe_path and fix temp-dir leak (#6503) The -c/-r fix did not cover UV_OVERRIDE, which uv also truncates at the first space. On Apple Silicon the overrides file is handed to uv via UV_OVERRIDE at install time (install_python_stack.py) and during the MLX self-heal (utils.mlx_repair), so a repo under a path containing a space still broke every uv call there. Move _uv_safe_path into backend.utils.uv_path_safety so both sites share it, and route UV_OVERRIDE through it. Also stop leaking the temp dir when shutil.copyfile fails after mkdtemp, and add tests for the UV_OVERRIDE channel, the TMPDIR-with-space fallback, the atexit cleanup, and the no-leak path. --------- Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-06-24 04:02:24 -07:00
Daniel Han	7a25266203	Fix _SameTaskStreamingResponse disconnect test bypassing __init__ (#6627) ... * Fix _SameTaskStreamingResponse disconnect test bypassing __init__ test_same_task_response_closes_body_iterator_on_send_disconnect builds the response via __new__ to skip Starlette's __init__, then wires body_iterator, background, and stream_response by hand. It never set _unstarted_cleanup, so the disconnect-before-first-chunk branch of __call__ raised AttributeError instead of ClientDisconnect, failing the Backend CI "Repo tests (CPU)" job on main. Set response._unstarted_cleanup = None in the manual construction, matching the default __init__ assigns. * Shorten the _unstarted_cleanup comment to one line	2026-06-24 03:47:35 -07:00
Vineeth Sai	d42256a5c5	Fix construct_chat_template leaking {INPUT}/{OUTPUT} sentinel into the chat template (#6531) ... * Fix construct_chat_template leaking {INPUT}/{OUTPUT} sentinel into the template In construct_chat_template's inner process() helper, the branch handling a section that starts with the {INPUT}/{OUTPUT} sentinel sliced the part from part.find(which) (which is 0 in that branch), so the literal sentinel was re-included in the generated Jinja chat template. The endswith branch already slices correctly with part[:part.find(which)]; this slices past the sentinel with part[len(which):], so a template whose input or output section begins with the sentinel (for example a user turn that starts with {INPUT}) renders correctly instead of emitting a literal {INPUT}/{OUTPUT}. Added a regression test covering {INPUT}-leading and {OUTPUT}-leading sections. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-24 00:40:53 -07:00
Vineeth Sai	e3c7d4dc4c	Match IGNORED_TOKENIZER_NAMES case-insensitively (#6620)	2026-06-23 19:40:50 -03:00
Vineeth Sai	7dc0857267	Fix SyntheticDataKit.chunk_data dropping single-chunk documents (#6595) ... * Fix SyntheticDataKit.chunk_data dropping single-chunk documents chunk_data turns the n boundary points from np.linspace into n-1 ranges via the boundaries[:-1] / [1:] pairing. When a document fits in a single chunk (n_chunks == 1) that produces zero ranges, so the loop writes no files and the whole document is silently dropped. Emit the full [0, length] range when n_chunks <= 1; the multi-chunk path is unchanged. Added a regression test covering the single-chunk and multi-chunk cases. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * chunk_data: emit nothing for an empty document (no empty chunk file) Addresses review feedback: when the input document is empty (length == 0), return no chunks instead of writing a single empty chunk file. Added a regression test for the empty-document case. * chunk_data: reject overlap >= chunk size (non-positive stride) Per review feedback: when overlap >= max_tokens the chunk stride is non-positive, which would divide by zero or silently emit one oversized chunk. Raise a clear RuntimeError for that unusable configuration. Added a regression test. * Broaden single-chunk guard to length <= max_tokens (also fixes sub-overlap docs); expand tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2026-06-23 16:39:38 -03:00
oobabooga	1cc785e5a0	Studio: remove OpenEnv and other unused packages (#6585) ... * Studio: drop OpenEnv and unused ExecuTorch/open_spiel install deps * Studio: drop 8 more unused install deps from extras * Studio: restore tomli<3.11 for kernels; tidy dep-cleanup comments and tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: refresh scan-packages baseline for scipy _external + unsloth-zoo tests scipy moved its vendored array_api_compat from scipy/_lib to scipy/_external, so the four allowlisted array_api_compat __init__.py entries stopped matching and resurfaced as unsuppressed CRITICAL "Downloads and executes remote code" findings on all three pip scan-packages shards (extras, hf-stack, studio). Add the _external paths next to the existing _lib ones so both scipy layouts stay covered. Allowlist two unsloth-zoo test-file false positives now present in the hf-stack shard: tests/test_mlx_save_export_regressions.py (writes to /tmp dropper) and tests/test_mlx_trainer_internals.py (obfuscation plus exec/eval). Drop nine stale entries for packages removed from the Studio requirements and no longer in any shard closure (evaluate, pytest, hypothesis, kgb, langid), confirmed absent via with-deps resolution of all three shards. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-06-23 07:20:47 -07:00
Daniel Han	b91cdc8793	Fix Qwen3 NaN loss: delegate pad_token repair to shared unsloth_zoo.pad_token (#6524) ... * Fix Qwen3 NaN: self-heal vision pad_token in load_correct_tokenizer Text-only Qwen3 (and Qwen2.5) models share Qwen3-VL's vocab, so their Hub tokenizer configs ship <|vision_pad|> as pad_token. Padding text-only training with a vision token corrupts attention/loss and produces NaN losses and gradients on affected stacks. patch_tokenizer already heals this, but only when a model with config is passed. The standalone load_correct_tokenizer path (and custom training loops) still returned <|vision_pad|>. This adds a model-independent guard in load_correct_tokenizer that replaces a vision pad_token on text-only tokenizers with the first safe text token (<|endoftext|>, <pad>, [PAD], <unk>), falling back to eos_token only if it differs from pad_token. The result now matches upstream Qwen configs (pad_token <|endoftext|>, id 151643) with no new token added. Vision processors (image_processor present) and non-vision pad tokens (Llama, Qwen2) are left untouched. Fixes #3155 * Format pad_token helper for ruff kwarg-spacing hook (pre-commit) * Harden vision pad_token fix: drop unk candidate, guard get_vocab, skip vision eos * Tighten code comments (no logic change) * Delegate pad_token fix to shared unsloth_zoo.pad_token Generalize the narrow _fix_vision_pad_token by delegating to unsloth_zoo's shared fix_pad_token (single source of truth, AGPL-3.0), which scans the reserved-token families instead of only the vision-pad case. A guarded import keeps this working against an older unsloth_zoo that has not shipped the module yet: on ImportError it falls back to _fix_vision_pad_token. allow_add=False is passed so the early load_correct_tokenizer call stays side-effect free (no model here to resize embeddings); the later model-aware patch_tokenizer call finishes the job and is idempotent. Adds tests/python/test_pad_token_fix.py covering both dispatch paths offline.	2026-06-23 06:29:55 -07:00
Daniel Han	a963ec92e7	Fix CPOTrainer crash with multimodal processors (Gemma 4) (#6522) ... * Fix CPOTrainer crash with multimodal processors CPOTrainer shares build_tokenized_answer/tokenize_row and __init__ with ORPOTrainer, but the ORPO replacement functions that route tokenization through the underlying text tokenizer and resolve pad_token_id were only registered for orpo_trainer. With a multimodal processing class (e.g. Gemma4Processor) the positional self.processing_class(prompt, ...) call binds prompt to images=, leaving text=None and raising TypeError: 'NoneType' object is not subscriptable. Register the existing orpo_trainer_text_tokenizer and orpo_trainer_processor_pad_token under cpo_trainer as well so CPO/SimPO fine-tuning of multimodal models works. No change for plain tokenizers. * Add CPO processor tokenizer regression test Static, CPU-only checks that cpo_trainer registers the same orpo_trainer_text_tokenizer and orpo_trainer_processor_pad_token rewriters as orpo_trainer, and that the rewriter drops the broken positional self.processing_class(prompt, ...) call. Guards against issue #4952 regressing. * Format CPO test assert for ruff line length (pre-commit) * Bind CPO __init__ pad/eos token reads to underlying tokenizer TRL 0.28+ CPOTrainer.__init__ reads bare processing_class.pad_token and processing_class.eos_token before pad_token_id, which raises AttributeError for multimodal processors (e.g. Gemma) where those live on .tokenizer. Extend orpo_trainer_processor_pad_token to route that block through the underlying tokenizer, and add a regression test. * Tighten code comments (no logic change) * Make CPO/ORPO rewriters reach the trainer on TRL 1.x TRL 1.x moved CPOTrainer and ORPOTrainer out of trl.trainer into trl.experimental.<algo> and dropped the trl.trainer.<algo>_trainer shim that older TRL (0.26 - 0.28) kept. patch_trl_rl_trainers() discovers trainers via dir(trl.trainer), so on TRL 1.x cpo_trainer and orpo_trainer are never found and the multimodal-processor tokenization fix (#4952) silently stops applying, even though the rewriters themselves still match the source. Re-expose experimental-only trainers that Unsloth has rewriters for (RL_FUNCTIONS keys) under trl.trainer before discovery, so the existing patch machinery and its thin-wrapper resolution work unchanged. The alias is a no-op on older TRL where trl.trainer.<algo>_trainer already exists. Also rebind the patched Trainer/Config into every already-imported trl.* module that holds the original class so the fix is visible at the experimental import site (from trl.experimental.cpo import CPOTrainer), not only via trl.trainer. Verified on transformers 4.57.6 + trl 0.22.2, transformers 4.57.6 + trl 0.27.1, and transformers 5.12.1 + trl 1.6.0: CPOTrainer with a multimodal processor tokenizes through the underlying text tokenizer with no crash on all three, and the SFT/GRPO/DPO patch paths are unchanged. * Format for ruff (pre-commit) * Simplify CPO fix to mirror ORPO registrations (#4952) Register the existing ORPO row-tokenizer/pad-token rewriters for cpo_trainer. Under the trl<=0.24.0 pin CPOTrainer lives in trl.trainer.cpo_trainer (found by dir(trl.trainer)), shares ORPO's build_tokenized_answer and uses processing_class.pad_token_id, so the two registrations are sufficient. Drop the trl 1.x experimental aliasing/rebind machinery in rl.py and the bare-pad_token rewriter: trl 1.x (CPO in trl.experimental) and the bare pad_token pattern (trl>=0.28) are not installable under the pin. * CPO: route bare pad_token/eos_token default through inner tokenizer TRL 1.x CPO/ORPO __init__ (the trl.experimental source unsloth resolves on TRL 0.26+) defaults processing_class.pad_token from processing_class.eos_token before tokenizing. Multimodal processors (Gemma3/Gemma4 Processor) expose those attributes on .tokenizer, not on the processor, so that bare access raises AttributeError during __init__ even with the pad_token_id fallback registered. Extend orpo_trainer_processor_pad_token to rewrite that defaulting block to run on the inner tokenizer. The pinned TRL range (<=0.24.0) has no such block, so the regex is a no-op there and only the existing pad_token_id fallback applies. Verified the rewrite against the real trl 1.6.0 experimental CPOTrainer.__init__ (bare access removed, result compiles, a processor without pad_token no longer raises) and added offline regression tests for both the rewrite and its no-op. --------- Co-authored-by: Daniel Han <michaelhan2050@gmail.com>	2026-06-23 06:29:22 -07:00
Wasim Yousef Said	37166efcfc	Fix Gemma 4 GGUF OpenAI API streams (#6476) ... * Fix Gemma 4 GGUF OpenAI API streams * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Avoid duplicate Responses stream disconnect watcher * Keep reasoning-only Responses output hidden * Address Gemma stream review comments * Avoid Responses stream task-group cleanup * Harden OpenAI chat completion streams * Address OpenAI stream review issues * Clean up Studio OpenAI stream helpers * Fix Studio passthrough cold stream timeout * Fix tool parser compatibility exports lint * Preserve audio stream disconnect cancellation * Avoid synthetic finish after passthrough errors * Address stream cleanup and Gemma parser reviews * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gemma 4: parse bare-string tool args and keep safetensors tools for native <|tool_call> - Quote bare unquoted string values in Gemma native tool-call args (e.g. {location:Tokyo,unit:celsius}) so they parse; JSON scalars stay typed. - Stop _detect_safetensors_features from suppressing supports_tools for templates that emit Gemma native <|tool_call>, which the shared parser now reads. - Add tests for both. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden Gemma tool-call parsing and stream-error detection Address three issues in the Gemma-native tool-call path: - _quote_gemma_object_keys stopped a bare (unquoted) string value at the first comma, so an argument like `location:New York, NY` was split mid-value and the synthesized JSON failed to parse, dropping the whole tool call. A bare value now ends only at `}` or a comma that begins the next `key:` pair. - parse_tool_calls_from_text scanned the entire response for Gemma markers even inside a tool call already parsed from a `<tool_call>{...}` JSON block, so a marker-like string inside an argument (data) was promoted to a second, unintended tool call. Matches inside an already-consumed call span are now skipped. - _openai_passthrough_stream relied on _monitor_openai_sse_line to flag a stream error, which returns early when monitor_id is None (skip_api_monitor), so an upstream error chunk left saw_stream_error unset and the synthetic-finish guard emitted a successful finish_reason after a failed stream. Error chunks are now detected independently of API monitoring. Adds tests/test_gemma_tool_parse_edge_cases.py covering the comma and marker-injection cases. * Emit the terminal finish_reason chunk in GGUF streams The OpenAI chat-completions GGUF tool stream and plain stream both built a final ChatCompletionChunk carrying finish_reason but never yielded it, so clients received the optional usage chunk and [DONE] with no chunk carrying finish_reason. OpenAI-compatible consumers rely on that terminal choice to distinguish stop/length/tool_calls. Yield it before the usage chunk and [DONE], matching the other streaming paths. * Parse tool calls in document order and skip nested markers both ways Unify the JSON- and Gemma-format tool-call passes into a single position-ordered scan: - Calls are now emitted in byte order across both formats, so a mixed output like `<|tool_call>call:create{...}<tool_call|> ... <tool_call> {"name":"read",...}</tool_call>` executes create before read, matching the order they appear in (tools run in returned order). - A candidate that starts inside an already-accepted call's span is skipped, in both directions: a JSON marker inside a Gemma argument and a Gemma marker inside a JSON argument are treated as data, not promoted to a second executable tool call. Extends tests/test_gemma_tool_parse_edge_cases.py with the ordering and JSON-in-Gemma nesting cases. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Quote bare Gemma array elements; order finish before trailing usage - _quote_gemma_object_keys skipped array values, so a Gemma call with a bare-string array argument like labels:[bug,ui] produced invalid JSON and the whole tool call was dropped. Array values are now scanned and bare string elements quoted, while numbers, quoted strings, and JSON literals are preserved. - In the OpenAI passthrough stream, a trailing usage-only chunk (stream_options.include_usage) that arrived before any finish chunk was relayed before the synthetic finish, producing usage -> finish -> [DONE]. Emit the synthetic finish before that usage chunk so the order matches the other streams (finish -> usage -> [DONE]). Extends tests/test_gemma_tool_parse_edge_cases.py with the bare-array cases. * Harden Gemma array parsing, XML-parameter guard, and stream teardown Address five review findings on the Gemma tool-call and OpenAI passthrough streaming paths: - parse_tool_calls_from_text collected JSON and Gemma markers without the _inside_open_parameter guard, so a marker embedded in an existing <function=...><parameter=...> value was promoted to a separate tool call. Candidates that start inside an open XML parameter are now skipped, matching the guard the XML-style parser already applies. - _quote_gemma_array_elements preserved array elements starting with { or [ verbatim, so an array of objects (items:[{path:a}]) or a nested array failed json.loads and the whole call was dropped. Object and nested-array elements are now normalised recursively. - _openai_passthrough_stream synthesized a finish chunk before a trailing usage-only chunk and set saw_finish_reason, which made the EOF guard skip the [DONE] sentinel. The EOF path now emits [DONE] whenever the upstream omitted it, even after a finish chunk was already synthesized. - /generate/stream drove generation through asyncio.to_thread with no disconnect watcher, so a client disconnect during a long generation went unnoticed until the next send. It now runs _await_disconnect_then_cancel against the request, matching the other local streaming endpoints. - _SameTaskStreamingResponse closed the body iterator with aclose() on a send-side disconnect, raising GeneratorExit so the generators' cancellation handlers (which finish the api_monitor entry) never ran. It now throws CancelledError, falling back to aclose() when athrow is unavailable. Extends tests/test_gemma_tool_parse_edge_cases.py with array-of-objects, nested-array, and marker-inside-XML-parameter cases. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Watch disconnects on Anthropic streams; keep timestamps in Gemma values Two follow-ups on the streaming and tool-parse paths: - _anthropic_tool_stream and _anthropic_plain_stream drove generation through asyncio.to_thread(next, gen, ...) and only polled is_disconnected() between events, so a client disconnect during prefill or a long generation/tool step held the decode slot until the next event or a failed send. Both now run the _await_disconnect_then_cancel watcher used by the other local streams, stop it in finally, and break promptly when cancel_event is set. - _GEMMA_NEXT_KEY_RE treated any comma followed by word-chars-then-colon as the next key, so a bare value such as "meet at 10:00, 11:00 tomorrow" was split into bogus keys. The next-key token must now be identifier-shaped (start with a letter or underscore), so a comma before a timestamp, ratio, or other numeric-then-colon text stays part of the value. Adds a timestamp-in-bare-value regression test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard nested markers, reset on disconnect, clean unstarted streams Three follow-ups on the tool-parse and streaming paths: - parse_tool_calls_from_text only skipped markers that fell inside a span it had already parsed successfully, so when an unquoted Gemma argument contained a literal marker (code:<|tool_call>call:terminal{...}<tool_call|>) the outer object failed to normalize, its span was never recorded, and the inner marker was promoted to a standalone terminal call. Candidates nested inside any other candidate's brace span are now skipped regardless of whether the enclosing candidate parsed, so a marker in malformed outer data is never executed. - /generate/stream skipped backend.reset_generation_state() when the disconnect watcher set cancel_event between chunks: the loop broke and the finally's reset is guarded on cancel_event being unset. A subprocess backend kept decoding after the client left. The cancel-break path now resets the backend. - _SameTaskStreamingResponse threw CancelledError / called aclose() on the body iterator on a send-side disconnect, but neither runs the try/finally of a generator that never started (early disconnect on http.response.start), so the passthrough's eagerly-opened upstream httpx stream and cancel-registry entry leaked. It now tracks whether the body started and, when it did not, runs an optional unstarted_cleanup hook; the OpenAI passthrough wires it to close the upstream resp/client and exit the cancel tracker. Adds a nested-unquoted-marker regression test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-06-23 06:13:56 -07:00
Daniel Han	935f6c50ef	studio: tighten torchao Windows-ROCm comments and test docstrings (#6610)	2026-06-23 05:49:25 -07:00
Daniel Han	55c392ff7c	studio: fix sentence-transformers RAG embedder on Windows ROCm (torchao) (#6608) ... torchao has no working Windows ROCm build. transformers.quantizers imports it, and it loads torch's c10d distributed backend at module level, which the AMD Windows wheels omit (no RCCL). The import aborts, transformers can no longer expose PreTrainedModel, and the sentence-transformers embedder silently falls back to the llama-server GGUF embedder. Linux ROCm and NVIDIA are unaffected (the c10d ops are present / torchao is real there). The training and export workers already install the shared torchao stub before importing transformers, but the RAG embedder runs in the main backend process, which never did. Two fixes, both no-ops off Windows ROCm: - embeddings.py: install_torchao_windows_rocm_stub() before the first sentence-transformers import, so an already-installed torchao is neutralized (fixes existing venvs). - install_python_stack.py: stop installing torchao on Windows ROCm; it can only crash on import there, so new venvs never ship it. Add tests covering the embedder stub call and the install skip.	2026-06-23 05:39:02 -07:00
Daniel Han	9780cdcca1	Fix FlashAttention fp32 crash with DoRA (use_dora=True) (#6526) ... * Fix FlashAttention fp32 crash with DoRA (use_dora=True) DoRA upcasts lora_magnitude_vector to fp32 for the optimizer, which promotes the q/k/v_proj output to fp32. FlashAttention only accepts fp16/bf16, so the fp32 q/k/v raised 'FlashAttention only support fp16 and bf16 data type'. Downcast q/k/v to the compute dtype before the flash kernels. Fixes #1013 * Apply kwarg-spacing format hook to DoRA dtype test (pre-commit) * DoRA+FA2: downcast any fp32 among Q/K/V and clamp to a flash-supported dtype * Tighten code comments (no logic change) --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com>	2026-06-23 01:29:19 -07:00
Daniel Han	7b208bc35c	Fix misleading 'only for image models' error for Qwen3-VL when torchvision is missing (#6525) ... * Fix misleading 'only for image models' error for Qwen3-VL when torchvision is missing transformers >= 5.4 hard-requires torchvision for VLM image/video processors and no longer falls back to a slow processor. Without torchvision the processor load raises ImportError, unsloth degrades to a text-only tokenizer, and the vision data collator later fails with 'UnslothVisionDataCollator is only for image models!'. Detect this case at load time and raise a clear, actionable error pointing at the missing torchvision dependency instead. Fixes unslothai/unsloth#4202 * Apply kwarg-spacing format hook to vision torchvision guard (pre-commit) * Make torchvision-missing detection precise: check availability first, match specific error text * Tighten code comments (no logic change) * Make missing-torchvision VLM error version-agnostic The raise also fires on transformers 4.57.x for VLMs with a video processor (Qwen2.5-VL, Qwen3-VL), where AutoVideoProcessor requires torchvision. The old message claimed 'transformers >= 5.4 requires torchvision', which is inaccurate on 4.57.x. Reword to state torchvision is required for this model's vision processors without a version-specific claim. --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com>	2026-06-23 01:28:09 -07:00
Daniel Han	eae59b25b6	fix: use EMPTY_LOGITS on the fused-CE not-return_dict path (#2068) (#6482) ... * fix: use EMPTY_LOGITS on the fused-CE not-return_dict path (#2068) CausalLM_fast_forward's fused cross-entropy path (small batch, labels set, UNSLOTH_RETURN_LOGITS off) computes the loss straight from hidden_states via unsloth_fused_ce_loss and never materializes `logits`. The return_dict=True branch returns EMPTY_LOGITS, but the `not return_dict` branch returned `(logits,) + outputs[1:]`, raising "UnboundLocalError: cannot access local variable 'logits'" whenever it ran (e.g. training with return_dict=False). Same bug in the llama and mistral fast-forward paths. Return EMPTY_LOGITS on that branch too, matching the adjacent return_dict output. Verified on GPU: a forward(return_dict=False, labels=...) that raised UnboundLocalError now returns (loss, EMPTY_LOGITS, ...) and backward() succeeds. Adds tests/test_fused_ce_not_return_dict_logits.py, a CPU source-drift guard (the fused path itself is GPU/triton only) asserting both fast-forward paths keep using EMPTY_LOGITS there. * Address review: parse the fused-CE drift line with whitespace-tolerant regexes The drift detector sliced the source with exact string matching (source.index("output = (") + the next newline), so a formatter respacing or rewrapping the assignment would break the parse. Switch to anchored regexes that tolerate whitespace and line wrapping, keeping the match anchored after the fused guard so it targets the fused-CE branch and not the normal output = (logits,) path. Behavior and the two drift assertions are unchanged. * Tighten code comments (no logic change) --------- Co-authored-by: Daniel Han <michaelhan2050@gmail.com>	2026-06-23 01:26:55 -07:00
Daniel Han	70926822db	studio/setup.sh: guard empty CUDA arch detection in the source build (#5854) (#6481) ... * studio/setup.sh: guard empty CUDA arch detection in the source build PR #5826 hardened setup.sh for fresh CUDA toolkits, but the source build still set -DCMAKE_CUDA_ARCHITECTURES only when nvidia-smi reported a compute capability. When that query returns nothing the build proceeded with no explicit arch list, so llama.cpp built PTX only. On a driver older than the toolkit that binary fails at runtime with "the provided PTX was compiled with an unsupported toolchain" - the build succeeds, so neither the build-time check nor the CPU fallback caught it (issue #5854). Resolve the arch list before committing to a CUDA build. A new pure helper _resolve_cuda_archs parses and de-duplicates the nvidia-smi compute_cap output and honors an explicit UNSLOTH_LLAMA_CUDA_ARCHS override. When the result is empty, build CPU llama.cpp instead of a PTX-only binary, with a clear message pointing at the override - so the user still ends up with a working llama-server. The override also lets advanced users force a native build on hosts where nvidia-smi cannot report compute_cap. No behavior change when an arch is detected: -DGGML_CUDA=ON plus the arch, CUDA flags and NVCC_PREPEND_FLAGS are assembled exactly as before. Adds tests/sh/test_resolve_cuda_archs.sh (single/multi/dedup/empty/garbage/ whitespace/override cases), wired into tests/run_all.sh and the studio-backend-ci.yml shell-test loop. * studio/setup.sh: resolve nvidia-smi via /usr/bin fallback for arch detection Addresses review feedback on the empty-CUDA-arch guard: _setup_has_usable_nvidia_gpu classifies a host as NVIDIA-usable using nvidia-smi on PATH OR /usr/bin/nvidia-smi, but the new arch detection probed only `command -v nvidia-smi`. On a GPU host where nvidia-smi is off PATH (reachable only at /usr/bin), arch detection returned empty and the new empty-arch branch dropped the build to CPU, losing CUDA. Mirror the same PATH-then-/usr/bin resolution so those hosts still get a native CUDA build. Also scope _resolve_cuda_archs locals with `local` (no behavior change; it already runs under command substitution). * tests: update compute_cap-probe assertion for $_smi_bin resolution The nvidia-smi /usr/bin fallback parameterized the binary in the compute_cap probe (_setup_run_smi "$_smi_bin" ...), so the literal-string assertion in test_compute_cap_probe_timeout_wrapped no longer matched. Assert the probe is preceded by _setup_run_smi (timeout-wrapped) instead, scanning all occurrences so the comment mention is ignored. Same intent, binary-agnostic. * tests: ruff-format the compute_cap probe assertion (pre-commit) Collapse the backslash-continued assert onto one line and normalize slice spacing so the ruff-format pre-commit hook (0.6.9) is satisfied. Formatting only; no behavior change. * Tighten code comments (no logic change) * studio(windows): build CPU when CUDA arch is undetectable (#5854) The Windows source build added -DGGML_CUDA=ON unconditionally but only set -DCMAKE_CUDA_ARCHITECTURES when $CudaArch was detected. With no detectable compute capability that produced a PTX-only binary, the same hole the Linux fix closed. Build CPU llama.cpp in that case, and honor UNSLOTH_LLAMA_CUDA_ARCHS to force a CUDA build, matching setup.sh. Detected-arch builds are unchanged. * test: anchor NVCC_PREPEND_FLAGS scope check on the final CPU branch The undetectable-arch CPU fallback adds an earlier -DGGML_CUDA=OFF, so the ordering check now anchors on -DGGML_CUDA=ON and the last -DGGML_CUDA=OFF instead of the first. --------- Co-authored-by: Daniel Han <michaelhan2050@gmail.com>	2026-06-23 01:26:43 -07:00
Daniel Han	3a9fc34fcf	Studio Playwright: snooze update banner before sending (#6576) ... * Studio Playwright: snooze update banner before sending The llama.cpp update banner is a fixed bottom-right toast (z-9998). When an update is available it overlaps the composer's Send button and its subtree intercepts the click, so send_and_wait times out (flaky; surfaces on the Windows studio UI smoke, passes otherwise). Snooze the banner if it is showing before each send, then wait for it to detach. * Also snooze the web update banner before sending The web update banner (web-update-banner, z-9999) is a fixed bottom-right toast like the llama.cpp one and can overlap the Send button too. Loop over both banners and snooze whichever is showing.	2026-06-22 08:27:18 -07:00
Sanat Bhargava	1fc8bf53c7	Add Hugging Face dataset streaming mode to Studio (#4946) ... * Add HF dataset streaming mode to Studio * Added default value for datasetStreaming in training-config-store.ts * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Handle None max_steps for streaming validation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: fast-fail streaming validation and guard incompatible modes Reject dataset_streaming at the API boundary when hf_dataset is empty, the dataset is vision/audio, or max_steps is not set. Probe eval split with get_dataset_split_names before the streaming load so typos fail immediately instead of mid-training. Guard column_names=None after map on iterables. Hide the UI toggle for non-text configurations and clear the stale flag when config becomes incompatible. * studio: add streaming dataset tests, iterable helper, and streaming template/format support (WIP) Work-in-progress on top of feat/studio-dataset-streaming-mode (PR #4946): - new test_training_streaming.py and iterable.py dataset helper - streaming support in chat_templates.py and format_conversion.py - additional streaming guards in trainer.py / models / routes - frontend streaming wiring in params-section and training-config-store Committed to preserve uncommitted work before merging latest main. * studio: fix review-team findings for streaming + main merge BLOCKER: streaming + raw-text/CPT crashed on len(IterableDataset). Guard it in the start route (reject format_type=="raw" or training_type=="Continued Pretraining") and in isStreamingSupported (datasetFormat !== "raw"). Also: - models/training.py: validate hf_dataset/subset/split (charset+length, block ..//); cap dataset slice indices (le=1e9); note validator ordering - chat_templates.py: guard _apply_custom_mapping .map() for streaming - trainer.py: warn when packing+streaming - training-config-store.ts: persist-migration bump to v11 (standalone datasetStreaming backfill); add isVisionModel to NON_PERSISTED; toast on silent streamingCompatiblePatch mutations in the 4 indirect setters - tests: route rejections (max_steps, raw/cpt), slice cap, unsafe hf_dataset * studio: enable raw-text/CPT dataset streaming + streaming UX polish - raw_text: keep the lazy filter but skip len()-based row counting for IterableDatasets so raw-text / CPT can stream; guard the eval-size log - routes/trainer: drop the raw/CPT streaming block; add a defensive not-streaming guard on the eval auto-split (train_test_split) - dataset-section: streaming toggle is visible-but-disabled and lists the exact unmet requirement(s) in its tooltip; block embedding models - training-start-overlay: show "streaming (no full download)" instead of a stuck download bar for streaming runs - trim the streaming test suite to the high-value cases * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: address streaming review (MLX/embedding guards, sliced eval split, rehydrate timing) - routes: reject dataset_streaming for embedding training and on Apple Silicon (MLX); both loaders materialize the full dataset instead of streaming - trainer: validate the base eval split name so streaming eval accepts HF slice syntax such as "validation[:1000]" - training-config-store: defer the onRehydrateStorage setState to a microtask so it doesn't hit the store's TDZ during synchronous hydration - test: streaming start rejects embedding models * studio: harden HF dataset streaming (column_names, split slicing, empty/eval bounds, gating) Address a deeper streaming review: - raw_text: resolve_column_names() guards IterableDataset.column_names=None (from_generator / unresolved features) so raw-text and CPT streaming no longer raise TypeError before training - models/routes: reject HF slice syntax in train_split/eval_split when streaming (load_dataset(streaming=True) raises "Bad split"); reject mixed sources (local/S3) and embedding/MLX streaming at the API, not just in the UI - trainer: an empty post-slice/filter stream fails preflight with a clear message; streaming eval is capped (STREAMING_EVAL_MAX_SAMPLES) so each eval terminates; the manual-slice shortcut falls back to a regular load when train_split is sliced - format_conversion: streaming conversions preflight the first mapped row so format errors surface before training, not mid-iteration - frontend: block streaming on Apple Silicon; clear datasetStreaming when a dataset is detected as image/audio at start * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: fix CI for streaming PR (lint blocker + no-torch sandbox + preflight test) - trainer.py: drop unused `IterableDataset` import (hoist safety-net blocker). - test_training_streaming.py: only select real classes (isinstance type) when locating the trainer class, so a MagicMock-stubbed global is never passed to object.__new__ (fixes TypeError on the Python 3.10-3.13 jobs). - no-torch import sandboxes (test_e2e_no_torch_sandbox.py, test_studio_import_no_torch.py): teach the chat_templates/format_conversion exec stubs and the full-import-chain copy list about the new `.iterable` module so the AFTER/runtime cases import without torch again. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com> Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai> Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>	2026-06-22 17:48:18 +03:00
Daniel Han	dbc13f02c9	Studio: fix Ctrl+C shutdown ordering (installer shell + uvicorn thread wait) (#6566) ... * Installer: respect a declined Studio auto-start and keep Ctrl+C shutdown logs ordered The `curl | sh` Studio auto-start prompt had two issues on Linux/macOS/WSL (install.sh). install.ps1 already gates on input redirection, so Windows is unaffected. 1. Typing n, or any closed/EOF /dev/tty, still launched Studio. The read fallbacks defaulted to "y" (read failure, and the no-tty branch), so any answer other than a cleanly delivered y/n line auto-started a blocking foreground server. Default those to "n"; a real Enter still counts as yes via ${_reply:-y}. 2. On Ctrl+C the shell prompt printed in the middle of Studio's shutdown logs. The non-interactive installer shell took the default SIGINT action and died before the child finished its graceful shutdown, so the prompt raced ahead of "All subprocesses cleaned up". trap '' INT in the installer shell so it waits for Studio's own graceful shutdown. * Studio: wait for the uvicorn thread before the terminal returns on Ctrl+C Builds on #6565 by @Imagineer99. The studio server runs uvicorn in a daemon thread, so on Ctrl+C the process could return to the shell while that thread was still writing its shutdown logs, interleaving them with the prompt. Retain the uvicorn thread and join it (flushing stdout/stderr) before terminal entrypoints return, from run.py's main shutdown path and the CLI shutdown paths. Refinements over #6565: - Bound the join at 5s (_SERVER_SHUTDOWN_JOIN_TIMEOUT, matching the existing _graceful_shutdown subprocess timeouts) so a stalled uvicorn shutdown cannot hang the terminal; the timeout warning branch is now reachable. - Restore SIG_DFL for SIGINT/SIGTERM at the start of the signal handler so a second Ctrl+C force-quits, and drop the redundant in-handler wait (the post-loop wait already covers the signal path). Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: keep child Ctrl+C working and restore SIGBREAK - install.sh: run studio in a subshell that resets INT to default (trap - INT; exec ...) so the foreground child does not inherit the installer shell's ignored SIGINT, which would otherwise swallow the studio process's own Ctrl+C and graceful shutdown. - run.py: also restore SIGBREAK to SIG_DFL in the signal handler so a second Ctrl+Break force-quits on Windows, matching SIGINT/SIGTERM. * install.sh: capture studio exit with || under set -e so the migration hint still prints * Trim shutdown-fix comments to be terser (comments only, no code change) * Dedup CLI shutdown-wait into finally blocks (review follow-up) --------- Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-22 07:41:10 -07:00
Daniel Han	86d65f3d4a	Add regression tests for the stray-forward compile-cache reset (#6569) ... * Add regression tests for the stray-forward compile-cache reset Follow-up to #6511, which fixed the bug but whose squash merge did not include the tests. These cover the two issues that fix addressed, under the GPU-free tests/conftest.py harness: - _unsloth_reset_stray_compile_cache is an exported module-level symbol in unsloth.models._utils (it previously lived only inside the RL trainer template string, so every non-RL import silently no-op'd) - _unsloth_install_pretrain_detector keeps a recorded "seen" forward on an idempotent reinstall with a live hook, and only resets it after teardown - only a grad-enabled pre-train forward marks the cache poisoned - the reset warns and clears seen when a stray forward was seen, tears the hook down even on the clean path, and walks the .model/.base_model/.module wrapper chain to reach a nested marker * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin UNSLOTH_COMPILE_DISABLE in the warn-path reset tests The reset only warns and resets Dynamo when UNSLOTH_COMPILE_DISABLE != "1". A GPU-free CI env that sets it to "1" would make the warn assertion in test_reset_clears_seen_and_warns_when_a_stray_forward_was_seen flaky. monkeypatch it to "0" in both warn-path tests so the warn / no-warn assertions are deterministic and test the seen flag, not the env. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-22 07:22:47 -07:00
Daniel Han	e83d4ae072	Windows installer: fix DiskPart UAC mid-install, drive-root cache, and spurious unsloth.exe rename warning (#6296) ... * Windows installer: fix DiskPart UAC, drive-root cache, spurious rename warning, CPU-base messaging amd-smi gate (DiskPart UAC mid-install): the AMD torch wheel ships hipInfo.exe inside the venv, and the bitsandbytes fix prepends that venv Scripts dir to PATH. shutil.which("hipinfo") then found it and flipped _amd_smi_allowed() to True, so the post-install AMD probe fell through to `amd-smi list` (the venv hipInfo failed to report gcnArchName, which is why the arch came from the GPU-name table) and amd-smi elevated, popping the DiskPart UAC. Fix: a hipinfo resolved inside the active venv (sys.prefix) is the torch-wheel binary, not a HIP SDK, and must not open the gate. Mirrored in install_python_stack.py, install_llama_prebuilt.py, and backend utils/hardware/amd.py (the runtime VRAM poller had the same latent prompt). TORCHINDUCTOR_CACHE_DIR: move from C:\tc to <StudioHome>\TORCHINDUCTOR_CACHE_DIR so the inductor/Triton cache lives under the user's Studio home, not the system drive root. Long paths are already enabled above so deep inductor paths still fit. unsloth.exe rename: skip the rename (and its "pip may fail with WinError 32" warning) when SKIP_STUDIO_BASE=1. In the install.ps1 flow base packages are not reinstalled, so unsloth.exe is never rewritten; the self-rename only failed because setup runs via unsloth.exe (the running launcher holds its own file). The 'studio update' flow still attempts it. CPU PyTorch messaging: clarify that the CPU base is temporary and setup replaces it with GPU ROCm wheels, and print an explicit "GPU ROCm PyTorch installed" line after the AMD wheels land, so the log makes clear the final install is GPU-accelerated. Adds two regression tests covering the venv-internal vs external hipInfo gate. Verified end-to-end on a Strix Halo box (Radeon 8060S / gfx1151): install.ps1 --local from this branch completed exit 0 with no DiskPart prompt, no rename warning, the cache under the Studio home, and "GPU ROCm PyTorch installed (gfx1151)"; Studio then booted and detected "ROCm (HIP 7.13.99004) -- AMD Radeon 8060S Graphics". * Windows installer: drop the unreliable unsloth.exe rename and its WinError 32 warning setup.ps1 used to rename the running unsloth.exe out of the way before the base-package upgrade so pip could replace it. That rename never actually worked: setup runs *via* unsloth.exe, so renaming our own running uv-trampoline launcher failed with a sharing violation (WinError 32) and only printed a scary 'could not rename unsloth.exe; pip may fail with WinError 32' warning on every Windows install and update. It also was not needed. pip tolerates a running/locked console-script .exe: it moves the old one aside and writes the new one. The base upgrade routes through pip on Windows, so the upgrade succeeds (or, in the install.ps1 flow with SKIP_STUDIO_BASE=1, the base is not touched at all) and unsloth.exe is left intact either way. Removing the rename block and its failed-install restore block removes the false warning for all Windows devices in both the install and update flows. * Windows installer: gate venv-internal hipInfo.exe in PowerShell amd-smi probe; harden venv path checks Follow-up to PR #6296. - install.ps1 and setup.ps1: ignore the AMD torch wheel hipInfo.exe that lives inside the Studio venv when probing for a HIP SDK, so amd-smi no longer reopens the DiskPart UAC during install/update. Mirrors _path_inside_venv in the Python installers, which already do this. - amd.py, install_llama_prebuilt.py, install_python_stack.py: normcase the venv containment check (Windows paths are case-insensitive) and run the HIP_PATH/ROCM_PATH candidate through it too. - setup.ps1: fall back to a short TORCHINDUCTOR cache dir when long paths are unavailable, and create the dir wildcard-safely. - tests: isolate sys.prefix in the gate helper, add HIP_PATH/ROCM_PATH cases, and assert the PowerShell venv exclusion. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows installer: install ROCm PyTorch directly for a known AMD arch When the GPU arch is known (name-inferred from the GPU-name table) but ROCm could not be probe-verified (no HIP SDK, no amd-smi), the bootstrap installed a CPU PyTorch base that setup.ps1 then force-reinstalled as ROCm. The repo.amd.com wheels bundle their own runtime (no HIP SDK required), which setup.ps1 already relies on, so the CPU base was a pure wasted download/install. - Gate the ROCm index on a known arch, not only on probe-verified ROCm, so a mapped arch installs ROCm torch directly. Unmapped arches and no-GPU hosts still get CPU (unchanged). - Fall back to a CPU base if the ROCm-index install fails, so a transient repo.amd.com outage does not abort the install (setup.ps1 retries ROCm). - Correct the stale comment that claimed ROCm wheels need a confirmed HIP SDK. - Add a regression test for the arch-based gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows installer: correct the unsloth.exe rename-removal comment The comment claimed the base upgrade 'routes through pip on Windows' and that pip 'moves the old unsloth.exe aside, then writes the new one'. That is not what the code does. install_python_stack tries uv first; on a locked launcher uv aborts and falls back to pip, but the pip fallback strips --upgrade-package and base.txt lists only bare unsloth/unsloth-zoo, so pip finds them already satisfied and no-ops. The running unsloth.exe is left intact at its current version either way. Reword the comment to describe the real uv-first / pip-fallback-no-op behavior. No functional change. * Windows installer: close two gaps in the venv-internal hipinfo exclusion Review follow-up. The amd-smi/DiskPart gate could still reopen in two cases: - setup.ps1 ran the HIP probe long before $VenvDir is assigned, so without VIRTUAL_ENV (the `unsloth studio update` path) $venvRoots was empty and the venv-internal hipInfo.exe was not recognized. Seed the venv root from UNSLOTH_SETUP_PYTHON and the default Studio home too (both installers). - The HIP_PATH/ROCM_PATH candidate was accepted without the venv filter, so an env var pointing into the venv (AMD wheel) still set $HipSdkInstalled. Run Test-HipinfoIsVenvInternal on the candidate as well (both installers). Extend the PS gate test to assert both. Both .ps1 parse clean; install tests pass (the venv-internal / HIP probe coverage at 359 passed). * Windows installer: correct the CPU-base message for arches with no ROCm wheels After gating the ROCm index on a known arch, a mapped arch sets $ROCmIndexUrl and installs ROCm directly, so it no longer reaches the "temporary CPU base" branch. That branch is now reached only by a name-inferred arch with no ROCm wheels (e.g. RDNA2 gfx103X), where setup.ps1 does NOT install ROCm. The old text ("setup replaces it with GPU ROCm wheels ... the final install IS GPU-accelerated") was therefore always wrong there. Say plainly that PyTorch stays on CPU for this GPU. * Windows installer: seed the venv-internal hipInfo check from a custom Studio home Test-HipinfoIsVenvInternal seeded the venv root from VIRTUAL_ENV, VenvDir, the setup python, and the default %USERPROFILE% path only. A standalone `unsloth studio update` with a custom UNSLOTH_STUDIO_HOME (or STUDIO_HOME alias) and none of those set would not recognize the venv hipInfo on PATH, reopening the amd-smi/DiskPart gate. Seed the custom home too, in both installers, and assert it in the gate test. * Studio installer: resolve venv aliases and expand ~ in the hipInfo venv filter Two review points on the amd-smi/DiskPart UAC gate: 1. _path_inside_venv compared os.path.abspath of sys.prefix and the hipInfo path, which does not resolve symlinks, junctions, or 8.3 short names. A venv reached through an aliased path then fails the check, so its bundled hipInfo.exe is mistaken for an external HIP SDK and amd-smi runs (the DiskPart prompt this fix exists to suppress). Switch to os.path.realpath in all three copies (amd.py, install_llama_prebuilt.py, install_python_stack.py). 2. setup.ps1's early venv-internal hipInfo probe seeded the venv root from a custom Studio home (UNSLOTH_STUDIO_HOME / STUDIO_HOME) without expanding a leading ~, while the canonical resolver does. With a tilde form, [IO.Path]::GetFullPath kept the literal ~ relative to cwd, so the custom-home hipInfo escaped the filter and reopened the gate. Expand ~ in the probe the same way as the resolver. tests/studio/install/test_pr5940_followups.py: 30 passed (adds a symlink realpath case and a setup.ps1 tilde-expansion guard). * Studio installer: mirror the hipInfo venv filter and ROCm wheel pins into install.ps1 Follow-up review on the same install.ps1 paths: 1. install.ps1's venv-internal hipInfo probe (Test-HipinfoIsVenvInternal) seeded the venv root from a custom Studio home without expanding a leading ~, unlike the canonical resolver and setup.ps1. A tilde form left [IO.Path]::GetFullPath with the literal ~ (relative to cwd), so the custom-home hipInfo escaped the filter and reopened the amd-smi/DiskPart gate. Expand ~ in the probe, matching the setup.ps1 fix. 2. The AMD ROCm path installed torchvision/torchaudio bare while pinning torch to below 2.12. AMD's per-arch index publishes the companions independently and may ship torchvision 0.27 (for torch 2.12) before removing 0.26, so a bare resolve can pick an ABI-incompatible set and fall back to CPU. Add torchvision/torchaudio floor maps and pass the pinned specs, mirroring setup.ps1 and install_python_stack.py. 3. The ROCm-to-CPU fallback torch install used Invoke-InstallCommand (no retry), the only torch step in the file without it. Switch to Invoke-InstallCommandRetry so the recovery path survives a transient index failure. tests/studio/install/test_pr5940_followups.py: 33 passed (parametrized tilde check over both installers, a torch/companion floor-map parity test, and a CPU-fallback retry guard). * Studio installer: scan all PATH hipinfo so the venv copy can't shadow a real HIP SDK The amd-smi HIP-SDK probe used shutil.which("hipinfo") / Get-Command hipinfo, which return only the first hit on PATH. The AMD torch wheel ships hipInfo.exe inside the venv and the bnb fix (plus the Studio backend) prepend the venv Scripts dir to PATH, so that venv-internal copy lands first. When a real HIP SDK hipinfo sits later on PATH with HIP_PATH/ROCM_PATH unset, the first-hit probe stopped at the venv copy, treated it as "not a HIP SDK", and closed the amd-smi gate -- AMD users in that PATH-only SDK setup lost amd-smi telemetry and could fall back to CPU. Scan every PATH entry and keep the first hipinfo that is not venv-internal; only the venv copy is ignored, so the UAC/DiskPart suppression is unchanged. Applied to all three Python copies (install_llama_prebuilt.py, install_python_stack.py, backend/utils/hardware/amd.py) via a new _external_hipinfo_on_path helper, and both PowerShell callers (install.ps1, setup.ps1) now use Get-Command hipinfo -All filtered by Test-HipinfoIsVenvInternal. tests/studio/install/test_pr5940_followups.py: 36 passed (real-PATH scan tests, a shadow-regression test for the exact venv-first ordering, and a parity check that every Python copy uses the scanning helper). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio uninstallers: fix leftovers (false "removed", shared icon, llama lock) Auditing a dual native+WSL uninstall on a real device surfaced three leftovers: 1. uninstall.ps1 removed the data dir (which holds unsloth.ico) before the shortcuts that reference that icon, so Explorer's icon cache briefly held it open. Remove-Item -Recurse reported success yet left the locked file, and the dir was never re-attempted, so it orphaned with a false "removed" log. _RemovePath now verifies the path is actually gone (retrying transient locks) and reports honestly, and the data dir is re-swept after the shortcuts go. 2. install.sh writes a shared unsloth.ico to %LOCALAPPDATA%\Unsloth Studio for the WSL shortcut, but uninstall.sh never removed it, orphaning the icon (and dir) after a WSL uninstall. uninstall.sh now drops that icon and the dir when empty, in both the powershell.exe and drvfs-fallback paths. 3. ~/.unsloth/.llama.cpp.install.lock was never removed, so the rmdir of ~/.unsloth failed and the dir lingered. Both uninstallers now remove the lock. Verified by running both uninstallers on a real dual install: device fully clean (no install dirs, shortcuts, PATH/registry entries, shared icon, or lock left). * install.sh: auto-route Strix Halo WSL to an existing Ubuntu 24.04 ROCm-on-WSL is the GPU runtime for Strix Halo and only targets Ubuntu 24.04. When the installer runs in a newer default distro (e.g. 26.04) it cannot enable the GPU and silently falls back to CPU. If a 24.04 distro already exists, re-run the install there and stop in the current one so the GPU path is taken without the user having to know about the distro requirement. Runs before venv creation so the wrong distro is left untouched, guards against re-route loops via UNSLOTH_WSL_REROUTED, leaves a working ROCm distro alone (librocdxg present), and skips the GGUF-only / opt-out / non-Strix cases. When no 24.04 distro exists we keep today's behaviour: continue to CPU and print the `wsl --install Ubuntu-24.04` guidance, never auto-downloading a distro. Adds tests/sh/test_strixhalo_wsl_reroute.sh (hermetic: extracts the function, rewrites its paths to fixtures, mocks wsl.exe) covering the full decision matrix, wired into tests/run_all.sh. * uninstall.ps1: keep shared unsloth.ico for a surviving WSL shortcut A dual native+WSL install shares %LOCALAPPDATA%\Unsloth Studio\unsloth.ico: install.sh points the WSL shortcut's icon there while the native install owns the dir. The native uninstaller removed the whole dir unconditionally, so uninstalling native while keeping WSL left the WSL shortcut with a blank icon. The old code only avoided this when Explorer happened to hold the icon open, which is unreliable; on a real dual install the dir was deleted and the WSL shortcut went blank. _RemoveDataDirKeepingWslIcon now scans the Start Menu + Desktop for a surviving "Unsloth Studio (WSL ...).lnk" and, if found, removes everything in the data dir except unsloth.ico (keeping the dir) instead of deleting it; with no WSL shortcut it removes the dir as before. uninstall.sh still drops the icon and the empty dir when WSL itself is uninstalled, so every uninstall order ends clean. Adds tests/studio/test_uninstall_dual_install_icon.ps1 (AST-extracts the helper and runs it against a temp dir with controlled shortcut dirs) covering the dual, native-only, empty, and missing-dir cases, wired into the windows-inference smoke workflow. Verified on a real dual install: native uninstall now keeps unsloth.ico and the WSL shortcut's icon stays intact. * installer: condense AMD/ROCm code comments (no behavior change) Tighten the comments added for the Strix Halo native+WSL installer work so they are shorter and clearer without losing intent: the venv-internal hipInfo amd-smi gate, the ROCm torch/companion floor maps, the WSL 24.04 reroute, and the dual-install uninstall icon handling. Comment-only; code paths unchanged. 107 insertions, 166 deletions across 11 files. * install.sh: run the Strix Halo WSL reroute before any STUDIO_HOME write The reroute fired after mkdir -p "$STUDIO_HOME" and the legacy-venv migration, so rerouting 26.04 -> 24.04 left an empty ~/.unsloth/studio stub in the origin distro (and ran venv migration in the distro about to be abandoned). Move the reroute ahead of the venv section so the origin distro is left untouched, matching the function's own comment. Behavior is identical on every non-reroute path. * installer: fix ROCm CPU-fallback, hipinfo gate edge cases, uninstall icon, WSL 22.04 - install.ps1: clear $ROCmIndexUrl/$ROCmTorchFloor after the CPU fallback so the flavor-repair block does not retry the failed ROCm index and abort the install; pin the ROCm companion specs ($visionSpec/$audioSpec) in the repair path too. - install.ps1 + setup.ps1: skip a bare drive root in Test-HipinfoIsVenvInternal so a non-venv UNSLOTH_SETUP_PYTHON does not match the whole drive; iterate HIP_PATH/HIP_PATH_57/ROCM_PATH and take the first non-venv hipinfo. - amd.py, install_llama_prebuilt.py, install_python_stack.py: strip surrounding quotes from PATH entries before probing for hipinfo. - install.sh: pipefail the WSL reroute curl|sh; do not reroute supported Ubuntu 22.04. - uninstall.sh: keep the shared unsloth.ico while any Unsloth shortcut (native or another WSL distro) still references it, in both the powershell and drvfs paths. - tests: regression coverage for all of the above. * installer: forward reroute options, guard ROCm bootstrap, harden hipinfo gate - install.sh: forward the caller's --package/--python/--verbose/--tauri and a custom UNSLOTH_STUDIO_HOME into the WSL reroute (was a bare default install); bail on --local; run the reroute BEFORE dependency/uv install so the origin distro is left untouched; set UNSLOTH_SKIP_ROCM_WSL_SETUP after a failed reroute so the later ROCm-on-WSL bootstrap does not install into the unsupported origin distro. - install.ps1 + setup.ps1: Get-Command hipinfo -CommandType Application so only real executables match (not an alias/function named hipinfo). - uninstall.ps1: guard $env:APPDATA when building the default shortcut search dirs. - tests: cover option forwarding, --local bail, the bootstrap guard, and the gate change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * installer: guard origin ROCm bootstrap on every CPU-only fallback; harden ~ expansion WSL reroute: the no-wsl.exe, no-24.04-target and --local fallbacks all tell the user the install continues CPU-only, but only the failed-reroute branch set UNSLOTH_SKIP_ROCM_WSL_SETUP=1. The later _maybe_bootstrap_rocm_wsl gate keys off that flag, so the other three branches could still install ROCm into the unsupported origin distro (e.g. 26.04). Set the skip guard on all of them. Forward UNSLOTH_ROCM_WSL_AUTO into the reroute so a Tauri/consented GPU bootstrap carries through to the rerouted 24.04 child instead of dropping to the prompt path. install.ps1/setup.ps1: guard the venv-probe ~ expansion on a non-empty $env:USERPROFILE so Join-Path does not throw on a profile-less service account. Tests: add no-wsl.exe and UNSLOTH_ROCM_WSL_AUTO reroute cases, the USERPROFILE guard assertion, and route shell-test fixtures through a single trap-cleaned root. * installer: pin + soften Windows ROCm Python repair, reroute to 22.04, harden gates install_python_stack.py: the Windows AMD ROCm repair in _ensure_rocm_torch() installed bare torch/torchvision/torchaudio via the fatal pip_install -- the same asymmetry already fixed on the PowerShell side. A transient repo.amd.com failure could abort the whole install even after install.ps1/setup.ps1 fell back to CPU. Pin companions per-arch (gfx120X/Strix -> the rocm7.2 trio, mirroring the PS floor maps) and make the retry nonfatal: keep the existing build and let the user re-run update to retry ROCm, so the chain install.ps1 -> setup.ps1 -> stack stays CPU-safe. install.sh: reroute now targets an installed Ubuntu 24.04 OR 22.04 (24.04 preferred); both are AMD-supported for ROCm-on-WSL, matching the leave-alone set, so a box with only 22.04 reaches the GPU instead of staying CPU-only. install.ps1/setup.ps1: a bare ~ for UNSLOTH_STUDIO_HOME left an empty Join-Path child (PS 5.1 throws); fall back to USERPROFILE directly and only join a real remainder. _path_inside_venv (amd.py + both installers): guard a root-dir sys.prefix so commonpath can't classify every path on the drive as venv-internal (defensive; venv never at root). uninstall.sh: guard an empty LOCALAPPDATA in the PS-interop icon cleanup (mirror APPDATA). Tests: add 22.04-target reroute cases, Windows ROCm pin+nonfatal coverage (text + behavioral), root-dir guard coverage, and bare-~/LOCALAPPDATA guard assertions. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install.sh: match WSL reroute target by exact distro name, not substring The 24.04/22.04 reroute target was chosen with grep -F (substring), so a custom distro such as 'Ubuntu-24.04-test' (with no exact Ubuntu-24.04) was picked as the target; the later 'wsl -d Ubuntu-24.04' then fails and the Strix Halo install stays CPU-only. Match whole lines (grep -ixF) and reuse the matched name so only a real Ubuntu-24.04/22.04 is targeted. Adds substring-rejection + exact-vs-custom tests. * install.sh: keep the WSL reroute target to Ubuntu 24.04 (helper-supported only) The ROCm-on-WSL bootstrap (scripts/install_rocm_wsl_strixhalo.sh) dies on any VERSION_ID other than 24.04 and pins the noble repo, so treating 22.04 as GPU-supported let the parent report a successful reroute while the child fell back to CPU. Drop 22.04 from the supported set and the reroute target list; 24.04 stays the sole target (keeping the exact whole-line distro match). An already-working ROCm on any other version is still left alone by the librocdxg check above. tests: reroute 22.04 cases updated to the 24.04-only behavior; make the "no wsl.exe" case hermetic so a real host wsl.exe can't leak in on dev boxes; stop the tauri exit-order check from mis-flagging the reroute helper's [ "$TAURI_MODE" = true ] && ... --tauri one-liner. * installer: tighten comment wording across the Strix Halo install/uninstall paths Condense the verbose multi-line comment blocks (amd-smi hipinfo gate, ROCm torch install + CPU fallback, WSL reroute, uninstall icon-keep) into fewer, clearer lines. Comments and a few docstrings only; no code, logic, or behavior change. Verified with bash -n, the PowerShell parser, and ast.parse, and the installer test suite still passes. * add AGPL-3.0 SPDX headers to the .sh/.ps1 scripts missing them Every shell and PowerShell script under the Studio/installer surface now carries the standard SPDX-License-Identifier: AGPL-3.0-only + copyright header (after the shebang where present): the installer (install.sh, install.ps1), build.sh, the .github and src-tauri scripts, the installer test suite, and the moe kernel test. Header-only, line endings preserved; bash -n, the PowerShell parser, and the installer tests all pass. * installer: drop the duplicate AGPL header from install.sh and install.ps1 Both already carry an SPDX-License-Identifier: AGPL-3.0-only header below their usage comment block; the prior header pass added a second one at the top because it only scanned the first few lines. Remove the duplicate so each file keeps a single original header. * installer: force-reinstall CPU fallback torch; propagate Tauri NEED_SUDO from reroute install.ps1/setup.ps1: when the AMD ROCm wheel install fails and we fall back to a CPU base, force-reinstall the torch/vision/audio triplet. A failed ROCm install can leave an unpinned ROCm torch (e.g. 2.10.0+rocm on gfx110X/gfx90a) that still satisfies the CPU torch>=2.4,<2.11.0 range, so without --force-reinstall uv keeps the ROCm build and only swaps the companions -- a mismatched venv the flavor-repair block won't fix. setup.ps1 scopes the forced reinstall to the ROCm-fallback path () so the genuine CPU-only install stays fast. install.sh: the Strix Halo WSL reroute treated every nonzero child exit as a reroute failure and fell back to CPU. In --tauri mode the child uses exit 2 ([TAURI:NEED_SUDO]) to ask the desktop app to elevate for the target distro; capture the child's exit code and propagate exit 2 in Tauri mode (the child already printed the NEED_SUDO line) instead of masking it. CLI mode still falls back to CPU on a generic failure. Tests: reroute Tauri exit-2 propagation (and non-Tauri CPU-fallback) cases; run_func now preserves the child exit code; force-reinstall assertions for both PowerShell installers. Note: codex's _rr_q apostrophe finding is a false positive -- the helper already emits POSIX-correct 'O'\''Brien' and round-trips under both sh and bash. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * setup.ps1: fix $cpuForce array collapse in the ROCm->CPU torch fallback An if-expression assignment ($cpuForce = if ($ROCmCpuFallback) { @("--force-reinstall") }) collapses the single-element array to a scalar string, so @cpuForce splatting enumerated it character-by-character into broken single-letter args (- - f o r c e ...), which made uv/pip reject the install and aborted the whole Studio setup on the AMD ROCm->CPU fallback path. Build $cpuForce as a real array assigned outside the if-expression so the splat passes a single --force-reinstall arg. Genuine CPU-only installs stay fast (empty array, no flag). Test now asserts the array-build form and rejects the if-expression form. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * uninstall: remove the isolated Node.js runtime (~/.unsloth/node) The isolated Node.js runtime (install_node_prebuilt.py, added with the managed-Node change) installs to ~/.unsloth/node in default mode -- a sibling of studio, so deleting <studio> leaves it behind (~200MB orphaned after uninstall). Both uninstallers already remove the other default-mode siblings (llama.cpp/.cache/.staging); add node alongside them. uninstall.ps1 also adds it to the handle-lock sweep so a held node.exe can't block the delete. Env/custom mode nests node under the custom root, removed with that root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-22 03:09:08 -07:00
Daniel Han	378e33c8a5	Studio macOS: faster startup, MLX self-heal, drop obsolete prebuilt pins (#6494) ... * Studio: defer llama.cpp update probes and self-heal MLX on macOS Two macOS startup problems shared one root area in the FastAPI lifespan: - The llama.cpp capability + freshness probes ran inline before the server yielded, so a cold/slow/flaky network on the GitHub freshness check blocked 'Application startup complete' (~34s on CI, longer in the field). Move both probes to a daemon thread; app.state stays None until ready (status routes already re-probe at request time). Opt out with UNSLOTH_DISABLE_UPDATE_CHECK=1. - Train and Export were greyed out because mlx/mlx-lm/mlx-vlm arrive only transitively and a resolver backtrack silently drops them, so CHAT_ONLY stayed true. Add utils/mlx_repair.py: when Apple Silicon is detected without MLX, reinstall mlx/mlx-lm/mlx-vlm by name on a daemon thread and re-run hardware detection (opt out UNSLOTH_DISABLE_MLX_AUTOREPAIR=1). Surface a chat_only_reason in /api/health plus a sidebar tooltip so a greyed Train/Export explains itself instead of failing silently. * Studio: guard model defaults against a None model name load_model_defaults(None) called model_name.lower() with no guard, raising 'Error loading model defaults for None' before any model is selected. Return an empty dict for a falsy/non-str name. * Studio: drop obsolete upstream macOS + Windows Blackwell prebuilt pins Both pins worked around gaps in ggml-org upstream prebuilts, but Studio now routes every GPU host and all of macOS to the unslothai/llama.cpp fork (published_repo_for_host), which ships the needed bundles, so both pins are dead code on the default install path: - macOS b9415: macOS always routes to the fork (its own macOS bundles), and host_supports_macos_minos() is the backstop. The pin only fired under an explicit --published-repo ggml-org override. - Windows Blackwell b9360: Windows-NVIDIA routes to the fork, whose windows-x64-cuda13 bundle covers Blackwell (manifest max_sm 120, toolkit 13.3), so the pin's self-disable check makes it dormant on every default install; it could only activate under the same upstream override on a 13.0-13.2 driver. Remove the pin constants, functions, and call sites. Keep the Blackwell capability detection (_drop_blackwell_incapable_windows_cuda, _host_is_blackwell, _windows_cuda_attempt_covers_blackwell) that still drops a non-sm_120 cuda-12.4 build on a Blackwell host. After this, an explicit --published-repo ggml-org override on a Blackwell 13.0-13.2 host loses its GPU fallback and lands on CPU; the default fork path is unaffected. Update the install selection-logic and macOS-compat unit tests for the new no-pin behavior. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: walk back deeper on the macOS upstream prebuilt path After removing the b9415 macOS pin, the explicit --published-repo ggml-org upstream path still used the default 2-release fallback, so a pre-macOS-26 host behind a run of macOS-26-only builds would exhaust two too-new plans (minos is only checked post-download) and drop to a source build before reaching a loadable older release. Walk back as deep as the fork macOS path (DEFAULT_MAX_MACOS_RELEASE_FALLBACKS), turning the removed static pin into dynamic discovery. Addresses review feedback on the macOS upstream fallback. * Studio: pin transformers during MLX self-heal so it cannot break Studio mlx-lm/mlx-vlm declare transformers>=5, but the single-env install pins transformers==4.57.6. The self-heal used --upgrade with no constraint, so it could upgrade transformers in the live venv and break the rest of Studio just to make import mlx.core pass. Pin transformers to the installed version via a constraint file: the resolver either finds an mlx build compatible with it or fails (we stay chat-only), never upgrading transformers underneath Studio. Addresses review feedback on the MLX repair install. * Studio: harden MLX self-heal against an unsupported mlx-vlm Pinning transformers alone made uv backtrack mlx-vlm to 0.3.9 (below unsloth-zoo's mlx-vlm>=0.4.4), which imports but breaks VLM Train/Export -- so the self-heal could clear chat-only onto a broken stack. Mirror the main installer: set UV_OVERRIDE=overrides-darwin-arm64.txt so a current mlx-vlm coexists with the transformers pin, require the same minimum versions unsloth-zoo declares, and gate/validate on a full mlx_stack_available() check (not a bare import) so an old or partial stack stays chat-only. Addresses PR review. * Studio: filter Blackwell-incapable CUDA in resolve_upstream_asset_choice resolve_upstream_asset_choice returned the first windows-cuda choice unfiltered, so a Blackwell host could be handed an sm_120-incapable cuda-12.4 build while the sibling planners drop it. Apply _drop_blackwell_incapable_windows_cuda here too and fall through to the CPU bundle on a Blackwell host with no capable GPU asset. Addresses PR review. * Studio: re-poll health so MLX self-heal reaches an open UI The sidebar cached the initial /api/health, so a successful background MLX self-heal (chat_only flips false) did not re-enable Train/Export until a manual reload. While chat-only for the recoverable mlx_unavailable reason, re-poll /api/health and stop once Train/Export become available. Addresses PR review. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make the disabled Train/Export tooltip reachable The greyed Train/Export items pass a tooltip explaining why (e.g. MLX missing), but a disabled <button> fires no pointer events and SidebarMenuButton only showed tooltips while collapsed, so the explanation never appeared. Wrap a disabled button in a focusable span and show its tooltip while expanded too; enabled items keep the collapsed-only behavior. Addresses PR review. * Studio: gate Train/Export on the full MLX stack, not bare mlx.core detect_hardware enabled MLX training whenever `import mlx.core` worked, but the MLX self-heal (utils/mlx_repair) treats a stack without mlx-lm/mlx-vlm at the versions unsloth-zoo requires as inadequate. That asymmetry let the UI enable Train/Export on exactly the partial/backtracked stack the self-heal is trying to repair (greyed-in-but-broken VLM export). Gate on the same mlx_stack_available() criterion so a partial stack stays chat-only (reason mlx_unavailable) and the background repair restores it. Addresses PR review. * Fix MLX repair and health auth for PR #6494 * Fix macOS upstream prebuilt fallback for PR #6494 * Fix MLX stack validation for PR #6494 * Fix MLX self-heal validation for PR #6494 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Review fixes: isolate hardware-state test, robust transformers pin - test_chat_only_reason.py: detect_hardware() assigns module globals directly, which monkeypatch does not revert; the autouse fixture now saves and restores DEVICE/CHAT_ONLY/CHAT_ONLY_REASON/IS_ROCM so a chat-only verdict here cannot leak into other backend tests (e.g. test_utils.py) on a GPU host. - mlx_repair.py: read the transformers version from importlib.metadata instead of importing transformers, so the install pin is not silently dropped when transformers has valid metadata but fails to import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix CI: model full MLX stack in dispatch tests, keep selection test offline dispatch (macOS) job: - detect_hardware now gates MLX on the full stack (mlx_stack_available imports mlx_lm/mlx_vlm and checks dist versions), so faking only mlx.core makes the apple_silicon_mlx profile resolve to CPU. The dispatch tests assert the routing decision when the stack IS usable, so model a complete stack: test_hardware_dispatch_matrix patches utils.mlx_repair.mlx_stack_available and test_is_mlx_dispatch_gate patches hardware._has_usable_mlx_stack. The stack predicate's own internals stay covered by test_mlx_repair.py. Repo tests (CPU) job: - test_no_cuda_attempt_on_published_path_for_13_1 fell through to a live github_release_assets() upstream fetch after the Blackwell filter dropped every published attempt, which the offline security scanner blocks. Stub that fetch so the walk-back deterministically finds no usable CUDA build and raises PrebuiltFallback without network. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden MLX self-heal: prepare transformers constraint inside the try attempt_mlx_repair runs on a daemon thread, but _transformers_constraint_args was called before the try. A failure there (e.g. tempfile.mkstemp on a full disk or a bad TMPDIR) would propagate unhandled and silently kill the self-heal thread. Move the call inside the try and initialize constraint_path so any such failure is caught and leaves Studio chat-only instead of crashing the thread. --------- Co-authored-by: Daniel Han <michaelhan2050@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: wasimysaid <wasimysdev@gmail.com>	2026-06-22 02:20:08 -07:00
Daniel Han	f89c829fcf	Fix save crash for legacy list-form _tied_weights_keys (NemotronH) (#6540) ... * Fix save crash for legacy list-form _tied_weights_keys (NemotronH) transformers >= 5 save_pretrained reads module._tied_weights_keys.keys(), which raises 'list' object has no attribute 'keys' for modules that still declare the attribute as a list (e.g. NemotronH backbone.layers.N.mixer.*_proj), crashing GGUF export and merged saves part-way through. Coerce any legacy list/tuple _tied_weights_keys into the dict form transformers 5.x expects, mapping each key to itself. Only the keys are read (as dedup patterns) so behaviour is preserved, and older transformers that iterate the attribute directly see the same keys. The helper is idempotent and best-effort so a save never fails over it. Called from unsloth_save_model, unsloth_save_pretrained_gguf and unsloth_generic_save after tokenizer patching. Adds version-independent unit tests covering list/tuple coercion, dict and None/empty pass-through, idempotency and odd-object tolerance. * Coerce empty/set _tied_weights_keys too transformers only skips _tied_weights_keys when it is None, so an empty list, tuple or set still reaches .keys() and raises the same AttributeError. Coerce every non-dict container (including the empty case and sets) to a dict, and add tests for empty/set inputs. * Tighten comments in tied-weights save fix * Scope tied-weights-keys coercion to the save call Coercing legacy list-form _tied_weights_keys to {k: k} fixed the transformers 5 save crash, but persisted a self-mapping on the live model. transformers 5 re-ties from the dict's values, so a later resize/re-tie would no-op the tie instead of pointing the output weights back at the input embeddings. Replace the in-place mutation with a decorator that coerces before the save and restores the originals afterwards (including on exception), so the save sees the dict form transformers needs while the model keeps its original tie metadata. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim comments to be more succinct --------- Co-authored-by: Daniel Han <michaelhan2050@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-22 02:11:46 -07:00
Daniel Han	a41b8c7a44	Make Visual Studio + CMake optional on Windows (prebuilt llama.cpp needs no build tools) (#6499) ... * studio/setup.ps1: complete Visual Studio 2026 support for the CUDA llama.cpp build Builds on #6038 (VS 2026 / v18 detection). Once the generator is detected as Visual Studio 18 2026, two things still broke the CUDA llama.cpp build: - the CUDA to VS MSBuild integration copied the CUDA .targets into a hardcoded VC\v170 (VS 2022) BuildCustomizations folder, so a VS 2026 (v180) toolchain saw no CUDA toolset and cmake failed with "No CUDA toolset found". - cmake was installed with no version check, but the "Visual Studio 18 2026" generator requires CMake 4.2+. This adds Get-VcBuildCustomizationsDir (derives v160/v170/v180 from the detected generator, falls back to v170), a CMake 4.2 guard for the VS 2026 generator (upgrades via winget once, else fails with a clear message), and routes both the copy target and the failure hint through the derived path. No behavior change for VS 2022/2019/2017: the folder resolves to v170 and the guard is skipped. Adds windows-latest Pester unit tests (tests/studio_setup_ps1) plus a workflow that runs them. * Address review: make VS 2026 self-contained + gate CMake guard to source build - Find-VsBuildTools now detects VS 2026: vswhere catalog_productLineVersion 2026 -> "Visual Studio 18 2026", and the filesystem scan covers the "18"/"2026" dirs (incl. non-standard editions like Preview). Adapted from #6038 by @LeoBorcherding, so the v180 BuildCustomizations path and the CMake guard are actually reachable on a VS 2026-only host. - Move the CMake 4.2 guard out of Phase 1 into the committed-source-build branch. The preferred prebuilt llama.cpp path never reaches it, so a VS 2026 host on CMake < 4.2 is no longer blocked from using the prebuilt. - winget upgrade -> install fallback when the on-PATH cmake is not the Kitware winget package, and log winget failures instead of swallowing them. - Add a windows-latest Find-VsBuildTools VS 2026 discovery regression test. * tests(vs2026): define New-FakeVsTree in BeforeAll so It blocks can see it The Find-VsBuildTools discovery tests are Windows-only (-Skip on non-Windows), so they first ran on the windows-latest Pester job, where New-FakeVsTree raised CommandNotFoundException: it was defined in the Describe body, which Pester 5 executes only during discovery, so the function did not persist into the run-phase It scope. Move it into a BeforeAll block (which runs in the run phase and is visible to the It blocks). No production code change. * Address review: probe cmake generator support, fall back to older VS, fix cmake PATH after winget The VS 2026 CMake guard previously gated only on the cmake version (>= 4.2) and hard-failed otherwise. Review on #6473 raised three real gaps: - A VS-bundled cmake below 4.2 can still drive the VS 2026 generator. Probe cmake --help (Test-CmakeListsGenerator / Test-CmakeCanDriveGenerator) and accept it when the generator is advertised, not just on the version floor. - After winget upgrade/install, an older cmake earlier on PATH kept being resolved. Add-DefaultCmakeToPath prepends the default install dir so the new cmake wins before re-probing. - When cmake cannot drive VS 2026 but an older Visual Studio (2022/2019/2017) is installed and usable, fall back to it (Get-FallbackVsGenerator) instead of hard-failing, preserving the pre-VS-2026 build path. Tests mock the cmake command rather than dropping a shim on PATH: PowerShell caches its application-path table, so a real cmake on the runner (present on windows-latest) wins over a PATH shim. A function mock is resolved first and is cache-proof cross-platform. * Detect VS installed under the Preview edition dir for older versions Find-VsBuildTools already scans every subdir for VS 2026, but the older-version (2017/2019/2022) filesystem fallback and Get-FallbackVsGenerator only checked BuildTools/Community/Professional/Enterprise. A Preview-channel install lives under a 'Preview' edition folder, so it was missed when vswhere was also unavailable. Add 'Preview' to both edition lists and guard each with a Windows Pester test. * Add real-VS integration matrix: detect actual VS 2022 and VS 2026 in parallel The unit tests validate VS detection logic with mocked vswhere and fake install trees (all five versions). This adds a parallel integration job that runs the real Find-VsBuildTools / Get-VcBuildCustomizationsDir against the Visual Studio actually preinstalled on GitHub-hosted runners: - windows-2022 -> real Visual Studio 2022, expect generator v170 - windows-2025-vs2026 -> real Visual Studio 2026, expect generator v180 It asserts our detection matches the real install, the install path exists, the derived toolset matches, and that the derived v-number is a real folder on the VS install. VS 2017/2019/2015 are retired from hosted images, so only 2022 and 2026 can be exercised against a genuine install; the rest stay covered by the mocks. * Detect VS 2026 via vswhere: it reports productLineVersion '18', not '2026' Real-VS CI on the windows-2025-vs2026 runner showed vswhere reports catalog_productLineVersion='18' (the internal major) for Visual Studio 2026, not the marketing year '2026' that VS <= 2022 report. The vswhere map only had '2026', so on a real VS 2026 host the vswhere branch returned null and detection survived only via the filesystem scan (Source='filesystem'); a VS 2026 installed outside the default Program Files location would not be found at all. Extract a pure Resolve-VsGeneratorFromLabel that accepts both the year and the internal-major form ('18'/'17'/'16'/'15' as well as '2026'/'2022'/'2019'/'2017') and use it for both the vswhere and filesystem branches. Add pure unit tests (cross-platform) for the mapping, including the '18' -> VS 2026 case. * ci: dot-source Resolve-VsGeneratorFromLabel in the real-VS integration job Find-VsBuildTools now calls Resolve-VsGeneratorFromLabel, so the integration step must extract it too; without it the job failed with the helper not recognized. * Defer Visual Studio + CMake to the llama.cpp source build (prebuilt path needs no build tools) The Windows installer required Visual Studio Build Tools and CMake eagerly in Phase 1 (winget install + exit 1 if absent), before the llama.cpp prebuilt-vs- source decision. But the preferred path downloads a prebuilt llama.cpp (no compiler), the backend only shells out to the prebuilt llama-server.exe, and PyTorch is pip wheels -- so VS and CMake are only needed for the from-source build last resort. The eager requirement forced every Windows user to install multi-GB Visual Studio + CMake they never use, or the installer failed. Change (mirrors the already-lazy Resolve-CudaToolkit / OpenSSL): - Phase 1c/1d now only DETECT cmake / VS and log; they never winget-install or exit. The prebuilt install runs zero build-tool installs and is unblocked on hosts without build tools. - New Ensure-BuildToolsForLlamaSourceBuild installs CMake (best effort) + VS (hard requirement, exit 1 with the existing guidance if it cannot be found), called only when a source build is actually committed, before Resolve-CudaToolkit. git stays eager (pip needs it for git+ deps). Tests: - Pester: the early probe (Find-VsBuildTools) returns null without exiting when no VS is present; Ensure-BuildToolsForLlamaSourceBuild no-ops when VS is already detected. - New studio-windows-no-vs-smoke.yml: Job A renames Visual Studio + vswhere away and hides cmake, runs the real install.ps1 --local --no-torch, and asserts the prebuilt llama.cpp installed (no source-build fallback, no VS/CMake install), PyTorch CPU imports, the backend is healthy, and a /v1/chat/completions inference returns a reply -- all with no Visual Studio. Job B confirms the GPU CUDA prebuilt is available and the resolver runs without VS. * Fix VS 2026 CUDA source build ordering and fallback VS discovery Same fix as on the stacked base branch (studio-vs2026-cuda-msbuild): - Move Resolve-CudaToolkit below the CMake gate/fallback in the source build path. It copies the CUDA MSBuild .targets into the current VS generator's BuildCustomizations folder, so running it before a VS 2026 to older-VS fallback left the .targets under v180 while cmake configured v170 ("No CUDA toolset found"). It now runs after the final generator is selected. - Get-FallbackVsGenerator now queries vswhere first, matching Find-VsBuildTools, so a VS installed outside the default Program Files roots is found instead of failing with a hard exit. - Add Pester regression tests: the source build resolves CUDA after the fallback, and the fallback queries vswhere. * Ensure the Visual C++ Redistributable is present for the prebuilt llama.cpp and PyTorch The prebuilt llama-server.exe and the PyTorch wheels dynamically link the MSVC runtime (VCRUNTIME140.dll, MSVCP140.dll, VCRUNTIME140_1.dll). The Universal CRT ships with Windows 10+, but the VC++ 2015-2022 redistributable does not, so a clean box can fail to launch llama-server or import torch with a missing VCRUNTIME140.dll. - Add Test-VCRedistInstalled (System32 vcruntime140_1.dll, with a registry fallback gated on version 14.20+) and Ensure-VCRedist (winget Microsoft.VCRedist.2015+.x64, non-fatal), called as Phase 1b.5 so it runs even on the no-build-tools prebuilt path. It is a no-op when the runtime is already present, which is the common case. - Add Pester tests for the detection: present via the DLL, present via the registry, absent, and an old 2015-only redist that is too low. * Add a CI job that validates the VC++ runtime detection on a real Windows runner Runs on windows-latest and windows-2025-vs2026: asserts Test-VCRedistInstalled reports present on the stock image, removes both detection signals (the System32 DLL via a redirected SystemRoot and the HKLM runtime keys, restorably) to confirm detection fires on a genuinely clean box, then does a literal uninstall/reinstall round trip with the official installer and the Ensure-VCRedist winget path. The runtime is restored before the job ends. * Dot-source the full logging closure in the VC++ runtime CI job Ensure-VCRedist calls step/substep, which reach Write-StudioStdoutMirror and Get-StudioAnsi; extract those too so the job does not fail with an unrecognized command. Also note that the runtime is ref-counted by Visual Studio on the hosted image, so the literal package uninstall is a no-op there (the clean-box section already proves detection fires when the runtime is genuinely absent). * Tighten comments in setup.ps1, the VS2026 tests and workflow Comment-only: condense the verbose helper/test/CI comments to one or two lines, drop the obvious ones, keep the non-obvious rationale. Verified comment-only by comparing the PowerShell code-token stream before and after (no code tokens changed); Pester suite still green. * Fold the no-VS and setup.ps1 VS2026 Windows CI into studio-windows-inference-smoke.yml Move the no-vs-cpu/no-vs-gpu-resolve and pester/vs-integration/vcredist-clean-box jobs into the existing Windows GGUF CI workflow and delete the two standalone files, so a studio change triggers one Windows workflow instead of three. Path filter gains tests/studio_setup_ps1/**; job keys and artifact names stay unique. * CI: assert a Windows ROCm prebuilt exists in the no-VS resolve job The no-vs-gpu-resolve job confirmed a Windows CUDA asset but never a ROCm one, and the resolver step resolves to CPU on hosted runners (no AMD GPU), so the AMD no-VS guarantee rode only on shared resolver code. Grep the per-gfx windows-x64-rocm-gfx bundles in the same asset-availability step so a release that drops the Windows ROCm prebuilts fails loudly. --------- Co-authored-by: Daniel Han <michaelhan2050@gmail.com>	2026-06-22 01:11:09 -07:00
Daniel Han	d77845ebc0	Update studio root-resilience tests for the inference-backend refactor (#6490) (#6553) ... * Update studio root-resilience tests for the inference-backend refactor #6490 moved the studio_root() probe and its (ImportError, OSError, ValueError) handler out of _find_llama_server_binary / _kill_orphaned_servers into the shared _resolved_studio_root_and_is_legacy() classifier, and switched the WSL ROCm lib-dir ordering to lib_dirs.extend(_wsl_system_rocm_lib_dirs()). These source-introspection tests still asserted the old inline structure, so they fail on main (surfaced by any PR that trips the Repo tests path filter, e.g. the Windows installer PRs). Point them at the new structure and assert the defense in its new home; no runtime change. * Address review: qualify the classifier call and harden helper-body extraction Assert the callers invoke LlamaCppBackend._resolved_studio_root_and_is_legacy() through the class namespace (more precise than the bare name), and end the helper-source slice at the next sibling def/decorator at the same indent instead of the literal @staticmethod string, so a future docstring that mentions a decorator can't truncate the helper mid-body and break exec(). --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com>	2026-06-22 01:10:49 -07:00
Daniel Han	821cfe561a	Harden evaluate_fetch against execution-context-destroyed during navigation (#6549) ... * Harden evaluate_fetch against execution-context-destroyed during navigation The Mac Studio chat-UI Playwright test intermittently failed with 'Page.evaluate: Execution context was destroyed, most likely because of a navigation' (e.g. run 27904470348). evaluate_fetch already retried transport failures (the JS result status==0), but the page.evaluate call itself can throw at the Python level when a navigation or auth refresh destroys the execution context mid-call, which was uncaught and crashed the script. Wrap the evaluate in a try/except that retries this transient class of error (execution context destroyed, frame detached, target closed) within the existing attempt budget, letting the page settle via wait_for_load_state before retrying. Real or persistent errors still propagate (re-raised on the final attempt or for non-transient messages). * Add reusable robust_evaluate and route auth-token reads through it Promote the execution-context-destroyed retry from evaluate_fetch into a reusable robust_evaluate(page_or_locator, expression, arg) helper, and have evaluate_fetch use it (no behaviour change to the transport retry). Route the post-login localStorage auth-token reads in playwright_chat_ui.py and playwright_extra_ui.py through it too, since those direct evaluates run right after auth redirects and are the same navigation-race class. The stable composer/IME evaluates that never overlap a navigation are left as-is. Helper retry semantics covered by unit checks (transient retries then succeeds, non-transient re-raises, persistent re-raises after the budget, locator settles via .page). * Apply repo ruff-format kwarg-spacing to the hardened Playwright evaluates * Don't replay single-use POSTs (auth/refresh) when robust_evaluate retries a context loss * Match context-loss markers case-insensitively and only replay idempotent fetches Playwright varies the casing of the detached-frame/destroyed-context error across versions, so the substring check now lowercases both sides. evaluate_fetch no longer replays mutating methods on a mid-call context loss: the default now retries only GET/HEAD/OPTIONS, so a duplicate POST /api/inference/load (rejected while the first is still loading) or a spent POST /api/auth/refresh is never re-sent. Callers can still override per call. --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com>	2026-06-21 21:57:25 -07:00
Daniel Han	9f39cc2c39	Studio: use an isolated Node.js for the frontend build instead of replacing the system Node/npm (#6533) ... * Studio: use an isolated Node.js for the frontend build instead of replacing the system Node/npm * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: address Node isolation review (no-Node probe crash, PATH refresh, OXC provisioning, venv python, runtime node resolver) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix/adjust Node isolation for PR #6533 * Studio Node: don't cache a negative node resolution; accept Node metadata in setup.sh ownership guard - node_runtime: memoize only a version-adequate executable so a Node installed by a separate-process 'studio update' is picked up without a backend restart. - setup.sh: _studio_owned_adoptable also accepts UNSLOTH_NODE_PREBUILT_INFO.json, matching the setup.ps1 Node ownership guard (custom-home parity). * Studio setup.ps1: skip OXC npm install gracefully when npm is absent Mirror setup.sh's `command -v npm` guard so a pip-installed Studio with no system Node skips the OXC runtime install (validator degrades at runtime) instead of exit 1 aborting the whole setup. Tighten test_node_probe_guard.ps1's probe regex so it only matches the two system-version probes, not this new npm guard. * Wire test_node_probe_guard.ps1 into Windows CI for PR #6533 * Harden isolated Node install and probes for PR #6533 - install_node_prebuilt.py: keep an existing, still-usable isolated Node when nodejs.org's dist index is unreachable instead of aborting the update on a transient outage (existing_install_usable + tolerant fetch). - install_node_prebuilt.py: pin NPM_CONFIG_PREFIX/npm_config_prefix and drop NODE_PATH in _run_node so any npm -g stays inside the isolated prefix; Windows npm otherwise writes to %APPDATA%\npm. - install_node_prebuilt.py: resolve tar hard-link targets against the archive root (symlink targets stay link-parent relative). - setup.ps1: wrap the system node/npm probes in try/catch so a present but broken shim degrades to the bundled Node instead of aborting setup. - setup.ps1: run the isolated Node install with the handed-off/venv Python (ReusedSetupPython); the main resolver runs later and bare python may be a Store stub this early. - setup.sh: log when the OXC validator runtime is skipped for missing npm, matching setup.ps1. - node_runtime.py: move the version-floor comment onto _version_meets_floor. - Tests for the offline-reuse and broken-shim paths. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim verbose comments across the Studio Node installer for PR #6533 Comments-only pass: collapse the multi-line section banners to single lines, drop comments that restate obvious code, and tighten the remaining docstrings and "why" notes without losing intent. No code changes (verified with an AST comment-only check on the Python files and a non-comment-diff scan on setup.sh and setup.ps1). Net 109 fewer lines; the install, decision, and probe-guard suites stay green. * Harden Node install from review: validated Python, version floor, legacy home, lock race For PR #6533, addressing the latest review pass: - setup.ps1: run the isolated Node install with the validated reused/venv Python. An incompatible reused interpreter (old venv, conda, stale UNSLOTH_SETUP_PYTHON) is no longer used; fall back to the resolved python instead. - setup.ps1: a STUDIO_HOME/UNSLOTH_STUDIO_HOME override equal to the legacy default now uses the legacy sibling node dir (~/.unsloth/node), matching the runtime resolver and setup.sh, so OXC can find the Node it installed. - install_node_prebuilt.py: reject an explicit --node-version below the floor (^20.19 || >=22.12 || >=23) instead of installing a Node the build cannot use. - install_node_prebuilt.py: atomically rename a stale install lock before unlinking so two concurrent runs without filelock cannot both acquire it. Tests added for the version floor (parametrized + explicit-below-floor rejection). Full install suite: 937 passed, 1 skipped; setup.ps1 parses; decision tests green. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address latest review: armv7l + later-fetch offline reuse for PR #6533 - install_node_prebuilt.py: reject 32-bit ARM (armv7l) up front. Node 24 LTS ships no linux-armv7l build, so the old path failed late with a confusing "no sha256"; it now fails fast with a clear unsupported-architecture error. - install_node_prebuilt.py: extend the offline-reuse fallback to the SHASUMS and archive fetches. If index.json resolves a newer Node but a later download fails and a usable isolated Node is already on disk, keep it instead of aborting a non-force update. Tests added: armv7l/armhf are unsupported; a SHASUMS failure keeps an existing usable Node and re-raises when none is present. Full install suite: 941 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add UNSLOTH_STUDIO_HOME node-dir tests (install side + resolver) for PR #6533 * Add regression tests pinning the reuse path read-only and isolating installer writes Lock in the two invariants behind the isolated-Node design: reusing a good system Node never mutates the user's Node/npm, and the installer's own npm calls only ever write inside its install_dir. - tests/studio/install/test_install_node_prebuilt_logic.py: assert _run_node redirects NPM_CONFIG_PREFIX/npm_config_prefix into install_dir and drops an inherited NODE_PATH; assert _ensure_npm_floor scopes the npm self-upgrade to install_dir (never -g against the system) and is a no-op once npm meets the floor. - tests/sh/test_system_node_readonly.sh (new, wired into studio-backend-ci.yml): the setup.sh NODE_SOURCE=system arm runs no global install and sets no NPM_CONFIG_PREFIX, with a positive control that the bundled arm does. - tests/studio/test_node_decision.ps1: symmetric structural guard that the prefix pin and the only global install (bun) live in the bundled branch, not the system arm. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: wasimysaid <wasimysdev@gmail.com>	2026-06-21 21:17:29 -07:00
Parvesh Saini	17e9714a98	studio: run /generate/stream's sync generator off the event loop to avoid blocking it (#6466) ... * studio: run /generate/stream's sync generator off the event loop to avoid blocking it * fix: close generator in finally on client disconnect in generate_stream * Fix/adjust generate stream test for PR #6466 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix/adjust generate stream cancellation for PR #6466 * Fix/adjust generate stream cleanup for PR #6466 * fix: cancel incomplete generate stream cleanup --------- Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: wasimysaid <wasimysdev@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com>	2026-06-19 14:19:04 +01:00
Daniel Han	a65672d947	Installer: repair stale/CPU-only PyTorch and warn on silent CPU fallback (NVIDIA + AMD, Win/Linux/Mac/WSL) (#5942) ... * Windows installer: repair a stale CPU PyTorch instead of looping forever A Windows machine with an NVIDIA CUDA 13 driver (e.g. RTX 6000 Pro on enterprise drivers) could get permanently stuck at: Stale venv detected (torch cpu != required cu130). [ERROR] The existing Studio environment needs repair. Re-run install.ps1 so it can replace the environment safely with rollback. Re-running install.ps1 did not help. install.ps1 installs torch with "torch>=2.4,<2.11.0" --index-url .../cu130 but no --force-reinstall, so when a torch==X+cpu is already present uv treats it as satisfying the range (PEP 440 ignores the +cpu/+cuXXX local label) and makes no change -- the CPU wheel is never replaced. setup.ps1 then rejects the venv as cpu != cu130 and exits, but it cannot create a venv or install torch, so the loop never resolves. The migrated- venv branch also preserves existing torch and never reinstalls it. After the install step, detect the installed torch flavor (cuXXX/cpu/rocm) and, when it does not match the tag implied by the selected index, force-reinstall the torch/torchvision/torchaudio triplet from the correct index via three --reinstall-package flags. No-op on a healthy matching venv; skipped for --no-torch, ROCm (already --force-reinstalls), and CPU-only machines. Adds two pure helpers (ConvertTo-TorchFlavorTag, Get-ExpectedTorchFlavorTag), a PowerShell unit test (tests/studio/test_torch_flavor.ps1), and a CI parse gate for install.ps1 (previously unparsed). * install.sh: repair a stale CPU PyTorch on Linux too (parity with install.ps1) install.sh has the same latent bug as the Windows installer: the CUDA torch install uses "torch>=2.4,<2.11.0" --index-url .../cuXXX with no --force-reinstall, so an already-present torch==X+cpu satisfies the version range (PEP 440 ignores the +cpu/+cuXXX local label) and uv leaves it in place. The migrated-venv branch also preserves existing torch. Unlike Windows there is no stale-venv check in setup.sh, so on Linux the symptom is silent CPU training rather than a hard loop -- same root cause. Mirror the install.ps1 fix: after the install block, detect the installed torch flavor (_torch_flavor_tag) and, when it does not match the index tag (_expected_torch_flavor_tag), force-reinstall the torch/torchvision/torchaudio triplet from the selected index via --reinstall-package. No-op on a healthy matching venv; skipped for --no-torch, ROCm (its own repair force-reinstalls), and CPU-only / macOS hosts. Adds tests/sh/test_torch_flavor.sh (run in studio-backend-ci and run_all.sh). * Installer: catch CPU-fallback on AMD/WSL too (repair ROCm, warn when unfixable) Extend the torch-flavor safety net beyond NVIDIA: - install.sh now auto-repairs a stale CPU torch on standard pytorch.org ROCm indexes too (the rocm-index install path lacked --force-reinstall, unlike the Windows ROCm install). Reuses the rocm-adjusted $TORCH_CONSTRAINT + rocm index, so it pulls the correct ROCm wheels. - Both installers gain a universal post-install warning: when a GPU build was expected (cuXXX / rocm, including the repo.amd.com gfx* arch indexes) but torch is still CPU-only, warn loudly instead of silently training on CPU. This catches the cases auto-repair cannot safely fix (AMD gfx arch indexes that need --find-links, a migrated AMD venv on Windows where the ROCm install was skipped). - Mac / Intel / CPU-only hosts resolve to the cpu index -> expected == installed -> no-op, no false warning. WSL uses install.sh, so the NVIDIA repair + warning apply there. Adds Get-InstalledTorchTag (ps1) and _torch_index_repairable (sh) helpers and extends both unit tests. gfx*/AMD indexes now map to the 'rocm' expected flavor. * Installer: tighten torch-flavor comments (no logic change) Condense the rationale comments added for the stale/CPU PyTorch repair in install.ps1, install.sh and the two helper unit tests; same intent, fewer lines. Comment-only: AST parse of install.ps1/setup.ps1 clean, helper unit tests (15 ps1, 24 sh under bash and dash) and the integration sims (24 ps1, 28 sh) still pass, banner markers the sims slice on are unchanged. * Installer: bound torch probe, auto-repair gfx, fix ROCm gate parity install.ps1: in Get-InstalledTorchTag, call WaitForExit(30000) and drain stdout and stderr asynchronously instead of reading stdout synchronously first, so a hung or noisy "import torch" (a wedged CUDA/driver, the exact failure this PR targets) can no longer block the probe past the timeout. install.sh and install.ps1: treat the repo.amd.com gfx* indexes as plain --index-url reinstallable. They are PEP 503 simple indexes uv resolves in full (torch plus every transitive dep) via --index-url, the same URLs the fresh ROCm install paths already use, so a stale CPU torch on AMD Strix now auto-repairs to the correct ROCm build instead of only warning. install.sh: include */gfx* alongside */rocm* in the bitsandbytes install and ROCm torch repair gates, so a custom UNSLOTH_AMD_ROCM_MIRROR whose path lacks /rocm/ still installs the AMD bitsandbytes build and repairs ROCm torch. tests/sh/test_torch_flavor.sh: gfx indexes now assert repairable, plus a gfx1151 case and an unknown-mirror not-repairable case. * install.ps1: guard Get-InstalledTorchTag against an empty python path Make the early return explicit for an empty $PythonExe instead of relying on Test-Path -LiteralPath '' returning false, so the probe stays safe under Set-StrictMode or a future refactor that drops the [string] annotation.	2026-06-18 08:57:17 -07:00
Daniel Han	18f8869829	Load DeepSeek-OCR and other VLMs that register AutoModel in auto_map (#6421) ... * Load repo-code VLMs that register AutoModel in auto_map FastModel.from_pretrained already falls back from the VLM auto class to AutoModelForCausalLM for repo-code VL models that register only that class in their auto_map (e.g. Nemotron-VL). Models like DeepSeek-OCR and DeepSeek-OCR-2 instead register their architecture under AutoModel, so they fell through to AutoModelForImageTextToText and raised "Unrecognized configuration class ... for AutoModelForImageTextToText". Generalize the guard: when neither vision auto class is registered, fall back to whichever generic auto class the repo actually registered (AutoModelForCausalLM, else AutoModel). * Do not hard-error on a newly initialized position_ids buffer RaiseUninitialized turns transformers' "some weights of ... were not initialized" warning into a hard error. position_ids is a deterministic arange buffer that transformers itself lists in _keys_to_ignore_on_load_missing, so re-initializing it is correct rather than a sign of a corrupt checkpoint. Some VLMs (e.g. DeepSeek-OCR) ship it non-persistently, which tripped the guard. Allowlist position_ids alongside the existing classifier/predictions head weights. * Only ignore missing-weight records that are exclusively position_ids The previous substring check skipped the whole "Some weights of ..." record whenever position_ids appeared anywhere in it. Transformers reports every missing key in one record, so a corrupt or incompatible checkpoint missing a real parameter could load with randomly initialized weights as long as one missing key contained position_ids. Parse the "newly initialized: [...]" list and suppress only when every listed key is a position_ids buffer; otherwise raise as before. * Match the concrete VLM auto class name when checking auto_map Transformers resolves remote code by the exact auto class name being called, and AutoModelForVision2Seq aliases to AutoModelForImageTextToText on transformers >= 5. Checking for both spellings treated a config that only registers the legacy AutoModelForVision2Seq key as having a supported VLM class, skipping the AutoModelForCausalLM fallback that used to load it and failing as an unrecognized config under AutoModelForImageTextToText. Match only the concrete class name we would actually pass, keeping the AutoModel and AutoModelForCausalLM fallbacks. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Preserve VLM mode on the vLLM path when falling back to AutoModel A repo-code VLM that registers only AutoModel or AutoModelForCausalLM (DeepSeek-OCR, Nemotron-VL) routes to that generic class, so is_vlm, derived from the resolved auto class, is False. That is correct for processor selection (these repos ship no AutoProcessor) but wrong for the vLLM path, where is_vision_model=is_vlm made vLLM treat a vision_config model as text-only and skip the VLM guard and conversion. Add is_vlm_config, derived from the config vision_config (and gated on not text_only so a text-only resolve still wins), and use it for the fast_inference VLM guard and the is_vision_model flags passed to load_vllm, get_vllm_state_dict and convert_vllm_to_huggingface. Processor selection still uses is_vlm, so DeepSeek-OCR keeps loading via its tokenizer. DeepSeek-OCR with fast_inference now raises the clear 'Fast inference is only supported for ...' error instead of being mishandled as text-only. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-18 07:03:43 -07:00
Daniel Han	07c7f9bfca	Package scanners: close fail-open gaps in the sdist fallback and hidden-payload paths (#6359) ... * Package scanners: close fail-open gaps in the sdist fallback and hidden-payload paths Follow-up hardening on the now-blocking scanners so the enforcing gate cannot report clean while a malicious artifact goes unscanned. scan_packages.py - Hidden payload: also flag a network call AND an os/subprocess exec that live only in a blanked docstring/string of an exec/eval file (the fetch-then-run shape of an exec(__doc__) dropper). Either alone in real code was already covered; hidden together they are the payload. - Pinned releases fail closed: _release_files no longer falls back to the latest artifact when a pinned version is missing or empty, so a yanked/bad pin is an error instead of a different file being scanned in its place. - requires_dist is read from the pinned release's metadata, not the project-level (latest) document, so a sdist-only pin follows its own dependency tree. - Environment markers are evaluated (PEP 508) instead of dropping any marker that merely contains the word extra, so default-true markers like extra != 'dev' are kept; conservative fallback keeps a dep on any uncertainty. - Transitive recovery is a depth-bounded worklist: a wheel dependency whose own child is sdist-only is fetched (--no-deps) and scanned, then its children are recovered in turn, rather than being silently skipped. scan_npm_packages.py - Baseline keys use the package-relative path instead of the basename, so the same basename in a different directory is not over-suppressed. Tests cover each case; full scripts pass AST and ruff checks. * Address review: tighten marker scope, decoy-proof the dropper check, fail closed on missing pin metadata - Markers: keep any dep whose marker can hold on another install target (sys_platform == 'win32', python_version == '3.13'); only drop a marker that depends solely on extra and is false with no extra. A scanner runs on one target but must cover code installed on others. Pure-extra markers are evaluated against default_environment() with extra unset. - Hidden dropper: the network+exec docstring check now inspects the removed (blanked) span directly, so a benign visible network or subprocess call cannot mask a payload that still lives in a docstring. Carrier checks stay blanked-only (an in-code carrier is already caught by the normal check), so corpus findings are unchanged. - requires_dist: a pinned version whose own metadata cannot be fetched recovers nothing rather than substituting the latest release's dependency tree. - Transitive recovery: the last-ditch direct-sdist branch also chases the recovered package's declared deps, matching the other branches. - npm baseline: schema bumped to v2 (package-relative keys); a pre-v2 baseline with entries is ignored (fail closed) instead of mis-applying basename keys. Tests cover each case; scripts pass AST, ruff, and the import-hoist verifier. * Scanner: exclude comments from hidden-payload check, flag missing pin metadata as incomplete Hidden network+exec detection now inspects only docstring/string spans (what exec(__doc__)/exec(<str>) can actually run), so a real exec() beside comments that mention a network and a subprocess call no longer false-positives. Missing pinned-release metadata in transitive recovery records a download_error so the --with-deps path fails closed instead of treating it as no dependencies. Adds regression tests for both. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-18 06:50:16 -07:00
Parvesh Saini	892d2983b0	Fix: scan_packages.py --fix crash on download_packages() tuple return (#6413) ... * Fix scan_packages.py --fix crash on download_packages() tuple return `download_packages()` returns `(results, download_errors)`, but the two `--fix`-path call sites still treated the return value as the bare results list. `find_safe_version` did `downloaded = download_packages(...)` followed by `if not downloaded:` (always false: a 2-tuple is truthy) and `for _, archive_path in downloaded:`, which unpacked the results list into two variables -> ValueError in the normal single-archive `--no-deps` case. `_run_fix` indexed `downloaded[0][1]`, i.e. the second archive of the results list instead of the first archive's path -> IndexError. So `--fix` crashed exactly when a CRITICAL finding needed remediation. The main scan path already unpacks the tuple; this aligns the two `--fix` sites with it. Adds CPU-only regression tests for both sites. Closes #6412 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Update scripts/scan_packages.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update scripts/scan_packages.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	2026-06-18 06:00:07 -07:00