mirror of
https://github.com/unslothai/unsloth.git
synced 2026-07-09 15:58:41 +00:00
mirror of https://github.com/unslothai/unsloth
agentaideepseekdeepseek-r1fine-tuninggemmagemma3llamallama3llmllmsloramistralopenaipytorchqwenqwen3text-to-speechttsunsloth
98a01e70cd
Some checks failed
Core / Core (HF=default + TRL=default) (push) Waiting to run
Core / Core (HF=4.57.6 + TRL<1) (push) Waiting to run
Core / Core (HF=latest + TRL=latest) (push) Waiting to run
Core / llama.cpp build + smoke (push) Waiting to run
Lint CI / Source lint (Python + shell + YAML + JSON + safety nets) (push) Waiting to run
MLX CI on Mac M1 / dispatch (push) Waiting to run
Security audit / advisory audit (pip + npm + cargo) (push) Waiting to run
Security audit / pip scan-packages :: extras (push) Waiting to run
Security audit / pip scan-packages :: studio (push) Waiting to run
Security audit / pip scan-packages :: hf-stack (push) Waiting to run
Security audit / npm scan-packages (Studio frontend tarballs) (push) Waiting to run
Security audit / workflow-trigger lint (pull_request_target / cache-poisoning) (push) Waiting to run
Security audit / pytest tests/security (push) Waiting to run
Security audit / npm provenance + new install-script diff (push) Waiting to run
Studio API CI / Studio API & Auth Tests (push) Waiting to run
Backend CI / (Python 3.10) (push) Waiting to run
Backend CI / (Python 3.11) (push) Waiting to run
Backend CI / (Python 3.12) (push) Waiting to run
Backend CI / (Python 3.13) (push) Waiting to run
Backend CI / Repo tests (CPU) (push) Waiting to run
Frontend CI / Frontend build + bundle sanity (push) Waiting to run
Studio GGUF CI / OpenAI, Anthropic API tests (push) Waiting to run
Studio GGUF CI / Tool calling Tests (push) Waiting to run
Studio GGUF CI / JSON, images (push) Waiting to run
Mac Studio API CI / Studio API & Auth Tests (push) Waiting to run
Mac Studio GGUF CI / OpenAI, Anthropic API tests (push) Waiting to run
Mac Studio GGUF CI / Tool calling Tests (push) Waiting to run
Mac Studio GGUF CI / JSON, images (push) Waiting to run
Mac Studio Install Matrix CI / Install + load (macos-14) (push) Waiting to run
Mac Studio Install Matrix CI / Install + load (macos-15) (push) Waiting to run
Mac Studio Install Matrix CI / Install + load (macos-26) (push) Waiting to run
Mac Studio Install Matrix CI / Install + load (macos-15-intel) (push) Waiting to run
Mac Studio Install Matrix CI / Install + load (macos-26-intel) (push) Waiting to run
Mac Studio UI CI / Chat UI Tests (push) Waiting to run
Mac Studio Update CI / Studio Updating Tests (push) Waiting to run
Studio Tauri CI / Tauri Linux debug build (no codesign) (push) Waiting to run
Studio UI CI / Chat UI Tests (push) Waiting to run
Studio Update CI / Studio Updating Tests (push) Waiting to run
Windows Studio API CI / Studio API & Auth Tests (push) Waiting to run
Windows Studio GGUF CI / OpenAI, Anthropic API tests (push) Waiting to run
Windows Studio GGUF CI / Tool calling Tests (push) Waiting to run
Windows Studio GGUF CI / JSON, images (push) Waiting to run
Windows Studio GGUF CI / Studio install + inference without Visual Studio (push) Waiting to run
Windows Studio GGUF CI / GPU prebuilt resolves without Visual Studio (push) Waiting to run
Windows Studio GGUF CI / setup.ps1 unit tests (VS 2026 / CMake guard) (push) Waiting to run
Windows Studio GGUF CI / real-VS detection (VS 2022) (push) Waiting to run
Windows Studio GGUF CI / real-VS detection (VS 2026) (push) Waiting to run
Windows Studio GGUF CI / VC++ runtime detect + install round-trip (windows-2025-vs2026) (push) Waiting to run
Windows Studio GGUF CI / VC++ runtime detect + install round-trip (windows-latest) (push) Waiting to run
Windows Studio UI CI / Chat UI Tests (push) Waiting to run
Windows Studio Update CI / Studio Updating Tests (push) Waiting to run
Wheel CI / Wheel build + content sanity + import smoke (push) Waiting to run
Studio load-orchestrator CI / test (push) Has been cancelled
* Studio: restore tensor parallelism for vision/mmproj GGUFs #6416 disabled --split-mode tensor for any GGUF that ships an mmproj projector to dodge a GGML_ASSERT crash (#6415) seen on an older llama.cpp build with consumer Blackwell (sm_120). The blanket skip silently dropped tensor_parallel=true for every multimodal/MTP GGUF (e.g. Qwen3.6-35B-A3B-MTP); on hardware where the model fits on one GPU the load then collapsed to a single GPU. mmproj + --split-mode tensor works on current builds (verified end to end on B200/sm_100), so the skip was disabling a working configuration. Make the vision skip self-healing per binary: - attempt tensor for vision models by default - skip upfront only on a binary already seen to abort on tensor + mmproj this session (_vision_tensor_split_aborts), recorded when such a launch crashes at startup (_record_vision_tensor_split_abort). Process scoped, so a studio update re-probes the new build. The route-level layer-split fallback stays the net. - add _select_gpus(min_gpus=...) so a downgraded tensor request can keep multiple GPUs instead of collapsing to one (default 1, no behavior change). Add tests/test_tp_vision_regression.py: an AST allowlist guard over the tensor_parallel drop sites (which would have flagged #6416), plus cache and _select_gpus coverage. No GPU required. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: address review on vision tensor-parallel self-healing Three fixes from the PR review: - Record a vision-tensor abort only after every startup retry fails. The first version cached the binary on the first spawn crash, which on every build (including capable ones) is the benign --fit step abort that the existing --fit off retry resolves. That poisoned the cache so the next vision load in the same process skipped tensor. Recording now happens at the post-retry failure block (after fit-off, flash-attn-off and MTP-drop), so a binary that actually works is never cached. - Gate the record on the tensor/mmproj crash signature: a hard signal fault (_is_signal_crash) with no non-tensor cause (_output_has_nonprojector_diagnostic excludes OOM and unknown-arch), so an OOM, bad extra args, or MTP/flash-attn crash no longer marks an otherwise capable binary incompatible. - Preserve the multi-GPU request on the cached downgrade. The vision gate now raises _layer_min_gpus to the visible GPU count and threads it through the layer-split GPU selection (_select_gpus min_gpus and the subset loops), so a downgraded tensor request still spreads across GPUs instead of collapsing to a single card the model happens to fit. Verified two vision+tensor loads in one backend process both tensor-split across 4 GPUs (the benign fit abort no longer poisons the cache). Tests updated. * Studio: harden vision tensor-parallel self-healing (review round 2) Address the second review round on the vision/mmproj tensor-parallel fix: - Preserve vision on the first load: a --split-mode tensor + --mmproj GGML_ASSERT now raises so the route-level tensor->layer fallback retries layer split with the projector intact, instead of stripping --mmproj and silently loading text-only (which returned success and skipped the fallback, losing vision on the first load until the next cached load). - Symmetric multi-GPU preservation: the pooled-VRAM tensor downgrade now raises _layer_min_gpus from the usable tensor GPUs like the vision downgrade, so it no longer collapses a multi-GPU request to a single card. - Base the layer fallback minimum on usable GPUs: _select_gpus caps min_gpus to the count of cards with usable VRAM, so a downgrade never forces a nearly-full card in (or trips --fit) just to hit the count. - Re-probe after in-app updates: key the per-binary abort cache on (path, mtime) like _capability_cache, so POST /api/llama/update swapping the binary in place (no backend restart) re-probes the new build instead of inheriting the old build's abort. - Bump _layer_min_gpus for a known-bad vision binary independent of the tensor drop, so the route fallback's layer retry (tensor already off) still spreads across GPUs. Adds deterministic non-GPU regression tests for each. * Studio: gate cached-vision layer minimum on the current tensor request The cached-vision _layer_min_gpus bump fired for every later vision load on a binary recorded as tensor+mmproj-incompatible, including loads that did not request tensor parallelism. A plain non-tensor vision load that fits on one card would then grab every GPU just because an earlier TP attempt aborted in the same backend process. Re-tie the bump to the current tensor request (back inside the tensor-drop guard), so only a downgraded tensor request preserves the multi-GPU spread; a non-tensor vision load minimizes device count as before. * Studio: preserve GPU count + confirm assert on vision tensor fallback Third review round on the vision/mmproj tensor-parallel fix: - Preserve multi-GPU on the first tensor->layer fallback. The route-level retry runs tensor-off, so the in-function downgrades can't see the original tensor request and a fits-on-one-card model loaded the first successful fallback on a single GPU. The GGUF load closure now passes preserve_multi_gpu_on_layer (the toggle asked for tensor, this attempt is layer) and load_model raises _layer_min_gpus for it, so the downgrade still spreads across GPUs. - Cap the auto-context layer loops to usable GPUs. They bypass _select_gpus, so a raised _layer_min_gpus could force a nearly-full card into the subset (or trip --fit). They now start from _auto_min_gpus, capped to the GPUs with usable VRAM. - Confirm the tensor/mmproj assert before caching. Recording (and the layer-retry raise) now require the ggml assert marker via _is_tensor_split_assert, not the bare-signal predicate shared with the projector-incompat branch, so a corrupt or too-new projector that SIGSEGVs independent of split mode is no longer cached as tensor/mmproj-incompatible. Adds deterministic non-GPU regression tests for each. * Studio: extend multi-GPU fallback to extra/env tensor + overhead-aware cap Fourth review round on the vision/mmproj tensor-parallel fix: - Preserve multi-GPU fallback for all tensor requests, not just the UI toggle. Tensor can also be requested via --split-mode tensor in extra args or an inherited LLAMA_ARG_SPLIT_MODE=tensor env; the fallback retries those too, so the preserve_multi_gpu_on_layer hint now keys off _effective_tensor_parallel (the same check the fallback uses), comparing the overall request against the current attempt instead of only request.tensor_parallel. - Cap the auto-context layer fallback to GPUs that can pay the per-device layer overhead. The cap counted any card with positive usable VRAM, so a nearly-full GPU with a few MiB free stayed eligible and could be exposed to llama.cpp and OOM. It now mirrors _select_gpus: a card counts only if usable VRAM exceeds the per-device pipeline overhead. Adds deterministic non-GPU regression tests for both. * Studio: match the #6415 split-axis assert + replay layer-preserve hint Fifth review round on the vision/mmproj tensor-parallel fix: - Narrow the tensor/mmproj crash signature. _is_tensor_split_assert matched any GGML_ASSERT/GGML_ABORT, so an unrelated invariant a corrupt GGUF or projector trips with --mmproj present could be cached as tensor/mmproj-incompatible. It now matches the specific #6415 warmup assertion (GGML_ASSERT(src_ss[0].axis != GGML_BACKEND_SPLIT_AXIS_0) in ggml-backend-meta), whose split-axis signature is inherent to tensor splitting. A reworded future assert just re-crashes-then-falls-back (vision preserved via layer split) instead of poisoning the cache for other models. - Persist the layer-preserve hint for respawns. A successful tensor->layer fallback committed _last_load_kwargs without preserve_multi_gpu_on_layer, so _respawn_if_dead replayed only --split-mode layer + tensor_parallel=False and a mid-session respawn of a fits-on-one-card model came back single-GPU. The hint is now in the replay snapshot, so recovery keeps the multi-GPU placement. Adds deterministic non-GPU regression tests for both. * Studio: tighten comments on the vision tensor-parallel fix Make the comments and docstrings added by this PR succinct: collapse the multi-line block comments in llama_cpp.py / inference.py to one or two lines, trim the verbose test docstrings (the names and assert messages already carry the intent), and shorten the module docstring. No code changes; verified comment-only with scripts/comment_tools.py check --strip-docstrings. * Studio: cache vision tensor abort only on the split-axis token _is_tensor_split_assert also accepted any GGML_ASSERT/GGML_ABORT from ggml-backend-meta, but that file holds many asserts, so an unrelated scheduler/projector/model invariant on an --mmproj launch could cache the binary as tensor/mmproj-incompatible and make later compatible vision models skip tensor parallelism. Match the GGML_BACKEND_SPLIT_AXIS_* token itself (unique to the #6415 warmup assert), not the source file name. * Studio: don't leak the httpx test stub into later tests The regression module stubbed httpx via sys.modules.setdefault, which installs the lightweight stub even when real httpx is present but not yet imported. The stub then persists for the whole pytest process, so provider/HF tests collected later (importing httpx or huggingface_hub.errors) got a module missing HTTPError/Response. Mirror the neighboring llama_cpp helper tests: import real httpx first and only fall back to a stub on ImportError. * Studio: latch the #6415 tensor-split abort on the first spawn, key it per model The self-heal recorded the --split-mode tensor abort only in the post-retry failure block, after the flash-attn-off retry. But SPLIT_MODE_TENSOR requires flash_attn, so the flash-off retry can't run tensor and its output no longer carries the warmup split-axis assert (ggml-backend-meta :541). The record therefore never fired on the real reproducer and the crash loop repeated on every load (reported by oobabooga on #6659). Latch instead on the first spawn that shows the signal crash + split-axis marker: record it, kill the process, and raise straight to the route's layer fallback, skipping the futile flash-attn/MTP retry ladder for this crash. The crash is a tensor-split geometry limit (e.g. MQA n_head_kv=1 splitting to GGML_BACKEND_SPLIT_AXIS_0), not a vision/mmproj property: it reproduces without --mmproj and even single-GPU tensor. So drop the vision/mmproj scoping, rename _vision_tensor_* -> _tensor_split_*, and key the session cache on (binary, mtime, model) rather than (binary, mtime) so one model's abort no longer skips tensor for every other model on the same build. Regression tests updated to pin the early-spawn record, the per-model cache, and that an unrelated ggml-backend-meta assert is not treated as the marker. * Studio: reload on explicit tensor-off after a multi-GPU layer fallback When a tensor load is downgraded to layer but kept multi-GPU to honor the tensor request (preserve_multi_gpu_on_layer, the geometry-cache gate, or the budget downgrade), the server reports tensor_parallel=False with --split-mode layer stored. A later Apply that explicitly turns the tensor toggle off then matched the loaded state and deduped to already_loaded, so Studio kept the fallback's all-GPU CUDA_VISIBLE_DEVICES placement instead of re-selecting normal placement (a single GPU for a model that fits on one card). Latch a _layer_preserves_tensor_intent flag in load_model whenever a tensor request is downgraded to layer with the multi-GPU floor raised (_layer_min_gpus > 1), clear it when tensor stays on or on unload, and force a reload in _request_matches_loaded_settings when the user explicitly turns the tensor toggle off while that flag is set. An Apply that does not touch the toggle still dedupes, so a working multi-GPU layer server is not churned. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: address reviewer.py findings on the tensor-split self-heal P1 (dedup): tensor intent can be dropped via extras, not only the toggle. An explicit llama_extra_args=["--split-mode", "layer"] matches the stored fallback extras, so _request_matches_loaded_settings deduped to the preserved all-GPU placement instead of reloading. Now reload when layer_preserves_tensor_intent and the user explicitly drops tensor via the toggle OR via extras (_effective_tensor_parallel of the explicit extras is false). P1 (downgrade symmetry): the len(tp_gpus) < 2 compute-buffer downgrade cleared tensor_parallel without raising _layer_min_gpus, unlike the budget and geometry downgrades. GPUs below tensor's replicated compute-buffer reserve can still take layer split's lower overhead, so keep the multi-GPU request (len(gpus) >= 2) and let _select_gpus cap unusable cards. P2 (cache key): key the tensor-split abort cache on st_mtime_ns, so a binary replaced in place within the same second after an abort is re-probed instead of inheriting the stale entry. P2 (test hygiene): load routes/inference.py via importlib in the regression tests instead of importing the routes package, which runs routes/__init__.py and pulls in every router (e.g. python-multipart). Added regression coverage for the extras-off reload, the compute-buffer multi-GPU preservation, and the same-second nanosecond cache invalidation. * Studio: record the tensor-split abort on the Windows CRT abort exit too The first-spawn split-axis latch only recorded when _is_signal_crash matched (POSIX signal or 0xC0000000+ NTSTATUS). On MSVC builds GGML_ASSERT terminates through the CRT abort() path with exit code 3, which is neither, so the cache never filled on Windows and every later load of the same bad binary/model repeated the tensor crash before falling back to layer. The split-axis marker is definitive, so accept either a signal crash or the Windows abort() exit (3) when the marker is present. Add _is_abort_exit and a unit test, and assert the early latch honors it. * Studio: fix UnboundLocalError on --fit-on fallback, reload backend fast path Two follow-ups from review on the tensor-split self-heal: UnboundLocalError: _layer_min_gpus was initialized inside the GPU-selection try. If NVML probing or GGUF/mmproj sizing raised, the except path logged "using --fit on" and fell through to the command builder, where the new self._layer_preserves_tensor_intent = _layer_min_gpus > 1 then raised, turning a safe --fit-on layer fallback into a hard load failure. Bind _layer_min_gpus before the try so the except path always has it. Backend fast path: _request_matches_loaded_settings forces a reload when a preserved tensor->layer fallback gets an explicit tensor-off request, but load_model's own _already_in_target_state still matched the tensor-off/layer settings and short-circuited, so the placement re-selection never ran. Mirror the guard there: reload when layer_preserves_tensor_intent and the request drops tensor intent. The flag clears on that reload, so there's no loop. Added regression coverage for both. * Studio: testable tensor-split record decision; skip futile fit-off retry Follow-ups from a deeper review of the tensor-split self-heal: Extract the record decision into _should_record_tensor_split_abort(rc, output) (marker AND (signal crash OR Windows abort)) and call it from the early latch. The combined boolean was only covered by source-inspection substring checks, so an or->and typo would silently stop recording on Windows (CRT abort exit 3 is not a signal) with every test still green. Add a behavioral test over the POSIX / Windows / NTSTATUS / clean-exit / SIGKILL / no-marker matrix. Skip the --fit off retry inside _spawn_and_wait when the crash already shows the split-axis marker: that abort is fit-independent, so the retry just warms up and crashes a second time before the latch records it. Skipping it lets the caller latch immediately and corrects the latch comment. Also clarify the dedup-guard comments (toggle read from model_fields_set vs extras via _effective_tensor_parallel without env; the backend fast path is intentionally broader and only ever forces a reload). * Studio: don't reload-loop tensor-off requests under env tensor The preserved-fallback reload guard fired on the raw tensor toggle, ignoring LLAMA_ARG_SPLIT_MODE=tensor. For an env-driven tensor user, an explicit tensor_parallel=false request then forced a reload that re-engaged tensor via the env and re-created the same preserved layer fallback, so every /load reloaded -- bypassing the env-downgrade matching that exists to avoid exactly this loop. Gate the guard on the env-aware effective tensor state: reload only when an explicit toggle/extras change leaves _effective_tensor_parallel (which consults the env) off. If the env still forces tensor, fall through to the existing env-downgrade match, which dedupes instead of looping. Added a regression test with LLAMA_ARG_SPLIT_MODE=tensor set. * Studio: tighten comments and test docstrings on the TP self-heal Condense the verbose comments and test docstrings added across the review rounds into fewer, succinct lines without changing their intent: the early-latch and downgrade-site rationale, the cache/key and helper docstrings, the dedup-guard comments, and the per-test docstrings. No code changes (AST-verified comments and docstrings only); tests and lint unchanged. * Studio: clear preserved tensor flag on diffusion; carry it across non-drop reloads Two follow-ups on the preserved-fallback machinery: Diffusion: the DiffusionGemma path early-returns from load_model before the command builder that sets/clears _layer_preserves_tensor_intent, so the flag from a prior tensor->layer fallback leaked onto a later diffusion load and forced needless reloads of the diffusion server on tensor-off/extra Applies. Clear it when starting diffusion. Settings reload: the preserve hint was recomputed only from the new request, so a reload for an unrelated setting (e.g. max_seq_length) with the tensor toggle omitted dropped a preserved multi-GPU layer placement back to one GPU. Carry llama_backend.layer_preserves_tensor_intent into the hint when the request is not an explicit tensor-off/extras-off drop, so a fitting model stays multi-GPU. Added regression tests for the diffusion clear, the carry-forward, and the updated tensor-intent computation. * Studio: gate the preserve carry-forward on the same model being loaded The tensor-intent carry-forward read llama_backend.layer_preserves_tensor_intent without checking it belonged to the model being loaded. On a direct model switch (load B without an explicit /unload of A), the flag is still set from A's downgrade (it isn't reset until B's load_model reaches the command builder, after the route reads it), so a plain load of B got preserve_multi_gpu_on_layer=True and was spread across all GPUs even though it fits on one and the user never requested tensor for it. The backend dedup doesn't have this leak (it checks model_identifier first); the leak was only in the route hint. Extract the decision into _carry_preserved_tensor_intent(preserved, same_model, explicit_drop) and gate it on the backend still holding the same model. Add a behavioral truth-table test (catches a `not` inversion and a missing same-model guard) and tighten the compute-buffer downgrade test to bound its source window. * Studio: match the HF quant too when carrying preserved tensor intent The same-model guard on the preserve carry-forward compared only model_identifier, which is variant-agnostic for HF repos. A later load of the same repo with a different gguf_variant (which already bypassed dedupe on the variant mismatch) was treated as the same model, so a request that omits tensor settings inherited the prior variant's preserved intent and forced multi-GPU layer placement for a quant that never requested tensor. Also require the loaded hf_variant to match for HF repos (local direct-file loads already differ by model_identifier path). Added a regression test for the variant guard. * Studio: match the loaded GGUF by path too when carrying preserved tensor intent A local directory holding multiple GGUF variants keeps one variant-agnostic model_identifier (the directory) while config.gguf_file selects the file, so the same-model guard let variant B inherit variant A's preserved tensor->layer fallback and forced B onto multi-GPU. Mirror _already_in_target_state's identity logic: match by resolved path when both sides have a local file, else by HF variant. #6659 * Studio: let implicit same-settings reloads dedupe after a preserved fallback The backend _already_in_target_state mirror forced a reload on ANY effective tensor-off request once a tensor->layer fallback was preserved. In the HF auto-pick / local-directory flows the route-level dedup is skipped, so an identical /load with tensor omitted reached this guard and reloaded every time even without an explicit drop. Thread the route's preserve_multi_gpu_on_layer decision in so only an explicit drop reloads; implicit carry-forward dedupes. #6659 * Studio: only an explicit tensor/split-mode change drops preserved intent The explicit-drop test treated request.llama_extra_args is not None as a drop, so a same-model reload that merely added an unrelated pass-through arg (e.g. --top-k 20) without touching the tensor field or --split-mode disabled the carry-forward and collapsed a fitting model back to one GPU. A drop now requires an explicit tensor_parallel field change or a non-tensor --split-mode override, via a shared _is_explicit_tensor_drop helper used by both the already-loaded dedup and the load carry-forward so the two readers agree. #6659 * Studio: treat an explicit clear of extras as a tensor drop When tensor intent was extras-driven (--split-mode tensor) and fell back to a preserved layer split, a later request that explicitly clears extras (llama_extra_args=[]) but omits tensor_parallel left the empty list with no split-mode override, so the carry-forward kept the model pinned multi-GPU instead of returning to normal layer selection. _is_explicit_tensor_drop now also counts an explicit empty-list clear as a drop, while an unrelated extra (--top-k) or inherit (None) still carries the preserved intent. #6659 * Studio: don't treat the UI's tensor_parallel echo as a tensor drop The Studio frontend always sends tensor_parallel and copies the /load response's resolved value back into its state, so after a tensor->layer fallback every ctx/settings reload carries tensor_parallel=false even though the user never changed it. Keying the drop on the field (or on an empty extras clear) collapsed the preserved multi-GPU placement on the next reload. A fallback also always stores --split-mode layer, never a tensor split mode, so a clear never wipes tensor intent. _is_explicit_tensor_drop now drops only on an explicit non-tensor --split-mode override; the bare field echo, an empty clear, an unrelated extra, and inherit all keep the preserved placement, and --split-mode tensor / tensor_parallel=true re-engage tensor. #6659 * Studio: match the resolved config.identifier when carrying tensor intent The same-model guard for the carry-forward compared the raw request id, but ModelConfig.from_identifier normalizes it (adds the unsloth/ prefix for a shorthand, fixes repo-id case) before load_model stores config.identifier. So a ctx/settings reload using the shorthand id missed the match, dropped _carry_preserved_tensor_intent, and could collapse a preserved multi-GPU layer placement to one GPU. Compare against config.identifier (what the backend stores), keeping it symmetric with _already_in_target_state. #6659 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|---|---|---|
| .github | ||
| images | ||
| scripts | ||
| studio | ||
| tests | ||
| unsloth | ||
| unsloth_cli | ||
| .gitattributes | ||
| .gitignore | ||
| .pre-commit-ci.yaml | ||
| .pre-commit-config.yaml | ||
| build.sh | ||
| cli.py | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| COPYING | ||
| install.ps1 | ||
| install.sh | ||
| LICENSE | ||
| pyproject.toml | ||
| README.md | ||
| unsloth-cli.py | ||
Unsloth Studio lets you run and train models locally.
Features • Quickstart • Notebooks • Documentation
⚡ Get started
macOS, Linux, WSL:
curl -fsSL https://unsloth.ai/install.sh | sh
Windows:
irm https://unsloth.ai/install.ps1 | iex
Community:
⭐ Features
Unsloth Studio (Beta) lets you run and train text, audio, embedding, vision models on Windows, Linux and macOS.
Inference
- Search + download + run models including GGUF, LoRA adapters, safetensors
- Export models: Save or export models to GGUF, 16-bit safetensors and other formats.
- Tool calling: Support for self-healing tool calling and web search
- Code execution: lets LLMs test code in Claude artifacts and sandbox environments
- API inference endpoint: Deploy and run local LLMs in Claude Code, Codex tools with Unsloth
- Auto set inference settings and customize chat templates.
- We work directly with teams behind gpt-oss, Qwen3, Llama 4, Mistral, Gemma 1-3, and Phi-4, where we’ve fixed bugs that improve model accuracy.
- Chat with images, audio, PDFs, code, DOCX and more. Connect API providers (OpenAI, Anthropic) or servers (vLLM, Ollama).
Training
- Train and RL 500+ models up to 2x faster with up to 70% less VRAM, with no accuracy loss.
- Custom Triton and mathematical kernels. See some collabs we did with PyTorch and Hugging Face.
- Data Recipes: Auto-create datasets from PDF, CSV, DOCX etc. Edit data in a visual-node workflow.
- Reinforcement Learning (RL): The most efficient RL library, using 80% less VRAM for GRPO, FP8 etc.
- Supports full fine-tuning, RL, pretraining, 4-bit, 16-bit and, FP8 training.
- Observability: Monitor training live, track loss and GPU usage and customize graphs.
- Multi-GPU training is supported, with major improvements coming soon.
📥 Install
Unsloth can be used in two ways: through Unsloth Studio, the web UI, or through Unsloth Core, the code-based version. Each has different requirements.
Unsloth Studio (web UI)
Unsloth Studio (Beta) works on Windows, Linux, WSL and macOS.
- CPU: Supported for Chat and Data Recipes currently
- NVIDIA: Training works on RTX 30/40/50, Blackwell, DGX Spark, Station and more
- macOS: Training, MLX and GGUF inference are ALL supported.
- AMD: Chat + Data works. Train with Unsloth Core. Studio support is out soon.
- Multi-GPU: Available now, with a major upgrade on the way
macOS, Linux, WSL:
curl -fsSL https://unsloth.ai/install.sh | sh
Use the same command to update.
Windows:
irm https://unsloth.ai/install.ps1 | iex
Use the same command to update.
Launch
unsloth studio -p 8888
For cloud or global access, add -H 0.0.0.0. By default, Unsloth is accessible only locally.
To reach Studio over HTTPS, use unsloth studio --secure. Studio stays bound to localhost and is reached only through a free Cloudflare tunnel, which publishes it at a public https://*.trycloudflare.com URL (it fails closed if the tunnel can't start, so the raw port is never exposed). This makes Studio reachable from the internet, so anyone with the link and API key can use it and run code: keep your API key private (see Remote access below).
Docker
Use our Docker image unsloth/unsloth container. Run:
docker run -d -e JUPYTER_PASSWORD="mypassword" \
-p 8888:8888 -p 8000:8000 -p 2222:22 \
-v $(pwd)/work:/workspace/work \
--gpus all \
unsloth/unsloth
Developer, Nightly, Uninstall
To see developer, nightly and uninstallation etc. instructions, see advanced installation.
Unsloth Core (code-based)
Linux, WSL:
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv unsloth_env --python 3.13
source unsloth_env/bin/activate
uv pip install unsloth --torch-backend=auto
Windows:
winget install -e --id Python.Python.3.13
winget install --id=astral-sh.uv -e
uv venv unsloth_env --python 3.13
.\unsloth_env\Scripts\activate
uv pip install unsloth --torch-backend=auto
For Windows, pip install unsloth works only if you have PyTorch installed. Read our Windows Guide.
You can use the same Docker image as Unsloth Studio.
AMD, Intel:
For RTX 50x, B200, 6000 GPUs: uv pip install unsloth --torch-backend=auto. Read our guides for: Blackwell and DGX Spark.
To install Unsloth on AMD and Intel GPUs, follow our AMD Guide and Intel Guide.
📒 Free Notebooks
Train for free with our notebooks. You can use our new free Unsloth Studio notebook to run and train models for free in a web UI. Read our guide. Add dataset, run, then deploy your trained model.
| Model | Free Notebooks | Performance | Memory use |
|---|---|---|---|
| Gemma 4 (E2B) | ▶️ Start for free | 1.5x faster | 50% less |
| Qwen3.5 (4B) | ▶️ Start for free | 1.5x faster | 60% less |
| gpt-oss (20B) | ▶️ Start for free | 2x faster | 70% less |
| Qwen3.5 GSPO | ▶️ Start for free | 2x faster | 70% less |
| gpt-oss (20B): GRPO | ▶️ Start for free | 2x faster | 80% less |
| Qwen3: Advanced GRPO | ▶️ Start for free | 2x faster | 70% less |
| embeddinggemma (300M) | ▶️ Start for free | 2x faster | 20% less |
| Mistral Ministral 3 (3B) | ▶️ Start for free | 1.5x faster | 60% less |
| Llama 3.1 (8B) Alpaca | ▶️ Start for free | 2x faster | 70% less |
| Llama 3.2 Conversational | ▶️ Start for free | 2x faster | 70% less |
| Orpheus-TTS (3B) | ▶️ Start for free | 1.5x faster | 50% less |
- See all our notebooks for: Kaggle, GRPO, TTS, embedding & Vision
- See all our models and all our notebooks
- See detailed documentation for Unsloth here
🦥 Unsloth News
- Connections: Connect any API provider (OpenAI, Anthropic) or server (vLLM, Ollama). Guide
- MTP: Run Qwen3.6 MTP in Unsloth. MTP settings are autoset specific to your hardware. Guide
- API inference endpoint: Deploy and run local LLMs in Claude Code, Codex tools. Guide
- Qwen3.6: Qwen3.6-35B-A3B can now be trained and run in Unsloth Studio. Blog
- Gemma 4: Run and train Google’s new models directly in Unsloth. Blog
- Introducing Unsloth Studio: our new web UI for running and training LLMs. Blog
- Qwen3.5 - 0.8B, 2B, 4B, 9B, 27B, 35-A3B, 112B-A10B are now supported. Guide + notebooks
- Train MoE LLMs 12x faster with 35% less VRAM - DeepSeek, GLM, Qwen and gpt-oss. Blog
- Embedding models: Unsloth now supports ~1.8-3.3x faster embedding fine-tuning. Blog • Notebooks
- New 7x longer context RL vs. all other setups, via our new batching algorithms. Blog
- New RoPE & MLP Triton Kernels & Padding Free + Packing: 3x faster training & 30% less VRAM. Blog
- 500K Context: Training a 20B model with >500K context is now possible on an 80GB GPU. Blog
- FP8 & Vision RL: You can now do FP8 & VLM GRPO on consumer GPUs. FP8 Blog • Vision RL
📥 Advanced Installation
The below advanced instructions are for Unsloth Studio. For Unsloth Core advanced installation, view our docs.
Developer / Nightly / Experimental installs: macOS, Linux, WSL:
The developer install builds from the main branch, which is the latest (nightly) source.
git clone https://github.com/unslothai/unsloth
cd unsloth
./install.sh --local
unsloth studio -p 8888
To install into an isolated location (its own virtual env, auth/, studio.db, cache and llama.cpp build), set UNSLOTH_STUDIO_HOME and pass it again at launch:
UNSLOTH_STUDIO_HOME="$PWD/.studio" ./install.sh --local
UNSLOTH_STUDIO_HOME="$PWD/.studio" unsloth studio -p 8888
Then to update :
cd unsloth && git pull
./install.sh --local
unsloth studio -p 8888
Developer / Nightly / Experimental installs: Windows PowerShell:
The developer install builds from the main branch, which is the latest (nightly) source.
git clone https://github.com/unslothai/unsloth.git
cd unsloth
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\install.ps1 --local
unsloth studio -p 8888
To install into an isolated location (its own virtual env, auth/, studio.db, cache and llama.cpp build), set UNSLOTH_STUDIO_HOME and pass it again at launch:
$env:UNSLOTH_STUDIO_HOME="$PWD\.studio"; .\install.ps1 --local
$env:UNSLOTH_STUDIO_HOME="$PWD\.studio"; unsloth studio -p 8888
Then to update :
cd unsloth; git pull
.\install.ps1 --local
unsloth studio -p 8888
Remote access: --secure (HTTPS tunnel) vs raw port
By default unsloth studio binds to 127.0.0.1 (this machine only). To reach it from another device, pick one of:
--secure(recommended): serve only through a free Cloudflare HTTPS link. Studio stays bound to localhost and the tunnel provides the public URL; it fails closed (does not start) if the tunnel can't come up, so the raw port is never exposed.
unsloth studio --secure -p 8888
-H 0.0.0.0: bind the raw port on all network interfaces, reachable from anywhere on the network. Only use this on a trusted network.
unsloth studio -H 0.0.0.0 -p 8888
Server-side tools (web search, Python and terminal code execution) run as your user and are on by default. Anyone who can reach the server with the API key can run code on this machine, so keep your API key private and pass --disable-tools when exposing Studio.
Advanced launch options
Installer options can be passed as environment variables. On macOS, Linux and WSL place the variable after the pipe so the shell passes it to sh; on Windows set it with $env: before piping to iex.
Skip PyTorch (GGUF-only mode):
curl -fsSL https://unsloth.ai/install.sh | UNSLOTH_NO_TORCH=1 sh
$env:UNSLOTH_NO_TORCH=1; irm https://unsloth.ai/install.ps1 | iex
Pin the Python version:
curl -fsSL https://unsloth.ai/install.sh | UNSLOTH_PYTHON=3.12 sh
$env:UNSLOTH_PYTHON='3.12'; irm https://unsloth.ai/install.ps1 | iex
Install to a custom location with UNSLOTH_STUDIO_HOME:
curl -fsSL https://unsloth.ai/install.sh | UNSLOTH_STUDIO_HOME=/abs/path sh
$env:UNSLOTH_STUDIO_HOME='C:\path'; irm https://unsloth.ai/install.ps1 | iex
On macOS, the installer defaults to the system certificate store (UV_SYSTEM_CERTS=1) so uv trusts the CAs in your Keychain, needed behind TLS-inspecting proxies (Cisco Umbrella, Zscaler, etc.). Opt out with:
curl -fsSL https://unsloth.ai/install.sh | UV_SYSTEM_CERTS=0 sh
Point the frontend build at a corporate npm mirror/proxy with UNSLOTH_NPM_REGISTRY (for the developer install behind a firewall that blocks registry.npmjs.org):
UNSLOTH_NPM_REGISTRY=https://artifactory.example.com/api/npm/npm/ ./install.sh --local
$env:UNSLOTH_NPM_REGISTRY='https://artifactory.example.com/api/npm/npm/'; .\install.ps1 --local
It is threaded as --registry into the Studio frontend npm/bun installs; the supply-chain locks (7-day min-release-age, exact version pins) stay in force.
Cap Studio's native CPU thread pools on high-core hosts: UNSLOTH_CPU_THREADS=8 unsloth studio -p 8888.
Uninstall
The recommended way to fully remove Unsloth Studio is the matching uninstall script for your OS. It stops any running servers, removes the install dir, the launcher data dir, the desktop shortcut, and any platform-specific entries (macOS .app bundle + Launch Services on Mac; Start Menu, HKCU\Software\Unsloth registry key and user PATH entries on Windows):
- MacOS, WSL, Linux:
curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.sh | sh - Windows (PowerShell):
irm https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.ps1 | iex
If you only want to drop the install dir and keep the launcher/shortcut for a later reinstall, you can instead run rm -rf ~/.unsloth/studio (Mac/Linux/WSL) or Remove-Item -Recurse -Force "$HOME\.unsloth\studio" (Windows). The model cache at ~/.cache/huggingface is not touched by any of these.
For more info, see our docs.
Deleting model files
You can delete old model files either from the bin icon in model search or by removing the relevant cached model folder from the default Hugging Face cache directory. By default, HF uses:
- MacOS, Linux, WSL:
~/.cache/huggingface/hub/ - Windows:
%USERPROFILE%\.cache\huggingface\hub\
💚 Community and Links
| Type | Links |
|---|---|
 Discord |
Join Discord server |
 r/unsloth Reddit |
Join Reddit community |
| 📚 Documentation & Wiki | Read Our Docs |
_logo_late_2025.svg){kind=logo} Twitter (aka X) |
Follow us on X |
| 🔮 Our Models | Unsloth Catalog |
| ✍️ Blog | Read our Blogs |
Citation
You can cite the Unsloth repo as follows:
@software{unsloth,
author = {Daniel Han, Michael Han and Unsloth team},
title = {Unsloth},
url = {https://github.com/unslothai/unsloth},
year = {2023}
}
If you trained a model with 🦥Unsloth, you can use this cool sticker!
License
Unsloth uses a dual-licensing model of Apache 2.0 and AGPL-3.0. The core Unsloth package remains licensed under Apache 2.0, while certain optional components, such as the Unsloth Studio UI are licensed under the open-source license AGPL-3.0.
This structure helps support ongoing Unsloth development while keeping the project open source and enabling the broader ecosystem to continue growing.
Thank You to
- The llama.cpp library that lets users run and save models with Unsloth
- The Hugging Face team and their libraries: transformers and TRL
- The Pytorch and Torch AO team for their contributions
- NVIDIA for their NeMo DataDesigner library and their contributions
- And of course for every single person who has contributed or has used Unsloth!