mirror of
https://github.com/unslothai/unsloth.git
synced 2026-07-09 15:58:41 +00:00
72e67ae5a6
* Studio: Add Tensor-Parallel llama.cpp support * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden Tensor-Parallel fallback and GPU selection * Studio: reconcile split-mode extras and harden tensor-split planning * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: reconcile split-mode extras in backend duplicate-load guard * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: preserve inherited non-tensor split modes on reload * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: honor cancellation in tensor fallback, preserve tensor mode on rollback, and don't raise an explicit small context * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: reconcile split-mode in reload check and strip it on tensor downgrade * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Strip --tensor-split alongside --split-mode so inherited ratios don't override the tensor planner An inherited or stale --tensor-split in llama_extra_args was appended after Studio's computed --tensor-split and won last in llama.cpp, re-introducing the asymmetric-GPU OOM tensor mode is meant to prevent. Group -ts/--tensor-split into the split-mode shadow set so it is stripped on inherit and on the layer fallback; parse_split_mode_override still keys on the mode value only. * Drop quantized KV for the tensor attempt and report native max context Tensor mode aborts on a quantized KV cache, so a user with q8_0/q4_1 etc. who enabled Tensor Parallelism silently fell back to layer split. Clear the cache type (and strip inherited/explicit --cache-type) for the tensor attempt only; the layer fallback re-runs with tensor off and keeps the user's choice. Also report max_available_ctx from the native context, not an explicit small -c, so the context slider no longer warns too early in tensor mode. * Reconcile inherited split-mode extras in the already-loaded check When a same-model load omitted llama_extra_args, the tensor comparison resolved the raw (None) request and treated an inherited --split-mode tensor server as a mismatch, forcing a needless reload. Compare using the stored extras stripped the same way the reload strips them. * Pass tensor_parallel through compare-mode loads The generalized compare path loaded each GGUF without tensor_parallel, so compare ran layer split even with the toggle on and left the settings sheet stale. Send the toggle and hydrate the loaded state from the response, matching the main chat and recipe load paths. * Add --tensor-parallel flag to unsloth studio run The headless one-liner could only reach tensor mode by passing --split-mode tensor as a raw llama.cpp extra. Add a first-class --tensor-parallel/ --no-tensor-parallel option that sets the tensor_parallel field on the /api/inference/load payload, forwarded through the studio-venv re-exec like the other polarity flags. Matches the web UI toggle and the API field. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <michaelhan2050@gmail.com> |
||
|---|---|---|
| .. | ||
| commands | ||
| tests | ||
| __init__.py | ||
| _inference.py | ||
| _tool_policy.py | ||
| config.py | ||
| options.py | ||