mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-25 08:42:25 +00:00
5ec9a599ee
* Studio: filter DSpark/DFlash drafters out of the GGUF quant picker unsloth/DeepSeek-V4-Flash-0731-GGUF ships its speculative-decoding drafters under dspark/, and those filenames carry BF16 and Q8_0 quant tokens. The repo has no real root-level BF16 or Q8_0, so the picker gained two phantom quants at 11 GB in a repo whose real quants are all 87 GB and up, and is_main_gguf_variant_path matched them, making a drafter the main file of a bf16 plan. Broaden the drafter predicate to mtp, dspark and dflash (the same DeepSeek drafter under its general.architecture name), matching by basename prefix or exact parent directory only. The kind names double as family names, so Qwen3.6-27B-MTP-Q4_K_M.gguf and Qwen3.6-35B-A3B-DFlash-Q4_K_M.gguf stay selectable. Keep _cached_repo_mtp_drafter narrow via a new _is_mtp_only_drafter_path. It uses the companion predicate inversely to find a drafter to launch, and DSpark needs --spec-type draft-dspark plus --fit off, so it must not pick one up. * Tighten drafter predicate comments * Match dflash drafters by basename prefix only, not by directory A local GGUF under a folder named dflash/ (say /models/dflash/Qwen3.6-35B-A3B-DFlash-Q4_K_M.gguf) was classified as a drafter, so detect_gguf_model returned None from both the direct-file and the directory scan and the model vanished. DFlash is a speculative-decoding technique name, so it lands in hundreds of Hub repo and model names and is a folder name a user picks for real weights. No published repo uses dflash/ as a companion directory, while mtp/ and dspark/ only ever appear as one. Real DFlash drafters keep the dflash- basename prefix (ggml-org/Qwen3.6-27B-GGUF ships dflash-Qwen3.6-27B-BF16.gguf next to the full Qwen3.6-27B-BF16.gguf), so prefix matching alone still catches them. * Keep drafter-named repos listable when they have no main model The dflash- prefix has to stay: ggml-org/Qwen3.6-27B-GGUF ships a 3.47 GB dflash-Qwen3.6-27B-BF16.gguf next to the real 53.81 GB one, and without the prefix rule the drafter merges into that quant group. But the prefix also matches repos whose family name IS the model name, and those lost every quant: mradermacher names each file after the repo, so all 11 real quants of DFlash-Qwen3.5-27B-Uncensored-GGUF carried it. Drafter-only repos went the same way. A companion is only a companion when it has something to accompany, so a prefix-named drafter is kept when no main GGUF sits beside it in the same listing. A directory-named one (mtp/, dspark/) is never reprieved: the publisher laid it out as a companion, and a snapshot holding only that is a half-downloaded repo rather than a model. Threaded through the listers, the plan builder and the local inventory so the layers cannot disagree. Swept 459 real GGUF repos: 5 change, and they are exactly the ones that listed nothing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope the drafter reprieve to remote listings and fix two holes in it Three problems with the reprieve added in 09d8652, all found in review. The utils mirror never got it, so a local folder of DFlash-named GGUFs still vanished from detect_gguf_model and the utils variant lister while the hub lister happily listed it. Chasing that down showed the reprieve cannot work on local directories at all: a folder holding only a companion is a half-downloaded repo, and it is structurally identical to a folder holding a drafter-named model. The existing companion-only contracts are the right behaviour there, so the reprieve is now scoped to whole-repo listings (the remote picker feed, download plans, detect_gguf_model_remote) and the local scanners are back to plain per-path filtering. preferred_mtp_sibling had already folded a reprieved root mtp-*.gguf into companions_expected, so the same file landed in a plan twice and doubled main_size_bytes and download_size_bytes. drafter_paths_in dropped non-GGUF files before asking whether a main model was present, so a repo of safetensors plus a root drafter got the drafter advertised as a selectable GGUF variant. A reprieved file must also carry a quant token, which keeps the bare mtp-<model>.gguf rejected: loading it as the model would pair it with itself (-m drafter --model-draft drafter). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Give every whole-repo consumer the same drafter set Scoping the reprieve to remote listings left four consumers still asking the per-path predicate, so each disagreed with what the picker advertised. The loader's variant resolution rejected every candidate in a drafter-named repo and fell through to a fabricated <repo>-<variant>.gguf name, so an advertised quant could not be loaded. Auto-download admission skipped the same siblings and cached the repo as not servable. Both now use the listing-aware set. An HF cache snapshot is the same whole-repo listing the remote path sees, so list_local_gguf_variants takes a whole_repo flag and the two snapshot callers pass it. Without that a fully cached drafter-named repo vanished from the picker as soon as the network went away. An arbitrary folder still gets the stricter per-path filter. common.py went back to main. It had kept a listing-aware _is_main_gguf_filename from an earlier revision of this branch, which let inventory advertise a lone companion GGUF that the load path then refused to resolve. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: restrict the drafter reprieve to family-named kinds A drafter GGUF with no main weight beside it in the same repo listing is that listing's own weight, which is what keeps repos like mradermacher/DFlash-Qwen3.5-27B-Uncensored-GGUF selectable. Only kinds whose names are also model family names can be reprieved: no family is named MTP, so an mtp- file is always a companion and can never be promoted into becoming its own --model-draft. Also make the whole-repo consumers agree with the picker: the HF cache detection, cache inventory scans, the loader's quant-token mirror, the low-disk fallback and the whole-repo scan all read the full listing now, so offline resolves what online resolves. * Studio: finish converting the whole-repo drafter consumers The reprieve needs the whole listing, so every consumer holding one has to read it that way or it disagrees with the picker. Remaining ones: the runtime listers behind /load and the memory guard, the cached variant resolvers behind the KV estimate and the cached-path endpoints, the local model resolver, the chat-template scan, and three sites in gguf_variants. Deletion moves as one unit. Per path it matched nothing on a repo whose quants all carry the family prefix, so deleting a variant was a no-op; fixing only the matcher would then arm the companion sweep against every other quant the user had downloaded. Both now come from the same reprieve-aware main set, so a companion is only what the main set excludes. Two listing bugs behind it: a dangling symlink is a collected blob, not an absent file, and dropping it hid a repo's main weight and reprieved the drafter in its place; and the loader read mmproj off the basename, so an mmproj/ subdir copy counted as a weight for the drafter to accompany. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: drop the drafter predicates deletion no longer imports The main/companion split now comes from the reprieve-aware set, so the three per-path helpers are dead here. * Studio: filter drafters unconditionally, dropping the listing reprieve Keeping drafter-named repos listable meant deciding "drafter or model?" from the whole listing, which a per-path predicate cannot do alone. Every consumer holding a listing then had to agree: the listers, the plan builder, the loader, cache inventory, deletion, the KV estimate and the cached-path endpoints. Review kept finding another that did not, and the reprieve could arm the companion sweep in variant deletion against a repo's other quants. Back to what this set out to do: mtp-, dspark- and dflash- named GGUFs are companions, always. The predicate is per path and the exclusion call sites inherit it unchanged, so nothing else in the backend moves. A repo naming every quant after the drafter family now lists none, which is the accepted cost. MTP is untouched: the root mtp-*.gguf is still fetched with every variant and still launched as --spec-type draft-mtp. * Studio: keep opt-in DSpark drafters out of the companion delete sweep Deleting a repo's last main quant reclaims its companions, on the grounds that they were downloaded with every variant. DSpark is deliberately in no variant plan, so Studio never fetched it, and broadening the selection predicate had put it in that sweep: deleting a quant would take an ~11 GB file the user fetched deliberately, plus its blob. The sweep now uses is_mtp_only_drafter_path, which covers exactly the companions a variant download actually brings in. * Studio: tighten the drafter predicate comments * Studio: import the MTP-only drafter check where it is used Routing it through hub.services.models.common made it an unused import there, which the import-hoist lint blocks. deletion.py already imports from hub.utils.gguf, so take it straight from the canonical module. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
111 lines
3.7 KiB
Python
111 lines
3.7 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""Tests for :func:`routes.models._resolve_quant_gguf` (PR #6364 follow-up).
|
|
|
|
The /kv-cache-estimate resolver must mirror list_local_gguf_variants:
|
|
- read the quant label from the snapshot-relative path so nested layouts like
|
|
``BF16/model.gguf`` resolve (not just basenames),
|
|
- skip MTP drafter files so a ``...-Q8_0-MTP.gguf`` drafter is never returned as
|
|
the Q8_0 weights, and
|
|
- when several cache snapshots hold the quant, pick the most complete (largest
|
|
total) so a partial older revision can't underestimate the weight bytes.
|
|
|
|
No GPU/network. The resolver only stats sizes and parses file names, so the
|
|
GGUF files can be arbitrary bytes.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sys
|
|
import types
|
|
from pathlib import Path
|
|
|
|
# Keep this test runnable without optional logging deps (mirrors
|
|
# test_cached_gguf_routes.py).
|
|
if "structlog" not in sys.modules:
|
|
|
|
class _DummyLogger:
|
|
def __getattr__(self, _name):
|
|
return lambda *args, **kwargs: None
|
|
|
|
sys.modules["structlog"] = types.SimpleNamespace(
|
|
BoundLogger = _DummyLogger,
|
|
get_logger = lambda *args, **kwargs: _DummyLogger(),
|
|
)
|
|
|
|
import routes.models as models_route
|
|
|
|
|
|
def _write(path: Path, size: int) -> Path:
|
|
path.parent.mkdir(parents = True, exist_ok = True)
|
|
path.write_bytes(b"\0" * size)
|
|
return path
|
|
|
|
|
|
def test_resolves_quant_from_parent_directory_layout(tmp_path):
|
|
# A repo that puts the quant label in a parent dir (BF16/model.gguf).
|
|
root = tmp_path / "repo"
|
|
f = _write(root / "BF16" / "model.gguf", 1234)
|
|
|
|
path, total = models_route._resolve_quant_gguf(str(root), "BF16", is_local = True)
|
|
|
|
assert path == str(f)
|
|
assert total == 1234
|
|
|
|
|
|
def test_skips_mtp_drafter_for_main_weights(tmp_path):
|
|
# Main Q8_0 weights next to a same-quant MTP drafter that sorts first by name.
|
|
root = tmp_path / "repo"
|
|
main = _write(root / "model-Q8_0.gguf", 100)
|
|
_write(root / "MTP" / "model-Q8_0-MTP.gguf", 50)
|
|
|
|
path, total = models_route._resolve_quant_gguf(str(root), "Q8_0", is_local = True)
|
|
|
|
assert path == str(main)
|
|
# Drafter bytes are excluded from the weight total.
|
|
assert total == 100
|
|
|
|
|
|
def test_skips_dspark_drafter_for_main_weights(tmp_path):
|
|
# Same contract for a DSpark drafter, whose filename carries a Q8_0 token.
|
|
root = tmp_path / "repo"
|
|
main = _write(root / "model-Q8_0.gguf", 100)
|
|
_write(root / "dspark" / "dspark-model-Q8_0.gguf", 50)
|
|
|
|
path, total = models_route._resolve_quant_gguf(str(root), "Q8_0", is_local = True)
|
|
|
|
assert path == str(main)
|
|
assert total == 100
|
|
|
|
|
|
def test_prefers_the_complete_snapshot(tmp_path, monkeypatch):
|
|
cache = tmp_path / "hub"
|
|
snaps = cache / "models--org--repo" / "snapshots"
|
|
# Partial older snapshot: one small shard.
|
|
_write(snaps / "aaaa" / "model-Q4_K_M.gguf", 10)
|
|
# Complete newer snapshot: two larger shards.
|
|
complete_first = _write(snaps / "bbbb" / "model-00001-of-00002-Q4_K_M.gguf", 30)
|
|
_write(snaps / "bbbb" / "model-00002-of-00002-Q4_K_M.gguf", 40)
|
|
|
|
monkeypatch.setattr(
|
|
"utils.hf_cache_settings.known_hf_hub_caches",
|
|
lambda: [cache],
|
|
)
|
|
|
|
path, total = models_route._resolve_quant_gguf("org/repo", "Q4_K_M", is_local = False)
|
|
|
|
# The most complete snapshot (70 bytes) wins over the partial one (10).
|
|
assert total == 70
|
|
# Shard 1 (metadata) of the complete snapshot is returned.
|
|
assert path == str(complete_first)
|
|
|
|
|
|
def test_returns_none_when_quant_absent(tmp_path):
|
|
root = tmp_path / "repo"
|
|
_write(root / "model-Q4_K_M.gguf", 100)
|
|
|
|
path, total = models_route._resolve_quant_gguf(str(root), "Q8_0", is_local = True)
|
|
|
|
assert path is None
|
|
assert total == 0
|