mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-13 19:04:10 +00:00
Studio: stop a multi-checkpoint GGUF repo collapsing into one row per quant (#8222)
* Studio: key GGUF picker rows on the file a load reads, not the quant token A repo holding several checkpoints at the same quant labels collapsed into one row per quant. unsloth/LTX-2.3-GGUF ships 63 GGUFs across three checkpoints (repo root, distilled/, distilled-1.1/); the picker showed 22 rows, every one of them resolving into distilled-1.1/, so the dev checkpoint could not be selected at all, and each row advertised the sum of all three copies. Rows are now keyed on gguf_variant_key(), which stays the bare quant token when that token names the file within its path (the shape almost every repo uses, so the persisted key is unchanged there) and falls back to the file's shard family otherwise. A row also describes one shard family rather than every file sharing a key, so a repo that ships the same quant twice no longer advertises double. * Studio: resolve a chosen GGUF row to its own checkpoint The loader, llama.cpp file selection and the picker's chat-template lookup all resolved a variant by quant label and took the first match by name, so in a repo holding several checkpoints at one quant a request for the repo-root Q6_K row opened distilled/'s copy instead. Each now prefers the files whose own variant key is the requested one, via a mirror of gguf_variant_key in model_config (utils cannot import hub). * Studio: carry the GGUF display label to the picker, and cover multi-checkpoint repos The gguf-variants route rebuilt each row and dropped display_label, so a path-qualified variant would have read as its whole relative path. Variant deletion matched on the basename, which for a repo holding several checkpoints at one quant deleted all of them for a single row. * Studio: resolve a cached GGUF path by variant key as well as quant label _resolve_quant_gguf and _resolve_cached_model_path compared only the quant label, so a path-qualified variant found nothing in the cache and a load or an export fell back to a repo-wide walk. * Studio: only a non-quant directory qualifies a GGUF variant key A quant-named directory says nothing the basename does not already say, and the basename still decides which quant a file is (Q8_0/model-Q4_K_M.gguf IS the Q4_K_M file), so only a directory naming something else marks a separate checkpoint. Also stop qualifying on a quant token that is not at the end of the name: <model>-<QUANT>-MTP is an established spelling. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Declare display_label on the response model the route actually builds The qualified label was assembled and passed, then dropped. routes/models.py builds models.models.GgufVariantDetail, but only its hub.schemas.inventory twin declared display_label, so pydantic discarded the kwarg without a word and every qualified row reached the picker labelled with its whole relative path instead of "Q6_K . distilled". Found by looking at the picker in a browser; no unit test could see it, because both halves were individually correct. * Keep a GGUF row inside its own checkpoint Three ways a row still reached outside itself: - The plan narrowed only main_files to one shard family. target_filenames, required_hashes and download_size_bytes still held the discarded copy, so the worker fetched both, reclaim deleted the unchosen one as not-ours, and the finished job reported partial and fetched it again. - looks_like_quant full-matches a quant token, which no slash-qualified key can satisfy, so an absent qualified checkpoint fell through to the first local variant and answered under the requested model id. - The bare quant label names every checkpoint carrying that quant, so the repo-root row matched the qualified files too and took whichever sorted first. Exact keys now rank above the legacy label spelling, which stays as the fallback for rows that have no qualified one. Also: two checkpoints in ONE directory shared a display label, which is two rows a user cannot tell apart, so a colliding scope shows the file as well. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Advertise the qualified keys to the resolvers that read these listers The picker builds its rows from the hub copy, but three consumers take their variant identities from utils/models/model_config and openai_auto_download instead: the /v1 local index, the remote VRAM preflight, and the auto-download size map. All three still grouped on the bare quant label, so a qualified row was invisible to them, and since a slash-qualified suffix is now read as an explicit variant that miss is a 404 rather than a fallback onto another checkpoint. They key on the variant key now. The endian test still takes the quant token, which is what it reads, and a repo whose quant token already names its file lists exactly what it listed before, so no stored pin and no /v1 model id changes. * Confine the qualified key to the rows that need one Keying the listers on the variant key outright also renamed rows with no recognised quant token, whose label here has always been the last hyphenated segment rather than the whole stem (Qwen3.6-27B-MTP-001-of-002.gguf listed as MTP). That difference is old and has nothing to do with several checkpoints sharing a quant, and renaming those rows breaks the pins that hold them for no gain. The key is used only where a recognised token is qualified by a non-quant directory. * Keep a bare id meaning what it always meant Three ways the qualified keys leaked into places that should still answer to a bare name: - The automatic remote pick converted the winning filename back to a bare quant label, handing the load a name several checkpoints answer to. It keeps the identity the lister advertised. - A repo that files every variant under one shared container qualifies every key, since the key is a pure function of the path and cannot know the directory disambiguates nothing. Stored pins then missed the plan map and the worker exited with 'No GGUF shards matching variant'. plan_for_variant falls back to the bare quant when exactly one plan carries it, resolved at lookup so the key stays pure and the rows stay one per checkpoint. A repo that really does hold several checkpoints at one quant gets no fallback, because there the bare name names none of them. - A bare local id ranked across the qualified rows too, so /v1 could answer with a distilled checkpoint while a plain load took the root. The root rows are ranked alone when there are any. * Keep the bpw precisions apart, and let a parent-only quant through the endian filter The label this module extracts carries the bpw modifier that keeps byteshape's IQ4_XS at 3.53, 3.97 and 4.19 separately selectable; the token extractor drops it. Routing the listers through the key merged all three, so the qualified key is used only when it is actually path qualified and the label stands everywhere else. And the plan builder handed the qualified key to _is_big_endian_gguf_path, which reads a quant token so it can tell a parent-only quant from a big-endian build. Misread that way it dropped distilled/Q4_K_M/foo.gguf from every plan, leaving a row that could be shown but never downloaded. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the bare-quant fallback reach admission, and the endian test the last predicate Admission rejects against its own size map before the worker plan is consulted, so a repo that files every quant under one shared container still answered a legacy org/repo:Q4_K_M with a 404 and the plan-side fallback was never reached. Same unambiguous rule on both. is_main_gguf_variant_path runs the endian test as well, and handed the qualified key it cannot see a parent-only quant, so the plan came back with no main files and an interrupted download had no hashes to resume against. * Keep a qualified key's punctuation, and the endian test on the quant token _normalized_quant_label folds out hyphens and underscores, which is right for a bare quant and wrong for a path: it made exp-a/model-Q6_K and expa/model-Q6_K the same ask, so each advertised checkpoint answered to the other's key. The exact-key test compares a qualified key verbatim and case-insensitively, the callers hand the request over unfolded, and the legacy folding stays where it belongs, on the bare aliases it was written for. Two more endian call sites in the variant service were reading the qualified key as a quant token, which dropped a parent-only quant from the local blob scan and left update detection with nothing to compare. * Carry the requested qualified variant into the local config _find_local_gguf_by_variant picks the right file, but the returned ModelConfig dropped the variant, so the load intent carried none and llama.cpp recorded the bare label off the filename. /status then named the root row for a qualified checkpoint, and the deletion guard compared that bare label against the selected key and let the delete through. * Accept the unambiguous bare quant when deleting a container variant, and show the display label in the picker A repo filing its sole Q4_K_M under a shared container (weights/model-Q4_K_M.gguf) qualifies that key, because the key is a pure function of the path and cannot know the directory disambiguates nothing. plan_for_variant already admits the bare quant for the download when exactly one plan carries it; the deletion matcher did not, so deleting the same explicit or previously stored bare variant answered 404 and left the cached weights on disk. Resolve the same alias, under the same unambiguity rule, and keep an ambiguous bare name deleting nothing. The variants route returns display_label but the primary model picker typed variants without it and rendered v.quant, so a multi-checkpoint row read as its internal key (distilled/ltx-2.3-22b-distilled-Q6_K) instead of "Q6_K . distilled". Render the label there; quant stays the selection identity everywhere else. * Drop the extract_quant_label imports the variant-key switch left unused Source lint's import-hoist safety net blocks on them: both call sites now read gguf_variant_key, so the hoisted import is dead. * Run the endian filter on the quant token, and show display labels in the remaining selectors _gguf_files_for_variant prefiltered with the requested variant, which is now a path-qualified key. _is_big_endian_gguf_path reads a quant TOKEN -- it decides whether the quant came from the parent directory only -- so handed distilled/Q4_K_M/foo-be it finds that string in neither the basename nor the parent, calls the file big-endian and drops it before the owned-key comparison: the row is advertised and downloadable but never resolves for loading. Pass the file's own label, the same way gguf_plan and the auto-download map already do. The recipe-studio local model selector and the agents settings tab still rendered variant.quant, so a multi-checkpoint row read as its internal path there. Render the display label; the value submitted stays the key. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the bpw modifier in the variant key, and keep a bare auto-download on the root Two builds of one base quant at different bits-per-weight are two checkpoints (byteshape ships IQ4_XS at 3.53, 3.97 and 4.19). The loader's label always kept the modifier and the local export lister advertises it, but gguf_variant_key dropped it, so this PR's delete predicate -- which replaced the bpw-preserving _extract_quant_label -- answered 404 for the advertised name and unlinked EVERY build for the collapsed one. Carry the suffix in the key and its mirror, which also lines the plan and the auto-download map up with the lister; a key that is only the token plus its bpw suffix reads as a label already, so the display pass leaves it alone. preferred_quant is order-sensitive, so once the auto-download map carried a key per checkpoint a bare org/repo could pick distilled/model-Q6_K over the root model-Q6_K -- while the same id resolves to the root locally, i.e. one model id serving two different sets of weights. Apply the root-first filter local_model_resolver already applies. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the bpw suffix in the bare aliases, and size the inference lister per shard family The three compatibility fallbacks that accept a bare name for a path-qualified key -- the plan lookup, auto-download admission and deletion -- compared on extract_quant_label, which drops the bpw modifier the key now carries, so a persisted IQ4_XS-3.53bpw resolved nothing. One shared bare_quant_alias for all three; it also stops re-stemming a key that is already extension- stripped, which cut at the dot in '3.53bpw' and in 'ltx-2.3'. The inference-side listers still summed every shard family into one row while the hub lister and the plan keep only the family holding the lexicographically first file. routes/inference.py bills that size to the VRAM guard, so a repo shipping one quant twice (QwQ-32B's BF16 under two shard names) was charged double and could be refused a load that fits. Mirror group_gguf_variant_files in both listers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Show the display label in the closed agents quantization trigger too The list rows read as "Q6_K . distilled" but the trigger still printed the raw selectedVariant, so the label vanished the moment the popover closed -- which is where the choice is actually read. The variants are already in scope there; the value bound to the Select is unchanged. * Drop the last dead extract_quant_label import, and point the picker contract at the key Source lint's import-hoist safety net blocks on the import deletion.py no longer uses (the bare alias goes through bare_quant_alias now). tests/studio pinned the literal extract_quant_label(gguf_files[0].name) in common.py, which the variant-key switch re-pointed. For a standalone .gguf there is no directory to qualify, so the key IS the quant token -- and it keeps the bpw modifier, which is what makes it agree with the loader's own _extract_quant_label, the equality the sibling contract test depends on. The hub label drops that modifier, so the key is the more correct target here, not merely the new one. * Read the bpw modifier off the quant directory too, and root-first the remote default The quant-directory layout carries the modifier upstairs (IQ4_XS-3.53bpw/model.gguf), which is exactly where extract_quant_token looks next, so reading only the basename gave both builds the bare IQ4_XS key: grouping and the plan kept one family, and deleting that advertised key unlinked both. The suffix now comes from whichever segment named the quant, and the walk stops there so a modifier further up the tree is not claimed. pick_best_gguf keeps whichever filename it met first among equals, so a repo with model-Q6_K.gguf beside distilled/model-Q6_K.gguf could hand the picker the distilled checkpoint as its automatic default -- while a bare repo id means the ROOT checkpoint to _match_variant(None, ...) and to local_model_resolver. Prefer root rows when there are any, as those two already do. * Apply the root-first default on every branch, and teach the load guard the delete's alias /gguf-variants answers from three branches (remote, cached, partial-local) and only the remote one preferred root rows, so the automatic default depended on which branch served the request -- and in the offline and prefer_local_cache flows a bare repo id could select the distilled checkpoint while _match_variant(None, ...) and local_model_resolver both mean the root. One helper, all three. Deletion accepts an unambiguous bare quant for a path-qualified key, but the load-state guard still compared the two spellings literally, so a model loaded through a legacy bare pin could be deleted through its advertised qualified row without the 'Unload the model before deleting' refusal -- unlinking the resident model's snapshot and blob. The guard is alias-aware now, and deliberately loose: a false match only refuses a delete, a false miss loses weights. * Root-first the load path's auto-select, and size the kv-cache estimate on one shard family ModelConfig.from_identifier is the LOAD path and was the fourth resolver of a bare repo id: _pick_best_gguf keeps whichever filename it met first among equals, so an LTX-style listing that puts distilled/...-Q6_K before the root ...-Q6_K made a bare id load the distilled checkpoint, while local_model_resolver, the auto-download map and /gguf-variants all mean the root. _resolve_quant_gguf summed every file that matched the key, so a snapshot holding the same quant twice (QwQ-32B's two BF16 shard sets) reported double the weights the loader opens -- /kv-cache-estimate turns that into a false exceeds-memory warning, and it can make a snapshot look more complete purely for holding a redundant copy. Narrow to the one shard family the loader reads, the same rule group_gguf_variant_files applies; a genuinely split GGUF is one family and survives whole. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Daniel Han
GitHub
parent
d38ef17c20
commit
995921268a
20 changed files with 1780 additions and 113 deletions
|
|
@ -1723,6 +1723,18 @@ def _gguf_extra_shards(files: Iterable[str], first_shard: str) -> list[str]:
|
|||
return sorted(f for f in files if f != first_shard and sibling_pat.match(f))
|
||||
|
||||
|
||||
def _quant_label_for_endian(path: str) -> Optional[str]:
|
||||
"""The file's own quant label, for ``_is_big_endian_gguf_path``, or None when unreadable.
|
||||
|
||||
Best-effort: with no label the caller keeps its previous argument rather than dropping the
|
||||
endian test, which would admit a genuinely big-endian build."""
|
||||
try:
|
||||
from hub.utils.gguf import extract_quant_token
|
||||
return extract_quant_token(path)
|
||||
except Exception: # noqa: BLE001 -- an unreadable label must not sink the resolution
|
||||
return None
|
||||
|
||||
|
||||
def _gguf_files_for_variant(files: Iterable[str], variant: str) -> list[str]:
|
||||
"""Return main GGUF files matching a requested variant.
|
||||
|
||||
|
|
@ -1735,15 +1747,33 @@ def _gguf_files_for_variant(files: Iterable[str], variant: str) -> list[str]:
|
|||
for f in files
|
||||
if f.lower().endswith(".gguf")
|
||||
and not _is_companion_gguf_path(f)
|
||||
and not _is_big_endian_gguf_path(f, variant_key)
|
||||
# The endian predicate reads a quant TOKEN -- it decides whether the quant came from the
|
||||
# parent directory only -- so it gets the file's own label, never the requested key. Handed
|
||||
# a path-qualified key it cannot find that string in either the basename or the parent and
|
||||
# reads distilled/Q4_K_M/foo-be.gguf as big-endian, dropping the one file the key owns:
|
||||
# the row is advertised and downloadable but never resolves for loading. Same reason
|
||||
# gguf_plan and the auto-download map pass the label here.
|
||||
and not _is_big_endian_gguf_path(f, _quant_label_for_endian(f) or variant_key)
|
||||
]
|
||||
if not variant_key:
|
||||
return sorted(main_files)
|
||||
|
||||
try:
|
||||
from utils.models.model_config import _extract_quant_label
|
||||
from utils.models.model_config import _extract_quant_label, _gguf_variant_key
|
||||
except Exception:
|
||||
_extract_quant_label = None
|
||||
_gguf_variant_key = None
|
||||
|
||||
if _gguf_variant_key is not None:
|
||||
try:
|
||||
# The variant's own files first. In a repo holding several checkpoints at
|
||||
# one quant the label alone names all of them, and handing llama-server a
|
||||
# mixed set makes it read another checkpoint's weights as a shard.
|
||||
owned = sorted(f for f in main_files if _gguf_variant_key(f).lower() == variant_key)
|
||||
if owned:
|
||||
return owned
|
||||
except Exception as e:
|
||||
logger.warning("Failed to derive GGUF variant keys: %s", e)
|
||||
|
||||
if _extract_quant_label is not None:
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -127,7 +127,14 @@ def _local_gguf_entry(loader_id: str, info) -> Optional[_LocalGgufEntry]:
|
|||
# load would take: answering with the largest can evict a model and then OOM.
|
||||
from core.inference.openai_auto_download import preferred_quant
|
||||
|
||||
best = preferred_quant(quants)
|
||||
# Rank the ROOT checkpoints alone when there are any. A plain local load resolves
|
||||
# through non-recursive detect_gguf_model and so always takes the repo root, while
|
||||
# preferred_quant ranks on the key text and would hand a bare id an equally-good
|
||||
# ``distilled/...`` row that sorts earlier -- the same id serving different weights
|
||||
# depending on which resolver answered it. The qualified rows stay advertised; they
|
||||
# simply are not what a bare id means.
|
||||
unqualified = tuple(q for q in quants if "/" not in q)
|
||||
best = preferred_quant(unqualified or quants)
|
||||
if best and quants[0] != best:
|
||||
quants = (best, *(q for q in quants if q != best))
|
||||
return _LocalGgufEntry(loader_id, str(load_dir), quants)
|
||||
|
|
|
|||
|
|
@ -148,6 +148,13 @@ def looks_like_quant(variant: Optional[str]) -> bool:
|
|||
return False
|
||||
# _extract_quant_label can append a bpw modifier (IQ4_XS-3.53bpw); still a quant.
|
||||
label = re.sub(r"-[0-9]+(?:\.[0-9]+)?bpw$", "", variant.strip(), flags = re.IGNORECASE)
|
||||
# A path-qualified variant key (``distilled/model-Q6_K``) is one of OUR advertised rows: a
|
||||
# repo with several checkpoints at one quant keys each on its path. No foreign tag has that
|
||||
# shape -- Ollama's is ``:latest``, LiteLLM's namespace sits before the colon -- so it must
|
||||
# be read as an explicit checkpoint request and MISS when absent. Falling through instead
|
||||
# served the caller a different checkpoint under the model id they asked for.
|
||||
if "/" in label.replace("\\", "/"):
|
||||
return True
|
||||
return _GGUF_KNOWN_QUANT_RE.fullmatch(label) is not None
|
||||
|
||||
|
||||
|
|
@ -235,6 +242,7 @@ def _gguf_variants(siblings) -> dict[str, int]:
|
|||
quant, so the disk reserve is measured against what the worker fetches.
|
||||
"""
|
||||
from hub.utils.gguf import extract_quant_label as canonical_quant_label
|
||||
from hub.utils.gguf import gguf_variant_key
|
||||
from hub.utils.gguf_plan import build_gguf_variant_plans
|
||||
from utils.models.model_config import (
|
||||
_extract_quant_label,
|
||||
|
|
@ -250,13 +258,18 @@ def _gguf_variants(siblings) -> dict[str, int]:
|
|||
name = getattr(sibling, "rfilename", "") or ""
|
||||
if not name.lower().endswith(".gguf"):
|
||||
continue
|
||||
quant = _extract_quant_label(name)
|
||||
label = _extract_quant_label(name)
|
||||
# The identity the PLAN is keyed on. A repo holding several checkpoints at one quant
|
||||
# advertises a qualified key per checkpoint, and keying this map on the bare label left
|
||||
# every one of those rows a hard miss here: a 404 instead of the download.
|
||||
quant = gguf_variant_key(name)
|
||||
if not looks_like_quant(quant):
|
||||
# With no recognized quant token the extractors part ways: this one takes
|
||||
# the last hyphenated segment ("7b" of llama-7b) while the plan and worker
|
||||
# key the whole stem, so advertising ours dispatches an unresolvable variant.
|
||||
quant = canonical_quant_label(name) or quant
|
||||
if _is_mmproj(name) or _is_mtp_drafter(name) or _is_big_endian_gguf_path(name, quant):
|
||||
# The endian test reads a quant TOKEN, so it gets the label, not the path-qualified key.
|
||||
if _is_mmproj(name) or _is_mtp_drafter(name) or _is_big_endian_gguf_path(name, label):
|
||||
continue
|
||||
plan = plans.get(quant.lower())
|
||||
if plan is not None:
|
||||
|
|
@ -676,10 +689,10 @@ async def _admit_and_start(
|
|||
)
|
||||
|
||||
expected_bytes = variants[variant]
|
||||
from hub.utils.gguf_plan import build_gguf_variant_plans
|
||||
from hub.utils.gguf_plan import build_gguf_variant_plans, plan_for_variant
|
||||
|
||||
plan = build_gguf_variant_plans(list(getattr(info, "siblings", None) or [])).get(
|
||||
variant.lower()
|
||||
plan = plan_for_variant(
|
||||
build_gguf_variant_plans(list(getattr(info, "siblings", None) or [])), variant
|
||||
)
|
||||
if require_vision and not (plan and plan.mmproj_filenames):
|
||||
_release(active)
|
||||
|
|
@ -735,6 +748,26 @@ def preferred_quant(labels) -> Optional[str]:
|
|||
return synthetic.get(best) if best else None
|
||||
|
||||
|
||||
def _bare_quant_alias(wanted: str, lowered: dict[str, str]) -> Optional[str]:
|
||||
"""The one qualified variant whose quant token is *wanted*, or None when it names 0 or 2+.
|
||||
|
||||
A key is a pure function of the path, so a repo that files every quant under one shared
|
||||
container qualifies all of them even though the directory disambiguates nothing, and the bare
|
||||
spelling every stored id uses then matches no key at all.
|
||||
"""
|
||||
from hub.utils.gguf import bare_quant_alias
|
||||
|
||||
target = (wanted or "").strip().lower()
|
||||
if not target:
|
||||
return None
|
||||
matches = [
|
||||
name
|
||||
for key, name in lowered.items()
|
||||
if "/" in key and bare_quant_alias(key).lower() == target
|
||||
]
|
||||
return matches[0] if len(matches) == 1 else None
|
||||
|
||||
|
||||
def _match_variant(wanted: Optional[str], variants: dict[str, int]) -> Optional[str]:
|
||||
"""Resolve the requested quant against what the repo actually has.
|
||||
|
||||
|
|
@ -749,10 +782,23 @@ def _match_variant(wanted: Optional[str], variants: dict[str, int]) -> Optional[
|
|||
# and defaulting past one would fetch a model nobody asked for.
|
||||
lowered = {name.lower(): name for name in variants}
|
||||
exact = lowered.get(wanted.strip().lower())
|
||||
if exact is None:
|
||||
# Same bare-quant fallback the plan lookup makes, and it has to be made HERE too:
|
||||
# admission rejects against this map first, so a repo that files every quant under
|
||||
# one shared container answered a legacy org/repo:Q4_K_M with a 404 and the worker's
|
||||
# fallback was never reached. Unambiguous only, for the same reason.
|
||||
exact = _bare_quant_alias(wanted, lowered)
|
||||
if exact is not None or looks_like_quant(wanted):
|
||||
# A quant-shaped suffix that matches nothing is a miss, never a swap.
|
||||
return exact
|
||||
return preferred_quant(variants)
|
||||
# A BARE org/repo means the ROOT checkpoint, so a qualified sibling must not be ranked
|
||||
# against it: preferred_quant is order-sensitive, and once the map carried a key per
|
||||
# checkpoint a repo with distilled/model-Q6_K beside model-Q6_K could serve the sibling for
|
||||
# a bare id -- the same id that resolves to the root locally. Same filter
|
||||
# local_model_resolver._local_gguf_entry applies, so both resolvers answer one id one way.
|
||||
# A repo with nothing at the root falls back to the whole set rather than refusing.
|
||||
unqualified = {name: size for name, size in variants.items() if "/" not in name}
|
||||
return preferred_quant(unqualified or variants)
|
||||
|
||||
|
||||
async def _dispatch(
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ from hub.schemas.inventory import (
|
|||
ModelRuntime,
|
||||
)
|
||||
from hub.utils.gguf import (
|
||||
extract_quant_label,
|
||||
gguf_variant_key,
|
||||
is_gguf_filename as _is_gguf_filename,
|
||||
is_mmproj_filename as _is_mmproj_filename,
|
||||
is_mtp_drafter_path as _is_mtp_drafter_path,
|
||||
|
|
@ -769,7 +769,7 @@ def _classify_local_path(
|
|||
if gguf_files:
|
||||
gguf_size_bytes = _sum_file_sizes(gguf_files)
|
||||
variant = (
|
||||
extract_quant_label(gguf_files[0].name)
|
||||
gguf_variant_key(gguf_files[0].name)
|
||||
if scan_path.is_file() and len(gguf_files) == 1
|
||||
else None
|
||||
)
|
||||
|
|
|
|||
|
|
@ -17,8 +17,9 @@ from hub.utils import download_manifest
|
|||
from hub.utils import download_registry
|
||||
from hub.utils import inventory_scan as hf_cache_scan
|
||||
from hub.utils.gguf import (
|
||||
extract_quant_label,
|
||||
bare_quant_alias,
|
||||
extract_quant_token,
|
||||
gguf_variant_key,
|
||||
is_reclaimable_drafter_path as _is_reclaimable_drafter_path,
|
||||
)
|
||||
from hub.utils.hf_cache_state import (
|
||||
|
|
@ -131,7 +132,11 @@ def _remove_empty_variant_dirs(target_repos: list, variant: str) -> tuple[int, l
|
|||
"""Remove now-empty ``snapshots/<rev>/<quant>/`` folders for *variant* (the
|
||||
quant label names the folder); only empty dirs go, so siblings are safe.
|
||||
Returns (count removed, removal failures other than a concurrent refill)."""
|
||||
variant_key = (extract_quant_token(variant) or variant).lower()
|
||||
# A path-qualified variant key names its own folder; its quant token belongs to
|
||||
# sibling checkpoints too, so it must not reach for a <quant>/ dir it does not own.
|
||||
variant_key = (
|
||||
variant.lower() if "/" in variant else (extract_quant_token(variant) or variant).lower()
|
||||
)
|
||||
removed = 0
|
||||
failures: list[str] = []
|
||||
for target_repo in target_repos:
|
||||
|
|
@ -197,6 +202,33 @@ def _remove_empty_snapshot_dirs(target_repos: list) -> tuple[int, list[str]]:
|
|||
return removed, failures
|
||||
|
||||
|
||||
def _variant_keys_to_delete(target_repo, variant: str) -> set[str]:
|
||||
"""The variant keys in *target_repo* that *variant* names, lowercased.
|
||||
|
||||
Its own key, always. Plus the unambiguous bare-quant alias the download side already admits
|
||||
(``gguf_plan.plan_for_variant``): a repo filing its sole Q4_K_M under a shared container
|
||||
(``weights/model-Q4_K_M.gguf``) qualifies that key, because the key is a pure function of the
|
||||
path and cannot know the directory disambiguates nothing, so every stored pin and every
|
||||
explicit ``repo:Q4_K_M`` names it by quant alone. Admitting the alias for the download and not
|
||||
for the delete answered "not found" and left the weights on disk.
|
||||
|
||||
Only when it is unambiguous, exactly as the download side decides it: a repo that really does
|
||||
hold several checkpoints at one quant gets no fallback, because there the bare name genuinely
|
||||
does not name one of them and deleting the wrong one is unrecoverable.
|
||||
"""
|
||||
wanted = (variant or "").strip().lower()
|
||||
if not wanted or "/" in wanted:
|
||||
return {wanted}
|
||||
keys = {
|
||||
gguf_variant_key(name).lower()
|
||||
for _snap, _blob, name in _repo_file_matches(target_repo, _is_main_gguf_filename)
|
||||
}
|
||||
if wanted in keys:
|
||||
return {wanted}
|
||||
aliased = {key for key in keys if "/" in key and bare_quant_alias(key).lower() == wanted}
|
||||
return aliased if len(aliased) == 1 else {wanted}
|
||||
|
||||
|
||||
def _delete_gguf_variant_from_repos(
|
||||
repo_id: str,
|
||||
variant: str,
|
||||
|
|
@ -214,10 +246,11 @@ def _delete_gguf_variant_from_repos(
|
|||
|
||||
for target_repo in target_repos:
|
||||
repo_dir = Path(target_repo.repo_path) if getattr(target_repo, "repo_path", None) else None
|
||||
wanted_keys = _variant_keys_to_delete(target_repo, variant)
|
||||
matched = _repo_file_matches(
|
||||
target_repo,
|
||||
lambda name: _is_main_gguf_filename(name)
|
||||
and extract_quant_label(name).lower() == variant.lower(),
|
||||
lambda name, keys = wanted_keys: _is_main_gguf_filename(name)
|
||||
and gguf_variant_key(name).lower() in keys,
|
||||
)
|
||||
|
||||
for snap, _blob, name in matched:
|
||||
|
|
@ -440,7 +473,7 @@ def reclaim_replaced_gguf_variant(
|
|||
matches = _repo_file_matches(
|
||||
target_repo,
|
||||
lambda name: _is_main_gguf_filename(name)
|
||||
and extract_quant_label(name).lower() == variant_key,
|
||||
and gguf_variant_key(name).lower() == variant_key,
|
||||
)
|
||||
for snap, blob, name in matches:
|
||||
# Prune only a file we can identify as a real, stale cache blob. A
|
||||
|
|
|
|||
|
|
@ -30,6 +30,7 @@ from hub.utils.hf_cache_state import (
|
|||
from hub.utils.gguf import (
|
||||
GgufVariantInfo,
|
||||
extract_quant_label,
|
||||
gguf_variant_key,
|
||||
iter_hf_cache_snapshots,
|
||||
is_big_endian_gguf_path,
|
||||
list_empty_gguf_variant_dirs,
|
||||
|
|
@ -392,8 +393,11 @@ def _local_main_gguf_blobs_by_quant(
|
|||
str(blob) for blob in hashes if blob
|
||||
)
|
||||
continue
|
||||
quant = extract_quant_label(normalized).lower()
|
||||
if is_big_endian_gguf_path(normalized, quant):
|
||||
quant = gguf_variant_key(normalized).lower()
|
||||
# The endian predicate reads a quant TOKEN so it can tell a parent-only quant
|
||||
# from a big-endian build; the qualified key makes it misread the path and drop
|
||||
# the blob, which leaves update detection with no local main files to compare.
|
||||
if is_big_endian_gguf_path(normalized, extract_quant_label(normalized)):
|
||||
continue
|
||||
bucket = result.setdefault(quant, {}).setdefault(normalized, set())
|
||||
bucket.update(str(blob) for blob in hashes if blob)
|
||||
|
|
@ -870,6 +874,20 @@ class VariantsAnswer(NamedTuple):
|
|||
context_source: Optional[str]
|
||||
|
||||
|
||||
def _default_variant_candidates(variants) -> list[str]:
|
||||
"""The filenames the automatic default may be picked from: ROOT rows when there are any.
|
||||
|
||||
``pick_best_gguf`` keeps whichever filename it met first among equals, so a repo with
|
||||
``model-Q6_K.gguf`` beside ``distilled/model-Q6_K.gguf`` could make the qualified sibling the
|
||||
default -- and then a bare repo id would mean one checkpoint here and another to
|
||||
``_match_variant(None, ...)`` and ``local_model_resolver``, which both define it as the root.
|
||||
Every branch of this service (remote, cached, partial-local) has to apply it, or the answer
|
||||
depends on which one served the request. Nothing at the root falls back to the whole set.
|
||||
"""
|
||||
root_rows = [v.filename for v in variants if "/" not in v.quant]
|
||||
return root_rows or [v.filename for v in variants]
|
||||
|
||||
|
||||
async def get_gguf_variants_answer(
|
||||
repo_id: str,
|
||||
prefer_local_cache: bool = False,
|
||||
|
|
@ -917,8 +935,8 @@ async def get_gguf_variants_answer(
|
|||
|
||||
# The default comes from the ready rows; with none ready every row is the fallback.
|
||||
ready = [v for v in variants if _downloaded(v)]
|
||||
best = pick_best_gguf([v.filename for v in (ready or variants)])
|
||||
default_variant = extract_quant_label(best) if best else None
|
||||
best = pick_best_gguf(_default_variant_candidates(ready or variants))
|
||||
default_variant = gguf_variant_key(best) if best else None
|
||||
|
||||
return GgufVariantsResponse(
|
||||
repo_id = response_repo_id,
|
||||
|
|
@ -941,9 +959,8 @@ async def get_gguf_variants_answer(
|
|||
def _partial_local_response(
|
||||
response_repo_id: str, variants, has_vision: bool
|
||||
) -> GgufVariantsResponse:
|
||||
filenames = [v.filename for v in variants]
|
||||
best = pick_best_gguf(filenames)
|
||||
default_variant = extract_quant_label(best) if best else None
|
||||
best = pick_best_gguf(_default_variant_candidates(variants))
|
||||
default_variant = gguf_variant_key(best) if best else None
|
||||
return GgufVariantsResponse(
|
||||
repo_id = response_repo_id,
|
||||
variants = [
|
||||
|
|
@ -1156,9 +1173,8 @@ async def get_gguf_variants_answer(
|
|||
if fallback is not None:
|
||||
return fallback
|
||||
|
||||
filenames = [v.filename for v in variants]
|
||||
best = pick_best_gguf(filenames)
|
||||
default_variant = extract_quant_label(best) if best else None
|
||||
best = pick_best_gguf(_default_variant_candidates(variants))
|
||||
default_variant = gguf_variant_key(best) if best else None
|
||||
|
||||
# Per-snapshot accounting: a variant counts as present only when one
|
||||
# snapshot holds all its files (split GGUFs need every shard together),
|
||||
|
|
@ -1192,8 +1208,8 @@ async def get_gguf_variants_answer(
|
|||
by_filename[key] = max(by_filename.get(key, 0), size)
|
||||
if _is_mmproj_filename(f.name) or _is_mtp_drafter_path(rel):
|
||||
continue
|
||||
q = extract_quant_label(rel)
|
||||
if is_big_endian_gguf_path(rel, q):
|
||||
q = gguf_variant_key(rel)
|
||||
if is_big_endian_gguf_path(rel, extract_quant_label(rel)):
|
||||
continue
|
||||
q = q.lower()
|
||||
by_quant[q] = by_quant.get(q, 0) + size
|
||||
|
|
|
|||
|
|
@ -279,6 +279,35 @@ def extract_quant_token(filename: str) -> Optional[str]:
|
|||
return None
|
||||
|
||||
|
||||
# A bits-per-weight modifier trailing the quant token. Two builds of one base quant at different
|
||||
# bpw are two checkpoints (byteshape ships IQ4_XS at 3.53, 3.97 and 4.19), and the loader's own
|
||||
# label keeps the modifier for exactly that reason. The variant KEY has to keep it too: without it
|
||||
# the lister advertises IQ4_XS-3.53bpw while the plan, the download map and the delete predicate
|
||||
# all say IQ4_XS, so the advertised name 404s and the collapsed one unlinks every build.
|
||||
_GGUF_BPW_SUFFIX_RE = re.compile(r"-[0-9]+(?:\.[0-9]+)?bpw$", re.IGNORECASE)
|
||||
|
||||
|
||||
def _gguf_bpw_suffix(filename: str) -> str:
|
||||
"""``-3.53bpw`` from whichever path segment names the quant, else ``""``.
|
||||
|
||||
Not the basename alone: the quant-directory layout carries it upstairs
|
||||
(``IQ4_XS-3.53bpw/model.gguf``), which is exactly where ``extract_quant_token`` looks next, so
|
||||
reading only ``model`` gave both bpw builds the bare ``IQ4_XS`` key. The walk stops at the
|
||||
segment that named the quant -- a modifier further up the tree belongs to something else.
|
||||
"""
|
||||
path = filename.replace("\\", "/")
|
||||
parents = path.rpartition("/")[0]
|
||||
for segment in (_gguf_stem(path), *reversed(parents.split("/"))):
|
||||
if not segment:
|
||||
continue
|
||||
match = _GGUF_BPW_SUFFIX_RE.search(segment)
|
||||
if match:
|
||||
return match.group(0)
|
||||
if _select_quant_match(segment) is not None:
|
||||
return ""
|
||||
return ""
|
||||
|
||||
|
||||
def _unknown_gguf_variant_key(filename: str) -> str:
|
||||
stem = _gguf_stem(filename)
|
||||
if "/" not in filename:
|
||||
|
|
@ -287,19 +316,144 @@ def _unknown_gguf_variant_key(filename: str) -> str:
|
|||
return f"{parents}/{stem}" if parents and stem else stem or "gguf"
|
||||
|
||||
|
||||
def gguf_variant_family(filename: str) -> str:
|
||||
"""The shard family *filename* belongs to: its directory plus its shard-stripped name.
|
||||
|
||||
The unit a row and a download plan describe. Every shard of one split GGUF shares
|
||||
a family, which is why summing sizes within a family is right; two files that do
|
||||
NOT share one are two different checkpoints, so summing across them is not.
|
||||
"""
|
||||
return _unknown_gguf_variant_key(filename)
|
||||
|
||||
|
||||
def extract_quant_label(filename: str) -> str:
|
||||
return extract_quant_token(filename) or _unknown_gguf_variant_key(filename)
|
||||
|
||||
|
||||
def bare_quant_alias(key: str) -> str:
|
||||
"""The bare quant spelling a path-qualified *key* also answers to.
|
||||
|
||||
``weights/model-IQ4_XS-3.53bpw`` -> ``IQ4_XS-3.53bpw``. The bpw modifier is part of the
|
||||
identity now, so the three compatibility fallbacks that accept a bare name for a qualified
|
||||
key (the plan lookup, auto-download admission, deletion) have to keep it -- comparing on the
|
||||
token alone means a persisted ``IQ4_XS-3.53bpw`` resolves nothing.
|
||||
"""
|
||||
# A key is already shard- and extension-stripped, so re-stemming it would cut at the dot in
|
||||
# "3.53bpw" (and at the one in "ltx-2.3"). Hand the extractors a name they expect instead.
|
||||
probe = f'{(key or "").replace(chr(92), "/").rsplit("/", 1)[-1]}.gguf'
|
||||
return f"{extract_quant_label(probe)}{_gguf_bpw_suffix(probe)}"
|
||||
|
||||
|
||||
def _is_quant_directory(segment: str) -> bool:
|
||||
"""Whether a path segment names a quant (``Q6_K/``, ``Llama-3.3-70B-Instruct-Q6_K/``).
|
||||
|
||||
Such a directory says only how the file was quantized, which its name already says,
|
||||
so it adds nothing to the file's identity. A directory naming something else
|
||||
(``distilled/``) is a different checkpoint and does. The basename still wins on
|
||||
which quant it is: ``Q8_0/model-Q4_K_M.gguf`` IS the Q4_K_M file.
|
||||
"""
|
||||
return _select_quant_match(segment) is not None
|
||||
|
||||
|
||||
def gguf_variant_key(filename: str) -> str:
|
||||
"""The persisted identity of a selectable GGUF variant.
|
||||
|
||||
The bare quant token when that token names the file within its path -- the shape
|
||||
almost every repo uses, so this is byte-identical to the historical key there and
|
||||
every stored pin, manifest and marker keeps resolving. When the token does NOT
|
||||
single the file out, because a sibling directory holds another checkpoint at the
|
||||
same quant (``distilled/`` and ``distilled-1.1/`` beside the repo root), the key
|
||||
is the file's :func:`gguf_variant_family` instead, which is unique within the
|
||||
repo and which the loader already accepts as a spelling
|
||||
(``model_config._find_local_gguf_by_variant`` matches its shard-stripped relative
|
||||
path, as does ``llama_cpp._gguf_files_for_variant``).
|
||||
|
||||
A pure function of the path, deliberately: the remote listing sees a whole repo
|
||||
while a cache scan sees whatever was downloaded, and a key that depended on the
|
||||
set would disagree between them and strand a finished download as incomplete.
|
||||
"""
|
||||
path = filename.replace("\\", "/")
|
||||
quant = extract_quant_token(path)
|
||||
if quant is None:
|
||||
return _unknown_gguf_variant_key(path)
|
||||
parents = path.rpartition("/")[0]
|
||||
if any(segment and not _is_quant_directory(segment) for segment in parents.split("/")):
|
||||
return _unknown_gguf_variant_key(path)
|
||||
return f"{quant}{_gguf_bpw_suffix(path)}"
|
||||
|
||||
|
||||
def _variant_scope_label(filename: str, *, with_stem: bool = False) -> str:
|
||||
"""The part of a qualified variant's path that tells it apart from its namesakes.
|
||||
|
||||
The directory alone reads best and is enough for the usual shape, where each checkpoint
|
||||
has its own. ``with_stem`` adds the filename back for the case where it is not: two
|
||||
checkpoints in ONE directory at one quant are two rows, and a label naming only the
|
||||
directory would print the same text on both.
|
||||
"""
|
||||
parents = filename.replace("\\", "/").rpartition("/")[0].strip("/")
|
||||
stem = _gguf_stem(filename)
|
||||
if not parents:
|
||||
return stem
|
||||
return f"{parents}/{stem}" if with_stem else parents
|
||||
|
||||
|
||||
def _apply_gguf_display_labels(variants: list[GgufVariantInfo]) -> None:
|
||||
unknown_variants = [
|
||||
variant for variant in variants if extract_quant_token(variant.filename) is None
|
||||
]
|
||||
if not unknown_variants:
|
||||
return
|
||||
ambiguous = len(unknown_variants) > 1
|
||||
for variant in unknown_variants:
|
||||
variant.display_label = f"GGUF · {variant.filename}" if ambiguous else "GGUF"
|
||||
|
||||
# The bpw modifier is part of the key but reads perfectly well on its own
|
||||
# ("IQ4_XS-3.53bpw"), so a key that is only the token plus its bpw suffix is NOT a
|
||||
# path-qualified one and needs no scope label.
|
||||
def _plain_key(variant) -> Optional[str]:
|
||||
token = extract_quant_token(variant.filename)
|
||||
return None if token is None else f"{token}{_gguf_bpw_suffix(variant.filename)}"
|
||||
|
||||
qualified = [
|
||||
variant
|
||||
for variant in variants
|
||||
if (plain := _plain_key(variant)) is not None and variant.quant.lower() != plain.lower()
|
||||
]
|
||||
# A scope shared by two rows does not tell them apart, so those rows show the file too.
|
||||
scopes: dict[str, int] = {}
|
||||
for variant in qualified:
|
||||
scope = _variant_scope_label(variant.filename).lower()
|
||||
scopes[scope] = scopes.get(scope, 0) + 1
|
||||
for variant in variants:
|
||||
token = extract_quant_token(variant.filename)
|
||||
if token is None:
|
||||
variant.display_label = f"GGUF · {variant.filename}" if ambiguous else "GGUF"
|
||||
elif variant.quant.lower() != (_plain_key(variant) or "").lower():
|
||||
# A key qualified by path: show the quant, plus what distinguishes it.
|
||||
collides = scopes.get(_variant_scope_label(variant.filename).lower(), 0) > 1
|
||||
variant.display_label = (
|
||||
f"{token} · {_variant_scope_label(variant.filename, with_stem = collides)}"
|
||||
)
|
||||
|
||||
|
||||
def group_gguf_variant_files(entries) -> dict[str, tuple[str, int]]:
|
||||
"""``variant key -> (first filename, size of that variant's shard family)``.
|
||||
|
||||
*entries* is an iterable of ``(path, size)`` for main GGUFs only, already filtered
|
||||
of mmproj, drafters and big-endian builds.
|
||||
|
||||
Sizes are summed across the shards of ONE family, never across families. A repo
|
||||
that ships the same quant twice (``BF16/QwQ-32B-BF16-*`` beside
|
||||
``BF16/QwQ-32B.BF16-*``) therefore advertises what a load would actually read
|
||||
rather than the total of both copies. The family kept is the one holding the
|
||||
lexicographically first file, which is the shard the lister and the loader open.
|
||||
"""
|
||||
families: dict[str, dict[str, list[tuple[str, int]]]] = {}
|
||||
for path, size in entries:
|
||||
families.setdefault(gguf_variant_key(path), {}).setdefault(
|
||||
gguf_variant_family(path), []
|
||||
).append((path, int(size or 0)))
|
||||
grouped: dict[str, tuple[str, int]] = {}
|
||||
for key, by_family in families.items():
|
||||
chosen = min(by_family.values(), key = lambda members: min(path for path, _ in members))
|
||||
grouped[key] = (min(path for path, _ in chosen), sum(size for _, size in chosen))
|
||||
return grouped
|
||||
|
||||
|
||||
def _env_offline() -> bool:
|
||||
|
|
@ -606,10 +760,8 @@ def list_gguf_variants(
|
|||
return _ready_cached_variants(cached)
|
||||
raise
|
||||
|
||||
variants: list[GgufVariantInfo] = []
|
||||
has_vision = False
|
||||
quant_totals: dict[str, int] = {}
|
||||
quant_first_file: dict[str, str] = {}
|
||||
main_files: list[tuple[str, int]] = []
|
||||
|
||||
for sibling in info.siblings:
|
||||
filename = getattr(sibling, "rfilename", None)
|
||||
|
|
@ -622,24 +774,18 @@ def list_gguf_variants(
|
|||
if is_mmproj_filename(filename):
|
||||
has_vision = True
|
||||
continue
|
||||
quant = extract_quant_label(filename)
|
||||
# The two extractors disagree on F16-be-checkpoint-Q4_K_M shapes; judge with the
|
||||
# loader's label so no row is advertised for a file the remote detector refuses.
|
||||
from utils.models.model_config import _extract_quant_label as _loader_quant
|
||||
|
||||
if is_big_endian_gguf_path(filename, _loader_quant(filename)):
|
||||
continue
|
||||
quant_totals[quant] = quant_totals.get(quant, 0) + int(getattr(sibling, "size", 0) or 0)
|
||||
quant_first_file.setdefault(quant, filename)
|
||||
main_files.append((filename, int(getattr(sibling, "size", 0) or 0)))
|
||||
|
||||
for quant, total_size in quant_totals.items():
|
||||
variants.append(
|
||||
GgufVariantInfo(
|
||||
filename = quant_first_file[quant],
|
||||
quant = quant,
|
||||
size_bytes = total_size,
|
||||
)
|
||||
)
|
||||
variants = [
|
||||
GgufVariantInfo(filename = filename, quant = quant, size_bytes = size)
|
||||
for quant, (filename, size) in group_gguf_variant_files(main_files).items()
|
||||
]
|
||||
|
||||
variants.sort(key = lambda variant: -variant.size_bytes)
|
||||
_apply_gguf_display_labels(variants)
|
||||
|
|
@ -677,8 +823,7 @@ def list_local_gguf_variants(
|
|||
else _registered_custom_model_root(directory)
|
||||
)
|
||||
|
||||
quant_totals: dict[str, int] = {}
|
||||
quant_first_file: dict[str, str] = {}
|
||||
main_files: list[tuple[str, int]] = []
|
||||
has_vision = False
|
||||
# Match the cache dir of ANY H3 bundle repo, not just one of them: the same aggregation runs
|
||||
# over whichever mirror the user actually downloaded.
|
||||
|
|
@ -704,23 +849,17 @@ def list_local_gguf_variants(
|
|||
rel = file.relative_to(root).as_posix()
|
||||
if _is_local_mtp_drafter(file, custom_root, rel):
|
||||
continue
|
||||
quant = extract_quant_label(rel)
|
||||
# The two extractors disagree on F16-be-checkpoint-Q4_K_M shapes; judge with the
|
||||
# loader's label so no row is listed for a file the local detector refuses.
|
||||
from utils.models.model_config import _extract_quant_label as _loader_quant
|
||||
|
||||
if is_big_endian_gguf_path(rel, _loader_quant(rel)):
|
||||
continue
|
||||
quant_totals[quant] = quant_totals.get(quant, 0) + size
|
||||
quant_first_file.setdefault(quant, rel)
|
||||
main_files.append((rel, size))
|
||||
|
||||
variants = [
|
||||
GgufVariantInfo(
|
||||
filename = quant_first_file[quant],
|
||||
quant = quant,
|
||||
size_bytes = size,
|
||||
)
|
||||
for quant, size in quant_totals.items()
|
||||
GgufVariantInfo(filename = filename, quant = quant, size_bytes = size)
|
||||
for quant, (filename, size) in group_gguf_variant_files(main_files).items()
|
||||
]
|
||||
variants.sort(key = lambda variant: -variant.size_bytes)
|
||||
_apply_gguf_display_labels(variants)
|
||||
|
|
|
|||
|
|
@ -8,7 +8,10 @@ from typing import Optional, Sequence
|
|||
|
||||
from hub.utils.download_manifest import ExpectedFile
|
||||
from hub.utils.gguf import (
|
||||
bare_quant_alias,
|
||||
extract_quant_label,
|
||||
gguf_variant_family,
|
||||
gguf_variant_key,
|
||||
is_big_endian_gguf_path,
|
||||
is_gguf_filename,
|
||||
is_mmproj_filename,
|
||||
|
|
@ -68,12 +71,21 @@ def is_companion_gguf_path(path: str) -> bool:
|
|||
|
||||
|
||||
def is_main_gguf_variant_path(path: str, variant: str) -> bool:
|
||||
"""Whether *path* is one of *variant*'s own weight files.
|
||||
|
||||
Keyed on :func:`gguf_variant_key`, in lockstep with the listers: a row built under
|
||||
one identity and matched under another produces a variant that can be shown but
|
||||
not downloaded.
|
||||
"""
|
||||
return (
|
||||
is_gguf_filename(path)
|
||||
and not is_mmproj_filename(path)
|
||||
and not is_mtp_drafter_path(path)
|
||||
and not is_big_endian_gguf_path(path, variant)
|
||||
and extract_quant_label(path).lower() == variant.lower()
|
||||
# The endian predicate reads a quant TOKEN, so it gets the label: handed the qualified key
|
||||
# it cannot see a parent-only quant and drops the file, leaving the plan with no main
|
||||
# files at all and an interrupted download with no hashes to resume against.
|
||||
and not is_big_endian_gguf_path(path, extract_quant_label(path))
|
||||
and gguf_variant_key(path).lower() == variant.lower()
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -145,8 +157,11 @@ def build_gguf_variant_plans(siblings: Sequence) -> dict[str, GgufVariantPlan]:
|
|||
# (the root mtp-*.gguf carries a quant label, e.g. Q8_0).
|
||||
if is_mmproj_filename(name) or is_mtp_drafter_path(name):
|
||||
continue
|
||||
quant = extract_quant_label(name).lower()
|
||||
if is_big_endian_gguf_path(name, quant):
|
||||
quant = gguf_variant_key(name).lower()
|
||||
# The endian predicate reads a quant TOKEN -- it decides whether the quant came from the
|
||||
# parent directory only -- so a qualified key would make it misread the path and drop the
|
||||
# file from every plan.
|
||||
if is_big_endian_gguf_path(name, extract_quant_label(name)):
|
||||
continue
|
||||
main.setdefault(quant, []).append(sibling)
|
||||
|
||||
|
|
@ -167,6 +182,52 @@ def build_gguf_variant_plans(siblings: Sequence) -> dict[str, GgufVariantPlan]:
|
|||
return plans
|
||||
|
||||
|
||||
def plan_for_variant(plans: dict[str, GgufVariantPlan], variant: str) -> Optional[GgufVariantPlan]:
|
||||
"""The plan for *variant*, accepting a bare quant when exactly one plan carries it.
|
||||
|
||||
A repo that files every variant under one shared container (``weights/model-Q4_K_M.gguf``)
|
||||
qualifies every key, because the key is a pure function of the path and cannot know that the
|
||||
directory disambiguates nothing. Every stored pin and every explicit ``repo:Q4_K_M`` then
|
||||
missed the plan map and the worker exited with "No GGUF shards matching variant".
|
||||
|
||||
Resolved at LOOKUP rather than by aliasing the map, so the key stays a pure function of the
|
||||
path -- the remote listing and a partial cache scan have to agree on it -- and the advertised
|
||||
rows stay one per checkpoint. Only when the bare name is UNAMBIGUOUS: a repo that really does
|
||||
hold several checkpoints at one quant gets no fallback, because there the bare name genuinely
|
||||
does not name one of them.
|
||||
"""
|
||||
wanted = (variant or "").strip().lower()
|
||||
if not wanted:
|
||||
return None
|
||||
exact = plans.get(wanted)
|
||||
if exact is not None:
|
||||
return exact
|
||||
matches = [key for key in plans if "/" in key and bare_quant_alias(key).lower() == wanted]
|
||||
return plans[matches[0]] if len(matches) == 1 else None
|
||||
|
||||
|
||||
def _one_shard_family(main_files: Sequence[ExpectedFile]) -> tuple[ExpectedFile, ...]:
|
||||
"""Narrow a variant's weight files to the single shard family a load would read.
|
||||
|
||||
A repo can ship one quant twice under names that share a variant key -- the same
|
||||
BF16 as ``QwQ-32B-BF16-*`` and ``QwQ-32B.BF16-*``, or one Q6_K under both ``Q6_K/``
|
||||
and ``<model>-Q6_K/``. Fetching both doubles the download and leaves the variant
|
||||
permanently short of its expected bytes, because the loader only ever opens one.
|
||||
Keep the family holding the lexicographically first file, the shard the lister
|
||||
advertises and the loader opens. A genuinely split GGUF is one family, so all of
|
||||
its shards survive this untouched.
|
||||
"""
|
||||
if len(main_files) < 2:
|
||||
return tuple(main_files)
|
||||
families: dict[str, list[ExpectedFile]] = {}
|
||||
for file in main_files:
|
||||
families.setdefault(gguf_variant_family(file.path), []).append(file)
|
||||
if len(families) < 2:
|
||||
return tuple(main_files)
|
||||
chosen = min(families.values(), key = lambda group: min(file.path for file in group))
|
||||
return tuple(chosen)
|
||||
|
||||
|
||||
def plan_from_expected_files(
|
||||
variant: str,
|
||||
expected_files: Sequence[ExpectedFile],
|
||||
|
|
@ -175,7 +236,14 @@ def plan_from_expected_files(
|
|||
all_mmproj_hashes: frozenset[str] | None = None,
|
||||
) -> GgufVariantPlan:
|
||||
expected = tuple(expected_files)
|
||||
main_files = tuple(file for file in expected if is_main_gguf_variant_path(file.path, variant))
|
||||
all_main = tuple(file for file in expected if is_main_gguf_variant_path(file.path, variant))
|
||||
main_files = _one_shard_family(all_main)
|
||||
# A discarded family has to leave the plan ENTIRELY, not just its main files. It is
|
||||
# target_filenames, required_hashes and download_size_bytes that the worker fetches and the
|
||||
# manifest checks against; leaving the copy there downloaded it, then reclaim deleted it as
|
||||
# not-ours (it is absent from main_hashes) and the job reported partial and fetched it again.
|
||||
kept = {file.path for file in main_files}
|
||||
expected = tuple(file for file in expected if file not in all_main or file.path in kept)
|
||||
companion_files = tuple(file for file in expected if is_companion_gguf_path(file.path))
|
||||
# Manifest-resume fallback for the mmproj fields below: companion_files
|
||||
# also holds the MTP drafter, so keep an mmproj-only view.
|
||||
|
|
|
|||
|
|
@ -29,7 +29,7 @@ from loggers import get_logger
|
|||
logger = get_logger(__name__)
|
||||
|
||||
from hub.utils.gguf import (
|
||||
extract_quant_label,
|
||||
gguf_variant_key,
|
||||
is_big_endian_gguf_path,
|
||||
is_gguf_filename,
|
||||
is_mmproj_filename,
|
||||
|
|
@ -776,7 +776,7 @@ def _completed_gguf_variants(snapshot_dir: Optional[Path]) -> set[str]:
|
|||
rel = path.relative_to(snapshot_dir).as_posix()
|
||||
if not is_gguf_filename(rel) or is_mmproj_filename(rel) or is_mtp_drafter_path(rel):
|
||||
continue
|
||||
quant = extract_quant_label(rel)
|
||||
quant = gguf_variant_key(rel)
|
||||
# Mirror the lister: a big-endian build is never offered, so it cannot vouch for the
|
||||
# quant. Judged with the loader's label, since the two extractors disagree on
|
||||
# F16-be-checkpoint-Q4_K_M and this file must not mark Q4_K_M complete.
|
||||
|
|
|
|||
|
|
@ -50,6 +50,7 @@ from hub.utils.snapshot_filters import (
|
|||
from hub.utils.gguf_plan import (
|
||||
GgufVariantPlan,
|
||||
build_gguf_variant_plans,
|
||||
plan_for_variant,
|
||||
plan_from_expected_files,
|
||||
sibling_sha256,
|
||||
)
|
||||
|
|
@ -615,7 +616,10 @@ def _gguf_variant_target_plan(
|
|||
raise RuntimeError(
|
||||
f"Metadata unavailable while resolving GGUF variant '{variant}' " f"for {repo_id}"
|
||||
) from e
|
||||
return build_gguf_variant_plans(list(info.siblings)).get(variant.lower())
|
||||
# plan_for_variant, not .get: a repo that files every variant under one shared container
|
||||
# qualifies every key, and a stored pin or an explicit repo:Q4_K_M then missed the map and
|
||||
# the worker exited with "No GGUF shards matching variant".
|
||||
return plan_for_variant(build_gguf_variant_plans(list(info.siblings)), variant)
|
||||
|
||||
|
||||
def _download_gguf_variant(repo_id: str, variant: str, hf_token: str | None, mode: str) -> None:
|
||||
|
|
|
|||
|
|
@ -134,7 +134,13 @@ class GgufVariantDetail(BaseModel):
|
|||
"""A single GGUF quantization variant in a HuggingFace repo."""
|
||||
|
||||
filename: str = Field(..., description = "GGUF filename (e.g., 'gemma-3-4b-it-Q4_K_M.gguf')")
|
||||
quant: str = Field(..., description = "Quantization label (e.g., 'Q4_K_M')")
|
||||
quant: str = Field(..., description = "Quantization label or internal GGUF variant key")
|
||||
# Mirrors hub.schemas.inventory.GgufVariantDetail. The route builds THIS model, so a field
|
||||
# that exists only on the hub twin is dropped by pydantic without a word, and a qualified
|
||||
# row falls back to rendering its whole relative path.
|
||||
display_label: Optional[str] = Field(
|
||||
None, description = "Optional user-facing label when quant is an internal key"
|
||||
)
|
||||
size_bytes: int = Field(0, description = "File size in bytes")
|
||||
download_size_bytes: int = Field(0, description = "Total bytes needed to download this variant")
|
||||
downloaded: bool = Field(
|
||||
|
|
|
|||
|
|
@ -244,6 +244,12 @@ def _iter_ggufs(dir_path: Path) -> list[Path]:
|
|||
|
||||
|
||||
def _variant_matches(relative_path: str, needle: str) -> bool:
|
||||
from hub.utils.gguf import gguf_variant_key
|
||||
|
||||
# The variant's own key first: in a repo holding several checkpoints at one quant
|
||||
# the bare label names every one of them, so it cannot pick between them.
|
||||
if gguf_variant_key(relative_path).lower() == needle:
|
||||
return True
|
||||
quant = _extract_quant_label(relative_path).lower()
|
||||
if quant == needle:
|
||||
return True
|
||||
|
|
@ -276,13 +282,22 @@ def _find_gguf_in_dir(dir_path: Path, gguf_variant: Optional[str]) -> Optional[P
|
|||
return None
|
||||
needle = (gguf_variant or "").strip().lower()
|
||||
if needle:
|
||||
for path in ggufs:
|
||||
from hub.utils.gguf import gguf_variant_key
|
||||
|
||||
def _relative(path: Path) -> str:
|
||||
try:
|
||||
relative = path.relative_to(dir_path).as_posix()
|
||||
return path.relative_to(dir_path).as_posix()
|
||||
except ValueError:
|
||||
relative = path.name
|
||||
if _variant_matches(relative, needle):
|
||||
return path
|
||||
return path.name
|
||||
|
||||
# Files this variant owns outright before ones its label merely also names.
|
||||
for owned in (True, False):
|
||||
for path in ggufs:
|
||||
relative = _relative(path)
|
||||
if owned != (gguf_variant_key(relative).lower() == needle):
|
||||
continue
|
||||
if _variant_matches(relative, needle):
|
||||
return path
|
||||
return None
|
||||
candidates = [path for path in ggufs if not _is_nonfirst_gguf_split(path)] or ggufs
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -2752,13 +2752,44 @@ def _prune_empty_parents(start: Path, stop_at: Path) -> None:
|
|||
parent = parent.parent
|
||||
|
||||
|
||||
def _variant_names_same_checkpoint(a: Optional[str], b: Optional[str]) -> bool:
|
||||
"""Whether two variant spellings can name the SAME checkpoint, for the load-state guard.
|
||||
|
||||
Deletion accepts an unambiguous bare quant for a path-qualified key (the shared-container
|
||||
layout, ``weights/model-Q4_K_M.gguf``), so a guard comparing the two spellings literally lets
|
||||
a model loaded through a legacy bare pin be deleted through its advertised qualified row --
|
||||
unlinking the resident model's snapshot and blob. Deliberately loose: a false match only
|
||||
refuses a delete, a false miss loses weights.
|
||||
"""
|
||||
from hub.utils.gguf import bare_quant_alias
|
||||
|
||||
left = (a or "").strip().lower()
|
||||
right = (b or "").strip().lower()
|
||||
if not left or not right:
|
||||
return False
|
||||
if left == right:
|
||||
return True
|
||||
for key, bare in ((left, right), (right, left)):
|
||||
if "/" in key and "/" not in bare and bare_quant_alias(key).lower() == bare:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _delete_gguf_variant_files(root: Path, variant: str) -> tuple[int, int]:
|
||||
deleted_count = 0
|
||||
deleted_bytes = 0
|
||||
for path in root.rglob("*"):
|
||||
if not path.is_file() or not _is_main_gguf_filename(path.name):
|
||||
continue
|
||||
if _extract_quant_label(path.name).lower() != variant.lower():
|
||||
# Keyed on the path, not the basename: a repo holding several checkpoints at
|
||||
# one quant would otherwise delete every one of them for a single row.
|
||||
from utils.models.model_config import _gguf_variant_key
|
||||
|
||||
try:
|
||||
relative = path.relative_to(root).as_posix()
|
||||
except ValueError:
|
||||
relative = path.name
|
||||
if _gguf_variant_key(relative).lower() != variant.lower():
|
||||
continue
|
||||
try:
|
||||
deleted_bytes += path.stat().st_size
|
||||
|
|
@ -2892,7 +2923,9 @@ async def delete_finetuned_model(
|
|||
and (
|
||||
not gguf_variant
|
||||
or not llama_backend.hf_variant
|
||||
or llama_backend.hf_variant.lower() == gguf_variant.lower()
|
||||
# Alias-aware: the delete below accepts a bare quant for a qualified key, so a
|
||||
# literal comparison here would wave through the very spelling it then deletes.
|
||||
or _variant_names_same_checkpoint(llama_backend.hf_variant, gguf_variant)
|
||||
)
|
||||
):
|
||||
raise HTTPException(
|
||||
|
|
@ -2909,7 +2942,7 @@ async def delete_finetuned_model(
|
|||
and (
|
||||
not gguf_variant
|
||||
or not llama_backend.hf_variant
|
||||
or llama_backend.hf_variant.lower() == gguf_variant.lower()
|
||||
or _variant_names_same_checkpoint(llama_backend.hf_variant, gguf_variant)
|
||||
)
|
||||
):
|
||||
raise HTTPException(
|
||||
|
|
@ -3319,25 +3352,34 @@ def _resolve_quant_gguf(repo_id: str, quant: str, is_local: bool) -> tuple[Optio
|
|||
if snaps.is_dir():
|
||||
roots.extend(s for s in snaps.iterdir() if s.is_dir())
|
||||
|
||||
want = _normalized_quant_label(quant)
|
||||
want = (quant or "").strip()
|
||||
best_total = 0
|
||||
best_first: Optional[str] = None
|
||||
for root in roots:
|
||||
matches: list[tuple[str, Path]] = []
|
||||
total = 0
|
||||
ranked: dict[int, list[tuple[str, Path, int]]] = {0: [], 1: []}
|
||||
for f in _iter_gguf_paths(root):
|
||||
try:
|
||||
rel = f.relative_to(root).as_posix()
|
||||
except ValueError:
|
||||
rel = f.name
|
||||
q = _main_variant_gguf_label(rel)
|
||||
if q is None or _normalized_quant_label(q) != want:
|
||||
rank = _main_variant_rank(rel, want)
|
||||
if rank is None:
|
||||
continue
|
||||
try:
|
||||
total += f.stat().st_size
|
||||
size = f.stat().st_size
|
||||
except OSError:
|
||||
continue
|
||||
matches.append((rel, f))
|
||||
ranked[rank].append((rel, f, size))
|
||||
# Exact keys alone when any exist: summing them with the label matches counts other
|
||||
# checkpoints' bytes into this row's estimate and can reveal one of their files.
|
||||
# ... and within those, ONE shard family, the same rule group_gguf_variant_files
|
||||
# applies: a snapshot holding the same quant twice (QwQ-32B's two BF16 shard sets)
|
||||
# would otherwise report double the weights the loader opens, which /kv-cache-estimate
|
||||
# turns into a false exceeds-memory warning and which can make a snapshot look
|
||||
# "more complete" purely for holding a redundant copy.
|
||||
chosen = _one_shard_family_of(ranked[0] or ranked[1])
|
||||
matches = [(rel, f) for rel, f, _size in chosen]
|
||||
total = sum(size for _rel, _f, size in chosen)
|
||||
# Prefer the most complete snapshot so a partial older revision can't underestimate bytes.
|
||||
if matches and total > best_total:
|
||||
matches.sort(key = lambda m: m[0])
|
||||
|
|
@ -3453,6 +3495,9 @@ async def get_gguf_variants(
|
|||
GgufVariantDetail(
|
||||
filename = v.filename,
|
||||
quant = v.quant,
|
||||
# A path-qualified key is not a label a picker can show; without this
|
||||
# the row reads as its whole relative path.
|
||||
display_label = getattr(v, "display_label", None),
|
||||
size_bytes = v.size_bytes,
|
||||
download_size_bytes = int(
|
||||
getattr(v, "download_size_bytes", v.size_bytes) or v.size_bytes
|
||||
|
|
@ -3640,6 +3685,62 @@ def _main_variant_gguf_label(rel_path: str) -> Optional[str]:
|
|||
return label
|
||||
|
||||
|
||||
def _one_shard_family_of(entries: list) -> list:
|
||||
"""*entries* narrowed to the single shard family the loader would open.
|
||||
|
||||
``(rel, path, size)`` triples. Same rule as ``hub.utils.gguf.group_gguf_variant_files``:
|
||||
every shard of one split GGUF shares a family, two files that do not are two checkpoints, and
|
||||
the family kept is the one holding the lexicographically first file. A genuinely split GGUF is
|
||||
one family and survives whole.
|
||||
"""
|
||||
if len(entries) < 2:
|
||||
return list(entries)
|
||||
from hub.utils.gguf import gguf_variant_family
|
||||
|
||||
families: dict[str, list] = {}
|
||||
for entry in entries:
|
||||
families.setdefault(gguf_variant_family(entry[0]), []).append(entry)
|
||||
if len(families) < 2:
|
||||
return list(entries)
|
||||
return min(families.values(), key = lambda group: min(e[0] for e in group))
|
||||
|
||||
|
||||
def _main_variant_rank(rel_path: str, want: str) -> Optional[int]:
|
||||
"""How well *want* names this file's variant: 0 for its own key, 1 for the legacy
|
||||
quant-label spelling, None for neither.
|
||||
|
||||
*want* is the request VERBATIM: the bare-quant folding is applied per comparison, because
|
||||
doing it once up front strips a qualified key's own path punctuation and folds ``exp-a/`` into
|
||||
``expa/``. Both spellings have to resolve -- a stored pin predates the qualified keys -- but
|
||||
they cannot rank equally. In a repo holding several checkpoints at one quant the bare label
|
||||
names every one of them, so a request for the repo-root ``Q6_K`` matched the qualified
|
||||
files too and then took whichever sorted first. Exact keys are used alone whenever any
|
||||
exist, and the label is the fallback for the rows that have no qualified spelling.
|
||||
"""
|
||||
from utils.models.model_config import _gguf_variant_key
|
||||
|
||||
label = _main_variant_gguf_label(rel_path)
|
||||
if label is None:
|
||||
return None
|
||||
if _variant_keys_match(_gguf_variant_key(rel_path), want):
|
||||
return 0
|
||||
return 1 if _normalized_quant_label(label) == _normalized_quant_label(want) else None
|
||||
|
||||
|
||||
def _variant_keys_match(key: str, want: str) -> bool:
|
||||
"""Whether *want* is *key*, for the exact-key test.
|
||||
|
||||
``_normalized_quant_label`` strips hyphens and underscores, which is right for a bare quant
|
||||
(``UD-Q4_K_XL`` and ``udq4kxl`` are the same ask) and wrong for a path: it folds ``exp-a/`` and
|
||||
``expa/`` into one, so two advertised checkpoints both answered to the other's key. A qualified
|
||||
key keeps its punctuation and compares case-insensitively; the legacy folding applies to the
|
||||
bare aliases it was written for.
|
||||
"""
|
||||
if "/" in key or "/" in want:
|
||||
return key.strip().lower() == want.strip().lower()
|
||||
return _normalized_quant_label(key) == _normalized_quant_label(want)
|
||||
|
||||
|
||||
def _normalized_quant_label(label: str) -> str:
|
||||
return label.lower().replace("-", "").replace("_", "")
|
||||
|
||||
|
|
@ -4421,7 +4522,7 @@ def _resolve_cached_model_path(repo_id: str, variant: Optional[str]) -> Path:
|
|||
raise HTTPException(status_code = 404, detail = "Model not found in cache")
|
||||
|
||||
if variant:
|
||||
want = _normalized_quant_label(variant)
|
||||
want = (variant or "").strip()
|
||||
candidate_revisions = sorted(
|
||||
(rev for repo_info in matching_repos for rev in repo_info.revisions),
|
||||
key = lambda rev: getattr(rev, "last_modified", 0) or 0,
|
||||
|
|
@ -4429,7 +4530,7 @@ def _resolve_cached_model_path(repo_id: str, variant: Optional[str]) -> Path:
|
|||
)
|
||||
for rev in candidate_revisions:
|
||||
snapshot = getattr(rev, "snapshot_path", None)
|
||||
matches = []
|
||||
ranked: dict[int, list[tuple[str, Path]]] = {0: [], 1: []}
|
||||
for f in rev.files:
|
||||
p = Path(f.file_path)
|
||||
rel = f.file_name
|
||||
|
|
@ -4438,11 +4539,13 @@ def _resolve_cached_model_path(repo_id: str, variant: Optional[str]) -> Path:
|
|||
rel = p.relative_to(snapshot).as_posix()
|
||||
except ValueError:
|
||||
pass
|
||||
label = _main_variant_gguf_label(rel)
|
||||
if label is None or _normalized_quant_label(label) != want:
|
||||
rank = _main_variant_rank(rel, want)
|
||||
if rank is None:
|
||||
continue
|
||||
if p.exists() or p.is_symlink():
|
||||
matches.append((rel, p))
|
||||
ranked[rank].append((rel, p))
|
||||
# Exact keys alone when any exist, else the legacy label spelling.
|
||||
matches = ranked[0] or ranked[1]
|
||||
if matches:
|
||||
# Path-sorted so a sharded quant deterministically yields its first split.
|
||||
return sorted(matches, key = lambda m: m[0].lower())[0][1]
|
||||
|
|
|
|||
1024
studio/backend/tests/test_gguf_variant_rows.py
Normal file
1024
studio/backend/tests/test_gguf_variant_rows.py
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -2299,6 +2299,111 @@ _GGUF_KNOWN_QUANT_RE = re.compile(
|
|||
)
|
||||
|
||||
|
||||
_FLOAT_PRECISION_QUANTS = frozenset({"BF16", "F16", "F32"})
|
||||
_GGUF_SPLIT_SUFFIX_RE = re.compile(r"-\d{3,}-of-\d{3,}", re.IGNORECASE)
|
||||
|
||||
|
||||
def _select_known_quant_match(text: str):
|
||||
fallback = None
|
||||
for match in _GGUF_KNOWN_QUANT_RE.finditer(text):
|
||||
if match.group(2).upper() in _FLOAT_PRECISION_QUANTS:
|
||||
if fallback is None:
|
||||
fallback = match
|
||||
continue
|
||||
return match
|
||||
return fallback
|
||||
|
||||
|
||||
def _gguf_variant_stem(filename: str) -> str:
|
||||
basename = filename.rsplit("/", 1)[-1]
|
||||
return _GGUF_SPLIT_SUFFIX_RE.sub("", basename.rsplit(".", 1)[0]).strip()
|
||||
|
||||
|
||||
def _gguf_variant_token(filename: str) -> Optional[str]:
|
||||
match = _select_known_quant_match(_gguf_variant_stem(filename))
|
||||
if not match and "/" in filename:
|
||||
for segment in reversed(filename.rsplit("/", 1)[0].split("/")):
|
||||
parent_match = _select_known_quant_match(segment)
|
||||
if parent_match:
|
||||
match = parent_match
|
||||
break
|
||||
return f"{match.group(1) or ''}{match.group(2)}" if match else None
|
||||
|
||||
|
||||
def _gguf_variant_family(filename: str) -> str:
|
||||
stem = _gguf_variant_stem(filename)
|
||||
if "/" not in filename:
|
||||
return stem or "gguf"
|
||||
parents = filename.rsplit("/", 1)[0].strip("/")
|
||||
return f"{parents}/{stem}" if parents and stem else stem or "gguf"
|
||||
|
||||
|
||||
def _gguf_bpw_suffix(filename: str) -> str:
|
||||
"""``-3.53bpw`` from whichever path segment names the quant, else ``""``.
|
||||
|
||||
MIRROR of ``hub.utils.gguf._gguf_bpw_suffix``; see ``_gguf_variant_key``. The quant-directory
|
||||
layout carries the modifier upstairs (``IQ4_XS-3.53bpw/model.gguf``), so the basename alone
|
||||
gave both bpw builds one key. The walk stops at the segment that named the quant.
|
||||
"""
|
||||
path = filename.replace("\\", "/")
|
||||
parents = path.rpartition("/")[0]
|
||||
for segment in (_gguf_variant_family(path).rsplit("/", 1)[-1], *reversed(parents.split("/"))):
|
||||
if not segment:
|
||||
continue
|
||||
match = re.search(r"-[0-9]+(?:\.[0-9]+)?bpw$", segment, re.IGNORECASE)
|
||||
if match:
|
||||
return match.group(0)
|
||||
if _select_known_quant_match(segment) is not None:
|
||||
return ""
|
||||
return ""
|
||||
|
||||
|
||||
def _gguf_variant_key(filename: str) -> str:
|
||||
"""MIRROR of ``hub.utils.gguf.gguf_variant_key``; utils cannot import hub.
|
||||
|
||||
The two must change in lockstep: the hub builds the picker rows with its copy
|
||||
and this one decides which file a chosen row loads, so a disagreement is a row
|
||||
that resolves to another checkpoint's weights.
|
||||
``tests/test_gguf_variant_rows.py`` asserts they agree.
|
||||
"""
|
||||
path = filename.replace("\\", "/")
|
||||
quant = _gguf_variant_token(path)
|
||||
if quant is None:
|
||||
return _gguf_variant_family(path)
|
||||
for segment in path.rpartition("/")[0].split("/"):
|
||||
# A quant-named directory adds nothing the basename does not already say; a
|
||||
# directory naming something else is a different checkpoint and qualifies.
|
||||
if segment and _select_known_quant_match(segment) is None:
|
||||
return _gguf_variant_family(path)
|
||||
# ... and the bpw modifier stays on, so two builds of one base quant keep two identities.
|
||||
return f"{quant}{_gguf_bpw_suffix(path)}"
|
||||
|
||||
|
||||
def _qualified_variant_name(filename: str, label: str) -> str:
|
||||
"""The name these listers advertise for *filename*, given its quant *label*.
|
||||
|
||||
The path-qualified key when a recognised quant token is qualified by a non-quant directory,
|
||||
because there the label alone names several checkpoints at once and the consumers reading
|
||||
these listers -- the /v1 local index, the remote VRAM preflight -- would never see the row
|
||||
they are asked for.
|
||||
|
||||
The label everywhere else, including a bare quant at the repo root and a file with no
|
||||
recognised quant token at all. This
|
||||
module's label for those is the last hyphenated segment while the variant key is the whole
|
||||
stem, and that difference is old, deliberate elsewhere, and nothing to do with several
|
||||
checkpoints sharing a quant. Changing it here would rename every such row and break the pins
|
||||
that hold them (``Qwen3.6-27B-MTP-001-of-002.gguf`` is listed as ``MTP``).
|
||||
"""
|
||||
if _gguf_variant_token(filename) is None:
|
||||
return label
|
||||
key = _gguf_variant_key(filename)
|
||||
# Only a PATH-qualified key. Without the slash the key is the bare quant token, and this
|
||||
# module's label is the richer of the two: it carries the bpw modifier that keeps
|
||||
# ``model-IQ4_XS-3.53bpw.gguf`` and ``model-IQ4_XS-3.97bpw.gguf`` separately selectable, which
|
||||
# the token extractor drops. Swapping in the token would merge them.
|
||||
return key if "/" in key else label
|
||||
|
||||
|
||||
def _is_big_endian_gguf_path(path: str, quant: str = "") -> bool:
|
||||
normalized = path.replace("\\", "/")
|
||||
name = normalized.rsplit("/", 1)[-1]
|
||||
|
|
@ -2471,8 +2576,8 @@ def list_gguf_variants(
|
|||
variants: list[GgufVariantInfo] = []
|
||||
has_vision = False
|
||||
|
||||
quant_totals: dict[str, int] = {} # quant -> total bytes
|
||||
quant_first_file: dict[str, str] = {} # quant -> first filename (display)
|
||||
# (name, quant, size); grouped per shard FAMILY below, not summed across copies.
|
||||
main_files: list[tuple[str, str, int]] = []
|
||||
|
||||
for sibling in info.siblings:
|
||||
fname = sibling.rfilename
|
||||
|
|
@ -2488,17 +2593,15 @@ def list_gguf_variants(
|
|||
if _is_mtp_drafter(fname):
|
||||
continue
|
||||
|
||||
quant = _extract_quant_label(fname)
|
||||
if _is_big_endian_gguf_path(fname, quant):
|
||||
label = _extract_quant_label(fname)
|
||||
if _is_big_endian_gguf_path(fname, label):
|
||||
continue
|
||||
quant_totals[quant] = quant_totals.get(quant, 0) + size
|
||||
if quant not in quant_first_file:
|
||||
quant_first_file[quant] = fname
|
||||
main_files.append((fname, _qualified_variant_name(fname, label), size))
|
||||
|
||||
for quant, total_size in quant_totals.items():
|
||||
for quant, (first_file, total_size) in _group_gguf_variant_files(main_files).items():
|
||||
variants.append(
|
||||
GgufVariantInfo(
|
||||
filename = quant_first_file[quant],
|
||||
filename = first_file,
|
||||
quant = quant,
|
||||
size_bytes = total_size,
|
||||
)
|
||||
|
|
@ -2510,6 +2613,29 @@ def list_gguf_variants(
|
|||
return variants, has_vision
|
||||
|
||||
|
||||
def _group_gguf_variant_files(entries: list[tuple[str, str, int]]) -> dict[str, tuple[str, int]]:
|
||||
"""``quant -> (first filename, size of that quant's shard family)``.
|
||||
|
||||
MIRROR of ``hub.utils.gguf.group_gguf_variant_files`` over ``(name, quant, size)`` triples.
|
||||
Sizes are summed across the shards of ONE family, never across families: a repo shipping the
|
||||
same quant twice (QwQ-32B's BF16 as ``QwQ-32B-BF16-*`` beside ``QwQ-32B.BF16-*``) would
|
||||
otherwise charge both copies to a row the loader only ever opens one of, and
|
||||
``routes/inference.py`` bills this ``size_bytes`` to the VRAM guard, which then refuses a load
|
||||
that fits. The family kept is the one holding the lexicographically first file, which is the
|
||||
shard this lister advertises and the loader opens.
|
||||
"""
|
||||
families: dict[str, dict[str, list[tuple[str, int]]]] = {}
|
||||
for name, quant, size in entries:
|
||||
families.setdefault(quant, {}).setdefault(_gguf_variant_family(name), []).append(
|
||||
(name, int(size or 0))
|
||||
)
|
||||
grouped: dict[str, tuple[str, int]] = {}
|
||||
for quant, by_family in families.items():
|
||||
chosen = min(by_family.values(), key = lambda members: min(n for n, _ in members))
|
||||
grouped[quant] = (min(n for n, _ in chosen), sum(s for _, s in chosen))
|
||||
return grouped
|
||||
|
||||
|
||||
def _resolve_gguf_dir(p: Path) -> Optional[Path]:
|
||||
"""Resolve a path to the directory containing GGUF variants.
|
||||
|
||||
|
|
@ -2551,8 +2677,7 @@ def list_local_gguf_variants(
|
|||
else _registered_custom_model_root(directory)
|
||||
)
|
||||
|
||||
quant_totals: dict[str, int] = {}
|
||||
quant_first_file: dict[str, str] = {}
|
||||
main_files: list[tuple[str, str, int]] = []
|
||||
has_vision = False
|
||||
|
||||
# Recurse so variant-specific subdirs (``BF16/...gguf``) are picked up. Result filenames
|
||||
|
|
@ -2569,20 +2694,18 @@ def list_local_gguf_variants(
|
|||
rel = f.relative_to(p).as_posix()
|
||||
if _is_local_mtp_drafter(f, root, rel):
|
||||
continue
|
||||
quant = _extract_quant_label(rel)
|
||||
if _is_big_endian_gguf_path(rel, quant):
|
||||
label = _extract_quant_label(rel)
|
||||
if _is_big_endian_gguf_path(rel, label):
|
||||
continue
|
||||
quant_totals[quant] = quant_totals.get(quant, 0) + size
|
||||
if quant not in quant_first_file:
|
||||
quant_first_file[quant] = rel
|
||||
main_files.append((rel, _qualified_variant_name(rel, label), size))
|
||||
|
||||
variants = [
|
||||
GgufVariantInfo(
|
||||
filename = quant_first_file[q],
|
||||
quant = q,
|
||||
size_bytes = s,
|
||||
filename = first_file,
|
||||
quant = quant,
|
||||
size_bytes = total_size,
|
||||
)
|
||||
for q, s in quant_totals.items()
|
||||
for quant, (first_file, total_size) in _group_gguf_variant_files(main_files).items()
|
||||
]
|
||||
variants.sort(key = lambda v: -v.size_bytes)
|
||||
return variants, has_vision
|
||||
|
|
@ -2656,6 +2779,7 @@ def _find_local_gguf_by_variant(
|
|||
# Recurse so variants under a quant-named subdir (``BF16/foo-BF16-00001-of-00002.gguf``)
|
||||
# are found. Match the relative path so the quant label can come from the dir name.
|
||||
matches = []
|
||||
owned = []
|
||||
for f in _iter_gguf_files(p, recursive = True):
|
||||
rel = f.relative_to(p).as_posix()
|
||||
if _is_mmproj(f.name) or _is_local_mtp_drafter(f, root, rel):
|
||||
|
|
@ -2675,7 +2799,15 @@ def _find_local_gguf_by_variant(
|
|||
) or _is_big_endian_gguf_path(rel, quant):
|
||||
continue
|
||||
matches.append(f)
|
||||
# A repo holding several checkpoints at one quant offers a row per checkpoint,
|
||||
# and only one of them is this variant's. Name order alone would hand a request
|
||||
# for the repo-root ``Q6_K`` the alphabetically earlier ``distilled/`` copy.
|
||||
if _variant_matches(variant, _gguf_variant_key(rel)):
|
||||
owned.append(f)
|
||||
matches.sort()
|
||||
owned.sort()
|
||||
if owned:
|
||||
return str(_local_gguf_load_path(owned[0]))
|
||||
if matches:
|
||||
return str(_local_gguf_load_path(matches[0]))
|
||||
return None
|
||||
|
|
@ -3482,6 +3614,12 @@ class ModelConfig:
|
|||
is_lora = False,
|
||||
is_gguf = True,
|
||||
gguf_file = gguf_file,
|
||||
# The identity the CALLER asked for, carried through. Dropping it left the
|
||||
# load intent with no variant, so llama.cpp recorded the bare label off the
|
||||
# filename: /status named the root row for a qualified checkpoint, and the
|
||||
# deletion guard compared that bare label against the selected key and let
|
||||
# the delete through.
|
||||
gguf_variant = gguf_variant,
|
||||
gguf_mmproj_file = mmproj_file,
|
||||
gguf_mtp_file = mtp_file,
|
||||
gguf_dspark_file = dspark_file,
|
||||
|
|
@ -3532,10 +3670,23 @@ class ModelConfig:
|
|||
f"Available variants: {available}"
|
||||
)
|
||||
if not variant: # auto-select best quant
|
||||
variant_filenames = [v.filename for v in variants]
|
||||
# ROOT rows when there are any. _pick_best_gguf keeps whichever filename it
|
||||
# met first among equals, so an LTX-style listing that puts distilled/...-Q6_K
|
||||
# before the root ...-Q6_K made a bare repo id load the distilled checkpoint --
|
||||
# while local_model_resolver, the auto-download map and /gguf-variants all
|
||||
# define a bare id as the root. This is the LOAD path, so it has to agree.
|
||||
root_rows = [
|
||||
v.filename
|
||||
for v in variants
|
||||
if "/" not in _qualified_variant_name(v.filename, v.quant)
|
||||
]
|
||||
variant_filenames = root_rows or [v.filename for v in variants]
|
||||
best = _pick_best_gguf(variant_filenames)
|
||||
if best:
|
||||
variant = _extract_quant_label(best)
|
||||
# The SAME identity the lister advertised for that file. Converting the
|
||||
# winner back to a bare label handed the load a name several checkpoints
|
||||
# answer to, and it then resolved whichever sorted first.
|
||||
variant = _qualified_variant_name(best, _extract_quant_label(best))
|
||||
else:
|
||||
variant = "Q4_K_M" # Fallback — llama-server's own default
|
||||
|
||||
|
|
|
|||
|
|
@ -134,7 +134,10 @@ export interface ValidateModelResponse {
|
|||
|
||||
export interface GgufVariantDetail {
|
||||
filename: string;
|
||||
/** Selection identity. Path-qualified when a repo holds several checkpoints at one quant. */
|
||||
quant: string;
|
||||
/** What to SHOW for `quant` ("Q6_K · distilled"); absent when the key already reads as a label. */
|
||||
display_label?: string | null;
|
||||
size_bytes: number;
|
||||
download_size_bytes?: number;
|
||||
downloaded?: boolean;
|
||||
|
|
|
|||
|
|
@ -40,6 +40,7 @@ import {
|
|||
TrainIcon,
|
||||
TransportConflictDialog,
|
||||
deleteCachedModel,
|
||||
ggufVariantDisplayLabel,
|
||||
invalidateGgufVariantsCache,
|
||||
listGgufVariants as listGgufVariantsCached,
|
||||
useGgufVariantsCacheVersions,
|
||||
|
|
@ -1435,7 +1436,10 @@ function GgufVariantExpander({
|
|||
<span
|
||||
className={cn(oom && "!text-gray-500 dark:!text-gray-400")}
|
||||
>
|
||||
{v.quant}
|
||||
{/* The key is the selection identity and can be path-qualified
|
||||
("distilled/ltx-2.3-22b-distilled-Q6_K"); the label is what that reads
|
||||
as ("Q6_K · distilled"). Everything below still keys on v.quant. */}
|
||||
{ggufVariantDisplayLabel(v)}
|
||||
</span>
|
||||
{unusableLocal ? (
|
||||
<span className="ml-1.5 text-ui-9 font-sans font-medium text-amber-700 dark:text-amber-300">
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@ import {
|
|||
import { cn } from "@/lib/utils";
|
||||
import { Link } from "@tanstack/react-router";
|
||||
import { ChevronDownIcon, ChevronRightIcon, RefreshCwIcon } from "lucide-react";
|
||||
import { ggufVariantDisplayLabel } from "@/features/hub";
|
||||
import {
|
||||
type ComponentPropsWithoutRef,
|
||||
type ReactElement,
|
||||
|
|
@ -207,7 +208,10 @@ function LocalGgufVariantList({
|
|||
)}
|
||||
>
|
||||
<span className="min-w-0 flex-1 truncate font-mono">
|
||||
{variant.quant}
|
||||
{/* The key is the selection identity and can be path-qualified
|
||||
("distilled/ltx-2.3-22b-distilled-Q6_K"); the label is what it reads as.
|
||||
onSelect and the recommended check still key on variant.quant. */}
|
||||
{ggufVariantDisplayLabel(variant)}
|
||||
</span>
|
||||
{variant.quant === defaultVariant ? (
|
||||
<Badge variant="secondary" className="h-4 px-1.5 text-ui-10">
|
||||
|
|
|
|||
|
|
@ -54,6 +54,7 @@ import {
|
|||
import { SettingsSection } from "../components/settings-section";
|
||||
import { psSingle, shSingle } from "../components/usage-examples";
|
||||
import { useSettingsPanelPrefsStore } from "../stores/settings-panel-prefs-store";
|
||||
import { ggufVariantDisplayLabel } from "@/features/hub";
|
||||
|
||||
const DOCS_URL = "https://unsloth.ai/docs/integrations/unsloth-start";
|
||||
const EXAMPLE_MODEL_REPO = "unsloth/gemma-4-E4B-it-GGUF";
|
||||
|
|
@ -1473,7 +1474,14 @@ export function AgentsTab() {
|
|||
: t("settings.agents.noQuantizations")
|
||||
}
|
||||
>
|
||||
{selectedVariant}
|
||||
{/* The closed trigger is where the choice is READ, so it shows the label
|
||||
too; SelectValue renders this instead of the raw item text. The value
|
||||
bound to the Select is still the key. */}
|
||||
{ggufVariantDisplayLabel(
|
||||
variants.find((v) => v.quant === selectedVariant) ?? {
|
||||
quant: selectedVariant ?? "",
|
||||
},
|
||||
)}
|
||||
</SelectValue>
|
||||
</SelectTrigger>
|
||||
<SelectContent align="start" className="min-w-[16rem]">
|
||||
|
|
@ -1492,7 +1500,9 @@ export function AgentsTab() {
|
|||
className="[&>span:last-child]:w-full [&>span:last-child]:justify-between"
|
||||
>
|
||||
<span className="font-mono text-xs whitespace-nowrap">
|
||||
{variant.quant}
|
||||
{/* The key is the selection identity and can be path-qualified; the
|
||||
label is what it reads as. The SelectItem value stays the key. */}
|
||||
{ggufVariantDisplayLabel(variant)}
|
||||
</span>
|
||||
{size ? (
|
||||
<span className="text-ui-10 whitespace-nowrap text-muted-foreground">
|
||||
|
|
|
|||
|
|
@ -2257,7 +2257,11 @@ def test_a_standalone_gguf_has_one_settings_key():
|
|||
|
||||
common = _read_backend("hub/services/models/common.py")
|
||||
# The rule this mirrors: a variant is derived only for a single scanned file.
|
||||
assert "extract_quant_label(gguf_files[0].name)" in common
|
||||
# gguf_variant_key, not the hub's extract_quant_label: for a standalone file there is no
|
||||
# directory to qualify, so the key IS the quant token -- and it keeps the bpw modifier, which
|
||||
# is what makes it agree with the loader's own _extract_quant_label (the equality the sibling
|
||||
# test below depends on). The hub label drops that modifier.
|
||||
assert "gguf_variant_key(gguf_files[0].name)" in common
|
||||
assert "if scan_path.is_file() and len(gguf_files) == 1" in common
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue