Each case spawns the real CLI with live hardware detection. Alone a run
takes about 20 s, but inside the full suite on a loaded laptop it went
past the 45 s limit, which failed npm test and therefore npm publish.
Both tests describe Linux hosts, but UnifiedDetector only collects the
generic GPU inventory on Linux and Windows, so on macOS systemGpu stayed
null and the suite failed. Force process.platform to linux for the
duration of each fixture and restore it afterwards.
npm audit --omit=dev now reports GHSA-6qxp-vccf-f47h (@modelcontextprotocol/sdk
<= 1.30.1) and GHSA-jqcg-44mw-7w3h (proxy-addr <= 2.0.7). Lockfile-only update
within the declared ranges: sdk 1.32.1, proxy-addr 2.0.8, plus brace-expansion
and js-yaml patch releases.
Built with the new pipeline (sync --exact, task-planned Hugging Face
listings with official publishers and per-repo enrichment, ModelScope,
Docker Hub, and benchmark scores):
- Ollama: 230 local models, 7,392 tags with exact sizes, quantizations,
parameter counts, context windows, inputs and licenses
- Registry: 10,965 repos, 55,064 artifacts (Hugging Face 39,640,
ModelScope 7,914, Ollama 6,425, Docker 1,053, GPT4All 32); every artifact
has an observed size
- Quality: 6,619 benchmark scores from 7 sources
The file grows from 43 MB to 92 MB (npm tarball ~14.5 MB).
lookup() scanned every quality row for every candidate. With benchmark
scores present (now shipped in the snapshot, previously after any
`quality-sync`), `recommend --no-registry` evaluated ~7k variants x 7
categories x ~6.6k rows and took ~52 s. sameCheckpoint requires equal
labels, so rows are indexed by label and the result is unchanged: the same
command takes ~3 s.
The base-variant ranking test compares estimates only, since the local
catalog may hold measured scores.
With native SQLite the snapshot is attached and each table copied in one
statement over the columns both schemas share, instead of ~50k row-by-row
upserts: a first run after an upgrade drops from ~10 s to ~2.6 s. The
sql.js backend keeps the row path. The packaged file is never written.
Recognise the Stability AI non-commercial and community licenses, the
Microsoft Research license, the GLM-4 license, Mistral's non-production
license, NVIDIA's AI Foundation Models license, EXAONE, Databricks Open
Model, Health AI Developer Foundations, Yi and Falcon licenses, which were
reported as 'other'. An acceptable-use policy layer is not a license and no
longer stands in for one.
ModelScope publishes the same repo layout as Hugging Face and is the hub
reachable where huggingface.co is not. `registry-sync --sources modelscope`
(and the seed build, 1500 repos by default) lists repos per language task
by downloads, reads each repo's file list (sizes, SHA-256) and config.json
(context window, quantization, vision tower), and normalises them with the
Hugging Face rules (shard sets, exclusions, size checks). Artifacts download
from ModelScope: `modelscope download --model` commands, `resolve/master`
URLs, and `VLLM_USE_MODELSCOPE=True` for vLLM. Loose `chunks/` files are no
longer treated as models.
- `check` on Apple Silicon ranked only the 26 static definitions; it now
ranks the same static + SQLite pool as other platforms (extracted into
buildCombinedModelPool).
- `check` built catalog models from family guesses (llama3 -> 8B, gemma ->
7B, phi -> 3B); it reads the default tag's parameter count and size.
- `ai-check` ignored variants and assumed 7B / params x 0.6 / Q4_K_M; it
reads the default tag's parameters, size, context window and inputs.
- `ai-run` scored catalog models through a 14-model table and a 7B
estimate; the selector now takes the catalog's per-tag facts and keeps
the table only for curated quality scores.
- Refresh the curated fallback catalog from registry data: qwen3/phi4/
deepseek-r1 families, real sizes and context windows, llama3.3:70b and
granite3.3:8b instead of tags that do not exist.
- Fix static definitions (Llama 3.2 context windows, Qwen 2.5 7B
specialization, Mistral Small 3.1 tag and size) and remove
scripts/update-models.js, unused, which injected invented models.
Docker Hub's official `ai/` namespace publishes ~105 language models with
exact sizes per tag (GGUF, safetensors for vLLM, MLX).
- `registry-sync --sources docker` (now in the default and seed sources)
reads the namespace and each repo's tags in both name orders, since
anonymous reads stop at offset 100, and waits out 429 responses.
- Tags sharing a digest are one artifact with aliases; a re-pushed
capitalisation keeps the newest push. Image, video and speech repos are
skipped, and the size/parameter consistency check applies.
- New `docker` runtime: `docker model pull` / `docker model run` commands,
install hints for Docker Desktop and the Linux plugin, a llama.cpp-like
MoE profile, and format-aware hardware support (MLX tags on Apple Silicon,
safetensors/vLLM tags on NVIDIA, GGUF everywhere).
`npm audit --omit=dev` fails CI with advisories published after the last
green run on main (GHSA-hrr3-gc8f-f4qj, GHSA-gqvv-2mrq-wpjv and others,
GHSA-rpw4-54j3-4h4q and others). `npm audit fix` moves the three transitive
dependencies of @modelcontextprotocol/sdk to patched releases within the
existing ranges; package.json is unchanged.
- `sync:seed` runs scripts/update-quality-seed.js, which refreshes every
quality source inside the snapshot (a failed source keeps its previous
scores). A fresh install had no benchmark data and ranked every model by
parameter count until the user ran `quality-sync`.
- Existing databases import a snapshot's quality source when it is newer
than their own copy, and rebuild the catalog cohort.
- New `hf_eval_results` source: GPQA, MMLU-Pro, HLE, SWE-bench,
Terminal-Bench, GSM8K, AIME/HMMT, IFEval and MMMU results attached to Hub
repos. Self-reported, marked not independent, last in each category's
metric preference.
- LiveBench discovers releases from the site bundle and adds those newer
than the curated list; older question sets stay out.
`sync --exact` reads the manifest's license layer once per distinct blob
and classifies it (Llama community licenses, Gemma terms, Qwen and Qwen
Research, Apache-2.0, MIT, CC BY-NC, ...). Licenses are per tag:
qwen2.5:3b ships under the non-commercial Qwen Research License while its
other sizes are Apache-2.0. The model and registry rows take the license
instead of 'unknown'; an unrecognised text is 'other', never guessed.
- Base/pretrained variants (`-base`, `-text`, `-Base`, `-pt`) lose 15
quality points in every generation category. Base coder variants carried
the `coder` tag, so the existing non-instruct coding penalty skipped them
and `deepseek-coder-v2:16b-lite-base` led coding recommendations.
- Ollama tags sharing a digest become one registry artifact with the
others recorded as aliases, and the exact registry parameter count beats
the nominal size in the tag. `:latest` and `:16b` were listed twice.
- Quantizations of one base checkpoint (bartowski, unsloth,
lmstudio-community, ...) collapse into one distinct recommendation.
- The calibration end-to-end test builds its own catalog instead of
depending on the packaged snapshot and the host's hardware.
Exact sizes exposed artifacts that were never the model they claimed:
- collection repos (cortexso/deepseek-r1) gave every distill the
largest model's 70B; the size in a file's own name now wins unless the
header count agrees with it
- speculative-decoding heads (MTP, DFlash, DSpark, EAGLE, draft), LoRA in
GGUF, vision/audio towers, training checkpoints, adapters and per-layer
splits were stored as full models
- `qwen2-1_5b` parsed as 5B
- Kimi-K2's `model-1-of-61` shards were not recognised as one set
A file below a quarter of its precision's nominal bytes per parameter
cannot hold the stated model and is skipped. The seed build also vacuums
the snapshot so replaced rows do not ship as free pages.
- `sync:seed` runs the Ollama sync with `--exact` (registry byte sizes,
quantizations and parameter counts).
- The registry seed sweeps official publishers and enriches each Hugging
Face repo with tree sizes, hashes and config.json context windows; the
default Hugging Face budget is 6000 repos split by task.
- The weekly workflow passes the optional HF_TOKEN secret for a larger
rate-limit window and allows the longer build.
- Document what the snapshot contains, the new sync flags, and the
changes in the changelog.
`recommend` reads `--category`; the positional value the MCP `recommend`
and `project_recommend` tools passed was ignored, so every category
returned general recommendations.
Hugging Face
- List per language task (text-generation, image-text-to-text, embeddings)
instead of the global download ranking, which is mostly ASR, diffusion and
classifiers: the 3000-repo limit kept ~1,400 usable repos and the weekly
seed build failed its 2,500-repo floor every week since July.
- Request `expand[]` metadata (full=true omits it): exact parameter counts and
dtypes from safetensors headers, GGUF parameters/architecture/context, card
license and base-model lineage.
- Sweep official model publishers by author, and accept repos without a
pipeline tag that ship for vLLM/mistral-common (official Mistral releases
were all rejected).
- Optional enrichment (`registry-sync --deep`): tree API file sizes and
SHA-256, config.json context window, quantization and vision tower. Honors
the Hub RateLimit header and HF_TOKEN.
- One artifact per complete shard set (including 0-based gpt-oss shards);
drop mmproj/imatrix companions, OpenVINO/ONNX/original/metal exports and
duplicate .bin/consolidated weight sets; store compact metadata.
- Bound the task/modality patterns: `vllm` tagged repos as vision, `encoder`
as coding and every nomic-ai repo as embeddings.
Ollama
- Parse each tags-page row: digest, size (stored in GiB), context window and
input types. Skip cloud-only tags. Aliases sharing a digest share params
and quantization, so `:latest` is no longer params=null/Q4_0.
- Stop fabricating values: no params x 0.5 sizes, no Q4_0 or 4096 defaults;
fix q4_1/q5_1/bf16/mxfp4 mapping and MoE totals (8x7b).
- Capabilities from Ollama's badges instead of whole-page regexes; library
models are official; decode numeric HTML entities.
- `sync --exact` reads registry.ollama.ai manifests and config blobs for exact
bytes, file_type, parameter count, MoE family and vision projectors.
- variants gain digest/size_bytes/blob_sha256 columns (migrated in place).
Consumers
- The deterministic selector reads each tag's own context window and inputs
(it used a model-level field that is never set, so every tag got 4096).
- An existing user database adopts a newer packaged snapshot per part; before,
only the first run ever saw the snapshot.
- A catalog sync rebuilds the registry's Ollama rows; `sync` deleted them and
registry recommendations lost every Ollama model until registry-sync.
`calculateQualityPrior` treated quality as a lookup table on parameter count
(0.5B->45, 7B->75, 70B->95), plus hand-written family bumps and a
`name.includes('code')` match for the coding category. Nothing measured
anything, and four of the seven categories — reasoning, creative, talking,
reading — had no task signal at all, so they ranked purely by size. A coding
model came first for "Creative".
quality-evals.js ingests four public leaderboards into new tables and
`lookupMeasuredQuality` uses them where they exist:
BigCodeBench pass@1, instruct and complete prompt modes
EvalPlus HumanEval+, MBPP+
LiveBench per-category scores that map onto our own categories
MMMU multimodal, val and pro kept apart
Coverage is 54 of 239 models. The rest still fall back to the size estimate and
say so: every recommendation carries a `qualitySource` of either
`{kind:'measured', metric, rawScore, source}` or
`{kind:'estimated', basis:'parameter count'}`, so a guess is never presented as
a measurement.
Three things this gets right that are easy to get wrong:
- New tables rather than the existing `benchmarks` one, which holds local speed
telemetry and is emptied by `ModelDatabase.clear()` on every `fullSync()`.
Quality data stored there would be wiped by a routine catalog refresh.
- The join key is (family_key, params_b), never a variant id: a board measures
an fp16 base checkpoint while a variant is a local Q4 build. A benchmark row
that publishes a size which disagrees is refused rather than attributed, so a
7B build can never inherit a 70B score. Rows with no published size match the
family and are flagged `sizeUnknown`.
- Percentiles rank within the local catalog cohort, not the whole board.
Ranking open models against frontier hosted ones puts every one of them in the
bottom percentile: qwen3@30B measured scored 20.8 against deepseek-r1@14B
estimated at 88.0, so a model was punished for having been benchmarked. The
percentile is then mapped onto the same 45-95 band the size estimator uses, so
measured and estimated stay comparable.
25 new tests cover family normalisation, the size-mismatch refusal, each source
parser, and the cohort. Full suite: 58/58.
