Each case spawns the real CLI with live hardware detection. Alone a run
takes about 20 s, but inside the full suite on a loaded laptop it went
past the 45 s limit, which failed npm test and therefore npm publish.
Both tests describe Linux hosts, but UnifiedDetector only collects the
generic GPU inventory on Linux and Windows, so on macOS systemGpu stayed
null and the suite failed. Force process.platform to linux for the
duration of each fixture and restore it afterwards.
lookup() scanned every quality row for every candidate. With benchmark
scores present (now shipped in the snapshot, previously after any
`quality-sync`), `recommend --no-registry` evaluated ~7k variants x 7
categories x ~6.6k rows and took ~52 s. sameCheckpoint requires equal
labels, so rows are indexed by label and the result is unchanged: the same
command takes ~3 s.
The base-variant ranking test compares estimates only, since the local
catalog may hold measured scores.
Recognise the Stability AI non-commercial and community licenses, the
Microsoft Research license, the GLM-4 license, Mistral's non-production
license, NVIDIA's AI Foundation Models license, EXAONE, Databricks Open
Model, Health AI Developer Foundations, Yi and Falcon licenses, which were
reported as 'other'. An acceptable-use policy layer is not a license and no
longer stands in for one.
ModelScope publishes the same repo layout as Hugging Face and is the hub
reachable where huggingface.co is not. `registry-sync --sources modelscope`
(and the seed build, 1500 repos by default) lists repos per language task
by downloads, reads each repo's file list (sizes, SHA-256) and config.json
(context window, quantization, vision tower), and normalises them with the
Hugging Face rules (shard sets, exclusions, size checks). Artifacts download
from ModelScope: `modelscope download --model` commands, `resolve/master`
URLs, and `VLLM_USE_MODELSCOPE=True` for vLLM. Loose `chunks/` files are no
longer treated as models.
- `check` on Apple Silicon ranked only the 26 static definitions; it now
ranks the same static + SQLite pool as other platforms (extracted into
buildCombinedModelPool).
- `check` built catalog models from family guesses (llama3 -> 8B, gemma ->
7B, phi -> 3B); it reads the default tag's parameter count and size.
- `ai-check` ignored variants and assumed 7B / params x 0.6 / Q4_K_M; it
reads the default tag's parameters, size, context window and inputs.
- `ai-run` scored catalog models through a 14-model table and a 7B
estimate; the selector now takes the catalog's per-tag facts and keeps
the table only for curated quality scores.
- Refresh the curated fallback catalog from registry data: qwen3/phi4/
deepseek-r1 families, real sizes and context windows, llama3.3:70b and
granite3.3:8b instead of tags that do not exist.
- Fix static definitions (Llama 3.2 context windows, Qwen 2.5 7B
specialization, Mistral Small 3.1 tag and size) and remove
scripts/update-models.js, unused, which injected invented models.
Docker Hub's official `ai/` namespace publishes ~105 language models with
exact sizes per tag (GGUF, safetensors for vLLM, MLX).
- `registry-sync --sources docker` (now in the default and seed sources)
reads the namespace and each repo's tags in both name orders, since
anonymous reads stop at offset 100, and waits out 429 responses.
- Tags sharing a digest are one artifact with aliases; a re-pushed
capitalisation keeps the newest push. Image, video and speech repos are
skipped, and the size/parameter consistency check applies.
- New `docker` runtime: `docker model pull` / `docker model run` commands,
install hints for Docker Desktop and the Linux plugin, a llama.cpp-like
MoE profile, and format-aware hardware support (MLX tags on Apple Silicon,
safetensors/vLLM tags on NVIDIA, GGUF everywhere).
- `sync:seed` runs scripts/update-quality-seed.js, which refreshes every
quality source inside the snapshot (a failed source keeps its previous
scores). A fresh install had no benchmark data and ranked every model by
parameter count until the user ran `quality-sync`.
- Existing databases import a snapshot's quality source when it is newer
than their own copy, and rebuild the catalog cohort.
- New `hf_eval_results` source: GPQA, MMLU-Pro, HLE, SWE-bench,
Terminal-Bench, GSM8K, AIME/HMMT, IFEval and MMMU results attached to Hub
repos. Self-reported, marked not independent, last in each category's
metric preference.
- LiveBench discovers releases from the site bundle and adds those newer
than the curated list; older question sets stay out.
`sync --exact` reads the manifest's license layer once per distinct blob
and classifies it (Llama community licenses, Gemma terms, Qwen and Qwen
Research, Apache-2.0, MIT, CC BY-NC, ...). Licenses are per tag:
qwen2.5:3b ships under the non-commercial Qwen Research License while its
other sizes are Apache-2.0. The model and registry rows take the license
instead of 'unknown'; an unrecognised text is 'other', never guessed.
- Base/pretrained variants (`-base`, `-text`, `-Base`, `-pt`) lose 15
quality points in every generation category. Base coder variants carried
the `coder` tag, so the existing non-instruct coding penalty skipped them
and `deepseek-coder-v2:16b-lite-base` led coding recommendations.
- Ollama tags sharing a digest become one registry artifact with the
others recorded as aliases, and the exact registry parameter count beats
the nominal size in the tag. `:latest` and `:16b` were listed twice.
- Quantizations of one base checkpoint (bartowski, unsloth,
lmstudio-community, ...) collapse into one distinct recommendation.
- The calibration end-to-end test builds its own catalog instead of
depending on the packaged snapshot and the host's hardware.
Exact sizes exposed artifacts that were never the model they claimed:
- collection repos (cortexso/deepseek-r1) gave every distill the
largest model's 70B; the size in a file's own name now wins unless the
header count agrees with it
- speculative-decoding heads (MTP, DFlash, DSpark, EAGLE, draft), LoRA in
GGUF, vision/audio towers, training checkpoints, adapters and per-layer
splits were stored as full models
- `qwen2-1_5b` parsed as 5B
- Kimi-K2's `model-1-of-61` shards were not recognised as one set
A file below a quarter of its precision's nominal bytes per parameter
cannot hold the stated model and is skipped. The seed build also vacuums
the snapshot so replaced rows do not ship as free pages.
Hugging Face
- List per language task (text-generation, image-text-to-text, embeddings)
instead of the global download ranking, which is mostly ASR, diffusion and
classifiers: the 3000-repo limit kept ~1,400 usable repos and the weekly
seed build failed its 2,500-repo floor every week since July.
- Request `expand[]` metadata (full=true omits it): exact parameter counts and
dtypes from safetensors headers, GGUF parameters/architecture/context, card
license and base-model lineage.
- Sweep official model publishers by author, and accept repos without a
pipeline tag that ship for vLLM/mistral-common (official Mistral releases
were all rejected).
- Optional enrichment (`registry-sync --deep`): tree API file sizes and
SHA-256, config.json context window, quantization and vision tower. Honors
the Hub RateLimit header and HF_TOKEN.
- One artifact per complete shard set (including 0-based gpt-oss shards);
drop mmproj/imatrix companions, OpenVINO/ONNX/original/metal exports and
duplicate .bin/consolidated weight sets; store compact metadata.
- Bound the task/modality patterns: `vllm` tagged repos as vision, `encoder`
as coding and every nomic-ai repo as embeddings.
Ollama
- Parse each tags-page row: digest, size (stored in GiB), context window and
input types. Skip cloud-only tags. Aliases sharing a digest share params
and quantization, so `:latest` is no longer params=null/Q4_0.
- Stop fabricating values: no params x 0.5 sizes, no Q4_0 or 4096 defaults;
fix q4_1/q5_1/bf16/mxfp4 mapping and MoE totals (8x7b).
- Capabilities from Ollama's badges instead of whole-page regexes; library
models are official; decode numeric HTML entities.
- `sync --exact` reads registry.ollama.ai manifests and config blobs for exact
bytes, file_type, parameter count, MoE family and vision projectors.
- variants gain digest/size_bytes/blob_sha256 columns (migrated in place).
Consumers
- The deterministic selector reads each tag's own context window and inputs
(it used a model-level field that is never set, so every tag got 4096).
- An existing user database adopts a newer packaged snapshot per part; before,
only the first run ever saw the snapshot.
- A catalog sync rebuilds the registry's Ollama rows; `sync` deleted them and
registry recommendations lost every Ollama model until registry-sync.
`calculateQualityPrior` treated quality as a lookup table on parameter count
(0.5B->45, 7B->75, 70B->95), plus hand-written family bumps and a
`name.includes('code')` match for the coding category. Nothing measured
anything, and four of the seven categories — reasoning, creative, talking,
reading — had no task signal at all, so they ranked purely by size. A coding
model came first for "Creative".
quality-evals.js ingests four public leaderboards into new tables and
`lookupMeasuredQuality` uses them where they exist:
BigCodeBench pass@1, instruct and complete prompt modes
EvalPlus HumanEval+, MBPP+
LiveBench per-category scores that map onto our own categories
MMMU multimodal, val and pro kept apart
Coverage is 54 of 239 models. The rest still fall back to the size estimate and
say so: every recommendation carries a `qualitySource` of either
`{kind:'measured', metric, rawScore, source}` or
`{kind:'estimated', basis:'parameter count'}`, so a guess is never presented as
a measurement.
Three things this gets right that are easy to get wrong:
- New tables rather than the existing `benchmarks` one, which holds local speed
telemetry and is emptied by `ModelDatabase.clear()` on every `fullSync()`.
Quality data stored there would be wiped by a routine catalog refresh.
- The join key is (family_key, params_b), never a variant id: a board measures
an fp16 base checkpoint while a variant is a local Q4 build. A benchmark row
that publishes a size which disagrees is refused rather than attributed, so a
7B build can never inherit a 70B score. Rows with no published size match the
family and are flagged `sizeUnknown`.
- Percentiles rank within the local catalog cohort, not the whole board.
Ranking open models against frontier hosted ones puts every one of them in the
bottom percentile: qwen3@30B measured scored 20.8 against deepseek-r1@14B
estimated at 88.0, so a model was punished for having been benchmarked. The
percentile is then mapped onto the same 45-95 band the size estimator uses, so
measured and estimated stay comparable.
25 new tests cover family normalisation, the size-mismatch refusal, each source
parser, and the cohort. Full suite: 58/58.
Closes#122
fullSync() called ModelDatabase.clear(), which DELETE FROM benchmarks
and dropped locally measured tokens/s telemetry. Snapshot those rows
keyed by model_id + tag before clear, then reattach them to the
recreated variants after scrape.
Replace serial execSync/spawnSync probes with async execFile and
Promise.all so check wall time approaches the slowest probe instead of
the sum. Keep the same detect() result shape, including monkey-patched
sync detect() for existing tests.
Closes#124
Verify-before-load structural safety validation powered by modelvet
(MIT, vendored at 748ac72), compiled to a 19.8 KB WebAssembly module
shipped in the npm package — pure JavaScript, zero native dependencies.
- New `verify <file>` command (exit 0=ACCEPT, 1=REJECT, 2=error; --json)
- Verification gates: `installed --verify` audits installed model blobs;
`ai-run --verify` refuses to run a REJECTED model after pull
- New `structural_validation` policy rule (audit/enforce) with
`verification` fields in audit export reports (JSON/CSV/SARIF)
- New MCP tool `verify_model`; `verify` added to MCP CLI allowlist
- `mcp-setup` is now multi-client: claude, codex, cursor, windsurf,
gemini, kimi, grok, generic (with --apply/--npx/--json)
- Shared Ollama blob resolver (src/security/ollama-blobs.js)
- Vendored amalgamation + rebuild script (vendor/modelvet/,
scripts/build-modelvet-wasm.sh)
New tests: modelvet-verify, verify-gates, policy-structural-validation,
mcp-multiclient. Full suite 53/53.
Virtual monitors from streaming hosts (Apollo/Sunshine via IddSampleDriver), VR
headsets (Meta Quest, Oculus) and remote-desktop tools (Parsec, spacedesk) were
classified as dedicated GPUs. That faked multi-GPU and buried the real card, so a
Radeon RX 7900 XTX was ignored and the checker returned CPU-only results.
- Expanded isRemoteDisplayModel (unified detector, used by the Windows
systeminformation fallback and inventory normalization) and added
isVirtualDisplayAdapter (legacy detector) to drop these adapters.
- Guarded against dropping real GPUs: an Intel "Apollo Lake" iGPU and rare vGPUs
carrying a real-GPU signature (geforce/rtx/radeon rx/arc/a100/...) are kept.
Bumps to 3.7.6. New tests/virtual-monitor-gpu-detection.test.js; full suite 49/49.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`search` and `list-models` were Ollama-catalog only, so the ~32k HF/Ollama/GPT4All
registry was invisible unless you knew the separate registry-* commands.
- `search <query> --registry` (or `--source <ollama|huggingface|gpt4all>`) now
searches the multi-source registry with --max-params/--min-params/--runtime/
--format/--quant/--max-size filters and exact install commands. Default (no
--registry/--source) keeps the Ollama-catalog behavior.
- `list-models --registry`/`--source` lists the registry.
- Extracted the registry search+render into one shared executeRegistrySearch()
used by registry-search and the new search/list-models paths (enum validation
included).
Bumps to 3.7.5. New cases in tests/registry-cli-validation.test.js; full suite 48/48.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- isModelArtifactFile: exclude LoRA/PEFT adapters + optimizer/training files that
were ingested as tiny "models" with the repo's full param count; keep
consolidated.*.pth weights.
- inferQuantization: F16/FP16/BF16 are precisions, not quantizations (no longer
mislabel full-precision HF models as quantized).
- normalizeGpt4AllEntry: parse comma-formatted file sizes; adopt the HF repo id as
canonical_model_id when the download is HF-backed (cross-source dedup).
- Drop the dead idx_model_artifacts_runtime index (JSON column only queried with
LIKE); schema drops it from existing DBs on open.
- Regenerated the packaged seed DB (3 sources, 3,259 repos, 32,779 artifacts).
Bumps to 3.7.4. New tests/registry-ingestor-quality.test.js; full suite 48/48.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- registry-search / registry-recommend reject invalid --source/--format/--runtime
/--optimize with a clear error (exit 1, JSON {error}) instead of returning
"no results" or echoing a bogus runtime.
- selectCategory returns an empty result when no registry artifacts match, so a
registry command never silently substitutes the deterministic selector's
built-in catalog (which mislabeled non-registry rows as registry results).
Bumps to 3.7.3. New tests/registry-cli-validation.test.js; full suite 47/47.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Multi-area bug-hunt pass fixing several causes of FALSE "fits" and dropped picks:
- estimateMemoryBreakdown: size MoE weights by TOTAL params; a real observed
artifact size always wins (removed the sparse-inference override that made a
236B MoE look ~14GB). Fixes the Ollama-catalog path the 3.7.0 registry fix
didn't cover.
- resolveVariantParamsB: disambiguate model_sizes with the variant's own artifact
size, so a `:latest` variant can't inherit model_sizes[0] and poison a sibling
(qwen3:latest 30B->~9B; qwen3:30b keeps 19GB and no longer "fits" 16GB).
- normalizeHardwareProfile: don't multiply a total `vramGB` by gpuCount (2x24=48
stays 48, not 96).
- filterByCategory: guard tags/modalities/name so a malformed pool row can't
throw and silently empty a category.
- registry-recommender: applySourceDiversity preserves the genuine best-by-score
(reserve ~60% of slots) instead of dropping runners-up for obscure sources;
modelDiversityKey no longer collapses distinct unknown-param models; sharded HF
weight files no longer use a per-shard size as the whole-model size.
- mcp-server: add registry-sync/search/recommend to the cli_exec allowlist and
export it.
Bumps to 3.7.2. Full suite 46/46; new tests/selector-memory-sizing.test.js plus
added cases in registry-diversity, model-registry-param-parsing, mcp-server, and
a corrected MoE assertion in deterministic-model-pool-check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Users on non-Apple hardware always saw only Ollama in the registry
recommendations: other sources were scored but never surfaced, and the top was
filled with near-identical quant/shard variants of one model.
- collapseToDistinctModels: collapse quant/shard/tag variants of the same model
(keyed by name + params, ignoring :tag/quant) to a single best-scoring entry,
so the top picks are distinct models (also fixes HF `layers-N.safetensors`
shard spam).
- applySourceDiversity: guarantee a slot for each source whose best candidate
scores within a margin of the top (floor + margin gates), so Hugging Face /
GPT4All artifacts surface in recommend/check/registry-recommend when they are
competitive — without ever promoting a clearly worse model.
- selectCategory ranks a wider window, then collapses + diversifies before
trimming to the caller's limit.
Bumps to 3.7.1. Full suite 45/45; new tests/registry-diversity.test.js.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The registry could recommend Mixture-of-Experts models that cannot physically
fit the user's hardware (e.g. a 397B-A17B model reported as "fits in 11GB" on a
24GB Mac). Two root causes:
1. The recommender read the stored/active parameter count for memory sizing, so
an MoE saved as a single expert (Mixtral 8x7B -> 7B) or by its active count
(397B-A17B -> 17B) looked tiny. It now sizes by the TOTAL, re-derived from the
model name (MoE-aware) and max'd with the stored column, so a stale DB value
can never shrink a model.
2. Setting activeParamsB switched the selector to a "sparse inference" memory
model (sizing by active params), which is false for Ollama/Metal/vLLM (all
experts resident). We now flag MoE + carry totalParamsB but NOT activeParamsB,
keeping memory conservative; active params remain a name-derived speed signal.
Also: ingestor stores the MoE total in parameter_count_b (HF + Ollama paths), and
the packaged seed DB was regenerated so stored MoE totals are correct
(Mixtral-8x7B 7B->56B, Qwen3.5-397B-A17B 17B->397B total / 17B active).
Full suite 44/44. New assertions in tests/model-registry-param-parsing.test.js
including an end-to-end memory check that a 397B MoE requires >100GB.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review fixes for the multi-source registry (PR #99):
- parseParamsB (ingestors + recommender): "NxMB" MoE naming (e.g. Mixtral 8x7B)
now sizes as experts * per-expert total, not a single 7B expert.
- Drop 'k'/'thousand' as parameter units and clamp round-to-zero, so context
tokens like "128k" are no longer misread as ~0B and silently dropped.
- model-database.searchModelArtifacts: escape LIKE wildcards in the runtime
filter so a value like "lla_a" can't over-match "llama".
- normalizeGpt4AllEntry: recover the name from the last non-empty URL segment
so trailing-slash URLs aren't dropped.
- Bump version to 3.7.0 (feature release) and add an integration test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
