data/hf_models.json and data/benchmark_cache.json were byte-identical
copies of the files in llmfit-core/data/ (the ones actually embedded via
include_str!), adding ~11 MB of redundant git-tracked data that could
silently drift. Scrapers now write only the canonical llmfit-core/data/
path; scripts and docs updated to match.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The HuggingFace model scraper was silently broken — the `offset` parameter
is ignored by the HF API, so pagination only ever fetched the first page.
This switches to cursor-based pagination via the Link header, adds multiple
sort strategies (downloads, trendingScore, likes30d), makes the database
additive so historical models are never lost, and allows quantization/repack
orgs (bartowski, unsloth, etc.) since users actually run those variants.
Key changes:
- Fix pagination: use HF API cursor-based pagination (Link header)
- Multi-strategy discovery: downloads + trendingScore + likes30d, merged
- Additive database: existing models retained, fresh data updates in place
- Allow repack orgs: only trl-internal-testing is blocked now
- Config.json fallback: estimate params when safetensors metadata missing
- Discovery enabled by default with 1000 model limit
- Add feature-extraction pipeline for embedding models
- Filter statistics printed every run for transparency
Result: 1520 models (up from 1023), including 364 retained historical.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: Add Qwen3.5 Claude Opus 4.6 reasoning models
* feat: Support parallel models update
* fix: Fix cursor navigate in tui
* feat: Support parallel models enriching
* feat: Support marquee animation for model name column auto if text too long
* chore: Ignore __pycache__
* chore: Remove hf_models backup
* chore: Remove duplicate Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distilled-GGUF in scrape hf models
