mirror of
https://github.com/Helldez/BigMoeOnEdge.git
synced 2026-10-03 03:25:42 +00:00
adbe3caaf2
* feat(moe): stream split multi-shard ggufs natively + DeepSeek V4 Flash recipe Hugging Face rejects single files above 50 GB, so every large model ships as -00001-of-0000N.gguf shards; until now the streamer assumed one file, forcing a merge with double the disk. gguf_offsets now fans the first shard out to the whole set and resolves every tensor to (shard, offset); the expert streamer and the dense loader open one positioned reader per shard and route each read by the tensor's shard index. Pass the first shard, exactly as llama.cpp takes it; a missing sibling fails the load with the shard named. Add the deepseek4 recipe row: V3.2-style routing (256 routed experts, a per-expert bias like lfm2moe, an always-on shared expert that stays resident) over the standard split expert suffixes. The V4 compressed-attention machinery is dense-side llama.cpp code, invisible to the streaming seam. The byte-identity gates gain a 4-shard qwen3moe fixture (metadata-only first shard, the layout large quants actually use); make-tiny-moe.py learns --split-max-tensors. All gates pass, split included. * fix(moe): cache auto must budget for the anon dense conversion The auto budget read MemAvailable while the dense weights were still reclaimable page cache, then dense-weights=anon converted them into buffers the kernel cannot take back: the same bytes planned twice. Latent since the anon policy shipped (dense sets were 2-3 GiB and explicit budgets were the benched path); DeepSeek V4 Flash's 6.5 GiB dense set turned it into a device-taking overcommit on first load. The budget now deducts the pending conversion and says so in the log. * fix(moe): review pass on the multi-shard path Three defects the split rewrite introduced, none of which the gates could see: - The shard index rode in an int8_t, so a model past 127 shards wrapped to a negative index into the reader vector. The bounds check could never catch it: it validated the untruncated value. Widened to int16_t, which covers the whole -%05d-of-%05d filename space. - DenseWeights::warm() reused one flag as both the inner loop condition and the partial-warm report, so the first shard that failed to open silently skipped the warm-up of every later shard. Per-shard condition, sticky report. - The dense readers stayed allocated for the session after read_anonymous had copied and rebound every tensor: fds and a per-lane bounce buffer per shard, sitting next to a cache counting every MiB. Released at the end of init. Also: the streaming banner read O_DIRECT off shard 0, which under the small-first-shard layout is metadata only and too short to verify, so it could claim a mode the shards carrying experts had not got. It now reports the weakest of the readers. * build: the engine version says 0.19.0, like the changelog does The version is declared in CMakeLists.txt and reported by `--version` and by the run-parameter preamble of every metrics CSV, so a committed benchmark file names the engine that produced it. This release section landed while the number stayed at 0.18.0, which would have stamped the wrong engine on every CSV this branch produces, defeating the one purpose the string has. |
||
|---|---|---|
| .. | ||
| bench-analyze.py | ||
| bench-lib.ps1 | ||
| bench-matrix-rework.ps1 | ||
| bench-matrix.ps1 | ||
| bench-pr23-c2000.ps1 | ||
| bench-pr23-summary.py | ||
| bench-prefetch.ps1 | ||
| bench-run.sh | ||
| bench-warmonly.sh | ||
| build-android.ps1 | ||
| build-host.sh | ||
| decode-analyze.py | ||
| gptoss-matrix.sh | ||
| gptoss-mmap.sh | ||
| make-tiny-moe.py | ||
| route-analyze.py | ||
| route-drop-replay.py | ||
| route-replay.py | ||
| route-viewer.py | ||
| trace_io.py | ||