mirror of
https://github.com/Helldez/BigMoeOnEdge.git
synced 2026-10-03 03:25:42 +00:00
47924565c1
llama.cpp maps every gguf it loads and keeps the mapping for the model's lifetime. On Windows that is expensive in a way nothing had attributed: while a section of a file is alive, NTFS serialises concurrent unbuffered reads on that file, and a lane opened while the section existed keeps serialising against it after the section is gone. Four I/O lanes therefore delivered exactly one lane's throughput, which is why lanes and threads have always measured dead on the desktop host and why the engine read at about a third of what the drive can serve. --release-mmap hands the mapping back after load: unmap the file, close its section, reopen the reader lanes. Both halves are needed. Whether it is safe is decided by looking rather than by reasoning, the engine asks the OS whether any weight the capture pass observed still points inside a mapping of the model files, and declines if any does. Off by default, because the check answers for the pointers the capture saw and for no others. Host A/B on Qwen3.6-35B-A3B Q4_K_M: 3.16 to 4.63 tok/s (+46%), flash stall per token 0.182 to 0.074, with bytes read, hit rate, evictions and re-reads identical to the digit and the generated text byte-identical. On the phone the read path is flat (f2fs does not serialise) but CPU per token falls about 9%; that cell is two short runs per variant and is recorded as a direction, not a number. Also fixes a bug this uncovered, independent of the flag: when a gguf carries no output.weight, llama.cpp builds the output head from the token embedding table and the model holds two identically named tensors over the same bytes. The capture pass keyed its map by name, so --dense-weights anon and ahwb rebound one and left the twin reading the mmap for the whole run, which on a model past RAM means the output projection served by page faults from flash. The capture now records every distinct leaf object by address and the dense policy rebinds every tensor over one file range onto the same buffer. Adds bmoe-iobench --mmap / --reopen-lanes / --range-mb / --fresh, the cells that isolate the mechanism, a mapping_release unit test on both platforms, the app switch "Release the model mapping", and the bench findings. README, architecture, AGENTS and roadmap updated, the last correcting a diagnosis this refutes. |
||
|---|---|---|
| .. | ||
| CMakeLists.txt | ||
| iobench.cpp | ||
| membench.cpp | ||