mirror of
https://github.com/Helldez/BigMoeOnEdge.git
synced 2026-10-03 11:35:50 +00:00
674e7dc136
A first command with nothing but -m, -p and -t ran plain llama.cpp on mmap: streaming off, cache off, and a dense policy that only applies once streaming is on. Nothing in the report said so, because the moe-stream: block only prints when streaming is enabled, so a baseline run read as a measurement of this engine and got reported as one (#186). A mode: line is now printed on every run. With streaming off on a MoE architecture the build has a recipe for it says the run is a baseline and names the flag; on any other model it says the architecture is not one this build streams. RunSummary carries the model's arch so the CLI can tell those apart. --cache-mb defaults to auto whenever --moe-stream is on. The previous default of 0 meant the cache was off, which re-reads every routed expert from flash every token. On a 16 GB host streaming Qwen3.6-35B-A3B Q4_K_M, 63 tokens, -t 8 --overlap: 1.164 to 2.351 tok/s and 585 to 238 MiB per token, same output. The default is resolved in the CLI, not in the library, so an embedder passing 0 still means no cache; an explicit --cache-mb or BMOE_CACHE_MB still wins, including an explicit 0. Reported by @eiffel31.
145 lines
10 KiB
Markdown
145 lines
10 KiB
Markdown
# Expert cache sizing
|
||
|
||
The expert cache budget (`--cache-mb`) is the dominant throughput lever, but the right value is
|
||
device- and model-specific: too small and the hit rate collapses **to exactly zero — see the
|
||
cliff below, it does not degrade gracefully**; too large and the pinned cache
|
||
plus the mmap-resident model push `MemAvailable` to zero and the Android low-memory-killer takes
|
||
the process (this is exactly why Gemma cannot use a 4000 MiB cache on an 11 GB phone — see
|
||
[benchmarks.md](benchmarks.md)). `--cache-mb auto` removes the guess: the engine sizes the cache to
|
||
the device once at load and holds that budget for the whole run.
|
||
|
||
## The cliff: below one token cycle the hit rate is 0 %, not "low"
|
||
|
||
Global LRU drops the least *recently* used entry. The model visits its layers in a fixed cycle
|
||
0..N-1, so the least recently used entry is also the **soonest to be needed again**. While the
|
||
budget holds a whole token cycle the cold end is genuinely stale history and this never shows.
|
||
Once it does not, the cache evicts precisely what it is about to read, and the hit rate does not
|
||
taper — it goes to **exactly 0.0 %**.
|
||
|
||
Measured on device (2026-07-20): gpt-oss-120b at top-4 has a `token_demand_MiB` of 1815, and a
|
||
**1500 MiB** budget returns 0.0 % hit while reading 1817.05 MiB/token — *identical to running with
|
||
no cache at all* (1817.0) — and still spends 0.233 s/token, 24 % of the token, managing it, on top
|
||
of holding 1500 MiB of RAM. Dropping the budget 2000 → 1500 does not cost a quarter of the hit
|
||
rate; it costs all of it.
|
||
|
||
Two traps worth knowing:
|
||
|
||
- **1500 is a legal value.** It is exactly `cache_min_mb`; no `--force-cache` is needed. Today's
|
||
only protection is that this floor happens to sit above the cycle for the shipped models at
|
||
their default top-k. Raise `--n-expert-used` and the cycle moves above the floor (gpt-oss: 908
|
||
MiB at k=2, 1815 at k=4) and the protection silently stops holding.
|
||
- **The existing floor is a different quantity.** `layer_demand_MiB` (the widest single layer, 50.4
|
||
on that run) is the mechanical minimum; the cliff is at `token_demand_MiB` (1815). They are
|
||
roughly `n_layer` apart. Both are already printed in the `# summary` line; nothing sizes from them.
|
||
|
||
So when a budget is being chosen by hand, check it against `token_demand_MiB` from a previous run
|
||
of the same model *and top-k*, not against `cache_min_mb`. Below the cycle, `--cache-mb 0` is
|
||
strictly better than a cache: same reads, none of the RAM, none of the management cost.
|
||
|
||
Fixing this by changing the eviction policy was tried and rejected — a per-layer partition is
|
||
immune to the cliff by construction but costs ~30 % throughput
|
||
([bench-data/2026-07-20-cache-replay/layer-lfu-verdict.md](bench-data/2026-07-20-cache-replay/layer-lfu-verdict.md)).
|
||
The cheap fix was a guard, and it is in: the worst-case cycle is priced at init from the model's
|
||
shape alone — every bound layer's expert-entry bytes times `min(top_k, n_expert)`, at the top-k the
|
||
run actually applies. The engine records it in every metrics preamble as `cache_cycle_mb` next to
|
||
the budget it is being compared against, and prints one line to stderr at load when the budget is
|
||
under it:
|
||
|
||
```
|
||
bmoe: WARNING expert cache 1500 MiB is below this model's worst-case token cycle of 1815 MiB
|
||
at top-k 4 — no entry can survive to the next token, so expect a hit rate near zero.
|
||
```
|
||
|
||
It states the fact and stops there; the advice above — that `--cache-mb 0` is strictly better below
|
||
the cliff — stays here rather than in the engine's output. Note this is the *worst case* (a disjoint
|
||
routed set at every layer), so it is an upper bound on the measured `token_demand_MiB`, and a budget
|
||
above it is safe for any routing. Recording it matters as much as the warning: a committed CSV whose
|
||
budget sat under the cycle now says so, without needing a run of the same model to compare against.
|
||
|
||
## What it does
|
||
|
||
- **At init**, once the full expert-set size is known, the budget is set to
|
||
`available_RAM − cache_floor_mb`, clamped to `[cache_min_mb, total expert bytes]`. Available memory
|
||
is read from the platform (`/proc/meminfo` `MemAvailable` on Linux/Android, `GlobalMemoryStatusEx`
|
||
on Windows); if it is unknown the budget falls back to the `cache_min_mb` floor.
|
||
- **Also at init**, the dense (non-expert) regions of the gguf — header, embeddings, attention,
|
||
norms, lm_head, the tensors the streamer leaves mmap-resident — are warmed into the page cache with
|
||
one sequential buffered sweep (reported as `bmoe: dense warm-up`), so the first tokens do not pay
|
||
for them as random 4 KiB faults. On a model far larger than RAM this is the difference between a
|
||
fast first token and a ~20-token slow-start ramp: measured on gpt-oss-120b, the first-five-token
|
||
wall average drops ~20× (see [benchmarks.md](benchmarks.md)). On models whose dense set is small it
|
||
is a harmless no-op. This sweep is the `--dense-weights warm` policy; `--dense-weights mmap`
|
||
disables it for A/B runs, and `--dense-weights anon` (the default) replaces it with an O_DIRECT
|
||
read into anonymous buffers, which is the better answer once the model is well past RAM — the
|
||
case the engine targets.
|
||
The warm-up is deliberately kept *out* of the budget: it only pre-faults the mmap-resident pages,
|
||
it does not pin or reserve them, so the expert-cache budget above is unchanged and its hit rate is
|
||
identical with and without it. (An alternative that folds the dense bytes into the floor —
|
||
reserving RAM so the expert cache can never evict them — was measured and rejected: on a
|
||
cache-sensitive model it lowers the budget and the hit rate, e.g. Gemma budget 4000→2909 MiB, hit
|
||
83%→73%, trading throughput for OOM headroom that the warm-up already avoids needing. See
|
||
[bench-data/2026-07-14-warmup/](bench-data/2026-07-14-warmup/).)
|
||
- **During generation, nothing resizes it.** `auto` is one shot at load, not a control loop: the
|
||
budget chosen at init is held for the whole run. A runtime governor that tracked free RAM and
|
||
shrank the budget under pressure did exist and was **retired** — it was measured a net loss on
|
||
the models it was built for (see [pressure.md](pressure.md)). The `moe-cache:` summary reports the
|
||
budget and what actually stayed resident:
|
||
|
||
```
|
||
moe-cache: 77.1% hit, resident 4000.0 MiB
|
||
```
|
||
|
||
Because expert reads use O_DIRECT they never enter the page cache, shrinking this budget is what
|
||
actually hands expert RAM back to the rest of the system. It is not the engine's only large
|
||
allocation, though: under `--dense-weights anon` (the default) the whole dense set also lives in
|
||
anon buffers. That footprint is **not** part of the cache budget and is not subtracted from it —
|
||
see the ordering warning below.
|
||
|
||
> **The budget is not only a throughput knob — it is what the kernel judges you by.** On Android the
|
||
> LRU promotes a page to the protected list only on a *second* reference, and a cache hit is that
|
||
> second reference: a cache with a high hit rate defends itself, one with a low hit rate is correctly
|
||
> read as cold and reclaimed. Measured on gpt-oss-120b, where 3000 MiB covers 5.2% of the expert bank
|
||
> and returns a 13% hit, the cache is taken back *while decoding* and the fight costs far more than
|
||
> the hits are worth. `MemAvailable` also over-states the headroom here, since it counts the page
|
||
> cache holding this model's own dense weights as free. Before trusting `auto` on a model whose
|
||
> expert set dwarfs the budget, read [android-memory.md](android-memory.md).
|
||
|
||
> **`auto` sizes before the dense weights are allocated.** The budget is chosen in
|
||
> `ExpertStreamSource::init` as soon as the expert-set size is known
|
||
> (`core/src/moe/expert_stream_source.cpp:68`); the dense policy runs later in the same init
|
||
> (`:200`), and under the default `anon` mode it then allocates the entire dense set into anon
|
||
> buffers. So the `MemAvailable` reading `auto` sizes from still counts that RAM as free. On a model
|
||
> with a large dense set the over-ask is roughly the dense size — `--cache-ceil-mb` is the lever that
|
||
> bounds it, which is why the Android example ships a 3000 MiB ceiling by default.
|
||
|
||
> **`auto` sizes from a signal that lies, so keep it modest.** `auto` reads `MemAvailable`, which
|
||
> reports memory the device will not actually concede (it counts the model's own mmap'd weights as
|
||
> free), so it over-asks — and an over-ask is not a wasted budget but a running fight. The runtime
|
||
> governor that once tried to correct this from the other end (`--cache-dynamic`) was retired as a
|
||
> net loss (see [pressure.md](pressure.md)); `auto` now sizes **once at load** and stays fixed, so
|
||
> bound it with `--cache-ceil-mb` on a model whose expert set dwarfs the device, or use cache-off.
|
||
|
||
## Flags
|
||
|
||
| Flag | Meaning |
|
||
|---|---|
|
||
| `--cache-mb auto` | size the cache to the device instead of a fixed MiB (mutually exclusive with a numeric `--cache-mb`). **The CLI's default whenever `--moe-stream` is on**: pass a number, or `0` for no cache, to override it. The library's own default is still no cache, so an embedder passing 0 keeps meaning it |
|
||
| `--cache-floor-mb N` | RAM to leave free for the rest of the system when auto-sizing (default 1536) |
|
||
| `--cache-ceil-mb N` | upper bound on the auto-sized budget (0 = no cap). Use it — uncapped `auto` over-asks |
|
||
| `--dense-weights mmap\|warm\|anon` | the dense (non-expert) weight policy. `warm` is the load-time page-cache sweep described above; `mmap` skips it; `anon` (default) reads the dense set via O_DIRECT into anonymous buffers instead, which is the right answer well past RAM — see [benchmarks-gpt-oss.md](benchmarks-gpt-oss.md). `--no-warm-dense` and `--dense-odirect` are deprecated aliases for `mmap` and `anon` |
|
||
|
||
`auto` is a real LRU cache, so it satisfies the cache requirement of `--prefetch`.
|
||
|
||
## Explicit control
|
||
|
||
Embedders that link the engine can also resize the cache directly with
|
||
`Session::set_cache_budget_mb(int)` — for an app's own memory-pressure callback. It must be called
|
||
between generations (never during a decode); it evicts to the new budget immediately. The Android
|
||
example does not use it: it runs `bmoe-cli` as a subprocess, so its "Auto" cache setting simply
|
||
passes `--cache-mb auto` and the load-time sizing above applies.
|
||
|
||
## Gate
|
||
|
||
**S3** proves a runtime resize is byte-safe: it opens a session with a warm cache, drops the budget
|
||
to force a full eviction, and asserts the next generation still matches the resident reference —
|
||
only residency changes, never the produced bytes.
|