BigMoeOnEdge/docs/cache-sizing.md

Expert cache sizing

The expert cache budget (--cache-mb) is the dominant throughput lever, but the right value is device- and model-specific: too small and the hit rate collapses to exactly zero — see the cliff below, it does not degrade gracefully; too large and the pinned cache plus the mmap-resident model push MemAvailable to zero and the Android low-memory-killer takes the process (this is exactly why Gemma cannot use a 4000 MiB cache on an 11 GB phone — see benchmarks.md). --cache-mb auto removes the guess: the engine sizes the cache to the device once at load and holds that budget for the whole run.

The cliff: below one token cycle the hit rate is 0 %, not "low"

Global LRU drops the least recently used entry. The model visits its layers in a fixed cycle 0..N-1, so the least recently used entry is also the soonest to be needed again. While the budget holds a whole token cycle the cold end is genuinely stale history and this never shows. Once it does not, the cache evicts precisely what it is about to read, and the hit rate does not taper — it goes to exactly 0.0 %.

Measured on device (2026-07-20): gpt-oss-120b at top-4 has a token_demand_MiB of 1815, and a 1500 MiB budget returns 0.0 % hit while reading 1817.05 MiB/token — identical to running with no cache at all (1817.0) — and still spends 0.233 s/token, 24 % of the token, managing it, on top of holding 1500 MiB of RAM. Dropping the budget 2000 → 1500 does not cost a quarter of the hit rate; it costs all of it.

Two traps worth knowing:

• 1500 is a legal value. It is exactly cache_min_mb; no --force-cache is needed. Today's only protection is that this floor happens to sit above the cycle for the shipped models at their default top-k. Raise --n-expert-used and the cycle moves above the floor (gpt-oss: 908 MiB at k=2, 1815 at k=4) and the protection silently stops holding.
• The existing floor is a different quantity. layer_demand_MiB (the widest single layer, 50.4 on that run) is the mechanical minimum; the cliff is at token_demand_MiB (1815). They are roughly n_layer apart. Both are already printed in the # summary line; nothing sizes from them.

So when a budget is being chosen by hand, check it against token_demand_MiB from a previous run of the same model and top-k, not against cache_min_mb. Below the cycle, --cache-mb 0 is strictly better than a cache: same reads, none of the RAM, none of the management cost.

Fixing this by changing the eviction policy was tried and rejected — a per-layer partition is immune to the cliff by construction but costs ~30 % throughput (bench-data/2026-07-20-cache-replay/layer-lfu-verdict.md). The cheap fix was a guard, and it is in: the worst-case cycle is priced at init from the model's shape alone — every bound layer's expert-entry bytes times min(top_k, n_expert), at the top-k the run actually applies. The engine records it in every metrics preamble as cache_cycle_mb next to the budget it is being compared against, and prints one line to stderr at load when the budget is under it:

bmoe: WARNING expert cache 1500 MiB is below this model's worst-case token cycle of 1815 MiB
at top-k 4 — no entry can survive to the next token, so expect a hit rate near zero.

It states the fact and stops there; the advice above — that --cache-mb 0 is strictly better below the cliff — stays here rather than in the engine's output. Note this is the worst case (a disjoint routed set at every layer), so it is an upper bound on the measured token_demand_MiB, and a budget above it is safe for any routing. Recording it matters as much as the warning: a committed CSV whose budget sat under the cycle now says so, without needing a run of the same model to compare against.

What it does

• At init, once the full expert-set size is known, the budget is set to available_RAM − cache_floor_mb, clamped to [cache_min_mb, total expert bytes]. Available memory is read from the platform (/proc/meminfo MemAvailable on Linux/Android, GlobalMemoryStatusEx on Windows); if it is unknown the budget falls back to the cache_min_mb floor.

• Also at init, the dense (non-expert) regions of the gguf — header, embeddings, attention, norms, lm_head, the tensors the streamer leaves mmap-resident — are warmed into the page cache with one sequential buffered sweep (reported as bmoe: dense warm-up), so the first tokens do not pay for them as random 4 KiB faults. On a model far larger than RAM this is the difference between a fast first token and a ~20-token slow-start ramp: measured on gpt-oss-120b, the first-five-token wall average drops ~20× (see benchmarks.md). On models whose dense set is small it is a harmless no-op. This sweep is the --dense-weights warm policy; --dense-weights mmap disables it for A/B runs, and --dense-weights anon (the default) replaces it with an O_DIRECT read into anonymous buffers, which is the better answer once the model is well past RAM — the case the engine targets. The warm-up is deliberately kept out of the budget: it only pre-faults the mmap-resident pages, it does not pin or reserve them, so the expert-cache budget above is unchanged and its hit rate is identical with and without it. (An alternative that folds the dense bytes into the floor — reserving RAM so the expert cache can never evict them — was measured and rejected: on a cache-sensitive model it lowers the budget and the hit rate, e.g. Gemma budget 4000→2909 MiB, hit 83%→73%, trading throughput for OOM headroom that the warm-up already avoids needing. See bench-data/2026-07-14-warmup/.)

• During generation, nothing resizes it. auto is one shot at load, not a control loop: the budget chosen at init is held for the whole run. A runtime governor that tracked free RAM and shrank the budget under pressure did exist and was retired — it was measured a net loss on the models it was built for (see pressure.md). The moe-cache: summary reports the budget and what actually stayed resident:

  moe-cache: 77.1% hit, resident 4000.0 MiB
  

Because expert reads use O_DIRECT they never enter the page cache, shrinking this budget is what actually hands expert RAM back to the rest of the system. It is not the engine's only large allocation, though: under --dense-weights anon (the default) the whole dense set also lives in anon buffers. That footprint is not part of the cache budget and is not subtracted from it — see the ordering warning below.

The budget is not only a throughput knob — it is what the kernel judges you by. On Android the LRU promotes a page to the protected list only on a second reference, and a cache hit is that second reference: a cache with a high hit rate defends itself, one with a low hit rate is correctly read as cold and reclaimed. Measured on gpt-oss-120b, where 3000 MiB covers 5.2% of the expert bank and returns a 13% hit, the cache is taken back while decoding and the fight costs far more than the hits are worth. MemAvailable also over-states the headroom here, since it counts the page cache holding this model's own dense weights as free. Before trusting auto on a model whose expert set dwarfs the budget, read android-memory.md.

auto sizes before the dense weights are allocated. The budget is chosen in ExpertStreamSource::init as soon as the expert-set size is known (core/src/moe/expert_stream_source.cpp:68); the dense policy runs later in the same init (:200), and under the default anon mode it then allocates the entire dense set into anon buffers. So the MemAvailable reading auto sizes from still counts that RAM as free. On a model with a large dense set the over-ask is roughly the dense size — --cache-ceil-mb is the lever that bounds it, which is why the Android example ships a 3000 MiB ceiling by default.

auto sizes from a signal that lies, so keep it modest. auto reads MemAvailable, which reports memory the device will not actually concede (it counts the model's own mmap'd weights as free), so it over-asks — and an over-ask is not a wasted budget but a running fight. The runtime governor that once tried to correct this from the other end (--cache-dynamic) was retired as a net loss (see pressure.md); auto now sizes once at load and stays fixed, so bound it with --cache-ceil-mb on a model whose expert set dwarfs the device, or use cache-off.

Flags

Flag	Meaning
--cache-mb auto	size the cache to the device instead of a fixed MiB (mutually exclusive with a numeric --cache-mb)
--cache-floor-mb N	RAM to leave free for the rest of the system when auto-sizing (default 1536)
--cache-ceil-mb N	upper bound on the auto-sized budget (0 = no cap). Use it — uncapped auto over-asks
--dense-weights mmap|warm|anon	the dense (non-expert) weight policy. warm is the load-time page-cache sweep described above; mmap skips it; anon (default) reads the dense set via O_DIRECT into anonymous buffers instead, which is the right answer well past RAM — see benchmarks-gpt-oss.md. --no-warm-dense and --dense-odirect are deprecated aliases for mmap and anon

auto is a real LRU cache, so it satisfies the cache requirement of --prefetch.

Explicit control

Embedders that link the engine can also resize the cache directly with Session::set_cache_budget_mb(int) — for an app's own memory-pressure callback. It must be called between generations (never during a decode); it evicts to the new budget immediately. The Android example does not use it: it runs bmoe-cli as a subprocess, so its "Auto" cache setting simply passes --cache-mb auto and the load-time sizing above applies.

Gate

S3 proves a runtime resize is byte-safe: it opens a session with a warm cache, drops the budget to force a full eviction, and asserts the next generation still matches the resident reference — only residency changes, never the produced bytes.