BigMoeOnEdge/docs/expert-dropping.md

Cache-aware expert dropping

--drop-cold-experts F skips a routed expert when it is not in the cache and the router weighted it below F × (1 / top-k) — that is, below F of the uniform share each of the k selected experts would get if the router split its mass evenly. Off by default.

It is the second lossy knob in the engine, after turbo top-k, and it exists because the first one spends quality in a place it does not have to.

Why cache state belongs in the decision

--n-expert-used k drops the routing's tail unconditionally: slot 7 and slot 8 go, whether or not they were already sitting in RAM. But an expert that is already resident costs no flash read — and on a streamed decode, flash reads are what the token is waiting for (decode is I/O-bound). Dropping a resident expert pays quality for nothing.

Turn that around and the policy writes itself: spend quality only where it buys I/O. Keep every resident expert however small its weight; consider dropping only the ones that would cost a read, and only when the router says they barely matter.

What it costs and what it buys

Replayed over the committed route traces (docs/bench-data/2026-07-15-route-trace/), decode phase, threshold at the uniform share (F = 1.0):

policy	flash reads avoided	router weight discarded
--drop-cold-experts 1.0	66%	9.5%
--n-expert-used 5	23%	10.6%
--n-expert-used 3	59%	36.8%

(Qwen3-30B-A3B at k=6; Gemma-4-26B-A4B is within a point and a half on both columns: 67.4% / 8.2%. On gpt-oss-120b at k=2 the policy matches --n-expert-used 1's read saving while discarding 25% of the weight mass instead of 42%.)

At a comparable quality cost the cache-aware policy avoids roughly three times the reads. The reason is visible in the third column of the trace: about 80% of decode routings are cache hits, and the policy leaves every one of them alone.

F is a curve, not a switch. At F = 0.75 the same model trades 4.4% of the weight mass for 37% of the reads — still better than --n-expert-used 5 on both axes.

These are replay numbers, and they were an upper bound only where dropping is light — see the measurement below, where at full strength reality beat the prediction.

Measured on device

Qwen3.6-35B-A3B (top-k 8 of 256), in-app, cache 3000 MiB, one variable changed:

F	tok/s	flash read	routings dropped
off	2.549	248 GiB	0
0.50	2.564	231 GiB	2.7%
0.75	3.938	163 GiB	14.2%
1.00	4.702	48 GiB	28.4%

Per-token bootstrap intervals separate every pair except off vs 0.50, which overlaps: at half the uniform share the policy finds almost nothing to drop and buys nothing. That is also the useful negative control — the machinery costs nothing measurable when it is not firing.

Run order was 1.00, off, 0.50, 0.75, so the two fastest cells are the first and the last: thermal drift would have made the last the worst. Full data, caveats and the run-order argument: bench-data/2026-07-22-drop-cold-experts.

Quality checked, no loss detected. 15 GSM8K questions through the same configuration: 12/15 with dropping off, 13/15 at every threshold including 1.0, where 28% of routings are discarded. Twelve of the fifteen questions give an identical final answer in all four cells; the variation sits on two questions and flips in both directions rather than worsening with the threshold. Reply length is flat, so the model neither rambles nor truncates. Details, per-question replies and the grading rule: bench-data/2026-07-22-drop-quality.

Decoding is greedy, so there is no sampling noise: every difference between cells is caused by the policy, which is what makes twelve identical answers a real statement. But 13 against 12 is not an improvement — perturbing a problem the model already got wrong can land either side of the right answer, and at 6.7 points per question this sample cannot establish the sign of the effect. It rules out a collapse, not a subtle cost.

Two properties worth knowing

A routing is never emptied. The largest weight in a routing is always at least the uniform share, so at F ≤ 1.0 the top expert can never fall below the threshold. validate() rejects F > 1.0 for that reason, and the implementation additionally pins the top-weighted expert, so the guarantee does not rest on the bound alone.

It requires the expert cache. With --cache-mb 0 every expert reads as a miss, so the policy would stop being cache-aware and become an unconditional weight cut — which is what --n-expert-used already does, without claiming to consult residency. validate() rejects the combination, the same way it rejects --prefetch without a cache.

It changes what --prefetch means. Speculation is normally output-neutral by construction. Here residency is an input to the policy, so a correct guess un-drops an expert that would otherwise have been discarded: prefetch depth becomes an output-affecting setting. The decision point also settles pending speculation a few nodes after it was issued, which shortens the overlap window the prefetch exists for — treat the two as interacting, not composable.

Prefill is excluded by default. With a cold cache almost every expert is a miss, and the same threshold discards ~42% of the weight mass instead of ~9%. Prefill is compute-bound anyway, so there is little to win. --drop-in-prefill arms it for experiments.

Narrow routings are a different regime, and the engine says so

The threshold is a fraction of the uniform share 1/top-k, so what it removes depends on how wide the routing is. At top-k 8 — where all the evidence here was collected — F = 0.75 means "below 9.4% of the routing", a tail trim. At top-k 4 the same F means "below 18.8%", and at top-k 2 it means "below 37.5%", which on a miss discards the entire minority expert. That is closer to halving the routing than to trimming it, and nothing in this document measured it.

So the engine warns once at load when dropping is armed on a routing of four experts or fewer:

bmoe: WARNING drop-cold-experts=0.75 with top-k 2 — the threshold is 37.5% of the routing here,
against 12.5% at the top-k 8 this was measured on. Expect it to discard much more, and check
output quality on your own task.

It warns rather than clamping or refusing. The engine cannot know whether that trade is acceptable for a given model and task, and silently adjusting a number the caller chose would be worse than a loud caveat. MoeStreamConfig::drop_low_topk_warn is an evidence boundary, not a physical one: nothing in the streaming path reads it, it only decides whether the caller is told.

The app shows the same warning inline under the setting, computed from the width the loaded model reports at BMOE_READY — and only once a model is loaded, since guessing would be worse than staying quiet. gpt-oss is the case to watch: it routes 4 of 128.

The output is no longer reproducible

This is the real novelty, and the reason the flag is off by default and named the way it is.

--n-expert-used is lossy but deterministic: same prompt, same config, same tokens. Dropping is lossy and state-dependent — what gets discarded depends on what the cache happened to hold, which depends on everything decoded before it. The same prompt can produce different text across runs, and a benchmark cell is noisier because the drop rate itself varies.

The greedy byte-identity gates therefore do not cover the policy's output, and cannot: there is nothing stable to compare against. They cover the machinery instead (see below).

How it is implemented

The decision needs the final router weights, and those are produced several graph nodes after the topk node where the streamer normally loads. So with the policy armed, load_layer() is postponed from the topk node to the terminal node of the layer's weight chain — the last node before the expert matmul consumes either the ids or the weights.

Which node is terminal depends on the model's gating (_norm, _softmax, _scaled, or none), so the hook learns it from the graph instead of carrying a per-architecture table: the first graph of a run records the chain, and dropping starts from the second. A layer whose shape has not been seen yet simply loads at its topk node, undropped. That costs a run its first token's dropping and nothing else, and it keeps hard rule 4 — no model-specific constants in the streaming path.

At the decision point two edits happen, both before anything reads them:

1. the dropped slot's weight is zeroed, and with drop_renorm (default on) the survivors are scaled so the routing keeps its original total mass;
2. the dropped slot's id is repointed at the routing's top-weighted expert.

The second edit is not cosmetic. An expert the engine declines to read may sit in a reserved-but-uncommitted slot, and mul_mat_id would still touch it. Pointing the slot at an expert that is certainly resident makes the kernel read valid memory and multiply it by exactly zero. It costs a duplicate matmul — the right trade on a decode bound by flash rather than arithmetic.

Renormalisation matters more than it looks: without it the layer's expert output is systematically scaled down by the discarded mass, a perturbation of the residual stream the model never sees in training. --drop-no-renorm exists to A/B that claim.

Cost of the extra barriers: the policy asks for each layer's weight nodes, a handful more synchronisation points per MoE layer on tensors of a few floats. The same asks a route trace makes.

Measuring it

The engine reports what the policy actually did, which the flag alone cannot tell you — the threshold is fixed, the drop rate is not:

moe-drop: <dropped>/<routed> routed experts dropped (<pct>%), threshold <F> x uniform

The route trace gains a dropped column: weight and residency stay as the router produced them, dropped records what the policy then did, and expert_bytes is 0 for a dropped routing because it costs no read. That is enough to replay a real run against the offline model and check whether the upper bound held. See telemetry.md.

Gates

bmoe_moe_gates covers the machinery, not the policy's output:

• G8a — with a threshold below any weight the router can produce, nothing is dropped and the output is byte-identical to the undropped stream. This proves the deferral and the learned terminal node are transparent, separating "the plumbing is correct" from "the policy is lossy" — a regression in the first would otherwise hide behind the expected difference. G8a' asserts the count separately (experts_routed > 0, experts_dropped == 0), so "a weight happened to fall under the threshold" fails legibly instead of as a mysterious byte mismatch.
• G8b — at full strength against a cache small enough to be evicting constantly, so dropped experts really do land on slots the cache has released. Generation still completes: the id repointing means no matmul ever reads reserved-but-uncommitted memory. (The gates deliberately do not run this with the cache off — there the shared-slot path has no uncommitted memory, so the safety property the repointing exists for would go untested.)
• G8c — forcing top-k to 1 makes every routed expert the top one, so dropping must be a no-op at any threshold and the output must match the undropped k=1 run byte for byte. This pins both the top-expert guarantee and the fact that the threshold is taken against the effective top-k discovered at runtime — a hardcoded width would not survive the override.

Defaults, and where the numbers do and do not come from

The CLI defaults it off, and will keep doing so: the byte-identity gates need a deterministic default, and an instrument should not quietly change the thing it measures.

The app ships it at 75%, under Speed / quality → Drop cold experts, with rungs 50 / 75 / 100 as percentages of the uniform share. It is disabled there in mmap mode and with the cache off, the same two conditions validate() enforces.

75% rather than 100% is the deliberate choice: it is where the measured curve turns, taking the larger part of the throughput win (+55%) for half the discarded routings (14% against 28%). With quality unquantified, the conservative end of a measured range is the defensible default.

What is still owed before this is recommended beyond that:

• a published decode A/B against --n-expert-used at matched tok/s, with the device state recorded the way benchmark-method.md requires;
• a quality comparison at that matched speed — the whole thesis is that this knob buys the same throughput for less damage, and only a side-by-side can support it;
• a re-run of the replay against a real traced run with the dropped column, to see how far the static upper bound overstated the win.

The layer-lfu entry in the roadmap is the standing reminder for why the third one matters: it simulated exactly as predicted and was ~30% slower in reality.