BigMoeOnEdge/docs/benchmark-method.md

Benchmark method

This is how to measure. The measured results — the full config matrix for Qwen and Gemma, with mean/min/max/median/p95 tok/s — live in benchmarks.md.

On-device (the headline number)

Hardware for the reference numbers: OnePlus 15R, Snapdragon-class SoC, 11.3 GB RAM, UFS 4.x storage. Model: Qwen3-30B-A3B-Q4_K_M (18.5 GB), ~1.7× device RAM.

Reproducible drivers (used for benchmarks.md): scripts/bench-run.sh (one device-side run, prompt baked in so no quoting has to survive adb, and it also samples the device-pressure axis — peak RSS, MemAvailable floor, CPU/battery temperature), scripts/bench-matrix.ps1 (the full matrix over adb — 8 configs × 2 models, including two --overlap rows), scripts/bench-analyze.py (mean/min/max + median/p5/p95 and the pressure table from the CSVs + .metrics). Use a fixed prompt and a fixed -n (≥256 tokens, so the expert cache reaches steady state) across every config.

Push the model to a real filesystem. The app's external files dir under /storage/emulated is FUSE-backed, where an O_DIRECT open succeeds but reads can return wrong data — the engine detects this and falls back to buffered I/O, so a model staged there does not measure the O_DIRECT path the rest of this method assumes. /sdcard/Download and /data/local/tmp are real filesystems; check the o_direct=1 field in the run's telemetry before trusting a number.

# push the model
adb push Qwen3-30B-A3B-Q4_K_M.gguf /sdcard/Download/

# or, running the CLI directly over adb shell (binary staged by build-android.ps1):
bmoe-cli -m Qwen3-30B-A3B-Q4_K_M.gguf --moe-stream \
  --cache-mb 4000 --io-threads 4 -t 4 -n 256 --progress \
  --csv /sdcard/.../sweep-4000.csv -p "Explain mixture-of-experts routing."

Report s/token and the compute + flash I/O split from the moe-stream: line, and the moe-cache: hit rate.

Sweep

Vary one axis at a time:

Axis	Values	Expectation
cache-mb	0, 2000, 4000, 6000	monotone improvement; 0-or-≥2000 only. On a device whose resident model already fills RAM (Gemma), the top of this range OOMs — watch the MemAvailable floor
io-threads	1, 4	4 ≈ 3× the serial read bandwidth
threads (-t)	2, 4, 8	U-shape, 4 optimal, 8 regresses
overlap	off, on	net gain only over a warm cache (hides residual flash wait behind compute); a net loss on a cold cache-0 stream, where I/O dwarfs compute
n-expert-used	default, 6	fewer active experts cut compute + I/O ~linearly (8→6 ≈ −25%), changes the output
drop-cold-experts	off, 0.75, 1.0	the second lossy axis, and the only non-deterministic one: what is skipped depends on cache state, so cells are noisier and the drop rate must be reported with the tok/s. Needs the cache on
dense-weights	warm, anon	decisive well past RAM, near-neutral near it: on gpt-oss (5.2× RAM) anon drops majflt/token from the hundreds to 6–10 and compute with it; on Qwen (1.64× RAM) there is little dense-fault pressure to remove. Watch majflt/token, not just tok/s

When sweeping --n-expert-used, run it as a matched A/B against the model's own default in the same session (same config, both cells entering from the same measured device state — see Cool on a condition) rather than against an older table — a cool-vs-warm device shifts the baseline enough to swamp the effect, and has been measured inverting it outright. Greedy decode makes the output diverge once routing narrows, so inspect the generated text for quality alongside tok/s; it is a speed/quality trade-off, not a free speedup. See the Turbo top-k section in benchmarks.md for a measured pair.

gpt-oss / harmony models

gpt-oss uses the harmony chat format, whose template always opens an analysis (chain-of-thought) channel before the answer — a plain run spends its whole -n budget reasoning, so a short probe never reaches the answer and per-token timing is measuring analysis tokens. Pass --no-think: on harmony models the engine primes the final channel directly, so the model answers immediately with no analysis tokens. Without it a throughput run is timing chain-of-thought.

Three things to keep honest when reporting gpt-oss numbers:

• --no-think is a speed mode, not a free lunch. Forcing the final channel removes the model's scratch space, so reasoning-dependent tasks degrade — on 17 × 23 the default top-4 answers wrong while k=2/3 answer right (greedy, so it depends only on k). Always inspect the generated text; report speed and correctness separately.
• Use the 256-token protocol, not a short probe. A 24-token run leaves the expert cache warming (hit 10–20 % vs 27–32 % at 256 tokens here) and is dominated by the cold head, so it is a floor rather than a result. The 2026-07-14 sweep in benchmarks-gpt-oss.md was 24-token probes and reads ~3× low against the current steady-state rows for that reason (among others).
• Set --dense-weights anon. This is the single biggest lever on a model this far past RAM, and not setting it silently changes what you are measuring: with the dense weights left in the page cache the kernel reclaims them mid-decode and the refault cost is billed to compute, so the model looks compute-bound when it is actually fault-bound. majflt/token (on the engine's compute: line) tells you which regime you are in — hundreds means the dense set is thrashing, single digits means it is not.

gpt-oss must be read from the real /data partition (e.g. /data/local/tmp/...), not /sdcard — the latter is FUSE and O_DIRECT silently falls back to buffered there.

Cool on a condition, not a timer — and log the entry state

This is the rule that most often decides whether a matrix means anything. A fixed sleep between cells does not return the device to baseline, so throughput tracks execution order and the matrix silently measures the order instead of the config. Measured 2026-07-17 over a 6-cell matrix with a 45 s cooldown (the value this page used to prescribe): the free-RAM floor and the major-fault rate degraded monotonically with position — run 1 entered at a 1.61 GB floor and 149 majflt/token, run 3 at a 0.63 GB floor, 1894 majflt/token and 88.8 °C. It inverted a well-established result (Turbo top-k measured −10.7 % where it reproducibly gives +24 %) purely because the k=6 cell ran second. The cause is reclaim hysteresis: the kernel takes memory away in seconds and returns it over minutes (see android-memory.md).

So:

• Gate each cell on a measured condition — CPU temperature and MemAvailable back under a threshold — with a bounded give-up that is recorded when it fires. bench-data/2026-07-17/driver-lanes.sh is a working example.
• Read the CPU sensor, not the battery. Battery temperature lags behind the SoC by minutes and will rank two cells backwards: in the 2026-07-17 lane pair, battery said cell A entered 2.7 °C cooler while the CPU said it entered 8.5 °C hotter and peaked at 93.8 °C. The CPU sensor is the one that governs compute throttling.
• Log the entry state next to every number (scaling_max_freq, CPU temp, MemAvailable), so a contaminated cell is visible in the data instead of being discovered later — or worse, published.
• Prefer a cold, idle, unplugged-then-reattached device. On USB power this phone idles around 38 °C and will not reach a 36 °C gate at all.
• On a >>RAM model (gpt-oss at 5.2× RAM) budget minutes, not seconds: a 58 GB run leaves the device hot and its free RAM depressed long after the process exits.

Two tells that a cell is contaminated rather than informative:

1. compute s/token rises as top-k falls. Physically impossible — fewer active experts cannot make the same kernels slower. It is fault-service time landing inside the compute bucket.
2. majflt/token jumps an order of magnitude between cells that should do comparable work.

Re-run such a cell; do not publish it. And sanity-check any matrix by reversing the run order — cells that move were measuring device state.

The reversal check does not work under --drop-cold-experts. There a cell can move because the drop rate moved — the policy reads live cache state, so the same command legitimately discards a different number of experts on a different run. That is the feature working, not the device contaminating the cell, and the two tells above cannot tell them apart. Always record experts_dropped/experts_routed (or the moe-drop: line) next to the tok/s: a dropping cell without its drop rate is uninterpretable, because the flag fixes a threshold and not a rate. Note also that a dropping run pays the same extra per-MoE-layer barriers a route-traced run does, so an A/B against --n-expert-used is not overhead-matched — see expert-dropping.md.

Caveats

• Thermal. Sustained decode throttles. Warm up, then measure a steady window; discard the first few tokens. Ignore cpu-hw-trip-* sensors — those are static 95 °C trip points, not live temperatures.
• Report the distribution, not just the mean. min/max tok/s are single-token extremes (a lone eviction stall crushes min); pair them with median and p5/p95 so an unstable config (wide spread) is distinguishable from a slow-but-steady one.
• Expect 0.27–0.6 s/token across the good part of the sweep (4000 MiB cache, 4 lanes, 256-token steady state); shorter runs read slower because the cache is still warming.

Device pressure (throughput is only half the story)

tok/s does not capture what a config does to the rest of the phone. mmap-only faults the whole model through the page cache and evicts other apps, so the device goes sluggish; streaming with a bounded cache + O_DIRECT bypasses the page cache and keeps the system responsive. Record a pressure indicator next to tok/s. Accessible over adb without root:

• Temperature — /sys/class/thermal/thermal_zone*/temp with matching .../type (CPU cpu-*, GPU gpuss-*, skin zones are readable by the shell user). dumpsys battery (temperature, deci-°C) is easy to read but lags the SoC by minutes — record it if you like, decide with the CPU zone.
• Free-RAM floor — /proc/meminfo MemAvailable, sampled before / mid-run / after; its collapse under mmap is the pressure signal.
• Major faults — majflt/token on the engine's own compute: line. This is the one that distinguishes "the kernels are slow" from "the dense weights are refaulting"; nothing else separates those two, because the kernel bills the refault to the compute call.
• Throttling state — dumpsys thermalservice, plus scaling_max_freq (see Caveats).

Kernel PSI (/proc/pressure/{memory,io,cpu}) is the cleanest stall metric but returns Permission denied without root on this device. Protocol: bring every config to a common baseline by measurement (see Cool on a condition) so sustained-decode throttling doesn't confound the comparison, then produce a tok/s vs. thermal rise and free-RAM floor table alongside the throughput one.

Host (correctness + a sanity number)

• Gates (mandatory before release): cd build && ctest --output-on-failure. These prove streamed == resident on the tiny synthetic model.
• Real small MoE (release checklist): run Qwen1.5-MoE-A2.7B-Q4_K_M streamed vs resident on the dev host and confirm identical output. It is too large for CI.

Desktop over-RAM run (measured)

Streaming also delivers on desktop when the model exceeds the machine's RAM. Qwen3-30B-A3B-Q4_K_M (17.3 GiB, 128 experts, 48 layers) on a Windows PC with 14.8 GiB RAM — 1.17× RAM, so it cannot be held resident — cache 4000 MiB, 4 I/O lanes, 4 threads:

generation: 16 tokens, 0.388 s/token (2.58 tok/s)
moe-stream: read 13774 MiB (861 MiB/token), 1.28 GiB/s O_DIRECT
moe-cache: 44.8% hit, resident 3997 MiB

Output is coherent; running the same model resident is impossible on this machine (OOM). On a desktop where the model does fit in RAM, run it resident — streaming only pays off above the RAM ceiling.

The host streamer works on Linux/macOS/Windows, but throughput targets apply to Android/Linux on UFS storage; Windows VirtualAlloc commit-per-slice is heavier (see limitations.md).