vrr/BigMoeOnEdge
2
0
Fork 0
mirror of https://github.com/Helldez/BigMoeOnEdge.git synced 2026-10-03 03:25:42 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

docs: split the benchmarks, and say what the engine grew

Browse source 
benchmarks.md was two documents in one file: 460 lines, two H1s, and four
section names appearing twice. roadmap.md's link to #reading-the-numbers
resolved to whichever came first, which happened to be the intended one — a
coincidence, not a design. Split the gpt-oss-120b half into its own doc and
cross-link the two.

That also fixes the anchor at the old benchmarks.md:30: GitHub slugs an em-dash
heading to a double hyphen, so #device-pressure-not-just-tokens never jumped
anywhere. The README already had this right for its own gpt-oss link, so the
convention was there — this one was just wrong.

README: route traces and the app's Markdown answers have been in main for
several commits with no mention, and a feature nobody can find is a feature
nobody has. Trimmed the O_DIRECT and cache bullets in exchange: both re-taught
mechanism the linked docs already own, which is what a landing page delegates.
This commit is contained in:
Helldez 2026-07-15 20:32:30 +02:00
parent f9e408f542
commit bb318cbd2e
4 changed files with 148 additions and 144 deletions
Download patch file Download diff file Expand all files Collapse all files

27
README.md
View file

@ -40,16 +40,13 @@ much smaller memory footprint — is worth making.
flash. Loads `use_mmap=true`, repack off, and rebinds each expert tensor onto a streaming
buffer in the native gguf layout. Fails fast if the model is not MoE.
- **LRU expert cache with an auto budget and ceiling** (`--cache-mb N|auto`, `--cache-ceil-mb`) —
a fixed MiB budget or one sized to the device (free RAM minus a floor), clamped to
`[1.5 GiB, full expert-set size]` and re-checked during generation so it shrinks and grows with
available memory. Cache size is the single biggest throughput lever.
a fixed MiB budget, or one sized to the device and re-checked during generation so it tracks
available memory. The single biggest throughput lever; see
[docs/adaptive-cache.md](docs/adaptive-cache.md).
- **Direct-from-flash reads, O_DIRECT** (`--io-threads 1..8`, `--no-odirect`) — each expert slice
is read straight from flash into the engine's own buffer, skipping the operating system's page
cache. That cache normally keeps a second copy of everything you read in spare RAM; here it would
only duplicate weights the engine is already caching itself, waste memory, and evict the user's
other apps. Reading direct keeps memory bounded and read latency predictable. Several read lanes
run in parallel (4 is the UFS 4.x sweet spot), and the engine falls back to normal buffered reads
on the odd filesystem that mishandles O_DIRECT.
is read straight into the engine's own buffer, bypassing the page cache that would otherwise hold
a second copy of weights the engine already caches. Several read lanes run in parallel (4 is the
UFS 4.x sweet spot), with a buffered fallback where O_DIRECT misbehaves.
- **Intra-layer I/O–compute overlap** (`--overlap`) — pipelines each layer's async expert reads
with its FFN compute, hiding flash latency behind the matmul; byte-identical to the serial path.
Top throughput lever over a warm cache. Requires the fork submodule.
@ -62,11 +59,15 @@ much smaller memory footprint — is worth making.
- **Honest, per-token telemetry** — `--progress`/`--csv` emit a per-token breakdown: compute vs
cache-management vs flash-I/O vs stall seconds, cache hit rate, flash bytes read, cache
residency and resizes. The Android panel renders it live.
- **Routing traces** (`--route-trace PATH`) — records which experts every token actually routed to,
per layer, for offline analysis (`scripts/route-analyze.py`, `scripts/route-viewer.py`). A
diagnostic: it perturbs the run, so its tok/s are not comparable with the benchmark tables.
- **Experimental, default-off**: temporal prefetch (`--prefetch K`, a cold-start/TTFT tool) reads
the next layers' likely experts on idle I/O lanes. An honest toggle kept for provability; it does
not help steady-state throughput on current hardware — see [Benchmarks](#benchmarks).
- **Android demo APK** ([`examples/android`](examples/android)) — a multi-turn chat app with a live
telemetry panel and every streaming knob exposed with a one-line note on what it does.
- **Android demo APK** ([`examples/android`](examples/android)) — a multi-turn chat app with
Markdown-rendered answers, a live telemetry panel, and every streaming knob exposed with a
one-line note on what it does.
## Supported models and architectures
@ -176,7 +177,7 @@ gpt-oss is heavily **compute-bound** (each expert is large), so **top-k is the d
is ~3× faster than the default k=4 — and prefetch only hurts. These are exploratory **24-token** probes
(cache still warming, 13–21% hit), not the 256-token steady state above; treat them as a floor. Full
matrix, the k=4 interruption caveat, and a **quality** note (`--no-think` drops gpt-oss's reasoning, so
default k=4 answers `17×23` *wrong* while k=2/3 get it right): [docs/benchmarks.md](docs/benchmarks.md#gpt-oss-120b--a-58-gb-model-at-52-device-ram).
default k=4 answers `17×23` *wrong* while k=2/3 get it right): [docs/benchmarks-gpt-oss.md](docs/benchmarks-gpt-oss.md).
### Desktop is not the target (for now)
@ -247,6 +248,8 @@ it, or reproduce the measurements. The entry points most people want:
- [docs/seam.md](docs/seam.md) — the exact contract with llama.cpp's public API.
- [docs/adding-a-model.md](docs/adding-a-model.md) — supporting a new MoE architecture.
- [docs/telemetry.md](docs/telemetry.md) — the `BMOE_*` line protocol and CSV schema.
- [docs/android-memory.md](docs/android-memory.md) — what reclaims the engine's memory on a phone,
and which levers actually exist.
- [docs/benchmarks.md](docs/benchmarks.md) — measured results, and [how they were
produced](docs/benchmark-method.md).

3
docs/README.md
View file

@ -28,7 +28,8 @@ for the idea the project is built on.
| Doc | What it answers |
|---|---|
| [benchmarks.md](benchmarks.md) | Measured results per model, with device-pressure numbers. |
| [benchmarks.md](benchmarks.md) | Measured results per model on Android, with device-pressure numbers. |
| [benchmarks-gpt-oss.md](benchmarks-gpt-oss.md) | gpt-oss-120b: a 58 GB model at 5.2× device RAM, and what it costs. |
| [benchmark-method.md](benchmark-method.md) | How the numbers are produced, so you can reproduce them. |
| [warmup-analysis.md](warmup-analysis.md) | Why first tokens are slow, and the two regimes behind it. |
| [bench-data/](bench-data/) | Raw per-run CSVs and session notes. A dated archive — see its README. |

127
docs/benchmarks-gpt-oss.md Normal file
View file

@ -0,0 +1,127 @@
# gpt-oss-120b — a 58 GB model at 5.2× device RAM
The same engine streams **OpenAI gpt-oss-120b** — a 58.46 GB MoE — on the same 11.3 GB phone.
That is **5.2× device RAM**: the model cannot be held resident by any means, and to our knowledge
this is the first time a 120B / 58 GB model has generated tokens on a phone at all. The run below
is an exploratory sweep (top-k × read-lanes × prefetch), not the polished 256-token matrix used for
Qwen/Gemma in [benchmarks.md](benchmarks.md) — read the two caveats before the numbers.
## Environment
| | |
|---|---|
| Device | OnePlus 15R (`CPH2769`), Android 16, 11.3 GB RAM — same as [benchmarks.md](benchmarks.md) |
| Model | `gpt-oss-120b-Q4_K_M.gguf` — 58.46 GB, 36 layers, 128 experts, **top-4** default, MXFP4 expert weights |
| Device path | `/data/local/tmp/shardllm/` — the real `/data` partition, **required** for working O_DIRECT (`/sdcard` is FUSE and silently falls back to buffered) |
| Fraction of RAM | ≈**5.2×** (58.46 GB / 11.3 GB) — resident load is impossible, so there is no in-RAM baseline, only `mmap` page-cache thrash |
| Engine | `bmoe-cli` built from `feat/harmony-nothink-final-channel` @ `4d12b75` (arm64, NDK r26, `armv8.2-a+dotprod+fp16`) |
| Fixed config | `--cache-mb auto --cache-ceil-mb 3000` (auto-sized, capped 3000 MiB), O_DIRECT on, `--overlap` on, `-t 4`, `--no-think` |
| Swept | top-k ∈ {2, 3, 4}, read-lanes ∈ {4, 8}, prefetch ∈ {off, 4} — 12 cells, plus a 2-cell `mmap` baseline |
| Probe | `-n 24`, prompt *"What is 17 times 23? Then name the capital of Australia."* (a short, checkable probe — see Quality) |
`--no-think` matters here. gpt-oss uses the harmony format, whose template **always** opens an
`analysis` (chain-of-thought) channel — so a normal run spends its whole budget reasoning before it
answers. `--no-think` now primes the `final` channel directly (see the engine fix on this branch), so
the model answers immediately with no analysis tokens. That is what makes a 24-token probe meaningful
— but it also removes the model's scratch space, which the Quality section below shows has a cost.
## Two caveats (both narrow the numbers, honestly)
1. **Short probe, not steady state.** These are **24-token** runs, not the 256-token runs used for
Qwen/Gemma. The expert cache is still warming — hit rate sits at **13–21 %** (vs 76 % for Qwen at
256 tokens), so flash-read/token is high and the absolute tok/s is a **floor**: a warm, longer run
would read less and decode faster.
2. **The k=4 rows were interrupted.** The phone was physically unplugged several times during the
k=4 cells; model-load and TTFT balloon there (k4 · io8 · pf0: load 90 s, TTFT 120 s). That row's
decode is **not trustworthy** — its compute drops to 2.042 s/tok against 3.869 s/tok for the *same*
k=4 at 4 lanes, but compute is lane-independent, so the gap is device state (cooler / less contended
after the interruption), not a lane effect. It is marked † and excluded from every conclusion. Read
k=4 from the **io4 · pf0** row (4.489 s/tok).
## Results
s/token and tok/s are the engine's `generation:` line; `compute` and `flash read/token` are from its
`moe-stream:` line; `cache hit` from `moe-cache:`. `--overlap` is on, so `flash I/O` runs concurrently
with compute and the **stall** (residual flash wait not hidden behind compute) is the honest I/O cost —
it stays ~0.2–0.3 s/tok throughout, i.e. overlap hides almost all of the flash read.
| top-k | lanes | prefetch | tok/s | s/token | compute (s/tok) | flash read/token | cache hit |
|---:|---:|---:|---:|---:|---:|---:|---:|
| **2** | **4** | **off** | **0.687** | **1.455** | 1.159 | 535.75 MiB | 20.4 % |
| 2 | 4 | 4 | 0.532 | 1.878 | 1.529 | 640.91 MiB | 21.2 % |
| 2 | 8 | off | 0.620 | 1.613 | 1.310 | 535.75 MiB | 20.4 % |
| 2 | 8 | 4 | 0.516 | 1.937 | 1.647 | 640.91 MiB | 21.2 % |
| **3** | **4** | **off** | **0.391** | **2.556** | 2.118 | 925.74 MiB | 16.9 % |
| 3 | 4 | 4 | 0.304 | 3.293 | 2.854 | 1100.68 MiB | 18.1 % |
| 3 | 8 | off | 0.279 | 3.581 | 3.074 | 925.74 MiB | 16.9 % |
| 3 | 8 | 4 | 0.295 | 3.394 | 2.924 | 1100.68 MiB | 18.1 % |
| **4** | **4** | **off** | **0.223** | **4.489** | 3.869 | 1402.75 MiB | 13.4 % |
| 4 | 4 | 4 | 0.188 | 5.327 | 4.701 | 1619.72 MiB | 14.7 % |
| 4 | 8 | off † | *0.383* | *2.613* | *2.042* | 1402.75 MiB | 13.4 % |
| 4 | 8 | 4 | 0.213 | 4.704 | 4.045 | 1623.57 MiB | 14.7 % |
| mmap | — | — | 0.089 | 11.240 | — | 0 (page cache) | — |
| mmap (k=4) | — | — | 0.075 | 13.337 | — | 0 (page cache) | — |
† Interrupted run — see caveat 2. Excluded from conclusions.
## Reading the numbers
- **Streaming vs `mmap`: 3–8×.** k=2 streams at 1.455 s/tok against **11.240 s/tok** for a plain
`mmap` load of the same file — **7.7× faster**; k=4 is 4.489 vs 13.337 — **3.0×**. `mmap`-ing 58 GB
onto 11 GB of RAM thrashes the page cache on *every* token (10–13 s each); the bounded 3 GB O_DIRECT
cache replaces that with reads the engine controls, and leaves the rest of RAM for the system.
- **top-k is the dominant lever — it cuts compute *and* I/O.** Both scale almost linearly with k:
compute 1.16 → 2.12 → 3.87 s/tok and flash-read 536 → 926 → 1403 MiB/tok across k = 2 → 3 → 4 (4
lanes). k=2 is ~**3× faster** than k=4. This is the same knob as Qwen/Gemma's Turbo top-k, but it
matters far more here because gpt-oss is heavily compute-bound.
- **Compute-bound, hard.** Even at these low hit rates the *compute* share dominates at k ≥ 3 (k=4:
3.87 s of the 4.49 s decode), because each gpt-oss expert is large (d_ff 2880 — several × a Qwen
expert), so top-4 is a lot of MAC per token. Overlap already hides almost all flash wait (stall
~0.2–0.3 s/tok), so the remaining cost is kernels, not the seam — exactly as on Qwen at a warm cache.
- **prefetch=4 always regresses.** Every `pf 4` row is slower than its `pf off` sibling. Prefetch
reads 20–25 % *more* per token speculatively (k=2: 640.91 vs 535.75 MiB/tok) but only **12–15 %** of
those experts are ever used — on a compute-bound model that wasted flash bandwidth buys nothing and
costs cache churn. Leave prefetch off for gpt-oss.
- **Lanes 4 vs 8: 4 wins where it's trustworthy.** At k=2 io4 beats io8 (1.455 vs 1.613) — with the
flash wait already overlapped, extra lanes only add contention. The k=4 lane comparison is confounded
(caveat 2), so no lane claim is made there.
- **A 24-token probe is mostly warm-up.** Unlike Qwen/Gemma, gpt-oss at 5.2× RAM warms up *inside*
compute: the first tokens fault the mmap-resident, non-expert working set in from flash (`compute_ms`
~18 s), settling to sub-second once hot. The mean over 24 tokens is therefore a floor dominated by that
cold head, and the steady tail is several × faster. This memory-residency warm-up — distinct from the
gentle, I/O-bound cache warm-up on Qwen/Gemma — is analysed token-by-token in
[warmup-analysis.md](warmup-analysis.md).
## Quality — the cost of dropping reasoning
Because these runs use `--no-think` (forced `final` channel, **no** chain-of-thought), the model
answers with no scratch work — and decode is greedy/deterministic, so the answer depends only on k
(identical under streaming and `mmap`):
| top-k | `17 × 23 =` | capital |
|---:|---|---|
| 2 | **391** ✅ | Canberra ✅ |
| 3 | **391** ✅ | Canberra ✅ |
| 4 (default) | **387** ❌ | Canberra ✅ |
The model's *default* top-4 gets the arithmetic **wrong** (387) while the narrower k=2/k=3 get it
**right** (391). This is not "smaller k is smarter" — it is that **without the analysis channel there
is no scratch space to compute 17 × 23**, so the answer is a one-shot guess whose correctness is
prompt- and k-specific. Takeaway: `--no-think` (forced-final) is a **latency/throughput mode** — use
it for direct-answer UX and for benchmarking decode speed; for arithmetic or any multi-step task, drop
`--no-think` and let gpt-oss spend analysis tokens. The capital is correct at every k.
## Provenance
Measured 2026-07-14 with `bmoe-cli` @ `4d12b75` (branch `feat/harmony-nothink-final-channel`). The
summary log for all 14 cells and the two `mmap` per-token CSVs are committed under
[`bench-data/2026-07-14/`](bench-data/2026-07-14/); the streaming cells were run without `--csv` (this
sweep captured only the `moe-stream:` / `moe-cache:` summary lines), so there are no per-token CSVs for
them — a follow-up warm 256-token run with `--csv` and a compute-thread sweep (`-t 4/6/8`) is the next
step. Drivers: [`scripts/gptoss-matrix.sh`](../scripts/gptoss-matrix.sh) (the 12-cell streaming sweep)
and [`scripts/gptoss-mmap.sh`](../scripts/gptoss-mmap.sh) (the 2-cell mmap baseline), prompt and flags
baked in. The model was merged from its two HF shards with `llama-gguf-split --merge` before use.
| Model | Device path | Size |
|---|---|---|
| gpt-oss-120b | `/data/local/tmp/shardllm/gpt-oss-120b-Q4_K_M.gguf` | 58.46 GB |

135
docs/benchmarks.md
View file

@ -5,6 +5,9 @@ the model, across the full configuration matrix, for two MoE families. These are
numbers the README table and the headline claim are drawn from; the how-to lives in
[benchmark-method.md](benchmark-method.md).
For the far end of the ratio — a 58 GB model at 5.2× device RAM — see
[benchmarks-gpt-oss.md](benchmarks-gpt-oss.md).
## TL;DR
- Streaming a >RAM MoE model is **stable and usable**: ~1.6–1.7 tok/s with no cache, and up
@ -27,7 +30,7 @@ numbers the README table and the headline claim are drawn from; the how-to lives
*aggregate* throughput collapses (Gemma **0.36** tok/s) because a handful of
page-cache-eviction stalls (single tokens as slow as 8 s) dominate the total — and during
those stalls the phone is effectively unusable for anything else (see
[Device pressure](#device-pressure-not-just-tokens)).
[Device pressure](#device-pressure--not-just-tokens)).
- **With a warm cache, decode is compute-bound, not I/O-bound.** The engine reports the
decode split as `compute + flash I/O`. At Qwen's cache 4000 the serial flash I/O share is
~0.13 s/token against ~0.16 s of compute; with overlap the flash wait (stall) falls to
@ -328,133 +331,3 @@ Model files (both Q4_K_M GGUF, staged on the device at `/sdcard/Download/`):
Sizes are the on-disk GGUF byte counts; both are stock Q4_K_M conversions, unmodified by the
engine (it loads `use_mmap=true`, rebinds expert tensors to the native gguf layout, no repack).
---
# gpt-oss-120b — a 58 GB model at 5.2× device RAM
The same engine streams **OpenAI gpt-oss-120b** — a 58.46 GB MoE — on the same 11.3 GB phone.
That is **5.2× device RAM**: the model cannot be held resident by any means, and to our knowledge
this is the first time a 120B / 58 GB model has generated tokens on a phone at all. The run below
is an exploratory sweep (top-k × read-lanes × prefetch), not the polished 256-token matrix used for
Qwen/Gemma above — read the two caveats before the numbers.
## Environment
| | |
|---|---|
| Device | OnePlus 15R (`CPH2769`), Android 16, 11.3 GB RAM — same as above |
| Model | `gpt-oss-120b-Q4_K_M.gguf` — 58.46 GB, 36 layers, 128 experts, **top-4** default, MXFP4 expert weights |
| Device path | `/data/local/tmp/shardllm/` — the real `/data` partition, **required** for working O_DIRECT (`/sdcard` is FUSE and silently falls back to buffered) |
| Fraction of RAM | ≈**5.2×** (58.46 GB / 11.3 GB) — resident load is impossible, so there is no in-RAM baseline, only `mmap` page-cache thrash |
| Engine | `bmoe-cli` built from `feat/harmony-nothink-final-channel` @ `4d12b75` (arm64, NDK r26, `armv8.2-a+dotprod+fp16`) |
| Fixed config | `--cache-mb auto --cache-ceil-mb 3000` (auto-sized, capped 3000 MiB), O_DIRECT on, `--overlap` on, `-t 4`, `--no-think` |
| Swept | top-k ∈ {2, 3, 4}, read-lanes ∈ {4, 8}, prefetch ∈ {off, 4} — 12 cells, plus a 2-cell `mmap` baseline |
| Probe | `-n 24`, prompt *"What is 17 times 23? Then name the capital of Australia."* (a short, checkable probe — see Quality) |
`--no-think` matters here. gpt-oss uses the harmony format, whose template **always** opens an
`analysis` (chain-of-thought) channel — so a normal run spends its whole budget reasoning before it
answers. `--no-think` now primes the `final` channel directly (see the engine fix on this branch), so
the model answers immediately with no analysis tokens. That is what makes a 24-token probe meaningful
— but it also removes the model's scratch space, which the Quality section below shows has a cost.
## Two caveats (both narrow the numbers, honestly)
1. **Short probe, not steady state.** These are **24-token** runs, not the 256-token runs used for
Qwen/Gemma. The expert cache is still warming — hit rate sits at **13–21 %** (vs 76 % for Qwen at
256 tokens), so flash-read/token is high and the absolute tok/s is a **floor**: a warm, longer run
would read less and decode faster.
2. **The k=4 rows were interrupted.** The phone was physically unplugged several times during the
k=4 cells; model-load and TTFT balloon there (k4 · io8 · pf0: load 90 s, TTFT 120 s). That row's
decode is **not trustworthy** — its compute drops to 2.042 s/tok against 3.869 s/tok for the *same*
k=4 at 4 lanes, but compute is lane-independent, so the gap is device state (cooler / less contended
after the interruption), not a lane effect. It is marked † and excluded from every conclusion. Read
k=4 from the **io4 · pf0** row (4.489 s/tok).
## Results
s/token and tok/s are the engine's `generation:` line; `compute` and `flash read/token` are from its
`moe-stream:` line; `cache hit` from `moe-cache:`. `--overlap` is on, so `flash I/O` runs concurrently
with compute and the **stall** (residual flash wait not hidden behind compute) is the honest I/O cost —
it stays ~0.2–0.3 s/tok throughout, i.e. overlap hides almost all of the flash read.
| top-k | lanes | prefetch | tok/s | s/token | compute (s/tok) | flash read/token | cache hit |
|---:|---:|---:|---:|---:|---:|---:|---:|
| **2** | **4** | **off** | **0.687** | **1.455** | 1.159 | 535.75 MiB | 20.4 % |
| 2 | 4 | 4 | 0.532 | 1.878 | 1.529 | 640.91 MiB | 21.2 % |
| 2 | 8 | off | 0.620 | 1.613 | 1.310 | 535.75 MiB | 20.4 % |
| 2 | 8 | 4 | 0.516 | 1.937 | 1.647 | 640.91 MiB | 21.2 % |
| **3** | **4** | **off** | **0.391** | **2.556** | 2.118 | 925.74 MiB | 16.9 % |
| 3 | 4 | 4 | 0.304 | 3.293 | 2.854 | 1100.68 MiB | 18.1 % |
| 3 | 8 | off | 0.279 | 3.581 | 3.074 | 925.74 MiB | 16.9 % |
| 3 | 8 | 4 | 0.295 | 3.394 | 2.924 | 1100.68 MiB | 18.1 % |
| **4** | **4** | **off** | **0.223** | **4.489** | 3.869 | 1402.75 MiB | 13.4 % |
| 4 | 4 | 4 | 0.188 | 5.327 | 4.701 | 1619.72 MiB | 14.7 % |
| 4 | 8 | off † | *0.383* | *2.613* | *2.042* | 1402.75 MiB | 13.4 % |
| 4 | 8 | 4 | 0.213 | 4.704 | 4.045 | 1623.57 MiB | 14.7 % |
| mmap | — | — | 0.089 | 11.240 | — | 0 (page cache) | — |
| mmap (k=4) | — | — | 0.075 | 13.337 | — | 0 (page cache) | — |
† Interrupted run — see caveat 2. Excluded from conclusions.
## Reading the numbers
- **Streaming vs `mmap`: 3–8×.** k=2 streams at 1.455 s/tok against **11.240 s/tok** for a plain
`mmap` load of the same file — **7.7× faster**; k=4 is 4.489 vs 13.337 — **3.0×**. `mmap`-ing 58 GB
onto 11 GB of RAM thrashes the page cache on *every* token (10–13 s each); the bounded 3 GB O_DIRECT
cache replaces that with reads the engine controls, and leaves the rest of RAM for the system.
- **top-k is the dominant lever — it cuts compute *and* I/O.** Both scale almost linearly with k:
compute 1.16 → 2.12 → 3.87 s/tok and flash-read 536 → 926 → 1403 MiB/tok across k = 2 → 3 → 4 (4
lanes). k=2 is ~**3× faster** than k=4. This is the same knob as Qwen/Gemma's Turbo top-k, but it
matters far more here because gpt-oss is heavily compute-bound.
- **Compute-bound, hard.** Even at these low hit rates the *compute* share dominates at k ≥ 3 (k=4:
3.87 s of the 4.49 s decode), because each gpt-oss expert is large (d_ff 2880 — several × a Qwen
expert), so top-4 is a lot of MAC per token. Overlap already hides almost all flash wait (stall
~0.2–0.3 s/tok), so the remaining cost is kernels, not the seam — exactly as on Qwen at a warm cache.
- **prefetch=4 always regresses.** Every `pf 4` row is slower than its `pf off` sibling. Prefetch
reads 20–25 % *more* per token speculatively (k=2: 640.91 vs 535.75 MiB/tok) but only **12–15 %** of
those experts are ever used — on a compute-bound model that wasted flash bandwidth buys nothing and
costs cache churn. Leave prefetch off for gpt-oss.
- **Lanes 4 vs 8: 4 wins where it's trustworthy.** At k=2 io4 beats io8 (1.455 vs 1.613) — with the
flash wait already overlapped, extra lanes only add contention. The k=4 lane comparison is confounded
(caveat 2), so no lane claim is made there.
- **A 24-token probe is mostly warm-up.** Unlike Qwen/Gemma, gpt-oss at 5.2× RAM warms up *inside*
compute: the first tokens fault the mmap-resident, non-expert working set in from flash (`compute_ms`
~18 s), settling to sub-second once hot. The mean over 24 tokens is therefore a floor dominated by that
cold head, and the steady tail is several × faster. This memory-residency warm-up — distinct from the
gentle, I/O-bound cache warm-up on Qwen/Gemma — is analysed token-by-token in
[warmup-analysis.md](warmup-analysis.md).
## Quality — the cost of dropping reasoning
Because these runs use `--no-think` (forced `final` channel, **no** chain-of-thought), the model
answers with no scratch work — and decode is greedy/deterministic, so the answer depends only on k
(identical under streaming and `mmap`):
| top-k | `17 × 23 =` | capital |
|---:|---|---|
| 2 | **391** ✅ | Canberra ✅ |
| 3 | **391** ✅ | Canberra ✅ |
| 4 (default) | **387** ❌ | Canberra ✅ |
The model's *default* top-4 gets the arithmetic **wrong** (387) while the narrower k=2/k=3 get it
**right** (391). This is not "smaller k is smarter" — it is that **without the analysis channel there
is no scratch space to compute 17 × 23**, so the answer is a one-shot guess whose correctness is
prompt- and k-specific. Takeaway: `--no-think` (forced-final) is a **latency/throughput mode** — use
it for direct-answer UX and for benchmarking decode speed; for arithmetic or any multi-step task, drop
`--no-think` and let gpt-oss spend analysis tokens. The capital is correct at every k.
## Provenance
Measured 2026-07-14 with `bmoe-cli` @ `4d12b75` (branch `feat/harmony-nothink-final-channel`). The
summary log for all 14 cells and the two `mmap` per-token CSVs are committed under
[`bench-data/2026-07-14/`](bench-data/2026-07-14/); the streaming cells were run without `--csv` (this
sweep captured only the `moe-stream:` / `moe-cache:` summary lines), so there are no per-token CSVs for
them — a follow-up warm 256-token run with `--csv` and a compute-thread sweep (`-t 4/6/8`) is the next
step. Drivers: [`scripts/gptoss-matrix.sh`](../scripts/gptoss-matrix.sh) (the 12-cell streaming sweep)
and [`scripts/gptoss-mmap.sh`](../scripts/gptoss-mmap.sh) (the 2-cell mmap baseline), prompt and flags
baked in. The model was merged from its two HF shards with `llama-gguf-split --merge` before use.
| Model | Device path | Size |
|---|---|---|
| gpt-oss-120b | `/data/local/tmp/shardllm/gpt-oss-120b-Q4_K_M.gguf` | 58.46 GB |