From bb318cbd2ee4c7306a90ba623ca2cc77c329bb02 Mon Sep 17 00:00:00 2001 From: Helldez <74499579+Helldez@users.noreply.github.com> Date: Wed, 15 Jul 2026 20:32:30 +0200 Subject: [PATCH] docs: split the benchmarks, and say what the engine grew MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit benchmarks.md was two documents in one file: 460 lines, two H1s, and four section names appearing twice. roadmap.md's link to #reading-the-numbers resolved to whichever came first, which happened to be the intended one — a coincidence, not a design. Split the gpt-oss-120b half into its own doc and cross-link the two. That also fixes the anchor at the old benchmarks.md:30: GitHub slugs an em-dash heading to a double hyphen, so #device-pressure-not-just-tokens never jumped anywhere. The README already had this right for its own gpt-oss link, so the convention was there — this one was just wrong. README: route traces and the app's Markdown answers have been in main for several commits with no mention, and a feature nobody can find is a feature nobody has. Trimmed the O_DIRECT and cache bullets in exchange: both re-taught mechanism the linked docs already own, which is what a landing page delegates. --- README.md | 27 ++++---- docs/README.md | 3 +- docs/benchmarks-gpt-oss.md | 127 ++++++++++++++++++++++++++++++++++ docs/benchmarks.md | 135 ++----------------------------------- 4 files changed, 148 insertions(+), 144 deletions(-) create mode 100644 docs/benchmarks-gpt-oss.md diff --git a/README.md b/README.md index d4b1018..555265c 100644 --- a/README.md +++ b/README.md @@ -40,16 +40,13 @@ much smaller memory footprint — is worth making. flash. Loads `use_mmap=true`, repack off, and rebinds each expert tensor onto a streaming buffer in the native gguf layout. Fails fast if the model is not MoE. - **LRU expert cache with an auto budget and ceiling** (`--cache-mb N|auto`, `--cache-ceil-mb`) — - a fixed MiB budget or one sized to the device (free RAM minus a floor), clamped to - `[1.5 GiB, full expert-set size]` and re-checked during generation so it shrinks and grows with - available memory. Cache size is the single biggest throughput lever. + a fixed MiB budget, or one sized to the device and re-checked during generation so it tracks + available memory. The single biggest throughput lever; see + [docs/adaptive-cache.md](docs/adaptive-cache.md). - **Direct-from-flash reads, O_DIRECT** (`--io-threads 1..8`, `--no-odirect`) — each expert slice - is read straight from flash into the engine's own buffer, skipping the operating system's page - cache. That cache normally keeps a second copy of everything you read in spare RAM; here it would - only duplicate weights the engine is already caching itself, waste memory, and evict the user's - other apps. Reading direct keeps memory bounded and read latency predictable. Several read lanes - run in parallel (4 is the UFS 4.x sweet spot), and the engine falls back to normal buffered reads - on the odd filesystem that mishandles O_DIRECT. + is read straight into the engine's own buffer, bypassing the page cache that would otherwise hold + a second copy of weights the engine already caches. Several read lanes run in parallel (4 is the + UFS 4.x sweet spot), with a buffered fallback where O_DIRECT misbehaves. - **Intra-layer I/O–compute overlap** (`--overlap`) — pipelines each layer's async expert reads with its FFN compute, hiding flash latency behind the matmul; byte-identical to the serial path. Top throughput lever over a warm cache. Requires the fork submodule. @@ -62,11 +59,15 @@ much smaller memory footprint — is worth making. - **Honest, per-token telemetry** — `--progress`/`--csv` emit a per-token breakdown: compute vs cache-management vs flash-I/O vs stall seconds, cache hit rate, flash bytes read, cache residency and resizes. The Android panel renders it live. +- **Routing traces** (`--route-trace PATH`) — records which experts every token actually routed to, + per layer, for offline analysis (`scripts/route-analyze.py`, `scripts/route-viewer.py`). A + diagnostic: it perturbs the run, so its tok/s are not comparable with the benchmark tables. - **Experimental, default-off**: temporal prefetch (`--prefetch K`, a cold-start/TTFT tool) reads the next layers' likely experts on idle I/O lanes. An honest toggle kept for provability; it does not help steady-state throughput on current hardware — see [Benchmarks](#benchmarks). -- **Android demo APK** ([`examples/android`](examples/android)) — a multi-turn chat app with a live - telemetry panel and every streaming knob exposed with a one-line note on what it does. +- **Android demo APK** ([`examples/android`](examples/android)) — a multi-turn chat app with + Markdown-rendered answers, a live telemetry panel, and every streaming knob exposed with a + one-line note on what it does. ## Supported models and architectures @@ -176,7 +177,7 @@ gpt-oss is heavily **compute-bound** (each expert is large), so **top-k is the d is ~3× faster than the default k=4 — and prefetch only hurts. These are exploratory **24-token** probes (cache still warming, 13–21% hit), not the 256-token steady state above; treat them as a floor. Full matrix, the k=4 interruption caveat, and a **quality** note (`--no-think` drops gpt-oss's reasoning, so -default k=4 answers `17×23` *wrong* while k=2/3 get it right): [docs/benchmarks.md](docs/benchmarks.md#gpt-oss-120b--a-58-gb-model-at-52-device-ram). +default k=4 answers `17×23` *wrong* while k=2/3 get it right): [docs/benchmarks-gpt-oss.md](docs/benchmarks-gpt-oss.md). ### Desktop is not the target (for now) @@ -247,6 +248,8 @@ it, or reproduce the measurements. The entry points most people want: - [docs/seam.md](docs/seam.md) — the exact contract with llama.cpp's public API. - [docs/adding-a-model.md](docs/adding-a-model.md) — supporting a new MoE architecture. - [docs/telemetry.md](docs/telemetry.md) — the `BMOE_*` line protocol and CSV schema. +- [docs/android-memory.md](docs/android-memory.md) — what reclaims the engine's memory on a phone, + and which levers actually exist. - [docs/benchmarks.md](docs/benchmarks.md) — measured results, and [how they were produced](docs/benchmark-method.md). diff --git a/docs/README.md b/docs/README.md index 9f7ea9e..cdc11c0 100644 --- a/docs/README.md +++ b/docs/README.md @@ -28,7 +28,8 @@ for the idea the project is built on. | Doc | What it answers | |---|---| -| [benchmarks.md](benchmarks.md) | Measured results per model, with device-pressure numbers. | +| [benchmarks.md](benchmarks.md) | Measured results per model on Android, with device-pressure numbers. | +| [benchmarks-gpt-oss.md](benchmarks-gpt-oss.md) | gpt-oss-120b: a 58 GB model at 5.2× device RAM, and what it costs. | | [benchmark-method.md](benchmark-method.md) | How the numbers are produced, so you can reproduce them. | | [warmup-analysis.md](warmup-analysis.md) | Why first tokens are slow, and the two regimes behind it. | | [bench-data/](bench-data/) | Raw per-run CSVs and session notes. A dated archive — see its README. | diff --git a/docs/benchmarks-gpt-oss.md b/docs/benchmarks-gpt-oss.md new file mode 100644 index 0000000..29aac8c --- /dev/null +++ b/docs/benchmarks-gpt-oss.md @@ -0,0 +1,127 @@ +# gpt-oss-120b — a 58 GB model at 5.2× device RAM + +The same engine streams **OpenAI gpt-oss-120b** — a 58.46 GB MoE — on the same 11.3 GB phone. +That is **5.2× device RAM**: the model cannot be held resident by any means, and to our knowledge +this is the first time a 120B / 58 GB model has generated tokens on a phone at all. The run below +is an exploratory sweep (top-k × read-lanes × prefetch), not the polished 256-token matrix used for +Qwen/Gemma in [benchmarks.md](benchmarks.md) — read the two caveats before the numbers. + +## Environment + +| | | +|---|---| +| Device | OnePlus 15R (`CPH2769`), Android 16, 11.3 GB RAM — same as [benchmarks.md](benchmarks.md) | +| Model | `gpt-oss-120b-Q4_K_M.gguf` — 58.46 GB, 36 layers, 128 experts, **top-4** default, MXFP4 expert weights | +| Device path | `/data/local/tmp/shardllm/` — the real `/data` partition, **required** for working O_DIRECT (`/sdcard` is FUSE and silently falls back to buffered) | +| Fraction of RAM | ≈**5.2×** (58.46 GB / 11.3 GB) — resident load is impossible, so there is no in-RAM baseline, only `mmap` page-cache thrash | +| Engine | `bmoe-cli` built from `feat/harmony-nothink-final-channel` @ `4d12b75` (arm64, NDK r26, `armv8.2-a+dotprod+fp16`) | +| Fixed config | `--cache-mb auto --cache-ceil-mb 3000` (auto-sized, capped 3000 MiB), O_DIRECT on, `--overlap` on, `-t 4`, `--no-think` | +| Swept | top-k ∈ {2, 3, 4}, read-lanes ∈ {4, 8}, prefetch ∈ {off, 4} — 12 cells, plus a 2-cell `mmap` baseline | +| Probe | `-n 24`, prompt *"What is 17 times 23? Then name the capital of Australia."* (a short, checkable probe — see Quality) | + +`--no-think` matters here. gpt-oss uses the harmony format, whose template **always** opens an +`analysis` (chain-of-thought) channel — so a normal run spends its whole budget reasoning before it +answers. `--no-think` now primes the `final` channel directly (see the engine fix on this branch), so +the model answers immediately with no analysis tokens. That is what makes a 24-token probe meaningful +— but it also removes the model's scratch space, which the Quality section below shows has a cost. + +## Two caveats (both narrow the numbers, honestly) + +1. **Short probe, not steady state.** These are **24-token** runs, not the 256-token runs used for + Qwen/Gemma. The expert cache is still warming — hit rate sits at **13–21 %** (vs 76 % for Qwen at + 256 tokens), so flash-read/token is high and the absolute tok/s is a **floor**: a warm, longer run + would read less and decode faster. +2. **The k=4 rows were interrupted.** The phone was physically unplugged several times during the + k=4 cells; model-load and TTFT balloon there (k4 · io8 · pf0: load 90 s, TTFT 120 s). That row's + decode is **not trustworthy** — its compute drops to 2.042 s/tok against 3.869 s/tok for the *same* + k=4 at 4 lanes, but compute is lane-independent, so the gap is device state (cooler / less contended + after the interruption), not a lane effect. It is marked † and excluded from every conclusion. Read + k=4 from the **io4 · pf0** row (4.489 s/tok). + +## Results + +s/token and tok/s are the engine's `generation:` line; `compute` and `flash read/token` are from its +`moe-stream:` line; `cache hit` from `moe-cache:`. `--overlap` is on, so `flash I/O` runs concurrently +with compute and the **stall** (residual flash wait not hidden behind compute) is the honest I/O cost — +it stays ~0.2–0.3 s/tok throughout, i.e. overlap hides almost all of the flash read. + +| top-k | lanes | prefetch | tok/s | s/token | compute (s/tok) | flash read/token | cache hit | +|---:|---:|---:|---:|---:|---:|---:|---:| +| **2** | **4** | **off** | **0.687** | **1.455** | 1.159 | 535.75 MiB | 20.4 % | +| 2 | 4 | 4 | 0.532 | 1.878 | 1.529 | 640.91 MiB | 21.2 % | +| 2 | 8 | off | 0.620 | 1.613 | 1.310 | 535.75 MiB | 20.4 % | +| 2 | 8 | 4 | 0.516 | 1.937 | 1.647 | 640.91 MiB | 21.2 % | +| **3** | **4** | **off** | **0.391** | **2.556** | 2.118 | 925.74 MiB | 16.9 % | +| 3 | 4 | 4 | 0.304 | 3.293 | 2.854 | 1100.68 MiB | 18.1 % | +| 3 | 8 | off | 0.279 | 3.581 | 3.074 | 925.74 MiB | 16.9 % | +| 3 | 8 | 4 | 0.295 | 3.394 | 2.924 | 1100.68 MiB | 18.1 % | +| **4** | **4** | **off** | **0.223** | **4.489** | 3.869 | 1402.75 MiB | 13.4 % | +| 4 | 4 | 4 | 0.188 | 5.327 | 4.701 | 1619.72 MiB | 14.7 % | +| 4 | 8 | off † | *0.383* | *2.613* | *2.042* | 1402.75 MiB | 13.4 % | +| 4 | 8 | 4 | 0.213 | 4.704 | 4.045 | 1623.57 MiB | 14.7 % | +| mmap | — | — | 0.089 | 11.240 | — | 0 (page cache) | — | +| mmap (k=4) | — | — | 0.075 | 13.337 | — | 0 (page cache) | — | + +† Interrupted run — see caveat 2. Excluded from conclusions. + +## Reading the numbers + +- **Streaming vs `mmap`: 3–8×.** k=2 streams at 1.455 s/tok against **11.240 s/tok** for a plain + `mmap` load of the same file — **7.7× faster**; k=4 is 4.489 vs 13.337 — **3.0×**. `mmap`-ing 58 GB + onto 11 GB of RAM thrashes the page cache on *every* token (10–13 s each); the bounded 3 GB O_DIRECT + cache replaces that with reads the engine controls, and leaves the rest of RAM for the system. +- **top-k is the dominant lever — it cuts compute *and* I/O.** Both scale almost linearly with k: + compute 1.16 → 2.12 → 3.87 s/tok and flash-read 536 → 926 → 1403 MiB/tok across k = 2 → 3 → 4 (4 + lanes). k=2 is ~**3× faster** than k=4. This is the same knob as Qwen/Gemma's Turbo top-k, but it + matters far more here because gpt-oss is heavily compute-bound. +- **Compute-bound, hard.** Even at these low hit rates the *compute* share dominates at k ≥ 3 (k=4: + 3.87 s of the 4.49 s decode), because each gpt-oss expert is large (d_ff 2880 — several × a Qwen + expert), so top-4 is a lot of MAC per token. Overlap already hides almost all flash wait (stall + ~0.2–0.3 s/tok), so the remaining cost is kernels, not the seam — exactly as on Qwen at a warm cache. +- **prefetch=4 always regresses.** Every `pf 4` row is slower than its `pf off` sibling. Prefetch + reads 20–25 % *more* per token speculatively (k=2: 640.91 vs 535.75 MiB/tok) but only **12–15 %** of + those experts are ever used — on a compute-bound model that wasted flash bandwidth buys nothing and + costs cache churn. Leave prefetch off for gpt-oss. +- **Lanes 4 vs 8: 4 wins where it's trustworthy.** At k=2 io4 beats io8 (1.455 vs 1.613) — with the + flash wait already overlapped, extra lanes only add contention. The k=4 lane comparison is confounded + (caveat 2), so no lane claim is made there. +- **A 24-token probe is mostly warm-up.** Unlike Qwen/Gemma, gpt-oss at 5.2× RAM warms up *inside* + compute: the first tokens fault the mmap-resident, non-expert working set in from flash (`compute_ms` + ~18 s), settling to sub-second once hot. The mean over 24 tokens is therefore a floor dominated by that + cold head, and the steady tail is several × faster. This memory-residency warm-up — distinct from the + gentle, I/O-bound cache warm-up on Qwen/Gemma — is analysed token-by-token in + [warmup-analysis.md](warmup-analysis.md). + +## Quality — the cost of dropping reasoning + +Because these runs use `--no-think` (forced `final` channel, **no** chain-of-thought), the model +answers with no scratch work — and decode is greedy/deterministic, so the answer depends only on k +(identical under streaming and `mmap`): + +| top-k | `17 × 23 =` | capital | +|---:|---|---| +| 2 | **391** ✅ | Canberra ✅ | +| 3 | **391** ✅ | Canberra ✅ | +| 4 (default) | **387** ❌ | Canberra ✅ | + +The model's *default* top-4 gets the arithmetic **wrong** (387) while the narrower k=2/k=3 get it +**right** (391). This is not "smaller k is smarter" — it is that **without the analysis channel there +is no scratch space to compute 17 × 23**, so the answer is a one-shot guess whose correctness is +prompt- and k-specific. Takeaway: `--no-think` (forced-final) is a **latency/throughput mode** — use +it for direct-answer UX and for benchmarking decode speed; for arithmetic or any multi-step task, drop +`--no-think` and let gpt-oss spend analysis tokens. The capital is correct at every k. + +## Provenance + +Measured 2026-07-14 with `bmoe-cli` @ `4d12b75` (branch `feat/harmony-nothink-final-channel`). The +summary log for all 14 cells and the two `mmap` per-token CSVs are committed under +[`bench-data/2026-07-14/`](bench-data/2026-07-14/); the streaming cells were run without `--csv` (this +sweep captured only the `moe-stream:` / `moe-cache:` summary lines), so there are no per-token CSVs for +them — a follow-up warm 256-token run with `--csv` and a compute-thread sweep (`-t 4/6/8`) is the next +step. Drivers: [`scripts/gptoss-matrix.sh`](../scripts/gptoss-matrix.sh) (the 12-cell streaming sweep) +and [`scripts/gptoss-mmap.sh`](../scripts/gptoss-mmap.sh) (the 2-cell mmap baseline), prompt and flags +baked in. The model was merged from its two HF shards with `llama-gguf-split --merge` before use. + +| Model | Device path | Size | +|---|---|---| +| gpt-oss-120b | `/data/local/tmp/shardllm/gpt-oss-120b-Q4_K_M.gguf` | 58.46 GB | diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 1ba9939..c3433ea 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -5,6 +5,9 @@ the model, across the full configuration matrix, for two MoE families. These are numbers the README table and the headline claim are drawn from; the how-to lives in [benchmark-method.md](benchmark-method.md). +For the far end of the ratio — a 58 GB model at 5.2× device RAM — see +[benchmarks-gpt-oss.md](benchmarks-gpt-oss.md). + ## TL;DR - Streaming a >RAM MoE model is **stable and usable**: ~1.6–1.7 tok/s with no cache, and up @@ -27,7 +30,7 @@ numbers the README table and the headline claim are drawn from; the how-to lives *aggregate* throughput collapses (Gemma **0.36** tok/s) because a handful of page-cache-eviction stalls (single tokens as slow as 8 s) dominate the total — and during those stalls the phone is effectively unusable for anything else (see - [Device pressure](#device-pressure-not-just-tokens)). + [Device pressure](#device-pressure--not-just-tokens)). - **With a warm cache, decode is compute-bound, not I/O-bound.** The engine reports the decode split as `compute + flash I/O`. At Qwen's cache 4000 the serial flash I/O share is ~0.13 s/token against ~0.16 s of compute; with overlap the flash wait (stall) falls to @@ -328,133 +331,3 @@ Model files (both Q4_K_M GGUF, staged on the device at `/sdcard/Download/`): Sizes are the on-disk GGUF byte counts; both are stock Q4_K_M conversions, unmodified by the engine (it loads `use_mmap=true`, rebinds expert tensors to the native gguf layout, no repack). - ---- - -# gpt-oss-120b — a 58 GB model at 5.2× device RAM - -The same engine streams **OpenAI gpt-oss-120b** — a 58.46 GB MoE — on the same 11.3 GB phone. -That is **5.2× device RAM**: the model cannot be held resident by any means, and to our knowledge -this is the first time a 120B / 58 GB model has generated tokens on a phone at all. The run below -is an exploratory sweep (top-k × read-lanes × prefetch), not the polished 256-token matrix used for -Qwen/Gemma above — read the two caveats before the numbers. - -## Environment - -| | | -|---|---| -| Device | OnePlus 15R (`CPH2769`), Android 16, 11.3 GB RAM — same as above | -| Model | `gpt-oss-120b-Q4_K_M.gguf` — 58.46 GB, 36 layers, 128 experts, **top-4** default, MXFP4 expert weights | -| Device path | `/data/local/tmp/shardllm/` — the real `/data` partition, **required** for working O_DIRECT (`/sdcard` is FUSE and silently falls back to buffered) | -| Fraction of RAM | ≈**5.2×** (58.46 GB / 11.3 GB) — resident load is impossible, so there is no in-RAM baseline, only `mmap` page-cache thrash | -| Engine | `bmoe-cli` built from `feat/harmony-nothink-final-channel` @ `4d12b75` (arm64, NDK r26, `armv8.2-a+dotprod+fp16`) | -| Fixed config | `--cache-mb auto --cache-ceil-mb 3000` (auto-sized, capped 3000 MiB), O_DIRECT on, `--overlap` on, `-t 4`, `--no-think` | -| Swept | top-k ∈ {2, 3, 4}, read-lanes ∈ {4, 8}, prefetch ∈ {off, 4} — 12 cells, plus a 2-cell `mmap` baseline | -| Probe | `-n 24`, prompt *"What is 17 times 23? Then name the capital of Australia."* (a short, checkable probe — see Quality) | - -`--no-think` matters here. gpt-oss uses the harmony format, whose template **always** opens an -`analysis` (chain-of-thought) channel — so a normal run spends its whole budget reasoning before it -answers. `--no-think` now primes the `final` channel directly (see the engine fix on this branch), so -the model answers immediately with no analysis tokens. That is what makes a 24-token probe meaningful -— but it also removes the model's scratch space, which the Quality section below shows has a cost. - -## Two caveats (both narrow the numbers, honestly) - -1. **Short probe, not steady state.** These are **24-token** runs, not the 256-token runs used for - Qwen/Gemma. The expert cache is still warming — hit rate sits at **13–21 %** (vs 76 % for Qwen at - 256 tokens), so flash-read/token is high and the absolute tok/s is a **floor**: a warm, longer run - would read less and decode faster. -2. **The k=4 rows were interrupted.** The phone was physically unplugged several times during the - k=4 cells; model-load and TTFT balloon there (k4 · io8 · pf0: load 90 s, TTFT 120 s). That row's - decode is **not trustworthy** — its compute drops to 2.042 s/tok against 3.869 s/tok for the *same* - k=4 at 4 lanes, but compute is lane-independent, so the gap is device state (cooler / less contended - after the interruption), not a lane effect. It is marked † and excluded from every conclusion. Read - k=4 from the **io4 · pf0** row (4.489 s/tok). - -## Results - -s/token and tok/s are the engine's `generation:` line; `compute` and `flash read/token` are from its -`moe-stream:` line; `cache hit` from `moe-cache:`. `--overlap` is on, so `flash I/O` runs concurrently -with compute and the **stall** (residual flash wait not hidden behind compute) is the honest I/O cost — -it stays ~0.2–0.3 s/tok throughout, i.e. overlap hides almost all of the flash read. - -| top-k | lanes | prefetch | tok/s | s/token | compute (s/tok) | flash read/token | cache hit | -|---:|---:|---:|---:|---:|---:|---:|---:| -| **2** | **4** | **off** | **0.687** | **1.455** | 1.159 | 535.75 MiB | 20.4 % | -| 2 | 4 | 4 | 0.532 | 1.878 | 1.529 | 640.91 MiB | 21.2 % | -| 2 | 8 | off | 0.620 | 1.613 | 1.310 | 535.75 MiB | 20.4 % | -| 2 | 8 | 4 | 0.516 | 1.937 | 1.647 | 640.91 MiB | 21.2 % | -| **3** | **4** | **off** | **0.391** | **2.556** | 2.118 | 925.74 MiB | 16.9 % | -| 3 | 4 | 4 | 0.304 | 3.293 | 2.854 | 1100.68 MiB | 18.1 % | -| 3 | 8 | off | 0.279 | 3.581 | 3.074 | 925.74 MiB | 16.9 % | -| 3 | 8 | 4 | 0.295 | 3.394 | 2.924 | 1100.68 MiB | 18.1 % | -| **4** | **4** | **off** | **0.223** | **4.489** | 3.869 | 1402.75 MiB | 13.4 % | -| 4 | 4 | 4 | 0.188 | 5.327 | 4.701 | 1619.72 MiB | 14.7 % | -| 4 | 8 | off † | *0.383* | *2.613* | *2.042* | 1402.75 MiB | 13.4 % | -| 4 | 8 | 4 | 0.213 | 4.704 | 4.045 | 1623.57 MiB | 14.7 % | -| mmap | — | — | 0.089 | 11.240 | — | 0 (page cache) | — | -| mmap (k=4) | — | — | 0.075 | 13.337 | — | 0 (page cache) | — | - -† Interrupted run — see caveat 2. Excluded from conclusions. - -## Reading the numbers - -- **Streaming vs `mmap`: 3–8×.** k=2 streams at 1.455 s/tok against **11.240 s/tok** for a plain - `mmap` load of the same file — **7.7× faster**; k=4 is 4.489 vs 13.337 — **3.0×**. `mmap`-ing 58 GB - onto 11 GB of RAM thrashes the page cache on *every* token (10–13 s each); the bounded 3 GB O_DIRECT - cache replaces that with reads the engine controls, and leaves the rest of RAM for the system. -- **top-k is the dominant lever — it cuts compute *and* I/O.** Both scale almost linearly with k: - compute 1.16 → 2.12 → 3.87 s/tok and flash-read 536 → 926 → 1403 MiB/tok across k = 2 → 3 → 4 (4 - lanes). k=2 is ~**3× faster** than k=4. This is the same knob as Qwen/Gemma's Turbo top-k, but it - matters far more here because gpt-oss is heavily compute-bound. -- **Compute-bound, hard.** Even at these low hit rates the *compute* share dominates at k ≥ 3 (k=4: - 3.87 s of the 4.49 s decode), because each gpt-oss expert is large (d_ff 2880 — several × a Qwen - expert), so top-4 is a lot of MAC per token. Overlap already hides almost all flash wait (stall - ~0.2–0.3 s/tok), so the remaining cost is kernels, not the seam — exactly as on Qwen at a warm cache. -- **prefetch=4 always regresses.** Every `pf 4` row is slower than its `pf off` sibling. Prefetch - reads 20–25 % *more* per token speculatively (k=2: 640.91 vs 535.75 MiB/tok) but only **12–15 %** of - those experts are ever used — on a compute-bound model that wasted flash bandwidth buys nothing and - costs cache churn. Leave prefetch off for gpt-oss. -- **Lanes 4 vs 8: 4 wins where it's trustworthy.** At k=2 io4 beats io8 (1.455 vs 1.613) — with the - flash wait already overlapped, extra lanes only add contention. The k=4 lane comparison is confounded - (caveat 2), so no lane claim is made there. -- **A 24-token probe is mostly warm-up.** Unlike Qwen/Gemma, gpt-oss at 5.2× RAM warms up *inside* - compute: the first tokens fault the mmap-resident, non-expert working set in from flash (`compute_ms` - ~18 s), settling to sub-second once hot. The mean over 24 tokens is therefore a floor dominated by that - cold head, and the steady tail is several × faster. This memory-residency warm-up — distinct from the - gentle, I/O-bound cache warm-up on Qwen/Gemma — is analysed token-by-token in - [warmup-analysis.md](warmup-analysis.md). - -## Quality — the cost of dropping reasoning - -Because these runs use `--no-think` (forced `final` channel, **no** chain-of-thought), the model -answers with no scratch work — and decode is greedy/deterministic, so the answer depends only on k -(identical under streaming and `mmap`): - -| top-k | `17 × 23 =` | capital | -|---:|---|---| -| 2 | **391** ✅ | Canberra ✅ | -| 3 | **391** ✅ | Canberra ✅ | -| 4 (default) | **387** ❌ | Canberra ✅ | - -The model's *default* top-4 gets the arithmetic **wrong** (387) while the narrower k=2/k=3 get it -**right** (391). This is not "smaller k is smarter" — it is that **without the analysis channel there -is no scratch space to compute 17 × 23**, so the answer is a one-shot guess whose correctness is -prompt- and k-specific. Takeaway: `--no-think` (forced-final) is a **latency/throughput mode** — use -it for direct-answer UX and for benchmarking decode speed; for arithmetic or any multi-step task, drop -`--no-think` and let gpt-oss spend analysis tokens. The capital is correct at every k. - -## Provenance - -Measured 2026-07-14 with `bmoe-cli` @ `4d12b75` (branch `feat/harmony-nothink-final-channel`). The -summary log for all 14 cells and the two `mmap` per-token CSVs are committed under -[`bench-data/2026-07-14/`](bench-data/2026-07-14/); the streaming cells were run without `--csv` (this -sweep captured only the `moe-stream:` / `moe-cache:` summary lines), so there are no per-token CSVs for -them — a follow-up warm 256-token run with `--csv` and a compute-thread sweep (`-t 4/6/8`) is the next -step. Drivers: [`scripts/gptoss-matrix.sh`](../scripts/gptoss-matrix.sh) (the 12-cell streaming sweep) -and [`scripts/gptoss-mmap.sh`](../scripts/gptoss-mmap.sh) (the 2-cell mmap baseline), prompt and flags -baked in. The model was merged from its two HF shards with `llama-gguf-split --merge` before use. - -| Model | Device path | Size | -|---|---|---| -| gpt-oss-120b | `/data/local/tmp/shardllm/gpt-oss-120b-Q4_K_M.gguf` | 58.46 GB |