BigMoeOnEdge/docs/row-gathered-tables.md

Row-gathered dense tables (--row-stream)

A dense weight that is multiplied is read whole, every token: making it resident is the only sensible policy, and that is what --dense-weights decides between. A dense weight the graph only gathers rows from is a different object. The token embedding table is the pure case, one row per decoded token out of a vocabulary of hundreds of thousands, and on a big model it is hundreds of MiB of RAM bought for a kilobyte of use per token.

--row-stream binds such a table to reserved address space and pulls in only the rows the graph is about to read, from flash. Measured on the 12 GB test phone:

model	table	resident dense, off	resident dense, on	flash read for it
Qwen3.6-35B-A3B UD-Q3_K_XL	token_embd 515 MiB	2436 MiB	1921 MiB	0.4 MiB / 12 tokens
Qwen3.8-Flash-Next UD-IQ3_XXS	token_embd 497 MiB	4313 MiB	3816 MiB	0.4 MiB / 24 tokens

Output is byte-identical in every cell. That is not a quality claim but a structural fact: see Why identity is the whole test.

Which tables qualify, and why there is no list of them

Nothing in the engine names a tensor. During the streamer's capture decode every graph node is already offered to the eval callback, and each reference to a dense weight is classified by how the node used it. A table qualifies only when both hold:

1. Every reference to it in the captured graph is a row gather (GGML_OP_GET_ROWS) with the table in the source position. One reference of any other kind disqualifies it.
2. Every such gather's index tensor is already materialized when the node runs: a graph input, or a pure view of one. An index computed inside the graph cannot be read before the node that produces it has run.

Both conditions are properties of the graph, so the rule carries to architectures nobody had in mind when it was written:

• A model with tied embeddings, where token_embd is also the output head, fails rule 1 on the head's matmul and keeps the residency it had. No special case, on any architecture.
• Qwen3.8-Flash-Next's n-gram table (per_layer_token_embd, ~27 GB) is row-gathered, but by hashes computed inside the graph, so it fails rule 2 and falls through to the size guard that was measured for it: mmap'd with random-access advice (see architecture.md and the 0.22.0 changelog entry).

The policy is also restricted to tables that could have been resident at all, which is to say it runs after that size guard. That is deliberate: the fallback below has to be able to pull a whole table in, and a table larger than memory could not honour it.

Mechanism

The reservation mirrors the tensor's own byte layout, so ggml's address arithmetic (data + i*nb1) is unchanged and the gather kernel is untouched. It is the same trick the expert cache plays on mul_mat_id, at row granularity instead of expert granularity.

• Slabs, not rows. A row is ~1 KiB, far below any device's request floor, so the unit fetched is a 16 KiB page-aligned slab: a read that size costs what the floor costs anyway, and adjacent vocabulary rows recur. This is a property of the storage, not of a model.
• A bounded window. --row-stream-mb (default 64) caps what all row-streamed tables hold resident together, evicted LRU. In practice a run never approaches it, since the tables above hold well under a MiB, but the ceiling is what makes the residency claim a guarantee rather than a hope. A single gather that alone exceeds the window overshoots it for that one node rather than reading a row at a time, and the slabs the gather in flight needs are never evicted under it.
• Ordered reads. A gather's slabs are deduplicated and issued in ascending file order.
• A fallback with teeth. IRowSource::materialize() pulls the whole table in and says so on stderr. It fires if a graph shape the capture pass never saw reads a served table any other way, or if a gather's index turns out to be unreadable. The engine is then simply back to the behaviour it would have had without the flag.

The policy applies under every --dense-weights mode, because what it changes is not how a mode works but which tensors the mode is applied to. A table it takes over leaves the resident dense set entirely: not read, not rebound, not warmed, and outside the residency sensor. A sensor that counted it would report a set that is resident by design only in part.

Why identity is the whole test

The tensor is bound to reserved, not allocated, memory. A row the policy failed to fetch is therefore not a slightly wrong weight: it is memory that was never written, and the output diverges on the first token that touches it. Byte-identity to the resident reference is proof that every gathered row arrived.

Gates G15a and G15b in tests/moe_gates.cpp assert exactly that on all three tiny models, G15b with a window of one slab so that nearly every gather has to re-read what the previous one handed back. What a gate cannot prove is that a table qualified at all. That is what the run's own moe-rows: line reports, on the real model.

What it costs, and what it is for

The read cost is negligible in both directions measured: 0.4 MiB across a whole generation, against 4.5 GB of expert reads on Qwen3.8. Expert bytes, cache hit rate, evictions and re-reads are identical to the digit with the flag on and off, so the expert path is not touched.

Throughput is, so far, neutral. Interleaved host cells on Qwen3.6-35B-A3B Q4_K_M gave 2.22 and 2.30 tok/s with the flag off, 2.27 and 2.31 with it on, a spread smaller than the one between the two baseline runs. The flag is off by default until a long on-device run says more.

What it is actually for is the RAM. On the 12 GB phone, Qwen3.8-Flash-Next pins 4.3 GB of dense weights, and the kernel's reclaim accounting cannot see them: MemAvailable falls under ~0.9 GB and the anonymous expert cache starts being compacted to zram, which is what a 2 to 7 second token is (see the 0.22.0 changelog entry). Half a gigabyte handed back to the system addresses that cause directly, and is also half a gigabyte the expert cache can take instead.

Flags

--row-stream        serve row-gathered dense tables from flash instead of RAM
--row-stream-mb N   resident window for those tables, in MiB (default 64)

Requires --moe-stream, since the tables are discovered by the capture pass and fed by the eval callback. In the Android app: Stream row-gathered tables, next to the dense-weight mode.

Telemetry: the moe-rows: end-of-run line and the row_* keys in the CSV summary trailer, both in telemetry.md.