feat(prefill): the NPU prefill reads only routed experts (+ --decide-probe); 0.28.0 (#208)

The NPU prefill's expert arena read every expert of every layer ahead of its
routing. It now reads, ahead of a layer's routing, the experts the previous
graph routed there, and at the routing node whatever the routing adds. The
matmul reads only routed experts, so the output is bit for bit the same. A layer
routing more than --prefill-routed-full (0.85) of its experts gets the next one
read whole; --no-prefill-routed restores whole layers everywhere.

Phone, Hexagon v81 NPU, top-4, same session, every answer identical:
Qwen3.6-35B-A3B Q4_0 7.68 -> 4.16 s, Q4_K_M 9.95 -> 5.37 s, Gemma 4 26B-A4B
Q4_K_M 6.69 -> 3.70 s, Nemotron 3.5 30B-A3B Q4_0 7.42 -> 6.81 s.

Also: --decide-probe (experimental per-decision expert usage and layer-exit
answers), BMOE_DECIDE prefill_dev_* counters, gates G17f/G17g, app 0.28.0.
This commit is contained in:
Raffaele 2026-09-29 16:08:23 +02:00 • committed by GitHub
parent 1d37057fac
commit 3170385fad
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
28 changed files with 869 additions and 43 deletions

View file

@ -183,7 +183,7 @@ processor with its own requirements: a Snapdragon with a Hexagon NPU of generati
2 onwards), and a model the NPU kernels take (Q4_0, Q8_0, MXFP4, or a Q4_K_M; not the Q3 and Q2
builds). On a phone without the NPU the switch is shown disabled with the reason. It runs the prompt
on the NPU and keeps decode on the CPU, and it widens the prompt batch to 2048 tokens, because each
batch reads the experts from flash once; short prompts do not gain. If the NPU does not open (an
batch reads the experts it routes to from flash once; the shorter the prompt, the fewer it reads. If the NPU does not open (an
older Snapdragon), the prompt simply runs on the CPU. See `../../docs/npu-prefill.md`.
Descriptions in the UI say what a setting does, without measured figures or flag names, because a

View file

@ -51,8 +51,8 @@ android {
applicationId 'io.bigmoeonedge.example'
minSdk 29
targetSdk 34
versionCode 42
versionName '0.27.0'
versionCode 43
versionName '0.28.0'
buildConfigField 'String', 'GIT_SHA', "\"${gitSha}\""
ndk {
// The engine ships as prebuilt arm64 binaries staged by build-android.ps1.