mirror of
https://github.com/Helldez/BigMoeOnEdge.git
synced 2026-10-03 03:25:42 +00:00
feat(prefill): the NPU prefill reads only routed experts (+ --decide-probe); 0.28.0 (#208)
The NPU prefill's expert arena read every expert of every layer ahead of its routing. It now reads, ahead of a layer's routing, the experts the previous graph routed there, and at the routing node whatever the routing adds. The matmul reads only routed experts, so the output is bit for bit the same. A layer routing more than --prefill-routed-full (0.85) of its experts gets the next one read whole; --no-prefill-routed restores whole layers everywhere. Phone, Hexagon v81 NPU, top-4, same session, every answer identical: Qwen3.6-35B-A3B Q4_0 7.68 -> 4.16 s, Q4_K_M 9.95 -> 5.37 s, Gemma 4 26B-A4B Q4_K_M 6.69 -> 3.70 s, Nemotron 3.5 30B-A3B Q4_0 7.42 -> 6.81 s. Also: --decide-probe (experimental per-decision expert usage and layer-exit answers), BMOE_DECIDE prefill_dev_* counters, gates G17f/G17g, app 0.28.0.
This commit is contained in:
parent
1d37057fac
commit
3170385fad
28 changed files with 869 additions and 43 deletions
|
|
@ -183,7 +183,7 @@ processor with its own requirements: a Snapdragon with a Hexagon NPU of generati
|
|||
2 onwards), and a model the NPU kernels take (Q4_0, Q8_0, MXFP4, or a Q4_K_M; not the Q3 and Q2
|
||||
builds). On a phone without the NPU the switch is shown disabled with the reason. It runs the prompt
|
||||
on the NPU and keeps decode on the CPU, and it widens the prompt batch to 2048 tokens, because each
|
||||
batch reads the experts from flash once; short prompts do not gain. If the NPU does not open (an
|
||||
batch reads the experts it routes to from flash once; the shorter the prompt, the fewer it reads. If the NPU does not open (an
|
||||
older Snapdragon), the prompt simply runs on the CPU. See `../../docs/npu-prefill.md`.
|
||||
|
||||
Descriptions in the UI say what a setting does, without measured figures or flag names, because a
|
||||
|
|
|
|||
|
|
@ -51,8 +51,8 @@ android {
|
|||
applicationId 'io.bigmoeonedge.example'
|
||||
minSdk 29
|
||||
targetSdk 34
|
||||
versionCode 42
|
||||
versionName '0.27.0'
|
||||
versionCode 43
|
||||
versionName '0.28.0'
|
||||
buildConfigField 'String', 'GIT_SHA', "\"${gitSha}\""
|
||||
ndk {
|
||||
// The engine ships as prebuilt arm64 binaries staged by build-android.ps1.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue