BigMoeOnEdge

mirror of https://github.com/Helldez/BigMoeOnEdge.git synced 2026-10-03 11:35:50 +00:00

History

Raffaele 3170385fad feat(prefill): the NPU prefill reads only routed experts (+ --decide-probe); 0.28.0 (#208 ) The NPU prefill's expert arena read every expert of every layer ahead of its routing. It now reads, ahead of a layer's routing, the experts the previous graph routed there, and at the routing node whatever the routing adds. The matmul reads only routed experts, so the output is bit for bit the same. A layer routing more than --prefill-routed-full (0.85) of its experts gets the next one read whole; --no-prefill-routed restores whole layers everywhere. Phone, Hexagon v81 NPU, top-4, same session, every answer identical: Qwen3.6-35B-A3B Q4_0 7.68 -> 4.16 s, Q4_K_M 9.95 -> 5.37 s, Gemma 4 26B-A4B Q4_K_M 6.69 -> 3.70 s, Nemotron 3.5 30B-A3B Q4_0 7.42 -> 6.81 s. Also: --decide-probe (experimental per-decision expert usage and layer-exit answers), BMOE_DECIDE prefill_dev_* counters, gates G17f/G17g, app 0.28.0.	2026-09-29 16:08:23 +02:00
..
src	feat: --decide, choose from a list from one prefill, with no decode (#201 )	2026-09-28 12:01:25 +02:00
build.gradle	feat(prefill): the NPU prefill reads only routed experts (+ --decide-probe); 0.28.0 (#208 )	2026-09-29 16:08:23 +02:00

feat(prefill): the NPU prefill reads only routed experts (+ --decide-probe); 0.28.0 (#208 )

The NPU prefill's expert arena read every expert of every layer ahead of its
routing. It now reads, ahead of a layer's routing, the experts the previous
graph routed there, and at the routing node whatever the routing adds. The
matmul reads only routed experts, so the output is bit for bit the same. A layer
routing more than --prefill-routed-full (0.85) of its experts gets the next one
read whole; --no-prefill-routed restores whole layers everywhere.

Phone, Hexagon v81 NPU, top-4, same session, every answer identical:
Qwen3.6-35B-A3B Q4_0 7.68 -> 4.16 s, Q4_K_M 9.95 -> 5.37 s, Gemma 4 26B-A4B
Q4_K_M 6.69 -> 3.70 s, Nemotron 3.5 30B-A3B Q4_0 7.42 -> 6.81 s.

Also: --decide-probe (experimental per-decision expert usage and layer-exit
answers), BMOE_DECIDE prefill_dev_* counters, gates G17f/G17g, app 0.28.0.

2026-09-29 16:08:23 +02:00

src

feat: --decide, choose from a list from one prefill, with no decode (#201 )

2026-09-28 12:01:25 +02:00

build.gradle

feat(prefill): the NPU prefill reads only routed experts (+ --decide-probe); 0.28.0 (#208 )

2026-09-29 16:08:23 +02:00