BigMoeOnEdge/examples/android

BigMoeOnEdge — Android example

A minimal chat app that validates the throughput claim on a real phone: pick a pushed .gguf, type a prompt, and watch the answer stream in while a live panel shows tok/s and the per-token compute-vs-flash-I/O split and cache hit rate.

It runs the engine as the bmoe-cli binary (shipped as libbmoe-cli.so) via ProcessBuilder from a foreground service — no JNI. This is the same pattern used by the research harness and keeps the app a thin driver over the CLI.

Build

1. Cross-compile and stage the engine binaries (needs the Android NDK):

   pwsh ../../scripts/build-android.ps1
   

   This fills app/src/main/jniLibs/arm64-v8a/ with libbmoe-cli.so and the libllama/libggml shared libraries.

   For Prefill on the NPU the engine needs the Hexagon backend, which builds inside upstream's Snapdragon toolchain container instead (it carries the Hexagon SDK), then stages the same way:

   docker run --rm -v <repo>:/workspace ghcr.io/snapdragon-toolchain/arm64-android:v0.7 \
       bash /workspace/scripts/build-hexagon-android.sh
   powershell -File ../../scripts/stage-hexagon-jnilibs.ps1
   

   The release APK is built this way by CI, with a skel for every NPU generation the backend supports (v73 to v81). A local APK built with build-android.ps1 has no Hexagon backend: its NPU switch is shown disabled, and ignored if a previous install saved it on.

2. Build and install the APK. Open this folder in Android Studio, or use the committed Gradle wrapper directly. The app has two distribution flavors (see below); build the one you want:

   ./gradlew assembleDevDebug
   adb install app/build/outputs/apk/dev/debug/app-dev-debug.apk
   

   The app's pure-Kotlin rules (catalog status and the like) have JVM unit tests under app/src/test; CI runs them, and so does ./gradlew testDevDebugUnitTest.

   Published sideload builds are signed with a stable key instead, so an update installs over the previous one rather than being refused. That needs a keystore.properties next to app/ (gitignored — it points at the keystore and holds its passwords); without it, builds fall back to debug signing. The APKs attached to a GitHub release are built by the release-apk workflow from a clean checkout of the tag when the release is published, signed with the same stable key from repository secrets — no locally built artifact is uploaded by hand.

Flavors

Two build flavors differ only in how a model reaches the device:

• dev — sideloaded (this is what CI attaches to releases). Keeps all-files access, so it can also read a model adb-pushed to shared storage. Application id …​.example.dev.
• play — Play-Store-compliant. No broad storage permission: models come only through the in-app downloader or the file picker. ./gradlew assemblePlayDebug.

Both declare android:appCategory="game". That is a performance decision rather than a claim about what the app is: vendor layers read the attribute to pick a CPU governor profile, and on the OxygenOS test device it lifted the foreground ceiling from 1.9/1.65 GHz to the hardware maximum of 3.32/3.80 GHz. Decode is the most CPU-hungry thing a phone does outside a game. The effect is the vendor's, not Android's — neutral on stock builds, and Samsung's game service has historically throttled apps it classifies this way — so treat any figure as a per-device measurement. It cannot be toggled at runtime; a manifest attribute is fixed at install, and the only lever would be a per-flavor manifest. Numbers measured in the app before this landed are not comparable with numbers measured after it.

Getting a model onto the device

The picker lists every MoE .gguf it finds (dense models are filtered out by a gguf-header check). Nothing below needs a storage permission except the last option.

1. Built-in catalog (both flavors) — the "Get a model" card offers the models this engine is measured on, each a single tap: Qwen3-30B-A3B-Q4_K_M (~18.6 GB, the reference model), Qwen3.6-35B-A3B-Q4_K_M (~22.3 GB, a hybrid attention/SSM MoE, comfortably past device RAM), Gemma-4-26B-A4B-it-Q4_K_M (~17 GB) and Nemotron-3.5-Lightning-30B-A3B-Q4_0 (~18.9 GB, a hybrid Mamba2/attention MoE with gate-less experts, from ggml-org). Downloads run in a foreground worker, survive the app being killed, resume an interrupted transfer instead of restarting, and appear in the picker when done.

2. Any other model — under Other model, paste a direct gguf URL (e.g. a Hugging Face …/resolve/main/model.gguf link), or pick a .gguf already on the device to import it.

   You do not need a special file for Guess ahead → Model's own head (MTP): the catalog's Qwen3.6 entry already carries the nextn block the MTP head lives in, as do Qwen3.6's ordinary quantisations generally. A gguf named -MTP- is the same head at a different quantisation. On a model with no head — anything that is not Qwen3.5/3.6 — the engine refuses to open rather than silently decoding one token at a time, so a wrong file fails immediately and says why. Guess ahead → Repeated text (n-gram) has no such requirement: it guesses from the text rather than from the weights, so it works on every model in the catalog.

   In-app downloads and picker imports both land in the app's internal storage (filesDir, a real f2fs/ext4 volume), so the streamed expert reads use O_DIRECT at full speed. Only models read from the emulated external dirs (adb-pushed to /sdcard/Download) fall back to buffered I/O. A download needs free space equal to the model size — no temporary second copy.

3. adb push (dev flavor only — needs all-files access, which the dev build requests):

   adb push Qwen3-30B-A3B-Q4_K_M.gguf /sdcard/Download/
   # /data/local/tmp/bmoe avoids duplicating a model too big to copy, and is on a real
   # filesystem where O_DIRECT works (the emulated dirs fall back to buffered I/O)
   adb push Qwen3-30B-A3B-Q4_K_M.gguf /data/local/tmp/bmoe/
   

   This directory was named shardllm before v0.8.0. To keep models already pushed there:

   adb shell mv /data/local/tmp/shardllm /data/local/tmp/bmoe
   

Sharded models (gpt-oss-120b, DeepSeek V4 Flash, Qwen3.8-Flash-Next)

Models above Hugging Face's 50 GB per-file limit ship as several shard files (-00001-of-0000N.gguf). The engine streams a split set natively, so these download in-app like any other catalog entry: the shards are fetched one at a time (each resumable), the row shows one progress bar over the whole set, and the model list offers the FIRST shard, which is the file the engine opens; it finds the siblings next to it. A merged single-file gpt-oss from an earlier release keeps working and still shows as on-device.

For adb-pushed models the same rule applies: push all shards to the same directory and pass the first one:

adb push DeepSeek-V4-Flash-0731-UD-IQ2_M-0000*-of-00003.gguf /data/local/tmp/bmoe/

Mind the space: DeepSeek V4 Flash UD-IQ2_M is ~91 GB on disk, Qwen3.8-Flash-Next UD-IQ3_XXS ~82 GB and its Q2_K build ~80 GB.

Qwen3.8-Flash-Next needs Dense weights = Pinned (dma-buf) in Settings. Its dense side is 4.3 GB and every token walks it: with Anon the kernel swaps it to zram and single tokens stall for 10-20 s, with Mmap it is refaulted from flash every token. Its 51B n-gram table is held back automatically and stays mmap'd whatever the setting; the engine says so on stderr at load. Keep the expert cache at 1000-1500 MiB on a 12 GB phone: the pinned dense set leaves no room for more.

The catalog also offers a Q2_K build of Qwen3.8-Flash-Next (DevQuasar) whose dense side is at 2-4 bit: 2.4 GB pinned instead of 4.3, which is ~2 GB the expert cache can have back. It is a plain llama-quantize build without an importance matrix, so its experts are coarser than the UD-IQ3_XXS ones; pick it when the cache, not expert precision, is what limits the phone. Every published dynamic quant of this model keeps the dense side at 5-8 bit whatever its overall size, which is why the two builds sit side by side.

Stream row-gathered tables takes ~500 MiB more off that pinned set on this model, and 515 MiB on Qwen3.6: the token embedding table is read one row per token, so it does not need to be in RAM at all. The reply is identical either way. It is off by default until a long run on a phone says whether the RAM it hands back is worth the reads, which is exactly the kind of thing this app exists to find out.

Expected numbers

On a phone with UFS 4.x storage and ~12 GB RAM, streaming Qwen3-30B-A3B-Q4_K_M with the expert cache at 4000 MiB, 4 I/O lanes and 4 compute threads, decode settles around 0.55–0.6 s/token (~1.8 tok/s) — a model ~1.7× the device RAM, lossless. That 4000 MiB is a sweep point from the benchmark protocol, not the app default: the app ships a fixed 2000 MiB expert cache. See ../../docs/benchmark-method.md for the full procedure and the cache/thread sweep.

Choose from options

The Choose from options switch on the chat screen turns the prompt into a question and adds a field for options, one per line. The model picks one without writing an answer: the options are lettered, the whole request is one prefill with no decode, and each option comes back with the probability the model put on it (they need not add up to 100%; the rest is the model being unsure). Asking the same question again with other options reuses the question instead of reading it again. A Choose turn ends the chat conversation, so the next chat message starts a new one. The session always accepts these requests, so switching between Chat and Choose never reloads the model. The engine side is docs/decide.md.

How Settings are organised

Each category shows the recommended configuration first and folds everything else into a collapsed Experimental group: the levers measured on one device, measured once, or still owed a measurement. They ship in the release build deliberately, because testing them on hardware other than the one test phone is what this app is for.

The NPU section holds "Prefill on the NPU (Snapdragon only)" (--prefill-device HTP0) and its loader threads. It is off by default and sits apart from Experimental because it is a different processor with its own requirements: a Snapdragon with a Hexagon NPU of generation v73 or newer (8 Gen 2 onwards), and a model the NPU kernels take (Q4_0, Q8_0, MXFP4, or a Q4_K_M; not the Q3 and Q2 builds). On a phone without the NPU the switch is shown disabled with the reason. It runs the prompt on the NPU and keeps decode on the CPU, and it widens the prompt batch to 2048 tokens, because each batch reads the experts it routes to from flash once; the shorter the prompt, the fewer it reads. If the NPU does not open (an older Snapdragon), the prompt simply runs on the CPU. See ../../docs/npu-prefill.md.

Descriptions in the UI say what a setting does, without measured figures or flag names, because a number needs the device, the model and the day beside it to be worth anything. The mapping to the CLI flags and the evidence behind each one lives in the docs, and the metrics screen keeps the flag names so a reading there can be matched against a CSV column.

Two worth knowing before you turn them on:

• "Ask the next layer what it wants" (--predict-prefetch) predicts each layer's experts one layer early and fetches or retains what the prediction names. It is markedly more accurate than the previous-token guess it replaces, and a better guess still did not buy throughput: in thermally matched pairs the read-ahead lost, because the flash is already saturated. See ../../docs/expert-prediction.md before drawing conclusions from a run.
• "Decide the experts early" (--route-ahead) commits each layer's routing before that layer runs, so the reads can never be wasted. It changes the reply, and it is refused alongside guessing ahead. See ../../docs/route-ahead.md.
• "Stream row-gathered tables" (--row-stream) serves the token embedding table out of flash instead of RAM. Lossless, and which tables it applies to is read off the model's own graph, so on a model where none qualify it does nothing. See ../../docs/row-gathered-tables.md.
• "Release the model mapping" (--release-mmap) hands the model file's mapping back to the kernel once every weight has been copied into the app's own memory, so it needs Dense weights on Anon or Pinned and is disabled otherwise. Lossless. The mechanism that makes it worth +46% on a Windows desktop does not exist here — on f2fs the read lanes measure the same either way — but keeping a 20 GB mapping registered costs a kernel under memory pressure, and dropping it took ~9% off CPU per token. Two 48-token cells on a device that spreads 20%: a direction, not a number, which is why it is off by default.
• "Prefer cached experts" (--expert-substitute) steers each routing toward experts already in RAM, so the same number of experts runs but fewer are read from flash. It changes the reply, and past 20% the reply keeps reading well while the model behind it is much worse: judge it on answers you can check. See ../../docs/cache-aware-substitution.md.