mirror of https://github.com/Helldez/BigMoeOnEdge.git synced 2026-10-03 11:35:50 +00:00

feat(io): release the model file's mapping after load (--release-mmap) (#185 )

llama.cpp maps every gguf it loads and keeps the mapping for the model's lifetime. On
Windows that is expensive in a way nothing had attributed: while a section of a file is
alive, NTFS serialises concurrent unbuffered reads on that file, and a lane opened while
the section existed keeps serialising against it after the section is gone. Four I/O lanes
therefore delivered exactly one lane's throughput, which is why lanes and threads have
always measured dead on the desktop host and why the engine read at about a third of what
the drive can serve.

--release-mmap hands the mapping back after load: unmap the file, close its section, reopen
the reader lanes. Both halves are needed. Whether it is safe is decided by looking rather
than by reasoning, the engine asks the OS whether any weight the capture pass observed still
points inside a mapping of the model files, and declines if any does. Off by default,
because the check answers for the pointers the capture saw and for no others.

Host A/B on Qwen3.6-35B-A3B Q4_K_M: 3.16 to 4.63 tok/s (+46%), flash stall per token 0.182
to 0.074, with bytes read, hit rate, evictions and re-reads identical to the digit and the
generated text byte-identical. On the phone the read path is flat (f2fs does not serialise)
but CPU per token falls about 9%; that cell is two short runs per variant and is recorded as
a direction, not a number.

Also fixes a bug this uncovered, independent of the flag: when a gguf carries no
output.weight, llama.cpp builds the output head from the token embedding table and the model
holds two identically named tensors over the same bytes. The capture pass keyed its map by
name, so --dense-weights anon and ahwb rebound one and left the twin reading the mmap for
the whole run, which on a model past RAM means the output projection served by page faults
from flash. The capture now records every distinct leaf object by address and the dense
policy rebinds every tensor over one file range onto the same buffer.

Adds bmoe-iobench --mmap / --reopen-lanes / --range-mb / --fresh, the cells that isolate the
mechanism, a mapping_release unit test on both platforms, the app switch "Release the model
mapping", and the bench findings. README, architecture, AGENTS and roadmap updated, the last
correcting a diagnosis this refutes.

2026-09-07 20:43:02 +02:00

11 KiB

Raw Blame History

BigMoeOnEdge — Android example

A minimal chat app that validates the throughput claim on a real phone: pick a pushed .gguf, type a prompt, and watch the answer stream in while a live panel shows tok/s and the per-token compute-vs-flash-I/O split and cache hit rate.

It runs the engine as the bmoe-cli binary (shipped as libbmoe-cli.so) via ProcessBuilder from a foreground service — no JNI. This is the same pattern used by the research harness and keeps the app a thin driver over the CLI.

Build

Cross-compile and stage the engine binaries (needs the Android NDK):
```
pwsh ../../scripts/build-android.ps1
```
This fills app/src/main/jniLibs/arm64-v8a/ with libbmoe-cli.so and the libllama/libggml shared libraries.
Build and install the APK. Open this folder in Android Studio, or use the committed Gradle wrapper directly. The app has two distribution flavors (see below); build the one you want:
```
./gradlew assembleDevDebug
adb install app/build/outputs/apk/dev/debug/app-dev-debug.apk
```
The app's pure-Kotlin rules (catalog status and the like) have JVM unit tests under app/src/test; CI runs them, and so does ./gradlew testDevDebugUnitTest.

Published sideload builds are signed with a stable key instead, so an update installs over the previous one rather than being refused. That needs a keystore.properties next to app/ (gitignored — it points at the keystore and holds its passwords); without it, builds fall back to debug signing. The APKs attached to a GitHub release are built by the release-apk workflow from a clean checkout of the tag when the release is published, signed with the same stable key from repository secrets — no locally built artifact is uploaded by hand.

Flavors

Two build flavors differ only in how a model reaches the device:

dev — sideloaded (this is what CI attaches to releases). Keeps all-files access, so it can also read a model adb-pushed to shared storage. Application id ….example.dev.
play — Play-Store-compliant. No broad storage permission: models come only through the in-app downloader or the file picker. ./gradlew assemblePlayDebug.

Both declare android:appCategory="game". That is a performance decision rather than a claim about what the app is: vendor layers read the attribute to pick a CPU governor profile, and on the OxygenOS test device it lifted the foreground ceiling from 1.9/1.65 GHz to the hardware maximum of 3.32/3.80 GHz. Decode is the most CPU-hungry thing a phone does outside a game. The effect is the vendor's, not Android's — neutral on stock builds, and Samsung's game service has historically throttled apps it classifies this way — so treat any figure as a per-device measurement. It cannot be toggled at runtime; a manifest attribute is fixed at install, and the only lever would be a per-flavor manifest. Numbers measured in the app before this landed are not comparable with numbers measured after it.

Getting a model onto the device

The picker lists every MoE .gguf it finds (dense models are filtered out by a gguf-header check). Nothing below needs a storage permission except the last option.

Built-in catalog (both flavors) — the "Get a model" card offers the models this engine is measured on, each a single tap: Qwen3-30B-A3B-Q4_K_M (~18.6 GB, the reference model), Qwen3.6-35B-A3B-Q4_K_M (~22.3 GB, a hybrid attention/SSM MoE, comfortably past device RAM) and Gemma-4-26B-A4B-it-Q4_K_M (~17 GB). Downloads run in a foreground worker, survive the app being killed, resume an interrupted transfer instead of restarting, and appear in the picker when done.
Any other model — under Other model, paste a direct gguf URL (e.g. a Hugging Face …/resolve/main/model.gguf link), or pick a .gguf already on the device to import it.

You do not need a special file for Guess ahead → Model's own head (MTP): the catalog's Qwen3.6 entry already carries the nextn block the MTP head lives in, as do Qwen3.6's ordinary quantisations generally. A gguf named -MTP- is the same head at a different quantisation. On a model with no head — anything that is not Qwen3.5/3.6 — the engine refuses to open rather than silently decoding one token at a time, so a wrong file fails immediately and says why. Guess ahead → Repeated text (n-gram) has no such requirement: it guesses from the text rather than from the weights, so it works on every model in the catalog.

In-app downloads and picker imports both land in the app's internal storage (filesDir, a real f2fs/ext4 volume), so the streamed expert reads use O_DIRECT at full speed. Only models read from the emulated external dirs (adb-pushed to /sdcard/Download) fall back to buffered I/O. A download needs free space equal to the model size — no temporary second copy.

adb push (dev flavor only — needs all-files access, which the dev build requests):

adb push Qwen3-30B-A3B-Q4_K_M.gguf /sdcard/Download/
# /data/local/tmp/bmoe avoids duplicating a model too big to copy, and is on a real
# filesystem where O_DIRECT works (the emulated dirs fall back to buffered I/O)
adb push Qwen3-30B-A3B-Q4_K_M.gguf /data/local/tmp/bmoe/

This directory was named shardllm before v0.8.0. To keep models already pushed there:

adb shell mv /data/local/tmp/shardllm /data/local/tmp/bmoe

Sharded models (gpt-oss-120b, DeepSeek V4 Flash, Qwen3.8-Flash-Next)

Models above Hugging Face's 50 GB per-file limit ship as several shard files (-00001-of-0000N.gguf). The engine streams a split set natively, so these download in-app like any other catalog entry: the shards are fetched one at a time (each resumable), the row shows one progress bar over the whole set, and the model list offers the FIRST shard, which is the file the engine opens; it finds the siblings next to it. A merged single-file gpt-oss from an earlier release keeps working and still shows as on-device.

For adb-pushed models the same rule applies: push all shards to the same directory and pass the first one:

adb push DeepSeek-V4-Flash-0731-UD-IQ2_M-0000*-of-00003.gguf /data/local/tmp/bmoe/

Mind the space: DeepSeek V4 Flash UD-IQ2_M is ~91 GB on disk, Qwen3.8-Flash-Next UD-IQ3_XXS ~82 GB and its Q2_K build ~80 GB.

Qwen3.8-Flash-Next needs Dense weights = Pinned (dma-buf) in Settings. Its dense side is 4.3 GB and every token walks it: with Anon the kernel swaps it to zram and single tokens stall for 10-20 s, with Mmap it is refaulted from flash every token. Its 51B n-gram table is held back automatically and stays mmap'd whatever the setting; the engine says so on stderr at load. Keep the expert cache at 1000-1500 MiB on a 12 GB phone: the pinned dense set leaves no room for more.

The catalog also offers a Q2_K build of Qwen3.8-Flash-Next (DevQuasar) whose dense side is at 2-4 bit: 2.4 GB pinned instead of 4.3, which is ~2 GB the expert cache can have back. It is a plain llama-quantize build without an importance matrix, so its experts are coarser than the UD-IQ3_XXS ones; pick it when the cache, not expert precision, is what limits the phone. Every published dynamic quant of this model keeps the dense side at 5-8 bit whatever its overall size, which is why the two builds sit side by side.

Stream row-gathered tables takes ~500 MiB more off that pinned set on this model, and 515 MiB on Qwen3.6: the token embedding table is read one row per token, so it does not need to be in RAM at all. The reply is identical either way. It is off by default until a long run on a phone says whether the RAM it hands back is worth the reads, which is exactly the kind of thing this app exists to find out.

Expected numbers

On a phone with UFS 4.x storage and ~12 GB RAM, streaming Qwen3-30B-A3B-Q4_K_M with the expert cache at 4000 MiB, 4 I/O lanes and 4 compute threads, decode settles around 0.55–0.6 s/token (~1.8 tok/s) — a model ~1.7× the device RAM, lossless. That 4000 MiB is a sweep point from the benchmark protocol, not the app default: the app ships a fixed 2000 MiB expert cache. See ../../docs/benchmark-method.md for the full procedure and the cache/thread sweep.

How Settings are organised

Each category shows the recommended configuration first and folds everything else into a collapsed Experimental group: the levers measured on one device, measured once, or still owed a measurement. They ship in the release build deliberately, because testing them on hardware other than the one test phone is what this app is for.

Descriptions in the UI say what a setting does, without measured figures or flag names, because a number needs the device, the model and the day beside it to be worth anything. The mapping to the CLI flags and the evidence behind each one lives in the docs, and the metrics screen keeps the flag names so a reading there can be matched against a CSV column.

Two worth knowing before you turn them on:

"Ask the next layer what it wants" (--predict-prefetch) predicts each layer's experts one layer early and fetches or retains what the prediction names. It is markedly more accurate than the previous-token guess it replaces, and a better guess still did not buy throughput: in thermally matched pairs the read-ahead lost, because the flash is already saturated. See ../../docs/expert-prediction.md before drawing conclusions from a run.
"Decide the experts early" (--route-ahead) commits each layer's routing before that layer runs, so the reads can never be wasted. It changes the reply, and it is refused alongside guessing ahead. See ../../docs/route-ahead.md.
"Stream row-gathered tables" (--row-stream) serves the token embedding table out of flash instead of RAM. Lossless, and which tables it applies to is read off the model's own graph, so on a model where none qualify it does nothing. See ../../docs/row-gathered-tables.md.
"Release the model mapping" (--release-mmap) hands the model file's mapping back to the kernel once every weight has been copied into the app's own memory, so it needs Dense weights on Anon or Pinned and is disabled otherwise. Lossless. The mechanism that makes it worth +46% on a Windows desktop does not exist here — on f2fs the read lanes measure the same either way — but keeping a 20 GB mapping registered costs a kernel under memory pressure, and dropping it took ~9% off CPU per token. Two 48-token cells on a device that spreads 20%: a direction, not a number, which is why it is off by default.
"Prefer cached experts" (--expert-substitute) steers each routing toward experts already in RAM, so the same number of experts runs but fewer are read from flash. It changes the reply, and past 20% the reply keeps reading well while the model behind it is much worse: judge it on answers you can check. See ../../docs/cache-aware-substitution.md.

11 KiB Raw Blame History Unescape Escape