BigMoeOnEdge/docs/android-memory.md

Android memory reclaim, from the engine's point of view

Why a >RAM streaming engine loses its working set on an Android phone, which levers exist (almost none), and which numbers to check before believing any theory about it. Measured on a OnePlus 15R (Android 16, kernel 6.12, 11.4 GB RAM, 12 GB swap, not rooted) while running gpt-oss-120b; the mechanisms are generic, the constants are this device's.

Everything below is either read from /proc on the device or read from vendor GPL source. Where a value could not be read live, it says so — several of the load-bearing ones are root-only.

The shape of the problem

Reclaim is not an event to recover from, it is a standing condition. It happens while the engine decodes, not only while it idles. Through one in-app turn, sampled every 4 s:

t=4    RssAnon 2.02 GB   swap 463 MB     <- full cache resident
t=12   RssAnon 1.86 GB   swap 620 MB     <- the kernel is taking it
t=20   RssAnon 1.57 GB   swap 917 MB
t=44   RssAnon 1.46 GB   swap 1.02 GB
t=68   RssAnon 2.02 GB   swap 468 MB     <- the engine faults it back
t=88   RssAnon 1.88 GB   swap 596 MB     <- and loses it again

That oscillation, not any single reclaim, is what costs the turn.

Free memory, and why 95 MB is not a crisis

Watermarks act on MemFree — physically empty page frames — not on MemAvailable. Linux keeps MemFree near zero deliberately: unused RAM is wasted RAM, so everything spare becomes page cache. A healthy phone therefore lives at its watermarks, and kswapd waking is normal, not pathological.

Zone Normal on this device (/proc/zoneinfo), against 11.4 GB of RAM:

watermark	pages	bytes	effect
min	5,792	22.6 MiB	below: direct reclaim — the allocating thread reclaims, synchronously
low	24,262	94.8 MiB	below: kswapd wakes
high	42,732	166.9 MiB	kswapd reclaims up to here, then stops

Both gaps are 18,470 pages; since gap = max(min/4, managed × wsf / 10000) with managed = 2,841,569, the implied watermark_scale_factor ≈ 65 against a stock default of 10 — kswapd here is 6.5× more hysteretic than stock, chasing 167 MiB free.

The signal that something is actually wrong is not "kswapd is running". It is allocstall_* and pgsteal_direct in /proc/vmstat: direct reclaim is billed to your thread as uninterruptible sleep, inside llama_decode, where no tok/s counter can see it. System-wide here: pgsteal_kswapd 270,019,486 vs pgsteal_direct 80,282,940 — 23% of all steals are direct.

Two kinds of memory, and why ours is the expensive kind

	reclaim cost	restore cost
clean file-backed (mmap'd weights, page cache)	free — dropped, no I/O	re-read from the file
anonymous (our expert cache buffers)	compress + write to zram, maybe writeback to NAND	read + decompress

The expert cache is a copy of bytes that already exist in the gguf, held in the one form the kernel must write out to reclaim. pswpout on this device: 36.8 M pages ≈ 140 GB written since boot. Worse, Q4_K_M weights are high-entropy, so zram stores them near-uncompressed: the swapout burns CPU both ways and frees almost nothing. (Inference — /sys/block/zram0/mm_stat is root-only; check huge_pages vs pages_stored with root to settle it.)

This is the unnoticed cost of the O_DIRECT design. O_DIRECT buys large sequential reads instead of a 4 KiB demand-fault storm — real and worth having — but it also converts free evictions into paid ones. Any future design should weigh both halves.

The hit rate is what decides whether the kernel keeps your cache

The LRU promotes a page from inactive to active only on a second reference — and a cache hit is a second reference. So the cache hit rate is not merely "flash reads avoided": it is literally the signal the kernel uses to decide whether your memory deserves to stay.

model	file	cache	hit	tok/s	what the kernel concludes
Qwen3-30B-A3B	18.5 GB	4000 MiB	77.1%	4.57	touched twice → active → protected
Gemma-4-26B-A4B	17.0 GB	4000 MiB	82.9%	4.76	same
gpt-oss-120b	58.5 GB	3000 MiB	13.4%	1.15	touched once → inactive → swapped

And the hit rate is geometry, not tuning:

gpt-oss-120b: 56.8 GB experts / (36 layers × 128) = 12.3 MB per expert slot
              3000 MiB cache covers  5.2% of the bank  ->  13% hit
Qwen3-30B:    16.3 GB experts / (48 layers × 128) =  2.6 MB per expert slot
              4000 MiB cache covers 24.5% of the bank  ->  77% hit

gpt-oss's bank is 3.5× larger with 4.6× fatter slices, so the same cache covers 5× less of it. Cover 5% of a 128-expert bank with near-uniform routing and you hit ~13%. On that model the kernel is right: it is being asked to hold 2 GB of pages the engine itself touches once. A cache below some coverage ratio does not merely fail to pay — it actively harms, by evicting the dense weights it competes with and by driving the allocation churn that wakes kswapd.

The vendor tilts the field against anonymous memory

/dev/memcg/.../memory.swappiness reads 100, and that number is a decoy. The module oplus_bsp_zram_opt is verified loaded (/proc/modules) and hooks android_vh_set_swappiness, overriding the sysctl inside the reclaim path (zram_opt.c, GPL, © Oplus):

static int g_direct_swappiness = 60;
static int g_swappiness = 160;

static void zo_set_swappiness(void *data, int *swappiness) {
    if (current_is_kswapd()) *swappiness = g_swappiness;      /* 160 */
    else                     *swappiness = g_direct_swappiness; /* 60 */
}
static void zo_set_inactive_ratio(void *data, unsigned long *inactive_ratio, bool file) {
    if (file) *inactive_ratio = min(2UL, *inactive_ratio);
    else      *inactive_ratio = 1;
}

Swappiness is a 0–200 ratio of the assumed relative I/O cost of swap vs filesystem paging (see vm.rst); 100 is parity. So kswapd scans anon with 4× the pressure of file cache, which is why 2 GB of our cache goes to zram while 4.9 GB of page cache stays. inactive_ratio = 1 for anon additionally forces it ~50% deactivated — parked, by construction, on the list reclaim eats first. Corroborated live: Active(anon) 459 MB vs Inactive(anon) 3,123 MB (87% inactive).

⚠️ Load-bearing and unverified. The 160 comes from OPLUS's 5.4 source; the module is verified loaded on this 6.12 device, but /sys/module/oplus_bsp_zram_opt/parameters/vm_swappiness is root-only and was not read. High-confidence inference, not a measurement. With root, one command settles it.

Note the asymmetry: kswapd (160) prefers stealing our anon, while our own direct-reclaim stalls (60) steal file cache — we evict our own dense weights when we allocate too fast.

Nothing protects anonymous memory. Every lever, and why it is closed

lever	verdict here
mlock / mlockall / MAP_LOCKED	dead — RLIMIT_MEMLOCK = 65536 bytes, soft and hard (/proc/<pid>/limits; AOSP init.rc sets it). 64 KB of a 2 GB cache = 0.003%
setrlimit to raise it	dead — raising the hard limit needs CAP_SYS_RESOURCE; an app has neither that nor CAP_IPC_LOCK
cgroup memory.min / memory.low	absent — memcg here is v1 at /dev/memcg (0700 root:system, limit_in_bytes unlimited); those knobs are v2-only, and cgroup2 has no memory controller
memory.reclaim	v2-only, and unused by AOSP anyway
/proc/<pid>/reclaim	does not exist — a Qualcomm patch, never mainline; AOSP moved to process_madvise() in Android 12
MGLRU knobs	CONFIG_LRU_GEN=y and default-enabled, but /sys/kernel/mm/lru_gen/enabled = 0x0000 — vendor-disabled at runtime. Classic active/inactive LRU applies
MADV_HUGEPAGE	no-op — THP is [never]
oom_score_adj, foreground service	affect kill selection only. Kernel reclaim is LRU-based and process-agnostic; it never consults them
onTrimMemory	notification only, and largely deprecated — apps are not notified of most levels since API 34
Memory Advice API	deprecated; only warns, never keeps anything resident
MAP_POPULATE	prefaults, does not pin. Load speed only
MADV_COLD / MADV_PAGEOUT (5.4+)	the one real lever — unprivileged. Can't stop reclaim, but chooses its victims: volunteer your own cold tail so kswapd finds cheap targets and you cut direct-reclaim stalls
dma-buf via AHardwareBuffer	the one lever that works — those pages stay pinned because a device may DMA from them at any time, and no capability is needed. Full anonymous-memory read speed, and +17.9% decode with the dense weights in it (2026-07-21). Capped at 2047 MiB per buffer: AHardwareBuffer_lock fails with EINVAL at 2^31 bytes even though allocation reaches the 4 GiB format cap. Shipped as --dense-weights ahwb, default off — see below

lmkd never reclaims — it only kills — and here it is silent by design. It early-returns while SwapFree >= swap_free_low_percentage (10%) of total; we sit at 70–79%. The 12 GB swap is exactly why nothing protects us: from lmkd's view, a machine with free swap is a healthy machine. The vendor's Osense/"nirvana" chases purposeFreeMB=390 but kills type=cached apps only and cannot touch a top-app process; its kill cnt=0 log lines are the sound of the system running out of other victims before kswapd comes for us.

For contrast: iOS exposes os_proc_available_memory() (a real per-process budget) and an entitlement to raise it. Android has no equivalent of either. No on-device runtime surveyed (llama.cpp, MLC, MediaPipe/LiteRT, ONNX Runtime, ExecuTorch) pins weights on Android or publishes a "stay under X% of RAM" rule; llama.cpp's --mlock fails here for the reason above, and ExecuTorch explicitly maps Android to NoMlock.

The dma-buf exception, and what it does not solve

Everything above is about memory the kernel is allowed to take. There is one allocation it is not: a dma-buf, whose pages are pinned for the lifetime of the buffer because a device may DMA from them at any moment. An app reaches one through AHardwareBuffer with format BLOB — no capability, no root, and it sidesteps RLIMIT_MEMLOCK entirely, since it is not mlock at all.

Measured properties on the test device (data):

property	value
throughput vs anon, long generation	+17.9% (see below)
read bandwidth vs anonymous memory	1.00× — 30.4 GiB/s and 45.2 GiB/s on the two clusters, 59.0 at 4 threads, all matching anon within 0.5%
CPU_READ_OFTEN vs CPU_READ_RARELY	no difference; the hint does not have to be coaxed
max usable size	2047 MiB per buffer — AHardwareBuffer_lock returns EINVAL at 2^31 bytes, a signed-32-bit boundary, though allocation reaches the 4 GiB format cap
allocation cost	~7–11 GiB/s, i.e. a few hundred ms once, at load

That the bandwidth is ordinary is the load-bearing result, because it was the plausible way for the idea to be dead on arrival: gralloc chooses per allocation whether a buffer is CPU-cacheable, and an uncached mapping would read at or below flash bandwidth — making pinned weights slower than refaulting them.

--dense-weights ahwb puts the dense weights there. It is anon with one substitution — the buffer comes from pio::pinned_alloc instead of the heap — so an A/B against anon moves a single variable, and it refuses to start where the platform has no such allocation rather than falling back, which would let the comparison quietly become a mode against itself.

Measured, in-app, on a long generation (2026-07-21):

	anon	ahwb
decode tok/s	2.588	3.053 (+17.9%, CIs disjoint)
dense_resident_frac	0.848	1.000
majflt / token	265	257 — equal
compute_ms	298	241
io_ms, stall_ms, cache hit %	-	all unchanged
swap	562 MiB	294 MiB

Reclaim-exemption holds: 1.000 exactly, minimum included, in every pinned run.

But the win is not the one that was predicted, and the correction matters more than the win. Major faults are equal. anon already keeps the dense weights off the flash — what it does not prevent is the kernel taking ~15% of them into zram. A page reclaimed to zram is not a major fault when touched again; it is a minor fault plus a decompression. That cost appears in no I/O counter and no fault counter, so it lands in compute_ms — which is a residual, not a measurement of arithmetic. Hence the entire delta showing up there while everything else stays flat.

So: anon protects the dense weights from flash; ahwb also protects them from zram, and the premium anon was quietly paying is worth ~18%. The feared trade — pinned memory starving the expert cache — does not appear at this size: the dense set (~1.6 GiB) is small next to a 3000 MiB cache budget, and the hit rate is identical to the decimal.

Two caveats that keep this default off: in the decisive pair ahwb ran first and an order effect cannot be excluded (the reversed pair is owed), and it is one device, one model, one config.

The general lesson outlives the flag: compute_ms has been absorbing zram decompression all along, so any earlier conclusion of the form "this regime is compute-bound" deserves re-examination.

A dense tensor larger than RAM

Every mode above assumes the dense set fits. qwen4exp (Qwen3.8-Flash-Next) breaks that with one tensor: per_layer_token_embd, a 51B-parameter n-gram embedding table, ~28.8 GB at IQ4_NL, that the graph gathers sixteen rows from per token. No mode can hold it, and each would fail differently: anon asks for an allocation the OS kills the process over, ahwb hits the 2047 MiB dma-buf ceiling, warm would sweep 28.8 GB through a page cache that evicts it as it goes, and the residency sensor would report a dense set that can never be resident. So the engine applies one rule before any mode: a dense tensor larger than the kernel's MemAvailable is held back. It stays mmap'd, leaves the warm sweep and the sensor, and the auto cache budget does not reserve for a conversion that will not happen. One stderr line names what was held back.

The held-back mapping also gets MADV_RANDOM. Default readahead brings in a window of hundreds of KiB per fault, right for a weight read whole and wrong by two orders of magnitude for a row gather: the neighbours are never touched and they take page cache the expert cache is competing for. With the advice a fault maps one page. Per token that is sixteen 4 KiB reads, under 1 % of a decode step; the cost that remains is prefill, where every prompt token faults its rows on the compute thread. Not measured on a device yet.

The size bound is deliberate and the access shape is not the criterion: a row-gathered table that fits keeps its mode, because demand-faulting it costs more in prefill than reading it once at load (token_embd left mmap'd measured −16 % decode, PR #135).

Why restoring reclaimed pages cannot win

MADV_WILLNEED on anon does swap them back in — but read_swap_cache_async puts them on the inactive list (folio_add_lru, mm/swap_state.c); nothing marks them accessed. Only a real second touch promotes to active. So a bulk restore hands the kernel a pile of pages at the front of the eviction queue, and at swappiness 160 with inactive_ratio = 1 the next pass comes almost immediately.

Measured: restoring 1.76 GB took 6.4 s and the kernel had it back in 8 s, with the turn still at 0.3 tok/s. That 8 s is a property of the LRU, not of the code.

The corollary is the useful part: a restore is only worth anything if the pages get hit right after — which makes its value a function of the hit rate, and predicts it helps exactly the high-hit-rate models it was not designed for.

Checklist before believing any theory here

• Read MemFree and MemAvailable and Cached — free ≈ 100 MB is normal, not a symptom.
• Split RSS: /proc/<pid>/status → RssAnon vs RssFile vs VmSwap. VmSwap sees only the anonymous half; file-backed reclaim is invisible to it, and it was 27× larger here (workingset_refault_file 114.8 M vs workingset_refault_anon 4.2 M).
• VmHWM tells you whether a process ever held what you think it held — it catches "this is a fresh process", which looks identical to "this process was stripped" in a bare RSS sample.
• Rates, not counters: /proc/vmstat totals are since boot. Delta them over a window.
• Measure in the app. The app's engine drops 3.5 GiB → 3.5 MiB within ~5 s of a reply; an adb session took 4 minutes to lose half that, and a second adb run barely lost anything. Every bench script in this repo is single-shot and never idles, so none of them can see this class of bug. Seeing it takes two turns with a timed idle gap between them, sampling /proc through the gap.
• Beware pgrep -f <name>: the shell running the command matches its own command line. A 3.8 MB "engine" is your own sh.