mirror of
https://github.com/Helldez/BigMoeOnEdge.git
synced 2026-10-03 03:25:42 +00:00
7eff9ee47b
Compute buffers are reserved for the worst-case graph, and this engine had always set n_ubatch = n_ctx so that any fitting prompt prefills in one pass. That quietly ties RESIDENT MEMORY to the context rather than to the work: 320 MiB reserved at n_ctx 2048, falling to 80 MiB at 512, scaling exactly with the context and nothing else. On an engine whose entire problem is that the expert cache and the dense weights compete for the same RAM, that is a real budget and it was invisible. Decode is unaffected -- a decode graph is one token wide whatever this says -- so the cost is prefill throughput, which processes a long prompt in more, smaller passes. Default 0 keeps the previous behaviour exactly. validate() rejects a negative value and one above n_ctx: reserving for a batch that cannot arrive is the inverse of what the knob is for. Found while chasing what looked like a catastrophic slowdown and turned out to be this reservation pushing an already-tight system into reclaim at nearly 6 000 major faults per token, none of them the workload's fault. That happened during GPU work, where the reservation was largest, but the coupling it exposed is the engine's own and applies to every configuration. Salvaged here because the GPU offload it was found alongside measured negative and was closed (#104). Also fixed, all found alongside and unrelated to each other: * A stray % in the --dense-weights help text was read by printf as a conversion specifier, so that line printed garbage and read past the argument list. * scripts/build-android.ps1 judged cmake by whether it wrote to stderr rather than by its exit code, which broke it under Windows PowerShell 5.1 -- the NDK toolchain prints progress to stderr and 5.1 turns that into a terminating error under ErrorActionPreference = Stop. * .gitignore now covers third_party/opencl-sdk/, 5.3 MB of Khronos headers and an ICD loader that fetch-opencl-android.ps1 drops into the tree and nothing was ignoring.
54 lines
1.1 KiB
Text
54 lines
1.1 KiB
Text
# Build output
|
|
/build*/
|
|
/out/
|
|
cmake-build-*/
|
|
|
|
# OpenCL headers + ICD loader, fetched by scripts/fetch-opencl-android.ps1 for the optional
|
|
# GPU build. Third-party sources we do not vendor: a build prerequisite, not our code.
|
|
/third_party/opencl-sdk/
|
|
|
|
# Models and large artifacts (never commit weights)
|
|
*.gguf
|
|
*.bin
|
|
*.apk
|
|
|
|
# Local test scratch (downloaded models/APKs, container build scripts/logs)
|
|
/.models/
|
|
/.apk/
|
|
/.bench/
|
|
# Dated bench working dirs (raw CSV/log/metrics before curation into docs/bench-data/)
|
|
/.bench-*/
|
|
*.safetensors
|
|
/*.log
|
|
# Scratch notes and raw run samples at the root. Curated CSVs live in docs/bench-data/.
|
|
/*.csv
|
|
/NEXT-SESSION-PROMPT.md
|
|
|
|
# Android
|
|
examples/android/.gradle/
|
|
examples/android/build/
|
|
examples/android/app/build/
|
|
examples/android/local.properties
|
|
examples/android/.idea/
|
|
examples/android/app/src/main/jniLibs/**/*.so
|
|
examples/android/captures/
|
|
|
|
# IDE / editor
|
|
.vscode/
|
|
.idea/
|
|
*.swp
|
|
*~
|
|
|
|
# OS
|
|
.DS_Store
|
|
Thumbs.db
|
|
|
|
# Python
|
|
__pycache__/
|
|
*.pyc
|
|
.venv/
|
|
|
|
# Android release signing — secrets, never commit (repo is written to be opened)
|
|
*.jks
|
|
*.keystore
|
|
keystore.properties
|