vrr/unsloth
2
0
Fork 0
mirror of https://github.com/unslothai/unsloth.git synced 2026-08-23 07:44:06 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 29

studio: refuse a gguf that cannot fit in free vram plus available ram (#8883)

Browse source 
* studio: refuse a gguf whose cpu offload is larger than available ram

`--fit on` places what it can in VRAM and spills the rest into host memory with
no ceiling of its own, and nothing upstream of llama-server prices that spill.
The weights are memory-mapped, so an oversized remainder does not fail an
allocation: the kernel evicts and re-reads the mapping every token until the
machine stops responding, and systemd-oomd then kills the whole app slice.

llama_cpp.py already refuses this on Apple unified memory and on AMD APUs, but
_amd_apu_wants_unified_memory is false for a discrete card, so CUDA and ROCm
hosts had no guard. _host_offload_shortfall_message adds the same ceiling,
priced on the spill rather than on the whole model.

* price the kv floor, page-locked mappings and vulkan igpus in the host guard

Three cases where the host-offload guard read a footprint the launch does not
produce.

Manual + Auto layers omits -c and leaves effective_ctx at 0, so _kv_bytes(0)
returned zero while the launch still floors --fit-ctx at 8192 and allocates that
KV. Price the floor, now a shared _AUTO_FIT_MIN_CTX so the guard and the emitted
flag cannot drift.

The Model Memory page-lock setting emits mmap+mlock for a --fit on launch, which
pins the whole mapping in host RAM including the layers copied to the GPU.
Subtracting free VRAM there under-prices by the GPU-resident portion, so charge
the whole mapping when should_mlock() is set.

A Vulkan iGPU reports shared system RAM as its free VRAM and total 0. Subtracting
it and then charging the remainder against that same RAM counts the pool twice,
so a 20 GB model on a 14 GB host read as a 6 GB requirement. Exclude a total-0
Vulkan device from the VRAM offset.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* scope the host guard to the placement the launch actually builds

Manual mode with an explicit layer count emits --gpu-layers N --fit off, but that
only clears use_fit while the command is built, long after the guard reads the
seeded True. At 0 layers the GPUs hold nothing, so subtracting their free VRAM
under-priced the whole model; a positive count cannot be sized per layer here, so
the guard abstains instead of pricing a placement it cannot compute.

--no-kv-offload keeps the whole KV cache in host RAM, where it is not fungible
with weights the GPUs can take. Charge it after the VRAM subtraction rather than
inside it, so 12 GB of weights that fit the card no longer absorb an 8 GB
CPU-resident cache.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* keep host-pinned bytes out of the vram subtraction, and gate the apu check

On a mixed ROCm host the arch gate can leave gpu_indices None while pinning the
launch to a discrete card. _amd_apu_wants_unified_memory then saw the whole
physical set, read the unsupported APU as unified memory and skipped this guard,
while the APU guard above had already cleared its own refusal for the same
placement. Ask about the devices the launch will pin instead.

A drafter pinned with --spec-draft-ngl 0 leaves the VRAM budget but keeps its
weights in host RAM, and nothing was charging them. Size it before the CPU
nulling and add it to the host figure.

Both need the spill clamped at zero before host-pinned bytes are added, or spare
VRAM absorbs them: a 4 GB target on a 14 GB card was cancelling an 8 GB CPU
drafter to a negative footprint.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* resolve the effective memory state and the cgroup ceiling in the host guard

With both Model Memory toggles off, apply_model_memory_policy preserves a
caller's --mlock or --no-mmap, so the launch keeps a full host copy while
should_mlock() reports false. Ask resolve_effective_memory_state, which already
answers for argv and env together, instead of the setting alone.

A --device naming no GPU leaves the child nothing to offload onto, so its free
VRAM is credit the launch never gets. _device_selection_is_cpu now zeroes the
offset, alongside the paravirtual and arch-gate cases already there.

A CPU-pinned drafter's KV cache was dropped with its weights when
_mtp_draft_for_budget is nulled; size both before that and charge them to the
host.

_available_system_memory_mib now caps its reading by _cgroup_free_bytes. Inside a
container or systemd scope the binding memory.max is what the OOM killer
enforces, and the host MemAvailable can be tens of GiB above it. The APU guard
reads the same helper and gains the ceiling too.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* close the gaps the previous round's fixes left open

A CPU device override now triggers the host check on its own. A successful fit
clears use_fit, so gating the whole guard on it skipped a retained --device none
that still loads the model on the CPU.

resolve_effective_memory_state gets the launch environment, not just argv.
scrub_memory_env keeps an inherited LLAMA_ARG_MLOCK or a reserving
LLAMA_ARG_LOAD_MODE when both toggles are off, and argv alone cannot see them.

The CPU drafter's KV is sized at the fit floor, matching the main KV term:
Manual + Auto leaves the context at 0, where _mtp_draft_kv_bytes returns None
while the launch still emits --fit-ctx.

The cgroup reading goes through _shared_policy instead of importing
unsloth.dataset_num_proc directly, which would run the package __init__ and load
the model stack in the middle of a llama.cpp load.

* stop the guard refusing loads that fit, and price the fit context it will run at

Three ways the previous rounds' fixes over-charged a viable load.

An explicit gpu_ids pick strips the device arguments in the command builder, so a
stale --device none no longer zeroes the VRAM credit for a launch that does use
the GPUs.

Page-locking duplicates the weight mapping in host RAM, not the KV cache. The
mlock branch charged the whole combined footprint, so a partially offloaded model
could be refused over a KV cache that never leaves the card. Charge the mapping
and let the GPU pool still cover the GPU-resident KV and MTP terms.

"Don't reserve system RAM" strips --mlock and --no-mmap in
apply_model_memory_policy, so resolving the raw flags claimed a full host copy the
child never keeps. Read both settings and let no-reserve win, as should_mlock does.

Alongside those, the guard context now honours a pass-through --fit-ctx instead of
assuming the 8192 floor, the MTP reserve is recomputed at that same context, and
the CPU drafter's KV is sized with the cache types, SWA, unified and ubatch
options the GPU-side call already passes.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* resolve the effective layer placement, and keep companion vram out of the charge

An -ngl 0 in the extras is appended after Studio's own flags and wins, so it now
counts as a CPU-only placement alongside --device none. A device override also
beats the positive-manual exemption: that exemption exists because a partial
split cannot be sized, which stops being true once the launch reaches no GPU.

Manual mode at 0 layers leaves the weights in host RAM but a companion-visible
launch still holds its KV and drafter on the card. Zeroing the whole VRAM credit
there charged those to RAM and could refuse a CPU model that fits, so it now
shares the page-lock treatment: the weight mapping goes to host, the GPU pool
still covers the GPU-resident terms.

The auto-fit floor is priced only when the build supports --fit-ctx, since
_ctx_integrity_flags omits the flag otherwise and the fitter may choose less.

_available_system_memory_mib adds back the cgroup's reclaimable inactive_file.
memory.current charges the GGUF's own page cache, which the kernel reclaims
rather than OOM on, so subtracting it counted those pages twice.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* run the host guard on the finished command instead of mid-fit state

The check sat inside the GPU-fit block, where none of its inputs were final. use_fit
is not settled until the command builder emits --gpu-layers/--fit off,
apply_model_memory_policy decides the mlock and load-mode flags later still, and
_cpu_only_zero_offload masks the child's GPUs after that. Every input it read could
be overwritten by code below it, and each round of review found another placement it
had guessed wrong.

_launch_host_shortfall_message now runs after the argv and env are built and reads
both: the model, projector and drafter paths (so the sizes come from the files the
child opens), -c and --fit-ctx for the context the caches are allocated at, -ngl and
--device for placement, the visibility mask for whether a GPU exists at all, and
resolve_effective_memory_state over the finished command for the memory mode.
_visible_gpu_free_mib credits only the devices the mask and any device pin leave
reachable.

That subsumes most of what the previous rounds added, so this deletes the seeded
use_fit workaround, the manual-layer and CPU-override predicates, the guard-context
recompute with its --fit-ctx capability gate, the deferred CPU-drafter sizing and
_extra_args_fit_ctx. Four behaviours fall out of the new placement rather than
needing their own patches: a device pin that narrows the pool, a full host-memory
mode on a proven GPU fit, an extras -ngl 0, and a companion-free manual zero whose
KV cache follows the weights into RAM.

All 23 existing regression tests keep their behaviour, re-pointed at the new call
site. The placement harness stubs the preflight off by default, as it does the APU
one, since it now runs on every launch.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* abstain on an unprobed gpu pool, and read one visibility mask

Two ways the relocated guard could refuse a load that fits.

An empty detected pool means the GPU probe threw, not that the child has no GPU:
the fallback still builds a --fit on command for whatever it discovers. Pricing
that as zero VRAM charged the whole model to host. Only a visibility mask or a
device pin is authoritative about there being nothing to offload to, so an
unsized pool now abstains.

The visibility lookup intersected every mask it found. ROCr names physical
devices while a prefer_rocr pin leaves the CUDA ordinals relative to the filtered
set, so intersecting their numeric strings compares different spaces and can
empty the pool. Take the first mask instead, ROCr first since it is the physical
one.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rewrite the guard as a weights-only floor

Nine review rounds converged on the wrong shape. The check re-derived about
fifteen placement quantities from argv (KV geometry, cache types, projector and
drafter placement, mlock and load-mode state, visibility masks, device pins,
cgroup ceilings), and every derivation was its own completeness surface. Round 9
returned ten items, more than any round before it, and none of them were the
ordering defect the previous relocation fixed. That list does not terminate
against llama.cpp's placement surface.

Price the weights alone against the whole free VRAM pool instead. That is a
strict lower bound on what the launch must hold: KV, projector, drafter and
compute buffers only add to it, and a layer or device pin only narrows the VRAM
actually reachable. Every omitted term moves the estimate down, so no missing
term can turn an allowed load into a refused one, and there is no placement
model to keep in step.

The field case still refuses: a 13.3 GB GGUF with 4877 MiB free on the card
needs about 8.5 GB of host RAM, which a 10 GB host cannot hold.

Drops _visible_gpu_free_mib, _cgroup_reclaimable_file_bytes, the nine argv
constants and the cgroup coupling in _available_system_memory_mib, which is back
to what main has.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* price a launch that masks the child off every gpu

The empty-pool abstention treated two different states as one. A pool that came
back empty means the probe threw, so abstaining is right. But the arch gate
(llama_cpp.py:13542) empties the pool deliberately when the installed build has
no kernels for any device on the host, and then masks every card from the child
at :15888. That is a known CPU placement, not an unknown one, and the whole
model runs from RAM with no preflight at all.

Manual zero-offload reaches the same mask with a pool the planner did probe, so
it was crediting VRAM the child cannot allocate on.

Pass the launch's own masked-off state, the disjunction that writes the -1 mask,
and take no VRAM credit when it is set. Reported by Codex on the arch-gate path.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* take no vram credit on a build that ships no gpu backend

_backend_lacks_gpu_lib identifies a split-library llama-server whose lib
directory carries cpu or base and none of cuda, hip or vulkan. That build cannot
offload, but the hardware probe still enumerates the host's cards, so the guard
was subtracting VRAM the child can never allocate on: a 20 GiB model with 16 GiB
free read as a 4 GiB spill an 8 GiB host holds, while the child placed all
20 GiB in RAM.

Fold it into the same flag the arch gate and manual zero-offload already set,
renamed to child_has_no_gpu since a CPU-only build is not masked off anything.
The helper fails open on a static or unrecognised layout, so a custom GPU build
keeps its credit and cannot be refused.

Reported by Codex.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* price a host whose gpu enumeration found nothing

_get_gpu_memory returns [] for "no supported GPU is reachable", not only for a
probe that raised: the torch fallback exists so callers can read [] as "no GPU"
and drop to CPU. The abstention treated both the same, so an ordinary CPU-only
machine running a GPU-capable build skipped the guard while the whole GGUF went
to host RAM.

Record whether the enumeration ran and let the launch vouch for an empty pool.
A probe that throws leaves the flag unset and still abstains. _detected_gpus is
captured before manual mode empties the planner pool, so a GPU host cannot reach
this with an empty list.

Drops test_an_unprobed_gpu_pool_abstains_rather_than_refusing, which asserted the
old reading of an empty pool. test_a_failed_enumeration_still_abstains covers the
case it was written for.

Reported by Codex.

* abstain on an empty gpu pool again, and state the reserve in the refusal

_get_gpu_memory catches its own probe failures and returns [] (the torch fallback
at llama_cpp.py:6673), so a flag set on "the call did not raise" marks a host
whose every probe failed as GPU-less. The guard then priced the full model and
could refuse a load llama-server's own enumeration still places on a card. That
is a false refusal, which the floor is built never to produce, so the inference
comes out and an empty pool abstains again. A host with no GPU at all stays
uncovered, in the permissive direction the check is documented to take.

The refusal message also read as contradictory arithmetic when the spill fit in
available RAM but not inside the headroom: 7 GB refused against 8 GB available.
Name the reserve and the usable figure, rounding the need up and the usable down
so the printed pair keeps the ordering that produced the refusal.

Both reported by Codex.

* recognise every gpu backend, and abstain on an rpc launch

Two false refusals in the CPU-only signal.

_installed_ggml_backends scans for base, cpu, cuda, hip and vulkan only, so a
split-library build shipping SYCL, OpenCL, MUSA or CANN recorded no GPU backend
and _backend_lacks_gpu_lib called it CPU-only. The guard then priced the whole
GGUF against RAM and refused a load the accelerator can hold.
_binary_ships_no_gpu_backend reads the same lib directory through
_GGML_GPU_BACKEND_RE, which already knows all nine names, and still requires a
proven split-library layout so a static build keeps its credit. The narrower
helper is left alone: its other caller gates a device pin, not a refusal.

--rpc places layers on remote devices, and the pass-through is not stripped, so
sizing a launch against local capacity alone refused a viable distributed run.
Abstain when it carries a value.

Both reported by Codex.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* read the environment twins, and drop advice the guard cannot act on

Both remaining false refusals were the environment form of something already
read from argv or from disk.

llama.cpp accepts RPC servers through LLAMA_ARG_RPC as well as --rpc, so a
distributed launch configured that way was still priced against local capacity.
GGML_BACKEND_PATH points the child at backend plugins outside the directory
beside the executable, so a cpu-only layout there is no longer proof the child
cannot offload. The guard now takes the finished child environment and both
answer by abstaining.

The refusal also told users to lower the context length. This prices weights
only and omits the KV cache entirely, so context changes none of its inputs and
the advice sent them through a reload that fails identically.

All three reported by Codex.

* charge a paravirtual metal launch

_paravirtual_cpu_forced rewrites the finished command to --gpu-layers 0 --device
none on a virtualised Apple GPU, and Metal hosts leave the probed pool empty, so
the guard read the placement as inconclusive and abstained. It is the fourth
state the launch already knows reaches no card, so it belongs beside the other
three rather than falling through to the empty-pool abstention.

Reported by Codex.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* abstain on a gpu-less host, and give the refusal a way through

child_has_no_gpu took _binary_ships_no_gpu_backend, which reads the build layout
and nothing about the hardware. Studio installs a CPU-only prebuilt on a host with
no GPU, so that host both probes an empty pool and reports a build shipping no GPU
backend, and the arm defeated the empty-pool abstention: a 7.5 GB GGUF with 9 GB
MemAvailable was refused, and the message blamed GPU memory on a machine that has
none. _cpu_only_zero_offload has the same shape, since gpu_memory_mode and
gpu_layers are read from the load request rather than from a probe. Both now
require a probed pool. On an empty pool free_vram_mib sums to zero either way, so
the gate moves the abstention only, never the charge. _arch_gate_forced_cpu and
_paravirtual_cpu_forced each name a device that was seen and ruled out, so they
stay ungated.

_get_gpu_memory returns [] on every Apple Silicon Mac, so before this every Mac
Manual zero-layer load of a GGUF over MemAvailable - 2 GiB was refused against a
Metal device that was never enumerated.

The refusal was also unconditional on a path the picker still offers, where
classifyGgufFit labels the variant "oom" and leaves the row selectable, and no
load field carries a force. UNSLOTH_ALLOW_HOST_OFFLOAD=1 now abstains and the
refusal names it, the same shape as UNSLOTH_ALLOW_PARAVIRTUAL_METAL.

That guard reads the finished argv, so it cannot answer until the route has taken
acquire_for(CHAT), evicted a resident Images/Video pipeline, cancelled the running
generations and unloaded the previous model. Picking an oversized variant
therefore tore down a working session to deliver an error.
host_offload_refusal_for_intent asks the same function ahead of that handoff,
crediting the ungated probe that every narrowing the launch applies only shrinks,
so it refuses a strict subset and fails open into the launch copy. Same placement
and same reason as non_chat_gguf_refusal_for_intent.

Reported by oobabooga.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* reprice the spill when the arch-crash retry narrows the pool

The guard runs once, against the aggregate _detected_gpus pool. The HIP
kernel-image recovery then drops the crashed device and respawns masked onto
_remaining, so a card that supplied most of the credited VRAM leaves the
narrowed launch spilling far more into RAM than the preflight allowed. On two
discrete cards holding 40000 and 4000 MiB free, a 30 GB model clears the
aggregate check and then has about 26 GB to place on a 20 GB host.

Same gap the APU preflight above it already closes for the unified-memory
mirror, so the recheck sits beside it and reuses the same function against the
surviving rows.

Reported by Codex.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* credit the physical total in the route preflight, not free vram

The preflight probed free VRAM, which still counts the resident llama-server,
Unsloth model and Images/Video pipeline that acquire_for(CHAT) and load_model go
on to reclaim. A 13.3 GB switch on a 24 GB card holding 900 MiB free was refused
with a 400 against memory that was about to come back, so switching models on a
busy GPU stopped working.

Each device's physical total is the ceiling on what the launch can ever see, and
every narrowing the launch applies to the pool only shrinks it further, so the
preflight still charges no more than the launch copy. What survives is the pick
no reclaim can rescue, which is the one worth catching before the teardown. A
device reporting total 0, an iGPU or a MIG/vGPU line, leaves the ceiling unknown
and abstains.

Reported by Codex.

* price the route preflight against total ram, not memavailable

The VRAM side was fixed for the reclaim the handoff performs; the host side had
the same defect. A resident llama-server holds RAM through its host KV cache,
CPU-offloaded weights and locked mappings, and an active Unsloth model holds more,
all of it released by the route at inference.py:8550 and by load_model's kill of
the previous server. Reading MemAvailable before any of that charged a spill
against memory that was about to come back, so a 30 GB target with 6.7 GB on the
host was refused with a 400 on a machine holding 3 GB free and 64 GB total.

_total_system_memory_mib reads MemTotal, the ceiling MemAvailable can never
exceed, and _launch_host_shortfall_message takes it through a new avail_mib
override so the launch keeps reading what is actually available. Unreadable total
RAM abstains, as an unreadable available figure already did.

Reported by Codex.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix studio gguf ram admission accounting

* discount reclaimable cgroup cache

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
This commit is contained in:
Maheswar Kumar 2026-08-18 12:10:43 +12:00 • committed by GitHub
parent 31c42e872b
commit ee68d9e2ad
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 1056 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

375
studio/backend/core/inference/llama_cpp.py
View file

@ -3391,6 +3391,8 @@ def _report_live_llama_timings(callback, chunk) -> None:
# margin (default 1024 MiB per device). ggml reports an iGPU's "VRAM" as shared
# system RAM, so hold back the same margin rather than inventing a larger one.
_IGPU_HOST_RESERVE_MIB = 1024
_CGROUP_ROOT = "/sys/fs/cgroup"
_PROC_SELF_CGROUP = "/proc/self/cgroup"
def _apply_igpu_host_reserve_mib(free_mib: int, is_igpu: bool) -> int:
@ -6938,25 +6940,262 @@ class LlamaCppBackend:
)
return gpus
@staticmethod
def _cgroup_available_memory_mib() -> Optional[int]:
"""Memory this process can still charge to an enforcing cgroup.
``psutil`` and ``/proc/meminfo`` expose host-wide availability in many
containers. Walk the process's cgroup plus its ancestors and pair each
limit with that same directory's usage; an ancestor slice can be the
binding limit and includes sibling usage that a leaf does not see.
Supports cgroup v2 and the legacy v1 memory controller. ``None`` means
no finite readable limit, so callers retain their host reading.
"""
def _first_line(path: str) -> Optional[str]:
try:
with open(path, "r", encoding = "utf-8") as f:
return f.readline().strip()
except OSError:
return None
def _integer(raw: Optional[str], *, limit: bool = False) -> Optional[int]:
if not raw or raw == "max":
return None
try:
value = int(raw)
except ValueError:
return None
# cgroup v1 spells unlimited as a near-2^63 sentinel.
if value < 0 or (limit and value >= 1 << 60):
return None
return value
def _stat_integer(path: str, *keys: str) -> int:
"""Read the first requested byte counter present in memory.stat."""
try:
with open(path, "r", encoding = "utf-8") as f:
values = {}
for line in f:
parts = line.split()
if len(parts) == 2 and parts[0] in keys:
value = _integer(parts[1])
if value is not None:
values[parts[0]] = value
except OSError:
return 0
return next((values[key] for key in keys if key in values), 0)
def _directories(root: str, relative: Optional[str]) -> list[str]:
root = os.path.abspath(root)
current = os.path.normpath(os.path.join(root, (relative or "/").lstrip("/")))
try:
if os.path.commonpath((root, current)) != root:
return [root]
except ValueError:
return [root]
out = []
while True:
out.append(current)
if current == root:
return out
current = os.path.dirname(current)
try:
with open(_PROC_SELF_CGROUP, "r", encoding = "utf-8") as f:
lines = [line.strip() for line in f if line.strip()]
except OSError:
lines = []
remaining: list[int] = []
v2_relative = next((line[3:] for line in lines if line.startswith("0::")), None)
for directory in _directories(_CGROUP_ROOT, v2_relative):
limit = _integer(_first_line(os.path.join(directory, "memory.max")), limit = True)
if limit is None:
continue
used = _integer(_first_line(os.path.join(directory, "memory.current")))
if used is not None:
# memory.current includes file-backed cache. Inactive file pages
# are reclaimable under pressure, so price the launch against the
# cgroup working set rather than charging cached GGUF pages twice.
used = max(0, used - _stat_integer(os.path.join(directory, "memory.stat"), "inactive_file"))
remaining.append(limit if used is None else limit - used)
v1_root = os.path.join(_CGROUP_ROOT, "memory")
v1_relative = None
for line in lines:
parts = line.split(":", 2)
if len(parts) == 3 and "memory" in parts[1].split(","):
v1_relative = parts[2]
break
for directory in _directories(v1_root, v1_relative):
limit = _integer(
_first_line(os.path.join(directory, "memory.limit_in_bytes")), limit = True
)
if limit is None:
continue
used = _integer(_first_line(os.path.join(directory, "memory.usage_in_bytes")))
if used is not None:
# v1 usage is hierarchical when use_hierarchy is enabled, so its
# matching counter is total_inactive_file. Fall back to the local
# counter for non-hierarchical controllers.
used = max(
0,
used
- _stat_integer(
os.path.join(directory, "memory.stat"), "total_inactive_file", "inactive_file"
),
)
remaining.append(limit if used is None else limit - used)
return max(min(remaining), 0) // (1024 * 1024) if remaining else None
@staticmethod
def _available_system_memory_mib() -> Optional[int]:
"""Available system RAM in MiB (psutil, then /proc/meminfo), or None if
neither is readable. On a unified-memory APU this, not the ROCm-reported
VRAM, is the real ceiling: the weights load into shared system RAM."""
neither is readable, capped by this process's cgroup remainder. On a
unified-memory APU this, not the ROCm-reported VRAM, is the real ceiling:
the weights load into shared system RAM."""
available = None
try:
import psutil
return int(psutil.virtual_memory().available // (1024 * 1024))
available = int(psutil.virtual_memory().available // (1024 * 1024))
except Exception:
pass
if available is None:
try:
with open("/proc/meminfo", encoding = "utf-8") as f:
for line in f:
if line.startswith("MemAvailable:"):
available = int(line.split()[1]) // 1024 # kB -> MiB
break
except Exception:
pass
cgroup_available = LlamaCppBackend._cgroup_available_memory_mib()
if available is None:
return cgroup_available
return min(available, cgroup_available) if cgroup_available is not None else available
@staticmethod
def _total_system_memory_mib() -> Optional[int]:
"""Total system RAM in MiB (psutil, then /proc/meminfo), or None if neither is
readable. The ceiling on what ``MemAvailable`` can ever become, so a preflight that
runs before the resident model and pipeline are torn down can price against it
without charging for host memory that is about to come back."""
try:
import psutil
return int(psutil.virtual_memory().total // (1024 * 1024))
except Exception:
pass
try:
with open("/proc/meminfo", encoding = "utf-8") as f:
for line in f:
if line.startswith("MemAvailable:"):
if line.startswith("MemTotal:"):
return int(line.split()[1]) // 1024 # kB -> MiB
except Exception:
pass
return None
_ARGV_MODEL = frozenset({"-m", "--model"})
_ARGV_RPC = frozenset({"--rpc"})
@staticmethod
def _binary_ships_no_gpu_backend(
binary: Optional[str] = None, env: Optional[Mapping[str, str]] = None
) -> bool:
"""Whether a split-library build beside ``binary`` carries no GPU backend at all.
Stricter than ``_backend_lacks_gpu_lib``, which reads cuda, hip and vulkan only:
a SYCL, MUSA, CANN or OpenCL build offloads too, and pricing its weights against
host RAM would refuse a load the accelerator can hold. A static or unrecognised
layout, a directory this cannot read, and a GGML_BACKEND_PATH pointing the child
at plugins elsewhere all answer False so a GPU build keeps its VRAM credit.
"""
binary = binary or LlamaCppBackend._find_llama_server_binary()
if not binary:
return False
source = os.environ if env is None else env
if str(source.get("GGML_BACKEND_PATH", "") or "").strip():
return False
try:
files = tuple(path.name for path in _llama_lib_dir(binary).iterdir() if path.is_file())
except OSError:
return False
cpu_stem = "ggml-cpu" if sys.platform == "win32" else "libggml-cpu"
base_stem = "ggml-base" if sys.platform == "win32" else "libggml-base"
split_library = any(name.startswith((cpu_stem, base_stem)) for name in files)
return split_library and not any(_GGML_GPU_BACKEND_RE.match(name) for name in files)
def _launch_host_shortfall_message(
self,
cmd: Iterable[str],
detected_gpus: Iterable[tuple],
env: Optional[Mapping[str, str]] = None,
*,
child_has_no_gpu: bool = False,
avail_mib: Optional[int] = None,
shared_gpu_ids: Iterable[int] = (),
) -> Optional[str]:
"""Refusal when the weights alone cannot fit in free VRAM plus available RAM.
Weights only, against the whole free pool, is a strict lower bound on what the
launch must hold: the KV cache, projector, drafter and compute buffers all add
to it, and a layer or device pin only narrows the VRAM actually reachable. So
every term this leaves out moves the estimate down, never up, and no missing
term can turn an allowed load into a refused one. That is what keeps the check
a floor with no placement modelling to keep in step with llama.cpp.
Read from the finished argv, so the model path is the one the child opens. An
unsized model abstains rather than guessing, and so does an empty GPU pool,
which means the probe threw. ``child_has_no_gpu`` is the launch reporting a
placement it already knows reaches no card, rather than one it could not read:
there the whole model is host-resident and takes no VRAM credit.
UNSLOTH_ALLOW_HOST_OFFLOAD=1 abstains outright, so a user who accepts the
paging can still load a variant the picker offers.
``avail_mib`` overrides the host figure for a caller that runs before the resident
owners are released; the launch reads what is available now. ``shared_gpu_ids``
names Vulkan iGPUs whose reported free memory is the same host pool, so it must
not also be credited as dedicated VRAM.
"""
argv = [str(a) for a in cmd or ()]
if not argv:
return None
model_path = _extra_args_device(argv, self._ARGV_MODEL)
if not model_path:
return None
# the user's own opt-out, so read the real environment, not the curated child env
if os.environ.get("UNSLOTH_ALLOW_HOST_OFFLOAD", "").strip().lower() in (
"1",
"true",
"yes",
):
logger.info("UNSLOTH_ALLOW_HOST_OFFLOAD set: skipping the host-RAM preflight.")
return None
# rpc places layers on remote devices this cannot size, in either spelling
_env = os.environ if env is None else env
if (_extra_args_device(argv, self._ARGV_RPC) or "").strip() or str(
_env.get("LLAMA_ARG_RPC", "") or ""
).strip():
return None
try:
model_bytes = self._get_gguf_size_bytes(model_path)
except Exception:
return None
gpus = [] if child_has_no_gpu else list(detected_gpus or ())
# _get_gpu_memory swallows a failed probe as [], so an empty pool it did not
# vouch for cannot be told from a host that has no gpu at all
if not model_bytes or (not gpus and not child_has_no_gpu):
return None
shared = set(shared_gpu_ids or ())
free_vram_mib = sum(max(0, row[1]) for row in gpus if row[0] not in shared)
return self._host_offload_shortfall_message(
model_bytes - free_vram_mib * 1024 * 1024,
self._available_system_memory_mib() if avail_mib is None else avail_mib,
)
@staticmethod
def _apu_ram_shortfall_message(
model_size_bytes: int,
@ -6980,6 +7219,36 @@ class LlamaCppBackend:
"(on WSL, raise the memory limit in .wslconfig)."
)
@staticmethod
def _host_offload_shortfall_message(
offload_bytes: int,
avail_mib: Optional[int],
headroom_mib: int = 2048,
) -> Optional[str]:
"""On a discrete GPU, return a user-facing refusal when the part of a load
that misses VRAM cannot fit in available system RAM (else None). The spill is
mmap'd, so an oversized one thrashes the mapping instead of failing, until the
OS kills the app. Priced against free VRAM with no margin subtracted, so it
under-states the spill and only an unambiguous shortfall refuses. None avail
(unknown RAM), and anything VRAM-resident, never refuse."""
if offload_bytes <= 0 or avail_mib is None:
return None
need_mib = offload_bytes / (1024 * 1024)
if need_mib <= avail_mib - headroom_mib:
return None
# need up, usable down, so the printed pair cannot round into a tie
need_gb = math.ceil(need_mib / 1024)
usable_gb = math.floor(max(0, avail_mib - headroom_mib) / 1024)
return (
f"About {need_gb} GB of this model does not fit in GPU memory and would run "
f"from system RAM. Only about {avail_mib / 1024:.0f} GB is available and "
f"{headroom_mib / 1024:.0f} GB of that is kept free for the rest of the "
f"system, leaving about {usable_gb} GB usable. The weights are memory-mapped, "
"so the machine pages them in and out until it stops responding and the OS "
"kills the app. Use a smaller or more quantized GGUF, free memory, or set "
"UNSLOTH_ALLOW_HOST_OFFLOAD=1 to load it anyway."
)
# Skip the wait when the last kill is older than this; the driver has
# already reclaimed the prior process's allocations.
_VRAM_SETTLE_WINDOW_S: float = 15.0
@ -8642,6 +8911,60 @@ class LlamaCppBackend:
logger.debug("Non-chat GGUF preflight failed for the route: %s", e)
return None
def host_offload_refusal_for_intent(self, intent) -> Optional[str]:
"""The host-RAM verdict for a resolved load intent, or None.
``_launch_host_shortfall_message`` stays authoritative, since it reads the finished
argv, but it runs after the ROUTE has evicted a resident Images/Video pipeline via
``acquire_for(CHAT)`` and cancelled the running generations. Asking here first spares
both, exactly as the non-chat header check above does.
Both capacities are read as PHYSICAL TOTALS, not as what is free right now. The
resident llama-server, Unsloth model and Images/Video pipeline hold VRAM and, through
a host KV cache, CPU-offloaded weights and locked mappings, host RAM as well, and the
route and ``load_model`` reclaim all of it after this runs. Pricing against the free
readings would refuse a switch to a model the reclaimed machine holds easily. Each
total is the ceiling on what the launch can ever see, and every narrowing the launch
applies to the pool (the ROCm arch gate, the Vulkan discrete preference, a ``gpu_ids``
pin) only shrinks it further, so this charges no more than the launch copy and can
refuse nothing the launch would allow. What survives is the pick no reclaim can
rescue, which is the one worth catching before the teardown.
Only a local path is priced. An HF repo may not be downloaded yet, and resolving one
here would start a download the route has not committed to.
"""
try:
gguf_path = getattr(intent, "gguf_path", None)
if not gguf_path or getattr(intent, "hf_repo", None):
return None
if not Path(gguf_path).is_file():
return None
binary = self._find_llama_server_binary()
if not binary:
return None
# extras are appended after -m at launch and llama.cpp is last-wins, so read them
argv = [
binary,
"-m",
str(gguf_path),
*(str(arg) for arg in getattr(intent, "extra_args", None) or ()),
]
total_ram_mib = self._total_system_memory_mib()
if total_ram_mib is None:
return None
probed = self._get_gpu_memory(binary)
# an igpu and a MIG/vGPU line report total 0, leaving the ceiling unknown
if any(total <= 0 for _idx, _free, total in probed):
return None
return self._launch_host_shortfall_message(
argv,
[(idx, total) for idx, _free, total in probed],
avail_mib = total_ram_mib,
)
except Exception as e: # noqa: BLE001 -- a probe that failed is not a verdict
logger.debug("Host-RAM preflight failed for the route: %s", e)
return None
# Each ask is a repo listing, a cache verification and a range request, every one bounded
# at 15s, so the route HANDS OVER what it learned rather than making a slow Hub pay twice.
# Written only by the route entry point and taken once by the load it belongs to, so a
@ -13872,6 +14195,7 @@ class LlamaCppBackend:
# empty `gpus` so the speculative defaults stay GPU-aware and the
# CPU-fallback check still knows GPUs were present.
_detected_gpus: list[tuple[int, int]] = []
_shared_gpu_ids: set[int] = set()
# Set when the arch gate emptied a non-empty GPU pool, so the env
# block below masks the child onto the CPU. Bound before the try for
# the same reason as _detected_gpus: the except path (--fit on) falls
@ -13995,6 +14319,17 @@ class LlamaCppBackend:
# GPU-aware speculative defaults; the list feeds the
# CPU-fallback check.
_detected_gpus = list(gpus)
# Vulkan reports total 0 only for integrated GPUs. Their
# free "VRAM" is the same host pool the RAM guard prices.
_shared_gpu_ids = (
{
idx
for idx, _free in _detected_gpus
if total_by_idx.get(idx, 1) <= 0
}
if is_vulkan_backend
else set()
)
# The --fit fallback is llama.cpp's own fitter, which knows nothing
# about this budget: it keeps its own margin and packs the rest on,
# so the slider never reached the path that runs when the fit is
@ -16589,6 +16924,29 @@ class LlamaCppBackend:
# The whole visible set stays in use, only its order is fixed.
self._pin_visible_gpu_order_for_split(env)
# reads the argv the child gets, not the mid-fit state that produced it
_offload_msg = self._launch_host_shortfall_message(
cmd,
_detected_gpus,
env,
child_has_no_gpu = (
# each names a device present but unusable, so it owns the empty pool
_arch_gate_forced_cpu
or _paravirtual_cpu_forced
# neither says a device exists, so an empty pool stays unreadable
or (
bool(_detected_gpus)
and (
_cpu_only_zero_offload
or self._binary_ships_no_gpu_backend(binary, env)
)
)
),
shared_gpu_ids = _shared_gpu_ids,
)
if _offload_msg:
raise RuntimeError(_offload_msg)
# Captured before any text-only fallback strips it from cmd.
launched_with_mmproj = "--mmproj" in cmd
@ -17018,6 +17376,15 @@ class LlamaCppBackend:
if _retry_ram_msg:
self._kill_process()
raise RuntimeError(_retry_ram_msg)
# host guard credited the whole pool; the respawn reaches only _remaining
_retry_offload_msg = self._launch_host_shortfall_message(
cmd,
[row for row in _detected_gpus if row[0] in set(_remaining)],
env,
)
if _retry_offload_msg:
self._kill_process()
raise RuntimeError(_retry_offload_msg)
logger.warning(
f"llama-server crashed with a HIP kernel-image error on "
f"GPU(s) {_crashed} -- the llama.cpp build has no kernels "

7
studio/backend/routes/inference.py
View file

@ -8641,6 +8641,13 @@ async def _load_model_impl(
if _non_chat:
logger.error("Refusing non-chat GGUF before the GPU handoff: %s", _non_chat)
raise HTTPException(status_code = 400, detail = _non_chat)
# same reason: the host-RAM guard reads the finished argv, so it answers too late
_host_offload = await asyncio.to_thread(
llama_backend.host_offload_refusal_for_intent, gguf_intent
)
if _host_offload:
logger.error("Refusing an oversized GGUF before the GPU handoff: %s", _host_offload)
raise HTTPException(status_code = 400, detail = _host_offload)
if chat_load_needs_gpu:
await asyncio.to_thread(

44
studio/backend/tests/test_gpu_arch_gate_os_matrix_7624.py
View file

@ -782,6 +782,7 @@ def _run_auto_load(
capture = None,
intent_kwargs = None,
apu_ram_stub = None,
host_offload_stub = None,
backend = None,
):
"""Drive a real automatic (no explicit GPU pick) llama-server load with the real
@ -820,6 +821,8 @@ def _run_auto_load(
# Off by default: the APU RAM preflight is not what most of these cells are
# about. A test that IS about it passes its own recording stub.
backend._apu_ram_shortfall_message = apu_ram_stub or (lambda *_args, **_kwargs: None)
# same, off: model_bytes here is sized to force --fit on, not to describe a host
backend._host_offload_shortfall_message = host_offload_stub or (lambda *_args, **_kwargs: None)
backend._find_llama_server_binary = lambda include_denied = False: binary
backend._fit_off_retry_eligible = lambda *_args, **_kwargs: False
backend.probe_server_capabilities = lambda _binary: {"found": True}
@ -1209,6 +1212,47 @@ class TestArchCrashRetryEnv:
assert _retry, "the arch-crash retry did not fire"
assert all(env.get("GGML_CUDA_ENABLE_UNIFIED_MEMORY") == "1" for env in _retry)
def _big_then_small_discrete(self, monkeypatch):
"""Both cards discrete, so neither pool reading is capped against system RAM and
the survivor is genuinely too small to hold what the crashed card held."""
_apply_os(monkeypatch, "linux", is_rocm = True)
monkeypatch.setattr(LlamaCppBackend, "_rocm_unified_memory_gpu_ids", staticmethod(set))
monkeypatch.setattr(
LlamaCppBackend, "_available_system_memory_mib", staticmethod(lambda: 20_000)
)
return _fake_torch(
[_device("gfx1030", free_mib = 40_000), _device("gfx900", free_mib = 4_000)],
vendor = "amd",
)
def test_the_retry_reprices_the_spill_against_the_narrowed_pool(
self, tmp_path, monkeypatch, probe_env
):
"""The host guard ran against the aggregate pool, and the retry masks the child onto
the survivor. When the crashed card supplied most of that credit the narrowed launch
spills far more into RAM than the preflight allowed, which is the OOM this guard
exists to stop. A 30 GB model is held by the 40000 MiB card the launch pins; the
4000 MiB survivor leaves about 26 GB for a host with 20 GB."""
torch = self._big_then_small_discrete(monkeypatch)
capture = {}
launches = _run_auto_load(
monkeypatch,
tmp_path,
torch,
None,
returncode = 1,
output = "ROCm error: device kernel image is invalid",
model_bytes = 30 * 1024**3,
host_offload_stub = LlamaCppBackend._host_offload_shortfall_message,
capture = capture,
)
assert launches, "the first launch was refused, so the retry is not what was tested"
assert not [
env for _c, env in launches if env.get("ROCR_VISIBLE_DEVICES") == "1"
], "the respawn on the narrowed pool was not refused"
assert "does not fit in GPU memory" in str(capture.get("error"))
class TestManualSplitLaunchesRespectTheGate:
"""Manual memory mode is not an explicit GPU pick, so the probe still opts into

148
studio/backend/tests/test_host_offload_ram_guard.py Normal file
View file

@ -0,0 +1,148 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""A GGUF that misses VRAM spills into host RAM under `--fit on`, unpriced. When
that spill is larger than available RAM the mmap'd weights thrash rather than
fail, the desktop stops responding and the OS kills the app, so the load must be
refused before llama-server is spawned."""
from __future__ import annotations
import sys
from types import SimpleNamespace
import core.inference.llama_cpp as llama_cpp_module
from core.inference.llama_cpp import LlamaCppBackend
_GB = 1024**3
_MIB_PER_GB = 1024
# Module-level (not a class attr) so it stays a plain function, not a bound method.
_shortfall = LlamaCppBackend._host_offload_shortfall_message
class TestHostOffloadShortfall:
def test_field_case_refuses(self):
# 13.3 GB GGUF + 1.1 GB mmproj + 1.8 GB KV on a 6 GB RTX 4050 laptop holding
# 4.8 GB free, against ~10 GB of RAM: about 11 GB has to run from host memory.
offload = int(16.2 * _GB) - int(4.8 * _GB)
msg = _shortfall(offload, 10 * _MIB_PER_GB)
assert msg is not None
# need rounds up and usable rounds down, so the pair never reads as a tie
assert "12 GB" in msg and "10 GB" in msg and "8 GB usable" in msg
assert "quantized GGUF" in msg
# the guard prices weights only, so context length cannot change its verdict
assert "context" not in msg
def test_same_spill_on_a_large_ram_host_allows(self):
# Deliberate CPU offload is a supported mode; only a shortfall refuses.
offload = int(16.2 * _GB) - int(4.8 * _GB)
assert _shortfall(offload, 64 * _MIB_PER_GB) is None
def test_vram_resident_load_never_refuses(self):
# More VRAM than the load needs, so the subtraction goes negative.
assert _shortfall(-4 * _GB, 1 * _MIB_PER_GB) is None
assert _shortfall(0, 1 * _MIB_PER_GB) is None
def test_unknown_available_never_refuses(self):
assert _shortfall(40 * _GB, None) is None
def test_boundary_at_headroom(self):
# 20 GB spill, headroom 2 GB. avail 23 GB -> fits; 21 GB -> refuse.
assert _shortfall(20 * _GB, 23 * _MIB_PER_GB) is None
assert _shortfall(20 * _GB, 21 * _MIB_PER_GB) is not None
def test_a_refusal_names_the_escape(self):
"""The picker still offers a variant this refuses, so the message has to say how
to load it anyway."""
msg = _shortfall(20 * _GB, 21 * _MIB_PER_GB)
assert msg is not None
assert "UNSLOTH_ALLOW_HOST_OFFLOAD=1" in msg
def test_a_refusal_never_prints_a_need_at_or_under_the_usable_figure(self):
"""A spill inside available RAM but inside the headroom too is still refused, so
the message must not read as 7 GB not fitting in 8 GB."""
msg = _shortfall(7 * _GB, 8 * _MIB_PER_GB)
assert msg is not None
assert "About 7 GB" in msg and "6 GB usable" in msg
def test_available_ram_is_capped_by_cgroup_v2_remainder(tmp_path, monkeypatch):
"""A container sees host-wide MemAvailable through psutil, but can only charge
memory.max - memory.current before the kernel enforces its own OOM boundary."""
root = tmp_path / "cgroup"
leaf = root / "studio.slice"
leaf.mkdir(parents = True)
(leaf / "memory.max").write_text(str(16 * _GB), encoding = "utf-8")
(leaf / "memory.current").write_text(str(4 * _GB), encoding = "utf-8")
proc_cgroup = tmp_path / "self.cgroup"
proc_cgroup.write_text("0::/studio.slice\n", encoding = "utf-8")
monkeypatch.setattr(llama_cpp_module, "_CGROUP_ROOT", str(root))
monkeypatch.setattr(llama_cpp_module, "_PROC_SELF_CGROUP", str(proc_cgroup))
monkeypatch.setitem(
sys.modules,
"psutil",
SimpleNamespace(virtual_memory = lambda: SimpleNamespace(available = 64 * _GB)),
)
assert LlamaCppBackend._available_system_memory_mib() == 12 * _MIB_PER_GB
backend = object.__new__(LlamaCppBackend)
backend._get_gguf_size_bytes = lambda _path: 20 * _GB
msg = backend._launch_host_shortfall_message(
["llama-server", "-m", str(tmp_path / "model.gguf")],
[(0, 4 * _MIB_PER_GB)],
)
assert msg is not None
assert "16 GB" in msg and "10 GB usable" in msg
def test_cgroup_v2_reclaims_inactive_file_cache_for_ram_admission(tmp_path, monkeypatch):
"""Cached GGUF pages are reclaimable, not another permanent host-RAM charge."""
root = tmp_path / "cgroup"
leaf = root / "studio.slice"
leaf.mkdir(parents = True)
(leaf / "memory.max").write_text(str(16 * _GB), encoding = "utf-8")
(leaf / "memory.current").write_text(str(12 * _GB), encoding = "utf-8")
(leaf / "memory.stat").write_text(f"inactive_file {8 * _GB}\n", encoding = "utf-8")
proc_cgroup = tmp_path / "self.cgroup"
proc_cgroup.write_text("0::/studio.slice\n", encoding = "utf-8")
monkeypatch.setattr(llama_cpp_module, "_CGROUP_ROOT", str(root))
monkeypatch.setattr(llama_cpp_module, "_PROC_SELF_CGROUP", str(proc_cgroup))
monkeypatch.setitem(
sys.modules,
"psutil",
SimpleNamespace(virtual_memory = lambda: SimpleNamespace(available = 64 * _GB)),
)
assert LlamaCppBackend._available_system_memory_mib() == 12 * _MIB_PER_GB
backend = object.__new__(LlamaCppBackend)
backend._get_gguf_size_bytes = lambda _path: 12 * _GB
assert (
backend._launch_host_shortfall_message(
["llama-server", "-m", str(tmp_path / "model.gguf")],
[(0, 4 * _MIB_PER_GB)],
)
is None
)
def test_cgroup_v1_reclaims_hierarchical_inactive_file_cache(tmp_path, monkeypatch):
root = tmp_path / "cgroup"
leaf = root / "memory" / "studio.slice"
leaf.mkdir(parents = True)
(leaf / "memory.limit_in_bytes").write_text(str(16 * _GB), encoding = "utf-8")
(leaf / "memory.usage_in_bytes").write_text(str(12 * _GB), encoding = "utf-8")
(leaf / "memory.stat").write_text(
f"inactive_file {2 * _GB}\ntotal_inactive_file {8 * _GB}\n",
encoding = "utf-8",
)
proc_cgroup = tmp_path / "self.cgroup"
proc_cgroup.write_text("5:memory:/studio.slice\n", encoding = "utf-8")
monkeypatch.setattr(llama_cpp_module, "_CGROUP_ROOT", str(root))
monkeypatch.setattr(llama_cpp_module, "_PROC_SELF_CGROUP", str(proc_cgroup))
assert LlamaCppBackend._cgroup_available_memory_mib() == 12 * _MIB_PER_GB

486
studio/backend/tests/test_llama_cpp_placement.py
View file

@ -95,6 +95,9 @@ def _backend(tmp_path: Path, *, vulkan: bool, memory):
backend._mmproj_vram_bytes = lambda _path: 0
backend._resolve_launch_mmproj_path = lambda **kwargs: None
backend._apu_ram_shortfall_message = lambda *args, **kwargs: None
# Off by default: the host-RAM preflight is not what most of these cells are about,
# and it now runs on every launch. The tests that ARE about it restore the real one.
backend._launch_host_shortfall_message = lambda *args, **kwargs: None
backend._amd_apu_wants_unified_memory = lambda *args, **kwargs: False
backend._find_llama_server_binary = lambda include_denied = False: "/fake/llama-server"
backend._is_vulkan_backend = lambda _binary = None: vulkan
@ -1531,3 +1534,486 @@ def test_a_subset_that_can_shrink_to_hold_both_is_where_the_decision_lands(tmp_p
assert "--model-draft" not in cmd
assert backend.spec_fallback_reason == "drafter_no_vram"
assert cmd[cmd.index("-c") + 1] == "8192"
def _restore_host_guard(backend):
"""Put the real preflight back on a harness that stubs it off by default."""
backend._launch_host_shortfall_message = LlamaCppBackend._launch_host_shortfall_message.__get__(
backend
)
return backend
def _offload_backend(tmp_path, *, gguf_gb, free_mib, avail_mib, monkeypatch, **kwargs):
backend, gguf = _backend(tmp_path, vulkan = False, memory = [(0, free_mib, 6141)])
_restore_host_guard(backend)
backend._get_gguf_size_bytes = lambda _path: int(gguf_gb * 1024**3)
# no subset holds the model, so --fit on owns placement and spills to host ram
backend._select_gpus = lambda *args, **kw: (None, True)
for name, value in kwargs.items():
setattr(backend, name, value)
monkeypatch.setattr(
LlamaCppBackend, "_available_system_memory_mib", staticmethod(lambda: avail_mib)
)
return backend, gguf
def test_weights_larger_than_vram_plus_ram_are_refused(tmp_path, monkeypatch):
"""The field case: a 13.3 GB GGUF on a 6 GB laptop card holding 4877 MiB free needs
about 8.5 GB of host RAM, which a 10 GB host cannot hold. Unrefused, the mmap'd
remainder thrashes until the OS kills Studio and the desktop session."""
backend, gguf = _offload_backend(
tmp_path, gguf_gb = 13.3, free_mib = 4877, avail_mib = 10_000, monkeypatch = monkeypatch
)
with pytest.raises(RuntimeError, match = "does not fit in GPU memory"):
_launch(backend, gguf)
def test_the_same_load_on_a_large_ram_host_still_launches(tmp_path, monkeypatch):
"""Deliberate CPU offload stays supported; only a shortfall refuses."""
backend, gguf = _offload_backend(
tmp_path, gguf_gb = 13.3, free_mib = 4877, avail_mib = 64_000, monkeypatch = monkeypatch
)
assert "--fit" in _launch(backend, gguf)["cmd"]
def test_free_vram_offsets_the_charge(tmp_path, monkeypatch):
"""Same model and same host RAM as the refusal above, but a card big enough to hold
it. The VRAM credit is what separates the two, so the charge is the shortfall and
not the model size."""
backend, gguf = _offload_backend(
tmp_path, gguf_gb = 13.3, free_mib = 20_000, avail_mib = 10_000, monkeypatch = monkeypatch
)
assert "--fit" in _launch(backend, gguf)["cmd"]
def test_vulkan_igpu_shared_memory_is_not_counted_twice(tmp_path, monkeypatch):
"""A Vulkan iGPU's free memory and MemAvailable describe the same unified pool.
Crediting both let a 20 GiB model through on a 14 GiB host (12 + 14 on paper)."""
backend, gguf = _backend(
tmp_path,
vulkan = True,
memory = [(0, 12 * 1024, 0)],
)
_restore_host_guard(backend)
backend._get_gguf_size_bytes = lambda _path: 20 * 1024**3
backend._select_gpus = lambda *args, **kwargs: (None, True)
monkeypatch.setattr(
LlamaCppBackend, "_available_system_memory_mib", staticmethod(lambda: 14 * 1024)
)
with pytest.raises(RuntimeError, match = "does not fit in GPU memory"):
_launch(backend, gguf)
def test_unknown_available_ram_abstains(tmp_path, monkeypatch):
backend, gguf = _offload_backend(
tmp_path, gguf_gb = 13.3, free_mib = 4877, avail_mib = None, monkeypatch = monkeypatch
)
assert _launch(backend, gguf)["cmd"]
def test_an_unsized_model_abstains(tmp_path, monkeypatch):
"""A GGUF whose size cannot be read leaves nothing to price."""
backend, gguf = _offload_backend(
tmp_path, gguf_gb = 13.3, free_mib = 4877, avail_mib = 10_000, monkeypatch = monkeypatch
)
backend._get_gguf_size_bytes = lambda _path: (_ for _ in ()).throw(OSError("stat failed"))
assert _launch(backend, gguf)["cmd"]
@pytest.mark.parametrize(
"extra_args",
[
["-ngl", "0"],
["--mlock"],
["--no-mmap"],
["--device", "none"],
["--no-kv-offload"],
],
ids = ["zero-layers", "mlock", "no-mmap", "cpu-device", "cpu-kv"],
)
def test_placement_flags_never_turn_an_allowed_load_into_a_refusal(
tmp_path, monkeypatch, extra_args
):
"""The floor prices weights against the whole free pool and models no placement.
Each of these moves bytes onto the host or narrows the reachable VRAM, so a guard
that read them could only refuse MORE. Leaving them out cannot invent a refusal,
which is the property that keeps this check free of llama.cpp placement modelling."""
backend, gguf = _offload_backend(
tmp_path, gguf_gb = 13.3, free_mib = 4877, avail_mib = 64_000, monkeypatch = monkeypatch
)
assert _launch(backend, gguf, extra_args = extra_args)["cmd"]
def test_the_guard_reads_the_model_the_child_opens(tmp_path, monkeypatch):
"""Sizing comes from the argv path, not from the planner's earlier pick, so a
fallback that rewrote -m is priced as launched."""
seen = []
backend, gguf = _offload_backend(
tmp_path, gguf_gb = 13.3, free_mib = 4877, avail_mib = 10_000, monkeypatch = monkeypatch
)
real_size = backend._get_gguf_size_bytes
def _record(path):
seen.append(str(path))
return real_size(path)
backend._get_gguf_size_bytes = _record
with pytest.raises(RuntimeError, match = "does not fit in GPU memory"):
_launch(backend, gguf)
assert str(gguf) in seen
def test_the_env_escape_loads_a_variant_the_guard_refuses(tmp_path, monkeypatch):
"""The picker still offers a variant `classifyGgufFit` calls "oom", and no load
field carries a force, so an unconditional refusal leaves that selection with no way
through. UNSLOTH_ALLOW_HOST_OFFLOAD=1 abstains, and the refusal names it."""
refused, gguf = _offload_backend(
tmp_path, gguf_gb = 13.3, free_mib = 4877, avail_mib = 10_000, monkeypatch = monkeypatch
)
with pytest.raises(RuntimeError, match = "UNSLOTH_ALLOW_HOST_OFFLOAD=1"):
_launch(refused, gguf)
allowed_dir = tmp_path / "allowed"
allowed_dir.mkdir()
allowed, gguf2 = _offload_backend(
allowed_dir, gguf_gb = 13.3, free_mib = 4877, avail_mib = 10_000, monkeypatch = monkeypatch
)
monkeypatch.setenv("UNSLOTH_ALLOW_HOST_OFFLOAD", "1")
assert "--fit" in _launch(allowed, gguf2)["cmd"]
def _load_intent(gguf, **kwargs):
return GgufLoadIntent(gguf_path = str(gguf), model_identifier = "test", **kwargs)
def _host_totals(
monkeypatch,
backend,
*,
vram_total_mib,
ram_total_mib,
vram_free_mib = None,
):
"""Pin what the preflight reads: the physical ceilings, and a free VRAM figure low
enough to stand for a card the resident model has not given back yet."""
free = vram_total_mib if vram_free_mib is None else vram_free_mib
backend._get_gpu_memory = lambda _binary = None, **_kw: [(0, free, vram_total_mib)]
monkeypatch.setattr(
LlamaCppBackend, "_total_system_memory_mib", staticmethod(lambda: ram_total_mib)
)
def test_the_route_precheck_refuses_before_the_gpu_handoff(tmp_path, monkeypatch):
"""`acquire_for(CHAT)` evicts a resident Images/Video pipeline and the reload
confirmation cancels the running generations, both before the launch guard can read the
finished argv. The route asks first, so a pick no reclaim can rescue, 100 GB against a
24 GB card and 10 GB of RAM, tears nothing down."""
backend, gguf = _offload_backend(
tmp_path, gguf_gb = 100, free_mib = 20_000, avail_mib = 10_000, monkeypatch = monkeypatch
)
_host_totals(monkeypatch, backend, vram_total_mib = 24_000, ram_total_mib = 32_000)
verdict = backend.host_offload_refusal_for_intent(_load_intent(gguf))
assert verdict is not None and "does not fit in GPU memory" in verdict
def test_the_route_precheck_credits_capacity_the_handoff_is_about_to_reclaim(tmp_path, monkeypatch):
"""The resident llama-server, Unsloth model and media pipeline hold VRAM, and through a
host KV cache, CPU-offloaded weights and locked mappings they hold RAM too. The route and
load_model reclaim all of it after this runs, so pricing against either free reading
refused a switch the reclaimed machine handles outright and made switching on a busy
machine impossible. Both physical totals are what bound the launch.
30 GB against a 24 GB card leaves about 6.7 GB on the host, which 3 GB of MemAvailable
cannot hold and the machine's own 64 GB holds easily."""
backend, gguf = _offload_backend(
tmp_path, gguf_gb = 30, free_mib = 900, avail_mib = 3_000, monkeypatch = monkeypatch
)
# 900 MiB free VRAM and 3 GB MemAvailable: the model being replaced still holds both
_host_totals(
monkeypatch, backend, vram_total_mib = 24_000, ram_total_mib = 64_000, vram_free_mib = 900
)
assert backend.host_offload_refusal_for_intent(_load_intent(gguf)) is None
def test_the_route_precheck_only_refuses_what_the_launch_would(tmp_path, monkeypatch):
"""Abstains on an undownloaded repo, a device whose total the probe cannot read, an
unreadable pool, unreadable total RAM and the escape. So it can never reject a load the
launch would allow."""
backend, gguf = _offload_backend(
tmp_path, gguf_gb = 100, free_mib = 20_000, avail_mib = 10_000, monkeypatch = monkeypatch
)
_host_totals(monkeypatch, backend, vram_total_mib = 24_000, ram_total_mib = 32_000)
assert backend.host_offload_refusal_for_intent(_load_intent(gguf, hf_repo = "org/repo")) is None
# an igpu or a MIG/vGPU line reports total 0, so the ceiling is unknown
backend._get_gpu_memory = lambda _binary = None, **_kw: [(0, 20_000, 0)]
assert backend.host_offload_refusal_for_intent(_load_intent(gguf)) is None
backend._get_gpu_memory = lambda _binary = None, **_kw: []
assert backend.host_offload_refusal_for_intent(_load_intent(gguf)) is None
_host_totals(monkeypatch, backend, vram_total_mib = 24_000, ram_total_mib = None)
assert backend.host_offload_refusal_for_intent(_load_intent(gguf)) is None
_host_totals(monkeypatch, backend, vram_total_mib = 24_000, ram_total_mib = 32_000)
monkeypatch.setenv("UNSLOTH_ALLOW_HOST_OFFLOAD", "1")
assert backend.host_offload_refusal_for_intent(_load_intent(gguf)) is None
def test_an_arch_gated_cpu_launch_prices_the_whole_model(tmp_path, monkeypatch):
"""The arch gate empties the pool AND masks every card, so the child is knowingly
on the CPU rather than unprobed. Abstaining there ran an oversized GGUF wholly from
RAM with no preflight, which is the OOM this guard exists to stop."""
backend, gguf = _backend(tmp_path, vulkan = False, memory = [])
_restore_host_guard(backend)
backend._get_gguf_size_bytes = lambda _path: int(13.3 * 1024**3)
backend._select_gpus = lambda *args, **kw: (None, True)
monkeypatch.setattr(
LlamaCppBackend, "_available_system_memory_mib", staticmethod(lambda: 10_000)
)
assert (
backend._launch_host_shortfall_message(
["llama-server", "-m", str(gguf)], [], child_has_no_gpu = True
)
is not None
)
def test_a_masked_off_child_takes_no_vram_credit(tmp_path, monkeypatch):
"""Manual zero-offload masks the child off cards the planner still probed. Crediting
that VRAM would offset a spill the child cannot place there."""
backend, gguf = _backend(tmp_path, vulkan = False, memory = [(0, 20_000, 24_000)])
_restore_host_guard(backend)
backend._get_gguf_size_bytes = lambda _path: int(13.3 * 1024**3)
monkeypatch.setattr(
LlamaCppBackend, "_available_system_memory_mib", staticmethod(lambda: 10_000)
)
argv = ["llama-server", "-m", str(gguf)]
assert backend._launch_host_shortfall_message(argv, [(0, 20_000)]) is None
assert (
backend._launch_host_shortfall_message(argv, [(0, 20_000)], child_has_no_gpu = True)
is not None
)
def test_an_unprobed_pool_still_abstains_when_nothing_was_masked(tmp_path, monkeypatch):
"""The abstention survives: only the launch saying it masked the child off every
card prices the full model, not a pool that merely came back empty."""
backend, gguf = _backend(tmp_path, vulkan = False, memory = [])
_restore_host_guard(backend)
backend._get_gguf_size_bytes = lambda _path: int(13.3 * 1024**3)
monkeypatch.setattr(
LlamaCppBackend, "_available_system_memory_mib", staticmethod(lambda: 10_000)
)
assert backend._launch_host_shortfall_message(["llama-server", "-m", str(gguf)], []) is None
def test_a_gpu_less_host_running_a_cpu_only_build_still_abstains(tmp_path, monkeypatch):
"""Studio installs a CPU-only prebuilt on a host with no GPU, so that host probes an
empty pool AND reports a build with no GPU backend. Letting the build state alone
charge the whole model refused a 7.5 GB GGUF with 9 GB of RAM, which loads on main,
and blamed GPU memory on a machine that has no GPU."""
backend, gguf = _backend(tmp_path, vulkan = False, memory = [])
_restore_host_guard(backend)
backend._get_gguf_size_bytes = lambda _path: int(7.5 * 1024**3)
backend._select_gpus = lambda *args, **kw: (None, True)
backend._binary_ships_no_gpu_backend = lambda _binary = None, _env = None: True
monkeypatch.setattr(
LlamaCppBackend, "_available_system_memory_mib", staticmethod(lambda: 9_216)
)
assert _launch(backend, gguf)["cmd"]
def test_a_gpu_less_host_still_abstains_on_a_zero_offload_request(tmp_path, monkeypatch):
"""gpu_layers=0 is a request, not a probe result, so it says nothing about whether a
card exists. Charging the whole model on an empty pool repeats the CPU-only-build
refusal on the same GPU-less host."""
backend, gguf = _backend(tmp_path, vulkan = False, memory = [])
_restore_host_guard(backend)
backend._get_gguf_size_bytes = lambda _path: int(7.5 * 1024**3)
backend._select_gpus = lambda *args, **kw: (None, True)
monkeypatch.setattr(
LlamaCppBackend, "_available_system_memory_mib", staticmethod(lambda: 9_216)
)
assert _launch(backend, gguf, gpu_memory_mode = "manual", gpu_layers = 0)["cmd"]
def test_a_cpu_only_build_takes_no_vram_credit(tmp_path, monkeypatch):
"""A split-library build shipping no cuda/hip/vulkan backend cannot offload, so the
cards the hardware probe still enumerates are unreachable. Crediting their VRAM
priced a spill the child never takes: it places the whole model in RAM."""
backend, gguf = _backend(tmp_path, vulkan = False, memory = [(0, 16_384, 24_000)])
_restore_host_guard(backend)
backend._get_gguf_size_bytes = lambda _path: 20 * 1024**3
monkeypatch.setattr(
LlamaCppBackend, "_available_system_memory_mib", staticmethod(lambda: 8_192)
)
argv = ["llama-server", "-m", str(gguf)]
# 20 GiB - 16 GiB free VRAM reads as a 4 GiB spill an 8 GiB host can hold.
assert backend._launch_host_shortfall_message(argv, [(0, 16_384)]) is None
assert (
backend._launch_host_shortfall_message(argv, [(0, 16_384)], child_has_no_gpu = True)
is not None
)
def test_an_unknown_backend_layout_keeps_its_vram_credit(tmp_path):
"""Fails open on a static or unrecognised layout, so a custom GPU build is never
mistaken for a CPU-only one and refused."""
assert LlamaCppBackend._binary_ships_no_gpu_backend("/nonexistent/llama-server") is False
def test_the_launch_reports_a_cpu_only_build_to_the_guard(tmp_path, monkeypatch):
"""End to end: the call site must pass the CPU-only-build state, not just accept it.
A 20 GiB model over 16 GiB of free VRAM reads as a 4 GiB spill an 8 GiB host holds,
so only the build state separates the launch from the refusal."""
gpu_build, gguf = _offload_backend(
tmp_path, gguf_gb = 20, free_mib = 16_384, avail_mib = 8_192, monkeypatch = monkeypatch
)
gpu_build._binary_ships_no_gpu_backend = lambda _binary = None, _env = None: False
assert _launch(gpu_build, gguf)["cmd"]
cpu_dir = tmp_path / "cpu"
cpu_dir.mkdir()
cpu_build, gguf2 = _offload_backend(
cpu_dir,
gguf_gb = 20,
free_mib = 16_384,
avail_mib = 8_192,
monkeypatch = monkeypatch,
)
cpu_build._binary_ships_no_gpu_backend = lambda _binary = None, _env = None: True
with pytest.raises(RuntimeError, match = "does not fit in GPU memory"):
_launch(cpu_build, gguf2)
def test_an_empty_gpu_pool_abstains(tmp_path, monkeypatch):
"""_get_gpu_memory swallows a failed probe as [], so an empty pool cannot be told
from a host with no GPU. Pricing the full model there would refuse a load that
llama-server's own enumeration can still place on a card."""
backend, gguf = _backend(tmp_path, vulkan = False, memory = [])
_restore_host_guard(backend)
backend._get_gguf_size_bytes = lambda _path: int(13.3 * 1024**3)
monkeypatch.setattr(
LlamaCppBackend, "_available_system_memory_mib", staticmethod(lambda: 10_000)
)
assert _launch(backend, gguf)["cmd"]
@pytest.mark.parametrize("accelerator", ["sycl", "opencl", "musa", "cann"])
def test_a_non_cuda_accelerator_build_keeps_its_vram_credit(tmp_path, accelerator):
"""_installed_ggml_backends reads only cuda, hip and vulkan, so a split-library build
shipping any other supported ggml accelerator looked CPU-only. Pricing its weights
against RAM refused loads the accelerator can hold."""
binary = tmp_path / "llama-server"
binary.write_bytes(b"x")
lib_dir = tmp_path / "lib"
lib_dir.mkdir()
prefix = "" if sys.platform == "win32" else "lib"
extension = "dll" if sys.platform == "win32" else "so"
(lib_dir / f"{prefix}ggml-cpu.{extension}").write_bytes(b"x")
(lib_dir / f"{prefix}ggml-{accelerator}.{extension}").write_bytes(b"x")
with patch("core.inference.llama_cpp._llama_lib_dir", return_value = lib_dir):
assert LlamaCppBackend._binary_ships_no_gpu_backend(str(binary)) is False
# the narrower pre-existing helper is what misreads this layout
assert LlamaCppBackend._backend_lacks_gpu_lib(str(binary)) is True
def test_a_genuinely_cpu_only_layout_is_still_recognised(tmp_path):
binary = tmp_path / "llama-server"
binary.write_bytes(b"x")
lib_dir = tmp_path / "lib"
lib_dir.mkdir()
prefix = "" if sys.platform == "win32" else "lib"
extension = "dll" if sys.platform == "win32" else "so"
(lib_dir / f"{prefix}ggml-cpu.{extension}").write_bytes(b"x")
(lib_dir / f"{prefix}ggml-base.{extension}").write_bytes(b"x")
with patch("core.inference.llama_cpp._llama_lib_dir", return_value = lib_dir):
assert LlamaCppBackend._binary_ships_no_gpu_backend(str(binary)) is True
def test_an_rpc_launch_abstains(tmp_path, monkeypatch):
"""--rpc places layers on remote devices this cannot size, so refusing on local
capacity alone would block a viable distributed launch."""
backend, gguf = _offload_backend(
tmp_path, gguf_gb = 13.3, free_mib = 4877, avail_mib = 10_000, monkeypatch = monkeypatch
)
argv = ["llama-server", "-m", str(gguf)]
assert backend._launch_host_shortfall_message(argv, [(0, 4877)]) is not None
assert (
backend._launch_host_shortfall_message([*argv, "--rpc", "10.0.0.2:50052"], [(0, 4877)])
is None
)
assert backend._launch_host_shortfall_message([*argv, "--rpc", " "], [(0, 4877)]) is not None
def test_an_rpc_env_launch_abstains(tmp_path, monkeypatch):
"""llama.cpp reads LLAMA_ARG_RPC as the environment twin of --rpc, so the guard has
to see the child environment or it refuses the same distributed launch."""
backend, gguf = _offload_backend(
tmp_path, gguf_gb = 13.3, free_mib = 4877, avail_mib = 10_000, monkeypatch = monkeypatch
)
argv = ["llama-server", "-m", str(gguf)]
assert backend._launch_host_shortfall_message(argv, [(0, 4877)], {}) is not None
assert (
backend._launch_host_shortfall_message(
argv, [(0, 4877)], {"LLAMA_ARG_RPC": "10.0.0.2:50052"}
)
is None
)
def test_an_external_backend_path_keeps_its_vram_credit(tmp_path):
"""GGML_BACKEND_PATH points the child at plugins outside the lib directory, so a
cpu-only layout beside the binary is no longer proof the child cannot offload."""
binary = tmp_path / "llama-server"
binary.write_bytes(b"x")
lib_dir = tmp_path / "lib"
lib_dir.mkdir()
prefix = "" if sys.platform == "win32" else "lib"
extension = "dll" if sys.platform == "win32" else "so"
(lib_dir / f"{prefix}ggml-cpu.{extension}").write_bytes(b"x")
with patch("core.inference.llama_cpp._llama_lib_dir", return_value = lib_dir):
assert LlamaCppBackend._binary_ships_no_gpu_backend(str(binary), {}) is True
assert (
LlamaCppBackend._binary_ships_no_gpu_backend(
str(binary), {"GGML_BACKEND_PATH": "/opt/ggml-cuda"}
)
is False
)
def test_a_paravirtual_metal_launch_prices_the_whole_model(tmp_path, monkeypatch):
"""A virtualised Apple GPU rewrites the command to --gpu-layers 0 --device none, and
Metal hosts leave the pool empty, so the abstention swallowed a placement the launch
already knew was CPU-only."""
backend, gguf = _backend(tmp_path, vulkan = False, memory = [])
_restore_host_guard(backend)
backend._get_gguf_size_bytes = lambda _path: int(13.3 * 1024**3)
monkeypatch.setattr(
LlamaCppBackend, "_available_system_memory_mib", staticmethod(lambda: 10_000)
)
argv = ["llama-server", "-m", str(gguf), "--gpu-layers", "0", "--device", "none"]
assert backend._launch_host_shortfall_message(argv, [], {}) is None
assert backend._launch_host_shortfall_message(argv, [], {}, child_has_no_gpu = True) is not None