mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-24 08:13:59 +00:00
300292b2f0
* Read ROCm VRAM through amd-smi so the backend keeps no HIP context torch.cuda.mem_get_info creates a CUDA/HIP primary context that the process never releases: 612 MiB measured on a B200 with CUDA 13, 692-712 MiB reported elsewhere. get_device_properties and get_device_capability do not, and agree with mem_get_info on total memory exactly. LlamaCppBackend._get_gpu_memory asks nvidia-smi first, so NVIDIA hosts were already clean. ROCm hosts, and NVIDIA hosts without nvidia-smi on PATH, fell straight through to torch, so a backend serving GGUF models held ~700 MiB for a number it only reads, on a card whose models run in a llama-server child that then cannot use it. Two docstrings promised the probe was torch-free. amd-smi answers the same question from a subprocess. The new branch sits between nvidia-smi and torch, reuses utils/hardware/amd.py's runner, unit parsing and circuit breaker, honours the visibility mask, and gives a unified-memory APU the same host reserve and total 0 as the torch branch. The torch fallback stays for hosts with no smi tool. Callers read an empty result as no GPU (_resolve_auto picks a backend from it, _gpu_available gates the embedder, the loader fit drops to CPU), so an unknown there would silently move inference off the GPU; the context is the lesser cost. Parity on GPU 5, amd-smi output synthesised from the live nvidia-smi reading of the same card: free 179165 MiB and total 183359 MiB both branches, 0 MiB context on the new path against 612 MiB on the torch path. NVIDIA numbers are untouched, and the branch is inert off ROCm. Rebased over #7670, which added for_llama_server and the ROCm arch gate to the same function. The new branch returns before the gate ran, so it would have put #7624 back for llama-server placement. _rocm_arch_gate_keep now holds the gate, and both branches call it, so the two cannot drift. It costs no context of its own: get_device_properties measured 0MiB against mem_get_info's 620MiB on GPU 5. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Defer to torch on unified-memory APUs, and read a single-GPU amd-smi dict for PR #8669 amd-smi reports only the dedicated VRAM carve-out on a unified-memory APU, while HIP reports the far larger GTT pool the models actually run in. On a 128GiB Strix Halo with an 8GiB BIOS carve-out that is 8GiB against ~100GiB, so the new branch would have handed the GGUF fitter the slice and refused models that fit today. The branch now declines the whole host when any visible device is a unified-memory APU: discrete ROCm cards keep the context saving, APU hosts keep byte-identical pre-PR numbers, and the two branches cannot disagree because only one ever answers. Reconstructing the pool from total_gtt was the other option and was rejected: HIP's mem_get_info on an APU is not documented as equal to GTT alone, and the older vram / fb_memory_usage shapes carry no GTT fields, so the two branches would drift, which is what _rocm_arch_gate_keep was unified to prevent. Separately, _gpu_entries' documented bare single-GPU dict fallback was dead: data.get("gpu", [data]) returns the numeric id for that shape and enumerate(int) raises. An envelope key now only counts when its value is really a list, which also closes enumerate(None) on {"gpus": null}. * Translate amd-smi gpu ids into HIP ids, and answer for every visible device amd-smi numbers its devices in discovery order over its own KFD view; HIP numbers them by KFD node id (amdsmi computes hip_id as node_id - smallest_node_id). The two agree on most hosts and not on all of them, and the branch was keying VRAM rows by amd-smi's number, comparing it against HIP/ROCR/CUDA visibility tokens and then pinning the child with it. On a host where they differ, the larger card's memory lands on the smaller card's HIP id and a load that fit before this PR can OOM. amd-smi list -e (ROCm 6.4.0+, amdsmi_get_gpu_enumeration_info) is the mapping AMD publishes for exactly this. Read it and re-key, and decline when it is not readable: an older CLI rejects -e, hip_id reads N/A when the library cannot reach the device's KFD node, and identity is then an assumption rather than a fact. A single-GPU host skips the call, since one card on the host forces the mapping, so the ROCm desktops this branch mostly runs on keep the context saving on any amd-smi version. That -e failure is kept out of the circuit breaker: an old CLI is not an amd-smi outage, and three of them would disable the utilization poller too. _resolve_visible_physical_ids returns None both for "no mask" and for a mask it cannot parse, and the branch read the second as the first. ROCr accepts UUID tokens (ROCR_VISIBLE_DEVICES=GPU-<uuid>), and amd-smi ignores the mask entirely, so a deliberately hidden card was being offered for automatic ranking. Use the existing _visibility_mask_is_unmappable and defer to torch, which enumerates the masked set. Separately, get_gpu_vram_mib drops a row whose total reads 0, and what was left was a non-empty list the caller takes as the whole host. On an APU beside a dGPU that turned two devices into one, so the "fewer than 2 usable GPUs" arm switched off tensor parallelism on a host main reports as two. get_gpu_vram_report now also returns every id amd-smi enumerated, and the branch answers for every visible device or declines entirely. The existing rocm fixture only patched _resolve_visible_physical_ids and left the shell's own CUDA_VISIBLE_DEVICES in the environment, which now reads as a mask set but unresolvable; clear it there too, the same trap as #8662. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Decline the amd-smi VRAM branch when HIP cannot open a device amd-smi reads the driver over sysfs and libdrm, HIP needs /dev/kfd and a matching HSA runtime. They come apart on a container started with --device=/dev/dri but no --device=/dev/kfd, and on a torch wheel built against a ROCm the installed runtime does not match: amd-smi lists the card and hipGetDeviceCount returns 0, so torch.cuda.is_available() is False while _torch_is_rocm() stays True. The llama-server child is a HIP process too, so it dies exactly where torch did. Before this branch existed the torch fallback returned [] for such a host and the load went to CPU. The amd-smi branch returns first and hands placement a device nothing can open. Neither existing guard catches it, since both are torch readers that fail open: _rocm_unified_memory_gpu_ids answers set() and _rocm_arch_by_physical_id answers {} the moment is_available() is False, so a unified-memory APU is not even recognised as one and its dedicated carve-out is taken for the whole pool. The gate costs nothing this path was not already paying: _rocm_unified_memory_gpu_ids calls is_available() and get_device_properties() further down the same function. Neither creates a primary context (measured: no compute-apps entry, against 612 MiB for mem_get_info), because is_available() is hipGetDeviceCount rather than _lazy_init. Both rocm fixtures now declare HIP reachability instead of inheriting the host's, the same trap as #8662: unpatched they read the test machine's own GPU and 16 of them fail on a CPU-only runner. * Answer the amd-smi VRAM branch only for HIP's own inventory for PR #8669 GPU_DEVICE_ORDINAL is a ROCm visibility variable that _resolve_visible_physical_ids does not read, so the branch took no mask from a process that had one. Defer to torch while it is set, the way hardware.py::_rocm_visibility_mask_active gates the system-wide overlay. Compare the visible device count against hipGetDeviceCount as well: a device-cgroup container given one /dev/dri/renderD* node and no env var leaves amd-smi enumerating the whole host, and a row amd-smi omitted leaves a single-GPU shape the id-mapping shortcut accepts as a one-card host. Compare the totals too, with the same 10% margin the System tab's overlay uses. An APU _rocm_classify_unified_memory cannot name (a gfx1103 Phoenix wheel with no is_integrated flag) has amd-smi reporting the BIOS carve-out where HIP reports the GTT pool, and a partition reads the other way round. Devices torch cannot describe fail open. * Decline the amd-smi VRAM branch when ROCm visibility masks are stacked ROCR_VISIBLE_DEVICES filters and renumbers agents inside the ROCr runtime, and clr's Device::init then indexes HIP_VISIBLE_DEVICES (or its CUDA_VISIBLE_DEVICES twin) into the agent vector ROCr left behind, so the two compose: ROCR_VISIBLE_DEVICES=1,2 with HIP_VISIBLE_DEVICES=0 opens physical GPU 1. _active_gpu_visibility_mask returns the single highest-precedence value, which reads [0] there, and on a host whose cards share a model every later check on the amd-smi branch agrees with it: HIP opens one device, the mask names one, and the totals match. The branch then reported physical GPU 0's free VRAM for a process that can only reach GPU 1. Defer to torch instead, which enumerates in HIP's own space; composing the masks would need the physical to ROCr mapping ROCr just renumbered away. HIP and CUDA together are one layer, not two, and Windows has no ROCr layer, so neither defers. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| amd.py | ||
| apple.py | ||
| hardware.py | ||
| nvidia.py | ||
| VRAM_ESTIMATION.md | ||
| vram_estimation.py | ||