86 commits

Author	SHA1	Message	Date
Daniel Han	34ee2bf1fa	Studio: name what is keeping Train off on Apple Silicon (#8303) ... * Studio: name what is keeping Train off on Apple Silicon The MLX gate is all-or-nothing across mlx, mlx-lm and mlx-vlm, and every failure came out as one verdict with one message: "Training needs MLX. Run `unsloth studio update` to enable Train." That is a dead end for the usual cause, which is an update that already ran and a resolver backtrack that left one package missing, too old, or unable to import under the pinned transformers. Nothing said which. mlx_stack_blockers() reports the same checks mlx_stack_available() makes, in the same order, as lines a person can act on: "mlx-vlm 0.1.0 is older than 0.4.4", "mlx_vlm does not import (ImportError: ...)". Detection records the first few on the mlx_unavailable verdict, /api/health ships them as chat_only_detail, and the greyed-out Train row reads "Training needs MLX: mlx-vlm 0.1.0 is older than 0.4.4. Run `unsloth studio update` to enable Train." A backend without the field falls back to the message as it is today. The installer runs the same check on Apple Silicon after it finishes. It used to report success and let the app come up chat-only, which is how a user ends up being told to run the update that has just completed. Advisory only: the install still succeeds, chat still works, and the background self-heal still gets its go. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the MLX blocker detail with the verdict it explains The detail only means anything beside the reason it explains, and it was not travelling with it: detect_hardware() snapshots the published verdict so a raising pass can put it back, and the tuple did not include the detail. A failed forced re-detect restored mlx_unavailable with the blocker gone, which is the generic message again. _discard_detection_locked() clears the verdict for a retired epoch and left the detail behind, and ensure_hardware_detected()'s detection_failed fallback could publish a detail recorded for a different reason. /api/health read the global after _hardware_snapshot() returned, outside the seqlock, so a re-detect starting in between could pair one pass's reason with another pass's detail, or with none. The snapshot is a 3-tuple now and the response reads it from there. Tests cover all four: restore after a raising pass, discard, and that the snapshot is taken as one read and does not move when the globals do. * Measure the MLX stack once, and bound what it reports * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stub the blocker gate in the hardware dispatch matrix * Re-measure the MLX blocker after an install that changed the stack * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Count a half-applied reinstall as changing the stack * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-10 06:40:22 -07:00
oobabooga	495ba21ebd	Studio: add MiniMax H3 video generation (#7989) ... * Add MiniMax H3 video generation * Improve MiniMax H3 memory routing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address MiniMax H3 review feedback * Keep H3's VAE off the CPU path, so low_vram stops aborting low_vram maps to the `model` policy, and offload_flags emits --vae-on-cpu for it unconditionally. On H3 that kills the process: ggml/src/ggml-cpu/ops.cpp:6321: GGML_ASSERT(src0->type == GGML_TYPE_F16) failed deterministically, SIGABRT, exit 134. Bisected on the flags: --vae-on-cpu with --audio-vae aborts, the same command without --audio-vae renders in 95.87s, and an fp16-converted audio VAE aborts too. So the trigger is the audio VAE, not the video one, and the F32 type is imposed inside stable-diffusion.cpp rather than by the file: ggml_conv_1d hardcodes an F16 im2col destination (ggml/src/ggml.c), ggml_compute_forward_im2col_f16 then asserts the KERNEL is F16, and audio_conv_weight_type (src/model/vae/ltx_audio_vae.hpp) maps only BF16 to F16 and lets F32 through. It cannot be fixed by shipping a different checkpoint. low_vram is the one mode a small-card user reaches for, so this drops the flag rather than the mode. offload_flags takes vae_on_cpu, defaulting True so no other family changes, and the H3 native path passes False. --offload-to-cpu and --clip-on-cpu still apply, which is where the saving actually is: the denoiser dominates, and with --offload-to-cpu the whole model peaks at 13.14 GiB. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Point the H3 GGUF pick at the unsloth mirror Main added test_curated_gguf_repos_are_unsloth_mirrors, which requires every curated video gguf_repo to live under unsloth/ so a one-click pick cannot 404 when a community repack is renamed or taken down. H3 was the one family still on a community repo. This was meant to be part of the merge commit but was left in the worktree, so CI on that commit still saw the old value. unsloth/MiniMax-H3-GGUF is still private and has to be made public before this merges, or the pick will 401. No CI check reads it. * Pin H3's native cfg-scale under test H3 is distilled and CFG-free: its empty unconditional prompt encodes to zero tokens, and the transposed tensor that produces trips GGML_ASSERT(!ggml_is_transposed(a)) in ggml.c. SIGABRT, exit 134. Measured: cfg 1.0 renders, cfg 1.5 and cfg 4.0 both abort. sd.cpp defaults cfg-scale to 7.0, so this is a crash a plausible refactor reintroduces by forwarding guidance the way every other family does. The native path already hardcodes 1.0 and the family sets supports_cfg = False, but nothing held either in place. supports_cfg only gates the diffusers path; the native path builds its own params. Checked the test fails when cfg_scale is changed to forward guidance, so it is not passing vacuously. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin diffusers by source archive instead of git, so macOS installs work The macos-15-intel leg fails deterministically, not flakily: process didn't exit successfully: `/usr/bin/git init` (exit status: 1) --- stderr xcode-select: note: No developer tools were found, requesting install. That runner has no Xcode, so /usr/bin/git is the developer-tools shim and exits 1 for everything. uv needs a working git to resolve a git+https dependency, so the diffusers pin this branch added cannot install there at all. Main is unaffected because it depends on plain diffusers. GitHub serves the same commit as a source archive, which uv installs with no git involved. Verified by stubbing git to fail exactly the way the macOS shim does: the git+ form reproduces the CI error, the archive form installs diffusers 0.40.0.dev0 from the same SHA with MiniMaxH3Transformer3DModel present and exported. Also drops a full clone of diffusers from every install. * Allow the diffusers source build on the clean-machine legs With the archive pin the macOS leg gets past `git init` and installs, but then trips the nobuild guard: built from source: diffusers -- these must resolve to wheels on a clean machine There is no wheel to resolve to. MiniMax-H3 support is not in any diffusers release, so this branch has to pin a commit, and neither a git URL nor a source archive can produce a wheel from an index. Added to the same allowlist that already carries triton-kernels for the same reason. Checked against the bar the comment there sets: diffusers builds with plain setuptools, declares no ext_modules, and its tree has zero .c/.cpp/.pyx/.rs/.cu files and no shipped binaries, so the PEP 517 build is a pure-Python copy step and needs no toolchain. The separate compiler-invocation check in both scripts is untouched and still fires if one is ever needed. Verified the allowlist logic still rejects a non-allowlisted source build (a log building both diffusers and numpy reports only numpy). Remove this entry once a diffusers release carries H3 and the requirement goes back to a version specifier. Noted in both scripts. * Baseline the hf-hub retry loop reopened by the 1.x upgrade `pip scan-packages :: hf-stack` fails on this branch with 1 unbaselined CRITICAL: C2 polling/beaconing loop detected Package: huggingface-hub File: huggingface_hub/utils/_http.py L461: while True: This is on us, not upstream drift. Main pins huggingface-hub==0.36.2; this branch needs >=1.23.0,<2.0 because diffusers at the pinned commit requires it, so the resolved version moves 0.36.2 -> 1.27.0. The baseline already carries this exact file and check at L462 and L298, from earlier versions. It did not carry over because the key hashes the matched code, not the line number, and the surrounding code changed across the major version. That is the guard behaving correctly: changed code in a baselined file reopens for review rather than staying suppressed. Reviewed it rather than just re-suppressing. L461 is the retry loop in `http_backoff`: bounded by `nb_tries > max_retries`, the URL comes from the caller, there is no hardcoded endpoint and nothing is exfiltrated. Same benign construct as the entries it replaces. Entry generated with the scanner's own _evidence_hash rather than hand-written, and inserted beside its siblings so the diff stays 8 lines. Verified: with the baseline the scan is exit 0 with 4 suppressed, without it exit 1, so the guard still bites. * Lift the macOS-arm huggingface-hub cap that this branch made unsatisfiable `mac macos-15 / trace / file` fails on this branch and passes on main. uv reports: No solution found when resolving dependencies: Because you require huggingface-hub>=0.34.0,<1.0 and huggingface-hub>=1.23.0,<2.0, your requirements are unsatisfiable. This branch moved base.txt, no-torch-runtime.txt, studio.txt and constraints.txt to hub >=1.23.0,<2.0, because diffusers at the pinned commit requires it, but left the flat <1.0 cap in overrides-darwin-arm64.txt. That file is macOS-arm only, which is why only the mac legs see it and Linux and Windows stayed green. The failure then presents as something else entirely: uv gives up, the installer falls back to pip, and the clean-machine trace fails on "installer invoked toolchain: rustc" rather than on the resolution. The cap's own comment explains it exists so the resolver can never pair hub 1.x with a pinned transformers 4.57.6 / hub 0.36.2 stack. That is still true below python 3.10 and still capped there. At 3.10 and above this branch is on transformers 5.5.0 and hub 1.x, so the premise is gone, and mlx-audio's own >=1.0 floor is satisfied by the 1.x window anyway. Checked by collecting every hub specifier that applies per Python version across all five files: py3.9 resolves to 0.36.2 as intended, py3.13 to 1.23.0/1.27.0. Before this change py3.13 resolved to nothing. * Make H3's native download use the repo the family advertises The curated-mirror test main added only inspects VideoFamily.gguf_repo. H3's native path does not read that field: video_minimax_h3.py has its own H3_GGUF_REPO constant, used for both the transformer and the Qwen3-VL encoder. So pointing the family at the unsloth mirror in the previous commit satisfied that test while the actual one-click download still came from a community repack, which is the exact failure the test exists to prevent. Pointed the constant at the same mirror and added test_the_h3_native_repo_matches_the_family_gguf_repo to pin the pair, so the two cannot drift apart again. Verified it fails when the constant is put back to leejet, so it is not passing vacuously. The mirror now carries the Qwen3-VL encoder quants alongside the denoisers, byte-identical in size to the community ones, so this repo alone satisfies both of h3_native_hub_files' hub entries. The encoder is part of MiniMaxAI/MiniMax-H3 itself (FL2VA/text_encoder) which we already mirror publicly at unsloth/MiniMax-H3 under the same licence, so shipping a quantization of it beside the denoisers is the same act. Also checked the encoder-tier routing survives the dynamic rung names: -UD-Q2_K_XL selects the Q2_K_M encoder and -UD-Q3_K_XL the Q4_K_M one, asserted in the new test. Updated the download-plan test, which hardcoded the old repo id. NOTE: unsloth/MiniMax-H3-GGUF is private. Unlike before, that now really does gate this: the native path downloads from it. It has to be public before this merges. * Pin H3's companion-checkpoint guard under test validate_h3_transformer_filename had no test. That mattered less when the denoisers lived alone; the mirror now ships the Qwen3-VL encoder quants in the same repo, so the picker lists both and a user can name either. Loading a 12-17 GB encoder as the transformer would fail deep inside sd-cli instead of at the boundary. The accept cases include the dynamic rung names on purpose. The guard is a prefix/suffix check and `-UD-Q2_K_XL` is a shape it had never seen when it was written; it happens to pass, and now that is asserted rather than assumed. Checked the test fails when the prefix check is dropped, so it is not vacuous. * Record why H3 drops --vae-on-cpu, now that the abort is fixed The comment justified the drop entirely by an sd.cpp abort. That abort is fixed in the Unsloth fork, which would have made this look like a stale workaround to revert once the fix reaches the pinned prebuilt. Measured on a build carrying the fix, 640x384, 25 frames, 4 steps, q4_K, with --offload-to-cpu --clip-on-cpu already applied: adding --vae-on-cpu moved peak VRAM 12.42 -> 12.42 GiB and wall time 20.9s -> 100.4s. Under --offload-to-cpu the peak is set by the streamed denoiser, so the flag saves nothing and costs 4.8x. It stays off on its own merits. * Pin the sd.cpp prebuilt that actually renders MiniMax-H3 The pin was master-812-ea7f0c8, a stock upstream build, and on a stock build H3 does not work: it aborts on the default --cfg-scale, aborts again on --vae-on-cpu, and its 1-D norms are quantized into an output uncorrelated with its own bf16 reference (LPIPS 0.981). The Studio side worked around the first by pinning cfg to 1.0 and the second by dropping the flag; the third had no workaround on the consumer side at all. All three are fixed in unslothai/stable-diffusion.cpp and open upstream as leejet/stable-diffusion.cpp#1861, #1862 and #1863. The mirror's prebuilt pipeline now applies them on top of the aged upstream tag it already builds, and marks such a build with a -u<id> suffix naming the patch set, so master-813-bfbef5b-u0665242 is upstream master-813-bfbef5b plus those three patches and nothing else. The patches are deleted once upstream releases them, at which point this pin goes back to a plain tag. Verified on the published Linux x86_64 asset, not on a local build: both new error strings are in the shipped sd-cli, and running it on a q4_K H3 denoiser without --mode vid_gen now exits 1 with the instruction instead of core dumping on a ggml assert. test_video_backend's fake engine returned the old tag as its version string, which read like a second pin; it only needs a non-None value, so it now says so. * Close two gaps the H3 mirror switch opened Both are consequences of the two preceding commits, found in review. The prebuilt pin is now mirror-only (master-813-bfbef5b-u0665242), and _resolve_with_fallback still asked leejet for it. That request is a guaranteed 404 by construction, since the -u<id> suffix marks a build only the mirror makes, so it was a wasted round trip on every install. Worse, when the mirror genuinely cannot serve a host the fallback lands on leejet's latest, which has none of the H3 fixes. For every other model that is the right trade, better a stock native engine than none. For H3 it is not visible: it aborts on the default cfg-scale, aborts on --vae-on-cpu, and a blanket --type renders a broken video rather than failing. A user who saw only the generic 'falling back to leejet' line had nothing connecting that to the output. It now says so. Second, hub/utils/gguf.py filtered H3 companion GGUFs by the old community repo id only. The mirror the family and catalog now advertise carries the Qwen3-VL encoder quants beside the denoisers, so a 12 GB text encoder was being aggregated as if it were a selectable transformer quant. Both bundle repos are now recognised, case-insensitively, and the cache-dir match follows. Tests are mutation-verified rather than assumed: restoring the upstream 404 attempt fails the ordering test, removing the H3 warning fails the fallback test, and dropping the mirror from the bundle set fails three. The ordering test deliberately makes the mirror serve nothing, because with the mirror serving the first attempt succeeds and the upstream attempts are never reached, which made an earlier version of it pass under its own mutation. A third test pins the native loader's H3_GGUF_REPO to the bundle set, since those live in different files and a future repo move that updated only the loader would silently reintroduce the same aggregation bug. * Exclude MiniMax-H3's small-M projections from int8 _INT8_FAMILY_EXCLUDE_NAME_TOKENS has entries for qwen-image and hunyuanvideo-1.5 but none for minimax-h3, and H3 needs one for the same reason they do. H3's adaLN projection is named adaln_proj, which no token in the generic list matches: 'norm' is the closest and does not appear in the name. On the dense checkpoint that projection is Linear(2688 -> 96768), so it clears min_features = 512, gets quantized, and then runs at M = 1. Inductor lowers int8 matmul to _int_mm, which requires M > 16, so it raises 'self.size(0) needs to be greater than 16, but got 1' at the first denoise. The offline prequant builder bakes it in happily, which is exactly the drift the exclude list exists to prevent for Flux and Qwen. The pruned-modulation form hides this rather than fixing it: there adaln_proj is Linear(8 -> 96768) and falls under min_features anyway. So this exclusion is what makes the DENSE path correct and is a no-op on the pruned one. context_embedder and token_refiner are added for the same reason hunyuanvideo-1.5 excludes its text stream. Measured at M = 10 text tokens against the video stream's thousands, they are 3.47% of GEMM time even in the slow eager int8 path, so leaving them bf16 costs nothing measurable. This is what made an earlier measurement conclude int8 does not work on H3. It does: on the pruned form int8 compiles and is 4.24% +-0.54% faster than fp8 at identical memory, paired over 12 renders. * Say which H3 component could not be downloaded, and why H3 pulls four files from two repos, and the Hub returns the same 'Repository Not Found ... make sure you are authenticated' for a repo that does not exist, one that is private, and one your token does not cover. A user reading that has no way to tell which of the four failed, and the wording points away from the real cause whenever the repo exists but is not public. That is the state the GGUF mirror is in today: it is unpublished, so picking H3 fails with a message suggesting the user fix their token, which will not help. This replaces it with the repo, the component, and the actual remedy, and says the other components are unaffected so the failure is not read as total. Gated repos get different wording, since accepting a licence is a different action from waiting for a repo to be published. Anything that is not a recognised access error is passed back unchanged rather than reworded, so a timeout or a full disk still reads as itself. The helper returns the exception instead of raising, so the caller keeps raise-from and the original traceback survives. Mutation-verified three ways: rewording every error (timeouts included), fixing the component name to 'denoiser', and giving gated repos the private wording each fail the test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop MiniMax-H3 holding two copies of its video VAE Two thirds of an H3 render's peak is not activations. Measured at 640x384 across 124 frames, a 20.25 GB int8 denoiser peaks at 36.96 GB, and the gap is almost all weights: the video VAE alone is 10.42 GB because diffusers pins it to float32, and a further 4.91 GB is autocast's own float16 copy of those same weights. A memory snapshot puts 92.9% of the transient in 437 blocks allocated from nn.Linear, largest 67.1 MB, which is the decoder's [2048, 16384] SwiGLU projection in float16. MiniMaxH3VideoDecodeStep wraps vae.decode in torch.autocast(float16), and autocast caches every weight it casts for the lifetime of the region, so the float32 original and its float16 twin are both resident through the whole decode. Storing those weights as float16 up front makes the cast a no-op and removes both. This is not an approximation: x.to(float16).to(float16) is x.to(float16), and the four regression tests check that on a real matmul rather than on the reasoning. The audio VAE decode is not under autocast, so it keeps float32. t2va starts from noise and never encodes, so vae.encoder and vae.quant_conv go too. That part is gated on the workflow name rather than dropped unconditionally, because an image-conditioned workflow needs them. Measured 36.96 -> 28.37 GB peak with the encoder drop and the pre-cast, 28.27 with expandable_segments as well, over 5 prompts x 2 seeds. Speed is unchanged (-0.05% +-0.86% eager, -0.34% +-2.23% compiled), and every arm hashes identically to its control on latents, audio_latents, frames and audio. The estimator's base still reads 68.5 GB. That figure was measured on the bfloat16 modular components, not the int8 arm above, so it stays put until it is re-measured in the same configuration rather than adjusted by arithmetic. * Pad MiniMax-H3's small-M int8 linears instead of leaving them dense torch._int_mm asserts self.size(0) > 16. torchao's eager path never trips it (safe_int_mm falls back to a widened matmul), but inductor lowers the same quantized linear straight to _int_mm, so any quantized Linear invoked at a small activation row count crashes under torch.compile. Until now the fix was to leave those linears dense bf16, which on H3 meant excluding context_embedder and both token_refiner blocks: 13 linears, 798M parameters, 0.80 GB of weights the int8 checkpoint was not allowed to touch. Pad instead. diffusion_quant_pad.PadToMinM pads the flattened row count up to 32, runs the GEMM and slices the result back, so the module compiles with no change to the quantization config and the caller's rows come back bitwise unchanged. Verified bitwise on all 65 (module, M) cases across H3's 13 linears at M = 10, 13, 14, 17, 19, on real torchao-quantized weights; compiling those same modules unpadded raises the _int_mm assert at M = 10, 13 and 14. Two properties carry that exactness and both are asserted rather than assumed. The pad rows replicate row 0, not zeros: an all-zero row has amax 0, so the activation quantizer divides by zero. And the activation scale must be per row, so each kept row's scale comes from that row alone; a quantized Linear whose granularity cannot be proven per row raises instead of being quietly skipped, because a half-padded transformer compiles on the wrapped modules and crashes on the rest. Everything below pad_to normalises to pad_to rather than only what is below the floor, so one inductor graph covers every prompt length in the range. H3's seven eval prompts run at M = 10..19, which straddles the floor, and padding only to 17 would leave three shapes behind. The wrapper reparents the Linear, so it runs after quantize_ on the runtime path and after load_state_dict on the prequant one. The offline builder drives quantize_ directly and saves the state dict, so it never sees a wrapper; PadToMinM is also state-dict transparent as a second line of defence, saving and loading under its own prefix so a wrapped transformer still writes context_embedder.weight. Scoped to minimax-h3. qwen-image, qwen-image-edit and hunyuanvideo-1.5 have the same small-M shape but published int8 prequant checkpoints whose metadata bakes the current exclusion set, and _validate_checkpoint compares that set against exclude_tokens_for_scheme, so they move only together with a rebuild. adaln_proj stays excluded for a different reason: on the dense checkpoint it is Linear(2688 -> 96768) and runs at M = 1, while on the pruned form it is Linear(8 -> 96768) and falls under min_features anyway (verified: the filter rejects all 51 for min_features). Measured on B200, 640x384 x 124 frames, 4 steps, 7 prompts x 2 seeds, the two arms alternated within each cell so drift on a shared box cancels: checkpoint 21.052 -> 20.254 GB (-0.798, -3.8%) transformer 21.051 -> 20.253 GB (-0.798) load peak 21.137 -> 20.336 GB (-0.801) render peak 37.766 -> 36.966 GB (-0.800, -2.1%) step time +0.0597 s +-0.0057 eager, +0.0533 s +-0.0086 compiled The time is torchao's un-fused eager activation quantization on 13 modules that run ONCE per render, so it is a fixed cost rather than a per-step one, and it does not scale with steps or resolution. Compiling those modules alongside the blocks removes it: +0.0034 s +-0.0040, no detectable difference. Quality is unchanged as far as n = 14 can resolve: against the same bf16 twin the padded arm sits +0.0070 +-0.0151 LPIPS from the excluded one, which rules out a degradation larger than 0.022. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Classify the MiniMax-H3 GGUF bundle as video in the cached inventory The H3 GGUFs are stable-diffusion.cpp conversions and carry no metadata keys at all (kv_count 0), so general.architecture is absent where the LTX-2 and Wan video GGUFs declare ltxv/wan. _arch_to_task therefore left the downloaded repo with no task, which drops it from the Video picker's On Device list and hands it to chat as a plain GGUF. Key the two bundle repo ids before the arch is consulted. * Release the VIDEO GPU claim when H3 native falls back to the CPU build On a CUDA/ROCm host /video/load acquires the VIDEO arbiter owner because the resolved device target is not CPU. _run_load_h3_native then asks for an accelerator-matched sd-cli, and the pinned prebuilt release publishes no Linux CUDA/ROCm archive, so ensure_sd_cpp_binary returns None and the load commits the CPU build with native_device = cpu. Nothing dropped the VIDEO claim, so the next chat or image acquire evicted and unloaded an H3 runtime that holds no VRAM. Release the claim once the CPU fallback is committed, through release_if so the token check is atomic against a newer load that already took ownership. Mirrors the CPU-only native release /images/load already does. * Video: protect the native H3 companion repos, cancel the modular denoise, forward the hub token - VideoBackend.loaded_repo_ids() publishes the repos the committed native H3 runtime re-reads every generation (Qwen encoder + both VAEs), and the delete-cached guard consults it, so deleting a companion under a loaded model is refused. - The H3 modular workflow no longer falls back to a null progress context: the denoise loop drives pipe.scheduler.step once per step, so the existing wrapper gives it the same per-step progress and cancellation the other callback-less pipeline gets. - load_components() gets the Settings token, so gated/private component loads are not issued anonymously. * Reject stable-diffusion.cpp builds that predate MiniMax-H3 support ensure_sd_cpp_binary hands back whatever find_sd_cpp_binary locates and only probes that it runs, so an install upgraded from an older Studio kept serving its pre-H3 managed sd-cli. The H3 load's only gate is SdCppEngine.version(), which that binary passes, so the load reported ready and the failure surfaced on the first generation, after the whole bundle had downloaded. Gate the H3 path alone on the capability instead of a version string: upstream added --ref-video and the other H3-only options in the same commit that added MiniMax-H3 (leejet/stable-diffusion.cpp#1854, master-812-ea7f0c8), and the release prebuilts report 'version unknown, commit unknown' because they are built without a .git directory, so --help is the only usable signal. Image generation keeps accepting any user-supplied build. A stale copy under the installer-owned root is removed so the pinned prebuilt reinstalls; a build the user supplied is left in place and the load fails naming it, the same ownership split _usable_or_discard_managed makes. A --help that cannot be read means 'cannot tell', never 'no H3'. * Distinguish a reused CPU sd-cli from an accelerator build on an H3 load On a Linux CUDA host the first H3 load installs the CPU prebuilt through the fallback and correctly commits native_device = cpu, because the pinned tag publishes no Linux CUDA, ROCm or Vulkan asset. Every later load then calls ensure_sd_cpp_binary(accelerator = cuda), which finds that same CPU binary and returns it without looking at what it was built for, so the fallback was skipped, native_device stayed cuda, and Studio applied GPU offload policy and retained the VIDEO gpu_arbiter claim while sd-cli ran wholly on the CPU. A later chat or image acquire then evicted an unrelated GPU model. Fall back on what the binary offers rather than on whether one was returned: sd-cli --list-devices prints one name/description line per available ggml backend device, so a CPU-only build answers with CPU alone. The second load now reaches the same cpu conclusion as the first, which is what lets the existing release_if drop the stale claim. Unreadable output, or an older build that rejects the flag, keeps the GPU: neither says the accelerator is missing. * Guard the H3 companion repos while a native video load is downloading _run_load_h3_native downloads from repo_id, the H3 GGUF companion and the H3 component repo, but the in-flight state only carried repo_id and base_repo. The cached-model delete guard reads loading_repo_ids(), so it allowed deleting Comfy-Org/MiniMax-H3, and the GGUF companion when the load comes from another mirror or a local file, while those files were still downloading, which fails the load. Carry the companions on the loading state, the way the image backend's _SdLoading already does, and publish them from loading_repo_ids(). This is the in-flight twin of loaded_repo_ids() and covers the same repos. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep curated Recommended rows searchable and give them their metadata Two things went wrong with the curated video models the Recommended list paints from the catalog rather than from a live Hub listing. Search dropped them. The Recommended search matched the query against `recommendedIds`, which filters out every id already on disk because a downloaded model gets its own On Device row. The unfiltered Recommended list does not filter that way: it renders the curated seeds and badges the downloaded ones. So a curated model was visible in the list and unfindable by typing its name the moment it was downloaded, and only a live listing row could bring it back, which a repo the listing does not return never gets. `searchableRecommendedIds` unions the seed ids with the listing ids, seeds first, deduped case insensitively (the HF cache lowercases repo ids), so both lists agree on what exists. Rows rendered bare. Everything past the id came from the listing alone, so a curated row the listing never returns showed no parameter chip and no capability glyph while its neighbours showed both. The catalog now carries the two facts nothing else can supply: `totalParams` per artifact and `capabilities` per group, read through `curatedTotalParamsFor` and `curatedCapabilitiesFor`. Both are fallbacks only. A listing row wins wherever there is one, because real tags and a Hub-reported total outrank anything hand written here. The parameter counts are measured, not guessed. The MiniMax-H3 figure is the sum of the tensor shapes in the BF16 GGUF of the FL2VA denoiser that repo publishes; the LTX-2.3 figure is what the Hub reports for its repo, carried so the row looks the same offline or rate limited. One more inconsistency fell out of the same place: `searchRowFits` hides anything it cannot size (`requireKnown`), and it could not size a curated repo with no listing row and no "<n>B" token in its id, so turning on "Fits on device" hid from search a row the unfiltered list still showed. It now falls back to the curated total the same way it already fell back to the curated size. Covered by studio/frontend/tests/recommended-curated-row-metadata.test.ts: 15 assertions over the search pool, the two catalog lookups, the fit check, and the four picker call sites that read them. * Load MiniMax-H3 from a hosted pre-quantized denoiser Video families had no way to reach a hosted pre-quantized DENOISER. VideoFamily carried gguf_repo and te_prequant_repos (the text encoder) only, and nothing in video.py consulted a denoiser table, so a MiniMax-H3 load either pulled the full 66.3 GB bfloat16 DiT or nothing at all. The image side already solves this with DiffusionFamily.prequant_repos plus family_prequant_repo(), so this follows that shape rather than inventing a second one. Four parts. VideoFamily gains prequant_repos, prequant_variant_repos and prequant_subfolder, resolved by video_family_prequant_repo() / video_family_prequant_schemes(), mirroring the image resolver. The registries stay separate, as the module header requires, so the base-id normaliser is local rather than imported. The shared resolver learns an optional subfolder. The hosted video checkpoints nest theirs one level down instead of keeping it at the repo root, and the prefix has to reach BOTH candidate names or the primary 404 is followed by a second one and the load silently falls back to the dense download. Always a literal forward slash: these are Hub repo paths, and a Windows join would miss the cache. The cache and download plumbing already handled a nested name, so only the filename builder changed, and every existing call site is byte-identical. The modular workflow builds its denoiser through its own component loader, so there is no dense module to quantise in place. A hosted checkpoint is therefore the only way to run that transformer quantized, and pre-seeding it with update_components() before load_components() is also what stops the dense download: load_components(names=None) skips a component whose attribute is already set. Passing names= instead would have forfeited the workflow's own block pruning, which is what already avoids the 61.7 GB Ref2VA transformer. Those checkpoints carry the pruned adaLN, where the modulation is a rank-8 affine factorization of the time-embedding curve plus a shared table, and roughly 40% of the released model's parameters go. Against the base repo's dense config the model is four keys short, one over and fifty-one shapes wrong, so the strict load fails and the checkpoint is unloadable by every route. video_minimax_h3_adaln.py reshapes the model between from_config and load_state_dict: table lookup with interpolation instead of the timestep MLP, no SiLU (the table already holds the activation's own output projected onto the basis), and the modulation cast to the block stack's dtype, without which the first quantized matmul dies on mismatched dtypes. Bound per instance, so a dense load in the same process is untouched. Finally the refusal becomes honest. A single-file pick on a modular family used to reach the loader only after ~98.7 GB had downloaded AND after the resident pipeline had been evicted to make room for it, because download-plan returned 200 and validation passed. Both refusals now run in validate_load_request, ahead of the diffusers availability probe so they still fire where diffusers cannot be imported, and each names what to pick instead. download-plan forwards transformer_quant to validation and to the plan, without which the quant-keyed refusal never fires on the route that stages the download and the plan stages shards the load never opens. * Point both MiniMax-H3 schemes at one hosted pre-quantized repo The two hosted pre-quantized denoisers were split across two repos with the checkpoint nested one level down, so reaching them needed a mechanism the image side has never had: a VideoFamily.prequant_subfolder field, a prequant_subfolder_prefix() normaliser, and a subfolder keyword threaded through both resolve_prequant_source() and usable_prequant_source(). Both schemes now live in ONE hosted repo, at the root, named <Model>-<SCHEME>.pt. That is the layout every image-side prequant repo already uses, and it is exactly what prequant_repo_filename() builds unaided, so the whole mechanism goes. Match an existing convention and the code should shrink, not grow: -22 lines in diffusion_prequant.py, -6 in video_families.py, -6 in video.py, with no new concept to carry. Landing on the primary name also fixes a memory-planning under-credit. cached_checkpoint_path() deliberately credits only the PRIMARY filename, so that a cached legacy artifact cannot pin a stale name once a repo ships the real one. While these checkpoints were published as transformer_<scheme>.pt the primary never existed: every hit came through fallback_filename, and planning therefore read an already-cached checkpoint as "this would have to download" and handed the pick to GGUF. The primary is now the published name, so the probe hits it. fallback_filename stays. It still covers repos that have not been renamed, and dropping it is a separate decision from this one. Tests: the subfolder-prefix cases are replaced by the naming they now guard -- both schemes resolving to one repo, the primary resolving to a root-level <Model>-<SCHEME>.pt with no directory component on any platform, the cache probe being asked for that primary name, and the repo's own scheme suffix being stripped and replaced rather than carried through. Five mutations run, each caught by the named test and reverted: two repos again (M1), the primary nested under prequant/ again (M2), the suffix strip removed (M3), the cache probe keyed on the fallback (M4), the fallback name dropped (M5). * Add MiniMax-H3 image and reference video conditioning * Improve H3 finalization progress * Report real sd.cpp progress on the Video page instead of a frozen 0 of 30 A native (GGUF) video generation reported phase "denoise", step 0/30 for its entire run and then flipped straight to "completed". Two separate things were wrong, and the progress endpoint could not move until both were fixed. The parser looked for r"(?:step|sampling)\D+(\d+)/(\d+)". sd-cli's sampling bar contains neither word. It prints |=========> | 7/30 - 21.50s/it so nothing ever matched. Anchor the pattern on the bar and on the trailing speed unit instead. A bare "n/m" is deliberately not enough: an unrelated ratio in some other log line must not drive the progress bar. The reader also delivered every redraw one step late. sd-cli redraws in place, and its carriage return LEADS the record while the newline only arrives on the final step, so a reader keyed on CR/LF cannot produce step 1 until step 2 has been flushed. Treat the erase-to-end-of-line that closes each redraw as a terminator too, and read the pipe with buffer.read1 so a record that carries no newline is not stuck behind a blocking readline. Escapes are stripped before a record reaches on_log or the error tail. Streams without a raw .buffer keep the old line iteration. The same bar shape is printed by three different things, so the parser now tells them apart rather than reporting whichever came last. Weight load prints it with a byte rate, and tiled VAE decode prints an identical s/it bar counting TILES: without a guard a run finished sampling at 30/30 and then jumped backwards to "step 1/16". Load and decode are real work with no sampling step, so they report their own phase and a null step rather than a fake 0 of 30. ETA is measured from the first step, not from job start, so the one-off weight load is not charged to every remaining step. Verified end to end against a real CUDA sd-cli MiniMax-H3 generation: the step advances 1..6 over 6.2s..9.7s of wall clock, load and decode are reported as themselves, and the ETA tracks. * Read sd-cli's in-place progress redraws so the Video bar moves during sampling The native H3 progress bar had two independent causes and fixing either alone changed nothing observable. The bar pattern is now correct, but the reader still was not. sd-cli redraws its sampling bar in place: one printf per step shaped "\r<bar> <n>/<total> - <speed>\033[K", with a newline only on the final step of a phase. The drain loop did `for raw in proc.stdout`, which terminates on LF only, so every redraw sat in the buffer until the next one's carriage return arrived and the last one until sampling was already over. The Video page saw nothing. Split the raw pipe into records on CR, LF, or the trailing erase-to-end-of-line, reading through buffer.read1 with an incremental UTF-8 decoder so a multi-byte character straddling two reads survives, and strip the CSI escapes before the record reaches on_log or the error tail. Streams without a raw .buffer (test doubles, non-pipes) fall back to line iteration. The new backend test drives the real byte stream through both halves, one flush per read, and asserts each step is visible on the read that carried it rather than one redraw later. * Give MiniMax-H3 first and last frame conditioning in the video backend MiniMax-H3's released transformer is the FL2VA one: text-to-video is the same checkpoint run with no keyframes. Studio only ever ran it text-only, so the Video generate request had nowhere to attach a reference frame. The load used to prune the block graph to t2va. That argument prunes STATICALLY, so an fl2va-pruned pipeline runs the keyframe blocks on every request and cannot serve a text-only one at all: it raises packing an empty conditioning list. The load now keeps the whole auto graph, which selects per request, and bounds load_components to the keyframe workflow's component set instead, so the 61.7 GB Ref2VA partition is no more loaded than before. Measured against the released checkpoint: a text-only request through this pipeline is bit-identical, video and audio, to the same request through a t2va-pruned one. A keyframe is a geometry anchor, not just conditioning, so the canvas comes from its aspect ratio through the released arithmetic (768 short edge, area capped at 768x1344, both axes rounded to 32) rather than from whatever resolution preset was selected. An arbitrary size produces a garbled clip rather than an error. sd.cpp already implements the same conditioning, so that side is the existing --init-img / --end-img flags with the frames staged as PNGs. Only MiniMax-H3 declares the capability, and status reports it, so Wan and LTX do not grow a control that does nothing. * Cover the MiniMax-H3 keyframe path with tests Registry (which families declare it, and the canvas a keyframe resolves to), the load wiring (whole block graph, component set still bounded, VAE encoder kept), request handling (decode, refusal, canvas override, what reaches the pipeline call) and the sd-cli argv. * Add the reference-frame controls to the Video page First and last frame pickers, shown only for a family whose status declares keyframe conditioning, so Wan and LTX are unchanged. The Images page's source picker moves to a shared component rather than the Video page growing a second upload path; both send the same data URL to the same backend decoder. While a frame is attached the Resolution preset is disabled and says why: the frame's aspect ratio resolves the clip's size, the way the model itself does. The gallery recipe records which ends were pinned. * Check the keyframe canvas against the pipeline's own resolver The canvas rule is a checkpoint contract, so pin it to the released implementation rather than only to hand-written expectations. Skipped where diffusers does not ship MiniMax-H3, which is most runners. * Revert the standalone H3 keyframe implementation oobabooga/unsloth#121 covers first frame, last frame, first-and-last, Ref2VA and the canvas rule, and it reached the same load construction independently. Two implementations of the same feature on one branch is worse than either, so this takes mine back out and leaves the branch ready for that work to land whole. The one finding worth keeping from it is already reflected there: passing workflow= to ModularPipeline.from_pretrained prunes the block graph statically, so the pipeline must be built unpruned and only load_components bounded. * Keep the pre-quantized MiniMax-H3 denoiser resident so a generation can run Loading H3 with a hosted pre-quantized denoiser worked, but every generation died on its first denoise step: Attempted to set the storage of a tensor on device "cuda:0" to a storage on different device "cpu". This is no longer allowed; the devices must match. ComponentsManager.enable_auto_cpu_offload parks every component on the CPU and moves each one onto the accelerator inside its own pre_forward, that is from within the block that is already executing. The text encoder and the VAEs survive that; a torchao-quantized denoiser does not, because the device change reaches return_and_correct_aliasing, which tries to alias a CPU storage to an accelerator tensor. Moving the same module at load time, outside any executing block, works. So place it once at load and take it out of the offload rotation: drop its hook, unlist it from the other components' eviction candidates, and move it. Everything else is unchanged, and the encoder and VAEs still offload around it. Keeping it resident is what asking for a quantized denoiser buys in the first place: the hosted checkpoint is about 20 GB against 66.3 GB dense. Verified end to end on a B200: MiniMaxAI/MiniMax-H3 loaded with the hosted fp8 denoiser, then a 1280x768, 124-frame clip generated from a start frame in 167s. The clip's first frame matches the supplied image and the motion is coherent. * Apply the pinned Diffusers revision on a fresh install, not just an update MiniMax-H3 needs a Diffusers revision newer than any published release, and Studio refuses to load it otherwise. The pin was in studio/backend/requirements/base.txt, and a clean install.sh run still ended up on diffusers 0.39.0 from PyPI, every time, with nothing in the log to say so. base.txt is never installed by install.sh. install.sh installs unsloth itself, whose own metadata pulls a diffusers release in transitively, and then runs install_python_stack.py with SKIP_STUDIO_BASE=1, where the base-packages step is a bare `pass`. So the pin applied on `unsloth studio update` and on the no-torch path (install.sh installs no-torch-runtime.txt directly) and was dead on exactly the path a new user takes. Reproduced on a clean install into a throwaway prefix before and after: 0.39.0, then 0.40.0.dev0 with MiniMaxH3Transformer3DModel present. The revision now lives in its own diffusers-pin.txt, installed by a step that sits outside every skip_base / NO_TORCH branch and after every other requirements file, so nothing left in the run can re-resolve diffusers back to a release behind it. No forced reinstall is needed: a direct URL requirement is not satisfied by a resident registry install, so the step is a no-op once the environment is already on the pin. tests/studio/install/test_diffusers_pin.py holds the shape in place: exactly one requirements file may name diffusers, the pin must be a full commit sha rather than a moving ref, the install step must sit at function top level rather than under a conditional, and it must come after every other requirements install. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: let the Video and Images pickers see a local diffusers pipeline A model already on disk never reached the pickers' On Device list unless it happened to keep a weight file beside a root config.json. Every image and video model downloaded as a pipeline keeps its weights in component subdirs under a root model_index.json instead, so the hub inventory scan behind /api/hub/local rejected it. Pointing Custom Folders at one was worse than empty: the LM Studio publisher walk descended into the pipeline and offered vae, transformer, text_encoder and audio_vae as four separate models, none of them loadable. Teach the hub scanner the same pipeline-root test routes/models.py already applies, in the three places that only accepted a root config plus loose weights, and keep a pipeline row through the custom-folder format filter: the layout has no loose weight to classify, so the row is "unknown" by construction rather than by fault. * Cover the local diffusers pipeline scan with tests * Pin what the pipeline exemption must not let through Three gaps in the cover added with the scan change, each found by mutating the fix and watching the suite stay green. The custom-folder format filter now waves a row through on its shape, and nothing said what it still has to reject: replacing the whole predicate with True passed. A folder holding a config.json and no weights, which an aborted download leaves behind, reports the same "unknown" format and no loader can start it, so it pins the boundary. The predicate is applied to every row the filter did not already accept, and a row's path can be a GGUF file rather than a directory. A missing path, a file, and a directory whose model_index.json is itself a directory must answer False rather than raise, because an exception there fails the scan and empties the picker. The publisher walk was only covered one level up. Adding the model folder itself as a scan folder is the obvious thing to do and used to publish vae, transformer and text_encoder as three models. * Drop the unused H3_TASK_KEYFRAMES import from the video backend video.py only branches on H3_TASK_REFERENCES; the keyframe constant is read from video_minimax_h3 directly by the tests that need it. The hoisted-import safety net in Source lint flags the unused name as a blocker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage the hosted pre-quantized H3 denoiser in the video download plan The plan already drops the dense transformer shards whenever a hosted pre-quantized checkpoint covers them, but nothing put that checkpoint back: an int8 or fp8 H3 stage skipped 66.3 GB of base shards and added none of the 20.25 GB artifact the load actually opens. The byte total under-reported the stage by the size of the checkpoint, the disk preflight cleared a volume that could not hold it, and an offline stage completed without the one file the load needs. _denoiser_prequant_hub_files mirrors the pre-cast encoder helper: it resolves the family's hosted checkpoint, confirms the file really exists on the Hub, and prefers the repo-root name over the legacy scheme name in the same order the load tries them. An unreachable repo is logged and yields no files, so a gated or renamed artifact keeps the dense shards instead of sinking the plan. The progress-bar estimate is deliberately left alone: it counts cached bytes for the checkpoint and base repos only, so adding a third repo there would leave the bar permanently short of 100 percent. * List both H3 denoiser partitions in the picker, not only FL2VA The bundle-repo filter accepted only minimax_h3_fl2va*, so every published minimax_h3_ref2va* quant was hidden from the variant listing. The loader disagrees: validate_h3_transformer_filename accepts either partition, on the grounds that which one is picked IS the task, and h3_transformer_task routes Ref2VA to the reference-video workflow this PR adds. The community bundle repo publishes three Ref2VA quants today, so the reference path was unreachable from the remote catalog. Accept both prefixes from one shared tuple and keep excluding the Qwen3-VL encoder and VAE companions, which are never picks for either partition. The filter test asserted the old behaviour and is updated with it. * Bound H3 reference-audio decoding to the trained window The reference-video decoder already selects, resizes and refuses incrementally because the encoded size says nothing about the decoded size. The audio decoder did not: it appended every resampled block to a list and then allocated a second full buffer in np.concatenate, with no duration or sample cap. The route accepts 32 MiB of encoded audio, which is over half an hour of compressed stereo. That lands as roughly 1.9 GB of float32 and doubles again in the concatenate, and up to three references are accepted per request, so an ordinary long music or podcast file picked by mistake could exhaust the host before the background job even started. H3's reference window is 15 seconds, so anything past it is unusable rather than merely large. Refuse it while decoding, with the same shape of message the video guard uses, instead of decoding it first. * Refuse a quantized H3 reference load instead of seeding the keyframe denoiser The hosted pre-quantized checkpoints are FL2VA (keyframe) denoisers. Ref2VA shares their module shapes and the same base model, so resolve_prequant_source handed one back for a reference load, it passed every metadata check, and seeding it made load_components skip the real Ref2VA transformer. The request then generated from the wrong partition rather than failing, which is the worst of the three outcomes. The route accepts h3_task, so this was reachable from the public API even though the picker does not expose the choice yet. validate_load_request now refuses the pairing with a message naming the workable alternatives, in the same place it already refuses a scheme with no hosted checkpoint, and the modular loader drops to the released components if a direct call reaches it. Nothing changes for keyframe loads, which are what the checkpoints are. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Skip modular-workflow families in the dense text-encoder plan assertion The plan-unchanged sweep from main walks every family and asserts the dense budget plan_diffusion_memory received. MiniMax-H3 is the first modular-workflow family to reach that list, and load_pipeline dispatches to the workflow's own loader before the planner runs: each component is built by its own from_pretrained, so there is no single dense pipeline to budget and no plan call to assert on. Skip it the way the sweep already skips wan2.2-t2v-a14b. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope the paired-axes canvas rule to keyframe requests A half-specified width/height is long-standing, documented API behaviour: the backend resolves the missing axis from the family's default preset, in both validate_video_request_shape and _resolve_keyframes. Applying the new paired-axes rule as an unconditional request validator rejected those calls with a 422 before family validation ever ran, breaking existing LTX, Wan, Hunyuan and prompt-only H3 clients. The rule still holds where it means something: with a keyframe present the canvas is matched to the source aspect whenever either axis is missing, so the axis the caller sent would be silently discarded. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the CUDA sd-cli pin and translate a mirror-only tag upstream The merge with main brings in the accelerator-aware installer, whose fallback translates a mirror-only -u<id> pin back to the upstream release it was built from instead of skipping the upstream attempt. That is strictly better: skipping kept the round trip cheap but dropped the pin entirely on every host the mirror does not build, leaving them on upstream latest. test_a_mirror_only_pin_is_never_requested_upstream asserted the old shape, that the fallback settled for upstream latest. It now pins the new one: never the literal -u<id> string, the translated release instead, and no latest attempt at all because the translated pin succeeds. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-08-09 10:05:58 -07:00
Etherl	968d2e50ff	Studio: fix Windows desktop updates failing integrity checks (#8185) ... * Fix Windows desktop updater integrity checks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Exclude nested compiled cache packages * Tighten updater comments * Keep package discovery test dependency-free * Shorten the einx Windows pin comment * Drop the einx pin comment --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-08-09 01:13:37 -07:00
Daniel Han	310a4b5b40	Studio: keep the upgrade intent when the pip fallback runs (#8112) ... * Studio: keep the upgrade intent when the pip fallback runs pip_install runs uv and falls back to pip on any nonzero exit. The fallback built its command with _build_pip_cmd, which dropped --upgrade-package and its value because pip has no such flag. On the update path that made the fallback a no-op. install_python_stack passes --upgrade-package unsloth --upgrade-package unsloth-zoo with req = base.txt, and base.txt lists a bare unsloth-zoo and unsloth, so pip found both requirements already satisfied, installed nothing, and exited 0. _fail_if_install_damaged checks file integrity rather than versions, so nothing downstream noticed, and unsloth studio update reported success having upgraded nothing. This was not limited to the Windows in-use launcher that motivated #8109. Any uv failure reaches the fallback, including a network or index failure, so the same silent no-op was possible on Linux and macOS. Translate the flag instead of dropping it, and pin --upgrade-strategy to only-if-needed rather than relying on pip's default, since that default is what keeps the existing torch build from being re-resolved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-07 08:22:04 -07:00
oobabooga	5659a9c47e	Installer: keep status messages off the progress bar line (#8052) ... * Installer: keep status messages off the progress bar line * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix blank lines and stale progress state in PR #8052 Follow-ups on the centralised progress-line close: - Four ROCm messages still opened with a literal \n, which used to be the only line terminator. _safe_print() emits one too now, so inferred-gfx, Strix and gfx906 installs got a blank line. - _end_progress_line() caught only OSError, so a closed or detached stdout took down messages bound for stderr, including the manifest error paths. - install_python_stack() reset _STEP but not _PROGRESS_LINE_ACTIVE. Only _step() read it before; every _safe_print() does now, so an aborted run left a stray newline on the next run's first message. - _note() aligned to the value column in verbose mode, where there is no bar and no step line to align to. - Dropped the _end_progress_line() call that _safe_print() now makes itself. Tests: patch _HAS_COLOR in _render() so the layout assertions hold under FORCE_COLOR=1, plus AST guards for leading-newline messages and direct sys.stdout writes, and coverage for wrapping, verbose, colour, closed stdout and the entry-point reset. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten comments in the progress-line changes --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-08-07 01:20:47 -07:00
Long Yixing	d6b1b7dcf2	Studio: drop the mlx-lm 0.31.3 exclusion so current mlx-vlm resolves (#7061)	2026-08-07 02:41:04 -03:00
Tai An	0c12d6473f	fix(studio/rocm): don't install for a shadowing iGPU on mixed AMD hosts (#7776) (#7778) ... * fix(studio/rocm): don't install for a shadowing iGPU on mixed AMD hosts (#7776) On a board with both an AMD APU and a discrete Radeon, HIP enumerates the iGPU first, so _detect_windows_gfx_arch picked index 0 and the installer pulled the iGPU's wheel family -- a gfx1036 Raphael iGPU shadowing a gfx1200 RX 9060 XT, leaving the discrete card unused until the reporter set HIP_VISIBLE_DEVICES=1 by hand. When no visible-device mask is pinned and more than one distinct arch is enumerated, skip a leading shadowing APU arch so the discrete card decides the wheel family, and print which GPU was chosen plus the HIP_VISIBLE_DEVICES override. An explicit HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES value still wins verbatim. The Strix arches (gfx1150/1151/1152) are deliberately excluded from the skip set: they are first-class unified-memory training targets, so their selection is unchanged. Signed-off-by: Tai An <antai12232931@outlook.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio/rocm): mirror the dGPU repick in setup.ps1 and honour CUDA_VISIBLE_DEVICES Addresses both review findings on #7778. 1. setup.ps1 resolved the gfx arch itself (hipinfo and amd-smi paths) and built $ROCmIndexUrl from it *before* invoking install_python_stack.py, and _ensure_rocm_torch() returns early once UNSLOTH_ROCM_TORCH_INSTALLED=1 -- so a fresh Windows install on a gfx1036 + gfx1200 host still received gfx103X-all wheels and never reached the Python-side repick. Resolve-ShadowingGfxPick mirrors _dedup_pick() and is applied at both PowerShell pick sites. 2. HIP honours CUDA_VISIBLE_DEVICES with the same semantics as its own masks -- _pick_rocm_gfx_target in install_llama_prebuilt.py already resolves all three identically -- so a ROCm install launched with only CUDA_VISIBLE_DEVICES set was treated as unpinned and could be overridden by the iGPU skip. It now counts as a pin on both sides ("" / "-1" still mean "no mask"). Tests: CUDA_VISIBLE_DEVICES pin + empty-is-not-a-pin cases, a setup.ps1 <-> Python parity check on the shadowing-arch list (the list now exists in two places), and the pre-existing shadowing tests now clear CUDA_VISIBLE_DEVICES so CI runners that export it cannot flip the assertions. Signed-off-by: Tai An <antai12232931@outlook.com> * fix(studio/rocm): keep a supported APU over a discrete card with no Windows wheels The shadowing-iGPU preference returned the first non-integrated arch in the enumeration regardless of whether AMD ships Windows wheels for it. On an unpinned gfx1036 + gfx1010 host that deposed a supported Raphael APU for a discrete card absent from _GFX_TO_AMD_INDEX_ARCH, so _windows_rocm_index_url resolved to None and the install fell back to CPU -- strictly worse than the shadowing the preference exists to undo. Only prefer the discrete arch when it actually has an index, unless the integrated pick has none either, in which case the swap costs nothing and the discrete card still wins. Both directions are covered: gfx1036+gfx1010 keeps the APU (fails without this change, returning gfx1010), gfx1013+gfx1010 still yields to the discrete card so the guard is not over-tightened. Signed-off-by: Tai An <antai12232931@outlook.com> * fix(studio/rocm): close two setup.ps1 gaps in the shadowing-iGPU preference Both halves of the #7776 preference existed in install_python_stack.py but only half of it in the PowerShell mirror, which resolves the arch and builds $ROCmIndexUrl itself before the Python installer ever runs. - Resolve-ShadowingGfxPick deposed a supported APU for any discrete arch, even one AMD ships no Windows wheels for (gfx1036 + an older gfx1010): the repick resolved to no index at all and dropped the host to CPU, strictly worse than the shadowing it undoes. It now consults $archFamilyMap, mirroring the _pick_has_wheels guard in _dedup_pick(). The map moves to script scope so detection can read it; contents are unchanged, so the four-way parity test still sees the same 18 entries. - The WMI fallback took the first AMD adapter before name -> arch inference, so an Adrenalin-only host listing a 780M ahead of an RX 9060 XT still inferred gfx1103 and installed gfx110X-all. It now keeps every AMD adapter, infers an arch for each, and runs the same preference over the list. Regression tests fail on the previous revision and pass on this one. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio/rocm): index the enumeration with CUDA_VISIBLE_DEVICES too _visible_devices_pinned() treats CUDA_VISIBLE_DEVICES as a pin, but _pick_visible_index() only read HIP/ROCR. On the probes that enumerate every GPU regardless of the masks (amd-smi, WMI), CUDA_VISIBLE_DEVICES=1 on a gfx1036 + gfx1200 host therefore suppressed the shadowing skip *and* resolved to index 0, installing the iGPU's wheels for the device the user masked away. Same mismatch in setup.ps1's $_hipVisIdx and $visGpu picks. All three masks are now read at every site, matching _pick_rocm_gfx_target in install_llama_prebuilt.py. Also corrects the _detect_windows_gfx_arch docstring, which still claimed the first GPU always wins without a mask. Signed-off-by: Tai An <antai12232931@outlook.com> * Share one mask resolver across every ROCm pick site for PR #7778 The shadowing-iGPU preference was correct, but each pick site still resolved HIP/ROCR/CUDA_VISIBLE_DEVICES with its own inline expression and they disagreed, so a mask the pin check honoured could resolve to a different GPU than the one the user asked for. On a mixed host that lands on index 0, which is the iGPU the preference exists to skip. setup.ps1 - Add Resolve-VisibleGpuIndex and use it at all four pick sites (hipinfo, amd-smi list, amd-smi static --asic, WMI name inference). Previously the hipinfo expression rejected " 1 " and the amd-smi one rejected "1,0". - The static --asic branch now collects every gfx token and runs the repick instead of taking the first regex match. - WMI inference indexes the adapter list rather than the inferred arch list, so an unrecognised name cannot shift a mask onto the wrong physical card, and it only repicks when every adapter mapped: an unknown name may itself be the discrete card. - Filter WMI adapters on ConfigManagerErrorCode so a disabled or driver-errored Radeon cannot depose a working iGPU. Get-CimInstance to match the rest of the repo. install_python_stack.py - _pick_visible_index now skips "" and "-1" and reads the next mask, matching _visible_devices_pinned. Before, HIP_VISIBLE_DEVICES="" with CUDA_VISIBLE_DEVICES=1 counted as pinned while the index resolved to GPU 0. - Out-of-range and unparseable masks warn instead of silently using GPU 0. - Strip the ":sramecc+:xnack-" suffix from gcnArchName like setup.ps1 does; a suffixed token matched neither the wheel table nor the skip set. - Prefer a wheel-backed candidate whenever one exists, not only when the picked arch has wheels, so gfx1036 + gfx1010 + gfx1200 no longer stops at gfx1010 and drops the host to CPU torch. Arch list - Drop gfx1037: it is not an AMDGPU target in LLVM, so no Windows tool emits it. - Add gfx1033 (Van Gogh) and gfx1153 (Krackan Point 2), both APUs. gfx1033 has a wheel family, so leaving it out let it act as the "discrete" card. - gfx1013 is Cyan Skillfish, not Van Gogh. Both advisories now tell the user to setx HIP_VISIBLE_DEVICES so the chosen GPU is used at runtime, not just at install time: the wheels alone do not change which device HIP enumerates first. Tests - TestSetupPs1ShadowingBehaviour actually executes Resolve-ShadowingGfxPick and Resolve-VisibleGpuIndex under pwsh, slicing them out by AST. The existing parity class only greps text, so a rename failed it while a semantic bug passed. - Regression tests for each fix above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Warn on a pinned wheel-less GPU and silence a bogus range warning Both found by simulating the change across the OS x GPU-vendor product rather than by reading it. - _dedup_pick now says so when an honoured pin selects a GPU AMD ships no Windows wheels for while another enumerated GPU has them. The pin is still honoured verbatim, but the install drops to CPU torch and the mask is the reason, which was previously invisible. - _pick_visible_index takes warn=False for callers whose list is deduplicated. The Linux Strix reroute indexes _detect_amd_gfx_codes(), which collapses duplicates, so a dual same-arch box (two gfx1151) has a 1-element list and a perfectly valid HIP_VISIBLE_DEVICES=1 read as out of range. That printed a false "out of range" warning on a healthy Linux host. The Windows arch-selection path still warns, where the index space really is devices. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align the visible-device masks with the ROCm runtime, and two repick fixes Three review items on the last round, all confirmed against the code at head. Mask semantics (mine to fix: 6d0ac82 got this wrong). "" and "-1" do not mean "no mask", they select no GPU at all, so falling through to the next variable was wrong. The ROCm runtime stores an explicitly empty var as " " (clr flags.cpp), then picks the HIP mask whenever its first byte is not NUL (paldevice.cpp on Windows, rocdevice.cpp on Linux), so an empty HIP_VISIBLE_DEVICES shadows CUDA_VISIBLE_DEVICES rather than deferring to it; parseRequestedDeviceList surfaces zero devices for " " and "-1", which ROCR states outright in amd_filter_device.h. _visible_devices_pinned and _pick_visible_index are now first-set-wins and treat any set value as a deliberate selection, matching _pick_rocm_gfx_target in install_llama_prebuilt.py and PyTorch's own _parse_visible_devices. Resolve-VisibleGpuIndex and Resolve-ShadowingGfxPick mirror it. Three tests asserted the old premise and now assert the runtime's. Resolve-ShadowingGfxPick did not prefer wheel-backed cards when the APU has no wheels either. The predicate went vacuously true and took the first non-integrated arch, so gfx90c,gfx1010,gfx1200 picked gfx1010, left $ROCmIndexUrl null and installed CPU torch despite the supported gfx1200. Now mirrors _dedup_pick's `_withWheels or (...)`. The Python WMI probe listed disabled adapters. setup.ps1 filters on ConfigManagerErrorCode but `(Get-CimInstance Win32_VideoController).Name` did not, so on a driver-only laptop a disabled RX 9060 could depose a working 780M and pull wheels for a GPU Windows never exposes. Same filter both sides. * Stop double-applying the mask to hipinfo, and two selection fixes hipinfo is itself a HIP application, so under a mask the runtime filtered and renumbered its device list before we ever read it. Indexing that output again applied the mask twice: with HIP_VISIBLE_DEVICES=1,0 on a gfx1036 + gfx1200 host, HIP exposes [gfx1200, gfx1036] and the second lookup landed on the iGPU, installing its wheel family for the card the mask put first. _dedup_pick now takes mask_resolved for the hipinfo probe and setup.ps1 reads $_hipAllArches[0]; amd-smi and WMI list every GPU regardless of the masks, so they keep the explicit index. The repo already stated this in _hip_visible_device_mask_set: "hipinfo, itself a HIP application, so under a mask it enumerates the VISIBLE devices, not the physical ones". The advisory hard-coded device 1. On gfx1036,gfx1010,gfx1200 the selected card is device 2, so following the message exposed the gfx1010 the installed wheels do not target. Both messages now name the selected arch's real index. The WMI path substituted another adapter's arch when the selected one had an unrecognised name. Unpinned that is the point (the #7776 iGPU has no entry in the name table, so the named discrete card should decide), but under a mask it installed wheels for a GPU the user masked away. The fallback is now gated on Test-VisibleDevicesPinned, which also replaces the inline pin loop in Resolve-ShadowingGfxPick so both sides share one definition. Two existing tests described a host that cannot exist: unfiltered hipinfo output under a mask. They now model the filtering HIP actually performs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep setup's dGPU repick for llama.cpp, and index WMI by adapter * Reinstall Windows ROCm torch when the wheel family changes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the WMI arch probe silent on non-AMD adapters * Read the active ROCm family from the rocm meta-package * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve the visible-device mask against devices, not deduplicated arches * Harden the WMI probe and the PowerShell index parse * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the platform in the two new Linux reroute tests * Pin the arch too in the new Linux reroute tests * Parse rocminfo per agent and honour ROCR filtering on Linux * Tighten comments --------- Signed-off-by: Tai An <antai12232931@outlook.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-08-06 05:46:38 -07:00
oobabooga	f8730f4339	Installer: select CUDA wheels that cover the host's GPUs (#7814) ... * Installer: select CUDA wheels that cover the host's GPUs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix Windows venv wipe and warning dedupe for PR #7814 - Windows pins torch<2.11, whose cu128 still ships sm_70, so capping a Volta to cu126 there rewrote a working family. The stale-venv check then read that as drift and deleted the venv on a direct "unsloth studio update", which cannot recreate it. Make the pre-Turing floor per-family (70 for cu128). - Repair an unpinned cu* -> cu* move in place instead of rebuilding the venv. - Decide the cu126 advice before deduping the uncovered-host warning: the host facts are release invariant but the artifact list is not, so the release walk-back let an unhelpful release swallow the remedy. - Gate the new coverage repair and the cu126 advice on x86_64, matching the cap. - Add tests/studio/test_pre_turing_cap.ps1: the parity test only greps for the call spelling, so neither PowerShell copy had behavioural coverage. * Tighten comments for PR #7814 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-08-04 06:55:29 -07:00
benj	6a58ea0f0e	Add Intel Arc GPU detection and XPU PyTorch install to Windows installer (#7706) ... * Add Intel Arc GPU detection and XPU PyTorch install to Windows installer The installer's GPU detection chain (NVIDIA -> AMD ROCm -> else) has no Intel Arc/SYCL/XPU branch, so Intel Arc GPUs fall into the "none (chat-only / GGUF)" branch and get CPU PyTorch despite PyTorch publishing XPU wheels at download.pytorch.org/whl/xpu. This adds: - WMI-based Intel GPU detection (Arc, Iris, UHD, HD Graphics) - Torch XPU availability check for migrated/upgraded environments - An XPU PyTorch install path with the whl/xpu index - CPU fallback with a pointer to the Intel oneAPI docs when XPU isn't available - Updated messaging from "NVIDIA or AMD ROCm" to include Intel Arc The XPU wheels ship their own oneAPI runtime (intel-sycl-rt et al.) so no Intel oneAPI Base Toolkit is required for GPU training. Tested on: Windows 11, Intel Arc 140V GPU (8GB), PyTorch 2.9.0+xpu Co-authored-by: CommandCodeBot <noreply@commandcode.ai> * Fix Intel XPU detection and install path for PR #7706 The XPU index selected during GPU detection was overwritten by Get-TorchIndexUrl before the install branch read it, so Intel hosts still got CPU PyTorch while being told XPU wheels were being installed. - Move the XPU reroute after Get-TorchIndexUrl, and let an explicit pin win - Detect via Get-CimInstance (Get-WmiObject is absent in PowerShell 7) - Match only Arc / Data Center GPU, so UHD / HD / Iris Xe are not promised XPU - Split Intel GPU present from XPU-capable so the CPU fallback hint works - Bound the XPU torch trio like every other index (bare names resolved torch 2.13.0 + torchaudio 2.11.0 and pulled unsloth back to an old release) - Clear the XPU state after a CPU fallback, mirroring the ROCm path - Teach the index family, GPU branch and torch flavor helpers about xpu * Keep install.sh diagnostics in parity with the install.ps1 xpu family install.ps1 now classifies an /xpu index leaf as family xpu / branch xpu, so mirror the same two cases in _tauri_torch_index_family and _tauri_gpu_branch. These feed the [TAURI:DIAG] line only, and a Linux user can already reach the xpu index via UNSLOTH_TORCH_INDEX_FAMILY, where it previously reported auto/unknown. Linux Intel auto-detection is not added here. * Tighten the Intel XPU comments in install.ps1 Comment and whitespace only, no code change. * Address Codex review on the Intel XPU path - Run the Intel scan before the GPU report chain instead of inside its final else. A WMI-named-only AMD adapter set ROCmGpuLabel and took that chain, so a discrete Arc card next to an AMD CPU's integrated Radeon was never detected. The scan is gated on no usable NVIDIA or AMD, and the Intel branch ranks above the two AMD-present-but-unusable branches, so a usable AMD host is unaffected. - Let a migrated env's torch veto the hardware match only when it is itself an XPU build. A CPU build reports torch.xpu.is_available() False for lacking XPU support, not for unsuitable hardware, and was blocking the CPU to XPU upgrade. - Detect Intel in studio/setup.ps1 too. It only knew NVIDIA and AMD, so every successful Intel install printed none (chat-only / GGUF) right after install.ps1 reported a usable Arc GPU. Self-contained so studio update works. * Address the second Codex round on the Intel XPU path - Reset $script:IsIntelXpu at the start of each invocation. Under the documented irm | iex path $script: is the caller's session scope, so a second run in the same session inherited a stale true, skipped the scan on a now-NVIDIA host and still rerouted to the xpu index. Reproduced in pwsh before fixing. - Gate the Intel scan on whether AMD actually gets a wheel, not on whether an AMD arch was seen. An arch missing from the family map has no ROCm wheels and lands on CPU torch, so it must not outrank a usable Arc card. The map is hoisted above the scan and consumed unchanged by the AMD reroute. - Select the XPU index in studio/setup.ps1, not just report it. Previously setup printed Intel GPU detected and then installed CPU torch, so studio update never migrated an Arc box off CPU. Adds a bounded XPU install with a CPU fallback, teaches the stale-venv check about +xpu, and mirrors the wheel-aware AMD gate so the two files agree instead of wiping the venv on every update. * Address the third Codex round on the Intel XPU path - Force the dependency pass on an Arc host whose torch is not XPU-capable, the Intel counterpart of the existing AMD escape. Without it the fast up-to-date path skipped the install block, so the xpu index selection was never reached and a CPU venv never migrated. - Confirm a working XPU runtime before treating an xpu venv as stale. If CIM is unavailable or returns an Intel name outside the Arc match, the expected tag fell through to cpu and a valid XPU environment was rebuilt and lost. - Force-reinstall the XPU trio only when the installed wheel is not already +xpu, or the pin changed. It was unconditional, so a fresh install re-fetched multiple GB immediately and again on every update. - Warn when torch.xpu.is_available() is false after installing XPU torch, naming the Intel driver floor. Otherwise the installer promised GPU training while unsloth raised NotImplementedError at import on a stale driver. - Stop the detection probe vetoing the hardware match. Its cpu fallback could not displace the installed +xpu wheel, so it only mislabelled a capable GPU as unusable; the driver warning covers that case honestly, and setup.ps1 agrees. * Bound the XPU probes, repair xpu pins in install.sh, and floor bitsandbytes on the Intel path install.sh: teach _torch_flavor_tag, _expected_torch_flavor_tag and _torch_index_repairable about the xpu leaf. The diagnostic already reported gpu_branch=xpu, but an xpu pin fell to the custom arm so a migrated env kept its CPU wheel. The +xpu flavor arm is required alongside, otherwise a correct 2.10.0+xpu wheel reads as cpu and gets force-reinstalled every run. install.ps1 / studio/setup.ps1: route every torch probe through a new bounded Invoke-BoundedPythonProbe (ProcessStartInfo, both streams drained async, WaitForExit, kill on timeout). A hanging Intel driver init is exactly what these probes detect, and an unbounded one would hang the installer instead of reaching the warning. Timeouts read as not-available. Get-InstalledTorchTag now shares the helper rather than carrying a second copy of the pattern. install.ps1: install bitsandbytes>=0.50.0 on the XPU path. unsloth's floor is >=0.45.5, so a migrated venv keeps a pre-0.49 wheel with no XPU library and 4-bit QLoRA silently turns off. Same floor the AMD paths use, since <=0.49.2 NaNs at 4-bit decode and an Arc card can sit next to a Radeon. * Floor bitsandbytes on the Studio XPU migration and on an explicit xpu pin studio/setup.ps1: `unsloth studio update` migrating a CPU venv to XPU replaced only the torch trio. install_python_stack.py then upgrades unsloth and unsloth-zoo alone, so an installed bitsandbytes 0.45.x kept satisfying the base floor while carrying no Windows XPU kernels, and 4-bit QLoRA silently turned off. Adds the same bitsandbytes>=0.50.0 --no-deps pass install.ps1 got, placed after the stack so it is the last word, gated on $XpuIndexUrl (the CPU fallback clears it, no-torch never sets it) and still inside the -not $SkipPythonDeps block so the up-to-date escape does not reach it. install.ps1: key the bitsandbytes pass off the index leaf instead of $script:IsIntelXpu. An explicit UNSLOTH_TORCH_INDEX_FAMILY=xpu pin on a non-Intel host skips the XPU branch but still installs the trio from the xpu index, so torch is +xpu and needs the same floor. The CPU fallback rewrites $TorchIndexUrl, so a failed XPU install reads as cpu and stays quiet. * Tighten the Intel XPU comments across the three installers Comment-only pass now that the review has settled: several blocks grew over successive rounds and were restating the code or narrating the review. Net 36 lines removed, with the load-bearing facts kept -- why ProcessStartInfo rather than the call operator, why both probe streams drain async, why the helper is defined above the Intel scan, the 0.50.0 bitsandbytes floor and why not the curated extra, and why PEP 440 means a migrated env can confirm but never veto the Intel match. Also records why the Studio bitsandbytes pass must stay above the ErrorActionPreference restore: Fast-Install needs EAP=Continue or PS 5.1 turns pip stderr into a terminating error. No code tokens changed; verified with a PowerShell token-stream diff of install.ps1 and setup.ps1, and by hand for install.sh. * Bound the Intel WMI scan, bound the stale flavor probe, and stop CUDA Triton shadowing XPU studio/setup.ps1: the stale-venv flavor probe read StandardOutput.ReadToEnd() before WaitForExit, so the timeout was unreachable and a wedged import torch hung studio setup forever; stderr was never drained either. Routed through Invoke-BoundedPythonProbe, which already drains both streams and kills on timeout. A timeout now reads as unreadable flavor, so the venv rebuilds. install.ps1 / studio/setup.ps1: bound the Win32_VideoController query and add a registry fallback. -ErrorAction suppresses errors but bounds nothing, and -OperationTimeoutSec is not enforced for the local COM session this uses, so a degraded WMI repository blocks forever. install_llama_prebuilt.py already runs this query out of process for the same reason and documents an Arc A770 being misrouted by it. The registry class key answers in-process; it is the fallback rather than the fast path because a stale driver config can outlive the hardware, and here a false positive would install XPU torch on a host with no Arc. studio/setup.ps1: replace triton-windows with torch's own XPU triton after the stack. Both distributions own the top-level triton package, sharing 151 paths including __init__.py and _C/libtriton.pyd, so an in-place cu-to-xpu repair leaves the CUDA build shadowing the XPU one. Removing it alone would delete the shared files the XPU wheel overwrote, and unsloth declares triton-windows as a win32 dependency so an earlier removal is reinstalled by the stack: uninstall and reinstall, after the stack, only while triton-windows is present. The spec is read from the installed torch, since the name changed from pytorch-triton-xpu to triton-xpu in torch 2.10. * Tighten the comments added with the bounded scan and Triton replacement Comment-only pass over the previous commit's additions, which had not been through one: 15 lines removed across the two bounded-scan headers, the two registry-fallback headers and the Triton block. Kept the facts that cost measurement: -OperationTimeoutSec not being enforced for a local COM session, Ok being false on an empty answer because a Windows host always has an adapter, the registry class key being fallback rather than fast path here, the 151 shared Triton paths, and why the uninstall has to be paired with a reinstall after the stack. No code tokens changed; verified with a PowerShell token-stream diff of both files, which also confirms the two helper copies stay identical. * Stage the Triton replacement behind a download so the uninstall cannot strand the venv The replacement uninstalled triton-windows and then installed the XPU triton from the index. A failure between the two left the venv with a partially deleted triton, since the uninstall drops the paths shared with the XPU distribution, and the warning made that look like a skipped optional repair. The uninstall cannot go last, because it removes the paths in triton-windows' own record and those are the shared ones. So fetch first: pip download the wheel, confirm one is actually on disk (exit 0 alone is not enough, an sdist-only mirror satisfies that), and only then uninstall and install the local file. A local wheel installs with the network refused, so nothing after the destructive step depends on the index. A failed fetch leaves triton-windows in place, which is the pre-existing shadowing rather than a broken venv, and says so. Past that point only disk or permissions can fail, so restore triton-windows if the local install does, leaving a triton that imports. If both fail the message is loud and carries the repair command, with the index URL redacted since a mirror pin can carry a token. pip only: uv has no pip download (astral-sh/uv#3163). * Windows: harden the Intel registry fallback and declare the XPU install state up front Get-IntelRegistryAdapterNames wrapped the whole enumeration in a single try, so one unreadable subkey discarded every adapter found before it. windows_intel_gpu_in_registry(), the in-process Python probe over the same class key, skips per subkey and continues; the PowerShell copy now does too. It also matched on the PCI vendor id but returned DriverDesc, which the callers re-filter on "Intel", so a localized or OEM-branded Arc was found here and dropped there. Both installers carry the same copy and a test asserts they stay identical. setup.ps1 read $installedTorchTag and $XpuIndexUrl from outside the blocks that assign them. Unset and $null are both falsy so behaviour is unchanged, but a caller running with Set-StrictMode -Version Latest turned those reads into terminating errors, and install.ps1 is documented as irm | iex into the caller's own session. Two comment corrections: 0.48.2, not 0.49.0, is the first win_amd64 bitsandbytes wheel carrying libbitsandbytes_xpu.dll, and the triton package overlap is version-dependent rather than a fixed 151 paths. The new test drives the shipped helper with the registry cmdlets mocked rather than reading a hive, so it runs on Linux and macOS as well as Windows. * Studio: show the Intel XPU runtime row in the About tab hardware.py has always emitted versions["xpu"], but HardwareInfo only ever declared cuda and rocm. On an Arc host both of those are null, so the runtime row disappeared entirely while the GPU name and VRAM rows still rendered, leaving a host that looks half detected. That was unreachable on Windows until the installer learned to select XPU wheels, which is what makes it worth fixing here. The three-way choice is lifted into a helper at module scope: inlining it pushes AboutTab past the cognitive-complexity ceiling. The label is a proper noun, so every locale carries the same literal. * Windows: reach Intel XPU through a localized name, a stale fast path and an old wheel Four holes in the XPU paths, all found by driving the shipped code rather than reading it. The registry fallback only ran when the CIM scan failed. When it succeeds and returns a localized adapter name, which on non-English Windows carries no ASCII "Intel", the filter dropped the adapter and the host went to CPU torch. The registry now re-labels an adapter WMI already reported, matched by name so an entry naming nothing WMI listed stays ignored: a driver record outliving its card still cannot promote a host WMI answered for. The XPU trio accepted torch 2.4 and 2.5, which unsloth/models/_utils.py rejects at import for an XPU device. An xpu mirror carrying only an older wheel produced an install that reported success and then failed on the first import, and an existing 2.5+xpu venv was kept because it satisfied the range. The floor is 2.6 on the XPU paths only; the CPU fallback keeps 2.4. The "package is up to date" fast path escaped for an Arc host on CPU torch, but not for one already on XPU torch whose bitsandbytes predates the XPU kernels or whose triton-windows still shadows the XPU Triton. Those two live in the dependency pass, so a venv that reached +xpu without them, an explicit pin or an update whose first pass ran the pre-XPU setup.ps1, never got them on any later update either. An unreadable version reads as stale. install_python_stack.py writes its completion manifest immediately before returning, so an interrupt between the triton-windows uninstall and the XPU wheel install left a venv with no triton that the next update read as complete. The manifest is now held aside across the swap and restored only once a triton is importable again. * Windows: move the install manifest across the Triton swap instead of rewriting it Two problems with the hold added in 2603fc809, both on the restore side. Reading and rewriting the file cannot survive a manifest carrying a non-ASCII path. Windows PowerShell 5.1 writes Set-Content in the ANSI code page by default, and its -Encoding utf8 emits a BOM that install_manifest.read_manifest's json.load rejects outright ("Unexpected UTF-8 BOM"); Get-Content is ANSI on a BOM-less file too, so the read lost bytes before the write got a chance to. The manifest is now MOVED into the wheel's temp directory and moved back, so no encoding is involved at either end. That directory is already removed in the finally, which is what keeps an unrestored manifest gone. A manifest that would not move left the old valid one in place for the whole destructive window, since the failure only cleared the saved copy and carried on into the uninstall. That is the case the hold exists for, so it now skips the swap entirely and says so: triton-windows keeps shadowing the XPU Triton, which costs torch.compile on the GPU and is repairable on the next run, rather than risking a venv with no Triton that reads as complete. * Windows: confirm the install manifest actually moved before the Triton swap Move-Item across volumes is a copy followed by a delete, and it reports success when only the delete fails, leaving the original exactly where it was. So the guard added in af928dd88 could believe it had set the manifest aside while a valid one sat there for the whole destructive window, which is the case that guard exists to prevent. Found by modelling the manifest in the setup.ps1 scenario matrix, which this had no coverage for: with the parent directory read-only the swap still ran, and the locked scenario passed for the wrong reason. The move is now confirmed by testing the source path afterwards, and a manifest still standing aborts the swap like any other failure to move it. Four new scenarios cover it: the swap keeping a byte-identical manifest, a swap where neither Triton reinstalls correctly leaving it gone, a failed fetch never touching it, and a manifest that cannot move aborting the swap. * Windows: key the XPU fast-path remediation off the installed wheel, not just the GPU scan $HasNvidiaSmi suppresses the Intel scan, so on a mixed NVIDIA + Intel box under an explicit xpu pin $script:IsIntelXpu stays false while the pin still lands the venv on a +xpu wheel. The staleness check added in 2603fc809 was gated on that flag alone, so those hosts kept taking the fast path and never reached the bitsandbytes floor or the Triton replacement. This is the same gating mistake the bitsandbytes pass had in round 4, where the fix was to key off the index leaf rather than the scan. The leaf is not resolved yet at the fast path, but the installed flavor tag is, and whatever put the venv on a +xpu wheel the two remediations still apply. The runtime probe above stays on the scan: reinstalling XPU torch is only right where an Intel GPU was actually found. A pure NVIDIA host on a cu wheel never runs the probe, which the matrix asserts alongside the two new mixed-host rows. * Windows: reconcile Intel names for hybrid GPUs, and stop the XPU escapes firing where XPU is unreachable Five fixes from a review of the XPU work so far. The registry reconciliation was gated on "no ASCII Intel name present", so a hybrid laptop reporting its Intel UHD alongside a localized Arc stopped at the UHD and left the Arc unrecognised. It is now gated on the absence of an XPU match, and the regex behind both that gate and the classification is defined once so they cannot drift. The two fast-path escapes cleared $SkipPythonDeps for any Intel host, but the XPU install and its two remediations are all gated on $XpuIndexUrl, which an explicit cpu / rocm / custom-leaf pin never sets, and no-torch mode has no torch pass at all. Those hosts ran the whole dependency pass, installed nothing new, and re-fired the identical condition on every later update. Both escapes now require XPU to be reachable. The manifest path was learned by a subprocess whose output parsing could not work: `& python` returns one array element per line, interpolating that joins on $OFS, a SPACE, so splitting on newlines yields a single element and a banner ahead of the answer arrives glued to the path. Any such failure then skipped the hold silently and swapped anyway, which is the window the hold exists to close. manifest_path() is venv_root()/MANIFEST_NAME and venv_root() is sys.prefix, which is $VenvDir here, so it is assembled like Get-PersistedNoTorch already does. A test asserts the literal still matches MANIFEST_NAME. The uninstall's exit code was discarded. A triton-windows that will not uninstall, which on Windows means Studio is running and holding libtriton.pyd open, still shadows the XPU Triton, so installing over it achieved nothing and restored the manifest onto a venv this pass was supposed to have changed. The restore had no verification and an empty catch, while the finally deletes the held copy either way, so a failed restore lost the manifest with nothing on screen. * Windows: keep the WMI adapter list an array so the Intel re-label appends instead of concatenating `$_gpuNames = if (...) { @(...) } else { @(...) }` wraps each branch, and a one-element array unrolls on its way out of the if, so on any single-adapter host $_gpuNames was a String. The `+=` that re-labels a localized adapter then concatenated two strings rather than appending a name, and the GPU reported to the user came out doubled: Intel(R) UHD Graphics 620Intel Intel(R) UHD Graphics 620 No install decision changes. The re-label only appends a registry name that already contains the WMI name, so the concatenation matches the Arc / Data Center regex exactly when the registry name alone would, and every scenario in the matrix records the same verdict either way. It is the displayed adapter name that was wrong. Widened by the previous commit: gating on the absence of an XPU match rather than of any Intel name brought ordinary single Intel iGPU hosts into the re-label for the first time. @() now wraps the whole if in both installers, with a test asserting it stays that way. * Windows: give pin-only XPU installs the 2.6 floor, and treat an unreadable dependency probe as stale The XPU install branch required $script:IsIntelXpu as well as an xpu index leaf, so an explicit FAMILY=xpu or URL pin on a host whose Intel scan never ran -- a mixed NVIDIA box, where $HasNvidiaSmi suppresses it -- fell through to the generic branch and its torch>=2.4. Against a mirror carrying an older +xpu wheel that installs a torch unsloth rejects at import. Keyed off the leaf alone now, which is what the bitsandbytes gate below it already does and says in its own comment. install.sh had the same gap from the other direction: its xpu leaf is reachable only by an explicit pin and kept the generic floor, so it gets the same 2.6 trio. The fast-path dependency probe treated "did not answer" as "nothing to do". A timeout, or a malformed .dist-info making distributions() raise, then left the fast path intact and an XPU migration never reached the bitsandbytes floor or the Triton replacement on any later update either. It now clears the fast path, the same direction an unparseable version already took. Two install.ps1 rows move, both FAMILY=xpu pins on non-Intel hosts, both onto the XPU branch. The CPU fallback after a failed XPU install keeps its 2.4 floor. * Windows XPU: probe the preserved venv, drop torchaudio on ARM64, and give POSIX XPU the bitsandbytes floor Three fixes to the Intel XPU paths. install.ps1, migrated-runtime probe: a rerun over an existing install moves the old venv to $script:StudioVenvRollbackDir and creates an empty one in its place, both before this probe runs, so it always asked an interpreter with no torch and answered "no XPU". Ask the preserved environment when there is one, which is the migrated runtime the fallback exists for. install.ps1, Windows on ARM: no win_arm64 torchaudio wheel exists on any index. Keying the XPU branch off the index leaf alone routes an arm64 interpreter into a branch that hardcoded the trio, so the install aborted. Ask the interpreter for its platform tag, as the generic path already does, and drop that one pin on arm64. The CPU fallback below it gets the same treatment. install.sh, XPU pins: bitsandbytes ships XPU kernels (libbitsandbytes_xpu2025.so and _xpu2026.so) from 0.50.0 on manylinux, and nothing on the POSIX side raised the floor for them, so a migrated environment kept a pre-XPU build and lost 4-bit QLoRA on a torch that otherwise works. Matches what the Windows XPU pass already installs. * Studio: stop the xpu label test from forbidding a partial locale check-parity.ts states the contract plainly: "Locale files may be partial; missing keys must fall back to English." The new test required every overlay to carry the xpu label, which contradicts that and breaks on the next locale anyone adds. It already did: it.ts landed on main after this branch, so the merged tree fails on all three runners even though nothing about the label is wrong there. The label is a proper noun, so the English fallback is byte-identical to a translation and the requirement bought nothing. Assert what actually renders wrong instead: en.ts must carry the key, because it is the fallback every locale resolves to, and no overlay may define a value that disagrees with it. Both halves were checked against a merged working tree, and both still fail when the condition they guard is broken. * Linux XPU: hoist the bitsandbytes pass out of the fresh-install arm It sat inside `elif [ -n "$TORCH_INDEX_URL" ]`, which a migrated environment never enters because the `_MIGRATED` arm above it wins, so the one environment the pass existed for was the one that skipped it. The AMD passes handle this by existing twice, once per arm; this gate needs nothing branch-specific, so it moves past the chain instead and both arms reach a single copy. tests/sh/test_xpu_bitsandbytes_reachable.sh guards both halves: the block must be placed where every arm reaches it, and it must still fire only on the xpu leaf. 25 checks over [migrated, fresh] x [xpu, mirrored xpu, cuda, rocm, cpu, none] x [torch, no-torch], run against the block and the leaf parser extracted from install.sh. Moving the block back inside an arm fails it. * Report the XPU runtime before the hardware summary, and show every runtime in About setup.ps1: the hardware report runs ~1300 lines before the torch.xpu.is_available() check that keeps an XPU environment, so a host the WMI scan and the registry fallback both miss (wedged CIM service, an Intel part outside the Arc|Data Center regex) was told "none (chat-only / GGUF)" and then watched setup keep the XPU venv. Ask the same question before printing, so the report and the decision cannot disagree. A free disk read gates the interpreter launch: torch/version.py carries the local label, so a CPU-only host never pays for an `import torch` on every `studio update` just to be told it has no Intel GPU. The dist-info name cannot be used for this -- pip normalises the local label out of it (torch-2.9.1.dist-info for a +cu128 wheel). The promotion carries its own try: it must still run when the scan threw, which is the case it exists for, and a junk UNSLOTH_STUDIO_HOME would otherwise abort setup from Join-Path. about-tab.tsx: hardware.py reads versions["cuda"] off torch.version.cuda and sets versions["xpu"] from an independent torch.xpu.is_available() probe, and UNSLOTH_FORCE_XPU=1 is a supported configuration where CUDA is present but XPU is selected. Both are non-null there, so returning the first match hid the XPU row on exactly the host it was added for. Collect every reported runtime instead. tests/studio/test_setup_xpu_runtime_prereport.ps1 covers the two new helpers with the filesystem mocked, so it runs on all three runners: override precedence, ~ expansion, the four wheel flavours, a missing or unreadable version.py, and wiring assertions that the promotion precedes the report and that the cheap read gates the probe. The About-tab test gains a case that fails if the picker returns early again. * POSIX: recognise a working XPU runtime, and raise the bitsandbytes floor on the update path The hardware summary tested NVIDIA, AMD and Apple Silicon and then fell through to "none (chat-only / GGUF)", so a Linux host running the +xpu wheel install.sh had just installed was told training needs an NVIDIA or AMD GPU. Added an arm ranked below both, matching setup.ps1. The bitsandbytes floor was also unreachable on the route an existing XPU user actually takes. `unsloth studio update` runs this file, never install.sh (see the note at the top of setup.sh), and neither this file nor install_python_stack.py had an XPU floor, while unsloth's own dep floor is 0.45.5 -- which a pre-XPU wheel satisfies indefinitely. So 4-bit QLoRA stayed unavailable on a torch that otherwise works. One detection serves both, but they read different signals on purpose. The floor keys on the WHEEL (+xpu, read off torch/version.py) and the summary keys on the RUNTIME (torch.xpu.is_available()): a +xpu wheel installs fine on a host whose driver never initialises, and that host should still get the kernels while no GPU is claimed for it. The disk read gates the interpreter launch, so a CPU-only host pays nothing per update. tests/sh/test_setup_xpu_posix_summary.sh builds real venv trees, version.py files and stub interpreters rather than mocking, so the disk read and the runtime probe genuinely execute: 13 checks over the four wheel flavours, working/dead/missing runtime, no venv, and the arm's rank. Removing the arm fails four of them. * POSIX XPU: make the bitsandbytes step nonfatal, bound the probe, and act on an XPU pin Three defects in the POSIX XPU code from the previous commit. run_quiet routes failure to setup_fail and exits, so the best-effort bitsandbytes upgrade could abort an otherwise fine `studio update` over a transient download, and the warning after it was unreachable. run_quiet_no_exit is the nonfatal wrapper. The runtime probe had no timeout. A stalled Intel driver wedges inside `import torch`, which is exactly the host this probe classifies, so it could hang every update forever. Bounded at 60s rather than the 10s the smi probes use: a cold `import torch` takes seconds by itself and a short bound would read a healthy host as having no GPU. Systems without coreutils timeout keep the previous behaviour rather than losing detection. An explicit XPU pin was protected but never acted on. An xpu leaf names no family the cuda/rocm repair helpers know, so _explicit_unknown_family_torch_index_url makes both skip it, and `unsloth studio update` never runs install.sh -- so switching a CPU install to UNSLOTH_TORCH_INDEX_FAMILY=xpu left the CPU wheel in place indefinitely. The fix goes in install_python_stack.py, which already parses the pin, rather than setup.sh, which has no pin awareness at all: _ensure_xpu_torch mirrors the existing _ensure_cpu_torch, the xpu leaf is classified so the backend is no longer unknown, and the ROCm helper skips an xpu backend so it cannot treat the pin as an AMD host. Windows is excluded because setup.ps1 owns torch there and installs the trio itself. That put the XPU trio in a third file, so tests/sh/test_xpu_torch_spec_parity.sh asserts the floors match across install.sh, install_python_stack.py and install.ps1 plus the wiring. Each of its four structural guards was mutation-tested: a drifted floor, a lost classification, wiring at one call site instead of two, and the ROCm skip removed all fail it. The POSIX summary suite gains checks for the nonfatal wrapper and the bound. * Linux XPU: swap generic Triton, gate the pin repair on the version, and escape the fast path Three defects in the XPU code from the previous commit. _ensure_xpu_torch returned on the +xpu tag alone, so a migrated 2.5+xpu venv was left in place even though unsloth/models/_utils.py raises at import for an XPU device below 2.6. It now returns only when the flavour and the supported range both match. That repair was also unreachable on the route it was written for. setup.sh skips install_python_stack entirely when the package version is current, and that pass is the only thing that acts on an XPU pin, so a CPU install switched to the xpu family stayed CPU. Added a third fast-path escape beside the anyio and incomplete-manifest ones. Generic triton and torch's pytorch-triton-xpu / triton-xpu both own the top-level triton package, and resolving unsloth against a pinned +xpu torch pulls both -- uv reports pytorch-triton-xpu 3.5.0 alongside triton 3.7.1 -- so the CUDA-oriented build lands last and torch.compile loads the wrong library on an Intel GPU. This is the POSIX half of the Windows swap: the spec is read from torch's own metadata, so the pytorch-triton-xpu to triton-xpu rename at torch 2.10 needs no hardcoding, and the fetch happens before the uninstall because the uninstall drops the shared paths from generic triton's own record. test_torch_installs_do_not_use_deprecated_index_url forbade --index-url on "$TORCH_INDEX_URL" anywhere in install.sh. That rule is about uv, which deprecated the flag in favour of --default-index; pip never had --default-index, so the pre-fetch legitimately uses it. The assertion is now per occurrence and exempts pip download only, and it joins backslash continuations first, since the flag and its command are routinely on different physical lines. Both a same-line and a continuation-line uv offender were mutation-tested and are still caught. tests/sh/test_xpu_triton_swap_posix.sh asserts the swap by execution -- ordering, the rename, no generic triton, torch wanting CUDA triton, non-xpu index, no-torch, empty index, and a dead mirror that must warn without removing anything. * XPU: move the Triton swap where both routes reach it, and bootstrap pip for it Five defects in the XPU code from the previous commits. The Triton pre-fetch could never have run. `uv venv` is created without --seed, so a fresh venv has no pip and `python -m pip download` fails with "No module named pip" every time, leaving the swap a no-op that only ever warns. My shell test missed it because its stub interpreter answered pip commands. install.sh already bootstraps pip this way before its pre-release bitsandbytes wheel. The swap also never ran on `unsloth studio update`, which runs setup.sh and never install.sh. Both fixes fall out of moving it: install.sh runs setup.sh, which runs install_python_stack.py, so that module is the one place both routes pass through. The install.sh copy is deleted rather than duplicated, and the shell test is replaced by tests/studio/test_xpu_triton_swap.py, which covers the no-pip case and asserts install.sh carries no second copy. The fast-path pin match missed authenticated and fragmented mirrors (https://mirror/whl/xpu?token=...), which read as "no XPU pin" and skipped the repair; query and fragment are now stripped before the leaf test. That escape also launched an interpreter, which a wedged Intel driver hangs inside. It now reads the local label out of torch/version.py instead: nothing to bound, and a CPU-only host pays nothing per update. setup.ps1's fast path asked only whether XPU was available. A 2.5+xpu build answers yes and is still rejected by unsloth/models/_utils.py at import, so it now checks the supported range too, via Test-TorchXpuVersionSupported. The POSIX suite is up to 22 checks; the three new guards were mutation-tested by removing the query strip, the fragment strip, and by making the escape launch an interpreter. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * XPU: make a failed Triton swap unsurvivable, and widen the fast-path escapes Five defects in the XPU code from the last two commits. The Triton uninstall ignored its return code. A read-only or locked venv leaves generic triton registered, so installing over it lets a later upgrade of that distribution delete the shared files again, and every dependency pass repeats the swap. A failed uninstall now changes nothing at all. Past the uninstall the venv has no triton, because the uninstall takes the shared top-level files with it, so a warning there let the caller write a completion manifest over a venv whose torch.compile is broken -- and the next update fast-paths straight past it, since no generic distribution is left to trigger on. That install is now fatal. _ensure_xpu_torch returned when the probe timed out. On this path a wedged `import torch` is evidence rather than noise: the usual cause is a stalled Intel driver under an unsupported +xpu wheel, which the resolver keeps because it satisfies the base range. An authoritative pin now repairs on an inconclusive probe. This deliberately differs from the CPU counterpart, where a wedge has no such likely cause. The fast-path pin match stripped one trailing slash, so a ".../whl/xpu//" pin still read as no pin. It now strips them all, like the shared leaf parsers. Moving the Triton swap into the Python stack left the fast path with no reason to run it: a migrated environment with supported +xpu torch and a leftover generic triton kept the CUDA-oriented build forever. A stale generic triton now forces the dependency pass too, detected from the dist-info name so no interpreter is launched. The POSIX suite is up to 26 checks and the Triton tests to 16. Two of the guards were rebuilt after their own negative controls found them vacuous: the stale-triton check matched the detection loop rather than the branch that acts on it, and a fixed line window had drifted off the code it was meant to cover, so it is now anchored on the block. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not require the XPU pin again after install Three Intel paths still assumed the pin was still in the environment, or that every XPU host looks like x64 Linux. install_python_stack.py: the generic-Triton swap returned unless UNSLOTH_TORCH_INDEX_URL / _FAMILY was set. That pin is one-shot -- a user who ran UNSLOTH_TORCH_INDEX_FAMILY=xpu ./install.sh has nothing left in the environment by the next plain `unsloth studio update`, yet that update's dependency pass can pull generic triton back in and shadow torch's XPU build again. The installed +xpu wheel is the durable signal (setup.sh already raises the bitsandbytes floor off it), so fall back to it and to the default xpu index. The label is read off disk: importlib.metadata drops the local version label, and `import torch` loads the SYCL runtime, which can wedge. install.ps1: the flavor repair built its own XPU trio including torchaudio, which has no win_arm64 wheel on any index. A migrated ARM64 venv skips the fresh XPU branch and takes this path, so the repair failed outright before setup.ps1 could reach its ARM-aware fallback. One builder now serves both sites, since the two copies drifted the moment only one learned about ARM. install.sh: adding the xpu tag made the final flavor guard reachable on an Intel pin, and it probes with an unbounded `import torch`. On a host whose driver initialization wedges that hangs the installer, with no timeout anywhere before setup.sh's bounded probes. The xpu path reads torch/version.py off disk instead; every other family keeps the interpreter read unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the setup XPU paths on the installed wheel, not the pin Follow-up to d07c179f: install_python_stack now treats the installed +xpu wheel as the durable signal, but the two callers upstream of it did not. setup.sh fast path: the escape only ran under `case $_setup_pin in *xpu`, so after a one-shot UNSLOTH_TORCH_INDEX_FAMILY=xpu install every later `studio update` saw no pin, kept _SKIP_PYTHON_DEPS=true and never reached the Triton swap at all -- generic triton kept shadowing the XPU build forever. The disk read now happens unconditionally and the swap escape keys on the wheel. The pin leaf is also compared exactly, like the shared index parsers: a custom mirror ending in -xpu was classified as the curated family, which cleared the skip flag on every up-to-date run while _ensure_xpu_torch declined to act. setup.ps1: bounding the flavour probe turned a timeout into "rebuild", and the host most likely to time out inside `import torch` is an Arc box whose compute driver stalled -- where torch/version.py still names a good +xpu wheel. With no currently exported pin the stale path then deleted the venv. It now falls back to the same disk check and warns about the driver. Other families still rebuild on an unreadable flavour. setup.sh summary: a +xpu wheel whose runtime will not initialise fell through to "none (chat-only / GGUF)", telling an Arc owner their hardware is unsupported and hiding the driver update that fixes it. It gets its own arm. * Stop the XPU paths from stranding or wiping a venv Four ways the Intel paths could still leave a user worse off than before they ran anything. setup.ps1 stale check: on a hybrid NVIDIA + Arc host the XPU promotion is gated on -not $HasNvidiaSmi, so a pinless `unsloth studio update` expects a cu* tag, calls the working Arc venv stale and DELETES it -- then exits, because only install.ps1 creates venvs. A direct update now keeps any +xpu venv and says to re-run install.ps1, which rebuilds with a rollback copy. setup.ps1 Triton swap: when the staged XPU wheel failed to install after triton-windows was removed AND the generic restore also failed, the branch only printed. $stackExit stayed 0, so setup reported success and install.ps1 committed a venv with no importable triton over its rollback. It now carries the real failure code into the existing handler. install_python_stack: the `pip download` that stages the XPU Triton wheel inherited the user's pip index environment. PIP_NO_INDEX makes pip ignore --index-url outright, and PIP_EXTRA_INDEX_URL / PIP_FIND_LINKS are consulted in addition to it, so the fetch could fail (leaving generic Triton shadowing the XPU build) or serve the wheel from an index the pin never named. It now takes the same _install_env_for_cmd scrub every other pinned install gets. setup.sh runtime probe: the arm taken when coreutils `timeout` is absent ran the probe with no deadline, on exactly the stalled-driver host the bounding exists for. The deadline now lives inside the probe as signal.alarm, which terminates the process even while the driver blocks in C. * Keep a preserved XPU venv on the XPU index Follow-up to 10ba6e31c, which stopped a direct update wiping a +xpu venv on a hybrid NVIDIA + Arc host but left the rest of the pass believing the host was CUDA. The index chain prefers NVIDIA over Intel, and the CUDA arm does not --reinstall-package torch, so uv left the +xpu wheel in place as satisfied while installing triton-windows over torch's XPU triton -- and with $XpuIndexUrl null nothing swapped it back. A half-converted venv is worse than either end state, so the preserved case now selects the xpu leaf, ahead of the NVIDIA arm and behind an explicit pin. The hardware report is untouched: there really is an NVIDIA GPU in the machine. install_python_stack: an inconclusive XPU probe was always read as a flavour mismatch, but on a stalled Intel driver under a SUPPORTED wheel that is two 90-second hangs and two force-reinstalls of the whole trio on every update, repairing nothing. The disk answers what the probe cannot, so a supported wheel now yields the driver warning and an unsupported or missing one still repairs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Lowercase the setup.sh pin leaf like every other index parser install.sh's _torch_index_url_leaf, setup.ps1's Get-TorchIndexLeaf and install_python_stack's _torch_index_leaf all lowercase before classifying. This copy did not, so UNSLOTH_TORCH_INDEX_FAMILY=XPU (or a URL ending in /XPU) left the leaf uppercase, the equality test against "xpu" failed, and the fast path stayed on. Those same classifiers call that pin XPU once they are reached, so the wheel was never migrated and the update silently repaired nothing. The comment above the line already claimed to match the shared parsers; now it does. Three cases added to tests/sh/test_setup_xpu_fastpath_escape.sh (FAMILY=XPU, FAMILY=Xpu, a URL ending /XPU), plus one that lowercasing must not widen the match: a custom leaf like PRIVATE-XPU stays an unknown family. All three fail against the previous line and pass now. * Do not promise CPU training when the XPU runtime will not start The unavailable-runtime arm said training and GPU inference run on CPU until the driver is fixed. They do not: with neither CUDA nor XPU available, get_device_type() in unsloth/device_type.py raises NotImplementedError, so importing unsloth fails outright rather than falling back. llama.cpp is unaffected, which is what chat and GGUF actually run on, so say that instead. The drift guard added with it needed two passes to be worth anything. Anchoring the arm on the flag name alone matched the bitsandbytes block instead, whose own "4-bit QLoRA may be unavailable" warning made both assertions pass on any wording; and the arm's explanatory comment quotes the phrase it must not use, so comment lines have to go before the grep. Restoring the old message now fails both checks. * Let an explicit non-XPU pin migrate off an XPU wheel Two halves of the same gap: asking for CUDA/ROCm/CPU on a host already running +xpu did nothing. setup.sh: the fast-path escape fired only when the pin itself was xpu, or when a stale generic triton shadowed the build. With an up-to-date install, a +xpu wheel and the pin switched to another family, neither arm matched, install_python_stack never ran, and the authoritative pin was ignored. Added an arm for that case, digit-gated like the shared classifiers so a custom verbatim leaf (rocm-current, cu-private) stays UNKNOWN and does not force a pass that repairs nothing. install_python_stack: _ensure_cpu_torch classifies the installed build and returns early on "already a CPU build". Its probe tested hip, rocm, cuda and +cu<digits>; an XPU wheel sets neither torch.version.cuda nor .hip, so it read as CPU and an explicit CPU pin over it did nothing at all. Keyed on the +xpu local label, since torch.version.xpu is None on some builds. Additive: +cu128 and +rocm still read gpu, +cpu and untagged still read cpu. The escape suite's extractor stopped after the second _SKIP_PYTHON_DEPS assignment, so adding a third arm truncated the block and the new cases failed while the old ones passed. It now stops at the next outer arm and asserts exactly three arms extract, so a future arm fails loudly instead of disappearing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop a wedged Intel driver from blocking the paths that repair it Both of these are fallout from making the CPU repair XPU-aware: it now has to classify an XPU wheel, and every route to that classification went through `import torch`, which loads the SYCL runtime and blocks on the exact host these paths exist to rescue. install_python_stack: the classifier probe times out after 90s and the except branch returned, so an explicit CPU pin over a wedged +xpu venv stayed a no-op. Classify off disk on timeout via _installed_torch_label_on_disk (find_spec, no interpreter) and fall through to the repair. Gated on a GPU label so a slow but healthy CPU-only host does not force-reinstall torch every update. install.sh: the rollback preservation probe read torch.__version__ through the interpreter at venv-replacement time, ahead of every bounded probe in setup.sh, so a hang there took the whole installer with it. It now reads torch/version.py, the same source _installed_torch_version_for_tag already uses for this reason. The interpreter stays as the fallback for a layout without one, where torch is absent and the import fails fast. The install.sh test executes the block against a fake venv whose stub interpreter records being called, so "read off disk" is proven by the interpreter never running rather than by reading the source. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match torch index families exactly, and judge the XPU fast path on the wheel studio/setup.sh classified a pin as a known non-XPU family with prefix globs (cu[0-9]*, rocm[0-9]*), so cu128-private, cu128rc1, cu128.1, rocm7.2-private, rocm7. and rocm7.2.1 all read as known while install_python_stack calls every one of them UNKNOWN and runs no repair: the fast path was cleared and the dependency pass that followed applied nothing, every update. It now matches exact families like install.sh _is_pip_rocm_family_leaf and install_python_stack _is_cuda_family_leaf: cpu, cu<digits>, rocm<digits>[.<digits>], gfx<digit>... (gfx stays a prefix on all three sides, since gfx120x-all is a real Radeon index leaf). studio/setup.ps1 keyed the same escape on torch.xpu.is_available(), which is also false for a supported +xpu wheel on an old or wedged compute driver. No dependency pass can repair a driver, and the pass force-reinstalls nothing when the flavour already matches, so each studio update repeated the bounded probes and a full resolution just to reach the warning Assert-XpuRuntimeReady already prints. The escape now asks Test-VenvTorchIsXpuSupported, which reads torch/version.py off disk and applies the same 2.6 <= v < 2.11 window, matching what setup.sh does on POSIX and removing the last import torch from a path an Arc host with a stalled driver is most likely to hit. Its only caller gone, Test-TorchXpuVersionSupported is removed. Tests: the escape test now also asks install_python_stack itself about a 28-leaf corpus and asserts the shell predicate agrees leaf for leaf, so the two cannot drift again (75 checks; 11 fail against the previous globs). The pre-report test covers the new helper and asserts the fast-path escape names no readiness probe and launches no interpreter. * Trim comments across the Intel XPU detection changes * Normalise setup.ps1 line endings before the wiring regexes A Windows checkout returns CRLF, so the fast-path escape pattern, which is anchored on a literal \n, matched nothing on windows-latest: the region came back empty, "the escape was found" failed, and the two -not checks inside it reported PASS with nothing to look at. Cross-platform parity caught it on windows-latest with 3 failures. $setupText is now normalised to LF once at the read, which covers both literal newline patterns in the file, and a new check asserts the raw CRLF form does NOT match the same pattern, so it is the normalisation rather than luck that makes this work. Verified against a CRLF copy of setup.ps1: the previous test fails there with exactly those 3 checks and the new one passes. * Run the Triton swap after every torch migration, not between two of them _ensure_xpu_triton keys off the installed +xpu label when no explicit XPU pin is set, and it ran ahead of _ensure_cpu_torch. So an existing +xpu venv updated with an explicit CPU pin had generic triton removed and XPU triton installed, and only then did _ensure_cpu_torch replace torch with the CPU build: a CPU environment whose top-level triton package is the XPU implementation, with the generic triton its own dependency set declares now gone. The CUDA and ROCm repairs already ran ahead of the swap, so their pins left the label correct by the time it read it; CPU was the one migration that did not. Moving the swap to the end of both repair blocks fixes it for every family at once and removes the ordering assumption entirely. The new test asserts the order on the AST at both call sites, so a reflow cannot fake it; against the previous order it fails on the first assertion. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bound the wedged-driver probe without GNU timeout macOS ships no GNU timeout (Homebrew coreutils installs it as gtimeout), so on the macOS parity leg the `timeout 30 python3 ...` line exited 127 the instant it was called. The test reads only the exit code, and 127 is non-zero with an elapsed time of 0, so both assertions passed without python ever starting: the alarm behaviour they exist to prove was never exercised on macOS. Replaced with the script's own background watchdog, which behaves the same on every platform, and added a lower bound on the elapsed time. The alarm is 2s, so a run that returns instantly did not execute the probe, which is precisely how the missing-timeout case looked. Verified by shimming `timeout` to exit 127: the previous test still reports 37 passed, and by shimming python3 to return instantly: the previous test still reports 37 passed while this one fails on the deadline check. * Trim comments in the Intel XPU detection changes --------- Co-authored-by: CommandCodeBot <noreply@commandcode.ai> Co-authored-by: danielhanchen <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-08-04 03:03:10 -07:00
Daniel Han	b6781a8bfe	CI: prove the installer works on a machine with no developer toolchain (#7551) ... * CI: prove the installer works on a machine with no developer toolchain No job has ever run the installer on a machine without one. studio-mac-install-matrix.yml is the only macOS installer job and it runs 'bash install.sh --local --no-torch' on runners that already have the Xcode CLT selected and setup-python preinstalled, so the CLT gate never fires there, and --local is precisely the mode that legitimately needs git. Repo-wide there was zero coverage of xcode-select or CommandLineTools outside install.sh itself. clean-machine-install-ci.yml runs the installer on a genuinely stripped machine. macOS legs move /var/db/xcode_select_link, /Library/Developer/CommandLineTools, /Applications/Xcode*.app and Homebrew aside, so xcode-select -p, git, cc and clang really do fail, and restore unconditionally afterwards. Removing the select-link alone is not enough: xcode-select falls through to a full Xcode.app and re-arms /usr/bin/git. Linux legs use containers, which are genuinely clean. Windows legs cover winget visible and masked, plus windows-11-arm. A WSL leg covers the 126 lines of WSL-specific install.sh logic that had no runtime test. Each macOS leg runs four deliveries: pipe (the advertised command, and the shape that turns an early exit into curl (56)), file (separates installer logic from pipe delivery), no-torch, and tauri (stdin closed, no tty, as the desktop app invokes it). One leg records every toolchain invocation and asserts the trace, which is the real deliverable: proof the installer never reached for a compiler rather than proof it happened to succeed. The asserts test that tools do NOT WORK rather than that they are absent from PATH. On a real virgin Mac /usr/bin/git and /usr/bin/cc exist as CLT stubs, so 'command -v git' succeeds and only running it tells the truth. desktop-app-clean-machine-ci.yml installs and launches the SHIPPED desktop app release on a stripped machine, covering Gatekeeper and quarantine on macOS, NSIS silent install on Windows, and Xvfb with WebKit2GTK on Linux. Known limit, stated plainly: hosted macOS runners are developer machines. Masking reproduces this bug and proves the installer does not invoke a toolchain, but it cannot prove no hidden dependency exists on a truly virgin Mac. An ephemeral-VM lane is the follow-up. * Point the llama assert at the right root, and name the Intel limitation The tauri leg installs to the legacy root because --tauri refuses a custom UNSLOTH_STUDIO_HOME. Its install succeeds end to end, but llama.cpp lives at <root>/llama.cpp while the venv is at <root>/studio, so the assert was pointed one level too deep. On macos-15-intel /usr/bin/git keeps working once the CLT are gone, so it is not CLT-provided there and no masking can remove it, while cc and clang do become stubs. Calling that 'masking failed' was wrong. That leg allowlists git explicitly and says why, so the assert stays strict everywhere else. * Make the clean-machine legs able to fail The toolchain strip never ran on the automatic triggers: inputs exists only for workflow_dispatch, and GitHub coerces '' and false alike to 0, so `inputs.strip_toolchain != false` was false. Confirmed on a pull_request run where the strip step reports skipped. Gate on the event instead. Also: scrub the Machine and User registry PATH, since install.ps1 rebuilds $env:Path from them mid-install and the toolchain came back; stop dropping WindowsApps unconditionally, which removed winget on the winget=visible leg too; fail rather than annotate when a bundle ships no installer or no CLI; run the bundled installer, which a headless launch never reaches; resolve the newest desktop-v* release instead of a pinned immutable tag; and give the two macOS matrix rows distinct artifact names. * Make the Windows and Linux clean-machine legs honest The Windows scrub only touched PATH, so the legs were green while not clean: run 30365014702 logged "python ABSENT" and then "Python 3.13 already installed" with uv resolving C:\hostedtoolcache\windows\Python\3.13.14\arm64\python.exe. py.exe lives in C:\Windows and uv discovers interpreters itself, so take the toolcache off disk and fail when tooling survives, instead of only printing it. The Linux desktop legs never stripped anything, and the tauri.log step was all || true so it could not fail. Run the bundled installer the way install.rs does, with --tauri alone, and assert torch: passing --no-torch skipped the slowest half of first launch and let the venv check pass over it. Pin the WSL rootfs to a dated build; current/ is a rolling alias and the digest next to it is fixed. * Give the Linux and WSL legs an assertion that can fail The Linux rows' only post-install gate was nobuild, a log grep, so an installer exiting 0 having produced nothing kept a required leg green. The WSL job and the Windows job both already check the install runs; the Linux job now does too. The WSL detection half only printed its Select-String, and the alternation also matches "platform linux", so a regression that skipped every WSL-specific branch would still pass as a plain-Linux install. Assert the exact marker, stripping ANSI first since step writes the label in reverse video. Probed against three fixtures: real wsl log passes, platform linux fails, missing log fails. * Tighten the clean-machine comments Compress the comment blocks across the clean-machine workflows and scripts. The explanations of why each check is written the way it is stay; the padding, restatement and duplication go. No code or workflow logic changes. * Point the nightly at the repo that publishes, and let its checks fail REL_REPO defaulted to unsloth-test/unsloth-test, which holds one release frozen at 2026-07-27, while release-desktop.yml publishes into github.repository. The schedule was re-testing the same fixture forever and could never see a broken production bundle. The windows job carried a blanket continue-on-error, so its NSIS assertions could not gate. lipo -archs prints and exits 0 for a thin binary and `|| true` swallowed even that, so the architecture was never checked; fall back to file, which survives the CLT mask. And require the preflight disposition line rather than the mere existence of tauri.log, which setup_logging creates at process start regardless. * Stop four clean-machine checks from passing over a real failure Re-run `absent` after the install on the masked macOS legs. It only ran before, so an installer that quietly selected the Xcode CLT or installed a compiler left the leg green while every later source build could succeed, which is the one thing clean-machine-assert.sh says `absent` guards the whole run against. Fail the Windows simulation when py.exe can still start an interpreter. The launcher binary itself may stay, but Find-CompatiblePython probes `py` first (install.ps1:1130-1153), so an interpreter registered outside the two renamed toolcache directories gets reused and Python bootstrap is never exercised. Exempting `py` without ever running it left that unchecked. Propagate the WSL installer exit code. It was printed and discarded, and the CLI check does not compensate: install.sh links the `unsloth` shim (4174-4182) before it reports a failing studio/setup.sh (4219-4230), so a late setup failure leaves a shim whose --version succeeds. Run the bundled installer in the Linux desktop jobs. The launch step only proves the process stayed alive, and on a fresh home preflight reports not_installed and the app waits on the install screen, so both required rows passed after 90 seconds without ever touching the shipped install.sh. Locate the resource in the deb payload or the extracted AppImage, run it the way install.rs does, and require a managed venv that can import torch. * Prove the trace wrapper records before trusting an empty trace The `notools` check reads an absence: it passes when the trace file contains no compiler, git or brew invocation. A shim directory that never reached PATH produces exactly the same empty file as an installer that touched nothing, so the single leg carrying that assertion would stay green no matter what the installer did. "Verify the simulation actually took effect" only ran for mask mode, which left the trace leg with nothing checking its own instrumentation. Call git explicitly after sourcing the environment and require it to appear in the trace, then truncate the file so the self-test entry does not count against the install. The call has to be explicit because macOS reaches _has_working_git only under STUDIO_LOCAL_INSTALL (install.sh:2026), so no consumer leg on that platform probes git on its own. * Stop the Windows clean-machine check failing on its own probe exit code All three Windows legs failed "Verify the simulation took effect" with no ::error:: printed at all. The check itself was right: the mask step logged "masked toolcache python: C:\hostedtoolcache\windows\Python", python/git/cmake/cl were ABSENT, no `py -3.x` probe started an interpreter, and the winget assertions were satisfied. The step still exited 1. The cause is $LASTEXITCODE leaking out of the step. The last external command is the `py -3.13` probe, which is SUPPOSED to fail; Get-Command and Write-Host are cmdlets and never reset $LASTEXITCODE, and the runner appends `if ((Test-Path -LiteralPath variable:\LASTEXITCODE)) { exit $LASTEXITCODE }` to every pwsh step (actions/runner#351). So a clean machine reported failure, and because this step runs before Install, no Windows leg has ever reached the installer. Clear $LASTEXITCODE after the probe loop and end with an explicit exit 0. The leak detection is untouched: a surviving python/git/cmake/cl, or a `py -3.x` that actually starts, still exits 1. Also print each probe's exit code and output, so the next failure here explains itself instead of being silent, and label `py -0p` as what it is. The launcher reads the registry, which the on-disk toolcache rename cannot rewrite, so -0p keeps naming paths that no longer exist. Unlabelled it reads like a leak. Accept the Fedora leg's real outcome instead of a message that can be absent The fedora assertion only accepted the unsupported-package-manager hard exit. That is still what this ref's install.sh does, but the pending installer change replaces it with a warning that lets the install continue, at which point the old grep matches nothing and the step fails for the wrong reason. Handle both, strictly. If the log shows the newer "using prebuilt llama.cpp (missing:" warning, the Linux gate demonstrably did not hard-stop, and the only tolerated failure past that point is release lag: install.sh comes from this ref while unsloth comes from PyPI, and the released studio/install_python_stack.py has no "skip triton kernels when git is missing" guard, so it still fetches the git+https triton_kernels requirement on a machine with no git. Anything else after that warning fails the step. Otherwise the old hard-exit message is still required. A missing log, a bootstrap outage or any unrecognised failure all remain errors, and the step retires to a plain success assertion once a release ships the no-git skip. * Make the AppImage Linux row actually extract, and hold Linux to the macOS preflight bar The appimage row invoked the extractor by bare filename, and a command word with no slash is resolved through PATH rather than the working directory, so the extraction exited 127 and the bundled-installer assertion below it never ran. Prefix it with ./ so the row exercises what it claims to. The Linux log step also asserted nothing: it skipped a missing log with continue and discarded the grep with || true. The launch step only proves the process stayed alive for 90 seconds, and the bundled-installer checks do not exercise the Rust preflight path, so an app that hung before preflight completed passed both required Linux rows. Require the same desktop_preflight completed disposition= record the macOS rows already do. * Put the branch's own Python under test on the clean-machine legs install.sh and install.ps1 come from the ref under test, but they install unsloth from PyPI, which is the consumer path and has to stay that way. That left everything Python-side coming out of the released wheel: studio/setup.sh, studio/setup.ps1, studio/install_python_stack.py, and every requirements and constraints file those resolve through Path(__file__). A branch that changes constraints.txt or setup.ps1 therefore got a green run that proved nothing about the change, and some legs proved less than they looked. The Fedora assertion was already carrying a hand-written workaround for exactly this, tolerating a triton/git failure on the grounds that the released package lags the ref. Legs marked overlay: true now re-point the venv at the ref just before studio setup runs, through UNSLOTH_CI_SOURCE_OVERLAY: a --no-deps editable install of the checkout. That makes import studio resolve to the working tree, so the existing setup-script lookup finds the ref's setup.sh / setup.ps1 and install_python_stack reads the ref's constraints, with no other change to either installer. Not --local: --local additionally installs unsloth-zoo from a git+https URL, which genuinely needs git, and git absence is the whole point of the masked legs. The overlay resolves no dependencies and clones nothing, so it holds up with git, cmake and the compilers all gone. It is not a consumer knob either: no flag, no usage entry, ignored unless the variable names a directory with a pyproject.toml in it. Four legs stay on the released package deliberately, each for its own reason, recorded in the header: the mac pipe legs keep an end-to-end signal on what a user actually runs; the trace leg would otherwise answer its own question, since the editable build calls git through setuptools-scm's file finder; the non-root Linux leg dies before a venv exists; and WSL only ever receives install.sh, not a source tree. Two supporting fixes the overlay depends on or exposes: install_python_stack.py discarded uv's output whenever a step succeeded, so the nobuild assertion, which reads the install log, could not see a source build in the dependency phase at all. That is the phase that installs studio.txt, where an sdist-only dependency actually turns up, and it reported "built: none" regardless. It now echoes successful output under UNSLOTH_VERBOSE, matching what install.sh's run_install_cmd already does. nobuild now ignores "Building <name> @ file://" lines. A local-path build is something the caller pointed at, never a dependency resolution chose, and index dependencies always print <name>==<version>, so a real sdist from PyPI is still caught, including one named unsloth. Each overlaid leg also asserts it really was overlaid, so an unset variable cannot quietly put the whole matrix back on the released wheel. * Allowlist the triton-kernels pure-Python sdist, and record why Windows on ARM is red The two ubuntu2404 root legs went red at "Assert no source build" reporting triton-kernels. That is not a regression in what the installer does. Those builds have always happened; they only became visible now that pip_install stopped discarding uv's output on success, which is what finally let the nobuild check read the dependency phase at all. So the question was whether each build actually needs a compiler. Checked against the real artifacts rather than assumed: openai-whisper 20250625, randomname 0.2.1, argbind 0.3.9 -- no version of any of the three has ever published a wheel; antlr4-python3-runtime is pinned at 4.9.3, below the first release that ships one. All four sdists use setuptools.build_meta, declare no ext_modules, and contain no .c/.cpp/.pyx/.rs file. Already allowlisted, correctly. triton-kernels is the same category and was the only name failing. It is pinned to the triton repo's python/triton_kernels subdirectory; that tree is 75 files of Python, a four-line pyproject.toml, no setup.py and no native source at all. The kernels are Triton DSL compiled at runtime, not at install time. It is also a direct URL the installer names itself rather than something resolution picked, and only Linux reaches it. It belongs in the allowlist, so add it with that reasoning written down. The allowlist match is now lowercased and underscore-folded on both sides. The requirement spells the package triton_kernels while uv prints triton-kernels, and an allowlist that matched only one spelling would pass by luck rather than by intent. A plain pyarrow sdist is still caught. The two data-designer @ file:// plugin builds needed nothing: they are in-tree local paths, already dropped by the same rule that exempts the source overlay's own build. Separately, the windows-11-arm leg fails for a real reason and should keep failing. The ARM handling itself works, the log shows torchaudio being skipped and torch plus torchvision installing from wheels. What stops it is that pyarrow and hf-transfer publish no win_arm64 wheel at all, so uv falls back to their sdists and they fail on CMake configure and on openssl-sys wanting perl. That is a product gap on the platform, not a gap in the simulation, so the leg stays experimental and keeps reporting it. Record that above the matrix entry so the next reader does not re-diagnose it. * Exercise the bundled Windows installer, and stop mislabelling installer sources Four things that let a leg go green while proving nothing. The desktop Windows job installed the bundle and launched it, and that was all. On a fresh profile preflight reports not_installed and the app sits on the install screen waiting for a click, so the process happily stays alive for 90 seconds without the bundled install.ps1 ever running. A bundle that shipped no install.ps1 resource, or a broken one, passed this job -- which is the packaged app failure the workflow exists to catch. macOS and Linux already invoke their bundled script directly; Windows now does the same, via the resource NSIS laid down next to the exe, invoked the way install.rs invokes it, then asserts the managed venv exists and can import torch. Its timeout goes to 60 minutes because a full torch install on a Windows runner is the slowest of the three. A manual run that selects installer_source: published only redirected the macOS and Linux jobs. WSL kept copying the checked-out install.sh and Windows kept running the checked-out install.ps1, so a run asking whether the script on unsloth.ai works reported on this ref under the published label. Both now honor the selection; install.ps1 advertises its own unsloth.ai URL, so published has a meaning on Windows too. Both branches stay empty on pull_request and push, so automatic runs are unchanged. The push-to-main filter listed only install.sh, install.ps1 and this workflow, while the PR filter also covers setup.sh, setup.ps1, install_python_stack.py and the clean-machine helpers. A direct push touching those skipped the workflow entirely, so the post-merge backstop never ran for the files the source overlay was added to cover. The two lists now match. Neither filter covered studio/backend/requirements, even though the overlay exists precisely so a constraints change is resolved on a machine with no compiler and no cached wheels. The update-smoke workflows cannot stand in: they start from a preinstalled Python and full developer tooling. * Make the Linux and Windows desktop legs clean, and honour published on every macOS delivery The desktop workflow claims all three platforms are stripped, but only macOS and Windows had a strip step and the Windows one scrubbed the process PATH only. Both gaps let a bundle that needs a developer toolchain pass the one workflow whose premise is that it must not. Linux: the job ignored strip_toolchain entirely and ran the bundled install.sh with the runner's git, gcc, cmake and make in /usr/bin. clean-machine-env.sh now has a Linux --remove branch that moves the resolved tool binaries aside, recorded in restore.sh, and the job calls it plus `assert absent` after the apt step (the .deb install needs dpkg) and before the bundled installer, with a restore step to match macOS. The loop repeats per tool so a name present in both /usr/bin and /usr/local/bin is fully masked rather than half masked. Windows: rewriting $env:PATH does not survive the bundled install.ps1, which calls Refresh-SessionPath (318-337) and rebuilds $env:Path from the Machine and User registry values, and py.exe in C:\Windows reaches the toolcache whatever PATH says. Ported the on-disk toolcache rename, the Machine/User registry scrub and the py -3.11/-3.12/-3.13 start probe from clean-machine-install-ci.yml, so the strip is proven rather than assumed. Windows preflight: the log step was Test-Path, Get-Content and Select-String, none of which can fail, so an app that hangs before preflight passed on the 90 second liveness check alone. It now asserts a tauri.log exists and carries a `desktop_preflight completed disposition=` line, the same unconstrained check macOS and Linux already make. The disposition VALUE is deliberately not constrained: ManagedReady over an unbootable venv is the reported bug. installer_source on macOS: only the pipe delivery branched on it, so a `published` dispatch ran the checked-out script on six of the eight macOS rows while the run was labelled published. The script is now resolved once at the top of the Install step and used by the file and tauri deliveries; pipe still re-fetches through the live transport, because that is half of what it tests. Linux, WSL and Windows already honoured the input. Also shortened the comments across the changed files, keeping the reasoning that says why each check exists. * Run the Windows installer under PowerShell 5.1, the only shell a clean machine has The Windows Install step ran `& $script` inside a `shell: pwsh` step, so install.ps1 was executing under PowerShell 7. A genuinely clean Windows box does not have PowerShell 7: Windows ships powershell.exe (Windows PowerShell 5.1) and pwsh is a separate install that the hosted runner image happens to preinstall. So the one workflow whose premise is a machine that has never seen a developer toolchain was testing the installer under a shell that machine would not have, and no other Windows job anywhere in .github exercises install.ps1 under 5.1. Invoke it the way the desktop does (install.rs:325-339, and the bundled installer step in desktop-app-clean-machine-ci.yml): powershell.exe with -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Bypass -File. The pwsh step wrapper stays, since it is only the installer that has to be under 5.1. Calling powershell.exe with `&` keeps the output in the pipeline, so Tee-Object still fills logs/install.log, and $LASTEXITCODE after the pipeline is the child's real exit code, so $rc and `exit $rc` are unchanged. install.ps1 and studio/setup.ps1 hold no PowerShell 7-only constructs: no `#Requires` above 5.1, no `&&`/`||` chain operators, no ternary, no null-coalescing, no ForEach-Object -Parallel, no $IsWindows/$PSStyle, and no 6+ cmdlets or parameters. setup.ps1 declares `#Requires -Version 5.1`, and its three $PSVersionTable branches gate a 7-only preference on the 7 side with a 5.1 fallback. Every Invoke-WebRequest already passes -UseBasicParsing, which 5.1 needs because it otherwise reaches for the IE engine. * Assert the Windows desktop strip actually took effect The desktop job's Windows masking renamed the toolcache Python, scrubbed the Machine and User registry PATH, and probed `py`, but nothing checked that `python`, `git`, `cmake` or `cl` were gone. The drop list is heuristic path fragment matching, so a runner image that moves any of those outside those fragments leaves the bundled install.ps1 reusing hosted developer tooling while the job still reports a clean machine. PATH written to $GITHUB_ENV only applies to later steps, so the check has to live in a step of its own; it carries the same event gate as the strip, exempts `py` (it lives in C:\Windows and stays, which is why the start probe is the real evidence), and resets $LASTEXITCODE before exiting 0 so an intentionally failing probe cannot fail a clean machine. Also correct the no-winget matrix note: that leg is not failing for an unfixed product reason. It stops at the unconditional git gate in setup.ps1 only on this ref, and with that gate relaxed it passes along with every other leg, so the row is a merge order dependency and stays required. * Resolve the desktop release including drafts, the convention this repo ships All three desktop legs died at the download step with an empty REL_TAG. The resolver passed --exclude-drafts while REL_REPO now defaults to github.repository, and every desktop-v* release in unslothai/unsloth is a draft: desktop-v0.1.50-beta and desktop-v0.1.471-beta are both drafts carrying the .dmg, .deb, .AppImage and setup.exe, while only the non-desktop tags like v0.1.501-beta are published. Excluding drafts therefore matched nothing and no leg could ever run against a production bundle. Drop --exclude-drafts so the newest desktop-v* release is found. A draft has no tag ref, so releases/tags/<tag> 404s for one, but gh resolves drafts over GraphQL and gh release download <tag> fetches their assets normally, so the download call is unchanged. Listing drafts requires push access, which for GITHUB_TOKEN means contents: write, so the workflow permission is raised from read and annotated. When nothing resolves the leg still fails hard rather than skipping: with no bundle to install there is nothing to prove, so a green run would be a lie. The error now names both causes, no release cut yet or a token that cannot see drafts. Also stop the restore step swallowing its own failure. `bash .clean-machine/restore.sh || true` printed "No such file or directory" whenever an earlier step failed before the toolchain was stripped, and hid a genuinely broken restore just the same. Skip explicitly when the file is absent and let a real restore failure surface. Same fix in clean-machine-install-ci.yml, which had the identical line. * Skip the desktop jobs on fork PRs instead of failing them Every desktop-v* release in this repo is a draft, and GitHub lists drafts only to a token with push access, which is why resolving one needs contents: write. A pull request from a fork receives a read-only token no matter what the workflow declares, so on those runs the resolver cannot see any release and the job died on "no desktop-v* release visible", accusing the repo of having no bundle when the real cause is the trigger. This workflow runs on pull_request for changes to itself and the stripping scripts, so an outside contributor editing either would have hit that. Guard the three jobs on the head repo not being a fork. A skipped job is honest here: it does not claim to have tested a bundle it was never able to download, and it is not reported as a pass. * Close the free headroom in the clean-machine simulation Assert arch and signature on every downloaded Mach-O. This is the one genuine gap the simulation had: Rosetta 2 is preinstalled on hosted runners and absent from a factory-fresh Mac, so an x86_64-only llama.cpp, whisper.cpp, Node or uv payload runs green here and dies with "bad CPU type in executable" for the user. llama-server launching under `assert-llama-loads.sh` does not rule that out, because Rosetta makes it launch. The new `macho` check reads `file -b` (`lipo` is an xcrun shim and is gone after masking, as the desktop lane already notes) and keys the expected arch off `uname -m`, so macos-15-intel expects x86_64. It also requires at least an ad-hoc signature on arm64, which closes the AMFI "Killed: 9" class that uv has already been bitten by; the check is skipped on x86_64, where unsigned code loads fine and so is not the same defect. It fails when the scan finds nothing, since an empty scan reads exactly like a clean one. Make absence real rather than PATH-hidden. uv probes well-known interpreter locations and the framework loader ignores PATH entirely, so hiding the toolcache only hid it from `command -v`. Empty /usr/local (it EXISTS on a factory-fresh Mac as a SIP-exempt firmlink, and is empty; it is /usr/local/bin that is absent, so the directory itself stays), move the hosted toolcache and /Library/Frameworks/Python.framework aside, and clear the developer dotdirs and caches. A populated uv or pip cache can also satisfy a resolution that would fail on a user's machine. Every removal goes through --remove and is recorded in the generated restore.sh, guarded so a path the install recreated is not buried inside its own restore. Unset CI, GITHUB_* and RUNNER_* for the installer process only. An installer branching on CI=true is a hidden dependency no consumer exercises. Scoped to the child so the step's own $GITHUB_OUTPUT still resolves. Record spctl --status and csrutil status. Neither is documented for these images and both change what a binary is allowed to do. * Pin the two failures no change here can fix, and add the virgin Windows container lane Three red checks, two of which test something this branch does not own. desktop linux deb / appimage run the SHIPPED bundle's own install.sh, and desktop-v0.1.50-beta was cut on 2026-07-21, before #7547 merged on 07-29. That bundle still carries the old optional-dependency gate, so on a stripped runner it exits 2 at [TAURI:NEED_SUDO] cmake git build-essential libcurl4-openssl-dev and never creates a venv. Current main's _check_linux_deps runs the same set through _SMART_APT_OPTIONAL, which suppresses every escalation path, so only a new release can change this. The step now pins that exact outcome: the exit code must be 2 and the log must carry exactly that package list, anything else still fails, and finding _SMART_APT_OPTIONAL in the extracted install.sh (the guard #7547 added) turns into a hard error saying to delete the pin. The venv and torch assertions stay and still run whenever the installer succeeds. win windows-11-arm gets a native ARM64 CPython, and torchaudio publishes no win_arm64 wheel at any version, so the PyTorch step cannot resolve. The fix is in install.ps1 on #7549, still open. Same treatment: the Install step is continue-on-error and a new step requires all three of the PyTorch step, the torchaudio resolution error and the missing win_arm64 platform tag, so any other failure is red. The row leaves experimental so the job is required, and the pin errors out as soon as the venv interpreter reports anything but win-arm64, which is what #7549 landing looks like. Adds the virgin Windows container lane as two jobs here rather than a sibling workflow: same premise as the win legs, same path filters, and masked-versus-real reads better side by side. The hosted Windows legs cannot test the VC++ 2015-2022 runtime (it ships in the runner image's System32) or a Windows with no Microsoft Store, and a servercore:ltsc2022 container on windows-2022 answers both. The probe asserts no python, py, git, cmake, cl, winget or uv on PATH, on disk or in the registry, and now also asserts vcruntime140.dll, vcruntime140_1.dll and msvcp140.dll are absent, which is the one thing the hosted runner cannot un-ship. Both container install rows stop at studio/setup.ps1's winget-only git gate on this branch, since #7549 is what relaxes it, so both are pinned the same way. The overlay row additionally requires the UNSLOTH_CI_SOURCE_OVERLAY hook to have fired, unconditionally: without that it would be indistinguishable from the released-wheel row, and the hook is this branch's own feature. Container notes carried over from the spike: never docker pull when the image is cached, since MCR has shipped an image ahead of the runner host before; wait for the Docker daemon, because one leg died in 21s on npipe:////./pipe/docker_engine and that flake misreads as "Windows containers unavailable"; drive docker from a run: step, because the job-level container: key is Linux-only. The root CA store is seeded after the virginity assertion, restoring what a real Windows already has, because studio/install_node_prebuilt.py downloads Node with bare urllib.request.urlopen and hits CERTIFICATE_VERIFY_FAILED against the empty container ROOT store. That product bug is left alone here. * Check signatures on Mach-O main executables only The macho check asserted a valid signature for every Mach-O under the studio home, and failed the macos-15 mask/pipe leg on 29 files: lxml, charset_normalizer, cygrpc, _upb, fontTools, caio, brotli and a bundled libportaudio.dylib. Those are MH_BUNDLE and MH_DYLIB images dlopen'd into a process without library validation, they ship unsigned in the wheels, and the same run had already installed and imported them with the installer exiting 0. Key the signature half off the Mach-O filetype and run it only on main executables. Report an absent seal separately from one that fails to verify, and capture codesign output instead of piping it into grep, which returned the unsigned exit status through pipefail and called every unsigned binary broken. The architecture half is unchanged and still a hard failure: it is what closes the Rosetta 2 gap. The zero-Mach-O guard is unchanged. The .venv_t5_* sidecars stay in scope; setup.sh creates them during a normal install and transformers_version.py puts them on sys.path, so they are payload. * Make the WSL job gate, assert Windows installed no toolchain, strip before the .deb * Assert the root Linux legs did not compile llama.cpp with the apt-installed toolchain * Pin the macOS desktop legs on the same pre-7547 release lag The Linux rows already pin the shipped bundle's own install.sh exiting 2 at the NEED_SUDO handshake. macos-15 and macos-26 fail the same way for the same reason: desktop-v0.1.50-beta predates #7547, so the bundled installer still hard-exits on the Xcode CLT gate that #7547 turned into a warning. Accept exit 1 plus that exact gate line, and nothing else. _check_macos_deps is the function #7547 added, so its presence in the bundle means the release caught up and the block errors out asking for the pin to be deleted. * Pin the WSL pipe truncation and the masked-winget git gate The WSL leg dies at install.sh:2082 with an unterminated quoted string. Nothing is wrong with that line: piping the script into sh is not atomic. dash reads it from the pipe in 8192-byte blocks and runs each command as it parses, and install.sh:2007 calls _maybe_reroute_strixhalo_to_2404, which on WSL alone shells out to Windows interop; interop relays the stdin it inherited and drains the pipe. dash has 11 blocks buffered at that point, ending at byte 90112, which falls inside "$STUDIO_LOCAL_INSTALL" on line 2082. Truncating install.sh at 90112 and parsing it reproduces the message verbatim, and running the whole file under a stdin-draining interop stub reproduces the exit code too. #7548 wraps the body in _unsloth_main so sh parses everything before running anything, and the same reproduction against its head is clean. The eight green staging runs cited when this job's continue-on-error came off were all on trees that already carried #7548, so that evidence never covered this branch. Pin the exact signature instead: exit 2 plus the shell's own unterminated-quoted-string error, with the _unsloth_main marker read back out of the distro as the flip condition. Pin winget=masked the same way. studio/setup.ps1:1655-1669 gates on git unconditionally and can only fetch it through winget, so masking winget leaves no way to satisfy it. #7549 relaxes the gate, and its wording appearing in the tree retires the pin. * Retire the WSL pipe pin now that #7548 is in main The pin flipped exactly as designed: it looks for _unsloth_main in the installer it actually ran, and #7548 put it there. Delete the pin and the CLI waiver, and assert the opposite instead. WSL is the only platform whose install shells out to Windows interop mid-script, and interop relays the stdin it inherited, so this job is the one that can catch the pipe being drained again. A truncation here is now a hard failure. * Gate the no-elevation Linux install and split off the no-transport case * Assert no source build on the hosted Windows legs and keep winget for the desktop lane * Retry the container root CA seeding instead of failing on one Windows Update timeout * Run the clean-machine workflow for the prebuilt installer helpers it overlays * Narrow the container pin to its own gates and scan uv and the venv interpreter for arch * Tighten the clean-machine comments * Re-assert toolchain absence after the desktop .deb pulls its dependencies * Retire the #7549 pins and add a wget-only Linux leg #7549 is in main, so the three known-outcome pins that were waiting on it are stale and would now hard-error by design. Each is replaced by the assertion it was standing in for rather than deleted: win windows-11-arm now gates. The x64-on-ARM64 resolver is asserted as an outcome: the venv interpreter reports win-amd64 from its own sysconfig, and torchaudio (no win_arm64 wheel at any version) is installed. Measured on the integration branch before #7549 merged: "only a native ARM64 Python 3.13 was found" -> "installing x64 Python" -> torchaudio 2.10.0+cpu, install green. win windows-latest / winget=masked now gates. The relaxed git gate is asserted from both sides: the old unconditional message must be absent, the no-git branch must have been reached (so the row cannot pass because git leaked back onto PATH), and setup.ps1 must report git as absent-but-not-required. Both Windows rows, and the visible one, gained the usability check the Linux legs have had and Windows never did: a managed interpreter, an unsloth CLI on disk, and that CLI actually running. nobuild and the toolchain check only read the log, so an installer that exited 0 having produced nothing satisfied them. The torch assert also loses its fallback to whatever `python` resolves to. The virgin container overlay row gates, and asserts what only that lane can: it is the one environment whose System32 does not already ship the VC++ 2015-2022 runtime, so it is the only place Ensure-VCRedist's direct aka.ms download can be proved to run rather than be short-circuited. The overlay=false row keeps a pin, with a new reason: it installs unsloth from PyPI on purpose, and setup.ps1 inside 2026.7.5 (uploaded the 23rd) predates #7549, so it still stops at the old gate. That is release lag, it flips on the next release, and the pinned signature is now the old wording rather than "#7549 has not landed". Also adds linux ubuntu2404-nonroot-wget. install.sh's download() takes curl or wget and _transport_missing is true only when both are gone, so a wget-only box is supported on paper, but the gating nonroot leg provisions ca-certificates AND curl, so curl won every probe and the wget branch had never run. Same image, same no-sudo user, same asserts, wget instead of curl, and curl proved absent on disk for root and for tester before AND after the install, so the claim is that every download went through wget rather than that curl happened to be unused. * Tighten the clean-machine CI comments Comments only, no assertion logic, pins or leg definitions touched. Reflowed every rationale block to denser wording and removed the duplication that had built up across repeated steps: the desktop workflow repeated the fork-PR skip, the desktop-v* tag resolution and the restore-runner note once per platform, and the installer workflow repeated its path-filter rationale in both the pull_request and push blocks. Those now point at the first copy. Every WHY is kept: why the masked legs avoid install.sh --local, what UNSLOTH_CI_SOURCE_OVERLAY is for, why `absent` tests "must not work" rather than command -v, why the .venv_t5_* sidecars are in the macho scan scope, why the signature check is main-executables-only, why each nobuild allowlist entry is a pure-Python sdist, why the WSL job gates and what the pipe truncation was, and why the virgin container's overlay=false row is still pinned. Proved comments-only three ways: both workflow revisions parsed with yaml.safe_load_all and every leaf walked (only `run:` scalars differ); every changed bash body and .sh compared byte-for-byte after `bash --pretty-print -n`; every changed pwsh body and .ps1 compared as a token stream with Comment and NewLine tokens dropped. A negative control injecting one non-comment line into each layer makes all of them fail. * Clean machine CI: strip Strawberry, make the Fedora pin gating, run the Linux CLI desktop windows failed the strip verification because windows-latest ships a MinGW toolchain under C:\Strawberry\c\bin, which matches none of the drop fragments; the installer workflow already scrubs it. Fedora sat behind job-level continue-on-error, so its outcome pin could not fail the run. Tolerate the install step instead, as the no-transport row does. The Linux usable-install check only tested the executable bit; Windows and WSL already execute the CLI. The macho scan now fails when no venv interpreter was scanned, rather than letting uv alone satisfy the outside-root guard. * Clean machine CI: tighten the comments Round 12 comment reduction: compress wording, keep every reason. Comments only, verified with a YAML leaf walk (differences only inside run: scalars, only on # lines), bash --pretty-print -n byte comparison, a PowerShell token-stream diff and a Python AST comparison. * Clean machine CI: dereference the venv interpreter, pin the deb deps and the Windows disposition file did not follow the <venv>/bin/python symlink find -L printed, so it answered 'symbolic link to ...' and the Mach-O test dropped the one interpreter the Rosetta scan exists to check. Read with file -Lb and count what was classified, not what was found. apt treats a toolchain package the strip only renamed as already installed, so a .deb that started declaring git or cmake would never restore it and the absent re-check would still pass. Assert the declared Depends instead. The Windows lane accepted any preflight disposition although the bundled installer was already required to build a working venv; NotInstalled or ManagedStale there means the app cannot boot what it just installed. * Clean machine CI: assert every masked tool, and re-select the developer dir last clean-machine-env.sh moves ten tools aside and only warns when a move fails, but absent checked four of them, so a surviving gcc -- which install.sh probes for build-essential -- went unnoticed. restore.sh ran xcode-select --switch before the line that moved CommandLineTools back, so it named a still-masked directory, failed into || true and left the selection link unrestored. Capture the original selection and re-apply it after both directory restores. --------- Co-authored-by: danielhanchen <unslothai@gmail.com>	2026-07-29 06:21:40 -07:00
Daniel Han	df63522369	Installer: stop requiring a developer toolchain on the consumer path (#7547) ... * Installer: stop requiring a developer toolchain on the consumer path A brand new Mac cannot install Studio at all. install.sh gates on `xcode-select -p` and exits 1 with 'Xcode Command Line Tools are required', and Linux exits 1 on any non-apt distro over cmake/gcc/git/libcurl headers. Nothing under either gate needs a toolchain. uv is a prebuilt binary, CPython comes from uv's managed python-build-standalone, llama.cpp and whisper.cpp are prebuilt downloads, Node is a pinned nodejs.org archive, and triton is skipped on macOS. unslothai/llama.cpp b10107-mix-1911198 publishes macos-arm64, macos-x64, linux-x64 and linux-arm64 builds covering cpu, cuda12, cuda13, rocm and vulkan. PR #6617 already dropped the Homebrew/cmake stop on macOS for this reason and just left the CLT stop behind. macOS: warn and continue when the CLT are absent. Linux: only a download transport (curl or wget) is fatal; build tooling warns. Both keep a hard git requirement for --local, which installs unsloth-zoo from a git+https URL. Both gates move into functions so tests/sh can extract them. The old inline form could not be reached by the tests/sh convention, which is why this shipped broken and stayed broken. test_macos_clt_gate.sh (19 assertions) and test_linux_deps_gate.sh (25) cover the clean machine, the CLT-stub shape where /usr/bin/git exists but fails, the non-apt distro, and the --local paths. Writing the Linux test caught a latent bug: the gate trimmed its list with $(echo ... | sed ...), so on a minimal image without sed the substitution yields empty and it reports 'all system dependencies found' on a machine with none of them. Replaced with parameter expansion. Also caps av<16 in the single-env constraints. av 16+ ships no cp313 macOS arm64 wheel, and it is a C extension over FFmpeg, so uv would silently fall back to a source build needing both a compiler and FFmpeg headers. Verified on GitHub-hosted macOS runners with /var/db/xcode_select_link, /Library/Developer/CommandLineTools, /Applications/Xcode*.app and Homebrew moved aside. macos-14, macos-15 and macos-26 fail on main and install cleanly with this; the recorded tool-invocation trace for the whole install is a single `xcode-select -p`, so nothing compiled and nothing installed a toolchain. * Linux: auto-install git rather than dropping it, and skip triton kernels without it Making git optional on Linux was too broad. studio/backend/requirements/ triton-kernels.txt line 2 is a git+https URL, so step 6/14 died with 'Cannot find command git' and failed the whole setup on ubuntu2404-root, ubuntu2404-arm-root and fedora41, all of which had been passing. The claim that nothing on the consumer path needs git holds on macOS, where triton is skipped, but not here. install.sh now auto-installs git through apt with the other optional tooling, so Debian and Ubuntu are unchanged. The triton kernels step skips with a message when git is absent instead of failing: they are a training speedup, not a boot requirement, and a GGUF chat install has no use for them. Six more assertions pin both halves. * macOS Intel: skip the one package with no x86_64 wheel The Intel clean-machine leg installed with the toolchain masked, then died in studio setup: subprocess.CalledProcessError: Command '['cmake', ...]' returned non-zero ERROR: Failed building wheel for pytorch_tokenizers pytorch_tokenizers publishes wheels for macOS arm64, linux x86_64, linux aarch64 and windows, but none for macOS x86_64 at any Python version, so uv falls back to an sdist that shells out to cmake. Nothing passes --only-binary, so the compiler-free property was an assumption rather than a contract, and Intel is where it broke. Marked so it installs everywhere except Intel macOS. Apple Silicon is unaffected. * Stop the optional dep gate from aborting the install _smart_apt_install exits rather than returns, and `|| true` does not catch an exit, so a box missing cmake or git aborted at the gate added to let it continue. Verified in sh, dash and bash. Run it in a subshell and re-raise only code 2, the NEED_SUDO handshake install.rs answers with an elevation prompt. install.sh treats a present-but-broken git as missing, but the Python side tested only shutil.which, so it promised to skip the git+https triton requirement and then fetched it anyway. Same check on both sides now. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Never elevate for optional build tools Re-raising code 2 turned the optional set into a NEED_SUDO handshake, so a box missing cmake or gcc got the desktop's mandatory permission dialog, whose Cancel drops back to not-installed. That re-imposes through a prompt the build-tool requirement this gate removes, and none of those tools are needed to run. Suppress the handshake for optional callers; a required package still elevates. Verified in sh, dash and bash. Also advance the progress bar on the no-git triton skip, which otherwise ends at 14/15. * Tighten the comments on the dependency gate * Correct why the PyAV cap is needed 16.0.0 does ship cp313-cp313-macosx_14_0_arm64; the comment claimed no cp313 wheel exists. The actual reason is the deployment target: 15.1.0 is macosx_13_0 and 16+ is macosx_14_0, so the cap is what keeps macOS 13 off a source build. * Tighten the installer gate comments * Cap cryptography on x86_64 macOS so the consumer install needs no Rust cryptography 49.0.0 (2026-06-12) dropped the macosx_10_9_universal2 wheel and now ships macosx_11_0_arm64 only, so x86_64 macOS has no wheel and uv falls back to the sdist. That build calls maturin, which pulls Rust and then fails at 'linking with cc failed' on a clean Mac without the Xcode Command Line Tools. It surfaced in the clean-machine leg mac macos-15-intel / mask / file, several minutes into the studio dependency step, which is exactly the up-front toolchain requirement this branch removes. 48.0.1 is the newest release carrying a universal2 wheel, and its cp39-abi3 / cp311-abi3 tags cover the 3.12 and 3.13 interpreters the installer creates. The cap is marker-scoped to darwin + x86_64, so arm64 macOS and every other platform still resolve to the latest. Lift it when cryptography ships an x86_64-capable macOS wheel again. Resolution of studio/backend/requirements/studio.txt under this constraints file gives 48.0.1 on x86_64-apple-darwin and 49.0.0 on aarch64-apple-darwin and x86_64-unknown-linux-gnu, on both 3.12 and 3.13. * Correct the av note now that cryptography also compiles on macOS * Never escalate for optional apt packages outside Tauri mode The optional bypass sat inside the TAURI_MODE branch, so a plain curl | sh install on a non-root Debian or Ubuntu box still fell through to the escalation branch and showed the default-yes permission prompt for cmake, GCC and the libcurl headers. That is exactly the toolchain this change set declared unnecessary on the consumer path, so the prompt asked for a password to install packages nothing here uses, and a headless run failed the same way instead of falling through to prebuilt llama.cpp. Move the check above the mode split so optional callers return 2 in both modes. Required packages such as curl still escalate unchanged. --------- Co-authored-by: danielhanchen <unslothai@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-28 18:50:38 -07:00
Leo Borcherding	411cb86d62	amd: require bitsandbytes>=0.50.0 in the amd extra (fixes ROCm 4-bit NaNs) (#7535) ... * amd: require bitsandbytes>=0.50.0 in the amd extra bnb <= 0.49.2 NaNs at decode shape on every AMD GPU. The ROCm 4-bit GEMV fix (bnb PR #1887) first ships in 0.50.0, on PyPI since 2026-07-24, so the old >=0.49.1 floor could still resolve the broken range. Mirrors the same change made on the pip release branch in #7278. * amd: cite the 0.50.0 ROCm work accurately in the bnb floor comment The comment credited bnb PR #1887 as "the ROCm 4-bit GEMV fix" for every AMD GPU. #1887 decouples blocksize from warp size and fixes a hardcoded warp size of 32 in kgemm_4bit_inference_naive, which is a CDNA problem by construction. The RDNA-side work is #1979 (fused 4-bit SIMT GEMM) and #2012 (RDNA3/4 workgroup resonance). All three first ship in 0.50.0, so the >=0.50.0 floor is unchanged; only the justification was wrong. * amd: raise the installer bitsandbytes fallback floors to 0.50.0 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * amd: stop reporting the bitsandbytes PyPI fallback as broken * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten AMD bnb floor comments * Keep the amd extra citation and the AMD install guide reference * amd: do not promise aarch64 a ROCm 4-bit backend it never gets * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * amd: fall back to the PyPI bitsandbytes floor on Windows ROCm too * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-28 18:12:26 -07:00
Lee Jackson	d7594ec10f	Fix Windows no-torch setup (#7511) ... * Fix Windows no-torch setup * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix no-torch env normalization on Windows * Accept on for Windows no-torch mode * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep no-torch mode across studio update on Windows Guarding the direct torch/Triton install made `install.ps1 --no-torch` actually produce a torch-free venv, which then broke the next `unsloth studio update`. That path exports no UNSLOTH_NO_TORCH, so $NoTorchMode was false, the stale-venv check read the missing torch as a broken venv, and setup tried to delete the venv it was running out of: [ERROR] Could not remove stale venv: Access to the path 'python.exe' is denied. That teardown can never succeed there, because setup.ps1 runs via unsloth.exe out of that same venv. The same gap also let the shared dependency pass reinstall torch from PyPI, unpinned, into a GGUF-only environment. install_python_stack.py now records the mode in the install manifest and setup.ps1 reads it back when no env var is exported, then re-exports a canonical value for the dependency pass (setup.ps1 drops the manifest before invoking it, so the child cannot repeat the lookup). The key is additive and MANIFEST_SCHEMA is unchanged, so existing manifests stay valid and a missing key keeps today's behaviour. Also: - read_manifest() caught only OSError, but UnicodeDecodeError is a ValueError. That is now on the installer's import path, so a manifest re-saved as ANSI or truncated mid-write would abort every install. - The env predicate now trims surrounding whitespace, matching the Python side. - The Windows update smoke workflow asserts the update leaves the venv GGUF-only, which is what would have caught this. Known follow-up, pre-existing: an install killed between the manifest drop and the dependency pass leaves no recorded mode, so a later update still walks the stale-venv path. Closing that needs a marker the installer never drops. * Persist no-torch mode in a marker the dependency pass cannot drop The install manifest alone was not enough. Both setup.ps1 and install_python_stack.py remove it before every dependency pass, and it is only rewritten on success, so a no-torch install interrupted in between left nothing recording the mode. The next update then resolved no-torch as false, read the expected missing torch as a stale venv, and tried to delete the environment whose python.exe was running it, which leaves the install unrepairable from the CLI. Add .unsloth-no-torch next to the existing .unsloth-studio-owned marker, written before the pass and cleared when torch is wanted. setup.ps1 writes it as soon as the mode resolves, so the window between the manifest drop and its own torch install is covered too. Read order stays manifest key first, then marker, so migrating out of no-torch is never blocked by a marker an earlier run left behind. Neither present still reads as "install torch", so nothing changes for installs made before either existed. Also adds the AGPL-3.0 header the new test file was missing. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothai@gmail.com>	2026-07-28 05:54:25 -07:00
Daniel Han	1781770bee	Studio: detect an interrupted dependency install instead of launching a backend that cannot import (#7492) ... * Studio: detect an interrupted dependency install instead of launching a backend that cannot import An installer killed part-way leaves a venv with a working CLI but without studio.txt's dependencies. Nothing recorded that, so three separate places all reported it healthy: - the desktop preflight probed only `unsloth -h` (typer + rich) and a hardcoded desktop-capabilities dict, neither of which touches studio.backend, so it returned ManagedReady and spawned a backend that died on `import structlog`; - setup.sh's fast path compared the installed unsloth version against PyPI, which matches on a half-built venv because unsloth is installed early, so `unsloth studio update` printed "up to date" and repaired nothing; - start_managed_repair calls that update and then re-checks with the same blind probes, so Repair reported success without fixing anything. install_python_stack.py now clears a completion manifest before the dependency pass and writes it only after the final step. `unsloth studio verify-install` and desktop-capabilities' new studio_install_ok field read it, the preflight turns a false answer into ManagedStale so auto-repair runs, and setup.sh / setup.ps1 gain an escape hatch next to the existing anyio one. Separately, the wheel ships studio/ and studio.backend* but declared none of their dependencies, so `unsloth train`, `export`, `chat`, `inference` and `studio` all ended in a rich traceback after a plain pip install. structlog is the only hard module-level import that chain reaches once starlette's annotation-only import moves under TYPE_CHECKING, so it becomes a core dependency and the rest of the server stack becomes a [studio] extra mirroring studio.txt. The CLI import sites now report missing dependencies as a sentence with two remedies. Fixes #4701, #5260, #7147 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the trimmed comments merged on the pip branch * Put the install manifest in the preflight fingerprint for PR #7492 The capability cache keyed the venv on pyvenv.cfg, uv.lock, requirements.txt, the interpreter and site-packages/unsloth_cli/commands/studio.py, none of which a repair touches when it only reinstalls studio.txt. So an entry cached while the install was healthy stayed valid after the manifest was dropped, and the probe returned Ready on exactly the half-built venv this is meant to catch. * Address the review findings on PR #7492 Fail the install when the completion manifest cannot be written, instead of exiting 0 without the record every later check requires, which is a repair loop by construction. Compare the version of the package the manifest names, so `studio update --package X` does not read as a permanent version change. Read the manifest from the venv that owns it when the CLI runs outside the managed venv, and drop the dependency verdict in that case: the walk ran against the wrong interpreter and says nothing about that venv. Name the import that actually failed. `unsloth train` reaches torch through the same guard, and the studio extra does not carry it, so recommending that extra alone left the command failing in the same place. * Declare click, which typer stopped providing, for PR #7492 unsloth_cli/commands/start.py imports click at module scope and unsloth_cli/__init__.py imports that module, so every unsloth command needs it. typer carried click through 0.19 and dropped it in 0.27, and the declared floor is typer>=0.12.0, so a fresh resolve gets no click. On the published wheel it still arrives because huggingface_hub requires click<9,>=8.4.2, which is luck rather than a declaration. A wheel built from this branch's dependency list has neither, and every command dies at import. Verified: before, `unsloth --help` on a fresh venv raised ModuleNotFoundError for click; after, it exits 0. The drift test now covers it. * Keep a running backend from the previous app version manageable The manageability bump gated two unrelated things through one constant. For the managed CLI probe 2 is right: a CLI reporting 1 cannot answer studio_install_ok. For a RUNNING backend it is wrong, because a process already started cannot change what it reports, so bumping studio/backend/main.py in lockstep does not help one the previous app version spawned. That backend is proven ours by root id and ownership token, but lifecycle_control_block_reason returned Unmanageable, and that branch never calls adopt_verified_backend. has_owned_backend() stays false, so Repair falls into block_external_conflict, which finds the same process and refuses: the app could no longer stop a backend it owns the token for. The same regression in backend.rs turned a terminal-launched same-root server from AttachedReady into ExternalConflict. Split the constant: DESKTOP_BACKEND_MANAGEABILITY_VERSION = 1 for the two live-backend probes, DESKTOP_MANAGEABILITY_VERSION = 2 for the CLI probe. Every real gate (protocol, auth, ownership, desktop-login, MIN_DESKTOP_BACKEND_VERSION) is untouched, so an old backend still reaches OwnedStale, adopt, stop, repair. Also stop the installer when the stale manifest cannot be removed. Windows raises on a read-only or locked file, and the pass would then run behind a marker that still names this version and these digests, so a run killed part-way would verify as complete. * Answer for the managed venv, not the one the CLI happens to run in The guard matched ModuleNotFoundError.name, an import name, against missing_requirements(), which returns distribution names. So a missing PyJWT printed 'pip install jwt', and jwt, docx and fitz are each a real but unrelated PyPI project (fitz is a neuroimaging workflow tool), so following the advice installed the wrong package and left the backend just as broken. Map the import to its distribution before deciding, and never offer the import itself. install_state() verified the caller's own prefix. The wheel ships studio/, so a CLI installed outside the managed venv always finds its own copy of the helper first, and a healthy managed install reported studio_install_incomplete with a missing list copied from the wrong venv. Selecting the root is not enough: _installed_version() reads the running interpreter and req_root defaults to the caller's studio.txt, so both checks still answered for the wrong venv. Hand verify_install() that venv's own metadata, enumerated through Distribution.discover(context = ...path), which does not fall back to sys.path. The candidate order is untouched, so shadowed-tree detection is unchanged. setup.ps1 replaces pip, torch and triton before install_python_stack.py runs, so the manifest it drops is not dropped before the first mutation. A run killed in between kept a marker that still verifies while torch was half-replaced; drop it at the top of the dependency pass instead. setup.sh is unaffected, the stack is the first thing its pass runs, and a test now pins both. pip uninstall rewrites nothing that was fingerprinted, and cache_matches re-reads the cached studio_install_ok rather than re-checking, so a venv that lost a studio.txt package kept being served the healthy verdict. Fold a sorted hash of the installed dist-info names into the marker hash. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * A missing manifest helper is a torn install, not an old one studio/install_manifest.py ships in the same wheel as _studio_deps.py, so nothing legitimately has one without the other: a CLI predating both never reaches this code, and the desktop already calls such a CLI stale on desktop_manageability_version. Returning ok=true there reported a healthy install for a tree the package update had half replaced, and the preflight then launched a backend whose own run.py could be just as absent. Report it incomplete so repair runs. * Tighten comments across the install-detection changes * Validate Studio dependency readiness --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>	2026-07-28 10:57:20 +02:00
Leo Borcherding	f03e669442	AMD: enable ROCm torch on gfx906 (MI50 / Radeon VII) on Linux (#7354) ... * Add community-maintained legacy support path for gfx906 (MI50 / Radeon VII) rocm6.4+/7.x torch wheels bundle ROCm libraries whose Tensile kernels dropped gfx906 (rocBLAS 'TensileLibrary.dat ... not read for gfx906', ROCm/TheRock#1844), so on MI50/Vega 20 hosts with newer ROCm the installer picked wheels that fail at the first BLAS call. The rocm6.3 index is the last one whose wheels run on gfx906 (torch 2.7.0 verified on MI50 32GB, up to 2.9 in community use). Dynamo/Inductor codegen is also broken on this arch, crashing compiled graphs that train fine in eager mode. - install.sh: when the runtime GPU is gfx906 and the picked index is newer than rocm6.3, reroute torch to the rocm6.3 index and reset the constraint trio to the default <2.11 window (a rocm7.2 pick raises the floor to 2.11, which rocm6.3 cannot satisfy), with a legacy-path warning. - install_python_stack.py: mirror the reroute in _ensure_rocm_torch using the _default pkg specs, including repairing an existing +rocm7.x torch and leaving a working rocm6.3 install alone. - device_type.py: default TORCHDYNAMO_DISABLE / TORCH_COMPILE_DISABLE / UNSLOTH_COMPILE_DISABLE on gfx906 (setdefault, user override wins). Windows allowlists are untouched: repo.amd.com publishes no gfx906 wheel family (verified in the RDNA2 enablement PR). 16-bit LoRA and full finetuning work out of the box; 4-bit QLoRA needs a source-built bitsandbytes for gfx906. Based on the verified MI50 32GB setup in namnguyen0503/mi50-gfx906-unsloth-bnb4bit-lab. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * gfx906: second Codex pass (bnb skip under pin, override beats Strix) - Compute the gfx906 runtime-target flag independently of any torch-index pin or Strix override, so the bitsandbytes skip still applies when a user pins the ROCm index and sets UNSLOTH_ROCM_GFX_ARCH=gfx906 (the pin suppresses the torch reroute, not the bnb skip). Probe only when no pin is set (an explicit pin means don't second-guess it, matching the Strix path's asserted no-probe invariant); an explicit gfx906 override needs no probe. - Let UNSLOTH_ROCM_GFX_ARCH=gfx906 suppress the Strix reroute (both install.sh and install_python_stack.py) so a mixed Strix + MI50 host routes to rocm6.3 instead of the gfx1151 wheels probe order would pick. - Fix test_hardcoded_torch_constraint: the default <2.11 window literal now legitimately appears on two TORCH_CONSTRAINT= assignments (default + the gfx906 reroute reset after the rocm7.2 floor bump); assert it only ever appears on assignment lines, never on a pip install line (its real intent). New tests: bnb skipped under an explicit pin, gfx906 override wins over Strix, install.sh suppresses Strix on the override. rocm_support + selection + cross-platform parity: 667 passed; structural constraint 9/9. * gfx906: collapse single-line asserts to match pre-commit formatting * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * gfx906: keep bnb skip + rocm6.3 routing correct under pins and suffixed overrides Address the four Codex P2 findings on #7354: - bnb skip under a pinned index (install.sh + install_python_stack.py): a real gfx906 host that pins UNSLOTH_TORCH_INDEX_URL to rocm6.3 without also setting UNSLOTH_ROCM_GFX_ARCH no longer reinstalls the generic bitsandbytes wheel over a source-built gfx906 bnb. A pin now suppresses only the torch reroute, not the gfx906 detection used for the bnb skip (Python drops the pin gate on _runtime_is_gfx906; bash _is_gfx906_bnb_skip probes via _probe_amd_gfx_arch when the index is pinned). - clear the Radeon marketing-name flag for every gfx906 target, not only when the >=6.4 reroute fires, so a Radeon VII already on rocm6.3 does not divert to the repo.radeon.com branch (whose wheels lack gfx906 kernels). - normalize a copied HIP gcnArchName (gfx906:sramecc-:xnack- -> gfx906) before the exact comparisons in install.sh and install_python_stack.py, mirroring device_type.py. Tests: relax the three Strix-pin tests (the gfx probe may now run for the bnb flag but must not reroute the pinned index) and add coverage for the pinned bnb skip, the suffixed override, and the bash Radeon-clear / pinned-probe paths. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * gfx906: log skipped vLLM aimv2 fix + robust source-scan test bounds Follow-up review polish: - import_fixes: log at info level when the vLLM aimv2 fix is skipped because the dist metadata is unreadable, so the skip is diagnosable instead of silent. - test_rocm_support: bound the gfx906 install.sh source-scan on the ';;' that closes its case arm via a shared _gfx906_reroute_block helper, replacing the brittle fixed-length (3200/3800) slices that shift when the block grows. * gfx906: trim whitespace on UNSLOTH_ROCM_GFX_ARCH in install.sh (py parity) The bash gfx906 comparisons lowercased and stripped the gfx906:… feature suffix but not surrounding whitespace, while the Python paths do .strip(). A stray newline (e.g. export UNSLOTH_ROCM_GFX_ARCH=$(cmd)) would make bash miss gfx906 while Python catches it. Trim with `tr -d '[:space:]'` at both comparison sites so the reroute target and bnb-skip agree across bash/Python. * gfx906: remove generic bitsandbytes pulled in transitively after the skip --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothai@gmail.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-07-27 05:22:19 -07:00
Daniel Han	1daaa5cbb4	Let a decode failure degrade instead of escaping a fail-closed helper (#7487) ... * Let a decode failure degrade instead of escaping a fail-closed helper Pinning utf-8 makes a read that used to return mojibake on Windows raise instead. 33 of those reads sit under a handler catching OSError or json.JSONDecodeError but not UnicodeDecodeError, which subclasses ValueError, so a corrupt file would now escape a helper written to return a default. Adds UnicodeDecodeError to those tuples only. * Treat an undecodable install lock as stale instead of retrying forever	2026-07-27 03:26:08 -07:00
Daniel Han	3fd948eb95	Pin utf-8 on shipping-code text I/O instead of the operator locale (#7486) ... * Pin utf-8 on shipping-code text I/O instead of the operator locale 113 read_text/write_text/open call sites across unsloth, studio and unsloth_cli let locale.getencoding() decide the encoding. That is utf-8 on the Linux and macOS runners and cp1252 on a stock Windows install, so the same file decodes differently for a Windows user and silently produces mojibake or raises UnicodeDecodeError. Adds tests/test_runtime_text_encoding.py to keep it that way. It resolves openers through each file's own imports rather than a fixed list of module names, so an aliased tarfile.open or a local from PIL.Image import open is not asked for an encoding it does not take. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scan tracked files only and resolve the unbound Path calling forms * Honour PEP 263 when scanning sources and migrate a legacy JSONL before appending * Scope guard imports lexically and only migrate a legacy file when it round-trips * Leave a legacy JSONL untouched and resolve path aliases in the foreign-opener check * Tighten comments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-27 02:14:20 -07:00
Leo Borcherding	3ea6d14c39	AMD: CI coverage for recent fixes, plus three wrong gfx ids (#7431) ... * ROCm/AMD CI coverage: arch-table parity, native-Linux lib prepend, RDNA4 grouped_mm, discovery-based shell suite Three merged ROCm fixes shipped without tests, and the CI wiring that would have run them was gated on files the fixes do not touch. Tests added (113): tests/studio/install/test_rocm_arch_table_parity.py (27) diffs the four duplicated gfx -> AMD pip-index tables across install.sh, install.ps1, studio/setup.ps1 and install_python_stack.py, plus the GPU-name -> arch tables and the torch 2.11 pin allowlist. tests/studio/install/test_rocm_native_linux_lib_dirs.py (26) covers #7233: system-ROCm lib dirs prepended ahead of bundled libggml-hip, the /dev/kfd + not-WSL + libhsa gate, the opt-out env var, root resolution order, and source parity between the two copies. studio/backend/tests/test_grouped_mm_rdna4_fallback.py (46) covers #7292: registration on the CUDA dispatch key, grouped and ungrouped numerics, bias/dtype promotion, and the Linux HIP<7.13 + RDNA4 name gate, executed from the shipped source rather than a copy. tests/studio/test_ci_shell_suite_coverage.py (14) fails if either shell runner goes back to a hardcoded list or skips a file without a recorded reason. CI wiring: studio-backend-ci.yml: add install.sh / install.ps1 to the path filter (the suites it runs assert against those two files, so install-only changes -- the shape most AMD/ROCm routing fixes take -- skipped it), and replace the 13-file hardcoded shell list with directory discovery. That list had fallen seven files behind, including test_strixhalo_wsl_reroute.sh, the only shell coverage of the ROCm WSL reroute, which had never run on a PR. tests/run_all.sh: same discovery loop so local and CI agree. * Test review fixes: assert on outcomes, not on the code under test Self-review of the previous commit found four tests that passed for the wrong reason. 1. The arch-table parity test pinned expected gfx ids copied out of the shipped tables, which enshrined three upstream inaccuracies as correct: RX 9070 (non-XT) is gfx1201 not gfx1200, RX 7800 XT is gfx1101 not gfx1100, and PRO V710 is gfx1101 not gfx1102 per AMD's ROCm compatibility matrix. The expectation is now the AMD pip index leaf -- the thing the tables exist to produce, and what a wrong answer costs the user. The three known drifts are listed explicitly with a test asserting they stay cosmetic, i.e. that the wrong and right ids still map to the same wheel index. That test turns red the day one of them starts routing users to the wrong wheel. 2. The RDNA4 device-name test extracted the regex from worker.py and then matched with it, so it could not fail. Widening the pattern -- the dangerous edit, since it forces the slow Python mm fallback onto RDNA3 users -- would have been silently accepted. It now reads the live pattern and checks it against fixed cases, plus asserts the name match stays guarded by `not _lin_arch` and that the name is lowercased before matching. 3. The CI-coverage test matched a verbatim line of studio-backend-ci.yml, so reindenting the step would fail the build while a real regression to a hardcoded list could slip past a reformat. It now parses the YAML, finds the step by name, and asserts on the glob plus the absence of individual filenames. The path-filter test likewise reads the parsed trigger instead of scanning raw text. 4. A set comprehension in the parity helper had a ternary whose branches were identical. Mutation-tested: widening the RDNA4 regex, desyncing one copy of the name table, dropping install.sh from the path filter, and re-skipping the ROCm WSL shell suite each fail at least two tests. Verified on Linux (WSL Ubuntu 24.04) with CI's torch pin: 86 + 48 pass. * Fix three wrong gfx ids in the GPU-name arch tables The name -> gfx tables disagreed with AMD's ROCm compatibility matrix on three entries. Corrected against the "Radeon GPU" list at rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html: RX 9070, RX 9070 GRE gfx1200 -> gfx1201 (Navi 48, same die as the XT) RX 7800 XT, RX 7700 XT gfx1100 -> gfx1101 (Navi 32, not Navi 31) PRO W7700 gfx1100 -> gfx1101 PRO V710 gfx1102 -> gfx1101 (Navi 32, not Navi 33) No wheel changes for anyone: gfx1200/gfx1201 both resolve to gfx120X-all and gfx1100/gfx1101/gfx1102 all resolve to gfx110X-all, in all four copies of the index-family map. That collapse is why the errors survived being copied into six places -- the leaf-level tests could not see them. It was not purely cosmetic, though. install.sh's second copy feeds "Tip: set UNSLOTH_ROCM_GFX_ARCH=<arch>", so a 7800 XT user following the printed advice exported gfx1100 and made a wrong id authoritative for every later run. It would also have become a real misroute the moment AMD split a family across index leaves, as they already do for gfx1151/gfx1150. Fixed in all six places, which is two more than the table's own "kept in sync with" comments claim exist: install.sh _infer_amd_gfx_arch_from_gpu_name install.sh case "$_gpu_disp_mkt" (banner + env tip; undocumented) studio/setup.sh install.ps1 studio/setup.ps1 studio/install_python_stack.py Ordering is preserved: the gfx1102 arm still precedes gfx1101 in the shell copies so "RX 7700S" cannot fall onto the "RX 7700" glob, and the PowerShell copies keep the (?!S) lookahead. Test changes: - test_rocm_arch_table_parity.py gains _AMD_DOCUMENTED_ARCH, exact gfx ids transcribed from AMD rather than from the tables. Agreement between six copies proves nothing when all six were transcribed from the same mistake, so the ground truth has to come from outside. Verified it catches the bug: against the pre-fix tables it fails 6 tests. - The parity check now covers all six copies. It had four; the two install.sh copies were being treated as one, and _WIN_GPU_NAME_ARCH_TABLE was not checked at all. - test_rocm_support.py's TestGfxArchNameFallback pinned two of the wrong ids as expected values; updated, and extended with a 9060 XT and a 7900 XTX case so each RDNA3/4 die is represented. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard against unregistered copies of the GPU-name arch table Counting the copies by hand is what let them drift: the in-code "kept in sync with" comments claimed four, the arch-id fix found six, and scanning the tree turns up a seventh. TestNoUnregisteredArchTable rediscovers the copies from the source tree instead of trusting a hand-maintained list. A table line is one that names a card and gives its arch; real tables score 9-17 such lines and the only other hits in the repo are two single-line prose comments, so the three-line threshold is not load-bearing. A companion test asserts the scan still finds the known copies, so the heuristic cannot go blind and pass by finding nothing. The seventh copy is tests/_zoo_rocm_spoof.py, the fixture other ROCm tests build their fake AMD host from. It states the mapping backwards (gfx -> the name torch should report), which makes it an independent witness: it had gfx1101 -> RX 7800 XT and gfx1201 -> RX 9070 XT right while all six installer copies were wrong, and nothing compared the two. Now they are round-tripped against each other. RX 6700 XT is pinned as a known divergence rather than normalised. AMD's compatibility matrix documents no consumer RX 6000 card and no gfx1031 at all, the installer arm is commented "gfx103X family", and gfx1031 appears only as an index-family key, never as a value a name table emits. With no external source to correct against, changing shipped behaviour would be guesswork. A test fails if the divergence ever disappears, so the exemption cannot go stale. Also adds the reverse of the AMD-matrix check: a documented card that matches no arm anywhere is a silent CPU fallback rather than a wrong id. This cannot detect hardware nobody transcribed, which would need a live fetch of AMD's matrix and a non-hermetic suite; the docstring says so rather than implying coverage that is not there. Verified on Linux: 478 passed, plus all five new guards mutation-tested to confirm each fails when its invariant is broken. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Docstring said six copies; the list under it now has seven * tests: run discovered shell tests with bash, not sh tests/run_all.sh discovered tests/sh/ instead of listing files, but still invoked each one with sh. Every file there declares a bash shebang, and on Debian/Ubuntu /bin/sh is dash: test_apt_distro_prompt.sh, test_studio_home_node_dir.sh and test_with_llama_cpp_dir_link_behavior.sh fail on bashisms under dash and pass under bash. The old hand-written list happened to name only dash-clean files, so switching to discovery is what surfaced it. Backend CI already used bash, so this was a local-only break. Guarded by a new test asserting both runners invoke tests/sh/ with bash. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix Krackan Point (Radeon 860M/840M) routed to the gfx1150 wheel index The GPU-name tables map 860M/840M and the Ryzen AI 7 350 / AI 5 340 CPU strings to gfx1150, but Krackan Point is gfx1152. AMD's own lemonade table (src/cpp/server/system_info.cpp) maps both Krackan iGPUs to gfx1152. Unlike the three ids already fixed here, this one is not wheel-neutral: repo.amd.com publishes gfx1150 and gfx1152 as separate index leaves with separately built torch wheels, so these laptops were installing wheels built for a different LLVM target. gfx1152 was absent from the codebase entirely, so it needed the index-family maps, the torch 2.11 floor lists (same _grouped_mm bug as gfx1150/1151), the Strix reroute set and the Windows arch allowlist as well as the seven name tables. The parity test added in this PR did not catch it because its AMD-matrix expectations stopped at 890M/880M. Added the APU rows, so the case that actually changes a wheel is now covered: reverting the tables fails 9 tests naming 860M, 840M and Krackan. gfx1153 (Ryzen AI 5 430 era) is left alone; AMD publishes no gfx1153 wheel family, so there is nothing to route it to. Verified: bash -n on both shell installers, PowerShell AST parse on both .ps1 files, python ast.parse on all touched modules, install suite 1334 passed with no new failures against main, shell suite 20 files. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gfx1152 to unified-memory classifiers, make parity allowlist set-based Krackan Point (gfx1152, Radeon 860M/840M) is the third RDNA 3.5 APU and shares one GPU/system-RAM pool exactly like Strix Point (gfx1150) and Strix Halo (gfx1151), but only the installers knew about it. The two runtime classifiers still had two-element arch sets, so a Krackan laptop got the 0.90 discrete headroom factor on a shared pool and ran llama.cpp without GGML_CUDA_ENABLE_UNIFIED_MEMORY. - worker.py _rocm_classify_unified_memory: add gfx1152 to the arch set, and 860m/840m to the device-name fallback. The NVIDIA GeForce 840M cannot collide there: the function is only reached under _hw.IS_ROCM. - llama_cpp.py _amd_apu_wants_unified_memory: add gfx1152 to the arch set. - Tests for both, including the :sramecc-:xnack- suffix form. TestGfx211AllowlistParity compared four hardcoded allowlist strings, so adding gfx1152 to all four installers correctly turned three assertions red without any installer actually disagreeing with another. Each test now extracts the set its installer holds and compares it to one EXPECTED constant. Order and spacing are free, membership is not, and the next leaf is a one-line edit instead of four. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-07-25 18:58:02 -05:00
Souravrajvi0	978ae4745b	fix(install): infer Strix gfx when ROCm runtime is absent (#7305) ... * fix(install): infer Strix gfx when ROCm runtime is absent When /dev/kfd and rocminfo are missing on Linux (e.g. Arch/CachyOS Strix Halo), route to AMD per-arch wheels via cpuinfo/lspci inference instead of CPU-only PyTorch. Mirrors install.ps1 Windows behavior and fixes studio update via install_python_stack.py (unslothai#7301). * Map Radeon 8065S to gfx1151 in the Linux gfx inference (Codex P2) install.sh _infer_amd_gfx_arch_from_gpu_name missed 8065S, so a Strix Halo host that only exposes 'AMD Radeon 8065S' via lspci (no Ryzen AI Max branding in /proc/cpuinfo) was left on CPU torch. setup.sh and setup.ps1 already list 8065S -> gfx1151. Added it, and widened the cpuinfo regexes (install.sh and install_python_stack.py) from Radeon 80[0-9]0S to 80[0-9][05]S to match the 80X5S naming, consistent with the display-side check already in install.sh. Tests cover the 8065S name and the cpuinfo-only case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gate the Linux gfx inference out of WSL without the ROCDXG runtime for PR #7305 On WSL /proc/cpuinfo and lspci still see the host APU, so a standalone 'unsloth studio update' could infer gfx1151 and install per-arch ROCm wheels into a WSL env whose ROCDXG bridge (librocdxg) was never bootstrapped, i.e. one that cannot expose the GPU. Skip the cpuinfo/lspci inference on WSL unless librocdxg is present; an explicit UNSLOTH_ROCM_GFX_ARCH override still wins. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address Codex review on PR #7305 (WSL runtime gate, Linux mirror, arch guard) - install.sh _infer_linux_amd_gfx_arch: skip the cpuinfo/lspci inference on WSL unless librocdxg is present (the ROCDXG bridge), mirroring the Python fix, so a WSL box whose ROCm bootstrap was skipped keeps the CPU fallback instead of installing AMD wheels that cannot reach the GPU. The explicit UNSLOTH_ROCM_GFX_ARCH override still returns first, so it stays authoritative. - install.sh: guard the inferred-gfx reroute on x86_64|amd64. ROCm torch wheels are not published for arm64, so an inferred/overridden gfx no longer pushes an arm64 host to the AMD arch index (get_torch_index_url returns CPU there). - install_python_stack.py _amd_arch_index_url: honour UNSLOTH_AMD_ROCM_MIRROR on Linux (the same var install.sh uses) instead of the Windows mirror var, so a mirrored/air-gapped Linux 'unsloth studio update' reaches the index install.sh chose. Windows still delegates unchanged; both default to repo.amd.com. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scan all AMD display controllers in the lspci fallback for PR #7305 (Codex P2) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio): keep inferred AMD wheels from being overwritten After a successful inferred-gfx install, skip the generic pytorch.org ROCm reinstall so readable ROCm userland without /dev/kfd cannot undo the per-arch repair (Codex P1 on #7305). Also merge latest main. * Only take the inferred-gfx install when the runtime sees no GPU for PR #7305 (Codex P1) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Isolate three updater tests from the host cpuinfo for PR #7305 (Strix dev box leak) * Gate the reroute on invisible ROCm and forward the inferred gfx to setup.sh for PR #7305 (Codex P2s) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Require AMD PCI display evidence for cpuinfo inference; honor gfx override with visible ROCm for PR #7305 (Codex P2s) --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: LeoBorcherding <borchborchmail@gmail.com>	2026-07-22 20:16:45 -05:00
Souravrajvi0	84b762228c	fix(install): route Strix to AMD gfx index on ROCm 7.14 (#7300) ... * fix(install): route Strix to AMD gfx index on ROCm 7.14 When ROCm 7.3+ caps to the generic pytorch.org rocm7.2 index (or the Radeon repo is unavailable), gfx1150/gfx1151 hosts were left on torch 2.11+rocm7.2 instead of AMD's arch-specific wheels. Broaden the Strix reroute in install.sh and studio/install_python_stack.py so `studio update` repairs the same path as fresh installs (unslothai#7280). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-07-22 05:03:51 -07:00
Daniel Han	2c492c8d9b	Recognize Radeon 8065S (Gorgon Halo / Ryzen AI Max 400) as gfx1151 (#7290) ... * Recognize Radeon 8065S (Gorgon Halo / Ryzen AI Max 400) as gfx1151 * Classify Radeon 8065S (Gorgon Halo) as unified memory in ROCm OOM guard * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-21 18:07:41 -07:00
Daniel Han	35f887d795	Installer: enable ROCm torch on RDNA2 (gfx1030-1036) on Windows (#7277) ... * Installer: enable ROCm torch on RDNA2 (gfx1030-1036) on Windows repo.amd.com publishes a gfx103X-all wheel family with win_amd64 torch 2.9.1/2.10.0/2.11.0+rocm7.13.0 (cp310-313), but both Windows allowlists omitted RDNA2, so RX 6000 cards (gfx1030/1032, etc.) fell back to CPU-only torch. Map gfx1030-1036 to gfx103X-all in install.ps1 ($archFamilyMap) and install_python_stack.py (_GFX_TO_AMD_INDEX_ARCH). No torch floor (mirrors gfx110X-all: newest wheel, no _grouped_mm bug on RDNA2). NVIDIA/Mac/CPU and Linux paths untouched; gfx906 stays CPU (no wheels published). * Sync studio/setup.ps1 RDNA2 (gfx1030-1036) allowlists for PR #7277	2026-07-21 03:54:25 -07:00
Daniel Han	3ab8dce97a	install: let UNSLOTH_TORCH_INDEX_FAMILY / _URL override CUDA wheel detection (#6692) ... * install: let UNSLOTH_TORCH_INDEX_FAMILY / _URL override CUDA wheel detection get_torch_index_url (and the studio-update mirror _detect_cuda_torch_index_url) chose the torch wheel family solely by probing the host GPU, with no override. In a headless / container / CI build the host driver is visible via the /proc/driver/nvidia/gpus fallback but nvidia-smi cannot report a CUDA version, so the function fell back to its cu126 default and installed the wrong wheels (e.g. a cu128 image got cu126 torch). Add an explicit override checked before any probing, in both the shell installer and the Python studio-update path: - UNSLOTH_TORCH_INDEX_URL full index URL, used verbatim (wins) - UNSLOTH_TORCH_INDEX_FAMILY family (cpu, cu128, rocm6.4, ...) appended to the mirror base (UNSLOTH_PYTORCH_MIRROR still honoured) This matches how the published GPU images select CUDA -- vLLM and SGLang take the CUDA version from an explicit build ARG rather than detecting it, and the Unsloth Docker base image already pins the cu128 index directly. Desktop installs are unchanged: with no override set, detection runs exactly as before. Adds test_get_torch_index_url.sh cases for the override (family, full URL, precedence, mirror base, trailing-slash strip, empty-ignored). * install: make the torch-index override authoritative across ROCm paths Address review feedback on the override added in this PR so a pinned index is honoured everywhere, not just in get_torch_index_url: - Skip the WSL ROCm bootstrap (root privilege + large downloads, probes /dev/dxg) when UNSLOTH_TORCH_INDEX_URL / _FAMILY is set; it previously ran before the override was consulted. - Skip the Radeon/Strix rerouting (which re-probes the GPU and overwrites the resolved URL with repo.radeon.com / repo.amd.com) when the index is pinned, so an explicit ROCm override (e.g. UNSLOTH_TORCH_INDEX_FAMILY=rocm6.4) is kept. - install_python_stack.py: derive _TORCH_BACKEND from the override when UNSLOTH_TORCH_BACKEND is unset (standalone studio update), so _ensure_rocm_torch / _ensure_cuda_torch repair to the requested family instead of re-detecting. - Strip ALL leading/trailing slashes in the shell override to match the Python side (avoids 404s on strict pip proxies). Adds test cases for double-slash and leading/trailing-slash overrides. * install: honor pinned torch index in CUDA/ROCm repair paths Follow-up to the override work in this PR: the get_torch_index_url / install.sh reroute already respect a pinned UNSLOTH_TORCH_INDEX_URL / _FAMILY, but the Python repair helpers in install_python_stack.py still re-probed the GPU and could overwrite the pinned family. Make the pin authoritative there too: - _ensure_cuda_torch: an explicit cu* pin commits to CUDA wheels, so repair a ROCm-poisoned venv even when no NVIDIA GPU is visible here (headless / container / CI cross-install), instead of bailing on the GPU-presence gate. - _ensure_rocm_torch: skip the AMD per-gfx (Strix) reroute when a ROCm index is pinned, and in the generic reinstall path install from the pinned URL verbatim rather than re-detecting the host ROCm version. gfx*/rocm7.2 indexes serve torch 2.11+, so select the 2.11 package specs for a gfx leaf. - install.sh: raise the torch constraint to 2.11 for */gfx* indexes too, matching rocm7.2, so a pinned full-URL/family override that returns early keeps a valid constraint. Add _explicit_torch_index_url / _explicit_rocm_torch_index_url helpers and tests covering the no-GPU CUDA pin repair and the explicit gfx index honored verbatim. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: honor torch-index override on the Windows installers too The pinned-index work landed for install.sh and install_python_stack.py, but the Windows installers still picked the wheel index from GPU probing. Extend the same UNSLOTH_TORCH_INDEX_URL / _FAMILY contract so a pinned index wins on every platform: - install.ps1: Get-TorchIndexUrl returns the pinned URL/family before nvidia-smi probing; the AMD ROCm reroute is skipped when the index is pinned, so an explicit cpu/cu* pin on an AMD host is not overwritten. - studio/setup.ps1: add shared Get-PinnedTorchIndexUrl / Get-TorchIndexLeaf helpers; the stale-venv check, the install selection and the AMD reroute all honor the pin, and the CPU/CUDA install pulls from the resolved index URL. - tests: parity test that all four installers read both override vars and the two Windows installers gate the AMD reroute on the pinned flag. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: complete pinned-index handling for ROCm/Windows edge cases Follow-ups to the override work flagged in review: - install.ps1: a pinned gfx*/rocm>=7.2 index previously skipped the AMD reroute that sets the torch>=2.11 floor, so the generic install used torch>=2.4,<2.11 and could resolve the known-bad _grouped_mm wheel. Route a pinned ROCm index through the ROCm install path with the 2.11 floor + companions, and guard the companion-spec lookup so a skipped reroute block cannot null-deref. - studio/setup.ps1: the stale-venv check compared the installed flavor (cuXXX/cpu, with +rocm misread as cpu) against the raw pinned leaf (gfx1151 / rocm6.4), so a correct pinned ROCm venv was always marked stale. Classify +rocm wheels as the generic 'rocm' flavor and normalize a pinned rocm*/gfx* leaf to 'rocm' before comparing (cu* stays specific so cu126-vs-cu128 still rebuilds). - install_python_stack.py: _ensure_cuda_torch now also reinstalls from a pinned CUDA index when the venv carries a CPU wheel (headless CPU-venv-to-CUDA cross-install via 'studio update'), not only when it finds a ROCm build. - tests: parity assertions already cover all four installers honoring the override. * install: finish pinned ROCm/CUDA edge cases on Windows + repair path Follow-ups to the previous round: - studio/setup.ps1: a pinned gfx*/rocm>=7.2 index now routes through the ROCm install path with the 2.11 floor + companions (it previously fell through to the CUDA branch with bare torch/torchvision/torchaudio against the ROCm index). The CPU/CUDA fallback index is forced to the CPU wheel index when a ROCm index is active, so a failed pinned-ROCm install does not retry the ROCm mirror. - studio/setup.ps1: the stale-venv check no longer treats an unrecognized pinned URL leaf (e.g. a PEP 503 mirror ending in /simple) as a torch flavor tag, which was marking a correct venv stale; cu*/cpu/rocm/gfx leaves are still compared. - install.ps1: the post-failure CPU fallback uses an explicit CPU index instead of , which for a pinned ROCm index was the ROCm mirror itself (so the 'fallback' just retried the failing index and aborted the installer). - install_python_stack.py: _ensure_cuda_torch now also reinstalls when the venv's CUDA family differs from a pinned one (installed cu126 vs pinned cu128), not only CPU->CUDA; the probe reports the installed cuXXX tag for the comparison. * install: keep the ROCm to CPU fallback install inside the retry-helper window The pinned-ROCm CPU fallback computes an explicit CPU index, but the comment explaining why it cannot reuse $TorchIndexUrl pushed the actual Invoke-InstallCommandRetry / --force-reinstall call more than 600 chars past the "ROCm PyTorch install failed" message, so test_pr5940_followups's window check no longer saw the retry helper. Move the CPU-index computation and its comment above the failure substep so the retrying force-reinstall stays adjacent to the message. No behavior change: same explicit CPU index, same retry, same --force-reinstall. * install: address #6692 review round 5 (ROCm/CPU pin edge cases) setup.ps1: - Stale-venv check: treat an AMD/ROCm host (HasROCm or a resolved gfx arch) with no explicit pin as expecting "rocm", not "cpu", so a healthy +rocm venv is not flagged stale (which made installer-managed setup exit and direct update rebuild). - Pinned-ROCm install failure now routes into the force-reinstall CPU branch: CuTag stays the rocm/gfx leaf on failure, so the condition also checks ROCmCpuFallback; otherwise the CUDA branch installed from the CPU index without --force-reinstall and kept the partial ROCm torch. - Explicit ROCm pin compare no longer collapses gfx*/rocm* to a generic "rocm": it compares the +rocmX.Y version (and the torch 2.11 line for gfx pins) so changing the pinned family (e.g. rocm6.4 -> gfx1151) rebuilds and applies it. install_python_stack.py: - _ensure_rocm_torch: an explicit ROCm wheel-index pin now bypasses the NVIDIA-present / no-AMD-GPU / unreadable-ROCm gates (headless/container/CI cross-install), mirroring the explicit-CUDA-pin bypass in _ensure_cuda_torch. - Add _ensure_cpu_torch: an explicit CPU pin (FAMILY=cpu or /cpu URL) now has a repair path that reinstalls CPU torch over an existing CUDA/ROCm build on a standalone update (which skips install.sh's flavor enforcement). install.sh: - Pin torchvision/torchaudio companions alongside torch for the rocm7.2 / per-gfx index and the Strix reroute (those AMD indexes publish companions independently and a bare name can resolve a torch-2.12-built wheel, an ABI mismatch). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * torch-index override: classify CUDA pin by leaf; trim blank shell overrides _ensure_cuda_torch only overrode the NVIDIA-presence gate for *any* pinned index, so a non-CUDA mirror URL (or a ROCm/CPU pin) on a non-NVIDIA host with ROCm torch could force a CUDA reinstall over a working ROCm venv. Add _explicit_cuda_torch_index_url() (leaf cu*), matching the ROCm/CPU helpers, and gate on it instead. install.sh::get_torch_index_url treated a whitespace-only UNSLOTH_TORCH_INDEX_URL / _FAMILY as authoritative (yielding an invalid index), unlike the Python .strip() and PowerShell IsNullOrWhiteSpace paths; trim leading/trailing whitespace first. * install: honor pinned torch index over CVD/GPU gates and fix leaf-based ROCm classification - install_python_stack.py: an explicit cu* pin now clears the CUDA_VISIBLE_DEVICES empty/-1 hide gate as well as the NVIDIA-presence gate, so CVD=-1 UNSLOTH_TORCH_INDEX_FAMILY=cu128 studio update repairs to CUDA wheels (parity with install.sh's get_torch_index_url override, which skips all GPU probing). Unpinned CVD=-1 still skips. - install_python_stack.py: _ensure_cpu_torch installs the bounded _CPU_TORCH_PKG_SPEC instead of a bare torch/torchvision/torchaudio trio; the /cpu index now also serves torch 2.11+, which is outside the supported <2.11 range. - install.sh: the torch>=2.11 constraint case matches the index leaf (rocm7.2|gfx*) instead of the whole URL, so a mirror base path containing a gfx/rocm7.2 segment with a cu*/cpu family is not false-matched onto the 2.11 line. - setup.ps1: the stale-venv check expects rocm torch only for arches the install path maps to a repo.amd.com wheel index; an unmapped/unreadable arch installs CPU, so a correct CPU venv is no longer marked stale. - Tests for each of the above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: tighten pinned torch-index override edge cases - install.sh: trim whitespace-only UNSLOTH_TORCH_INDEX_URL/_FAMILY before the _torch_index_pinned guard, matching get_torch_index_url, so a blank override no longer skips the WSL bootstrap and Radeon/Strix reroutes while detection still picks the normal index. - install.sh / install.ps1 / setup.ps1 / install_python_stack.py: force the torch 2.11 floor only for the gfx families with the <2.11 _grouped_mm bug (gfx120X-all, gfx1151, gfx1150). A pinned override to gfx110X-all/gfx90a/gfx908 stays on the default range, matching the automatic AMD path. - install_python_stack.py _ensure_cuda_torch: treat an untagged CUDA build under a CUDA pin as a family mismatch (reinstall), and match cuXXX pins narrowly (cu + digits) so a custom/current mirror leaf no longer forces CUDA over a CPU/ROCm venv. - install_python_stack.py _ensure_rocm_torch: reinstall when an explicit ROCm pin names a different ROCm family than the already-installed ROCm torch (the ROCm analogue of the CUDA cuXXX mismatch repair). Adds tests for each case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: fix second-order edge cases in pinned torch-index ROCm/CUDA handling Parse the ROCm torch probe positionally so an empty HIP marker is kept: CPU/CUDA torch no longer reads as HIP, so the ROCm reinstall is not skipped. Emit one "<marker>|<version>" line (like the CUDA probe) for a robust parse. Limit the gfx torch 2.11 expectation to the install allowlist (gfx120X-all/gfx1151/gfx1150). A pinned gfx110X-all/gfx90a/gfx908 index stays on the default <2.11 specs, so a correct 2.10+rocm wheel is no longer judged a mismatch and force-reinstalled every update. Distinguish an AMD per-arch wheel (three-part +rocmA.B.C) from a generic pytorch.org wheel (two-part +rocmA.B): a gfx per-arch pin over a generic 2.11 wheel now reinstalls the per-arch wheel, while an already-installed per-arch wheel is not re-flagged (no reinstall loop). Mirror all of the above in setup.ps1 via new Test-RocmGfx211Leaf / Test-CudaFamilyLeaf / Get-RocmPinStaleTags helpers, reused by both the install-spec path and the stale-venv check so they cannot diverge again. Require a digit after "cu" (^cu[0-9]) in setup.ps1, install.ps1 and install.sh so a mirror leaf like /custom or /current is not branded CUDA and does not rebuild the venv every run. Add tests: CPU/CUDA probe -> has_hip_torch False; gfx110X-all pin + 2.10 wheel not stale; gfx1151 pin + generic 2.11 wheel stale; gfx1151 pin + per-arch wheel not stale; /custom and /current not CUDA; plus cross-language allowlist and cu-digit parity guards, and a PowerShell unit test for the new setup.ps1 helpers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix ROCm/gfx pin case normalization, ROCm-tag requirement, and CUDA-leaf classification Normalize torch-index leaves to lowercase before the gfx*/rocm*/cu* allowlist matches so the canonical gfx120X-all (capital X) gets the torch 2.11 floor in install.sh (leaf, flavor and repairable helpers). Require an installed +rocm local tag before a rocmX.Y or non-2.11 gfx pin is judged satisfied in setup.ps1 Get-RocmPinStaleTags and the Python _rocm_pin_family_mismatch, so an untagged CPU/CUDA wheel never leaves the pin unapplied. Classify a leaf as CUDA only via ^cu[0-9]: the Python _TORCH_BACKEND derivation now uses _is_cuda_family_leaf, and install.sh brands cuda only on cu[0-9]* (unset on an unknown /current /custom mirror leaf) so the stack probes the GPU instead of skipping ROCm repair. Add bash, Python and PowerShell tests for capital gfx120X-all floor, current/custom not-cuda, and untagged-wheel ROCm pins. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: converge torch-index pin detection via a per-venv marker Introduce a torch-index MARKER that records the exact wheel --index-url used after each successful torch install, so `unsloth studio update` / repair makes the "did the pinned index change?" decision by an EXACT string compare rather than inferring it from the wheel +rocm/+cu version tag. The tag cannot encode the AMD per-arch gfx family (two 2.11 gfx indexes both install +rocm7.13.0), so the tag heuristic missed a gfx1151 -> gfx120X-all switch and a custom-URL swap. Marker path is per-venv (.unsloth-torch-index), one line = the resolved index URL, written atomically (temp + rename). Path, format and normalization are shared across all four installers (install.sh, install_python_stack.py, setup.ps1, install.ps1). - Reapply gfx pins on a per-arch target change: the marker's exact compare reinstalls when the pinned index differs, even when both wheels share a tag. - Honor custom ROCm URL pins during repair: an explicit index whose leaf is not rocm/gfx/cu/cpu (e.g. simple, current) now reinstalls torch VERBATIM from the pin when it differs from the marker ("URL wins verbatim"). - Align the KNOWN-2.11 rocm/gfx set to exactly rocm7.2 plus the gfx allowlist gfx120x-all/gfx1151/gfx1150 in every language; stop treating an unknown newer rocm (rocm7.3, which does not exist) as the 2.11 line speculatively. Backward compatible: with no marker (old venvs, torch installed out-of-band) the existing +rocm/version-tag heuristics still decide, and a matching marker never reinstall-loops. A cu128 CUDA pin stays a CUDA pin; custom and current leaves are not CUDA. Adds marker tests (py/sh/ps) plus cross-installer parity checks. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: keep the torch-index marker additive to flavor validation Three narrow fixes in the marker-based stale-venv detection: - setup.ps1: a matching marker no longer overwrites the detected installed flavor. The marker compare is now an additional rebuild trigger, so a stale wheel (torch swapped to a +cpu build while the marker still records a cuXXX pin) is still caught by the flavor check instead of being masked as up to date. - setup.ps1: a supported AMD arch carrying CPU torch is no longer marked stale and wiped. The downstream AMD Windows ROCm override upgrades CPU torch to ROCm in place, so wiping first would delete the venv and abort with "Virtual environment not found". Only a genuinely wrong CUDA wheel still rebuilds. - install.sh: the Radeon --find-links path records its repo.radeon.com base in the marker instead of the generic pytorch.org ROCm fallback index, so a later pin to that generic family correctly reinstalls rather than comparing equal. Mirrors install.ps1/setup.ps1, which already record the real AMD index. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: honor custom pins and repair pinned venvs in place Four follow-ups to the torch-index marker work: - install_python_stack.py: _ensure_cuda_torch/_ensure_rocm_torch now bail when an explicit custom-index pin names no known torch family, so a verbatim URL override (a private/simple mirror) is not clobbered by auto-detected CUDA/ROCm wheels before _ensure_verbatim_torch_index applies it. - install_python_stack.py: the ROCm marker is additive, not a substitute -- a matching marker still runs the family/version check so a wheel swapped after the marker was written is caught. Mirrors setup.ps1. - setup.ps1: a stale venv under an explicit pin, whose torch still imports, is repaired in place (force-reinstall torch from the pin in the dependency pass) instead of wiped. The wipe path only delegates to install.ps1, so on a direct update it stranded the user at "Virtual environment not found" instead of applying the new pin. A broken venv or unpinned drift still wipes/delegates. - install.ps1: when a pinned ROCm install fails over to a CPU base, the marker now records the CPU index actually used instead of the ROCm pin, so the next managed setup does not see CPU torch under a ROCm pin and abort as stale. * setup.ps1: keep the ROCm CPU-fallback force line the pr5940 test guards 5c93ffd4 folded the pin-change force-reinstall into the ROCm CPU-fallback condition on one line, so the exact literal that test_pr5940_followups.py checks (if ($ROCmCpuFallback) { $cpuForce = @("--force-reinstall") }) no longer appeared and the test failed. Split the two conditions into separate if lines: the ROCm fallback line is restored verbatim and the pin-change force is its own line. Both still set $cpuForce to the array, so @splat passes one arg. * install: honor exact CUDA/custom index URL pins in the torch-index marker Address three Codex review findings on the torch-index marker mechanism: - install.sh: after the ROCm CPU repair reinstalls torch from the generic $TORCH_INDEX_URL, record that as the marker source. A Radeon --find-links install set _TORCH_MARKER_INDEX_URL to its repo.radeon.com base earlier, so leaving it made the marker misreport Radeon wheels and a later Radeon pin would compare equal and skip a needed reinstall. - install_python_stack.py: _ensure_cuda_torch now consults the exact-URL marker (_marker_pin_mismatch) when the installed +cuXXX tag matches the pinned leaf, so a same-leaf CUDA mirror change (official cu128 to an internal cu128 mirror) is reinstalled and re-recorded instead of skipped. - _normalize_index_url / _normalize_family_leaf (install.sh, setup.ps1, install_python_stack.py): lowercase only KNOWN wheel-family leaves (rocm/gfx/ cpu/cuXXX) so gfx120X-all still matches gfx120x-all, while a custom (unknown-family) leaf keeps its case so a verbatim URL pin like /Current does not compare equal to /current. Tests updated to assert the refined behavior. * install: fix 3 torch-index marker edge cases (CPU mirror pin, Radeon leaf, migrated venv) Addresses three review findings on the torch-index override path: 1. CPU index URL change on an already-CPU venv. _ensure_cpu_torch returned early whenever torch was already a CPU build, so a standalone update that moved the pin (official /cpu -> a private UNSLOTH_PYTORCH_MIRROR /cpu, same +cpu tag) never reinstalled. It now consults the exact-URL marker and reinstalls only when _marker_pin_mismatch reports a different index, mirroring the CUDA/ROCm same-family handling. A matching marker (or none) still leaves CPU torch untouched, so there is no reinstall loop. 2. Radeon find-links directory misclassified as a pip ROCm family. A repo.radeon.com/.../rocm-rel-7.2.1 leaf starts with "rocm" but is a find-links listing, not a pip --index-url. The old startswith(("rocm", "gfx")) test routed it into a --index-url reinstall that fails against find-links. New _is_pip_rocm_family_leaf gates on ^rocm\d / gfx (matching install.sh's rocm[0-9]* and setup.ps1's ^(rocm[0-9]|gfx)), so a Radeon URL routes to the verbatim/marker path instead. 3. Migrated venv rewriting its marker to a pin it did not install. install.sh and install.ps1 write the marker unconditionally, so a migration that preserves existing torch recorded the newly requested pin and a later update then found a matching marker and skipped the reinstall the pin needs (e.g. a per-arch gfx1151 -> gfx120X-all switch, identical +rocm tag). Both now track _TORCH_INSTALLED_THIS_RUN and write the marker only when torch was actually installed or repaired this run. Also add Get-NormalizedFamilyLeaf to the setup.ps1 helper-extraction list in test_torch_index_marker.ps1 (it was added to setup.ps1 and the shell test in an earlier round but missed here) and add two unit tests covering findings 1 and 2. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: keep pinned torch repairs on the pinned index Two fixes for explicit index pins (UNSLOTH_TORCH_INDEX_FAMILY / _URL): 1. install_python_stack.py's repair paths ran uv without clearing the inherited uv index env vars. uv resolves the default index (--index-url or --default-index) at the LOWEST priority, so a UV_INDEX or UV_EXTRA_INDEX_URL mirror in the environment won for any package it served: a cu128-pinned repair could install torch from the mirror and then record the cu128 marker it never used. Verified empirically: with UV_EXTRA_INDEX_URL=.../cu126 exported, uv pip install torch --index-url .../cu128 resolves torch 2.13.0+cu126. Strip the four uv index env vars for pinned-index commands only, mirroring the gate install.sh, install.ps1 and setup.ps1 already have; non-pinned installs keep the user's mirror. 2. install.ps1 routed any pinned leaf matching rocm* through the ROCm --default-index path, so a custom find-links leaf like rocm-rel-7.2.1 was treated as a PEP 503 ROCm index and could silently fall back to CPU torch on resolution failure. Require a digit after rocm, matching install.sh's rocm[0-9]* and install_python_stack.py's ^rocm\d. Adds parity + unit tests for both (11 new tests). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: keep pinned repairs off UV_TORCH_BACKEND and narrow setup.ps1's rocm pin match Round 2 of the pinned-index hardening: 1. _build_uv_cmd converted UV_TORCH_BACKEND into --torch-backend before the new env isolation could act, and uv's torch backend redirects torch resolution to its own per-backend index even when --index-url is given (verified: a cu128-pinned dry run with UV_TORCH_BACKEND=cpu resolves torch 2.13.0+cpu). Pinned-index commands now never receive the flag and UV_TORCH_BACKEND joins the stripped env vars, so uv cannot re-read it. 2. setup.ps1's pinned reroute had the same bare rocm* glob install.ps1 had: a custom find-links leaf like rocm-rel-7.2.1 was routed through the ROCm --index-url path instead of the verbatim unknown-pin path. Now requires a digit after rocm, matching install.ps1, install.sh and _is_pip_rocm_family_leaf. 3. The marker test's case-normalization checks used -eq, which is case-insensitive in PowerShell, making them vacuous, and the unknown-leaf expectation was written lowercased while the implementation deliberately preserves custom-leaf case. Tightened to -ceq with the case-preserving expected value. Adds unit + parity tests for 1 and 2 (5 new tests). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: extend the pinned-index guards to every remaining surface Round 3 of the pinned-index hardening, closing the same holes on the surfaces the earlier rounds missed: 1. install.sh's pinned-install env scrub now clears UV_TORCH_BACKEND (uv's torch backend redirects torch resolution to its own per-backend index even against --default-index), and both PowerShell wrappers clear it in their pinned-install scrubs, matching install_python_stack.py. 2. setup.ps1's marker stale check still classified any rocm* leaf as a PyTorch ROCm family while the install selection is digit-gated, so a custom rocm-current / rocm-rel-7.2.1 pin stale-compared as not-rocm vs rocm and force-reinstalled on every studio update. The stale check now uses the same ^rocm\d gate. 3. install_python_stack.py's pinned-command scrub also strips PIP_EXTRA_INDEX_URL for the pip fallback: pip adds the env extra index in addition to --index-url, so an inherited mirror could satisfy torch off the pin while the marker recorded the pinned URL. PIP_INDEX_URL needs no strip since the explicit --index-url flag overrides it. Parity + unit tests extended (4 new tests). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: scrub find-links and carry the pinned scrub through pip fallbacks Round 4 of the pinned-index hardening: 1. UV_FIND_LINKS joins every pinned-install scrub (install.sh, install.ps1, setup.ps1, install_python_stack.py): uv's --find-links locations can satisfy torch off the pinned index the same way an extra index does. 2. setup.ps1's Fast-Install restored the scrubbed vars in its finally BEFORE the pip fallback ran, and never touched the pip env vars at all, so a failed uv attempt fell back to python -m pip with an inherited PIP_EXTRA_INDEX_URL / PIP_FIND_LINKS able to win over the pinned --index-url. The scrub now wraps the whole function (uv attempt + pip fallback) and includes the pip vars; restore happens after both. 3. install_python_stack.py's scrub also strips PIP_FIND_LINKS for its own pip fallback, completing the PIP_EXTRA_INDEX_URL fix from round 3. Parity tests extended (2 new tests). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: digit-gate rocm leaves in marker normalization and ROCm side effects Round 5 of the pinned-index hardening (three custom-rocm-leaf edge cases): 1. _normalize_family_leaf lowercased every leaf starting with rocm, so a custom mirror leaf like rocm-Current compared equal to its lowercase form and a case-only pin change was skipped. URL paths can be case-sensitive. The rocm prefix is now digit-gated (rocm[0-9]*, matching _is_pip_rocm_family_leaf) in install.sh, setup.ps1 and install_python_stack.py, so only true family leaves (rocm7.2) are lowercased; a custom rocm-* leaf keeps its case. 2. setup.ps1 Test-MarkerPinMismatch compared normalized URLs with -ne, which is case-insensitive in PowerShell, so a case-only marker change (Simple vs simple) was treated as matching and the reinstall skipped. Now -cne. 3. install.sh gated the AMD bitsandbytes install and the "repair ROCm torch" --default-index reinstall on a bare whole-URL rocm glob, so a custom CPU/CUDA/private index whose leaf merely starts with rocm (rocm-current) was force-repaired from the wrong ROCm-only path whenever torch.version.hip was empty. Both now gate on _torch_index_is_rocm_family, computed once from the digit-gated leaf (rocm[0-9]*/gfx*). Tests: 4 new parity assertions plus 2 case-sensitivity marker checks. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: apply an explicit custom torch-index pin on the first update Round 6: an explicitly-set custom (unknown-family) UNSLOTH_TORCH_INDEX_URL was silently ignored on the first `studio update` of a venv that predates the marker feature, on both platforms, because the no-marker case was treated as "do nothing" and the version-tag heuristics cannot judge an unknown leaf. 1. install_python_stack.py _ensure_verbatim_torch_index now reinstalls verbatim when the marker is ABSENT (None), not only when it differs, and short-circuits only when the marker already records this exact pin. It then writes the marker, so every later update is a no-op. A user who did not set the override gets pin=None and is untouched, so an out-of-band torch install is never clobbered. 2. setup.ps1: for an unknown-family pin on a marker-less venv the stale-venv check now sets PinChangedForceReinstall so the torch block reinstalls in place from the pin. It deliberately does NOT set shouldRebuild, which would wipe the venv and strand a direct `studio update`. 3. setup.sh (the Linux `studio update` entry point) skipped install_python_stack.py entirely when unsloth was already current, so the marker-driven reinstall (both the verbatim custom pin and the cu/rocm flavor and family-change repair, e.g. gfx1151 to gfx120X-all) never ran. It now forces the dependency pass when a torch-index pin env var is set; the pass is idempotent and no-ops when the marker already matches. This mirrors setup.ps1's stale-venv pre-check. Tests: 3 new parity assertions. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * test: expect first-update reinstall for a no-marker custom index pin Follow-up to d671d8fb2: _ensure_verbatim_torch_index now applies an explicit unknown-family URL pin verbatim on the first update when the marker is absent (instead of no-op), so the old test_verbatim_custom_url_no_marker_is_noop assertion was stale. Rewritten as test_verbatim_custom_url_no_marker_reinstalls_once: asserts the one verbatim reinstall from the pinned URL, that the marker is written, and that a second call with the pin still set is idempotent (no reinstall loop). * install: gate the pinned update pass on the marker and record a pin baseline Round 8, two follow-ups to the round-6 first-update pin fix: 1. setup.sh forced the full dependency pass on EVERY `studio update` while a torch-index pin stayed exported, even after the marker already recorded the same pin, turning quick updates into the expensive pass every time. It now probes install_python_stack.py --torch-pin-needs-apply (which reuses the exact marker normalization) and forces the pass only when the pin is not yet applied (marker absent or different); an already-applied persistent pin keeps the fast path. A probe error fails safe toward running the pass. setup.ps1 gets the same probe in its fast path for parity. 2. A known-family full-URL pin on a venv predating the marker (e.g. an installed cu128 build and UNSLOTH_TORCH_INDEX_URL pointing at a same-family mirror) left the marker absent forever: the _ensure_* helpers deliberately do not force a multi-GB reinstall of identical-family wheels on an old venv, so nothing recorded the pin and every update re-entered the pass. _record_torch_index_pin_baseline now records the resolved pin as a baseline after the ensure sequence when the family already matches and no marker exists, so the pin is tracked (a later genuine change is detected and applied) and the update loop is broken, without the redundant reinstall. Tests: 3 new baseline unit tests, 4 new parity assertions, and the CLI probe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * setup.sh: keep the pin probe's exit 1 from killing the update under set -e The --torch-pin-needs-apply probe deliberately exits 1 for the common steady-state answer (pin already recorded, keep the fast path), but it ran as a bare command under set -euo pipefail, so the whole studio update aborted before the exit code was even captured. Absorb the status with || _PIN_NEEDS_APPLY=$? and pre-seed 0 so all three outcomes route as documented: 0 runs the pass, 1 keeps the fast path, anything else fails safe into the pass. Parity test asserts the guard. * install: strip pin credentials, disable uv config discovery, bound verbatim installs Four verified fix groups from a 12-reviewer audit of the torch-index override feature, each reproduced before fixing: 1. Credential persistence: all four marker writers stored the raw pin URL, so an authenticated pin (https://user:token@mirror/simple) persisted its credentials in .unsloth-torch-index (mode 0644 under a default POSIX umask) and install_python_stack.py printed pin URLs verbatim in repair messages. Userinfo is now stripped before persisting and in every log/substep that interpolates a pin, via lockstep helpers (_strip_index_url_credentials in install.sh / install_python_stack.py, Remove-IndexUrlCredentials in install.ps1 / setup.ps1). The three normalizers strip too, so an OLD marker that already carries credentials still compares equal to the same pin: no reinstall loop on upgrade. Query strings deliberately stay in the marker; two indexes distinguished only by query must not compare equal. 2. uv configuration discovery beat the explicit pin: with a discovered uv.toml declaring torch-backend = "cpu" or a [[index]] entry, uv 0.10.12 resolves torch 2.13.0+cpu against an explicit --index-url/.../cu126 pin; UV_NO_CONFIG=1 restores +cu126 (reproduced both ways). The pinned-install scrub in all four installers now sets UV_NO_CONFIG=1 and drops UV_CONFIG_FILE. 3. The verbatim custom-index update path installed a bare, unconstrained torch trio while fresh installs from the same unknown-leaf pin apply the supported range; _ensure_verbatim_torch_index now installs the bounded trio spec, closing the fresh-vs-update asymmetry. 4. Query-bearing pins (.../cu128?token=x) classified by raw leaf split and force-reinstalled on every update (the installed cu128 never equals cu128?token=x). Query/fragment are now stripped before leaf classification in all four implementations; the marker comparison keeps the query per (1). Rejected after verification (no change): the pin-baseline record cannot produce a wrong later decision (every pin change still mismatches and reinstalls from the new pin); the venv temp-file symlink scenarios require an attacker who already owns the environment; pathological inputs like " / cu128 / " have no realistic caller and fail loudly. Parity, stack, rocm-support, marker (sh + ps1), pin-stale, index-url and flavor suites all pass (455 python + full shell/ps1 batteries). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: harden custom-pin repair against clobber, broken torch, and pip config Four follow-ups to the pinned-index audit fixes: 1. setup.ps1 routed an unknown-leaf custom pin through the CUDA branch with a bare torch trio while install.ps1 (fresh) and the Python verbatim path bound the supported range; the pinned unknown-leaf route now applies the same torch>=2.4,<2.11.0 bound. Known cu* leaves and unpinned runs are unchanged. 2. The final torch safety pass could not repair a clobbered unknown-family pin: intermediate dependency steps can pull torch from PyPI (the pass exists for exactly that reason), but the verbatim helper short-circuited on marker==pin and no flavor tag exists to probe. The helper now keeps a per-run snapshot of the installed trio (taken after a verbatim reinstall or on the first matching-marker pass) and reinstalls from the pin when the final pass sees the trio drifted. Probe failure skips the comparison; a reinstall refreshes the snapshot, so no loop. 3. _record_torch_index_pin_baseline could freeze a known-family pin as applied on a venv whose torch is missing or broken (every family helper returns without reinstalling when its probe fails), making --torch-pin-needs-apply report done forever. The baseline now probes the installed flavor and records only on a match: a cuXXX pin requires the matching +cuXXX tag, cpu requires a cpu build, rocm/gfx requires hip; probe failure records nothing. 4. The pinned pip fallback stripped PIP_* env vars but user/site pip config files still applied (a configured global.extra-index-url can satisfy torch off the pin). PIP_CONFIG_FILE is now pointed at the null device for pinned commands (pip loads no config files then), in _install_env_for_cmd and setup.ps1's Fast-Install pinned scrub. install.sh / install.ps1 have no pip fallback (uv-only), verified. Tests: 7 new rocm_support tests (snapshot reset fixture), 1 stack test, 2 parity tests. Full battery green (464 python, sh and ps1 suites). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: complete the pin-repair coverage across the fast path and platforms Three cross-platform follow-ups to the round-2 pin-repair fixes: 1. The --torch-pin-needs-apply probe only compared marker==pin, so a torch trio clobbered to the wrong family (a cpu wheel replacing cu128 via a later pip install) with a still-matching marker reported "already applied" and the _ensure_{cuda,rocm,cpu} repair never ran on the Linux fast path. The probe is now a testable _torch_pin_needs_apply() that also checks the installed flavor against a known-family pin (via a shared _torch_flavor_matches_pin() helper, so the baseline and the probe cannot drift). An unknown-family pin has no flavor to validate and a failed probe cannot prove drift, so both keep the fast path. 2. macOS ARM (real CPU/MPS torch, not NO_TORCH) never applied an unknown- family custom pin on update: both the verbatim path and the baseline returned on IS_MACOS while fresh install.sh honors the pin, so the marker was never written and setup.sh forced the dependency pass on every update forever. The guards are now IS_MAC_INTEL (Intel mac is already NO_TORCH), and the final pass applies the pin on macOS ARM. 3. The round-2 final verbatim repair sat in the step-13 sequence guarded not IS_WINDOWS, so on Windows a dependency step that clobbered torch after the pin was applied was masked by the matching marker (setup.ps1 does not re-validate the main venv's torch after calling this script -- verified). Step 13 now runs the verbatim snapshot-drift repair on Windows and macOS ARM too; the Linux-oriented cuda/rocm/cpu family helpers stay Linux-only. Tests: 13 new rocm_support cases (flavor drift, macOS ARM, Windows repair), parity updates. Full battery green (475 python, sh and ps1 suites). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: strip query tokens from the marker and tighten the pin-drift probe Four follow-ups to the round-3 pin-repair fixes: 1. The credential stripper feeding the torch-index marker and the logged repair messages dropped only user:pass@ userinfo, so a private feed that carries its auth token in the query string (.../simple?token=SECRET) persisted the token in the world-readable marker (mode 0644 under a default umask) and printed it in substep output. All four strippers (install.sh, install.ps1, studio/setup.ps1, install_python_stack.py) now drop the query and fragment before building the sanitized URL. A query is not part of a PEP 503 index's identity, so this also stops a rotated token from spuriously mismatching the marker and forcing a needless reinstall. 2. The --torch-pin-needs-apply fast-path probe accepted an untagged CUDA build (no +cuXXX local tag) under a specific cuXXX pin, but _ensure_cuda_torch reinstalls exactly that build to enforce the pin. The probe was more lenient than the repair, so the repair pass was skipped on the fast path. _torch_flavor_matches_pin now reports a mismatch for an untagged build under a cuXXX pin, forcing the pass. 3. The probe's ROCm branch accepted any HIP build for a rocm/gfx pin, while _ensure_rocm_torch decides a reinstall with the per-arch _rocm_pin_family_mismatch predicate (a generic +rocm7.2 wheel under a per-arch gfx pin, or a wrong ROCm version, is a mismatch). The probe now reuses that predicate, so it is as strict as the repair. This needs the installed torch version, so _probe_torch_flavor now returns (marker, cutag, version) and _torch_flavor_matches_pin takes the pin URL (extracting the leaf internally). 4. On Windows a known-family cu*/cpu pin is applied to the main venv by setup.ps1 before install_python_stack.py runs; a later dependency step can clobber it, and the GPU-aware _ensure_{cuda,cpu}_torch self-skip on Windows while the verbatim helper handles only unknown-family pins, so nothing repaired the clobber (setup.ps1 does not re-validate the main venv's torch afterward, verified). New _ensure_pinned_known_family_torch reinstalls a drifted cu*/cpu pin in the step-13 Windows/macOS-ARM branch; rocm/gfx per-arch specs stay owned by setup.ps1, unknown-family by the verbatim helper. A speculative ROCm 2.11 floor was also raised but is unreachable: the rocm7.2 index publishes no 2.x wheel below 2.11.0, and an unknown newer rocm is not floored speculatively. Tests: query/fragment strip cases in the sh + ps1 marker suites and the Python strip/marker tests; the tri-state helper and the probe/baseline harnesses moved to the (marker, cutag, version) flavor with matching versions; new probe cases (untagged CUDA, generic-rocm-under-gfx) and 8 _ensure_pinned_known_family_torch tests; a four-way query-strip parity assertion. Full battery green (1150 python, sh 26/26 marker, ps1 marker/flavor/pin-stale). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: reinstall markerless gfx pins and cap custom-index updates at torch 2.11 Two follow-ups from the pin-marker audit: 1. A markerless venv with a gfx per-arch 2.11 pin trusted the wheel version tag, which is byte-identical (+rocm7.13.0) across gfx120X-all / gfx1151 / gfx1150. A pre-marker install holding one gfx arch's wheel that is now pinned to a DIFFERENT gfx index was therefore never switched: _rocm_pin_family_mismatch returns no-mismatch for any three-part +rocm 2.11 wheel, and _ensure_rocm_torch's absent-marker branch fell through to that heuristic. _ensure_rocm_torch now forces a one-time reinstall when the marker is absent AND the pin leaf is a 2.11 gfx per-arch index; the reinstall writes the marker, so the next update compares exactly and does not loop (the correctly-pinned no-reinstall guarantee then comes from the exact marker compare, not the ambiguous tag). Non-gfx-2.11 pins (rocmX.Y, non-2.11 gfx) stay on the tag heuristic -- their tags are distinguishable. 2. The verbatim custom-index update path used _CUDA_TORCH_PKG_SPEC (torch <2.12.0) while a FRESH install of the same unknown leaf caps torch at <2.11.0 (install.sh's default TORCH_CONSTRAINT, and setup.ps1's custom-pin branch), so a private /simple mirror publishing torch 2.11 could upgrade a `studio update` to a state the fresh installer never produces. Added _CUSTOM_INDEX_TORCH_PKG_SPEC (torch>=2.4,<2.11.0), used only by the verbatim path; companions stay pinned for the same exclusive --index-url ABI reason as _CUDA_TORCH_PKG_SPEC (a bare name could pull a torch-2.12-built torchvision). _CUDA_TORCH_PKG_SPEC is unchanged (known-family cu/cpu repair correctly tracks install.sh's widened cu ceiling). Tests: 2 new markerless-gfx cases (one-time reinstall + marker write + no-loop second run, and the rocmX.Y absent-marker no-op), the pre-existing markerless gfx no-reinstall test flipped to assert the one-time reinstall (it had encoded the old tag-trusting behavior), and the custom-index bound assertions. 488 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: a matching marker must not mask a broken, clobbered, or misclassified torch Four round-6 follow-ups, all closing cases where a matching torch-index marker wrongly vouched for a torch that is not actually the pinned one: 1. _is_cuda_family_leaf matched cu+digits by PREFIX (^cu[0-9]), so a custom mirror leaf like cu128-private classified as CUDA family; the flavor check then compared the installed cu128 tag to the whole leaf cu128-private and forced a reinstall on EVERY update (never converging). The cu family is now matched EXACTLY (re.fullmatch cu[0-9]+), so a cu-suffixed custom leaf routes through the verbatim/unknown path with a stable marker. Mirrored in install.sh (_normalize_family_leaf: strip cu, require an all-digit remainder) and setup.ps1 / install.ps1 (^cu[0-9]+$). 2. _torch_pin_needs_apply returned False on a failed torch probe (missing or unimportable) under a matching marker, so setup.sh kept the fast path and a broken torch was never repaired. A failed probe now forces the pass: the marker cannot vouch for a torch that does not import, forcing is idempotent, and once torch imports again the probe succeeds and the forcing stops (self-resolving). Reverses the round-4 conservative choice for this case. 3. _ensure_verbatim_torch_index snapshotted the installed trio on the first pass with a matching marker and treated an unimportable torch (snapshot None) as "no drift, skip", so a torch clobbered to a broken state before the run was masked. A None snapshot now reapplies the pin. A torch clobbered to a WORKING-but-wrong build under an unknown-family pin remains undetectable from metadata (no flavor tag; reinstalling every update would be the loop this avoids) and is documented as a known limitation. 4. The step-13 Windows final repair reran only the verbatim (unknown-family) and known-family cu*/cpu paths, so a clobbered explicit rocm/gfx pin (the wheel setup.ps1 installed from AMD's per-arch index) was left in place. The branch now also runs _ensure_rocm_torch on Windows for an explicit rocm/gfx pin; it has a Windows path and no-ops when torch already links HIP, so it only reinstalls a genuinely clobbered ROCm venv (loop-safe). Tests: the round-4 failed-probe-trusts-marker test flipped to force the pass; new cases for the cu-suffix no-loop, the broken-torch verbatim reinstall, and the Windows rocm final-repair structure; item-2 exact-cu parity assertions. 490 passed. sh/ps1 marker + flavor + pin-stale suites all green. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: repair Windows ROCm pins from the pinned URL and honor NO_TORCH Four round-7 review items, two of them regressions in the round-6 work: 1. _torch_pin_needs_apply ignored UNSLOTH_NO_TORCH. With a torch-index env var set and no marker, the failed-probe branch forced the dependency pass on every `studio update`, and the pass (which also honors NO_TORCH) never installs torch or writes a marker, so nothing could ever stop the forcing. It now returns False immediately under NO_TORCH: the pin only matters once torch is actually installed. 2. The step-13 Windows final repair (round-6) restored a clobbered explicit rocm/gfx pin by calling _ensure_rocm_torch, whose Windows path reinstalls from the arch AUTO-DETECTED via hipinfo, not from the pin. A user pinning a different gfx family or a private mirror was restored from the wrong source (and the wrong marker written), and a headless box was skipped entirely (the arch probe returns nothing). The repair now goes through _ensure_pinned_known_family_torch, which reinstalls from the PINNED url with the same per-arch floor setup.ps1 uses (2.11-line gfx leaves) or a bare trio (older arches, rocmN mirrors). It is gated on IS_WINDOWS since macOS ARM has no ROCm, and the existing flavor check keeps it loop-safe (a matching HIP wheel is left alone). 3. _ensure_verbatim_torch_index's broken-torch check (round-6) used "_installed_trio_snapshot() is None", but that helper reports a REMOVED torch as "torch==absent" (a non-None tuple) and a broken import as the stale on-disk version, so a missing or unimportable torch under a matching marker was read as "no drift" and skipped. The matching-marker path now confirms torch health with an import probe (_probe_torch_flavor): a torch that does not import reapplies the pin, while a healthy torch keeps the snapshot-based intra-run drift detection. 4. A unit test for _ensure_cpu_torch did not pin NO_TORCH False like its siblings, so a suite run with UNSLOTH_NO_TORCH=1 in the environment made the guard return early and the reinstall assertions fail spuriously. Tests: the round-6 broken-torch verbatim test re-encodes the non-None "torch==absent" snapshot case (the exact state the old "is None" check missed); new Windows-ROCm pinned-repair cases (reinstall from the pin, per-arch floor vs bare spec, matching-wheel no-op, off-Windows no-op); a NO_TORCH fast-path probe case; the parity test now asserts the Windows final branch does not auto-detect the ROCm index and that the helper reinstalls from the explicit pin. 494 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: floor the rocm7.2 index in the Windows pin repair; isolate marker tests Three round-8 review items, two of them downstream of the round-7 changes: 1. _ensure_pinned_known_family_torch gave a rocm<d> index leaf a bare torch/torchvision/torchaudio trio while flooring only gfx* leaves, so a Windows venv clobbered under an explicit rocm7.2 pin could reinstall an unbounded or ABI-mismatched trio from that exclusive --index-url. It now mirrors the spec the initial ROCm paths pin: the rocm7.2 floor for 2.11-line gfx leaves and rocm<d> leaves that serve torch 2.11, the <2.11 default for older rocm versions, and a bare trio only for older gfx per-arch leaves (which publish no floor), matching _ROCM_TORCH_PKG_SPECS / _ensure_rocm_torch. 2. test_verbatim_custom_url_no_marker_reinstalls_once called _ensure_verbatim_torch_index twice; the second call now hits the matching-marker health probe, and with pip_install mocked torch never becomes importable, so in a no-torch environment _probe_torch_flavor returned None and forced another reinstall, failing the idempotence assertion. The test now pins a healthy flavor so the idempotence check is about the marker, not ambient torch. 3. The TestEnsureRocmTorchMarker fixture patched os.environ per test but not _TORCH_BACKEND, which install_python_stack.py computes once at import from UNSLOTH_TORCH_BACKEND. A runner starting with a cuda/cpu backend made _ensure_rocm_torch early-return and skip the mocked repair these tests exercise. The fixture now neutralizes _TORCH_BACKEND so the marker tests are independent of the caller's installer-pin environment. Tests: the Windows floor-spec test now asserts a rocm7.2 mirror pin uses the rocm7.2 floor (not bare), plus a new rocm7.1 case that must fall back to the <2.11 default; the marker suite passes under a hostile UNSLOTH_TORCH_BACKEND=cuda / UNSLOTH_TORCH_INDEX_URL env. 495 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: apply same-flavor pin repoints, keep ROCm fallback nonfatal, bound custom companions Four round-9 review items, two of them regressions in the round-7 pin helper: 1. _ensure_pinned_known_family_torch returned as satisfied whenever the installed flavor matched the pin, so a same-flavor SOURCE change (one /cpu or /cu128 mirror to another, or a gfx1151 -> gfx120x-all per-arch switch, both carrying the same wheel tag) was never applied, while _torch_pin_needs_apply kept forcing the pass on the marker mismatch forever. It now also reinstalls when the marker records a DIFFERENT index of the same flavor, rewriting the marker so the next update matches (no loop), exactly as the Linux _ensure_{cuda,cpu}_torch helpers do. An absent marker on an already-matching venv is still left to the baseline recorder (no forced reinstall of a correct pre-marker venv). 2. That helper reinstalled a Windows ROCm pin with the FATAL pip_install, so when setup.ps1 had taken its CPU fallback (the pinned AMD index unavailable), the final repair re-hit the same missing index and aborted the whole install. The ROCm reinstall is now nonfatal (pip_install_try): on failure it leaves the CPU base in place and writes no ROCm marker, so the install completes -- matching _ensure_rocm_torch's Windows path. cu*/cpu pins stay fatal (authoritative source). 3. install.sh left torchvision/torchaudio bare for a pinned custom/unknown-leaf index (a private /simple mirror), unlike the Python update path's _CUSTOM_INDEX_TORCH_PKG_SPEC, so a mirror also exposing newer companion wheels could resolve a torch-2.12-built torchvision against the capped <2.11 torch. It now bounds the companions (torchvision>=0.19,<0.26.0 / torchaudio>=2.4,<2.11.0) for a custom leaf, gated on an empty _expected_torch_flavor_tag so known families keep their curated bare/floored companions. 4. install.sh's _expected_torch_flavor_tag matched cu[0-9]* by prefix, so a custom leaf like cu128-private classified as the cu128 family and force-reinstalled a correct +cu128 wheel on every run. It now requires exact cu+digits (routing the suffixed leaf to the custom path), matching the Python re.fullmatch(cu[0-9]+) and PowerShell, and feeding item 3's custom-leaf detection. Tests: new cases for the same-flavor marker-change reinstall, the nonfatal ROCm fallback (no marker on failure), the rocm7.2/older-rocm floor selection now split across the nonfatal path, cu-suffixed custom leaves in test_torch_flavor.sh, and the custom-leaf companion bounds in test_torch_constraint.sh. 497 python + 143 shell assertions pass; the marker suite still passes under a hostile UNSLOTH_TORCH_BACKEND=cuda env. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: bound custom-pin companions on the Windows setup path; isolate pin-probe tests Two round-10 review items: 1. setup.ps1's custom/unknown-leaf pin branch capped only torch ($cudaTorchSpec) and still asked the exclusive index for bare torchvision/torchaudio, so a private mirror that also serves newer companion wheels could install a torch<2.11 wheel alongside a torchvision>=0.26 / torchaudio>=2.11 built for a newer torch ABI, after which the marker records the pin as applied. It now bounds the whole trio (torch>=2.4,<2.11.0 / torchvision>=0.19,<0.26.0 / torchaudio>=2.4,<2.11.0) for a pinned non-cu-family leaf, matching install.sh, install.ps1's fresh pinned install, and install_python_stack.py's _CUSTOM_INDEX_TORCH_PKG_SPEC. This completes the companion-bounds fix across all three installers; known cu* leaves keep bare specs (the family index bounds them). 2. The _torch_pin_needs_apply probe tests did not pin NO_TORCH False, so a test process launched with UNSLOTH_NO_TORCH=1 short-circuited the probe (the round-7 guard) and returned False for cases that expect the pass to run. The _needs_apply helper now patches NO_TORCH (default False) around the call, and the dedicated no-torch case passes no_torch=True explicitly. Tests: the cross-platform parity test now asserts setup.ps1 bounds the full trio (not just torch) for a custom leaf; the pin-probe suite passes under a hostile UNSLOTH_NO_TORCH=1 environment. setup.ps1 parses clean; 497 python + shell suites green. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: bound custom rocm-* pins, redact diag tokens, snapshot custom pins before base update Three round-11 review items, all reproduced before fixing: 1. install.sh's custom-index companion bounds gated on _expected_torch_flavor_tag returning empty, but that helper returned "rocm" for ANY rocm* leaf, so a custom mirror whose leaf starts with rocm but is not a pip family (a private rocm-current mirror, a Radeon find-links rocm-rel-7.2.1) escaped the bounds and installed bare torchvision/torchaudio. It now digit-gates rocm to rocm[0-9]* (matching the Python _is_pip_rocm_family_leaf ^rocm\d), so those custom leaves return "" and the <2.11 companion caps apply; real rocm7.2 / gfx per-arch indexes still classify as rocm. 2. _tauri_torch_index_family classified by the raw last path segment, so a pinned URL carrying auth in the query (.../rocm7.2?token=SECRET) had the token echoed verbatim into the emitted [TAURI:DIAG] line. It now strips query/fragment before classifying (mirroring the marker/log credential stripping), so no token reaches the diagnostic output; as a side effect .../cu128?token=x now classifies as cu128 instead of auto. 3. On studio update, the core package step (a newer unsloth can require a torch the custom pin does not satisfy, pulling a default PyPI trio) runs BEFORE the step-2b verbatim check, which then recorded the already-clobbered trio as the baseline for a matching marker and left the pin unapplied. A new _capture_verbatim_baseline() records the pre-clobber trio before the core step, so the verbatim pass detects the drift and reapplies the pin. Captures only for a matching custom pin with importable torch; a mismatched/absent marker or broken torch is left to _ensure_verbatim_torch_index. Tests: _expected_torch_flavor_tag rocm-current / rocm-rel cases; _tauri_torch_index_family token/fragment redaction with a no-leak regression guard; _capture_verbatim_baseline record/skip cases plus an end-to-end clobber-detection scenario; a structural guard that the capture runs before the core step. 501 python + shell suites pass; install.sh bash -n clean, shellcheck unchanged from base. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: match rocm family leaves exactly, enforce the rocm7.2 torch line, repair a broken pinned torch A pinned index is a pip ROCm --index-url family only when its leaf is an exact rocm<digits> / rocm<digits>.<digits> (rocm7.2) or a gfx* per-arch leaf. The prior ^rocm[0-9] prefix match also caught suffixed private-mirror leaves (rocm7.2-private, rocm7-current), routing them through the ROCm/companion-family path instead of the verbatim pin: the companion bounds were skipped and, on a pre-marker venv with a compatible +rocm wheel, the pin was never applied. Match the family exactly through one shared helper at every site: - install_python_stack.py: _is_pip_rocm_family_leaf (re.fullmatch), plus the two other loose gates it feeds (_normalize_family_leaf, _torch_flavor_matches_pin). - install.sh: a new _is_pip_rocm_family_leaf routes _expected_torch_flavor_tag, _torch_index_repairable, _normalize_family_leaf and the ROCm side-effect gate. - setup.ps1: a new Test-PipRocmFamilyLeaf routes Get-NormalizedFamilyLeaf and both pinned reroutes; install.ps1 anchors its reroute regex. _rocm_pin_family_mismatch (and its setup.ps1 mirror Get-RocmPinStaleTags) compared only the ROCm version, so a +rocm7.2 wheel whose torch release drifted off the 2.11 line (2.12/2.13 from an out-of-band upgrade or a custom rocm7.2 mirror) satisfied the family check while violating _ROCM_TORCH_PKG_SPECS['rocm7.2'] (torch>=2.11,<2.12). Flag it stale so the repair reinstalls to floor; >=2.11 alone is not enough, so the release is compared exactly against the 2.11 line for a KNOWN-2.11 rocm pin. _ensure_pinned_known_family_torch returned on a failed import probe, but _torch_pin_needs_apply forces the dependency pass on that same failed probe: a broken torch under a known-family pin was left in place and the pass was forced on every update. Treat an unimportable torch as drift and reinstall the pinned trio (the spec and marker derive from the pinned leaf, not the absent flavor); once it lands the probe succeeds and the fast path returns. Tests: exact-match cases across test_torch_flavor.sh, test_rocm_support.py, test_cross_platform_parity.py and the two .ps1 helper suites; the rocm7.2 release-line and broken-probe-reinstall cases; extraction lists updated for the new helpers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: anchor the PS pinned-ROCm floor gate and bound install.ps1 custom-pin companions Round 12 made every family CLASSIFIER exact, but the Windows install-flow floor gate reads $_pinRocm211 directly from the raw pinned leaf with an unanchored -match '^rocm(\d+)\.(\d+)' BEFORE any exact classification runs. A suffixed custom leaf (rocm7.2-private) matches that rocm7.2 prefix, so it takes the 2.11-floor branch and is force-routed through the ROCm install path before the exact-match elseif can send it to the verbatim install. Anchor the match ($) in both install.ps1 and setup.ps1 so only an exact rocmX.Y leaf is floored; a suffixed or newer-suffix leaf falls through to the verbatim path. The Python floor selection is already exact (dict lookups gated on _is_pip_rocm_family_leaf), so only the two PS scripts needed this. install.ps1's custom (non-cu-family) pinned-torch install bounded torch>=2.4,<2.11.0 but left torchvision/torchaudio bare, so a private mirror serving newer companions could pull a wheel built for a newer torch ABI while the marker records the pin as applied. Bound both companions (torchvision>=0.19,<0.26.0 / torchaudio>=2.4,<2.11.0) when the leaf is not a cu<digits> family index (a cu index bounds its own resolution), matching setup.ps1's Test-CudaFamilyLeaf gate and _CUSTOM_INDEX_TORCH_PKG_SPEC. Tests: parity guards for the anchored floor gate in both PS scripts and for install.ps1's bounded custom-pin companions. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: tighten comments in the torch-index-override paths Collapse the verbose comment and docstring blocks added across the installer scripts and their tests to fewer, clearer lines without changing behaviour. Remove a duplicated CUDA-spec comment block. Comments/docstrings only; no code changes (AST-verified). * install: repair a broken pinned torch on Linux, strip trailing slash in tauri family, count the final step _ensure_cuda_torch / _ensure_cpu_torch returned on a failed import probe (torch present but unimportable). With an explicit CUDA/CPU pin, _torch_pin_needs_apply forces the dependency pass on that same failed probe, and the base package update does not force-reinstall an already-installed torch distribution, so the broken torch was left in place and the pass reran every update without repairing it. Treat a failed probe under a pin as drift and reinstall from the pinned index (the reinstall rewrites the marker and the next probe imports, so no loop). This is the Linux counterpart of the known-family repair fix. _tauri_torch_index_family stripped the query/fragment before classifying but not a trailing slash, so a token-authenticated pin like .../cu128/?token=x collapsed to .../cu128/ and fell through the exact-suffix */cu128 and */cpu arms to "auto". Strip a trailing slash too, mirroring _torch_index_url_leaf. The Windows / macOS-ARM final torch-repair step (_ensure_pinned_known_family_torch) runs a progress step that base_total never counted (the final-step increment was gated to Linux), so _STEP ran one past _TOTAL on those platforms. Add the missing increment. Tests: broken-probe reinstall for the CUDA (family and URL pins) and CPU paths; trailing slash / slash+token cases for _tauri_torch_index_family; a full-flow progress-count guard asserting _STEP == _TOTAL on Windows and Linux. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: tighten comments in the torch-index-override paths * install: harden the torch-index pin across all four installers Redact index-URL credentials from captured install logs before they print on failure. uv/pip failure text embeds the failing --index-url verbatim, so a user:token@ or ?token= secret could leak into the console. Add a shared redaction pass (_redact_install_output / Redact-InstallOutput) wired into the error-output dump in install.sh, install.ps1, setup.ps1 and install_python_stack.py. Verbose mode still streams live uncaptured output, so it is intentionally left unredacted (developer opt-in). Trim trailing slashes on the PATH only for a verbatim UNSLOTH_TORCH_INDEX_URL override, preserving a ?query/#fragment token. A whole-URL rstrip corrupted a base64 token ending in "/", and a single-slash strip left .../cu128// classifying as an empty leaf. Add _trim_index_path_slashes / Trim-IndexPathSlashes and route the override through it; strip ALL trailing slashes in the backend-branding leaf classifier so a double slash still yields the real leaf. Reject a trailing-dot ROCm leaf (rocm7.) in the bash family validator so it matches Python re.fullmatch(rocm\d+(?:\.\d+)?) and the PowerShell regex: both the major and the minor must be non-empty digits, so rocm7. is a custom verbatim pin, not a pip ROCm family. Scrub PIP_NO_INDEX and PIP_INDEX_URL for a pinned install in the two installers that have a plain-pip fallback (install_python_stack.py, setup.ps1): PIP_NO_INDEX=1 makes the fallback ignore every index including the pinned --index-url, and PIP_INDEX_URL replaces it. install.sh and install.ps1 install via uv --default-index (which ignores pip config/env), so they are unaffected. Add unit tests (bash, Python, PowerShell) and cross-platform parity tests covering credential redaction, path-only slash trimming, the rocm7. validator, the double-slash leaf, and the PIP_NO_INDEX/PIP_INDEX_URL scrub. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: redact captured torch-install output and warn on a failed pinned ROCm repair Close a redaction gap the earlier pass missed: setup.ps1's direct `Fast-Install ... | Out-String` branches (ROCm from $ROCmIndexUrl, CPU/CUDA from $TorchInstallIndexUrl, plus the Triton and T5 sub-venv installs) printed the captured $output verbatim on failure, bypassing Redact-InstallOutput. A private index carrying userinfo or a ?token= in the pin could leak into Windows Studio setup logs. Route every `Write-Host $output` through Redact-InstallOutput. Warn on a failed pinned Windows ROCm reinstall in _ensure_pinned_known_family_torch: the branch printed "reinstalling from it" then called pip_install_try, but had no else, so a failure continued silently and left the user believing the pin was applied while the old CPU/wrong torch survived. Mirror the auto-ROCm Windows path and warn, telling the user to retry. * install: redact captured output on the pip fallback and optional-install failure paths The uv install path already redacted its captured output, but pip_install's pip fallback runs through run(), which printed result.stdout verbatim on failure, and _print_optional_install_failure did the same. A pinned --index-url carrying userinfo or a ?token= could still leak there when uv is unavailable or the pip fallback also fails. Route both through _redact_install_output. The verbose pip_install_try path stays raw (developer opt-in), matching the other installers. * install: split the survive-updates marker subsystem into a follow-up The torch-index override PR grew a persisted per-venv marker plus repair machinery (stale-pin detection, verbatim re-apply, update-time reinstall triggers) that roughly doubled it. That subsystem is orthogonal to the core feature and is being reworked in a follow-up (versioned/hashed marker, full-URL pin baseline), so it moves there wholesale instead of shipping twice. What this PR still does: UNSLOTH_TORCH_INDEX_URL / UNSLOTH_TORCH_INDEX_FAMILY pick the torch wheel index at install time in all four installers, with the exact rocm/gfx/cpu/cu leaf classification, the torch 2.11 floor for the per-arch AMD indexes, bounded companions for custom leaves, credential redaction of captured installer output, path-only slash trimming, and the uv/pip index env scrubs. Flavor-based repair keeps honoring the pin: a wrong family under an explicit pin still reinstalls from the pinned URL, and setup.ps1 repairs a pinned stale venv in place instead of wiping it. What moves to the follow-up: the .unsloth-torch-index marker file and its writers/readers/normalizers, exact-URL pin-change detection on update (same-tag gfx switches, custom-mirror repoints), the verbatim trio snapshot and clobber re-apply, the pin-baseline recorder, and the --torch-pin-needs-apply fast-path probe in setup.sh / setup.ps1. Their tests (the marker sh/ps1 suites, the stale-pin suite, and the marker classes in the rocm/cuda/parity suites) move with them; the removed code is preserved on a local archive branch to seed that PR. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: re-apply a ROCm pin over an existing HIP wheel via the version tag The subsystem split left an explicit ROCm/gfx pin unenforced on `studio update` whenever the venv already imported ANY ROCm torch: the pinned reinstall lived inside the `elif not has_hip_torch` branch, so a rocm6.4 to rocm7.2 switch, a gfx1151 pin over a generic +rocm7.2 wheel, or a broken 2.12+rocm7.2 drift never re-applied the pin. Restore the markerless half of that detection: _rocm_pin_family_mismatch compares the pinned leaf against the installed wheel tag (exact rocmX.Y compare, the 2.11 gfx per-arch allowlist, the untagged-wheel rule), the HIP probe emits "<hip_marker>|<version>" again so the installed tag is available, and _ensure_rocm_torch reinstalls from the pinned URL when the tag mismatches even though HIP torch is present. setup.ps1 mirrors it: the stale-venv check routes a pinned rocm/gfx leaf through Get-RocmPinStaleTags instead of collapsing it to a generic "rocm" flavor, and the existing pinned in-place repair (no wipe) applies the change. What still waits for the follow-up marker PR, by design: pin changes the wheel tag cannot see -- a per-arch switch between two 2.11 gfx indexes (identical +rocm7.13.0 tag), a custom-mirror URL repoint under the same family leaf, and unknown-family verbatim pins. Those need the persisted index record. Tests restored with the code: the _rocm_pin_family_mismatch table, the five update-path cases (older-rocm reinstall, gfx-over-pre-2.11 reinstall, matching-pin no-reinstall, non-2.11 gfx no-reinstall, gfx-over-generic-2.11 reinstall), the "|" probe-format guards, and the AST-extracted Get-RocmPinStaleTags suite for setup.ps1. * install: compare major-only rocm pins, redact URL fragments, bound pinned CPU trio Three review fixes on the restored pin-repair path. The family classifier accepts a major-only rocm<d> leaf (rocm7), but the mismatch comparators only parsed rocmX.Y, so a rocm7 pin fell through to the 2.11-line fallback and INVERTED both verdicts: an installed +rocm6.4 wheel compared as satisfied (pin never re-applied) while a matching +rocm7.2 wheel compared as stale (reinstall loop). Major-only pins now compare on the major alone in _rocm_pin_family_mismatch and Get-RocmPinStaleTags: rocm6.x under a rocm7 pin is a mismatch, any rocm7.x satisfies it, an untagged wheel never does, and a bare +rocm tag with an unreadable version is accepted (matching the existing lenient unreadable fallback). The output redactors scrubbed userinfo and ?query= values but not #fragments, so a pin like https://mirror/whl/cu128#token=secret leaked the secret in captured uv/pip failure text -- inconsistent with the URL handling itself, which already treats fragments as sensitive. All four redactors gain a URL-anchored fragment rule (anchored so a bare "# comment" line in tool output is never touched). setup.ps1's CPU branch installed a bare torch/torchvision/torchaudio trio; fine for the unpinned host default, but a PINNED cpu index routes through the same branch and the /cpu index serves newer torch, so a fresh pinned CPU install could land an unsupported trio that _ensure_cpu_torch then keeps (it accepts any CPU build). Under a pin the branch now installs the bounded trio mirroring _CPU_TORCH_PKG_SPEC (torch>=2.4,<2.12.0 and matching companions); the unpinned path is unchanged. Tests: major-only rows in the Python mismatch table and the AST-extracted setup.ps1 suite; fragment + query-plus-fragment + bare-hash-comment cases in all four redactor suites; a parity check that the pinned CPU trio bounds exist, are gated on the pin, and mirror the Python repair spec. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install: tighten comments in the torch index override paths * tests: track the moved pass-through inheritance in the gguf order check Main moved the llama_extra_args pass-through inheritance out of the GGUF branch into _resolve_inherited_extra_args, which runs before it, so the source-order assertion's "if request.llama_extra_args is None" anchor no longer exists inside the branch and the check failed after the main merge. The test now asserts the same property in the current shape: inheritance before the GGUF branch (a carried --no-mmproj still shapes the hub guard's companion requirement), and marker, hub guard, unload in order within the branch. Full file passes (32 tests). * tests: anchor the inheritance order check on the call, not the definition source.index("_resolve_inherited_extra_args(") matched the function definition, which always precedes the endpoint, so the ordering assertion was vacuously true. Anchoring on "= _resolve_inherited_ extra_args(" pins the first call site inside the load endpoint (line 4505), which is the statement whose position relative to the GGUF branch the test is meant to guard. 32 tests pass. * tests: align the gguf order test with main Main fixed the stale ordering assertion in PR 7252; adopting its version verbatim removes this file from the branch diff entirely and avoids a conflict on the next main merge. 32 tests pass. * install: bound the companion constraints to torch's window everywhere A full platform x vendor validation matrix over this branch surfaced a real trio mismatch on the cpu/mac paths: torch is capped <2.11 (installs 2.10.0+cpu) but the bare torchaudio companion resolves 2.11.0+cpu, because torchaudio 2.11 dropped its exact torch pin. Reproduced in a sandboxed end to end cpu install. torchvision still exact-pins torch and self-corrected. The default companion constraints are now bounded to torch's window (<0.26 / <2.11) and widen together with the cu* torch window (<0.27 / <2.12), so every leaf resolves a paired trio. Verified with uv dry-runs on the cpu, cu130, and rocm6.4 leaves (2.10.0/0.25.0/2.10.0, 2.11.0/0.26.0/2.11.0, 2.9.1/0.24.1/2.9.1) and a rerun of the sandboxed cpu install, which now lands torch 2.10.0+cpu with torchaudio 2.10.0+cpu. The Strix WSL reroute now also forwards UNSLOTH_TORCH_INDEX_URL and UNSLOTH_TORCH_INDEX_FAMILY into the rerouted 24.04 distro; dropping them silently reverted the child install to auto-detection, defeating the pin this branch introduces. test_torch_constraint.sh updated: the bounded companions must appear at the defaults and the custom-leaf block, no bare companion may remain, and the cu* widen must carry the companions with it. * install: harden the override path against reroute drift and credential leaks Review sweep focused on default-path idempotency found no defects on the unset path; these fixes cover the override path and failure reporting. install.sh: - The early WSL Strix Halo distro reroute now honors an explicit index pin (UNSLOTH_TORCH_INDEX_URL / _FAMILY): the pin is used in the current distro instead of probing the GPU and re-entering another distribution, matching the contract of the later Radeon and Strix guards. Whitespace only values do not gate, in parity with get_torch_index_url. - Verbose mode now streams installer output through the credential redactor; it previously bypassed the redaction the quiet path applies. The exit code survives the pipe via an rc file since the script runs under plain sh with no pipefail. - The kept-release fallback warning now strips credentials from the index URL before printing it. install.ps1: - Bounded torchvision and torchaudio next to every capped torch install (custom pin, ROCm CPU fallback, CUDA flavor repair). torchaudio 2.11 dropped its exact torch pin from the wheel metadata, so a bare companion beside torch<2.11 can resolve a mismatched 2.11.0 build, cu family indexes included. Mirrors the install.sh companion bounds. studio/install_python_stack.py: - The verbose failure path now redacts index URLs in pip and uv output before printing, matching every other output site in the file. All sh, ps1 and python installer test suites pass (the host-defaults suite has a known pre-existing failure unrelated to this change). * install: redact verbose Windows installer output and repair the parity tests Follow-ups to the override-hardening commit, from review: - install.ps1 Invoke-InstallCommand and setup.ps1 Invoke-SetupCommand now pipe verbose output through Redact-InstallOutput per record, and the three verbose Fast-Install torch call sites (ROCm, CPU, CUDA) do the same: uv and pip echo the pinned index URL, credentials included, in their errors, and verbose mode previously bypassed the redaction the quiet paths apply. ForEach-Object and Out-Host leave $LASTEXITCODE untouched, verified with a native command exiting 7 behind the pipe. - test_cross_platform_parity.py: the install.ps1 companion-bounds assertion now matches the implemented behavior (bounds on every index, no cu-family exemption, since torchaudio 2.11 dropped its exact torch pin) instead of requiring the removed $_pinCuLeaf gate. - test_rocm_support.py: the WSL reroute guard test slices the whole function body to its closing brace instead of a fixed 1200-character window, which the new pin-gate preamble had outgrown. 428 tests pass across the parity, install stack and rocm support suites; the sh and ps1 installer suites pass unchanged. * install: tighten comments in the torch-index and ROCm/CUDA repair paths * install: digit-gate the gfx family leaf and honor ROCm pins in the Windows repair Two review follow-ups on the override path: - The pip ROCm family predicate accepted ANY gfx-prefixed leaf, so a custom verbatim pin like /gfx-private classified as a ROCm family and enabled the ROCm-only side effects (AMD bitsandbytes, ROCm torch repair) on a mirror that may serve CPU/CUDA wheels. gfx now requires a following digit (gfx90a, gfx1151, gfx120X-all), consistently in install.sh, install_python_stack.py, install.ps1 (family gate and expected-flavor classifier) and setup.ps1, matching the strictness the rocm side already had (rocm7.2-private stays verbatim). The broader backend BRANDING globs are unchanged on purpose: radeon repo leaves (rocm-rel-X.Y) must still brand the rocm backend without being force-repaired as a family. - The Windows branch of the ROCm torch repair always installed from the public per-arch index, ignoring an explicit ROCm-family pin: after a pinned setup.ps1 install failed to a CPU base, the repair retried repo.amd.com instead of the pinned index. The branch now resolves _explicit_rocm_torch_index_url() first, uses it as the install index when set, and mirrors the Linux pin contract by skipping the NVIDIA and gfx-detection gates a pin is documented to override. Source-assertion tests updated to the tightened predicate and the new repair label. 1165 tests pass across the parity, install stack and studio install suites; the sh and ps1 suites pass; both PowerShell installers parse clean. * Remove scratch archives accidentally committed with the comment pass The temp/ archive copies of installer and test files were working scratch, not PR content, and inflated the diff by about nine thousand lines. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-20 00:58:52 -07:00
Michael Han	6d8c18cd1a	Replace standalone Studio wording with Unsloth (#7221) ... * Replace standalone Studio wording with Unsloth Replace the single word Studio with Unsloth wherever it is used as shorthand for Unsloth Studio in docs, CLI output, UI strings, i18n locales, workflow display names, comments and docstrings. Kept unchanged: the full name Unsloth Studio, third party product names (LM Studio, Visual Studio, Mac Studio), feature names (Recipe Studio, Fine-tuning Studio and its translations), and all identifiers such as env vars, commands, paths and filenames. * Address review feedback on the Studio wording rename Use "an" before Unsloth where the rename left the article as "a". Restore the split brand where Unsloth and Studio render as two halves of the full product name: the onboarding sidebar subtitle and the IPv6 localhost warning. Scope two messages to the full name Unsloth Studio where plain Unsloth was misleading: the AMD README bullet and the CLI studio setup error.	2026-07-19 00:47:04 -07:00
Thomas Eric 🇧🇷	03cbe211a3	Studio: fix flash-attn and torchao install on Blackwell (sm_100+) GPUs (Closes #6961) (#6970) ... * fix: Remove moot has_blackwell_gpu() function Fixes unslothai/unsloth#6961. This function skipped flash-attn on Blackwell GPUs because no prebuilt wheel existed; Dao-AILab now ships one and url_exists() already gates resolution. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: use torchao 0.17.0 for Blackwell Fixes #6961. Torchao 0.16.0's cpp extensions are built against CUDA 12, so on a CUDA-13 torch (cu130 / Blackwell) they fail to load with "libcudart.so.12: cannot open shared object file". Select 0.17.0 there instead: its cpp targets torch 2.11, so it is skipped cleanly rather than crashing. CUDA-12 / ROCm / CPU torch 2.10 keeps 0.16.0 and its working kernels. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Condense torchao version-selection comments (no behavior change) * Support torch 2.11 in the Studio installer via the torch2.10 prebuilt wheels Map torch 2.11 to the torch2.10 prebuilt wheels for flash-attn, causal-conv1d, and mamba through wheel_utils.prebuilt_wheel_torch_mm, applied in direct_wheel_url (filename) and flash_attn_wheel_url (version). Those torch2.10 CUDA wheels load and pass each project's own test suite on torch 2.11 (verified on B200), so a torch 2.11 environment gets the prebuilt accelerators instead of skipping or building from source. Raise _CUDA_TORCH_PKG_SPEC to <2.12.0 (torchvision <0.27.0, torchaudio <2.12.0) so the CUDA torch repair path can install torch 2.11, where torchao 0.17's cpp kernels load cleanly. Add tests for the mapping. * Keep has_blackwell_gpu as a False stub for future arch gating * Restore has_blackwell_gpu as a return-False probe kept for future arch gating Keep the nvidia-smi compute_cap detection and its two call sites, but short-circuit with return False at the top so flash-attn is no longer skipped on Blackwell (sm_100+ now has prebuilt wheels and url_exists gates resolution). Drop the early return to re-enable arch-based detection later. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-07-08 06:38:10 -07:00
Daniel Han	c2a7b78f6b	Studio: exclude mlx-lm 0.31.3 (broke gemma4/qwen3_5 QK-norm load on Apple Silicon) (#6803) ... * Studio: exclude mlx-lm 0.31.3 (broke gemma4/qwen3_5 QK-norm load) mlx-lm 0.31.3 regressed the QK-norm archs: its strict load_weights rejects the q_norm/k_norm tensors with "Received N parameters not in model", so gemma4 and qwen3_5 checkpoints fail to load. Studio installs the MLX stack unpinned at latest, which pulls 0.31.3. Verified on a real macos-14 runner: gemma4 fails to load on 0.31.3 but loads and generates coherently on 0.31.2 and on git-main (future 0.31.4). See mlx-lm #1242. Exclude just that release (!=0.31.3) in the installer and the self-heal floor so --upgrade still resolves to the newest good build, and treat an already-installed 0.31.3 as unsatisfied so the self-heal replaces it. * Studio MLX: cover fresh-install path + robust bad-version compare Address PR review: - Fresh install.sh (Apple Silicon) runs the base 'uv pip install unsloth' with SKIP_STUDIO_BASE=1, skipping the guarded MLX-stack step, so transitive resolution could still pull mlx-lm 0.31.3. install.sh already exports UV_OVERRIDE -> overrides-darwin-arm64.txt before that install, so exclude mlx-lm 0.31.3 there too; this also strengthens the self-heal (same override). - Match the known-bad version with parsed packaging.Version so 0.31.3 == 0.31.3.0 (trailing-zero normalization) instead of raw string equality. * Studio: exclude mlx-lm 0.31.3 on the fresh Apple Silicon install too The overrides file only applies via UV_OVERRIDE when it exists relative to the script, which is not true for a curl-piped install, and the guarded MLX step in install_python_stack.py is skipped there (SKIP_STUDIO_BASE=1). So the base install could still resolve the transitive mlx-lm to the broken 0.31.3. Append mlx-lm!=0.31.3 to the base install on Apple Silicon (empty elsewhere), so the fresh path pins away from 0.31.3 without waiting for the runtime self-heal. * Studio: exclude mlx-lm 0.31.3 on the migrated install; keep the >=0.22.0 floor The with-deps migrated install did not append ${_MLX_LM_EXCLUDE_ARG:-}, so a curl-piped Apple Silicon migration (no repo overrides file, UV_OVERRIDE unset) could resolve mlx-lm 0.31.3 transitively. Append the exclusion there, matching the fresh install path. The no-torch migration is left alone since --no-deps never resolves mlx-lm (same as the fresh no-torch path). Also restore the >=0.22.0 floor in overrides-darwin-arm64.txt: a uv override replaces the transitive constraint, so a bare !=0.31.3 could let the resolver drop below the supported minimum that mlx_repair.py enforces at runtime. * Triage huggingface_hub 1.22.0 / fastapi / multiprocess scanner false positives The scan-packages gate red-failed on all three shards after transitive deps bumped. Every new CRITICAL is a benign false positive, verified against upstream: - huggingface_hub 1.22.0 added _sandbox.py for the remote HF sandbox feature. Its job-startup bootstrap string (fetch sbx-server into the container /tmp and exec it) and the SandboxPool host-reservation loop trip the staged-dropper and C2-loop heuristics; that script runs inside a remote HF container, not on the user machine. The bump also re-hashed the already-reviewed benign polling loops in hf_api.py and utils/_http.py. The PyPI artifact is byte-identical to the official v1.22.0 tag. - fastapi 0.139.0 routing.py re-hashed the websocket keepalive while-True loop; byte-identical to upstream 0.139.0. - multiprocess 0.70.19 forkserver.py and tests/__init__.py re-hashed the AF_UNIX fork-server IPC and fd-inheritance tests; genuine uqfoundation release, local IPC not network. Added 7 reviewed allowlist entries (no blind regenerate). All three shards (hf-stack, studio, extras) exit 0 locally. * Tighten mlx-lm 0.31.3 exclusion comments * Trim mlx-lm 0.31.3 exclusion comments	2026-07-06 19:40:06 -07:00
Ayushman	c356427f30	Guard Windows ROCm torchao override skip (#6837) ... * Fix: skip fp16/bf16 validation for full finetuning in RL trainers When doing full finetuning (FFT) of a bfloat16 model, the fp16/bf16 mismatch validation fires before the corrective logic runs, causing a misleading error even though the code would properly handle it downstream. Skip the validation when full_finetuning is active. Fixes #6731 * Fix: auto-correct fp16/bf16 mismatches for full finetuning before validation Instead of entirely skipping validation (which could let mismatches through when mixed_precision_dtype is float32), auto-correct explicit fp16/bf16 settings that conflict with the model's dtype for FFT. This way the existing validation still catches real mismatches for non-FFT cases, and the corrective logic below handles the normalized settings. Fixes the issue raised in Codex review of PR #6813. * Guard Windows ROCm torchao override skip Detect installed ROCm torch directly before applying the torchao override so Windows ROCm environments never install the crashing torchao package even if the earlier ROCm-installed flag is missing. * Update unsloth/models/rl.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update studio/install_python_stack.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Harden ROCm probe and sync RL precision flags Tolerate stray stdout noise when probing Windows ROCm torch installs by checking the last non-empty output line, matching the existing torch version probe behavior. Also keep args.fp16 and args.bf16 synchronized with the full-finetuning precision auto-corrections in the RL trainer patch so downstream eval settings see a consistent TrainingArguments state. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add MLX trainer compatibility shims Patch imported MLXTrainer and MLXTrainingConfig objects to preserve the expected dataclass field ordering and to provide a _train_dataset_for_batches fallback when older trainers or test doubles only expose train_dataset. Also add focused worker tests covering both compatibility paths. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope PR to Windows ROCm torchao guard * Restore PR scope to Windows ROCm guard * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * test: cover Windows ROCm torchao skip behavior * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Ayushman Paul <ayushman@HP> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com>	2026-07-03 19:24:29 +01:00
Abdul Moiz	91f4ec7ba7	Studio: self-heal a pre-#6483-fix anyio>=4.14 stuck in existing installs (#6805) ... * Studio: self-heal a pre-#6483-fix anyio>=4.14 stuck in existing installs The <4.14 cap in constraints.txt/no-torch-runtime.txt only constrains new anyio resolutions. An install made before that cap existed can already be sitting on anyio 4.14+, and since it already satisfies mcp/fastmcp's anyio>=4.5 floor, every later constrained install skips it as already-satisfied -- so affected installs never recover and keep hitting the cancel-scope RuntimeError on every request (#6797, a recurrence of #6483). Force-reinstall anyio<4.14 whenever a stuck 4.14+ is detected. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: also repair anyio on the update fast path setup.sh's _SKIP_PYTHON_DEPS and setup.ps1's $SkipPythonDeps skip install_python_stack.py entirely once the installed package version already matches PyPI latest, so an install stuck on anyio>=4.14 with an otherwise up-to-date package never reaches the repair added in install_python_stack.py. Probe anyio on that fast path too and fall through to the full dependency pass when it's still >=4.14, mirroring the existing ROCm/CPU-torch override right below it. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-02 15:49:40 +02:00
Saicharan Ramineni	9d53656614	Make _uv_safe_path space-safe on macOS/Linux (#6503) (#6534) ... * Copy uv `-c`/`-r` paths to a space-free temp dir on macOS/Linux uv 0.11.x truncates a constraints/requirements path passed via `-c`/`-r` at the first space, so `unsloth studio` setup from a repo cloned under a path containing a space (e.g. `/Users/me/Open Source/unsloth`) fails with: error: File not found: `/Users/me/Open` _uv_safe_path() already worked around this on Windows via the 8.3 short path but returned the space-containing path unchanged on macOS/Linux, which have no 8.3 equivalent. Extend it to copy the (small, flat) constraints/requirements file into a space-free temp dir and hand uv the copy; the temp dirs are removed at process exit. Falls back to the original path on any error, so it is never worse than before. Refs unslothai/unsloth#6503 * Route UV_OVERRIDE through _uv_safe_path and fix temp-dir leak (#6503) The -c/-r fix did not cover UV_OVERRIDE, which uv also truncates at the first space. On Apple Silicon the overrides file is handed to uv via UV_OVERRIDE at install time (install_python_stack.py) and during the MLX self-heal (utils.mlx_repair), so a repo under a path containing a space still broke every uv call there. Move _uv_safe_path into backend.utils.uv_path_safety so both sites share it, and route UV_OVERRIDE through it. Also stop leaking the temp dir when shutil.copyfile fails after mkdtemp, and add tests for the UV_OVERRIDE channel, the TMPDIR-with-space fallback, the atexit cleanup, and the no-leak path. --------- Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-06-24 04:02:24 -07:00
oobabooga	1cc785e5a0	Studio: remove OpenEnv and other unused packages (#6585) ... * Studio: drop OpenEnv and unused ExecuTorch/open_spiel install deps * Studio: drop 8 more unused install deps from extras * Studio: restore tomli<3.11 for kernels; tidy dep-cleanup comments and tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: refresh scan-packages baseline for scipy _external + unsloth-zoo tests scipy moved its vendored array_api_compat from scipy/_lib to scipy/_external, so the four allowlisted array_api_compat __init__.py entries stopped matching and resurfaced as unsuppressed CRITICAL "Downloads and executes remote code" findings on all three pip scan-packages shards (extras, hf-stack, studio). Add the _external paths next to the existing _lib ones so both scipy layouts stay covered. Allowlist two unsloth-zoo test-file false positives now present in the hf-stack shard: tests/test_mlx_save_export_regressions.py (writes to /tmp dropper) and tests/test_mlx_trainer_internals.py (obfuscation plus exec/eval). Drop nine stale entries for packages removed from the Studio requirements and no longer in any shard closure (evaluate, pytest, hypothesis, kgb, langid), confirmed absent via with-deps resolution of all three shards. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-06-23 07:20:47 -07:00
Daniel Han	935f6c50ef	studio: tighten torchao Windows-ROCm comments and test docstrings (#6610)	2026-06-23 05:49:25 -07:00
Daniel Han	55c392ff7c	studio: fix sentence-transformers RAG embedder on Windows ROCm (torchao) (#6608) ... torchao has no working Windows ROCm build. transformers.quantizers imports it, and it loads torch's c10d distributed backend at module level, which the AMD Windows wheels omit (no RCCL). The import aborts, transformers can no longer expose PreTrainedModel, and the sentence-transformers embedder silently falls back to the llama-server GGUF embedder. Linux ROCm and NVIDIA are unaffected (the c10d ops are present / torchao is real there). The training and export workers already install the shared torchao stub before importing transformers, but the RAG embedder runs in the main backend process, which never did. Two fixes, both no-ops off Windows ROCm: - embeddings.py: install_torchao_windows_rocm_stub() before the first sentence-transformers import, so an already-installed torchao is neutralized (fixes existing venvs). - install_python_stack.py: stop installing torchao on Windows ROCm; it can only crash on import there, so new venvs never ship it. Add tests covering the embedder stub call and the install skip.	2026-06-23 05:39:02 -07:00
Daniel Han	e83d4ae072	Windows installer: fix DiskPart UAC mid-install, drive-root cache, and spurious unsloth.exe rename warning (#6296) ... * Windows installer: fix DiskPart UAC, drive-root cache, spurious rename warning, CPU-base messaging amd-smi gate (DiskPart UAC mid-install): the AMD torch wheel ships hipInfo.exe inside the venv, and the bitsandbytes fix prepends that venv Scripts dir to PATH. shutil.which("hipinfo") then found it and flipped _amd_smi_allowed() to True, so the post-install AMD probe fell through to `amd-smi list` (the venv hipInfo failed to report gcnArchName, which is why the arch came from the GPU-name table) and amd-smi elevated, popping the DiskPart UAC. Fix: a hipinfo resolved inside the active venv (sys.prefix) is the torch-wheel binary, not a HIP SDK, and must not open the gate. Mirrored in install_python_stack.py, install_llama_prebuilt.py, and backend utils/hardware/amd.py (the runtime VRAM poller had the same latent prompt). TORCHINDUCTOR_CACHE_DIR: move from C:\tc to <StudioHome>\TORCHINDUCTOR_CACHE_DIR so the inductor/Triton cache lives under the user's Studio home, not the system drive root. Long paths are already enabled above so deep inductor paths still fit. unsloth.exe rename: skip the rename (and its "pip may fail with WinError 32" warning) when SKIP_STUDIO_BASE=1. In the install.ps1 flow base packages are not reinstalled, so unsloth.exe is never rewritten; the self-rename only failed because setup runs via unsloth.exe (the running launcher holds its own file). The 'studio update' flow still attempts it. CPU PyTorch messaging: clarify that the CPU base is temporary and setup replaces it with GPU ROCm wheels, and print an explicit "GPU ROCm PyTorch installed" line after the AMD wheels land, so the log makes clear the final install is GPU-accelerated. Adds two regression tests covering the venv-internal vs external hipInfo gate. Verified end-to-end on a Strix Halo box (Radeon 8060S / gfx1151): install.ps1 --local from this branch completed exit 0 with no DiskPart prompt, no rename warning, the cache under the Studio home, and "GPU ROCm PyTorch installed (gfx1151)"; Studio then booted and detected "ROCm (HIP 7.13.99004) -- AMD Radeon 8060S Graphics". * Windows installer: drop the unreliable unsloth.exe rename and its WinError 32 warning setup.ps1 used to rename the running unsloth.exe out of the way before the base-package upgrade so pip could replace it. That rename never actually worked: setup runs *via* unsloth.exe, so renaming our own running uv-trampoline launcher failed with a sharing violation (WinError 32) and only printed a scary 'could not rename unsloth.exe; pip may fail with WinError 32' warning on every Windows install and update. It also was not needed. pip tolerates a running/locked console-script .exe: it moves the old one aside and writes the new one. The base upgrade routes through pip on Windows, so the upgrade succeeds (or, in the install.ps1 flow with SKIP_STUDIO_BASE=1, the base is not touched at all) and unsloth.exe is left intact either way. Removing the rename block and its failed-install restore block removes the false warning for all Windows devices in both the install and update flows. * Windows installer: gate venv-internal hipInfo.exe in PowerShell amd-smi probe; harden venv path checks Follow-up to PR #6296. - install.ps1 and setup.ps1: ignore the AMD torch wheel hipInfo.exe that lives inside the Studio venv when probing for a HIP SDK, so amd-smi no longer reopens the DiskPart UAC during install/update. Mirrors _path_inside_venv in the Python installers, which already do this. - amd.py, install_llama_prebuilt.py, install_python_stack.py: normcase the venv containment check (Windows paths are case-insensitive) and run the HIP_PATH/ROCM_PATH candidate through it too. - setup.ps1: fall back to a short TORCHINDUCTOR cache dir when long paths are unavailable, and create the dir wildcard-safely. - tests: isolate sys.prefix in the gate helper, add HIP_PATH/ROCM_PATH cases, and assert the PowerShell venv exclusion. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows installer: install ROCm PyTorch directly for a known AMD arch When the GPU arch is known (name-inferred from the GPU-name table) but ROCm could not be probe-verified (no HIP SDK, no amd-smi), the bootstrap installed a CPU PyTorch base that setup.ps1 then force-reinstalled as ROCm. The repo.amd.com wheels bundle their own runtime (no HIP SDK required), which setup.ps1 already relies on, so the CPU base was a pure wasted download/install. - Gate the ROCm index on a known arch, not only on probe-verified ROCm, so a mapped arch installs ROCm torch directly. Unmapped arches and no-GPU hosts still get CPU (unchanged). - Fall back to a CPU base if the ROCm-index install fails, so a transient repo.amd.com outage does not abort the install (setup.ps1 retries ROCm). - Correct the stale comment that claimed ROCm wheels need a confirmed HIP SDK. - Add a regression test for the arch-based gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows installer: correct the unsloth.exe rename-removal comment The comment claimed the base upgrade 'routes through pip on Windows' and that pip 'moves the old unsloth.exe aside, then writes the new one'. That is not what the code does. install_python_stack tries uv first; on a locked launcher uv aborts and falls back to pip, but the pip fallback strips --upgrade-package and base.txt lists only bare unsloth/unsloth-zoo, so pip finds them already satisfied and no-ops. The running unsloth.exe is left intact at its current version either way. Reword the comment to describe the real uv-first / pip-fallback-no-op behavior. No functional change. * Windows installer: close two gaps in the venv-internal hipinfo exclusion Review follow-up. The amd-smi/DiskPart gate could still reopen in two cases: - setup.ps1 ran the HIP probe long before $VenvDir is assigned, so without VIRTUAL_ENV (the `unsloth studio update` path) $venvRoots was empty and the venv-internal hipInfo.exe was not recognized. Seed the venv root from UNSLOTH_SETUP_PYTHON and the default Studio home too (both installers). - The HIP_PATH/ROCM_PATH candidate was accepted without the venv filter, so an env var pointing into the venv (AMD wheel) still set $HipSdkInstalled. Run Test-HipinfoIsVenvInternal on the candidate as well (both installers). Extend the PS gate test to assert both. Both .ps1 parse clean; install tests pass (the venv-internal / HIP probe coverage at 359 passed). * Windows installer: correct the CPU-base message for arches with no ROCm wheels After gating the ROCm index on a known arch, a mapped arch sets $ROCmIndexUrl and installs ROCm directly, so it no longer reaches the "temporary CPU base" branch. That branch is now reached only by a name-inferred arch with no ROCm wheels (e.g. RDNA2 gfx103X), where setup.ps1 does NOT install ROCm. The old text ("setup replaces it with GPU ROCm wheels ... the final install IS GPU-accelerated") was therefore always wrong there. Say plainly that PyTorch stays on CPU for this GPU. * Windows installer: seed the venv-internal hipInfo check from a custom Studio home Test-HipinfoIsVenvInternal seeded the venv root from VIRTUAL_ENV, VenvDir, the setup python, and the default %USERPROFILE% path only. A standalone `unsloth studio update` with a custom UNSLOTH_STUDIO_HOME (or STUDIO_HOME alias) and none of those set would not recognize the venv hipInfo on PATH, reopening the amd-smi/DiskPart gate. Seed the custom home too, in both installers, and assert it in the gate test. * Studio installer: resolve venv aliases and expand ~ in the hipInfo venv filter Two review points on the amd-smi/DiskPart UAC gate: 1. _path_inside_venv compared os.path.abspath of sys.prefix and the hipInfo path, which does not resolve symlinks, junctions, or 8.3 short names. A venv reached through an aliased path then fails the check, so its bundled hipInfo.exe is mistaken for an external HIP SDK and amd-smi runs (the DiskPart prompt this fix exists to suppress). Switch to os.path.realpath in all three copies (amd.py, install_llama_prebuilt.py, install_python_stack.py). 2. setup.ps1's early venv-internal hipInfo probe seeded the venv root from a custom Studio home (UNSLOTH_STUDIO_HOME / STUDIO_HOME) without expanding a leading ~, while the canonical resolver does. With a tilde form, [IO.Path]::GetFullPath kept the literal ~ relative to cwd, so the custom-home hipInfo escaped the filter and reopened the gate. Expand ~ in the probe the same way as the resolver. tests/studio/install/test_pr5940_followups.py: 30 passed (adds a symlink realpath case and a setup.ps1 tilde-expansion guard). * Studio installer: mirror the hipInfo venv filter and ROCm wheel pins into install.ps1 Follow-up review on the same install.ps1 paths: 1. install.ps1's venv-internal hipInfo probe (Test-HipinfoIsVenvInternal) seeded the venv root from a custom Studio home without expanding a leading ~, unlike the canonical resolver and setup.ps1. A tilde form left [IO.Path]::GetFullPath with the literal ~ (relative to cwd), so the custom-home hipInfo escaped the filter and reopened the amd-smi/DiskPart gate. Expand ~ in the probe, matching the setup.ps1 fix. 2. The AMD ROCm path installed torchvision/torchaudio bare while pinning torch to below 2.12. AMD's per-arch index publishes the companions independently and may ship torchvision 0.27 (for torch 2.12) before removing 0.26, so a bare resolve can pick an ABI-incompatible set and fall back to CPU. Add torchvision/torchaudio floor maps and pass the pinned specs, mirroring setup.ps1 and install_python_stack.py. 3. The ROCm-to-CPU fallback torch install used Invoke-InstallCommand (no retry), the only torch step in the file without it. Switch to Invoke-InstallCommandRetry so the recovery path survives a transient index failure. tests/studio/install/test_pr5940_followups.py: 33 passed (parametrized tilde check over both installers, a torch/companion floor-map parity test, and a CPU-fallback retry guard). * Studio installer: scan all PATH hipinfo so the venv copy can't shadow a real HIP SDK The amd-smi HIP-SDK probe used shutil.which("hipinfo") / Get-Command hipinfo, which return only the first hit on PATH. The AMD torch wheel ships hipInfo.exe inside the venv and the bnb fix (plus the Studio backend) prepend the venv Scripts dir to PATH, so that venv-internal copy lands first. When a real HIP SDK hipinfo sits later on PATH with HIP_PATH/ROCM_PATH unset, the first-hit probe stopped at the venv copy, treated it as "not a HIP SDK", and closed the amd-smi gate -- AMD users in that PATH-only SDK setup lost amd-smi telemetry and could fall back to CPU. Scan every PATH entry and keep the first hipinfo that is not venv-internal; only the venv copy is ignored, so the UAC/DiskPart suppression is unchanged. Applied to all three Python copies (install_llama_prebuilt.py, install_python_stack.py, backend/utils/hardware/amd.py) via a new _external_hipinfo_on_path helper, and both PowerShell callers (install.ps1, setup.ps1) now use Get-Command hipinfo -All filtered by Test-HipinfoIsVenvInternal. tests/studio/install/test_pr5940_followups.py: 36 passed (real-PATH scan tests, a shadow-regression test for the exact venv-first ordering, and a parity check that every Python copy uses the scanning helper). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio uninstallers: fix leftovers (false "removed", shared icon, llama lock) Auditing a dual native+WSL uninstall on a real device surfaced three leftovers: 1. uninstall.ps1 removed the data dir (which holds unsloth.ico) before the shortcuts that reference that icon, so Explorer's icon cache briefly held it open. Remove-Item -Recurse reported success yet left the locked file, and the dir was never re-attempted, so it orphaned with a false "removed" log. _RemovePath now verifies the path is actually gone (retrying transient locks) and reports honestly, and the data dir is re-swept after the shortcuts go. 2. install.sh writes a shared unsloth.ico to %LOCALAPPDATA%\Unsloth Studio for the WSL shortcut, but uninstall.sh never removed it, orphaning the icon (and dir) after a WSL uninstall. uninstall.sh now drops that icon and the dir when empty, in both the powershell.exe and drvfs-fallback paths. 3. ~/.unsloth/.llama.cpp.install.lock was never removed, so the rmdir of ~/.unsloth failed and the dir lingered. Both uninstallers now remove the lock. Verified by running both uninstallers on a real dual install: device fully clean (no install dirs, shortcuts, PATH/registry entries, shared icon, or lock left). * install.sh: auto-route Strix Halo WSL to an existing Ubuntu 24.04 ROCm-on-WSL is the GPU runtime for Strix Halo and only targets Ubuntu 24.04. When the installer runs in a newer default distro (e.g. 26.04) it cannot enable the GPU and silently falls back to CPU. If a 24.04 distro already exists, re-run the install there and stop in the current one so the GPU path is taken without the user having to know about the distro requirement. Runs before venv creation so the wrong distro is left untouched, guards against re-route loops via UNSLOTH_WSL_REROUTED, leaves a working ROCm distro alone (librocdxg present), and skips the GGUF-only / opt-out / non-Strix cases. When no 24.04 distro exists we keep today's behaviour: continue to CPU and print the `wsl --install Ubuntu-24.04` guidance, never auto-downloading a distro. Adds tests/sh/test_strixhalo_wsl_reroute.sh (hermetic: extracts the function, rewrites its paths to fixtures, mocks wsl.exe) covering the full decision matrix, wired into tests/run_all.sh. * uninstall.ps1: keep shared unsloth.ico for a surviving WSL shortcut A dual native+WSL install shares %LOCALAPPDATA%\Unsloth Studio\unsloth.ico: install.sh points the WSL shortcut's icon there while the native install owns the dir. The native uninstaller removed the whole dir unconditionally, so uninstalling native while keeping WSL left the WSL shortcut with a blank icon. The old code only avoided this when Explorer happened to hold the icon open, which is unreliable; on a real dual install the dir was deleted and the WSL shortcut went blank. _RemoveDataDirKeepingWslIcon now scans the Start Menu + Desktop for a surviving "Unsloth Studio (WSL ...).lnk" and, if found, removes everything in the data dir except unsloth.ico (keeping the dir) instead of deleting it; with no WSL shortcut it removes the dir as before. uninstall.sh still drops the icon and the empty dir when WSL itself is uninstalled, so every uninstall order ends clean. Adds tests/studio/test_uninstall_dual_install_icon.ps1 (AST-extracts the helper and runs it against a temp dir with controlled shortcut dirs) covering the dual, native-only, empty, and missing-dir cases, wired into the windows-inference smoke workflow. Verified on a real dual install: native uninstall now keeps unsloth.ico and the WSL shortcut's icon stays intact. * installer: condense AMD/ROCm code comments (no behavior change) Tighten the comments added for the Strix Halo native+WSL installer work so they are shorter and clearer without losing intent: the venv-internal hipInfo amd-smi gate, the ROCm torch/companion floor maps, the WSL 24.04 reroute, and the dual-install uninstall icon handling. Comment-only; code paths unchanged. 107 insertions, 166 deletions across 11 files. * install.sh: run the Strix Halo WSL reroute before any STUDIO_HOME write The reroute fired after mkdir -p "$STUDIO_HOME" and the legacy-venv migration, so rerouting 26.04 -> 24.04 left an empty ~/.unsloth/studio stub in the origin distro (and ran venv migration in the distro about to be abandoned). Move the reroute ahead of the venv section so the origin distro is left untouched, matching the function's own comment. Behavior is identical on every non-reroute path. * installer: fix ROCm CPU-fallback, hipinfo gate edge cases, uninstall icon, WSL 22.04 - install.ps1: clear $ROCmIndexUrl/$ROCmTorchFloor after the CPU fallback so the flavor-repair block does not retry the failed ROCm index and abort the install; pin the ROCm companion specs ($visionSpec/$audioSpec) in the repair path too. - install.ps1 + setup.ps1: skip a bare drive root in Test-HipinfoIsVenvInternal so a non-venv UNSLOTH_SETUP_PYTHON does not match the whole drive; iterate HIP_PATH/HIP_PATH_57/ROCM_PATH and take the first non-venv hipinfo. - amd.py, install_llama_prebuilt.py, install_python_stack.py: strip surrounding quotes from PATH entries before probing for hipinfo. - install.sh: pipefail the WSL reroute curl|sh; do not reroute supported Ubuntu 22.04. - uninstall.sh: keep the shared unsloth.ico while any Unsloth shortcut (native or another WSL distro) still references it, in both the powershell and drvfs paths. - tests: regression coverage for all of the above. * installer: forward reroute options, guard ROCm bootstrap, harden hipinfo gate - install.sh: forward the caller's --package/--python/--verbose/--tauri and a custom UNSLOTH_STUDIO_HOME into the WSL reroute (was a bare default install); bail on --local; run the reroute BEFORE dependency/uv install so the origin distro is left untouched; set UNSLOTH_SKIP_ROCM_WSL_SETUP after a failed reroute so the later ROCm-on-WSL bootstrap does not install into the unsupported origin distro. - install.ps1 + setup.ps1: Get-Command hipinfo -CommandType Application so only real executables match (not an alias/function named hipinfo). - uninstall.ps1: guard $env:APPDATA when building the default shortcut search dirs. - tests: cover option forwarding, --local bail, the bootstrap guard, and the gate change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * installer: guard origin ROCm bootstrap on every CPU-only fallback; harden ~ expansion WSL reroute: the no-wsl.exe, no-24.04-target and --local fallbacks all tell the user the install continues CPU-only, but only the failed-reroute branch set UNSLOTH_SKIP_ROCM_WSL_SETUP=1. The later _maybe_bootstrap_rocm_wsl gate keys off that flag, so the other three branches could still install ROCm into the unsupported origin distro (e.g. 26.04). Set the skip guard on all of them. Forward UNSLOTH_ROCM_WSL_AUTO into the reroute so a Tauri/consented GPU bootstrap carries through to the rerouted 24.04 child instead of dropping to the prompt path. install.ps1/setup.ps1: guard the venv-probe ~ expansion on a non-empty $env:USERPROFILE so Join-Path does not throw on a profile-less service account. Tests: add no-wsl.exe and UNSLOTH_ROCM_WSL_AUTO reroute cases, the USERPROFILE guard assertion, and route shell-test fixtures through a single trap-cleaned root. * installer: pin + soften Windows ROCm Python repair, reroute to 22.04, harden gates install_python_stack.py: the Windows AMD ROCm repair in _ensure_rocm_torch() installed bare torch/torchvision/torchaudio via the fatal pip_install -- the same asymmetry already fixed on the PowerShell side. A transient repo.amd.com failure could abort the whole install even after install.ps1/setup.ps1 fell back to CPU. Pin companions per-arch (gfx120X/Strix -> the rocm7.2 trio, mirroring the PS floor maps) and make the retry nonfatal: keep the existing build and let the user re-run update to retry ROCm, so the chain install.ps1 -> setup.ps1 -> stack stays CPU-safe. install.sh: reroute now targets an installed Ubuntu 24.04 OR 22.04 (24.04 preferred); both are AMD-supported for ROCm-on-WSL, matching the leave-alone set, so a box with only 22.04 reaches the GPU instead of staying CPU-only. install.ps1/setup.ps1: a bare ~ for UNSLOTH_STUDIO_HOME left an empty Join-Path child (PS 5.1 throws); fall back to USERPROFILE directly and only join a real remainder. _path_inside_venv (amd.py + both installers): guard a root-dir sys.prefix so commonpath can't classify every path on the drive as venv-internal (defensive; venv never at root). uninstall.sh: guard an empty LOCALAPPDATA in the PS-interop icon cleanup (mirror APPDATA). Tests: add 22.04-target reroute cases, Windows ROCm pin+nonfatal coverage (text + behavioral), root-dir guard coverage, and bare-~/LOCALAPPDATA guard assertions. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install.sh: match WSL reroute target by exact distro name, not substring The 24.04/22.04 reroute target was chosen with grep -F (substring), so a custom distro such as 'Ubuntu-24.04-test' (with no exact Ubuntu-24.04) was picked as the target; the later 'wsl -d Ubuntu-24.04' then fails and the Strix Halo install stays CPU-only. Match whole lines (grep -ixF) and reuse the matched name so only a real Ubuntu-24.04/22.04 is targeted. Adds substring-rejection + exact-vs-custom tests. * install.sh: keep the WSL reroute target to Ubuntu 24.04 (helper-supported only) The ROCm-on-WSL bootstrap (scripts/install_rocm_wsl_strixhalo.sh) dies on any VERSION_ID other than 24.04 and pins the noble repo, so treating 22.04 as GPU-supported let the parent report a successful reroute while the child fell back to CPU. Drop 22.04 from the supported set and the reroute target list; 24.04 stays the sole target (keeping the exact whole-line distro match). An already-working ROCm on any other version is still left alone by the librocdxg check above. tests: reroute 22.04 cases updated to the 24.04-only behavior; make the "no wsl.exe" case hermetic so a real host wsl.exe can't leak in on dev boxes; stop the tauri exit-order check from mis-flagging the reroute helper's [ "$TAURI_MODE" = true ] && ... --tauri one-liner. * installer: tighten comment wording across the Strix Halo install/uninstall paths Condense the verbose multi-line comment blocks (amd-smi hipinfo gate, ROCm torch install + CPU fallback, WSL reroute, uninstall icon-keep) into fewer, clearer lines. Comments and a few docstrings only; no code, logic, or behavior change. Verified with bash -n, the PowerShell parser, and ast.parse, and the installer test suite still passes. * add AGPL-3.0 SPDX headers to the .sh/.ps1 scripts missing them Every shell and PowerShell script under the Studio/installer surface now carries the standard SPDX-License-Identifier: AGPL-3.0-only + copyright header (after the shebang where present): the installer (install.sh, install.ps1), build.sh, the .github and src-tauri scripts, the installer test suite, and the moe kernel test. Header-only, line endings preserved; bash -n, the PowerShell parser, and the installer tests all pass. * installer: drop the duplicate AGPL header from install.sh and install.ps1 Both already carry an SPDX-License-Identifier: AGPL-3.0-only header below their usage comment block; the prior header pass added a second one at the top because it only scanned the first few lines. Remove the duplicate so each file keeps a single original header. * installer: force-reinstall CPU fallback torch; propagate Tauri NEED_SUDO from reroute install.ps1/setup.ps1: when the AMD ROCm wheel install fails and we fall back to a CPU base, force-reinstall the torch/vision/audio triplet. A failed ROCm install can leave an unpinned ROCm torch (e.g. 2.10.0+rocm on gfx110X/gfx90a) that still satisfies the CPU torch>=2.4,<2.11.0 range, so without --force-reinstall uv keeps the ROCm build and only swaps the companions -- a mismatched venv the flavor-repair block won't fix. setup.ps1 scopes the forced reinstall to the ROCm-fallback path () so the genuine CPU-only install stays fast. install.sh: the Strix Halo WSL reroute treated every nonzero child exit as a reroute failure and fell back to CPU. In --tauri mode the child uses exit 2 ([TAURI:NEED_SUDO]) to ask the desktop app to elevate for the target distro; capture the child's exit code and propagate exit 2 in Tauri mode (the child already printed the NEED_SUDO line) instead of masking it. CLI mode still falls back to CPU on a generic failure. Tests: reroute Tauri exit-2 propagation (and non-Tauri CPU-fallback) cases; run_func now preserves the child exit code; force-reinstall assertions for both PowerShell installers. Note: codex's _rr_q apostrophe finding is a false positive -- the helper already emits POSIX-correct 'O'\''Brien' and round-trips under both sh and bash. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * setup.ps1: fix $cpuForce array collapse in the ROCm->CPU torch fallback An if-expression assignment ($cpuForce = if ($ROCmCpuFallback) { @("--force-reinstall") }) collapses the single-element array to a scalar string, so @cpuForce splatting enumerated it character-by-character into broken single-letter args (- - f o r c e ...), which made uv/pip reject the install and aborted the whole Studio setup on the AMD ROCm->CPU fallback path. Build $cpuForce as a real array assigned outside the if-expression so the splat passes a single --force-reinstall arg. Genuine CPU-only installs stay fast (empty array, no flag). Test now asserts the array-build form and rejects the if-expression form. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * uninstall: remove the isolated Node.js runtime (~/.unsloth/node) The isolated Node.js runtime (install_node_prebuilt.py, added with the managed-Node change) installs to ~/.unsloth/node in default mode -- a sibling of studio, so deleting <studio> leaves it behind (~200MB orphaned after uninstall). Both uninstallers already remove the other default-mode siblings (llama.cpp/.cache/.staging); add node alongside them. uninstall.ps1 also adds it to the handle-lock sweep so a held node.exe can't block the delete. Env/custom mode nests node under the custom root, removed with that root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-22 03:09:08 -07:00
Lee Jackson	1eb15162d9	fix: clean up Studio warning log formatting (#6265) ... * feat: queue chat prompts during generation * fix: address prompt queue review edge cases * fix: harden queued prompt dispatch * fix: track queued prompt run state by thread * fix: preserve prompt queue ordering * fix: isolate prompt queue on new chat * fix: clean up Studio warning log formatting * Fix export log markup --------- Co-authored-by: wasimysaid <wasimysdev@gmail.com>	2026-06-19 05:06:03 -07:00
Daniel Han	f5f9e217c1	studio: select torchao version from the installed torch (#6400) ... * studio: select torchao version from the installed torch The Studio installer pins CUDA torch to torch>=2.4,<2.11 and its driver ladder selects the cu130 wheel index on recent NVIDIA drivers, so pip resolves torch 2.10.0. overrides.txt hard-pinned torchao==0.14.0, whose C++ extensions are built against torch 2.9.0, so torchao skipped its cpp kernels ("Skipping import of cpp extensions due to incompatible torch version 2.10.0+cu130 for torchao version 0.14.0") and fell back to the slow Python path. Every CUDA index now tops out at torch 2.10.0, so this hit most modern installs, not just cu130. Pick the torchao version matching the torch actually installed in the venv (table: pytorch/ao#2919): torch 2.10.x -> torchao 0.16.0, 2.11.x -> torchao 0.17.0, otherwise the previous 0.14.0 (so torch <=2.9 is unchanged). The installer reads torch.__version__ from the venv via a cross-platform sys.executable probe (probe_torch_wheel_env is Linux-only) and passes the computed spec positionally to the existing force-reinstall override step; overrides.txt becomes a pointer to that logic. torchao's Python API (Float8Tensor, used by unsloth/kernels/utils.py) imports cleanly on 0.16.0/0.17.0, verified against torch 2.9.1. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: address review on torchao selection - Clean the torch minor of pre-release/dev suffixes before parsing (e.g. '2.10rc1' -> minor 10), matching wheel_utils.probe_torch_wheel_env. - Pass _windows_hidden_subprocess_kwargs() to the torch-version probe so it does not flash a console window on Windows (no-op elsewhere). - Use _safe_print for the selection log line, consistent with the file's other status output (safe on non-UTF-8 consoles). --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-17 04:09:40 -07:00
Matt Van Horn	08c3878919	fix: use partial hipinfo output on crash to avoid CPU fallback (RDNA 4 / gfx1200) (#6292) ... * fix: use partial hipinfo output on crash to avoid CPU fallback (#6043) `hipinfo.exe` on some RDNA 4 hosts (e.g. RX 9060 XT / gfx1200) exits with STATUS_ACCESS_VIOLATION (0xC0000005) after printing the gcnArchName line. The previous guard `$LASTEXITCODE -eq 0` in studio/setup.ps1 and `if result.returncode == 0` in install_python_stack.py discarded this partial-but-valid output, causing the installer to fall through to WMI name inference which sets HasROCm=false and installs CPU PyTorch instead of the ROCm wheel. Fix: check for gcnArchName in stdout first; accept the arch regardless of exit code. Only fall through to the amd-smi / WMI path when no gcnArchName is present at all (crash before any output, or a genuine "no device" error). A cyan INFO substep is emitted when the arch is recovered from a crashed hipinfo run so users can see what happened. Adds a regression test covering the crash-with-valid-output path. Fixes #6043 * Fix/adjust hipinfo crash fallback for PR #6292 --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: wasimysaid <wasimysdev@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>	2026-06-15 13:26:04 +02:00
oobabooga	5300c047b6	Installer: drop the lemonade ROCm fallback now the fork ships identical per-gfx prebuilts (#6225) ... --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-06-12 11:53:26 -03:00
Daniel Han	2db9fad4b5	Installer: GPU detection follow-ups after #6174 (poisoned venv repair, llama.cpp routing, probe bounds) (#6183) ... * Installer: harden GPU detection follow-ups after #6174 Ports the NVIDIA-priority and /proc/driver/nvidia/gpus hardening from #6174 to the remaining pathways and adds recovery for already-poisoned venvs: - install_python_stack.py: add _ensure_cuda_torch so 'unsloth studio update' force-reinstalls CUDA torch when the venv carries a ROCm build on an NVIDIA Linux host (the pre-#6174 poisoning signature). Honors UNSLOTH_TORCH_BACKEND, UNSLOTH_ROCM_TORCH_INSTALLED, and CUDA_VISIBLE_DEVICES=-1/'' opt-outs; never touches healthy CUDA, deliberate CPU wheels, macOS, or Windows. - install_llama_prebuilt.py: detect_host gains the /proc NVIDIA fallback and skips ROCm probes when NVIDIA is usable; forwarded --rocm-gfx/--has-rocm overrides still win. - setup.sh: GPU summary classifies NVIDIA first through a timeout-bounded probe with the /proc fallback; AMD probes are bounded and gain a KFD vendor_id 4098 fallback; the llama.cpp source build only selects GGML_CUDA/GGML_HIP when the matching GPU is actually detected. - install.sh: bound both nvidia-smi calls with a 10s timeout (no behavior change when healthy or when the timeout binary is absent); classify the exported UNSLOTH_TORCH_BACKEND on the final index path segment so custom mirrors containing 'rocm'/'gfx' in their base path are not mislabeled. - install.ps1 + setup.ps1: NVIDIA probes now require a real 'GPU N:' row from nvidia-smi -L under a 10s bound instead of bare exit code 0; later CUDA version and compute_cap queries are bounded too. Tests: 3 new test files (50+ tests), suite at 788 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix Resolve-CudaToolkit driver probe for extracted-function unit test tests/studio/test_resolve_cuda_toolkit.ps1 extracts Resolve-CudaToolkit alone into a child pwsh and stubs nvidia-smi with a .ps1 script. The bounded runner is not in scope there (and ProcessStartInfo cannot dispatch .ps1 stubs), so the DriverMaxCuda parse silently returned nothing and the major-mismatch scenarios failed. Fall back to direct invocation when Invoke-NvidiaSmiBounded is unavailable; production setup.ps1 always has it defined and keeps the 10s bound. * Treat CUDA_VISIBLE_DEVICES empty or -1 as hidden in NVIDIA-first guards The NVIDIA-first guards added in this branch only special-cased CUDA_VISIBLE_DEVICES=-1 at two setup.sh gates and ignored the empty-string form entirely, while the Python detector (install_llama_prebuilt.py) already treats both as hidden. On a mixed AMD+NVIDIA host steered to the AMD card via CUDA_VISIBLE_DEVICES, the guards suppressed the AMD probes, so setup.sh fell to a CPU llama.cpp build and install.sh picked CUDA wheels instead of ROCm. Move the policy into the helpers so every consumer agrees: - install.sh: new _cvd_hides_nvidia checked first in _has_usable_nvidia_gpu - studio/setup.sh: same via _setup_cvd_hides_nvidia; the two ad-hoc CUDA_VISIBLE_DEVICES=-1 gate conditions are now redundant and removed - studio/install_python_stack.py: _has_usable_nvidia_gpu returns False when CUDA_VISIBLE_DEVICES is set to or -1 (whitespace tolerated) Tests: 5 new sh scenarios (hidden via , -1, padded -1, visible device, and mixed host with hidden NVIDIA restoring the ROCm route) plus a pytest class covering all three implementations behaviourally. Addresses the review comment on the NVIDIA-first setup.sh block. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Retrigger CI after PyPI 503 outage during the previous run --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-11 05:06:02 -07:00
Leo Borcherding	bf2cd745b1	Fix installer selecting ROCm torch on NVIDIA Linux hosts (#6174) ... * fix: prevent ROCm torch from installing on NVIDIA Linux hosts NVIDIA's open kernel module (driver 560+) registers GPU topology nodes in the KFD sysfs hierarchy with non-zero gpu_id values. The _has_amd_rocm_gpu (install.sh) and _has_rocm_gpu (install_python_stack.py) sysfs fallbacks previously treated any non-zero gpu_id as proof of an AMD GPU, so an NVIDIA-only host with the open kernel driver was misrouted to the ROCm install path, replacing the correctly-installed CUDA torch with ROCm wheels. Fixes: 1. install.sh _has_amd_rocm_gpu sysfs fallback: require vendor_id 4098 (AMD 0x1002) in the KFD node properties file before declaring an AMD GPU present. NVIDIA KFD nodes carry vendor_id 4318 (0x10DE) and are now skipped. 2. install_python_stack.py _has_rocm_gpu sysfs fallback: same vendor_id guard. Also preserves the existing fallback for older kernels that don't ship a properties file (trusts gpu_id alone there). 3. install.sh now exports UNSLOTH_TORCH_BACKEND ("cuda"/"rocm"/"cpu") immediately after get_torch_index_url() resolves the wheel family. install_python_stack.py reads this as _TORCH_BACKEND and short-circuits _ensure_rocm_torch() entirely on cuda/cpu hosts, providing a second layer of defense that is independent of subprocess GPU detection. Tests: 9 new cases in TestHasRocmGpuKfdVendorGuard, TestEnsureRocmTorch, and TestInstallShStructure cover all three changes. Full test_rocm_support.py suite: 289 passed, 2 skipped, 0 failed. Closes #6172 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: show actual torch backend in progress step labels The 'ROCm torch check' and 'ROCm torch (final)' step labels were hardcoded regardless of whether the installer was targeting CUDA, ROCm, or CPU. On NVIDIA hosts they showed 'ROCm' even though no ROCm wheels were being installed, which was misleading. Add _torch_step_label(suffix) which reads UNSLOTH_TORCH_BACKEND (set by install.sh) and formats the label as e.g. 'torch check (cuda)' or 'torch final (rocm)'. Falls back to live GPU detection for standalone studio update runs that bypass install.sh. * fix: make KFD sysfs vendor check conservative -- skip if no properties file The previous implementation fell through to `return True` when the KFD node's properties file was missing (OSError), intending to support older kernels. But NVIDIA open driver KFD nodes can also lack a properties file on some kernel versions, so the fallback still produced a false positive. Change the `except OSError: pass` to `continue` so any node without a readable properties file is skipped rather than trusted. KFD properties files exist on every kernel version that actually exposes /sys/class/kfd, so this does not regress real AMD GPU detection -- if the directory exists at all, properties files will be present for genuine GPU nodes. * fix: bulletproof NVIDIA vs AMD GPU detection Four changes that together ensure ROCm torch can never be installed on an NVIDIA host regardless of which detection path fires: 1. _has_rocm_gpu() (Python): NVIDIA guard at the top -- returns False immediately when _has_usable_nvidia_gpu() is True, blocking rocminfo, amd-smi, and KFD sysfs from producing a false positive even when ROCm tools are co-installed alongside the NVIDIA driver. 2. _has_amd_rocm_gpu() (install.sh): same NVIDIA guard -- calls _has_usable_nvidia_gpu first and returns 1 if it succeeds. 3. _has_usable_nvidia_gpu() (Python): adds /proc/driver/nvidia/gpus/ sysfs fallback. The NVIDIA driver populates this directory on Linux regardless of nvidia-smi state, so a subprocess PATH gap, timeout, or driver initialisation race can no longer silence NVIDIA detection. 4. _has_usable_nvidia_gpu() (install.sh): same /proc/driver/nvidia/gpus fallback, tried after nvidia-smi -L rather than instead of it. Together: NVIDIA wins at every decision point. If nvidia-smi works, it confirms NVIDIA. If it fails, /proc/driver/nvidia confirms NVIDIA. If somehow both fail, _has_rocm_gpu still checks NVIDIA first before any AMD path runs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: two KFD/proc-only corner cases from Codex review 1. KFD awk state not reset per node file (Ryzen+NVIDIA false positive): The awk glob processes all topology node properties files in one pass. Without FNR==1 reset, a Ryzen+NVIDIA host where an AMD CPU-agent node sets amd=1 (vendor_id 4098, gpu_id 0) can combine with a later NVIDIA node setting gpu=1 (gpu_id > 0), triggering found=1 before vendor_id 4318 is seen. Added FNR==1{ gpu=0; amd=0 } to reset per file. 2. proc-only NVIDIA not reaching CUDA wheel selection: _has_usable_nvidia_gpu returning true via /proc/driver/nvidia fallback left _smi empty, so get_torch_index_url entered the AMD/CPU branch and selected CPU wheels despite NVIDIA being confirmed. Introduced _nvidia_detected flag (separate from _smi) so the AMD branch is skipped whenever NVIDIA is confirmed by any path, while _cuda_ver reads from _smi when available (with the existing cu126 fallback when _smi is absent). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-10 21:00:11 -07:00
Daniel Han	265c9f5db4	Fix UnboundLocalError in ROCm version detection dpkg/rpm fallback (#6149) ... * Fix UnboundLocalError in _detect_rocm_version dpkg/rpm fallback A leftover local import re inside the amd-smi branch made re function local for the whole scope. When amd-smi and hipconfig are absent and dpkg-query or rpm reports rocm-core, the epoch strip at the dpkg/rpm fallback hit re.sub before any local binding existed and crashed the installer with UnboundLocalError. Drop the local import (the module already imports re at top level) and add a regression test covering the dpkg path without hipconfig. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-10 08:10:55 -07:00
Bubu	8af9fe63a3	fix: persist Windows ROCm BNB version (#6048) ... * fix: persist Windows ROCm BNB version * style: apply kwarg spacing hook * fix: avoid persisting caller ROCm overrides * fix: redetect managed BNB ROCm defaults * style: apply ROCm guard test formatting --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-06-10 06:01:05 -07:00
Daniel Han	62191c4765	Windows/WSL installer: fix winget msstore cert failure, amd-smi DiskPart prompt, and enable AMD GPU (Strix Halo gfx1151) (#5940) ... * Fix Windows installer winget msstore certificate failure `winget install` was invoked without `--source winget`, so winget also queried the msstore source. When msstore fails certificate pinning (error 0x8a15005e, "The server certificate did not match any of the expected values") winget aborts and demands `--source`, so the Python (and uv) install fails even though the package exists in the winget source. - Pass `--source winget` to all winget install calls (Python x2, uv). Both packages live in the winget source, so this is strictly correct and skips the failing msstore round-trip entirely. - Add a python.org fallback (Install-PythonFromPythonOrg) that downloads the official installer and runs it silently per-user (no admin/UAC) when winget is unavailable or fails for any reason. Mirrors the existing uv -> astral.sh fallback so Python installs without manual steps. Resolves the latest 3.13.x from python.org with a pinned fallback, and selects the amd64/arm64/x86 installer per architecture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Pin remaining setup.ps1 winget calls to --source winget Two winget invocations in studio/setup.ps1 still queried all sources and could hit the same msstore certificate-pinning failure (0x8a15005e) that broke the Python install in install.ps1: - `winget show Nvidia.CUDA --versions` (CUDA Toolkit version probe) - `winget install ... ShiningLight.OpenSSL.Dev` (OpenSSL dev for llama-server) Every other winget call in this file already passes `--source winget` (Git, CMake, VS Build Tools, CUDA install, Node.js, and setup.ps1's own Python 3.12 install), so these two were stragglers. Both packages live in the winget source; pinning it makes setup robust to an unhealthy msstore source, matching the rest of the file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Stop amd-smi GPU probe from popping a DiskPart UAC prompt On Windows, AMD GPU detection in install.ps1 and studio/setup.ps1 runs `amd-smi list` / `static --asic` / `version`. amd-smi (shipped in System32 by the Adrenalin driver) auto-elevates to read GPU/APU memory details, surfacing a confusing DiskPart UAC prompt mid-install. The Studio backend already documents and circuit-breaks on this in studio/backend/utils/hardware/amd.py, but the installers did not. Add an Invoke-AmdSmiNoElevate helper (both scripts) that runs amd-smi via Start-Process under __COMPAT_LAYER=RunAsInvoker so it cannot auto-elevate (no prompt), with a 30s timeout (matching amd.py) so a flaky amd-smi cannot stall the install for minutes. On failure/timeout the existing WMI name -> gfx fallback still resolves the arch, so detection is unchanged on working hosts. Verified on a Strix Halo (Radeon 8060S / gfx1151) box: the prompt is gone and the probe is bounded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Add experimental ROCm-on-WSL setup helper for Strix Halo (gfx1151) install.sh already routes gfx1151 (Radeon 8060S / Strix Halo) to the repo.amd.com/rocm/whl/gfx1151 wheels once a ROCm runtime is present, but it does not install AMD's driver/ROCm stack -- a large, admin-gated prerequisite. scripts/install_rocm_wsl_strixhalo.sh automates the Linux side on a dedicated Ubuntu 24.04 WSL2 distro: ROCm 7.2 (wsl usecase), the rocr4wsl HSA runtime, a librocdxg build, env setup, and a PyTorch gfx1151 GPU smoke test. A hard preflight refuses to run until the Adrenalin >=26.3.1 driver is actually present, so it cannot half-install. Procedure adapted from AMD's ROCm-on-WSL docs and community gfx1151 notes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Detect AMD GPUs by name so native Windows gets a GPU llama.cpp The gfx-arch inference from the WMI GPU name was gated behind $HasROCm, which the hipinfo/amd-smi probe leaves false on the common Windows case (Adrenalin driver only, no HIP SDK -- and amd-smi often cannot read the arch without elevation). So an AMD GPU was detected by name but never mapped to a gfx target, --rocm-gfx was not forwarded, and studio setup fell back to a CPU llama.cpp build. Un-gate the inference (install.ps1 + studio/setup.ps1) so it runs whenever an AMD GPU name is available. The inferred gfx is forwarded as --rocm-gfx, which makes install_llama_prebuilt.py download the matching lemonade-sdk ROCm prebuilt (e.g. llama-bNNNN-windows-rocm-gfx1151-x64.zip) -- a GPU-accelerated llama.cpp that bundles its own ROCm runtime, so it runs with just the Adrenalin driver. PyTorch's ROCm wheels still require a confirmed HIP SDK ($HasROCm), so this only affects llama.cpp / inference and never pulls broken ROCm torch. Also broaden the name->arch table to every family lemonade ships Windows assets for: gfx120X (RDNA 4), gfx110X (RDNA 3), gfx1151/gfx1150 (RDNA 3.5), and gfx103X (RDNA 2). Unknown names still fall back to CPU. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Suppress amd-smi DiskPart UAC prompt in the Python install/runtime paths The earlier PowerShell guard covered install.ps1 / setup.ps1, but the Python installer (install_llama_prebuilt.py detect_host, install_python_stack.py ROCm probes) and the Studio backend monitor (amd.py) also shell out to amd-smi on Windows, where it auto-elevates and pops the same DiskPart UAC prompt mid-install / at runtime. Inject __COMPAT_LAYER=RunAsInvoker into the amd-smi subprocess env on Windows so it runs un-elevated (no prompt). Callers already tolerate an empty/failed result and fall back to WMI / name detection (installer) or the existing circuit breaker (amd.py). Gated to Windows so Linux/macOS amd-smi behaviour is unchanged. - install_llama_prebuilt.py: handled centrally in run_capture (covers detect_host's `amd-smi list` and the version probe). - install_python_stack.py: new _amd_smi_env() helper on its 3 raw subprocess.run amd-smi calls. - amd.py: merge RunAsInvoker into the existing child env. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Tighten AMD GPU name->arch patterns to avoid mismatches The W9[0-9]{3} and RX 90[0-9]{2} patterns added for RDNA 4 were speculative and over-broad: W9xxx would also match old GCN FirePro W9100/W9000 cards (wrong gfx1201 -> a lemonade gfx120X download that fails validation), and RX 90[0-9]{2} was redundant with the explicit 9070/9060 entries. Drop both; keep only confirmed RDNA 4 SKUs. Unmatched AMD names still fall back cleanly to CPU. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Fetch the llama.cpp validation model via huggingface_hub The prebuilt validation downloads a tiny GGUF test model from huggingface via bare urllib. On Windows / proxy setups where the server sends an incomplete TLS chain, urllib cannot complete the Amazon CA chain (it does no AIA intermediate fetching) and fails with CERTIFICATE_VERIFY_FAILED, so a perfectly good GPU prebuilt is rejected and the installer falls back to a CPU source build. Route the validation-model download through huggingface_hub (hf_hub_download) -- the same mechanism Studio uses for model downloads, which completes the chain where urllib cannot -- keeping the direct URL as a fallback. This lets the lemonade ROCm prebuilt validate and install on cert-restricted machines (verified: hf_hub_download succeeds where urllib returns CERTIFICATE_VERIFY_FAILED). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Guard the remaining raw amd-smi version probe via run_capture A ROCm-version detector in install_llama_prebuilt.py called amd-smi version through a raw subprocess.run that bypassed run_capture's Windows RunAsInvoker guard, so it still triggered the DiskPart UAC prompt during setup. Route it through run_capture like the other amd-smi calls. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Forward --rocm-gfx even when the ROCm runtime is unconfirmed setup.ps1 forwarded --rocm-gfx (and picked the windows-hip llama.cpp prebuilt) only inside `if ($HasROCm)`. On Adrenalin-only hosts (amd-smi present but no HIP SDK, so $HasROCm stays false) the gfx arch was name-inferred but never forwarded, so install_llama_prebuilt.py saw has_rocm=False and installed the CPU build -- even though the lemonade gfx1151 GPU prebuilt runs fine there (it bundles its own ROCm runtime; verified: llama-cli --list-devices -> ROCm0: AMD Radeon 8060S, 69 GB). Forward --rocm-gfx whenever a gfx arch is known (it is authoritative and implies ROCm in install_llama_prebuilt.py), and treat a known gfx arch as windows-hip in the existing-install mismatch check. --has-rocm stays gated on the confirmed-runtime signal. Verified on Radeon 8060S / gfx1151: the installer now selects, validates, and installs llama-b1286-windows-rocm-gfx1151-x64.zip (ROCm DLLs present) instead of the CPU build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Install AMD ROCm PyTorch on name-inferred gfx hosts (enables Train/Export) setup.ps1 picked the AMD ROCm PyTorch wheels only inside `if ($HasROCm ...)`. On Adrenalin-only hosts (amd-smi present but no HIP SDK, so $HasROCm is false) the gfx arch was name-inferred but the ROCm-wheel branch never ran, so the host got torch+cpu. With CPU torch, torch.cuda.is_available() is False, so the Studio backend sets CHAT_ONLY=True and hides Train/Export. Un-gate the ROCm PyTorch index resolution on a known gfx arch (mirrors the llama.cpp --rocm-gfx fix). AMD's per-arch Windows wheels (repo.amd.com/rocm/whl/<gfx>) bundle the ROCm runtime, so they work without a HIP SDK; a failed install still falls back to CPU. Verified on Radeon 8060S / gfx1151: torch 2.11.0+rocm7.13.0 installs and torch.cuda.is_available() -> True, device "AMD Radeon(TM) 8060S Graphics", GPU matmul OK -> CHAT_ONLY=False -> Train/Export enabled. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Force amd-smi un-elevated process-wide in the Python installers Guarding individual amd-smi call sites kept missing some (install_python_stack.py's probe loop and its Windows GPU re-check), so the DiskPart UAC prompt kept reappearing. Set __COMPAT_LAYER=RunAsInvoker process-wide at the top of install_python_stack.py and install_llama_prebuilt.py on Windows so every amd-smi subprocess (current and future) runs un-elevated with no per-call guard. Safe: these scripts only spawn amd-smi/rocminfo/hipinfo probes and pip/uv. setup.ps1 keeps per-call guards because it also spawns winget installers that need elevation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Fix Invoke-AmdSmiNoElevate exit code on PS 5.1 + RX 7700S arch match Start-Process -PassThru leaves the returned process object's .ExitCode $null after WaitForExit on Windows PowerShell 5.1, so the helper set $LASTEXITCODE to $null and every caller's `if ($LASTEXITCODE -eq 0 ...)` was always false -- the amd-smi GPU / gfx-token / ROCm-version detection branch was effectively dead (masked only because the un-gated WMI name->gfx inference still ran). Reproduced on PS 5.1.26100. Rewrite the helper to use [System.Diagnostics.Process]::Start with a ProcessStartInfo (UseShellExecute=false), whose .ExitCode is reliable, with async stream reads (ReadToEndAsync) to avoid a pipe-buffer deadlock and WaitForExit(timeout) to bound a flaky amd-smi. __COMPAT_LAYER= RunAsInvoker (inherited via the process env) still suppresses the auto-elevation / DiskPart prompt. Also drops the temp files and the empty-ArgumentList edge case. Verified: exit code propagates (7 -> $LASTEXITCODE=7), output captured, env restored. Also fix the gfx1100 name pattern `RX 7700(?! S)` -> `RX 7700(?!S)` so the spaceless retail name "RX 7700S" is correctly excluded (it belongs to the gfx1102 row). Both found by PR review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address PR review follow-ups (install.sh table, update path, tests, WSL) From the multi-agent PR review: - install.sh: sync the AMD name->arch table with install.ps1 / setup.ps1 (the bash table had drifted to the old narrow patterns). Adds RDNA 2 (gfx103X), workstation PRO W SKUs, and more Strix Halo/Point names, and orders gfx1102 before gfx1100 so the spaceless retail name "RX 7700S" resolves correctly (bash case has no negative lookahead). AMD-ROCm-only: the name inference stays gated behind _has_amd_rocm_gpu(), so NVIDIA / CPU / macOS are unaffected. - setup.ps1: the "dependencies up to date" fast path skipped the torch reinstall, so an existing user who had CPU torch (installed before ROCm-wheel support) stayed stuck in CHAT_ONLY. Now, when an AMD gfx arch is known AND the installed torch is CPU-only, don't skip -- force the dependency pass so the ROCm wheels install. - scripts/install_rocm_wsl_strixhalo.sh: resolve the real /opt/rocm dir instead of hardcoding ROCM_VER for LD_LIBRARY_PATH / the librocdxg symlink (breaks if amdgpu-install lays ROCm under a patch-version dir); add a LIBROCDXG_REF pin knob and a "verified against" freshness header. - tests/studio/install/test_pr5940_followups.py: cover _hf_resolve_url_parts, _fetch_validation_model_bytes (hf path + urllib fallback), run_capture's Windows-only amd-smi RunAsInvoker injection, and install.ps1 vs setup.ps1 name-table parity (catches future drift). 14 tests, all passing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix DiskPart UAC prompt: skip amd-smi on Windows without a HIP SDK On Windows, amd-smi re-initialises the ROCm runtime on every invocation (even `amd-smi version`) and, on hosts without a working HIP runtime (consumer APUs/dGPUs with only the Adrenalin driver), elevates a child process at runtime -- popping a UAC/DiskPart prompt. amd-smi's own manifest is asInvoker, so __COMPAT_LAYER=RunAsInvoker cannot suppress that runtime elevation (verified: even `amd-smi version` hangs and times out with RunAsInvoker set). Replace the ineffective RunAsInvoker-only approach with a real gate: only spawn amd-smi on Windows when a HIP SDK is detectable (hipinfo present, so amd-smi runs un-elevated) or the user opts in with UNSLOTH_ENABLE_AMD_SMI=1. The gfx arch is already resolved from WMI name inference (forwarded via --rocm-gfx), so ROCm wheel + lemonade llama.cpp selection is unaffected. Linux/macOS amd-smi never elevates and is untouched (no regression). RunAsInvoker is kept as harmless belt-and-suspenders for tools that DO use manifest elevation. Applied consistently across: - studio/backend/utils/hardware/amd.py (runtime GPU polling) - install.ps1, studio/setup.ps1 (install-time detection) - studio/install_llama_prebuilt.py (prebuilt arch probe + version) - studio/install_python_stack.py (ROCm version + arch probe) Verified live on AMD Radeon 8060S (gfx1151), native Windows: fresh install detects the GPU, installs ROCm torch (torch.cuda.is_available() True), launches Studio with no DiskPart prompt, and inference, tool calling, web search, LoRA finetuning, and GGUF export all run on the GPU. Tests: add 6 _amd_smi_allowed() gating tests + PowerShell-installer gate assertions; update the three amd-smi monitoring tests to opt in (they mock amd-smi as available). Full suite: 267 passed, 2 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install.sh: helpful WSL message when the GPU isn't exposed to ROCm In WSL, an AMD GPU's ROCm-on-WSL runtime is only available with a recent Adrenalin driver AND a distro AMD supports (currently Ubuntu 24.04). When neither is in place, GPU detection (rocminfo/_has_amd_rocm_gpu) finds nothing and we silently fall back to CPU. Add an actionable hint in the CPU-fallback path, shown only on WSL and only AFTER detection has already failed -- so it is forward-compatible: the moment a driver/distro DOES expose the GPU (e.g. if AMD later adds Ubuntu 26.04 support), detection succeeds and the hint never fires. The message: - notes a GPU is plumbed in (/dev/dxg) but no ROCm runtime is exposed, - lists the two prerequisites (Adrenalin driver + Ubuntu 24.04), - if the distro is not 24.04, says AMD may not support it yet, - tells the user to `wsl --install Ubuntu-24.04` and re-run, - links AMD's ROCm-on-WSL guide + the experimental Strix Halo helper. Verified live: on Ubuntu-24.04 the hint shows (version-warning omitted) and the CPU install completes; on Ubuntu-26.04 the extra "this distro may not be supported" line appears and points to 24.04. Also fix the experimental scripts/install_rocm_wsl_strixhalo.sh: AMD's repo.radeon.com/amdgpu-install/ is indexed by unified installer version (30.30, 31.30, ...), NOT ROCm version, so the hard-coded amdgpu-install/7.2.0/ path 404'd. Scan the installer dirs newest-first for a noble .deb matching the target ROCm major.minor (ROCm 7.2 -> 30.30.x/amdgpu-install_7.2.x), falling back to the newest available. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * WSL: fix shortcut collision + pin ROCm-on-WSL driver reqs from AMD docs Two WSL-related fixes informed by AMD's official ROCm-on-WSL docs and field reports for Strix Halo / Ryzen AI Max+ (Radeon 8060S, gfx1151): 1. Shortcut collision (real bug). install.sh's WSL branch wrote "Unsloth Studio.lnk" to the SAME Desktop / Start Menu folder as the native-Windows installer (install.ps1 New-StudioShortcuts). Running install.sh in WSL therefore silently retargeted the native shortcut at the WSL launcher (wt.exe -> wsl.exe), so the desktop/start-menu icon stopped launching native GPU Studio. Now the WSL shortcut uses a DISTINCT name -- "Unsloth Studio (WSL - <distro>).lnk" -- and fetches the Unsloth .ico to %LOCALAPPDATA%\Unsloth Studio so it shows the proper icon. Native and WSL shortcuts now coexist. 2. Precise ROCm-on-WSL prerequisites. Research (AMD radeon-ryzen WSL compatibility matrix, gianni.rosagallina.com Feb-2026 guide, ROCm/ROCm#4952/#5509/#6022) confirms WSL GPU on Strix Halo requires AMD Adrenalin Edition >= 26.1.1 (26.2.2+ is the first production ROCDXG/WSL release) + ROCm 7.2.1 + Ubuntu 24.04; an older driver does not inject the ROCm/DXG runtime into /usr/lib/wsl/lib, so rocminfo sees only the CPU. install.sh's WSL hint and the experimental install_rocm_wsl_strixhalo.sh header/preflight now state the exact driver version (was a guessed ">=26.3.1"), bump ROCM_VER to 7.2.1, link AMD's radeon-ryzen docs, and document the known librocdxg caveat that usable VRAM is currently capped at the .wslconfig memory setting. bash -n clean; install test suite 267 passed, 2 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * installer: hint when the AMD driver is too old for ROCm-on-WSL Adds a detect-and-guide hook for the optional WSL-GPU path. An AMD GPU on native Windows can also be used inside WSL2, but only with AMD Adrenalin Edition >= 26.2.2 (the first production ROCDXG/WSL release). Native Windows GPU works with any recent driver, so this is purely about enabling the WSL path. We intentionally do NOT auto-install the driver: AMD referrer-gates driver downloads (scripted curl/Invoke-WebRequest are blocked) and does not publish Adrenalin via winget, so no installer can reliably fetch it -- and silently swapping a live display driver is risky. Instead we point the user at AMD's official download page (one click), after which the existing WSL detection lights up automatically. - install.ps1: new Show-AmdWslDriverHint -- when an AMD GPU is present and the installed driver predates the 26.2.2 release (DriverDate < 2026-02-01), print a concise tip with the AMD download URL. Handles DriverDate as either a CIM DateTime or a WMI string. Suppress with UNSLOTH_SKIP_AMD_DRIVER_HINT=1. - install.sh (WSL hint): add the direct Adrenalin 26.2.2 download URL and note that AMD downloads are referrer-gated (open in a browser). Verified: hint fires on a Sept-2025 driver, auto-suppresses on >= 2026-02-01; install.ps1 parses; install.sh bash -n clean; suite 267 passed, 2 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * install.ps1: refresh shell icon cache after creating the shortcut After writing the Desktop / Start Menu .lnk, nudge Explorer to refresh its icon (ie4uinit.exe -show). Without this, a stale icon cache can show a blank shortcut icon until the next explorer restart -- most visible when a shortcut of the same name was rewritten (e.g. a native install followed by a WSL install, which previously shared the name; now they use distinct names, but the cache nudge makes the icon appear immediately regardless). Best-effort and wrapped in try/catch so it never fails the install. The bundled unsloth.ico itself is valid (verified it renders). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * setup.ps1: don't silently CPU-build llama.cpp on an AMD GPU For AMD, GPU acceleration comes from the lemonade ROCm prebuilt (it bundles the ROCm runtime, no HIP SDK needed) and is the preferred/default path. The source-build fallback is CPU-only -- a HIP/ROCm *source* build would need the full HIP SDK + ROCm clang toolchain, which the prebuilt exists to avoid. Previously, if an AMD-GPU host ever fell through to the source build (e.g. the prebuilt could not be downloaded), it printed "building llama.cpp (CPU-only, no NVIDIA GPU detected)" and quietly produced a CPU binary -- masking the lost GPU acceleration. Now that case emits a loud [WARN] explaining the GPU prebuilt is the AMD path and how to restore it (re-run / check network / set UNSLOTH_LLAMA_RELEASE_TAG), so AMD never silently degrades to CPU. No behavior change on the happy path: AMD still gets the GPU prebuilt (verified on gfx1151: ggml-hip.dll bundled, ~80% GPU compute during inference). NVIDIA (CUDA source build) and CPU-only hosts are unchanged. setup.ps1 parses; install suite 267 passed, 2 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * uninstall: remove shared llama.cpp build, kill lock-holders, match WSL shortcut Three gaps found by running a real uninstall on a native-Windows + WSL host; all fixes are scoped to Unsloth-owned paths and no-op on the other pathways (env/custom-root, NVIDIA/AMD/CPU, Mac) so nothing else regresses. uninstall.ps1: - Remove the default-mode SHARED llama.cpp build + cache. setup.ps1 installs them at ~/.unsloth/llama.cpp and ~/.unsloth/.cache -- SIBLINGS of studio, not under it -- so deleting <studio> left hundreds of MB behind. Now removed explicitly, then ~/.unsloth is dropped ONLY if empty (never nukes unrelated content). No-op in env/custom mode (llama.cpp nests under the custom root, removed already) and when absent. UNSLOTH_LLAMA_CPP_PATH (user-owned) is kept. - New _StopProcessesLockingRoots: _StopStudioProcesses only matched the venv unsloth/python/studio exe, so it missed (a) llama-server.exe under llama.cpp and (b) an orphaned multiprocessing python fork that ran from the SYSTEM python but loaded a venv DLL (bitsandbytes) -- on Windows an open DLL handle blocks the directory delete, leaving a half-removed install. The new helper kills any process whose image path OR loaded module is under a target root (module scan scoped to python/unsloth/llama-server names; vendor-agnostic). - _RemovePath now retries (transient post-kill handle release). uninstall.sh: - Remove the default-mode ~/.unsloth/llama.cpp + ~/.unsloth/.cache; rmdir ~/.unsloth only if empty. - WSL Windows-side shortcut cleanup now matches by TARGET (any "Unsloth Studio*.lnk" whose target launches wsl.exe), covering both the legacy "Unsloth Studio.lnk" and the new "Unsloth Studio (WSL - <distro>).lnk" -- and never removes a native-Windows shortcut (which launches wscript.exe). uninstall.ps1 parses; uninstall.sh passes sh -n and bash -n. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * install.ps1: invalidate Win11 Start Menu tile cache after creating shortcut The Start Menu shortcut kept showing a blank/generic icon even after the Explorer icon-cache rebuild, because Windows 11's StartMenuExperienceHost keeps its OWN pre-rendered tile-icon cache (%LOCALAPPDATA%\Packages\Microsoft.Windows.StartMenuExperienceHost_cw5n1h2txyewy\ TempState\TileCache_*.bin + StartUnifiedTileModelCache.dat), separate from Explorer's iconcache_*.db. ie4uinit and an explorer.exe restart do not touch it, and they don't recycle the host -- so a rewritten same-name shortcut keeps showing the first-rendered (often the generic wscript ">") tile until the host restarts on its own. Fix: after creating the shortcut, drop only the Start Menu RENDER caches (TileCache_* + StartUnifiedTileModelCache.dat) and stop StartMenuExperienceHost (Windows auto-relaunches it), so the tile re-resolves the real icon via the shell image factory. start2.bin (the user's pinned layout) is deliberately preserved. Guarded by Test-Path (Windows 10 has no such host -> skipped) and wrapped in try/catch so it can never fail the install. Windows-only (install.ps1); no effect on Linux/macOS/Studio. Verified live: rendering the shortcut via IShellItemImageFactory::GetImage (the API StartMenuExperienceHost uses) returns the Unsloth sloth icon, color-matched, after this invalidation -- previously it returned the generic script tile. install.ps1 parses; install suite 267 passed, 2 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ROCm-on-WSL for AMD Strix Halo (gfx1151): auto-setup + runtime enablement Make Unsloth Studio set up ROCm-on-WSL automatically for AMD Strix Halo (Radeon 8060S / gfx1151) and use the GPU at runtime, validated end-to-end on a Ryzen AI Max+ PRO 395 (ROCm 7.2.1 + librocdxg + Adrenalin Apr-2026): rocminfo enumerates gfx1151, torch.cuda True, ~85.8 GB UMA pool. Every change is a strict no-op for all other configs (NVIDIA/CUDA, discrete + native-Linux AMD ROCm, macOS/MLX, Windows, CPU-only, non-Strix WSL) and can never abort the installer. - scripts/install_rocm_wsl_strixhalo.sh: rewrite to the validated recipe. Fixes that would have broken a working box: drop the /usr/lib/wsl/lib preflight (a working ROCDXG host has only d3d12/dxcore there); remove the obsolete rocr4wsl step (gone from the 7.2.1 repo; would hard-fail and also rips out the standard hsa-rocr ROCDXG needs); dynamic librocdxg soname (was hardcoded 1.1.0; build is 1.2.0); direct apt-repo install; Windows SDK auto-discovery; persist env to /etc/profile.d + ~/.bashrc; idempotent. - install.sh: _maybe_bootstrap_rocm_wsl auto-offers/runs the helper when it detects a Strix Halo APU in WSL (/dev/dxg) with no ROCm runtime, then loads the env so detection routes to the gfx1151 wheels. Fast-path when already configured. Fix an inaccurate WSL hint line. - studio/backend/main.py + worker.py: set HSA_ENABLE_DXG_DETECTION=1 in-process before torch (gated on /dev/dxg AND librocdxg.so), so the worker uses the GPU even when launched outside a login shell. Mirrors the existing BNB_ROCM_VERSION injection. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * uninstall: clean up ROCm-on-WSL artifacts + Start Menu tile cache - uninstall.sh: remove the ROCm-on-WSL helper artifacts -- the librocdxg build clone (~/.unsloth/librocdxg, which otherwise blocks the empty-dir rmdir of ~/.unsloth), the throwaway smoke-test venv, the persisted env (/etc/profile.d/unsloth-rocm-wsl.sh) and the ~/.bashrc block. The system ROCm userspace is a shared prereq like CUDA and is kept by default; UNSLOTH_UNINSTALL_ROCM=1 removes it too. No-ops on macOS / non-Strix Linux. - uninstall.ps1: invalidate the Win11 Start Menu tile cache after removing the shortcut so its tile disappears promptly (mirrors install.ps1), preserving start2.bin. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * installer: accurate AMD ROCm messaging (HIP SDK optional, not required) The Windows installer printed "HIP SDK not found - GPU-accelerated training unavailable" / "ROCm wheels require the HIP SDK" whenever the HIP SDK was absent. That is misleading: for a detected AMD GPU arch (gfx1151 etc.), setup.ps1 installs AMD's bundled-runtime ROCm PyTorch wheels (repo.amd.com) which ship their own ROCm runtime and do NOT need the HIP SDK -- verified end-to-end (torch 2.11.0+rocm7.13.0, cuda True, QLoRA training on GPU) on a Radeon 8060S with no HIP SDK installed. Gate the GPU-detection + rocm-step messages on a detected gfx arch: when one is known, state that GPU PyTorch uses bundled-runtime wheels and the HIP SDK is optional; only when the arch is unknown fall back to the HIP-SDK hint. Behavior (torch routing) is unchanged; this is messaging only. No-op for NVIDIA/CUDA, HIP-SDK-present, and CPU paths (they hit earlier branches). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * installer: fix /opt/rocm data-loss + make WSL shortcut create/remove interop-robust Two fixes from the 3-reviewer regression audit + live testing on a systemd-enabled WSL distro (interop disabled): F1 (data-loss, install_rocm_wsl_strixhalo.sh): the /opt/rocm symlink-repair could force-delete a pre-existing REAL ROCm install. The guard only checked that /opt/rocm is a real directory, not that it is the stray librocdxg stub. Now it only touches /opt/rocm when it is NOT a real install (no bin/rocminfo, bin/hipcc, or .info/version present), and MOVES it aside (rocm.unsloth-stub-bak) instead of deleting it, so a wrong guess can never lose data. WSL interop robustness (install.sh + uninstall.sh): both relied on `command -v powershell.exe`, which is true even when WSL interop cannot EXECUTE it (on systemd distros powershell.exe fails with "Exec format error"). Result: the WSL shortcut silently failed to create (install) and to remove (uninstall). - uninstall.sh: test that powershell.exe actually runs; if not, remove the "Unsloth Studio (WSL...).lnk" files directly via drvfs (/mnt/<drive>), which works without interop. The name is WSL-install-specific, so a native install's "Unsloth Studio.lnk" is never touched. - install.sh: when the shortcut cannot be created, warn with the manual launch command + how to re-enable interop, instead of failing silently. No behavior change on the interop-on path. The regression audit otherwise found no regressions on Linux/Mac/Windows/CPU/NVIDIA install paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * install.sh: fast-path fully restores ROCm-on-WSL env when the drop-in is gone Reinstall regression found by uninstall->reinstall testing: after a Studio uninstall that removed /etc/profile.d/unsloth-rocm-wsl.sh but KEPT the shared ROCm (the default), a non-login reinstall hit the bootstrap fast-path (librocdxg present) and its else-branch only set HSA_ENABLE_DXG_DETECTION -- NOT PATH/LD_LIBRARY_PATH. So rocminfo was not on PATH, GPU detection failed, and the installer fell back to CPU-only PyTorch. Fix: when librocdxg is present but the env drop-in is missing, restore the FULL env inline (HSA + TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL + PATH + LD_LIBRARY_PATH) so rocminfo is found and detection routes to the GPU, and recreate /etc/profile.d/unsloth-rocm-wsl.sh so future shells and the Studio worker get it too. No change to the env-present fast-path or any other host. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * installer: clear Explorer icon cache so shortcut icons aren't blank Root cause of the persistent blank Desktop + Start Menu icons: Explorer caches each shortcut's icon in iconcache_*.db and does NOT re-read the .ico when a same-name .lnk is recreated across reinstalls. The .ico and .lnk are correct (the shell renders them non-blank via IShellItemImageFactory; the .ico has real image data at 16/32/48/128 px), but the stale cache entry wins. The previous fix only ran a weak `ie4uinit -show` + the Start Menu tile-cache clear -- it never invalidated Explorer's icon cache, so the desktop icon stayed blank. Fix (native install.ps1 New-StudioShortcuts AND the WSL shortcut path in install.sh): - ie4uinit -ClearIconCache (thorough; replaces -show as the primary refresh) - SHChangeNotify(SHCNE_ASSOCCHANGED) to force a live desktop/taskbar refresh WITHOUT restarting explorer - keep the Win11 Start Menu tile-cache invalidation (and add it to the WSL shortcut path too, preserving start2.bin) Non-disruptive (no explorer restart). install.ps1 parses clean; install.sh passes bash -n + dash -n; the heredoc-generated WSL PowerShell parses clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * installer: per-item SHChangeNotify(UPDATEITEM) reliably fixes blank icons The blank Desktop/Start Menu shortcut icons are a stale Explorer PER-ITEM icon cache: when a same-name .lnk is recreated across reinstalls, Explorer caches the previously-resolved (often generic "white page") icon for that item and won't re-extract the .ico on its own. The .ico and the .lnk's IconLocation are correct (every icon API renders the sloth) -- only Explorer's cached display is stale. The previous refresh (ie4uinit -ClearIconCache + a GLOBAL SHCNE_ASSOCCHANGED broadcast) does NOT recover a stale item -- confirmed by reproduction. The reliable, NON-disruptive fix (no explorer restart) is a PER-ITEM SHChangeNotify(SHCNE_UPDATEITEM, SHCNF_PATHW, <lnk path>) for each created shortcut, which forces Explorer to re-read that exact item's icon. Verified end-to-end: deliberately staled a shortcut to the generic icon, ran the installer's exact new refresh code, and the sloth icon recovered with NO explorer restart (confirmed by capturing the live desktop via PrintWindow). Applied to both native install.ps1 (New-StudioShortcuts) and the WSL shortcut path in install.sh. Still clears the on-disk icon cache (ie4uinit) and the Win11 Start Menu tile cache (preserving start2.bin). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * uninstall: remove leftover llama.cpp .staging root so ~/.unsloth is cleaned The llama.cpp atomic-install staging root (install_llama_prebuilt.py INSTALL_STAGING_ROOT_NAME=.staging) is a sibling of the llama.cpp install dir (~/.unsloth/.staging in default mode). It is normally pruned after a successful activate, but an interrupted or retained build can leave a <name>.staging-XXXX tree behind. The uninstallers removed llama.cpp and .cache but not .staging, so the final empty-dir cleanup of ~/.unsloth failed and the directory lingered. Reproduced on WSL (Ubuntu-24.04) where an empty llama.cpp.staging-XXXX dir kept ~/.unsloth alive after uninstall. Remove ~/.unsloth/.staging in both uninstall.sh and uninstall.ps1. No-op in env/custom mode (staging nests under the custom root removed already) and when absent. Cross-platform fix (the staging logic is platform-agnostic). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * installer: WSL-absent hint + fix here-string lint false positive install.ps1: in the AMD WSL-ROCm driver hint, detect when wsl.exe is absent and add a one-line "wsl --install -d Ubuntu-24.04" pointer so a Strix Halo user with no WSL yet gets an actionable next step (the hint previously assumed an Ubuntu-24.04 distro already existed). Best-effort, informational only. test_rocm_support.py: test_no_here_strings did a crude substring check that false-positived on the conda-style block marker printf '# <<< Unsloth ROCm-on-WSL (gfx1151) <<<' -- a string literal written into the /etc/profile.d drop-in, also used as a sed delimiter pair by uninstall.sh, not a here-string. Strip quoted spans before the check so the lint still catches a real here-string operator but ignores quoted literals. install.sh remains POSIX-clean (sh -n / dash -n / bash -n all pass). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * installer: address PR review comments (gfx1150 mapping, amd-smi opt-out, WSL bootstrap, SDK path, make) Apply the valid bot review findings on #5940; reject the ones that don't hold. Fixed: - AMD name->gfx table (setup.ps1 + install.ps1): Radeon 890M and Ryzen AI 9 HX 370/375 are Strix POINT (gfx1150), not Strix Halo (gfx1151). Move 890M / HX 37x / AI 9 HX to the gfx1150 row and drop the bogus HX 38x pattern (no such Strix Halo SKU). Matches the runtime classifier in worker.py (890M/880M -> gfx1150; 8060S/8050S -> gfx1151). Prevents Strix Point hosts from getting the wrong ROCm prebuilt/wheels. - amd-smi opt-out (setup.ps1 + install.ps1): an explicit UNSLOTH_ENABLE_AMD_SMI= 0/false/no/off now wins over the HIP-SDK heuristic, so a host with a HIP SDK binary but a broken runtime no longer gets the DiskPart/UAC prompt the opt-out exists to avoid. - amd-smi warning probes (install_python_stack.py): _has_rocm_gpu and _detect_amd_gfx_codes now gate amd-smi behind _amd_smi_allowed() (and pass _amd_smi_env()), closing the last unguarded amd-smi spawn on Windows. - WSL ROCm bootstrap (install.sh): the "already-usable ROCm?" early return now requires rocminfo to enumerate the real gfx1151 agent instead of the generic _has_amd_rocm_gpu (whose broad gfx[1-9][0-9] match accepts a fallback "gfx11-generic" ISA), so a Strix Halo box missing the ROCDXG bridge is no longer skipped. The shared helper is untouched (no gfx90a regression). - install_rocm_wsl_strixhalo.sh: * Quote-safe Windows SDK discovery: the old for-in-$(ls -d "...Program Files (x86)/...") word-split on the space and never matched; use find + read loop. * Add `make` to apt prereqs (cmake only recommends it; minimal images lacked it and the librocdxg `make -j` build failed). * Verification requires gfx1151 exactly (not gfx1[0-9]) so a generic ISA or an unrelated RDNA GPU can't pass while the real GPU is absent. Reviewed but NOT changed: - "Forward inferred ROCm arch without HasROCm" (setup.ps1): already correct -- --rocm-gfx is forwarded under `if ($script:ROCmGfxArch)`, not `if ($HasROCm)`. - "Route inferred arch into install.ps1 torch path": not a bug -- install.ps1 installs CPU torch as a base by design and setup.ps1 swaps in the ROCm wheel for the inferred arch (gate `($HasROCm -or $ROCmGfxArch) -and cpu`); verified live the native install ends on torch 2.11.0+rocm7.13.0. - "$p null guard after Start-Process" (install.ps1/setup.ps1): redundant -- the amd-smi runner uses [Process]::Start wrapped in try/catch, so a null process already returns "" with LASTEXITCODE=1 (no uncaught exception). - "ls -> find for /usr/lib/wsl/lib" (gemini): stale -- that heuristic was removed; only a comment about it remains. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * installer(rocm-wsl): auto-install the Windows 11 SDK via winget (fewer manual steps) librocdxg's build needs the Windows SDK 'shared' headers on the Windows host. Previously the helper just die()d with "install the Windows 11 SDK and re-run" if they were missing -- a manual prerequisite that broke the otherwise-seamless `curl ... install.sh | sh` one-liner on Strix Halo. Now, when the headers aren't found, the helper installs the Windows 11 SDK on the Windows host from inside WSL via winget (powershell.exe interop), then re-discovers them. The SDK installer elevates -> ONE UAC prompt on the Windows desktop; the headers appear under /mnt/c immediately (drvfs is live, no reboot). The user already consented to the ROCm-on-WSL setup, so no extra prompt is added beyond the OS UAC gate. - New _find_win_sdk (space-safe find of the newest installed SDK 'shared' dir) and _install_windows_sdk_via_winget helpers. - winget IDs tried newest-stable first: Microsoft.WindowsSDK.10.0.26100, then .22621. The presence of the headers (re-check) is the source of truth, not winget's exit code. </dev/null so winget never consumes a piped `curl|sh` stdin. - Best-effort + non-fatal: interop-off / no-winget / declined-UAC all fall through to the existing clear manual-install die(). Opt out with UNSLOTH_SKIP_WIN_SDK_INSTALL=1. Removes the last avoidable manual step from the WSL Strix Halo path; only the AMD Adrenalin driver (AMD referrer-gates the download) remains manual. Verified _find_win_sdk resolves the spaced "Program Files (x86)" path; bash -n clean; all winget flags validated against `winget install --help`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * installer(amd): gate install-time amd-smi probe to fix DiskPart UAC prompt install_python_stack.py's Windows "AMD GPU detected but ROCm torch missing" warning probe ran `amd-smi list` whenever amd-smi was on PATH -- and amd-smi ships in C:\Windows\System32 with the AMD Adrenalin driver -- without the _amd_smi_allowed() gate that every other amd-smi call site in the file uses. On Adrenalin-only hosts (no HIP SDK) amd-smi elevates a child at runtime and pops a UAC/DiskPart prompt that __COMPAT_LAYER=RunAsInvoker cannot suppress (amd-smi's manifest is asInvoker). The probe also ran before the ROCm-torch-installed check, so it fired on every Windows AMD install. Gate it behind _amd_smi_allowed() and pass _amd_smi_env(), matching _has_rocm_gpu()/_detect_amd_gfx_codes(). When skipped, the only loss is the best-effort "AMD GPU detected" note on HIP-SDK-less hosts. Adds a per-function AST regression test asserting every function in install_python_stack.py that names the amd-smi command and spawns a subprocess also references _amd_smi_allowed() (flags the pre-fix code; passes after). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * studio(cli): fix `unsloth studio stop` crashing on Windows `stop` used the POSIX `os.kill(pid, 0)` liveness probe, but on Windows CPython raises OSError (WinError 87, "The parameter is incorrect") for *every* pid -- alive or dead. `stop` only catches ProcessLookupError / PermissionError, so the OSError propagated and the command crashed with a traceback before ever reaching its (correct) `taskkill /F` path. Add a cross-platform `_pid_alive(pid)` helper (tasklist on Windows, signal-0 elsewhere) and use it for both the pre-check and the post-kill wait loop. The actual kill path is unchanged. Verified on Windows (Python 3.13): os.kill(pid,0) raises WinError 87 for both a live and a dead pid; `_pid_alive` returns True/False correctly and the full stop() flow (alive -> taskkill -> dead -> "stopped") passes end-to-end against a throwaway process. Adds tests/studio/test_cli_studio_stop_windows.py (AST guard against a bare os.kill(pid,0) liveness probe + mock-only _pid_alive behaviour for the win32 tasklist branch and the POSIX signal-0 branch). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * installer(amd): fix install.sh name->arch table misrouting Strix Point to gfx1151 The bash name->arch inference table in install.sh placed Strix Point identifiers (Radeon 890M, "Ryzen AI 9 HX 370/375", "AI 9 HX") in the gfx1151 (Strix Halo) row, diverging from the install.ps1 / setup.ps1 PowerShell tables which correctly map them to gfx1150. It also carried a stray "HX 38" token absent from the PowerShell source-of-truth. Align install.sh with the PowerShell tables: gfx1151 row: 8060S|8050S|8040S|Strix Halo|Ryzen AI Max|AI Max gfx1150 row: 890M|880M|860M|840M|Strix Point|Krackan|HX 37|AI 9 HX|... Impact is low (the bash table only feeds the display label _gpu_disp_gfx and the "set UNSLOTH_ROCM_GFX_ARCH=..." hint; wheel selection is driven by the detected ROCm version, not this name string) but a Strix Point user would otherwise see/copy the wrong gfx arch. Add a parity test (test_install_sh_name_arch_agrees_with_ps_for_strix_and_non_amd) that parses install.sh's case table and asserts Strix Halo->gfx1151, Strix Point->gfx1150, RX 7700S->gfx1102, and NVIDIA/Intel->no match, cross-checking against install.ps1 (the previous parity test only compared install.ps1 <-> setup.ps1, missing install.sh). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * setup.ps1: keep prebuilt-llama ownership guard within the test's block window The AMD additions to the prebuilt-llama.cpp block (the windows-hip vs windows-cpu existing-install kind validation) pushed the install_llama_prebuilt.py invocation to ~1999 chars after the "installing prebuilt llama.cpp bundle (preferred path)" anchor, right at the edge of the 2000-char window that test_setup_ps1_prebuilt_llama_cpp_has_ownership_guard slices -- so the helper string was truncated and the test failed with "substring not found" (CI: Repo tests (CPU)). The ownership-guard invariant (Assert-StudioOwnedOrAbsent precedes the install_llama_prebuilt.py call) was already satisfied; only the proximity to the anchor regressed. Move the "installing prebuilt..." substep to immediately before the install (after the existing-install pre-cleanup), which also reads better (validate/clean existing -> then "installing"), shrinking anchor->helper from 1999 to 413 chars. Behaviour is unchanged (console message ordering only). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * install.sh: auto-run Strix Halo ROCm-on-WSL setup by default `curl -fsSL https://unsloth.ai/install.sh | sh` should make a Strix Halo (gfx1151) GPU usable inside WSL with no extra commands. Previously the ROCm-on-WSL bootstrap was opt-in: it required UNSLOTH_ROCM_WSL_AUTO=1 or an interactive [Y/n] at a TTY, and silently skipped under a pipe (no /dev/tty), so the piped one-liner never set the GPU up automatically. Flip it to auto-by-default for the single narrow case the existing guards allow (WSL + Strix Halo + /dev/dxg + no usable ROCm yet) -- exactly the GPU setup the user ran the installer for. Opt out with UNSLOTH_SKIP_ROCM_WSL_SETUP=1. The Tauri desktop app keeps its own consent UI (only auto-runs when it passes UNSLOTH_ROCM_WSL_AUTO=1). All hardware/OS guards are unchanged, so non-Strix / non-WSL / NVIDIA / native-Linux / macOS / CPU paths are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * PR comments: condense to be succinct (comments/docstrings only) Shorten the verbose explanatory comments and docstrings this PR added across the installer, scripts, backend shims, CLI, and tests -- tighter, fewer lines, while preserving every non-obvious "why" (os.kill WinError 87, amd-smi RunAsInvoker/UAC, /dev/dxg + librocdxg gating, the ROCm-on-WSL bootstrap guard chain, ownership guards, etc.). No executable code, string literals, messages, or behavior changed. Verified comments-only: docstring-normalized AST equality (Python, 9 files), non-comment token equality (PowerShell, 3 files), comment-stripped diff + sh -n / bash -n (shell, 3 files). Behavior re-confirmed: get_torch_index_url + gfx name->arch table 44/44 under dash & bash; rocm_support / pr5940_followups / cli_studio_stop tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Installer: address PR review (amd-smi opt-out, pipefail, multi-distro, non-root) Fixes valid findings from the Codex/Gemini PR review: - install.ps1 / setup.ps1: gate the `amd-smi version` ROCm-version fallback with $amdSmiAllowed so UNSLOTH_ENABLE_AMD_SMI=0 opt-out is honored (the device probe was gated but this fallback wasn't), avoiding the DiskPart/UAC prompt. - install_rocm_wsl_strixhalo.sh: make the post-verification rocminfo summary best-effort (|| true) so head's early pipe-close under `set -o pipefail` can't fail the bootstrap after gfx1151 was already enumerated; pin the Windows SDK `winget install` to --source winget (matches the msstore-cert fix rationale). - install.ps1: python.org fallback installs the py launcher per-user (InstallLauncherAllUsers=0, avoids admin), and derives the fallback full version from the requested minor so a non-default UNSLOTH_PYTHON (e.g. 3.12) isn't silently replaced with 3.13 when the listing is unreachable. - install.sh: recreate /etc/profile.d/unsloth-rocm-wsl.sh via `sudo tee` for a non-root reinstall (a plain redirect failed silently, dropping the ROCm env). - uninstall.sh: scope WSL Windows-side shortcut removal to the current WSL_DISTRO_NAME (per-distro name or -d "<distro>" arg) so uninstalling one distro no longer deletes other distros' launchers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Studio ROCm Windows: fix field-reported issues from Strix Halo testers Four fixes from PR #5940 field reports (Win11 native, gfx1151): 1. bitsandbytes arch-probe spam: bnb's get_rocm_gpu_arch() runs hipinfo.exe via subprocess PATH at import; the AMD torch wheel ships hipInfo.exe in the venv Scripts dir, which is only on PATH for activated venvs. Every bnb import logged "Could not detect ROCm GPU architecture: [WinError 2]" ERROR + WARNING (even with the HIP SDK installed, whose bin dir is not on PATH either). Prepend the Scripts dir to PATH before bnb imports in main.py, worker.py, and install_python_stack.py, gated on the file existing (only AMD wheels ship it). Verified on gfx1151: ROCM_GPU_ARCH now resolves to gfx1151 with zero errors. 2. OOM-guard double-tax on native Windows unified APUs: mem_get_info's total is the WDDM budget the driver grants HIP (BIOS carve + ~half of remaining RAM) -- the OS share is already outside it. The 0.80 unified cap on top denied loads that fit (field report: 48.49 GiB budget -> "38.79 GiB allowed" OOM for a 47.29 GiB load with 48.08 free). Use 1.0 on win32 unified; Linux keeps 0.80, discrete 0.90. 3. "Missing VRAM" confusion: log the WDDM budget vs physical RAM with the fix (BIOS UMA frame buffer / AMD Software Variable Graphics Memory) when the grant is under 75% of RAM, so a 48 GiB cap on a 96 GiB box reads as policy, not a Studio bug. 4. llama-server fit-step crash (Qwen3.6-27B-MTP + mmproj, lemonade gfx1151): --fit defaults to 'on' upstream, so the fit step runs even when Studio already placed the model via -ngl -1, and aborts in ggml-cuda.cu on some ROCm hosts. Retry the spawn once with --fit off when the server crashes during startup and Studio's own VRAM math had placed the model (never when use_fit or an explicit fit flag was passed). Also keep the TAIL of crash output in the error log (the diagnostic line prints last; head-truncation cut exactly that) and reference the full on-disk log. Verified live on Radeon 8060S: bnb import clean, Qwen3.5-4B-MTP loads and generates through the new spawn loop, stub-crash retry appends --fit off and recovers, fraction probes confirm WDDM overcommit and sub-1.0-only enforcement on current AMD wheels. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio ROCm Windows: GPU-name fallbacks so nothing depends on amd-smi amd-smi does not reliably exist on Windows: the HIP SDK never ships a CLI, inbox Windows Update drivers do not, and only some full Adrenalin packages drop amd-smi.exe into System32 (field report: fresh Win11 + Adrenalin + HIP SDK, still no amd-smi anywhere). Make every consumer work without it: - install_python_stack._detect_windows_gfx_arch: two new probes after hipinfo/amd-smi -- (2b) the venv Scripts hipInfo.exe shipped by AMD torch wheels (drives `studio update` on driver-only hosts), and (4) a last-resort GPU marketing-name -> gfx table via WMI (Win32_VideoController), mirroring setup.ps1's $nameArchTable so a standalone repair resolves the arch with zero AMD tooling installed. - install_llama_prebuilt._resolve_exe: also probe the venv Scripts dir so a standalone rerun finds hipInfo.exe without HIP_PATH. - hardware/amd.py _run_amd_smi: which() guard before spawning -- absence now disables the poller in one step instead of burning the 3-strike circuit breaker on FileNotFoundError; corrected the stale comment claiming Adrenalin ships amd-smi. Simulated against the real detection functions on gfx1151: amd-smi absent, present-but-crashing (exit 1), present-but-hanging (60s sleep vs 5-10s probe timeouts), and hard opt-out -- all resolve gfx1151, no exceptions, bounded time. Full adversarial install (broken amd-smi stub first on PATH + UNSLOTH_ENABLE_AMD_SMI=1, fresh uninstall first): exit 0, name-table arch inference, lemonade gfx1151 b1292 prebuilt, torch 2.11.0+rocm7.13.0 cuda_avail=True on the 8060S, Studio boots healthy and stops cleanly. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: per-attempt llama-server log names + amd-smi test portability Found by cross-platform simulation of the --fit off retry (Windows + Linux sandboxes, real load_model with stub servers): - llama-server log filename now carries the spawn-attempt index. The retry can respawn within the same epoch second; reusing the name opened the same file with "w" and truncated the crash log the retry warning had just pointed the user at (proven with a frozen time.time: one file, crash evidence gone; with the suffix both attempts keep their logs). Regression-pinned in test_llama_cpp_wait_for_health.py. - test_amd_primary_gpu_with_mock now mocks shutil.which alongside subprocess.run: the amd-smi absence guard which()-checks before spawning, so on hosts without a real amd-smi (Linux CI, driver-only Windows) the subprocess mock was never reached and the test failed. Surfaced by running the suite in a clean Linux sandbox. Simulation coverage on both OSes: 67-case platform/edge matrix (real shipped code blocks under win32/linux/darwin spoofs: OOM-guard fractions + VGM-hint boundary, bnb PATH-prepend gates, retry eligibility incl. equals-forms and decoy tokens, GPU-name table adversarial set, WMI fallback without powershell, monitor absence semantics), 6-scenario live retry matrix (crash-once/crash-always/ exit-zero/explicit-fit/hang/log-collision) against real llama-server spawns on Windows and WSL (GPU success legs on the 8060S), and a 3-engine browser matrix (chromium/firefox/webkit) driving the live backend's health + authed /v1 chat completion. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: classify unified-memory via props.is_integrated first Align the ROCm OOM-guard classifier with PR #5988's UMA gate: consult hipDeviceProp_t.integrated (props.is_integrated) before the hardcoded arch set. Strictly additive -- truthy upgrades to unified; 0/absent falls through to the existing gfx1150/gfx1151 + device-name logic, so wheels that omit or zero the field cannot downgrade the known APU set. Extends correct unified-cap treatment to APUs outside that set (e.g. gfx1103 Phoenix iGPUs) and keeps Studio's two unified-memory consumers on one driver signal. Verified live on gfx1151 (is_integrated == 1 on the AMD Windows wheel -> ('gfx1151', True) via the new path). * AMD detection: probe rocminfo with HSA_ENABLE_DXG_DETECTION and sync setup.sh gfx table Fleet validation on a Strix Halo WSL2 box showed the system rocminfo (HSA 1.18, ROCm 7.2.1) only enumerates the GPU over /dev/dxg when HSA_ENABLE_DXG_DETECTION=1, and that rocminfo can sit at /opt/rocm/bin off PATH outside login shells. Detection probes that miss either of these report no GPU on a working ROCDXG host and select the CPU build even though the lemonade bundle offloads fine (95.7 tok/s measured vs 64.5 CPU on the same laptop). Seed the env (a no-op on bare metal) and the PATH fallback in install.sh, studio/setup.sh, and the installer's Linux rocm probe, mirroring what main.py/worker.py already do for the runtime. Also sync studio/setup.sh's name->gfx table with install.sh: 890M and the HX 37/AI 9 HX SKUs are Strix Point (gfx1150, not gfx1151), RX 7700S must match gfx1102 before the gfx1100 row, and the RDNA2/workstation rows were missing. New parity test pins the two bash tables together so they cannot drift again. * Studio: persist server session logs + native-crash stacks to disk Field report (Strix Halo, 96 GB UMA carve, WSL and native Windows): "the studio just terminates without a warning". A native crash in the GPU runtime kills the process with no Python traceback, and a desktop- shortcut console closes before anything can be read. The server only ever logged to the console, so there was nothing to send back. run_server now tees stdout/stderr to ~/.unsloth/studio/logs/server/server-<ts>-pid<n>.log (console behavior unchanged; file copy is best-effort), arms faulthandler at the same file so access violations / SIGSEGV leave a stack trace on disk, and exports PYTHONFAULTHANDLER=1 so training workers inherit crash dumps on their captured stderr. Armed before `from main import app` so even import-time failures leave evidence. Keeps the newest 20 session logs; opt out with UNSLOTH_STUDIO_NO_FILE_LOG=1. Prints "Session log: <path>" at startup so users know what to attach. Verified on this box: a forced real segfault (faulthandler._sigsegv) leaves the full session output plus "Fatal Python error: Segmentation fault" and the thread stack in the file while the console shows nothing; a normal server boot captures the startup banner and serves health as before. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * AMD probe: honor a pre-set HSA_ENABLE_DXG_DETECTION value Match the shell helpers, which use the parameter-default form: a user who exports HSA_ENABLE_DXG_DETECTION=0 to deliberately hide the GPU from DXG detection should not have the probe override it. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <michaelhan2050@gmail.com>	2026-06-10 04:24:49 -07:00
Daniel Han	187144d4e7	Reduce and tighten code comments and docstrings repo-wide (#6095) ... Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.	2026-06-08 23:09:51 -07:00
Daniel Han	8292e699e4	Studio: make code comments and docstrings more succinct (#6029) ... Trim and tighten code comments and docstrings across studio/ Python. Comment-only: every changed file verified code-identical to main via AST/token comparison.	2026-06-08 23:07:28 -07:00
Daniel Han	3ce187da02	Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) ... Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.	2026-06-08 04:24:13 -07:00
Daniel Han	c6e86d5e77	Update Install Scripts (#5968) ... * Update Install Scripts Add SPDX AGPL-3.0 headers to the installer scripts and let the piped web installs take their common options from the environment. - install.sh / install.ps1: read UNSLOTH_NO_TORCH (and UNSLOTH_PYTHON for install.sh) so a piped install needs no positional flags. Flags and the pipe forms still work; an explicit flag wins. - Fix the UNSLOTH_STUDIO_HOME example so the variable sits after the pipe and reaches sh instead of curl. - Add SPDX headers to install.sh, install.ps1, the uninstall scripts, and the MLX install scripts. - Drop the internal test package names from the studio install comments. * Mirror UNSLOTH_PYTHON env var to install.ps1 install.ps1 now reads UNSLOTH_PYTHON to pin the Python version, matching install.sh, and lists all three env vars (UNSLOTH_NO_TORCH, UNSLOTH_PYTHON, UNSLOTH_STUDIO_HOME) in the header examples. The requested version is preferred during detection and used as the winget install target; behavior is unchanged when the variable is unset.	2026-06-03 05:39:42 -07:00
Daniel Han	8ec9a74fd3	studio: ROCm cleanups follow-up to #5301 (#5874) ... Follow-up cleanups to the merged AMD ROCm support PR #5301: 1. De-duplicate the torchao Windows-ROCm import stub into a single shared module (studio/backend/core/_torchao_stub.py); both workers call one install_torchao_windows_rocm_stub() entrypoint. 2. Align the gfx name/arch comment columns in setup.sh and setup.ps1. 3. Isolate the float16 dtype fallback to AMD without native bf16; NVIDIA keeps dtype=None so unsloth's own bf16/fp16/FORCE_FLOAT32 detection is honored. 4. Hoist unconditional stdlib imports (gc, glob, re, subprocess, copy, types, sys, importlib.metadata) from function bodies to module top across the PR #5301-touched files; heavy/optional/relative imports stay lazy. 5. bitsandbytes Windows-ROCm install now uses plain pip (force_pip=True) instead of UV_SKIP_WHEEL_FILENAME_CHECK, per the AMD hackathon docs. Also adds scripts/verify_import_hoist.py (a scope-aware LEGB AST resolver that catches dangling-alias and rename-clash bugs in import-hoist refactors) and wires it into the Lint CI source-lint job as a self-test plus a pull_request compare gate.	2026-05-30 03:06:47 -07:00
Leo Borcherding	b6d5636cc0	fix/strix halo and windows AMD ROCm support (#5301) ... * fix(studio): set HIP_VISIBLE_DEVICES in apply_gpu_ids for ROCm training workers Training workers are spawned via multiprocessing spawn before detect_hardware() runs, so IS_ROCM is still False. If the user never set HIP_VISIBLE_DEVICES in their shell, _inherits_rocm_visibility is also False, leaving the worker with only CUDA_VISIBLE_DEVICES set. On ROCm hosts the HIP runtime honors HIP_VISIBLE_DEVICES over CUDA_VISIBLE_DEVICES, so the worker saw the full device list and torch raised "no usable HIP accelerator" on some setups. Fall back to probing torch.version.hip (a build-time attribute, safe to read before GPU init) to detect ROCm when neither IS_ROCM nor inherited env vars are available. Mirrors the existing fix in llama_cpp.py for llama-server subprocess GPU pinning. Fixes https://github.com/unslothai/unsloth/issues/5180 * test: tighten apply_gpu_ids ROCm fallback assertions Replace loose OR chain with exact string matches, split into three focused tests, and add a guard check for the try/except wrapper. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: detect ROCm unified memory (Strix Halo / AMD iGPU) via torch fallback amd-smi on iGPUs with shared/unified memory (e.g. Radeon 8060S on Strix Halo) reports only the dedicated VRAM slice (~512 MB) in its metric output, so get_visible_gpu_utilization() was returning usable_gb ≈ 0.35 GB instead of the full GTT pool (~128 GB). torch.cuda.mem_get_info() already surfaces the correct unified-pool size. Add _reconcile_rocm_unified_memory(): after amd-smi returns a valid result on a ROCm device, cross-check each device's vram_total_gb against torch.cuda.mem_get_info(). When torch reports a larger total, replace the amd-smi VRAM fields in-place. No-op for discrete AMD GPUs where the two sources agree. Fixes: "Falling back to all visible GPUs -- model may not fit" on AMD iGPU machines even when 100+ GB of unified memory is available. * Apply unified-memory reconciliation in get_gpu_utilization too The visible-GPU path was already corrected for AMD iGPUs with unified memory (Strix Halo / Radeon 8060S), but get_gpu_utilization was still returning the raw 512 MB amd-smi VRAM slice. Studio's /api/train/hardware endpoint and the live GPU monitor read from this primary path, so users continued seeing the wrong total even after auto_select_gpu_ids picked the right device. Refactor to share the per-device correction: * _apply_unified_memory_correction(metrics, torch_info) -- the actual replacement logic, in-place on a single metrics dict. * _reconcile_rocm_unified_memory(...) -- multi-device, iterates utilization["devices"] (visible-GPU path). * _reconcile_primary_rocm_unified_memory(...) -- single flat metrics dict (primary-GPU path), uses parent_visible_spec to pick the primary index, falls back to ordinal 0 when no visibility env is set. get_gpu_utilization now calls the primary reconciler under IS_ROCM, so both endpoints surface the real unified-memory pool on iGPUs while leaving discrete AMD GPUs untouched (torch_total <= smi_total -> no replace). * Use 'is not None' and log debug on torch.version.hip probe failures Two small follow-ups to the apply_gpu_ids ROCm fallback: 1. Match detect_hardware()'s 'getattr(torch.version, "hip", None) is not None' form so the entire codebase has one canonical 'this torch was built with HIP' check. On every shipping torch wheel hip is either None or a non-empty version string, so the new form agrees with the old bool() form on every real install. 2. Log the probe failure at debug level instead of swallowing it silently. The broad 'except Exception' is intentional (we never want apply_gpu_ids to crash a worker over a probe), but the silent pass made it impossible to tell whether the fallback was firing or being skipped. * fix(studio): honour HIP_VISIBLE_DEVICES in _get_parent_visible_gpu_spec before IS_ROCM is set When a user has HIP_VISIBLE_DEVICES set in their shell (e.g. "1" to select GPU 1) but detect_hardware() has not yet run in the Studio parent process, IS_ROCM is still False. _get_parent_visible_gpu_spec() was gated on IS_ROCM so it fell through to CUDA_VISIBLE_DEVICES (unset), saw all physical GPUs, and auto-selected index 0. apply_gpu_ids then overwrote HIP_VISIBLE_DEVICES with "0", making the intended GPU invisible to ROCm torch in the worker, which triggered the "no usable HIP accelerator" error (issue #5180). Apply the same _inherits_rocm_visibility pattern already used in apply_gpu_ids: check for HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES in the environment regardless of IS_ROCM so the correct GPU index is preserved. * fix(install): harden AMD ROCm GPU detection for multi-GPU and env-filtered setups The previous rocminfo awk pattern could miss discrete GPUs on machines where HIP_VISIBLE_DEVICES/ROCR_VISIBLE_DEVICES is used to mask an integrated GPU — the env vars filter rocminfo output but may not propagate into the install script subprocess, causing detection to fail entirely. Two changes: - Tighten rocminfo pattern from /gfx[0-9]/ && !/gfx000/ to /gfx[1-9][0-9]/ — simpler and correctly excludes the CPU agent (gfx000) without a negative lookahead - Add sysfs KFD topology fallback: reads /sys/class/kfd/kfd/topology/nodes/*/gpu_id which is a kernel-level view unaffected by HIP_VISIBLE_DEVICES or ROCR_VISIBLE_DEVICES Fixes detection failure reported in Discord by Chains (gfx1201 + iGPU machine where env var exclusion of the iGPU caused rocminfo to return no usable device). * Fix KFD sysfs awk fallback to read properties file The fallback added by this PR reads /sys/class/kfd/kfd/topology/nodes/*/gpu_id files but matches the literal token 'gpu_id' against their content. Those files contain only a single decimal value (e.g. '0' for CPU agents, '50432' for GPU agents), so the regex never matches and 'found' stays 0, making the fallback a no-op on every host. The properties file in the same directory contains key/value lines like 'gpu_id 50432' which is what the existing awk pattern expects. Reproduced with a synthetic sysfs layout: against gpu_id files awk exits 1; against properties files awk exits 0 when any node reports gpu_id > 0. * fix(setup.ps1): detect AMD ROCm GPU on Windows, bring to parity with setup.sh setup.ps1 only checked nvidia-smi and fell straight to "gpu: none" on AMD machines. setup.sh already probed rocminfo/amd-smi/hipconfig/hipinfo. Add three-tier detection mirroring install_llama_prebuilt.py's detect_host(): 1. hipinfo: gcnArchName in output confirms a real HIP GPU (not just SDK) 2. amd-smi list: "GPU: <digit>" data rows as fallback 3. WMI Win32_VideoController: last resort -- detects AMD GPU even without HIP SDK, then guides user to install it rather than silently going CPU Also corrects the "none" message to mention AMD ROCm alongside NVIDIA so users with AMD hardware understand the requirement. Fixes: rohit-style install where Strix Halo (Radeon 8060S) showed "gpu: none" even with the HIP SDK present. * fix(install.ps1): detect AMD ROCm GPU on Windows, bring to parity with setup.ps1 install.ps1 had the same nvidia-smi-only GPU detection as setup.ps1 before the setup.ps1 fix. Applies the same three-tier AMD detection: 1. hipinfo: gcnArchName confirms real HIP GPU 2. amd-smi list: GPU data rows as fallback 3. WMI Win32_VideoController: detects AMD GPU without HIP SDK and guides user to install it Fixes: install.ps1 showing "gpu: none" while setup.ps1 correctly showed "AMD GPU detected" on the same machine (reported by rohit, RX 7600 XT). * fix(install.ps1): suppress 'No NVIDIA GPU detected' when AMD GPU is present * feat: add Windows AMD ROCm PyTorch wheel installation install_python_stack.py: - Add _ROCM_WINDOWS_WHEEL_BASE and _ROCM_WINDOWS_RELEASES constants pointing to AMD repo.radeon.com (ROCm 7.2 -> torch 2.9.1+rocm7.2.1) - Extend _ensure_rocm_torch() with a Windows branch: detects ROCm via _has_rocm_gpu() / _detect_rocm_version(), requires Python 3.12 (cp312 is the only ABI AMD publishes for Windows), installs the direct wheel URL from repo.radeon.com install.ps1: - Capture ROCmVersion during AMD detection via hipconfig --version / amd-smi version (needed for wheel URL selection) - After Get-TorchIndexUrl, add an AMD wheel override block: when HasROCm and Python 3.12 detected, set ROCmTorchWheelUrl to AMD wheel URL - Expand torch install branch to handle ROCmTorchWheelUrl with uv pip install --force-reinstall --no-cache-dir * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: also install torchvision and torchaudio from AMD Windows repo AMD publishes matching torchvision-0.24.1+rocm7.2.1 and torchaudio-2.9.1+rocm7.2.1 cp312 wheels at the same repo.radeon.com release folder. Install all three in both install.ps1 and install_python_stack.py Windows ROCm path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feat: add ROCm 7.1.1 Windows wheel mapping AMD uses a different version string for 7.1.1 wheels: 2.9.0+rocmsdk20251116 (date-tagged) instead of +rocm7.1.1. Adds the 7.1.1 release folder to both install.ps1 and install_python_stack.py so users with ROCm 7.1 get ROCm torch instead of falling back to CPU. * fix: install rocm_sdk_core and rocm_sdk_libraries_custom alongside torch The AMD Windows torch wheels declare rocm[libraries]==<ver> as a hard dependency. Without installing rocm_sdk_core and rocm_sdk_libraries_custom from the same AMD release folder, uv cannot resolve the dependency and fails with 'No solution found'. Include all 5 wheels in one install call. * fix: expand ROCm wheel array to scalars for Invoke-InstallCommand @array splatting inside a scriptblock only works when the native command is prefixed with '&'. Invoke-InstallCommand uses '& $Command' to run the block, so @ROCmAllWheelUrls was not being expanded. Extract to scalar variables $rw0-$rw4 which are captured correctly by the closure. * fix: use --no-deps for AMD Windows torch wheel install uv's resolver looks up rocm[libraries]==0.1.dev0 on PyPI during dependency resolution before downloading any wheels, and fails because the package doesn't exist on PyPI. --no-deps skips resolution entirely and installs all 5 AMD wheels directly. The GPU runtime dependency is satisfied by the HIP SDK, not a Python package. * fix: setup.ps1 and install_python_stack.py now install ROCm torch on Windows setup.ps1 was always setting CuTag='cpu' for non-NVIDIA hosts and installing cpu-only PyTorch, overwriting the ROCm torch installed by install.ps1. Adds the same AMD wheel selection logic (ROCm version detection, Python 3.12 check, 5-wheel install with --no-deps) to setup.ps1's torch install block. install_python_stack.py: remove IS_WINDOWS guard from _ensure_rocm_torch() call site so the Windows path in _ensure_rocm_torch() is reachable during 'unsloth studio update' as well. * fix: suppress manual-install warning when ROCm torch already present; fix progress counter - Gate the 'must be installed manually' warning on torch.version.hip being empty so it doesn't fire when our ROCm torch install succeeded - Update _TOTAL counter to include the 3 ROCm steps on Windows now that _ensure_rocm_torch() is called there (fixes 10/9 display) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feat: add rocm step display in setup.ps1; fix warning and progress counter - Add 'rocm' step after 'cuda' in setup.ps1 showing ROCm version or HIP SDK missing - Move ROCm version detection up to GPU detection block so it's available early - Suppress 'must be installed manually' warning when torch.version.hip is set - Fix _TOTAL counter to include ROCm steps on Windows (fixes 10/9 display) * fix: detect AMD SDK ROCm torch via __version__ when torch.version.hip is unset AMD's repo.radeon.com wheels (e.g. 2.9.0+rocmsdk20251116) do not set torch.version.hip, leaving it None. All three probes that relied solely on torch.version.hip now also check for 'rocm' in torch.__version__.lower(): - hardware.py detect_hardware(): IS_ROCM was never set, causing the studio to report 'Hardware detected: CPU' even after AMD wheels were installed and HIP DLLs were on PATH. - install_python_stack.py _ensure_rocm_torch(): skip-if-already-installed probe would always reinstall on subsequent runs. - install_python_stack.py Windows AMD warning: suppression check always failed, so the 'must be installed manually' note kept appearing after a successful AMD wheel install. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * perf: drop --no-cache-dir from AMD ROCm torch wheel installs uv caches downloaded wheels by default; passing --no-cache-dir forced a full redownload of the ~2 GB torch wheel on every install run. CUDA installs never had this flag -- AMD was the only path affected. * fix: use install-state flag instead of subprocess probe for AMD Windows warning Replace the subprocess torch probe in the post-install warning block with a module-level _rocm_windows_torch_installed flag set by _ensure_rocm_torch(). Subprocess re-import of torch is unnecessary and fragile -- the install function already knows whether it succeeded. * fix: hoist global declaration to top of _ensure_rocm_torch Python requires the global statement to appear before any assignment to the variable within a function. Moving it to the function top fixes the SyntaxError on line 354. * fix: pass AMD torch install status via env var to suppress false warning setup.ps1 now sets UNSLOTH_ROCM_TORCH_INSTALLED=1 after a successful AMD wheel install. install_python_stack.py reads this at the top of _ensure_rocm_torch() to skip both the subprocess probe and the warning -- no re-import of torch needed, and the warning message now correctly says 'could not be auto-installed' rather than 'must be installed manually'. * fix: register ROCm DLL directory before torch import on Windows Python 3.8+ ignores PATH for extension DLL loading on Windows; amdhip64.dll and other HIP runtime DLLs must be registered via os.add_dll_directory(). Without this, torch.cuda.is_available() always returns False on AMD ROCm Windows even when HIP_PATH is correctly set in system environment variables. Reads HIP_PATH / ROCM_PATH env vars first, then falls back to scanning common ROCm install roots (C:\Program Files\AMD\ROCm, F:\ROCm, C:\ROCm). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: remove hardcoded non-standard ROCm paths from DLL directory scan Only use HIP_PATH/ROCM_PATH (set by AMD installer) and the standard C:\Program Files\AMD\ROCm\<version>\bin location. Custom drive paths like F:\ROCm are user-specific and should not be hardcoded. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: prevent torchao overrides step from overwriting AMD ROCm torch torchao==0.14.0 in overrides.txt declares torch as a dependency. Without --no-deps, uv resolves torch from PyPI and installs 2.11.0+cpu on top of the AMD ROCm wheels (2.9.0+rocmsdk20251116). This was the root cause of 'Hardware detected: CPU' -- the AMD wheels were installed but then immediately overwritten by the overrides step. When _rocm_windows_torch_installed is True, add --no-deps to the overrides pip_install call so torchao is installed without pulling in CPU torch. * fix: add rocm_sdk namespace tarball to Windows ROCm wheel installs torch/_rocm_init.py calls `import rocm_sdk` at startup, which requires the rocm namespace tarball (rocm-*.tar.gz) in addition to the SDK wheel packages. This tarball was missing from both install.ps1 and setup.ps1, causing ModuleNotFoundError on first torch import. - Add rocm-0.1.dev0.tar.gz to ROCm 7.1.1 install (provides rocm_sdk namespace) - Add rocm-7.2.1.tar.gz + rocm_sdk_devel to ROCm 7.2.1 install - Install tarball in a dedicated step before main SDK/torch wheels - Switch to @array splatting in install.ps1 scriptblock for dynamic wheel count - Remove --no-cache-dir from Python-side ROCm wheel install (prevents ~2GB redownload) * feat: enable ROCm 7.2 torch install + warn on gfx1151 with ROCm < 7.2 Chigoma333 (AMD Radeon 8060S / gfx1151, Strix Halo) confirmed that ROCm 7.1 segfaults when tensors are moved to GPU, but ROCm 7.2 + torch 2.11.0+rocm7.2 works fully including training. Changes: - Uncomment (7,2): "rocm7.2" in _ROCM_TORCH_INDEX (was blocked by <2.11.0) - Add _ROCM_TORCH_PKG_SPECS dict with per-tag version bounds: rocm7.2 → torch>=2.11.0,<2.12.0; all older tags → <2.11.0 - Add _detect_amd_gfx_codes() helper that parses rocminfo output - Warn on gfx1151/gfx1150 (Strix Halo) when ROCm < 7.2 is installed, pointing users at the known segfault and recommending upgrade - install.sh get_torch_index_url(): enable rocm7.2 case (previously capped to rocm7.1), cap unknown future tags to rocm7.2 - install.sh: override TORCH_CONSTRAINT to >=2.11.0,<2.12.0 when rocm7.2 index is selected, so pip can actually resolve torch 2.11.0 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: prefer Python 3.12 for AMD ROCm users when 3.13 is also installed After GPU detection, if ROCm HIP SDK is found and the selected Python is not 3.12, run a second pass to locate a 3.12 install via py.exe and PATH (catches uv-managed installs). Switch $DetectedPython to 3.12 so the venv is created with a compatible interpreter for the cp312-only AMD Windows torch wheels. NVIDIA and Intel GPU paths are unaffected -- the re-detection block only runs when $HasROCm is true. Fixes: #5301 * fix: also check uv-managed Python 3.12 for AMD ROCm #5301 * fix: hide amd-smi console popups on Windows, guard torch.distributed.is_initialized for ROCm #5301 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: suppress remaining console popups on Windows, patch torch.distributed.is_initialized for ROCm #5301 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: stub all missing torch.distributed attrs for ROCm Windows wheel #5301 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: inject torch.distributed stub when C backend missing in ROCm Windows wheel #5301 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(rocm/windows): pre-stub torch._C._distributed_c10d + raise amd-smi timeout Two fixes for Windows ROCm regressions reported by electroglyph on #5301: 1. worker.py — torch.distributed stub now fires unconditionally on Windows The previous stub only injected sys.modules in the except branch, meaning it was silently skipped when `import torch.distributed` happened to succeed (the C backend is lazily resolved). The crash then hit later when transformers/trl triggered the lazy load. Fix: on win32 we pre-populate sys.modules['torch._C._distributed_c10d'] AND set the attribute on the torch._C extension module *before* attempting the import, covering both the early-ImportError and lazy-load failure modes. 2. amd.py — increase amd-smi timeout from 5 s to 30 s on Windows (10 s Linux) amd-smi on Windows must cold-init the ROCm runtime on first invocation; 5 s was consistently too short, producing repeated 'Command timed out' warnings in the server log. 30 s gives enough headroom without blocking indefinitely on broken installs. 3. install.ps1 — widen Python 3.12 enforcement to ROCmGpuLabel (WMI-only path) Users whose HIP SDK is not on PATH were detected via WMI but not switched to Python 3.12 before the install started, causing a second pass. Guard now fires on (HasROCm -or ROCmGpuLabel). * fix(rocm): guard c10d stub, fix TorchIndexFamily for 7.1, clean dead code + comments - worker.py: wrap c10d stub injection in `if _c10d_key not in sys.modules` so Windows NVIDIA users with a real torch.distributed are never affected - install.ps1: fix Get-TauriTorchIndexFamily receiving hardcoded "rocm7.2" even when ROCm 7.1 wheels are installed; now branches on $ROCmVersion - main.py: remove dead `import ctypes as _ctypes` (ctypes is never called) - hardware.py, install_python_stack.py, worker.py, install.ps1: shorten verbose multi-line comment blocks throughout - tests: update 4 stale assertions that expected rocm7.2 to be absent/capped * fix(tests): match windows AMD warning assertion to actual source string * chore: trim verbose comment blocks across all ROCm-related files * fix: guard reconcile call against None numeric_ids; add torchvision lower bounds * fix(install.ps1): recreate venv with Python 3.12 after ROCm switch Venv was created with 3.13 before GPU detection ran; switching $DetectedPython to 3.12 had no effect since $VenvPython still pointed to the 3.13 interpreter inside the already-created venv. * ux: detect AMD GPU before Python selection to avoid double venv creation - Early hipinfo + WMI probe runs before Find-CompatiblePython so Python 3.12 is selected upfront when AMD is detected; venv is now created exactly once instead of 3.13 then immediately 3.12. - Post-venv recreation block replaced with a simple warning for the rare case where AMD was missed by the early probe. - setup.ps1: show venv's actual Python version (e.g. 3.12) instead of the system Python found by the pre-activation search (was showing 3.13). * fix(rocm/win): auto-stub all _distributed_c10d symbols via PEP-562 __getattr__ The bare ModuleType stub caused ImportError when torch._dynamo was imported (triggered by trainer.py accessing torch._dynamo.config at load time). torch._dynamo pulls in torch.distributed.fsdp._flat_param which does: from torch._C._distributed_c10d import FakeProcessGroup and potentially other symbols. Adding module __getattr__ auto-creates a stub class for any missing symbol so all such imports succeed without enumerating every individual symbol. Applied to both the primary stub and the fallback stub in the except branch. * chore: trim c10d stub comment * fix(rocm/win): auto-stub missing torch.distributed attrs (Store, ProcessGroup, …) * fix(rocm/win): pre-stub fsdp submodules in sys.modules; fix __getattr__ subpackage clash * feat(rocm/win): arch-aware wheel selector always picks newest ROCm release Replace HIP-SDK-version-gated wheel selection with GPU arch-based logic. Select-ROCmWheelRelease (PS) and _select_windows_rocm_release (Python) map gcnArchName → minimum ROCm version, then pick the newest available release that satisfies it (currently always rocm-rel-7.2.1 for any supported GPU). Wheels bundle their own ROCm runtime so the installed HIP SDK 7.1 does not prevent using 7.2.1 wheels on gfx1200 (RX 9060 XT) and similar RDNA 4 GPUs. Also installs the bitsandbytes Windows ROCm continuous-release wheel and sets BNB_ROCM_VERSION=72 in worker.py before ML imports so bnb loads the libbitsandbytes_rocm72.dll that ships in that wheel. * fix(rocm/win): stub class metaclass for ProcessGroup.BackendType; amd-smi circuit breaker torchao.float8.inference accesses ProcessGroup.BackendType as a class-level attribute. Plain type() stubs have no __getattr__ on the metaclass so this raises AttributeError. Introduce _StubClassMeta whose __getattr__ returns child stub classes, fixing the torchao import chain. Add an amd-smi circuit breaker in amd.py: after 3 consecutive failures the module stops spawning the process, eliminating the repeated Windows UAC / DiskPart elevation prompts caused by polling a non-functional amd-smi. Also guard BNB_ROCM_VERSION=72 behind a DLL existence check so bitsandbytes fails with its own detection message rather than a harder "DLL not found" when the Windows ROCm bnb wheel is not yet installed. * fix: stub __members__ so torchao float8 enum check doesn't crash on ROCm Windows torchao.float8.inference accesses ProcessGroup.BackendType.__members__ expecting a Python Enum registry dict. _StubClassMeta.__getattr__ was blocking all dunder attributes, causing AttributeError. Return {} for __members__ specifically so the isinstance/iteration checks pass cleanly. * fix: stub distributed tensor/functional_collectives to prevent missing C++ op crash on ROCm Windows torch._dynamo.trace_rules eagerly loads torch.distributed.tensor at import time, which pulls in _functional_collectives.py. That file registers Meta kernels for _c10d_functional C++ ops, but those ops are only registered by torch._C._distributed_c10d — a C extension absent from ROCm Windows wheels. Pre-stubbing the affected modules in sys.modules prevents the real import chain from running and avoids the "operator does not exist" crash. * fix: give mod stubs __path__ and pre-stub _tensor to fix 'not a package' import error _make_mod_stub now sets __path__=[] so Python treats stub modules as packages. Without it, any import of a submodule raises "is not a package". Also pre-stub torch.distributed._tensor and its submodules so that _tensor/__init__.py (which re-exports from torch.distributed.tensor) never runs and torchao's `from torch.distributed._tensor import DTensor` gets a harmless stub instead of crashing. * fix: stub torch.ops._c10d_functional namespace with hashable op sentinels torchao.dtypes.nf4tensor uses _c10d_functional ops as dict keys at import time (all_gather_into_tensor.default, wait_tensor.default) and torch.ops.c10d.scatter_.default. None of these ops are registered on ROCm Windows because torch._C._distributed_c10d (the C extension) doesn't ship. Replace the whole _c10d_functional namespace with a custom stub whose ops return hashable .default objects, so dict-key construction doesn't crash. Also inject a scatter_ stub into torch.ops.c10d if it's missing. * fix: stub entire torchao package on ROCm Windows instead of individual ops torchao is not supported on ROCm Windows and its import chain transitively requires torch._C._distributed_c10d (absent from the ROCm Windows wheel). Rather than stub each missing op one by one, stub the whole torchao package upfront. Unsloth uses bitsandbytes for quantization, not torchao, so this has no functional impact. transformers gracefully handles an importable-but- empty torchao by disabling TorchAoHfQuantizer. * fix: set __spec__ on mod stubs so importlib.util.find_spec doesn't raise Manually-injected sys.modules entries have __spec__=None by default. importlib.util.find_spec() raises ValueError when it finds a module in sys.modules with __spec__=None (transformers.utils.import_utils hits this when checking if torchao is available). Give every stub a minimal ModuleSpec(name, loader=None, is_package=True) to satisfy find_spec. * fix: add meta path finder to auto-stub subpackages of stub modules `import torchao.prototype` goes through the import machinery, not __getattr__, so an empty __path__ means ModuleNotFoundError. Rather than list every submodule explicitly, register a MetaPathFinder that intercepts any import whose parent is one of our stubs (detected by loader=None in the parent's ModuleSpec). Real installed packages always have a SourceFileLoader so they are never intercepted. Also register child stubs in sys.modules from __getattr__ as a belt-and-suspenders measure. * fix: use _unsloth_stub sentinel instead of loader=None for stub detection The import machinery overwrites module.__spec__ with the spec returned by find_spec (which has loader=_StubSubpackageLoader, not None), so the loader=None check broke for second-level subpackages. Switch to a custom _unsloth_stub object identity sentinel set directly on each stub module -- it survives __spec__ being replaced and correctly identifies stubs at any depth (torchao.prototype.safetensors, etc.). * refactor(rocm/win): switch to repo.amd.com arch-aware index, remove stubs AMD recommends repo.amd.com/rocm/whl/{arch}/ as the Windows ROCm wheel source. These wheels bundle their own ROCm runtime, support all Python versions (not just cp312), and include the full torch._C extension set (including _distributed_c10d) that the old repo.radeon.com wheel omitted. Changes: - install.ps1: remove Select-ROCmWheelRelease + hardcoded cp312 wheel URLs; remove Python 3.12 forced-preference logic; install via --index-url repo.amd.com/rocm/whl/{arch-family}/ - studio/setup.ps1: same -- remove Select-ROCmWheelRelease, switch to repo.amd.com arch-aware index URL - studio/install_python_stack.py: replace _ROCM_WINDOWS_RELEASES / _select_windows_rocm_release with _windows_rocm_index_url() using the _GFX_TO_AMD_INDEX_ARCH map; drop Python 3.12 restriction - studio/backend/core/training/worker.py: remove all stub machinery (_make_mod_stub, _StubSubpackageFinder, _StubSubpackageLoader, _StubClassMeta, torchao/fsdp/dtensor stubs, _c10d_functional ops stubs, BNB DLL detection) -- no longer needed with new wheel source * fix(rocm/win): restore _distributed_c10d + torchao stubs; fix BNB install repo.amd.com torch wheels also omit torch._C._distributed_c10d on Windows (RCCL is not shipped on Windows). torch/distributed/__init__.py imports from it unconditionally at module level, so the stub must land in sys.modules before any torch.distributed import. torchao (pulled in by transformers.quantizers) walks torchao.float8.distributed_utils -> torch.distributed._functional_collectives -> distributed_c10d at import time. Stubbing torchao up-front short-circuits that chain. worker.py: - Restore _make_mod_stub / _StubSubpackageFinder / _StubSubpackageLoader - Restore _StubClassMeta for ProcessGroup.BackendType attribute access - Restore _distributed_c10d stub with __getattr__ (Windows only) - Restore torchao stubs (5 modules, Windows only) install_python_stack.py: - BNB AMD wheel install was inside the early-return branch that fires when torch is already a ROCm build (installed by install.ps1). Move BNB install outside that branch so it always runs on Windows ROCm — the PyPI bitsandbytes has only CUDA DLLs and fails to load on ROCm. * worker: remove _distributed_c10d stub; stub only torchao The installed torch/distributed/__init__.py from repo.amd.com (torch==2.10.0+rocm7.12.0) is now properly guarded with `if is_available():`, so `import torch.distributed` alone is safe. The crash only comes via torchao's import chain: torchao.float8.distributed_utils → torch.distributed._functional_collectives (unguarded import) → torch.distributed.distributed_c10d → torch._C._distributed_c10d ← absent on Windows ROCm Stubbing torchao short-circuits the chain entirely. No need to stub _distributed_c10d. Remove _StubClassMeta and the _c10d stub block; keep only _make_mod_stub + _StubSubpackageFinder + torchao seeds. * fix: BNB AMD wheel skipped + torch.compile segfault on Windows ROCm install_python_stack.py: the UNSLOTH_ROCM_TORCH_INSTALLED=1 early-return path (set by setup.ps1 when it installed torch itself) returned before ever reaching the AMD BNB prerelease wheel install. The PyPI bitsandbytes==0.49.x ships only CUDA DLLs, so loading it on ROCm fails with "libbitsandbytes_rocm72.dll not found". Now installs the AMD Windows BNB wheel before returning on that path too. worker.py: torch._grouped_mm crashes on gfx1200 (null HIP kernel pointer, 0xC0000005) when torch.compile's JitDecomp system dispatches it during the first forward pass. Detect Windows ROCm via torch.version.hip (already in sys.modules from section 1e) and set TORCHDYNAMO_DISABLE=1 to bypass the broken kernel dispatch. * fix: BNB AMD wheel install fails uv wheel filename check The bitsandbytes continuous-release wheel is intentionally mismatched: filename encodes 1.33.7.preview (= 1.33.7rc0 in PEP 440) but wheel metadata reports 0.50.0.dev0. uv rejects this by default. Introduce _install_bnb_windows_rocm() helper that sets UV_SKIP_WHEEL_FILENAME_CHECK=1 only for this specific install, then restores the previous env value. Both BNB install call sites (the UNSLOTH_ROCM_TORCH_INSTALLED early-return path and the normal Windows ROCm path) now use this helper. * worker: patch _grouped_mm CUDA dispatch on Windows ROCm (gfx1200 null kernel) TORCHDYNAMO_DISABLE=1 stopped the compiler frontend but not the autograd JitDecomp system, which also dispatches _grouped_mm and hits the same null HIP kernel crash (0xC0000005). Verified that torch.library.Library("aten","IMPL").impl("_grouped_mm", fn, "CUDA") successfully overrides the broken HIP kernel with a Python mm fallback on torch==2.10.0+rocm7.12.0. Schema: _grouped_mm(Tensor self, Tensor mat2, Tensor? offs=None, Tensor? bias=None, ScalarType? out_dtype=None) -> Tensor The fallback handles both the simple case (offs=None → torch.mm) and the grouped case (offs provided → split self by offsets, multiply each group against the corresponding slice of mat2, then cat results). Keep _WINDOWS_ROCM_GROUPED_MM_LIB alive at function scope to prevent the C++ dispatch registration from being freed by GC. * worker: fix torchao stub — return stub classes not modules for isinstance() peft/tuners/lora/torchao.py does: from torchao.dtypes import AffineQuantizedTensor, LinearActivationQuantizedTensor isinstance(weight, (AffineQuantizedTensor, LinearActivationQuantizedTensor)) The stub __getattr__ was returning stub modules, which isinstance() rejects with "arg 2 must be a type, a tuple of types, or a union". Add _StubTypeMeta metaclass whose __instancecheck__ always returns False, and _make_stub_type() to create stub classes via it. Change _make_mod_stub __getattr__ to return stub classes instead of stub modules for leaf attribute access, so isinstance() gets a valid type and returns False. _StubSubpackageFinder still handles import-style subpackage creation (those still need module objects in sys.modules); __getattr__ only fires for from-import or direct attribute access, which are the isinstance paths. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests: add coverage for Windows ROCm install paths and worker patches Add conftest.py to fix pre-existing sys.path issue that prevented test_rocm_support.py from running at all (install_python_stack.py imports from backend.utils.wheel_utils which needs studio/ on sys.path). New test classes cover everything added in this session: - TestWindowsRocmIndexUrl: arch → AMD pip index URL mapping (gfx120X-all, gfx1151, gfx1150, gfx110X-all, unknown → None, trailing slash) - TestDetectWindowsGfxArch: hipinfo output parsing, missing/timeout/bad returncode/no-gcnArchName paths - TestInstallBnbWindowsRocm: UV_SKIP_WHEEL_FILENAME_CHECK set+restored, env restored on exception, no-op when URL missing - TestRocmTorchInstalledEnvVar: UNSLOTH_ROCM_TORCH_INSTALLED=1 skips pip_install, calls _install_bnb_windows_rocm, sets flag - TestWorkerWindowsRocmPatches: _grouped_mm CUDA dispatch override, offs/grouped variant handling, GC-prevention sentinel, _StubTypeMeta __instancecheck__, _StubSubpackageFinder registration, torchao key submodule pre-stubbing, TORCHDYNAMO_DISABLE guard - TestRocmTorchPkgSpecs: rocm7.2 torch 2.11.x spec, default <2.11 cap, 3-tuple shape, _GFX_TO_AMD_INDEX_ARCH RDNA4/3.5/3 coverage * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests: fix encoding, IS_WINDOWS patching, and wrong assertion - Add encoding="utf-8" to all read_text() calls (54 occurrences) so tests pass on Windows where the default codec is cp1252 and source files contain UTF-8 emoji (e.g. ⚠️ in install_python_stack.py) - Add @patch.object(stack_mod, "IS_WINDOWS", False) to Linux-path TestEnsureRocmTorch tests so they reach the Linux code path when run on a Windows machine instead of short-circuiting into the Windows branch - Fix test_grouped_mm_patch_guarded_by_windows_and_hip_check: the source uses getattr(_torch_for_rocm, "version", None) not torch.version, so check for '"version"' and '"hip"' substrings instead 137 passed, 2 skipped * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: pin BNB_ROCM_VERSION=72 for torch==2.11.0+rocm7.13.0 compatibility AMD's pip index now ships torch==2.11.0+rocm7.13.0 (ROCm 7.13). bitsandbytes auto-detects HIP 7.13 from torch.version.hip and looks for libbitsandbytes_rocm713.dll, which the AMD Windows prerelease wheel does not ship (it only ships rocm72.dll), causing a load error at training start. Fix: - worker.py section 1f: set BNB_ROCM_VERSION=72 (via setdefault) before section 2 ML imports, so bitsandbytes always loads rocm72.dll on Windows ROCm - install_python_stack.py: set BNB_ROCM_VERSION=72 in _install_bnb_windows_rocm() for any post-install imports; update comment to document root cause - tests: 4 new assertions covering the fix (141 passed, 2 skipped) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: detect BNB ROCm DLL suffix dynamically instead of hardcoding '72' BNB_ROCM_VERSION was pinned to '72' which works today (AMD wheel ships rocm72.dll) but would break again if AMD ships a future wheel with a different DLL suffix (e.g. rocm713.dll). Add _detect_bnb_rocm_dll_ver() to install_python_stack.py: scans the installed bitsandbytes package dir for libbitsandbytes_rocm{VER}.dll using importlib.util.find_spec (no BNB import needed) and returns the suffix. '72' remains the fallback when detection fails. Apply the same detection inline in worker.py section 1f. Both paths still respect a pre-set BNB_ROCM_VERSION (caller override wins). Tests: +8 cases covering detection logic and fallback (147 passed, 2 skipped). * fix: patch torch.distributed stubs in server process for Windows ROCm On Windows ROCm, torch.distributed ships without process-group helpers (is_initialized, is_available, get_rank, get_world_size). The worker subprocess already patches these in section 1e, but the main server process calls _determine_attention_impl_for_gpu_estimate() which calls unsloth's resolve_attention_implementation() → is_initialized(), causing: "Could not resolve attention implementation for '...': module 'torch.distributed' has no attribute 'is_initialized'" Fix: patch the missing attrs onto torch.distributed at the top of _determine_attention_impl_for_gpu_estimate, matching the same stubs already applied in worker.py section 1e. No-ops on Linux/CUDA where torch.distributed is fully populated. * fix: gate _grouped_mm dispatch patch on HIP < 7.13 AMD fixed the gfx1200 null HIP kernel in ROCm 7.13 (torch 2.11+). Users on the new wheel now get the real GPU _grouped_mm kernel for MoE workloads instead of the Python mm fallback. Changes: - worker.py: add _hip_ver_at_least() helper; wrap full _grouped_mm patch in `if not _hip_ver_at_least(7, 13):` with else branch that logs the skip reason; update section-1f comment to document the fix - test_rocm_support.py: add 5 tests covering the helper definition, the (7, 13) gate expression, the else branch, the skip log message, and the AMD-format version string parsing (.split(".")[:2]) Verified: torch==2.11.0+rocm7.13.0 — 3D batch and grouped (offs) variants both succeed; null crash only present on rocm7.12 and earlier. * fix: stub is_torchelastic_launched on torch.distributed for Windows ROCm resolve_attention_implementation calls is_torchelastic_launched() which does not exist in the incomplete torch.distributed shipped with the Windows ROCm wheel, causing a warning on every model config load in the server process. Add it to the stub table alongside the four helpers already patched in _determine_attention_impl_for_gpu_estimate. Also adds two tests: one confirming the new stub and one confirming all five core distributed helpers are covered. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: explicit warnings on AMD ROCm arch/version fallbacks + Fast-Install arg order setup.ps1: - Fix Fast-Install argument order: packages before flags, consistent with all other Fast-Install calls in the file (was: Fast-Install --force-reinstall --index-url $url torch ...) (now: Fast-Install torch torchvision torchaudio --force-reinstall --index-url $url) - Add explicit [WARN] substep when $HasROCm is true but arch mapping fails: - GPU arch detected but not in supported wheel list → names the arch and lists supported families so user knows exactly what to report - HIP SDK present (amd-smi path) but gcnArchName unreadable → instructs user to re-install the HIP SDK; previously fell back silently to CPU install.sh: - Add [WARN] to stderr before silent CPU fallback when AMD GPU is confirmed (rocminfo/amd-smi) but ROCm version cannot be read from any source (amd-smi, /opt/rocm/.info/version, hipconfig, dpkg, rpm) - Add [WARN] to stderr when ROCm version is too old (< 6.0) with upgrade link install.ps1 and setup.sh: no changes needed (already handle these paths correctly) * fix: robust gfx arch detection for Strix Halo / HIP-runtime-only installs Covers users who have the HIP runtime (amd-smi available) but not the full HIP SDK (no hipinfo), which is common on Strix Halo iGPU systems. Without this, $ROCmGfxArch stays null and the installer silently falls back to CPU-only PyTorch despite a working GPU. Detection waterfall (setup.ps1 + install.ps1): 1. hipinfo gcnArchName -- full HIP SDK (existing, unchanged) 2. amd-smi list gfx pattern -- newer amd-smi versions embed arch 3. amd-smi static --asic -- ROCm 6+ ASIC details with GFX target 4. UNSLOTH_ROCM_GFX_ARCH env -- manual override escape hatch 5. GPU name → arch table -- best-effort from marketing name: 890M / Strix Halo → gfx1151 (RDNA 3.5 iGPU, Strix Halo) 880M / Strix Point → gfx1150 (RDNA 3.5 iGPU, Strix Point) 780M / Phoenix → gfx1103 (RDNA 3 iGPU) RX 7900/7800/7700 → gfx1100 (RDNA 3 desktop) RX 9070 XT / 9080 → gfx1201 (RDNA 4) RX 9070 / 9060 XT → gfx1200 (RDNA 4) When arch is inferred from name, a Cyan substep tells the user to set UNSLOTH_ROCM_GFX_ARCH to skip inference on future installs. WMI block intentionally does not set $HasROCm (no runtime confirmation). Tests: 11 new tests in TestStrixHaloGfxArchDetection covering all five detection levels, WMI safety, and gfx regex in both ps1 files. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: resolve hipinfo/hipconfig via HIP_PATH/ROCM_PATH when not on PATH AMD HIP SDK sets HIP_PATH on Windows but does not always add the bin directory to PATH. Get-Command hipinfo therefore silently fails and detection falls through to WMI, which cannot provide a gfx arch, leaving the user with a CPU-only PyTorch install and no warning. Changes: - setup.ps1 / install.ps1: before falling through to amd-smi, attempt to locate hipinfo.exe and hipconfig.exe under $env:HIP_PATH\bin (then $env:ROCM_PATH\bin) when Get-Command returns nothing - Emit a [WARN] with the resolved path and a one-liner to permanently fix PATH via SetEnvironmentVariable - Emit a [WARN] when HIP_PATH/ROCM_PATH is set but the exe is still not found (incomplete SDK install) - Emit a [WARN] with the first hipinfo output line when hipinfo runs but returns a non-zero exit code (e.g. "no ROCm-capable device detected") - 18 new tests in TestHipSdkEnvPathResolution; total 183 passed, 2 skipped * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feat: print HIP SDK path and full hipconfig version in terminal on AMD detection Both install.ps1 and setup.ps1 now emit substeps under the gpu step when AMD ROCm is detected: gpu AMD ROCm (gfx1200) HIP SDK: C:\Program Files\AMD\ROCm\7.1 hipconfig: 7.1.51803-d3a86bd04 Previously only the gpu label (e.g. "AMD ROCm (gfx1200)") was shown with no indication of where the SDK was found or which exact build was active. The full hipconfig build string (e.g. 7.1.51803-d3a86bd04 instead of just 7.1) is now stored in ROCmVersionFull and also used in setup.ps1's 'rocm' step label. 9 new tests in TestHipSdkDetectedSubstep; total 192 passed, 2 skipped * fix: Strix rocm7.1 segfault bypass + Ubuntu 24.04 HIP gcc-install-dir Issue 1 (install.sh): gfx1151/gfx1150 + ROCm 7.1 causes a segfault in torch._grouped_mm (moe_utils.py:167). The Radeon repo now ships cp313 wheels for rocm-rel-7.1, so _amd_gpu_radeon=true silently lands on the broken combo. When Strix Halo/Point is detected and TORCH_INDEX_URL is rocm7.1, override to rocm7.2 PyTorch index, update TORCH_CONSTRAINT, and set _amd_gpu_radeon=false to bypass the Radeon repo entirely. Emits a clear [WARN] explaining the segfault and linking to the ROCm upgrade docs. Issue 2 (setup.sh): ROCm 7.x ships clang-20 which on Ubuntu 24.04+ picks /usr/lib/gcc/x86_64-linux-gnu/14/ (runtime dir, no C++ headers), causing 'cstdlib file not found' and a failed llama.cpp HIP build. Iterate gcc versions 14→11 to find the first install dir that has both runtime and /usr/include/c++/<ver> headers, then pass --gcc-install-dir to clang via CMAKE_HIP_FLAGS. Fix confirmed by h34v3nzc0dex (llama.cpp 417/417 clean). 11 new tests across TestStrixRocm71Override and TestSetupShGccInstallDir; total 203 passed, 2 skipped * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: BNB_ROCM_VERSION in server process + torch._C._distributed_c10d stubs Two errors visible in training logs on Windows ROCm: 1. Server process bitsandbytes crash: "Configured ROCm binary not found at libbitsandbytes_rocm713.dll" The installed BNB wheel ships rocm72.dll (not rocm713.dll). The training worker already sets BNB_ROCM_VERSION=72 via DLL detection but the server process (main.py) imported bitsandbytes before that ran. Fix: add the same DLL-scan + BNB_ROCM_VERSION assignment to main.py inside the existing win32 guard, before any downstream import can pull in bitsandbytes. 2. torch.distributed import failure: "No module named 'torch._C._distributed_c10d'; torch._C is not a package" torch._C is a C extension on Windows ROCm — Python cannot do submodule imports from it, so torch.distributed fails to import before our attribute stubs could ever run. Fix: inject empty ModuleType stubs for _distributed_c10d, _distributed_autograd and _distributed_rpc into sys.modules inside the win32 guard in hardware.py BEFORE importing torch.distributed, so the import succeeds and our attribute stubs take effect. 9 new tests in TestServerStartupRocmFixes; total 212 passed, 2 skipped * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(win32): populate distributed c10d stub with dummy symbols torch.distributed tries to `from torch._C._distributed_c10d import FakeProcessGroup` (and ProcessGroup, Work, Store, etc.). The previous empty ModuleType stub caused an AttributeError on those names. Populate every stub with a _Dummy class for each known symbol so the import chain completes silently on Windows ROCm where torch._C is a compiled extension and its _distributed_c10d submodule doesn't exist. Adds four new tests in TestServerStartupRocmFixes covering FakeProcessGroup, ProcessGroup, setattr population, and all three _distributed_* siblings. * fix(win32): distinguish HIP SDK installed vs GPU not ROCm-accessible Previously, when hipinfo was found but exited non-zero (e.g. "no ROCm-capable device detected"), both install.ps1 and setup.ps1 fell through to the WMI-label-only branch and printed "AMD GPU detected -- HIP SDK not found" -- factually wrong since the SDK binary is present. Add $HipSdkInstalled flag (set true when hipinfo binary is found, regardless of exit code). When HipSdkInstalled && !HasROCm: - Show "AMD GPU detected -- not ROCm-accessible (HIP <ver>)" instead - Explain this is a driver issue, not an SDK issue, with a link - Still run hipconfig version capture so version shows in output - CPU-only hint now says "GPU not ROCm-accessible" not "require HIP SDK" Also applies to setup.ps1 (same detection block, same branches). Adds TestHipSdkInstalledButDeviceInaccessible (11 tests). * fix(win32): scope ROCm workarounds to AMD hosts only Three Codex-flagged issues where Windows ROCm workarounds incorrectly applied to Windows CUDA (NVIDIA) machines: main.py (P1): BNB_ROCM_VERSION was set unconditionally on all win32 hosts. On NVIDIA, bitsandbytes sees BNB_ROCM_VERSION and looks for a ROCm DLL that doesn't exist, breaking bitsandbytes initialisation. Fix: gate the block on HIP_PATH/ROCM_PATH being present (ROCm hosts only). worker.py (P2): torchao stubs were seeded for all win32 runs, shadowing real torchao on Windows CUDA and silently disabling torchao quantization for NVIDIA users. Fix: gate on HIP_PATH/ROCM_PATH (win32 ROCm only). install_python_stack.py (P1): _detect_windows_gfx_arch() only checked shutil.which("hipinfo"), skipping the HIP_PATH/ROCM_PATH fallback that the PowerShell installers use. On installs where the HIP SDK bin dir is not on PATH, _ensure_rocm_torch() returned early without installing ROCm wheels or bitsandbytes. Fix: mirror the env-var fallback. * fix(linux): route Strix + ROCm 7.1 to AMD arch-specific index Instead of falling back to pytorch.org/rocm7.2, the Strix override now routes to repo.amd.com/rocm/whl/gfx1151/ (or gfx1150/) which serves torch 2.11.0+rocm7.13.0 -- AMD's build containing the actual _grouped_mm kernel fix, verified on real gfx1151 hardware by h34v3nzc0dex. This exercises the real GPU kernel path rather than the rocm7.2 workaround. UNSLOTH_AMD_ROCM_MIRROR can override the base URL for air-gapped installs. Also teaches _tauri_torch_index_family to recognise AMD arch-specific URLs (repo.amd.com/rocm/whl/gfx*) and return the rocm7.13 family label so _tauri_gpu_branch correctly classifies these installs as rocm. Suggested by h34v3nzc0dex based on hardware-verified probe results. * fix(studio/rocm): gate ROCm-only side-effects on active torch runtime Address five edge cases flagged during PR review: 1. studio/backend/main.py: BNB_ROCM_VERSION was set whenever HIP_PATH or ROCM_PATH was present in the environment. A Windows CUDA user who once installed the HIP SDK and reverted to a CUDA torch wheel still has those env vars set, so bitsandbytes would try to load libbitsandbytes_rocm72.dll against a CUDA torch and crash. Now probe torch.version.hip inside the env-var guard (worker.py already does this). 2. studio/backend/main.py: os.add_dll_directory returned handles were discarded. Per CPython docs, the directory leaves the DLL search list when the handle is garbage collected. Retain handles in module-level _ROCM_DLL_HANDLES list so they survive process lifetime. 3. studio/install_python_stack.py: _install_bnb_windows_rocm() returned None regardless of pip_install_try outcome, and the caller flipped _rocm_windows_torch_installed to True unconditionally. On a failed BNB install the post-install "manual install may be required" warning was suppressed and the user was misled. Helper now returns bool; caller gates on it. 4. studio/install_python_stack.py: _detect_windows_gfx_arch returned the raw capture group, so mixed-case hipinfo output ("Gfx1151") missed the lowercase keys in _GFX_TO_AMD_INDEX_ARCH and silently fell back to CPU torch. Lowercase the token. 5. studio/install_python_stack.py: UNSLOTH_ROCM_TORCH_INSTALLED=1 early- return trusted the env var even when the venv was wiped between runs. Subprocess-probe torch importability first; fall through to the full install path if the probe fails. Tests: 231 passed, 1 skipped in tests/studio/install/test_rocm_support.py (adds one new test for case 5 fall-through). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio/rocm): worker.py parity + don't roll back ROCm torch on bnb failure Addresses findings from a 10x reviewer pass on the prior fix commit: 1. studio/backend/core/training/worker.py (parity with main.py): - Gate the torchao stub block on torch.version.hip / 'rocm' in torch.__version__ instead of HIP_PATH / ROCM_PATH env-var presence. Same root cause as main.py: HIP SDK env vars stick around on CUDA hosts. - Add module-level Windows ROCm DLL registration block. Worker subprocesses inherit env vars but not the parent's add_dll_directory handles, so the first `import torch` in the worker could fail to find amdhip64.dll when HIP_PATH\bin is not on PATH. Mirrors main.py setup. Handles retained at module scope via _ROCM_DLL_HANDLES. - Promote _WINDOWS_ROCM_GROUPED_MM_LIB to module scope with `global` in run_training_process so the torch.library.Library registration survives past function return / mid-run garbage collection. - Harden _torch_has_hip() to also accept 'rocm' in torch.__version__ (AMD SDK / Radeon wheels may not set torch.version.hip). 2. studio/install_python_stack.py: - Don't roll back ROCm torch when bitsandbytes install fails. The prior commit gated _rocm_windows_torch_installed on _install_bnb_windows_rocm() returning True; if torch installed successfully but bnb failed, the flag stayed False and later install steps could overwrite ROCm torch with the generic CPU torch wheel. Set the flag after torch install; surface bnb failure as a separate warning instead. - _detect_windows_gfx_arch now probes in three tiers: UNSLOTH_ROCM_GFX_ARCH env-var override (matches the PowerShell installer), then hipinfo (PATH or HIP_PATH\bin), then amd-smi (`static --asic`, `list`). Without the amd-smi fallback, runtime-only Radeon installs without hipinfo on PATH made `studio update` return early and leave the venv on CPU torch. - Linux torch-already-rocm probe in _ensure_rocm_torch now matches the Windows probe shape: accepts torch.version.hip OR 'rocm' in torch.__version__ to cover AMD SDK / Radeon Linux wheels. 3. studio/backend/utils/hardware/hardware.py: - apply_gpu_ids() final-fallback torch probe accepts 'rocm' in torch.__version__ in addition to torch.version.hip, matching detect_hardware(). AMD SDK wheels could otherwise leak through with CUDA-only visibility masks on a spawned ROCm worker. Tests: 231 passed, 1 skipped in tests/studio/install/test_rocm_support.py (no test changes needed; the probe shape that prints the hip version (or 'rocm' sentinel) preserves the existing non-empty-string contract). Not addressed in this commit (deferred or out of scope): - Tag drift / lemonade checksum (PR 5303 surface, not this PR). - install.sh rocm7.2.1 URL: small fix, separate. - install.ps1 / setup.ps1 'Radeon 8060S' marketing-name fallback table. - Strix Halo + ROCm 7.1 routing asymmetry in Python update path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio/rocm): robustness pass - rocm tag normalisation, Strix routing parity, hardened detection Robustness pass on top of 76137b2d. Four targeted fixes: 1. install.sh ROCm-tag routing normalisation. `rocm7.2.1` would route to https://download.pytorch.org/whl/rocm7.2.1 which does not exist (PyTorch publishes major.minor URLs only). Same for any future patch-level tag. Normalise every rocm{maj.min}* pattern to the bare {maj.min} index URL. 2. install.ps1 + studio/setup.ps1 marketing-name fallback. The gfx1151 row matched 890M / Strix Halo / HX 37x / HX 38x / AI 9 HX but not the actual retail name 'AMD Radeon 8060S Graphics' shipped by OEMs (Ryzen AI MAX+ 395). Add '8060S' to the regex. 3. install_python_stack.py Strix + ROCm 7.1 routing parity with install.sh. The shell installer reroutes Strix Halo / Point + ROCm 7.1 to repo.amd.com/rocm/whl/{gfx}/ (which serves torch 2.11.0+rocm7.13.0 with the upstream _grouped_mm fix). The Python `studio update` path only warned and still installed the broken generic rocm7.1 wheel. Mirror the override: detect gfx1151/gfx1150 on ROCm 7.1, route to the AMD per-gfx index, honour UNSLOTH_AMD_ROCM_MIRROR override. 4. _detect_windows_gfx_arch amd-smi parsing tightened. The amd-smi fallback added in the prior commit used a bare `\bgfx[1-9][0-9a-z]{2,3}\b` match against the lowercased stdout, which could pick up stray gfx references in warnings / device-name strings. Anchor on labelled lines first (Target_Graphics_Version, ASIC, Arch, gfx) and fall back to the bare match only when no labelled line is present. Tests: 231 passed, 1 skipped in tests/studio/install/test_rocm_support.py; sim_5301 23 cases pass (6 new sims for the Strix override + amd-smi parsing). * fix(studio/rocm): multi-GPU selection, Strix sibling handling, defensive cleanups Round 4 robustness pass based on 5 parallel Opus reviewers of head 21773215. Seven items from across regression / edge-case / error-paths / architecture reviews: 1. studio/backend/main.py BNB gate: aligned with the broad ROCm check used everywhere else in this PR (torch.version.hip OR 'rocm' in __version__). AMD SDK / Radeon Linux wheels do not always populate torch.version.hip; without this, main.py would silently skip BNB_ROCM_VERSION while worker.py set it. 2. studio/install_python_stack.py _install_bnb_windows_rocm: init _ok = False before the try block. Without this, if pip_install_try itself raises (e.g. OSError on uv binary missing), the finally block restored env vars correctly but the subsequent `if not _ok:` raised UnboundLocalError, masking the original exception. 3. studio/install_python_stack.py _detect_windows_gfx_arch: - Rewrote to use re.findall (not re.search) on both hipinfo and amd-smi output, dedup tokens preserving order, and select via new _pick_visible_index() helper. - HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES (first comma entry, integer) now picks the right GPU on multi-AMD-GPU hosts. Out-of-range or non-int values fall back to the first GPU (matches detect_host behaviour in install_llama_prebuilt.py). 4. studio/install_python_stack.py Strix override now consults the runtime target before flipping: - Previous behaviour intersected gfx_codes with {gfx1151, gfx1150} and picked the first Strix arch, ignoring whether HIP_VISIBLE_DEVICES selected a non-Strix sibling (e.g. discrete RX 7900 in a mixed APU+dGPU box). Could install Strix-specific wheels onto a gfx1100 dGPU. - Now resolves the runtime gfx via _pick_visible_index() and only overrides when that runtime target is in the Strix set. 5. studio/backend/main.py + studio/backend/core/training/worker.py: ROCm version dir scan no longer sorts lexically. Previous sort placed "10.0" before "7.0" alphabetically, which would mis-prioritise ROCm 10.x bin dirs once AMD ships them. New _ver_key() splits on "." and sorts numerically with a string fallback. 6. install.sh Strix override URL: replaced ${var%/} (strips one trailing slash) with a while-loop that strips all trailing slashes, matching Python's .rstrip("/"). A user setting UNSLOTH_AMD_ROCM_MIRROR with "http://corp/whl///" no longer ends up with "http://corp/whl///gfx1151/" which strict pip proxies (artifactory, sonatype) 404 on. 7. studio/install_python_stack.py: bumped torch import probe timeout from 30s to 90s. PyTorch's lazy .so loading can take 60-90s on cold NFS or USB-backed venvs. The shorter timeout was producing a false "torch missing" classification and reinstalling a working ROCm torch. Tests: 231 passed, 1 skipped. sim_5301 30 cases pass (added 7 new sims for multi-GPU detection, Strix sibling handling, and _ok-init regression). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio/rocm): worker BNB/grouped_mm broad gate, install.sh Strix visibility, runtime-only ROCm detection Round-5 robustness pass based on 20 parallel reviewers of head 96b9e465. 1. studio/backend/core/training/worker.py - BNB version pin / dynamo disable / _grouped_mm fallback block was still gated on torch.version.hip alone despite the torchao stub block above already using the broad check. AMD SDK / Radeon Windows wheels (torch.__version__ contains "rocm" but torch.version.hip is None) silently skipped the Windows ROCm runtime patches. Aligned to the same broad check (8/20 reviewers). 2. studio/backend/core/training/worker.py - _hip_ver_at_least() now also parses the ROCm version out of torch.__version__ (e.g. "2.11.0+rocm7.13.0") when torch.version.hip is missing, so the kernel-fix gate is correct for SDK / Radeon wheels too. 3. studio/backend/core/training/worker.py - _grouped_mm_safe_impl with offs=None now picks torch.bmm/matmul for 3-D inputs instead of always calling torch.mm. The real _grouped_mm accepts 3-D batched matmul; the prior fallback raised "self must be a matrix" on MoE workloads (2/20). 4. studio/backend/main.py - dropped the HIP_PATH / ROCM_PATH env-var gate from the BNB block; probe torch directly. Runtime-only Radeon / AMD SDK Windows installs do not set those SDK env vars but still ship ROCm torch (5/20 reviewers). 5. install.sh - Strix override now collects every gfx token from rocminfo / amd-smi (in enumeration order), then indexes by HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES so a mixed Strix iGPU + non- Strix dGPU host where the user selected the dGPU does NOT get rerouted to the Strix per-gfx index. Mirrors the Python update path (5/20 reviewers). 6. install.sh - Strix detection chain now also probes `amd-smi static --asic`, matching the PowerShell installer (1/20). Closes the gap on runtime-only Strix hosts where `amd-smi list` does not surface a gfx token. 7. studio/install_python_stack.py - _has_rocm_gpu() now has the sysfs KFD topology fallback (/sys/class/kfd/kfd/topology/nodes/*/gpu_id), matching install.sh. On minimal package-managed installs without rocminfo / amd-smi GUI tools, `studio update` can now detect the GPU and repair the venv instead of returning early (2/20). 8. studio/install_python_stack.py - _detect_amd_gfx_codes() now falls back to `amd-smi list` and `amd-smi static --asic` when rocminfo is missing (2/20). Strix routing on runtime-only Radeon hosts now matches what install.sh has done for a while. 9. studio/install_python_stack.py - Strix override now applies even when has_hip_torch is True. The whole point of the override is to repair an existing broken torch.version.hip == "7.1" install; skipping the reinstall left users on the known _grouped_mm segfaulting stack (3/20). Tests: 231 passed, 1 skipped. sim_5301 30 cases pass. sim_cross 12 pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio/rocm): code review hardening pass - main.py: numeric DLL sort (string sort picked rocm72 over rocm713); add basename() to regex; log warning on detection failure; log info when BNB_ROCM_VERSION is set (mirrors worker.py) - worker.py: explicit len-guard in _hip_ver_at_least() with warning logs instead of silent IndexError/ValueError swallow - hardware.py: isinstance(result, dict) guard before result.get() in _smi_query() to prevent AttributeError on non-dict backend returns - amd.py: round() before int() on parsed GPU IDs; log warning when truncation occurs (defensive against malformed amd-smi output) - setup.sh: quote --gcc-install-dir value in CMAKE_HIP_FLAGS so paths with spaces do not break the CMake argument - install.ps1, setup.ps1: apply colon-split + ToLower() to hipinfo gcnArchName match (consistent with each other and with setup.sh) - install.sh: tighten ROCm tag case patterns to explicit rocmX.Y|rocmX.Y.* to avoid unintended prefix matches * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio/training): GPU OOM guard to prevent system freeze on VRAM exhaustion On RDNA 4 (gfx1200/gfx1201) and other ROCm GPUs, exhausting VRAM can cause a HIP driver hang that freezes the entire system rather than raising a recoverable Python exception. Two-part fix: - set_per_process_memory_fraction(0.90) caps the HIP/CUDA allocator at 90% of VRAM so PyTorch raises OutOfMemoryError before hitting the hardware limit, keeping the driver alive and the system responsive - top-level exception handler detects OOM errors by type and message and surfaces a clear actionable message to the UI (reduce max_seq_length, enable gradient_checkpointing, lower batch size) instead of the raw CUDA/HIP error string * fix(studio/rocm): OOM guard ROCm-only + unified memory, multi-GPU arch selection OOM guard (worker.py): - Scope to _hw.IS_ROCM only -- NVIDIA CUDA has a graceful OOM path and does not need the allocator cap - Detect unified memory by comparing torch VRAM against psutil system RAM; use 0.80 on unified-memory APUs (gfx1151 Strix Halo) where the GPU pool is carved from host RAM, 0.90 on discrete cards Multi-GPU arch selection: - install.ps1 / setup.ps1: replace -match (first hit only) with [regex]::Matches() to collect all gcnArchName entries, then index by HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES - install_python_stack.py: index into full token list before dedup so HIP_VISIBLE_DEVICES=2 on [gfx1100, gfx1100, gfx1151] resolves gfx1151 - install.sh: remove awk dedup from gfx token collection for same reason GCC multiarch (setup.sh): - Only append -linux-gnu when gcc -print-multiarch does not already return the full triple, fixing double-suffix on Ubuntu 24.04 * fix(tests): update ROCm version cap expectations from rocm7.1 to rocm7.2 Daniel's normalisation commit updated the cap from rocm7.1 to rocm7.2 since PyTorch now publishes that index and rocm7.2 ships torch 2.11.0. Test expectations were stale. * fix(tests): correct MLX smoke test losses_per_step assertion logging_steps=1 with max_steps=30 produces 30 loss entries, not 7. The assertion was stale from a previous config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(studio/worker): detect unified-memory APU by GPU name not VRAM/RAM ratio The previous heuristic (VRAM > 50 % of system RAM) false-positived on discrete cards in low-RAM systems — e.g. RX 9060 XT 16 GB on a 16 GB or 24 GB machine would trip the unified-memory path and log "unified memory host" when it should say "discrete". AMD iGPUs (gfx1150/gfx1151 Strix Halo, Strix Point, etc.) expose names with a digit+M suffix ("AMD Radeon 890M"), while discrete cards use "RX NNNN [XT|XTX]" naming. Matching that suffix is reliable across all current ROCm-capable AMD consumer GPUs and does not require psutil. Also includes the device name in the log line to ease future debugging. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(install/setup.ps1): force array on hipinfo gcnArchName parse to fix single-GPU arch truncation When [regex]::Matches() finds exactly one match, PowerShell's pipeline unwraps the result to a scalar string. Indexing a scalar string with [0] returns the first *character*, so a one-GPU system would parse gcnArchName "gfx1200" as "g", which is not in the supported arch map and triggers the CPU-only fallback. Wrapping with @() forces the result to remain an array regardless of match count. On a single-GPU machine the arch is now correctly read as "gfx1200" (or whatever the full name is) so the ROCm wheel index is selected. Reproducer: hipinfo exits 0 and outputs exactly one gcnArchName line. Without @(), $_hipAllArches = "gfx1200" (String); $_hipAllArches[0] = 'g'. With @(), $_hipAllArches = @("gfx1200") (Object[]); $_hipAllArches[0] = "gfx1200". * fix(studio/rocm): classify unified-memory APU via VRAM/RAM ratio, not arch list Replace the gcnArchName allowlist {gfx1150, gfx1151} with a psutil-based heuristic: unified APUs expose the entire system RAM as the HIP pool (ratio ≥ 0.90), discrete cards are well below that. No arch name required — future APUs classify correctly without code changes. Also removes the stale import re / \d[Mm]\b device-name regex that 5d84704 left behind, and logs vram/sys GiB for easier on-hardware verification. Addresses h34v3nzc0dex review: Radeon 8060S (gfx1151, 128 GiB unified) now correctly gets 0.80 cap instead of 0.90. * fix(studio/rocm): revert to gcnArchName for unified-memory APU classification VRAM/RAM ratio >= 0.90 false-positives on machines where discrete VRAM equals system RAM (e.g. RX 9060 XT 16 GB + 16 GB system RAM → ratio 1.0, incorrectly classified as unified → wrong 0.80 cap applied). gcnArchName is the correct signal: naming-independent, stable within a product family, and already parsed throughout this PR. Unified set is {gfx1150, gfx1151} (Strix Point + Strix Halo). * fix(studio/llama-prebuilt): resolve hipinfo via HIP_PATH/ROCM_PATH on Windows shutil.which("hipinfo") returns None when the HIP SDK bin dir is not on PATH -- the HIP SDK installer sets HIP_PATH/ROCM_PATH but does not always add the bin dir to PATH. This caused has_rocm=False in the prebuilt asset selector, so AMD ROCm machines got the CPU llama.cpp zip instead of the HIP one, silently running all chat inference on CPU. Add _resolve_exe() that falls back to %HIP_PATH%\bin and %ROCM_PATH%\bin when shutil.which() finds nothing, mirroring the same fallback already present in setup.ps1. * fix(studio/llama-prebuilt): pass --has-rocm from setup.ps1 to skip re-detection The Python prebuilt installer re-detects ROCm independently via shutil.which("hipinfo"), which fails when hipinfo is not on PATH (HIP SDK sets HIP_PATH but doesn't always add the bin dir to PATH). This caused has_rocm=False and downloaded the CPU llama.cpp zip even on confirmed AMD ROCm machines. setup.ps1 already performs reliable ROCm detection with its own HIP_PATH/ROCM_PATH fallback. Add --has-rocm flag to install_llama_prebuilt.py so setup.ps1 can forward its result directly, and pass it whenever $HasROCm is true. The Python script then overrides has_rocm=True in the HostInfo without re-probing. * fix(studio/llama-prebuilt): add HIP asset to simple-policy Windows path direct_upstream_release_plan (used by --simple-policy, which setup.ps1 always passes) only checked has_usable_nvidia on Windows and fell straight to CPU for AMD ROCm machines, ignoring has_rocm entirely. The --has-rocm override had no effect because the simple-policy code path never reached resolve_asset_choice where has_rocm was checked. Add an elif branch for has_rocm that tries the upstream HIP asset (llama-TAG-bin-win-hip-radeon-x64.zip) before falling through to the CPU fallback, consistent with the non-simple-policy path. * fix(studio/setup.ps1): auto-remove mismatched llama.cpp install kind When an existing llama.cpp install is the wrong kind for the current GPU (e.g. windows-cpu on an AMD ROCm machine that should have windows-hip), the prebuilt installer skips on tag match and never upgrades. Read install_kind from UNSLOTH_PREBUILT_INFO.json before invoking the installer and remove the directory if the kind doesn't match, forcing a fresh download of the correct variant. * fix(studio/setup.ps1): show live PyTorch install output in verbose mode for ROCm The ROCm torch reinstall (setup.ps1 phase) always silently captured output, so in --verbose mode the torch downgrade mid-install (2.11.0+rocm → 2.10.0 → 2.11.0+rocm) looked like the final state was 2.10.0. Match the CPU/CUDA blocks which show live uv output when $script:UnslothVerbose is set. * fix(rocm/windows): set ROCBLAS_TENSILE_LIBPATH for bundled rocblas.dll The llama.cpp ROCm prebuilt bundles rocblas.dll next to the binary but not the Tensile kernel library files it depends on at runtime (rocblas/library/TensileLibrary*.dat + *.hsaco). The bundled DLL searches for these files relative to its own location by default, i.e. <binary_dir>/rocblas/library/, which does not exist in the prebuilt install tree. This causes a silent crash on the very first GEMM (prefill) with no output from llama-server, seen by the caller as WinError 10054 / 10061. Model load and the single-token warmup pass because they use simpler code paths that do not trigger rocBLAS GEMM. Fix: set ROCBLAS_TENSILE_LIBPATH in the subprocess env to <HIP_PATH>/bin/rocblas/library so the bundled DLL finds the kernel files from the system ROCm installation. Uses setdefault so a user- supplied env var is never overwritten. No-ops on CUDA and CPU (no HIP_PATH) and on Linux (win32 branch only). Reproducer log: rocBLAS error: Cannot read .../Release/rocblas/library/TensileLibrary.dat rocBLAS error: Could not initialize Tensile host: directory_iterator: The system cannot find the path specified. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(install.sh): restore gfx token dedup in Strix multi-GPU awk indexer 536a54df removed the per-source `| awk '!seen[$0]++'` dedup from the _gfx_all collection step but left the indexer awk as bare NF, so on a mixed-arch host (e.g. dGPU gfx1100 + Strix iGPU gfx1151) where rocminfo emits each gfx token twice (Name: field + ISA triple), HIP_VISIBLE_DEVICES=1 indexed vals[1] = the second gfx1100 occurrence instead of gfx1151, triggering the Strix routing on the wrong GPU. Add !seen[$0]++ to the indexer awk so duplicate tokens from the same GPU collapse to one entry before the HIP_VISIBLE_DEVICES index is applied -- matching exactly what the Python side does with dict.fromkeys() in _detect_amd_gfx_codes(). The comment above the block ("skip duplicates") already documented this as the intended behaviour. * fix(studio/install): correct _TOTAL progress count on Windows base_total += 3 fired for all non-macOS platforms including Windows, but flash-attn (line 1620) and ROCm torch final (line 1705) are both guarded by 'not IS_WINDOWS and not IS_MACOS', so on Windows with torch enabled _TOTAL was 13 while only 11 _progress() calls actually execute. Split into +1 for the ROCm torch check (all non-macOS) and +2 for the two Linux-only steps, so Windows gets _TOTAL=11 and Linux gets 14. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(install.ps1): enforce torch>=2.11.0 for gfx120X and Strix on Windows The AMD arch-specific index (repo.amd.com/rocm/whl/gfx120X-all/ and gfx1151/) publishes torch wheels from 2.7.1 through 2.11.0. Without a version floor pip can resolve to torch 2.10.0+rocm7.12 on RDNA 4 (gfx120X) or torch 2.10.0+rocm7.1 on Strix (gfx1151/gfx1150), both of which have a null-pointer crash in torch._C._grouped_mm (TheRock issues #5284 / #3284). torch 2.11.0+rocm7.13 contains the fix. Add $ROCmTorchFloor alongside $ROCmIndexUrl: set to torch>=2.11.0 for the two affected arch families, null for all others. Wire it into the uv pip install call so the broken wheels are never selected. * fix(rocm/windows): address Codex nits - deterministic DLL suffix, CUDA llama.cpp kind, HIP_VISIBLE_DEVICES arch indexing - install_python_stack.py / worker.py: _detect_bnb_rocm_dll_ver() and the inline worker probe now collect ALL libbitsandbytes_rocm*.dll suffixes and return max() by numeric value instead of stopping at the first glob hit. Filesystem glob order is not guaranteed; this ensures '713' always wins over '72' when both variants are present in the wheel. - setup.ps1 (expectedKind): add 'windows-cuda' branch so NVIDIA hosts are not treated as 'windows-cpu'. Previously an existing windows-cuda prebuilt was always considered a mismatch on non-ROCm machines, forcing an unnecessary re-download on every update. - setup.ps1 (amd-smi gfx arch): collect ALL gfx tokens from amd-smi list output in GPU order and honour HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES when selecting which arch to use. On mixed-arch AMD systems where the visible GPU is not the first enumerated one, this prevents installing an incompatible wheel index. Falls back to index 0 (same as before) when the visibility var is unset or is a comma-separated list. - test_rocm_support.py: add test_picks_highest_suffix_when_multiple_dlls to cover the multi-DLL case that was previously untested. * fix(rocm): misleading amd-smi log, BNB spec consistency, torch ceiling for AMD index amd.py: split 'returncode != 0 or not stdout' into two separate branches. Previously, exit-0 with empty output logged 'amd-smi returned code 0' (which reads as success, not a warning) and incorrectly incremented the circuit-breaker counter. Now: non-zero exit logs the code and counts toward the limit as before; empty stdout on exit 0 logs at DEBUG level and does not penalise the counter (amd-smi --json always emits at least [] on exit 0, so this branch is rare and is not a tool failure). main.py: replace spec.origin / os.path.dirname() with spec.submodule_search_locations to match install_python_stack.py and worker.py. For normal wheel installs both approaches reach the same directory, but using submodule_search_locations is the canonical way and handles editable bitsandbytes installs correctly. Also use max() by numeric suffix (same as the other two sites) instead of a sort-then-break loop. install.ps1: add <2.12.0 ceiling to the torch constraint for gfx120X (RDNA 4) and gfx1151/gfx1150 (Strix). AMD actively publishes new versions on their per-arch index; without a ceiling, a future 2.12.0+rocmX.Y wheel would be pulled in automatically before being validated on these architectures. The ceiling matches the existing Linux install_python_stack.py constraint for the same arches. Bump both when 2.12.x is confirmed working. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(rocm): torch floor in setup.ps1, torchvision pin for Strix, rocmsdk in _hip_ver_at_least setup.ps1: add \ (mirrors install.ps1) and derive \ from it. Previously the AMD index install called 'Fast-Install torch torchvision torchaudio --force-reinstall --index-url \' with no version constraint, so pip could resolve torch 2.10.0+rocm7.12 for gfx1151/gfx1200 -- the exact broken wheel the PR is meant to avoid. Now gfx120X and Strix enforce 'torch>=2.11.0,<2.12.0', matching install.ps1 and the Linux constraint. install_python_stack.py: pin torchvision and torchaudio in _strix_override_pkgs. The Strix Linux override uses --index-url (exclusive, no PyPI fallback); bare unversioned 'torchvision' and 'torchaudio' could resolve a build from AMD's index targeting a different torch major, causing ABI/version mismatches at runtime. Now pinned to '>=0.26.0,<0.27.0' and '>=2.11.0,<2.12.0' respectively, matching _ROCM_TORCH_CONSTRAINT['rocm7.2']. worker.py: extend _hip_ver_at_least to handle AMD SDK wheel version strings. The fallback regex r'rocm(\d+)\.(\d+)' cannot match '2.9.0+rocmsdk20251116' (no rocmX.Y component), so the function always returned False on SDK/Radeon wheels -- installing the Python _grouped_mm workaround on wheels that already have the working HIP kernel. Added a second check: if the version string contains '+rocmsdk', assume >= 7.13 (the rocmsdk format post-dates the gfx120X null-kernel fix) and skip the fallback. * fix(rocm): warn on OOB HIP_VISIBLE_DEVICES, bail on empty numeric_ids mask - setup.ps1: when HIP/ROCR_VISIBLE_DEVICES names an index beyond the detected GPU count, emit a yellow warning and fall back to GPU 0 instead of silently reading allGfxArches[-1] (wrong arch) - hardware.py _reconcile_primary_rocm_unified_memory: distinguish numeric_ids=None (no env var, use torch ordinal 0) from numeric_ids=[] (empty mask / HIP_VISIBLE_DEVICES=-1, no GPU visible); bail out early in the empty case to avoid querying torch.device(0) incorrectly * fix(rocm): gate StubSubpackageFinder on win32 ROCm, add gcnArchName fallbacks - worker.py _StubSubpackageFinder: the meta_path append was running on every platform on every call to run_training_process; moved it inside the if _is_win32_rocm: block since stubs are only seeded there and the finder is a pure accumulation on Linux/Windows CUDA - worker.py OOM guard: AMD SDK / Radeon wheels may not populate gcnArchName, causing Strix Halo to be misclassified as discrete and get the 0.90 cap (12.8 GB OS headroom) instead of 0.80 (25.6 GB); now tries gcn_arch_name / arch_name / gfx_arch_name variants first, then falls back to device-name matching (890M -> Strix Halo, 880M -> Strix Point) with a debug log when the fallback fires * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(rocm): pin torchvision/torchaudio in setup.ps1, remove -Unique from arch array - setup.ps1 ROCm torch install: torchvision and torchaudio were passed bare alongside pinned torch>=2.11.0,<2.12.0 for gfx1151/gfx1200 arches. AMD publishes packages independently so a future torchvision 0.27 (for torch 2.12) on the same arch index would cause pip ResolutionImpossible or an ABI-incompatible install. Added torchvisionFloorMap and torchaudioFloorMap mirroring install_python_stack.py's strix override (torchvision>=0.26.0,<0.27.0, torchaudio>=2.11.0,<2.12.0) and derived ROCmVisionSpec/ROCmAudioSpec used in all three Fast-Install call sites. - setup.ps1 amd-smi arch detection: Select-Object -Unique was collapsing same-arch multi-GPU arrays (e.g. two gfx1151 APUs -> 1-element array) causing HIP_VISIBLE_DEVICES=1 to trigger a false out-of-range warning and fall back to GPU 0 even though the correct GPU would have been at index 1. Removed -Unique; added comment noting the positional-index assumption and its non-contiguous-GPU limitation. * fix(rocm): add 8060s/8050s to OOM guard device-name fallback, extract classifier helper Path 3 of the OOM guard device-name fallback only checked for 890m/880m (gfx1150 Strix Point SKU names). Strix Halo (gfx1151) ships as Radeon 8060S (Ryzen AI MAX+ 395) and Radeon 8050S (cut-down SKU) -- neither matches, so the fallback returned is_unified=False and applied the 0.90 fraction instead of 0.80, leaving ~12.8 GiB OS headroom on a 128 GiB pool instead of ~25.6 GiB. Fix: add 8060s and 8050s to the name-match set. Also correct the comment that mislabelled 890M as a Strix Halo name (it is Strix Point). Refactor: extract the three-path classifier into _rocm_classify_unified_memory() so it can be unit-tested directly. Add 31 test cases in test_rocm_oom_guard.py covering all three paths and the regression case (Radeon 8060S Graphics). Reported-by: h34v3nzc0dex * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(rocm): pass explicit dtype on bf16-unsupported hardware (RDNA2) dtype=None lets unsloth auto-detect the model dtype. On RDNA2 (gfx103x, e.g. RX 6600) is_bfloat16_supported() incorrectly returns True, so unsloth picks bf16 and the first bf16 kernel dispatch triggers: LLVM ERROR: Cannot select: intrinsic %llvm.amdgcn.fdot2.bf16.bf16 Replace every dtype=None in load_model() with _auto_dtype which resolves to None when bf16 is supported (all modern NVIDIA + RDNA3+) and torch.float16 otherwise. This gives RDNA2 users a working float16 training path without touching NVIDIA behaviour at all. Fixes: https://github.com/unslothai/unsloth/issues/5337 * fix: reduce log noise for expected non-issues on Windows ROCm Three log lines fired at warning/error level for conditions that are completely expected on a Windows HIP SDK-only setup: amd.py - amd-smi WinError 2 (FileNotFoundError): downgrade warning -> debug. amd-smi ships with Adrenalin, not the HIP SDK; absence is normal. - 'disabling' message: downgrade warning -> info with clearer text 'not available (not installed; expected on HIP SDK-only systems); GPU VRAM polling disabled' hardware.py - torch.distributed.Store missing: downgrade warning -> debug. The distributed stub added in this PR intentionally omits Store; the attention-impl fallback to eager is expected and non-actionable. worker.py - causal-conv1d: add early Windows exit (info) in both _ensure_causal_conv1d_fast_path and _causal_conv1d_install hook; no cp313/win_amd64 wheel exists, so the install always fails. - FLA: add early Windows exit (info) in _ensure_flash_linear_attention_unconditional; triton dependency has no cp313/win_amd64 wheel. - Defense-in-depth: _install_package_wheel_first non-HIP PyPI failure logs info+debug on Windows instead of error; FLA failure logs info+debug on Windows instead of warning. * [AMD] FIx installation of bitsandbytes when it's from .dev and skip rebuilding llama.cpp if we build it manually. * fix: use force_pip for Windows ROCm bitsandbytes prebuilt wheel install uv rejects the bnb continuous-release wheel due to filename/metadata version mismatch (1.33.7.preview vs 0.50.0.dev0). Switch to force_pip=True (pip bypass) instead of the UV_SKIP_WHEEL_FILENAME_CHECK env var workaround -- cleaner and consistent with how the Linux path handles it. BNB_ROCM_VERSION is still set post-install to the detected DLL suffix so the worker subprocess loads the correct libbitsandbytes_rocm{VER}.dll even when torch.version.hip reports a newer HIP version than the wheel ships. * fix: three small correctness fixes found in PR review - _install_bnb_windows_rocm: use UV_SKIP_WHEEL_FILENAME_CHECK=1 with try/finally instead of force_pip=True so the env var is always restored and the failing CI test passes - _determine_attention_impl_for_gpu_estimate: gate torch._C distributed stubs on IS_ROCM so Windows CUDA users keep the real extension - install.ps1 amd-smi fallback: collect all gfx tokens and index by HIP_VISIBLE_DEVICES, matching the hipinfo path on multi-GPU hosts * fix: stub torchao in export subprocess on Windows ROCm On Windows, the ROCm build of PyTorch ships without the distributed C extension (torch._C._distributed_c10d). torchao, which is pulled in transitively by transformers.quantizers at import time, walks into torch.distributed._functional_collectives -> distributed_c10d and crashes with: No module named 'torch._C._distributed_c10d'; 'torch._C' is not a package This only affected the export subprocess because the training subprocess already applied an identical torchao stub (introduced separately to fix the same root cause). The export subprocess had no such guard and died during 'Importing Unsloth...' before any model loading could happen. Fix: apply the same _StubSubpackageFinder / torchao stub pattern to the export subprocess entry point, gated on Windows ROCm detection, before any import of transformers or unsloth_zoo. Root cause tracked in ROCm/TheRock#3284 (libuv / torch.distributed missing on Windows ROCm builds). Ref: https://github.com/ROCm/TheRock/issues/3284 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install.sh, setup.sh: add GPU arch step logging to match PS1 scripts Both shell scripts were missing the step "gpu" terminal log block that install.ps1 and setup.ps1 emit. This adds equivalent output: GPU label with gfx arch (e.g. "AMD ROCm (gfx1151)"), ROCm root path, hipconfig version, and marketing name substep. Includes the same gfx arch detection chain (rocminfo → amd-smi list → amd-smi static --asic), UNSLOTH_ROCM_GFX_ARCH env override, and name-based arch inference table (Strix Halo/Point, RDNA 3/4) as the PS1 versions. install.sh also replaces bare echo blocks for the AMD ROCm and CPU-only cases with formatted substep output. * Fix BNB_ROCM_VERSION gate, ROCm GPU mask preference, APU unified memory and Release build for PR #5301 - main.py: gate BNB_ROCM_VERSION on the rocm bnb DLL or HIP_PATH/ROCM_PATH instead of importing torch on every Windows host - hardware.py: prefer HIP/ROCR visible-device masks only on ROCm hosts so a stale mask cannot override CUDA_VISIBLE_DEVICES on NVIDIA - llama_cpp.py: set GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 only for unified-memory APUs (gfx1150/gfx1151) - setup.sh: pass -DCMAKE_BUILD_TYPE=Release for the HIP source build - add test_amd_apu_unified_memory.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: guard recompile_limit + fix AMD VRAM monitor fallback trainer.py: torch._dynamo.config.recompile_limit does not exist in some ROCm torch builds (e.g. pytorch.org/whl/rocm6.2 wheels). Guard the assignment so training doesn't crash on RDNA2/RDNA3. hardware.py: when amd-smi/nvidia-smi is unavailable or returns no usable data (HIP SDK-only Windows, Docker, unexpected JSON format), the existing fallback used torch.cuda.memory_allocated() which is process-specific and reads near-zero even with a fully loaded model. Switch to torch.cuda.mem_get_info() via _torch_get_per_device_info() which reports system-wide VRAM occupancy so the GPU monitor shows real usage on all AMD systems without requiring amd-smi. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: Windows VRAM monitor via Performance Counter API When amd-smi/nvidia-smi is unavailable on Windows, query dedicated GPU VRAM via Windows Performance Counters (same source as Task Manager). This gives system-wide cross-process usage, fixing the near-zero reading caused by torch.cuda.mem_get_info only seeing the Studio server process. Linux fallback path unchanged (mem_get_info is system-wide on ROCm). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: rename to _rocm_windows_perf_counter_vram_gb, scope to IS_ROCM Function is AMD ROCm specific — amd-smi absent on Windows when only the HIP SDK is installed. Scoped to IS_ROCM so NVIDIA Windows path is untouched (nvidia-smi handles that case). * fix: AMD VRAM monitor — Linux DRM sysfs + Windows perf counter Linux: read /sys/class/drm/card*/device/mem_info_vram_used|total for system-wide GPU memory across all processes. No tools required, always present on Linux AMD systems. Windows: Windows Performance Counter API (already added). Both paths are gated on IS_ROCM and only fire when amd-smi is absent. torch mem_get_info remains as last resort (process-local). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: AMD GPU monitor — utilization, temperature, and power for Windows and Linux fallback paths - Windows: GPU utilization via \GPU Engine(*engtype_3D*)\Utilization Percentage perf counter - Windows: temperature and power via ADL (atiadlxx.dll, ships with Adrenalin) - Linux: GPU utilization via DRM sysfs gpu_busy_percent - Linux: temperature via hwmon temp1_input (millidegrees C) - Linux: power via hwmon power1_average / power1_input (microwatts) All paths are no-op fallbacks (None) when the source is unavailable. Mirrors what nvidia-smi provides on the CUDA path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: remove ADL ctypes — does not support AMD iGPU (Strix Halo) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: Erland366 <erland.pg366@gmail.com> Co-authored-by: danielhanchen <michaelhan2050@gmail.com>	2026-05-29 22:29:56 -07:00
Daniel Han	ca55acbb5f	Studio: unblock install on Linux ARM64 + Windows ARM64 + Intel Mac (#5790) ... * Studio: unblock cross-platform install on Linux ARM64 + Windows ARM64 Three independent bugs that together prevent `install.sh` / `install.ps1` from completing on the ARM machines GitHub Actions now ships (`ubuntu-24.04-arm`, `windows-11-arm`) and on equivalent real hosts (Ampere Altra, Raspberry Pi 5, Snapdragon X Elite, ...). Validated on the staging-2 cross-OS smoke suite -- five per-OS workflows pinned to `ubuntu-latest`, `ubuntu-24.04-arm`, `macos-14`, `macos-15-intel`, `windows-11-arm`. Before this change Windows ARM exits 1 in the winget gate and Linux ARM source-builds llama.cpp because the prebuilt selector returns 0 attempts; with it both reach healthy /api/health. 1. studio/install_llama_prebuilt.py -- resolve_simple_install_release_plans had explicit branches for windows+x86_64, macos+arm64, macos+x86_64 and linux+x86_64 only. Upstream ggml-org/llama.cpp ships `llama-bNNNN-bin-ubuntu-arm64.tar.gz` and `llama-bNNNN-bin-win-cpu-arm64.zip` (visible in the b9334 release manifest), so the missing elif branches force every Linux ARM64 and Windows ARM64 host into a source build even when a perfectly good upstream prebuilt is one HTTP GET away. Two new branches mirror the existing CPU variants; runtime_patterns_for_choice and runtime_payload_health_groups gain `linux-arm64` (.so layout) and `windows-arm64` (.dll layout) so the health-check pass-through matches the asset shape. 2. studio/setup.sh -- the helper-release-repo selector routed any non-x86_64 Linux to `unslothai/llama.cpp`, which only publishes the Linux CUDA bundle set. The result on Linux ARM64 was a guaranteed `direct_linux_release_plan` raise of "no compatible Linux prebuilt asset was found" on every release in the scan, then a source-build fallback. Pin Linux ARM64 (CPU-only) to `ggml-org/llama.cpp` so the new branch in (1) can see the upstream asset. setup.ps1 already hardcodes `ggml-org/llama.cpp`, so Windows ARM64 picks up (1) without an additional change. 3. install.ps1 -- the winget pre-check hard-failed before Python or uv detection. `windows-11-arm` runners (and many corporate Windows hosts without the Microsoft Store) ship without winget but already have a usable Python plus the Astral uv PowerShell installer reachable. Demote the winget check to a soft warning, defer the hard failure to the Python install branch (which is the only path that genuinely needs winget), and let the uv install fall through to `https://astral.sh/uv/install.ps1` when winget is absent. The uv PowerShell installer was already the existing fallback for the "winget present but uv install failed" case; this just makes it the primary path on hosts without winget. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: filter torchcodec on platforms without wheels torchcodec 0.10.0 ships wheels for manylinux_2_28_x86_64, macosx_12_0_arm64, and win_amd64 only -- visible on its PyPI page and in the resolver error reported by #4446. install_python_stack.py pulls torchcodec via extras-no-deps.txt, which is now installed unconditionally during `unsloth studio update --local` (the update command has no --no-torch flag). Result on Linux aarch64 / Windows ARM64 / Intel Mac (when invoked outside the install.sh auto-skip-torch path): ERROR: Could not find a version that satisfies the requirement torchcodec==0.10.0 (from versions: 0.0.0.dev0, ...) ERROR: No matching distribution found for torchcodec==0.10.0 error Installing extras (no-deps) (pip) failed (exit code 1) `NO_TORCH_SKIP_PACKAGES` already lists torchcodec but only fires when NO_TORCH is true -- the update path inherits no NO_TORCH from the original install and inferrence falls back to IS_MAC_INTEL only, so Linux aarch64 / Windows ARM64 sail past the guard. Adds a platform predicate PLATFORM_LACKS_TORCHCODEC_WHEEL and applies the torchcodec filter unconditionally there, independent of NO_TORCH. Surfaced by the staging-2 cross-OS smoke `unsloth studio update` step on ubuntu-24.04-arm; verified the same step is green with this patch overlaid. * Studio: skip librosa on no-torch hosts (unblocks Intel Mac install) Closes the last cross-platform install gap surfaced by the staging-2 cross-OS smoke (see unslothai/unsloth#5046 for the original report): `install.sh --local` on macos-15-intel fails at × Failed to build `llvmlite==0.47.0` error: failed-wheel-build-for-install ╰─> llvmlite error studio setup failed (exit code 1) Root cause: upstream llvmlite dropped the macosx_x86_64 wheel between 0.42.0 and 0.46.0 (https://pypi.org/project/llvmlite/0.47.0/#files -- only macosx_arm64 / manylinux / win_amd64 remain). pip falls back to a from-source build of llvmlite's FFI, which needs LLVM 14/15 dev headers and matching llvm-config -- not present in Xcode Command Line Tools' libclang and not installed by install.sh's MAC_INTEL deps branch. llvmlite enters Studio's tree via librosa -> numba -> llvmlite in extras.txt. openai-whisper (extras.txt:28) would also pull numba but is already filtered on no-torch hosts. Adding librosa to the same NO_TORCH_SKIP_PACKAGES set makes the install go through cleanly on Intel Mac (auto-detected NO_TORCH=true via the MAC_INTEL branch) and on any user-passed --no-torch host where torch-dependent audio pipelines would not run anyway. Tracked / verified on the danielhanchen/unsloth-staging-2#154 smoke matrix (macos-15-intel). * Studio UI tests: retry evaluate_fetch on transport-level failure (PR #5790) Mac Studio UI CI on this PR (run 26496820814, job 78026959359) failed with /api/models/list status=0 error='TypeError: Failed to fetch'. The artifact studio.log shows the server answered the two preceding /api/models/list calls from the React mount (both 200) but never received the third call from the test script: the browser reused a kept-alive HTTP/1.1 socket that uvicorn (5s keep_alive_timeout) had closed ~130ms earlier. Chromium under --single-process on macos-14 free runners is most prone to this; the post /api/auth/change-password session churn accelerates it. A rerun on the same SHA passed, which is the classic flake signature. evaluate_fetch in tests/studio/_playwright_robust.py already returns a structured {status: 0, body: None, error: "..."} on JS-side throws, but every caller treats status=0 as fatal. Add a bounded retry inside the helper so the one class of failure recovers transparently: status != 0 -> real HTTP response (incl. 4xx/5xx); propagate. error has "AbortError" -> caller's AbortSignal deadline; propagate. else (status==0) -> stale-keepalive or other transport failure; retry after 250ms / 500ms backoff so the pool evicts the dead socket before the next attempt. Defaults transport_retries=2, transport_backoff_ms=250 (max added latency on the happy path is zero; on a transport failure: up to 750ms of sleep). Callers keep the existing {status, body, error} shape; no call-site changes needed. Verified: tests/studio/_playwright_robust.py compiles; signature gains two kwonly args (transport_retries, transport_backoff_ms); 8 evaluate_fetch call sites in playwright_chat_ui.py + playwright_extra_ui.py pick up the retry without change. --------- Co-authored-by: danielhanchen <info@unsloth.ai> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-05-27 04:53:38 -07:00
Daniel Han	849da89605	Fix unsloth studio update silently downgrading on macOS arm64 (#5767) ... * Fix unsloth studio update silently downgrading on macOS arm64 Root cause: studio/install_python_stack.py's "Updating base packages" step passes `--upgrade-package unsloth -r base.txt -c constraints.txt` with base.txt's `unsloth` and `unsloth-zoo` entries unpinned. On macOS arm64 the resolver silently backtracks to an older unsloth (2026.5.2 or even 2025.7.2) whenever a transitive constraint (the most common one is bitsandbytes wheel availability: 0.49.0+ ships macosx_14_0_arm64 wheels, older versions do not) makes the unpinned requirement satisfiable by an older release. install.sh already maintains an explicit `unsloth>=N.N.N` floor for the same reason, but the floor was missing from the in-venv update path. Reproduced on macos-14 across 2026.3.18 / 2026.4.8 / 2026.5.2 / 2026.5.6 starting states. All four ended on unsloth==2026.5.2 after a clean `unsloth studio update` invocation (2026.5.6 was a true downgrade, others were stale or partial advances). Fix mirrors install.sh: query PyPI at runtime for the current latest version of unsloth and unsloth-zoo, then pass `unsloth>=<latest>` and `unsloth-zoo>=<latest>` as extra positional pins alongside the existing `--upgrade-package` flags. Network failures fall back to the historical unpinned behaviour so offline installs continue to work. Applied to all three upgrade branches (standard update, local-repo overlay, no-torch). Also fix the cosmetic `Hardware detected: MLX -- Apple Silicon (i386)` banner. platform.processor() reads `uname -p` which returns "i386" on many universal2-shaped Python builds even on a native arm64 interpreter; platform.machine() is the reliable source ("arm64" once is_apple_silicon has gated us). * Dedup floor-pin call sites + LRU cache PyPI lookup Three upgrade branches each rebuilt the same conditional `unsloth>=` / `unsloth-zoo>=` arg list with two PyPI round-trips per branch -- six round-trips per `unsloth studio update` invocation. Extract a `_pin_floor_args(*, include_unsloth=True)` helper and wrap `_resolve_latest_pypi_version` in `functools.lru_cache` so the three branches share a single PyPI request per package. Functionally equivalent; pure cleanup on top of the previous commit. * Warn when PyPI is unreachable so the silent fallback is visible If `_resolve_latest_pypi_version` returns None for either lookup the floor args are silently dropped, which restores the pre-fix resolver behaviour. Print a single cyan `warning` line in `_pin_floor_args` when that happens so users behind a proxy / captive portal / firewalled PyPI mirror know the upgrade has degraded -- and can supply network egress or a `--index-url` mirror and retry. * Soft floor with unpinned-fallback for hosts where floor is unsatisfiable Reviewer found that the unconditional unsloth-zoo>=LATEST floor turns a previously-resolvable macOS 13 arm64 update into a hard resolver failure: unsloth-zoo 2026.5.4 requires mlx-vlm>=0.4.4 -> mlx>=0.30.0, and mlx 0.30+ only publishes macosx_14_0_arm64 wheels. The pre-fix behaviour backtracked to an older unsloth instead of erroring. We should not turn "stale" into "fail". Add pip_install_with_floor_fallback: first try the install with the floor appended; if the resolver cannot satisfy it (subprocess exit code != 0), retry the install without the floor and print a clear warning. The fall-through preserves the legacy "succeed-but-stale" contract on hosts where wheel availability is the bottleneck. Also extend pip_install_try with a req= kwarg so the floor attempt can pass `-r base.txt` like pip_install does, and add an UNSLOTH_NO_PYPI_FLOOR=1 opt-out for air-gapped CI / corporate PyPI mirrors that intentionally do not expose pypi.org directly. All three upgrade branches (standard, local-repo, no-torch) now go through the helper so the fallback behaviour is consistent. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add second fallback level: floor without constraints macOS arm64 floored attempt with -c constraints.txt fails because the single-env constraint `transformers==4.57.6` conflicts with the new unsloth-zoo 2026.5.4 -> mlx-vlm 0.4.4+ -> transformers>=5.1.0 chain. First fallback level retries the floored install without constraints (transformers freely resolves to a mlx-vlm-compatible version); downstream pip_install calls still apply constraints.txt to anything that doesn't transitively conflict. If THAT still fails (wheel availability rather than constraint conflict), drop the floor and fall back unpinned as before. Verified locally with uv pip compile against aarch64-apple-darwin python-3.13: strict-constrained floor errors, no-constraint floor resolves cleanly to unsloth==2026.5.7 + unsloth-zoo==2026.5.4 + transformers==5.5.0 + mlx-vlm==0.5.0. * setup.sh/.ps1: also gate fast-path on unsloth-zoo being up to date The version-check fast-path in setup.sh / setup.ps1 only looked at unsloth itself. If unsloth was at the PyPI latest but unsloth-zoo was stale, the gate set _SKIP_PYTHON_DEPS=true and install_python_stack.py never ran -- so the new floor pin from PR #5767 had no effect for the exact "unsloth at latest, zoo behind" state several reviewers flagged. Probe both packages' installed-vs-latest versions and only skip the deps step when BOTH match. When either is behind, fall through to install_python_stack.py so the new resolver fix gets a chance to run. Verified setup.sh with `bash -n`; the setup.ps1 change uses PowerShell if-expressions for the null-default pattern rather than bash-style ${var:-default} which is not valid PowerShell. * Skip unsloth-zoo floor too for custom no-torch test packages Reviewer found the asymmetric guard: the no-torch branch was already gating the unsloth floor on package_name == "unsloth" (test side packages may not publish to PyPI), but the unsloth-zoo floor was still added unconditionally. A custom no-torch update that ships its own forked zoo metadata could now hit a public PyPI floor that does not match the fork's published version. Add a symmetric `include_zoo` parameter to `_pin_floor_args` and gate both pins on the same `package_name == "unsloth"` check. * Address review feedback: simpler except clause + private-index note Gemini flagged TimeoutError in the PyPI fetch exception list. OSError already covers socket timeouts and the 3.11+ TimeoutError subclass on every supported Python, so drop the redundant entry and explain what each remaining exception catches. Codex flagged that floor lookups against pypi.org could break installs behind a lagging private mirror. Step 3 of pip_install_with_floor_fallback already recovers transparently in that case; expand the docstring so the behavior is discoverable without reading the body. * extras-no-deps: skip transformers==4.57.6 on macOS arm64 Reviewer flagged that the resolver-selected transformers from the no-constraints base step on macOS arm64 (transformers 5.x for mlx-vlm 0.4.4+) gets silently downgraded back to 4.57.6 by extras-no-deps.txt during the very next step, breaking mlx-vlm imports at runtime even though unsloth itself reports as latest. Add a PEP 508 platform marker so the pin only applies off macOS arm64. constraints.txt still enforces 4.57.6 everywhere else; mlx-vlm only publishes wheels for darwin arm64, so other platforms are unaffected. * setup.sh/.ps1: gate fast-path zoo probe on _PKG_NAME == unsloth Reviewer found the asymmetric custom-package regression: the new zoo-aware fast-path probes public unsloth-zoo unconditionally, but a custom STUDIO_PACKAGE_NAME side build may ship its own zoo fork via dependency metadata and not install public unsloth-zoo at all. The previous behaviour (skip Python deps if the custom package itself is at its declared latest) is preserved by only running the zoo probe when the managed package literally IS unsloth. Matches the include_zoo gate already in _pin_floor_args() at install_python_stack.py. * install_python_stack: all-or-nothing floor + uv-to-pip retry Two reviewer findings on the floor-pin helpers: 1. _pin_floor_args() previously kept a half-floor if one PyPI lookup succeeded and the other failed. With unsloth at latest but the zoo lookup down, the resolver could still backtrack zoo while we required unsloth at latest, defeating the pin. Return [] on any lookup failure so the unpinned legacy path runs cleanly. 2. pip_install_try() ran ONLY uv when USE_UV was true; a uv-specific failure short-circuited to False even when pip itself could have applied the floor. Mirror pip_install()'s uv-to-pip fallback: try uv, fall through to pip on non-zero exit, and only then give up. * extras-no-deps: rewrite marker without `not` for PEP 508 parsers pip's vendored packaging rejects `not (...)` in PEP 508 markers; the grammar only specifies `and` / `or` between boolean atoms. The staging macos-14 matrix failed every job at "Installing extras (no-deps)" with `Expected a marker variable or quoted string`. Apply De Morgan's law so the marker uses `or` between two `!=` checks, which both pip and uv parse cleanly. Behaviour identical: skip the 4.57.6 pin only on darwin arm64; pin everywhere else. * constraints: skip transformers==4.57.6 pin on macOS arm64 too Marker-gating the extras-no-deps.txt pin was not sufficient. Every subsequent pip_install in the update pipeline passes -c single-env/constraints.txt, and constraints.txt itself pinned transformers==4.57.6 unconditionally. The latest staging-2 run shows the base step's no-constraints fallback installed transformers 5.5.0 correctly, but a later constrained step (extras / studio / data-designer deps) silently downgraded it back to 4.57.6, leaving mlx-vlm 0.5.0 in the venv with an unsatisfied transformers>=5.5.0 requirement. Apply the same `sys_platform != "darwin" or platform_machine != "arm64"` marker to the constraints.txt entry so it is inert on darwin arm64. Other platforms still pin 4.57.6 because mlx-vlm only publishes wheels for darwin arm64; no other platform is affected. * constraints: carve out darwin arm64 from every == pin Marker-gating only transformers was not enough; staging-2 still failed with the same `transformers==4.57.6 in venv after the update` outcome because the resolver hit a `huggingface-hub==0.36.2` (and adjacent) conflict with mlx-vlm's `huggingface-hub>=1.5.0` requirement, then fell back to a stale stack even after my no-constraints level fired on the base step. Apply the same `sys_platform != "darwin" or platform_machine != "arm64"` marker to every == pin in constraints.txt. Range pins (mcp, fastmcp, websockets) stay active everywhere because they do not conflict with the mlx-vlm chain. mlx-vlm only publishes wheels for darwin arm64, so no other platform is affected. * install_python_stack: also --upgrade-package transformers and mlx-vlm Staging-2 showed that even after the constraints.txt carve-out for darwin arm64, the venv still ended up with the OLD `transformers==4.57.6` paired with a NEW `mlx-vlm==0.5.0` from unsloth-zoo's transitive upgrade. The resolver's --upgrade-package flag only freshens the named packages and their newly-pulled transitive deps; transformers was already installed at a version that satisfied unsloth-zoo's range (`>=4.51.3,<=5.5.0` with exclusions), so the resolver did not upgrade it -- even though mlx-vlm 0.5.0 requires `transformers>=5.5.0`. Add `--upgrade-package transformers` and `--upgrade-package mlx-vlm` to all three base-step branches. Both are no-ops when the package is absent (mlx-vlm only ships wheels on darwin arm64); on darwin arm64 this is what nudges the resolver to upgrade both together so the final venv is internally consistent. On Linux/Windows, transformers stays at 4.57.6 because constraints.txt still pins it there and mlx-vlm never enters the resolution. * install_python_stack: explicit mlx-vlm + transformers realign on macOS arm64 Even with --upgrade-package hints, uv leaves the venv with the already-installed transformers (4.57.6 inherited from the OLD venv's constrained install) when that version still happens to satisfy unsloth's own metadata range -- but it does not also re-resolve mlx-vlm's stricter `transformers>=5.5.0` requirement, so the venv ends up with mlx-vlm 0.5.0 paired with transformers 4.57.6 and mlx-vlm imports break at runtime. After the base step, on darwin arm64 only, run an explicit `pip install --upgrade mlx-vlm transformers` with constrain=False. This forces both packages through the resolver again as direct top-level requirements, so transformers is pulled up to whatever mlx-vlm's metadata requires (5.5.0 today). No effect on any other platform because mlx-vlm has no wheels off darwin arm64 and the branch is gated on IS_MAC_ARM. * requirements: marker-gate every == pin that conflicts with mlx-vlm chain Staging-2 kept ending up with transformers==4.57.6 even after the realign step, because studio.txt unconditionally pins huggingface-hub==0.36.2 (and datasets==4.3.0). Installing studio.txt with constraints active pulls the resolver back to a huggingface-hub that only recent transformers (4.x) supports, which silently downgrades the realigned 5.5.0 to 4.57.6 -- exactly the inconsistency we tried to prevent. Also extras-no-deps.txt still pinned trl==0.23.1 unconditionally; the 0.23.1 wheel transitively requires huggingface-hub<1, same coupling. Marker-gate all three. The carve-out is identical to constraints.txt's: inactive on darwin arm64 (where the mlx-vlm chain dictates newer versions), active everywhere else (where Linux/Windows users rely on the single-env pins). mlx-vlm only publishes wheels for darwin arm64 so no other platform is affected. * realign: --force-reinstall mlx-vlm + transformers + huggingface_hub Plain --upgrade does not force uv to re-resolve mlx-vlm's transformers requirement when the already-installed transformers happens to satisfy unsloth's own range. Switch to --force-reinstall on the three packages so the resolver tears them down and brings them back together with consistent versions. Include huggingface_hub because transformers 5.x requires hf-hub>=1.5.0 and the resolver would not touch it otherwise. * realign: pin transformers via mlx-vlm's own metadata spec `pip install --force-reinstall mlx-vlm transformers` still resolved to an already-installed transformers 4.57.6 because uv treats it as satisfying unsloth's transformers range without re-checking mlx-vlm's stricter requirement. Pull mlx-vlm's actual transformers specifier from its installed metadata at runtime and pass it as an explicit version requirement (e.g. `transformers>=5.5.0` for mlx-vlm 0.5.0). That removes the resolver's wiggle room: it MUST pick a transformers satisfying mlx-vlm AND unsloth, which on darwin arm64 with the latest unsloth-zoo means transformers==5.5.0. Falls back to unpinned `transformers` if metadata read fails, so this never errors. * realign: uninstall-then-install to bypass uv's incumbent bias Every flag-based approach failed: --upgrade, --upgrade-package, --force-reinstall, and even an explicit `transformers>=5.5.0` requirement all left the venv with transformers==4.57.6 because uv treats the already-installed version as satisfying unsloth-zoo's range and refuses to disturb it, even when it does not satisfy mlx-vlm's stricter requirement. Replace the realign step with an explicit uninstall of the conflicting trio (transformers / mlx-vlm / huggingface_hub) followed by a fresh install. With no transformers in the venv, the resolver MUST pick a version satisfying every installed package's metadata, which on darwin arm64 with the latest unsloth-zoo is uniquely 5.5.0. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim verbose comments across PR #5767 changes * Simplify mac-arm64 fix: install MLX stack with --no-deps The previous approach (PyPI floor pin + 3-level fallback + macOS arm64 realign step + marker carve-outs on every == pin) was fighting symptoms. The root cause is that unsloth-zoo declares mlx-vlm>=0.4.4 as a darwin arm64 dep, and mlx-vlm 0.5.0's metadata pulls in transformers>=5.5.0, which conflicts with the main venv's transformers==4.57.6 pin and forces the resolver to backtrack unsloth. Severing that chain at its source: install mlx + mlx-metal + mlx-lm + mlx-vlm with --no-deps BEFORE unsloth-zoo. The resolver sees mlx-vlm already installed (>=0.4.4) and never inspects its transformers metadata. Per-model transformers version routing is already handled at runtime by the side-car venvs in utils/transformers_version.py (.venv_t5_530 for Ministral/GLM/Qwen3 MoE, .venv_t5_550 for Gemma 4). Net change: -224 / +71 lines across install.sh, install_python_stack.py and the three requirements files. Reverted: - _resolve_latest_pypi_version + _pin_floor_args + pip_install_with_floor_fallback - macOS arm64 realign step (pip uninstall + reinstall) - --upgrade-package transformers --upgrade-package mlx-vlm in base steps - All ; sys_platform != "darwin" or platform_machine != "arm64" markers in constraints.txt, studio.txt, extras-no-deps.txt - pip_install_try restored to its pre-PR signature Added: - install.sh: Apple Silicon MLX --no-deps install before unsloth (both fresh and migrated branches) - install_python_stack.py: same step gated on IS_MAC_ARM and not skip_base Kept (independent bugs): - setup.sh / setup.ps1 dual-package zoo version check - platform.processor() -> platform.machine() hardware-detect fix * Minimise PR to mac-arm64-specific changes only Revert setup.sh and setup.ps1 to main -- the dual-package zoo check was defensive and not strictly needed once mlx-vlm is installed --no-deps (the resolver-backtrack scenario that produced stale zoo no longer happens). Tighten remaining comments in install.sh and install_python_stack.py. Final PR-attributable changes: install.sh +24/-5 (MLX --no-deps in 2 places) studio/install_python_stack.py +19 (MLX --no-deps + IS_MAC_ARM) studio/backend/utils/hardware/hardware.py +6/-6 (processor() -> machine()) studio/backend/requirements/*.txt unchanged * Revert "Minimise PR to mac-arm64-specific changes only" This reverts commit 9470daa855. * Revert "Simplify mac-arm64 fix: install MLX stack with --no-deps" This reverts commit f8a43b87e8. * Revert "Trim verbose comments across PR #5767 changes" This reverts commit c3f293a10f. * Simplify mac-arm64 fix: --no-deps MLX + METADATA patch Root cause: unsloth-zoo declares mlx-vlm>=0.4.4 as a darwin-arm64 dep, and mlx-vlm 0.5.0's published metadata declares transformers>=5.5.0. Every subsequent resolver run with constraints.txt's transformers==4.57.6 sees the conflict and backtracks unsloth to escape it (user-reported downgrade). The aggressive pin doesn't reflect what mlx-vlm actually requires at top-level import time -- the symbols it loads (AutoProcessor, AutoTokenizer, ProcessorMixin, BatchFeature) are stable across transformers 4.51+. Model- specific submodules that genuinely need 5.x APIs are only loaded once the 3-tier transformers dispatcher (utils/transformers_version.py) has activated the matching .venv_t5_530 / .venv_t5_550 side-car at runtime. Fix: on Apple Silicon, install the MLX stack with --no-deps then rewrite mlx-vlm/mlx-lm's installed METADATA to declare transformers>=4.51.3. Now the resolver sees mlx-vlm 0.5.0 as compatible with the main venv's transformers==4.57.6 and there's nothing to backtrack. Reverts the previous heavy machinery: - _resolve_latest_pypi_version, _pin_floor_args, pip_install_with_floor_fallback - macOS arm64 realign step (pip uninstall + reinstall) - --upgrade-package transformers --upgrade-package mlx-vlm in base steps - All ; sys_platform != "darwin" or platform_machine != "arm64" markers in constraints.txt / studio.txt / extras-no-deps.txt - setup.sh / setup.ps1 dual-package zoo check (Windows never had the bug; with this fix in place stale zoo no longer happens on macOS either) - pip_install_try restored to pre-PR signature Kept: - install.sh: MLX --no-deps install in fresh + migrated branches - install_python_stack.py: same step gated on IS_MAC_ARM and not skip_base - _relax_mlx_metadata() helper, called immediately after each MLX install - studio/backend/utils/hardware/hardware.py: platform.processor() -> platform.machine() cosmetic fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Use UV_OVERRIDE to relax mlx-vlm transformers pin uv supports --overrides / UV_OVERRIDE which globally overrides any package's stated dependency requirement. mlx-vlm 0.5.0 declares transformers>=5.5.0 and mlx-lm 0.31.3 declares transformers>=5.0.0; neither is true at top-level import time (their imports use AutoProcessor / AutoTokenizer / ProcessorMixin / BatchFeature which are stable across transformers 4.51+). Per-model 5.x routing is handled at runtime via the .venv_t5_530 / .venv_t5_550 side-cars. Override file (overrides-darwin-arm64.txt) declares transformers>=4.51.3 ; exported via UV_OVERRIDE env var on Apple Silicon by both install.sh and install_python_stack.py. uv then resolves mlx-vlm as compatible with the main venv's transformers==4.57.6 (constraints.txt) and unsloth advances cleanly to LATEST. Drops, vs. the previous attempts: - _resolve_latest_pypi_version + _pin_floor_args + pip_install_with_floor_fallback (floor-pin machinery -- replaced by single UV_OVERRIDE line) - macOS arm64 realign step (pip uninstall + reinstall) - --upgrade-package transformers --upgrade-package mlx-vlm in base steps - All ; sys_platform != "darwin" or platform_machine != "arm64" markers - _relax_mlx_metadata() helper + sed METADATA patch (uv reads from index, not dist-info, so dist-info patches were ineffective) Kept: - install.sh / install_python_stack.py: MLX latest install on Apple Silicon (now without --no-deps, the override lets the resolver pick a consistent set) - studio/backend/utils/hardware/hardware.py: platform.machine() cosmetic fix * Trim UV_OVERRIDE comments; bump override floor to 4.57.6 Match the main venv's constraints.txt pin exactly so the override file reads as the actual installed version rather than mlx-vlm's API floor. Comments collapsed to one-liners where possible. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-05-26 07:23:13 -07:00