1 commit

Author	SHA1	Message	Date
Daniel Han	52a9601032	Keep import unsloth working when bitsandbytes is absent (#7502) ... * Keep `import unsloth` working when bitsandbytes is absent device_type.py already prints "bitsandbytes is not installed - 4bit QLoRA unallowed, but 16bit and full finetuning works" and clears ALLOW_BITSANDBYTES / ALLOW_PREQUANTIZED_MODELS, but the import chain then hard-required the module anyway, so `import unsloth` raised instead. #7354 made this reachable: the gfx906 install path uninstalls the generic bitsandbytes wheel (no gfx906 kernels in it), which leaves an MI50 / Radeon VII host unable to import unsloth at all, not on the 16bit path the message promises. - kernels/utils.py: guard the bnb import; bind get_ptr and the five 4bit ctypes handles to a stub that raises a clear message if a 4bit path is entered. HAS_CUDA_STREAM stays False, which is the correct route. - save.py, models/granite.py: guard Bnb_Linear4bit and peft's Linear4bit (peft exports it only when bnb imported cleanly) with placeholder classes. Both names only feed isinstance checks, so nothing matching is exact. - _gpu_init.py: same degradation on the xpu branch as the cuda branch above. Verified on a Strix Halo (gfx1151, DEVICE_TYPE=hip, torch 2.11.0+rocm7.13.0) by blocking bitsandbytes with sys.modules["bitsandbytes"] = None, so find_spec returns None and the import raises exactly as when the package is absent. Before: ModuleNotFoundError at kernels/utils.py:136. After: import succeeds, FastLanguageModel/FastModel import, ALLOW_BITSANDBYTES=False, ALLOW_PREQUANTIZED=False, and the 4bit stub raises with the real cause. With bitsandbytes present, every binding is unchanged. New test walks the `import unsloth` module graph with ast and fails on any unguarded bitsandbytes (or peft Linear4bit) import; verified it catches the old code. Targeted suites: 702 passed, 18 skipped. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address the review on #7502: zoo coupling, non-hip flags, py3.9 collection Three findings, each reproduced first and negative-controlled after. 1. The fix still needed an unreleased unsloth_zoo (P1). save.py imported unsloth_zoo.saving_utils at module scope, and any zoo without the companion #953 fix imports bitsandbytes there, so `import unsloth` kept failing for a dependency set pyproject.toml allows. Raising the floor was not an option: PyPI's newest zoo is 2026.7.6 and #953 is merged but unreleased, so a bump would break every install today. Both names it pulled in are used only inside functions, so the import is now lazy at those two call sites, matching what determine_base_model_source in the same file already does. Verified against a real pre-#953 zoo checkout with bitsandbytes blocked: import succeeds, and restoring the eager import reproduces the failure at saving_utils.py:70. This PR no longer depends on a zoo release. 2. Capability flags were only cleared on hip (P2). device_type.py probed bitsandbytes inside its DEVICE_TYPE == "hip" branch, so a cuda or xpu host without bnb imported fine but still reported ALLOW_BITSANDBYTES=True, and the default load_in_4bit=True path in models/loader.py would select a 4bit checkpoint before failing. Clear both flags whenever the module is absent, on every backend, via find_spec so a working install pays nothing. A cuda host with bnb blocked now reports False/False; with bnb present nothing changes. 3. The new test could not be collected on Python 3.9 (P2). `Path | None` is a PEP 604 union and requires-python still allows 3.9, so pytest raised TypeError at import. Added `from __future__ import annotations`. Checked in real uv venvs on 3.9, 3.10 and 3.13: 2 passed each; removing the future import reproduces "unsupported operand type(s) for |" on 3.9 only. The xpu branch in _gpu_init.py needs no separate flag handling now that the probe is backend-independent. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address the second review on #7502: guarded probe, and 8bit in the same guard 1. The capability probe used find_spec while the fallbacks in kernels/utils.py and _gpu_init.py treat any import failure as unavailable, so an installed but unusable wheel would leave ALLOW_BITSANDBYTES true while the kernels had already bound the stub. Probe with the same guarded import instead, so all three agree by construction. No new cost on any path: _gpu_init.py already imports bnb before device_type is reached on cuda, and device_type's own hip block imports it a few lines later. Worth recording that the state this prevents is currently unreachable for an unrelated reason: a broken wheel takes `import unsloth` down earlier, in transformers/integrations/bitsandbytes.py:20 via unsloth_zoo/patching_utils.py:680, whichever exception it raises (OSError also escapes the zoo moe_utils `except ImportError`). So this is correctness for when those imports get guarded, not an observable fix today. 2. Both loader guards printed for load_in_4bit or load_in_8bit but only cleared load_in_4bit, so an explicit load_in_8bit=True survived and reached Transformers, which builds the bnb quantizer and fails there. Clear both. The message no longer says AMD either: the flag now goes false whenever bnb is unusable on any backend. Tests: the probe must not use find_spec, and an ast walk requires every ALLOW_BITSANDBYTES guard in loader.py to clear both flags, so a third guard cannot be added with the same omission. Dropping either fix reddens them (1 and 2 failures respectively). 4 passed on 3.9, 3.13 and the ROCm venv; absent and healthy bnb both stay consistent across hip and cuda. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop the importlib import left over from the find_spec probe on #7502 * Address the third review on #7502: exact-name bypass and a forwarded bnb config Both findings hold up, so both are fixed. 1. use_exact_model_name=True skipped the guard entirely. load_in_4bit defaults to True, so on a host without bitsandbytes FastLanguageModel.from_pretrained(name, use_exact_model_name=True) kept 4bit set and failed downstream. That option suppresses repo-name remapping and cannot make bitsandbytes available, so it has no business gating a capability check. Ungated at both sites. 2. A user-supplied quantization_config survived the fallback. It sets load_in_4bit/8bit at the top of from_pretrained and stays in kwargs, so clearing the local flags still let Transformers rebuild the bnb quantizer. Now dropped as part of the fallback. One correction to the second suggestion: it cannot be dropped whenever the fallback runs. quantization_config also carries GPTQ, AWQ, fp8 and torchao configs, which have nothing to do with bitsandbytes and must reach the loader untouched. The pop is gated on the config actually requesting load_in_4bit or load_in_8bit, reusing the same dict/attr probe from the top of the function. Behaviour, exercising the real guard block against synthetic inputs with use_exact_model_name=True and bnb unusable: default 4bit, no cfg 4bit=False 8bit=False explicit 8bit, no cfg 4bit=False 8bit=False BitsAndBytesConfig(4bit/8bit) 4bit=False 8bit=False config dropped dict bnb config 4bit=False 8bit=False config dropped GPTQ config 4bit=False 8bit=False config SURVIVES fp8 dict 4bit=False 8bit=False config SURVIVES Nothing changes when bitsandbytes works: the whole block is inside `if not ALLOW_BITSANDBYTES`. Tests: an ast walk requires neither guard to reference use_exact_model_name in its test, and requires each to pop quantization_config behind a _wants_bnb check, so an unconditional pop fails too. Re-gating one guard or removing one pop reddens a test each. 6 passed on 3.9, 3.13 and the ROCm venv. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address the fourth review on #7502: FastModel never reached the 16bit path Both findings are real, and the second one meant this PR did not actually deliver what it advertises for FastModel or vision loads. Reproduced first. 1. patch_compiling_bitsandbytes() ran unguarded at the top of FastModel.from_pretrained, and unsloth_zoo's copy imports bitsandbytes unconditionally (patching_utils.py:40). So every FastModel call on a bnb-less host died there, whatever the arguments: FastModel(load_in_16bit=True) -> ModuleNotFoundError at patching_utils.py:40 FastModel(full_finetuning=True) -> ModuleNotFoundError at patching_utils.py:40 The FastLanguageModel path already wraps this call in try/except with a warning, and its comment even says "Mirror FastModel" - FastModel was the unwrapped one. Wrapped it the same way, so behaviour is unchanged wherever bitsandbytes imports. 2. The mode-exclusivity check ran before the capability fallback. load_in_4bit defaults to True, so load_in_16bit=True made int(load_in_4bit) + int(load_in_16bit) == 2 and raised "Can only load in 4bit or 8bit or 16bit" before the fallback could clear the unavailable 4bit request. Moved the fallback ahead of that check. After both, the same three calls get past every bitsandbytes gate and reach model resolution, failing only on the deliberately fake repo name used by the probe. Nothing changes when bitsandbytes works: the fallback is still inside `if not ALLOW_BITSANDBYTES`, and the wrapper only swallows an import that previously crashed the load. Tests: the mode check must be preceded by an ALLOW_BITSANDBYTES fallback in the same function, and no call to patch_compiling_bitsandbytes may sit outside a try. The ordering assertion is scoped to the enclosing function on purpose - my first version compared line numbers file-wide, so the other loader's guard satisfied it and the negative control passed when it should have failed. With the scoping fixed, moving the fallback back after the mode check reddens it, as does unwrapping the patch call. 8 passed on 3.9, 3.13 and the ROCm venv. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-28 08:03:54 -07:00