mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-17 12:53:59 +00:00
85a036a195
* Keep an explicitly requested float32 model in float32 without bfloat16
A model loaded with dtype = torch.float32 was being wrapped in float16
autocast on V100/T4 even when the caller set fp16 = False and bf16 = False.
The mixed precision block in rl.py reads "neither flag set" as "caller did
not choose" and picks for them:
use_bf16_amp = (not float16) and _bf16_supported()
args.fp16 = not use_bf16_amp
so on a GPU without bf16 the answer is always float16. float16 has five
exponent bits against float32's eight, so a value the model was loaded wide
enough to hold overflows to inf and then NaN.
Spark_TTS_(0_5B) is the case that surfaced it: it loads with
dtype = torch.float32 ("Spark seems to only work on float32 for now"), sets
fp16 = False, bf16 = False, and on a T4 logs nan for every step, then dies
at inference inside torch.multinomial, which refuses a distribution
containing NaN. The sampler assert is two steps downstream of the cause.
Suppressing the autocast for every float32 model is too wide: full
finetuning upcasts trainable weights to float32 by itself, and float16
autocast over float32 master weights is the ordinary V100/T4 recipe
(issue #4082, tests/python/test_v100_fullft_precision.py). So
from_pretrained now records whether float32 was ASKED for in
UNSLOTH_USER_FLOAT32, as opposed to arrived at by upcasting, and only an
explicit request suppresses the autocast. It is set in both loaders, since
FastLanguageModel and FastBaseModel resolve dtype independently, and written
on every load so a previous run cannot leave a stale "1" behind.
Only the float16 fallback changes. bf16 has float32's exponent range, so a
bf16 GPU keeps autocasting and keeps the memory saving. An explicit
fp16 = True or bf16 = True is still obeyed, float16 and bfloat16 models are
untouched, and UNSLOTH_FORCE_FLOAT32 models still take their earlier branch.
tests/test_float32_no_fp16_autocast.py pulls the mixed precision block out
of rl.py by AST and executes it against fake args/model objects with the
bf16 capability check stubbed, so the no-bf16 path is covered without a
V100/T4. No GPU, no network, no trl import.
Also, separately: do not let one optional audio dependency kill the whole
test session. require_python_package calls sys.exit(1) when a package is
missing, and every caller is a test module invoking it at import time, so
under pytest that is not a skip of one module -- SystemExit propagates out
of collection and the run ends with
INTERNALERROR> SystemExit: 1
with no report at all. A missing xcodec2 took out 1000+ unrelated tests. It
now skips the module under pytest; outside pytest the exit is unchanged,
since these helpers are also used by standalone scripts. test_whisper.py had
the same shape for a different reason: it downloads an audio fixture at
import and did `assert False` on failure. Wikimedia rate limits and this URL
really did answer 429 during a batch run, and an asset we could not fetch
says nothing about unsloth, so it skips too.
* Record the float32 request on the model, and let GRPO honour a disabled autocast
Two real gaps in the float32 gating, plus the provenance problem underneath both.
The gate never reached most users. llama, mistral, gemma, gemma2, qwen2 and
qwen3 LoRA/QLoRA loads go through FastLanguageModel to dispatch_model.from_pretrained,
which is neither of the two loaders that recorded the request, so user_float32
stayed false and the fp16 autocast this branch exists to avoid ran anyway. That
is most of the notebooks.
The value was also process-global and derived rather than asked for. A program
that loads two models before building a trainer described whichever loaded last,
and a dtype inferred from a 4bit config's bnb_4bit_compute_dtype was recorded as
an explicit model-wide request, which is especially wrong under full finetuning
where the loader then turns 4bit off again. Both entry points now read the
argument as the caller wrote it, before any normalisation, and the answer is
attached to the model. The outermost caller wins, since only it saw the raw
argument; the two delegating returns in FastLanguageModel re-stamp it for that
reason.
Then ACCELERATE_MIXED_PRECISION = 'no' was read as a two-way switch in the GRPO
replacements, so 'no' came out as bfloat16 and autocast was entered anyway. torch
does not ignore that on a T4 or V100, it raises "Current CUDA Device does not
support bfloat16", which is exactly the hardware this branch targets. Verified
against torch's own autocast __init__: the check is gated on `enabled`, so
disabling autocast is both the correct reading of 'no' and the minimal fix.
Full finetuning already set 'no', so this was reachable before this PR too.
Tests: 4384 passed against 4369 on the same tree with these changes stashed,
identical 1310 environment failures either way, and nothing failing on this
branch that is not also failing on its base.
* Keep forced float32 on fp16 autocast, size GRPO chunks by the flag, and run the tests off a GPU
Three follow-ups on the autocast change.
UNSLOTH_FORCE_FLOAT32 exports ACCELERATE_MIXED_PRECISION 'no' as well, and the
new gate read only that, so the injected _prepare_inputs header disabled
autocast for it while the dtype expression right beside it was still choosing
float16. That left Gemma3 and gpt-oss generation in full float32, and it
disagreed with the two other consumers, which set _autocast_enabled True for
exactly this case. The header honours the flag now.
The GRPO chunk autotuner derived dtype_bytes from _autocast_dtype alone, which
stays bfloat16 when autocast is off, so it sized hidden states and logits as
16-bit for a forward running in float32. That is half the real figure, on the
explicit-float32 and full-finetuning runs this branch is for.
The new tests drove torch.amp.autocast(device_type = "cuda") on a box that has
CUDA. On a CPU runner torch warns "CUDA is not available. Disabling" and hands
back a no-op, so the premise test would not have raised and every enabled case
would have read False, failing for reasons unrelated to the code. Measured, then
fixed by claiming the device rather than skipping, since a CPU job is where these
would most need to run. They pass with the GPUs visible and with
CUDA_VISIBLE_DEVICES empty.
Tests: 4386 passed, same 1310 environment failures as the branch base, nothing
failing here that is not also failing there.
* Size the GRPO chunk on the dtype the forward runs in
The autocast-off branch assumed float32, but pure bfloat16 full finetuning
also disables autocast (ACCELERATE_MIXED_PRECISION='no') while keeping bf16
weights and a bf16 forward. That path got dtype_bytes=32 where main gave 16,
cutting the chunk ~1.5x and doubling the forward-pass count on 10-22GB cards.
Read lm_head.dtype instead: float32 for an explicit float32 load, bfloat16
here. Retargets the test that pinned the old value.
* Apply ruff kwarg-spacing formatting
pre-commit.ci's ruff-format-with-kwargs hook reported "files were modified by
this hook" and could not push the result back, so the check stayed red.
Applied locally instead.
Three source-asserting tests had to be made robust rather than re-pinned,
since the reformatting is legitimate and will happen again:
- test_the_flag_is_recorded_beside_the_dtype counted the literal
"self._autocast_enabled = (", which the formatter collapsed onto one line.
Now pairs each _autocast_dtype initialiser with a nearby flag assignment by
position, so it still catches a genuinely missing one.
- test_the_original_error_is_still_reported anchored on a whole f-string
literal, which the formatter merged with the hint that follows it. Anchors
on the message text alone now; either shape satisfies the intent.
- test_the_source_beats_the_version_fallback exposed a real fragility rather
than a test problem: the detector matched the literal "kw_only=True", so any
whitespace in transformers' own source would have read an install that needs
nothing as one that needs patching. Made whitespace-tolerant.
* Latch the GRPO autocast decision on the trainer
* Add the license header to the new test files
* Stamp the forced float32 answer on the model
UNSLOTH_FORCE_FLOAT32 is process wide and from_pretrained clears it on every
load, so a model loaded after a Gemma3 or gpt-oss trainer was built left '0'
behind and GRPO's first generation dropped the float16 autocast that rl.py's
'no' was written expecting, running generation in full float32.
The loaders now stamp the answer on the model they loaded, next to the
existing float32 request marker, and the trainer reads it from there. Models
without the stamp fall back to the environment as before.
* Preserve a bf16 trainer when a forced-float32 model was loaded (#7867)
* Stamp the forced float32 answer on every loader return path (#7867)
* Stamp the float32 answers on the text-diffusion dispatch (#7867)
* Apply the repo ruff-format hook to a test touched by this PR
* Apply the repo ruff-format hook (ruff 0.6.9) to files this PR touches
* Fix the trainer reading the process-wide forced float32 flag for PR #7867
* Fix a missing optional dependency aborting the whole pytest session for PR #7867
require_package exited the process when a system package was absent. The TTS
test modules call it at import time, so under pytest the SystemExit escaped
collection and ended the run with INTERNALERROR / exit code 3 instead of
skipping one file. It now degrades to a module-level skip under pytest, the
same treatment require_python_package already had, and keeps the exit for the
standalone script callers.
test_whisper.py also imported the unsloth runtime unguarded, which is a
collection error on the Windows runner where triton is absent; guard it with a
module-level skip.
* Inherit an outer autocast by omitting dtype, not by a sentinel
* Read the forced float32 stamp inside native fast generation
* Stamp the full finetuning mode on the model the trainer reads
* Keep one missing-dependency helper after the merge with main
The merge left this branch's _skip_if_pytest alongside main's _missing_dependency, doing the same job, plus two near-duplicate test files. Fold both into main's helper and its test module, keeping this branch's extra coverage of require_package and of the call at module scope.
* Make the generation autocast tests independent of the host having CUDA
torch.autocast(device_type = 'cuda') disables itself when CUDA is absent, so reading _enabled off the constructed object measured the runner instead of the branch, and these could never pass on a CPU-only machine, a Mac or Windows. Record the arguments the code asks for instead. Also skip before importing rl_replacements, which needs unsloth_zoo.
* Apply the repo's ruff kwarg-spacing formatting
* Format these three files with the ruff the hook actually pins
The ruff-format-with-kwargs hook installs ruff==0.6.9 through
additional_dependencies, and 0.6.9 and 0.15 disagree on where the message
goes in a multi-line assert and on how a long lambda signature wraps. I
had run the script with the workspace ruff, so pre-commit.ci kept
reformatting these back and failing, with nothing to show in the diff.
AST is identical on all three.
* Drop the nullcontext import this PR no longer uses
Left over from the earlier shape of the autocast gating, which now omits dtype
instead of passing a sentinel context. Source lint's import-hoist check treats an
added-but-unused hoisted import as a blocker, and it is right: nothing in this
file references it.
* Tighten comments for the float32 dtype gating change
---------
Co-authored-by: danielhanchen <unslothshared@gmail.com>
93 lines
3 KiB
Python
93 lines
3 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
"""A missing optional dependency must skip one module, not kill the session."""
|
|
|
|
import subprocess
|
|
import sys
|
|
import textwrap
|
|
|
|
import pytest
|
|
|
|
from tests.utils.os_utils import require_package, require_python_package
|
|
|
|
|
|
_ABSENT = "unsloth_definitely_not_a_real_package_xyz"
|
|
_ABSENT_SYSTEM = "unsloth-nonexistent-system-package"
|
|
_ABSENT_EXECUTABLE = "unsloth-nonexistent-executable"
|
|
|
|
|
|
def _repo_root():
|
|
import pathlib
|
|
return pathlib.Path(__file__).resolve().parents[1]
|
|
|
|
|
|
def test_a_missing_package_skips_the_module_under_pytest():
|
|
with pytest.raises(pytest.skip.Exception):
|
|
require_python_package(_ABSENT)
|
|
|
|
|
|
def test_a_missing_system_package_skips_the_module_too():
|
|
with pytest.raises(pytest.skip.Exception):
|
|
require_package(_ABSENT_SYSTEM, _ABSENT_EXECUTABLE)
|
|
|
|
|
|
def test_an_installed_package_is_a_no_op():
|
|
require_python_package("sys", import_name = "sys")
|
|
require_package("python", sys.executable)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"helper, args",
|
|
[
|
|
("require_package", f"{_ABSENT_SYSTEM!r}, {_ABSENT_EXECUTABLE!r}"),
|
|
("require_python_package", f"{_ABSENT!r}"),
|
|
],
|
|
)
|
|
def test_the_call_at_module_scope_does_not_kill_the_session(tmp_path, helper, args):
|
|
"""The real regression: the helper at import time, in its own pytest session."""
|
|
module = tmp_path / f"test_absent_{helper}.py"
|
|
module.write_text(
|
|
textwrap.dedent(f"""
|
|
import sys
|
|
sys.path.insert(0, {str(_repo_root())!r})
|
|
from tests.utils.os_utils import {helper}
|
|
{helper}({args})
|
|
|
|
def test_unreachable():
|
|
raise AssertionError("the module-level skip should have stopped this")
|
|
"""),
|
|
encoding = "utf-8",
|
|
)
|
|
|
|
result = subprocess.run(
|
|
[sys.executable, "-m", "pytest", str(module), "-q", "-p", "no:cacheprovider"],
|
|
capture_output = True,
|
|
text = True,
|
|
cwd = str(tmp_path),
|
|
)
|
|
output = result.stdout + result.stderr
|
|
assert "INTERNALERROR" not in output, output
|
|
assert "SystemExit" not in output, output
|
|
assert "skipped" in output, output
|
|
# 5 is "no tests ran", which a module-level skip legitimately produces.
|
|
assert result.returncode in (0, 5), f"exit code {result.returncode}\n{output}"
|
|
|
|
|
|
def test_the_standalone_script_path_still_exits():
|
|
# A subprocess, so pytest is genuinely absent from sys.modules, not faked.
|
|
script = textwrap.dedent(f"""
|
|
import sys
|
|
sys.path.insert(0, {str(_repo_root())!r})
|
|
for name in list(sys.modules):
|
|
if name == "pytest" or name.startswith("pytest."):
|
|
del sys.modules[name]
|
|
from tests.utils.os_utils import require_python_package
|
|
require_python_package({_ABSENT!r})
|
|
print("NO EXIT")
|
|
""")
|
|
result = subprocess.run(
|
|
[sys.executable, "-c", script],
|
|
capture_output = True,
|
|
text = True,
|
|
)
|
|
assert result.returncode == 1, result.stdout + result.stderr
|
|
assert "NO EXIT" not in result.stdout
|