53 commits

Author	SHA1	Message	Date
oobabooga	1840f7bd7f	Keep xFormers attention masks on the GPU running each layer (#8516)	2026-08-19 19:28:34 -03:00
Daniel Han	7b13a787cf	Studio: tokenize the dataset online for plain-text single-pass runs (#8960) ... * Studio: tokenize the dataset online for plain-text single-pass runs TRL's tokenizing map is the largest fixed cost of starting a text run: 71s of the 78s of preparation on 100k rows of OpenMathReasoning, with dataset_num_proc already at 8. It is per-row string work, so it can run in DataLoader workers while the GPU is busy instead of blocking the start. Four parts, all needed together: * datasets.with_transform attaches a batched tokenizer that runs on __getitem__. with_transform, not set_transform: the caller's split is also held by the preview and the row-count checks. * TRL gets dataset_kwargs = {"skip_prepare_dataset": True} so it does not run its own map over the view. Feature-detected on SFTConfig's fields plus SFTTrainer.__init__'s source, never assumed. * dataloader_num_workers / prefetch_factor / persistent_workers, sized from the same shared policy that sizes dataset_num_proc and capped at four. * a prewarm barrier inside _preflight_first_batch, which already built a loader and pulled a batch. It now drains max(grad_accum, workers * prefetch) microbatches, and memoizes the train loader -- transformers caches only the eval ones, so without that train() forks a second worker set and drops everything the barrier filled. The transform reproduces unsloth_zoo's sft_prepare_dataset tokenize step exactly: same truncation and max_length, the same double-BOS rule, and the tokenizer's whole output rather than input_ids alone, because the collator and the attention dispatcher both branch on which keys are present. Default ON only for: Linux, plain text, plain tokenizer, map-style datasets.Dataset, packing off, no custom collator, no completion masking, not already tokenized, no token_type_ids, a raw eval split or none, at least 10k rows, and at most one pass over the data. Everything else takes today's path with config_args and the dataset wrapper untouched, and any failure in the gate or the attach degrades the same way. UNSLOTH_STUDIO_ONLINE_TOKENIZATION=0 forces it off; =1 lifts the two cost gates but never a correctness gate. The one-pass rule is what the measurements support: within a single pass the workers stay ahead and there is no steady-state cost (225.45s eager vs 225.33s online over 200 steps), while a lazy view re-tokenizes on every further pass where Arrow would just be read. rl.py: a split may now attest its own truncation width through _unsloth_truncated_to, and the max_length enforcement believes it instead of scanning. Scanning a lazily-tokenizing split reads every row, which is the whole eager tokenize pass again, run inside __init__ where nothing overlaps it -- and the fallback it would then take turns padding-free off. Both copies of the scan honour it, the module-level one and the one inlined into every generated trainer. Measured on one B200, Qwen3-0.6B + LoRA, 100k rows, cold datasets cache: preparation 71.2s -> 0.4s, time to first step 91.9s -> 17.7s. Losses match: the largest per-step gap between the eager and online arms is 7e-4, smaller than the 9e-4 between two eager runs of the same seed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Shut the online path's workers down, and refuse the rows it would fail late on Review of the online tokenization path found two things it got wrong once it was running, rather than in the gate. The persistent DataLoader workers were never torn down. Persistence is what lets the prewarm barrier's workers survive into `train()`, and the memo holds the loader that owns them, so after `train()` returned nothing dropped the last reference: four `pt_data_worker` processes, 5.27 GB resident between them, still alive through merging, quantizing and GGUF export, each a fork of a process that had already initialised CUDA. Measured on one B200, Qwen3-0.6B + LoRA, 12k rows: 4 workers still there fifteen seconds after training ended, and only the process exiting cleared them. `release_train_dataloader` shuts them down and puts the real `get_train_dataloader` back, called from a `finally` around `train()` so it runs before `_finalize_training` rather than after it, and again from the outer `finally` for the paths that return before training starts -- the preflight error has already forked the workers. Same measurement after: 4 workers before the release, 0 after. An accelerate wrapper and the loader inside it share one iterator, so the walk counts a worker set once and clears the reference on both. The second is the one asymmetry the gate did not cover. A null or non-string row fails the eager map inside the trainer constructor, in seconds, before anything else has happened; the lazy view reads a row only when the sampler draws it, so the same dataset trained twenty steps and exited clean, and would have died at whatever step drew row 137. That is the one way this feature can make a failing run worse rather than slower. Both checks are metadata -- the dtype off the schema, `null_count` off Arrow's per-chunk statistics -- so neither reads a row, and a `select`ed split over-reports, which vetoes a split that might have been fine and never the reverse. No runtime fallback on top: switching a running job to the eager path would tokenize the whole split mid-run and hide the bad data, where an error naming the transform says what is actually wrong. Also: - The Linux gate tested `sys.platform`, but the hazard it names is `spawn` re-importing the entry point against a `sys.path` Studio modified in process. A Linux host whose start method is set to spawn or forkserver is the identical hazard and a platform check cannot see it. Read the start method instead, via `allow_none` and the method list, since resolving it the other way pins the context and makes a later `set_start_method()` raise. - The transform truncated to the `max_seq_length` the user asked for, while the generated `__init__` reduces that to the model's own cap before deriving `max_length` from it. Read the same cap, or the two paths stop producing the same rows and the attestation claims a width nothing applied. - Delete `prewarm_dataloader`. It was called from nowhere, and its docstring described tearing the loader down so the workers do not survive, which is the opposite of what the shipped barrier does on purpose. - `scripts/online_tokenization_ab.py` defaulted `--dataset` and `--model` to paths under one workspace. `--dataset` is required now and the rest resolve without them. - Note in the module docstring that the pass gate counts train passes only: an eval split is re-tokenized on every evaluation, where the eager map tokenized it once. The gate was well covered and the mechanism was not. Neutering `attach`, `online_config_args` and the memo while leaving the gate saying yes left 64 of 72 tests passing. `test_online_tokenization_runtime.py` pins the three claims that needed a real DataLoader with real forked workers to establish: the prewarm re-iterates from the start instead of continuing (a sequential sampler makes it exact -- continuing the prewarmed iterator loses exactly `prewarm * batch` rows and starts at the wrong one), the loader the barrier filled is the one handed back afterwards, and the workers are gone once training is over. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Probe the eval split's own BOS convention instead of reusing the train split's TRL calls _prepare_dataset once per split, so the eager path derives add_special_tokens from each split's own first row. The online path reused the train split's answer for the eval view, which tokenizes eval differently from the map it stands in for whenever the two splits disagree about a leading BOS. Also correct the prewarm barrier's docstring. torch answers a second iter() on a persistent-workers loader with _iterator._reset(), which restarts the sampler at row 0 and drops what is in flight, so the drained batches are tokenized again rather than handed to step 1. No rows are lost; what the barrier buys is workers that are already forked and past their first tokenizer touch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Release the memoized eval loader's workers alongside the train loader dataloader_num_workers and dataloader_persistent_workers are TrainingArguments settings, so an online run with evaluation on forks the same workers for the eval loader. Transformers parks that prepared loader in _eval_dataloaders (Trainer._get_dataloader, unchanged from 4.51.3 through 5.5.0) and torch never drops _iterator on a persistent-workers loader once it has been iterated, so those workers outlived train() and sat resident through the merge and export that the existing cleanup exists to protect. Drain and drop the memo too. * Stop the online tokenization tests depending on the runner's TRL and torch Two CPU CI environments were red for reasons that had nothing to do with what the tests cover. The gate tests read the installed TRL through trl_supports_skip_prepare_dataset, and the CPU job installs no TRL, so every refusal reported the missing hook instead of the gate under test. Pin it in the autouse fixture, the way sys.platform is already pinned, and cover the detector and its veto directly instead. The wiring tests import UnslothTrainer, which imports torch, at module scope, so a runner without torch failed collection and interrupted the whole run rather than skipping the module. Guard it with importorskip, as the runtime tests already do. * Tighten online tokenization comments * Route a Hugging Face dataset id through dataset_source in the A/B harness --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-16 04:55:53 -07:00
Daniel Han	2886dfc383	Make the packed-boundary guard reachable on the fused cross-entropy path (#8959) ... * Make the packed-boundary guard reachable on the fused cross-entropy path `mask_packed_sequence_boundaries` stops a packed document from being trained to predict the first token of the next document. It was called zero times on every reachable packed training path. Its only call site sits after the fused-CE branch has already returned, so it runs only when UNSLOTH_RETURN_LOGITS=1 - and that flag is itself in the padding-free blocklist, which force-clears packing and padding_free. The only flag that made the guard run was the flag that disabled the feature it guards. The guard could not simply be moved: it operates on already-shifted labels, and every fused-CE path shifts internally and receives the raw labels. Add `mask_packed_boundary_labels`, the pre-shift equivalent (masking raw slot `cumsum` is exactly masking shifted slot `cumsum - 1`), and call it in the fused-CE branches of llama.py and mistral.py, plus in the two collator wrappers that create `packed_seq_lengths`, so compiled forwards for non-Llama architectures are covered too. Out-of-place, so no caller's batch is mutated, and a strict no-op when `packed_seq_lengths` is absent - the non-packed path is untouched. Idempotent: TRL's padding-free collator already sets exactly these positions to -100, so the default path is unchanged. Measured on Qwen3-0.6B, xformers and SDPA arms, via scripts/w2_equivalence.py: self-built labels, xformers: 13.209772109985352 -> 13.210536956787110 (= the boundaries-masked reference exactly; delta 0.0, was 7.65e-4) self-built labels, SDPA: 13.204041481018066 -> 13.204877853393555 (delta vs masked reference 0.0, was 8.36e-4) TRL collator path, both arms: bitwise identical before and after (0x00000080cb6b2a40 and 0x000000c0e5682a40 unchanged) Guard invocations on a labelled packed forward: 0 -> 1 on both arms. The helper compiles fullgraph with zero graph breaks and one graph across three token counts, and performs no device sync: out-of-range cumulative sums are redirected to slot 0, which the shift discards, instead of being filtered with a data-dependent mask. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fused-CE call sites and the collator wrappers with tests that fail without them The four tests added alongside the guard pass with every production hunk reverted: two exercise only the helper, and the two collator ones drive TRL's own collator, which already masks the same positions, so they cannot observe the wrapper at all. The fused-CE call sites in llama.py and mistral.py had no test. These drive the real fused branch against a stub that captures the labels unsloth_fused_ce_loss receives, and use a padding-free collator that does not pre-mask, which is the only way the wrapper hunks are observable. Verified by reverting: the call-site tests fail with llama.py and mistral.py reverted, the wrapper tests fail with the two collator hunks neutered. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Give the fused-CE test stub a training flag so it runs without xformers MistralForCausalLM_fast_forward reaches its `elif self.training:` mask branch only when xformers is absent, which is the case on the CI runners but not on a dev box. The stub omitted the attribute, so the new test passed locally and raised AttributeError on all three Core matrix jobs. * Tighten comments around the packed-boundary label guard * Keep the boundary targets in the collator so num_items_in_batch stays correct unsloth_zoo's _unsloth_get_batch_samples already subtracts the N-1 packed boundary targets from num_items_in_batch. Masking them in the collator as well deducted them twice on TRL < 0.24, which does not pre-mask them, so the loss and gradients were scaled up by targets / (targets - (N - 1)). The guard stays where it belongs, on the fused cross-entropy path in the forward, which is what this PR set out to make reachable. * Tighten the packed-boundary guard comments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-16 03:50:23 -07:00
Daniel Han	2cf7a28880	Keep xFormers working when flash-attn 4 is installed, and guard the varlen int32 overflow (#8957) ... * Keep xFormers alive next to flash-attn 4, and guard the varlen int32 overflow Two failures found while benchmarking attention on Blackwell. 1. flash-attn 4 silently disables xFormers. The `flash-attn-4` wheel installs its CuTe build at `flash_attn/cute/` with no `flash_attn/__init__.py`, so `flash_attn` resolves as an implicit namespace package with no `flash_attn_func`, no `flash_attn_varlen_func` and no `flash_attn.flash_attn_interface`. xFormers gates on `find_spec("flash_attn")` and then imports `flash_attn.flash_attn_interface` unguarded, so `import xformers.ops` raises, `models/_utils.py` swallows it into `xformers = None`, `HAS_XFORMERS` goes False and every fast-path model drops to plain SDPA without a word. Measured on a B200 at seq_len 8192 with Qwen3-0.6B + LoRA: 547 -> 2154 ms/step and 2.69 -> 19.02 GB peak. `fix_flash_attn_4_namespace_shadow` imports xFormers once with `flash_attn` hidden from `find_spec`, which sends it down the next branch of its own elif chain exactly as on a machine with no flash-attn. A real flash-attn 2 install is detected by `flash_attn.flash_attn_interface` resolving -- the exact import xFormers performs -- so flash-attn 2 alone, and flash-attn 2 alongside flash-attn 4, are both left untouched. Nothing is written to any third-party package. If the repair cannot work the state is reported once with its cost. 2. Packed rows with thousands of documents abort the CUDA context. flash-attn 2's varlen backward allocates `dq_accum = zeros(total_q + 128 * n_seqs, n_heads, round_up(head_dim, 32))` and indexes it with int32, so it faults with "CUDA error: an illegal memory access was encountered" once that element count reaches 2**31 -- and that poisons the context, so the run dies with an opaque error far from the cause. xFormers dispatches a BlockDiagonal* bias to the same kernel, which is where it showed up. Bisecting document count on a B200 puts the predicted limit within one document of the observed one across seven shapes (16/128/len1: 8129, 16/96: 10838, 16/64: 16257, 8/128: 16257, 16/128/len2: ~8065, 16/128/len4: 7944, and 4/128 no failure at all, correctly below the bound). Forward-only allocates no dq_accum and ran clean at 20000 documents, so the guard is conditioned on a backward being possible, keyed so gradient checkpointing picks the same backend in both of its passes. Over the bound both varlen backends fall back to SDPA with a one-time warning naming the cost; `UNSLOTH_DISABLE_VARLEN_INT32_GUARD=1` opts out. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Classify the flash_attn layout without importing flash_attn find_spec("flash_attn.flash_attn_interface") resolves the dotted name by importing the parent first, so the classifier was executing flash_attn/__init__.py (and loading flash_attn_2_cuda) during import unsloth on every machine with a real flash-attn 2, including users who never touch flash attention. Probe the package's own search locations on disk instead: same classification, a stat() instead of an import, and no side effects. Adds a test that pins it for all four layouts. Also point the remedy at flash-attn>=2.7.1 rather than >=2.6.3, since xformers enforces FLASH_VER_MIN = 2.7.1 and a pinned 2.6.3 reproduces the failure the message is about, and record the measured (~1s) width of the find_spec window instead of asserting the process is single-threaded. * Do not drop Gemma 2 softcapping when the int32 guard falls back Gemma 2 hands attn_logit_softcapping to the fast kernels through flash_varlen_kwargs only (unsloth/models/gemma2.py:157-168) and the SDPA branch of run_attention has no softcap at all, so swapping the backend to SDPA would keep a packed Gemma 2 run alive while training it on uncapped logits and uncapped gradients. That is worse than the fault the guard exists to prevent, and it is reachable: at 16 heads and head_dim 256 the threshold is around four thousand documents in a row. When a softcap is configured, raise with the element count and the way out instead of falling back. Everything without a softcap keeps the rescue unchanged. * Let the flash-attn layout tests run on the CPU-only job They imported the module as unsloth.import_fixes, which runs unsloth/__init__.py, which refuses to import without an accelerator. So every subprocess in the file exited 1 on Repo tests (CPU), and check = True hid the child's stderr behind a CalledProcessError, which is why the cause never appeared in the log. Load import_fixes.py by path instead, in the parent and in each subprocess, and assert on the return code with stdout and stderr attached. Verified both ways: 35 passed with a GPU visible, and 35 passed under CUDA_VISIBLE_DEVICES="". * Stub the top-level xformers package in the broken-xformers test The test that checks find_spec is restored after a failed xformers import only intercepted xformers.ops, so on a machine without xformers installed the repair returned early at its find_spec("xformers") gate and never warned. The Repo tests (CPU) job collects this file and installs no xformers, so the assertion would fail there. Serve a stub spec for the top-level package as well, which makes the case exercise the same path with and without xformers present. * Tighten the comments added by this PR Comment text only, no code change. The measured B200 numbers, the dq_accum formula and its int32 bound, the bisection table, the reason the classifier never resolves a dotted flash_attn name, the two-term backward test and the _gpu_init ordering constraint all stay. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-16 02:39:37 -07:00
Daniel Han	7088df88ea	Carry the sliding window into SDPA, and treat a zero window as no window (#8253) ... * Carry the sliding window into SDPA, and treat a zero window as no window Three related holes, all of which let a Mistral-style model attend past its configured sliding window without saying anything. MistralAttention_fast_forward passed the window to the flash path through window_size but never to run_attention, so with xFormers off and FlashAttention absent nothing carried it. Training takes the `elif self.training: pass` branch, so no 4D mask is synthesized either, and every sequence longer than config.sliding_window attended its whole causal history. run_attention had no way to apply a window when there was nothing to hang it on. With no packing and no padding mask it fell through to SDPA's is_causal, which is full causal and has no window at all. It now builds the band mask for that case. A non-positive window was passed through as a real width. That makes window_size (0, 0) for flash, and an SDPA mask whose lower bound sits above its causal upper bound, hiding every position from every other. Both sites now read it as "no local attention", which is how the mask builders already read an absent one. llama.py also recomputed HAS_XFORMERS as `xformers is not None`, overriding the dispatcher, which turns it off when the library imports but has no kernel that runs here. Model code then stayed on the xFormers path the dispatcher had already left, and Mistral answers xFormers by skipping the 4D mask, which is how the first hole above is reached in practice. * Build the dense window mask once per shape, not once per layer Every layer of a Mistral-style model asks the SDPA fallback for the identical q_len x kv_len mask. At 32K that tensor is 1 GiB, and the two comparisons plus their conjunction keep two more alive while it is built, so rebuilding it per layer is how this path OOMs a run that xFormers or flash would have carried -- on exactly the no-xFormers, no-FlashAttention configuration the mask exists for. Cached per device and keyed on shape and window, the same single-entry shape _SDPA_MASK_CACHE already uses for the packed path. Read-only by contract; the one caller that combines a mask with another does so out of place. * Free the outgoing window mask before allocating its replacement * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Daniel Han <moonshotaisubstack@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-09 05:33:38 -07:00
Daniel Han	85a036a195	Keep an explicitly requested float32 model in float32 without bfloat16 (#7867) ... * Keep an explicitly requested float32 model in float32 without bfloat16 A model loaded with dtype = torch.float32 was being wrapped in float16 autocast on V100/T4 even when the caller set fp16 = False and bf16 = False. The mixed precision block in rl.py reads "neither flag set" as "caller did not choose" and picks for them: use_bf16_amp = (not float16) and _bf16_supported() args.fp16 = not use_bf16_amp so on a GPU without bf16 the answer is always float16. float16 has five exponent bits against float32's eight, so a value the model was loaded wide enough to hold overflows to inf and then NaN. Spark_TTS_(0_5B) is the case that surfaced it: it loads with dtype = torch.float32 ("Spark seems to only work on float32 for now"), sets fp16 = False, bf16 = False, and on a T4 logs nan for every step, then dies at inference inside torch.multinomial, which refuses a distribution containing NaN. The sampler assert is two steps downstream of the cause. Suppressing the autocast for every float32 model is too wide: full finetuning upcasts trainable weights to float32 by itself, and float16 autocast over float32 master weights is the ordinary V100/T4 recipe (issue #4082, tests/python/test_v100_fullft_precision.py). So from_pretrained now records whether float32 was ASKED for in UNSLOTH_USER_FLOAT32, as opposed to arrived at by upcasting, and only an explicit request suppresses the autocast. It is set in both loaders, since FastLanguageModel and FastBaseModel resolve dtype independently, and written on every load so a previous run cannot leave a stale "1" behind. Only the float16 fallback changes. bf16 has float32's exponent range, so a bf16 GPU keeps autocasting and keeps the memory saving. An explicit fp16 = True or bf16 = True is still obeyed, float16 and bfloat16 models are untouched, and UNSLOTH_FORCE_FLOAT32 models still take their earlier branch. tests/test_float32_no_fp16_autocast.py pulls the mixed precision block out of rl.py by AST and executes it against fake args/model objects with the bf16 capability check stubbed, so the no-bf16 path is covered without a V100/T4. No GPU, no network, no trl import. Also, separately: do not let one optional audio dependency kill the whole test session. require_python_package calls sys.exit(1) when a package is missing, and every caller is a test module invoking it at import time, so under pytest that is not a skip of one module -- SystemExit propagates out of collection and the run ends with INTERNALERROR> SystemExit: 1 with no report at all. A missing xcodec2 took out 1000+ unrelated tests. It now skips the module under pytest; outside pytest the exit is unchanged, since these helpers are also used by standalone scripts. test_whisper.py had the same shape for a different reason: it downloads an audio fixture at import and did `assert False` on failure. Wikimedia rate limits and this URL really did answer 429 during a batch run, and an asset we could not fetch says nothing about unsloth, so it skips too. * Record the float32 request on the model, and let GRPO honour a disabled autocast Two real gaps in the float32 gating, plus the provenance problem underneath both. The gate never reached most users. llama, mistral, gemma, gemma2, qwen2 and qwen3 LoRA/QLoRA loads go through FastLanguageModel to dispatch_model.from_pretrained, which is neither of the two loaders that recorded the request, so user_float32 stayed false and the fp16 autocast this branch exists to avoid ran anyway. That is most of the notebooks. The value was also process-global and derived rather than asked for. A program that loads two models before building a trainer described whichever loaded last, and a dtype inferred from a 4bit config's bnb_4bit_compute_dtype was recorded as an explicit model-wide request, which is especially wrong under full finetuning where the loader then turns 4bit off again. Both entry points now read the argument as the caller wrote it, before any normalisation, and the answer is attached to the model. The outermost caller wins, since only it saw the raw argument; the two delegating returns in FastLanguageModel re-stamp it for that reason. Then ACCELERATE_MIXED_PRECISION = 'no' was read as a two-way switch in the GRPO replacements, so 'no' came out as bfloat16 and autocast was entered anyway. torch does not ignore that on a T4 or V100, it raises "Current CUDA Device does not support bfloat16", which is exactly the hardware this branch targets. Verified against torch's own autocast __init__: the check is gated on `enabled`, so disabling autocast is both the correct reading of 'no' and the minimal fix. Full finetuning already set 'no', so this was reachable before this PR too. Tests: 4384 passed against 4369 on the same tree with these changes stashed, identical 1310 environment failures either way, and nothing failing on this branch that is not also failing on its base. * Keep forced float32 on fp16 autocast, size GRPO chunks by the flag, and run the tests off a GPU Three follow-ups on the autocast change. UNSLOTH_FORCE_FLOAT32 exports ACCELERATE_MIXED_PRECISION 'no' as well, and the new gate read only that, so the injected _prepare_inputs header disabled autocast for it while the dtype expression right beside it was still choosing float16. That left Gemma3 and gpt-oss generation in full float32, and it disagreed with the two other consumers, which set _autocast_enabled True for exactly this case. The header honours the flag now. The GRPO chunk autotuner derived dtype_bytes from _autocast_dtype alone, which stays bfloat16 when autocast is off, so it sized hidden states and logits as 16-bit for a forward running in float32. That is half the real figure, on the explicit-float32 and full-finetuning runs this branch is for. The new tests drove torch.amp.autocast(device_type = "cuda") on a box that has CUDA. On a CPU runner torch warns "CUDA is not available. Disabling" and hands back a no-op, so the premise test would not have raised and every enabled case would have read False, failing for reasons unrelated to the code. Measured, then fixed by claiming the device rather than skipping, since a CPU job is where these would most need to run. They pass with the GPUs visible and with CUDA_VISIBLE_DEVICES empty. Tests: 4386 passed, same 1310 environment failures as the branch base, nothing failing here that is not also failing there. * Size the GRPO chunk on the dtype the forward runs in The autocast-off branch assumed float32, but pure bfloat16 full finetuning also disables autocast (ACCELERATE_MIXED_PRECISION='no') while keeping bf16 weights and a bf16 forward. That path got dtype_bytes=32 where main gave 16, cutting the chunk ~1.5x and doubling the forward-pass count on 10-22GB cards. Read lm_head.dtype instead: float32 for an explicit float32 load, bfloat16 here. Retargets the test that pinned the old value. * Apply ruff kwarg-spacing formatting pre-commit.ci's ruff-format-with-kwargs hook reported "files were modified by this hook" and could not push the result back, so the check stayed red. Applied locally instead. Three source-asserting tests had to be made robust rather than re-pinned, since the reformatting is legitimate and will happen again: - test_the_flag_is_recorded_beside_the_dtype counted the literal "self._autocast_enabled = (", which the formatter collapsed onto one line. Now pairs each _autocast_dtype initialiser with a nearby flag assignment by position, so it still catches a genuinely missing one. - test_the_original_error_is_still_reported anchored on a whole f-string literal, which the formatter merged with the hint that follows it. Anchors on the message text alone now; either shape satisfies the intent. - test_the_source_beats_the_version_fallback exposed a real fragility rather than a test problem: the detector matched the literal "kw_only=True", so any whitespace in transformers' own source would have read an install that needs nothing as one that needs patching. Made whitespace-tolerant. * Latch the GRPO autocast decision on the trainer * Add the license header to the new test files * Stamp the forced float32 answer on the model UNSLOTH_FORCE_FLOAT32 is process wide and from_pretrained clears it on every load, so a model loaded after a Gemma3 or gpt-oss trainer was built left '0' behind and GRPO's first generation dropped the float16 autocast that rl.py's 'no' was written expecting, running generation in full float32. The loaders now stamp the answer on the model they loaded, next to the existing float32 request marker, and the trainer reads it from there. Models without the stamp fall back to the environment as before. * Preserve a bf16 trainer when a forced-float32 model was loaded (#7867) * Stamp the forced float32 answer on every loader return path (#7867) * Stamp the float32 answers on the text-diffusion dispatch (#7867) * Apply the repo ruff-format hook to a test touched by this PR * Apply the repo ruff-format hook (ruff 0.6.9) to files this PR touches * Fix the trainer reading the process-wide forced float32 flag for PR #7867 * Fix a missing optional dependency aborting the whole pytest session for PR #7867 require_package exited the process when a system package was absent. The TTS test modules call it at import time, so under pytest the SystemExit escaped collection and ended the run with INTERNALERROR / exit code 3 instead of skipping one file. It now degrades to a module-level skip under pytest, the same treatment require_python_package already had, and keeps the exit for the standalone script callers. test_whisper.py also imported the unsloth runtime unguarded, which is a collection error on the Windows runner where triton is absent; guard it with a module-level skip. * Inherit an outer autocast by omitting dtype, not by a sentinel * Read the forced float32 stamp inside native fast generation * Stamp the full finetuning mode on the model the trainer reads * Keep one missing-dependency helper after the merge with main The merge left this branch's _skip_if_pytest alongside main's _missing_dependency, doing the same job, plus two near-duplicate test files. Fold both into main's helper and its test module, keeping this branch's extra coverage of require_package and of the call at module scope. * Make the generation autocast tests independent of the host having CUDA torch.autocast(device_type = 'cuda') disables itself when CUDA is absent, so reading _enabled off the constructed object measured the runner instead of the branch, and these could never pass on a CPU-only machine, a Mac or Windows. Record the arguments the code asks for instead. Also skip before importing rl_replacements, which needs unsloth_zoo. * Apply the repo's ruff kwarg-spacing formatting * Format these three files with the ruff the hook actually pins The ruff-format-with-kwargs hook installs ruff==0.6.9 through additional_dependencies, and 0.6.9 and 0.15 disagree on where the message goes in a multi-line assert and on how a long lambda signature wraps. I had run the script with the workspace ruff, so pre-commit.ci kept reformatting these back and failing, with nothing to show in the diff. AST is identical on all three. * Drop the nullcontext import this PR no longer uses Left over from the earlier shape of the autocast gating, which now omits dtype instead of passing a sentinel context. Source lint's import-hoist check treats an added-but-unused hoisted import as a blocker, and it is right: nothing in this file references it. * Tighten comments for the float32 dtype gating change --------- Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-08-09 05:12:02 -07:00
Daniel Han	1fe8c7a136	Stop padding-free SFT from tripping the TRL >= 1.0.0 max_length guard (#7951) ... * Stop padding-free SFT from tripping the TRL >= 1.0.0 max_length guard TRL 1.0.0 added a guard that refuses to build an SFTTrainer when padding-free is on, packing is off and args.max_length is set. Unsloth auto-enables padding-free whenever padding_free is left at its None default, and the max_length_check codegen always wrote args.max_length from max_seq_length, so every default SFTTrainer(...) raised on TRL >= 1.0.0. rl.py now clears args.max_length for a TRL that carries the guard, and routes the truncation length through max_seq_length instead, which is what Unsloth's own dataset prep reads. An explicit user max_length is honoured there rather than dropped, with a one-line notice when it differs from the resolved length. The block is only emitted when the guard text is present in the TRL source, so older TRLs receive exactly what they did before. rl_replacements.py makes the Zoo's truncation-length seed None-safe, and trainer.py grows a back-off for a TRL that words the same guard differently. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the resolved length and TRL's guard when padding-free swaps max_length Review follow-ups on the padding-free / max_length swap: Only route the cap into max_seq_length when Unsloth's dataset prep will really run its truncating tokenize pass. It does not for dataset_kwargs= {'skip_prepare_dataset': True}, nor for a dataset that already carries input_ids / labels, which makes the Zoo's sft_prepare_dataset skip tokenization. Clearing args.max_length there told TRL the caller had supplied truncated rows while TRL hands its collator max_length=None under padding-free, so 402-token rows reached training against a 128 cap. Turn padding-free off and keep max_length for those datasets instead, so TRL's own collator truncates. Drop the _unsloth_requested_max_length override. The block above already resolves the length under Unsloth's documented precedence (max_seq_length, capped by the model, beats max_length), so re-reading the raw user max_length inverted it: max_seq_length=4096, max_length=512 truncated at 512 on TRL >= 1.0.0 and at 4096 everywhere else. The resolved length is now used as-is. Tests cover both, and fail on the previous commit. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix padding-free test on CPU-only runners for PR #7951 UNSLOTH_ALLOW_CPU=1 makes `import unsloth` skip both halves of the SFT patch on purpose, so cross-platform CI saw stock trl.SFTTrainer and every construction test failed with "SFT patch did not apply". Request the codegen swap and the __init__ wrapper explicitly, in _gpu_init's order. * Run the padding-free suite in the version-compat jobs that install the deps * Read the yielded row schema, not column_names, before clearing max_length * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Copy the length cap unconditionally before clearing max_length * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments on the padding-free max_length handshake * Truncate pre-tokenized rows instead of leaving the cap unenforced Disabling padding-free stopped TRL's guard from firing but did not make max_length mean anything. TRL enforces the cap in _prepare_dataset via truncate_dataset, not in the collator: the LM collator it builds for the language-modeling case is constructed without a max_length. The Zoo's _prepare_dataset returns already-tokenized rows untouched, so a 402-token row still reached the model under a 128-token request. Truncate those rows here instead, by the same rule TRL uses (slice every per-row list column, so input_ids, labels, attention_mask and the masks stay aligned). That restores TRL's own contract rather than inventing one, and padding-free can stay on instead of being dropped, so the speed win survives. Written out rather than imported from trl.data_utils: that module pulls in the processor stack, and an ImportError there would silently drop the cap. A failure to truncate now prints rather than passing quietly, since a swallowed error reads exactly like the cap being enforced. Two cases keep the old behaviour on purpose. skip_prepare_dataset means the user asked for the dataset to be left alone and TRL skips truncation there too. A with_transform dataset reports column_names as its backing columns while yielding input_ids, so mapping it would truncate the wrong thing. Tests split accordingly: pre-tokenized rows are truncated, the cap is consumed and padding-free stays on; skip_prepare_dataset and the transformed dataset keep the cap and lose padding-free. The truncated case asserts a collated width of 2 x cap, because padding-free concatenates the batch, and asserting the bare cap there would be asserting padding-free was off. Verified against trl 1.9.2, which has the guard, as well as 0.25.1, which does not. The suite is vacuous without the guard: every assertion sits behind trl_has_guard, so a green run on a guardless TRL proves nothing. * Truncate every eval split, and columns that are not lists Two gaps in the truncation branch, both found in review. TRL accepts eval_dataset as a dict of named splits, and a dict has no .map of its own, so the splits were skipped while max_length was still cleared below. Evaluation then ran at the full length the cap was meant to stop. Each split is mapped now. A dataset with a torch or numpy format hands batched map() tensors rather than lists. raises on a tensor's ambiguous truth value, and the isinstance(list) check would have left the column alone in any case, so the rows stayed long while the cap was cleared. The predicate now asks for a per-row sequence via __len__, excluding str and bytes so a text column is never sliced. Both new tests fail against the previous version: the formatted dataset keeps overlength rows, and the named eval splits are left untouched. * Verify the truncation instead of assuming it worked Three more findings, and together they say the shape was wrong: every round found a new way an unconditional dataset rewrite corrupts something. So the branch now decides per DATASET and then checks its own work. A raw eval split was being sliced as if it held tokens. messages is a per-row sequence too, so a blanket map cut conversation turns off the end and silently corrupted evaluation. The truncatable test now runs against each split on its own, so a raw split stays raw for the tokenizer pass that follows. A with_transform dataset whose BACKING table carries input_ids passed the schema test, but map() writes that table while the transform keeps rebuilding overlength rows on read, and the cap was cleared anyway. Custom-format datasets are now rejected outright. Dataset.map ran single-process regardless of dataset_num_proc, which TRL's own truncate_dataset honours through its map_kwargs. It is forwarded now. The real change is the last one: after rewriting, a row is read back and compared against the cap, and anything still over it restores the original datasets, keeps max_length and drops padding-free. Every predicate above is a guess about what a dataset will do; this is the only step that observes it, so an unforeseen shape degrades to the safe path rather than training uncapped. Confirmed independent: the with_transform case only fails when BOTH the format guard and this read-back are removed. 19 passed on trl 1.9.2 (guard present), 13 passed 6 skipped on 0.25.1. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse a cap nothing can enforce, and cover every split Three Codex items on the padding-free handshake. Dropping padding-free keeps max_length for TRL's collator, and no TRL from 0.22.2 to main truncates there: truncation lives only in _prepare_dataset, which returns pre-tokenized rows untouched. So for a with_transform dataset that already yields input_ids there was no enforcement path left, and construction now succeeded where TRL's own guard used to refuse. Verify a row and raise when one is over. The cap was consumed on the train split's word alone. A tokenized eval split in a shape the truncation cannot rewrite is left as it was, and prep does not re-tokenize rows that carry input_ids, so evaluation ran over the cap. Every split is checked now. The truncation map forwarded args.dataset_num_proc raw. The config layer writes serial as 1 and datasets >= 4.1 builds a Pool(1) for it, so an explicit dataset_num_proc=1 or UNSLOTH_DATASET_NUM_PROC=0 could still fork a tokenizer worker. Resolved through the same helper as every other map site. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Check every row for the cap, not just the first A short row 0 in front of a long row 5000 read as within the cap, and in the fallback branch nothing downstream truncates it. A map-style split is now read in full; a stream cannot be rewound, so a bounded prefix is all there is and the check says so rather than pretending otherwise. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cap every split, and truncate a stream instead of sampling it The truncation block was gated on the train split, so a raw train set beside a pre-tokenized eval set skipped it entirely, consumed `max_length`, and left evaluation uncapped: preparation does not re-tokenize rows that already carry `input_ids`. Each split is now capped and checked on its own. `IterableDataset.map` takes no `num_proc`, so forwarding the auto-sized one raised TypeError, the catch restored the stream, and the run died on "cannot be enforced". The map kwargs are chosen per split now, and the lazy map caps every row a stream will ever yield, which the 1024-row prefix scan could not promise. A stream that cannot be rewritten stays unverified and is refused. Enforcement is no longer confused with observation. A `with_transform` split that happens to sit under the cap rebuilds its rows on every read, so it holds `max_length` and turns padding-free off, the same answer the train side already gave. Splits that were rewritten keep their truncation when another split cannot be enforced; rolling them back put an overlength train set back and turned a healthy run into the same error. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match TRL's own truncation: keep_end, packed splits, and masked rows Three ways the manual truncation diverged from what TRL's `_prepare_dataset` does, each of them silent because this path consumes `max_length` and stops TRL from doing it. `truncation_mode = 'keep_end'` slices `[-max_length:]` upstream, and callers use it when the completion sits at the tail of a long prompt. Always keeping the prefix trained on the wrong half of every row. A packed split carries `seq_lengths`, which holds document lengths rather than tokens. Slicing it by the cap left it describing the pre-truncation row, so padding-free built position ids for more tokens than `input_ids` still held. TRL skips truncation entirely when packing, so such a split is refused rather than cut, and the remaining columns are matched by row length against `input_ids` so a sidecar list is never mistaken for a token sequence. TRL drops rows left fully masked immediately after truncating, since a prompt that alone fills the cap has every label at -100 and contributes no loss. Without that filter, completion-only datasets fed batches with no supervised tokens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Three fixes to the pre-tokenized max_length handshake A 0-dimensional tensor has __len__ and raises on it, so under set_format("torch") a scalar id column read as a per-token sequence and the later len() threw TypeError. The outer catch then restored the overlength dataset and a truncatable run died on "cannot be enforced". Probe with len() instead of hasattr. Supervision is carried three ways and only labels was filtered after truncation. A completion-only or assistant-only row whose prompt fills the cap is left with an all-zero completion_mask/assistant_masks, which TRL's collator turns into all -100: no supervised token in the row, and a batch made of them is a NaN loss. skip_prepare_dataset exempted the overlength check, which made it the one route to a silently uncapped run: TRL then neither truncates nor builds its collator with a truncation length, so the oversized rows reach the model with max_length set and ignored. The behavioural tests only run on a TRL that ships the guard, so the three changes are pinned in the emitted source as well. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gate the mask filters on their loss mode, and fix the test I contradicted A mask is only supervision when its loss mode is on. With completion_only_loss = False TRL ignores completion_mask and trains from the normal labels, so a row whose retained mask is all zero is still a good row; filtering it biased the split or emptied it. completion_only_loss defaults to None, which TRL reads as on for a prompt-completion dataset, so only an explicit False opts out. assistant_only_loss defaults to False and has to be asked for. labels stays unconditional: it IS the supervision. test_unprepared_datasets_keep_their_length_cap passed an OVERLENGTH split with skip_prepare_dataset, which is now the one configuration that raises, so on a TRL carrying the guard it asserted the opposite of test_skip_prepare_dataset_does_not_excuse_an_overlength_row. Rewritten against rows that already fit, which is what the flag's promise is about. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Filter assistant_masks on presence, the way TRL's collator reads it * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cap a pre-tokenized eval split handed to evaluate() after construction * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve the loss mode like TRL, intersect the masks, cap predict too * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the late eval cap agree with the one at construction, and leave a packed eval split to the packer The evaluate/predict wrapper is the same cap arriving late, so it has to match the construction-time cap on every detail. Four ways it did not. A stream came back uncapped, and silently: dataset[0] does not raise on one, because datasets 4.x reads the 0 as a COLUMN name and returns an IterableColumn whose len() then threw TypeError into the catch that returns the original. So neither this wrapper nor TRL enforced anything. The prefix scan is now skipped for a stream and map() is applied directly, which is lazy and covers every row it will ever yield. truncation_mode was ignored: [:cap] always, while TRL and the constructor slice [-cap:] for keep_end. Callers who set it are the ones whose completion sits at the tail, so eval ran on the wrong half of every long row. Rows left with no supervised token were kept. TRL filters those right after its own truncation, but args.max_length is None on this path so TRL does not run, and a batch of them reports NaN. Same labels-then-mask-intersection rule as the constructor, with completion_only_loss resolved from the dataset shape. A packed split was partially sliced. seq_lengths describes documents, not tokens, so cutting input_ids under a stale one makes padding-free build position ids for tokens the row no longer has. The constructor refuses this shape; so does this now. Separately, eval_packing is resolved apart from packing (packing = args.packing if args.eval_packing is None else args.eval_packing), so packing = False with eval_packing = True reaches a branch gated on not packing. TRL then packs that eval split instead of truncating it, and wrapped packing concatenates the whole token stream before chunking, so truncating each row first evaluates on a truncated corpus. That case now drops the enforcement claim rather than the split: max_length stays and padding-free turns off, which is also what packing itself requires. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Let the eval packer own its split without raising, and make supervision one intersection Sparing an eval split for TRL's packer set _unsloth_capped = False, which drops into the branch that scans every split and raises on an overlength row. That split is overlength on purpose, so the previous commit turned a working eval-packing run into a ValueError and denied wrapped and bfd_split the overflow they exist to handle. The scan now skips the eval splits the packer owns, and keeps scanning the train split, which nothing packs. _unsloth_eval_packing moved one level out so the fallback can read it even when skip_prepare_dataset skips the truncation block, and it no longer requires an eval split at construction: a split handed to evaluate() later would otherwise find max_length already gone. The late evaluate/predict cap makes the same call now, and returns a split the eval packer will take untouched. completion_only_loss is resolved once, from the training sample, because that is what TRL does (dataset_sample = next(iter(train_dataset))). Resolving it per split disagreed with the collator whenever the schemas differ: prompt/completion training data makes the collator apply completion_mask, while a pre-tokenized eval split carrying only input_ids and completion_mask read as full-sequence loss, so rows whose mask truncated to all zeros survived and went all -100. The resolved value is parked on args so the late cap uses it too. labels joined the mask intersection instead of being a filter of its own. The collator applies the masks onto the labels, so a row supervised at one position and masked in at another passed both filters separately and still went out with no supervised token. truncation_mode now has to be keep_start or keep_end. TRL's SFT path never reads that attribute, so nothing downstream would catch a third value and mapping it to the default cuts from the side the caller asked us not to; the enforcement claim is dropped instead. A late split with no column_names is read from a row rather than assumed raw. A custom map-style dataset carries none, and args.max_length is already cleared on that path, so nothing else would have capped it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cap a late eval split that cannot be rewritten, and give the mask-filter test the mode it asserts The late cap called .map(), which belongs to datasets. A torch.utils.data.Dataset or a plain list has neither map nor filter, so the call raised AttributeError into the broad catch, which returns the original, and the split reached the collator uncapped on a path where max_length is already None. A with_transform dataset is the same problem from the other side: it HAS map, but rebuilds its rows on every read, so mapping writes a backing table nobody reads, and its column_names still reports the backing schema while it yields input_ids, so it did not even read as pre-tokenized. Both are now capped on read through a small wrapper: rows are sliced on __getitem__ and __iter__, which is how the collator reaches them either way, and the surviving indices are resolved once up front since dropping unsupervised rows changes the length. Columns are read from metadata AND from a yielded row, because either alone misses one of these two shapes. Separately, test_rows_whose_mask_is_truncated_away_are_dropped now passes completion_only_loss = True. Its split has no prompt/completion columns, so a None resolves to False from the train sample, the collator ignores completion_mask entirely, and filtering on it would be deleting rows that still carry full-sequence supervision. The test is about the filter, so it has to ask for the mode that makes the mask supervision. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Close four gaps in the late eval cap: streams, predict, the stored split, short rows * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cap the late splits nothing else will, and keep the packer bypass out of it Reconciles the four late-cap gaps into one implementation. Nothing TRL owns runs on a split handed over after construction: _prepare_dataset is called from __init__ and nowhere else, and SFTTrainer overrides neither evaluate nor predict nor get_eval_dataloader. So the eval_packing bypass was wrong on both entry points, not just predict: no packer will ever see a late split, and skipping the cap there hands the collator raw overlength rows with max_length already cleared. The test asserts that premise off the installed TRL source rather than assuming it. A stream capped on read has to still BE an IterableDataset, so the wrapper subclasses one, and the shared base no longer carries the raising __len__ that made a stream look map-style. Its __getattr__ refuses dunders and its own state as well, since a DataLoader worker pickles the split and a __setstate__ arriving before __init__ would recurse on _inner forever. Being under the cap is not the same as being supervised, so the length scan now only decides whether anything needs truncating; the supervision filter runs either way, as it does at construction time. It hands back the caller's own object when it drops nothing. evaluate() with no argument falls back to self.eval_dataset. That split is swapped onto the trainer for the call rather than passed down as an argument, because HF recurses over a dict of splits by NAME when nothing was passed, and passing the dict turns that into an override. Capping is memoized per split object: every eval during training comes through here, and the scan materializes the whole input_ids column. * Make the late-cap wrappers picklable, probe non-destructively, keep predict whole * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the mode refusal, refuse a one-shot stream, require the seed rewrite Three from the latest review round. `_unsloth_capped` is seeded from the truncation mode and the eval branches combine into it with `and`, but the train branch assigned straight over it. So a `truncation_mode` that is neither keep_start nor keep_end printed "not being enforced here" and was then served as keep_start anyway, with `max_length` cleared and padding-free still on. Now the same `and` shape the eval splits use. The fallback cap scan iterated the split to check it, which for a one-shot stream IS consuming it: the trainer would get an exhausted dataset, or one short by the whole 1024-row prefix. Two `iter()` calls returning the same object marks one -- true for a generator and for a torch IterableDataset with shared iterator state, false for a datasets.IterableDataset, which restarts -- and neither call reads a row. The answer on a hit is the same as for an unfinished prefix: not proven within the cap, so the enforcement claim is dropped and every row stays. The max_length seed rewrite is not the optional worker-count edit `_replace_or_fallback` was written for, whose warning text it was borrowing. The generated trainer clears `args.max_length`, so an unrewritten seed reads that None rather than the 0 that makes the guard fall through, and a raw dataset stops being truncated. `required = True` keeps the two-anchor tolerance and raises only when both miss, which is where the neighbouring _require_replace edits in the same function would raise a line later anyway. Both anchors match the installed unsloth_zoo, so this changes nothing today. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Use one one-shot signal everywhere a split is probed The cap scan learned that two `iter()` calls returning the same object marks a single-pass stream; the three other probe sites kept `iterator is dataset`, which is true for a bare generator and false for an `IterableDataset` whose `__iter__` hands back one stored generator. Just as single-pass, and each site read a row off it and moved on. `_column_names` now uses both tests and chains the probed row back for either. A `datasets.IterableDataset` restarts, answers False and is rewound rather than chained, which is what stops row 0 being duplicated. The two probes in the generated block read the first TRAINING example and had nothing to chain it back to, so the run began at row 2. Writing the test turned up a second one beyond the reported site: the prep-truncation probe just above. Both now ask metadata first and fall back to a row only when the stream can spare one; the completion-only probe then resolves to False, which is what TRL answers for a split with neither `prompt` nor `completion`. Not chained back at these two, because rebinding `train_dataset` to an `itertools.chain` inside the constructor would hand TRL an object without the dataset API it goes on to use. `_memo_token` refuses a split whose format type is custom. `_fingerprint` covers the backing table, not the transform, so a transform closing over mutable state yields different rows under an unchanged token and the memo replayed a supervision filter decided against the old ones. One existing assertion moved off a fixed 600-byte window and onto the next anchor: a comment added inside the block had pushed it out of range. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cap the split a string eval key names, and stop trusting a transform's schema Three things the same optimistic reading of dataset metadata was behind. evaluate(eval_dataset = 'validation') is the supported way to pick one split out of a stored dict: get_eval_dataloader resolves it as self.eval_dataset[key], so capping the key was a no-op and the split it named reached the collator over max_seq_length with max_length = None. Cap the stored split in place instead and hand the key straight back. column_names describes the BACKING table, so a with_transform split storing text while yielding input_ids read as raw and the cap was cleared for rows nothing then truncates. Discard the metadata when the split has a custom format and probe the yielded row, which is free there: such a split rebuilds its rows on read. A stream that cannot spare a probe row cannot be ruled tokenized either, and leaving _unsloth_prep_truncates at its optimistic seed cleared the cap on that guess. Refuse instead, which costs padding-free on those streams and keeps max_length for TRL's own collator. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cap the dataloader paths, judge token columns by value, and stop the last destructive probe Three from the same round, all ways the late cap could be walked past. get_eval_dataloader and get_test_dataloader are public API and neither goes through evaluate/predict, so a caller building a dataloader directly reached the padding-free collator with args.max_length already cleared and nothing capping the split. Same wrapper, same argument position. The per-token allow-list also treated every token-shaped NAME as sliceable. An optional column stored as token_type_ids = None made the map raise, and the broad catch around it handed back the uncapped split; a 2-D position_ids sliced on the wrong axis and came out misaligned with the truncated input_ids. Judged by a row now, the way the construction-time truncation already does. And _unsloth_pretokenized still read a row off a one-shot stream, which the schema probe and the cap scan had both stopped doing: read raw it declares the split safe and training starts at row 2, read tokenized it rejects a caller-owned stream it has already mutated. Schema first, a row only when one is free, and an unprobeable stream holds max_length rather than clearing it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cap a split once, probe a transform, and leave an unknown mode alone The dataloader wrappers made `evaluate()` reach the cap twice: it caps, stores, and the original then calls `get_eval_dataloader`, which is wrapped too. Over a one-shot stream that was destructive, since `_CappedStream` hands out a fresh generator over the same exhausting source rather than rewinding, so the second pass's probes ate rows. `_cap` now stamps what it capped to and hands a matching split straight back. `_unsloth_pretokenized` returned on `column_names`, which for a transform describes the BACKING table, so a split storing `text` and yielding overlength `input_ids` was called raw and had its cap cleared. The transform rule is now one helper both schema probes read. An unknown `truncation_mode` seeded the refusal but still sliced, so the fallback scanned an already-trimmed split and only turned padding-free off: every row silently cut from the start. Both mutation sites are gated now. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align every column, stay lazy, key the cache, and refuse an unknown mode - `_column_names` read a row and threw it away, so on a one-shot stream `_sliceable_per_token` had no widths to compare and cut `input_ids` alone, leaving `labels` and `attention_mask` overlength. It hands the row back now. - `_CappedRows` built an identity index even with no supervision to filter on, a whole extra pass over a `with_transform` split before the dataloader starts. No supervision means no index. - The eval-cap memo omitted `truncation_mode`, so keep_start then keep_end reused the cached prefixes. - The late cap took an unknown `truncation_mode` as keep_start, cutting from the side the caller ruled out. It refuses, like the construction path. - A retained `max_length` was read as proof the cap is enforced downstream. It is what the block leaves behind when it turns padding-free OFF instead, and `_prepare_dataset` never runs for a late split, so an overlength one reached the model uncapped. This reverses an earlier test whose premise -- that TRL truncates in its own prep -- does not hold for the late path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse the padding-free retry when nothing enforces the cap * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Validate the late cap per row, and bound what it remembers The read-side wrapper chose its sliceable columns from one probed row and then applied that slice to every row, so an optional column that is None or a different width further in raised inside the dataloader. It now measures each row the way the map path already does. A user-defined per-token field was never on the allow-list, so it kept its full length while input_ids was cut and a custom collator saw mismatched rows. Those ride along when the row proves them aligned and flat. The cap mark was trusted forever on the caller's own object, which a set_transform can mutate under it. It now carries the fingerprint the memo already requires, and the memo itself is bounded: every entry pinned both the split and its capped copy for the trainer's lifetime. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cut nested token fields, and stop three probes reading the wrong source A [seq_len, channels] field has its token axis first, so the slice is correct there; the nested test that rejected it was aimed at the channel-major shape, which the length check already catches. Left uncut, it handed a custom collator the old sequence length beside capped tokens. The construction-time batch map classified a column from its first row and then called len() on every value, so an optional field that is None further in raised, the handler restored the overlength split, and a truncatable run died on cannot be enforced. Per value now, as the late cap already does. completion-only resolved from column_names, which under set_format(columns = [...], output_all_columns = False) still lists the backing table while the rows yield only the named columns. The format's own list is what is yielded. The fallback cap scan now resolves eval_packing: TRL's eval packer owns and chunks that overflow, and the generated path already excludes those splits, so scanning them refused a configuration that path accepts. And a source edit whose result is already present is no longer an edit that failed. old is a prefix of new for the max_length seed, so the wide anchor matched the normalized line and appended a second or 0, while a Zoo that spells it differently matched nothing and raised under required = True. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Run the pre-truncation rewrite under the rank window, and read the mode TRL resolved Every rank reaches the inserted map and filter before TRL's _prepare_dataset, and TRL runs its own preparation maps under main_process_first. Without the same window, eight ranks each start num_proc workers against one Arrow cache: 64 processes doing identical work and writing the same files at once. The body moved into _unsloth_cap_one so the map AND the filter sit inside it; a single process gets a no-op context manager. The late cap read completion-only from args, which only the generated block sets. That block does not always run -- a TRL whose guard did not match, or padding-free off from the start -- and the fallback then read the LATE split's schema, answering False for one carrying only input_ids and completion_mask. Rows whose completion was cut away entirely survived and the collator turned them into all -100, i.e. a NaN eval loss. TRL's own resolved value is what the collator uses, so that is what is read now. And the idempotence check compares quote-normalized text. The narrow regex already accepts either style, so a Zoo carrying the replacement single-quoted matched neither anchor and raised under required = True. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Honour eval_packing on a late split when TRL packs one TRL 1.7.0 gave SFTTrainer its own `evaluate`, which prepares a split passed straight to it and packs it under `packing = args.packing if args.eval_packing is None else args.eval_packing`. The late cap assumed the opposite -- that nothing TRL owns ever runs on a late split -- so on 1.7.0 and newer it cut each row at the cap before the packer could redistribute the overflow. `_trl_prepares_late_evals` reads that off the class rather than off a version number, and `packs_late` is passed per entry point because `predict` and the two dataloader builders stay the base Trainer's on every TRL. It joins the memo key, since `evaluate` and `get_eval_dataloader` see the same object in one call and only the first may skip the cut. A split the supervision filter emptied now says so. Every TRL 1.x reads `next(iter(train_dataset))` in `__init__` to resolve `completion_only_loss` and `_is_vision_dataset`, so an emptied split came back out as a bare `StopIteration` naming nothing the caller could act on. The tests that asserted the old premise are rewritten to cover both sides of 1.7.0 with stub trainers, so they pin the wrapper's logic rather than whichever TRL is installed, and the version fact is pinned separately. Three more expected a transformed or unprobeable overlength split to construct quietly; that shape keeps `max_length` and reports the rows instead, which is the behaviour the block already had, so they now assert the report and a within-cap companion shows it is about the rows and not the transform. 147 tests pass on trl 0.24.0, 1.0.0, 1.5.1, 1.6.0, 1.7.0, 1.8.0 and 1.9.2. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not mark a split capped when only the packer was meant to own it * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Lift the quote-normalising helper into the anchor-helper test namespace * Give the drift fixture the Zoo's real max_length seed line * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: danielhanchen <unslothai@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>	2026-08-09 04:04:58 -07:00
Daniel Han	ee76b5fb84	Revert "Say when xFormers is installed but its kernels cannot load (#8157)" (#8237) ... This reverts commit 72aec0fa0a.	2026-08-09 01:27:48 -07:00
Daniel Han	72aec0fa0a	Say when xFormers is installed but its kernels cannot load (#8157) ... * Detect and report a mismatched xformers build at import xformers wheels are compiled against one torch release and one CUDA major, but nothing at install time enforces it: the wheels are abi3/none-tagged and --no-deps skips Requires-Dist entirely, so a cu128 wheel drops cleanly next to a cu130 runtime and only fails inside torch.ops.load_library, which xformers catches and downgrades to a logger warning. _utils.py made that worse in two ways: it silenced the xformers logger to ERROR immediately before the import, and it reported the resulting failure only under UNSLOTH_ENABLE_LOGGING. The net effect was a silent fall back to SDPA attention. Its torch<->xformers version table also stopped at torch 2.4. New unsloth/xformers_compat.py (stdlib only, imports neither torch nor xformers) carries the version knowledge up to torch 2.10 / xformers 0.0.35 and reads the installed wheel's own xformers/cpp_lib.json to learn what it was really built against. The tables and the cpp_lib.json shape were read off the published wheels, not inferred. 0.0.35 is the case worth calling out: it is the first release to declare a range (torch>=2.10) while its _C is still a single build against 2.10.0, so pip will pair it with torch 2.11+ and the extension will not load. No xformers release is built for torch 2.11 or later, so xformers_for_torch returns None there rather than recommending the install that produces the bug. Because the check runs off the file before the import, the mismatch is known in time to leave xformers' own diagnostic un-silenced in exactly the case where it is the bug report, and to print a single actionable message naming the torch, CUDA and Python the wheel was built for versus what is running. A Python version difference alone is never reported as a break: _C loads through torch.ops.load_library, not the CPython ABI. XFORMERS_BUILD_METADATA and XFORMERS_BROKEN_REASON are exported so callers can report what broke instead of guessing. * Probe the xformers kernel on every CUDA capability, and keep the reason _xformers_disabled_for_capability returned early below sm_120 with the comment "Below sm_120 xformers always works; skip the probe". That holds only when the wheel matches the runtime. A cu128-built xformers on a cu130 torch is dead on an sm_90 Hopper exactly as it is on an sm_120, and the early return meant the one piece of code that actually runs the kernel never looked. The gate also skipped the probe whenever flash-attn was installed, so those machines could never report a dead xformers at all. Now the real op decides on every capability, including with flash-attn present. The extra cost is one 1x8x1x64 forward on a CUDA context torch has already initialised, since _XFORMERS_FP32_UNSUPPORTED just below forces the same lazy init unconditionally. The probe also caches why it failed in XFORMERS_PROBE_REASON, surfaced as XFORMERS_DISABLED_REASON alongside XFORMERS_BROKEN_REASON from _utils, so a caller can report what is wrong rather than only that xformers is off. It still never raises: it runs at import, so a diagnostic must not be what breaks the import. The #4631 regressions are kept and extended: a working kernel is still kept on sm_120, and now the probe-means-disable failure mode is pinned on every other capability too. * Report the accelerator stack in the Studio hardware detail get_package_versions() covered unsloth, torch, transformers and torch's CUDA/HIP version, and nothing else. An xformers built for a different torch reports its version happily and has no memory-efficient attention, so the report a user sends with a bug looked identical whether the acceleration stack worked or not. There was no Python version either, which is half of what a mismatch report needs. get_package_versions() now also carries python, xformers, flash_attn, torchao and bitsandbytes as plain version strings. Still flat str-or-None, so every existing consumer is untouched. The part that costs a real import lives in the new get_accelerator_report(), returned under /api/system/hardware?include_details=true only -- the default response is polled for training-method selection and must stay cheap. Per package it separates installed / imports / runs, and for xformers reads the wheel's own cpp_lib.json to say what it was compiled against, which is the part that names the fix. The cpp_lib.json read is a deliberate copy of the one in unsloth.xformers_compat rather than a call to it: this module has to keep working with no torch installed, and importing unsloth would execute unsloth/__init__.py. Cached per process, deep-copied out, never raises, and skippable with UNSLOTH_SKIP_ACCELERATOR_PROBE=1 for an install where importing a broken native wheel is worse than not knowing. Deliberately does not gate startup. NVIDIA asked to block release when optimized kernels cannot load; blocking the app instead would turn a slower-but-working install into a dead one. Detection and loud reporting, not refusal to run. Ran it here and it immediately found a real one: torchao 0.18.0+cu130 against torch 2.9.1+cu128 does not import. * Surface dead optimized kernels in Settings The About tab showed a version string per runtime and nothing else, which a mismatched xformers reports exactly as happily as a working one, and no Python version at all -- the other half of every "built for 3.10, running 3.13" report. About gains a Python row and an Optimized kernels section listing xFormers, FlashAttention, torchao and bitsandbytes, each as installed / not installed / working / not loading rather than as a version. A broken one shows what its wheel was built for, with the full reason behind the row's info hint. The banner sits in the Settings dialog rather than on the About tab, above whichever tab is showing: a kernel that is installed and cannot load produces no symptom other than being slower, so a user has no reason to go looking in About. It mounts inside DialogContent, which Radix only renders while the dialog is open, so the detail fetch and the native imports behind it stay off app startup. Status is never colour-only -- each state carries its own word -- so it survives a greyscale screenshot in a bug report, which is where this will mostly be read. Parsing moved to hooks/accelerator-report.ts so it is importable without react or the auth chain, which is what makes it unit testable. A backend that predates the field parses to null, not to an empty healthy report: "cannot tell" must not render as "all fine". Also drops the misleading half of the CLI fix hint. On a torch newer than any xformers release there is no version to pin, and naming one would send the user back into the same mismatch, so it now says so and points at a source build. * Move the accelerator-report tests into the hardware test namespace The suite is routinely run as `pytest studio/backend/tests -k "hardware or system"`, and only 1 of the 15 matched under the old filename. It is the hardware module's report, so name it that way. * Never report a degraded stack on a host these kernels do not run on bitsandbytes is a dependency on every platform but only loads on CUDA/XPU, so on a Mac or a CPU-only host its import failure is the expected outcome, not a broken acceleration stack. Probing there pinned a permanent false banner to those installs, which is exactly the credibility problem that makes people stop reading banners. The probe now runs only when get_device() is CUDA or XPU, and "we did not look" stays distinguishable from "you told us not to look": "not used on this device" versus "not probed". The probe tests pinned get_device to CUDA as well. Without that they passed on a CPU-only runner by never probing at all -- green for the wrong reason. * Keep the About Python row on a host with no GPU It sat inside the Hardware section, which only renders when there is a GPU or an accelerator runtime to show. On a CPU-only host that hid the one field a bug report most needs. * Probe the accelerator stack out of process, and stop guessing at the reason Review of the first pass turned up four things that were worse than the problem being reported, three of them measured on this host. Importing the packages in the web-server process was wrong three ways. A package whose __init__ raises leaves every submodule it already executed in sys.modules, so the next import returns a half-built module -- this repo already documents that hazard and ships purge_partial_import for it (unslothai/unsloth#7580); torchao left 40 stale submodules behind here. `import bitsandbytes` latches a CUDA context, and the backend deliberately never had one: main.py pins CUDA_DEVICE_ORDER before any torch import and the export planner budgets from free VRAM read before a context exists, so a diagnostic was about to take several hundred MB off every later VRAM reading for the rest of the session. And a badly broken native wheel can abort the interpreter rather than raise, which in the server means the app dies on exactly the installs this is meant to describe. It now runs in one throwaway child with CUDA_VISIBLE_DEVICES emptied. Measured after the change: no CUDA context, no leaked modules, 3.5s once per process. _describe_xformers_break asserted a version mismatch whenever build metadata merely existed, without ever comparing it to the runtime, and dropped the real exception on the floor. [WinError 126] with a perfectly matching wheel -- a missing VC++ runtime or CUDA DLL, the most common Windows xformers failure that is NOT a mismatch -- was reported as "built for X but runs X". It now compares first and hands back the real error otherwise. The Mac/CPU gate let ROCm and Intel XPU through: ROCm hosts report DeviceType.CUDA internally, and the stock bitsandbytes is CUDA-only on both. Same permanent false banner, just moved to AMD and Intel users. The report also moves off include_details onto its own include_accelerators flag. The detail path is read by Export, Video and onboarding, none of which should pay for this, so the frontend gets a separate hook used only by the Settings surfaces. Also: read xformers' already-recorded _cpp_library_load_exception instead of re-calling the private _register_extensions (whose absence was being reported as "broken"), cap reason strings that land in an aria-label, name the subprocess encoding so Windows paths survive, and carry hip through to the UI so a ROCm wheel shows a build at all. Tests: the probe seam is faked everywhere, so none of them import the host's native wheels; added the matching-build, python-only, cuda-minor, ROCm, XPU and could-not-be-checked cases that let those defects through. * Do not let the probe silently disable xformers, or mislabel what it caught Review of the first pass found the new probe reintroducing the failure mode it was written to remove, plus two ways of stating the wrong cause. The gate turned any probe exception into HAS_XFORMERS = False for the whole process, and printed nothing. Before, capability[0] < 12 returned early, so sm_70-sm_100 could not lose xformers this way at all. A full device 0, a device claimed by another rank under EXCLUSIVE_PROCESS, a MIG slice: each of those now cost memory-efficient attention for the session, silently, because of the diagnostic. Those failures are now inconclusive -- xformers stays on and the real forward decides -- and a conclusive one is announced through the same path as an ABI mismatch, so XFORMERS_DISABLED_REASON is no longer written and never read. The probe also targeted device 0 for everybody. Under torchrun each rank owns a different GPU, and on a mixed box device 0 is often the small display card, so a wheel with no kernel for the weakest GPU disabled xformers on the good ones. It now probes LOCAL_RANK's device. The loud warning was reflowing every exception into "its optimized kernels cannot load ... install the matching build". That arm also catches the sm_100/110/120 FA3 guard and the three old-torch guards, whose messages are multi-line, fenced and already actionable; the template stated the wrong cause and mangled the text. Only a real build mismatch gets the version-pin treatment now. xformers_compat: a .dev/rc build no longer folds onto its release. The fourteen 0.0.35.devNNNN wheels on PyPI are built against torch nightlies, so answering "0.0.35, therefore torch 2.10.0" for one was confidently wrong; unknown is the honest answer. And declared_torch_pin's docstring claimed it returns None for a range pin when it falls back to the build table -- the code is right, the docstring was not. Tests: test_warning_is_printed_once was vacuous (a fresh subprocess reaches the announcement once whether or not the guard exists), so it now counts in-process; _mismatched_build synthesizes a CUDA-major difference and is skipped where there is no running CUDA major; the probe tests patch the globals they write through instead of leaking them into the session; and there is now a test that a healthy install prints nothing at all, which is the half of "never cry wolf" nothing covered. * Let the P0 xformers tests run on a host with no GPU Both subprocess tests in tests/utils/test_xformers_broken_warning.py failed on the cross-platform runners for two independent reasons, neither of them the warning they are meant to prove. First, the child ran a bare `python -c`. Everything else under tests/ gets tests/conftest.py, the GPU-free harness that forces DEVICE_TYPE to "cuda" and stubs the torch.cuda probes unsloth fires at import time; the child got none of it, so unsloth/_gpu_init.py's module-scope torch.cuda.get_device_capability() raised "Found no NVIDIA driver" and the import exited 1. That is also why the in-process `import unsloth` in test_xformers_capability_gate.py was green on the same runner. The child now loads that same harness by path before importing unsloth, so the subprocess and the parent agree about the host. On a real accelerator the harness is a no-op. Second, the stub xformers was an importable module with no .dist-info, and `import unsloth` asks importlib.metadata for the xformers version twice. With no xformers on the host that raised PackageNotFoundError straight out of fix_xformers_performance_issue(); with one, it silently answered from the HOST's xformers, which is exactly what this file's docstring promises not to depend on. The stub now ships its own metadata and the child asserts the version it reads back is the stub's. test_a_healthy_install_says_nothing additionally needs a healthy install to exist, so it now skips when importlib.util.find_spec("xformers") is None instead of reaching its "this host's xformers is genuinely broken" skip, which would have been the wrong reason for a host that simply has no xformers. Tests only - no change to the detection in unsloth/models/_utils.py, unsloth/xformers_compat.py or unsloth/utils/attention_dispatch.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the probe on this rank's device, and stop it crashing the import Widening the probe from sm_120-only to every capability put it on the standard multi-GPU path for the first time, and it does not survive there. Both of these reproduce on a real two-GPU torchrun; neither happens on main. The probe placed q on cuda:{LOCAL_RANK} but built the attn_bias with BlockDiagonalCausalMask.from_seqlens, which allocates on the CURRENT device -- still cuda:0 on every rank at import time, because launchers set LOCAL_RANK in the environment while torch.cuda.set_device happens later inside the trainer. So the two tensors disagreed, xformers rejected the pair with "Attention bias and Query/Key/Value should be on the same device", and since that text is not an inconclusive marker the gate hard-disabled xformers. Measured on two healthy identical B200s: rank 0 kept xformers, rank 1 lost it. On an 8-GPU node that is 7 ranks silently dropping to SDPA at double the attention memory -- the exact regression the inconclusive guard was written to prevent, caused by the diagnostic itself, and reported to the user as a broken install. It also allocated on cuda:0 from every rank, pinning a second context per rank, about 700 MB each. Running the whole probe under torch.cuda.device fixes both. Separately, LOCAL_RANK is a rank and not an index into the devices a process can see. Slurm with --gpus-per-task=1, and anything that narrows CUDA_VISIBLE_DEVICES per rank, gives a rank one visible device while still exporting its global rank; accelerate and transformers both use -1 to mean "not distributed". torch.cuda.get_device_capability raises on an invalid ordinal and that call is at module scope, so `import unsloth` died outright with AssertionError: Invalid device id. Verified for LOCAL_RANK of 3, 1, 07 and -1 against one visible device; all four now import and fall back to device 0, which is the only device such a rank has. Clamping is the fix rather than a try/except around the capability read: with an out-of-range index the probe fails with "invalid device ordinal", which is not an inconclusive marker either, so catching the crash alone would have converted it into the silent disable above. Those two strings are added to the inconclusive list regardless, so that path can never again be read as evidence about the build. Two more things this turned up. The probe's reason is a captured exception, and xformers answers a capability rejection with a dozen lines listing every operator it considered -- announced verbatim, that is what every RTX 50-series user would see on every import. Print the first line and put the rest behind UNSLOTH_ENABLE_LOGGING, truncating at the call site because the announcer's other callers pass deliberately fenced, copy-pasteable text that has to arrive intact. And re-silence the xformers logger once the extensions turn out to have loaded: skipping the silencing on a PREDICTED break is right, but the prediction is read off cpp_lib.json and is wrong in the healthy direction whenever a wheel records a torch patch it still loads against, and leaving the logger open is permanent and process-wide. _XFORMERS_FP32_UNSUPPORTED now reads the same device the probe used, which is what its comment already claimed. Tests: the rank-targeting test stubs torch.cuda.device so it runs on a host with any GPU count and asserts the context index, not just the tensor's; new tests cover the clamp invariant and a real subprocess import under five hostile LOCAL_RANK values, which is the regression the mocked test could not catch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tell the accelerator probe child to emit UTF-8 The probe decodes its child as UTF-8, but the child's own stdout defaults to locale.getpreferredencoding(), which on Windows is the ANSI code page. That is exactly where this probe's value is: native import errors quote DLL and file paths, and a non-ASCII path would come back mangled. Route the env through utf8_child_env, as the repo's own contract test requires. * Report what the accelerator stack can actually do, not what imported - the xFormers probe picked bf16 from device 0 while probing this rank's GPU, so a mixed box with a Turing card wrote off a healthy install. - the fix hint pinned a version but no index; all three CUDA families publish the same version string and PyPI carries only cu128, so on the reported case it reinstalled the identical broken wheel. - both mismatch diagnoses (unsloth and Studio) called a stable-ABI wheel on a later torch a mismatch, and the Studio one compared full version strings, so 2.10.0+cu126 vs 2.10.0+cu128 hid the real error before the CUDA-major check could give the accurate one. - the probe reported bitsandbytes and torchao as working on an import alone, which is exactly the state where 4-bit dies mid-run and torchao has no kernels; both now check their native side, bitsandbytes through the same leaf module the loader gates on. - xFormers loading its library is not the same as having a kernel for this GPU; a capability rejection now reads as degraded instead of Working. - ROCm hosts probed nothing at all, though Unsloth enables flash-attn there. - and a child that dies without answering no longer erases the whole report: the survivors keep their answers and the package that killed it is named. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Ask the GPU the app can actually use, and let each row say if it was checked Five things the probe got wrong once it started answering "does this run". nvidia-smi lists physical devices and ignores CUDA_VISIBLE_DEVICES, so on a mixed host masked to the sm_120 card the verdict was computed against row 0's sm_90 op table, and xFormers read as working on a GPU that has no kernel for it. The mask is resolved in the parent now, by index or by (abbreviated) UUID, and handed to the child as UNSLOTH_PROBE_DEVICE_CC, because the child is started with the mask cleared and cannot recover it. bitsandbytes picks its native library from torch.cuda.is_available(), so hiding the GPUs from the child made a healthy CUDA install report dead handles and light the banner. It gets its own child now, with the mask the app itself runs under. The server still never holds a context: that child exits immediately. On Intel XPU the applicable set was empty, so a broken XPU bitsandbytes wheel was "not used on this device" and never reached degraded, while the loader gates 4-bit on native_kernels_ready(_bnb_probe, DEVICE_TYPE) with XPU symbols. It is probed. torch.ops.<ns> materialises an empty _OpNamespace on attribute access, and its dir() already lists __name__ and friends, so dir(torch.ops.torchao) was non-empty for the no-extension case the check exists to catch. Counted off the dispatcher instead. And the About table read one report-wide "probed" flag onto every row, so a package the backend deliberately skipped rendered "Not loading" as soon as any other package was probed -- red for something that is fine, and absent from the banner. Each entry carries its own probed flag; the report-wide one is the fallback for an older backend. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the capability read out of module scope, and let the model code see the verdict Two ways the probe could hurt the thing it was added to protect. The gate read torch.cuda.get_device_capability() at the call site, at module scope, and CUDA refuses that query when the device is busy, in exclusive-compute mode or otherwise unavailable. There it raised before the probe could classify the failure as inconclusive, so a diagnostic whose worst answer is "keep xformers on and let the forward decide" turned `import unsloth` into a crash. The argument is gone (the gate asks the kernel, not the number), and the fp32 read below it now degrades to unknown through a guarded helper. A test walks the module's top-level statements and fails on any capability query that runs at import. And llama.py recomputed HAS_XFORMERS as `xformers is not None`, so turning the dispatcher's flag off never reached the model code. Mistral answers "xFormers is on" by skipping the 4D sliding-window mask and letting the xFormers bias carry the window; with the dispatcher already on SDPA, that mask is the only thing making the window local, so a sequence longer than config.sliding_window attended to the whole causal history. It takes the probed flag from the dispatcher now. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Answer for every visible GPU, and say unknown where the probe cannot tell Five follow-ups, all the same shape: a diagnostic that overstates what it knows. Mistral's training branch takes `elif self.training: pass`, so no 4D mask is synthesized, and its AttentionContext never carried sliding_window at all. With xFormers off and flash-attn absent, nothing was left holding the window and every sequence past config.sliding_window attended its whole causal history. The context carries the window now, and the SDPA path builds a windowed keep mask even with no padding mask to hang it off, since is_causal is full causal and has no window of its own. The capability resolver truncated the mask to its first entry, so with CUDA_VISIBLE_DEVICES=0,1 across a mixed pair a rank on the second card fell back to SDPA while the verdict from the first said Working. Every visible device is resolved and handed to the child now, and one uncovered card makes the whole install degraded. flash-attn recorded runs=true on the strength of importing its interface. Our own installer refuses prebuilt wheels on sm_100+ because none covers those cards, so a wheel that got there another way could load and still fail on its first launch. That is unknown, with the reason, rather than a verdict this child would have to launch a kernel to earn. The About table read runs=null as Working. A probe that ran and could not decide -- an unrecognised xformers layout, a missing bitsandbytes checker, a torch with no dispatcher table, the flash-attn case above -- is unknown, and nothing else would have corrected it since those packages are not in degraded either. And xformers_for_torch returned None above 2.10, which contradicted this module's own describe_xformers_mismatch: it accepts a 2.10-built wheel on 2.11 under the stable ABI, then the fix hint told the user no release exists and to downgrade torch or build from source. Above the floor the answer is 0.0.35, and the hint already names the CUDA index that repairs the family. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop the probe calling a supported fallback broken, and normalise a zero window Five more, all cases where the report claimed more than it knew, plus one bug of mine. torchao with no registered operators is the MANAGED stack: install_python_stack pins 0.17 on torch 2.10+cu130 knowing its extension is cleanly skipped, and torchao keeps running through its Python fallbacks with only the optimized kernels gone. Calling that degraded put a destructive banner on the standard configuration. Unknown, with the reason, so the row says it without the banner. flash-attn is unknown on every card now, not just above sm_100. A build with no cubin or PTX image for this architecture imports fine and fails its first launch, which is as true of a source build or an older wheel as of a Blackwell card; flash-attn exposes no list of the architectures it was compiled for, and this child never launches a kernel. A failed import is still broken. "device is currently in use by another process" is the driver saying the GPU is someone else's right now. It matched none of the inconclusive markers, so a transient turned into a process-wide fallback to SDPA on a wheel that was never tested. The probe device honoured LOCAL_RANK and otherwise took device 0, ignoring a caller that had already run torch.cuda.set_device(1). That can disable xformers over a card nothing will touch, and creates a context on it to do so. LOCAL_RANK still wins when usable. And sliding_window = 0 means "no local attention", the same as absent. Passing it through made window_size (0, 0) for flash and, now that the dispatcher honours the window, an all-false SDPA mask: the lower bound q_pos - (0 - 1) sits above the causal upper bound, so the layer returned nothing at all. Normalised in Mistral and again in the dispatcher. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the xFormers kernel verdict unknown until a launch proves it The op table admitting this GPU is not evidence the build ships a kernel image for it: CUDA_MINIMUM/MAXIMUM_COMPUTE_CAPABILITY are class constants, and a source build compiled for other architectures registers the same op and fails its first launch with 'no kernel image is available'. The probe holds no CUDA context by design, so it cannot establish coverage. The negative verdict is sound and stays; every other exit is now unknown, matching probe_flash_attn. The About row renders the reason on an unknown result too. Most unknowns are deliberate and come with an explanation, and the row was showing 'Not checked' and discarding it, so a skipped native extension looked like a probe that never ran. Also imports UNSLOTH_ENABLE_LOGGING explicitly in attention_dispatch: it is not in _utils.__all__, so the newly reachable inconclusive-probe branch raised NameError during import unsloth instead of keeping xformers on. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop three xFormers diagnoses from stating more than they know The no-metadata path (a source or editable 0.0.34+ build) fell back to the recorded torch pin and called every later release a mismatch, contradicting the stable-ABI rule the build-metadata branch two blocks up already applies. An unrelated extension failure was therefore diagnosed as a torch-version problem with reinstall instructions that fix nothing. The breakage announcement claimed a fallback to SDPA and more memory even when FlashAttention is installed, which select_attention_backend prefers over xformers anyway. The consequence sentence now follows what actually picks up the work; the breakage is still reported either way. A numeric CUDA_VISIBLE_DEVICES entry is a CUDA ordinal in the current CUDA_DEVICE_ORDER, and nvidia-smi's index is the PCI_BUS_ID ordering. The pin is a setdefault, so a user who exported FASTEST_FIRST keeps it and the ordinal cannot be inverted without initialising CUDA. That case now answers unknown instead of probing the wrong card; UUID masks are unaffected. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Four more honest answers from the accelerator report An unresolvable device mask reached the probe child as an empty UNSLOTH_PROBE_DEVICE_CC, which the child reads as no override at all and answers from nvidia-smi over every physical GPU: the whole-box verdict the resolver returned None to avoid. It now travels as an explicit unknown sentinel, and the child refuses to fall back on it. An xformers whose _cpp_lib raises a native load error left runs at None, so the package stayed out of degraded and the About row read 'Not checked' with no banner, on exactly the corrupt install this report is for. That is now broken; a layout with no _cpp_lib at all stays unknown. bitsandbytes is probed on ROCm. device_type.py imports it on every backend and gates 4-bit and prequantized loading on native_kernels_ready, so a dead ROCm wheel silently costs quantized loading while the report called the package 'not used on this device'. And the fix hint redacts the mirror URL. UNSLOTH_PYTORCH_MIRROR can carry credentials or a signed-URL query, and that hint prints to stdout on the default import path, into logs and shared notebooks. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Find the bitsandbytes checker from the checkout too find_spec is the right answer for an installed unsloth, but CI installs a published wheel that predates bnb_availability.py, so the lookup landed on a site-packages copy without the file and every bitsandbytes verdict came back unknown. This file's own location settles it: studio/backend/utils/hardware to the repo root. * Stop three accelerator messages from overstating what is known A transient probe failure is not a broken wheel. bitsandbytes keeps CUDA visible on purpose, so opening Settings while a trainer has filled the GPU, or holds it in EXCLUSIVE_PROCESS, fails at context creation on a healthy install -- and that answer is cached for the life of the backend, so the banner sat there until restart. Busy, out-of-memory and unavailable now read as not probed, with the reason shown, using the same classification the in-process xformers probe makes. The mirror redactor no longer returns its input when it cannot parse it. A malformed authority is exactly the URL whose credentials or signed query this was added to keep out of stdout, so it drops the index instead and the hint omits --index-url. And the Settings banner no longer promises a slower fallback. When xFormers is the dead package and FlashAttention loaded, the dispatcher prefers FlashAttention and there is no slower path; the banner now names what cannot load and points at the list, in every locale. * Pin two accelerator-report tests to a CUDA host Both assert on the probe's classification, and on a CPU-only runner the report never probes at all: every row comes back "not used on this device" and degraded is empty, so the two assertions failed on CI while passing on any machine with a GPU. The on_accelerator fixture the rest of the file already uses is what makes the probe applicable. --------- Co-authored-by: Daniel Han <moonshotaisubstack@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-09 01:25:31 -07:00
Daniel Han	68e5c3f04b	tests: make pytest tests collect cleanly and drop the deleted QVQ registry mirror (#8206) ... * tests: make `pytest tests` collect cleanly and drop the deleted QVQ registry mirror Collection was interrupted by 17 errors, so nothing after them ran, and tests/test_model_registry.py::test_model_registration[qwen] was red against the live Hub. Registry: - unsloth/QVQ-72B-Preview no longer exists on the Hub. Authenticated as an unsloth org member the API answers 404 and an org-wide search returns only unsloth/QVQ-72B-Preview-bnb-4bit, so it is not private and not gated (a gated repo still serves public metadata). Drop QuantType.NONE from the QVQ meta; the bnb-4bit mirror and the upstream Qwen/QVQ-72B-Preview stay. - The registry check now treats only RepositoryNotFoundError as "missing". Every other hub error (429, 5xx, DNS, timeout) skips the test instead of reporting all 129 models missing. Collection: - tests/saving files with no test items are standalone GPU scripts whose body runs at import, so bare collection downloaded checkpoints, trained and pushed to the Hub. They now call require_opt_in() and are visible skips unless UNSLOTH_RUN_SAVING_SCRIPTS=1. Running them directly is unchanged. - Four filenames contained a dot, which pytest's importer reads as a package separator; renamed to underscores. - tests/test_raw_text.py left its datasets stub in sys.modules for the rest of the session, which broke collection of tests/utils/test_packing.py. The stub is now restored after raw_text is loaded. - The two sentencepiece tests imported the legacy transformers.utils.sentencepiece_model_pb2, which fails on protobuf >= 4. Use convert_slow_tokenizer.import_protobuf(), the accessor the code under test uses. - tests/test_collection_hygiene.py guards all three regressions. tests --collect-only: 7629 collected + 17 errors in 214s (exit 2) -> 7703 collected, 0 errors in 3s (exit 0). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-08 20:03:55 -07:00
Daniel Han	36020c004a	Skip a test module with a missing optional dep instead of exiting pytest (#7971) ... * Skip a test module with a missing optional dep instead of exiting pytest require_package and require_python_package are called at module import time by the four tests/saving/text_to_speech_models files, and they called sys.exit(1) when a package was absent. Under pytest that lands during collection, where SystemExit becomes an INTERNALERROR and aborts the whole session. On a host without xcodec2, snac or soundfile, `pytest tests/` therefore ran zero tests and reported no failures, which reads as a clean run. Removing one offending module does not help: ignoring test_lasa.py just moves the abort to test_orpheus.py on snac. Route both helpers through one place that skips at module level under pytest and keeps the historical exit for the standalone-script path these files also support. Nothing else in the repo calls either helper. CI is unaffected: every workflow runs targeted paths rather than the whole tests/ tree, which is why this was only reachable when running the suite locally. * Tighten comments in the skip-instead-of-exit test helpers * Trim comments in the skip-instead-of-exit tests --------- Co-authored-by: danielhanchen <unslothai@gmail.com>	2026-08-06 00:39:21 -07:00
Daniel Han	cd3aef8a70	Bound dataset_num_proc by memory, and stop treating 1 as "no multiprocessing" (#7831) ... * Bound dataset_num_proc by memory, and stop treating 1 as "no multiprocessing" Training intermittently died with "One of the subprocesses has abruptly died during map operation", then succeeded on the next run (#2693, and a fresh Studio report). Measuring the tokenization map on an 8000-row dataset with a fast tokenizer found the mechanism: - Each pool task dill-pickles the tokenizer closure over a pipe, 5,369,755 bytes, once per worker per map. This happens under fork too: datasets does `from multiprocess import Pool`, and multiprocess/queues.py pickles every task regardless of start method. - Each worker peaks around 680 MB RSS. The old auto count was min(max(cpu_count + 4, 2), 64), so a large host forked up to 64 workers for roughly 43 GB resident. On a smaller box the OOM killer takes one and the parent reports only the generic message above, because datasets/utils/py_utils.py compares pool PIDs and never reads the child's exit status. Killing a worker with SIGKILL or SIGSEGV reproduces the error character for character, at any num_proc including 1. Two further defects made it worse: - The guard asked stdlib multiprocessing for the start method while datasets uses multiprocess, which keeps an independent default context. It was reading the wrong module. - num_proc=1 was used as the "no multiprocessing" sentinel. On datasets 4.3.0 (the Studio pin) map() takes the pool branch for any num_proc >= 1, so 1 still builds a Pool(1). Measured, num_proc=1 is 51% slower than None while buying no parallelism. Only None is in-process on every supported release. Changes: - New unsloth/utils/dataset_num_proc.py, one policy instead of four drifted copies. It asks multiprocess about the start method, caps the auto count at 8, and bounds any count, explicit ones included, by available memory at roughly 1 GB per worker over half of free RAM. Studio's explicit cpu_count // 4 previously bypassed every bound, which is how a 192-core host reached 48 workers. UNSLOTH_DATASET_NUM_PROC remains an uncapped escape hatch. - The config layer records intent and the map() call site makes it safe. These cannot be collapsed: unsloth_zoo reads a config None as "auto-size me", so writing None for a user who asked for 1 would inflate it. - worker.py no longer forces stdlib multiprocessing onto fork. It never reached Dataset.map, and Linux already defaults to fork. - A dead worker now raises with the start method, the worker count, the approximate memory cost and the escape hatch, chained from the original. No CUDA guard: 300 forced-fork map() runs on an initialized CUDA context produced no failures, and the child only runs the tokenizer. Since detect_hardware() always initializes CUDA, such a guard would cost every CUDA run its tokenization parallelism for no measured benefit. Known gap: the 1 -> None normalisation reaches SFT only, since that is the path sft_prepare_dataset owns. DPO, KTO, CPO, ORPO, Reward, PRM, PPO and BCO read args.dataset_num_proc in their own _prepare_dataset, so an explicit 1 there still builds a Pool(1). The memory bound does apply to all of them, so the OOM mechanism is covered everywhere. Reported by Eyera, who traced it to the commit and the call chain. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop UNSLOTH_DATASET_NUM_PROC=0 from inflating the worker count The env override returned before the serial encoding, so at the config layer it wrote None. unsloth_zoo.sft_prepare_dataset reads a config None as "auto-size me" and re-derives with its own uncapped min(max(cpu+4,2),64). So a user hitting a dead worker, following the escape hatch the new diagnostics message tells them to use, could get 64 workers. The hatch did the opposite of what it advertises, in exactly the OOM scenario it exists for. Affected 0, none, null, false, "" and 1: 14 of 42 config-layer cells. Measured end to end with the num_proc anchor skipped: before: requested auto / explicit 64 / Studio cpu//4 -> config None -> up to 64 after: all three -> config 1 -> bounded Also make the unsloth_zoo num_proc anchor non-required. It is the one anchor whose absence is harmless: the Zoo then reads args.dataset_num_proc, which the config layer has already bounded, so the memory ceiling still holds. unsloth_zoo is a floor dependency rather than a pin, and this is the block whose policy this branch changes, so it is likelier than the others to drift upstream. Hard-failing every install to recover an optimisation is the wrong trade. This is also what made the bug above reachable, so the two belong together. Verified across 630 cells: Python 3.10-3.14 x datasets 3.4.1-4.3.0 x fork/spawn/forkserver x four memory levels. Policy identical throughout. Zero config-layer None cells remain. 52 tests pass; reverting either fix fails them. Worth recording from that matrix: on Python 3.14 stdlib multiprocessing defaults to forkserver while multiprocess still defaults to fork, so the two disagree. Reading multiprocess, as this branch now does, is what matches what datasets will actually do; the old stdlib read would have silently disabled multiprocessing that was available. * Bound train_on_responses_only, the most-travelled map() in the library unsloth_zoo.dataset_utils.train_on_responses_only is a third copy of the same heuristic, and nothing in this branch touched it. unsloth re-exports it verbatim from chat_templates, it appears in essentially every Unsloth SFT notebook, and it is how Studio's apply_completion_masking reaches a map(). So the up-to-64-workers exposure sat on the most-travelled path while the branch fixed the quieter ones. Measured on a 192-core host with datasets 4.3.0, auto over a large split: 64 workers before, 8 after. Two constraints shaped the wrapper, both discovered before writing it: None cannot mean "in-process" at this boundary. The zoo's first act is `_num_proc_was_auto = num_proc is None or ...`, so a None arriving from outside reads as "size it for me" and triggers the very heuristic being bounded. Passing None for a caller who asked for 1 would inflate to 64, the same class of bug as the UNSLOTH_DATASET_NUM_PROC=0 one fixed earlier in this branch. So 1, not None, is the serial value here. The consequence is that an explicit 1 still builds a Pool(1) on datasets >= 4.0, unchanged from the raw zoo, so this is not a regression but the branch's "1 -> in-process" claim does not extend here. An explicit count also disables the zoo's 5000-row guard, which it applies to train_dataset and eval_dataset independently and only when it chose the count itself. Substituting unconditionally would hand workers to a small eval split that never had them. So substitute only when some split is actually at or above the threshold, which keeps the guard wherever it was doing work. A drift canary reads the zoo's constant off disk so the duplicated 5000 cannot diverge silently. 17 tests. Pool spy on a real masking run confirms a small auto dataset still builds no pool. Mutations: threshold drift 1 fail, explicit-serial returning None 2 fails, auto ignoring split size 6 fails. Known hole: when any split has no length (IterableDataset), this returns None and lets the zoo decide, so a huge train split beside a streaming eval split stays unbounded. Correct per the zoo's own rule, but it is a hole. * Keep the config layer serial on spawn, and stop an unsized split hiding a sized one Two genuine bugs from review, both of them regressions this branch introduced. The config sentinel leaked onto spawn platforms. On a non-fork start method the config layer wrote 1, and only SFT's map site rewrites that back to None. DPO, KTO, CPO, ORPO, Reward and PRM hand args.dataset_num_proc straight to Dataset.map, where datasets >= 4.1 builds a Pool(1) whose spawned child re-executes the user's __main__ (the Windows spawn loop, #3211/#3397). origin/main carried None there for the auto path, so this was a regression. The sentinel exists only to stop a downstream auto-sizer re-inflating serial, and no auto-sizer can do that when forking is unavailable, so it is now conditioned on fork. An unsized split masked a sized sibling. _largest_split_rows returned None the moment any split had no length, so a trainer with a large sized split next to a streaming one took the shortcut and returned a bare None past the env check. The Zoo reads that None as "auto" and picked 64 workers on this host: the exact inflation this branch exists to remove, and it happened with no env var set at all. Unsized splits are now skipped rather than allowed to veto, and an explicit UNSLOTH_DATASET_NUM_PROC wins on the shortcut too. Codex suggested resolving the override before the shortcut; that would return 1 for a small split and build a Pool(1) on datasets >= 4.1, so the fix is split in two instead. Also corrects the datasets boundary throughout: it is 4.1.0, not 4.0. huggingface/datasets#7702 flipped `num_proc > 1` to `>= 1`; 4.0.0 still ran num_proc=1 in-process. Verified against the 3.6.0, 4.0.0, 4.1.0 and 4.3.0 tags. One studio test asserted on 4.0.0 and would have failed on exactly that release. New file headers switched from LGPL to Apache 2.0, byte-identical to unsloth/dataprep/raw_text.py and tests/utils/data_utils.py, matching the repo LICENSE. 18 new tests. 87 pass; reverting the helper alone fails 16. * Keep macOS in-process by policy, not by a wrong start-method probe The probe read multiprocess.get_all_start_methods()[0]. multiprocess copies that function from the stdlib verbatim, darwin branch included, but not the darwin default that goes with it: its _default_context is still fork, carrying a literal '#FIXME: spawn'. So on macOS the probe said spawn while Dataset.map actually forks, and the dead-worker diagnostics printed the wrong method. Read the default context's own name instead, which fixes the report, and add _workers_unusable_reason() so the macOS refusal survives the corrected probe. Forking on macOS is what CPython itself declared unsafe when it moved the default to spawn in 3.8 (bpo-33725), and this parent has already loaded Torch and a threaded BLAS, so macOS stays in-process -- now as a stated policy rather than as a side effect of a misreport. Also run both num_proc suites in CI. They were never on the consolidated workflow's tests/utils allowlist, so all 87 guards were dead weight. * Bound the worker count Studio computes for itself A simulation across the platform x start-method x cpu x memory x request x env product found the one path that still reached Dataset.map unbounded. Studio's numbers are backend heuristics: trainer.py asks for cpu_count // 4 and safe_num_proc's own auto path is cpu_count // 3. By the time this module sees them they are explicit ints, which it reads as deliberate user intent and clamps by free memory only -- so a large host with RAM to spare kept every one of them. Measured end to end: 64 cores gave 16 workers, 96 gave 24, 192 gave 48 at ~1GB each, against a cap of 8 that the auto path has obeyed all along. The benchmark in dataset_num_proc.py has 32 workers at 14.2s versus 6.3s in-process, so those counts were slower as well as heavier, and Studio on a big machine is the configuration issue #2693 was reported from. Cap in safe_num_proc, which every Studio map() site routes through, and before the multi-GPU cap so the tighter of the two still wins. The constant is duplicated rather than imported, because importing it would pull unsloth's whole __init__ into hardware detection; a canary asserts the two stay equal, the same arrangement the Zoo's row threshold already uses in the other direction. UNSLOTH_DATASET_NUM_PROC is unaffected: it is read downstream and bypasses this. The simulation is scripts/matrix_numproc_policy.py. After the fix all 17280 cells hold every invariant: no workers on a start method that cannot support them, never 1 at a map() call site, never None at the config layer while forking works, never more workers than memory covers, never over the cap on the auto path, the env var obeyed verbatim, and deterministic throughout. * Say what UNSLOTH_DATASET_NUM_PROC=0 actually does The dead-worker message told the reader to tokenize in-process with UNSLOTH_DATASET_NUM_PROC=0. That is true almost everywhere and false in the one case the message is most likely to be read: train_on_responses_only on fork, with a split at or over the Zoo's 5000-row threshold, resolves to 1 rather than None, and datasets >= 4.1 turns 1 into a Pool(1). So the recovery advice offered for a large-dataset worker death did not remove the workers. The value is still right. A bare None there is read by the Zoo as 'size it for me' and would inflate to its uncapped count, and unsloth_zoo's _effective_num_proc returns num_proc unchanged when it is None or 1, so no value expresses in-process on fork for a large split without changing the Zoo. What was wrong was the sentence, so the sentence is now specific: fewest workers this path can use, in-process everywhere except that case, one worker there. Two tests. One reads the rendered message and requires it to name the exception, the path and the row threshold. The other drives resolve_responses_only_num_proc on both sides of the threshold and asserts 1 and None, so the message cannot claim a behaviour the resolver does not have. * Make the studio num_proc tests runnable off Linux and without torch The cross-platform staging legs failed all three, and the file's own docstring claimed it ran on any host, so both halves of that were wrong. dataset_map_num_proc returns None outright on win32 and darwin, so every assertion expecting a worker count was really an assertion about Linux and failed on the macOS and Windows runners. An autouse fixture pins the platform; the parametrised spawn-platform test sets its own value afterwards and still wins. _patch_runtime imported torch directly, which is a hard failure on a runner that has none. Worse than the error: dataset_map_num_proc treats an ImportError as "runtime not touched yet", so a torch-less host turns the XPU guard into a no-op and the test asserting None would have been passing for the wrong reason wherever it did not outright fail. It now falls back to a stub module in sys.modules, which a real "import torch" finds. Verified by reproducing both runner conditions locally rather than waiting on CI: 9 passed with torch and 9 passed with it removed. That harness needed correcting too -- it first blocked __import__ unconditionally, which is stricter than any real runner, since real Python consults sys.modules first and that is exactly what the stub relies on. * Do not trust a start method the host does not offer The cross-platform legs found a real bug in the probe, not just in its tests. On a Windows runner the private default-context chain answered "fork" while get_all_start_methods() was ["spawn"]. Those attributes are private and not consistent across builds, and a start method the platform does not offer cannot be the one in use. Believing it read Windows as forkable, so _workers_unusable_reason() returned None and workers were allowed through - the spawn re-import loop of #3211 / #3397 that this module exists to prevent. The probe now cross-checks its answer against the available methods and falls back to the documented list, with a regression test that reproduces the exact shape: spawn-only host, private chain saying fork, result None at both layers. The test failures around it were mine too, and they share a cause: the macOS policy added earlier made sys.platform load-bearing in get_dataset_num_proc, so a batch of tests that assert a worker count became platform-dependent. They passed on the Linux runner and failed on macOS. The module fixture pins the platform; the tests that are about the platform set their own value afterwards. The two studio tests that build real worker processes now skip when the host cannot fork. Under spawn inside pytest the pool fails for reasons that have nothing to do with the claim being made (WinError 10038 closing a handle, os.WNOHANG missing), and the version split they check is also asserted without processes in tests/utils. * Import multiprocess before the tests spoof the platform The Windows leg of staging CI failed inside a real worker pool with AttributeError: module 'os' has no attribute 'WNOHANG'. multiprocess picks its concrete contexts at import time from sys.platform, and both test files spoof that to linux, so the first import under the spoof handed a Windows runner the POSIX fork contexts. get_all_start_methods() then reported fork, the skip guard did not fire, and the pool tried to reap a child the way only POSIX can. Import multiprocess at module scope, before any fixture runs, and read the real platform there too so the two real-pool tests skip on Windows even if something later lies about it. * Tighten the comments added by this PR * Import the num_proc policy from the zoo, not back into unsloth The trainer source rl.py generates ran `from unsloth.utils.dataset_num_proc import ...`. unsloth/__init__.py is what generates that source, so the import reaches back into the package mid-flight, and it also drags unsloth/utils/__init__.py -> packing -> attention_dispatch -> models._utils, which means a module whose only imports are contextlib, os, sys and typing arrives through torch and the whole model stack. Nothing circular in practice: the injected imports are function-body imports that run at config construction and dataset prep, and tripping them mid-import at rl, attention_dispatch, packing, llama and chat_templates all resolved. But the coupling is real. Cold, in a process that imports only the compiled trainer cache, it costs a 9.7s `import unsloth`, and it inherits any unrelated failure in that import: on a box with a torchao/torch mismatch the stdlib-only helper failed to import along with everything else. The policy now lives in unsloth_zoo.dataset_num_proc (unslothai/unsloth-zoo#984), which unsloth already depends on and which never imports unsloth. Every call site tries the zoo first and falls back to the copy here, so upgrading unsloth alone still fixes the bug on an older zoo, and a new zoo takes over with no further change. test_the_two_copies_have_not_drifted compares the two, with docstrings stripped, whenever both are importable, so they cannot silently disagree about a worker count. Verified both directions end to end: with the zoo module present the generated config imports unsloth_zoo.dataset_num_proc and never touches the unsloth copy, and with it absent the fallback runs and the config still comes out bounded. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments this PR adds * Mirror the zoo's container and start-method-split fixes unslothai/unsloth-zoo#984 review found three holes in this policy, and the copy here is a code twin of that module, so it takes the same changes. psutil reports the HOST inside a cgroup, so a 2GB container on a large box read as having room for the full worker set, and a one-core pinned job auto-sized workers that contended for that core. Memory is now the smaller of the host reading and the cgroup limit less its current usage, and the CPU count the smallest of the host, the affinity mask and any cgroup quota. resolve_responses_only_num_proc handed the zoo a bare None to mean serial, but the zoo's own veto reads stdlib multiprocessing. Where multiprocess is on spawn while stdlib is on fork, that None is read as "size it for me". It re-encodes as 1 when the two disagree. The CPU-count test patches move to the resolved count: patching psutil alone would let a 4-vCPU runner override a test that asks for 128. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Mirror the cgroup usage-path correction from the zoo /sys/fs/cgroup/memory.current is the whole machine's usage at the root, so subtracting it from a systemd unit's own MemoryMax left every run with nothing free. Usage now comes from the directories the limit was resolved from. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-run CI Every check on the previous head was cancelled at 16:33 by an Actions-level event that hit both repos and many unrelated branches, main included. * Bound the paths the review found still unbounded Five findings, all about a value reaching Dataset.map without the policy. The config sentinel was written as 1 for every patched trainer. Only SFT has a downstream auto-sizer to defend it against: DPO, KTO, CPO, ORPO, Reward and PRM hand args.dataset_num_proc straight to Dataset.map, where nothing can inflate a None but a 1 is a Pool(1) on datasets >= 4.1 -- one worker holding its own tokenizer copy, on the low-memory host that had just refused workers. The codegen now picks the encoding per trainer. train_on_responses_only with UNSLOTH_DATASET_NUM_PROC=0 and an explicit count returned 1, which bypasses the small-split guard and builds that Pool(1) even on a 100-row split. Under the threshold the guard is in-process, so None is what expresses the request exactly, and that is what it now returns. Studio's dataset_map_num_proc handed its own count straight to callers in format_conversion.py and chat_templates.py with no memory ceiling and no environment override, though the cap's log line advertised one. It now runs the count through the shared policy when unsloth_zoo has it, so those paths get the memory and cgroup clamp and the escape hatch, and the log line no longer names a variable that path never read. The fallback copy moves to unsloth/dataset_num_proc.py. Under unsloth/utils it sat behind an __init__ that imports .packing (torch) and .attention_dispatch (unsloth.models._utils), so a torch-free MLX host with an older zoo raised before train_on_responses_only could delegate. Also found while running the wider suite: the tokenizing map() anchor was required, so a Zoo release moving that line would hard-fail every SFT run over a diagnostic wrapper. It is optional now, like the selection anchor above it, and the drift canary is what reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Mirror the zoo cgroup fix into the fallback copy unsloth_zoo.dataset_num_proc is the source of truth; this copy exists only so upgrading unsloth alone still fixes the bug, and test_the_two_copies_have_not_drifted holds the two together. Also moves the three new files onto the AGPL-3.0 header the repo now uses for new sources. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep serial requests serial across the config boundary and the drifting anchor Three review points. Studio computes dataset_num_proc for a config, not for a map() call, and the shared policy was applied with the map-site sentinel: the audio and CUDA-audio paths ask for 1, that became None, and SFTConfig read None as "auto-size me" and returned 8. Measured on the generated class, not simulated. The XPU leg was worse: unlike win32 and darwin, forking still works there, so a config None was auto-sized back up and forked the Level-Zero context the guard protects. dataset_map_num_proc now takes serial_as_none, default True so the seven map-site callers are untouched, and the trainer passes False. The spawn platforms keep None at both layers, where nothing can inflate it and a 1 would reach Dataset.map from DPO and friends as a Pool(1). The sft_prepare_dataset num_proc anchor was optional on the grounds that its absence was harmless. It was not: the config layer encodes serial as 1 for that rewrite to turn back into None, and an un-rewritten zoo hands the 1 to Dataset.map, which pools for any count from datasets 4.1. It now falls back to the assignment the block ends with, unchanged in the zoo since Aug 2025 while the block around it was rewritten three times in 2026, and warns only when both anchors miss. Hard-failing instead would break every install on a newer zoo. Two cgroup tests read the host tree once the fallback reader stopped needing unsloth_zoo, so they passed on a laptop and failed in a limited container. They are isolated now, and the six unaided-reader tests from the zoo copy came with them. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Skip the map-site half of the new tests when the policy is absent Two of the new config-boundary tests asserted dataset_map_num_proc(1) is None without guarding on the policy import, so they failed on all three cross-platform runners: unsloth_zoo there has no dataset_num_proc module yet (it ships in unslothai/unsloth-zoo#984), and _bounded_by_the_shared_policy returns the count unchanged in that case by design. Same pytest.importorskip guard the three memory and env tests beside them already use. With an unsloth_zoo that lacks the module the file is 15 passed, 6 skipped instead of 2 failed. * Let the shared policy see the request Studio was actually given Three points on the hardware.py side. safe_num_proc materialized an auto request before the policy could see it, so the policy never ran its own auto path: it reads this process's CPU affinity and cgroup quota, while safe_num_proc reads the host os.cpu_count(). A 2-core container on a 64-core box asked for cpu_count // 3 workers and was bounded only by memory. The request now passes through as written, and Studio's caps are applied to whatever the policy chose, since the multi-GPU fork-deadlock cap is knowledge the policy does not have. For an explicit count the two orders are equivalent, both being min(studio cap, request, affordable). The escape hatch is unvetoed by contract, but the win32/darwin return fired before the policy could read it, so UNSLOTH_DATASET_NUM_PROC was silently ignored on the platforms whose dead-worker message recommends it. It is now checked before that veto, and Studio's caps never apply to it. The older-zoo path returned the Studio count unchanged rather than trying unsloth.dataset_num_proc, the byte-identical fallback every other call site uses. It is used now, but only when unsloth is already imported: importing it from here would make hardware detection patch torch and pull in the model stack. The torch-less XPU branch routes through the policy too, having been the one path that ignored both the ceiling and the hatch. Seven new tests, 28 total, 17 passed and 11 skipped against an unsloth_zoo without the module. Reverting each of the three fails its own test. * Mirror the zoo test isolation and prose The fallback copy tracks unslothai/unsloth-zoo#984: the dnp fixture pins the memory ceiling at its sources, so a memory-limited runner cannot turn a start-method test into a clamp test, and the dead-worker advice now says that the single-worker exception applies to a Zoo older than the one that reads 1 as in-process. * Honour the hatch on XPU, leave the ordinary case to the policy, ignore typos Three follow-ups to the previous round, all of the same shape as fixes already made one line away. The XPU-initialized return bypassed the policy the way the spawn platforms did before this, so UNSLOTH_DATASET_NUM_PROC was ignored there too. It takes the same route now: the guard exists because fork corrupts the Level-Zero context, but a user who set the variable has accepted that, and unset the veto stands at both layers. The trainer's non-audio branch passed max(1, os.cpu_count() // 4), which the policy reads as an explicit request and so skips its own auto path, the only one that consults this process's affinity mask and cgroup quota. It passes None now, and Studio's caps still apply to whatever the policy chooses. The override probe treated any non-empty value as active, but the policy warns about and ignores an unparseable or negative one, so a typo skipped the multi-GPU cap while contributing nothing. It reads the parsed result through the zoo's new environment_override(), falling back to presence on a copy that predates it. Four new tests, 32 total. Reverting the XPU check or the override probe fails three of them; the trainer's None is pinned by an AST guard, since dropping it changes only the worker count on a container. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the encoding on the cgroup reads tests/test_runtime_text_encoding.py caught both call sites the unaided cgroup reader added: a locale-dependent text read crashes or produces mojibake on a Windows console codepage, and the gate is absolute even for ASCII kernel files. Mirrors unslothai/unsloth-zoo#984. * Neutralise the zoo cgroup readers by name in the num_proc fixture Pinning hf_xet_tuning.CGROUP_ROOT only works against a zoo that has that global. An older one still exposes the private dir helpers the policy prefers, so monkeypatch finds nothing to pin and the readers walk the runner's real cgroup: under a 2GB memory.max that turns a test about the start method into a test of the clamp. * Patch the zoo cgroup readers through the cache the policy actually uses unsloth_zoo/__init__ imports hf_xet_tuning near the top and only raises "Please install Unsloth" at the end, so a failed package import drops unsloth_zoo from sys.modules and leaves unsloth_zoo.hf_xet_tuning behind. The policy reaches the submodule through that surviving cache entry, so treating the failure as absence left the real readers live on the runner's own /sys/fs/cgroup and every sizing assertion silently became a test of the container's memory limit. * Make the Studio half of this PR actually run, and three tests mean what they say studio-backend-ci is the only job that executes studio/backend/tests, and it installs studio.txt, which carries no unsloth_zoo: 14 of the 32 cases importorskip away there, and with no policy installed the survivors fall back to the pre-PR safe_num_proc, so they would pass with the whole wiring deleted. Run the file in the hard-gate step instead, which has an editable unsloth_zoo, and pin the memory ceiling so the counts are not really assertions about the runner's free RAM. test_env_override_is_uncapped never exercised the exemption it is named for: the fixture leaves room for 512 workers, so asking for 100 was never near the clamp. test_unrelated_errors_pass_through_untouched held under 'except Exception' too, since the guard re-raises the same object; it now also passes a non-RuntimeError carrying the dead-worker text. And the codegen tests supplied their own copy of rl.py's serial_as_none rule, which made them self-fulfilling -- they now read it out of rl.py's AST, so flipping SFT to True fails the behavioural test and not only the literal match. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reach the policy without importing unsloth, so the gate is not inert The CI step I added runs the Studio file in a job with an editable unsloth_zoo, but the policy module is not on unsloth_zoo main -- it is in the companion PR -- and the job clones main, so all 14 cases that reach the policy still skipped and the survivors still exercised the pre-PR path. The claim in that step's comment was wrong. The same gap is live in production: _shared_policy only fell back to the in-repo copy when unsloth was already imported, and no Studio backend module imports it, so the API process reaching format conversion got no policy at all on every install whose zoo predates the module -- the 2GB container with eight cores this PR exists to fix. It now loads the file off disk when the package is not imported, which is safe because the module is stdlib-only by design, and memoises through sys.modules so the warn-once state and the cgroup reads are not redone per map() call. The tests ask _shared_policy for the same object, so they patch what production uses: 32 pass with the zoo copy blocked, where 18 passed and 14 skipped before. Also parenthesise the source segment _rl_serial_as_none evals, matching its sibling: a formatter reflowing that ternary in rl.py turned all eight codegen tests into an IndentationError. And mirror the two cgroup and escape-hatch tests just added on the zoo side. * Count pools at the class, not at a module attribute datasets moved The hard gate I added surfaced this the first time the file ran against HF=latest: datasets 3.x and 4.x do 'from multiprocess import Pool', so datasets.arrow_dataset.Pool exists, but 5.x calls mp.Pool() and a spawn context instead and the attribute is simply gone, so the spy raised AttributeError on four Python versions. Patching multiprocess.pool.Pool's __init__ catches every route. Verified against a real datasets 5.0.1: num_proc=None builds no pool, num_proc=1 builds one, so the claim the test makes about 4.1+ still holds there. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-05 05:31:27 -07:00
Daniel Han	d4825e7a57	Fix the three CI failures blocking every open PR (#7722) ... * Build the packing test collator from TRL's live signature TRL 1.7.0 removed completion_only_loss from trl.trainer.sft_trainer.DataCollatorForLanguageModeling when label construction moved from collation into dataset preparation (huggingface/trl#6037). tests/utils/test_packing.py passed that kwarg on every construction attempt, so on TRL 1.x every attempt raised TypeError, _DummyTrainer.data_collator was never assigned, and test_enable_sample_packing failed with AttributeError on the HF=latest + TRL=latest matrix leg. Filter the collator kwargs against the installed dataclass signature so the dummy tracks whatever fields TRL currently exposes, and add a regression test pinning the production contract that enable_sample_packing only requires torch_call on the collator. (cherry picked from commit 5d8513bac7) * Retire the virgin container release-lag pin now that the released wheel carries #7549 The "virgin win container / overlay=false" row installs unsloth from PyPI on purpose, so its studio/setup.ps1 comes out of the released wheel. A Server Core container has no Microsoft Store and so no winget, and the released setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime, so that row could not install end to end. #7549 relaxed both gates but merged on 2026-07-29, after the then newest wheel 2026.7.5 (2026-07-23), so the row was held under continue-on-error and a step that asserted it failed only on that known signature. unsloth 2026.7.6 shipped later the same day and is the first wheel to carry the relaxed gates: unpacking it shows the old "Git is required but could not be installed automatically" wording gone and both "so git is not needed" and "downloading the runtime directly" present. The row now installs end to end and the pin's own tripwire fired to say so. Drop continue-on-error from the Install step and delete the release-lag assertion, so the released-wheel row gates like the overlay row. The Install step is already passing on this row in recent runs, so nothing else is being tolerated by the escape hatch being removed. (cherry picked from commit cf0b25ea06) * Check the harness verdict on both container rows Removing the released row's tripwire made a green released row reachable for the first time, and the only log-based verdict check sat behind `if: matrix.overlay`. Every later step is `if: always()` and exits 0, so that row was gated on `docker exec`'s exit code alone -- which the file already notes can be 0 for an exec that lost its container. Hoisted the marker check out of the overlay step so both rows read the harness's own verdict; the overlay-specific assertions stay gated. (cherry picked from commit 7613460e81) * Wait for the floating monitor's system payload before baselining its geometry The geometry check baselines the panel with a single bounding_box() taken as soon as the caller observes an /api/system request go out. The response has not been applied yet at that point, so the panel is still painting use-system.ts's zero-filled DEFAULT_SYSTEM, which reports no GPU. On a host that does report one the VRAM row then lands and adds a row permanently, so the panel never returns to the sampled baseline and "content shrink" waits out its whole 5s deadline before failing. That is why this looked flaky but is not: macos-14 reports a single MLX device and failed on every PR, while ubuntu-latest and windows-latest have no GPU row to add and passed. Three jobs share the name "Chat UI Tests", so the one consistent platform failure read as one of three parallel runs failing at random. Wait for the payload to be rendered and for the panel to have finished resizing to it before sampling the baseline. The gate requires a non-zero RAM total, the scroll region to exactly fit the content it was reconciled against, and that geometry to hold for two consecutive animation frames, which is how Playwright defines a stable element and is what wait_for_function's default polling="raf" provides. (cherry picked from commit 7d5cc8c190)	2026-08-01 02:59:03 -07:00
Taranum Wasu	7367955bcb	fix: replace deprecated transformers attention mask imports (#6880) ... * fix: replace deprecated transformers attention mask imports Add a local attention-mask compat module adapted from Transformers so Unsloth no longer imports modeling_attn_mask_utils directly. This keeps training working when the module is removed in Transformers v5.10 and eliminates FutureWarning spam during forward passes. Closes #6860 Co-authored-by: Cursor <cursoragent@cursor.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: use device-agnostic mask inversion in attn compat Replace torch.tensor(1.0) subtraction with 1.0 - expanded_mask to avoid CPU/GPU device mismatches and float16 sub_cpu errors on some platforms. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(attn-mask-compat): fall back when transformers < 4.52 lacks is_tracing The compat module imports `is_tracing` from `transformers.utils.import_utils`, but that symbol is only exported from transformers >= 4.52 (added when the upstream `_prepare_4d_attention_mask_for_sdpa` rewrite landed). Unsloth declares `transformers>=4.51.3` in pyproject.toml, so on the lower bound tested by CI (`__from_pyproject__` matrix cell) the new top-level import raises ImportError before any patched module can import the compat helpers. Mirror the conservative pre-`is_tracing` upstream behavior: when the symbol is missing, define a local `is_tracing(tensor=None)` that only consults `is_torchdynamo_compiling`. The `tensor` argument is accepted but ignored — the older release has no public API for fake-tensor / JAX-jit detection via `import_utils` either, and the previously-used upstream code path on transformers 4.51.x guarded the same checks by `is_torchdynamo_compiling` alone. Tests: - `test_import_falls_back_when_is_tracing_missing` reloads the compat module with `is_tracing` removed from `transformers.utils.import_utils` and asserts the local fallback is in use (returns False when dynamo is idle, accepts the optional tensor positional arg). - All 26 tests in `tests/utils/test_attn_mask_compat.py` pass. This addresses the Codex review feedback (P1) on PR #6880. Signed-off-by: Taranum Wasu <taranumwasu@Taranums-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix(attn-mask-compat): preserve JIT/FX tracing detection in fallback Codex review follow-up on PR #6880 (P2). The previous fallback (``def is_tracing(tensor=None): return is_torchdynamo_compiling()``) was strictly less conservative than what upstream ``transformers==4.51.3`` did inline before ``is_tracing`` was added to ``transformers.utils.import_utils``. The pre-4.52 upstream expression was: is_tracing = torch.jit.is_tracing() or isinstance( inputs_embeds, torch.fx.Proxy ) or is_torchdynamo_compiling() The previous fallback only consulted Dynamo. That meant callers tracing or exporting under transformers 4.51.x would silently hit the data-dependent ``torch.all(attention_mask == 1)`` branch in ``_ignore_causal_mask_sdpa`` (and the equivalent in ``_prepare_4d_attention_mask_for_sdpa``) — failing on proxy control flow or baking the wrong SDPA causal-mask path. The local fallback now mirrors the legacy upstream expression: - ``torch.jit.is_tracing()`` for ``torch.jit.trace`` / ``torch.jit.script`` flows. - ``isinstance(tensor, torch.fx.Proxy)`` for ``symbolic_trace`` and ``torch.export`` paths that don't go through Dynamo. - ``is_torchdynamo_compiling()`` for ``torch.compile`` and ``torch._dynamo`` paths. The CUDA stream capture, FakeTensor, and JAX (torchax) checks the modern ``is_tracing`` does are out of scope: those need newer ``import_utils`` helpers, and the conservative dynamo fallback is the right choice when those helpers aren't available. This matches the behavior of the upstream ``is_tracing`` helpers that landed in 4.52 in the first place — those were new additions, not behaviour that pre-existed in 4.51.x. Tests: - ``test_import_falls_back_when_is_tracing_missing`` now exercises all three branches (dynamo idle → False; ``torch.fx.Proxy`` arg → True; patched ``torch.jit.is_tracing()`` → True). - All 26 tests in ``tests/utils/test_attn_mask_compat.py`` pass. - ``ruff check`` and ``ruff format`` clean on both files. Signed-off-by: Taranum Wasu <taranumwasu@Taranums-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope _unmask_unattended to the explicit-mask branch for PR #6880 Upstream calls _unmask_unattended only when the caller supplied an attention_mask (modeling_attn_mask_utils.py:411 on 4.57.6, :427 on main). The vendored copy had it at function scope, so it also ran on the attention_mask is None -> to_causal_4d path. Values were never wrong there: a causal or sliding-window row always attends to its own diagonal, so no row is entirely -inf and the multiply is a no-op. But to_causal_4d returns a stride-0 expand view, and .mul() materialises it into a dense [bsz, 1, q, kv] tensor. Measured on a B200, bf16: bsz 32, q_len 1, kv 8192 1.34x slower, storage 0.50MB vs 0.02MB bsz 8, q_len 4096 7.06x slower, peak 544MB vs 128MB After this change both match upstream exactly (1.00-1.03x, identical storage). The existing equivalence test could not catch it: it builds inputs_embeds on CPU, where the device.type in ("cuda", "xpu") gate is False for both sides. Added a CUDA counterpart that asserts stride and storage as well as values. Also corrected the is_tracing version note. The symbol first appears in transformers 5.0.0, not 4.52, so the fallback is the live path for all of 4.x rather than an edge case at the 4.51.3 floor. Restored HuggingFace's copyright line alongside ours, and replaced the "v5.10+" claim in the module docstring with what is actually true today: the module still ships in 5.14.1, but transformers stopped importing it itself after 5.5.0 (135 internal references at 4.51.3, none from 5.5.0), so it can be dropped at any release. Verified: 34560 differential checks CPU+CUDA with 0 value mismatches and 0 layout divergences; 2880 checks in each of ten isolated venvs covering Python 3.9-3.14, transformers 4.51.3/4.57.6/5.5.0/5.14.1 and torch 2.6.0/2.7.1/2.8.0/2.9.1; bit-identical logits and generated token ids against main on Llama-3.2-1B and gemma-2-2b-it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore upstream mask inversion form and fix test license header for PR #6880 _expand_mask was changed to a Python float literal (1.0 - expanded_mask), which reverts huggingface/transformers#38637. Upstream uses a 0-dim tensor from 4.53.0 onward so ExecuTorch edge-dialect lowering does not see an fp32 scalar against an fp16/bf16 mask. Both forms are bitwise equal on every supported torch, so this costs nothing and keeps the vendored copy matching upstream. Also switch the new test file to the AGPL-3.0 SPDX header used by the rest of tests/ instead of LGPL-3.0, which is not used anywhere in the repo. * Add upstream drift test for the vendored attention-mask helpers (PR #6880) _attn_mask_compat.py is a hand-copied subset of transformers' modeling_attn_mask_utils.py, and hand-copied code drifts silently. Both defects found while reviewing this PR were invisible in the diff and only surfaced under differential testing, so add a test that compares the two ASTs directly. Stylistic differences are normalised away first: docstrings, annotation text, the deprecation warnings the copy exists to drop, the 4.x inline is_tracing expression against the 5.x helper, single-use temporaries and a dead else after a return. Two deliberate forward-ports (the xpu device gate from 5.x, the 0-dim inversion tensor from 4.53.0) are relaxed only below the version that introduced them, so newer installs still compare them exactly. Verified 11/11 symbols match on transformers 4.51.3, 4.57.6, 5.5.0 and 5.14.1, that the test skips when the upstream module is absent, and that it fails on both defects this PR already fixed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Taranum Wasu <taranumwasu@Taranums-MacBook-Pro.local> Co-authored-by: Taranum Wasu <taranumwasu@Taranums-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Etherl <61019402+Etherll@users.noreply.github.com> Co-authored-by: danielhanchen <unslothai@gmail.com>	2026-07-31 04:25:56 -07:00
Daniel Han	5017a6145c	Make tests/utils runnable again and cover test_packing.py in CI (#7642) ... * Make tests/utils runnable: fake configs need to_dict, torchao and CUDA guards * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-30 03:16:33 -07:00
Elia	2e1862cbe2	fix(trainer): warn loudly when requested packing is silently disabled (#7234) ... * fix(trainer): warn loudly when requested packing is silently disabled When packing=True is requested but the model is a VLM / processor-based / uses a custom collator, auto-packing is turned off. Previously this only printed a terse 'Sample packing skipped' line and never mentioned the consequence: sequences longer than the max sequence length are truncated rather than split, so long-document datasets (e.g. raw-text CPT) can silently lose a large fraction of their tokens. Escalate to logger.warning, state that overlength samples are TRUNCATED (not split), include the effective max sequence length when available, and point users at pre-splitting/pre-packing. Messaging-only; behaviour unchanged. Fixes #7206 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the packing skip warning for PR #7234 Two corrections to the message: - Name UNSLOTH_RETURN_LOGITS instead of falling back to "custom data collator". unsloth sets that env var itself for compute_metrics, so the warning blamed a collator the user never passed. - Drop the token count. It is read before max_seq_length, max_length and the model's own limit are reconciled, so it printed 1024 for a 2048 model and 512 for a 2048 model in the common notebook flows. Adds a regression test. * Keep the custom-collator reason and drop the split claim for PR #7234 A stray else on the reason chain overwrote "custom data collator" whenever a collator was passed and no other cause matched, so the message named an env var that was not set. Guard the fallback on data_collator is None instead. Also drop "rather than split into additional sequences": TRL's default packing strategy is bfd, which truncates to max_length anyway, so packing would not have preserved those tokens. * Scope the data-loss claim to wrapped packing and drop the setter name for PR #7234 * Treat legacy trl as wrapped and skip the claim when prep is skipped for PR #7234 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Shorten the packing skip warning for PR #7234 * Reduce PR #7234 to a one-line packing skip warning --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-07-30 02:58:08 -07:00
JoshuaL3000	e662af769b	fix: enable XPU support and update hardcoded CUDA selections for tests (#7401) ... * fix: add XPU device support and update hardcoded CUDA selections * fix: add XPU device support for pytest CUDA skipped tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix device handling for PR #7401 - perplexity_eval.py: use DEVICE_TYPE_TORCH, not DEVICE_TYPE. The latter can be "hip" or "mlx", which .to() rejects, so this regressed ROCm. - test_batched_leftpad_generation_gpu.py: XPU diverges here today, so mark it non-strict xfail on XPU instead of reverting to a CUDA-only guard. Keeps the real XPU gap visible and turns green once it is fixed. - Guard torch.xpu.is_available() with hasattr, matching device_type.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-enable the flash varlen attention test in CI for PR #7401 attention_dispatch.py now predefines flash_attn_func / flash_attn_varlen_func as None, so test_run_attention_flash_varlen_receives_window_and_softcap no longer needs flash_attn importable to be monkeypatched. Verified on a runner shaped like the CPU-only one: the test fails against main's attention_dispatch and passes at this head, so the deselect is now dead weight. * Tighten comments for PR #7401 Drop the hasattr rationale: torch.xpu has existed since torch 2.3 and the dependency floor is 2.4, so no supported build predates the namespace. The guard stays as cheap defence, but the comment claimed something untrue. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-07-28 15:38:57 -07:00
Leo Borcherding	1dd2fc4583	tests: read checked-in files as UTF-8 instead of the platform default (#7438) ... * tests: read checked-in files as UTF-8 instead of the platform default Path.read_text() with no encoding uses locale.getpreferredencoding(), which is UTF-8 on the Linux runners and cp1252 on a stock Windows install. Nine module-level reads of checked-in source files were relying on that default. studio/backend/routes/inference.py carries the DeepSeek tool-call token regexes, so it holds U+FF5C and U+2581. Under cp1252 that read raised UnicodeDecodeError on byte 0x81 at position 97806, and because the reads run at import time it took test_cancel_atomicity.py and test_cancel_id_wiring.py out at collection, not as failures. Green on CI, permanently broken for a Windows contributor running the suite locally. Adds a guard: at module scope there is no tmp_path fixture, so a bare read_text()/write_text()/open() there is always touching a checked-in file. That makes the rule mechanical enough to enforce with no allowlist, while staying quiet about temp-dir I/O inside test bodies where the platform default is harmless. The repo already spells this correctly in 464 other places; this only stops the stragglers coming back. * tests: cover import-time helper reads and keep the guard py3.9-safe Follows up on the Codex review: - add `from __future__ import annotations`, since `str | None` in `_offender` is evaluated at import on Python 3.9 and pyproject declares requires-python ">=3.9,<3.15". - widen the guard from module scope to import time. Class bodies and the bodies of module-level helpers called from an executing statement run during collection too, so `CODE = _extract_mixed_precision_code()` was the same hazard as an inline read. `if __name__ == "__main__":` blocks are skipped: pytest never executes them. - scan studio/backend/tests/ as well as tests/. Both trees are collected on Windows by separate CI jobs, and the offender that started this, test_tool_xml_strip.py reading routes/inference.py, lives there. Widening it surfaced seven more import-time reads of checked-in sources; all now name utf-8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden the import-time encoding guard for PR #7438 Close the detector gaps raised in review, all of which I reproduced against the actual AST before changing anything. False negatives (the guard let a real hazard through): - _is_main_guard ignored the comparison operator, so if __name__ != "__main__" counted as script-only even though its body runs at import. - The else arm of a main guard was discarded with the rest of the If node. - Decorators and argument defaults on a module-level def were skipped with the body, though both are evaluated when the def executes. - Path.open() in text mode was invisible; only builtin open() was matched. - encoding = None and encoding = "locale" both re-select the platform default, but the keyword merely being present counted as pinned. False positives (the guard would have blocked a compliant contributor): - A non-literal mode fell through to the "r" default, so open(p, mode) was flagged even when mode is "rb", where adding encoding= is a ValueError and there is no edit that satisfies the rule. - Same for open(*args) and a **kwargs splat, which hide the mode and can hide an encoding. - Lambda bodies and comprehension elements were walked even though neither runs at definition. Verified: still reports the same 22 offenders on unpatched main, green on this branch and on the tree merged with latest main (557 files), and an adversarial corpus of 33 cases now scores zero false positives and zero false negatives. Also corrected two docstring claims: neither collecting job runs on Windows, and the read is governed by locale.getencoding(). * Walk eager comprehensions and treat io.open as the builtin Two regressions from the previous commit, both reproduced against the AST before changing anything. Lumping list, set and dict comprehensions in with generator expressions was wrong. Only a genexp is lazy; the other three run their element expression, their filters and their nested iterators immediately, so CONTENTS = [p.read_text() for p in PATHS] at module scope is an import-time read the guard was silently missing. Comprehensions are now walked in full and only the genexp keeps the outermost-iterable-only treatment. io was also in the not-a-path-opener list, but io.open is the builtin, with the same mode position and the same platform default. io.open(CHECKED_IN_FILE) is exactly the hazard this guard exists for, so it is matched now, with binary modes and a pinned encoding still exempt. tarfile.open and fitz.open stay exempt since neither has an encoding to name. Verified: 13 targeted cases covering all five eager comprehension forms and io.open in text, binary and pinned shapes all classify correctly; still 22 offenders on unpatched main; green on this branch and on the tree merged with latest main. * Close three more walker gaps in the import-time guard All three reproduced against the AST first. A generator expression handed straight to a call is consumed there, so DATA = "".join(p.read_text() for p in paths) runs its element at import. Only an unconsumed genexp bound to a name stays lazy, so the walker now follows the consumed ones in full and keeps the outermost-iterable-only treatment for the rest. if "__main__" == __name__ is an equivalent and accepted spelling of the main guard, but requiring __name__ on the left meant its body was treated as import-time code. That is a false positive on a block pytest never runs, so both operand orders are recognised now. The helper table was built from module-level defs only, so a def in a class body invoked while the class is constructed was never followed, contradicting the walker's stated coverage of class bodies. Helpers are now collected from the module body and from class bodies at any nesting. Verified: 15 targeted cases including all three fixes and the earlier ones still classify correctly; still 22 offenders on unpatched main; green on this branch and on the tree merged with latest main. * Handle positional read_text encodings, lazy generators and nested helpers * Guard reads reached from test bodies, unbound Path calls and __file__ paths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow derived paths, skip lazy generator helpers, cover compressed openers * Guard the CLI tests, helper parameters and unbound Path arguments * Discover test roots and follow literal, in-place and tuple-derived paths * Identify module openers by import, unwrap starred paths, pin subprocess snippets * Resolve import origins, seed helper locals, follow named generators and parametrize * Scope imports lexically, list tracked test files, bind unpacked names * Resolve aliased openers, keyword-only params, destructured targets, next() * Pin the encoding on subprocess snippets, workflow lint and CLI output for PR #7438 * Harden the CLI encoding guard against detached streams for PR #7438 * Tighten the encoding guard's path and scope analysis for PR #7438 * Resolve path provenance more precisely and keep POSIX stream encodings for PR #7438 * Resolve qualified path classes and scope conditional imports for PR #7438 * Scope CLI stream setup to the entry point and align two encoding pairs for PR #7438 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-07-26 23:31:56 -07:00
Daniel Han	07272b9278	Experimental: correct varlen sample packing for hybrid linear-attention models (#7249) ... * Fix text-only VLM CPT packing truncation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Handle streaming vision datasets in packing * Harden multimodal packing detection * Preserve safe packing boundaries * Scope stream packing checks to VLMs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Narrow VLM packing detection * Align packing mode and eval safety * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add qwen3_5/qwen3_next to PADDING_FREE_BLOCKLIST to avoid packed-sequence contamination * Detect hybrid linear-attention models structurally instead of by name for packing guard * Add experimental varlen packing for hybrid linear-attention models Feed seq_idx to the causal conv and cu_seqlens to the gated-delta scan so sample packing / padding-free reset state at sequence boundaries for hybrid linear-attention models (Qwen3.5, Qwen3-Next). Gated behind UNSLOTH_EXPERIMENTAL_HYBRID_PACKING and fail-closed: when the flag is off or the accelerated kernels (causal_conv1d + fla) are unavailable, the guard keeps these models on the padded path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden hybrid linear-attention varlen packing shim Make patch_hybrid_linear_attention_varlen robust across transformers 4.57.6 through 5.x and TRL 0.22.2 through 1.x, following the import_fixes.py style: - Read UNSLOTH_EXPERIMENTAL_HYBRID_PACKING at call time so the flag takes effect when set after importing unsloth. - Idempotent: repeat calls on a patched model return True without re-validating the wrappers or double-wrapping; signatures are checked on captured originals. - Prefer the authoritative packed_seq_lengths (via get_packed_info_from_kwargs) over position_ids resets, handling pad_to_multiple_of trailing tokens. - Suppress injection for cached forwards (use_cache / past_key_values) so generation and eval are left on the untouched decode path. - Validate every gated-delta module before mutating any (transactional). - Bind position_ids / use_cache from both positional and keyword args. - Verify dispatch at runtime (Unsloth wraps each module forward, so the mixer source is not statically inspectable) and warn once if the shim is never hit. - Emit one deduped diagnostic on each fail-closed path. Add CPU unit tests covering the hybrid guard detection, the boundary builders, and the shim (fail-closed, active, idempotent, cached no-op, runtime handshake). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Abort hybrid packing when the varlen shim is not fully dispatched The runtime handshake used a single per-module hit flag written by both the conv and scan wrappers, so a partial dispatch (only one kernel routed through self.<kernel>) passed the any() check and trained on contaminated data, and a missing dispatch only logged a warning. Track conv and scan dispatch separately, require both on every gated-delta module on the first packed forward, and raise before loss/backward when either is missing (the batch is already flattened, so there is no padded recovery at that point). Also skip an empty packed_seq_lengths before it reaches max(), and document the position_ids fallback's left-pad assumption. Add tests for no-dispatch and partial (conv-only / scan-only) abort, the packed_seq_lengths preference over a competing position_ids, MRoPE 3D position ids, and the pad_to_multiple_of trailing-segment path through the metadata builder. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Import the hybrid packing patch from its submodule to satisfy the import-hoist lint * Fail closed for hybrid packing on encoder-decoder, chunked-loss, and string-name models The varlen shim only helps decoder-only hybrid models that run their mixer through self.<kernel> on a live nn.Module forward. Three cases slipped past the guard: - Encoder-decoder configs (is_encoder_decoder) reached the packing path even though flattening a cross-attention batch is unsound. Block them explicitly. - TRL's chunked_nll loss (the 1.x default) calls the backbone directly and bypasses model.forward, so the per-instance forward wrapper that refreshes the varlen stash never runs. Detect that path and keep the model padded. - A string model_name reaches the trainer before the module exists, so the instance shim has nothing to patch. Resolve the config up front and keep string hybrids on the padded path. Adds encoder-decoder / decoder-only / chunked-loss / string-model tests. * Harden the SFT source-injection replacements and forward auth args for string models The wrapped-packing injection rewrote the sourced unsloth_zoo sft_prepare_dataset with str.replace anchored on the exact 'All Unsloth Zoo code licensed under LGPLv3' comment. str.replace never raises on a missing anchor, so a supported newer unsloth_zoo (the dependency is only lower-bounded) that moved that header would silently drop the setup while the truncation and pack_dataset edits still referenced _unsloth_wrapped_packing / _inspect, raising NameError on every SFT dataset preparation. - Install the setup at the sft_prepare_dataset signature via re.subn (a structural anchor that always exists) and raise if even that is missing. - Route the remaining edits through a _require_replace helper that fails loudly on a missing required anchor (or warns once for an optional one), formalizing the verify-then-replace idiom the DPO patchers in this file already use. - Reuse the guarded _unsloth_pack_has_strategy at the pack_dataset call instead of re-calling inspect.signature(pack_dataset) unguarded, so a non-introspectable pack_dataset cannot crash there after the setup already handled it. - _resolve_string_model_config now forwards token / use_auth_token / cache_dir / code_revision, so a private hybrid resolves its config instead of falling through as non-hybrid and enabling packing without the varlen shim. Adds regression tests for the drift-resistant injection, the helper, and the string-model auth forwarding. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Honor top-level SFTConfig.trust_remote_code when resolving a string model TRL merges the top-level args.trust_remote_code into the load via model_init_kwargs.setdefault("trust_remote_code", args.trust_remote_code) before create_model_from_path, so a remote-code hybrid is commonly set with SFTConfig(trust_remote_code=True) rather than inside model_init_kwargs. The config probe only read model_init_kwargs, so AutoConfig could fail for such a model, leave model_config None, and let the guard treat it as non-hybrid, enabling packing without the varlen shim. Mirror TRL's setdefault (model_init_kwargs wins). * Tighten hybrid-packing comments for concision * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: alkinun <alkinunl@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Etherl <61019402+Etherll@users.noreply.github.com>	2026-07-20 00:57:02 -07:00
alkinun	9e334d552c	Fix text-only VLM CPT packing truncation (#7211) ... * Fix text-only VLM CPT packing truncation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Handle streaming vision datasets in packing * Harden multimodal packing detection * Preserve safe packing boundaries * Scope stream packing checks to VLMs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Narrow VLM packing detection * Align packing mode and eval safety * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add qwen3_5/qwen3_next to PADDING_FREE_BLOCKLIST to avoid packed-sequence contamination * Detect hybrid linear-attention models structurally instead of by name for packing guard * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Install wrapped-packing setup at the signature, not the Zoo license comment The _unsloth_wrapped_packing / _inspect setup block was injected by matching the exact 'All Unsloth Zoo code licensed under LGPLv3' comment line in the sourced sft_prepare_dataset. The unsloth_zoo dependency is only lower-bounded, so a newer Zoo that moves or drops that header made the setup a silent no-op while the truncation and pack_dataset rewrites still emitted references to those names, raising NameError on every SFT dataset preparation. Anchor the setup on the function signature instead (a structural location that always exists) and fail loudly if it cannot be found, so the helper variables are always defined before they are referenced across Zoo versions. Adds a regression test that patches in a Zoo source without the license header. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Etherl <61019402+Etherll@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-07-20 00:23:37 -07:00
oobabooga	ed42702730	Probe xformers support on sm_120 instead of disabling it by version (#6828)	2026-07-14 00:01:25 -03:00
Daniel Han	534c877d21	Keep native RoPE scaling when extending context; carry rope_theta for linear (#7028) ... * Keep native RoPE scaling when extending context; carry rope_theta for linear When max_seq_length exceeds a model's native window, the loader overwrote the model's rope_scaling with linear scaling. For models that already ship a scaled RoPE (llama3/yarn/longrope) that is far worse for long context, and on transformers v5 the linear dict omitted rope_theta (v5 keeps it under rope_parameters), so the rotary base fell back to 10000 and broke past ~8K tokens. Keep the native scaling and just widen the window; only synthesize linear for plain-RoPE models, and carry rope_theta so v5 keeps the real base. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only preserve native llama3 when extending context; keep linear fallback otherwise The patched attention constructor (patch_llama_rope_scaling) rebuilds only linear, llama3 and longrope and its longrope branch reads a top-level original_max_position_embeddings, so preserving yarn or a nested-only longrope config would raise during construction on transformers <= 4.47.1. Keep only llama3 native; yarn/longrope/other types fall back to the linear override, still carrying rope_theta. * Correct long-context extension comment to match llama3-only preservation --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-09 04:20:41 -07:00
Daniel Han	bdb958e052	Guard RoPE scaling against the transformers v5 buffer blank; honor extended RoPE factor (#6925) ... * Guard RoPE scaling against the transformers v5 buffer blank; honor extended factor Add a family-agnostic guard that builds each rotary from a scaled config, blanks its non-persistent buffers (what transformers v5 does on load), runs loader._fix_rope_inv_freq, and asserts every buffer is restored to its scaled value (llama3 and longrope). This catches the whole bug class, not just the one call site, and is validated to fail on the pre-fix repair. Also make LlamaExtendedRotaryEmbedding read the llama3 factor from the config instead of hardcoding 8 (wrong for Llama-3.2, factor 32), falling back to the Llama-3.1 defaults when built without a config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pass config into extended rotary codegen; skip v5 round-trip on transformers 4.x - patch_llama_rope_scaling now builds the llama3 extended rotary with config=self.config so it reads the real factor (32 for Llama-3.2) instead of falling back to 8; the template already references self.config. - test_v5_blank_repair_roundtrip now skips when loader._NEEDS_ROPE_FIX is False, since _fix_rope_inv_freq is a no-op on transformers 4.x and cannot restore the blanked buffers there. * Raise stream deadlock-guard timeouts from 0.2s to 5.0s in passthrough tests These asyncio.wait_for guards bound test setup and cross-task event signaling that complete near-instantly on success; the 0.2s budget is a latency assertion in disguise and times out under CI scheduling load (seen on the 3.11 matrix leg while 3.10/3.12/3.13 pass the same commit). 5.0s matches the timeout used elsewhere in the suite and still fails fast on a real hang. No test relies on the guard expiring. * Extended rotary reads rope_parameters as well as rope_scaling transformers v5 stores llama3 scaling under config.rope_parameters and exposes rope_scaling only as a back-compat property. Reading that property works on 5.0-5.13 (verified: factor resolves to 32 for Llama-3.2), but a future release may drop the shim, after which the subclass path would fall back to factor 8. Read either field so the factor survives the rename. Adds test_extended_rotary_reads_rope_parameters_v5 (fails on the old single-field read: rope_parameters-only config resolves to 8, not 32). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-07 04:16:57 -07:00
Daniel Han	2fada48ef5	Fix llama3 RoPE scaling dropped on transformers v5 (#6907) ... * Fix llama3 RoPE scaling dropped on transformers v5 transformers v5 loads on meta then blanks non-persistent buffers, so _fix_rope_inv_freq rebuilds inv_freq after load. It recomputed a vanilla inv_freq and applied _apply_inv_freq_scaling, a no-op on the base LlamaRotaryEmbedding used by the config/llama3 path, so inv_freq ended up divided by 1 instead of the config factor (8 for Llama 3.1, 32 for Llama 3.2). This corrupts long-range positions and inflates long-context loss about 3-5x. transformers 4.x was unaffected. Route __init__ and the v5 repair through one _unsloth_recompute_inv_freq so they cannot diverge, and stash the config on the rotary module so the repair can rebuild the same scaled value. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add test for llama3 RoPE scaling under the transformers v5 repair * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update RoPE drift guard for the recompute refactor and guard the v5 repair The drift guard's AST tripwire asserted the config-scaling call lived in the if config is not None branch of LlamaRotaryEmbedding.__init__. The fix moved that into _unsloth_recompute_inv_freq, so follow it there (with a fallback to the old inline branch) and add a guard that loader._fix_rope_inv_freq rebuilds inv_freq through the same helper. Also add a CPU functional check of the helper and drop the redundant standalone test. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 09:13:14 -07:00
Daniel Han	9780cdcca1	Fix FlashAttention fp32 crash with DoRA (use_dora=True) (#6526) ... * Fix FlashAttention fp32 crash with DoRA (use_dora=True) DoRA upcasts lora_magnitude_vector to fp32 for the optimizer, which promotes the q/k/v_proj output to fp32. FlashAttention only accepts fp16/bf16, so the fp32 q/k/v raised 'FlashAttention only support fp16 and bf16 data type'. Downcast q/k/v to the compute dtype before the flash kernels. Fixes #1013 * Apply kwarg-spacing format hook to DoRA dtype test (pre-commit) * DoRA+FA2: downcast any fp32 among Q/K/V and clamp to a flash-supported dtype * Tighten code comments (no logic change) --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com>	2026-06-23 01:29:19 -07:00
Daniel Han	a6dc10dad2	Reduce and tighten comments and docstrings across the test suite (#6429) ... * Reduce and tighten comments and docstrings in tests Shorten verbose comments and docstrings across the test suite without changing any test logic. Remove narration that restates the next line, collapse long module and test docstrings to a single line, and drop banner separators. Keep regression context (issue and PR references, run ids), skip reasons, mocking and timing rationale, license headers, lint and type directives, and commented-out code. Comments and docstrings only: an AST signature check confirms no code, assertions, or string literals changed, and the suite byte-compiles cleanly. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-18 01:07:09 -07:00
Daniel Han	6dae2f525b	Stop false RoPE 'default' warning and fix rope drift gate on transformers 5 (#6223) ... * Handle rope_type 'default' on transformers 5 to stop false RoPE warning transformers 5 reports rope_type="default" for every plain (unscaled) config and dropped "default" from ROPE_INIT_FUNCTIONS. _compute_config_rope_inv_freq then did ROPE_INIT_FUNCTIONS["default"], hit KeyError, returned None and logged "Could not apply RoPE scaling 'default'; long-context generation may degrade" on every model load. The inv_freq was still correct (the constructor recomputes vanilla on None), but the warning is a false alarm for unscaled models. Compute the unscaled inv_freq directly for rope_type "default"/None instead of going through ROPE_INIT_FUNCTIONS, so plain configs return the right value with no warning. Scaled types (llama3/linear/yarn/...) are unchanged. Also skip test_object_style_rope_scaling_on_config_delegates_correctly when transformers strict-validates rope_scaling (5.x): it rejects a non-dict object on config.rope_scaling, so the object-style delegation path cannot be set up there. The test still runs and asserts on transformers <5. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-11 20:37:01 -07:00
Daniel Han	d24ee77f17	Fix Llama 3.1+ rope scaling dropped on the FastLanguageModel path (long inputs become gibberish past ~29K tokens) (#6197) ... * Fix config.rope_scaling being dropped by the replaced rotary embedding (#2405) On modern transformers, LlamaModel builds its rotary embedding from config using unsloth's replacement LlamaRotaryEmbedding class, whose config path computed vanilla inv_freq and ignored config.rope_scaling entirely. The llama3/linear/longrope dispatch in patch_llama_rope_scaling rewrites LlamaAttention.__init__, which no longer constructs rotary embeddings, so it never fires; the model-level rotary is then copied onto every attention layer. Result: Llama-3.1/3.2/3.3 ran with unscaled RoPE on the FastLanguageModel path and collapsed into repetition loops past roughly 29K tokens (PASS at 28867, FAIL at 31767 in needle retrieval). FastModel was unaffected because vision.py keeps transformers' own rotary. qwen2, qwen3, qwen3_moe, mistral and cohere assign the same base class, so any rope-scaled config of those families was equally exposed. The fix makes the base class config path compute inv_freq and attention_scaling via transformers' ROPE_INIT_FUNCTIONS (covers llama3, linear, dynamic, yarn, longrope), with an inline llama3 fallback reading factors from config for older transformers, degrading to prior behavior on any failure. attention_scaling is applied in _set_cos_sin_cache (1.0 default, exact no-op for unscaled paths) and persists across extend_rope_embedding. A type(self) guard prevents double-scaling via the legacy scaled subclasses. Adds tests/utils/test_rope_scaling_drift.py (AST tripwire + behavioral inv_freq/cos-cache/extension checks, validated to fail 4 of 5 on the unfixed code) and wires it into the existing consolidated CI HARD GATE step. Verified on GPU: 48K-token needle retrieval flips FAIL to PASS for FastLanguageModel in bf16 and 4bit, 20K stays PASS, scaled inv_freq matches transformers exactly, and the left-padded batch generation guard still gets exact solo-vs-batched token matches. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: normalize object-style rope_scaling, vectorize llama3 fallback config.rope_scaling can be a config object rather than a dict on newer transformers; _rope_scaling_as_dict normalizes it (to_dict/dict/vars fallbacks) before any .get() access, with a regression test using a dataclass stand-in. The inline llama3 fallback now uses torch.where instead of a per-frequency Python loop; verified bit-for-bit equal to transformers ROPE_INIT_FUNCTIONS for factor 8 (Llama-3.1) and factor 32 (Llama-3.2). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: CPU-safe rope guard tests, normalized config for delegation The rotary constructor builds per-device CUDA caches, so the behavioral tests that instantiate it cannot run on GPU-less CI. Restructured into three layers: the AST tripwire now also asserts the constructor stays wired to _compute_config_rope_inv_freq; the CPU layer tests that pure helper directly (llama3 dict, llama3 object, linear object, default type) with no instantiation; the instantiation and cache tests are gated behind a real CUDA probe (actual tensor allocation, so import-time CUDA spoofs cannot fool the gate). Verified: 9 passed with GPU; 5 passed 4 skipped with CUDA hidden; 5 failed 4 skipped on the unfixed code in CPU mode. Delegation to ROPE_INIT_FUNCTIONS now retries with a shallow config copy carrying the normalized rope_scaling dict when the original was an object the installed transformers cannot read; covered by a linear-object test, which has no inline fallback and passes only through that retry path. * Tighten comments in rope scaling fix and guard test Comment and docstring reduction only; verified code-identical with scripts/comment_tools.py check --strip-docstrings (AST signature match on both Python files). All guard tests unchanged: 20 passed with GPU, 5 passed 4 skipped with CUDA hidden. * Apply repo kwarg-spacing format --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-11 07:48:21 -07:00
Daniel Han	184141db99	Tests + CI guard: batched left-padded generation can never silently regress again (#1066, #3699) (#6145) ... * Add regression guard for batched left-padded generation (#1066, #3699) Three layers of tests plus a path-filtered CI workflow so the left-padding position_ids / attention-mask bug class cannot silently return: - tests/utils/test_prepare_inputs_ast_guard.py: import-free AST checks on _fast_prepare_inputs_for_generation (cumsum-from-mask branch present, cache_position only as fallback, no mask truncation, model families wired) - tests/utils/test_prepare_inputs_leftpad.py: CPU behavioral unit test with synthetic left-padded masks and fake caches; exact expected position_ids for prefill and cached decode - tests/utils/test_batched_leftpad_generation_gpu.py: optional GPU e2e, solo vs batched prefix match, skipped without CUDA - .github/workflows/batch-inference-guard.yml: ubuntu-latest CPU job running the two deterministic layers on PRs touching unsloth/models/** Validated: all pass on main; both CPU layers fail at 6d0f8643~1 (pre #4100) and at 332eabf3~1 (pre #2216), reproducing the historical bug signatures. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cite staging proof in batch-inference-guard header (staging-2 PRs 170/171) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fold left-padding guard into consolidated Core CI; merge AST + behavioral tests No new workflow and no new CI job: the guard now runs as one HARD GATE step inside consolidated-tests-ci.yml, right after the callback signature drift detector, where the CPU torch stack is already installed. The AST structural checks and the behavioral unit tests live in a single file (tests/utils/test_prepare_inputs_leftpad.py); the AST layer stays stdlib-only with unsloth imported lazily inside the behavioral tests, so import breakage cannot mask the structural checks. Revalidated after the merge: 11 assertions pass on main, 8 fail at 6d0f8643~1 (pre #4100). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update staging proof reference for consolidated gate (PRs 170/172) --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-10 08:00:28 -07:00
Daniel Han	187144d4e7	Reduce and tighten code comments and docstrings repo-wide (#6095) ... Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.	2026-06-08 23:09:51 -07:00
Daniel Han	3ce187da02	Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) ... Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.	2026-06-08 04:24:13 -07:00
Leo Borcherding	f4873182e0	Add None/empty content detection for conversation datasets (#4438) ... Adds studio/backend/utils/datasets/dataset_none_detect.py, a standalone scanner that reports None/empty content turns in alpaca, chatml, sharegpt, and gptoss datasets without modifying data, plus generator and runner scripts under tests/utils. Depends only on the datasets library and is not wired into the package init, so it stays import-light.	2026-06-03 05:03:43 -07:00
Avaya Aggarwal	7c5464ad71	feat: Add cactus QAT scheme support (#4679) ... * feat: Add cactus QAT scheme support * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * test(qat): add tests for cactus QAT scheme and fix missing import * Fix cactus QAT scheme: correct MappingType import, tighten PerGroup filter - Drop the broken `from torchao.dtypes import MappingType` import. `MappingType` lives in `torchao.quantization` (and `torchao.quantization.quant_primitives`); it is not exported from `torchao.dtypes` in any supported torchao release (verified on 0.14, 0.16, 0.17). The previous code raised `ImportError` on every cactus call and was masked as a misleading 'torchao not found' error. - Since `IntxWeightOnlyConfig` already defaults `mapping_type` to `MappingType.SYMMETRIC`, drop the explicit kwarg entirely and remove the import. Behavior is unchanged. - Introduce a named `group_size = 32` constant (matches the int4 / fp8-int4 pattern in the surrounding branches) and add a `% group_size == 0` divisibility guard to the filter. `PerGroup(32)` requires `in_features % 32 == 0` at `quantize_()` time, otherwise torchao raises `ValueError: in_features (N) % group_size (32) must be == 0`. The old `in_features >= 32` filter would admit non-aligned widths (e.g. 33, 48, 65, 127) and crash `_prepare_model_for_qat` for those shapes. * Warn when cactus QAT skips non-divisible Linear layers Multiple reviewers flagged that the divisibility guard added in the previous commit can silently leave Linear layers in full precision when their in_features is not a multiple of 32. For currently supported Unsloth models (Qwen, Llama, Gemma, Mistral, Phi) every Linear width is already a multiple of 32/64/128 so this never triggers, but surfacing the coverage gap is cheap and avoids users assuming 100% QAT coverage when they bring a custom model with unusual shapes. Emit a UserWarning listing up to the first 8 skipped layers whenever the cactus filter excludes any Linear due to the modulo guard. This keeps the lenient silent-skip behavior (consistent with int4 / fp8-int4), but stops making it silent. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-04-15 07:40:03 -07:00
Avaya Aggarwal	45d0a343b5	feat: Implement Q-GaLore optimizer and custom embedding learning rate… (#4511) ... * feat: Implement Q-GaLore optimizer and custom embedding learning rate in the Unsloth trainer. * feat: Implement QGaLoreAdamW8bit optimizer with 8-bit states, GaLore low-rank gradient projection, and optional INT8 weight quantization, along with supporting projector and tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feat: Introduce Q-GaLore AdamW optimizer with low-rank quantized gradient projection and integrate into the trainer, along with dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feat: Implement Q-GaLore AdamW optimizer with gradient projection and quantization, including trainer integration and corresponding tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix 3 bugs in Q-GaLore optimizer and add weight_quant forward hooks 1. Fix use-after-delete crash: move `del p._saved_data` after the weight decay block so decoupled weight decay can reference the current weights correctly (p.data). 2. Fix substring matching in make_q_galore_param_groups: split parameter names on "." and check exact component matches to prevent false positives (e.g. "not_q_proj" matching "q_proj"). 3. Implement forward pre-hooks for weight_quant: after the optimizer quantizes weights to INT8, replace p.data with a 1-element placeholder to free float memory. A register_forward_pre_hook dequantizes back to float before each forward pass. The trainer calls install_weight_quant_hooks() when weight_quant is enabled. 4. Update test_weight_decay_uses_saved_data to match the fixed code path (decoupled decay uses p.data, expected value 2.7). Add test_weight_quant_hook_restores_float to verify the INT8-to-float hook round-trip. All 24/24 Q-GaLore tests pass. Benchmarked on Llama-3.2-1B-Instruct FFT: Q-GaLore saves 32% VRAM (10.63 -> 7.24 GB) with better loss convergence (1.3 vs 2.0 at step 100). No regressions in 31-notebook sweep across Llama, Qwen, Mistral, Phi, Gemma, vision, and GRPO. * Default weight_quant to False in QGaloreConfig Benchmarks show weight_quant=True adds ~1 GB on Llama-3.2-1B due to INT8 copy/scale overhead exceeding savings from the placeholder trick. Users can still opt in explicitly. The optimizer logic is unchanged. * Optimize Q-GaLore projector and optimizer step performance Projector (q_galore_projector.py): - Use torch.svd_lowrank with oversampling p=10 (Halko et al. 2009) instead of full SVD for large matrices. Falls back to full SVD when min(m,n) <= 2*rank. SVD steps are 6-8x faster on Llama-3.2-1B (22s -> 3s for first step). - Cache the dequantized ortho matrix between project() and project_back() to avoid redundant dequantization when quant=True. - Replace F.cosine_similarity with torch.dot for 1-D unit vectors in the adaptive schedule. Remove unused torch.nn.functional import. - Use collections.deque(maxlen=queue_size) instead of list with manual pop(0). Optimizer (q_galore_adamw.py): - Remove redundant .clone() on dequantized weights (line 151) and on float data before re-quantization (line 211). _dequantize already returns a fresh tensor and _quantize/_quantize_stochastic only reads its input. - Consolidate per-group torch.cuda.synchronize() into a single call after all param groups complete. - Use torch.empty instead of torch.zeros for the scalar placeholder tensor that is never read. Verified: 24/24 unit tests pass. Llama-3.2-1B 61-step training produces losses within 0.24% relative diff (correlation >0.9999) of the original. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-03-25 01:03:10 -07:00
Daniel Han	3bddfed117	Patch trunc_normal_ for low-precision stability (#4027) ... * Fix low-precision trunc_normal initialization instability * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Document TorchTitan trunc_normal low-precision failure mode * Fix trunc_normal generator positional compatibility * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix trunc_normal generator TypeError fallback --------- Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-02-19 04:40:14 -08:00
Mohammad Miadh Angkad	336bec216a	Refactor Ollama template wiring and harden packing helpers (#3890) ... * Refactor Ollama template wiring and harden packing helpers Signed-off-by: Mohammad Miadh Angkad <MAngkad.BSDSBA2027@aim.edu> * Fix Qwen3 and Gemma3n template bindings and tidy packing test helper * Fix gptoss Ollama comment and tinyllama stop parameter - Fix wrong comment referencing gemma3n for gptoss_ollama in chat_templates.py - Add missing stop keyword to tinyllama PARAMETER in ollama_template_mappers.py * Fix _DummyTrainer compatibility across TRL versions The try/except only handled the removal of return_position_ids (TRL v0.24+) but not the absence of padding_free (TRL v0.18.2). Gracefully degrade through all optional collator flags so the test works from trl>=0.18.2 through v0.27+. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Mohammad Miadh Angkad <MAngkad.BSDSBA2027@aim.edu> Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-02-09 04:04:48 -08:00
electroglyph	d80e69258c	add weight-only int8 QAT scheme and update tests for torchao 0.15.0 (#3859) ... * add int8 weight-only QAT scheme, add test, fix tests for current torchao version * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * change quantization to PerAxis * lambda =/ * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add torchao messages, remove group_size from int8 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * raise exception on missing torchao * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * touch up the torchao imports * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-01-16 09:32:29 +05:30
Dan Saunders	75e0d7ce62	Auto-enable padding-free SFT (#3672) ... * implement (sdpa, xformers, fa2) sample packing * attention dispatching * ddp working OOTB with CLI * packed SWA and softcap support * enable batch flattening * LGPL license headers * mask packed sequence boundaries * auto-enable sample packing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add explicit toggle for sample packing * Add explicit toggle for sample packing * Update __init__.py * Update unsloth/kernels/rope_embedding.py * Update unsloth/kernels/rope_embedding.py * remove grad output clones; restore deleted FastLanguageModel arg * fix * restore rope embedding clones * xformers mask cache * implement (sdpa, xformers, fa2) sample packing * attention dispatching * ddp working OOTB with CLI * packed SWA and softcap support * enable batch flattening * LGPL license headers * mask packed sequence boundaries * auto-enable sample packing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add explicit toggle for sample packing * Add explicit toggle for sample packing * Update __init__.py * Update unsloth/kernels/rope_embedding.py * Update unsloth/kernels/rope_embedding.py * remove grad output clones; restore deleted FastLanguageModel arg * fix * restore rope embedding clones * xformers mask cache * add back accidental deletion * Update unsloth/kernels/rope_embedding.py Co-authored-by: Daniel Han <danielhanchen@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix merge conflicts * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add **kwargs * add back clobbered * Update rope_embedding.py * Update rope_embedding.py * simplify trl warnings filter * docstring * nit * bugfix * add padding-free seqlen metadata * auto-enable padding free * gemma2 disable * Apply suggestion from @danielhanchen * Update trainer.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update trainer.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2025-12-10 03:07:29 -08:00
Dan Saunders	496f84ff6b	SFT sample packing (#3566) ... * implement (sdpa, xformers, fa2) sample packing * attention dispatching * ddp working OOTB with CLI * packed SWA and softcap support * enable batch flattening * LGPL license headers * mask packed sequence boundaries * auto-enable sample packing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add explicit toggle for sample packing * Add explicit toggle for sample packing * Update __init__.py * Update unsloth/kernels/rope_embedding.py * Update unsloth/kernels/rope_embedding.py * remove grad output clones; restore deleted FastLanguageModel arg * fix * restore rope embedding clones * xformers mask cache * implement (sdpa, xformers, fa2) sample packing * attention dispatching * ddp working OOTB with CLI * packed SWA and softcap support * enable batch flattening * LGPL license headers * mask packed sequence boundaries * auto-enable sample packing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add explicit toggle for sample packing * Add explicit toggle for sample packing * Update __init__.py * Update unsloth/kernels/rope_embedding.py * Update unsloth/kernels/rope_embedding.py * remove grad output clones; restore deleted FastLanguageModel arg * fix * restore rope embedding clones * xformers mask cache * add back accidental deletion * Update unsloth/kernels/rope_embedding.py Co-authored-by: Daniel Han <danielhanchen@gmail.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix merge conflicts * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add **kwargs * add back clobbered * Update rope_embedding.py * Update rope_embedding.py * simplify trl warnings filter * docstring * nit * bugfix * Apply suggestion from @danielhanchen * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update unsloth/trainer.py * Update unsloth/trainer.py * Update unsloth/trainer.py * Update unsloth/trainer.py --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2025-12-09 17:36:45 -08:00
Daniel Han	66649d18bd	Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" ... This reverts commit cad158a56c.	2025-12-01 07:24:58 -08:00
pre-commit-ci[bot]	cad158a56c	[pre-commit.ci] auto fixes from pre-commit.com hooks ... for more information, see https://pre-commit.ci	2025-12-01 15:24:34 +00:00
Daniel Han	487a951914	Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" ... This reverts commit 964c9fef95.	2025-12-01 07:24:21 -08:00
pre-commit-ci[bot]	964c9fef95	[pre-commit.ci] auto fixes from pre-commit.com hooks ... for more information, see https://pre-commit.ci	2025-12-01 15:23:44 +00:00
Daniel Han	5f27bc4db5	Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" ... This reverts commit d34e0454ac.	2025-12-01 07:23:31 -08:00
pre-commit-ci[bot]	d34e0454ac	[pre-commit.ci] auto fixes from pre-commit.com hooks ... for more information, see https://pre-commit.ci	2025-12-01 15:20:22 +00:00
Daniel Han	ba2897a318	Revert "[FIX] Vllm guided decoding params (#3662)" ... This reverts commit fb4f0fdf56.	2025-12-01 05:43:45 -08:00
Datta Nimmaturi	fb4f0fdf56	[FIX] Vllm guided decoding params (#3662) ... * vllm sampling params fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * do not patch base_trainer * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * seperate vllm fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Apply suggestion from @danielhanchen * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" This reverts commit 58b483dc0d1790f99580665801d3fa0d7267c533. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" This reverts commit b2497519659a9f301e7a633795d9efdafdc2b277. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" This reverts commit de3daaf429f81aceb6632932b0cb1af5149652a8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2025-12-01 05:42:37 -08:00
Daniel Han	d6bb89ad44	Formatting & bug fixes (#3563) ... * Update rl.py * Fix CE Loss * Versioning * Update loader.py * Update loader.py * extract_model_type_from_config * Model types * Update loader.py * get_transformers_model_type * Update loader.py * Update loader.py * Update loader.py * Update rl.py * Update pyproject.toml * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Versioning * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update vision.py * Update vision.py * Fix DataParallel * Update _utils.py * Update rl.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update mapper.py * Versioning * Update loader.py * Update loader.py * Update rl.py * Versioning * Update _utils.py * Fix auto_mapping * Update loader.py * Update loader.py * Update vision.py * Update vision.py * Update loader.py * Message * Update vision.py * Update loader.py * Update vision.py * cache_implementation * Update vision.py * Update loader.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Save max_seq_length * Update _utils.py * Update rl.py * Update vision.py * Update llama.py * Mistral3 vllm (#3349) * [WIP] use vLLM for vision language models * Update README.md Editing icon sizes * Update README.md Updating icon sizes * Update README.md (#2885) * MoE kernels AGPLv3 * versioning * Many bug fixes (#2908) * add deepseek v3 * add deepseek r1 base * add deepseek r1 zero * add deepseek distill llama * add deepseek distill models * remove redundant code when constructing model names * add mistral small to registry * rename model registration methods * rename deepseek registration methods * refactor naming for mistral and phi * add global register models * refactor model registration tests for new registry apis * add model search method * remove deprecated registration api * add quant type test * add registry readme * make llama registration more specific * clear registry when executing individual model registration file * more registry readme updates * Update _auto_install.py * Llama4 * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Synthetic data * Update mapper.py * Xet and Synthetic * Update synthetic.py * Update loader.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update pyproject.toml * Delete .gitignore * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update _utils.py * Update pyproject.toml * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update chat_templates.py * Seasame force float16 / float32 * Fix Seasame * Update loader.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * is_multimodal * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update vision.py * Update vision.py * Update vision.py * UNSLOTH_DISABLE_STATIC_GENERATION * Update vision.py * Auto vision detection * Sesame * Whisper * Update loader.py * Update loader.py * Update loader.py * Update mapper.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update _utils.py * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * logging * Update pyproject.toml * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * logits / temperature * Update rl_replacements.py * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Debugging only * Update llama.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Generic efficient GRPO * Update rl_replacements.py * Update rl_replacements.py * Remove debugging * Update rl_replacements.py * Update rl_replacements.py * Update vision.py * Update llama.py * Update rl_replacements.py * versioning * Update _utils.py * Update vision.py * Update mapper.py * Update loader.py * Update mapper.py * Update vision.py * Update loader.py * Update vision.py * Update loader.py * Update _utils.py * Update vision.py * gradient checkpointing * Gemma 3N fixes * Update loader.py * Versioning * Gemma 3N fixes * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Fix setup.py * setup.py * Prints * Update setup.py * Update setup.py * Update setup.py * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update vision.py * Update vision.py * Update pyproject.toml * Update vision.py * Update _utils.py * Update __init__.py * Update __init__.py --------- Co-authored-by: jeromeku <jerome.ku@gmail.com> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> * silienty skip falcon h1 import is transformers_version < 4.53.0 (#2912) * Dynamically adjust get_per_token_logps function and patch as well (#2911) * add intel gpu with vllm support (#2903) * [bugs] fix for casual mask (#2868) * fix for casual mask * use un_casual in sdpa * add missing mask * fix for type * Explicitly check if xformers exists for attention (#2889) * Update __init__.py * Update llama.py * if mlp doesn't exist in layer module check for feed_forward name for falcon h1 (#2913) * Move inputs to right devices. (#2919) * Move tensors to right devices * fix multi gpu for non mistral models * multi GPU RoPE for gemma2 * Finish up multi GPU inference * Make multiGPU rope a list * Remove unnecessary transfer to CPU * Remove unnecessary move to CPU * Donot move inputs to device yet will be handled separately in another PR * Move inputs to appropriate decoder device * Make device count global variable * Cleanup RoPE device code * Fixup num_gpu to device count * Cleanup device counts * Use device index for RoPE get_cache * Donot typecast * Use tuple instead of list for tensors. Use device index directly * fixup move to device logic * WIP VLM vLLM * Make vLLM patch a function * Add save and load lora functions * Make fast_inference setup depend on the flag * Improve fast inference patching mechanism * Make vision setting depend on checks in fastbasemodel * Check LoRA and vLLM intercompatibility for vision models * Comment pointing to vLLM LoRA check * Improve lora validation on vLLM * Error out on no vLLM and increase max lora rank * Bug fixes (#3017) * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update pyproject.toml * Delete .gitignore * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update _utils.py * Update pyproject.toml * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update chat_templates.py * Seasame force float16 / float32 * Fix Seasame * Update loader.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * is_multimodal * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update vision.py * Update vision.py * Update vision.py * UNSLOTH_DISABLE_STATIC_GENERATION * Update vision.py * Auto vision detection * Sesame * Whisper * Update loader.py * Update loader.py * Update loader.py * Update mapper.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update _utils.py * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * logging * Update pyproject.toml * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * logits / temperature * Update rl_replacements.py * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Debugging only * Update llama.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Generic efficient GRPO * Update rl_replacements.py * Update rl_replacements.py * Remove debugging * Update rl_replacements.py * Update rl_replacements.py * Update vision.py * Update llama.py * Update rl_replacements.py * versioning * Update _utils.py * Update vision.py * Update mapper.py * Update loader.py * Update mapper.py * Update vision.py * Update loader.py * Update vision.py * Update loader.py * Update _utils.py * Update vision.py * gradient checkpointing * Gemma 3N fixes * Update loader.py * Versioning * Gemma 3N fixes * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Fix setup.py * setup.py * Prints * Update setup.py * Update setup.py * Update setup.py * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update vision.py * Update vision.py * Update pyproject.toml * Update vision.py * Update _utils.py * Update __init__.py * Update __init__.py * Small fixes * Update vision.py * Update vision.py * versioning * Update __init__.py * Update llama.py * Update rl.py * Update rl.py * Update _utils.py * Update vision.py * Update vision.py * compiler stance * Update _utils.py * Update pyproject.toml * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Revert "Revert "Add Qwen2.5-VL-32B-Instruct mapping to fix quantized model me…" (#2990) This reverts commit 4021da634a. * skip_guard_eval_unsafe fix * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update llama.py * Update llama.py * Fix `quantization_method` * versioning * fix for casual mask (#3011) * [intel] add for intel path for llama.py (#3012) * fix for intel path * remove unuse code * Update unsloth/models/llama.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Update llama.py * Fix Gemma 2 (#3024) * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update pyproject.toml * Delete .gitignore * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update _utils.py * Update pyproject.toml * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update chat_templates.py * Seasame force float16 / float32 * Fix Seasame * Update loader.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * is_multimodal * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update vision.py * Update vision.py * Update vision.py * UNSLOTH_DISABLE_STATIC_GENERATION * Update vision.py * Auto vision detection * Sesame * Whisper * Update loader.py * Update loader.py * Update loader.py * Update mapper.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update _utils.py * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * logging * Update pyproject.toml * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * logits / temperature * Update rl_replacements.py * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Debugging only * Update llama.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Generic efficient GRPO * Update rl_replacements.py * Update rl_replacements.py * Remove debugging * Update rl_replacements.py * Update rl_replacements.py * Update vision.py * Update llama.py * Update rl_replacements.py * versioning * Update _utils.py * Update vision.py * Update mapper.py * Update loader.py * Update mapper.py * Update vision.py * Update loader.py * Update vision.py * Update loader.py * Update _utils.py * Update vision.py * gradient checkpointing * Gemma 3N fixes * Update loader.py * Versioning * Gemma 3N fixes * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Fix setup.py * setup.py * Prints * Update setup.py * Update setup.py * Update setup.py * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update vision.py * Update vision.py * Update pyproject.toml * Update vision.py * Update _utils.py * Update __init__.py * Update __init__.py * Small fixes * Update vision.py * Update vision.py * versioning * Update __init__.py * Update llama.py * Update rl.py * Update rl.py * Update _utils.py * Update vision.py * Update vision.py * compiler stance * Update _utils.py * Update pyproject.toml * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Revert "Revert "Add Qwen2.5-VL-32B-Instruct mapping to fix quantized model me…" (#2990) This reverts commit 4021da634a. * skip_guard_eval_unsafe fix * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update llama.py * Update llama.py * Fix `quantization_method` * versioning * Update _utils.py * Update _utils.py * Update _utils.py * falcon force float32 on sm<75 machines (#3026) * Fix torch compile issues (#3028) * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update pyproject.toml * Delete .gitignore * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update _utils.py * Update pyproject.toml * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update chat_templates.py * Seasame force float16 / float32 * Fix Seasame * Update loader.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * is_multimodal * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update vision.py * Update vision.py * Update vision.py * UNSLOTH_DISABLE_STATIC_GENERATION * Update vision.py * Auto vision detection * Sesame * Whisper * Update loader.py * Update loader.py * Update loader.py * Update mapper.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update _utils.py * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * logging * Update pyproject.toml * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * logits / temperature * Update rl_replacements.py * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Debugging only * Update llama.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Generic efficient GRPO * Update rl_replacements.py * Update rl_replacements.py * Remove debugging * Update rl_replacements.py * Update rl_replacements.py * Update vision.py * Update llama.py * Update rl_replacements.py * versioning * Update _utils.py * Update vision.py * Update mapper.py * Update loader.py * Update mapper.py * Update vision.py * Update loader.py * Update vision.py * Update loader.py * Update _utils.py * Update vision.py * gradient checkpointing * Gemma 3N fixes * Update loader.py * Versioning * Gemma 3N fixes * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Fix setup.py * setup.py * Prints * Update setup.py * Update setup.py * Update setup.py * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update vision.py * Update vision.py * Update pyproject.toml * Update vision.py * Update _utils.py * Update __init__.py * Update __init__.py * Small fixes * Update vision.py * Update vision.py * versioning * Update __init__.py * Update llama.py * Update rl.py * Update rl.py * Update _utils.py * Update vision.py * Update vision.py * compiler stance * Update _utils.py * Update pyproject.toml * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Revert "Revert "Add Qwen2.5-VL-32B-Instruct mapping to fix quantized model me…" (#2990) This reverts commit 4021da634a. * skip_guard_eval_unsafe fix * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update llama.py * Update llama.py * Fix `quantization_method` * versioning * Update _utils.py * Update _utils.py * Update _utils.py * check stride * Cleanup * Update rope_embedding.py * Update gemma2.py * Fix `set_stance` * Update pyproject.toml * Update _utils.py * Fixup patch vllm * Disable mllama * Use variables to decide VLM support * Better attn_impl handling * Patch TF protobuf incompatability * Torch 2.8 (#3186) * Fix mamba * Update loader.py * Update vision.py * Update loader.py * Filter vLLM standby logs (#3131) * filter vLLM standby logs * safeguard standby logger patch * Update unsloth/models/_utils.py * Update unsloth/models/_utils.py * Update unsloth/models/_utils.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Update loader.py * Add scaler * Update llama.py * Update _utils.py * Versioning * GPT OSS fix * GPT OSS fix * Update loader.py * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Update vision.py * Update llama.py * Update llama.py * Update llama.py * Versioning * Update mapper.py * Update vision.py * Update vision.py * Update vision.py * Upcast norms * Update loader.py * Update vision.py * Upcast layernorms * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update save.py * Update rl.py * Update pyproject.toml * Update rl.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl.py * Update _utils.py * Update __init__.py * Torch 2.8 * Update rl_replacements.py --------- Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com> * Update _auto_install.py * Update pyproject.toml * Update rl.py * Protobuf issue * Update pyproject.toml * Fix extras transformers typo in pyproject.toml * Update _utils.py * Bug fixes (#3195) * Fix mamba * Update loader.py * Update vision.py * Update loader.py * Filter vLLM standby logs (#3131) * filter vLLM standby logs * safeguard standby logger patch * Update unsloth/models/_utils.py * Update unsloth/models/_utils.py * Update unsloth/models/_utils.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Update loader.py * Add scaler * Update llama.py * Update _utils.py * Versioning * GPT OSS fix * GPT OSS fix * Update loader.py * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Update vision.py * Update llama.py * Update llama.py * Update llama.py * Versioning * Update mapper.py * Update vision.py * Update vision.py * Update vision.py * Upcast norms * Update loader.py * Update vision.py * Upcast layernorms * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update save.py * Update rl.py * Update pyproject.toml * Update rl.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl.py * Update _utils.py * Update __init__.py * Torch 2.8 * Update rl_replacements.py * Update loader.py * UNSLOTH_ENABLE_CCE * Fix * Update loader.py * Update loader.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Import fixes * Update loader.py * Fix aimv2 issue * Update loader.py * Update import_fixes.py * Update import_fixes.py * Update loader.py * Update loader.py * Update loader.py * Upgrade * Update loader.py * Update loader.py * Update loader.py * Update loader.py --------- Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com> * adallow float32 dtype in FastLanguageModel (#3204) * Update loader.py * Update vision.py * Suppress message and use unsloth sampling params * Use trl sampling params for now * Improve error message * fixup quantized fast inference model name * Add mistral 3 support --------- Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: jeromeku <jerome.ku@gmail.com> Co-authored-by: DoubleMathew <mmathew23@gmail.com> Co-authored-by: Lei Zhenyuan <zhenyuan.lei@intel.com> Co-authored-by: parth2510 <parthguptapg7326@gmail.com> * Set padding to 0 * Fix patch * fixup patch (#3359) Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com> * Update vision.py * Versioning * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * MXFP4 dequant * Update loader.py * Update vision.py * load_in_16bit * Update vision.py * Update vision.py * Update vision.py * Update rl.py * Update vision.py * offload_embedding * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update rl_replacements.py * Update loader.py * Fix padding issue * Update pyproject.toml * Update _utils.py * Update pyproject.toml * Update _utils.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * New models * Update llama.py * Versioning * Update _utils.py * Update llama.py * Update _utils.py * Update llama.py * Fix AMD * Update _utils.py * Update llama.py * Update vision.py * DEVICE_TYPE_TORCH * Update __init__.py * Update __init__.py * Update _utils.py * Move DEVICE_TYPE * Update rl_replacements.py * Update loader.py * AMD install script * Move AMD * Update _amd_install.sh * Update pyproject.toml * Update pyproject.toml * Delete _amd_install.sh * Update device_type.py * Update loader.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update tokenizer_utils.py * Versioning * Update pyproject.toml * Update loader.py * Update _utils.py * Update pyproject.toml * Update pyproject.toml * Update _utils.py * Update pyproject.toml * Update _utils.py * Update _utils.py * Update loader.py * Update _utils.py * Update _utils.py * local_files_only * Cut Cross Entropy * Update llama.py * Update vision.py * Update vision.py * Update vision.py * Qwen 3 VL vLLM (#3489) * Update __init__.py * patch_torchao * torchao_logger * Update rl_replacements.py * Fix * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update _utils.py * Versioning * fbgemm fp8 block quant support (>=1.4.0) (#3531) * fbgemm fp8 block quant support (>=1.4.0) * Verify for fp8 support before proceeding * Use unsloth zoo's Version and improve comments * spacessss * Update vision.py * Update vision.py * Update rl.py * vllm_sampling_params * Update rl.py * Update rl.py * Update rl.py * Add `ruff` pre-commit hook and apply it (#3424) * Add Ruff pre-commit config and workflow * Add kwarg spacing enforcement helper * Apply Ruff formatting * Update fp8.py * Revert ruff on some files * Update * force-exclude = true * Datasets issue * Ruff * Remove mapper * Update mapper.py * Update pyproject.toml --------- Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> Co-authored-by: jeromeku <jerome.ku@gmail.com> Co-authored-by: DoubleMathew <mmathew23@gmail.com> Co-authored-by: Lei Zhenyuan <zhenyuan.lei@intel.com> Co-authored-by: parth2510 <parthguptapg7326@gmail.com> Co-authored-by: Dan Saunders <danjsaund@gmail.com>	2025-11-07 06:00:22 -08:00
andrewor14	3ffb3bdcfe	Fix QAT + LoRA fast path, add tests (#3307) ... **Summary:** The existing QAT + LoRA path only applied fake quantization to the original slow path, but the default is the fast path that calls unsloth's fast LoRA primitives. This commit integrates fake quantization into these fast primitives as well, and add unit tests to assert that fake quantization is actually taking place. **Test Plan:** Unit tests: ``` pytest tests/utils/test_qat.py ``` End-to-end test: https://gist.github.com/andrewor14/6360dd69b5784c71c46e80c14f53e6b6 Full fine-tuning Llama3.1-8B with and without QAT + LoRA on yahma/alpaca-cleaned for 1 epoch: - Batch size = 8 (no grad accum) - Learning rate = 2e-4 - Quantization scheme = int4 weight only (with bf16 activations) Wikitext perplexity: - Baseline = int4 quantized model finetuned without QAT - QAT int4 quantized model (with this PR) achieved 33% lower perplexity than the int4 baseline - QAT int4 quantized model without this PR was worse than the int4 baseline ``` ==> unsloth_model_lora_baseline_output/lm_eval_float.log <== | | |none | 0|word_perplexity|↓ |7.5551|± | N/A| ==> unsloth_model_lora_baseline_output/lm_eval_quantized.log <== | | |none | 0|word_perplexity|↓ |8.7655|± | N/A| ==> unsloth_model_lora_qat_int4_output/lm_eval_quantized.log <== | | |none | 0|word_perplexity|↓ |8.3548|± | N/A| ```	2025-09-17 15:18:17 -07:00
leopardracer	c6e0366e0d	Fix Typos in Documentation and Comments (#2721) ... * Update ocr_eval.md * Update backward.py	2025-06-17 04:34:51 -07:00