50 commits

Author	SHA1	Message	Date
Daniel Han	48ea5560ee	Use Kaggle's large overlay for saves, and refuse a GGUF export that cannot fit (#8439) ... * Refuse a GGUF export that cannot fit, and use Kaggle's large /tmp overlay Two disk failures seen in real notebook runs, fixed in front of the writes rather than after them. Kaggle exhausts /kaggle/working, which is about 20GB, while the overlay mounted at /tmp on the same kernel has over a terabyte free. A merge or GGUF export pointed at a relative path under the working directory that cannot fit there now goes to /tmp instead, with one printed line saying where the files are and that /tmp is scratch space Kaggle does not save as kernel output. An absolute path is never moved, because silently relocating a directory the caller named would be worse than the disk error it avoids, and a push_to_hub save_directory is a repo id rather than a path so it is left alone entirely. Gemma4 26B A4B Vision, Gemma4 31B Vision and Qwen3 32B each trained, ran inference and wrote a complete merged_16bit, then died partway through a GGUF shard. The size in front of them counted the model twice; the real peak is the pre-warmed base in the Hugging Face cache, the merge, the intermediate GGUF and the quants, all on one filesystem at once. _preflight_gguf_disk sizes all four before the merge starts. When the export fits but a cached base as well does not, it drops the pre-warm rather than refusing, because that is an optimization for the next export and this one still runs. When nothing fits it raises with the two numbers and what to do about them. Neither guard blocks on a guess: an unmeasurable model or an unmeasurable filesystem proceeds exactly as before, as does UNSLOTH_DISK_PREFLIGHT=0. Kaggle detection moves to unsloth_zoo.disk_utils, with a fallback for an older installed unsloth_zoo that keeps the environment question answered correctly and turns the new guard into a no-op. 30 new tests, each watched to fail against a perturbed fix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the Kaggle merge preflight measurable, and size what the export really writes The merge preflight passed keep_intermediate_gguf to unsloth_zoo.disk_utils.estimate_gguf_export_bytes, which has no such parameter, so every call raised TypeError into the surrounding except and no save_pretrained_merged was ever redirected on Kaggle. A merge writes no GGUF at all, so it is now sized with model_16bit_bytes rather than the GGUF export estimate, which always prices an intermediate conversion. Also, on the same path: - every compressed export (fp8, nvfp4, mxfp8, w4a16, ...) keeps the 16-bit merge and writes a quantized sibling, so all of them are measured, and the sibling is counted, not just mxfp4's merge; - save_method spellings are normalized the way unsloth_save_model normalizes them, so "merged 16bit" is measured like merged_16bit; - the non-PEFT fallback save_pretrained writes a full checkpoint, so the GGUF estimate counts it instead of assuming a non-PEFT model has one on disk; - imatrix_file=False disables the imatrix in _resolve_imatrix_file, so the preflight no longer sizes a two-pass conversion for it; - Kaggle and Colab never pre-warm the hub cache, so the redirect decision stops pricing a cache copy that cannot exist there and sending an export that fits in /kaggle/working to /tmp, which is not kept as output. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Size the Kaggle merge redirect the way the merge itself measures it The redirect asked for the raw tensor bytes, but merge_and_overwrite_lora compares the save against int(free * 0.95), so a merge that only just fit was left in /kaggle/working and then refused outright instead of moving to /tmp. Ask for the same effective figure. Also size a quantized sibling by what a weight-only scheme actually shrinks: compressed-tensors and torchao quantize Linear weights only, so the embeddings and an untied lm_head stay 16-bit. The torchao portable exports now take the redirect too; their 16-bit merge is staged in a temp directory, so only the sibling is priced at save_directory. And the non-PEFT GGUF fallback save_pretrained()s the model at its own dtype, which for a float32 load is twice what the estimate budgeted. * Do not redirect a torchao export onto a /tmp that its staging merge also fills The portable torchao exports now reach the Kaggle redirect, but the sibling is the only artifact the redirect sizes. _unsloth_save_torchao merges into tempfile.mkdtemp() and keeps that 16-bit staging checkpoint until quantization finishes, and on a Kaggle kernel tempfile resolves to the same /tmp the redirect moves the sibling to, so both are on the destination at once. A working directory with room for the sibling but not the 5% the merge guard reserves therefore had its export moved to a /tmp that then ran out, when leaving it in place would have worked. The staging bytes are checked against the destination rather than added to need_bytes: nothing stages in the working directory, so charging it there would relocate exports that fit into /tmp, which is not kept as notebook output. The check can only cancel a redirect, never cause one. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stat the real redirect destination in the torchao staging test The staging guard's own tests stub _same_filesystem, so nothing exercised the os.stat pair it is built on, and nothing recorded why stat-ing the destination is safe: kaggle_tmp_redirect returns a message only after os.makedirs has succeeded, and the fallback in unsloth/disk_utils.py never returns one, so the guard is unreachable with a destination that does not exist. Two tests. The first cancels a redirect with _same_filesystem left unstubbed and a real directory created under the tempfile default. Cancelling is the one outcome the helper cannot reach by accident, since every failure inside it, os.stat included, returns True and takes the redirect, so a real destination that is really rejected proves both stats resolved. The second pins the invariant the first depends on, against whichever kaggle_tmp_redirect is installed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Preflight a full model saved as LoRA, and measure the GGUF sibling's disk Two gaps in the Kaggle disk guard, both of which let an export pass a check that measured the wrong thing. A model with no adapter saved with save_method="lora" is not a LoRA save: unsloth_generic_save and unsloth_save_model both fall back to save_pretrained and write the entire model, so a full fine-tune asked for "lora" fills /kaggle/working exactly like a merge. The preflight skipped it on the method name alone. It is now sized whenever the model is not a PeftModel, from the parameters' own storage rather than as a 16-bit merge, because that fallback casts nothing: an fp32 model writes four bytes per parameter and a 4-bit one its packed storage. A real PeftModel still writes adapters and is still skipped, and a model that cannot be measured is left exactly as before. The GGUF files land in save_directory + "_gguf", which is a sibling, so they sit on the parent's filesystem. That is the same disk as save_directory unless that path is itself a mount point or a symlink onto another one, and then the aggregate estimate was checked against a filesystem the quants never touch. The sibling is now probed separately and compared against the export minus the checkpoint, which is what actually goes there. Only a strictly tighter sibling can change the outcome, so a single filesystem behaves exactly as before, an unmeasurable path leaves the decision alone, and an estimator that cannot answer the new question leaves the main guard standing. The export and the preflight now share one definition of that directory so the two cannot drift. * Charge the compressed sibling for every module the recipe refuses to quantize The compressed recipe ignores more than the embeddings: `lm_head`, `re:.*\.linear_attn\..*`, `re:.*\.visual\..*`, `re:.*mtp.*`, and on an MoE `re:.*\.gate$` and `re:.*\.shared_expert_gate$`. The sizing helper walked only `get_input_embeddings` and `get_output_embeddings`, so a VLM's vision tower, a Qwen3-Next hybrid's linear attention, an MTP head and the MoE routers were all priced at 4 or 8 bits for bytes the export copies across at 16. That under-counts `need`, and an under-counted `need` is how a merge that should have gone to /tmp stays in /kaggle/working. The four lines that build `ignore` are now a module-level `compressed_ignore_patterns(config)` in `_compressed_quantize.py`, which `main()` calls and which `save.py` reads, so the sizing cannot drift from the recipe again. That module's imports stay stdlib only. `_unquantized_parameter_bytes` takes the pattern list and adds the 16-bit bytes of every module the recipe ignores, matching compressed-tensors' `match_name`: a `re:` prefix is `re.match` against the fully qualified module name, a plain entry is an exact name or a parent class name. Ids are deduplicated against the embeddings, so an `lm_head` that is also `get_output_embeddings()` is counted once and a nested match under an already counted tower is not counted twice. The torchao branch passes no patterns. `_unsloth_save_torchao` quantizes with a bare weight-only config and has no ignore list, so charging it these would over-count, and over-counting relocates an export that fits into a /tmp that Kaggle does not keep as notebook output. Every new path degrades to the old estimate rather than raising: a missing or renamed symbol, a model whose modules cannot be walked, a module that will not answer `parameters()`, or an unparseable pattern. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Charge each filesystem only for what the export writes to it The GGUF preflight measured the sibling directory's filesystem but still compared the aggregate estimate against the checkpoint's, so a save directory that is a mount point or a symlink onto another mount had the intermediate conversion and every quant charged to a disk they never touch. A checkpoint that fits, with a sibling filesystem that has ample room for the quants, was refused. Both halves now hang off one predicate computed once, so they cannot disagree about whether the storage is split. When it is split the checkpoint portion, need minus need_sibling, is charged to the save directory's free space and need_sibling to the sibling's; when it is not, nothing changes. The predicate is the device id of the filesystem each probe measures, rather than the sibling reporting less free space, because two disk_usage calls on one filesystem can disagree when something else writes between them and reading that as two filesystems would charge a single-filesystem export the larger of its two halves instead of their sum. An unreadable or zero device id means not split. The sibling refusal no longer requires the sibling to be the tighter of the two: once the checkpoint is charged only its own portion, a sibling that is roomier than the save directory and still short of need_sibling has to be caught there, because the aggregate comparison that used to catch it is gone. The new branch can only turn a refusal into a pass or change which message is raised, and it runs after the redirect has been decided, so it still cannot cause one. An estimator that cannot size the sibling leaves need_sibling at zero, which makes the checkpoint portion the whole estimate again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Charge the GGUF export for what is on disk at once, and where Three corrections to the disk preflight, all of them cases where the figure it compares against free space is not the figure the export produces. A disposable merge is not charged for all three artefacts at once. `_free_merge_if_disk_is_tight` deletes this export's own merge once the intermediate GGUF exists and before the first quantize pass, so the peak is the larger of "merge plus intermediate" and "intermediate plus quants", not their sum. Nemotron-3-Nano-30B-A3B on a 132GB disk peaks at 128GB and runs; the aggregate 147GB refused it. `_preflight_gguf_disk` takes the same `merge_is_disposable` flag the reclamation acts on, and relaxes only when a quantize pass follows, only on one filesystem (the reclamation declines across two), only when the output directory holds no weights this export did not write, and only when the peak is lower than the aggregate. The working directory the initial conversion writes to is measured. `convert_to_gguf` passes a bare `--outfile`, which llama.cpp resolves against the process CWD, and the finished file is moved into the `_gguf` directory afterwards. So a Kaggle export redirected to /tmp still writes its largest staging artefact into the 20GB working directory, on a filesystem nothing here looked at. When that directory is a different filesystem from where the file ends up, it now has to hold the intermediate, and the export is refused with the directory named if it cannot. Ignored modules are sized from their logical shapes. `model_16bit_bytes` sizes the merge through unsloth_zoo's `logical_numel`, which reads `quant_state.shape`; `_unquantized_parameter_bytes` was subtracting a figure built from `numel()`, which on a 4-bit `Params4bit` is the packed uint8 count and roughly half. A Qwen3-Next linear-attention subtree was priced at 8 bits for bytes the export writes at 16, and an under-counted need is a Kaggle redirect that never happens. It now calls `logical_numel` itself, through `named_parameters` so the name reaches it, which is the only way MXFP4 packing is identifiable. Every new path degrades to the previous behaviour rather than raising: an unreadable working directory, an unmeasurable free figure, an estimator that cannot size a phase, a directory that cannot be listed, a module that cannot name its parameters. * Keep the merge guard's reserve when the estimate is split `_preflight_gguf_disk` charges the checkpoint's own portion to the filesystem holding `save_directory` once the GGUF sibling is on another one. That portion is two bytes per parameter exactly, and `merge_and_overwrite_lora` refuses to write a merge unless `free * 0.95` covers it, so 16GB of checkpoint on 16GB of disk passed the preflight and died in the merge seconds later. The aggregate branch never needed the reserve because it charges the quants as well. Clamped at `need`, so the split still cannot refuse an export the aggregate allowed, and skipped where no merge is written or where the sibling could not be sized and the figure is a fallback rather than a checkpoint. Also size a full-model `"lora"` save from the caller's `state_dict` when there is one. `save_pretrained` writes that dict, only `"16bit" in save_method` casts it, and both `_preflight_merge_disk` call sites accept one and document `"lora"`, so an fp32 dict over fp16 parameters was priced at half. * Say which disk ran out when the staging merge is on its own filesystem `_destination_holds_torchao_staging` asks whether the redirect DESTINATION can hold the staging merge as well, which is the whole question on Kaggle, where `tempfile` and the destination are both /tmp. When they are separate mounts it returns True and nothing has measured the staging filesystem at all, so a 4GB tmpfs is handed a 60GB merge and `_unsloth_save_torchao` dies inside `tempfile.mkdtemp` without naming TMPDIR. A warning rather than a refusal or a cancelled redirect: the preflight never raises, and the staging merge is written to TMPDIR whether or not the export was relocated, so declining the move leaves the identical failure and puts the output on the smaller disk too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Compare filesystems by the ancestor a fresh destination writes to `_same_filesystem` stat-ed both paths directly, and `_preflight_merge_disk` runs before anything has been created, so on a first torchao export the destination was a name and not a directory. The stat raised FileNotFoundError, the helper's broad handler swallowed the whole probe, and an undersized TMPDIR went unmentioned in exactly the case the warning was added for. It now resolves the nearest existing ancestor, which is the filesystem the write really lands on and the same resolution `free_bytes` and `_filesystem_id` already use. A path neither side can identify still raises, because both callers read that as "cannot tell" and cancel the probe rather than guess. * Reserve in the TMPDIR warning what the staging merge really needs The warning compared the free space on a separate TMPDIR against the raw size of the staging merge, but that merge is written by `merge_and_overwrite_lora`, which refuses to write anything `int(free * 0.95)` does not cover. Between the merge's size and that size over 0.95 the export therefore died with "Failed saving - no disk space left" while the diagnostic that exists to name the disk stayed silent, and on a separate TMPDIR the merge's own temp-folder fallback lands on the same filesystem, so nothing recovers it. Compared against the reserved figure now, the same one `_preflight_merge_disk` asks the redirect for, and the printed size is that figure so the number and the threshold agree. * Charge the merge guard's reserve only where that guard runs The split branch added `merge_and_overwrite_lora`'s 5% reserve to the checkpoint whenever `needs_merge` was set, but `needs_merge` is also true for a non-PEFT model with no reusable local `_name_or_path`: the GGUF path has to write a checkpoint there too, and it writes it with a bare `self.save_pretrained`, which consults no guard and reserves nothing. A filesystem holding exactly that checkpoint was refused for headroom the writer never asks for. Gated on the model really being a PEFT one, which is the only case that reaches the merge. The tests that cover the reserve now use a model the preflight recognises as PEFT, so they exercise the path the reserve belongs to. * Follow the reused checkpoint when an unwritable CWD moves the conversion `convert_to_gguf` passes a bare `--outfile`, and when the process CWD cannot be written to llama.cpp's output is redirected into the converter's input folder. That input folder is not always `save_directory`: a non-PEFT model with a local `_name_or_path` is converted from its own checkpoint, which `unsloth_save_pretrained_gguf` swaps in before calling `save_to_gguf`. The preflight probed the requested output instead, so with the two on different filesystems it measured a disk nothing was written to while the intermediate GGUF filled the checkpoint's. The conversion probe now resolves the same input folder the export will use, by the same condition `_gguf_writes_16bit_checkpoint` already reads. * Charge a split export for the conversion that lands beside its checkpoint The split branch charges the filesystem holding `save_directory` for the checkpoint alone, and the working-directory check charged whatever filesystem the intermediate conversion is written to for the conversion alone. When those are the same disk - `save_directory` a mount whose `_gguf` sibling is elsewhere, with the process working directory on the mount - both artefacts sit on it at once and neither check ever added them up. A 60GB checkpoint and a 60GB conversion both passed on 100GB, and then it filled. The conversion is now added to the split branch's figure when it lands on that same filesystem. `_shares_filesystem` answers that, and unlike `_on_separate_filesystems` it says no to a path it cannot identify, because here the answer adds a charge rather than removing one. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reserve for the merge alone, and follow the dict the writer follows Four corrections to where the merge guard's 5% reserve is charged, and to how a caller's state dict is read. `_preflight_merge_disk` reserved around the whole estimate. Only the 16-bit merge is written by `merge_and_overwrite_lora`, so only that part is measured by its `free * 0.95`. A full-model `"lora"` save goes straight through `save_pretrained`, and the torchao path merges into a temp directory and leaves only the quantized sibling here, so neither pays a reserve at all. On a 20GB Kaggle working directory the over-charge relocated every output between 19.0GB and 20.0GB into a /tmp the kernel does not keep. The GGUF split path compared the checkpoint plus its cache copy against the reserved checkpoint with `max`, when the cache is written first and is still resident when the guard runs. 16GB of checkpoint with a 14GB cache on 30.5GB passed at 30GB and then the merge saw 16.5GB and refused 16GB. Added, this band drops the optional pre-warm rather than failing the export. `_full_model_checkpoint_bytes` selected the caller's dict on truthiness, while `unsloth_generic_save` forwards it on `is not None`. So an explicitly empty dict wrote no tensors and was priced as the whole resident model. And compressed and torchao build their output as `save_directory + "-<suffix>"`, which is lexical, so a symlinked `save_directory` puts the sibling on the filesystem the probe never asked about. Warned rather than redirected: the caller derives the sibling from whatever this returns, so a redirect already moves both, and the uncovered case is the one where no redirect fires. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow the dict a 16-bit save writes, and phase the merge reserve The generic writer only builds a state dict when it was handed none, so a caller-supplied one is what the 16-bit checkpoint costs; the other writer rebuilds it from the merged layers and the resident model is still the right figure there. A split export whose conversion lands beside its checkpoint was charged the merge reserve and the conversion together. The merge guard runs before the conversion is written, so the requirement is the taller of the two phases: 60GB of merge and 60GB of conversion fit in 122GB, and the sum asked 123.2GB and refused. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Ask each writer what it does with the dict, and the config what dtype it holds Four sizing inputs the preflights were guessing at. `unsloth_save_model` rebuilds the state dict from the merged layers only on the path that walks `.model.layers`. Anything else takes its generic fallback, which calls `save_pretrained(**save_pretrained_settings)` with the caller's dictionary still in it and with no cast, so the checkpoint is that dictionary's own bytes at its own dtypes. A 4GB fp32 dictionary was priced as a 2GB merge. compressed-tensors and torchao merge to 16 bits through `unsloth_generic_save` as well, forwarding the same dictionary, so their kept or staged merge is sized from it too and not only a literal `merged_16bit` request. `_preflight_gguf_disk` was left on its `model_dtype = "f16"` default while `save_to_gguf` reads the config. `estimate_gguf_export_bytes` drops a requested output that equals the initial conversion, so a bf16 model asked for ["f16", "q4_k_m"] was charged one 16-bit file where the export writes a bf16 intermediate and a separate f16 output: 37.2GB asked against 52.4GB written on Qwen3-8B. And the pre-warmed base model is charged to the filesystem that actually holds the Hugging Face cache. `save_directory` being a mount is the premise of the split branch, and then `~/.cache` and the lexical `_gguf` sibling are both on the parent disk: charging the checkpoint's filesystem dropped a pre-warm that had room and let the sibling accept the GGUF files alone on a disk the base was about to be downloaded onto. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop bf16 to f16 before sizing, and charge the cache where the conversion lands Two filesystems' worth of undercount, both in the GGUF preflight. `save_to_gguf` switches a bf16 initial conversion to f16 on hardware with no bf16, and it does so AFTER resolving one. The preflight only normalized the dtype it was told, so an explicit `first_conversion="bf16"` went through untouched, and so did the single direct-convert method `_choose_first_conversion` hands back. The estimate omits an output equal to the initial conversion, so `["bf16"]` on a T4 was priced as one 16-bit file where the export writes an f16 intermediate and a separate bf16 output: 30.5GB charged against 45.8GB written on Qwen3-8B, 15.3GB short. The other is a third filesystem. With the output on an external drive and the writable CWD sharing the machine's own disk with the Hugging Face cache, the conversion check charged the intermediate alone. The pre-warm downloads the base onto that same disk first and leaves it there, and its own gate does not catch it either: it asks for two base copies free, and an f32 conversion is two base copies on its own. 38.1GB free clears the pre-warm's 30.5GB threshold and the 30.5GB conversion check, then the conversion writes 30.5GB into the 22.8GB the cached base left. The cache is now resolved once, above the split, and charged on the conversion's filesystem too - dropping the pre-warm rather than refusing, since the raise for a conversion that does not fit at all still runs first. The pre-warm's skip message travels with the flag now, because more than one filesystem can clear it and each has to name the one it measured. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reserve the merge guard's 5% only where the guard runs `merge_and_overwrite_lora` and its `free * 0.95` have one caller here: `unsloth_generic_save`, on its adapter branch. The preflight was charging the reserve to every 16-bit merge that was not written verbatim, including `unsloth_generic_save_pretrained_merged`'s no-adapter save, which casts the supplied dict and writes it with a bare `save_pretrained`. The reserve is now driven by whether that writer runs, which is separate from the sizing: a compressed export really is cast to two bytes by the same writer and keeps that sizing whether or not there is an adapter to merge. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Charge the cache pre-warm to the filesystem it is written to The single-filesystem branch of the GGUF preflight added the pre-warmed base copy to need_with_cache and then compared that total against the free space on save_directory, whatever filesystem the Hugging Face cache is actually on. HF_HOME on a data volume is the ordinary layout on a machine with more than one disk, and there the cached base never lands on the export disk at all. Measured with a 60GB checkpoint, 40GB of quants and a 60GB cached base on 120GB free: the export needs 100GB on that disk and fits, but the fictitious cache copy takes the ask to 160GB and the pre-warm is dropped, so the next export downloads the whole base again. That is the re-download the pre-warm exists to stop. With the cache on its own filesystem the ask is now 100GB and the pre-warm survives; with the cache on the export disk it is still 160GB and the pre-warm is still dropped. The split branch already did this accounting for the checkpoint and the sibling, so the predicate moves above the split and both branches read one answer. The figure can only fall, and only the pre-warm decision reads it, so no export that fit before is refused now. An unresolvable cache stays charged where it was. * Size the Kaggle redirect from the phased peak, and say when a cancelled one leaves no room Two holes on the Kaggle side of the preflight. The GGUF redirect was asked for the aggregate while the refusal below it reads the phased peak of a disposable merge. On the Nemotron shape - a 63GB merge, a 60GB intermediate, an 18GB Q4_K_M - the ask was 141.0GB and the peak is 123.0GB, so a /kaggle/working with 132.0GB free measured too small and the export was relocated to /tmp, which the kernel does not keep as notebook output. The redirect now asks the same 123.0GB, under the same predicates as the branch that lowers the figure, read against the directory before any move. It can only ever lower the ask. A cancelled torchao redirect was silent. With a 10GB staging merge and a 5.0GB fp8 sibling, /kaggle/working at 4.0GB free and /tmp at 12.0GB, the redirect fires because the sibling does not fit here and is then cancelled because /tmp cannot hold the sibling and the staging merge together. That is still the right move, but it hands the export back a filesystem measured too small for it, and nothing downstream measures that: the merge is staged in TMPDIR, so the merge guard's free * 0.95 covers the staging disk only, and the sibling is written at the very end of a long quantization. It now says so, with both figures. A warning and not a refusal, like the two warnings beside it: this preflight never raises and the sibling is an estimate. An unmeasured move (UNSLOTH_KAGGLE_USE_TMP=1) stays silent. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-ask the Kaggle redirect at the phased peak when this directory cannot be reclaimed A save directory that already holds a checkpoint yields nothing for _free_merge_if_disk_is_tight to reclaim, so the redirect above it is asked for the aggregate. The move, though, writes the merge into a directory of the export's own, where the reclamation is available: on the Nemotron-3-Nano-30B-A3B shape (63GB merge, 60GB intermediate, 18GB Q4_K_M) that is a 141GB ask against a 130GB overlay, declined, and the export refused in place at 141GB on 100GB free, when relocating peaks at 123GB and fits. The first ask keeps the aggregate. Only after the move is declined, and only where this filesystem is already short of the figure the refusal will read, is it asked again at the peak. Asking the peak outright instead is a regression: 130GB free here holds the peak but not the aggregate, so the lower ask cancels a move the export needs and the refusal reads 141GB anyway. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-13 18:56:43 -07:00
Daniel Han	71656a0318	Free the intermediate 16-bit merge when the GGUF quants will not fit (#8500) ... * Free the intermediate 16-bit merge when the GGUF quants will not fit Nemotron-3-Nano-30B-A3B trained, ran inference, merged to 16-bit, and then died in llama-quantize at tensor 88 of 401: llama_model_quantize: failed to quantize: basic_ios::clear: iostream error That is llama.cpp's own fout.exceptions(std::ofstream::failbit), commented upstream as 'fail fast on write errors', firing on a write with nowhere to go. On the 132GB disk it ran on: a 63GB intermediate 16-bit merge, a 60GB BF16 GGUF and a Q4_K_M needing about 18GB is 141GB. The merge is the wasted copy. It is written, converted to GGUF, and never read again, because llama-quantize reads the GGUF. So when the quants do not fit, reclaim it: only the weight files, so config and tokenizer survive for later steps, and only when the free space is actually short, so anyone with room keeps the merge they asked for. The input filename is a red herring worth recording. unsloth passes initial_files[0], which for this model was BF16-00001-of-00002.gguf, the first shard of a split. llama.cpp's llama_model_quantize_impl walks ml.weights_map across every input shard and, with keep_split false, collapses them into one output, so handing it shard one is correct usage. Second, separable fix: the advice to git clone llama.cpp and rebuild was gated on IS_KAGGLE_ENVIRONMENT, so this Colab run was told to spend a long compile on a build that was fine. A disk-shaped failure now says so, with the free space measured, wherever it happens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Narrow the merge reclamation to intermediates it is safe to delete Five things it could get wrong, each with a test that fails without the fix. A non-PEFT save_pretrained_gguf does not write an intermediate at all: it points the converter at the local checkpoint the model was loaded from. On a tight disk the reclamation would have deleted the user's own model, so it now runs only for a merge the export wrote itself, off by default. gguf_directory can put the outputs on another filesystem, where deleting the merge frees bytes the quantize pass cannot reach: the export runs out of space anyway and the merge is gone too. Compare st_dev first. f32 off an f16 or bf16 base is twice the file it reads, so bounding every pass by the base size called a disk roomy that was about to fill. Size each pass by its own output type. _gguf_failure_looks_like_disk let the working directory answer for a roomy output directory, which would blame disk for a quantizer failure and bury the llama.cpp advice that fixes it. The directory that has to hold the file decides; the working directory is only the fallback. os.listdir sat outside the helper's never-raises contract. An unreadable directory the quantizer no longer needs must not be what fails the export. * Keep the SentenceTransformer export's own module directory The GGUF export it drives is pointed at 0_Transformer, which the wrapper wrote in step 1 and uploads as part of the folder in step 7. That is the caller's deliverable, not a throwaway on the way to the GGUF, so reclaiming it on a tight disk would hand back a directory that no longer loads as a SentenceTransformer. save_pretrained_gguf takes the flag now and defaults it to the behaviour it already had; the wrapper is the one caller that opts out. * Retitle the section header now that it covers more than four cases * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reclaim only the merge shards, and price each quant by its own width The reclamation deleted every top-level .safetensors, .bin, .pth and .pt in the merge directory. A merge routinely lands in a training output_dir, so that took training_args.bin, optimizer.pt and rng_state.pth with it, and an adapter sitting alongside. Match the names save_pretrained actually writes instead: the shard index when it wrote one, else transformers' own -NNNNN-of-NNNNN shape and the canonical unsharded spellings. The per-pass size estimate charged every quantized output a whole copy of the base GGUF, so a Q4_K_M off a 60GB base was budgeted 60GB rather than about 21GB and a disk with room to spare had its merge deleted anyway, against the stated promise that the merge survives whenever the export already fits. Price each pass from the width its name carries, rounding the base down to nominal and the output up by the block overhead k- and i-quants store, so the estimate stays an upper bound. That also fixes the base itself: q8_0 is a direct-convert outtype, so first_conversion is not always 16-bit. A VLM conversion also emits an -mmproj projector that llama-quantize copies rather than quantizes. It was charged to every requested quant; filter it the way the RAM budget a few lines up already does. * Reclaim only shard sets the merge itself could have written The shard alternative in the merge-weight matcher accepted any stem, so a `-NNNNN-of-NNNNN` set that belongs to the user, sitting in the directory the export was pointed at, was deleted along with the merge. transformers is narrower than that when it clears stale shards from a save directory: the name must start with the weights stem as well as carry the shard shape. Match the same pair here. A merge that does use another stem is still reclaimed, because a sharded save_pretrained always writes the index and the index names its own shards. * Reclaim only the serialization a disposable merge is written in * Read only the merge's own shard index, not an older one beside it * Tighten the comments around the merge reclamation * Validate the shard index before letting it widen the deletion Reading the index is the one way a merge under a name the convention misses still gets reclaimed, which makes it the one way a file the convention protects gets deleted. transformers writes an index only when a save shards, and its stale-shard sweep never removes one: reg.fullmatch on the shard shape does not match model.safetensors.index. So an unsharded merge lands in a directory that still holds an earlier save's index, and every name that index carries was handed to a permanent delete, stem rule or not. A user's users_own-00001-of-00002.safetensors went with it. An index that is live is self-consistent in a way a stale one has no reason to be, so require that: one stem, one of-000NN, exactly that many entries, all of them still on disk. A sharded merge under an unknown stem still reclaims; a mixed or partial listing falls back to the naming convention. The existing index test built its fixture as a single non-shard file, which is a shape save_pretrained never writes, since a save either shards and writes an index or does neither. Rebuilt as a whole shard set, which is what the case is actually about. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Record what the export owns instead of inferring it from the names Reclamation was deciding ownership from file naming, and the names a merge writes are exactly the names an earlier save wrote. Validating the index was not enough: it rejects a partial stale listing, but a previous completed sharded save leaves an index that is self-consistent, names one stem, and has every shard present. transformers removes neither, so after the merge that set is indistinguishable from one this run produced, and a tight disk deleted it. The same hole covered consolidated.safetensors, a name the shard selection drops whenever other shards coexist, so a reused output directory could lose a checkpoint this export never wrote. Both are answered by recording the directory contents before the merge writes, where the difference still exists, and reclaiming only what appeared after. An export that cannot read the directory beforehand reclaims nothing, since the deletion is permanent and a guess is not good enough to justify it. Also sizes the disk-full diagnosis against the output rather than a fixed 2GB floor. Room is a relation between free space and the write that failed: a 400MB quant with 1.5GB free has all it needs, and calling that a full disk hid the rebuild advice that would have fixed the real problem. Note for callers: save_to_gguf(merge_is_disposable = True) now also needs preexisting_weights before it will reclaim anything. Erring toward keeping files seemed right for the one case where being wrong is unrecoverable. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Credit a failed pass's partial output back before calling the disk full llama-quantize streams into its output file, so a pass that dies partway leaves those bytes on disk. The disk check then measured the remaining free space against an estimate of the whole output and read a 10GB export that started with 12GB free and failed on an unsupported tensor as a full disk, suppressing the llama.cpp rebuild advice that addressed the real failure. Add the partial file back, on its own filesystem only. * Price the failure diagnosis low and reclaim the index with its shards * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-13 04:42:31 -07:00
Daniel Han	85a036a195	Keep an explicitly requested float32 model in float32 without bfloat16 (#7867) ... * Keep an explicitly requested float32 model in float32 without bfloat16 A model loaded with dtype = torch.float32 was being wrapped in float16 autocast on V100/T4 even when the caller set fp16 = False and bf16 = False. The mixed precision block in rl.py reads "neither flag set" as "caller did not choose" and picks for them: use_bf16_amp = (not float16) and _bf16_supported() args.fp16 = not use_bf16_amp so on a GPU without bf16 the answer is always float16. float16 has five exponent bits against float32's eight, so a value the model was loaded wide enough to hold overflows to inf and then NaN. Spark_TTS_(0_5B) is the case that surfaced it: it loads with dtype = torch.float32 ("Spark seems to only work on float32 for now"), sets fp16 = False, bf16 = False, and on a T4 logs nan for every step, then dies at inference inside torch.multinomial, which refuses a distribution containing NaN. The sampler assert is two steps downstream of the cause. Suppressing the autocast for every float32 model is too wide: full finetuning upcasts trainable weights to float32 by itself, and float16 autocast over float32 master weights is the ordinary V100/T4 recipe (issue #4082, tests/python/test_v100_fullft_precision.py). So from_pretrained now records whether float32 was ASKED for in UNSLOTH_USER_FLOAT32, as opposed to arrived at by upcasting, and only an explicit request suppresses the autocast. It is set in both loaders, since FastLanguageModel and FastBaseModel resolve dtype independently, and written on every load so a previous run cannot leave a stale "1" behind. Only the float16 fallback changes. bf16 has float32's exponent range, so a bf16 GPU keeps autocasting and keeps the memory saving. An explicit fp16 = True or bf16 = True is still obeyed, float16 and bfloat16 models are untouched, and UNSLOTH_FORCE_FLOAT32 models still take their earlier branch. tests/test_float32_no_fp16_autocast.py pulls the mixed precision block out of rl.py by AST and executes it against fake args/model objects with the bf16 capability check stubbed, so the no-bf16 path is covered without a V100/T4. No GPU, no network, no trl import. Also, separately: do not let one optional audio dependency kill the whole test session. require_python_package calls sys.exit(1) when a package is missing, and every caller is a test module invoking it at import time, so under pytest that is not a skip of one module -- SystemExit propagates out of collection and the run ends with INTERNALERROR> SystemExit: 1 with no report at all. A missing xcodec2 took out 1000+ unrelated tests. It now skips the module under pytest; outside pytest the exit is unchanged, since these helpers are also used by standalone scripts. test_whisper.py had the same shape for a different reason: it downloads an audio fixture at import and did `assert False` on failure. Wikimedia rate limits and this URL really did answer 429 during a batch run, and an asset we could not fetch says nothing about unsloth, so it skips too. * Record the float32 request on the model, and let GRPO honour a disabled autocast Two real gaps in the float32 gating, plus the provenance problem underneath both. The gate never reached most users. llama, mistral, gemma, gemma2, qwen2 and qwen3 LoRA/QLoRA loads go through FastLanguageModel to dispatch_model.from_pretrained, which is neither of the two loaders that recorded the request, so user_float32 stayed false and the fp16 autocast this branch exists to avoid ran anyway. That is most of the notebooks. The value was also process-global and derived rather than asked for. A program that loads two models before building a trainer described whichever loaded last, and a dtype inferred from a 4bit config's bnb_4bit_compute_dtype was recorded as an explicit model-wide request, which is especially wrong under full finetuning where the loader then turns 4bit off again. Both entry points now read the argument as the caller wrote it, before any normalisation, and the answer is attached to the model. The outermost caller wins, since only it saw the raw argument; the two delegating returns in FastLanguageModel re-stamp it for that reason. Then ACCELERATE_MIXED_PRECISION = 'no' was read as a two-way switch in the GRPO replacements, so 'no' came out as bfloat16 and autocast was entered anyway. torch does not ignore that on a T4 or V100, it raises "Current CUDA Device does not support bfloat16", which is exactly the hardware this branch targets. Verified against torch's own autocast __init__: the check is gated on `enabled`, so disabling autocast is both the correct reading of 'no' and the minimal fix. Full finetuning already set 'no', so this was reachable before this PR too. Tests: 4384 passed against 4369 on the same tree with these changes stashed, identical 1310 environment failures either way, and nothing failing on this branch that is not also failing on its base. * Keep forced float32 on fp16 autocast, size GRPO chunks by the flag, and run the tests off a GPU Three follow-ups on the autocast change. UNSLOTH_FORCE_FLOAT32 exports ACCELERATE_MIXED_PRECISION 'no' as well, and the new gate read only that, so the injected _prepare_inputs header disabled autocast for it while the dtype expression right beside it was still choosing float16. That left Gemma3 and gpt-oss generation in full float32, and it disagreed with the two other consumers, which set _autocast_enabled True for exactly this case. The header honours the flag now. The GRPO chunk autotuner derived dtype_bytes from _autocast_dtype alone, which stays bfloat16 when autocast is off, so it sized hidden states and logits as 16-bit for a forward running in float32. That is half the real figure, on the explicit-float32 and full-finetuning runs this branch is for. The new tests drove torch.amp.autocast(device_type = "cuda") on a box that has CUDA. On a CPU runner torch warns "CUDA is not available. Disabling" and hands back a no-op, so the premise test would not have raised and every enabled case would have read False, failing for reasons unrelated to the code. Measured, then fixed by claiming the device rather than skipping, since a CPU job is where these would most need to run. They pass with the GPUs visible and with CUDA_VISIBLE_DEVICES empty. Tests: 4386 passed, same 1310 environment failures as the branch base, nothing failing here that is not also failing there. * Size the GRPO chunk on the dtype the forward runs in The autocast-off branch assumed float32, but pure bfloat16 full finetuning also disables autocast (ACCELERATE_MIXED_PRECISION='no') while keeping bf16 weights and a bf16 forward. That path got dtype_bytes=32 where main gave 16, cutting the chunk ~1.5x and doubling the forward-pass count on 10-22GB cards. Read lm_head.dtype instead: float32 for an explicit float32 load, bfloat16 here. Retargets the test that pinned the old value. * Apply ruff kwarg-spacing formatting pre-commit.ci's ruff-format-with-kwargs hook reported "files were modified by this hook" and could not push the result back, so the check stayed red. Applied locally instead. Three source-asserting tests had to be made robust rather than re-pinned, since the reformatting is legitimate and will happen again: - test_the_flag_is_recorded_beside_the_dtype counted the literal "self._autocast_enabled = (", which the formatter collapsed onto one line. Now pairs each _autocast_dtype initialiser with a nearby flag assignment by position, so it still catches a genuinely missing one. - test_the_original_error_is_still_reported anchored on a whole f-string literal, which the formatter merged with the hint that follows it. Anchors on the message text alone now; either shape satisfies the intent. - test_the_source_beats_the_version_fallback exposed a real fragility rather than a test problem: the detector matched the literal "kw_only=True", so any whitespace in transformers' own source would have read an install that needs nothing as one that needs patching. Made whitespace-tolerant. * Latch the GRPO autocast decision on the trainer * Add the license header to the new test files * Stamp the forced float32 answer on the model UNSLOTH_FORCE_FLOAT32 is process wide and from_pretrained clears it on every load, so a model loaded after a Gemma3 or gpt-oss trainer was built left '0' behind and GRPO's first generation dropped the float16 autocast that rl.py's 'no' was written expecting, running generation in full float32. The loaders now stamp the answer on the model they loaded, next to the existing float32 request marker, and the trainer reads it from there. Models without the stamp fall back to the environment as before. * Preserve a bf16 trainer when a forced-float32 model was loaded (#7867) * Stamp the forced float32 answer on every loader return path (#7867) * Stamp the float32 answers on the text-diffusion dispatch (#7867) * Apply the repo ruff-format hook to a test touched by this PR * Apply the repo ruff-format hook (ruff 0.6.9) to files this PR touches * Fix the trainer reading the process-wide forced float32 flag for PR #7867 * Fix a missing optional dependency aborting the whole pytest session for PR #7867 require_package exited the process when a system package was absent. The TTS test modules call it at import time, so under pytest the SystemExit escaped collection and ended the run with INTERNALERROR / exit code 3 instead of skipping one file. It now degrades to a module-level skip under pytest, the same treatment require_python_package already had, and keeps the exit for the standalone script callers. test_whisper.py also imported the unsloth runtime unguarded, which is a collection error on the Windows runner where triton is absent; guard it with a module-level skip. * Inherit an outer autocast by omitting dtype, not by a sentinel * Read the forced float32 stamp inside native fast generation * Stamp the full finetuning mode on the model the trainer reads * Keep one missing-dependency helper after the merge with main The merge left this branch's _skip_if_pytest alongside main's _missing_dependency, doing the same job, plus two near-duplicate test files. Fold both into main's helper and its test module, keeping this branch's extra coverage of require_package and of the call at module scope. * Make the generation autocast tests independent of the host having CUDA torch.autocast(device_type = 'cuda') disables itself when CUDA is absent, so reading _enabled off the constructed object measured the runner instead of the branch, and these could never pass on a CPU-only machine, a Mac or Windows. Record the arguments the code asks for instead. Also skip before importing rl_replacements, which needs unsloth_zoo. * Apply the repo's ruff kwarg-spacing formatting * Format these three files with the ruff the hook actually pins The ruff-format-with-kwargs hook installs ruff==0.6.9 through additional_dependencies, and 0.6.9 and 0.15 disagree on where the message goes in a multi-line assert and on how a long lambda signature wraps. I had run the script with the workspace ruff, so pre-commit.ci kept reformatting these back and failing, with nothing to show in the diff. AST is identical on all three. * Drop the nullcontext import this PR no longer uses Left over from the earlier shape of the autocast gating, which now omits dtype instead of passing a sentinel context. Source lint's import-hoist check treats an added-but-unused hoisted import as a blocker, and it is right: nothing in this file references it. * Tighten comments for the float32 dtype gating change --------- Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-08-09 05:12:02 -07:00
oobabooga	e8fcbd68d1	Studio: only clean up GGUF artifacts the export owns (#7940) ... * Studio: only clean up GGUF artifacts the export owns * Studio: reject symlinked GGUF export artifacts * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix undefined temp-root cleanup and drop unused glob import for PR #7940 * Keep the materialized imatrix out of the GGUF export outputs for PR #7940 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let the imatrix retain the temporary export root for PR #7940 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the imatrix by path so a name clash cannot drop the real GGUF for PR #7940 * Apply the repo kwarg spacing format for PR #7940 * Keep an optional Modelfile relocation failure from sinking the export for PR #7940 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com>	2026-08-09 03:59:59 -07:00
Daniel Han	68e5c3f04b	tests: make pytest tests collect cleanly and drop the deleted QVQ registry mirror (#8206) ... * tests: make `pytest tests` collect cleanly and drop the deleted QVQ registry mirror Collection was interrupted by 17 errors, so nothing after them ran, and tests/test_model_registry.py::test_model_registration[qwen] was red against the live Hub. Registry: - unsloth/QVQ-72B-Preview no longer exists on the Hub. Authenticated as an unsloth org member the API answers 404 and an org-wide search returns only unsloth/QVQ-72B-Preview-bnb-4bit, so it is not private and not gated (a gated repo still serves public metadata). Drop QuantType.NONE from the QVQ meta; the bnb-4bit mirror and the upstream Qwen/QVQ-72B-Preview stay. - The registry check now treats only RepositoryNotFoundError as "missing". Every other hub error (429, 5xx, DNS, timeout) skips the test instead of reporting all 129 models missing. Collection: - tests/saving files with no test items are standalone GPU scripts whose body runs at import, so bare collection downloaded checkpoints, trained and pushed to the Hub. They now call require_opt_in() and are visible skips unless UNSLOTH_RUN_SAVING_SCRIPTS=1. Running them directly is unchanged. - Four filenames contained a dot, which pytest's importer reads as a package separator; renamed to underscores. - tests/test_raw_text.py left its datasets stub in sys.modules for the rest of the session, which broke collection of tests/utils/test_packing.py. The stub is now restored after raw_text is loaded. - The two sentencepiece tests imported the legacy transformers.utils.sentencepiece_model_pb2, which fails on protobuf >= 4. Use convert_slow_tokenizer.import_protobuf(), the accessor the code under test uses. - tests/test_collection_hygiene.py guards all three regressions. tests --collect-only: 7629 collected + 17 errors in 214s (exit 2) -> 7703 collected, 0 errors in 3s (exit 0). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-08-08 20:03:55 -07:00
Vineeth Sai Varikuntla	d32b93c061	Guard VLM detection against config.architectures being None in the save paths (#7372)	2026-07-31 23:37:34 -03:00
JoshuaL3000	e662af769b	fix: enable XPU support and update hardcoded CUDA selections for tests (#7401) ... * fix: add XPU device support and update hardcoded CUDA selections * fix: add XPU device support for pytest CUDA skipped tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix device handling for PR #7401 - perplexity_eval.py: use DEVICE_TYPE_TORCH, not DEVICE_TYPE. The latter can be "hip" or "mlx", which .to() rejects, so this regressed ROCm. - test_batched_leftpad_generation_gpu.py: XPU diverges here today, so mark it non-strict xfail on XPU instead of reverting to a CUDA-only guard. Keeps the real XPU gap visible and turns green once it is fixed. - Guard torch.xpu.is_available() with hasattr, matching device_type.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-enable the flash varlen attention test in CI for PR #7401 attention_dispatch.py now predefines flash_attn_func / flash_attn_varlen_func as None, so test_run_attention_flash_varlen_receives_window_and_softcap no longer needs flash_attn importable to be monkeypatched. Verified on a runner shaped like the CPU-only one: the test fails against main's attention_dispatch and passes at this head, so the deselect is now dead weight. * Tighten comments for PR #7401 Drop the hasattr rationale: torch.xpu has existed since torch 2.3 and the dependency floor is 2.4, so no supported build predates the namespace. The guard stays as cheap defence, but the comment claimed something untrue. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-07-28 15:38:57 -07:00
Souravrajvi0	8b9ee5facb	avoid Hub metadata probe when loading tokenizers with local_files_only (#7482) ... * fix: keep offline GGUF export off the Hub for VLM tokenizers (#7481) Resolve cached snapshot directories before loading PreTrainedTokenizerFast during VLM processor fallback so transformers does not call is_base_mistral() -> model_info() when HF_HUB_OFFLINE is set. Also probe the local cache in _has_tokenizer_model instead of model_info when offline. Fixes unslothai/unsloth#7481 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * test: add real-cache offline GGUF integration checks for #7481 Download unsloth/gemma-3-270m-it-bnb-4bit (~430MB) and verify offline snapshot resolution and tokenizer load with network blocked. Full unsloth import tests remain GPU-gated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: address Codex review on offline GGUF tokenizer paths (#7481) - Only rewrite Hub repo ids to cached snapshot dirs when offline - Copy tokenizer.model from cache offline in preserve_sentencepiece - Do not cache negative offline tokenizer.model probe results - Add regression tests for all three review items * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: probe HF cache before model_info for local-only GGUF saves (#7481) Always resolve tokenizer.model from the local Hub cache before calling model_info, and skip Hub metadata when the tokenizer was loaded with local_files_only or offline env vars. Fixes Codex review on PR #7482. * Fix lint blocker, false-green tests and offline defaults for PR #7482 Drop the two unused _env_says_offline imports that fail the Source lint import-hoist check. test_has_tokenizer_model_offline_skips_model_info and its local_files_only twin set model_info.side_effect = AssertionError, but _has_tokenizer_model wraps that call in "except Exception: return False", so the AssertionError was swallowed and both passed on the merge base with the fix absent. Assert model_info.call_count == 0 instead; both now fail on the base with assert 1 == 0. The real-cache integration tests called hf_hub_download and PreTrainedTokenizerFast directly, so they exercised plain huggingface_hub and passed identically on both trees. Route them through the resolver this PR adds, and gate the file at module level since importing unsloth needs a GPU host either way. _resolve_hub_repo_local_dir and _resolve_hub_repo_cached_file defaulted to local_files_only = False, so a helper named "resolve local dir" would download with backoff retries when called without the flag. Every caller already passes it explicitly, so default it closed. Use tempfile.gettempdir() rather than a hardcoded /tmp, which silently skipped both files on Windows, the platform in the bug report. Patch socket.socket connect rather than replacing the class, which broke isinstance checks. Wire the unit tests into the Bucket-A CI list; Repo tests (CPU) ignores tests/saving, so none of these ran anywhere. * docs: note transformers 4.57.2-5.5.4 window for local tokenizer resolve Name the version range where from_pretrained still probes model_info under local_files_only, and point at the 5.6.0 upstream fix so the helper can be removed once the supported floor moves past it. * fix: enable real-cache suite in offline GGUF integration runner Pass UNSLOTH_INTEGRATION_IMPORT=1 into the pytest subprocess so the documented runner actually executes the real-cache tests instead of reporting success after only the fake-cache unit file runs. * docs: note integration runner enables UNSLOTH_INTEGRATION_IMPORT Document that the runner sets the gate itself and still needs a host that can import unsloth. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep an explicit local_files_only load local-only at save time transformers takes local_files_only as an explicit from_pretrained parameter, so it never lands in tokenizer.init_kwargs, and _offline_aware_load restores HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE as soon as the load window closes. A VLM loaded with local_files_only = True but no offline env var therefore came back with the Hub repo id in name_or_path and nothing recording the request, so _tokenizer_wants_local_only returned False on the later save and _has_tokenizer_model fell through to HfApi.model_info - and then _preserve_sentencepiece_tokenizer_assets fetched tokenizer.model from the Hub with local_files_only = False. On a disconnected host that is a network wait before the export gives up. Stamp the load's local-only mode onto the returned processor and its tokenizer inside the forced-offline window, and honour that stamp in _tokenizer_wants_local_only, so the save path inherits the load's contract. Verified against a real hub-cache layout whose snapshot has tokenizer metadata but no tokenizer.model: before, one model_info call plus an hf_hub_download with local_files_only = False; after, zero model_info calls and cache probes only. Two tests added to tests/saving/test_offline_gguf_vlm_tokenizer_7481.py; both fail with the loader_utils hunk reverted and pass with it in place. * Carry the load's cache_dir through to saving for PR #7482 The local-only stamp added in e7b7400de preserved only the boolean. Saving still derived its cache from HF_HUB_CACHE or HF_HOME, which does not see a caller-supplied cache_dir, and FastBaseModel.from_pretrained threads one all the way down. So a local_files_only load against a custom cache missed on the probe, and the stamp then stopped the Hub fallback that used to cover it, and tokenizer.model was silently left out of the GGUF staging directory. Stamp the cache_dir alongside the local-only marker and prefer it at both sites in save.py that derive one from the environment. Reverting save.py alone, with the helper still present, fails the new test on behaviour. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Merge main and drop an unused import for PR #7482 Brings the branch up to date with main, which clears the stale Source lint blocker inherited from #7476 by taking studio/backend/utils/hardware/__init__.py out of this PR's changed-file set. pytest was imported in the new test file and never used, which the import-hoist check flags in its own right. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-07-27 05:59:48 -07:00
Leo Borcherding	1dd2fc4583	tests: read checked-in files as UTF-8 instead of the platform default (#7438) ... * tests: read checked-in files as UTF-8 instead of the platform default Path.read_text() with no encoding uses locale.getpreferredencoding(), which is UTF-8 on the Linux runners and cp1252 on a stock Windows install. Nine module-level reads of checked-in source files were relying on that default. studio/backend/routes/inference.py carries the DeepSeek tool-call token regexes, so it holds U+FF5C and U+2581. Under cp1252 that read raised UnicodeDecodeError on byte 0x81 at position 97806, and because the reads run at import time it took test_cancel_atomicity.py and test_cancel_id_wiring.py out at collection, not as failures. Green on CI, permanently broken for a Windows contributor running the suite locally. Adds a guard: at module scope there is no tmp_path fixture, so a bare read_text()/write_text()/open() there is always touching a checked-in file. That makes the rule mechanical enough to enforce with no allowlist, while staying quiet about temp-dir I/O inside test bodies where the platform default is harmless. The repo already spells this correctly in 464 other places; this only stops the stragglers coming back. * tests: cover import-time helper reads and keep the guard py3.9-safe Follows up on the Codex review: - add `from __future__ import annotations`, since `str | None` in `_offender` is evaluated at import on Python 3.9 and pyproject declares requires-python ">=3.9,<3.15". - widen the guard from module scope to import time. Class bodies and the bodies of module-level helpers called from an executing statement run during collection too, so `CODE = _extract_mixed_precision_code()` was the same hazard as an inline read. `if __name__ == "__main__":` blocks are skipped: pytest never executes them. - scan studio/backend/tests/ as well as tests/. Both trees are collected on Windows by separate CI jobs, and the offender that started this, test_tool_xml_strip.py reading routes/inference.py, lives there. Widening it surfaced seven more import-time reads of checked-in sources; all now name utf-8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden the import-time encoding guard for PR #7438 Close the detector gaps raised in review, all of which I reproduced against the actual AST before changing anything. False negatives (the guard let a real hazard through): - _is_main_guard ignored the comparison operator, so if __name__ != "__main__" counted as script-only even though its body runs at import. - The else arm of a main guard was discarded with the rest of the If node. - Decorators and argument defaults on a module-level def were skipped with the body, though both are evaluated when the def executes. - Path.open() in text mode was invisible; only builtin open() was matched. - encoding = None and encoding = "locale" both re-select the platform default, but the keyword merely being present counted as pinned. False positives (the guard would have blocked a compliant contributor): - A non-literal mode fell through to the "r" default, so open(p, mode) was flagged even when mode is "rb", where adding encoding= is a ValueError and there is no edit that satisfies the rule. - Same for open(*args) and a **kwargs splat, which hide the mode and can hide an encoding. - Lambda bodies and comprehension elements were walked even though neither runs at definition. Verified: still reports the same 22 offenders on unpatched main, green on this branch and on the tree merged with latest main (557 files), and an adversarial corpus of 33 cases now scores zero false positives and zero false negatives. Also corrected two docstring claims: neither collecting job runs on Windows, and the read is governed by locale.getencoding(). * Walk eager comprehensions and treat io.open as the builtin Two regressions from the previous commit, both reproduced against the AST before changing anything. Lumping list, set and dict comprehensions in with generator expressions was wrong. Only a genexp is lazy; the other three run their element expression, their filters and their nested iterators immediately, so CONTENTS = [p.read_text() for p in PATHS] at module scope is an import-time read the guard was silently missing. Comprehensions are now walked in full and only the genexp keeps the outermost-iterable-only treatment. io was also in the not-a-path-opener list, but io.open is the builtin, with the same mode position and the same platform default. io.open(CHECKED_IN_FILE) is exactly the hazard this guard exists for, so it is matched now, with binary modes and a pinned encoding still exempt. tarfile.open and fitz.open stay exempt since neither has an encoding to name. Verified: 13 targeted cases covering all five eager comprehension forms and io.open in text, binary and pinned shapes all classify correctly; still 22 offenders on unpatched main; green on this branch and on the tree merged with latest main. * Close three more walker gaps in the import-time guard All three reproduced against the AST first. A generator expression handed straight to a call is consumed there, so DATA = "".join(p.read_text() for p in paths) runs its element at import. Only an unconsumed genexp bound to a name stays lazy, so the walker now follows the consumed ones in full and keeps the outermost-iterable-only treatment for the rest. if "__main__" == __name__ is an equivalent and accepted spelling of the main guard, but requiring __name__ on the left meant its body was treated as import-time code. That is a false positive on a block pytest never runs, so both operand orders are recognised now. The helper table was built from module-level defs only, so a def in a class body invoked while the class is constructed was never followed, contradicting the walker's stated coverage of class bodies. Helpers are now collected from the module body and from class bodies at any nesting. Verified: 15 targeted cases including all three fixes and the earlier ones still classify correctly; still 22 offenders on unpatched main; green on this branch and on the tree merged with latest main. * Handle positional read_text encodings, lazy generators and nested helpers * Guard reads reached from test bodies, unbound Path calls and __file__ paths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Follow derived paths, skip lazy generator helpers, cover compressed openers * Guard the CLI tests, helper parameters and unbound Path arguments * Discover test roots and follow literal, in-place and tuple-derived paths * Identify module openers by import, unwrap starred paths, pin subprocess snippets * Resolve import origins, seed helper locals, follow named generators and parametrize * Scope imports lexically, list tracked test files, bind unpacked names * Resolve aliased openers, keyword-only params, destructured targets, next() * Pin the encoding on subprocess snippets, workflow lint and CLI output for PR #7438 * Harden the CLI encoding guard against detached streams for PR #7438 * Tighten the encoding guard's path and scope analysis for PR #7438 * Resolve path provenance more precisely and keep POSIX stream encodings for PR #7438 * Resolve qualified path classes and scope conditional imports for PR #7438 * Scope CLI stream setup to the entry point and align two encoding pairs for PR #7438 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-07-26 23:31:56 -07:00
Michael Han	6d8c18cd1a	Replace standalone Studio wording with Unsloth (#7221) ... * Replace standalone Studio wording with Unsloth Replace the single word Studio with Unsloth wherever it is used as shorthand for Unsloth Studio in docs, CLI output, UI strings, i18n locales, workflow display names, comments and docstrings. Kept unchanged: the full name Unsloth Studio, third party product names (LM Studio, Visual Studio, Mac Studio), feature names (Recipe Studio, Fine-tuning Studio and its translations), and all identifiers such as env vars, commands, paths and filenames. * Address review feedback on the Studio wording rename Use "an" before Unsloth where the rename left the article as "a". Restore the split brand where Unsloth and Studio render as two halves of the full product name: the onboarding sidebar subtitle and the IPv6 localhost warning. Scope two messages to the full name Unsloth Studio where plain Unsloth was misleading: the AMD README bullet and the CLI studio setup error.	2026-07-19 00:47:04 -07:00
Andrew Chen	4e09328c3b	fix(tokenizer): check for tokenizer.model after saving it, not before (#7194) ... * fix(tokenizer): check for tokenizer.model after saving it, not before `fix_sentencepiece_tokenizer` creates its temporary directory, then returns early unless that directory already contains a tokenizer.model: if not os.path.exists(temporary_location): os.makedirs(temporary_location) # fresh, empty if not os.path.isfile(f"{temporary_location}/tokenizer.model"): return new_tokenizer # always true old_tokenizer.save_pretrained(temporary_location) # writes that file The file only appears on the line after the check, so the guard is always true and the body never runs. Nothing else writes that path either -- `convert_to_fast_tokenizer` saves into a per-name subdirectory, not `{temporary_location}/tokenizer.model`. Both call sites are in `get_chat_template` and are commented "Must fix the sentence piece tokenizer since there's no tokenizer.model file!" -- the guard defeats the exact intent the caller states. The effect is silent: the caller still gets a working `new_tokenizer`, but the sentencepiece piece rename is skipped, so the mapped token (e.g. the eos token remapped to `<|im_end|>`) is missing from tokenizer.model and GGUF/llama.cpp exports carry the old piece. `check_if_sentencepiece_model` in save.py does the same probe in the right order -- makedirs, save_pretrained, then isfile. Match it. Tests are added under tests/saving/ next to the existing sentencepiece coverage, and to the two Bucket-A lists in consolidated-tests-ci.yml, since Repo tests (CPU) --ignores tests/saving and these need protobuf. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Clear stale tokenizer.model before the sentencepiece guard The guard now runs after old_tokenizer.save_pretrained, but the default temporary_location is a fixed reusable directory. A fast-only tokenizer writes no tokenizer.model, so a stale file from an earlier sentencepiece call could pass the guard and patch the wrong model (e.g. mixing models in one process, like a long-running server). Remove any existing tokenizer.model first, and add a regression test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Empty the reusable sentencepiece scratch directory each call The final AutoTokenizer.from_pretrained reloads the whole temporary_location, so removing only a stale tokenizer.model still let other artifacts from a previous tokenizer (added_tokens.json, chat template, etc.) leak into the reload when the default reusable directory is used across models in one process. Recreate the directory instead, and add a regression test for the leaked-artifact case. * Clear only top-level scratch files, keep subdirectories Recreating the whole reusable directory deleted the {name} subtree that convert_to_fast_tokenizer stores a converted tokenizer's source vocab in, so old_tokenizer.save_pretrained could not copy tokenizer.model and the guard returned the tokenizer unpatched for those legacy converted tokenizers. Remove only stale top-level files (all the final reload reads) and leave subdirectories intact. Add a regression test for the converted-source subdirectory. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the current tokenizer's own source vocab when clearing On a repeated get_chat_template(map_eos_token=True) call, the returned tokenizer's vocab_file points back at the top-level tokenizer.model, and the cleanup deleted that source before old_tokenizer.save_pretrained could re-emit it, so the guard returned the tokenizer unpatched. Skip removing the old tokenizer's own source vocab while still clearing stale files from a different tokenizer, and add a regression test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Use a per-call temporary directory for the sentencepiece fix The scratch directory defaulted to a single shared path, so concurrent or repeated get_chat_template(map_eos_token=True) calls could delete or overwrite each other's tokenizer.model between save and reload (tripping the piece assertion or reloading the wrong model), and stale files from an earlier tokenizer could leak into the reload. Work in a unique per-call subdirectory instead: this isolates every call without deleting anything the caller owns, and replaces the earlier per-file cleanup. Tests updated to read the patched model from the reloaded directory and to cover isolation and source-vocab preservation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pass only the applied token mappings into the sentencepiece fix get_chat_template mirrors token remaps into tokenizer.model via fix_sentencepiece_tokenizer, but two caller paths passed a mapping that did not match what they wrote to the fast tokenizer JSON, so once the sentencepiece patch runs the model and JSON disagree: - the mapped-token path skipped entries whose target already existed but still passed the full mapping, renaming a piece the JSON never changed; - the EOS-swap path swapped both tokens in the JSON but passed only one direction, leaving two stop_word pieces and no old EOS piece. Pass the applied mapping (and both swap directions) instead. Add regression tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten sentencepiece guard comments * Add SPDX license identifier to sentencepiece guard test * Reclaim the per-call sentencepiece scratch directory The per-call tempfile.mkdtemp fixed the shared-directory race but never cleaned up, so a long-running process leaked one scratch dir per call. The dir cannot be deleted eagerly for sentencepiece tokenizers because the returned tokenizer's vocab_file points into it (a later save_pretrained copies the patched tokenizer.model from there). Reclaim it correctly instead: remove the dir right away on the fast-only path (the returned tokenizer never references it), and attach a weakref.finalize so the sentencepiece dir is removed once its tokenizer is garbage collected. Add regression tests for both. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the scratch-dir reclaim comment --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com>	2026-07-18 05:53:33 -07:00
Andrew Chen	b508c8fe89	fix(save): unsloth_push_to_hub_gguf(save_method="lora") raises NameError (#7193) ... * fix(save): unsloth_push_to_hub_gguf(save_method="lora") raises NameError unsloth_push_to_hub_gguf reads is_main_process at save.py:3181 but never declares it. Its twin unsloth_save_pretrained_gguf declares it (2783) and uses it the same way (2839) -- the LoRA branch was copied between the twins, the parameter it depends on was not. There is no module-level global, so the name resolves as a global load and the branch raises NameError 100% of the time. save_pretrained_gguf(save_method="lora", push_to_hub=True) raises a ValueError that tells users to "use .push_to_hub_gguf(save_method='lora') instead" -- the documented escape hatch is the broken call. Add is_main_process to the signature, positioned as in the twin, and forward it to unsloth_save_pretrained_gguf on the merged path so the parameter is not silently ignored there. Default stays True, so nothing changes for existing callers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(save): preserve GGUF push compatibility --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>	2026-07-17 11:55:34 +03:00
dylanschroers	8cfd1a2173	fix: single-pass GGUF export for directly convertible outtypes in save.py (#7090) ... * Single-pass GGUF export for direct outtypes + parallel multi-quant save_to_gguf defaulted first_conversion to model_dtype before the block that picks the optimal base conversion, leaving that block dead since it landed (#3356). Every default export (fast_quantized -> q8_0) therefore ran two passes: convert HF -> 16-bit GGUF, then llama-quantize -> q8_0, writing a 2x-size intermediate that the cleanup step deletes again. - Route single-output exports whose type convert_hf_to_gguf.py emits directly (f32/f16/bf16/q8_0) through one conversion pass with no 16-bit intermediate. Measured on Qwen2.5-0.5B-Instruct (8-core CPU): bytes written 1525 MB -> 531 MB (2.9x less), peak extra disk 994 MB -> 0, wall time neutral on local NVMe (14.8s vs 15.4s). The dequantized q8_0 tensors are bit-identical to the two-pass output (max diff 0 over all 290 tensors, same quant-type table). On disk-capped runtimes (Kaggle 20 GB, Colab) the removed intermediate is the difference between an export that fits and one that dies - see the Kaggle error text this file already carries. imatrix runs keep the two-pass route since only llama-quantize can apply one; explicit first_conversion is still honored. - Run independent llama-quantize passes two at a time when several quant methods are requested (thread budget split between workers, outputs byte-identical, order preserved). Measured 1.38x wall-clock on q4_k_m+q5_k_m+q6_k. Sequential under UNSLOTH_ENABLE_LOGGING=1 to keep subprocess logs readable; kill switch UNSLOTH_PARALLEL_GGUF_QUANTS=0. Duplicate methods now quantize once. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard parallel GGUF quant on Kaggle and make multi-quant failures atomic Each llama-quantize pass loads the whole model into RAM, so running two at once on Kaggle can OOM a host that succeeded sequentially; skip the parallel path there. On a failed multi-quant export, stop launching queued passes and remove orphaned quant outputs so a failure leaves no partial GGUFs behind, keeping the 16-bit base for retry. Also accept 0/false/no/off/empty for UNSLOTH_PARALLEL_GGUF_QUANTS so a well-meant 'false' actually disables parallelism, and add tests/saving/test_gguf_single_pass_export.py to the CI saving bucket so the new tests run. * Preserve pre-existing outputs for canceled quant passes on failure The parallel cleanup unlinked every requested output name, so a failed rerun could delete a valid model.<METHOD>.gguf left by an earlier successful export for a method whose pass was canceled and never ran this session. Skip canceled futures and only remove outputs from passes that actually executed. * Gate parallel GGUF quant on available memory and preserve prior outputs Skip the two-worker path when RAM cannot hold two full-model quantizations at once (and on Colab as well as Kaggle), so a multi-quant export that fit sequentially no longer OOMs. On failure, remove only outputs this run newly created, tracked against a pre-launch snapshot, so a rerun into an existing _gguf directory never deletes a valid artifact from an earlier export. --------- Co-authored-by: djs <dschroers2@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <michaelhan2050@gmail.com>	2026-07-15 05:25:03 -07:00
Michael Han	af93868760	Fix repeated base model downloads across checkpoint exports (#6896) ... * Fix repeated base model downloads across checkpoint exports (#6890) Pre-warm the HF hub cache with the 16bit base weights before merge_and_overwrite_lora runs. The merge fetches shards with hf_hub_download(local_dir=...), which never populates the hub cache, so temporary merge directories (GGUF checkpoint exports) forced a full re-download of the base model for every checkpoint. The first export now downloads once into the cache and later exports copy from it. Skips itself when already cached, offline, on Kaggle/Colab, for local or nf4/fp4 bases, non-downloading save methods, or low disk. Opt out with UNSLOTH_PREWARM_HUB_CACHE=0. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Show MB for small base models in the pre-warm download message * Harden pre-warm: getattr for model config, abspath for relative HF_HUB_CACHE - Read config._name_or_path via getattr so a model without a config skips cleanly instead of taking the outer error path. - abspath the cache probe so a relative HF_HUB_CACHE walks up to a real root rather than "", which would zero the free-space check and skip pre-warm. Both from PR review; each covered by a test that fails without the fix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pre-warm the live-env HF cache so runtime redirects still hit (#6890) Resolve the hub cache the same way the merge does (unsloth_zoo _active_caches, live env) instead of huggingface_hub's import-time-frozen constants.HF_HUB_CACHE, and pass it as cache_dir to the cached probe, disk check and snapshot_download. Without this, a runtime HF_HOME/HF_HUB_CACHE redirect (unsloth_zoo redirect_hf_cache_if_readonly on a read-only default cache, or Studio) makes the pre-warm populate a different directory than the one the merge reads, so the cache-copy fast path misses and the base re-downloads on every export anyway. Adds 3 regression tests covering the cache_dir threading and the redirect case. * Apply ruff-format kwarg spacing to the pre-warm cache-dir changes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pre-warm the 16bit sibling for FP8 bases so their merged_16bit exports reuse the cache too For a merged_16bit export of an FP8 base with an existing 16bit sibling, the merge swaps to the sibling and downloads that (unsloth_zoo _resolve_fp8_16bit_sibling), so pre-warming the FP8 repo missed the cache and re-downloaded the sibling every export. Mirror the swap and pre-warm the sibling. No sibling still caches the FP8 repo for the in-place dequant path. Adds 2 regression tests. * Filter pre-warm shards through the safetensors index like the merge does Repos that ship a leftover shard set the index does not reference (e.g. granite-3.2) made the disk gate over-count and snapshot_download fetch shards the merge never reads. Mirror the merge: on the download path, keep only index-referenced shards. Runs after the already-cached check so the cached fast path stays network-free. Adds 2 tests. * Tighten pre-warm comments --------- Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>	2026-07-06 23:38:29 -07:00
Anas Khan	c44d94f1ae	fix: map None quant method to q8_0 before lowercasing in GGUF export (#6889)	2026-07-06 07:11:49 -07:00
Daniel Han	64f6526160	Fix export-time trust_remote_code bypass in FP8/INT8/GGUF-LoRA export (#6869) ... * Fix export-time trust_remote_code bypass in FP8/INT8/GGUF-LoRA export The torchao, compressed-tensors, and LoRA GGUF export paths re-read the merged checkpoint and used to set trust_remote_code from the checkpoint config's static auto_map (the torchao path also scanned the staged tokenizer/processor configs). A model that loads with built-in Transformers classes can carry an auto_map entry, which skips the load-time remote-code consent scan (that only runs when the load already requested trust_remote_code) yet flips trust_remote_code on at export, running unvetted custom code. Derive the reload trust_remote_code from the approved load decision instead: a new _loaded_via_remote_code() checks whether the in-memory model / tokenizer was itself loaded from custom code (its class lives in the transformers_modules package), walking PEFT / wrapper layers. Built-in-loaded models no longer gain trust from config metadata; genuine custom-code models (loaded with consent) still reload correctly. Add CPU-only regression tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden _loaded_via_remote_code against a None/missing __module__ Read type(node).__module__ via getattr and require a string before startswith, so a dynamically created or C-extension class with a None module does not raise during export. Add a regression test. * Split model and tokenizer trust for the compressed subprocess, walk processor components The compressed-tensors export collapsed model and tokenizer trust into one --trust-remote-code flag, so an approved custom tokenizer would have let an unapproved model's custom code run inside the quantization subprocess. The subprocess now takes --trust-remote-code-tokenizer for the processor load and keeps --trust-remote-code for the model loads, matching the torchao path's separate model_trust / tok_trust. _loaded_via_remote_code now also walks processor components (tokenizer, image_processor, feature_extractor, video_processor), so an approved custom tokenizer held inside a built-in ProcessorMixin keeps its trust on the export reload instead of failing with trust_remote_code=False. The walk is a bounded BFS with a seen set so wrapper cycles terminate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-05 05:16:39 -07:00
Daniel Han	ec4c044e70	Pin llm-compressor auto-install to a vetted version range (#6778) ... * Pin llm-compressor auto-install to a vetted version range install_llm_compressor() auto-installs llm-compressor on first use of an FP8/FP4 compressed export when it is not already present. The install command used the bare package name, so pip resolved to whatever the configured index served; a compromised, dependency-confused, or inflated-version ("999.0.0") release could then run under the Unsloth process at install and import time. Bound the automatic install to a vetted range (_LLM_COMPRESSOR_SPEC = "llmcompressor>=0.8.0,<0.13"), which the oneshot / QuantizationModifier API this uses supports, so pip can no longer jump to an arbitrary future or inflated version. An already-installed newer llm-compressor is still used as-is (the import short-circuits), so this only constrains the auto-install, never a user's own install. Add UNSLOTH_DISABLE_LLM_COMPRESSOR_AUTOINSTALL=1 to forbid the automatic install entirely and require a manual, vetted install, for locked-down or air-gapped environments. Update the manual-install hints to the pinned spec. Add tests/saving/test_llm_compressor_install_pin.py: static (ast) guards that the spec stays a bounded pin, that the install command never passes an unpinned llmcompressor literal, and that the opt-out env gate is evaluated before any install runs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Loosen llm-compressor auto-install ceiling to <1.0 so new models still export The earlier <0.13 ceiling was too tight: brand-new architectures (for example Qwen3_5ForConditionalGeneration / qwen3_5, gemma-4 MoE) can require a newer llm-compressor, and _unsloth_save_compressed_tensors already fails with "requires a newer llm-compressor" when a scheme is unavailable. Capping the auto-install at 0.12 would block getting that newer release and break compressed export for new models. Widen to llmcompressor>=0.8.0,<1.0. pip still auto-installs the latest 0.x (where new-architecture support lands), while the <1.0 ceiling continues to block a jump to an inflated-version ("999.0.0") or 1.0+ dependency-confusion release. An already-installed newer llm-compressor is still used as-is (the import short-circuits), and UNSLOTH_DISABLE_LLM_COMPRESSOR_AUTOINSTALL still forbids the automatic install entirely for locked-down environments. * Lower llm-compressor floor to 0.6.0 so supported old torch still resolves The >=0.8.0 floor conflicts with the torch this install pins in its constraints file. Unsloth supports torch>=2.4, but llm-compressor 0.7.0+ require torch>=2.7 (0.10+ need >=2.9, 0.12+ need >=2.10). On a supported torch 2.4-2.6 box pip then has no candidate in [0.8.0, 1.0) and FP8/FP4 export fails before quantization. Lower the floor to 0.6.0 (its metadata only needs torch>=1.7), which never conflicts with any supported torch. pip still prefers the newest compatible release, so modern torch continues to get the latest 0.x (0.12.0). The <1.0 ceiling that blocks an inflated-version supply-chain jump is unchanged. Add a regression test asserting the floor stays <= 0.6.0. * Cap llm-compressor auto-install ceiling to a vetted minor (<0.13) A bare <1.0 ceiling still admits any 0.x, so an inflated "0.999.0" served by a compromised or misconfigured index would win pip's highest-version selection -- the same dependency-confusion this pin is meant to block. Cap the ceiling to the current vetted minor (<0.13) so that jump is blocked; bump it deliberately, after vetting, when a newer llm-compressor is needed (e.g. for a brand-new architecture scheme). Current new models are unaffected: 0.12.0 is < 0.13 and supports them. The 0.6.0 floor (torch>=1.7 compatible) is unchanged, so resolution still works across Unsloth's whole supported torch range (2.4 -> 0.6.0 ... 2.12 -> 0.12.0). Add a regression test asserting the ceiling admits the current vetted release but blocks an inflated 0.x and the next major. * Trim comments in the llm-compressor pin (comment-only, no code change) * Cap llm-compressor auto-install to the exact vetted patch (<=0.12.0) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-01 04:48:38 -07:00
Daniel Han	8cc05ac89c	Reduce comments across recent fixes (#6776) ... Condense the verbose comments and docstrings added by the recent chat template, GPT-OSS detection, PEFT tensor-parallel, and Studio inference proxy fixes. Comments and whitespace only; no code changes.	2026-06-30 23:13:36 -07:00
Daniel Han	9369dd47e6	Add FP8/FP4 compressed export to save_pretrained_merged (#6706) ... * Add FP8/FP4 compressed export to save_pretrained_merged Adds compressed-tensors export (for vLLM) to save_pretrained_merged / push_to_hub_merged via llm-compressor, alongside the existing lora / merged_16bit / merged_4bit / gguf / torchao paths: model.save_pretrained_merged("model", tokenizer, save_method="fp8") Supported save_method values: fp8 (FP8_DYNAMIC), mxfp4, nvfp4 (W4A4) and mxfp8. The LoRA is merged to 16bit at save_directory, then a quantized checkpoint is written to save_directory + "-<fmt>". nvfp4 needs a small calibration set (defaults to ultrachat, overridable via calibration_dataset). Notes: - llm-compressor is installed lazily on first use, pinning the current torch and transformers via a constraints file so they are not upgraded (a plain install pulls transformers>=5 and breaks Unsloth). - Quantization runs in a separate process (unsloth/_compressed_quantize.py, launched by file path) so Unsloth's transformers attention patches do not interfere with the forward llm-compressor runs during calibration, mirroring how GGUF export shells out to llama.cpp. - mxfp8 needs a newer llm-compressor (transformers>=5); it is recognised and raises a clear error until that stack is available. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: main-process guard, calibration subsampling, tokenizer + dtype handling - Route the 16bit merge through unsloth_generic_save for both LoRA and full finetuned models, so non-PEFT models are written in 16bit consistently instead of saving the original (possibly quantized) weights directly. - Honor is_main_process: only the main process quantizes and writes the compressed output, so distributed ranks do not race on the same dirs. - Subsample an in-memory calibration Dataset before save_to_disk so large training sets are not fully copied to a temp dir. - Tolerate a missing tokenizer in the converter (data-free exports); still require one for calibration based schemes. - Open config.json via a context manager in both files. - Drop the redundant nvfp4 entry from the unsupported-name check (fp4 covers it). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add direct LoRA to GGUF export and harden FP8/FP4 compressed export - Run llm-compressor install and scheme check before the 16bit merge so unsupported schemes (e.g. mxfp8) fail fast without writing a checkpoint - Only the main process installs, merges, quantizes and uploads; isolate hub pushes to a temp dir and clean all temp dirs in a finally - Forward standard save kwargs (state_dict, max_shard_size, ...) to the merge - Fall back to the first dataset split for Hub calibration ids - Export LoRA adapters to GGUF via convert_lora_to_gguf.py: modernize save_pretrained_ggml/push_to_hub_ggml and add save_method="lora" to save_pretrained_gguf/push_to_hub_gguf; resolve base from the adapter config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix LoRA GGUF shell-injection test and compressed export trailing-slash path - Update tests/saving/test_save_shell_injection.py for the new delegation: the LoRA to GGUF conversion now lives in _unsloth_save_lora_gguf, so assert it passes argv as a list with no shell=True and that the legacy ggml wrappers delegate to it instead of calling subprocess.Popen directly - Normalize the local save_directory before building the "<dir>-<fmt>" sibling so a trailing slash no longer nests the compressed output inside the 16bit dir * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Polish FP8/FP4 and LoRA GGUF export after review - Warn (not silently downgrade) when an explicit quantization_method is not a valid LoRA GGUF outtype; default stays f16 - Correct the inference hardware note: MXFP8 is 8-bit (cc >= 8.9), only FP4 needs Blackwell for full activation quantization - Document that a local fp8/fp4 save keeps the 16bit merge at save_directory and writes the quantized checkpoint to save_directory + "-<fmt>" * Use sequential calibration pipeline and validate Hub access early - nvfp4 calibration no longer forces the memory-hungry "basic" pipeline. The quantization runs in a clean subprocess, so llm-compressor's default sequential pipeline (layer-by-layer onloading) works and lets large models that do not fit at once still calibrate; fall back to "basic" only if tracing fails - For push_to_hub compressed exports, create/validate the repo up front so a bad token or denied repo fails before the merge and quantization instead of after * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden compressed export: explicit sequential pipeline, base-tokenizer calibration, GPU memory - nvfp4 calibration now passes pipeline="sequential" explicitly (layer-by-layer onloading) instead of relying on the inferred default, with a "basic" fallback - Calibration datasets with a messages column no longer require a chat template: base / non-chat tokenizers fall back to concatenating message contents - Free the in-memory model's CUDA memory before the quantize subprocess loads its own copy from disk (best-effort, single-device non-quantized only; restored afterward), so a single GPU need not hold two copies at once - Create the calibration temp dir in the system temp location instead of next to the save directory, avoiding stray dirs in the workspace * Free the failed calibration model before the basic-pipeline retry In the sequential -> basic NVFP4 fallback, release the partially-processed model and clear the CUDA cache before loading a fresh copy, so the retry does not transiently hold two model copies on the GPU. * Harden calibration data handling and compressed-export edge cases - Calibration messages without a chat template now handle multimodal (list) content, None content, and null message rows instead of crashing on join - Raise a clear error when the calibration dataset is empty after subsampling - Reset llm-compressor's global session before freeing the model in the sequential -> basic NVFP4 fallback, so the old model is actually released - LoRA GGUF export accepts a single-element list quantization_method - Attach datasets metadata to the pushed repo on compressed hub exports - Warn (instead of silently) if the model cannot be restored to its device - Raise a clear error if the LoRA base model id cannot be determined * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Handle DatasetDict calibration, MoE routers, and MTP models in compressed export - Reduce an in-memory DatasetDict calibration set to a single split before row subsampling, so save_to_disk does not copy every split to the temp dir - For MoE models, keep the router/gate unquantized and pass moe_calibrate_all_experts so every expert is calibrated - Warn when a model carries MTP / speculative-decoding tensors that the compressed export does not include * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Support many more compressed-tensors schemes and address review - Expand save_method to cover the full set of compressed-tensors preset schemes: FP8 (dynamic/static/block), INT8, W8A8, W8A16, W4A16(+asym), W4A8, W4AFP8, MXFP4(+A16), NVFP4(+A16), plus the gated MXFP8; calibration is used only for the static-activation schemes (FP8 static, NVFP4) - Broaden the near-miss save_method error to cover int/w-prefixed names - MoE: also keep the Qwen shared-expert gate unquantized - Strip non-model-input columns from already-tokenized calibration data so the collator does not choke on a leftover messages column - Forward the Hub token to the LoRA converter and the quantize subprocess so gated/private base models and calibration datasets work without a global login * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Collapse compressed-tensors export help line so ruff-format converges The print line in print_quantization_methods needed two ruff-format passes to reach a fixpoint (merge implicit string concat, then collapse the single-arg print). pre-commit.ci applies one pass per run, so it kept reformatting. Land the converged single-line form directly. * Add CPU-only regression tests for the export API Cover all export paths without a GPU, for slow CPU-only CI: - pure-function checks of the compressed-tensors scheme registry and save_method normalization (aliases, calibration flags, near-miss errors) - AST checks that every merged saver dispatches compressed export, the GGUF savers expose the lora branch, torchao routes PTQ/QAT, the public methods stay attached, and the export subprocesses remain shell-safe (argv list, sys.executable, no shell) - monkeypatched dispatch checks that fp8/nvfp4/merged_16bit, the LoRA-GGUF outtype resolution, and torchao PTQ/QAT reach the right helper with the right arguments * Run the CPU-only export tests in consolidated CI tests/saving is --ignored by the Repo tests (CPU) job, so the new GPU-free export tests are added by path to consolidated-tests-ci.yml (collection sanity + Bucket-A run), alongside the existing CPU saving tests, so they actually execute on CPU CI. * Add GPU GGUF export + llama-cli inference smoke test tests/saving/test_gguf_export_and_inference.py: skipif no CUDA. Trains a tiny phrase-imprinting LoRA, exports a full-model q8_0 GGUF (merge -> convert_hf_to_gguf -> llama-quantize), asserts a valid GGUF (magic + size), and - when a llama-cli binary is available - runs one bounded generation (byte cap + watchdog kill) and asserts the trained phrase round-trips through HF -> GGUF -> quantize -> inference. The llama-cli step skips gracefully since the export only builds llama-quantize. * Fix variant mismatch in compressed (FP8/FP4) export save_pretrained_merged(..., save_method=fp8/nvfp4, variant=...) forwarded the variant into the intermediate 16bit merge, so Transformers wrote variant-named shards (model.<variant>.safetensors). The converter subprocess then reloaded that directory with the default weight filenames, so the compressed export failed after doing the merge. Pop the variant out of the intermediate merge (internal staging that the subprocess reloads with default names) and forward it via --variant so it is applied to the final compressed checkpoint instead. Add a CPU AST guard for the contract. * Harden export paths from review - install_llm_compressor: fall back to uv pip when this interpreter has no pip seeded (uv-created/relocatable venvs), instead of failing with No module named pip. - LoRA GGUF export: if convert_lora_to_gguf.py is missing (a prebuilt or reused CWD llama.cpp install carries binaries but not the converter script), force a dedicated source checkout that ships it. - push_to_hub_gguf(save_method=lora): return on non-main ranks, matching the local save_pretrained_gguf lora branch, so only rank 0 converts/uploads. - compressed export VLM detection: require a vision_config or a ForVisionText2Text architecture; a bare *ForConditionalGeneration also matches text seq2seq models (T5/BART/Whisper) and is no longer treated as a VLM on its own. - GGUF GPU smoke test: drop SFTConfig(max_length=1024), which raises under newer TRL padding-free training; length enforcement is not needed here. * Add imatrix option to GGUF export, enabling IQ low-bit quants save_pretrained_gguf / push_to_hub_gguf gain imatrix_file: None -> no imatrix (unchanged) '/path' -> pass to llama-quantize --imatrix (a *.gguf_file is renamed to *.gguf) True -> download the upstream unsloth/<base>-GGUF imatrix (imatrix_unsloth.dat or .gguf_file), raising a clear error if none exists An importance matrix unlocks the IQ low-bit quants (iq2_xxs, iq4_xs, ...), which were hard disabled before. They are gated: requesting one without an imatrix raises a clear error. - _resolve_imatrix_file resolves path/True (PEFT base first, normalized via get_model_name, derives unsloth/<base>-GGUF, copies out of the HF cache before renaming *.gguf_file). - IMATRIX_QUANTS registry replaces the old commented-out IQ entries; save_to_gguf accepts a resolved imatrix and threads it into the quantize calls. - The --imatrix flag is emitted by unsloth_zoo's quantize_gguf (companion change). save.py fails fast with an upgrade hint if the installed unsloth_zoo lacks the imatrix kwarg. Tests: tests/saving/test_imatrix_export.py (CPU: resolution, repo derivation, IQ gate, --imatrix wiring) wired into CI; tests/saving/test_gguf_export_and_inference.py extended with GPU iq2_xxs/iq4_xs export + inference. Verified end to end on Llama-3.2-1B: imatrix auto-downloaded, iq2_xxs/iq4_xs exported and run via llama.cpp. Note: requires the companion unsloth_zoo quantize_gguf imatrix change. * Address imatrix/compressed review feedback: unsloth org GGUF repo, fail-fast, calibration split - imatrix auto-resolve (imatrix_file=True): derive the upstream repo as unsloth/<base>-GGUF instead of <org>/<base>-GGUF, so official bases (e.g. meta-llama/Llama-3.1-8B-Instruct) find the matching Unsloth GGUF imatrix repo rather than failing on a nonexistent meta-llama/...-GGUF. - Resolve/validate the imatrix before the 16-bit merge in save_pretrained_gguf, so a bad path or an unavailable upstream imatrix fails fast instead of after a long, multi-GB merge. - Compressed calibration: when a Hub dataset has no "train" split, resolve the first split name and slice it, instead of materializing the whole dataset just to take num_samples rows. Keeps the original materialize-then-subselect path as a last resort. Tests: add unsloth/<base>-GGUF mapping for an official base id, and create the imatrix file in the quantize_gguf flag test (quantize_gguf now validates the imatrix exists). --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-30 03:40:16 -07:00
Vineeth Sai	677ec0cc20	Fix gpt-oss detection in save: config.architectures is a list, not a string (#6711)	2026-06-29 02:44:00 -03:00
Daniel Han	e83d4ae072	Windows installer: fix DiskPart UAC mid-install, drive-root cache, and spurious unsloth.exe rename warning (#6296) ... * Windows installer: fix DiskPart UAC, drive-root cache, spurious rename warning, CPU-base messaging amd-smi gate (DiskPart UAC mid-install): the AMD torch wheel ships hipInfo.exe inside the venv, and the bitsandbytes fix prepends that venv Scripts dir to PATH. shutil.which("hipinfo") then found it and flipped _amd_smi_allowed() to True, so the post-install AMD probe fell through to `amd-smi list` (the venv hipInfo failed to report gcnArchName, which is why the arch came from the GPU-name table) and amd-smi elevated, popping the DiskPart UAC. Fix: a hipinfo resolved inside the active venv (sys.prefix) is the torch-wheel binary, not a HIP SDK, and must not open the gate. Mirrored in install_python_stack.py, install_llama_prebuilt.py, and backend utils/hardware/amd.py (the runtime VRAM poller had the same latent prompt). TORCHINDUCTOR_CACHE_DIR: move from C:\tc to <StudioHome>\TORCHINDUCTOR_CACHE_DIR so the inductor/Triton cache lives under the user's Studio home, not the system drive root. Long paths are already enabled above so deep inductor paths still fit. unsloth.exe rename: skip the rename (and its "pip may fail with WinError 32" warning) when SKIP_STUDIO_BASE=1. In the install.ps1 flow base packages are not reinstalled, so unsloth.exe is never rewritten; the self-rename only failed because setup runs via unsloth.exe (the running launcher holds its own file). The 'studio update' flow still attempts it. CPU PyTorch messaging: clarify that the CPU base is temporary and setup replaces it with GPU ROCm wheels, and print an explicit "GPU ROCm PyTorch installed" line after the AMD wheels land, so the log makes clear the final install is GPU-accelerated. Adds two regression tests covering the venv-internal vs external hipInfo gate. Verified end-to-end on a Strix Halo box (Radeon 8060S / gfx1151): install.ps1 --local from this branch completed exit 0 with no DiskPart prompt, no rename warning, the cache under the Studio home, and "GPU ROCm PyTorch installed (gfx1151)"; Studio then booted and detected "ROCm (HIP 7.13.99004) -- AMD Radeon 8060S Graphics". * Windows installer: drop the unreliable unsloth.exe rename and its WinError 32 warning setup.ps1 used to rename the running unsloth.exe out of the way before the base-package upgrade so pip could replace it. That rename never actually worked: setup runs *via* unsloth.exe, so renaming our own running uv-trampoline launcher failed with a sharing violation (WinError 32) and only printed a scary 'could not rename unsloth.exe; pip may fail with WinError 32' warning on every Windows install and update. It also was not needed. pip tolerates a running/locked console-script .exe: it moves the old one aside and writes the new one. The base upgrade routes through pip on Windows, so the upgrade succeeds (or, in the install.ps1 flow with SKIP_STUDIO_BASE=1, the base is not touched at all) and unsloth.exe is left intact either way. Removing the rename block and its failed-install restore block removes the false warning for all Windows devices in both the install and update flows. * Windows installer: gate venv-internal hipInfo.exe in PowerShell amd-smi probe; harden venv path checks Follow-up to PR #6296. - install.ps1 and setup.ps1: ignore the AMD torch wheel hipInfo.exe that lives inside the Studio venv when probing for a HIP SDK, so amd-smi no longer reopens the DiskPart UAC during install/update. Mirrors _path_inside_venv in the Python installers, which already do this. - amd.py, install_llama_prebuilt.py, install_python_stack.py: normcase the venv containment check (Windows paths are case-insensitive) and run the HIP_PATH/ROCM_PATH candidate through it too. - setup.ps1: fall back to a short TORCHINDUCTOR cache dir when long paths are unavailable, and create the dir wildcard-safely. - tests: isolate sys.prefix in the gate helper, add HIP_PATH/ROCM_PATH cases, and assert the PowerShell venv exclusion. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows installer: install ROCm PyTorch directly for a known AMD arch When the GPU arch is known (name-inferred from the GPU-name table) but ROCm could not be probe-verified (no HIP SDK, no amd-smi), the bootstrap installed a CPU PyTorch base that setup.ps1 then force-reinstalled as ROCm. The repo.amd.com wheels bundle their own runtime (no HIP SDK required), which setup.ps1 already relies on, so the CPU base was a pure wasted download/install. - Gate the ROCm index on a known arch, not only on probe-verified ROCm, so a mapped arch installs ROCm torch directly. Unmapped arches and no-GPU hosts still get CPU (unchanged). - Fall back to a CPU base if the ROCm-index install fails, so a transient repo.amd.com outage does not abort the install (setup.ps1 retries ROCm). - Correct the stale comment that claimed ROCm wheels need a confirmed HIP SDK. - Add a regression test for the arch-based gate. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Windows installer: correct the unsloth.exe rename-removal comment The comment claimed the base upgrade 'routes through pip on Windows' and that pip 'moves the old unsloth.exe aside, then writes the new one'. That is not what the code does. install_python_stack tries uv first; on a locked launcher uv aborts and falls back to pip, but the pip fallback strips --upgrade-package and base.txt lists only bare unsloth/unsloth-zoo, so pip finds them already satisfied and no-ops. The running unsloth.exe is left intact at its current version either way. Reword the comment to describe the real uv-first / pip-fallback-no-op behavior. No functional change. * Windows installer: close two gaps in the venv-internal hipinfo exclusion Review follow-up. The amd-smi/DiskPart gate could still reopen in two cases: - setup.ps1 ran the HIP probe long before $VenvDir is assigned, so without VIRTUAL_ENV (the `unsloth studio update` path) $venvRoots was empty and the venv-internal hipInfo.exe was not recognized. Seed the venv root from UNSLOTH_SETUP_PYTHON and the default Studio home too (both installers). - The HIP_PATH/ROCM_PATH candidate was accepted without the venv filter, so an env var pointing into the venv (AMD wheel) still set $HipSdkInstalled. Run Test-HipinfoIsVenvInternal on the candidate as well (both installers). Extend the PS gate test to assert both. Both .ps1 parse clean; install tests pass (the venv-internal / HIP probe coverage at 359 passed). * Windows installer: correct the CPU-base message for arches with no ROCm wheels After gating the ROCm index on a known arch, a mapped arch sets $ROCmIndexUrl and installs ROCm directly, so it no longer reaches the "temporary CPU base" branch. That branch is now reached only by a name-inferred arch with no ROCm wheels (e.g. RDNA2 gfx103X), where setup.ps1 does NOT install ROCm. The old text ("setup replaces it with GPU ROCm wheels ... the final install IS GPU-accelerated") was therefore always wrong there. Say plainly that PyTorch stays on CPU for this GPU. * Windows installer: seed the venv-internal hipInfo check from a custom Studio home Test-HipinfoIsVenvInternal seeded the venv root from VIRTUAL_ENV, VenvDir, the setup python, and the default %USERPROFILE% path only. A standalone `unsloth studio update` with a custom UNSLOTH_STUDIO_HOME (or STUDIO_HOME alias) and none of those set would not recognize the venv hipInfo on PATH, reopening the amd-smi/DiskPart gate. Seed the custom home too, in both installers, and assert it in the gate test. * Studio installer: resolve venv aliases and expand ~ in the hipInfo venv filter Two review points on the amd-smi/DiskPart UAC gate: 1. _path_inside_venv compared os.path.abspath of sys.prefix and the hipInfo path, which does not resolve symlinks, junctions, or 8.3 short names. A venv reached through an aliased path then fails the check, so its bundled hipInfo.exe is mistaken for an external HIP SDK and amd-smi runs (the DiskPart prompt this fix exists to suppress). Switch to os.path.realpath in all three copies (amd.py, install_llama_prebuilt.py, install_python_stack.py). 2. setup.ps1's early venv-internal hipInfo probe seeded the venv root from a custom Studio home (UNSLOTH_STUDIO_HOME / STUDIO_HOME) without expanding a leading ~, while the canonical resolver does. With a tilde form, [IO.Path]::GetFullPath kept the literal ~ relative to cwd, so the custom-home hipInfo escaped the filter and reopened the gate. Expand ~ in the probe the same way as the resolver. tests/studio/install/test_pr5940_followups.py: 30 passed (adds a symlink realpath case and a setup.ps1 tilde-expansion guard). * Studio installer: mirror the hipInfo venv filter and ROCm wheel pins into install.ps1 Follow-up review on the same install.ps1 paths: 1. install.ps1's venv-internal hipInfo probe (Test-HipinfoIsVenvInternal) seeded the venv root from a custom Studio home without expanding a leading ~, unlike the canonical resolver and setup.ps1. A tilde form left [IO.Path]::GetFullPath with the literal ~ (relative to cwd), so the custom-home hipInfo escaped the filter and reopened the amd-smi/DiskPart gate. Expand ~ in the probe, matching the setup.ps1 fix. 2. The AMD ROCm path installed torchvision/torchaudio bare while pinning torch to below 2.12. AMD's per-arch index publishes the companions independently and may ship torchvision 0.27 (for torch 2.12) before removing 0.26, so a bare resolve can pick an ABI-incompatible set and fall back to CPU. Add torchvision/torchaudio floor maps and pass the pinned specs, mirroring setup.ps1 and install_python_stack.py. 3. The ROCm-to-CPU fallback torch install used Invoke-InstallCommand (no retry), the only torch step in the file without it. Switch to Invoke-InstallCommandRetry so the recovery path survives a transient index failure. tests/studio/install/test_pr5940_followups.py: 33 passed (parametrized tilde check over both installers, a torch/companion floor-map parity test, and a CPU-fallback retry guard). * Studio installer: scan all PATH hipinfo so the venv copy can't shadow a real HIP SDK The amd-smi HIP-SDK probe used shutil.which("hipinfo") / Get-Command hipinfo, which return only the first hit on PATH. The AMD torch wheel ships hipInfo.exe inside the venv and the bnb fix (plus the Studio backend) prepend the venv Scripts dir to PATH, so that venv-internal copy lands first. When a real HIP SDK hipinfo sits later on PATH with HIP_PATH/ROCM_PATH unset, the first-hit probe stopped at the venv copy, treated it as "not a HIP SDK", and closed the amd-smi gate -- AMD users in that PATH-only SDK setup lost amd-smi telemetry and could fall back to CPU. Scan every PATH entry and keep the first hipinfo that is not venv-internal; only the venv copy is ignored, so the UAC/DiskPart suppression is unchanged. Applied to all three Python copies (install_llama_prebuilt.py, install_python_stack.py, backend/utils/hardware/amd.py) via a new _external_hipinfo_on_path helper, and both PowerShell callers (install.ps1, setup.ps1) now use Get-Command hipinfo -All filtered by Test-HipinfoIsVenvInternal. tests/studio/install/test_pr5940_followups.py: 36 passed (real-PATH scan tests, a shadow-regression test for the exact venv-first ordering, and a parity check that every Python copy uses the scanning helper). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio uninstallers: fix leftovers (false "removed", shared icon, llama lock) Auditing a dual native+WSL uninstall on a real device surfaced three leftovers: 1. uninstall.ps1 removed the data dir (which holds unsloth.ico) before the shortcuts that reference that icon, so Explorer's icon cache briefly held it open. Remove-Item -Recurse reported success yet left the locked file, and the dir was never re-attempted, so it orphaned with a false "removed" log. _RemovePath now verifies the path is actually gone (retrying transient locks) and reports honestly, and the data dir is re-swept after the shortcuts go. 2. install.sh writes a shared unsloth.ico to %LOCALAPPDATA%\Unsloth Studio for the WSL shortcut, but uninstall.sh never removed it, orphaning the icon (and dir) after a WSL uninstall. uninstall.sh now drops that icon and the dir when empty, in both the powershell.exe and drvfs-fallback paths. 3. ~/.unsloth/.llama.cpp.install.lock was never removed, so the rmdir of ~/.unsloth failed and the dir lingered. Both uninstallers now remove the lock. Verified by running both uninstallers on a real dual install: device fully clean (no install dirs, shortcuts, PATH/registry entries, shared icon, or lock left). * install.sh: auto-route Strix Halo WSL to an existing Ubuntu 24.04 ROCm-on-WSL is the GPU runtime for Strix Halo and only targets Ubuntu 24.04. When the installer runs in a newer default distro (e.g. 26.04) it cannot enable the GPU and silently falls back to CPU. If a 24.04 distro already exists, re-run the install there and stop in the current one so the GPU path is taken without the user having to know about the distro requirement. Runs before venv creation so the wrong distro is left untouched, guards against re-route loops via UNSLOTH_WSL_REROUTED, leaves a working ROCm distro alone (librocdxg present), and skips the GGUF-only / opt-out / non-Strix cases. When no 24.04 distro exists we keep today's behaviour: continue to CPU and print the `wsl --install Ubuntu-24.04` guidance, never auto-downloading a distro. Adds tests/sh/test_strixhalo_wsl_reroute.sh (hermetic: extracts the function, rewrites its paths to fixtures, mocks wsl.exe) covering the full decision matrix, wired into tests/run_all.sh. * uninstall.ps1: keep shared unsloth.ico for a surviving WSL shortcut A dual native+WSL install shares %LOCALAPPDATA%\Unsloth Studio\unsloth.ico: install.sh points the WSL shortcut's icon there while the native install owns the dir. The native uninstaller removed the whole dir unconditionally, so uninstalling native while keeping WSL left the WSL shortcut with a blank icon. The old code only avoided this when Explorer happened to hold the icon open, which is unreliable; on a real dual install the dir was deleted and the WSL shortcut went blank. _RemoveDataDirKeepingWslIcon now scans the Start Menu + Desktop for a surviving "Unsloth Studio (WSL ...).lnk" and, if found, removes everything in the data dir except unsloth.ico (keeping the dir) instead of deleting it; with no WSL shortcut it removes the dir as before. uninstall.sh still drops the icon and the empty dir when WSL itself is uninstalled, so every uninstall order ends clean. Adds tests/studio/test_uninstall_dual_install_icon.ps1 (AST-extracts the helper and runs it against a temp dir with controlled shortcut dirs) covering the dual, native-only, empty, and missing-dir cases, wired into the windows-inference smoke workflow. Verified on a real dual install: native uninstall now keeps unsloth.ico and the WSL shortcut's icon stays intact. * installer: condense AMD/ROCm code comments (no behavior change) Tighten the comments added for the Strix Halo native+WSL installer work so they are shorter and clearer without losing intent: the venv-internal hipInfo amd-smi gate, the ROCm torch/companion floor maps, the WSL 24.04 reroute, and the dual-install uninstall icon handling. Comment-only; code paths unchanged. 107 insertions, 166 deletions across 11 files. * install.sh: run the Strix Halo WSL reroute before any STUDIO_HOME write The reroute fired after mkdir -p "$STUDIO_HOME" and the legacy-venv migration, so rerouting 26.04 -> 24.04 left an empty ~/.unsloth/studio stub in the origin distro (and ran venv migration in the distro about to be abandoned). Move the reroute ahead of the venv section so the origin distro is left untouched, matching the function's own comment. Behavior is identical on every non-reroute path. * installer: fix ROCm CPU-fallback, hipinfo gate edge cases, uninstall icon, WSL 22.04 - install.ps1: clear $ROCmIndexUrl/$ROCmTorchFloor after the CPU fallback so the flavor-repair block does not retry the failed ROCm index and abort the install; pin the ROCm companion specs ($visionSpec/$audioSpec) in the repair path too. - install.ps1 + setup.ps1: skip a bare drive root in Test-HipinfoIsVenvInternal so a non-venv UNSLOTH_SETUP_PYTHON does not match the whole drive; iterate HIP_PATH/HIP_PATH_57/ROCM_PATH and take the first non-venv hipinfo. - amd.py, install_llama_prebuilt.py, install_python_stack.py: strip surrounding quotes from PATH entries before probing for hipinfo. - install.sh: pipefail the WSL reroute curl|sh; do not reroute supported Ubuntu 22.04. - uninstall.sh: keep the shared unsloth.ico while any Unsloth shortcut (native or another WSL distro) still references it, in both the powershell and drvfs paths. - tests: regression coverage for all of the above. * installer: forward reroute options, guard ROCm bootstrap, harden hipinfo gate - install.sh: forward the caller's --package/--python/--verbose/--tauri and a custom UNSLOTH_STUDIO_HOME into the WSL reroute (was a bare default install); bail on --local; run the reroute BEFORE dependency/uv install so the origin distro is left untouched; set UNSLOTH_SKIP_ROCM_WSL_SETUP after a failed reroute so the later ROCm-on-WSL bootstrap does not install into the unsupported origin distro. - install.ps1 + setup.ps1: Get-Command hipinfo -CommandType Application so only real executables match (not an alias/function named hipinfo). - uninstall.ps1: guard $env:APPDATA when building the default shortcut search dirs. - tests: cover option forwarding, --local bail, the bootstrap guard, and the gate change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * installer: guard origin ROCm bootstrap on every CPU-only fallback; harden ~ expansion WSL reroute: the no-wsl.exe, no-24.04-target and --local fallbacks all tell the user the install continues CPU-only, but only the failed-reroute branch set UNSLOTH_SKIP_ROCM_WSL_SETUP=1. The later _maybe_bootstrap_rocm_wsl gate keys off that flag, so the other three branches could still install ROCm into the unsupported origin distro (e.g. 26.04). Set the skip guard on all of them. Forward UNSLOTH_ROCM_WSL_AUTO into the reroute so a Tauri/consented GPU bootstrap carries through to the rerouted 24.04 child instead of dropping to the prompt path. install.ps1/setup.ps1: guard the venv-probe ~ expansion on a non-empty $env:USERPROFILE so Join-Path does not throw on a profile-less service account. Tests: add no-wsl.exe and UNSLOTH_ROCM_WSL_AUTO reroute cases, the USERPROFILE guard assertion, and route shell-test fixtures through a single trap-cleaned root. * installer: pin + soften Windows ROCm Python repair, reroute to 22.04, harden gates install_python_stack.py: the Windows AMD ROCm repair in _ensure_rocm_torch() installed bare torch/torchvision/torchaudio via the fatal pip_install -- the same asymmetry already fixed on the PowerShell side. A transient repo.amd.com failure could abort the whole install even after install.ps1/setup.ps1 fell back to CPU. Pin companions per-arch (gfx120X/Strix -> the rocm7.2 trio, mirroring the PS floor maps) and make the retry nonfatal: keep the existing build and let the user re-run update to retry ROCm, so the chain install.ps1 -> setup.ps1 -> stack stays CPU-safe. install.sh: reroute now targets an installed Ubuntu 24.04 OR 22.04 (24.04 preferred); both are AMD-supported for ROCm-on-WSL, matching the leave-alone set, so a box with only 22.04 reaches the GPU instead of staying CPU-only. install.ps1/setup.ps1: a bare ~ for UNSLOTH_STUDIO_HOME left an empty Join-Path child (PS 5.1 throws); fall back to USERPROFILE directly and only join a real remainder. _path_inside_venv (amd.py + both installers): guard a root-dir sys.prefix so commonpath can't classify every path on the drive as venv-internal (defensive; venv never at root). uninstall.sh: guard an empty LOCALAPPDATA in the PS-interop icon cleanup (mirror APPDATA). Tests: add 22.04-target reroute cases, Windows ROCm pin+nonfatal coverage (text + behavioral), root-dir guard coverage, and bare-~/LOCALAPPDATA guard assertions. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * install.sh: match WSL reroute target by exact distro name, not substring The 24.04/22.04 reroute target was chosen with grep -F (substring), so a custom distro such as 'Ubuntu-24.04-test' (with no exact Ubuntu-24.04) was picked as the target; the later 'wsl -d Ubuntu-24.04' then fails and the Strix Halo install stays CPU-only. Match whole lines (grep -ixF) and reuse the matched name so only a real Ubuntu-24.04/22.04 is targeted. Adds substring-rejection + exact-vs-custom tests. * install.sh: keep the WSL reroute target to Ubuntu 24.04 (helper-supported only) The ROCm-on-WSL bootstrap (scripts/install_rocm_wsl_strixhalo.sh) dies on any VERSION_ID other than 24.04 and pins the noble repo, so treating 22.04 as GPU-supported let the parent report a successful reroute while the child fell back to CPU. Drop 22.04 from the supported set and the reroute target list; 24.04 stays the sole target (keeping the exact whole-line distro match). An already-working ROCm on any other version is still left alone by the librocdxg check above. tests: reroute 22.04 cases updated to the 24.04-only behavior; make the "no wsl.exe" case hermetic so a real host wsl.exe can't leak in on dev boxes; stop the tauri exit-order check from mis-flagging the reroute helper's [ "$TAURI_MODE" = true ] && ... --tauri one-liner. * installer: tighten comment wording across the Strix Halo install/uninstall paths Condense the verbose multi-line comment blocks (amd-smi hipinfo gate, ROCm torch install + CPU fallback, WSL reroute, uninstall icon-keep) into fewer, clearer lines. Comments and a few docstrings only; no code, logic, or behavior change. Verified with bash -n, the PowerShell parser, and ast.parse, and the installer test suite still passes. * add AGPL-3.0 SPDX headers to the .sh/.ps1 scripts missing them Every shell and PowerShell script under the Studio/installer surface now carries the standard SPDX-License-Identifier: AGPL-3.0-only + copyright header (after the shebang where present): the installer (install.sh, install.ps1), build.sh, the .github and src-tauri scripts, the installer test suite, and the moe kernel test. Header-only, line endings preserved; bash -n, the PowerShell parser, and the installer tests all pass. * installer: drop the duplicate AGPL header from install.sh and install.ps1 Both already carry an SPDX-License-Identifier: AGPL-3.0-only header below their usage comment block; the prior header pass added a second one at the top because it only scanned the first few lines. Remove the duplicate so each file keeps a single original header. * installer: force-reinstall CPU fallback torch; propagate Tauri NEED_SUDO from reroute install.ps1/setup.ps1: when the AMD ROCm wheel install fails and we fall back to a CPU base, force-reinstall the torch/vision/audio triplet. A failed ROCm install can leave an unpinned ROCm torch (e.g. 2.10.0+rocm on gfx110X/gfx90a) that still satisfies the CPU torch>=2.4,<2.11.0 range, so without --force-reinstall uv keeps the ROCm build and only swaps the companions -- a mismatched venv the flavor-repair block won't fix. setup.ps1 scopes the forced reinstall to the ROCm-fallback path () so the genuine CPU-only install stays fast. install.sh: the Strix Halo WSL reroute treated every nonzero child exit as a reroute failure and fell back to CPU. In --tauri mode the child uses exit 2 ([TAURI:NEED_SUDO]) to ask the desktop app to elevate for the target distro; capture the child's exit code and propagate exit 2 in Tauri mode (the child already printed the NEED_SUDO line) instead of masking it. CLI mode still falls back to CPU on a generic failure. Tests: reroute Tauri exit-2 propagation (and non-Tauri CPU-fallback) cases; run_func now preserves the child exit code; force-reinstall assertions for both PowerShell installers. Note: codex's _rr_q apostrophe finding is a false positive -- the helper already emits POSIX-correct 'O'\''Brien' and round-trips under both sh and bash. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * setup.ps1: fix $cpuForce array collapse in the ROCm->CPU torch fallback An if-expression assignment ($cpuForce = if ($ROCmCpuFallback) { @("--force-reinstall") }) collapses the single-element array to a scalar string, so @cpuForce splatting enumerated it character-by-character into broken single-letter args (- - f o r c e ...), which made uv/pip reject the install and aborted the whole Studio setup on the AMD ROCm->CPU fallback path. Build $cpuForce as a real array assigned outside the if-expression so the splat passes a single --force-reinstall arg. Genuine CPU-only installs stay fast (empty array, no flag). Test now asserts the array-build form and rejects the if-expression form. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * uninstall: remove the isolated Node.js runtime (~/.unsloth/node) The isolated Node.js runtime (install_node_prebuilt.py, added with the managed-Node change) installs to ~/.unsloth/node in default mode -- a sibling of studio, so deleting <studio> leaves it behind (~200MB orphaned after uninstall). Both uninstallers already remove the other default-mode siblings (llama.cpp/.cache/.staging); add node alongside them. uninstall.ps1 also adds it to the handle-lock sweep so a held node.exe can't block the delete. Env/custom mode nests node under the custom root, removed with that root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-22 03:09:08 -07:00
Daniel Han	f89c829fcf	Fix save crash for legacy list-form _tied_weights_keys (NemotronH) (#6540) ... * Fix save crash for legacy list-form _tied_weights_keys (NemotronH) transformers >= 5 save_pretrained reads module._tied_weights_keys.keys(), which raises 'list' object has no attribute 'keys' for modules that still declare the attribute as a list (e.g. NemotronH backbone.layers.N.mixer.*_proj), crashing GGUF export and merged saves part-way through. Coerce any legacy list/tuple _tied_weights_keys into the dict form transformers 5.x expects, mapping each key to itself. Only the keys are read (as dedup patterns) so behaviour is preserved, and older transformers that iterate the attribute directly see the same keys. The helper is idempotent and best-effort so a save never fails over it. Called from unsloth_save_model, unsloth_save_pretrained_gguf and unsloth_generic_save after tokenizer patching. Adds version-independent unit tests covering list/tuple coercion, dict and None/empty pass-through, idempotency and odd-object tolerance. * Coerce empty/set _tied_weights_keys too transformers only skips _tied_weights_keys when it is None, so an empty list, tuple or set still reaches .keys() and raises the same AttributeError. Coerce every non-dict container (including the empty case and sets) to a dict, and add tests for empty/set inputs. * Tighten comments in tied-weights save fix * Scope tied-weights-keys coercion to the save call Coercing legacy list-form _tied_weights_keys to {k: k} fixed the transformers 5 save crash, but persisted a self-mapping on the live model. transformers 5 re-ties from the dict's values, so a later resize/re-tie would no-op the tie instead of pointing the output weights back at the input embeddings. Replace the in-place mutation with a decorator that coerces before the save and restores the originals afterwards (including on exception), so the save sees the dict form transformers needs while the model keeps its original tie metadata. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim comments to be more succinct --------- Co-authored-by: Daniel Han <michaelhan2050@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-22 02:11:46 -07:00
Daniel Han	a6dc10dad2	Reduce and tighten comments and docstrings across the test suite (#6429) ... * Reduce and tighten comments and docstrings in tests Shorten verbose comments and docstrings across the test suite without changing any test logic. Remove narration that restates the next line, collapse long module and test docstrings to a single line, and drop banner separators. Keep regression context (issue and PR references, run ids), skip reasons, mocking and timing rationale, license headers, lint and type directives, and commented-out code. Comments and docstrings only: an AST signature check confirms no code, assertions, or string literals changed, and the suite byte-compiles cleanly. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-18 01:07:09 -07:00
dylanschroers	51f1c8732d	fix: decode subprocess output as UTF-8 in save.py on Windows (#6218) ... * Fix UnicodeDecodeError on Windows reading subprocess output in save path On Windows the default text encoding is the locale code page (cp1252), not UTF-8. The text-mode subprocess calls in save.py (text=True / universal_newlines=True) set no explicit encoding, so they decode llama.cpp / Ollama output with cp1252. When a child process emits a byte undefined in cp1252 -- e.g. 0x9d, which appears inside the UTF-8 encoding of common punctuation / box-drawing glyphs and in non-ASCII file paths -- the read raises UnicodeDecodeError and aborts GGUF export. Add encoding="utf-8", errors="replace" to all 8 text-mode subprocess calls. errors="replace" also avoids silent mojibake for inputs whose bytes happen to be valid-but-wrong in cp1252. Add tests/saving/test_save_subprocess_utf8_encoding.py: - an AST drift detector asserting every text-mode subprocess call in save.py pins encoding="utf-8" (runs without importing torch/unsloth_zoo) - a behavioural test reproducing the cp1252 failure and the utf-8 fix Relates-to: #2660 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-06-12 01:31:31 -07:00
Daniel Han	187144d4e7	Reduce and tighten code comments and docstrings repo-wide (#6095) ... Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.	2026-06-08 23:09:51 -07:00
Datta Nimmaturi	b30e2b4b15	Merge qwen35_export CI fixes ... Merged latest main, resolved save.py/KTO test conflicts, fixed TRL/GRPO KTO drift	2026-06-08 22:02:05 +05:30
Daniel Han	3ce187da02	Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) ... Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.	2026-06-08 04:24:13 -07:00
Anmol Mishra	4e9d772d36	fix: preserve tokenizer eos token on merged saves (#5451) ... Preserves the source tokenizer's eos_token in tokenizer_config.json after merged saves so runtimes such as vLLM read the correct stop token. Centralized inside the patched tokenizer save_pretrained so all save paths (merged_16bit, GGUF, torchao, push_to_hub) benefit, with filename_prefix support. Fixes #5386	2026-05-17 06:44:23 -07:00
luo jiyin	06ed94da0d	chore: fix typo cleanup across tests and backend strings (#5152) ... * chore: fix typos in studio/backend/routes/models.py * chore: fix typos in tests/saving/non_peft/test_mistral_non_peft.py * chore: fix typos in tests/saving/non_peft/test_whisper_non_peft.py * chore: fix typos in tests/saving/vision_models/test_index_file_sharded_model.py * chore: fix typos in tests/saving/vision_models/test_push_to_hub_merged.py * chore: fix typos in tests/saving/vision_models/test_save_merge_qwen2.5vl32B_model_ocr_benchmark.py * chore: fix typos in tests/saving/vision_models/test_save_merge_vision_model_ocr_benchmark.py * chore: fix typos in unsloth/import_fixes.py * Split: keep only 6 file(s) --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>	2026-04-24 12:51:27 +01:00
Octopus	72c1c3b254	fix: patch CONTROL type for special tokens in sentencepiece GGUF export (#5080) ... * fix: patch CONTROL type for special tokens in sentencepiece GGUF export (fixes #5070) When converting a Gemma 3 fine-tune to GGUF via save_pretrained_gguf, tokens like <start_of_turn> (id=105) and <end_of_turn> (id=106) are already present in the sentencepiece model but are typed as NORMAL (1) instead of CONTROL (3). llama.cpp only recognises CONTROL tokens when parse_special=True is active, so these tokens get BPE-split during chat inference and the model produces garbage output. fix_sentencepiece_gguf now reads tokenizer.json's added_tokens list and, for any token with "special": true whose ID falls within the existing sentencepiece vocabulary, updates its type from NORMAL to CONTROL before writing the patched tokenizer.model to disk. The same CONTROL type is also applied when new tokens are appended for the out-of-range case, so both code paths are consistent. * Wire fix_sentencepiece_gguf into tokenizer save path and guard np.diff - save.py: call fix_sentencepiece_gguf inside unsloth_tokenizer_save_pretrained after _preserve_sentencepiece_tokenizer_assets. The helper was previously unreferenced in the repo, so the PR's CONTROL-type patch never actually ran during save_pretrained_gguf. - tokenizer_utils.py: add an early-return guard for len(added_tokens_ids) < 2 before the existing np.diff contiguity check. np.diff on a single-element array returns [] and .min() raises ValueError, which would discard the new in-vocab CONTROL patch; the guard flushes tokenizer.model first. Guard is inserted before the existing lines (diff = np.diff(...) and the min/max check) so their blame is unchanged. Dropped the separate refactor to fold the four duplicated "if patched > 0: write tokenizer.model" blocks into a helper because doing so re-indents lines whose blame is "Formatting & bug fixes"; the duplication remains the author's pattern. * Fix review findings: negative token_id guard and np.diff single-element - tokenizer_utils.py:481: add 0 <= lower bound to the special_token_ids bounds check. Previously a negative token_id from tokenizer.json passed 'token_id < sentence_piece_size' and Python's negative indexing wrapped tokenizer_file.pieces[-1] to silently corrupt the last piece to CONTROL. - tokenizer_utils.py:513: replace the loop-1 'if len < 2: return' guard (which was too broad: it silently skipped vocab extension for single-entry added_tokens.json) with a pre-pass that substitutes a trivially-contiguous 2-element sentinel for the contiguity check, then restores the original array before the append loop. Lines 519 ('diff = np.diff(added_tokens_ids)') and 520-529 (min/max/boundary checks and early-return write blocks) are left literally unchanged so blame remains intact. * Restore real added_tokens_ids before min boundary check Move the '_real_added_tokens_ids' restore above the 'added_tokens_ids.min() != sentence_piece_size' check. With the previous order the sentinel [sentence_piece_size, sentence_piece_size + 1] was still in scope when the min check ran, so any single-entry added_tokens .json with an out-of-range start id (e.g. 99 when sentence_piece_size=2) bypassed the boundary check and fell through to the append loop. * Scope fix_sentencepiece_gguf to GGUF export path only Previously wired fix_sentencepiece_gguf into unsloth_tokenizer_save_pretrained, which is the generic monkey-patch replacement for every tokenizer.save_pretrained call. That caused the GGUF-specific mutation (and the unconditional protobuf import in fix_sentencepiece_gguf) to run on every LoRA / merged 16-bit / push_to_hub / torchao save, where it has no purpose and can abort the entire save if the protobuf runtime is unavailable. - save.py: remove fix_sentencepiece_gguf call from unsloth_tokenizer_save_pretrained. - save.py: add the call inside unsloth_save_pretrained_gguf immediately before save_to_gguf, wrapped in try/except so a protobuf import failure logs a warning and lets GGUF conversion proceed rather than aborting the save. * Broaden special-token retag to USER_DEFINED and narrow save.py except - tokenizer_utils.py:483: the in-vocab retag previously only promoted NORMAL pieces to CONTROL, but the real Gemma tokenizer (e.g. unsloth/functiongemma -270m-it) stores <start_of_turn>/<end_of_turn> as USER_DEFINED (type 4). Extend the predicate to cover both NORMAL and USER_DEFINED so tokens marked "special": true in tokenizer.json are promoted regardless of their current sentencepiece type. Only tokens explicitly flagged special are touched, so non-special USER_DEFINED pieces are unchanged; already-CONTROL pieces stay unchanged. The warning message is generalised accordingly. - save.py:2294: narrow the except clause from Exception to ImportError. The loop-3 try/except was added to tolerate a missing protobuf runtime; leaving it broad also swallows OSError/PermissionError mid-write, which would ship a corrupted tokenizer.model to save_to_gguf. ImportError still covers the protobuf case while letting I/O errors propagate to the outer save handler. * Harden fix_sentencepiece_gguf: widen except, protobuf fallback, revert USER_DEFINED widen, guard entry id - save.py:2294: widen except from ImportError back to Exception. The loop-4 narrowing let JSONDecodeError / KeyError / OSError / PermissionError from fix_sentencepiece_gguf abort the entire GGUF export, a regression vs pre-PR behavior. The outer save_to_gguf try/except still covers GGUF-side failures; any fix-side failure now logs a typed warning and lets conversion proceed. - tokenizer_utils.py:445: the direct 'from transformers.utils import sentencepiece_model_pb2' raises TypeError ("Descriptors cannot be created directly") on modern protobuf runtimes. Prepend a sys.modules.setdefault pre-population using transformers.convert_slow_tokenizer.import_protobuf() so the subsequent from-import finds a compatible module via the module cache. The original import line is left verbatim at its place as the final resolver. - tokenizer_utils.py:483: revert loop-4 widening; retag only NORMAL pieces to CONTROL. Retagging USER_DEFINED pieces caused a concrete tokenization regression where an intentionally-USER_DEFINED in-vocab special token had its sentencepiece encoding broken ('<user> hello' changed from [11, 3, 8] to [11, 0, 12, 21, 0, 8]). The PR's stated scope is the NORMAL->CONTROL Gemma case; USER_DEFINED handling is deferred. - tokenizer_utils.py:475: defensive guard around entry["id"]. A malformed added_tokens entry missing the "id" field or with a non-int id is now skipped rather than raising KeyError / inserting garbage. * Add review tests for sentencepiece GGUF fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: octo-patch <octo-patch@github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-04-23 03:11:35 -07:00
Datta Nimmaturi	77756faa46	Fix tokenizer save gemma (#5115) ... * [WIP] Fast inference for qwen3.5 * fix tokenizer not saving properly * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * extend to VLM and clenaup * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * gate tokenizer.model saving * fix for gated/private models * Fix tokenizer save review findings - save.py:261 restore dict-based _TOKENIZER_MODEL_CACHE so negative results are cached; the set() in 0129fb5e regressed non-SentencePiece tokenizer saves to a fresh HfApi.model_info call on every checkpoint. Don't cache on exception so gated/private repos can retry later with a valid token. - save.py:282 guard `repo_info.siblings` with `or []`; huggingface_hub types this Optional and returns None for empty or new repos, which made any() raise TypeError out of save_pretrained. - save.py:3487 split push_to_hub into local save + _preserve + push so uploaded tokenizer_config.json/tokenizer.model include the fix rather than the unfixed copies written before the upload. - save.py:3352 call patch_saving_functions on tokenizers passed to unsloth_save_pretrained_torchao to match the other three save entrypoints; previously torchao saves skipped the preservation patch. * Fix push_to_hub repo_id conflict and torchao token forwarding - save.py:3493-3496 pop `repo_id` from kwargs (defaulting to `save_directory`) before calling `self.push_to_hub(repo_id, **kwargs)`. The previous `self.push_to_hub(save_directory, **kwargs)` passed `save_directory` as the first positional `repo_id` while also forwarding a user-supplied `repo_id` through kwargs, raising `TypeError: got multiple values for argument 'repo_id'` on the standard `save_pretrained(local_path, push_to_hub=True, repo_id=...)` call shape. This regression was introduced by the earlier iteration that split push_to_hub into an explicit second step. - save.py:3314 forward `token=token` on the torchao non-PEFT `tokenizer.save_pretrained(torchao_save_directory)` call so the patched wrapper can reach gated repos when HF_TOKEN is not in the environment. Left the sibling `unsloth_generic_save` call at 3063 untouched (blame points at an earlier full-finetuned save_pretrained_merged fix and the token gap there is lower risk). * Fix torchao tokenizer reload and push_to_hub repo_id default - save.py:3283 after `auto_processor.from_pretrained(save_directory)` re-runs `patch_saving_functions(tokenizer)` on the freshly loaded tokenizer. The rebind at 3283 was overwriting the patched tokenizer passed into `unsloth_save_pretrained_torchao`, so the subsequent `tokenizer.push_to_hub` (3309) and `tokenizer.save_pretrained` (3314) bypassed `_preserve_sentencepiece_tokenizer_assets` and left `{save_directory}-torchao` without `tokenizer.model` / restored `added_tokens_decoder`. - save.py:3497 fall back to `os.path.basename(save_directory)` for `repo_id` instead of the raw `save_directory`. The round-2 fallback diverged from `transformers.PreTrainedTokenizerBase.save_pretrained`, which defaults `repo_id = save_directory.split(os.path.sep)[-1]`; nested local paths like `./out/my-repo` now resolve to `my-repo` (the Hub id) instead of the full filesystem path. * Revert tokenizer save_pretrained repo_id basename fallback - save.py:3497 default `repo_id` back to `save_directory` as-is rather than `os.path.basename(save_directory)`. The basename fallback (added last iteration to match upstream transformers) stripped the user namespace from the Unsloth convention `tokenizer.save_pretrained( "user/repo", push_to_hub=True)`, redirecting the upload to `{current_user}/repo`. save.py itself treats `save_directory` as the repo id at 572, 593, 1723, 1779, 1836, 1844, 1858, and 3025, so the wrapper should follow the same convention. Users who pass a nested filesystem path with `push_to_hub=True` can supply explicit `repo_id=...`. * Guard processor.tokenizer recursion against None save.py:3511 change `elif hasattr(model, "tokenizer")` to `elif getattr(model, "tokenizer", None) is not None`. The previous guard only checked attribute existence; a ProcessorMixin that sets `tokenizer = None` (audio-only or manually constructed) would enter the branch and crash inside the recursive patch_saving_functions on `model.push_to_hub.__name__`. * Add review tests for tokenizer save * Consolidate review tests Drop redundant assertion in test_patch_saving_functions_still_patches_non_none_tokenizer. The hasattr check already proves the patch applied; the or-chained repeat assertion added no signal. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-04-22 09:03:20 -07:00
Daniel Han	dc0729aadf	Add regression test for shell injection fix in GGML conversion (#4773) ... AST-based test ensures subprocess.Popen calls in GGML conversion functions use argv lists instead of shell=True. Companion to PR #4768.	2026-04-02 00:10:47 -07:00
Daniel Han	66649d18bd	Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" ... This reverts commit cad158a56c.	2025-12-01 07:24:58 -08:00
pre-commit-ci[bot]	cad158a56c	[pre-commit.ci] auto fixes from pre-commit.com hooks ... for more information, see https://pre-commit.ci	2025-12-01 15:24:34 +00:00
Daniel Han	487a951914	Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" ... This reverts commit 964c9fef95.	2025-12-01 07:24:21 -08:00
pre-commit-ci[bot]	964c9fef95	[pre-commit.ci] auto fixes from pre-commit.com hooks ... for more information, see https://pre-commit.ci	2025-12-01 15:23:44 +00:00
Daniel Han	5f27bc4db5	Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" ... This reverts commit d34e0454ac.	2025-12-01 07:23:31 -08:00
pre-commit-ci[bot]	d34e0454ac	[pre-commit.ci] auto fixes from pre-commit.com hooks ... for more information, see https://pre-commit.ci	2025-12-01 15:20:22 +00:00
Daniel Han	ba2897a318	Revert "[FIX] Vllm guided decoding params (#3662)" ... This reverts commit fb4f0fdf56.	2025-12-01 05:43:45 -08:00
Datta Nimmaturi	fb4f0fdf56	[FIX] Vllm guided decoding params (#3662) ... * vllm sampling params fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * do not patch base_trainer * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * seperate vllm fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Apply suggestion from @danielhanchen * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" This reverts commit 58b483dc0d1790f99580665801d3fa0d7267c533. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" This reverts commit b2497519659a9f301e7a633795d9efdafdc2b277. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" This reverts commit de3daaf429f81aceb6632932b0cb1af5149652a8. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2025-12-01 05:42:37 -08:00
Daniel Han	d6bb89ad44	Formatting & bug fixes (#3563) ... * Update rl.py * Fix CE Loss * Versioning * Update loader.py * Update loader.py * extract_model_type_from_config * Model types * Update loader.py * get_transformers_model_type * Update loader.py * Update loader.py * Update loader.py * Update rl.py * Update pyproject.toml * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Versioning * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update vision.py * Update vision.py * Fix DataParallel * Update _utils.py * Update rl.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update mapper.py * Versioning * Update loader.py * Update loader.py * Update rl.py * Versioning * Update _utils.py * Fix auto_mapping * Update loader.py * Update loader.py * Update vision.py * Update vision.py * Update loader.py * Message * Update vision.py * Update loader.py * Update vision.py * cache_implementation * Update vision.py * Update loader.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Save max_seq_length * Update _utils.py * Update rl.py * Update vision.py * Update llama.py * Mistral3 vllm (#3349) * [WIP] use vLLM for vision language models * Update README.md Editing icon sizes * Update README.md Updating icon sizes * Update README.md (#2885) * MoE kernels AGPLv3 * versioning * Many bug fixes (#2908) * add deepseek v3 * add deepseek r1 base * add deepseek r1 zero * add deepseek distill llama * add deepseek distill models * remove redundant code when constructing model names * add mistral small to registry * rename model registration methods * rename deepseek registration methods * refactor naming for mistral and phi * add global register models * refactor model registration tests for new registry apis * add model search method * remove deprecated registration api * add quant type test * add registry readme * make llama registration more specific * clear registry when executing individual model registration file * more registry readme updates * Update _auto_install.py * Llama4 * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Synthetic data * Update mapper.py * Xet and Synthetic * Update synthetic.py * Update loader.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update pyproject.toml * Delete .gitignore * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update _utils.py * Update pyproject.toml * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update chat_templates.py * Seasame force float16 / float32 * Fix Seasame * Update loader.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * is_multimodal * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update vision.py * Update vision.py * Update vision.py * UNSLOTH_DISABLE_STATIC_GENERATION * Update vision.py * Auto vision detection * Sesame * Whisper * Update loader.py * Update loader.py * Update loader.py * Update mapper.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update _utils.py * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * logging * Update pyproject.toml * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * logits / temperature * Update rl_replacements.py * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Debugging only * Update llama.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Generic efficient GRPO * Update rl_replacements.py * Update rl_replacements.py * Remove debugging * Update rl_replacements.py * Update rl_replacements.py * Update vision.py * Update llama.py * Update rl_replacements.py * versioning * Update _utils.py * Update vision.py * Update mapper.py * Update loader.py * Update mapper.py * Update vision.py * Update loader.py * Update vision.py * Update loader.py * Update _utils.py * Update vision.py * gradient checkpointing * Gemma 3N fixes * Update loader.py * Versioning * Gemma 3N fixes * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Fix setup.py * setup.py * Prints * Update setup.py * Update setup.py * Update setup.py * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update vision.py * Update vision.py * Update pyproject.toml * Update vision.py * Update _utils.py * Update __init__.py * Update __init__.py --------- Co-authored-by: jeromeku <jerome.ku@gmail.com> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> * silienty skip falcon h1 import is transformers_version < 4.53.0 (#2912) * Dynamically adjust get_per_token_logps function and patch as well (#2911) * add intel gpu with vllm support (#2903) * [bugs] fix for casual mask (#2868) * fix for casual mask * use un_casual in sdpa * add missing mask * fix for type * Explicitly check if xformers exists for attention (#2889) * Update __init__.py * Update llama.py * if mlp doesn't exist in layer module check for feed_forward name for falcon h1 (#2913) * Move inputs to right devices. (#2919) * Move tensors to right devices * fix multi gpu for non mistral models * multi GPU RoPE for gemma2 * Finish up multi GPU inference * Make multiGPU rope a list * Remove unnecessary transfer to CPU * Remove unnecessary move to CPU * Donot move inputs to device yet will be handled separately in another PR * Move inputs to appropriate decoder device * Make device count global variable * Cleanup RoPE device code * Fixup num_gpu to device count * Cleanup device counts * Use device index for RoPE get_cache * Donot typecast * Use tuple instead of list for tensors. Use device index directly * fixup move to device logic * WIP VLM vLLM * Make vLLM patch a function * Add save and load lora functions * Make fast_inference setup depend on the flag * Improve fast inference patching mechanism * Make vision setting depend on checks in fastbasemodel * Check LoRA and vLLM intercompatibility for vision models * Comment pointing to vLLM LoRA check * Improve lora validation on vLLM * Error out on no vLLM and increase max lora rank * Bug fixes (#3017) * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update pyproject.toml * Delete .gitignore * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update _utils.py * Update pyproject.toml * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update chat_templates.py * Seasame force float16 / float32 * Fix Seasame * Update loader.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * is_multimodal * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update vision.py * Update vision.py * Update vision.py * UNSLOTH_DISABLE_STATIC_GENERATION * Update vision.py * Auto vision detection * Sesame * Whisper * Update loader.py * Update loader.py * Update loader.py * Update mapper.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update _utils.py * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * logging * Update pyproject.toml * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * logits / temperature * Update rl_replacements.py * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Debugging only * Update llama.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Generic efficient GRPO * Update rl_replacements.py * Update rl_replacements.py * Remove debugging * Update rl_replacements.py * Update rl_replacements.py * Update vision.py * Update llama.py * Update rl_replacements.py * versioning * Update _utils.py * Update vision.py * Update mapper.py * Update loader.py * Update mapper.py * Update vision.py * Update loader.py * Update vision.py * Update loader.py * Update _utils.py * Update vision.py * gradient checkpointing * Gemma 3N fixes * Update loader.py * Versioning * Gemma 3N fixes * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Fix setup.py * setup.py * Prints * Update setup.py * Update setup.py * Update setup.py * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update vision.py * Update vision.py * Update pyproject.toml * Update vision.py * Update _utils.py * Update __init__.py * Update __init__.py * Small fixes * Update vision.py * Update vision.py * versioning * Update __init__.py * Update llama.py * Update rl.py * Update rl.py * Update _utils.py * Update vision.py * Update vision.py * compiler stance * Update _utils.py * Update pyproject.toml * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Revert "Revert "Add Qwen2.5-VL-32B-Instruct mapping to fix quantized model me…" (#2990) This reverts commit 4021da634a. * skip_guard_eval_unsafe fix * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update llama.py * Update llama.py * Fix `quantization_method` * versioning * fix for casual mask (#3011) * [intel] add for intel path for llama.py (#3012) * fix for intel path * remove unuse code * Update unsloth/models/llama.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Update llama.py * Fix Gemma 2 (#3024) * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update pyproject.toml * Delete .gitignore * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update _utils.py * Update pyproject.toml * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update chat_templates.py * Seasame force float16 / float32 * Fix Seasame * Update loader.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * is_multimodal * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update vision.py * Update vision.py * Update vision.py * UNSLOTH_DISABLE_STATIC_GENERATION * Update vision.py * Auto vision detection * Sesame * Whisper * Update loader.py * Update loader.py * Update loader.py * Update mapper.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update _utils.py * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * logging * Update pyproject.toml * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * logits / temperature * Update rl_replacements.py * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Debugging only * Update llama.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Generic efficient GRPO * Update rl_replacements.py * Update rl_replacements.py * Remove debugging * Update rl_replacements.py * Update rl_replacements.py * Update vision.py * Update llama.py * Update rl_replacements.py * versioning * Update _utils.py * Update vision.py * Update mapper.py * Update loader.py * Update mapper.py * Update vision.py * Update loader.py * Update vision.py * Update loader.py * Update _utils.py * Update vision.py * gradient checkpointing * Gemma 3N fixes * Update loader.py * Versioning * Gemma 3N fixes * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Fix setup.py * setup.py * Prints * Update setup.py * Update setup.py * Update setup.py * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update vision.py * Update vision.py * Update pyproject.toml * Update vision.py * Update _utils.py * Update __init__.py * Update __init__.py * Small fixes * Update vision.py * Update vision.py * versioning * Update __init__.py * Update llama.py * Update rl.py * Update rl.py * Update _utils.py * Update vision.py * Update vision.py * compiler stance * Update _utils.py * Update pyproject.toml * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Revert "Revert "Add Qwen2.5-VL-32B-Instruct mapping to fix quantized model me…" (#2990) This reverts commit 4021da634a. * skip_guard_eval_unsafe fix * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update llama.py * Update llama.py * Fix `quantization_method` * versioning * Update _utils.py * Update _utils.py * Update _utils.py * falcon force float32 on sm<75 machines (#3026) * Fix torch compile issues (#3028) * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update pyproject.toml * Delete .gitignore * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update _utils.py * Update pyproject.toml * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update chat_templates.py * Seasame force float16 / float32 * Fix Seasame * Update loader.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * is_multimodal * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update vision.py * Update vision.py * Update vision.py * UNSLOTH_DISABLE_STATIC_GENERATION * Update vision.py * Auto vision detection * Sesame * Whisper * Update loader.py * Update loader.py * Update loader.py * Update mapper.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update _utils.py * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * logging * Update pyproject.toml * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * logits / temperature * Update rl_replacements.py * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Debugging only * Update llama.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Generic efficient GRPO * Update rl_replacements.py * Update rl_replacements.py * Remove debugging * Update rl_replacements.py * Update rl_replacements.py * Update vision.py * Update llama.py * Update rl_replacements.py * versioning * Update _utils.py * Update vision.py * Update mapper.py * Update loader.py * Update mapper.py * Update vision.py * Update loader.py * Update vision.py * Update loader.py * Update _utils.py * Update vision.py * gradient checkpointing * Gemma 3N fixes * Update loader.py * Versioning * Gemma 3N fixes * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Fix setup.py * setup.py * Prints * Update setup.py * Update setup.py * Update setup.py * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update vision.py * Update vision.py * Update pyproject.toml * Update vision.py * Update _utils.py * Update __init__.py * Update __init__.py * Small fixes * Update vision.py * Update vision.py * versioning * Update __init__.py * Update llama.py * Update rl.py * Update rl.py * Update _utils.py * Update vision.py * Update vision.py * compiler stance * Update _utils.py * Update pyproject.toml * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Revert "Revert "Add Qwen2.5-VL-32B-Instruct mapping to fix quantized model me…" (#2990) This reverts commit 4021da634a. * skip_guard_eval_unsafe fix * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update llama.py * Update llama.py * Fix `quantization_method` * versioning * Update _utils.py * Update _utils.py * Update _utils.py * check stride * Cleanup * Update rope_embedding.py * Update gemma2.py * Fix `set_stance` * Update pyproject.toml * Update _utils.py * Fixup patch vllm * Disable mllama * Use variables to decide VLM support * Better attn_impl handling * Patch TF protobuf incompatability * Torch 2.8 (#3186) * Fix mamba * Update loader.py * Update vision.py * Update loader.py * Filter vLLM standby logs (#3131) * filter vLLM standby logs * safeguard standby logger patch * Update unsloth/models/_utils.py * Update unsloth/models/_utils.py * Update unsloth/models/_utils.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Update loader.py * Add scaler * Update llama.py * Update _utils.py * Versioning * GPT OSS fix * GPT OSS fix * Update loader.py * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Update vision.py * Update llama.py * Update llama.py * Update llama.py * Versioning * Update mapper.py * Update vision.py * Update vision.py * Update vision.py * Upcast norms * Update loader.py * Update vision.py * Upcast layernorms * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update save.py * Update rl.py * Update pyproject.toml * Update rl.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl.py * Update _utils.py * Update __init__.py * Torch 2.8 * Update rl_replacements.py --------- Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com> * Update _auto_install.py * Update pyproject.toml * Update rl.py * Protobuf issue * Update pyproject.toml * Fix extras transformers typo in pyproject.toml * Update _utils.py * Bug fixes (#3195) * Fix mamba * Update loader.py * Update vision.py * Update loader.py * Filter vLLM standby logs (#3131) * filter vLLM standby logs * safeguard standby logger patch * Update unsloth/models/_utils.py * Update unsloth/models/_utils.py * Update unsloth/models/_utils.py --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> * Update loader.py * Add scaler * Update llama.py * Update _utils.py * Versioning * GPT OSS fix * GPT OSS fix * Update loader.py * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Update vision.py * Update llama.py * Update llama.py * Update llama.py * Versioning * Update mapper.py * Update vision.py * Update vision.py * Update vision.py * Upcast norms * Update loader.py * Update vision.py * Upcast layernorms * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update save.py * Update rl.py * Update pyproject.toml * Update rl.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl.py * Update _utils.py * Update __init__.py * Torch 2.8 * Update rl_replacements.py * Update loader.py * UNSLOTH_ENABLE_CCE * Fix * Update loader.py * Update loader.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Import fixes * Update loader.py * Fix aimv2 issue * Update loader.py * Update import_fixes.py * Update import_fixes.py * Update loader.py * Update loader.py * Update loader.py * Upgrade * Update loader.py * Update loader.py * Update loader.py * Update loader.py --------- Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com> * adallow float32 dtype in FastLanguageModel (#3204) * Update loader.py * Update vision.py * Suppress message and use unsloth sampling params * Use trl sampling params for now * Improve error message * fixup quantized fast inference model name * Add mistral 3 support --------- Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: jeromeku <jerome.ku@gmail.com> Co-authored-by: DoubleMathew <mmathew23@gmail.com> Co-authored-by: Lei Zhenyuan <zhenyuan.lei@intel.com> Co-authored-by: parth2510 <parthguptapg7326@gmail.com> * Set padding to 0 * Fix patch * fixup patch (#3359) Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com> * Update vision.py * Versioning * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * MXFP4 dequant * Update loader.py * Update vision.py * load_in_16bit * Update vision.py * Update vision.py * Update vision.py * Update rl.py * Update vision.py * offload_embedding * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update rl_replacements.py * Update loader.py * Fix padding issue * Update pyproject.toml * Update _utils.py * Update pyproject.toml * Update _utils.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * New models * Update llama.py * Versioning * Update _utils.py * Update llama.py * Update _utils.py * Update llama.py * Fix AMD * Update _utils.py * Update llama.py * Update vision.py * DEVICE_TYPE_TORCH * Update __init__.py * Update __init__.py * Update _utils.py * Move DEVICE_TYPE * Update rl_replacements.py * Update loader.py * AMD install script * Move AMD * Update _amd_install.sh * Update pyproject.toml * Update pyproject.toml * Delete _amd_install.sh * Update device_type.py * Update loader.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update _utils.py * Update tokenizer_utils.py * Versioning * Update pyproject.toml * Update loader.py * Update _utils.py * Update pyproject.toml * Update pyproject.toml * Update _utils.py * Update pyproject.toml * Update _utils.py * Update _utils.py * Update loader.py * Update _utils.py * Update _utils.py * local_files_only * Cut Cross Entropy * Update llama.py * Update vision.py * Update vision.py * Update vision.py * Qwen 3 VL vLLM (#3489) * Update __init__.py * patch_torchao * torchao_logger * Update rl_replacements.py * Fix * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update _utils.py * Versioning * fbgemm fp8 block quant support (>=1.4.0) (#3531) * fbgemm fp8 block quant support (>=1.4.0) * Verify for fp8 support before proceeding * Use unsloth zoo's Version and improve comments * spacessss * Update vision.py * Update vision.py * Update rl.py * vllm_sampling_params * Update rl.py * Update rl.py * Update rl.py * Add `ruff` pre-commit hook and apply it (#3424) * Add Ruff pre-commit config and workflow * Add kwarg spacing enforcement helper * Apply Ruff formatting * Update fp8.py * Revert ruff on some files * Update * force-exclude = true * Datasets issue * Ruff * Remove mapper * Update mapper.py * Update pyproject.toml --------- Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> Co-authored-by: jeromeku <jerome.ku@gmail.com> Co-authored-by: DoubleMathew <mmathew23@gmail.com> Co-authored-by: Lei Zhenyuan <zhenyuan.lei@intel.com> Co-authored-by: parth2510 <parthguptapg7326@gmail.com> Co-authored-by: Dan Saunders <danjsaund@gmail.com>	2025-11-07 06:00:22 -08:00
Roland Tannous	2011859430	Add TorchAO quantization tests with FP16 models and serialization workarounds (#3269) ... * Add TorchAO quantization tests with FP16 models and serialization workarounds * remove unrelated files * cleaned submission	2025-09-04 17:22:07 -07:00
Roland Tannous	0135d126df	fixed save_pretrained_torchao and associated tests (#3264)	2025-09-03 20:24:12 -07:00
Jerry Zhang	969c6a0bd8	Support saving locally in model.save_pretrained_torchao (#3263) ... Summary: Previously the test was not ran correctly and the save to local path is not tested this PR added support for that and tries to test properly Note: `python tests/saving/test_unsloth_save.py` doesn't run test Test Plan: pytest tests/saving/test_unsloth_save.py -k test_save_torchao Reviewers: Subscribers: Tasks: Tags:	2025-09-03 17:51:33 -07:00
Roland Tannous	711ec4a3ac	tests for mxfp4 and quantized models merge fix unsloth zoo pr 254 (#3223)	2025-08-29 01:30:48 -07:00
Jerry Zhang	f3ab8c21af	Support model.save_pretrained_torchao (#3111) ... Summary: Allow users merge the LoRA weights and then do a post training quantization with torchao Usage: ``` from torchao.quantization import Int8DynamicActivationInt8WeightConfig torchao_config = Int8DynamicActivationInt8WeightConfig() model.save_pretrained_torchao( save_path, tokenizer=tokenizer, torchao_config=torchao_config, ) ``` Test Plan: python tests/saving/test_unsloth_save.py Reviewers: Subscribers: Tasks: Tags:	2025-08-26 04:53:39 -07:00
Daniel Han	ce6a73986d	Revert "Revert "Add Qwen2.5-VL-32B-Instruct mapping to fix quantized model me…" (#2990) ... This reverts commit 4021da634a.	2025-07-17 15:37:23 -07:00
Roland Tannous	efe2cc43a7	tests for additional merge fix unsloth zoo pr 163 (#2719) ... * tests for additional merge fix unsloth zoo pr 163 * fixed load_dataset indent in mistral perplexity test file	2025-06-11 14:08:41 -07:00
Roland Tannous	58f3a6e29d	reroute merge logic language models + comprehensive tests + eval kits (#2673)	2025-06-02 20:32:57 -07:00
Erland366	ed16a50bf9	feat: Add validation for 4bit save method and implement corresponding error handling	2025-04-19 20:36:30 +00:00