vrr/unsloth
2
0
Fork 0
mirror of https://github.com/unslothai/unsloth.git synced 2026-07-10 00:08:58 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 29
unsloth/tests/utils
History
Daniel Han 9780cdcca1
Fix FlashAttention fp32 crash with DoRA (use_dora=True) (#6526) 
* Fix FlashAttention fp32 crash with DoRA (use_dora=True)

DoRA upcasts lora_magnitude_vector to fp32 for the optimizer, which promotes
the q/k/v_proj output to fp32. FlashAttention only accepts fp16/bf16, so the
fp32 q/k/v raised 'FlashAttention only support fp16 and bf16 data type'.
Downcast q/k/v to the compute dtype before the flash kernels.

Fixes #1013

* Apply kwarg-spacing format hook to DoRA dtype test (pre-commit)

* DoRA+FA2: downcast any fp32 among Q/K/V and clamp to a flash-supported dtype

* Tighten code comments (no logic change)

---------

Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
2026-06-23 01:29:19 -07:00
..
__init__.py Revert "[FIX] Vllm guided decoding params (#3662)" 2025-12-01 05:43:45 -08:00
aime_eval.md reroute merge logic language models + comprehensive tests + eval kits (#2673) 2025-06-02 20:32:57 -07:00
aime_eval.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
cleanup_utils.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
data_utils.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
generate_dataset_with_none.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
hf_utils.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
ocr_eval.md Fix Typos in Documentation and Comments (#2721) 2025-06-17 04:34:51 -07:00
ocr_eval.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
os_utils.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
perplexity_eval.md reroute merge logic language models + comprehensive tests + eval kits (#2673) 2025-06-02 20:32:57 -07:00
perplexity_eval.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
run_none_detect_tests.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_attention_dispatch_dora_dtype.py Fix FlashAttention fp32 crash with DoRA (use_dora=True) (#6526) 2026-06-23 01:29:19 -07:00
test_attention_masks.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
test_batched_leftpad_generation_gpu.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_packing.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_prepare_inputs_leftpad.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_q_galore.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_qat.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_rope_scaling_drift.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
test_trunc_normal_patch.py Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00