* Do not select Flash Attention 2 for float32 model loads
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pass the load dtype to resolve_attention_implementation in the llama path
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tell the resolver the attention dtype, not the checkpoint load dtype
The float32 rule added here turns Flash Attention off for a dtype the attention
kernels never see on three archs. UNSLOTH_FORCE_CUSTOM_DTYPE loads csm, falcon_h1
and nemotron_h in float32 so the Mamba kernels keep ieee Triton precision, then
casts every projection back to correct_dtype (float16) in the exec(custom_datatype)
loop right after the load. Attention runs in float16 there, so flash_attention_2 is
usable and was being used.
Loading unsloth/Falcon-H1-0.5B-Instruct with dtype = torch.float16 on a machine with
flash-attn installed went from flash_attention_2 to sdpa and printed "the model loads
in float32", which is neither what the user asked for nor what the weights end up as.
Pass correct_dtype when the custom-dtype path set one, otherwise the load dtype.
UNSLOTH_FORCE_FLOAT32 already had the same treatment one line up: it loads bfloat16
despite the name, and reporting that keeps flash on.
* Keep the float32 rule from re-answering questions that are not about flash
A float32 load says something about the load, not about the model, so unlike the
config-driven disable reasons it must veto Flash Attention and nothing else. Setting
disable_reason for it made flash_attention_disabled true everywhere, and two other
decisions read that flag.
An explicit non-flash request was rerouted into the flash fallback ladder, which
answers differently. gemma3 with attn_implementation = "sdpa" resolves to eager,
because its SDPA is known-broken and the tail guard catches it, but the ladder
prefers gemma3's flex_attention. The same request meant one thing in bfloat16 and
another in float32.
A checkpoint shipping "attn_implementation": "eager" in its config.json was dragged
from sdpa down to eager. With no flash-specific reason that config value never
steered the choice at all; the early return in _disable_flash_attention_if_needed
that honors it is meant for models that genuinely cannot do flash.
Gate both on float32 being the only reason. Also stop the model-resolution stage
announcing a downgrade that an explicit request is about to override, again only for
the float32 reason so no existing message changes.
* Shorten the attention dtype comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* fix(gpt-oss): prefer flex attention over sdpa
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(gpt-oss): use eager config for unsupported backends
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
