mirror of
https://github.com/unslothai/unsloth.git
synced 2026-07-14 18:28:43 +00:00
b91cdc8793
* Fix Qwen3 NaN: self-heal vision pad_token in load_correct_tokenizer Text-only Qwen3 (and Qwen2.5) models share Qwen3-VL's vocab, so their Hub tokenizer configs ship <|vision_pad|> as pad_token. Padding text-only training with a vision token corrupts attention/loss and produces NaN losses and gradients on affected stacks. patch_tokenizer already heals this, but only when a model with config is passed. The standalone load_correct_tokenizer path (and custom training loops) still returned <|vision_pad|>. This adds a model-independent guard in load_correct_tokenizer that replaces a vision pad_token on text-only tokenizers with the first safe text token (<|endoftext|>, <pad>, [PAD], <unk>), falling back to eos_token only if it differs from pad_token. The result now matches upstream Qwen configs (pad_token <|endoftext|>, id 151643) with no new token added. Vision processors (image_processor present) and non-vision pad tokens (Llama, Qwen2) are left untouched. Fixes #3155 * Format pad_token helper for ruff kwarg-spacing hook (pre-commit) * Harden vision pad_token fix: drop unk candidate, guard get_vocab, skip vision eos * Tighten code comments (no logic change) * Delegate pad_token fix to shared unsloth_zoo.pad_token Generalize the narrow _fix_vision_pad_token by delegating to unsloth_zoo's shared fix_pad_token (single source of truth, AGPL-3.0), which scans the reserved-token families instead of only the vision-pad case. A guarded import keeps this working against an older unsloth_zoo that has not shipped the module yet: on ImportError it falls back to _fix_vision_pad_token. allow_add=False is passed so the early load_correct_tokenizer call stays side-effect free (no model here to resize embeddings); the later model-aware patch_tokenizer call finishes the job and is idempotent. Adds tests/python/test_pad_token_fix.py covering both dispatch paths offline. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| conftest.py | ||
| test_construct_chat_template_validation.py | ||
| test_cpo_processor_text_tokenizer.py | ||
| test_cross_platform_parity.py | ||
| test_dpo_vision_processor_passthrough.py | ||
| test_e2e_no_torch_sandbox.py | ||
| test_fast_language_model_text_only.py | ||
| test_fast_model_config_passthrough.py | ||
| test_fast_sentence_transformer_redirect_lifecycle.py | ||
| test_flash_attn_install_python_stack.py | ||
| test_gpu_init_ldconfig_guard.py | ||
| test_install_python_stack.py | ||
| test_no_torch_filtering.py | ||
| test_orpo_processor_text_tokenizer.py | ||
| test_pad_token_fix.py | ||
| test_patch_trl_rl_trainers_defensive.py | ||
| test_studio_import_no_torch.py | ||
| test_tokenizers_and_torch_constraint.py | ||
| test_unsloth_run_tool_policy_resolver.py | ||
| test_vision_lora_targeting.py | ||