mirror of
https://github.com/unslothai/unsloth.git
synced 2026-07-10 00:08:58 +00:00
4929c5f769
* Keep pad-named pad_tokens; defer pad repair to shared unsloth_zoo.pad_token A pad-named token (e.g. <|vision_pad|>) is a valid pad. The narrow fallback that stripped vision pad tokens on text-only models is now a no-op; the active path delegates to the shared fix_pad_token in unsloth_zoo, which keeps pad-named tokens and only heals missing / eos-collision / out-of-range pads. This fixes the Qwen3-4B-Base load crash (its config ships pad_token=<|vision_pad|>): the old swap could not find a safe text pad (eos is <|endoftext|>, no unk_token) and left the tokenizer broken. Removes the unused _VISION_PAD_TOKENS / _SAFE_TEXT_PAD_TOKENS sets. Tests updated. Pairs with unslothai/unsloth-zoo#831. * Remove _fix_vision_pad_token; inline the no-op fallback A pad-named token (e.g. <|vision_pad|>) is a valid pad, so the old vision-pad swap helper has no purpose. _fix_pad_token now returns the tokenizer unchanged when the shared unsloth_zoo.pad_token module is unavailable, instead of routing through a no-op helper. Test WANTED set updated. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| conftest.py | ||
| test_construct_chat_template_validation.py | ||
| test_cpo_processor_text_tokenizer.py | ||
| test_cross_platform_parity.py | ||
| test_dpo_vision_processor_passthrough.py | ||
| test_e2e_no_torch_sandbox.py | ||
| test_fast_language_model_text_only.py | ||
| test_fast_model_config_passthrough.py | ||
| test_fast_sentence_transformer_redirect_lifecycle.py | ||
| test_flash_attn_install_python_stack.py | ||
| test_gpu_init_ldconfig_guard.py | ||
| test_install_python_stack.py | ||
| test_install_uv_override_space.py | ||
| test_no_torch_filtering.py | ||
| test_orpo_processor_text_tokenizer.py | ||
| test_pad_token_fix.py | ||
| test_patch_trl_rl_trainers_defensive.py | ||
| test_studio_import_no_torch.py | ||
| test_tokenizers_and_torch_constraint.py | ||
| test_unsloth_run_tool_policy_resolver.py | ||
| test_vision_lora_targeting.py | ||