mirror of
https://github.com/unslothai/unsloth.git
synced 2026-07-24 23:23:56 +00:00
a963ec92e7
* Fix CPOTrainer crash with multimodal processors CPOTrainer shares build_tokenized_answer/tokenize_row and __init__ with ORPOTrainer, but the ORPO replacement functions that route tokenization through the underlying text tokenizer and resolve pad_token_id were only registered for orpo_trainer. With a multimodal processing class (e.g. Gemma4Processor) the positional self.processing_class(prompt, ...) call binds prompt to images=, leaving text=None and raising TypeError: 'NoneType' object is not subscriptable. Register the existing orpo_trainer_text_tokenizer and orpo_trainer_processor_pad_token under cpo_trainer as well so CPO/SimPO fine-tuning of multimodal models works. No change for plain tokenizers. * Add CPO processor tokenizer regression test Static, CPU-only checks that cpo_trainer registers the same orpo_trainer_text_tokenizer and orpo_trainer_processor_pad_token rewriters as orpo_trainer, and that the rewriter drops the broken positional self.processing_class(prompt, ...) call. Guards against issue #4952 regressing. * Format CPO test assert for ruff line length (pre-commit) * Bind CPO __init__ pad/eos token reads to underlying tokenizer TRL 0.28+ CPOTrainer.__init__ reads bare processing_class.pad_token and processing_class.eos_token before pad_token_id, which raises AttributeError for multimodal processors (e.g. Gemma) where those live on .tokenizer. Extend orpo_trainer_processor_pad_token to route that block through the underlying tokenizer, and add a regression test. * Tighten code comments (no logic change) * Make CPO/ORPO rewriters reach the trainer on TRL 1.x TRL 1.x moved CPOTrainer and ORPOTrainer out of trl.trainer into trl.experimental.<algo> and dropped the trl.trainer.<algo>_trainer shim that older TRL (0.26 - 0.28) kept. patch_trl_rl_trainers() discovers trainers via dir(trl.trainer), so on TRL 1.x cpo_trainer and orpo_trainer are never found and the multimodal-processor tokenization fix (#4952) silently stops applying, even though the rewriters themselves still match the source. Re-expose experimental-only trainers that Unsloth has rewriters for (RL_FUNCTIONS keys) under trl.trainer before discovery, so the existing patch machinery and its thin-wrapper resolution work unchanged. The alias is a no-op on older TRL where trl.trainer.<algo>_trainer already exists. Also rebind the patched Trainer/Config into every already-imported trl.* module that holds the original class so the fix is visible at the experimental import site (from trl.experimental.cpo import CPOTrainer), not only via trl.trainer. Verified on transformers 4.57.6 + trl 0.22.2, transformers 4.57.6 + trl 0.27.1, and transformers 5.12.1 + trl 1.6.0: CPOTrainer with a multimodal processor tokenizes through the underlying text tokenizer with no crash on all three, and the SFT/GRPO/DPO patch paths are unchanged. * Format for ruff (pre-commit) * Simplify CPO fix to mirror ORPO registrations (#4952) Register the existing ORPO row-tokenizer/pad-token rewriters for cpo_trainer. Under the trl<=0.24.0 pin CPOTrainer lives in trl.trainer.cpo_trainer (found by dir(trl.trainer)), shares ORPO's build_tokenized_answer and uses processing_class.pad_token_id, so the two registrations are sufficient. Drop the trl 1.x experimental aliasing/rebind machinery in rl.py and the bare-pad_token rewriter: trl 1.x (CPO in trl.experimental) and the bare pad_token pattern (trl>=0.28) are not installable under the pin. * CPO: route bare pad_token/eos_token default through inner tokenizer TRL 1.x CPO/ORPO __init__ (the trl.experimental source unsloth resolves on TRL 0.26+) defaults processing_class.pad_token from processing_class.eos_token before tokenizing. Multimodal processors (Gemma3/Gemma4 Processor) expose those attributes on .tokenizer, not on the processor, so that bare access raises AttributeError during __init__ even with the pad_token_id fallback registered. Extend orpo_trainer_processor_pad_token to rewrite that defaulting block to run on the inner tokenizer. The pinned TRL range (<=0.24.0) has no such block, so the regex is a no-op there and only the existing pad_token_id fallback applies. Verified the rewrite against the real trl 1.6.0 experimental CPOTrainer.__init__ (bare access removed, result compiles, a processor without pad_token no longer raises) and added offline regression tests for both the rewrite and its no-op. --------- Co-authored-by: Daniel Han <michaelhan2050@gmail.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| conftest.py | ||
| test_construct_chat_template_validation.py | ||
| test_cpo_processor_text_tokenizer.py | ||
| test_cross_platform_parity.py | ||
| test_dpo_vision_processor_passthrough.py | ||
| test_e2e_no_torch_sandbox.py | ||
| test_fast_language_model_text_only.py | ||
| test_fast_model_config_passthrough.py | ||
| test_fast_sentence_transformer_redirect_lifecycle.py | ||
| test_flash_attn_install_python_stack.py | ||
| test_gpu_init_ldconfig_guard.py | ||
| test_install_python_stack.py | ||
| test_no_torch_filtering.py | ||
| test_orpo_processor_text_tokenizer.py | ||
| test_patch_trl_rl_trainers_defensive.py | ||
| test_studio_import_no_torch.py | ||
| test_tokenizers_and_torch_constraint.py | ||
| test_unsloth_run_tool_policy_resolver.py | ||
| test_vision_lora_targeting.py | ||