mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-16 04:13:54 +00:00
c3230a749a
* Hand the LoRA to vLLM on TRL 1.x GRPO rollouts
TRL 1.10.0 dropped the collective_rpc("reload_weights") call from
VLLMGeneration.generate, so the regex anchored on it matched nothing and
raised. The lora_request injection ran after that anchor in the same
function, so it never happened, and _init_vllm/sync_weights had already
been installed by then. fast_inference=True GRPO therefore sampled every
rollout from the base model with the adapter ignored, and reported finite
losses while doing it.
Wrap VLLMGeneration.generate instead of rewriting its source, and
intercept on the vLLM engine (generate/chat/collective_rpc), which is the
part Unsloth owns. Restore all three methods if any patch fails, so the
half-patched state is unreachable, and wake a sleeping engine before
sync_weights returns early.
* Raise the TRL cap to 1.10.0, and open the datasets cap to 4.x
Two metadata caps, and both have to move together or neither does anything.
trl<=0.24.0 is metadata only. Nothing in unsloth or unsloth_zoo reads the TRL
version and refuses 1.x at runtime; the only comparisons are floors at
trainer.py:725 and :993. Measured against real TRL 1.10.0 with the trees on
PYTHONPATH so no resolver ever sees the cap: import ok, tests/ -k grpo 991
passed, tests/version_compat 1702 passed, and four real 3-step GRPO runs whose
grad-norms and reward stats match TRL 0.22.2 to every digit. The 5-test gap
against 0.22.2 is this repo's own TRL >= 1.7.0 gated assertions, which 0.22.2
skips and 1.10.0 runs and passes.
datasets<4.4.0 is the second cap, and raising only the TRL one ships a pin no
user exercises: with the datasets cap in place, a resolver asked for
trl<=1.10.0 settles on trl==0.29.1, because TRL 1.10.0 requires
datasets>=4.7.0. So the datasets cap has to open at the same time.
It opens as a hole rather than a floor bump. A datasets>=4.7.0 floor would drop
Python 3.9 and force 4.x on TRL 0.22.2 users who do not need it. The hole
!=4.4.*,!=4.5.0 is exactly what patch_datasets in import_fixes.py already
refuses at runtime over the RLock recursion bug, so metadata and the runtime
guard now agree instead of the metadata being the stricter of the two. Verified
on datasets 4.8.5 with TRL 1.10.0: 991 / 1702 passed, exit 0 both, and the GRPO
smoke numbers are identical to the datasets 3.6.0 run to every digit.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Test that the GRPO rollout adapter reaches vLLM on every TRL generate shape
The failure this locks down is silent: with TRL 1.10.0's generate, the old
source patch anchored on collective_rpc("reload_weights") matched nothing,
raised, and took the lora_request injection down with it, so fast_inference
GRPO sampled every rollout from the base model at exit 0.
The tests are shape-driven rather than version-driven. A synthetic
trl.generation.vllm_generation goes into sys.modules with a VLLMGeneration
whose _init_vllm and sync_weights carry the anchors the source patches need,
and whose generate reaches a fake engine the way a given TRL era reaches it:
sync_weights on 1.10.0, collective_rpc on 0.22.2, chat for conversational
rollouts, and no engine at all in server mode. Then the real
vllm_generation_init_patch runs over it and the fake engine records what
arrived. No vLLM, no GPU, no network, and importlib_version is monkeypatched
so the installed TRL is irrelevant - these pass against the 0.25.1 in this
env, which the patch would otherwise skip outright.
Also covered: the engine methods are restored after the call so a later
model.fast_generate is clean, an engine that already carried its own generate
gets it back rather than deleted, a TRL-created engine is left alone, a
sleeping engine is woken before sync_weights returns early, a failed sibling
patch rolls all three methods back, and patching twice does not double wrap.
Seven of the ten fail on
|
||
|---|---|---|
| .. | ||
| __init__.py | ||
| _fetch.py | ||
| test_bitsandbytes_pinned_symbols.py | ||
| test_import_leaves_torch_globals_alone.py | ||
| test_peft_conversion_symbol_backfill.py | ||
| test_peft_pinned_symbols.py | ||
| test_sentence_transformers_pinned_symbols.py | ||
| test_transformers_pinned_symbols.py | ||
| test_trl_fake_train_cpu.py | ||
| test_trl_grpo_fake_run.py | ||
| test_trl_grpo_pinned_symbols.py | ||
| test_trl_loss_normalization_contract.py | ||
| test_trl_padding_free_max_length.py | ||
| test_trl_vllm_generation_lora_patch.py | ||
| test_unsloth_zoo_save_merged_pinned_symbols.py | ||