mirror of
https://github.com/unslothai/unsloth.git
synced 2026-07-10 00:08:58 +00:00
534c877d21
* Keep native RoPE scaling when extending context; carry rope_theta for linear When max_seq_length exceeds a model's native window, the loader overwrote the model's rope_scaling with linear scaling. For models that already ship a scaled RoPE (llama3/yarn/longrope) that is far worse for long context, and on transformers v5 the linear dict omitted rope_theta (v5 keeps it under rope_parameters), so the rotary base fell back to 10000 and broke past ~8K tokens. Keep the native scaling and just widen the window; only synthesize linear for plain-RoPE models, and carry rope_theta so v5 keeps the real base. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only preserve native llama3 when extending context; keep linear fallback otherwise The patched attention constructor (patch_llama_rope_scaling) rebuilds only linear, llama3 and longrope and its longrope branch reads a top-level original_max_position_embeddings, so preserving yarn or a nested-only longrope config would raise during construction on transformers <= 4.47.1. Keep only llama3 native; yarn/longrope/other types fall back to the linear override, still carrying rope_theta. * Correct long-context extension comment to match llama3-only preservation --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| aime_eval.md | ||
| aime_eval.py | ||
| cleanup_utils.py | ||
| data_utils.py | ||
| generate_dataset_with_none.py | ||
| hf_utils.py | ||
| ocr_eval.md | ||
| ocr_eval.py | ||
| os_utils.py | ||
| perplexity_eval.md | ||
| perplexity_eval.py | ||
| run_none_detect_tests.py | ||
| test_attention_dispatch_dora_dtype.py | ||
| test_attention_masks.py | ||
| test_batched_leftpad_generation_gpu.py | ||
| test_packing.py | ||
| test_prepare_inputs_leftpad.py | ||
| test_q_galore.py | ||
| test_qat.py | ||
| test_rope_scaling_drift.py | ||
| test_trunc_normal_patch.py | ||