mirror of
https://github.com/unslothai/unsloth.git
synced 2026-07-09 15:58:41 +00:00
d4fbc81d3a
* Restore dropped FP8 weight_scale_inv tensors on load Some block-scale FP8 checkpoints (for example Qwen3.6-27B-FP8, issue #6200) load with transformers leaving an mlp.gate_proj as a plain bf16 Linear instead of an fp8 module. Its raw quantized values are read into the bf16 weight and the weight_scale_inv is dropped as an unexpected key, so the weight is used un-scaled and the base model is garbage (perplexity around 2 million). After load, for every checkpoint weight_scale_inv whose live weight is not fp8, dequantize the orphaned weight in place using the block scale from the checkpoint index. Modules that were converted correctly keep an fp8 weight and are skipped, so healthy checkpoints and single-file checkpoints are a no-op. Verified on Qwen3.6-27B-FP8: 64 gate_proj scales restored, perplexity 2028902 to 8.9. No-op on Qwen3-8B-FP8 (all scales already live). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden FP8 weight_scale_inv restore from review - Skip restore when the model has no fp8 weights, so an intentionally dequantized load (load_in_16bit) is never re-scaled and corrupted. - Thread revision, subfolder and cache_dir through the index and shard downloads so scales come from the same snapshot as the weights. - Cover unsharded single-file model.safetensors checkpoints (no index). - Handle transposed block-scale layouts and skip on a true grid mismatch instead of applying a wrong scale. - Match text-only VLM loads where the language_model prefix was stripped. - Restore on the FastLanguageModel text path too, not only vision. - Handle a scalar weight_block_size; per-tensor error handling so one bad tensor cannot abort the rest or hide a partial mutation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address second review round on FP8 scale restore - Bound peak memory: dequantize block views in place with the fp32 scale broadcast instead of materializing a full expanded scale and fp32 copy, so a near-VRAM-limit load is not pushed into OOM by the repair. - Restore on the sequence-classification load path too. - Cover more VLM key remappings (language_model.model.* to model.language_model.*) when matching modules. - Skip the restore for variant loads (variant=...) rather than risk applying default-checkpoint scales to variant weights. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align FP8 scale restore revision with the loaded weights and warn on disk-offloaded layers In llama.py the CausalLM/SequenceClassification weight loads resolve model_name on its default branch (revision is not forwarded there), so read the dropped weight_scale_inv tensors from the same default branch instead of the requested revision, avoiding rescaling default-branch weights with scales from another revision. In loader_utils.py a disk-offloaded layer keeps its weight on the meta device until the offload hook materializes it, so the scale cannot be applied in place. Skip such layers explicitly and print a warning rather than silently leaving them unscaled. * Tighten comments in the FP8 scale restore path --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| notebooks | ||
| python | ||
| qlora | ||
| saving | ||
| security | ||
| sh | ||
| studio | ||
| studio_setup_ps1 | ||
| utils | ||
| version_compat | ||
| vllm_compat | ||
| __init__.py | ||
| _zoo_aggressive_cuda_spoof.py | ||
| _zoo_rocm_spoof.py | ||
| conftest.py | ||
| run_all.sh | ||
| test_attention_implementation.py | ||
| test_attn_impl_honor_explicit.py | ||
| test_bad_mappings_redirect.py | ||
| test_callback_signature_drift.py | ||
| test_cli_export_unpacking.py | ||
| test_enforce_kwargs_spacing.py | ||
| test_fast_gemv_dispatch.py | ||
| test_fast_generate_slow_guard.py | ||
| test_finetune_last_n_layers.py | ||
| test_fp8_device_context.py | ||
| test_fp8_restore_dropped_scale.py | ||
| test_fp8_tiny_e8m0.py | ||
| test_fused_ce_not_return_dict_logits.py | ||
| test_gemma4_chat_template.py | ||
| test_gemma_2b_mapper_key.py | ||
| test_generate_kwarg_gate.py | ||
| test_get_model_name.py | ||
| test_gradient_checkpointing_restore.py | ||
| test_ignored_tokenizer_casing.py | ||
| test_import_fixes_drift.py | ||
| test_loader_glob_skip.py | ||
| test_missing_torchvision_vlm.py | ||
| test_model_registry.py | ||
| test_moe_lora_targets.py | ||
| test_multi_image_grpo_chunking.py | ||
| test_nvfp4_quant_load.py | ||
| test_offline_loading_helpers.py | ||
| test_offload_embedding_hooks.py | ||
| test_offload_tied_guard.py | ||
| test_peft_tensor_parallel_compat.py | ||
| test_peft_weight_converter_compat.py | ||
| test_prefetch_snapshot_scope.py | ||
| test_pretrain_compile_reset.py | ||
| test_public_api_surface.py | ||
| test_raw_text.py | ||
| test_resolve_model_class.py | ||
| test_studio_install_workspace_guard.py | ||
| test_studio_root_resilience.py | ||
| test_studio_shutdown_thread_wait.py | ||
| test_synthetic_chunk_data.py | ||
| test_tool_mask_zoo_compat.py | ||
| test_uninitialized_position_ids.py | ||
| test_video_path_validation.py | ||
| test_vllm_broken_detection.py | ||
| test_windows_rocm_bnb_version.py | ||