mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-07-10 01:28:39 +00:00
ef6c47f9d2
* [feat](kt-kernel): AVX2 MXFP4 MoE MXFP4 dispatch
- Add AVX2 MXFP4 MoE kernel (mxfp4-moe.hpp) with 4-token M-blocking
- Wire AVX2MXFP4_MOE binding in ext_bindings.cpp
- Support TP_MOE down_proj slicing and multi-pool per-expert loading
- Add test_fp4_moe_avx2.py integration test
* [fix](kt-kernel): address PR #2010 review — memory leaks, alignment, dynamic expert update
- Track aligned_alloc pointers in AVX2_MOE_BASE::owned_aligned_allocs_ and
free them in the destructor (fixes BufferB backing memory leak on destroy).
- Track per-TP down_buf allocations in TP_MOE::tp_owned_down_bufs_ with
nullptr checks and size rounding to alignment boundary.
- Add nibble-alignment runtime check for per_tp_interm in MXFP4 TP K-split.
- Add write_weight_scale_to_buffer override to TP_MOE<AVX2_MXFP4_MOE_TP>,
enabling dynamic expert update with kt-threadpool-count>=2.
- Guard against ZeroDivisionError in test_fp4_moe_avx2.py.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [fix](kt-kernel): add intermediate_size parity check in MXFP4 TP flat-buffer path
The per-expert path validates that intermediate_size is even (required for
nibble-aligned FP4 addressing), but the flat-buffer path was missing this
check — an odd value would silently truncate /2 divisions, corrupting
memcpy sizes and offsets.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(avx2-moe): fix TP offset calculation and add safety checks
C1-C4: Fix incorrect TP offset calculations in load_weights()
- Per-expert mode used per_tp_interm instead of full_interm for offsets
- This caused segfault when TP > 1 due to invalid pointer arithmetic
H1-H3: Add safety checks
- H1: Validate source weight pointers are not null
- H2: Check lid index is within bounds
- H3: Check BufferB.b is not null in gemm_mxfp4
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(avx2-moe): revert incorrect C2/C4 offset changes, keep safety checks
Reverts the incorrect offset calculation changes from previous commit.
The original per_tp_interm-based offsets were correct:
- gate/up weights are N-split (along intermediate dim)
- Each TP partition handles per_tp_interm rows
- Offset = i * per_tp_interm * hidden / 2 (not full_interm)
Keeps H1-H3 safety checks:
- H1: Validate source weight pointers are not null
- H2: Check lid index is within bounds
- H3: Check BufferB.b is not null in gemm_mxfp4
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(avx2): copy weights to owned buffers in per-expert mode
Previously, AVX2 MXFP4 MoE per-expert mode directly pointed BufferB.b
into mmap'd safetensor data. This caused use-after-free when Python
layer releases the mmap after load_weights() returns.
Now AVX2 copies weights into owned buffers via memcpy/from_raw_mat(),
matching AMX behavior. This decouples the MoE weights from mmap lifecycle.
Changes:
- buffer_b_required_size_impl: always allocate full buffer (weights + scales)
- make_buffer_b_impl: always create full BufferB with owned storage
- Single-TP per-expert: use from_raw_mat() instead of direct pointer
- TP_MOE per-expert: add gate/up owned buffers with memcpy
- Destructor: free gate/up buffers alongside down
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Revert "[fix] Add runtime AMX BF16 check to prevent SIGILL on pre-Sapphire Rapids CPUs (#2018)"
This reverts commit
|
||
|---|---|---|
| .. | ||
| .gitignore | ||
| bench_moe_amx_int8.py | ||
| configuration_deepseek_v3.py | ||
| modeling_deepseek_v3.py | ||
| repro_llamafile_re.py | ||
| test-debug.py | ||
| test_apply_rope.py | ||
| test_attention.py | ||
| test_awq_moe_amx.py | ||
| test_bf16_moe.py | ||
| test_deepseekv3.py | ||
| test_deepseekv3_prefill.py | ||
| test_deepseekv3_prefill_speed.py | ||
| test_fp4_moe_amx.py | ||
| test_fp4_moe_avx2.py | ||
| test_fp4_moe_v4.py | ||
| test_fp8_moe.py | ||
| test_fp8_perchannel_moe.py | ||
| test_gate.py | ||
| test_k2_moe_amx.py | ||
| test_k2_write_buffer.py | ||
| test_linear.py | ||
| test_mla.py | ||
| test_mla_qlen.py | ||
| test_mla_quant.py | ||
| test_mla_simple.py | ||
| test_mla_torch.py | ||
| test_mlp.py | ||
| test_moe.py | ||
| test_moe_amx.py | ||
| test_moe_kernel.py | ||
| test_moe_kml.py | ||
| test_rope.cpp | ||
| test_rope.py | ||
| test_softmax.py | ||
| test_write_buffer.py | ||
| torch_attention.py | ||