vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-08-06 15:05:31 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

llama-quant : exclude i32 ffn_gate_tid2eid routing table from quantization (#25787)

Browse source 
DeepSeek-V4's ffn_gate_tid2eid tensor is an i32 token-id -> expert-id
index table, not weights. It was never added to the name-based
exclusion list alongside ffn_gate_inp.weight, so llama-quantize tries
to quantize it and fails since i32 cannot convert to a float type.

Fixes ggml-org/llama.cpp#25754

Signed-off-by: Yash Raj Pandey <yashpn62@gmail.com>
This commit is contained in:
Yash Raj Pandey 2026-07-18 07:43:18 -04:00 • committed by GitHub
parent 86a9c79f86
commit 4937ca83f4
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 3 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

3
src/llama-quant.cpp
View file

@ -306,6 +306,9 @@ static bool tensor_allows_quantization(const llama_model_quantize_params * param
// NOTE: can't use LLM_TN here because the layer number is not known
quantize &= name.find("ffn_gate_inp.weight") == std::string::npos;
// do not quantize the i32 token-id -> expert-id routing table (DeepSeek-V4)
quantize &= name.find("ffn_gate_tid2eid.weight") == std::string::npos;
// these are very small (e.g. 4x4)
quantize &= name.find("altup") == std::string::npos;
quantize &= name.find("laurel") == std::string::npos;