vrr/kvcache-ai-ktransformers
2
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-05-01 05:09:47 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

support smt and glm4

Browse source
This commit is contained in:
djw 2025-07-24 09:39:19 +00:00
parent b66d96db97
commit 613f0b7c37
8 changed files with 115 additions and 28 deletions
Download patch file Download diff file Expand all files Collapse all files

2
ktransformers/optimize/optimize_rules/Smallthinker-serve.yaml
View file

@ -49,7 +49,7 @@
- match:
name: "^model\\.layers\\..*\\.block_sparse_moe\\.experts$"
replace:
class: ktransformers.operators.experts.KTransformersExpertsV2 # custom MoE Kernel with expert paralleism
class: ktransformers.operators.experts.KSmallthinkerExperts # custom MoE Kernel with expert paralleism
kwargs:
prefill_device: "cuda"
prefill_op: None