vrr/kvcache-ai-ktransformers
2
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2025-09-10 23:34:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

yaml: fix Marlin AssertionError

Browse source 
Marlin quantized linear only supports GPU device, when change generate_op
to "KLinearMarlin", generate_device need to be changed to "cuda" accordingly.

Fixes: e5b001d76f ("Update readme; Format code; Add example yaml.")
This commit is contained in:
Aubrey Li 2025-03-21 23:58:20 +08:00
parent 05f6cede37
commit a12e8ab46e
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
ktransformers/optimize/optimize_rules/DeepSeek-V2-Lite-Chat.yaml
View file

@ -22,7 +22,7 @@
replace: replace:
class: ktransformers.operators.linear.KTransformersLinear class: ktransformers.operators.linear.KTransformersLinear
kwargs: kwargs:
generate_device: "cpu" generate_device: "cuda"
prefill_device: "cuda" prefill_device: "cuda"
generate_op: "KLinearMarlin" generate_op: "KLinearMarlin"
prefill_op: "KLinearTorch" prefill_op: "KLinearTorch"