vrr/kvcache-ai-ktransformers
2
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2025-09-05 20:19:51 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2
698 commits 41 branches 19 tags 46 MiB
Commit graph

68 commits

Author SHA1 Message Date
djw
3f9bbf1181 support qwen3, dont speak human language 2025-04-28 08:44:47 +00:00
chenht2022
f3d842a0ca support AMX 2025-04-25 14:47:16 +00:00
root
921061666c fix some bugs 2025-04-17 00:48:09 +08:00
Atream
3b9e16cec7
Update attention.py 2025-04-09 10:54:00 +08:00
Aubrey Li
12a4c631df Fix TypeError when invoke KLinearCPUInfer.forward() 
Fix the following error:

  File "/home/aubrey/work/ktransformers/ktransformers/operators/linear.py", line 825, in forward
    y = self.generate_linear.forward(x, bsz_tensor)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: KLinearCPUInfer.forward() takes 2 positional arguments but 3 were given
 2025-04-07 12:03:35 +08:00
dongjw
1b7672937b update install doc and fix local_chat bug 2025-04-03 12:42:41 +08:00
Azure-Tang
31677181c3 Fix ktransformers-server flashinfer wrapper position arg issue; 
Fix db position issue
 2025-04-01 07:30:23 +00:00
Azure-Tang
203b853c75 rm KMoEGateDeepSeekV3, fall back to KMoEGate 2025-04-01 07:13:05 +00:00
Azure-Tang
3a5330b215 Merge branch 'main' into work-concurrent 2025-04-01 06:48:19 +00:00
Atream
25cee5810e add balance-serve, support concurrence 2025-03-31 22:55:32 +08:00
Atream
633af5d235
Update gate.py 2025-03-20 14:54:01 +08:00
Atream
b453333f60
Update gate.py 2025-03-19 16:14:54 +08:00
Atream
44599229cd
Update gate.py 2025-03-19 12:16:48 +08:00
Atream
114995355b
fix-gate-compile 2025-03-19 11:27:18 +08:00
Atream
3934b9dfc1 rollback-triton-prefill 2025-03-15 14:21:21 +00:00
Azure
3986e2d2cf
Merge pull request #178 from fxzjshm/hip 
[Feat] Port to ROCm/HIP
 2025-03-15 02:31:07 +08:00
Azure-Tang
e5b001d76f Update readme; Format code; Add example yaml. 2025-03-14 14:25:52 -04:00
Atream
a889288fc1 use compile for gate, slight performance improvement 2025-03-14 12:43:28 +00:00
Azure-Tang
ed8437413b merge main; Add torch q8 linear 2025-03-14 05:52:07 -04:00
Atream
6f43bbe55f fix-singleton 2025-03-14 04:16:53 +00:00
Lander-Hatsune
d166fb9f6e cpuinfer: filter repeated backend instantiation 2025-03-10 22:03:04 +08:00
Atream
d453c320f1 fix flashinfer precision 2025-03-07 14:07:00 +00:00
Yi Pan
01755a60c0
fix: wrong shape in KLinearMarlin. 2025-03-03 17:34:45 +08:00
Atream
f35e8d41d8 support chunk prefill, support 139K context for 24G VRAM 2025-03-01 11:28:25 +00:00
Atream
e645d84794 use generation config from json file in official repo 2025-02-27 11:48:34 +00:00
liam
ffb86c66e3 ⚡ fix experts torch 2025-02-26 15:04:40 +08:00
Atream
477ac28a9c fix-update-flashinfer_wrapper_local_chat 2025-02-25 12:47:31 +00:00
Atream
b443c7dfa2
Merge pull request #657 from kvcache-ai/feat-absorb-for-long-prefill 
Feat absorb for long prefill
 2025-02-25 16:53:21 +08:00
Atream
f4c198bd42 support absorb for prefill long context 2025-02-25 08:52:02 +00:00
Azure
ca7366d2db Merge remote-tracking branch 'upstream/develop-0.2.2' into support-fp8 2025-02-24 11:58:10 +00:00
Azure
581a524f65 Add data loader to read special weights for fp8; Add special weight process script 2025-02-24 11:34:17 +00:00
Atream
e9b1216a9a Merge branch 'main' into feat-absorb-for-long-prefill 2025-02-24 09:44:17 +00:00
Atream
e8e02e5ccc support Moonlight 2025-02-23 14:21:18 +00:00
DDong Jianwei
95d937c51d tmp 2025-02-23 18:51:42 +08:00
Atream
006e8c6abc remove causal mask 2025-02-23 07:40:47 +00:00
Azure
7b7c6a657d Add fp8 linear kernel;\n Add empty cache to fit in 16G VRAM; By 'wkGCaSS - 知乎 https://zhuanlan.zhihu.com/p/25491611225' 2025-02-22 13:05:08 +00:00
Atream
024009675e Merge branch 'main' into feat-more-context 2025-02-22 06:17:39 +00:00
Atream
5ec33d046d optimize gguf dequant, save mem, support Q2_K 
use marlin for lm_head, lm_head only calc last token for prefill
extend context window to 19K for DeepSeek-V3/R1 within 24GB VRAM
 2025-02-22 06:13:01 +00:00
Atream
a529518346 clean PR code and disable flashinfer 2025-02-19 04:42:47 +00:00
ceerrep
2ffb43f974 fix: adapt prefix cache in forward_linux_flashinfer 2025-02-18 11:40:15 +08:00
ceerrep
bb1cadfff3 Merge branch 'fix_precision_MLA' of https://github.com/kvcache-ai/ktransformers into server-prefix-cache 2025-02-17 18:08:04 +08:00
Atream
038bc30888 fix precision bug imported by position_ids in 0.2.0 2025-02-17 09:23:14 +00:00
ceerrep
5ac266085e fix: use flash_attn for faster prefill 2025-02-17 00:11:24 +08:00
Azure
ff6b265e53 Mock triton mla due to precision issue 2025-02-16 06:03:12 +00:00
Atream
c5f036e8a4
Merge pull request #333 from kvcache-ai/feat_experts_gpu 
toy support for experts on GPU, no CUDA Graph
 2025-02-15 23:30:24 +08:00
Atream
c189d55bd1 toy support for experts on GPU, no CUDA Graph 2025-02-15 15:16:00 +00:00
Atream
92399283b6
Update attention.py 2025-02-15 15:43:35 +08:00
Atream
d90749d35d
Update triton_attention.py 2025-02-15 15:41:01 +08:00
Atream
885a91e7db
Merge pull request #294 from kvcache-ai/feat-fast-MLA 
Feat fast mla
 2025-02-14 19:40:36 +08:00
Atream
1084d4e4b4 linux support triton MLA kernel 2025-02-14 11:38:55 +00:00