vrr/kvcache-ai-ktransformers
2
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2025-09-05 20:19:51 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2
764 commits 41 branches 19 tags 46 MiB
Commit graph

75 commits

Author SHA1 Message Date
wang jiahao
8456222852
Merge pull request #1276 from kvcache-ai/support_load_safetensor
Some checks failed
Book-CI / test (push) Has been cancelled
Details
Deploy / deploy (macos-latest) (push) Has been cancelled
Details
Deploy / deploy (ubuntu-latest) (push) Has been cancelled
Details
Deploy / deploy (windows-latest) (push) Has been cancelled
Details 
support safetensor load, delete architectures argument
 2025-05-12 11:10:26 +08:00
qiyuxinlin
c6aa379de2 support safetensor load, delete architectures argument 2025-05-09 10:38:29 +00:00
Atream
30eab48a75
Merge pull request #799 from aubreyli/cpu_offloading
Some checks failed
Book-CI / test (push) Has been cancelled
Details
Deploy / deploy (macos-latest) (push) Has been cancelled
Details
Deploy / deploy (ubuntu-latest) (push) Has been cancelled
Details
Deploy / deploy (windows-latest) (push) Has been cancelled
Details 
Restore CPU offloading capability
 2025-05-09 00:38:54 -06:00
Atream
b0318fc01c fix-hopper-flashinfer 2025-04-29 11:06:34 +08:00
qiyuxinlin
27990dc6fb fix load bug 2025-04-28 21:08:13 +00:00
djw
0da3792b27 support qwen3 2025-04-28 14:05:24 +00:00
djw
3f9bbf1181 support qwen3, dont speak human language 2025-04-28 08:44:47 +00:00
chenht2022
f3d842a0ca support AMX 2025-04-25 14:47:16 +00:00
root
921061666c fix some bugs 2025-04-17 00:48:09 +08:00
Atream
3b9e16cec7
Update attention.py 2025-04-09 10:54:00 +08:00
Aubrey Li
12a4c631df Fix TypeError when invoke KLinearCPUInfer.forward() 
Fix the following error:

  File "/home/aubrey/work/ktransformers/ktransformers/operators/linear.py", line 825, in forward
    y = self.generate_linear.forward(x, bsz_tensor)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: KLinearCPUInfer.forward() takes 2 positional arguments but 3 were given
 2025-04-07 12:03:35 +08:00
dongjw
1b7672937b update install doc and fix local_chat bug 2025-04-03 12:42:41 +08:00
Azure-Tang
31677181c3 Fix ktransformers-server flashinfer wrapper position arg issue; 
Fix db position issue
 2025-04-01 07:30:23 +00:00
Azure-Tang
203b853c75 rm KMoEGateDeepSeekV3, fall back to KMoEGate 2025-04-01 07:13:05 +00:00
Azure-Tang
3a5330b215 Merge branch 'main' into work-concurrent 2025-04-01 06:48:19 +00:00
Atream
25cee5810e add balance-serve, support concurrence 2025-03-31 22:55:32 +08:00
Aubrey Li
f4d52d1f0c Restore CPU offloading capability 2025-03-21 10:04:31 +08:00
Atream
633af5d235
Update gate.py 2025-03-20 14:54:01 +08:00
Atream
b453333f60
Update gate.py 2025-03-19 16:14:54 +08:00
Atream
44599229cd
Update gate.py 2025-03-19 12:16:48 +08:00
Atream
114995355b
fix-gate-compile 2025-03-19 11:27:18 +08:00
Atream
3934b9dfc1 rollback-triton-prefill 2025-03-15 14:21:21 +00:00
Azure
3986e2d2cf
Merge pull request #178 from fxzjshm/hip 
[Feat] Port to ROCm/HIP
 2025-03-15 02:31:07 +08:00
Azure-Tang
e5b001d76f Update readme; Format code; Add example yaml. 2025-03-14 14:25:52 -04:00
Atream
a889288fc1 use compile for gate, slight performance improvement 2025-03-14 12:43:28 +00:00
Azure-Tang
ed8437413b merge main; Add torch q8 linear 2025-03-14 05:52:07 -04:00
Atream
6f43bbe55f fix-singleton 2025-03-14 04:16:53 +00:00
Lander-Hatsune
d166fb9f6e cpuinfer: filter repeated backend instantiation 2025-03-10 22:03:04 +08:00
Atream
d453c320f1 fix flashinfer precision 2025-03-07 14:07:00 +00:00
Yi Pan
01755a60c0
fix: wrong shape in KLinearMarlin. 2025-03-03 17:34:45 +08:00
Atream
f35e8d41d8 support chunk prefill, support 139K context for 24G VRAM 2025-03-01 11:28:25 +00:00
Atream
e645d84794 use generation config from json file in official repo 2025-02-27 11:48:34 +00:00
liam
ffb86c66e3 ⚡ fix experts torch 2025-02-26 15:04:40 +08:00
Atream
477ac28a9c fix-update-flashinfer_wrapper_local_chat 2025-02-25 12:47:31 +00:00
Atream
b443c7dfa2
Merge pull request #657 from kvcache-ai/feat-absorb-for-long-prefill 
Feat absorb for long prefill
 2025-02-25 16:53:21 +08:00
Atream
f4c198bd42 support absorb for prefill long context 2025-02-25 08:52:02 +00:00
Azure
ca7366d2db Merge remote-tracking branch 'upstream/develop-0.2.2' into support-fp8 2025-02-24 11:58:10 +00:00
Azure
581a524f65 Add data loader to read special weights for fp8; Add special weight process script 2025-02-24 11:34:17 +00:00
Atream
e9b1216a9a Merge branch 'main' into feat-absorb-for-long-prefill 2025-02-24 09:44:17 +00:00
Atream
e8e02e5ccc support Moonlight 2025-02-23 14:21:18 +00:00
DDong Jianwei
95d937c51d tmp 2025-02-23 18:51:42 +08:00
Atream
006e8c6abc remove causal mask 2025-02-23 07:40:47 +00:00
Azure
7b7c6a657d Add fp8 linear kernel;\n Add empty cache to fit in 16G VRAM; By 'wkGCaSS - 知乎 https://zhuanlan.zhihu.com/p/25491611225' 2025-02-22 13:05:08 +00:00
Atream
024009675e Merge branch 'main' into feat-more-context 2025-02-22 06:17:39 +00:00
Atream
5ec33d046d optimize gguf dequant, save mem, support Q2_K 
use marlin for lm_head, lm_head only calc last token for prefill
extend context window to 19K for DeepSeek-V3/R1 within 24GB VRAM
 2025-02-22 06:13:01 +00:00
Atream
a529518346 clean PR code and disable flashinfer 2025-02-19 04:42:47 +00:00
ceerrep
2ffb43f974 fix: adapt prefix cache in forward_linux_flashinfer 2025-02-18 11:40:15 +08:00
ceerrep
bb1cadfff3 Merge branch 'fix_precision_MLA' of https://github.com/kvcache-ai/ktransformers into server-prefix-cache 2025-02-17 18:08:04 +08:00
Atream
038bc30888 fix precision bug imported by position_ids in 0.2.0 2025-02-17 09:23:14 +00:00
ceerrep
5ac266085e fix: use flash_attn for faster prefill 2025-02-17 00:11:24 +08:00