vrr/kvcache-ai-ktransformers
2
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2025-09-15 09:39:42 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
857 commits 42 branches 19 tags 46 MiB
Commit graph

323 commits

Author SHA1 Message Date
TangJingqi
6735beb5b6 Fix cannot offload whole layer in cpu 2024-08-29 19:10:14 +08:00
TangJingqi
abd4214b56 fix readme; adjust param 2024-08-29 10:40:08 +08:00
chenxl
4d1d561d28 [feature] release 0.1.3 2024-08-28 16:11:43 +00:00
chenxl
b9f0819a86 None for load config 2024-08-22 15:52:25 +00:00
molamooo
29f4151ebc
[fix] f16 dequantize device ignored 2024-08-22 15:10:06 +08:00
TangJingqi
170b7a6001 fix server don't accept yaml path as param; fix server static cache device problem 2024-08-21 14:19:43 +08:00
TangJingqi
de3faaf55d Update readme; add pipeline tutorial; add detailed inject tutorial 2024-08-15 20:42:54 +08:00
TangJingqi
c47205dce9 fix name 2024-08-15 11:25:12 +08:00
TangJingqi
67043b4b5c [fix] format classes and files name 2024-08-15 10:44:59 +08:00
chenxl
1db4a67dca [feature] add github action for pre compile 2024-08-14 16:54:50 +00:00
Atream
412055d450 [feature] experts can be injected using CPUInfer 
[fix] fix ktransformers interface when use new CUDAGraphRunner
[fix] fix YAML and optimize logic, the top rule has the highest priority
 2024-08-14 16:10:54 +08:00
BITcyman
7c4cb520bd [feature] support q2_k & q3_k dequantize on gpu 2024-08-12 12:53:12 +00:00
chenxl
650c368c18 Merge remote-tracking branch 'upstream/main' into develop-0.1.2 2024-08-12 12:31:49 +00:00
Atream
3c675af61a
Update task_queue.h 2024-08-12 20:06:19 +08:00
chenxl
f5f79f5c0e [ADD] support multi-gpu qlen>1 q5_k 2024-08-12 11:41:26 +00:00
chenxl
782a17e4e6 [feature] add bat for windows, update readme 2024-08-09 09:39:42 +00:00
chenht2022
c1cc7d2cd2 1) Linear and MLP operators support qlen>1; 2) All operators now share a single memory buffer; 3) Refactor CPUInfer submit/sync logic. 2024-08-08 09:04:36 +00:00
Atream
1f92f7cc61 [fix] linux and windows can all find CPUInfer in current Directory 2024-08-08 16:02:42 +08:00
Atream
0e613b602d [fix] recover fp16 support 2024-08-08 15:42:21 +08:00
chenxl
1d9d397525 fix some bug in compile in linux 2024-08-08 15:34:19 +08:00
Atream
0a2fd52cea support windows support q4_0 and q5_0 dequant on cpu Add CopyRight from pygguf(It was added before, but disappear after merge). Add some TODO in the code. 2024-08-08 15:34:02 +08:00
chenxl
112cb3c962 [feature] support python 310 and multi instruction 2024-07-31 13:58:17 +00:00
chenxl
18c42e67df Initial commit 2024-07-27 16:06:58 +08:00