25 commits

Author	SHA1	Message	Date
qiyuxinlin	c6aa379de2	support safetensor load, delete architectures argument	2025-05-09 10:38:29 +00:00
root	921061666c	fix some bugs	2025-04-17 00:48:09 +08:00
Aubrey Li	12a4c631df	Fix TypeError when invoke KLinearCPUInfer.forward() ... Fix the following error: File "/home/aubrey/work/ktransformers/ktransformers/operators/linear.py", line 825, in forward y = self.generate_linear.forward(x, bsz_tensor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: KLinearCPUInfer.forward() takes 2 positional arguments but 3 were given	2025-04-07 12:03:35 +08:00
Atream	25cee5810e	add balance-serve, support concurrence	2025-03-31 22:55:32 +08:00
Azure	3986e2d2cf	Merge pull request #178 from fxzjshm/hip ... [Feat] Port to ROCm/HIP	2025-03-15 02:31:07 +08:00
Azure-Tang	e5b001d76f	Update readme; Format code; Add example yaml.	2025-03-14 14:25:52 -04:00
Atream	a889288fc1	use compile for gate, slight performance improvement	2025-03-14 12:43:28 +00:00
Azure-Tang	ed8437413b	merge main; Add torch q8 linear	2025-03-14 05:52:07 -04:00
Atream	6f43bbe55f	fix-singleton	2025-03-14 04:16:53 +00:00
Yi Pan	01755a60c0	fix: wrong shape in KLinearMarlin.	2025-03-03 17:34:45 +08:00
Azure	ca7366d2db	Merge remote-tracking branch 'upstream/develop-0.2.2' into support-fp8	2025-02-24 11:58:10 +00:00
Azure	581a524f65	Add data loader to read special weights for fp8; Add special weight process script	2025-02-24 11:34:17 +00:00
Azure	7b7c6a657d	Add fp8 linear kernel;\n Add empty cache to fit in 16G VRAM; By 'wkGCaSS - 知乎 https://zhuanlan.zhihu.com/p/25491611225'	2025-02-22 13:05:08 +00:00
Atream	5ec33d046d	optimize gguf dequant, save mem, support Q2_K ... use marlin for lm_head, lm_head only calc last token for prefill extend context window to 19K for DeepSeek-V3/R1 within 24GB VRAM	2025-02-22 06:13:01 +00:00
Atream	038bc30888	fix precision bug imported by position_ids in 0.2.0	2025-02-17 09:23:14 +00:00
Atream	c189d55bd1	toy support for experts on GPU, no CUDA Graph	2025-02-15 15:16:00 +00:00
liam	c74453d8ca	📝 add doc support and fix bug in qwen2	2025-02-13 16:37:43 +08:00
Azure	c4d9bc6670	support KExpertsMarlin backend	2025-02-07 05:57:40 +00:00
Azure	f748cd29f0	fix rope; update moegate	2025-02-01 18:05:45 +00:00
Azure	476b1d8dc6	support deepseekv3; runable but have precition problem	2025-01-31 08:27:24 +00:00
TangJingqi	6735beb5b6	Fix cannot offload whole layer in cpu	2024-08-29 19:10:14 +08:00
TangJingqi	67043b4b5c	[fix] format classes and files name	2024-08-15 10:44:59 +08:00
Atream	412055d450	[feature] experts can be injected using CPUInfer ... [fix] fix ktransformers interface when use new CUDAGraphRunner [fix] fix YAML and optimize logic, the top rule has the highest priority	2024-08-14 16:10:54 +08:00
chenxl	f5f79f5c0e	[ADD] support multi-gpu qlen>1 q5_k	2024-08-12 11:41:26 +00:00
chenxl	18c42e67df	Initial commit	2024-07-27 16:06:58 +08:00