46 commits

Author	SHA1	Message	Date
liam	04cebec4bb	⚡ rm opt config path default value and fix some config logic bug	2024-11-14 20:02:30 +08:00
liam	c2b4dc805c	🚑️:roll back transformer.py and find that it's multiple chat hsitory have minor accurate error	2024-11-04 14:02:19 +08:00
liam	a148da2cfe	✨: rm sensitive info in config.yaml, add readme of makefile. support old model_path config	2024-11-04 14:02:19 +08:00
anyanqilin	9a2e7057c8	wjh fix change	2024-11-04 14:02:19 +08:00
anyanqilin	a72dc6ed15	wjh change	2024-11-04 14:02:19 +08:00
anyanqilin	2d67016d14	wjh-change	2024-11-04 14:02:19 +08:00
liam	7c94df4bcf	🚑️: back transformer.py bugs version, and fix typo error in local_chat.py	2024-11-04 14:02:19 +08:00
liam	dd1d8667f3	✨: refactor local_chat and fix message slice bug in server	2024-11-04 14:02:19 +08:00
Chen Hongtao	43fc7f44a6	Merge pull request #99 from chenht2022/main ... Adapt Windows	2024-10-09 19:09:58 +08:00
chenht2022	14869b55ad	Adapt Windows	2024-10-09 11:08:32 +00:00
UnicornChan	a81a7ffe21	Merge pull request #77 from TKONIY/fix-prefill-and-generate ... Fix: Wrong type of token list returned by prefill_and_generate	2024-10-09 19:04:27 +08:00
Chen Hongtao	b4904537e3	Merge pull request #83 from sayap/task-queue-cond-var ... Use cond var to avoid busy loop	2024-10-09 18:57:17 +08:00
UnicornChan	43e8848ddd	Merge pull request #86 from xhedit/main ... typo fix: KMisrtal -> KMistral	2024-09-19 10:37:55 +08:00
Azure	3758afb526	fix some dequant function dosen't support multi gpu bug	2024-09-13 08:34:23 +00:00
xhedit	234faf7987	typo fix: KMisrtal -> KMistral	2024-09-12 15:58:01 +00:00
Yap Sok Ann	6666d62237	Use cond var to avoid busy loop	2024-09-11 16:10:54 +07:00
UnicornChan	3ed8a0437f	Merge pull request #72 from sayap/dequantize-iq4-xs ... Support IQ4_XS dequantize	2024-09-06 10:06:10 +08:00
yangshen	ee72cee050	Fix: the tokens return by prefill_and_generate	2024-09-05 05:29:23 +00:00
UnicornChan	be81269e38	Merge pull request #71 from Azure-Tang/main ... [fix] Fix qlen > chunk_size mask is none error	2024-09-02 14:51:05 +08:00
Azure	c55de02f7b	fix qlen > 1000 mask is none error	2024-09-02 02:58:10 +00:00
Yap Sok Ann	be356c1b8d	Support IQ4_XS dequantize	2024-09-02 09:10:19 +07:00
chenxl	49cce0c437	[fix] bugs about Qwen57B, install requirement, Dockerfile	2024-08-30 09:51:32 +00:00
TangJingqi	8747c099f2	update yaml example; update version idx; update docker file	2024-08-29 22:39:20 +08:00
TangJingqi	6735beb5b6	Fix cannot offload whole layer in cpu	2024-08-29 19:10:14 +08:00
TangJingqi	abd4214b56	fix readme; adjust param	2024-08-29 10:40:08 +08:00
chenxl	4d1d561d28	[feature] release 0.1.3	2024-08-28 16:11:43 +00:00
chenxl	b9f0819a86	None for load config	2024-08-22 15:52:25 +00:00
molamooo	29f4151ebc	[fix] f16 dequantize device ignored	2024-08-22 15:10:06 +08:00
TangJingqi	170b7a6001	fix server don't accept yaml path as param; fix server static cache device problem	2024-08-21 14:19:43 +08:00
TangJingqi	de3faaf55d	Update readme; add pipeline tutorial; add detailed inject tutorial	2024-08-15 20:42:54 +08:00
TangJingqi	c47205dce9	fix name	2024-08-15 11:25:12 +08:00
TangJingqi	67043b4b5c	[fix] format classes and files name	2024-08-15 10:44:59 +08:00
chenxl	1db4a67dca	[feature] add github action for pre compile	2024-08-14 16:54:50 +00:00
Atream	412055d450	[feature] experts can be injected using CPUInfer ... [fix] fix ktransformers interface when use new CUDAGraphRunner [fix] fix YAML and optimize logic, the top rule has the highest priority	2024-08-14 16:10:54 +08:00
BITcyman	7c4cb520bd	[feature] support q2_k & q3_k dequantize on gpu	2024-08-12 12:53:12 +00:00
chenxl	650c368c18	Merge remote-tracking branch 'upstream/main' into develop-0.1.2	2024-08-12 12:31:49 +00:00
Atream	3c675af61a	Update task_queue.h	2024-08-12 20:06:19 +08:00
chenxl	f5f79f5c0e	[ADD] support multi-gpu qlen>1 q5_k	2024-08-12 11:41:26 +00:00
chenxl	782a17e4e6	[feature] add bat for windows, update readme	2024-08-09 09:39:42 +00:00
chenht2022	c1cc7d2cd2	1) Linear and MLP operators support qlen>1; 2) All operators now share a single memory buffer; 3) Refactor CPUInfer submit/sync logic.	2024-08-08 09:04:36 +00:00
Atream	1f92f7cc61	[fix] linux and windows can all find CPUInfer in current Directory	2024-08-08 16:02:42 +08:00
Atream	0e613b602d	[fix] recover fp16 support	2024-08-08 15:42:21 +08:00
chenxl	1d9d397525	fix some bug in compile in linux	2024-08-08 15:34:19 +08:00
Atream	0a2fd52cea	support windows support q4_0 and q5_0 dequant on cpu Add CopyRight from pygguf(It was added before, but disappear after merge). Add some TODO in the code.	2024-08-08 15:34:02 +08:00
chenxl	112cb3c962	[feature] support python 310 and multi instruction	2024-07-31 13:58:17 +00:00
chenxl	18c42e67df	Initial commit	2024-07-27 16:06:58 +08:00