vrr/kvcache-ai-ktransformers
2
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2025-09-10 23:34:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

support windows support q4_0 and q5_0 dequant on cpu Add CopyRight from pygguf(It was added before, but disappear after merge). Add some TODO in the code.

Browse source
This commit is contained in:
Atream 2024-08-07 12:19:06 +08:00
parent 442e13bc97
commit 0a2fd52cea
32 changed files with 248 additions and 108 deletions
Download patch file Download diff file Expand all files Collapse all files

2
ktransformers/local_chat.py
View file

@ -112,4 +112,4 @@ def local_chat(
generated = prefill_and_generate(model, tokenizer, input_tensor.cuda(), max_new_tokens)
if __name__ == "__main__":
fire.Fire(local_chat)
fire.Fire(local_chat)