vrr/kvcache-ai-ktransformers
2
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2025-09-11 07:44:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

fix local_chat and update balance-serve and SUMMARY doc

Browse source
This commit is contained in:
dongjw 2025-04-03 12:19:43 +08:00
parent 8acb270c90
commit ab0b0f4ea1
3 changed files with 6 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

1
doc/README.md
View file

@ -22,6 +22,7 @@ Our vision for KTransformers is to serve as a flexible platform for experimentin
<h2 id="Updates">🔥 Updates</h2> <h2 id="Updates">🔥 Updates</h2>
* **Apr 2, 2025**: Support Multi-concurrency. ([Tutorial](./en/balance-serve.md)).
* **Mar 27, 2025**: Support Multi-concurrency. * **Mar 27, 2025**: Support Multi-concurrency.
* **Mar 15, 2025**: Support ROCm on AMD GPU ([Tutorial](./en/ROCm.md)). * **Mar 15, 2025**: Support ROCm on AMD GPU ([Tutorial](./en/ROCm.md)).
* **Mar 5, 2025**: Support unsloth 1.58/2.51 bits weights and [IQ1_S/FP8 hybrid](./en/fp8_kernel.md) weights. Support 139K [Longer Context](./en/DeepseekR1_V3_tutorial.md#v022-longer-context) for DeepSeek-V3 and R1 in 24GB VRAM. * **Mar 5, 2025**: Support unsloth 1.58/2.51 bits weights and [IQ1_S/FP8 hybrid](./en/fp8_kernel.md) weights. Support 139K [Longer Context](./en/DeepseekR1_V3_tutorial.md#v022-longer-context) for DeepSeek-V3 and R1 in 24GB VRAM.

1
doc/SUMMARY.md
View file

@ -11,6 +11,7 @@
- [Multi-GPU Tutorial](en/multi-gpu-tutorial.md) - [Multi-GPU Tutorial](en/multi-gpu-tutorial.md)
- [Use FP8 GPU Kernel](en/fp8_kernel.md) - [Use FP8 GPU Kernel](en/fp8_kernel.md)
- [Use AMD GPU](en/ROCm.md) - [Use AMD GPU](en/ROCm.md)
- [Use Multi-concurrency](en/balance-serve.md)
# Server # Server
- [Server](en/api/server/server.md) - [Server](en/api/server/server.md)
- [Website](en/api/server/website.md) - [Website](en/api/server/website.md)

5
doc/en/balance-serve.md
View file

@ -74,6 +74,9 @@ strings ~/anaconda3/envs/ktransformers/lib/libstdc++.so.6 | grep GLIBCXX
```bash ```bash
sudo apt install libtbb-dev libssl-dev libcurl4-openssl-dev libaio1 libaio-dev libfmt-dev libgflags-dev zlib1g-dev patchelf sudo apt install libtbb-dev libssl-dev libcurl4-openssl-dev libaio1 libaio-dev libfmt-dev libgflags-dev zlib1g-dev patchelf
pip3 install packaging ninja cpufeature numpy openai
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
``` ```
### 3. Build ktransformers ### 3. Build ktransformers
@ -87,7 +90,7 @@ git submodule update --init --recursive
# Install single NUMA dependencies # Install single NUMA dependencies
USE_BALANCE_SERVE=1 bash ./install.sh USE_BALANCE_SERVE=1 bash ./install.sh
# Or Install Dual NUMA dependencies # For those who have two cpu and 1T RAM（Dual NUMA）:
USE_BALANCE_SERVE=1 USE_NUMA=1 bash ./install.sh USE_BALANCE_SERVE=1 USE_NUMA=1 bash ./install.sh
``` ```