vrr/kvcache-ai-ktransformers
2
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2025-09-06 12:40:02 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Merge pull request #1301 from kvcache-ai/update-readme
Some checks failed
Book-CI / test (push) Has been cancelled
Details
Deploy / deploy (macos-latest) (push) Has been cancelled
Details
Deploy / deploy (ubuntu-latest) (push) Has been cancelled
Details
Deploy / deploy (windows-latest) (push) Has been cancelled
Details

Browse source 
update readme
This commit is contained in:
wang jiahao 2025-05-14 21:15:55 +08:00 • committed by GitHub
commit 2d3aaef8b6
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 8 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
doc/en/balance-serve.md
View file

@ -100,8 +100,10 @@ git submodule update --init --recursive
# Install single NUMA dependencies
USE_BALANCE_SERVE=1 bash ./install.sh
pip install third_party/custom_flashinfer/
# For those who have two cpu and 1T RAM（Dual NUMA）:
USE_BALANCE_SERVE=1 USE_NUMA=1 bash ./install.sh
pip install third_party/custom_flashinfer/
```
## Running DeepSeek-R1-Q4KM Models

2
doc/en/install.md
View file

@ -117,11 +117,13 @@ Download source code and compile:
```shell
USE_BALANCE_SERVE=1 bash ./install.sh
pip install third_party/custom_flashinfer/
```
- For Multi-concurrency with two cpu and 1T RAM:
```shell
USE_BALANCE_SERVE=1 USE_NUMA=1 bash ./install.sh
pip install third_party/custom_flashinfer/
```
- For Windows (Windows native temporarily deprecated, please try WSL)

2
doc/en/llama4.md
View file

@ -68,8 +68,10 @@ pip3 install torch torchvision torchaudio --index-url https://download.pytorch.o
```bash
# Install single NUMA dependencies
USE_BALANCE_SERVE=1 bash ./install.sh
pip install third_party/custom_flashinfer/
# For those who have two cpu and 1T RAM（Dual NUMA）:
USE_BALANCE_SERVE=1 USE_NUMA=1 bash ./install.sh
pip install third_party/custom_flashinfer/
```
### 4. Use our custom config.json

2
doc/zh/DeepseekR1_V3_tutorial_zh.md
View file

@ -127,8 +127,10 @@ cd ktransformers
git submodule update --init --recursive
# 如果使用双 numa 版本
USE_BALANCE_SERVE=1 USE_NUMA=1 bash ./install.sh
pip install third_party/custom_flashinfer/
# 如果使用单 numa 版本
USE_BALANCE_SERVE=1 bash ./install.sh
pip install third_party/custom_flashinfer/
# 启动命令
python ktransformers/server/main.py --model_path <your model path> --gguf_path <your gguf path> --cpu_infer 62 --optimize_config_path <inject rule path> --port 10002 --chunk_size 256 --max_new_tokens 1024 --max_batch_size 4 --port 10002 --cache_lens 32768 --backend_type balance_serve
```