mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2025-09-07 13:09:50 +00:00
88 lines
No EOL
4.5 KiB
Markdown
88 lines
No EOL
4.5 KiB
Markdown
# Report
|
|
## Prerequisites
|
|
We run our best performance tests (V0.2) on <br>
|
|
CPU: Intel (R) Xeon (R) Gold 6454S 1T DRAM (2 NUMA nodes) <br>
|
|
GPU: 4090D 24G VRAM <br>
|
|
## Bench Result
|
|
### V0.2
|
|
#### Settings
|
|
- Model: DeepseekV3-q4km (int4)<br>
|
|
- CPU: cpu_model_name: Intel (R) Xeon (R) Gold 6454S, 32 cores per socket, 2 sockets, 2 numa nodes
|
|
- GPU: 4090D 24G VRAM
|
|
- We test after enough warm up
|
|
#### Memory consumption:
|
|
- Single socket: 382G DRAM, at least 12G VRAM
|
|
- Dual socket: 1T DRAM, at least 12G VRAM
|
|
|
|
#### Benchmark Results
|
|
|
|
"6 experts" case is part of V0.3's preview
|
|
|
|
| Prompt<br>(500 tokens) | Dual socket Ktrans (6 experts) | Dual socket Ktrans (8 experts) | Single socket Ktrans (6 experts) | Single socket Ktrans (8 experts)| llama.cpp (8 experts) |
|
|
| --- | --- | --- | --- | --- | --- |
|
|
| Prefill token/s | 97.32 | 82.94 | 65.14 | 54.21 | 10.31 |
|
|
| Decode token/s | 13.69 | 12.208 | 10.303 | 8.73 |4.51 |
|
|
|
|
**The highest speedup reaches up to <u>3.03x</u> in decoding and <u>9.44x</u> in prefill.**
|
|
|
|
### V0.3-Preview
|
|
#### Settings
|
|
- Model: DeepseekV3-BF16 (online quant into int8 for CPU and int4 for GPU)
|
|
- CPU: cpu_model_name: Intel (R) Xeon (R) Gold 6454S, 32 cores per socket, 2 socket, 2 numa nodes
|
|
- GPU: (1~4)x 4090D 24GVRAM (requires more VRAM for longer prompt)
|
|
|
|
#### Memory consumptions:
|
|
- 644GB DRAM, at least 12GB VRAM
|
|
|
|
#### Benchmark results
|
|
| Prompt length | 1K | 2K | 4K | 8K |
|
|
|---------------|-----|-----|-----|-----|
|
|
| KTrans (8 experts) Prefill token/s | 185.96 | 255.26 | 252.58 | 195.62 |
|
|
| KTrans (6 experts) Prefill token/s | 203.70 | 286.55 | 271.08 | 207.20 |
|
|
|
|
**The prefill of KTrans V0.3 is up to <u>3.45x</u> times faster than KTrans V0.2, and is up to <u>63.53x</u> times faster than llama.cpp.**
|
|
**The decoding speed is the same as KTrans V0.2 (6 experts version) so it is omitted**
|
|
|
|
The main acceleration comes from
|
|
- Intel AMX instruction set and our specially designed cache friendly memory layout
|
|
- Expert selection strategy that selects fewer experts based on offline profile results of out of domain data
|
|
|
|
|
|
*From our research on DeepSeekV2, DeepSeekV3 and DeepSeekR1,
|
|
when we slightly decrease the activation experts num in inference,
|
|
the output quality doesn't change. But the speed of decoding and prefill
|
|
is speed up which is inspiring. So our showcase makes use of this finding*
|
|
|
|
## How to Run
|
|
### V0.2 Showcase
|
|
#### Single socket version (32 cores)
|
|
Our local_chat test command is:
|
|
``` shell
|
|
git clone https://github.com/kvcache-ai/ktransformers.git
|
|
cd ktransformers
|
|
numactl -N 1 -m 1 python ./ktransformers/local_chat.py --model_path <your model path> --gguf_path <your gguf path> --prompt_file <your prompt txt file> --cpu_infer 33 --cache_lens 1536
|
|
<when you see chat, then press enter to load the text prompt_file>
|
|
```
|
|
\<your model path\> can be local or set from online hugging face like deepseek-ai/DeepSeek-V3. If online encounters connection problem, try use mirror (hf-mirror.com) <br>
|
|
\<your gguf path\> can also be online, but as its large we recommend you download it and quantize the model to what you want <br>
|
|
The command numactl -N 1 -m 1 aims to advoid data transfer between numa nodes
|
|
#### Dual socket version (64 cores)
|
|
Make suer before you install (use install.sh or `make dev_install`), setting the env var `USE_NUMA=1` by `export USE_NUMA=1` (if already installed, reinstall it with this env var set) <br>
|
|
Our local_chat test command is:
|
|
``` shell
|
|
git clone https://github.com/kvcache-ai/ktransformers.git
|
|
cd ktransformers
|
|
export USE_NUMA=1
|
|
make dev_install # or sh ./install.sh
|
|
python ./ktransformers/local_chat.py --model_path <your model path> --gguf_path <your gguf path> --prompt_file <your prompt txt file> --cpu_infer 65 --cache_lens 1536
|
|
<when you see chat, then press enter to load the text prompt_file>
|
|
```
|
|
The parameters' meaning is the same. But As we use dual socket, we set cpu_infer to 65
|
|
## Some Explanations
|
|
1. Also we want to make further use of our two NUMA nodes on Xeon Gold cpu.
|
|
To avoid the cost of data transfer between nodes, we "copy" the critical matrix on
|
|
both nodes which takes more memory consumption but accelerates the prefill and decoding process.
|
|
But this method takes huge memory and slow when loading weights, So be patient when loading
|
|
and monitor the memory usage. (we are considering to make this method as an option). We are going to optimize this huge memory overhead. Stay tuned~ <br>
|
|
2. The command args `--cpu_infer 65` specifies how many cores to use (it's ok that it exceeds the physical number,
|
|
but it's not the more the better. Adjust it slightly lower to your actual number of cores)<br> |