vrr/kvcache-ai-ktransformers
2
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2025-09-09 05:54:06 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

[feature] update docker image and entrypoint

Browse source
This commit is contained in:
chenxl 2025-02-15 07:55:33 +00:00
parent 1548c99234
commit 0e4b7a3929
3 changed files with 9 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

2
Dockerfile
View file

@ -32,4 +32,4 @@ CPU_INSTRUCT=NATIVE KTRANSFORMERS_FORCE_BUILD=TRUE TORCH_CUDA_ARCH_LIST="8.0;8.
pip cache purge pip cache purge
EOF EOF
ENTRYPOINT [ "/opt/conda/bin/ktransformers" ] ENTRYPOINT ["tail", "-f", "/dev/null"]

8
doc/en/Docker.md
View file

@ -7,7 +7,7 @@
## Images ## Images
There is a Docker image available for our project, you can pull the docker image by： There is a Docker image available for our project, you can pull the docker image by：
``` ```
docker pull approachingai/ktransformers:0.1.1 docker pull approachingai/ktransformers:0.2.1
``` ```
**Notice**: In this image, we compile the ktransformers in AVX512 instuction CPUs, if your cpu not support AVX512, it is suggested to recompile and install ktransformer in the /workspace/ktransformers directory within the container. **Notice**: In this image, we compile the ktransformers in AVX512 instuction CPUs, if your cpu not support AVX512, it is suggested to recompile and install ktransformer in the /workspace/ktransformers directory within the container.
@ -16,14 +16,16 @@ docker pull approachingai/ktransformers:0.1.1
- finish, execute - finish, execute
```bash ```bash
docker build -t approachingai/ktransformers:v0.1.1 . docker build -t approachingai/ktransformers:0.2.1 .
``` ```
## Usage ## Usage
Assuming you have the [nvidia-container-toolkit](https://github.com/NVIDIA/nvidia-container-toolkit) that you can use the GPU in a Docker container. Assuming you have the [nvidia-container-toolkit](https://github.com/NVIDIA/nvidia-container-toolkit) that you can use the GPU in a Docker container.
``` ```
docker run --gpus all -v /path/to/models:/models -p 10002:10002 approachingai/ktransformers:v0.1.1 --port 10002 --gguf_path /models/path/to/gguf_path --model_path /models/path/to/model_path --web True docker run --gpus all -v /path/to/models:/models --name ktransformers -itd approachingai/ktransformers:0.2.1
docker exec -it ktransformers /bin/bash
python -m ktransformers.local_chat --gguf_path /models/path/to/gguf_path --model_path /models/path/to/model_path --cpu_infer 33
``` ```
More operators you can see in the [readme](../../README.md) More operators you can see in the [readme](../../README.md)

6
ktransformers/__init__.py
View file

@ -5,7 +5,7 @@ Description :
Author : kkk1nak0 Author : kkk1nak0
Date : 2024-08-15 07:34:46 Date : 2024-08-15 07:34:46
Version : 1.0.0 Version : 1.0.0
LastEditors : unicornchan LastEditors : chenxl
LastEditTime : 2025-02-10 00:59:53 LastEditTime : 2025-02-15 03:53:02
''' '''
__version__ = "0.2.0" __version__ = "0.2.1"