mirror of
https://github.com/LostRuins/koboldcpp.git
synced 2025-09-11 09:34:37 +00:00
81ac0e5656
# Conflicts: # .devops/full-cuda.Dockerfile # .devops/full-rocm.Dockerfile # .devops/full.Dockerfile # .devops/llama-cpp-clblast.srpm.spec # .devops/llama-cpp-cuda.srpm.spec # .devops/llama-cpp.srpm.spec # .devops/nix/package.nix # .devops/server-cuda.Dockerfile # .devops/server-intel.Dockerfile # .devops/server-rocm.Dockerfile # .devops/server-vulkan.Dockerfile # .devops/server.Dockerfile # .github/workflows/build.yml # .github/workflows/code-coverage.yml # .github/workflows/docker.yml # .github/workflows/editorconfig.yml # .github/workflows/gguf-publish.yml # .github/workflows/nix-ci-aarch64.yml # .github/workflows/nix-ci.yml # .github/workflows/python-check-requirements.yml # .github/workflows/python-lint.yml # .github/workflows/server.yml # .github/workflows/zig-build.yml # CMakeLists.txt # Makefile # README-sycl.md # README.md # ci/run.sh # examples/gguf-split/gguf-split.cpp # flake.lock # flake.nix # llama.cpp # scripts/compare-llama-bench.py # scripts/sync-ggml-am.sh # scripts/sync-ggml.last # scripts/sync-ggml.sh # tests/CMakeLists.txt # tests/test-backend-ops.cpp # tests/test-chat-template.cpp |
||
|---|---|---|
| .. | ||
| CMakeLists.txt | ||
| quantize.cpp | ||
| README.md | ||
quantize
TODO
Llama 2 7B
| Quantization | Bits per Weight (BPW) |
|---|---|
| Q2_K | 3.35 |
| Q3_K_S | 3.50 |
| Q3_K_M | 3.91 |
| Q3_K_L | 4.27 |
| Q4_K_S | 4.58 |
| Q4_K_M | 4.84 |
| Q5_K_S | 5.52 |
| Q5_K_M | 5.68 |
| Q6_K | 6.56 |
Llama 2 13B
| Quantization | Bits per Weight (BPW) |
|---|---|
| Q2_K | 3.34 |
| Q3_K_S | 3.48 |
| Q3_K_M | 3.89 |
| Q3_K_L | 4.26 |
| Q4_K_S | 4.56 |
| Q4_K_M | 4.83 |
| Q5_K_S | 5.51 |
| Q5_K_M | 5.67 |
| Q6_K | 6.56 |
Llama 2 70B
| Quantization | Bits per Weight (BPW) |
|---|---|
| Q2_K | 3.40 |
| Q3_K_S | 3.47 |
| Q3_K_M | 3.85 |
| Q3_K_L | 4.19 |
| Q4_K_S | 4.53 |
| Q4_K_M | 4.80 |
| Q5_K_S | 5.50 |
| Q5_K_M | 5.65 |
| Q6_K | 6.56 |