vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-05-10 04:00:53 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

move nix example to a standalone file (+1 squashed commits)

Browse source 
Squashed commits:

[fef38bab5] move nix example to a standalone file
This commit is contained in:
Concedo 2024-11-26 16:27:03 +08:00
parent d82d7b60e6
commit 5b35790343
2 changed files with 27 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

23
README.md
View file

@ -129,28 +129,7 @@ when you can't use the precompiled binary directly, we provide an automated buil
- Arch Linux Packages: [CUBLAS](https://aur.archlinux.org/packages/koboldcpp-cuda), and [HIPBLAS](https://aur.archlinux.org/packages/koboldcpp-hipblas).
- Unofficial Dockers: [korewaChino](https://github.com/korewaChino/koboldCppDocker) and [noneabove1182](https://github.com/noneabove1182/koboldcpp-docker)
- Nix & NixOS: KoboldCpp is available on Nixpkgs and can be installed by adding just `koboldcpp` to your `environment.systemPackages` *(or it can also be placed in `home.packages`)*.
- In order to enable NVIDIA CUDA support, you will need to set `nixpkgs.config.allowUnfree`, `hardware.opengl.enable` (*`hardware.graphics.enable` if you're using 24.11 or the unstable channel*) and `nixpkgs.config.cudaSupport` to `true`, and set `nixpkgs.config.cudaArches` *(e.g., if you have an RTX 2080, you need to set `cudaArches` to `[ "sm_75" ]`)* to your GPU architecture. Find your architecture here: [Matching CUDA arch and CUDA gencode for various NVIDIA architectures](https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/)
- Make sure to configure `nixpkgs` settings again/separately for home-manager if `home-manager.useGlobalPkgs` is set to `false`.
- Metal is enabled by default on macOS, Vulkan support is enabled by default on both Linux and macOS, ROCm support isn't available yet.
- Example Nix Setup:
```nix
nixpkgs.config = {
allowUnfree = true;
cudaSupport = true;
cudaArches = [ "sm_75" ];
};
# NixOS 24.05
hardware.opengl.enable = true;
# NixOS 24.11 or unstable
# hardware.graphics.enable = true;
environment.systemPackages = [ pkgs.koboldcpp ];
# If you're using home-manager to install KoboldCpp
# home.packages = [ pkgs.koboldcpp ];
```
- [Example Nix Setup and further information](examples/nix_example.md)
- If you face any issues with running KoboldCpp on Nix, please open an issue [here](https://github.com/NixOS/nixpkgs/issues/new?assignees=&labels=0.kind%3A+bug&projects=&template=bug_report.md&title=).
- [GPTLocalhost](https://gptlocalhost.com/demo#KoboldCpp) - KoboldCpp is supported by GPTLocalhost, a local Word Add-in for you to use KoboldCpp in Microsoft Word. A local alternative to "Copilot in Word."

26
examples/nix_example.md Normal file
View file

@ -0,0 +1,26 @@
# Guide to Nix for KoboldCpp
- KoboldCpp is available on Nixpkgs and can be installed by adding just `koboldcpp` to your `environment.systemPackages` *(or it can also be placed in `home.packages`)*.
## KoboldCpp Nix - CUDA Support
- In order to enable NVIDIA CUDA support, you will need to set `nixpkgs.config.allowUnfree`, `hardware.opengl.enable` (*`hardware.graphics.enable` if you're using 24.11 or the unstable channel*) and `nixpkgs.config.cudaSupport` to `true`, and set `nixpkgs.config.cudaArches` *(e.g., if you have an RTX 2080, you need to set `cudaArches` to `[ "sm_75" ]`)* to your GPU architecture. Find your architecture here: [Matching CUDA arch and CUDA gencode for various NVIDIA architectures](https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/)
- Make sure to configure `nixpkgs` settings again/separately for home-manager if `home-manager.useGlobalPkgs` is set to `false`.
- Metal is enabled by default on macOS, Vulkan support is enabled by default on both Linux and macOS, ROCm support isn't available yet.
## Example KoboldCpp Nix Configuration
```nix
nixpkgs.config = {
allowUnfree = true;
cudaSupport = true;
cudaArches = [ "sm_75" ];
};
# NixOS 24.05
hardware.opengl.enable = true;
# NixOS 24.11 or unstable
# hardware.graphics.enable = true;
environment.systemPackages = [ pkgs.koboldcpp ];
# If you're using home-manager to install KoboldCpp
# home.packages = [ pkgs.koboldcpp ];
```
## Getting Help for KoboldCpp Nix
- If you face any issues with running KoboldCpp on Nix, please open an issue [here](https://github.com/NixOS/nixpkgs/issues/new?assignees=&labels=0.kind%3A+bug&projects=&template=bug_report.md&title=)