vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2025-09-13 02:19:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Expand nix section (#1255)

Browse source 
* examples/nix_example.md: fix markdownlint complaints

* examples/nix_example.md: add home manager section

* examples/nix_example.md: add hardware support section

* examples/nix_example.md: rewrite CUDA section

---------

Co-authored-by: Concedo <39025047+LostRuins@users.noreply.github.com>
This commit is contained in:
DontEatOreo 2024-12-10 16:58:48 +02:00 • committed by GitHub
parent 00a686fc72
commit f6aebb1449
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 50 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

50
examples/nix_example.md
View file

@ -1,12 +1,48 @@
# Guide to Nix for KoboldCpp
- KoboldCpp is available on Nixpkgs and can be installed by adding just `koboldcpp` to your `environment.systemPackages` *(or it can also be placed in `home.packages`)*.
## KoboldCpp Nix - CUDA Support
- In order to enable NVIDIA CUDA support, you will need to set `nixpkgs.config.allowUnfree`, `hardware.opengl.enable` (*`hardware.graphics.enable` if you're using 24.11 or the unstable channel*) and `nixpkgs.config.cudaSupport` to `true`, and set `nixpkgs.config.cudaArches` *(e.g., if you have an RTX 2080, you need to set `cudaArches` to `[ "sm_75" ]`)* to your GPU architecture. Find your architecture here: [Matching CUDA arch and CUDA gencode for various NVIDIA architectures](https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/)
- Make sure to configure `nixpkgs` settings again/separately for home-manager if `home-manager.useGlobalPkgs` is set to `false`.
- Metal is enabled by default on macOS, Vulkan support is enabled by default on both Linux and macOS, ROCm support isn't available yet.
In order to enable NVIDIA CUDA support, you'll need to configure several settings:
- Enable required options:
```nix
nixpkgs.config.allowUnfree = true; # Allow proprietary software
nixpkgs.config.cudaSupport = true; # Enable CUDA functionality
```
- Enable graphics support based on your NixOS version:
```nix
# For NixOS 24.05:
hardware.opengl.enable = true;
# For NixOS 24.11 or unstable:
hardware.graphics.enable = true;
```
- Set your GPU architecture:
```nix
nixpkgs.config.cudaArches = [ "sm_75" ]; # Example for RTX 2080
```
To find your GPU's architecture code:
1. Visit the [NVIDIA Architecture Guide](https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/)
2. Locate your GPU Architecture
3. Use the corresponding `sm_XX` code in your configuration
## Hardware Support
- ✅ Vulkan: Enabled by default on Linux and macOS
- ✅ Metal: Enabled by default on macOS
- ❌ ROCm: Not currently available
## Example KoboldCpp Nix Configuration
```nix
nixpkgs.config = {
allowUnfree = true;
@ -22,5 +58,13 @@ environment.systemPackages = [ pkgs.koboldcpp ];
# home.packages = [ pkgs.koboldcpp ];
```
## KoboldCpp - Home Manager
The setup for Home Manager is the same as regular Nix, with one exception regarding Home Manager's instance of nixpkgs. By default, Home Manager manages its own isolated instance of nixpkgs, which has two implications:
1. You can keep your private Home Manager nixpkgs instance and simply repeat your `nixpkgs.config` in home manager.
2. You can set `home-manager.useGlobalPkgs = true;` to copy your module system's nixpkgs instance. This way, you only need to define it in your `configuration.nix`, and Home Manager will "inherit" this configuration.
## Getting Help for KoboldCpp Nix
- If you face any issues with running KoboldCpp on Nix, please open an issue [here](https://github.com/NixOS/nixpkgs/issues/new?assignees=&labels=0.kind%3A+bug&projects=&template=bug_report.md&title=)