mirror of
https://github.com/LostRuins/koboldcpp.git
synced 2025-09-08 08:09:06 +00:00
6b7d2349a7
added dpe colab (+8 squashed commit) Squashed commit: [b8362da4] updated lite [ed6c037d] move nsigma into the regular sampler stack [ac5f61c6] relative filepath fixed [05fe96ab] export template [ed0a5a3e] nix_example.md: refactor (#1401) * nix_example.md: add override example * nix_example.md: drop graphics example, already basic nixos knowledge * nix_example.md: format * nix_example.md: Vulkan is disabled on macOS Disabled in:1ccd253acc* nix_examples.md: nixpkgs.config.cuda{Arches -> Capabilities} Fixes: https://github.com/LostRuins/koboldcpp/issues/1367 [675c62f7] AutoGuess: Phi 4 (mini) (#1402) [4bf56982] phrasing [b8c0df04] Add Rep Pen to Top N Sigma sampler chain (#1397) - place after nsigma and before xtc (+3 squashed commit) Squashed commit: [87c52b97] disable VMM from HIP [ee8906f3] edit description [e85c0e69] Remove Unnecessary Rep Counting (#1394) * stop counting reps * fix range-based initializer * strike that - reverse it
70 lines
2.3 KiB
Markdown
70 lines
2.3 KiB
Markdown
# Guide to Nix for KoboldCpp
|
|
|
|
- KoboldCpp is available on Nixpkgs and can be installed by adding just
|
|
`koboldcpp` to your `environment.systemPackages` *(or it can also be placed
|
|
in `home.packages`)*.
|
|
|
|
## KoboldCpp Nix - CUDA Support
|
|
|
|
In order to enable NVIDIA CUDA support, you'll need to configure several
|
|
settings:
|
|
|
|
- Enable required options:
|
|
|
|
```nix
|
|
nixpkgs.config.allowUnfree = true; # Allow proprietary software
|
|
nixpkgs.config.cudaSupport = true; # Enable CUDA functionality
|
|
```
|
|
|
|
- Set your GPU architecture:
|
|
|
|
```nix
|
|
nixpkgs.config.cudaCapabilities = [ "sm_75" ]; # Example for RTX 2080
|
|
```
|
|
|
|
To find your GPU's architecture code:
|
|
|
|
1. Visit the [NVIDIA Architecture Guide](https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/)
|
|
2. Locate your GPU Architecture
|
|
3. Use the corresponding `sm_XX` code in your configuration
|
|
|
|
## Hardware Support
|
|
|
|
- ✅ Vulkan: Enabled by default on Linux
|
|
- ✅ Metal: Enabled by default on macOS
|
|
- ❌ ROCm: Not currently available
|
|
|
|
## Example KoboldCpp Nix Configuration
|
|
|
|
```nix
|
|
nixpkgs.config = {
|
|
allowUnfree = true;
|
|
cudaSupport = true;
|
|
cudaCapabilities = [ "sm_75" ];
|
|
};
|
|
environment.systemPackages = [ pkgs.koboldcpp ];
|
|
# If you're using home-manager to install KoboldCpp
|
|
# home.packages = [ pkgs.koboldcpp ];
|
|
|
|
# You can also just override koboldcpp to add your CUDA architecture:
|
|
# environment.systemPackages = [ (koboldcpp.override { cudaArches = ["sm_75"]; }) ]
|
|
# or
|
|
# home.packages = [ (koboldcpp.override { cudaArches = ["sm_75"]; }) ];
|
|
```
|
|
|
|
## KoboldCpp - Home Manager
|
|
|
|
The setup for Home Manager is the same as regular Nix, with one exception
|
|
regarding Home Manager's instance of nixpkgs. By default, Home Manager manages
|
|
its own isolated instance of nixpkgs, which has two implications:
|
|
|
|
1. You can keep your private Home Manager nixpkgs instance and simply repeat
|
|
your `nixpkgs.config` in home manager.
|
|
2. You can set `home-manager.useGlobalPkgs = true;` to copy your module
|
|
system's nixpkgs instance. This way, you only need to define it in your
|
|
`configuration.nix`, and Home Manager will "inherit" this configuration.
|
|
|
|
## Getting Help for KoboldCpp Nix
|
|
|
|
- If you face any issues with running KoboldCpp on Nix, please open an issue
|
|
[here](https://github.com/NixOS/nixpkgs/issues/new?assignees=&labels=0.kind%3A+bug&projects=&template=bug_report.md&title=)
|