mirror of https://github.com/LostRuins/koboldcpp.git synced 2025-09-07 15:49:05 +00:00

Concedo 6b7d2349a7 Rewrite history to fix bad vulkan shader commits without increasing repo size

added dpe colab (+8 squashed commit)

Squashed commit:

[b8362da4] updated lite

[ed6c037d] move nsigma into the regular sampler stack

[ac5f61c6] relative filepath fixed

[05fe96ab] export template

[ed0a5a3e] nix_example.md: refactor (#1401)

* nix_example.md: add override example

* nix_example.md: drop graphics example, already basic nixos knowledge

* nix_example.md: format

* nix_example.md: Vulkan is disabled on macOS

Disabled in: 1ccd253acc

* nix_examples.md: nixpkgs.config.cuda{Arches -> Capabilities}

Fixes: https://github.com/LostRuins/koboldcpp/issues/1367

[675c62f7] AutoGuess: Phi 4 (mini) (#1402)

[4bf56982] phrasing

[b8c0df04] Add Rep Pen to Top N Sigma sampler chain (#1397)

- place after nsigma and before xtc (+3 squashed commit)

Squashed commit:

[87c52b97] disable VMM from HIP

[ee8906f3] edit description

[e85c0e69] Remove Unnecessary Rep Counting (#1394)

* stop counting reps

* fix range-based initializer

* strike that - reverse it

2025-03-05 00:02:20 +08:00

2.3 KiB

Raw Permalink Blame History

Guide to Nix for KoboldCpp

KoboldCpp is available on Nixpkgs and can be installed by adding just koboldcpp to your environment.systemPackages (or it can also be placed in home.packages).

KoboldCpp Nix - CUDA Support

In order to enable NVIDIA CUDA support, you'll need to configure several settings:

Enable required options:

nixpkgs.config.allowUnfree = true;    # Allow proprietary software
nixpkgs.config.cudaSupport = true;    # Enable CUDA functionality

Set your GPU architecture:

nixpkgs.config.cudaCapabilities = [ "sm_75" ];  # Example for RTX 2080

To find your GPU's architecture code:

Visit the NVIDIA Architecture Guide
Locate your GPU Architecture
Use the corresponding sm_XX code in your configuration

Hardware Support

✅ Vulkan: Enabled by default on Linux
✅ Metal: Enabled by default on macOS
❌ ROCm: Not currently available

Example KoboldCpp Nix Configuration

nixpkgs.config = {
  allowUnfree = true;
  cudaSupport = true;
  cudaCapabilities = [ "sm_75" ];
};
environment.systemPackages = [ pkgs.koboldcpp ];
# If you're using home-manager to install KoboldCpp
# home.packages = [ pkgs.koboldcpp ];

# You can also just override koboldcpp to add your CUDA architecture:
# environment.systemPackages = [ (koboldcpp.override { cudaArches = ["sm_75"]; }) ]
# or
# home.packages = [ (koboldcpp.override { cudaArches = ["sm_75"]; }) ];

KoboldCpp - Home Manager

The setup for Home Manager is the same as regular Nix, with one exception regarding Home Manager's instance of nixpkgs. By default, Home Manager manages its own isolated instance of nixpkgs, which has two implications:

You can keep your private Home Manager nixpkgs instance and simply repeat your nixpkgs.config in home manager.
You can set home-manager.useGlobalPkgs = true; to copy your module system's nixpkgs instance. This way, you only need to define it in your configuration.nix, and Home Manager will "inherit" this configuration.

Getting Help for KoboldCpp Nix

If you face any issues with running KoboldCpp on Nix, please open an issue here

2.3 KiB Raw Permalink Blame History