vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2025-09-11 01:24:36 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Rewrite history to fix bad vulkan shader commits without increasing repo size

Browse source 
added dpe colab (+8 squashed commit)

Squashed commit:

[b8362da4] updated lite

[ed6c037d] move nsigma into the regular sampler stack

[ac5f61c6] relative filepath fixed

[05fe96ab] export template

[ed0a5a3e] nix_example.md: refactor (#1401)

* nix_example.md: add override example

* nix_example.md: drop graphics example, already basic nixos knowledge

* nix_example.md: format

* nix_example.md: Vulkan is disabled on macOS

Disabled in: 1ccd253acc

* nix_examples.md: nixpkgs.config.cuda{Arches -> Capabilities}

Fixes: https://github.com/LostRuins/koboldcpp/issues/1367

[675c62f7] AutoGuess: Phi 4 (mini) (#1402)

[4bf56982] phrasing

[b8c0df04] Add Rep Pen to Top N Sigma sampler chain (#1397)

- place after nsigma and before xtc (+3 squashed commit)

Squashed commit:

[87c52b97] disable VMM from HIP

[ee8906f3] edit description

[e85c0e69] Remove Unnecessary Rep Counting (#1394)

* stop counting reps

* fix range-based initializer

* strike that - reverse it
This commit is contained in:
Concedo 2025-03-05 00:02:20 +08:00
parent 50eae1ffeb
commit 6b7d2349a7
114 changed files with 6666 additions and 2642 deletions
Download patch file Download diff file Expand all files Collapse all files

46
examples/nix_example.md
View file

@ -1,10 +1,13 @@
# Guide to Nix for KoboldCpp
- KoboldCpp is available on Nixpkgs and can be installed by adding just `koboldcpp` to your `environment.systemPackages` *(or it can also be placed in `home.packages`)*.
- KoboldCpp is available on Nixpkgs and can be installed by adding just
`koboldcpp` to your `environment.systemPackages` *(or it can also be placed
in `home.packages`)*.
## KoboldCpp Nix - CUDA Support
In order to enable NVIDIA CUDA support, you'll need to configure several settings:
In order to enable NVIDIA CUDA support, you'll need to configure several
settings:
- Enable required options:
@ -13,20 +16,10 @@ nixpkgs.config.allowUnfree = true; # Allow proprietary software
nixpkgs.config.cudaSupport = true; # Enable CUDA functionality
```
- Enable graphics support based on your NixOS version:
```nix
# For NixOS 24.05:
hardware.opengl.enable = true;
# For NixOS 24.11 or unstable:
hardware.graphics.enable = true;
```
- Set your GPU architecture:
```nix
nixpkgs.config.cudaArches = [ "sm_75" ]; # Example for RTX 2080
nixpkgs.config.cudaCapabilities = [ "sm_75" ]; # Example for RTX 2080
```
To find your GPU's architecture code:
@ -37,7 +30,7 @@ To find your GPU's architecture code:
## Hardware Support
- ✅ Vulkan: Enabled by default on Linux and macOS
- ✅ Vulkan: Enabled by default on Linux
- ✅ Metal: Enabled by default on macOS
- ❌ ROCm: Not currently available
@ -47,24 +40,31 @@ To find your GPU's architecture code:
nixpkgs.config = {
allowUnfree = true;
cudaSupport = true;
cudaArches = [ "sm_75" ];
cudaCapabilities = [ "sm_75" ];
};
# NixOS 24.05
hardware.opengl.enable = true;
# NixOS 24.11 or unstable
# hardware.graphics.enable = true;
environment.systemPackages = [ pkgs.koboldcpp ];
# If you're using home-manager to install KoboldCpp
# home.packages = [ pkgs.koboldcpp ];
# You can also just override koboldcpp to add your CUDA architecture:
# environment.systemPackages = [ (koboldcpp.override { cudaArches = ["sm_75"]; }) ]
# or
# home.packages = [ (koboldcpp.override { cudaArches = ["sm_75"]; }) ];
```
## KoboldCpp - Home Manager
The setup for Home Manager is the same as regular Nix, with one exception regarding Home Manager's instance of nixpkgs. By default, Home Manager manages its own isolated instance of nixpkgs, which has two implications:
The setup for Home Manager is the same as regular Nix, with one exception
regarding Home Manager's instance of nixpkgs. By default, Home Manager manages
its own isolated instance of nixpkgs, which has two implications:
1. You can keep your private Home Manager nixpkgs instance and simply repeat your `nixpkgs.config` in home manager.
2. You can set `home-manager.useGlobalPkgs = true;` to copy your module system's nixpkgs instance. This way, you only need to define it in your `configuration.nix`, and Home Manager will "inherit" this configuration.
1. You can keep your private Home Manager nixpkgs instance and simply repeat
your `nixpkgs.config` in home manager.
2. You can set `home-manager.useGlobalPkgs = true;` to copy your module
system's nixpkgs instance. This way, you only need to define it in your
`configuration.nix`, and Home Manager will "inherit" this configuration.
## Getting Help for KoboldCpp Nix
- If you face any issues with running KoboldCpp on Nix, please open an issue [here](https://github.com/NixOS/nixpkgs/issues/new?assignees=&labels=0.kind%3A+bug&projects=&template=bug_report.md&title=)
- If you face any issues with running KoboldCpp on Nix, please open an issue
[here](https://github.com/NixOS/nixpkgs/issues/new?assignees=&labels=0.kind%3A+bug&projects=&template=bug_report.md&title=)