vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-04-28 11:40:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

server : host-memory prompt caching (#16391)

Browse source 
* minor : code style

* server : fix prompt similarity calculation

* server : initial host-memory prompt caching

* cont

* server : refactor

* cont

* cont : make the server task of the slot const

* cont : minor [no ci]

* server : cache prompts and checkpoints only for completion tasks

* server : improve prompt caching logic

* cont : fix check for number of cached prompts [no ci]

* server : improve caching logic, add -cram CLI arg

* server : print prompt mismatch info

* cont : better naming [no ci]

* server : improve prompt cache loading logic

* server : add option to debug the slot contents (#16482)

* server : add option to debug the slot contents

* Update tools/server/server.cpp

---------

Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>

* server : add option to disable prompt cache

---------

Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
This commit is contained in:
Georgi Gerganov 2025-10-09 18:54:51 +03:00 • committed by GitHub
parent 8328fd4bae
commit d00cbea63c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
10 changed files with 813 additions and 471 deletions
Download patch file Download diff file Expand all files Collapse all files

3
tools/server/tests/unit/test_basic.py
View file

@ -66,8 +66,7 @@ def test_server_slots():
assert len(res.body) == server.n_slots
assert server.n_ctx is not None and server.n_slots is not None
assert res.body[0]["n_ctx"] == server.n_ctx / server.n_slots
assert "params" in res.body[0]
assert res.body[0]["params"]["seed"] == server.seed
assert "params" not in res.body[0]
def test_load_split_model():