mirror of
https://github.com/LostRuins/koboldcpp.git
synced 2026-10-03 11:35:46 +00:00
160bd031b2
* server: fix LRU hang on multiple requests same model * server: keep a queued model out of the victim pool until its waiters leave A waiter that gave up while its model was still loading left the model idle with no request behind it, and nothing recounted the free slots, so a second request queued behind it stayed queued forever. tick() was only driven by requests: join, claim and the end of a proxied request. Keep the queue entry alive after a successful claim so the model coming up is never picked as a victim before its waiters use it, and recount the slots on every status change and whenever a waiter abandons the queue. The model is then evicted as soon as it comes up with nobody left to serve. --------- Co-authored-by: Pascal <admin@serveurperso.com> |
||
|---|---|---|
| .. | ||
| batched-bench | ||
| cli | ||
| completion | ||
| cvector-generator | ||
| export-lora | ||
| fit-params | ||
| gguf-split | ||
| imatrix | ||
| llama-bench | ||
| mtmd | ||
| perplexity | ||
| quantize | ||
| results | ||
| rpc | ||
| server | ||
| tokenize | ||
| tts | ||
| tuning | ||
| ui | ||
| CMakeLists.txt | ||