vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2025-09-10 17:14:36 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

API: add /props route (#1222)

Browse source 
* API: add an /extra/chat_template route

A lot of manual tweaking is done when swapping between models. We can automate or make better assumptions about some of them by having more information, such as chat template. This PR adds an endpoint /extra/chat_template which returns the model chat template string as is in a 'chat_template' key. The front end can then use this to derive the proper templates or use it as is, or at least warn the user when they are trying to use e.g. a Mistral preset with a Llama 3.1 model.

* switch to pre-established /props endpoint for chat template

* bug-fix (upstream): one-off in string juggling
This commit is contained in:
kallewoof 2024-11-21 11:58:32 +09:00 • committed by GitHub
parent 8ab3eb89a8
commit 547ab2aebb
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 35 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

10
expose.cpp
View file

@ -275,6 +275,16 @@ extern "C"
return (int)last_stop_reason;
}
const char* get_chat_template() {
// we need to keep this around
static std::string* ct = nullptr;
if (ct == nullptr) {
ct = new std::string();
}
*ct = gpttype_get_chat_template();
return ct->c_str();
}
const char* get_pending_output() {
return gpttype_get_pending_output().c_str();
}