vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-08-21 22:35:48 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Add Mistral [THINK]/[/THINK] thinking format (mistral3 arch) (#2380)

Browse source 
The reasoning budget derived from reasoning_effort never applied to Mistral
models. gpttype_adapter.cpp picks the think delimiters from a switch on the
model architecture, and mistral3 has no case, so it falls back to <think> /
</think>. Those are not vocabulary tokens for Ministral-3, so TokenizeString
returns more than one token each, the expected_start/end_tokens guard clears
all three vectors, and apply_reasoning_budget() returns at its first if.
The parameter is accepted, converted and passed down to the sampler, then
dropped on a size check, with nothing logged.

Adding the mistral3 case arms the budget. [THINK] and [/THINK] are single
vocabulary tokens (ids 34 and 35 on Ministral-3), so the size guard passes.

The thinkformats entry is a separate fix for a separate defect: without it the
thinking block was never split out, so it leaked into content with its [THINK]
marker still in it, instead of going to reasoning_content.

Measured on Ministral-3-14B-Reasoning-2512 (IQ4_XS, ctx 8192, --jinja), 5 real
prompts x 3 samples per cell, max_tokens 3000 (so a 750-token budget at "low"):

  reasoning_effort   thinking words before      thinking words after
  none               311 - 2314                 7 (the forced-close phrase)
  low                340 - 2255                 521 - 574

Forced closes: 0/15 before, 14/15 after at "low" and 15/15 at "none". Three
samples per cell because this model's variance at temperature 0.7 spans a
factor of 4 on an identical payload — a single sample per cell cannot tell an
effect from noise.

No regression on a non-reasoning mistral3 model: Ministral-3-8B-Instruct with
reasoning_effort "low" returns finish_reason "stop", a normal answer and zero
forced closes, since apply_reasoning_budget() bails out when the start marker
never appears.
This commit is contained in:
Julien BODIN 2026-08-05 12:51:55 +02:00 • committed by GitHub
parent 4dc9df05f6
commit 152e080b6a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 7 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

5
gpttype_adapter.cpp
View file

@ -5859,6 +5859,11 @@ generation_outputs gpttype_generate(const generation_inputs inputs)
end = "<|END_THINKING|>";
budget_exceeded = "\n(Reasoning budget exceeded)\nTime to respond now.\n<|END_THINKING|>";
break;
case llm_arch::LLM_ARCH_MISTRAL3:
start = "[THINK]";
end = "[/THINK]";
budget_exceeded = "\n(Reasoning budget exceeded)\nTime to respond now.\n[/THINK]";
break;
default:
break;
}

3
koboldcpp.py
View file

@ -186,7 +186,8 @@ thinkformats = [{"start":"<|channel|>analysis<|message|>","end":"<|start|>assist
{"start":"<think>","end":"</think>"},
{"start":"<seed:think>","end":"</seed:think>"},
{"start":"<|START_THINKING|>","end":"<|END_THINKING|>"},
{"start":"<|channel>thought","end":"<channel|>"}]
{"start":"<|channel>thought","end":"<channel|>"},
{"start":"[THINK]","end":"[/THINK]"}]
tool_call_pairs = [ #third element is optional str to match in chat template before we use this pair, fourth element is whether its stream-handleable
("<tool_call>", "</tool_call>", None, True), #qwen, glm
("<seed:tool_call>", "</seed:tool_call>", None, True), #seed oss