vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-07-10 01:18:32 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
koboldcpp/models
History
Aldehir Rojas c818263f2a
chat : implement minicpm5 parser (#24889) 
* Add minicpm5 tool call parser

* Refactor MiniCPM5 PEG parser per review feedback

* Fix jinja min/max API to match Jinja2

* modify by review

* MiniCPM5: use autoparser for XML tool calls and fix grammar preserved-token triggers

* MiniCPM5: fix streaming tool-arg placeholder and remove alt XML markers

* skip min/max attribute tests in -py mode

* test-jinja: use real expected output for min/max attribute tests

* MiniCPM5: revert shared mapper and history fallbacks per review

Drop streaming tool-arg placeholder workarounds from the generic PEG
mapper and restore strict tool-call argument JSON parsing so MiniCPM5
support stays limited to autoparser/diff-analyzer changes.

* chat : refactor minicpm5 back to dedicated parser

* cont : simplify grammar

* cont : refactor

* cont : fixes

* cont : rename template to openbmb-MiniCPM5-1B.jinja

* cont : add message delimiters

* cont : fix tests

---------

Co-authored-by: zhangtao <zhangtao2@modelbest.cn>
Co-authored-by: 张涛 <>
2026-06-28 16:53:32 +02:00
..
templates chat : implement minicpm5 parser (#24889) 2026-06-28 16:53:32 +02:00
.editorconfig gguf : new file format with flexible meta data (beta) (#2398) 2023-08-21 23:07:43 +03:00
ggml-vocab-aquila.gguf Work on the BPE tokenizer (#3252) 2023-10-03 09:16:26 +02:00
ggml-vocab-baichuan.gguf Add more tokenizer tests (#3742) 2023-10-24 09:17:17 +02:00
ggml-vocab-bert-bge.gguf llama : fix BPE pre-tokenization (#6920) 2024-04-29 16:58:41 +03:00
ggml-vocab-bert-bge.gguf.inp convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-bert-bge.gguf.out convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-command-r.gguf command-r : add BPE pre-tokenization (#7063) 2024-05-05 08:19:30 +03:00
ggml-vocab-command-r.gguf.inp convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-command-r.gguf.out convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-deepseek-coder.gguf llama : fix BPE pre-tokenization (#6920) 2024-04-29 16:58:41 +03:00
ggml-vocab-deepseek-coder.gguf.inp convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-deepseek-coder.gguf.out convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-deepseek-llm.gguf llama : fix BPE pre-tokenization (#6920) 2024-04-29 16:58:41 +03:00
ggml-vocab-deepseek-llm.gguf.inp convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-deepseek-llm.gguf.out convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-falcon.gguf llama : fix BPE pre-tokenization (#6920) 2024-04-29 16:58:41 +03:00
ggml-vocab-falcon.gguf.inp convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-falcon.gguf.out convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-gemma-4.gguf vocab: add gemma4 tokenizer tests, fix edge case (#21534) 2026-04-09 11:41:14 +02:00
ggml-vocab-gemma-4.gguf.inp vocab: add gemma4 tokenizer tests, fix edge case (#21534) 2026-04-09 11:41:14 +02:00
ggml-vocab-gemma-4.gguf.out vocab: add gemma4 tokenizer tests, fix edge case (#21534) 2026-04-09 11:41:14 +02:00
ggml-vocab-gpt-2.gguf llama : fix BPE pre-tokenization (#6920) 2024-04-29 16:58:41 +03:00
ggml-vocab-gpt-2.gguf.inp convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-gpt-2.gguf.out convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-gpt-neox.gguf Add more tokenizer tests (#3742) 2023-10-24 09:17:17 +02:00
ggml-vocab-llama-bpe.gguf llama : fix BPE pre-tokenization (#6920) 2024-04-29 16:58:41 +03:00
ggml-vocab-llama-bpe.gguf.inp convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-llama-bpe.gguf.out convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-llama-spm.gguf llama : fix BPE pre-tokenization (#6920) 2024-04-29 16:58:41 +03:00
ggml-vocab-llama-spm.gguf.inp convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-llama-spm.gguf.out convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-mpt.gguf llama : fix BPE pre-tokenization (#6920) 2024-04-29 16:58:41 +03:00
ggml-vocab-mpt.gguf.inp convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-mpt.gguf.out convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-nomic-bert-moe.gguf tests : improve UGM tokenizer test coverage (#13773) 2025-05-25 16:22:29 +02:00
ggml-vocab-phi-3.gguf Per token attributes (#7685) 2024-06-04 09:17:17 +02:00
ggml-vocab-phi-3.gguf.inp convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-phi-3.gguf.out convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-qwen2.gguf llama : add BPE pre-tokenization for Qwen2 (#7114) 2024-05-08 15:06:43 +03:00
ggml-vocab-qwen2.gguf.inp convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-qwen2.gguf.out convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-qwen35.gguf unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regr… (#22110) 2026-05-14 11:03:40 +02:00
ggml-vocab-qwen35.gguf.inp unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regr… (#22110) 2026-05-14 11:03:40 +02:00
ggml-vocab-qwen35.gguf.out unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regr… (#22110) 2026-05-14 11:03:40 +02:00
ggml-vocab-refact.gguf tests : add test-tokenizer-0.sh + fix some tokenizers (#7036) 2024-05-04 08:32:32 +03:00
ggml-vocab-refact.gguf.inp convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-refact.gguf.out convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-starcoder.gguf llama : fix BPE pre-tokenization (#6920) 2024-04-29 16:58:41 +03:00
ggml-vocab-starcoder.gguf.inp convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00
ggml-vocab-starcoder.gguf.out convert : allow partial update to the chkhsh pre-tokenizer list (#13847) 2025-05-30 12:24:37 +02:00