vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2025-09-14 02:49:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

vocab : JetBrains Mellum pre-tokenizer (#15045)

Browse source
This commit is contained in:
Csaba Kecskemeti 2025-08-03 12:38:18 -07:00 • committed by GitHub
parent 83bc2f288c
commit 97366dc6ab
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 6 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

3
src/llama-vocab.cpp
View file

@ -1856,7 +1856,8 @@ void llama_vocab::impl::load(llama_model_loader & ml, const LLM_KV & kv) {
tokenizer_pre == "gigachat" ||
tokenizer_pre == "jina-v2-es" ||
tokenizer_pre == "jina-v2-de" ||
tokenizer_pre == "a.x-4.0") {
tokenizer_pre == "a.x-4.0" ||
tokenizer_pre == "mellum") {
pre_type = LLAMA_VOCAB_PRE_TYPE_GPT2;
} else if (
tokenizer_pre == "jina-v1-en" ||