vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-07-09 17:08:33 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
14123 commits 3 branches 248 tags 814 MiB
Commit graph

14123 commits

This branch
This branch
All branches
Author SHA1 Message Date
Concedo
3090ae0bf7 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/s390x.Dockerfile
#	.dockerignore
#	.github/workflows/docker.yml
#	.github/workflows/release.yml
#	docs/android.md
#	ggml/src/ggml-cpu/amx/mmq.cpp
#	ggml/src/ggml-hexagon/htp/ssm-conv.c
#	tests/peg-parser/test-gbnf-generation.cpp
#	tests/test-arg-parser.cpp
#	tests/test-chat.cpp
#	tests/test-jinja.cpp
#	tests/test-json-schema-to-grammar.cpp
#	tools/server/README.md
 2026-06-22 18:23:59 +08:00
Pascal
d0f9d2e5ac
server: fix edit_file crash on append at end of file (line_start -1) (#24893)
Some checks failed
Python Type-Check / python type-check (push) Has been cancelled
Details
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Details 
line_start -1 normalized to n+1, so append inserted at lines.begin() + n + 1,
one past end() -> heap-buffer-overflow in vector::_M_range_insert.

Normalize -1 to n (insert at end()), restrict -1 to append mode and reject it
for replace/delete instead of silently clobbering the last line. Parenthesize
the insert offset so empty-file append computes the position as int first,
avoiding a transient begin() - 1 on a null vector data pointer.
 2026-06-22 10:55:28 +02:00
aafsmarak
0ef6f06d55
docs/android.md: Add dependency libandroid-spawn for building in termux (#21812) 
Fixes https://github.com/ggml-org/llama.cpp/issues/18615
 2026-06-22 05:48:31 +02:00
Aldehir Rojas
52b3df0023
common/peg : implement ac parser for stricter grammar generation (#24869) 
* common/peg : implement ac parser

* cont : extract functions

* cont : tidy up

* cont : remove a test

* cont : move ac() def
 2026-06-21 16:20:58 -05:00
Concedo
dfa1c573c4 fix router mode on lcpp ui 2026-06-22 00:21:19 +08:00
Xuan-Son Nguyen
7c082bc417
server: fix report progress for loading spec models, add "stages" list (#24870) 
* server: fix report progress for loading spec models, add "stages" list

* improve

* nits

* nits 2
 2026-06-21 17:36:52 +02:00
henk717
a072dd8304
llama-ui update (#2281) 2026-06-21 23:02:16 +08:00
Concedo
08fbef5049 lcpp ui think budget 2026-06-21 22:56:10 +08:00
Concedo
44bcead521 mcp fixes 2026-06-21 22:28:47 +08:00
Concedo
1afe5a730a minor fixes to handler newer lcpp ui 2026-06-21 22:21:10 +08:00
Xuan-Son Nguyen
bddfd2b113
server: refactor batch construction (#24843) 
* server: refactor batch construction

* wip

* wip 2

* wip 3

* wip 4

* add abort_all_slots

* handle batch full more carefully

* fix assert

* rm debug log

* small nits

* (debug) add timings

* debug: force llama_synchronize for accurate timings

* address comments

* disable DEBUG_TIMINGS
 2026-06-21 14:16:11 +02:00
Xuan-Son Nguyen
0d135df48c
mtmd: fix mtmd_get_memory_usage (#24867) 2026-06-21 14:12:15 +02:00
Sigbjørn Skjæret
bf533823cd
jinja : implement call statement (#24847) 
* implement call statement

* undo unintended change

* de-lambda

* simplify

* move caller context inside function handler
 2026-06-21 14:04:52 +02:00
Xuan-Son Nguyen
2f89acc2bc
mtmd: add load progress callback (#24865) 2026-06-21 13:40:52 +02:00
Xuan-Son Nguyen
bfa3219177
server: add "verbose" field to schema (#24864) 2026-06-21 13:03:14 +02:00
Xuan-Son Nguyen
d6d899580d
server: real-time model load progress tracking via /models/sse (#24828) 
* server: real-time model load progress tracking via /models/sse

* update docs

* add mutex for notify_to_router

* correct docs
 2026-06-21 11:58:14 +02:00
Georgi Gerganov
8a118ee86c
minor : clean-up whitespaces (#24862) 
[no ci]
 2026-06-21 11:37:12 +03:00
YiChen Lv
d789527482
spec : Support Step3.5/3.7 flash mtp3 (#24340) 
* add mtp_layer_offset + include nextn flags in graph reuse

* add llama_set_mtp_layer_offset + llama_model_n_nextn_layer API

* offset head select + require all MTP blocks

* speculative multi-head process()

* speculative multi-head draft()

* gather outputs via inp_out_ids

* cleanup

* fix core

* minor cleanup

* merged draft_multi_head into draft()

* mtp rename nextn

* Apply suggestions from code review

Co-authored-by: Aman Gupta <amangupta052@gmail.com>

* clean-up comments

* fix for multi seq

* apply suggestions && chain-heads comment

* add a reference for chain_heads discussion

---------

Co-authored-by: Aman Gupta <amangupta052@gmail.com>
 2026-06-21 11:33:18 +03:00
Concedo
f202c0a457 mtp init -2 2026-06-21 10:21:48 +08:00
Aldehir Rojas
063d9c156e
common/peg : refactor until gbnf grammar generation (#24839) 
* common/peg : refactor until gbnf grammar into an ac automaton

* cont : add a test with multiple strings

* cont : pad state with 0s so rules line up

* cont : clean up comments

* cont : use set everywhere

* cont : inline state num string padding

* cont : add a ref to PR

* cont : fix regression in server-tools.cpp
 2026-06-20 21:15:06 -05:00
Aldehir Rojas
c57607016a
common/json-schema-to-grammar : align spacing rules with parsers (#24835) 2026-06-20 17:43:04 -05:00
Guanhuai Zhang
4a80943174
fix(hexagon): use padded stride for ssm-conv weights (#24470) 2026-06-20 14:58:49 -07:00
Adrien Gallouët
84de01a1f1
llama : use LLM_KV for quantization_version & file_type (#24802) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-20 20:07:01 +02:00
Xuan-Son Nguyen
75f460ac28
arg: try fixing test-args-parser randomly fails (#24826) 
* arg: try fixing test-args-parser randomly fails

* return ref

* try triggering the workflow

* exception wrapper

* wip

* test

* test 2

* arg: guard win32 utf8 argv override

make_utf8_argv rebuilds argv from GetCommandLineW to fix utf8 handling of
non ascii arguments on windows. the override runs unconditionally inside
common_params_parse, so it also clobbers a programmatic argv passed by a
caller. test-arg-parser builds a synthetic argv but then sees the real
process command line instead, the model argument is never parsed, and the
assert that expects success aborts via fastfail (0xC0000409). this shows up
as a random failure in the openvino windows workflow.

only override argv when its length matches the caller argc, so the utf8
repair still applies to real binaries while a programmatic argv stays intact.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
 2026-06-20 19:45:27 +02:00
Muhammad Salem
8452824611
release: add missing link for win opencl adreno arm64 (#24809) 2026-06-20 23:08:59 +08:00
Concedo
a5019767c3 docs 2026-06-20 23:02:56 +08:00
Matti4
e27f308597
server: avoid forwarding auth headers in CORS proxy (#24373) 
* server: avoid forwarding auth headers in CORS proxy

* format

* fix test

* fix e2e test

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
 2026-06-20 15:34:47 +02:00
Aldehir Rojas
67e9fd3b74
docker : prebuild web UI for s390x build [no release] (#24829) 2026-06-20 05:54:42 -05:00
davidrhodus
796f41bedc
model : glm-dsa load DSA indexer tensors as optional (#24770) 
GLM-5.2 ships the DSA "lightning indexer" on only a subset of layers (the
"full" layers; others omit it), but the GLM_DSA loader created the five
indexer tensors on every layer as required, so loading any GLM-5.2 GGUF
failed with e.g. `missing tensor 'blk.3.indexer.k_norm.weight'`.

GLM_DSA's graph is llama_model_deepseek2::graph (plain MLA) and does not use
the indexer tensors (indexer runtime not yet implemented), so they are
loaded-but-unused. Marking them TENSOR_NOT_REQUIRED lets layers without an
indexer load as nullptr and the model runs as full MLA attention.

DeepSeek-V3.2 (uniform indexer on all layers) is unaffected.
 2026-06-20 13:48:24 +03:00
Adrien Gallouët
37a77fb057
ggml : optimize AMX (#24806) 
Flatten the partition over n_batch * M so every thread participates in
the quantization

    | CPU                             | Model                         | Test   |   t/s OLD |   t/s NEW |   Speedup |
    |:--------------------------------|:------------------------------|:-------|----------:|----------:|----------:|
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B IQ4_NL - 4.5 bpw  | pp512  |    730.71 |    779.86 |      1.07 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B IQ4_NL - 4.5 bpw  | tg128  |     87.88 |     86.79 |      0.99 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B IQ4_XS - 4.25 bpw | pp512  |    725.09 |   1023.31 |      1.41 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B IQ4_XS - 4.25 bpw | tg128  |     83.64 |     83.62 |      1.00 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_0              | pp512  |    820.51 |    924.05 |      1.13 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_0              | tg128  |     90.59 |     92.46 |      1.02 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_1              | pp512  |    776.88 |    872.79 |      1.12 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_1              | tg128  |     89.39 |     90.94 |      1.02 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_K_M            | pp512  |    719.28 |   1009.27 |      1.40 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_K_M            | tg128  |     80.62 |     80.86 |      1.00 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_K_S            | pp512  |    732.29 |   1077.29 |      1.47 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_K_S            | tg128  |     86.42 |     83.53 |      0.97 |

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-20 13:43:06 +03:00
Sigbjørn Skjæret
f4043fec01
convert : more consistent handling of rope_parameters (#24833) 2026-06-20 13:42:36 +03:00
Concedo
73cc7d9287 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/cann.Dockerfile
#	.devops/cpu.Dockerfile
#	.devops/cuda.Dockerfile
#	.devops/intel.Dockerfile
#	.devops/musa.Dockerfile
#	.devops/openvino.Dockerfile
#	.devops/rocm.Dockerfile
#	.devops/s390x.Dockerfile
#	.devops/vulkan.Dockerfile
#	.devops/zendnn.Dockerfile
#	.dockerignore
#	.pi/gg/SYSTEM.md
#	ggml/CMakeLists.txt
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	scripts/sync-ggml.last
#	scripts/sync_vendor.py
#	tools/cli/README.md
#	tools/cli/cli.cpp
#	tools/mtmd/clip.cpp
#	tools/server/README.md
 2026-06-20 17:16:19 +08:00
Concedo
84b8856295 Merge commit '32eddaf2ea' into concedo_experimental 
# Conflicts:
#	docs/multimodal.md
#	docs/preset.md
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/hex-dma.h
#	ggml/src/ggml-hexagon/htp/hex-utils.h
#	ggml/src/ggml-hexagon/htp/hmx-flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/hmx-matmul-ops.c
#	ggml/src/ggml-hexagon/htp/hmx-queue.c
#	ggml/src/ggml-hexagon/htp/hmx-queue.h
#	ggml/src/ggml-hexagon/htp/htp-ctx.h
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.c
#	scripts/snapdragon/ggml-hexagon-profile.py
#	scripts/ui-assets.cmake
#	tools/export-lora/README.md
#	tools/server/CMakeLists.txt
 2026-06-20 11:18:24 +08:00
Masashi Yoshimura
f449e05537
ggml-webgpu: add adapter toggles for F16 on Vulkan + NVIDIA
Some checks failed
Python Type-Check / python type-check (push) Has been cancelled
Details
2026-06-20 08:12:32 +09:00
Xuan-Son Nguyen
2b686a9120
server: refactor child --> router communication (#24821) 
* server: refactor child --> router communication

* fix wakeup case

* add docs

* improve update_status()

* nits
 2026-06-20 01:02:26 +02:00
Adrien Gallouët
4b48a53b6c
server : optimize get_token_probabilities (#24796) 
Use std::partial_sort to order only the requested top-n tokens instead
of the full vocabulary

    logprobs sort: vocab=128000 n_top=0 iters=100
    full    sort:   8555.6 us/op
    partial sort:    704.3 us/op

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-19 23:26:54 +02:00
Xuan-Son Nguyen
e475fa2b5f
mtmd, arg: fix utf8 handling on windows (#24779) 
* mtmd, arg: fix utf8 handling on windows

* also fix ggml_fopen

* fix build fail

* also fix CLI
 2026-06-19 22:28:38 +02:00
Xuan-Son Nguyen
175147e8f6
server: remove all internal mentions about "webui" (#24817) 2026-06-19 22:12:46 +02:00
Concedo
2fb3406be7 added ideogram 4 support 2026-06-20 00:29:48 +08:00
Mikolaj Kucharski
fabde3bf51
arg: Add comment line support to --api-key-file (#23168) 2026-06-19 17:33:54 +02:00
Alessandro de Oliveira Faria (A.K.A.CABELO)
0d2d9ccbf6
vendor : update cpp-httplib to 0.48.0 (#24787) 2026-06-19 22:16:35 +08:00
Xuan-Son Nguyen
8c2d6f6475
server: add --agent arg, remove redundant webui naming compat (#24801) 
* server: add --agent arg, remove redundant webui naming compat

* corrent env

* fix the test

* llama-gen-docs

* nits: wordings
 2026-06-19 16:06:13 +02:00
Concedo
9b1e2fa8b8 advanced onready bypass 2026-06-19 21:44:37 +08:00
Aldehir Rojas
38724ab593
docker : build the UI (#24794) 
* docker : build the UI

* cont : use existing APP_VERSION
 2026-06-19 15:32:31 +02:00
Xuan-Son Nguyen
e2e7a9b2d0
mtmd: several bug fixes (#24784) 
* mtmd: several bug fixes

* fix build

* fix gemma4ua

* add sanity check in get_u32()

* fix build (2)

* area() avoid overflow
 2026-06-19 12:18:36 +02:00
Ruixiang Wang
b14e3fb90c
spec: support eagle3 for qwen3.5 & 3.6 (#24593) 
* spec: support qwen3.5 & 3.6 eagle3 draft

* eagle3: Add deferred boundary checkpoints restore support for hybrid models

* apply suggestions

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* spec: adapt to API change

* spec: fix naming

* cont : add TODO

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-06-19 13:08:50 +03:00
Xuan-Son Nguyen
159d093a43
server: fix non-bound n_discard value (ctx shifting) (#24786) 
* server: fix non-bound n_discard value

* Update tools/server/server-context.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-06-19 10:53:44 +02:00
Georgi Gerganov
5fd2dc2c41 sync : ggml 2026-06-19 10:19:14 +03:00
Georgi Gerganov
1868af13ac ggml : bump version to 0.15.2 (ggml/1548) 2026-06-19 10:19:14 +03:00
Concedo
df08e951d0 response format returns actual mp3 filename if requested 2026-06-19 15:18:53 +08:00