vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-08-04 13:39:56 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
14598 commits 3 branches 251 tags 847 MiB
Commit graph

525 commits

Author SHA1 Message Date
Concedo
49dbdaaab5 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	AGENTS.md
#	CODEOWNERS
#	CONTRIBUTING.md
#	docs/backend/OPENCL.md
#	docs/development/HOWTO-add-model.md
#	examples/training/finetune.cpp
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp-drv.cpp
#	ggml/src/ggml-hexagon/htp/act-ops.c
#	ggml/src/ggml-hexagon/htp/dma-queue.c
#	ggml/src/ggml-hexagon/htp/dma-queue.h
#	ggml/src/ggml-hexagon/htp/flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/flash-attn-ops.h
#	ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h
#	ggml/src/ggml-hexagon/htp/htp-ctx.h
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/htp-tensor.c
#	ggml/src/ggml-hexagon/htp/htp-tensor.h
#	ggml/src/ggml-hexagon/htp/hvx-fa-kernels.h
#	ggml/src/ggml-hexagon/htp/hvx-reduce.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.h
#	ggml/src/ggml-hexagon/htp/unary-ops.c
#	ggml/src/ggml-hexagon/htp/unary-ops.h
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	scripts/compare-llama-bench.py
#	scripts/snapdragon/ggml-hexagon-profile.py
#	scripts/snapdragon/ggml-hexagon-trace.py
#	scripts/sync_vendor.py
#	tests/test-arg-parser.cpp
#	tests/test-chat.cpp
#	tests/test-model-load-cancel.cpp
#	tests/test-quantize-stats.cpp
#	tools/cli/README.md
#	tools/completion/README.md
#	tools/llama-bench/llama-bench.cpp
#	tools/server/README.md
#	tools/ui/src/lib/constants/settings-registry.ts
 2026-07-25 12:20:51 +08:00
Concedo
eea06b0a89 Merge commit 'f534da26e4' into concedo_experimental 
# Conflicts:
#	.github/workflows/release.yml
#	docs/ops.md
#	docs/ops/WebGPU.csv
#	ggml/src/CMakeLists.txt
#	ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-openvino/ggml-openvino.cpp
#	ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	models/templates/deepseek-ai-DeepSeek-V4.jinja
#	tests/test-backend-ops.cpp
#	tests/test-chat-auto-parser.cpp
#	tests/test-chat.cpp
#	tests/test-llama-archs.cpp
 2026-07-25 12:00:23 +08:00
Aaron Teo
e6dd0e29a6
args: refactor mlock/mmap/directio into load-mode (#20834) 
* args: overhaul mmap/mlock/dio into single arg

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: update docs with llama-gen-docs

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* chore: satisfy code quality

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* args: make the `+` sign an actual modifier now

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* chore: general code clean up + comments

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* arg: fix deprecated flags support

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* arg: quick sanity check

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* bench: sync llama-bench argument parsing

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* fix: bugfix variable behaviour + llama-bench lm column size

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* arg: inverse commands should do the opposite instead of doing nothing

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* bench: fix incorrect dash

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* bench: fix missing modifiers for deprecated flags

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* llama: switch back to thread_local

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* arg: switch back to single enum

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: update arg docs

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* chore: fix missing `mlock` from llama_load_mode_from_str + cleanup llama-bench

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* llama: fix mlock not activating

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* arg: add deprecation warning when old and new flags are combined

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* arg: cont add comment for todo in the future

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: sync with upstream

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* docs: re-sync with upstream again

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2026-07-23 20:32:56 +08:00
Pascal
6d5a910c50
common: infer the speculative type from the draft repo sidecars (#25989) 
With -hfd pointing to a repo that ships mtp-/dflash-/eagle3- sidecars
and no --spec-type given, the draft resolved to a full model while the
sidecar was the intended draft.

When the speculative types are still at their default, discover the
sidecars of the draft repo, pick the first available following the
existing mtp > dflash > eagle3 priority, and set the corresponding
type, so this now works without any extra flag:

llama-server -hf repo:Q3_K_M -hfd repo:Q8_0

An explicit --spec-type disables the inference, and a draft repo
without sidecars keeps resolving to a full model as before.
 2026-07-22 13:06:35 +02:00
Pascal
60f6a17704
common: resolve draft repo to its requested sidecar (#25955) 
With -hfd pointing to a repo shipping speculative sidecars, the draft
resolved to the main model of that repo, since find_best_model()
excludes sidecar files, and the explicit draft plan suppressed the
sidecar discovery on the -hf repo.

The draft plan already discovers its sidecars, they were just never
consumed. Wire them as the draft, following the fallback pattern of
the main plan, so this now works as expected:

llama-server -hf repo -hfd repo --spec-type draft-dflash
 2026-07-21 18:03:43 +02:00
Concedo
0a0b88a5c0 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	docs/backend/OPENCL.md
#	ggml/CMakeLists.txt
#	ggml/src/ggml-cpu/CMakeLists.txt
#	ggml/src/ggml-cpu/kleidiai/kernels.cpp
#	ggml/src/ggml-cpu/kleidiai/kernels.h
#	ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp-drv.cpp
#	ggml/src/ggml-hexagon/htp-drv.h
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/act-ops.c
#	ggml/src/ggml-hexagon/htp/argsort-ops.c
#	ggml/src/ggml-hexagon/htp/binary-ops.c
#	ggml/src/ggml-hexagon/htp/cumsum-ops.c
#	ggml/src/ggml-hexagon/htp/flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/hex-dma.h
#	ggml/src/ggml-hexagon/htp/hex-profile.h
#	ggml/src/ggml-hexagon/htp/hex-utils.h
#	ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h
#	ggml/src/ggml-hexagon/htp/hmx-queue.c
#	ggml/src/ggml-hexagon/htp/hmx-queue.h
#	ggml/src/ggml-hexagon/htp/htp-ctx.h
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.h
#	ggml/src/ggml-hexagon/htp/rope-ops.c
#	ggml/src/ggml-hexagon/htp/unary-ops.c
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/gemm_moe_mxfp4_q8_1_dp4a.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_q4_0_q8_1_dp4a.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_q4_k_q8_1_dp4a.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_q6_k_q8_1_dp4a.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_q8_1_dp4a.cl
#	ggml/src/ggml-opencl/kernels/gemm_noshuffle_q4_k_f32.cl
#	ggml/src/ggml-opencl/kernels/gemm_noshuffle_q4_k_q8_1_dp4a.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_k_f32.cl
#	ggml/src/ggml-opencl/kernels/mul_mv_q4_k_f32_flat.cl
#	ggml/src/ggml-opencl/kernels/mul_mv_q5_k_f32_flat.cl
#	ggml/src/ggml-sycl/dmmv.cpp
#	scripts/snapdragon/ggml-hexagon-profile.py
#	scripts/sync-ggml.last
#	tests/CMakeLists.txt
#	tests/test-backend-ops.cpp
#	tests/test-llama-archs.cpp
#	tests/test-recurrent-state-rollback.cpp
#	tools/cli/README.md
#	tools/completion/README.md
#	tools/server/README.md
 2026-07-18 12:26:15 +08:00
Concedo
4c25a3d829 Merge commit '79bba02a67' into concedo_experimental 
# Conflicts:
#	.github/workflows/build-self-hosted.yml
#	.github/workflows/release.yml
#	.github/workflows/server-self-hosted.yml
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/flash_attn_f32_f16.cl
#	ggml/src/ggml-opencl/kernels/flash_attn_f32_q4_0.cl
#	ggml/src/ggml-opencl/kernels/flash_attn_f32_q8_0.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_mxfp4_f32_ns.cl
#	ggml/src/ggml-opencl/kernels/mul_mv_f16_f32_l4.cl
#	ggml/src/ggml-opencl/kernels/mul_mv_q4_k_f32.cl
#	tests/test-backend-ops.cpp
#	tools/tokenize/tokenize.cpp
#	tools/ui/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionAdd/ChatFormActionAddToolsSubmenu.svelte
 2026-07-18 11:04:46 +08:00
Georgi Gerganov
635cdd5fcc
common : auto-download dflash- and eagle3- HF sidecars (#25811) 
* common: auto-download dflash- and eagle3- HF sidecars

Mirror the existing mtp- sidecar logic to support auto-discovery and
download of DFlash (dflash-) and Eagle3 (eagle3-) speculative decoding
sidecars from Hugging Face repos.

Changes:
- Add --dflash and --eagle3 CLI flags to trigger sidecar download
- Add find_best_dflash() and find_best_eagle3() using find_best_sibling
- Exclude dflash- and eagle3- filenames from primary model selection
- Filter dflash- and eagle3- from cached model listings
- Wire download tasks that set speculative.draft.mparams as fallback

Assisted-by: pi:llama.cpp/Qwen3.6-27B

* docs : regen
 2026-07-17 12:15:30 +03:00
Adrien Gallouët
aff6eb6e75
tokenize : drop --stdin mutual-exclusion check (#25672) 
match cli and completion, which don't enforce it
 2026-07-15 18:41:51 +02:00
Concedo
9001369da0 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	ggml/src/ggml-cpu/CMakeLists.txt
#	ggml/src/ggml-cpu/kleidiai/kernels.cpp
#	ggml/src/ggml-cpu/kleidiai/kernels.h
#	ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
#	ggml/src/ggml-cpu/ops.cpp
#	ggml/src/ggml-cuda/mmq.cuh
#	ggml/src/ggml-hexagon/htp/hmx-queue.c
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_iq4_nl_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q1_0_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_0_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_0_f32_spec.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_1_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_k_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q5_0_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q5_1_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q5_k_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q6_k_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q8_0_f32.cl
#	ggml/src/ggml-opencl/kernels/mul_mv_f16_f16.cl
#	ggml/src/ggml-opencl/kernels/mul_mv_f16_f32.cl
#	ggml/src/ggml-opencl/kernels/mul_mv_f16_f32_1row.cl
#	ggml/src/ggml-sycl/fattn-vec.hpp
#	tests/test-backend-ops.cpp
#	tests/test-chat-auto-parser.cpp
#	tests/test-export-graph-ops.cpp
#	tests/test-jinja.cpp
#	tests/test-llama-archs.cpp
#	tools/tokenize/tokenize.cpp
 2026-07-15 15:39:31 +08:00
Xuan-Son Nguyen
6e52db5b72
server: add --cors-* options (#25655) 
* server: add --cors-* options

* add special "localhost" value

* add tests

* fix test

* add link to PR
 2026-07-14 17:23:44 +02:00
Adrien Gallouët
8ff8c4299d
tokenize : align usage by using common args (#25516) 
Migrate the tokenize tool to common_params_parse, replacing its
hand-rolled argv parsing, Windows UTF-8 handling and file reading
with the shared common helpers.

Expose the model-sourcing flags (-m, -mu, -dr, -hf, -hff, --offline,
HF_TOKEN) to LLAMA_EXAMPLE_TOKENIZE, and register --ids, --stdin,
--no-bos, --no-parse-special and --show-count as common args.
parse_special defaults to true for TOKENIZE to preserve the old
behavior. Errors now go through LOG_ERR instead of fprintf(stderr).

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-07-14 15:20:53 +02:00
Christian Kastner
ec0dbef816
arg: Flush log before exiting after usage() (#25504) 
Under certain conditions, it's possible for messages emitted via LOG()
to get lost before exit, apparently because they are emitted by another
thread. common_params_print_usage() uses printf directly, and is not
affected.

Flushing the log before exit seems to resolve this.
 2026-07-14 12:03:22 +03:00
Concedo
f57cd915a9 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.github/workflows/ui-publish.yml
#	CODEOWNERS
#	docs/ops.md
#	ggml/CMakeLists.txt
#	ggml/src/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/argsort-ops.c
#	scripts/sync-ggml.last
#	tools/mtmd/tests/test-deepseek-ocr.py
 2026-07-11 11:36:59 +08:00
Concedo
9d8a50378a Merge commit '961e4b26a7' into concedo_experimental 
# Conflicts:
#	.github/workflows/hip-quality-check.yml
#	AGENTS.md
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/concat-ops.c
#	ggml/src/ggml-hexagon/htp/flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/flash-attn-ops.h
#	ggml/src/ggml-hexagon/htp/hmx-fa-kernels.h
#	ggml/src/ggml-hexagon/htp/hmx-queue.c
#	ggml/src/ggml-hexagon/htp/hmx-queue.h
#	ggml/src/ggml-hexagon/htp/hmx-utils.h
#	ggml/src/ggml-hexagon/htp/htp-ctx.h
#	ggml/src/ggml-hexagon/htp/hvx-utils.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.c
#	ggml/src/ggml-hexagon/htp/rope-ops.c
#	ggml/src/ggml-hexagon/htp/unary-ops.c
#	ggml/src/ggml-hexagon/htp/worker-pool.c
#	ggml/src/ggml-hexagon/htp/worker-pool.h
#	ggml/src/ggml-hip/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/flash_attn_f32_f16.cl
#	ggml/src/ggml-opencl/kernels/flash_attn_f32_q4_0.cl
#	ggml/src/ggml-opencl/kernels/flash_attn_f32_q8_0.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_mxfp4_f32_ns.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_q4_0_f32_ns.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_q4_1_f32_ns.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_q4_k_f32_ns.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_q5_0_f32_ns.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_q5_1_f32_ns.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_q5_k_f32_ns.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_q6_k_f32_ns.cl
#	ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
#	ggml/src/ggml-webgpu/wgsl-shaders/flash_attn_vec_split.wgsl
#	tests/CMakeLists.txt
#	tests/test-backend-ops.cpp
#	tools/cli/CMakeLists.txt
#	tools/cli/cli.cpp
#	tools/llama-bench/llama-bench.cpp
 2026-07-11 11:13:27 +08:00
Concedo
0f0245161e Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	tests/test-backend-ops.cpp
#	tests/test-quantize-fns.cpp
#	tools/server/README.md
#	tools/ui/src/lib/constants/settings-registry.ts
 2026-07-11 09:20:40 +08:00
Xuan-Son Nguyen
22b69b6e92
arg: prevent duplicate spec model downloads (#25527) 2026-07-10 16:53:26 +02:00
Xuan-Son Nguyen
c4ae9a88f8
server: improve tools, remove apply_diff (#25498) 
* server: improve tools, remove apply_diff

* improve edit tool

* add tools_io abstraction

* add tools_io_basic

* fix build

* move utils to class member

* add const
 2026-07-10 11:52:59 +02:00
Xuan-Son Nguyen
3de7dd4c8f
cli: add --output option (#25484) 2026-07-09 19:37:39 +02:00
Xuan-Son Nguyen
c264f65ff9
cli : move to HTTP-based implementation (#24948) 
* cli: move to HTTP-based implementation

* wip

* working

* remote server ok

* cli support router mode

Co-authored-by: Piotr Wilkin <ilintar@gmail.com>

* case: router with only one model

* Apply suggestions from code review

Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

* remove outdated comment

* use destructor instead

* add ftype

* cli-view --> cli-ui

* pimpl

* no more json in header

* nits fixes

* also show model aliases

---------

Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
 2026-07-08 14:52:43 +02:00
rankaiyx
f296fdfbed
common: auto-create prompts-log-dir at argument parsing, so all tools using the flag benefit (#25322) 2026-07-08 09:45:28 +02:00
Concedo
56d11ad4e8 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	docs/backend/OPENCL.md
#	ggml/src/ggml-hexagon/CMakeLists.txt
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/hex-dma.h
#	ggml/src/ggml-hexagon/htp/hmx-utils.h
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/hvx-base.h
#	ggml/src/ggml-hexagon/htp/hvx-exp.h
#	ggml/src/ggml-hexagon/htp/hvx-sigmoid.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.c
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/cvt.cl
#	scripts/snapdragon/adb/run-completion.sh
#	scripts/snapdragon/adb/run-tool.sh
#	scripts/snapdragon/ggml-hexagon-profile.py
#	tests/test-backend-ops.cpp
 2026-07-02 21:42:36 +08:00
Adrien Gallouët
a6647b1a32
common : use hf primary split as model path (#25194) 
Fixes #25181
 2026-07-01 18:33:00 +02:00
Concedo
3b867bd4b1 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.github/workflows/release.yml
#	SECURITY.md
#	common/CMakeLists.txt
#	docs/speculative.md
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/cvt.cl
#	ggml/src/ggml-opencl/kernels/flash_attn_f16.cl
#	ggml/src/ggml-opencl/kernels/flash_attn_f32.cl
#	ggml/src/ggml-opencl/kernels/flash_attn_f32_f16.cl
#	ggml/src/ggml-opencl/kernels/set_rows.cl
#	ggml/src/ggml-openvino/ggml-openvino.cpp
#	ggml/src/ggml-sycl/norm.cpp
#	tests/CMakeLists.txt
#	tests/test-backend-ops.cpp
#	tests/test-chat-template.cpp
#	tests/test-chat.cpp
#	tests/test-export-graph-ops.cpp
#	tests/test-jinja.cpp
#	tests/test-llama-archs.cpp
#	tools/rpc/CMakeLists.txt
#	tools/rpc/README.md
 2026-06-29 16:43:44 +08:00
Xuan-Son Nguyen
b3fed31b99
jinja, chat: add --reasoning-preserve flag (#25105) 
* jinja, chat: add --reasoning-preserve flag

* correct help message
 2026-06-28 23:33:51 +02:00
Adrien Gallouët
c1a1c8ee94
common : allow --offline in llama download (#25091) 
Expose the existing --offline flag to `llama download` so a script can
run it to check whether a model is already cached and ready to be served
without touching the network.

Also fix a latent use-after-free in the URL-task on_done callback:
first_path is block-scoped and was captured by reference, but invoked
after the block ends.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-28 12:34:11 +02:00
Concedo
e27861e14e Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/cann.Dockerfile
#	.devops/cpu.Dockerfile
#	.devops/cuda.Dockerfile
#	.devops/intel.Dockerfile
#	.devops/musa.Dockerfile
#	.devops/openvino.Dockerfile
#	.devops/rocm.Dockerfile
#	.devops/s390x.Dockerfile
#	.devops/vulkan.Dockerfile
#	.devops/zendnn.Dockerfile
#	.github/workflows/build-cache.yml
#	.github/workflows/build-openvino.yml
#	.github/workflows/build-self-hosted.yml
#	.github/workflows/release.yml
#	app/llama.cpp
#	build-xcframework.sh
#	docs/backend/OPENVINO.md
#	ggml/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-openvino/ggml-decoder.cpp
#	ggml/src/ggml-openvino/openvino/op/add_id.cpp
#	ggml/src/ggml-openvino/openvino/op/glu_swiglu.cpp
#	ggml/src/ggml-openvino/openvino/op/mul_mat_id.cpp
#	ggml/src/ggml-openvino/openvino/op/softmax.cpp
#	ggml/src/ggml-openvino/openvino/op_table.cpp
#	ggml/src/ggml-openvino/openvino/op_table.h
#	ggml/src/ggml-sycl/softmax.cpp
#	scripts/sync-ggml.last
#	tests/test-backend-ops.cpp
#	tests/test-quantize-fns.cpp
#	tools/server/CMakeLists.txt
#	tools/ui/src/lib/services/chat.service.ts
 2026-06-27 10:33:29 +08:00
Concedo
4e43c21e58 Merge commit '9d5d882d8c' into concedo_experimental 
# Conflicts:
#	.github/labeler.yml
#	app/CMakeLists.txt
#	app/llama.cpp
#	build-xcframework.sh
#	common/CMakeLists.txt
#	common/download.h
#	docs/backend/SYCL.md
#	docs/backend/snapdragon/CMakeUserPresets.json
#	docs/speculative.md
#	ggml/CMakeLists.txt
#	ggml/include/ggml-sycl.h
#	ggml/src/ggml-hexagon/CMakeLists.txt
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/cmake-toolchain.cmake
#	ggml/src/ggml-hexagon/htp/flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/hex-dma.h
#	ggml/src/ggml-hexagon/htp/hex-utils.h
#	ggml/src/ggml-hexagon/htp/hmx-flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/htp-ctx.h
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/htp_iface.idl
#	ggml/src/ggml-hexagon/htp/hvx-base.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.c
#	ggml/src/ggml-hexagon/libggml-htp.inf
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/norm.cl
#	ggml/src/ggml-sycl/conv3d.cpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	scripts/snapdragon/adb/run-completion.sh
#	scripts/snapdragon/adb/run-tool.sh
#	scripts/snapdragon/ggml-hexagon-profile.py
#	tests/CMakeLists.txt
#	tests/test-backend-ops.cpp
#	tests/test-thread-safety.cpp
#	tools/llama-bench/llama-bench.cpp
#	tools/mtmd/CMakeLists.txt
#	tools/mtmd/tests/test-deepseek-ocr.py
 2026-06-27 10:18:52 +08:00
Xuan-Son Nguyen
024930c6ad
arg: fix handling --spec-draft-hf and --hf-repo-v (#25043) 
* arg: fix handling --spec-draft-hf and --hf-repo-v

* fix missing mparams.hf_file
 2026-06-26 14:36:03 +02:00
Xuan-Son Nguyen
60bc8866b1
common: refactor model handling (#24980) 
* common: refactor models handling

* remote preset

* cont

* rm skip_download option

* missing header

* fix plan.model_files

* fix --offline case

* move hf_plan to download

* refactor

* rm redundant curr_ex, add comments

* adapt
 2026-06-25 15:17:51 +02:00
Adrien Gallouët
683b04cc4a
app : add the llama download subcommand (#24982) 
* app : add the download command (with llama-download)

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Remove llama-download tool for now

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-25 13:36:36 +02:00
Concedo
579229d157 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	CODEOWNERS
#	README.md
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q8_0_f32.cl
#	ggml/src/ggml-sycl/binbcast.cpp
#	ggml/src/ggml-sycl/element_wise.cpp
#	ggml/src/ggml-vulkan/CMakeLists.txt
#	ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_id_vec.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec_acc.tmpl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec_q_acc.tmpl
#	ggml/src/ggml-webgpu/wgsl-shaders/quantize_q8.wgsl
#	tests/test-backend-ops.cpp
#	tests/test-chat.cpp
#	tests/test-sampling.cpp
#	tools/server/README.md
 2026-06-24 23:28:21 +08:00
Xuan-Son Nguyen
75ad0b23ed
server: fix remote preset handling, add test (#24938) 
* server: add test for remote preset

* fix remote preset handling

* fix

* fix test
 2026-06-23 13:28:34 +02:00
Xuan-Son Nguyen
721354fbdf
server: (router) move model downloading to dedicated process (#24834) 
* server: real-time model load progress tracking via /models/sse

* update docs

* server: move model download to child process

* rm unused

* fix most problems

* clean up

* nit fixes

* fix test case

* do not detact() thread

* shorter MODEL_DOWNLOAD_TIMEOUT in test

* throttle
 2026-06-22 18:24:04 +02:00
Concedo
3090ae0bf7 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/s390x.Dockerfile
#	.dockerignore
#	.github/workflows/docker.yml
#	.github/workflows/release.yml
#	docs/android.md
#	ggml/src/ggml-cpu/amx/mmq.cpp
#	ggml/src/ggml-hexagon/htp/ssm-conv.c
#	tests/peg-parser/test-gbnf-generation.cpp
#	tests/test-arg-parser.cpp
#	tests/test-chat.cpp
#	tests/test-jinja.cpp
#	tests/test-json-schema-to-grammar.cpp
#	tools/server/README.md
 2026-06-22 18:23:59 +08:00
Xuan-Son Nguyen
75f460ac28
arg: try fixing test-args-parser randomly fails (#24826) 
* arg: try fixing test-args-parser randomly fails

* return ref

* try triggering the workflow

* exception wrapper

* wip

* test

* test 2

* arg: guard win32 utf8 argv override

make_utf8_argv rebuilds argv from GetCommandLineW to fix utf8 handling of
non ascii arguments on windows. the override runs unconditionally inside
common_params_parse, so it also clobbers a programmatic argv passed by a
caller. test-arg-parser builds a synthetic argv but then sees the real
process command line instead, the model argument is never parsed, and the
assert that expects success aborts via fastfail (0xC0000409). this shows up
as a random failure in the openvino windows workflow.

only override argv when its length matches the caller argc, so the utf8
repair still applies to real binaries while a programmatic argv stays intact.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
 2026-06-20 19:45:27 +02:00
Concedo
73cc7d9287 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/cann.Dockerfile
#	.devops/cpu.Dockerfile
#	.devops/cuda.Dockerfile
#	.devops/intel.Dockerfile
#	.devops/musa.Dockerfile
#	.devops/openvino.Dockerfile
#	.devops/rocm.Dockerfile
#	.devops/s390x.Dockerfile
#	.devops/vulkan.Dockerfile
#	.devops/zendnn.Dockerfile
#	.dockerignore
#	.pi/gg/SYSTEM.md
#	ggml/CMakeLists.txt
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	scripts/sync-ggml.last
#	scripts/sync_vendor.py
#	tools/cli/README.md
#	tools/cli/cli.cpp
#	tools/mtmd/clip.cpp
#	tools/server/README.md
 2026-06-20 17:16:19 +08:00
Concedo
84b8856295 Merge commit '32eddaf2ea' into concedo_experimental 
# Conflicts:
#	docs/multimodal.md
#	docs/preset.md
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/hex-dma.h
#	ggml/src/ggml-hexagon/htp/hex-utils.h
#	ggml/src/ggml-hexagon/htp/hmx-flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/hmx-matmul-ops.c
#	ggml/src/ggml-hexagon/htp/hmx-queue.c
#	ggml/src/ggml-hexagon/htp/hmx-queue.h
#	ggml/src/ggml-hexagon/htp/htp-ctx.h
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.c
#	scripts/snapdragon/ggml-hexagon-profile.py
#	scripts/ui-assets.cmake
#	tools/export-lora/README.md
#	tools/server/CMakeLists.txt
 2026-06-20 11:18:24 +08:00
Xuan-Son Nguyen
2b686a9120
server: refactor child --> router communication (#24821) 
* server: refactor child --> router communication

* fix wakeup case

* add docs

* improve update_status()

* nits
 2026-06-20 01:02:26 +02:00
Xuan-Son Nguyen
e475fa2b5f
mtmd, arg: fix utf8 handling on windows (#24779) 
* mtmd, arg: fix utf8 handling on windows

* also fix ggml_fopen

* fix build fail

* also fix CLI
 2026-06-19 22:28:38 +02:00
Mikolaj Kucharski
fabde3bf51
arg: Add comment line support to --api-key-file (#23168) 2026-06-19 17:33:54 +02:00
Xuan-Son Nguyen
8c2d6f6475
server: add --agent arg, remove redundant webui naming compat (#24801) 
* server: add --agent arg, remove redundant webui naming compat

* corrent env

* fix the test

* llama-gen-docs

* nits: wordings
 2026-06-19 16:06:13 +02:00
Xuan-Son Nguyen
552258c535
server: (router) rework -hf preset repo (#24739) 
* server: temporary remove HF remote preset

* rework remove preset.ini support

* rm unused get_remote_preset_whitelist()

* print warning

* add docs

* rm stray file
 2026-06-18 12:45:23 +02:00
Concedo
ea0351c71a Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.github/workflows/release.yml
#	.github/workflows/ui-build-self-hosted.yml
#	.github/workflows/ui-build.yml
#	.github/workflows/ui-publish.yml
#	.github/workflows/ui-self-hosted.yml
#	.github/workflows/ui.yml
#	.gitignore
#	README.md
#	docs/ops.md
#	docs/ops/Vulkan.csv
#	ggml/CMakeLists.txt
#	scripts/sync-ggml.last
#	scripts/sync_vendor.py
#	scripts/ui-assets.cmake
#	tests/test-jinja.cpp
#	tests/test-llama-archs.cpp
 2026-06-14 11:26:16 +08:00
Xuan-Son Nguyen
e37abd6b5f
mtmd: add batching API (#24384) 
* mtmd: add batching API

* wip

* first working version (gemma4v)

* add arg

* nits

* wire up support_batch()

* fix 0.0 output embd

* fix audio

* nits

* refactor a bit

* nits

* fix non-batching case

* fix comment
 2026-06-13 00:10:29 +02:00
Concedo
afc8de2c6b Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/cpu.Dockerfile
#	.devops/cuda.Dockerfile
#	.devops/intel.Dockerfile
#	.devops/musa.Dockerfile
#	.devops/openvino.Dockerfile
#	.devops/rocm.Dockerfile
#	.devops/vulkan.Dockerfile
#	.devops/zendnn.Dockerfile
#	.github/workflows/build-webgpu.yml
#	.github/workflows/release.yml
#	ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	ggml/src/ggml-webgpu/wgsl-shaders/binary.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/concat.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_decls.tmpl
#	ggml/src/ggml-webgpu/wgsl-shaders/scale.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/unary.wgsl
#	tests/CMakeLists.txt
#	tests/test-backend-ops.cpp
#	tests/test-mtmd-c-api.c
#	tools/cli/cli.cpp
#	tools/mtmd/CMakeLists.txt
#	tools/server/README.md
 2026-06-10 17:21:05 +08:00
Concedo
dd2f966f4b Merge commit 'c2b1518fd4' into concedo_experimental 
# Conflicts:
#	.devops/intel.Dockerfile
#	ggml/CMakeLists.txt
#	scripts/sync-ggml.last
#	tests/test-backend-ops.cpp
#	tests/test-llama-archs.cpp
#	tools/cli/cli.cpp
 2026-06-10 00:56:44 +08:00
jacekpoplawski
1e912561dd
server: log prompts to directory (#22031) 
* server: log prompts to directory

Add `--log-prompts-dir` to write each prompt to a separate text file in
the specified directory.

* Apply suggestion from @ngxson

---------

Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
 2026-06-09 12:09:07 +02:00
fiesh
961e9a3e46
server : do not clear slots without unified KV cache (#24190) 
* Always export idle slots to RAM

Without this, a slot's VRAM cache may not be written to RAM.  If this
slot happens to be busy then later on, this triggers needless
preprocessing in another slot.

* cont : clean-up

---------

Co-authored-by: Christoph Weiss <weiss@wsoptics.de>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-06-09 10:45:16 +03:00
Xuan-Son Nguyen
8f83d6c271
mtmd : add video input support (#24269) 
* wip

* ok: lazy bitmap API

* remember to free lazy text

* wip

* add mtmd_helper_video

* support video input on server (base64 input)

* add MTMD_VIDEO config

* add timestamp

* update CLI

* cli: allow auto-completion for video

* add --video arg

* fix build

* update docs

* rename as suggested
 2026-06-08 14:40:12 +03:00