Concedo
e541554d70
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/ui-build.yml
# .github/workflows/ui-publish.yml
# .github/workflows/ui-self-hosted.yml
# CMakeLists.txt
# app/CMakeLists.txt
# app/llama.cpp
# common/arg.cpp
# ggml/CMakeLists.txt
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-opencl/ggml-opencl.cpp
# scripts/snapdragon/ggml-hexagon-profile.py
# scripts/sync-ggml.last
# src/CMakeLists.txt
# tests/test-llama-archs.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/mtmd/CMakeLists.txt
# tools/mtmd/tests/test-deepseek-ocr.py
# tools/server/README.md
2026-05-30 01:54:53 +08:00
Concedo
baacd640c0
Merge commit ' dd1557907a' into concedo_experimental
...
# Conflicts:
# .github/workflows/build-vulkan.yml
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/hmx-flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/hmx-matmul-ops.c
# tests/test-backend-ops.cpp
# tests/test-chat-template.cpp
# tests/test-chat.cpp
# tests/test-llama-archs.cpp
# tools/perplexity/perplexity.cpp
# tools/server/README.md
2026-05-30 01:40:09 +08:00
Xuan-Son Nguyen
06d26dfdff
download: add option to skip_download ( #23059 )
...
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Waiting to run
Python check requirements.txt / check-requirements (push) Waiting to run
Python Type-Check / python type-check (push) Waiting to run
* download: add option to skip_download
* fix
* fix 2
* if file doesn't exist, respect skip_download flag
2026-05-29 16:30:55 +02:00
Xuan-Son Nguyen
cb47092b00
server: bump timeout to 3600s ( #23842 )
...
* server: bump timeout to 3600s
* nits: change wording
2026-05-29 10:23:17 +02:00
Adrien Gallouët
98e480a32e
app : move licences to llama-app ( #23824 )
...
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-05-29 07:46:11 +02:00
Mikolaj Kucharski
7fb1e70b59
arg: Add LLAMA_ARG_API_KEY_FILE environment variable for --api-key-file ( #23167 )
2026-05-28 16:25:40 +02:00
Funtowicz Morgan
0b246862b9
server: minor tweaks to use more cpp features ( #23785 )
...
* misc(server): add default port to impl RAII
* misc(server): register_gcp_compat() can be const
* misc(server): use proper cpp const/auto methods
* misc(server): do not reset a unique_ptr, use make_unique instead to be exception safe
2026-05-28 14:00:25 +02:00
Markus Tavenrath
d205df6812
server, ui : Add support for HTTP ETags in llama-server ( #23701 )
...
* allow caching of ui elements in llama-server
* use fnv_hash
* Update tools/server/server-http.cpp
etag has to be set always
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
---------
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
2026-05-28 12:21:24 +02:00
Concedo
ca942eb172
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/actions/windows-setup-cuda/action.yml
# .github/workflows/build-3rd-party.yml
# .github/workflows/build-android.yml
# .github/workflows/build-apple.yml
# .github/workflows/build-cann.yml
# .github/workflows/build-cuda-ubuntu.yml
# .github/workflows/build-ibm.yml
# .github/workflows/build-msys.yml
# .github/workflows/build-opencl.yml
# .github/workflows/build-openvino.yml
# .github/workflows/build-riscv.yml
# .github/workflows/build-rpc.yml
# .github/workflows/build-sanitize.yml
# .github/workflows/build-self-hosted.yml
# .github/workflows/build-sycl.yml
# .github/workflows/build-vulkan.yml
# .github/workflows/build-webgpu.yml
# .github/workflows/hip-quality-check.yml
# .github/workflows/release.yml
# .github/workflows/server-sanitize.yml
# .github/workflows/server-self-hosted.yml
# .github/workflows/server.yml
# .github/workflows/ui-self-hosted.yml
# .github/workflows/ui.yml
# CONTRIBUTING.md
# ci/run.sh
# docs/autoparser.md
# docs/multimodal/granitevision.md
# examples/model-conversion/README.md
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/gated-delta-net-ops.c
# ggml/src/ggml-hexagon/htp/hmx-matmul-ops.c
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-vulkan/CMakeLists.txt
# ggml/src/ggml-vulkan/ggml-vulkan.cpp
# ggml/src/ggml-vulkan/vulkan-shaders/CMakeLists.txt
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/cpy.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_id_gather.wgsl
# ggml/src/ggml-zendnn/ggml-zendnn.cpp
# pyproject.toml
# scripts/snapdragon/adb/run-completion.sh
# scripts/snapdragon/adb/run-tool.sh
# scripts/sync_vendor.py
# tests/gguf-model-data.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/server/README.md
2026-05-28 15:28:29 +08:00
Georgi Gerganov
6b4e4bd582
common : fix env names to all have LLAMA_ARG_ prefix ( #23778 )
2026-05-27 14:52:47 +03:00
Radoslav Gerganov
7085492c6f
server : fix the log message when using SSL ( #23393 )
...
When llama-server is started with SSL key and cert, the log says that it
listens on http instead of https. This patch fixes this.
2026-05-27 08:06:30 +03:00
Concedo
9204f78926
Merge commit ' ae251b5ff2' into concedo_experimental
...
# Conflicts:
# .github/actions/linux-setup-spacemit/action.yml
# .github/actions/unarchive-tar/action.yml
# .github/workflows/build-android.yml
# .github/workflows/build-cmake-pkg.yml
# .github/workflows/build-cross.yml
# .github/workflows/build-self-hosted.yml
# .github/workflows/build.yml
# .github/workflows/check-vendor.yml
# .github/workflows/code-style.yml
# .github/workflows/editorconfig.yml
# .github/workflows/pre-tokenizer-hashes.yml
# .github/workflows/python-check-requirements.yml
# .github/workflows/python-lint.yml
# .github/workflows/python-type-check.yml
# .github/workflows/server-self-hosted.yml
# .github/workflows/ui-build.yml
# .github/workflows/ui.yml
# .github/workflows/update-ops-docs.yml
# ci/run.sh
# docs/build-riscv64-spacemit.md
# examples/convert_legacy_llama.py
# ggml/cmake/ggml-config.cmake.in
# ggml/src/CMakeLists.txt
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# scripts/sync_vendor.py
# tests/test-chat-auto-parser.cpp
# tests/test-chat.cpp
# tests/test-gguf.cpp
# tools/cli/README.md
# tools/perplexity/perplexity.cpp
# tools/server/README.md
2026-05-26 22:01:57 +08:00
Aman Gupta
6c4cbdc70b
server: MTP layer kv-cache should respect draft type ctk ( #23646 )
2026-05-25 16:46:23 +08:00
jacekpoplawski
e2ef8fe42c
server: fix checkpoints creation ( #22929 )
...
* common : add common_chat_split_by_role
* cont : fix spans to reach end of message
* server: fix checkpoints creation
- extract message_spans from chat templates
- find the prompt token position before the latest user message
- split prompt batching at that position
- create a context checkpoint before the latest user input
- avoid periodic mid-prompt checkpoints when that position is known
- handle multimodal prompts when mapping text/template positions to server prompt tokens
- add --checkpoint-min-step to control minimum spacing between checkpoints
* cont : clean-up
* Support autoparser detection for message barriers
* server: fix message span delimiter and update docs
---------
Co-authored-by: Alde Rojas <hello@alde.dev>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
2026-05-25 08:56:18 +03:00
Aman Gupta
83eebe9d08
server: add margin for draft model for fit ( #23485 )
2026-05-24 14:43:08 +08:00
Concedo
8ca4283f55
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/release.yml
# .github/workflows/server.yml
# .github/workflows/ui-build.yml
# .github/workflows/ui-publish.yml
# CMakeLists.txt
# docs/autoparser.md
# docs/backend/snapdragon/CMakeUserPresets.json
# docs/backend/snapdragon/README.md
# docs/backend/snapdragon/windows.md
# docs/function-calling.md
# examples/model-conversion/scripts/embedding/run-original-model.py
# ggml/src/ggml-hexagon/htp/hmx-flash-attn-ops.c
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/cvt.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_mxfp4_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q4_0_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q4_1_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q4_k_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q5_0_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q5_1_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q5_k_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q6_k_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_mxfp4_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_q4_0_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_q4_1_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_q4_k_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_q5_0_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_q5_1_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_q5_k_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_q6_k_f32_ns.cl
# ggml/src/ggml-sycl/common.hpp
# ggml/src/ggml-sycl/dmmv.cpp
# ggml/src/ggml-sycl/gated_delta_net.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-vulkan/CMakeLists.txt
# ggml/src/ggml-zendnn/CMakeLists.txt
# ggml/src/ggml-zendnn/ggml-zendnn.cpp
# requirements/requirements-convert_hf_to_gguf.txt
# scripts/snapdragon/windows/setup-build.ps1
# tools/perplexity/perplexity.cpp
2026-05-24 13:55:44 +08:00
Aldehir Rojas
b22ff4b7b4
cmake/ui : refactor the build ( #23352 )
2026-05-23 17:08:22 -04:00
Aldehir Rojas
1acee6bf89
server: only parse empty msg if continuing an assistant msg ( #23506 )
2026-05-22 11:58:15 -04:00
Concedo
632c41a72f
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/build-apple.yml
# .github/workflows/build-cmake-pkg.yml
# .github/workflows/release.yml
# .pi/gg/SYSTEM.md
# CMakeLists.txt
# CODEOWNERS
# README.md
# build-xcframework.sh
# ci/run.sh
# docs/build.md
# examples/CMakeLists.txt
# examples/llama.android/lib/build.gradle.kts
# ggml/src/ggml-webgpu/wgsl-shaders/flash_attn_tile.wgsl
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tests/test-save-load-state.cpp
# tools/batched-bench/CMakeLists.txt
# tools/cli/CMakeLists.txt
# tools/completion/CMakeLists.txt
# tools/llama-bench/CMakeLists.txt
# tools/perplexity/CMakeLists.txt
# tools/quantize/CMakeLists.txt
# tools/server/CMakeLists.txt
2026-05-22 20:42:51 +08:00
Georgi Gerganov
bbce619adb
cmake : add install() for impl libraries + fix apple builds ( #23511 )
...
* pi : update
* ci : fix ios build
* ci : fix andoroid
* ci : fix apple builds
* cmake : add install() for impl libraries
Add install(TARGETS <target> LIBRARY) for all -impl libraries that were
changed from STATIC to shared (controlled by BUILD_SHARED_LIBS) in
commit bb28c1fe2https://github.com/ggml-org/llama.cpp/issues/23494#issuecomment-4512912515
Assisted-by: llama.cpp:local pi
* ci : fix xcframework build
2026-05-22 11:46:26 +03:00
Georgi Gerganov
bb28c1fe24
cmake : remove STATIC from impl libraries, enable LLAMA_BUILD_APP by default ( #23462 )
...
* cmake : remove STATIC from impl libraries, allow BUILD_SHARED_LIBS control
Remove explicit STATIC from all -impl libraries (server, cli, completion, bench,
batched-bench, fit-params, quantize, perplexity) so BUILD_SHARED_LIBS controls
shared vs static linkage.
Add WINDOWS_EXPORT_ALL_SYMBOLS ON for proper DLL export on Windows.
Assisted-by: llama.cpp:local pi
* cmake : enable LLAMA_BUILD_APP by default
Assisted-by: llama.cpp:local pi
* ci : disable app in build-cmake-pkg.yml
2026-05-21 21:13:59 +03:00
Concedo
718dc159b6
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# CMakeLists.txt
# docs/speculative.md
# ggml/src/ggml-cuda/CMakeLists.txt
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/hmx-matmul-ops.c
# ggml/src/ggml-hexagon/htp/hmx-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# ggml/src/ggml-hexagon/htp/rope-ops.c
# ggml/src/ggml-hexagon/htp/ssm-conv.c
# ggml/src/ggml-opencl/ggml-opencl.cpp
# scripts/snapdragon/adb/run-bench.sh
# scripts/snapdragon/adb/run-cli.sh
# scripts/snapdragon/adb/run-completion.sh
# scripts/snapdragon/adb/run-mtmd.sh
# scripts/snapdragon/windows/run-bench.ps1
# scripts/snapdragon/windows/run-cli.ps1
# scripts/snapdragon/windows/run-completion.ps1
# scripts/snapdragon/windows/run-mtmd.ps1
# src/llama-vocab.cpp
# tests/test-backend-ops.cpp
# tools/batched-bench/CMakeLists.txt
# tools/batched-bench/batched-bench.cpp
# tools/cli/CMakeLists.txt
# tools/cli/README.md
# tools/cli/cli.cpp
# tools/completion/CMakeLists.txt
# tools/completion/README.md
# tools/llama-bench/CMakeLists.txt
# tools/llama-bench/llama-bench.cpp
# tools/mtmd/CMakeLists.txt
# tools/mtmd/tests/test-deepseek-ocr.py
# tools/mtmd/tests/tests-requirements.txt
# tools/perplexity/CMakeLists.txt
# tools/perplexity/perplexity.cpp
# tools/quantize/CMakeLists.txt
# tools/server/CMakeLists.txt
# tools/server/README.md
# ty.toml
2026-05-21 23:47:21 +08:00
Concedo
54af9aada9
Merge commit ' e6b4acfe86' into concedo_experimental
...
# Conflicts:
# .devops/cann.Dockerfile
# .devops/cpu.Dockerfile
# .devops/cuda.Dockerfile
# .devops/intel.Dockerfile
# .devops/musa.Dockerfile
# .devops/openvino.Dockerfile
# .devops/rocm.Dockerfile
# .devops/s390x.Dockerfile
# .devops/vulkan.Dockerfile
# tools/mtmd/clip.cpp
# tools/mtmd/clip.h
2026-05-21 23:31:32 +08:00
ScrewTSW
b65bb4baae
server: expose prompt token counts in /slots endpoint ( #23454 )
...
Add n_prompt_tokens, n_prompt_tokens_processed, and n_prompt_tokens_cache
to the /slots JSON response. These fields are already tracked internally
but were not exposed, making it impossible for clients to monitor prompt
evaluation progress during processing.
2026-05-21 13:29:13 +02:00
Aman Gupta
52fb93a2bd
server : free draft/MTP resources on sleep to fix VRAM leak ( #23461 )
...
The destroy() function in server_context_impl only cleaned up the main
model and context (via llama_init.reset()) but did not free the speculative
decoder (spec), draft context (ctx_dft), or draft model (model_dft).
For MTP (Multi-Token Prediction) models, ctx_dft holds GPU-allocated
resources (KV cache, compute buffers) that are not freed when entering
the sleeping state. On each sleep/resume cycle, new resources are
allocated without the old ones being freed, leading to a VRAM leak
that eventually crashes the server with out-of-memory errors.
Fix by explicitly resetting spec, ctx_dft, and model_dft in destroy()
before resetting llama_init, ensuring proper cleanup order to avoid
use-after-free.
ref: https://github.com/ggml-org/llama.cpp/issues/23395
Assisted-by: llama.cpp:local pi
2026-05-21 16:11:11 +08:00
Pascal
c9021714e8
server: re-inject subcommand when router spawns children under unified binary ( #23442 )
2026-05-21 10:09:19 +02:00
wendadawen
6a257d4463
mtmd, model : merge HunyuanOCR into HunyuanVL and fix OCR vision precision ( #23329 )
...
- HunyuanOCR shares the same HF arch and vision layout as HunyuanVL butwas split into a separate path that skipped the +0.1 bilinear sampler used by the HF reference.
- Collapse OCR into the HUNYUANVL projector + HUNYUAN_VL text arch
2026-05-21 00:35:37 +02:00
Adrien Gallouët
29f1482221
app : introduce the llama unified executable ( #23296 )
...
* app : introduce the llama unified executable
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Use serve for server
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Hide completion and bench, add help command
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Remove STATIC
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Use -impl targets instead of -lib
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Revert "Remove STATIC"
This reverts commit cc44caccb9902b34a3531633edac911e5b3d65cd.
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-05-20 13:22:22 +02:00
Concedo
7d987af23a
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/cann.Dockerfile
# .devops/cpu.Dockerfile
# .devops/cuda.Dockerfile
# .devops/intel.Dockerfile
# .devops/llama-cli-cann.Dockerfile
# .devops/musa.Dockerfile
# .devops/openvino.Dockerfile
# .devops/rocm.Dockerfile
# .devops/s390x.Dockerfile
# .devops/vulkan.Dockerfile
# .github/ISSUE_TEMPLATE/011-bug-results.yml
# .github/ISSUE_TEMPLATE/019-bug-misc.yml
# .github/workflows/build-and-test-snapdragon.yml
# .github/workflows/docker.yml
# .github/workflows/server-self-hosted.yml
# .github/workflows/ui-ci.yml
# .pi/gg/SYSTEM.md
# README.md
# common/arg.cpp
# docs/backend/SYCL.md
# docs/backend/snapdragon/CMakeUserPresets.json
# docs/backend/snapdragon/README.md
# docs/speculative.md
# examples/save-load-state/save-load-state.cpp
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/rope-ops.c
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/cvt.cl
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/gated_delta_net.wgsl
# tools/cli/README.md
# tools/server/README.md
2026-05-20 18:48:34 +08:00
Xuan-Son Nguyen
e2b129e1bf
mtmd: fit_params now take into account mmproj ( #21489 )
...
* mtmd: fit_params now take into account mmproj
* rename alloc_compute_meta to reserve_compute_meta
* rm unused functions
* add ggml_backend_dev_t support
* add debug log
2026-05-20 11:27:44 +02:00
Georgi Gerganov
d14ce3dab4
llama : MTP clean-up ( #23269 )
...
* llama : disable equal splits for recurrent memory with partial rollback
* spec : re-enable p-min with MTP drafts
* spec : re-enable ngram spec in combination with RS rollback
* spec : fix ngram-map-* params
* spec : fix acceptance logic in combined ngram + draft configs
* graph : fix reuse for combined `token` + `embd` batches
* spec : log parameters for each speculative implementation
- add LOG_INF in each constructor with implementation type and parameters
- extract device string logic into common_speculative_get_devices_str()
- move 'adding speculative implementation' log from init into constructors
Assisted-by: llama.cpp:local pi
* spec : extend --spec-default with ngram-map-k4v
Assisted-by: llama.cpp:local pi
* minor : fix n_embd log
* args : update draft.n_max == 3 + regen docs
* spec : relax ngram-mod rejection thold to 0.25 @ 5 low
* logs : improve
* docs : update speculative decoding CLI argument documentation
- Add missing draft model CPU scheduling and tensor override parameters
- Update --spec-type to include all available types (excluding draft-eagle3 WIP)
- Fix default values to match implementation (n_max=3, n_min=0, p_min=0.0)
- Remove deprecated options (spec-draft-ctx-size, spec-draft-replace)
- Add environment variables for new parameters
Assisted-by: llama.cpp:local pi
* arg : step-back on adding k4v to the default spec config
* cont : fix name
2026-05-19 15:32:58 +03:00
Pascal
ccee426426
server-context: guarantee there is at least 1 token to decode ( #23280 )
2026-05-19 09:49:01 +03:00
Georgi Gerganov
3c81c8deea
server : print graphs reused in slot timings ( #23279 )
...
Add graphs reused counter to the per-slot timing output, printed via
llama_perf_context().
Assisted-by: llama.cpp:local pi
Co-authored-by: ggerganov <ggerganov@users.noreply.github.com>
2026-05-19 09:46:58 +03:00
Aleksander Grygier
1ff0fc1384
ui: Refactor models store, MCP service, and gate logs behind VITE_DEBUG ( #23236 )
...
* refactor: Scope console logs to `DEV` + `VITE_DEBUG` env vars
* refactor: skip MCP proxy probe when no server requires it
* refactor: suppress expected disconnect errors during MCP client shutdown
* refactor: Deduplicate requests
* refactor: deduplicate model fetching across ROUTER and MODEL modes
* refactor: Clean up models logic
* chore: Add `.env.example` file
* refactor: replace client-side CORS proxy probe with server status flag
* refactor: Post-review fixes
* test: add vitest client setup with API fetch mocks
2026-05-18 16:09:40 +02:00
Concedo
fecf2dc3fa
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/server-self-hosted.yml
# CMakeLists.txt
# CODEOWNERS
# ci/run.sh
# cmake/llama-config.cmake.in
# common/chat.cpp
# examples/sycl/start-svr.sh
# examples/sycl/test.sh
# examples/sycl/win-start-svr.bat
# examples/sycl/win-test.bat
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/vecdotq.hpp
# ggml/src/ggml-vulkan/CMakeLists.txt
# scripts/wc2wt.sh
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
2026-05-18 21:27:23 +08:00
Aldehir Rojas
87589042ca
cmake : fix LLAMA_BUILD_UI logic ( #23190 )
2026-05-17 14:42:26 -04:00
Aman Gupta
3e12fbdea5
llama: avoid copying logits during prompt decode in MTP ( #23198 )
...
* llama: avoid copying logits during prompt decode in MTP
* review: update comment
* llama-graph: call set_output for t_h_pre_norm
2026-05-17 23:30:25 +08:00
Aldehir Rojas
39cf5d6191
common : delegate assistant continuation to underlying template handlers ( #23089 )
...
* common : delegate assistant continuation to template handler
* server : implement echo parameter to exclude assistant prefill in the response
* server : fix tests for prefill
* server : use existing llama template
* cont : clean up
2026-05-17 13:36:05 +02:00
Rares Vernica
1a68ec9378
server : honor --embd-normalize CLI arg ( #23125 )
...
The --embd-normalize flag was registered only for the embedding and debug
examples, so llama-server rejected it and the /embedding handler used a
hard-coded default of 2 (L2). Add LLAMA_EXAMPLE_SERVER to the flag's
example set and read params.embd_normalize as the handler's default. The
per-request "embd_normalize" body field continues to override.
2026-05-17 09:39:04 +03:00
Concedo
1e828ccabf
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# common/common.cpp
# ggml/CMakeLists.txt
# scripts/sync-ggml.last
# scripts/sync_vendor.py
# src/llama-context.cpp
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/server/README.md
2026-05-17 11:26:18 +08:00
Xuan-Son Nguyen
b64739ea39
server: (router) alloc tmp buffer on heap ( #23159 )
2026-05-16 23:42:16 +02:00
Pascal
64b38b561b
server: skip device enumeration in router mode to avoid creating CUDA primary context ( #23137 )
2026-05-16 21:21:06 +02:00
Concedo
9203b6a051
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/labeler.yml
# .github/workflows/build-self-hosted.yml
# .github/workflows/release.yml
# .github/workflows/server-sanitize.yml
# .github/workflows/server-self-hosted.yml
# .github/workflows/server.yml
# .github/workflows/ui-build.yml
# .github/workflows/ui-ci.yml
# .github/workflows/ui-publish.yml
# .gitignore
# CMakeLists.txt
# CODEOWNERS
# scripts/ui-download.cmake
# scripts/xxd.cmake
# tests/test-backend-ops.cpp
# tests/test-reasoning-budget.cpp
# tools/CMakeLists.txt
# tools/server/CMakeLists.txt
# tools/server/README.md
2026-05-16 22:56:33 +08:00
Aman Gupta
255582687b
llama + spec: MTP Support ( #22673 )
...
* spec: support MTP
* fix batch size
* rename files
* cont : simplify (#7 )
* MTP: clean-up (#9 )
* MTP: clean-up
* review: use llama_context_type instead of llama_graph_type
* review: remove llama_model_has_mtp
* review: fix convert issues
* convert: fix pycheck
* review: formatting
* use `mtp-` for identifying mtp models
* convert: fix mtp conversion
* mtp -> draft-mtp
* remove unused llama_arch
* add need_embd in speculative
* llama: allow partial seq_rm for GDN models for speculative decoding
Currently speculative checkpoint needs to restart from a checkpoint
after some draft tokens are not accepted, this leads to some wastage in
running the target again. This PR adds the ability to rollback upto
`draft_max` by storing the GDN intermediates.
* fix pending state
* vulkan: add GDN partial rollback
* meta: extend check to axis 1
* metal: add GDN partial rollback
Extend the gated delta net kernel to store intermediate states for
partial rollback support on the Metal backend.
- Add K (snapshot slot count) as a function constant
- Read input state from slot 0 of the 3D state tensor
- Write intermediate states to different slots during token loop
- For K=1, maintain backward-compatible single-slot behavior
Ref: 8c05923630#11 )
* server : adjust checkpoint logic
* cont : rm asserts
* server-context: fix early exit
* spec : fix compatibility with n-gram and add TODOs (#13 )
* metal : cleanup
* llama : fix faulty bitwise check in recurrent memory
* server : disable RS-based MTP in combination with other spec types
* spec : add TODOs
* cont : fix comment
* cont : update comment
* common : fix logic for ngram + mtp compat
* llama-memory: enable checkpointing with partial rollback
* cont: add test-case for loading into a dirty ctx
* llama-memory-recurrent: clear rs_idx in clear
* download: fix mtp path
* llama-arch: fix enorm op
* docs: update docs
* conversion: fix type annotations
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-05-16 20:06:23 +08:00
Aleksander Grygier
59778f0196
ui: Restructure repo to use tools/ui folder and ui / UI / llama-ui / LLAMA_UI naming ( #23064 )
...
* webui: Move static build output from `tools/server/public` to `build/ui` directory
* refactor: Move to `tools/ui`
* refactor: rename CMake variables and preprocessor defines
- Rename LLAMA_BUILD_WEBUI -> LLAMA_BUILD_UI (old kept as deprecated)
- Rename LLAMA_USE_PREBUILT_WEBUI -> LLAMA_USE_PREBUILT_UI (old kept as deprecated)
- Backward compat: old vars auto-forward to new ones with DEPRECATION warning
- Rename internal vars: WEBUI_SOURCE -> UI_SOURCE, WEBUI_SOURCE_DIR -> UI_SOURCE_DIR, etc.
- Rename HF bucket: LLAMA_WEBUI_HF_BUCKET -> LLAMA_UI_HF_BUCKET
- Emit both LLAMA_BUILD_WEBUI and LLAMA_BUILD_UI preprocessor defines
- Emit both LLAMA_WEBUI_DEFAULT_ENABLED and LLAMA_UI_DEFAULT_ENABLED
* refactor: rename CLI flags (--webui -> --ui) with backward compat
- Add --ui/--no-ui (old --webui/--no-webui kept as deprecated aliases)
- Add --ui-config (old --webui-config kept as deprecated alias)
- Add --ui-config-file (old --webui-config-file kept as deprecated alias)
- Add --ui-mcp-proxy/--no-ui-mcp-proxy (old --webui-mcp-proxy kept as deprecated)
- Add new env vars: LLAMA_ARG_UI, LLAMA_ARG_UI_CONFIG, LLAMA_ARG_UI_CONFIG_FILE, LLAMA_ARG_UI_MCP_PROXY
- C++ struct fields: params.ui, params.ui_config_json, params.ui_mcp_proxy added alongside old fields
- Backward compat: old fields synced to new ones in g_params_to_internals
* refactor: update C++ server internals with backward compat
- Rename json_webui_settings -> json_ui_settings (both kept in server_context_meta)
- Rename params.webui usage -> params.ui (both synced, old still works)
- JSON API emits both "ui"/"ui_settings" and "webui"/"webui_settings" keys
- Server routes use params.ui_mcp_proxy || params.webui_mcp_proxy
- Preprocessor guards use #if defined(LLAMA_BUILD_UI) || defined(LLAMA_BUILD_WEBUI)
* refactor: rename CI/CD workflows, artifacts, and build script
- Rename webui-build.yml -> ui-build.yml; artifact webui-build -> ui-build
- Rename webui-publish.yml -> ui-publish.yml; var HF_BUCKET_WEBUI_STATIC_OUTPUT -> HF_BUCKET_UI_STATIC_OUTPUT
- Rename server-webui.yml -> server-ui.yml; job webui-build/checks -> ui-build/checks
- Update server.yml: job/artifact refs webui-build -> ui-build
- Update release.yml: all webui-build/publish refs -> ui-build/publish; HF_TOKEN_WEBUI_STATIC_OUTPUT -> HF_TOKEN_UI_STATIC_OUTPUT
- Update server-self-hosted.yml: webui-build -> ui-build
- Update build-self-hosted.yml: HF_WEBUI_VERSION -> HF_UI_VERSION
- Rename webui-download.cmake -> ui-download.cmake (internal refs updated)
- Update labeler.yml: server/webui -> server/ui path label
* docs: update CODEOWNERS and server README docs
- Update CODEOWNERS: team ggml-org/llama-webui -> ggml-org/llama-ui, path /tools/server/webui/ -> /tools/ui/
- Update server README.md: CLI tables show --ui flags with deprecated --webui aliases
- Update server README-dev.md: "WebUI" -> "UI", paths updated to tools/ui/
* fix: Small fixes for UI build
* fix: CMake.txt syntax
* chore: Formatting
* fix: `.editorconfig` for llama-ui
* chore: Formatting
* refactor: Use `APP_NAME` in Error route
* refactor: Cleanup
* refactor: Single migration service
* make llama-ui a linkable target
* fix: UI Build output
* fix: Missing change
* fix: separate llama-ui npm build output into build/tools/ui/dist subfolder + use cmake npm build instead of downloading ui-build.yml artifacts in CI
* refactor: UI workflows cleanup
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-05-16 02:02:40 +02:00
Julien Chaumond
6831fe470c
docs: document usage object in server timings response ( #23110 )
...
* docs: document `usage` object in server timings response
Co-Authored-By: julien-agent <Agents+cyolo@huggingface.co>
* Apply suggestion from @julien-c
---------
Co-authored-by: julien-agent <Agents+cyolo@huggingface.co>
2026-05-15 19:33:12 +02:00
Pascal
8be1786707
webui: fix theme from --webui-config-file not applied on first load (fresh localStorage) ( #22902 )
2026-05-15 19:25:38 +02:00
Pascal
d528444580
webui: preserve partial response on streaming error ( #23090 )
2026-05-15 11:18:11 +02:00
Concedo
da2cc90723
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/labeler.yml
# .github/workflows/build-and-test-snapdragon.yml
# .github/workflows/build-self-hosted.yml
# .github/workflows/release.yml
# .github/workflows/server-self-hosted.yml
# .github/workflows/server-webui.yml
# .github/workflows/server.yml
# .gitignore
# CMakeLists.txt
# CONTRIBUTING.md
# README.md
# ggml/src/ggml-cuda/fattn.cu
# ggml/src/ggml-hexagon/htp/cpy-ops.c
# ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# grammars/README.md
# scripts/snapdragon/qdc/run_qdc_jobs.py
# scripts/snapdragon/qdc/tests/run_backend_ops_posix.py
# scripts/snapdragon/qdc/tests/run_bench_tests_posix.py
# scripts/snapdragon/qdc/tests/utils.py
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
# tools/server/CMakeLists.txt
# tools/server/README.md
# tools/server/webui/src/lib/components/app/server/ServerLoadingSplash.svelte
# tools/server/webui/src/routes/(chat)/chat/[id]/+page.svelte
# ty.toml
2026-05-15 17:09:48 +08:00
Sid Shaytay
91e84fed64
Support for Codex CLI by skipping unsupported Responses tools ( #23041 )
...
Python Type-Check / python type-check (push) Waiting to run
* Support for Codex CLI by skipping unsupported Responses tools
* Warn on skipped Responses tools and preserve gpt-oss apply_patch rejection
* Revert gpt-oss apply_patch special handling
2026-05-15 09:03:24 +02:00
