Concedo
c30a386932
fix tools build
2026-05-30 08:55:37 +08:00
Concedo
e541554d70
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/ui-build.yml
# .github/workflows/ui-publish.yml
# .github/workflows/ui-self-hosted.yml
# CMakeLists.txt
# app/CMakeLists.txt
# app/llama.cpp
# common/arg.cpp
# ggml/CMakeLists.txt
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-opencl/ggml-opencl.cpp
# scripts/snapdragon/ggml-hexagon-profile.py
# scripts/sync-ggml.last
# src/CMakeLists.txt
# tests/test-llama-archs.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/mtmd/CMakeLists.txt
# tools/mtmd/tests/test-deepseek-ocr.py
# tools/server/README.md
2026-05-30 01:54:53 +08:00
Concedo
baacd640c0
Merge commit ' dd1557907a' into concedo_experimental
...
# Conflicts:
# .github/workflows/build-vulkan.yml
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/hmx-flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/hmx-matmul-ops.c
# tests/test-backend-ops.cpp
# tests/test-chat-template.cpp
# tests/test-chat.cpp
# tests/test-llama-archs.cpp
# tools/perplexity/perplexity.cpp
# tools/server/README.md
2026-05-30 01:40:09 +08:00
Xuan-Son Nguyen
06d26dfdff
download: add option to skip_download ( #23059 )
...
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Waiting to run
Python check requirements.txt / check-requirements (push) Waiting to run
Python Type-Check / python type-check (push) Waiting to run
* download: add option to skip_download
* fix
* fix 2
* if file doesn't exist, respect skip_download flag
2026-05-29 16:30:55 +02:00
Xuan-Son Nguyen
cb47092b00
server: bump timeout to 3600s ( #23842 )
...
* server: bump timeout to 3600s
* nits: change wording
2026-05-29 10:23:17 +02:00
Omid Azizi
b000431a0b
ngram-mod : Add missing include ( #23857 )
...
[no release]
Signed-off-by: Omid Azizi <oazizi@gimletlabs.ai>
2026-05-29 09:21:37 +03:00
Adrien Gallouët
98e480a32e
app : move licences to llama-app ( #23824 )
...
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-05-29 07:46:11 +02:00
Mikolaj Kucharski
7fb1e70b59
arg: Add LLAMA_ARG_API_KEY_FILE environment variable for --api-key-file ( #23167 )
2026-05-28 16:25:40 +02:00
Concedo
ca942eb172
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/actions/windows-setup-cuda/action.yml
# .github/workflows/build-3rd-party.yml
# .github/workflows/build-android.yml
# .github/workflows/build-apple.yml
# .github/workflows/build-cann.yml
# .github/workflows/build-cuda-ubuntu.yml
# .github/workflows/build-ibm.yml
# .github/workflows/build-msys.yml
# .github/workflows/build-opencl.yml
# .github/workflows/build-openvino.yml
# .github/workflows/build-riscv.yml
# .github/workflows/build-rpc.yml
# .github/workflows/build-sanitize.yml
# .github/workflows/build-self-hosted.yml
# .github/workflows/build-sycl.yml
# .github/workflows/build-vulkan.yml
# .github/workflows/build-webgpu.yml
# .github/workflows/hip-quality-check.yml
# .github/workflows/release.yml
# .github/workflows/server-sanitize.yml
# .github/workflows/server-self-hosted.yml
# .github/workflows/server.yml
# .github/workflows/ui-self-hosted.yml
# .github/workflows/ui.yml
# CONTRIBUTING.md
# ci/run.sh
# docs/autoparser.md
# docs/multimodal/granitevision.md
# examples/model-conversion/README.md
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/gated-delta-net-ops.c
# ggml/src/ggml-hexagon/htp/hmx-matmul-ops.c
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-vulkan/CMakeLists.txt
# ggml/src/ggml-vulkan/ggml-vulkan.cpp
# ggml/src/ggml-vulkan/vulkan-shaders/CMakeLists.txt
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/cpy.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_id_gather.wgsl
# ggml/src/ggml-zendnn/ggml-zendnn.cpp
# pyproject.toml
# scripts/snapdragon/adb/run-completion.sh
# scripts/snapdragon/adb/run-tool.sh
# scripts/sync_vendor.py
# tests/gguf-model-data.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/server/README.md
2026-05-28 15:28:29 +08:00
Georgi Gerganov
6b4e4bd582
common : fix env names to all have LLAMA_ARG_ prefix ( #23778 )
2026-05-27 14:52:47 +03:00
Concedo
9204f78926
Merge commit ' ae251b5ff2' into concedo_experimental
...
# Conflicts:
# .github/actions/linux-setup-spacemit/action.yml
# .github/actions/unarchive-tar/action.yml
# .github/workflows/build-android.yml
# .github/workflows/build-cmake-pkg.yml
# .github/workflows/build-cross.yml
# .github/workflows/build-self-hosted.yml
# .github/workflows/build.yml
# .github/workflows/check-vendor.yml
# .github/workflows/code-style.yml
# .github/workflows/editorconfig.yml
# .github/workflows/pre-tokenizer-hashes.yml
# .github/workflows/python-check-requirements.yml
# .github/workflows/python-lint.yml
# .github/workflows/python-type-check.yml
# .github/workflows/server-self-hosted.yml
# .github/workflows/ui-build.yml
# .github/workflows/ui.yml
# .github/workflows/update-ops-docs.yml
# ci/run.sh
# docs/build-riscv64-spacemit.md
# examples/convert_legacy_llama.py
# ggml/cmake/ggml-config.cmake.in
# ggml/src/CMakeLists.txt
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# scripts/sync_vendor.py
# tests/test-chat-auto-parser.cpp
# tests/test-chat.cpp
# tests/test-gguf.cpp
# tools/cli/README.md
# tools/perplexity/perplexity.cpp
# tools/server/README.md
2026-05-26 22:01:57 +08:00
jacekpoplawski
e2ef8fe42c
server: fix checkpoints creation ( #22929 )
...
* common : add common_chat_split_by_role
* cont : fix spans to reach end of message
* server: fix checkpoints creation
- extract message_spans from chat templates
- find the prompt token position before the latest user message
- split prompt batching at that position
- create a context checkpoint before the latest user input
- avoid periodic mid-prompt checkpoints when that position is known
- handle multimodal prompts when mapping text/template positions to server prompt tokens
- add --checkpoint-min-step to control minimum spacing between checkpoints
* cont : clean-up
* Support autoparser detection for message barriers
* server: fix message span delimiter and update docs
---------
Co-authored-by: Alde Rojas <hello@alde.dev>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
2026-05-25 08:56:18 +03:00
Aman Gupta
83eebe9d08
server: add margin for draft model for fit ( #23485 )
2026-05-24 14:43:08 +08:00
Concedo
8ca4283f55
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/release.yml
# .github/workflows/server.yml
# .github/workflows/ui-build.yml
# .github/workflows/ui-publish.yml
# CMakeLists.txt
# docs/autoparser.md
# docs/backend/snapdragon/CMakeUserPresets.json
# docs/backend/snapdragon/README.md
# docs/backend/snapdragon/windows.md
# docs/function-calling.md
# examples/model-conversion/scripts/embedding/run-original-model.py
# ggml/src/ggml-hexagon/htp/hmx-flash-attn-ops.c
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/cvt.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_mxfp4_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q4_0_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q4_1_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q4_k_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q5_0_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q5_1_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q5_k_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemm_moe_q6_k_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_mxfp4_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_q4_0_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_q4_1_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_q4_k_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_q5_0_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_q5_1_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_q5_k_f32_ns.cl
# ggml/src/ggml-opencl/kernels/gemv_moe_q6_k_f32_ns.cl
# ggml/src/ggml-sycl/common.hpp
# ggml/src/ggml-sycl/dmmv.cpp
# ggml/src/ggml-sycl/gated_delta_net.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-vulkan/CMakeLists.txt
# ggml/src/ggml-zendnn/CMakeLists.txt
# ggml/src/ggml-zendnn/ggml-zendnn.cpp
# requirements/requirements-convert_hf_to_gguf.txt
# scripts/snapdragon/windows/setup-build.ps1
# tools/perplexity/perplexity.cpp
2026-05-24 13:55:44 +08:00
Aldehir Rojas
b22ff4b7b4
cmake/ui : refactor the build ( #23352 )
2026-05-23 17:08:22 -04:00
Aldehir Rojas
1acee6bf89
server: only parse empty msg if continuing an assistant msg ( #23506 )
2026-05-22 11:58:15 -04:00
Concedo
718dc159b6
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# CMakeLists.txt
# docs/speculative.md
# ggml/src/ggml-cuda/CMakeLists.txt
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/hmx-matmul-ops.c
# ggml/src/ggml-hexagon/htp/hmx-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# ggml/src/ggml-hexagon/htp/rope-ops.c
# ggml/src/ggml-hexagon/htp/ssm-conv.c
# ggml/src/ggml-opencl/ggml-opencl.cpp
# scripts/snapdragon/adb/run-bench.sh
# scripts/snapdragon/adb/run-cli.sh
# scripts/snapdragon/adb/run-completion.sh
# scripts/snapdragon/adb/run-mtmd.sh
# scripts/snapdragon/windows/run-bench.ps1
# scripts/snapdragon/windows/run-cli.ps1
# scripts/snapdragon/windows/run-completion.ps1
# scripts/snapdragon/windows/run-mtmd.ps1
# src/llama-vocab.cpp
# tests/test-backend-ops.cpp
# tools/batched-bench/CMakeLists.txt
# tools/batched-bench/batched-bench.cpp
# tools/cli/CMakeLists.txt
# tools/cli/README.md
# tools/cli/cli.cpp
# tools/completion/CMakeLists.txt
# tools/completion/README.md
# tools/llama-bench/CMakeLists.txt
# tools/llama-bench/llama-bench.cpp
# tools/mtmd/CMakeLists.txt
# tools/mtmd/tests/test-deepseek-ocr.py
# tools/mtmd/tests/tests-requirements.txt
# tools/perplexity/CMakeLists.txt
# tools/perplexity/perplexity.cpp
# tools/quantize/CMakeLists.txt
# tools/server/CMakeLists.txt
# tools/server/README.md
# ty.toml
2026-05-21 23:47:21 +08:00
Gaurav Garg
ad27757261
Move to backend sampling for MTP draft path ( #23287 )
...
* Move to backend sampling for MTP draft path
Run top_k(10) on the draft backend. D2H transfers happen only for the top 10 logits
Make backend sampling more robust and fallback to CPU on failure cases, such as with "-sm tensor" or when a backend doesn't support TOP_K.
* Allow sampler chains to be partially offloaded to backend
* Add --spec-draft-backend-sampling argument. Enabled by default.
2026-05-20 22:34:45 +05:30
Georgi Gerganov
510b5c2a35
common/speculative : fix nullptr crash in get_devices_str ( #23386 )
...
ggml_backend_dev_by_name always appends a nullptr sentinel to the devices
vector. Skipping nullptr entries prevents assertion failure in
ggml_backend_dev_name.
Assisted-by: llama.cpp:local pi
2026-05-20 19:44:30 +03:00
Concedo
7d987af23a
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/cann.Dockerfile
# .devops/cpu.Dockerfile
# .devops/cuda.Dockerfile
# .devops/intel.Dockerfile
# .devops/llama-cli-cann.Dockerfile
# .devops/musa.Dockerfile
# .devops/openvino.Dockerfile
# .devops/rocm.Dockerfile
# .devops/s390x.Dockerfile
# .devops/vulkan.Dockerfile
# .github/ISSUE_TEMPLATE/011-bug-results.yml
# .github/ISSUE_TEMPLATE/019-bug-misc.yml
# .github/workflows/build-and-test-snapdragon.yml
# .github/workflows/docker.yml
# .github/workflows/server-self-hosted.yml
# .github/workflows/ui-ci.yml
# .pi/gg/SYSTEM.md
# README.md
# common/arg.cpp
# docs/backend/SYCL.md
# docs/backend/snapdragon/CMakeUserPresets.json
# docs/backend/snapdragon/README.md
# docs/speculative.md
# examples/save-load-state/save-load-state.cpp
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/rope-ops.c
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/cvt.cl
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/gated_delta_net.wgsl
# tools/cli/README.md
# tools/server/README.md
2026-05-20 18:48:34 +08:00
Johannes Gäßler
57cb35c886
common: fix --help for --verbosity ( #23278 )
2026-05-19 21:34:04 +02:00
Johannes Gäßler
7256fce047
common: fix --fit verbosity with --verbosity 4 ( #23282 )
2026-05-19 21:33:23 +02:00
Georgi Gerganov
d14ce3dab4
llama : MTP clean-up ( #23269 )
...
* llama : disable equal splits for recurrent memory with partial rollback
* spec : re-enable p-min with MTP drafts
* spec : re-enable ngram spec in combination with RS rollback
* spec : fix ngram-map-* params
* spec : fix acceptance logic in combined ngram + draft configs
* graph : fix reuse for combined `token` + `embd` batches
* spec : log parameters for each speculative implementation
- add LOG_INF in each constructor with implementation type and parameters
- extract device string logic into common_speculative_get_devices_str()
- move 'adding speculative implementation' log from init into constructors
Assisted-by: llama.cpp:local pi
* spec : extend --spec-default with ngram-map-k4v
Assisted-by: llama.cpp:local pi
* minor : fix n_embd log
* args : update draft.n_max == 3 + regen docs
* spec : relax ngram-mod rejection thold to 0.25 @ 5 low
* logs : improve
* docs : update speculative decoding CLI argument documentation
- Add missing draft model CPU scheduling and tensor override parameters
- Update --spec-type to include all available types (excluding draft-eagle3 WIP)
- Fix default values to match implementation (n_max=3, n_min=0, p_min=0.0)
- Remove deprecated options (spec-draft-ctx-size, spec-draft-replace)
- Add environment variables for new parameters
Assisted-by: llama.cpp:local pi
* arg : step-back on adding k4v to the default spec config
* cont : fix name
2026-05-19 15:32:58 +03:00
Georgi Gerganov
cd963fee6a
save-load-state : refactor tests and improve readability ( #23196 )
...
* save-load-state : refactor into separate phase functions
- Split monolithic main() into 4 self-contained phase functions, each
managing its own context/sampler/batch lifecycle
- Each function tokenizes internally using its local ctx instance
- main() is now a clean orchestrator: init -> run phases -> assert results
- Proper resource cleanup on every exit path (return {} on error)
Assisted-by: llama.cpp:local pi
* save-load-state : use params.out_file instead of separate state_file
- Remove state_file parameter from all phase functions
- Each function accesses params.out_file directly
- Initialize params.out_file in main alongside params.prompt
Assisted-by: llama.cpp:local pi
* save-load-state : use smart pointers for ctx and smpl
- Replace raw llama_context* with llama_context_ptr
- Replace raw llama_sampler* with llama_sampler_ptr
- Remove all manual llama_free() and llama_sampler_free() calls
- Keep llama_batch as raw (managed manually with llama_batch_free)
Assisted-by: llama.cpp:local pi
* save-load-state : add local llama_batch_ptr RAII wrapper
- Add llama_batch_ptr struct holding llama_batch by value
- Calls llama_batch_free() in destructor
- Eliminates all manual llama_batch_free() calls
Assisted-by: llama.cpp:local pi
* save-load-state : replace printf/fprintf with logging macros
- Add log.h include
- Replace fprintf(stderr, ...) errors with LOG_ERR
- Replace fprintf(stderr, ...) info with LOG_TRC
- Replace printf output with LOG
Assisted-by: llama.cpp:local pi
* save-load-state : refactor tests to check results inline
Each follow-up phase now accepts an expected result and performs
the comparison internally instead of collecting results in main().
Assisted-by: llama.cpp:local pi
* save-load-state : improve test output readability
Add phase labels, remove redundant run prefixes, and show
PASS after each test.
Assisted-by: llama.cpp:local pi
* pi : add rule about git signing
* save-load-state : simplify llama_batch_ptr
Change get() to return a reference and remove operator*().
Use batch.get() throughout for consistency.
Assisted-by: llama.cpp:local pi
* save-load-state : extract generate_tokens helper
Factor out the repeated token generation loop into a shared
helper function used by all phases.
Assisted-by: llama.cpp:local pi
* save-load-state : update comments to use test terminology
Replace "Phase" with "Test" and list each test's steps
as bullet points.
Assisted-by: llama.cpp:local pi
* save-load-state : rename test functions
Rename to test_baseline, test_state_load, test_seq_cp_host,
test_seq_cp_device. Update comments and logs accordingly.
Assisted-by: llama.cpp:local pi
* pi : add rule to never git push without confirmation
Assisted-by: llama.cpp:local pi
* common : add model_only option to common_init_from_params
Add bool model_only parameter to skip context creation,
sampler init, and context-dependent setup.
Use in save-load-state to initialize only the model,
with each test creating its own context.
Assisted-by: llama.cpp:local pi
---------
Co-authored-by: ggerganov <ggerganov@users.noreply.github.com>
2026-05-19 09:46:34 +03:00
Adrien Gallouët
45b455e66f
common : remove hf cache migration ( #23266 )
...
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-05-18 17:11:47 +02:00
Concedo
fecf2dc3fa
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/server-self-hosted.yml
# CMakeLists.txt
# CODEOWNERS
# ci/run.sh
# cmake/llama-config.cmake.in
# common/chat.cpp
# examples/sycl/start-svr.sh
# examples/sycl/test.sh
# examples/sycl/win-start-svr.bat
# examples/sycl/win-test.bat
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/vecdotq.hpp
# ggml/src/ggml-vulkan/CMakeLists.txt
# scripts/wc2wt.sh
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
2026-05-18 21:27:23 +08:00
Aldehir Rojas
87589042ca
cmake : fix LLAMA_BUILD_UI logic ( #23190 )
2026-05-17 14:42:26 -04:00
Aman Gupta
3e12fbdea5
llama: avoid copying logits during prompt decode in MTP ( #23198 )
...
* llama: avoid copying logits during prompt decode in MTP
* review: update comment
* llama-graph: call set_output for t_h_pre_norm
2026-05-17 23:30:25 +08:00
Aldehir Rojas
39cf5d6191
common : delegate assistant continuation to underlying template handlers ( #23089 )
...
* common : delegate assistant continuation to template handler
* server : implement echo parameter to exclude assistant prefill in the response
* server : fix tests for prefill
* server : use existing llama template
* cont : clean up
2026-05-17 13:36:05 +02:00
Aldehir Rojas
f4cc787b9f
common : enable streaming JSON argument values ( #23173 )
...
* common : remove atomic from json arguments
* common : remove parsing logic on JSON arguments
2026-05-17 03:44:34 -05:00
Rares Vernica
1a68ec9378
server : honor --embd-normalize CLI arg ( #23125 )
...
The --embd-normalize flag was registered only for the embedding and debug
examples, so llama-server rejected it and the /embedding handler used a
hard-coded default of 2 (L2). Add LLAMA_EXAMPLE_SERVER to the flag's
example set and read params.embd_normalize as the handler's default. The
per-request "embd_normalize" body field continues to override.
2026-05-17 09:39:04 +03:00
ddh0
a16cce81d3
ngram : reduce noisy logs ( #23185 )
...
* ngram : reduce noisy logs
* ngram : reduce noisy logs
2026-05-17 09:38:17 +03:00
Concedo
1e828ccabf
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# common/common.cpp
# ggml/CMakeLists.txt
# scripts/sync-ggml.last
# scripts/sync_vendor.py
# src/llama-context.cpp
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/server/README.md
2026-05-17 11:26:18 +08:00
Pascal
64b38b561b
server: skip device enumeration in router mode to avoid creating CUDA primary context ( #23137 )
2026-05-16 21:21:06 +02:00
Concedo
9203b6a051
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/labeler.yml
# .github/workflows/build-self-hosted.yml
# .github/workflows/release.yml
# .github/workflows/server-sanitize.yml
# .github/workflows/server-self-hosted.yml
# .github/workflows/server.yml
# .github/workflows/ui-build.yml
# .github/workflows/ui-ci.yml
# .github/workflows/ui-publish.yml
# .gitignore
# CMakeLists.txt
# CODEOWNERS
# scripts/ui-download.cmake
# scripts/xxd.cmake
# tests/test-backend-ops.cpp
# tests/test-reasoning-budget.cpp
# tools/CMakeLists.txt
# tools/server/CMakeLists.txt
# tools/server/README.md
2026-05-16 22:56:33 +08:00
Aman Gupta
255582687b
llama + spec: MTP Support ( #22673 )
...
* spec: support MTP
* fix batch size
* rename files
* cont : simplify (#7 )
* MTP: clean-up (#9 )
* MTP: clean-up
* review: use llama_context_type instead of llama_graph_type
* review: remove llama_model_has_mtp
* review: fix convert issues
* convert: fix pycheck
* review: formatting
* use `mtp-` for identifying mtp models
* convert: fix mtp conversion
* mtp -> draft-mtp
* remove unused llama_arch
* add need_embd in speculative
* llama: allow partial seq_rm for GDN models for speculative decoding
Currently speculative checkpoint needs to restart from a checkpoint
after some draft tokens are not accepted, this leads to some wastage in
running the target again. This PR adds the ability to rollback upto
`draft_max` by storing the GDN intermediates.
* fix pending state
* vulkan: add GDN partial rollback
* meta: extend check to axis 1
* metal: add GDN partial rollback
Extend the gated delta net kernel to store intermediate states for
partial rollback support on the Metal backend.
- Add K (snapshot slot count) as a function constant
- Read input state from slot 0 of the 3D state tensor
- Write intermediate states to different slots during token loop
- For K=1, maintain backward-compatible single-slot behavior
Ref: 8c05923630#11 )
* server : adjust checkpoint logic
* cont : rm asserts
* server-context: fix early exit
* spec : fix compatibility with n-gram and add TODOs (#13 )
* metal : cleanup
* llama : fix faulty bitwise check in recurrent memory
* server : disable RS-based MTP in combination with other spec types
* spec : add TODOs
* cont : fix comment
* cont : update comment
* common : fix logic for ngram + mtp compat
* llama-memory: enable checkpointing with partial rollback
* cont: add test-case for loading into a dirty ctx
* llama-memory-recurrent: clear rs_idx in clear
* download: fix mtp path
* llama-arch: fix enorm op
* docs: update docs
* conversion: fix type annotations
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-05-16 20:06:23 +08:00
Aleksander Grygier
59778f0196
ui: Restructure repo to use tools/ui folder and ui / UI / llama-ui / LLAMA_UI naming ( #23064 )
...
* webui: Move static build output from `tools/server/public` to `build/ui` directory
* refactor: Move to `tools/ui`
* refactor: rename CMake variables and preprocessor defines
- Rename LLAMA_BUILD_WEBUI -> LLAMA_BUILD_UI (old kept as deprecated)
- Rename LLAMA_USE_PREBUILT_WEBUI -> LLAMA_USE_PREBUILT_UI (old kept as deprecated)
- Backward compat: old vars auto-forward to new ones with DEPRECATION warning
- Rename internal vars: WEBUI_SOURCE -> UI_SOURCE, WEBUI_SOURCE_DIR -> UI_SOURCE_DIR, etc.
- Rename HF bucket: LLAMA_WEBUI_HF_BUCKET -> LLAMA_UI_HF_BUCKET
- Emit both LLAMA_BUILD_WEBUI and LLAMA_BUILD_UI preprocessor defines
- Emit both LLAMA_WEBUI_DEFAULT_ENABLED and LLAMA_UI_DEFAULT_ENABLED
* refactor: rename CLI flags (--webui -> --ui) with backward compat
- Add --ui/--no-ui (old --webui/--no-webui kept as deprecated aliases)
- Add --ui-config (old --webui-config kept as deprecated alias)
- Add --ui-config-file (old --webui-config-file kept as deprecated alias)
- Add --ui-mcp-proxy/--no-ui-mcp-proxy (old --webui-mcp-proxy kept as deprecated)
- Add new env vars: LLAMA_ARG_UI, LLAMA_ARG_UI_CONFIG, LLAMA_ARG_UI_CONFIG_FILE, LLAMA_ARG_UI_MCP_PROXY
- C++ struct fields: params.ui, params.ui_config_json, params.ui_mcp_proxy added alongside old fields
- Backward compat: old fields synced to new ones in g_params_to_internals
* refactor: update C++ server internals with backward compat
- Rename json_webui_settings -> json_ui_settings (both kept in server_context_meta)
- Rename params.webui usage -> params.ui (both synced, old still works)
- JSON API emits both "ui"/"ui_settings" and "webui"/"webui_settings" keys
- Server routes use params.ui_mcp_proxy || params.webui_mcp_proxy
- Preprocessor guards use #if defined(LLAMA_BUILD_UI) || defined(LLAMA_BUILD_WEBUI)
* refactor: rename CI/CD workflows, artifacts, and build script
- Rename webui-build.yml -> ui-build.yml; artifact webui-build -> ui-build
- Rename webui-publish.yml -> ui-publish.yml; var HF_BUCKET_WEBUI_STATIC_OUTPUT -> HF_BUCKET_UI_STATIC_OUTPUT
- Rename server-webui.yml -> server-ui.yml; job webui-build/checks -> ui-build/checks
- Update server.yml: job/artifact refs webui-build -> ui-build
- Update release.yml: all webui-build/publish refs -> ui-build/publish; HF_TOKEN_WEBUI_STATIC_OUTPUT -> HF_TOKEN_UI_STATIC_OUTPUT
- Update server-self-hosted.yml: webui-build -> ui-build
- Update build-self-hosted.yml: HF_WEBUI_VERSION -> HF_UI_VERSION
- Rename webui-download.cmake -> ui-download.cmake (internal refs updated)
- Update labeler.yml: server/webui -> server/ui path label
* docs: update CODEOWNERS and server README docs
- Update CODEOWNERS: team ggml-org/llama-webui -> ggml-org/llama-ui, path /tools/server/webui/ -> /tools/ui/
- Update server README.md: CLI tables show --ui flags with deprecated --webui aliases
- Update server README-dev.md: "WebUI" -> "UI", paths updated to tools/ui/
* fix: Small fixes for UI build
* fix: CMake.txt syntax
* chore: Formatting
* fix: `.editorconfig` for llama-ui
* chore: Formatting
* refactor: Use `APP_NAME` in Error route
* refactor: Cleanup
* refactor: Single migration service
* make llama-ui a linkable target
* fix: UI Build output
* fix: Missing change
* fix: separate llama-ui npm build output into build/tools/ui/dist subfolder + use cmake npm build instead of downloading ui-build.yml artifacts in CI
* refactor: UI workflows cleanup
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-05-16 02:02:40 +02:00
Aman Gupta
ac33f032ac
reasoning-budget: clone should do a deep-copy ( #23095 )
2026-05-15 11:59:07 +02:00
Concedo
da2cc90723
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/labeler.yml
# .github/workflows/build-and-test-snapdragon.yml
# .github/workflows/build-self-hosted.yml
# .github/workflows/release.yml
# .github/workflows/server-self-hosted.yml
# .github/workflows/server-webui.yml
# .github/workflows/server.yml
# .gitignore
# CMakeLists.txt
# CONTRIBUTING.md
# README.md
# ggml/src/ggml-cuda/fattn.cu
# ggml/src/ggml-hexagon/htp/cpy-ops.c
# ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# grammars/README.md
# scripts/snapdragon/qdc/run_qdc_jobs.py
# scripts/snapdragon/qdc/tests/run_backend_ops_posix.py
# scripts/snapdragon/qdc/tests/run_bench_tests_posix.py
# scripts/snapdragon/qdc/tests/utils.py
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
# tools/server/CMakeLists.txt
# tools/server/README.md
# tools/server/webui/src/lib/components/app/server/ServerLoadingSplash.svelte
# tools/server/webui/src/routes/(chat)/chat/[id]/+page.svelte
# ty.toml
2026-05-15 17:09:48 +08:00
Aleksander Grygier
253ba110bc
webui: Move static build output from repo code to HF Bucket ( #22937 )
...
* ci: add workflow to publish webui to Hugging Face bucket
* ci: add webui release job to release workflow
* ci: test webui release job
* chore: Return to default minification strategy for build output files
* ci: extract webui build into separate workflow and job
* chore: Ignore webui static output + clean up references
* chore: Delete legacy webui static output
* chore: Ignore webui build static output
* fix: Workflow
* fix: Versioning naming
* chore: Update package name
* test: Test CI fix
* refactor: Naming
* server: implement webui build strategy with HF Bucket support
* chore: Remove test workflow
* chore: Use WebUI build workflow call in other workflows
* server: HF Buckets fallback for WebUI build
* refactor: App name variable
* refactor: Naming
* fix: Retrieve loading.html
* fix: workflow syntax
* fix: Rewrite malformed release.yml
* fix: Req param
* test: Re-add missing Playwright installation for CI tests
* refactor: Logic & security improvements
* refactor: Retrieve publishing jobs and DRY the workflows
* fix: Test workflow syntax
* fix: Upstream Release Tag for test workflow
* chore: Remove test workflow
* ci: Run WebUI jobs on `ubuntu-24.04-arm`
* refactor: Post-CR cleanup
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
* refactor: CI cleanup
* refactor: Cleanup
* test: Test workflow
* refactor: use LLAMA_BUILD_NUMBER instead of LLAMA_BUILD_TAG for HF Bucket webui downloads
* server: add fallback mechanism for HF Bucket webui downloads from latest directory
* fix: Incorrect argument order in file(SHA256) calls for checksum verification
* refactor: Use cmake script for handling the HF Bucket download on build time
* feat: support local npm build for WebUI assets
* refactor: add `HF_ENABLED` flag to control WebUI build/download provisioning
* refactor: Cleanup
* chore: Remove test workflow
* fix: remove s390x from release workflow
* fix: add webui-build dependency to ubuntu-22-rocm and windows-hip
* Revert "fix: remove s390x from release workflow"
This reverts commit debcfffa9bc1e3112eae41f2d29741b682e4eb19.
* fix: Release workflow file
* fix: Proper release tag used for HF Bucket upload
* fix: Remove duplicate steps in release workflow
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
2026-05-14 13:21:41 +02:00
Concedo
cc82c3164e
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/intel.Dockerfile
# .github/workflows/build-cross.yml
# .github/workflows/build-sycl.yml
# .github/workflows/build.yml
# .github/workflows/editorconfig.yml
# .github/workflows/release.yml
# cmake/riscv64-spacemit-linux-gnu-gcc.cmake
# docs/backend/OPENVINO.md
# docs/backend/SYCL.md
# docs/build-riscv64-spacemit.md
# docs/ops.md
# docs/ops/WebGPU.csv
# embd_res/ggml-vocab-qwen35.gguf
# embd_res/ggml-vocab-qwen35.gguf.inp
# embd_res/ggml-vocab-qwen35.gguf.out
# examples/model-conversion/Makefile
# ggml/CMakeLists.txt
# ggml/src/ggml-cpu/CMakeLists.txt
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/hmx-flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/hmx-matmul-ops.c
# ggml/src/ggml-hexagon/htp/hmx-utils.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/hvx-utils.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/cvt.cl
# ggml/src/ggml-sycl/CMakeLists.txt
# ggml/src/ggml-sycl/common.cpp
# ggml/src/ggml-sycl/common.hpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/common_decls.tmpl
# ggml/src/ggml-webgpu/wgsl-shaders/flash_attn_tile.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/flash_attn_vec_reduce.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/flash_attn_vec_split.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/get_rows.wgsl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_decls.tmpl
# ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec_acc.tmpl
# ggml/src/ggml-webgpu/wgsl-shaders/unary.wgsl
# ggml/src/ggml-zendnn/CMakeLists.txt
# ggml/src/ggml-zendnn/ggml-zendnn.cpp
# scripts/snapdragon/adb/run-completion.sh
# tests/CMakeLists.txt
# tools/cli/README.md
# tools/completion/README.md
# tools/mtmd/clip-impl.h
# tools/mtmd/clip.cpp
# tools/mtmd/clip.h
# tools/server/README.md
2026-05-14 19:04:04 +08:00
Georgi Gerganov
67b2b7f2f2
logs : reduce ( #23021 )
...
Python Type-Check / python type-check (push) Waiting to run
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* logs : reduce
* args : fix envs
* server : fix build
* common : print verbosity level at start
* server : clean-up logs
* server : print prompt processing timings + sampling params
* minor : whitespaces
2026-05-14 13:05:52 +03:00
Xuan-Son Nguyen
e75cd5efb5
download: do not exit() on error ( #23008 )
2026-05-13 15:14:58 +02:00
Georgi Gerganov
634275fbbb
spec : update CLI arguments for better consistency ( #22964 )
...
* spec : update CLI arguments for better consistency
* cont : fix CLI arg message
2026-05-13 09:15:39 +03:00
Xuan-Son Nguyen
7bfe120c21
mtmd, server, common: expose modalities to /v1/models ( #22952 )
...
* mtmd, server, common: expose modalities to /v1/models
* fix build
* rename to mtmd_caps
2026-05-12 19:08:07 +02:00
Concedo
f7923b261f
need to fix cuda compile. Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/python-type-check.yml
# examples/speculative-simple/README.md
# examples/speculative-simple/speculative-simple.cpp
# ggml/src/ggml-cuda/im2col.cu
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/cvt.cl
# tests/test-backend-ops.cpp
# tools/cli/README.md
# tools/mtmd/CMakeLists.txt
# tools/server/README.md
2026-05-12 20:47:07 +08:00
Georgi Gerganov
68e7ea3eab
spec : parallel drafting support ( #22838 )
...
* spec : refactor
* spec : drop support for incompatible vocabs
* spec : update common_speculative_init()
* cont : pass seq_id
* cont : dedup ctx_seq_rm_type
* server : sketch the ctx_dft decode loop
* server : draft prompt cache and checkpoints
* server : improve ctx names
* server, spec : transition to unified spec context
* cont : sync main and drft contexts
* cont : async drft eval when possible
* cont : handle non-ckpt models
* cont : pass correct n_past for drafting
* cont : process images throught the draft context
* spec : handle draft running out of context
* server : fix mtmd draft processing
* server : fix URL for draft model
* server : add comment
* server : clean-up + dry
* speculative-simple : update
* spec : fix n_past type
* server : fix slot ctx_drft ptr
* tools : update readme
* naming : improve consistency
* spec : refactor for multi-sequence speculative context
* cont : prepare params
* cont : prepare params
* spec : support parallel drafts
* server : support parallel drafting
* llama : reuse device buffers when possible
* server, spec : clean-up
* cont : clean-up
* cont : minor
* spec : reset `drafting` flag at the end
* spec : introduce `common_speculative_process()`
* spec : allow for multiple spec types (chain of speculators)
* replace old type field of type common_speculative_type in the
common_params_speculative struct with a vector to allow multiple
types to be specified
* introduce common_get_enabled_speculative_impls(const std::vector<enum common_speculative_type>)
to figure out which implementations the user has enabled
* introduce common_speculative_type_from_names(const std::vector<std::string> & names)
to parse the already user provided spec types
* all speculators run sequentially, best one wins (we verify its drafted tokens)
* maximize expected accepted tokens for current round by calculating the
product between the probability of accepting current token (n_acc_tokens / n_gen_drafts)
and the draft's length
---------
Co-authored-by: Petros Sideris <petros.sideris@nokia.com>
2026-05-11 19:09:43 +03:00
Concedo
2771e16fbc
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/intel.Dockerfile
# .devops/nix/package.nix
# .gitignore
# docs/backend/SYCL.md
# docs/ops.md
# docs/ops/SYCL.csv
# ggml/CMakeLists.txt
# ggml/src/ggml-cuda/fattn.cu
# ggml/src/ggml-cuda/ggml-cuda.cu
# ggml/src/ggml-sycl/CMakeLists.txt
# ggml/src/ggml-sycl/common.hpp
# ggml/src/ggml-sycl/convert.cpp
# ggml/src/ggml-sycl/dequantize.hpp
# ggml/src/ggml-sycl/fattn-common.hpp
# ggml/src/ggml-sycl/getrows.cpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/im2col.cpp
# ggml/src/ggml-sycl/im2col.hpp
# ggml/src/ggml-sycl/mmvq.cpp
# ggml/src/ggml-sycl/quants.hpp
# ggml/src/ggml-sycl/vecdotq.hpp
# ggml/src/ggml-virtgpu/ggml-backend-device.cpp
# scripts/sync-ggml.last
# scripts/sync_vendor.py
# tests/test-backend-ops.cpp
2026-05-11 16:18:28 +08:00
Concedo
9b0b36b5ef
Merge commit ' 66001722aa' into concedo_experimental
...
# Conflicts:
# README.md
# docs/ops.md
# docs/ops/SYCL.csv
# examples/sycl/start-svr.sh
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/cvt.cl
# ggml/src/ggml-sycl/gated_delta_net.hpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/pad.cpp
# ggml/src/ggml-sycl/ssm_conv.cpp
# tests/test-backend-ops.cpp
# tests/test-reasoning-budget.cpp
# tools/server/README.md
# tools/server/webui/src/lib/constants/settings-config.ts
2026-05-11 15:40:10 +08:00
Tim Neumann
2e97c5f96f
backend sampling: support returning post-sampling probs ( #22622 )
...
* server: Never return 0.0 post-sampling probabilities
* backend sampling: support returning post-sampling probs
2026-05-10 19:12:02 +02:00
