Concedo
e944cca86f
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# scripts/sync_vendor.py
# src/llama-model-loader.cpp
# tests/test-backend-ops.cpp
# tests/test-chat-auto-parser.cpp
# tests/test-chat.cpp
# tools/cli/cli.cpp
# tools/server/README.md
2026-07-05 11:30:12 +08:00
Ruixiang Wang
152d337fad
spec: support spec-draft-p-min in DFlash ( #25246 )
...
* spec: support spec-draft-p-min in DFlash
* dflash: add n_min guard
* dflash: guard both n_min and n_max
2026-07-03 15:40:06 +02:00
Concedo
3b867bd4b1
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/release.yml
# SECURITY.md
# common/CMakeLists.txt
# docs/speculative.md
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/cvt.cl
# ggml/src/ggml-opencl/kernels/flash_attn_f16.cl
# ggml/src/ggml-opencl/kernels/flash_attn_f32.cl
# ggml/src/ggml-opencl/kernels/flash_attn_f32_f16.cl
# ggml/src/ggml-opencl/kernels/set_rows.cl
# ggml/src/ggml-openvino/ggml-openvino.cpp
# ggml/src/ggml-sycl/norm.cpp
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tests/test-chat-template.cpp
# tests/test-chat.cpp
# tests/test-export-graph-ops.cpp
# tests/test-jinja.cpp
# tests/test-llama-archs.cpp
# tools/rpc/CMakeLists.txt
# tools/rpc/README.md
2026-06-29 16:43:44 +08:00
Ruixiang Wang
d1b34251bc
spec : add DFlash support ( #22105 )
...
* spec: add DFlash v2 support
* dflash: support sliding window attention per layer_types
* docs: add dflash section
---------
Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>
2026-06-28 16:01:34 +03:00
Georgi Gerganov
27c8bb4f63
logs : reduce v2 ( #25078 )
...
* server : reduce logs
* cont : common
* cont : spec
* cont : CMN_ -> COM_
2026-06-28 08:52:15 +03:00
Concedo
3090ae0bf7
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/s390x.Dockerfile
# .dockerignore
# .github/workflows/docker.yml
# .github/workflows/release.yml
# docs/android.md
# ggml/src/ggml-cpu/amx/mmq.cpp
# ggml/src/ggml-hexagon/htp/ssm-conv.c
# tests/peg-parser/test-gbnf-generation.cpp
# tests/test-arg-parser.cpp
# tests/test-chat.cpp
# tests/test-jinja.cpp
# tests/test-json-schema-to-grammar.cpp
# tools/server/README.md
2026-06-22 18:23:59 +08:00
Georgi Gerganov
8a118ee86c
minor : clean-up whitespaces ( #24862 )
...
[no ci]
2026-06-21 11:37:12 +03:00
YiChen Lv
d789527482
spec : Support Step3.5/3.7 flash mtp3 ( #24340 )
...
* add mtp_layer_offset + include nextn flags in graph reuse
* add llama_set_mtp_layer_offset + llama_model_n_nextn_layer API
* offset head select + require all MTP blocks
* speculative multi-head process()
* speculative multi-head draft()
* gather outputs via inp_out_ids
* cleanup
* fix core
* minor cleanup
* merged draft_multi_head into draft()
* mtp rename nextn
* Apply suggestions from code review
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
* clean-up comments
* fix for multi seq
* apply suggestions && chain-heads comment
* add a reference for chain_heads discussion
---------
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
2026-06-21 11:33:18 +03:00
Concedo
73cc7d9287
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/cann.Dockerfile
# .devops/cpu.Dockerfile
# .devops/cuda.Dockerfile
# .devops/intel.Dockerfile
# .devops/musa.Dockerfile
# .devops/openvino.Dockerfile
# .devops/rocm.Dockerfile
# .devops/s390x.Dockerfile
# .devops/vulkan.Dockerfile
# .devops/zendnn.Dockerfile
# .dockerignore
# .pi/gg/SYSTEM.md
# ggml/CMakeLists.txt
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# scripts/sync-ggml.last
# scripts/sync_vendor.py
# tools/cli/README.md
# tools/cli/cli.cpp
# tools/mtmd/clip.cpp
# tools/server/README.md
2026-06-20 17:16:19 +08:00
Ruixiang Wang
b14e3fb90c
spec: support eagle3 for qwen3.5 & 3.6 ( #24593 )
...
* spec: support qwen3.5 & 3.6 eagle3 draft
* eagle3: Add deferred boundary checkpoints restore support for hybrid models
* apply suggestions
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* spec: adapt to API change
* spec: fix naming
* cont : add TODO
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-06-19 13:08:50 +03:00
Concedo
382ce55fb7
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/intel.Dockerfile
# README.md
# docs/backend/SYCL.md
# docs/install.md
# docs/ops.md
# docs/ops/SYCL.csv
# ggml/src/ggml-sycl/binbcast.cpp
# ggml/src/ggml-sycl/concat.cpp
# ggml/src/ggml-sycl/dmmv.cpp
# ggml/src/ggml-sycl/element_wise.cpp
# ggml/src/ggml-sycl/element_wise.hpp
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/mmvq.cpp
# ggml/src/ggml-sycl/mmvq.hpp
# tests/peg-parser/test-gbnf-generation.cpp
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
# tools/llama-bench/llama-bench.cpp
2026-06-16 17:55:04 +08:00
Ruixiang Wang
a1824902b5
spec: add backend sampling support for eagle3 ( #24655 )
Update Operations Documentation / update-ops-docs (push) Waiting to run
Python Type-Check / python type-check (push) Has been cancelled
2026-06-16 12:05:52 +03:00
Ruixiang Wang
635b65ad7a
spec: add spec metrics mean acceptance length and acceptance rate per position ( #24536 )
...
* spec: add spec metrics mean acceptance length and acceptance per pos
* fix as suggestion
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* fix as suggestion
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* fix as suggestion
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* fix as suggestions
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-06-16 10:23:09 +03:00
Concedo
be1750a3b8
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/cuda.Dockerfile
# .github/workflows/build-sycl.yml
# .github/workflows/release.yml
# ggml/CMakeLists.txt
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/gated-delta-net-ops.c
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/cvt.cl
# ggml/src/ggml-opencl/kernels/gated_delta_net.cl
# ggml/src/ggml-sycl/gated_delta_net.cpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/gated_delta_net.wgsl
# scripts/sync-ggml.last
# tests/test-backend-ops.cpp
# tests/test-llama-archs.cpp
2026-06-12 15:39:57 +08:00
Ruixiang Wang
88a39274ec
spec: add EAGLE3 speculative decoding support ( #18039 )
...
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
* llama : enable layer input extraction
* spec: support eagle3
* eagle3: fix params bug
* eagle3: support Gemma4 eagle3 from RedHatAI
* eagle3: set sync when get features from target
Co-authored-by: tnhnyzc <115956684+tnhnyzc@users.noreply.github.com>
* eagle3 : fix ubatch handling in embd_layer_inp extraction and encoder
Co-authored-by: Doğaç Eldenk <dogacel@gmail.com>
* eagle3: adapt to upstream changes
* eagle3: fix rebase issues and adapt to upstream changes
* eagle3:exclude the eagle3 arch from test-llama-archs
* eagle3: fix editorconfig check failures
* eagle3: fix multi-seq issue in d2t vocab mapping
* cont : minor style / clean-up
* spec : remove `common_speculative_setup_draft_model()`
* llama : clean-up unused API
* eagle3: set d2t vocab mapping in decode graph
* cont : assert layer inputs are configured
* hparams : use n_embd_inp instead of n_embd_target_features
* eagle3: make output.weight optional and inherit from target model when needed
* haparams : generic norm-before-residual param
* llama-ext : consistent names
* cont : fix
* hparams : remove target_hidden_size
* cparams : rename output_layer_inp -> embeddings_layer_inp
* arch : reuse ATTN_NORM_2 instead of adding new hidden norm
* llama : clean-up names
* cont : add assert + comment
* Update conversion/llama.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: tnhnyzc <115956684+tnhnyzc@users.noreply.github.com>
Co-authored-by: Doğaç Eldenk <dogacel@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
2026-06-12 10:21:06 +03:00
Concedo
a860ec0b37
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/winget.yml
2026-06-10 23:16:06 +08:00
ddh0
d2e22ed975
speculative : fix "ngram-map-k4v" name in logging ( #24253 )
...
This is a non-functional change.
When using `--spec-type ngram-map-k4v`, the log messages at startup and
runtime say `ngram-map-k`. Added logic in the in the constructor of
`common_speculative_impl_ngram_map_k` to pass the correct
`COMMON_SPECULATIVE_TYPE_NGRAM_MAP_K4V` when `config.key_only` is
`false`.
After this change, the log messages use the correct name.
2026-06-10 09:31:35 +02:00
Concedo
dd2f966f4b
Merge commit ' c2b1518fd4' into concedo_experimental
...
# Conflicts:
# .devops/intel.Dockerfile
# ggml/CMakeLists.txt
# scripts/sync-ggml.last
# tests/test-backend-ops.cpp
# tests/test-llama-archs.cpp
# tools/cli/cli.cpp
2026-06-10 00:56:44 +08:00
Aman Gupta
04eb4c446d
llama : add Gemma4 MTP ( #23398 )
2026-06-07 20:50:54 +08:00
Sigbjørn Skjæret
8a091c47ab
spec : fix vocab compatibility check ( #24256 )
2026-06-07 14:43:52 +03:00
Concedo
7fb55b1e32
does not work for e4b
2026-06-04 10:44:40 +08:00
Aman Gupta
166fe29492
qwen35: use post-norm hidden state for MTP ( #24025 )
...
* qwen35: use post-norm hidden state for MTP
* rename pre_norm to nextn
* fix step35
2026-06-04 01:29:09 +08:00
Concedo
13eaa04269
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/nix/package.nix
# .github/workflows/server.yml
# docs/development/HOWTO-add-model.md
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/argsort-ops.c
# ggml/src/ggml-hexagon/htp/concat-ops.c
# ggml/src/ggml-hexagon/htp/flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/gated-delta-net-ops.c
# ggml/src/ggml-hexagon/htp/hmx-flash-attn-ops.c
# ggml/src/ggml-hexagon/htp/hmx-matmul-ops.c
# ggml/src/ggml-hexagon/htp/hmx-ops.h
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/hvx-utils.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# ggml/src/ggml-hexagon/htp/pad-ops.c
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/cvt.cl
# ggml/src/ggml-webgpu/wgsl-shaders/cpy.wgsl
# scripts/sync_vendor.py
# src/llama-context.cpp
# tests/test-backend-sampler.cpp
# tools/server/README.md
# tools/ui/tests/stories/ChatScreenForm.a11y.stories.svelte
2026-06-02 19:28:39 +08:00
Georgi Gerganov
5dcb711666
speculative : fix n_outputs_max and remove draft-simple auto-enable ( #23988 )
...
* speculative : add common_speculative_n_max helper function
Extract the speculative max-draft-size logic from server_n_outputs_max
into a reusable common_speculative_n_max() function in common/speculative.
Assisted-by: llama.cpp:local pi
* cont : draft context always has n_parallel outputs
* llama : log n_outputs_max
* speculative : remove draft-simple auto-enable
* ci : enable server tests on PRs
2026-06-01 22:26:58 +03:00
Concedo
718dc159b6
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# CMakeLists.txt
# docs/speculative.md
# ggml/src/ggml-cuda/CMakeLists.txt
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/hmx-matmul-ops.c
# ggml/src/ggml-hexagon/htp/hmx-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/matmul-ops.c
# ggml/src/ggml-hexagon/htp/rope-ops.c
# ggml/src/ggml-hexagon/htp/ssm-conv.c
# ggml/src/ggml-opencl/ggml-opencl.cpp
# scripts/snapdragon/adb/run-bench.sh
# scripts/snapdragon/adb/run-cli.sh
# scripts/snapdragon/adb/run-completion.sh
# scripts/snapdragon/adb/run-mtmd.sh
# scripts/snapdragon/windows/run-bench.ps1
# scripts/snapdragon/windows/run-cli.ps1
# scripts/snapdragon/windows/run-completion.ps1
# scripts/snapdragon/windows/run-mtmd.ps1
# src/llama-vocab.cpp
# tests/test-backend-ops.cpp
# tools/batched-bench/CMakeLists.txt
# tools/batched-bench/batched-bench.cpp
# tools/cli/CMakeLists.txt
# tools/cli/README.md
# tools/cli/cli.cpp
# tools/completion/CMakeLists.txt
# tools/completion/README.md
# tools/llama-bench/CMakeLists.txt
# tools/llama-bench/llama-bench.cpp
# tools/mtmd/CMakeLists.txt
# tools/mtmd/tests/test-deepseek-ocr.py
# tools/mtmd/tests/tests-requirements.txt
# tools/perplexity/CMakeLists.txt
# tools/perplexity/perplexity.cpp
# tools/quantize/CMakeLists.txt
# tools/server/CMakeLists.txt
# tools/server/README.md
# ty.toml
2026-05-21 23:47:21 +08:00
Gaurav Garg
ad27757261
Move to backend sampling for MTP draft path ( #23287 )
...
* Move to backend sampling for MTP draft path
Run top_k(10) on the draft backend. D2H transfers happen only for the top 10 logits
Make backend sampling more robust and fallback to CPU on failure cases, such as with "-sm tensor" or when a backend doesn't support TOP_K.
* Allow sampler chains to be partially offloaded to backend
* Add --spec-draft-backend-sampling argument. Enabled by default.
2026-05-20 22:34:45 +05:30
Georgi Gerganov
510b5c2a35
common/speculative : fix nullptr crash in get_devices_str ( #23386 )
...
ggml_backend_dev_by_name always appends a nullptr sentinel to the devices
vector. Skipping nullptr entries prevents assertion failure in
ggml_backend_dev_name.
Assisted-by: llama.cpp:local pi
2026-05-20 19:44:30 +03:00
Concedo
7d987af23a
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .devops/cann.Dockerfile
# .devops/cpu.Dockerfile
# .devops/cuda.Dockerfile
# .devops/intel.Dockerfile
# .devops/llama-cli-cann.Dockerfile
# .devops/musa.Dockerfile
# .devops/openvino.Dockerfile
# .devops/rocm.Dockerfile
# .devops/s390x.Dockerfile
# .devops/vulkan.Dockerfile
# .github/ISSUE_TEMPLATE/011-bug-results.yml
# .github/ISSUE_TEMPLATE/019-bug-misc.yml
# .github/workflows/build-and-test-snapdragon.yml
# .github/workflows/docker.yml
# .github/workflows/server-self-hosted.yml
# .github/workflows/ui-ci.yml
# .pi/gg/SYSTEM.md
# README.md
# common/arg.cpp
# docs/backend/SYCL.md
# docs/backend/snapdragon/CMakeUserPresets.json
# docs/backend/snapdragon/README.md
# docs/speculative.md
# examples/save-load-state/save-load-state.cpp
# ggml/src/ggml-hexagon/ggml-hexagon.cpp
# ggml/src/ggml-hexagon/htp/CMakeLists.txt
# ggml/src/ggml-hexagon/htp/htp-ctx.h
# ggml/src/ggml-hexagon/htp/htp-ops.h
# ggml/src/ggml-hexagon/htp/main.c
# ggml/src/ggml-hexagon/htp/rope-ops.c
# ggml/src/ggml-hexagon/htp/unary-ops.c
# ggml/src/ggml-opencl/CMakeLists.txt
# ggml/src/ggml-opencl/ggml-opencl.cpp
# ggml/src/ggml-opencl/kernels/cvt.cl
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-webgpu/ggml-webgpu.cpp
# ggml/src/ggml-webgpu/wgsl-shaders/gated_delta_net.wgsl
# tools/cli/README.md
# tools/server/README.md
2026-05-20 18:48:34 +08:00
Georgi Gerganov
d14ce3dab4
llama : MTP clean-up ( #23269 )
...
* llama : disable equal splits for recurrent memory with partial rollback
* spec : re-enable p-min with MTP drafts
* spec : re-enable ngram spec in combination with RS rollback
* spec : fix ngram-map-* params
* spec : fix acceptance logic in combined ngram + draft configs
* graph : fix reuse for combined `token` + `embd` batches
* spec : log parameters for each speculative implementation
- add LOG_INF in each constructor with implementation type and parameters
- extract device string logic into common_speculative_get_devices_str()
- move 'adding speculative implementation' log from init into constructors
Assisted-by: llama.cpp:local pi
* spec : extend --spec-default with ngram-map-k4v
Assisted-by: llama.cpp:local pi
* minor : fix n_embd log
* args : update draft.n_max == 3 + regen docs
* spec : relax ngram-mod rejection thold to 0.25 @ 5 low
* logs : improve
* docs : update speculative decoding CLI argument documentation
- Add missing draft model CPU scheduling and tensor override parameters
- Update --spec-type to include all available types (excluding draft-eagle3 WIP)
- Fix default values to match implementation (n_max=3, n_min=0, p_min=0.0)
- Remove deprecated options (spec-draft-ctx-size, spec-draft-replace)
- Add environment variables for new parameters
Assisted-by: llama.cpp:local pi
* arg : step-back on adding k4v to the default spec config
* cont : fix name
2026-05-19 15:32:58 +03:00
Concedo
fecf2dc3fa
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/server-self-hosted.yml
# CMakeLists.txt
# CODEOWNERS
# ci/run.sh
# cmake/llama-config.cmake.in
# common/chat.cpp
# examples/sycl/start-svr.sh
# examples/sycl/test.sh
# examples/sycl/win-start-svr.bat
# examples/sycl/win-test.bat
# ggml/src/ggml-sycl/ggml-sycl.cpp
# ggml/src/ggml-sycl/vecdotq.hpp
# ggml/src/ggml-vulkan/CMakeLists.txt
# scripts/wc2wt.sh
# tests/test-backend-ops.cpp
# tests/test-chat.cpp
2026-05-18 21:27:23 +08:00
Aman Gupta
3e12fbdea5
llama: avoid copying logits during prompt decode in MTP ( #23198 )
...
* llama: avoid copying logits during prompt decode in MTP
* review: update comment
* llama-graph: call set_output for t_h_pre_norm
2026-05-17 23:30:25 +08:00
Concedo
1e828ccabf
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# common/common.cpp
# ggml/CMakeLists.txt
# scripts/sync-ggml.last
# scripts/sync_vendor.py
# src/llama-context.cpp
# tests/CMakeLists.txt
# tests/test-backend-ops.cpp
# tools/cli/README.md
# tools/completion/README.md
# tools/server/README.md
2026-05-17 11:26:18 +08:00
Aman Gupta
255582687b
llama + spec: MTP Support ( #22673 )
...
* spec: support MTP
* fix batch size
* rename files
* cont : simplify (#7 )
* MTP: clean-up (#9 )
* MTP: clean-up
* review: use llama_context_type instead of llama_graph_type
* review: remove llama_model_has_mtp
* review: fix convert issues
* convert: fix pycheck
* review: formatting
* use `mtp-` for identifying mtp models
* convert: fix mtp conversion
* mtp -> draft-mtp
* remove unused llama_arch
* add need_embd in speculative
* llama: allow partial seq_rm for GDN models for speculative decoding
Currently speculative checkpoint needs to restart from a checkpoint
after some draft tokens are not accepted, this leads to some wastage in
running the target again. This PR adds the ability to rollback upto
`draft_max` by storing the GDN intermediates.
* fix pending state
* vulkan: add GDN partial rollback
* meta: extend check to axis 1
* metal: add GDN partial rollback
Extend the gated delta net kernel to store intermediate states for
partial rollback support on the Metal backend.
- Add K (snapshot slot count) as a function constant
- Read input state from slot 0 of the 3D state tensor
- Write intermediate states to different slots during token loop
- For K=1, maintain backward-compatible single-slot behavior
Ref: 8c05923630#11 )
* server : adjust checkpoint logic
* cont : rm asserts
* server-context: fix early exit
* spec : fix compatibility with n-gram and add TODOs (#13 )
* metal : cleanup
* llama : fix faulty bitwise check in recurrent memory
* server : disable RS-based MTP in combination with other spec types
* spec : add TODOs
* cont : fix comment
* cont : update comment
* common : fix logic for ngram + mtp compat
* llama-memory: enable checkpointing with partial rollback
* cont: add test-case for loading into a dirty ctx
* llama-memory-recurrent: clear rs_idx in clear
* download: fix mtp path
* llama-arch: fix enorm op
* docs: update docs
* conversion: fix type annotations
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-05-16 20:06:23 +08:00
Georgi Gerganov
67b2b7f2f2
logs : reduce ( #23021 )
...
Python Type-Check / python type-check (push) Waiting to run
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* logs : reduce
* args : fix envs
* server : fix build
* common : print verbosity level at start
* server : clean-up logs
* server : print prompt processing timings + sampling params
* minor : whitespaces
2026-05-14 13:05:52 +03:00
Georgi Gerganov
634275fbbb
spec : update CLI arguments for better consistency ( #22964 )
...
* spec : update CLI arguments for better consistency
* cont : fix CLI arg message
2026-05-13 09:15:39 +03:00
Georgi Gerganov
68e7ea3eab
spec : parallel drafting support ( #22838 )
...
* spec : refactor
* spec : drop support for incompatible vocabs
* spec : update common_speculative_init()
* cont : pass seq_id
* cont : dedup ctx_seq_rm_type
* server : sketch the ctx_dft decode loop
* server : draft prompt cache and checkpoints
* server : improve ctx names
* server, spec : transition to unified spec context
* cont : sync main and drft contexts
* cont : async drft eval when possible
* cont : handle non-ckpt models
* cont : pass correct n_past for drafting
* cont : process images throught the draft context
* spec : handle draft running out of context
* server : fix mtmd draft processing
* server : fix URL for draft model
* server : add comment
* server : clean-up + dry
* speculative-simple : update
* spec : fix n_past type
* server : fix slot ctx_drft ptr
* tools : update readme
* naming : improve consistency
* spec : refactor for multi-sequence speculative context
* cont : prepare params
* cont : prepare params
* spec : support parallel drafts
* server : support parallel drafting
* llama : reuse device buffers when possible
* server, spec : clean-up
* cont : clean-up
* cont : minor
* spec : reset `drafting` flag at the end
* spec : introduce `common_speculative_process()`
* spec : allow for multiple spec types (chain of speculators)
* replace old type field of type common_speculative_type in the
common_params_speculative struct with a vector to allow multiple
types to be specified
* introduce common_get_enabled_speculative_impls(const std::vector<enum common_speculative_type>)
to figure out which implementations the user has enabled
* introduce common_speculative_type_from_names(const std::vector<std::string> & names)
to parse the already user provided spec types
* all speculators run sequentially, best one wins (we verify its drafted tokens)
* maximize expected accepted tokens for current round by calculating the
product between the probability of accepting current token (n_acc_tokens / n_gen_drafts)
and the draft's length
---------
Co-authored-by: Petros Sideris <petros.sideris@nokia.com>
2026-05-11 19:09:43 +03:00
Georgi Gerganov
d6e7b033a4
llama : add option to save memory in device buffers ( #22679 )
...
* llama : add option to save memory in device buffers
* tests : extend llama-save-load-state
2026-05-05 06:35:07 +03:00
Georgi Gerganov
80afa33aad
spec : fix draft model checkpoints ( #22521 )
...
* spec : fix draft model checkpoints
* cont : clean-up
* cont : gate the ngram-mod reset warning behind verbose flag
2026-04-30 08:32:18 +03:00
Georgi Gerganov
683c5acb90
spec : disacard last drafted token with low prob ( #22506 )
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Waiting to run
Python check requirements.txt / check-requirements (push) Waiting to run
Python Type-Check / python type-check (push) Waiting to run
2026-04-29 17:00:00 +03:00
Georgi Gerganov
14e733e36f
spec : refactor params ( #22397 )
...
* spec : refactor params
* cont : fix
* cont : rename "sparam" to "sampling"
* cont : add spec params category
* cont : add info about removed arguments
* cont : skip param length check for spec params
* cont : adapt server tests
2026-04-28 09:07:33 +03:00
Georgi Gerganov
98dc1418ea
spec : fix vocab compat checks ( #22358 )
2026-04-25 20:11:35 +03:00
Paul Dubs
72d693e4fb
spec : reset i_last when low acceptance streak occurs ( #22168 )
...
By resetting i_last to zero, we will include the current context when rebuilding the speculative map.
2026-04-21 21:29:07 +03:00
Georgi Gerganov
de71b5f81c
server : refactor "use checkpoint" logic ( #22114 )
2026-04-20 08:42:37 +03:00
Sascha Rogmann
455d8e4be8
server : speculative checkpointing ( #19493 )
...
* server : speculative decoding using checkpoints
* server : fix draft check with checkpoints
* server : rename spec vars
* server : log levels
* server : refactored spec logic to speculative.cpp
* server : renamed spec checkpoints option
* server : fix spec checkpoints, logging
* speculative : checkpoints with draft model, logging
* server : n_tokens_cur and create_checkpoint in draft
* server : fix server_speculative_callback (slot.id)
* spec : fix ngram-map/begin idx_last_check
* spec : init ckpt (begin() wasn't called)
* chore: update webui build output
* server : restore sampler in spec checkpoint and clear mem
* cont : avoid --spec-use-checkpoints argument
* cont : remove server_prompt_checkpoint_with_size
* spec : rename (leave_draft_state)
* cont : clean-up
* cont : do not ignore partial drafts even if the are short
* cont : spec callback owned by session
* cont : simplify
* cont : avoid empty speculative session
* cont : simplify
* cont : simplify
* cont : enable mtmd speculative decoding
* cont : keep the spec sampler alive
* cont : simplify
* cont : fix nullptr deref + draft checkpoints
* cont : remove common_speculative_accept_response
* cont : remove callback
* cont : simplify
* cont : minor
* cont : simplify
* cont : fix accepted number
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-04-19 10:24:06 +03:00
Sascha Rogmann
292f6908cd
spec : remove check rate ( #19377 )
...
* spec: remove parameter spec-ngram-check-rate
* spec : renamed statistics vars
* spec : add n_call_begin, n_call_accept
* spec : don't enable key-map-stats
2026-02-09 15:30:50 +02:00
Georgi Gerganov
dfde5993ea
common : add common_speculative_is_compat() ( #19270 )
...
* llama : add llama_memory_can_rm_suffix()
* Revert "llama : add llama_memory_can_rm_suffix()"
This reverts commit d30e59b62a15ef4266a6503e3f4eba770aec001b.
* spec : check if the target context is compatible for spec decoding
2026-02-06 16:47:22 +02:00
Georgi Gerganov
d838c22bb3
spec : fix the check-rate logic of ngram-simple ( #19261 )
...
* spec : fix the check-rate logic of ngram-simple
* cont : refactor + fix checks
2026-02-04 10:39:53 +02:00
Georgi Gerganov
aeb827a3cc
spec : simplify time measurement using common_time_meas ( #19262 )
2026-02-03 08:20:15 +02:00
Sascha Rogmann
b4d05a3d2f
spec : various improvements ton ngram-map + docs ( #19253 )
...
* spec: ngram-map and reasoning chats
* spec: add t_begin and t_accept
* ngram-map : add internal hash map
* docs : update ngram-map, add ngram-mod
* docs : fix ngram-map-k
* docs : differences between implementations
2026-02-02 08:26:58 +02:00
Georgi Gerganov
dabaa2e77a
spec : add ngram-mod ( #19164 )
...
* spec : add ngram-mod
* cont : simplify + keep track of occupancy
* cont : cleanup
* cont : move initialization to common/speculative
* cont : cleanup
* cont : cleanup
* cont : fix
2026-01-30 18:21:48 +02:00
