vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-08-23 23:37:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
14251 commits 3 branches 252 tags 878 MiB
Commit graph

1475 commits

Author SHA1 Message Date
Concedo
849ec89bad restructure some compilation units 2026-07-01 18:51:25 +08:00
Concedo
61ad97cbc1 Merge commit '8c146a8366' into concedo_experimental 
# Conflicts:
#	src/CMakeLists.txt
#	tests/test-llama-archs.cpp
 2026-06-30 22:00:03 +08:00
Aman Gupta
8c146a8366
DeepSeek V4 (#24162) 
* convert: add dsv4 conversion

* add basic setup

* add llm_graph_input_dsv4

* add save-load state

* add sinkhorn eps - correction by @fairydreaming

* add rope fix

* cleanup dead code

* fix bugs

* support pro model: added by @fairydreaming

* remove redundant V cache

* Chat template

* remove debugging leftovers

* Add mechanism for inlining templates based on architecture

* s/deepseek-v4-flash/deepseek4/g

* s/deepseek-v4-flash/deepseek4/g continued

* enable graph reuse

* enable FA

* fix test llama archs

* rename

* compatibility with antirez ds4 GGUFs

* simplified set_gguf_parameters() by calling super class method, replaced moe.score_func with expert_gating_func.

* reserve worst-case kv-cache

* revert max split inputs

* address review comments

* add padding to enable FA

* pad only the final value of plan.n_kv to 256

* remove built-in cpp chat template

* cont: remove cpp built-in template

* rm outdated test

* replace ggml_view_3d() with ggml_reshape_3d()

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* only support n_seq=1 for now

* remove unused var

* cont: remove unused var

* use scale bias

* use correct ptr for can_reuse

* remove gen-chat-inline-templates.py

* simplify graph reuse

* cont: cleanup

* remove unused inputs

* enable partial checkpointing

* add correct shape for kq_mask + set llama_model_n_swa to 0 for dsv4

* precompute source_idx + add comment about dummy write

* support multi-seq

* remove restored_trim_pos

* use split_equal when possible

* fix indent

* address review comments

* use LLM_KV

* fix ci

---------

Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-06-29 16:58:51 +08:00
Concedo
3b867bd4b1 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.github/workflows/release.yml
#	SECURITY.md
#	common/CMakeLists.txt
#	docs/speculative.md
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/cvt.cl
#	ggml/src/ggml-opencl/kernels/flash_attn_f16.cl
#	ggml/src/ggml-opencl/kernels/flash_attn_f32.cl
#	ggml/src/ggml-opencl/kernels/flash_attn_f32_f16.cl
#	ggml/src/ggml-opencl/kernels/set_rows.cl
#	ggml/src/ggml-openvino/ggml-openvino.cpp
#	ggml/src/ggml-sycl/norm.cpp
#	tests/CMakeLists.txt
#	tests/test-backend-ops.cpp
#	tests/test-chat-template.cpp
#	tests/test-chat.cpp
#	tests/test-export-graph-ops.cpp
#	tests/test-jinja.cpp
#	tests/test-llama-archs.cpp
#	tools/rpc/CMakeLists.txt
#	tools/rpc/README.md
 2026-06-29 16:43:44 +08:00
Ruixiang Wang
d1b34251bc
spec : add DFlash support (#22105) 
* spec: add DFlash v2 support

* dflash: support sliding window attention per layer_types

* docs: add dflash section

---------

Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>
 2026-06-28 16:01:34 +03:00
Georgi Gerganov
27c8bb4f63
logs : reduce v2 (#25078) 
* server : reduce logs

* cont : common

* cont : spec

* cont : CMN_ -> COM_
 2026-06-28 08:52:15 +03:00
Concedo
e27861e14e Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/cann.Dockerfile
#	.devops/cpu.Dockerfile
#	.devops/cuda.Dockerfile
#	.devops/intel.Dockerfile
#	.devops/musa.Dockerfile
#	.devops/openvino.Dockerfile
#	.devops/rocm.Dockerfile
#	.devops/s390x.Dockerfile
#	.devops/vulkan.Dockerfile
#	.devops/zendnn.Dockerfile
#	.github/workflows/build-cache.yml
#	.github/workflows/build-openvino.yml
#	.github/workflows/build-self-hosted.yml
#	.github/workflows/release.yml
#	app/llama.cpp
#	build-xcframework.sh
#	docs/backend/OPENVINO.md
#	ggml/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-openvino/ggml-decoder.cpp
#	ggml/src/ggml-openvino/openvino/op/add_id.cpp
#	ggml/src/ggml-openvino/openvino/op/glu_swiglu.cpp
#	ggml/src/ggml-openvino/openvino/op/mul_mat_id.cpp
#	ggml/src/ggml-openvino/openvino/op/softmax.cpp
#	ggml/src/ggml-openvino/openvino/op_table.cpp
#	ggml/src/ggml-openvino/openvino/op_table.h
#	ggml/src/ggml-sycl/softmax.cpp
#	scripts/sync-ggml.last
#	tests/test-backend-ops.cpp
#	tests/test-quantize-fns.cpp
#	tools/server/CMakeLists.txt
#	tools/ui/src/lib/services/chat.service.ts
 2026-06-27 10:33:29 +08:00
Concedo
4e43c21e58 Merge commit '9d5d882d8c' into concedo_experimental 
# Conflicts:
#	.github/labeler.yml
#	app/CMakeLists.txt
#	app/llama.cpp
#	build-xcframework.sh
#	common/CMakeLists.txt
#	common/download.h
#	docs/backend/SYCL.md
#	docs/backend/snapdragon/CMakeUserPresets.json
#	docs/speculative.md
#	ggml/CMakeLists.txt
#	ggml/include/ggml-sycl.h
#	ggml/src/ggml-hexagon/CMakeLists.txt
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/cmake-toolchain.cmake
#	ggml/src/ggml-hexagon/htp/flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/hex-dma.h
#	ggml/src/ggml-hexagon/htp/hex-utils.h
#	ggml/src/ggml-hexagon/htp/hmx-flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/htp-ctx.h
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/htp_iface.idl
#	ggml/src/ggml-hexagon/htp/hvx-base.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.c
#	ggml/src/ggml-hexagon/libggml-htp.inf
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/norm.cl
#	ggml/src/ggml-sycl/conv3d.cpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	scripts/snapdragon/adb/run-completion.sh
#	scripts/snapdragon/adb/run-tool.sh
#	scripts/snapdragon/ggml-hexagon-profile.py
#	tests/CMakeLists.txt
#	tests/test-backend-ops.cpp
#	tests/test-thread-safety.cpp
#	tools/llama-bench/llama-bench.cpp
#	tools/mtmd/CMakeLists.txt
#	tools/mtmd/tests/test-deepseek-ocr.py
 2026-06-27 10:18:52 +08:00
Arsen Arutunan
960d628f46
mamba2: remove hardcoded 2x expansion factor and invalid d_inner % d_state check (#23082) 
* mamba2: remove hardcoded 2x expansion factor, support any expand value

* mamba2: remove invalid d_inner %% d_state check (unrelated parameters)

* Update convert_hf_to_gguf.py: make expand optional with default 2

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* mamba2: apply expand fix to refactored conversion/mamba.py

* also check for mamba_expand

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
 2026-06-26 08:50:54 +03:00
Tarek Dakhran
9d5d882d8c
model : Add label for LFM2.5-230M (#25008) 2026-06-25 18:58:52 +02:00
Sigbjørn Skjæret
b3ce5cedf4
quant : fix quantizing moe with mtp (#24986) 2026-06-25 08:36:49 +03:00
Concedo
579229d157 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	CODEOWNERS
#	README.md
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q8_0_f32.cl
#	ggml/src/ggml-sycl/binbcast.cpp
#	ggml/src/ggml-sycl/element_wise.cpp
#	ggml/src/ggml-vulkan/CMakeLists.txt
#	ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_id_vec.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec_acc.tmpl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec_q_acc.tmpl
#	ggml/src/ggml-webgpu/wgsl-shaders/quantize_q8.wgsl
#	tests/test-backend-ops.cpp
#	tests/test-chat.cpp
#	tests/test-sampling.cpp
#	tools/server/README.md
 2026-06-24 23:28:21 +08:00
Tarek Dakhran
88636e178f
model : Add LFM2.5-ColBERT-350M and LFM2.5-Embedding-350M (#24913) 
* model : Add LFM2.5-ColBERT-350M and LFM2.5-Embedding-350M

* Restore LFM2 models in README.md
 2026-06-24 09:49:46 +03:00
Tim Neumann
37957e8531
sampling : remove unconditional softmax+sort in top-n-sigma sampler (#22645) 2026-06-22 14:08:32 +03:00
Concedo
3090ae0bf7 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/s390x.Dockerfile
#	.dockerignore
#	.github/workflows/docker.yml
#	.github/workflows/release.yml
#	docs/android.md
#	ggml/src/ggml-cpu/amx/mmq.cpp
#	ggml/src/ggml-hexagon/htp/ssm-conv.c
#	tests/peg-parser/test-gbnf-generation.cpp
#	tests/test-arg-parser.cpp
#	tests/test-chat.cpp
#	tests/test-jinja.cpp
#	tests/test-json-schema-to-grammar.cpp
#	tools/server/README.md
 2026-06-22 18:23:59 +08:00
YiChen Lv
d789527482
spec : Support Step3.5/3.7 flash mtp3 (#24340) 
* add mtp_layer_offset + include nextn flags in graph reuse

* add llama_set_mtp_layer_offset + llama_model_n_nextn_layer API

* offset head select + require all MTP blocks

* speculative multi-head process()

* speculative multi-head draft()

* gather outputs via inp_out_ids

* cleanup

* fix core

* minor cleanup

* merged draft_multi_head into draft()

* mtp rename nextn

* Apply suggestions from code review

Co-authored-by: Aman Gupta <amangupta052@gmail.com>

* clean-up comments

* fix for multi seq

* apply suggestions && chain-heads comment

* add a reference for chain_heads discussion

---------

Co-authored-by: Aman Gupta <amangupta052@gmail.com>
 2026-06-21 11:33:18 +03:00
Adrien Gallouët
84de01a1f1
llama : use LLM_KV for quantization_version & file_type (#24802) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-20 20:07:01 +02:00
davidrhodus
796f41bedc
model : glm-dsa load DSA indexer tensors as optional (#24770) 
GLM-5.2 ships the DSA "lightning indexer" on only a subset of layers (the
"full" layers; others omit it), but the GLM_DSA loader created the five
indexer tensors on every layer as required, so loading any GLM-5.2 GGUF
failed with e.g. `missing tensor 'blk.3.indexer.k_norm.weight'`.

GLM_DSA's graph is llama_model_deepseek2::graph (plain MLA) and does not use
the indexer tensors (indexer runtime not yet implemented), so they are
loaded-but-unused. Marking them TENSOR_NOT_REQUIRED lets layers without an
indexer load as nullptr and the model runs as full MLA attention.

DeepSeek-V3.2 (uniform indexer on all layers) is unaffected.
 2026-06-20 13:48:24 +03:00
Concedo
73cc7d9287 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/cann.Dockerfile
#	.devops/cpu.Dockerfile
#	.devops/cuda.Dockerfile
#	.devops/intel.Dockerfile
#	.devops/musa.Dockerfile
#	.devops/openvino.Dockerfile
#	.devops/rocm.Dockerfile
#	.devops/s390x.Dockerfile
#	.devops/vulkan.Dockerfile
#	.devops/zendnn.Dockerfile
#	.dockerignore
#	.pi/gg/SYSTEM.md
#	ggml/CMakeLists.txt
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	scripts/sync-ggml.last
#	scripts/sync_vendor.py
#	tools/cli/README.md
#	tools/cli/cli.cpp
#	tools/mtmd/clip.cpp
#	tools/server/README.md
 2026-06-20 17:16:19 +08:00
Ruixiang Wang
b14e3fb90c
spec: support eagle3 for qwen3.5 & 3.6 (#24593) 
* spec: support qwen3.5 & 3.6 eagle3 draft

* eagle3: Add deferred boundary checkpoints restore support for hybrid models

* apply suggestions

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* spec: adapt to API change

* spec: fix naming

* cont : add TODO

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-06-19 13:08:50 +03:00
Concedo
6591c33667 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.github/workflows/release.yml
#	app/llama.cpp
#	common/download.cpp
#	docs/backend/SYCL.md
#	docs/ops.md
#	docs/ops/SYCL.csv
#	ggml/CMakeLists.txt
#	ggml/src/CMakeLists.txt
#	ggml/src/ggml-cpu/CMakeLists.txt
#	ggml/src/ggml-sycl/CMakeLists.txt
#	ggml/src/ggml-sycl/backend.hpp
#	ggml/src/ggml-sycl/common.cpp
#	ggml/src/ggml-sycl/common.hpp
#	ggml/src/ggml-sycl/convert.cpp
#	ggml/src/ggml-sycl/dequantize.hpp
#	ggml/src/ggml-sycl/dmmv.cpp
#	ggml/src/ggml-sycl/dpct/helper.hpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-sycl/mmvq.cpp
#	ggml/src/ggml-sycl/outprod.cpp
#	ggml/src/ggml-sycl/vecdotq.hpp
#	tools/server/README.md
 2026-06-18 21:00:52 +08:00
Dev-iL
b4024af6c2
llama : skip main_gpu validation when no devices are available (#23405) 2026-06-17 17:30:26 +03:00
Ruixiang Wang
1a2dea29b9
spec: fix segfault error on long prompts for eagle3 (#24707) 2026-06-17 17:29:49 +03:00
Concedo
b8b7763c76 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/openvino.Dockerfile
#	.github/workflows/build-cache.yml
#	.github/workflows/build-openvino.yml
#	.github/workflows/build-self-hosted.yml
#	.github/workflows/release.yml
#	docs/backend/OPENVINO.md
#	docs/backend/SYCL.md
#	docs/ops.md
#	docs/ops/SYCL.csv
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/mul_mv_f16_f32_l4.cl
#	ggml/src/ggml-openvino/.clang-format
#	ggml/src/ggml-openvino/CMakeLists.txt
#	ggml/src/ggml-openvino/ggml-decoder.cpp
#	ggml/src/ggml-openvino/ggml-decoder.h
#	ggml/src/ggml-openvino/ggml-openvino-extra.cpp
#	ggml/src/ggml-openvino/ggml-openvino-extra.h
#	ggml/src/ggml-openvino/ggml-openvino.cpp
#	ggml/src/ggml-openvino/ggml-quants.cpp
#	ggml/src/ggml-openvino/ggml-quants.h
#	ggml/src/ggml-openvino/openvino/decoder.h
#	ggml/src/ggml-openvino/openvino/frontend.h
#	ggml/src/ggml-openvino/openvino/input_model.h
#	ggml/src/ggml-openvino/openvino/node_context.h
#	ggml/src/ggml-openvino/openvino/op/cont.cpp
#	ggml/src/ggml-openvino/openvino/op/cpy.cpp
#	ggml/src/ggml-openvino/openvino/op/flash_attn_ext.cpp
#	ggml/src/ggml-openvino/openvino/op/get_rows.cpp
#	ggml/src/ggml-openvino/openvino/op/glu_geglu.cpp
#	ggml/src/ggml-openvino/openvino/op/glu_swiglu.cpp
#	ggml/src/ggml-openvino/openvino/op/mulmat.cpp
#	ggml/src/ggml-openvino/openvino/op/permute.cpp
#	ggml/src/ggml-openvino/openvino/op/reshape.cpp
#	ggml/src/ggml-openvino/openvino/op/rms_norm.cpp
#	ggml/src/ggml-openvino/openvino/op/rope.cpp
#	ggml/src/ggml-openvino/openvino/op/set_rows.cpp
#	ggml/src/ggml-openvino/openvino/op/softmax.cpp
#	ggml/src/ggml-openvino/openvino/op/transpose.cpp
#	ggml/src/ggml-openvino/openvino/op/unary_silu.cpp
#	ggml/src/ggml-openvino/openvino/op/view.cpp
#	ggml/src/ggml-openvino/openvino/op_table.cpp
#	ggml/src/ggml-openvino/openvino/op_table.h
#	ggml/src/ggml-openvino/openvino/pass/mark_decompression_convert_constant_folding.h
#	ggml/src/ggml-openvino/openvino/translate_session.cpp
#	ggml/src/ggml-openvino/openvino/translate_session.h
#	ggml/src/ggml-openvino/openvino/utils.cpp
#	ggml/src/ggml-openvino/openvino/utils.h
#	ggml/src/ggml-openvino/utils.cpp
#	ggml/src/ggml-openvino/utils.h
#	ggml/src/ggml-sycl/common.hpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
 2026-06-17 16:06:00 +08:00
Oliver Simons
02810c7aa8
Fix and restrict NVFP4 edge-cases in llama-graph (#24331) 
* Move post-GEMM MUL required for dequant b4 lora and bias add

see https://github.com/ggml-org/llama.cpp/pull/23484 :
1. For lora, I would presume we want fully dequantized values before
   doing the residuals, but this depends on how the LORAs were
generated. Literature tells me LORA happens post-mul but pre-bias add https://github.com/ggml-org/llama.cpp/pull/8332
2. For ModelOPT, bias-add should happen on [fully-dequantized
   values](b49f9b9e2d/modelopt/torch/quantization/backends/nvfp4_gemm.py (L59-L64))

* Restrict build_ffn for NVFP4 to supported combinations
 2026-06-16 11:52:38 +02:00
Concedo
71801743c4 added support for cohere north mini 2026-06-16 00:24:30 +08:00
Concedo
a91357614a added lfm2 tool call template support 2026-06-15 22:55:24 +08:00
Concedo
bdbcd049b1 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/cpu.Dockerfile
#	.devops/cuda.Dockerfile
#	.devops/intel.Dockerfile
#	.devops/llama-cli-cann.Dockerfile
#	.devops/musa.Dockerfile
#	.devops/openvino.Dockerfile
#	.devops/rocm.Dockerfile
#	.devops/s390x.Dockerfile
#	.devops/vulkan.Dockerfile
#	.devops/zendnn.Dockerfile
#	.github/labeler.yml
#	docs/backend/CUDA-FEDORA.md
#	docs/ops.md
#	docs/ops/SYCL.csv
#	examples/sycl/build.sh
#	examples/sycl/win-build-sycl.bat
#	ggml/src/ggml-sycl/common.cpp
#	ggml/src/ggml-sycl/common.hpp
#	ggml/src/ggml-sycl/cpy.hpp
#	ggml/src/ggml-sycl/dmmv.cpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-sycl/mmvq.cpp
#	ggml/src/ggml-sycl/presets.hpp
#	ggml/src/ggml-sycl/set_rows.cpp
#	ggml/src/ggml-sycl/softmax.cpp
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_decls.tmpl
#	grammars/README.md
#	tests/test-backend-ops.cpp
#	tests/test-chat-auto-parser.cpp
#	tests/test-chat.cpp
#	tests/test-jinja.cpp
#	tools/cli/cli.cpp
 2026-06-15 20:00:57 +08:00
Bartowski
8ed274ef46
Add cohere2moe to llama-vocab for TINY_AYA (#24601) 2026-06-14 09:04:46 +02:00
Concedo
ea0351c71a Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.github/workflows/release.yml
#	.github/workflows/ui-build-self-hosted.yml
#	.github/workflows/ui-build.yml
#	.github/workflows/ui-publish.yml
#	.github/workflows/ui-self-hosted.yml
#	.github/workflows/ui.yml
#	.gitignore
#	README.md
#	docs/ops.md
#	docs/ops/Vulkan.csv
#	ggml/CMakeLists.txt
#	scripts/sync-ggml.last
#	scripts/sync_vendor.py
#	scripts/ui-assets.cmake
#	tests/test-jinja.cpp
#	tests/test-llama-archs.cpp
 2026-06-14 11:26:16 +08:00
Michael Wand
4988f6e866
Add arch support for cohere2-MoE (#24260) 
* Add arch support for cohere2-MoE

* Removed redundant gating_func checks

* Changed ffn lookup to prefer prefix_dense_intermediate_size

* Renamed arch to cohere2moe

* Removed redundant lmhead check and chat template changes

* Removed lm_head.weight check from modify tensors, load output tensor not required, fallback to token_embd.weight

* Changed to (routed+shared)*0.5 for shared expert combined avg

* fixed sliding_window_pattern issue and pattern

* Fixed transformers crash 'first_k_dense_replace' error

* Remove comment

* Removed cohere2-moe as a tokenizer type and kept as tiny_aya.  Renamed North-Mini-Code-1.0.

* Fixed MTP fail, changed to use iSWA

* Fixed remaining todos: cohere2moe renamed, changed swa parsing to use get_key_or_arr, removed extra get_arr use

* Force metadata usage

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Remove Cohere2 checkpoint comment

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Remove MTP comment

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Regenerate cohere2moe tokenizer hash

* Add cohere2moe to Llama Model Saver supported list

* Check for zerobios tensors and add support for Command to use LayerNorm

* Map expert_selection_fn to sigmoid in base.py instead of command.py

* use bools for foundnorm/foundnormrms

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-06-13 19:49:00 +02:00
Georgi Gerganov
02182fc5b9
fit : avoid including llama-ext.h in fit.h (#24506) 2026-06-12 15:57:05 +03:00
Concedo
be1750a3b8 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/cuda.Dockerfile
#	.github/workflows/build-sycl.yml
#	.github/workflows/release.yml
#	ggml/CMakeLists.txt
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp/gated-delta-net-ops.c
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/cvt.cl
#	ggml/src/ggml-opencl/kernels/gated_delta_net.cl
#	ggml/src/ggml-sycl/gated_delta_net.cpp
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	ggml/src/ggml-webgpu/wgsl-shaders/gated_delta_net.wgsl
#	scripts/sync-ggml.last
#	tests/test-backend-ops.cpp
#	tests/test-llama-archs.cpp
 2026-06-12 15:39:57 +08:00
Ruixiang Wang
88a39274ec
spec: add EAGLE3 speculative decoding support (#18039)
Some checks failed
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Details
Python check requirements.txt / check-requirements (push) Has been cancelled
Details
Python Type-Check / python type-check (push) Has been cancelled
Details 
* llama : enable layer input extraction

* spec: support eagle3

* eagle3: fix params bug

* eagle3: support Gemma4 eagle3 from RedHatAI

* eagle3: set sync when get features from target

Co-authored-by: tnhnyzc <115956684+tnhnyzc@users.noreply.github.com>

* eagle3 : fix ubatch handling in embd_layer_inp extraction and encoder

Co-authored-by: Doğaç Eldenk <dogacel@gmail.com>

* eagle3: adapt to upstream changes

* eagle3: fix rebase issues and adapt to upstream changes

* eagle3:exclude the eagle3 arch from test-llama-archs

* eagle3: fix editorconfig check failures

* eagle3: fix multi-seq issue in d2t vocab mapping

* cont : minor style / clean-up

* spec : remove `common_speculative_setup_draft_model()`

* llama : clean-up unused API

* eagle3: set d2t vocab mapping in decode graph

* cont : assert layer inputs are configured

* hparams : use n_embd_inp instead of n_embd_target_features

* eagle3: make output.weight optional and inherit from target model when needed

* haparams : generic norm-before-residual param

* llama-ext : consistent names

* cont : fix

* hparams : remove target_hidden_size

* cparams : rename output_layer_inp -> embeddings_layer_inp

* arch : reuse ATTN_NORM_2 instead of adding new hidden norm

* llama : clean-up names

* cont : add assert + comment

* Update conversion/llama.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: tnhnyzc <115956684+tnhnyzc@users.noreply.github.com>
Co-authored-by: Doğaç Eldenk <dogacel@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-06-12 10:21:06 +03:00
o7si
68f30663cf
vocab : refactor normalizer flags into options struct, add strip_accents (#24371) 
* vocab : refactor normalizer flags into options struct, add strip_accents

* Update src/llama-vocab.h

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Update src/llama-vocab.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-06-11 10:36:50 +03:00
Gaurav Garg
e95dae18d6
Remove padding and multiple D2D copies for MTP (#24086) 
* Make ggml_gated_delta_net take only the initial recurrent state (D, 1, n_seqs) and passes the snapshot count K as an op parameter instead of inferring it from state->ne[1].

Remove the padding hack and copy all emitted snapshots into the recurrent cache with a single strided ggml_cpy

* Make GDN changes in all backends. Address review comments.

* Fix CI build errors
 2026-06-10 23:21:16 +05:30
Concedo
a860ec0b37 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.github/workflows/winget.yml
 2026-06-10 23:16:06 +08:00
Concedo
afc8de2c6b Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/cpu.Dockerfile
#	.devops/cuda.Dockerfile
#	.devops/intel.Dockerfile
#	.devops/musa.Dockerfile
#	.devops/openvino.Dockerfile
#	.devops/rocm.Dockerfile
#	.devops/vulkan.Dockerfile
#	.devops/zendnn.Dockerfile
#	.github/workflows/build-webgpu.yml
#	.github/workflows/release.yml
#	ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	ggml/src/ggml-webgpu/wgsl-shaders/binary.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/concat.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_decls.tmpl
#	ggml/src/ggml-webgpu/wgsl-shaders/scale.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/unary.wgsl
#	tests/CMakeLists.txt
#	tests/test-backend-ops.cpp
#	tests/test-mtmd-c-api.c
#	tools/cli/cli.cpp
#	tools/mtmd/CMakeLists.txt
#	tools/server/README.md
 2026-06-10 17:21:05 +08:00
Aarnav Pai
d73cd07674
graph: Fix granite speech model inference by applying embedding scale when deepstack is not used (#24357) 
* llama-graph : apply embedding scale when deepstack is not used

* nits: remove non-existant hunyuan-vl from the tests

* apply suggestion from @gabe-l-hart

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
 2026-06-09 19:46:27 +02:00
Concedo
dd2f966f4b Merge commit 'c2b1518fd4' into concedo_experimental 
# Conflicts:
#	.devops/intel.Dockerfile
#	ggml/CMakeLists.txt
#	scripts/sync-ggml.last
#	tests/test-backend-ops.cpp
#	tests/test-llama-archs.cpp
#	tools/cli/cli.cpp
 2026-06-10 00:56:44 +08:00
Sigbjørn Skjæret
f0152efe40
models : fix plamo2 attention_key/value_length regression (#24317) 2026-06-09 10:26:44 +03:00
Max Krasnyansky
7d2b45b4f7
mtp: support for gemma-4 E2B and E4B assistants (#24282) 
* models: update converter to support smaller assistants

* models: add masked_embd tensors to gemma4-assist arch

* gemma-4: remove temp debug for conversion

* gemma-4-mtp: filter out masked_embedding tensors during conversion
 2026-06-08 13:48:52 -07:00
Pascal
a66d50588b
graph: guard iswa kq_mask on its own buffer (#24294) 
A SWA-only draft head (e.g. StepFun MTP) leaves the base sub-cache
empty, so its kq_mask buffer stays null and asserts at load. Guard
each mask on its own buffer in set_input and can_reuse, base and swa.

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-06-08 19:20:28 +02:00
Georgi Gerganov
379ac6673b
kv-cache : avoid kv cells copies (#24277) 2026-06-07 21:42:54 +03:00
Pascal
f0156d1401
kv-cache: follow the source cache size when sharing cells (#24267) 
A fitted target context can end up smaller than the draft default, the
oversized assistant views then overflow the shared K/V tensors and trip
the ggml_view_4d size assert during graph reserve.
 2026-06-07 18:33:00 +03:00
Aman Gupta
04eb4c446d
llama : add Gemma4 MTP (#23398) 2026-06-07 20:50:54 +08:00
Concedo
e40e5c660e Merge commit '5a69c97439' into concedo_experimental 
# Conflicts:
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/concat.cl
#	ggml/src/ggml-opencl/kernels/cpy.cl
#	ggml/src/ggml-opencl/kernels/get_rows.cl
#	ggml/src/ggml-opencl/kernels/mul_mv_q6_k_f32_flat.cl
#	tests/test-chat.cpp
#	tools/mtmd/CMakeLists.txt
#	tools/mtmd/clip.cpp
#	tools/mtmd/clip.h
 2026-06-07 17:46:09 +08:00
Concedo
202aaef40b Merge commit '6effcecd0b' into concedo_experimental 
# Conflicts:
#	.github/workflows/build-msys.yml
#	.pi/gg/SYSTEM.md
#	common/CMakeLists.txt
#	examples/speculative-simple/speculative-simple.cpp
#	ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-sycl/mmvq.cpp
#	scripts/ui-assets.cmake
#	src/llama-model.cpp
#	tools/cli/cli.cpp
#	tools/imatrix/imatrix.cpp
#	tools/quantize/quantize.cpp
#	tools/server/README.md
#	tools/ui/tests/stories/a11y/ChatScreenForm.a11y.stories.svelte
 2026-06-07 17:10:58 +08:00
Sigbjørn Skjæret
5343f4502a
model : rename local n_layer_all variable (#24209) 2026-06-06 07:07:20 +03:00
Sigbjørn Skjæret
603300b008
context : fix off-by-one comparisons to n_gpu_layers (#24208) 2026-06-06 07:06:47 +03:00