vrr/koboldcpp
2
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-08-26 00:34:30 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
14708 commits 3 branches 252 tags 884 MiB
Commit graph

348 commits

Author SHA1 Message Date
Concedo
98926f27c1 Merge commit '11b068d066' into concedo_experimental 
# Conflicts:
#	CONTRIBUTING.md
#	docs/backend/SYCL.md
#	docs/install.md
#	docs/speculative.md
#	ggml/src/ggml-hip/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-sycl/common.hpp
#	ggml/src/ggml-sycl/element_wise.cpp
#	ggml/src/ggml-sycl/fattn-onednn.cpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	ggml/src/ggml-webgpu/wgsl-shaders/glu.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/ssm_scan.wgsl
#	tests/test-backend-ops.cpp
#	tests/test-chat.cpp
#	tests/test-llama-archs.cpp
#	tools/cli/README.md
#	tools/llama-bench/llama-bench.cpp
#	tools/mtmd/CMakeLists.txt
#	tools/server/README.md
 2026-07-31 17:06:19 +08:00
Aman Gupta
f5b9bd39b5
RPC: add tensor_memset (#25912) 2026-07-29 15:04:30 +08:00
Concedo
0a0b88a5c0 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	docs/backend/OPENCL.md
#	ggml/CMakeLists.txt
#	ggml/src/ggml-cpu/CMakeLists.txt
#	ggml/src/ggml-cpu/kleidiai/kernels.cpp
#	ggml/src/ggml-cpu/kleidiai/kernels.h
#	ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp-drv.cpp
#	ggml/src/ggml-hexagon/htp-drv.h
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/act-ops.c
#	ggml/src/ggml-hexagon/htp/argsort-ops.c
#	ggml/src/ggml-hexagon/htp/binary-ops.c
#	ggml/src/ggml-hexagon/htp/cumsum-ops.c
#	ggml/src/ggml-hexagon/htp/flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/hex-dma.h
#	ggml/src/ggml-hexagon/htp/hex-profile.h
#	ggml/src/ggml-hexagon/htp/hex-utils.h
#	ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h
#	ggml/src/ggml-hexagon/htp/hmx-queue.c
#	ggml/src/ggml-hexagon/htp/hmx-queue.h
#	ggml/src/ggml-hexagon/htp/htp-ctx.h
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.h
#	ggml/src/ggml-hexagon/htp/rope-ops.c
#	ggml/src/ggml-hexagon/htp/unary-ops.c
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/gemm_moe_mxfp4_q8_1_dp4a.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_q4_0_q8_1_dp4a.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_q4_k_q8_1_dp4a.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_q6_k_q8_1_dp4a.cl
#	ggml/src/ggml-opencl/kernels/gemm_moe_q8_1_dp4a.cl
#	ggml/src/ggml-opencl/kernels/gemm_noshuffle_q4_k_f32.cl
#	ggml/src/ggml-opencl/kernels/gemm_noshuffle_q4_k_q8_1_dp4a.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_k_f32.cl
#	ggml/src/ggml-opencl/kernels/mul_mv_q4_k_f32_flat.cl
#	ggml/src/ggml-opencl/kernels/mul_mv_q5_k_f32_flat.cl
#	ggml/src/ggml-sycl/dmmv.cpp
#	scripts/snapdragon/ggml-hexagon-profile.py
#	scripts/sync-ggml.last
#	tests/CMakeLists.txt
#	tests/test-backend-ops.cpp
#	tests/test-llama-archs.cpp
#	tests/test-recurrent-state-rollback.cpp
#	tools/cli/README.md
#	tools/completion/README.md
#	tools/server/README.md
 2026-07-18 12:26:15 +08:00
Aman Gupta
0dc74e332e
DeepseekV4: Add fused hyper-connection ops (#25585) 
* dsv4 hc-ops

* add missing files;

* add cparams

* update rpc version

* address review comments

* address review comments
 2026-07-17 00:33:33 +08:00
Rajendra Matcha
f15bd60901
kleidiai: Add SME vs SME2 distinction in kernel dispatch (#25478) 
The current integration treats SME as a single capability (CPU_FEATURE_SME)
with no distinction between SME(v1) and SME2. The kernels dispatched under
CPU_FEATURE_SME use SME2-specific instructions, making dispatch incorrect
on SME(v1)-only hardware.

We introduce build-time and runtime distinction between SME and SME2, and
wire SME(v1) and SME2 kernels based on actual hardware support.
 2026-07-16 08:57:04 -07:00
Concedo
9001369da0 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	ggml/src/ggml-cpu/CMakeLists.txt
#	ggml/src/ggml-cpu/kleidiai/kernels.cpp
#	ggml/src/ggml-cpu/kleidiai/kernels.h
#	ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
#	ggml/src/ggml-cpu/ops.cpp
#	ggml/src/ggml-cuda/mmq.cuh
#	ggml/src/ggml-hexagon/htp/hmx-queue.c
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_iq4_nl_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q1_0_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_0_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_0_f32_spec.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_1_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_k_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q5_0_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q5_1_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q5_k_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q6_k_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q8_0_f32.cl
#	ggml/src/ggml-opencl/kernels/mul_mv_f16_f16.cl
#	ggml/src/ggml-opencl/kernels/mul_mv_f16_f32.cl
#	ggml/src/ggml-opencl/kernels/mul_mv_f16_f32_1row.cl
#	ggml/src/ggml-sycl/fattn-vec.hpp
#	tests/test-backend-ops.cpp
#	tests/test-chat-auto-parser.cpp
#	tests/test-export-graph-ops.cpp
#	tests/test-jinja.cpp
#	tests/test-llama-archs.cpp
#	tools/tokenize/tokenize.cpp
 2026-07-15 15:39:31 +08:00
fairydreaming
a7312ae94f
ggml : add a set of functions for checking contiguity of inner tensor dimensions (#25650) 
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
 2026-07-14 14:37:52 +02:00
Concedo
fa21872f97 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	docs/backend/SYCL.md
#	ggml/src/ggml-cpu/ggml-cpu.c
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/cvt.cl
#	ggml/src/ggml-opencl/kernels/gemv_moe_mxfp4_f32_ns.cl
#	ggml/src/ggml-opencl/kernels/gemv_moe_q4_k_f32_ns.cl
#	ggml/src/ggml-sycl/backend.hpp
#	ggml/src/ggml-sycl/common.hpp
#	ggml/src/ggml-sycl/dmmv.cpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	scripts/sync_vendor.py
#	tests/CMakeLists.txt
#	tests/test-alloc.cpp
#	tests/test-backend-ops.cpp
#	tests/test-chat.cpp
#	tests/test-gguf.cpp
#	tests/test-save-load-state.cpp
 2026-07-13 20:53:37 +08:00
QuintinShaw
ad8d821991
gguf : add tensor shape accessor (#24405) 
* gguf : add tensor shape accessors

* gguf : return tensor shape as const int64_t *

* gguf : remove n_dims accessor, keep only gguf_get_tensor_ne
 2026-07-13 13:55:15 +03:00
fairydreaming
00f5442cc4
ggml : add GGML_OP_LIGHTNING_INDEXER that implements DeepSeek V3.2/V4 lightning indexer (#24231) 
* ggml : add GGML_OP_LIGHTNING_INDEXER that implements DeepSeek V3.2/V4 lightning indexer

* ggml : remove scale parameters from lightning indexer OP, add f16 mask parameter

* tests : add GGML_OP_LIGHTNING_INDEXER tests

* ggml : bump RPC version

* chore : check if lightning indexer input tensors are not transposed

* tests : count flops instead of bandwidth in lightning indexer test

* chore : add missing const

* chore : whitespace

* ggml : renamed variables in CPU lightning indexer implementation

* ggml : fix lightning indexer mask broadcasting

* tests : tests for lightning indexer mask broadcasting

* chore : whitespace

* llama : use GGML_OP_LIGHTNING_INDEXER in DeepSeek V3.2 and DeepSeek V4 models

---------

Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
 2026-07-11 11:39:07 +02:00
Concedo
f57cd915a9 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.github/workflows/ui-publish.yml
#	CODEOWNERS
#	docs/ops.md
#	ggml/CMakeLists.txt
#	ggml/src/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/argsort-ops.c
#	scripts/sync-ggml.last
#	tools/mtmd/tests/test-deepseek-ocr.py
 2026-07-11 11:36:59 +08:00
Concedo
0f0245161e Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	tests/test-backend-ops.cpp
#	tests/test-quantize-fns.cpp
#	tools/server/README.md
#	tools/ui/src/lib/constants/settings-registry.ts
 2026-07-11 09:20:40 +08:00
Concedo
fc6e197fdd Merge commit '33ca0dcb9d' into concedo_experimental 
# Conflicts:
#	docs/backend/SYCL.md
#	docs/build.md
#	docs/ops.md
#	docs/ops/SYCL.csv
#	ggml/src/ggml-cuda/ggml-cuda.cu
#	ggml/src/ggml-hip/CMakeLists.txt
#	ggml/src/ggml-opencl/fa_tune.h
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/flash_attn_f16.cl
#	ggml/src/ggml-opencl/kernels/flash_attn_f32.cl
#	ggml/src/ggml-opencl/kernels/flash_attn_f32_f16.cl
#	ggml/src/ggml-opencl/kernels/flash_attn_f32_q4_0.cl
#	ggml/src/ggml-opencl/kernels/flash_attn_f32_q8_0.cl
#	ggml/src/ggml-opencl/kernels/mul_mv_f16_f32_l4.cl
#	ggml/src/ggml-sycl/backend.hpp
#	ggml/src/ggml-sycl/common.hpp
#	ggml/src/ggml-sycl/cpy.cpp
#	ggml/src/ggml-sycl/cpy.hpp
#	ggml/src/ggml-sycl/dmmv.cpp
#	ggml/src/ggml-sycl/element_wise.cpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-sycl/presets.hpp
#	src/llama-graph.cpp
#	src/llama-kv-cache.cpp
 2026-07-11 08:46:16 +08:00
Martin Chang
082b326fc7
ggml-et: Initial ET backend (#24179) 
* ggml-et: Add performance logging

* ggml-et: Quants helpers

* ggml-et: Add MUL_MAT kernel

* ggml-et: Add ROPE kernel

* ggml-et: Add RMS_NORM kernel

* ggml-et: Add GLU kernel

* ggml-et: Add SOFT_MAX kernel

* ggml-et: Add GET_ROWS kernel

* ggml-et: Add CONT kernel

* ggml-et: Add SET_ROWS kernel

* ggml-et: Add MUL_MAT_ID kernel

* ggml-et: Build et kernels as part of ggml

* ggml-et: Embed kernels with fs fallback

* ggml-et: Build fixes

* ggml-et: Add MUL_MAT F32xF32 op

* ggml_et: Add MUL_MAT_ID op

* ggml-et: Disable offloading for debug

* ggml-et: Refactor out block ops

* ggml-et: ggml backend API changes

* ggml-et: Add RESHAPE/TRANSPOSE to supported

* ggml-et: Add CONT_F16

* ggml-et: Add supported ops doc

* gglm-et: Initial doc

* ggml-et: Remove  runtime import hacks

We can now import the runtime by a simple find_package(), so we
can cleanup the CMakeLists.txt.

* ggml-et: Fix GET_ROWS kernel

Fix lost batch dimension.

Also clean vibe-comments.

* ggml-et: Fix SET_ROWS kernel

Remove incorrect broadcasting guard.

* ggml-et: Use custom instruction for fp32->fp16

* ggml-et: Vectorize set_rows fp32->fp16

* ggml-et: Fix ROPE kernel (yarn)

ggml-et: fix et_logf

WIP: Fix ramp

WIP: fix ROPE!

* ggml-et: Better sinf

* ggml-et: Fix SOFT_MAX

Add `max_bias` and `sink` support.

* ggml-et: Fix CONT

Reorder from contiguous write to read with atomic stores.

* ggml-et: Fix elmap kernel

Remainder handlin

* ggml-et: Fix MUL_MAT MUL_MAT_ID remainders

* ggml-et: Fix ET-SOC reference

* ggml-et: Fix embed kernels scripts for old python

This allows GGML-ET to build on pre-3.8 python.

* Add sysemu support with compile time flag `-DGGML_ET_SYSEMU=ON` (#6)

* Example using ET-Soc-1 emulator configuration

Example usage:
```bash
cmake -B build -DGGML_CUDA=OFF -DGGML_ET=ON -DLLAMA_CURL=OFF -DGGML_CCACHE=ON
cmake --build build --config Release -j $(nproc)

time ./build/bin/test-backend-ops

./build/bin/llama-server \
    --model Qwen3-0.6B-Q8_0.gguf \
    --alias Qwen3-0.6B-Q8_0 \
    -fa 0 \
    --ctx-size 1024 \
    --no-warmup \
    --host 127.0.0.1 \
    --port 8080
```

* build: proper dep tracking for kernels

* support host using MOLD linker

* initial multi core GET_ROW F32 implementation

* vectorized q8 dequant

* wip: cland warning clenaups and initial logging refactor

* wip: message default message cleanup

* chore: message cleanups

* cmake cleanup

* migrate to use platform provided functions

* cmake back into subdir

* support et_print() in kernels

* fix: repair kernel building

* perf: operations run async by default

* debug: proper kernel dep tracking and error detection on kenrel launch

* fix: kernel binary dep tracking and fixing get_rows_f32 erroring

* perf: back to doing async kernel runs by default

* perf: vectorize and parallel device memset

* merge matmul work

* misc: align allocation and enable all offload

* misc: delete deadcode and respect memory limits

* fix: repair tensor debug print

* fix: loosen RMS_NORM op percision

* feat: Q4_0 GET_ROWS

* perf: FP32 MUL_MAT using TensorFMA

* update limitations

* perf: redue L1 load in compute_block_dot_product_q8_0

* feat: save kernel mapping (name to id) when profiling is enabled

* chore: memops cleanup

* perf: parallelize softmax by rows

* perf: vectorize 2nd phase of softmax

* perf: ban GET_ROWS from offloaded

* perf: vectorize and non-atomic for eltwise ops and sub support

* perf: vectorize normal rope

* perf: glu runs in parallel

* merge: manually merge saqib's work on kernel fixes

* perf: more vectorized RoPE

* perf: parallelize mul_mat_id

* perf: parallelize set_rows_f32

* perf: vectorize softmax

* feat: support kernel fusion and fuse RMS_NORM + MUL

* fix: mostly resolve test-backend-ops failure in SOFT_MAX and ROPE

* fix: bump max rope dims for gemma

* feat: GeGLU and SCALE support to fully offload Gemma

* perf: faster device memset

* feat: get_rows supporting Q4_K and avoid cont cache coherent issues

* better F32 MM

* feat: NORM for ET backend

* feat: SQR for ET backend

* feat: UNARY on ET

* feat: el_map support broadcasting for ET

* feat: SUM_ROWS in ET backend

* feat: more ops in ET backend

* feat: WKV* operators in ET backend

* perf: parallelize operators across cacheline instead of row

* perf: parallelize get_rows on cacheline

* wip: baseline FlashAttention for ET backend

* wip: enough FA and CPY f32->f16 to run llama 3.1 fully offloaded with FA on

* feat: f16 x f16 -> f32 MM using matrix engine

* wip: f16 FlashAttention using matrix engine

* wip: clean up

* feat: barriers

* perf: optimize FA_F16 in ET

* perf: vectorize pack_k_for_transpose16

* perf: prefetch next loop matrix tile

* perf: FlashAttention 2nd MM uses TensorFMA and optimizations

* cleanup: flashattention reorg

* perf: optimizations and fixes

* feat: L2SCP API and make FlashAttention support DV = 256 for gemma

* perf: parallelize norms beyond single row

* feat: GATED_DELTA_NET support and relaxed L2_NORM requirment

* feat: loosen RMS_NORM, NORM, ROPE contingous req too

* feat: repeat supports brocasting on dim 0 and loosen cont check

* feat: FILL and DIAG operator

* feat: loosen UNARY support chcek

* feat: TRI support

* feat: SOLVE_TRI support

* feat: basic SET support

* feat: loosen CONT req

* perf: fp16_to_fp32 use ASM

* feat: IMROPE support

* feat: PAD support

* feat: global barrier

* fix: view must live on the same backend as backing tensor

* feat: relax CONCAT in ET backend

* feat: dead simple CUMSUM implementation

* feat: basic SSM_CONV support

* feat: loosen CONCAT req

* feat: relax GATED_DELTA_NET and add SET support proper

* cleanup: cleanup LCM math

* feat: SWIGLU single input

* feat: SSM_SCAN support

* feat: el_map supports non aligned tensors in best effort

* feat: basic GROUP_NORM support

* feat: loosen MUL_MAT capablities slightly

* feat: loosen MUL_MAT and GET_ROWS and add IM2COL

* feat: special case for softmax 1x1x1x1

* feat: loosen SOFT_MAX req in ET backend

* fix: el_map unaligned acse fixes

* perf: optimize zero_acc_vec in flash_attn_ext_f16_me

* perf: use hart 1 for packing in MM and FA for FP16

* feat: kernel semaphore

* perf: better instruction sequence in FlashAttention

* fix: gated_delta_net with proper masking

* perf: better parallelization for GATED_DELTA_NET

* perf: parallelize SSM_CONV over nr

* perf: vectorize SSM_CONV

* perf: optimize MUL_MAT for q8

* feat: support Gemma 4

* fix: support multi-device

* feat: broader GLU support

* feat: unary ops supports view

* fix: repair fp16 MM using matrix engine

* perf: handle large N GEMV better

* perf: better q8_0 MM

* perf: better set_rows

* add back deleted files

* fix: repair after merge

* feat: POC version of uberkernel

* feat: RMS_NORM in uberkernel

* feat: add more kernels into usage

* chore: clean up uberkernel compilation

* perf: faster flash attention

* perf: opt flash attention for large seq length

* feat: loosen op bounds. clamp and mean support

* perf: vectorize ssm_scan

* perf: slightly faster FA

* perf: FlashAttention parallel MM and load

* perf: fuse Q8 MM and ADD

* feat: basic conv kernel for ET

* softMAx_test

* set_rows_f32

* get_rows and cont

* testing

* set_rows_exp

* Junk addition

* Narrowing the issue

* Update flash_attn_ext_f16_me.c

Focusing FA_ext_f16_me

* test

* Eviction updated

* Detailed cache eviction debug

* mulmat

* removeal of `BUILD_FOR_UBERKERNEL` flag

* cleaning...

* fix: balance FCC0 count

* feat: implement mul_mat and mul_mat_id for Q4_0 type

* optimize uberkernel plan upload

* add mul_mat q4 into uberkernel

* enable gating flush to just uberkernel

* update docs for ET

* update op support for ET

* et-backend: optimize Q4_0 and Q8_0 mul_mat_id row accumulations

* et-backend: specialize mul_mat_id kernels for Q4_0 and Q8_0

* et-backend: fix RoPE YaRN corr_dim formula and handle degenerate inputs

* test-backend-ops: add DeepSeek-V2-Lite RoPE test coverage

* et-backend: add Q4_0 mul_mat matrix-engine kernel using TensorFMA32

* et-backend: vectorize Q4_0 matrix-engine dequantization

* et-backend: support hybrid matrix/vector engine execution for Q4_0 mul_mat tail

* et-backend: run partial-N tiles on matrix engine for Q4_0 mul_mat

* et-backend: route Q4_0 mul_mat N < 53 to vecdot for better prefill latency

* Update uberkernel.c

* Update unary_f32.c

* gemma 4

* bisect gemma4: enable scale_f32 only

* bisect gemma4: +rms_norm_f32

* bisect gemma4: +rms_norm_mul_f32

* bisect gemma4: disable rms_norm_mul_f32 -- BREAKS OUTPUT

* bisect gemma4: +rope_f32 (skip rms_norm_mul)

* bisect gemma4: +el_map_f32

* bisect gemma4: +softmax_f32

* bisect gemma4: +get_rows_f32

* bisect gemma4: +glu_f32

* bisect gemma4: +mul_mat_f32 +mul_mat_f32_matrix_engine

* bisect gemma4: +mul_mat_f16 +mul_mat_f16_matrix_engine

* bisect gemma4: +mul_mat_Q8_0 +mul_mat_Q4_0

* bisect gemma4: +flash_attn_ext_f32 +flash_attn_ext_f16_me

* bisect gemma4: +mul_mat_id_f32

* bisect gemma4: +sum_rows_f32

* bisect gemma4: +cont_f16

* bisect gemma4: +fill_f32

* bisect gemma4: +unary_f32 (all ops re-enabled except rms_norm_mul)

* Update rms_norm_mul_f32.c

* bisect2 gemma4 n64: +scale_f32 only

* bisect2 gemma4 n64: +rms_norm_f32 +rope_f32

* bisect2 gemma4 n64: +rms_norm_mul_f32 (with ET_UBERKERNEL eviction fix)

* bisect2 gemma4 n64: +el_map +get_rows +glu +softmax (skip rms_norm_mul)

* bisect2 gemma4 n64: all ops enabled except rms_norm_mul

* bisect2 n64: test unary+cont+fill+sum_rows (no mul_mat/flash_attn)

* bisect2 n64: +mul_mat_f32 +mul_mat_f32_matrix_engine

* bisect2 n64: +mul_mat_f16 +mul_mat_f16_matrix_engine

* bisect2 n64: +mul_mat_Q8_0 +mul_mat_Q4_0

* bisect2 n64: +mul_mat_Q8_0 only (disable Q4_0)

* bisect2 n64: +mul_mat_Q4_0 only (Q8_0 breaks)

* bisect2 n64: +mul_mat_id +flash_attn_ext (skip Q8_0)

* run-3: matmul + rms_norm_mul

* run-4

* Revert "run-4"

* run5

* changes after cleanup

* cleanup before upstream

* restrict changes into ET backend

* move kernel embedding from Python to CMake

* move uberkernel gen into CMake

* apply clang format

* update CMake style

* update to match C and C++ style

* use source ggml and quant headers instead of ET's

* MROPE support

* absorb view ops into same branch as none

* fix bad rebase

* add marty1885 to codeowners

* oops

* remove redundant newline

* fix CI editor warnings

---------

Co-authored-by: Vidas <vidas@nuolat.lt>
Co-authored-by: Gianluca Guida <glguida@tlbflush.org>
Co-authored-by: Gianluca Guida <gianluca@nekko.ai>
Co-authored-by: ubergarm <leimgrub@gmail.com>
Co-authored-by: SaqibAkram-10xE <saqib.akram@10xengineers.ai>
Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai>
 2026-07-10 12:38:34 +08:00
Pasha Khosravi
bec4772f6a
Add Q2_0 quantization: type definition and CPU backend (#24448) 2026-07-07 12:05:47 -07:00
Johannes Gäßler
74976e1aef
CUDA: remove -sm row, refactor cuBLAS (#24216) 
* CUDA: remove -sm row, refactor cuBLAS

* fix CDNA + BF16 logic

* fix bad return

* fix src0 strides, contiguous requirements

* fix GGML_CUDA_FORCE_CUBLAS

* fix casts to BF16
 2026-07-06 20:04:53 +02:00
David Spruill
e9fb3b3fc0
sycl : support --split-mode tensor (#24152) 
* Sycl tp stage1 (#1)

* SYCL: tensor parallelism (--split-mode tensor) for dual-GPU

Adds the comm_init/comm_free/comm_allreduce_tensor trio that the
meta-backend queries via get_proc_address to enable backend-specific
all-reduce, mirroring the pattern used by ggml-cuda.cu.

For N=2 (the common dual-GPU case) implements a degenerate ring
all-reduce with two size-branched paths:

  * Small (nelem < 32768): FP32 direct memcpy + per-device ADD kernel
    chained via depends_on(memcpy_event). 4 SYCL submissions/call.

  * Large (nelem >= 32768): BF16-compressed. Each device compresses
    FP32 -> BF16 in a local outbox, cross-device memcpys to the peer's
    inbox (HALF the PCIe bytes), then decompresses + adds into the
    local FP32 partial. 6 SYCL submissions/call but PCIe bytes halved
    -- wins for any tensor where PCIe dominates kernel time.

Threshold and BF16 path pattern mirror the CUDA NCCL allreduce.

Storage: ONE persistent uint8_t buffer per device, 4 * nelem bytes
(matches both path layouts: FP32 nelem floats; BF16 outbox+inbox =
2 * nelem uint16_t each). Single alloc+free per device keeps the
SYCL pool's strict-LIFO invariant trivial.

Initial impl handles N=2 FP32 contiguous tensors. Other cases return
false, causing the meta-backend to use its generic butterfly fallback.

Per-call sync is intentionally omitted. SYCL in-order queue semantics
ensure that the meta-backend's next compute on the same per-device
queue waits for our final ADD, and the next allreduce's first op on
the same persistent buffer waits via the same queue. Only comm_free
does an explicit final wait.

OneCCL is NOT used: OneCCL 2021.17 hardcodes single-device-per-process
in communicator_impl.hpp:47 (condition devices.size() == 1), which is
incompatible with llama.cpp's single-process multi-GPU model.

Measured on dual Intel Arc Pro B70 (NEO 26.05.x, oneAPI 2025.3 +
DPC++ nightly):

  Llama-3.3-70B Q4_K_M, -sm tensor -fa 1 -ctk f16 -ctv f16:
    pp512 = 377.08 t/s  (vs 313.65 layer mode = +20.2%)
    tg128 = 17.40 t/s   (vs   9.74 layer mode = +78.6%)

  Qwen3-Coder-Next-80B-A3B Q3_K_M (MoE):
    pp512 = 216.56 t/s  (vs 156.58 meta-backend butterfly = +38.3%)
    tg128 = 17.60 t/s   (vs  14.31 meta-backend butterfly = +23.0%)

  Qwen3-4B Q4_K_M:
    pp64  = 984.51 t/s, tg16 = 49.29 t/s

Llama-3.3-70B in SYCL TP now comfortably beats production layer mode
on both prefill and decode. Coder-Next-80B-A3B (MoE) also wins on
both — the BF16 path is what unlocks the many-medium-allreduces
prefill pattern.

Build/CMake: no changes. No new dependencies. ~210 lines added across
ggml-sycl.h and ggml-sycl.cpp.

* Fix comments

* documentation update to address PR feedback

* Bring over my device-to-device memcpy chagnes

* move the dev2dev_memcpy calls to the upstream 7-parameter variety

* Fix a typo and remove a trailing whitespace
 2026-06-25 08:35:21 +03:00
Concedo
be1750a3b8 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/cuda.Dockerfile
#	.github/workflows/build-sycl.yml
#	.github/workflows/release.yml
#	ggml/CMakeLists.txt
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp/gated-delta-net-ops.c
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/cvt.cl
#	ggml/src/ggml-opencl/kernels/gated_delta_net.cl
#	ggml/src/ggml-sycl/gated_delta_net.cpp
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	ggml/src/ggml-webgpu/wgsl-shaders/gated_delta_net.wgsl
#	scripts/sync-ggml.last
#	tests/test-backend-ops.cpp
#	tests/test-llama-archs.cpp
 2026-06-12 15:39:57 +08:00
Gaurav Garg
e95dae18d6
Remove padding and multiple D2D copies for MTP (#24086) 
* Make ggml_gated_delta_net take only the initial recurrent state (D, 1, n_seqs) and passes the snapshot count K as an op parameter instead of inferring it from state->ne[1].

Remove the padding hack and copy all emitted snapshots into the recurrent cache with a single strided ggml_cpy

* Make GDN changes in all backends. Address review comments.

* Fix CI build errors
 2026-06-10 23:21:16 +05:30
Concedo
afc8de2c6b Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/cpu.Dockerfile
#	.devops/cuda.Dockerfile
#	.devops/intel.Dockerfile
#	.devops/musa.Dockerfile
#	.devops/openvino.Dockerfile
#	.devops/rocm.Dockerfile
#	.devops/vulkan.Dockerfile
#	.devops/zendnn.Dockerfile
#	.github/workflows/build-webgpu.yml
#	.github/workflows/release.yml
#	ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	ggml/src/ggml-webgpu/wgsl-shaders/binary.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/concat.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_decls.tmpl
#	ggml/src/ggml-webgpu/wgsl-shaders/scale.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/unary.wgsl
#	tests/CMakeLists.txt
#	tests/test-backend-ops.cpp
#	tests/test-mtmd-c-api.c
#	tools/cli/cli.cpp
#	tools/mtmd/CMakeLists.txt
#	tools/server/README.md
 2026-06-10 17:21:05 +08:00
Pascal
26021699bc
ggml : add GGML_OP_COL2IM_1D (#24206) 
* cpu: add GGML_OP_COL2IM_1D

Add the overlap-add (scatter-add) step of a 1D transposed convolution.
A ConvTranspose1d factorizes as a GEMM followed by col2im: a weight
pre-permuted to [IC, K*OC] is contracted against the [IC, T_in] input
with mul_mat to produce a column matrix [K*OC, T_in], and col2im_1d
scatters those columns back into the [T_out, OC] signal, with
T_out = (T_in - 1)*s0 + K - 2*p0.

Keeping the contraction as a plain mul_mat leaves the heavy work on the
optimized (and quantizable) matmul kernels, so col2im_1d only does the
cheap overlap-add.

CPU uses a gather formulation parallelized over output channels,
supporting F32, F16 and BF16 with an F32 accumulator.

* tests: add backend coverage for GGML_OP_COL2IM_1D

Add test_col2im_1d next to the conv_transpose_1d cases, covering F32,
F16 and BF16 across eight geometries: the canonical kernel = 2*stride
DAC upsampling shape, overlap, no overlap, cropping (p0 = 1 and
p0 = stride/2), kernel < stride with zeroed gaps, kernel not a
multiple of stride, and a single column unfold.

Perf mode gets three real vocoder stage shapes reporting memory
bandwidth. max_nmse_err relaxes to 5e-4 for F16 and BF16.

* cpu: harden GGML_OP_COL2IM_1D

ggml_col2im_1d validates s0, oc, p0 and input contiguity at graph
build time, before the oc division, protecting every backend at once.
The kernel asserts the contiguity its flat indexing assumes and its
doc states the full output length including the crop term.

The kernel parallelizes over the time axis: the split stays balanced
down to OC = 1, where the previous channel split was single threaded.
Values are bit identical on the three real vocoder chains, two out of
three improve.

* tests: extend the GGML_OP_COL2IM_1D grid

The eval grid grows to eleven geometries: OC = 1 (mono output stage),
K = 1 with stride > 1 (sparse scatter, every gap position zeroed) and
a crop down to T_out = 2 where all the gather bounds act at once.

* tests: add col2im_1d equivalence test

tests/test-col2im-1d.cpp proves mul_mat + col2im_1d matches the
native ggml_conv_transpose_1d on the CPU backend, F32 bit exact, F16
and BF16 through casts of the column matrix. test-backend-ops cannot
cover this for a CPU only op since the CPU backend is its own
reference there.

* rpc: bump protocol patch version for GGML_OP_COL2IM_1D

GGML_OP_COUNT goes from 96 to 97 with the new op, which trips the
static_assert in ggml-rpc.h. Bump RPC_PROTO_PATCH_VERSION since the
op is appended and no existing op code shifts.
 2026-06-09 12:01:37 +03:00
Concedo
153214c66a Merge commit '5aba5364d9' into concedo_experimental 
# Conflicts:
#	.github/workflows/build-apple.yml
#	.github/workflows/build-cpu.yml
#	.github/workflows/build-openvino.yml
#	.github/workflows/build-rpc.yml
#	.github/workflows/build-self-hosted.yml
#	.github/workflows/build-vulkan.yml
#	.github/workflows/build-webgpu.yml
#	.github/workflows/server.yml
#	SECURITY.md
#	docs/backend/SYCL.md
#	docs/backend/ZenDNN.md
#	docs/build.md
#	docs/ops.md
#	docs/ops/SYCL.csv
#	ggml/src/ggml-sycl/common.hpp
#	ggml/src/ggml-sycl/convert.cpp
#	ggml/src/ggml-sycl/dequantize.hpp
#	ggml/src/ggml-sycl/dmmv.cpp
#	ggml/src/ggml-sycl/fattn-common.hpp
#	ggml/src/ggml-sycl/getrows.cpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-sycl/mmvq.cpp
#	ggml/src/ggml-sycl/quants.hpp
#	ggml/src/ggml-sycl/vecdotq.hpp
#	tests/test-reasoning-budget.cpp
#	tools/llama-bench/README.md
#	tools/llama-bench/llama-bench.cpp
#	tools/mtmd/CMakeLists.txt
 2026-06-02 18:30:40 +08:00
Johannes Gäßler
8e6fff84de
TP: quantized KV cache support (#23792) 
* TP: quantized KV cache support

* fix partial view

* remove overly strict assert
 2026-06-01 12:30:10 +02:00
Concedo
9fbc0cd7dc Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.github/workflows/build-and-test-snapdragon.yml
#	.github/workflows/build-android.yml
#	.github/workflows/build-cache.yml
#	.github/workflows/build-cann.yml
#	.github/workflows/build-cross.yml
#	.github/workflows/build-openvino.yml
#	.github/workflows/build-riscv.yml
#	.github/workflows/build-sanitize.yml
#	.github/workflows/build-self-hosted.yml
#	.github/workflows/build-sycl.yml
#	.github/workflows/build-vulkan.yml
#	.github/workflows/build.yml
#	.github/workflows/release.yml
#	.github/workflows/server-sanitize.yml
#	ci/run.sh
#	docs/backend/SYCL.md
#	docs/backend/snapdragon/README.md
#	ggml/CMakeLists.txt
#	ggml/src/ggml-cuda/ggml-cuda.cu
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/cpy-ops.c
#	ggml/src/ggml-hexagon/htp/get-rows-ops.c
#	ggml/src/ggml-hexagon/htp/hmx-flash-attn-ops.c
#	ggml/src/ggml-hexagon/htp/htp-ctx.h
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/hvx-utils.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-hexagon/htp/rope-ops.c
#	ggml/src/ggml-hexagon/htp/set-rows-ops.c
#	ggml/src/ggml-hexagon/htp/unary-ops.c
#	ggml/src/ggml-sycl/common.hpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	ggml/src/ggml-webgpu/wgsl-shaders/common_decls.tmpl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec_acc.tmpl
#	scripts/sync-ggml.last
#	tests/gguf-model-data.cpp
#	tests/test-backend-ops.cpp
 2026-05-27 00:26:08 +08:00
Concedo
9204f78926 Merge commit 'ae251b5ff2' into concedo_experimental 
# Conflicts:
#	.github/actions/linux-setup-spacemit/action.yml
#	.github/actions/unarchive-tar/action.yml
#	.github/workflows/build-android.yml
#	.github/workflows/build-cmake-pkg.yml
#	.github/workflows/build-cross.yml
#	.github/workflows/build-self-hosted.yml
#	.github/workflows/build.yml
#	.github/workflows/check-vendor.yml
#	.github/workflows/code-style.yml
#	.github/workflows/editorconfig.yml
#	.github/workflows/pre-tokenizer-hashes.yml
#	.github/workflows/python-check-requirements.yml
#	.github/workflows/python-lint.yml
#	.github/workflows/python-type-check.yml
#	.github/workflows/server-self-hosted.yml
#	.github/workflows/ui-build.yml
#	.github/workflows/ui.yml
#	.github/workflows/update-ops-docs.yml
#	ci/run.sh
#	docs/build-riscv64-spacemit.md
#	examples/convert_legacy_llama.py
#	ggml/cmake/ggml-config.cmake.in
#	ggml/src/CMakeLists.txt
#	ggml/src/ggml-cpu/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	scripts/sync_vendor.py
#	tests/test-chat-auto-parser.cpp
#	tests/test-chat.cpp
#	tests/test-gguf.cpp
#	tools/cli/README.md
#	tools/perplexity/perplexity.cpp
#	tools/server/README.md
 2026-05-26 22:01:57 +08:00
Ori Pekelman
b251f74f49 ggml.h: correct ggml_silu_back arg docstring (a=dy, b=x) (ggml/1500) 2026-05-25 12:38:01 +03:00
Gilad S.
66efd13375
ggml: gguf_init_from_callback and gguf_init_from_buffer (#22341) 
* ggml: implement `gguf_init_from_buffer`

* test: `gguf_init_from_buffer`

* fix: memory breakdown for a model loaded with `no_alloc` from a file is consistent with being loaded from a buffer

* fix: use `GGML_UNUSED`

Co-authored-by: Copilot <copilot@github.com>

* fix: remove `total_size` from `gguf_reader`

* fix: file offset calculation, rename `offset` to `data_offset`

Co-authored-by: Copilot <copilot@github.com>

* refactor: extract model loader bug fixes to another PR

* feat: add `gguf_init_from_callback`

* fix: always require a max expected size

* fix: change `gguf_reader_callback_t`'s `output` type to `void *`, change `max_expected_size` and offsets to `uint64_t`

* fix: harden against offset overflow in buffer read

* fix: remove seek behavior from the callback

* feat: `max_chunk_read == 0` means `SIZE_MAX`

* fix: seeking in a gguf file with no tensors

---------

Co-authored-by: Copilot <copilot@github.com>
 2026-05-25 11:33:29 +02:00
Johannes Gäßler
fff63b5108
TP: fix entirely zero-sized slices per device (#23525) 2026-05-24 08:19:33 +02:00
Concedo
1e828ccabf Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	common/common.cpp
#	ggml/CMakeLists.txt
#	scripts/sync-ggml.last
#	scripts/sync_vendor.py
#	src/llama-context.cpp
#	tests/CMakeLists.txt
#	tests/test-backend-ops.cpp
#	tools/cli/README.md
#	tools/completion/README.md
#	tools/server/README.md
 2026-05-17 11:26:18 +08:00
Aman Gupta
255582687b
llama + spec: MTP Support (#22673) 
* spec: support MTP

* fix batch size

* rename files

* cont : simplify (#7)

* MTP: clean-up (#9)

* MTP: clean-up

* review: use llama_context_type instead of llama_graph_type

* review: remove llama_model_has_mtp

* review: fix convert issues

* convert: fix pycheck

* review: formatting

* use `mtp-` for identifying mtp models

* convert: fix mtp conversion

* mtp -> draft-mtp

* remove unused llama_arch

* add need_embd in speculative

* llama: allow partial seq_rm for GDN models for speculative decoding

Currently speculative checkpoint needs to restart from a checkpoint
after some draft tokens are not accepted, this leads to some wastage in
running the target again. This PR adds the ability to rollback upto
`draft_max` by storing the GDN intermediates.

* fix pending state

* vulkan: add GDN partial rollback

* meta: extend check to axis 1

* metal: add GDN partial rollback

Extend the gated delta net kernel to store intermediate states for
partial rollback support on the Metal backend.

- Add K (snapshot slot count) as a function constant
- Read input state from slot 0 of the 3D state tensor
- Write intermediate states to different slots during token loop
- For K=1, maintain backward-compatible single-slot behavior

Ref: 8c05923630

Assisted-by: llama.cpp:local pi

* delta_net_base: use ggml_pad instead of new_tensor

* review: add need_rs_seq

* review: rename part_bounded to n_rs

* review: deslop comments

* review: rename, add asserts

* server : adjust checkpoint logic (#11)

* server : adjust checkpoint logic

* cont : rm asserts

* server-context: fix early exit

* spec : fix compatibility with n-gram and add TODOs (#13)

* metal : cleanup

* llama : fix faulty bitwise check in recurrent memory

* server : disable RS-based MTP in combination with other spec types

* spec : add TODOs

* cont : fix comment

* cont : update comment

* common : fix logic for ngram + mtp compat

* llama-memory: enable checkpointing with partial rollback

* cont: add test-case for loading into a dirty ctx

* llama-memory-recurrent: clear rs_idx in clear

* download: fix mtp path

* llama-arch: fix enorm op

* docs: update docs

* conversion: fix type annotations

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-05-16 20:06:23 +08:00
Concedo
9b0b36b5ef Merge commit '66001722aa' into concedo_experimental 
# Conflicts:
#	README.md
#	docs/ops.md
#	docs/ops/SYCL.csv
#	examples/sycl/start-svr.sh
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/htp-ctx.h
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-hexagon/htp/unary-ops.c
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/cvt.cl
#	ggml/src/ggml-sycl/gated_delta_net.hpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-sycl/pad.cpp
#	ggml/src/ggml-sycl/ssm_conv.cpp
#	tests/test-backend-ops.cpp
#	tests/test-reasoning-budget.cpp
#	tools/server/README.md
#	tools/server/webui/src/lib/constants/settings-config.ts
 2026-05-11 15:40:10 +08:00
Johannes Gäßler
a8fd165fec
CUDA: lower-case PCI bus id, standardize for ggml (#22820) 2026-05-08 10:09:38 +02:00
Concedo
9e9497f0cc Merge remote-tracking branch 'origin/upstream' into concedo_experimental 
# Conflicts:
#	examples/save-load-state/save-load-state.cpp
#	ggml/CMakeLists.txt
#	ggml/src/ggml-cpu/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/hmx-matmul-ops.c
#	ggml/src/ggml-hexagon/htp/matmul-ops.c
#	ggml/src/ggml-opencl/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-opencl/kernels/gemm_noshuffle_q4_0_f32.cl
#	ggml/src/ggml-opencl/kernels/gemm_noshuffle_q8_0_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_0_f32.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q4_0_f32_spec.cl
#	ggml/src/ggml-opencl/kernels/gemv_noshuffle_q8_0_f32.cl
#	ggml/src/ggml-rpc/ggml-rpc.cpp
#	scripts/sync-ggml.last
#	scripts/sync_vendor.py
#	src/llama-graph.cpp
#	tests/test-backend-ops.cpp
#	tests/test-state-restore-fragmented.cpp
 2026-05-06 21:20:06 +08:00
Ismail
a817a22bc6
ggml : implement fast walsh-hadamard transform for kv rotation (#21352) (#22631) 2026-05-05 10:05:05 +08:00
Concedo
768527b031 Merge commit '1e796eb41f' into concedo_experimental 
# Conflicts:
#	.devops/nix/package.nix
#	.github/workflows/build-riscv.yml
#	.github/workflows/build-vulkan.yml
#	.github/workflows/build.yml
#	docs/backend/SYCL.md
#	docs/build.md
#	docs/development/HOWTO-add-model.md
#	embd_res/templates/Reka-Edge.jinja
#	ggml/CMakeLists.txt
#	ggml/src/ggml-rpc/CMakeLists.txt
#	ggml/src/ggml-rpc/ggml-rpc.cpp
#	ggml/src/ggml-sycl/CMakeLists.txt
#	ggml/src/ggml-sycl/convert.cpp
#	ggml/src/ggml-sycl/dequantize.hpp
#	ggml/src/ggml-sycl/dmmv.cpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	ggml/src/ggml-webgpu/wgsl-shaders/common_decls.tmpl
#	ggml/src/ggml-webgpu/wgsl-shaders/get_rows.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_decls.tmpl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_id.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_reg_tile.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_subgroup_matrix.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_vec.wgsl
#	ggml/src/ggml-webgpu/wgsl-shaders/unary.wgsl
#	tests/test-chat.cpp
#	tools/rpc/README.md
 2026-04-17 21:47:29 +08:00
Johannes Gäßler
014dca49d6
CUDA: manage NCCL communicators in context (#21891) 
* CUDA: manage NCCL communicators in context

* add check that all backends are CUDA

* remove unused vector, limit init to > 1 GPUs

* fix warnings

* fix cuda device, cache allreduce
 2026-04-15 15:58:40 +02:00
Valeriy Dubov
adb541a6ad
rpc : add native RDMA transport for RPC backend (RoCEv2) (#20590) 2026-04-15 16:44:02 +03:00
Xuan-Son Nguyen
80d8770804
docs: more extensive RoPE documentation [no ci] (#21953) 
* more extensive ggml_rope documentation

* add more docs

* nits
 2026-04-15 14:45:16 +02:00
Concedo
ac29e6f0c0 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.devops/vulkan.Dockerfile
#	.github/workflows/build-self-hosted.yml
#	.github/workflows/build.yml
#	.github/workflows/release.yml
#	.github/workflows/server-self-hosted.yml
#	docs/build.md
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/hex-utils.h
#	ggml/src/ggml-hexagon/htp/hmx-matmul-ops.c
#	ggml/src/ggml-hexagon/htp/hmx-utils.h
#	ggml/src/ggml-hexagon/htp/htp-ctx.h
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/hvx-base.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	tests/test-backend-ops.cpp
#	tests/test-mtmd-c-api.c
 2026-04-15 15:15:19 +08:00
Xuan-Son Nguyen
fae3a28070
ggml : remove ggml-ext.h (#21869) 
* ggml: correct placement of ggml-ext.h

* ggml : remove ggml-ext.h

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-04-14 17:32:58 +03:00
Concedo
a165a73120 Merge commit 'd6f3030047' into concedo_experimental 
# Conflicts:
#	examples/model-conversion/scripts/causal/run-casual-gen-embeddings-org.py
#	examples/model-conversion/scripts/utils/semantic_check.py
#	ggml/CMakeLists.txt
#	ggml/src/CMakeLists.txt
#	ggml/src/ggml-cann/ggml-cann.cpp
#	ggml/src/ggml-cpu/amx/amx.cpp
#	ggml/src/ggml-cuda/CMakeLists.txt
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hip/CMakeLists.txt
#	ggml/src/ggml-opencl/ggml-opencl.cpp
#	ggml/src/ggml-openvino/ggml-openvino.cpp
#	ggml/src/ggml-rpc/ggml-rpc.cpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-virtgpu/ggml-backend-buffer.cpp
#	ggml/src/ggml-virtgpu/ggml-backend.cpp
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	ggml/src/ggml-zdnn/ggml-zdnn.cpp
#	ggml/src/ggml-zendnn/ggml-zendnn.cpp
#	pyproject.toml
#	requirements/requirements-convert_legacy_llama.txt
#	requirements/requirements-tool_bench.txt
#	src/llama-model.cpp
#	src/llama.cpp
#	tests/test-llama-archs.cpp
#	tests/test-tokenizer-0.py
#	tests/test-tokenizer-random.py
#	tools/llama-bench/llama-bench.cpp
#	tools/perplexity/perplexity.cpp
 2026-04-11 11:10:55 +08:00
Johannes Gäßler
d6f3030047
ggml: backend-agnostic tensor parallelism (experimental) (#19378) 
* ggml: backend-agnostic tensor parallelism

* support for GPT-OSS, Qwen 3 MoE

* partial Vulkan fix

* add support for 4/8 GPUs

* unconditional peer access

* re-use buffers + ggml contexts

* fix output pattern

* NCCL support

* GGML: HIP: add RCCL support

* Remove shfl and AllReduce from backend interface

* move allocation workaround out of ggml-alloc.c

* 2d tensor set/get support

* Fix the seg fault without NCCL

* Apply suggestion from JohannesGaessler

* support for tensor dims % n_devs != 0

* fix view_offs scaling

* arbitrary num. of GPUs/tensor split

* fix compilation

* better granularity estimate

* Support device-specific host buffer types if all underlying backends expose the same type. This allows using pinned memory instead of pageable memory for CUDA.

Fix compilation errors.

* partial Qwen 3 Next support

* Fix qwen3 30b (#8)

* Fix crash with Qwen-30B-A3B Q4_0

Qwen-30B-A3B Q4_0 has an intermediate dimension of 768. Using a granularity of 256 forces an uneven split between GPUs, which is not supported by the current implementation.

* Decide block size based on tensor quantization type

* Fix crashes due to KV cache serialization (#9)

KV cache serialization requires non-zero offsets on the tensor. Add support in the meta backend to set/get a tensor with a non-zero offset.

* metal : fix build (#7)

* static memory allocations, fix usage count

* fix tensor granularity

* more even memory distribution

* use BF16 for allreduce

* rebase fixup

* better error message for unsupported architectures

* Fix device mismatch during scatter of allReduce. (#11)

There is a mismatch between the dst buffer device and the backend device, causing the use of sync copies

* Enable the previous allreduce implementation. It is better in both perf and stability (#12)

* delay AllReduce for Moe for less I/O

* build : clean-up compile warnings

* backend : move most of the meta backend API to ggml-backend-impl.h

* cont : hide unused public API in the implementation

* llama : use llama_device + remove ggml_backend_dev_is_meta()

* ggml-backend : remove unused alloc include

* minor : remove regex include

* ggml : introduce ggml-ext.h for staging new APIs

* rebase fixup

* fix tests

* llama : more robust logic for determining Meta devices (#16)

* llama : more robust logic for determining Meta devices

* cont : fix devs size check

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* cont : fix log type

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* disable roundtrip for meta backend

* fix arch selection

* Qwen 3.5 support

* fix Gemma 4 MoE

* fix OpenVino, SYCL

* fix test-llama-archs for CPU-only builds

* Fix Qwen 3.5 MoE

* disable meta backend tests for WebGPU

* tests : filter CPU-based devices from the Meta backend tests (#17)

* meta : formatting, naming, indentation (#18)

* formatting : llama-model.cpp

* formatting : ggml-ext.h

* formatting : ggml-backend-meta.cpp

* meta : add TODO

* add documentation

* better error messages

* fix GPT-OSS

---------

Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz>
Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-04-09 16:42:19 +02:00
Concedo
5529748a01 Merge commit 'de1aa6fa73' into concedo_experimental 
# Conflicts:
#	docs/build.md
#	docs/ops.md
#	docs/ops/WebGPU.csv
#	ggml/src/ggml-sycl/dequantize.hpp
#	ggml/src/ggml-sycl/dmmv.cpp
#	ggml/src/ggml-sycl/ggml-sycl.cpp
#	ggml/src/ggml-sycl/mmvq.cpp
#	ggml/src/ggml-sycl/quants.hpp
#	ggml/src/ggml-sycl/vecdotq.hpp
#	ggml/src/ggml-webgpu/ggml-webgpu-shader-lib.hpp
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	ggml/src/ggml-webgpu/wgsl-shaders/mul_mat_decls.tmpl
#	tests/test-backend-ops.cpp
#	tests/test-quantize-fns.cpp
 2026-04-09 17:16:33 +08:00
Georgi Gerganov
22fc79134e
ggml : deprecate GGML_OP_ADD1 (#21363) 
* ggml : deprecate GGML_OP_ADD1

* cont : remove tests

* cont : re-enable vulkan check
 2026-04-07 15:28:27 +03:00
Pasha Khosravi
2e1f0a889e
ggml: add Q1_0 1-bit quantization support (CPU) (#21273) 
* ggml: add Q1_0 and Q1_0_g128 1-bit quantization support (CPU)

* add generic fallback for x86

* remove Q1_0 (group size 32)

* rename Q1_0_g128 => Q1_0

* fix Q1_0 LlamaFileType Enum

* Fix trailing spaces; add generic fallback for othre backends

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* fix /r/n spacing + arch-fallback

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
 2026-04-06 20:55:21 +02:00
Concedo
3ec6381123 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	.github/workflows/build-self-hosted.yml
#	.github/workflows/build.yml
#	.github/workflows/copilot-setup-steps.yml
#	.github/workflows/gguf-publish.yml
#	ci/run.sh
#	docs/backend/OPENVINO.md
#	examples/llama.android/lib/src/main/cpp/ai_chat.cpp
#	ggml/src/ggml-sycl/add-id.cpp
#	requirements/requirements-pydantic.txt
#	tests/test-gguf.cpp
#	tests/test-jinja.cpp
#	tests/test-llama-archs.cpp
#	tools/gguf-split/README.md
#	tools/llama-bench/llama-bench.cpp
 2026-03-28 01:18:20 +08:00
Johannes Gäßler
36dafba5c4
llama: fix llama-model-saver (#20503) 
* llama : add fd-based model loading via llama_model_load_from_fd

* llama : address review feedback for fd-based model loading

* llama : use FILE pointer instead of fd in public API

* llama : use FILE pointer consistently, address review feedback

* fixup

* fix tensor names

* fix llama-model-saver

* roundtrip tests

* fixup

* refactor tests

* fix prints

* fix model saving

* fix CI, disable Chameleon

* print seed

---------

Co-authored-by: Siddhesh2377 <siddheshsonar2377@gmail.com>
 2026-03-25 12:53:16 +02:00
Concedo
48f914e374 Merge branch 'upstream' into concedo_experimental 
# Conflicts:
#	ci/run.sh
#	ggml/CMakeLists.txt
#	ggml/src/ggml-cpu/arch/riscv/repack.cpp
#	ggml/src/ggml-cpu/arch/x86/repack.cpp
#	ggml/src/ggml-cpu/repack.cpp
#	ggml/src/ggml-hexagon/ggml-hexagon.cpp
#	ggml/src/ggml-hexagon/htp/CMakeLists.txt
#	ggml/src/ggml-hexagon/htp/htp-msg.h
#	ggml/src/ggml-hexagon/htp/htp-ops.h
#	ggml/src/ggml-hexagon/htp/hvx-base.h
#	ggml/src/ggml-hexagon/htp/hvx-exp.h
#	ggml/src/ggml-hexagon/htp/hvx-sigmoid.h
#	ggml/src/ggml-hexagon/htp/main.c
#	ggml/src/ggml-hexagon/htp/softmax-ops.c
#	ggml/src/ggml-hexagon/htp/unary-ops.c
#	ggml/src/ggml-webgpu/ggml-webgpu.cpp
#	scripts/sync-ggml.last
#	tests/test-backend-sampler.cpp
#	tests/test-chat.cpp
#	tests/test-jinja.cpp
#	tools/cli/cli.cpp
 2026-03-19 02:23:06 +08:00
Georgi Gerganov
79187f2fb8 ggml : restore ggml_type_sizef() to aboid major version bump (ggml/1441) 2026-03-18 15:17:28 +02:00
Concedo
f3d2f58fa8 note: smartcache is broken for rnn currently 2026-03-15 11:31:47 +08:00