* CUDA: runtime GGML_CUDA_MMVQ_MAX to tune the mvq->MMQ decode crossover
Add a runtime override of the mul_mat_vec_q -> MMQ batch crossover
(default MMVQ_MAX_BATCH_SIZE). Lowering it routes batches above the
threshold from the CUDA-core vector kernel to the int8 MMQ tensor-core
path, which is faster once quantized decode becomes compute-bound at
B>1 (measured +23-41% at B=8 on RTX 5090 for Q4_K dense, no low-batch loss).
The value is parsed once and clamped to [1, MMVQ_MAX_BATCH_SIZE], since
mul_mat_vec_q asserts ncols_dst <= that; invalid input warns and falls
back to the default. The override is applied consistently in both the
mul_mat_vec_q and MUL_MAT_ID dispatch paths. Default behavior unchanged.
* Added Blackwell specific switch point, to reduce dependence on runtime env var.
* Add per-HW switch point values for DGX Spark and removing runtime env var
* Adding switch points for Ada, tested on RTX 4090
* Modifying DGX Spark numbers based on latest run and adding some comments and small functional changes relating to MoE
* Reverting an unnecessary conditional
* Update ggml/src/ggml-cuda/mmvq.cu
---------
Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* metal: dequantize q8_0 KV to f16 before flash attention
Add a preprocessing pass for GGML_OP_FLASH_ATTN_EXT on the Metal backend:
when the KV cache is quantized (Q8_0 for now), dequantize K and V into a
contiguous F16 scratch buffer and run the existing F16 flash attention
kernels on it, instead of the in-kernel dequantization path.
- new kernel kernel_flash_attn_ext_dequant_to_f16<block_t, QK, deq_t4x4>:
one thread per quant block (K then V), stride-aware so permuted KV is
supported; instantiated for Q8_0 (extending to Q4_0/Q4_1/Q5_0/Q5_1 is
one instantiation + one gate case)
- the gate is type-only: dequantize whenever the KV is quantized,
regardless of head sizes, GQA ratio or n_kv; the attention kernels
themselves are untouched
- the F16 copies live in the op's own scratch allocation
(ggml_metal_op_flash_attn_ext_extra_dequant_f16); the KV pad kernel
reads the dequantized buffers when the path is active
- the FA pipeline getters gain a use_f16_kv flag selecting the existing
f16 kernels and contiguous strides
- ref: https://github.com/ggml-org/llama.cpp/pull/25556
Verification (M2 Ultra):
- test-backend-ops test -o FLASH_ATTN_EXT: 4798/4798 pass, including the
new q8_0 eval cases (decode/prompt, permuted, sinks+ALiBi+softcap,
kv=113 pad path, kv=16384)
- llama-perplexity on Qwen2.5-0.5B with -ctk q8_0 -ctv q8_0 matches the
f16 KV reference (PPL 1.0008 vs 1.0008)
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : launch the FA KV dequant kernel separately for K and V
Simplify kernel_flash_attn_ext_dequant_to_f16: it now dequantizes a single
tensor (its own ne/nb and dst) with no is_v branching, and the op dispatches
it twice with the same pipeline - once for K and once for V. The kargs
struct shrinks to a single ne/nb set plus nblocks.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : dequantize q4_0, q4_1, q5_0 and q5_1 KV to f16 before flash attention
The dequant pass now covers all quantized KV types supported by the Metal
flash attention kernels. The dequant kernel, kargs, scratch allocation and
dispatch are type-generic, so each type is one kernel instantiation plus one
gate case.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : skip the redundant V dequant when V is a view of K
In MLA-based models, the V of the FA op is a view of K (the first ne20
elements of each K row); the dequantized V is then a view of the dequantized
K, so skip the second dequant dispatch, do not reserve the V scratch region,
and let the pad and attention kernels read V from the K F16 buffer with K's
strides. The detection follows the CUDA backend:
V->view_src && (V->view_src == K || (V->view_src == K->view_src && V->view_offs == K->view_offs))
Also fix the FA pipeline getters: ns10/ns20 are function constants baked into
the kernels and must be the actual K/V row widths as seen by the kernel. The
dispatch now passes them explicitly (nb11_attn/nb10_attn, nb21_attn/nb20_attn)
instead of the getters assuming contiguous F16 KV (ns20 = dv), which was wrong
when V is read from K with K's row pitch (e.g. 576 vs 512).
New test cases: 576/512 q8_0 (MLA shape, V is a view of K) at kv=113 (KV pad),
nb=1 (vec) and nb=64 (non-vec).
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* test : remove backend-specific wording from test-backend-ops comments
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* pi : avoid backend mentions in test-backend-ops comments
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : rename the FA dequant_f16 identifiers to kv_f16
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : clean-up
* cont : remove TODO
* ggml: fix backend split scheduler race condition
splits without input were running concurrently with other splits, while potentially reusing memory the other split is accessing
* only sync when split has no inputs
* convert: fix get block count error for Nemotron
Signed-off-by: Rock Chen <rockchen.tw@gmail.com>
* fix this in NemotronHModel.__init__ instead.
This reverts commit ca689cbc8792ba69ea1fd5d8b3ae0485c47536ec.
---------
Signed-off-by: Rock Chen <rockchen.tw@gmail.com>
* provide static workspace for cuBLAS handles
* account for concurrent streams when using GGML_CUDA_GRAPH_OPT
* drop cublas_handle overloads and remove direct cublasSetStream calls
* Update ggml/src/ggml-cuda/common.cuh
---------
Co-authored-by: Oliver Simons <osimons@nvidia.com>
build_attn with the llm_graph_input_attn_k_iswa input was using the cached K
tensor itself as V. Create V as a view of K (the first v_cur->ne[0] elements
of each row), like the other K-only build_attn overloads.
The deepseek4 MTP call site now passes the kv tensor as v_cur.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* backend: propagate buffer usage in meta backend
* ggml-meta: make sure to call init_tensor for all new tensors
* meta: remove explicit check for meta backend in ggml_backend_meta_get_split_state
I can't seem to reproduce the original failure in the latest code.
* hexagon: fix FA HMX queue ordering in the pipelined path
* hexagon: double buffer D matrix, store diagonal tile only
* format code
* align the indentation
* opencl: port fused ssm_scan kernel (Mamba-2, d_state in {128, 256})
Fold the fused per-token SSM_SCAN recurrent step from opencl/gdn-qwen36-35b
onto the unified base. Previously SSM_SCAN fell back to CPU here; now scalar-A
Mamba-2 with d_state in {128,256}, all-f32, runs on GPU. Other shapes (incl.
Mamba-1 element-wise A) still fall back. test-backend-ops -o SSM_SCAN passes on
Adreno X2-90. opt-out via GGML_OPENCL_DISABLE_SSM_SCAN=1.
* opencl: cleanup
* opencl: require K == 1
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* ggml-cpu: gate __fp16 on __ARM_FP16_FORMAT_IEEE
__ARM_NEON only signals NEON availability. The __fp16 type also needs
the IEEE half format, implied on AArch64 but selected with
-mfp16-format=ieee on 32 bit Arm, where the compiler otherwise rejects
the type.
The guard keeps every toolchain that provides the type on the same code
and sends that one configuration to the generic lookup path.
* ggml-cpu: gate the NEON+FMA block on __ARM_FP16_FORMAT_IEEE
Both halves of the F16 section dereference __fp16, so armv7 with
neon-vfpv4 hits the same unknown type error. Without the IEEE
format the configuration now falls back to the scalar path.
Address review from @JonathanC-ARM
* vulkan : dequant q8_0 KV once in coopmat1
Assisted-by: Claude (Opus 4.8)
* vulkan : fall back instead of aborting when FA scratch exceeds maxStorageBufferRange
* vulkan : require KV-cache layout in FA dequant path
Assisted-by: Claude (Opus 4.8)
* vulkan : skip FA dequant path on coopmat2
Assisted-by: Claude (Opus 4.8)
* tests : add contiguously-allocated quant K/V FA tests
Assisted-by: Claude (Opus 4.8)
* vulkan : trim comments
* vulkan : tighten permutation checks for FA path
* vulkan : set prealloc_x_need_sync after the FA dispatch
* vulkan : exclude Intel Xe1 from FA dequant path
* feat(convert): Add conversion for GraniteSWAForCausalLM
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob, OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(llama): Add granite_swa support
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob, OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(conversion): Add conversion infra for rope_pattern array
NOTE: There is other work also targeting this, so this may be
removed depending on merge order.
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix(conversion): Fix SWA pattern logic and support for non-rope layers
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(conversion): Add support for GraniteMoeSWA
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Add llama_hparams::has_rope and arch constants
NOTE: This shadows the work done for Granite Speech
https://github.com/ggml-org/llama.cpp/pull/25107
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Add support for per-layer rope determination
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: Fix failing flake8 for extra newlines
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* test: Write out SLIDING_WINDOW_PATTERN in llama-model-saver
Branch: GraniteSWAForCausalLM
AI-usage: full (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix(convert): Fix missing registration for GraniteMoeSWAForCausalLM
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Load MoE params as optional
Branch: GraniteSWAForCausalLM
AI-usage: draft (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Handle MoE params in conversion
branch: GraniteSWAForCausalLM
AI-usage: full (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: Remove unnecessary newline
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Remove unnecessary tensor additions to GRANITE architecture
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Correctly handle naming for ffn gate inp
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Always default hparams.rope_pattern to 1s
This isn't strictly necessary, but it will allow other models to rely on
hparams.has_rope(il) without needting to prepopulate.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Move to has_rope for all granite model architectures
Now that we have a proper hparam for this, it's better to use it and not
require a hacky fallback in the hparam method itself.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: No hacky rope_finetuned fallback in has_rope
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Fully remove rope hparam filling in granitemoe
There are no granitemoe models that use NoPE (it's not actually used in the
layer building below), so this was just dead code.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Save out rope_pattern in model-saver
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Set hparams.rope_finetuned for round trip
Since the value is _read_ from rope_finetuned, we need to persist it when
the model is saved with the saver.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Code review cleanup
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* refactor: Keep gate/up fused for MoE path
Branch: GraniteSWAForCausalLM
AI-usage: full (Claude + Sonnet 5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Skip GRANITE_SWA in model saver
https://github.com/ggml-org/llama.cpp/pull/25505#discussion_r3773175651
Keeping is_swa_impl in the saver can break other models.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* add sliding window pattern for model in test
* style: Fix indentation
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Fix \r\n
Thanks Claude!
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Keep shared expert fused
Branch: GraniteSWAForCausalLM
AI-usage: full (Claude + Sonnet 5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: More indentation fixes
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* add params
* cpu kernel
* metal kernel
* add test backend ops
* gate other backends
* ggml: (cuda) support ggml_rope_set_offset (#27121)
* rm cuda supports_op guard, fix webgpu clang-format
* ggml: support ggml_rope_set_offset on vulkan (#27344)
* ggml: support ggml_rope_set_offset on vulkan
* remove inplace optimization
The route loads run ahead of the root layout script, so validateApiKey
read the settings store while it still held factory defaults and probed
/props without the stored key. initStores() now hands the same startup
promise to every caller and the chat loads await it before probing.
The one-time admin baseline no longer overwrites a key the user has
already set: on a first visit the config carries factory values only, so
a diverging key comes from the user and wins.
* vulkan: tiled transpose for 0<->2 permuted CONT
-ggml_vk_get_cpy_pipeline only routed to the tiled shared-memory transpose
shader when dim1 was the innermost dimension, i.e. ggml_transpose (a 0<->1
swap). A 0<->2 swap -- ggml_cont(ggml_permute(x, 2, 1, 0, 3)) -- fell back to
the generic per-element strided copy, whose source reads stride by ne0*ne1
elements: one cache line per lane.
-DeepSeek-V4's lightning indexer performs exactly that permute on a
[n_kv, n_tokens, n_head] tensor. On Vulkan/RADV gfx1151 it ran at ~1-9 GB/s of
a ~200 GB/s part and accounted for 43% of total prefill time.
-Add copy_transpose_02.comp, mirroring copy_transpose.comp but tiling over dst
dims (0, 2) with dims 1 and 3 as the batch, so reads walk src dim2 and writes
walk dst dim0 -- both contiguous. The selection condition additionally requires
a non-contiguous source and a contiguous destination so it cannot take cases
the contiguous-copy shader already handles.
-test-backend-ops only exercised ggml_transpose for CONT, so the strided path
was untested. Add test_cont_permute covering (2,1,0,3), (1,2,0,3) and (0,2,1,3)
over f32/f16 at tile-aligned, tile-unaligned and large shapes. The large shapes
are in the eval set rather than only in perf because perf mode does not verify
results.
-Measured on gfx1151, ne=[n_kv,64,64,1], perm=(2,1,0,3), f32:
n_kv=1024: 9.08 -> 579.85 GB/s
n_kv=1280: 20.03 -> 153.71 GB/s
n_kv=2048: 7.11 -> 91.68 GB/s
n_kv=2304: 16.24 -> 86.49 GB/s
-The ~2.2x penalty previously seen at power-of-two n_kv (destination-stride
aliasing) is gone. End to end, DeepSeek-V4-Flash IQ3_XXS prefill on a 9k-token
prompt goes from 56.33 t/s to 103.74 t/s (+84%).
-Note: at n_tokens=512 a single slow-path dispatch takes ~273 ms and looping it
in perf mode can trip the GPU watchdog, so the perf cases use n_tokens=64.
* tests: fold test_cont_permute into test_cont, add L2-exceeding perf shapes
Review feedback: test_cont gains a permute parameter ({0,0,0,0} = none),
matching test_mul_mat's pattern, and the separate struct is gone. Perf
adds [n_kv, 512, 64, 1] variants (~0.5 GB per run) that exceed GPU L2,
since the 64-token shapes fit in cache on large parts and read above
memory bandwidth.
* tests: trim perf-case comment to the two-line summary
* vulkan: trim comments on the 0<->2 transpose path
Drop the shader file header, the read/write block comments and the
rationale prose in the CONT test cases. Keep the tile-shape and
bank-conflict notes and the permute parameter documentation.
---------
Co-authored-by: Kevin Hopper <no-reply@maestro.press>
* gguf-py : add size guards to GGUFReader
Guard kv_count, tensor_count, string length, and array length
against crafted values that cause unbounded allocation or hangs.
Assisted-by: opencode
* gguf : validate tensor data section fits within file
When no_alloc=true, gguf_init_from_reader accepted files where the
tensor data section (computed from header claims) exceeded the remaining
file size. This allowed crafted GGUF files to pass validation while
having insufficient data, leading to OOB reads when the loader later
mapped tensor data from the file.
Assisted-by: opencode
* gguf-py : move size limits into gguf_reader.py
Per review feedback, the limits are not part of gguf.h but are
arbitrary limits defined in gguf.cpp, so define them locally in
the reader instead of exporting them from constants.
Assisted-by: opencode
* remove gguf.ccp changes
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* ui: Move stream lookup and replay fetches into ChatService
chatStore called fetch() directly for /v1/streams/lookup and the
/v1/stream replay. These now live next to the other stream-session
methods in ChatService, so services stay the only API I/O layer.
* ui: Move /models/sse feed reader into ModelsService
ModelsService.watchModelEvents owns the byte stream, reconnect loop
and SSE record parsing; modelsStore keeps only event routing and
state.
* ui: Extract conversation import/export into ConversationTransferService
The JSONL session format, ZIP archiving and browser downloads are
pure I/O with no store state, so they move out of
conversationsStore. The store keeps the DB orchestration
(bulkExportConversations, downloadConversation,
importConversationsData) and delegates the format work.
* ui: Consolidate active model resolution into modelsStore.activeModelId
The same resolution chain was duplicated in useChatScreenActiveModel,
ChatForm, ChatFormActionModels and contextStatsStore, with slight
drift in the single-model fallback. The canonical getter now lives in
modelsStore, and the shared last-assistant-model lookup moved to
utils as getConversationModel.
* ui: Initialize stores explicitly via initStores()
Store constructors and module-level side effects ran migrations and
localStorage reads in import order. Migrations rename and rewrite
localStorage keys, so a settings load racing ahead of them could
clobber migrated values. initStores() is called once from the root
layout and runs migrations first, then the stores that read
localStorage, then the conversations DB load.
* refactor: Constants for stream query params
* ui: Remove dead code from stores
- persisted() helper was exported but never used
- messageUpdateCallback / registerMessageUpdateCallback were never wired up
- conversationsStore.initialize() alias, single caller moved to init()
* ui: Merge device, theme and viewport into a single deviceStore
All three are reactive browser-environment signals, now exposed as one
class store: deviceStore.isMobile, deviceStore.isIOSDevice / isIOSSafari
/ isWKWebView / isStandalone and deviceStore.systemTheme.isDark. The
systemTheme name disambiguates the OS preference from the user theme
preference in settingsStore. Drops the unused viewport export (only
isMobile was consumed).
* ui: Merge build info into version store
One VersionStore class with build (llama.cpp build number from
build.json) and frontend (PWA version from _app/version.json),
matching the class pattern of the other stores.
* ui: Colocate context gauge popup state with its components
The gauge popup state is local UI state shared only by the
ChatFormContextGauge subtree, so it lives next to its consumers
instead of the app-scope stores barrel.
Kernel is a port of `ggml-cuda/fwht.cu`
(us/run, median):
```
m x n x k GEMM FWHT speedup
64 x 1 x 64 10.20 2.93 3.48x
64 x 2048 x 64 10.75 2.71 3.97x
128 x 1 x 128 10.33 2.88 3.59x
128 x 32 x 128 9.20 2.77 3.33x
128 x 2048 x 128 16.46 2.76 5.95x
256 x 1 x 256 10.19 2.77 3.68x
256 x 2048 x 256 16.69 3.41 4.89x
512 x 2048 x 512 54.16 12.89 4.20x
```
The collapsed \p{S} class was missing '~', which split " ~" into
separate pre-tokens and prevented the Ġ~ BPE merge used by DeepSeek V4.
This caused re-tokenized prompts to diverge from sampled tokens and
broke KV cache reuse.
Assisted-by: Codex
* update to ov-2026.3, update device drivers
* ci: skip nemotron-h rollback test on OpenVINO
The OpenVINO backend does not support SSM_SCAN, so the Nemotron-H recurrent state rollback graph is split and cannot preserve the recurrent cache output shape. Keep the test enabled for other backends and retain the qwen35 OpenVINO rollback coverage.
---------
Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
