* sd: minor API path handling cleanup
* sd: sync with master-749-b11c95a
* sd: use original API parameters at the internal C++ API
* sd: split_mode and auto_fit backend support
* sd: sync with master-758-c674225
* sd: sync with master-765-bb84971
* sd: sync with master-767-885f01a
The matmul_tiled path uses large local stack buffers for A_pack and B_pack. On AIX this can trigger a segmentation fault, so reduce the buffer footprint there to keep the tiled path usable.
Performance Impact:
~ 2x gains in PP_Speed for FP32, Q4_0 and Q8_0 models tested with llama-bench, llama-batched-bench and llama-cli.
Models used: Llama3.2 3b Instruct F32, qwen 2.5 3b Q4_0 and Q8_0
* meta: fix tensor split metadata for GQA attention
* Tidied the code a bit to match existing style
* Revert "Tidied the code a bit to match existing style"
This reverts commit b90c6c6300091fe09e2350a3d4edcfcf15db8d2e.
* Reverted the ggml-backend-meta asset hack.
Tensor parallelism (-sm tensor) combined with -ncmoe (CPU-offloaded MoE
experts) aborts during warm-up on MoE models with
GGML_ASSERT(ggml_is_contiguous(tensor)) in ggml-backend-meta.cpp.
The failing tensor is the MoE router output (ffn_moe_topk): it is mirrored
(GGML_BACKEND_SPLIT_AXIS_MIRRORED, replicated across backends since routing
must be identical) and happens to be a non-contiguous view.
ggml_backend_meta_buffer_{get,set}_tensor asserted contiguity before
consulting the split state, so a mirrored non-contiguous tensor tripped the
assert even though the GGML_BACKEND_SPLIT_AXIS_MIRRORED case right below
already handles it.
Move the split-state lookup above the assert and allow the mirrored case in
both get_tensor and set_tensor.
Diagnosis credit to the reporter (@nathanmp).
Fixes#24886
Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com>
* Update ggml-cuda.cu - Turing P2P access fix.
* Add original code as fallback behaviour when NCCL or P2P is not set/true.
* Update ggml/src/ggml-cuda/ggml-cuda.cu to add comment as per suggestion
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* cuda : concat implementation for quantized types
* chore : apply am17an clever suggestion to shorten the code
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
llm_graph_input_attn_kv::set_input and llm_graph_input_attn_kv_iswa::set_input
call set_input_k_rot / set_input_v_rot whenever the rotation tensor pointer is
non-null, but the tensor's buffer can be unallocated (NULL) when a graph only
stores K/V without attending -- e.g. DFlash speculative decoding's KV-injection
pass. set_input_k_rot then calls ggml_backend_buffer_is_host() on a NULL buffer
and aborts with GGML_ASSERT(buffer).
Guard the four k_rot/v_rot inputs with the same "&& ->buffer" check that the
adjacent kq_mask inputs already use in these two functions. When the buffer is
unallocated there is no data to upload, so skipping is correct.
Fixes#25191
Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com>
* chat: trim messages sent to StepFun parser (fixes long reasoning loops)
* add regression test; remove duplicate template
* chat: trim StepFun content parts before rendering
The StepFun trim workaround ran on the already-rendered messages, where
typed content parts have been concatenated into a single string, so the
per-part whitespace could no longer be reached. Move the trim ahead of
rendering and apply it to content_parts text as well as the string
content and reasoning_content. Adds a content-parts regression test.
Co-Authored-By: Piotr Wilkin <ilintar@gmail.com>
Assisted-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: tarruda <tpadilha84@gmail.com>
* ui: Improve performance when streaming
* ui: build sibling info map in branching utils
Moves the node map and sibling map construction from the
.by block into buildSiblingInfoMap() in branching.ts.
The map is built once per structural change and only read
afterwards, so it does not need SvelteMap reactivity. Keeping
the construction in plain TypeScript fixes the
svelte/prefer-svelte-reactivity lint error and groups the
branching logic where it already lives.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* cuda: enable topk-moe fusion for 288 experts
The topk-moe fusion only accepted power-of-2 expert counts (or the
special-cased 576), so models with 288 experts (e.g. Step-3.7-Flash)
fell back to the unfused per-layer routing chain: softmax/sigmoid,
argsort, get_rows, sum_rows, div, clamp, scale. At batch size 1 that
is ~330 extra tiny graph nodes per token.
288 is a multiple of the warp size, so the existing kernel already
handles it; this adds the missing template instantiation and accepts
288 in the eligibility check.
Measured on gfx1151 with Step-3.7-Flash IQ4_XS (llama-bench,
-b 4096 -ub 4096 -fa 1 -dio 1 -ctk q8_0 -ctv q8_0; machine idle,
before/after paired so pp4096 stays matched as a load control):
test | before | after
----------------+----------------+----------------
pp4096 | 460.99 ± 0.45 | 462.47 ± 0.34 (unchanged)
tg128 | 19.10 ± 0.04 | 19.56 ± 0.03 (+2.4%)
tg128 @ d30000 | 12.68 ± 0.04 | 12.69 ± 0.03 (unchanged)
Prompt processing is unaffected (the fusion only touches decode
routing). The decode gain is ~+2.4% at shallow context and fades with
depth: by 30k tokens each step is attention-bound over the KV cache,
so removing the fixed routing overhead is no longer visible.
Assisted-By: Claude Fable 5 <noreply@anthropic.com>
* Update tests/test-backend-ops.cpp
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* Add comment for case 288 in topk-moe.cu
---------
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* ui: migrate legacy string-encoded booleans in persisted config
* ui: enable thinking by default
Fresh users and legacy conversations without a persisted thinking
preference now default to enabled. The per-conversation toggle and
the persisted localStorage choice keep taking precedence.
Picks up the enable_thinking default from #24876.
* server + ui: ping silent SSE streams every 1s and kick only after 3s so slow prefill never drops healthy connections
* server + ui: sse_ping_interval becomes a per-request body field
Address review from ngxson: the global default returns to 30 so API
clients see no behavior change, and the WebUI sends sse_ping_interval: 1
in the request body since it owns the 3s visibility-kick contract and
declares the cadence it needs. Positive values keep the existing > 0
gate, -1 keeps its disabled semantics.
* server: move sse_ping_interval into the request schema
Address review from ngxson: the field is now a typed field_num with
hard limits (-1, INT32_MAX) bound to task_params, seeded from the CLI
default alongside the other inherited parameters. The raw json_value
read and its redundant comment are gone, and schema evaluation brings
type and range validation for free.
* feat: ui: Add predefined recommended MCP servers to settings
* feat: ui: Add MCP server recommendation dialog with custom server support
* feat: Auto-focus input fields on mount and dynamic addition
* feat: Add header validation to MCP server add and edit forms
* feat: Persist recommended MCP server opt-in selections
* test: Cover MCP configuration with tests
* chore: Format & cleanup
* feat: Centralize MCP server overrides to settings config and improve recommendation UI
* fix: Capture index before mutation to prevent focus drift
* refactor: Extract MCP_CARD_VISIBLE_TOOL_LIMIT to shared constants
* refactor: Support arbitrary authorization header schemes
* refactor: Consolidate MCP recommendations dismissal into existing storage key
* fix: Use case-insensitive comparison for MCP server ID prefix check
* refactor: Centralize MCP server visibility logic and extract recommendations hook
* refactor: Cleanup
* Remove redundant CUDA copies after gated_delta_net.
Currently, GDN writes recurrent state snapshots into its output tail, then the graph immediately copies those snapshots into ssm_states_all. With MTP draft length 3, target decode uses K=4, so that becomes 4 extra ggml_cuda_cpy calls.
The change detects that gated_delta_net -> view -> cpy pattern and makes the CUDA GDN kernel write the state snapshot(s) directly into the recurrent cache, skipping the intermediate tail writes and copy kernels when safe.
* Address review comments
* llama : add llama_model_ftype_name()
Expose the model file type (quantization) name, e.g. "Q8_0" or
"Q4_K - Medium", through a new public C API. The returned pointer is
valid for the lifetime of the model and nullptr when the model is
invalid or the file type is unknown.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Export enum
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* s/llama_model_ftype_name/llama_ftype_name/
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Move "(guessed)" to the front in llama_ftype_name
Prepend the "(guessed)" label instead of appending it. This allows removing
the non-thread-safe static std::string, making the function allocation-free.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Add LLAMA_FTYPE_PREFIX
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Dont check for model
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* hex-mm: fold mm quant tasks into the main matmul threads
* hex-mm: minor formatting fixes
* hex-mm: cleanup is_quant checks in dma dispatch
* hex-mm: fix dst-spad alignment
* hex-mm: move fp kernels in the hvx-mm-kernels header
* hex-mm: fuse with ADD
* hex-fa: factor out ukernels into separate headers and unify the rest
* hex-fa: move kernel-params compute into the host
* hex-fa: refactor vtcm alloc for consistency
* hex-fa: add support for FA_SELECT
* hex-fa: update tracing insrumentation to cover all functions
* hex-fa: update hvx fallback thresholds to recover t/g regressions
* hex-fa: update tracing instrumentation
* hex-fa: improved tracing with additional events
* hex-fa: optimize mask processing (fastdiv, etc)
* hex-fa: improve mask dma caching
* hmx-fa: change loop order to maximize mask cache hits
* hex-fa: remove over instrumentation
* hex-fa: breakdown QKV prep trace events
* hmx-fa: further mask proc optimizations
* hex-fa: mask broadcast is the common case, optimize for that
* hex-fa: use aligned loads where possible
* hex-fa: update loops to use uint32_t indices
* hmx-fa: fold vtcm init into q prep task
* hex-fa: update rest of the hmx funcs to use uint32_t
* hmx-fa: fold build_d into the main softmax loop
* hmx-fa: start kv dmas earlier
* hmx-fa: start mask dma a bit earlier
* hex-fa: precompute rows per task to avoid divs
* hmx-fa: specialize fa_o_store for f16 and f32
* hmx-fa: prelim support for Sinks
* hmx-fa: keep softmax accumulators in fp32
* hex-fa: add tanh_f16 and exp2_f16 and use that in FA
* hex-fa: use fp16 math in the hvx kernel
* hex-fa: avoid expensive float -> __fp16 cast for slopes and softcap
* hex-fa: replace most vec_exp_f32 with vec_exp2_f16
* hmx-fa: vectorize sinks update
* hex-fa: minor formatting
* hmx-fa: fold softcap loop into the tile load
* hmx-fa: use vectoralias to populate sinks
* hex-fa: remove redudant check
* hex-fa: fix vtcm size compute to use fp32 for accumulators
* hex-mm: fix trailing spaces
* hmx-fa: dont use -inf to init mask to avoid conversion overflows
* hex-fa: no need to explicitly guard -inf in the f16->f32 converter now
* hmx-fa: cleanup fa sinks handling
* hex-mm: fixed src2 stride handling when mm is fused with add
* hex-fa: make lto happy
