* chore(ui): pin package versions to currently installed
- Update all dependencies and devDependencies to match exactly what's in package-lock.json
- This ensures reproducible builds by locking to specific versions rather than semver ranges
* chore: Update packages
* chore: Move remaining dependencies to devDependencies
* fix: Add missing `mermaid` package
* chore: Update `cookie` package to `v1.1.1`
* chore: Formatting
* test: Update test configs
* ggml: vectorize ggml_vec_dot_q4_1_q8_1 with WASM SIMD128
Optimize the inner loop of ggml_vec_dot_q4_1_q8_1_generic using
WASM SIMD128 intrinsics, gated behind #ifdef __wasm_simd128__ so
non-wasm builds are completely unaffected.
Approach:
- single wasm_v128_load covers all 32 packed 4-bit weights
- nibbles unpacked via AND/SHR into two u8x16 registers
- widened to i16 before multiply (WASM SIMD has no i8*i8 instruction)
- 4x wasm_i32x4_dot_i16x8 calls accumulate all 32 element pairs
- horizontal reduce via 4x wasm_i32x4_extract_lane
Benchmark (node v25, emcc -O3 -msimd128, 64 blocks x QK8_1=32,
200k iterations):
| impl | ns/call | speedup |
|--------|---------|---------|
| scalar | 880.7 | 1.00x |
| simd | 257.8 | 3.42x |
Correctness verified against scalar reference across 10 random seeds
with exact output match.
* ggml: move q4_1_q8_1 WASM SIMD implementation to wasm backend
Relocate the SIMD128 implementation of ggml_vec_dot_q4_1_q8_1 to ggml/src/ggml-cpu/arch/wasm/quants.c to follow architecture-specific layout. Restore the generic implementation in ggml/src/ggml-cpu/quants.c.
Move for loop in the else block.
* ggml: use generic q4_1_q8_1 fallback in wasm backend
* server: avoid unnecessary checkpoint restore when new tokens are present
The pos_min_thold calculation unconditionally subtracts 1 to ensure at
least one token is evaluated for logits when no new tokens exist.
However, when the request contains new tokens beyond the cached prefix,
this -1 is overly conservative and may trigger an unnecessary checkpoint
restore.
Conditionally apply the -1 only when n_past >= task.n_tokens() (no new
tokens), avoiding redundant KV state restoration when there is actual
work to do.
* cont : add ref
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* webui: fix tool selector toggle/counter, key tools by stable identity
Key the disabled set, counts and toggles by a stable per-tool key
instead of bare function name, deduped from one canonical list. Per-tool
checkboxes become presentational (single row handler, no nested button),
category checkboxes drop the tristate (n/total carries partial). One
getEnabledToolsForLLM keeps normalized MCP schemas and dedupes by name.
* ui: use SvelteSet and SvelteMap for local tool collections to satisfy svelte/prefer-svelte-reactivity
The XCFramework generated by build-xcframework.sh creates a module map
that manually lists public headers.
That list can fall out of sync with the framework's Headers directory.
The module map is currently missing ggml-opt.h, which is present in the
framework headers. This can cause downstream Apple builds to fail with:
Include of non-modular header inside framework module 'llama'
Use the framework's Headers directory itself as the module map umbrella
instead of maintaining a manual header list. This makes all public headers
under the generated framework's Headers directory part of the llama module.
* tests : refactor test-save-load-state to accept token input
- Default prompt is now empty; when not provided, generate n_batch
random tokens (useful for models without a tokenizer)
- Tokenization happens once upfront; pass token vector to test functions
- generate_tokens prints token IDs instead of decoded pieces
- Use llama_model_get_vocab / llama_vocab_n_tokens API
- Upgrade log level from LOG_TRC to LOG_INF for visibility
Assisted-by: llama.cpp:local pi
* cont : use llama_tokens alias
* Start work on flash_attn refactor
* Refactor
* Split k/v quantization
* Refactor and abstract quantization logic for flash_attn and mul_mat
* Add quantization support to tile path
* formatting
* Move to functions, add a check
* Tidy up SYCL doc a bit
- Add explicit links to referenced items
- Fix spelling errors
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Correct documented default for GGML_SYCL_GRAPH
The default is ON, not OFF:
$ cmake -LAH -B build | grep GGML_SYCL_GRAPH
...
GGML_SYCL_GRAPH:BOOL=ON
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Move docker instructions from SYCL.md to docker.md
This makes them directly accesible from the Quick Start section
of the top-level README.md.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Refer to intel.Dockerfile for ARGs and their defaults
The defaults are always changing; this avoids accuracy errors
from duplicating the information.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Remove mention of Nvidia in SYCL row of backend table
This support was removed in 2026.02 - refer to the SYCL.md News.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
---------
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Removes __restrict__ from PDL kernel headers due to incompatibility with
PDL. Adds preprocessor directives based on arch in kernel body to add
__restrict__ to retain performance on older architectures.
* Simplifies new __restrict__ usage via macro
* Add hopper to PDL __restrict__ fix.
Co-authored-by: Oliver Simons <osimons@nvidia.com>
---------
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* cuda: reserve space for quantize kv-cache at startup
* address review comments
* remove forward decl
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* remove assert in ggml-cuda.cu
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Add support for the ibm-granite/granite-embedding-{97m,311m}-multilingual-r2 embedding models:
* Added a version of the gpt4o tokenizer that has a fixed regex (better handling of marks), and different token merging setting for the 97m model
* Reused gemma4 tokenizer for the 311m model
* granite-embedding-*-multilingual-r2 : add support SwiGLU FFN for Granite Embedding Multilingual R2
* added new GGUF key <arch>.hidden_activation (LLM_KV_HIDDEN_ACT) + writer
* added a forward declaration of llm_ffn_op_type to llama-hparams.h
* added llm_ffn_op in hparams
* added LLM_FFN_NONE = 0 sentinel to llm_ffn_op_type (value-initialization), modern-bert: explicitly assigns LLM_FFN_GEGLU before reading GGUF (unchanged).
* centralized hidden_act mapping in llama-model.cpp, added llm_ffn_op_type_from_string() helper, mirroring rope_scaling_type/llama_rope_scaling_type_from_string()
* modern-bert reads the GGUF key (when present) and uses the resulting op in its FFN graph
* Added granite-embedding-{97m,311m}-multilingual-r2 to the converter code
* Added the hashes for the granite embedding multilingual R2 models
* Set the hidden_activation in the GGUF if the field is present in config.json (such as for the granite embedding models)
* common : fix state save in common_prompt_batch_decode
This commit addresses a bug in common_prompt_batch_decode that affects
the session state store/restore in completion.cpp and
save-load-state.cpp.
The motivation for this is that currently the code is saving n-1 tokens
in both the session_tokens and in the KV cache. Then when loading the
session tokens, and if the prompt matches, it would replay the last
saved token (n-1) into the next position, effectively replaying the
same token in the wrong position.
The fix is to store all n tokens in session_tokens, while the memory
state only reflects n-1 processed tokens as the saving happens before
the last token is decoded in common_prompt_batch_decode.
I ran both completion.cpp and save-load-state.cpp with a transformer, a
recurrent, and a hybrid model.
Resolves: https://github.com/ggml-org/llama.cpp/issues/23400
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
