* server: return 400 instead of 500 on validation error with X-Conversation-Id
set_req() attaches the spipe as soon as the header is present, before the request
body is parsed. When params validation throws, set_next() never runs and next_orig
stays empty, so on_complete() called it and crashed with std::bad_function_call,
turning the prepared 400 JSON into a generic 500.
on_complete() now treats an empty next_orig as "streaming never started" and evicts
the session installed by set_req(), so a failed request leaves nothing behind for
discovery or replay. This also covers valid requests that carry the header but do
not stream, which previously left an empty finalized session in the map until the
GC TTL.
* ui: do not send the backend_sampling placeholder
On a fresh profile the syncable settings hold the empty string placeholder meaning
"let the server decide". Every neighbor field goes through the hasValue() guard
that filters it, except backend_sampling, which sent the placeholder verbatim and
made every default settings completion fail validation.
Guard the field with hasValue() like its neighbors. hasValue(false) is true, so an
explicit false still reaches the server and the intent of #18781 (send both true
and false) is preserved. Only the placeholder is filtered.
* common: auto-download dflash- and eagle3- HF sidecars
Mirror the existing mtp- sidecar logic to support auto-discovery and
download of DFlash (dflash-) and Eagle3 (eagle3-) speculative decoding
sidecars from Hugging Face repos.
Changes:
- Add --dflash and --eagle3 CLI flags to trigger sidecar download
- Add find_best_dflash() and find_best_eagle3() using find_best_sibling
- Exclude dflash- and eagle3- filenames from primary model selection
- Filter dflash- and eagle3- from cached model listings
- Wire download tasks that set speculative.draft.mparams as fallback
Assisted-by: pi:llama.cpp/Qwen3.6-27B
* docs : regen
read_file with append_loc emits "{n}\u2192 {line}". The space after the
arrow is meant as a separator, but it is indistinguishable from real
indentation. Models strip "{n}\u2192" yet keep the space, so the old_text
passed to edit_file carries a phantom leading space and never matches
(normalize_for_fuzzy_match trims trailing whitespace only, never leading).
Drop the separator space so the arrow abuts content: stripping "{n}\u2192"
now yields the exact line with its real indentation preserved, and the
failure mode cannot occur by construction. Update the description example
to match the new format.
* server : clear checkpoints upon prompt clear
* server : move the prompt state data to the server_prompt_cache
Assisted-by: pi:llama.cpp/Qwen3.6-27B
* server : handle batched slot being cleared
* server: honour per-request reasoning_budget_tokens in chat completions
The reasoning-budget block in oaicompat_chat_params_parse read only the
server-level default (opt.reasoning_budget, typically -1) and the
Anthropic-style alias thinking_budget_tokens, but never the canonical
reasoning_budget_tokens field from the request body. Because the key
was then written into llama_params before the generic body-copy loop
ran, the copy loop found the key already present and silently skipped
the caller-supplied value. Any per-request override (e.g. 0 to
suppress thinking entirely) was therefore discarded.
Fix: read reasoning_budget_tokens from the request body first, so the
value that reaches the sampling layer is the one the caller intended.
Add a unit test in test-chat.cpp that exercises this path via
oaicompat_chat_params_parse with a Qwen3 template (which the autoparser
detects as a thinking-capable model) and asserts the returned
llama_params carries reasoning_budget_tokens == 0.
* server: honour per-request reasoning_budget_message in chat completions
The reasoning-budget block in oaicompat_chat_params_parse wrote
reasoning_budget_message into llama_params straight from the server-level
default (opt.reasoning_budget_message) and never read the canonical
reasoning_budget_message field from the request body. Because the key
was written before the generic body-copy loop ran, that loop found the
key already present and silently skipped the caller-supplied value. Any
per-request override of the message injected before the end tag when the
budget is exhausted was therefore discarded, even though server-task.cpp
already reads reasoning_budget_message from that data.
This mirrors the reasoning_budget_tokens bug fixed in the previous commit.
Fix: read reasoning_budget_message from the request body first, falling
back to the server default, so the value that reaches the sampling layer
is the one the caller intended.
While here, collapse the adjacent reasoning_budget_tokens override to a
single json_value() call; json_value already falls back to the default on
a missing/null/wrong-type key, so the explicit body.contains() guard was
redundant. No behavioral change.
Add a unit test in test-chat.cpp that exercises this path via
oaicompat_chat_params_parse with a Qwen3 template (which the autoparser
detects as a thinking-capable model) and asserts the returned
llama_params carries the per-request reasoning_budget_message rather than
the server default.
* cleanup
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
If mmproj is explicitly disabled via the model preset or command-line
parameters then the model won't be able to handle image/audio inputs and
this shouldn't be declared as supported input modality on the /v1/models
endpoint.
* mtmd: fix silent prompt truncation on embedded NUL
mtmd_input_text carried the prompt as a bare const char* with no
length, so a NUL byte in message content cut the prompt at the
tokenizer boundary and dropped every later message plus the assistant
marker, with no log. Add an explicit text_len and thread it through,
matching llama_tokenize and the text only path.
* cleanup
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* server : fix image blocks in tool_result being dropped during Anthropic→OpenAI conversion
server_chat_convert_anthropic_to_oai() silently discarded image blocks
inside Anthropic tool_result content. This broke multimodal tool outputs
(e.g. a tool that returns an image) because the model never received the
image.
When tool_result contains image blocks, convert them to OpenAI
multimodal content parts (text + image_url array). Plain-text results
remain simple strings for backwards compatibility.
* server : add test for image blocks in Anthropic tool_result conversion
* server: accept null sampling params
Extend the schema validation to treat a null value as absent, so
clients can send null on nullable params (temperature, top_p, ...)
to request the server default. This matches the OpenAI spec and the
json_value convention used elsewhere.
Add has_field() to skip null in the field eval guards.
* has_field -> has_value
A model whose chat template parses at init but fails parser generation
at apply time (e.g. uses {% call %}) throws std::invalid_argument from
common_chat_templates_support_enable_thinking(), which ran outside the
try/catch guarding common_chat_templates_init(). The throw was uncaught
and llama-cli aborted (SIGABRT) instead of failing to load. Moved the
probe inside that try/catch so an apply-time error fails load the same
way an init parse error does.
Signed-off-by: Jesse LaRose <jesse@taey.ai>
* cli: move to HTTP-based implementation
* wip
* working
* remote server ok
* cli support router mode
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
* case: router with only one model
* Apply suggestions from code review
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
* remove outdated comment
* use destructor instead
* add ftype
* cli-view --> cli-ui
* pimpl
* no more json in header
* nits fixes
* also show model aliases
---------
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
* server-stream : pimpl
* server-stream: prefix free functions with server_stream_
address review from ggerganov: scope the public stream functions under the
server_stream_ prefix, matching server_stream_session_manager_start/stop.
* server-stream: guard session and manager state with the mutex
address review from ggerganov: make done, completed_ts and the GC running flag plain members under their
mutex and set the condvar predicates under the lock. keep cancelled atomic for
the lock-free should_stop poll.
* server-stream: trim comments to the non-obvious
address review from ggerganov: drop comments that restate the code, keep the
concurrency, lifetime and ordering rationale. de-stale a few comments left by the
pimpl: g_stream_sessions is now internal and the /v1/streams listing is gone.
* server-stream: update dev docs for the pimpl and prefix
reflect server_stream_session_manager_start/stop and the server_stream_ prefix,
note the manager is now a file-static singleton hidden in the .cpp
* server-stream: move stream traces to debug level
keep the bring-up traces for diagnostics but off the default log: skip
drain, draining, drain ended, DELETE evict, attach_pipe, and the router
stream resume proxy.
* server-stream: align router stream resume proxy trace with upstream
the child-side bring-up traces are already SRV_TRC on master, move the
router stream resume proxy trace to the same level.
* server-stream: move stream_read_status enum to the cpp
it is only used by the hidden session and consumer types, so it belongs
with them behind the pimpl boundary, not on the public header surface.
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* fix: draft model fit vs load inconsistency
* refactor(server): unify draft/mtp parameter initialization, model, and context load
- moves speculative init to speculative.cpp
- changes server_context_impl model_dft and ctx_dft to use raw pointers
- fix: don't throttle progress callback when loading draft model
- refactor: rename draft model/ctx load method
* fix: valign
Before this commit, --cache-ram was not a hard limit:
- The cache always kept at least one entry, even if that entry exceeded the
RAM/token limits.
- Old entries were only evicted for the RAM/token limits after saving the new
one, which could cause the cache to temporarily exceed the RAM/token limits
even if individual entries were below the limit.
Now, ensure that the RAM limit is strict with these changes:
- Skip saving state to cache if by itself it exceeds the RAM limit.
- Evict old entries as necessary to make the new entry fit.
Additionally, token-limit cleanup may now evict the last remaining cache entry
instead of always preserving one.
* server: fix deadlock in load_models() when erasing a finished download
The download monitoring thread acquires the models mutex on its way out,
but load_models() joined it from the erase loop while holding that mutex.
Join it outside the lock via threads_to_join like the other monitoring
threads.
* server: add default timeout to test requests
A hung server now fails the test after 10 minutes instead of stalling
the CI job for hours. Explicit timeouts are unchanged.
* server + ui: ping silent SSE streams every 1s and kick only after 3s so slow prefill never drops healthy connections
* server + ui: sse_ping_interval becomes a per-request body field
Address review from ngxson: the global default returns to 30 so API
clients see no behavior change, and the WebUI sends sse_ping_interval: 1
in the request body since it owns the 3s visibility-kick contract and
declares the cadence it needs. Positive values keep the existing > 0
gate, -1 keeps its disabled semantics.
* server: move sse_ping_interval into the request schema
Address review from ngxson: the field is now a typed field_num with
hard limits (-1, INT32_MAX) bound to task_params, seeded from the CLI
default alongside the other inherited parameters. The raw json_value
read and its redundant comment are gone, and schema evaluation brings
type and range validation for free.
* llama : add llama_model_ftype_name()
Expose the model file type (quantization) name, e.g. "Q8_0" or
"Q4_K - Medium", through a new public C API. The returned pointer is
valid for the lifetime of the model and nullptr when the model is
invalid or the file type is unknown.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Export enum
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* s/llama_model_ftype_name/llama_ftype_name/
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Move "(guessed)" to the front in llama_ftype_name
Prepend the "(guessed)" label instead of appending it. This allows removing
the non-thread-safe static std::string, making the function allocation-free.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Add LLAMA_FTYPE_PREFIX
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Dont check for model
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* common,server: handle bracketed IPv6 literals in URL authority
Parse the [host]:port form (RFC 3986) and bracket IPv6 hosts when
formatting a URL authority: listening log, proxy Host header, proxy
log, client rebuild. The per-request remote_addr stays bare.
* common: restore unsupported scheme throw in url parser
Address @ngxson review: keep the explicit reject in port resolution so
the block stays self-contained. Non-http(s) schemes still throw (also
gated at the top of common_http_parse_url).
