From c00c1e70c8a9f5a4cdbac61fc73b55e89c52be08 Mon Sep 17 00:00:00 2001 From: Daniel Han Date: Mon, 6 Jul 2026 15:40:46 -0700 Subject: [PATCH] studio: tool calling for DeepSeek (R1/V3/V3.1), GLM 4.x, Kimi K2 on safetensors + MLX (#5624) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * studio: tool calling for Llama-3, Mistral, Gemma 4 on safetensors + MLX (#5615) Adds tool calling for Llama-3, Mistral (pre-v11 + v11+ + [ARGS]), and Gemma 4 to the safetensors / transformers and MLX backends. Parser patched against llama.cpp / vLLM / SGLang per-family parsers and normalises to OpenAI shape. 96 targeted unit tests + cross-OS staging CI (ubuntu / macos-14 / windows) green on the multi-format probe. * studio: tool-call healing parity between safetensors / MLX and GGUF After the multi-format parser landed in #5615, the safetensors / MLX agentic loop and the GGUF loop still differed on healing behaviour. This commit closes the gaps in both directions so the two backends react the same way to identical model output. Changes: 1. core/inference/llama_cpp.py -- the GGUF BUFFERING state machine now wakes on every emission marker the shared parser knows. Was ("", " / Mistral [TOOL_CALLS] / Gemma 4 <|tool_call>). Stream cleanup is delegated to the same shared strip_tool_markup so leaked markup from any family is removed from assistant content. 2. core/inference/llama_cpp.py -- per-tool canonical heal key. When a tool arguments field is a bare string and JSON parsing fails, the GGUF path now heals to {"code": raw_args} for python, {"command": raw_args} for terminal, and {"query": raw_args} for everything else. Was hard-coded to {"query": raw_args}, which silently routed every python / terminal emission through web_search. Mirrors safetensors_agentic._CANONICAL_HEAL_ARG. 3. core/inference/safetensors_agentic.py -- re-prompt on plan- without-action. When the model emits a short forward-looking intent ("I'll search for that", "Let me check", "First, I will...") and no tool call, the loop nudges the model to act instead of silently returning a plan-only answer. Up to _MAX_REPROMPTS=3 (matches GGUF). The intent regex, character cap, and instruction text are byte-identical to the GGUF path. The buffer-end fall-through is unified so a buffered intent emission that never exits the BUFFERING state still triggers the re-prompt. 4. core/inference/safetensors_agentic.py -- extra iteration slots for re-prompts. The loop now budgets max_tool_iterations + _MAX_REPROMPTS + 1 total iterations and tracks the tool-call count separately, so a stalling model can be nudged 3x without eating the caller's tool-call budget. Mirrors the _extra slot reservation in the GGUF path. Tests (14 new safetensors-side units; 5 GGUF parity pins): TestLoopRePrompt -- intent-trigger, plain-answer, no-tools, cap-at-three, budget preserved, buffer-end intent. TestLoopCanonicalHealKey -- python / terminal / unknown. TestGGUFSafetensorsHealingParity -- shared markers used, shared strip used, canonical heal keys identical, intent regex matches same phrases, _MAX_REPROMPTS equal on both backends. All 110 targeted tests pass locally; the broader tool / inference / model-config / sandbox / anthropic / mlx suites stay green. Why this matters Without this parity, Llama-3.2 / Mistral / Gemma 4 emissions on Mac (MLX) and Linux-safetensors stop the agentic loop as soon as the model says "Let me...", because the GGUF re-prompt logic never existed on these backends. The two-marker GGUF BUFFERING tuple also let non-Qwen tool emissions stream out as plain prose when llama-server's structured channel did not pick them up. Both paths now drain the same way, heal the same way, and re-prompt the same way -- so a tool call that works on GGUF works identically on safetensors / MLX. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: fix tool-call parser bugs from gemini review on #5620 Three high-priority gemini findings on the tool-call parsing additions: 1. unicode_escape on UTF-8 bytes corrupts non-ASCII literals (e.g. ✨ becomes â\x9c¨). Replace with json.loads on a quoted string -- preserves emoji / CJK / RTL while still handling \n \t \uXXXX escapes. 2. Llama-3 sentinel stripping is order-dependent. A leading `<|eot_id|><|begin_of_text|>` left `<|begin_of_text|>` behind because the loop had already passed that sentinel. Loop until no sentinel matches at the start. 3. Mistral v11+ `[TOOL_CALLS] name { json }` regex uses non-greedy `\{.*?\}` which truncates at the first `}` of a nested JSON argument, leaking the tail (e.g. `}}`) into user-visible streamed text. Same problem for the v0.3 array pattern with nested brackets. Strip those with balanced brace/bracket scanning via a new `_strip_mistral_closed_calls` helper called from `strip_tool_markup`. Also fix the inference routes' parallel `_TOOL_XML_RE`: - Same nested-JSON truncation in the Mistral patterns; route the strip through the parser's balanced-scan helper via a thin `_strip_tool_xml` wrapper that all existing callers now use. - Llama-3 `<|python_tag|>[^\n<]*` stopped at any `<`, leaking the tail of any tool call whose argument contained a literal `<` (queries, code snippets). Relax to `[^\n]*` which keeps the strip confined to the actual end-of-line. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: tool calling for DeepSeek (R1/V3/V3.1), GLM 4.x, Kimi K2 Adds three more emission-family parsers to tool_call_parser.py so the shared safetensors / MLX / GGUF agentic loop covers the major open- weight reasoning families. Patterns ported from llama.cpp (common/chat-parser.cpp legacy pre-PEG branch), vLLM (tool_parsers/deepseekv3*, glm4_moe, kimi_k2), and SGLang (function_call/deepseekv31_detector, glm4_moe_detector, kimik2_detector). All three references are MIT (llama.cpp) or Apache-2.0 (vLLM, SGLang). Formats covered: DeepSeek R1 <｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function <｜tool▁sep｜>NAME\n```json\n{...}\n```<｜tool▁call▁end｜> <｜tool▁calls▁end｜> -- args wrapped in a Markdown json fence, ``function`` literal prefix per llama.cpp common_chat_parse_ deepseek_r1 (chat-parser.cpp:801-820) DeepSeek V3/V3.1 <｜tool▁calls▁begin｜><｜tool▁call▁begin｜>NAME <｜tool▁sep｜>{json}<｜tool▁call▁end｜><｜tool▁calls▁end｜> -- bare JSON, no code fence, no ``function`` prefix per llama.cpp common_chat_parse_deepseek_v3_1 (chat-parser.cpp:822-879) GLM 4.5/4.6/4.7 NAME\nk1 \nv1... -- strings raw, non-strings JSON-encoded per chat_template.jinja; multi-call is back-to-back blocks. Per llama.cpp common_chat_parse_glm_4_5 (chat-parser.cpp:1040-1052) Kimi K2 <|tool_calls_section_begin|><|tool_call_begin|> functions.NAME:IDX<|tool_call_argument_begin|>{json} <|tool_call_end|><|tool_calls_section_end|> -- bare name recovered by stripping ``functions.`` prefix and ``:IDX`` suffix; full id preserved as tool_calls[i].id so the roundtrip replays verbatim. Per llama.cpp common_chat_parse_kimi_k2 (chat-parser.cpp:896-913) Marker collisions GLM uses the same ```` opener as Qwen but with a bare function name + ```` body (Qwen has ``\s*{`` after the tag). The dispatch keeps Qwen first; Qwen's _TC_JSON_START_RE returns no matches on a GLM emission, so the fall-through to _parse_glm_tool_ calls handles it correctly. Existing Qwen tests confirm zero regression. Streaming buffer TOOL_XML_SIGNALS extended from 5 markers to 12 so the BUFFERING state machine wakes on every new family's section opener. Added the DeepSeek alternative markers (ASCII underscores, short ``<｜tool▁calls｜>`` form) because real checkpoints emit those variants. Strip patterns _TOOL_CLOSED_PATS adds DeepSeek envelope (``<｜tool▁calls▁begin｜>... <｜tool▁calls▁end｜>``) and Kimi section (``<|tool_calls_section_begin|> ...<|tool_calls_section_end|>``). _TOOL_ALL_PATS adds the same plus the unclosed-tail variants so a truncated stream does not leak markup. Route gate _detect_safetensors_features._PARSER_MARKERS grows to include DeepSeek and Kimi markers plus ```` (the unique GLM signal). _TOOL_XML_RE (the route-layer markup-strip regex) gets DeepSeek and Kimi closed-pair patterns. _TOOL_TEMPLATE_MARKERS in llama_cpp.py adds ``message['role'] == 'tool'``, ``message['tool_calls']``, and ``tool_calls is defined`` so the classifier recognises DeepSeek's subscripted-access template style (it has no top-level ``{% if tools %}`` block). Tests (39 new): TestParserDeepSeek (7) -- R1 fence, short-form opener, V3.1 bare, multi-call, with-reasoning, strip, signal-wakes-streaming TestParserGLM (6) -- single, mixed types, multi-call, unclosed-heal, no-Qwen-regression, strip TestParserKimi (6) -- single, multi-call, dotted-name, unclosed, strip, signal-wakes-streaming TestParserCrossFormatRouting (2) -- dispatch routing, signal coverage TestLoopBasic loop integration (3) -- DeepSeek / GLM / Kimi end-to-end Capability advertise (3) -- DeepSeek / GLM / Kimi templates flip supports_tools=True All 398 targeted tests pass locally (115 safetensors + 27 capability + rest of tool / inference / sandbox / model-config suites). Builds on PR #5620 (parser + healing parity for Llama-3 / Mistral / Gemma 4); will rebase cleanly onto main once #5620 lands. PR opened as draft - do not merge until validated against real models for each family. Sources - llama.cpp common/chat-parser.cpp lines 801-913, 1040-1052 (MIT) - vLLM vllm/tool_parsers/deepseekv31_tool_parser.py (Apache-2.0) - vLLM vllm/tool_parsers/glm4_moe_tool_parser.py (Apache-2.0) - vLLM vllm/tool_parsers/kimi_k2_tool_parser.py (Apache-2.0) - SGLang python/sglang/srt/function_call/{deepseekv31,glm4_moe,kimik2}_ detector.py (Apache-2.0) - Live chat templates: deepseek-ai/DeepSeek-V3.1, zai-org/GLM-4.6, moonshotai/Kimi-K2-Instruct, unsloth/DeepSeek-V3-0324, unsloth/GLM-4.5-Air, unsloth/Kimi-K2-Instruct * studio/routes: make python_tag strip multi-line aware Earlier revisions of _TOOL_XML_RE in studio.backend.routes.inference oscillated between two bug shapes: 5615 r"<\|python_tag\|>[^\n<]*" -- stopped at any literal "<" so code='if x < 10: pass' leaked '< 10: pass)' to the user. 5620.1 r"<\|python_tag\|>[^\n]*" -- single-line only; the second line of python.call(code="a\nb") leaked. The full parser (_parse_llama3_python_tag) already handles both via balanced-brace scanning, so the parsing path was fine; the LEAK was in the streaming strip path that runs on every cumulative emission while content is still arriving. Switch to r"<\|python_tag\|>(?:[^<]|<(?!\|))*" so the strip consumes: * any character that is not a "<" (newlines, JSON, code, ...), * a "<" only when it is NOT followed by "|" (i.e. NOT a Llama-3 sentinel start like <|eot_id|>, <|eom_id|>, <|begin_of_text|>). This means: * code='if x < 10' stays inside the strip (5615 fix preserved), * multi-line code stays inside the strip (5620 round 2), * the strip terminates at the next Llama-3 sentinel so trailing assistant content survives. Tests: TestRoutesPythonTagStrip (8 cases) pytest test_safetensors_tool_loop.py test_safetensors_capability_advertise.py -> 118 passed in 1.81s (was 110). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: review follow-ups for DeepSeek / GLM / Kimi tool calling Four fixes addressing review of the parent commit: 1. GLM coercion: tighten the json.loads -> ast.literal_eval -> raw cascade to only deserialize when the body unambiguously looks like a JSON literal (object, array, JSON-encoded string, true/false/null, or numeric). Strings like ``True`` / ``None`` (Python literals, not JSON) and arbitrary prose now stay raw. The bare-numeric / bare-boolean ambiguity with string args remains an inherent limitation of the template without schema access -- documented in the new comment. Drops the ast import entirely (closes Gemini's :1036 suggestion). 2. Kimi K2 bare-counter ids (e.g. ``<|tool_call_begin|>3``) are now dropped rather than surfaced as a tool literally named "3". Matches vLLM behaviour; SGLang's schema-infer fallback is out of scope at the parse site. Real Kimi K2 emissions use ``functions.NAME:IDX`` so this is the exception path. 3. Restore the elaborate ``<|python_tag|>(?:[^<]|<(?!\|))*`` clause in routes.inference._TOOL_XML_RE -- the simpler ``[^\n<]*`` form regressed PR #5620's multi-line / literal-``<`` python_tag fix. Restore ``TestRoutesPythonTagStrip`` (8 tests) adapted to call ``_TOOL_XML_RE.sub`` directly since the ``_strip_tool_xml`` helper was inlined this PR. 4. Add the spaced and backslash-escaped DeepSeek opener variants (``<｜tool calls begin｜>``, ``<｜tool\_calls\_begin｜>``) to ``TOOL_XML_SIGNALS`` for streaming-gate parity with ``_DEEPSEEK_BEGIN_RE``. Also updates the llama.cpp / vLLM citations in the parser docstrings: ``common/chat-parser.cpp`` was split into ``common/chat.cpp`` + ``common/chat-peg-parser.cpp`` by llama.cpp PR #18675, and vLLM moved the tool parsers from ``vllm/entrypoints/openai/tool_parsers/`` to ``vllm/tool_parsers/``. Pin to pre-refactor commit ``51fa458a92d6`` where the cited line numbers still resolve. New regression tests in ``test_pr5624_regressions.py`` cover the GLM coercion heuristic shapes, GLM literal-``<`` in arg_value, Kimi K2 dotted name, Kimi K2 bare-counter drop, DeepSeek V3.1 truncated mid-stream, and routes-layer strip across all three new families. Tests: pytest studio/backend/tests/test_safetensors_tool_loop.py studio/backend/tests/test_safetensors_capability_advertise.py studio/backend/tests/test_pr5624_regressions.py -q -> 170 passed in 1.91s * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: tighten verbose comments in tool-call parser sections Comments were narrating what the code already says. Cut historical "earlier revisions used X, then Y" narratives down to one-line WHY notes where the footgun still matters (canonical heal-key parity, balanced-brace vs non-greedy regex, ``(?:[^<]|<(?!\|))*`` over ``[^\n<]*``/``[^\n]*``). Drop section-header banners. No behaviour change. Re-ran: pytest studio/backend/tests/test_safetensors_tool_loop.py \ studio/backend/tests/test_safetensors_capability_advertise.py -q -> 118 passed. Regression replay (parser + _coerce_arguments on the 5 #5615 inputs) -> 21/21. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: GLM 4.7 no-newline emission + Kimi multi-section parity Two fixes surfaced by triple-confirm verification against the live HF chat templates and upstream llama.cpp / vLLM / SGLang parsers. 1. GLM 4.7 silent drop ``zai-org/GLM-4.7/chat_template.jinja`` line 65 uses ``{{- '' + tc.name -}}`` which Jinja strips trailing whitespace from, so the first ```` follows the function name with NO ``\n`` between them. Real emissions look like ``get_weathercityLondon ``. The previous ``_GLM_TC_OPEN_RE`` ended the name with ``\n`` so GLM-4.7 calls were silently dropped (parser returned ``[]``). Fix: relax the name terminator to a lookahead that accepts EITHER ``\n`` OR the next ````: _GLM_TC_OPEN_RE = re.compile( r"\s*([^\n<{][^\n<]*?)\s*(?=\n|)" ) The first-char restriction ``[^\n<{]`` still excludes Qwen's ``{json}`` form so the Qwen-vs-GLM dispatch remains mutually exclusive. 2. Kimi multi-section parity with vLLM / SGLang ``vllm/tool_parsers/kimi_k2_tool_parser.py`` and SGLang's ``kimik2_detector.py`` both use ``re.findall`` and so collect every ``<|tool_calls_section_begin|>...<|tool_calls_section_end|>`` block in a single stream. The previous implementation stopped at the first ``<|tool_calls_section_end|>``. Kimi K2 doesn't emit multi-section in practice, but parity is cheap. Fix: wrap the existing per-call body parser in an outer loop that advances past each ``<|tool_calls_section_end|>`` and continues to the next ``<|tool_calls_section_begin|>``. Body parsing extracted to ``_parse_kimi_section_body`` for clarity. Truncated final section is still surfaced via the existing in-body balanced-brace walk. Verified independently against the live HF templates: * GLM-4.7 emission constructed from the live template parses to the expected ``{name, arguments}`` shape. * GLM-4.5 / 4.6 newline shape continues to parse (the lookahead also matches ``\n``). * Qwen ``{json}`` still dispatches to the Qwen path -- the first-char restriction stops the GLM regex from biting JSON bodies. * Kimi two-section stream surfaces both calls in order with full ids preserved. * Bare-counter Kimi ids still drop. Tests added in ``test_pr5624_regressions.py``: * ``test_glm_4_7_no_newlines_between_name_and_arg_key`` * ``test_glm_4_7_no_newlines_multi_call`` * ``test_glm_4_7_does_not_break_qwen_path`` * ``test_kimi_two_sections_in_one_stream_both_parse`` pytest studio/backend/tests/test_safetensors_tool_loop.py studio/backend/tests/test_safetensors_capability_advertise.py studio/backend/tests/test_pr5624_regressions.py -q -> 174 passed in 1.93s pytest studio/backend/tests/ -q -k 'not gpu and not llama_cpp_integration' -> 2038 passed, 15 failed (pre-existing CI gaps). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: parser robustness fixes for PR #5620 Three surgical extensions to the multi-format tool-call parser, each covering a real fine-tune / template emission shape that the current parser silently drops. No path narrows; all changes widen what is accepted. 1. `_parse_tool_call_json` now accepts both `arguments` and `parameters` keys. A Hermes / Qwen `{json}` wrapper around a Llama-3.2 fine-tune that emits the `parameters` key was extracting the tool name and silently discarding the args, producing a working-shaped call with an empty payload. The bare-JSON and python_tag paths already accepted both keys; this path now matches them. 2. `_TC_FUNC_START_RE`, `_TC_PARAM_START_RE`, and `_TC_PARAM_CLOSE_RE` now also match the attribute form `v` used by MiniCPM-5 and MiniMax-M2. Names land in either capture group, and `` is accepted as a short close. 3. `_parse_llama3_bare_json` sentinel-strip now consumes the role label inserted between `<|start_header_id|>` and `<|end_header_id|>` by Meta's official Llama-3.x chat template. Without this, every assistant turn re-fed through the template prefix `<|start_header_id|>assistant<|end_header_id|>\n\n{json}` parsed to zero calls, so any history-with-tool-call round-trip in production silently dropped. Tests in `studio/backend/tests/test_safetensors_tool_loop.py`: * `TestParserRobustness::test_tool_call_json_accepts_parameters_key` * `TestParserRobustness::test_function_xml_attribute_form` * `TestParserRobustness::test_function_xml_attribute_form_multi_param` * `TestParserRobustness::test_function_xml_legacy_equals_form_still_works` (regression guard for the existing `` syntax) * `TestParserRobustness::test_llama3_chat_template_round_trip` * `TestParserRobustness::test_llama3_round_trip_all_roles` * `TestParserRobustness::test_llama3_round_trip_with_eot_prefix` `pytest studio/backend/tests/test_safetensors_tool_loop.py studio/backend/tests/test_safetensors_capability_advertise.py -q` goes from 118 to 125 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim verbose comments in tool-call parser sections for PR #5624 Pure comment / docstring tightening on top of the GLM 4.7 + Kimi multi-section fixes. No behavioural change. * Drop multi-paragraph prelude and post-refactor citation chatter in the DeepSeek, GLM and Kimi parser docstrings; keep the shape and upstream-commit pin. * Collapse ``parse_tool_calls_from_text``'s 9 per-family blocks into a single ordered loop with one combined comment. * Tighten the GLM coercion, Kimi bare-counter and ``_TOOL_XML_RE`` comments to one or two lines each. * Same trim pass on ``_PARSER_MARKERS`` and the regression-test docstrings. Tests: pytest studio/backend/tests/test_safetensors_tool_loop.py studio/backend/tests/test_safetensors_capability_advertise.py studio/backend/tests/test_pr5624_regressions.py -q -> 174 passed in 2.00s * Fix O(N^2) DeepSeek V3.1 backtracking for PR #5624 Adversarial input ``<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>fn<｜tool▁sep｜>`` followed by a long body that does NOT contain a closing brace caused the V3 path's ``([^\n<]+?)<｜tool▁sep｜>`` regex to backtrack quadratically: at each position the lazy quantifier extends one char at a time looking for a sep that isn't there, taking ~19s on 50k chars. Replace the regex search with ``str.find`` on the sep marker plus a left-walk to recover the name. ``str.find`` is O(N); the walk stops on ``\n`` (turn boundary), ``<`` (start of a tag), or ``>`` (end of an optional ``<｜tool▁call▁begin｜>`` prefix). Same observable behaviour as the regex on every canonical input. Tests: test_deepseek_v3_1_huge_truncated_body_is_linear (new) -- 50k chars must parse in < 1s. pytest studio/backend/tests/test_safetensors_tool_loop.py studio/backend/tests/test_safetensors_capability_advertise.py studio/backend/tests/test_pr5624_regressions.py -q -> 175 passed in 1.97s pytest studio/backend/tests/ -q -k 'not gpu and not llama_cpp_integration' -> 2038 passed, 15 pre-existing failures unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: terminate function-XML body at , not just `_parse_function_xml` was looking for `` (the Hermes wrapper) as the body terminator. When a model emits a standalone `v` followed by explanatory prose (which models routinely do), no `` is present, so the body extended to end-of-string and the trailing prose leaked into the LAST parameter value. Pre-existing on main (the legacy `` form had this bug too). Same affects PR #5620's new attribute-form `v` emission used by MiniCPM-5 / MiniMax-M2. Fix: `_TC_END_TAG_RE` now matches either `` OR ``. The existing `_TC_FUNC_CLOSE_RE` / `_TC_PARAM_CLOSE_RE` strips are unchanged. Multi-call inputs still bound each function at the next `` is preserved because the embedded close tag is ``, not ``). `pytest studio/backend/tests/test_safetensors_tool_loop.py studio/backend/tests/test_safetensors_capability_advertise.py -q` goes from 125 to 127 passed. * Studio: tighten Llama-3.2 bare-JSON guard A fuzz pass on PR #5811 turned up that ``_parse_llama3_bare_json`` accepted ``parameters`` as a string, contradicting the docstring's "parameters or arguments is a dict" guard. Prose JSON like ``{"name":"foo","parameters":"a sentence"}`` would wrongly fire the parser, which the agentic loop would then heal into a real ``foo(query="a sentence")`` call. Same code lives on this branch, so the same fix applies here. Tightened guard: - ``parameters`` must be a dict (Llama-3 spec). - ``arguments`` may be a dict, or a JSON-encoded string that decodes to a dict (OpenAI shape, e.g. ``"arguments":"{\"q\":\"x\"}"``). Plain non-JSON strings or JSON-strings of lists / scalars / null no longer pass. Mirrors the fix landed in PR #5811 commit 615b8608. Adds the same 4 regression tests under TestParserMultiFormat. Existing test suite stays green: 127 -> 131 passing. * Studio: skip non-scalar args in python_tag JSON form The JSON sub-path of ``_parse_llama3_python_tag`` was fabricating ``{"value": args}`` when the model emitted a non-dict / non-string ``arguments`` value (e.g. ``42``, ``[1,2,3]``, ``null``, ``true``). This silently turned a malformed emission into a real tool call, which the agentic loop would then execute with arguments the model never intended. Tightened: skip the call instead of fabricating. The same behaviour now matches the bare-JSON guard tightened earlier (strict-guard merge from PR #5620, inherited via merge here). Added a regression test covering the four non-scalar shapes. Pass count on this branch: 158 -> 159. Sites in ``_parse_tool_call_json`` and ``_consume_mistral_call`` keep the existing looser behaviour for now; both are reached only after explicit ```` / ``[TOOL_CALLS]`` markers so the false-positive surface there is much narrower. * studio: fix safetensors tool-call parser gaps vs llama.cpp (Mistral CALL_ID / THINK, attribute-form signal) Three GGUF-parity fixes to the safetensors tool-call parser, each matching llama.cpp's reference behaviour: - Mistral Small 3.2 emits [TOOL_CALLS]name[CALL_ID][ARGS]{json}. The parser stopped after the name on seeing [CALL_ID] (neither [ARGS] nor {), dropping the call. Skip an optional [CALL_ID] segment in both the parse and strip paths. llama.cpp parses this (test-chat.cpp:4785). - Magistral wraps reasoning in [THINK]...[/THINK]. A [TOOL_CALLS] inside the reasoning was parsed as a real call, producing a phantom call. Strip a leading [THINK] block before scanning so only the post-reasoning call counts (test-chat.cpp:2285); a literal [THINK] inside a later argument is left intact. - The standalone MiniCPM-5 / MiniMax-M2 attribute form parsed correctly but was absent from TOOL_XML_SIGNALS and the markup strip patterns, so the streaming safety-net parse was gated off (dropping the call) and markup leaked into displayed text. Add the signal and broaden the strip regexes. Adds regression tests for all three. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: fix GLM and Kimi K2 safetensors tool-call parser gaps vs llama.cpp Four GGUF-parity fixes for the GLM and Kimi K2 families: - GLM 4.7 zero-argument inline call name was dropped: the open-tag lookahead only allowed \n or after the name. Allow too so a no-arg call parses to empty args (vLLM / SGLang / llama.cpp all parse it). - GLM string argument values were stripped, losing significant leading / trailing whitespace in code / diff arguments. Keep the raw value for the string fallback and only strip the copy used to probe for a JSON literal, matching vLLM glm4_moe which never strips string args. - Kimi K2 calls emitted without the <|tool_calls_section_begin|> wrapper were dropped. llama.cpp makes the section optional (Kimi can call a tool straight after reasoning without opening a section); parse a bare <|tool_call_begin|> when no section is present. - Kimi K2 malformed / truncated JSON in one call dropped every later call in the section. Skip the bad call and keep parsing so valid subsequent calls are recovered (vLLM parity). Adds regression tests for all four. * studio: fire safetensors tool calls for the bare-JSON (Llama-3.2) form The agentic loop's streaming safety-net parse was gated on has_tool_signal(), which is False for the Llama-3.1 / 3.2 bare-JSON tool form {"name":..,"parameters":..} (no XML marker). Real tool calls were therefore dropped: the loop logged "model planned without calling tools", re-prompted three times, then gave up with zero tool calls, while GGUF's llama-server parses the same emission natively. Run parse_tool_calls_from_text() unconditionally in the safety net. The parser is strict (only fires on a valid tool-call shape) so plain answers are unaffected. Reproduced on a real unsloth/Llama-3.1-8B-Instruct run: the model emits {"name":"web_search","parameters":{...}} which now executes the tool instead of being re-prompted into a no-op. Adds a loop regression test for the bare-JSON form. * studio: fire safetensors tool calls for Gemma 4 (native template + stripped parser) Gemma-4 safetensors fired no tools while its GGUF fired reliably. Three gaps: - The Studio swaps in the Unsloth "gemma-4" chat template, which does not render the tools schema (the model's native template does), so the model never saw the tools. Fall back to the model's native template when the override template renders identically with and without tools. Same fix helps any family whose override template drops tools. - skip_special_tokens strips the <|tool_call> wrapper and <|"|> string markers, so a streamed Gemma-4 call arrives as a bare call:NAME{k:v, ...} with unquoted values. Parse that form, keeping commas/braces inside a code or command value, normalising surrounding quotes, and stripping the leaked markup from the final answer. - Without a grammar a small model can loop, repeating one call for the whole tool budget. Collapse exact-duplicate calls within a turn and force a final answer after a turn that made no new tool progress (llama-server's lazy grammar prevents this loop on the GGUF side). Adds parser tests for the bare/stripped Gemma-4 form. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: complete strict-mode contract and fix parser import paths Address review findings on the multi-format tool-call parser: - Honor allow_incomplete=False in the remaining sub-parsers. The Llama-3 <|python_tag|>NAME.call(...) parser, the pre-v11 Mistral [TOOL_CALLS] array parser, and the Gemma 4 <|tool_call> parser ignored strict mode, so a truncated call (missing closing paren, ], or ) was still healed and executed with Auto-Heal disabled. Thread strictness through and reject the unclosed forms, matching the JSON and function-XML paths. - Drop the duplicate tool_call_parser import block in llama_cpp.py and the redundant un-aliased TOOL_XML_SIGNALS; only the _SHARED_TOOL_XML_SIGNALS alias is used as a value. - Import _strip_mistral_closed_calls from core.inference.tool_call_parser in routes/inference.py instead of studio.backend.core... The self-contained run.py launch mode only puts studio/backend on sys.path, so the absolute package path raised ModuleNotFoundError on the server-tool strip path. Add strict-mode regression tests for the truncated Llama-3 dot-call and the unclosed Mistral array. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden DeepSeek/Kimi tool-call parsing and strip Address review findings on the DeepSeek and Kimi parsers: - Honor allow_incomplete=False for DeepSeek. An envelope with no closing <｜tool▁calls▁end｜> is truncated mid-stream; reject it in strict mode instead of healing the body out to EOF, matching the strict XML and Mistral paths. - Do not skip a following tool call when the current call's end marker is missing. The DeepSeek V3 and Kimi loops advanced by searching forward for the next <｜tool▁call▁end｜> / <|tool_call_end|>, which could land on a later call's end marker and drop the call in between. Advance by the JSON end; the loop re-locates the next call marker from there. - Strip truncated DeepSeek and Kimi section blocks in the route-level display regex. The patterns required the closing marker; add the end-of-text alternative so a block truncated by EOS does not leak raw markup to the UI. Add regression tests for the truncated DeepSeek envelope, and for DeepSeek and Kimi multi-call recovery when the first call's end marker is missing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: preserve XML param indentation and alias Mistral array parameters Two parser-correctness fixes found by auditing against the model chat templates and the SGLang / vLLM reference parsers: - Qwen3.5 XML parameter values lost their leading indentation. The chat template emits \nVALUE\n, but the parameter-start regex ate the wrapping newline AND the value's first-line indentation with a trailing \s*, then str.strip() removed the rest. Narrow the trailing class to horizontal whitespace only and trim exactly one wrapping newline (via _trim_param_value), preserving indentation in code/diff arguments. Matches SGLang's qwen3_coder detector. Applies to both _parse_function_xml (tool_call_parser.py) and the XML path in tool_healing.py. - Mistral pre-v11 array objects keyed on parameters dropped their payload. _consume_mistral_call read only the arguments key; alias parameters the same way the JSON/XML paths and SGLang's base detector do. Add regression tests for preserved multi-line indentation and the array parameters alias. * Studio: DeepSeek strip sync, Gemma nested args, GLM/Kimi strict mode Parser-correctness fixes found by auditing DeepSeek/GLM/Kimi against vLLM, SGLang, and the model chat templates: - DeepSeek: the short <｜tool▁calls｜> opener (and the space / escaped-underscore spellings) was parsed but never stripped, so a short-opener envelope leaked raw markup to the UI. Share one opener alternation between _DEEPSEEK_BEGIN_RE and the strip patterns (and the route-level display regex) so a signal we parse can never be left un-stripped. - Gemma wrapper-less stream: a nested object/array argument (loc:{city:NYC}, labels:[bug,ui]) was kept as a literal string. Parse it recursively when the bare value is a balanced {} / [], falling back to the raw string for a truncated value. - GLM and Kimi ignored allow_incomplete. With Auto-Heal off, a GLM block with no , a Kimi section with no <|tool_calls_section_end|>, or a Kimi call with no <|tool_call_end|> are truncated and must be rejected, matching the strict behavior of the JSON/XML/Mistral/DeepSeek paths and vLLM/SGLang. Add regression tests for the short-opener strip, the Gemma nested args, and GLM / Kimi strict-mode rejection. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten tool-call parser comments Make the comments in the multi-format tool-call parser and its callers succinct: compress verbose docstrings/blocks to one or two lines, drop ones that restate the code, and trim the tiny balanced-scanner helpers. Correctness rationale and upstream provenance (SGLang/llama.cpp parity, the strict-mode / Auto-Heal contract, whitespace-preservation, and the Unicode / full-width-pipe notes) are kept in compact form. Comment-only: no code or behavior change (verified with comment_tools.py check --strip-docstrings; parser suite green). * Studio: tighten DeepSeek/GLM/Kimi parser comments Compress the comments added for the DeepSeek/GLM/Kimi parsers and the Gemma wrapper-less helpers to one or two lines, keeping the upstream provenance (llama.cpp 51fa458a92d6), the O(N^2) / strict-mode rationale, and the vLLM parity notes intact. Comment-only: no code or behavior change (verified with comment_tools.py check --strip-docstrings; parser suite green). * Studio: make DeepSeek R1 / GLM parsing linear and close routes strip gaps Review follow-up for the DeepSeek/GLM/Kimi parser: - DeepSeek R1 detection used a greedy ``([^\n]+)\n```json`` regex that backtracks O(N^2) on a fence-less truncated body; scan with str.find instead (mirrors the V3 path). - GLM arg pairs used a lazy-group finditer that rescanned to EOF from each bare in an unclosed body (O(N^2)); walk pairs with str.find. - The route display strip (_TOOL_XML_RE) accepted fewer DeepSeek openers than the parser (missed the space / escaped-underscore spellings) and missed bare section-less Kimi calls, so a call we parse could leak raw markup to the UI. Reuse the parser's shared _DEEPSEEK_OPEN_RE_SRC and add a bare-Kimi arm. Add ReDoS-linearity regressions for the R1 and GLM paths, a positive R1 fenced-json parse test, and routes-strip tests for the space/escaped DeepSeek openers and the bare Kimi call. * Studio: fix test_mcp_servers _TOOL_XML_RE reconstruction after _DS_OPEN_SRC reuse The routes strip fix made _TOOL_XML_RE reference the module-level _DS_OPEN_SRC variable. test_mcp_servers reconstructs the regex by exec-ing the extracted compile() source in a namespace that only defined _re, so it raised NameError. Inject _DS_OPEN_SRC into that namespace, matching the same fix already applied in test_tool_xml_strip. * Studio: make Llama-3 .call and Mistral-array healing parsing linear Two more O(n^2) ReDoS paths in the multi-format parser, both reachable from the agentic loop on a long truncated body with no length cap: - _LLAMA3_KV_RE.finditer over a .call(...) body retried at every offset of a long word run / unterminated quote (40K -> 14s). Replace with a hand-scan that reuses the same key/number/literal sub-regexes via anchored match and walks the string body by hand, so an unterminated quote is O(n). Verified byte-identical to the old regex over 200K fuzzed inputs. - _parse_mistral_array healing ran _balanced_brace_end from every { in the body (20K -> 17s). Walk top-level objects, advancing past each balanced {...}; this also drops the phantom call the old scan emitted from a nested argument object. Add adversarial-length linearity regressions plus positive .call kwargs and unclosed-array recovery coverage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: strengthen #5624 regression assertions and strip-test harness guards - test_strip_tool_markup_handles_deepseek_envelope used `A or B` where B was the preservation property the next line already asserts, masking the real check. Replace with an explicit assertion that the call name and args are stripped. - The test_tool_xml_strip source-extraction harness reconstructs _TOOL_XML_RE and _strip_tool_xml_for_display from routes/inference.py via lazy regexes that could silently grab a shorter slice. Assert the extracted regex carries the DeepSeek / bare-Kimi arms and the helper body reached the _TOOL_XML_RE.sub call. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: honor strict mode in safety-net, keep empty Gemma args, strip attribute-form function XML - safetensors safety-net parser now forwards allow_incomplete=auto_heal_tool_calls, matching the draining path, so a late incomplete tool call is not healed and executed when Auto-Heal is off. - Gemma empty bare value ({k:}) now serialises as "" instead of invalid {"k":}, which previously dropped the whole call. - Route _TOOL_XML_RE also strips the attribute form (MiniCPM-5 / MiniMax-M2) so it no longer leaks to the UI. * Studio: linearize wrapper-less Gemma nested-arg parsing and correct parser provenance - _gemma_parse_value/_gemma_parse_mapping/_gemma_parse_array now parse nested {}/[] in a single forward pass instead of pre-scanning each subtree with a balanced-brace walk and re-parsing it. Deeply nested wrapper-less Gemma args were O(n^2); they are now ~linear (and ~40x faster at depth 400). - Correct the DeepSeek/GLM/Kimi provenance comments: the cited commit 51fa458a92d6 is unrelated, and GLM/Kimi were never standalone common_chat_parse_* functions (llama.cpp uses common_chat_params_init_glm_4_5 plus a generalized XML parser, PRs #15904 / #16932). - Add tests: Gemma deep-nesting linearity, nested object/array preservation, same-turn distinct-call cap, and the native-template tool-render fallback. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: guard Gemma value parser against non-advancement and missing tokenizer Addresses Gemini review: - _gemma_parse_value now consumes one character when a stray }/]/, sits where a value is expected, so _gemma_parse_array can never stall at the same index on malformed input (a latent infinite loop). - _render_with_native_template returns None when neither a tokenizer nor a processor is present instead of raising AttributeError. - Tests for both. * Studio: fix attribute-form function-XML literal close tag and zero-arg strict call Addresses Codex review of the attribute form in _parse_function_xml (MiniCPM-5 / MiniMax-M2): - End the call body at the LAST / within the call's window, so a literal close tag inside a code/search argument (e.g. print("")) is preserved instead of truncating the call. - Accept a closed call with no parameters as a valid zero-argument call in strict mode (the function close is already required), instead of rejecting it as a truncated call. - Tests for both, mirroring the legacy coverage. * Studio: drop scratch review/planning artifacts from the branch * Studio: fix tool-call parser/loop review findings on the multi-format path Address the live code-review findings on the safetensors/MLX + GGUF tool path: - routes: include the attribute form in the safetensors capability whitelist so MiniCPM-5 / MiniMax-M2 templates keep the tool pill (parser already handles the form; the post-filter wrongly suppressed it). - safetensors loop: build the plan-without-action re-prompt from the active tools instead of a hardcoded web_search/python string, and gate it on auto_heal_tool_calls, matching the GGUF loop. - safetensors loop: hold a leading bare-JSON object ({"name":..,"parameters":..}) during BUFFERING until it closes, then drain it as a tool call instead of streaming the raw JSON to clients. The DRAINING/STREAMING resolvers still recover a plain JSON answer, so this can never drop content. - parser: anchor the Llama-3 <|python_tag|>NAME.call(...) scan to the tag and chain ; -separated calls, so all semicolon-separated built-ins parse and a literal <|python_tag|>x.call(...) inside a JSON string argument no longer fires the wrong tool. - parser: consume the optional trailing after a named Mistral [TOOL_CALLS]name{json} call, mirroring the array shape. - GGUF streaming strip: use the shared parser patterns (which know [TOOL_CALLS] and <|python_tag|>) so a textual tool call entering DRAINING is stripped instead of leaking the marker to streaming clients. - routes: hoist the _strip_mistral_closed_calls import to module level. Adds regression tests covering each fix; existing parser suite stays green. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fix DeepSeek/GLM/Gemma tool-call review findings Address the live code-review findings specific to the DeepSeek / GLM / Kimi and native-template additions: - parser: in strict mode (Auto-Heal off) require the per-call <｜tool▁call｜end｜> terminator for DeepSeek V3 calls instead of executing on a bare balanced object closed only by the envelope end. - parser: keep GLM string arguments that begin with a quote verbatim (drop the leading-quote case from the JSON-decode probe) so a quoted search query is not decoded down to its inner text. - parser: reject a GLM call with an unclosed in strict mode, and under Auto-Heal keep the partial value rather than dropping it to a no-arg call. - parser: add a balanced wrapper-less Gemma strip (call:NAME{...}) so a nested object/array argument is removed whole instead of leaving a trailing brace; run the balanced Mistral and Gemma strips on the streaming display paths too. - safetensors loop: buffer a leading wrapper-less Gemma call:NAME{...} so it drains and executes instead of streaming the raw call text. - inference: render the native-template fallback on a shallow tokenizer copy instead of mutating the shared tokenizer outside the generation lock, and load the native template from base_model for LoRA adapters. Adds regression tests for each; existing parser suite stays green. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden multi-format tool-call detection from review findings Apply five targeted fixes from the review pass over the multi-format tool path: - routes: route display strip delegates to _strip_tool_xml so Mistral [TOOL_CALLS] blocks with nested JSON are removed from streamed display text, not just the XML forms. - tool_call_parser: skip function/parameter starts that fall inside an already-open parameter block (_inside_open_parameter) so nested example payloads are not mis-parsed as new calls; extract strip_llama3_leading_sentinels so the bare-JSON guard is shared. - safetensors_agentic: probe bare JSON through strip_llama3_leading_sentinels before the balanced-brace check so a leaked header sentinel does not defeat the guard. - tool_healing: allow dotted tool names in the Gemma wrapped start pattern. - llama_cpp (GGUF): buffer wrapper-less Llama-3.2 {"name":..} calls that carry no XML signal, drain a complete object silently and hold an incomplete one, and run the end-of-stream safety net unconditionally so markerless calls are detected and never leak the raw JSON (including truncated fragments). Adds regression tests for the GGUF bare-JSON streaming path and the Mistral display strip. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop bare-JSON tool calls leaking at EOF, oversized, and into history The second review pass flagged that the Llama-3.2 bare-JSON tool-call handling still leaked raw JSON in several spots; ``strip_tool_markup`` only knows XML/bracket markup, so the bare-JSON form survived it. Fix them symmetrically across the safetensors and GGUF loops: - Safetensors stream-end resolver now routes a held bare-JSON fragment to DRAINING (mirroring GGUF) so a truncated ``{"name":..`` cut off by the end of the stream is dropped instead of flushed as assistant content. The 7/10 reviewer finding. - Both loops now drain (suppress) an oversized still-open bare-JSON call once it passes ``_MAX_BARE_JSON_BUFFER`` instead of streaming the raw prefix, gated on a ``"name"`` key so a giant plain JSON answer still streams; a complete oversized call still executes via the safety net. - Add a shared ``strip_leading_bare_json_call`` helper and apply it to the content kept for the assistant turn in both loops, so an executed bare-JSON call is not replayed as visible text or fed back as next-turn history. Plain JSON answers without a ``"name"`` key are untouched throughout. Adds regression tests for the EOF, oversized, and next-turn cases on both backends plus unit tests for the helper. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: bound the Llama-3 python_tag strip on real control sentinels The route display strip's <|python_tag|> arm ran to the next <| of any kind. A tool-call argument carrying a literal <|...|> token (for example <|cite|> inside a string value) truncated the strip early and leaked the call tail into the visible response. Narrow the stop condition to the genuine Llama control sentinels (eot_id, eom_id, python_tag, start/end_header_id, begin_of_text, finetune_right_pad_id) so embedded markup and JSON are consumed while real header/turn boundaries still bound the strip. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden GLM/Gemma parsing, cap GGUF textual calls, share native-template fallback GLM 4.x parser walked a body pre-bounded by the first , so a string argument containing a literal (e.g. code that prints it) was truncated. Walk arg_key/arg_value pairs against the full content instead, since each is delimited by its own and the call's real close is the that precedes the next . Add a truncated wrapper-less Gemma pattern (call:NAME{... with no closing brace) to the markup strip so a call cut off mid-arguments does not leak raw into the visible stream. It runs after the closed form, so a complete call keeps trailing prose. Cap and dedup tool calls parsed from the GGUF TEXTUAL fallback at _MAX_TOOL_CALLS_PER_TURN, mirroring the safetensors loop. Structured delta.tool_calls are grammar-bounded by llama-server, but text parsed straight from content is not, so one runaway turn could fan out into dozens of executions. Extract the native-chat-template fallback into chat_template_helpers (render_native_template / render_with_native_template_fallback) so the transformers and MLX text backends share one implementation. The MLX text path now applies it too, so an Unsloth override template that drops the tools schema no longer silently stops MLX from advertising tools. The MLX VLM path renders via the processor for image tokens and is intentionally left on its own render. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate markerless bare JSON on enabled tools and close parser/strip asymmetries The Llama-3.2 custom_tools bare-JSON form has no marker, so any JSON object with a name key was read as a tool call. An ordinary JSON answer like {"name":"Alice","parameters":{"age":30}} was misclassified as a call to a disabled tool and dropped from the visible response. Gate the markerless form on the enabled tool names (threaded through parse_tool_calls_from_text and strip_leading_bare_json_call, supplied by both streaming loops): an object whose name is not an enabled tool is ordinary content. The marker-based forms keep their name-agnostic behaviour (an explicit signal is a real call attempt), and unrestricted mode stays ungated. Also fix two parser/strip asymmetries the parser already tolerated: - A literal inside a parameter value (print("")) truncated both the core and route strips at the first close, leaking the tail. Extend the strip to the call's real close (last before the next opener), mirroring the parser, without merging separate calls. - The single-object Mistral [TOOL_CALLS]{...} shape parsed but _strip_mistral_closed_calls left it, leaking the raw object into display. Strip the balanced object while keeping trailing prose, matching the array and name shapes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio tools: fix strip/parse symmetry and native-template token for DeepSeek/GLM/Kimi Pass-3 review follow-ups on the multi-format tool parser: - Bare Kimi call (<|tool_call_begin|>...<|tool_call_end|> with no section wrapper) is accepted by the parser, so add it to the closed strip patterns so the streaming (non-final) display strip removes it instead of leaking the markup mid-generation. - Route display strip now also runs the wrapper-less Gemma cleanup, so a Gemma 4 call:NAME{..} no longer leaks into the visible answer. - MLX model record carries base_model for a LoRA adapter so the native-template fallback loads the base repo template rather than the adapter's (often template-less) tokenizer. - Native-template reload forwards the load-time HF token so a gated/private model's repo template can still be fetched (transformers and MLX text paths). - GGUF end-of-stream bare-call heuristic is gated on the enabled tool names so a truncated ordinary JSON object ({"name":"Alice","age":) streams as the answer instead of being dropped as a tool call. Adds regression tests for each case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio tools: gate GGUF bare-JSON suppression on enabled tools and fix python-tag exponent parsing Pass-4 review follow-ups on the GGUF tool loop and Llama-3 parser: - The GGUF bare-JSON suppression sites still keyed off a raw "name" substring, so an ordinary JSON answer whose name is not an enabled tool was dropped when it was truncated, oversized, or reached the no-tool DRAINING fallback (the parser, helper, and safetensors paths were already gated). All three sites now use the shared enabled-name gate, and a held bare-JSON buffer that turns out not to be an enabled call is shown as the answer instead of dropped at stream end. - The Llama-3 python-tag numeric kwarg regex matched only the mantissa, so scientific notation was truncated to its leading digits (1e-3 parsed as 1) and a tool executed with the wrong value. The regex now accepts exponent and decimal forms, and the int/float classification keys off the exponent too. Adds regression tests for the truncated / oversized disabled-name JSON cases (and a counterpart that a truncated enabled call still does not leak) plus the scientific-notation kwargs. * Studio: drop accidentally committed async worker transcripts Eight generated reviewer / async-worker transcripts were committed under studio/backend/async_task_outputs/. They are not imported or referenced by any code and carry only internal task state, so they should never ship in the repo. Remove them and gitignore the directory so they cannot be re-added. * Studio tools: gate safetensors bare-JSON drain, fix nested-name gate and function-XML strip Pass-4 review follow-ups on the shared parser / safetensors loop: - The safetensors oversized and end-of-stream bare-JSON drain branches keyed off a raw "name" substring, so a large or truncated ordinary JSON answer whose name is not an enabled tool was drained instead of streamed. Both now use the shared enabled-tool-name gate, matching the GGUF path. - strip_leading_bare_json_call matched the first "name" anywhere, so a plain JSON answer with a nested name equal to an enabled tool ({"result":{"name":"web_search"}}) was wrongly suppressed. It now extracts the TOP-LEVEL name only, walking past nested objects/arrays and keeping the text when a top-level value is truncated. - The function-XML display strip used a regex negative-lookahead that stopped at a literal opener inside a parameter value and then dropped the rest of the answer to EOF. A scan-based strip mirrors the parser (ignores openers inside an open via _inside_open_parameter) and closes each call at its real , so trailing assistant text after such a call survives. Adds regression tests for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep tools prompt when native-template probe raises; make helper tests hermetic Pass-4 review follow-ups on the native-template fallback: - render_with_native_template_fallback re-renders the live template with tools=None to detect whether it dropped the schema. A template that requires tools can raise on that probe; that must not discard the already-valid tools prompt. The probe is now wrapped so any error returns the original formatted_prompt (transformers would otherwise fall back to manual formatting and lose the schema; MLX would let the exception escape). - The native-template helper tests imported InferenceBackend just to reach the thin wrapper, which pulls in unsloth and its optional vllm package metadata. They now call the dependency-light render_native_template helper directly so they pass in a backend/test environment without vllm. Adds a probe-raises regression test. * Tool parsing: 3.9 import safety, disabled-Auto-Heal contract, capability gate Round-2 review follow-ups on the multi-format tool-call parser: - tool_call_parser: add `from __future__ import annotations`. The module is dependency-light by design (external llama-server wrappers import it standalone) and the package targets python >=3.9, where its PEP 604 `int | None` return annotations would raise TypeError on import. - safetensors + GGUF drain fallback: gate the leading bare-JSON strip on auto_heal_tool_calls. With Auto-Heal off, a truncated enabled-name fragment that did not parse now stays visible, matching the XML strip in the same branch and the disabled-Auto-Heal contract. With Auto-Heal on it is still suppressed. - safetensors capability gate: match the bare-JSON `{"name":` template marker with a whitespace/escape-tolerant regex so a pretty-printed `{ "name" :` or JSON-escaped `{\"name\":` template is not mis-classified as tool-less. The parser already accepts that whitespace via raw_decode, so the gate must too. Regression tests added for each case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GLM tool-call display strip: treat literal close tag in arg value as data Round-2 review follow-up on the GLM 4.x tool-call format. The GLM call shape is NAMEkv .... The parser was hardened to walk arg_key / arg_value pairs so a literal inside an argument value (e.g. print("")) is treated as data and the call's real close is the that precedes the next . The display strips still used a non-greedy .*? regex, which stopped at the literal and leaked the call's tail into visible content and stale history. Add _strip_glm_calls, a scan that mirrors the parser's close detection, and run it before the regex arms in every strip pipeline: the core strip_tool_markup, the route _strip_tool_xml display/history cleanup, and the safetensors + GGUF streaming strips. Qwen / Hermes {json} has no NAME token after the opener, so it is left to the regex arms unchanged. Regression tests cover the literal-close-tag leak (core + route), normal GLM calls, back-to-back GLM calls, zero-arg GLM, truncated GLM, and untouched Qwen. * Tool parsing: symmetric "function" bare-JSON alias and route strip parity Round-3 review follow-ups, all parser/strip symmetry fixes. - Bare-JSON "function" alias: the markerless parser accepts a call name via obj.get("name") or obj.get("function"), but the strip/gates only knew "name", so a {"function":} call executed while its raw JSON leaked. Teach _top_level_bare_json_name the alias (with "name" precedence and the same nested and truncated-name guards), and widen the guards in strip_leading_bare_json_call, the safetensors and GGUF _looks_like_enabled_bare_json gates, and the route capability marker regex. - Route display/history cleanup: strip a tail-only alias close (the parser accepts ...), and run the parser's guarded function-XML scan (_inside_open_parameter) before _TOOL_XML_RE so a literal nested inside an argument value does not truncate the strip and leak the tail. Regression tests added for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio tools: fix DeepSeek strict recovery, Kimi dotted names, Gemma spaced streaming Round 3 review fixes for the DeepSeek / GLM / Kimi tool-call parsing path. - DeepSeek R1 and V3/V3.1 strict parsing (Auto-Heal off): when a call is truncated (missing closing fence or terminator), skip it and keep scanning for later well-formed calls instead of breaking out and dropping the rest of the envelope. This matches the Kimi strict parser's recovery behaviour. - Kimi dotted tool names: keep the full name after stripping only the functions. prefix and :idx suffix, e.g. functions.mcp.server-list:0 stays mcp.server-list. The previous split on "." truncated dotted MCP names to their last segment. This matches current vLLM (tool_id.split(":")[0].removeprefix("functions.")) and SGLang (^(?:functions\.)?(?P[\w.\-]+):(?P\d+)$). - Gemma wrapper-less call streaming: hold the whitespace-tolerant prefix (call : NAME) in the streaming suppression buffer, matching the parser's _GEMMA_BARE_TC_RE, so the spaced spelling split across chunks is buffered instead of leaking as visible text. Applied to both the safetensors and llama.cpp streaming paths. - Remove dead _render_with_native_template method and the now-unused copy import from inference.py; the live path uses render_with_native_template_fallback. Adds regression tests for DeepSeek R1/V3 strict recovery, Kimi full dotted name preservation, and the Gemma spaced-call streaming suppression. * Studio tools: honor tool budget in GGUF loop and guard function-XML streaming strip Round 4 review fixes. Both are asymmetric-fix bugs where the final/steady path got a guard the analogous streaming/loop path did not. - GGUF tool-call budget: the safetensors loop counts real tool-call turns against max_tool_iterations (re-prompt stalls excepted), but the GGUF loop only bounded the turn count by the enlarged range (max_tool_iterations + _MAX_REPROMPTS). Since this PR raised _MAX_REPROMPTS from 1 to 3, a model that keeps making valid tool calls could run up to three extra tool rounds (with max_tool_iterations=1, four rounds instead of one). Add a _tool_iters_done counter that increments only when a tool actually executed in the turn, and stop once the caller's budget is spent so the post-loop final-answer nudge fires. A duplicate/disabled no-op turn is a correction turn (like a plan-without-action re-prompt) and does not consume budget, preserving the existing "already completed" re-prompt behavior. - Streaming display strip: the final strip runs the guarded _strip_function_xml_calls scanner (a literal inside a parameter value is data, not a nested call), but the GGUF and safetensors streaming strips still used only the open-ended regex arms. When a tool-call argument contained literal function markup, the regex tail ate everything to end-of-text and dropped the real trailing prose after the call's true . Run the guarded scanner (and the balanced Mistral strip) before the regex arms in both streaming paths so streaming and final display agree. Adds regression tests: GGUF valid tool calls respect max_tool_iterations, and the streaming strip keeps trailing prose after a function-XML call with a literal marker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio tools: safetensors tool budget counts only executed turns (GGUF parity) Follow-up to the GGUF budget fix. The safetensors loop charged max_tool_iterations per non-re-prompt iteration (iteration + 1 - reprompt_count), so a duplicate/disabled no-op turn spent a budget slot even though no tool ran. With a small cap this dropped real work: for max_tool_iterations=2, a model that made a valid call, repeated it (an internal no-op correction turn), then made a distinct valid call executed only the first -- the third turn was sent with no tools and the distinct call was ignored. Track whether a turn actually executed a tool (set on record_result) and count only those turns against the cap, matching the GGUF loop. A duplicate/disabled no-op is a correction turn -- like a plan-without-action re-prompt -- and no longer consumes budget, so the model still gets its "already completed" nudge and another tool-enabled turn. Adds a regression test for the small-cap duplicate-then-distinct-call flow. * Studio tools: fix stale Kimi dotted-name regression test test_pr5624_regressions.py still expected functions.my.tool:0 to resolve to the last segment (tool). The parser now preserves the full dotted name (my.tool) after removing only the functions. prefix and :idx suffix, matching current vLLM/SGLang so dotted MCP names like mcp.server-list survive. Update the assertion, name, and module docstring to the corrected contract (the raw id is still preserved on the call). * Studio: render the reasoning block for safetensors and MLX like GGUF enable_thinking chat templates (Qwen3/Qwen3.5/GLM) prefill an unclosed into the generation prompt, so the model emits only the closing then the answer. The safetensors/MLX chat stream emitted that as plain content, so the reasoning showed inline with no collapsible thinking block, while GGUF (which surfaces reasoning via reasoning_content) rendered one. This brings safetensors and MLX to parity. - _ResponsesReasoningExtractor gains a reasoning_prefilled mode that starts inside the reasoning block and splits on the first ; default False keeps GGUF and every existing caller byte-identical. It suppresses a stray re-emitted and holds partial markers back across chunk boundaries. - _sf_reasoning_prefill_mode gates the mode on reasoning being enabled for the request, an enable_thinking or enable_thinking_effort style, and the template actually using the standard / markers. Models with a bespoke reasoning channel (e.g. gemma's <|think|>/<|channel>) are excluded so their answer is never swallowed; gpt-oss (Harmony) and thinking-off requests are excluded too. - sf_tool_stream and stream_chunks (the latter also serves MLX) feed text through the extractor, emitting reasoning_content then content deltas, with a per-turn reset in the tool loop and a flush before each tool_start; only the visible delta reaches the monitor reply. The two non-streaming drains split reasoning_content the same way. - Tests: extractor prefilled mode (streaming and edge cases), the gate matrix including the gemma-style exclusion, and a route-replay of the tool-loop reasoning stream. * Studio: render the reasoning block for safetensors and MLX like GGUF enable_thinking chat templates (Qwen3/Qwen3.5/GLM) prefill an unclosed into the generation prompt, so the model emits only the closing then the answer. The safetensors/MLX chat stream emitted that as plain content, so the reasoning showed inline with no collapsible thinking block, while GGUF (which surfaces reasoning via reasoning_content) rendered one. This brings safetensors and MLX to parity. - _ResponsesReasoningExtractor gains a reasoning_prefilled mode that starts inside the reasoning block and splits on the first ; default False keeps GGUF and every existing caller byte-identical. It suppresses a stray re-emitted and holds partial markers back across chunk boundaries. - _sf_reasoning_prefill_mode gates the mode on reasoning being enabled for the request, an enable_thinking or enable_thinking_effort style, and the template actually using the standard / markers. Models with a bespoke reasoning channel (e.g. gemma's <|think|>/<|channel>) are excluded so their answer is never swallowed; gpt-oss (Harmony) and thinking-off requests are excluded too. - sf_tool_stream and stream_chunks (the latter also serves MLX) feed text through the extractor, emitting reasoning_content then content deltas, with a per-turn reset in the tool loop and a flush before each tool_start; only the visible delta reaches the monitor reply. The two non-streaming drains split reasoning_content the same way. - Tests: extractor prefilled mode (streaming and edge cases), the gate matrix including the gemma-style exclusion, and a route-replay of the tool-loop reasoning stream. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: don't force a tool re-prompt on a negated intent (safetensors parity) The safetensors _INTENT_SIGNAL claimed to mirror GGUF but was missing the negative lookahead, so a refusal like "I will not search the web for that" matched the "i will" intent and triggered the plan-without-action re-prompt (STOP... you MUST call a tool), overriding a valid no-tool answer. GGUF already excludes not/never. Add the same (?!\s+(?:not|never)\b) lookahead so both backends agree. Extends the intent parity test with negated refusals. * studio: parse the outer envelope before DeepSeek/Kimi markers embedded in its args parse_tool_calls_from_text ran the DeepSeek/Kimi marker pre-pass before the shared / parser. When a Qwen/Hermes call's argument contained literal Kimi/DeepSeek markup (for example a user asking the model to explain that syntax), the pre-pass matched the embedded marker and returned it, executing the wrong tool and dropping the real call. Skip the pre-pass when a or envelope opens before the first DeepSeek/Kimi marker, so the shared parser takes the outer call; a genuine marker-led call (no leading envelope) still goes through the pre-pass. Tests for the embedded-marker case and the control. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: trim redundant comments (comment-only, AST-verified) * Studio: trim redundant comments (comment-only, AST-verified) * Studio: prevent Gemma tool-parser DoS on stray delimiters _gemma_parse_value returned the input index unchanged when text[i] was a stray delimiter (,}]), so the list and mapping caller loops that advance on the returned index spun forever at 100% CPU on malformed input such as [},]. Advance past the delimiter so parsing always terminates. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: strip Magistral [THINK] reasoning from final display/history strip_tool_markup removed [TOOL_CALLS] and markup but left a leading Magistral [THINK]...[/THINK] block intact, so its bracket-form reasoning (not the the reasoning channel renders) leaked into the safetensors display and conversation history while GGUF/llama.cpp routes it natively. Drop the leading reasoning block at end-of-turn (final=True) via the existing _strip_mistral_reasoning helper; streaming is untouched. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep times in wrapper-less Gemma tool arguments The wrapper-less Gemma value scanner used _GEMMA_KEY_RE = [\w.\-]+ for keys, which also matches a digit-leading token, so a comma followed by a time or ratio inside a value (call:web_search{query:meet at 10:00, 11:00 tomorrow}) was misread as a new 11: key, truncating the query and injecting a bogus argument. Require keys to start with a letter or underscore, matching the identifier-start rule the wrapped path already uses (_GEMMA_NEXT_KEY_RE). Add a regression test. * Studio: treat markers/close-tags inside tool-call arguments as data Four parser correctness fixes where a valid argument string was mistaken for structure: - DeepSeek: find the envelope-end token outside JSON strings, so a query/code argument containing the literal token no longer truncates the body and drops the whole call. - GLM: locate the real as the one whose next token is / / end, so a value containing a literal (or ) is kept instead of executing the tool with corrupted arguments. - Attribute-form envelopes now count in the embedded-marker guard, so a DeepSeek/Kimi marker inside a parameter value does not hijack the outer call and run the wrong tool. - Wrapper-less Gemma call:NAME{...} is gated on the enabled tool names (parse and display strip), mirroring the Llama bare-JSON gate, so a disabled/example name in prose is not stolen as a call and the real answer is preserved. Add regression tests for each. * Gate route Gemma wrapperless strip by enabled tools; make Kimi section-end search string-aware Route-level display stripping now threads the enabled tool-name set into the Gemma wrapperless-call strip, so prose that mentions a disabled tool (call:foo{...}) is preserved while active tool calls are still stripped. This mirrors the parser-level gate already used in tool_call_parser. The Kimi section-end lookup now searches outside JSON string literals, so a section-end marker appearing inside an argument string no longer triggers a false truncation that drops a valid tool call. * Run DeepSeek/Kimi pre-pass when a closed tool-call example precedes a real block The marker pre-pass was skipped whenever any / opener appeared before the first DeepSeek/Kimi marker, even when that opener was a CLOSED syntax example in prose that ends before the real block. In that case parse_tool_calls_from_text skipped the DeepSeek/Kimi parsers and the genuine tool call was dropped while a phantom tool named in the example ran instead. Only treat a marker as embedded in a leading envelope when removing the closed outer / envelopes also removes every marker (the marker actually sat inside one). A marker left standing is a real call, so the pre-pass runs. The legitimate case of a marker inside a closed outer envelope's arguments is preserved. * Honor reasoning_effort none in safetensors prefill; strip Magistral reasoning while streaming Two safetensors/MLX reasoning fixes surfaced in review: _sf_reasoning_prefill_mode only checked enable_thinking, so an enable_thinking_effort (GLM-5.2) request that disables thinking via reasoning_effort=none (without enable_thinking=False) still began in prefilled- mode. A plain answer with no was then swallowed whole into reasoning_content and the visible response came back empty. Thread reasoning_effort into the predicate and treat none as disabled, mirroring _request_reasoning_kwargs. strip_tool_markup_streaming stripped tool markup but not the leading Magistral [THINK]...[/THINK] bracket block, so the raw chain-of-thought leaked into the streamed safetensors content instead of the reasoning drawer (GGUF routes it natively). Apply _strip_mistral_reasoning first, matching the final strip; an unclosed [THINK] is held from the marker on so nothing flickers. * Heal truncated outer tool envelopes and keep quoted Gemma args intact Two follow-ups from review of the marker pre-pass and Gemma parsing: The leading-envelope guard only removed CLOSED outer / envelopes before deciding whether a DeepSeek/Kimi marker was embedded, so a truncated outer call missing its close tag (whose argument embeds a marker) was treated as a standalone marker and the embedded sample ran instead of the intended outer call being Auto-Healed. Decide on the last outer opener before the marker and whether it closed before the marker instead, so a closed syntax example still runs the pre-pass while a real closed-or-truncated outer call keeps it. The wrapper-less Gemma argument scan tracked bracket depth but not quotes, so a quoted value containing a comma followed by a key-like token (a search query such as "weather, location: Boston") was split mid-string, truncating the value and fabricating an extra argument. Track quote state (with escapes) so the top-level comma boundary is only taken outside quoted spans. * Span outer envelopes to their real close when locating embedded markers Locating the DeepSeek/Kimi marker relative to a leading outer envelope used the FIRST close tag after the opener, so a literal or inside an argument value (for example python code that contains the text) was mistaken for the envelope boundary. The marker after it was then treated as a standalone call and the embedded sample ran instead of the intended outer call. Match the closed outer envelopes with the shared patterns that already extend to the real final close (a literal close inside a value is data), and treat a marker that survives their removal as embedded only when a still-open (truncated) outer opener precedes it, so Auto-Heal still repairs a truncated outer call. A closed syntax example before a genuine block still runs the pre-pass. * Span the tool_call outer envelope to its real close in the marker guard The leading-envelope check reused the lazy .*? strip pattern, so a Qwen/Hermes JSON argument containing a literal ended the span early. A DeepSeek/Kimi sample later in that same string then survived the closed-envelope removal, and the pre-pass executed the embedded call instead of the outer . The arm already spanned to its real close; give the same real-close pattern (with the negative lookahead that keeps back-to-back calls separate) so a literal close inside a value is data. * Preserve no-tool Gemma prose and keep later R1 calls when healing a close Two review follow-ups: _gemma_strip_gate returned None when no tools were enabled, and None means strip every markerless call:NAME{...} block, so a no-tool answer that documents the syntax (or the Anthropic display path, which passes an empty tool list as None) had that prose deleted. It is a display/history gate, so return the enabled-name set instead -- an empty set when no tool is enabled, which strips nothing because every call:NAME{...} is then prose. The DeepSeek R1 heal path located the close fence with an unbounded forward search, so when a first call had balanced JSON but omitted its fence the search landed on a LATER call's terminator and pos advanced past that valid call, dropping it. Match the close immediately after the JSON (whitespace-skipped) like the strict path, and advance by just the JSON when it is absent, so a multi-call turn keeps its later well-formed calls (heal is now a superset of strict). * Resume wrapper-less Gemma scan past a consumed call's balanced body The markerless call:NAME{...} scan used finditer, which resumes right after the opening call: token, so a nested call:OTHER{...} mentioned inside the first call's own quoted string argument (for example a web_search query that quotes the Gemma tool syntax) was re-matched and returned as a spurious second tool call, executing an unintended tool. Walk with a manual cursor that resumes after the outer call's balanced body (brace matching already skips quoted braces), so a call's arguments are never rescanned. Genuinely separate back-to-back calls and disabled/example prose are unaffected. * Mistral outer call wins over XML literals; align healer signals with its parser Two follow-ups on the shared-parser ordering after the healing-passthrough merge: - A well-formed [TOOL_CALLS] call whose JSON arguments quote tool XML parsed the literal instead of the outer call (executing the wrong tool). When the first XML signal sits inside a leading balanced Mistral body it is argument data, so the Mistral parser now runs first; an XML signal before the trigger keeps the normal order, so a [TOOL_CALLS] literal inside an XML call's arguments still stays data. - passthrough_healing buffered streams on the parser module's broadened signal list (now including <|python_tag|> and [TOOL_CALLS]) but promotes with core.tool_healing, which does not parse those forms: a streamed Mistral or Llama text call was held until finalization and flushed as prose. The healer keeps its own signal list limited to the formats it can promote, restoring immediate streaming for the rest. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: Gemma wrapper-less marker literals and quotes, GLM embedded close pair - The Gemma fallback deferral now keys on an actual wrapped opener (_GEMMA_TC_RE), not the wrapper literal anywhere in content: a wrapper-less call whose argument merely mentions <|tool_call> has nothing tool_healing can parse, and deferring it lost the call entirely (not executed and stripped from display). - New _gemma_body_brace_end boundary scanner honors single- and double-quoted strings like _gemma_parse_stripped_body, shared by parse and strip, so a quoted brace in a code argument (code:print('}')) no longer truncates the executed arguments or the strip span. - _glm_value_close now requires a structural to sit at balanced quote state: the full pair embedded inside a string literal is data, not an early close. When no candidate balances, the first token-valid close wins as before. * Address review: leading envelopes win over rehearsed literals - New _first_foreign_tool_signal shared by the leading-envelope guards adds <|python_tag|> to the protected signal set: the spelled-out literal inside a Mistral call's arguments (a query about Llama built-in tool syntax) executed the inner literal instead of the outer call. - New _xml_signal_inside_leading_bare_json guard, sibling of the Mistral one: a leading bare-JSON call whose string argument quotes tool XML (a code value citing ) had the literal promoted by the shared XML pass before the bare-JSON parser ran. - Magistral [THINK]...[/THINK] is dropped once at parse entry instead of only inside the Mistral parser, so a call rehearsed in the think block in a foreign format can no longer be promoted while the real call after the block is lost. Parse now agrees with the display strip. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: a disabled leading bare-JSON object keeps its literals as data When the leading bare-JSON object is ordinary content (name not an enabled tool), the guard proved the first tool signal sits inside it, so falling through to the XML/python_tag passes promoted quoted string data as a real call. Drop the object and parse only the tail: a real call after the object still parses, nothing inside it can be promoted. * Address review: apostrophes in raw Gemma values, GLM strict key contract, per-model template token - Quote openers in the wrapper-less Gemma boundary and body scanners now require value-start context (after : { [ ( , =): an apostrophe inside an unquoted value (query:what's the weather) opened quote mode, swallowed the real closing brace, and lost the whole call on common contraction queries. Quoted values keep hiding delimiters as before. - A GLM with no tag now rejects the call in strict mode, matching the unclosed-value contract, instead of executing the tool with the argument silently dropped; Auto-Heal keeps the lenient skip. - The native-template fallback reads the hf_token stored on the model record instead of the instance-wide last-load token, so a later token-less load cannot break template fetches for a previously loaded gated model (both the transformers and MLX backends). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: Mistral literals inside leading JSON, whitespace-tolerant wrapped Gemma opener - The leading bare-JSON guard now treats the [TOOL_CALLS] trigger as a foreign signal: the Mistral parser runs before the bare-JSON one, so a literal quoted inside the leading object's strings was promoted over the outer call (or over ordinary JSON content). - tool_healing's wrapped Gemma opener tolerates whitespace around call and the colon: sampling drift emits call: name{ and call : name{, and rejecting those lost the call entirely because no fallback re-parses the wrapped form. Strict mode still requires the closing tag. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: DeepSeek/Kimi markers inside leading JSON and Mistral envelopes stay data The DeepSeek/Kimi pre-pass runs before the outer-call parsers, and _marker_inside_leading_envelope only protected XML envelopes: a marker quoted inside a leading bare-JSON or Mistral call's argument strings was promoted as a separate no-arg call and the real outer call dropped. The guard now recognizes those two leading envelopes as well; standalone DeepSeek/Kimi calls keep parsing. * Address review: accept dotted Gemma argument keys in the key-quoting scanner The scanner quoted keys of [alnum_-] only, so a dotted key (user.name:...) was left unquoted, json.loads failed, and the whole wrapped call was lost (parse empty, strip wipes the markup). Dots now match the parser's own key/name charset. * Address review: a real DeepSeek/Kimi call after a disabled leading JSON object still parses DeepSeek/Kimi markers are foreign signals for the leading bare-JSON guard too: a marker literal inside a disabled leading object made the envelope guard skip the pre-pass for the whole message, so a real DeepSeek/Kimi call after the object was dropped. Routing the case through the guard's drop-and-parse-the-tail recursion reaches the real call while the literal inside the object stays data. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: leading Mistral call owns the turn, dotted keys after bare values - A LEADING parseable [TOOL_CALLS] call now runs the Mistral parser first unconditionally: literal XML in trailing prose after the call was promoted by the earlier shared XML pass, executing the quoted example instead of the real leading call. XML leading keeps the normal order. - _GEMMA_NEXT_KEY_RE accepts dots so a dotted key after a bare value (query:foo,user.name:bob) ends the value at the comma instead of being swallowed into it, matching the round-earlier key-quoting charset. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: a leading wrapper-less Gemma call owns the turn A quoted foreign literal inside a leading wrapper-less Gemma call's argument (a query citing another tool syntax) was promoted by tool_healing before the Gemma fallback ran, executing the quoted example and dropping the outer call. New leading guard, sibling of the Mistral and bare-JSON ones, gated on an enabled name since the form is markerless. Foreign markup leading keeps the normal order. * Fix merge resolution: restore both leading-guard test classes intact * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: markup quoted inside a nameless leading JSON answer stays data The leading bare-JSON guard required a top-level name, so a structured JSON answer quoting tool markup in its strings (a response_format turn documenting a tool's syntax) had the literal promoted by the later passes. A nameless leading object that parses as real JSON now routes through the same decline-then-parse-the-tail path; non-JSON braced prose keeps the old behaviour, and a real call after the answer still parses. * Address review: JSON answers stay data, nested Gemma quotes, earliest envelope, no failure caching - A whole-content JSON value is a structured answer: the markerless Gemma scan and its strip no longer promote or strip a quoted example of an enabled tool's syntax inside it. - Nested stripped-stream Gemma values now unquote quoted string leaves recursively, so {loc:{city:"New York"}} hands the tool New York, matching the top-level coercion. - The DeepSeek/Kimi pre-pass dispatches by earliest envelope opener, so a leading real call wins over a trailing example of the sibling format in either direction. - A failed native-template fetch is no longer cached as no-template: the next call retries after the model record's token is fixed or a transient Hub error clears; only definitive loads are cached. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: closed calls precede the marker pre-pass, truncated Gemma scan stops, quoted nested delimiters - A closed non-DeepSeek/Kimi call preceding the first DS/Kimi marker owns the turn: a trailing syntax example, or one quoted inside a wrapped Gemma argument, was promoted by the pre-pass and dropped the real leading call. Wrapped Gemma joins the outer-envelope pattern sets. - An unbalanced wrapper-less Gemma call now stops the scan (mirroring the strip contract) instead of resuming inside its own argument text, where a quoted enabled call would be promoted. - Raw-quoted strings in nested stripped-stream Gemma values hide delimiters, so {city:"New, York"} is one value instead of a split pair, returned unquoted like the top-level coercion. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: string-marker literals in wrapper-less args, mid-value quoted phrases - The wrapper-less deferral guard no longer keys on the <|"|> literal: a real call whose argument merely mentions the string marker was deferred to tool_healing, which has no wrapped opener to parse, losing the call. The wrapped-opener check alone owns the deferral. - Double quotes now also open at the start of a word, so a quoted phrase mid-value (query:find "weather, location: Boston", limit:3) hides its delimiters instead of splitting the value into garbage keys; apostrophes keep the value-start-only rule so contractions stay prose. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: strict GLM refuses in-quote close fallback, Gemma guard covers preambles - _glm_value_close gains a strict flag: a truncated value whose only close candidates sit inside a string literal rejects the call in strict mode (Auto-Heal keeps the lenient partial), restoring the strict contract the quote-aware fallback had weakened. - The leading wrapper-less Gemma guard no longer requires the call to open the response: a visible preamble before call:NAME{...} is the normal shape, and the quoted foreign literal inside the argument was promoted again in that shape. An enabled balanced call beginning before the first foreign signal owns it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: contextual GLM quote openers, disabled Gemma examples stay prose, JSON array answers - The GLM value-close quote tracker uses the same contextual openers as the Gemma scanners (single quote after punctuation context, double quote also at word start), so strict mode accepts a normal apostrophe value again while still rejecting a truncated value whose only close candidates sit inside a string literal. - A disabled wrapper-less Gemma call is prose by design, so a tool literal quoted inside it no longer promotes: the span is dropped for parsing and the tail parsed, mirroring the nameless-JSON guard. - Leading JSON ARRAY answers join the leading-JSON envelope guard, so a marker quoted inside a structured array response stays data. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align closed-envelope regression test with the document-order contract The test asserted the pre-round-13 behavior (trailing DeepSeek/Kimi block wins over a leading closed envelope) while the shipped rule is document order: the leading closed call owns the turn. Rename the test and assert the leading call so the suite matches the contract exercised by test_leading_xml_call_wins_over_trailing_kimi_example. * Parse a leading Llama-3.2 bare-JSON call before the markerless Gemma scan The bare-JSON form only ever matches a leading call object, and document order says that call owns the turn. Running the Gemma wrapper-less scan first let an enabled call:NAME{...} snippet quoted inside the leading call's string arguments steal the turn when the JSON was not the whole content (trailing prose or a second ;-separated call), executing the quoted tool instead of the real one. Reordering cannot take a leading Gemma call's turn since that content never starts with an object brace. * Leading-call ownership: Mistral trigger in Gemma guards, closed bare JSON before markers, depth-aware nested Gemma values Three parser gaps against the document-order contract: The wrapperless Gemma leading guards did not count [TOOL_CALLS] as a foreign signal, so a leading Gemma call quoting a Mistral snippet in its argument lost the turn to the quoted literal. Both the enabled-call and disabled-example guards now include the trigger, matching the bare-JSON guard's local inclusion. _marker_inside_leading_envelope required the DeepSeek/Kimi marker to sit inside the first closed bare-JSON or Mistral call. A marker after that closed call (a trailing example or data in a later ;-chained call's strings) now also defers to the leading call, the same inside-or-after rule the closed XML envelope patterns already applied. The nested Gemma primitive value scan split on every comma, corrupting arguments like opts:{code:print(1,2),lang:py}. It now applies the same paren/brace depth, contextual quote openers, and comma-only-before-a-key mapping rule as the top-level scan. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gemma leading guard: a closed enabled call preceding the signal owns the turn The wrapperless Gemma guard only claimed the turn when the first foreign signal sat inside the first enabled balanced call. When that call closed before the signal (a second call quoting a Mistral or Kimi literal, or a trailing prose example), the guard forfeited the turn and the foreign parser promoted the quoted literal, dropping the real Gemma calls. Apply the same inside-or-after ownership rule as the closed bare-JSON and Mistral envelopes, gated on an enabled name so the name-agnostic legacy path is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Marker guard: only an executable leading bare-JSON call owns the turn The bare-JSON branch of the leading-envelope marker guard claimed the turn for any NAMED leading object. A disabled-name object is prose by design (the bare-JSON parser will not execute it), so deferring the DeepSeek/Kimi pre-pass to it lost the real later call entirely. Gate the ownership claim on the enabled set (or the name-agnostic None path). A marker inside the disabled object's own strings stays data, matching the tail-exclusion contract; a marker after it now falls through so the pre-pass parses the real call. The Mistral branch stays ungated since [TOOL_CALLS] parsing is never name-gated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gemma scan skips leading JSON answers; GLM heal bounds values at structural tags Two fixes to the document-order data contracts: The markerless Gemma scan only exempted whole-content JSON, so a leading JSON answer followed by prose had an enabled call:NAME{...} snippet inside its strings promoted to a real executed call and stripped from the displayed answer. Both the parse and strip scans now start after a balanced json-valid leading value span, keeping parse and strip mirrored. Real calls after the answer still parse; mid-prose JSON gets no exemption. The GLM heal fallback for a missing closing arg_value tag took the entire remainder as the value, executing markup-contaminated arguments like city="NYC" and swallowing trailing prose. The healed value now stops at the next arg_key or tool_call close and the pair walk resumes there. EOF-truncated values keep the partial heal, strict mode still rejects, and closed values holding a literal close tag in quotes are untouched. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Compress docstrings in the multi-format tool parser to their contract essence * Condense parser guard comments and test narration to contract essentials * verify_import_hoist: exempt __future__ imports and same-diff relocations Two false positives fired on this PR's refactor. A from __future__ import is a compiler directive whose name never appears as a runtime load, so HOISTED-IMPORT-UNUSED can never see it used, yet the file requires it for PEP 604 annotations on Python 3.9. TARGET-CHANGED flagged the deliberate move of the strip-pattern constants into core.inference.tool_call_parser as a silent re-point even though the old module-level target was removed and the new one added in the same diff. Both get narrow exemptions; a re-point to a pre-existing target is still caught, and the self-test negative controls all pass unchanged. * Leading bare-JSON calls own the turn; function calls end at the first balanced close The XML-signal guard for a leading bare-JSON call required the signal strictly inside the object, so a trailing XML example stole the turn from the leading call; it now applies the same inside-or-after rule as the Mistral guard. Function-XML calls also ended at the LAST close tag, which let prose after a closed call that mentions a literal close tag get swallowed into the final parameter value; calls now end at the first close tag that is not inside an open parameter, and the strip mirrors the same rule so parse and strip agree. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Attribute-form calls end at the first balanced close; bare-JSON strip requires the call shape The attribute form parser still kept the last close tag in the call window, folding prose after a closed call into the final parameter value. It now takes the first close not inside an open parameter, the same rule the equals form and the strip already use. The leading bare-JSON strip deleted any closed object whose top-level name matched an enabled tool, including plain JSON answers the parser correctly rejects as non-calls. The strip (and the drain gate that delegates to it) now requires the parser's exact call shape, so answers like {"name":"web_search","result":...} stream and display intact. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * False-alarm markers keep the answer; the bare-JSON strip consumes the whole chain The trailing strip arms dropped everything from a bare marker to EOF, so a normal answer that mentions [TOOL_CALLS] or another marker literally was truncated (or fully swallowed when it started with the literal) after the no-call drain fallback. Those arms now require a call-shaped lookahead or marker-at-EOF before dropping; truncated real calls still strip. Chained bare-JSON turns executed both calls but stripped only the first object, so the second call's raw JSON replayed into the next assistant history message alongside the structured tool_calls. The strip now consumes the entire chained run of call-shaped enabled objects while non-call answers, disabled names, and trailing prose stay intact. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * DeepSeek and Kimi trailing strip arms require a call-shaped lookahead Same false-alarm rule as the bare-word markers: a prose answer that mentions a DeepSeek or Kimi marker literally keeps its tail, while truncated real envelopes and bare end-of-text fragments still drop. * Attribute-form containment, parameter-close-decides rule, preamble-tolerant Mistral guard, strict strip shape Four document-order and containment fixes. A leading attribute-form call now parses before the shared XML pass, so markup quoted in its parameter stays data. The open-parameter scan lets the parameter's own close tag decide, so any number of literal function closes inside one value stay data, restoring the pre-close-scan behavior for multi-close arguments. The leading-Mistral guard tolerates a visible preamble, with the leading-bare-JSON guard running first so a trigger quoted inside a leading JSON object stays data. The bare-JSON strip requires the parser's top-level name in every mode, so nested-name JSON answers survive name-agnostic stripping. * Keep buffering long wrapper-less Gemma tool names instead of leaking the prefix The streaming buffer stopped holding a call:NAME prefix at a fixed 32-char cap, so a Gemma wrapper-less call to a tool whose name exceeds that (OpenAI allows 64 chars, MCP names run longer) streamed its raw call:longname text as visible content before the end-of-turn parser executed it. Hold the variable-length prefix while it still matches the call: shape, bounded like the bare-JSON path and self-terminating into prose, draining once the opening brace arrives. * Keep prose that only mentions DeepSeek/Kimi markers in the route display strip The route-level _TOOL_XML_RE DeepSeek/Kimi arms consumed from an opener up to the end of text whenever the marker appeared, so an answer that merely refers to a marker (for example "See <|tool_call_begin|> in the docs") had the rest of the reply truncated. The parser-level _TOOL_ALL_PATS already gates these arms with a call-shaped lookahead. Mirror it here so a marker is only stripped when a real call follows it or it is a bare fragment at end of text. * Tighten tool-calling parser and backend comments * Pass trust_remote_code when reloading native tokenizers The native-template fallback re-fetches a model's native chat template from its repo when an Unsloth override template drops the tools schema. The secondary AutoTokenizer.from_pretrained threaded hf_token but not trust_remote_code, so for a model loaded with trust_remote_code=True whose tokenizer repo carries custom code the reload raised, was swallowed, and the request silently kept the tool-dropping prompt for a model that supports tools. Store the loaded trust_remote_code on each backend's per-model info dict and source it in render_native_template, so the reload re-uses exactly the consent granted at load. For a LoRA adapter the reload targets the base model, whose remote code was gated and loaded under the same stored flag, so re-passing it executes no unconsented code. Falsy stored flag preserves the prior behaviour. Adds a regression test that fails without the flag (custom-code reload raises, returns None) and passes with it (tools-advertising native prompt returned). * Treat <|python_tag|> as an outer marker envelope A Llama-3 <|python_tag|> tool call (built-in NAME.call(...) or custom {json} form) whose argument quotes a complete DeepSeek/Kimi example was hijacked by the DeepSeek/Kimi marker pre-pass: the embedded example (for example delete_all) executed instead of the real outer call. python_tag is Llama-3's tool-call envelope, so a marker quoted inside its arguments is data, the same as for , , bare JSON, Mistral and wrapper-less Gemma, which the guard already covers. Add <|python_tag|> to _OUTER_ENVELOPE_OPEN_RE with a call-shaped lookahead (mirroring the _TOOL_ALL_PATS python_tag arm) so the marker pre-pass is suppressed when a python_tag call opens before the first marker, while a bare prose <|python_tag|> mention is left untouched. * Tighten tool-call parser comments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen Co-authored-by: Daniel Han Co-authored-by: danielhanchen --- .gitignore | 2 + scripts/verify_import_hoist.py | 15 +- .../core/inference/chat_template_helpers.py | 184 +- studio/backend/core/inference/inference.py | 34 + studio/backend/core/inference/llama_cpp.py | 124 +- .../backend/core/inference/mlx_inference.py | 32 + .../core/inference/passthrough_healing.py | 9 +- .../core/inference/safetensors_agentic.py | 138 +- .../core/inference/tool_call_parser.py | 1546 +++++++++++++++-- studio/backend/routes/inference.py | 168 +- .../tests/test_gemma_tool_parse_edge_cases.py | 61 +- .../backend/tests/test_llama_cpp_tool_loop.py | 136 +- studio/backend/tests/test_mcp_servers.py | 4 +- .../tests/test_mlx_inference_backend.py | 51 +- .../test_native_template_trust_remote_code.py | 176 ++ .../backend/tests/test_pr5624_regressions.py | 1011 +++++++++++ .../tests/test_responses_tool_passthrough.py | 14 +- .../test_safetensors_capability_advertise.py | 133 +- .../test_safetensors_reasoning_stream.py | 12 +- .../tests/test_safetensors_tool_loop.py | 1247 ++++++++++++- .../tests/test_tool_call_parser_strict.py | 632 ++++++- studio/backend/tests/test_tool_xml_strip.py | 194 ++- 22 files changed, 5472 insertions(+), 451 deletions(-) create mode 100644 studio/backend/tests/test_native_template_trust_remote_code.py create mode 100644 studio/backend/tests/test_pr5624_regressions.py diff --git a/.gitignore b/.gitignore index 9f7d4b8c6..39ca2226c 100644 --- a/.gitignore +++ b/.gitignore @@ -11,6 +11,8 @@ outputs/ exports/ /datasets/ studio/backend/assets/datasets/ +# Generated async worker / reviewer transcripts (never part of the product). +studio/backend/async_task_outputs/ unsloth_training_checkpoints/ *.gguf *.safetensors diff --git a/scripts/verify_import_hoist.py b/scripts/verify_import_hoist.py index 2d30265ab..22a21a2eb 100644 --- a/scripts/verify_import_hoist.py +++ b/scripts/verify_import_hoist.py @@ -564,7 +564,10 @@ def compare(before_src: str, after_src: str, path: str) -> list[tuple[str, str]] for n, tids in b["module_import_targets"].items(): if tids & after_used: continue # resolved -> fine - # `from __future__ import ...` is a compiler directive whose name is never loaded; skip it. + # `from __future__ import ...` is a compiler directive, not a runtime + # binding: the name (`annotations`, ...) is never loaded, so it can never + # "resolve" to a use. Skip it so a legitimately-added future import + # (e.g. `annotations` for lazy PEP 604 `X | None` on py3.9) is not flagged. if all(t.startswith("from:__future__:") for t in tids): continue newly_added = bool(tids - before_module_targets) @@ -592,9 +595,13 @@ def compare(before_src: str, after_src: str, path: str) -> list[tuple[str, str]] # `import urllib.error` next to `import urllib.request`). Nothing the name # resolved to before is lost, so no reference is re-pointed -- skip it. # - # A deliberate *relocation* is also benign: a name's import source moves A -> B in - # THIS diff (old `from A import x` removed, new `from B import x` added). Mirrors the - # TARGET-MISSING tolerance. Re-pointing to a pre-existing target (clash) is NOT exempted. + # A deliberate *relocation* is also benign and must not block: when a name + # keeps its spelling but its import source is moved A -> B in THIS diff (the + # old `from A import x` is removed at module level and a new `from B import x` + # is added), the swap is intentional, not a silent re-point to a pre-existing + # different object. This mirrors the relocation tolerance already applied to + # TARGET-MISSING. The dangerous case -- the name now resolving to a target + # that already existed before (shadow/clash) -- is NOT exempted. removed_module_targets = before_module_targets - after_module_targets for key, tafter in b["target_by_use"].items(): tbefore = a["target_by_use"].get(key) diff --git a/studio/backend/core/inference/chat_template_helpers.py b/studio/backend/core/inference/chat_template_helpers.py index f58c93b7f..dfd4c1c0b 100644 --- a/studio/backend/core/inference/chat_template_helpers.py +++ b/studio/backend/core/inference/chat_template_helpers.py @@ -3,13 +3,19 @@ """ Dependency-light wrapper around tokenizer.apply_chat_template with a kwarg -fallback for templates that reject reasoning/tools args. +fallback for templates that reject reasoning/tools args, plus the shared +native-chat-template fallback used by the transformers and MLX backends. """ +import copy import json +import logging from typing import Optional +logger = logging.getLogger(__name__) + + def _normalize_tool_call_arguments(messages: list) -> list: """Coerce each assistant ``tool_calls[].function.arguments`` from a JSON string to a dict. @@ -110,3 +116,179 @@ def apply_chat_template_for_generation( if normalized is messages: raise return _render(normalized) + + +def render_native_template( + *, + model_info: dict, + active_model_name: Optional[str], + messages: list, + tools: list, + enable_thinking: Optional[bool] = None, + reasoning_effort: Optional[str] = None, + preserve_thinking: Optional[bool] = None, + apply_fn = None, + hf_token: Optional[str] = None, +) -> Optional[str]: + """Render ``messages`` + ``tools`` with the model's NATIVE chat template. + + Some Unsloth override templates (e.g. ``mistral``, ``gemma-4``) do not emit + the ``tools`` schema, so a tool-calling turn silently stops advertising tools. + The native template ships in the model repo and carries the family's + tool-calling syntax. It is loaded straight from the repo (bypassing any + override on the live tokenizer) and cached on ``model_info``. Returns the + rendered prompt only if the native template actually emits the tools (render + differs with vs without tools); otherwise ``None``. + + ``hf_token`` is the token the model was loaded with -- passed to the repo load + so a gated/private model's native template can still be fetched (otherwise the + fallback fails silently and keeps the override prompt that dropped tools). + + ``trust_remote_code`` is sourced from ``model_info`` (the value the model was + actually loaded with) rather than a call-site argument, so the native-template + reload uses exactly the consent already granted at load. A custom-code tokenizer + repo raises in ``AutoTokenizer.from_pretrained`` unless ``trust_remote_code`` is + passed, so without this the fallback fails silently and keeps the tool-dropping + prompt for a model the user already consented to run remote code for. For a LoRA + adapter the reload targets the base model, whose remote code was gated and loaded + under the same stored flag, so re-passing it executes no unconsented code. + """ + # ``apply_fn`` lets a backend inject its own render; defaults to the module helper. + if apply_fn is None: + apply_fn = apply_chat_template_for_generation + native_tpl = model_info.get("native_chat_template") + if native_tpl is None: + # A LoRA adapter's native template lives on the base model, not the adapter id. + template_source = model_info.get("base_model") or active_model_name + # Re-use the load-time trust_remote_code so a custom-code tokenizer repo can + # instantiate its class (the stored flag already covers template_source). + trust_remote_code = bool(model_info.get("trust_remote_code", False)) + try: + from transformers import AutoTokenizer + nt = AutoTokenizer.from_pretrained( + template_source, + token = hf_token if hf_token and hf_token.strip() else None, + trust_remote_code = trust_remote_code, + ) + native_tpl = nt.chat_template or False + except Exception as exc: + logger.warning( + "Could not load native chat template for '%s': %s", + template_source, + exc, + ) + # A failed fetch is not "no template": leave the sentinel unset so the next + # call retries (caching False would pin the tool-dropping override). + return None + model_info["native_chat_template"] = native_tpl + if not native_tpl: + return None + + tokenizer = model_info.get("tokenizer") or model_info.get("processor") + if tokenizer is None: + return None + tokenizer = getattr(tokenizer, "tokenizer", tokenizer) + # Render on a shallow copy: mutating the shared tokenizer.chat_template (outside the + # generation lock) races concurrent requests. + try: + render_tokenizer = copy.copy(tokenizer) + render_tokenizer.chat_template = native_tpl + except Exception as exc: + logger.warning( + "Could not clone tokenizer for native-template render of '%s': %s", + active_model_name, + exc, + ) + return None + try: + with_tools = apply_fn( + render_tokenizer, + messages, + tools = tools, + enable_thinking = enable_thinking, + reasoning_effort = reasoning_effort, + preserve_thinking = preserve_thinking, + ) + no_tools = apply_fn( + render_tokenizer, + messages, + tools = None, + enable_thinking = enable_thinking, + reasoning_effort = reasoning_effort, + preserve_thinking = preserve_thinking, + ) + except Exception as exc: + logger.warning( + "Native-template tool render failed for '%s': %s", + active_model_name, + exc, + ) + return None + return with_tools if with_tools != no_tools else None + + +def render_with_native_template_fallback( + *, + formatted_prompt: str, + tokenizer, + model_info: dict, + active_model_name: Optional[str], + messages: list, + tools: Optional[list], + enable_thinking: Optional[bool] = None, + reasoning_effort: Optional[str] = None, + preserve_thinking: Optional[bool] = None, + apply_fn = None, + hf_token: Optional[str] = None, +) -> str: + """Return ``formatted_prompt``, swapping in a native-template render when an + override template dropped the ``tools`` schema. + + If ``tools`` were requested but the live render is identical with and without + them (detected by comparison, robust against tool names in the system prompt), + re-render with the model's native template. Shared by the transformers and MLX + backends so both advertise tools consistently. ``hf_token`` is forwarded so a + gated/private model's native template can still be fetched.""" + if not tools: + return formatted_prompt + if apply_fn is None: + apply_fn = apply_chat_template_for_generation + # Probe whether the live template dropped the schema. A tools-requiring template + # can raise here; on any error keep the valid tools prompt rather than lose it. + try: + probe_no_tools = apply_fn( + tokenizer, + messages, + tools = None, + enable_thinking = enable_thinking, + reasoning_effort = reasoning_effort, + preserve_thinking = preserve_thinking, + ) + except Exception as exc: + logger.warning( + "No-tools probe failed for '%s'; keeping the existing tools prompt: %s", + active_model_name, + exc, + ) + return formatted_prompt + if formatted_prompt != probe_no_tools: + return formatted_prompt # template already emits the tools schema + native_prompt = render_native_template( + model_info = model_info, + active_model_name = active_model_name, + messages = messages, + tools = tools, + enable_thinking = enable_thinking, + reasoning_effort = reasoning_effort, + preserve_thinking = preserve_thinking, + apply_fn = apply_fn, + hf_token = hf_token, + ) + if native_prompt: + logger.info( + "Override template for '%s' dropped tool schemas; using the model's " + "native template for this tool-calling turn.", + active_model_name, + ) + return native_prompt + return formatted_prompt diff --git a/studio/backend/core/inference/inference.py b/studio/backend/core/inference/inference.py index eaee5a213..164f20268 100644 --- a/studio/backend/core/inference/inference.py +++ b/studio/backend/core/inference/inference.py @@ -269,6 +269,9 @@ class InferenceBackend: gpu_ids: Optional[list[int]] = None, ) -> bool: """Load any model: base, LoRA adapter, text, or vision.""" + # Keep the token so the native-template fallback can fetch a + # gated model's repo template later during generation. + self._hf_token = hf_token # GGUF uses max_seq_length=0 as "model default"; Unsloth crashes on it. if max_seq_length <= 0: max_seq_length = 2048 @@ -279,6 +282,8 @@ class InferenceBackend: # Already loaded? if model_name in self.models and self.models[model_name].get("model"): logger.info(f"Model {model_name} already loaded") + if hf_token: + self.models[model_name]["hf_token"] = hf_token self.active_model_name = model_name return True @@ -294,6 +299,14 @@ class InferenceBackend: ) self.models[model_name] = { + # Per-model token: the native-template fallback must use the + # token this model was loaded with, not whichever loaded last. + "hf_token": hf_token, + # Per-model consent: the native-template reload must re-use the + # exact trust_remote_code this model (and a LoRA's base) was loaded + # with, so a custom-code tokenizer repo can be re-fetched without + # executing any code the user did not already consent to. + "trust_remote_code": trust_remote_code, "is_vision": config.is_vision, "is_lora": config.is_lora, "is_audio": config.is_audio, @@ -1040,6 +1053,27 @@ class InferenceBackend: reasoning_effort = reasoning_effort, preserve_thinking = preserve_thinking, ) + + # If tools were requested but the (possibly overridden) template ignored + # them, fall back to the model's native template (shared with MLX). + from core.inference.chat_template_helpers import ( + render_with_native_template_fallback, + ) + + formatted_prompt = render_with_native_template_fallback( + formatted_prompt = formatted_prompt, + tokenizer = tokenizer, + model_info = model_info, + active_model_name = self.active_model_name, + messages = template_messages, + tools = tools, + enable_thinking = enable_thinking, + reasoning_effort = reasoning_effort, + preserve_thinking = preserve_thinking, + apply_fn = self._apply_chat_template_for_generation, + hf_token = model_info.get("hf_token"), + ) + logger.debug(f"Formatted prompt: {formatted_prompt[:200]}...") except Exception as e: logger.error(f"Error applying chat template: {e}") diff --git a/studio/backend/core/inference/llama_cpp.py b/studio/backend/core/inference/llama_cpp.py index 5e67f6b48..455d1d084 100644 --- a/studio/backend/core/inference/llama_cpp.py +++ b/studio/backend/core/inference/llama_cpp.py @@ -40,11 +40,15 @@ from core.inference.llama_server_args import ( ) # Share strip / signal constants with the multi-format parser so BUFFERING also -# catches Llama-3 / Mistral / Gemma 4. +# catches Llama-3 / Mistral / Gemma 4 (legacy helper only knew / str: if not (auto_heal_tool_calls or force): return text - return _shared_strip_tool_markup(text, final = final) + return _shared_strip_tool_markup( + text, final = final, enabled_tool_names = _enabled_tool_names + ) def _strip_tool_markup_streaming(text: str, *, force: bool = False) -> str: if not (auto_heal_tool_calls or force): return text - # Shared patterns so a textual Mistral/Llama call entering DRAINING is stripped, not - # leaked. Mistral first; no final trim so incremental length comparisons hold. + # Shared parser patterns (not the legacy tool_healing set) so textual + # Mistral/python_tag calls entering DRAINING never leak. Balanced strips + # first (nested JSON removed whole); no final trim so length compares hold. text = _strip_mistral_closed_calls(text) - # Parser-accurate function-XML scan before the regex arms so a literal ```` - # in a value doesn't make the tail eat trailing prose after the real ````. + text = _strip_gemma_wrapperless_calls(text, _enabled_tool_names) + # Parser-accurate scans close at each call's REAL terminator before + # the regex arms: literal markup inside a value is data. text = _strip_function_xml_calls(text, final = True) + text = _strip_glm_calls(text, final = True) for pat in _TOOL_ALL_PATS: text = pat.sub("", text) return text @@ -8507,8 +8531,8 @@ class LlamaCppBackend: # "Hello!" won't match. Pattern compiled at module level # (_INTENT_SIGNAL). _reprompt_count = 0 - # Gates ``max_tool_iterations`` on real tool turns so reserved re-prompt slots don't - # extend the budget. Mirrors the safetensors guard. + # Gates ``max_tool_iterations`` on real tool turns (not the enlarged range) so reserved + # re-prompt slots don't extend the budget. Mirrors the safetensors guard. _tool_iters_done = 0 _forced_tool_call_pending = False @@ -8525,13 +8549,13 @@ class LlamaCppBackend: if not active_tools: _append_budget_exhausted_nudge = False break - # Gate the markerless bare-JSON form on enabled names so a JSON answer isn't misread as a call. + # Gate the markerless bare-JSON form on enabled names so an ordinary JSON answer isn't misread as a call. _enabled_tool_names = { (tool.get("function") or {}).get("name") for tool in active_tools if (tool.get("function") or {}).get("name") } - # Shared signal tuple so GGUF BUFFERING wakes on every format the parser knows. + # Shared signal tuple so GGUF BUFFERING wakes on every format the parser knows (like safetensors). _tool_xml_signals = _SHARED_TOOL_XML_SIGNALS # Build payload -- stream: True so we detect tool signals @@ -8815,8 +8839,9 @@ class LlamaCppBackend: is_prefix = True break - # Bare Llama-3.2 {"name":..} has no XML signal: hold an - # incomplete object, drain a complete one (mirrors safetensors). + # Signal-less call shapes (mirror the safetensors + # loop): Llama-3.2 bare {"name":..} and Gemma + # call:NAME{...} would otherwise stream raw. _hold_buffer = False # Whole buffer is the call (no visible prefix) -- drain silently. _drain_silently = False @@ -8829,9 +8854,9 @@ class LlamaCppBackend: elif _looks_like_enabled_bare_json( _bare, _enabled_tool_names ): - # Oversized still-open ENABLED-tool call: stop - # holding (memory bound) but DRAIN, not leak; - # a giant ordinary JSON answer still streams. + # Oversized still-open enabled call: drain + # rather than leak; a giant ordinary JSON + # answer still streams. _drain_silently = True elif self._parse_tool_calls_from_text( content_buffer, @@ -8839,6 +8864,17 @@ class LlamaCppBackend: enabled_tool_names = _enabled_tool_names, ): _drain_silently = True + elif ( + "call:".startswith(stripped_buf) + or _GEMMA_BARE_TC_PREFIX_RE.match(stripped_buf) + is not None + or _GEMMA_BARE_TC_RE.match(stripped_buf) is not None + ): + # Whitespace-tolerant like the parser. + if _GEMMA_BARE_TC_RE.match(stripped_buf): + _drain_silently = True + elif len(stripped_buf) < _MAX_BUFFER_CHARS: + _hold_buffer = True if _drain_silently: # No visible prefix -- the buffered text IS @@ -8890,9 +8926,10 @@ class LlamaCppBackend: # ── Resolve BUFFERING at stream end ── if detect_state == _S_BUFFERING: stripped_buf = content_buffer.lstrip() - # A held bare-JSON fragment has no XML signal; route it to DRAINING. + # A held bare-JSON fragment has no XML signal; route it to DRAINING (the signal-only + # gate below would flush the raw JSON to the user). _bare_eos = strip_llama3_leading_sentinels(stripped_buf) - # Gate on enabled names so a JSON answer isn't routed to DRAINING and dropped. + # Gate on enabled names so an ordinary JSON answer isn't routed to DRAINING and dropped. _is_bare_tc = bool(active_tools) and _looks_like_enabled_bare_json( _bare_eos, _enabled_tool_names ) @@ -8925,8 +8962,8 @@ class LlamaCppBackend: "text": cumulative_display, } else: - # No tool signal and no enabled bare-JSON call: a leading ``{`` is an ordinary - # JSON answer and must be shown; any other partial-markup prefix is dropped. + # Held buffer was no tool signal and no enabled bare-JSON call: a leading ``{`` is an + # ordinary JSON answer and must be shown; any other partial-markup prefix is dropped. _held = strip_llama3_leading_sentinels(content_buffer.lstrip()) if _held.startswith("{") and not _suppress_visible_output: yield {"type": "content", "text": _held} @@ -8934,10 +8971,12 @@ class LlamaCppBackend: # ── STREAMING path: no tool call ── if detect_state == _S_STREAMING: - # Safety net: re-parse the full content for tool calls. The route layer resets - # prev_text on tool_start, so post-tool synthesis streams correctly even if - # content was emitted before the tool XML. Unconditional (not gated on - # _tool_xml_signals): bare-JSON and Gemma wrapper-less calls carry no signal. + # Safety net: re-parse the full content for tool calls. The + # route layer resets prev_text on tool_start, so post-tool + # synthesis streams correctly even if content was emitted + # before the tool XML. + # Unconditional (not gated on _tool_xml_signals): bare-JSON and Gemma wrapper-less + # calls carry no XML signal, so a signal gate would let them slip past. _safety_tc = self._parse_tool_calls_from_text( content_accum, allow_incomplete = auto_heal_tool_calls, @@ -9060,8 +9099,8 @@ class LlamaCppBackend: if (tool_calls_acc[i].get("function", {}).get("name", "").strip()) ] or None if not tool_calls: - # Unconditional re-parse: DRAINING means the buffer looked like a call, and - # bare-JSON / Gemma wrapper-less calls carry no XML signal to gate on. + # Unconditional re-parse: we only reach DRAINING when the buffer looked like a + # call, and bare-JSON / Gemma wrapper-less calls carry no XML signal to gate on. tool_calls = self._parse_tool_calls_from_text( content_accum, allow_incomplete = auto_heal_tool_calls, @@ -9073,8 +9112,8 @@ class LlamaCppBackend: final = True, force = True, ) - # ``_strip_tool_markup`` only knows XML; also drop a leading bare-JSON call - # so the executed call isn't replayed as text or next-turn history. + # ``_strip_tool_markup`` only knows XML; also drop a leading bare-JSON call so the + # executed call isn't replayed as text or next-turn history. content_text = strip_leading_bare_json_call( content_text, _enabled_tool_names ) @@ -9091,8 +9130,8 @@ class LlamaCppBackend: if content_accum: # Strip leaked tool-call XML before yielding. content_accum = _strip_tool_markup(content_accum, final = True) - # A truncated bare-JSON call has no XML to strip and didn't parse. With - # Auto-Heal on drop a leading ENABLED-tool fragment (plain JSON untouched); + # A truncated bare-JSON call has no XML markup to strip and didn't parse. With + # Auto-Heal on, drop a leading ENABLED-tool fragment (ordinary JSON answers untouched); # off keeps it visible per the strict contract. if content_accum and active_tools and auto_heal_tool_calls: content_accum = strip_leading_bare_json_call( @@ -9115,6 +9154,29 @@ class LlamaCppBackend: _accumulated_predicted_ms += _it.get("predicted_ms", 0) _accumulated_predicted_n += _it.get("predicted_n", 0) + # Collapse exact-duplicate calls and cap the count for the TEXTUAL + # fallback (mirrors the safetensors loop; see _MAX_TOOL_CALLS_PER_TURN). + if tool_calls and not has_structured_tc and len(tool_calls) > 1: + _seen_keys: set = set() + _deduped: list = [] + for _tc in tool_calls: + _fn = _tc.get("function", {}) or {} + _key = (_fn.get("name", ""), str(_fn.get("arguments", ""))) + if _key in _seen_keys: + continue + _seen_keys.add(_key) + _deduped.append(_tc) + if len(_deduped) >= _MAX_TOOL_CALLS_PER_TURN: + break + if len(_deduped) != len(tool_calls): + logger.info( + "GGUF textual fallback: collapsed %d repeated tool call(s) " + "in one turn to %d", + len(tool_calls), + len(_deduped), + ) + tool_calls = _deduped + # disable_parallel_tool_use: execute only the first tool call # this turn. Truncate before building assistant_msg so the # conversation stays consistent and extra calls are never executed. @@ -9265,8 +9327,8 @@ class LlamaCppBackend: if tool_controller.force_final_answer or not tool_controller.active_tools(): _append_budget_exhausted_nudge = False break - # Count only real tool turns against the cap so reserved re-prompt slots can't - # become extra tool rounds; a no-op turn doesn't consume budget (GGUF parity). + # Count only real tool turns against the cap so reserved re-prompt slots can't become + # extra tool rounds; a no-op correction turn doesn't consume budget (GGUF parity). if _turn_executed_real_tool: _tool_iters_done += 1 if _tool_iters_done >= max_tool_iterations: diff --git a/studio/backend/core/inference/mlx_inference.py b/studio/backend/core/inference/mlx_inference.py index 5c7799152..45f46fef2 100644 --- a/studio/backend/core/inference/mlx_inference.py +++ b/studio/backend/core/inference/mlx_inference.py @@ -104,6 +104,9 @@ class MLXInferenceBackend: ) -> bool: import mlx.core as mx + # Keep the token so the native-template fallback can fetch a + # gated model's repo template later during generation. + self._hf_token = hf_token model_name = config.identifier if hasattr(config, "identifier") else str(config) is_vision = getattr(config, "is_vision", False) @@ -168,11 +171,20 @@ class MLXInferenceBackend: self.active_model_name = model_name self.models[model_name] = { + # Per-model token for the native-template fallback (matches transformers). + "hf_token": hf_token, + # Per-model consent for the native-template reload: re-use the exact + # trust_remote_code this model was loaded with (matches transformers). + "trust_remote_code": trust_remote_code, "model": self._model, "tokenizer": self._tokenizer, "processor": self._processor, "is_vision": is_vision, "is_lora": getattr(config, "is_lora", False), + # For a LoRA adapter the native chat template lives on the base model. + "base_model": getattr(config, "base_model", None) + if getattr(config, "is_lora", False) + else None, "is_audio": False, "audio_type": None, "has_audio_input": False, @@ -355,6 +367,7 @@ class MLXInferenceBackend: from core.inference.chat_template_helpers import ( apply_chat_template_for_generation, + render_with_native_template_fallback, ) prompt = apply_chat_template_for_generation( @@ -368,6 +381,25 @@ class MLXInferenceBackend: if prompt is None: raise RuntimeError("apply_chat_template returned None — tokenizer may be incompatible") + # Same parity fix as the transformers backend: if the template dropped the + # requested tools, fall back to the native template so MLX text models keep + # advertising them. ``self._tokenizer`` is this entry's model_info tokenizer, + # so probe and native render share a renderer. (The VLM path renders via the + # processor for image tokens and is intentionally not wired here.) + model_info = self.models.get(self.active_model_name, {}) + prompt = render_with_native_template_fallback( + formatted_prompt = prompt, + tokenizer = self._tokenizer, + model_info = model_info, + active_model_name = self.active_model_name, + messages = messages, + tools = tools, + enable_thinking = enable_thinking, + reasoning_effort = reasoning_effort, + preserve_thinking = preserve_thinking, + hf_token = model_info.get("hf_token"), + ) + sampler = make_sampler( temp = temperature, top_p = top_p, diff --git a/studio/backend/core/inference/passthrough_healing.py b/studio/backend/core/inference/passthrough_healing.py index 35855cc34..fe1aca0e4 100644 --- a/studio/backend/core/inference/passthrough_healing.py +++ b/studio/backend/core/inference/passthrough_healing.py @@ -32,9 +32,12 @@ from typing import Any, Optional from core.inference.tool_loop_controller import coerce_tool_arguments from core.tool_healing import parse_tool_calls_from_text -# Only the formats this healer can promote. The parser's broader list adds Llama -# <|python_tag|> / Mistral [TOOL_CALLS], but buffering those here would flush a -# streamed call as prose, so keep a healer-aligned list. +# Signals limited to the formats parse_tool_calls_from_text (core.tool_healing) +# actually promotes. The parser module's broader signal list also covers Llama +# <|python_tag|> and Mistral [TOOL_CALLS] for the streaming DRAIN buffers whose +# full parser handles them; buffering those here would hold a streamed +# client-tool call until finalization and then flush it as prose (this healer +# cannot promote them), so the passthrough keeps its own aligned list. _HEAL_SIGNALS = ( "", "<|tool_call>", diff --git a/studio/backend/core/inference/safetensors_agentic.py b/studio/backend/core/inference/safetensors_agentic.py index b67c6cf7e..8e86d0975 100644 --- a/studio/backend/core/inference/safetensors_agentic.py +++ b/studio/backend/core/inference/safetensors_agentic.py @@ -21,9 +21,13 @@ from typing import Callable, Generator, Optional from loggers import get_logger from core.inference.tool_call_parser import ( + _GEMMA_BARE_TC_PREFIX_RE, + _GEMMA_BARE_TC_RE, _TOOL_ALL_PATS, _balanced_brace_end, _strip_function_xml_calls, + _strip_gemma_wrapperless_calls, + _strip_glm_calls, _strip_mistral_closed_calls, _strip_mistral_reasoning, BUDGET_EXHAUSTED_NUDGE, @@ -59,8 +63,8 @@ _MAX_BUFFER_CHARS = 32 # Memory bound for holding a leading bare-JSON object whose top-level "{" never balances. _MAX_BARE_JSON_BUFFER = 16384 -# Forward-looking intent ("I'll", "First,", "Step 1:") = planning; nudge a call. Negative -# lookahead drops negated forms ("I will not"). Mirrors GGUF. +# Forward-looking intent ("I'll", "First,", "Step 1:") = planning, not answering; nudge a call. +# Negative lookahead drops negated forms ("I will not") so a refusal doesn't trigger it. Mirrors GGUF. _INTENT_SIGNAL = re.compile( r"(?i)(" r"\b(i['’](ll|m going to|m gonna)|i am (going to|gonna)|i will|i shall|let me|allow me)\b(?!\s+(?:not|never)\b)" @@ -70,11 +74,15 @@ _INTENT_SIGNAL = re.compile( ) _MAX_REPROMPTS = 3 _REPROMPT_MAX_CHARS = 2000 -# Templated so the nudge names the caller's enabled tools. Mirrors GGUF tool_hint. +# Templated so the nudge names the caller's enabled tools, not a hardcoded set. Mirrors GGUF tool_hint. _REPROMPT_INSTRUCTION_TEMPLATE = ( "STOP. Do NOT write code or explain. You MUST call a tool NOW. Call {tool_hint} immediately." ) +# No grammar constraint here (unlike llama-server's lazy grammar): collapse +# exact-duplicate calls and cap the count so a runaway turn cannot fan out. +_MAX_TOOL_CALLS_PER_TURN = 8 + def _active_tool_names(active_tools: list[dict]) -> list[str]: names = [ @@ -90,16 +98,25 @@ def strip_tool_markup_streaming( *, auto_heal_tool_calls: bool = True, tool_protocol_active: bool = False, + enabled_tool_names: Optional[set] = None, ) -> str: - """Strip open-ended tool XML from display text without trimming whitespace.""" + """Strip open-ended tool XML from display text without trimming whitespace. + ``enabled_tool_names`` gates the markerless Gemma ``call:NAME{...}`` strip so a + disabled/example name in prose is kept (mirrors the parser gate).""" if not (auto_heal_tool_calls or tool_protocol_active): return text - # Mirror the final strip (no final trim): drop a leading Magistral ``[THINK]...[/THINK]`` - # block, then Mistral calls, then a parser-accurate function-XML scan before the regex - # arms. An unclosed ``[THINK]`` holds until ``[/THINK]`` so text stays monotonic. + # Mirror the final strip's scan order so streaming and final display agree: + # balanced strips first (nested JSON removed whole), then the guarded + # function-XML/GLM scans that close at each call's REAL terminator, so literal + # markup inside argument values is data and trailing prose survives. No final + # trim so streaming length comparisons hold. Leading Magistral [THINK]...[/THINK] + # is dropped (bracket form, not the reasoning channel's ); an unclosed + # [THINK] holds until [/THINK] so the cleaned text stays monotonic. text = _strip_mistral_reasoning(text) text = _strip_mistral_closed_calls(text) + text = _strip_gemma_wrapperless_calls(text, enabled_tool_names) text = _strip_function_xml_calls(text, final = True) + text = _strip_glm_calls(text, final = True) for pat in _TOOL_ALL_PATS: text = pat.sub("", text) return text @@ -110,10 +127,11 @@ def _strip_tool_markup_final( *, auto_heal_tool_calls: bool, tool_protocol_active: bool = False, + enabled_tool_names: Optional[set] = None, ) -> str: if not (auto_heal_tool_calls or tool_protocol_active): return text - return strip_tool_markup(text, final = True) + return strip_tool_markup(text, final = True, enabled_tool_names = enabled_tool_names) def _status_for_tool(tool_name: str, arguments: dict) -> str: @@ -247,8 +265,9 @@ def run_safetensors_tool_loop( final_attempt_done = False next_call_id = 0 reprompt_count = 0 - # Only turns that executed a tool count against ``max_tool_iterations``; a no-op or - # re-prompt turn must not consume budget (GGUF parity). + # Real tool-call turns completed. Only turns that actually executed a tool count + # against ``max_tool_iterations``; a duplicate/disabled no-op correction turn (and a + # plan-without-action re-prompt) must not consume budget, matching the GGUF loop. _executed_tool_iters = 0 def _tool_succeeded(tool_name: str) -> bool: @@ -285,7 +304,7 @@ def run_safetensors_tool_loop( tool_protocol_active = not final_attempt_done and (unrestricted_tools or bool(active_tools)) tool_xml_signals = TOOL_XML_SIGNALS if tool_protocol_active else () - # Gate the markerless bare-JSON form on enabled names so a JSON answer isn't misread as a call. + # Gate the markerless bare-JSON form on enabled names so an ordinary JSON answer isn't misread as a call. _enabled_tool_names = None if unrestricted_tools else set(_active_tool_names(active_tools)) detect_state = _state_buffering @@ -373,6 +392,7 @@ def run_safetensors_tool_loop( before_tool, auto_heal_tool_calls = auto_heal_tool_calls, tool_protocol_active = tool_protocol_active, + enabled_tool_names = _enabled_tool_names, ) if len(cleaned_before) > len(last_emitted): last_emitted = cleaned_before @@ -403,6 +423,7 @@ def run_safetensors_tool_loop( cumulative_display, auto_heal_tool_calls = auto_heal_tool_calls, tool_protocol_active = tool_protocol_active, + enabled_tool_names = _enabled_tool_names, ) if len(cleaned) > len(last_emitted): last_emitted = cleaned @@ -425,8 +446,9 @@ def run_safetensors_tool_loop( is_prefix = True break - # Bare Llama-3.2 ``{"name":..,"parameters":..}`` carries no XML signal. Hold a leading - # ``{`` (after any sentinel) until it closes: drain if it parses as a call, else stream. + # Llama-3.2 ``custom_tools`` emits a bare ``{"name":..,"parameters":..}`` with no XML + # signal. Hold a leading ``{`` (after any sentinel) until it closes: drain if it parses + # as a call, else stream as content. Non-call text is always recovered downstream. bare_probe = strip_llama3_leading_sentinels(stripped) if ( not is_match @@ -439,7 +461,7 @@ def run_safetensors_tool_loop( continue # object still open -- keep buffering elif _looks_like_enabled_bare_json(bare_probe, _enabled_tool_names): # Oversized still-open ENABLED-tool call: stop holding (memory bound) but - # DRAIN, not leak; a giant ordinary JSON answer still streams. + # DRAIN instead of leaking the raw prefix; a giant ordinary JSON answer still streams. detect_state = _state_draining continue elif parse_tool_calls_from_text( @@ -453,6 +475,35 @@ def run_safetensors_tool_loop( continue # Closed non-call object (or oversized non-call) -- stream as text. + # Gemma wrapper-less ``call:NAME{...}`` has no tool_xml_signals entry: + # buffer it here or it streams raw until the end-of-turn safety net. + # ``(? len(last_emitted): last_emitted = cleaned @@ -493,6 +545,7 @@ def run_safetensors_tool_loop( cumulative_display, auto_heal_tool_calls = auto_heal_tool_calls, tool_protocol_active = tool_protocol_active, + enabled_tool_names = _enabled_tool_names, ) if len(cleaned) > len(last_emitted): last_emitted = cleaned @@ -515,23 +568,25 @@ def run_safetensors_tool_loop( elif tool_protocol_active and _looks_like_enabled_bare_json( _bare_eos, _enabled_tool_names ): - # Held ENABLED-tool bare-JSON fragment has no XML signal; DRAIN it (a JSON answer - # falls through to the else and streams, GGUF parity). + # A held bare-JSON ENABLED-tool fragment has no XML signal; DRAIN it (an ordinary + # JSON answer falls through to the else and streams as content, GGUF parity). detect_state = _state_draining else: # Drain and fall through to STREAMING so the intent re-prompt + safety-net parser # still fire on short emissions like "Let me search." that never exit BUFFERING. if content_buffer: cumulative_display += content_buffer - cleaned = strip_tool_markup(cumulative_display, final = True) + cleaned = strip_tool_markup( + cumulative_display, final = True, enabled_tool_names = _enabled_tool_names + ) if len(cleaned) > len(last_emitted): last_emitted = cleaned yield {"type": "content", "text": cleaned} detect_state = _state_streaming if detect_state == _state_streaming: - # Run the parser even with no XML signal (bare-JSON carries none); it's strict so - # plain answers stay untouched. Mirrors GGUF. + # Run the parser even with no XML signal (the Llama-3.2 bare-JSON form carries none); it's + # strict so plain answers stay untouched. Mirrors GGUF. safety_tc = parse_tool_calls_from_text( content_accum, id_offset = next_call_id, @@ -539,8 +594,8 @@ def run_safetensors_tool_loop( enabled_tool_names = _enabled_tool_names, ) if not safety_tc: - # Re-prompt only when the model planned without acting (intent signal); - # "4" / "Hello!" never trigger. Mirrors GGUF. + # Re-prompt only when the model planned without acting (intent + # signal); "4" / "Hello!" never trigger. Mirrors GGUF. _stripped = content_accum.strip() if ( tools @@ -569,9 +624,9 @@ def run_safetensors_tool_loop( yield {"type": "status", "text": ""} continue - # Final answer. If a literal tool marker in prose was buffered but never - # parsed as a call, restore the raw text so the prose surfaces; route - # cleanup still applies the Auto-Heal policy. + # Final answer. If a literal tool marker in prose was buffered but + # never parsed as a call, restore the raw text so the prose surfaces + # in full; route-level cleanup still applies the Auto-Heal policy. if content_accum and any(sig in content_accum for sig in tool_xml_signals): yield {"type": "content", "text": content_accum} yield {"type": "status", "text": ""} @@ -581,6 +636,7 @@ def run_safetensors_tool_loop( content_accum, auto_heal_tool_calls = auto_heal_tool_calls, tool_protocol_active = True, + enabled_tool_names = _enabled_tool_names, ) logger.info( "Safetensors safety net: parsed %d tool call(s) from streamed content", @@ -603,9 +659,10 @@ def run_safetensors_tool_loop( content_accum, auto_heal_tool_calls = auto_heal_tool_calls, tool_protocol_active = False, + enabled_tool_names = _enabled_tool_names, ) - # Drained bare-JSON call that didn't parse: with Auto-Heal on drop the fragment - # (plain JSON untouched); off keeps it visible per the strict contract. + # Drained bare-JSON call that didn't parse: with Auto-Heal on, drop the fragment + # (plain JSON answers are left untouched); off keeps it visible per the strict contract. if tool_protocol_active and auto_heal_tool_calls: _drain_text = strip_leading_bare_json_call(_drain_text, _enabled_tool_names) if _drain_text: @@ -625,12 +682,13 @@ def run_safetensors_tool_loop( content_accum, auto_heal_tool_calls = auto_heal_tool_calls, tool_protocol_active = True, + enabled_tool_names = _enabled_tool_names, ) if tool_calls: next_call_id += len(tool_calls) - # Strip a leading bare-JSON call so it isn't replayed as text or next-turn history - # (``_strip_tool_markup_final`` only knows XML). No-op for plain JSON answers. + # Strip a leading bare-JSON call from the kept content so it isn't replayed as text or + # next-turn history (``_strip_tool_markup_final`` only knows XML). No-op for plain JSON answers. content_text = strip_leading_bare_json_call(content_text, _enabled_tool_names) if final_attempt_done: @@ -640,6 +698,27 @@ def run_safetensors_tool_loop( yield {"type": "status", "text": ""} return + # Collapse exact-duplicate calls and cap the count (runaway-turn guard). + if tool_calls: + seen_keys: set = set() + deduped: list = [] + for _tc in tool_calls: + _fn = _tc.get("function", {}) or {} + _key = (_fn.get("name", ""), str(_fn.get("arguments", ""))) + if _key in seen_keys: + continue + seen_keys.add(_key) + deduped.append(_tc) + if len(deduped) >= _MAX_TOOL_CALLS_PER_TURN: + break + if len(deduped) != len(tool_calls): + logger.info( + "Safetensors: collapsed %d repeated tool call(s) in one turn to %d", + len(tool_calls), + len(deduped), + ) + tool_calls = deduped + assistant_msg: dict = {"role": "assistant", "content": content_text} assistant_appended = False @@ -771,7 +850,8 @@ def run_safetensors_tool_loop( if not unrestricted_tools and not tool_controller.active_tools(): final_attempt_done = True continue - # Count only real tool turns against the cap so a no-op turn doesn't consume budget (GGUF parity). + # Count only turns that executed a tool against the cap; a no-op correction turn doesn't + # consume budget so the model gets its nudge and another tool-enabled turn (GGUF parity). if _turn_executed_real_tool: _executed_tool_iters += 1 if _executed_tool_iters >= max_tool_iterations and not final_attempt_done: diff --git a/studio/backend/core/inference/tool_call_parser.py b/studio/backend/core/inference/tool_call_parser.py index 9e82e40de..08a6bf418 100644 --- a/studio/backend/core/inference/tool_call_parser.py +++ b/studio/backend/core/inference/tool_call_parser.py @@ -14,18 +14,22 @@ safetensors + MLX agentic loop sees the same call shape llama-server gives GGUF: - ``[TOOL_CALLS]name{json}`` (Mistral v11+ / Magistral) - ``[TOOL_CALLS]name[ARGS]{json}`` (Ministral / Mistral Large 3) - ``<|tool_call>call:NAME{k:<|"|>v<|"|>}`` (Gemma 4) + - ``<｜tool▁calls▁begin｜>...function<｜tool▁sep｜>NAME\\n``\\`\\`\\`json\\n{...}\\n\\`\\`\\`...`` (DeepSeek R1) + - ``<｜tool▁calls▁begin｜>...<｜tool▁call▁begin｜>NAME<｜tool▁sep｜>{json}<｜tool▁call▁end｜>...`` (DeepSeek V3 / V3.1) + - ``NAME\\nk\\nv...`` (GLM 4.5 / 4.6 / 4.7) + - ``<|tool_calls_section_begin|>...<|tool_call_begin|>functions.NAME:IDX<|tool_call_argument_begin|>{json}<|tool_call_end|>...`` (Kimi K2) Missing closing tags / brackets are tolerated: models often truncate mid-stream. """ -# Keeps PEP 604 `X | None` lazy for python 3.9 (imported standalone by external servers). +# Lazy annotations keep the standalone python 3.9 import working. from __future__ import annotations import json import re from typing import Any, Optional -# Shared parser handles Qwen/Hermes, Qwen3.5 XML, Gemma 4; this module adds Llama-3, Mistral, bare JSON. +# Qwen/Hermes, Qwen3.5 XML and Gemma 4 live in core.tool_healing; this module adds the rest. from core import tool_healing as _tool_healing @@ -37,14 +41,31 @@ TOOL_XML_SIGNALS = ( "<|python_tag|>", "[TOOL_CALLS]", "<|tool_call>", + # DeepSeek R1 / V3 / V3.1 -- 5 opener variants llama.cpp keeps. + "<｜tool▁calls▁begin｜>", + "<｜tool▁call▁begin｜>", + "<｜tool_calls_begin｜>", + "<｜tool▁calls｜>", + "<｜tool calls begin｜>", + "<｜tool\\_calls\\_begin｜>", + # Kimi K2 / Moonshot. + "<|tool_calls_section_begin|>", + "<|tool_call_begin|>", ) -# Closed pairs only (mid-stream); _TOOL_ALL_PATS eats unclosed tails at end-of-turn. +# DeepSeek opener variants; shared by parse and strip so a parsed signal is always stripped. +_DEEPSEEK_OPEN_ALT = ( + r"tool▁calls▁begin|tool_calls_begin|tool calls begin|tool\\_calls\\_begin|tool▁calls" +) +_DEEPSEEK_OPEN_RE_SRC = r"<｜(?:" + _DEEPSEEK_OPEN_ALT + r")｜>" + +# Closed pairs only (mid-stream); _TOOL_ALL_PATS also eats unclosed tails at +# end-of-turn. ``[\w-]+`` on ```` tracks OpenAI's +# ``^[a-zA-Z0-9_-]{1,64}$`` so hyphenated MCP names parse like built-ins. _TOOL_CLOSED_PATS = [ re.compile(r".*?", re.DOTALL), - # Match to the real ```` (lookahead, not greedy ``.*``) so a literal - # ```` in a value doesn't truncate and each call stays separate. + # Span to the real ```` so a literal one inside a value can't truncate the strip. re.compile( r'' r'(?:(?!).)*' @@ -52,12 +73,21 @@ _TOOL_CLOSED_PATS = [ re.DOTALL, ), re.compile(r"<\|tool_call>.*?", re.DOTALL), + re.compile(r"\[TOOL_CALLS\]\s*\[.*?\](?:\s*)?", re.DOTALL), + # Mistral v11+ ``[TOOL_CALLS]name{json}`` (may chain), close at ``}``. + re.compile(r"\[TOOL_CALLS\]\s*[\w\.\-]+\s*(?:\[ARGS\])?\s*\{.*?\}", re.DOTALL), + # DeepSeek R1 / V3 / V3.1: full envelope (any opener variant) ... end. + re.compile(_DEEPSEEK_OPEN_RE_SRC + r".*?<｜tool▁calls▁end｜>", re.DOTALL), + # Kimi K2: ``<|tool_calls_section_begin|>...<|tool_calls_section_end|>``. + re.compile(r"<\|tool_calls_section_begin\|>.*?<\|tool_calls_section_end\|>", re.DOTALL), + # Kimi K2 section-less closed call; else the catch-all below eats trailing prose to EOS. + re.compile(r"<\|tool_call_begin\|>.*?<\|tool_call_end\|>", re.DOTALL), ] _TOOL_ALL_PATS = _TOOL_CLOSED_PATS + [ re.compile(r".*$", re.DOTALL), re.compile(r'.*$', re.DOTALL), - # Bare-word markers drop a trailing truncated call only when the next chars look like - # a call start, so prose mentioning the marker is kept; a marker at end-of-text drops. + # Bare-word markers drop a trailing truncated call only when a call-shaped start + # follows; a prose mention (``See [TOOL_CALLS] docs...``) keeps its tail. Bare marker at EOF drops. re.compile(r"<\|tool_call>(?=\s*call\s*:|\s*$).*$", re.DOTALL), re.compile( r"\[TOOL_CALLS\](?=\s*(?:[\[{]|[A-Za-z_][\w.\-]*[\[{])|\s*$).*$", @@ -67,6 +97,22 @@ _TOOL_ALL_PATS = _TOOL_CLOSED_PATS + [ r"<\|python_tag\|>(?=\s*(?:\{|[A-Za-z_][\w.]*\()|\s*$).*$", re.DOTALL, ), + # DeepSeek envelopes truncated mid-stream (any opener); same call-shaped lookahead as above. + re.compile( + _DEEPSEEK_OPEN_RE_SRC + r"(?=\s*(?:<｜tool▁call▁begin｜>|function)|\s*$).*$", + re.DOTALL, + ), + re.compile(r"<｜tool▁call▁begin｜>(?=\s*function|\s*$).*$", re.DOTALL), + # Kimi K2 envelope truncated. + re.compile( + r"<\|tool_calls_section_begin\|>(?=\s*<\|tool_call_begin\|>|\s*$).*$", + re.DOTALL, + ), + re.compile( + r"<\|tool_call_begin\|>(?=\s*[A-Za-z_][\w.\-]*:\d|\s*$).*$", + re.DOTALL, + ), + # Gemma wrapper-less ``call:NAME{...}`` is handled by ``_strip_gemma_wrapperless_calls`` (enabled-name gate). ] @@ -105,8 +151,7 @@ BUDGET_EXHAUSTED_NUDGE = ( "any more tools." ) -# The exact-args dup guard misses paraphrased re-searches, so also cap executed -# KB searches per turn, then nudge. +# The exact-args dup guard misses paraphrased re-searches, so also cap KB searches per turn. RAG_MAX_SEARCHES_PER_TURN = 3 RAG_SEARCH_CAP_NUDGE = ( "You have already searched the knowledge base several times this turn. " @@ -117,14 +162,16 @@ RAG_SEARCH_CAP_NUDGE = ( # Qwen / Hermes ``{json}``. _TC_JSON_START_RE = re.compile(r"\s*\{") -# Qwen3.5 ```` plus attribute form ```` (MiniCPM-5, -# MiniMax-M2); name in group(1) or group(2). +# Qwen3.5 ```` and the attribute form ```` +# (MiniCPM-5, MiniMax-M2); name class ``[\w.\-]+`` lands in group(1) or group(2). _TC_FUNC_START_RE = re.compile(r'\s*') -# Body ends at ```` or ```` so trailing prose stays out of args. +# Body ends at ```` (Hermes) or ```` (Qwen3.5 / MiniCPM-5) +# so it stops at the close even when prose follows (else prose leaked into args). _TC_END_TAG_RE = re.compile(r"") _TC_FUNC_CLOSE_RE = re.compile(r"\s*\s*$") -# Horizontal whitespace only so the wrapping newline + indent survive (``_trim_param_value`` -# trims one newline), preserving code indent. +# Horizontal whitespace only (``[^\S\n]*``, not ``\s*``) so the wrapping newline + +# first-line indentation survive; ``_trim_param_value`` trims one newline, preserving +# code indentation (SGLang qwen3_coder). _TC_PARAM_START_RE = re.compile( r'<(?:parameter|param)(?:=([\w\.\-]+)|\s+name="([\w\.\-]+)")>[^\S\n]*' ) @@ -135,35 +182,81 @@ _LLAMA3_PYTHON_TAG = "<|python_tag|>" _LLAMA3_PY_CALL_RE = re.compile( r"<\|python_tag\|>\s*([\w\.\-]+)\s*\.\s*call\s*\(", ) -# Anchored at the char after ``<|python_tag|>`` plus the ``; NAME.call(`` chain sep, so -# a ``.call(`` inside JSON args is ignored. +# Anchored at a fixed offset (char after ``<|python_tag|>``) plus the ``; NAME.call(`` +# chain separator; fixed-offset (not a free scan) ignores ``.call(`` inside JSON args. _LLAMA3_PY_CALL_HEAD_RE = re.compile(r"\s*([\w\.\-]+)\s*\.\s*call\s*\(") _LLAMA3_CALL_CHAIN_RE = re.compile(r"\s*;\s*([\w\.\-]+)\s*\.\s*call\s*\(") -# ``.call(k=v)`` kwarg tokens, hand-scanned below (not finditer) to stay linear on a -# truncated body (ReDoS). +# Llama-3 ``.call(k=v)`` kwarg tokens, hand-scanned below (not finditer) to stay +# linear on a truncated body; finditer retries every offset of a long run (ReDoS). _LLAMA3_KEY_RE = re.compile(r"\w+") _LLAMA3_WS_RE = re.compile(r"\s*") -# ints, decimals, sci notation; trailing ``(?![\w.])`` stops ``1.2.3`` truncating to ``1.2``. +# ints, decimals (1.5, 1., .5) and sci notation; trailing ``(?![\w.])`` stops a token +# like ``1.2.3`` being truncated to ``1.2`` (which would mis-parse the remainder). _LLAMA3_NUM_RE = re.compile(r"-?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?(?![\w.])") _LLAMA3_LIT_RE = re.compile(r"true|false|null") -# Mistral ``[TOOL_CALLS]`` trigger. v11+ chains ``name{json}`` (Magistral) or -# ``name[ARGS]{json}`` (Ministral / Large 3). +# Mistral ``[TOOL_CALLS]`` trigger. v11+ chains them, each followed by a bare name +# plus ``{json}`` (Magistral) or ``[ARGS]{json}`` (Ministral / Large 3). _MISTRAL_TRIGGER = "[TOOL_CALLS]" _MISTRAL_ARGS_MARKER = "[ARGS]" -# Mistral Small 3.2 emits ``name[CALL_ID][ARGS]{json}`` (absent on Ministral / Magistral). +# Mistral Small 3.2 emits ``name[CALL_ID][ARGS]{json}`` (absent on Ministral / +# Magistral); llama.cpp distinguishes the two on ``[CALL_ID]`` (common/chat.cpp). _MISTRAL_CALL_ID_MARKER = "[CALL_ID]" -# Magistral wraps reasoning in ``[THINK]...[/THINK]``; a ``[TOOL_CALLS]`` inside is not a real call. +# Magistral wraps reasoning in ``[THINK]...[/THINK]``; a ``[TOOL_CALLS]`` inside +# that block is chain-of-thought, not a real call. _MISTRAL_THINK_OPEN = "[THINK]" _MISTRAL_THINK_CLOSE = "[/THINK]" _MISTRAL_V11_NAME_RE = re.compile(r"\s*([\w\.\-]+)\s*") +# DeepSeek markers (full-width pipe U+FF5C, block U+2581); five outer-open variants like llama.cpp. +_DEEPSEEK_BEGIN_RE = re.compile(_DEEPSEEK_OPEN_RE_SRC) +_DEEPSEEK_END = "<｜tool▁calls▁end｜>" +_DEEPSEEK_CALL_BEGIN = "<｜tool▁call▁begin｜>" +_DEEPSEEK_SEP = "<｜tool▁sep｜>" +_DEEPSEEK_CALL_END = "<｜tool▁call▁end｜>" +# R1 wraps args in a ```json fence with a ``function`` prefix; V3/V3.1 do not. +# Scanned with ``str.find`` -- the regex forms are O(N^2) on truncated bodies. +_DEEPSEEK_R1_FUNC_MARKER = "function" + _DEEPSEEK_SEP +_DEEPSEEK_R1_FENCE = "\n```json\n" +_DEEPSEEK_R1_CLOSE_RE = re.compile(r"```[\s\r\n]*" + re.escape(_DEEPSEEK_CALL_END)) + +# GLM 4.5-4.7: ``NAME[\n]K...``; the lookahead also allows a +# direct ````/```` (4.7 drops the newline, zero-arg calls close at once). +# Name class ``[\w.\-]+`` keeps prose like ``not a call`` unparsed; +# ``{`` stays with the Qwen JSON parser. +_GLM_TC_OPEN_RE = re.compile(r"\s*([\w.\-]+)\s*(?=\n||)") +_GLM_TC_CLOSE = "" +_GLM_ARG_KEY_OPEN = "" +_GLM_ARG_KEY_CLOSE = "" +_GLM_ARG_VAL_OPEN = "" +_GLM_ARG_VAL_CLOSE = "" +# Strings arrive raw, non-strings via tojson; only unambiguous JSON literals decode +# (bare ``42``/``true``/``null`` stay strings). +_GLM_JSON_NUMERIC_RE = re.compile(r"-?\d+(?:\.\d+)?(?:[eE][+-]?\d+)?") + +# Kimi K2 / Moonshot (ASCII pipes). Id ``functions.NAME:IDX`` -- strip ``functions.``/``:N`` for the name. +_KIMI_SECTION_BEGIN = "<|tool_calls_section_begin|>" +_KIMI_SECTION_END = "<|tool_calls_section_end|>" +_KIMI_CALL_BEGIN = "<|tool_call_begin|>" +_KIMI_ARG_BEGIN = "<|tool_call_argument_begin|>" +_KIMI_CALL_END = "<|tool_call_end|>" +_KIMI_ID_RE = re.compile(r"^(?:functions\.)?([\w\.\-]+)(?::(\d+))?$") + # Gemma 4: ``<|tool_call>call:NAME{...}``, ``<|"|>`` wraps strings. _GEMMA_TC_RE = re.compile(r"<\|tool_call>\s*call\s*:\s*([\w\.\-]+)\s*\{") _GEMMA_STR_BEGIN = '<|"|>' _GEMMA_STR_END = '<|"|>' _GEMMA_TC_END = "" +# skip_special_tokens strips the wrapper and ``<|"|>`` markers, so streamed Gemma calls +# arrive as bare ``call:NAME{k:v, ...}``; ``(? int | None: """Index of the ``]`` matching ``[`` at ``text[start]`` (ignores brackets in JSON strings).""" @@ -215,7 +308,8 @@ def _skip_mistral_call_id(text: str, pos: int) -> int: def _strip_mistral_reasoning(content: str) -> str: - """Drop a leading Magistral ``[THINK]`` block so rehearsed calls inside reasoning are not promoted; unclosed drops to EOF.""" + """Drop a leading Magistral ``[THINK]...[/THINK]`` so a ``[TOOL_CALLS]`` inside + reasoning is not taken as a real call; an unclosed ``[THINK]`` drops from it on.""" i = 0 n = len(content) while i < n and content[i] in " \t\n\r": @@ -229,7 +323,10 @@ def _strip_mistral_reasoning(content: str) -> str: def _strip_mistral_closed_calls(text: str) -> str: - """Strip cleanly-closed ``[TOOL_CALLS]`` blocks via balanced scanning (a non-greedy regex would truncate nested JSON); unclosed runs wait for ``final=True``.""" + """Strip cleanly-closed ``[TOOL_CALLS]`` blocks (array, ``name{json}``, + ``name[ARGS]{json}``) via balanced scanning -- a non-greedy ``\\{.*?\\}`` would + truncate at the first ``}`` and lose nested JSON. Unclosed runs are left for + ``final=True`` cleanup.""" n = len(text) out = [] cursor = 0 @@ -254,7 +351,8 @@ def _strip_mistral_closed_calls(text: str) -> str: if text.startswith("", cursor): cursor += len("") continue - # Single-object shape ``[TOOL_CALLS] { json }``: the parser accepts it, so strip it too. + # Single-object shape ``[TOOL_CALLS] { json }`` (no name/array): the parser + # accepts it, so the display strip must remove it too (else it leaks). if i < n and text[i] == "{": end = _balanced_brace_end(text, i) if end is None: @@ -287,12 +385,50 @@ def _strip_mistral_closed_calls(text: str) -> str: out.append(text[idx:]) break cursor = end + 1 - # Consume the optional EOS marker so ``...{json}`` doesn't leave ```` as content. + # Consume the optional EOS marker too, mirroring the array shape, so a + # ``[TOOL_CALLS]name{json}`` tail doesn't leave ```` as content. if text.startswith("", cursor): cursor += len("") return "".join(out) +def _strip_gemma_wrapperless_calls(text: str, enabled_tool_names: Optional[set] = None) -> str: + """Strip closed wrapper-less Gemma ``call:NAME{...}`` calls with balanced brace + scanning (nested arguments are removed whole). ``enabled_tool_names`` gates the + strip like the parser gate: a disabled/example name stays visible; ``None`` + strips every closed call.""" + if _whole_content_is_json_value(text): + return text + n = len(text) + out = [] + # Mirror the parse scan: a leading JSON answer's span is data, kept visible. + cursor = _leading_json_value_end(text) or 0 + if cursor: + out.append(text[:cursor]) + while cursor < n: + m = _GEMMA_BARE_TC_RE.search(text, cursor) + if not m: + out.append(text[cursor:]) + break + disabled = enabled_tool_names is not None and m.group(1) not in enabled_tool_names + brace = m.end() - 1 # _GEMMA_BARE_TC_RE consumes through the opening ``{`` + # Same boundary scanner as the parser: strip exactly what it consumed. + end = _gemma_body_brace_end(text, brace) + closed = end is not None + next_index = (end + 1) if closed else len(text) + if not closed: + # Unclosed call: drop an enabled call to EOS; keep a disabled/example name as prose. + out.append(text[cursor:] if disabled else text[cursor : m.start()]) + break + if disabled: + # Disabled/example name is prose: keep it whole. + out.append(text[cursor:next_index]) + else: + out.append(text[cursor : m.start()]) + cursor = next_index # already past the matching ``}`` + return "".join(out) + + _FUNC_CLOSE_TAG_RE = re.compile(r"") @@ -327,16 +463,131 @@ def _strip_function_xml_calls(text: str, *, final: bool) -> str: return "".join(out) -def strip_tool_markup(text: str, *, final: bool = False) -> str: - """Strip tool-call markup; ``final=True`` also drops trailing unclosed runs and trims.""" +def _glm_value_close( + text: str, + vs: int, + *, + strict: bool = False, +) -> int: + """Index of the ```` that really ends the GLM value at ``vs``: the + first one whose next non-space token is ````, ```` or + end-of-text AND that sits at balanced quote state (an embedded literal pair + like ``print("")`` lives inside a still-open string). + Quote openers are contextual (single quote only after punctuation, so + apostrophes are prose; double quote also at word start), mirroring the Gemma + scanners. If no candidate balances, the first token-valid one wins -- except + in ``strict`` mode (Auto-Heal off), which refuses the in-quote fallback rather + than execute truncated arguments. Returns -1 if unclosed.""" + n = len(text) + search = vs + first_candidate = -1 + quote = "" + prev = ":" + prev_raw = ":" + qpos = vs # quote-state cursor; advanced incrementally to each candidate + while True: + ve = text.find(_GLM_ARG_VAL_CLOSE, search) + if ve < 0: + return -1 if strict else first_candidate + j = ve + len(_GLM_ARG_VAL_CLOSE) + while j < n and text[j] in " \t\r\n": + j += 1 + if j >= n or text.startswith(_GLM_ARG_KEY_OPEN, j) or text.startswith(_GLM_TC_CLOSE, j): + while qpos < ve: + ch = text[qpos] + if quote: + if ch == "\\" and qpos + 1 < ve: + qpos += 2 + continue + if ch == quote: + quote = "" + elif ch in "\"'" and (prev in ":{[(,=" or (ch == '"' and prev_raw.isspace())): + quote = ch + if not ch.isspace(): + prev = ch + prev_raw = ch + qpos += 1 + if not quote: + return ve + if first_candidate < 0: + first_candidate = ve + search = ve + len(_GLM_ARG_VAL_CLOSE) + + +def _strip_glm_calls(text: str, *, final: bool) -> str: + """Strip GLM 4.x calls by scanning to each call's REAL ```` (the one + after the last consumed ````, mirroring ``_parse_glm_tool_calls``), so + a literal ```` inside a value is data. Qwen ``{json}`` has + no NAME token and is left to the regex arms. ``final`` drops a truncated call to + EOS; otherwise it stays buffered.""" + out: list[str] = [] + cursor = 0 + n = len(text) + while True: + m = _GLM_TC_OPEN_RE.search(text, cursor) + if not m: + break + apos = m.end() + close = -1 + while True: + ks = text.find(_GLM_ARG_KEY_OPEN, apos) + tc = text.find(_GLM_TC_CLOSE, apos) + if tc >= 0 and (ks < 0 or tc < ks): + close = tc + break + if ks < 0: + break # no close and no more keys -- truncated body + ke = text.find(_GLM_ARG_KEY_CLOSE, ks + len(_GLM_ARG_KEY_OPEN)) + if ke < 0: + break + vstart = ke + len(_GLM_ARG_KEY_CLOSE) + while vstart < n and text[vstart] in " \t\r\n": + vstart += 1 + if not text.startswith(_GLM_ARG_VAL_OPEN, vstart): + apos = ke + len(_GLM_ARG_KEY_CLOSE) + continue + vs = vstart + len(_GLM_ARG_VAL_OPEN) + ve = _glm_value_close(text, vs) + if ve < 0: + break # unclosed -- truncated + apos = ve + len(_GLM_ARG_VAL_CLOSE) + if close >= 0: + out.append(text[cursor : m.start()]) + cursor = close + len(_GLM_TC_CLOSE) + continue + # Truncated GLM call (no real close yet). + if final: + out.append(text[cursor : m.start()]) + cursor = n + # Non-final: leave the unclosed call (and any tail) buffered as-is. + break + out.append(text[cursor:]) + return "".join(out) + + +def strip_tool_markup( + text: str, + *, + final: bool = False, + enabled_tool_names: Optional[set] = None, +) -> str: + """Strip tool-call markup. ``final=False`` keeps in-progress markup buffered; + ``final=True`` also drops trailing unclosed runs and trims. ``enabled_tool_names`` + gates the markerless Gemma ``call:NAME{...}`` strip so a disabled/example name in + prose is kept (mirrors the parser gate); ``None`` strips every closed call.""" if final: - # End-of-turn only: drop a leading Magistral ``[THINK]...[/THINK]`` block (bracket form, - # not the ```` reasoning channel) so raw reasoning doesn't leak into display/history. + # Drop a leading Magistral ``[THINK]...[/THINK]`` at end-of-turn; its bracket + # form is not the ```` the reasoning channel renders. text = _strip_mistral_reasoning(text) text = _strip_mistral_closed_calls(text) - # Scan-strip the function-XML form first (parser-accurate: a literal ```` in - # a value is data, not a call); the regex arms below cover the other formats. + if final: + text = _strip_gemma_wrapperless_calls(text, enabled_tool_names) + # Scan-strip the function-XML form (a literal ```` inside a value is + # data). The regex arms below cover the other formats but no-op on function calls here. text = _strip_function_xml_calls(text, final = final) + # GLM 4.x: scan to the call's real so a literal one inside a value is data, + # not a leak. Qwen {json} is left to the regex arms. + text = _strip_glm_calls(text, final = final) pats = _TOOL_ALL_PATS if final else _TOOL_CLOSED_PATS for pat in pats: text = pat.sub("", text) @@ -347,8 +598,77 @@ def has_tool_signal(text: str) -> bool: return any(s in text for s in TOOL_XML_SIGNALS) +# A Qwen/Hermes ````/```` envelope whose arguments carry literal +# DeepSeek/Kimi markers must parse as the OUTER call. Detect it opening before the first +# marker so the pre-pass skips it. +_EMBEDDED_MARKER_RE = re.compile( + _DEEPSEEK_OPEN_RE_SRC + "|" + re.escape(_KIMI_SECTION_BEGIN) + "|" + re.escape(_KIMI_CALL_BEGIN) +) +# Covers ```` and the attribute form. ``<|python_tag|>`` is Llama-3's +# envelope too (built-in ``NAME.call(`` and custom ``{json}``), so a quoted DeepSeek/Kimi +# example is data; the call-shaped lookahead mirrors the ``_TOOL_ALL_PATS`` python_tag arm +# so a bare prose ``<|python_tag|>`` mention isn't treated as one. +_OUTER_ENVELOPE_OPEN_RE = re.compile( + r'|' + r"|<\|python_tag\|>(?=\s*(?:\{|[A-Za-z_][\w.]*\())" +) +# CLOSED outer envelopes, each spanning to its REAL final close so a literal +# ````/```` inside a value is data. Wrapped Gemma counts too. +_OUTER_ENVELOPE_CLOSED_PATS = ( + re.compile(r"(?:(?!).)*", re.DOTALL), + _TOOL_CLOSED_PATS[1], + re.compile(r"<\|tool_call>.*?", re.DOTALL), +) + + +def _marker_inside_leading_envelope(content: str, enabled_tool_names: Optional[set] = None) -> bool: + first_marker = _EMBEDDED_MARKER_RE.search(content) + if first_marker is None: + return False + # A leading bare-JSON or Mistral [TOOL_CALLS] call is an outer envelope too: + # a DS/Kimi marker in its argument strings is data. + i = 0 + n = len(content) + while i < n and content[i] in " \t\n\r": + i += 1 + if content.startswith("{", i): + end = _balanced_brace_end(content, i) + if end is not None and i < first_marker.start(): + name = _top_level_bare_json_name(content[i : end + 1]) + if name is not None and (enabled_tool_names is None or name in enabled_tool_names): + # The closed leading call owns the turn: a marker inside it is argument + # data, one after it a trailing example (same rule as the XML envelopes below). + return True + if name is not None and first_marker.start() <= end: + # A disabled-name leading object is prose (can't own the turn), but a marker + # inside its own strings stays data. A marker AFTER it falls through to the pre-pass. + return True + elif content.startswith(_MISTRAL_TRIGGER, i): + end = _mistral_region_end(content, i) + if end is not None and i < first_marker.start(): + return True + # A closed outer call PRECEDING the first marker owns the turn; the pre-pass must + # not steal a trailing example or argument data. + for _pat in _OUTER_ENVELOPE_CLOSED_PATS: + m = _pat.search(content) + if m is not None and m.start() < first_marker.start(): + return True + residue = content + for _pat in _OUTER_ENVELOPE_CLOSED_PATS: + residue = _pat.sub("", residue) + marker = _EMBEDDED_MARKER_RE.search(residue) + if marker is None: + return True + # A marker still stands; any opener left in the residue is UNCLOSED. One before the + # marker is a truncated outer call holding the marker as data: skip the pre-pass. + opener = _OUTER_ENVELOPE_OPEN_RE.search(residue) + return opener is not None and opener.start() < marker.start() + + def _mistral_region_end(text: str, idx: int) -> int | None: - """Exclusive end of the balanced ``[TOOL_CALLS]`` call at ``idx``, or ``None`` when truncated (array, object, and named forms).""" + """Exclusive end of the balanced ``[TOOL_CALLS]`` call starting at ``idx``, + or ``None`` when truncated/unrecognised (same shapes as the strip scan: + array, single-object, and named ``name [CALL_ID]? [ARGS]? {json}``).""" n = len(text) i = idx + len(_MISTRAL_TRIGGER) while i < n and text[i] in " \t\n\r": @@ -384,8 +704,10 @@ def _xml_signal_inside_leading_mistral(content: str) -> bool: first_xml = _first_foreign_tool_signal(content) if first_xml is not None and first_xml < trig: return False - # Only plain prose precedes the trigger (preamble-tolerant); prose merely mentioning - # the marker has no parseable region and keeps the normal order. + # Only plain prose precedes the trigger: a visible preface must not hand + # the turn to a later XML literal (preamble-tolerant, like the + # wrapperless-Gemma guard). Prose that merely mentions the marker has no + # parseable region and keeps the normal order. return _mistral_region_end(content, trig) is not None @@ -393,7 +715,8 @@ _ATTR_FUNC_OPEN_RE = re.compile(r' int | None: - """Offset of the first signal a non-envelope parser would fire on (XML forms plus the Llama-3 ``<|python_tag|>`` marker).""" + """Offset of the first tool signal a non-envelope parser would fire on + (XML forms plus ``<|python_tag|>``, which also runs before the Mistral parser).""" first = None for sig in ("", "<|tool_call>", ""): p = content.find(sig) @@ -402,37 +725,126 @@ def _first_foreign_tool_signal(content: str) -> int | None: attr = _ATTR_FUNC_OPEN_RE.search(content) if attr is not None and (first is None or attr.start() < first): first = attr.start() + # DeepSeek/Kimi markers are foreign to a JSON envelope too: a marker inside a leading + # object routes through the same guard (and, if disabled, the drop-and-parse-the-tail + # recursion, so a real call after the object is still reached). + marker = _EMBEDDED_MARKER_RE.search(content) + if marker is not None and (first is None or marker.start() < first): + first = marker.start() return first def _xml_signal_inside_leading_bare_json(content: str) -> bool: - """True when the first foreign signal sits inside a LEADING bare-JSON call's balanced body: quoted argument data, so the bare-JSON parser takes the outer call first.""" + """True when the first foreign tool signal is a quoted literal inside a + LEADING bare-JSON call object or JSON answer -- data, not a real call + (sibling of ``_xml_signal_inside_leading_mistral``).""" i = 0 n = len(content) while i < n and content[i] in " \t\n\r": i += 1 - if i >= n or content[i] != "{": + if i >= n or content[i] not in "{[": return False + if content[i] == "[": + # A leading array is only ever a structured answer; its literals are data. + end = _balanced_bracket_end(content, i) + if end is None: + return False + try: + json.loads(content[i : end + 1]) + except ValueError: + return False + first_xml = _first_foreign_tool_signal(content) + trig = content.find(_MISTRAL_TRIGGER) + if trig >= 0 and (first_xml is None or trig < first_xml): + first_xml = trig + return first_xml is not None and i < first_xml < end end = _balanced_brace_end(content, i) if end is None: return False if _top_level_bare_json_name(content[i : end + 1]) is None: - # Not a call object, but a nameless object that parses as real JSON is an envelope - # too (markup in its strings is data); non-JSON braced prose keeps the old behaviour. + # A NAMELESS object that parses as real JSON is a structured answer / envelope too: + # quoted markup is data, and the decline path drops it and parses the tail. + # Non-JSON braced prose keeps the old behaviour. try: json.loads(content[i : end + 1]) except ValueError: return False first_xml = _first_foreign_tool_signal(content) - # The Mistral trigger is foreign to a JSON envelope too, so fold it into first_xml. + # The Mistral trigger is foreign to a JSON envelope too (its parser runs first). trig = content.find(_MISTRAL_TRIGGER) if trig >= 0 and (first_xml is None or trig < first_xml): first_xml = trig - # Inside the balanced body the signal is quoted argument data, so the leading call owns - # the turn; a non-call object takes the decline path (dropped, only the tail parsed). + # Inside the balanced body the signal is quoted data; after the closed object the + # leading call still owns the turn (mirrors the leading-Mistral rule). return first_xml is not None and i < first_xml +def _signal_inside_leading_wrapperless_gemma( + content: str, enabled_tool_names: Optional[set] +) -> bool: + """True when the first foreign tool signal is a quoted literal inside (or + after) a LEADING enabled wrapper-less Gemma call (sibling of the + Mistral/bare-JSON leading guards). Markerless form, so gated on an enabled + name (``None`` keeps the name-agnostic behaviour).""" + first = _first_foreign_tool_signal(content) + # The Mistral trigger is foreign to a Gemma call too (its parser runs first). + trig = content.find(_MISTRAL_TRIGGER) + if trig >= 0 and (first is None or trig < first): + first = trig + if first is None: + return False + # A preamble before ``call:NAME{...}`` is normal; what matters is an ENABLED balanced + # call beginning before the first foreign signal. + cursor = 0 + while True: + m = _GEMMA_BARE_TC_RE.search(content, cursor) + if m is None or m.start() > first: + return False + if enabled_tool_names is not None and m.group(1) not in enabled_tool_names: + cursor = m.end() + continue + end = _gemma_body_brace_end(content, m.end() - 1) + if end is None: + return False + if m.end() - 1 < first <= end: + return True + # An enabled call that CLOSES before the signal still owns the turn (inside-or-after + # rule, as for closed bare-JSON/Mistral envelopes), gated on an enabled name. + return enabled_tool_names is not None and end < first + + +def _disabled_gemma_call_end_containing_signal( + content: str, enabled_tool_names: Optional[set] +) -> int | None: + """End offset (exclusive) of the earliest DISABLED wrapper-less Gemma call + whose balanced body contains the first foreign signal, else None. A disabled + name is prose, so the quoted literal is data: the caller drops the span and + recurses on the tail. An ENABLED call defers to the enabled-call guard.""" + if enabled_tool_names is None: + return None + first = _first_foreign_tool_signal(content) + # Mirror the enabled-call guard: the Mistral trigger is foreign here too. + trig = content.find(_MISTRAL_TRIGGER) + if trig >= 0 and (first is None or trig < first): + first = trig + if first is None: + return None + cursor = 0 + while True: + m = _GEMMA_BARE_TC_RE.search(content, cursor) + if m is None or m.start() > first: + return None + if m.group(1) in enabled_tool_names: + return None + end = _gemma_body_brace_end(content, m.end() - 1) + if end is None: + cursor = m.end() + continue + if m.end() - 1 < first <= end: + return end + 1 + cursor = end + 1 + + def parse_tool_calls_from_text( content: str, *, @@ -440,26 +852,35 @@ def parse_tool_calls_from_text( allow_incomplete: bool = True, enabled_tool_names: Optional[set] = None, ) -> list[dict]: - """Return OpenAI-format tool calls, first-match wins. ``allow_incomplete`` heals truncated calls (``False`` = strict closed-only); ``enabled_tool_names`` gates the markerless bare-JSON form.""" - # Drop Magistral reasoning before any dispatch so a rehearsed call inside - # [THINK]...[/THINK] is not promoted; keeps the parse path aligned with the display strip. + """Return OpenAI-format tool calls, first-match wins so calls are never double-counted. + + ``allow_incomplete=True`` (default) heals truncated calls (missing close tag / + unclosed parameter); ``False`` accepts only well-formed closed calls (trailing + prose tolerated), matching llama-server's strict path when Auto-Heal is off. + + ``enabled_tool_names`` gates only the markerless Llama-3.2 bare-JSON form (the + marker-based forms carry an explicit signal, so a disabled-tool name there is a + real call attempt). ``None`` keeps the name-agnostic behaviour.""" + # Drop Magistral [THINK]...[/THINK] BEFORE dispatch: a rehearsed call inside it must + # never be promoted, and the parse path must agree with the display strip. content = _strip_mistral_reasoning(content) - # A leading bare-JSON value is decided FIRST so markup quoted in its arguments stays - # data. Must precede the Mistral guard, whose preamble tolerance would else claim a - # trigger quoted inside the leading object. + # A leading bare-JSON value is decided FIRST: a string argument quoting tool markup + # (XML or a Mistral trigger) must stay data, so the bare-JSON parser takes the outer + # call before any other pass. Precedes the Mistral guard, whose preamble tolerance + # would otherwise claim a trigger quoted inside the leading object. if _xml_signal_inside_leading_bare_json(content): calls = _parse_llama3_bare_json( content, id_offset = id_offset, enabled_tool_names = enabled_tool_names ) if calls: return calls - # Disabled/example name: the leading object is ordinary content. Drop it and parse - # only the tail -- a real call after it still parses, nothing inside it is promoted. + # Disabled/example name: the leading object is content. Drop it and parse the tail. i = 0 while i < len(content) and content[i] in " \t\n\r": i += 1 - end = _balanced_brace_end(content, i) # guard guarantees a balanced object + # The guard guarantees a balanced leading value (object or array). + end = (_balanced_brace_end if content[i] == "{" else _balanced_bracket_end)(content, i) return parse_tool_calls_from_text( content[end + 1 :], id_offset = id_offset, @@ -467,8 +888,32 @@ def parse_tool_calls_from_text( enabled_tool_names = enabled_tool_names, ) + # A leading enabled wrapper-less Gemma call is decided BEFORE the Mistral guard: its + # body reads as prose to the preamble tolerance below, so a quoted [TOOL_CALLS] would + # otherwise steal the turn. + if _signal_inside_leading_wrapperless_gemma(content, enabled_tool_names): + calls = _parse_gemma_tool_calls( + content, + id_offset = id_offset, + allow_incomplete = allow_incomplete, + enabled_tool_names = enabled_tool_names, + ) + if calls: + return calls + + # A DISABLED wrapper-less Gemma call is prose: drop the span and parse the tail BEFORE + # the Mistral guard, whose preamble tolerance would otherwise parse a quoted trigger. + _prose_end = _disabled_gemma_call_end_containing_signal(content, enabled_tool_names) + if _prose_end is not None: + return parse_tool_calls_from_text( + content[_prose_end:], + id_offset = id_offset, + allow_incomplete = allow_incomplete, + enabled_tool_names = enabled_tool_names, + ) + # A [TOOL_CALLS] call that is the first tool emission owns the turn: XML quoted in its - # arguments or in trailing prose is not promoted, and a plain-prose preface keeps it. + # arguments or trailing prose is not promoted over it, nor does a prose preface forfeit it. if _xml_signal_inside_leading_mistral(content): calls = _parse_mistral_tool_calls( content, id_offset = id_offset, allow_incomplete = allow_incomplete @@ -476,8 +921,29 @@ def parse_tool_calls_from_text( if calls: return calls - # A leading MiniCPM/MiniMax ```` call owns the turn: tool_healing - # does not know the wrapper, so gate it here. A signal before the opener keeps normal order. + # DeepSeek/Kimi markers are unique, so try them first -- unless an outer envelope + # opens before the first marker (then the marker is argument data). + if not _marker_inside_leading_envelope(content, enabled_tool_names): + # Dispatch by earliest opener so a quoted DS example inside a Kimi call (or vice + # versa) can't hijack the turn via fixed parser order. + _ds = _DEEPSEEK_BEGIN_RE.search(content) + _ds_pos = _ds.start() if _ds else len(content) + _km_section = content.find(_KIMI_SECTION_BEGIN) + _km_bare = content.find(_KIMI_CALL_BEGIN) + _km_pos = min(p for p in (_km_section, _km_bare, len(content)) if p >= 0) + pre_pass = [ + (_ds_pos, _parse_deepseek_tool_calls), + (_km_pos, _parse_kimi_tool_calls), + ] + pre_pass.sort(key = lambda pair: pair[0]) + for _pos, parser in pre_pass: + calls = parser(content, id_offset = id_offset, allow_incomplete = allow_incomplete) + if calls: + return calls + + # A leading MiniCPM/MiniMax attribute-form call owns the turn: tool_healing doesn't know + # the wrapper, so a quoted in its parameter would beat + # the outer call. Any earlier signal keeps normal order. attr = _ATTR_FUNC_OPEN_RE.search(content) if attr is not None: first_other = None @@ -518,8 +984,9 @@ def parse_tool_calls_from_text( if calls: return calls - # Qwen/Hermes, Qwen3.5 XML, and Gemma 4 use the shared tool_healing parser (the - # strict/Auto-Heal + nested-marker + ``<|"|>`` handling GGUF relies on). + # Qwen/Hermes, Qwen3.5 XML, and Gemma 4 go through the shared tool_healing + # parser (strict/Auto-Heal contract + nested-marker, trailing-prose, and + # ``<|"|>`` quoted-string handling the GGUF path relies on). calls = _tool_healing.parse_tool_calls_from_text( content, id_offset = id_offset, @@ -528,11 +995,11 @@ def parse_tool_calls_from_text( if calls: return calls - # Formats tool_healing does not cover: ```` (MiniCPM-5 / MiniMax-M2), - # Llama-3 and Mistral. Run only after tool_healing found nothing, so a strict-rejected - # call is never re-healed here. Blank any JSON/Gemma marker coverage first: markup inside - # a marker's span (even one that failed to parse) is that call's data, not a sibling, so - # a nested ```` / ``<|python_tag|>`` / ``[TOOL_CALLS]`` must not be promoted. + # Formats tool_healing does not cover; these run only after it finds + # nothing, so a strict-rejected call is never re-healed here. Blank any + # JSON/Gemma marker coverage first: markup inside a marker's span (even one + # that failed to parse) is that call's data, not a sibling, so a nested + # ```` / ``<|python_tag|>`` / ``[TOOL_CALLS]`` must not be promoted. fallback_content = content coverage = _tool_healing.marker_coverage(content) if coverage: @@ -542,6 +1009,7 @@ def parse_tool_calls_from_text( chars[i] = " " fallback_content = "".join(chars) for parser in ( + _parse_glm_tool_calls, # GLM 4.x name _parse_function_xml, # attribute form _parse_llama3_python_tag, # Llama-3 <|python_tag|> _parse_mistral_tool_calls, # Mistral [TOOL_CALLS] @@ -550,11 +1018,22 @@ def parse_tool_calls_from_text( if calls: return calls - # Llama-3.2 bare ``{"name":..., "parameters":...}``. Strict (starts with ``{`` - # and parses to the right shape) so plain prose stays untouched. - return _parse_llama3_bare_json( + # Llama-3.2 bare ``{"name":..., "parameters":...}`` (strict shape). Only a LEADING call + # object matches and owns the turn, so an enabled ``call:NAME{...}`` in its arguments + # stays data (Gemma never starts ``{``). + calls = _parse_llama3_bare_json( content, id_offset = id_offset, enabled_tool_names = enabled_tool_names ) + if calls: + return calls + + # Gemma wrapper-less ``call:NAME{...}``: markerless, so the same enabled-name gate applies. + return _parse_gemma_tool_calls( + content, + id_offset = id_offset, + allow_incomplete = allow_incomplete, + enabled_tool_names = enabled_tool_names, + ) def _parse_tool_call_json( @@ -569,8 +1048,9 @@ def _parse_tool_call_json( end = _balanced_brace_end(content, brace_start) if end is None: continue - # Strict mode: a balanced body that never closed its ```` is truncated - # (trailing prose after the close is still tolerated). + # Strict mode: a balanced JSON body that never closed its ```` + # is a truncated call, not a finished one. Trailing prose after the close + # is still tolerated (matches the GGUF strict path). if not allow_incomplete and not content[end + 1 :].lstrip().startswith(""): continue try: @@ -578,7 +1058,7 @@ def _parse_tool_call_json( except (json.JSONDecodeError, ValueError): continue name = obj.get("name", "") - # Accept both ``arguments`` (Hermes/Qwen) and ``parameters`` (Llama-3 drift). + # Accept ``arguments`` (Hermes/Qwen) and ``parameters`` (Llama-3 drift). args = obj.get("arguments") if args is None: args = obj.get("parameters", {}) @@ -601,7 +1081,10 @@ def _parse_tool_call_json( def _trim_param_value(val: str) -> str: - """Trim only the template's wrapping newline around an XML parameter value; ``str.strip()`` destroyed code/diff indentation.""" + """Trim one wrapping newline the template adds around an XML parameter value + (``\nVALUE\n``), preserving inner indentation. + ``str.strip()`` destroyed code/diff indentation; SGLang's qwen3_coder trims only + the wrapping newline.""" if val.startswith("\n"): val = val[1:] if val.endswith("\n"): @@ -610,14 +1093,19 @@ def _trim_param_value(val: str) -> str: def _inside_open_parameter(text: str, pos: int) -> bool: - """True if ``pos`` is inside an unclosed ```` block, i.e. the opener at ``pos`` is literal argument data, not a nested call.""" + """True if ``pos`` sits inside an unclosed ````/```` block -- + i.e. a ```` / ```` opener at ``pos`` is a literal inside an + argument value (e.g. code that prints tool-call XML), not a real nested call. + Compares the last parameter opener before ``pos`` against the last + parameter/function close before it.""" last_param_open = -1 for m in _TC_PARAM_START_RE.finditer(text, 0, pos): last_param_open = m.start() if last_param_open < 0: return False - # The parameter's OWN close tag decides: if it closes after ``pos`` the position is - # argument data (even across literal ````); an unclosed one falls back to func close. + # The parameter's OWN close tag decides: while it closes after ``pos`` the position is + # argument data, even across several literal function closes. Only an unclosed + # parameter (heal mode) falls back to the first function close. own_closes = [ c for c in ( @@ -647,7 +1135,7 @@ def _parse_function_xml( ) -> list[dict]: out: list[dict] = [] # Skip ```` openers that are literals inside an open parameter value, - # else the nested marker becomes a second call and truncates the real argument. + # else the nested marker is promoted to a second call and truncates the real argument. func_starts = [ fm for fm in _TC_FUNC_START_RE.finditer(content) @@ -658,9 +1146,10 @@ def _parse_function_xml( func_name = fm.group(1) or fm.group(2) body_start = fm.end() next_func = func_starts[idx + 1].start() if idx + 1 < len(func_starts) else len(content) - # The call ends at the FIRST / not inside an open parameter: - # a literal close in an argument is skipped as data, prose after the real close is not - # folded in (mirrors _strip_function_xml_calls). + # The call ends at the FIRST / not inside an open + # parameter: a literal close in a code/search argument is skipped as data, and + # prose after the real close isn't folded into the last argument (mirrors + # _strip_function_xml_calls and tool_healing._func_close_index). close_match = None for cm in _TC_END_TAG_RE.finditer(content, body_start, next_func): if not _inside_open_parameter(content, cm.start()): @@ -671,14 +1160,14 @@ def _parse_function_xml( body_end = close_match.start() else: body_end = min(len(content), next_func) - # Strict mode: a call that never reached its close is truncated; do not heal it. + # Strict mode: an unclosed function call is truncated -- do not heal it. if not allow_incomplete and not has_close: continue body = _TC_FUNC_CLOSE_RE.sub("", content[body_start:body_end]) args: dict = {} param_unclosed = False - # Same nested-literal guard: a ```` opener inside an open value is literal text. + # A ```` opener inside an open parameter value is literal text. param_starts = [ pm for pm in _TC_PARAM_START_RE.finditer(body) @@ -703,8 +1192,8 @@ def _parse_function_xml( val = _TC_PARAM_CLOSE_RE.sub("", raw_val) args[pm.group(1) or pm.group(2)] = _trim_param_value(val) - # Strict mode: every parameter must close; a dangling one means the call was cut off. - # A closed call with no parameters is a valid zero-argument call, so keep it. + # Strict mode: a dangling parameter means the call was cut off; a closed + # zero-parameter call stays valid. if not allow_incomplete and param_unclosed: continue @@ -719,7 +1208,8 @@ def _parse_function_xml( def _llama3_kv_value(body: str, p: int, n: int) -> tuple[Any, int | None]: - """One ``.call`` value at ``body[p:]``; returns ``(value, len)`` or ``(None, None)``.""" + """One ``.call`` value (string/number/true/false/null) at ``body[p:]``. + Returns ``(value, consumed_len)`` or ``(None, None)`` if none matches.""" if p >= n: return None, None if body[p] == '"': @@ -745,7 +1235,8 @@ def _llama3_kv_value(body: str, p: int, n: int) -> tuple[Any, int | None]: nm = _LLAMA3_NUM_RE.match(body, p) if nm: v = nm.group(0) - # Sci notation and decimals decode as float; a bare integer stays int. + # Scientific notation (1e-3, -2E+4, 0.5e2) and decimals decode as float; a bare + # integer stays int. ``"." in v`` alone missed the exponent forms (1e-3 -> 1). return (float(v) if any(c in v for c in ".eE") else int(v)), nm.end() - p lm = _LLAMA3_LIT_RE.match(body, p) if lm: @@ -754,7 +1245,8 @@ def _llama3_kv_value(body: str, p: int, n: int) -> tuple[Any, int | None]: def _parse_llama3_kv_args(body: str) -> dict[str, Any]: - """Left-to-right ``k=v`` kwargs from a ``.call(...)`` body (linear scan; later keys win).""" + """``k=v, ...`` kwargs from a ``.call(...)`` body, left to right (later keys win). + Linear hand-scan replacing the quadratic ``_LLAMA3_KV_RE.finditer`` walk.""" args: dict[str, Any] = {} n = len(body) i = 0 @@ -783,13 +1275,17 @@ def _parse_llama3_python_tag( id_offset: int, allow_incomplete: bool = True, ) -> list[dict]: - """Parse Llama-3 ``<|python_tag|>`` emissions: ``NAME.call(...)``, bare JSON, ``; `` multi-call, ``parameters``/``arguments`` keys.""" + """Parse the Llama-3 emissions: ``<|python_tag|>NAME.call(...)`` (built-in), + ``<|python_tag|>{"name":..., "parameters":...}`` (custom), multi-call via + ``; ``, ``parameters`` or ``arguments`` key.""" out: list[dict] = [] if _LLAMA3_PYTHON_TAG not in content: return out - # 1. ``NAME.call(...)`` built-in form, anchored to ``<|python_tag|>`` (optionally - # ``; ``-chained) so a ``.call(...)`` inside a JSON string argument isn't mistaken for one. + # 1. ``NAME.call(...)`` built-in form, anchored to ``<|python_tag|>`` and optionally + # ``; ``-chained within one emission. Anchoring to the tag boundary (not a free scan) + # keeps a literal ``<|python_tag|>x.call(...)`` quoted in a custom-form JSON argument + # from being mistaken for a real built-in call. pos = content.find(_LLAMA3_PYTHON_TAG) truncated = False while pos >= 0 and not truncated: @@ -824,7 +1320,8 @@ def _parse_llama3_python_tag( if depth == 0: break i += 1 - # Truncated ``.call(...)`` (no closing paren): reject in strict mode. + # Truncated ``.call(...)`` with no closing paren: reject in strict mode + # instead of executing a partial. if not allow_incomplete and depth > 0: truncated = True break @@ -848,7 +1345,8 @@ def _parse_llama3_python_tag( # Past the consumed region: a second ``<|python_tag|>`` may carry more calls. pos = content.find(_LLAMA3_PYTHON_TAG, i + 1) - # 2. ``<|python_tag|>{"name":.., "parameters":..}``; raw_decode peels ``; ``-separated objects. + # 2. ``<|python_tag|>{"name":..., "parameters":...}``. ``raw_decode`` peels multiple + # ``; ``-separated objects from one emission. if not out: decoder = json.JSONDecoder() idx = content.find(_LLAMA3_PYTHON_TAG) @@ -873,12 +1371,14 @@ def _parse_llama3_python_tag( continue name = obj.get("name") or obj.get("function") or "" args = obj.get("parameters") if "parameters" in obj else obj.get("arguments", {}) + # Skip rather than fabricate ``{"value": args}`` for a non-dict/non-string value. if isinstance(args, dict): args_str = json.dumps(args) elif isinstance(args, str): args_str = args else: - args_str = json.dumps({"value": args}) + cursor = brace + end_offset + continue if name: out.append( { @@ -892,7 +1392,8 @@ def _parse_llama3_python_tag( return out -# Llama-3 special-token sentinels (chainable, any order) plus the header role label. +# Llama-3 special-token sentinels (chainable, any order) plus the role label the +# template inserts between ``<|start_header_id|>`` and ``<|end_header_id|>``. _LLAMA3_BARE_JSON_SENTINELS = ( "<|begin_of_text|>", "<|eot_id|>", @@ -904,7 +1405,10 @@ _LLAMA3_HEADER_ROLES = ("assistant", "user", "system", "tool", "ipython") def strip_llama3_leading_sentinels(content: str) -> str: - """Strip leading Llama-3 sentinels leaked from a prior turn; shared by the parser and the streaming guards.""" + """Strip leading Llama-3 special-token sentinels (and the role label after + ``<|start_header_id|>``) that can leak from a prior turn before a bare-JSON tool + call. Shared by the parser and the streaming buffering guards so a + sentinel-prefixed ``{"name":...}`` is recognised the same everywhere.""" stripped = content.lstrip() while True: stripped = stripped.lstrip() @@ -930,7 +1434,9 @@ def _parse_llama3_bare_json( allow_incomplete: bool = True, enabled_tool_names: Optional[set] = None, ) -> list[dict]: - """Llama-3.2 bare ``{"name":.., "parameters":..}`` (strict). ``enabled_tool_names`` keeps ordinary JSON answers from being misread; ``None`` is name-agnostic.""" + """Llama-3.2 ``custom_tools`` bare ``{"name":.., "parameters":{..}}`` (no ``<|python_tag|>``), + strict so prose/echoes don't fire. ``enabled_tool_names`` gates on the parsed name so an + ordinary JSON answer isn't misread as a call to a disabled tool; ``None`` is name-agnostic.""" out: list[dict] = [] stripped = strip_llama3_leading_sentinels(content) if not stripped.startswith("{"): @@ -954,11 +1460,12 @@ def _parse_llama3_bare_json( name = obj.get("name") or obj.get("function") or "" if not isinstance(name, str) or not name: break - # Markerless JSON is ambiguous: only a call when the name is an enabled tool. + # Markerless JSON is ambiguous: treat it as a call only when the name is an enabled + # tool, else it is an ordinary JSON answer. if enabled_tool_names is not None and name not in enabled_tool_names: break - # ``parameters`` must be a dict (Llama-3 spec); ``arguments`` may be a dict or a - # JSON-string of one (OpenAI). + # ``parameters`` must be a dict (Llama-3 spec); ``arguments`` may be a dict or + # JSON-string of one (OpenAI). Looser would fire on ``{"name":"x","parameters":"sentence"}``. if "parameters" in obj: args = obj.get("parameters") if not isinstance(args, dict): @@ -997,14 +1504,15 @@ def _parse_mistral_tool_calls( id_offset: int, allow_incomplete: bool = True, ) -> list[dict]: - """Parse Mistral ``[TOOL_CALLS]`` emissions: pre-v11 array/object and v11+ named forms.""" + """Parse all Mistral emissions: pre-v11 ``[TOOL_CALLS][...]`` / ``[TOOL_CALLS]{...}`` + and v11+ ``[TOOL_CALLS]name{json}`` / ``[TOOL_CALLS]name[ARGS]{json}``.""" out: list[dict] = [] content = _strip_mistral_reasoning(content) idx = content.find(_MISTRAL_TRIGGER) if idx < 0: return out - # Disambiguate the first occurrence: array / single object (pre-v11) or bare-name (v11+). + # Disambiguate the first occurrence: array / single object (pre-v11), or bare-name (v11+). j = idx + len(_MISTRAL_TRIGGER) k = j while k < len(content) and content[k] in " \t\n\r": @@ -1016,7 +1524,7 @@ def _parse_mistral_tool_calls( return _parse_mistral_array(content, k, id_offset, allow_incomplete = allow_incomplete) if content[k] == "{": - # Pre-v11 single ``{"name":...}``; fall through to v11+ if it carries no ``name``. + # Pre-v11 single ``{"name":...}``; fall through without a ``name`` so v11+ still runs. end = _balanced_brace_end(content, k) if end is not None: try: @@ -1027,7 +1535,8 @@ def _parse_mistral_tool_calls( except (json.JSONDecodeError, ValueError): pass - # v11+: walk every ``[TOOL_CALLS]``, parsing ``name{json}`` or ``name[ARGS]{json}``. + # v11+: walk every ``[TOOL_CALLS]``, parsing ``name{json}`` or + # ``name[ARGS]{json}`` after each trigger. pos = idx while pos >= 0: cur = pos + len(_MISTRAL_TRIGGER) @@ -1101,7 +1610,8 @@ def _parse_mistral_array( if depth == 0: break j += 1 - # An unclosed array (no matching ]) is truncated; reject in strict mode. + # An unclosed array (no matching ]) is a truncated call. In strict mode reject it + # instead of recovering objects by hand below. if not allow_incomplete and depth != 0: return out body = content[start : j + 1] if depth == 0 else content[start:] @@ -1117,8 +1627,8 @@ def _parse_mistral_array( if not allow_incomplete: return out - # Healing path for unclosed arrays: walk top-level objects, advancing past each - # balanced ``{...}`` (re-scanning from every ``{`` would be quadratic ReDoS). + # Healing path for unclosed arrays: walk top-level objects, advancing past each balanced + # ``{...}`` instead of re-scanning from every ``{`` (quadratic ReDoS). pos = 0 blen = len(body) while pos < blen: @@ -1141,7 +1651,8 @@ def _consume_mistral_call(obj_text: str, out: list[dict], id_offset: int) -> Non if not isinstance(obj, dict): return name = obj.get("name") or "" - # Mistral uses ``arguments``; accept the ``parameters`` alias too. + # Mistral uses ``arguments``; accept the ``parameters`` alias too (sibling paths and + # SGLang's base detector alias it) so an array object keyed on it keeps args. args = obj.get("arguments") if args is None: args = obj.get("parameters", {}) @@ -1161,28 +1672,86 @@ def _consume_mistral_call(obj_text: str, out: list[dict], id_offset: int) -> Non ) +def _whole_content_is_json_value(text: str) -> bool: + """True when the entire content is one valid JSON value (a structured + answer, e.g. a response_format turn). Markerless scans must treat text + inside it as data: an answer documenting an enabled tool's syntax must + not execute that tool or have the example stripped from display.""" + t = text.strip() + if t[:1] not in "{[": + return False + try: + json.loads(t) + except ValueError: + return False + return True + + +def _leading_json_value_end(text: str) -> int | None: + """End index (exclusive) of a balanced LEADING JSON value that parses as + JSON: a structured answer possibly followed by prose. Markerless scans treat + its contents as data (extends ``_whole_content_is_json_value``); leading-keyed, + so a JSON blob mid-prose is not an answer span.""" + i = 0 + n = len(text) + while i < n and text[i].isspace(): + i += 1 + if i >= n or text[i] not in "{[": + return None + end = (_balanced_brace_end if text[i] == "{" else _balanced_bracket_end)(text, i) + if end is None: + return None + try: + json.loads(text[i : end + 1]) + except ValueError: + return None + return end + 1 + + def _parse_gemma_tool_calls( content: str, *, id_offset: int, allow_incomplete: bool = True, + enabled_tool_names: Optional[set] = None, ) -> list[dict]: - """Gemma 4: ``<|tool_call>call:NAME{k:<|"|>v<|"|>, ...}``.""" + """Gemma 4: ``<|tool_call>call:NAME{k:<|"|>v<|"|>, ...}``, plus the + ``skip_special_tokens`` stream where the wrapper and string markers were + stripped (bare ``call:NAME{k:v, ...}``). + + ``enabled_tool_names`` gates on the parsed name: the wrapper-less shape is + indistinguishable from prose documenting the syntax, so a disabled/example + name must not be stolen as a call. ``None`` keeps the name-agnostic behaviour.""" out: list[dict] = [] - for m in _GEMMA_TC_RE.finditer(content): + # The WRAPPED form (strict + nested-marker handling) is tool_healing's, which runs + # first: defer content with a wrapped opener. A marker literal alone is not enough -- + # a wrapper-less call mentioning ``<|tool_call>`` would be lost if deferred. + if _GEMMA_TC_RE.search(content): + return out + # A whole-content JSON value is a structured answer: quoted examples must not become calls. + if _whole_content_is_json_value(content): + return out + # Manual cursor: resume AFTER each consumed balanced body so a nested ``call:OTHER{...}`` + # in an argument is never re-matched. A leading JSON answer's span is data -- scan after it. + cursor = _leading_json_value_end(content) or 0 + while True: + m = _GEMMA_BARE_TC_RE.search(content, cursor) + if m is None: + break name = m.group(1) body_start = m.end() - 1 - end_marker = content.find(_GEMMA_TC_END, body_start) - # No closing tag: truncated call, reject in strict mode. - if not allow_incomplete and end_marker < 0: - continue - scan_end = end_marker if end_marker >= 0 else len(content) - end = _gemma_balanced_brace_end(content, body_start, scan_end) + end = _gemma_body_brace_end(content, body_start) if end is None: + # Unclosed call: nothing parseable follows (mirrors the strip contract); + # scanning on would promote quoted argument text. + break + cursor = end + 1 + # Markerless: a disabled/example name is prose, not a call. + if enabled_tool_names is not None and name not in enabled_tool_names: continue body = content[body_start + 1 : end] try: - args = _gemma_parse_mapping_body(body) + args = _gemma_parse_stripped_body(body) except Exception: args = {} out.append( @@ -1225,11 +1794,54 @@ def _balanced_brace_end(text: str, brace_pos: int) -> int | None: return None +def _gemma_body_brace_end(text: str, brace_pos: int) -> int | None: + """Index of the ``}`` closing the wrapper-less Gemma body at ``brace_pos``. + + Values are raw after ``skip_special_tokens``, so quoted strings (single or + double) hide braces; the quote rules mirror ``_gemma_parse_stripped_body`` so + the boundary always agrees with the body parser. Contextual openers: a single + quote opens only at value-start context (after ``:{[(,=`` -- apostrophes in + ``what's the weather`` are prose), a double quote also at word start (so + ``query:find "a, b"`` hides its delimiters).""" + if brace_pos >= len(text) or text[brace_pos] != "{": + return None + depth = 0 + quote = "" + prev = "" + prev_raw = "" + i = brace_pos + n = len(text) + while i < n: + ch = text[i] + if quote: + if ch == "\\" and i + 1 < n: + i += 2 + continue + if ch == quote: + quote = "" + elif ch in "\"'" and (prev in ":{[(,=" or (ch == '"' and prev_raw.isspace())): + quote = ch + elif ch == "{": + depth += 1 + elif ch == "}": + depth -= 1 + if depth == 0: + return i + if not ch.isspace(): + prev = ch + prev_raw = ch + i += 1 + return None + + _BARE_JSON_NAME_RE = re.compile(r'"name"\s*:\s*"([^"]+)"') def _top_level_bare_json_name(probe: str) -> Optional[str]: - """Top-level ``"name"`` (or ``"function"`` alias) of a bare-JSON object, else None; nested objects are skipped and truncated tails return None.""" + """TOP-LEVEL ``"name"`` (or ``"function"`` alias, name wins) of a bare-JSON object, else None. + + Skips nested objects/arrays so a nested ``"name"`` isn't mistaken for the call name; a + truncated tail returns None so the caller keeps the text.""" if not probe.startswith("{"): return None decoder = json.JSONDecoder() @@ -1240,7 +1852,7 @@ def _top_level_bare_json_name(probe: str) -> Optional[str]: while i < n and probe[i] in " \t\r\n,": i += 1 if i >= n or probe[i] == "}": - # End of object, no top-level ``"name"``: fall back to the ``"function"`` alias. + # End of the object with no top-level ``"name"``: fall back to a recorded ``"function"`` alias. return function_value if probe[i] != '"': return None @@ -1267,7 +1879,8 @@ def _top_level_bare_json_name(probe: str) -> Optional[str]: return value if isinstance(value, str) else None return None if key == "function" and function_value is None and i < n and probe[i] == '"': - # ``"function"`` is an alias; record it but keep scanning (``"name"`` wins). + # ``"function"`` aliases the call name. Record it but keep scanning: a top-level + # ``"name"`` still wins. try: value, consumed = decoder.raw_decode(probe[i:]) except (json.JSONDecodeError, ValueError): @@ -1276,7 +1889,8 @@ def _top_level_bare_json_name(probe: str) -> Optional[str]: function_value = value i += consumed continue - # Skip a non-name top-level value; a truncated one returns None (keep the text). + # Skip a non-name top-level value; a truncated one can't prove a top-level name + # exists, so return None (keep the text). if i < n and probe[i] == "{": end = _balanced_brace_end(probe, i) if end is None: @@ -1314,16 +1928,18 @@ def strip_leading_bare_json_call(text: str, enabled_tool_names: Optional[set] = if not (probe.startswith("{") and ('"name"' in probe or '"function"' in probe)): return probe.lstrip() if stripped_any else text if enabled_tool_names is not None: - # Only suppress when the leading object's TOP-LEVEL name is an enabled tool - # (a nested ``"name"`` is data); an unknown name is kept. + # Only suppress when the leading object's TOP-LEVEL name is an enabled tool. A + # nested ``"name"`` (e.g. {"result":{"name":"web_search",...}}) is data, not the + # call name, so it must not gate the strip. An un-extractable name is kept. name = _top_level_bare_json_name(probe) if name not in enabled_tool_names: return probe.lstrip() if stripped_any else text end = _balanced_brace_end(probe, 0) if end is None: return "" # truncated bare-JSON call -- nothing recoverable - # A closed object must have the CALL SHAPE the parser accepts; an ordinary JSON - # answer it rejects is content, so keep it visible. + # A closed object must have the CALL SHAPE the parser accepts (dict ``parameters``, + # or dict / JSON-string ``arguments``). An ordinary JSON answer like + # {"name":"web_search","result":"no call"} is content, so the strip keeps it visible. try: obj = json.loads(probe[: end + 1]) except (json.JSONDecodeError, ValueError): @@ -1338,7 +1954,8 @@ def _bare_json_call_shaped(obj) -> bool: """The shape gate ``_parse_llama3_bare_json`` applies to a decoded object.""" if not isinstance(obj, dict): return False - # The parser requires a TOP-LEVEL name; a nested one is data, not the call name. + # The parser requires a TOP-LEVEL name; a nested one (e.g. in a "result" value of an + # ordinary JSON answer) is data, and stripping it name-agnostically would delete content. name = obj.get("name") or obj.get("function") or "" if not isinstance(name, str) or not name: return False @@ -1379,101 +1996,670 @@ def _gemma_balanced_brace_end(text: str, brace_pos: int, hard_stop: int) -> int return None -def _gemma_parse_value(text: str, i: int): - """Parse one Gemma arg value at ``i``; returns ``(value, next_index)``.""" +def _gemma_parse_value( + text: str, + i: int, + *, + in_mapping: bool = False, +): + """Parse one Gemma arg value at ``i`` in a single O(n) forward pass; returns + ``(value, next_index, closed)``. ``closed`` is False when a string/object/array + runs off the end without its terminator, so the caller can fall back to raw. + ``in_mapping`` applies the top-level rule that a comma only ends the value + when a ``key:`` follows (array elements split on every top-level comma).""" if text.startswith(_GEMMA_STR_BEGIN, i): close = text.find(_GEMMA_STR_END, i + len(_GEMMA_STR_BEGIN)) if close < 0: - return text[i + len(_GEMMA_STR_BEGIN) :], len(text) - return text[i + len(_GEMMA_STR_BEGIN) : close], close + len(_GEMMA_STR_END) + return text[i + len(_GEMMA_STR_BEGIN) :], len(text), False + return text[i + len(_GEMMA_STR_BEGIN) : close], close + len(_GEMMA_STR_END), True if text[i] == "{": - end = _gemma_balanced_brace_end(text, i, len(text)) - if end is None: - return {}, len(text) - return _gemma_parse_mapping_body(text[i + 1 : end]), end + 1 + return _gemma_parse_mapping(text, i) if text[i] == "[": - j, depth = i, 0 - while j < len(text): - if text.startswith(_GEMMA_STR_BEGIN, j): - k = text.find(_GEMMA_STR_END, j + len(_GEMMA_STR_BEGIN)) - if k < 0: - j = len(text) - break - j = k + len(_GEMMA_STR_END) + return _gemma_parse_array(text, i) + if text[i] in "\"'": + # Raw-quoted string: delimiters inside are data (``{city:"New, York"}`` is one + # value); returned unquoted like the top-level scalar coercion. + quote = text[i] + j = i + 1 + n = len(text) + while j < n: + if text[j] == "\\" and j + 1 < n: + j += 2 continue - ch = text[j] - if ch == "[": - depth += 1 - elif ch == "]": - depth -= 1 - if depth == 0: - break + if text[j] == quote: + return text[i + 1 : j], j + 1, True j += 1 - body = text[i + 1 : j] - items: list[Any] = [] - k = 0 - while k < len(body): - if body[k] in " \t\n\r,": - k += 1 - continue - v, k = _gemma_parse_value(body, k) - items.append(v) - return items, j + 1 - # Primitive: number / true/false/null / bare identifier. + return text[i + 1 :], n, False + # Primitive / unquoted code: same delimiter rules as the top-level scan (bracket depth + # + contextual quote openers hide commas and closers). end = i - while end < len(text) and text[end] not in ",}]" and not text.startswith(_GEMMA_STR_BEGIN, end): + n = len(text) + depth = 0 + quote = "" + prev = ":" + prev_raw = ":" + while end < n and not text.startswith(_GEMMA_STR_BEGIN, end): + ch = text[end] + if quote: + if ch == "\\" and end + 1 < n: + end += 2 + continue + if ch == quote: + quote = "" + elif ch in "\"'" and (prev in ":{[(,=" or (ch == '"' and prev_raw.isspace())): + quote = ch + elif ch in "{[(": + depth += 1 + elif ch in "}])": + if depth == 0: + break + depth -= 1 + elif ch == "," and depth == 0: + if not in_mapping or _GEMMA_KEY_RE.match(text, end + 1): + break + if not ch.isspace(): + prev = ch + prev_raw = ch end += 1 if end == i: - # Stray delimiter, nothing consumed: advance past it so callers can't spin forever. - return "", i + 1 + # Stray delimiter where a value was expected: consume one char so callers always + # advance (no infinite loop on malformed input). + return "", i + 1, True raw = text[i:end].strip() if raw == "true": - return True, end + return True, end, True if raw == "false": - return False, end + return False, end, True if raw == "null": - return None, end + return None, end, True try: - return int(raw), end + return int(raw), end, True except ValueError: pass try: - return float(raw), end + return float(raw), end, True except ValueError: pass - return raw, end + return raw, end, True -def _gemma_parse_mapping_body(body: str) -> dict[str, Any]: - """Parse a Gemma argument mapping (content between `{` and `}`).""" - out: dict[str, Any] = {} - i = 0 - n = len(body) +def _gemma_parse_array(text: str, start: int): + """Parse a Gemma ``[...]`` array at ``text[start] == '['`` in one forward + pass; returns ``(list, next_index, closed)``.""" + items: list[Any] = [] + i, n = start + 1, len(text) while i < n: - while i < n and body[i] in " \t\n\r,": + while i < n and text[i] in " \t\n\r,": i += 1 + if i < n and text[i] == "]": + return items, i + 1, True if i >= n: break - if body.startswith(_GEMMA_STR_BEGIN, i): - close = body.find(_GEMMA_STR_END, i + len(_GEMMA_STR_BEGIN)) + v, i, _closed = _gemma_parse_value(text, i) + items.append(v) + return items, i, False + + +def _gemma_coerce_scalar(raw: str) -> Any: + """Coerce an unquoted Gemma value to bool/int/float/None, else keep str + (quotes stripped first so quoted/unquoted variants compare identical).""" + raw = raw.strip() + if len(raw) >= 2 and raw[0] == raw[-1] and raw[0] in "\"'": + return raw[1:-1] + if raw == "true": + return True + if raw == "false": + return False + if raw == "null": + return None + try: + return int(raw) + except ValueError: + pass + try: + return float(raw) + except ValueError: + pass + return raw + + +def _gemma_strip_quoted_leaves(value: Any) -> Any: + """Recursively unquote quoted string leaves of a nested stripped-stream value, + so nested ``city:"New York"`` matches the top-level coercion (no stray quotes).""" + if isinstance(value, str): + v = value.strip() + if len(v) >= 2 and v[0] == v[-1] and v[0] in "\"'": + return v[1:-1] + return value + if isinstance(value, dict): + return {k: _gemma_strip_quoted_leaves(v) for k, v in value.items()} + if isinstance(value, list): + return [_gemma_strip_quoted_leaves(v) for v in value] + return value + + +def _gemma_parse_stripped_body(body: str) -> dict[str, Any]: + """Parse a quote-less Gemma arg body ``key:value, key2:value2`` (the + ``skip_special_tokens`` stream with ``<|"|>`` markers removed). Each value runs + to the next top-level ``, key:`` boundary, tracking ``{}``/``[]``/``()`` depth so + commas/braces inside a ``code`` / ``command`` value aren't truncated.""" + out: dict[str, Any] = {} + i, n = 0, len(body) + while i < n: + m = _GEMMA_KEY_RE.match(body, i) + if not m: + break + key = m.group(1) + i = m.end() + vstart = i + depth = 0 + quote = "" + # Contextual quote openers mirror _gemma_body_brace_end. + prev = ":" + prev_raw = ":" + while i < n: + ch = body[i] + if quote: + # A ``, key:`` shape inside the quoted string is not a boundary. + if ch == "\\" and i + 1 < n: + i += 2 + continue + if ch == quote: + quote = "" + elif ch in "\"'" and (prev in ":{[(,=" or (ch == '"' and prev_raw.isspace())): + quote = ch + elif ch in "{[(": + depth += 1 + elif ch in "}])": + if depth > 0: + depth -= 1 + elif ch == "," and depth == 0 and _GEMMA_KEY_RE.match(body, i + 1): + break + if not ch.isspace(): + prev = ch + prev_raw = ch + i += 1 + raw_val = body[vstart:i].strip() + if raw_val[:1] in "{[": + # Nested object/array: accept only a fully consumed, closed parse; a + # truncated/malformed value falls back to the raw string. + parsed, end, closed = _gemma_parse_value(raw_val, 0) + out[key] = ( + _gemma_strip_quoted_leaves(parsed) + if (closed and end == len(raw_val)) + else _gemma_coerce_scalar(raw_val) + ) + else: + out[key] = _gemma_coerce_scalar(raw_val) + if i < n and body[i] == ",": + i += 1 + return out + + +def _gemma_parse_mapping(text: str, start: int): + """Parse a Gemma ``{key:value, ...}`` mapping at ``text[start] == '{'`` in one + forward pass; returns ``(dict, next_index, closed)`` (``closed`` True iff the + matching ``}`` was reached).""" + out: dict[str, Any] = {} + i, n = start + 1, len(text) + while i < n: + while i < n and text[i] in " \t\n\r,": + i += 1 + if i < n and text[i] == "}": + return out, i + 1, True + if i >= n: + break + if text.startswith(_GEMMA_STR_BEGIN, i): + close = text.find(_GEMMA_STR_END, i + len(_GEMMA_STR_BEGIN)) if close < 0: break - key = body[i + len(_GEMMA_STR_BEGIN) : close] + key = text[i + len(_GEMMA_STR_BEGIN) : close] i = close + len(_GEMMA_STR_END) else: kstart = i - while i < n and body[i] != ":": + while i < n and text[i] not in ":}": i += 1 - key = body[kstart:i].strip() - while i < n and body[i] in " \t\n\r": + key = text[kstart:i].strip() + while i < n and text[i] in " \t\n\r": i += 1 - if i < n and body[i] == ":": + if i < n and text[i] == ":": i += 1 - while i < n and body[i] in " \t\n\r": + while i < n and text[i] in " \t\n\r": i += 1 if i >= n: out[key] = None break - v, i = _gemma_parse_value(body, i) + if text[i] == "}": + out[key] = None + return out, i + 1, True + v, i, _closed = _gemma_parse_value(text, i, in_mapping = True) out[key] = v + return out, i, False + + +# ── DeepSeek R1 / V3 / V3.1 ───────────────────────────────────────── + + +def _find_outside_json_strings(text: str, needle: str, start: int) -> int: + """Index of ``needle`` at/after ``start`` OUTSIDE any JSON string, or -1: a + marker inside an argument string must not be taken as the structural terminator.""" + i = start + n = len(text) + in_string = False + esc = False + while i < n: + ch = text[i] + if in_string: + if esc: + esc = False + elif ch == "\\": + esc = True + elif ch == '"': + in_string = False + i += 1 + continue + if ch == '"': + in_string = True + i += 1 + continue + if text.startswith(needle, i): + return i + i += 1 + return -1 + + +def _parse_deepseek_tool_calls( + content: str, + *, + id_offset: int, + allow_incomplete: bool = True, +) -> list[dict]: + """DeepSeek R1 / V3 / V3.1. + + R1: ``<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>NAME\\n``\\`\\`\\`json\\n{...}\\n\\`\\`\\`<｜tool▁call▁end｜>...`` + V3.x: ``<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>NAME<｜tool▁sep｜>{json}<｜tool▁call▁end｜>...`` + + Mirrors llama.cpp's pre-autoparser ``common_chat_parse_deepseek_r1`` / + ``_v3_1`` handling; tolerates the 5 opener variants llama.cpp keeps. + """ + out: list[dict] = [] + begin = _DEEPSEEK_BEGIN_RE.search(content) + if not begin: + return out + scan_start = begin.end() + # Envelope end OUTSIDE JSON strings: an argument may contain the literal end token, + # and a raw find would truncate the call. + end_pos = _find_outside_json_strings(content, _DEEPSEEK_END, scan_start) + # Strict mode: an unclosed envelope is truncated; reject, don't heal to EOF. + if not allow_incomplete and end_pos < 0: + return out + scan_end = end_pos if end_pos >= 0 else len(content) + body = content[scan_start:scan_end] + + # R1 path first: ``function<｜tool▁sep｜>NAME\n```json\n{...}\n```<｜tool▁call▁end｜>``. + pos = 0 + while pos < len(body): + fpos = body.find(_DEEPSEEK_R1_FUNC_MARKER, pos) + if fpos < 0: + break + name_start = fpos + len(_DEEPSEEK_R1_FUNC_MARKER) + nl = body.find("\n", name_start) + if nl < 0: + break + if not body.startswith(_DEEPSEEK_R1_FENCE, nl): + pos = name_start + continue + name = body[name_start:nl].strip() + json_start = nl + len(_DEEPSEEK_R1_FENCE) + # Walk a balanced ``{`` even if the trailing fence is truncated. + if json_start >= len(body) or body[json_start] != "{": + pos = json_start + continue + brace_end = _balanced_brace_end(body, json_start) + if brace_end is None: + break + try: + args = json.loads(body[json_start : brace_end + 1]) + except (json.JSONDecodeError, ValueError): + pos = brace_end + 1 + continue + if not isinstance(args, dict): + pos = brace_end + 1 + continue + # The closing fence + <｜tool▁call▁end｜> must IMMEDIATELY follow the JSON, else an + # unbounded search lands on a LATER call's terminator. Absent close: heal past the + # JSON (strict rejects); later well-formed calls are still kept. + after = brace_end + 1 + while after < len(body) and body[after] in " \t\r\n": + after += 1 + close_m = _DEEPSEEK_R1_CLOSE_RE.match(body, after) + if not allow_incomplete and close_m is None: + pos = brace_end + 1 + continue + if name: + out.append( + { + "id": f"call_{id_offset + len(out)}", + "type": "function", + "function": { + "name": name, + "arguments": json.dumps(args), + }, + } + ) + pos = close_m.end() if close_m else brace_end + 1 + if out: + return out + + # V3 / V3.1: name then bare JSON. Use ``str.find`` for the sep marker and walk + # back for the name (a ``[^\n<]+`` regex search is O(N^2) on truncated bodies). + pos = 0 + while pos < len(body): + sep_pos = body.find(_DEEPSEEK_SEP, pos) + if sep_pos < 0: + break + # Walk left from sep_pos to the name start; stop at ``\n`` (turn boundary), ``<`` + # (tag start), or ``>`` (end of an optional ``<｜tool▁call▁begin｜>``). + name_start = sep_pos + while name_start > pos and body[name_start - 1] not in "\n<>": + name_start -= 1 + name = body[name_start:sep_pos].strip() + json_start = sep_pos + len(_DEEPSEEK_SEP) + while json_start < len(body) and body[json_start] in " \t\n\r": + json_start += 1 + if json_start >= len(body) or body[json_start] != "{": + pos = sep_pos + len(_DEEPSEEK_SEP) + continue + brace_end = _balanced_brace_end(body, json_start) + if brace_end is None: + break + # Strict mode: a real V3 call closes with the per-call <｜tool▁call▁end｜>; without + # it the call is truncated/merged, so skip it but keep scanning for a later + # well-formed call (matches Kimi strict). + if not allow_incomplete: + after = brace_end + 1 + while after < len(body) and body[after] in " \t\r\n": + after += 1 + if not body.startswith(_DEEPSEEK_CALL_END, after): + pos = brace_end + 1 + continue + try: + args = json.loads(body[json_start : brace_end + 1]) + except (json.JSONDecodeError, ValueError): + pos = brace_end + 1 + continue + if not isinstance(args, dict): + pos = brace_end + 1 + continue + if name: + out.append( + { + "id": f"call_{id_offset + len(out)}", + "type": "function", + "function": { + "name": name, + "arguments": json.dumps(args), + }, + } + ) + # Advance just past the JSON; seeking the optional <｜tool▁call▁end｜> could land on + # a LATER call's end marker and skip the call between. + pos = brace_end + 1 + return out + + +# ── GLM 4.5 / 4.6 / 4.7 ───────────────────────────────────────────── + + +def _parse_glm_tool_calls( + content: str, + *, + id_offset: int, + allow_incomplete: bool = True, +) -> list[dict]: + """GLM 4.5 / 4.6 / 4.7. + + ``NAME[\\n]K[\\n]V + ...``. Multi-call is back-to-back blocks, no envelope. + Mirrors llama.cpp's GLM 4.x tool-call handling (``common_chat_params_init_glm_4_5`` + plus its generalized XML-style parser, llama.cpp PRs #15904 / #16932). + """ + out: list[dict] = [] + pos = 0 + while pos < len(content): + m = _GLM_TC_OPEN_RE.search(content, pos) + if not m: + break + name = m.group(1).strip() + apos = m.end() # absolute position in ``content``; advances past each pair + + args: dict[str, Any] = {} + valid = True + close = -1 + # Walk arg pairs directly against ``content``: a value may contain a literal + # , so the real close is the before the next . + # ``str.find`` keeps this linear. + while True: + ks = content.find(_GLM_ARG_KEY_OPEN, apos) + tc = content.find(_GLM_TC_CLOSE, apos) + if tc >= 0 and (ks < 0 or tc < ks): + close = tc + break + if ks < 0: + break # no close and no more keys -- truncated body + ke = content.find(_GLM_ARG_KEY_CLOSE, ks + len(_GLM_ARG_KEY_OPEN)) + if ke < 0: + break + vstart = ke + len(_GLM_ARG_KEY_CLOSE) + while vstart < len(content) and content[vstart] in " \t\r\n": + vstart += 1 + if not content.startswith(_GLM_ARG_VAL_OPEN, vstart): + # Key without : strict rejects the call; Auto-Heal skips it. + if not allow_incomplete: + valid = False + apos = ke + len(_GLM_ARG_KEY_CLOSE) + continue + vs = vstart + len(_GLM_ARG_VAL_OPEN) + # A first-match find on would truncate values containing literal + # close tags and execute corrupted arguments. + ve = _glm_value_close(content, vs, strict = not allow_incomplete) + key = content[ks + len(_GLM_ARG_KEY_OPEN) : ke].strip() + if ve < 0: + # Unclosed : strict rejects the whole call; Auto-Heal keeps the + # partial value (a truncated query is not a no-arg call). + if not allow_incomplete: + valid = False + break + # Bound the healed value at the next structural tag, not EOF, so a value + # missing only its can't swallow the markup after it. + nk = content.find(_GLM_ARG_KEY_OPEN, vs) + tc = content.find(_GLM_TC_CLOSE, vs) + bounds = [b for b in (nk, tc) if b >= 0] + if not bounds: + args[key] = content[vs:].rstrip() + break + bound = min(bounds) + args[key] = content[vs:bound].rstrip() + apos = bound + continue + raw_val = content[vs:ve] + apos = ve + len(_GLM_ARG_VAL_CLOSE) + # Decode only unambiguous JSON literals; else keep the value RAW so whitespace + # in string args survives (matches vLLM glm4_moe). ``"`` is left out of the + # probe: a verbatim string's quotes are meaningful. + probe = raw_val.strip() + if ( + probe[:1] in "{[" + or probe in ("true", "false", "null") + or _GLM_JSON_NUMERIC_RE.fullmatch(probe) + ): + try: + args[key] = json.loads(probe) + continue + except (json.JSONDecodeError, ValueError): + pass + args[key] = raw_val + + # Strict mode: a block with no is truncated; reject it. + if not allow_incomplete and close < 0: + valid = False + + if name and valid: + out.append( + { + "id": f"call_{id_offset + len(out)}", + "type": "function", + "function": { + "name": name, + "arguments": json.dumps(args), + }, + } + ) + pos = close + len(_GLM_TC_CLOSE) if close >= 0 else len(content) + return out + + +# ── Kimi K2 / Moonshot ────────────────────────────────────────────── + + +def _parse_kimi_tool_calls( + content: str, + *, + id_offset: int, + allow_incomplete: bool = True, +) -> list[dict]: + """Kimi K2. + + ``<|tool_calls_section_begin|><|tool_call_begin|>functions.NAME:IDX + <|tool_call_argument_begin|>{json}<|tool_call_end|>... + <|tool_calls_section_end|>``. Full id is preserved on ``tool_calls + [i].id`` for round-trip through the chat template. Outer loop walks + every section in the stream (vLLM / SGLang parity); mirrors llama.cpp's + Kimi K2 handling via its generalized XML-style parser (llama.cpp PR #16932). + """ + out: list[dict] = [] + outer_pos = 0 + while True: + section_start = content.find(_KIMI_SECTION_BEGIN, outer_pos) + if section_start < 0: + break + scan_start = section_start + len(_KIMI_SECTION_BEGIN) + # Section end OUTSIDE JSON strings: an argument may contain the literal end token, + # and a raw find would drop the later valid call. + section_end = _find_outside_json_strings(content, _KIMI_SECTION_END, scan_start) + scan_end = section_end if section_end >= 0 else len(content) + body = content[scan_start:scan_end] + # Truncated tail: parse what we have, then exit. In strict mode a section with no + # <|tool_calls_section_end|> is truncated; reject it instead. + if section_end < 0: + if allow_incomplete: + out.extend( + _parse_kimi_section_body( + body, id_offset = id_offset + len(out), allow_incomplete = True + ) + ) + return out + outer_pos = section_end + len(_KIMI_SECTION_END) + out.extend( + _parse_kimi_section_body( + body, id_offset = id_offset + len(out), allow_incomplete = allow_incomplete + ) + ) + + # The section wrapper is optional (llama.cpp): a bare <|tool_call_begin|> call parses + # as one section when the loop matched nothing. + if not out and _KIMI_CALL_BEGIN in content: + out.extend( + _parse_kimi_section_body( + content, id_offset = id_offset, allow_incomplete = allow_incomplete + ) + ) + return out + + +def _parse_kimi_section_body( + body: str, + *, + id_offset: int, + allow_incomplete: bool = True, +) -> list[dict]: + """Parse one Kimi K2 section body (between begin / end markers).""" + out: list[dict] = [] + pos = 0 + while pos < len(body): + call_start = body.find(_KIMI_CALL_BEGIN, pos) + if call_start < 0: + break + id_start = call_start + len(_KIMI_CALL_BEGIN) + arg_begin = body.find(_KIMI_ARG_BEGIN, id_start) + if arg_begin < 0: + break + full_id = body[id_start:arg_begin].strip() + m = _KIMI_ID_RE.match(full_id) + if m: + # group(1) is the whole name; do NOT split on ``.`` -- a dotted MCP name stays intact. + name = m.group(1) + else: + base = full_id.split(":")[0] + name = base[len("functions.") :] if base.startswith("functions.") else base + # Drop bare-counter ids (``3``, ``42``) -- matches vLLM; SGLang infers the name + # from the tool schema, which we don't have here. + if name.isdigit(): + json_start = arg_begin + len(_KIMI_ARG_BEGIN) + brace_end = ( + _balanced_brace_end(body, json_start) + if (json_start < len(body) and body[json_start] == "{") + else None + ) + if brace_end is None: + pos = arg_begin + len(_KIMI_ARG_BEGIN) + else: + pos = brace_end + 1 + continue + json_start = arg_begin + len(_KIMI_ARG_BEGIN) + # Balanced brace lets a truncated trailing end marker still surface a call. + while json_start < len(body) and body[json_start] in " \t\n\r": + json_start += 1 + if json_start >= len(body) or body[json_start] != "{": + pos = arg_begin + len(_KIMI_ARG_BEGIN) + continue + brace_end = _balanced_brace_end(body, json_start) + if brace_end is None: + # Malformed / truncated JSON: skip this call but keep parsing later ones + # instead of dropping the rest of the section (vLLM recovers them). + nxt = body.find(_KIMI_CALL_BEGIN, json_start) + if nxt < 0: + break + pos = nxt + continue + try: + args = json.loads(body[json_start : brace_end + 1]) + except (json.JSONDecodeError, ValueError): + pos = brace_end + 1 + continue + if not isinstance(args, dict): + pos = brace_end + 1 + continue + if not allow_incomplete: + # Strict mode: this call must close with <|tool_call_end|> before the next + # <|tool_call_begin|>; otherwise it is truncated, so reject it. + end_marker = body.find(_KIMI_CALL_END, brace_end + 1) + next_call = body.find(_KIMI_CALL_BEGIN, brace_end + 1) + if end_marker < 0 or (next_call >= 0 and end_marker > next_call): + pos = brace_end + 1 + continue + if name: + out.append( + { + "id": full_id or f"call_{id_offset + len(out)}", + "type": "function", + "function": { + "name": name, + "arguments": json.dumps(args), + }, + } + ) + # Advance past the JSON; seeking <|tool_call_end|> could skip a following call + # when this one's end marker is missing. + pos = brace_end + 1 return out diff --git a/studio/backend/routes/inference.py b/studio/backend/routes/inference.py index 1a1a93400..3341a9c62 100644 --- a/studio/backend/routes/inference.py +++ b/studio/backend/routes/inference.py @@ -1150,7 +1150,13 @@ from core.inference.key_exchange import decrypt_api_key from core.inference.model_ids import public_model_id from core.inference.api_monitor import api_monitor from core.inference.llama_http import nonstreaming_client -from core.inference.tool_call_parser import _strip_function_xml_calls, _strip_mistral_closed_calls +from core.inference.tool_call_parser import ( + _strip_function_xml_calls, + _strip_gemma_wrapperless_calls, + _strip_glm_calls, + _strip_mistral_closed_calls, +) +from core.inference.tool_call_parser import TOOL_XML_SIGNALS as _PARSER_TOOL_SIGNALS from core.inference.passthrough_healing import ( StreamToolCallHealer, heal_gate, @@ -1309,8 +1315,8 @@ async def artifact_preview_frame(allow_network: bool = False): ) -# Whitespace/escape-tolerant bare-JSON tool-template detector: matches pretty-printed and -# JSON-escaped ``{"name":`` plus the ``"function"`` alias. +# Whitespace/escape-tolerant bare-JSON tool-template detector (matches pretty-printed and +# JSON-escaped ``{"name":`` plus the ``"function"`` alias), mirroring the parser's tolerance. _BARE_JSON_NAME_MARKER_RE = _re.compile(r'\{\s*\\?"(?:name|function)\\?"\s*:') @@ -1324,15 +1330,16 @@ def _detect_safetensors_features(backend, chat_template: Optional[str]) -> dict: model_identifier = model_id, log_source = "safetensors", ) - # Markers the parser recognises; drop the pill if a template advertises tools but uses none. - # The bare-JSON ``{"name":`` form is matched whitespace-tolerantly below. + # Markers any supported parser recognises (template advertises tools but + # uses none -> drop the pill). Reuse the parser's own signal list so this + # gate never drifts (a hand-maintained copy lost the DeepSeek variants); + # ```` is GLM's unique signal, absent from the shared set. The + # bare-JSON ``{"name":`` form is matched below with the whitespace/escape- + # tolerant ``_BARE_JSON_NAME_MARKER_RE`` so pretty-printed or escaped + # templates are not mis-classified as tool-less. _PARSER_MARKERS = ( - "", - "", - "[TOOL_CALLS]", - "<|tool_call>", + *_PARSER_TOOL_SIGNALS, + "", ) if ( flags.get("supports_tools") @@ -1365,7 +1372,12 @@ def _sf_reasoning_prefill_mode( template: Optional[str] = None, reasoning_effort: Optional[str] = None, ) -> bool: - """Whether this request begins inside an unclosed ```` (Qwen3/GLM prefill it). Gated on the standard markers; bespoke channels, gpt-oss, and thinking-disabled requests are excluded. ``enable_thinking=None`` defaults ON.""" + """Whether this request begins INSIDE an unclosed ```` (Qwen3/Qwen3.5/GLM prefill it). + + Gated on the STANDARD ````/```` markers: a bespoke reasoning channel (e.g. gemma) + never emits ````, so prefilled mode would swallow the whole answer -- excluded, as are + gpt-oss and thinking-disabled requests. ``enable_thinking=None`` defaults ON, so plain requests prefill. + """ if features.get("reasoning_style") not in ("enable_thinking", "enable_thinking_effort"): return False tpl = template or "" @@ -1377,8 +1389,11 @@ def _sf_reasoning_prefill_mode( return False if enable_thinking is False: return False - # reasoning_effort="none" disables thinking on enable_thinking_effort (GLM-5.2) models like - # enable_thinking=False; without this the answer is swallowed into empty reasoning_content. + # A reasoning_effort="none" request disables thinking for enable_thinking_effort + # (GLM-5.2) models the same way enable_thinking=False does (see + # ``_request_reasoning_kwargs``). Without this, the model emits no ```` and + # a plain answer is swallowed whole into reasoning_content, leaving the visible + # response empty. if features.get("reasoning_style") == "enable_thinking_effort" and reasoning_effort == "none": return False return True @@ -1654,41 +1669,83 @@ def _apply_rag_nudge(nudge: str, tools: list[dict], *, rag_scope) -> str: return nudge + " " + _RAG_GROUNDING_NUDGE -# Strip leaked tool-call markup: every shared-parser format plus the leak shapes -# ``llama_cpp.py``'s speculative buffer splits across the visible/DRAIN boundary. Mistral -# ``[TOOL_CALLS]`` uses the parser's balanced-brace helper (``\{.*?\}`` would truncate nested JSON). +# Strip leaked tool-call markup: every shared-parser format plus the four leak +# shapes llama_cpp.py's speculative buffer splits across the visible/DRAIN +# boundary. Mistral [TOOL_CALLS] uses the parser's balanced-brace helper (a +# non-greedy regex would truncate nested JSON); the DeepSeek opener alternation +# is the parser's own, so a signal we parse is never left un-stripped. +from core.inference.tool_call_parser import _DEEPSEEK_OPEN_RE_SRC as _DS_OPEN_SRC + _TOOL_XML_RE = _re.compile( - # Hyphen in the name char-class matches MCP tool names with dashes - # (mcp__srv__list-issues) that would otherwise leak past this strip. - # The ``<|python_tag|>`` arm runs to the next REAL Llama sentinel or EOF, so a literal - # ``<|...|>`` token in an argument (e.g. ``<|cite|>``) doesn't truncate the strip. - # ```` plus the ```` attribute form; name class mirrors the parser. - # A CLOSED ``...`` extends to the last ```` before the next - # opener (so a literal ```` in a value can't truncate); this arm runs first. + # Arm order/notes: the closed ```` arm runs first and extends + # to the call's REAL close so a literal ```` in a value does not + # leak the tail; the combined arm still catches ```` and orphan + # tails. The python_tag arm bounds only on REAL Llama control sentinels + # (stopping at any ``<|`` truncated on literal ``<|x|>`` tokens in values). + # The last arms cover DeepSeek envelopes (all opener variants), Kimi section + # blocks, and bare Kimi calls. Name class ``[\w.\-]`` mirrors the parser. + # Those three arms carry a call-shaped lookahead (matching the parser's + # ``_TOOL_ALL_PATS``): a prose answer that merely mentions a marker + # (``See <|tool_call_begin|> in the docs``) is only stripped when a real + # call actually follows the marker, or the marker is a bare fragment at EOF. r'(?:(?!).)*' r'|<(?:tool_call|function(?:=[\w.\-]+|\s+name="[\w.\-]+"))>.*?(?:|\Z)' r"|<\|tool_call>.*?(?:|\Z)" r"|" r"|" r"|<\|python_tag\|>(?:[^<]|<(?!\|(?:eot_id|eom_id|python_tag|start_header_id|end_header_id|begin_of_text|finetune_right_pad_id)\|))*" - # ```` is the attribute-form alias of ````; strip a tail-only orphan. + r"|" + + _DS_OPEN_SRC + + r"(?=\s*(?:<｜tool▁call▁begin｜>|function)|\s*$).*?(?:<｜tool▁calls▁end｜>|\Z)" + r"|<\|tool_calls_section_begin\|>(?=\s*<\|tool_call_begin\|>|\s*$).*?(?:<\|tool_calls_section_end\|>|\Z)" + r"|<\|tool_call_begin\|>(?=\s*[A-Za-z_][\w.\-]*:\d|\s*$).*?(?:<\|tool_call_end\|>|\Z)" + # ```` is the attribute-form alias of ```` (the parser accepts + # both); strip a tail-only orphan close of either spelling. r"|\s*\Z", _re.DOTALL, ) -def _strip_tool_xml(text: str) -> str: - """Mistral balanced-brace helper + guarded function-XML scan + ``_TOOL_XML_RE`` (skips openers inside an open ````).""" - return _TOOL_XML_RE.sub( - "", _strip_function_xml_calls(_strip_mistral_closed_calls(text), final = True) +def _gemma_strip_gate(tools) -> set: + """Enabled tool NAMES gating the wrapper-less Gemma strip (mirrors the + parser/loop gate: only an enabled ``call:foo{...}`` is a call). With NO tools + enabled this returns an EMPTY set, not ``None``: every ``call:NAME{...}`` is + then prose, and ``None`` would strip-all and delete a legitimate answer.""" + names = { + (t.get("function") or {}).get("name") + for t in (tools or []) + if isinstance(t, dict) and isinstance(t.get("function"), dict) + } + names.discard(None) + return names + + +def _strip_tool_xml(text: str, enabled_tool_names: Optional[set] = None) -> str: + """Combine the parser's scan-based strips (Mistral balanced-brace, gated + Gemma wrapper-less, GLM real-close, guarded function-XML) with + ``_TOOL_XML_RE`` -- the scan strips close at each call's REAL terminator so + literal markup inside argument values is data, not a leaked tail. + ``enabled_tool_names`` gates the Gemma strip; ``None`` strips every closed call.""" + cleaned = _strip_glm_calls( + _strip_gemma_wrapperless_calls(_strip_mistral_closed_calls(text), enabled_tool_names), + final = True, ) + cleaned = _strip_function_xml_calls(cleaned, final = True) + return _TOOL_XML_RE.sub("", cleaned) -def _strip_tool_xml_for_display(text: str, *, auto_heal_tool_calls: bool) -> str: - """Route-level tool-call leak cleanup (Auto-Heal only) via ``_strip_tool_xml``.""" +def _strip_tool_xml_for_display( + text: str, + *, + auto_heal_tool_calls: bool, + enabled_tool_names: Optional[set] = None, +) -> str: + """Route-level leak cleanup (Auto-Heal only). Delegates to ``_strip_tool_xml`` + so the Mistral balanced-brace pass runs too (``_TOOL_XML_RE`` alone has no + ``[TOOL_CALLS]`` arm). ``enabled_tool_names`` gates the Gemma strip.""" if not auto_heal_tool_calls: return text - return _strip_tool_xml(text) + return _strip_tool_xml(text, enabled_tool_names) logger = get_logger(__name__) @@ -5960,6 +6017,7 @@ async def openai_chat_completions( _msg["content"] = _strip_tool_xml_for_display( _msg["content"], auto_heal_tool_calls = _gguf_auto_heal_tool_calls, + enabled_tool_names = _gemma_strip_gate(tools_to_use), ).strip() def gguf_generate_with_tools(): @@ -6093,6 +6151,7 @@ async def openai_chat_completions( clean_cumulative = _strip_tool_xml_for_display( raw_cumulative, auto_heal_tool_calls = _gguf_auto_heal_tool_calls, + enabled_tool_names = _gemma_strip_gate(tools_to_use), ) new_text = clean_cumulative[len(prev_text) :] prev_text = clean_cumulative @@ -6199,6 +6258,7 @@ async def openai_chat_completions( full_text = _strip_tool_xml_for_display( event.get("text", ""), auto_heal_tool_calls = _gguf_auto_heal_tool_calls, + enabled_tool_names = _gemma_strip_gate(tools_to_use), ) return full_text, usage, finish finally: @@ -6572,7 +6632,7 @@ async def openai_chat_completions( _sf_features = _detect_safetensors_features(backend, _sf_tpl) # Split prefilled-```` output into reasoning_content deltas (GGUF parity) so the UI - # renders the thinking block for safetensors and MLX. + # renders the thinking block for safetensors AND MLX. _sf_parse_think = bool( _sf_features.get("supports_reasoning") or _sf_features.get("reasoning_always_on") ) @@ -6673,6 +6733,7 @@ async def openai_chat_completions( "content": _strip_tool_xml_for_display( _msg["content"], auto_heal_tool_calls = _sf_auto_heal_tool_calls, + enabled_tool_names = _gemma_strip_gate(_sf_tools_to_use), ).strip(), } ) @@ -6731,7 +6792,7 @@ async def openai_chat_completions( reasoning_extractor = _new_sf_reasoning_extractor() def _sf_flush_reasoning(): - # Drain the extractor at a turn boundary / stream end; only visible text reaches the monitor. + # Drain the extractor at a turn boundary / stream end (GGUF parity); only visible text reaches the monitor. fr, fv = reasoning_extractor.finish() out = [] if fr: @@ -6757,7 +6818,7 @@ async def openai_chat_completions( if event["type"] == "status": if not event["text"]: - # Turn boundary: flush reasoning, then start a fresh extractor. + # Iteration boundary: flush reasoning, then start a fresh extractor for the next turn. for _c in _sf_flush_reasoning(): yield _c prev_text = "" @@ -6773,7 +6834,7 @@ async def openai_chat_completions( if event["type"] in ("tool_start", "tool_end"): if event["type"] == "tool_start": - # Flush reasoning before tool_start so the thinking block closes ahead of the tool card. + # Flush reasoning before the tool_start line so the thinking block closes ahead of the tool card. for _c in _sf_flush_reasoning(): yield _c prev_text = "" @@ -6786,6 +6847,7 @@ async def openai_chat_completions( clean_cumulative = _strip_tool_xml_for_display( raw_cumulative, auto_heal_tool_calls = _sf_auto_heal_tool_calls, + enabled_tool_names = _gemma_strip_gate(_sf_tools_to_use), ) new_text = clean_cumulative[len(prev_text) :] prev_text = clean_cumulative @@ -6877,11 +6939,12 @@ async def openai_chat_completions( full_text = _strip_tool_xml_for_display( event.get("text", ""), auto_heal_tool_calls = _sf_auto_heal_tool_calls, + enabled_tool_names = _gemma_strip_gate(_sf_tools_to_use), ) return full_text content_text = await asyncio.to_thread(_drain_to_text) - # Split prefilled reasoning from the visible answer; monitor gets visible text only. + # Split prefilled reasoning out of the visible answer (GGUF parity); monitor gets visible text only. _reasoning_text, _visible_text = _extract_responses_reasoning( content_text, parse_think_markers = _sf_parse_think, @@ -6980,7 +7043,7 @@ async def openai_chat_completions( yield _chat_role_chunk(completion_id, created, model_name) prev_text = "" - # Split prefilled into reasoning_content deltas. Single turn (no per-turn reset); also MLX. + # Split prefilled into reasoning_content deltas (GGUF parity). Single turn (no per-turn reset); also serves MLX. reasoning_extractor = _new_sf_reasoning_extractor() # Run the sync generator in a thread pool to avoid blocking the # event loop. Critical for compare mode: two SSE requests arrive @@ -7087,7 +7150,7 @@ async def openai_chat_completions( for token in generate(): full_text = token - # Split prefilled reasoning from the visible answer; also covers MLX. + # Split prefilled reasoning from the visible answer (GGUF parity); also covers MLX. _reasoning_text, _visible_text = _extract_responses_reasoning( full_text, parse_think_markers = _sf_parse_think, @@ -7937,8 +8000,8 @@ class _ResponsesReasoningExtractor: reasoning_prefilled: bool = False, ) -> None: self._buffer = "" - # ``reasoning_prefilled``: output begins inside an unclosed ```` (Qwen3/GLM prefill), - # so start in reasoning to capture leading text until the first ````. + # ``reasoning_prefilled``: output begins INSIDE an unclosed ```` (Qwen3/GLM prefill), + # so start in reasoning to capture leading text until the first ````. Callers default False. self._in_reasoning = reasoning_prefilled # Splitting requires marker parsing; a prefilled open implies it. self._parse_think_markers = parse_think_markers or reasoning_prefilled @@ -7970,7 +8033,7 @@ class _ResponsesReasoningExtractor: self._buffer = self._buffer[close_idx + len(_RESPONSES_THINK_CLOSE) :] self._in_reasoning = False continue - # Hold back a trailing partial of either marker: the close (clean chunk-boundary split) + # Hold back a trailing partial of EITHER marker: the close (clean chunk-boundary split) # and a stray open (so a re-emitted ```` isn't leaked into the reasoning drawer). keep = _responses_marker_holdback( self._buffer, (_RESPONSES_THINK_CLOSE, _RESPONSES_THINK_OPEN) @@ -9859,7 +9922,9 @@ async def anthropic_messages( # Strip stale tool-call XML from conversation for _msg in openai_messages: if _msg.get("role") == "assistant" and isinstance(_msg.get("content"), str): - _msg["content"] = _strip_tool_xml(_msg["content"]).strip() + _msg["content"] = _strip_tool_xml( + _msg["content"], _gemma_strip_gate(openai_tools) + ).strip() def _run_tool_gen(): return llama_backend.generate_chat_completion_with_tools( @@ -9904,6 +9969,7 @@ async def anthropic_messages( message_id, model_name, disable_parallel_tool_use = _disable_parallel, + openai_tools = openai_tools, ) ) @@ -10010,7 +10076,7 @@ async def _anthropic_tool_stream( # content event that was purely tool XML doesn't count as text. if etype == "content": event = dict(event) - event["text"] = _strip_tool_xml(event["text"]) + event["text"] = _strip_tool_xml(event["text"], _gemma_strip_gate(openai_tools)) # disable_parallel_tool_use: keep only the first tool_use block, # dropping every later tool_start and its paired tool_end (robust # to empty tool-call ids — tracked by state, not id matching). @@ -10165,6 +10231,7 @@ async def _anthropic_tool_non_streaming( message_id, model_name, disable_parallel_tool_use = False, + openai_tools = None, ): """Non-streaming response for the tool-calling path. @@ -10193,7 +10260,7 @@ async def _anthropic_tool_non_streaming( etype = event.get("type", "") if etype == "content": # Strip leaked tool-call XML - clean = _strip_tool_xml(event["text"]) + clean = _strip_tool_xml(event["text"], _gemma_strip_gate(openai_tools)) new = clean[len(prev_text) :] prev_text = clean if new: @@ -10662,11 +10729,14 @@ async def _anthropic_passthrough_non_streaming( else: text = message.get("content") or "" if text: - # Keep unpromoted bytes when healing is active; legacy stripping is only for opted-out - # or no-client-tool requests. _strip_tool_xml also cleans Mistral [TOOL_CALLS] and - # guarded function-XML, not just _TOOL_XML_RE. + # Keep unpromoted bytes when healing is active; legacy stripping is + # only for opted-out or no-client-tool requests. Use the full + # _strip_tool_xml pass so Mistral [TOOL_CALLS] and guarded + # function-XML leaks are cleaned too, not just _TOOL_XML_RE forms, + # with the Gemma display gate so a disabled/example call:NAME{...} + # in prose survives. if not healing_active: - text = _strip_tool_xml(text) + text = _strip_tool_xml(text, _gemma_strip_gate(openai_tools)) text = text.strip() if text: content_blocks.append(AnthropicResponseTextBlock(text = text)) diff --git a/studio/backend/tests/test_gemma_tool_parse_edge_cases.py b/studio/backend/tests/test_gemma_tool_parse_edge_cases.py index fff6b240c..7b653f47a 100644 --- a/studio/backend/tests/test_gemma_tool_parse_edge_cases.py +++ b/studio/backend/tests/test_gemma_tool_parse_edge_cases.py @@ -41,7 +41,7 @@ def test_normal_multi_key_arguments_still_split(): def test_empty_bare_value_becomes_empty_string_not_dropped(): - # An empty bare value (``{query:}``) must serialise as ``""`` (``{"query":}`` is invalid JSON). + # An empty bare value (``{query:}``) must serialise as ``""`` (``{"query":}`` is invalid JSON and dropped the call). calls = parse_tool_calls_from_text("<|tool_call>call:search{query:,unit:celsius}") assert len(calls) == 1, calls assert _args(calls[0]) == {"query": "", "unit": "celsius"} @@ -60,6 +60,15 @@ def test_bare_value_with_timestamps_after_comma_is_kept(): assert _args(calls[0]) == {"query": "meet at 10:00, 11:00 tomorrow", "priority": "high"} +def test_wrapperless_bare_value_with_timestamps_after_comma_is_kept(): + # The wrapper-less Gemma form (no <|tool_call> markers) goes through the + # _gemma_parse_stripped_body scanner and its _GEMMA_KEY_RE. + calls = parse_tool_calls_from_text("call:web_search{query:meet at 10:00, 11:00 tomorrow}") + assert len(calls) == 1, calls + assert calls[0]["function"]["name"] == "web_search" + assert _args(calls[0]) == {"query": "meet at 10:00, 11:00 tomorrow"} + + def test_marker_inside_json_argument_is_not_a_second_call(): content = ( '{"name":"python","arguments":{"code":' @@ -97,8 +106,8 @@ def test_json_marker_inside_gemma_argument_is_not_a_second_call(): def test_nested_gemma_marker_in_unquoted_arg_does_not_run_inner_call(): - # The outer object fails to normalize, but the nested marker is covered by - # its span; safe outcome is no executed call at all. + # An UNQUOTED Gemma value containing a literal marker: the marker is nested in the outer + # candidate span, so it must not be promoted to a standalone `terminal` call (no tool call). content = "<|tool_call>call:python{code:<|tool_call>call:terminal{command:ls}}" calls = parse_tool_calls_from_text(content) assert "terminal" not in [c["function"]["name"] for c in calls], calls @@ -151,6 +160,43 @@ def test_json_marker_inside_xml_parameter_is_not_a_second_call(): assert [c["function"]["name"] for c in calls] == ["python"], calls +def test_wrapperless_nested_object_argument_is_parsed(): + # skip_special_tokens stream: wrapper and <|"|> markers stripped, so a nested object arrives bare. + calls = parse_tool_calls_from_text("call:f{loc:{city:NYC},n:3}") + assert len(calls) == 1 + assert _args(calls[0]) == {"loc": {"city": "NYC"}, "n": 3} + + +def test_wrapperless_array_argument_is_parsed(): + calls = parse_tool_calls_from_text("call:label{labels:[bug,ui],n:2}") + assert len(calls) == 1 + assert _args(calls[0]) == {"labels": ["bug", "ui"], "n": 2} + + +def test_wrapperless_deeply_nested_object_and_array_are_preserved(): + # The single-pass parser must keep multi-level nesting (objects inside + # objects, arrays inside arrays) intact, not flatten or drop it. + calls = parse_tool_calls_from_text( + "call:f{loc:{city:NYC,geo:{lat:1,lng:2}},tags:[a,b,[c,d]],n:3}" + ) + assert len(calls) == 1 + assert _args(calls[0]) == { + "loc": {"city": "NYC", "geo": {"lat": 1, "lng": 2}}, + "tags": ["a", "b", ["c", "d"]], + "n": 3, + } + + +def test_gemma_parse_array_advances_on_stray_brace(): + # Regression: a stray '}' / ']' / ',' where an array element is expected must + # not stall _gemma_parse_value at the same index (it looped forever before). + from core.inference.tool_call_parser import _gemma_parse_array + + items, end, closed = _gemma_parse_array("[a,}]", 0) + assert end == 5 and closed is True # consumed through the closing ']' + assert items[0] == "a" + + def test_gemma_close_marker_inside_quoted_arg_is_not_leaked_when_stripping(): # Parse keeps the quoted close marker as data; strip removes the whole span. text = '<|tool_call>call:python{code:<|"|>print("")<|"|>}' @@ -312,16 +358,17 @@ def test_valid_call_after_close_less_marker_with_quoted_close_token_is_recovered def test_gemma_parse_value_always_advances_on_stray_delimiter(): # A stray delimiter (`,`, `}`, `]`) at the primitive position must still advance the - # parser, or a looping caller spins forever (DoS). + # index by at least one, or a caller looping on it spins forever at 100% CPU (DoS). for delim in (",", "}", "]"): text = delim + "rest" - value, nxt = _gemma_parse_value(text, 0) + value, nxt, _explicit = _gemma_parse_value(text, 0) assert nxt > 0, (delim, value, nxt) def test_malformed_gemma_array_does_not_hang(): - # ``[},]`` (stray ``}`` in a list body) hung the buggy parser; the timeout fails - # the regression loudly instead of blocking CI forever. + # ``[},]`` puts a stray ``}`` at the primitive position inside a list body. + # On the buggy parser this hangs the server; guard with a wall-clock timeout + # so the regression fails loudly instead of blocking CI forever. import threading result: dict = {} diff --git a/studio/backend/tests/test_llama_cpp_tool_loop.py b/studio/backend/tests/test_llama_cpp_tool_loop.py index 8977d6e92..dcc759a21 100644 --- a/studio/backend/tests/test_llama_cpp_tool_loop.py +++ b/studio/backend/tests/test_llama_cpp_tool_loop.py @@ -1040,7 +1040,7 @@ def test_render_html_success_does_not_reprompt_render_html_intent(monkeypatch): def test_internal_reprompt_attempts_do_not_duplicate_visible_text(monkeypatch): """No-tool re-prompt attempts should not concatenate into the UI.""" - # One initial response plus one stream per re-prompt (count from the shared cap). + # One initial response plus one stream per re-prompt; derive the count from the shared cap. streams = [[_sse({"content": "I will use render_html now."}), _done()]] streams += [ [_sse({"content": "Understood. I will use render_html now."}), _done()] @@ -1207,8 +1207,8 @@ def test_auto_heal_disabled_parses_well_formed_xml_when_tools_enabled(monkeypatc def test_textual_mistral_marker_not_leaked_when_inline_with_preface(monkeypatch): - # Inline Mistral ``[TOOL_CALLS]`` after a visible preface: the DRAINING flush must use the - # shared parser patterns (the legacy set leaked the marker to clients). + # Textual Mistral ``[TOOL_CALLS]`` inline with visible preface: the DRAINING flush must use the + # shared parser patterns (which know ``[TOOL_CALLS]``); the legacy set leaked the marker to clients. streams = [ [_sse({"content": 'Let me search. [TOOL_CALLS]web_search{"query":"cats"}'}), _done()], [_sse({"content": "done"}), _done()], @@ -1836,6 +1836,7 @@ def test_bare_json_tool_call_streamed_is_not_leaked_and_executes(monkeypatch): ) ) + # The tool ran with the parsed arguments. assert calls == [("web_search", {"query": "weather in Sydney"})] assert any( event.get("type") == "tool_end" and event.get("tool_name") == "web_search" @@ -1903,6 +1904,37 @@ def test_incomplete_bare_json_truncation_is_not_leaked(monkeypatch): assert all('{"name"' not in t for t in content_texts), content_texts +def test_gguf_truncated_ordinary_json_with_name_key_is_shown_not_suppressed(monkeypatch): + """A truncated markerless object whose "name" is NOT an enabled tool (a person + record cut off mid-stream, ``{"name":"Alice","age":``) must still be shown. The + end-of-stream ``_is_bare_tc`` heuristic routed any ``{...,"name",...}`` fragment + to DRAINING (dropped); it is now gated on the enabled tool names so only a real + truncated tool call is suppressed, ordinary JSON streams through.""" + + truncated = '{"name": "Alice", "age": 30, "bio": "loves ' + stream = _streamed_content(truncated) + payloads: list[dict] = [] + backend = _make_backend(monkeypatch, [stream], payloads) + + calls: list[tuple[str, dict]] = [] + monkeypatch.setattr( + "core.inference.tools.execute_tool", + lambda n, a, **_k: (calls.append((n, a)) or "x"), + ) + + events = list( + backend.generate_chat_completion_with_tools( + messages = [{"role": "user", "content": "start a person record"}], + tools = [{"type": "function", "function": {"name": "web_search"}}], + max_tool_iterations = 1, + ) + ) + + assert calls == [], calls + content_texts = [e.get("text", "") for e in events if e.get("type") == "content"] + assert any("Alice" in t for t in content_texts), content_texts + + def test_gguf_truncated_disabled_name_json_is_preserved_when_tools_active(monkeypatch): """A truncated JSON answer with a non-enabled name must still be shown (resolvers are gated on enabled names).""" @@ -1987,6 +2019,40 @@ def test_gguf_oversized_disabled_name_json_is_preserved(monkeypatch): assert any("Alice" in t for t in content_texts), content_texts[:1] +def test_gemma_wrapperless_call_streamed_is_not_leaked_and_executes(monkeypatch): + """Gemma 4 GGUF (skip_special_tokens) streams a wrapper-less ``call:NAME{..}`` + with no XML signal. Like bare JSON, the BUFFERING scan must recognise it via + _GEMMA_BARE_TC_RE, drain it silently, and execute the tool -- never leaking + the ``call:`` markup to the user-visible stream.""" + + gemma_call = 'call:web_search{query:"weather in Sydney"}' + first_stream = _streamed_content(gemma_call) + final_stream = [_sse({"content": "It is sunny in Sydney."}), _done()] + payloads: list[dict] = [] + backend = _make_backend(monkeypatch, [first_stream, final_stream], payloads) + + calls: list[tuple[str, dict]] = [] + + def fake_execute_tool(name, arguments, **_kwargs): + calls.append((name, arguments)) + return "Weather: sunny, 22C." + + monkeypatch.setattr("core.inference.tools.execute_tool", fake_execute_tool) + + events = list( + backend.generate_chat_completion_with_tools( + messages = [{"role": "user", "content": "weather in Sydney?"}], + tools = [{"type": "function", "function": {"name": "web_search"}}], + max_tool_iterations = 1, + ) + ) + + assert calls == [("web_search", {"query": "weather in Sydney"})] + content_texts = [e.get("text", "") for e in events if e.get("type") == "content"] + assert all("call:" not in t for t in content_texts), content_texts + assert any("sunny in Sydney" in t for t in content_texts), content_texts + + def _usage_done(usage: dict, finish_reason: str = "stop") -> str: """A terminal SSE chunk carrying llama-server's ``usage`` block, the way the real server reports it on the final chunk of a completion.""" @@ -2124,6 +2190,66 @@ def test_gguf_bare_json_call_not_replayed_in_next_turn_content(monkeypatch): assert asst and not any('"name"' in (m.get("content") or "") for m in asst), asst +def test_gguf_textual_fallback_caps_distinct_tool_calls_per_turn(monkeypatch): + """A single textual-fallback turn that parses many DISTINCT tool calls must be + capped at _MAX_TOOL_CALLS_PER_TURN (structured delta.tool_calls are grammar + bounded by llama-server; text parsed from content is not). Mirrors the + safetensors loop so one runaway turn cannot fan out into dozens of executions.""" + from core.inference.llama_cpp import _MAX_TOOL_CALLS_PER_TURN + + n = _MAX_TOOL_CALLS_PER_TURN + 4 + blocks = "".join( + '{"name":"t%d","arguments":{"i":%d}}' % (i, i) for i in range(n) + ) + first_stream = [_sse({"content": blocks}), _done()] + final_stream = [_sse({"content": "done"}), _done()] + payloads: list[dict] = [] + backend = _make_backend(monkeypatch, [first_stream, final_stream], payloads) + + calls: list[tuple[str, dict]] = [] + monkeypatch.setattr( + "core.inference.tools.execute_tool", + lambda name, arguments, **_k: (calls.append((name, arguments)) or "OK"), + ) + + list( + backend.generate_chat_completion_with_tools( + messages = [{"role": "user", "content": "go"}], + tools = [{"type": "function", "function": {"name": f"t{i}"}} for i in range(n)], + max_tool_iterations = 1, + ) + ) + + assert len(calls) == _MAX_TOOL_CALLS_PER_TURN, [c[0] for c in calls] + # The cap keeps the first calls in order (no reordering / drop of leading ones). + assert [c[0] for c in calls] == [f"t{i}" for i in range(_MAX_TOOL_CALLS_PER_TURN)] + + +def test_gguf_textual_fallback_collapses_duplicate_tool_calls(monkeypatch): + """Exact-duplicate textual calls in one turn collapse to a single execution.""" + blocks = '{"name":"web_search","arguments":{"query":"cats"}}' * 5 + first_stream = [_sse({"content": blocks}), _done()] + final_stream = [_sse({"content": "done"}), _done()] + payloads: list[dict] = [] + backend = _make_backend(monkeypatch, [first_stream, final_stream], payloads) + + calls: list[tuple[str, dict]] = [] + monkeypatch.setattr( + "core.inference.tools.execute_tool", + lambda name, arguments, **_k: (calls.append((name, arguments)) or "OK"), + ) + + list( + backend.generate_chat_completion_with_tools( + messages = [{"role": "user", "content": "cats"}], + tools = [{"type": "function", "function": {"name": "web_search"}}], + max_tool_iterations = 1, + ) + ) + + assert len(calls) == 1, [c[0] for c in calls] + + def test_gguf_drain_truncated_enabled_name_json_preserved_when_auto_heal_disabled(monkeypatch): """Auto-Heal OFF keeps a truncated enabled-name fragment visible; ON suppresses it (strip gated on auto_heal_tool_calls).""" @@ -2159,8 +2285,8 @@ def test_gguf_drain_truncated_enabled_name_json_preserved_when_auto_heal_disable def test_gguf_valid_tool_calls_respect_max_tool_iterations(monkeypatch): """Re-prompt slots must not extend the tool budget: stop after ``max_tool_iterations`` executed rounds.""" - # More tool-call streams than the budget: leaked re-prompt slots would run 2+3=5 rounds; - # honouring the budget stops after 2, then a tool-less final-answer pass. + # More tool-call streams than the budget: if re-prompt slots leaked into the budget (the bug) the + # loop would run 2+3=5 rounds; honouring it stops after 2, then a tool-less final-answer pass. streams = [ _structured_tool_call("web_search", {"query": f"q{i}"}, f"call_{i}") for i in range(6) ] diff --git a/studio/backend/tests/test_mcp_servers.py b/studio/backend/tests/test_mcp_servers.py index 12239e711..6d26d075c 100644 --- a/studio/backend/tests/test_mcp_servers.py +++ b/studio/backend/tests/test_mcp_servers.py @@ -587,10 +587,12 @@ def test_tool_xml_strip_handles_hyphenated_function_names(): import re as _re from pathlib import Path + from core.inference.tool_call_parser import _DEEPSEEK_OPEN_RE_SRC as _DS_OPEN_SRC + src = (Path(__file__).resolve().parent.parent / "routes/inference.py").read_text() m = _re.search(r"_TOOL_XML_RE = _re\.compile\((.*?)\n\)", src, _re.DOTALL) assert m, "could not extract _TOOL_XML_RE" - ns: dict = {"_re": _re} + ns: dict = {"_re": _re, "_DS_OPEN_SRC": _DS_OPEN_SRC} exec(f"_TOOL_XML_RE = _re.compile({m.group(1)})", ns) rx = ns["_TOOL_XML_RE"] stripped = rx.sub( diff --git a/studio/backend/tests/test_mlx_inference_backend.py b/studio/backend/tests/test_mlx_inference_backend.py index 9871965ce..ac4088fb2 100644 --- a/studio/backend/tests/test_mlx_inference_backend.py +++ b/studio/backend/tests/test_mlx_inference_backend.py @@ -100,6 +100,32 @@ def test_mlx_inference_text_load_forwards_studio_settings(monkeypatch): ] assert backend._is_vlm is False assert isinstance(backend._tokenizer, _DummyTokenizer) + # Non-LoRA text model: no base_model on the record. + assert backend.models["fake/text"]["base_model"] is None + + +def test_mlx_text_lora_record_keeps_base_model_for_native_template(monkeypatch): + # A LoRA adapter's own tokenizer often ships no chat template; the native tool-calling template + # lives on the base model. + _install_fake_mlx(monkeypatch) + calls = [] + _install_fake_fast_mlx(monkeypatch, calls) + + from core.inference.mlx_inference import MLXInferenceBackend + + backend = MLXInferenceBackend() + config = SimpleNamespace( + identifier = "fake/text-adapter", + is_vision = False, + is_lora = True, + base_model = "fake/text-base", + ) + + assert backend.load_model(config, max_seq_length = 4096, hf_token = "hf-token") + + record = backend.models["fake/text-adapter"] + assert record["is_lora"] is True + assert record["base_model"] == "fake/text-base" def test_mlx_inference_vlm_lora_uses_unsloth_loader_without_native_adapter_rewrite( @@ -188,12 +214,12 @@ def test_mlx_generate_text_forwards_kwargs_into_template_helper(monkeypatch): _install_fake_mlx(monkeypatch) from core.inference.mlx_inference import MLXInferenceBackend - captured = {} + # The text path renders once with tools, then the native-template fallback makes a second no- + # tools probe call (tools=None) to detect whether the template dropped the schema. + captured_calls = [] def _fake_apply(tokenizer, messages, **kwargs): - captured["tokenizer"] = tokenizer - captured["messages"] = messages - captured["kwargs"] = kwargs + captured_calls.append({"tokenizer": tokenizer, "messages": messages, "kwargs": kwargs}) return "" monkeypatch.setattr( @@ -248,8 +274,15 @@ def test_mlx_generate_text_forwards_kwargs_into_template_helper(monkeypatch): ) ) assert out == ["hi"] - # The toggled kwargs must reach the chat-template helper. - assert captured["kwargs"]["tools"] == [{"function": {"name": "web_search"}}] - assert captured["kwargs"]["enable_thinking"] is True - assert captured["kwargs"]["reasoning_effort"] == "medium" - assert captured["kwargs"]["preserve_thinking"] is True + # The toggled kwargs must reach the chat-template helper on the real render + # (one of the calls carries the tools; the fallback probe passes tools=None). + tool_renders = [ + c + for c in captured_calls + if c["kwargs"].get("tools") == [{"function": {"name": "web_search"}}] + ] + assert tool_renders, captured_calls + render = tool_renders[0] + assert render["kwargs"]["enable_thinking"] is True + assert render["kwargs"]["reasoning_effort"] == "medium" + assert render["kwargs"]["preserve_thinking"] is True diff --git a/studio/backend/tests/test_native_template_trust_remote_code.py b/studio/backend/tests/test_native_template_trust_remote_code.py new file mode 100644 index 000000000..60dc80f64 --- /dev/null +++ b/studio/backend/tests/test_native_template_trust_remote_code.py @@ -0,0 +1,176 @@ +# SPDX-License-Identifier: AGPL-3.0-only +# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0 + +"""Regression tests for trust_remote_code in the native-template fallback. + +``render_native_template`` re-fetches a model's native chat template from its +repo when an Unsloth override template (mistral, gemma-4) dropped the tools +schema. For a model loaded with ``trust_remote_code=True`` whose tokenizer repo +carries custom code, the secondary ``AutoTokenizer.from_pretrained`` must re-use +that same consent or transformers raises (it requires ``trust_remote_code`` to +instantiate a custom tokenizer class), the ``except`` swallows it, and the +request silently keeps the tool-dropping prompt even though the user already +consented to remote code for the model load. + +These tests pin that the stored ``trust_remote_code`` is threaded to the reload, +that the reload is skipped (returns ``None`` without executing code) when no +consent is stored, and that both backend ``model_info`` dicts persist the flag at +load time so the read lands on a value ``load_model`` actually set. +""" + +from __future__ import annotations + +import importlib.util +import sys +from pathlib import Path + +import pytest + + +_BACKEND_DIR = str(Path(__file__).resolve().parent.parent) +if _BACKEND_DIR not in sys.path: + sys.path.insert(0, _BACKEND_DIR) + +# ``chat_template_helpers`` is dependency-light (copy / logging / typing, with the +# transformers import deferred inside the function). Load it directly so the test +# runs without importing the heavy ``core.inference`` package (unsloth / torch). +_HELPERS_PATH = Path(_BACKEND_DIR) / "core" / "inference" / "chat_template_helpers.py" +_spec = importlib.util.spec_from_file_location("_native_tpl_trc_test", _HELPERS_PATH) +chat_template_helpers = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(chat_template_helpers) + +render_native_template = chat_template_helpers.render_native_template + + +# A native template that emits a tools section only when tools are provided, so the +# with-tools vs no-tools render differs and ``render_native_template`` accepts it. +_NATIVE_TEMPLATE = ( + "{% for m in messages %}{{ m['role'] }}: {{ m['content'] }}\n{% endfor %}" + "{% if tools %}[AVAILABLE_TOOLS]{{ tools }}[/AVAILABLE_TOOLS]\n{% endif %}" + "{% if add_generation_prompt %}assistant:{% endif %}" +) + +_MESSAGES = [{"role": "user", "content": "what is the weather"}] +_TOOLS = [{"type": "function", "function": {"name": "get_weather"}}] + + +class _JinjaTokenizer: + """Minimal tokenizer whose ``apply_chat_template`` renders ``self.chat_template``. + + Stands in for the live model tokenizer that ``render_native_template`` shallow- + copies and re-points at the native template before rendering. + """ + + def __init__(self, chat_template): + self.chat_template = chat_template + + def apply_chat_template( + self, + messages, + tokenize = False, + add_generation_prompt = True, + tools = None, + **kwargs, + ): + from jinja2 import BaseLoader, Environment + env = Environment(loader = BaseLoader()) + return env.from_string(self.chat_template).render( + messages = messages, + tools = tools, + add_generation_prompt = add_generation_prompt, + ) + + +def _install_custom_code_tokenizer(monkeypatch): + """Patch ``AutoTokenizer.from_pretrained`` to mimic a custom-code repo: raise + unless ``trust_remote_code`` is truthy, else return a tokenizer carrying the + native template. Records the ``trust_remote_code`` it was called with.""" + pytest.importorskip("jinja2") + from transformers import AutoTokenizer + + calls = {} + + def fake_from_pretrained( + model_id, + *args, + trust_remote_code = False, + token = None, + **kwargs, + ): + calls["trust_remote_code"] = trust_remote_code + calls["model_id"] = model_id + calls["token"] = token + if not trust_remote_code: + # Mirrors transformers.dynamic_module_utils.resolve_trust_remote_code: + # has_remote_code and not has_local_code and not trust_remote_code -> ValueError. + raise ValueError( + f"The repository {model_id} contains custom code which must be executed " + "to correctly load the model. Please pass the argument " + "`trust_remote_code=True` to allow custom code to be run." + ) + return _JinjaTokenizer(_NATIVE_TEMPLATE) + + monkeypatch.setattr(AutoTokenizer, "from_pretrained", staticmethod(fake_from_pretrained)) + return calls + + +def _model_info(trust_remote_code): + return { + "native_chat_template": None, # force the repo reload path + "base_model": None, # non-LoRA: template_source == active_model_name + "trust_remote_code": trust_remote_code, + # Live tokenizer that gets shallow-copied + re-pointed at the native template. + "tokenizer": _JinjaTokenizer("OVERRIDE-THAT-DROPS-TOOLS"), + } + + +def test_native_reload_passes_stored_trust_remote_code(monkeypatch): + """With ``trust_remote_code`` stored on ``model_info`` the custom-code reload + succeeds and the tools-advertising native prompt is returned. This FAILS before + the fix (reload omits the flag, raises, is swallowed, returns None).""" + calls = _install_custom_code_tokenizer(monkeypatch) + model_info = _model_info(trust_remote_code = True) + + out = render_native_template( + model_info = model_info, + active_model_name = "acme/custom-tokenizer-model", + messages = _MESSAGES, + tools = _TOOLS, + ) + + assert out is not None, "native fallback should render the tools prompt with consent" + assert "[AVAILABLE_TOOLS]" in out + assert "get_weather" in out + assert calls["trust_remote_code"] is True # the stored consent was threaded through + # A successful fetch is cached so the next tool turn skips the reload. + assert model_info["native_chat_template"] == _NATIVE_TEMPLATE + + +def test_native_reload_without_consent_returns_none(monkeypatch): + """Without stored consent the custom-code reload raises, is swallowed, and + ``render_native_template`` returns None (no unconsented code execution). Proves + the stored flag -- not a hard-coded True -- drives the reload.""" + calls = _install_custom_code_tokenizer(monkeypatch) + model_info = _model_info(trust_remote_code = False) + + out = render_native_template( + model_info = model_info, + active_model_name = "acme/custom-tokenizer-model", + messages = _MESSAGES, + tools = _TOOLS, + ) + + assert out is None + assert calls["trust_remote_code"] is False + # A failed fetch must not be cached as "no template" (would pin the tool drop). + assert model_info["native_chat_template"] is None + + +def test_backend_model_info_persists_trust_remote_code(): + """Both backends must store ``trust_remote_code`` on their per-model info dict so + ``render_native_template`` can source the consent value. Guards against the read + landing on a key ``load_model`` never sets (which would silently no-op the fix).""" + inf = (Path(_BACKEND_DIR) / "core" / "inference" / "inference.py").read_text() + mlx = (Path(_BACKEND_DIR) / "core" / "inference" / "mlx_inference.py").read_text() + assert '"trust_remote_code": trust_remote_code,' in inf + assert '"trust_remote_code": trust_remote_code,' in mlx diff --git a/studio/backend/tests/test_pr5624_regressions.py b/studio/backend/tests/test_pr5624_regressions.py new file mode 100644 index 000000000..4f5471675 --- /dev/null +++ b/studio/backend/tests/test_pr5624_regressions.py @@ -0,0 +1,1011 @@ +# SPDX-License-Identifier: AGPL-3.0-only +# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0 + +""" +Regression tests for PR #5624 (DeepSeek R1/V3.x, GLM 4.x, Kimi K2 tool +parsing). Each test pins a specific edge case surfaced during the +review: + +* GLM string-vs-JSON-encoded value coercion (template emits strings + raw and non-strings JSON-encoded; the parser must not coerce a + bare string ``"42"`` into ``42``). +* GLM ```` containing a literal ``<`` (e.g. ``if x < 10``). +* Kimi K2 dotted name ``functions.my.tool:0`` keeps its full name + (``my.tool``) after stripping only the ``functions.`` prefix and + ``:idx`` suffix, while the full id is preserved on the call. +* Kimi K2 bare-counter id (no ``functions.`` prefix, no ``:IDX``) is + dropped rather than surfaced under a numeric name. +* DeepSeek V3.1 truncated mid-stream produces an empty result without + raising. +* ``routes.inference._strip_tool_xml`` strips the DeepSeek envelope and + the Kimi section markers added by this PR. +""" + +import json + +import pytest + +from core.inference.tool_call_parser import ( + parse_tool_calls_from_text, + strip_tool_markup, +) + + +# GLM string-vs-JSON-encoded value coercion (finding B in plan) + + +@pytest.mark.parametrize( + "raw_val, expected_python", + [ + # Bare numeric / bool / null shapes are still treated as JSON + # literals (ambiguous with strings; the template doesn't tell us). + ("42", 42), + ("true", True), + ("false", False), + ("null", None), + ("3.14", 3.14), + ("-7", -7), + ("1e3", 1000.0), + ], +) +def test_glm_numeric_and_bool_literals_are_json_decoded(raw_val, expected_python): + text = ( + "n\n" + f"v\n" + f"{raw_val}\n" + "" + ) + calls = parse_tool_calls_from_text(text) + assert len(calls) == 1 + args = json.loads(calls[0]["function"]["arguments"]) + assert args["v"] == expected_python + + +@pytest.mark.parametrize( + "raw_val", + [ + "hello world", # plain prose + "True", # Python literal, NOT JSON -- no longer eaten by ast.literal_eval + "None", # Python literal, NOT JSON -- no longer eaten by ast.literal_eval + "if x < 10: pass", # code with literal < (well, < not in arg_value here) + "{not valid json", # looks like an object but is malformed -- must stay raw + "[oops", # looks like an array but is malformed + ], +) +def test_glm_non_json_shapes_stay_raw(raw_val): + text = ( + "n\n" + f"v\n" + f"{raw_val}\n" + "" + ) + calls = parse_tool_calls_from_text(text) + assert len(calls) == 1 + args = json.loads(calls[0]["function"]["arguments"]) + assert args["v"] == raw_val + assert isinstance(args["v"], str) + + +def test_glm_json_object_arg_decoded(): + text = ( + "nest\n" + "opts\n" + '{"limit": 10}\n' + "" + ) + calls = parse_tool_calls_from_text(text) + args = json.loads(calls[0]["function"]["arguments"]) + assert args["opts"] == {"limit": 10} + + +def test_glm_json_array_arg_decoded(): + text = ( + "nest\n" + "ids\n" + "[1, 2, 3]\n" + "" + ) + calls = parse_tool_calls_from_text(text) + args = json.loads(calls[0]["function"]["arguments"]) + assert args["ids"] == [1, 2, 3] + + +def test_glm_arg_value_with_literal_less_than(): + text = ( + "run\n" + "code\n" + "if x < 10: pass\n" + "" + ) + calls = parse_tool_calls_from_text(text) + assert len(calls) == 1 + args = json.loads(calls[0]["function"]["arguments"]) + assert args["code"] == "if x < 10: pass" + + +# GLM 4.7 no-newline emission shape + + +def test_glm_4_7_no_newlines_between_name_and_arg_key(): + """GLM 4.7 strips the ``\\n`` after the name (``{{- ... -}}`` in the + template) so ```` follows directly. Parser must accept both.""" + text = ( + "get_weather" + "cityLondon" + "unitscelsius" + "" + ) + calls = parse_tool_calls_from_text(text) + assert len(calls) == 1 + assert calls[0]["function"]["name"] == "get_weather" + args = json.loads(calls[0]["function"]["arguments"]) + assert args == {"city": "London", "units": "celsius"} + + +def test_glm_4_7_no_newlines_multi_call(): + """Back-to-back GLM 4.7 calls without intervening newlines.""" + text = ( + "ax1" + "by2" + ) + calls = parse_tool_calls_from_text(text) + assert len(calls) == 2 + assert calls[0]["function"]["name"] == "a" + assert calls[1]["function"]["name"] == "b" + + +def test_glm_4_7_does_not_break_qwen_path(): + """Qwen ``{json}`` still dispatches to Qwen; GLM's + first-char ``[^\\n<{]`` excludes ``{``.""" + text = '{"name":"web_search","arguments":{"q":"x"}}' + calls = parse_tool_calls_from_text(text) + assert len(calls) == 1 + assert calls[0]["function"]["name"] == "web_search" + + +# Kimi K2 dotted name + bare counter (finding C in plan) + + +def test_kimi_dotted_namespace_keeps_full_dotted_name(): + # A dotted Kimi id keeps its FULL name; only the ``functions.`` prefix and ``:idx`` suffix drop (vLLM parity). + text = ( + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>functions.my.tool:0" + "<|tool_call_argument_begin|>{}" + "<|tool_call_end|>" + "<|tool_calls_section_end|>" + ) + calls = parse_tool_calls_from_text(text) + assert len(calls) == 1 + assert calls[0]["function"]["name"] == "my.tool" + assert calls[0]["id"] == "functions.my.tool:0" + + +def test_kimi_two_sections_in_one_stream_both_parse(): + """Outer loop walks every ``<|tool_calls_section_begin|>...end|>`` + so vLLM / SGLang parity holds even on multi-section streams.""" + text = ( + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>functions.a:0" + '<|tool_call_argument_begin|>{"x":1}' + "<|tool_call_end|>" + "<|tool_calls_section_end|>" + " some prose between sections " + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>functions.b:0" + '<|tool_call_argument_begin|>{"y":2}' + "<|tool_call_end|>" + "<|tool_calls_section_end|>" + ) + calls = parse_tool_calls_from_text(text) + assert len(calls) == 2 + assert calls[0]["function"]["name"] == "a" + assert calls[1]["function"]["name"] == "b" + assert calls[0]["id"] == "functions.a:0" + assert calls[1]["id"] == "functions.b:0" + + +def test_kimi_bare_counter_id_is_dropped(): + """Bare-digit id (``3``) is dropped (matches vLLM); SGLang infers + name from schema, which we don't have at parse time.""" + text = ( + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>3" + "<|tool_call_argument_begin|>{}" + "<|tool_call_end|>" + "<|tool_calls_section_end|>" + ) + calls = parse_tool_calls_from_text(text) + assert calls == [] + + +# DeepSeek truncated mid-stream + + +def test_deepseek_v3_1_huge_truncated_body_is_linear(): + """Adversarial input: DeepSeek envelope with no JSON brace and a + 50k-char body. A regex-based ``[^\\n<]+?`` name capture is O(N^2) + here; the parser uses ``str.find`` on the sep marker so it stays + linear. Budget 1s to flag any future regression.""" + import time as _time + + text = "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>fn<｜tool▁sep｜>" + "x" * 50_000 + start = _time.time() + calls = parse_tool_calls_from_text(text) + elapsed = _time.time() - start + assert elapsed < 1.0, f"V3 path is non-linear: {elapsed:.2f}s" + assert calls == [] + + +def test_deepseek_r1_huge_fenceless_body_is_linear(): + """R1 detection used a greedy ``([^\\n]+)\\n```json`` regex that is O(N^2) on a + fence-less body of repeated ``function`` tokens. The parser now scans with + ``str.find``; budget 1s to flag any regression.""" + import time as _time + + text = "<｜tool▁calls▁begin｜>" + "function<｜tool▁sep｜>a" * 40_000 + start = _time.time() + calls = parse_tool_calls_from_text(text) + elapsed = _time.time() - start + assert elapsed < 1.0, f"R1 path is non-linear: {elapsed:.2f}s" + assert calls == [] + + +def test_glm_unclosed_body_many_arg_keys_is_linear(): + """An unclosed GLM ```` body runs to EOF; a lazy-group ``finditer`` + over many bare ```` tokens was O(N^2). The parser now walks pairs with + ``str.find``; budget 1s.""" + import time as _time + + text = "foo\n" + "k" * 40_000 + start = _time.time() + parse_tool_calls_from_text(text) + elapsed = _time.time() - start + assert elapsed < 1.0, f"GLM path is non-linear: {elapsed:.2f}s" + + +def test_deepseek_r1_fenced_json_parses(): + """R1 wraps args in a ```json fence after ``functionNAME``.""" + import json as _json + + text = ( + "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n" + "```json\n" + '{"city":"NYC","unit":"c"}\n' + "```<｜tool▁call▁end｜><｜tool▁calls▁end｜>" + ) + calls = parse_tool_calls_from_text(text) + assert len(calls) == 1 + assert calls[0]["function"]["name"] == "get_weather" + assert _json.loads(calls[0]["function"]["arguments"]) == {"city": "NYC", "unit": "c"} + + +def test_deepseek_v3_1_truncated_arguments_drops_call_without_crash(): + text = ( + "<｜tool▁calls▁begin｜>" + "<｜tool▁call▁begin｜>get_time" + "<｜tool▁sep｜>" + '{"city":"Tokyo"' # no closing brace, no end markers + ) + calls = parse_tool_calls_from_text(text) + assert calls == [] + + +def test_deepseek_v3_1_truncated_after_end_marker_still_yields_call(): + text = ( + "<｜tool▁calls▁begin｜>" "<｜tool▁call▁begin｜>get_time" "<｜tool▁sep｜>" '{"city":"Tokyo"}' + # neither <｜tool▁call▁end｜> nor <｜tool▁calls▁end｜> + ) + calls = parse_tool_calls_from_text(text) + assert len(calls) == 1 + assert calls[0]["function"]["name"] == "get_time" + assert json.loads(calls[0]["function"]["arguments"]) == {"city": "Tokyo"} + + +# Routes-layer strip across the three new families + + +def test_routes_layer_strip_removes_deepseek_envelope(): + from routes.inference import _strip_tool_xml as _routes_strip + + text = ( + "before " + "<｜tool▁calls▁begin｜>" + "<｜tool▁call▁begin｜>get_time" + '<｜tool▁sep｜>{"city":"Tokyo"}' + "<｜tool▁call▁end｜>" + "<｜tool▁calls▁end｜>" + " after" + ) + stripped = _routes_strip(text) + assert stripped == "before after" + + +def test_routes_layer_strip_removes_kimi_section(): + from routes.inference import _strip_tool_xml as _routes_strip + + text = ( + "before " + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>functions.web_search:0" + '<|tool_call_argument_begin|>{"q":"x"}' + "<|tool_call_end|>" + "<|tool_calls_section_end|>" + " after" + ) + stripped = _routes_strip(text) + assert stripped == "before after" + + +def test_routes_layer_strip_removes_glm_block(): + """``.*?`` covers GLM via the Qwen pattern.""" + from routes.inference import _strip_tool_xml as _routes_strip + + text = ( + "before " + "web_search\n" + "q\nx\n" + "" + " after" + ) + stripped = _routes_strip(text) + assert stripped == "before after" + + +# strip_tool_markup (parser-level finalise path) over the new families + + +def test_strip_tool_markup_handles_deepseek_envelope(): + text = ( + "before " + "<｜tool▁calls▁begin｜>" + "<｜tool▁call▁begin｜>get_time" + '<｜tool▁sep｜>{"city":"Tokyo"}' + "<｜tool▁call▁end｜>" + "<｜tool▁calls▁end｜>" + " after" + ) + stripped = strip_tool_markup(text, final = True) + assert "before" in stripped and "after" in stripped + assert "｜tool▁" not in stripped + assert "get_time" not in stripped and "Tokyo" not in stripped + + +def test_strip_tool_markup_handles_kimi_section(): + text = ( + "before " + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>functions.web_search:0" + '<|tool_call_argument_begin|>{"q":"x"}' + "<|tool_call_end|>" + "<|tool_calls_section_end|>" + " after" + ) + stripped = strip_tool_markup(text, final = True) + assert "before" in stripped and "after" in stripped + assert "tool_calls_section_begin" not in stripped + + +# Round-2 review findings: GLM quoted-string / unclosed-arg, DeepSeek +# strict terminator, nested wrapper-less Gemma strip + + +def test_glm_quoted_string_arg_keeps_its_quotes(): + # A GLM string value emitted verbatim that itself begins with a quote. + text = ( + "web_search\n" + "query\n" + '"exact phrase"\n' + "" + ) + calls = parse_tool_calls_from_text(text) + args = json.loads(calls[0]["function"]["arguments"]) + assert args["query"] == '"exact phrase"' + + +def test_glm_unclosed_arg_value_is_rejected_in_strict_mode(): + # Closing present but a value never closes: strict mode must reject + # the whole call rather than execute it with the argument silently dropped. + text = ( + "web_search\n" + "query\n" + "Tokyo weather" # no + "" + ) + assert parse_tool_calls_from_text(text, allow_incomplete = False) == [] + # With Auto-Heal the partial value is kept, not dropped to a no-arg call. + healed = parse_tool_calls_from_text(text, allow_incomplete = True) + assert len(healed) == 1 + args = json.loads(healed[0]["function"]["arguments"]) + assert "Tokyo weather" in args.get("query", "") + + +def test_deepseek_v3_missing_call_terminator_rejected_in_strict_mode(): + # Envelope closes but the per-call <｜tool▁call▁end｜> is absent. Strict mode + # must reject (it is truncated/merged); Auto-Heal still parses it. + text = ( + "<｜tool▁calls▁begin｜>" + "<｜tool▁call▁begin｜>get_time" + '<｜tool▁sep｜>{"city":"Tokyo"}' + "<｜tool▁calls▁end｜>" # envelope end only, no per-call end + ) + assert parse_tool_calls_from_text(text, allow_incomplete = False) == [] + healed = parse_tool_calls_from_text(text, allow_incomplete = True) + assert len(healed) == 1 + assert healed[0]["function"]["name"] == "get_time" + + +def test_deepseek_v3_with_call_terminator_parses_in_strict_mode(): + # Sanity: a well-formed V3 call (with the per-call end marker) still parses + # under strict mode after the terminator check. + text = ( + "<｜tool▁calls▁begin｜>" + "<｜tool▁call▁begin｜>get_time" + '<｜tool▁sep｜>{"city":"Tokyo"}' + "<｜tool▁call▁end｜>" + "<｜tool▁calls▁end｜>" + ) + calls = parse_tool_calls_from_text(text, allow_incomplete = False) + assert len(calls) == 1 + assert calls[0]["function"]["name"] == "get_time" + + +def test_strip_tool_markup_removes_nested_wrapperless_gemma_call(): + # Wrapper-less Gemma call with a NESTED object arg: the balanced helper must strip the whole call, not leave a trailing ``}``. + text = "answer: call:f{loc:{city:NYC},n:3} done" + stripped = strip_tool_markup(text, final = True) + assert "call:f" not in stripped + assert "}" not in stripped + assert "answer:" in stripped and "done" in stripped + + +# Pass-3 review findings: bare-Kimi streaming (non-final) strip symmetry +# and the wrapper-less Gemma route-display strip + + +def test_strip_tool_markup_non_final_removes_bare_kimi_call(): + # A bare ``<|tool_call_begin|>...<|tool_call_end|>`` (no section wrapper): the CLOSED (final=False) strip must remove it too. + text = ( + "before " + "<|tool_call_begin|>functions.web_search:0" + '<|tool_call_argument_begin|>{"q":"x"}' + "<|tool_call_end|>" + " after" + ) + stripped = strip_tool_markup(text, final = False) + assert "tool_call_begin" not in stripped + assert "tool_call_end" not in stripped + assert "before" in stripped and "after" in stripped + + +def test_routes_layer_strip_removes_wrapperless_gemma_call(): + # Gemma 4 (skip_special_tokens) emits a wrapper-less ``call:NAME{..}`` with no XML markers. + from routes.inference import _strip_tool_xml as _routes_strip + + text = 'before call:web_search{query:"weather in Sydney"} after' + stripped = _routes_strip(text) + assert "call:web_search" not in stripped + assert "before" in stripped and "after" in stripped + + +def test_deepseek_envelope_end_inside_arg_string_is_not_a_truncation(): + # A DeepSeek V3.1 call whose argument string contains the literal envelope-end token must not be dropped. + content = ( + "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>web_search<｜tool▁sep｜>" + '{"query":"what does <｜tool▁calls▁end｜> mean"}' + "<｜tool▁call▁end｜><｜tool▁calls▁end｜>" + ) + calls = parse_tool_calls_from_text(content) + assert len(calls) == 1, calls + assert calls[0]["function"]["name"] == "web_search" + assert json.loads(calls[0]["function"]["arguments"]) == { + "query": "what does <｜tool▁calls▁end｜> mean" + } + + +def test_glm_value_containing_literal_arg_value_close_is_preserved(): + # A GLM string argument may legitimately contain . + content = ( + "runcode" + 'print("")' + ) + calls = parse_tool_calls_from_text(content) + assert len(calls) == 1, calls + assert json.loads(calls[0]["function"]["arguments"]) == {"code": 'print("")'} + + +def test_attribute_form_function_with_embedded_marker_runs_outer_call(): + # is a supported envelope; a DeepSeek/Kimi marker inside one of its + # parameter values is data, not a second call. + content = ( + '' + "The Kimi format is <|tool_call_begin|>functions.delete_all:0" + "<|tool_call_argument_begin|>{}<|tool_call_end|>" + "" + ) + calls = parse_tool_calls_from_text(content) + assert [c["function"]["name"] for c in calls] == ["respond"], calls + + +def test_wrapperless_gemma_call_gated_by_enabled_tools(): + # Once skip_special_tokens removes the <|tool_call> wrapper, call:NAME{...} is + # indistinguishable from prose documenting the Gemma syntax. + prose = "Here is an example of the syntax: call:foo{x:1}. That shows how tools work." + assert parse_tool_calls_from_text(prose, enabled_tool_names = {"web_search"}) == [] + # The display strip is gated the same way, so the example survives in the answer. + assert "call:foo{x:1}" in strip_tool_markup( + prose, final = True, enabled_tool_names = {"web_search"} + ) + # An enabled name is still a real call (parsed, and stripped from display). + real = "Answer. call:web_search{query:hi}" + calls = parse_tool_calls_from_text(real, enabled_tool_names = {"web_search"}) + assert [c["function"]["name"] for c in calls] == ["web_search"], calls + assert "call:web_search" not in strip_tool_markup( + real, final = True, enabled_tool_names = {"web_search"} + ) + + +def test_kimi_section_end_inside_arg_string_is_not_a_truncation(): + # In a multi-call Kimi section, a later call whose argument holds the literal section-end token must not truncate the section. + content = ( + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>functions.search:0<|tool_call_argument_begin|>" + '{"q":"cats"}<|tool_call_end|>' + "<|tool_call_begin|>functions.explain:1<|tool_call_argument_begin|>" + '{"text":"the token <|tool_calls_section_end|> means end"}<|tool_call_end|>' + "<|tool_calls_section_end|>" + ) + calls = parse_tool_calls_from_text(content) + assert [c["function"]["name"] for c in calls] == ["search", "explain"], calls + assert json.loads(calls[1]["function"]["arguments"]) == { + "text": "the token <|tool_calls_section_end|> means end" + } + + +def test_closed_envelope_before_deepseek_block_owns_turn(): + # Document order is the contract: a CLOSED / call that precedes a + # DeepSeek/Kimi block owns the turn, even when prose frames it as an example. + deepseek = ( + "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>search_web\n" + "```json\n" + '{"query":"weather in Paris"}\n' + "```" + "<｜tool▁call▁end｜><｜tool▁calls▁end｜>" + ) + prose = ( + 'A Qwen call looks like {"name":"example_tool","arguments":{}}.\n' + ) + calls = parse_tool_calls_from_text(prose + deepseek) + assert [c["function"]["name"] for c in calls] == ["example_tool"], calls + + kimi = ( + "<|tool_calls_section_begin|><|tool_call_begin|>functions.lookup:0" + '<|tool_call_argument_begin|>{"id":7}<|tool_call_end|><|tool_calls_section_end|>' + ) + calls_k = parse_tool_calls_from_text("Example: {} and now:\n" + kimi) + assert [c["function"]["name"] for c in calls_k] == ["demo"], calls_k + + +def test_marker_inside_closed_outer_envelope_still_runs_outer_call(): + # The guard must fire when the marker sits INSIDE a closed outer / envelope's arguments: the OUTER call wins. + outer = ( + "what does <｜tool▁calls▁begin｜> mean" + ) + calls = parse_tool_calls_from_text(outer) + # The outer envelope is the real call; the embedded DeepSeek marker must not + # hijack the parse into a spurious tool. + assert [c["function"]["name"] for c in calls] == ["lookup"], calls + assert json.loads(calls[0]["function"]["arguments"]) == { + "q": "what does <｜tool▁calls▁begin｜> mean" + } + + +def test_truncated_outer_envelope_with_embedded_marker_heals_outer_call(): + # A TRUNCATED outer call embedding a DeepSeek/Kimi marker in its argument still Auto-Heals as the outer call. + trunc = 'x = "<｜tool▁calls▁begin｜>sample"' + calls = parse_tool_calls_from_text(trunc) + assert [c["function"]["name"] for c in calls] == ["python"], calls + + +def test_python_tag_call_with_embedded_marker_runs_outer_call(): + # ``<|python_tag|>`` is Llama-3's tool-call envelope, so a DeepSeek/Kimi example quoted + # in its argument is data: the OUTER python_tag call (``web_search``) must run, not the + # embedded marker (``delete_all``). + kimi = ( + "<|tool_calls_section_begin|><|tool_call_begin|>functions.delete_all:0" + "<|tool_call_argument_begin|>{}<|tool_call_end|><|tool_calls_section_end|>" + ) + deepseek = ( + "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>delete_all<｜tool▁sep｜>{}" + "<｜tool▁call▁end｜><｜tool▁calls▁end｜>" + ) + for embedded in (kimi, deepseek): + builtin = '<|python_tag|>web_search.call(query="explain ' + embedded + '")' + calls = parse_tool_calls_from_text(builtin, enabled_tool_names = {"web_search"}) + assert [c["function"]["name"] for c in calls] == ["web_search"], calls + custom = ( + '<|python_tag|>{"name":"web_search","parameters":' + '{"query":"explain ' + embedded + '"}}' + ) + calls = parse_tool_calls_from_text(custom, enabled_tool_names = {"web_search"}) + assert [c["function"]["name"] for c in calls] == ["web_search"], calls + + # A bare ``<|python_tag|>`` prose mention (no call shape) must NOT be treated as an + # envelope: a real Kimi call after it still parses (the call-shaped lookahead guard). + prose = "The token <|python_tag|> is used. " + kimi + calls = parse_tool_calls_from_text(prose) + assert [c["function"]["name"] for c in calls] == ["delete_all"], calls + + +def test_gemma_wrapperless_quoted_value_with_comma_not_split(): + # A wrapper-less Gemma call whose quoted value contains ``, key:``. + text = 'call:web_search{query:"weather, location: Boston", limit:3}' + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search"}) + assert [c["function"]["name"] for c in calls] == ["web_search"], calls + assert json.loads(calls[0]["function"]["arguments"]) == { + "query": "weather, location: Boston", + "limit": 3, + } + + +def test_literal_close_tag_in_xml_arg_before_marker_runs_outer_call(): + # A literal ```` inside an outer XML argument (before a marker) is not the envelope close: the span reaches the REAL final close. + text = ( + 'x = " ' + "<|tool_call_begin|>functions.delete_all:0<|tool_call_argument_begin|>{}" + '<|tool_call_end|>"' + ) + calls = parse_tool_calls_from_text(text) + assert [c["function"]["name"] for c in calls] == ["python"], calls + + +def test_literal_tool_call_close_in_qwen_json_before_marker_runs_outer_call(): + # A Qwen/Hermes whose JSON argument holds a literal then a marker must run the OUTER call. + text = ( + '{"name":"search","arguments":{"query":"explain then ' + "<|tool_call_begin|>functions.delete_all:0<|tool_call_argument_begin|>{}" + '<|tool_call_end|>"}}' + ) + calls = parse_tool_calls_from_text(text) + assert [c["function"]["name"] for c in calls] == ["search"], calls + # Back-to-back Qwen calls still parse independently (real-close span must keep the + # negative-lookahead that separates adjacent calls). + bb = ( + '{"name":"a","arguments":{}}' + '{"name":"b","arguments":{}}' + ) + assert [c["function"]["name"] for c in parse_tool_calls_from_text(bb)] == ["a", "b"] + + +def test_r1_heal_keeps_later_call_when_first_omits_close_fence(): + # DeepSeek R1 multi-call where the FIRST call has balanced JSON but omits its close + # fence/terminator, followed by a well-formed second call. + text = ( + "<｜tool▁calls▁begin｜>" + "<｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n```json\n" + '{"city":"SF"}\n```' # no <｜tool▁call▁end｜> + "<｜tool▁call▁begin｜>function<｜tool▁sep｜>get_time\n```json\n" + '{"tz":"UTC"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>' + ) + heal = [c["function"]["name"] for c in parse_tool_calls_from_text(text)] + assert "get_time" in heal, heal + # Strict keeps the later well-formed call; heal must be a superset. + strict = [ + c["function"]["name"] for c in parse_tool_calls_from_text(text, allow_incomplete = False) + ] + assert set(strict) <= set(heal), (strict, heal) + + +def test_wrapperless_gemma_nested_call_in_arg_is_not_a_second_call(): + # A wrapper-less Gemma call whose quoted argument mentions another enabled tool must not execute that nested name. + text = 'call:web_search{query:"explain call:delete_all{target:files}"}' + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search", "delete_all"}) + assert [c["function"]["name"] for c in calls] == ["web_search"], calls + assert json.loads(calls[0]["function"]["arguments"]) == { + "query": "explain call:delete_all{target:files}" + } + # Two genuinely separate calls still both parse. + two = "call:web_search{query:hi}call:get_time{tz:UTC}" + assert [ + c["function"]["name"] + for c in parse_tool_calls_from_text(two, enabled_tool_names = {"web_search", "get_time"}) + ] == ["web_search", "get_time"] + + +def test_leading_bare_json_call_owns_quoted_gemma_snippet(): + # Document order: a leading Llama-3.2 bare-JSON call with trailing prose owns the turn. + text = ( + '{"name":"lookup","parameters":{"note":"use call:web_search{query:cats} for this"}}\n' + "That is the call I would make." + ) + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"lookup", "web_search"}) + assert [c["function"]["name"] for c in calls] == ["lookup"], calls + assert json.loads(calls[0]["function"]["arguments"]) == { + "note": "use call:web_search{query:cats} for this" + } + + # Same with the ``;`` inter-call separator: both real calls parse, the + # quoted snippet still does not. + two = ( + '{"name":"lookup","parameters":{"note":"see call:web_search{query:cats}"}};' + '{"name":"lookup","parameters":{"q":"second"}}' + ) + calls_two = parse_tool_calls_from_text(two, enabled_tool_names = {"lookup", "web_search"}) + assert [c["function"]["name"] for c in calls_two] == ["lookup", "lookup"], calls_two + + +def test_leading_gemma_call_still_wins_over_trailing_json_example(): + # Reverse control: a real leading Gemma call followed by a bare-JSON example keeps the Gemma call (bare JSON matches only a LEADING object). + text = 'call:web_search{query:cats} Example JSON: {"name":"demo_tool","parameters":{}}' + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search", "demo_tool"}) + assert [c["function"]["name"] for c in calls] == ["web_search"], calls + + # And prose-only enabled Gemma syntax (no leading JSON) still promotes: the + # markerless by-design behaviour is unchanged. + prose = "You can run call:web_search{query:cats} to search." + calls_p = parse_tool_calls_from_text(prose, enabled_tool_names = {"web_search"}) + assert [c["function"]["name"] for c in calls_p] == ["web_search"], calls_p + + +def test_leading_gemma_call_owns_quoted_mistral_trigger(): + # A leading wrapper-less Gemma call whose argument quotes a Mistral trigger must win: the [TOOL_CALLS] literal is data. + text = 'call:web_search{query:"docs say [TOOL_CALLS]delete_all{}"}' + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search", "delete_all"}) + assert [c["function"]["name"] for c in calls] == ["web_search"], calls + assert json.loads(calls[0]["function"]["arguments"]) == { + "query": "docs say [TOOL_CALLS]delete_all{}" + } + + # Reverse control: a real leading Mistral call still parses normally. + real = '[TOOL_CALLS]delete_all{"x":1}' + calls_m = parse_tool_calls_from_text(real, enabled_tool_names = {"web_search", "delete_all"}) + assert [c["function"]["name"] for c in calls_m] == ["delete_all"], calls_m + + # A DISABLED Gemma example quoting the trigger is dropped as prose and a + # real call after it still parses (drop-the-span recursion). + mixed = ( + 'Example: call:demo{note:"see [TOOL_CALLS]delete_all{}"}\n' + '[TOOL_CALLS]web_search{"q":"real"}' + ) + calls_d = parse_tool_calls_from_text(mixed, enabled_tool_names = {"web_search", "delete_all"}) + assert [c["function"]["name"] for c in calls_d] == ["web_search"], calls_d + + +def test_chained_bare_json_owns_kimi_marker_in_later_call(): + # Document order: two ;-chained bare-JSON calls own the turn even when the second's argument quotes a complete Kimi snippet. + kimi = ( + "<|tool_call_begin|>functions.delete_all:0" + "<|tool_call_argument_begin|>{}<|tool_call_end|>" + ) + two = ( + '{"name":"lookup","parameters":{"q":"first"}};' + '{"name":"lookup","parameters":{"note":"' + kimi + '"}}' + ) + calls = parse_tool_calls_from_text(two, enabled_tool_names = {"lookup", "delete_all"}) + assert [c["function"]["name"] for c in calls] == ["lookup", "lookup"], calls + + # Reverse control: prose followed by a real Kimi block still parses. + real = "Let me check.\n<|tool_calls_section_begin|>" + kimi + "<|tool_calls_section_end|>" + calls_k = parse_tool_calls_from_text(real, enabled_tool_names = {"lookup", "delete_all"}) + assert [c["function"]["name"] for c in calls_k] == ["delete_all"], calls_k + + # A closed leading Mistral call preceding a trailing Kimi example owns the + # turn too (same closed-call-precedes-marker rule). + mistral = '[TOOL_CALLS]lookup{"q":"first"} then example ' + kimi + calls_m = parse_tool_calls_from_text(mistral, enabled_tool_names = {"lookup", "delete_all"}) + assert [c["function"]["name"] for c in calls_m] == ["lookup"], calls_m + + +def test_nested_gemma_values_keep_commas_and_parens(): + # Nested wrapper-less Gemma mappings/arrays use the top-level delimiter rules, so nested arguments are not split. + calls = parse_tool_calls_from_text( + "call:python{opts:{code:print(1,2),lang:py}}", enabled_tool_names = {"python"} + ) + assert [c["function"]["name"] for c in calls] == ["python"], calls + assert json.loads(calls[0]["function"]["arguments"]) == { + "opts": {"code": "print(1,2)", "lang": "py"} + } + + arr = parse_tool_calls_from_text( + "call:python{opts:[1,2,{a:f(1,2)}]}", enabled_tool_names = {"python"} + ) + assert json.loads(arr[0]["function"]["arguments"]) == {"opts": [1, 2, {"a": "f(1,2)"}]} + + prose_comma = parse_tool_calls_from_text( + "call:python{opts:{note:hello, world}}", enabled_tool_names = {"python"} + ) + assert json.loads(prose_comma[0]["function"]["arguments"]) == {"opts": {"note": "hello, world"}} + + quoted = parse_tool_calls_from_text( + 'call:python{opts:{q:say "a, b" now,n:3}}', enabled_tool_names = {"python"} + ) + assert json.loads(quoted[0]["function"]["arguments"]) == { + "opts": {"q": 'say "a, b" now', "n": 3} + } + + # Controls: nested quoted values and multi-key mappings are unchanged, and + # a truncated nested value still falls back to the raw string. + nested_q = parse_tool_calls_from_text( + 'call:python{loc:{city:"New York"}}', enabled_tool_names = {"python"} + ) + assert json.loads(nested_q[0]["function"]["arguments"]) == {"loc": {"city": "New York"}} + multi = parse_tool_calls_from_text( + "call:python{opts:{a:1,b:2},n:3}", enabled_tool_names = {"python"} + ) + assert json.loads(multi[0]["function"]["arguments"]) == {"opts": {"a": 1, "b": 2}, "n": 3} + trunc = parse_tool_calls_from_text( + "call:python{opts:{code:print(1,2}}", enabled_tool_names = {"python"} + ) + assert json.loads(trunc[0]["function"]["arguments"]) == {"opts": "{code:print(1,2}"} + + +def test_multi_gemma_calls_own_turn_over_signal_in_later_call(): + # Document order: when the first enabled Gemma call closes before the first foreign signal, the leading call still owns the turn. + en = {"get_time", "web_search", "delete_all"} + both = parse_tool_calls_from_text( + 'call:get_time{} call:web_search{query:"docs say [TOOL_CALLS]delete_all{}"}', + enabled_tool_names = en, + ) + assert [c["function"]["name"] for c in both] == ["get_time", "web_search"], both + assert json.loads(both[1]["function"]["arguments"]) == { + "query": "docs say [TOOL_CALLS]delete_all{}" + } + + # XML and Kimi markers in the later call's strings stay data too. + xml = parse_tool_calls_from_text( + 'call:get_time{} call:web_search{query:"see delete_all"}', + enabled_tool_names = en, + ) + assert [c["function"]["name"] for c in xml] == ["get_time", "web_search"], xml + kimi = parse_tool_calls_from_text( + 'call:get_time{} call:web_search{query:"see <|tool_call_begin|>' + 'functions.delete_all:0<|tool_call_argument_begin|>{}<|tool_call_end|>"}', + enabled_tool_names = en, + ) + assert [c["function"]["name"] for c in kimi] == ["get_time", "web_search"], kimi + + # A trailing prose example after the closed leading call defers the same way. + prose = parse_tool_calls_from_text( + "call:get_time{} Example: [TOOL_CALLS]delete_all{}", enabled_tool_names = en + ) + assert [c["function"]["name"] for c in prose] == ["get_time"], prose + + +def test_multi_gemma_ownership_reverse_controls(): + # A real leading Mistral/XML call with a trailing Gemma example keeps the leading call; a signal before every Gemma call keeps normal order. + en = {"get_time", "web_search", "delete_all"} + mistral = parse_tool_calls_from_text( + '[TOOL_CALLS][{"name":"delete_all","arguments":{}}] Example: call:web_search{query:cats}', + enabled_tool_names = en, + ) + assert [c["function"]["name"] for c in mistral] == ["delete_all"], mistral + xml_first = parse_tool_calls_from_text( + '{"name":"delete_all","arguments":{}} call:web_search{query:cats}', + enabled_tool_names = en, + ) + assert [c["function"]["name"] for c in xml_first] == ["delete_all"], xml_first + agnostic = parse_tool_calls_from_text( + 'call:foo{} {"name":"delete_all","arguments":{}}' + ) + assert [c["function"]["name"] for c in agnostic] == ["delete_all"], agnostic + + +def test_disabled_leading_bare_json_does_not_hide_later_marker_call(): + # A leading bare-JSON object with a NOT-enabled name is prose: the real DeepSeek/Kimi call after it still parses. + kimi = ( + "<|tool_calls_section_begin|><|tool_call_begin|>functions.web_search:0" + '<|tool_call_argument_begin|>{"q":"cats"}<|tool_call_end|><|tool_calls_section_end|>' + ) + calls = parse_tool_calls_from_text( + '{"name":"draft","parameters":{}} ' + kimi, enabled_tool_names = {"web_search"} + ) + assert [c["function"]["name"] for c in calls] == ["web_search"], calls + assert json.loads(calls[0]["function"]["arguments"]) == {"q": "cats"} + + deepseek = ( + "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>web_search\n" + '```json\n{"q":"cats"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>' + ) + calls_ds = parse_tool_calls_from_text( + '{"name":"draft","parameters":{}} ' + deepseek, enabled_tool_names = {"web_search"} + ) + assert [c["function"]["name"] for c in calls_ds] == ["web_search"], calls_ds + + +def test_disabled_leading_bare_json_ownership_controls(): + kimi_delete = ( + "<|tool_calls_section_begin|><|tool_call_begin|>functions.delete_all:0" + "<|tool_call_argument_begin|>{}<|tool_call_end|><|tool_calls_section_end|>" + ) + # ENABLED leading name still owns the turn (document order, the shipped + # inside-or-after rule). + owns = parse_tool_calls_from_text( + '{"name":"web_search","parameters":{"q":"first"}} ' + kimi_delete, + enabled_tool_names = {"web_search", "delete_all"}, + ) + assert [c["function"]["name"] for c in owns] == ["web_search"], owns + # A marker INSIDE the disabled object's own strings stays data: the span + # is prose, the tail holds no call, so nothing parses. + inside = parse_tool_calls_from_text( + '{"name":"draft","parameters":{"note":"see <|tool_call_begin|>functions.delete_all:0' + '<|tool_call_argument_begin|>{}<|tool_call_end|>"}}\nsome trailing prose', + enabled_tool_names = {"web_search", "delete_all"}, + ) + assert inside == [], inside + # Nameless leading JSON answers keep recursing to the real call. + nameless = parse_tool_calls_from_text( + '{"answer":42} ' + kimi_delete, enabled_tool_names = {"delete_all"} + ) + assert [c["function"]["name"] for c in nameless] == ["delete_all"], nameless + # Name-agnostic path unchanged: the leading object is the call. + agnostic = parse_tool_calls_from_text('{"name":"draft","parameters":{}} ' + kimi_delete) + assert [c["function"]["name"] for c in agnostic] == ["draft"], agnostic + + +def test_leading_json_answer_with_prose_keeps_quoted_gemma_snippet_as_data(): + # A LEADING JSON answer followed by prose is data (same contract as the whole-content JSON exemption). + obj = '{"summary":"use call:web_search{query:cats} to search"}\nHope that helps!' + assert parse_tool_calls_from_text(obj, enabled_tool_names = {"web_search"}) == [] + arr = '["use call:web_search{query:cats} to search"]\nHope that helps!' + assert parse_tool_calls_from_text(arr, enabled_tool_names = {"web_search"}) == [] + assert strip_tool_markup(obj, enabled_tool_names = {"web_search"}) == obj + + # A REAL call in the tail after the answer still parses (and strips). + tail = '{"summary":"done"}\ncall:web_search{query:cats}' + calls = parse_tool_calls_from_text(tail, enabled_tool_names = {"web_search"}) + assert [c["function"]["name"] for c in calls] == ["web_search"], calls + + # A leading brace run that is NOT valid JSON gets no exemption. + not_json = "{not json} call:web_search{query:cats}" + calls_nj = parse_tool_calls_from_text(not_json, enabled_tool_names = {"web_search"}) + assert [c["function"]["name"] for c in calls_nj] == ["web_search"], calls_nj + + +def test_glm_heal_bounds_unclosed_value_at_tool_call_close(): + # Auto-Heal: a value missing only its before the block's heals to the + # value text, not the close tag and everything after it swallowed into the argument. + one = "get_weathercityNYC" + calls = parse_tool_calls_from_text(one, allow_incomplete = True) + assert [c["function"]["name"] for c in calls] == ["get_weather"], calls + assert json.loads(calls[0]["function"]["arguments"]) == {"city": "NYC"} + + # Trailing prose after the close stays out of the healed value. + two = one + "\nLet me check that for you." + calls_two = parse_tool_calls_from_text(two, allow_incomplete = True) + assert json.loads(calls_two[0]["function"]["arguments"]) == {"city": "NYC"} + + # Strict mode still rejects the unclosed value outright. + assert parse_tool_calls_from_text(one, allow_incomplete = False) == [] + + # A value truncated at EOF (no structural tag follows) keeps the partial heal, and a proper + # close whose value holds a literal is untouched by the bounding. + eof = "get_weathercityNew York Ci" + calls_eof = parse_tool_calls_from_text(eof, allow_incomplete = True) + assert json.loads(calls_eof[0]["function"]["arguments"]) == {"city": "New York Ci"} + lit = ( + "get_weathercity" + 'print("")' + ) + calls_lit = parse_tool_calls_from_text(lit, allow_incomplete = True) + assert json.loads(calls_lit[0]["function"]["arguments"]) == {"city": 'print("")'} + + +def test_prose_mentioning_ds_kimi_markers_survives_final_strip(): + # False-alarm literals: the trailing strip arms require a call-shaped + # lookahead, so an answer documenting a marker keeps its tail. + from core.inference.tool_call_parser import strip_tool_markup + + for text in [ + "The Kimi marker <|tool_calls_section_begin|> starts a section.", + "DeepSeek uses <｜tool▁calls▁begin｜> to open calls.", + "See <|tool_call_begin|> in the docs.", + ]: + assert strip_tool_markup(text, final = True) == text + + # Truncated REAL calls still drop, and a bare marker at EOF is a fragment. + truncated_kimi = ( + "<|tool_calls_section_begin|><|tool_call_begin|>functions.web_search:0" + '<|tool_call_argument_begin|>{"q' + ) + assert strip_tool_markup(truncated_kimi, final = True) == "" + assert strip_tool_markup("prefix <|tool_calls_section_begin|>", final = True) == "prefix" diff --git a/studio/backend/tests/test_responses_tool_passthrough.py b/studio/backend/tests/test_responses_tool_passthrough.py index ce5688be3..89a0b3879 100644 --- a/studio/backend/tests/test_responses_tool_passthrough.py +++ b/studio/backend/tests/test_responses_tool_passthrough.py @@ -1990,8 +1990,8 @@ class TestTranslatedMessagesValidate: ChatMessage(**m.model_dump(exclude_none = True)) -# reasoning_prefilled: Qwen3/GLM enable_thinking templates prefill an unclosed , so generation -# begins inside the think block and emits only the closing ; extractor starts in reasoning. +# reasoning_prefilled mode: Qwen3/GLM enable_thinking templates prefill an unclosed , so +# generation begins inside the think block and emits only the closing ; the extractor starts in reasoning. class TestReasoningPrefilledExtractor: def test_prefilled_single_feed_splits_lone_close(self): # T1: reasoning...answer with a prefilled (unseen) open tag. @@ -2055,7 +2055,8 @@ class TestReasoningPrefilledExtractor: assert visible == "\n\nanswer" def test_prefilled_stray_open_tag_is_suppressed(self): - # T7: a re-emitted literal inside prefilled reasoning is dropped, not leaked. + # T7: a re-emitted literal inside prefilled reasoning is dropped, + # not leaked into the drawer (covers enable_thinking_effort full-tag output). reasoning, visible = _extract_responses_reasoning( "abc", parse_think_markers = True, @@ -2076,7 +2077,9 @@ class TestReasoningPrefilledExtractor: assert visible == "hi" def test_not_prefilled_lone_close_preserves_current_behavior(self): - # T9: without prefilled, a lone keeps pre-fix behavior (reasoning stays visible, tag dropped). + # T9: GGUF-parity guard -- WITHOUT prefilled, a lone keeps the + # pre-fix behavior (reasoning stays visible, tag dropped). Ensures GGUF and + # every existing caller are byte-identical. reasoning, visible = _extract_responses_reasoning( "reasoningans", parse_think_markers = True, @@ -2096,7 +2099,8 @@ class TestReasoningPrefilledExtractor: assert visible == "v" def test_prefilled_ignored_when_markers_not_parsed(self): - # T11: a non-reasoning model (parse_think_markers False) passes text straight through. + # T11: a non-reasoning model (parse_think_markers False) still passes text + # straight through even if reasoning_prefilled were mistakenly set False. reasoning, visible = _extract_responses_reasoning( "just an answer", parse_think_markers = False, diff --git a/studio/backend/tests/test_safetensors_capability_advertise.py b/studio/backend/tests/test_safetensors_capability_advertise.py index 643d64af7..3701a00dd 100644 --- a/studio/backend/tests/test_safetensors_capability_advertise.py +++ b/studio/backend/tests/test_safetensors_capability_advertise.py @@ -11,6 +11,8 @@ from pathlib import Path from types import SimpleNamespace from unittest.mock import MagicMock +import pytest + _backend_root = Path(__file__).resolve().parent.parent if str(_backend_root) not in sys.path: sys.path.insert(0, str(_backend_root)) @@ -127,8 +129,8 @@ def test_detect_safetensors_features_gptoss_disables_tools(): assert flags["supports_tools"] is False -# Llama-3 / Mistral / Gemma 4 tool-call formats are parser-supported, so supports_tools stays True; -# only templates matching none of the known markers are suppressed. +# Llama-3 / Mistral / Gemma 4 tool-call formats are now parser-supported, so supports_tools=True +# must hold for all of them; only templates matching none of the five known markers are suppressed. LLAMA3_TEMPLATE = """ {%- if tools %} @@ -198,6 +200,86 @@ def test_detect_safetensors_features_gemma4_template_keeps_tools_on(): assert flags["supports_tools"] is True +# DeepSeek V3 / V3.1 / R1 emit ``<｜tool▁calls▁begin｜>...`` blocks. +# Note the full-width pipe (U+FF5C) and lower-1/8-block (U+2581). +DEEPSEEK_TEMPLATE = """ +{%- if tools %} + {%- for tool in tools %} + {{- tool | tojson }} + {%- endfor %} +{%- endif %} +{%- for message in messages %} + {%- if message.role == 'assistant' and message.tool_calls %} + {%- for tc in message.tool_calls %} + {{- '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tc.function.name + + '<｜tool▁sep｜>' + tc.function.arguments + '<｜tool▁call▁end｜>' }} + {%- endfor %} + {%- endif %} +{%- endfor %} +""" + + +def test_detect_safetensors_features_deepseek_template_keeps_tools_on(): + """DeepSeek emits ``<｜tool▁calls▁begin｜>...``; parser now supports it.""" + from routes.inference import _detect_safetensors_features + + backend = SimpleNamespace(active_model_name = "unsloth/DeepSeek-V3.1") + flags = _detect_safetensors_features(backend, DEEPSEEK_TEMPLATE) + assert flags["supports_tools"] is True + + +# GLM 4.5 / 4.6 / 4.7 emit ``NAME\n...... +GLM_TEMPLATE = """ +{%- if tools %} + For each function call, output the function name and arguments within + the following XML format: + {function-name} + {arg-key} + {arg-value} + + {%- for tool in tools %} + {{- tool | tojson }} + {%- endfor %} +{%- endif %} +""" + + +def test_detect_safetensors_features_glm_template_keeps_tools_on(): + """GLM 4.x emits ``NAME\\n...``; parser handles it.""" + from routes.inference import _detect_safetensors_features + + backend = SimpleNamespace(active_model_name = "unsloth/GLM-4.6") + flags = _detect_safetensors_features(backend, GLM_TEMPLATE) + assert flags["supports_tools"] is True + + +# Kimi K2 / Moonshot uses ``<|tool_calls_section_begin|>...`` blocks +# with ``functions.NAME:IDX`` as the per-call id. +KIMI_TEMPLATE = """ +{%- if tools %} + <|im_system|>tool_declare<|im_middle|>{{ tools | tojson }}<|im_end|> +{%- endif %} +{%- for message in messages %} + {%- if message.role == 'assistant' and message.tool_calls %} + <|tool_calls_section_begin|> + {%- for tc in message.tool_calls %} + <|tool_call_begin|>{{ tc.id }}<|tool_call_argument_begin|>{{ tc.function.arguments | tojson }}<|tool_call_end|> + {%- endfor %} + <|tool_calls_section_end|> + {%- endif %} +{%- endfor %} +""" + + +def test_detect_safetensors_features_kimi_template_keeps_tools_on(): + """Kimi K2 emits ``<|tool_calls_section_begin|>...``; parser handles it.""" + from routes.inference import _detect_safetensors_features + + backend = SimpleNamespace(active_model_name = "unsloth/Kimi-K2-Instruct") + flags = _detect_safetensors_features(backend, KIMI_TEMPLATE) + assert flags["supports_tools"] is True + + LLAMA3_2_BARE_JSON_TEMPLATE = """ {%- if tools %} {{- 'Given the following functions, respond with JSON for a function call.' }} @@ -534,7 +616,34 @@ def test_route_layer_emits_supports_tools_true_for_qwen3_safetensors(): assert flags["supports_preserve_thinking"] is True -# Templates advertising tools whose ``{"name":`` example is pretty-printed or JSON-escaped. +@pytest.mark.parametrize( + "opener", + [ + "<｜tool▁calls▁begin｜>", # canonical + "<｜tool_calls_begin｜>", # ASCII underscores + "<｜tool▁calls｜>", # short form + "<｜tool calls begin｜>", # spaces + "<｜tool\\_calls\\_begin｜>", # escaped underscores + ], +) +def test_detect_safetensors_features_deepseek_opener_variants_keep_tools_on(opener): + # Every DeepSeek opener the parser accepts must keep supports_tools on; the route gate derives + # its markers from the parser's TOOL_XML_SIGNALS so it can no longer drift behind the parser ... + from routes.inference import _detect_safetensors_features + + tpl = ( + "{%- if tools %}tools{%- endif %}" + + opener + + "<｜tool▁call▁begin｜>function<｜tool▁sep｜>get_time{}" + "<｜tool▁call▁end｜><｜tool▁calls▁end｜>" + ) + backend = SimpleNamespace(active_model_name = "unsloth/DeepSeek-V3.1") + flags = _detect_safetensors_features(backend, tpl) + assert flags["supports_tools"] is True + + +# Templates that advertise tools ({%- if tools %}) and prompt the bare-JSON +# call form, but whose ``{"name":`` example is pretty-printed or JSON-escaped. _WHITESPACE_BARE_JSON_TEMPLATE = ( "{%- if tools %}\n" "To call a tool, output JSON of the form:\n" @@ -554,7 +663,8 @@ _TOOLS_ADVERTISED_NO_PARSEABLE_FORM = ( def test_detect_safetensors_features_keeps_tools_for_pretty_printed_bare_json(): - # Pretty-printed bare-JSON (``{ "name" :``) keeps supports_tools: parser accepts the whitespace. + # A pretty-printed bare-JSON example (``{ "name" :``) must keep supports_tools since the parser + # accepts that whitespace via raw_decode. from routes.inference import _detect_safetensors_features backend = SimpleNamespace(active_model_name = "unsloth/Llama-3.2-3B-Instruct") @@ -571,7 +681,8 @@ def test_detect_safetensors_features_keeps_tools_for_escaped_bare_json(): def test_detect_safetensors_features_drops_tools_when_no_parseable_form(): - # Negative control: tools advertised but no parser-recognised emission form -> pill dropped. + # Negative control: tools advertised but no parser-recognised emission form at + # all -> the pill is still dropped (the gate is not now matching everything). from routes.inference import _detect_safetensors_features backend = SimpleNamespace(active_model_name = "unsloth/Llama-3.2-3B-Instruct") @@ -580,7 +691,8 @@ def test_detect_safetensors_features_drops_tools_when_no_parseable_form(): def test_detect_safetensors_features_keeps_tools_for_function_alias_bare_json(): - # The {"function":...} bare-JSON alias keeps supports_tools, mirroring {"name":...}. + # A template documenting the parser-supported {"function":...} bare-JSON alias + # must keep supports_tools, mirroring the {"name":...} form. from routes.inference import _detect_safetensors_features tpl = ( @@ -594,9 +706,10 @@ def test_detect_safetensors_features_keeps_tools_for_function_alias_bare_json(): assert flags["supports_tools"] is True -# _sf_reasoning_prefill_mode gates the prefilled- extractor for enable_thinking models. +# _sf_reasoning_prefill_mode gates the prefilled- extractor so safetensors/MLX reach +# GGUF reasoning-block parity for enable_thinking models. class TestSafetensorsReasoningPrefillGate: - # Qwen3-style template with the standard / markers. + # A minimal Qwen3-style template with the standard / markers. _QWEN_TPL = "{% if enable_thinking %}{% endif %}......" # gemma-style bespoke reasoning channel -- no standard markers. _GEMMA_TPL = "{% if enable_thinking %}<|think|>{% endif %}<|channel>thought" @@ -650,8 +763,8 @@ class TestSafetensorsReasoningPrefillGate: assert _sf_reasoning_prefill_mode(feats, False, self._QWEN_TPL) is True def test_g8_gemma_bespoke_channel_excluded(self): - # G8: gemma's <|think|>/<|channel> format has no -> NOT prefilled (else the - # whole answer is swallowed as reasoning). Regression guard. + # G8: gemma's <|think|>/<|channel> format has no -> NOT prefilled + # (would otherwise swallow the whole answer as reasoning). Regression guard. from routes.inference import _sf_reasoning_prefill_mode assert _sf_reasoning_prefill_mode(self._features(), True, self._GEMMA_TPL) is False diff --git a/studio/backend/tests/test_safetensors_reasoning_stream.py b/studio/backend/tests/test_safetensors_reasoning_stream.py index 9158d1ad5..4a5423fa8 100644 --- a/studio/backend/tests/test_safetensors_reasoning_stream.py +++ b/studio/backend/tests/test_safetensors_reasoning_stream.py @@ -34,7 +34,7 @@ def _replay_sf_reasoning_stream(events: list[dict], *, prefilled: bool) -> dict: visible_deltas: list[str] = [] monitor: list[str] = [] tool_starts: list[dict] = [] - order: list[str] = [] # "reasoning" | "visible" | "tool_start" sequence + order: list[str] = [] # sequence of ("reasoning"|"visible"|"tool_start") events def _flush(): fr, fv = extractor.finish() @@ -155,8 +155,11 @@ _THINK_TPL = "...{% if enable_thinking %}{% endif %}......" def test_s6_reasoning_effort_none_disables_prefill_for_enable_thinking_effort(): - # GLM-5.2 enable_thinking_effort + reasoning_effort="none" disables thinking like - # enable_thinking=False, so prefilled must be OFF (else the answer is swallowed into reasoning). + # GLM-5.2-style enable_thinking_effort: a request with reasoning_effort="none" (and + # enable_thinking omitted) disables thinking exactly like enable_thinking=False, so + # prefilled mode must be OFF. Otherwise the model emits no and a plain + # answer is swallowed whole into reasoning_content, leaving the visible response + # empty (the exact bug: prefilled=True below eats the whole answer). feats = {"reasoning_style": "enable_thinking_effort", "supports_reasoning": True} assert _sf_reasoning_prefill_mode(feats, None, _THINK_TPL, "none") is False # Thinking on (effort level or default) still prefills. @@ -171,7 +174,8 @@ def test_s6_reasoning_effort_none_disables_prefill_for_enable_thinking_effort(): plain = {"reasoning_style": "enable_thinking", "supports_reasoning": True} assert _sf_reasoning_prefill_mode(plain, None, _THINK_TPL, "none") is True - # End-to-end: with prefilled=False, a plain no- answer stays visible. + # End-to-end: with the corrected prefilled=False, a plain no- answer is + # emitted as visible content rather than swallowed into the thinking drawer. events = [{"type": "content", "text": "The capital of France is Paris."}] out = _replay_sf_reasoning_stream(events, prefilled = False) assert out["visible"] == "The capital of France is Paris." diff --git a/studio/backend/tests/test_safetensors_tool_loop.py b/studio/backend/tests/test_safetensors_tool_loop.py index 984d5f8ae..38b30fe8f 100644 --- a/studio/backend/tests/test_safetensors_tool_loop.py +++ b/studio/backend/tests/test_safetensors_tool_loop.py @@ -139,7 +139,7 @@ class TestParser: assert "print('hi')" in result[0]["function"]["arguments"] def test_xml_param_preserves_leading_indentation(self): - # Only the wrapping newline is trimmed, so code indentation survives. + # Only the wrapping newline is trimmed, so code-argument indentation survives (str.strip() destroyed it). text = ( "\n" " indented = 1\n" @@ -204,7 +204,9 @@ class TestParser: assert strip_tool_markup(text) == "before after" def test_strip_named_mistral_call_consumes_trailing_eos(self): - # The named [TOOL_CALLS]name{json} shape must eat the optional trailing . + # The named ``[TOOL_CALLS]name{json}`` shape must eat the optional + # trailing ```` like the array shape, so the EOS marker is not left + # behind as visible content. text = '[TOOL_CALLS]web_search{"query":"cats"}' assert strip_tool_markup(text) == "" text = '[TOOL_CALLS]web_search{"query":"cats"} and then' @@ -235,8 +237,27 @@ class TestParser: == "before " ) + def test_streaming_strip_handles_nested_mistral_json(self): + # The non-greedy [TOOL_CALLS]name{...} pattern truncates nested JSON at the first }; the + # balanced helper must remove the whole call so no trailing brace leaks to the streaming ... + raw = 'ok [TOOL_CALLS]foo{"a":{"b":1}} tail' + out = strip_tool_markup_streaming(raw) + assert "[TOOL_CALLS]" not in out + assert "}" not in out + assert "ok " in out and "tail" in out + + def test_streaming_strip_handles_nested_wrapperless_gemma(self): + # Same class of bug for the wrapper-less Gemma call:NAME{...} form with a + # nested object argument. + raw = "ok call:f{loc:{city:NYC},n:3} tail" + out = strip_tool_markup_streaming(raw) + assert "call:f" not in out + assert "}" not in out + assert "ok " in out and "tail" in out + def test_streaming_strip_keeps_prose_after_function_xml_with_literal_marker(self): - # A literal in a value is data: the strip closes at the REAL , keeping prose. + # A literal ```` in a value is data: the strip must close at the REAL + # ```` and keep trailing prose (the open-ended regex ate to EOF). raw = ( "pref " 'print("") tail' @@ -246,15 +267,19 @@ class TestParser: assert strip_tool_markup_streaming(raw) == strip_tool_markup(raw, final = True) def test_streaming_strip_drops_leading_magistral_reasoning(self): - # Magistral reasoning is a leading [THINK]...[/THINK] block; the streaming strip must drop it. + # Magistral emits reasoning as a leading ``[THINK]...[/THINK]`` bracket block + # (not the ```` the reasoning channel renders). The streaming display + # strip must drop it so the raw chain-of-thought does not leak into the + # safetensors content; GGUF routes it to reasoning_content natively. closed = "[THINK]Let me think. 2+2 is 4.[/THINK]The answer is 4." assert strip_tool_markup_streaming(closed) == "The answer is 4." assert strip_tool_markup_streaming(closed) == strip_tool_markup(closed, final = True) - # Unclosed mid-stream reasoning is held; cleaned text grows only after [/THINK]. + # Unclosed mid-stream reasoning is held from the marker on (nothing leaks, and + # the cleaned text only grows as the answer streams in after ``[/THINK]``). assert strip_tool_markup_streaming("[THINK]still thinking") == "" assert strip_tool_markup_streaming("[THINK]r[/THINK]The") == "The" assert strip_tool_markup_streaming("[THINK]r[/THINK]The answer") == "The answer" - # A non-leading [THINK] is ordinary prose, left untouched. + # A non-leading ``[THINK]`` is ordinary prose and is left untouched. assert strip_tool_markup_streaming("hi [THINK] later") == "hi [THINK] later" @@ -294,7 +319,7 @@ class TestParserMultiFormat: assert args == {"query": "hi", "n": 5} def test_llama3_python_tag_json_form_with_eom(self): - # Llama-3 emits <|eom_id|> after the JSON; must not break parsing. + # Llama-3 emits ``<|eom_id|>`` after the JSON; must not break parsing. import json text = '<|python_tag|>{"name":"python","parameters":{"code":"print(2+2)"}}<|eom_id|>' @@ -307,10 +332,22 @@ class TestParserMultiFormat: text = '<|python_tag|>brave_search.call(query="x")' assert strip_tool_markup(text, final = True) == "" - # Llama-3.2 bare JSON ``custom_tools`` + def test_llama3_python_tag_json_form_non_scalar_args_skipped(self): + # Should NOT fabricate ``{"value": args}`` when the JSON form + # has a non-dict / non-string ``arguments`` value. + for bad in ( + '<|python_tag|>{"name":"foo","arguments":42}', + '<|python_tag|>{"name":"foo","arguments":[1,2,3]}', + '<|python_tag|>{"name":"foo","arguments":null}', + '<|python_tag|>{"name":"foo","arguments":true}', + ): + assert parse_tool_calls_from_text(bad) == [], bad + + # ── Llama-3.2 bare JSON ``custom_tools`` ───────────────────── def test_llama3_2_bare_json_parameters(self): - # Llama-3.2-Instruct emits bare JSON directly as content, no <|python_tag|> prefix. + # Llama-3.2-Instruct emits bare JSON directly as content; no + # <|python_tag|> prefix per its training template. import json text = '{"name":"web_search","parameters":{"query":"Tokyo weather"}}' @@ -330,7 +367,7 @@ class TestParserMultiFormat: assert args == {"a": 1, "b": 2} def test_llama3_2_bare_json_multi_call(self): - # Llama-3 may chain calls with "; " per training template. + # Llama-3 may chain calls with ``; `` per training template. text = '{"name":"a","parameters":{}}; {"name":"b","parameters":{}}' result = parse_tool_calls_from_text(text) assert len(result) == 2 @@ -356,7 +393,8 @@ class TestParserMultiFormat: assert parse_tool_calls_from_text(text) == [] def test_llama3_2_bare_json_embedded_in_prose_does_not_fire(self): - # Defensive: JSON embedded in prose must NOT fire (content must START with `{`). + # Defensive: JSON embedded in prose must NOT fire (parser is + # strict about content STARTING with `{`). text = 'The tool result was: {"name":"foo"}' assert parse_tool_calls_from_text(text) == [] @@ -373,12 +411,14 @@ class TestParserMultiFormat: assert parse_tool_calls_from_text(text) == [] def test_llama3_2_bare_json_string_parameters_does_not_fire(self): - # Llama-3 spec: parameters must be a dict; a string value must NOT trigger. + # Llama-3 spec: parameters must be a dict. Prose like + # ``{"name":"foo","parameters":"a sentence"}`` must NOT trigger. text = '{"name":"foo","parameters":"this is a sentence"}' assert parse_tool_calls_from_text(text) == [] def test_llama3_2_bare_json_string_arguments_not_json_does_not_fire(self): - # OpenAI arguments may be a JSON-string of a dict, but a plain non-JSON string must not pass. + # OpenAI ``arguments`` may be a JSON-string of a dict, but a + # plain non-JSON string must not pass the guard. text = '{"name":"foo","arguments":"not json"}' assert parse_tool_calls_from_text(text) == [] @@ -417,7 +457,8 @@ class TestParserMultiFormat: def test_mistral_array_parameters_key_alias(self): import json - # Array object keyed on parameters (not arguments) must keep its payload. + # Array object keyed on ``parameters`` (not ``arguments``) must keep its + # payload, matching the JSON/XML paths and SGLang's base detector. text = '[TOOL_CALLS] [{"name":"get_weather","parameters":{"city":"Paris"}}]' result = parse_tool_calls_from_text(text) assert len(result) == 1 @@ -435,7 +476,7 @@ class TestParserMultiFormat: assert result[1]["function"]["name"] == "b" def test_mistral_pre_v11_unclosed_array(self): - # Closing ] truncated: parser must heal off individual objects. + # Closing ``]`` truncated -- parser must heal off individual objects. text = '[TOOL_CALLS] [{"name":"web_search","arguments":{"q":"x"},"id":"id"}' result = parse_tool_calls_from_text(text) assert len(result) == 1 @@ -444,7 +485,7 @@ class TestParserMultiFormat: # Mistral v11+ def test_mistral_v11_single(self): - # Magistral / Mistral Small 3.1: bare name{json} after trigger. + # Magistral / Mistral Small 3.1: bare ``name{json}`` after trigger. import json text = '[TOOL_CALLS]add{"a":3.5,"b":4}' @@ -454,7 +495,7 @@ class TestParserMultiFormat: assert json.loads(result[0]["function"]["arguments"]) == {"a": 3.5, "b": 4} def test_mistral_v11_parallel(self): - # v11+ parallel: [TOOL_CALLS]a{...}[TOOL_CALLS]b{...}. + # v11+ parallel: ``[TOOL_CALLS]a{...}[TOOL_CALLS]b{...}``. text = '[TOOL_CALLS]add{"a":1}[TOOL_CALLS]sub{"b":2}' result = parse_tool_calls_from_text(text) assert len(result) == 2 @@ -462,7 +503,7 @@ class TestParserMultiFormat: assert result[1]["function"]["name"] == "sub" def test_mistral_v11_with_args_marker(self): - # Ministral / Mistral Large 3: [TOOL_CALLS]name[ARGS]{json}. + # Ministral / Mistral Large 3: ``[TOOL_CALLS]name[ARGS]{json}``. import json text = '[TOOL_CALLS]add[ARGS]{"a":1,"b":2}' @@ -476,7 +517,9 @@ class TestParserMultiFormat: assert strip_tool_markup(text, final = True) == "" def test_mistral_call_id_form(self): - # Mistral Small 3.2: the [CALL_ID] segment must be skipped, not treated as a stop (llama.cpp test-chat.cpp:4785). + # Mistral Small 3.2: ``[TOOL_CALLS]name[CALL_ID][ARGS]{json}``. + # The ``[CALL_ID]`` segment must be skipped, not treated as a stop + # (llama.cpp test-chat.cpp:4785 parses this to one call). import json text = '[TOOL_CALLS]special_function[CALL_ID]123456789[ARGS]{"arg1": 1}' @@ -501,7 +544,9 @@ class TestParserMultiFormat: assert strip_tool_markup(text, final = True) == "" def test_mistral_think_reasoning_ignored(self): - # A [TOOL_CALLS] inside [THINK]...[/THINK] is reasoning; only the call after [/THINK] counts (llama.cpp test-chat.cpp:2285). + # Magistral wraps reasoning in ``[THINK]...[/THINK]``. A ``[TOOL_CALLS]`` + # inside the reasoning is chain-of-thought, not a real call; only the + # call after ``[/THINK]`` counts (llama.cpp test-chat.cpp:2285). import json text = ( @@ -514,12 +559,14 @@ class TestParserMultiFormat: assert json.loads(result[0]["function"]["arguments"]) == {"y": 2} def test_mistral_think_reasoning_no_real_call(self): - # Reasoning that mentions a call but emits none after [/THINK] yields no calls. + # Reasoning that merely mentions a tool call but does not emit one + # after ``[/THINK]`` yields no calls. text = '[THINK]I might call [TOOL_CALLS]fake[ARGS]{"x":1}[/THINK]Done.' assert parse_tool_calls_from_text(text) == [] def test_mistral_think_literal_in_argument_preserved(self): - # A literal [THINK] inside a real tool argument must not be stripped or corrupt the parse. + # A literal ``[THINK]`` inside a real tool argument (after the call) + # must not be stripped or corrupt the parse. import json text = '[TOOL_CALLS]search[ARGS]{"q":"explain the [THINK] token"}' @@ -554,7 +601,7 @@ class TestParserMultiFormat: assert args == {"enabled": True, "attempts": 5, "threshold": 1.5, "nickname": None} def test_gemma4_nested_args(self): - # Gemma 4 nests dicts / lists with bare keys and <|"|> strings. + # Gemma 4 nests dicts / lists with bare keys and ``<|"|>`` strings. import json text = ( @@ -585,7 +632,65 @@ class TestParserMultiFormat: text = "<|tool_call>call:foo{x:1}" assert strip_tool_markup(text, final = True) == "" - # Cross-format sentinels + # ── Gemma 4 wrapper-less (skip_special_tokens stripped) ─────────── + + def test_gemma4_bare_stripped_call(self): + # skip_special_tokens removes <|tool_call>/ and <|"|>, + # leaving a bare call:NAME{...} with an unquoted value. + import json + + text = "call:web_search{query:weather in San Francisco right now}" + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + assert result[0]["function"]["name"] == "web_search" + args = json.loads(result[0]["function"]["arguments"]) + assert args == {"query": "weather in San Francisco right now"} + + def test_gemma4_bare_code_with_commas(self): + # A code value with commas must not truncate at the first comma. + import json + + text = ( + "call:python{code:def f(n):\n a, b = 0, 1\n" + " for _ in range(2, n+1):\n a, b = b, a + b\n" + " return b\n\nprint(f(30))}" + ) + result = parse_tool_calls_from_text(text) + assert result[0]["function"]["name"] == "python" + code = json.loads(result[0]["function"]["arguments"])["code"] + assert "a, b = 0, 1" in code and "print(f(30))" in code + + def test_gemma4_bare_quotes_normalized(self): + # The same value quoted vs unquoted must parse identically so the + # agentic loop can collapse a looping model's repeated calls. + import json + + a = parse_tool_calls_from_text('call:web_search{query:"foo bar"}') + b = parse_tool_calls_from_text("call:web_search{query:foo bar}") + assert json.loads(a[0]["function"]["arguments"]) == {"query": "foo bar"} + assert json.loads(a[0]["function"]["arguments"]) == json.loads( + b[0]["function"]["arguments"] + ) + + def test_gemma4_bare_multi_arg(self): + import json + + text = "call:web_search{query:pytorch latest, url:https://pytorch.org}" + result = parse_tool_calls_from_text(text) + args = json.loads(result[0]["function"]["arguments"]) + assert args == {"query": "pytorch latest", "url": "https://pytorch.org"} + + def test_gemma4_bare_not_matched_in_prose(self): + # A word ending in "call:" must not trigger a bare tool call. + text = "I will recall:that the function{ } is helpful." + result = parse_tool_calls_from_text(text) + assert result == [] + + def test_gemma4_bare_strip_markup_final(self): + text = "Here you go: call:web_search{query:weather today}" + assert "call:web_search" not in strip_tool_markup(text, final = True) + + # ── Cross-format sentinels ──────────────────────────────────── def test_all_markers_in_tool_xml_signals(self): # Streaming buffer wakes up on every emission marker. @@ -703,6 +808,553 @@ def _make_loop( ), exec_fn +class TestParserDeepSeek: + """DeepSeek R1 / V3 / V3.1 coverage. Markers use full-width pipes + (U+FF5C) and lower-one-eighth-block (U+2581). R1 wraps args in a + Markdown ``` ```json ``` ``` fence; V3 / V3.1 emit bare JSON.""" + + def test_r1_simple_call_with_code_fence(self): + import json as _json + + text = ( + "<｜tool▁calls▁begin｜>" + "<｜tool▁call▁begin｜>function" + "<｜tool▁sep｜>special_function\n" + "```json\n" + '{"arg1": 1}\n' + "```" + "<｜tool▁call▁end｜>" + "<｜tool▁calls▁end｜>" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + assert result[0]["function"]["name"] == "special_function" + assert _json.loads(result[0]["function"]["arguments"]) == {"arg1": 1} + + def test_r1_short_form_outer_marker(self): + # llama.cpp accepts ``<｜tool▁calls｜>`` as the short-form opener. + import json as _json + + text = ( + "<｜tool▁calls｜>function" + "<｜tool▁sep｜>get_time\n" + "```json\n" + '{"city": "Paris"}\n' + "```" + "<｜tool▁call▁end｜>" + "<｜tool▁calls▁end｜>" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + assert result[0]["function"]["name"] == "get_time" + + def test_v3_1_bare_json(self): + # V3 / V3.1 omit the ``function`` prefix and the code fence. + import json as _json + + text = ( + "<｜tool▁calls▁begin｜>" + "<｜tool▁call▁begin｜>get_time" + "<｜tool▁sep｜>" + '{"city": "Tokyo"}' + "<｜tool▁call▁end｜>" + "<｜tool▁calls▁end｜>" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + assert result[0]["function"]["name"] == "get_time" + assert _json.loads(result[0]["function"]["arguments"]) == {"city": "Tokyo"} + + def test_v3_1_multi_call_shares_envelope(self): + # Parallel calls share one outer envelope; each inner call has + # its own ``<｜tool▁call▁begin｜>...<｜tool▁call▁end｜>``. + text = ( + "<｜tool▁calls▁begin｜>" + "<｜tool▁call▁begin｜>get_time" + "<｜tool▁sep｜>" + '{"city": "Paris"}' + "<｜tool▁call▁end｜>" + "<｜tool▁call▁begin｜>get_weather" + "<｜tool▁sep｜>" + '{"city": "Paris"}' + "<｜tool▁call▁end｜>" + "<｜tool▁calls▁end｜>" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 2 + assert result[0]["function"]["name"] == "get_time" + assert result[1]["function"]["name"] == "get_weather" + + def test_v3_1_with_reasoning(self): + # Reasoning ... precedes the tool block. + text = ( + "I'm thinking\n" + "<｜tool▁calls▁begin｜>" + "<｜tool▁call▁begin｜>get_time" + "<｜tool▁sep｜>" + '{"city": "Tokyo"}' + "<｜tool▁call▁end｜>" + "<｜tool▁calls▁end｜>" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + assert result[0]["function"]["name"] == "get_time" + + def test_v3_1_strict_rejects_unclosed_envelope(self): + # Envelope truncated mid-stream (no <｜tool▁calls▁end｜>): healed by + # default, rejected with Auto-Heal off. + text = ( + "<｜tool▁calls▁begin｜>" + "<｜tool▁call▁begin｜>get_time" + "<｜tool▁sep｜>" + '{"city": "Tokyo"}' + ) + assert len(parse_tool_calls_from_text(text)) == 1 + assert parse_tool_calls_from_text(text, allow_incomplete = False) == [] + + def test_v3_1_multi_call_recovers_when_first_end_marker_missing(self): + # First inner call omits its <｜tool▁call▁end｜>; the second must still be parsed. + text = ( + "<｜tool▁calls▁begin｜>" + "<｜tool▁call▁begin｜>get_time" + "<｜tool▁sep｜>" + '{"city": "Paris"}' + "<｜tool▁call▁begin｜>get_weather" + "<｜tool▁sep｜>" + '{"city": "Paris"}' + "<｜tool▁call▁end｜>" + "<｜tool▁calls▁end｜>" + ) + result = parse_tool_calls_from_text(text) + assert [c["function"]["name"] for c in result] == ["get_time", "get_weather"] + + def test_v3_1_strict_recovers_after_missing_call_end(self): + # Strict mode (Auto-Heal off): the FIRST inner call is missing its <｜tool▁call▁end｜> + # terminator, so it is skipped -- but the parser must keep scanning and still return the ... + text = ( + "<｜tool▁calls▁begin｜>" + "<｜tool▁call▁begin｜>get_weather" + "<｜tool▁sep｜>" + '{"city": "SF"}' + "<｜tool▁call▁begin｜>get_time" + "<｜tool▁sep｜>" + '{"tz": "PST"}' + "<｜tool▁call▁end｜>" + "<｜tool▁calls▁end｜>" + ) + # Auto-Heal keeps both; strict skips the truncated first, keeps the second. + assert [c["function"]["name"] for c in parse_tool_calls_from_text(text)] == [ + "get_weather", + "get_time", + ] + strict = parse_tool_calls_from_text(text, allow_incomplete = False) + assert [c["function"]["name"] for c in strict] == ["get_time"] + + def test_r1_strict_recovers_after_missing_close_fence(self): + # R1 form. + text = ( + "<｜tool▁calls▁begin｜>" + "function<｜tool▁sep｜>get_weather\n```json\n" + '{"city": "SF"}' + "function<｜tool▁sep｜>get_time\n```json\n" + '{"tz": "PST"}' + "\n```<｜tool▁call▁end｜>" + "<｜tool▁calls▁end｜>" + ) + strict = parse_tool_calls_from_text(text, allow_incomplete = False) + assert [c["function"]["name"] for c in strict] == ["get_time"] + + def test_deepseek_strip_markup(self): + text = ( + "before " + "<｜tool▁calls▁begin｜>" + "<｜tool▁call▁begin｜>foo" + "<｜tool▁sep｜>" + "{}" + "<｜tool▁call▁end｜>" + "<｜tool▁calls▁end｜>" + " after" + ) + assert strip_tool_markup(text, final = True) == "before after" + + def test_deepseek_signal_wakes_streaming(self): + # The streaming buffer state machine must wake on the DeepSeek opener so the rest of the + # section is drained instead of leaked. + text = "<｜tool▁calls▁begin｜>..." + assert has_tool_signal(text) + + def test_deepseek_short_opener_is_stripped(self): + # The short ``<｜tool▁calls｜>`` opener is parsed, so its markup must also be stripped (the + # strip patterns used to require ...calls_begin and left the short-opener markup leaking to ... + text = ( + "before " + "<｜tool▁calls｜>" + "<｜tool▁call▁begin｜>foo" + "<｜tool▁sep｜>" + "{}" + "<｜tool▁call▁end｜>" + "<｜tool▁calls▁end｜>" + " after" + ) + assert strip_tool_markup(text, final = True) == "before after" + + +class TestParserGLM: + """GLM 4.5 / 4.6 / 4.7 coverage. Marker collides with Qwen's + ```` but the body shape is XML kv pairs instead of JSON, + so the dispatch order keeps both formats working.""" + + def test_glm_simple_call(self): + import json as _json + + text = ( + "web_search\n" + "query\n" + "weather Tokyo\n" + "" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + assert result[0]["function"]["name"] == "web_search" + args = _json.loads(result[0]["function"]["arguments"]) + # Strings come through raw; the parser does not double-quote. + assert args == {"query": "weather Tokyo"} + + def test_glm_mixed_types_decode_correctly(self): + # Per the chat_template.jinja, strings are emitted raw and non-strings are JSON-encoded. + import json as _json + + text = ( + "complex_function\n" + "name\nJohn Doe\n" + "age\n30\n" + "active\ntrue\n" + "score\n95.5\n" + "" + ) + result = parse_tool_calls_from_text(text) + args = _json.loads(result[0]["function"]["arguments"]) + assert args == {"name": "John Doe", "age": 30, "active": True, "score": 95.5} + + def test_glm_multi_call_back_to_back(self): + # GLM emits parallel calls as consecutive ``... + # `` blocks with no outer envelope. + text = ( + "a\nx\n1\n" + "b\ny\n2\n" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 2 + assert result[0]["function"]["name"] == "a" + assert result[1]["function"]["name"] == "b" + + def test_glm_unclosed_tool_call_does_not_lose_value(self): + # Truncated mid-stream (no ) -- the parser must + # still surface what it found rather than dropping the call. + text = "web_search\nquery\npartial" + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + assert result[0]["function"]["name"] == "web_search" + + def test_glm_does_not_break_qwen_path(self): + # Real Qwen emission must still be parsed by the Qwen branch, + # not silently misrouted to GLM (the marker is shared). + text = '{"name":"web_search","arguments":{"q":"x"}}' + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + assert result[0]["function"]["name"] == "web_search" + + def test_glm_strip_markup(self): + text = ( + "before " + "a\nx\n1\n" + " after" + ) + assert strip_tool_markup(text, final = True) == "before after" + + def test_glm_zero_arg_inline_call(self): + # GLM 4.7 emits a no-argument call inline as ``name`` (name followed + # straight by the close tag, no \n / ). + import json as _json + + text = "get_current_date" + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + assert result[0]["function"]["name"] == "get_current_date" + assert _json.loads(result[0]["function"]["arguments"]) == {} + + def test_glm_zero_arg_call_in_parallel_batch(self): + # A no-arg call alongside a normal one must not make either vanish. + text = ( + "get_current_date" + "get_weather\ncity\n" + "Tokyo" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 2 + assert result[0]["function"]["name"] == "get_current_date" + assert result[1]["function"]["name"] == "get_weather" + + def test_glm_string_value_whitespace_preserved(self): + # The template emits string args verbatim, so significant leading / trailing whitespace + # (code, diffs) must survive. + import json as _json + + text = ( + "run\ncode\n" + " indented code " + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + args = _json.loads(result[0]["function"]["arguments"]) + assert args == {"code": " indented code "} + + +class TestParserKimi: + """Kimi K2 / Moonshot coverage. ASCII pipes only (NOT full-width). + Name arrives as ``functions.NAME:IDX``; the parser strips the + prefix and the index to recover the bare callable name while + preserving the full id for round-trip rendering.""" + + def test_kimi_simple_call(self): + import json as _json + + text = ( + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>functions.special_function:0" + "<|tool_call_argument_begin|>" + '{"arg1": 1}' + "<|tool_call_end|>" + "<|tool_calls_section_end|>" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + # Bare name recovered; full id preserved verbatim. + assert result[0]["function"]["name"] == "special_function" + assert result[0]["id"] == "functions.special_function:0" + assert _json.loads(result[0]["function"]["arguments"]) == {"arg1": 1} + + def test_outer_tool_call_with_embedded_kimi_marker_parses_outer(self): + # A Qwen/Hermes whose argument contains literal Kimi markup (a user asking + # about that syntax) must execute the OUTER call, not the embedded marker via the ... + text = ( + '{"name":"web_search","arguments":{"query":' + '"explain <|tool_call_begin|>functions.evil:0' + '<|tool_call_argument_begin|>{}<|tool_call_end|>"}}' + "" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + assert result[0]["function"]["name"] == "web_search" + + def test_genuine_kimi_call_without_envelope_still_parses(self): + # Control: a real Kimi call with no leading envelope must + # still go through the pre-pass. + text = ( + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>functions.web_search:0" + '<|tool_call_argument_begin|>{"query":"x"}<|tool_call_end|>' + "<|tool_calls_section_end|>" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + assert result[0]["function"]["name"] == "web_search" + + def test_kimi_multi_call_with_index(self): + # Multiple consecutive calls inside a single section, each + # with its own monotonically incrementing ``:IDX``. + text = ( + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>functions.read_file:0" + "<|tool_call_argument_begin|>" + '{"path":"a"}' + "<|tool_call_end|>" + "<|tool_call_begin|>functions.web_search:1" + "<|tool_call_argument_begin|>" + '{"query":"x"}' + "<|tool_call_end|>" + "<|tool_calls_section_end|>" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 2 + assert result[0]["function"]["name"] == "read_file" + assert result[0]["id"].endswith(":0") + assert result[1]["function"]["name"] == "web_search" + assert result[1]["id"].endswith(":1") + + def test_kimi_dotted_name_keeps_full_dotted_name(self): + # A dotted Kimi id keeps its FULL name after stripping only the ``functions.`` prefix and + # ``:idx`` suffix -- matching current vLLM ... + text = ( + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>a.b.c:2" + "<|tool_call_argument_begin|>" + "{}" + "<|tool_call_end|>" + "<|tool_calls_section_end|>" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + assert result[0]["function"]["name"] == "a.b.c" + + def test_kimi_dotted_mcp_name_with_functions_prefix(self): + # ``functions.mcp.server-list:0`` must resolve to ``mcp.server-list`` + # (only the ``functions.`` prefix and ``:idx`` are removed). + text = ( + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>functions.mcp.server-list:0" + "<|tool_call_argument_begin|>" + "{}" + "<|tool_call_end|>" + "<|tool_calls_section_end|>" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + assert result[0]["function"]["name"] == "mcp.server-list" + + def test_kimi_multi_call_recovers_when_first_end_marker_missing(self): + # First call omits its <|tool_call_end|>; the second must still parse. + text = ( + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>functions.read_file:0" + "<|tool_call_argument_begin|>" + '{"path":"a"}' + "<|tool_call_begin|>functions.web_search:1" + "<|tool_call_argument_begin|>" + '{"query":"x"}' + "<|tool_call_end|>" + "<|tool_calls_section_end|>" + ) + result = parse_tool_calls_from_text(text) + assert [c["function"]["name"] for c in result] == ["read_file", "web_search"] + + def test_kimi_handles_unclosed_section(self): + # End marker missing -- the parser must still extract the call. + text = ( + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>functions.foo:0" + "<|tool_call_argument_begin|>" + '{"a":1}' + "<|tool_call_end|>" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + assert result[0]["function"]["name"] == "foo" + + def test_kimi_strip_markup(self): + text = ( + "before " + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>functions.x:0" + "<|tool_call_argument_begin|>" + "{}" + "<|tool_call_end|>" + "<|tool_calls_section_end|>" + " after" + ) + assert strip_tool_markup(text, final = True) == "before after" + + def test_kimi_signal_wakes_streaming(self): + text = "<|tool_calls_section_begin|>..." + assert has_tool_signal(text) + + def test_kimi_call_without_section_wrapper(self): + # llama.cpp makes the ``<|tool_calls_section_begin|>`` wrapper optional -- Kimi K2 can emit + # a bare ``<|tool_call_begin|>`` call. + import json as _json + + text = ( + "<|tool_call_begin|>functions.execute_command:0" + "<|tool_call_argument_begin|>" + '{"cmd":"ls"}' + "<|tool_call_end|>" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + assert result[0]["function"]["name"] == "execute_command" + assert _json.loads(result[0]["function"]["arguments"]) == {"cmd": "ls"} + + def test_kimi_malformed_json_recovers_later_calls(self): + # A call with malformed / truncated JSON must not drop the valid calls that follow it in + # the same section (the bad call is skipped, the good one is recovered). + import json as _json + + text = ( + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>functions.a:0" + '<|tool_call_argument_begin|>{"city":"Beijing"' # missing closing brace + "<|tool_call_end|>" + "<|tool_call_begin|>functions.b:1" + '<|tool_call_argument_begin|>{"city":"Shanghai"}' + "<|tool_call_end|>" + "<|tool_calls_section_end|>" + ) + result = parse_tool_calls_from_text(text) + assert len(result) == 1 + assert result[0]["function"]["name"] == "b" + assert _json.loads(result[0]["function"]["arguments"]) == {"city": "Shanghai"} + + +class TestParserCrossFormatRouting: + """Ensure the per-format dispatch order doesn't misroute any + family. Real emissions for each new family + every old family + must still parse correctly when intermixed.""" + + def test_dispatch_routes_each_family_correctly(self): + cases = [ + ( + "Qwen", + '{"name":"a","arguments":{"x":1}}', + "a", + ), + ( + "DeepSeek V3.1", + "<｜tool▁calls▁begin｜>" + "<｜tool▁call▁begin｜>get_time" + "<｜tool▁sep｜>" + '{"city":"Tokyo"}' + "<｜tool▁call▁end｜>" + "<｜tool▁calls▁end｜>", + "get_time", + ), + ( + "GLM", + "web_search\n" + "q\nx\n" + "", + "web_search", + ), + ( + "Kimi", + "<|tool_calls_section_begin|>" + "<|tool_call_begin|>functions.add:0" + "<|tool_call_argument_begin|>" + '{"a":1}' + "<|tool_call_end|>" + "<|tool_calls_section_end|>", + "add", + ), + ] + for label, text, expected_name in cases: + result = parse_tool_calls_from_text(text) + assert len(result) == 1, f"{label}: parser missed the call" + assert result[0]["function"]["name"] == expected_name, ( + f"{label}: got {result[0]['function']['name']!r}, " f"expected {expected_name!r}" + ) + + def test_all_new_markers_in_tool_xml_signals(self): + # The safetensors / MLX streaming buffer must wake on every supported emission marker -- + # otherwise the BUFFERING state leaks tool content to the user before parse. + from core.inference.tool_call_parser import TOOL_XML_SIGNALS + for marker in ( + "<｜tool▁calls▁begin｜>", + "<｜tool▁call▁begin｜>", + "<|tool_calls_section_begin|>", + "<|tool_call_begin|>", + ): + assert marker in TOOL_XML_SIGNALS, f"streaming loop would not wake on {marker!r}" + + def test_active_tools_are_passed_to_single_turn_after_render_html_success(): captured_tool_names: list[list[str]] = [] exec_fn = FakeExecuteTool(["Rendered HTML canvas."]) @@ -739,7 +1391,8 @@ def test_active_tools_are_passed_to_single_turn_after_render_html_success(): def test_safety_net_honors_disabled_auto_heal_for_late_incomplete_call(): - # A late unclosed heals only with Auto-Heal on; off, it must not execute. + # A late call caught by the safety net: an unclosed ```` heals only with Auto-Heal on; + # off, the safety net must not pass ``allow_incomplete=True`` and execute a truncated call. prose = "Sure, let me look that up for you right now. " incomplete = '{"name":"web_search","arguments":{"query":"weather in Sydney"}}' @@ -764,7 +1417,9 @@ def test_safety_net_honors_disabled_auto_heal_for_late_incomplete_call(): def test_bare_json_tool_call_is_not_streamed_as_content(): - # Llama-3.2 bare form carries no XML signal: BUFFER until the object closes, never leak the JSON. + # Llama-3.2 ``custom_tools`` bare form ``{"name":..,"parameters":..}`` carries no + # XML signal. The loop must BUFFER it until the object closes and execute it via + # the safety net, never leaking the raw JSON to streaming clients as content. bare = '{"name":"web_search","parameters":{"query":"cats"}}' loop, exec_fn = _make_loop( turns = [[bare], ["Here are the results."]], @@ -779,7 +1434,9 @@ def test_bare_json_tool_call_is_not_streamed_as_content(): def test_ordinary_json_with_name_key_is_shown_not_treated_as_tool_call(): - # Markerless JSON whose "name" is not an enabled tool must be shown, not dropped. + # Markerless JSON whose "name" is not an enabled tool (e.g. a person record + # ``{"name":"Alice",...}``) must be shown as the answer, not misread as a call + # to a disabled tool and dropped. _make_loop enables web_search/python/terminal. answer = '{"name":"Alice","parameters":{"age":30}}' loop, exec_fn = _make_loop(turns = [[answer]], max_tool_iterations = 1) events = _collect_events(loop) @@ -789,7 +1446,8 @@ def test_ordinary_json_with_name_key_is_shown_not_treated_as_tool_call(): def test_bare_json_tool_call_split_across_chunks_is_not_streamed(): - # Same as above but the bare object arrives split mid-key, held across chunks until it balances. + # Same as above but the bare object arrives split mid-key, so the buffer is + # held open across chunks before it balances. loop, exec_fn = _make_loop( turns = [ ['{"name":"web_', 'search","parameters":{"query":"cats"}}'], @@ -804,8 +1462,68 @@ def test_bare_json_tool_call_split_across_chunks_is_not_streamed(): assert not any('"name"' in t or "web_search" in t for t in contents), contents +def test_gemma_wrapperless_call_is_not_streamed_as_content(): + # Gemma 4 wrapper-less ``call:NAME{...}`` has no XML signal; the loop must hold + # it (BUFFERING) and execute it, never streaming the raw call text. + loop, exec_fn = _make_loop( + turns = [["call:web_search{query:cats}"], ["Found."]], + exec_results = ["RESULT"], + max_tool_iterations = 3, + ) + events = _collect_events(loop) + assert exec_fn.calls == [("web_search", {"query": "cats"})], exec_fn.calls + contents = [e["text"] for e in events if e["type"] == "content"] + assert not any("call:web_search" in t for t in contents), contents + + +def test_gemma_wrapperless_call_with_whitespace_is_suppressed_when_streamed(): + # Gemma may emit ``call : NAME{...}`` with whitespace around the colon, split across stream + # chunks. + loop, exec_fn = _make_loop( + turns = [["call", " : ", "web_search", "{query:cats}"], ["Found."]], + exec_results = ["RESULT"], + max_tool_iterations = 3, + ) + events = _collect_events(loop) + assert exec_fn.calls == [("web_search", {"query": "cats"})], exec_fn.calls + contents = [e["text"] for e in events if e["type"] == "content"] + assert not any("call" in t for t in contents), contents + + +def test_long_gemma_tool_name_is_not_streamed_as_content(): + # A tool name longer than the small buffer cap (OpenAI 64 chars, MCP longer) + # must still be held: the ``call:NAME`` prefix keeps buffering until ``{`` + # instead of leaking ``call:longname`` as visible text. + long_name = "mcp__github__list_repository_issues" # 35 chars + turns = iter([list('call:%s{repo:"octo/hello"}' % long_name), ["Done."]]) + + def _gen(_messages): + try: + chunks = next(turns) + except StopIteration: + return + acc = "" + for c in chunks: + acc += c + yield acc + + exec_fn = FakeExecuteTool(["RESULT"]) + loop = run_safetensors_tool_loop( + single_turn = _gen, + messages = [{"role": "user", "content": "hi"}], + tools = [{"type": "function", "function": {"name": long_name}}], + execute_tool = exec_fn, + max_tool_iterations = 3, + ) + events = _collect_events(loop) + assert exec_fn.calls == [(long_name, {"repo": "octo/hello"})], exec_fn.calls + contents = [e["text"] for e in events if e["type"] == "content"] + assert not any("call:" in t for t in contents), contents + + def test_leading_json_answer_is_not_dropped(): - # A leading {...} that is NOT a call must still surface; the hold only delays it. + # A leading ``{...}`` that is NOT a tool call must still surface as content: + # the bare-JSON hold can only ever delay it to end-of-object, never drop it. obj = '{"answer": 42, "note": "done"}' loop, exec_fn = _make_loop( turns = [[obj]], @@ -844,7 +1562,8 @@ def _reprompt_loop(*, auto_heal_tool_calls): def test_reprompt_names_only_active_tools_not_hardcoded(): - # The nudge must name the tools actually enabled, not hardcoded web_search/python. + # The plan-without-action nudge must name the tools actually enabled, never the + # old hardcoded ``web_search``/``python`` (which a restricted set would reject). captured, _events = _reprompt_loop(auto_heal_tool_calls = True) assert len(captured) >= 2, "intent prose should have triggered a re-prompt turn" reprompt = captured[1][-1] @@ -855,7 +1574,8 @@ def test_reprompt_names_only_active_tools_not_hardcoded(): def test_reprompt_suppressed_when_auto_heal_disabled(): - # With Auto-Heal off the nudge stays silent for GGUF parity, so only the initial generation runs. + # With Auto-Heal off the safetensors nudge must stay silent for backend parity + # with the GGUF loop, so only the single initial generation runs. captured, events = _reprompt_loop(auto_heal_tool_calls = False) assert len(captured) == 1, captured contents = [e["text"] for e in events if e["type"] == "content"] @@ -922,7 +1642,8 @@ class TestLoopBasic: assert "Result: 1" in contents[-1]["text"] def test_llama3_python_tag_form(self): - # The loop must recognise Llama-3's <|python_tag|> marker, drain the turn, and execute the call. + # The agentic loop must recognise Llama-3's <|python_tag|> + # marker, drain the rest of the turn, and execute the call. loop, exec_fn = _make_loop( turns = [ [ @@ -940,8 +1661,12 @@ class TestLoopBasic: assert "sunny" in contents[-1]["text"].lower() def test_llama3_bare_json_form_fires_tool(self): - # Llama-3.1/3.2 bare-JSON calls carry no XML signal; the safety-net parse must still fire - # the tool. Regression for the has_tool_signal gate that dropped these. + # Llama-3.1 / 3.2 emit a bare-JSON tool call + # ``{"name":..,"parameters":..}`` with NO XML signal. The loop's + # safety-net parse must still fire the tool instead of treating the + # turn as "planned without calling tools" and re-prompting the model + # into giving up. Regression for the has_tool_signal gate that + # dropped these; GGUF's llama-server parses them natively. loop, exec_fn = _make_loop( turns = [ ['{"name": "web_search", "parameters": {"query": "weather in SF"}}'], @@ -955,7 +1680,7 @@ class TestLoopBasic: assert "sunny" in contents[-1]["text"].lower() def test_mistral_pre_v11_form(self): - # Pre-v11 Mistral emission: [TOOL_CALLS] [{...}]. + # Pre-v11 Mistral emission: ``[TOOL_CALLS] [{...}]``. loop, exec_fn = _make_loop( turns = [ [ @@ -973,7 +1698,7 @@ class TestLoopBasic: assert tool_start["tool_call_id"] == "abc" def test_mistral_v11_form(self): - # v11+ Mistral emission: bare name{json} after the trigger. + # v11+ Mistral emission: bare ``name{json}`` after the trigger. loop, exec_fn = _make_loop( turns = [ ['[TOOL_CALLS]web_search{"query":"hi"}'], @@ -985,7 +1710,7 @@ class TestLoopBasic: assert exec_fn.calls == [("web_search", {"query": "hi"})] def test_gemma4_form(self): - # Gemma 4 emission: <|tool_call>call:NAME{...}. + # Gemma 4 emission: ``<|tool_call>call:NAME{...}``. loop, exec_fn = _make_loop( turns = [ [ @@ -1000,6 +1725,70 @@ class TestLoopBasic: events = _collect_events(loop) assert exec_fn.calls == [("web_search", {"query": "weather"})] + def test_deepseek_v3_1_form(self): + # DeepSeek V3.1 emission inside the agentic loop -- the buffer state machine must wake on + # ``<｜tool▁calls▁begin｜>`` and the parser must extract the V3.1 bare-JSON body. + loop, exec_fn = _make_loop( + turns = [ + [ + "<｜tool▁calls▁begin｜>", + "<｜tool▁call▁begin｜>web_search", + "<｜tool▁sep｜>", + '{"query":"Tokyo weather"}', + "<｜tool▁call▁end｜>", + "<｜tool▁calls▁end｜>", + ], + ["The weather is sunny."], + ], + exec_results = ["Sunny, 22C"], + ) + events = _collect_events(loop) + assert exec_fn.calls == [("web_search", {"query": "Tokyo weather"})] + contents = [e for e in events if e["type"] == "content"] + assert contents and "sunny" in contents[-1]["text"].lower() + + def test_glm_form(self): + # GLM 4.x emission: ``NAME\n...``. + loop, exec_fn = _make_loop( + turns = [ + [ + "web_search\n", + "query\n", + "Tokyo\n", + "", + ], + ["found"], + ], + exec_results = ["..."], + ) + events = _collect_events(loop) + assert exec_fn.calls == [("web_search", {"query": "Tokyo"})] + + def test_kimi_form(self): + # Kimi K2 emission ``<|tool_calls_section_begin|>...``. + loop, exec_fn = _make_loop( + turns = [ + [ + "<|tool_calls_section_begin|>", + "<|tool_call_begin|>functions.web_search:0", + "<|tool_call_argument_begin|>", + '{"query":"Tokyo"}', + "<|tool_call_end|>", + "<|tool_calls_section_end|>", + ], + ["done"], + ], + exec_results = ["..."], + ) + events = _collect_events(loop) + # The bare name must reach execute_tool, even though the model + # emitted ``functions.web_search:0`` as the formatted id. + assert exec_fn.calls == [("web_search", {"query": "Tokyo"})] + # tool_start carries the original full id so the conversation + # roundtrip can replay it verbatim. + tool_start = next(e for e in events if e["type"] == "tool_start") + assert tool_start["tool_call_id"] == "functions.web_search:0" + def test_render_html_emits_provisional_tool_start(self): exec_fn = FakeExecuteTool(["Rendered HTML canvas."]) turn_iter = iter( @@ -1360,8 +2149,12 @@ class TestLoopBehaviour: assert captured_tool_names[2] == ["web_search", "python"] def test_duplicate_noop_does_not_consume_budget_at_small_cap(self): - # A duplicate no-op turn must NOT spend the tool budget: only turns that execute a tool - # count (GGUF parity), so a distinct call can still follow at max_tool_iterations=2. + # A duplicate/disabled no-op turn is a correction turn and must NOT spend the + # caller's tool budget, so with max_tool_iterations=2 the model can still make a + # DISTINCT valid call after repeating one. Only turns that actually execute a + # tool count -- matching the GGUF loop. (The budget used to be charged per + # non-re-prompt iteration, so the duplicate burned the second slot and the third + # turn was sent with no tools, dropping the ``python`` call.) captured_tool_names: list[list[str]] = [] turns = iter( [ @@ -1657,7 +2450,8 @@ class TestLoopRePrompt: assert contents and contents[-1]["text"].strip() == "4" def test_max_reprompts_capped_at_three(self): - # Model keeps stalling with intent -- after 3 re-prompts the loop must give up. + # Model keeps stalling with intent -- after 3 re-prompts the + # loop must give up rather than burn forever. turns = [["Let me search for that."]] * 6 # well over the cap loop, exec_fn = _make_loop( turns = turns, @@ -1670,7 +2464,9 @@ class TestLoopRePrompt: assert statuses and statuses[-1]["text"] == "" def test_short_intent_below_buffer_threshold_triggers_reprompt(self): - # Short emission that never exits BUFFERING must still trigger the intent re-prompt. + # Short emission that never exits BUFFERING (< 32 chars + no + # marker prefix). The unified buffer-end path must still + # trigger the intent re-prompt, not silently terminate. loop, exec_fn = _make_loop( turns = [ ["Let me check."], @@ -1683,7 +2479,9 @@ class TestLoopRePrompt: assert exec_fn.calls == [("web_search", {"query": "x"})] def test_reprompt_does_not_consume_tool_budget(self): - # max_tool_iterations=1: the re-prompt must not eat the slot, so the real call still runs. + # max_tool_iterations=1: one re-prompt, then one real tool call, + # then the budget-exhausted final answer must still fire. If the + # re-prompt ate the slot the tool call would never run. loop, exec_fn = _make_loop( turns = [ # 1. Intent stall (re-prompt 1/3). @@ -1714,7 +2512,8 @@ class TestLoopCanonicalHealKey: exec_results = ["1\n"], ) events = _collect_events(loop) - # The bare string must heal to {"code": ...}, not {"query": ...}, so the python sandbox runs it. + # The bare string must heal to {"code": "print(1)"}, not + # {"query": ...}, so the python sandbox actually executes it. assert exec_fn.calls == [("python", {"code": "print(1)"})] def test_terminal_bare_string_heals_to_command(self): @@ -1744,7 +2543,10 @@ class TestGGUFSafetensorsHealingParity: """Pin GGUF vs safetensors/MLX loop parity so a regression on either side breaks CI.""" def test_gguf_imports_shared_signal_markers(self): - # The GGUF BUFFERING machine must wake on every shared emission marker, else calls slip past as prose. + # The GGUF BUFFERING state machine must wake on every emission + # marker the shared parser knows -- otherwise Llama-3 / Mistral + # / Gemma 4 emissions slip past as plain prose when the + # llama-server structured channel fails. import inspect from core.inference.llama_cpp import LlamaCppBackend @@ -1756,7 +2558,10 @@ class TestGGUFSafetensorsHealingParity: ) def test_gguf_uses_shared_strip_helper(self): - # The GGUF stream-cleanup must delegate to the shared strip_tool_markup for every family. + # The GGUF stream-cleanup function must delegate to the shared + # strip_tool_markup so closed-pair markup is removed for every + # emission family (Llama-3 <|python_tag|>, Mistral [TOOL_CALLS], + # Gemma 4 <|tool_call>...). import inspect from core.inference.llama_cpp import LlamaCppBackend @@ -1767,7 +2572,11 @@ class TestGGUFSafetensorsHealingParity: ), "GGUF stream cleanup must delegate to the shared strip_tool_markup helper" def test_gguf_uses_canonical_heal_keys(self): - # GGUF and safetensors heal a bare-string argument to the same canonical key via the shared coerce_tool_arguments. + # GGUF and safetensors heal a bare-string ``arguments`` to the same + # per-tool canonical key -- ``code`` for python, ``command`` for + # terminal, ``query`` for everything else. The mapping is centralised in + # the shared ToolLoopController (both backends route bare-string args + # through ``coerce_tool_arguments``), so the two paths cannot drift. from core.inference.tool_loop_controller import ( _CANONICAL_HEAL_ARG, coerce_tool_arguments, @@ -1786,7 +2595,9 @@ class TestGGUFSafetensorsHealingParity: } def test_intent_regex_matches_same_phrases_as_gguf(self): - # The intent re-prompt regex must match the SAME phrases on both backends. + # The intent re-prompt regex must match the SAME forward-looking + # phrases on both backends so behaviour is the same on Mac (MLX + # / safetensors) and on Linux (GGUF). from core.inference.llama_cpp import _INTENT_SIGNAL as gguf_re from core.inference.safetensors_agentic import ( _INTENT_SIGNAL as sf_re, @@ -1811,7 +2622,8 @@ class TestGGUFSafetensorsHealingParity: "I can help with that.", "I should mention", "Let's go.", - # Negated intent is a refusal, not a plan: neither backend may re-prompt on it. + # Negated intent is a refusal, not a plan: neither backend may + # force a tool-call re-prompt on it. "I will not search the web for that.", "I'll never call that tool.", ): @@ -2240,6 +3052,28 @@ class TestGuardrails: and event.get("type") in {"tool_start", "tool_end"} ] + def test_same_turn_distinct_calls_are_capped(self): + # >_MAX_TOOL_CALLS_PER_TURN DISTINCT calls in one turn must be capped so a runaway turn + # cannot fan out into many executions (the GGUF path is held back by llama-server's lazy ... + from core.inference.safetensors_agentic import _MAX_TOOL_CALLS_PER_TURN + + n = _MAX_TOOL_CALLS_PER_TURN + 4 + turn = "".join( + '{"name":"web_search","arguments":{"query":"q%d"}}' % i + for i in range(n) + ) + loop, exec_fn = _make_loop( + turns = [[turn], ["final"]], + exec_results = ["r"] * n, + max_tool_iterations = 2, + ) + _collect_events(loop) + assert len(exec_fn.calls) == _MAX_TOOL_CALLS_PER_TURN + # The first N distinct queries executed, in document order. + assert [a["query"] for _name, a in exec_fn.calls] == [ + "q%d" % i for i in range(_MAX_TOOL_CALLS_PER_TURN) + ] + def test_coerce_string_args_python_uses_code_key(self): assert _coerce_arguments("print(1)", heal = True, tool_name = "python") == {"code": "print(1)"} @@ -2283,7 +3117,8 @@ class TestRoutesPythonTagStrip: """``_TOOL_XML_RE`` must consume multi-line code, embedded JSON, and bare ``<`` (earlier ``[^\n<]*`` / ``[^\n]*`` revisions leaked tails); the streaming route-level strip is the regression-prone path.""" def _strip(self, text: str) -> str: - # Import inside the test so a routes-module import error doesn't fail collection. + # Import inside the test so a routes-module import error does + # not blow up the entire test file at collection time. from routes.inference import _strip_tool_xml return _strip_tool_xml(text) @@ -2293,7 +3128,8 @@ class TestRoutesPythonTagStrip: assert self._strip(text) == "" def test_python_tag_with_less_than_in_code(self): - # 5615 regression: a literal < inside code must NOT terminate the strip early. + # 5615 regression: literal ``<`` inside code must NOT terminate + # the strip early. text = '<|python_tag|>python.call(code="if x < 10: pass")' assert self._strip(text) == "" @@ -2303,7 +3139,7 @@ class TestRoutesPythonTagStrip: assert self._strip(text) == "" def test_python_tag_multiline_with_less_than(self): - # Combined: multi-line code AND literal < in code. + # Combined: multi-line code AND literal ``<`` in code. text = ( '<|python_tag|>python.call(code="for i in range(10):\n' " if i < 5:\n" @@ -2312,7 +3148,8 @@ class TestRoutesPythonTagStrip: assert self._strip(text) == "" def test_python_tag_stops_at_eom_sentinel(self): - # Strip stops at the next Llama-3 <| sentinel so trailing assistant content survives. + # Strip stops at the next Llama-3 ``<|`` sentinel so any + # trailing assistant content survives. text = '<|python_tag|>python.call(code="multi\nline")' "<|eom_id|>final answer text" assert self._strip(text) == "<|eom_id|>final answer text" @@ -2326,20 +3163,25 @@ class TestRoutesPythonTagStrip: assert self._strip(text) == "" def test_python_tag_with_eom_then_trailing_python_tag(self): - # Two python_tag emissions back-to-back across a sentinel: both strip independently. + # Two python_tag emissions back-to-back across a sentinel: both + # should strip independently. text = ( '<|python_tag|>brave_search.call(query="a")' "<|eom_id|>" '<|python_tag|>python.call(code="x=1")' ) - # <|eom_id|> between the two strips remains; both python_tag blocks are consumed. + # ``<|eom_id|>`` between the two strips remains; both + # python_tag blocks are fully consumed. assert self._strip(text) == "<|eom_id|>" # Robustness fixes uncovered while validating against vLLM / sglang. class TestParserRobustness: def test_tool_call_json_accepts_parameters_key(self): - # Hermes wrapper using parameters instead of arguments; this path now accepts both keys. + # Hermes wrapper around a Llama-3.2 bare-JSON object that uses + # ``parameters`` instead of ``arguments``. The bare-JSON and + # python_tag paths already accept both keys; this path now does + # too. Was extracting name only and silently dropping the args. import json text = "\n" '{"name": "search", "parameters": {"q": "ramen"}}\n' "" @@ -2349,7 +3191,8 @@ class TestParserRobustness: assert json.loads(result[0]["function"]["arguments"]) == {"q": "ramen"} def test_function_xml_attribute_form(self): - # MiniCPM-5 / MiniMax-M2 attribute syntax: v. + # MiniCPM-5 / MiniMax-M2 attribute syntax: + # ``v``. import json text = '' 'Tokyo' "" @@ -2373,7 +3216,8 @@ class TestParserRobustness: assert args == {"city": "Tokyo", "unit": "celsius"} def test_function_xml_legacy_equals_form_still_works(self): - # Regression guard: the old v syntax must keep parsing after the regex broadening. + # Regression guard: the old ``v`` + # syntax must keep parsing after the regex broadening. import json text = "Tokyo" @@ -2383,17 +3227,24 @@ class TestParserRobustness: assert json.loads(result[0]["function"]["arguments"]) == {"city": "Tokyo"} def test_function_attribute_form_has_tool_signal(self): - # The standalone form must flip the streaming buffer, else the call is dropped. + # The standalone ```` attribute form must flip + # the streaming buffer; otherwise the end-of-turn safety-net parse in + # the agentic loop is gated off and the real call is dropped. assert has_tool_signal('') is True def test_function_attribute_form_strip_markup(self): - # The attribute form must also be stripped from displayed text, like . + # The attribute form must also be stripped from displayed text, like + # the legacy ```` form. text = 'result X' assert strip_tool_markup(text, final = True) == "result" def test_llama3_chat_template_round_trip(self): - # Llama-3.x prefixes assistant turns with <|start_header_id|>...<|end_header_id|>; the - # sentinel-strip must reach past the role label to the JSON body, else history calls drop. + # Meta's official Llama-3.x chat template prefixes every + # assistant turn with + # ``<|start_header_id|>assistant<|end_header_id|>\n\n``. The + # sentinel-strip in ``_parse_llama3_bare_json`` must reach past + # the role label to the JSON body, else every round-tripped + # tool call in history silently drops. import json text = ( @@ -2418,7 +3269,8 @@ class TestParserRobustness: assert json.loads(result[0]["function"]["arguments"]) == {"x": 1} def test_llama3_round_trip_with_eot_prefix(self): - # Prior turn closes with <|eot_id|>, then the new header opens; both sentinels + role must be consumed. + # Prior assistant turn closes with ``<|eot_id|>``, then the + # new header opens. Both sentinels + the role must be consumed. import json text = ( @@ -2430,7 +3282,10 @@ class TestParserRobustness: assert result[0]["function"]["name"] == "f" def test_function_xml_followed_by_prose(self): - # Body must terminate at even without a wrapper, else prose leaks into the value. + # Models routinely follow a tool call with explanatory prose. + # Body must terminate at ```` even without a + # ```` wrapper, else trailing prose leaks into the + # last parameter value. import json text = ( @@ -2456,8 +3311,236 @@ class TestParserRobustness: assert json.loads(result[0]["function"]["arguments"]) == {"city": "Tokyo"} +def test_render_with_native_template_returns_render_only_when_tools_emitted(): + # The native-template fallback re-renders with the model's repo template when an override drops + # the tools schema. + from types import SimpleNamespace + + from core.inference.chat_template_helpers import render_native_template + + messages = [{"role": "user", "content": "hi"}] + tools = [{"type": "function", "function": {"name": "web_search"}}] + model_info = { + "native_chat_template": "TPL", + "tokenizer": SimpleNamespace(chat_template = "OVERRIDE"), + } + + def emitting(tokenizer, msgs, *, tools, **_kw): + body = "".join(m["content"] for m in msgs) + return body + ("|TOOLS=" + ",".join(t["function"]["name"] for t in tools) if tools else "") + + def ignoring(tokenizer, msgs, *, tools, **_kw): + return "".join(m["content"] for m in msgs) # never reflects tools + + out = render_native_template( + model_info = dict(model_info), + active_model_name = "x", + messages = messages, + tools = tools, + apply_fn = emitting, + ) + assert out == "hi|TOOLS=web_search" + # The native template must be restored on the live tokenizer after probing. + assert model_info["tokenizer"].chat_template == "OVERRIDE" + + assert ( + render_native_template( + model_info = dict(model_info), + active_model_name = "x", + messages = messages, + tools = tools, + apply_fn = ignoring, + ) + is None + ) + + # No tokenizer and no processor -> return None instead of an AttributeError. + no_tok = {"native_chat_template": "TPL"} + assert ( + render_native_template( + model_info = no_tok, + active_model_name = "x", + messages = messages, + tools = tools, + apply_fn = emitting, + ) + is None + ) + + +def test_render_with_native_template_does_not_mutate_shared_tokenizer(): + # The shared tokenizer must never carry the temporary native template, even mid-render: this + # runs outside the generation lock, so a concurrent request could otherwise render with the ... + from types import SimpleNamespace + + from core.inference.chat_template_helpers import render_native_template + + shared = SimpleNamespace(chat_template = "OVERRIDE") + seen = [] + + def capture(tokenizer, msgs, *, tools, **_kw): + seen.append((tokenizer is shared, shared.chat_template)) + body = "".join(m["content"] for m in msgs) + return body + ("|T" if tools else "") + + model_info = {"native_chat_template": "TPL", "tokenizer": shared} + render_native_template( + model_info = model_info, + active_model_name = "x", + messages = [{"role": "user", "content": "hi"}], + tools = [{"type": "function", "function": {"name": "web_search"}}], + apply_fn = capture, + ) + # Rendering happened on a copy, and the shared tokenizer stayed "OVERRIDE" + # throughout (never the temporary "TPL"). + assert seen and all(not is_shared for is_shared, _ in seen) + assert all(tpl == "OVERRIDE" for _, tpl in seen) + assert shared.chat_template == "OVERRIDE" + + +def test_native_template_loads_from_base_model_for_lora(monkeypatch): + # For a LoRA adapter the chat template lives on the base model; active_model_name + # is the adapter id and may ship no template. The loader must read base_model. + from types import SimpleNamespace + + import transformers + + from core.inference.chat_template_helpers import render_native_template + + captured = {} + + def fake_from_pretrained(name, *args, **kwargs): + captured["source"] = name + return SimpleNamespace(chat_template = "BASE_TPL") + + monkeypatch.setattr(transformers.AutoTokenizer, "from_pretrained", fake_from_pretrained) + + def emitting(tokenizer, msgs, *, tools, **_kw): + body = "".join(m["content"] for m in msgs) + return body + ("|T" if tools else "") + + model_info = { + "base_model": "base/model-id", + "tokenizer": SimpleNamespace(chat_template = "OVERRIDE"), + } + out = render_native_template( + model_info = model_info, + active_model_name = "adapter/path", + messages = [{"role": "user", "content": "hi"}], + tools = [{"type": "function", "function": {"name": "web_search"}}], + apply_fn = emitting, + ) + assert captured["source"] == "base/model-id" + assert out == "hi|T" + + +def test_render_with_native_template_fallback_swaps_when_override_drops_tools(): + # The shared gate (used by the transformers and MLX backends): when the live render is + # identical with and without tools, re-render with the native template and return it. + from types import SimpleNamespace + + from core.inference.chat_template_helpers import render_with_native_template_fallback + + messages = [{"role": "user", "content": "hi"}] + tools = [{"type": "function", "function": {"name": "web_search"}}] + + # apply_fn that IGNORES tools -> live render drops the schema. + def ignoring(tokenizer, msgs, *, tools, **_kw): + return "".join(m["content"] for m in msgs) + + model_info = { + "native_chat_template": "TPL", + "tokenizer": SimpleNamespace(chat_template = "OVERRIDE"), + } + + # Native render emits the tools, so the fallback swaps to it. + def native_emits(tokenizer, msgs, *, tools, **_kw): + body = "".join(m["content"] for m in msgs) + return body + ("|TOOLS" if tools else "") + + out = render_with_native_template_fallback( + formatted_prompt = ignoring(None, messages, tools = tools), + tokenizer = SimpleNamespace(), + model_info = dict(model_info), + active_model_name = "x", + messages = messages, + tools = tools, + apply_fn = lambda tok, msgs, *, tools, **kw: ( + native_emits(tok, msgs, tools = tools) + if getattr(tok, "chat_template", None) == "TPL" + else ignoring(tok, msgs, tools = tools) + ), + ) + assert out == "hi|TOOLS", out + + +def test_render_with_native_template_fallback_keeps_prompt_when_tools_emitted(): + # Live render already differs with vs without tools -> no fallback, returned + # unchanged. Also a no-tools call is a passthrough. + from types import SimpleNamespace + + from core.inference.chat_template_helpers import render_with_native_template_fallback + + messages = [{"role": "user", "content": "hi"}] + tools = [{"type": "function", "function": {"name": "web_search"}}] + + def emitting(tokenizer, msgs, *, tools, **_kw): + body = "".join(m["content"] for m in msgs) + return body + ("|T" if tools else "") + + kept = render_with_native_template_fallback( + formatted_prompt = emitting(None, messages, tools = tools), + tokenizer = SimpleNamespace(), + model_info = {"native_chat_template": "TPL", "tokenizer": SimpleNamespace()}, + active_model_name = "x", + messages = messages, + tools = tools, + apply_fn = emitting, + ) + assert kept == "hi|T", kept + + # No tools -> passthrough (native template never consulted). + passthrough = render_with_native_template_fallback( + formatted_prompt = "hi", + tokenizer = SimpleNamespace(), + model_info = {}, + active_model_name = "x", + messages = messages, + tools = None, + apply_fn = emitting, + ) + assert passthrough == "hi" + + +def test_render_with_native_template_fallback_keeps_prompt_when_no_tools_probe_raises(): + # A template that REQUIRES tools can raise on the no-tools probe. + from types import SimpleNamespace + + from core.inference.chat_template_helpers import render_with_native_template_fallback + + messages = [{"role": "user", "content": "hi"}] + tools = [{"type": "function", "function": {"name": "web_search"}}] + + def raises_without_tools(tokenizer, msgs, *, tools, **_kw): + if not tools: + raise RuntimeError("template requires tools") + return "".join(m["content"] for m in msgs) + "|T" + + out = render_with_native_template_fallback( + formatted_prompt = "hi|T", + tokenizer = SimpleNamespace(), + model_info = {"native_chat_template": "TPL", "tokenizer": SimpleNamespace()}, + active_model_name = "x", + messages = messages, + tools = tools, + apply_fn = raises_without_tools, + ) + assert out == "hi|T", out + + def test_truncated_bare_json_at_eof_is_not_leaked(): - # Stream ends mid bare-JSON: the held fragment must be dropped at EOF, not flushed as content. + # Stream ends mid bare-JSON object: the held fragment must be dropped at the + # EOF resolver, not flushed as plain assistant content (GGUF parity). loop, _exec = _make_loop( turns = [['{"name":"web_search","parameters":{"query":"weather in S']], max_tool_iterations = 1, @@ -2468,7 +3551,9 @@ def test_truncated_bare_json_at_eof_is_not_leaked(): def test_oversized_bare_json_call_is_not_leaked_and_executes(): - # A bare-JSON call exceeding _MAX_BARE_JSON_BUFFER must DRAIN, not stream the prefix, and still execute. + # A bare-JSON call whose arguments exceed _MAX_BARE_JSON_BUFFER must DRAIN + # (suppress) rather than stream the raw JSON prefix, and still execute once + # the full object is parsed by the safety net. from core.inference.safetensors_agentic import _MAX_BARE_JSON_BUFFER big = "A" * (_MAX_BARE_JSON_BUFFER + 5000) @@ -2483,7 +3568,8 @@ def test_oversized_bare_json_call_is_not_leaked_and_executes(): def test_oversized_plain_json_answer_still_streams(): - # A giant plain JSON answer (no "name" key) is NOT a call and must still stream. + # A giant plain JSON answer (no "name" key) is NOT a tool call and must still + # stream -- the oversized DRAIN route is gated on a "name" key. from core.inference.safetensors_agentic import _MAX_BARE_JSON_BUFFER big = "A" * (_MAX_BARE_JSON_BUFFER + 5000) @@ -2496,7 +3582,9 @@ def test_oversized_plain_json_answer_still_streams(): def test_oversized_disabled_name_json_answer_still_streams(): - # A giant still-open JSON answer whose "name" is NOT an enabled tool must stream, not drain. + # A giant still-open JSON answer whose "name" is NOT an enabled tool must stream: + # the oversized DRAIN branch was gated only on the presence of a "name" key, so a + # large ordinary record ({"name":"Alice",...}) was drained instead of shown. from core.inference.safetensors_agentic import _MAX_BARE_JSON_BUFFER big = "A" * (_MAX_BARE_JSON_BUFFER + 5000) @@ -2510,7 +3598,8 @@ def test_oversized_disabled_name_json_answer_still_streams(): def test_truncated_disabled_name_json_is_shown_at_eof(): - # A truncated JSON answer whose name is not an enabled tool must be shown at EOF. + # A truncated ordinary JSON answer whose name is not an enabled tool, held to EOF, + # must be shown -- the EOF bare-JSON DRAIN branch was gated only on a "name" key. truncated = '{"name":"Alice","parameters":{"age":' loop, exec_fn = _make_loop(turns = [[truncated]], max_tool_iterations = 1) events = _collect_events(loop) @@ -2520,7 +3609,9 @@ def test_truncated_disabled_name_json_is_shown_at_eof(): def test_truncated_plain_json_with_nested_enabled_name_is_visible(): - # A truncated answer with only a NESTED "name" must be shown: the gate uses the TOP-LEVEL name. + # A truncated ordinary JSON answer with a NESTED ``"name"`` matching an enabled + # tool ({"result":{"name":"web_search",...) must be shown, not suppressed: the + # gate now extracts the TOP-LEVEL name only, so the nested field is just data. loop, exec_fn = _make_loop( turns = [['{"result":{"name":"web_search","age":']], max_tool_iterations = 1, @@ -2532,7 +3623,8 @@ def test_truncated_plain_json_with_nested_enabled_name_is_visible(): def test_bare_json_call_not_replayed_in_next_turn_content(): - # After a bare-JSON call executes, the next-turn assistant content must not contain the raw call. + # After a complete bare-JSON call executes, the assistant content fed to the + # next turn must not contain the raw call (next-turn contamination). captured: list[list[dict]] = [] exec_fn = FakeExecuteTool(["RESULT"]) @@ -2562,7 +3654,10 @@ if __name__ == "__main__": def test_drain_truncated_enabled_name_json_preserved_when_auto_heal_disabled(): - # With Auto-Heal OFF a truncated enabled-name bare-JSON fragment stays visible; with it ON, suppressed. + # F3: with Auto-Heal OFF, a truncated ENABLED-name bare-JSON fragment that did + # not parse must stay visible (disabled-Auto-Heal contract: malformed markup is + # preserved), matching the XML strip in the same drain branch. With Auto-Heal ON + # the same fragment is suppressed. trunc = '{"name":"web_search","parameters":{"query":"weather' off, exec_off = _make_loop(turns = [[trunc]], max_tool_iterations = 1, auto_heal_tool_calls = False) events_off = _collect_events(off) @@ -2578,7 +3673,9 @@ def test_drain_truncated_enabled_name_json_preserved_when_auto_heal_disabled(): def test_looks_like_enabled_bare_json_accepts_function_alias(): - # The buffering gate must recognise the "function" bare-JSON alias, so it is buffered, not streamed. + # The safetensors buffering gate must recognise the "function" bare-JSON alias + # the parser accepts, so a truncated/complete {"function":} call is + # buffered/healed instead of streaming as visible content. from core.inference.safetensors_agentic import _looks_like_enabled_bare_json enabled = {"web_search"} @@ -2591,7 +3688,8 @@ def test_looks_like_enabled_bare_json_accepts_function_alias(): class TestFalseAlarmMarkerProse: def test_leading_marker_prose_streams_intact(self): - # An answer starting with a literal marker is a false alarm: the full prose must reach the client. + # An answer that starts with a literal marker is a false alarm: the + # drain finds no calls and the full prose must reach the client. text = "[TOOL_CALLS] is the Mistral tool marker. More prose after." loop, exec_fn = _make_loop(turns = [[text]]) events = _collect_events(loop) @@ -2600,7 +3698,8 @@ class TestFalseAlarmMarkerProse: assert texts and texts[-1] == text def test_chained_bare_json_calls_not_replayed_in_history(self): - # Both chained calls execute; the next-turn history must not contain the second call's raw JSON. + # Both chained calls execute; the kept content (next-turn assistant + # history) must not contain the second call's raw JSON. chained = ( '{"name":"web_search","parameters":{"q":"first"}};' '{"name":"python","parameters":{"code":"x"}}' diff --git a/studio/backend/tests/test_tool_call_parser_strict.py b/studio/backend/tests/test_tool_call_parser_strict.py index fded2a844..7f47140b8 100644 --- a/studio/backend/tests/test_tool_call_parser_strict.py +++ b/studio/backend/tests/test_tool_call_parser_strict.py @@ -72,10 +72,8 @@ class TestFunctionStyleTrailingText: assert call == {"name": "python", "arguments": {"code": 'print("")'}} def test_closed_function_with_trailing_prose_heal_path(self): - # Regression: the heal / finalize path (allow_incomplete=True) used to fold - # and the trailing prose into the argument and drop - # the prose from visible content. It must now match the strict path -- keep a - # clean argument and leave the trailing prose outside the call span. + # Regression: the heal path (allow_incomplete=True) must match the strict path -- + # keep a clean argument and leave trailing prose outside the call span. text = "cats trailing words" calls = parse_tool_calls_from_text(text, allow_incomplete = True) assert len(calls) == 1 @@ -103,7 +101,8 @@ class TestFunctionStyleTrailingText: assert parse_tool_calls_from_text(text, allow_incomplete = False) == [] def test_attribute_form_literal_close_tag_is_preserved(self): - # Attribute form ends at the LAST , so a literal close inside code survives. + # The attribute form (MiniCPM-5 / MiniMax-M2) also ends at the + # LAST , so a literal close tag inside a code argument survives. text = ( '' 'print("")' @@ -113,7 +112,8 @@ class TestFunctionStyleTrailingText: assert call == {"name": "python", "arguments": {"code": 'print("")'}} def test_closed_zero_param_attribute_call_is_accepted_in_strict_mode(self): - # A closed zero-param call is valid; strict mode must not treat it as truncated. + # A closed call with no parameters is a valid zero-argument call; strict + # mode must not treat the empty parameter list as a truncated call. assert _only('') == {"name": "ping", "arguments": {}} # A no-arg call that never closes is still rejected as truncated. assert parse_tool_calls_from_text('', allow_incomplete = False) == [] @@ -231,9 +231,8 @@ class TestHealingPathUnaffected: assert calls[0]["function"]["name"] == "web_search" def test_closed_function_call_keeps_trailing_prose_out_of_arguments(self): - # allow_incomplete exists for truncated output; a call that DID close - # must parse identically to strict mode, leaving prose after - # out of the last parameter and out of the removal span. + # A call that DID close must parse identically to strict mode, leaving prose after + # out of the last parameter and the removal span. from core.tool_healing import parse_tool_calls_from_text as parse_with_spans text = "cats trailing" @@ -246,7 +245,8 @@ class TestHealingPathUnaffected: ) def test_wrapperless_fallback_calls_carry_spans(self): - # The wrapperless fallback must report spans so consumers strip exactly the markup. + # The wrapperless function-XML fallback must report spans too, so with_spans + # consumers strip exactly the promoted markup (through when closed). from core.tool_healing import parse_tool_calls_from_text as parse_with_spans closed = "before cats after" @@ -266,6 +266,50 @@ class TestHealingPathUnaffected: assert healed[span[0] : span[1]] == "dogs" +class TestGlmStrict: + def test_closed_glm_call_is_accepted(self): + text = ( + "get_weather\n" + "city\nParis\n" + "" + ) + calls = parse_tool_calls_from_text(text, allow_incomplete = False) + assert len(calls) == 1 + assert calls[0]["function"]["name"] == "get_weather" + + def test_unclosed_glm_call_is_rejected(self): + # No close: truncated, reject with Auto-Heal off. + text = "get_weather\ncity\nParis" + assert parse_tool_calls_from_text(text, allow_incomplete = False) == [] + assert len(parse_tool_calls_from_text(text, allow_incomplete = True)) == 1 + + +class TestKimiStrict: + _SB = "<|tool_calls_section_begin|>" + _KB = "<|tool_call_begin|>" + _AB = "<|tool_call_argument_begin|>" + _KE = "<|tool_call_end|>" + _SE = "<|tool_calls_section_end|>" + + def test_full_kimi_call_is_accepted(self): + text = self._SB + self._KB + "functions.x:0" + self._AB + '{"a":1}' + self._KE + self._SE + calls = parse_tool_calls_from_text(text, allow_incomplete = False) + assert len(calls) == 1 + assert calls[0]["function"]["name"] == "x" + + def test_kimi_call_without_call_end_is_rejected(self): + # Section closed but the call lacks <|tool_call_end|>: reject in strict. + text = self._SB + self._KB + "functions.x:0" + self._AB + '{"a":1}' + self._SE + assert parse_tool_calls_from_text(text, allow_incomplete = False) == [] + assert len(parse_tool_calls_from_text(text, allow_incomplete = True)) == 1 + + def test_kimi_without_section_end_is_rejected(self): + # No <|tool_calls_section_end|>: truncated section, reject in strict. + text = self._SB + self._KB + "functions.x:0" + self._AB + '{"a":1}' + self._KE + assert parse_tool_calls_from_text(text, allow_incomplete = False) == [] + assert len(parse_tool_calls_from_text(text, allow_incomplete = True)) == 1 + + class TestParserLinearity: """Llama-3 ``.call`` kwargs and Mistral-array healing must stay linear (a regex-per-offset blew up on long truncated bodies).""" @@ -293,6 +337,27 @@ class TestParserLinearity: parse_tool_calls_from_text(text, allow_incomplete = True) assert time.perf_counter() - t0 < 2.0 + def test_gemma_wrapperless_deep_nesting_is_linear(self): + # Wrapper-less Gemma ``call:f{a:{a:{...}}}`` deep nesting must parse in linear time (no quadratic re-scan). + import time + + def nested(d): + return "call:f{a:" + "{a:" * d + "x:1" + "}" * d + "}" + + def best_ms(depth): + text = nested(depth) + best = float("inf") + for _ in range(5): + t0 = time.perf_counter() + calls = parse_tool_calls_from_text(text) + best = min(best, time.perf_counter() - t0) + assert calls and json.loads(calls[0]["function"]["arguments"]), "nested args dropped" + return best + + t200 = best_ms(200) + t400 = best_ms(400) + assert t400 < t200 * 3.0, (t200, t400) + def test_llama3_call_kwargs_still_parse(self): text = '<|python_tag|>do.call(s="hi 😀", n=42, f=1.5, b=true, z=null)' calls = parse_tool_calls_from_text(text, allow_incomplete = True) @@ -334,7 +399,8 @@ class TestLlamaBuiltinChainAndNesting: assert json.loads(calls[1]["function"]["arguments"]) == {"y": 2} def test_nested_python_tag_in_json_string_arg_is_not_a_call(self): - # A <|python_tag|> literal inside a code arg is data: the outer "python" call wins. + # A code arg literally containing a <|python_tag|>...call(...) string: the real call is the + # outer "python", not the nested "os" -- the scan stays anchored to the first tag. text = ( '<|python_tag|>{"name":"python","parameters":' '{"code":"<|python_tag|>os.call(\'rm -rf /\')"}}' @@ -353,6 +419,41 @@ class TestLlamaBuiltinChainAndNesting: assert json.loads(calls[0]["function"]["arguments"]) == {"query": "cats"} +def test_glm_open_does_not_parse_spaced_prose_as_tool_name(): + # The GLM NAME opener must reject spaced literal prose (V10); only a + # valid [\w.\-]+ name (followed by newline//) is a call. + assert parse_tool_calls_from_text("not a call") == [] + ok = parse_tool_calls_from_text( + "get_weather\ncity\nNYC\n" + ) + assert [c["function"]["name"] for c in ok] == ["get_weather"] + + +def test_deepseek_r1_missing_call_terminator_rejected_in_strict_mode(): + # R1 must reject a fenced call whose closing ``` + <｜tool▁call▁end｜> never + # arrived when Auto-Heal is off, matching V3/V3.1 strictness (V6). + text = ( + "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n" + "```json\n" + '{"city":"NYC"}' + "<｜tool▁calls▁end｜>" + ) + assert parse_tool_calls_from_text(text, allow_incomplete = False) == [] + assert len(parse_tool_calls_from_text(text, allow_incomplete = True)) == 1 + + +def test_deepseek_r1_complete_call_accepted_in_strict_mode(): + # A fully-terminated R1 call (close fence + per-call end) is still accepted. + text = ( + "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n" + "```json\n" + '{"city":"NYC"}\n' + "```<｜tool▁call▁end｜><｜tool▁calls▁end｜>" + ) + calls = parse_tool_calls_from_text(text, allow_incomplete = False) + assert len(calls) == 1 and calls[0]["function"]["name"] == "get_weather" + + def test_strip_leading_bare_json_call_drops_complete_call(): from core.inference.tool_call_parser import strip_leading_bare_json_call @@ -386,6 +487,57 @@ def test_strip_leading_bare_json_call_preserves_plain_json_and_prose(): assert strip_leading_bare_json_call("just a sentence.") == "just a sentence." +def test_glm_literal_close_tag_in_string_arg_not_truncated(): + import json + + from core.inference.tool_call_parser import parse_tool_calls_from_text + + # A GLM string argument may legitimately contain the literal close tag ````. + text = ( + "run_code\n" + "code\n" + 'print("")\n' + "" + ) + calls = parse_tool_calls_from_text(text, allow_incomplete = True) + assert len(calls) == 1 + args = json.loads(calls[0]["function"]["arguments"]) + assert args["code"] == 'print("")', args + + +def test_glm_truncated_block_rejected_in_strict_mode_but_healed_otherwise(): + from core.inference.tool_call_parser import parse_tool_calls_from_text + + # No close: strict mode (Auto-Heal off) rejects the truncated + # block; with Auto-Heal it keeps the partial call. + text = "get_weather\ncity\nNYC" + assert parse_tool_calls_from_text(text, allow_incomplete = False) == [] + healed = parse_tool_calls_from_text(text, allow_incomplete = True) + assert len(healed) == 1 and healed[0]["function"]["name"] == "get_weather" + + +def test_truncated_wrapperless_gemma_call_is_stripped(): + from core.inference.tool_call_parser import strip_tool_markup + + # A wrapper-less Gemma ``call:NAME{...`` cut off mid-arguments (no closing + # brace) must not leak the raw call into the visible stream. + text = 'Sure!\ncall:web_search{"query": "weather in San Fr' + stripped = strip_tool_markup(text, final = True) + assert "call:web_search" not in stripped, repr(stripped) + assert stripped.strip() == "Sure!" + + +def test_complete_wrapperless_gemma_call_keeps_trailing_prose(): + from core.inference.tool_call_parser import strip_tool_markup + + # The truncation pattern must run AFTER the closed form, so a complete call + # followed by prose keeps the prose instead of eating to EOS. + text = 'call:web_search{"query": "cats"} Here you go.' + stripped = strip_tool_markup(text, final = True) + assert "call:web_search" not in stripped + assert stripped.strip() == "Here you go." + + def test_bare_json_gated_on_enabled_tool_names(): from core.inference.tool_call_parser import parse_tool_calls_from_text @@ -421,7 +573,8 @@ def test_strip_leading_bare_json_call_gated_on_enabled_tool_names(): def test_function_xml_strip_keeps_literal_close_tag_in_param_value(): from core.inference.tool_call_parser import strip_tool_markup - # Strip uses the LAST so a literal in a value survives; calls strip independently. + # The strip uses the LAST (like the parser) so a literal in a value doesn't + # truncate it; separate calls still strip independently. text = 'print("") done' assert strip_tool_markup(text, final = True) == "done" two = ( @@ -434,7 +587,8 @@ def test_function_xml_strip_keeps_literal_close_tag_in_param_value(): def test_function_xml_strip_keeps_trailing_text_after_literal_open_tag(): from core.inference.tool_call_parser import parse_tool_calls_from_text, strip_tool_markup - # A literal opener inside a value is data: the strip keeps " done". + # A literal ```` opener inside a parameter value is data, not a call: the scan-based + # strip keeps " done" (the old negative-lookahead regex ate the trailing prose). text = 'print("") done' assert parse_tool_calls_from_text(text)[0]["function"]["name"] == "python" assert strip_tool_markup(text, final = True) == "done" @@ -446,10 +600,11 @@ def test_function_xml_strip_keeps_trailing_text_after_literal_open_tag(): def test_final_strip_removes_magistral_think_reasoning(): from core.inference.tool_call_parser import strip_tool_markup - # Magistral reasoning is [THINK]...[/THINK]; end-of-turn must drop it. + # Magistral emits reasoning as ``[THINK]...[/THINK]`` (bracket form, not ````); + # at end-of-turn it must be dropped so it doesn't leak into display / history. text = "[THINK]The user greeted me, I should say hi.[/THINK]Hello! How can I help?" assert strip_tool_markup(text, final = True) == "Hello! How can I help?" - # A [TOOL_CALLS] living inside the reasoning goes with it. + # A ``[TOOL_CALLS]`` living inside the reasoning goes with it. with_call = '[THINK]Maybe I should search.[/THINK][TOOL_CALLS]search{"q":"x"}' assert strip_tool_markup(with_call, final = True) == "" @@ -457,7 +612,8 @@ def test_final_strip_removes_magistral_think_reasoning(): def test_streaming_strip_keeps_magistral_think_buffered(): from core.inference.tool_call_parser import strip_tool_markup - # Mid-stream (final=False) leaves the reasoning block intact; only end-of-turn removes it. + # Mid-stream (final=False) the reasoning block is left intact; only the + # end-of-turn pass removes it. text = "[THINK]still thinking" assert strip_tool_markup(text, final = False) == text @@ -465,7 +621,7 @@ def test_streaming_strip_keeps_magistral_think_buffered(): def test_final_strip_leaves_non_magistral_bracket_text_untouched(): from core.inference.tool_call_parser import strip_tool_markup - # Only a LEADING [THINK] block is reasoning; unrelated bracketed prose stays. + # Only a LEADING ``[THINK]`` block is reasoning; unrelated bracketed prose stays. text = "See [THINK about it] later" assert strip_tool_markup(text, final = True) == "See [THINK about it] later" @@ -473,7 +629,8 @@ def test_final_strip_leaves_non_magistral_bracket_text_untouched(): def test_strip_leading_bare_json_call_ignores_nested_name(): from core.inference.tool_call_parser import strip_leading_bare_json_call - # A nested "name" must NOT gate the strip; the JSON answer is kept verbatim. + # A nested ``"name"`` must NOT gate the strip (only a TOP-LEVEL enabled name is a call); the + # ordinary JSON answer is kept verbatim, truncated or complete. nested_trunc = '{"result":{"name":"web_search","age":' nested_full = '{"result":{"name":"web_search","age":1}}' assert strip_leading_bare_json_call(nested_trunc, {"web_search"}) == nested_trunc @@ -493,7 +650,8 @@ def test_mistral_single_object_call_is_stripped_for_display(): parse_tool_calls_from_text, ) - # The parser accepts single-object [TOOL_CALLS]{...}, so the strip must remove it too. + # The parser accepts the single-object [TOOL_CALLS]{...} shape, so the display + # strip must remove it too (asymmetry would leak the raw object). text = '[TOOL_CALLS]{"name":"web_search","arguments":{"filters":{"date":"2024"}}} tail' assert [c["function"]["name"] for c in parse_tool_calls_from_text(text)] == ["web_search"] assert _strip_mistral_closed_calls(text) == " tail" @@ -502,7 +660,8 @@ def test_mistral_single_object_call_is_stripped_for_display(): def test_tool_call_parser_declares_future_annotations_for_py39_import(): - # PEP 604 X | None annotations need `from __future__ import annotations` on py3.9; guard it stays. + # F1: the parser is imported standalone on python >=3.9, where its PEP 604 ``X | None`` + # annotations need ``from __future__ import annotations``; guard that the import stays. from pathlib import Path src = ( Path(__file__).resolve().parent.parent / "core" / "inference" / "tool_call_parser.py" @@ -510,8 +669,23 @@ def test_tool_call_parser_declares_future_annotations_for_py39_import(): assert "from __future__ import annotations" in src +def test_glm_strip_treats_literal_close_tag_in_arg_value_as_data(): + # Core strip parity: a literal inside a GLM is argument data, so the whole call is stripped (no leaked tail). + from core.inference.tool_call_parser import strip_tool_markup + + text = ( + "web_search\nquery\n" + "see tag\n tail" + ) + assert strip_tool_markup(text, final = True) == "tail" + calls = parse_tool_calls_from_text(text) + assert [c["function"]["name"] for c in calls] == ["web_search"] + assert json.loads(calls[0]["function"]["arguments"]) == {"query": "see tag"} + + def test_bare_json_function_alias_parses_and_strips_symmetrically(): - # The "function" alias for the call name must parse and strip symmetrically. + # The bare-JSON parser accepts the "function" alias for the call name; + # strip_leading_bare_json_call must recognise it too (parser/strip symmetry). from core.inference.tool_call_parser import ( parse_tool_calls_from_text, strip_leading_bare_json_call, @@ -585,6 +759,92 @@ class TestHealerSignalAlignment: assert not list(healer.finalize()) or all(k == "text" for k, _v in healer.finalize()) +class TestGemmaWrapperlessLiteralMarkers: + """Wrapper-less Gemma calls whose ARGUMENTS mention Gemma's own markup. + + The tool_healing deferral must key on an actual wrapped opener + (``<|tool_call>call:...``), not the wrapper literal anywhere in content: + a query about the marker has nothing tool_healing can parse, and deferring + it loses the call entirely (not executed AND stripped from display).""" + + def test_marker_literal_in_argument_still_parses(self): + text = 'call:web_search{query:"what does <|tool_call> mean"}' + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search"}) + assert len(calls) == 1 + args = json.loads(calls[0]["function"]["arguments"]) + assert args["query"] == "what does <|tool_call> mean" + + def test_real_wrapped_call_still_deferred_to_tool_healing(self): + from core.inference.tool_call_parser import _parse_gemma_tool_calls + + # An actual wrapped opener present: the Gemma fallback must keep + # deferring to the shared tool_healing parser that owns that form. + text = '<|tool_call>call:web_search{query:<|"|>cats<|"|>}' + assert _parse_gemma_tool_calls(text, id_offset = 0) == [] + + def test_single_quoted_brace_does_not_truncate_code(self): + text = "call:python{code:print('}')}" + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"python"}) + assert len(calls) == 1 + args = json.loads(calls[0]["function"]["arguments"]) + assert args["code"] == "print('}')" + + def test_single_quoted_brace_strip_span_covers_whole_call(self): + from core.inference.tool_call_parser import strip_tool_markup + + text = "call:python{code:print('}')} Done." + stripped = strip_tool_markup(text, final = True, enabled_tool_names = {"python"}) + assert "call:python" not in stripped + assert "')}" not in stripped + assert stripped.strip() == "Done." + + +class TestGlmEmbeddedClosePair: + """A GLM value whose string literal embeds the full close-tag pair + ```` (code documenting the GLM format) must not be + truncated at the embedded pair: a structural close sits at balanced quote + state, an embedded one is inside an open string literal.""" + + def test_embedded_pair_inside_quoted_value_not_structural(self): + text = ( + "python\n" + "code\n" + 'print("")\nx = 1\n' + "" + ) + calls = parse_tool_calls_from_text(text, allow_incomplete = True) + assert len(calls) == 1 + args = json.loads(calls[0]["function"]["arguments"]) + assert args["code"] == 'print("")\nx = 1' + + def test_strip_covers_the_full_call(self): + from core.inference.tool_call_parser import strip_tool_markup + + text = ( + "python\n" + "code\n" + 'print("")\nx = 1\n' + " Done." + ) + stripped = strip_tool_markup(text, final = True) + assert "arg_value" not in stripped + assert stripped.strip() == "Done." + + def test_unbalanced_apostrophe_falls_back_to_first_candidate(self): + # Prose-like value with an apostrophe: no candidate reaches balanced + # quote state, so the first token-valid close wins (prior behavior). + text = ( + "web_search\n" + "query\n" + "it's fine\n" + "" + ) + calls = parse_tool_calls_from_text(text, allow_incomplete = True) + assert len(calls) == 1 + args = json.loads(calls[0]["function"]["arguments"]) + assert args["query"] == "it's fine" + + class TestPythonTagLiteralInsideMistralArgs: """A python_tag LITERAL inside a leading Mistral call's arguments is data; the outer call executes.""" @@ -719,6 +979,60 @@ class TestMagistralThinkRehearsal: assert parse_tool_calls_from_text(text) == [] +class TestGemmaUnquotedApostrophes: + """Quotes open strings only at value-start context: an apostrophe inside + an unquoted wrapper-less value (contractions, possessives) is prose, and + treating it as an opener swallowed the closing brace and lost the call.""" + + def test_contraction_in_unquoted_query_parses(self): + text = "call:web_search{query:what's the weather}" + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search"}) + assert len(calls) == 1 + args = json.loads(calls[0]["function"]["arguments"]) + assert args["query"] == "what's the weather" + + def test_contraction_does_not_swallow_next_key(self): + text = "call:web_search{query:what's up, n:3}" + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search"}) + assert len(calls) == 1 + args = json.loads(calls[0]["function"]["arguments"]) + assert args["query"] == "what's up" + assert args["n"] == 3 + + def test_contraction_strip_span_covers_whole_call(self): + from core.inference.tool_call_parser import strip_tool_markup + + text = "call:web_search{query:what's the weather} Done." + stripped = strip_tool_markup(text, final = True, enabled_tool_names = {"web_search"}) + assert "call:web_search" not in stripped + assert stripped.strip() == "Done." + + def test_quoted_values_still_hide_delimiters(self): + text = 'call:web_search{query:"weather, location: Boston", n:2}' + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search"}) + assert len(calls) == 1 + args = json.loads(calls[0]["function"]["arguments"]) + assert args["query"] == "weather, location: Boston" + assert args["n"] == 2 + + +class TestGlmKeyWithoutValue: + """A GLM with no tag: strict mode rejects the call + (same contract as an unclosed value) instead of executing it with the + argument silently dropped; Auto-Heal keeps the lenient skip.""" + + def test_strict_rejects_key_without_value(self): + text = "web_search\nquery\n" + assert parse_tool_calls_from_text(text, allow_incomplete = False) == [] + + def test_heal_keeps_the_lenient_skip(self): + text = "web_search\nquery\n" + calls = parse_tool_calls_from_text(text, allow_incomplete = True) + assert len(calls) == 1 + assert calls[0]["function"]["name"] == "web_search" + assert json.loads(calls[0]["function"]["arguments"]) == {} + + class TestDisabledBareJsonLiteralNotPromoted: """A leading non-enabled-name object is content: nothing inside promotes, and a call after it still parses.""" @@ -742,6 +1056,38 @@ class TestDisabledBareJsonLiteralNotPromoted: assert [c["function"]["name"] for c in calls] == ["web_search"] +class TestDeepSeekMarkerInsideLeadingEnvelopes: + """A DeepSeek/Kimi marker quoted inside a leading bare-JSON or Mistral + call's argument strings is data: the pre-pass must not promote the + embedded no-arg literal and drop the real outer call.""" + + def test_marker_inside_leading_json_call_stays_data(self): + text = ( + '{"name": "web_search", "arguments": ' + '{"query": "what is <｜tool▁calls▁begin｜>...{}..."}}' + ) + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search"}) + assert [c["function"]["name"] for c in calls] == ["web_search"] + args = json.loads(calls[0]["function"]["arguments"]) + assert "tool▁calls▁begin" in args["query"] + + def test_marker_inside_leading_mistral_call_stays_data(self): + text = ( + '[TOOL_CALLS] [{"name": "web_search", "arguments": ' + '{"query": "docs on <｜tool▁calls▁begin｜> markers"}}]' + ) + calls = parse_tool_calls_from_text(text) + assert [c["function"]["name"] for c in calls] == ["web_search"] + + def test_standalone_deepseek_call_still_parses(self): + text = ( + "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>web_search\n" + '```json\n{"query": "cats"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>' + ) + calls = parse_tool_calls_from_text(text) + assert [c["function"]["name"] for c in calls] == ["web_search"] + + class TestMistralLiteralInsideLeadingJson: """A [TOOL_CALLS] literal quoted inside a leading JSON object must not be promoted over it.""" @@ -776,6 +1122,28 @@ class TestGemmaWrappedWhitespace: assert parse_tool_calls_from_text(text, allow_incomplete = False) == [] +class TestDisabledJsonBeforeDeepSeekCall: + """A disabled leading bare-JSON object whose strings mention a + DeepSeek/Kimi marker is dropped and the tail parsed, so a REAL + DeepSeek/Kimi call after the object still executes instead of the whole + message skipping the pre-pass.""" + + _DS = ( + "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>web_search\n" + '```json\n{"query": "cats"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>' + ) + + def test_real_deepseek_call_after_disabled_json_parses(self): + text = '{"name": "Alice", "note": "<｜tool▁calls▁begin｜>"} ' + self._DS + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search"}) + assert [c["function"]["name"] for c in calls] == ["web_search"] + assert json.loads(calls[0]["function"]["arguments"]) == {"query": "cats"} + + def test_disabled_json_with_marker_alone_stays_data(self): + text = '{"name": "Alice", "note": "<｜tool▁calls▁begin｜>"}' + assert parse_tool_calls_from_text(text, enabled_tool_names = {"web_search"}) == [] + + class TestGemmaDottedArgumentKeys: """Dotted Gemma keys (namespaced schemas) must survive key-quoting or the call is lost.""" @@ -787,6 +1155,29 @@ class TestGemmaDottedArgumentKeys: assert args == {"user.name": "bob", "query": "x"} +class TestLeadingWrapperlessGemmaOverEmbeddedMarkers: + """A leading wrapper-less Gemma call to an enabled tool owns the turn: a + quoted foreign literal inside its argument (a query citing another tool + syntax) is data, and tool_healing must not promote it before the Gemma + fallback runs. Foreign markup leading keeps the normal order.""" + + def test_leading_gemma_wins_over_quoted_xml_literal(self): + text = ( + 'call:web_search{query:"explain ' + '{"name":"evil","arguments":{}}"}' + ) + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search", "evil"}) + assert [c["function"]["name"] for c in calls] == ["web_search"] + + def test_xml_leading_keeps_normal_order(self): + text = ( + '{"name":"web_search","arguments":' + '{"query":"call:evil{x:1} example"}}' + ) + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search", "evil"}) + assert [c["function"]["name"] for c in calls] == ["web_search"] + + class TestLeadingMistralCallOwnsTheTurn: """A leading Mistral call wins in document order over literal XML in trailing prose.""" @@ -798,7 +1189,7 @@ class TestLeadingMistralCallOwnsTheTurn: calls = parse_tool_calls_from_text(text) assert [c["function"]["name"] for c in calls] == ["web_search"] - def test_xml_leading_keeps_normal_order(self): + def test_function_xml_leading_keeps_normal_order(self): text = ( "x " "[TOOL_CALLS]evil[ARGS]{}" @@ -816,6 +1207,63 @@ class TestGemmaDottedKeyAfterBareValue: assert args == {"query": "foo", "user.name": "bob"} +class TestJsonAnswersAreDataForMarkerlessScans: + """A whole-content JSON value is a structured answer: a quoted example of + an enabled tool's syntax inside it must not execute the tool, and the + display strip must not mutilate the answer.""" + + def test_gemma_example_inside_json_answer_not_promoted(self): + text = '{"answer":"Gemma syntax is call:web_search{query:hi}"}' + assert parse_tool_calls_from_text(text, enabled_tool_names = {"web_search"}) == [] + + def test_gemma_example_inside_json_answer_not_stripped(self): + from core.inference.tool_call_parser import strip_tool_markup + text = '{"answer":"Gemma syntax is call:web_search{query:hi}"}' + assert strip_tool_markup(text, final = True, enabled_tool_names = {"web_search"}) == text + + def test_kimi_marker_inside_json_answer_not_promoted(self): + text = ( + '{"answer":"<|tool_call_begin|>functions.web_search:0' + '<|tool_call_argument_begin|>{}<|tool_call_end|>"}' + ) + assert parse_tool_calls_from_text(text, enabled_tool_names = {"web_search"}) == [] + + +class TestGemmaNestedQuotedLeaves: + def test_nested_object_and_array_values_are_unquoted(self): + text = 'call:f{loc:{city:"New York"},items:["a","b"],n:3}' + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"f"}) + assert len(calls) == 1 + args = json.loads(calls[0]["function"]["arguments"]) + assert args == {"loc": {"city": "New York"}, "items": ["a", "b"], "n": 3} + + +class TestEarliestEnvelopeWinsAcrossDeepSeekKimi: + """The DeepSeek/Kimi pre-pass dispatches by earliest envelope opener: a + leading real call wins over a trailing example of the sibling format in + either direction (document order, like the other leading guards).""" + + _DS = ( + "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>evil\n" + '```json\n{"x": 1}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>' + ) + _KIMI = ( + "<|tool_calls_section_begin|><|tool_call_begin|>functions.web_search:0" + '<|tool_call_argument_begin|>{"query": "cats"}<|tool_call_end|>' + "<|tool_calls_section_end|>" + ) + + def test_leading_kimi_wins_over_trailing_deepseek_example(self): + text = self._KIMI + " For reference: " + self._DS + calls = parse_tool_calls_from_text(text) + assert [c["function"]["name"] for c in calls] == ["web_search"] + + def test_leading_deepseek_wins_over_trailing_kimi_example(self): + text = self._DS + " Kimi format: " + self._KIMI + calls = parse_tool_calls_from_text(text) + assert [c["function"]["name"] for c in calls] == ["evil"] + + class TestNamelessLeadingJsonAnswerIsData: """A nameless leading JSON answer is an envelope: quoted markup stays data, and a call after it parses.""" @@ -832,6 +1280,140 @@ class TestNamelessLeadingJsonAnswerIsData: assert [c["function"]["name"] for c in calls] == ["web_search"] +class TestClosedCallPrecedesMarkerPrePass: + """A closed non-DeepSeek/Kimi call that precedes the first DS/Kimi marker + owns the turn: a trailing example (or an example quoted inside a wrapped + Gemma argument) must not be promoted by the pre-pass.""" + + _KIMI_EVIL = ( + "<|tool_calls_section_begin|><|tool_call_begin|>functions.evil:0" + '<|tool_call_argument_begin|>{"x": 1}<|tool_call_end|>' + "<|tool_calls_section_end|>" + ) + + def test_kimi_example_inside_wrapped_gemma_arg_stays_data(self): + text = ( + '<|tool_call>call:web_search{query:<|"|>explain ' + + self._KIMI_EVIL + + '<|"|>}' + ) + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search", "evil"}) + assert [c["function"]["name"] for c in calls] == ["web_search"] + + def test_leading_xml_call_wins_over_trailing_kimi_example(self): + text = ( + '{"name":"web_search","arguments":{"query":"cats"}}' + " For reference: " + self._KIMI_EVIL + ) + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search", "evil"}) + assert [c["function"]["name"] for c in calls] == ["web_search"] + + def test_standalone_kimi_call_still_parses(self): + calls = parse_tool_calls_from_text(self._KIMI_EVIL) + assert [c["function"]["name"] for c in calls] == ["evil"] + + +class TestTruncatedWrapperlessGemmaStopsScan: + def test_call_quoted_inside_truncated_arg_not_promoted(self): + text = 'call:python{code:example("call:web_search{query:hi}") and then it cut' + assert parse_tool_calls_from_text(text, enabled_tool_names = {"python", "web_search"}) == [] + + +class TestGemmaQuotedNestedDelimiters: + def test_comma_inside_quoted_nested_string_not_a_split(self): + text = 'call:f{loc:{city:"New, York"},n:1}' + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"f"}) + assert len(calls) == 1 + args = json.loads(calls[0]["function"]["arguments"]) + assert args == {"loc": {"city": "New, York"}, "n": 1} + + +class TestGemmaStringMarkerLiteralInArgs: + def test_string_marker_literal_does_not_lose_the_call(self): + text = "call:web_search{query:'what does <|\"|> mean in Gemma'}" + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search"}) + assert len(calls) == 1 + args = json.loads(calls[0]["function"]["arguments"]) + assert args["query"] == 'what does <|"|> mean in Gemma' + + +class TestGemmaMidValueQuotedPhrase: + def test_quoted_phrase_mid_value_hides_delimiters(self): + text = 'call:web_search{query:find "weather, location: Boston", limit:3}' + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search"}) + assert len(calls) == 1 + args = json.loads(calls[0]["function"]["arguments"]) + assert args == {"query": 'find "weather, location: Boston"', "limit": 3} + + def test_apostrophes_still_prose_mid_value(self): + text = "call:web_search{query:what's on at the museum, n:2}" + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search"}) + args = json.loads(calls[0]["function"]["arguments"]) + assert args == {"query": "what's on at the museum", "n": 2} + + +class TestGlmStrictRefusesInQuoteFallback: + """A truncated GLM value whose only close candidates sit inside a string + literal must reject in strict mode instead of executing truncated + arguments; Auto-Heal keeps the lenient partial value.""" + + _TRUNC = ( + 'python\ncode\nprint("")' + ) + + def test_strict_rejects_truncated_in_string_close(self): + assert parse_tool_calls_from_text(self._TRUNC, allow_incomplete = False) == [] + + def test_heal_keeps_partial_value(self): + calls = parse_tool_calls_from_text(self._TRUNC, allow_incomplete = True) + assert len(calls) == 1 and calls[0]["function"]["name"] == "python" + + +class TestGemmaGuardCoversPreambles: + def test_preamble_then_gemma_call_quoting_xml_wins(self): + text = ( + "Sure, searching now. call:web_search{query:" + '"explain {"name":"evil","arguments":{}}"}' + ) + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"web_search", "evil"}) + assert [c["function"]["name"] for c in calls] == ["web_search"] + + +class TestGlmStrictAcceptsApostrophes: + def test_apostrophe_value_parses_in_strict_mode(self): + text = ( + "web_search\nquery\n" + "what's the weather\n" + ) + calls = parse_tool_calls_from_text(text, allow_incomplete = False) + assert len(calls) == 1 + args = json.loads(calls[0]["function"]["arguments"]) + assert args == {"query": "what's the weather"} + + +class TestDisabledGemmaCallLiteralsAreData: + def test_literal_inside_disabled_call_not_promoted(self): + text = 'call:foo{query:"x"}' + assert parse_tool_calls_from_text(text, enabled_tool_names = {"python", "web_search"}) == [] + + def test_real_call_after_disabled_example_still_parses(self): + text = ( + 'call:foo{query:"x"}' + " call:web_search{query:hi}" + ) + calls = parse_tool_calls_from_text(text, enabled_tool_names = {"python", "web_search"}) + assert [c["function"]["name"] for c in calls] == ["web_search"] + + +class TestLeadingJsonArrayAnswerIsData: + def test_kimi_marker_inside_json_array_answer_not_promoted(self): + text = ( + '[{"answer": "<|tool_call_begin|>functions.web_search:0' + '<|tool_call_argument_begin|>{}<|tool_call_end|>"}]' + ) + assert parse_tool_calls_from_text(text, enabled_tool_names = {"web_search"}) == [] + + class TestLeadingBareJsonOwnsTurnOverTrailingXml: """Document order: a leading closed bare-JSON call owns the turn even when tool XML appears AFTER it (inside-or-after, mirroring the Mistral rule).""" @@ -855,7 +1437,8 @@ class TestLeadingBareJsonOwnsTurnOverTrailingXml: assert [c["function"]["name"] for c in calls] == ["lookup", "lookup"], calls def test_non_call_leading_object_defers_to_trailing_real_call(self): - # Nameless/disabled-name objects decline: dropped, and the real trailing call still parses. + # Nameless answers and disabled-name objects take the decline path: + # the object is dropped and the real trailing call still parses. for lead in ('{"answer": 42}', '{"name":"draft","parameters":{}}'): text = lead + ' {"name":"delete_all","arguments":{}}' calls = parse_tool_calls_from_text(text, enabled_tool_names = {"delete_all"}) @@ -891,7 +1474,8 @@ class TestProseCloseTagAfterClosedFunctionCall: assert json.loads(calls[0]["function"]["arguments"]) == {"code": 'print("")'} def test_attribute_form_arguments_do_not_swallow_prose(self): - # The attribute form shares the first-balanced-close rule: prose closes never fold in. + # The attribute form shares the first-balanced-close + # rule: prose mentioning a literal close tag never folds into arguments. text = ( 'cats' " Done. The tag closes a call." diff --git a/studio/backend/tests/test_tool_xml_strip.py b/studio/backend/tests/test_tool_xml_strip.py index 7fe52a664..d50c27130 100644 --- a/studio/backend/tests/test_tool_xml_strip.py +++ b/studio/backend/tests/test_tool_xml_strip.py @@ -24,19 +24,39 @@ import re as _re _src = (Path(_BACKEND_DIR) / "routes" / "inference.py").read_text() _m = _re.search(r"_TOOL_XML_RE = _re\.compile\((.*?)\n\)", _src, _re.DOTALL) assert _m, "could not extract _TOOL_XML_RE source" -# Provide both helpers so the extracted _strip_tool_xml_for_display resolves. -from core.inference.tool_call_parser import _strip_function_xml_calls, _strip_mistral_closed_calls +# The lazy ``(.*?)\n\)`` could grab a shorter expression if an arm is ever wrapped; +# pin the DeepSeek + bare-Kimi arms so a silent truncation fails loudly here. +assert "_DS_OPEN_SRC" in _m.group(1) and "tool_call_begin" in _m.group( + 1 +), "extracted _TOOL_XML_RE is missing expected arms (extraction truncated?)" +# The regex reuses the parser's shared DeepSeek opener alternation; provide it so the extracted +# ``_re.compile`` expression resolves the same source. +from core.inference.tool_call_parser import _DEEPSEEK_OPEN_RE_SRC as _DS_OPEN_SRC +from core.inference.tool_call_parser import ( + _strip_function_xml_calls, + _strip_gemma_wrapperless_calls, + _strip_glm_calls, + _strip_mistral_closed_calls, +) + +from typing import Optional as _Optional _ns = { "_re": _re, + "_DS_OPEN_SRC": _DS_OPEN_SRC, + "Optional": _Optional, "_strip_mistral_closed_calls": _strip_mistral_closed_calls, + "_strip_gemma_wrapperless_calls": _strip_gemma_wrapperless_calls, + "_strip_glm_calls": _strip_glm_calls, "_strip_function_xml_calls": _strip_function_xml_calls, } exec(f"_TOOL_XML_RE = _re.compile({_m.group(1)})", _ns) _TOOL_XML_RE = _ns["_TOOL_XML_RE"] +# Signatures may span multiple lines and now carry the enabled_tool_names gate; match +# the whole (possibly multi-line) signature up to ``-> str:`` then the indented body. _xml_helper = _re.search( - r"def _strip_tool_xml\(text: str\) -> str:\n(?: .+\n)+", + r"def _strip_tool_xml\((?:.|\n)*?\) -> str:\n(?: .+\n)+", _src, ) assert _xml_helper, "could not extract _strip_tool_xml source" @@ -44,17 +64,27 @@ assert "_strip_mistral_closed_calls" in _xml_helper.group( 0 ), "extracted _strip_tool_xml no longer runs the Mistral balanced strip" exec(_xml_helper.group(0), _ns) +_strip_tool_xml = _ns["_strip_tool_xml"] _helper = _re.search( - r"def _strip_tool_xml_for_display\(text: str, \*, auto_heal_tool_calls: bool\) -> str:\n" - r"(?: .+\n)+", + r"def _strip_tool_xml_for_display\((?:.|\n)*?\) -> str:\n(?: .+\n)+", _src, ) assert _helper, "could not extract _strip_tool_xml_for_display source" +# After the V1 fix the display helper delegates to _strip_tool_xml; confirm the +# extracted body actually reached that call rather than truncating early. assert "_strip_tool_xml(" in _helper.group(0), "display helper no longer delegates" exec(_helper.group(0), _ns) _strip_tool_xml_for_display = _ns["_strip_tool_xml_for_display"] +_gate_src = _re.search( + r"def _gemma_strip_gate\((?:.|\n)*?\) -> set:\n(?: .+\n)+", + _src, +) +assert _gate_src, "could not extract _gemma_strip_gate source" +exec(_gate_src.group(0), _ns) +_gemma_strip_gate = _ns["_gemma_strip_gate"] + # ── Well-formed pairs ───────────────────────────────────────────── @@ -66,7 +96,8 @@ def test_route_display_strip_respects_disabled_auto_heal_contract(): def test_route_display_strip_removes_mistral_tool_calls_with_nested_json(): - # [TOOL_CALLS] with nested JSON needs the Mistral balanced-brace strip, not the regex. + # _TOOL_XML_RE has no [TOOL_CALLS] arm, so the helper delegates to _strip_tool_xml for the Mistral + # balanced-brace strip (a non-greedy \{.*?\} would truncate nested JSON). text = 'ok [TOOL_CALLS]web_search{"filters":{"date":"2024"},"query":"cats"} tail' assert _strip_tool_xml_for_display(text, auto_heal_tool_calls = False) == text out = _strip_tool_xml_for_display(text, auto_heal_tool_calls = True) @@ -102,7 +133,8 @@ def test_strips_function_only_well_formed(): def test_strips_function_attribute_form(): - # Attribute form must strip from the route too; dotted/hyphenated names included. + # Attribute form ```` (MiniCPM-5 / MiniMax-M2) must strip from the route too + # (it previously leaked into the UI); a dotted/hyphenated name also strips. text = ( 'Sure.\n\n' "\nSydney\n\n\nDone." @@ -330,9 +362,84 @@ def test_no_catastrophic_backtracking_on_orphan_opening_spam(): assert "" not in cleaned +# ── DeepSeek opener variants + bare Kimi (parse/strip symmetry) ── + + +def test_strips_deepseek_space_opener_variant(): + # The space-separated opener is parsed by the parser, so the display strip + # must remove it too (the shared opener alternation is reused here). + text = ( + "pre <｜tool calls begin｜><｜tool▁call▁begin｜>get_x<｜tool▁sep｜>" + '{"a":1}<｜tool▁call▁end｜><｜tool▁calls▁end｜> post' + ) + cleaned = _TOOL_XML_RE.sub("", text) + assert "tool" not in cleaned.replace("post", "").replace("pre", "") + assert cleaned == "pre post" + + +def test_strips_deepseek_escaped_underscore_opener_variant(): + text = ( + "pre <｜tool\\_calls\\_begin｜><｜tool▁call▁begin｜>get_y<｜tool▁sep｜>" + '{"a":1}<｜tool▁call▁end｜><｜tool▁calls▁end｜> post' + ) + cleaned = _TOOL_XML_RE.sub("", text) + assert cleaned == "pre post" + + +def test_strips_bare_kimi_call_without_section_wrapper(): + # Kimi can emit a bare <|tool_call_begin|>...<|tool_call_end|> with no + # section wrapper; the parser accepts it, so the strip must cover it. + text = ( + "pre <|tool_call_begin|>functions.get_w:0<|tool_call_argument_begin|>" + '{"a":1}<|tool_call_end|> post' + ) + cleaned = _TOOL_XML_RE.sub("", text) + assert "tool_call_begin" not in cleaned + assert cleaned == "pre post" + + +@pytest.mark.parametrize( + "text", + [ + # Prose that merely names a Kimi/DeepSeek marker (no real call follows) must + # survive: the call-shaped lookahead fires only on a real call or a bare EOF + # fragment, so an answer discussing the protocol is never truncated. + "See <|tool_call_begin|> in the docs. More prose after it.", + "The <|tool_calls_section_begin|> marker opens a batch. Read on.", + "DeepSeek uses <｜tool▁calls▁begin｜> to start a call block, then continues.", + ], +) +def test_deepseek_kimi_false_alarm_prose_is_kept(text): + # Regression for the route arm truncating a prose answer that references a marker + # without a following call (parser _TOOL_ALL_PATS already had this lookahead). + assert _TOOL_XML_RE.sub("", text) == text + + +def test_deepseek_kimi_real_calls_still_strip_after_false_alarm_fix(): + # The lookahead must not weaken real-call stripping: closed, truncated, and bare + # EOF-fragment forms all still get removed. + closed = ( + "answer <|tool_call_begin|>functions.get_w:0<|tool_call_argument_begin|>" + '{"a":1}<|tool_call_end|> tail' + ) + assert _TOOL_XML_RE.sub("", closed) == "answer tail" + eof_fragment = "prefix <|tool_call_begin|>" + assert _TOOL_XML_RE.sub("", eof_fragment) == "prefix " + deepseek = ( + "reply <｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_x<｜tool▁sep｜>" + '{"a":1}<｜tool▁call▁end｜><｜tool▁calls▁end｜>' + ) + assert _TOOL_XML_RE.sub("", deepseek) == "reply " + + +# ── Llama-3 <|python_tag|> arm bounds on REAL sentinels only ────── + + # Llama-3 <|python_tag|> arm bounds on REAL sentinels only def test_python_tag_strip_consumes_literal_sentinel_in_arg(): - # A literal <|...|> token inside the arg must not end the strip early. + # A <|python_tag|> tool call whose JSON argument carries a literal <|...|> + # token (here <|cite|>) must be stripped whole. The old `<(?!\|)` arm stopped + # at any `<|`, leaking the call tail (e.g. `<|cite|> here"}}`) into display. text = '<|python_tag|>{"name": "send", "parameters": {"text": "use <|cite|> here"}}' cleaned = _TOOL_XML_RE.sub("", text) assert cleaned == "", f"python_tag call leaked at literal sentinel: {cleaned!r}" @@ -348,7 +455,8 @@ def test_python_tag_strip_consumes_literal_sentinel_in_arg(): ], ) def test_python_tag_strip_stops_at_real_sentinel(sentinel): - # A real control sentinel bounds the strip so following text survives. + # A genuine Llama control sentinel still bounds the strip so following + # assistant text is preserved (the arm must not swallow past it). text = f'<|python_tag|>{{"name": "x", "parameters": {{}}}}{sentinel}visible answer' cleaned = _TOOL_XML_RE.sub("", text) assert ( @@ -357,14 +465,37 @@ def test_python_tag_strip_stops_at_real_sentinel(sentinel): def test_python_tag_strip_restarts_on_second_python_tag(): - # A second <|python_tag|> opens a new region; both are stripped. + # A second <|python_tag|> opens a new tool-call region, so the whole pair is + # stripped (the arm bounds the first, then the next match consumes the rest). text = '<|python_tag|>{"name": "a"}<|python_tag|>{"name": "b"}' cleaned = _TOOL_XML_RE.sub("", text) assert cleaned == "", f"second python_tag region leaked: {cleaned!r}" +def test_glm_call_with_literal_close_tag_in_arg_value_is_stripped_whole(): + # GLM 4.x emits NAMEkv .... + text = ( + "web_search\nquery\n" + "find here\n done" + ) + out = _strip_tool_xml_for_display(text, auto_heal_tool_calls = True) + assert "" not in out + assert "" not in out + assert out.strip() == "done" + + +def test_glm_normal_and_qwen_calls_still_stripped_by_route(): + # Regression: a normal GLM call (no literal close tag) and a Qwen + # {json} are still stripped; trailing prose is kept. + glm = "get_time\ntz\nUTC\n ok" + assert _strip_tool_xml_for_display(glm, auto_heal_tool_calls = True).strip() == "ok" + qwen = '{"name":"web_search","arguments":{"q":"x"}} after' + assert _strip_tool_xml_for_display(qwen, auto_heal_tool_calls = True).strip() == "after" + + def test_route_strip_removes_param_alias_close_tag(): - # Orphan (attribute-form alias of ) must strip too. + # The parser accepts the ... attribute-form alias of + # ; the route tail cleanup must strip an orphan close too. assert _strip_tool_xml_for_display("answer ", auto_heal_tool_calls = True) == "answer " assert ( _strip_tool_xml_for_display("answer ", auto_heal_tool_calls = True) == "answer " @@ -372,13 +503,44 @@ def test_route_strip_removes_param_alias_close_tag(): def test_route_strip_uses_guarded_function_scan_for_literal_nested_markup(): - # A literal in a value must not truncate the strip. + # A literal in a value must not truncate the strip: the route runs the + # parser's guarded function-XML scan before the regex, matching the core strip. text = " tail" assert _strip_tool_xml_for_display(text, auto_heal_tool_calls = True).strip() == "tail" +def test_route_strip_gates_wrapperless_gemma_by_enabled_tools(): + # The route strip must gate the markerless Gemma call:NAME{...} form on the enabled tool names, + # like the parser/loop, so a disabled/example name in prose is preserved in ... + prose = "To document syntax you write call:foo{query:example}. That shows the format." + assert "call:foo{query:example}" in _strip_tool_xml(prose, {"web_search"}) + # An enabled name is still a real call and stripped. + assert "call:web_search" not in _strip_tool_xml( + "Answer. call:web_search{query:x}", {"web_search"} + ) + # No gate (legacy) strips every closed call. + assert "call:foo" not in _strip_tool_xml(prose) + + +def test_gemma_strip_gate_empty_tools_preserves_prose(): + # With NO tools enabled the gate must return an EMPTY set (strip nothing), not None: None falls + # back to strip-all and deletes an answer that documents the call:NAME{...} syntax. + assert _gemma_strip_gate([]) == set() + assert _gemma_strip_gate(None) == set() + assert _gemma_strip_gate([{"function": {"name": "web_search"}}]) == {"web_search"} + prose = "To document syntax you write call:foo{query:example}. That shows the format." + assert "call:foo{query:example}" in _strip_tool_xml(prose, _gemma_strip_gate([])) + assert "call:foo{query:example}" in _strip_tool_xml(prose, _gemma_strip_gate(None)) + # An enabled tool's real call is still stripped. + assert "call:web_search" not in _strip_tool_xml( + "Answer. call:web_search{query:x}", + _gemma_strip_gate([{"function": {"name": "web_search"}}]), + ) + + def test_strip_keeps_prose_after_closed_function_call_with_literal_close(): - # The call ends at its first non-data close; prose after (even a literal ) survives. + # The call ends at its first non-data close: prose after it survives the + # strip even when it mentions a literal . from core.inference.tool_call_parser import strip_tool_markup text = ( "cats" @@ -388,7 +550,8 @@ def test_strip_keeps_prose_after_closed_function_call_with_literal_close(): def test_final_strip_keeps_prose_mentioning_bare_markers(): - # A false-alarm marker in prose must not drop trailing text; only call-start-shaped text drops. + # A false-alarm marker in a normal answer must not lose everything after + # it; only text that looks like that family's call start drops. from core.inference.tool_call_parser import strip_tool_markup for text in ( "See [TOOL_CALLS] docs for details. More prose after.", @@ -413,7 +576,8 @@ def test_final_strip_still_drops_truncated_marker_calls(): def test_chained_bare_json_strip_consumes_all_calls(): - # Next-turn history must not keep an executed call, else it replays. + # The loops keep this text as next-turn history: a leftover executed call + # would be replayed alongside the structured tool_calls. from core.inference.tool_call_parser import strip_leading_bare_json_call enabled = {"web_search", "python"}