6083 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Wasim Yousef Said	49d1fb3863	Speed up Studio startup path (#6899) ... * Speed up Studio startup path * Studio: recheck managed binary executability on preflight cache hit and ignore stale unauthenticated platform fetches Preflight: a matching capability cache fingerprint no longer skips the runnability check when the managed binary's executable bit was cleared (size and mtime unchanged, since chmod bumps ctime not mtime). The cache fast path now confirms the binary is still executable, otherwise it falls back to the CLI help probe so preflight reports Stale and can repair, instead of returning Ready and failing later at backend start. Adds a regression test. Frontend: now that first render is no longer gated on fetchDeviceType, the initial unauthenticated health call can resolve after an authenticated platform fetch. Guard the store so a late unauthenticated or failed non-forced response cannot overwrite an already authoritative device type, tunnel URL, or secure flag. Forced refreshes and the first unauthenticated load are unaffected. * Studio: use access(X_OK) for the preflight cache executability guard A mode bitmask treats any execute bit as launchable, but the executable bits can be set only for another owner or group, or be denied by an ACL, so the current user could still hit PermissionDenied at launch and the cached fast path would wrongly return Ready. access(X_OK) checks real executability for the calling user, so an ownership or permission change correctly falls back to the CLI help probe and the Stale repair path. * Studio: ignore any stale non-forced platform fetch once authoritative Extend the platform store guard so a non-forced health response never overwrites an already authoritative result, not only unauthenticated ones. With a saved token the post-render non-forced request can be authenticated but older than a later forced refresh that already picked up the tunnel URL and secure flag; if that earlier request resolves last it would null those fields. Now any non-forced response is dropped once the store holds a server-reported platform. Forced refreshes and the first authoritative write are unaffected. * Studio: run the managed CLI help probe before trusting the preflight cache Restore running the managed CLI help probe before returning Ready from the desktop capability cache, so a managed install whose venv interpreter or a runtime dependency is broken (while path, size, mtime, and markers are unchanged) is reported Stale for repair rather than proceeding to a backend start that cannot spawn. The capability cache still skips the heavier desktop-capabilities probe on a hit, so a warm cache runs one probe instead of two. Removes the executable-access shortcut, which the help probe now subsumes. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-07-07 18:08:07 -07:00
Daniel Han	01b8085dc2	Create ossf.yml (#6952)	2026-07-07 17:10:01 -07:00
oobabooga	a9db53e189	Studio: stream reasoning tokens in the tool-loop generator (fixes DeepSeek thinking not streaming with a pill on) (#6947)	2026-07-07 19:50:40 -03:00
Ayushman	304b8eca7a	fix: match qwen3-thinking double-newline in train_on_responses_only response pattern (#6926) ... * fix: match qwen3-thinking chat template double-newline in response pattern The Qwen3-thinking chat template generates `<think>\n\n` (double newline) after the think tag, but `train_on_responses_only` was looking for `<think>\n` (single newline). `\n\n` is token 271 while `\n` is token 198 -- different tokens, so the pattern match in `train_on_responses_only` fails, masking ALL tokens and dropping 100% of training samples. Update the response pattern from `<think>\n` to `<think>\n\n` to match what the actual qwen3-thinking template generates. Fixes #6919 * fix qwen3 thinking response marker --------- Co-authored-by: Ayushman Paul <ayushman@HP> Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>	2026-07-07 21:28:18 +03:00
oobabooga	93c9d6d0dd	Studio: render \[ \] and \( \) LaTeX delimiters in chat (#6914)	2026-07-07 15:13:53 -03:00
Nilay	07ecdb34c0	Sort chat recents by last activity (#6844) ... * show chat by by last activity * Update chat thread updated_at logic and enhance sidebar chat item handling * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-07 17:54:32 +01:00
Daniel Han	37075c5422	Bump install.sh / install.ps1 pin to unsloth>=2026.7.1 (#6943) ... Co-authored-by: danielhanchen <unslothai@gmail.com>	2026-07-07 07:49:59 -07:00
Daniel Han	8efcc17f47	Studio: account for DeepSeek-V4 compute buffer in context auto-fit (#6940) ... * Studio: account for DeepSeek-V4 compute buffer in context auto-fit DeepSeek-V4-Flash's lightning indexer plus compressed sparse attention reserve a large context-scaling compute buffer that _compute_buffer_ctx_bytes did not model (the KQ-mask and dequant-scratch rates both miss it, even with an f16 cache). Measured on UD-Q4_K_XL at ub 512 it is about 65.5 GiB at 1M context, which the mask estimate puts near 1.5 GiB, so the auto-fit kept the full 1M train context and llama-server OOM'd allocating the ~70 GB buffer, then spilled to CPU (~4 tok/s). Add a deepseek4-gated flat plus per-token term so the fit caps the context (about 256k on a B200) and the model stays fully on GPU. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-07 07:20:31 -07:00
Etherll	ba450b437e	Studio: add assistant response details panel (#6842) ... * Studio: add assistant response details panel * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide model badge by default, show on hover/focus Wrap MessageResponseModelBadge in a span with hidden/group-hover visibility classes to reduce visual clutter. The badge now only displays when hovering or focusing on the assistant message, improving the UI presentation. Updated corresponding tests to verify the new CSS classes. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-07 06:26:00 -07:00
Daniel Han	10d8f985a2	Versioning	2026-07-07 06:19:25 -07:00
Daniel Han	411c4d1e50	Add DeepSeek-V4-Flash-GGUF to Studio with none/high/max reasoning (#6908) ... * Add DeepSeek-V4-Flash-GGUF to Studio with none/high/max reasoning Adds unsloth/DeepSeek-V4-Flash-GGUF as a default selectable model with the recommended decoding defaults (temperature 1.0, top_p 1.0 from the official generation_config.json) and its three tier reasoning control. The high/max ladder is surfaced for deepseek-v4 model ids and flows through the existing enable_thinking_effort reasoning style via chat_template_kwargs, so no frontend changes are needed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio DeepSeek-V4: segment-scope high, enable thinking for lone effort, render tests Match deepseek-v4 on whole repo-name segments so a future deepseek-v40 or deepseek40 cannot false-match the synthetic 'high'. In _request_reasoning_kwargs, emit enable_thinking when a named effort level is sent without it, so the newly exposed High mode renders thinking-on over the API (the UI already sent it explicitly). Add a none/high/max render-path test file (jinja behind importorskip) with a lone-high regression. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-07 06:13:43 -07:00
Daniel Han	59977f95c3	GRPO: default router_aux_loss_coef to 0 on TRL >= 1.7.0 (#6938) ... TRL 1.7.0 enables the MoE router load-balancing aux loss by default (router_aux_loss_coef = 0.001). Unsloth's optimized GRPO forward does not compute it, so default the coefficient to 0, matching pre-1.7.0 behaviour. Users can still opt in with router_aux_loss_coef > 0. No-op on TRL < 1.7.0.	2026-07-07 05:49:24 -07:00
Daniel Han	d79495dc96	Add RDNA 2/3/4 ROCm routing tests via a CPU-only torch spoof (#6935) ... * Add RDNA 2/3/4 ROCm routing tests via a CPU-only torch spoof Introduces tests/_zoo_rocm_spoof.py, the ROCm sibling of _zoo_aggressive_cuda_spoof.py: it reuses the CUDA spoof's torch.cuda no-op machinery and overlays an AMD Radeon identity (torch.version.hip, gcnArchName, capability) for any RDNA 2/3/4 gfx target, so hip code paths run on CPU-only CI with no AMD hardware. tests/studio/install/test_rocm_rdna_routing.py then asserts unsloth_zoo routes every RDNA arch (gfx1030/1031/1032/1034, gfx1100/1101/1102, gfx1150/1151, gfx1200/1201) correctly: device_type resolves to hip, llama.cpp target resolves to (rocm, gfx), and the per-family ROCm bundle suffix (gfx103X/gfx110X/gfx120X, or self for gfx1150/1151) is picked. The torch-facing checks run in a subprocess so the spoof never leaks into sibling tests and DEVICE_TYPE (cached at import) resolves from a clean process; the pure gfx-family mapping runs in-process. Guarded by importorskip so it runs where torch and unsloth_zoo are installed (the Repo tests CPU job) and skips elsewhere. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-07 04:41:37 -07:00
Daniel Han	414503745e	Run the malware gate on the RAG embedding model before it loads (#6887) ... * Run the malware gate on the RAG embedding model before it loads Setting the RAG embedding model through PUT /api/settings/embedding-model persisted an arbitrary repo and later handed it straight to SentenceTransformer, which deserializes pickle weights. Unlike the normal model-load paths, this route never ran evaluate_file_security, and force skipped verification entirely, so a repo Hugging Face flags as unsafe (or any repo under force) could be downloaded and loaded in the backend process without a scan. Run the malware/pickle scan at both ends: the settings endpoint now scans before persisting and returns 409 on a flagged repo even under force (force still only skips the is-embedding-model type check for offline or local repos), and the embedder scans again at the load sink so a name that arrives via env or default is covered too. Local paths and unreachable scans fail open inside evaluate_file_security, and the sink never bricks the embedder on a gate error. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Thread the load token into the embedding scan and hard-fail on a block The load-sink scan ran without a token, so evaluate_file_security (which passes token=False when none is given) could not reach a gated or private repo and failed open for exactly the model SentenceTransformer would still load. Resolve the loader's own token (HF_TOKEN env or the cached login) and pass it to the sink scan, and fall back to it in the settings endpoint when the request omits one. The sink previously raised a plain RuntimeError, which the llama-server fallback in encode() and _build_st_backend_or_fallback() swallowed as a routine ST failure, silently switching backends instead of blocking. Raise a distinct UnsafeEmbeddingModelError that both fallback paths re-raise, so a flagged model hard-fails. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scan sentence-transformers module dirs and scope the embedding pickle gate to the ST backend Extend the RAG embedding malware gate so a poisoned pickle under a SentenceTransformer module dir (for example 0_Transformer/pytorch_model.bin) blocks. Those dirs are read from the repo's modules.json and passed as load roots to evaluate_file_security at both the settings endpoint and the load sink, so such a pickle is treated as root-level there instead of an unreferenced nested shard that was previously allowed. Scope the ST pickle scan to the sentence-transformers backend. On the llama-server backend the embedder loads GGUF files (inert) from the -GGUF companion repo, never the ST repo's pickle, so a custom ST repo with a flagged pickle and a clean GGUF companion is no longer rejected. The existing GGUF availability checks already cover that path. Return 403 for the hard security block instead of 409. The settings UI routes every 409 into the forceable save-anyway flow, but this block cannot be bypassed by force, so it now uses a distinct status the client treats as non-forceable. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Base the embedding pickle scan on the actual backend, not just the resolver _llama_backend_active only consulted the auto resolver, so on a GPU box where auto resolves to sentence-transformers but the process already fell back to the llama-server backend at runtime (a torch or CUDA load/encode failure), it returned False and the settings endpoint hard-blocked a save whose ST pickle is flagged even though the process loads only inert GGUF. Add active_backend_is_llama, which reflects the actual built backend (True when the cached backend is a LlamaServerBackend, including a runtime fallback) and otherwise defers to the resolver as a fresh process would, and delegate _llama_backend_active to it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Report the cached embedding backend verbatim, not the resolver active_backend_is_llama() fell through to the config resolver whenever a backend was already built but was not llama-server, so a live sentence-transformers backend could report llama=True once the resolver picked llama (GPU heuristic or a runtime config change) and wrongly skip its pickle scan. Once a backend exists, return isinstance(backend, LlamaServerBackend) directly; only defer to the resolver before any backend is built. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-07 04:30:21 -07:00
Daniel Han	bdb958e052	Guard RoPE scaling against the transformers v5 buffer blank; honor extended RoPE factor (#6925) ... * Guard RoPE scaling against the transformers v5 buffer blank; honor extended factor Add a family-agnostic guard that builds each rotary from a scaled config, blanks its non-persistent buffers (what transformers v5 does on load), runs loader._fix_rope_inv_freq, and asserts every buffer is restored to its scaled value (llama3 and longrope). This catches the whole bug class, not just the one call site, and is validated to fail on the pre-fix repair. Also make LlamaExtendedRotaryEmbedding read the llama3 factor from the config instead of hardcoding 8 (wrong for Llama-3.2, factor 32), falling back to the Llama-3.1 defaults when built without a config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pass config into extended rotary codegen; skip v5 round-trip on transformers 4.x - patch_llama_rope_scaling now builds the llama3 extended rotary with config=self.config so it reads the real factor (32 for Llama-3.2) instead of falling back to 8; the template already references self.config. - test_v5_blank_repair_roundtrip now skips when loader._NEEDS_ROPE_FIX is False, since _fix_rope_inv_freq is a no-op on transformers 4.x and cannot restore the blanked buffers there. * Raise stream deadlock-guard timeouts from 0.2s to 5.0s in passthrough tests These asyncio.wait_for guards bound test setup and cross-task event signaling that complete near-instantly on success; the 0.2s budget is a latency assertion in disguise and times out under CI scheduling load (seen on the 3.11 matrix leg while 3.10/3.12/3.13 pass the same commit). 5.0s matches the timeout used elsewhere in the suite and still fails fast on a real hang. No test relies on the guard expiring. * Extended rotary reads rope_parameters as well as rope_scaling transformers v5 stores llama3 scaling under config.rope_parameters and exposes rope_scaling only as a back-compat property. Reading that property works on 5.0-5.13 (verified: factor resolves to 32 for Llama-3.2), but a future release may drop the shim, after which the subclass path would fall back to factor 8. Read either field so the factor survives the rename. Adds test_extended_rotary_reads_rope_parameters_v5 (fails on the old single-field read: rope_parameters-only config resolves to 8, not 32). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-07 04:16:57 -07:00
Leo Borcherding	296cacb5a1	ROCm-on-WSL: support discrete Radeon (RDNA 3/4) in WSL, not just Strix Halo (#6915) ... * WSL ROCm: generalize ROCm-on-WSL bootstrap from Strix-only to any RDNA arch install_rocm_wsl_strixhalo.sh hardcoded gfx1151, so its verify step died on discrete Radeon cards even though the ROCm + librocdxg setup is arch-agnostic. Auto-detect the GPU arch from rocminfo (override via UNSLOTH_WSL_GFX), verify any GPU agent enumerates over DXG, and map the arch to AMD's per-arch wheel family for the optional smoke test (injecting librocdxg into torch/lib so torch's bundled ROCr finds the DXG bridge). Verified on gfx1200 (Radeon RX 9060 XT) in WSL2 + Ubuntu 24.04 -- torch.cuda now enumerates the GPU. * WSL ROCm: trigger the ROCm-on-WSL bootstrap for discrete Radeon GPUs too _maybe_bootstrap_rocm_wsl only fired for Strix APUs (matched via /proc/cpuinfo, which discrete cards don't appear in). Add _wsl_amd_gpu_name() -- queries the Windows host via WMI -- and broaden the trigger gate plus the 'already-usable ROCm' rocminfo check from gfx1151-only to any real GPU agent (gfxNNNN, excluding the gfx11-generic fallback ISA). The generalized bootstrap then auto-detects the arch. Enables 'curl install.sh | sh' to set up ROCm-on-WSL on discrete Radeon RX 7000/9000 in WSL2 + Ubuntu 24.04, not just Strix Halo/Point. * WSL ROCm: address review -- filter generic ISA in bootstrap, bound the host GPU query - install_rocm_wsl_strixhalo.sh: exclude the gfx11-generic fallback ISA in arch detection (grep -v generic), matching install.sh's rocminfo check, so a generic agent listed before the real one can't be picked as the arch. - install.sh: wrap the powershell.exe Win32_VideoController query in _run_bounded (10s timeout) so an unstable WSL-interop / busy host can't hang the installer. * WSL ROCm: harden arch-detect + librocdxg copy under set -eo pipefail (review) - _detected_gfx: append '|| true' so a no-GPU rocminfo (empty pipeline, non-zero under pipefail) doesn't abort the assignment before the '[ -z ]' branch prints the diagnostic + die message. - smoke-test librocdxg copy: gate on '[ -d "$_tlib" ]' instead of '[ -n ]' so a non-directory value can't make cp rename librocdxg to 'lib'. * WSL ROCm: address Codex review (gfx000, 24.04 reroute for discrete, test locator) - Exclude gfx000 (the CPU agent) from the WSL 'usable ROCm' check and the bootstrap arch-detect: match gfx[1-9] (nonzero arch), so a partial ROCm install that only reports the CPU ISA no longer short-circuits the librocdxg setup. (P2) - Reuse the Ubuntu-24.04 reroute for discrete Radeon: broaden _maybe_reroute_strixhalo_to_2404's gate with the same _wsl_amd_gpu_name (WMI) fallback, so a discrete card on 26.04 reroutes to a 24.04 distro like Strix does instead of falling to CPU. Moved _wsl_amd_gpu_name above the reroute and made it self-contained + 10s-bounded (it runs before _run_bounded is defined). (P2) - Update TestInstallShDropinPersistence to locate the gate by its unique '!/generic/' clause now that the gfx1151 literal is gone. (P1) * Condense ROCm-on-WSL comments in install.sh and bootstrap helper * Guard WSL reroute from NVIDIA hybrid hosts and fix GFX-override pipefail check * Honor CUDA_VISIBLE_DEVICES-hidden NVIDIA in the WSL reroute guard * Reuse _has_usable_nvidia_gpu in the WSL reroute guard --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com>	2026-07-07 02:29:37 -07:00
Daniel Han	69f8e0b228	Clear stale yolo approval state on no-launch reruns (#6868) ... * Clear stale yolo approval state on no-launch reruns The no-launch session config dir is deliberately reused across runs, but the config writers only ever added the --yolo auto-approval settings and never removed them. After one --yolo --no-launch run, every later run without --yolo kept OpenClaw's tools.exec security=full/ask=off policy plus exec-approvals.json, and OpenCode's permission allow block, so tool execution stayed silently pre-approved. Non-yolo runs now reset that state: OpenClaw drops the exec policy keys and the yolo defaults in exec-approvals.json (approvals OpenClaw itself recorded are kept; the file is removed when only the yolo payload is left), and OpenCode drops the permission block. Launch mode is untouched since it already uses an ephemeral temp dir. * Strip only yolo-written values on non-yolo cleanup Match each field against the exact value the yolo path writes before removing it, so a stricter exec policy, approvals defaults set by the user or the OpenClaw UI, and deny/ask OpenCode permission entries all survive a plain no-launch rerun. An unparseable exec-approvals.json is left in place, matching how an unparseable config is handled. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Write a prompting policy on non-yolo instead of deleting to a permissive default OpenClaw and OpenCode both treat an omitted policy as permissive: OpenClaw's effective exec policy for an unset tools.exec is security=full/ask=off on the gateway host, and OpenCode defaults an unset permission to allow. So clearing the yolo values on a non-yolo run did not restore prompting, it fell back to those permissive defaults and left tool execution auto-approved. A non-yolo run now writes an explicit prompting policy: OpenClaw gets security=allowlist/ask=on-miss (verified to prompt even with the approvals file removed, since the stricter of config and approvals wins), and OpenCode gets edit/bash/webfetch=ask. Only a permissive/yolo value is tightened; a stricter deny (or an ask the user set) is preserved, and the yolo approvals defaults are still stripped. The file-edit CI path opts opencode/openclaw into --yolo, since those agents now prompt by default and the headless test needs auto-approval. * Respect existing exec mode, sandbox/node host, and global permission rules on non-yolo reset The non-yolo reset for openclaw/opencode assumed an omitted policy was the permissive yolo default and rewrote it, which corrupted or weakened stricter setups it should have preserved: - OpenClaw tools.exec.mode is the normalized policy knob and cannot be combined with explicit security/ask (OpenClaw rejects the whole config), so writing security+ask alongside a mode:deny/ask policy both broke the config and relaxed it. Leave a mode-based policy untouched. - host=sandbox defaults to security=deny and host=node routes to a paired node; neither is written by --yolo (which only writes host=gateway). Treating the missing security as full and popping host broadened those into gateway/auto exec. Only rewrite a gateway-routed permissive policy, and never pop a non-gateway host. - OpenCode permission can be a string ("deny") or a {"*": ...} catch-all. The old code dropped a string form and overrode a catch-all by writing per-tool ask, weakening a stricter user rule. Now a string is left in place, a catch-all governs absent tools, and only an effective allow is tightened. - The non-yolo ask policy only lived in OPENCODE_CONFIG, which loads below project opencode.json, so a project config allowing edit/bash/webfetch still auto-approved. Carry the ask policy in OPENCODE_CONFIG_CONTENT (above project config) too, symmetric to how yolo carries its allow. Also harden the openclaw path against a malformed non-dict tools value. Adds tests for mode/sandbox/node hosts, string and catch-all permissions, and the inline ask policy over a project config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scope non-yolo resets to the exact yolo fingerprint and preserve granular denies OpenClaw: reset only the exact host=gateway/security=full/ask=off policy --yolo writes, so an omitted or host=auto/sandbox/node policy (which can resolve to a sandbox security=deny default) is no longer broadened to allowlist/on-miss, and a deliberate tools.exec.mode is left alone (OpenClaw never migrates our security/ask write into a mode). OpenCode: carry a granular object or a deny inline verbatim so a per-tool user rule is not collapsed to a blanket ask, but floor any object that grants allow anywhere to the string ask (which fully replaces a project object) so no inline allow pattern can leak through into a silent auto-approve on a non-yolo session. * Stop overriding project config on non-yolo; require full approvals fingerprint The non-yolo OpenCode reset carried a session permission in OPENCODE_CONFIG_CONTENT, which outranks the project opencode.json we cannot read. That inline override could not correctly reflect the project: it weakened a project deny to a prompt, mishandled global string rules, leaked through a granular object's permissive default when no catch-all was present, collapsed an object with an allow (losing its deny), and missed per-agent permissions. All of these stem from forcing a value over an unknown project config. A non-yolo run now only undoes what --yolo wrote: it flips our own explicit per-tool allow back to ask in our config file and carries no permission inline, so the project's own permissions are honored as written. Clearing our persisted yolo state is the actual fix; --yolo still carries its allow inline so it works over a project config. OpenClaw approvals cleanup now strips the yolo defaults only when the full fingerprint (security=full, ask=off, askFallback=full) is present, so a mixed user policy that merely shares askFallback=full (whose omitted default is deny) is kept intact. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-07 00:06:48 -07:00
Daniel Han	08226c2475	Studio: fix torch CUDA undefined-symbol errors from a conflicting LD_LIBRARY_PATH (#6905) ... * Studio: re-exec to prepend torch's bundled CUDA libs to LD_LIBRARY_PATH On Linux the dynamic linker reads LD_LIBRARY_PATH before the RUNPATH baked into torch's .so files, so a pre-existing LD_LIBRARY_PATH pointing at a system CUDA (conda, a Docker base image, /usr/local/cuda-*/lib64) shadows torch's bundled nvidia/*/lib libraries and causes undefined-symbol errors when the Studio backend imports torch. Detect torch's lib dirs without importing torch, prepend them to LD_LIBRARY_PATH, and re-exec once (LD_LIBRARY_PATH is only read at process start). Linux-only, sentinel-guarded against re-exec loops, and called only from run.py's __main__ so library/embedder imports (e.g. Colab's `from run import run_server`) are never re-exec'd. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 23:48:23 -07:00
Michael Han	af93868760	Fix repeated base model downloads across checkpoint exports (#6896) ... * Fix repeated base model downloads across checkpoint exports (#6890) Pre-warm the HF hub cache with the 16bit base weights before merge_and_overwrite_lora runs. The merge fetches shards with hf_hub_download(local_dir=...), which never populates the hub cache, so temporary merge directories (GGUF checkpoint exports) forced a full re-download of the base model for every checkpoint. The first export now downloads once into the cache and later exports copy from it. Skips itself when already cached, offline, on Kaggle/Colab, for local or nf4/fp4 bases, non-downloading save methods, or low disk. Opt out with UNSLOTH_PREWARM_HUB_CACHE=0. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Show MB for small base models in the pre-warm download message * Harden pre-warm: getattr for model config, abspath for relative HF_HUB_CACHE - Read config._name_or_path via getattr so a model without a config skips cleanly instead of taking the outer error path. - abspath the cache probe so a relative HF_HUB_CACHE walks up to a real root rather than "", which would zero the free-space check and skip pre-warm. Both from PR review; each covered by a test that fails without the fix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pre-warm the live-env HF cache so runtime redirects still hit (#6890) Resolve the hub cache the same way the merge does (unsloth_zoo _active_caches, live env) instead of huggingface_hub's import-time-frozen constants.HF_HUB_CACHE, and pass it as cache_dir to the cached probe, disk check and snapshot_download. Without this, a runtime HF_HOME/HF_HUB_CACHE redirect (unsloth_zoo redirect_hf_cache_if_readonly on a read-only default cache, or Studio) makes the pre-warm populate a different directory than the one the merge reads, so the cache-copy fast path misses and the base re-downloads on every export anyway. Adds 3 regression tests covering the cache_dir threading and the redirect case. * Apply ruff-format kwarg spacing to the pre-warm cache-dir changes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pre-warm the 16bit sibling for FP8 bases so their merged_16bit exports reuse the cache too For a merged_16bit export of an FP8 base with an existing 16bit sibling, the merge swaps to the sibling and downloads that (unsloth_zoo _resolve_fp8_16bit_sibling), so pre-warming the FP8 repo missed the cache and re-downloaded the sibling every export. Mirror the swap and pre-warm the sibling. No sibling still caches the FP8 repo for the in-place dequant path. Adds 2 regression tests. * Filter pre-warm shards through the safetensors index like the merge does Repos that ship a leftover shard set the index does not reference (e.g. granite-3.2) made the disk gate over-count and snapshot_download fetch shards the merge never reads. Mirror the merge: on the download path, keep only index-referenced shards. Runs after the already-cached check so the cached fast path stays network-free. Adds 2 tests. * Tighten pre-warm comments --------- Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>	2026-07-06 23:38:29 -07:00
Daniel Han	5608081c35	Studio: apply presence_penalty on the safetensors and MLX inference paths (#6923) ... * Studio: apply presence_penalty on the safetensors and MLX inference paths The safetensors and MLX generate paths resolved the inference config and then dropped presence_penalty before generation, so the same model applied the configured value under GGUF and 0 under safetensors/MLX. Thread the already-resolved presence_penalty through the orchestrator command, worker gen_kwargs, and the safetensors/MLX generate calls, and apply it with a small logits processor (subtract once per distinct completion token, prompt excluded, presence not frequency, zero is a no-op, negatives raise). Backwards compatible: presence_penalty defaults to 0.0 (byte-identical output when unset) and the GGUF path is unchanged. Also forward min_p on the legacy /generate/stream route and add the missing min_p field to GenerateRequest. * Studio: bound presence_penalty generated ids to valid vocab range on both paths The presence-penalty logits processors index by generated token ids. The torch path filtered only the upper bound (seen < vocab_size), so a negative id would silently wrap to the wrong row; the MLX path had no bound at all, and MLX out-of-bounds indexing is documented undefined behavior (crash or memory corruption on Apple Silicon), unlike torch's harmless negative wrap. Bound generated ids to [0, vocab) consistently on both paths: - torch: seen[(seen >= 0) & (seen < vocab_size)] (zero-regression safety net; real completion tokens are always in range). - MLX: route out-of-range/negative ids to a discarded scratch slot via mx.where and a (vocab + 1)-wide scatter-assign mask, then subtract. MLX has no boolean-mask filtering (data-dependent output shape), so this keeps a fixed shape, stays on-device, and preserves once-per-distinct-token semantics without any torch/numpy dependency. Add torch tests for out-of-range and negative ids (only in-range distinct ids penalized, stray ids ignored, no wrong-index wrap) and a bound-documenting MLX test that runs on the arm64 macOS CI. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 22:24:47 -07:00
Daniel Han	9674e882c2	Studio: serialize the compare-mode dispatcher lifecycle to fix a start race (#6922) ... * Studio: serialize the compare-mode dispatcher lifecycle to fix a start race _generate_dispatched (compare mode) bypasses _gen_lock so two concurrent compare requests can both reach _start_dispatcher. The check-then-spawn there had no lock, so both could observe no live dispatcher and each spawn one. The extra dispatcher is orphaned (self._dispatcher_thread tracks only the last) and during a later unload it can consume the 'unloaded' reply off _resp_queue before unload_model's _wait_response, hanging the unload on its timeout. Add _dispatcher_lifecycle_lock and take it around the whole body of both _start_dispatcher and _stop_dispatcher, so start/stop cannot interleave and the second concurrent starter sees the dispatcher alive and returns. _start_dispatcher now returns whether it actually spawned the thread, and _generate_dispatched derives dispatcher_preexisting from that atomic result instead of a separate unlocked is_alive() read. No call site holds _mailbox_lock when calling start/stop, so joining the dispatcher (which takes _mailbox_lock) under the new lock cannot deadlock; the lock order is always _gen_lock then _dispatcher_lifecycle_lock and is never inverted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: refuse dispatcher start queued behind an unload's stop A compare request could pass the early _unload_pending check, then block in _start_dispatcher on _dispatcher_lifecycle_lock behind an unload's _stop_dispatcher. When the unload released the lock the start spawned a fresh dispatcher, which became the resp_queue reader and consumed the worker's unroutable 'unloaded' reply before unload_model's _wait_response saw it, hanging the unload for 300s. Gate _start_dispatcher on _unload_pending under the lifecycle lock, and set _unload_pending under the same lock ahead of the stop, so any start queued behind the stop observes the unload and refuses. Ordering stays _gen_lock -> _dispatcher_lifecycle_lock. Adds a regression test forcing the queued-behind-stop interleaving. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 22:09:41 -07:00
Daniel Han	3506371677	Studio: keep the nudge wiring test collectable without the unsloth stack (#6924) ... test_nudge_tool_calls_wiring.py imported InferenceBackend from core.inference.inference, which pulls in unsloth (and thus unsloth_zoo) at module scope. The dependency-light backend CI matrix job does not install unsloth_zoo, so the import raised at collection time and aborted the whole job (831 tests never ran). Guard that one import and fold the safetensors InferenceBackend checks in only when the unsloth stack is importable; the orchestrator/llama_cpp/safetensors_agentic wiring is still asserted unconditionally, and local/full-stack runs keep the InferenceBackend coverage.	2026-07-06 21:57:56 -07:00
Daniel Han	46ab683065	Studio: client-tool passthrough healing for safetensors and MLX (#6870) ... * Studio: client-tool passthrough healing for safetensors and MLX PR 6801 made response-side tool-call healing default-on for the client-tool passthrough, but only on the GGUF path: the passthrough branch in /v1/chat/completions is gated on using_gguf, and the safetensors section never reads payload.tools, so a client-tools request against a safetensors or MLX model silently dropped the tool schemas and returned prose with no tool_calls. Add the missing leg. When a non-GGUF model is loaded, the request declares client tools (or carries tool-role history), server-side tools are off, and the template supports tools, the route now: - renders the tools into the chat template for a single turn via the existing backend.generate_chat_response(..., tools=...) seam (worker templating already accepts role=tool and assistant.tool_calls messages, normalized with _openai_messages_for_passthrough); - non-streaming: promotes text-form calls with heal_openai_message, honors the opt-in nudge single retry (nudge_should_retry / nudge_messages), caps healed calls when parallel_tool_calls=false (covers the nudge retry too), and sets finish_reason=tool_calls with content null on a pure tool-call turn; - streaming: derives deltas from the worker's cumulative snapshots and feeds StreamToolCallHealer, emitting healed tool-call deltas and the correct finish chunk, guarded against repeated or shrinking snapshots. heal_gate semantics are identical to the GGUF passthrough: default on, auto_heal_tool_calls=false or UNSLOTH_DISABLE_TOOL_CALL_HEALING=1 relays verbatim, tool_choice narrows promotion, undeclared names stay text. MLX rides the same orchestrator seam, so both local backends gain the behavior. CompletionMessage.content becomes Optional so a promoted pure tool-call turn matches the OpenAI contract (content null when only tool_calls return). Adds tests/test_sf_client_tools_passthrough.py (22 cases: healing, gating, opt-outs, streaming deltas, tool-role history, dict-arguments history, forced tool_choice, parallel cap, usage, nudge on/off/double-failure, generator error hygiene, disconnect reset, empty output, MLX path). * Address review: tool_choice none, developer folding, retry fallback, monitor reply Four review follow-ups on the safetensors/MLX client-tool passthrough leg: - tool_choice="none" keeps the tool-history templating but no longer advertises the tools, so a forced final-answer turn is not prompted into emitting markup that the (correctly disabled) healer would relay as prose. Mirrors the GGUF passthrough where llama-server honors tool_choice itself. - OpenAI "developer" messages fold into a single leading system message via _set_or_prepend_system_message before templating; local templates reject the role and the fallback formatter drops it. - A nudge retry that fails or is cancelled after the original answer exists falls back to the first response instead of surfacing a 500, matching the GGUF nudge path. - The API monitor records the healed tool call summary instead of the raw markup on a promoted turn. Adds four regression tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: forced tool_choice templating, content-part flattening, stream monitor parity - A forced tool_choice function is now the only schema rendered into the local template, so the advertised tools and the healer allowlist can no longer disagree (llama-server enforces tool_choice itself on the GGUF path). - Content-part lists are flattened to their text parts before templating. Remote image URLs are not decodable locally, so such requests reached this path with part lists that raise inside apply_chat_template on text-only templates; the plain non-GGUF path has always flattened them. - The streaming monitor entry is now fed from the healed events the client actually receives, recording promoted calls as the [tool_calls] summary the non-streaming path records. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: gate passthrough on the engaged server path, deserialize templated arguments - The client-tools gate now keys on _sf_use_tools (whether the server-side tool path actually claimed the request) instead of the raw mcp_enabled flag: with an empty MCP registry or a CLI --disable-tools policy, a client that sets mcp_enabled while declaring its own tools fell through to plain generation with the tools silently dropped. The GGUF passthrough gate has no mcp_enabled clause either. - New _structured_tool_history_for_local_template deserializes assistant tool_calls[].function.arguments JSON strings into mappings for the templated copy only: spec-compliant clients send strings, but local chat templates iterate arguments as a mapping or raise on strings, which crashed or misrendered multi-turn tool history. The HTTP response and the GGUF wire shape keep strings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten comments and docstrings in the client-tools passthrough * Report first-attempt usage when a nudge retry is discarded When nudge_should_retry fires but the retry produces no healable tool call (or raises), the first response is still delivered to the client. The retry's generate() had already overwritten stats_holder, so _monitor_usage recorded the unseen retry's token counts against the request instead of the first attempt that was actually returned. Capture the first attempt's stats before the retry and restore them on both the no-heal and exception paths so the monitor reports the usage of the response the caller received. * Do not promote buffered tool markup when a stream is cancelled The streaming client-tool heal path breaks out of the token loop when cancel_event is set (the registry "Stop" path), but then still fell through to healer.finalize(), which heals incomplete tool markup at EOF (allow_incomplete) and emits a tool_calls delta plus finish_reason=tool_calls. Because the Stop request only sets the event and leaves the SSE socket open, the client received that promoted call and executed a tool the user had just cancelled. The disconnect path already returns before finalize; guard finalize and the finish_reason on cancel_event too, so a cancelled stream ends with finish_reason=stop and no tool call. Adds a regression test driving a Stop mid-emission with buffered markup. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim comments in the client-tools passthrough * Trim client-tools passthrough comments further * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 19:48:36 -07:00
Daniel Han	8ba46b566a	Studio: close switch/cancel races during model load (#6918) ... Fix six race conditions when a user switches or cancels a model while a previous load or generation is still in flight, across the inference orchestrator and the /load and /unload routes: - Cancel an in-flight generation on a safetensors/MLX model switch and serialize unload with load under the inference lifecycle gate. - Cancel an in-flight load off the lifecycle gate so a Stop-loading cancel does not wait out the multi-minute load; guard the dispatched mailbox against a racing unload. - Recheck the loading marker after spawn and again after the load response before publishing, so a load cancelled mid-flight is reaped instead of going live. - Discard the loading marker before tearing the subprocess down in cancel_load, closing a spawn-after-cancel window and an orphaned compare-mode dispatcher during unload. - Match the unload target before canceling an in-flight GGUF load and add an off-gate fast path for the still-loading GGUF case. - Run the Unsloth unload off the event loop so a paused SSE stream holding _gen_lock cannot block the loop. Adds studio/backend/tests/test_orchestrator_unload_cancel.py covering the unload/cancel/switch race paths.	2026-07-06 19:43:15 -07:00
Daniel Han	9dabe96786	Studio chat: tool-call nudging on by default (API stays opt-in) (#6883) ... * Studio chat: tool-call nudging on by default (API stays opt-in) Healing is already default-on everywhere and the nudge retry from the client-tool passthrough is opt-in on the API. Studio chat had neither signal: the frontend never sent nudge_tool_calls, and the safetensors and MLX server-side loop lacked the GGUF loop's plan-without-action re-prompt entirely. Backend: the re-prompt helpers move from llama_cpp.py into tool_call_parser.py (shared, cycle-free; the GGUF loop imports them under its old names with zero behavior change) and run_safetensors_tool_loop now re-prompts once at the streaming no-tool-call exit, gated on Auto-Heal, active tools, nothing executed yet, and short forward-looking text. Re-prompts do not consume tool iterations. Frontend: the chat adapter sends nudge_tool_calls from a new nudgeToolCalls runtime setting (default true) with the same persistence, hydration, and settings toggle plumbing as Auto-Heal. Request-model defaults are untouched, so raw API callers stay opt-in. * Address review: persist the nudge setting, consume the flag in the loops, skip the re-prompt after RAG autoinject ChatSettingsPayload uses extra forbid, so a settings patch containing nudgeToolCalls failed to persist any settings; the field is now typed and round-trips. nudge_tool_calls now plumbs into both server-side tool loops and gates the plan-without-action re-prompt with None meaning on, so API callers keep today's behavior, explicit false disables it, and Studio's default-on flag actually controls the path Studio chat runs. The safetensors loop no longer re-prompts after RAG autoinject: the injected retrieval bypasses the tool controller, so the nothing-executed gate saw an empty history and re-asked after a successful retrieval. * Safetensors loop: the plan-without-action retry requires an explicit nudge flag The retry is new on this loop, so an omitted nudge_tool_calls must not change existing API behavior; Studio opts in explicitly. The GGUF loop keeps None as on because its re-prompt predates the flag. * Suppress the plan-without-action re-prompt after a denied tool confirmation A denial appends TOOL_REJECTED_MESSAGE but records nothing in the tool controller history, so the nothing-executed gate re-prompted the model to call the tool the user had just rejected, producing another confirmation prompt. A denial now suppresses the re-prompt for the rest of the request, mirroring the RAG autoinject handling. * Tighten plan-without-action re-prompt comments * Tighten plan-without-action re-prompt comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: match unified plan-without-action nudge cap to GGUF default of 3 The shared MAX_ACT_REPROMPTS was set to 1, but GGUF's established default (llama_cpp.py) has re-prompted a stalling model up to 3 times since #5620. Restore the GGUF-matched cap so safetensors and MLX inherit the same behavior, and update the safetensors cap test to assert the cap dynamically. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 19:41:19 -07:00
Daniel Han	c2a7b78f6b	Studio: exclude mlx-lm 0.31.3 (broke gemma4/qwen3_5 QK-norm load on Apple Silicon) (#6803) ... * Studio: exclude mlx-lm 0.31.3 (broke gemma4/qwen3_5 QK-norm load) mlx-lm 0.31.3 regressed the QK-norm archs: its strict load_weights rejects the q_norm/k_norm tensors with "Received N parameters not in model", so gemma4 and qwen3_5 checkpoints fail to load. Studio installs the MLX stack unpinned at latest, which pulls 0.31.3. Verified on a real macos-14 runner: gemma4 fails to load on 0.31.3 but loads and generates coherently on 0.31.2 and on git-main (future 0.31.4). See mlx-lm #1242. Exclude just that release (!=0.31.3) in the installer and the self-heal floor so --upgrade still resolves to the newest good build, and treat an already-installed 0.31.3 as unsatisfied so the self-heal replaces it. * Studio MLX: cover fresh-install path + robust bad-version compare Address PR review: - Fresh install.sh (Apple Silicon) runs the base 'uv pip install unsloth' with SKIP_STUDIO_BASE=1, skipping the guarded MLX-stack step, so transitive resolution could still pull mlx-lm 0.31.3. install.sh already exports UV_OVERRIDE -> overrides-darwin-arm64.txt before that install, so exclude mlx-lm 0.31.3 there too; this also strengthens the self-heal (same override). - Match the known-bad version with parsed packaging.Version so 0.31.3 == 0.31.3.0 (trailing-zero normalization) instead of raw string equality. * Studio: exclude mlx-lm 0.31.3 on the fresh Apple Silicon install too The overrides file only applies via UV_OVERRIDE when it exists relative to the script, which is not true for a curl-piped install, and the guarded MLX step in install_python_stack.py is skipped there (SKIP_STUDIO_BASE=1). So the base install could still resolve the transitive mlx-lm to the broken 0.31.3. Append mlx-lm!=0.31.3 to the base install on Apple Silicon (empty elsewhere), so the fresh path pins away from 0.31.3 without waiting for the runtime self-heal. * Studio: exclude mlx-lm 0.31.3 on the migrated install; keep the >=0.22.0 floor The with-deps migrated install did not append ${_MLX_LM_EXCLUDE_ARG:-}, so a curl-piped Apple Silicon migration (no repo overrides file, UV_OVERRIDE unset) could resolve mlx-lm 0.31.3 transitively. Append the exclusion there, matching the fresh install path. The no-torch migration is left alone since --no-deps never resolves mlx-lm (same as the fresh no-torch path). Also restore the >=0.22.0 floor in overrides-darwin-arm64.txt: a uv override replaces the transitive constraint, so a bare !=0.31.3 could let the resolver drop below the supported minimum that mlx_repair.py enforces at runtime. * Triage huggingface_hub 1.22.0 / fastapi / multiprocess scanner false positives The scan-packages gate red-failed on all three shards after transitive deps bumped. Every new CRITICAL is a benign false positive, verified against upstream: - huggingface_hub 1.22.0 added _sandbox.py for the remote HF sandbox feature. Its job-startup bootstrap string (fetch sbx-server into the container /tmp and exec it) and the SandboxPool host-reservation loop trip the staged-dropper and C2-loop heuristics; that script runs inside a remote HF container, not on the user machine. The bump also re-hashed the already-reviewed benign polling loops in hf_api.py and utils/_http.py. The PyPI artifact is byte-identical to the official v1.22.0 tag. - fastapi 0.139.0 routing.py re-hashed the websocket keepalive while-True loop; byte-identical to upstream 0.139.0. - multiprocess 0.70.19 forkserver.py and tests/__init__.py re-hashed the AF_UNIX fork-server IPC and fd-inheritance tests; genuine uqfoundation release, local IPC not network. Added 7 reviewed allowlist entries (no blind regenerate). All three shards (hf-stack, studio, extras) exit 0 locally. * Tighten mlx-lm 0.31.3 exclusion comments * Trim mlx-lm 0.31.3 exclusion comments	2026-07-06 19:40:06 -07:00
Daniel Han	f109e7f0e6	Studio: parse Mistral [TOOL_CALLS] and rehearsal tool-call shapes (#5704) ... * Studio: parse Mistral [TOOL_CALLS] and rehearsal tool-call shapes Extends the rescue parsers in core/tool_healing.py and core/inference/tool_call_parser.py to recognise two extra serialisations local models commonly emit when bypassing native function calling: * [TOOL_CALLS]name{json_args} (Devstral-Small-2, Mistral-Small-3.x). * name[ARGS]{json_args} (reasoning-model rehearsal). Both extractors use a brace-balance scan that honours escapes and quoted strings so nested JSON args stay intact. Also pre-strips <think>...</think> and [THINK]...[/THINK] blocks before matching so calls emitted after a reasoning preamble are recognised regardless of position. Streaming gates (TOOL_XML_SIGNALS, llama_cpp.py _TOOL_XML_SIGNALS) and the SSE strip regex (routes/inference.py _TOOL_XML_RE) gain the new sentinels so the parser is actually invoked and the raw markup never leaks to the UI. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Strip unclosed think blocks and catch rehearsal [ARGS] mid-buffer The pre-existing ``_THINK_TAG_RE`` only matched closed thinking blocks (``<think>...</think>`` or ``[THINK]...[/THINK]``). During streaming the model is still inside the open block when the parser runs, so any tool-shaped markup the model is REHEARSING inside that block survived the strip and could be executed as a real call. Switch both copies of the regex (parser + healing) to accept the trailing block being terminated by end-of-string in addition to the explicit closer. The ``_TOOL_XML_SIGNALS`` list on the llama_cpp streaming buffer included ``[ARGS]`` to catch rehearsal syntax, but the gate used a ``startswith`` check against the buffer head -- rehearsal is shaped ``name[ARGS]{json}``, so the buffer never STARTS with ``[ARGS]`` and the signal had no effect. Add a substring fallback for the bracket-style signals so the BUFFERING window can still divert the stream into DRAINING when rehearsal markup arrives mid-buffer. Adds three regression tests covering rehearsal inside unclosed ``<think>`` / ``[THINK]`` blocks (must yield no calls) and the positive case after a closed think block (still parsed). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden bracket-tag tool-call parsing and streaming strip Address review findings on the Mistral [TOOL_CALLS] / rehearsal [ARGS] paths: - Accept hyphenated tool names in the bracket parsers and strip patterns. _MISTRAL_BRACKET_RE and _REHEARSAL_RE used \w+, which dropped or truncated MCP function names containing dashes (mcp__srv__list-issues). Use [\w-]+ to match the XML and Gemma parsers. - Strip a partial bracket marker streamed before its opening brace. The trailing-unclosed patterns required the {, so a [TOOL_CALLS]web_search or python[ARGS] split across deltas leaked the raw marker to the UI. Match the bare marker to end-of-text, mirroring how the bare open tags are stripped. Closed pairs are unchanged so in-progress markup stays buffered until parsed. - Strip a truncated bracket tail in the route-level display regex. _TOOL_XML_RE required a balanced JSON object; a tool call truncated by EOS now strips up to \Z, like the orphan-opening XML shapes. Complete calls still strip only their balanced JSON so following prose survives. Add regression tests for hyphenated names, the streaming partial-marker strip, and the unclosed-tail route strip. * Studio: preserve XML parameter indentation in tool_healing The chat template emits <parameter=k>\nVALUE\n</parameter>; the parameter-start regex consumed the wrapping newline AND the value's first-line indentation via a trailing \s*, then str.strip() removed the rest, corrupting code/diff arguments. Narrow the trailing class to horizontal whitespace and trim exactly one wrapping newline (_trim_param_value), preserving indentation. Matches SGLang's qwen3_coder detector and the same fix on the multi-format parser. Add a regression test. * Studio: tighten Mistral/rehearsal tool-call comments Compress the comments in the Mistral [TOOL_CALLS] / rehearsal [ARGS] healing shim and its callers to one or two lines, keeping the bracket-tag stripping rationale, the thinking-block handling note, and the forge attribution intact. Comment-only: no code or behavior change (verified with comment_tools.py check --strip-docstrings; tests green). * Studio: fix think-strip arg corruption and nested bracket-JSON strip Review follow-up for the Mistral/rehearsal healing shim: - The <think>/[THINK] strip ran unconditionally over the whole content before parsing, so a real tool argument that legitimately contained a <think> / [THINK] literal was silently corrupted. Don't delete the blocks: compute the reasoning-block spans and skip any tool-call candidate that STARTS inside one, across all parse paths (JSON, Gemma, XML, bracket, rehearsal). A rehearsed call inside reasoning is still ignored; a real call after </think> still parses. - The bracket-tag display strip used a fixed one-level-nesting regex, so a call with two-level-nested JSON args either leaked raw markup or, in final mode, let the catch-all eat the trailing prose. Add a balanced-brace _strip_bracket_tag_calls pass (any nesting depth) used by strip_tool_call_markup and the route display strip. Add regressions: <think>/[THINK] literal inside a real argument, rehearsal-inside- think with a real call after, and two-level-nested bracket/rehearsal strip keeping trailing prose. * Studio: correct think-block comments to match span-skip behavior The think-strip fix replaced the unconditional think-block strip with a span-skip (the block is kept and any tool-call candidate starting inside it is ignored), but two comments still described the old strip-first behavior. Update the _THINK_TAG_RE comment and the parse_tool_calls_from_text docstring. * Studio: parse Mistral arrays and call-ids, unify bracket parse/strip, keep it linear - Parse the canonical Mistral array form (TOOL_CALLS followed by a JSON list of calls) and emit every call; parse the v11 shape that carries an opaque CALL_ID token between the name and ARGS (the function name is the token after TOOL_CALLS, never the call-id); and parse a Mistral call plus a rehearsal call in one message (the second was dropped yet still stripped from display). - One shared balanced forward scan (_iter_bracket_spans) backs both the parser and the strip path, so they no longer diverge. It is linear: each regex is re-searched only once its cached match falls behind the cursor, replacing the per-match full-tail re-scan that was O(n^2) (O(n^3) over a stream). A length cap before the scan is a backstop. - strip_tool_call_markup preserves think/reasoning blocks verbatim (the parser skips tool markup inside them), stripping only the visible text around them. - _in_think uses bisect over the sorted think spans (was a linear scan per candidate). - GGUF streaming strip runs the balanced bracket pre-pass before the regex patterns so nested-arg calls do not leak or eat trailing prose, and the BUFFERING ARGS detector requires the rehearsal name-ARGS shape. - Tests: canonical array, array string-args, array strip keeps prose, Mistral plus rehearsal multi-call, v11 call-id name, think-rehearsal strip preservation, and bracket-strip linearity. * Studio: preserve reasoning blocks in the route and streaming strip paths too Addresses Gemini/Codex review: making strip_tool_call_markup preserve think blocks left the route display strip and the GGUF streaming strip inconsistent, so a rehearsed call inside a reasoning block was still deleted from the visible text on those paths. - Extract the think-block segmentation into one shared helper (strip_outside_think) and route all three strip paths through it: strip_tool_call_markup, _strip_tool_xml_for_display, and the GGUF _strip_tool_markup_streaming closure. - Add a route-strip regression test that a rehearsal inside a reasoning block is preserved while a real call outside it is still stripped. * Studio: fix bracket-tag strip/buffer review findings Address the live code-review findings on the Mistral bracket-tag / rehearsal tool-call rescue path: - tool_healing: a literal think block inside a tool-call argument is no longer treated as a reasoning block. strip_outside_think now excludes think spans that sit inside a complete tool-call span, so the call is stripped whole instead of the split hiding its open/close pair and leaking the raw call. - tool_healing: the rehearsal trailing-strip pattern requires a following brace or end-of-text, so prose that merely mentions name[ARGS] is not truncated as a phantom call. The bracket strip patterns are aligned with the parser regexes (whitespace, v11 [CALL_ID]/[ARGS] metadata, and the [CALL_ID] lookbehind). - routes: strip a truncated canonical Mistral array ([TOOL_CALLS] [{... with no closing bracket) that the balanced scan cannot remove, align the display regex with the parser regexes, and apply the same rehearsal-prose guard. - safetensors loop: mirror the GGUF [ARGS] rehearsal-substring check during BUFFERING so a rehearsal name does not stream before its [ARGS] arrives. Adds regression tests for each; existing parser suite stays green. * Studio: hold split rehearsal tool-name prefix in both streaming loops A reasoning-model rehearsal call can stream the tool name and its [ARGS] arm in separate chunks (web_search then [ARGS]{...}). The buffering detector only recognised the rehearsal once [ARGS] was present, so the bare tool name was emitted as visible content before the call drained and executed. Add _is_rehearsal_prefix (mirrored in the safetensors loop and the GGUF loop): when a no-signal buffer is a bare active-tool name -- or a partial prefix of NAME[ARGS] -- hold it as a prefix instead of streaming it, so the next chunk's [ARGS] flips it to a drain. A whitespace in the buffer means prose, not a split call, so ordinary text still streams. Adds regression tests for the split rehearsal in both loops and a guard that a plain non-tool word still streams. * Studio: route Anthropic tool-call cleanup through the protected display strip The Anthropic stream, non-stream, and passthrough paths cleaned content with raw _TOOL_XML_RE.sub instead of _strip_tool_xml_for_display, so a rehearsal call inside <think> was deleted from the reasoning and a nested [TOOL_CALLS] call dropped its trailing prose (the OpenAI-compatible paths already use the helper). Route all four sites (prior-assistant cleanup, streaming content events, non-stream aggregation, passthrough conversion) through the protected helper, and add a source-level guard test so raw _TOOL_XML_RE.sub stays confined to the helper itself. * Studio: stop split rehearsal tool names leaking once streaming, uncapped, or unrestricted The split-rehearsal guard (NAME in one chunk, [ARGS]{...} in the next) only held the name in the initial BUFFERING state. Three gaps remained where the bare tool name still streamed as visible content before the call drained: - STREAMING: after prose had already streamed, both loops emitted a trailing active-tool-name token (and the GGUF/safetensors [ARGS] boundary was not pulled back over the name). Hold the trailing rehearsal token and release it on the next chunk, with an end-of-stream flush so a plain answer that merely ends on a tool-name word is never dropped. - Buffer cap: a realistic MCP name longer than the 32-char _MAX_BUFFER_CHARS cap defeated the BUFFERING hold. A rehearsal prefix is self-bounding (it stops matching once it grows past NAME[ARGS]), so the generic cap no longer applies to it. - Unrestricted mode (tools=[]): with no declared tool list, any bare identifier may be a NAME[ARGS] rehearsal, so the prefix check now recognises one instead of leaking the name and mis-parsing the call. Regression tests cover the streaming, long-name, and unrestricted cases plus the plain-prose paths that must not be held or corrupted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio tools: protect think blocks in safetensors streaming, hold split rehearsal on initial flush, advertise Mistral tools Pass-3 review follow-ups on the Mistral [TOOL_CALLS] / rehearsal [ARGS] work: - Safetensors streaming display strip now preserves think / [THINK] reasoning verbatim (routes through strip_outside_think like the GGUF path). A call rehearsed inside a reasoning block was stripped mid-stream and then restored by the final strip, a non-monotonic shrink/grow that corrupted append-by-length stream consumers and the visible reasoning. - The first flush out of BUFFERING (safetensors and GGUF) now applies the same trailing-name hold the STREAMING branch uses, so a split rehearsal (prose plus a trailing active tool name in one chunk, [ARGS]{...} in the next) no longer leaks the bare name before the call drains. - Safetensors capability gate no longer suppresses tools for Mistral [TOOL_CALLS] templates, which the shared bracket-tag parser now handles end to end. Llama python_tag stays suppressed (still unparseable). - Route display strip applies the open-ended / bare-marker tail arms only on the segment after the last reasoning block (closed-only regex before it), matching strip_tool_call_markup, so a bare foo[ARGS] before a reasoning block is preserved while complete calls are still removed in every segment. Adds regression tests for each and updates the now-stale Mistral capability test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix tool-call think-marker and bracket-wrapper edge cases Round-1 review follow-ups on the Mistral/rehearsal tool-call healing: - tool_healing: a reasoning marker that opens INSIDE a tool call's arguments is argument data, not a reasoning block. Add _think_spans_outside_tool_markup (start-inside test) and use it in both parse_tool_calls_from_text and strip_outside_think so a literal marker in one call's args no longer hides a later call (parse) or leaks the raw markup (strip) when the greedy match runs past the call's closer. - tool_healing: strip the orphan Mistral v11 [/TOOL_CALLS] closer left behind after the balanced scan removes the call body. Add a route arm for the same closer in _TOOL_XML_RE / _TOOL_XML_CLOSED_RE. - safetensors + llama_cpp streaming strip: run the open-ended (EOS anchored) tail patterns only on the last segment; segments before a reasoning block use the closed-only patterns, matching the final strip and the route strip. A bare foo[ARGS] before a reasoning block is prose, not a truncated call. - safetensors streaming detector: validate each [ARGS] hit before draining. A bare foo[ARGS] in prose (no active tool name in front) no longer drains the rest of the turn; a later real NAME[ARGS] call is still found and the prose in between is preserved. Regression tests added for each case across the parser, strip helpers, and both streaming loops. * Strip incomplete-XML tool markup with literal think tags; widen render-html detector Round-2 review follow-ups. - tool_healing: an UNCLOSED <tool_call> / <function= call that the parser still executes via allow_incomplete leaked its markup when an argument contained a literal think marker. _tool_call_markup_spans only covered closed calls, so the literal was treated as a reasoning block to preserve. Extend it to the open-ended XML tail forms (shared as _TOOL_OPEN_XML_TAIL_PATS) so a think marker inside an unclosed call is argument data and the call's markup is stripped. A complete call's opener stays bounded to its closed span, and a real reasoning block with no tool call is still preserved. - safetensors render-html provisional card: _detect_render_html_tool_start was XML-only, so a Mistral [TOOL_CALLS]render_html or rehearsal render_html[ARGS] call executed but skipped the early card. Detect the earliest tool-call marker across every serialization the loop executes and fire when it is render_html. Regression tests added for both. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio tools: gate [ARGS] on active tools and skip think-block render_html rehearsal Round 3 review fixes for the Mistral / rehearsal tool-call parsing path. Both are asymmetric-fix bugs where one code path applied a guard the analogous paths did not. - [ARGS] active-tool gating: the streaming state already validates a rehearsal NAME[ARGS] against the active tool list before draining, but the BUFFERING detection and the end-of-stream safety-net checks (safetensors and GGUF) treated any word[ARGS] substring as a tool boundary. An answer containing a literal foo[ARGS]{...} in prose, where foo is not an enabled tool, was drained, parsed into a disabled foo no-op, and forced an extra generation turn. Gate those checks on the active tool name too (unrestricted mode still accepts any name), so inactive-name prose is neither drained nor parsed. Adds a shared _has_genuine_tool_signal helper (safetensors) and _gguf_rehearsal_signal_pos / _gguf_has_genuine_tool_signal (GGUF). - render_html provisional card vs think blocks: the parser skips tool candidates that start inside a <think>/[THINK] reasoning block, but the provisional render_html detector scanned raw content. A render_html rehearsed inside <think> followed by a real non-render_html call emitted a provisional render_html tool_start (reusing the later call's id) that the loop never executed. Drop candidates that start inside a think span and use the first marker of each shape outside the blocks. Also resolve the [TOOL_CALLS] [{...}] array shape through the parser so a nested "name" argument key no longer fires a false provisional card ahead of the real top-level tool name. Adds regression tests for both loops: inactive-name foo[ARGS]{...} is not drained into a disabled no-op or a retry turn, a think-block render_html rehearsal emits no provisional card, and the array top-level name is read correctly. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gate ambiguous bare-rehearsal parse and strip on the active tool list A bare NAME[ARGS]{json} is a genuine rehearsal call only when NAME is an active tool; otherwise it is prose. The earlier round gated only detection (so an inactive foo[ARGS] no longer drained the buffer or forced a retry turn), but the parse and strip stayed unrestricted, which produced two regressions: 1. An inactive foo[ARGS]{...} placed immediately before a real web_search[ARGS]{...} in the same content span made the real call fail to execute (parse consumed the phantom foo call). 2. An inactive foo[ARGS]{...} in a prose answer had its markup stripped from the visible text, corrupting the sentence to " is just syntax." Thread enabled_tool_names through the shared parser/strip so parse and strip apply the SAME active-tool gate as detection: - core/tool_healing.py: _iter_bracket_spans skips an inactive rehearsal span; parse_tool_calls_from_text, _strip_bracket_tag_calls, _strip_markup_segment and strip_tool_call_markup accept and thread the gate; apply_tool_strip_patterns keeps an inactive rehearsal match. - core/inference/tool_call_parser.py: wrappers forward the gate. - core/inference/safetensors_agentic.py and core/inference/llama_cpp.py: compute the gate from the active tool list (None when unrestricted, to keep the legacy strip-all behavior) and thread it into every parse and streaming/final strip site. - routes/inference.py: _strip_tool_xml_for_display accepts the gate and keeps an inactive rehearsal via a capture group on its rehearsal arm, so the display cleanup does not re-strip the already-correct loop output. The [TOOL_CALLS] control-token arms still strip unconditionally. Wire the current turn's active tool names into the GGUF and safetensors content-display sites. Tests: parse and strip gate coverage in test_tool_call_parser_strict.py, test_tool_xml_strip.py and test_safetensors_tool_loop.py; end-to-end GGUF coverage for the real-call-after-inactive-rehearsal case and a strengthened assertion that the inactive rehearsal prose survives intact. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: render the reasoning block for safetensors and MLX like GGUF enable_thinking chat templates (Qwen3/Qwen3.5/GLM) prefill an unclosed <think> into the generation prompt, so the model emits only the closing </think> then the answer. The safetensors/MLX chat stream emitted that as plain content, so the reasoning showed inline with no collapsible thinking block, while GGUF (which surfaces reasoning via reasoning_content) rendered one. This brings safetensors and MLX to parity. - _ResponsesReasoningExtractor gains a reasoning_prefilled mode that starts inside the reasoning block and splits on the first </think>; default False keeps GGUF and every existing caller byte-identical. It suppresses a stray re-emitted <think> and holds partial markers back across chunk boundaries. - _sf_reasoning_prefill_mode gates the mode on reasoning being enabled for the request, an enable_thinking or enable_thinking_effort style, and the template actually using the standard <think>/</think> markers. Models with a bespoke reasoning channel (e.g. gemma's <|think|>/<|channel>) are excluded so their answer is never swallowed; gpt-oss (Harmony) and thinking-off requests are excluded too. - sf_tool_stream and stream_chunks (the latter also serves MLX) feed text through the extractor, emitting reasoning_content then content deltas, with a per-turn reset in the tool loop and a flush before each tool_start; only the visible delta reaches the monitor reply. The two non-streaming drains split reasoning_content the same way. - Tests: extractor prefilled mode (streaming and edge cases), the gate matrix including the gemma-style exclusion, and a route-replay of the tool-loop reasoning stream. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: skip tool calls rehearsed in prefilled reasoning Reasoning models (Qwen3.5 enable_thinking) open <think> in the prompt, so the generated text starts inside the thought and emits only a closing </think> with no opener. _think_spans_outside_tool_markup only found spans with an explicit opener, so a NAME[ARGS]{...} or [TOOL_CALLS] call rehearsed in that leading thought was parsed and executed as a real call. Add a leading think span (offset 0 through the first close marker) when the content opens with a bare close, so the rehearsed call is skipped and the reasoning is preserved by strip_outside_think. Guarded by the existing call-span check: a literal </think> inside a real call's arguments does not trigger the span, so a genuine leading call still fires. Tests for both cases. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: do not start prefilled reasoning mode when reasoning_effort is none enable_thinking_effort models (e.g. GLM-5.2) express thinking-off via reasoning_effort="none" rather than enable_thinking=False, but _sf_reasoning_prefill_mode only looked at enable_thinking, so such a request started the extractor in prefilled mode. With thinking off the model never emits </think>, so the whole answer was captured as reasoning_content and the visible content/stream came back empty. Thread reasoning_effort through and return False when it is "none". Tests for none vs a real effort level. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: only treat a leading bare </think> as prefilled reasoning when a real call follows The prefilled-reasoning virtual span fired on any unmatched leading close marker, so a non-prefilled turn that emits a real call before a stray </think> (for example "Now web_search[ARGS]{...}</think> answer") had the call swallowed by the span and dropped. Require that a real tool call also appear after the close (the actual turn that follows the thought) before adding the span, so a stray close in a normal answer no longer suppresses a genuine leading call. The rehearse-then- call case still skips the rehearsal. Test for the stray-close case. * Studio: trim redundant comments (comment-only, AST-verified) * studio: keep tool_healing importable on Python 3.9 _balanced_json_span was annotated -> int | None. With no from __future__ import annotations, that PEP 604 union is evaluated at import time, so on Python 3.9 (which the package still supports, requires-python >=3.9, and where external inference servers import this module standalone) the def raises TypeError and the whole module fails to import before any parsing runs. Add from __future__ import annotations so annotations stay lazy strings, matching the prevailing convention across studio/backend. No behavior change: the module has no runtime annotation introspection. * Studio: gate the Anthropic tool-stream display strip on declared tools The Anthropic streaming and non-streaming tool paths called _strip_tool_xml_for_display without enabled_tool_names, so with the default strip-all behavior a final answer that literally contains an inactive-name NAME[ARGS]{json} (prose, not a call) lost those bytes in the delivered text. The GGUF and safetensors paths already pass _display_tool_name_gate(tools); these two sites were missed when that gate was threaded through. Compute the gate from the declared tools and pass it at both sites (threading openai_tools into _anthropic_tool_non_streaming and its caller), so an inactive-name rehearsal survives while an active-name one is still stripped. Add a regression test. * Studio: hold a split unrestricted rehearsal prefix at the bracket In unrestricted tool mode (tools=[]) the rehearsal-prefix regex required [A after the bracket, so a chunk boundary landing right after NAME[ (e.g. web_search[ then ARGS]{...}) failed the prefix check and streamed the partial tool markup web_search[ to the client before the call drained. Restricted mode already holds this via a startswith check. Make the bracket and each ARGS letter individually optional so NAME[ is held too, matching the documented intent. Add a regression test. * Studio: gate rehearsal detection and history strip on the original tool set Two display/loop gate fixes so a spent one-shot tool is handled consistently: - Rehearsal DETECTION (safetensors and GGUF loops) now uses the ORIGINAL tool list, matching the strip gate, instead of the post-removal active_tools. After a one-shot tool (render_html) runs it is dropped from active_tools; a repeat render_html[ARGS]{...} while another tool is still active was stripped from display yet never detected, so it was not routed to the render_html_repeat no-op and the turn ended as a blank continuation. Detection now fires for it. - The GGUF assistant-history sanitiser forwards the enabled-tool-name gate (like the live-response strip), so a prior turn documenting an inactive foo[ARGS]{...} shape is preserved in the replayed prompt context instead of being deleted. Add regression tests for both loops and the history strip. * Studio: thread the tool-name gate through the remaining rehearsal/history sites Follow-up to the rehearsal-detection and history-strip gate fixes, covering the sibling sites that were missed: - GGUF loop: the rehearsal-prefix and trailing-name hold checks now use the original tool list (_detect_tools) like the detection path, so a spent one-shot's split repeat (bare render_html then [ARGS]{...}) is held instead of flushed as visible text. - The safetensors and Anthropic assistant-history sanitisers and the Anthropic non-streaming passthrough now forward the enabled-tool-name gate to _strip_tool_xml_for_display, matching the GGUF history sanitiser and the live strips, so a prior turn documenting an inactive foo[ARGS]{...} example is preserved in the replayed prompt / final text instead of deleted. Add regression tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tile bracket-call spans per array item and include the v11 closer Two with_spans fixes for the Mistral bracket parser, both hit through the client-tool passthrough healers: - A multi-call [TOOL_CALLS] array carried its whole markup span on the first call and zero-width spans after, so a consumer that filters promotions by the declared tool set either re-emitted the full raw array as text next to the promoted call or silently dropped a filtered call's bytes. The region is now tiled across the call-producing items (each call's span covers its own JSON object plus the separator bytes before it; the last span runs to the region end), so promoted markup strips exactly once and a skipped call's bytes stay visible. - The v11 wrapper closer [/TOOL_CALLS] sat outside the reported span and leaked as stray text after promotion; the region now extends over an immediately-following closer. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: decouple healer signals from the loop signal set The passthrough healer buffered on every TOOL_XML_SIGNALS entry, so the bare [ARGS] rehearsal marker this branch adds for the loops (where it is gated on active tool names) put legitimate prose like 'Use foo[ARGS] in templates' into the holding state and stalled the stream until finalization. The healer can never promote a bare rehearsal call, so it now buffers only on formats its parser promotes: <tool_call>, <|tool_call>, <function=, [TOOL_CALLS]. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Condense comments in the Mistral tool-call rescue to contract essentials * verify_import_hoist: exempt __future__ imports and same-diff relocations Two false positives fired on this PR's refactor. A from __future__ import is a compiler directive whose name never appears as a runtime load, so HOISTED-IMPORT-UNUSED can never see it used, yet the file requires it for PEP 604 annotations on Python 3.9. TARGET-CHANGED flagged the deliberate move of the strip-pattern constants into core.inference.tool_call_parser as a silent re-point even though the old module-level target was removed and the new one added in the same diff. Both get narrow exemptions; a re-point to a pre-existing target is still caught, and the self-test negative controls all pass unchanged. * Drain the whole Mistral [TOOL_CALLS] array in streaming passthrough healing StreamToolCallHealer._drain promoted only the first parsed call per pass and dropped the rest of the buffer past that one span. For a well-formed Mistral parallel-tool-call array streamed through client-tool passthrough ([TOOL_CALLS][{...},{...}]), the per-item spans are contiguous, so after the first call was promoted the residue began with ,{...}] (no leading signal) and was flushed as raw text: every call after the first was lost. _drain now walks the contiguous run of parsed calls (adjacent tiled spans = one array), promoting each declared call and relaying undeclared ones as data, and stops at the first gap (prose) or incomplete trailing block so separate blocks still stream incrementally in document order. This mirrors the non-streaming heal_openai_message / finalize promote-or-flush loop and the server-side safetensors loop, which already handled multi-call arrays. Added regression tests: 2-call array in one feed and char-by-char, an undeclared middle call kept as text, and an array followed by trailing prose. * Drain comma-less Mistral tool-call arrays and normalize null arguments The array branch fed the whole body to a single json.loads, which rejects the comma-less multi-call form the repo's own Mistral/Ollama templates render (the range loop in ollama_template_mappers.py emits the objects with no separator) and so dropped every call. Decode elements individually with the existing comma-tolerant raw_decode helper, now _decode_array_items, which also returns the objects, so all calls are recovered while the span tiling is unchanged. Also normalize a non-object array argument such as arguments null to an empty object, matching the wrapped tool_call path, instead of serializing None to the string "null" that auto-heal would turn into a bogus query of "null". * Gate safetensors reasoning prefill on the rendered generation prompt reasoning_always_on fires on any paired <think></think> in the template, including markup that only renders PAST assistant history (Kimi-K2-Thinking) while the generation prompt opens no <think>. Starting the reasoning extractor in prefilled mode there captured a normal answer entirely as reasoning_content and returned blank visible content. Prefill only when rendering the generation prompt actually leaves <think> open (DeepSeek-R1 / QwQ / Qwen3-Thinking); history-only templates start the extractor in normal mode and parse the model's own <think>...</think>. Adds a Kimi-shape regression test. * Keep bare scalar Mistral array arguments raw instead of double-encoding A scalar string argument in the canonical Mistral [TOOL_CALLS] array (for example [TOOL_CALLS][{"name":"web_search","arguments":"weather"}]) was run through json.dumps, turning weather into the JSON string "weather". The downstream argument healer then wrapped that quoted form, so a single-string tool like web_search searched for the literal "weather" with quotes. The <tool_call> path already keeps a scalar argument raw; mirror it here so only a dict is serialized. Add a regression test asserting both paths yield the same healed arguments. * Tighten tool-call rescue and reasoning-prefill comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 18:52:13 -07:00
Daniel Han	233949cc9c	scan_packages: baseline transitive-dep drift in the supply-chain scan (#6917) ... The pip scan-packages gate (SCAN_ENFORCE=1) blocks on non-baselined CRITICAL/HIGH findings. Recent upstream releases of transitive dependencies added new files/loops that trip the pattern scanner, so all three shards (extras, hf-stack, studio) red-failed on legitimate library code. Add the 7 reviewed findings to scripts/scan_packages_baseline.json. Each entry is genuine upstream code from the official PyPI archive: - huggingface-hub huggingface_hub/_sandbox.py (staged dropper + C2 loop): the HF Jobs sandbox bootstrap string and its host-pool reservation loop. New in huggingface_hub 1.x (pulled via huggingface_hub>=0.34.0). - huggingface-hub huggingface_hub/hf_api.py, utils/_http.py (C2 loop): standard polling / retry while True loops. - fastapi fastapi/routing.py (C2 loop): websocket receive loop. - fastmcp-slim fastmcp/cli/apps_dev.py (fs enum + network): the FastMCP dev CLI (PrefectHQ) making httpx/socket calls. - cffi cffi/_cffi_gen_src.py (compile + exec): cffi generating and running C extension source, its core purpose. Additive only: no existing baseline entry is changed or removed. Verified by re-running the scanner over the full closure on Python 3.12.13 (the CI interpreter); it now exits 0 with only MEDIUM findings remaining.	2026-07-06 18:34:18 -07:00
Daniel Han	c00c1e70c8	studio: tool calling for DeepSeek (R1/V3/V3.1), GLM 4.x, Kimi K2 on safetensors + MLX (#5624) ... * studio: tool calling for Llama-3, Mistral, Gemma 4 on safetensors + MLX (#5615) Adds tool calling for Llama-3, Mistral (pre-v11 + v11+ + [ARGS]), and Gemma 4 to the safetensors / transformers and MLX backends. Parser patched against llama.cpp / vLLM / SGLang per-family parsers and normalises to OpenAI shape. 96 targeted unit tests + cross-OS staging CI (ubuntu / macos-14 / windows) green on the multi-format probe. * studio: tool-call healing parity between safetensors / MLX and GGUF After the multi-format parser landed in #5615, the safetensors / MLX agentic loop and the GGUF loop still differed on healing behaviour. This commit closes the gaps in both directions so the two backends react the same way to identical model output. Changes: 1. core/inference/llama_cpp.py -- the GGUF BUFFERING state machine now wakes on every emission marker the shared parser knows. Was ("<tool_call>", "<function="); is now the five-tuple imported from core.inference.tool_call_parser (Qwen / Qwen3.5 / Llama-3 <|python_tag|> / Mistral [TOOL_CALLS] / Gemma 4 <|tool_call>). Stream cleanup is delegated to the same shared strip_tool_markup so leaked markup from any family is removed from assistant content. 2. core/inference/llama_cpp.py -- per-tool canonical heal key. When a tool arguments field is a bare string and JSON parsing fails, the GGUF path now heals to {"code": raw_args} for python, {"command": raw_args} for terminal, and {"query": raw_args} for everything else. Was hard-coded to {"query": raw_args}, which silently routed every python / terminal emission through web_search. Mirrors safetensors_agentic._CANONICAL_HEAL_ARG. 3. core/inference/safetensors_agentic.py -- re-prompt on plan- without-action. When the model emits a short forward-looking intent ("I'll search for that", "Let me check", "First, I will...") and no tool call, the loop nudges the model to act instead of silently returning a plan-only answer. Up to _MAX_REPROMPTS=3 (matches GGUF). The intent regex, character cap, and instruction text are byte-identical to the GGUF path. The buffer-end fall-through is unified so a buffered intent emission that never exits the BUFFERING state still triggers the re-prompt. 4. core/inference/safetensors_agentic.py -- extra iteration slots for re-prompts. The loop now budgets max_tool_iterations + _MAX_REPROMPTS + 1 total iterations and tracks the tool-call count separately, so a stalling model can be nudged 3x without eating the caller's tool-call budget. Mirrors the _extra slot reservation in the GGUF path. Tests (14 new safetensors-side units; 5 GGUF parity pins): TestLoopRePrompt -- intent-trigger, plain-answer, no-tools, cap-at-three, budget preserved, buffer-end intent. TestLoopCanonicalHealKey -- python / terminal / unknown. TestGGUFSafetensorsHealingParity -- shared markers used, shared strip used, canonical heal keys identical, intent regex matches same phrases, _MAX_REPROMPTS equal on both backends. All 110 targeted tests pass locally; the broader tool / inference / model-config / sandbox / anthropic / mlx suites stay green. Why this matters Without this parity, Llama-3.2 / Mistral / Gemma 4 emissions on Mac (MLX) and Linux-safetensors stop the agentic loop as soon as the model says "Let me...", because the GGUF re-prompt logic never existed on these backends. The two-marker GGUF BUFFERING tuple also let non-Qwen tool emissions stream out as plain prose when llama-server's structured channel did not pick them up. Both paths now drain the same way, heal the same way, and re-prompt the same way -- so a tool call that works on GGUF works identically on safetensors / MLX. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: fix tool-call parser bugs from gemini review on #5620 Three high-priority gemini findings on the tool-call parsing additions: 1. unicode_escape on UTF-8 bytes corrupts non-ASCII literals (e.g. ✨ becomes â\x9c¨). Replace with json.loads on a quoted string -- preserves emoji / CJK / RTL while still handling \n \t \uXXXX escapes. 2. Llama-3 sentinel stripping is order-dependent. A leading `<|eot_id|><|begin_of_text|>` left `<|begin_of_text|>` behind because the loop had already passed that sentinel. Loop until no sentinel matches at the start. 3. Mistral v11+ `[TOOL_CALLS] name { json }` regex uses non-greedy `\{.*?\}` which truncates at the first `}` of a nested JSON argument, leaking the tail (e.g. `}}`) into user-visible streamed text. Same problem for the v0.3 array pattern with nested brackets. Strip those with balanced brace/bracket scanning via a new `_strip_mistral_closed_calls` helper called from `strip_tool_markup`. Also fix the inference routes' parallel `_TOOL_XML_RE`: - Same nested-JSON truncation in the Mistral patterns; route the strip through the parser's balanced-scan helper via a thin `_strip_tool_xml` wrapper that all existing callers now use. - Llama-3 `<|python_tag|>[^\n<]*` stopped at any `<`, leaking the tail of any tool call whose argument contained a literal `<` (queries, code snippets). Relax to `[^\n]*` which keeps the strip confined to the actual end-of-line. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: tool calling for DeepSeek (R1/V3/V3.1), GLM 4.x, Kimi K2 Adds three more emission-family parsers to tool_call_parser.py so the shared safetensors / MLX / GGUF agentic loop covers the major open- weight reasoning families. Patterns ported from llama.cpp (common/chat-parser.cpp legacy pre-PEG branch), vLLM (tool_parsers/deepseekv3*, glm4_moe, kimi_k2), and SGLang (function_call/deepseekv31_detector, glm4_moe_detector, kimik2_detector). All three references are MIT (llama.cpp) or Apache-2.0 (vLLM, SGLang). Formats covered: DeepSeek R1 <｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function <｜tool▁sep｜>NAME\n```json\n{...}\n```<｜tool▁call▁end｜> <｜tool▁calls▁end｜> -- args wrapped in a Markdown json fence, ``function`` literal prefix per llama.cpp common_chat_parse_ deepseek_r1 (chat-parser.cpp:801-820) DeepSeek V3/V3.1 <｜tool▁calls▁begin｜><｜tool▁call▁begin｜>NAME <｜tool▁sep｜>{json}<｜tool▁call▁end｜><｜tool▁calls▁end｜> -- bare JSON, no code fence, no ``function`` prefix per llama.cpp common_chat_parse_deepseek_v3_1 (chat-parser.cpp:822-879) GLM 4.5/4.6/4.7 <tool_call>NAME\n<arg_key>k1</arg_key> \n<arg_value>v1</arg_value>...</tool_call> -- strings raw, non-strings JSON-encoded per chat_template.jinja; multi-call is back-to-back blocks. Per llama.cpp common_chat_parse_glm_4_5 (chat-parser.cpp:1040-1052) Kimi K2 <|tool_calls_section_begin|><|tool_call_begin|> functions.NAME:IDX<|tool_call_argument_begin|>{json} <|tool_call_end|><|tool_calls_section_end|> -- bare name recovered by stripping ``functions.`` prefix and ``:IDX`` suffix; full id preserved as tool_calls[i].id so the roundtrip replays verbatim. Per llama.cpp common_chat_parse_kimi_k2 (chat-parser.cpp:896-913) Marker collisions GLM uses the same ``<tool_call>`` opener as Qwen but with a bare function name + ``<arg_key>`` body (Qwen has ``\s*{`` after the tag). The dispatch keeps Qwen first; Qwen's _TC_JSON_START_RE returns no matches on a GLM emission, so the fall-through to _parse_glm_tool_ calls handles it correctly. Existing Qwen tests confirm zero regression. Streaming buffer TOOL_XML_SIGNALS extended from 5 markers to 12 so the BUFFERING state machine wakes on every new family's section opener. Added the DeepSeek alternative markers (ASCII underscores, short ``<｜tool▁calls｜>`` form) because real checkpoints emit those variants. Strip patterns _TOOL_CLOSED_PATS adds DeepSeek envelope (``<｜tool▁calls▁begin｜>... <｜tool▁calls▁end｜>``) and Kimi section (``<|tool_calls_section_begin|> ...<|tool_calls_section_end|>``). _TOOL_ALL_PATS adds the same plus the unclosed-tail variants so a truncated stream does not leak markup. Route gate _detect_safetensors_features._PARSER_MARKERS grows to include DeepSeek and Kimi markers plus ``<arg_key>`` (the unique GLM signal). _TOOL_XML_RE (the route-layer markup-strip regex) gets DeepSeek and Kimi closed-pair patterns. _TOOL_TEMPLATE_MARKERS in llama_cpp.py adds ``message['role'] == 'tool'``, ``message['tool_calls']``, and ``tool_calls is defined`` so the classifier recognises DeepSeek's subscripted-access template style (it has no top-level ``{% if tools %}`` block). Tests (39 new): TestParserDeepSeek (7) -- R1 fence, short-form opener, V3.1 bare, multi-call, with-reasoning, strip, signal-wakes-streaming TestParserGLM (6) -- single, mixed types, multi-call, unclosed-heal, no-Qwen-regression, strip TestParserKimi (6) -- single, multi-call, dotted-name, unclosed, strip, signal-wakes-streaming TestParserCrossFormatRouting (2) -- dispatch routing, signal coverage TestLoopBasic loop integration (3) -- DeepSeek / GLM / Kimi end-to-end Capability advertise (3) -- DeepSeek / GLM / Kimi templates flip supports_tools=True All 398 targeted tests pass locally (115 safetensors + 27 capability + rest of tool / inference / sandbox / model-config suites). Builds on PR #5620 (parser + healing parity for Llama-3 / Mistral / Gemma 4); will rebase cleanly onto main once #5620 lands. PR opened as draft - do not merge until validated against real models for each family. Sources - llama.cpp common/chat-parser.cpp lines 801-913, 1040-1052 (MIT) - vLLM vllm/tool_parsers/deepseekv31_tool_parser.py (Apache-2.0) - vLLM vllm/tool_parsers/glm4_moe_tool_parser.py (Apache-2.0) - vLLM vllm/tool_parsers/kimi_k2_tool_parser.py (Apache-2.0) - SGLang python/sglang/srt/function_call/{deepseekv31,glm4_moe,kimik2}_ detector.py (Apache-2.0) - Live chat templates: deepseek-ai/DeepSeek-V3.1, zai-org/GLM-4.6, moonshotai/Kimi-K2-Instruct, unsloth/DeepSeek-V3-0324, unsloth/GLM-4.5-Air, unsloth/Kimi-K2-Instruct * studio/routes: make python_tag strip multi-line aware Earlier revisions of _TOOL_XML_RE in studio.backend.routes.inference oscillated between two bug shapes: 5615 r"<\|python_tag\|>[^\n<]*" -- stopped at any literal "<" so code='if x < 10: pass' leaked '< 10: pass)' to the user. 5620.1 r"<\|python_tag\|>[^\n]*" -- single-line only; the second line of python.call(code="a\nb") leaked. The full parser (_parse_llama3_python_tag) already handles both via balanced-brace scanning, so the parsing path was fine; the LEAK was in the streaming strip path that runs on every cumulative emission while content is still arriving. Switch to r"<\|python_tag\|>(?:[^<]|<(?!\|))*" so the strip consumes: * any character that is not a "<" (newlines, JSON, code, ...), * a "<" only when it is NOT followed by "|" (i.e. NOT a Llama-3 sentinel start like <|eot_id|>, <|eom_id|>, <|begin_of_text|>). This means: * code='if x < 10' stays inside the strip (5615 fix preserved), * multi-line code stays inside the strip (5620 round 2), * the strip terminates at the next Llama-3 sentinel so trailing assistant content survives. Tests: TestRoutesPythonTagStrip (8 cases) pytest test_safetensors_tool_loop.py test_safetensors_capability_advertise.py -> 118 passed in 1.81s (was 110). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: review follow-ups for DeepSeek / GLM / Kimi tool calling Four fixes addressing review of the parent commit: 1. GLM <arg_value> coercion: tighten the json.loads -> ast.literal_eval -> raw cascade to only deserialize when the body unambiguously looks like a JSON literal (object, array, JSON-encoded string, true/false/null, or numeric). Strings like ``True`` / ``None`` (Python literals, not JSON) and arbitrary prose now stay raw. The bare-numeric / bare-boolean ambiguity with string args remains an inherent limitation of the template without schema access -- documented in the new comment. Drops the ast import entirely (closes Gemini's :1036 suggestion). 2. Kimi K2 bare-counter ids (e.g. ``<|tool_call_begin|>3``) are now dropped rather than surfaced as a tool literally named "3". Matches vLLM behaviour; SGLang's schema-infer fallback is out of scope at the parse site. Real Kimi K2 emissions use ``functions.NAME:IDX`` so this is the exception path. 3. Restore the elaborate ``<|python_tag|>(?:[^<]|<(?!\|))*`` clause in routes.inference._TOOL_XML_RE -- the simpler ``[^\n<]*`` form regressed PR #5620's multi-line / literal-``<`` python_tag fix. Restore ``TestRoutesPythonTagStrip`` (8 tests) adapted to call ``_TOOL_XML_RE.sub`` directly since the ``_strip_tool_xml`` helper was inlined this PR. 4. Add the spaced and backslash-escaped DeepSeek opener variants (``<｜tool calls begin｜>``, ``<｜tool\_calls\_begin｜>``) to ``TOOL_XML_SIGNALS`` for streaming-gate parity with ``_DEEPSEEK_BEGIN_RE``. Also updates the llama.cpp / vLLM citations in the parser docstrings: ``common/chat-parser.cpp`` was split into ``common/chat.cpp`` + ``common/chat-peg-parser.cpp`` by llama.cpp PR #18675, and vLLM moved the tool parsers from ``vllm/entrypoints/openai/tool_parsers/`` to ``vllm/tool_parsers/``. Pin to pre-refactor commit ``51fa458a92d6`` where the cited line numbers still resolve. New regression tests in ``test_pr5624_regressions.py`` cover the GLM coercion heuristic shapes, GLM literal-``<`` in arg_value, Kimi K2 dotted name, Kimi K2 bare-counter drop, DeepSeek V3.1 truncated mid-stream, and routes-layer strip across all three new families. Tests: pytest studio/backend/tests/test_safetensors_tool_loop.py studio/backend/tests/test_safetensors_capability_advertise.py studio/backend/tests/test_pr5624_regressions.py -q -> 170 passed in 1.91s * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: tighten verbose comments in tool-call parser sections Comments were narrating what the code already says. Cut historical "earlier revisions used X, then Y" narratives down to one-line WHY notes where the footgun still matters (canonical heal-key parity, balanced-brace vs non-greedy regex, ``(?:[^<]|<(?!\|))*`` over ``[^\n<]*``/``[^\n]*``). Drop section-header banners. No behaviour change. Re-ran: pytest studio/backend/tests/test_safetensors_tool_loop.py \ studio/backend/tests/test_safetensors_capability_advertise.py -q -> 118 passed. Regression replay (parser + _coerce_arguments on the 5 #5615 inputs) -> 21/21. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: GLM 4.7 no-newline emission + Kimi multi-section parity Two fixes surfaced by triple-confirm verification against the live HF chat templates and upstream llama.cpp / vLLM / SGLang parsers. 1. GLM 4.7 silent drop ``zai-org/GLM-4.7/chat_template.jinja`` line 65 uses ``{{- '<tool_call>' + tc.name -}}`` which Jinja strips trailing whitespace from, so the first ``<arg_key>`` follows the function name with NO ``\n`` between them. Real emissions look like ``<tool_call>get_weather<arg_key>city</arg_key><arg_value>London </arg_value></tool_call>``. The previous ``_GLM_TC_OPEN_RE`` ended the name with ``\n`` so GLM-4.7 calls were silently dropped (parser returned ``[]``). Fix: relax the name terminator to a lookahead that accepts EITHER ``\n`` OR the next ``<arg_key>``: _GLM_TC_OPEN_RE = re.compile( r"<tool_call>\s*([^\n<{][^\n<]*?)\s*(?=\n|<arg_key>)" ) The first-char restriction ``[^\n<{]`` still excludes Qwen's ``<tool_call>{json}`` form so the Qwen-vs-GLM dispatch remains mutually exclusive. 2. Kimi multi-section parity with vLLM / SGLang ``vllm/tool_parsers/kimi_k2_tool_parser.py`` and SGLang's ``kimik2_detector.py`` both use ``re.findall`` and so collect every ``<|tool_calls_section_begin|>...<|tool_calls_section_end|>`` block in a single stream. The previous implementation stopped at the first ``<|tool_calls_section_end|>``. Kimi K2 doesn't emit multi-section in practice, but parity is cheap. Fix: wrap the existing per-call body parser in an outer loop that advances past each ``<|tool_calls_section_end|>`` and continues to the next ``<|tool_calls_section_begin|>``. Body parsing extracted to ``_parse_kimi_section_body`` for clarity. Truncated final section is still surfaced via the existing in-body balanced-brace walk. Verified independently against the live HF templates: * GLM-4.7 emission constructed from the live template parses to the expected ``{name, arguments}`` shape. * GLM-4.5 / 4.6 newline shape continues to parse (the lookahead also matches ``\n``). * Qwen ``<tool_call>{json}`` still dispatches to the Qwen path -- the first-char restriction stops the GLM regex from biting JSON bodies. * Kimi two-section stream surfaces both calls in order with full ids preserved. * Bare-counter Kimi ids still drop. Tests added in ``test_pr5624_regressions.py``: * ``test_glm_4_7_no_newlines_between_name_and_arg_key`` * ``test_glm_4_7_no_newlines_multi_call`` * ``test_glm_4_7_does_not_break_qwen_path`` * ``test_kimi_two_sections_in_one_stream_both_parse`` pytest studio/backend/tests/test_safetensors_tool_loop.py studio/backend/tests/test_safetensors_capability_advertise.py studio/backend/tests/test_pr5624_regressions.py -q -> 174 passed in 1.93s pytest studio/backend/tests/ -q -k 'not gpu and not llama_cpp_integration' -> 2038 passed, 15 failed (pre-existing CI gaps). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: parser robustness fixes for PR #5620 Three surgical extensions to the multi-format tool-call parser, each covering a real fine-tune / template emission shape that the current parser silently drops. No path narrows; all changes widen what is accepted. 1. `_parse_tool_call_json` now accepts both `arguments` and `parameters` keys. A Hermes / Qwen `<tool_call>{json}</tool_call>` wrapper around a Llama-3.2 fine-tune that emits the `parameters` key was extracting the tool name and silently discarding the args, producing a working-shaped call with an empty payload. The bare-JSON and python_tag paths already accepted both keys; this path now matches them. 2. `_TC_FUNC_START_RE`, `_TC_PARAM_START_RE`, and `_TC_PARAM_CLOSE_RE` now also match the attribute form `<function name="..."><param name="...">v</param></function>` used by MiniCPM-5 and MiniMax-M2. Names land in either capture group, and `</param>` is accepted as a short close. 3. `_parse_llama3_bare_json` sentinel-strip now consumes the role label inserted between `<|start_header_id|>` and `<|end_header_id|>` by Meta's official Llama-3.x chat template. Without this, every assistant turn re-fed through the template prefix `<|start_header_id|>assistant<|end_header_id|>\n\n{json}` parsed to zero calls, so any history-with-tool-call round-trip in production silently dropped. Tests in `studio/backend/tests/test_safetensors_tool_loop.py`: * `TestParserRobustness::test_tool_call_json_accepts_parameters_key` * `TestParserRobustness::test_function_xml_attribute_form` * `TestParserRobustness::test_function_xml_attribute_form_multi_param` * `TestParserRobustness::test_function_xml_legacy_equals_form_still_works` (regression guard for the existing `<function=name>` syntax) * `TestParserRobustness::test_llama3_chat_template_round_trip` * `TestParserRobustness::test_llama3_round_trip_all_roles` * `TestParserRobustness::test_llama3_round_trip_with_eot_prefix` `pytest studio/backend/tests/test_safetensors_tool_loop.py studio/backend/tests/test_safetensors_capability_advertise.py -q` goes from 118 to 125 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim verbose comments in tool-call parser sections for PR #5624 Pure comment / docstring tightening on top of the GLM 4.7 + Kimi multi-section fixes. No behavioural change. * Drop multi-paragraph prelude and post-refactor citation chatter in the DeepSeek, GLM and Kimi parser docstrings; keep the shape and upstream-commit pin. * Collapse ``parse_tool_calls_from_text``'s 9 per-family blocks into a single ordered loop with one combined comment. * Tighten the GLM coercion, Kimi bare-counter and ``_TOOL_XML_RE`` comments to one or two lines each. * Same trim pass on ``_PARSER_MARKERS`` and the regression-test docstrings. Tests: pytest studio/backend/tests/test_safetensors_tool_loop.py studio/backend/tests/test_safetensors_capability_advertise.py studio/backend/tests/test_pr5624_regressions.py -q -> 174 passed in 2.00s * Fix O(N^2) DeepSeek V3.1 backtracking for PR #5624 Adversarial input ``<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>fn<｜tool▁sep｜>`` followed by a long body that does NOT contain a closing brace caused the V3 path's ``([^\n<]+?)<｜tool▁sep｜>`` regex to backtrack quadratically: at each position the lazy quantifier extends one char at a time looking for a sep that isn't there, taking ~19s on 50k chars. Replace the regex search with ``str.find`` on the sep marker plus a left-walk to recover the name. ``str.find`` is O(N); the walk stops on ``\n`` (turn boundary), ``<`` (start of a tag), or ``>`` (end of an optional ``<｜tool▁call▁begin｜>`` prefix). Same observable behaviour as the regex on every canonical input. Tests: test_deepseek_v3_1_huge_truncated_body_is_linear (new) -- 50k chars must parse in &lt; 1s. pytest studio/backend/tests/test_safetensors_tool_loop.py studio/backend/tests/test_safetensors_capability_advertise.py studio/backend/tests/test_pr5624_regressions.py -q -> 175 passed in 1.97s pytest studio/backend/tests/ -q -k 'not gpu and not llama_cpp_integration' -> 2038 passed, 15 pre-existing failures unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: terminate function-XML body at </function>, not just </tool_call> `_parse_function_xml` was looking for `</tool_call>` (the Hermes wrapper) as the body terminator. When a model emits a standalone `<function=NAME><parameter=K>v</parameter></function>` followed by explanatory prose (which models routinely do), no `</tool_call>` is present, so the body extended to end-of-string and the trailing prose leaked into the LAST parameter value. Pre-existing on main (the legacy `<function=NAME>` form had this bug too). Same affects PR #5620's new attribute-form `<function name="NAME"><param name="K">v</param></function>` emission used by MiniCPM-5 / MiniMax-M2. Fix: `_TC_END_TAG_RE` now matches either `</tool_call>` OR `</function>`. The existing `_TC_FUNC_CLOSE_RE` / `_TC_PARAM_CLOSE_RE` strips are unchanged. Multi-call inputs still bound each function at the next `<function=` start, so no over-eager consumption. New tests: * `test_function_xml_followed_by_prose` (legacy form + prose) * `test_function_attribute_xml_followed_by_prose` (attribute form + prose) Existing `test_code_with_embedded_xml` still passes (a parameter value containing literal `<a></a>` is preserved because the embedded close tag is `</a>`, not `</function>`). `pytest studio/backend/tests/test_safetensors_tool_loop.py studio/backend/tests/test_safetensors_capability_advertise.py -q` goes from 125 to 127 passed. * Studio: tighten Llama-3.2 bare-JSON guard A fuzz pass on PR #5811 turned up that ``_parse_llama3_bare_json`` accepted ``parameters`` as a string, contradicting the docstring's "parameters or arguments is a dict" guard. Prose JSON like ``{"name":"foo","parameters":"a sentence"}`` would wrongly fire the parser, which the agentic loop would then heal into a real ``foo(query="a sentence")`` call. Same code lives on this branch, so the same fix applies here. Tightened guard: - ``parameters`` must be a dict (Llama-3 spec). - ``arguments`` may be a dict, or a JSON-encoded string that decodes to a dict (OpenAI shape, e.g. ``"arguments":"{\"q\":\"x\"}"``). Plain non-JSON strings or JSON-strings of lists / scalars / null no longer pass. Mirrors the fix landed in PR #5811 commit 615b8608. Adds the same 4 regression tests under TestParserMultiFormat. Existing test suite stays green: 127 -> 131 passing. * Studio: skip non-scalar args in python_tag JSON form The JSON sub-path of ``_parse_llama3_python_tag`` was fabricating ``{"value": args}`` when the model emitted a non-dict / non-string ``arguments`` value (e.g. ``42``, ``[1,2,3]``, ``null``, ``true``). This silently turned a malformed emission into a real tool call, which the agentic loop would then execute with arguments the model never intended. Tightened: skip the call instead of fabricating. The same behaviour now matches the bare-JSON guard tightened earlier (strict-guard merge from PR #5620, inherited via merge here). Added a regression test covering the four non-scalar shapes. Pass count on this branch: 158 -> 159. Sites in ``_parse_tool_call_json`` and ``_consume_mistral_call`` keep the existing looser behaviour for now; both are reached only after explicit ``<tool_call>`` / ``[TOOL_CALLS]`` markers so the false-positive surface there is much narrower. * studio: fix safetensors tool-call parser gaps vs llama.cpp (Mistral CALL_ID / THINK, attribute-form signal) Three GGUF-parity fixes to the safetensors tool-call parser, each matching llama.cpp's reference behaviour: - Mistral Small 3.2 emits [TOOL_CALLS]name[CALL_ID]<id>[ARGS]{json}. The parser stopped after the name on seeing [CALL_ID] (neither [ARGS] nor {), dropping the call. Skip an optional [CALL_ID]<id> segment in both the parse and strip paths. llama.cpp parses this (test-chat.cpp:4785). - Magistral wraps reasoning in [THINK]...[/THINK]. A [TOOL_CALLS] inside the reasoning was parsed as a real call, producing a phantom call. Strip a leading [THINK] block before scanning so only the post-reasoning call counts (test-chat.cpp:2285); a literal [THINK] inside a later argument is left intact. - The standalone MiniCPM-5 / MiniMax-M2 <function name="..."> attribute form parsed correctly but was absent from TOOL_XML_SIGNALS and the markup strip patterns, so the streaming safety-net parse was gated off (dropping the call) and markup leaked into displayed text. Add the signal and broaden the strip regexes. Adds regression tests for all three. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: fix GLM and Kimi K2 safetensors tool-call parser gaps vs llama.cpp Four GGUF-parity fixes for the GLM and Kimi K2 families: - GLM 4.7 zero-argument inline call <tool_call>name</tool_call> was dropped: the open-tag lookahead only allowed \n or <arg_key> after the name. Allow </tool_call> too so a no-arg call parses to empty args (vLLM / SGLang / llama.cpp all parse it). - GLM string argument values were stripped, losing significant leading / trailing whitespace in code / diff arguments. Keep the raw value for the string fallback and only strip the copy used to probe for a JSON literal, matching vLLM glm4_moe which never strips string args. - Kimi K2 calls emitted without the <|tool_calls_section_begin|> wrapper were dropped. llama.cpp makes the section optional (Kimi can call a tool straight after reasoning without opening a section); parse a bare <|tool_call_begin|> when no section is present. - Kimi K2 malformed / truncated JSON in one call dropped every later call in the section. Skip the bad call and keep parsing so valid subsequent calls are recovered (vLLM parity). Adds regression tests for all four. * studio: fire safetensors tool calls for the bare-JSON (Llama-3.2) form The agentic loop's streaming safety-net parse was gated on has_tool_signal(), which is False for the Llama-3.1 / 3.2 bare-JSON tool form {"name":..,"parameters":..} (no XML marker). Real tool calls were therefore dropped: the loop logged "model planned without calling tools", re-prompted three times, then gave up with zero tool calls, while GGUF's llama-server parses the same emission natively. Run parse_tool_calls_from_text() unconditionally in the safety net. The parser is strict (only fires on a valid tool-call shape) so plain answers are unaffected. Reproduced on a real unsloth/Llama-3.1-8B-Instruct run: the model emits {"name":"web_search","parameters":{...}} which now executes the tool instead of being re-prompted into a no-op. Adds a loop regression test for the bare-JSON form. * studio: fire safetensors tool calls for Gemma 4 (native template + stripped parser) Gemma-4 safetensors fired no tools while its GGUF fired reliably. Three gaps: - The Studio swaps in the Unsloth "gemma-4" chat template, which does not render the tools schema (the model's native template does), so the model never saw the tools. Fall back to the model's native template when the override template renders identically with and without tools. Same fix helps any family whose override template drops tools. - skip_special_tokens strips the <|tool_call> wrapper and <|"|> string markers, so a streamed Gemma-4 call arrives as a bare call:NAME{k:v, ...} with unquoted values. Parse that form, keeping commas/braces inside a code or command value, normalising surrounding quotes, and stripping the leaked markup from the final answer. - Without a grammar a small model can loop, repeating one call for the whole tool budget. Collapse exact-duplicate calls within a turn and force a final answer after a turn that made no new tool progress (llama-server's lazy grammar prevents this loop on the GGUF side). Adds parser tests for the bare/stripped Gemma-4 form. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: complete strict-mode contract and fix parser import paths Address review findings on the multi-format tool-call parser: - Honor allow_incomplete=False in the remaining sub-parsers. The Llama-3 <|python_tag|>NAME.call(...) parser, the pre-v11 Mistral [TOOL_CALLS] array parser, and the Gemma 4 <|tool_call> parser ignored strict mode, so a truncated call (missing closing paren, ], or <tool_call|>) was still healed and executed with Auto-Heal disabled. Thread strictness through and reject the unclosed forms, matching the JSON and function-XML paths. - Drop the duplicate tool_call_parser import block in llama_cpp.py and the redundant un-aliased TOOL_XML_SIGNALS; only the _SHARED_TOOL_XML_SIGNALS alias is used as a value. - Import _strip_mistral_closed_calls from core.inference.tool_call_parser in routes/inference.py instead of studio.backend.core... The self-contained run.py launch mode only puts studio/backend on sys.path, so the absolute package path raised ModuleNotFoundError on the server-tool strip path. Add strict-mode regression tests for the truncated Llama-3 dot-call and the unclosed Mistral array. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden DeepSeek/Kimi tool-call parsing and strip Address review findings on the DeepSeek and Kimi parsers: - Honor allow_incomplete=False for DeepSeek. An envelope with no closing <｜tool▁calls▁end｜> is truncated mid-stream; reject it in strict mode instead of healing the body out to EOF, matching the strict XML and Mistral paths. - Do not skip a following tool call when the current call's end marker is missing. The DeepSeek V3 and Kimi loops advanced by searching forward for the next <｜tool▁call▁end｜> / <|tool_call_end|>, which could land on a later call's end marker and drop the call in between. Advance by the JSON end; the loop re-locates the next call marker from there. - Strip truncated DeepSeek and Kimi section blocks in the route-level display regex. The patterns required the closing marker; add the end-of-text alternative so a block truncated by EOS does not leak raw markup to the UI. Add regression tests for the truncated DeepSeek envelope, and for DeepSeek and Kimi multi-call recovery when the first call's end marker is missing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: preserve XML param indentation and alias Mistral array parameters Two parser-correctness fixes found by auditing against the model chat templates and the SGLang / vLLM reference parsers: - Qwen3.5 XML parameter values lost their leading indentation. The chat template emits <parameter=k>\nVALUE\n</parameter>, but the parameter-start regex ate the wrapping newline AND the value's first-line indentation with a trailing \s*, then str.strip() removed the rest. Narrow the trailing class to horizontal whitespace only and trim exactly one wrapping newline (via _trim_param_value), preserving indentation in code/diff arguments. Matches SGLang's qwen3_coder detector. Applies to both _parse_function_xml (tool_call_parser.py) and the XML path in tool_healing.py. - Mistral pre-v11 array objects keyed on parameters dropped their payload. _consume_mistral_call read only the arguments key; alias parameters the same way the JSON/XML paths and SGLang's base detector do. Add regression tests for preserved multi-line indentation and the array parameters alias. * Studio: DeepSeek strip sync, Gemma nested args, GLM/Kimi strict mode Parser-correctness fixes found by auditing DeepSeek/GLM/Kimi against vLLM, SGLang, and the model chat templates: - DeepSeek: the short <｜tool▁calls｜> opener (and the space / escaped-underscore spellings) was parsed but never stripped, so a short-opener envelope leaked raw markup to the UI. Share one opener alternation between _DEEPSEEK_BEGIN_RE and the strip patterns (and the route-level display regex) so a signal we parse can never be left un-stripped. - Gemma wrapper-less stream: a nested object/array argument (loc:{city:NYC}, labels:[bug,ui]) was kept as a literal string. Parse it recursively when the bare value is a balanced {} / [], falling back to the raw string for a truncated value. - GLM and Kimi ignored allow_incomplete. With Auto-Heal off, a GLM block with no </tool_call>, a Kimi section with no <|tool_calls_section_end|>, or a Kimi call with no <|tool_call_end|> are truncated and must be rejected, matching the strict behavior of the JSON/XML/Mistral/DeepSeek paths and vLLM/SGLang. Add regression tests for the short-opener strip, the Gemma nested args, and GLM / Kimi strict-mode rejection. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten tool-call parser comments Make the comments in the multi-format tool-call parser and its callers succinct: compress verbose docstrings/blocks to one or two lines, drop ones that restate the code, and trim the tiny balanced-scanner helpers. Correctness rationale and upstream provenance (SGLang/llama.cpp parity, the strict-mode / Auto-Heal contract, whitespace-preservation, and the Unicode / full-width-pipe notes) are kept in compact form. Comment-only: no code or behavior change (verified with comment_tools.py check --strip-docstrings; parser suite green). * Studio: tighten DeepSeek/GLM/Kimi parser comments Compress the comments added for the DeepSeek/GLM/Kimi parsers and the Gemma wrapper-less helpers to one or two lines, keeping the upstream provenance (llama.cpp 51fa458a92d6), the O(N^2) / strict-mode rationale, and the vLLM parity notes intact. Comment-only: no code or behavior change (verified with comment_tools.py check --strip-docstrings; parser suite green). * Studio: make DeepSeek R1 / GLM parsing linear and close routes strip gaps Review follow-up for the DeepSeek/GLM/Kimi parser: - DeepSeek R1 detection used a greedy ``([^\n]+)\n```json`` regex that backtracks O(N^2) on a fence-less truncated body; scan with str.find instead (mirrors the V3 path). - GLM arg pairs used a lazy-group finditer that rescanned to EOF from each bare <arg_key> in an unclosed body (O(N^2)); walk pairs with str.find. - The route display strip (_TOOL_XML_RE) accepted fewer DeepSeek openers than the parser (missed the space / escaped-underscore spellings) and missed bare section-less Kimi calls, so a call we parse could leak raw markup to the UI. Reuse the parser's shared _DEEPSEEK_OPEN_RE_SRC and add a bare-Kimi arm. Add ReDoS-linearity regressions for the R1 and GLM paths, a positive R1 fenced-json parse test, and routes-strip tests for the space/escaped DeepSeek openers and the bare Kimi call. * Studio: fix test_mcp_servers _TOOL_XML_RE reconstruction after _DS_OPEN_SRC reuse The routes strip fix made _TOOL_XML_RE reference the module-level _DS_OPEN_SRC variable. test_mcp_servers reconstructs the regex by exec-ing the extracted compile() source in a namespace that only defined _re, so it raised NameError. Inject _DS_OPEN_SRC into that namespace, matching the same fix already applied in test_tool_xml_strip. * Studio: make Llama-3 .call and Mistral-array healing parsing linear Two more O(n^2) ReDoS paths in the multi-format parser, both reachable from the agentic loop on a long truncated body with no length cap: - _LLAMA3_KV_RE.finditer over a .call(...) body retried at every offset of a long word run / unterminated quote (40K -> 14s). Replace with a hand-scan that reuses the same key/number/literal sub-regexes via anchored match and walks the string body by hand, so an unterminated quote is O(n). Verified byte-identical to the old regex over 200K fuzzed inputs. - _parse_mistral_array healing ran _balanced_brace_end from every { in the body (20K -> 17s). Walk top-level objects, advancing past each balanced {...}; this also drops the phantom call the old scan emitted from a nested argument object. Add adversarial-length linearity regressions plus positive .call kwargs and unclosed-array recovery coverage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: strengthen #5624 regression assertions and strip-test harness guards - test_strip_tool_markup_handles_deepseek_envelope used `A or B` where B was the preservation property the next line already asserts, masking the real check. Replace with an explicit assertion that the call name and args are stripped. - The test_tool_xml_strip source-extraction harness reconstructs _TOOL_XML_RE and _strip_tool_xml_for_display from routes/inference.py via lazy regexes that could silently grab a shorter slice. Assert the extracted regex carries the DeepSeek / bare-Kimi arms and the helper body reached the _TOOL_XML_RE.sub call. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: honor strict mode in safety-net, keep empty Gemma args, strip attribute-form function XML - safetensors safety-net parser now forwards allow_incomplete=auto_heal_tool_calls, matching the draining path, so a late incomplete tool call is not healed and executed when Auto-Heal is off. - Gemma empty bare value ({k:}) now serialises as "" instead of invalid {"k":}, which previously dropped the whole call. - Route _TOOL_XML_RE also strips the <function name="..."> attribute form (MiniCPM-5 / MiniMax-M2) so it no longer leaks to the UI. * Studio: linearize wrapper-less Gemma nested-arg parsing and correct parser provenance - _gemma_parse_value/_gemma_parse_mapping/_gemma_parse_array now parse nested {}/[] in a single forward pass instead of pre-scanning each subtree with a balanced-brace walk and re-parsing it. Deeply nested wrapper-less Gemma args were O(n^2); they are now ~linear (and ~40x faster at depth 400). - Correct the DeepSeek/GLM/Kimi provenance comments: the cited commit 51fa458a92d6 is unrelated, and GLM/Kimi were never standalone common_chat_parse_* functions (llama.cpp uses common_chat_params_init_glm_4_5 plus a generalized XML parser, PRs #15904 / #16932). - Add tests: Gemma deep-nesting linearity, nested object/array preservation, same-turn distinct-call cap, and the native-template tool-render fallback. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: guard Gemma value parser against non-advancement and missing tokenizer Addresses Gemini review: - _gemma_parse_value now consumes one character when a stray }/]/, sits where a value is expected, so _gemma_parse_array can never stall at the same index on malformed input (a latent infinite loop). - _render_with_native_template returns None when neither a tokenizer nor a processor is present instead of raising AttributeError. - Tests for both. * Studio: fix attribute-form function-XML literal close tag and zero-arg strict call Addresses Codex review of the <function name="..."> attribute form in _parse_function_xml (MiniCPM-5 / MiniMax-M2): - End the call body at the LAST </function> / </tool_call> within the call's window, so a literal close tag inside a code/search argument (e.g. print("</function>")) is preserved instead of truncating the call. - Accept a closed call with no parameters as a valid zero-argument call in strict mode (the function close is already required), instead of rejecting it as a truncated call. - Tests for both, mirroring the legacy <function=...> coverage. * Studio: drop scratch review/planning artifacts from the branch * Studio: fix tool-call parser/loop review findings on the multi-format path Address the live code-review findings on the safetensors/MLX + GGUF tool path: - routes: include the attribute form <function name="..."> in the safetensors capability whitelist so MiniCPM-5 / MiniMax-M2 templates keep the tool pill (parser already handles the form; the post-filter wrongly suppressed it). - safetensors loop: build the plan-without-action re-prompt from the active tools instead of a hardcoded web_search/python string, and gate it on auto_heal_tool_calls, matching the GGUF loop. - safetensors loop: hold a leading bare-JSON object ({"name":..,"parameters":..}) during BUFFERING until it closes, then drain it as a tool call instead of streaming the raw JSON to clients. The DRAINING/STREAMING resolvers still recover a plain JSON answer, so this can never drop content. - parser: anchor the Llama-3 <|python_tag|>NAME.call(...) scan to the tag and chain ; -separated calls, so all semicolon-separated built-ins parse and a literal <|python_tag|>x.call(...) inside a JSON string argument no longer fires the wrong tool. - parser: consume the optional trailing </s> after a named Mistral [TOOL_CALLS]name{json} call, mirroring the array shape. - GGUF streaming strip: use the shared parser patterns (which know [TOOL_CALLS] and <|python_tag|>) so a textual tool call entering DRAINING is stripped instead of leaking the marker to streaming clients. - routes: hoist the _strip_mistral_closed_calls import to module level. Adds regression tests covering each fix; existing parser suite stays green. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fix DeepSeek/GLM/Gemma tool-call review findings Address the live code-review findings specific to the DeepSeek / GLM / Kimi and native-template additions: - parser: in strict mode (Auto-Heal off) require the per-call <｜tool▁call｜end｜> terminator for DeepSeek V3 calls instead of executing on a bare balanced object closed only by the envelope end. - parser: keep GLM string arguments that begin with a quote verbatim (drop the leading-quote case from the JSON-decode probe) so a quoted search query is not decoded down to its inner text. - parser: reject a GLM call with an unclosed <arg_value> in strict mode, and under Auto-Heal keep the partial value rather than dropping it to a no-arg call. - parser: add a balanced wrapper-less Gemma strip (call:NAME{...}) so a nested object/array argument is removed whole instead of leaving a trailing brace; run the balanced Mistral and Gemma strips on the streaming display paths too. - safetensors loop: buffer a leading wrapper-less Gemma call:NAME{...} so it drains and executes instead of streaming the raw call text. - inference: render the native-template fallback on a shallow tokenizer copy instead of mutating the shared tokenizer outside the generation lock, and load the native template from base_model for LoRA adapters. Adds regression tests for each; existing parser suite stays green. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden multi-format tool-call detection from review findings Apply five targeted fixes from the review pass over the multi-format tool path: - routes: route display strip delegates to _strip_tool_xml so Mistral [TOOL_CALLS] blocks with nested JSON are removed from streamed display text, not just the XML forms. - tool_call_parser: skip function/parameter starts that fall inside an already-open parameter block (_inside_open_parameter) so nested example payloads are not mis-parsed as new calls; extract strip_llama3_leading_sentinels so the bare-JSON guard is shared. - safetensors_agentic: probe bare JSON through strip_llama3_leading_sentinels before the balanced-brace check so a leaked header sentinel does not defeat the guard. - tool_healing: allow dotted tool names in the Gemma wrapped start pattern. - llama_cpp (GGUF): buffer wrapper-less Llama-3.2 {"name":..} calls that carry no XML signal, drain a complete object silently and hold an incomplete one, and run the end-of-stream safety net unconditionally so markerless calls are detected and never leak the raw JSON (including truncated fragments). Adds regression tests for the GGUF bare-JSON streaming path and the Mistral display strip. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop bare-JSON tool calls leaking at EOF, oversized, and into history The second review pass flagged that the Llama-3.2 bare-JSON tool-call handling still leaked raw JSON in several spots; ``strip_tool_markup`` only knows XML/bracket markup, so the bare-JSON form survived it. Fix them symmetrically across the safetensors and GGUF loops: - Safetensors stream-end resolver now routes a held bare-JSON fragment to DRAINING (mirroring GGUF) so a truncated ``{"name":..`` cut off by the end of the stream is dropped instead of flushed as assistant content. The 7/10 reviewer finding. - Both loops now drain (suppress) an oversized still-open bare-JSON call once it passes ``_MAX_BARE_JSON_BUFFER`` instead of streaming the raw prefix, gated on a ``"name"`` key so a giant plain JSON answer still streams; a complete oversized call still executes via the safety net. - Add a shared ``strip_leading_bare_json_call`` helper and apply it to the content kept for the assistant turn in both loops, so an executed bare-JSON call is not replayed as visible text or fed back as next-turn history. Plain JSON answers without a ``"name"`` key are untouched throughout. Adds regression tests for the EOF, oversized, and next-turn cases on both backends plus unit tests for the helper. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: bound the Llama-3 python_tag strip on real control sentinels The route display strip's <|python_tag|> arm ran to the next <| of any kind. A tool-call argument carrying a literal <|...|> token (for example <|cite|> inside a string value) truncated the strip early and leaked the call tail into the visible response. Narrow the stop condition to the genuine Llama control sentinels (eot_id, eom_id, python_tag, start/end_header_id, begin_of_text, finetune_right_pad_id) so embedded markup and JSON are consumed while real header/turn boundaries still bound the strip. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden GLM/Gemma parsing, cap GGUF textual calls, share native-template fallback GLM 4.x parser walked a body pre-bounded by the first </tool_call>, so a string argument containing a literal </tool_call> (e.g. code that prints it) was truncated. Walk arg_key/arg_value pairs against the full content instead, since each <arg_value> is delimited by its own </arg_value> and the call's real close is the </tool_call> that precedes the next <arg_key>. Add a truncated wrapper-less Gemma pattern (call:NAME{... with no closing brace) to the markup strip so a call cut off mid-arguments does not leak raw into the visible stream. It runs after the closed form, so a complete call keeps trailing prose. Cap and dedup tool calls parsed from the GGUF TEXTUAL fallback at _MAX_TOOL_CALLS_PER_TURN, mirroring the safetensors loop. Structured delta.tool_calls are grammar-bounded by llama-server, but text parsed straight from content is not, so one runaway turn could fan out into dozens of executions. Extract the native-chat-template fallback into chat_template_helpers (render_native_template / render_with_native_template_fallback) so the transformers and MLX text backends share one implementation. The MLX text path now applies it too, so an Unsloth override template that drops the tools schema no longer silently stops MLX from advertising tools. The MLX VLM path renders via the processor for image tokens and is intentionally left on its own render. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate markerless bare JSON on enabled tools and close parser/strip asymmetries The Llama-3.2 custom_tools bare-JSON form has no marker, so any JSON object with a name key was read as a tool call. An ordinary JSON answer like {"name":"Alice","parameters":{"age":30}} was misclassified as a call to a disabled tool and dropped from the visible response. Gate the markerless form on the enabled tool names (threaded through parse_tool_calls_from_text and strip_leading_bare_json_call, supplied by both streaming loops): an object whose name is not an enabled tool is ordinary content. The marker-based forms keep their name-agnostic behaviour (an explicit signal is a real call attempt), and unrestricted mode stays ungated. Also fix two parser/strip asymmetries the parser already tolerated: - A literal </function> inside a parameter value (print("</function>")) truncated both the core and route strips at the first close, leaking the tail. Extend the strip to the call's real close (last </function> before the next opener), mirroring the parser, without merging separate calls. - The single-object Mistral [TOOL_CALLS]{...} shape parsed but _strip_mistral_closed_calls left it, leaking the raw object into display. Strip the balanced object while keeping trailing prose, matching the array and name shapes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio tools: fix strip/parse symmetry and native-template token for DeepSeek/GLM/Kimi Pass-3 review follow-ups on the multi-format tool parser: - Bare Kimi call (<|tool_call_begin|>...<|tool_call_end|> with no section wrapper) is accepted by the parser, so add it to the closed strip patterns so the streaming (non-final) display strip removes it instead of leaking the markup mid-generation. - Route display strip now also runs the wrapper-less Gemma cleanup, so a Gemma 4 call:NAME{..} no longer leaks into the visible answer. - MLX model record carries base_model for a LoRA adapter so the native-template fallback loads the base repo template rather than the adapter's (often template-less) tokenizer. - Native-template reload forwards the load-time HF token so a gated/private model's repo template can still be fetched (transformers and MLX text paths). - GGUF end-of-stream bare-call heuristic is gated on the enabled tool names so a truncated ordinary JSON object ({"name":"Alice","age":) streams as the answer instead of being dropped as a tool call. Adds regression tests for each case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio tools: gate GGUF bare-JSON suppression on enabled tools and fix python-tag exponent parsing Pass-4 review follow-ups on the GGUF tool loop and Llama-3 parser: - The GGUF bare-JSON suppression sites still keyed off a raw "name" substring, so an ordinary JSON answer whose name is not an enabled tool was dropped when it was truncated, oversized, or reached the no-tool DRAINING fallback (the parser, helper, and safetensors paths were already gated). All three sites now use the shared enabled-name gate, and a held bare-JSON buffer that turns out not to be an enabled call is shown as the answer instead of dropped at stream end. - The Llama-3 python-tag numeric kwarg regex matched only the mantissa, so scientific notation was truncated to its leading digits (1e-3 parsed as 1) and a tool executed with the wrong value. The regex now accepts exponent and decimal forms, and the int/float classification keys off the exponent too. Adds regression tests for the truncated / oversized disabled-name JSON cases (and a counterpart that a truncated enabled call still does not leak) plus the scientific-notation kwargs. * Studio: drop accidentally committed async worker transcripts Eight generated reviewer / async-worker transcripts were committed under studio/backend/async_task_outputs/. They are not imported or referenced by any code and carry only internal task state, so they should never ship in the repo. Remove them and gitignore the directory so they cannot be re-added. * Studio tools: gate safetensors bare-JSON drain, fix nested-name gate and function-XML strip Pass-4 review follow-ups on the shared parser / safetensors loop: - The safetensors oversized and end-of-stream bare-JSON drain branches keyed off a raw "name" substring, so a large or truncated ordinary JSON answer whose name is not an enabled tool was drained instead of streamed. Both now use the shared enabled-tool-name gate, matching the GGUF path. - strip_leading_bare_json_call matched the first "name" anywhere, so a plain JSON answer with a nested name equal to an enabled tool ({"result":{"name":"web_search"}}) was wrongly suppressed. It now extracts the TOP-LEVEL name only, walking past nested objects/arrays and keeping the text when a top-level value is truncated. - The function-XML display strip used a regex negative-lookahead that stopped at a literal <function=...> opener inside a parameter value and then dropped the rest of the answer to EOF. A scan-based strip mirrors the parser (ignores openers inside an open <parameter> via _inside_open_parameter) and closes each call at its real </function>, so trailing assistant text after such a call survives. Adds regression tests for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep tools prompt when native-template probe raises; make helper tests hermetic Pass-4 review follow-ups on the native-template fallback: - render_with_native_template_fallback re-renders the live template with tools=None to detect whether it dropped the schema. A template that requires tools can raise on that probe; that must not discard the already-valid tools prompt. The probe is now wrapped so any error returns the original formatted_prompt (transformers would otherwise fall back to manual formatting and lose the schema; MLX would let the exception escape). - The native-template helper tests imported InferenceBackend just to reach the thin wrapper, which pulls in unsloth and its optional vllm package metadata. They now call the dependency-light render_native_template helper directly so they pass in a backend/test environment without vllm. Adds a probe-raises regression test. * Tool parsing: 3.9 import safety, disabled-Auto-Heal contract, capability gate Round-2 review follow-ups on the multi-format tool-call parser: - tool_call_parser: add `from __future__ import annotations`. The module is dependency-light by design (external llama-server wrappers import it standalone) and the package targets python >=3.9, where its PEP 604 `int | None` return annotations would raise TypeError on import. - safetensors + GGUF drain fallback: gate the leading bare-JSON strip on auto_heal_tool_calls. With Auto-Heal off, a truncated enabled-name fragment that did not parse now stays visible, matching the XML strip in the same branch and the disabled-Auto-Heal contract. With Auto-Heal on it is still suppressed. - safetensors capability gate: match the bare-JSON `{"name":` template marker with a whitespace/escape-tolerant regex so a pretty-printed `{ "name" :` or JSON-escaped `{\"name\":` template is not mis-classified as tool-less. The parser already accepts that whitespace via raw_decode, so the gate must too. Regression tests added for each case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GLM tool-call display strip: treat literal close tag in arg value as data Round-2 review follow-up on the GLM 4.x tool-call format. The GLM call shape is <tool_call>NAME<arg_key>k</arg_key><arg_value>v </arg_value>...</tool_call>. The parser was hardened to walk arg_key / arg_value pairs so a literal </tool_call> inside an argument value (e.g. print("</tool_call>")) is treated as data and the call's real close is the </tool_call> that precedes the next <arg_key>. The display strips still used a non-greedy <tool_call>.*?</tool_call> regex, which stopped at the literal and leaked the call's tail into visible content and stale history. Add _strip_glm_calls, a scan that mirrors the parser's close detection, and run it before the regex arms in every strip pipeline: the core strip_tool_markup, the route _strip_tool_xml display/history cleanup, and the safetensors + GGUF streaming strips. Qwen / Hermes <tool_call>{json} has no NAME token after the opener, so it is left to the regex arms unchanged. Regression tests cover the literal-close-tag leak (core + route), normal GLM calls, back-to-back GLM calls, zero-arg GLM, truncated GLM, and untouched Qwen. * Tool parsing: symmetric "function" bare-JSON alias and route strip parity Round-3 review follow-ups, all parser/strip symmetry fixes. - Bare-JSON "function" alias: the markerless parser accepts a call name via obj.get("name") or obj.get("function"), but the strip/gates only knew "name", so a {"function":<enabled tool>} call executed while its raw JSON leaked. Teach _top_level_bare_json_name the alias (with "name" precedence and the same nested and truncated-name guards), and widen the guards in strip_leading_bare_json_call, the safetensors and GGUF _looks_like_enabled_bare_json gates, and the route capability marker regex. - Route display/history cleanup: strip a tail-only </param> alias close (the parser accepts <param name="...">...</param>), and run the parser's guarded function-XML scan (_inside_open_parameter) before _TOOL_XML_RE so a literal nested <function=...></function> inside an argument value does not truncate the strip and leak the tail. Regression tests added for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio tools: fix DeepSeek strict recovery, Kimi dotted names, Gemma spaced streaming Round 3 review fixes for the DeepSeek / GLM / Kimi tool-call parsing path. - DeepSeek R1 and V3/V3.1 strict parsing (Auto-Heal off): when a call is truncated (missing closing fence or <tool_call_end> terminator), skip it and keep scanning for later well-formed calls instead of breaking out and dropping the rest of the envelope. This matches the Kimi strict parser's recovery behaviour. - Kimi dotted tool names: keep the full name after stripping only the functions. prefix and :idx suffix, e.g. functions.mcp.server-list:0 stays mcp.server-list. The previous split on "." truncated dotted MCP names to their last segment. This matches current vLLM (tool_id.split(":")[0].removeprefix("functions.")) and SGLang (^(?:functions\.)?(?P<name>[\w.\-]+):(?P<index>\d+)$). - Gemma wrapper-less call streaming: hold the whitespace-tolerant prefix (call : NAME) in the streaming suppression buffer, matching the parser's _GEMMA_BARE_TC_RE, so the spaced spelling split across chunks is buffered instead of leaking as visible text. Applied to both the safetensors and llama.cpp streaming paths. - Remove dead _render_with_native_template method and the now-unused copy import from inference.py; the live path uses render_with_native_template_fallback. Adds regression tests for DeepSeek R1/V3 strict recovery, Kimi full dotted name preservation, and the Gemma spaced-call streaming suppression. * Studio tools: honor tool budget in GGUF loop and guard function-XML streaming strip Round 4 review fixes. Both are asymmetric-fix bugs where the final/steady path got a guard the analogous streaming/loop path did not. - GGUF tool-call budget: the safetensors loop counts real tool-call turns against max_tool_iterations (re-prompt stalls excepted), but the GGUF loop only bounded the turn count by the enlarged range (max_tool_iterations + _MAX_REPROMPTS). Since this PR raised _MAX_REPROMPTS from 1 to 3, a model that keeps making valid tool calls could run up to three extra tool rounds (with max_tool_iterations=1, four rounds instead of one). Add a _tool_iters_done counter that increments only when a tool actually executed in the turn, and stop once the caller's budget is spent so the post-loop final-answer nudge fires. A duplicate/disabled no-op turn is a correction turn (like a plan-without-action re-prompt) and does not consume budget, preserving the existing "already completed" re-prompt behavior. - Streaming display strip: the final strip runs the guarded _strip_function_xml_calls scanner (a literal <function=...> inside a parameter value is data, not a nested call), but the GGUF and safetensors streaming strips still used only the open-ended regex arms. When a tool-call argument contained literal function markup, the regex tail ate everything to end-of-text and dropped the real trailing prose after the call's true </function>. Run the guarded scanner (and the balanced Mistral strip) before the regex arms in both streaming paths so streaming and final display agree. Adds regression tests: GGUF valid tool calls respect max_tool_iterations, and the streaming strip keeps trailing prose after a function-XML call with a literal marker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio tools: safetensors tool budget counts only executed turns (GGUF parity) Follow-up to the GGUF budget fix. The safetensors loop charged max_tool_iterations per non-re-prompt iteration (iteration + 1 - reprompt_count), so a duplicate/disabled no-op turn spent a budget slot even though no tool ran. With a small cap this dropped real work: for max_tool_iterations=2, a model that made a valid call, repeated it (an internal no-op correction turn), then made a distinct valid call executed only the first -- the third turn was sent with no tools and the distinct call was ignored. Track whether a turn actually executed a tool (set on record_result) and count only those turns against the cap, matching the GGUF loop. A duplicate/disabled no-op is a correction turn -- like a plan-without-action re-prompt -- and no longer consumes budget, so the model still gets its "already completed" nudge and another tool-enabled turn. Adds a regression test for the small-cap duplicate-then-distinct-call flow. * Studio tools: fix stale Kimi dotted-name regression test test_pr5624_regressions.py still expected functions.my.tool:0 to resolve to the last segment (tool). The parser now preserves the full dotted name (my.tool) after removing only the functions. prefix and :idx suffix, matching current vLLM/SGLang so dotted MCP names like mcp.server-list survive. Update the assertion, name, and module docstring to the corrected contract (the raw id is still preserved on the call). * Studio: render the reasoning block for safetensors and MLX like GGUF enable_thinking chat templates (Qwen3/Qwen3.5/GLM) prefill an unclosed <think> into the generation prompt, so the model emits only the closing </think> then the answer. The safetensors/MLX chat stream emitted that as plain content, so the reasoning showed inline with no collapsible thinking block, while GGUF (which surfaces reasoning via reasoning_content) rendered one. This brings safetensors and MLX to parity. - _ResponsesReasoningExtractor gains a reasoning_prefilled mode that starts inside the reasoning block and splits on the first </think>; default False keeps GGUF and every existing caller byte-identical. It suppresses a stray re-emitted <think> and holds partial markers back across chunk boundaries. - _sf_reasoning_prefill_mode gates the mode on reasoning being enabled for the request, an enable_thinking or enable_thinking_effort style, and the template actually using the standard <think>/</think> markers. Models with a bespoke reasoning channel (e.g. gemma's <|think|>/<|channel>) are excluded so their answer is never swallowed; gpt-oss (Harmony) and thinking-off requests are excluded too. - sf_tool_stream and stream_chunks (the latter also serves MLX) feed text through the extractor, emitting reasoning_content then content deltas, with a per-turn reset in the tool loop and a flush before each tool_start; only the visible delta reaches the monitor reply. The two non-streaming drains split reasoning_content the same way. - Tests: extractor prefilled mode (streaming and edge cases), the gate matrix including the gemma-style exclusion, and a route-replay of the tool-loop reasoning stream. * Studio: render the reasoning block for safetensors and MLX like GGUF enable_thinking chat templates (Qwen3/Qwen3.5/GLM) prefill an unclosed <think> into the generation prompt, so the model emits only the closing </think> then the answer. The safetensors/MLX chat stream emitted that as plain content, so the reasoning showed inline with no collapsible thinking block, while GGUF (which surfaces reasoning via reasoning_content) rendered one. This brings safetensors and MLX to parity. - _ResponsesReasoningExtractor gains a reasoning_prefilled mode that starts inside the reasoning block and splits on the first </think>; default False keeps GGUF and every existing caller byte-identical. It suppresses a stray re-emitted <think> and holds partial markers back across chunk boundaries. - _sf_reasoning_prefill_mode gates the mode on reasoning being enabled for the request, an enable_thinking or enable_thinking_effort style, and the template actually using the standard <think>/</think> markers. Models with a bespoke reasoning channel (e.g. gemma's <|think|>/<|channel>) are excluded so their answer is never swallowed; gpt-oss (Harmony) and thinking-off requests are excluded too. - sf_tool_stream and stream_chunks (the latter also serves MLX) feed text through the extractor, emitting reasoning_content then content deltas, with a per-turn reset in the tool loop and a flush before each tool_start; only the visible delta reaches the monitor reply. The two non-streaming drains split reasoning_content the same way. - Tests: extractor prefilled mode (streaming and edge cases), the gate matrix including the gemma-style exclusion, and a route-replay of the tool-loop reasoning stream. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: don't force a tool re-prompt on a negated intent (safetensors parity) The safetensors _INTENT_SIGNAL claimed to mirror GGUF but was missing the negative lookahead, so a refusal like "I will not search the web for that" matched the "i will" intent and triggered the plan-without-action re-prompt (STOP... you MUST call a tool), overriding a valid no-tool answer. GGUF already excludes not/never. Add the same (?!\s+(?:not|never)\b) lookahead so both backends agree. Extends the intent parity test with negated refusals. * studio: parse the outer envelope before DeepSeek/Kimi markers embedded in its args parse_tool_calls_from_text ran the DeepSeek/Kimi marker pre-pass before the shared <tool_call>/<function=...> parser. When a Qwen/Hermes call's argument contained literal Kimi/DeepSeek markup (for example a user asking the model to explain that syntax), the pre-pass matched the embedded marker and returned it, executing the wrong tool and dropping the real call. Skip the pre-pass when a <tool_call> or <function=...> envelope opens before the first DeepSeek/Kimi marker, so the shared parser takes the outer call; a genuine marker-led call (no leading envelope) still goes through the pre-pass. Tests for the embedded-marker case and the control. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: trim redundant comments (comment-only, AST-verified) * Studio: trim redundant comments (comment-only, AST-verified) * Studio: prevent Gemma tool-parser DoS on stray delimiters _gemma_parse_value returned the input index unchanged when text[i] was a stray delimiter (,}]), so the list and mapping caller loops that advance on the returned index spun forever at 100% CPU on malformed input such as [},]. Advance past the delimiter so parsing always terminates. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: strip Magistral [THINK] reasoning from final display/history strip_tool_markup removed [TOOL_CALLS] and <function> markup but left a leading Magistral [THINK]...[/THINK] block intact, so its bracket-form reasoning (not the <think> the reasoning channel renders) leaked into the safetensors display and conversation history while GGUF/llama.cpp routes it natively. Drop the leading reasoning block at end-of-turn (final=True) via the existing _strip_mistral_reasoning helper; streaming is untouched. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep times in wrapper-less Gemma tool arguments The wrapper-less Gemma value scanner used _GEMMA_KEY_RE = [\w.\-]+ for keys, which also matches a digit-leading token, so a comma followed by a time or ratio inside a value (call:web_search{query:meet at 10:00, 11:00 tomorrow}) was misread as a new 11: key, truncating the query and injecting a bogus argument. Require keys to start with a letter or underscore, matching the identifier-start rule the wrapped path already uses (_GEMMA_NEXT_KEY_RE). Add a regression test. * Studio: treat markers/close-tags inside tool-call arguments as data Four parser correctness fixes where a valid argument string was mistaken for structure: - DeepSeek: find the envelope-end token outside JSON strings, so a query/code argument containing the literal token no longer truncates the body and drops the whole call. - GLM: locate the real </arg_value> as the one whose next token is <arg_key> / </tool_call> / end, so a value containing a literal </arg_value> (or </tool_call>) is kept instead of executing the tool with corrupted arguments. - Attribute-form <function name="..."> envelopes now count in the embedded-marker guard, so a DeepSeek/Kimi marker inside a parameter value does not hijack the outer call and run the wrong tool. - Wrapper-less Gemma call:NAME{...} is gated on the enabled tool names (parse and display strip), mirroring the Llama bare-JSON gate, so a disabled/example name in prose is not stolen as a call and the real answer is preserved. Add regression tests for each. * Gate route Gemma wrapperless strip by enabled tools; make Kimi section-end search string-aware Route-level display stripping now threads the enabled tool-name set into the Gemma wrapperless-call strip, so prose that mentions a disabled tool (call:foo{...}) is preserved while active tool calls are still stripped. This mirrors the parser-level gate already used in tool_call_parser. The Kimi section-end lookup now searches outside JSON string literals, so a section-end marker appearing inside an argument string no longer triggers a false truncation that drops a valid tool call. * Run DeepSeek/Kimi pre-pass when a closed tool-call example precedes a real block The marker pre-pass was skipped whenever any <tool_call>/<function> opener appeared before the first DeepSeek/Kimi marker, even when that opener was a CLOSED syntax example in prose that ends before the real block. In that case parse_tool_calls_from_text skipped the DeepSeek/Kimi parsers and the genuine tool call was dropped while a phantom tool named in the example ran instead. Only treat a marker as embedded in a leading envelope when removing the closed outer <tool_call>/<function> envelopes also removes every marker (the marker actually sat inside one). A marker left standing is a real call, so the pre-pass runs. The legitimate case of a marker inside a closed outer envelope's arguments is preserved. * Honor reasoning_effort none in safetensors prefill; strip Magistral reasoning while streaming Two safetensors/MLX reasoning fixes surfaced in review: _sf_reasoning_prefill_mode only checked enable_thinking, so an enable_thinking_effort (GLM-5.2) request that disables thinking via reasoning_effort=none (without enable_thinking=False) still began in prefilled-<think> mode. A plain answer with no </think> was then swallowed whole into reasoning_content and the visible response came back empty. Thread reasoning_effort into the predicate and treat none as disabled, mirroring _request_reasoning_kwargs. strip_tool_markup_streaming stripped tool markup but not the leading Magistral [THINK]...[/THINK] bracket block, so the raw chain-of-thought leaked into the streamed safetensors content instead of the reasoning drawer (GGUF routes it natively). Apply _strip_mistral_reasoning first, matching the final strip; an unclosed [THINK] is held from the marker on so nothing flickers. * Heal truncated outer tool envelopes and keep quoted Gemma args intact Two follow-ups from review of the marker pre-pass and Gemma parsing: The leading-envelope guard only removed CLOSED outer <tool_call>/<function> envelopes before deciding whether a DeepSeek/Kimi marker was embedded, so a truncated outer call missing its close tag (whose argument embeds a marker) was treated as a standalone marker and the embedded sample ran instead of the intended outer call being Auto-Healed. Decide on the last outer opener before the marker and whether it closed before the marker instead, so a closed syntax example still runs the pre-pass while a real closed-or-truncated outer call keeps it. The wrapper-less Gemma argument scan tracked bracket depth but not quotes, so a quoted value containing a comma followed by a key-like token (a search query such as "weather, location: Boston") was split mid-string, truncating the value and fabricating an extra argument. Track quote state (with escapes) so the top-level comma boundary is only taken outside quoted spans. * Span outer envelopes to their real close when locating embedded markers Locating the DeepSeek/Kimi marker relative to a leading outer envelope used the FIRST close tag after the opener, so a literal </function> or </tool_call> inside an argument value (for example python code that contains the text) was mistaken for the envelope boundary. The marker after it was then treated as a standalone call and the embedded sample ran instead of the intended outer call. Match the closed outer envelopes with the shared patterns that already extend to the real final close (a literal close inside a value is data), and treat a marker that survives their removal as embedded only when a still-open (truncated) outer opener precedes it, so Auto-Heal still repairs a truncated outer call. A closed syntax example before a genuine block still runs the pre-pass. * Span the tool_call outer envelope to its real close in the marker guard The leading-envelope check reused the lazy <tool_call>.*?</tool_call> strip pattern, so a Qwen/Hermes JSON argument containing a literal </tool_call> ended the span early. A DeepSeek/Kimi sample later in that same string then survived the closed-envelope removal, and the pre-pass executed the embedded call instead of the outer <tool_call>. The <function> arm already spanned to its real close; give <tool_call> the same real-close pattern (with the negative lookahead that keeps back-to-back calls separate) so a literal close inside a value is data. * Preserve no-tool Gemma prose and keep later R1 calls when healing a close Two review follow-ups: _gemma_strip_gate returned None when no tools were enabled, and None means strip every markerless call:NAME{...} block, so a no-tool answer that documents the syntax (or the Anthropic display path, which passes an empty tool list as None) had that prose deleted. It is a display/history gate, so return the enabled-name set instead -- an empty set when no tool is enabled, which strips nothing because every call:NAME{...} is then prose. The DeepSeek R1 heal path located the close fence with an unbounded forward search, so when a first call had balanced JSON but omitted its fence the search landed on a LATER call's terminator and pos advanced past that valid call, dropping it. Match the close immediately after the JSON (whitespace-skipped) like the strict path, and advance by just the JSON when it is absent, so a multi-call turn keeps its later well-formed calls (heal is now a superset of strict). * Resume wrapper-less Gemma scan past a consumed call's balanced body The markerless call:NAME{...} scan used finditer, which resumes right after the opening call: token, so a nested call:OTHER{...} mentioned inside the first call's own quoted string argument (for example a web_search query that quotes the Gemma tool syntax) was re-matched and returned as a spurious second tool call, executing an unintended tool. Walk with a manual cursor that resumes after the outer call's balanced body (brace matching already skips quoted braces), so a call's arguments are never rescanned. Genuinely separate back-to-back calls and disabled/example prose are unaffected. * Mistral outer call wins over XML literals; align healer signals with its parser Two follow-ups on the shared-parser ordering after the healing-passthrough merge: - A well-formed [TOOL_CALLS] call whose JSON arguments quote tool XML parsed the literal instead of the outer call (executing the wrong tool). When the first XML signal sits inside a leading balanced Mistral body it is argument data, so the Mistral parser now runs first; an XML signal before the trigger keeps the normal order, so a [TOOL_CALLS] literal inside an XML call's arguments still stays data. - passthrough_healing buffered streams on the parser module's broadened signal list (now including <|python_tag|> and [TOOL_CALLS]) but promotes with core.tool_healing, which does not parse those forms: a streamed Mistral or Llama text call was held until finalization and flushed as prose. The healer keeps its own signal list limited to the formats it can promote, restoring immediate streaming for the rest. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: Gemma wrapper-less marker literals and quotes, GLM embedded close pair - The Gemma fallback deferral now keys on an actual wrapped opener (_GEMMA_TC_RE), not the wrapper literal anywhere in content: a wrapper-less call whose argument merely mentions <|tool_call> has nothing tool_healing can parse, and deferring it lost the call entirely (not executed and stripped from display). - New _gemma_body_brace_end boundary scanner honors single- and double-quoted strings like _gemma_parse_stripped_body, shared by parse and strip, so a quoted brace in a code argument (code:print('}')) no longer truncates the executed arguments or the strip span. - _glm_value_close now requires a structural </arg_value> to sit at balanced quote state: the full pair </arg_value></tool_call> embedded inside a string literal is data, not an early close. When no candidate balances, the first token-valid close wins as before. * Address review: leading envelopes win over rehearsed literals - New _first_foreign_tool_signal shared by the leading-envelope guards adds <|python_tag|> to the protected signal set: the spelled-out literal inside a Mistral call's arguments (a query about Llama built-in tool syntax) executed the inner literal instead of the outer call. - New _xml_signal_inside_leading_bare_json guard, sibling of the Mistral one: a leading bare-JSON call whose string argument quotes tool XML (a code value citing <function=...>) had the literal promoted by the shared XML pass before the bare-JSON parser ran. - Magistral [THINK]...[/THINK] is dropped once at parse entry instead of only inside the Mistral parser, so a call rehearsed in the think block in a foreign format can no longer be promoted while the real call after the block is lost. Parse now agrees with the display strip. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: a disabled leading bare-JSON object keeps its literals as data When the leading bare-JSON object is ordinary content (name not an enabled tool), the guard proved the first tool signal sits inside it, so falling through to the XML/python_tag passes promoted quoted string data as a real call. Drop the object and parse only the tail: a real call after the object still parses, nothing inside it can be promoted. * Address review: apostrophes in raw Gemma values, GLM strict key contract, per-model template token - Quote openers in the wrapper-less Gemma boundary and body scanners now require value-start context (after : { [ ( , =): an apostrophe inside an unquoted value (query:what's the weather) opened quote mode, swallowed the real closing brace, and lost the whole call on common contraction queries. Quoted values keep hiding delimiters as before. - A GLM <arg_key> with no <arg_value> tag now rejects the call in strict mode, matching the unclosed-value contract, instead of executing the tool with the argument silently dropped; Auto-Heal keeps the lenient skip. - The native-template fallback reads the hf_token stored on the model record instead of the instance-wide last-load token, so a later token-less load cannot break template fetches for a previously loaded gated model (both the transformers and MLX backends). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: Mistral literals inside leading JSON, whitespace-tolerant wrapped Gemma opener - The leading bare-JSON guard now treats the [TOOL_CALLS] trigger as a foreign signal: the Mistral parser runs before the bare-JSON one, so a literal quoted inside the leading object's strings was promoted over the outer call (or over ordinary JSON content). - tool_healing's wrapped Gemma opener tolerates whitespace around call and the colon: sampling drift emits call: name{ and call : name{, and rejecting those lost the call entirely because no fallback re-parses the wrapped form. Strict mode still requires the closing tag. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: DeepSeek/Kimi markers inside leading JSON and Mistral envelopes stay data The DeepSeek/Kimi pre-pass runs before the outer-call parsers, and _marker_inside_leading_envelope only protected XML envelopes: a marker quoted inside a leading bare-JSON or Mistral call's argument strings was promoted as a separate no-arg call and the real outer call dropped. The guard now recognizes those two leading envelopes as well; standalone DeepSeek/Kimi calls keep parsing. * Address review: accept dotted Gemma argument keys in the key-quoting scanner The scanner quoted keys of [alnum_-] only, so a dotted key (user.name:...) was left unquoted, json.loads failed, and the whole wrapped call was lost (parse empty, strip wipes the markup). Dots now match the parser's own key/name charset. * Address review: a real DeepSeek/Kimi call after a disabled leading JSON object still parses DeepSeek/Kimi markers are foreign signals for the leading bare-JSON guard too: a marker literal inside a disabled leading object made the envelope guard skip the pre-pass for the whole message, so a real DeepSeek/Kimi call after the object was dropped. Routing the case through the guard's drop-and-parse-the-tail recursion reaches the real call while the literal inside the object stays data. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: leading Mistral call owns the turn, dotted keys after bare values - A LEADING parseable [TOOL_CALLS] call now runs the Mistral parser first unconditionally: literal XML in trailing prose after the call was promoted by the earlier shared XML pass, executing the quoted example instead of the real leading call. XML leading keeps the normal order. - _GEMMA_NEXT_KEY_RE accepts dots so a dotted key after a bare value (query:foo,user.name:bob) ends the value at the comma instead of being swallowed into it, matching the round-earlier key-quoting charset. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: a leading wrapper-less Gemma call owns the turn A quoted foreign literal inside a leading wrapper-less Gemma call's argument (a query citing another tool syntax) was promoted by tool_healing before the Gemma fallback ran, executing the quoted example and dropping the outer call. New leading guard, sibling of the Mistral and bare-JSON ones, gated on an enabled name since the form is markerless. Foreign markup leading keeps the normal order. * Fix merge resolution: restore both leading-guard test classes intact * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: markup quoted inside a nameless leading JSON answer stays data The leading bare-JSON guard required a top-level name, so a structured JSON answer quoting tool markup in its strings (a response_format turn documenting a tool's syntax) had the literal promoted by the later passes. A nameless leading object that parses as real JSON now routes through the same decline-then-parse-the-tail path; non-JSON braced prose keeps the old behaviour, and a real call after the answer still parses. * Address review: JSON answers stay data, nested Gemma quotes, earliest envelope, no failure caching - A whole-content JSON value is a structured answer: the markerless Gemma scan and its strip no longer promote or strip a quoted example of an enabled tool's syntax inside it. - Nested stripped-stream Gemma values now unquote quoted string leaves recursively, so {loc:{city:"New York"}} hands the tool New York, matching the top-level coercion. - The DeepSeek/Kimi pre-pass dispatches by earliest envelope opener, so a leading real call wins over a trailing example of the sibling format in either direction. - A failed native-template fetch is no longer cached as no-template: the next call retries after the model record's token is fixed or a transient Hub error clears; only definitive loads are cached. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: closed calls precede the marker pre-pass, truncated Gemma scan stops, quoted nested delimiters - A closed non-DeepSeek/Kimi call preceding the first DS/Kimi marker owns the turn: a trailing syntax example, or one quoted inside a wrapped Gemma argument, was promoted by the pre-pass and dropped the real leading call. Wrapped Gemma joins the outer-envelope pattern sets. - An unbalanced wrapper-less Gemma call now stops the scan (mirroring the strip contract) instead of resuming inside its own argument text, where a quoted enabled call would be promoted. - Raw-quoted strings in nested stripped-stream Gemma values hide delimiters, so {city:"New, York"} is one value instead of a split pair, returned unquoted like the top-level coercion. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: string-marker literals in wrapper-less args, mid-value quoted phrases - The wrapper-less deferral guard no longer keys on the <|"|> literal: a real call whose argument merely mentions the string marker was deferred to tool_healing, which has no wrapped opener to parse, losing the call. The wrapped-opener check alone owns the deferral. - Double quotes now also open at the start of a word, so a quoted phrase mid-value (query:find "weather, location: Boston", limit:3) hides its delimiters instead of splitting the value into garbage keys; apostrophes keep the value-start-only rule so contractions stay prose. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: strict GLM refuses in-quote close fallback, Gemma guard covers preambles - _glm_value_close gains a strict flag: a truncated value whose only close candidates sit inside a string literal rejects the call in strict mode (Auto-Heal keeps the lenient partial), restoring the strict contract the quote-aware fallback had weakened. - The leading wrapper-less Gemma guard no longer requires the call to open the response: a visible preamble before call:NAME{...} is the normal shape, and the quoted foreign literal inside the argument was promoted again in that shape. An enabled balanced call beginning before the first foreign signal owns it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: contextual GLM quote openers, disabled Gemma examples stay prose, JSON array answers - The GLM value-close quote tracker uses the same contextual openers as the Gemma scanners (single quote after punctuation context, double quote also at word start), so strict mode accepts a normal apostrophe value again while still rejecting a truncated value whose only close candidates sit inside a string literal. - A disabled wrapper-less Gemma call is prose by design, so a tool literal quoted inside it no longer promotes: the span is dropped for parsing and the tail parsed, mirroring the nameless-JSON guard. - Leading JSON ARRAY answers join the leading-JSON envelope guard, so a marker quoted inside a structured array response stays data. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Align closed-envelope regression test with the document-order contract The test asserted the pre-round-13 behavior (trailing DeepSeek/Kimi block wins over a leading closed envelope) while the shipped rule is document order: the leading closed call owns the turn. Rename the test and assert the leading call so the suite matches the contract exercised by test_leading_xml_call_wins_over_trailing_kimi_example. * Parse a leading Llama-3.2 bare-JSON call before the markerless Gemma scan The bare-JSON form only ever matches a leading call object, and document order says that call owns the turn. Running the Gemma wrapper-less scan first let an enabled call:NAME{...} snippet quoted inside the leading call's string arguments steal the turn when the JSON was not the whole content (trailing prose or a second ;-separated call), executing the quoted tool instead of the real one. Reordering cannot take a leading Gemma call's turn since that content never starts with an object brace. * Leading-call ownership: Mistral trigger in Gemma guards, closed bare JSON before markers, depth-aware nested Gemma values Three parser gaps against the document-order contract: The wrapperless Gemma leading guards did not count [TOOL_CALLS] as a foreign signal, so a leading Gemma call quoting a Mistral snippet in its argument lost the turn to the quoted literal. Both the enabled-call and disabled-example guards now include the trigger, matching the bare-JSON guard's local inclusion. _marker_inside_leading_envelope required the DeepSeek/Kimi marker to sit inside the first closed bare-JSON or Mistral call. A marker after that closed call (a trailing example or data in a later ;-chained call's strings) now also defers to the leading call, the same inside-or-after rule the closed XML envelope patterns already applied. The nested Gemma primitive value scan split on every comma, corrupting arguments like opts:{code:print(1,2),lang:py}. It now applies the same paren/brace depth, contextual quote openers, and comma-only-before-a-key mapping rule as the top-level scan. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gemma leading guard: a closed enabled call preceding the signal owns the turn The wrapperless Gemma guard only claimed the turn when the first foreign signal sat inside the first enabled balanced call. When that call closed before the signal (a second call quoting a Mistral or Kimi literal, or a trailing prose example), the guard forfeited the turn and the foreign parser promoted the quoted literal, dropping the real Gemma calls. Apply the same inside-or-after ownership rule as the closed bare-JSON and Mistral envelopes, gated on an enabled name so the name-agnostic legacy path is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Marker guard: only an executable leading bare-JSON call owns the turn The bare-JSON branch of the leading-envelope marker guard claimed the turn for any NAMED leading object. A disabled-name object is prose by design (the bare-JSON parser will not execute it), so deferring the DeepSeek/Kimi pre-pass to it lost the real later call entirely. Gate the ownership claim on the enabled set (or the name-agnostic None path). A marker inside the disabled object's own strings stays data, matching the tail-exclusion contract; a marker after it now falls through so the pre-pass parses the real call. The Mistral branch stays ungated since [TOOL_CALLS] parsing is never name-gated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gemma scan skips leading JSON answers; GLM heal bounds values at structural tags Two fixes to the document-order data contracts: The markerless Gemma scan only exempted whole-content JSON, so a leading JSON answer followed by prose had an enabled call:NAME{...} snippet inside its strings promoted to a real executed call and stripped from the displayed answer. Both the parse and strip scans now start after a balanced json-valid leading value span, keeping parse and strip mirrored. Real calls after the answer still parse; mid-prose JSON gets no exemption. The GLM heal fallback for a missing closing arg_value tag took the entire remainder as the value, executing markup-contaminated arguments like city="NYC</tool_call>" and swallowing trailing prose. The healed value now stops at the next arg_key or tool_call close and the pair walk resumes there. EOF-truncated values keep the partial heal, strict mode still rejects, and closed values holding a literal close tag in quotes are untouched. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Compress docstrings in the multi-format tool parser to their contract essence * Condense parser guard comments and test narration to contract essentials * verify_import_hoist: exempt __future__ imports and same-diff relocations Two false positives fired on this PR's refactor. A from __future__ import is a compiler directive whose name never appears as a runtime load, so HOISTED-IMPORT-UNUSED can never see it used, yet the file requires it for PEP 604 annotations on Python 3.9. TARGET-CHANGED flagged the deliberate move of the strip-pattern constants into core.inference.tool_call_parser as a silent re-point even though the old module-level target was removed and the new one added in the same diff. Both get narrow exemptions; a re-point to a pre-existing target is still caught, and the self-test negative controls all pass unchanged. * Leading bare-JSON calls own the turn; function calls end at the first balanced close The XML-signal guard for a leading bare-JSON call required the signal strictly inside the object, so a trailing XML example stole the turn from the leading call; it now applies the same inside-or-after rule as the Mistral guard. Function-XML calls also ended at the LAST close tag, which let prose after a closed call that mentions a literal close tag get swallowed into the final parameter value; calls now end at the first close tag that is not inside an open parameter, and the strip mirrors the same rule so parse and strip agree. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Attribute-form calls end at the first balanced close; bare-JSON strip requires the call shape The attribute form parser still kept the last close tag in the call window, folding prose after a closed call into the final parameter value. It now takes the first close not inside an open parameter, the same rule the equals form and the strip already use. The leading bare-JSON strip deleted any closed object whose top-level name matched an enabled tool, including plain JSON answers the parser correctly rejects as non-calls. The strip (and the drain gate that delegates to it) now requires the parser's exact call shape, so answers like {"name":"web_search","result":...} stream and display intact. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * False-alarm markers keep the answer; the bare-JSON strip consumes the whole chain The trailing strip arms dropped everything from a bare marker to EOF, so a normal answer that mentions [TOOL_CALLS] or another marker literally was truncated (or fully swallowed when it started with the literal) after the no-call drain fallback. Those arms now require a call-shaped lookahead or marker-at-EOF before dropping; truncated real calls still strip. Chained bare-JSON turns executed both calls but stripped only the first object, so the second call's raw JSON replayed into the next assistant history message alongside the structured tool_calls. The strip now consumes the entire chained run of call-shaped enabled objects while non-call answers, disabled names, and trailing prose stay intact. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * DeepSeek and Kimi trailing strip arms require a call-shaped lookahead Same false-alarm rule as the bare-word markers: a prose answer that mentions a DeepSeek or Kimi marker literally keeps its tail, while truncated real envelopes and bare end-of-text fragments still drop. * Attribute-form containment, parameter-close-decides rule, preamble-tolerant Mistral guard, strict strip shape Four document-order and containment fixes. A leading attribute-form call now parses before the shared XML pass, so markup quoted in its parameter stays data. The open-parameter scan lets the parameter's own close tag decide, so any number of literal function closes inside one value stay data, restoring the pre-close-scan behavior for multi-close arguments. The leading-Mistral guard tolerates a visible preamble, with the leading-bare-JSON guard running first so a trigger quoted inside a leading JSON object stays data. The bare-JSON strip requires the parser's top-level name in every mode, so nested-name JSON answers survive name-agnostic stripping. * Keep buffering long wrapper-less Gemma tool names instead of leaking the prefix The streaming buffer stopped holding a call:NAME prefix at a fixed 32-char cap, so a Gemma wrapper-less call to a tool whose name exceeds that (OpenAI allows 64 chars, MCP names run longer) streamed its raw call:longname text as visible content before the end-of-turn parser executed it. Hold the variable-length prefix while it still matches the call: shape, bounded like the bare-JSON path and self-terminating into prose, draining once the opening brace arrives. * Keep prose that only mentions DeepSeek/Kimi markers in the route display strip The route-level _TOOL_XML_RE DeepSeek/Kimi arms consumed from an opener up to the end of text whenever the marker appeared, so an answer that merely refers to a marker (for example "See <|tool_call_begin|> in the docs") had the rest of the reply truncated. The parser-level _TOOL_ALL_PATS already gates these arms with a call-shaped lookahead. Mirror it here so a marker is only stripped when a real call follows it or it is a bare fragment at end of text. * Tighten tool-calling parser and backend comments * Pass trust_remote_code when reloading native tokenizers The native-template fallback re-fetches a model's native chat template from its repo when an Unsloth override template drops the tools schema. The secondary AutoTokenizer.from_pretrained threaded hf_token but not trust_remote_code, so for a model loaded with trust_remote_code=True whose tokenizer repo carries custom code the reload raised, was swallowed, and the request silently kept the tool-dropping prompt for a model that supports tools. Store the loaded trust_remote_code on each backend's per-model info dict and source it in render_native_template, so the reload re-uses exactly the consent granted at load. For a LoRA adapter the reload targets the base model, whose remote code was gated and loaded under the same stored flag, so re-passing it executes no unconsented code. Falsy stored flag preserves the prior behaviour. Adds a regression test that fails without the flag (custom-code reload raises, returns None) and passes with it (tools-advertising native prompt returned). * Treat <|python_tag|> as an outer marker envelope A Llama-3 <|python_tag|> tool call (built-in NAME.call(...) or custom {json} form) whose argument quotes a complete DeepSeek/Kimi example was hijacked by the DeepSeek/Kimi marker pre-pass: the embedded example (for example delete_all) executed instead of the real outer call. python_tag is Llama-3's tool-call envelope, so a marker quoted inside its arguments is data, the same as for <tool_call>, <function=...>, bare JSON, Mistral and wrapper-less Gemma, which the guard already covers. Add <|python_tag|> to _OUTER_ENVELOPE_OPEN_RE with a call-shaped lookahead (mirroring the _TOOL_ALL_PATS python_tag arm) so the marker pre-pass is suppressed when a python_tag call opens before the first marker, while a bare prose <|python_tag|> mention is left untouched. * Tighten tool-call parser comments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <michaelhan2050@gmail.com> Co-authored-by: Daniel Han <info@unsloth.ai> Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>	2026-07-06 15:40:46 -07:00
Daniel Han	eb1ef44255	Studio: Gemma tool-call streaming follow-ups + nested-XML escape fix (#6476) (#6611) ... * Quote-aware Gemma strip, symmetric unstarted cleanup, ReDoS anchor Address review findings on the tool-strip and streaming paths: - strip_tool_call_markup stripped Gemma-native spans with a plain regex that stops at the first <tool_call|>, so a literal close marker inside a <|"|>-quoted argument truncated the span and leaked its suffix into visible text. A brace/quote-aware _strip_gemma_native_spans now removes complete spans (keeping an incomplete one unless final), matching the parser's own balance logic. - The Gemma close pattern this PR added (<\|tool_call>.*?<tool_call\|>) had no \Z fallback, so a run of unclosed markers backtracked from every open position (quadratic, and the streaming stripper re-scans per token). It is now anchored to (?:<tool_call|>|\Z) like routes/inference.py's _TOOL_XML_RE, linear with identical output on well-formed input. - _SameTaskStreamingResponse added unstarted_cleanup for the OpenAI passthrough, but the local GGUF/safetensors streams that enter _TrackedCancel before returning only unregister in the generator finally, which never runs if the client disconnects before the body iterator starts, leaking cancel-registry entries. Each such stream now passes unstarted_cleanup to exit its tracker. - __call__ reads _unstarted_cleanup via getattr so a response built through __new__ (the cancel-timing test) without __init__ does not raise AttributeError; the test also sets the attribute explicitly. - Document that the verbatim /v1/chat/completions passthrough delegates <think>/<|tool_call> splitting to llama-server (--jinja, --reasoning-format auto) and is intentionally not re-parsed locally, noting the llama.cpp dependency. Adds a regression test for the close-marker-inside-quoted-argument strip. * Tighten comments on the tool-strip and streaming paths Compress the verbose comment blocks added with the Gemma tool-call / streaming work to crisp one or two liners, drop restatements of obvious code, and shorten docstrings, keeping the load-bearing rationale (ReDoS anchor, quote-aware strip, unstarted-cleanup, llama.cpp passthrough dependency). Code is unchanged (verified comment-only via AST/ast signature, docstrings stripped). * Harden Gemma parse/strip: span-aware XML fallback and quote-aware streaming - Security: the XML fallback in parse_tool_calls_from_text scanned the whole content for <function=...> markers and only skipped those inside an open XML parameter, not those inside a collected JSON/Gemma candidate span. A balanced but unparsable Gemma call whose argument data contained XML tool markup (<|tool_call>call:outer{code:<function=terminal>...}<tool_call|>) therefore fell through to the fallback and returned an executable terminal call. The fallback now also excludes <function=> markers inside any candidate span, including ones that failed to parse. - strip_tool_call_markup no longer skips the generic Gemma regex after running the quote-aware _strip_gemma_native_spans, so a closed Gemma span the helper cannot match (malformed, e.g. <|tool_call>{"name":"x"}<tool_call|>) is still stripped instead of leaking its opener and payload into visible text. - _strip_gemma_native_spans stops at the first unbalanced start instead of re-scanning every later start to EOF, keeping it linear on a run of unclosed markers rather than quadratic. - The GGUF and safetensors streaming strippers run _strip_gemma_native_spans before the regex patterns, so a well-formed streamed call whose quoted argument contains a literal close marker no longer leaks its suffix into incremental display. Adds regression tests for the nested-XML escape and the malformed-span strip. * Avoid remainder copy in _strip_gemma_native_spans Match the Gemma close marker with re pos directly on the buffer instead of slicing tail = text[brace_end + 1:] on every span. The streaming strippers re-scan a growing cumulative buffer per token, so the per-span remainder copy was quadratic. Behavior is unchanged. * Exclude unclosed Gemma/JSON starts from the XML tool-call fallback The nested-XML guard only skipped <function=> markers inside recorded candidate spans, but a span is recorded only when the braces balance. An unbalanced call such as <|tool_call>call:outer{code:<function=terminal>... recorded no span, so the fallback still promoted the inner <function=> to an executable terminal call. Treat unclosed JSON/Gemma starts as exclusion spans through EOF before scanning. Standalone <function=> calls with no preceding unclosed start still parse. Regression tests added. * Skip doomed tool-strip passes to avoid quadratic rescans The lazy closed-pair strip patterns (<tool_call>.*?</tool_call>, <function=...>.*?</function>) rescan to EOF from every opener when their close token is absent, which is O(n^2) and re-runs per streamed token. Add strip_tool_patterns, which skips a pass whose close token is not present in the text; output is identical to the per-pattern loop (verified by fuzz), and a degenerate run drops from ~minutes to milliseconds. Used by strip_tool_call_markup and the GGUF/safetensors streaming strippers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Use full tool-call envelopes to close nested-XML escape variants Key the parser and stripper off the full <|tool_call>...<tool_call|> / <tool_call>...</tool_call> envelope (start to close marker, searched after the braces; EOF if unclosed) instead of just the braces: - XML between the closing brace and the close marker (call:outer{broken:{x}}<function=terminal>...<tool_call|>) is now inside the envelope, so the fallback no longer promotes it to a tool call. - A balanced inner call inside an unclosed outer (call:outer{code:<|tool_call>call:terminal{...}<tool_call|>) is skipped via the envelope nested check, not just the XML fallback. - strip_tool_call_markup searches for the close marker after the braces, so junk before <tool_call|> is stripped through the close and text after it is preserved instead of truncated to EOF; a no-close run stops early (linear). Regression tests added; standalone XML and well-formed calls unaffected. * Fix non-final Gemma strip and missing-close recovery for PR #6611 Split the nested-skip from the XML fallback exclusion: nesting is decided by each marker's brace region, so a balanced call after one with a missing close marker is recovered instead of being swallowed to EOF. Only the XML fallback keeps the search-to-close envelope, so trailing nested markup still cannot escape as an executable call. Use a closed-only Gemma pattern in the non-final strip list so an incomplete block is preserved (matching the JSON and function paths); the final list keeps the close-or-EOF Gemma pattern in its original position, so streaming display output is byte-for-byte unchanged. Add regression tests for both cases. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Block gap-nested tool markers and fix XML strip order for PR #6611 Decide candidate nesting by a per-marker coverage region paired with a per-format stack (a close after the braces pops the nearest still-open marker of that format). A closed outer call now covers up to its own close marker, so a JSON or Gemma tool marker smuggled between the outer braces and that close is treated as data instead of being executed. An outer that balances but has no close of its own covers only its brace region, so a later sibling after an omitted close marker is still recovered (adjacent calls use an exclusive end bound so the next call is not misread as nested). Strip every closed pair (JSON, Gemma, function) before any to-EOF sweep, so a closed function call whose parameter text contains a bare Gemma opener is removed as a unit and the to-EOF sweep can no longer drop the visible text after the close. Add regression tests for both. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Strip closed tool blocks before the Gemma final sweep for PR #6611 The final display strip ran the quote-aware Gemma helper before the closed JSON/function patterns. A closed <tool_call>...</tool_call> or <function=...>...</function> block whose argument data held a call-form Gemma opener (e.g. a "<|tool_call>call:t{" string) was read as an incomplete Gemma span and truncated to EOF, dropping the block's close and any visible text after it. Strip closed JSON/function blocks first, so such a block is removed as a unit before the helper runs. Centralize the final strip order in a shared strip_tool_markup_final so strip_tool_call_markup and both streaming display wrappers (safetensors, llama_cpp) stay in sync, and apply the same closed-block pre-pass to the non-final path. Add regression tests for the JSON and function variants. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Recover XML/JSON siblings after a close-less tool marker for PR #6611 Two fixes so the XML fallback and marker coverage recover a later valid call after an earlier marker omits its close, matching the candidate loop: Reuse the candidate marker-coverage in the XML fallback instead of a separate search-to-close-or-EOF envelope. A balanced but close-less marker now covers only its brace region there too, so a following <function=...> sibling is recovered rather than filtered as nested data; an unbalanced marker still covers to EOF and a closed one still covers through its close, so nested XML stays blocked. Ignore a close token that falls inside another call's balanced braces when pairing closes in _marker_coverage. Such a token is that call's quoted argument data, so it no longer pops an earlier close-less marker and extends its coverage over a later valid sibling. Add regression tests for both. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the closed-block strip pre-pass Gemma-span-aware The final display strip ran the closed JSON/function regex pre-pass before removing Gemma-native spans, so a literal <function=...> quoted inside a Gemma argument plus any later </function> (a real call's close or even prose) was deleted across the Gemma boundary. That mangled the Gemma close marker, the quote-aware helper then saw an unclosed opener, and the whole visible tail after the call was truncated. The pre-pass now skips matches that start inside a complete Gemma span (that text is the span's argument data) and resumes scanning at the end of the covering span, so a real function-XML call after the Gemma call is still stripped. The original ordering rationale is preserved: a Gemma opener inside a JSON or function argument still cannot truncate that block, covered by regression tests for both directions. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim comments in the Gemma streaming and strip pipeline to essentials * Tighten comments in the Gemma strip and streaming disconnect paths * Fold marker-collection comment to two lines --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 10:39:37 -07:00
Daniel Han	e9ea45b6a5	Studio: coerce tool_call arguments to dict before chat templating (fixes MLX tool follow-up error) (#6807) ... * Studio: coerce tool_call arguments to dict before chat templating Strict tool chat templates (e.g. mlx-community Qwen3.5 checkpoints) iterate arguments.items() and raise "TypeError: Can only get item pairs from a mapping" when a prior assistant tool call is re-rendered on the next turn. The agentic loop stores arguments in the OpenAI JSON-string form (as_assistant_tool_call), which is correct on the wire and for llama-server, but the transformers / MLX paths apply_chat_template directly and hit the strict Jinja templates. Normalize each assistant tool_call's function.arguments from a JSON string to a dict inside apply_chat_template_for_generation (shared by both the MLX and safetensors paths). A dict renders on strict and lenient templates alike; non-JSON / non-dict values are left untouched, and the OpenAI-format as_assistant_tool_call (used by the GGUF path + API responses) is unchanged. Verified against the real mlx-community/Qwen3.5-2B-8bit template: string args raised the tester's error, the fix renders cleanly, and the lenient unsloth/Qwen3.5-0.8B template still works. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make tool-arg coercion a string-first fallback (non-regressive) Render the original OpenAI string-arg form first and only coerce arguments to a dict when the template raises the mapping TypeError, instead of always coercing. Any template that already renders is now byte-identical (a template that emits arguments verbatim keeps the JSON string, not a Python dict repr). Verified across Llama-3, Qwen2.5, Qwen3, Qwen3.5, Phi-3.5 (byte-identical) and mlx-community/Qwen3.5-2B-8bit (strict -> fixed). Gemma-3 / Mistral tool-template errors are unrelated (role alternation / tool-id length) and identical with or without the change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make core.inference package init lazy so dependency-light helpers import standalone Importing any core.inference submodule ran the package __init__, which eagerly imported orchestrator and llama_cpp; both pull loggers -> structlog (and httpx), so a dependency-light helper like chat_template_helpers dragged in the full heavy stack and its unit test failed to collect in a backend env without structlog. Defer those imports to attribute access via PEP 562 __getattr__, mirroring the lazy pattern already in core/__init__.py. The re-exports resolve unchanged on first access. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Retry dict-coercion for strict templates that raise non-TypeError apply_chat_template_for_generation only retried the OpenAI JSON-string arguments coercion when the first render raised TypeError (the arguments.items() form). The bundled gemma-4.jinja instead rejects string arguments with raise_exception, which surfaces as a Jinja error, so a second tool turn with string function.arguments propagated and failed rather than retrying with the parsed dict. Broaden the outer catch to Exception, still gated on there being a string arg to normalize (normalized is messages -> re-raise), so unrelated template errors and templates that already render are unaffected. * Tighten comments in tool-call argument coercion helper and tests * Tighten tool-call argument coercion comments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 10:12:22 -07:00
Daniel Han	e9f49c62dd	studio: deterministic backend tool-calling wiring test (#6836) ... * studio: deterministic backend tool-calling wiring test Add a deterministic, download-free test that exercises the shared tool-calling seam both inference backends use. InferenceBackend (transformers) and MLXInferenceBackend both render the prompt through apply_chat_template_for_generation(..., tools=...) and stream cumulative text into run_safetensors_tool_loop. The existing test_safetensors_tool_loop.py covers the parser and the loop state machine with fake generators but does not cover the backend's own tool-injection seam, so a regression that drops the tool schema before the tokenizer, or fails to feed a tool result back into generation, would slip through. The test drives that seam with fakes: a tokenizer that records the tools it is handed, a canned tool-call generation, and a stub executor. It asserts the full chain: tools reach the chat template, the loop parses the call, the tool is dispatched once with the parsed arguments, the result is fed back, generation re-enters, and the final answer streams after the tool result. It also guards that the raw tool-call markup never leaks to the client as content. The test imports no torch, unsloth, or mlx, so it runs in the portable Backend CI alongside the tool-call parser tests and stays sub-second. Follow-up to the parser test PRs #5620 and #5704. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: assert the tool result is fed back before the final turn Strengthen the wiring test so single_turn records each turn's conversation and the test asserts the tool result message is present in the conversation handed to the final generation turn. Event ordering alone did not catch a loop that stops appending the tool output before re-entering generation, because the fake generation ignores the conversation; this closes that gap. * studio: tighten comments in tool-calling wiring test * studio: shorten comments in tool-calling wiring test --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 10:08:39 -07:00
Daniel Han	f38672da65	Studio: stop chat generation on the assistant-turn-end token (fixes Qwen3.5 loop) (#6804) ... * Studio: stop chat generation on the assistant-turn-end token A small chat model (e.g. Qwen3.5-0.8B) looped on the safetensors path: it emitted a valid response or tool call, then ran past its turn and re-emitted the call, hallucinating <|im_start|>user turns. Root cause: the model's tokenizer.eos_token is synced to the config document terminator (<|endoftext|>, 248044) while chat turns actually end with <|im_end|> (248046), so generate_stream's single eos_token_id never stopped at the turn boundary. Stop on every assistant-turn-end marker the vocab defines (tokenizer.eos plus <|im_end|>, <|eot_id|>, <end_of_turn>, ...). Verified on the real weights: the single-eos control loops (400 tokens) while the fixed set yields a clean 38-token tool call and a clean answer from the tool result. No-op when eos is already the turn-ender (the id just dedups). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: repair chat generation_config.eos_token_id at load time Qwen3.5 / Qwen3.6 small chat checkpoints declare the chat turn-end as tokenizer.eos_token (<|im_end|>) but ship config.eos_token_id = <|endoftext|> and no generation_config.json (upstream shipped generation_config only on the large chat models). So every .generate() path that reads generation_config -- the vision path and tool loops, not just generate_stream -- never stops at the turn boundary and loops. At load time, when the tokenizer's own eos is a chat turn-end marker but generation_config.eos_token_id omits it, add it. This fixes the config once for all generation paths and complements the generate_stream turn-end stop. No-op for base models (eos is a plain document terminator) and already-correct configs. Verified on unsloth/Qwen3.5-0.8B: 248044 -> [248044, 248046]. * Studio: derive chat turn-end eos from the template, resolve once at load Address PR review of the turn-end stop handling: - Do not call tokenizer.get_vocab() per generation request (serializes the whole 100k+ vocab). Resolve the turn-end tokens once at load and cache them on model_info; generate_stream reads the cache. - Derive turn-end markers from the chat_template the model actually uses, not raw vocab membership, so a base/coder model that merely carries ChatML control tokens in a shared vocab is not stopped early, and a loader that synced tokenizer.eos to the document terminator is still covered. - Skip harmony/gpt-oss templates: <|end|> there is an intra-message channel delimiter, not the turn end (dropped <|return|> from the marker list too). - Move the logic to a dependency-light module (core.inference.chat_eos) so the unit test does not import the full unsloth/torch inference stack. Verified on unsloth/Qwen3.5-0.8B (gen_config 248044 -> [248044, 248046], clean 38-token tool call with generation_config-only stopping), Phi-3.5 (adds <|end|>), Llama-3 / Qwen3 (unchanged), and a harmony template (left untouched). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: refresh turn-end eos after the mapper installs its template For a MODEL_TO_TEMPLATE_MAPPER model whose own tokenizer ships no chat_template, the effective template is applied at generate time via get_chat_template, but the turn-end eos ids were resolved once at load when the template was still empty, so only the document eos was cached. Qwen2.5 / Yi base checkpoints (eos <|endoftext|>, ChatML turns end with <|im_end|>) then run past the assistant boundary in generate_stream and loop. Re-resolve the turn-end eos from the now-templated tokenizer and refresh the cached ids right after applying the mapper template, so generate_stream stops at the ChatML turn end. Add a regression test. * Studio: union turn-end eos refresh into load-time cache instead of overwriting get_chat_template can return a different tokenizer whose vocab was remapped (Gemma folds <end_of_turn> onto the eos id), while generate_stream re-reads the original model_info tokenizer. Overwriting the cache with the refreshed set dropped a valid load-time id (e.g. <end_of_turn>=107) and let generation run past the real turn marker. Union the refresh into the existing cache so it can only add ids, never drop a valid one. Add a regression test covering the destructive-swap case the prior test missed. * Studio: resolve refreshed turn-end ids on the generation tokenizer, add Gemma-4 marker Two residual gaps in the turn-end eos refresh: - For map_eos_token=True mapped templates (e.g. chatml on a Yi-6B base), get_chat_template returns a tokenizer whose vocab folds the turn-end token onto the document eos id, while generate_stream re-reads the original tokenizer. The refresh resolved ids on the returned tokenizer, so it stored the doc eos and missed the real turn-end id, and generation ran past the boundary. Read the turn-end marker strings from the mapped template but resolve their ids on the original generation tokenizer (new resolve_chat_turn_end_eos_ids_using). - Add Gemma-4's <turn|> turn terminator to the marker allowlist; those templates keep a document eos so resolve otherwise missed the real turn marker. Add regression tests for both. * Fix turn-end detection for Starling, multi-variant and vision templates; keep tests collectable The turn-end marker set missed OpenChat/Starling's barred <|end_of_turn|> (distinct from Gemma's unbarred form), so Starling generations ran past the assistant boundary. A dict/list chat_template (Hermes-3 style default+tool_use variants) hit an early non-string return and skipped detection; flatten and scan every variant. Vision models carry the chat_template on the ProcessorMixin, not the unwrapped inner tokenizer, so read markers from the template-carrying container while resolving ids on the generation tokenizer. The refresh test constructs the real backend, so it is guarded with a module-level skip when unsloth/unsloth_zoo is absent (the lightweight pytest matrix), and core.inference package init is made lazy so the dependency-light chat_eos tests collect without the heavy stack. * Studio: tighten chat turn-end eos comments * Studio: condense chat turn-end eos comments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 10:07:56 -07:00
Daniel Han	f0a5c52821	studio: tool calling + healing parity for Llama-3, Mistral, Gemma 4 on safetensors + MLX (#5620) ... * studio: tool calling for Llama-3, Mistral, Gemma 4 on safetensors + MLX (#5615) Adds tool calling for Llama-3, Mistral (pre-v11 + v11+ + [ARGS]), and Gemma 4 to the safetensors / transformers and MLX backends. Parser patched against llama.cpp / vLLM / SGLang per-family parsers and normalises to OpenAI shape. 96 targeted unit tests + cross-OS staging CI (ubuntu / macos-14 / windows) green on the multi-format probe. * studio: tool-call healing parity between safetensors / MLX and GGUF After the multi-format parser landed in #5615, the safetensors / MLX agentic loop and the GGUF loop still differed on healing behaviour. This commit closes the gaps in both directions so the two backends react the same way to identical model output. Changes: 1. core/inference/llama_cpp.py -- the GGUF BUFFERING state machine now wakes on every emission marker the shared parser knows. Was ("<tool_call>", "<function="); is now the five-tuple imported from core.inference.tool_call_parser (Qwen / Qwen3.5 / Llama-3 <|python_tag|> / Mistral [TOOL_CALLS] / Gemma 4 <|tool_call>). Stream cleanup is delegated to the same shared strip_tool_markup so leaked markup from any family is removed from assistant content. 2. core/inference/llama_cpp.py -- per-tool canonical heal key. When a tool arguments field is a bare string and JSON parsing fails, the GGUF path now heals to {"code": raw_args} for python, {"command": raw_args} for terminal, and {"query": raw_args} for everything else. Was hard-coded to {"query": raw_args}, which silently routed every python / terminal emission through web_search. Mirrors safetensors_agentic._CANONICAL_HEAL_ARG. 3. core/inference/safetensors_agentic.py -- re-prompt on plan- without-action. When the model emits a short forward-looking intent ("I'll search for that", "Let me check", "First, I will...") and no tool call, the loop nudges the model to act instead of silently returning a plan-only answer. Up to _MAX_REPROMPTS=3 (matches GGUF). The intent regex, character cap, and instruction text are byte-identical to the GGUF path. The buffer-end fall-through is unified so a buffered intent emission that never exits the BUFFERING state still triggers the re-prompt. 4. core/inference/safetensors_agentic.py -- extra iteration slots for re-prompts. The loop now budgets max_tool_iterations + _MAX_REPROMPTS + 1 total iterations and tracks the tool-call count separately, so a stalling model can be nudged 3x without eating the caller's tool-call budget. Mirrors the _extra slot reservation in the GGUF path. Tests (14 new safetensors-side units; 5 GGUF parity pins): TestLoopRePrompt -- intent-trigger, plain-answer, no-tools, cap-at-three, budget preserved, buffer-end intent. TestLoopCanonicalHealKey -- python / terminal / unknown. TestGGUFSafetensorsHealingParity -- shared markers used, shared strip used, canonical heal keys identical, intent regex matches same phrases, _MAX_REPROMPTS equal on both backends. All 110 targeted tests pass locally; the broader tool / inference / model-config / sandbox / anthropic / mlx suites stay green. Why this matters Without this parity, Llama-3.2 / Mistral / Gemma 4 emissions on Mac (MLX) and Linux-safetensors stop the agentic loop as soon as the model says "Let me...", because the GGUF re-prompt logic never existed on these backends. The two-marker GGUF BUFFERING tuple also let non-Qwen tool emissions stream out as plain prose when llama-server's structured channel did not pick them up. Both paths now drain the same way, heal the same way, and re-prompt the same way -- so a tool call that works on GGUF works identically on safetensors / MLX. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: fix tool-call parser bugs from gemini review on #5620 Three high-priority gemini findings on the tool-call parsing additions: 1. unicode_escape on UTF-8 bytes corrupts non-ASCII literals (e.g. ✨ becomes â\x9c¨). Replace with json.loads on a quoted string -- preserves emoji / CJK / RTL while still handling \n \t \uXXXX escapes. 2. Llama-3 sentinel stripping is order-dependent. A leading `<|eot_id|><|begin_of_text|>` left `<|begin_of_text|>` behind because the loop had already passed that sentinel. Loop until no sentinel matches at the start. 3. Mistral v11+ `[TOOL_CALLS] name { json }` regex uses non-greedy `\{.*?\}` which truncates at the first `}` of a nested JSON argument, leaking the tail (e.g. `}}`) into user-visible streamed text. Same problem for the v0.3 array pattern with nested brackets. Strip those with balanced brace/bracket scanning via a new `_strip_mistral_closed_calls` helper called from `strip_tool_markup`. Also fix the inference routes' parallel `_TOOL_XML_RE`: - Same nested-JSON truncation in the Mistral patterns; route the strip through the parser's balanced-scan helper via a thin `_strip_tool_xml` wrapper that all existing callers now use. - Llama-3 `<|python_tag|>[^\n<]*` stopped at any `<`, leaking the tail of any tool call whose argument contained a literal `<` (queries, code snippets). Relax to `[^\n]*` which keeps the strip confined to the actual end-of-line. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio/routes: make python_tag strip multi-line aware Earlier revisions of _TOOL_XML_RE in studio.backend.routes.inference oscillated between two bug shapes: 5615 r"<\|python_tag\|>[^\n<]*" -- stopped at any literal "<" so code='if x < 10: pass' leaked '< 10: pass)' to the user. 5620.1 r"<\|python_tag\|>[^\n]*" -- single-line only; the second line of python.call(code="a\nb") leaked. The full parser (_parse_llama3_python_tag) already handles both via balanced-brace scanning, so the parsing path was fine; the LEAK was in the streaming strip path that runs on every cumulative emission while content is still arriving. Switch to r"<\|python_tag\|>(?:[^<]|<(?!\|))*" so the strip consumes: * any character that is not a "<" (newlines, JSON, code, ...), * a "<" only when it is NOT followed by "|" (i.e. NOT a Llama-3 sentinel start like <|eot_id|>, <|eom_id|>, <|begin_of_text|>). This means: * code='if x < 10' stays inside the strip (5615 fix preserved), * multi-line code stays inside the strip (5620 round 2), * the strip terminates at the next Llama-3 sentinel so trailing assistant content survives. Tests: TestRoutesPythonTagStrip (8 cases) pytest test_safetensors_tool_loop.py test_safetensors_capability_advertise.py -> 118 passed in 1.81s (was 110). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: tighten verbose comments in tool-call parser sections Comments were narrating what the code already says. Cut historical "earlier revisions used X, then Y" narratives down to one-line WHY notes where the footgun still matters (canonical heal-key parity, balanced-brace vs non-greedy regex, ``(?:[^<]|<(?!\|))*`` over ``[^\n<]*``/``[^\n]*``). Drop section-header banners. No behaviour change. Re-ran: pytest studio/backend/tests/test_safetensors_tool_loop.py \ studio/backend/tests/test_safetensors_capability_advertise.py -q -> 118 passed. Regression replay (parser + _coerce_arguments on the 5 #5615 inputs) -> 21/21. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: parser robustness fixes for PR #5620 Three surgical extensions to the multi-format tool-call parser, each covering a real fine-tune / template emission shape that the current parser silently drops. No path narrows; all changes widen what is accepted. 1. `_parse_tool_call_json` now accepts both `arguments` and `parameters` keys. A Hermes / Qwen `<tool_call>{json}</tool_call>` wrapper around a Llama-3.2 fine-tune that emits the `parameters` key was extracting the tool name and silently discarding the args, producing a working-shaped call with an empty payload. The bare-JSON and python_tag paths already accepted both keys; this path now matches them. 2. `_TC_FUNC_START_RE`, `_TC_PARAM_START_RE`, and `_TC_PARAM_CLOSE_RE` now also match the attribute form `<function name="..."><param name="...">v</param></function>` used by MiniCPM-5 and MiniMax-M2. Names land in either capture group, and `</param>` is accepted as a short close. 3. `_parse_llama3_bare_json` sentinel-strip now consumes the role label inserted between `<|start_header_id|>` and `<|end_header_id|>` by Meta's official Llama-3.x chat template. Without this, every assistant turn re-fed through the template prefix `<|start_header_id|>assistant<|end_header_id|>\n\n{json}` parsed to zero calls, so any history-with-tool-call round-trip in production silently dropped. Tests in `studio/backend/tests/test_safetensors_tool_loop.py`: * `TestParserRobustness::test_tool_call_json_accepts_parameters_key` * `TestParserRobustness::test_function_xml_attribute_form` * `TestParserRobustness::test_function_xml_attribute_form_multi_param` * `TestParserRobustness::test_function_xml_legacy_equals_form_still_works` (regression guard for the existing `<function=name>` syntax) * `TestParserRobustness::test_llama3_chat_template_round_trip` * `TestParserRobustness::test_llama3_round_trip_all_roles` * `TestParserRobustness::test_llama3_round_trip_with_eot_prefix` `pytest studio/backend/tests/test_safetensors_tool_loop.py studio/backend/tests/test_safetensors_capability_advertise.py -q` goes from 118 to 125 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: terminate function-XML body at </function>, not just </tool_call> `_parse_function_xml` was looking for `</tool_call>` (the Hermes wrapper) as the body terminator. When a model emits a standalone `<function=NAME><parameter=K>v</parameter></function>` followed by explanatory prose (which models routinely do), no `</tool_call>` is present, so the body extended to end-of-string and the trailing prose leaked into the LAST parameter value. Pre-existing on main (the legacy `<function=NAME>` form had this bug too). Same affects PR #5620's new attribute-form `<function name="NAME"><param name="K">v</param></function>` emission used by MiniCPM-5 / MiniMax-M2. Fix: `_TC_END_TAG_RE` now matches either `</tool_call>` OR `</function>`. The existing `_TC_FUNC_CLOSE_RE` / `_TC_PARAM_CLOSE_RE` strips are unchanged. Multi-call inputs still bound each function at the next `<function=` start, so no over-eager consumption. New tests: * `test_function_xml_followed_by_prose` (legacy form + prose) * `test_function_attribute_xml_followed_by_prose` (attribute form + prose) Existing `test_code_with_embedded_xml` still passes (a parameter value containing literal `<a></a>` is preserved because the embedded close tag is `</a>`, not `</function>`). `pytest studio/backend/tests/test_safetensors_tool_loop.py studio/backend/tests/test_safetensors_capability_advertise.py -q` goes from 125 to 127 passed. * Studio: tighten Llama-3.2 bare-JSON guard A fuzz pass on PR #5811 turned up that ``_parse_llama3_bare_json`` accepted ``parameters`` as a string, contradicting the docstring's "parameters or arguments is a dict" guard. Prose JSON like ``{"name":"foo","parameters":"a sentence"}`` would wrongly fire the parser, which the agentic loop would then heal into a real ``foo(query="a sentence")`` call. Same code lives on this branch, so the same fix applies here. Tightened guard: - ``parameters`` must be a dict (Llama-3 spec). - ``arguments`` may be a dict, or a JSON-encoded string that decodes to a dict (OpenAI shape, e.g. ``"arguments":"{\"q\":\"x\"}"``). Plain non-JSON strings or JSON-strings of lists / scalars / null no longer pass. Mirrors the fix landed in PR #5811 commit 615b8608. Adds the same 4 regression tests under TestParserMultiFormat. Existing test suite stays green: 127 -> 131 passing. * studio: fix safetensors tool-call parser gaps vs llama.cpp (Mistral CALL_ID / THINK, attribute-form signal) Three GGUF-parity fixes to the safetensors tool-call parser, each matching llama.cpp's reference behaviour: - Mistral Small 3.2 emits [TOOL_CALLS]name[CALL_ID]<id>[ARGS]{json}. The parser stopped after the name on seeing [CALL_ID] (neither [ARGS] nor {), dropping the call. Skip an optional [CALL_ID]<id> segment in both the parse and strip paths. llama.cpp parses this (test-chat.cpp:4785). - Magistral wraps reasoning in [THINK]...[/THINK]. A [TOOL_CALLS] inside the reasoning was parsed as a real call, producing a phantom call. Strip a leading [THINK] block before scanning so only the post-reasoning call counts (test-chat.cpp:2285); a literal [THINK] inside a later argument is left intact. - The standalone MiniCPM-5 / MiniMax-M2 <function name="..."> attribute form parsed correctly but was absent from TOOL_XML_SIGNALS and the markup strip patterns, so the streaming safety-net parse was gated off (dropping the call) and markup leaked into displayed text. Add the signal and broaden the strip regexes. Adds regression tests for all three. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: fire safetensors tool calls for the bare-JSON (Llama-3.2) form The agentic loop's streaming safety-net parse was gated on has_tool_signal(), which is False for the Llama-3.1 / 3.2 bare-JSON tool form {"name":..,"parameters":..} (no XML marker). Real tool calls were therefore dropped: the loop logged "model planned without calling tools", re-prompted three times, then gave up with zero tool calls, while GGUF's llama-server parses the same emission natively. Run parse_tool_calls_from_text() unconditionally in the safety net. The parser is strict (only fires on a valid tool-call shape) so plain answers are unaffected. Reproduced on a real unsloth/Llama-3.1-8B-Instruct run: the model emits {"name":"web_search","parameters":{...}} which now executes the tool instead of being re-prompted into a no-op. Adds a loop regression test for the bare-JSON form. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: complete strict-mode contract and fix parser import paths Address review findings on the multi-format tool-call parser: - Honor allow_incomplete=False in the remaining sub-parsers. The Llama-3 <|python_tag|>NAME.call(...) parser, the pre-v11 Mistral [TOOL_CALLS] array parser, and the Gemma 4 <|tool_call> parser ignored strict mode, so a truncated call (missing closing paren, ], or <tool_call|>) was still healed and executed with Auto-Heal disabled. Thread strictness through and reject the unclosed forms, matching the JSON and function-XML paths. - Drop the duplicate tool_call_parser import block in llama_cpp.py and the redundant un-aliased TOOL_XML_SIGNALS; only the _SHARED_TOOL_XML_SIGNALS alias is used as a value. - Import _strip_mistral_closed_calls from core.inference.tool_call_parser in routes/inference.py instead of studio.backend.core... The self-contained run.py launch mode only puts studio/backend on sys.path, so the absolute package path raised ModuleNotFoundError on the server-tool strip path. Add strict-mode regression tests for the truncated Llama-3 dot-call and the unclosed Mistral array. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: preserve XML param indentation and alias Mistral array parameters Two parser-correctness fixes found by auditing against the model chat templates and the SGLang / vLLM reference parsers: - Qwen3.5 XML parameter values lost their leading indentation. The chat template emits <parameter=k>\nVALUE\n</parameter>, but the parameter-start regex ate the wrapping newline AND the value's first-line indentation with a trailing \s*, then str.strip() removed the rest. Narrow the trailing class to horizontal whitespace only and trim exactly one wrapping newline (via _trim_param_value), preserving indentation in code/diff arguments. Matches SGLang's qwen3_coder detector. Applies to both _parse_function_xml (tool_call_parser.py) and the XML path in tool_healing.py. - Mistral pre-v11 array objects keyed on parameters dropped their payload. _consume_mistral_call read only the arguments key; alias parameters the same way the JSON/XML paths and SGLang's base detector do. Add regression tests for preserved multi-line indentation and the array parameters alias. * Studio: tighten tool-call parser comments Make the comments in the multi-format tool-call parser and its callers succinct: compress verbose docstrings/blocks to one or two lines, drop ones that restate the code, and trim the tiny balanced-scanner helpers. Correctness rationale and upstream provenance (SGLang/llama.cpp parity, the strict-mode / Auto-Heal contract, whitespace-preservation, and the Unicode / full-width-pipe notes) are kept in compact form. Comment-only: no code or behavior change (verified with comment_tools.py check --strip-docstrings; parser suite green). * Studio: make Llama-3 .call and Mistral-array healing parsing linear Two more O(n^2) ReDoS paths in the multi-format parser, both reachable from the agentic loop on a long truncated body with no length cap: - _LLAMA3_KV_RE.finditer over a .call(...) body retried at every offset of a long word run / unterminated quote (40K -> 14s). Replace with a hand-scan that reuses the same key/number/literal sub-regexes via anchored match and walks the string body by hand, so an unterminated quote is O(n). Verified byte-identical to the old regex over 200K fuzzed inputs. - _parse_mistral_array healing ran _balanced_brace_end from every { in the body (20K -> 17s). Walk top-level objects, advancing past each balanced {...}; this also drops the phantom call the old scan emitted from a nested argument object. Add adversarial-length linearity regressions plus positive .call kwargs and unclosed-array recovery coverage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: honor strict mode in safety-net, keep empty Gemma args, strip attribute-form function XML - safetensors safety-net parser now forwards allow_incomplete=auto_heal_tool_calls, matching the draining path, so a late incomplete tool call is not healed and executed when Auto-Heal is off. - Gemma empty bare value ({k:}) now serialises as "" instead of invalid {"k":}, which previously dropped the whole call. - Route _TOOL_XML_RE also strips the <function name="..."> attribute form (MiniCPM-5 / MiniMax-M2) so it no longer leaks to the UI. * Studio: fix attribute-form function-XML literal close tag and zero-arg strict call Addresses Codex review of the <function name="..."> attribute form in _parse_function_xml (MiniCPM-5 / MiniMax-M2): - End the call body at the LAST </function> / </tool_call> within the call's window, so a literal close tag inside a code/search argument (e.g. print("</function>")) is preserved instead of truncating the call. - Accept a closed call with no parameters as a valid zero-argument call in strict mode (the function close is already required), instead of rejecting it as a truncated call. - Tests for both, mirroring the legacy <function=...> coverage. * Studio: fix tool-call parser/loop review findings on the multi-format path Address the live code-review findings on the safetensors/MLX + GGUF tool path: - routes: include the attribute form <function name="..."> in the safetensors capability whitelist so MiniCPM-5 / MiniMax-M2 templates keep the tool pill (parser already handles the form; the post-filter wrongly suppressed it). - safetensors loop: build the plan-without-action re-prompt from the active tools instead of a hardcoded web_search/python string, and gate it on auto_heal_tool_calls, matching the GGUF loop. - safetensors loop: hold a leading bare-JSON object ({"name":..,"parameters":..}) during BUFFERING until it closes, then drain it as a tool call instead of streaming the raw JSON to clients. The DRAINING/STREAMING resolvers still recover a plain JSON answer, so this can never drop content. - parser: anchor the Llama-3 <|python_tag|>NAME.call(...) scan to the tag and chain ; -separated calls, so all semicolon-separated built-ins parse and a literal <|python_tag|>x.call(...) inside a JSON string argument no longer fires the wrong tool. - parser: consume the optional trailing </s> after a named Mistral [TOOL_CALLS]name{json} call, mirroring the array shape. - GGUF streaming strip: use the shared parser patterns (which know [TOOL_CALLS] and <|python_tag|>) so a textual tool call entering DRAINING is stripped instead of leaking the marker to streaming clients. - routes: hoist the _strip_mistral_closed_calls import to module level. Adds regression tests covering each fix; existing parser suite stays green. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden multi-format tool-call detection from review findings Apply five targeted fixes from the review pass over the multi-format tool path: - routes: route display strip delegates to _strip_tool_xml so Mistral [TOOL_CALLS] blocks with nested JSON are removed from streamed display text, not just the XML forms. - tool_call_parser: skip function/parameter starts that fall inside an already-open parameter block (_inside_open_parameter) so nested example payloads are not mis-parsed as new calls; extract strip_llama3_leading_sentinels so the bare-JSON guard is shared. - safetensors_agentic: probe bare JSON through strip_llama3_leading_sentinels before the balanced-brace check so a leaked header sentinel does not defeat the guard. - tool_healing: allow dotted tool names in the Gemma wrapped start pattern. - llama_cpp (GGUF): buffer wrapper-less Llama-3.2 {"name":..} calls that carry no XML signal, drain a complete object silently and hold an incomplete one, and run the end-of-stream safety net unconditionally so markerless calls are detected and never leak the raw JSON (including truncated fragments). Adds regression tests for the GGUF bare-JSON streaming path and the Mistral display strip. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop bare-JSON tool calls leaking at EOF, oversized, and into history The second review pass flagged that the Llama-3.2 bare-JSON tool-call handling still leaked raw JSON in several spots; ``strip_tool_markup`` only knows XML/bracket markup, so the bare-JSON form survived it. Fix them symmetrically across the safetensors and GGUF loops: - Safetensors stream-end resolver now routes a held bare-JSON fragment to DRAINING (mirroring GGUF) so a truncated ``{"name":..`` cut off by the end of the stream is dropped instead of flushed as assistant content. The 7/10 reviewer finding. - Both loops now drain (suppress) an oversized still-open bare-JSON call once it passes ``_MAX_BARE_JSON_BUFFER`` instead of streaming the raw prefix, gated on a ``"name"`` key so a giant plain JSON answer still streams; a complete oversized call still executes via the safety net. - Add a shared ``strip_leading_bare_json_call`` helper and apply it to the content kept for the assistant turn in both loops, so an executed bare-JSON call is not replayed as visible text or fed back as next-turn history. Plain JSON answers without a ``"name"`` key are untouched throughout. Adds regression tests for the EOF, oversized, and next-turn cases on both backends plus unit tests for the helper. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: bound the Llama-3 python_tag strip on real control sentinels The route display strip's <|python_tag|> arm ran to the next <| of any kind. A tool-call argument carrying a literal <|...|> token (for example <|cite|> inside a string value) truncated the strip early and leaked the call tail into the visible response. Narrow the stop condition to the genuine Llama control sentinels (eot_id, eom_id, python_tag, start/end_header_id, begin_of_text, finetune_right_pad_id) so embedded markup and JSON are consumed while real header/turn boundaries still bound the strip. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate markerless bare JSON on enabled tools and close parser/strip asymmetries The Llama-3.2 custom_tools bare-JSON form has no marker, so any JSON object with a name key was read as a tool call. An ordinary JSON answer like {"name":"Alice","parameters":{"age":30}} was misclassified as a call to a disabled tool and dropped from the visible response. Gate the markerless form on the enabled tool names (threaded through parse_tool_calls_from_text and strip_leading_bare_json_call, supplied by both streaming loops): an object whose name is not an enabled tool is ordinary content. The marker-based forms keep their name-agnostic behaviour (an explicit signal is a real call attempt), and unrestricted mode stays ungated. Also fix two parser/strip asymmetries the parser already tolerated: - A literal </function> inside a parameter value (print("</function>")) truncated both the core and route strips at the first close, leaking the tail. Extend the strip to the call's real close (last </function> before the next opener), mirroring the parser, without merging separate calls. - The single-object Mistral [TOOL_CALLS]{...} shape parsed but _strip_mistral_closed_calls left it, leaking the raw object into display. Strip the balanced object while keeping trailing prose, matching the array and name shapes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio tools: gate GGUF bare-JSON suppression on enabled tools and fix python-tag exponent parsing Pass-4 review follow-ups on the GGUF tool loop and Llama-3 parser: - The GGUF bare-JSON suppression sites still keyed off a raw "name" substring, so an ordinary JSON answer whose name is not an enabled tool was dropped when it was truncated, oversized, or reached the no-tool DRAINING fallback (the parser, helper, and safetensors paths were already gated). All three sites now use the shared enabled-name gate, and a held bare-JSON buffer that turns out not to be an enabled call is shown as the answer instead of dropped at stream end. - The Llama-3 python-tag numeric kwarg regex matched only the mantissa, so scientific notation was truncated to its leading digits (1e-3 parsed as 1) and a tool executed with the wrong value. The regex now accepts exponent and decimal forms, and the int/float classification keys off the exponent too. Adds regression tests for the truncated / oversized disabled-name JSON cases (and a counterpart that a truncated enabled call still does not leak) plus the scientific-notation kwargs. * Studio tools: gate safetensors bare-JSON drain, fix nested-name gate and function-XML strip Pass-4 review follow-ups on the shared parser / safetensors loop: - The safetensors oversized and end-of-stream bare-JSON drain branches keyed off a raw "name" substring, so a large or truncated ordinary JSON answer whose name is not an enabled tool was drained instead of streamed. Both now use the shared enabled-tool-name gate, matching the GGUF path. - strip_leading_bare_json_call matched the first "name" anywhere, so a plain JSON answer with a nested name equal to an enabled tool ({"result":{"name":"web_search"}}) was wrongly suppressed. It now extracts the TOP-LEVEL name only, walking past nested objects/arrays and keeping the text when a top-level value is truncated. - The function-XML display strip used a regex negative-lookahead that stopped at a literal <function=...> opener inside a parameter value and then dropped the rest of the answer to EOF. A scan-based strip mirrors the parser (ignores openers inside an open <parameter> via _inside_open_parameter) and closes each call at its real </function>, so trailing assistant text after such a call survives. Adds regression tests for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tool parsing: 3.9 import safety, disabled-Auto-Heal contract, capability gate Round-2 review follow-ups on the multi-format tool-call parser: - tool_call_parser: add `from __future__ import annotations`. The module is dependency-light by design (external llama-server wrappers import it standalone) and the package targets python >=3.9, where its PEP 604 `int | None` return annotations would raise TypeError on import. - safetensors + GGUF drain fallback: gate the leading bare-JSON strip on auto_heal_tool_calls. With Auto-Heal off, a truncated enabled-name fragment that did not parse now stays visible, matching the XML strip in the same branch and the disabled-Auto-Heal contract. With Auto-Heal on it is still suppressed. - safetensors capability gate: match the bare-JSON `{"name":` template marker with a whitespace/escape-tolerant regex so a pretty-printed `{ "name" :` or JSON-escaped `{\"name\":` template is not mis-classified as tool-less. The parser already accepts that whitespace via raw_decode, so the gate must too. Regression tests added for each case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tool parsing: symmetric "function" bare-JSON alias and route strip parity Round-3 review follow-ups, all parser/strip symmetry fixes. - Bare-JSON "function" alias: the markerless parser accepts a call name via obj.get("name") or obj.get("function"), but the strip/gates only knew "name", so a {"function":<enabled tool>} call executed while its raw JSON leaked. Teach _top_level_bare_json_name the alias (with "name" precedence and the same nested and truncated-name guards), and widen the guards in strip_leading_bare_json_call, the safetensors and GGUF _looks_like_enabled_bare_json gates, and the route capability marker regex. - Route display/history cleanup: strip a tail-only </param> alias close (the parser accepts <param name="...">...</param>), and run the parser's guarded function-XML scan (_inside_open_parameter) before _TOOL_XML_RE so a literal nested <function=...></function> inside an argument value does not truncate the strip and leak the tail. Regression tests added for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio tools: honor tool budget in GGUF loop and guard function-XML streaming strip Round 4 review fixes. Both are asymmetric-fix bugs where the final/steady path got a guard the analogous streaming/loop path did not. - GGUF tool-call budget: the safetensors loop counts real tool-call turns against max_tool_iterations (re-prompt stalls excepted), but the GGUF loop only bounded the turn count by the enlarged range (max_tool_iterations + _MAX_REPROMPTS). Since this PR raised _MAX_REPROMPTS from 1 to 3, a model that keeps making valid tool calls could run up to three extra tool rounds (with max_tool_iterations=1, four rounds instead of one). Add a _tool_iters_done counter that increments only when a tool actually executed in the turn, and stop once the caller's budget is spent so the post-loop final-answer nudge fires. A duplicate/disabled no-op turn is a correction turn (like a plan-without-action re-prompt) and does not consume budget, preserving the existing "already completed" re-prompt behavior. - Streaming display strip: the final strip runs the guarded _strip_function_xml_calls scanner (a literal <function=...> inside a parameter value is data, not a nested call), but the GGUF and safetensors streaming strips still used only the open-ended regex arms. When a tool-call argument contained literal function markup, the regex tail ate everything to end-of-text and dropped the real trailing prose after the call's true </function>. Run the guarded scanner (and the balanced Mistral strip) before the regex arms in both streaming paths so streaming and final display agree. Adds regression tests: GGUF valid tool calls respect max_tool_iterations, and the streaming strip keeps trailing prose after a function-XML call with a literal marker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio tools: safetensors tool budget counts only executed turns (GGUF parity) Follow-up to the GGUF budget fix. The safetensors loop charged max_tool_iterations per non-re-prompt iteration (iteration + 1 - reprompt_count), so a duplicate/disabled no-op turn spent a budget slot even though no tool ran. With a small cap this dropped real work: for max_tool_iterations=2, a model that made a valid call, repeated it (an internal no-op correction turn), then made a distinct valid call executed only the first -- the third turn was sent with no tools and the distinct call was ignored. Track whether a turn actually executed a tool (set on record_result) and count only those turns against the cap, matching the GGUF loop. A duplicate/disabled no-op is a correction turn -- like a plan-without-action re-prompt -- and no longer consumes budget, so the model still gets its "already completed" nudge and another tool-enabled turn. Adds a regression test for the small-cap duplicate-then-distinct-call flow. * Studio: render the reasoning block for safetensors and MLX like GGUF enable_thinking chat templates (Qwen3/Qwen3.5/GLM) prefill an unclosed <think> into the generation prompt, so the model emits only the closing </think> then the answer. The safetensors/MLX chat stream emitted that as plain content, so the reasoning showed inline with no collapsible thinking block, while GGUF (which surfaces reasoning via reasoning_content) rendered one. This brings safetensors and MLX to parity. - _ResponsesReasoningExtractor gains a reasoning_prefilled mode that starts inside the reasoning block and splits on the first </think>; default False keeps GGUF and every existing caller byte-identical. It suppresses a stray re-emitted <think> and holds partial markers back across chunk boundaries. - _sf_reasoning_prefill_mode gates the mode on reasoning being enabled for the request, an enable_thinking or enable_thinking_effort style, and the template actually using the standard <think>/</think> markers. Models with a bespoke reasoning channel (e.g. gemma's <|think|>/<|channel>) are excluded so their answer is never swallowed; gpt-oss (Harmony) and thinking-off requests are excluded too. - sf_tool_stream and stream_chunks (the latter also serves MLX) feed text through the extractor, emitting reasoning_content then content deltas, with a per-turn reset in the tool loop and a flush before each tool_start; only the visible delta reaches the monitor reply. The two non-streaming drains split reasoning_content the same way. - Tests: extractor prefilled mode (streaming and edge cases), the gate matrix including the gemma-style exclusion, and a route-replay of the tool-loop reasoning stream. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: don't force a tool re-prompt on a negated intent (safetensors parity) The safetensors _INTENT_SIGNAL claimed to mirror GGUF but was missing the negative lookahead, so a refusal like "I will not search the web for that" matched the "i will" intent and triggered the plan-without-action re-prompt (STOP... you MUST call a tool), overriding a valid no-tool answer. GGUF already excludes not/never. Add the same (?!\s+(?:not|never)\b) lookahead so both backends agree. Extends the intent parity test with negated refusals. * Studio: trim redundant comments (comment-only, AST-verified) * Studio: prevent Gemma tool-parser DoS on stray delimiters _gemma_parse_value returned the input index unchanged when text[i] was a stray delimiter (,}]), so the list and mapping caller loops that advance on the returned index spun forever at 100% CPU on malformed input such as [},]. Advance past the delimiter so parsing always terminates. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: strip Magistral [THINK] reasoning from final display/history strip_tool_markup removed [TOOL_CALLS] and <function> markup but left a leading Magistral [THINK]...[/THINK] block intact, so its bracket-form reasoning (not the <think> the reasoning channel renders) leaked into the safetensors display and conversation history while GGUF/llama.cpp routes it natively. Drop the leading reasoning block at end-of-turn (final=True) via the existing _strip_mistral_reasoning helper; streaming is untouched. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Honor reasoning_effort none in safetensors prefill; strip Magistral reasoning while streaming Two safetensors/MLX reasoning fixes surfaced in review: _sf_reasoning_prefill_mode only checked enable_thinking, so an enable_thinking_effort (GLM-5.2) request that disables thinking via reasoning_effort=none (without enable_thinking=False) still began in prefilled-<think> mode. A plain answer with no </think> was then swallowed whole into reasoning_content and the visible response came back empty. Thread reasoning_effort into the predicate and treat none as disabled, mirroring _request_reasoning_kwargs. strip_tool_markup_streaming stripped tool markup but not the leading Magistral [THINK]...[/THINK] bracket block, so the raw chain-of-thought leaked into the streamed safetensors content instead of the reasoning drawer (GGUF routes it natively). Apply _strip_mistral_reasoning first, matching the final strip; an unclosed [THINK] is held from the marker on so nothing flickers. * Mistral outer call wins over XML literals; align healer signals with its parser Two follow-ups on the shared-parser ordering after the healing-passthrough merge: - A well-formed [TOOL_CALLS] call whose JSON arguments quote tool XML parsed the literal instead of the outer call (executing the wrong tool). When the first XML signal sits inside a leading balanced Mistral body it is argument data, so the Mistral parser now runs first; an XML signal before the trigger keeps the normal order, so a [TOOL_CALLS] literal inside an XML call's arguments still stays data. - passthrough_healing buffered streams on the parser module's broadened signal list (now including <|python_tag|> and [TOOL_CALLS]) but promotes with core.tool_healing, which does not parse those forms: a streamed Mistral or Llama text call was held until finalization and flushed as prose. The healer keeps its own signal list limited to the formats it can promote, restoring immediate streaming for the rest. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: leading envelopes win over rehearsed literals - New _first_foreign_tool_signal shared by the leading-envelope guards adds <|python_tag|> to the protected signal set: the spelled-out literal inside a Mistral call's arguments (a query about Llama built-in tool syntax) executed the inner literal instead of the outer call. - New _xml_signal_inside_leading_bare_json guard, sibling of the Mistral one: a leading bare-JSON call whose string argument quotes tool XML (a code value citing <function=...>) had the literal promoted by the shared XML pass before the bare-JSON parser ran. - Magistral [THINK]...[/THINK] is dropped once at parse entry instead of only inside the Mistral parser, so a call rehearsed in the think block in a foreign format can no longer be promoted while the real call after the block is lost. Parse now agrees with the display strip. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: a disabled leading bare-JSON object keeps its literals as data When the leading bare-JSON object is ordinary content (name not an enabled tool), the guard proved the first tool signal sits inside it, so falling through to the XML/python_tag passes promoted quoted string data as a real call. Drop the object and parse only the tail: a real call after the object still parses, nothing inside it can be promoted. * Address review: Mistral literals inside leading JSON, whitespace-tolerant wrapped Gemma opener - The leading bare-JSON guard now treats the [TOOL_CALLS] trigger as a foreign signal: the Mistral parser runs before the bare-JSON one, so a literal quoted inside the leading object's strings was promoted over the outer call (or over ordinary JSON content). - tool_healing's wrapped Gemma opener tolerates whitespace around call and the colon: sampling drift emits call: name{ and call : name{, and rejecting those lost the call entirely because no fallback re-parses the wrapped form. Strict mode still requires the closing tag. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: accept dotted Gemma argument keys in the key-quoting scanner The scanner quoted keys of [alnum_-] only, so a dotted key (user.name:...) was left unquoted, json.loads failed, and the whole wrapped call was lost (parse empty, strip wipes the markup). Dots now match the parser's own key/name charset. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: leading Mistral call owns the turn, dotted keys after bare values - A LEADING parseable [TOOL_CALLS] call now runs the Mistral parser first unconditionally: literal XML in trailing prose after the call was promoted by the earlier shared XML pass, executing the quoted example instead of the real leading call. XML leading keeps the normal order. - _GEMMA_NEXT_KEY_RE accepts dots so a dotted key after a bare value (query:foo,user.name:bob) ends the value at the comma instead of being swallowed into it, matching the round-earlier key-quoting charset. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address review: markup quoted inside a nameless leading JSON answer stays data The leading bare-JSON guard required a top-level name, so a structured JSON answer quoting tool markup in its strings (a response_format turn documenting a tool's syntax) had the literal promoted by the later passes. A nameless leading object that parses as real JSON now routes through the same decline-then-parse-the-tail path; non-JSON braced prose keeps the old behaviour, and a real call after the answer still parses. * Compress docstrings in the multi-format tool parser to their contract essence * verify_import_hoist: exempt __future__ imports and same-diff relocations Two false positives fired on this PR's refactor. A from __future__ import is a compiler directive whose name never appears as a runtime load, so HOISTED-IMPORT-UNUSED can never see it used, yet the file requires it for PEP 604 annotations on Python 3.9. TARGET-CHANGED flagged the deliberate move of the strip-pattern constants into core.inference.tool_call_parser as a silent re-point even though the old module-level target was removed and the new one added in the same diff. Both get narrow exemptions; a re-point to a pre-existing target is still caught, and the self-test negative controls all pass unchanged. * Leading bare-JSON calls own the turn; function calls end at the first balanced close The XML-signal guard for a leading bare-JSON call required the signal strictly inside the object, so a trailing XML example stole the turn from the leading call; it now applies the same inside-or-after rule as the Mistral guard. Function-XML calls also ended at the LAST close tag, which let prose after a closed call that mentions a literal close tag get swallowed into the final parameter value; calls now end at the first close tag that is not inside an open parameter, and the strip mirrors the same rule so parse and strip agree. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Attribute-form calls end at the first balanced close; bare-JSON strip requires the call shape The attribute form parser still kept the last close tag in the call window, folding prose after a closed call into the final parameter value. It now takes the first close not inside an open parameter, the same rule the equals form and the strip already use. The leading bare-JSON strip deleted any closed object whose top-level name matched an enabled tool, including plain JSON answers the parser correctly rejects as non-calls. The strip (and the drain gate that delegates to it) now requires the parser's exact call shape, so answers like {"name":"web_search","result":...} stream and display intact. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * False-alarm markers keep the answer; the bare-JSON strip consumes the whole chain The trailing strip arms dropped everything from a bare marker to EOF, so a normal answer that mentions [TOOL_CALLS] or another marker literally was truncated (or fully swallowed when it started with the literal) after the no-call drain fallback. Those arms now require a call-shaped lookahead or marker-at-EOF before dropping; truncated real calls still strip. Chained bare-JSON turns executed both calls but stripped only the first object, so the second call's raw JSON replayed into the next assistant history message alongside the structured tool_calls. The strip now consumes the entire chained run of call-shaped enabled objects while non-call answers, disabled names, and trailing prose stay intact. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Attribute-form containment, parameter-close-decides rule, preamble-tolerant Mistral guard, strict strip shape Four document-order and containment fixes. A leading attribute-form call now parses before the shared XML pass, so markup quoted in its parameter stays data. The open-parameter scan lets the parameter's own close tag decide, so any number of literal function closes inside one value stay data, restoring the pre-close-scan behavior for multi-close arguments. The leading-Mistral guard tolerates a visible preamble, with the leading-bare-JSON guard running first so a trigger quoted inside a leading JSON object stays data. The bare-JSON strip requires the parser's top-level name in every mode, so nested-name JSON answers survive name-agnostic stripping. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Let a leading <|python_tag|> call own the turn over quoted XML literals The leading-call ownership contract (a leading executable call owns the turn; foreign markup quoted in its string arguments or trailing prose stays data) was enforced for the bare-JSON, Mistral and attribute-form leading calls but not for the Llama-3 <|python_tag|> form. The shared tool_healing XML pass runs before _parse_llama3_python_tag and does not recognise <|python_tag|>, so a <function=...> / <tool_call> / [TOOL_CALLS] literal quoted inside a <|python_tag|> .call(...) string argument (or its JSON parameters) was promoted and the wrong tool executed. Well-formed single-format examples: <|python_tag|>web_search.call(query="... <function=foo> ...") -> foo <|python_tag|>python.call(code="<function=render_html>..</function>") -> render_html both returned the phantom inner tool instead of the real leading call. Add a leading-<|python_tag|> guard mirroring the other leading-call guards: when the tag is the first tool signal, parse it before tool_healing so quoted foreign markup stays data. A foreign signal before the tag keeps normal document order. Added TestPythonTagOuterOverXmlLiteral (7 cases). * studio: tighten tool-calling comments to be shorter and clearer * studio: shorten tool-format comments in changed files --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <michaelhan2050@gmail.com> Co-authored-by: Daniel Han <info@unsloth.ai> Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>	2026-07-06 10:06:06 -07:00
Daniel Han	cb9d902830	Add the second blank line before _fix_rope_inv_freq (#6910) ... ruff-format requires two blank lines before a top-level function. loader.py carried only one, so the ruff-format-with-kwargs pre-commit hook reformats it and the run fails. This restores the expected spacing.	2026-07-06 09:13:29 -07:00
Daniel Han	2fada48ef5	Fix llama3 RoPE scaling dropped on transformers v5 (#6907) ... * Fix llama3 RoPE scaling dropped on transformers v5 transformers v5 loads on meta then blanks non-persistent buffers, so _fix_rope_inv_freq rebuilds inv_freq after load. It recomputed a vanilla inv_freq and applied _apply_inv_freq_scaling, a no-op on the base LlamaRotaryEmbedding used by the config/llama3 path, so inv_freq ended up divided by 1 instead of the config factor (8 for Llama 3.1, 32 for Llama 3.2). This corrupts long-range positions and inflates long-context loss about 3-5x. transformers 4.x was unaffected. Route __init__ and the v5 repair through one _unsloth_recompute_inv_freq so they cannot diverge, and stash the config on the rotary module so the repair can rebuild the same scaled value. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add test for llama3 RoPE scaling under the transformers v5 repair * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update RoPE drift guard for the recompute refactor and guard the v5 repair The drift guard's AST tripwire asserted the config-scaling call lived in the if config is not None branch of LlamaRotaryEmbedding.__init__. The fix moved that into _unsloth_recompute_inv_freq, so follow it there (with a fallback to the old inline branch) and add a guard that loader._fix_rope_inv_freq rebuilds inv_freq through the same helper. Also add a CPU functional check of the helper and drop the redundant standalone test. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 09:13:14 -07:00
Daniel Han	46e2cf5dee	studio: label RAM and VRAM readouts as GiB not GB (#6895) ... The live resource monitor and GPU readouts derive memory from binary byte counts (bytes / 1024**3 for torch and psutil, MiB / 1024 for the nvidia-smi path), which is GiB, but the UI labeled the values "GB". On a B200 this showed "178.35 GB" for a card whose nvidia-smi total is 183359 MiB (179 GiB), so it looked like memory was missing. Relabel the measured RAM and VRAM readouts to GiB across the floating monitor, the resources tab, the studio live GPU panel, the hub header, the about tab and the onboarding summary. The numeric values are unchanged, so the training GPU selection and memory-fit logic that read the same fields are unaffected. Disk stays labeled GB because the backend reports it in decimal GB (bytes / 1e9), and model file sizes and download progress keep their decimal GB labels to match Hugging Face.	2026-07-06 08:27:26 -07:00
Anas Khan	cc99aab607	fix: correct class name in SyntheticDataKit.chunk_data guard message (#6901)	2026-07-06 07:11:57 -07:00
Anas Khan	c44d94f1ae	fix: map None quant method to q8_0 before lowercasing in GGUF export (#6889)	2026-07-06 07:11:49 -07:00
Anas Khan	f4d1dc541f	fix(fp8): use int64 offsets in weight_dequant_kernel (#6884)	2026-07-06 07:11:41 -07:00
Anas Khan	487b420948	CI: pin lockfile-audit actions to commit SHAs (#6902)	2026-07-06 07:11:33 -07:00
Daniel Han	cf4906dbe6	Note bundled flash-linear-attention kernels for gated-deltanet models (#6850) ... * Note the bundled flash-linear-attention kernels for gated-deltanet models Unsloth Zoo now bundles the flash-linear-attention (fla) gated-delta Triton kernels and injects them automatically, so gated-deltanet models (Qwen3-Next, Qwen3.5, Kimi-Linear) get the fast path with no pip install. Replace the old install advisory with a one-time note that fires only when the bundled kernels could not be enabled on the current setup (no CUDA, or torch < 2.7 / triton < 3.3), i.e. exactly when transformers falls back to the slow pure PyTorch path. * Tighten comments * Normalize model_types in fla install advisory for None and single string * Cover olmo_hybrid in the gated-deltanet fla advisory	2026-07-06 05:50:15 -07:00
Daniel Han	efcaffb17b	Sync FORCE_FLOAT32 fallback with unsloth-zoo (gemma4, glm4_moe, qwen3_moe) (#6865) ... * Add gemma4, glm4_moe and qwen3_moe to the FORCE_FLOAT32 fallback list Keeps the fallback list (used only if the unsloth_zoo import fails) in sync with unsloth_zoo/model_lists.py, which now force-float32s these MoE archs so a float16 request loads bf16 and trains finite instead of NaNing the grad_norm. * Union FORCE_FLOAT32 fallback so new archs force float32 with older unsloth_zoo	2026-07-06 05:48:10 -07:00
Daniel Han	95a73f0f51	Honor explicit load_in_16bit for local -bf16 directories (#6726) ... A model path ending in -bf16 unconditionally forced 16-bit loading, so a LOCAL checkpoint directory whose name happens to end in -bf16 could never be loaded in 4-bit, 8-bit or fp8: the suffix rule silently overrode the caller's quantization flags. Hub repo ids keep the existing behavior (the suffix is a publishing convention there), but for a local directory (expanduser-aware, so tilde paths are detected too) the requested quantization is preserved unless the caller explicitly passes load_in_16bit=True.	2026-07-06 05:47:26 -07:00
Daniel Han	c7b8666ce4	Auto-enable grouped MoE on loaded / PEFT'd models via loader hook (#6727) ... * Auto-enable grouped MoE on loaded / PEFT'd models via loader hook Wraps the FastLlamaModel and FastBaseModel from_pretrained / get_peft_model leaves with wrap_loader_for_grouped_moe so the grouped-GEMM MoE forward is installed on the live instance after the model and its compiled module are built. Gated by UNSLOTH_MOE_GROUPED and wrapped in try/except, so it is a no-op when the unsloth_zoo module is absent or no eligible MoE block exists. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Install grouped-MoE loader wrappers before PatchFastRL * Re-evaluate grouped MoE after loading a PEFT adapter When loading an existing adapter through FastLanguageModel.from_pretrained, the base model is evaluated for grouped MoE when the wrapped from_pretrained leaf returns, but the adapter is attached afterwards via PeftModel and patch_peft_model. Re-run auto_enable_grouped_moe on the final model so blocks whose experts gained LoRA are restored to the original loop, attention-only adapters keep the grouped path on their frozen experts, and recompute is re-derived from the final gradient-checkpointing state. Guarded so it never blocks adapter loading. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim comments in the grouped MoE loader hooks Shorten the loader re-eval and llama.py wrapper comments; code is unchanged (verified comment-only). * Re-evaluate grouped MoE after loading a PEFT adapter on the vision path --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 05:46:30 -07:00
Daniel Han	c520662c12	Honor an explicit sdpa or flex_attention request when flash is disabled (#6847) ... * Honor an explicit sdpa or flex_attention request when flash is disabled When flash attention is disabled for a model, the fallback selection could downgrade a caller who explicitly passed attn_implementation='sdpa' or 'flex_attention' to a different backend, because the disable reason is flash-specific. Keep an explicit non-flash request as-is; flash requests still fall back as before. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten comments * Gate honor-explicit attention on provenance and flex support Only honor an explicit non-flash attention request when it comes from the caller argument, not from a config value the loaders synthesize (the language path seeds attn_implementation=sdpa). Honor explicit flex_attention only when supports_flex_attention is True so excluded/broken configs (e.g. gpt_oss) fall back instead of selecting a known-broken backend. Explicit sdpa stays honored. * Honor explicit sdpa through the resolver guard * Keep SDPA exclusions when honoring an explicit sdpa request An explicit attn_implementation="sdpa" was re-enabling sdpa for models in _SDPA_EXCLUDED_MODELS (e.g. gpt_oss) where sdpa is known-broken: the helper honored the request and the resolver's final not-supports_sdpa guard skipped the eager downgrade for any explicit request. Honor an explicit sdpa only when the model is not sdpa-excluded, mirroring the flex guard that already falls back for _FLEX_EXCLUDED_MODELS via supports_flex_attention. Conservative supports_sdpa=False (large head dim / attention-sink models) still honors an explicit sdpa; a synthesized/default sdpa still downgrades to eager. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Honor DISABLE_SDPA_MODEL_NAMES when honoring explicit sdpa The honor-explicit-sdpa guard only skipped the sdpa->eager downgrade for models in _SDPA_EXCLUDED_MODELS (gpt_oss). Gemma3/Gemma3Text disable SDPA through the loader's DISABLE_SDPA_MODEL_NAMES (their bundled SDPA modules are wrong), so an explicit sdpa request bypassed the downgrade and re-enabled a known-wrong path. Extend _is_sdpa_excluded to also treat DISABLE_SDPA_MODEL_NAMES membership as excluded, replicating the loader's trailing-comma substring match so gemma3 and gemma3_text match but gemma3n does not. Move the constant into _utils.py (single source of truth, re-exported from loader.py) to avoid a loader -> _utils cycle. Conservative supports_sdpa=False models not in either list still honor explicit sdpa. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 05:45:45 -07:00
Daniel Han	7cc1752a64	Scope MoE expert LoRA detection to actual MLP projection targets (#6849) ... * Scope MoE expert LoRA detection to actual MLP projection targets _moe_target_set_from_string treated any regex containing the substring mlp or ffn as targeting the expert MLP projections. Unsloth's auto-generated attention-only regex lists mlp, ffn and feed_forward as allowed intermediate path segments while its final group matches only q_proj/k_proj/v_proj/o_proj, so attention-only finetuning on MoE models silently enabled expert LoRA as well: the experts were trained and every MoE layer paid the extra expert LoRA grouped matmuls. Detect expert intent from the projection names themselves (gate_proj/up_proj/down_proj/gate_up_proj) instead of the mlp substring. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten comments * Detect MoE expert LoRA via mlp path segment, not proj names The auto-generated target regex always lists every projection leaf (q/k/v/o and gate/up/down), so keying detection on a proj name mis-fired: it enabled expert LoRA for attention-only regexes and dropped the mlp/ffn path regexes. Key on the mlp/ffn/feed_forward/experts path segment instead, which is present only when the MLP/experts are actually targeted. Add a regression test for the attention-only case. * Scope expert LoRA targets to the leaves a regex names An mlp path alternative with attention-only leaves, for example (mlp|self_attn).(q_proj|o_proj), no longer enables expert LoRA, and a regex naming a single expert leaf such as .*experts.*down_proj now targets only that projection instead of the whole broad set. Generic mlp projections (.*mlp.*proj) and the auto regex mlp tag block keep the broad set for fused-expert models whose leaves are plain Parameters. * Route explicit leaf list into MoE expert detection An attention-only explicit target_modules list routed through get_peft_regex for family scoping (e.g. FastVisionModel with vision layers off) yields a regex carrying the full mlp|feed_forward|ffn|dense component block even though its leaf group only names q/k/v/o_proj. Keying expert detection on that regex trained the experts for a language-only/attention-only request. Use the caller's original leaf list for detection; only the auto path uses the regex, where the mlp block is the sole MLP-intent signal on fused-expert models. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Respect finetune_mlp_modules and finetune_language_layers scope for MoE expert detection When an explicit leaf list that names MLP projections (gate_proj/up_proj/down_proj) is routed through get_peft_regex under finetune_mlp_modules=False, the scoped regex correctly drops the MLP leaves, but MoE expert detection was still keyed on the original list and re-added mlp.experts.* via target_parameters, training the experts the caller had frozen. Same gap for finetune_language_layers=False on vision-only runs. Prefer the original list only when MLP and language families are both in scope (preserving the attention-only fix); otherwise honor the scoped result so the frozen family is respected. Factored the choice into _select_moe_detection_targets with unit tests over the full selection matrix. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 05:45:06 -07:00
Daniel Han	22bd86ecb7	Handle odd shapes and non-float scales in FP8BlockQuantLinear (#6848) ... * Handle odd shapes and non-float scales in FP8BlockQuantLinear Small fp8 checkpoints (e.g. tiny test models) break the block-quantized linear in three ways: weight scales stored in a float8 dtype such as float8_e8m0fnu have no triton dtype mapping; activations whose hidden dim is not a multiple of the activation quant block fail act_quant's divisibility assert; and weights whose dims are not multiples of the weight block cannot be tiled by the triton dequant kernel. Cast non-float scales to float32 on entry, and when the hidden dim does not divide into the activation block, dequantize the weight and run a plain matmul instead of the fp8 block matmul. The dequant goes through a new shape-safe helper that falls back to a torch-native scale expansion when the weight does not tile evenly; backward uses the same helper so the gradient path works for every shape the forward accepts. Full-size checkpoints are unaffected. * Add tiny / e8m0 fp8 block-quant regression test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix FP8 block-quant fallback: real block size in dequant and scalar-scale fast path * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route rectangular fp8 blocks through torch dequant and keep block_size across e8m0 upcast The triton weight_dequant kernel uses one BLOCK_SIZE for both axes, so rectangular blocks (block_size[0] != block_size[1]) mis-index the column scale and corrupt grad_X. Route those through the torch scale expansion, which handles each dimension independently, and keep the triton path for square blocks only. Also preserve a block_size attribute carried on the scale tensor across the e8m0 -> float32 upcast so the later lookup no longer falls back to [128, 128]. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-06 05:44:55 -07:00
Daniel Han	08e133cd6b	Add PrefixGrouper for GRPO: dedup the shared prompt across a group's completions (#6871) ... * GRPO: optional sequence packing for the no-grad old/ref logp path Add an opt-in sequence-packing fast path to _get_per_token_logps_and_entropies, enabled with UNSLOTH_GRPO_SEQ_PACKING=1. When the batch is text-only, the padded [B, Lmax] per-chunk forward is replaced by a single varlen [1, sum L] forward (BlockDiagonalCausalMask via packed_seq_lengths with reset position_ids). Per-token logps use the same float32 chunked_hidden_states_selective_log_softmax as the padded path, so the old and reference logps are bit-for-bit identical. Safety: the packed path is self-verified once against the padded ground truth on a batch that has at least two rows with real completion tokens (self._unsloth_seq_packing_nograd_ok), so cross-sample contamination would actually manifest; a degenerate all-pad / fully tool-masked batch leaves the verdict unset and re-verifies later. If a backend silently ignores packed_seq_lengths (flat batch run under a normal causal mask, samples leaking across boundaries), the packed logps will not match and packing is disabled instead of corrupting logps. It also forces use_cache=False (a populated past_key_value disables varlen packing), skips packing when a sliding window is shorter than the packed stream, runs the same GPT-OSS offload device_synchronize the padded loop uses, and falls back on any exception (UNSLOTH_GRPO_SEQ_PACKING_DEBUG=1 prints the reason). Default off, so existing behavior is unchanged. Pairs with the matching gradient-path change in unsloth_zoo so the full GRPO logp + loss + backward can run packed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GRPO no-grad packing: address review feedback - Cache the packed-vs-padded verdict per unwrapped model instead of on the trainer, so a separately forwarded reference model is verified on its own forward path rather than inheriting the policy model's verdict. - Force the padded path when token_type_ids or mm_token_type_ids are present, matching the extra vision kwargs the padded loop forwards. - Require the xformers varlen backend before packing. Without it the packed mask falls back to a dense O(T^2) SDPA mask that can OOM on the flattened batch, so we keep the padded loop in that case. - On any packed-forward failure (missing backend, OOM, unsupported forward) empty the cache on OOM, disable packing for that model, and fall back to the chunked padded loop instead of retrying every step. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GRPO no-grad packing: default-on, verify against per-row reference Redesign of the optional sequence-packing fast path for the no-grad old/ref logprob recompute, after establishing that the packed forward is the exact per-row computation and the padded batch forward is the side that mis-positions left-padded rows on long completions. - Default the packing on (UNSLOTH_GRPO_SEQ_PACKING, disable with 0). - Verify the packed logprobs against the per-row clean forward (each row's real tokens alone, reset 0-based positions, no padding), not the padded batch which is itself wrong for left-padding. Cross-sample contamination (a backend ignoring packed_seq_lengths) shows up as a large mismatch and falls back to the padded loop. - Make the trust decision shape and RoPE aware: re-verify whenever the packed total length or the longest segment grows past what was verified, so a later batch crossing a LongRoPE short/long cache boundary is re-checked instead of trusted blindly. - Run lm_head only on completion-prediction positions instead of every packed prompt token, so long-prompt/short-completion batches do not pay for projecting the whole packed prompt. - Drop the hard xformers import so the path also runs in FlashAttention-only environments; the per-row verification guards correctness regardless of backend. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GRPO no-grad packing: disable entirely on cross-sample mismatch When the per-row verification fails, distinguish the two failure modes by magnitude instead of by sequence length: - A large mismatch (>= 1.5) is the cross-sample contamination signature: the model's attention does not honor the block-diagonal packed mask (seen on some MoE / custom-attention models, e.g. qwen2_moe). Disable packing entirely for the model so later batches do not pay the verification cost again. - A moderate mismatch is more likely a length-boundary effect (a LongRoPE short/long cache switch): keep marking just that length region unsafe so packing still runs for smaller shapes. Validated: Qwen1.5-MoE falls back after a single verification (grad and no-grad ok flags go False, no re-verify on later steps); dense Llama-3.2 and Qwen3 still verify and engage packing. * GRPO no-grad packing: trim comments to be concise * GRPO no-grad packing: fix per-row completion boundary for left-padded rows The completion-target selection used a single global boundary (col >= L - logits_to_keep). After left-packing, each row's completion starts at (L - logits_to_keep) - left_pad[row], so for left-padded rows the first left_pad completion tokens fall below the global boundary and were dropped, leaving 0 logprobs at real completion positions that the loss mask keeps. Use the per-row boundary so packed coverage matches create_completion_attention_mask exactly, and widen the self-verify mask to the full per-row completion region so it can catch coverage gaps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GRPO no-grad packing: gate verification on real completion rows Count active rows via create_completion_attention_mask (the same mask the loss uses) instead of any non-pad token in the packed window. Prompt-only rows carry prompt-overflow tokens in the window and could otherwise satisfy the >= 2 verification guard, letting a batch with a single real completion row cache a trust decision. This matches the gradient path, which already gates on the completion mask. The same mask is reused for the self-verify comparison. * GRPO no-grad packing: gate debug logging on UNSLOTH_ENABLE_LOGGING Use the shared UNSLOTH_ENABLE_LOGGING global (import_fixes, re-exported by _utils) instead of a bespoke UNSLOTH_GRPO_SEQ_PACKING_DEBUG env var for the packing debug prints, matching the rest of the codebase. * GRPO packing: import UNSLOTH_ENABLE_LOGGING inside the injected logp function _get_per_token_logps_and_entropies is copied verbatim into the generated GRPO trainer via inspect.getsource, and that module never imported UNSLOTH_ENABLE_LOGGING, so the default-on packing verify path raised NameError (and the except handler re-raised it). Import the flag locally, before the try, so the name is defined in the generated module too. Drop it from the now-unused module-level import. * GRPO no-grad packing: harden unsafe-length skip, verify guard, fallback cleanup Three fixes to the no-grad logp packing path, mirroring the grad path: - skip the packed forward for known-unsafe lengths by reading unsafe_T and gating on it before the forward, instead of running the full packed pass and the result build only to discard them (wastes a pass, can OOM at large T) - only widen the verified T/seg envelope when >= 2 completion rows actually exercised cross-sample packing; a < 2 row batch cannot expose leakage, so it must not extend the trusted shape that later multi-row batches skip verify for - drop the packed intermediates (hidden/sel/result/ref) before the padded fallback loop so it does not run with the flattened hidden state still resident * GRPO no-grad packing: cap the flattened forward at one mini-batch budget The packed path built a single [1, sum L] forward over every row before any size check, so a large batch could exceed the memory the padded path bounds per mini-batch. Gate packing on _pk_T <= _pk_cap (B * seq_len, one padded mini-batch's token budget); larger batches fall back to the chunked padded loop. * GRPO no-grad packing: disable unless unsloth_zoo has the masked-column guard The packed path leaves masked prompt/pad logprob columns at 0, which only stays finite if unsloth_zoo grpo_compute_loss zeroes them before exp() (zoo#840). An older unsloth_zoo without that guard would NaN. Detect the guard once (cached on the model) via inspect.getsource and gate packing on it, so #6738 is safe with any unsloth_zoo version and re-enables packing automatically once a guarded zoo is installed, independent of the pinned lower bound. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GRPO packing: hoist env gates and zoo-guard detection to one-time module checks Read UNSLOTH_GRPO_SEQ_PACKING and detect the unsloth_zoo masked-column guard once at import time (module constants plus RL_PRE_ITEMS for the generated trainer cache) instead of per call, and drop the in-function UNSLOTH_ENABLE_LOGGING import for a module-top one. The UNSLOTH_GRPO_SEQ_PACKING_VERIFY force-verify debug knob is commented out, kept in place for hand re-enable; the first-use and envelope-growth self-verify stays active. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GRPO packing: cap the flattened forward by the padded chunk rows B counts chunks at this point, so B * seq_len understated (small runs) or overstated (large runs) the padded mini-batch token budget; use batch_size * seq_len, the rows the padded loop actually forwards per chunk. * Add PrefixGrouper for GRPO: dedup the shared prompt across a group's completions In GRPO every prompt spawns G=num_generations completions that share the prompt prefix, so the trunk logprob forward re-encodes that prefix G times. PrefixGrouper stores the prefix once and concatenates only the G suffixes behind a FlexAttention shared-prefix mask, cutting the forward from G*(P+R) to P+G*R tokens across both the no-grad old/ref forwards and the grad logp forward. Default off behind the UNSLOTH_GRPO_PREFIX_GROUPER env gate, so the gate-unset path is byte-identical to today. A tok_r auto-gate and a first-use self-verify (fall back and mark the shape unsafe on mismatch) keep it from ever shipping wrong logprobs silently. Wired for llama, mistral, qwen3, gemma2, cohere, granite and falcon_h1, plus qwen2 and gemma through the shared LlamaAttention_fast_forward. Stacked on the GRPO sequence-packing PR (#6738); the grad path lands in a companion unsloth-zoo PR. Also fixes a latent UNSLOTH_ENABLE_LOGGING NameError in the seq-packing no-grad verify path by defining the name as a generated-cache pre-item. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * PrefixGrouper: enforce the sliding-window cap, gate softcap models, bound the mask cache Add a max_segment_cap kwarg to build_group_layout so it falls back when a group's span (prefix + longest suffix) exceeds the model's local window, and pass the config sliding_window into the no-grad engage gate the same way the packed _pk guard derives it. Skip PrefixGrouper entirely for attn_logit_softcapping models, since the FlexAttention kernel never applies logit softcapping. Bound _BLOCK_MASK_CACHE to a FIFO of 8 so per-step lengths cannot pin BlockMasks forever, release the PG hidden before the verify forward, and align the UNSLOTH_ENABLE_LOGGING pre-item truthiness with the canonical form. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * PrefixGrouper: vectorize the real-column scan in build_group_layout Replace the per-row O(B*L) Python scan of the keep mask with a GPU-derived contiguous-run fast path (first real column + count per row), keeping the general scan only as a fallback for non-contiguous rows. Works for both call sites: the no-grad layout (left-padded prompt + right-padded completion, run does not start at column 0) and the grad layout (left-packed). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * PrefixGrouper: hoist the gate and kernel imports to one-time module checks, AGPLv3 headers Read UNSLOTH_GRPO_PREFIX_GROUPER and resolve the prefix_grouper imports once at module level (source constants plus an RL_PRE_ITEMS entry for the generated trainer cache) instead of per call, matching the sequence-packing gates. The prefix_grouper env helpers become one-time module reads with unchanged signatures, and attention_dispatch resolves the FlexAttention kernel once behind the same gate (lazy fallback kept). The two new prefix_grouper files move to AGPLv3 headers. * PrefixGrouper: length-envelope trust and hybrid SSM exclusion Verified signatures now record (max T, max segment) and re-verify when either grows, matching the packed path's envelope. Hybrid SSM models (FalconH1 etc.) are excluded at the gate since only attention gets the shared-prefix isolation, and the FalconH1 wiring is removed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * PrefixGrouper: defer the unverified no-grad forward until the packed reference exists Unverified shapes no longer run the whole-batch shared-prefix forward up front; it now runs at the verify site, only when the packed path produced a reference. A declined packed path (budget, window) therefore costs no wasted PG forward per step. Trusted shapes still run it first to skip the full-row forward, with the same fallback. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * PrefixGrouper: disable under vLLM (fast_inference=True) With colocated vLLM generation the rollout dominates the GRPO step, so the shared-prefix training forward saves little end-to-end and its first-use self-verify (which also runs the full-row path) is net overhead. Gate PG on not use_vllm so it only engages on the raw transformers path, where the training forward is on the critical path. Packing is unaffected. * PrefixGrouper: compile the FlexAttention kernel with dynamic shapes GRPO changes the packed length T almost every batch. With dynamic=False the flex forward+backward kernel recompiled on every new T (~14s each on a 4B trunk), which dominated the step and made PG a net loss. dynamic=True compiles once, then reuses the kernel across all lengths recompile-free (a new shape drops from ~14s to ~1.4ms after a two-graph warmup). T is still padded to a multiple of 128 for the backward block assertion. * PrefixGrouper: default on Enable PrefixGrouper by default (UNSLOTH_GRPO_PREFIX_GROUPER defaults to 1; set 0 to disable). Still auto-disabled under vLLM (fast_inference=True) and by the arch/softcap/ SSM/tok_r gates, and the first-use self-verify falls back on any mismatch, so this is a memory-first default on the raw-transformers path with no correctness risk. * GRPO PrefixGrouper: gate on zoo masked-column guard and exclude MoE - Require the zoo masked-column guard (zoo#840) before PrefixGrouper can engage. PG rides the sequence-packing path, so when the first-step self-verify is off the fast path trusts PG output directly; without the guard those masked columns feed NaN into the packed loss. Gate PG on the same UNSLOTH_ZOO_HAS_MASKED_COL_GUARD the packing path already checks. - Exclude MoE configs (num_experts, num_local_experts, n_routed_experts, moe_intermediate_size) alongside the hybrid-SSM markers. Only the threaded attention forwards carry the shared-prefix isolation, so a MoE decoder that does not forward prefix_seg_info would let suffixes leak across completions. - Refresh the stale default-off comments now that UNSLOTH_GRPO_PREFIX_GROUPER is on by default. * GRPO PrefixGrouper: import chunked_hidden_states_selective_log_softmax The shared-prefix forward passes chunked_hidden_states_selective_log_softmax into extract_logps, but the name was only ever provided by the generated trainer cache (rl.py injects grpo_selective_log_softmax_code), never bound in this module. Import it from unsloth_zoo.rl_replacements next to its sibling chunked_selective_log_softmax so the source resolves the name in every scope (the new _pg_run_forward closure included). No runtime change: the cache still defines the function via template injection. * GRPO PrefixGrouper: dropout gate, device-safe layout, Mistral mask skip Addresses three review findings on the shared-prefix path: - Skip PrefixGrouper when the model sets a nonzero attention_dropout. The normal backends apply config.attention_dropout while training (e.g. Granite dense flash/sdpa/xformers), but the FlexAttention shared-prefix path is deterministic, so gate PG off for those configs rather than train on mismatched activations. - Move the shared-prefix mask labels to the consumer (Q) device in get_block_mask and the target index maps to hidden.device in extract_logps, mirroring the packed path moving its metadata to the consumer device. Prevents cross-device indexing when the model is sharded across GPUs. - Do not synthesize a causal attention_mask in the Mistral forward when prefix_seg_info is present. On the no-xFormers path that synthetic mask tripped resolve_prefix_seg_info and forced PG to always fall back to the packed forward. * GRPO sequence packing: tighten comments * GRPO PrefixGrouper: tighten comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GRPO PrefixGrouper: persistent disable on runtime failure; build block-mask labels with inference mode disabled - rl_replacements: on a PG forward exception (FlexAttention/Triton compile failure or OOM), set a model-level _unsloth_prefix_grouper_nograd_disabled flag and consult it in the engage gate, mirroring the seq-packing handler, so a GPU-wide failure is not retried and re-paid every step. - prefix_grouper_kernel: move the .to(device) label copies inside the inference_mode(False) block so a cross-device (model-parallel shard) first build does not capture inference tensors, which otherwise cannot be saved for backward when the grad training forward reuses the cached BlockMask. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>	2026-07-06 05:26:24 -07:00
Daniel Han	9407d49193	GRPO: sequence packing for the no-grad old/ref logp path (default-on) (#6738) ... * GRPO: optional sequence packing for the no-grad old/ref logp path Add an opt-in sequence-packing fast path to _get_per_token_logps_and_entropies, enabled with UNSLOTH_GRPO_SEQ_PACKING=1. When the batch is text-only, the padded [B, Lmax] per-chunk forward is replaced by a single varlen [1, sum L] forward (BlockDiagonalCausalMask via packed_seq_lengths with reset position_ids). Per-token logps use the same float32 chunked_hidden_states_selective_log_softmax as the padded path, so the old and reference logps are bit-for-bit identical. Safety: the packed path is self-verified once against the padded ground truth on a batch that has at least two rows with real completion tokens (self._unsloth_seq_packing_nograd_ok), so cross-sample contamination would actually manifest; a degenerate all-pad / fully tool-masked batch leaves the verdict unset and re-verifies later. If a backend silently ignores packed_seq_lengths (flat batch run under a normal causal mask, samples leaking across boundaries), the packed logps will not match and packing is disabled instead of corrupting logps. It also forces use_cache=False (a populated past_key_value disables varlen packing), skips packing when a sliding window is shorter than the packed stream, runs the same GPT-OSS offload device_synchronize the padded loop uses, and falls back on any exception (UNSLOTH_GRPO_SEQ_PACKING_DEBUG=1 prints the reason). Default off, so existing behavior is unchanged. Pairs with the matching gradient-path change in unsloth_zoo so the full GRPO logp + loss + backward can run packed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GRPO no-grad packing: address review feedback - Cache the packed-vs-padded verdict per unwrapped model instead of on the trainer, so a separately forwarded reference model is verified on its own forward path rather than inheriting the policy model's verdict. - Force the padded path when token_type_ids or mm_token_type_ids are present, matching the extra vision kwargs the padded loop forwards. - Require the xformers varlen backend before packing. Without it the packed mask falls back to a dense O(T^2) SDPA mask that can OOM on the flattened batch, so we keep the padded loop in that case. - On any packed-forward failure (missing backend, OOM, unsupported forward) empty the cache on OOM, disable packing for that model, and fall back to the chunked padded loop instead of retrying every step. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GRPO no-grad packing: default-on, verify against per-row reference Redesign of the optional sequence-packing fast path for the no-grad old/ref logprob recompute, after establishing that the packed forward is the exact per-row computation and the padded batch forward is the side that mis-positions left-padded rows on long completions. - Default the packing on (UNSLOTH_GRPO_SEQ_PACKING, disable with 0). - Verify the packed logprobs against the per-row clean forward (each row's real tokens alone, reset 0-based positions, no padding), not the padded batch which is itself wrong for left-padding. Cross-sample contamination (a backend ignoring packed_seq_lengths) shows up as a large mismatch and falls back to the padded loop. - Make the trust decision shape and RoPE aware: re-verify whenever the packed total length or the longest segment grows past what was verified, so a later batch crossing a LongRoPE short/long cache boundary is re-checked instead of trusted blindly. - Run lm_head only on completion-prediction positions instead of every packed prompt token, so long-prompt/short-completion batches do not pay for projecting the whole packed prompt. - Drop the hard xformers import so the path also runs in FlashAttention-only environments; the per-row verification guards correctness regardless of backend. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GRPO no-grad packing: disable entirely on cross-sample mismatch When the per-row verification fails, distinguish the two failure modes by magnitude instead of by sequence length: - A large mismatch (>= 1.5) is the cross-sample contamination signature: the model's attention does not honor the block-diagonal packed mask (seen on some MoE / custom-attention models, e.g. qwen2_moe). Disable packing entirely for the model so later batches do not pay the verification cost again. - A moderate mismatch is more likely a length-boundary effect (a LongRoPE short/long cache switch): keep marking just that length region unsafe so packing still runs for smaller shapes. Validated: Qwen1.5-MoE falls back after a single verification (grad and no-grad ok flags go False, no re-verify on later steps); dense Llama-3.2 and Qwen3 still verify and engage packing. * GRPO no-grad packing: trim comments to be concise * GRPO no-grad packing: fix per-row completion boundary for left-padded rows The completion-target selection used a single global boundary (col >= L - logits_to_keep). After left-packing, each row's completion starts at (L - logits_to_keep) - left_pad[row], so for left-padded rows the first left_pad completion tokens fall below the global boundary and were dropped, leaving 0 logprobs at real completion positions that the loss mask keeps. Use the per-row boundary so packed coverage matches create_completion_attention_mask exactly, and widen the self-verify mask to the full per-row completion region so it can catch coverage gaps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GRPO no-grad packing: gate verification on real completion rows Count active rows via create_completion_attention_mask (the same mask the loss uses) instead of any non-pad token in the packed window. Prompt-only rows carry prompt-overflow tokens in the window and could otherwise satisfy the >= 2 verification guard, letting a batch with a single real completion row cache a trust decision. This matches the gradient path, which already gates on the completion mask. The same mask is reused for the self-verify comparison. * GRPO no-grad packing: gate debug logging on UNSLOTH_ENABLE_LOGGING Use the shared UNSLOTH_ENABLE_LOGGING global (import_fixes, re-exported by _utils) instead of a bespoke UNSLOTH_GRPO_SEQ_PACKING_DEBUG env var for the packing debug prints, matching the rest of the codebase. * GRPO packing: import UNSLOTH_ENABLE_LOGGING inside the injected logp function _get_per_token_logps_and_entropies is copied verbatim into the generated GRPO trainer via inspect.getsource, and that module never imported UNSLOTH_ENABLE_LOGGING, so the default-on packing verify path raised NameError (and the except handler re-raised it). Import the flag locally, before the try, so the name is defined in the generated module too. Drop it from the now-unused module-level import. * GRPO no-grad packing: harden unsafe-length skip, verify guard, fallback cleanup Three fixes to the no-grad logp packing path, mirroring the grad path: - skip the packed forward for known-unsafe lengths by reading unsafe_T and gating on it before the forward, instead of running the full packed pass and the result build only to discard them (wastes a pass, can OOM at large T) - only widen the verified T/seg envelope when >= 2 completion rows actually exercised cross-sample packing; a < 2 row batch cannot expose leakage, so it must not extend the trusted shape that later multi-row batches skip verify for - drop the packed intermediates (hidden/sel/result/ref) before the padded fallback loop so it does not run with the flattened hidden state still resident * GRPO no-grad packing: cap the flattened forward at one mini-batch budget The packed path built a single [1, sum L] forward over every row before any size check, so a large batch could exceed the memory the padded path bounds per mini-batch. Gate packing on _pk_T <= _pk_cap (B * seq_len, one padded mini-batch's token budget); larger batches fall back to the chunked padded loop. * GRPO no-grad packing: disable unless unsloth_zoo has the masked-column guard The packed path leaves masked prompt/pad logprob columns at 0, which only stays finite if unsloth_zoo grpo_compute_loss zeroes them before exp() (zoo#840). An older unsloth_zoo without that guard would NaN. Detect the guard once (cached on the model) via inspect.getsource and gate packing on it, so #6738 is safe with any unsloth_zoo version and re-enables packing automatically once a guarded zoo is installed, independent of the pinned lower bound. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GRPO packing: hoist env gates and zoo-guard detection to one-time module checks Read UNSLOTH_GRPO_SEQ_PACKING and detect the unsloth_zoo masked-column guard once at import time (module constants plus RL_PRE_ITEMS for the generated trainer cache) instead of per call, and drop the in-function UNSLOTH_ENABLE_LOGGING import for a module-top one. The UNSLOTH_GRPO_SEQ_PACKING_VERIFY force-verify debug knob is commented out, kept in place for hand re-enable; the first-use and envelope-growth self-verify stays active. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * GRPO packing: cap the flattened forward by the padded chunk rows B counts chunks at this point, so B * seq_len understated (small runs) or overstated (large runs) the padded mini-batch token budget; use batch_size * seq_len, the rows the padded loop actually forwards per chunk. * GRPO sequence packing: tighten comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>	2026-07-06 05:17:39 -07:00