vrr/qwen-code
2
0
Fork 0
mirror of https://github.com/QwenLM/qwen-code.git synced 2026-08-15 19:54:54 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 16
qwen-code/docs/design/toolsearch-preload-threshold.md
Dragon 18cb393e4b
feat(core): preload deferred tools within a context-window threshold (#7922) 
* feat(core): preload deferred tools within a context-window threshold

Adds tools.toolSearch.threshold (default 10, percent of the context
window). At session start, when the combined estimated schema footprint
of every deferred tool - bundled built-ins and MCP alike - fits within
the budget, all are revealed upfront so the declaration list stays
stable for the whole session and prefix KV caches survive; otherwise
everything stays deferred. Set 0 to always defer. Mirrors Claude Code's
ENABLE_TOOL_SEARCH=auto threshold mode, extended to bundled deferred
tools because here every reveal rewrites the declaration list and busts
the prompt-cache prefix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(core): log deferred-tool preload budget decision

Emit debugLogger diagnostics in preloadDeferredToolsWithinBudget covering
the computed budget, estimated token footprint, candidate count, and which
branch of the all-or-nothing gate was taken (no candidates, over budget, or
preloaded). Lets an operator diagnosing session-startup cost tell from debug
logs whether the deferred set fit the budget or was left behind ToolSearch,
without adding temporary instrumentation. No behavior change.

* fix(tools): bound toolSearch.threshold to 0-100%

The threshold setting is a percentage of the context window but had no
upper bound, so a value like 200 (a typo or misreading of the "(%)"
label) made the preload budget exceed the whole window and
unconditionally preloaded every deferred tool — the opposite of the
prefix-stability the threshold buys.

- Add minimum:0/maximum:100 to the setting schema (jsonSchemaOverride,
  like autoCompactThreshold) and regenerate the VS Code settings schema.
- Add a symmetric runtime upper guard next to the existing
  'thresholdPercent <= 0' lower guard in client.ts, clamping to 100% so a
  hand-edited settings file cannot slip a larger budget past validation.

Adds a client test asserting a 200% threshold clamps to a full-context
budget.

* test(core): cover configured preload budget

* fix(tool-search): harden preload threshold

* test(tool-search): cover preload exclusions

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 23:01:01 +00:00

4.1 KiB
Raw Permalink Blame History

ToolSearch preload threshold

Problem

Deferred tools (shouldDefer=true) are unconditionally hidden behind ToolSearch: every MCP tool (hardcoded in DiscoveredMCPTool) plus a set of bundled built-ins (web_search, web_fetch, cron, monitor, worktree, …). Deferral saves prompt tokens when the deferred set is large, but it is not free: every mid-session reveal rewrites the function-declaration list, which sits at the front of the tools→system→messages prefix, so a single ToolSearch load invalidates the entire prompt KV cache. For a small deferred set the deferral saves little and the cache damage plus the extra ToolSearch round-trip make it a net loss.

Claude Code models this tradeoff with ENABLE_TOOL_SEARCH=auto / auto:N: "tools load upfront if they fit within 10% of the context window, deferred otherwise" (code.claude.com/docs/en/agent-sdk/tool-search). This change adds the equivalent gate.

Design

New setting tools.toolSearch.threshold (number, percent, default 10).

At session start (GeminiClient.startChat, before the deferred-tools reminder is resolved), when ToolSearch is registered and the threshold is > 0:

  • Estimate the combined token footprint of every deferred tool schema — bundled built-ins and MCP alike (JSON.stringify(tool.schema).length / CHARS_PER_TOKEN).
  • If the total fits within threshold% of the context window (contentGeneratorConfig.contextWindowSize, falling back to tokenLimit(model)), reveal them all via the existing revealDeferredTool mechanism. All-or-nothing — a partial reveal would leave an arbitrary subset behind ToolSearch, and any tool left deferred can still bust the cache on first use.
  • Otherwise everything stays deferred (previous behavior). threshold: 0 restores the old behavior unconditionally.

Preloaded tools therefore land in the initial declaration list, are filtered out of the startup deferred-tools reminder, and the declaration list stays stable for the whole session.

Decisions

  • Session start only, never setTools(). Revealing a tool the startup reminder already announced would make queueAddedMcpToolsReminder flag it as "removed", and a mid-session declaration change busts the very cache the preload exists to protect. Tools from servers that connect later stay deferred (announced via the added-tools reminder, reachable through ToolSearch) until the next session start. /clear clears the revealed set and re-runs the decision.
  • One budget over the whole deferred set, bundled included. Claude Code's auto threshold covers MCP/SDK tools only (its built-ins are managed separately), but it can afford that split: deferred tools are stripped from the prompt prefix before the cache key is computed, and a discovered tool's definition is expanded inline via a tool_reference block — "The prefix is untouched, so prompt caching is preserved" (platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool). Here every reveal — bundled or MCP — goes through setTools() and rewrites the declaration list. Excluding the ~14 bundled deferred tools (web_search, web_fetch, …) would leave the prefix one common tool-load away from a full cache bust, forfeiting exactly the stability the preload buys. When the union exceeds the budget everything stays deferred, which matches the pre-threshold baseline for bundled tools.
  • Threshold defaults to 10 (auto-mode on), unlike Claude Code's default. Claude Code's unset default keeps MCP tools always deferred and makes auto opt-in — affordable there because a deferred tool's first use costs no cache invalidation. Here it costs a full prefix rebuild, so the auto-style gate is on by default; threshold: 0 reproduces Claude Code's always-defer default.
  • Already-revealed tools count toward the budget so repeated session starts (compression also passes through startChat) cannot ratchet the revealed set past the budget as servers come and go.
  • No preload when ToolSearch is unavailable — the existing eager-reveal branch in resolveDeferredToolsForReminder already exposes everything.