mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-07-10 01:29:17 +00:00
fix(core): stop repeated truncated write_file/edit retries from looping (#5934)
* fix(core): stop repeated truncated edit retries * fix(core): default output tokens to the model limit instead of the 8K cap The 8K CAPPED_DEFAULT_MAX_TOKENS made normal large responses (esp. file writes) truncate, forcing a truncate->escalate round-trip and, worst case, a retry loop. Default to the model's declared output limit instead; the existing escalation + multi-turn recovery stay as the truncation backstop. The 8K cap was a slot-reservation optimization. Claude Code keeps the same cap but gates it behind a feature flag that defaults OFF for third-party providers; qwen-code's providers are all third-party / OpenAI-compatible / self-hosted, so matching that default-off behavior is the safe choice. The capacity tradeoff stays opt-in via QWEN_CODE_MAX_OUTPUT_TOKENS. Refs #5756 * fix(core): use a truncation-specific stop directive for repeated truncated writes * docs: update max tokens configuration wording
This commit is contained in:
易良
GitHub
parent
a1f7846cd1
commit
f33dd61f8a
12 changed files with 183 additions and 126 deletions
|
|
@ -1,25 +1,25 @@
|
|||
# Adaptive Output Token Escalation Design
|
||||
# Output Token Limit and Escalation Design
|
||||
|
||||
> Reduces GPU slot over-reservation by ~4x through a "low default + escalate on truncation" strategy for output tokens, with multi-turn recovery for responses that exceed even the escalated limit.
|
||||
> Defaults to the model's declared output limit unless the user or environment configures `max_tokens`, then uses escalation and multi-turn recovery only when a response still hits `MAX_TOKENS`.
|
||||
|
||||
## Problem
|
||||
|
||||
Every API request reserves a fixed GPU slot proportional to `max_tokens`. The previous default of 32K tokens means each request reserves a 32K output slot, but 99% of responses are under 5K tokens. This over-reserves GPU capacity by 4-6x, limiting server concurrency and increasing cost.
|
||||
Every API request reserves a fixed GPU slot proportional to `max_tokens`. A low default can reduce slot reservation, but it also makes normal large responses more likely to truncate. For file-writing workflows, that can produce incomplete tool-call arguments and force the scheduler to reject the partial write.
|
||||
|
||||
## Solution
|
||||
|
||||
Use a capped default of **8K** output tokens. When a response is truncated (the model hits `max_tokens`):
|
||||
Use the model's declared output limit by default. When a response is truncated (the model hits `max_tokens`):
|
||||
|
||||
1. **Escalate** to the model's full output limit (with 64K as a floor for unknown models)
|
||||
1. **Escalate** to the model's full output limit (with 64K as a floor when the current limit is lower)
|
||||
2. If still truncated, **recover** by keeping the partial response in history and injecting a continuation message, up to 3 times
|
||||
3. If recovery is exhausted, fall back to the tool scheduler's truncation guidance
|
||||
|
||||
Since <1% of requests are actually truncated, this reduces average slot reservation significantly while preserving output quality for long responses.
|
||||
This favors correctness for large generation and file-edit tasks. Operators that need a lower reservation can still set `QWEN_CODE_MAX_OUTPUT_TOKENS`, and that explicit value is respected.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Request (max_tokens = 8K)
|
||||
Request (max_tokens = user/env value or model output limit)
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────┐
|
||||
|
|
@ -78,11 +78,11 @@ Request (max_tokens = 8K)
|
|||
|
||||
The effective `max_tokens` is resolved in the following priority order:
|
||||
|
||||
| Priority | Source | Value (known model) | Value (unknown model) | Escalation behavior |
|
||||
| ----------- | ---------------------------------------------------- | ---------------------------- | --------------------- | ----------------------------------------------- |
|
||||
| 1 (highest) | User config (`samplingParams.max_tokens`) | `min(userValue, modelLimit)` | `userValue` | No escalation |
|
||||
| 2 | Environment variable (`QWEN_CODE_MAX_OUTPUT_TOKENS`) | `min(envValue, modelLimit)` | `envValue` | No escalation |
|
||||
| 3 (lowest) | Capped default | `min(modelLimit, 8K)` | `min(32K, 8K)` = 8K | Escalates to model limit (64K floor) + recovery |
|
||||
| Priority | Source | Value (known model) | Value (unknown model) | Escalation behavior |
|
||||
| ----------- | ---------------------------------------------------- | ---------------------------- | ---------------------------------- | ----------------------------------------------- |
|
||||
| 1 (highest) | User config (`samplingParams.max_tokens`) | `min(userValue, modelLimit)` | `userValue` | No escalation |
|
||||
| 2 | Environment variable (`QWEN_CODE_MAX_OUTPUT_TOKENS`) | `min(envValue, modelLimit)` | `envValue` | No escalation |
|
||||
| 3 (lowest) | Model/default output limit | `modelLimit` | `DEFAULT_OUTPUT_TOKEN_LIMIT` = 32K | Escalates to model limit (64K floor) + recovery |
|
||||
|
||||
A "known model" is one that has an explicit entry in `OUTPUT_PATTERNS` (checked via `hasExplicitOutputLimit()`). For known models, the effective value is always capped at the model's declared output limit to avoid API errors. Unknown models (custom deployments, self-hosted endpoints) pass the user's value through directly, since the backend may support larger limits.
|
||||
|
||||
|
|
@ -143,11 +143,10 @@ The `isContinuation` flag is passed through to the UI so it can decide whether t
|
|||
|
||||
Defined in `geminiChat.ts` and `tokenLimits.ts`:
|
||||
|
||||
| Constant | Value | Purpose |
|
||||
| ------------------------------ | ------ | ------------------------------------------------------- |
|
||||
| `CAPPED_DEFAULT_MAX_TOKENS` | 8,000 | Default output token limit when no user override is set |
|
||||
| `ESCALATED_MAX_TOKENS` | 64,000 | Floor for escalation (used when model limit is unknown) |
|
||||
| `MAX_OUTPUT_RECOVERY_ATTEMPTS` | 3 | Max multi-turn recovery attempts after escalation |
|
||||
| Constant | Value | Purpose |
|
||||
| ------------------------------ | ------ | ------------------------------------------------- |
|
||||
| `ESCALATED_MAX_TOKENS` | 64,000 | Floor for escalation when the model limit is low |
|
||||
| `MAX_OUTPUT_RECOVERY_ATTEMPTS` | 3 | Max multi-turn recovery attempts after escalation |
|
||||
|
||||
The effective escalated limit is `max(ESCALATED_MAX_TOKENS, tokenLimit(model, 'output'))`:
|
||||
|
||||
|
|
@ -160,11 +159,12 @@ The effective escalated limit is `max(ESCALATED_MAX_TOKENS, tokenLimit(model, 'o
|
|||
|
||||
## Design decisions
|
||||
|
||||
### Why 8K default?
|
||||
### Why not use an 8K default?
|
||||
|
||||
- 99% of responses are under 5K tokens
|
||||
- 8K provides reasonable headroom for slightly longer responses without triggering unnecessary retries
|
||||
- Reduces average slot reservation from 32K to 8K (4x improvement)
|
||||
- An 8K default is a slot-reservation/capacity optimization, not a correctness requirement. It trades correctness (large responses truncate) for backend throughput (a request reserves a GPU slot proportional to `max_tokens`, so a lower value over-reserves less).
|
||||
- Large file generation and edit tool calls can legitimately exceed 8K, so an 8K default turns a normal request into a truncate → escalate round-trip (and, in the worst case, a retry loop).
|
||||
- Claude Code keeps the same 8K cap but gates it behind a feature flag (`tengu_otk_slot_v1`) that **defaults to off for third-party providers** ("not validated on Bedrock/Vertex") — i.e. its default behavior for non-first-party serving is exactly "use the model's declared limit." qwen-code's providers are all third-party / OpenAI-compatible / self-hosted, so matching that default-off behavior is the safe choice; assuming the low default is safe for every backend is not.
|
||||
- The capacity tradeoff is not lost, only made opt-in: operators on a capacity-constrained self-hosted backend can set `QWEN_CODE_MAX_OUTPUT_TOKENS` (e.g. `8000`) to restore the lower per-request reservation. A GrowthBook-style feature flag is intentionally not reintroduced — qwen-code has no such infrastructure, and the env var already covers the need.
|
||||
|
||||
### Why escalate to model limit instead of fixed 64K?
|
||||
|
||||
|
|
@ -174,7 +174,7 @@ The effective escalated limit is `max(ESCALATED_MAX_TOKENS, tokenLimit(model, 'o
|
|||
|
||||
### Why multi-turn recovery instead of progressive escalation?
|
||||
|
||||
- Progressive escalation (8K → 16K → 32K → 64K) requires regenerating the full response each time
|
||||
- Progressive escalation (for example 16K -> 32K -> 64K) requires regenerating the full response each time
|
||||
- Multi-turn recovery keeps the partial response and lets the model continue, saving tokens and latency
|
||||
- Recovery messages are cheap (~40 tokens each) compared to regenerating large responses
|
||||
- The 3-attempt limit prevents infinite loops while covering most practical cases
|
||||
|
|
|
|||
|
|
@ -189,17 +189,13 @@ Settings are organized into categories. Most settings should be placed within th
|
|||
}
|
||||
```
|
||||
|
||||
**max_tokens (adaptive output tokens):**
|
||||
**max_tokens (output token limit):**
|
||||
|
||||
When `samplingParams.max_tokens` is not set, Qwen Code uses an adaptive output token strategy to optimize GPU resource usage:
|
||||
When neither `samplingParams.max_tokens` nor `QWEN_CODE_MAX_OUTPUT_TOKENS` is set, Qwen Code generally uses the selected model's declared output limit as the request's default output limit. If the response still hits that limit, Qwen Code may retry with an escalated limit (using a 64K floor) and then recover across continuation turns.
|
||||
|
||||
1. Requests start with a default limit of **8K** output tokens
|
||||
2. If the response is truncated (the model hits the limit), Qwen Code automatically retries with **64K** tokens
|
||||
3. The partial output is discarded and replaced with the full response from the retry
|
||||
For OpenAI-compatible providers, `samplingParams` is also a wire-shape escape hatch: when it is set, its keys are passed through verbatim and Qwen Code does not synthesize a `max_tokens` default. Use this for provider-specific parameters such as `max_completion_tokens`.
|
||||
|
||||
This is transparent to users — you may briefly see a retry indicator if escalation occurs. Since 99% of responses are under 5K tokens, the retry happens rarely (<1% of requests).
|
||||
|
||||
To override this behavior, either set `samplingParams.max_tokens` in your settings or use the `QWEN_CODE_MAX_OUTPUT_TOKENS` environment variable.
|
||||
To force a fixed output limit, set `samplingParams.max_tokens` in your settings or use the `QWEN_CODE_MAX_OUTPUT_TOKENS` environment variable. Explicit limits disable automatic output-token escalation.
|
||||
|
||||
**toolResultContentFormat:**
|
||||
|
||||
|
|
@ -600,33 +596,33 @@ For authentication-related variables (like `OPENAI_*`) and the recommended `.qwe
|
|||
|
||||
### Environment Variables Table
|
||||
|
||||
| Variable | Description | Notes |
|
||||
| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `QWEN_HOME` | Customizes the global configuration directory (default: `~/.qwen`). Accepts an absolute or relative path (relative paths are resolved from the current working directory). Leading `~` is expanded to the user's home directory. | Stores credentials, settings, memory, skills, and other global state. When set, project-level `.qwen/` directories are unaffected. An empty string is treated as unset. |
|
||||
| `QWEN_RUNTIME_DIR` | Overrides the runtime output directory (conversations, logs, todos). When unset, defaults to the `QWEN_HOME` directory. | Use this to separate ephemeral runtime data from persistent config. Useful when `QWEN_HOME` is on a shared/slow filesystem. |
|
||||
| `QWEN_TELEMETRY_ENABLED` | Set to `true` or `1` to enable telemetry. Any other value is treated as disabling it. | Overrides the `telemetry.enabled` setting. |
|
||||
| `QWEN_TELEMETRY_TARGET` | Sets an informational label for the telemetry destination (`local` or `gcp`). Does not control routing; use `QWEN_TELEMETRY_OTLP_ENDPOINT` or `QWEN_TELEMETRY_OUTFILE` to configure where data is sent. | Overrides the `telemetry.target` setting. |
|
||||
| `QWEN_TELEMETRY_OTLP_ENDPOINT` | Sets the OTLP endpoint for telemetry. | Overrides the `telemetry.otlpEndpoint` setting. |
|
||||
| `QWEN_TELEMETRY_OTLP_PROTOCOL` | Sets the OTLP protocol (`grpc` or `http`). | Overrides the `telemetry.otlpProtocol` setting. |
|
||||
| `QWEN_TELEMETRY_LOG_PROMPTS` | Set to `true` or `1` to enable or disable logging of user prompts. Any other value is treated as disabling it. | Overrides the `telemetry.logPrompts` setting. |
|
||||
| `QWEN_TELEMETRY_INCLUDE_SENSITIVE_SPAN_ATTRIBUTES` | Set to `true` or `1` to attach verbatim user prompts, system prompts, tool I/O, and model responses to native OTel span attributes (and keep `prompt` / `function_args` / `response_text` on log-to-span bridge spans). Any other value disables it. | Overrides the `telemetry.includeSensitiveSpanAttributes` setting. ⚠️ Streams sensitive data to your OTLP backend. |
|
||||
| `QWEN_TELEMETRY_SENSITIVE_SPAN_ATTRIBUTE_MAX_LENGTH` | Sets the maximum JavaScript string length for each sensitive native OTel span attribute content payload. Must be a positive integer no greater than `104857600` (100 MiB). | Overrides the `telemetry.sensitiveSpanAttributeMaxLength` setting. Default is `1048576` (1 MiB); lower it if your collector or backend rejects large span attributes. |
|
||||
| `QWEN_TELEMETRY_OUTFILE` | Sets the file path to write telemetry to. When set, overrides OTLP export. | Overrides the `telemetry.outfile` setting. |
|
||||
| `QWEN_SANDBOX` | Alternative to the `sandbox` setting in `settings.json`. | Accepts `true`, `false`, `docker`, `podman`, or a custom command string. |
|
||||
| `QWEN_SANDBOX_IMAGE` | Overrides sandbox image selection for Docker/Podman. | Takes precedence over `tools.sandboxImage`. |
|
||||
| `SEATBELT_PROFILE` | (macOS specific) Switches the Seatbelt (`sandbox-exec`) profile on macOS. | `permissive-open`: (Default) Restricts writes to the project folder (and a few other folders, see `packages/cli/src/utils/sandbox-macos-permissive-open.sb`) but allows other operations. `strict`: Uses a strict profile that declines operations by default. `<profile_name>`: Uses a custom profile. To define a custom profile, create a file named `sandbox-macos-<profile_name>.sb` in your project's `.qwen/` directory (e.g., `my-project/.qwen/sandbox-macos-custom.sb`). |
|
||||
| `DEBUG` or `DEBUG_MODE` | (often used by underlying libraries or the CLI itself) Set to `true` or `1` to enable verbose debug logging, which can be helpful for troubleshooting. | **Note:** These variables are automatically excluded from project `.env` files by default to prevent interference with the CLI behavior. Use `.qwen/.env` files if you need to set these for Qwen Code specifically. |
|
||||
| `NO_COLOR` | Set to any value to disable all color output in the CLI. | |
|
||||
| `FORCE_HYPERLINK` | Override the OSC 8 clickable-link detection in the markdown renderer. Set to `1` (or any non-zero integer, or empty string) to force-enable; set to `0` or a non-numeric value such as `false` / `off` to force-disable. Honors `NO_COLOR` / `QWEN_DISABLE_HYPERLINKS` opt-outs above it. | Use this to opt into OSC 8 inside `tmux` / GNU `screen` (auto-detection refuses by default because the host terminal's capabilities are hidden behind the multiplexer). Requires `set -g allow-passthrough on` on tmux 3.3+. Also enables Hyper, which isn't auto-detected. |
|
||||
| `QWEN_DISABLE_HYPERLINKS` | Set to `1` to hard-disable OSC 8 clickable hyperlinks in the markdown renderer even on terminals that auto-detect as capable. | Useful when a terminal advertises support but breaks on long URLs, or when piping output through an intermediary that mangles escape sequences. The renderer falls back to plain `label (url)` rendering. |
|
||||
| `CLI_TITLE` | Set to a string to customize the title of the CLI. | |
|
||||
| `CODE_ASSIST_ENDPOINT` | Specifies the endpoint for the code assist server. | This is useful for development and testing. |
|
||||
| `QWEN_CODE_MAX_OUTPUT_TOKENS` | Overrides the default maximum output tokens per response. When not set, Qwen Code uses an adaptive strategy: starts with 8K tokens and automatically retries with 64K if the response is truncated. Set this to a specific value (e.g., `16000`) to use a fixed limit instead. | Takes precedence over the capped default (8K) but is overridden by `samplingParams.max_tokens` in settings. Disables automatic escalation when set. Example: `export QWEN_CODE_MAX_OUTPUT_TOKENS=16000` |
|
||||
| `QWEN_CODE_UNATTENDED_RETRY` | Set to `true` or `1` to enable persistent retry mode. When enabled, transient API capacity errors (HTTP 429 Rate Limit and 529 Overloaded) are retried indefinitely with exponential backoff (capped at 5 minutes per retry) and heartbeat keepalives every 30 seconds on stderr. | Designed for CI/CD pipelines and background automation where long-running tasks should survive temporary API outages. Must be set explicitly — `CI=true` alone does **not** activate this mode. See [Headless Mode](../features/headless#persistent-retry-mode) for details. Example: `export QWEN_CODE_UNATTENDED_RETRY=1` |
|
||||
| `QWEN_CODE_PROFILE_STARTUP` | Set to `1` to enable startup performance profiling. Writes a JSON timing report to `~/.qwen/startup-perf/` with per-phase durations. | Only active inside the sandbox child process (or with `QWEN_CODE_PROFILE_STARTUP_OUTER=1`). Zero overhead when not set. Example: `export QWEN_CODE_PROFILE_STARTUP=1` |
|
||||
| `QWEN_CODE_PROFILE_STARTUP_OUTER` | Set to `1` together with `QWEN_CODE_PROFILE_STARTUP=1` to also collect a startup profile in the outer (pre-sandbox) process. Outer-process reports get an `outer-` filename prefix to keep them distinct from the sandbox child's report. | Off by default — only the sandbox child collects, to avoid duplicate reports. Useful for local development where the cli isn't relaunched into a sandbox. |
|
||||
| `QWEN_CODE_PROFILE_STARTUP_NO_HEAP` | Set to `1` together with `QWEN_CODE_PROFILE_STARTUP=1` to skip the per-checkpoint `process.memoryUsage()` snapshots. Useful when measuring the profiler's own Heisenberg overhead. | Off by default. Heap snapshots cost ~50 µs each (well below 1% of total startup) so most users should leave this alone. |
|
||||
| `QWEN_CODE_LEGACY_MCP_BLOCKING` | Set to `1` to restore the pre-progressive-MCP behavior where `Config.initialize()` waits synchronously for every configured MCP server's discover handshake before returning. | Off by default. Modern qwen-code lets MCP servers come online in the background while the UI is already interactive; the model sees each batch of new tools within ~16 ms of the server settling. This flag is kept as a rollback escape hatch for ≥ 1 release. Example: `export QWEN_CODE_LEGACY_MCP_BLOCKING=1` |
|
||||
| Variable | Description | Notes |
|
||||
| ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `QWEN_HOME` | Customizes the global configuration directory (default: `~/.qwen`). Accepts an absolute or relative path (relative paths are resolved from the current working directory). Leading `~` is expanded to the user's home directory. | Stores credentials, settings, memory, skills, and other global state. When set, project-level `.qwen/` directories are unaffected. An empty string is treated as unset. |
|
||||
| `QWEN_RUNTIME_DIR` | Overrides the runtime output directory (conversations, logs, todos). When unset, defaults to the `QWEN_HOME` directory. | Use this to separate ephemeral runtime data from persistent config. Useful when `QWEN_HOME` is on a shared/slow filesystem. |
|
||||
| `QWEN_TELEMETRY_ENABLED` | Set to `true` or `1` to enable telemetry. Any other value is treated as disabling it. | Overrides the `telemetry.enabled` setting. |
|
||||
| `QWEN_TELEMETRY_TARGET` | Sets an informational label for the telemetry destination (`local` or `gcp`). Does not control routing; use `QWEN_TELEMETRY_OTLP_ENDPOINT` or `QWEN_TELEMETRY_OUTFILE` to configure where data is sent. | Overrides the `telemetry.target` setting. |
|
||||
| `QWEN_TELEMETRY_OTLP_ENDPOINT` | Sets the OTLP endpoint for telemetry. | Overrides the `telemetry.otlpEndpoint` setting. |
|
||||
| `QWEN_TELEMETRY_OTLP_PROTOCOL` | Sets the OTLP protocol (`grpc` or `http`). | Overrides the `telemetry.otlpProtocol` setting. |
|
||||
| `QWEN_TELEMETRY_LOG_PROMPTS` | Set to `true` or `1` to enable or disable logging of user prompts. Any other value is treated as disabling it. | Overrides the `telemetry.logPrompts` setting. |
|
||||
| `QWEN_TELEMETRY_INCLUDE_SENSITIVE_SPAN_ATTRIBUTES` | Set to `true` or `1` to attach verbatim user prompts, system prompts, tool I/O, and model responses to native OTel span attributes (and keep `prompt` / `function_args` / `response_text` on log-to-span bridge spans). Any other value disables it. | Overrides the `telemetry.includeSensitiveSpanAttributes` setting. ⚠️ Streams sensitive data to your OTLP backend. |
|
||||
| `QWEN_TELEMETRY_SENSITIVE_SPAN_ATTRIBUTE_MAX_LENGTH` | Sets the maximum JavaScript string length for each sensitive native OTel span attribute content payload. Must be a positive integer no greater than `104857600` (100 MiB). | Overrides the `telemetry.sensitiveSpanAttributeMaxLength` setting. Default is `1048576` (1 MiB); lower it if your collector or backend rejects large span attributes. |
|
||||
| `QWEN_TELEMETRY_OUTFILE` | Sets the file path to write telemetry to. When set, overrides OTLP export. | Overrides the `telemetry.outfile` setting. |
|
||||
| `QWEN_SANDBOX` | Alternative to the `sandbox` setting in `settings.json`. | Accepts `true`, `false`, `docker`, `podman`, or a custom command string. |
|
||||
| `QWEN_SANDBOX_IMAGE` | Overrides sandbox image selection for Docker/Podman. | Takes precedence over `tools.sandboxImage`. |
|
||||
| `SEATBELT_PROFILE` | (macOS specific) Switches the Seatbelt (`sandbox-exec`) profile on macOS. | `permissive-open`: (Default) Restricts writes to the project folder (and a few other folders, see `packages/cli/src/utils/sandbox-macos-permissive-open.sb`) but allows other operations. `strict`: Uses a strict profile that declines operations by default. `<profile_name>`: Uses a custom profile. To define a custom profile, create a file named `sandbox-macos-<profile_name>.sb` in your project's `.qwen/` directory (e.g., `my-project/.qwen/sandbox-macos-custom.sb`). |
|
||||
| `DEBUG` or `DEBUG_MODE` | (often used by underlying libraries or the CLI itself) Set to `true` or `1` to enable verbose debug logging, which can be helpful for troubleshooting. | **Note:** These variables are automatically excluded from project `.env` files by default to prevent interference with the CLI behavior. Use `.qwen/.env` files if you need to set these for Qwen Code specifically. |
|
||||
| `NO_COLOR` | Set to any value to disable all color output in the CLI. | |
|
||||
| `FORCE_HYPERLINK` | Override the OSC 8 clickable-link detection in the markdown renderer. Set to `1` (or any non-zero integer, or empty string) to force-enable; set to `0` or a non-numeric value such as `false` / `off` to force-disable. Honors `NO_COLOR` / `QWEN_DISABLE_HYPERLINKS` opt-outs above it. | Use this to opt into OSC 8 inside `tmux` / GNU `screen` (auto-detection refuses by default because the host terminal's capabilities are hidden behind the multiplexer). Requires `set -g allow-passthrough on` on tmux 3.3+. Also enables Hyper, which isn't auto-detected. |
|
||||
| `QWEN_DISABLE_HYPERLINKS` | Set to `1` to hard-disable OSC 8 clickable hyperlinks in the markdown renderer even on terminals that auto-detect as capable. | Useful when a terminal advertises support but breaks on long URLs, or when piping output through an intermediary that mangles escape sequences. The renderer falls back to plain `label (url)` rendering. |
|
||||
| `CLI_TITLE` | Set to a string to customize the title of the CLI. | |
|
||||
| `CODE_ASSIST_ENDPOINT` | Specifies the endpoint for the code assist server. | This is useful for development and testing. |
|
||||
| `QWEN_CODE_MAX_OUTPUT_TOKENS` | Overrides the default maximum output tokens per response. When not set, Qwen Code defaults to the model's declared output limit and, if a response is truncated, automatically escalates (64K floor) and recovers across turns. Set this to a specific value (e.g., `16000`) to use a fixed limit instead — useful for capacity-constrained self-hosted backends that want a lower per-request slot reservation. | Takes precedence over the model-limit default but is overridden by `samplingParams.max_tokens` in settings. Disables automatic escalation when set. Example: `export QWEN_CODE_MAX_OUTPUT_TOKENS=16000` |
|
||||
| `QWEN_CODE_UNATTENDED_RETRY` | Set to `true` or `1` to enable persistent retry mode. When enabled, transient API capacity errors (HTTP 429 Rate Limit and 529 Overloaded) are retried indefinitely with exponential backoff (capped at 5 minutes per retry) and heartbeat keepalives every 30 seconds on stderr. | Designed for CI/CD pipelines and background automation where long-running tasks should survive temporary API outages. Must be set explicitly — `CI=true` alone does **not** activate this mode. See [Headless Mode](../features/headless#persistent-retry-mode) for details. Example: `export QWEN_CODE_UNATTENDED_RETRY=1` |
|
||||
| `QWEN_CODE_PROFILE_STARTUP` | Set to `1` to enable startup performance profiling. Writes a JSON timing report to `~/.qwen/startup-perf/` with per-phase durations. | Only active inside the sandbox child process (or with `QWEN_CODE_PROFILE_STARTUP_OUTER=1`). Zero overhead when not set. Example: `export QWEN_CODE_PROFILE_STARTUP=1` |
|
||||
| `QWEN_CODE_PROFILE_STARTUP_OUTER` | Set to `1` together with `QWEN_CODE_PROFILE_STARTUP=1` to also collect a startup profile in the outer (pre-sandbox) process. Outer-process reports get an `outer-` filename prefix to keep them distinct from the sandbox child's report. | Off by default — only the sandbox child collects, to avoid duplicate reports. Useful for local development where the cli isn't relaunched into a sandbox. |
|
||||
| `QWEN_CODE_PROFILE_STARTUP_NO_HEAP` | Set to `1` together with `QWEN_CODE_PROFILE_STARTUP=1` to skip the per-checkpoint `process.memoryUsage()` snapshots. Useful when measuring the profiler's own Heisenberg overhead. | Off by default. Heap snapshots cost ~50 µs each (well below 1% of total startup) so most users should leave this alone. |
|
||||
| `QWEN_CODE_LEGACY_MCP_BLOCKING` | Set to `1` to restore the pre-progressive-MCP behavior where `Config.initialize()` waits synchronously for every configured MCP server's discover handshake before returning. | Off by default. Modern qwen-code lets MCP servers come online in the background while the UI is already interactive; the model sees each batch of new tools within ~16 ms of the server settling. This flag is kept as a rollback escape hatch for ≥ 1 release. Example: `export QWEN_CODE_LEGACY_MCP_BLOCKING=1` |
|
||||
|
||||
When both user-level `.env` files define the same variable, the Qwen-specific
|
||||
file wins: `<QWEN_HOME>/.env` (or `~/.qwen/.env` when `QWEN_HOME` is unset) is
|
||||
|
|
|
|||
|
|
@ -1597,7 +1597,7 @@ describe('AnthropicContentGenerator', () => {
|
|||
);
|
||||
});
|
||||
|
||||
it('uses conservative default when max_tokens is not explicitly configured', async () => {
|
||||
it('uses model default when max_tokens is not explicitly configured', async () => {
|
||||
const { AnthropicContentGenerator } = await importGenerator();
|
||||
anthropicState.createImpl.mockResolvedValue({
|
||||
id: 'anthropic-1',
|
||||
|
|
@ -1625,7 +1625,7 @@ describe('AnthropicContentGenerator', () => {
|
|||
const [anthropicRequest] =
|
||||
anthropicState.lastCreateArgs as AnthropicCreateArgs;
|
||||
expect(anthropicRequest).toEqual(
|
||||
expect.objectContaining({ max_tokens: 8000 }),
|
||||
expect.objectContaining({ max_tokens: 65536 }),
|
||||
);
|
||||
});
|
||||
|
||||
|
|
@ -1660,7 +1660,7 @@ describe('AnthropicContentGenerator', () => {
|
|||
const [anthropicRequest] =
|
||||
anthropicState.lastCreateArgs as AnthropicCreateArgs;
|
||||
expect(anthropicRequest).toEqual(
|
||||
expect.objectContaining({ max_tokens: 8000 }),
|
||||
expect.objectContaining({ max_tokens: 65536 }),
|
||||
);
|
||||
}
|
||||
});
|
||||
|
|
@ -1759,7 +1759,7 @@ describe('AnthropicContentGenerator', () => {
|
|||
const [anthropicRequest] =
|
||||
anthropicState.lastCreateArgs as AnthropicCreateArgs;
|
||||
expect(anthropicRequest).toEqual(
|
||||
expect.objectContaining({ max_tokens: 8000 }),
|
||||
expect.objectContaining({ max_tokens: 65536 }),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -38,7 +38,6 @@ import { createDebugLogger } from '../../utils/debugLogger.js';
|
|||
import { runtimeDiagnostics } from '../../utils/runtimeDiagnostics.js';
|
||||
import {
|
||||
tokenLimit,
|
||||
CAPPED_DEFAULT_MAX_TOKENS,
|
||||
hasExplicitOutputLimit,
|
||||
parsePositiveIntegerEnvValue,
|
||||
} from '../tokenLimits.js';
|
||||
|
|
@ -594,7 +593,7 @@ export class AnthropicContentGenerator implements ContentGenerator {
|
|||
? Math.min(userMaxTokens, modelLimit)
|
||||
: userMaxTokens;
|
||||
} else {
|
||||
// No explicit user config — check env var, then use capped default.
|
||||
// No explicit user config — check env var, then use the model limit.
|
||||
const envMaxTokens = parsePositiveIntegerEnvValue(
|
||||
process.env['QWEN_CODE_MAX_OUTPUT_TOKENS'],
|
||||
);
|
||||
|
|
@ -603,7 +602,7 @@ export class AnthropicContentGenerator implements ContentGenerator {
|
|||
? Math.min(envMaxTokens, modelLimit)
|
||||
: envMaxTokens;
|
||||
} else {
|
||||
maxTokens = Math.min(modelLimit, CAPPED_DEFAULT_MAX_TOKENS);
|
||||
maxTokens = modelLimit;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -5416,6 +5416,60 @@ describe('CoreToolScheduler truncated output protection', () => {
|
|||
);
|
||||
}
|
||||
});
|
||||
|
||||
it('should inject retry loop directive after repeated truncated write_file rejections', async () => {
|
||||
const writeFileConfig = {
|
||||
getProjectRoot: () => '/tmp',
|
||||
getTargetDir: () => '/tmp',
|
||||
getFileSystemService: () => ({
|
||||
readTextFile: vi.fn(),
|
||||
writeTextFile: vi.fn(),
|
||||
}),
|
||||
getDefaultFileEncoding: () => undefined,
|
||||
setApprovalMode: vi.fn(),
|
||||
} as unknown as Config;
|
||||
const writeFileTool = new WriteFileTool(writeFileConfig);
|
||||
const { scheduler, onAllToolCallsComplete } = createTruncationTestScheduler(
|
||||
writeFileTool,
|
||||
[WriteFileTool.Name],
|
||||
);
|
||||
|
||||
const messages: string[] = [];
|
||||
|
||||
for (let i = 1; i <= 3; i++) {
|
||||
await scheduler.schedule(
|
||||
[
|
||||
{
|
||||
callId: `truncated-write-file-${i}`,
|
||||
name: WriteFileTool.Name,
|
||||
args: { file_path: '/tmp/test.txt', content: 'partial' },
|
||||
isClientInitiated: false,
|
||||
prompt_id: `prompt-id-write-file-truncated-${i}`,
|
||||
wasOutputTruncated: true,
|
||||
},
|
||||
],
|
||||
new AbortController().signal,
|
||||
);
|
||||
|
||||
await vi.waitFor(() => {
|
||||
expect(onAllToolCallsComplete).toHaveBeenCalledTimes(i);
|
||||
});
|
||||
|
||||
const completedCalls = onAllToolCallsComplete.mock.calls.at(-1)?.[0] as
|
||||
| ToolCall[]
|
||||
| undefined;
|
||||
const completedCall = completedCalls?.[0];
|
||||
expect(completedCall?.status).toBe('error');
|
||||
if (completedCall?.status === 'error') {
|
||||
messages.push(completedCall.response.error?.message ?? '');
|
||||
}
|
||||
}
|
||||
|
||||
expect(messages[0]).toContain('truncated due to max_tokens limit');
|
||||
expect(messages[0]).not.toContain('RETRY LOOP DETECTED');
|
||||
expect(messages[1]).not.toContain('RETRY LOOP DETECTED');
|
||||
expect(messages[2]).toContain('RETRY LOOP DETECTED');
|
||||
});
|
||||
});
|
||||
|
||||
describe('CoreToolScheduler Sequential Execution', () => {
|
||||
|
|
|
|||
|
|
@ -795,6 +795,12 @@ const RETRY_LOOP_STOP_DIRECTIVE =
|
|||
'fundamentally different approach. If you cannot resolve the validation error, explain the issue to the user ' +
|
||||
'instead of retrying.';
|
||||
|
||||
/** Directive injected when a truncated file-modifying call repeats. */
|
||||
const TRUNCATION_RETRY_LOOP_DIRECTIVE =
|
||||
'\n\n⚠️ RETRY LOOP DETECTED: The same truncated file write has been rejected multiple times. ' +
|
||||
'STOP resending the same large content. Either split it into smaller write_file + incremental edit calls, ' +
|
||||
'or explain to the user that the content is too large to write safely in one call.';
|
||||
|
||||
const createErrorResponse = (
|
||||
request: ToolCallRequestInfo,
|
||||
error: Error,
|
||||
|
|
@ -1773,6 +1779,28 @@ export class CoreToolScheduler {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Increments the retry counter for a (tool, errorMessage) pair and prunes any
|
||||
* other error counters for the same tool, so a different failure on the same
|
||||
* tool restarts the count rather than tripping the loop threshold. Shared by
|
||||
* the truncated-Edit rejection path and the schema-validation failure path so
|
||||
* both feed the same RETRY LOOP DETECTED detector.
|
||||
*/
|
||||
private recordRetryableToolError(
|
||||
toolName: string,
|
||||
errorMessage: string,
|
||||
): number {
|
||||
const errorKey = `${toolName}:${errorMessage}`;
|
||||
const count = (this.validationRetryCounts.get(errorKey) ?? 0) + 1;
|
||||
for (const key of this.validationRetryCounts.keys()) {
|
||||
if (key.startsWith(`${toolName}:`) && key !== errorKey) {
|
||||
this.validationRetryCounts.delete(key);
|
||||
}
|
||||
}
|
||||
this.validationRetryCounts.set(errorKey, count);
|
||||
return count;
|
||||
}
|
||||
|
||||
private async _schedule(
|
||||
request: ToolCallRequestInfo | ToolCallRequestInfo[],
|
||||
signal: AbortSignal,
|
||||
|
|
@ -1879,7 +1907,15 @@ export class CoreToolScheduler {
|
|||
// Reject file-modifying calls when truncated to prevent
|
||||
// writing incomplete content, even if params failed schema validation.
|
||||
if (reqInfo.wasOutputTruncated && toolInstance.kind === Kind.Edit) {
|
||||
const truncationError = new Error(TRUNCATION_EDIT_REJECTION);
|
||||
const count = this.recordRetryableToolError(
|
||||
reqInfo.name,
|
||||
TRUNCATION_EDIT_REJECTION,
|
||||
);
|
||||
const truncationError = new Error(
|
||||
count >= VALIDATION_RETRY_LOOP_THRESHOLD
|
||||
? `${TRUNCATION_EDIT_REJECTION}${TRUNCATION_RETRY_LOOP_DIRECTIVE}`
|
||||
: TRUNCATION_EDIT_REJECTION,
|
||||
);
|
||||
newToolCalls.push({
|
||||
status: 'error',
|
||||
request: reqInfo,
|
||||
|
|
@ -1910,14 +1946,10 @@ export class CoreToolScheduler {
|
|||
// Track validation retry for loop detection. Counts accumulate per
|
||||
// (tool, error message) pair so a different validation mistake on
|
||||
// the same tool starts fresh rather than tripping the threshold.
|
||||
const errorKey = `${reqInfo.name}:${invocationOrError.message}`;
|
||||
const count = (this.validationRetryCounts.get(errorKey) ?? 0) + 1;
|
||||
for (const key of this.validationRetryCounts.keys()) {
|
||||
if (key.startsWith(`${reqInfo.name}:`) && key !== errorKey) {
|
||||
this.validationRetryCounts.delete(key);
|
||||
}
|
||||
}
|
||||
this.validationRetryCounts.set(errorKey, count);
|
||||
const count = this.recordRetryableToolError(
|
||||
reqInfo.name,
|
||||
invocationOrError.message,
|
||||
);
|
||||
|
||||
const finalError =
|
||||
count >= VALIDATION_RETRY_LOOP_THRESHOLD
|
||||
|
|
|
|||
|
|
@ -1973,9 +1973,8 @@ export class GeminiChat {
|
|||
cgConfig?.maxRetries ?? RATE_LIMIT_RETRY_OPTIONS.maxRetries;
|
||||
const extraRetryErrorCodes = cgConfig?.retryErrorCodes;
|
||||
|
||||
// Max output tokens escalation: when no user/env override is set,
|
||||
// the capped default (8K) is used. If the model hits MAX_TOKENS,
|
||||
// retry once with escalated limit (64K).
|
||||
// Max output tokens escalation: when no user/env override is set and
|
||||
// the model hits MAX_TOKENS, retry once with the escalated limit.
|
||||
let maxTokensEscalated = false;
|
||||
const hasUserMaxTokensOverride =
|
||||
(cgConfig?.samplingParams?.max_tokens !== undefined &&
|
||||
|
|
@ -2361,12 +2360,11 @@ export class GeminiChat {
|
|||
}
|
||||
}
|
||||
|
||||
// Max output tokens escalation: if the retry loop succeeded with
|
||||
// the capped default (8K) but hit MAX_TOKENS, retry once at the
|
||||
// model's full output limit. This ensures models with large output
|
||||
// limits (e.g., 128K for Claude Opus, GPT-5) are fully utilized,
|
||||
// while using ESCALATED_MAX_TOKENS (64K) as a floor for unknown
|
||||
// models.
|
||||
// Max output tokens escalation: if the retry loop succeeded but hit
|
||||
// MAX_TOKENS, retry once at the model's full output limit. This ensures
|
||||
// models with large output limits (e.g., 128K for Claude Opus, GPT-5)
|
||||
// are fully utilized, while using ESCALATED_MAX_TOKENS (64K) as a floor
|
||||
// for unknown models.
|
||||
// Placed outside the retry loop so that any errors from the
|
||||
// escalated stream propagate directly (not caught by retry logic).
|
||||
const requestedMaxOutputTokens = params.config?.maxOutputTokens;
|
||||
|
|
@ -2388,7 +2386,7 @@ export class GeminiChat {
|
|||
maxTokensEscalated = true;
|
||||
const startingLimitLabel =
|
||||
requestedMaxOutputTokens === undefined
|
||||
? 'capped default'
|
||||
? 'default max_tokens'
|
||||
: `${requestedMaxOutputTokens} tokens`;
|
||||
debugLogger.info(
|
||||
`Output truncated at ${startingLimitLabel}. Escalating to ${escalatedLimit} tokens.`,
|
||||
|
|
|
|||
|
|
@ -1251,7 +1251,7 @@ describe('DashScopeOpenAICompatibleProvider', () => {
|
|||
expect(result.max_tokens).toBe(1000); // Should remain unchanged
|
||||
});
|
||||
|
||||
it('should set conservative max_tokens default when not present in request', () => {
|
||||
it('should set model max_tokens default when not present in request', () => {
|
||||
const request: OpenAI.Chat.ChatCompletionCreateParams = {
|
||||
model: 'qwen3-max',
|
||||
messages: [{ role: 'user', content: 'Hello' }],
|
||||
|
|
@ -1260,12 +1260,10 @@ describe('DashScopeOpenAICompatibleProvider', () => {
|
|||
|
||||
const result = provider.buildRequest(request, 'test-prompt-id');
|
||||
|
||||
// Should set capped default (min of model limit and CAPPED_DEFAULT_MAX_TOKENS)
|
||||
// qwen3-max has 32K output limit, so min(32K, 8K) = 8K
|
||||
expect(result.max_tokens).toBe(8000);
|
||||
expect(result.max_tokens).toBe(32768);
|
||||
});
|
||||
|
||||
it('should set conservative max_tokens when null is provided', () => {
|
||||
it('should set model max_tokens when null is provided', () => {
|
||||
const request: OpenAI.Chat.ChatCompletionCreateParams = {
|
||||
model: 'qwen3-max',
|
||||
messages: [{ role: 'user', content: 'Hello' }],
|
||||
|
|
@ -1274,8 +1272,7 @@ describe('DashScopeOpenAICompatibleProvider', () => {
|
|||
|
||||
const result = provider.buildRequest(request, 'test-prompt-id');
|
||||
|
||||
// null is treated as not configured, so set capped default: min(32K, 8K) = 8K
|
||||
expect(result.max_tokens).toBe(8000);
|
||||
expect(result.max_tokens).toBe(32768);
|
||||
});
|
||||
|
||||
it('should respect user max_tokens for unknown models', () => {
|
||||
|
|
|
|||
|
|
@ -205,9 +205,7 @@ export class DashScopeOpenAICompatibleProvider extends DefaultOpenAICompatiblePr
|
|||
tools = updatedTools;
|
||||
}
|
||||
|
||||
// Apply output token limits using parent class logic
|
||||
// Uses capped default (min of model limit and CAPPED_DEFAULT_MAX_TOKENS=8K)
|
||||
// Requests hitting the cap get one clean retry at 64K (geminiChat.ts)
|
||||
// Apply output token limits using parent class logic.
|
||||
const requestWithTokenLimits = this.applyOutputTokenLimit(request);
|
||||
|
||||
const extraBody = this.contentGeneratorConfig.extra_body;
|
||||
|
|
|
|||
|
|
@ -206,7 +206,7 @@ describe('DefaultOpenAICompatibleProvider', () => {
|
|||
expect(result).not.toBe(originalRequest); // Should be a new object
|
||||
});
|
||||
|
||||
it('should set conservative max_tokens default when not configured', () => {
|
||||
it('should set model max_tokens default when not configured', () => {
|
||||
const requestWithoutMaxTokens: OpenAI.Chat.ChatCompletionCreateParams = {
|
||||
model: 'gpt-4',
|
||||
messages: [{ role: 'user', content: 'Hello' }],
|
||||
|
|
@ -217,9 +217,7 @@ describe('DefaultOpenAICompatibleProvider', () => {
|
|||
'prompt-id',
|
||||
);
|
||||
|
||||
// Should set capped default (min of model limit and CAPPED_DEFAULT_MAX_TOKENS)
|
||||
// GPT-4 has 16K output limit, so min(16K, 8K) = 8K
|
||||
expect(result.max_tokens).toBe(8000);
|
||||
expect(result.max_tokens).toBe(16384);
|
||||
});
|
||||
|
||||
it('should ignore malformed QWEN_CODE_MAX_OUTPUT_TOKENS values', () => {
|
||||
|
|
@ -233,7 +231,7 @@ describe('DefaultOpenAICompatibleProvider', () => {
|
|||
|
||||
const result = provider.buildRequest(request, 'prompt-id');
|
||||
|
||||
expect(result.max_tokens).toBe(8000);
|
||||
expect(result.max_tokens).toBe(16384);
|
||||
}
|
||||
});
|
||||
|
||||
|
|
@ -263,8 +261,7 @@ describe('DefaultOpenAICompatibleProvider', () => {
|
|||
expect(result.max_tokens).toBe(100000);
|
||||
});
|
||||
|
||||
it('should use capped default for unknown models when max_tokens not configured', () => {
|
||||
// Unknown models without user config: use CAPPED_DEFAULT_MAX_TOKENS
|
||||
it('should use default output limit for unknown models when max_tokens not configured', () => {
|
||||
const request: OpenAI.Chat.ChatCompletionCreateParams = {
|
||||
model: 'custom-deployment-alias',
|
||||
messages: [{ role: 'user', content: 'Hello' }],
|
||||
|
|
@ -272,8 +269,7 @@ describe('DefaultOpenAICompatibleProvider', () => {
|
|||
|
||||
const result = provider.buildRequest(request, 'prompt-id');
|
||||
|
||||
// Uses capped default (8K)
|
||||
expect(result.max_tokens).toBe(8000);
|
||||
expect(result.max_tokens).toBe(32000);
|
||||
});
|
||||
|
||||
it('should cap max_tokens for known models to avoid API errors', () => {
|
||||
|
|
@ -299,8 +295,7 @@ describe('DefaultOpenAICompatibleProvider', () => {
|
|||
|
||||
const result = provider.buildRequest(request, 'prompt-id');
|
||||
|
||||
// GPT-4 has 16K output limit, capped default is 8K: min(16K, 8K) = 8K
|
||||
expect(result.max_tokens).toBe(8000);
|
||||
expect(result.max_tokens).toBe(16384);
|
||||
});
|
||||
|
||||
it('should preserve all sampling parameters', () => {
|
||||
|
|
@ -340,10 +335,9 @@ describe('DefaultOpenAICompatibleProvider', () => {
|
|||
|
||||
const result = provider.buildRequest(minimalRequest, 'prompt-id');
|
||||
|
||||
// Should set conservative max_tokens default
|
||||
expect(result.model).toBe('gpt-4');
|
||||
expect(result.messages).toEqual(minimalRequest.messages);
|
||||
expect(result.max_tokens).toBe(8000); // GPT-4 has 16K limit, min(16K, 8K) = 8K
|
||||
expect(result.max_tokens).toBe(16384);
|
||||
});
|
||||
|
||||
it('should not inject max_tokens when samplingParams is set without it (e.g. GPT-5 / o-series)', () => {
|
||||
|
|
@ -395,11 +389,10 @@ describe('DefaultOpenAICompatibleProvider', () => {
|
|||
|
||||
const result = provider.buildRequest(streamingRequest, 'prompt-id');
|
||||
|
||||
// Should set conservative max_tokens default while preserving stream
|
||||
expect(result.model).toBe('gpt-4');
|
||||
expect(result.messages).toEqual(streamingRequest.messages);
|
||||
expect(result.stream).toBe(true);
|
||||
expect(result.max_tokens).toBe(8000); // GPT-4 has 16K limit, min(16K, 8K) = 8K
|
||||
expect(result.max_tokens).toBe(16384);
|
||||
});
|
||||
|
||||
it('should not modify the original request object', () => {
|
||||
|
|
@ -443,7 +436,7 @@ describe('DefaultOpenAICompatibleProvider', () => {
|
|||
|
||||
expect(result).toEqual({
|
||||
...originalRequest,
|
||||
max_tokens: 8000, // GPT-4 has 16K limit, min(16K, 8K) = 8K
|
||||
max_tokens: 16384,
|
||||
custom_param: 'custom_value',
|
||||
nested: { key: 'value' },
|
||||
});
|
||||
|
|
@ -458,11 +451,10 @@ describe('DefaultOpenAICompatibleProvider', () => {
|
|||
|
||||
const result = provider.buildRequest(originalRequest, 'prompt-id');
|
||||
|
||||
// Should preserve original params and set conservative max_tokens default
|
||||
expect(result.model).toBe('gpt-4');
|
||||
expect(result.messages).toEqual(originalRequest.messages);
|
||||
expect(result.temperature).toBe(0.7);
|
||||
expect(result.max_tokens).toBe(8000); // GPT-4 has 16K limit, min(16K, 8K) = 8K
|
||||
expect(result.max_tokens).toBe(16384);
|
||||
expect(result).not.toHaveProperty('custom_param');
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -7,7 +7,6 @@ import type { OpenAICompatibleProvider } from './types.js';
|
|||
import { buildRuntimeFetchOptions } from '../../../utils/runtimeFetchOptions.js';
|
||||
import {
|
||||
tokenLimit,
|
||||
CAPPED_DEFAULT_MAX_TOKENS,
|
||||
hasExplicitOutputLimit,
|
||||
parsePositiveIntegerEnvValue,
|
||||
} from '../../tokenLimits.js';
|
||||
|
|
@ -140,16 +139,15 @@ export class DefaultOpenAICompatibleProvider
|
|||
* configured value entirely (backend may support larger limits)
|
||||
* 2. If user didn't configure max_tokens:
|
||||
* - Check QWEN_CODE_MAX_OUTPUT_TOKENS env var first
|
||||
* - Otherwise use min(modelLimit, CAPPED_DEFAULT_MAX_TOKENS=8K)
|
||||
* - Requests hitting the 8K cap get one clean retry at 64K (geminiChat.ts)
|
||||
* - Otherwise use the model's output limit
|
||||
* 3. If model has no specific limit (tokenLimit returns default):
|
||||
* - Still apply CAPPED_DEFAULT_MAX_TOKENS as safeguard
|
||||
* - Use DEFAULT_OUTPUT_TOKEN_LIMIT
|
||||
*
|
||||
* Examples:
|
||||
* - User sets 4K, known model limit 64K → uses 4K (respects user preference)
|
||||
* - User sets 100K, known model limit 64K → uses 64K (capped to avoid API error)
|
||||
* - User sets 100K, unknown model → uses 100K (respects user, backend may support it)
|
||||
* - User not set, model limit 64K → uses 8K (capped default for slot optimization)
|
||||
* - User not set, model limit 64K → uses 64K
|
||||
* - User not set, model limit 4K → uses 4K (model limit is lower)
|
||||
* - User not set, env QWEN_CODE_MAX_OUTPUT_TOKENS=16000 -> uses 16K
|
||||
*
|
||||
|
|
@ -185,9 +183,7 @@ export class DefaultOpenAICompatibleProvider
|
|||
effectiveMaxTokens = userMaxTokens;
|
||||
}
|
||||
} else {
|
||||
// No explicit user config — check env var, then use capped default.
|
||||
// Capped default (8K) reduces GPU slot over-reservation by ~4×.
|
||||
// Requests hitting the cap get one clean retry at 64K (geminiChat.ts).
|
||||
// No explicit user config — check env var, then use the model limit.
|
||||
const envMaxTokens = parsePositiveIntegerEnvValue(
|
||||
process.env['QWEN_CODE_MAX_OUTPUT_TOKENS'],
|
||||
);
|
||||
|
|
@ -196,7 +192,7 @@ export class DefaultOpenAICompatibleProvider
|
|||
? Math.min(envMaxTokens, modelLimit)
|
||||
: envMaxTokens;
|
||||
} else {
|
||||
effectiveMaxTokens = Math.min(modelLimit, CAPPED_DEFAULT_MAX_TOKENS);
|
||||
effectiveMaxTokens = modelLimit;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -11,11 +11,6 @@ export type TokenLimitType = 'input' | 'output';
|
|||
export const DEFAULT_TOKEN_LIMIT: TokenCount = 131_072; // 128K (power-of-two)
|
||||
export const DEFAULT_OUTPUT_TOKEN_LIMIT: TokenCount = 32_000; // 32K tokens
|
||||
|
||||
// Capped default for slot-reservation optimization. 99% of outputs are under 5K
|
||||
// tokens, so 32K defaults over-reserve 4-6× slot capacity. With the cap
|
||||
// enabled, <1% of requests hit the limit; those get one clean retry at 64K
|
||||
// (see geminiChat.ts max_output_tokens escalation).
|
||||
export const CAPPED_DEFAULT_MAX_TOKENS: TokenCount = 8_000;
|
||||
export const ESCALATED_MAX_TOKENS: TokenCount = 64_000;
|
||||
|
||||
export function parsePositiveIntegerEnvValue(
|
||||
|
|
@ -194,7 +189,7 @@ const OUTPUT_PATTERNS: Array<[RegExp, TokenCount]> = [
|
|||
// Alibaba / Qwen
|
||||
[/^qwen3\.\d/, LIMITS['64k']],
|
||||
[/^coder-model$/, LIMITS['64k']],
|
||||
[/^qwen/, LIMITS['32k']], // Qwen fallback (VL, turbo, plus, etc.): 8K
|
||||
[/^qwen/, LIMITS['32k']], // Qwen fallback (VL, turbo, plus, etc.): 32K
|
||||
|
||||
// DeepSeek
|
||||
[/^deepseek-v4/, LIMITS['384k']], // DeepSeek V4 (flash, pro): 384K
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue