From 13b0f35e324d79e7d53a9842a997d93b56a51470 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 8 Jul 2026 22:16:26 +0000 Subject: [PATCH] docs: fix model-provider config shape and refresh feature/setting drift Audit findings against the current codebase: - model-providers.md, auth.md: the documented modelProviders shape used the reverted `{ protocol, models }` wrapper. The canonical shape is a bare `ModelConfig[]` array per provider id (a wrapped entry in a migrated settings file is silently skipped). Update all examples and prose, document the separate top-level `providerProtocol` map for custom provider ids, and correct the unknown-key behavior. - settings.md: correct the default for `model.chatCompression.screenshotTriggerThreshold` (20, not 50). - commands.md: add the missing `/reload-plugins` command and note that `/dream` and `/forget` are registered only when managed auto-memory is available. - Add a Computer Use feature page (on-by-default desktop automation via the cua-driver native driver) and wire it into the features nav and the qc-helper doc index. --- docs/users/configuration/auth.md | 95 ++-- docs/users/configuration/model-providers.md | 438 +++++++++--------- docs/users/configuration/settings.md | 2 +- docs/users/features/_meta.ts | 1 + docs/users/features/commands.md | 3 +- docs/users/features/computer-use.md | 77 +++ .../src/skills/bundled/qc-helper/SKILL.md | 1 + 7 files changed, 344 insertions(+), 273 deletions(-) create mode 100644 docs/users/features/computer-use.md diff --git a/docs/users/configuration/auth.md b/docs/users/configuration/auth.md index 2f0384d0e6..158d6c48e7 100644 --- a/docs/users/configuration/auth.md +++ b/docs/users/configuration/auth.md @@ -75,18 +75,15 @@ If you prefer to skip the interactive `/auth` flow, add the following to `~/.qwe ```json { "modelProviders": { - "openai": { - "protocol": "openai", - "models": [ - { - "id": "qwen3-coder-plus", - "name": "qwen3-coder-plus (Coding Plan)", - "baseUrl": "https://coding.dashscope.aliyuncs.com/v1", - "description": "qwen3-coder-plus from Alibaba Cloud Coding Plan", - "envKey": "BAILIAN_CODING_PLAN_API_KEY" - } - ] - } + "openai": [ + { + "id": "qwen3-coder-plus", + "name": "qwen3-coder-plus (Coding Plan)", + "baseUrl": "https://coding.dashscope.aliyuncs.com/v1", + "description": "qwen3-coder-plus from Alibaba Cloud Coding Plan", + "envKey": "BAILIAN_CODING_PLAN_API_KEY" + } + ] }, "env": { "BAILIAN_CODING_PLAN_API_KEY": "sk-sp-xxxxxxxxx" @@ -117,18 +114,15 @@ The simplest way to get started with API Key authentication is to put everything ```json { "modelProviders": { - "openai": { - "protocol": "openai", - "models": [ - { - "id": "qwen3-coder-plus", - "name": "qwen3-coder-plus", - "baseUrl": "https://dashscope.aliyuncs.com/compatible-mode/v1", - "description": "Qwen3-Coder via Dashscope", - "envKey": "DASHSCOPE_API_KEY" - } - ] - } + "openai": [ + { + "id": "qwen3-coder-plus", + "name": "qwen3-coder-plus", + "baseUrl": "https://dashscope.aliyuncs.com/compatible-mode/v1", + "description": "Qwen3-Coder via Dashscope", + "envKey": "DASHSCOPE_API_KEY" + } + ] }, "env": { "DASHSCOPE_API_KEY": "sk-xxxxxxxxxxxxx" @@ -183,37 +177,28 @@ Edit `~/.qwen/settings.json` (create it if it doesn't exist). You can mix multip ```json { "modelProviders": { - "openai": { - "protocol": "openai", - "models": [ - { - "id": "gpt-4o", - "name": "GPT-4o", - "envKey": "OPENAI_API_KEY", - "baseUrl": "https://api.openai.com/v1" - } - ] - }, - "anthropic": { - "protocol": "anthropic", - "models": [ - { - "id": "claude-sonnet-4-20250514", - "name": "Claude Sonnet 4", - "envKey": "ANTHROPIC_API_KEY" - } - ] - }, - "gemini": { - "protocol": "gemini", - "models": [ - { - "id": "gemini-2.5-pro", - "name": "Gemini 2.5 Pro", - "envKey": "GEMINI_API_KEY" - } - ] - } + "openai": [ + { + "id": "gpt-4o", + "name": "GPT-4o", + "envKey": "OPENAI_API_KEY", + "baseUrl": "https://api.openai.com/v1" + } + ], + "anthropic": [ + { + "id": "claude-sonnet-4-20250514", + "name": "Claude Sonnet 4", + "envKey": "ANTHROPIC_API_KEY" + } + ], + "gemini": [ + { + "id": "gemini-2.5-pro", + "name": "Gemini 2.5 Pro", + "envKey": "GEMINI_API_KEY" + } + ] } } ``` diff --git a/docs/users/configuration/model-providers.md b/docs/users/configuration/model-providers.md index 71f79508e6..e2c02dc831 100644 --- a/docs/users/configuration/model-providers.md +++ b/docs/users/configuration/model-providers.md @@ -4,7 +4,11 @@ Qwen Code allows you to configure multiple model providers through the `modelPro ## Overview -Use `modelProviders` to declare models per auth type that the `/model` picker can switch between. Keys must be valid auth types (`openai`, `anthropic`, `gemini`, etc.). Each auth type maps to a `ProviderConfig` object with a `protocol` field and a `models` field (the array of model definitions). Each entry in `models` requires an `id`; `envKey` is **optional and recommended** (when omitted, it falls back to the auth type's default env key, e.g. `OPENAI_API_KEY` for `openai`), with optional `name`, `description`, `baseUrl`, and `generationConfig`. Credentials are never persisted in settings; the runtime reads them from `process.env[envKey]`. Qwen OAuth models remain hard-coded and cannot be overridden. +Use `modelProviders` to declare models per provider id that the `/model` picker can switch between. Each key is a provider id and its value is **an array of model definitions** (`ModelConfig[]`). For built-in providers the key must be a valid auth type (`openai`, `anthropic`, `gemini`, `vertex-ai`); a custom provider id (e.g. `idealab`) is allowed as long as you map it to a protocol via the top-level [`providerProtocol`](#custom-provider-ids-providerprotocol) setting. Each model entry requires an `id`; `envKey` is **optional and recommended** (when omitted, it falls back to the auth type's default env key, e.g. `OPENAI_API_KEY` for `openai`), with optional `name`, `description`, `baseUrl`, and `generationConfig`. Credentials are never persisted in settings; the runtime reads them from `process.env[envKey]`. Qwen OAuth models remain hard-coded and cannot be overridden. + +> [!note] +> +> Earlier previews wrapped each provider's models in a `{ "protocol": ..., "models": [...] }` object. That shape has been reverted — the current value is the bare `ModelConfig[]` array shown throughout this page. A wrapped entry in an already-migrated (`$version: 4`) settings file is silently skipped, so update any old configs to the array form. > [!note] > @@ -31,7 +35,30 @@ The `modelProviders` object keys must be valid `authType` values. Currently supp | `vertex-ai` | Google Vertex AI (uses the `gemini` protocol and the `@google/genai` SDK in Vertex AI mode; selecting it sets `GOOGLE_GENAI_USE_VERTEXAI=true`) | > [!warning] -> If an unknown auth type key is used (e.g., a typo like `"openai-custom"`), a non-empty key is accepted as-is as its own auth-type group, but it will not map to a known protocol — so its models won't work as intended and won't behave correctly in the `/model` picker. Only blank (empty or whitespace-only) keys are skipped. Always use one of the supported auth type values listed above. +> A provider id that is neither a built-in protocol nor mapped via `providerProtocol` (e.g. a typo like `"openai-custom"`) cannot be routed, so its whole entry is **skipped** with a warning — its models simply won't appear in the `/model` picker. Use one of the supported auth type values above for built-in providers, or add a [`providerProtocol`](#custom-provider-ids-providerprotocol) mapping for a custom id. + +### Custom provider ids (`providerProtocol`) + +Built-in provider ids (`openai`, `gemini`, `anthropic`, `vertex-ai`, `qwen-oauth`) are routed to their SDK protocol automatically. To use a **custom** provider id — for example to group several OpenAI-compatible endpoints under a friendlier name — declare it under `modelProviders` and map it to a built-in protocol with the top-level `providerProtocol` setting: + +```json +{ + "modelProviders": { + "idealab": [ + { + "id": "my-model", + "envKey": "IDEALAB_API_KEY", + "baseUrl": "https://idealab.example.com/v1" + } + ] + }, + "providerProtocol": { + "idealab": "openai" + } +} +``` + +Without a matching `providerProtocol` entry, a custom provider id is skipped (see the warning above). ### SDKs Used for API Requests @@ -58,79 +85,76 @@ This auth type supports not only OpenAI's official API but also any OpenAI-compa "REQUESTY_API_KEY": "sk-your-actual-requesty-key-here" }, "modelProviders": { - "openai": { - "protocol": "openai", - "models": [ - { - "id": "gpt-4o", - "name": "GPT-4o", - "envKey": "OPENAI_API_KEY", - "baseUrl": "https://api.openai.com/v1", - "generationConfig": { - "timeout": 60000, - "maxRetries": 3, - "enableCacheControl": true, - "contextWindowSize": 128000, - "modalities": { - "image": true - }, - "customHeaders": { - "X-Client-Request-ID": "req-123" - }, - "extra_body": { - "enable_thinking": true, - "service_tier": "priority" - }, - "samplingParams": { - "temperature": 0.2, - "top_p": 0.8, - "max_tokens": 4096, - "presence_penalty": 0.1, - "frequency_penalty": 0.1 - } - } - }, - { - "id": "gpt-4o-mini", - "name": "GPT-4o Mini", - "envKey": "OPENAI_API_KEY", - "baseUrl": "https://api.openai.com/v1", - "generationConfig": { - "timeout": 30000, - "samplingParams": { - "temperature": 0.5, - "max_tokens": 2048 - } - } - }, - { - "id": "openai/gpt-4o", - "name": "GPT-4o (via OpenRouter)", - "envKey": "OPENROUTER_API_KEY", - "baseUrl": "https://openrouter.ai/api/v1", - "generationConfig": { - "timeout": 120000, - "maxRetries": 3, - "samplingParams": { - "temperature": 0.7 - } - } - }, - { - "id": "openai/gpt-4o-mini", - "name": "GPT-4o Mini (via Requesty)", - "envKey": "REQUESTY_API_KEY", - "baseUrl": "https://router.requesty.ai/v1", - "generationConfig": { - "timeout": 120000, - "maxRetries": 3, - "samplingParams": { - "temperature": 0.7 - } + "openai": [ + { + "id": "gpt-4o", + "name": "GPT-4o", + "envKey": "OPENAI_API_KEY", + "baseUrl": "https://api.openai.com/v1", + "generationConfig": { + "timeout": 60000, + "maxRetries": 3, + "enableCacheControl": true, + "contextWindowSize": 128000, + "modalities": { + "image": true + }, + "customHeaders": { + "X-Client-Request-ID": "req-123" + }, + "extra_body": { + "enable_thinking": true, + "service_tier": "priority" + }, + "samplingParams": { + "temperature": 0.2, + "top_p": 0.8, + "max_tokens": 4096, + "presence_penalty": 0.1, + "frequency_penalty": 0.1 } } - ] - } + }, + { + "id": "gpt-4o-mini", + "name": "GPT-4o Mini", + "envKey": "OPENAI_API_KEY", + "baseUrl": "https://api.openai.com/v1", + "generationConfig": { + "timeout": 30000, + "samplingParams": { + "temperature": 0.5, + "max_tokens": 2048 + } + } + }, + { + "id": "openai/gpt-4o", + "name": "GPT-4o (via OpenRouter)", + "envKey": "OPENROUTER_API_KEY", + "baseUrl": "https://openrouter.ai/api/v1", + "generationConfig": { + "timeout": 120000, + "maxRetries": 3, + "samplingParams": { + "temperature": 0.7 + } + } + }, + { + "id": "openai/gpt-4o-mini", + "name": "GPT-4o Mini (via Requesty)", + "envKey": "REQUESTY_API_KEY", + "baseUrl": "https://router.requesty.ai/v1", + "generationConfig": { + "timeout": 120000, + "maxRetries": 3, + "samplingParams": { + "temperature": 0.7 + } + } + } + ] } } ``` @@ -143,40 +167,37 @@ This auth type supports not only OpenAI's official API but also any OpenAI-compa "ANTHROPIC_API_KEY": "sk-ant-your-actual-anthropic-key-here" }, "modelProviders": { - "anthropic": { - "protocol": "anthropic", - "models": [ - { - "id": "claude-3-5-sonnet", - "name": "Claude 3.5 Sonnet", - "envKey": "ANTHROPIC_API_KEY", - "baseUrl": "https://api.anthropic.com/v1", - "generationConfig": { - "timeout": 120000, - "maxRetries": 3, - "contextWindowSize": 200000, - "samplingParams": { - "temperature": 0.7, - "max_tokens": 8192, - "top_p": 0.9 - } - } - }, - { - "id": "claude-3-opus", - "name": "Claude 3 Opus", - "envKey": "ANTHROPIC_API_KEY", - "baseUrl": "https://api.anthropic.com/v1", - "generationConfig": { - "timeout": 180000, - "samplingParams": { - "temperature": 0.3, - "max_tokens": 4096 - } + "anthropic": [ + { + "id": "claude-3-5-sonnet", + "name": "Claude 3.5 Sonnet", + "envKey": "ANTHROPIC_API_KEY", + "baseUrl": "https://api.anthropic.com/v1", + "generationConfig": { + "timeout": 120000, + "maxRetries": 3, + "contextWindowSize": 200000, + "samplingParams": { + "temperature": 0.7, + "max_tokens": 8192, + "top_p": 0.9 } } - ] - } + }, + { + "id": "claude-3-opus", + "name": "Claude 3 Opus", + "envKey": "ANTHROPIC_API_KEY", + "baseUrl": "https://api.anthropic.com/v1", + "generationConfig": { + "timeout": 180000, + "samplingParams": { + "temperature": 0.3, + "max_tokens": 4096 + } + } + } + ] } } ``` @@ -189,32 +210,29 @@ This auth type supports not only OpenAI's official API but also any OpenAI-compa "GEMINI_API_KEY": "AIza-your-actual-gemini-key-here" }, "modelProviders": { - "gemini": { - "protocol": "gemini", - "models": [ - { - "id": "gemini-2.0-flash", - "name": "Gemini 2.0 Flash", - "envKey": "GEMINI_API_KEY", - "baseUrl": "https://generativelanguage.googleapis.com", - "capabilities": { - "vision": true - }, - "generationConfig": { - "timeout": 60000, - "maxRetries": 2, - "contextWindowSize": 1000000, - "schemaCompliance": "auto", - "samplingParams": { - "temperature": 0.4, - "top_p": 0.95, - "max_tokens": 8192, - "top_k": 40 - } + "gemini": [ + { + "id": "gemini-2.0-flash", + "name": "Gemini 2.0 Flash", + "envKey": "GEMINI_API_KEY", + "baseUrl": "https://generativelanguage.googleapis.com", + "capabilities": { + "vision": true + }, + "generationConfig": { + "timeout": 60000, + "maxRetries": 2, + "contextWindowSize": 1000000, + "schemaCompliance": "auto", + "samplingParams": { + "temperature": 0.4, + "top_p": 0.95, + "max_tokens": 8192, + "top_k": 40 } } - ] - } + } + ] } } ``` @@ -231,54 +249,51 @@ Most local inference servers (vLLM, Ollama, LM Studio, etc.) provide an OpenAI-c "LMSTUDIO_API_KEY": "lm-studio" }, "modelProviders": { - "openai": { - "protocol": "openai", - "models": [ - { - "id": "qwen2.5-7b", - "name": "Qwen2.5 7B (Ollama)", - "envKey": "OLLAMA_API_KEY", - "baseUrl": "http://localhost:11434/v1", - "generationConfig": { - "timeout": 300000, - "maxRetries": 1, - "contextWindowSize": 32768, - "samplingParams": { - "temperature": 0.7, - "top_p": 0.9, - "max_tokens": 4096 - } - } - }, - { - "id": "llama-3.1-8b", - "name": "Llama 3.1 8B (vLLM)", - "envKey": "VLLM_API_KEY", - "baseUrl": "http://localhost:8000/v1", - "generationConfig": { - "timeout": 120000, - "maxRetries": 2, - "contextWindowSize": 128000, - "samplingParams": { - "temperature": 0.6, - "max_tokens": 8192 - } - } - }, - { - "id": "local-model", - "name": "Local Model (LM Studio)", - "envKey": "LMSTUDIO_API_KEY", - "baseUrl": "http://localhost:1234/v1", - "generationConfig": { - "timeout": 60000, - "samplingParams": { - "temperature": 0.5 - } + "openai": [ + { + "id": "qwen2.5-7b", + "name": "Qwen2.5 7B (Ollama)", + "envKey": "OLLAMA_API_KEY", + "baseUrl": "http://localhost:11434/v1", + "generationConfig": { + "timeout": 300000, + "maxRetries": 1, + "contextWindowSize": 32768, + "samplingParams": { + "temperature": 0.7, + "top_p": 0.9, + "max_tokens": 4096 } } - ] - } + }, + { + "id": "llama-3.1-8b", + "name": "Llama 3.1 8B (vLLM)", + "envKey": "VLLM_API_KEY", + "baseUrl": "http://localhost:8000/v1", + "generationConfig": { + "timeout": 120000, + "maxRetries": 2, + "contextWindowSize": 128000, + "samplingParams": { + "temperature": 0.6, + "max_tokens": 8192 + } + } + }, + { + "id": "local-model", + "name": "Local Model (LM Studio)", + "envKey": "LMSTUDIO_API_KEY", + "baseUrl": "http://localhost:1234/v1", + "generationConfig": { + "timeout": 60000, + "samplingParams": { + "temperature": 0.5 + } + } + } + ] } } ``` @@ -394,18 +409,15 @@ If you prefer to manually configure Coding Plan models, you can add them to your ```json { "modelProviders": { - "openai": { - "protocol": "openai", - "models": [ - { - "id": "qwen3-coder-plus", - "name": "qwen3-coder-plus", - "description": "Qwen3-Coder via Alibaba Cloud Coding Plan", - "envKey": "YOUR_CUSTOM_ENV_KEY", - "baseUrl": "https://coding.dashscope.aliyuncs.com/v1" - } - ] - } + "openai": [ + { + "id": "qwen3-coder-plus", + "name": "qwen3-coder-plus", + "description": "Qwen3-Coder via Alibaba Cloud Coding Plan", + "envKey": "YOUR_CUSTOM_ENV_KEY", + "baseUrl": "https://coding.dashscope.aliyuncs.com/v1" + } + ] } } ``` @@ -503,17 +515,14 @@ The following fields are treated as atomic objects - provider values completely // modelProviders configuration { "modelProviders": { - "openai": { - "protocol": "openai", - "models": [{ - "id": "gpt-4o", - "envKey": "OPENAI_API_KEY", - "generationConfig": { - "timeout": 60000, - "samplingParams": { "temperature": 0.2 } - } - }] - } + "openai": [{ + "id": "gpt-4o", + "envKey": "OPENAI_API_KEY", + "generationConfig": { + "timeout": 60000, + "samplingParams": { "temperature": 0.2 } + } + }] } } ``` @@ -539,25 +548,22 @@ The optional `reasoning` field under `generationConfig` controls how aggressivel ```jsonc { "modelProviders": { - "openai": { - "protocol": "openai", - "models": [ - { - "id": "deepseek-v4-pro", - "name": "DeepSeek V4 Pro", - "baseUrl": "https://api.deepseek.com/v1", - "envKey": "DEEPSEEK_API_KEY", - "generationConfig": { - // The four-tier scale: - // 'low' | 'medium' — server-mapped to 'high' on DeepSeek - // 'high' — default reasoning intensity - // 'max' — DeepSeek-specific extra-strong tier - // Or set `false` to disable reasoning entirely. - "reasoning": { "effort": "max" }, - }, + "openai": [ + { + "id": "deepseek-v4-pro", + "name": "DeepSeek V4 Pro", + "baseUrl": "https://api.deepseek.com/v1", + "envKey": "DEEPSEEK_API_KEY", + "generationConfig": { + // The four-tier scale: + // 'low' | 'medium' — server-mapped to 'high' on DeepSeek + // 'high' — default reasoning intensity + // 'max' — DeepSeek-specific extra-strong tier + // Or set `false` to disable reasoning entirely. + "reasoning": { "effort": "max" }, }, - ], - }, + }, + ], }, } ``` diff --git a/docs/users/configuration/settings.md b/docs/users/configuration/settings.md index f1e762d464..948d2af6b0 100644 --- a/docs/users/configuration/settings.md +++ b/docs/users/configuration/settings.md @@ -172,7 +172,7 @@ Settings are organized into categories. Most settings should be placed within th | `model.chatCompression.maxRecentFilesToRetain` | number | Number of most-recently-touched files whose current content is restored (embedded if small, otherwise referenced by path) into history after auto-compaction. `0` restores none. Env override: `QWEN_COMPACT_MAX_RECENT_FILES`. | `5` | | `model.chatCompression.maxRecentImagesToRetain` | number | Number of most-recent images (tool screenshots / user pastes) restored into history after auto-compaction. `0` restores none. Env override: `QWEN_COMPACT_MAX_RECENT_IMAGES`. | `3` | | `model.chatCompression.enableScreenshotTrigger` | boolean | When `true`, auto-compaction also fires once the number of tool-returned images accumulated in history reaches `screenshotTriggerThreshold`, independent of token usage — aimed at computer-use sessions where frequent screenshots dilute model attention. Counts only images returned inside tool results, not user-pasted images. Env override: `QWEN_COMPACT_SCREENSHOT_TRIGGER` (`1`/`true`/`0`/`false`). | `true` | -| `model.chatCompression.screenshotTriggerThreshold` | number | Tool-returned image count at or above which the screenshot trigger fires (only when `enableScreenshotTrigger`). Compaction resets the count — surviving images are re-embedded as top-level parts, which the trigger doesn't count — so it won't immediately re-fire. Env override: `QWEN_COMPACT_SCREENSHOT_THRESHOLD`. | `50` | +| `model.chatCompression.screenshotTriggerThreshold` | number | Tool-returned image count at or above which the screenshot trigger fires (only when `enableScreenshotTrigger`). Compaction resets the count — surviving images are re-embedded as top-level parts, which the trigger doesn't count — so it won't immediately re-fire. Env override: `QWEN_COMPACT_SCREENSHOT_THRESHOLD`. | `20` | | `model.skipNextSpeakerCheck` | boolean | Skip the next speaker check. | `true` | | `model.skipLoopDetection` | boolean | Disables streaming loop detection checks. Defaults to `true` (loop detection is skipped) to avoid false positives interrupting legitimate workflows. Set to `false` to re-enable streaming loop detection — useful as a guardrail in headless / non-interactive runs where stuck repetition can otherwise waste budget. | `true` | | `model.maxToolCallsPerTurn` | number | Hard cap on tool calls within a single turn (one model turn plus its tool-result continuations; blocking Stop-hook continuations such as `/goal` iterations start a fresh budget). Always-on circuit breaker against runaway turns, independent of `model.skipLoopDetection`; the pattern-based loop detectors fire long before this cap, which only bounds total volume. Set to `0` or a negative value to disable the cap. Choosing "Disable loop detection for this session" in the loop-detected dialog also suppresses it for the rest of the session. | `100` | diff --git a/docs/users/features/_meta.ts b/docs/users/features/_meta.ts index d4cafb29a4..247d04525c 100644 --- a/docs/users/features/_meta.ts +++ b/docs/users/features/_meta.ts @@ -16,6 +16,7 @@ export default { worktree: 'Worktrees', mcp: 'MCP', lsp: 'LSP (Language Server Protocol)', + 'computer-use': 'Computer Use', 'token-caching': 'Token Caching', sandbox: 'Sandboxing', language: 'i18n', diff --git a/docs/users/features/commands.md b/docs/users/features/commands.md index 47cf3868e7..f15c7fb1d7 100644 --- a/docs/users/features/commands.md +++ b/docs/users/features/commands.md @@ -104,6 +104,7 @@ Commands for managing AI tools and models. | `/forget` | Remove matching entries from auto-memory | `/forget ` | | `/dream` | Manually run auto-memory consolidation | `/dream` | | `/hooks` | Manage Qwen Code hooks | `/hooks`, `/hooks list` | +| `/reload-plugins` | Reload extension changes (commands, skills, agents, hooks, MCP/LSP servers) from disk | `/reload-plugins` | | `/permissions` | Manage permission rules | `/permissions` | | `/agents` | Manage subagents | `/agents manage`, `/agents create` | | `/arena` | Manage Arena sessions | `/arena start`, `/arena stop`, `/arena status`, `/arena select` (alias `choose`) | @@ -123,7 +124,7 @@ Commands for managing AI tools and models. > [!note] > -> `/workflows`, `/lsp`, and `/trust` are registered only when their feature is enabled — via the `QWEN_CODE_ENABLE_WORKFLOWS=1` env var, the `--experimental-lsp` CLI flag, and the `security.folderTrust.enabled` setting respectively. When disabled they won't appear and will report an unknown command. +> `/workflows`, `/lsp`, and `/trust` are registered only when their feature is enabled — via the `QWEN_CODE_ENABLE_WORKFLOWS=1` env var, the `--experimental-lsp` CLI flag, and the `security.folderTrust.enabled` setting respectively. When disabled they won't appear and will report an unknown command. Similarly, `/dream` and `/forget` are registered only when managed auto-memory is available; without it they won't appear. ### 1.5 Built-in Skills diff --git a/docs/users/features/computer-use.md b/docs/users/features/computer-use.md new file mode 100644 index 0000000000..3938e6d675 --- /dev/null +++ b/docs/users/features/computer-use.md @@ -0,0 +1,77 @@ +# Computer Use + +Qwen Code ships built-in **Computer Use** tools that let the agent drive your desktop — clicking, typing, scrolling, launching apps, reading window contents, and taking screenshots. This turns Qwen Code into a general desktop automation agent, not just a coding assistant confined to the terminal. + +Computer Use is powered by the [`cua-driver`](https://github.com/trycua/cua) native driver. The tools are registered as deferred (lazy-loaded) built-ins under the `computer_use__` prefix, so they only cost prompt space once the model actually reaches for them. + +> [!warning] +> +> Computer Use gives the agent control of your mouse, keyboard, and windows, and lets it read the contents of your screen. Only use it with trusted prompts and, where possible, in a sandboxed or disposable environment. The action tools (click, type, drag, etc.) go through the normal [approval flow](./approval-mode.md); read-only tools such as listing windows may run without a prompt. + +## Enabling and disabling + +Computer Use is **enabled by default**. The `computer_use__*` tools are registered automatically at startup. + +To disable it entirely — which also prevents the native driver from being downloaded or spawned — set `tools.computerUse.enabled` to `false` in your `settings.json`: + +```jsonc +{ + "tools": { + "computerUse": { + "enabled": false + } + } +} +``` + +This setting requires a restart to take effect. + +## First run and the native driver + +The first time the agent invokes a Computer Use tool, Qwen Code downloads a pinned, signed `cua-driver` binary (~20 MB) into `~/.qwen/computer-use/` and spawns it as a local process. Prebuilt binaries are published for macOS (Apple Silicon and Intel), Linux (x86_64), and Windows (x86_64). + +### macOS permissions + +On macOS, desktop automation requires two system permissions: + +- **Accessibility** — to read window/UI state and synthesize input +- **Screen Recording** — to capture screenshots + +On first use the driver walks you through granting these via the standard macOS system dialogs. The agent can also check permission status on demand (the `check_permissions` tool). Because macOS attributes permission grants to the *responsible* process, grants may need to be given to the terminal or IDE that launched Qwen Code. + +## What the agent can do + +The full `cua-driver` tool surface is exposed. Highlights: + +| Category | Tools (a selection) | +| ------------- | ----------------------------------------------------------------------------------- | +| Mouse | `click`, `double_click`, `right_click`, `drag`, `move_cursor`, `scroll` | +| Keyboard | `type_text`, `press_key`, `hotkey` | +| Windows / UI | `list_windows`, `get_window_state`, `get_accessibility_tree`, `set_value`, `zoom` | +| Apps | `launch_app`, `list_apps`, `bring_to_front`, `kill_app` | +| Browser pages | `page` (execute JavaScript, read text, query the DOM, click elements) | +| Screenshots | `get_window_state` (captures a PNG), `page` | +| Recording | `start_recording`, `stop_recording`, `replay_trajectory` (record/replay a session) | +| Sessions | `start_session`, `end_session`, agent-cursor overlay controls | + +Element-addressed actions are preferred over raw pixel coordinates: `get_window_state` returns a Markdown rendering of a window's accessibility tree with a stable `element_index` for each actionable element, which the input tools can target directly. + +Support is most complete on macOS; some tools are platform-specific (for example, `bring_to_front` is Windows-only, and `launch_app` targets macOS apps). + +## Configuration + +All Computer Use settings live under `tools.computerUse` in `settings.json`. See the [Settings reference](../configuration/settings.md) for the authoritative list. + +| Setting | Type | Default | Description | +| ---------------------------------- | ------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `tools.computerUse.enabled` | boolean | `true` | Register the `computer_use__*` tools. When `false`, the driver is never downloaded or spawned. | +| `tools.computerUse.maxImageDimension` | number | `-1` | Longest-edge pixel cap for screenshots. `-1` keeps the driver's default (1568); `0` disables resizing (full resolution); a positive value caps the longest edge. Lower caps cut vision-token cost. Env override: `QWEN_COMPUTER_USE_MAX_IMAGE_DIMENSION`. | +| `tools.computerUse.idleTimeoutMs` | number | `300000` | Milliseconds to keep the driver process alive after the last `computer_use__*` call (default 5 minutes). `0` keeps it running until Qwen Code exits. | + +All three settings require a restart to take effect. + +## See also + +- [Approval Mode](./approval-mode.md) — how tool executions are gated +- [Sandboxing](./sandbox.md) — isolating what tools can touch +- [Settings reference](../configuration/settings.md) — the full `tools.computerUse.*` schema diff --git a/packages/core/src/skills/bundled/qc-helper/SKILL.md b/packages/core/src/skills/bundled/qc-helper/SKILL.md index 0188db1d10..e379888679 100644 --- a/packages/core/src/skills/bundled/qc-helper/SKILL.md +++ b/packages/core/src/skills/bundled/qc-helper/SKILL.md @@ -60,6 +60,7 @@ Use this index to locate the right document for the user's question. Load only t | Slash commands | `docs/features/commands.md` | | Headless / non-interactive mode | `docs/features/headless.md` | | LSP integration | `docs/features/lsp.md` | +| Computer Use (desktop automation) | `docs/features/computer-use.md` | | Token caching | `docs/features/token-caching.md` | | Language / i18n | `docs/features/language.md` | | Arena mode | `docs/features/arena.md` |