vrr/openclaw
2
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-07-10 00:11:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 22

docs: rewrite published docs grounded in current source (#100142)

Browse source 
Source-grounded rewrite of 529 published docs pages with per-unit information-loss verification: 1,713 factual corrections cited to src/**, generated surfaces regenerated, frontmatter titles preserved for i18n, release notes pages untouched. All docs gates green.

Closes #100141
This commit is contained in:
Peter Steinberger 2026-07-05 00:32:47 -04:00 • committed by GitHub
parent e069cb26f7
commit f7d7148cf0
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
531 changed files with 31763 additions and 40928 deletions
Download patch file Download diff file Expand all files Collapse all files

105
docs/nodes/audio.md
View file

@ -5,36 +5,36 @@ read_when:
title: "Audio and voice notes"
---
## What works
## What it does
- **Media understanding (audio)**: If audio understanding is enabled (or auto-detected), OpenClaw:
1. Locates the first audio attachment (local path or URL) and downloads it if needed.
2. Enforces `maxBytes` before sending to each model entry.
3. Runs the first eligible model entry in order (provider or CLI).
4. If it fails or skips (size/timeout), it tries the next entry.
5. On success, it replaces `Body` with an `[Audio]` block and sets `{{Transcript}}`.
- **Command parsing**: When transcription succeeds, `CommandBody`/`RawBody` are set to the transcript so slash commands still work.
- **Verbose logging**: In `--verbose`, we log when transcription runs and when it replaces the body.
When audio understanding is enabled (or auto-detected), OpenClaw:
1. Locates the first audio attachment (local path or URL) and downloads it if needed.
2. Enforces `maxBytes` before sending to each model entry.
3. Runs the first eligible model entry in order (provider or CLI); if an entry fails or skips (size/timeout), the next entry is tried.
4. On success, replaces `Body` with an `[Audio]` block and sets `{{Transcript}}`.
When transcription succeeds, `CommandBody`/`RawBody` are also set to the transcript so slash commands still work. With `--verbose`, logs show when transcription runs and when it replaces the body.
## Auto-detection (default)
If you **don't configure models** and `tools.media.audio.enabled` is **not** set to `false`,
OpenClaw auto-detects in this order and stops at the first working option:
If you have not configured models and `tools.media.audio.enabled` is not `false`, OpenClaw auto-detects in this order and stops at the first working option:
1. **Active reply model** when its provider supports audio understanding.
2. **Local CLIs** (if installed)
- `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with encoder/decoder/joiner/tokens)
- `whisper-cli` (from `whisper-cpp`; uses `WHISPER_CPP_MODEL` or the bundled tiny model)
1. **Active reply model**, when its provider supports audio understanding.
2. **Configured provider auth** — any `models.providers.*` entry with auth available for a provider that supports audio transcription. This is checked before local CLIs, so a configured API key always wins over a local binary on `PATH`.
Provider priority when multiple are configured: Groq, OpenAI, xAI, Deepgram, Google, SenseAudio, ElevenLabs, Mistral.
3. **Local CLIs** (only if no provider auth resolved), checked in this order:
- `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with `tokens.txt`, `encoder.onnx`, `decoder.onnx`, and `joiner.onnx`)
- `whisper-cli` (from `whisper-cpp`; uses `WHISPER_CPP_MODEL` or a bundled tiny model)
- `whisper` (Python CLI; downloads models automatically)
3. **Provider auth**
- Configured `models.providers.*` entries that support audio are tried first
- Provider fallback order: OpenAI → Groq → xAI → Deepgram → Google → SenseAudio → ElevenLabs → Mistral
As of 2026-05-22, Gemini CLI auto-detect is no longer supported for media understanding. Google is transitioning Gemini CLI users to Antigravity CLI; audio should use local or provider transcription, while image/video CLI fallback should move to Antigravity CLI (`agy`).
Gemini CLI auto-detect for media understanding was replaced by a sandboxed Antigravity CLI (`agy`) fallback for image/video; audio does not use a CLI fallback beyond the local binaries above.
To disable auto-detection, set `tools.media.audio.enabled: false`.
To customize, set `tools.media.audio.models`.
Note: Binary detection is best-effort across macOS/Linux/Windows; ensure the CLI is on `PATH` (we expand `~`), or set an explicit CLI model with a full command path.
To disable auto-detection, set `tools.media.audio.enabled: false`. To customize, set `tools.media.audio.models`.
<Note>
Binary detection is best-effort across macOS/Linux/Windows. Make sure the CLI is on `PATH` (`~` is expanded), or set an explicit CLI model with a full command path.
</Note>
## Config examples
@ -48,7 +48,7 @@ Note: Binary detection is best-effort across macOS/Linux/Windows; ensure the CLI
enabled: true,
maxBytes: 20971520,
models: [
{ provider: "openai", model: "gpt-4o-mini-transcribe" },
{ provider: "openai", model: "gpt-4o-transcribe" },
{
type: "cli",
command: "whisper",
@ -74,7 +74,7 @@ Note: Binary detection is best-effort across macOS/Linux/Windows; ensure the CLI
default: "allow",
rules: [{ action: "deny", match: { chatType: "group" } }],
},
models: [{ provider: "openai", model: "gpt-4o-mini-transcribe" }],
models: [{ provider: "openai", model: "gpt-4o-transcribe" }],
},
},
},
@ -136,7 +136,7 @@ Note: Binary detection is best-effort across macOS/Linux/Windows; ensure the CLI
enabled: true,
echoTranscript: true, // default is false
echoFormat: '📝 "{transcript}"', // optional, supports {transcript}
models: [{ provider: "openai", model: "gpt-4o-mini-transcribe" }],
models: [{ provider: "openai", model: "gpt-4o-transcribe" }],
},
},
},
@ -147,51 +147,42 @@ Note: Binary detection is best-effort across macOS/Linux/Windows; ensure the CLI
- Provider auth follows the standard model auth order (auth profiles, env vars, `models.providers.*.apiKey`).
- Groq setup details: [Groq](/providers/groq).
- Deepgram picks up `DEEPGRAM_API_KEY` when `provider: "deepgram"` is used.
- Deepgram setup details: [Deepgram (audio transcription)](/providers/deepgram).
- Deepgram picks up `DEEPGRAM_API_KEY` when `provider: "deepgram"` is used. Setup details: [Deepgram](/providers/deepgram).
- Mistral setup details: [Mistral](/providers/mistral).
- SenseAudio picks up `SENSEAUDIO_API_KEY` when `provider: "senseaudio"` is used.
- SenseAudio setup details: [SenseAudio](/providers/senseaudio).
- SenseAudio picks up `SENSEAUDIO_API_KEY` when `provider: "senseaudio"` is used. Setup details: [SenseAudio](/providers/senseaudio).
- Audio providers can override `baseUrl`, `headers`, and `providerOptions` via `tools.media.audio`.
- Default size cap is 20MB (`tools.media.audio.maxBytes`). Oversize audio is skipped for that model and the next entry is tried.
- Tiny/empty audio files below 1024 bytes are skipped before provider/CLI transcription.
- Default `maxChars` for audio is **unset** (full transcript). Set `tools.media.audio.maxChars` or per-entry `maxChars` to trim output.
- OpenAI auto default is `gpt-4o-mini-transcribe`; set `model: "gpt-4o-transcribe"` for higher accuracy.
- Use `tools.media.audio.attachments` to process multiple voice notes (`mode: "all"` + `maxAttachments`).
- Audio files below 1024 bytes are skipped before provider/CLI transcription.
- Default `maxChars` for audio is **unset** (full transcript). Set `tools.media.audio.maxChars` or a per-entry `maxChars` to trim output.
- OpenAI auto-detect default is `gpt-4o-transcribe`; set `model: "gpt-4o-mini-transcribe"` for a cheaper/faster option.
- Use `tools.media.audio.attachments` to process multiple voice notes (`mode: "all"` plus `maxAttachments`, default 1).
- Transcript is available to templates as `{{Transcript}}`.
- `tools.media.audio.echoTranscript` is off by default; enable it to send transcript confirmation back to the originating chat before agent processing.
- `tools.media.audio.echoFormat` customizes the echo text (placeholder: `{transcript}`).
- CLI stdout is capped (5MB); keep CLI output concise.
- CLI `args` should use `{{MediaPath}}` for the local audio file path. Run `openclaw doctor --fix` to migrate deprecated `{input}` placeholders from older `audio.transcription.command` configs.
- `tools.media.audio.echoTranscript` is off by default; enable it to send a transcript confirmation back to the originating chat before agent processing.
- `tools.media.audio.echoFormat` customizes the echo text (placeholder: `{transcript}`; default `📝 "{transcript}"`).
- CLI stdout is capped at 5MB; keep CLI output concise.
- CLI `args` should use `{{MediaPath}}` for the local audio file path. Run `openclaw doctor --fix` to migrate deprecated `{input}` placeholders from older `audio.transcription.command` configs (retired key: `audio.transcription`, replaced by `tools.media.audio.models`).
### Proxy environment support
Provider-based audio transcription honors standard outbound proxy env vars:
Provider-based audio transcription honors standard outbound proxy env vars, matching undici's `EnvHttpProxyAgent` semantics:
- `HTTPS_PROXY`
- `HTTP_PROXY`
- `ALL_PROXY`
- `https_proxy`
- `http_proxy`
- `all_proxy`
- `HTTPS_PROXY` / `https_proxy`
- `HTTP_PROXY` / `http_proxy`
- `ALL_PROXY` / `all_proxy`
If no proxy env vars are set, direct egress is used. If proxy config is malformed, OpenClaw logs a warning and falls back to direct fetch.
Lowercase variables take precedence over uppercase; `NO_PROXY`/`no_proxy` entries (hostnames, `*.suffix`, or `host:port`) bypass the proxy. If no proxy env vars are set, direct egress is used. If proxy setup fails (malformed URL), OpenClaw logs a warning and falls back to direct fetch.
## Mention detection in groups
When `requireMention: true` is set for a group chat, OpenClaw now transcribes audio **before** checking for mentions. This allows voice notes to be processed even when they contain mentions.
When `requireMention: true` is set for a group chat, OpenClaw transcribes audio **before** checking for mentions. This lets voice notes pass the mention gate even when the message has no text body.
**How it works:**
1. If a voice message has no text body and the group requires mentions, OpenClaw performs a "preflight" transcription.
2. The transcript is checked for mention patterns (e.g., `@BotName`, emoji triggers).
1. If a voice message has no text body and the group requires mentions, OpenClaw performs a preflight transcription of the first audio attachment.
2. The transcript is checked for mention patterns (for example `@BotName`, emoji triggers).
3. If a mention is found, the message proceeds through the full reply pipeline.
4. The transcript is used for mention detection so voice notes can pass the mention gate.
**Fallback behavior:**
- If transcription fails during preflight (timeout, API error, etc.), the message is processed based on text-only mention detection.
- This ensures that mixed messages (text + audio) are never incorrectly dropped.
**Fallback behavior:** if preflight transcription fails (timeout, API error, etc.), the message falls back to text-only mention detection so mixed messages (text + audio) are never dropped.
**Opt-out per Telegram group/topic:**
@ -199,15 +190,15 @@ When `requireMention: true` is set for a group chat, OpenClaw now transcribes au
- Set `channels.telegram.groups.<chatId>.topics.<threadId>.disableAudioPreflight` to override per-topic (`true` to skip, `false` to force-enable).
- Default is `false` (preflight enabled when mention-gated conditions match).
**Example:** A user sends a voice note saying "Hey @Claude, what's the weather?" in a Telegram group with `requireMention: true`. The voice note is transcribed, the mention is detected, and the agent replies.
**Example:** a user sends a voice note saying "Hey @Claude, what's the weather?" in a Telegram group with `requireMention: true`. The voice note is transcribed, the mention is detected, and the agent replies.
## Gotchas
- Scope rules use first-match wins. `chatType` is normalized to `direct`, `group`, or `room`.
- Ensure your CLI exits 0 and prints plain text; JSON needs to be massaged via `jq -r .text`.
- Scope rules use first-match-wins; `chatType` is normalized to `direct`, `group`, or `channel`.
- Ensure your CLI exits 0 and prints plain text; JSON output needs to be massaged via `jq -r .text`.
- For `parakeet-mlx`, if you pass `--output-dir`, OpenClaw reads `<output-dir>/<media-basename>.txt` when `--output-format` is `txt` (or omitted); non-`txt` output formats fall back to stdout parsing.
- Keep timeouts reasonable (`timeoutSeconds`, default 60s) to avoid blocking the reply queue.
- Preflight transcription only processes the **first** audio attachment for mention detection. Additional audio is processed during the main media understanding phase.
- Preflight transcription only processes the **first** audio attachment for mention detection. Additional audio attachments are processed during the main media-understanding phase.
## Related

112
docs/nodes/camera.md
View file

@ -6,56 +6,44 @@ read_when:
title: "Camera capture"
---
OpenClaw supports **camera capture** for agent workflows:
OpenClaw supports camera capture for agent workflows on paired **iOS**, **Android**, and **macOS** nodes: capture a photo (`jpg`) or a short video clip (`mp4`, with optional audio) via Gateway `node.invoke`.
- **iOS node** (paired via Gateway): capture a **photo** (`jpg`) or **short video clip** (`mp4`, with optional audio) via `node.invoke`.
- **Android node** (paired via Gateway): capture a **photo** (`jpg`) or **short video clip** (`mp4`, with optional audio) via `node.invoke`.
- **macOS app** (node via Gateway): capture a **photo** (`jpg`) or **short video clip** (`mp4`, with optional audio) via `node.invoke`.
All camera access is gated behind **user-controlled settings**.
All camera access is gated behind a user-controlled setting per platform.
## iOS node
### User setting (default on)
### iOS user setting
- iOS Settings tab → **Camera** → **Allow Camera** (`camera.enabled`)
- iOS Settings tab → **Camera** → **Allow Camera** (`camera.enabled`).
- Default: **on** (missing key is treated as enabled).
- When off: `camera.*` commands return `CAMERA_DISABLED`.
### Commands (via Gateway `node.invoke`)
### iOS commands (via Gateway `node.invoke`)
- `camera.list`
- Response payload:
- `devices`: array of `{ id, name, position, deviceType }`
- Response payload: `devices` — array of `{ id, name, position, deviceType }`.
- `camera.snap`
- Params:
- `facing`: `front|back` (default: `front`)
- `maxWidth`: number (optional; default `1600` on the iOS node)
- `quality`: `0..1` (optional; default `0.9`)
- `maxWidth`: number (optional; default `1600`)
- `quality`: `0..1` (optional; default `0.9`, clamped to `[0.05, 1.0]`)
- `format`: currently `jpg`
- `delayMs`: number (optional; default `0`)
- `delayMs`: number (optional; default `0`, internally capped at `10000`)
- `deviceId`: string (optional; from `camera.list`)
- Response payload:
- `format: "jpg"`
- `base64: "<...>"`
- `width`, `height`
- Payload guard: photos are recompressed to keep the base64 payload under 5 MB.
- Response payload: `format: "jpg"`, `base64`, `width`, `height`.
- Payload guard: photos are recompressed to keep the base64-encoded payload under 5MB.
- `camera.clip`
- Params:
- `facing`: `front|back` (default: `front`)
- `durationMs`: number (default `3000`, clamped to a max of `60000`)
- `durationMs`: number (default `3000`, clamped to `[250, 60000]`)
- `includeAudio`: boolean (default `true`)
- `format`: currently `mp4`
- `deviceId`: string (optional; from `camera.list`)
- Response payload:
- `format: "mp4"`
- `base64: "<...>"`
- `durationMs`
- `hasAudio`
- Response payload: `format: "mp4"`, `base64`, `durationMs`, `hasAudio`.
### Foreground requirement
### iOS foreground requirement
Like `canvas.*`, the iOS node only allows `camera.*` commands in the **foreground**. Background invocations return `NODE_BACKGROUND_UNAVAILABLE`.
@ -63,89 +51,83 @@ Like `canvas.*`, the iOS node only allows `camera.*` commands in the **foregroun
The easiest way to get media files is via the CLI helper, which writes decoded media to a temp file and prints the saved path.
Examples:
```bash
openclaw nodes camera snap --node <id> # default: both front + back (2 MEDIA lines)
openclaw nodes camera snap --node <id> # default: both front + back (2 MEDIA lines)
openclaw nodes camera snap --node <id> --facing front
openclaw nodes camera clip --node <id> --duration 3000
openclaw nodes camera clip --node <id> --no-audio
```
Notes:
- `nodes camera snap` defaults to **both** facings to give the agent both views.
- Output files are temporary (in the OS temp directory) unless you build your own wrapper.
`nodes camera snap` defaults to `--facing both`, capturing both front and back to give the agent both views; pass `--device-id` with a single explicit facing (`both` is rejected when `--device-id` is set). Output files are temporary (in the OS temp directory) unless you build your own wrapper.
## Android node
### Android user setting (default on)
### Android user setting
- Android Settings sheet → **Camera** → **Allow Camera** (`camera.enabled`)
- Default: **on** (missing key is treated as enabled).
- When off: `camera.*` commands return `CAMERA_DISABLED`.
- Android Settings sheet → **Camera** → **Allow Camera** (`camera.enabled`).
- **Fresh installs default to off.** Existing installs that predate this setting are migrated to **on** so upgrades do not silently lose previously working camera access.
- When off: `camera.*` commands return `CAMERA_DISABLED: enable Camera in Settings`.
### Permissions
- Android requires runtime permissions:
- `CAMERA` for both `camera.snap` and `camera.clip`.
- `RECORD_AUDIO` for `camera.clip` when `includeAudio=true`.
- `CAMERA` is required for both `camera.snap` and `camera.clip`; missing/denied permission returns `CAMERA_PERMISSION_REQUIRED`.
- `RECORD_AUDIO` is required for `camera.clip` when `includeAudio` is `true`; missing/denied permission returns `MIC_PERMISSION_REQUIRED`.
If permissions are missing, the app will prompt when possible; if denied, `camera.*` requests fail with a
`*_PERMISSION_REQUIRED` error.
The app prompts for runtime permissions when possible.
### Android foreground requirement
Like `canvas.*`, the Android node only allows `camera.*` commands in the **foreground**. Background invocations return `NODE_BACKGROUND_UNAVAILABLE`.
Like `canvas.*`, the Android node only allows `camera.*` commands in the **foreground**. Background invocations return `NODE_BACKGROUND_UNAVAILABLE: command requires foreground`.
### Android commands (via Gateway `node.invoke`)
- `camera.list`
- Response payload:
- `devices`: array of `{ id, name, position, deviceType }`
- Response payload: `devices` — array of `{ id, name, position, deviceType }`.
### Payload guard
- `camera.snap`
- Params: `facing` (`front|back`, default `front`), `quality` (default `0.95`, clamped to `[0.1, 1.0]`), `maxWidth` (default `1600`), `deviceId` (optional; unknown id fails with `INVALID_REQUEST`).
- Response payload: `format: "jpg"`, `base64`, `width`, `height`.
- Payload guard: recompressed to keep base64 under 5MB (same budget as iOS).
Photos are recompressed to keep the base64 payload under 5 MB.
- `camera.clip`
- Params: `facing` (default `front`), `durationMs` (default `3000`, clamped to `[200, 60000]`), `includeAudio` (default `true`), `deviceId` (optional).
- Response payload: `format: "mp4"`, `base64`, `durationMs`, `hasAudio`.
- Payload guard: raw MP4 is capped at 18MB before base64 encoding; oversize clips fail with `PAYLOAD_TOO_LARGE` (reduce `durationMs` and retry).
## macOS app
### User setting (default off)
### macOS user setting
The macOS companion app exposes a checkbox:
- **Settings → General → Allow Camera** (`openclaw.cameraEnabled`)
- Default: **off**
- When off: camera requests return "Camera disabled by user".
- **Settings → General → Allow Camera** (`openclaw.cameraEnabled`).
- Default: **off**.
- When off: camera requests return `CAMERA_DISABLED: enable Camera in Settings`.
### CLI helper (node invoke)
Use the main `openclaw` CLI to invoke camera commands on the macOS node.
Examples:
```bash
openclaw nodes camera list --node <id> # list camera ids
openclaw nodes camera snap --node <id> # prints saved path
openclaw nodes camera list --node <id> # list camera ids
openclaw nodes camera snap --node <id> # prints saved path
openclaw nodes camera snap --node <id> --max-width 1280
openclaw nodes camera snap --node <id> --delay-ms 2000
openclaw nodes camera snap --node <id> --device-id <id>
openclaw nodes camera clip --node <id> --duration 10s # prints saved path
openclaw nodes camera clip --node <id> --duration-ms 3000 # prints saved path (legacy flag)
openclaw nodes camera clip --node <id> --duration 10s # prints saved path
openclaw nodes camera clip --node <id> --duration-ms 3000 # prints saved path (legacy flag)
openclaw nodes camera clip --node <id> --device-id <id>
openclaw nodes camera clip --node <id> --no-audio
```
Notes:
- `openclaw nodes camera snap` defaults to `maxWidth=1600` unless overridden.
- On macOS, `camera.snap` waits `delayMs` (default 2000ms) after warm-up/exposure settle before capturing.
- Photo payloads are recompressed to keep base64 under 5 MB.
- `camera.snap` waits `delayMs` (default 2000ms, clamped to `[0, 10000]`) after warm-up/exposure settle before capturing.
- Photo payloads are recompressed to keep base64 under 5MB.
## Safety + practical limits
- Camera and microphone access trigger the usual OS permission prompts (and require usage strings in Info.plist).
- Video clips are capped (currently `<= 60s`) to avoid oversized node payloads (base64 overhead + message limits).
- Camera and microphone access trigger the usual OS permission prompts (and require usage strings in `Info.plist`).
- Video clips are capped at 60s to avoid oversized node payloads (base64 overhead plus message limits).
## macOS screen video (OS-level)
@ -155,9 +137,7 @@ For _screen_ video (not camera), use the macOS companion:
openclaw nodes screen record --node <id> --duration 10s --fps 15 # prints saved path
```
Notes:
- Requires macOS **Screen Recording** permission (TCC).
Requires macOS **Screen Recording** permission (TCC).
## Related

67
docs/nodes/images.md
View file

@ -5,69 +5,80 @@ read_when:
title: "Image and media support"
---
The WhatsApp channel runs via **Baileys Web**. This document captures the current media handling rules for send, gateway, and agent replies.
The WhatsApp channel runs on Baileys Web. This page covers media handling rules for send, gateway, and agent replies.
## Goals
- Send media with optional captions via `openclaw message send --media`.
- Send media with an optional caption via `openclaw message send --media`.
- Allow auto-replies from the web inbox to include media alongside text.
- Keep per-type limits sane and predictable.
## CLI Surface
- `openclaw message send --media <path-or-url> [--message <caption>]`
- `--media` optional; caption can be empty for media-only sends.
- `--dry-run` prints the resolved payload; `--json` emits `{ channel, to, messageId, mediaUrl, caption }`.
`openclaw message send --target <dest> --media <path-or-url> [--message <caption>]`
- `--media <path-or-url>` — attach media (image/audio/video/document); accepts local paths or URLs. Optional; caption can be empty for media-only sends.
- `--gif-playback` — treat video media as GIF playback (WhatsApp only).
- `--force-document` — send media as a document to avoid channel compression (Telegram, WhatsApp); applies to images, GIFs, and videos.
- `--reply-to <id>`, `--thread-id <id>`, `--pin`, `--silent` — delivery/threading options shared with text-only sends.
- `--dry-run` — print the resolved payload and skip sending.
- `--json` — print the result as JSON: `{ action, channel, dryRun, handledBy, messageId?, payload }` (`payload` carries the channel-specific send result, including any media reference).
## WhatsApp Web channel behavior
- Input: local file path **or** HTTP(S) URL.
- Flow: load into a Buffer, detect media kind, and build the correct payload:
- **Images:** resize & recompress to JPEG (max side 2048px) targeting `channels.whatsapp.mediaMaxMb` (default: 50 MB).
- **Audio/Voice/Video:** pass-through up to 16 MB; audio is sent as a voice note (`ptt: true`).
- **Documents:** anything else, up to 100 MB, with filename preserved when available.
- WhatsApp GIF-style playback: send an MP4 with `gifPlayback: true` (CLI: `--gif-playback`) so mobile clients loop inline.
- MIME detection prefers magic bytes, then headers, then file extension.
- Flow: load into a buffer, detect media kind, then build the outbound payload per kind:
- **Images:** optimized to fit under `channels.whatsapp.mediaMaxMb` (default 50MB). Opaque images are recompressed to JPEG (default side ladder starts at 2048px, descending on repeated size misses); images with transparency are kept as PNG. If the source is already an acceptable JPEG/PNG/WebP within the size and side-length budget, the original bytes are preserved unchanged instead of being recompressed. Animated GIFs are never re-encoded, only size-checked.
- **Audio/voice:** unless already native voice audio (`.ogg`/`.opus`, or `audio/ogg`/`audio/opus`), outbound audio is transcoded via `ffmpeg` to Opus/OGG (48kHz mono, 64kbps, capped at 20 minutes) before sending as a voice note (`ptt: true`).
- **Video:** pass-through up to 16MB.
- **Documents:** anything else, up to 100MB, with filename preserved when available.
- WhatsApp GIF-style playback: send an MP4 with `gifPlayback: true` (CLI: `--gif-playback`) so mobile clients loop it inline.
- MIME detection prefers sniffed magic bytes, then the file extension, then response headers; a generic sniffed container (`application/octet-stream`, `zip`) never overrides a more specific extension mapping (for example XLSX vs ZIP).
- Caption comes from `--message` or `reply.text`; empty caption is allowed.
- Logging: non-verbose shows `↩️`/`✅`; verbose includes size and source path/URL.
<Note>
The 16MB audio/video and 100MB document figures above are the shared per-kind media defaults used when no explicit byte cap is passed. WhatsApp sends set an explicit cap from `channels.whatsapp.mediaMaxMb` (default 50MB), which applies uniformly across kinds for that account.
</Note>
## Auto-Reply Pipeline
- `getReplyFromConfig` returns `{ text?, mediaUrl?, mediaUrls? }`.
- `getReplyFromConfig` returns a reply payload (or array of payloads) with `text?`, `mediaUrl?`, and `mediaUrls?` among other fields.
- When media is present, the web sender resolves local paths or URLs using the same pipeline as `openclaw message send`.
- Multiple media entries are sent sequentially if provided.
## Inbound Media To Commands
- When inbound web messages include media, OpenClaw downloads to a temp file and exposes templating variables:
- `{{MediaUrl}}` pseudo-URL for the inbound media.
- `{{MediaPath}}` local temp path written before running the command.
- When a per-session Docker sandbox is enabled, inbound media is copied into the sandbox workspace and `MediaPath`/`MediaUrl` are rewritten to a relative path like `media/inbound/<filename>`.
- Media understanding (if configured via `tools.media.*` or shared `tools.media.models`) runs before templating and can insert `[Image]`, `[Audio]`, and `[Video]` blocks into `Body`.
- When inbound web messages include media, OpenClaw downloads it to a temp file and exposes templating variables:
- `{{MediaUrl}}` — pseudo-URL for the inbound media.
- `{{MediaPath}}` — local temp path written before running the command.
- When a per-session Docker sandbox is enabled, inbound media is copied into the sandbox workspace and `MediaPath`/`MediaUrl` are rewritten to a sandbox-relative path like `media/inbound/<filename>`.
- Media understanding (configured via `tools.media.*` or shared `tools.media.models`) runs before templating and can insert `[Image]`, `[Audio]`, and `[Video]` blocks into `Body`.
- Audio sets `{{Transcript}}` and uses the transcript for command parsing so slash commands still work.
- Video and image descriptions preserve any caption text for command parsing.
- If the active primary image model already supports vision natively, OpenClaw skips the `[Image]` summary block and passes the original image to the model instead.
- By default only the first matching image/audio/video attachment is processed; set `tools.media.<cap>.attachments` to process multiple attachments.
- If the active primary model already supports vision natively, OpenClaw skips the `[Image]` summary block and passes the original image to the model instead.
- By default only the first matching image/audio/video attachment is processed; set `tools.media.<capability>.attachments` to process multiple attachments.
## Limits and errors
**Outbound send caps (WhatsApp web send)**
- Images: up to `channels.whatsapp.mediaMaxMb` (default: 50 MB) after recompression.
- Audio/voice/video: 16 MB cap; documents: 100 MB cap.
- Oversize or unreadable media → clear error in logs and the reply is skipped.
- Images: up to `channels.whatsapp.mediaMaxMb` (default 50MB) after optimization.
- Audio/video: 16MB cap (shared default; overridden by `mediaMaxMb` when sending through WhatsApp).
- Documents: 100MB cap (shared default; overridden by `mediaMaxMb` when sending through WhatsApp).
- Oversize or unreadable media produces a clear error in logs, and the reply is skipped.
**Media understanding caps (transcription/description)**
- Image default: 10 MB (`tools.media.image.maxBytes`).
- Audio default: 20 MB (`tools.media.audio.maxBytes`).
- Video default: 50 MB (`tools.media.video.maxBytes`).
- Oversize media skips understanding, but replies still go through with the original body.
- Image default: 10MB (`tools.media.image.maxBytes`).
- Audio default: 20MB (`tools.media.audio.maxBytes`).
- Video default: 50MB (`tools.media.video.maxBytes`).
- Oversize media skips understanding, but the reply still goes through with the original body.
## Notes for Tests
- Cover send + reply flows for image/audio/document cases.
- Validate recompression for images (size bound) and voice-note flag for audio.
- Cover send and reply flows for image/audio/document cases.
- Validate size bounds after image optimization and the voice-note flag for audio.
- Ensure multi-media replies fan out as sequential sends.
## Related

203
docs/nodes/index.md
View file

@ -9,27 +9,17 @@ title: "Nodes"
A **node** is a companion device (macOS/iOS/Android/headless) that connects to the Gateway **WebSocket** (same port as operators) with `role: "node"` and exposes a command surface (e.g. `canvas.*`, `camera.*`, `device.*`, `notifications.*`, `system.*`) via `node.invoke`. Protocol details: [Gateway protocol](/gateway/protocol).
Legacy transport: [Bridge protocol](/gateway/bridge-protocol) (TCP JSONL;
historical only for current nodes).
Legacy transport: [Bridge protocol](/gateway/bridge-protocol) (TCP JSONL; historical only for current nodes).
macOS can also run in **node mode**: the menubar app connects to the Gateway's
WS server and exposes its local canvas/camera commands as a node (so
`openclaw nodes …` works against this Mac). In remote gateway mode, browser
automation is handled by the CLI node host (`openclaw node run` or the
installed node service), not by the native app node.
macOS can also run in **node mode**: the menubar app connects to the Gateway's WS server and exposes its local canvas/camera commands as a node (so `openclaw nodes …` works against this Mac). In remote gateway mode, browser automation is handled by the CLI node host (`openclaw node run` or the installed node service), not by the native app node.
Notes:
Nodes are **peripherals**, not gateways: they don't run the gateway service, and channel messages (Telegram, WhatsApp, etc.) land on the gateway, not on nodes.
- Nodes are **peripherals**, not gateways. They don't run the gateway service.
- Telegram/WhatsApp/etc. messages land on the **gateway**, not on nodes.
- Troubleshooting runbook: [/nodes/troubleshooting](/nodes/troubleshooting)
Troubleshooting runbook: [/nodes/troubleshooting](/nodes/troubleshooting)
## Pairing + status
**WS nodes use device pairing.** Nodes present a device identity during `connect`; the Gateway
creates a device pairing request for `role: node`. Approve via the devices CLI (or UI).
Quick CLI:
WS nodes use **device pairing**. A node presents a device identity during `connect`; the Gateway creates a device pairing request for `role: node`. Approve via the devices CLI (or UI).
```bash
openclaw devices list
@ -39,26 +29,12 @@ openclaw nodes status
openclaw nodes describe --node <idOrNameOrIp>
```
If a node retries with changed auth details (role/scopes/public key), the prior
pending request is superseded and a new `requestId` is created. Re-run
`openclaw devices list` before approving.
Notes:
Pending pairing requests expire after 5 minutes; see [Gateway-owned pairing](/gateway/pairing) for the full request/approve/token lifecycle. If a node retries with changed auth details (role/scopes/public key), the prior pending request is superseded and a new `requestId` is created — re-run `openclaw devices list` before approving.
- `nodes status` marks a node as **paired** when its device pairing role includes `node`.
- The device pairing record is the durable approved-role contract. Token
rotation stays inside that contract; it cannot upgrade a paired node into a
different role that pairing approval never granted.
- `node.pair.*` (CLI: `openclaw nodes pending/approve/reject/remove/rename`) is a separate gateway-owned
node pairing store; it does **not** gate the WS `connect` handshake.
- `openclaw nodes remove --node <id|name|ip>` removes a node pairing. For a
device-backed node it revokes the device's `node` role in `devices/paired.json`
and disconnects that device's node-role sessions — a mixed-role device keeps
its row and only loses the `node` role, while a node-only device row is
deleted. It also clears any matching entry from the separate gateway-owned node
pairing store. `operator.pairing` may remove non-operator node rows; a
device-token caller revoking its own node role on a mixed-role device
additionally needs `operator.admin`.
- The device pairing record is the durable approved-role contract. Token rotation stays inside that contract; it cannot upgrade a paired node into a role that pairing approval never granted.
- `node.pair.*` (CLI: `openclaw nodes pending/approve/reject/remove/rename`) is a separate, gateway-owned node pairing store that tracks the node's approved command/capability surface across reconnects. It does **not** gate the WS `connect` handshake — device pairing does that.
- `openclaw nodes remove --node <id|name|ip>` removes a node pairing. For a device-backed node it revokes the device's `node` role in `devices/paired.json` and disconnects that device's node-role sessions: a mixed-role device keeps its row and only loses the `node` role, while a node-only device row is deleted. It also clears any matching entry from the separate node pairing store. `operator.pairing` may remove non-operator node rows on other devices; a device-token caller revoking its own node role on a mixed-role device additionally needs `operator.admin`.
- Approval scope follows the pending request's declared commands:
- commandless request: `operator.pairing`
- non-exec node commands: `operator.pairing` + `operator.write`
@ -66,24 +42,19 @@ Notes:
## Remote node host (system.run)
Use a **node host** when your Gateway runs on one machine and you want commands
to execute on another. The model still talks to the **gateway**; the gateway
forwards `exec` calls to the **node host** when `host=node` is selected.
Use a **node host** when your Gateway runs on one machine and you want commands to execute on another. The model still talks to the **gateway**; the gateway forwards `exec` calls to the **node host** when `host=node` is selected.
### What runs where
- **Gateway host**: receives messages, runs the model, routes tool calls.
- **Node host**: executes `system.run`/`system.which` on the node machine.
- **Approvals**: enforced on the node host via `~/.openclaw/exec-approvals.json`.
| Role | Responsibility |
| ------------ | ---------------------------------------------------------------- |
| Gateway host | Receives messages, runs the model, routes tool calls. |
| Node host | Executes `system.run`/`system.which` on the node machine. |
| Approvals | Enforced on the node host via `~/.openclaw/exec-approvals.json`. |
Approval note:
- Approval-backed node runs bind exact request context.
- For direct shell/runtime file executions, OpenClaw also best-effort binds one concrete local
file operand and denies the run if that file changes before execution.
- If OpenClaw cannot identify exactly one concrete local file for an interpreter/runtime command,
approval-backed execution is denied instead of pretending full runtime coverage. Use sandboxing,
separate hosts, or an explicit trusted allowlist/full workflow for broader interpreter semantics.
- Approval-backed node runs bind exact request context. The exec path prepares a canonical `systemRunPlan` before approval; once granted, the gateway forwards that stored plan, not any later caller-edited command/cwd/session fields, and re-validates the working directory before running.
- For direct shell/runtime file executions, OpenClaw also best-effort binds one concrete local file operand and denies the run if that file changes before execution.
- If OpenClaw cannot identify exactly one concrete local file for an interpreter/runtime command, approval-backed execution is denied instead of pretending full runtime coverage. Use sandboxing, separate hosts, or an explicit trusted allowlist/full workflow for broader interpreter semantics.
### Start a node host (foreground)
@ -93,11 +64,11 @@ On the node machine:
openclaw node run --host <gateway-host> --port 18789 --display-name "Build Node"
```
`node run` also accepts `--context-path` (Gateway WS context path), `--tls`, `--tls-fingerprint <sha256>`, and `--node-id` (overriding it clears the pairing token).
### Remote gateway via SSH tunnel (loopback bind)
If the Gateway binds to loopback (`gateway.bind=loopback`, default in local mode),
remote node hosts cannot connect directly. Create an SSH tunnel and point the
node host at the local end of the tunnel.
If the Gateway binds to loopback (`gateway.bind=loopback`, default in local mode), remote node hosts cannot connect directly. Create an SSH tunnel and point the node host at the local end of the tunnel.
Example (node host -> gateway host):
@ -128,6 +99,8 @@ openclaw node start
openclaw node restart
```
`node install` also accepts `--context-path`, `--tls`, `--tls-fingerprint`, `--node-id`, `--runtime <node|bun>` (default: node), and `--force` to reinstall. `node status`, `node stop`, and `node uninstall` are also available.
### Pair + name
On the gateway host:
@ -138,12 +111,11 @@ openclaw devices approve <requestId>
openclaw nodes status
```
If the node retries with changed auth details, re-run `openclaw devices list`
and approve the current `requestId`.
If the node retries with changed auth details, re-run `openclaw devices list` and approve the current `requestId`.
Naming options:
- `--display-name` on `openclaw node run` / `openclaw node install` (persists in `~/.openclaw/node.json` on the node).
- `--display-name` on `openclaw node run` / `openclaw node install` (persists in `~/.openclaw/node.json` on the node, alongside the node id, token, and gateway connection info).
- `openclaw nodes rename --node <id|name|ip> --name "Build Node"` (gateway override).
### Allowlist the commands
@ -169,12 +141,11 @@ openclaw config set tools.exec.node "<id-or-name>"
Or per session:
```
```text
/exec host=node security=allowlist node=<id-or-name>
```
Once set, any `exec` call with `host=node` runs on the node host (subject to the
node allowlist/approvals).
Once set, any `exec` call with `host=node` runs on the node host (subject to the node allowlist/approvals).
`host=auto` will not implicitly choose the node on its own, but an explicit per-call `host=node` request is allowed from `auto`. If you want node exec to be the default for the session, set `tools.exec.host=node` or `/exec host=node ...` explicitly.
@ -186,11 +157,7 @@ Related:
### Local model inference
A desktop or server node can expose chat-capable models from an Ollama server
running on that node. Agents use the Ollama plugin's `node_inference` tool to
discover installed models and run a bounded prompt remotely; the Gateway does
not need direct network access to Ollama. See [Ollama node-local inference](/providers/ollama#node-local-inference)
for setup, model filtering, and direct verification commands.
A desktop or server node can expose chat-capable models from an Ollama server running on that node. Agents use the Ollama plugin's `node_inference` tool to discover installed models and run a bounded prompt remotely; the Gateway does not need direct network access to Ollama. See [Ollama node-local inference](/providers/ollama#node-local-inference) for setup, model filtering, and direct verification commands.
## Invoking commands
@ -200,33 +167,36 @@ Low-level (raw RPC):
openclaw nodes invoke --node <idOrNameOrIp> --command canvas.eval --params '{"javaScript":"location.href"}'
```
Higher-level helpers exist for the common "give the agent a MEDIA attachment" workflows.
`nodes invoke` blocks `system.run` and `system.run.prepare`; those commands only run through the `exec` tool with `host=node` (see above). Higher-level helpers exist for the common "give the agent a MEDIA attachment" workflows (canvas, camera, screen, location, below).
## Command policy
Node commands must pass two gates before they can be invoked:
1. The node must declare the command in its WebSocket `connect.commands` list.
2. The gateway's platform policy must allow the declared command.
2. The gateway's platform-and-approval-derived allowlist must include the declared command.
Windows and macOS companion nodes allow safe declared commands such as
`canvas.*`, `camera.list`, `location.get`, and `screen.snapshot` by default.
Trusted nodes that advertise the `talk` capability or declare `talk.*` commands
also allow declared push-to-talk commands (`talk.ptt.start`, `talk.ptt.stop`,
`talk.ptt.cancel`, `talk.ptt.once`) by default, independent of platform label.
Dangerous or privacy-heavy commands such as `camera.snap`, `camera.clip`, and
`screen.record` still require explicit opt-in with
`gateway.nodes.allowCommands`. `gateway.nodes.denyCommands` always wins over
defaults and extra allowlist entries.
Default allowlists by platform (before plugin defaults and `allowCommands`/`denyCommands` overrides):
Plugin-owned node commands can add a Gateway node-invoke policy. That policy
runs after the allowlist check and before forwarding to the node, so raw
`node.invoke`, CLI helpers, and dedicated agent tools share the same plugin
permission boundary. Dangerous plugin node commands still require explicit
`gateway.nodes.allowCommands` opt-in.
| Platform | Commands allowed by default |
| -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| iOS | `camera.list`, `location.get`, `device.info`, `device.status`, `contacts.search`, `calendar.events`, `reminders.list`, `photos.latest`, `motion.activity`, `motion.pedometer`, `system.notify` |
| Android | `camera.list`, `location.get`, `notifications.list`, `notifications.actions`, `system.notify`, `device.info`, `device.status`, `device.permissions`, `device.health`, `device.apps`, `contacts.search`, `calendar.events`, `callLog.search`, `reminders.list`, `photos.latest`, `motion.activity`, `motion.pedometer` |
| macOS | `camera.list`, `location.get`, `device.info`, `device.status`, `contacts.search`, `calendar.events`, `reminders.list`, `photos.latest`, `motion.activity`, `motion.pedometer`, `system.notify` |
| Windows | `camera.list`, `location.get`, `device.info`, `device.status`, `system.notify` |
| Linux | `system.notify` (node host commands like `system.run` are approval-gated, see below) |
After a node changes its declared command list, reject the old device pairing
and approve the new request so the gateway stores the updated command snapshot.
`canvas.*` commands (`canvas.present`, `canvas.hide`, `canvas.navigate`, `canvas.eval`, `canvas.snapshot`, `canvas.a2ui.*`) are a plugin default on iOS, Android, macOS, Windows, and unknown platforms (not Linux); all of them are foreground-restricted on iOS.
`talk.ptt.start`, `talk.ptt.stop`, `talk.ptt.cancel`, and `talk.ptt.once` are allowed by default for any node that advertises the `talk` capability or declares `talk.*` commands, independent of platform label.
Desktop host commands (`system.run`, `system.run.prepare`, `system.which`, `browser.proxy`, `screen.snapshot` on macOS/Windows) are not part of the static platform-default table above. They become available once the operator approves a pairing request that declares them, after which the node's approved command set carries them forward on reconnect.
Dangerous or privacy-heavy commands still require explicit opt-in with `gateway.nodes.allowCommands`, even if a node declares them: `camera.snap`, `camera.clip`, `screen.record`, `contacts.add`, `calendar.add`, `reminders.add`, `sms.send`, `sms.search`. `gateway.nodes.denyCommands` always wins over defaults and extra allowlist entries.
Plugin-owned node commands can add a Gateway node-invoke policy. That policy runs after the allowlist check and before forwarding to the node, so raw `node.invoke`, CLI helpers, and dedicated agent tools share the same plugin permission boundary. Dangerous plugin node commands still require explicit `gateway.nodes.allowCommands` opt-in.
After a node changes its declared command list, reject the old device pairing and approve the new request so the gateway stores the updated command snapshot.
## Config (`openclaw.json`)
@ -261,10 +231,7 @@ Node-related settings live under `gateway.nodes` and `tools.exec`:
}
```
Use exact node command names. `denyCommands` removes a command even when a
platform default or `allowCommands` entry would otherwise allow it. See
[Gateway configuration reference](/gateway/configuration-reference#gateway-field-details)
for gateway node pairing and command-policy field details.
Use exact node command names. `denyCommands` removes a command even when a platform default or `allowCommands` entry would otherwise allow it. See [Gateway configuration reference](/gateway/configuration-reference#gateway) for gateway node pairing and command-policy field details.
Per-agent exec node override:
@ -328,6 +295,7 @@ Photos (`jpg`):
openclaw nodes camera list --node <idOrNameOrIp>
openclaw nodes camera snap --node <idOrNameOrIp> # default: both facings (2 MEDIA lines)
openclaw nodes camera snap --node <idOrNameOrIp> --facing front
openclaw nodes camera snap --node <idOrNameOrIp> --device-id <id> --max-width 1200 --quality 0.9 --delay-ms 2000
```
Video clips (`mp4`):
@ -340,7 +308,7 @@ openclaw nodes camera clip --node <idOrNameOrIp> --duration 3000 --no-audio
Notes:
- The node must be **foregrounded** for `canvas.*` and `camera.*` (background calls return `NODE_BACKGROUND_UNAVAILABLE`).
- Clip duration is clamped (currently `<= 60s`) to avoid oversized base64 payloads.
- Nodes clamp clip duration to keep the base64 payload manageable (see [Camera capture](/nodes/camera) for exact per-platform limits). The `nodes` agent tool additionally caps requested `durationMs` at 300000 (5 minutes) before forwarding the call; the node itself enforces the tighter limit.
- Android will prompt for `CAMERA`/`RECORD_AUDIO` permissions when possible; denied permissions fail with `*_PERMISSION_REQUIRED`.
## Screen recordings (nodes)
@ -355,9 +323,9 @@ openclaw nodes screen record --node <idOrNameOrIp> --duration 10s --fps 10 --no-
Notes:
- `screen.record` availability depends on node platform.
- Screen recordings are clamped to `<= 60s`.
- The `nodes` agent tool caps requested `durationMs` at 300000 (5 minutes); the node may enforce a tighter limit to bound the returned payload.
- `--no-audio` disables microphone capture on supported platforms.
- Use `--screen <index>` to select a display when multiple screens are available.
- Use `--screen <index>` to select a display when multiple screens are available (0 = primary).
## Location (nodes)
@ -375,10 +343,11 @@ Notes:
- Location is **off by default**.
- "Always" requires system permission; background fetch is best-effort.
- The response includes lat/lon, accuracy (meters), and timestamp.
- Full parameter/response shape and error codes: [Location command](/nodes/location-command).
## SMS (Android nodes)
Android nodes can expose `sms.send` when the user grants **SMS** permission and the device supports telephony.
Android nodes can expose `sms.send` and `sms.search` when the user grants **SMS** permission and the device supports telephony. Both commands are dangerous-by-default: the gateway operator must also add them to `gateway.nodes.allowCommands` before they can be invoked (see [Command policy](#command-policy)).
Low-level invoke:
@ -391,21 +360,21 @@ Notes:
- The permission prompt must be accepted on the Android device before the capability is advertised.
- Wi-Fi-only devices without telephony will not advertise `sms.send`.
## Android device + personal data commands
## Device and personal data commands
Android nodes can advertise additional command families when the corresponding capabilities are enabled.
iOS, Android, and macOS nodes advertise several read-only data commands by default (see the [Command policy](#command-policy) table); Android additionally exposes a larger family gated by its own in-app settings.
Available families:
- `device.status`, `device.info`, `device.permissions`, `device.health`
- `device.apps` when Installed Apps sharing is enabled in Android Settings
- `notifications.list`, `notifications.actions`
- `photos.latest`
- `contacts.search`, `contacts.add`
- `calendar.events`, `calendar.add`
- `callLog.search`
- `sms.search`
- `motion.activity`, `motion.pedometer`
- `device.status`, `device.info` — iOS, Android, macOS, Windows.
- `device.permissions`, `device.health`, `device.apps` — Android only; `device.apps` requires Installed Apps sharing enabled in Android Settings and returns launcher-visible apps by default.
- `notifications.list`, `notifications.actions` — Android only.
- `photos.latest` — iOS, Android, macOS.
- `contacts.search` — iOS, Android, macOS (read-only default); `contacts.add` is dangerous and needs `gateway.nodes.allowCommands`.
- `calendar.events` — iOS, Android, macOS (read-only default); `calendar.add` is dangerous and needs `gateway.nodes.allowCommands`.
- `reminders.list` — iOS, Android, macOS (read-only default); `reminders.add` is dangerous and needs `gateway.nodes.allowCommands`.
- `callLog.search` — Android only.
- `motion.activity`, `motion.pedometer` — iOS, Android, macOS; capability-gated by available sensors.
Example invokes:
@ -416,15 +385,9 @@ openclaw nodes invoke --node <idOrNameOrIp> --command notifications.list --param
openclaw nodes invoke --node <idOrNameOrIp> --command photos.latest --params '{"limit":1}'
```
Notes:
- `device.apps` is opt-in and returns launcher-visible apps by default.
- Motion commands are capability-gated by available sensors.
## System commands (node host / mac node)
The macOS node exposes `system.run`, `system.notify`, and `system.execApprovals.get/set`.
The headless node host exposes `system.run`, `system.which`, and `system.execApprovals.get/set`.
The macOS node exposes `system.run`, `system.notify`, and `system.execApprovals.get/set`. The headless node host exposes `system.run`, `system.which`, and `system.execApprovals.get/set`.
Examples:
@ -438,25 +401,20 @@ Notes:
- `system.run` returns stdout/stderr/exit code in the payload.
- Shell execution now goes through the `exec` tool with `host=node`; `nodes` remains the direct-RPC surface for explicit node commands.
- `nodes invoke` does not expose `system.run` or `system.run.prepare`; those stay on the exec path only.
- The exec path prepares a canonical `systemRunPlan` before approval. Once an
approval is granted, the gateway forwards that stored plan, not any later
caller-edited command/cwd/session fields.
- `system.notify` respects notification permission state on the macOS app.
- The exec path prepares a canonical `systemRunPlan` before approval. Once an approval is granted, the gateway forwards that stored plan, not any later caller-edited command/cwd/session fields.
- `system.notify` respects notification permission state on the macOS app; supports `--priority <passive|active|timeSensitive>` and `--delivery <system|overlay|auto>`.
- Unrecognized node `platform` / `deviceFamily` metadata uses a conservative default allowlist that excludes `system.run` and `system.which`. If you intentionally need those commands for an unknown platform, add them explicitly via `gateway.nodes.allowCommands`.
- `system.run` supports `--cwd`, `--env KEY=VAL`, `--command-timeout`, and `--needs-screen-recording`.
- For shell wrappers (`bash|sh|zsh ... -c/-lc`), request-scoped `--env` values are reduced to an explicit allowlist (`TERM`, `LANG`, `LC_*`, `COLORTERM`, `NO_COLOR`, `FORCE_COLOR`).
- For allow-always decisions in allowlist mode, known dispatch wrappers (`env`, `flock`, `nice`, `nohup`, `stdbuf`, `timeout`) persist inner executable paths instead of wrapper paths. If unwrapping is not safe, no allowlist entry is persisted automatically.
- On Windows node hosts in allowlist mode, shell-wrapper runs via `cmd.exe /c` require approval (allowlist entry alone does not auto-allow the wrapper form).
- `system.notify` supports `--priority <passive|active|timeSensitive>` and `--delivery <system|overlay|auto>`.
- Node hosts ignore `PATH` overrides and strip dangerous startup/shell keys (`DYLD_*`, `LD_*`, `BASHOPTS`, `FPATH`, `KSH_ENV`, `NODE_OPTIONS`, `NODE_REDIRECT_WARNINGS`, `NODE_REPL_EXTERNAL_MODULE`, `NODE_REPL_HISTORY`, `NODE_V8_COVERAGE`, `PYTHON*`, `PERL*`, `RUBYOPT`, `SHELLOPTS`, `PS4`, `TCLLIBPATH`). If you need extra PATH entries, configure the node host service environment (or install tools in standard locations) instead of passing `PATH` via `--env`.
- On macOS node mode, `system.run` is gated by exec approvals in the macOS app (Settings → Exec approvals).
Ask/allowlist/full behave the same as the headless node host; denied prompts return `SYSTEM_RUN_DENIED`.
- On headless node host, `system.run` is gated by exec approvals (`~/.openclaw/exec-approvals.json`).
- Node hosts ignore `PATH` overrides in `--env` and strip a large, maintained set of interpreter/shell startup variables (for example `NODE_OPTIONS`, `PYTHONPATH`, `BASH_ENV`, `DYLD_*`, `LD_*`) before running a command. If you need extra PATH entries, configure the node host service environment (or install tools in standard locations) instead of passing `PATH` via `--env`.
- On macOS node mode, `system.run` is gated by exec approvals in the macOS app (Settings → Exec approvals). Ask/allowlist/full behave the same as the headless node host; denied prompts return `SYSTEM_RUN_DENIED`.
- On headless node host, `system.run` is gated by exec approvals (`~/.openclaw/exec-approvals.json`); on macOS specifically, see the exec-host routing env vars under [Headless node host](#headless-node-host-cross-platform) below.
## Exec node binding
When multiple nodes are available, you can bind exec to a specific node.
This sets the default node for `exec host=node` (and can be overridden per agent).
When multiple nodes are available, you can bind exec to a specific node. This sets the default node for `exec host=node` (and can be overridden per agent).
Global default:
@ -480,13 +438,11 @@ openclaw config unset 'agents.list[0].tools.exec.node'
## Permissions map
Nodes may include a `permissions` map in `node.list` / `node.describe`, keyed by permission name (e.g. `screenRecording`, `accessibility`) with boolean values (`true` = granted).
Nodes may include a `permissions` map in `node.list` / `node.describe`, keyed by permission name (e.g. `screenRecording`, `accessibility`, `location`) with boolean values (`true` = granted).
## Headless node host (cross-platform)
OpenClaw can run a **headless node host** (no UI) that connects to the Gateway
WebSocket and exposes `system.run` / `system.which`. This is useful on Linux/Windows
or for running a minimal node alongside a server.
OpenClaw can run a **headless node host** (no UI) that connects to the Gateway WebSocket and exposes `system.run` / `system.which`. This is useful on Linux/Windows or for running a minimal node alongside a server.
Start it:
@ -498,11 +454,8 @@ Notes:
- Pairing is still required (the Gateway will show a device pairing prompt).
- The node host stores its node id, token, display name, and gateway connection info in `~/.openclaw/node.json`.
- Exec approvals are enforced locally via `~/.openclaw/exec-approvals.json`
(see [Exec approvals](/tools/exec-approvals)).
- On macOS, the headless node host executes `system.run` locally by default. Set
`OPENCLAW_NODE_EXEC_HOST=app` to route `system.run` through the companion app exec host; add
`OPENCLAW_NODE_EXEC_FALLBACK=0` to require the app host and fail closed if it is unavailable.
- Exec approvals are enforced locally via `~/.openclaw/exec-approvals.json` (see [Exec approvals](/tools/exec-approvals)).
- On macOS, the headless node host executes `system.run` locally by default. Set `OPENCLAW_NODE_EXEC_HOST=app` to route `system.run` through the companion app exec host; add `OPENCLAW_NODE_EXEC_FALLBACK=0` to require the app host and fail closed if it is unavailable.
- Add `--tls` / `--tls-fingerprint` when the Gateway WS uses TLS.
## Mac node mode

39
docs/nodes/location-command.md
View file

@ -8,20 +8,14 @@ title: "Location command"
## TL;DR
- `location.get` is a node command (via `node.invoke`).
- `location.get` is a node command, invoked via `node.invoke` or `openclaw nodes location get`.
- Off by default.
- Android app settings use a selector: Off / While Using.
- Separate toggle: Precise Location.
- Precise Location is a separate toggle.
## Why a selector (not just a switch)
OS permissions are multi-level. We can expose a selector in-app, but the OS still decides the actual grant.
- iOS/macOS may expose **While Using** or **Always** in system prompts/Settings.
- Android app currently supports foreground location only.
- Precise location is a separate grant (iOS 14+ "Precise", Android "fine" vs "coarse").
Selector in UI drives our requested mode; actual grant lives in OS settings.
OS location permissions are multi-level (iOS/macOS expose While Using vs Always; Android currently supports foreground-only). Precise location is a separate OS grant too (iOS 14+ "Precise", Android "fine" vs "coarse"). The in-app selector drives the requested mode, but the OS still decides the actual grant.
## Settings model
@ -33,17 +27,22 @@ Per node device:
UI behavior:
- Selecting `whileUsing` requests foreground permission.
- If OS denies requested level, revert to the highest granted level and show status.
- If the OS denies the requested level, the app reverts to the highest granted level and shows status.
## Permissions mapping (node.permissions)
Optional. macOS node reports `location` via the permissions map; iOS/Android may omit it.
Optional. The macOS node reports `location` via the `permissions` map on `node.list`/`node.describe`; iOS/Android may omit it.
## Command: `location.get`
Called via `node.invoke`.
Called via `node.invoke`, or the CLI helper:
Params (suggested):
```bash
openclaw nodes location get --node <idOrNameOrIp>
openclaw nodes location get --node <idOrNameOrIp> --accuracy precise --max-age 15000 --location-timeout 10000
```
Params:
```json
{
@ -53,6 +52,8 @@ Params (suggested):
}
```
CLI flags map directly: `--location-timeout` -> `timeoutMs`, `--max-age` -> `maxAgeMs`, `--accuracy` -> `desiredAccuracy`.
Response payload:
```json
@ -73,21 +74,20 @@ Errors (stable codes):
- `LOCATION_DISABLED`: selector is off.
- `LOCATION_PERMISSION_REQUIRED`: permission missing for requested mode.
- `LOCATION_BACKGROUND_UNAVAILABLE`: app is backgrounded but only While Using allowed.
- `LOCATION_BACKGROUND_UNAVAILABLE`: app is backgrounded but only While Using is granted.
- `LOCATION_TIMEOUT`: no fix in time.
- `LOCATION_UNAVAILABLE`: system failure / no providers.
- `LOCATION_UNAVAILABLE`: system failure or no providers.
## Background behavior
- Android app denies `location.get` while backgrounded.
- Keep OpenClaw open when requesting location on Android.
- The Android app denies `location.get` while backgrounded; keep OpenClaw open when requesting location on Android.
- Other node platforms may differ.
## Model/tooling integration
- Tool surface: `nodes` tool adds `location_get` action (node required).
- Agent tool: the `nodes` tool's `location_get` action (node required).
- CLI: `openclaw nodes location get --node <id>`.
- Agent guidelines: only call when user enabled location and understands the scope.
- Agent guidelines: only call when the user enabled location and understands the scope.
## UX copy (suggested)
@ -97,6 +97,7 @@ Errors (stable codes):
## Related
- [Nodes overview](/nodes)
- [Channel location parsing](/channels/location)
- [Camera capture](/nodes/camera)
- [Talk mode](/nodes/talk)

282
docs/nodes/media-understanding.md
View file

@ -7,70 +7,41 @@ title: "Media understanding"
sidebarTitle: "Media understanding"
---
OpenClaw can **summarize inbound media** (image/audio/video) before the reply pipeline runs. It auto-detects when local tools or provider keys are available, and can be disabled or customized. If understanding is off, models still receive the original files/URLs as usual.
OpenClaw can summarize inbound media (image/audio/video) before the reply pipeline runs, so command parsing and routing work off short text instead of raw bytes. Understanding auto-detects local tools or provider keys, or you can configure explicit models. Original media is always delivered to the model as usual; when understanding fails or is disabled, the reply flow continues unchanged.
Vendor-specific media behavior is registered by vendor plugins, while OpenClaw core owns the shared `tools.media` config, fallback order, and reply-pipeline integration.
Vendor plugins register capability metadata (which provider supports which media type, default model, priority). OpenClaw core owns the shared `tools.media` config, fallback order, and reply-pipeline integration.
## Goals
- Optional: pre-digest inbound media into short text for faster routing + better command parsing.
- Preserve original media delivery to the model (always).
- Support **provider APIs** and **CLI fallbacks**.
- Allow multiple models with ordered fallback (error/size/timeout).
## High-level behavior
## How it works
<Steps>
<Step title="Collect attachments">
Collect inbound attachments (`MediaPaths`, `MediaUrls`, `MediaTypes`).
</Step>
<Step title="Select per-capability">
For each enabled capability (image/audio/video), select attachments per policy (default: **first**).
<Step title="Select per capability">
For each enabled capability (image/audio/video), select attachments per the `attachments` policy (default: first attachment only).
</Step>
<Step title="Choose model">
Choose the first eligible model entry (size + capability + auth).
<Step title="Choose a model">
Pick the first eligible model entry (size + capability + auth available).
</Step>
<Step title="Fallback on failure">
If a model fails or the media is too large, **fall back to the next entry**.
<Step title="Fall back on failure">
If a model errors, times out, or the media exceeds `maxBytes`, try the next entry.
</Step>
<Step title="Apply success block">
On success:
- `Body` becomes `[Image]`, `[Audio]`, or `[Video]` block.
- Audio sets `{{Transcript}}`; command parsing uses caption text when present, otherwise the transcript.
- Captions are preserved as `User text:` inside the block.
<Step title="Apply on success">
`Body` becomes an `[Image]`, `[Audio]`, or `[Video]` block. Audio also sets `{{Transcript}}`; command parsing uses caption text when present, otherwise the transcript. Captions are preserved as `User text:` inside the block.
</Step>
</Steps>
If understanding fails or is disabled, **the reply flow continues** with the original body + attachments.
## Config
## Config overview
`tools.media` supports **shared models** plus per-capability overrides:
<AccordionGroup>
<Accordion title="Top-level keys">
- `tools.media.models`: shared model list (use `capabilities` to gate).
- `tools.media.image` / `tools.media.audio` / `tools.media.video`:
- defaults (`prompt`, `maxChars`, `maxBytes`, `timeoutSeconds`, `language`)
- provider overrides (`baseUrl`, `headers`, `providerOptions`)
- Deepgram audio options via `tools.media.audio.providerOptions.deepgram`
- audio transcript echo controls (`echoTranscript`, default `false`; `echoFormat`)
- optional **per-capability `models` list** (preferred before shared models)
- `attachments` policy (`mode`, `maxAttachments`, `prefer`)
- `scope` (optional gating by channel/chatType/session key)
- `tools.media.concurrency`: max concurrent capability runs (default **2**).
</Accordion>
</AccordionGroup>
`tools.media` holds a shared model list plus per-capability overrides:
```json5
{
tools: {
media: {
concurrency: 2, // max concurrent capability runs (default)
models: [
/* shared list */
/* shared list, gate with capabilities */
],
image: {
/* optional overrides */
@ -88,9 +59,28 @@ If understanding fails or is disabled, **the reply flow continues** with the ori
}
```
Per-capability (`image`/`audio`/`video`) keys:
| Key | Type | Default | Notes |
| ----------------------------------------------- | --------- | ---------------------------------------------------- | ----------------------------------------------------------------------------------- |
| `enabled` | `boolean` | auto (`false` disables) | Set `false` to turn off auto-detect for this capability |
| `models` | array | none | Preferred before the shared `tools.media.models` list |
| `prompt` | `string` | `"Describe the {media}."` (+ maxChars guidance) | Image/video only by default |
| `maxChars` | `number` | `500` (image/video), unset (audio) | Output is trimmed if the model returns more |
| `maxBytes` | `number` | image `10485760`, audio `20971520`, video `52428800` | Oversized media skips to the next model |
| `timeoutSeconds` | `number` | `60` (image/audio), `120` (video) | |
| `language` | `string` | unset | Audio transcription hint |
| `baseUrl`/`headers`/`providerOptions`/`request` | - | - | Provider request overrides; see [Tools and custom providers](/gateway/config-tools) |
| `attachments` | object | `{ mode: "first", maxAttachments: 1 }` | See [Attachment policy](#attachment-policy) |
| `scope` | object | unset | Gate by channel/chatType/keyPrefix |
| `echoTranscript` | `boolean` | `false` | Audio only: echo the transcript back to the chat before agent processing |
| `echoFormat` | `string` | `'📝 "{transcript}"'` | Audio only: `{transcript}` placeholder |
Deepgram-specific options go under `providerOptions.deepgram` (the top-level `deepgram: { detectLanguage, punctuate, smartFormat }` field is deprecated but still read).
### Model entries
Each `models[]` entry can be **provider** or **CLI**:
Each `models[]` entry is a **provider** entry (default) or a **CLI** entry:
<Tabs>
<Tab title="Provider entry">
@ -103,7 +93,7 @@ Each `models[]` entry can be **provider** or **CLI**:
maxChars: 500,
maxBytes: 10485760,
timeoutSeconds: 60,
capabilities: ["image"], // optional, used for multi-modal entries
capabilities: ["image"], // optional, for multi-modal shared entries
profile: "vision-profile",
preferredProfile: "vision-fallback",
}
@ -128,26 +118,14 @@ Each `models[]` entry can be **provider** or **CLI**:
}
```
CLI templates can also use:
- `{{MediaDir}}` (directory containing the media file)
- `{{OutputDir}}` (scratch dir created for this run)
- `{{OutputBase}}` (scratch file base path, no extension)
CLI templates can also use `{{MediaDir}}` (directory containing the media file), `{{OutputDir}}` (scratch dir created for this run), and `{{OutputBase}}` (scratch file base path, no extension).
</Tab>
</Tabs>
### Provider credentials (`apiKey`)
### Provider credentials
Provider media understanding uses the same provider auth resolution as normal
model calls: auth profiles, environment variables, then
`models.providers.<providerId>.apiKey`.
`tools.media.*.models[]` entries do not accept an inline `apiKey` field. The
`provider` value in a media model entry, such as `openai` or `moonshot`, must
have credentials available through one of the standard provider auth sources.
Minimal example:
Provider media understanding uses the same auth resolution as normal model calls: auth profiles, environment variables, then `models.providers.<providerId>.apiKey`. `tools.media.*.models[]` entries do not accept an inline `apiKey` field.
```json5
{
@ -160,72 +138,52 @@ Minimal example:
}
```
For the full provider auth reference, including profiles, environment
variables, and custom base URLs, see [Tools and custom providers](/gateway/config-tools).
See [Tools and custom providers](/gateway/config-tools) for profiles, env vars, and custom base URLs.
## Defaults and limits
## Rules and behavior
Recommended defaults:
- Media exceeding `maxBytes` skips that model and tries the next one.
- Audio files under 1024 bytes are treated as empty/corrupt and skipped before transcription; the agent gets a deterministic placeholder transcript instead.
- If the active primary image model already supports vision natively, OpenClaw skips the `[Image]` summary block and passes the original image into the model directly. MiniMax is an exception: `minimax`, `minimax-cn`, `minimax-portal`, and `minimax-portal-cn` always route image understanding through the plugin-owned `MiniMax-VL-01` media provider, even if legacy MiniMax M2.x chat metadata claims image input (only `MiniMax-M3` and later are treated as natively vision-capable).
- If a Gateway/WebChat primary model is text-only, image attachments are preserved as offloaded `media://inbound/*` refs so image/PDF tools or a configured image model can still inspect them instead of losing the attachment.
- Explicit `openclaw infer image describe --file <path> --model <provider/model>` (alias: `openclaw capability image describe`) runs that image-capable provider/model directly, including Ollama refs such as `ollama/qwen2.5vl:7b` when a matching image-capable model is configured under `models.providers.ollama.models[]`.
- If `<capability>.enabled` is not `false` but no models are configured, OpenClaw tries the active reply model when its provider supports the capability.
- `maxChars`: **500** for image/video (short, command-friendly)
- `maxChars`: **unset** for audio (full transcript unless you set a limit)
- `maxBytes`:
- image: **10MB**
- audio: **20MB**
- video: **50MB**
### Auto-detect (default)
<AccordionGroup>
<Accordion title="Rules">
- If media exceeds `maxBytes`, that model is skipped and the **next model is tried**.
- Audio files smaller than **1024 bytes** are treated as empty/corrupt and skipped before provider/CLI transcription; inbound reply context receives a deterministic placeholder transcript so the agent knows the note was too small.
- If the model returns more than `maxChars`, output is trimmed.
- `prompt` defaults to simple "Describe the {media}." plus the `maxChars` guidance (image/video only).
- If the active primary image model already supports vision natively, OpenClaw skips the `[Image]` summary block and passes the original image into the model instead.
- If a Gateway/WebChat primary model is text-only, image attachments are preserved as offloaded `media://inbound/*` refs so the image/PDF tools or configured image model can still inspect them instead of losing the attachment.
- Explicit `openclaw infer image describe --model <provider/model>` requests are different: they run that image-capable provider/model directly, including Ollama refs such as `ollama/qwen2.5vl:7b`.
- If `<capability>.enabled: true` but no models are configured, OpenClaw tries the **active reply model** when its provider supports the capability.
</Accordion>
</AccordionGroup>
### Auto-detect media understanding (default)
If `tools.media.<capability>.enabled` is **not** set to `false` and you haven't configured models, OpenClaw auto-detects in this order and **stops at the first working option**:
When `tools.media.<capability>.enabled` is not `false` and no models are configured, OpenClaw tries these in order and stops at the first working option:
<Steps>
<Step title="Active reply model">
Active reply model when its provider supports the capability.
<Step title="Configured image model (image only)">
`agents.defaults.imageModel` primary/fallback refs, unless the active reply model already supports vision natively. Prefer `provider/model` refs; bare refs are qualified from configured image-capable provider model entries only when the match is unique.
</Step>
<Step title="agents.defaults.imageModel">
`agents.defaults.imageModel` primary/fallback refs (image only).
Prefer `provider/model` refs. Bare refs are qualified from configured image-capable provider model entries only when the match is unique.
<Step title="Active reply model">
The active reply model, when its provider supports the capability.
</Step>
<Step title="Provider auth (audio only, before local CLIs)">
Configured `models.providers.*` entries that support audio are tried before local CLIs. Bundled provider priority order (ties break alphabetically by provider id): Groq/OpenAI &rarr; xAI &rarr; Deepgram &rarr; OpenRouter &rarr; Google/SenseAudio &rarr; Deepinfra/ElevenLabs &rarr; Mistral.
</Step>
<Step title="Local CLIs (audio only)">
Local CLIs (if installed):
- `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with encoder/decoder/joiner/tokens)
- `whisper-cli` (`whisper-cpp`; uses `WHISPER_CPP_MODEL` or the bundled tiny model)
- `whisper` (Python CLI; downloads models automatically)
First installed local binary, in this order:
- `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with `tokens.txt`/`encoder.onnx`/`decoder.onnx`/`joiner.onnx`)
- `whisper-cli` (`whisper-cpp`; uses `WHISPER_CPP_MODEL` or a bundled tiny model)
- `whisper` (Python CLI; defaults to the `turbo` model, downloads automatically)
</Step>
<Step title="Gemini CLI">
`gemini` using `read_many_files`.
<Step title="Provider auth (image/video)">
Configured `models.providers.*` entries that support the capability are tried before the bundled fallback order. Image-only config providers with an image-capable model auto-register for media understanding even when they are not a bundled vendor plugin.
Bundled provider priority order (ties break alphabetically by provider id):
- Image: Anthropic/OpenAI &rarr; Google &rarr; MiniMax &rarr; Deepinfra &rarr; MiniMax Portal &rarr; Z.AI
- Video: Google &rarr; Qwen &rarr; Moonshot
</Step>
<Step title="Provider auth">
- Configured `models.providers.*` entries that support the capability are tried before the bundled fallback order.
- Image-only config providers with an image-capable model auto-register for media understanding even when they are not a bundled vendor plugin.
- Ollama image understanding is available when selected explicitly, for example through `agents.defaults.imageModel` or `openclaw infer image describe --model ollama/<vision-model>`.
Bundled fallback order:
- Audio: OpenAI → Groq → xAI → Deepgram → OpenRouter → Google → SenseAudio → ElevenLabs → Mistral
- Image: OpenAI → Anthropic → Google → MiniMax → MiniMax Portal → Z.AI
- Video: Google → Qwen → Moonshot
<Step title="Antigravity CLI (image/video only)">
First installed `agy` or `antigravity` binary (override with `OPENCLAW_ANTIGRAVITY_CLI`), sandboxed against the media's directory.
</Step>
</Steps>
To disable auto-detection, set:
To disable auto-detection for a capability:
```json5
{
@ -240,71 +198,59 @@ To disable auto-detection, set:
```
<Note>
Binary detection is best-effort across macOS/Linux/Windows; ensure the CLI is on `PATH` (we expand `~`), or set an explicit CLI model with a full command path.
Binary detection is best-effort across macOS/Linux/Windows; ensure the CLI is on `PATH` (`~` is expanded), or set an explicit CLI model entry with a full command path.
</Note>
### Proxy environment support (provider models)
### Proxy support (audio/video provider calls)
When provider-based **audio** and **video** media understanding is enabled, OpenClaw honors standard outbound proxy environment variables for provider HTTP calls:
Provider-based **audio** and **video** understanding honors standard outbound proxy environment variables, including `NO_PROXY`/`no_proxy` bypass rules: `HTTPS_PROXY`, `HTTP_PROXY`, `ALL_PROXY`, `https_proxy`, `http_proxy`, `all_proxy`. Lowercase vars take precedence over uppercase. If none are set, media understanding uses direct egress; if the proxy value is malformed, OpenClaw logs a warning and falls back to direct fetch. Image understanding does not go through this proxy path.
- `HTTPS_PROXY`
- `HTTP_PROXY`
- `ALL_PROXY`
- `https_proxy`
- `http_proxy`
- `all_proxy`
## Capabilities
If no proxy env vars are set, media understanding uses direct egress. If the proxy value is malformed, OpenClaw logs a warning and falls back to direct fetch.
Set `capabilities` on a `models[]` entry to restrict it to specific media types. For shared lists, OpenClaw infers defaults per bundled provider:
## Capabilities (optional)
| Provider | Capabilities |
| ------------------------------------------------------------------------ | --------------------- |
| `openai`, `anthropic`, `minimax` | image |
| `minimax-portal` | image |
| `moonshot` | image + video |
| `openrouter` | image + audio |
| `google` (Gemini API) | image + audio + video |
| `qwen` | image + video |
| `deepinfra` | image + audio |
| `mistral` | audio |
| `zai` | image |
| `groq`, `xai`, `deepgram`, `senseaudio` | audio |
| Any `models.providers.<id>.models[]` catalog with an image-capable model | image |
If you set `capabilities`, the entry only runs for those media types. For shared lists, OpenClaw can infer defaults:
For CLI entries, set `capabilities` explicitly to avoid surprising matches; if omitted, the entry is eligible for every capability list it appears in.
- `openai`, `anthropic`, `minimax`: **image**
- `minimax-portal`: **image**
- `moonshot`: **image + video**
- `openrouter`: **image + audio**
- `google` (Gemini API): **image + audio + video**
- `qwen`: **image + video**
- `mistral`: **audio**
- `zai`: **image**
- `groq`: **audio**
- `xai`: **audio**
- `deepgram`: **audio**
- Any `models.providers.<id>.models[]` catalog with an image-capable model: **image**
## Provider support matrix
For CLI entries, **set `capabilities` explicitly** to avoid surprising matches. If you omit `capabilities`, the entry is eligible for the list it appears in.
## Provider support matrix (OpenClaw integrations)
| Capability | Provider integration | Notes |
| ---------- | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Image | OpenAI, OpenAI Codex OAuth, Codex app-server, OpenRouter, Anthropic, Google, MiniMax, Moonshot, Qwen, Z.AI, config providers | Vendor plugins register image support; `openai/*` can use API-key or Codex OAuth routing; `codex/*` uses a bounded Codex app-server turn; MiniMax and MiniMax OAuth both use `MiniMax-VL-01`; image-capable config providers auto-register. |
| Audio | OpenAI, Groq, xAI, Deepgram, OpenRouter, Google, SenseAudio, ElevenLabs, Mistral | Provider transcription (Whisper/Groq/xAI/Deepgram/OpenRouter STT/Gemini/SenseAudio/Scribe/Voxtral). |
| Video | Google, Qwen, Moonshot | Provider video understanding via vendor plugins; Qwen video understanding uses the Standard DashScope endpoints. |
| Capability | Providers | Notes |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Image | Anthropic, Codex app-server, Deepinfra, Google, MiniMax, MiniMax Portal, Moonshot, OpenAI, OpenAI Codex OAuth, OpenRouter, Qwen, Z.AI, config providers | Vendor plugins register image support; `openai/*` can use API-key or Codex OAuth routing; `codex/*` uses a bounded Codex app-server turn; image-capable config providers auto-register. |
| Audio | Deepgram, Deepinfra, ElevenLabs, Google, Groq, Mistral, OpenAI, OpenRouter, SenseAudio, xAI | Provider transcription (Whisper/Groq/xAI/Deepgram/OpenRouter STT/Gemini/SenseAudio/Scribe/Voxtral). |
| Video | Google, Moonshot, Qwen | Provider video understanding via vendor plugins; Qwen video understanding uses the standard DashScope endpoints. |
<Note>
**MiniMax note**
- `minimax`, `minimax-cn`, `minimax-portal`, and `minimax-portal-cn` image understanding comes from the plugin-owned `MiniMax-VL-01` media provider.
- Automatic image routing keeps using `MiniMax-VL-01` even if legacy MiniMax M2.x chat metadata claims image input.
**MiniMax note**: `minimax`, `minimax-cn`, `minimax-portal`, and `minimax-portal-cn` image understanding always comes from the plugin-owned `MiniMax-VL-01` media provider, even if legacy MiniMax M2.x chat metadata claims image input.
</Note>
## Model selection guidance
- Prefer the strongest latest-generation model available for each media capability when quality and safety matter.
- Prefer the strongest current-generation model for each media capability when quality and safety matter.
- For tool-enabled agents handling untrusted inputs, avoid older/weaker media models.
- Keep at least one fallback per capability for availability (quality model + faster/cheaper model).
- CLI fallbacks (`whisper-cli`, `whisper`, `gemini`) are useful when provider APIs are unavailable.
- `parakeet-mlx` note: with `--output-dir`, OpenClaw reads `<output-dir>/<media-basename>.txt` when output format is `txt` (or unspecified); non-`txt` formats fall back to stdout.
- CLI fallbacks (`whisper-cli`, `whisper`, `gemini`) help when provider APIs are unavailable.
- `parakeet-mlx`: with `--output-dir`, OpenClaw reads `<output-dir>/<media-basename>.txt` when the output format is `txt` or unspecified; other formats fall back to stdout.
## Attachment policy
Per-capability `attachments` controls which attachments are processed:
<ParamField path="mode" type='"first" | "all"' default="first">
Whether to process the first selected attachment or all of them.
Process only the first selected attachment, or all of them.
</ParamField>
<ParamField path="maxAttachments" type="number" default="1">
Cap the number processed.
@ -315,16 +261,12 @@ Per-capability `attachments` controls which attachments are processed:
When `mode: "all"`, outputs are labeled `[Image 1/2]`, `[Audio 2/2]`, etc.
<AccordionGroup>
<Accordion title="File-attachment extraction behavior">
- Extracted file text is wrapped as **untrusted external content** before it is appended to the media prompt.
- The injected block uses explicit boundary markers like `<<<EXTERNAL_UNTRUSTED_CONTENT id="...">>>` / `<<<END_EXTERNAL_UNTRUSTED_CONTENT id="...">>>` and includes a `Source: External` metadata line.
- This attachment-extraction path intentionally omits the long `SECURITY NOTICE:` banner to avoid bloating the media prompt; the boundary markers and metadata still remain.
- If a file has no extractable text, OpenClaw injects `[No extractable text]`.
- If a PDF falls back to rendered page images in this path, OpenClaw forwards those page images to vision-capable reply models and keeps the placeholder `[PDF content rendered to images]` in the file block.
### File-attachment extraction
</Accordion>
</AccordionGroup>
- Extracted file text is wrapped as untrusted external content before it's appended to the media prompt, using boundary markers like `<<<EXTERNAL_UNTRUSTED_CONTENT id="...">>>` / `<<<END_EXTERNAL_UNTRUSTED_CONTENT id="...">>>` plus a `Source: External` metadata line.
- This path intentionally omits the long `SECURITY NOTICE:` banner to keep the media prompt short; the boundary markers and metadata still apply.
- A file with no extractable text gets `[No extractable text]`.
- If a PDF falls back to rendered page images, OpenClaw forwards those images to vision-capable reply models and keeps the placeholder `[PDF content rendered to images]` in the file block.
## Config examples
@ -404,7 +346,7 @@ When `mode: "all"`, outputs are labeled `[Image 1/2]`, `[Audio 2/2]`, etc.
}
```
</Tab>
<Tab title="Image-only">
<Tab title="Image only">
```json5
{
tools: {
@ -415,7 +357,7 @@ When `mode: "all"`, outputs are labeled `[Image 1/2]`, `[Audio 2/2]`, etc.
maxChars: 500,
models: [
{ provider: "openai", model: "gpt-5.5" },
{ provider: "anthropic", model: "claude-opus-4-6" },
{ provider: "anthropic", model: "claude-opus-4-8" },
{
type: "cli",
command: "gemini",
@ -475,19 +417,17 @@ When `mode: "all"`, outputs are labeled `[Image 1/2]`, `[Audio 2/2]`, etc.
## Status output
When media understanding runs, `/status` includes a short summary line:
When media understanding runs, `/status` includes a per-capability summary line:
```
📎 Media: image ok (openai/gpt-5.4) · audio skipped (maxBytes)
📎 Media: image ok (openai/gpt-5.5) · audio skipped (maxBytes)
```
This shows per-capability outcomes and the chosen provider/model when applicable.
## Notes
- Understanding is **best-effort**. Errors do not block replies.
- Understanding is best-effort. Errors do not block replies.
- Attachments are still passed to models even when understanding is disabled.
- Use `scope` to limit where understanding runs (e.g. only DMs).
- Use `scope` to limit where understanding runs (for example, only DMs).
## Related

118
docs/nodes/talk.md
View file

@ -6,41 +6,31 @@ read_when:
title: "Talk mode"
---
Talk mode has two runtime shapes:
Talk mode covers five runtime shapes:
- Native macOS/iOS/Android Talk uses local speech recognition, Gateway chat, and `talk.speak` TTS. Nodes advertise the `talk` capability and declare the `talk.*` commands they support.
- iOS Talk uses client-owned WebRTC for OpenAI realtime configurations that select `webrtc` or omit the transport. Explicit `gateway-relay`, `provider-websocket`, and non-OpenAI realtime configurations stay on the Gateway-owned relay; non-realtime configurations use the native speech loop.
- Browser Talk uses `talk.client.create` for client-owned `webrtc` and `provider-websocket` sessions, or `talk.session.create` for Gateway-owned `gateway-relay` sessions. `managed-room` is reserved for Gateway handoff and walkie-talkie rooms.
- Android Talk can opt into Gateway-owned realtime relay sessions with `talk.realtime.mode: "realtime"` and `talk.realtime.transport: "gateway-relay"`. Otherwise it stays on native speech recognition, Gateway chat, and `talk.speak`.
- Transcription-only clients use `talk.session.create({ mode: "transcription", transport: "gateway-relay", brain: "none" })`, then `talk.session.appendAudio`, `talk.session.cancelTurn`, and `talk.session.close` when they need captions or dictation without an assistant voice response.
- **Native macOS/iOS/Android Talk**: local speech recognition, Gateway chat, and `talk.speak` TTS. Nodes advertise the `talk` capability and declare which `talk.*` commands they support.
- **iOS Talk (realtime)**: client-owned WebRTC for OpenAI realtime configs that select `webrtc` transport or omit transport. Explicit `gateway-relay`, `provider-websocket`, and non-OpenAI realtime configs stay on the Gateway-owned relay; non-realtime configs use the native speech loop.
- **Browser Talk**: `talk.client.create` for client-owned `webrtc`/`provider-websocket` sessions, or `talk.session.create` for Gateway-owned `gateway-relay` sessions. `managed-room` is reserved for Gateway handoff and walkie-talkie rooms.
- **Android Talk (realtime)**: opt in with `talk.realtime.mode: "realtime"` and `talk.realtime.transport: "gateway-relay"`. Otherwise Android stays on native speech recognition, Gateway chat, and `talk.speak`.
- **Transcription-only clients**: `talk.session.create({ mode: "transcription", transport: "gateway-relay", brain: "none" })`, then `talk.session.appendAudio`, `talk.session.cancelTurn`, and `talk.session.close` for captions/dictation without an assistant voice response. One-shot uploaded voice notes still use the [media understanding](/nodes/media-understanding) audio path.
Native Talk is a continuous voice conversation loop:
Native Talk is a continuous loop: listen for speech, send the transcript to the model through the active session, wait for the response, then speak it via the configured Talk provider (`talk.speak`).
1. Listen for speech
2. Send transcript to the model through the active session
3. Wait for the response
4. Speak it via the configured Talk provider (`talk.speak`)
Client-owned realtime Talk forwards provider tool calls through `talk.client.toolCall` instead of calling `chat.send` directly. While a realtime consult is active, clients can call `talk.client.steer` or `talk.session.steer` to classify spoken input as `status`, `steer`, `cancel`, or `followup`. Accepted steering queues into the active embedded run; rejected steering returns a reason such as `no_active_run`, `not_streaming`, or `compacting`.
Client-owned realtime Talk forwards provider tool calls through `talk.client.toolCall`; those clients do not call `chat.send` directly for realtime consults.
While a realtime consult is active, Talk clients can use `talk.client.steer` or
`talk.session.steer` to classify spoken input as `status`, `steer`, `cancel`, or
`followup`. Accepted steering is queued into the active embedded run; rejected
steering returns a structured reason such as `no_active_run`, `not_streaming`,
or `compacting`.
Transcription-only Talk emits the same common Talk event envelope as realtime and STT/TTS sessions, but uses `mode: "transcription"` and `brain: "none"`. It is for captions, dictation, and observe-only speech capture; one-shot uploaded voice notes still use the media/audio path.
Transcription-only Talk emits the same Talk event envelope as realtime and STT/TTS sessions, but uses `mode: "transcription"` and `brain: "none"`. All Talk sessions broadcast events on the `talk.event` channel; clients subscribe to it for partial/final transcript updates (`transcript.delta`/`transcript.done`) and other session telemetry.
## Behavior (macOS)
- **Always-on overlay** while Talk mode is enabled.
- **Listening → Thinking → Speaking** phase transitions.
- On a **short pause** (silence window), the current transcript is sent.
- Replies are **written to WebChat** (same as typing).
- **Interrupt on speech** (default on): if the user starts talking while the assistant is speaking, we stop playback and note the interruption timestamp for the next prompt.
- Always-on overlay while Talk mode is enabled.
- **Listening &rarr; Thinking &rarr; Speaking** phase transitions.
- On a short pause (silence window), the current transcript is sent.
- Replies are written to WebChat (same as typing).
- **Interrupt on speech** (default on): if the user talks while the assistant is speaking, playback stops and the interruption timestamp is noted for the next prompt.
## Voice directives in replies
The assistant may prefix its reply with a **single JSON line** to control voice:
The assistant can prefix a reply with a single JSON line to control voice:
```json
{ "voice": "<voice-id>", "once": true }
@ -48,19 +38,11 @@ The assistant may prefix its reply with a **single JSON line** to control voice:
Rules:
- First non-empty line only.
- First non-empty line only; the JSON line is stripped before TTS playback.
- Unknown keys are ignored.
- `once: true` applies to the current reply only.
- Without `once`, the voice becomes the new default for Talk mode.
- The JSON line is stripped before TTS playback.
- `once: true` applies to the current reply only; without it, the voice becomes the new Talk mode default.
Supported keys:
- `voice` / `voice_id` / `voiceId`
- `model` / `model_id` / `modelId`
- `speed`, `rate` (WPM), `stability`, `similarity`, `style`, `speakerBoost`
- `seed`, `normalize`, `lang`, `output_format`, `latency_tier`
- `once`
Supported keys: `voice` / `voice_id` / `voiceId`, `model` / `model_id` / `modelId`, `speed`, `rate` (WPM), `stability`, `similarity`, `style`, `speakerBoost`, `seed`, `normalize`, `lang`, `output_format`, `latency_tier`, `once`.
## Config (`~/.openclaw/openclaw.json`)
@ -89,7 +71,7 @@ Supported keys:
openai: {
apiKey: "openai_api_key",
model: "gpt-realtime-2",
voice: "cedar",
speakerVoice: "cedar",
},
},
instructions: "Speak warmly and keep answers brief.",
@ -101,59 +83,51 @@ Supported keys:
}
```
Defaults:
| Key | Default | Notes |
| ---------------------------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `provider` | - | Active Talk TTS provider. Use `elevenlabs`, `mlx`, or `system` for macOS-local playback paths. |
| `providers.<id>.voiceId` | - | ElevenLabs falls back to `ELEVENLABS_VOICE_ID` / `SAG_VOICE_ID`, or the first available voice with an API key. |
| `providers.elevenlabs.modelId` | `eleven_v3` | |
| `providers.mlx.modelId` | `mlx-community/Soprano-80M-bf16` | |
| `providers.elevenlabs.apiKey` | - | Falls back to `ELEVENLABS_API_KEY` (or gateway shell profile if available). |
| `speechLocale` | device default | BCP 47 locale id for on-device Talk speech recognition on iOS/macOS. |
| `silenceTimeoutMs` | `700` ms macOS/Android, `900` ms iOS | Pause window before Talk sends the transcript. |
| `interruptOnSpeech` | `true` | |
| `outputFormat` | `pcm_44100` macOS/iOS, `pcm_24000` Android | Set `mp3_*` to force MP3 streaming. |
| `consultThinkingLevel` | unset | Thinking level override for the agent run behind realtime `openclaw_agent_consult` calls. |
| `consultFastMode` | unset | Fast-mode override for realtime `openclaw_agent_consult` calls. |
| `realtime.provider` | - | `openai` for WebRTC, `google` for provider WebSocket, or a bridge-only provider through Gateway relay. |
| `realtime.providers.<id>` | - | Provider-owned realtime config. Browsers receive only ephemeral/constrained session credentials, never a standard API key. |
| `realtime.providers.openai.speakerVoice` | `alloy` | Built-in OpenAI Realtime voice id (the older `voice` key still works but is deprecated). Current `gpt-realtime-2` voices: `alloy`, `ash`, `ballad`, `cedar`, `coral`, `echo`, `marin`, `sage`, `shimmer`, `verse`; `marin` and `cedar` are recommended for best quality. |
| `realtime.transport` | - | `webrtc`: client-owned OpenAI WebRTC on iOS and in the browser. `provider-websocket`: browser-owned, stays on Gateway relay on iOS. `gateway-relay`: keeps provider audio on the Gateway; Android uses realtime only with this transport. |
| `realtime.brain` | - | `agent-consult` routes realtime tool calls through Gateway policy; `direct-tools` is legacy direct-tool compatibility; `none` is for transcription/external orchestration. |
| `realtime.consultRouting` | - | `provider-direct` preserves the provider's direct reply when it skips `openclaw_agent_consult`; `force-agent-consult` routes finalized user transcripts through OpenClaw instead. |
| `realtime.instructions` | - | Appends provider-facing system instructions to OpenClaw's built-in realtime prompt (voice style/tone); the default `openclaw_agent_consult` guidance stays. |
- `interruptOnSpeech`: true
- `silenceTimeoutMs`: when unset, Talk keeps the platform default pause window before sending the transcript (`700 ms on macOS and Android, 900 ms on iOS`)
- `provider`: selects the active Talk provider. Use `elevenlabs`, `mlx`, or `system` for the macOS-local playback paths.
- `providers.<provider>.voiceId`: falls back to `ELEVENLABS_VOICE_ID` / `SAG_VOICE_ID` for ElevenLabs (or first ElevenLabs voice when API key is available).
- `providers.elevenlabs.modelId`: defaults to `eleven_v3` when unset.
- `providers.mlx.modelId`: defaults to `mlx-community/Soprano-80M-bf16` when unset.
- `providers.elevenlabs.apiKey`: falls back to `ELEVENLABS_API_KEY` (or gateway shell profile if available).
- `consultThinkingLevel`: optional thinking level override for the full OpenClaw agent run behind realtime `openclaw_agent_consult` calls.
- `consultFastMode`: optional fast-mode override for realtime `openclaw_agent_consult` calls.
- `realtime.provider`: selects the active realtime voice provider. Use `openai` for WebRTC, `google` for provider WebSocket, or a bridge-only provider through Gateway relay.
- `realtime.providers.<provider>` stores provider-owned realtime config. The browser receives only ephemeral or constrained session credentials, never a standard API key.
- `realtime.providers.openai.voice`: built-in OpenAI Realtime voice id. Current `gpt-realtime-2` voices are `alloy`, `ash`, `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`, `marin`, and `cedar`; `marin` and `cedar` are recommended for best quality.
- `realtime.transport`: `webrtc` uses client-owned OpenAI WebRTC on iOS and in the browser. `provider-websocket` is browser-owned but stays on the Gateway relay on iOS. `gateway-relay` keeps provider audio on the Gateway; Android uses realtime only for this transport and otherwise keeps its native STT/TTS loop.
- `realtime.brain`: `agent-consult` routes realtime tool calls through Gateway policy; `direct-tools` is legacy direct-tool compatibility behavior; `none` is for transcription or external orchestration.
- `realtime.consultRouting`: `provider-direct` preserves the provider's direct reply when it skips `openclaw_agent_consult`; `force-agent-consult` makes Gateway relay route finalized user transcripts through OpenClaw instead.
- `realtime.instructions`: appends provider-facing system instructions to OpenClaw's built-in realtime prompt. Use it for voice style and tone; OpenClaw keeps the default `openclaw_agent_consult` guidance.
- `talk.catalog` exposes canonical provider ids and registry aliases alongside each provider's valid modes, transports, brain strategies, realtime audio formats, capability flags, and the runtime-selected readiness result. First-party Talk clients should use that catalog instead of maintaining provider aliases locally; an older Gateway that omits group readiness is unverified rather than definitively unconfigured.
- Streaming transcription providers are discovered through `talk.catalog.transcription`. The current Gateway relay uses the Voice Call streaming provider config until the dedicated Talk transcription config surface is added.
- `speechLocale`: optional BCP 47 locale id for on-device Talk speech recognition on iOS/macOS. Leave unset to use the device default.
- `outputFormat`: defaults to `pcm_44100` on macOS/iOS and `pcm_24000` on Android (set `mp3_*` to force MP3 streaming)
`talk.catalog` exposes canonical provider ids and registry aliases, each provider's valid modes/transports/brain strategies/realtime audio formats/capability flags, and the runtime-selected readiness result. First-party Talk clients should read that catalog instead of maintaining provider aliases locally; treat an older Gateway that omits group readiness as unverified rather than definitively unconfigured. Streaming transcription providers are discovered through `talk.catalog.transcription`; the current Gateway relay uses the Voice Call streaming provider config until a dedicated Talk transcription config surface ships.
## macOS UI
- Menu bar toggle: **Talk**
- Config tab: **Talk Mode** group (voice id + interrupt toggle)
- Overlay:
- **Listening**: cloud pulses with mic level
- **Thinking**: sinking animation
- **Speaking**: radiating rings
- Click cloud: stop speaking
- Click X: exit Talk mode
- Overlay: Listening (cloud pulses with mic level) &rarr; Thinking (sinking animation) &rarr; Speaking (radiating rings). Click the cloud to stop speaking, click X to exit Talk mode.
## Android UI
- Voice tab toggle: **Talk**
- Manual **Mic** and **Talk** are mutually exclusive runtime capture modes.
- Manual Mic and realtime Talk prefer a connected Bluetooth Classic or BLE headset microphone. If it disconnects, the app requests another headset input or lets Android use the default microphone; stopping capture restores the default microphone preference.
- Manual **Mic** and **Talk** are mutually exclusive capture modes.
- Manual Mic and realtime Talk prefer a connected Bluetooth Classic or BLE headset microphone; if it disconnects, the app requests another headset input or falls back to the default microphone, restoring the default preference once capture stops.
- Manual Mic stops when the app leaves the foreground or the user leaves the Voice tab.
- Talk Mode keeps running until toggled off or the Android node disconnects, and uses Android's microphone foreground-service type while active.
- Talk Mode keeps running until toggled off or the node disconnects, using Android's microphone foreground-service type while active.
- Android supports `pcm_16000`, `pcm_22050`, `pcm_24000`, and `pcm_44100` output formats for low-latency `AudioTrack` streaming.
## Notes
- Requires Speech + Microphone permissions.
- Native Talk uses the active Gateway session and only falls back to history polling when response events are unavailable.
- Client-owned realtime Talk uses `talk.client.toolCall` for `openclaw_agent_consult` instead of exposing `chat.send` to provider-owned sessions.
- Transcription-only Talk uses `talk.session.create`, `talk.session.appendAudio`, `talk.session.cancelTurn`, and `talk.session.close`; clients subscribe to `talk.event` for partial/final transcript updates.
- The gateway resolves Talk playback through `talk.speak` using the active Talk provider. Android falls back to local system TTS only when that RPC is unavailable.
- macOS local MLX playback uses the bundled `openclaw-mlx-tts` helper when present, or an executable on `PATH`. Set `OPENCLAW_MLX_TTS_BIN` to point at a custom helper binary during development.
- `stability` for `eleven_v3` is validated to `0.0`, `0.5`, or `1.0`; other models accept `0..1`.
- `latency_tier` is validated to `0..4` when set.
- Android supports `pcm_16000`, `pcm_22050`, `pcm_24000`, and `pcm_44100` output formats for low-latency AudioTrack streaming.
- Voice directive value ranges (ElevenLabs): `stability`, `similarity`, and `style` accept `0..1`; `speed` accepts `0.5..2`; `latency_tier` accepts `0..4`.
## Related

49
docs/nodes/troubleshooting.md
View file

@ -18,7 +18,7 @@ openclaw doctor
openclaw channels status --probe
```
Then run node specific checks:
Then run node-specific checks:
```bash
openclaw nodes status
@ -29,12 +29,12 @@ openclaw approvals get --node <idOrNameOrIp>
Healthy signals:
- Node is connected and paired for role `node`.
- `nodes describe` includes the capability you are calling.
- Exec approvals show expected mode/allowlist.
- `nodes describe` includes the capability you're calling.
- Exec approvals show the expected mode/allowlist.
## Foreground requirements
`canvas.*`, `camera.*`, and `screen.*` are foreground only on iOS/Android nodes.
`canvas.*`, `camera.*`, and `screen.*` are foreground-only on iOS/Android nodes.
Quick check and fix:
@ -57,12 +57,14 @@ If you see `NODE_BACKGROUND_UNAVAILABLE`, bring the node app to the foreground a
## Pairing versus approvals
These are different gates:
Three separate gates control whether a node command succeeds:
1. **Device pairing**: can this node connect to the gateway?
2. **Gateway node command policy**: is the RPC command ID allowed by `gateway.nodes.allowCommands` / `denyCommands` and platform defaults?
3. **Exec approvals**: can this node run a specific shell command locally?
Node pairing is an identity/trust gate, not a per-command approval surface. For `system.run`, the per-node policy lives in that node's exec approvals file (`openclaw approvals get --node ...`), not in the gateway pairing record.
Quick checks:
```bash
@ -72,29 +74,24 @@ openclaw approvals get --node <idOrNameOrIp>
openclaw approvals allowlist add --node <idOrNameOrIp> "/usr/bin/uname"
```
If pairing is missing, approve the node device first.
If `nodes describe` is missing a command, check the gateway node command policy and whether the node actually declared that command on connect.
If pairing is fine but `system.run` fails, fix exec approvals/allowlist on that node.
- Pairing missing: approve the node device first.
- `nodes describe` missing a command: check the gateway node command policy and whether the node actually declared that command on connect.
- Pairing fine but `system.run` fails: fix exec approvals/allowlist on that node.
Node pairing is an identity/trust gate, not a per-command approval surface. For `system.run`, the per-node policy lives in that node's exec approvals file (`openclaw approvals get --node ...`), not in the gateway pairing record.
For approval-backed `host=node` runs, the gateway also binds execution to the
prepared canonical `systemRunPlan`. If a later caller mutates command/cwd or
session metadata before the approved run is forwarded, the gateway rejects the
run as an approval mismatch instead of trusting the edited payload.
For approval-backed `host=node` runs, the gateway also binds execution to the prepared canonical `systemRunPlan`. If a later caller mutates the command, cwd, or session metadata before the approved run is forwarded, the gateway rejects the run as an approval mismatch instead of trusting the edited payload.
## Common node error codes
- `NODE_BACKGROUND_UNAVAILABLE` → app is backgrounded; bring it foreground.
- `CAMERA_DISABLED` → camera toggle disabled in node settings.
- `*_PERMISSION_REQUIRED` → OS permission missing/denied.
- `LOCATION_DISABLED` → location mode is off.
- `LOCATION_PERMISSION_REQUIRED` → requested location mode not granted.
- `LOCATION_BACKGROUND_UNAVAILABLE` → app is backgrounded but only While Using permission exists.
- `SYSTEM_RUN_DENIED: approval required` → exec request needs explicit approval.
- `SYSTEM_RUN_DENIED: allowlist miss` → command blocked by allowlist mode.
On Windows node hosts, shell-wrapper forms like `cmd.exe /c ...` are treated as allowlist misses in
allowlist mode unless approved via ask flow.
| Code | Meaning |
| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `NODE_BACKGROUND_UNAVAILABLE` | App is backgrounded; bring it to the foreground. |
| `CAMERA_DISABLED` | Camera toggle disabled in node settings. |
| `*_PERMISSION_REQUIRED` | OS permission missing/denied. |
| `LOCATION_DISABLED` | Location mode is off. |
| `LOCATION_PERMISSION_REQUIRED` | Requested location mode not granted. |
| `LOCATION_BACKGROUND_UNAVAILABLE` | App is backgrounded but only While Using permission exists. |
| `SYSTEM_RUN_DENIED: approval required` | Exec request needs explicit approval. |
| `SYSTEM_RUN_DENIED: allowlist miss` | Command blocked by allowlist mode. On Windows node hosts, shell-wrapper forms like `cmd.exe /c ...` are treated as allowlist misses in allowlist mode unless approved via the ask flow. |
## Fast recovery loop
@ -108,9 +105,9 @@ openclaw logs --follow
If still stuck:
- Re-approve device pairing.
- Re-open node app (foreground).
- Re-open the node app (foreground).
- Re-grant OS permissions.
- Recreate/adjust exec approval policy.
- Recreate/adjust the exec approval policy.
## Related

79
docs/nodes/voicewake.md
View file

@ -6,46 +6,33 @@ read_when:
title: "Voice wake"
---
OpenClaw treats **wake words as a single global list** owned by the **Gateway**.
Wake words are **one global list owned by the Gateway** — there are no per-node custom lists. Any node or app UI can edit the list; the Gateway persists the change and broadcasts it to every connected client.
- There are **no per-node custom wake words**.
- **Any node/app UI may edit** the list; changes are persisted by the Gateway and broadcast to everyone.
- macOS and iOS keep local **Voice Wake enabled/disabled** toggles (local UX + permissions differ).
- Android currently keeps Voice Wake off and uses a manual mic flow in the Voice tab.
- **macOS**: local Voice Wake enable/disable toggle. Requires macOS 26+; see [Voice wake (macOS)](/platforms/mac/voicewake) for runtime/PTT details.
- **iOS**: local Voice Wake enable/disable toggle in Settings.
- **Android**: Voice Wake is force-disabled at runtime. The Voice tab uses manual mic capture instead of wake-word triggers.
## Storage (Gateway host)
## Storage
Wake words and routing rules are stored in the gateway state database:
- `~/.openclaw/state/openclaw.sqlite`
The active tables are:
- `voicewake_triggers`
- `voicewake_routing_config`
- `voicewake_routing_routes`
Legacy `settings/voicewake.json` and `settings/voicewake-routing.json` files are
doctor migration inputs only; runtime reads and writes the SQLite tables.
Wake words and routing rules live in the Gateway state database, `~/.openclaw/state/openclaw.sqlite` by default (override with `OPENCLAW_STATE_DIR`), tables `voicewake_triggers`, `voicewake_routing_config`, `voicewake_routing_routes`. Legacy `settings/voicewake.json` and `settings/voicewake-routing.json` are `openclaw doctor --fix` migration inputs only — runtime never reads them.
## Protocol
### Methods
### Trigger list
- `voicewake.get` → `{ triggers: string[] }`
- `voicewake.set` with params `{ triggers: string[] }` → `{ triggers: string[] }`
| Method | Params | Result |
| --------------- | ------------------------ | ------------------------ |
| `voicewake.get` | none | `{ triggers: string[] }` |
| `voicewake.set` | `{ triggers: string[] }` | `{ triggers: string[] }` |
Notes:
`voicewake.set` normalizes input: trims whitespace, drops empty entries, keeps at most 32 triggers, truncates each to 64 characters. An empty result falls back to the built-in defaults (`openclaw`, `claude`, `computer`).
- Triggers are normalized (trimmed, empties dropped). Empty lists fall back to defaults.
- Limits are enforced for safety (count/length caps).
### Routing (trigger to target)
### Routing methods (trigger → target)
- `voicewake.routing.get` → `{ config: VoiceWakeRoutingConfig }`
- `voicewake.routing.set` with params `{ config: VoiceWakeRoutingConfig }` → `{ config: VoiceWakeRoutingConfig }`
`VoiceWakeRoutingConfig` shape:
| Method | Params | Result |
| ----------------------- | ------------------------------------ | ------------------------------------ |
| `voicewake.routing.get` | none | `{ config: VoiceWakeRoutingConfig }` |
| `voicewake.routing.set` | `{ config: VoiceWakeRoutingConfig }` | `{ config: VoiceWakeRoutingConfig }` |
```json
{
@ -56,38 +43,28 @@ Notes:
}
```
Route targets support exactly one of:
Each route `target` supports exactly one of:
- `{ "mode": "current" }`
- `{ "agentId": "main" }`
- `{ "sessionKey": "agent:main:main" }`
Limits: at most 32 routes, trigger text at most 64 characters. Route triggers are normalized for matching and duplicate detection by lowercasing, stripping leading/trailing punctuation from each word, and collapsing whitespace (`"Hey, Bot!!"` and `"hey bot"` match and count as duplicates) — this is a stricter normalization than the plain trim used for the global trigger list above.
### Events
- `voicewake.changed` payload `{ triggers: string[] }`
- `voicewake.routing.changed` payload `{ config: VoiceWakeRoutingConfig }`
| Event | Payload |
| --------------------------- | ------------------------------------ |
| `voicewake.changed` | `{ triggers: string[] }` |
| `voicewake.routing.changed` | `{ config: VoiceWakeRoutingConfig }` |
Who receives it:
- All WebSocket clients (macOS app, WebChat, etc.)
- All connected nodes (iOS/Android), and also on node connect as an initial "current state" push.
Both broadcast to every WebSocket client with read scope (macOS app, WebChat, and similar) and to every connected node. A node also gets both as an initial snapshot push right after it connects.
## Client behavior
### macOS app
- Uses the global list to gate `VoiceWakeRuntime` triggers.
- Editing "Trigger words" in Voice Wake settings calls `voicewake.set` and then relies on the broadcast to keep other clients in sync.
### iOS node
- Uses the global list for `VoiceWakeManager` trigger detection.
- Editing Wake Words in Settings calls `voicewake.set` (over the Gateway WS) and also keeps local wake-word detection responsive.
### Android node
- Voice Wake is currently disabled in Android runtime/Settings.
- Android voice uses manual mic capture in the Voice tab instead of wake-word triggers.
- **macOS**: calls `voicewake.set`/`voicewake.get` and listens for `voicewake.changed` to stay in sync with other clients.
- **iOS**: calls `voicewake.set`/`voicewake.get` and listens for `voicewake.changed` to keep local wake-word detection responsive.
- **Android**: `VoiceWakeMode` (`Off`/`Foreground`/`Always`) and gateway sync code exist, but the app forces the mode to `Off` on startup — Voice Wake is not currently reachable from Android Settings.
## Related