openclaw/docs/nodes/media-understanding.md

summary	read_when	title	sidebarTitle
Inbound image/audio/video understanding (optional) with provider + CLI fallbacks	Designing or refactoring media understanding Tuning inbound audio/video/image preprocessing	Media understanding	Media understanding

OpenClaw can summarize inbound media (image/audio/video) before the reply pipeline runs, so command parsing and routing work off short text instead of raw bytes. Understanding auto-detects local tools or provider keys, or you can configure explicit models. Original media is always delivered to the model as usual; when understanding fails or is disabled, the reply flow continues unchanged.

Vendor plugins register capability metadata (which provider supports which media type, default model, priority). OpenClaw core owns the shared tools.media config, fallback order, and reply-pipeline integration.

How it works

Collect inbound attachments (`MediaPaths`, `MediaUrls`, `MediaTypes`). For each enabled capability (image/audio/video), select attachments per the `attachments` policy (default: first attachment only). Pick the first eligible model entry (size + capability + auth available). If a model errors, times out, or the media exceeds `maxBytes`, try the next entry. `Body` becomes an `[Image]`, `[Audio]`, or `[Video]` block. Audio also sets `{{Transcript}}`; command parsing uses caption text when present, otherwise the transcript. Captions are preserved as `User text:` inside the block.

Config

tools.media holds a shared model list plus per-capability overrides:

{
  tools: {
    media: {
      concurrency: 2, // max concurrent capability runs (default)
      models: [
        /* shared list, gate with capabilities */
      ],
      image: {
        /* optional overrides */
      },
      audio: {
        /* optional overrides */
        echoTranscript: true,
        echoFormat: '📝 "{transcript}"',
      },
      video: {
        /* optional overrides */
      },
    },
  },
}

Per-capability (image/audio/video) keys:

Key	Type	Default	Notes
enabled	boolean	auto (false disables)	Set false to turn off auto-detect for this capability
models	array	none	Preferred before the shared tools.media.models list
prompt	string	"Describe the {media}." (+ maxChars guidance)	Image/video only by default
maxChars	number	500 (image/video), unset (audio)	Output is trimmed if the model returns more
maxBytes	number	image 10485760, audio 20971520, video 52428800	Oversized media skips to the next model
timeoutSeconds	number	60 (image/audio), 120 (video)
language	string	unset	Audio transcription hint
baseUrl/headers/providerOptions/request	-	-	Provider request overrides; see Tools and custom providers
attachments	object	{ mode: "first", maxAttachments: 1 }	See Attachment policy
scope	object	unset	Gate by channel/chatType/keyPrefix
echoTranscript	boolean	false	Audio only: echo the transcript back to the chat before agent processing
echoFormat	string	'📝 "{transcript}"'	Audio only: {transcript} placeholder

Deepgram-specific options go under providerOptions.deepgram (the top-level deepgram: { detectLanguage, punctuate, smartFormat } field is deprecated but still read).

Model entries

Each models[] entry is a provider entry (default) or a CLI entry:

```json5 { type: "provider", // default if omitted provider: "openai", model: "gpt-5.5", prompt: "Describe the image in <= 500 chars.", maxChars: 500, maxBytes: 10485760, timeoutSeconds: 60, capabilities: ["image"], // optional, for multi-modal shared entries profile: "vision-profile", preferredProfile: "vision-fallback", } ``` ```json5 { type: "cli", command: "gemini", args: [ "-m", "gemini-3-flash", "--allowed-tools", "read_file", "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.", ], maxChars: 500, maxBytes: 52428800, timeoutSeconds: 120, capabilities: ["video", "image"], } ```

CLI templates can also use `{{MediaDir}}` (directory containing the media file), `{{OutputDir}}` (scratch dir created for this run), and `{{OutputBase}}` (scratch file base path, no extension).

Provider credentials

Provider media understanding uses the same auth resolution as normal model calls: auth profiles, environment variables, then models.providers.<providerId>.apiKey. tools.media.*.models[] entries do not accept an inline apiKey field.

{
  models: {
    providers: {
      openai: { apiKey: "<OPENAI_API_KEY>" },
      moonshot: { apiKey: "<MOONSHOT_API_KEY>" },
    },
  },
}

See Tools and custom providers for profiles, env vars, and custom base URLs.

Rules and behavior

• Media exceeding maxBytes skips that model and tries the next one.
• Audio files under 1024 bytes are treated as empty/corrupt and skipped before transcription; the agent gets a deterministic placeholder transcript instead.
• If the active primary image model already supports vision natively, OpenClaw skips the [Image] summary block and passes the original image into the model directly. MiniMax is an exception: minimax, minimax-cn, minimax-portal, and minimax-portal-cn always route image understanding through the plugin-owned MiniMax-VL-01 media provider, even if legacy MiniMax M2.x chat metadata claims image input (only MiniMax-M3 and later are treated as natively vision-capable).
• If a Gateway/WebChat primary model is text-only, image attachments are preserved as offloaded media://inbound/* refs so image/PDF tools or a configured image model can still inspect them instead of losing the attachment.
• Explicit openclaw infer image describe --file <path> --model <provider/model> (alias: openclaw capability image describe) runs that image-capable provider/model directly, including Ollama refs such as ollama/qwen2.5vl:7b when a matching image-capable model is configured under models.providers.ollama.models[].
• If <capability>.enabled is not false but no models are configured, OpenClaw tries the active reply model when its provider supports the capability.

Auto-detect (default)

When tools.media.<capability>.enabled is not false and no models are configured, OpenClaw tries these in order and stops at the first working option:

`agents.defaults.imageModel` primary/fallback refs, unless the active reply model already supports vision natively. Prefer `provider/model` refs; bare refs are qualified from configured image-capable provider model entries only when the match is unique. The active reply model, when its provider supports the capability. Configured `models.providers.*` entries that support audio are tried before local CLIs. Bundled provider priority order (ties break alphabetically by provider id): Groq/OpenAI → xAI → Deepgram → OpenRouter → Google/SenseAudio → Deepinfra/ElevenLabs → Mistral. First installed local binary, in this order: - `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with `tokens.txt`/`encoder.onnx`/`decoder.onnx`/`joiner.onnx`) - `whisper-cli` (`whisper-cpp`; uses `WHISPER_CPP_MODEL` or a bundled tiny model) - `whisper` (Python CLI; defaults to the `turbo` model, downloads automatically) Configured `models.providers.*` entries that support the capability are tried before the bundled fallback order. Image-only config providers with an image-capable model auto-register for media understanding even when they are not a bundled vendor plugin.

Bundled provider priority order (ties break alphabetically by provider id):
- Image: Anthropic/OpenAI → Google → MiniMax → Deepinfra → MiniMax Portal → Z.AI
- Video: Google → Qwen → Moonshot

First installed `agy` or `antigravity` binary (override with `OPENCLAW_ANTIGRAVITY_CLI`), sandboxed against the media's directory.

To disable auto-detection for a capability:

{
  tools: {
    media: {
      audio: {
        enabled: false,
      },
    },
  },
}

Binary detection is best-effort across macOS/Linux/Windows; ensure the CLI is on `PATH` (`~` is expanded), or set an explicit CLI model entry with a full command path.

Proxy support (audio/video provider calls)

Provider-based audio and video understanding honors standard outbound proxy environment variables, including NO_PROXY/no_proxy bypass rules: HTTPS_PROXY, HTTP_PROXY, ALL_PROXY, https_proxy, http_proxy, all_proxy. Lowercase vars take precedence over uppercase. If none are set, media understanding uses direct egress; if the proxy value is malformed, OpenClaw logs a warning and falls back to direct fetch. Image understanding does not go through this proxy path.

Capabilities

Set capabilities on a models[] entry to restrict it to specific media types. For shared lists, OpenClaw infers defaults per bundled provider:

Provider	Capabilities
openai, anthropic, minimax	image
minimax-portal	image
moonshot	image + video
openrouter	image + audio
google (Gemini API)	image + audio + video
qwen	image + video
deepinfra	image + audio
mistral	audio
zai	image
groq, xai, deepgram, senseaudio	audio
Any models.providers.<id>.models[] catalog with an image-capable model	image

For CLI entries, set capabilities explicitly to avoid surprising matches; if omitted, the entry is eligible for every capability list it appears in.

Provider support matrix

Capability	Providers	Notes
Image	Anthropic, Codex app-server, Deepinfra, Google, MiniMax, MiniMax Portal, Moonshot, OpenAI, OpenAI Codex OAuth, OpenRouter, Qwen, Z.AI, config providers	Vendor plugins register image support; openai/* can use API-key or Codex OAuth routing; codex/* uses a bounded Codex app-server turn; image-capable config providers auto-register.
Audio	Deepgram, Deepinfra, ElevenLabs, Google, Groq, Mistral, OpenAI, OpenRouter, SenseAudio, xAI	Provider transcription (Whisper/Groq/xAI/Deepgram/OpenRouter STT/Gemini/SenseAudio/Scribe/Voxtral).
Video	Google, Moonshot, Qwen	Provider video understanding via vendor plugins; Qwen video understanding uses the standard DashScope endpoints.

**MiniMax note**: `minimax`, `minimax-cn`, `minimax-portal`, and `minimax-portal-cn` image understanding always comes from the plugin-owned `MiniMax-VL-01` media provider, even if legacy MiniMax M2.x chat metadata claims image input.

Model selection guidance

• Prefer the strongest current-generation model for each media capability when quality and safety matter.
• For tool-enabled agents handling untrusted inputs, avoid older/weaker media models.
• Keep at least one fallback per capability for availability (quality model + faster/cheaper model).
• CLI fallbacks (whisper-cli, whisper, gemini) help when provider APIs are unavailable.
• parakeet-mlx: with --output-dir, OpenClaw reads <output-dir>/<media-basename>.txt when the output format is txt or unspecified; other formats fall back to stdout.

Attachment policy

Per-capability attachments controls which attachments are processed:

Process only the first selected attachment, or all of them. Cap the number processed. Selection preference among candidate attachments.

When mode: "all", outputs are labeled [Image 1/2], [Audio 2/2], etc.

File-attachment extraction

• Extracted file text is wrapped as untrusted external content before it's appended to the media prompt, using boundary markers like <<<EXTERNAL_UNTRUSTED_CONTENT id="...">>> / <<<END_EXTERNAL_UNTRUSTED_CONTENT id="...">>> plus a Source: External metadata line.
• This path intentionally omits the long SECURITY NOTICE: banner to keep the media prompt short; the boundary markers and metadata still apply.
• A file with no extractable text gets [No extractable text].
• If a PDF falls back to rendered page images, OpenClaw forwards those images to vision-capable reply models and keeps the placeholder [PDF content rendered to images] in the file block.

Config examples

```json5 { tools: { media: { models: [ { provider: "openai", model: "gpt-5.5", capabilities: ["image"] }, { provider: "google", model: "gemini-3-flash-preview", capabilities: ["image", "audio", "video"], }, { type: "cli", command: "gemini", args: [ "-m", "gemini-3-flash", "--allowed-tools", "read_file", "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.", ], capabilities: ["image", "video"], }, ], audio: { attachments: { mode: "all", maxAttachments: 2 }, }, video: { maxChars: 500, }, }, }, } ``` ```json5 { tools: { media: { audio: { enabled: true, models: [ { provider: "openai", model: "gpt-4o-mini-transcribe" }, { type: "cli", command: "whisper", args: ["--model", "base", "{{MediaPath}}"], }, ], }, video: { enabled: true, maxChars: 500, models: [ { provider: "google", model: "gemini-3-flash-preview" }, { type: "cli", command: "gemini", args: [ "-m", "gemini-3-flash", "--allowed-tools", "read_file", "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.", ], }, ], }, }, }, } ``` ```json5 { tools: { media: { image: { enabled: true, maxBytes: 10485760, maxChars: 500, models: [ { provider: "openai", model: "gpt-5.5" }, { provider: "anthropic", model: "claude-opus-4-8" }, { type: "cli", command: "gemini", args: [ "-m", "gemini-3-flash", "--allowed-tools", "read_file", "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.", ], }, ], }, }, }, } ``` ```json5 { tools: { media: { image: { models: [ { provider: "google", model: "gemini-3.1-pro-preview", capabilities: ["image", "video", "audio"], }, ], }, audio: { models: [ { provider: "google", model: "gemini-3.1-pro-preview", capabilities: ["image", "video", "audio"], }, ], }, video: { models: [ { provider: "google", model: "gemini-3.1-pro-preview", capabilities: ["image", "video", "audio"], }, ], }, }, }, } ```

Status output

When media understanding runs, /status includes a per-capability summary line:

📎 Media: image ok (openai/gpt-5.5) · audio skipped (maxBytes)

Notes

• Understanding is best-effort. Errors do not block replies.
• Attachments are still passed to models even when understanding is disabled.
• Use scope to limit where understanding runs (for example, only DMs).

Related

• Configuration
• Image & media support