openclaw/docs/plugins/sdk-runtime/media.md

summary	read_when	title	sidebarTitle
Speech, media understanding, image/video/music generation, web search, and media utilities	You are synthesizing speech or transcribing inbound audio You are generating images, video, or music from a plugin You need MIME detection, resizing, or QR helpers	Plugin runtime media helpers	Media helpers

Speech, media understanding, generation, web search, and the low-level media utilities. Part of the Plugin runtime helpers reference.

FFmpeg command discovery

Official plugin sources use resolveFfmpegBin from the private openclaw/plugin-sdk/media-ffmpeg runtime. It resolves the same trusted system paths as the media runtime and throws an installation hint when FFmpeg is unavailable. This narrow entry point keeps media generation and agent runtimes out of source audio worker startup. It has a JavaScript-only host export; its declarations are excluded from the package. Third-party plugins retain the existing resolveFfmpegBin export from openclaw/plugin-sdk/media-runtime.

The official plugin publication builder emits media-runtime for this private import, including worker entries, so published plugins keep working on supported hosts that predate media-ffmpeg. Both paths use the host's FFmpeg resolver.

Realtime voice playback

Official audio workers use createRealtimeVoiceOutputActivityTracker and isRealtimeVoiceAudioAudible from the private openclaw/plugin-sdk/realtime-voice-playback runtime to avoid loading voice session and agent runtimes from source. The publication builder emits the existing openclaw/plugin-sdk/realtime-voice host import for these helpers, including worker entries, to preserve supported hosts. The source facade has a JavaScript-only host export; its declarations are excluded from the package.

Media and generation namespaces

Text-to-speech synthesis.

```typescript
// Standard TTS
const clip = await api.runtime.tts.textToSpeech({
  text: "Hello from OpenClaw",
  cfg: api.config,
});

// Telephony-optimized TTS
const telephonyClip = await api.runtime.tts.textToSpeechTelephony({
  text: "Hello from OpenClaw",
  cfg: api.config,
});

// List available voices
const voices = await api.runtime.tts.listVoices({
  provider: "elevenlabs",
  cfg: api.config,
});
```

Uses core `tts` configuration and provider selection. Returns PCM audio buffer + sample rate. `textToSpeechStream` is also available for streaming synthesis.

Image, audio, and video analysis.

```typescript
// Describe an image
const image = await api.runtime.mediaUnderstanding.describeImageFile({
  filePath: "/tmp/inbound-photo.jpg",
  cfg: api.config,
  agentDir: "/tmp/agent",
});

// Prepare a capture limit before installing audio receive listeners.
const budget = await api.runtime.mediaUnderstanding.resolveAudioInputBudget({
  cfg: api.config,
});
// budget.enabled is false when audio understanding is disabled. Otherwise,
// budget.maxBytes includes the container header and covers the largest fallback.

// Transcribe audio
const { text } = await api.runtime.mediaUnderstanding.transcribeAudioFile({
  filePath: "/tmp/inbound-audio.ogg",
  cfg: api.config,
  mime: "audio/ogg", // optional, for when MIME cannot be inferred
});

// Describe a video
const video = await api.runtime.mediaUnderstanding.describeVideoFile({
  filePath: "/tmp/inbound-video.mp4",
  cfg: api.config,
});

// Generic file analysis
const result = await api.runtime.mediaUnderstanding.runFile({
  filePath: "/tmp/inbound-file.pdf",
  cfg: api.config,
});

// Structured image extraction through a specific provider/model.
// Include at least one image; text inputs are supplemental context.
// receiptImageBuffer is your own image bytes, not an SDK-provided value.
const evidence = await api.runtime.mediaUnderstanding.extractStructuredWithModel({
  provider: "codex",
  model: "gpt-6-astra",
  input: [
    {
      type: "image",
      buffer: receiptImageBuffer,
      fileName: "receipt.png",
      mime: "image/png",
    },
    { type: "text", text: "Prefer the printed total over handwritten notes." },
  ],
  instructions: "Extract vendor, total, and searchable tags.",
  schemaName: "receipt.evidence",
  jsonSchema: {
    type: "object",
    properties: {
      vendor: { type: "string" },
      total: { type: "number" },
      tags: { type: "array", items: { type: "string" } },
    },
    required: ["vendor", "total"],
  },
  cfg: api.config,
});
```

Returns `{ text: undefined }` when no output is produced (e.g. skipped input).

`describeImageFileWithModel(...)` describes an already-known image through a specific provider/model, bypassing the default active-model resolution that `describeImageFile(...)` uses.

Image generation.

```typescript
const result = await api.runtime.imageGeneration.generate({
  prompt: "A robot painting a sunset",
  cfg: api.config,
});

const providers = api.runtime.imageGeneration.listProviders({ cfg: api.config });
```

Video generation, mirroring the image generation shape.

```typescript
const result = await api.runtime.videoGeneration.generate({
  prompt: "A drone shot flying over a coastline at sunrise",
  cfg: api.config,
});

const providers = api.runtime.videoGeneration.listProviders({ cfg: api.config });
```

Music generation, mirroring the image generation shape.

```typescript
const result = await api.runtime.musicGeneration.generate({
  prompt: "An upbeat lo-fi track for a coding session",
  cfg: api.config,
});

const providers = api.runtime.musicGeneration.listProviders({ cfg: api.config });
```

Web search.

```typescript
const providers = api.runtime.webSearch.listProviders({ config: api.config });

const result = await api.runtime.webSearch.search({
  config: api.config,
  args: { query: "OpenClaw plugin SDK", count: 5 },
});
```

Search callers may supply a synchronous `assertCurrent` callback with `signal`
to retain their authority through provider preparation. Guarded HTTP requests
check it after transport preparation and before each request or redirect.
A registered search provider using another transport must call the execution
context's `assertCurrent` before each external side effect, after awaited
preparation. The callback belongs to the caller and expires when the search
finishes; providers must not replace it or treat its absence as permission.

Low-level media utilities.

```typescript
const webMedia = await api.runtime.media.loadWebMedia(url);
const mime = await api.runtime.media.detectMime({ buffer });
const kind = api.runtime.media.mediaKindFromMime("image/jpeg"); // "image"
const isVoice = api.runtime.media.isVoiceCompatibleAudio({ fileName: filePath });
const metadata = await api.runtime.media.getImageMetadata(buffer);
const resized = await api.runtime.media.resizeToJpeg({ buffer, maxSide: 800, quality: 85 });
```

QR helpers are exported by `openclaw/plugin-sdk/media-runtime`:

```typescript
import { resolvePreferredOpenClawTmpDir } from "openclaw/plugin-sdk/temp-path";

const qr = await import("openclaw/plugin-sdk/media-runtime");
const terminalQr = await qr.renderQrTerminal("https://openclaw.ai");
const pngQr = await qr.renderQrPngBase64("https://openclaw.ai", {
  scale: 6, // 1-12
  marginModules: 4, // 0-16
});
const pngQrDataUrl = await qr.renderQrPngDataUrl("https://openclaw.ai");
const tmpRoot = resolvePreferredOpenClawTmpDir();
const pngQrFile = await qr.writeQrPngTempFile("https://openclaw.ai", {
  tmpRoot,
  dirPrefix: "my-plugin-qr-",
  fileName: "qr.png",
});
```