mirror of
https://github.com/openclaw/openclaw.git
synced 2026-10-10 21:41:43 +00:00
Missing prerequisites for two test lanes, a copy-paste slip in an SDK sample, three undeclared identifiers in a quick start, a colon promising keys named two paragraphs later, duplicated Related lists, two unlinked pages that exist, an unbracketed placeholder, and a bare doctor invocation. The thirteenth fix, an ffmpeg prerequisite on docs/tools/tts/quickstart.md, is held back: every PR touching that page is refused by the secret scanner before review.
592 lines
31 KiB
Markdown
592 lines
31 KiB
Markdown
---
|
|
summary: "OpenClaw browser control API, CLI reference, and scripting actions"
|
|
read_when:
|
|
- Scripting or debugging the agent browser via the local control API
|
|
- Looking for the `openclaw browser` CLI reference
|
|
- Adding custom browser automation with snapshots and refs
|
|
title: "Browser control API"
|
|
---
|
|
|
|
For setup, configuration, and troubleshooting, see [Browser](/tools/browser).
|
|
This page is the reference for the local control HTTP API, the `openclaw browser`
|
|
CLI, and scripting patterns (snapshots, refs, waits, debug flows).
|
|
|
|
## Control API (optional)
|
|
|
|
For local integrations only, the Gateway exposes a small loopback HTTP API.
|
|
This standalone server is opt-in — set the environment variable
|
|
`OPENCLAW_EAGER_BROWSER_CONTROL_SERVER=1` in the gateway service environment
|
|
and restart the gateway before the HTTP endpoints become available. Without
|
|
this variable the browser control runtime still works through the CLI and
|
|
agent tools, but nothing listens on the loopback control port.
|
|
|
|
- Status/start/stop: `GET /`, `GET /doctor`, `POST /start`, `POST /stop`, `POST /reset-profile`
|
|
- Profiles: `GET /profiles`, `POST /profiles/create`, `DELETE /profiles/:name`
|
|
- Tabs: `GET /tabs`, `POST /tabs/open`, `POST /tabs/focus`, `DELETE /tabs/:targetId`, `POST /tabs/action`
|
|
- Snapshot/screenshot/stream: `GET /snapshot`, `POST /screenshot`, `POST /screencast`
|
|
- Actions: `POST /navigate`, `POST /act`
|
|
- Hooks: `POST /hooks/file-chooser`, `POST /hooks/dialog`
|
|
- Downloads: `POST /download`, `POST /wait/download`
|
|
- Permissions: `POST /permissions/grant`
|
|
- Debugging: `GET /console`, `GET /errors`, `GET /requests`, `GET /dialogs`, `POST /pdf`, `POST /trace/start`, `POST /trace/stop`, `POST /highlight`
|
|
- Network: `POST /response/body`
|
|
- State: `GET /cookies`, `POST /cookies/set`, `POST /cookies/clear`, `GET /storage/:kind`, `POST /storage/:kind/set`, `POST /storage/:kind/clear`
|
|
- Settings: `POST /set/offline`, `POST /set/headers`, `POST /set/credentials`, `POST /set/geolocation`, `POST /set/media`, `POST /set/timezone`, `POST /set/locale`, `POST /set/device`
|
|
|
|
`POST /tabs/action` is the batched form the CLI uses internally for
|
|
`browser tab` subcommands (`{"action":"new"|"label"|"select"|"close"|"list", ...}`).
|
|
Prefer the single-purpose tab routes above when scripting directly.
|
|
|
|
All endpoints accept `?profile=<name>`. `POST /start?headless=true` requests a
|
|
one-shot headless launch for local managed profiles without changing persisted
|
|
browser config. Attach-only, remote CDP, and existing-session profiles reject
|
|
that override because OpenClaw does not launch those browser processes.
|
|
|
|
For tab endpoints, `targetId` is the compatibility field name. Prefer passing
|
|
`suggestedTargetId` from `GET /tabs` or `POST /tabs/open`. Labels and `tabId`
|
|
handles such as `t1` are also accepted. Raw CDP target ids and unique raw
|
|
target-id prefixes still work, but they are volatile diagnostic handles.
|
|
Tab handles are scoped to a browser host or node and profile. Keep that route
|
|
with the handle when making follow-up requests.
|
|
|
|
For profiles configured with `driver: "extension"`, `GET /tabs` and the browser
|
|
tool can also return `webExtensionTabId`, the runtime-scoped numeric Chrome
|
|
WebExtensions tab ID for the same tab. This field is omitted for other drivers
|
|
or when extension metadata is unavailable. Use it only when calling a
|
|
WebExtensions API; continue to use `suggestedTargetId` or `tabId` for OpenClaw
|
|
browser actions because a `webExtensionTabId` can change after the browser or
|
|
extension reconnects.
|
|
|
|
The Control UI's `browser.request` Gateway method accepts `target: "host"` to
|
|
pin the Gateway host or `target: "node"` with `node: "<node-id>"` to pin a browser
|
|
node. Pass the profile in `query.profile`. Explicit routes do not fall back to
|
|
another host. Omitting them keeps the configured automatic routing. These
|
|
routing fields do not grant access or change browser policy.
|
|
|
|
Browser previews require a result from the `browser` tool with a known route.
|
|
Browser-shaped metadata from other tools does not trigger screenshots or change
|
|
the panel's selection. Those results remain ordinary tool output.
|
|
|
|
When URL validation fails during tab listing, the tab keeps its identity and
|
|
title but returns `url: ""` and `urlUnavailableReason`:
|
|
|
|
- `navigation_blocked`: navigation rules rejected the address.
|
|
- `navigation_check_failed`: OpenClaw could not validate the address, for example
|
|
because DNS lookup failed. Refresh to check again.
|
|
|
|
An empty URL alone does not indicate a policy denial. Navigation-policy errors
|
|
also carry `reason: "navigation_blocked"`. Raw blocked URLs and DNS details are
|
|
not included in that metadata. Tab listings are observations, not authorization:
|
|
every subsequent content read or action still enforces its own checks.
|
|
|
|
If shared-secret gateway auth is configured, browser HTTP routes require auth too:
|
|
|
|
- `Authorization: Bearer <gateway token>`
|
|
- `x-openclaw-password: <gateway password>` or HTTP Basic auth with that password
|
|
|
|
Notes:
|
|
|
|
- This standalone loopback browser API does **not** consume trusted-proxy or
|
|
Tailscale Serve identity headers.
|
|
- If `gateway.auth.mode` is `none` or `trusted-proxy`, these loopback browser
|
|
routes do not inherit those identity-bearing modes. Keep them loopback-only.
|
|
|
|
### Screencast stream
|
|
|
|
`POST /screencast` mints a single-use token for a live view of the selected tab.
|
|
Pass an optional `targetId`, `maxWidth`, `maxHeight`, and `quality` in the JSON
|
|
body. Dimensions default to 1280 and are clamped to integers from 320 to 2000.
|
|
JPEG quality defaults to 70 and is clamped from 30 to 90.
|
|
|
|
The response contains `token`, `wsPath`, `expiresAtMs`, `targetId`, and `url`.
|
|
Resolve `wsPath` against the Gateway URL and open a WebSocket there:
|
|
`/browser/screencast?token=<token>`. The 48-character hexadecimal token expires
|
|
after 60 seconds and can be consumed once. Invalid, expired, and reused tokens
|
|
are rejected with HTTP 401 before upgrade. Viewers send no application messages.
|
|
Binary viewer messages close the connection.
|
|
|
|
Tickets and viewers minted through the Gateway are bound to the requesting
|
|
Gateway connection. Ending that connection revokes unused tickets and closes
|
|
its viewers. Revoked (invalidated) connections are fenced immediately, before
|
|
the Gateway socket finishes closing: their tickets cannot upgrade, and their
|
|
viewers receive no further frames or metadata. The loopback HTTP control API
|
|
has no Gateway connection to bind, so its tickets remain TTL-only.
|
|
|
|
The plugin shares one CDP screencast per profile and tab. Chrome sends JPEG
|
|
frames on repaint, paced to approximately 20 frames per second. Slow viewers
|
|
skip frames instead of building a queue. Navigation immediately retires the
|
|
capture session. A new CDP session starts only after the address is allowed,
|
|
so delayed frames from the previous document cannot enter the new stream.
|
|
A rejected navigation stops the stream.
|
|
|
|
Text messages are JSON with `type` in `ready`, `meta`, or `error`. A `ready`
|
|
message includes `targetId`, `url`, and `title`. `meta` updates `url` and `title`
|
|
after allowed navigation and page load. Binary messages contain:
|
|
|
|
1. A four-byte unsigned big-endian JSON header length.
|
|
2. That many bytes of UTF-8 JSON: `{ "url", "cssWidth", "cssHeight", "scrollX", "scrollY", "ts" }`.
|
|
3. The JPEG bytes.
|
|
|
|
`cssWidth` and `cssHeight` come from CDP's `deviceWidth` and `deviceHeight`
|
|
metadata and describe the layout viewport in CSS pixels. `scrollX` and `scrollY`
|
|
come from `scrollOffsetX` and `scrollOffsetY`. `ts` is the CDP frame timestamp.
|
|
|
|
| Close code | Meaning |
|
|
| ---------- | ----------------------------------------------------------------------------------------------------------- |
|
|
| 4001 | Token invalid or expired (normally rejected before upgrade with HTTP 401) |
|
|
| 4003 | `navigation_blocked` |
|
|
| 4004 | `target_closed`, including a profile lifecycle change |
|
|
| 4005 | Unsupported streaming |
|
|
| 4006 | `authority_revoked` (the requesting Gateway connection ended or was invalidated, e.g. device token revoked) |
|
|
| 1012 | Gateway shutting down |
|
|
|
|
Chrome MCP existing-session profiles and missing Playwright return HTTP 501 with
|
|
`code: "SCREENCAST_UNSUPPORTED"` and `reason: "existing-session"` or `"playwright"`.
|
|
Node-routed requests fail before proxying with `INVALID_REQUEST` and details
|
|
`{ "code": "SCREENCAST_UNSUPPORTED", "reason": "node" }`. The Control UI falls
|
|
back to the existing screenshot route when streaming is unavailable.
|
|
Navigation metadata updates the tab and address bar. The displayed image keeps
|
|
its own URL and metrics until a replacement frame arrives. While Annotate or
|
|
Inspect is active, the Control UI pins the captured image and its URL, then
|
|
displays the latest held frame when capture mode ends.
|
|
|
|
### `/act` error contract
|
|
|
|
`POST /act` uses a structured error response for validation, policy, and
|
|
recognized interaction failures:
|
|
|
|
```json
|
|
{ "error": "<message>", "code": "ACT_*" }
|
|
```
|
|
|
|
Current `code` values:
|
|
|
|
- `ACT_KIND_REQUIRED` (HTTP 400): `kind` is missing or unrecognized.
|
|
- `ACT_INVALID_REQUEST` (HTTP 400): action payload failed normalization or validation.
|
|
- `ACT_SELECTOR_UNSUPPORTED` (HTTP 400): `selector` was used with an unsupported action kind.
|
|
- `ACT_EVALUATE_DISABLED` (HTTP 403): `evaluate` (or `wait --fn`) is disabled by config.
|
|
- `ACT_TARGET_ID_MISMATCH` (HTTP 403): top-level or batched `targetId` conflicts with request target.
|
|
- `ACT_OPERATION_FAILED` (HTTP 500): the selected element could not perform the action, such as a noneditable input, covered control, or ambiguous ref. The message describes the interaction failure without treating it as a browser connection outage.
|
|
- `ACT_EXISTING_SESSION_UNSUPPORTED` (HTTP 501): action is not supported for existing-session profiles.
|
|
|
|
Other runtime failures may still return `{ "error": "<message>" }` without a
|
|
`code` field.
|
|
|
|
### Playwright requirement
|
|
|
|
Some features (navigate/act/AI snapshot/role snapshot, element
|
|
screenshots, PDF) require Playwright. If Playwright isn't installed, those endpoints return
|
|
a clear 501 error.
|
|
|
|
What still works without Playwright:
|
|
|
|
- ARIA snapshots
|
|
- Role-style accessibility snapshots (`--interactive`, `--compact`,
|
|
`--depth`, `--efficient`) when a per-tab CDP WebSocket is available. This is
|
|
a fallback for inspection and ref discovery. Playwright remains the primary
|
|
action engine.
|
|
- Page screenshots for the managed `openclaw` browser when a per-tab CDP
|
|
WebSocket is available
|
|
- Page screenshots for `existing-session` / Chrome MCP profiles
|
|
- `existing-session` ref-based screenshots (`--ref`) from snapshot output
|
|
|
|
What still needs Playwright:
|
|
|
|
- `navigate`
|
|
- `act`
|
|
- AI snapshots that depend on Playwright's native AI snapshot format
|
|
- CSS-selector element screenshots (`--element`)
|
|
- full browser PDF export
|
|
|
|
Element screenshots also reject `--full-page`. The route returns `fullPage is
|
|
not supported for element screenshots`.
|
|
|
|
If you see `Playwright is not available in this gateway build`, the packaged
|
|
Gateway is missing the core browser runtime dependency. Reinstall or update
|
|
OpenClaw, then restart the gateway. For Docker, also install the Chromium
|
|
browser binaries as shown below.
|
|
|
|
#### Docker Playwright install
|
|
|
|
If your Gateway runs in Docker, avoid `npx playwright` (npm override conflicts).
|
|
For custom images, bake Chromium into the image:
|
|
|
|
```bash
|
|
OPENCLAW_INSTALL_BROWSER=1 ./scripts/docker/setup.sh
|
|
```
|
|
|
|
The browser also needs system libraries, so installing Chromium in a one-off
|
|
Compose container is not durable. Rebuild the image with
|
|
`OPENCLAW_INSTALL_BROWSER=1` instead. To persist browser downloads and other
|
|
caches, persist `/home/node` with `OPENCLAW_HOME_VOLUME` or a bind mount. See
|
|
[Docker](/install/docker).
|
|
|
|
## How it works (internal)
|
|
|
|
A small loopback control server accepts HTTP requests and connects to Chromium-based browsers via CDP. Advanced actions (click/type/snapshot/PDF) go through Playwright on top of CDP. When Playwright is missing, only non-Playwright operations are available. The agent sees one stable interface while local/remote browsers and profiles swap freely underneath.
|
|
|
|
## CLI quick reference
|
|
|
|
All commands accept `--browser-profile <name>` to target a specific profile, and `--json` for machine-readable output.
|
|
|
|
<AccordionGroup>
|
|
|
|
<Accordion title="Basics: status, tabs, open/focus/close">
|
|
|
|
```bash
|
|
openclaw browser status
|
|
openclaw browser doctor
|
|
openclaw browser doctor --deep # add a live snapshot probe
|
|
openclaw browser start
|
|
openclaw browser start --headless # one-shot local managed headless launch
|
|
openclaw browser stop # also clears emulation on attach-only/remote CDP
|
|
openclaw browser reset-profile # moves the profile's browser data to Trash
|
|
openclaw browser tabs
|
|
openclaw browser tab # shortcut for current tab
|
|
openclaw browser tab new
|
|
openclaw browser tab new --label research
|
|
openclaw browser tab label abcd1234 research
|
|
openclaw browser tab select 2
|
|
openclaw browser tab close 2
|
|
openclaw browser open https://example.com
|
|
openclaw browser focus abcd1234
|
|
openclaw browser close abcd1234
|
|
```
|
|
|
|
</Accordion>
|
|
|
|
<Accordion title="Profiles: list, create, delete">
|
|
|
|
```bash
|
|
openclaw browser profiles
|
|
openclaw browser create-profile --name research --color "#0066CC"
|
|
openclaw browser create-profile --name attach --driver existing-session --cdp-url http://127.0.0.1:9222
|
|
openclaw browser delete-profile --name research
|
|
```
|
|
|
|
</Accordion>
|
|
|
|
<Accordion title="Inspection: screenshot, snapshot, console, errors, requests">
|
|
|
|
```bash
|
|
openclaw browser screenshot
|
|
openclaw browser screenshot --full-page
|
|
openclaw browser screenshot --ref 12 # or --ref e12
|
|
openclaw browser screenshot --labels
|
|
openclaw browser snapshot
|
|
openclaw browser snapshot --format aria --limit 200
|
|
openclaw browser snapshot --interactive --compact --depth 6
|
|
openclaw browser snapshot --efficient
|
|
openclaw browser snapshot --labels
|
|
openclaw browser snapshot --urls
|
|
openclaw browser snapshot --selector "#main" --interactive
|
|
openclaw browser snapshot --frame "iframe#main" --interactive
|
|
openclaw browser snapshot --out snapshot.txt
|
|
openclaw browser console --level error
|
|
openclaw browser errors --clear
|
|
openclaw browser requests --filter api --clear
|
|
openclaw browser pdf
|
|
openclaw browser responsebody "**/api" --max-chars 5000
|
|
```
|
|
|
|
</Accordion>
|
|
|
|
<Accordion title="Actions: navigate, click, type, drag, wait, evaluate">
|
|
|
|
```bash
|
|
openclaw browser navigate https://example.com
|
|
openclaw browser resize 1280 720
|
|
openclaw browser click 12 --double # or e12 for role refs
|
|
openclaw browser click-coords 120 340 # viewport coordinates
|
|
openclaw browser type 23 "hello" --submit
|
|
openclaw browser press Enter
|
|
openclaw browser hover 44
|
|
openclaw browser scrollintoview e12
|
|
openclaw browser drag 10 11
|
|
openclaw browser select 9 OptionA OptionB
|
|
openclaw browser download e12 report.pdf
|
|
openclaw browser waitfordownload report.pdf
|
|
openclaw browser upload /tmp/openclaw/uploads/file.pdf
|
|
openclaw browser upload /tmp/openclaw/uploads/file.pdf --ref e12
|
|
openclaw browser upload media://inbound/file.pdf
|
|
openclaw browser fill --fields '[{"ref":"1","type":"text","value":"Ada"}]'
|
|
openclaw browser dialog --accept
|
|
openclaw browser dialog --dismiss --dialog-id d1
|
|
openclaw browser wait --text "Done"
|
|
openclaw browser wait "#main" --url "**/dash" --load networkidle --fn "window.ready===true"
|
|
openclaw browser evaluate --fn '(el) => el.textContent' --ref 7
|
|
openclaw browser evaluate --fn 'const title = document.title; return title;'
|
|
openclaw browser evaluate --timeout-ms 30000 --fn 'async () => { await window.ready; return true; }'
|
|
openclaw browser highlight e12
|
|
openclaw browser trace start
|
|
openclaw browser trace stop
|
|
```
|
|
|
|
</Accordion>
|
|
|
|
<Accordion title="State: cookies, storage, offline, headers, geo, device">
|
|
|
|
```bash
|
|
openclaw browser cookies
|
|
openclaw browser cookies set session abc123 --url "https://example.com"
|
|
openclaw browser cookies clear
|
|
openclaw browser storage local get
|
|
openclaw browser storage local set theme dark
|
|
openclaw browser storage session clear
|
|
openclaw browser set offline on
|
|
openclaw browser set headers --headers-json '{"X-Debug":"1"}'
|
|
openclaw browser set credentials user pass # --clear to remove
|
|
openclaw browser set geo 37.7749 -122.4194 --origin "https://example.com"
|
|
openclaw browser set media dark
|
|
openclaw browser set timezone America/New_York
|
|
openclaw browser set locale en-US
|
|
openclaw browser set device "iPhone 14"
|
|
```
|
|
|
|
</Accordion>
|
|
|
|
</AccordionGroup>
|
|
|
|
Notes:
|
|
|
|
- The agent-facing `browser` tool exposes `action=download` (required `ref` and
|
|
`path`) and `action=waitfordownload` (optional `path`). Both return the saved
|
|
download URL, suggested filename, and guarded local path. Explicit download
|
|
interception is available for managed Playwright profiles. Existing-session
|
|
profiles return an unsupported-operation error.
|
|
- Prefer atomic chooser uploads: pass the trigger `--ref` with the upload so OpenClaw arms and clicks in one request. Paths-only `upload` remains supported when a later trigger is intentional. Use `--input-ref` or `--element` to set a file input directly. `dialog` is an arming call. Run it before the click/press that triggers the dialog. If an action opens a modal, the action response includes `blockedByDialog` and `browserState.dialogs.pending`. Pass that `dialogId` to respond directly. Dialogs handled outside OpenClaw appear under `browserState.dialogs.recent`.
|
|
- Cancelling a pending locator click, typing, or upload operation leaves other tabs connected. Upload waiters belong to the selected tab. A new upload on that tab replaces its previous waiter.
|
|
- `click`/`type`/etc require a `ref` from `snapshot` (for example, Playwright ref `f1e12`, role ref `e12`, or actionable ARIA ref `ax12`). Copy the returned ref unchanged, including any frame prefix. CSS selectors are intentionally not supported for actions. Use `click-coords` when the visible viewport position is the only reliable target.
|
|
- Download and trace paths are constrained to OpenClaw temp roots: `/tmp/openclaw{,/downloads}` (fallback: `${os.tmpdir()}/openclaw/...`).
|
|
- `upload` accepts files from the OpenClaw temp uploads root and
|
|
OpenClaw-managed inbound media. Managed inbound media can be referenced as
|
|
`media://inbound/<id>`, sandbox-relative `media/inbound/<id>`, or a resolved
|
|
path inside the managed inbound media directory. Nested media refs,
|
|
traversal, symlinks, hardlinks, and arbitrary local paths are still rejected.
|
|
- `upload` can also set file inputs directly via `--input-ref` or `--element`; these operations honor the upload timeout.
|
|
- Dialog prompt text preserves whitespace and empty strings exactly. Reading all local or session storage preserves empty keys and keys such as `__proto__`.
|
|
|
|
Stable tab ids and labels survive Chromium raw-target replacement when OpenClaw
|
|
can prove the replacement tab, such as a unique old/new pair for the same URL or
|
|
a single old tab becoming a single new tab after form submission. Ambiguous
|
|
duplicate-URL replacements receive fresh handles. Raw target ids are still
|
|
volatile. Prefer `suggestedTargetId` from `tabs` in scripts.
|
|
|
|
Snapshot flags at a glance:
|
|
|
|
- `--format ai` (default with Playwright): AI snapshot with native Playwright refs, including frame-qualified refs such as `f1e12`.
|
|
- `--format aria`: accessibility tree with `axN` refs. When Playwright is available, OpenClaw binds refs with backend DOM ids to the live page. Follow-up actions can then use them. Otherwise treat the output as inspection-only.
|
|
- `--efficient` (or `--mode efficient`): compact role snapshot preset. Set `browser.snapshotDefaults.mode: "efficient"` to make this the default (see [Gateway configuration](/gateway/config-browser-ui-desktop#browser)).
|
|
- `--interactive`, `--compact`, `--depth`, `--selector` force a role snapshot with `ref=e12` refs. `--frame "<iframe>"` scopes role snapshots to an iframe.
|
|
- Selector- and frame-scoped role refs bind to the captured DOM controls, including shadow DOM and external `aria-owns` members. Reordering controls does not retarget those refs. Removed controls or failed bindings require a new snapshot instead of matching another control by name.
|
|
- In scoped role snapshots, ignored nodes and unnamed generic wrappers are transparent before depth filtering. Depth counts the remaining role nodes from the selected root; wrapper lines and ref numbers can differ from older snapshots. State attributes, URL appendices, and output limits still apply.
|
|
- A selector-scoped snapshot is a point-in-time observation. If no element matches at request time, it returns an empty snapshot immediately. It does not wait for the snapshot timeout. Use `openclaw browser wait "<selector>"` when the page is expected to add the element later.
|
|
- `--selector` does not change the behavior of page-wide or frame-scoped transport failures. Those still use the configured snapshot timeout and diagnostics.
|
|
- With Playwright, `--labels` adds a screenshot with overlayed ref labels
|
|
(prints `MEDIA:<path>`) plus an `annotations` array with each ref's bounding
|
|
box. On `screenshot`, Playwright-backed labels work with `--full-page`,
|
|
`--ref`, and `--element`. On `snapshot`, the accompanying screenshot remains
|
|
viewport-only. Existing-session/chrome-mcp profiles render overlay labels on
|
|
page screenshots but do not return `annotations` or use the Playwright
|
|
full-page/ref/element projection helper. Without Playwright or chrome-mcp,
|
|
labeled screenshots are not available.
|
|
- `--urls` appends discovered link destinations to AI snapshots.
|
|
|
|
## Snapshots and refs
|
|
|
|
OpenClaw supports three "snapshot" styles:
|
|
|
|
- **AI snapshot (native refs)**: `openclaw browser snapshot` (default, `--format ai`)
|
|
- Output: a text snapshot with refs such as `f1e12` and matching `refs` metadata.
|
|
- Actions: `openclaw browser click f1e12`, `openclaw browser type f1e23 "hello"` (use your snapshot's refs).
|
|
- Internally, the ref is resolved via Playwright's `aria-ref`.
|
|
|
|
- **Role snapshot (role refs like `e12`)**: `openclaw browser snapshot --interactive` (or `--compact`, `--depth`, `--selector`, `--frame`)
|
|
- Output: a role-based list/tree with `[ref=e12]` (and optional `[nth=1]`).
|
|
- Actions: `openclaw browser click e12`, `openclaw browser highlight e12`.
|
|
- Internally, the ref is resolved via `getByRole(...)` (plus `nth()` for duplicates).
|
|
- Names containing quotes, backslashes, or YAML punctuation remain actionable. Use the ref rather than reconstructing a locator from the displayed name.
|
|
- A missing displayed name can mean an empty accessible name or one above Playwright's 900 UTF-16-unit limit. Keep using the returned ref.
|
|
- Add `--labels` to include a screenshot with overlayed `e12` labels. On
|
|
Playwright-backed profiles this also returns per-ref bounding-box metadata
|
|
(`annotations[]`). A labeled element screenshot preserves the ref and frame
|
|
from the snapshot that produced it.
|
|
- Add `--urls` when link text is ambiguous and the agent needs concrete
|
|
navigation targets. With `--frame`, the URL appendix comes from that frame.
|
|
|
|
- **ARIA snapshot (ARIA refs like `ax12`)**: `openclaw browser snapshot --format aria`
|
|
- Output: the accessibility tree as structured nodes.
|
|
- Actions: `openclaw browser click ax12` works when the snapshot path can bind
|
|
the ref through Playwright and Chrome backend DOM ids.
|
|
- If Playwright is unavailable, ARIA snapshots can still be useful for
|
|
inspection, but refs may not be actionable. Re-snapshot with `--format ai`
|
|
or `--interactive` when you need action refs.
|
|
- When the driver exposes stable document identity, consecutive AI and role
|
|
snapshots for the same profile, tab, document, and option family append
|
|
`[new]` to ref-bearing lines absent from the previous snapshot. Navigation
|
|
starts a fresh unmarked baseline, including same-URL iframe reloads when the
|
|
snapshot includes that frame. Existing-session snapshots omit deltas.
|
|
The first snapshot establishes the baseline without markers. Later responses
|
|
also expose `newElements`, and add a count footer when the value is nonzero.
|
|
Structured `--format aria` snapshots with `axN` refs do not use delta markers.
|
|
- Contributors: the raw-CDP fallback path has a Docker proof lane, described in
|
|
[Docker test suites](/reference/test/docker).
|
|
|
|
Ref behavior:
|
|
|
|
- Refs are **not stable across navigations**. If something fails, re-run `snapshot` and use a fresh ref.
|
|
- A batch stops after a committed main-frame navigation—including a same-URL
|
|
reload—or after the page closes. Its `aborted` summary reports the action
|
|
number and skipped count. Take a fresh snapshot before issuing dependent
|
|
actions, or use separate act calls when navigation is expected.
|
|
- `/act` returns the current raw `targetId` after action-triggered replacement
|
|
when it can prove the replacement tab. Keep using stable tab ids/labels for
|
|
follow-up commands.
|
|
- A role snapshot taken with `--frame` scopes role refs to that iframe until the next role snapshot.
|
|
- Unknown or stale `axN` refs fail fast instead of falling through to
|
|
Playwright's `aria-ref` selector. Run a fresh snapshot on the same tab when
|
|
that happens.
|
|
|
|
## Browser batch CLI
|
|
|
|
`openclaw browser batch` runs an array of nested `/act` actions in one `/act`
|
|
call (the same `kind="batch"` runtime reached through the agent tool), so CLI
|
|
users and scripts can combine actions like `wait`, `click`, `type`, and
|
|
`evaluate` into a single replayable plan without per-action round trips. Each
|
|
entry in `actions[]` is a `BrowserActRequest` — the closed union the `/act`
|
|
route accepts (`click`, `clickCoords`, `type`, `press`, `hover`,
|
|
`scrollIntoView`, `drag`, `select`, `fill`, `resize`, `wait`, `evaluate`,
|
|
`close`, `batch`) — not arbitrary `openclaw browser` subcommands. `batch` is
|
|
not supported on `profile="user"` and other existing-session (chrome-mcp)
|
|
profiles. Send actions individually there.
|
|
|
|
- CLI: `openclaw browser batch --actions '<json>'`, `openclaw browser batch
|
|
--actions-file plan.json`, or `openclaw browser batch --actions-file -` to
|
|
read the JSON array from stdin. `--continue` sets `stopOnError=false`. The
|
|
default is to stop on first error. `--target-id` scopes the whole batch to
|
|
one tab. `--actions-file` and stdin input are capped at 1,000,000 bytes.
|
|
Split larger plans into multiple batch commands.
|
|
- Ref lifecycle: refs come from a `snapshot` run before the batch (snapshot is
|
|
not a nested action). A nested action that changes page state — such as a
|
|
`click` that triggers navigation, or an `evaluate` that mutates the DOM — can
|
|
invalidate earlier refs for the rest of the batch. Put state-changing actions
|
|
first, or split into a follow-up batch after re-snapshotting. Navigation and
|
|
re-snapshotting happen outside the batch (`openclaw browser navigate` /
|
|
`snapshot`), since `open`, `navigate`, and `snapshot` are not `/act` kinds.
|
|
- Target id conflicts: a nested action may omit `targetId` or repeat the
|
|
request-level `targetId`. An explicit nested `targetId` that resolves to a
|
|
different tab is rejected with `ACT_TARGET_ID_MISMATCH` before any action
|
|
runs. Batched actions share the request's tab by design.
|
|
- Error summary: the response is `{ "results": [{ "ok": true }, { "ok": false,
|
|
"error": "<message>" }, ...] }`, one entry per action in order. When
|
|
`stopOnError` is the default, the array ends at the first failure. With
|
|
`--continue` it covers every action. Any failed entry makes the CLI exit
|
|
nonzero. Pass `--json` to preserve the full ordered response for scripts.
|
|
- Nested batches occupy one parent result. If a child action fails, that result
|
|
reports the first child error. Each batch applies its own `stopOnError`:
|
|
continuing inside a nested batch does not make it successful or make its
|
|
parent continue.
|
|
|
|
## Wait power-ups
|
|
|
|
You can wait on more than just time/text:
|
|
|
|
- Wait for URL (globs supported by Playwright):
|
|
- `openclaw browser wait --url "**/dash"`
|
|
- Wait for load state:
|
|
- `openclaw browser wait --load networkidle`
|
|
- Supported on managed `openclaw` and raw/remote CDP profiles. Profiles using the `existing-session` driver (including the default `user` profile) reject `networkidle`. Use `--url`, `--text`, a selector, or `--fn` waits there.
|
|
- Wait for a JS predicate:
|
|
- `openclaw browser wait --fn "window.ready===true"`
|
|
- Wait for a selector to become visible:
|
|
- `openclaw browser wait "#main"`
|
|
|
|
These can be combined:
|
|
|
|
```bash
|
|
openclaw browser wait "#main" \
|
|
--url "**/dash" \
|
|
--load networkidle \
|
|
--fn "window.ready===true" \
|
|
--timeout-ms 15000
|
|
```
|
|
|
|
## Debug workflows
|
|
|
|
When an action fails (e.g. "not visible", "strict mode violation", "covered"):
|
|
|
|
1. `openclaw browser snapshot --interactive`
|
|
2. Use `click <ref>` / `type <ref>` (prefer role refs in interactive mode)
|
|
3. If it still fails: `openclaw browser highlight <ref>` to see what Playwright is targeting
|
|
4. If the page behaves oddly:
|
|
- `openclaw browser errors --clear`
|
|
- `openclaw browser requests --filter api --clear`
|
|
5. For deep debugging: record a trace:
|
|
- `openclaw browser trace start`
|
|
- reproduce the issue
|
|
- `openclaw browser trace stop` (prints `TRACE:<path>`)
|
|
|
|
## JSON output
|
|
|
|
`--json` is for scripting and structured tooling.
|
|
|
|
Examples:
|
|
|
|
```bash
|
|
openclaw browser --json status
|
|
openclaw browser --json snapshot --interactive
|
|
openclaw browser --json requests --filter api
|
|
openclaw browser --json cookies
|
|
```
|
|
|
|
Role snapshots in JSON include `refs` plus a small `stats` block (lines/chars/refs/interactive) so tools can reason about payload size and density.
|
|
|
|
## State and environment knobs
|
|
|
|
These are useful for "make the site behave like X" workflows:
|
|
|
|
- Cookies: `cookies`, `cookies set`, `cookies clear`
|
|
- Storage: `storage local|session get|set|clear`
|
|
- Offline: `set offline on|off`
|
|
- Headers: `set headers --headers-json '{"X-Debug":"1"}'` (or the positional form `set headers '{"X-Debug":"1"}'`)
|
|
- HTTP basic auth: `set credentials user pass` (or `--clear`)
|
|
- Geolocation: `set geo <lat> <lon> --origin "https://example.com"` (or `--clear`)
|
|
- Media: `set media dark|light|no-preference|none`
|
|
- Timezone / locale: `set timezone ...`, `set locale ...`
|
|
- Device / viewport:
|
|
- `set device "iPhone 14"` (Playwright device presets)
|
|
- `set viewport 1280 720`
|
|
|
|
## Security and privacy
|
|
|
|
- The openclaw browser profile may contain logged-in sessions. Treat it as sensitive.
|
|
- `browser act kind=evaluate` / `openclaw browser evaluate` and `wait --fn`
|
|
execute arbitrary JavaScript in the page context. Prompt injection can steer
|
|
this. Disable it with `browser.evaluateEnabled=false` if you do not need it.
|
|
- `openclaw browser evaluate --fn` accepts a function source, an expression, or
|
|
a statement body. Statement bodies are wrapped as async functions, so use
|
|
`return` for the value you want back. Use `--timeout-ms <ms>` when the
|
|
page-side function may need longer than the default evaluate timeout.
|
|
- For logins and anti-bot notes (X/Twitter, etc.), see [Browser login + X/Twitter posting](/tools/browser-login).
|
|
- Keep the Gateway/node host private (loopback or tailnet-only).
|
|
- Remote CDP endpoints are powerful. Tunnel and protect them.
|
|
|
|
Strict-mode example (block private/internal destinations by default):
|
|
|
|
```json5
|
|
{
|
|
browser: {
|
|
ssrfPolicy: {
|
|
dangerouslyAllowPrivateNetwork: false,
|
|
allowedHostnames: ["*.example.com", "example.com", "localhost"],
|
|
},
|
|
},
|
|
}
|
|
```
|
|
|
|
## Related
|
|
|
|
- [Browser](/tools/browser) - overview, configuration, profiles, security
|
|
- [Browser login](/tools/browser-login) - signing in to sites
|
|
- [Browser Linux troubleshooting](/tools/browser-linux-troubleshooting)
|
|
- [Browser WSL2 troubleshooting](/tools/browser-wsl2-windows-remote-cdp-troubleshooting)
|