mirror of
https://github.com/openclaw/openclaw.git
synced 2026-07-09 15:59:30 +00:00
feat(nodes): add computer use (computer.act) on macOS nodes via Peekaboo (#102776)
* feat(nodes): add computer use (computer.act) on macOS nodes via Peekaboo * fix(nodes): release held computer.act input on node disconnect/stop/disable * fix(nodes): release held computer.act input armed after a lifecycle release * fix(nodes): scope computer.act lifecycle catch-up release to the arming action * chore(nodes): register computer tool display metadata, sync native i18n, doc-comment constants * fix(nodes): preserve caught-error cause in computer tool resolver; regen docs map
This commit is contained in:
Peter Steinberger
GitHub
parent
d9f8fc6322
commit
f8dbd34ab8
35 changed files with 2632 additions and 127 deletions
|
|
@ -1697,6 +1697,7 @@
|
|||
{
|
||||
"group": "Node features",
|
||||
"pages": [
|
||||
"nodes/computer-use",
|
||||
"nodes/talk",
|
||||
"nodes/voicewake",
|
||||
"nodes/location-command"
|
||||
|
|
|
|||
|
|
@ -4679,6 +4679,17 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
|
|||
- H2: macOS screen video (OS-level)
|
||||
- H2: Related
|
||||
|
||||
## nodes/computer-use.md
|
||||
|
||||
- Route: /nodes/computer-use
|
||||
- Headings:
|
||||
- H2: Requirements
|
||||
- H2: The computer agent tool
|
||||
- H2: The computer.act node command
|
||||
- H2: Enable and arm
|
||||
- H2: Safety
|
||||
- H2: Relationship to other desktop-control paths
|
||||
|
||||
## nodes/images.md
|
||||
|
||||
- Route: /nodes/images
|
||||
|
|
|
|||
67
docs/nodes/computer-use.md
Normal file
67
docs/nodes/computer-use.md
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
---
|
||||
summary: "Agent-driven desktop control on a paired macOS node via the computer tool and computer.act node command"
|
||||
read_when:
|
||||
- Letting the gateway agent see and control a Mac desktop
|
||||
- Arming, permissions, or safety for computer use
|
||||
- Extending the computer.act node command or its fulfillers
|
||||
title: "Computer use"
|
||||
---
|
||||
|
||||
Computer use lets the gateway agent see and control a paired **macOS** desktop: it captures a screenshot with the existing `screen.snapshot` node command and drives the pointer and keyboard through a single dangerous node command, `computer.act`. The action set mirrors Anthropic's `computer_20251124` tool, so any vision-capable model can drive it through the built-in `computer` agent tool.
|
||||
|
||||
The agent emits one uniform command, `computer.act`; it cannot tell how a node fulfills it. A macOS node fulfills `computer.act` in-process with the embedded Peekaboo automation engine (correct TCC permissions, no extra process). Other platforms can fulfill the same command later without changing the agent-facing contract.
|
||||
|
||||
## Requirements
|
||||
|
||||
- A paired **macOS** node (the OpenClaw macOS app running in node mode).
|
||||
- macOS app setting **Allow Computer Control** enabled (default: off).
|
||||
- macOS **Accessibility** permission granted to OpenClaw (for pointer/keyboard injection) and **Screen Recording** permission (for `screen.snapshot`).
|
||||
- The `computer.act` command armed on the gateway (it is dangerous and disarmed by default).
|
||||
- A vision-capable agent model.
|
||||
|
||||
## The `computer` agent tool
|
||||
|
||||
The built-in `computer` tool takes one action per call. Coordinates are pixels in the most recent screenshot; the node maps them to display points.
|
||||
|
||||
- Reads: `screenshot`.
|
||||
- Pointer: `left_click`, `right_click`, `middle_click`, `double_click`, `triple_click`, `mouse_move`, `left_click_drag` (with `startCoordinate`), `left_mouse_down`, `left_mouse_up`.
|
||||
- Scroll: `scroll` with `scrollDirection` (`up|down|left|right`) and `scrollAmount` (wheel ticks).
|
||||
- Keyboard: `type` (text), `key` (combo such as `cmd+shift+t` or `Return`), `hold_key` (`text` combo held for `duration` seconds).
|
||||
- Pacing: `wait` (`duration` seconds).
|
||||
|
||||
Modifier keys ride the `text` field on click and scroll actions (`shift`, `ctrl`, `alt`, `cmd`). After an input action the tool returns a fresh screenshot so the model can observe the result. If more than one computer-capable node is connected, pass `node` explicitly.
|
||||
|
||||
Screenshots are kept **model-only**: they are never auto-delivered to the chat channel. Treat all on-screen content as untrusted input; the tool warns the model not to follow on-screen instructions that conflict with the user's request.
|
||||
|
||||
## The `computer.act` node command
|
||||
|
||||
`computer.act` is the single node command the tool routes input through (`node.invoke` with `command: "computer.act"`). It is:
|
||||
|
||||
- **Dangerous by default**: listed in the built-in dangerous node commands and excluded from the runtime allowlist until explicitly armed. A macOS node may still declare it at pairing so the surface is approved once.
|
||||
- **macOS-only** today: only advertised by a macOS node that has **Allow Computer Control** enabled.
|
||||
|
||||
Reads reuse `screen.snapshot`; there is no second capture path. See [Camera and screen nodes](/nodes/camera) for the shared capture command.
|
||||
|
||||
## Enable and arm
|
||||
|
||||
1. In the macOS app, enable **Settings -> Allow Computer Control**, then grant **Accessibility** and **Screen Recording** when prompted.
|
||||
2. Approve the pairing update on the gateway (a new command forces re-pairing).
|
||||
3. Arm `computer.act` for a bounded window. The `phone-control` plugin exposes a `computer` group:
|
||||
|
||||
```text
|
||||
/phone arm computer 30m
|
||||
/phone status
|
||||
/phone disarm
|
||||
```
|
||||
|
||||
Arming requires `operator.admin` (or the owner) and auto-expires. Arming only toggles what the gateway may invoke; the macOS app still enforces its **Allow Computer Control** setting and Accessibility permission. Operators can equivalently add `computer.act` to `gateway.nodes.allowCommands`.
|
||||
|
||||
## Safety
|
||||
|
||||
- Nothing is autonomous: `computer.act` stays disarmed until an operator arms it, and every layer (gateway allowlist, macOS setting, Accessibility permission) must agree.
|
||||
- Screenshots are model-only and never auto-sent to chat (issue [#44759](https://github.com/openclaw/openclaw/issues/44759)).
|
||||
- Treat screen content as untrusted; it can carry prompt injection.
|
||||
|
||||
## Relationship to other desktop-control paths
|
||||
|
||||
This is the agent-driven path. See [Peekaboo bridge](/platforms/mac/peekaboo) for how it relates to the PeekabooBridge host, Codex Computer Use, and the direct `cua-driver` MCP.
|
||||
|
|
@ -18,13 +18,14 @@ OpenClaw can host **PeekabooBridge** as a local, permission-aware UI automation
|
|||
|
||||
## Relationship to other desktop-control paths
|
||||
|
||||
OpenClaw has three desktop-control paths that intentionally stay separate:
|
||||
OpenClaw has four desktop-control paths that intentionally stay separate:
|
||||
|
||||
- **PeekabooBridge host**: OpenClaw.app hosts the local PeekabooBridge socket. The `peekaboo` CLI is the client and uses OpenClaw.app's macOS permissions for screenshots, clicks, menus, dialogs, Dock actions, and window management.
|
||||
- **Agent-driven computer use (`computer.act`)**: the gateway agent's built-in `computer` tool captures screenshots via `screen.snapshot` and drives the pointer and keyboard through the dangerous `computer.act` node command. A macOS node fulfills `computer.act` in-process using the same embedded Peekaboo automation services this bridge exposes (`UIAutomationService`), so agent-driven control reuses the engine without going through the PeekabooBridge socket or the `peekaboo` CLI. See [Computer use](/nodes/computer-use).
|
||||
- **Codex Computer Use**: the bundled `codex` plugin checks and can install Codex's `computer-use` MCP plugin (`extensions/codex/src/app-server/computer-use.ts`), then lets Codex own native desktop-control tool calls during Codex-mode turns. OpenClaw does not proxy those actions through PeekabooBridge.
|
||||
- **Direct `cua-driver` MCP**: OpenClaw can register TryCua's upstream `cua-driver mcp` server as a normal MCP server, giving agents the CUA driver's own schemas and pid/window/element-index workflow without routing through the Codex marketplace or the PeekabooBridge socket.
|
||||
|
||||
Use Peekaboo for the broad macOS automation surface via OpenClaw.app's permission-aware bridge host. Use Codex Computer Use when a Codex-mode agent should rely on Codex's native plugin. Use direct `cua-driver mcp` to expose the CUA driver to any OpenClaw-managed runtime as a normal MCP server.
|
||||
Use Peekaboo for the broad macOS automation surface via OpenClaw.app's permission-aware bridge host. Use agent-driven computer use when the gateway agent should see and control the desktop through a uniform `computer.act` node command that any vision model can drive. Use Codex Computer Use when a Codex-mode agent should rely on Codex's native plugin. Use direct `cua-driver mcp` to expose the CUA driver to any OpenClaw-managed runtime as a normal MCP server.
|
||||
|
||||
## Enable the bridge
|
||||
|
||||
|
|
|
|||
|
|
@ -307,7 +307,7 @@ must be in the same identity group.
|
|||
| ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `/dreaming [on\|off\|status\|help]` | Toggle memory dreaming (owner or Gateway admin). See [Dreaming](/concepts/dreaming) |
|
||||
| `/pair [qr\|status\|pending\|approve\|cleanup\|notify]` | Manage device pairing. See [Pairing](/channels/pairing) |
|
||||
| `/phone status\|arm ...\|disarm` | Temporarily arm high-risk phone node commands |
|
||||
| `/phone status\|arm ...\|disarm` | Temporarily arm high-risk node commands (camera/screen/computer/writes). See [Computer use](/nodes/computer-use) |
|
||||
| `/voice status\|list\|set <voiceId>` | Manage Talk voice config. Discord native name: `/talkvoice` |
|
||||
| `/card ...` | Send LINE rich card presets. See [LINE](/channels/line) |
|
||||
| `/codex <action> ...` | Bind, steer, and inspect the Codex app-server harness (status, threads, resume, model, fast, permissions, compact, review, mcp, skills, and more). See [Codex harness](/plugins/codex-harness) |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue