vrr/openclaw
2
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-07-09 15:59:30 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 22

feat(nodes): add computer use (computer.act) on macOS nodes via Peekaboo (#102776)

Browse source 
* feat(nodes): add computer use (computer.act) on macOS nodes via Peekaboo

* fix(nodes): release held computer.act input on node disconnect/stop/disable

* fix(nodes): release held computer.act input armed after a lifecycle release

* fix(nodes): scope computer.act lifecycle catch-up release to the arming action

* chore(nodes): register computer tool display metadata, sync native i18n, doc-comment constants

* fix(nodes): preserve caught-error cause in computer tool resolver; regen docs map
This commit is contained in:
Peter Steinberger 2026-07-09 15:28:09 +01:00 • committed by GitHub
parent d9f8fc6322
commit f8dbd34ab8
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
35 changed files with 2632 additions and 127 deletions
Download patch file Download diff file Expand all files Collapse all files

5
docs/platforms/mac/peekaboo.md
View file

@ -18,13 +18,14 @@ OpenClaw can host **PeekabooBridge** as a local, permission-aware UI automation
## Relationship to other desktop-control paths
OpenClaw has three desktop-control paths that intentionally stay separate:
OpenClaw has four desktop-control paths that intentionally stay separate:
- **PeekabooBridge host**: OpenClaw.app hosts the local PeekabooBridge socket. The `peekaboo` CLI is the client and uses OpenClaw.app's macOS permissions for screenshots, clicks, menus, dialogs, Dock actions, and window management.
- **Agent-driven computer use (`computer.act`)**: the gateway agent's built-in `computer` tool captures screenshots via `screen.snapshot` and drives the pointer and keyboard through the dangerous `computer.act` node command. A macOS node fulfills `computer.act` in-process using the same embedded Peekaboo automation services this bridge exposes (`UIAutomationService`), so agent-driven control reuses the engine without going through the PeekabooBridge socket or the `peekaboo` CLI. See [Computer use](/nodes/computer-use).
- **Codex Computer Use**: the bundled `codex` plugin checks and can install Codex's `computer-use` MCP plugin (`extensions/codex/src/app-server/computer-use.ts`), then lets Codex own native desktop-control tool calls during Codex-mode turns. OpenClaw does not proxy those actions through PeekabooBridge.
- **Direct `cua-driver` MCP**: OpenClaw can register TryCua's upstream `cua-driver mcp` server as a normal MCP server, giving agents the CUA driver's own schemas and pid/window/element-index workflow without routing through the Codex marketplace or the PeekabooBridge socket.
Use Peekaboo for the broad macOS automation surface via OpenClaw.app's permission-aware bridge host. Use Codex Computer Use when a Codex-mode agent should rely on Codex's native plugin. Use direct `cua-driver mcp` to expose the CUA driver to any OpenClaw-managed runtime as a normal MCP server.
Use Peekaboo for the broad macOS automation surface via OpenClaw.app's permission-aware bridge host. Use agent-driven computer use when the gateway agent should see and control the desktop through a uniform `computer.act` node command that any vision model can drive. Use Codex Computer Use when a Codex-mode agent should rely on Codex's native plugin. Use direct `cua-driver mcp` to expose the CUA driver to any OpenClaw-managed runtime as a normal MCP server.
## Enable the bridge