vrr/CodeNomad
2
0
Fork 0
mirror of https://github.com/NeuralNomadsAI/CodeNomad.git synced 2026-10-19 15:34:12 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

feat(automation): complete safe native window interactions (#1018)

Browse source 
Complete application-window interaction on the shared page engine introduced
by #1015, while preserving native window/run and document-scoped ref authority.

Add real pointer clicks, hover, stepped pointer dragging, wheel scrolling,
keyboard chords and bounded text waits. Flat tool commands are validated at
both bridge boundaries. Clicks refuse covered targets; press(ref) refuses a
target that cannot acquire focus, so Enter never reaches an unrelated draft.
Drag requires both ends visible instead of scrolling under a measured source.
Preferences remains outside automation; focusing Preferences or a remote
window falls back to the most recently used local application window.

Trusted native input carries monotonic provenance intervals. Human Form
decisions check the originating event timestamp, not the time a stalled
renderer handles it; delayed keys and their implicit submit clicks remain
automated after lease expiry. Fixed page functions keep their depth marker.
Native events use the page's epoch clock, and cleanup preserves the original
press timestamp, independently of configured command timeouts.
Failed sequences best-effort release held buttons and keys, including when
the cleanup mark write itself times out. An admitted input timeout is reported
as an unknown outcome with inspection-before-retry guidance: the renderer may
deliver the input or finish the click later. Never replay these inputs.

Validation covers the actual Mission editor and InterruptionDock in isolated
Chromium, including a 16-second renderer freeze and a control human click.
Targeted server tests cover key mapping, both bridge parsers, focus/target
fences, failure cleanup and late input outcomes. Full server validation,
isolated OpenCode 2.0.31 automation, bundle regression and Tauri compilation
also pass; typecheck diagnostics remain at baseline. No installed application
or shared daemon was changed or restarted.

Four independent gatekeeper passes; the final review reports zero actionable
P0-P3 findings at 0ffb5a53b425500336854902c7068cd2233695f4.

Refs #1008.
This commit is contained in:
Pascal André 2026-10-19 08:48:14 +02:00 • committed by GitHub
parent 7eb401eb18
commit b2d5b722ec
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
23 changed files with 1118 additions and 240 deletions
Download patch file Download diff file Expand all files Collapse all files

2
.opencode/skills/codenomad-automation/SKILL.md
View file

@ -8,7 +8,7 @@ description: Inspect, modify, rebuild, relaunch, and verify CodeNomad desktop bu
Use this workflow when changing CodeNomad itself:
1. Call `codenomad.inspect` before editing. It identifies the local CodeNomad window and returns an indented accessibility outline with refs, plus runtime diagnostics. Large parts collapse with `(+N hidden)`/`(+N more)`: pass `ref` to expand one, or `query` to find elements by text or role.
2. Use refs from the latest inspections with `codenomad.act`: `click`, `type` (replaces the field's text; `clear: false` appends) and `select` (native `<select>` options by label or value). Each result returns the field value or chosen options to check. Inspect again after navigation, reload or failed actions.
2. Use refs from the latest inspections with `codenomad.act`: `click` (a real pointer click, refused when the element is covered), `type` (replaces the field's text; `clear: false` appends), `select` (native `<select>` options by label or value), `hover`, `drag` onto another ref, `scroll`, `press` a key or chord such as `Enter`, `Escape`, `Shift+Tab` or `Mod+K`, and `wait` for text to appear or disappear. Results return the field value, chosen options or focused element to check. Inspect again after navigation, reload or failed actions.
3. Make the smallest source change and run the relevant typecheck, native test, or build. Automation does not own OpenCode: the shared `opencode2` daemon and this session remain alive if CodeNomad closes.
4. After updated artifacts are ready, call `codenomad.act` with `action: "restart"`. The call gracefully relaunches the host pinned by the latest inspection, waits for the same artifact identity to reconnect to the OpenCode session, and returns a fresh inspection. It never switches to another worktree's build.
5. Call `codenomad.screenshot` for final visual evidence and report the checks that ran.

2
AGENTS.md
View file

@ -156,7 +156,7 @@
- One bundled `codenomad.automation` V2 plugin uses native discovery and backend presence. All browser/developer tools are available without a Developer Mode toggle; native instrumentation starts with the desktop host. Loading, tool availability and execution targeting are separate: retain the authenticated bridge and session/window fences. Tauri preview children receive no application capabilities; primary-renderer reload must dispose them and final-window checks must count native windows. See `dev-docs/BROWSER_AUTOMATION.md`.
- Developer UI tools target the inspected native window/run, not the currently selected project or conversation. Keep backend ownership discovery separate from UI focus; agents must be able to inspect and navigate back from another session themselves. Accessibility refs expire on document/target replacement, not merely on session selection. Browser previews retain their own session attachments.
- Page automation (outline, refs, click/type/select) lives once in the transport-agnostic `packages/server/src/automation/page/` engine; inspections stay budgeted plain-text outlines that announce what they collapse, never a silent cut. In the CodeNomad window page functions carry `__codenomadAutomationDepth`, clicks stay programmatic and native text insertion yields only input events, so automation can never produce the trusted dock click/key press a human Form answer requires: never add unmarked trusted input or arbitrary script evaluation there. See `dev-docs/BROWSER_AUTOMATION.md`.
- Page automation (outline, refs, click/type/select/hover/drag/scroll/press/wait) lives once in the transport-agnostic `packages/server/src/automation/page/` engine, with tool input validated by its `parse.ts` on both sides of the bridge; inspections stay budgeted plain-text outlines that announce what they collapse, never a silent cut. Pointer actions aim at the visible centre and refuse covered elements. In the CodeNomad window page functions carry `__codenomadAutomationDepth` and every trusted DevTools input the expiring `__codenomadAutomationInputUntil` lease, both read by `lib/automation-input.ts`, so automation can never produce the trusted dock click/key press a human Form answer requires: never add unmarked trusted input or arbitrary script evaluation there. See `dev-docs/BROWSER_AUTOMATION.md`.
- Desktop automation and pruning provisioning follows the authenticated OpenCode connection and its `config.get` global discovery directory, including reconnects. Never derive a running daemon's roots from backend/startup environment or CLI `debug paths`. OpenCode watches the entire discovery root recursively: keep changing presence leases outside it, in the sibling `.codenomad/<root-hash>/` namespace. Retain existing outside-root storage; migrate inside-root storage while reading older backends' leases without writing there. WSL translates the daemon-reported paths through the selected distro only for filesystem access. Verify heartbeat stability with the isolated native automation fixture.
- Plugins register only at their next presence check, so a backend's first plugin RPC could arrive before registration and be refused. Followers acknowledge a lease with a `<lease>.ready` marker (current namespace only) once every follower of that directory in the native process has observed it while registered; routes calling bundled-plugin RPCs await `workspaceManager.desktopPluginsReady()` before reading or authorizing the session, so ownership checks and deletion-fence entry stay adjacent (bounded to 5 s, never rejects). The bound is paid in full only when no marker can arrive: no Location loaded yet, or pre-marker plugin copies still loaded after an upgrade. Do not replace this ordering with retries of refused reads.

10
dev-docs/BROWSER_AUTOMATION.md
View file

@ -20,13 +20,17 @@ The bridge remains loopback-only, token-authenticated, bounded, and session-loca
## Page engine
`codenomad.inspect` and `codenomad.act` (the CodeNomad window) run on the shared page engine in `packages/server/src/automation/page/`, which only needs a DevTools protocol channel: `snapshot.ts` renders the accessibility tree, `actions.ts` acts through fixed page functions, and `refs.ts` names DOM nodes. Browser-preview tools still use their native controllers.
`codenomad.inspect` and `codenomad.act` (the CodeNomad window) run on the shared page engine in `packages/server/src/automation/page/`, which only needs a DevTools protocol channel: `snapshot.ts` renders the accessibility tree, `actions.ts` performs actions, `input.ts` dispatches trusted pointer, wheel and keyboard input, `keys.ts` maps key chords, `parse.ts` validates tool input on both sides of the bridge, `protocol.ts` holds the shared protocol primitives and `refs.ts` names DOM nodes. Browser-preview tools still use their native controllers.
- Inspection is an indented outline with refs, read as plain text. Wrappers, unnamed images and text repeating its parent's name are dropped; fields show their value instead of their inner nodes; native selects list their options with `selected`/`disabled`. A character budget (32 000 by default) is shared among siblings: small parts stay whole while large ones collapse in place with `(+N hidden)` or `(+N more)`, keeping their first and last items, so nothing is cut silently. `ref` expands one subtree and `query` keeps elements whose name, value or description contains it, or whose role is exactly it, with their contents and ancestors.
- A ref names one node within one target, run and navigation epoch. It keeps its name and stays usable for the life of that page generation, whatever is inspected later; a node that left the page fails its action explicitly.
- `type` focuses the field and refuses one that cannot take focus (hidden, disabled, behind a modal dialog), because native text insertion goes to the focused element. It then replaces the field's text by default (select, then native insertion, so framework input handlers fire); `clear: false` appends. Date, time, range and color inputs are validated before anything changes, then set with `input`/`change`, keeping the browser's normalization; email and number inputs append as a value. `select` picks native `<select>` options by label, else value, and dispatches `input`/`change` only when the choice changes; clicking an option ref picks that exact option. Results return the field value (passwords masked, rich text with its line breaks) or the chosen labels.
- Trust rule: in the CodeNomad window page functions run under `__codenomadAutomationDepth` and clicks stay programmatic. Native text insertion is unmarked but produces only input events, which the human-answer check never accepts: it requires a trusted click or key press on the dock's submit (`packages/ui/src/lib/form-answer-interaction.ts`). Never add trusted CDP mouse or keyboard input to the application window without a mark, and never expose arbitrary script evaluation there.
- `packages/ui/tests/browser/automation-forms.test.ts` drives a real Chromium page and the real Mission editor through `DeveloperCdp`.
- `click` is a real pointer click at the centre of the element's visible part, refused when another element would receive it (hit testing respects `pointer-events: none`, and a label of the target counts as the target); an element without a visible box is activated programmatically. `click` on a native select asks for `select` instead of opening its out-of-page popup. `hover` and `scroll` (wheel on a ref or the page) use the same aiming. `drag` requires both ends already visible, without scrolling under a saved start point; it uses stepped pointer moves for pointer-based drag libraries (HTML drag-and-drop is not emulated). `press` dispatches a key or chord (`Enter`, `Shift+Tab`, `Control+K`, `Mod+K` = Meta on macOS) to the focused element, optionally focusing a ref first and refusing it unless focus actually moved there, and returns the newly focused element. Failed input sequences best-effort release held buttons/keys on their original channel without replaying the action. `wait` polls (≤ 15 s) for text in the rendered page or in labels, titles, placeholders and values, or for its disappearance. Native application-menu accelerators handled by the host may not see dispatched keys.
- Trust rule: in the CodeNomad window page functions run under `__codenomadAutomationDepth`, and every trusted DevTools input (pointer, wheel, keys, text insertion) under the short `__codenomadAutomationInputUntil` lease in `performance.now()` units. Ending it retains its end timestamp: `lib/automation-input.ts` checks the originating event's `timeStamp`, not when a stalled renderer handles it. Delayed automated keys and their same-turn implicit submit clicks remain automated after expiry. `form-answer-interaction.ts` refuses human provenance for those events, so automation can never produce the trusted dock click or key press a human Form answer requires. Never add unmarked trusted input to the application window, and never expose arbitrary script evaluation there.
- An input timeout has an **unknown outcome**, not a proven failure: a busy renderer can deliver an already admitted key or click later, including when cleanup releases its held input. Cleanup still attempts release when its marker write times out, but never repeats the press. The tool explicitly instructs inspection before retrying; never automatically retry these inputs.
- The marker returns the page's own epoch timestamp for CDP mouse/key events. Cleanup retains the original press timestamp even if its marker renewal times out, so delayed releases remain within the original provenance interval independently of command-timeout settings. Missing/invalid provenance timestamps refuse before input dispatch.
- When a Preferences or remote window has focus, window automation targets the most recently used application window, as the update flow does; the Preferences window itself is not an automation target.
- `packages/ui/tests/browser/automation-forms.test.ts` drives a real Chromium page and the real Mission editor through `DeveloperCdp`; `automation-input.test.ts` covers trusted input and proves that automation clicks and Enter in the real InterruptionDock never carry the human-answer header, while a real user click does.
## September 2026 architecture decision

4
packages/electron-app/electron/main/main.ts
View file

@ -191,9 +191,11 @@ function runPrimary(firstIntent: LaunchIntent) {
const developerMode = new DeveloperMode({
devtoolsDataPath: sessionDataPath,
nativeIdentity: developerNativeIdentity,
// A focused Preferences or remote window is not an automation target: fall back to the
// most recently used application window instead of reporting no target at all.
targetWindowId: () => {
const focused = BrowserWindow.getFocusedWindow()
return focused ? registry.resolve(focused.webContents)?.id : registry.mruRecord()?.id
return (focused ? registry.resolve(focused.webContents)?.id : undefined) ?? registry.mruRecord()?.id
},
requestRelaunch: () => lifecycle.requestRelaunch(),
})

211
packages/server/src/automation/page/actions.ts
View file

@ -1,33 +1,48 @@
import { aim, clickAt, dragBetween, hoverAt, pressKeys, waitForText, wheelAt, type MouseButton } from "./input"
import { callOn, markedInput, OBJECT_GROUP, releaseObjects, resolveElement, VALUE_LIMIT, type PageSession } from "./protocol"
import type { RefAction } from "./parse"
import { renderSnapshot, type AxNode, type Snapshot, type SnapshotOptions } from "./snapshot"
/** One page's protocol connection, whatever carries it (DevTools socket or a native pipe). */
export interface PageChannel {
send(method: string, params?: Record<string, unknown>): Promise<Record<string, unknown>>
}
export type { PageChannel, PageSession } from "./protocol"
export interface PageSession {
channel: PageChannel
/** Throws once the page changed since the operation began; runs between protocol steps. */
checkpoint(): Promise<void>
/** The trusted CodeNomad window: page-side automation is marked so it never counts as a
* human answer (`form-answer-interaction.ts`). Never set this in a user's web page. */
markAutomation: boolean
}
interface Target { ref: string; backendNodeId: number }
export type PageAction =
| { kind: "click"; ref: string; backendNodeId: number }
| { kind: "type"; ref: string; backendNodeId: number; text: string; clear: boolean }
| { kind: "select"; ref: string; backendNodeId: number; values: string[] }
| Target & { kind: "click"; button?: MouseButton }
| Target & { kind: "type"; text: string; clear: boolean }
| Target & { kind: "select"; values: string[] }
| Target & { kind: "hover" }
| Target & { kind: "drag"; to: Target }
| { kind: "scroll"; target?: Target; direction: "up" | "down" | "left" | "right"; amount: number }
| { kind: "press"; key: string; target?: Target }
| { kind: "wait"; text: string; gone: boolean; timeoutMs: number }
export interface PageActionResult {
/** The field's value after typing; passwords are masked. */
value?: string
/** The select's chosen option labels. */
selected?: string[]
/** The element focused after a key press. */
focused?: string
/** Whether wait found its text, and after how long. */
found?: boolean
waitedMs?: number
}
const OBJECT_GROUP = "codenomad-automation"
const VALUE_LIMIT = 200
/** Resolves a tool's refs to DOM nodes before any page access; a stale ref fails the whole action. */
export function bindRefs(action: RefAction, resolve: (ref: string) => number): PageAction {
const target = (ref: string): Target => ({ ref, backendNodeId: resolve(ref) })
switch (action.action) {
case "click": return { kind: "click", ...target(action.ref), ...(action.button ? { button: action.button } : {}) }
case "type": return { kind: "type", ...target(action.ref), text: action.text, clear: action.clear }
case "select": return { kind: "select", ...target(action.ref), values: action.values }
case "hover": return { kind: "hover", ...target(action.ref) }
case "drag": return { kind: "drag", ...target(action.ref), to: target(action.to) }
case "scroll": return { kind: "scroll", direction: action.direction, amount: action.amount, ...(action.ref ? { target: target(action.ref) } : {}) }
case "press": return { kind: "press", key: action.key, ...(action.ref ? { target: target(action.ref) } : {}) }
case "wait": return { kind: "wait", text: action.text, gone: action.gone, timeoutMs: action.timeout }
}
}
export async function readAccessibilityTree(session: PageSession): Promise<AxNode[]> {
const result = await session.channel.send("Accessibility.getFullAXTree")
@ -43,90 +58,84 @@ export async function snapshotPage(session: PageSession, issue: () => (backendNo
}
export async function performAction(session: PageSession, action: PageAction): Promise<PageActionResult> {
const { channel } = session
try {
await channel.send("DOM.scrollIntoViewIfNeeded", { backendNodeId: action.backendNodeId }).catch(() => undefined)
await session.checkpoint()
const element = await resolve(session, action)
if (action.kind === "click") {
const selected = await call(session, action, element, CLICK, [])
return Array.isArray(selected) ? { selected: selected.map(String) } : {}
switch (action.kind) {
case "click": return await click(session, action)
case "type": return await type(session, action)
case "select": {
const element = await resolveElement(session, action.ref, action.backendNodeId)
return { selected: (await callOn(session, action.ref, element, SELECT, [action.values]) as unknown[]).map(String) }
}
case "hover": {
const point = await aim(session, action.ref, action.backendNodeId)
if (!point) throw new Error(`${action.ref} has no visible box to hover`)
await hoverAt(session, point)
return {}
}
case "drag": {
// Both ends must already be visible: scrolling to the second target would
// invalidate the first point and could press a different row underneath it.
const from = await aim(session, action.ref, action.backendNodeId, { scroll: false })
const to = await aim(session, action.to.ref, action.to.backendNodeId, { scroll: false })
if (!from || !to) throw new Error(`${!from ? action.ref : action.to.ref} has no visible box to drag ${!from ? "from" : "to"}`)
await dragBetween(session, from, to)
return {}
}
case "scroll": {
const point = action.target ? await aim(session, action.target.ref, action.target.backendNodeId) : undefined
if (action.target && !point) throw new Error(`${action.target.ref} has no visible box to scroll`)
const sign = action.direction === "up" || action.direction === "left" ? -1 : 1
const vertical = action.direction === "up" || action.direction === "down"
await wheelAt(session, point, { x: vertical ? 0 : sign * action.amount, y: vertical ? sign * action.amount : 0 })
return {}
}
case "press": {
if (action.target) {
const element = await resolveElement(session, action.target.ref, action.target.backendNodeId)
await callOn(session, action.target.ref, element, FOCUS, [])
}
await pressKeys(session, action.key)
return { focused: await focusedElement(session) }
}
case "wait": return await waitForText(session, action.text, action.gone, action.timeoutMs)
}
if (action.kind === "select") {
return { selected: (await call(session, action, element, SELECT, [action.values]) as unknown[]).map(String) }
}
const field = await call(session, action, element, FIELD, [], false) as string
const mode = await call(session, action, field, PREPARE, [action.clear])
if (mode === "value" || mode === "append") await call(session, action, field, SET_VALUE, [action.text, mode === "append"])
else if (action.text) {
await channel.send("Input.insertText", { text: action.text })
await session.checkpoint()
} else if (action.clear) await call(session, action, field, DELETE_SELECTION, [])
return { value: String(await call(session, action, field, READ_VALUE, [])) }
} finally {
await channel.send("Runtime.releaseObjectGroup", { objectGroup: OBJECT_GROUP }).catch(() => undefined)
await releaseObjects(session)
}
}
async function resolve(session: PageSession, action: PageAction): Promise<string> {
let result: Record<string, unknown>
try {
result = await session.channel.send("DOM.resolveNode", { backendNodeId: action.backendNodeId, objectGroup: OBJECT_GROUP })
} catch {
throw new Error(`${action.ref} is no longer in the page; inspect again`)
}
async function click(session: PageSession, action: Target & { button?: MouseButton }): Promise<PageActionResult> {
const element = await resolveElement(session, action.ref, action.backendNodeId)
const selected = await callOn(session, action.ref, element, OPTION_CLICK, [])
if (Array.isArray(selected)) return { selected: selected.map(String) }
const point = await aim(session, action.ref, action.backendNodeId)
// An element without a visible box (zero-size or offscreen-only) is activated programmatically.
if (!point) {
if (action.button && action.button !== "left") throw new Error(`${action.ref} has no visible box to ${action.button}-click`)
await callOn(session, action.ref, element, PROGRAMMATIC_CLICK, [])
} else await clickAt(session, point, action.button)
return {}
}
async function type(session: PageSession, action: Target & { text: string; clear: boolean }): Promise<PageActionResult> {
const element = await resolveElement(session, action.ref, action.backendNodeId)
await session.channel.send("DOM.scrollIntoViewIfNeeded", { backendNodeId: action.backendNodeId }).catch(() => undefined)
await session.checkpoint()
const objectId = (result.object as { objectId?: unknown } | undefined)?.objectId
if (typeof objectId !== "string") throw new Error(`${action.ref} is no longer in the page; inspect again`)
return objectId
const field = await callOn(session, action.ref, element, FIELD, [], false) as string
const mode = await callOn(session, action.ref, field, PREPARE, [action.clear])
if (mode === "value" || mode === "append") await callOn(session, action.ref, field, SET_VALUE, [action.text, mode === "append"])
else if (action.text) await markedInput(session, dispatch => dispatch("Input.insertText", { text: action.text }))
else if (action.clear) await callOn(session, action.ref, field, DELETE_SELECTION, [])
return { value: String(await callOn(session, action.ref, field, READ_VALUE, [])) }
}
/** Runs a fixed page function on an element; `byValue: false` returns another element handle. */
async function call(session: PageSession, action: PageAction, objectId: string, body: string, args: unknown[], byValue = true): Promise<unknown> {
const result = await session.channel.send("Runtime.callFunctionOn", {
objectId,
functionDeclaration: pageFunction(body, session.markAutomation),
arguments: args.map(value => ({ value })),
returnByValue: byValue,
objectGroup: OBJECT_GROUP,
})
async function focusedElement(session: PageSession): Promise<string | undefined> {
const active = await session.channel.send("Runtime.evaluate", { expression: "document.activeElement", objectGroup: OBJECT_GROUP })
await session.checkpoint()
const failure = result.exceptionDetails as { exception?: { description?: unknown }; text?: unknown } | undefined
if (failure) throw pageError(action, String(failure.exception?.description ?? failure.text ?? ""))
const value = result.result as { value?: unknown; objectId?: unknown } | undefined
if (!byValue) {
if (typeof value?.objectId !== "string") throw new Error(`${action.ref} is no longer in the page; inspect again`)
return value.objectId
}
return value?.value
}
function pageFunction(body: string, mark: boolean): string {
const element = `const element = this.nodeType === Node.ELEMENT_NODE ? this : this.parentElement;
if (!(element instanceof Element) || !element.isConnected) throw new Error("codenomad:gone");`
return mark
? `function (...args) { ${element} globalThis.__codenomadAutomationDepth = (globalThis.__codenomadAutomationDepth || 0) + 1;
try { return (() => { ${body} })(); } finally { globalThis.__codenomadAutomationDepth--; } }`
: `function (...args) { ${element} ${body} }`
}
function pageError(action: PageAction, description: string): Error {
const [code, detail = ""] = /codenomad:([a-z-]+)(?::(.*))?/.exec(description.split("\n", 1)[0])?.slice(1) ?? []
const quoted = JSON.stringify(detail.slice(0, VALUE_LIMIT))
switch (code) {
case "gone": return new Error(`${action.ref} is no longer in the page; inspect again`)
case "not-select": return new Error(`${action.ref} is not a native select; click it to open its menu, then click the option`)
case "no-option": return new Error(`No option ${quoted} in ${action.ref}; inspect it to see its options`)
case "option-disabled": return new Error(`Option ${quoted} of ${action.ref} is disabled`)
case "single": return new Error(`${action.ref} takes exactly one value`)
case "disabled": return new Error(`${action.ref} is disabled`)
case "readonly": return new Error(`${action.ref} is read-only`)
case "not-editable": return new Error(`${action.ref} is not a text field`)
case "not-text": return new Error(`${action.ref} is not a text field; use click`)
case "unfocusable": return new Error(`${action.ref} cannot take focus (hidden, disabled or behind a dialog); reveal it, then inspect again`)
case "invalid-value": return new Error(`${quoted} is not a valid value for ${action.ref}`)
default: return new Error(`The page rejected the action on ${action.ref}: ${description.split("\n", 1)[0].slice(0, VALUE_LIMIT)}`)
}
const objectId = (active.result as { objectId?: unknown } | undefined)?.objectId
if (typeof objectId !== "string") return undefined
const described = await callOn(session, "the focused element", objectId, DESCRIBE, [])
return typeof described === "string" ? described : undefined
}
const LABEL = `const label = option => (option.label || option.textContent || "").replace(/\\s+/g, " ").trim();`
@ -159,11 +168,15 @@ const CHOOSE = `${LABEL}
return apply(select, picked);
};`
/** Clicking an option of a native select picks that exact option, since its popup is not part of the page. */
const CLICK = `${CHOOSE}
/** A native select's popup is not part of the page: clicking an option picks that exact option,
* and the select itself is changed with `select` rather than opened. */
const OPTION_CLICK = `${CHOOSE}
const option = element.closest("option"), select = option && option.closest("select");
if (option && select) return apply(select, [option]);
if (element instanceof HTMLElement) element.click();
if (element instanceof HTMLSelectElement) throw new Error("codenomad:is-select");
return null;`
const PROGRAMMATIC_CLICK = `if (element instanceof HTMLElement) element.click();
else element.dispatchEvent(new MouseEvent("click", { bubbles: true, cancelable: true, composed: true }));
return null;`
@ -173,6 +186,16 @@ const SELECT = `${CHOOSE}
if (!select) throw new Error("codenomad:not-select");
return choose(select, args[0]);`
const FOCUS = `if (element.matches(":disabled")) throw new Error("codenomad:disabled");
if (element instanceof HTMLElement || element instanceof SVGElement) element.focus();
if (element.getRootNode().activeElement !== element) throw new Error("codenomad:unfocusable");
return null;`
const DESCRIBE = `const role = element.getAttribute("role") || element.tagName.toLowerCase();
const name = (element.getAttribute("aria-label") || element.getAttribute("title") || element.getAttribute("placeholder")
|| (element.labels && element.labels[0] && element.labels[0].textContent) || element.textContent || "").replace(/\\s+/g, " ").trim().slice(0, 80);
return role + (name ? " " + JSON.stringify(name) : "");`
const NOT_TEXT = ["checkbox", "radio", "button", "submit", "reset", "file", "image", "hidden"]
const VALUE_TYPES = ["date", "time", "datetime-local", "month", "week", "range", "color"]
/** Types whose value sanitization empties an invalid value. */

65
packages/server/src/automation/page/input.test.ts Normal file
View file

@ -0,0 +1,65 @@
import assert from "node:assert/strict"
import { test } from "node:test"
import { clickAt, dragBetween, pressKeys } from "./input"
import type { PageSession } from "./protocol"
test("failed pointer and key sequences release potentially held inputs without replaying the action", async () => {
for (const kind of ["click", "drag", "keys"] as const) {
const calls: Array<{ method: string; params: Record<string, unknown> }> = []
let pressed = false, failed = false
const session: PageSession = {
markAutomation: true,
channel: { send: async (method, params = {}) => {
calls.push({ method, params })
if (params.type === "mousePressed" || params.type === "rawKeyDown") pressed = true
return method === "Runtime.evaluate" ? { result: { value: 1_700_000_000 } } : {}
} },
checkpoint: async () => { if (pressed && !failed) { failed = true; throw new Error("Page changed") } },
}
await assert.rejects(kind === "click" ? clickAt(session, { x: 10, y: 20 })
: kind === "drag" ? dragBetween(session, { x: 10, y: 20 }, { x: 50, y: 60 }) : pressKeys(session, "Control+K"), /Page changed/)
const inputs = calls.filter(call => call.method.startsWith("Input."))
assert.equal(inputs.filter(call => call.params.type === "mousePressed" || call.params.type === "rawKeyDown").length, 1,
"the action is admitted only once")
assert.equal(inputs.at(-1)?.params.type, kind === "keys" ? "keyUp" : "mouseReleased")
assert.equal(inputs.at(-1)?.params[kind === "keys" ? "key" : "button"], kind === "keys" ? "Control" : "left")
assert.match(String(calls.at(-1)?.params.expression), /Math.min/, "the retained input interval ends only after release")
}
})
test("a frozen mark write cannot suppress release, and late input effects are reported as unknown", async () => {
for (const recoversBeforeRelease of [true, false]) {
const calls: Array<{ method: string; params: Record<string, unknown> }> = []
let frozen = false, clicks = 0, queuedRelease: (() => void) | undefined
const session: PageSession = {
markAutomation: true,
checkpoint: async () => {},
channel: { send: async (method, params = {}) => {
calls.push({ method, params })
if (method === "Runtime.evaluate" && frozen) throw new Error("CDP mark timed out")
if (params.type === "mousePressed") { frozen = true; throw new Error("CDP input timed out") }
if (params.type === "mouseReleased") {
if (recoversBeforeRelease) clicks++
else { queuedRelease = () => { clicks++ }; throw new Error("CDP release timed out") }
}
return method === "Runtime.evaluate" ? { result: { value: 1_700_000_000 } } : {}
} },
}
await assert.rejects(clickAt(session, { x: 10, y: 20 }), /outcome is unknown.*may still be delivered; inspect before retrying/)
assert.equal(calls.filter(call => call.params.type === "mousePressed").length, 1)
assert.equal(calls.filter(call => call.params.type === "mouseReleased").length, 1,
"release is attempted even if its mark write times out")
assert.equal(calls.find(call => call.params.type === "mouseReleased")?.params.timestamp,
calls.find(call => call.params.type === "mousePressed")?.params.timestamp, "late release retains the original press timestamp")
queuedRelease?.()
assert.equal(clicks, 1, "cleanup can complete the admitted click after a timeout, but never replays its press")
}
})
test("input refuses when the page cannot supply a valid provenance timestamp", async () => {
const methods: string[] = []
await assert.rejects(pressKeys({ markAutomation: true, checkpoint: async () => {},
channel: { send: async method => { methods.push(method); return { result: { value: null } } } },
}, "Enter"), /Cannot establish automation input provenance/)
assert.ok(methods.every(method => method === "Runtime.evaluate"))
})

140
packages/server/src/automation/page/input.ts Normal file
View file

@ -0,0 +1,140 @@
import { keyChord } from "./keys"
import { callOn, Handle, markedInput, OBJECT_GROUP, resolveElement, type PageSession } from "./protocol"
/** Trusted pointer, wheel and keyboard input through DevTools, aimed like a user would:
* at the element's visible centre, only when that point actually reaches the element. */
export interface Point { x: number; y: number }
/** The point a user would click, or undefined for an element without a visible box. */
export async function aim(session: PageSession, ref: string, backendNodeId: number, options: { hitTest?: boolean; scroll?: boolean } = {}): Promise<Point | undefined> {
const { channel } = session
if (options.scroll !== false) await channel.send("DOM.scrollIntoViewIfNeeded", { backendNodeId }).catch(() => undefined)
await session.checkpoint()
const result = await channel.send("DOM.getContentQuads", { backendNodeId }).catch(() => ({} as Record<string, unknown>))
await session.checkpoint()
const quads = Array.isArray(result.quads) ? result.quads as number[][] : []
const viewport = await layoutViewport(session)
const quad = quads.find(item => item.length === 8 && area(item) >= 1)
if (!quad) return undefined
// The visible part of the box, so a half-clipped element is aimed where it shows.
const xs = [quad[0], quad[2], quad[4], quad[6]], ys = [quad[1], quad[3], quad[5], quad[7]]
const left = Math.max(0, Math.min(...xs)), right = Math.min(viewport.width, Math.max(...xs))
const top = Math.max(0, Math.min(...ys)), bottom = Math.min(viewport.height, Math.max(...ys))
if (right - left < 1 || bottom - top < 1) return undefined
const point = { x: (left + right) / 2, y: (top + bottom) / 2 }
if (options.hitTest !== false) await assertReceives(session, ref, backendNodeId, point)
return point
}
/** Refuses a point that another element covers, instead of acting on whatever is on top. */
async function assertReceives(session: PageSession, ref: string, backendNodeId: number, point: Point): Promise<void> {
// Like real events, hit testing passes through `pointer-events: none` layers (toasts, overlays' gaps).
const hit = await session.channel.send("DOM.getNodeForLocation", { x: Math.floor(point.x), y: Math.floor(point.y),
includeUserAgentShadowDOM: false, ignorePointerEventsNone: false }).catch(() => undefined)
await session.checkpoint()
const hitNode = hit?.backendNodeId
if (typeof hitNode !== "number" || hitNode === backendNodeId) return
const target = await resolveElement(session, ref, backendNodeId)
const other = await resolveElement(session, ref, hitNode).catch(() => undefined)
if (!other) return
const covered = await callOn(session, ref, target, RECEIVES, [new Handle(other)])
if (typeof covered === "string") throw new Error(`${ref} is covered by ${covered}; close or move it first, then inspect again`)
}
const RECEIVES = `const hit = args[0] && (args[0].nodeType === Node.ELEMENT_NODE ? args[0] : args[0].parentElement);
if (!hit || element === hit || element.contains(hit)) return null;
const label = hit.closest("label");
if (label && (label.control === element || label.contains(element))) return null;
const name = (hit.getAttribute("aria-label") || hit.getAttribute("title") || hit.innerText || "").replace(/\\s+/g, " ").trim().slice(0, 60);
return hit.tagName.toLowerCase() + (name ? " " + JSON.stringify(name) : "");`
async function layoutViewport(session: PageSession): Promise<{ width: number; height: number }> {
const metrics = await session.channel.send("Page.getLayoutMetrics").catch(() => ({} as Record<string, unknown>))
const viewport = (metrics.cssLayoutViewport ?? metrics.layoutViewport) as { clientWidth?: number; clientHeight?: number } | undefined
return { width: viewport?.clientWidth ?? Number.POSITIVE_INFINITY, height: viewport?.clientHeight ?? Number.POSITIVE_INFINITY }
}
function area(quad: number[]): number {
let sum = 0
for (let index = 0; index < 4; index++) {
const next = (index + 1) % 4
sum += quad[index * 2] * quad[next * 2 + 1] - quad[next * 2] * quad[index * 2 + 1]
}
return Math.abs(sum) / 2
}
const BUTTONS = { left: 1, right: 2, middle: 4 } as const
export type MouseButton = keyof typeof BUTTONS
export async function clickAt(session: PageSession, point: Point, button: MouseButton = "left"): Promise<void> {
await markedInput(session, async dispatch => {
await dispatch("Input.dispatchMouseEvent", { type: "mouseMoved", ...point })
await dispatch("Input.dispatchMouseEvent", { type: "mousePressed", ...point, button, buttons: BUTTONS[button], clickCount: 1 })
await dispatch("Input.dispatchMouseEvent", { type: "mouseReleased", ...point, button, buttons: 0, clickCount: 1 })
})
}
export async function hoverAt(session: PageSession, point: Point): Promise<void> {
await markedInput(session, dispatch => dispatch("Input.dispatchMouseEvent", { type: "mouseMoved", ...point }))
}
export async function wheelAt(session: PageSession, point: Point | undefined, delta: { x: number; y: number }): Promise<void> {
const at = point ?? await viewportCentre(session)
await markedInput(session, async dispatch => {
await dispatch("Input.dispatchMouseEvent", { type: "mouseMoved", ...at })
await dispatch("Input.dispatchMouseEvent", { type: "mouseWheel", ...at, deltaX: delta.x, deltaY: delta.y })
})
}
/** A pointer drag in small steps, so pointer-based drag libraries pass their activation distance. */
export async function dragBetween(session: PageSession, from: Point, to: Point): Promise<void> {
await markedInput(session, async dispatch => {
await dispatch("Input.dispatchMouseEvent", { type: "mouseMoved", ...from })
await dispatch("Input.dispatchMouseEvent", { type: "mousePressed", ...from, button: "left", buttons: 1, clickCount: 1 })
const steps = 10
for (let step = 1; step <= steps; step++) {
await dispatch("Input.dispatchMouseEvent", { type: "mouseMoved", x: from.x + (to.x - from.x) * step / steps,
y: from.y + (to.y - from.y) * step / steps, button: "left", buttons: 1 })
}
await dispatch("Input.dispatchMouseEvent", { type: "mouseReleased", ...to, button: "left", buttons: 0, clickCount: 1 })
})
}
export async function pressKeys(session: PageSession, chord: string): Promise<void> {
const events = keyChord(chord)
await markedInput(session, async dispatch => {
for (const event of events) await dispatch("Input.dispatchKeyEvent", event)
})
}
async function viewportCentre(session: PageSession): Promise<Point> {
const { width, height } = await layoutViewport(session)
return { x: Number.isFinite(width) ? width / 2 : 400, y: Number.isFinite(height) ? height / 2 : 300 }
}
/** Polls the page for text, in its rendered text or in labels, titles, placeholders and values. */
export async function waitForText(session: PageSession, text: string, gone: boolean, timeoutMs: number): Promise<{ found: boolean; waitedMs: number }> {
const started = Date.now()
for (;;) {
const root = await session.channel.send("Runtime.evaluate", { expression: "document.documentElement", objectGroup: OBJECT_GROUP })
await session.checkpoint()
const objectId = (root.result as { objectId?: unknown } | undefined)?.objectId
const found = typeof objectId === "string" && await callOn(session, "the page", objectId, CONTAINS, [text]) === true
const waitedMs = Date.now() - started
if (found !== gone) return { found, waitedMs }
if (waitedMs >= timeoutMs) {
throw new Error(gone ? `${JSON.stringify(text)} is still shown after ${timeoutMs} ms` : `${JSON.stringify(text)} did not appear within ${timeoutMs} ms`)
}
await new Promise(resolve => setTimeout(resolve, Math.min(200, timeoutMs - waitedMs)))
}
}
const CONTAINS = `const needle = String(args[0]).replace(/\\s+/g, " ").trim().toLowerCase();
const root = element;
const text = value => String(value || "").replace(/\\s+/g, " ").toLowerCase();
if (text(root.innerText || root.textContent).includes(needle)) return true;
for (const item of root.querySelectorAll("[aria-label], [title], [placeholder], input, textarea")) {
if ([item.getAttribute("aria-label"), item.getAttribute("title"), item.getAttribute("placeholder"), item.value].some(value => text(value).includes(needle))) return true;
}
return false;`

35
packages/server/src/automation/page/keys.test.ts Normal file
View file

@ -0,0 +1,35 @@
import assert from "node:assert/strict"
import { test } from "node:test"
import { keyChord } from "./keys"
const brief = (chord: string, platform: NodeJS.Platform = "win32") =>
keyChord(chord, platform).map(event => `${event.type}:${event.key}:${event.modifiers}${event.text ? `:${JSON.stringify(event.text)}` : ""}`)
test("plain keys type their text; named keys dispatch without text except Enter and Space", () => {
assert.deepEqual(brief("a"), ["keyDown:a:0:\"a\"", "keyUp:a:0"])
assert.deepEqual(brief("Enter"), ["keyDown:Enter:0:\"\\r\"", "keyUp:Enter:0"])
assert.deepEqual(brief("Space"), ["keyDown: :0:\" \"", "keyUp: :0"])
assert.deepEqual(brief("Escape"), ["rawKeyDown:Escape:0", "keyUp:Escape:0"])
assert.deepEqual(brief("esc"), brief("Escape"), "aliases and case-insensitive names")
assert.deepEqual(brief("arrowdown"), ["rawKeyDown:ArrowDown:0", "keyUp:ArrowDown:0"])
assert.equal(keyChord("F5")[0].windowsVirtualKeyCode, 116)
assert.deepEqual(keyChord("/").map(event => event.code), ["Slash", "Slash"])
})
test("chords hold modifiers around the key, type nothing, and release in reverse", () => {
assert.deepEqual(brief("Control+Shift+K"), [
"rawKeyDown:Control:2", "rawKeyDown:Shift:10", "rawKeyDown:K:10", "keyUp:K:10", "keyUp:Shift:2", "keyUp:Control:0"])
assert.deepEqual(brief("Shift+Tab"), ["rawKeyDown:Shift:8", "rawKeyDown:Tab:8", "keyUp:Tab:8", "keyUp:Shift:0"])
assert.deepEqual(brief("Shift+a"), ["rawKeyDown:Shift:8", "keyDown:A:8:\"A\"", "keyUp:A:8", "keyUp:Shift:0"], "shift types the capital")
assert.deepEqual(brief("Mod+k", "darwin").slice(0, 2), ["rawKeyDown:Meta:4", "rawKeyDown:k:4"])
assert.deepEqual(brief("Mod+k", "linux").slice(0, 2), ["rawKeyDown:Control:2", "rawKeyDown:k:2"])
assert.deepEqual(brief("ctrl+cmd+alt+x").slice(0, 3), ["rawKeyDown:Control:2", "rawKeyDown:Meta:6", "rawKeyDown:Alt:7"])
assert.deepEqual(brief("+"), ["keyDown:+:0:\"+\"", "keyUp:+:0"])
})
test("unknown keys and modifiers are refused with the accepted vocabulary", () => {
assert.throws(() => keyChord("Hyper+Q"), /Unknown modifier "Hyper".*Control, Shift, Alt, Meta or Mod/)
assert.throws(() => keyChord("Enterr"), /Unknown key "Enterr"; use a character or a key name/)
assert.throws(() => keyChord("Control+"), /Unknown key/)
assert.throws(() => keyChord("é"), /Unknown key "é"/)
})

115
packages/server/src/automation/page/keys.ts Normal file
View file

@ -0,0 +1,115 @@
/** Keyboard chords for `press`, as DevTools `Input.dispatchKeyEvent` sequences. */
interface KeyDefinition {
key: string
code: string
keyCode: number
/** Text a plain press inserts (Enter submits forms and breaks lines). */
text?: string
location?: number
}
export interface KeyEvent extends Record<string, unknown> {
type: "keyDown" | "rawKeyDown" | "keyUp"
key: string
code: string
windowsVirtualKeyCode: number
modifiers: number
text?: string
unmodifiedText?: string
location?: number
}
const MODIFIERS: Record<string, { definition: KeyDefinition; bit: number }> = {
Alt: { definition: { key: "Alt", code: "AltLeft", keyCode: 18, location: 1 }, bit: 1 },
Control: { definition: { key: "Control", code: "ControlLeft", keyCode: 17, location: 1 }, bit: 2 },
Meta: { definition: { key: "Meta", code: "MetaLeft", keyCode: 91, location: 1 }, bit: 4 },
Shift: { definition: { key: "Shift", code: "ShiftLeft", keyCode: 16, location: 1 }, bit: 8 },
}
const MODIFIER_ALIASES: Record<string, string> = { alt: "Alt", option: "Alt", control: "Control", ctrl: "Control",
meta: "Meta", cmd: "Meta", command: "Meta", shift: "Shift" }
const NAMED: Record<string, KeyDefinition> = {
Enter: { key: "Enter", code: "Enter", keyCode: 13, text: "\r" },
Tab: { key: "Tab", code: "Tab", keyCode: 9 },
Escape: { key: "Escape", code: "Escape", keyCode: 27 },
Backspace: { key: "Backspace", code: "Backspace", keyCode: 8 },
Delete: { key: "Delete", code: "Delete", keyCode: 46 },
Space: { key: " ", code: "Space", keyCode: 32, text: " " },
ArrowUp: { key: "ArrowUp", code: "ArrowUp", keyCode: 38 },
ArrowDown: { key: "ArrowDown", code: "ArrowDown", keyCode: 40 },
ArrowLeft: { key: "ArrowLeft", code: "ArrowLeft", keyCode: 37 },
ArrowRight: { key: "ArrowRight", code: "ArrowRight", keyCode: 39 },
Home: { key: "Home", code: "Home", keyCode: 36 },
End: { key: "End", code: "End", keyCode: 35 },
PageUp: { key: "PageUp", code: "PageUp", keyCode: 33 },
PageDown: { key: "PageDown", code: "PageDown", keyCode: 34 },
Insert: { key: "Insert", code: "Insert", keyCode: 45 },
ContextMenu: { key: "ContextMenu", code: "ContextMenu", keyCode: 93 },
...Object.fromEntries(Array.from({ length: 12 }, (_, index) => [`F${index + 1}`, { key: `F${index + 1}`, code: `F${index + 1}`, keyCode: 112 + index }])),
}
const NAMED_ALIASES: Record<string, string> = { esc: "Escape", return: "Enter", del: "Delete", up: "ArrowUp", down: "ArrowDown",
left: "ArrowLeft", right: "ArrowRight", pgup: "PageUp", pgdn: "PageDown", " ": "Space" }
/** US-layout punctuation: [code, key code, shifted character]. */
const PUNCTUATION: Record<string, [string, number, string?]> = {
"`": ["Backquote", 192, "~"], "-": ["Minus", 189, "_"], "=": ["Equal", 187, "+"], "[": ["BracketLeft", 219, "{"],
"]": ["BracketRight", 221, "}"], "\\": ["Backslash", 220, "|"], ";": ["Semicolon", 186, ":"], "'": ["Quote", 222, "\""],
",": ["Comma", 188, "<"], ".": ["Period", 190, ">"], "/": ["Slash", 191, "?"],
}
const SHIFTED_DIGITS = ")!@#$%^&*("
function character(value: string): KeyDefinition | undefined {
if (value.length !== 1) return undefined
if (/[a-z]/i.test(value)) return { key: value, code: `Key${value.toUpperCase()}`, keyCode: value.toUpperCase().charCodeAt(0), text: value }
if (/[0-9]/.test(value)) return { key: value, code: `Digit${value}`, keyCode: value.charCodeAt(0), text: value }
const shiftedDigit = SHIFTED_DIGITS.indexOf(value)
if (shiftedDigit >= 0) return { key: value, code: `Digit${shiftedDigit}`, keyCode: 48 + shiftedDigit, text: value }
for (const [base, [code, keyCode, shifted]] of Object.entries(PUNCTUATION)) {
if (value === base || value === shifted) return { key: value, code, keyCode, text: value }
}
return undefined
}
/** Parses "Enter", "a", "Shift+Tab", "Control+Shift+K" or "Mod+K" (Meta on macOS, Control elsewhere). */
export function keyChord(chord: string, platform: NodeJS.Platform = process.platform): KeyEvent[] {
const parts = chord === "+" ? ["+"] : chord.split("+").map(part => part.trim())
if (!parts.length || parts.some(part => !part) || chord.length > 64) throw new Error(`Unknown key ${JSON.stringify(chord)}`)
const main = parts.pop()!
const modifiers: string[] = []
for (const part of parts) {
const name = part.toLowerCase() === "mod" ? (platform === "darwin" ? "Meta" : "Control") : MODIFIER_ALIASES[part.toLowerCase()]
if (!name) throw new Error(`Unknown modifier ${JSON.stringify(part)} in ${JSON.stringify(chord)}; use Control, Shift, Alt, Meta or Mod`)
if (!modifiers.includes(name)) modifiers.push(name)
}
const named = NAMED[main] ?? NAMED[NAMED_ALIASES[main.toLowerCase()] ?? ""]
?? Object.values(NAMED).find(definition => definition.key.toLowerCase() === main.toLowerCase() && definition.key.length > 1)
let definition = named ?? character(main)
if (!definition) throw new Error(`Unknown key ${JSON.stringify(main)}; use a character or a key name such as Enter, Escape, Tab or ArrowDown`)
// Shift+a types "A", like a physical keyboard.
if (modifiers.includes("Shift") && definition.text && /^[a-z]$/.test(definition.text)) {
definition = { ...definition, key: definition.text.toUpperCase(), text: definition.text.toUpperCase() }
}
const events: KeyEvent[] = []
let bits = 0
for (const name of modifiers) {
const { definition: modifier, bit } = MODIFIERS[name]
bits |= bit
events.push(keyEvent("rawKeyDown", modifier, bits))
}
// A shortcut types nothing; a plain or shifted press produces its text.
const text = bits & ~MODIFIERS.Shift.bit ? undefined : definition.text
events.push(keyEvent(text ? "keyDown" : "rawKeyDown", definition, bits, text))
events.push(keyEvent("keyUp", definition, bits))
for (const name of [...modifiers].reverse()) {
const { definition: modifier, bit } = MODIFIERS[name]
bits &= ~bit
events.push(keyEvent("keyUp", modifier, bits))
}
return events
}
function keyEvent(type: KeyEvent["type"], definition: KeyDefinition, modifiers: number, text?: string): KeyEvent {
return { type, key: definition.key, code: definition.code, windowsVirtualKeyCode: definition.keyCode, modifiers,
...(text ? { text, unmodifiedText: text } : {}), ...(definition.location ? { location: definition.location } : {}) }
}

86
packages/server/src/automation/page/parse.ts Normal file
View file

@ -0,0 +1,86 @@
import { keyChord } from "./keys"
/** Page actions as tools receive them, with elements as refs from the latest inspection. Parsing is
* idempotent, so the same flat command crosses the bridge and is validated again by the backend. */
export type RefAction =
| { action: "click"; ref: string; button?: "left" | "right" }
| { action: "type"; ref: string; text: string; clear: boolean }
| { action: "select"; ref: string; values: string[] }
| { action: "hover"; ref: string }
| { action: "drag"; ref: string; to: string }
| { action: "scroll"; ref?: string; direction: "up" | "down" | "left" | "right"; amount: number }
| { action: "press"; key: string; ref?: string }
| { action: "wait"; text: string; gone: boolean; timeout: number }
export const PAGE_ACTIONS = ["click", "type", "select", "hover", "drag", "scroll", "press", "wait"] as const
export const MAX_WAIT_MS = 15_000
const MAX_SELECT_VALUES = 64
const MAX_TEXT_LENGTH = 200
const MAX_SCROLL = 10_000
/** Validates one page action; `undefined` when `action` names no page action. */
export function parsePageAction(value: Record<string, unknown>, inspect: string): RefAction | undefined {
const ref = (field = "ref") => {
if (typeof value[field] !== "string" || !value[field]) throw new Error(`${value.action} requires ${field}: a ref from the latest ${inspect}`)
return value[field] as string
}
const optionalRef = () => value.ref === undefined ? {} : { ref: ref() }
switch (value.action) {
case "click":
if (value.button !== undefined && value.button !== "left" && value.button !== "right") throw new Error("click button must be left or right")
return { action: "click", ref: ref(), ...(value.button === "right" ? { button: "right" as const } : {}) }
case "type":
if (typeof value.text !== "string") throw new Error("type requires ref and text")
if (value.clear !== undefined && typeof value.clear !== "boolean") throw new Error("type clear must be true or false")
return { action: "type", ref: ref(), text: value.text, clear: value.clear !== false }
case "select":
if (!Array.isArray(value.values) || value.values.length > MAX_SELECT_VALUES || !value.values.every(item => typeof item === "string")) {
throw new Error("select requires ref and values: the option labels or values to choose")
}
return { action: "select", ref: ref(), values: value.values as string[] }
case "hover":
return { action: "hover", ref: ref() }
case "drag":
return { action: "drag", ref: ref(), to: ref("to") }
case "scroll": {
if (!["up", "down", "left", "right"].includes(value.direction as string)) throw new Error("scroll requires direction: up, down, left or right")
const amount = value.amount ?? 600
if (typeof amount !== "number" || !Number.isInteger(amount) || amount < 1 || amount > MAX_SCROLL) {
throw new Error(`scroll amount must be a whole number of pixels from 1 to ${MAX_SCROLL}`)
}
return { action: "scroll", ...optionalRef(), direction: value.direction as "up" | "down" | "left" | "right", amount }
}
case "press":
if (typeof value.key !== "string" || !value.key) throw new Error("press requires key, such as Enter, Escape, Tab, ArrowDown or Control+K")
keyChord(value.key)
return { action: "press", key: value.key, ...optionalRef() }
case "wait": {
if (typeof value.text !== "string" || !value.text.trim() || value.text.length > MAX_TEXT_LENGTH) {
throw new Error(`wait requires text of at most ${MAX_TEXT_LENGTH} characters`)
}
if (value.gone !== undefined && typeof value.gone !== "boolean") throw new Error("wait gone must be true or false")
const timeout = value.timeout ?? 5_000
if (typeof timeout !== "number" || !Number.isInteger(timeout) || timeout < 0 || timeout > MAX_WAIT_MS) {
throw new Error(`wait timeout must be a whole number of milliseconds from 0 to ${MAX_WAIT_MS}`)
}
return { action: "wait", text: value.text, gone: value.gone === true, timeout }
}
default:
return undefined
}
}
/** JSON Schema properties shared by the tools that accept page actions. */
export const PAGE_ACTION_PROPERTIES = {
ref: { type: "string", description: "Element ref from the latest inspection (optional for scroll and press)" },
text: { type: "string", description: "type: text to enter; wait: text to wait for" },
clear: { type: "boolean", description: "type only: replace the field's current text (default true); false appends" },
values: { type: "array", items: { type: "string" }, description: "select only: option labels or values to choose; one for an ordinary select" },
button: { type: "string", enum: ["left", "right"], description: "click only: mouse button (default left)" },
to: { type: "string", description: "drag only: ref of the element to drop onto" },
direction: { type: "string", enum: ["up", "down", "left", "right"], description: "scroll only" },
amount: { type: "integer", minimum: 1, maximum: MAX_SCROLL, description: "scroll only: pixels (default 600)" },
key: { type: "string", description: "press only: a key or chord such as Enter, Escape, Tab, Shift+Tab, ArrowDown, a, Control+K or Mod+K (Meta on macOS)" },
gone: { type: "boolean", description: "wait only: wait until the text disappears instead" },
timeout: { type: "integer", minimum: 0, maximum: MAX_WAIT_MS, description: "wait only: milliseconds (default 5000)" },
} as const

151
packages/server/src/automation/page/protocol.ts Normal file
View file

@ -0,0 +1,151 @@
/** Protocol primitives shared by the page engine's actions. */
/** One page's protocol connection, whatever carries it (DevTools socket or a native pipe). */
export interface PageChannel {
send(method: string, params?: Record<string, unknown>): Promise<Record<string, unknown>>
}
export interface PageSession {
channel: PageChannel
/** Throws once the page changed since the operation began; runs between protocol steps. */
checkpoint(): Promise<void>
/** The trusted CodeNomad window: page-side automation and trusted input are marked so they
* never count as a human answer (`form-answer-interaction.ts`). Never set this in a user's web page. */
markAutomation: boolean
}
/** An element handle passed to a page function, as opposed to a plain value. */
export class Handle {
constructor(readonly objectId: string) {}
}
export const OBJECT_GROUP = "codenomad-automation"
export const VALUE_LIMIT = 200
export function gone(ref: string): Error {
return new Error(`${ref} is no longer in the page; inspect again`)
}
export async function resolveElement(session: PageSession, ref: string, backendNodeId: number): Promise<string> {
let result: Record<string, unknown>
try {
result = await session.channel.send("DOM.resolveNode", { backendNodeId, objectGroup: OBJECT_GROUP })
} catch {
throw gone(ref)
}
await session.checkpoint()
const objectId = (result.object as { objectId?: unknown } | undefined)?.objectId
if (typeof objectId !== "string") throw gone(ref)
return objectId
}
/** Runs a fixed page function on an element; inputs travel as protocol arguments, never as source.
* `byValue: false` returns another element handle. */
export async function callOn(session: PageSession, ref: string, objectId: string, body: string, args: unknown[], byValue = true): Promise<unknown> {
const result = await session.channel.send("Runtime.callFunctionOn", {
objectId,
functionDeclaration: pageFunction(body, session.markAutomation),
arguments: args.map(value => value instanceof Handle ? { objectId: value.objectId } : { value }),
returnByValue: byValue,
objectGroup: OBJECT_GROUP,
})
await session.checkpoint()
const failure = result.exceptionDetails as { exception?: { description?: unknown }; text?: unknown } | undefined
if (failure) throw pageError(ref, String(failure.exception?.description ?? failure.text ?? ""))
const value = result.result as { value?: unknown; objectId?: unknown } | undefined
if (!byValue) {
if (typeof value?.objectId !== "string") throw gone(ref)
return value.objectId
}
return value?.value
}
export async function releaseObjects(session: PageSession): Promise<void> {
await session.channel.send("Runtime.releaseObjectGroup", { objectGroup: OBJECT_GROUP }).catch(() => undefined)
}
function pageFunction(body: string, mark: boolean): string {
const element = `const element = this.nodeType === Node.ELEMENT_NODE ? this : this.parentElement;
if (!(element instanceof Element) || !element.isConnected) throw new Error("codenomad:gone");`
return mark
? `function (...args) { ${element} globalThis.__codenomadAutomationDepth = (globalThis.__codenomadAutomationDepth || 0) + 1;
try { return (() => { ${body} })(); } finally { globalThis.__codenomadAutomationDepth--; } }`
: `function (...args) { ${element} ${body} }`
}
function pageError(ref: string, description: string): Error {
const [code, detail = ""] = /codenomad:([a-z-]+)(?::(.*))?/.exec(description.split("\n", 1)[0])?.slice(1) ?? []
const quoted = JSON.stringify(detail.slice(0, VALUE_LIMIT))
switch (code) {
case "gone": return gone(ref)
case "not-select": return new Error(`${ref} is not a native select; click it to open its menu, then click the option`)
case "is-select": return new Error(`${ref} is a native select; use select with its option labels`)
case "no-option": return new Error(`No option ${quoted} in ${ref}; inspect it to see its options`)
case "option-disabled": return new Error(`Option ${quoted} of ${ref} is disabled`)
case "single": return new Error(`${ref} takes exactly one value`)
case "disabled": return new Error(`${ref} is disabled`)
case "readonly": return new Error(`${ref} is read-only`)
case "not-editable": return new Error(`${ref} is not a text field`)
case "not-text": return new Error(`${ref} is not a text field; use click`)
case "unfocusable": return new Error(`${ref} cannot take focus (hidden, disabled or behind a dialog); reveal it, then inspect again`)
case "invalid-value": return new Error(`${quoted} is not a valid value for ${ref}`)
default: return new Error(`The page rejected the action on ${ref}: ${description.split("\n", 1)[0].slice(0, VALUE_LIMIT)}`)
}
}
/** Marks trusted input (DevTools mouse/keyboard events) in the CodeNomad window. A lease, not a
* counter: if the clearing call is lost with the connection, the mark still expires. */
export async function markedInput<T>(session: PageSession, run: (dispatch: (method: string, params: Record<string, unknown>) => Promise<void>) => Promise<T>): Promise<T> {
const held = new Map<string, { method: string; params: Record<string, unknown> }>()
let point = { x: 0, y: 0 }
let attempted = false
const dispatch = async (method: string, params: Record<string, unknown>) => {
let timestamp: number | undefined
if (session.markAutomation) {
const result = await session.channel.send("Runtime.evaluate", { expression: MARK_INPUT, returnByValue: true })
const stamp = (result.result as { value?: unknown } | undefined)?.value
if (typeof stamp !== "number" || !Number.isFinite(stamp) || stamp <= 0) throw new Error("Cannot establish automation input provenance")
timestamp = stamp
}
const mouse = method === "Input.dispatchMouseEvent"
if (mouse && typeof params.x === "number" && typeof params.y === "number") point = { x: params.x, y: params.y }
const id = mouse ? `mouse:${params.button}` : `key:${params.code}`
const down = mouse ? params.type === "mousePressed" : params.type === "keyDown" || params.type === "rawKeyDown"
const up = mouse ? params.type === "mouseReleased" : params.type === "keyUp"
if (down) held.set(id, { method, params: { ...params, type: mouse ? "mouseReleased" : "keyUp", buttons: 0, modifiers: 0,
...(timestamp === undefined ? {} : { timestamp }) } })
attempted = true
await session.channel.send(method, { ...params,
...(timestamp === undefined || !["Input.dispatchMouseEvent", "Input.dispatchKeyEvent"].includes(method) ? {} : { timestamp }) })
if (up) held.delete(id)
await session.checkpoint()
}
try {
return await run(dispatch)
} catch (error) {
if (attempted && error instanceof Error && /timed out|timeout/i.test(error.message)) {
throw new Error("Input timed out: its outcome is unknown. The page was busy and the input may still be delivered; inspect before retrying.")
}
throw error
} finally {
// Failure may be after admission but before acknowledgement/checkpoint. Release
// anything potentially held, without a failed checkpoint suppressing cleanup.
for (const release of [...held.values()].reverse()) {
try {
// A frozen page can time out this write too. The release retains the
// original press's timestamp, so provenance never depends on the timeout.
if (session.markAutomation) await session.channel.send("Runtime.evaluate", { expression: MARK_INPUT, returnByValue: true }).catch(() => undefined)
const { text: _text, unmodifiedText: _unmodified, ...params } = release.params
await session.channel.send(release.method, { ...params, ...(release.method === "Input.dispatchMouseEvent" ? point : {}) })
} catch { /* Best effort, exact original channel only; never retry the action. */ }
}
if (session.markAutomation) await session.channel.send("Runtime.evaluate", { expression: UNMARK_INPUT, returnByValue: true }).catch(() => undefined)
}
}
export const INPUT_MARK_LEASE_MS = 15_000
// CDP timestamps use epoch seconds; DOM event.timeStamp uses the page's monotonic
// origin. Reading both in the page keeps delayed input within its original interval.
const MARK_INPUT = `globalThis.__codenomadAutomationInputUntil = performance.now() + ${INPUT_MARK_LEASE_MS}; Date.now() / 1000`
// Keep the interval's end for events queued before a stalled renderer recovered.
const UNMARK_INPUT = "globalThis.__codenomadAutomationInputUntil = Math.min(globalThis.__codenomadAutomationInputUntil || 0, performance.now()); true"

65
packages/server/src/developer-cdp.test.ts
View file

@ -74,14 +74,24 @@ class FakeChrome {
this.deferred.set(command.method, { socket, command })
return
}
queueMicrotask(() => socket.respond(command, this.result(command.method)))
queueMicrotask(() => socket.respond(command, command.method === "Runtime.evaluate" && String(command.params?.expression).includes("__codenomadAutomationInputUntil")
? { result: { value: 1_700_000_000 } } : this.result(command.method)))
}
/** Visible box of every element, or none; and the element hit at its centre. */
quads: number[][] = []
hit = 42
functionResult: unknown
result(method: string): Record<string, unknown> {
if (method === "Accessibility.getFullAXTree") return {
nodes: this.axNodes,
}
if (method === "DOM.getContentQuads") return { quads: this.quads }
if (method === "DOM.getNodeForLocation") return { backendNodeId: this.hit }
if (method === "Page.getLayoutMetrics") return { cssLayoutViewport: { clientWidth: 800, clientHeight: 600 } }
if (method === "Runtime.evaluate") return { result: { value: this.context } }
if (method === "Runtime.callFunctionOn") return { result: { value: this.functionResult } }
if (method === "Page.captureScreenshot") return { data: this.screenshotData }
if (method === "DOM.resolveNode") return { object: { objectId: "node-1" } }
return {}
@ -115,7 +125,7 @@ describe("DeveloperCdp", () => {
try {
const first = await client.inspect(window)
chrome.context = { ...chrome.context, sessionId: "another-session", instanceId: "another-project" }
await client.act({ ...window, kind: "click", ref: refOf(first) })
await client.act(window, { action: "click", ref: refOf(first) })
assert.equal((await client.inspect(window)).context.sessionId, "another-session")
chrome.context = { windowId: "window-1", sessionId: null, instanceId: null } as never
assert.equal((await client.inspect(window)).context.sessionId, null)
@ -175,7 +185,7 @@ describe("DeveloperCdp", () => {
assert.equal(second.target.id, "two")
assert.equal(oldSocket.closed, true)
assert.equal(chrome.sockets.length, 4)
await assert.rejects(client.act({ ...identity, kind: "click", ref: oldRef }), /stale ref/)
await assert.rejects(client.act(identity, { action: "click", ref: oldRef }), /stale ref/)
})
it("refreshes target metadata without reconnecting", async () => {
@ -191,15 +201,40 @@ describe("DeveloperCdp", () => {
assert.equal(chrome.sockets.length, 2)
})
it("activates clicks through the resolved accessibility node", async () => {
it("clicks a visible element with marked real pointer input at its visible centre", async () => {
const chrome = new FakeChrome()
chrome.quads = [[100, 40, 300, 40, 300, 80, 100, 80]]
const client = chrome.client()
const inspection = await client.inspect(identity)
chrome.commands.length = 0
await client.act({ ...identity, kind: "click", ref: refOf(inspection) })
await client.act(identity, { action: "click", ref: refOf(inspection) })
const sequence = chrome.commands.filter(command => command.method === "Input.dispatchMouseEvent"
|| (command.method === "Runtime.evaluate" && /__codenomadAutomationInputUntil/.test(String(command.params?.expression))))
.map(command => command.method === "Runtime.evaluate" ? (/Math.min/.test(String(command.params!.expression)) ? "unmark" : "mark")
: `${command.params!.type}@${command.params!.x},${command.params!.y}`)
assert.deepEqual(sequence, ["mark", "mouseMoved@200,60", "mark", "mousePressed@200,60", "mark", "mouseReleased@200,60", "unmark"],
"every trusted event in the CodeNomad window is marked, and the mark is cleared")
})
it("refuses a covered element and activates a box-less one programmatically", async () => {
const chrome = new FakeChrome()
chrome.quads = [[0, 0, 10, 0, 10, 10, 0, 10]]
chrome.hit = 7
const client = chrome.client()
const inspection = await client.inspect(identity)
chrome.commands.length = 0
chrome.functionResult = "div \"Overlay\""
await assert.rejects(client.act(identity, { action: "click", ref: refOf(inspection) }), /is covered by div "Overlay"; close or move it first/)
assert.equal(chrome.commands.some(command => command.method === "Input.dispatchMouseEvent"), false)
chrome.functionResult = undefined
chrome.quads = []
chrome.commands.length = 0
await client.act(identity, { action: "click", ref: refOf(inspection) })
assert.equal(chrome.commands.some((command) => command.method === "Input.dispatchMouseEvent"), false)
assert.equal(chrome.commands.find((command) => command.method === "Runtime.callFunctionOn")?.params?.objectId, "node-1")
assert.equal(chrome.commands.filter((command) => command.method === "Runtime.callFunctionOn").at(-1)?.params?.objectId, "node-1")
})
it("retries a failed WebSocket connection", async () => {
@ -234,7 +269,7 @@ describe("DeveloperCdp", () => {
chrome.sockets.at(-1)!.emit("Page.frameNavigated", { frame: { id: "main" } })
await assert.rejects(
client.act({ ...identity, kind: "type", ref: refOf(inspection), text: "hello", clear: true }),
client.act(identity, { action: "type", ref: refOf(inspection), text: "hello", clear: true }),
/stale ref/,
)
})
@ -246,7 +281,7 @@ describe("DeveloperCdp", () => {
chrome.sockets.at(-1)!.close()
await assert.rejects(
client.act({ ...identity, kind: "click", ref: refOf(inspection) }),
client.act(identity, { action: "click", ref: refOf(inspection) }),
/stale ref/,
)
assert.equal(chrome.sockets.length, 3)
@ -259,7 +294,7 @@ describe("DeveloperCdp", () => {
chrome.context = { ...chrome.context, sessionId: "session-2" }
await assert.rejects(
client.act({ ...identity, kind: "click", ref: refOf(inspection) }),
client.act(identity, { action: "click", ref: refOf(inspection) }),
/active session changed/,
)
assert.equal(chrome.commands.some((command) => command.method === "Input.dispatchMouseEvent"), false)
@ -273,7 +308,7 @@ describe("DeveloperCdp", () => {
const socket = chrome.sockets.at(-1)!
socket.onerror?.({})
await assert.rejects(client.act({ ...identity, kind: "click", ref: oldRef }), /stale ref/)
await assert.rejects(client.act(identity, { action: "click", ref: oldRef }), /stale ref/)
assert.equal(socket.closed, true)
})
@ -289,7 +324,7 @@ describe("DeveloperCdp", () => {
const client = chrome.client()
const inspection = await client.inspect(identity)
chrome.deferred.set("DOM.resolveNode", undefined as never)
const action = client.act({ ...identity, kind: "click", ref: refOf(inspection) })
const action = client.act(identity, { action: "click", ref: refOf(inspection) })
while (!chrome.deferred.get("DOM.resolveNode")) await new Promise<void>((resolve) => setImmediate(resolve))
const deferred = chrome.deferred.get("DOM.resolveNode")!
deferred.socket.emit("Page.frameNavigated", { frame: { id: "main" } })
@ -305,7 +340,7 @@ describe("DeveloperCdp", () => {
const inspection = await client.inspect(identity)
chrome.deferred.set("DOM.resolveNode", undefined as never)
const action = client.act({ ...identity, kind: "click", ref: refOf(inspection) })
const action = client.act(identity, { action: "click", ref: refOf(inspection) })
while (!chrome.deferred.get("DOM.resolveNode")) await new Promise<void>((resolve) => setImmediate(resolve))
chrome.context = { ...chrome.context, instanceId: "workspace-2" }
const deferred = chrome.deferred.get("DOM.resolveNode")!
@ -353,7 +388,7 @@ describe("DeveloperCdp", () => {
const save = /button "Save" \[(e\d+)\]/.exec(full.snapshot)![1]
const scoped = await client.inspect(identity, { ref: dialog })
assert.equal(scoped.snapshot, `- dialog "Rename" [${dialog}]\n - textbox "Title" [${/textbox "Title" \[(e\d+)\]/.exec(full.snapshot)![1]}] value="Old"`)
await client.act({ ...identity, kind: "click", ref: save })
await client.act(identity, { action: "click", ref: save })
assert.match((await client.inspect(identity, { query: "save" })).snapshot, new RegExp(`button "Save" \\[${save}\\]`))
assert.equal(/button "Save" \[(e\d+)\]/.exec((await client.inspect(identity)).snapshot)![1], save)
@ -363,9 +398,9 @@ describe("DeveloperCdp", () => {
chrome.axNodes[0] = { ...chrome.axNodes[0], childIds: ["toolbar"] }
await client.inspect(identity)
await assert.rejects(client.inspect(identity, { ref: dialog }), /is no longer in the page/)
await client.act({ ...identity, kind: "click", ref: dialog })
await client.act(identity, { action: "click", ref: dialog })
chrome.sockets.at(-1)!.emit("Page.frameNavigated", { frame: { id: "main" } })
await assert.rejects(client.act({ ...identity, kind: "click", ref: save }), /stale ref/)
await assert.rejects(client.act(identity, { action: "click", ref: save }), /stale ref/)
})
it("closes active target connections", async () => {

23
packages/server/src/developer-cdp.ts
View file

@ -1,5 +1,6 @@
import { WebSocket } from "undici"
import { performAction, snapshotPage, type PageActionResult, type PageSession } from "./automation/page/actions"
import { bindRefs, performAction, snapshotPage, type PageActionResult, type PageSession } from "./automation/page/actions"
import type { RefAction } from "./automation/page/parse"
import { RefTable } from "./automation/page/refs"
import { SnapshotScopeError } from "./automation/page/snapshot"
@ -58,11 +59,6 @@ export interface DeveloperCdpInspectOptions {
query?: string
}
export type DeveloperCdpAction =
| (DeveloperCdpIdentity & { kind: "click"; ref: string })
| (DeveloperCdpIdentity & { kind: "type"; ref: string; text: string; clear: boolean })
| (DeveloperCdpIdentity & { kind: "select"; ref: string; values: string[] })
export interface DeveloperCdpScreenshot {
mediaType: "image/png"
data: string
@ -165,15 +161,12 @@ export class DeveloperCdp {
})
}
async act(action: DeveloperCdpAction): Promise<PageActionResult> {
return this.exclusive(action.runId, async () => {
const state = await this.ensure(action)
const epoch = state.navigationEpoch
const backendNodeId = this.refs.resolve(action.ref, this.generation(state))
const session = this.session(state, epoch, "Page changed during action; inspect again")
return performAction(session, action.kind === "click" ? { kind: "click", ref: action.ref, backendNodeId }
: action.kind === "type" ? { kind: "type", ref: action.ref, backendNodeId, text: action.text, clear: action.clear }
: { kind: "select", ref: action.ref, backendNodeId, values: action.values })
async act(identity: DeveloperCdpIdentity, action: RefAction): Promise<PageActionResult> {
return this.exclusive(identity.runId, async () => {
const state = await this.ensure(identity)
const epoch = state.navigationEpoch, generation = this.generation(state)
const page = bindRefs(action, ref => this.refs.resolve(ref, generation))
return performAction(this.session(state, epoch, "Page changed during action; inspect again"), page)
})
}

27
packages/server/src/opencode/automation-plugin.test.ts
View file

@ -90,6 +90,33 @@ test("validates Developer Mode actions", () => {
assert.throws(() => parseDeveloperAction({ action: "click" }), /click requires ref/)
})
test("validates trusted-input actions, and parsing is idempotent across the bridge", () => {
const cases: Array<[Record<string, unknown>, unknown]> = [
[{ action: "click", ref: "e1", button: "right" }, { action: "click", ref: "e1", button: "right" }],
[{ action: "click", ref: "e1", button: "left" }, { action: "click", ref: "e1" }],
[{ action: "hover", ref: "e2" }, { action: "hover", ref: "e2" }],
[{ action: "drag", ref: "e2", to: "e3" }, { action: "drag", ref: "e2", to: "e3" }],
[{ action: "scroll", direction: "down" }, { action: "scroll", direction: "down", amount: 600 }],
[{ action: "scroll", ref: "e4", direction: "left", amount: 120 }, { action: "scroll", ref: "e4", direction: "left", amount: 120 }],
[{ action: "press", key: "Shift+Tab" }, { action: "press", key: "Shift+Tab" }],
[{ action: "press", key: "Enter", ref: "e5" }, { action: "press", key: "Enter", ref: "e5" }],
[{ action: "wait", text: "Ready" }, { action: "wait", text: "Ready", gone: false, timeout: 5_000 }],
[{ action: "wait", text: "Saving", gone: true, timeout: 0 }, { action: "wait", text: "Saving", gone: true, timeout: 0 }],
]
for (const [input, expected] of cases) {
assert.deepEqual(parseDeveloperAction(input), expected)
assert.deepEqual(parseDeveloperAction(parseDeveloperAction(input)), expected, "the backend re-parses the same flat command")
}
assert.throws(() => parseDeveloperAction({ action: "click", ref: "e1", button: "middle" }), /left or right/)
assert.throws(() => parseDeveloperAction({ action: "drag", ref: "e1" }), /drag requires to/)
assert.throws(() => parseDeveloperAction({ action: "scroll", direction: "sideways" }), /up, down, left or right/)
assert.throws(() => parseDeveloperAction({ action: "scroll", direction: "up", amount: 10_001 }), /from 1 to 10000/)
assert.throws(() => parseDeveloperAction({ action: "press", key: "" }), /press requires key/)
assert.throws(() => parseDeveloperAction({ action: "press", key: "Hyper+Q" }), /Unknown modifier/)
assert.throws(() => parseDeveloperAction({ action: "wait", text: " " }), /wait requires text/)
assert.throws(() => parseDeveloperAction({ action: "wait", text: "x", timeout: 20_000 }), /from 0 to 15000/)
})
test("an inspection reads as a plain outline with its window, collapse note and diagnostics", () => {
const text = formatInspection({
target: { title: "CodeNomad", url: "http://127.0.0.1:9899/" },

35
packages/server/src/opencode/automation-plugin.ts
View file

@ -7,6 +7,7 @@ import type { Plugin } from "@opencode/plugin"
import { MissionControlError } from "../missions/control-error"
import { signNativeRecurrenceControl, type RecurrenceControlProofBody } from "../missions/recurrence-control-proof"
import { missionRecoveryError } from "../missions/recovery-error"
import { PAGE_ACTION_PROPERTIES, PAGE_ACTIONS, parsePageAction, type RefAction } from "../automation/page/parse"
export const AUTOMATION_BRIDGE_PATH = "/api/opencode-plugin/automation"
const REQUEST_TIMEOUT_MS = 95_000
@ -26,13 +27,10 @@ let cachedWslWindowsLocalAppData: string | null | undefined
export type DeveloperAction =
| { action: "inspect"; ref?: string; query?: string }
| { action: "click"; ref: string }
| { action: "type"; ref: string; text: string; clear: boolean }
| { action: "select"; ref: string; values: string[] }
| RefAction
| { action: "restart" }
| { action: "screenshot" }
const MAX_SELECT_VALUES = 64
const MAX_QUERY_LENGTH = 200
export type BrowserAction =
@ -243,21 +241,11 @@ export function parseDeveloperAction(input: unknown): DeveloperAction {
case "restart":
case "screenshot":
return { action: value.action }
case "click":
if (typeof value.ref !== "string") throw new Error("click requires ref from the latest inspection")
return { action: "click", ref: value.ref }
case "type":
if (typeof value.ref !== "string" || typeof value.text !== "string") throw new Error("type requires ref and text")
if (value.clear !== undefined && typeof value.clear !== "boolean") throw new Error("type clear must be true or false")
return { action: "type", ref: value.ref, text: value.text, clear: value.clear !== false }
case "select":
if (typeof value.ref !== "string" || !Array.isArray(value.values) || value.values.length > MAX_SELECT_VALUES
|| !value.values.every(item => typeof item === "string")) {
throw new Error("select requires ref and values: the option labels or values to choose")
}
return { action: "select", ref: value.ref, values: value.values as string[] }
default:
throw new Error("Unsupported native automation action")
default: {
const page = parsePageAction(value, "inspection")
if (!page) throw new Error("Unsupported native automation action")
return page
}
}
}
@ -657,15 +645,12 @@ export function createDeveloperToolTransform() {
})
draft.add({
name: "act",
description: "Act on a ref from the latest codenomad.inspect: click it, type into a field (replacing its text unless clear is false), or select native <select> options by label or value. restart gracefully relaunches CodeNomad and waits for reconnection.",
description: "Act on refs from the latest codenomad.inspect like a user: click (real pointer at the element's visible centre, refused when covered), type into a field (replacing its text unless clear is false), select native <select> options by label or value, hover, drag onto another ref, scroll (a ref or the page), press a key or chord (optionally after focusing a ref), or wait for text to appear or disappear. restart gracefully relaunches CodeNomad and waits for reconnection.",
input: {
type: "object",
properties: {
action: { type: "string", enum: ["click", "type", "select", "restart"] },
ref: { type: "string", description: "Element ref from the latest inspection" },
text: { type: "string", description: "Text for type" },
clear: { type: "boolean", description: "type only: replace the field's current text (default true); false appends" },
values: { type: "array", items: { type: "string" }, description: "select only: option labels or values to choose; one for an ordinary select" },
action: { type: "string", enum: [...PAGE_ACTIONS, "restart"] },
...PAGE_ACTION_PROPERTIES,
},
required: ["action"],
additionalProperties: false,

10
packages/server/src/server/routes/automation-plugin.test.ts
View file

@ -69,11 +69,11 @@ test("targets the inspected window independently of the visible conversation", a
}
},
close: () => undefined,
act: async (identity: { sessionId?: string; instanceId?: string; kind: string; values?: string[] }) => {
act: async (identity: { sessionId?: string; instanceId?: string }, action: { action: string; values?: string[] }) => {
assert.equal(identity.sessionId, undefined)
assert.equal(identity.instanceId, undefined)
actions.push(identity)
return identity.kind === "select" ? { selected: identity.values } : {}
actions.push(action)
return action.action === "select" ? { selected: action.values } : {}
},
},
workspaceManager: {
@ -107,8 +107,10 @@ test("targets the inspected window independently of the visible conversation", a
const selected = await request({ mode: "developer-execute", sessionID: "session-1", command: { action: "select", ref: "e2", values: ["Exploration"] } })
assert.deepEqual(selected.json(), { result: { ok: true, selected: ["Exploration"] } })
await request({ mode: "developer-execute", sessionID: "session-1", command: { action: "type", ref: "e3", text: "x", clear: false } })
assert.deepEqual(actions.map(action => [action.kind, action.ref, action.clear ?? action.values]),
assert.deepEqual(actions.map(action => [action.action, action.ref, action.clear ?? action.values]),
[["click", "ax1", undefined], ["select", "e2", ["Exploration"]], ["type", "e3", false]])
assert.equal((await request({ mode: "developer-execute", sessionID: "session-1", command: { action: "press", key: "Hyper+Q" } })).statusCode, 400,
"the backend validates the flat command again")
const restart = await request({ mode: "developer-execute", sessionID: "session-1", command: { action: "restart" } })
assert.equal(restart.statusCode, 200)

5
packages/server/src/server/routes/automation-plugin.ts
View file

@ -212,10 +212,7 @@ export function registerAutomationPluginRoute(app: FastifyInstance, deps: Automa
const image = await deps.developerCdp.screenshot(identity)
return reply.send({ result: { image: { data: image.data, mime: image.mediaType } } })
}
const result = await deps.developerCdp.act(command.action === "type"
? { ...identity, kind: "type", ref: command.ref, text: command.text, clear: command.clear }
: command.action === "select" ? { ...identity, kind: "select", ref: command.ref, values: command.values }
: { ...identity, kind: "click", ref: command.ref })
const result = await deps.developerCdp.act(identity, command)
return reply.send({ result: { ok: true, ...result } })
} catch (error) {
return reply.code(409).send({ error: error instanceof Error ? error.message : String(error) })

4
packages/tauri-app/src-tauri/src/developer_mode.rs
View file

@ -123,8 +123,10 @@ impl DeveloperMode {
}
}
/// Like the update flow, automation targets an application window: a focused
/// Preferences or remote window falls back to the most recently used one.
fn focused_window_id(app: &AppHandle) -> Option<String> {
crate::local_windows::focused_local_window(app)
crate::local_windows::update_flow_window(app)
.and_then(|window| crate::identity::local_window_id(window.label()).ok())
}

7
packages/ui/src/lib/automation-input.ts Normal file
View file

@ -0,0 +1,7 @@
/** Whether CodeNomad's own window automation is acting right now (server `automation/page`):
* page functions run under a depth counter and trusted DevTools input under a short lease, so
* neither ever counts as a person's input. */
export function automationInputActive(eventTime = performance.now()): boolean {
const scope = globalThis as typeof globalThis & { __codenomadAutomationDepth?: number; __codenomadAutomationInputUntil?: number }
return Boolean(scope.__codenomadAutomationDepth) || eventTime <= (scope.__codenomadAutomationInputUntil ?? -1)
}

13
packages/ui/src/lib/form-answer-interaction.ts
View file

@ -1,4 +1,5 @@
import type { FormAnswer, FormInfo } from "@opencode/client"
import { automationInputActive } from "./automation-input"
export interface FormAnswerInteraction { readonly kind: "dock-input" }
const interactions = new WeakMap<FormAnswerInteraction, { form: string; answer: string }>()
@ -8,9 +9,17 @@ const identity = (form: FormInfo) => JSON.stringify([form.id, form.sessionID, fo
* too). This is first-party origin correlation, not physical-person attestation. */
export function createFormAnswerInteraction(form: () => FormInfo, answer: () => FormAnswer) {
let pending: FormAnswerInteraction | undefined
let automatedTurn = false
const capture = (event: MouseEvent | KeyboardEvent) => {
pending = undefined
if (!event.isTrusted || (globalThis as typeof globalThis & { __codenomadAutomationDepth?: number }).__codenomadAutomationDepth) return
if (!event.isTrusted) return
if (automatedTurn || automationInputActive(event.timeStamp)) {
// A delayed Enter/Space can synthesize a default-button click with a newer
// timestamp in the same event turn. It inherits the key's provenance.
automatedTurn = true
setTimeout(() => { automatedTurn = false }, 0)
return
}
const element = event.target
if (!(element instanceof HTMLElement)) return
const owner = event.currentTarget
@ -43,7 +52,7 @@ export function createFormAnswerInteraction(form: () => FormInfo, answer: () =>
/** One-use, exact request/schema/answer match at the actual SDK send boundary. */
export function consumeFormAnswerInteraction(token: FormAnswerInteraction | undefined, form: FormInfo, answer: FormAnswer): boolean {
if (!token || (globalThis as typeof globalThis & { __codenomadAutomationDepth?: number }).__codenomadAutomationDepth) return false
if (!token || automationInputActive()) return false
const captured = interactions.get(token)
interactions.delete(token)
return captured?.form === identity(form) && captured.answer === JSON.stringify(answer)

117
packages/ui/tests/browser/automation-forms.test.ts
View file

@ -1,22 +1,19 @@
import assert from "node:assert/strict"
import { after, before, test } from "node:test"
import { mkdtemp, readFile, rm } from "node:fs/promises"
import os from "node:os"
import path from "node:path"
import { fileURLToPath } from "node:url"
import { chromium, type BrowserContext, type Page } from "playwright"
import type { Page } from "playwright"
import { createServer, type ViteDevServer } from "vite"
import solid from "vite-plugin-solid"
import type { MissionMap } from "../../../server/src/api-types"
import { DeveloperCdp, type DeveloperCdpIdentity } from "../../../server/src/developer-cdp"
import { recurrenceSnapshotSchema } from "../../../server/src/missions/recurrence-control-contract"
import { launchAutomationBrowser, openAutomationPage } from "./automation-session"
import { createFixtureCache } from "./fixture-cache"
import { createFixtureShutdown } from "./fixture-shutdown"
import type {} from "./fixtures/mission-editor-lifetime"
// The real codenomad.inspect/act controller drives real pages over Chromium's DevTools
// endpoint, exactly as it drives the CodeNomad window (#1008).
let server: ViteDevServer, context: BrowserContext, profile: string, endpoint: string, base: string, windows = 0
let server: ViteDevServer, automation: Awaited<ReturnType<typeof launchAutomationBrowser>>, base: string
const FORM = `<!doctype html><html lang="en"><body><main>
<label>Title <input id="title" value="Prefilled"></label>
@ -52,6 +49,7 @@ const EDGES = `<!doctype html><html lang="en"><body><main>
for (const element of document.querySelectorAll("input")) for (const type of ["input", "change"]) element.addEventListener(type, () => window.events.push(type + ":" + element.id))
</script></body></html>`
before(async () => {
const cache = await createFixtureCache(), shutdown = createFixtureShutdown(cache)
server = await createServer({ configFile: false, root: fileURLToPath(new URL("../..", import.meta.url)), logLevel: "error", cacheDir: cache.cacheDir,
@ -66,44 +64,15 @@ before(async () => {
shutdown.own(server)
await server.listen()
base = `http://127.0.0.1:${(server.httpServer!.address() as { port: number }).port}`
profile = await mkdtemp(path.join(os.tmpdir(), "codenomad-automation-forms-"))
context = await chromium.launchPersistentContext(profile, { executablePath: process.env.CODENOMAD_BROWSER_PATH || undefined, locale: "en-US",
args: ["--remote-debugging-address=127.0.0.1", "--remote-debugging-port=0"] })
let port = ""
for (let attempt = 0; attempt < 100 && !port; attempt++) {
port = (await readFile(path.join(profile, "DevToolsActivePort"), "utf8").catch(() => "")).split(/\r?\n/, 1)[0]
if (!port) await new Promise(resolve => setTimeout(resolve, 50))
}
assert.ok(port, "Chromium published its DevTools port")
endpoint = `http://127.0.0.1:${port}`
automation = await launchAutomationBrowser()
})
after(async () => {
await context?.close()
await automation?.close()
await server?.close()
if (profile) await rm(profile, { recursive: true, force: true })
})
/** A page that identifies as one CodeNomad window, and the controller pinned to it. */
async function open(pathname: string, setup?: (page: Page) => Promise<void>) {
const windowId = `automation-window-${++windows}`
const page = await context.newPage()
page.setDefaultTimeout(15_000)
await page.addInitScript(`window.__CODENOMAD_WINDOW_ID__ = ${JSON.stringify(windowId)}`)
await setup?.(page)
await page.goto(`${base}${pathname}`)
const cdp = new DeveloperCdp({ timeoutMs: 10_000 })
const identity: DeveloperCdpIdentity = { endpoint, runId: `run-${windowId}`, windowId }
const refFor = async (pattern: RegExp, query?: string) => {
const { snapshot } = await cdp.inspect(identity, query ? { query } : {})
const match = new RegExp(`${pattern.source}[^\\n]*?\\[(e\\d+)\\]`).exec(snapshot)
assert.ok(match, `${pattern} in:\n${snapshot}`)
return match[1]
}
const act = (action: { kind: "click"; ref: string } | { kind: "type"; ref: string; text: string; clear?: boolean } | { kind: "select"; ref: string; values: string[] }) =>
cdp.act(action.kind === "type" ? { ...identity, ...action, clear: action.clear !== false } : { ...identity, ...action })
return { page, cdp, identity, refFor, act, close: async () => { cdp.close(); await page.close() } }
}
const open = (pathname: string, setup?: (page: Page) => Promise<void>) => openAutomationPage(automation, `${base}${pathname}`, setup)
test("typing replaces a prefilled field by default, clear:false appends, and selects fire input/change like a user", async () => {
const run = await open("/automation-form")
@ -114,32 +83,32 @@ test("typing replaces a prefilled field by default, clear:false appends, and sel
assert.match(outline, /combobox "Scenario" \[e\d+\] value="Flexible mission" collapsed\n {4}- option "Flexible mission" \[e\d+\] selected\n {4}- option "Exploration" \[e\d+\]\n {4}- option "Debugging" \[e\d+\] disabled/)
const title = await refFor(/textbox "Title"/)
assert.deepEqual(await act({ kind: "type", ref: title, text: "Mission title" }), { value: "Mission title" })
assert.deepEqual(await act({ kind: "type", ref: title, text: " suffix", clear: false }), { value: "Mission title suffix" })
assert.deepEqual(await act({ action: "type", ref: title, text: "Mission title" }), { value: "Mission title" })
assert.deepEqual(await act({ action: "type", ref: title, text: " suffix", clear: false }), { value: "Mission title suffix" })
assert.equal(await page.inputValue("#title"), "Mission title suffix")
assert.deepEqual(await act({ kind: "type", ref: await refFor(/textbox "Notes"/), text: "" }), { value: "" })
assert.deepEqual(await act({ kind: "type", ref: await refFor(/textbox "Rich"/), text: "New rich" }), { value: "New rich" })
assert.deepEqual(await act({ action: "type", ref: await refFor(/textbox "Notes"/), text: "" }), { value: "" })
assert.deepEqual(await act({ action: "type", ref: await refFor(/textbox "Rich"/), text: "New rich" }), { value: "New rich" })
const due = await refFor(/Date "Due"/)
assert.deepEqual(await act({ kind: "type", ref: due, text: "2026-10-19" }), { value: "2026-10-19" })
assert.deepEqual(await act({ action: "type", ref: due, text: "2026-10-19" }), { value: "2026-10-19" })
assert.match((await run.cdp.inspect(run.identity, { ref: due })).snapshot, /^- Date "Due" \[e\d+\][^\n]*$/, "a date field is one line, without its inner spinbuttons")
const scenario = await refFor(/combobox "Scenario"/)
await page.evaluate(() => { (window as any).events.length = 0 })
assert.deepEqual(await act({ kind: "select", ref: scenario, values: ["Exploration"] }), { selected: ["Exploration"] })
assert.deepEqual(await act({ action: "select", ref: scenario, values: ["Exploration"] }), { selected: ["Exploration"] })
assert.equal(await page.inputValue("#scenario"), "wayfinder")
assert.deepEqual(await page.evaluate(() => (window as any).events), ["input:scenario:1", "change:scenario:1"],
"framework handlers see input then change, marked as automation")
assert.equal(await page.evaluate(() => (globalThis as any).__codenomadAutomationDepth), 0)
assert.deepEqual(await act({ kind: "select", ref: scenario, values: ["custom"] }), { selected: ["Flexible mission"] }, "values match too")
await assert.rejects(act({ kind: "select", ref: scenario, values: ["Debugging"] }), /Option "Debugging" of e\d+ is disabled/)
await assert.rejects(act({ kind: "select", ref: scenario, values: ["Nope"] }), /No option "Nope" in e\d+/)
await assert.rejects(act({ kind: "select", ref: scenario, values: [] }), /takes exactly one value/)
await assert.rejects(act({ kind: "type", ref: scenario, text: "x" }), /is not a text field/)
await assert.rejects(act({ kind: "type", ref: await refFor(/checkbox "Agree"/), text: "x" }), /is not a text field; use click/)
await assert.rejects(act({ kind: "select", ref: title, values: ["x"] }), /is not a native select/)
assert.deepEqual(await act({ action: "select", ref: scenario, values: ["custom"] }), { selected: ["Flexible mission"] }, "values match too")
await assert.rejects(act({ action: "select", ref: scenario, values: ["Debugging"] }), /Option "Debugging" of e\d+ is disabled/)
await assert.rejects(act({ action: "select", ref: scenario, values: ["Nope"] }), /No option "Nope" in e\d+/)
await assert.rejects(act({ action: "select", ref: scenario, values: [] }), /takes exactly one value/)
await assert.rejects(act({ action: "type", ref: scenario, text: "x" }), /is not a text field/)
await assert.rejects(act({ action: "type", ref: await refFor(/checkbox "Agree"/), text: "x" }), /is not a text field; use click/)
await assert.rejects(act({ action: "select", ref: title, values: ["x"] }), /is not a native select/)
// An option ref is clickable: it picks that option, since the native popup is not in the page.
assert.deepEqual(await act({ kind: "click", ref: await refFor(/option "Exploration"/, "Scenario") }), { selected: ["Exploration"] })
assert.deepEqual(await act({ action: "click", ref: await refFor(/option "Exploration"/, "Scenario") }), { selected: ["Exploration"] })
assert.match((await run.cdp.inspect(run.identity, { query: "scenario" })).snapshot, /combobox "Scenario" \[e\d+\] value="Exploration"/)
} finally { await run.close() }
})
@ -159,40 +128,40 @@ test("typing never lands in another field, and normalized inputs keep the browse
await page.evaluate(() => { document.querySelector("details")!.open = true })
const hidden = await refFor(/textbox "Hidden"/)
await page.evaluate(() => { document.querySelector("details")!.open = false })
await assert.rejects(act({ kind: "type", ref: hidden, text: "Lost" }), /cannot take focus/)
await assert.rejects(act({ kind: "type", ref: await refFor(/textbox "Locked"/), text: "Lost" }), /is disabled/)
await assert.rejects(act({ action: "type", ref: hidden, text: "Lost" }), /cannot take focus/)
await assert.rejects(act({ action: "type", ref: await refFor(/textbox "Locked"/), text: "Lost" }), /is disabled/)
assert.equal(await page.evaluate(() => document.activeElement?.id), "outside")
assert.equal(await page.inputValue("#outside"), "Kept", "nothing was typed into the focused field instead")
const modalOutside = await refFor(/textbox "Outside"/)
await page.evaluate(() => (document.getElementById("modal") as HTMLDialogElement).showModal())
await assert.rejects(act({ kind: "type", ref: modalOutside, text: "Lost" }), /cannot take focus/)
await assert.rejects(act({ action: "type", ref: modalOutside, text: "Lost" }), /cannot take focus/)
assert.equal(await page.inputValue("#modal-field"), "", "the dialog's field received nothing")
await page.evaluate(() => (document.getElementById("modal") as HTMLDialogElement).close())
assert.deepEqual(await act({ kind: "type", ref: await refFor(/textbox "Mail"/), text: "c", clear: false }), { value: "abc" })
assert.deepEqual(await act({ kind: "type", ref: await refFor(/spinbutton "Count"/), text: "3", clear: false }), { value: "123" })
assert.deepEqual(await act({ kind: "type", ref: await refFor(/textbox "Mail"/), text: "x@y.z" }), { value: "x@y.z" }, "clearing works there too")
assert.deepEqual(await act({ action: "type", ref: await refFor(/textbox "Mail"/), text: "c", clear: false }), { value: "abc" })
assert.deepEqual(await act({ action: "type", ref: await refFor(/spinbutton "Count"/), text: "3", clear: false }), { value: "123" })
assert.deepEqual(await act({ action: "type", ref: await refFor(/textbox "Mail"/), text: "x@y.z" }), { value: "x@y.z" }, "clearing works there too")
// Through its label's text, which designates the labelled control.
assert.deepEqual(await act({ kind: "type", ref: await refFor(/text "Tint"/), text: "#FF0000" }), { value: "#ff0000" })
assert.deepEqual(await act({ action: "type", ref: await refFor(/text "Tint"/), text: "#FF0000" }), { value: "#ff0000" })
const volume = await refFor(/slider "Volume"/)
assert.deepEqual(await act({ kind: "type", ref: volume, text: "55" }), { value: "60" }, "ranges snap to their step")
assert.deepEqual(await act({ action: "type", ref: volume, text: "55" }), { value: "60" }, "ranges snap to their step")
await page.evaluate(() => { (window as any).events.length = 0 })
for (const text of ["0x50", " 70", ""]) {
await assert.rejects(act({ kind: "type", ref: volume, text }), /is not a valid value/, `range ${JSON.stringify(text)}`)
await assert.rejects(act({ action: "type", ref: volume, text }), /is not a valid value/, `range ${JSON.stringify(text)}`)
}
await assert.rejects(act({ kind: "type", ref: await refFor(/text "Tint"/), text: "" }), /is not a valid value/, "a color has no empty value")
assert.deepEqual(await act({ kind: "type", ref: await refFor(/textbox "Mail"/), text: "", clear: false }), { value: "x@y.z" })
await assert.rejects(act({ action: "type", ref: await refFor(/text "Tint"/), text: "" }), /is not a valid value/, "a color has no empty value")
assert.deepEqual(await act({ action: "type", ref: await refFor(/textbox "Mail"/), text: "", clear: false }), { value: "x@y.z" })
assert.deepEqual([await page.inputValue("#volume"), await page.inputValue("#tint")], ["60", "#ff0000"])
assert.deepEqual(await page.evaluate(() => (window as any).events), [], "refused or unchanged values fire nothing")
await assert.rejects(act({ kind: "type", ref: await refFor(/Date "Day"/), text: "2026-13-45" }), /"2026-13-45" is not a valid value/)
await assert.rejects(act({ action: "type", ref: await refFor(/Date "Day"/), text: "2026-13-45" }), /"2026-13-45" is not a valid value/)
assert.equal(await page.inputValue("#day"), "2026-01-02", "an invalid date changes nothing")
assert.deepEqual(await page.evaluate(() => (window as any).events), [])
const second = /option "Same" \[(e\d+)\][^\n]*\n\s*- option "Same" \[(e\d+)\]/.exec((await run.cdp.inspect(run.identity, { query: "Twins" })).snapshot)![2]
await act({ kind: "click", ref: second })
await act({ action: "click", ref: second })
assert.equal(await page.inputValue("#twins"), "s2", "an option click picks that exact option, not the first with its label")
assert.deepEqual(await act({ kind: "type", ref: await refFor(/textbox "Story"/), text: " end", clear: false }), { value: "one\ntwo end" })
assert.deepEqual(await act({ action: "type", ref: await refFor(/textbox "Story"/), text: " end", clear: false }), { value: "one\ntwo end" })
} finally { await run.close() }
})
@ -222,17 +191,17 @@ test("an agent fills the real Mission editor: the title is replaced, not concate
})
try {
const { page, act, refFor } = run
await act({ kind: "click", ref: await refFor(/button "Create mission"/, "Create mission") })
await act({ kind: "type", ref: await refFor(/textbox "What should the mission do\?"/, "What should the mission do?"), text: "Chart the module seams" })
await act({ action: "click", ref: await refFor(/button "Create mission"/, "Create mission") })
await act({ action: "type", ref: await refFor(/textbox "What should the mission do\?"/, "What should the mission do?"), text: "Chart the module seams" })
const title = await refFor(/textbox "Title"/, "Title")
await act({ kind: "type", ref: title, text: "First draft" })
assert.deepEqual(await act({ kind: "type", ref: title, text: "Seam survey" }), { value: "Seam survey" }, "a second type replaces the first")
assert.deepEqual(await act({ kind: "type", ref: title, text: " (pilot)", clear: false }), { value: "Seam survey (pilot)" })
await act({ kind: "click", ref: await refFor(/DisclosureTriangle "Options"/, "Options") })
assert.deepEqual(await act({ kind: "select", ref: await refFor(/combobox "Playbook"/, "Playbook"), values: ["Exploration"] }), { selected: ["Exploration"] })
await act({ action: "type", ref: title, text: "First draft" })
assert.deepEqual(await act({ action: "type", ref: title, text: "Seam survey" }), { value: "Seam survey" }, "a second type replaces the first")
assert.deepEqual(await act({ action: "type", ref: title, text: " (pilot)", clear: false }), { value: "Seam survey (pilot)" })
await act({ action: "click", ref: await refFor(/DisclosureTriangle "Options"/, "Options") })
assert.deepEqual(await act({ action: "select", ref: await refFor(/combobox "Playbook"/, "Playbook"), values: ["Exploration"] }), { selected: ["Exploration"] })
assert.match((await run.cdp.inspect(run.identity, { query: "Playbook" })).snapshot, /combobox "Playbook" \[e\d+\] value="Exploration"/)
await page.waitForFunction(() => !(document.querySelector('form.mission-editor button[type="submit"]') as HTMLButtonElement)?.disabled)
await act({ kind: "click", ref: await refFor(/button "Create"(?= \[)/, "Create") })
await act({ action: "click", ref: await refFor(/button "Create"(?= \[)/, "Create") })
await page.waitForFunction(() => document.querySelector("form.mission-editor") === null)
assert.equal(creates.length, 1)
assert.deepEqual([creates[0].title, creates[0].objective, creates[0].template], ["Seam survey (pilot)", "Chart the module seams", "wayfinder"])

186
packages/ui/tests/browser/automation-input.test.ts Normal file
View file

@ -0,0 +1,186 @@
import assert from "node:assert/strict"
import { after, before, test } from "node:test"
import { fileURLToPath } from "node:url"
import { createServer, type ViteDevServer } from "vite"
import solid from "vite-plugin-solid"
import { launchAutomationBrowser, openAutomationPage } from "./automation-session"
import { createFixtureCache } from "./fixture-cache"
import { createFixtureShutdown } from "./fixture-shutdown"
import { prepareInterruptionDock } from "./fixtures/interruption-dock-preparation"
// Trusted pointer, wheel and keyboard input from the real codenomad.act controller.
let server: ViteDevServer, automation: Awaited<ReturnType<typeof launchAutomationBrowser>>, base: string
const INPUTS = `<!doctype html><html lang="en"><body style="margin:0"><main>
<form id="search"><label>Query <input id="query"></label><button>Go</button></form>
<button id="hover-target">Hover me</button><output id="hover-state">idle</output>
<button id="menu-trigger" aria-haspopup="menu">Open menu</button><div id="menu" role="menu" hidden><button role="menuitem" id="item">Menu item</button></div>
<div id="scroller" role="region" aria-label="Scroller" style="height:100px;overflow:auto"><div style="height:2000px">tall</div></div>
<div style="position:relative;height:200px">
<div id="handle" role="button" aria-label="Handle" style="position:absolute;left:20px;top:20px;width:40px;height:40px;background:#888"></div>
<div id="zone" role="region" aria-label="Drop zone" style="position:absolute;left:300px;top:20px;width:100px;height:100px;border:1px solid"></div>
</div>
<div style="position:relative"><button id="under">Under</button>
<div id="cover" style="position:absolute;inset:0;background:#fff8" aria-label="Cover"></div></div>
<p id="later"></p>
</main><script>
const marked = () => Boolean(globalThis.__codenomadAutomationDepth) || (globalThis.__codenomadAutomationInputUntil || 0) >= performance.now()
window.log = []
for (const type of ["keydown", "pointerdown", "click"]) document.addEventListener(type, event => {
window.log.push(type + ":" + (event.target.id || event.target.tagName) + ":" + (event.key || "") + ":" + event.isTrusted + ":" + marked())
}, true)
document.getElementById("search").addEventListener("submit", event => { event.preventDefault(); window.log.push("submit:" + query.value) })
document.getElementById("hover-target").addEventListener("pointerenter", () => { document.getElementById("hover-state").textContent = "hovered" })
// Opens on pointerdown only, like Kobalte menus: a programmatic click never opens it.
document.getElementById("menu-trigger").addEventListener("pointerdown", () => { document.getElementById("menu").hidden = false })
document.getElementById("item").addEventListener("click", () => window.log.push("menu-item"))
let dragging = false
const handle = document.getElementById("handle"), zone = document.getElementById("zone")
handle.addEventListener("pointerdown", event => { dragging = true; handle.setPointerCapture(event.pointerId) })
handle.addEventListener("pointermove", event => { if (dragging) { handle.style.left = (event.clientX - 20) + "px"; handle.style.top = (event.clientY - handle.offsetParent.getBoundingClientRect().top - 20) + "px" } })
handle.addEventListener("pointerup", event => {
dragging = false
const box = zone.getBoundingClientRect()
window.dropped = event.clientX >= box.left && event.clientX <= box.right && event.clientY >= box.top && event.clientY <= box.bottom
})
setTimeout(() => { document.getElementById("later").textContent = "Ready now" }, 600)
</script></body></html>`
before(async () => {
const cache = await createFixtureCache(), shutdown = createFixtureShutdown(cache)
server = await createServer({ configFile: false, root: fileURLToPath(new URL("../..", import.meta.url)), logLevel: "error", cacheDir: cache.cacheDir,
plugins: [solid(), shutdown.plugin, { name: "automation-input", configureServer(s) {
s.middlewares.use("/automation-inputs", (_req, res) => { res.setHeader("Content-Type", "text/html"); res.end(INPUTS) })
s.middlewares.use("/automation-dock", async (_req, res) => {
res.setHeader("Content-Type", "text/html")
res.end(await s.transformIndexHtml("/automation-dock", '<html><body><div id="root"></div><script type="module" src="/tests/browser/fixtures/interruption-dock.tsx"></script></body></html>'))
})
} }], resolve: { dedupe: ["solid-js"] }, optimizeDeps: { exclude: ["lucide-solid"] }, server: { host: "127.0.0.1", port: 0, hmr: false, watch: null } })
shutdown.own(server)
await server.listen()
await prepareInterruptionDock(server)
base = `http://127.0.0.1:${(server.httpServer!.address() as { port: number }).port}`
automation = await launchAutomationBrowser()
})
after(async () => {
await automation?.close()
await server?.close()
})
const log = (page: import("playwright").Page) => page.evaluate(() => (window as any).log.splice(0) as string[])
test("keys, hover, wheel, drag and pointer clicks behave like a user and are always marked as automation", async () => {
const run = await openAutomationPage(automation, `${base}/automation-inputs`)
try {
const { page, act, refFor } = run
const query = await refFor(/textbox "Query"/)
await act({ action: "type", ref: query, text: "hello" })
await log(page)
assert.deepEqual(await act({ action: "press", key: "Enter", ref: query }), { focused: "input \"Query\"" })
assert.deepEqual(await log(page), ["keydown:query:Enter:true:true", "click:BUTTON::true:true", "submit:hello"],
"a trusted Enter submits the form through its default button; the key and the implicit click are both marked")
assert.deepEqual(await act({ action: "press", key: "Tab" }), { focused: "button \"Go\"" }, "Tab moves focus like a keyboard")
assert.equal(await page.evaluate(() => (globalThis as any).__codenomadAutomationInputUntil <= performance.now()), true,
"the mark ends after each action but retains its timestamp for queued events")
await act({ action: "hover", ref: await refFor(/button "Hover me"/) })
assert.equal(await page.textContent("#hover-state"), "hovered")
await log(page)
await act({ action: "click", ref: await refFor(/button "Open menu"/) })
assert.deepEqual((await log(page)).filter(entry => entry.startsWith("pointerdown")), ["pointerdown:menu-trigger::true:true"])
await act({ action: "click", ref: await refFor(/menuitem "Menu item"/, "Menu item") })
assert.ok((await log(page)).includes("menu-item"), "a pointerdown-only menu opens and its item is clickable")
await act({ action: "scroll", ref: await refFor(/region "Scroller"/), direction: "down", amount: 500 })
await page.waitForFunction(() => document.getElementById("scroller")!.scrollTop > 0)
await act({ action: "drag", ref: await refFor(/button "Handle"/), to: await refFor(/region "Drop zone"/) })
assert.equal(await page.evaluate(() => (window as any).dropped), true)
await assert.rejects(act({ action: "click", ref: await refFor(/button "Under"/) }), /is covered by div "Cover"/)
assert.deepEqual((await log(page)).filter(entry => entry.includes("under")), [], "a covered element receives nothing")
assert.equal((await act({ action: "wait", text: "Ready now", timeout: 5_000 })).found, true)
assert.equal((await act({ action: "wait", text: "Hover me", gone: true, timeout: 300 }).catch(error => error.message)), "\"Hover me\" is still shown after 300 ms")
await assert.rejects(act({ action: "press", key: "Hyper+Q" }), /Unknown modifier "Hyper"/)
} finally { await run.close() }
})
test("a hidden press target never submits the focused draft, and dragging cannot scroll under its start point", async () => {
const run = await openAutomationPage(automation, `${base}/automation-inputs`)
try {
const { page, act, refFor } = run
await page.evaluate(() => {
const details = document.createElement("details")
details.open = true
details.innerHTML = '<summary>Target</summary><input aria-label="Hidden target">'
document.body.append(details)
})
const hidden = await refFor(/textbox "Hidden target"/)
await page.evaluate(() => { document.querySelector("details")!.open = false })
await page.focus("#query")
await log(page)
await assert.rejects(act({ action: "press", ref: hidden, key: "Enter" }), /cannot take focus/)
assert.deepEqual(await log(page), [], "no key or implicit submit reached the focused field")
await page.evaluate(() => {
const list = document.createElement("section")
list.innerHTML = Array.from({ length: 35 }, (_, index) => `<button style="display:block;height:65px" aria-label="Row ${index + 1}">${index + 1}</button>`).join("")
document.body.append(list)
})
const first = await refFor(/button "Row 1"/), last = await refFor(/button "Row 35"/)
await page.getByRole("button", { name: "Row 1", exact: true }).scrollIntoViewIfNeeded()
const before = await page.evaluate(() => scrollY)
await log(page)
await assert.rejects(act({ action: "drag", ref: first, to: last }), /no visible box to drag to/)
assert.equal(await page.evaluate(() => scrollY), before, "dragging never scrolls the page under a saved start point")
assert.deepEqual(await log(page), [], "no row was pressed")
} finally { await run.close() }
})
/** Answers the real InterruptionDock question and returns the human-answer header it sent. */
async function answerDock(via: "click" | "press" | "press-stalled" | "click-stalled" | "human"): Promise<unknown> {
const run = await openAutomationPage(automation, `${base}/automation-dock`, async page => {
await page.route("**/api/**", route => route.fulfill({ contentType: "application/json", body: "{}" }))
})
try {
const { page, act, refFor } = run
await page.waitForFunction(() => Boolean((window as any).fixture?.snapshot().ids.length))
await page.evaluate(() => (window as any).fixture.ask())
const field = page.locator('.interruption-dock input[type="text"]:visible')
await field.waitFor()
if (via === "human") {
await field.fill("Human answer")
await page.locator('.interruption-dock button[type="submit"]:visible').click()
} else {
const ref = await refFor(/textbox/, "textbox")
await act({ action: "type", ref, text: `Automation ${via}` })
if (via === "press-stalled" || via === "click-stalled") {
await page.evaluate(({ event, duration }) => window.addEventListener(event, () => {
const end = performance.now() + duration
while (performance.now() < end) { /* Renderer stalled beyond the 15-second input lease. */ }
}, { capture: true, once: true }), { event: via === "press-stalled" ? "keydown" : "pointerdown", duration: via === "press-stalled" ? 16_000 : 25_000 })
const input = via === "press-stalled" ? { action: "press", key: "Enter", ref }
: { action: "click", ref: await refFor(/button "Submit"/, "Submit") }
await act(input).catch(error => {
if (!/timed out/.test(error.message)) throw error
})
} else if (via === "press") await act({ action: "press", key: "Enter", ref })
else await act({ action: "click", ref: await refFor(/button "Submit"/, "Submit") })
}
await page.waitForFunction(() => (window as any).fixture.replies.length === 1)
const sent = await page.evaluate(() => ({ answer: (window as any).fixture.replies[0].answer, options: (window as any).fixture.replyOptions[0] }))
assert.deepEqual(sent.answer, { q0: via === "human" ? "Human answer" : `Automation ${via}` })
return sent.options?.headers?.["x-codenomad-human-answer"]
} finally { await run.close() }
}
test("a dock answer given through trusted automation clicks or keys never carries the human-answer mark", async () => {
assert.equal(await answerDock("click"), undefined, "a trusted automation click is not a human answer")
assert.equal(await answerDock("press"), undefined, "a trusted automation Enter is not a human answer")
assert.equal(await answerDock("press-stalled"), undefined, "a key queued past lease expiry and its default click still are not human")
assert.equal(await answerDock("click-stalled"), undefined, "a cleanup release after two 10-second command timeouts retains its original provenance")
assert.equal(await answerDock("human"), "1", "control: a real user click on the same dock is")
})

45
packages/ui/tests/browser/automation-session.ts Normal file
View file

@ -0,0 +1,45 @@
import assert from "node:assert/strict"
import { mkdtemp, readFile, rm } from "node:fs/promises"
import os from "node:os"
import path from "node:path"
import { chromium, type BrowserContext, type Page } from "playwright"
import { DeveloperCdp, type DeveloperCdpIdentity } from "../../../server/src/developer-cdp"
import { parsePageAction } from "../../../server/src/automation/page/parse"
/** A real Chromium with its DevTools endpoint, driven by the real codenomad.inspect/act
* controller exactly as it drives the CodeNomad window. */
export async function launchAutomationBrowser() {
const profile = await mkdtemp(path.join(os.tmpdir(), "codenomad-automation-"))
const context = await chromium.launchPersistentContext(profile, { executablePath: process.env.CODENOMAD_BROWSER_PATH || undefined, locale: "en-US",
args: ["--remote-debugging-address=127.0.0.1", "--remote-debugging-port=0"] })
let port = ""
for (let attempt = 0; attempt < 100 && !port; attempt++) {
port = (await readFile(path.join(profile, "DevToolsActivePort"), "utf8").catch(() => "")).split(/\r?\n/, 1)[0]
if (!port) await new Promise(resolve => setTimeout(resolve, 50))
}
assert.ok(port, "Chromium published its DevTools port")
return { context, endpoint: `http://127.0.0.1:${port}`, close: async () => { await context.close(); await rm(profile, { recursive: true, force: true }) } }
}
let windows = 0
/** A page that identifies as one CodeNomad window, and the controller pinned to it. */
export async function openAutomationPage(browser: { context: BrowserContext; endpoint: string }, url: string, setup?: (page: Page) => Promise<void>) {
const windowId = `automation-window-${++windows}`
const page = await browser.context.newPage()
page.setDefaultTimeout(15_000)
await page.addInitScript(`window.__CODENOMAD_WINDOW_ID__ = ${JSON.stringify(windowId)}`)
await setup?.(page)
await page.goto(url)
const cdp = new DeveloperCdp({ timeoutMs: 10_000 })
const identity: DeveloperCdpIdentity = { endpoint: browser.endpoint, runId: `run-${windowId}`, windowId }
const refFor = async (pattern: RegExp, query?: string) => {
const { snapshot } = await cdp.inspect(identity, query ? { query } : {})
const match = new RegExp(`${pattern.source}[^\\n]*?\\[(e\\d+)\\]`).exec(snapshot)
assert.ok(match, `${pattern} in:\n${snapshot}`)
return match[1]
}
/** Tool-shaped input, validated by the same parser as codenomad.act. */
const act = async (input: Record<string, unknown>) => cdp.act(identity, parsePageAction(input, "inspection")!)
return { page, cdp, identity, refFor, act, close: async () => { cdp.close(); await page.close() } }
}