qwen-code/packages/cua-driver/rust/tests/integration/driver_client.py
顾盼 adc2bce414
feat(cua-driver): vendor qwen-cua-driver with opt-in 0–1000 relative coordinates (#5896)
* feat(cua-driver): vendor trycua/cua driver with 1000-normalized coordinate support

Vendor libs/cua-driver from trycua/cua into packages/cua-driver as the
basis for qwen-code's computer-use backend, adding an opt-in relative
(1000x1000 normalized) coordinate mode for Qwen-VL clients.

- coord_norm.rs: 0-1000 <-> pixel conversion, per-(pid,window_id) size
  cache, tools/list description rewrite (TDD, 27 tests)
- ToolRegistry: normalized field + invoke input/output hooks
- protocol.rs: system-instruction coordinate wording switched by mode
- serve.rs: daemon list path description rewrite (input_schema aware)
- main.rs: CUA_DRIVER_RS_COORDINATE_SPACE env seed

Default coordinate_space=pixels => zero behavior change for existing
pixel clients. Set CUA_DRIVER_RS_COORDINATE_SPACE=normalized_1000 to
enable. Excludes rust/target build output.

* feat(cua-driver): make normalized coordinate scale configurable

Add CUA_DRIVER_RS_COORDINATE_SCALE (default 1000) so the normalization
full-scale can absorb the Qwen 999-vs-1000 cookbook ambiguity without a
recompile. norm_to_px/px_to_norm now take an explicit scale; denormalize_args
reads the process-wide COORDINATE_SCALE seeded once at startup from env.

* ci(cua-driver): add cross-platform release workflow for vendored driver

Standalone GitHub Action that builds, signs, and releases the vendored
cua-driver under packages/cua-driver. Adapted from upstream trycua/cua
cd-rust-cua-driver.yml:

- macOS: universal binary (lipo arm64+x86_64), codesigned + notarized into
  CuaDriver.app using qwen-code's existing secrets (MAC_CSC_LINK cert +
  App Store Connect API key notarization); Developer ID identity is
  auto-discovered from the imported cert.
- Linux: x86_64 + arm64, built in debian:11 for a glibc 2.31 floor.
- Windows: x86_64 + arm64, unsigned (no EV cert, matches upstream).
- Release: softprops/action-gh-release on cua-driver-rs-v* tags or manual
  dispatch, prerelease.

Triggered by tag push (cua-driver-rs-v*) or workflow_dispatch.

* chore(cua-driver): rebrand vendored driver as qwen-cua-driver

Rename the vendored trycua/cua driver so the fork installs and runs
independently of any upstream trycua install:
- binary cua-driver -> qwen-cua-driver
- bundle CuaDriver.app -> QwenCuaDriver.app
- bundle id com.trycua.driver -> com.qwencode.cua-driver

Updates the cargo/uia manifests, Info.plist, bundle/proxy launch paths,
permission/health-report wording, the install/build scripts, and the
cross-platform release workflow.

* feat(cua-driver): finish relative-coordinate mode — toggle, scale, zoom/move_cursor

- CUA_DRIVER_RS_COORDINATE_SPACE is now a 1/0 toggle (via is_env_truthy);
  default off keeps pixel mode byte-identical to upstream.
- Thread CUA_DRIVER_RS_COORDINATE_SCALE through every coordinate surface
  (was hardcoded 1000): input denormalization already used it; now the
  rewritten screenshot dims, the tool/param descriptions, and the agent
  instructions track the configured scale too.
- Normalize zoom (window basis) and move_cursor (screen basis) inputs and
  rewrite their descriptions, alongside click/double_click/right_click/drag.
- Fix zoom on downscaled (Retina) windows: apply the get_window_state resize
  ratio so the crop lands on the region the agent saw. Normalized mode only;
  pixel-mode zoom unchanged.

All coordinate behavior stays gated on the normalized flag, so the default
(pixels) path is unchanged from upstream.

* chore(cua-driver): add upstream-sync script (git subtree unusable here)

`git subtree split --prefix=libs/cua-driver` hangs on a commit deep in
trycua/cua's history, so the subtree add/pull workflow isn't usable for
the vendored driver (and a pull would re-split + re-hang every time).

Add scripts/sync-from-upstream.sh instead: it git-diffs two upstream refs
(never walks the full history, so it dodges the hang), reprefixes the
libs/cua-driver delta to packages/cua-driver, and `git apply --reject`s it
on top of our local changes — conflicts land as *.rej for manual fixup.
Record the vendored version in .vendored-from and document the migration +
sync method in the design doc.

* chore(cua-driver): exclude vendored driver from qwen-code ESLint

The vendored packages/cua-driver tree carries upstream JS (e.g. the
test-harness Electron app) that doesn't follow qwen-code's lint rules and
fails CI. It is not a workspace package (no package.json) and is not
qwen-code TypeScript, so add it to eslint.config.js global ignores —
alongside packages/desktop/** — the standard treatment for vendored code.

* fix(cua-driver): let start_session revive an idle-reaped session

Ports the fix from upstream trycua/cua#2035 into the vendored driver.

When a session is reaped for idleness, a subsequent start_session with the
same id failed instead of resuming it. Revive the ended session in place so
the agent can continue rather than getting a hard error.

* fix(cua-driver): retry daemon socket writes on EAGAIN

Ports the fix from upstream trycua/cua#2036 into the vendored driver.

A non-blocking daemon socket can return EAGAIN/EWOULDBLOCK mid-write when the
peer's receive buffer is momentarily full. The driver treated that as fatal
and dropped the connection. Add a bounded retry/poll loop (mirror of the
read-side socket_io helper) so transient back-pressure no longer kills the
session; only a real timeout or hard error fails the write.

* fix(cua-driver/linux): stop reporting bare "Clicked" for X11 synthetic clicks

Ports the fix from upstream trycua/cua#2025 into the vendored driver.

On X11, clicks are delivered via XSendEvent synthetic events, which many
toolkits (GTK/SDL/Allegro) ignore because send_event is set. The driver still
reported a flat success ("Clicked"), masking that nothing happened. Report
the synthetic-delivery caveat honestly so the agent can fall back instead of
assuming the click landed.

(platform-linux crate is not built on macOS; verified by clean upstream apply
and covered by upstream + release-workflow Linux CI.)

* fix(cua-driver/windows): list empty-/null-title top-level windows

Ports the fix from upstream trycua/cua#2021 into the vendored driver.

list_windows filtered out any top-level window whose title was empty or null,
so legitimate targets (splash screens, some Electron/game windows, tool
windows) were invisible to the agent and unclickable. Include empty-title
windows, using class name / process as a fallback label.

(platform-windows crate is not built on macOS; verified by clean upstream
apply and covered by upstream + release-workflow Windows CI.)

* chore(cua-driver): track cherry-picked upstream PRs; fix vendored-from

The vendored copy is actually at cua-driver-rs-v0.6.7 (workspace version and
all 0.6.7->0.6.8 delta files confirm it), but .vendored-from had drifted to
0.6.8 during an earlier sync-script trial whose code delta was not kept. Left
as-is it would make a future sync diff 0.6.8->newer and silently skip the real
0.6.7->0.6.8 fixes. Correct it back to 0.6.7.

Also record the four not-yet-merged upstream PRs we carry as cherry-picks
(trycua/cua#2021/#2025/#2035/#2036) in .vendored-patches.md, and have
sync-from-upstream.sh point at it so the next sync reconciles them.

* ci(cua-driver): satisfy repo yamllint on the release workflow

The vendored-driver release workflow tripped 114 quoted-strings violations
under the repo's .yamllint (quote-type: single, required). Single-quote all
string scalars to match every other workflow in .github/workflows.

While reformatting, the release-notes body also got its paragraph blank lines
collapsed and still referenced the old CUA_DRIVER_RS_COORDINATE_SPACE=
normalized_1000 value — restore the blank lines and update it to the current
0/1 toggle (default 0 = off; optional CUA_DRIVER_RS_COORDINATE_SCALE=1000).

* chore(cua-driver): sync vendored driver to cua-driver-rs-v0.6.8

First real run of scripts/sync-from-upstream.sh: it 3-way-applied the upstream
0.6.7->0.6.8 delta onto our local fork. 10/12 files applied cleanly; the 2
rejects (install.ps1, _install-rust.sh) were already-applied baked-version
bumps (0.6.6->0.6.7, our copies were already at 0.6.7), i.e. no real conflict.

0.6.8 brings: Wayland input path (platform-linux), linux health_report +
overlay tweaks, a platform-macos build.rs step, and dependency bumps. Version
moved to 0.6.8 across the workspace.

Verified our work survived the sync untouched: the relative-coordinate shim
(coord_norm/protocol) and all four cherry-picked PRs (socket_io/session +
linux/windows) are intact — in particular the 0.6.8 edit to platform-linux
tools/impl_.rs landed alongside our #2025 change with no collision. macOS
cargo check + 132 core tests green. (platform-linux/windows + the binary
integration test build only on their own runners; upstream CI covers those.)

* ci(cua-driver): add a dry_run gate to the release workflow

Mirror the desktop-release / release dry-run pattern: a workflow_dispatch
dry_run boolean input (default true). The cross-platform build + package jobs
always run and upload their artifacts; the GitHub Release job now publishes
only on a tag push or an explicit dry_run=false dispatch.

Lets us rehearse the whole build/package pipeline (dry_run=true, notarize=false)
and inspect the produced artifacts without cutting a release. A branch push
(no tag, not a dispatch) likewise builds without releasing.
2026-06-26 13:06:43 +00:00

214 lines
7.3 KiB
Python

"""Minimal MCP stdio client used by cua-driver-rs integration tests.
Speaks JSON-RPC 2.0 line-framed over stdin/stdout. Keeps request IDs monotonic
and strips notifications from the response stream. Deliberately tiny — we want
the test failures to point at cua-driver-rs, not at a heavy client dependency.
"""
from __future__ import annotations
import json
import os
import subprocess
import threading
import time
from collections import deque
from typing import Any, Optional
class DriverClient:
def __init__(self, binary_path: str, subcommand: str = "mcp") -> None:
self.binary_path = binary_path
self.subcommand = subcommand
self.process: Optional[subprocess.Popen] = None
self._next_id = 0
self._lines: deque[str] = deque()
self._reader: Optional[threading.Thread] = None
self._stderr_reader: Optional[threading.Thread] = None
def __enter__(self) -> "DriverClient":
self.process = subprocess.Popen(
[self.binary_path, self.subcommand],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
bufsize=1,
)
self._reader = threading.Thread(target=self._pump_stdout, daemon=True)
self._reader.start()
# Drain stderr to prevent pipe-buffer deadlock: the driver writes tracing
# logs to stderr; if the 64KB OS pipe buffer fills the driver blocks and
# the test hangs indefinitely waiting for an MCP response that never comes.
self._stderr_reader = threading.Thread(target=self._drain_stderr, daemon=True)
self._stderr_reader.start()
self._handshake()
return self
def __exit__(self, *exc: Any) -> None:
if not self.process:
return
for stream in (self.process.stdin, self.process.stdout, self.process.stderr):
try:
if stream is not None:
stream.close()
except Exception:
pass
try:
self.process.terminate()
self.process.wait(timeout=3)
except Exception:
self.process.kill()
self.process.wait(timeout=1)
def _pump_stdout(self) -> None:
assert self.process and self.process.stdout
for line in self.process.stdout:
line = line.strip()
if line:
self._lines.append(line)
def _drain_stderr(self) -> None:
"""Discard stderr output to prevent pipe-buffer deadlock."""
assert self.process and self.process.stderr
try:
for _ in self.process.stderr:
pass
except Exception:
pass
def _handshake(self) -> None:
self._call(
"initialize",
{
"protocolVersion": "2025-06-18",
"capabilities": {},
"clientInfo": {"name": "cua-driver-rs-integration", "version": "0.0.1"},
},
)
self._notify("notifications/initialized")
def _notify(self, method: str, params: Optional[dict] = None) -> None:
payload = {"jsonrpc": "2.0", "method": method}
if params is not None:
payload["params"] = params
self._write(payload)
def _call(
self, method: str, params: Optional[dict] = None, timeout: float = 20.0
) -> dict:
self._next_id += 1
request_id = self._next_id
payload = {"jsonrpc": "2.0", "id": request_id, "method": method}
if params is not None:
payload["params"] = params
self._write(payload)
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
while self._lines:
line = self._lines.popleft()
try:
msg = json.loads(line)
except json.JSONDecodeError:
continue
if msg.get("id") == request_id:
if "error" in msg:
raise MCPCallError(msg["error"])
return msg["result"]
time.sleep(0.02)
raise TimeoutError(f"no response for {method} within {timeout}s")
def _write(self, payload: dict) -> None:
assert self.process and self.process.stdin
self.process.stdin.write(json.dumps(payload) + "\n")
self.process.stdin.flush()
def list_tools(self) -> list[dict]:
return self._call("tools/list")["tools"]
def call_tool(self, name: str, arguments: Optional[dict] = None) -> dict:
return self._call(
"tools/call",
{"name": name, "arguments": arguments or {}},
)
class MCPCallError(RuntimeError):
def __init__(self, error: dict) -> None:
super().__init__(f"MCP error {error.get('code')}: {error.get('message')}")
self.code = error.get("code")
self.message = error.get("message")
def default_binary_path() -> str:
"""Return the path to the cua-driver-rs binary.
Checks CUA_DRIVER_BINARY env var first, then falls back to the debug build
in the Rust workspace target directory.
"""
return os.environ.get(
"CUA_DRIVER_BINARY",
os.path.join(
os.path.dirname(os.path.abspath(__file__)),
"..",
"..",
"target",
"debug",
"cua-driver",
),
)
def reset_calculator(settle_s: float = 0.3) -> None:
"""Force a known clean state for Calculator.
Several tests drive Calculator as a 'freshly launched, never frontmost'
target. `pkill -x Calculator` + short settle is the shared boilerplate.
"""
subprocess.run(["pkill", "-x", "Calculator"], check=False)
time.sleep(settle_s)
def frontmost_bundle_id(client: "DriverClient") -> Optional[str]:
"""Return the bundle id of the currently-frontmost app, or None."""
apps = client.call_tool("list_apps")["structuredContent"]["apps"]
for app in apps:
if app.get("active"):
return app.get("bundle_id")
return None
def resolve_window_id(
client: "DriverClient", pid: int, require_on_current_space: bool = True
) -> int:
"""Pick a `window_id` for `pid` via `list_windows`.
Prefers windows that are on-screen AND on the user's current Space,
then max z_index. Falls back to any layer-0 window for the pid when
the preferred filter finds nothing (hidden-launched / minimized apps
still have layer-0 windows we can snapshot).
Raises `RuntimeError` when the pid has no layer-0 window at all.
"""
result = client.call_tool("list_windows", {"pid": pid})
windows = result["structuredContent"]["windows"]
if not windows:
raise RuntimeError(f"pid {pid} has no windows")
if require_on_current_space:
preferred = [
w for w in windows
if w.get("is_on_screen") and w.get("on_current_space") is not False
]
if preferred:
preferred.sort(key=lambda w: w.get("z_index", 0), reverse=True)
return preferred[0]["window_id"]
# Fallback: any layer-0 window. Pick the one with the biggest bounds.
def _area(w: dict) -> float:
b = w.get("bounds", {})
return float(b.get("width", 0)) * float(b.get("height", 0))
windows.sort(key=_area, reverse=True)
return windows[0]["window_id"]