mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-08-15 03:34:59 +00:00
* feat(cua-driver): vendor trycua/cua driver with 1000-normalized coordinate support
Vendor libs/cua-driver from trycua/cua into packages/cua-driver as the
basis for qwen-code's computer-use backend, adding an opt-in relative
(1000x1000 normalized) coordinate mode for Qwen-VL clients.
- coord_norm.rs: 0-1000 <-> pixel conversion, per-(pid,window_id) size
cache, tools/list description rewrite (TDD, 27 tests)
- ToolRegistry: normalized field + invoke input/output hooks
- protocol.rs: system-instruction coordinate wording switched by mode
- serve.rs: daemon list path description rewrite (input_schema aware)
- main.rs: CUA_DRIVER_RS_COORDINATE_SPACE env seed
Default coordinate_space=pixels => zero behavior change for existing
pixel clients. Set CUA_DRIVER_RS_COORDINATE_SPACE=normalized_1000 to
enable. Excludes rust/target build output.
* feat(cua-driver): make normalized coordinate scale configurable
Add CUA_DRIVER_RS_COORDINATE_SCALE (default 1000) so the normalization
full-scale can absorb the Qwen 999-vs-1000 cookbook ambiguity without a
recompile. norm_to_px/px_to_norm now take an explicit scale; denormalize_args
reads the process-wide COORDINATE_SCALE seeded once at startup from env.
* ci(cua-driver): add cross-platform release workflow for vendored driver
Standalone GitHub Action that builds, signs, and releases the vendored
cua-driver under packages/cua-driver. Adapted from upstream trycua/cua
cd-rust-cua-driver.yml:
- macOS: universal binary (lipo arm64+x86_64), codesigned + notarized into
CuaDriver.app using qwen-code's existing secrets (MAC_CSC_LINK cert +
App Store Connect API key notarization); Developer ID identity is
auto-discovered from the imported cert.
- Linux: x86_64 + arm64, built in debian:11 for a glibc 2.31 floor.
- Windows: x86_64 + arm64, unsigned (no EV cert, matches upstream).
- Release: softprops/action-gh-release on cua-driver-rs-v* tags or manual
dispatch, prerelease.
Triggered by tag push (cua-driver-rs-v*) or workflow_dispatch.
* chore(cua-driver): rebrand vendored driver as qwen-cua-driver
Rename the vendored trycua/cua driver so the fork installs and runs
independently of any upstream trycua install:
- binary cua-driver -> qwen-cua-driver
- bundle CuaDriver.app -> QwenCuaDriver.app
- bundle id com.trycua.driver -> com.qwencode.cua-driver
Updates the cargo/uia manifests, Info.plist, bundle/proxy launch paths,
permission/health-report wording, the install/build scripts, and the
cross-platform release workflow.
* feat(cua-driver): finish relative-coordinate mode — toggle, scale, zoom/move_cursor
- CUA_DRIVER_RS_COORDINATE_SPACE is now a 1/0 toggle (via is_env_truthy);
default off keeps pixel mode byte-identical to upstream.
- Thread CUA_DRIVER_RS_COORDINATE_SCALE through every coordinate surface
(was hardcoded 1000): input denormalization already used it; now the
rewritten screenshot dims, the tool/param descriptions, and the agent
instructions track the configured scale too.
- Normalize zoom (window basis) and move_cursor (screen basis) inputs and
rewrite their descriptions, alongside click/double_click/right_click/drag.
- Fix zoom on downscaled (Retina) windows: apply the get_window_state resize
ratio so the crop lands on the region the agent saw. Normalized mode only;
pixel-mode zoom unchanged.
All coordinate behavior stays gated on the normalized flag, so the default
(pixels) path is unchanged from upstream.
* chore(cua-driver): add upstream-sync script (git subtree unusable here)
`git subtree split --prefix=libs/cua-driver` hangs on a commit deep in
trycua/cua's history, so the subtree add/pull workflow isn't usable for
the vendored driver (and a pull would re-split + re-hang every time).
Add scripts/sync-from-upstream.sh instead: it git-diffs two upstream refs
(never walks the full history, so it dodges the hang), reprefixes the
libs/cua-driver delta to packages/cua-driver, and `git apply --reject`s it
on top of our local changes — conflicts land as *.rej for manual fixup.
Record the vendored version in .vendored-from and document the migration +
sync method in the design doc.
* chore(cua-driver): exclude vendored driver from qwen-code ESLint
The vendored packages/cua-driver tree carries upstream JS (e.g. the
test-harness Electron app) that doesn't follow qwen-code's lint rules and
fails CI. It is not a workspace package (no package.json) and is not
qwen-code TypeScript, so add it to eslint.config.js global ignores —
alongside packages/desktop/** — the standard treatment for vendored code.
* fix(cua-driver): let start_session revive an idle-reaped session
Ports the fix from upstream trycua/cua#2035 into the vendored driver.
When a session is reaped for idleness, a subsequent start_session with the
same id failed instead of resuming it. Revive the ended session in place so
the agent can continue rather than getting a hard error.
* fix(cua-driver): retry daemon socket writes on EAGAIN
Ports the fix from upstream trycua/cua#2036 into the vendored driver.
A non-blocking daemon socket can return EAGAIN/EWOULDBLOCK mid-write when the
peer's receive buffer is momentarily full. The driver treated that as fatal
and dropped the connection. Add a bounded retry/poll loop (mirror of the
read-side socket_io helper) so transient back-pressure no longer kills the
session; only a real timeout or hard error fails the write.
* fix(cua-driver/linux): stop reporting bare "Clicked" for X11 synthetic clicks
Ports the fix from upstream trycua/cua#2025 into the vendored driver.
On X11, clicks are delivered via XSendEvent synthetic events, which many
toolkits (GTK/SDL/Allegro) ignore because send_event is set. The driver still
reported a flat success ("Clicked"), masking that nothing happened. Report
the synthetic-delivery caveat honestly so the agent can fall back instead of
assuming the click landed.
(platform-linux crate is not built on macOS; verified by clean upstream apply
and covered by upstream + release-workflow Linux CI.)
* fix(cua-driver/windows): list empty-/null-title top-level windows
Ports the fix from upstream trycua/cua#2021 into the vendored driver.
list_windows filtered out any top-level window whose title was empty or null,
so legitimate targets (splash screens, some Electron/game windows, tool
windows) were invisible to the agent and unclickable. Include empty-title
windows, using class name / process as a fallback label.
(platform-windows crate is not built on macOS; verified by clean upstream
apply and covered by upstream + release-workflow Windows CI.)
* chore(cua-driver): track cherry-picked upstream PRs; fix vendored-from
The vendored copy is actually at cua-driver-rs-v0.6.7 (workspace version and
all 0.6.7->0.6.8 delta files confirm it), but .vendored-from had drifted to
0.6.8 during an earlier sync-script trial whose code delta was not kept. Left
as-is it would make a future sync diff 0.6.8->newer and silently skip the real
0.6.7->0.6.8 fixes. Correct it back to 0.6.7.
Also record the four not-yet-merged upstream PRs we carry as cherry-picks
(trycua/cua#2021/#2025/#2035/#2036) in .vendored-patches.md, and have
sync-from-upstream.sh point at it so the next sync reconciles them.
* ci(cua-driver): satisfy repo yamllint on the release workflow
The vendored-driver release workflow tripped 114 quoted-strings violations
under the repo's .yamllint (quote-type: single, required). Single-quote all
string scalars to match every other workflow in .github/workflows.
While reformatting, the release-notes body also got its paragraph blank lines
collapsed and still referenced the old CUA_DRIVER_RS_COORDINATE_SPACE=
normalized_1000 value — restore the blank lines and update it to the current
0/1 toggle (default 0 = off; optional CUA_DRIVER_RS_COORDINATE_SCALE=1000).
* chore(cua-driver): sync vendored driver to cua-driver-rs-v0.6.8
First real run of scripts/sync-from-upstream.sh: it 3-way-applied the upstream
0.6.7->0.6.8 delta onto our local fork. 10/12 files applied cleanly; the 2
rejects (install.ps1, _install-rust.sh) were already-applied baked-version
bumps (0.6.6->0.6.7, our copies were already at 0.6.7), i.e. no real conflict.
0.6.8 brings: Wayland input path (platform-linux), linux health_report +
overlay tweaks, a platform-macos build.rs step, and dependency bumps. Version
moved to 0.6.8 across the workspace.
Verified our work survived the sync untouched: the relative-coordinate shim
(coord_norm/protocol) and all four cherry-picked PRs (socket_io/session +
linux/windows) are intact — in particular the 0.6.8 edit to platform-linux
tools/impl_.rs landed alongside our #2025 change with no collision. macOS
cargo check + 132 core tests green. (platform-linux/windows + the binary
integration test build only on their own runners; upstream CI covers those.)
* ci(cua-driver): add a dry_run gate to the release workflow
Mirror the desktop-release / release dry-run pattern: a workflow_dispatch
dry_run boolean input (default true). The cross-platform build + package jobs
always run and upload their artifacts; the GitHub Release job now publishes
only on a tag push or an explicit dry_run=false dispatch.
Lets us rehearse the whole build/package pipeline (dry_run=true, notarize=false)
and inspect the produced artifacts without cutting a release. A branch push
(no tag, not a dispatch) likewise builds without releasing.
214 lines
7.3 KiB
Python
214 lines
7.3 KiB
Python
"""Minimal MCP stdio client used by cua-driver-rs integration tests.
|
|
|
|
Speaks JSON-RPC 2.0 line-framed over stdin/stdout. Keeps request IDs monotonic
|
|
and strips notifications from the response stream. Deliberately tiny — we want
|
|
the test failures to point at cua-driver-rs, not at a heavy client dependency.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import os
|
|
import subprocess
|
|
import threading
|
|
import time
|
|
from collections import deque
|
|
from typing import Any, Optional
|
|
|
|
|
|
class DriverClient:
|
|
def __init__(self, binary_path: str, subcommand: str = "mcp") -> None:
|
|
self.binary_path = binary_path
|
|
self.subcommand = subcommand
|
|
self.process: Optional[subprocess.Popen] = None
|
|
self._next_id = 0
|
|
self._lines: deque[str] = deque()
|
|
self._reader: Optional[threading.Thread] = None
|
|
self._stderr_reader: Optional[threading.Thread] = None
|
|
|
|
def __enter__(self) -> "DriverClient":
|
|
self.process = subprocess.Popen(
|
|
[self.binary_path, self.subcommand],
|
|
stdin=subprocess.PIPE,
|
|
stdout=subprocess.PIPE,
|
|
stderr=subprocess.PIPE,
|
|
text=True,
|
|
bufsize=1,
|
|
)
|
|
self._reader = threading.Thread(target=self._pump_stdout, daemon=True)
|
|
self._reader.start()
|
|
# Drain stderr to prevent pipe-buffer deadlock: the driver writes tracing
|
|
# logs to stderr; if the 64KB OS pipe buffer fills the driver blocks and
|
|
# the test hangs indefinitely waiting for an MCP response that never comes.
|
|
self._stderr_reader = threading.Thread(target=self._drain_stderr, daemon=True)
|
|
self._stderr_reader.start()
|
|
self._handshake()
|
|
return self
|
|
|
|
def __exit__(self, *exc: Any) -> None:
|
|
if not self.process:
|
|
return
|
|
for stream in (self.process.stdin, self.process.stdout, self.process.stderr):
|
|
try:
|
|
if stream is not None:
|
|
stream.close()
|
|
except Exception:
|
|
pass
|
|
try:
|
|
self.process.terminate()
|
|
self.process.wait(timeout=3)
|
|
except Exception:
|
|
self.process.kill()
|
|
self.process.wait(timeout=1)
|
|
|
|
def _pump_stdout(self) -> None:
|
|
assert self.process and self.process.stdout
|
|
for line in self.process.stdout:
|
|
line = line.strip()
|
|
if line:
|
|
self._lines.append(line)
|
|
|
|
def _drain_stderr(self) -> None:
|
|
"""Discard stderr output to prevent pipe-buffer deadlock."""
|
|
assert self.process and self.process.stderr
|
|
try:
|
|
for _ in self.process.stderr:
|
|
pass
|
|
except Exception:
|
|
pass
|
|
|
|
def _handshake(self) -> None:
|
|
self._call(
|
|
"initialize",
|
|
{
|
|
"protocolVersion": "2025-06-18",
|
|
"capabilities": {},
|
|
"clientInfo": {"name": "cua-driver-rs-integration", "version": "0.0.1"},
|
|
},
|
|
)
|
|
self._notify("notifications/initialized")
|
|
|
|
def _notify(self, method: str, params: Optional[dict] = None) -> None:
|
|
payload = {"jsonrpc": "2.0", "method": method}
|
|
if params is not None:
|
|
payload["params"] = params
|
|
self._write(payload)
|
|
|
|
def _call(
|
|
self, method: str, params: Optional[dict] = None, timeout: float = 20.0
|
|
) -> dict:
|
|
self._next_id += 1
|
|
request_id = self._next_id
|
|
payload = {"jsonrpc": "2.0", "id": request_id, "method": method}
|
|
if params is not None:
|
|
payload["params"] = params
|
|
self._write(payload)
|
|
|
|
deadline = time.monotonic() + timeout
|
|
while time.monotonic() < deadline:
|
|
while self._lines:
|
|
line = self._lines.popleft()
|
|
try:
|
|
msg = json.loads(line)
|
|
except json.JSONDecodeError:
|
|
continue
|
|
if msg.get("id") == request_id:
|
|
if "error" in msg:
|
|
raise MCPCallError(msg["error"])
|
|
return msg["result"]
|
|
time.sleep(0.02)
|
|
raise TimeoutError(f"no response for {method} within {timeout}s")
|
|
|
|
def _write(self, payload: dict) -> None:
|
|
assert self.process and self.process.stdin
|
|
self.process.stdin.write(json.dumps(payload) + "\n")
|
|
self.process.stdin.flush()
|
|
|
|
def list_tools(self) -> list[dict]:
|
|
return self._call("tools/list")["tools"]
|
|
|
|
def call_tool(self, name: str, arguments: Optional[dict] = None) -> dict:
|
|
return self._call(
|
|
"tools/call",
|
|
{"name": name, "arguments": arguments or {}},
|
|
)
|
|
|
|
|
|
class MCPCallError(RuntimeError):
|
|
def __init__(self, error: dict) -> None:
|
|
super().__init__(f"MCP error {error.get('code')}: {error.get('message')}")
|
|
self.code = error.get("code")
|
|
self.message = error.get("message")
|
|
|
|
|
|
def default_binary_path() -> str:
|
|
"""Return the path to the cua-driver-rs binary.
|
|
|
|
Checks CUA_DRIVER_BINARY env var first, then falls back to the debug build
|
|
in the Rust workspace target directory.
|
|
"""
|
|
return os.environ.get(
|
|
"CUA_DRIVER_BINARY",
|
|
os.path.join(
|
|
os.path.dirname(os.path.abspath(__file__)),
|
|
"..",
|
|
"..",
|
|
"target",
|
|
"debug",
|
|
"cua-driver",
|
|
),
|
|
)
|
|
|
|
|
|
def reset_calculator(settle_s: float = 0.3) -> None:
|
|
"""Force a known clean state for Calculator.
|
|
|
|
Several tests drive Calculator as a 'freshly launched, never frontmost'
|
|
target. `pkill -x Calculator` + short settle is the shared boilerplate.
|
|
"""
|
|
subprocess.run(["pkill", "-x", "Calculator"], check=False)
|
|
time.sleep(settle_s)
|
|
|
|
|
|
def frontmost_bundle_id(client: "DriverClient") -> Optional[str]:
|
|
"""Return the bundle id of the currently-frontmost app, or None."""
|
|
apps = client.call_tool("list_apps")["structuredContent"]["apps"]
|
|
for app in apps:
|
|
if app.get("active"):
|
|
return app.get("bundle_id")
|
|
return None
|
|
|
|
|
|
def resolve_window_id(
|
|
client: "DriverClient", pid: int, require_on_current_space: bool = True
|
|
) -> int:
|
|
"""Pick a `window_id` for `pid` via `list_windows`.
|
|
|
|
Prefers windows that are on-screen AND on the user's current Space,
|
|
then max z_index. Falls back to any layer-0 window for the pid when
|
|
the preferred filter finds nothing (hidden-launched / minimized apps
|
|
still have layer-0 windows we can snapshot).
|
|
|
|
Raises `RuntimeError` when the pid has no layer-0 window at all.
|
|
"""
|
|
result = client.call_tool("list_windows", {"pid": pid})
|
|
windows = result["structuredContent"]["windows"]
|
|
if not windows:
|
|
raise RuntimeError(f"pid {pid} has no windows")
|
|
|
|
if require_on_current_space:
|
|
preferred = [
|
|
w for w in windows
|
|
if w.get("is_on_screen") and w.get("on_current_space") is not False
|
|
]
|
|
if preferred:
|
|
preferred.sort(key=lambda w: w.get("z_index", 0), reverse=True)
|
|
return preferred[0]["window_id"]
|
|
|
|
# Fallback: any layer-0 window. Pick the one with the biggest bounds.
|
|
def _area(w: dict) -> float:
|
|
b = w.get("bounds", {})
|
|
return float(b.get("width", 0)) * float(b.get("height", 0))
|
|
|
|
windows.sort(key=_area, reverse=True)
|
|
return windows[0]["window_id"]
|