mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-08-15 03:34:59 +00:00
* feat(cua-driver): vendor trycua/cua driver with 1000-normalized coordinate support
Vendor libs/cua-driver from trycua/cua into packages/cua-driver as the
basis for qwen-code's computer-use backend, adding an opt-in relative
(1000x1000 normalized) coordinate mode for Qwen-VL clients.
- coord_norm.rs: 0-1000 <-> pixel conversion, per-(pid,window_id) size
cache, tools/list description rewrite (TDD, 27 tests)
- ToolRegistry: normalized field + invoke input/output hooks
- protocol.rs: system-instruction coordinate wording switched by mode
- serve.rs: daemon list path description rewrite (input_schema aware)
- main.rs: CUA_DRIVER_RS_COORDINATE_SPACE env seed
Default coordinate_space=pixels => zero behavior change for existing
pixel clients. Set CUA_DRIVER_RS_COORDINATE_SPACE=normalized_1000 to
enable. Excludes rust/target build output.
* feat(cua-driver): make normalized coordinate scale configurable
Add CUA_DRIVER_RS_COORDINATE_SCALE (default 1000) so the normalization
full-scale can absorb the Qwen 999-vs-1000 cookbook ambiguity without a
recompile. norm_to_px/px_to_norm now take an explicit scale; denormalize_args
reads the process-wide COORDINATE_SCALE seeded once at startup from env.
* ci(cua-driver): add cross-platform release workflow for vendored driver
Standalone GitHub Action that builds, signs, and releases the vendored
cua-driver under packages/cua-driver. Adapted from upstream trycua/cua
cd-rust-cua-driver.yml:
- macOS: universal binary (lipo arm64+x86_64), codesigned + notarized into
CuaDriver.app using qwen-code's existing secrets (MAC_CSC_LINK cert +
App Store Connect API key notarization); Developer ID identity is
auto-discovered from the imported cert.
- Linux: x86_64 + arm64, built in debian:11 for a glibc 2.31 floor.
- Windows: x86_64 + arm64, unsigned (no EV cert, matches upstream).
- Release: softprops/action-gh-release on cua-driver-rs-v* tags or manual
dispatch, prerelease.
Triggered by tag push (cua-driver-rs-v*) or workflow_dispatch.
* chore(cua-driver): rebrand vendored driver as qwen-cua-driver
Rename the vendored trycua/cua driver so the fork installs and runs
independently of any upstream trycua install:
- binary cua-driver -> qwen-cua-driver
- bundle CuaDriver.app -> QwenCuaDriver.app
- bundle id com.trycua.driver -> com.qwencode.cua-driver
Updates the cargo/uia manifests, Info.plist, bundle/proxy launch paths,
permission/health-report wording, the install/build scripts, and the
cross-platform release workflow.
* feat(cua-driver): finish relative-coordinate mode — toggle, scale, zoom/move_cursor
- CUA_DRIVER_RS_COORDINATE_SPACE is now a 1/0 toggle (via is_env_truthy);
default off keeps pixel mode byte-identical to upstream.
- Thread CUA_DRIVER_RS_COORDINATE_SCALE through every coordinate surface
(was hardcoded 1000): input denormalization already used it; now the
rewritten screenshot dims, the tool/param descriptions, and the agent
instructions track the configured scale too.
- Normalize zoom (window basis) and move_cursor (screen basis) inputs and
rewrite their descriptions, alongside click/double_click/right_click/drag.
- Fix zoom on downscaled (Retina) windows: apply the get_window_state resize
ratio so the crop lands on the region the agent saw. Normalized mode only;
pixel-mode zoom unchanged.
All coordinate behavior stays gated on the normalized flag, so the default
(pixels) path is unchanged from upstream.
* chore(cua-driver): add upstream-sync script (git subtree unusable here)
`git subtree split --prefix=libs/cua-driver` hangs on a commit deep in
trycua/cua's history, so the subtree add/pull workflow isn't usable for
the vendored driver (and a pull would re-split + re-hang every time).
Add scripts/sync-from-upstream.sh instead: it git-diffs two upstream refs
(never walks the full history, so it dodges the hang), reprefixes the
libs/cua-driver delta to packages/cua-driver, and `git apply --reject`s it
on top of our local changes — conflicts land as *.rej for manual fixup.
Record the vendored version in .vendored-from and document the migration +
sync method in the design doc.
* chore(cua-driver): exclude vendored driver from qwen-code ESLint
The vendored packages/cua-driver tree carries upstream JS (e.g. the
test-harness Electron app) that doesn't follow qwen-code's lint rules and
fails CI. It is not a workspace package (no package.json) and is not
qwen-code TypeScript, so add it to eslint.config.js global ignores —
alongside packages/desktop/** — the standard treatment for vendored code.
* fix(cua-driver): let start_session revive an idle-reaped session
Ports the fix from upstream trycua/cua#2035 into the vendored driver.
When a session is reaped for idleness, a subsequent start_session with the
same id failed instead of resuming it. Revive the ended session in place so
the agent can continue rather than getting a hard error.
* fix(cua-driver): retry daemon socket writes on EAGAIN
Ports the fix from upstream trycua/cua#2036 into the vendored driver.
A non-blocking daemon socket can return EAGAIN/EWOULDBLOCK mid-write when the
peer's receive buffer is momentarily full. The driver treated that as fatal
and dropped the connection. Add a bounded retry/poll loop (mirror of the
read-side socket_io helper) so transient back-pressure no longer kills the
session; only a real timeout or hard error fails the write.
* fix(cua-driver/linux): stop reporting bare "Clicked" for X11 synthetic clicks
Ports the fix from upstream trycua/cua#2025 into the vendored driver.
On X11, clicks are delivered via XSendEvent synthetic events, which many
toolkits (GTK/SDL/Allegro) ignore because send_event is set. The driver still
reported a flat success ("Clicked"), masking that nothing happened. Report
the synthetic-delivery caveat honestly so the agent can fall back instead of
assuming the click landed.
(platform-linux crate is not built on macOS; verified by clean upstream apply
and covered by upstream + release-workflow Linux CI.)
* fix(cua-driver/windows): list empty-/null-title top-level windows
Ports the fix from upstream trycua/cua#2021 into the vendored driver.
list_windows filtered out any top-level window whose title was empty or null,
so legitimate targets (splash screens, some Electron/game windows, tool
windows) were invisible to the agent and unclickable. Include empty-title
windows, using class name / process as a fallback label.
(platform-windows crate is not built on macOS; verified by clean upstream
apply and covered by upstream + release-workflow Windows CI.)
* chore(cua-driver): track cherry-picked upstream PRs; fix vendored-from
The vendored copy is actually at cua-driver-rs-v0.6.7 (workspace version and
all 0.6.7->0.6.8 delta files confirm it), but .vendored-from had drifted to
0.6.8 during an earlier sync-script trial whose code delta was not kept. Left
as-is it would make a future sync diff 0.6.8->newer and silently skip the real
0.6.7->0.6.8 fixes. Correct it back to 0.6.7.
Also record the four not-yet-merged upstream PRs we carry as cherry-picks
(trycua/cua#2021/#2025/#2035/#2036) in .vendored-patches.md, and have
sync-from-upstream.sh point at it so the next sync reconciles them.
* ci(cua-driver): satisfy repo yamllint on the release workflow
The vendored-driver release workflow tripped 114 quoted-strings violations
under the repo's .yamllint (quote-type: single, required). Single-quote all
string scalars to match every other workflow in .github/workflows.
While reformatting, the release-notes body also got its paragraph blank lines
collapsed and still referenced the old CUA_DRIVER_RS_COORDINATE_SPACE=
normalized_1000 value — restore the blank lines and update it to the current
0/1 toggle (default 0 = off; optional CUA_DRIVER_RS_COORDINATE_SCALE=1000).
* chore(cua-driver): sync vendored driver to cua-driver-rs-v0.6.8
First real run of scripts/sync-from-upstream.sh: it 3-way-applied the upstream
0.6.7->0.6.8 delta onto our local fork. 10/12 files applied cleanly; the 2
rejects (install.ps1, _install-rust.sh) were already-applied baked-version
bumps (0.6.6->0.6.7, our copies were already at 0.6.7), i.e. no real conflict.
0.6.8 brings: Wayland input path (platform-linux), linux health_report +
overlay tweaks, a platform-macos build.rs step, and dependency bumps. Version
moved to 0.6.8 across the workspace.
Verified our work survived the sync untouched: the relative-coordinate shim
(coord_norm/protocol) and all four cherry-picked PRs (socket_io/session +
linux/windows) are intact — in particular the 0.6.8 edit to platform-linux
tools/impl_.rs landed alongside our #2025 change with no collision. macOS
cargo check + 132 core tests green. (platform-linux/windows + the binary
integration test build only on their own runners; upstream CI covers those.)
* ci(cua-driver): add a dry_run gate to the release workflow
Mirror the desktop-release / release dry-run pattern: a workflow_dispatch
dry_run boolean input (default true). The cross-platform build + package jobs
always run and upload their artifacts; the GitHub Release job now publishes
only on a tag push or an explicit dry_run=false dispatch.
Lets us rehearse the whole build/package pipeline (dry_run=true, notarize=false)
and inspect the produced artifacts without cutting a release. A branch push
(no tag, not a dispatch) likewise builds without releasing.
324 lines
12 KiB
Python
324 lines
12 KiB
Python
"""Integration test: two cua-driver processes running concurrently.
|
|
|
|
Verifies that two independent cua-driver instances can coexist without:
|
|
- crashing each other
|
|
- stealing focus from FocusMonitorApp
|
|
- interfering with each other's operations
|
|
|
|
Scenario:
|
|
Driver A → AX-clicks Calculator (2 + 2 = 4)
|
|
Driver B → Concurrently moves its agent cursor and inspects window list
|
|
|
|
Run:
|
|
CUA_DRIVER_BINARY=../../target/debug/cua-driver python3 -m unittest test_concurrent_drivers -v
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import re
|
|
import subprocess
|
|
import sys
|
|
import threading
|
|
import time
|
|
import unittest
|
|
|
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
|
from driver_client import (
|
|
DriverClient,
|
|
default_binary_path,
|
|
frontmost_bundle_id,
|
|
resolve_window_id,
|
|
)
|
|
|
|
_THIS_DIR = os.path.dirname(os.path.abspath(__file__))
|
|
_DRIVER_RS_ROOT = os.path.dirname(os.path.dirname(_THIS_DIR))
|
|
_LIBS_ROOT = os.path.dirname(_DRIVER_RS_ROOT)
|
|
_FOCUS_APP_DIR = os.path.join(_LIBS_ROOT, "cua-driver", "Tests", "FocusMonitorApp")
|
|
_FOCUS_APP_EXE = os.path.join(
|
|
_FOCUS_APP_DIR, "FocusMonitorApp.app", "Contents", "MacOS", "FocusMonitorApp"
|
|
)
|
|
_LOSS_FILE = "/tmp/focus_monitor_losses.txt"
|
|
|
|
CALC_BUNDLE = "com.apple.calculator"
|
|
FOCUS_MONITOR_BUNDLE = "com.trycua.FocusMonitorApp"
|
|
|
|
|
|
def _build_focus_app() -> None:
|
|
if not os.path.exists(_FOCUS_APP_EXE):
|
|
subprocess.run(
|
|
[os.path.join(_FOCUS_APP_DIR, "build.sh")], check=True
|
|
)
|
|
|
|
|
|
def _launch_focus_app() -> tuple[subprocess.Popen, int]:
|
|
proc = subprocess.Popen(
|
|
[_FOCUS_APP_EXE],
|
|
stdout=subprocess.PIPE,
|
|
stderr=subprocess.DEVNULL,
|
|
text=True,
|
|
)
|
|
for _ in range(40):
|
|
line = proc.stdout.readline().strip()
|
|
if line.startswith("FOCUS_PID="):
|
|
return proc, int(line.split("=", 1)[1])
|
|
time.sleep(0.1)
|
|
proc.terminate()
|
|
raise RuntimeError("FocusMonitorApp did not print FOCUS_PID in time")
|
|
|
|
|
|
def _read_focus_losses() -> int:
|
|
try:
|
|
with open(_LOSS_FILE) as f:
|
|
return int(f.read().strip())
|
|
except (FileNotFoundError, ValueError):
|
|
return -1
|
|
|
|
|
|
def _find_calc_button(tree: str, label: str) -> int | None:
|
|
for line in tree.split("\n"):
|
|
if "AXButton" not in line:
|
|
continue
|
|
m = re.search(r'\[(\d+)\]', line)
|
|
if not m:
|
|
continue
|
|
if f'({label})' in line or f'id={label}' in line:
|
|
return int(m.group(1))
|
|
return None
|
|
|
|
|
|
class ConcurrentDriversTest(unittest.TestCase):
|
|
"""Two driver processes running side-by-side without interference."""
|
|
|
|
_calc_pid: int
|
|
_focus_proc: subprocess.Popen
|
|
_focus_pid: int
|
|
binary: str
|
|
|
|
@classmethod
|
|
def setUpClass(cls) -> None:
|
|
_build_focus_app()
|
|
|
|
# Kill stale processes from any prior crashed run.
|
|
subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)
|
|
subprocess.run(["pkill", "-x", "Calculator"], check=False)
|
|
time.sleep(0.3)
|
|
|
|
try:
|
|
os.remove(_LOSS_FILE)
|
|
except FileNotFoundError:
|
|
pass
|
|
|
|
cls.binary = default_binary_path()
|
|
|
|
# Launch Calculator via Driver A.
|
|
with DriverClient(cls.binary) as c:
|
|
result = c.call_tool("launch_app", {"bundle_id": CALC_BUNDLE})
|
|
cls._calc_pid = result["structuredContent"]["pid"]
|
|
print(f"\n Calculator pid: {cls._calc_pid}")
|
|
time.sleep(1.5)
|
|
|
|
# Launch FocusMonitorApp (becomes frontmost).
|
|
cls._focus_proc, cls._focus_pid = _launch_focus_app()
|
|
print(f" FocusMonitor pid: {cls._focus_pid}")
|
|
time.sleep(1.0)
|
|
|
|
with DriverClient(cls.binary) as c:
|
|
active = frontmost_bundle_id(c)
|
|
assert active == FOCUS_MONITOR_BUNDLE, (
|
|
f"Expected FocusMonitorApp frontmost, got {active}"
|
|
)
|
|
|
|
losses = _read_focus_losses()
|
|
assert losses == 0, f"Expected 0 focus losses at start, got {losses}"
|
|
|
|
@classmethod
|
|
def tearDownClass(cls) -> None:
|
|
if hasattr(cls, '_focus_proc'):
|
|
cls._focus_proc.terminate()
|
|
try:
|
|
cls._focus_proc.wait(timeout=3)
|
|
except subprocess.TimeoutExpired:
|
|
cls._focus_proc.kill()
|
|
if cls._focus_proc.stdout:
|
|
cls._focus_proc.stdout.close()
|
|
subprocess.run(["pkill", "-x", "FocusMonitorApp"], check=False)
|
|
subprocess.run(["pkill", "-x", "Calculator"], check=False)
|
|
try:
|
|
os.remove(_LOSS_FILE)
|
|
except FileNotFoundError:
|
|
pass
|
|
|
|
def setUp(self) -> None:
|
|
# Clear Calculator before each test so previous display state doesn't interfere.
|
|
with DriverClient(self.binary) as c:
|
|
window_id = resolve_window_id(c, self._calc_pid)
|
|
snap = c.call_tool("get_window_state", {
|
|
"pid": self._calc_pid, "window_id": window_id,
|
|
})
|
|
tree = snap.get("structuredContent", snap).get("tree_markdown", "")
|
|
btn_ac = _find_calc_button(tree, "All Clear") or _find_calc_button(tree, "Clear")
|
|
if btn_ac is not None:
|
|
c.call_tool("click", {
|
|
"pid": self._calc_pid, "window_id": window_id, "element_index": btn_ac,
|
|
})
|
|
time.sleep(0.2)
|
|
|
|
subprocess.run(
|
|
["osascript", "-e", 'tell application "FocusMonitorApp" to activate'],
|
|
check=False,
|
|
)
|
|
time.sleep(0.5)
|
|
self._losses_before = _read_focus_losses()
|
|
|
|
# ── tests ─────────────────────────────────────────────────────────────────
|
|
|
|
def test_two_drivers_run_concurrently(self) -> None:
|
|
"""Open Driver A and Driver B simultaneously; both must succeed without crashing."""
|
|
|
|
driver_a_result: dict = {}
|
|
driver_b_result: dict = {}
|
|
driver_a_error: list[str] = []
|
|
driver_b_error: list[str] = []
|
|
|
|
def run_driver_a() -> None:
|
|
try:
|
|
with DriverClient(self.binary) as ca:
|
|
# Driver A: AX-click 2 + 2 = on Calculator.
|
|
window_id = resolve_window_id(ca, self._calc_pid)
|
|
snap = ca.call_tool("get_window_state", {
|
|
"pid": self._calc_pid, "window_id": window_id,
|
|
})
|
|
tree = snap.get("structuredContent", snap).get("tree_markdown", "")
|
|
btn_2 = _find_calc_button(tree, "2")
|
|
btn_add = _find_calc_button(tree, "Add")
|
|
btn_eq = _find_calc_button(tree, "Equals")
|
|
if None in (btn_2, btn_add, btn_eq):
|
|
driver_a_error.append("Missing calculator buttons")
|
|
return
|
|
for idx in [btn_2, btn_add, btn_2, btn_eq]:
|
|
ca.call_tool("click", {
|
|
"pid": self._calc_pid,
|
|
"window_id": window_id,
|
|
"element_index": idx,
|
|
})
|
|
time.sleep(0.15)
|
|
snap2 = ca.call_tool("get_window_state", {
|
|
"pid": self._calc_pid, "window_id": window_id,
|
|
"query": "AXStaticText",
|
|
})
|
|
tree2 = snap2.get("structuredContent", snap2).get("tree_markdown", "")
|
|
driver_a_result["tree"] = tree2
|
|
except Exception as e:
|
|
driver_a_error.append(str(e))
|
|
|
|
def run_driver_b() -> None:
|
|
try:
|
|
with DriverClient(self.binary) as cb:
|
|
# Driver B: move its cursor, list windows, move cursor again.
|
|
cb.call_tool("move_cursor", {"x": 200.0, "y": 200.0, "cursor_id": "agent-b"})
|
|
wins = cb.call_tool("list_windows", {"on_screen_only": True})
|
|
count = len(wins.get("structuredContent", {}).get("windows", []))
|
|
cb.call_tool("move_cursor", {"x": 300.0, "y": 300.0, "cursor_id": "agent-b"})
|
|
driver_b_result["window_count"] = count
|
|
except Exception as e:
|
|
driver_b_error.append(str(e))
|
|
|
|
# Launch both threads simultaneously.
|
|
ta = threading.Thread(target=run_driver_a, daemon=True)
|
|
tb = threading.Thread(target=run_driver_b, daemon=True)
|
|
ta.start()
|
|
tb.start()
|
|
ta.join(timeout=30)
|
|
tb.join(timeout=30)
|
|
|
|
# Both must have completed without error.
|
|
self.assertFalse(
|
|
driver_a_error,
|
|
f"Driver A errored: {driver_a_error}",
|
|
)
|
|
self.assertFalse(
|
|
driver_b_error,
|
|
f"Driver B errored: {driver_b_error}",
|
|
)
|
|
|
|
print(f"\n Driver A result tree:\n{driver_a_result.get('tree', '(empty)')}")
|
|
print(f" Driver B window count: {driver_b_result.get('window_count', -1)}")
|
|
|
|
# Driver A should have produced "4".
|
|
self.assertIn(
|
|
"4",
|
|
driver_a_result.get("tree", ""),
|
|
"Calculator did not show 4 after Driver A clicked 2+2=",
|
|
)
|
|
|
|
# Driver B should have seen at least some windows.
|
|
self.assertGreater(
|
|
driver_b_result.get("window_count", 0),
|
|
0,
|
|
"Driver B saw no windows in list_windows",
|
|
)
|
|
|
|
# Focus must not have been stolen.
|
|
time.sleep(0.3)
|
|
losses = _read_focus_losses()
|
|
with DriverClient(self.binary) as c:
|
|
active = frontmost_bundle_id(c)
|
|
print(f" losses: {self._losses_before}->{losses}, frontmost: {active}")
|
|
self.assertEqual(
|
|
active, FOCUS_MONITOR_BUNDLE,
|
|
f"Focus stolen during concurrent operation — frontmost is {active}",
|
|
)
|
|
|
|
def test_sequential_driver_reuse(self) -> None:
|
|
"""Open/close/reopen a DriverClient — state from prior session does not bleed."""
|
|
|
|
# First session — clear Calculator.
|
|
with DriverClient(self.binary) as c:
|
|
window_id = resolve_window_id(c, self._calc_pid)
|
|
snap = c.call_tool("get_window_state", {
|
|
"pid": self._calc_pid, "window_id": window_id,
|
|
})
|
|
tree = snap.get("structuredContent", snap).get("tree_markdown", "")
|
|
btn_ac = _find_calc_button(tree, "All Clear") or _find_calc_button(tree, "Clear")
|
|
if btn_ac is not None:
|
|
c.call_tool("click", {
|
|
"pid": self._calc_pid, "window_id": window_id, "element_index": btn_ac,
|
|
})
|
|
time.sleep(0.2)
|
|
|
|
time.sleep(0.2)
|
|
|
|
# Second session — press 5.
|
|
with DriverClient(self.binary) as c:
|
|
window_id = resolve_window_id(c, self._calc_pid)
|
|
snap = c.call_tool("get_window_state", {
|
|
"pid": self._calc_pid, "window_id": window_id,
|
|
})
|
|
tree = snap.get("structuredContent", snap).get("tree_markdown", "")
|
|
btn_5 = _find_calc_button(tree, "5")
|
|
self.assertIsNotNone(btn_5, "Could not find '5' button in fresh session")
|
|
c.call_tool("click", {
|
|
"pid": self._calc_pid, "window_id": window_id, "element_index": btn_5,
|
|
})
|
|
time.sleep(0.3)
|
|
snap2 = c.call_tool("get_window_state", {
|
|
"pid": self._calc_pid, "window_id": window_id, "query": "AXStaticText",
|
|
})
|
|
tree2 = snap2.get("structuredContent", snap2).get("tree_markdown", "")
|
|
print(f"\n After pressing 5: {tree2[:200]}")
|
|
|
|
self.assertIn("5", tree2, "Calculator did not show 5 after clicking it in a new session")
|
|
|
|
# Focus check.
|
|
time.sleep(0.3)
|
|
with DriverClient(self.binary) as c:
|
|
active = frontmost_bundle_id(c)
|
|
self.assertEqual(
|
|
active, FOCUS_MONITOR_BUNDLE,
|
|
f"Focus stolen — frontmost is {active}",
|
|
)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
unittest.main(verbosity=2)
|