mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-07-09 17:19:02 +00:00
6 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
顾盼
|
21cdb4a4dc
|
chore(cua-driver): update version refs to 0.7.1 + add fix doc (#6616)
Bump baked version in install scripts, CD workflow default, and README examples to 0.7.1 so the release pipeline picks up the correct version. Add docs/0.7.1-coord-norm-fixes.md documenting all fixes in this release. |
||
|
顾盼
|
e06d3be2b3
|
fix(cua-driver): complete coordinate normalization for zoom/scroll/mouse tools (#6610)
* fix(cua-driver): denormalize from_zoom click coords in normalized mode When CUA_DRIVER_RS_COORDINATE_SPACE=1 (0-1000 normalized coordinates), from_zoom=true clicks were skipping denormalization entirely, causing the model's 0-1000 values to be treated as raw pixel offsets in the zoom image — landing clicks at the wrong position. - Add ZOOM_SIZE_CACHE: cache zoom image dimensions (width/height) from zoom tool results, keyed by pid (matching platform ZoomRegistry) - denormalize_args: when from_zoom=true and cache exists, denormalize x/y against zoom image dimensions instead of skipping - rewrite_coord_desc: rewrite from_zoom description from "pixel coordinates" to "0-1000 normalized coordinates" so the model sends normalized values - ingest_zoom_size: new output hook called after zoom tool execution to populate the cache * fix(cua-driver): error on missing coord cache + normalize scroll/mouse tools Two additional fixes for the 0-1000 normalized coordinate mode: 1. denormalize_args now returns Result<(), String> and errors when a required size basis is missing (SIZE_CACHE, SCREEN_SIZE, DESKTOP_SCREENSHOT_SIZE, ZOOM_SIZE_CACHE), instead of silently passing through unconverted normalized coordinates that would land clicks at wrong positions. The error messages guide the model to call the appropriate query tool (get_window_state, get_screen_size, get_desktop_state, zoom) first. 2. Add scroll, mouse_button_down, mouse_button_up, mouse_drag to input_coord_fields so their x/y coordinates are denormalized in normalized mode. scroll x/y specify where to deliver the wheel event (not scroll amount); Linux mouse tools are stateful press/drag/release with window-local coordinates. * chore(cua-driver): bump version to 0.7.1 * fix(cua-driver): stop rewriting screenshot_width/height in normalized mode normalize_result was rewriting get_window_state and get_desktop_state screenshot_width/height to 1000, conflicting with elements[].frame which stays in pixels. The model received contradictory coordinate information: screenshot dimensions said 1000 but element positions were in real pixels (e.g. x=1444). Query tool results should be returned unmodified. The model is guided to use 0-1000 coordinates through tool schema descriptions and MCP instructions, not by tampering with return values. * fix(cua-driver): address review — normalize_result + parallel_mouse_drag 1. Stop rewriting screenshot_width/height to 1000 in normalized mode. Query results now return real pixel values; the model is guided to use 0-1000 coords through schema descriptions and instructions only. 2. Fix parallel_mouse_drag per-item window_id lookup: each drag item resolves its own (pid, window_id) from SIZE_CACHE instead of using the caller-level screenshot_w/h (which is always 0 since the tool has no top-level pid/window_id). 3. Add x_from/x_to conversion for fn-expression domain bounds. 4. Prevent from_zoom early return from skipping nested coord handler. 5. Fix misleading comments about "non-empty entry" and "nested rewrite". |
||
|
顾盼
|
5605c1b60d
|
fix(cua-driver): migrate Windows scripts + README rewrite (#6515)
* fix(cua-driver): migrate Windows install/uninstall scripts to qwen-cua-driver install.ps1 and uninstall.ps1 were essentially un-migrated from upstream trycua/cua — binary name was `cua-driver.exe` instead of `qwen-cua-driver.exe`, repo pointed to trycua/cua, install paths used upstream vendor names. Changes: - Binary: cua-driver.exe → qwen-cua-driver.exe - Repo: trycua/cua → QwenLM/qwen-code - Install dir: Programs\Cua\cua-driver\ → Programs\Qwen\qwen-cua-driver\ - Home dir: .cua-driver → .qwen-cua-driver - Autostart task: cua-driver-serve → qwen-cua-driver-serve - All URLs and user-facing text updated - BAKED_VERSION: 0.6.8 → 0.7.0 - Added PATH restart hint after install - Docs URL in post-install-hints.txt and _install-rust.sh fallback * fix(cua-driver): correct cua-driver-uia.exe process name in install.ps1 The UIA helper binary ships as cua-driver-uia.exe in release tarballs (not qwen-prefixed), so taskkill and Get-Process must use the actual binary name. * docs(cua-driver): rewrite README with full install/verify/MCP guide Covers macOS, Windows, Linux installation; verification steps; MCP configuration for Qwen Code, Claude Code, and Codex; relative-coordinate mode; uninstall instructions; and platform support matrix. |
||
|
顾盼
|
e5ec6019cd
|
fix(cua-driver): bump BAKED_VERSION to 0.7.0 (#6241)
The install script's baked version was still pointing at 0.6.8, requiring users to manually specify CUA_DRIVER_RS_VERSION=0.7.0. |
||
|
顾盼
|
9605bcc25f
|
feat(cua-driver): sync vendored cua-driver 0.6.8 → 0.7.0 (#6212)
* feat(cua-driver): sync vendored cua-driver from 0.6.8 to 0.7.0 Upstream 0.7.0 brings action-time modality (get_window_state always returns AX tree + screenshot, capture_mode retired), honest verification (every action returns verified/path/effect/escalation), browser page (CDP) tools, desktop-scope (capture_scope=window|desktop, get_desktop_state, screen-absolute click/scroll), and Rust-only consolidation (Swift backend retired). Cherry-pick patch changes: - #2025 (X11 synthetic click): retired — superseded by 0.7.0 honest-verification - #2035 (session revive): retired — superseded by 0.7.0 revive_session/loud-reject/is_session_lifecycle_tool - #2036 (EAGAIN socket write retry): re-injected — merged upstream post-0.7.0 tag, EAGAIN os error 35 is not acceptable - #2021 (null/empty-title windows + resolve_uwp_host_window): retained, adapted to 0.7.0's new is_listable_top_level and window_title helpers Shim extensions: - coord_norm: desktop-scope support — denormalize_args falls back to screen basis when no window cache (desktop-scope click), normalize_result handles get_desktop_state, ingest_screen_size reads get_desktop_state's screen_width/height - serve.rs: rewrite_coord_desc re-injected at both tools/list sites Install scripts simplified to Rust-only (Swift backend retired upstream). Fix platform-linux cross-compilation: cfg-gate wayland::PORTAL_LIBEI_ENABLED reference in health_report.rs for non-linux hosts. * fix(cua-driver): distinguish desktop-scope from window cache miss in denormalize_args When screenshot_w == 0 (no cached window size), check whether the call has a pid/window_id target before falling back to screen-size basis. Window-scoped clicks with a cache miss now correctly leave coordinates as-is instead of mapping them against the screen dimensions. * fix(cua-driver): use screenshot pixels not logical points for desktop-scope coord basis ingest_screen_size was reading screen_width/height (logical display points from CGDisplayBounds) for get_desktop_state, but the model operates on the screenshot PNG whose dimensions are physical pixels. On Retina displays (2x) this caused every desktop-scope click to land at half the intended position. Read screenshot_width/height instead — the same basis normalize_result rewrites to 1000. Updated test to use realistic Retina values (screen 1920, screenshot 3840) to catch 1:1 masking. * fix(cua-driver): split screen cache into logical (move_cursor) and physical (desktop-scope) The single screen_cache was being written by both get_screen_size (logical points, e.g. 1920x1080) and get_desktop_state (physical pixels, e.g. 3840x2160). Since move_cursor operates in CGEvent screen points (logical) while desktop-scope clicks operate in screenshot pixels (physical), whichever tool ran last would corrupt the other's denormalization basis — causing 2x overshoot or 1/2 offset on Retina. Split into: - SCREEN_SIZE: logical points, written by get_screen_size, consumed by move_cursor (screen_basis=true) - DESKTOP_SCREENSHOT_SIZE: physical pixels, written by get_desktop_state, consumed by desktop-scope click fallback (screenshot_w==0, no pid) Added cross-contamination isolation tests. --------- Co-authored-by: Shaojin Wen <shaojin.wensj@alibaba-inc.com> |
||
|
顾盼
|
adc2bce414
|
feat(cua-driver): vendor qwen-cua-driver with opt-in 0–1000 relative coordinates (#5896)
* feat(cua-driver): vendor trycua/cua driver with 1000-normalized coordinate support
Vendor libs/cua-driver from trycua/cua into packages/cua-driver as the
basis for qwen-code's computer-use backend, adding an opt-in relative
(1000x1000 normalized) coordinate mode for Qwen-VL clients.
- coord_norm.rs: 0-1000 <-> pixel conversion, per-(pid,window_id) size
cache, tools/list description rewrite (TDD, 27 tests)
- ToolRegistry: normalized field + invoke input/output hooks
- protocol.rs: system-instruction coordinate wording switched by mode
- serve.rs: daemon list path description rewrite (input_schema aware)
- main.rs: CUA_DRIVER_RS_COORDINATE_SPACE env seed
Default coordinate_space=pixels => zero behavior change for existing
pixel clients. Set CUA_DRIVER_RS_COORDINATE_SPACE=normalized_1000 to
enable. Excludes rust/target build output.
* feat(cua-driver): make normalized coordinate scale configurable
Add CUA_DRIVER_RS_COORDINATE_SCALE (default 1000) so the normalization
full-scale can absorb the Qwen 999-vs-1000 cookbook ambiguity without a
recompile. norm_to_px/px_to_norm now take an explicit scale; denormalize_args
reads the process-wide COORDINATE_SCALE seeded once at startup from env.
* ci(cua-driver): add cross-platform release workflow for vendored driver
Standalone GitHub Action that builds, signs, and releases the vendored
cua-driver under packages/cua-driver. Adapted from upstream trycua/cua
cd-rust-cua-driver.yml:
- macOS: universal binary (lipo arm64+x86_64), codesigned + notarized into
CuaDriver.app using qwen-code's existing secrets (MAC_CSC_LINK cert +
App Store Connect API key notarization); Developer ID identity is
auto-discovered from the imported cert.
- Linux: x86_64 + arm64, built in debian:11 for a glibc 2.31 floor.
- Windows: x86_64 + arm64, unsigned (no EV cert, matches upstream).
- Release: softprops/action-gh-release on cua-driver-rs-v* tags or manual
dispatch, prerelease.
Triggered by tag push (cua-driver-rs-v*) or workflow_dispatch.
* chore(cua-driver): rebrand vendored driver as qwen-cua-driver
Rename the vendored trycua/cua driver so the fork installs and runs
independently of any upstream trycua install:
- binary cua-driver -> qwen-cua-driver
- bundle CuaDriver.app -> QwenCuaDriver.app
- bundle id com.trycua.driver -> com.qwencode.cua-driver
Updates the cargo/uia manifests, Info.plist, bundle/proxy launch paths,
permission/health-report wording, the install/build scripts, and the
cross-platform release workflow.
* feat(cua-driver): finish relative-coordinate mode — toggle, scale, zoom/move_cursor
- CUA_DRIVER_RS_COORDINATE_SPACE is now a 1/0 toggle (via is_env_truthy);
default off keeps pixel mode byte-identical to upstream.
- Thread CUA_DRIVER_RS_COORDINATE_SCALE through every coordinate surface
(was hardcoded 1000): input denormalization already used it; now the
rewritten screenshot dims, the tool/param descriptions, and the agent
instructions track the configured scale too.
- Normalize zoom (window basis) and move_cursor (screen basis) inputs and
rewrite their descriptions, alongside click/double_click/right_click/drag.
- Fix zoom on downscaled (Retina) windows: apply the get_window_state resize
ratio so the crop lands on the region the agent saw. Normalized mode only;
pixel-mode zoom unchanged.
All coordinate behavior stays gated on the normalized flag, so the default
(pixels) path is unchanged from upstream.
* chore(cua-driver): add upstream-sync script (git subtree unusable here)
`git subtree split --prefix=libs/cua-driver` hangs on a commit deep in
trycua/cua's history, so the subtree add/pull workflow isn't usable for
the vendored driver (and a pull would re-split + re-hang every time).
Add scripts/sync-from-upstream.sh instead: it git-diffs two upstream refs
(never walks the full history, so it dodges the hang), reprefixes the
libs/cua-driver delta to packages/cua-driver, and `git apply --reject`s it
on top of our local changes — conflicts land as *.rej for manual fixup.
Record the vendored version in .vendored-from and document the migration +
sync method in the design doc.
* chore(cua-driver): exclude vendored driver from qwen-code ESLint
The vendored packages/cua-driver tree carries upstream JS (e.g. the
test-harness Electron app) that doesn't follow qwen-code's lint rules and
fails CI. It is not a workspace package (no package.json) and is not
qwen-code TypeScript, so add it to eslint.config.js global ignores —
alongside packages/desktop/** — the standard treatment for vendored code.
* fix(cua-driver): let start_session revive an idle-reaped session
Ports the fix from upstream trycua/cua#2035 into the vendored driver.
When a session is reaped for idleness, a subsequent start_session with the
same id failed instead of resuming it. Revive the ended session in place so
the agent can continue rather than getting a hard error.
* fix(cua-driver): retry daemon socket writes on EAGAIN
Ports the fix from upstream trycua/cua#2036 into the vendored driver.
A non-blocking daemon socket can return EAGAIN/EWOULDBLOCK mid-write when the
peer's receive buffer is momentarily full. The driver treated that as fatal
and dropped the connection. Add a bounded retry/poll loop (mirror of the
read-side socket_io helper) so transient back-pressure no longer kills the
session; only a real timeout or hard error fails the write.
* fix(cua-driver/linux): stop reporting bare "Clicked" for X11 synthetic clicks
Ports the fix from upstream trycua/cua#2025 into the vendored driver.
On X11, clicks are delivered via XSendEvent synthetic events, which many
toolkits (GTK/SDL/Allegro) ignore because send_event is set. The driver still
reported a flat success ("Clicked"), masking that nothing happened. Report
the synthetic-delivery caveat honestly so the agent can fall back instead of
assuming the click landed.
(platform-linux crate is not built on macOS; verified by clean upstream apply
and covered by upstream + release-workflow Linux CI.)
* fix(cua-driver/windows): list empty-/null-title top-level windows
Ports the fix from upstream trycua/cua#2021 into the vendored driver.
list_windows filtered out any top-level window whose title was empty or null,
so legitimate targets (splash screens, some Electron/game windows, tool
windows) were invisible to the agent and unclickable. Include empty-title
windows, using class name / process as a fallback label.
(platform-windows crate is not built on macOS; verified by clean upstream
apply and covered by upstream + release-workflow Windows CI.)
* chore(cua-driver): track cherry-picked upstream PRs; fix vendored-from
The vendored copy is actually at cua-driver-rs-v0.6.7 (workspace version and
all 0.6.7->0.6.8 delta files confirm it), but .vendored-from had drifted to
0.6.8 during an earlier sync-script trial whose code delta was not kept. Left
as-is it would make a future sync diff 0.6.8->newer and silently skip the real
0.6.7->0.6.8 fixes. Correct it back to 0.6.7.
Also record the four not-yet-merged upstream PRs we carry as cherry-picks
(trycua/cua#2021/#2025/#2035/#2036) in .vendored-patches.md, and have
sync-from-upstream.sh point at it so the next sync reconciles them.
* ci(cua-driver): satisfy repo yamllint on the release workflow
The vendored-driver release workflow tripped 114 quoted-strings violations
under the repo's .yamllint (quote-type: single, required). Single-quote all
string scalars to match every other workflow in .github/workflows.
While reformatting, the release-notes body also got its paragraph blank lines
collapsed and still referenced the old CUA_DRIVER_RS_COORDINATE_SPACE=
normalized_1000 value — restore the blank lines and update it to the current
0/1 toggle (default 0 = off; optional CUA_DRIVER_RS_COORDINATE_SCALE=1000).
* chore(cua-driver): sync vendored driver to cua-driver-rs-v0.6.8
First real run of scripts/sync-from-upstream.sh: it 3-way-applied the upstream
0.6.7->0.6.8 delta onto our local fork. 10/12 files applied cleanly; the 2
rejects (install.ps1, _install-rust.sh) were already-applied baked-version
bumps (0.6.6->0.6.7, our copies were already at 0.6.7), i.e. no real conflict.
0.6.8 brings: Wayland input path (platform-linux), linux health_report +
overlay tweaks, a platform-macos build.rs step, and dependency bumps. Version
moved to 0.6.8 across the workspace.
Verified our work survived the sync untouched: the relative-coordinate shim
(coord_norm/protocol) and all four cherry-picked PRs (socket_io/session +
linux/windows) are intact — in particular the 0.6.8 edit to platform-linux
tools/impl_.rs landed alongside our #2025 change with no collision. macOS
cargo check + 132 core tests green. (platform-linux/windows + the binary
integration test build only on their own runners; upstream CI covers those.)
* ci(cua-driver): add a dry_run gate to the release workflow
Mirror the desktop-release / release dry-run pattern: a workflow_dispatch
dry_run boolean input (default true). The cross-platform build + package jobs
always run and upload their artifacts; the GitHub Release job now publishes
only on a tag push or an explicit dry_run=false dispatch.
Lets us rehearse the whole build/package pipeline (dry_run=true, notarize=false)
and inspect the produced artifacts without cutting a release. A branch push
(no tag, not a dispatch) likewise builds without releasing.
|