qwen-code/scripts/tests
Shaojin Wen 6b0d2bbca0
feat(triage): make /verify evidence screenshots actually possible (#8016)
* feat(triage): make /verify evidence screenshots actually possible

Measured across the 14 real verify reports on this repo: tables appear
in 14/14 (4 to 31 rows), images in 0/14. The hosting machinery has been
complete the whole time — pr-assets branch, PNG magic check, 8 images,
2 MB cap, sanitised names — and it has never once been fed.

Two independent causes, both fixed:

1. The agent physically could not produce a PNG. It runs as `node`
   under `env -i` with HOME pointed at a per-run scratch dir, so
   `playwright install --with-deps chromium` needs an apt it has no
   rights to and would re-download ~170 MB into a directory deleted at
   the end of every run. Chromium is now installed by ROOT in the tools
   step into a shared, world-readable PLAYWRIGHT_BROWSERS_PATH, and the
   agent env carries that path plus QWEN_VERIFY_CHROMIUM=1 — but ONLY
   when the install actually succeeded, so the variable's absence is a
   real signal rather than a stale promise. The install is best-effort:
   a failure warns and the run continues to a text-only report, which
   is what every run has produced anyway.

2. The skill discouraged it. Captures were "Optionally … use them when
   text cannot carry the oracle", with TUI rendering as the only
   example — so an agent verifying a daemon or a CLI correctly
   concluded text carried the oracle and skipped. It now says produce
   them whenever a harness ran, and says WHY: a table is the agent's
   claim about what happened; a capture of the run is a witness that
   the numbers came from a real execution, which is the part a reviewer
   cannot get any other way. It names the three highest-value shots
   (A/B cells side by side, the mutation matrix as printed, raw harness
   output behind a headline number) and forbids `playwright install`
   explicitly, so the failure mode of cause 1 cannot be rediscovered by
   an agent burning budget on it.

Affordable now because #8014 raised the agent budget to 120m; at 20m a
browser download alone was most of the run.

Mutation-verified 4/4: handing the agent the variable unconditionally,
failing the job on a chromium install failure, reverting the skill to
"Optionally", and dropping the do-not-install warning each turn the
test red.

90/90 tests; prettier, eslint, actionlint and shellcheck clean.

* fix(triage): match browser to lockfile and fix pr-assets branch conflict (#8016)

Split the chromium install into system deps (tools step, as root) and
browser binary (new post-checkout step, as node using the checkout's
own Playwright). This eliminates the version pin that produced an
unusable browser tree.

Change evidence hosting from a bare pr-assets leaf branch — which
cannot coexist with the existing pr-assets/* namespace — to per-PR
branches (pr-assets/pr<N>-verify) with orphan-init fallback.

Update terminal-capture skill to warn CI agents away from the
forbidden playwright install command, and close test blind spots
for conditionality (M1b), version matching (M5), and marker
ordering (M6).

* fix(triage): unpin Playwright install-deps so apt list tracks lockfile browser (#8016)

* fix(triage): assert the browser path reaches the agent, not just the flag

Review finding on #8016, verified by mutation before accepting it:
deleting `"PLAYWRIGHT_BROWSERS_PATH=$CHROMIUM_PATH"` from the agent env
left the test green. That is the nastiest arm of this feature — the
agent is TOLD chromium is available via QWEN_VERIFY_CHROMIUM=1, then
Playwright looks in the default ~/.cache/ms-playwright instead of the
shared install and every capture fails, so a successful install still
degrades to a text-only report.

The test now asserts both variables and that BOTH sit behind the
success guard. Mutation-verified 2/2: deleting the path line fails on
the containment assertion, and hoisting it outside the guard fails on
the ordering one (`expected 21928 to be less than 21834`).

Not changed, because the other two findings do not describe this
codebase — replies on the threads carry the evidence.

90/90 tests; prettier and eslint clean.

* fix(triage): cover orphan-init delivery and delete verify asset branches

Two review findings on #8016, both about the per-PR branch scheme the
autofix bot introduced on this branch (029beba3ef), and both correct.

**Orphan-init's success path had no test.** The existing scenarios both
take the clone-failed branch, but one seeded the branch (so clone
succeeds) and the other points at a non-repo (so the push fails too) —
neither proves orphan-init can DELIVER. A third scenario now points at
a valid bare remote with the branch deleted first: clone fails,
orphan-init runs, and the images land. It also asserts the branch has
exactly one commit, so a graft onto unrelated history would fail rather
than pass as "orphan". Without it, a dropped `git init` or
`remote add origin` would silently discard every image on every PR's
first verification run.

Mutation-verified: dropping `remote add origin` and dropping `git init`
each kill it. A third mutation — pointing `checkout --orphan` at the
wrong branch name — SURVIVES, and correctly: the push is
`HEAD:$assets_branch`, which names its destination explicitly, so the
local branch name cannot affect delivery. Recorded as a non-defect
rather than papered over with a test that pins an irrelevant detail.

**Per-PR asset branches were never deleted.** `pr-assets/*` had exactly
one producer and one deleter; the verify lane became a second producer
without being added, so every verified PR would leave a single-commit
branch in the base repo permanently. The cleanup workflow now deletes
both refs, is renamed to match what it does, and carries a note that
every new `pr-assets/*` producer must be added — a branch nothing
deletes is permanent.

The loop deliberately drops `set -e`: most PRs produce neither branch,
so a 404 on the first must not skip the second. A real delete failure
still warns and sets a non-zero exit. The new test executes the script
against a stubbed `gh` where the visuals branch 404s, and asserts the
verify branch is still deleted — which a `set -e` version would fail.

92/92 tests; prettier and eslint clean.

* fix(triage): match browser install to the harness Playwright and re-arm the strip guard (#8016)

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(triage): gate chromium marker on deps and resolve Playwright from the harness (#8016)

Address review feedback on the verify evidence-browser path:

- Resolve the Playwright CLI from the capture harness's own directory
  (require.resolve of the exported package.json, then join cli.js) instead
  of assuming npm hoists playwright to the root, so the installed binary
  tracks the harness's import even if the hoist layout changes. cli.js is
  not in the package's exports map, which is why a direct path was used
  before; resolving the exported package.json keeps that bypass while
  removing the hoist assumption.
- Gate the verify-chromium-path marker on a new verify-chromium-deps-ok
  marker written by the tools step. apt and the Playwright CDN are
  independent servers with no shared success signal, so a binary download
  alone must not promise chromium to the agent when system deps failed.
- Add a delete-failure case to the PR asset cleanup test so the status=1
  and exit path is executed, not just statically asserted.

* fix(triage): strip cache creds from browser install and guard cli.js resolution (#8016)

The evidence-browser install runs the PR-resolved Playwright CLI but did
not drop ACTIONS_RUNTIME_TOKEN/URL/CACHE_URL the way the adjacent prepare
and agent steps do, leaving runner-injected cache credentials readable by
PR-controlled code. Strip them in the same env -u chain. Also replace the
string-only guard on the require.resolve + cli.js join with an execution
guard that resolves the expression against the installed tree, so a
Playwright bump that relocates cli.js fails the test instead of silently
degrading evidence capture at runtime.

---------

Co-authored-by: wenshao <wenshao@example.com>
Co-authored-by: qwen-code-dev-bot <qwen-code-dev@service.alibaba.com>
Co-authored-by: qwen-code-ci-bot <qwen-code-ci-bot@users.noreply.github.com>
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Co-authored-by: qwen-code-dev-bot <qwen-code-dev-bot@users.noreply.github.com>
2026-07-30 05:37:42 +00:00
..
ai-release-notes-workflow.test.js fix(release): raise model timeouts and shrink batch size for slow networks (#8007) 2026-07-29 22:38:35 +00:00
audit-runtime-critical.test.js ci: keep the critical-audit gate honest when npm cannot answer (#7743) 2026-07-26 06:59:13 +00:00
build-and-publish-image-workflow.test.js ci(autofix): restore sandbox image flow (#6261) 2026-07-03 15:30:58 +00:00
check-build-status.test.js fix(review): report what the transcripts prove; build the roster in one call (#7033) 2026-07-18 00:43:57 +00:00
check-i18n.test.ts fix(cli): localize approval mode UI labels (#6592) 2026-07-11 00:07:03 +00:00
chrome-extension-package.test.js fix(ci): cover release integration regressions (#5994) 2026-06-29 11:54:11 +00:00
ci-flaky-rerun-workflow.test.js fix(ci): stop a slow patrol classifier from killing every flaky rerun (#7358) 2026-07-21 02:33:51 +00:00
ci-flaky-rerun.test.js feat(ci): auto-open a deflake fix issue for confirmed flaky tests (#7231) 2026-07-19 16:49:29 +00:00
clean-package-build-artifacts.test.js test(core): stabilize file history eviction test (#6637) 2026-07-10 06:39:52 +00:00
cli-entry.test.js fix(cli): update npm installs safely in background (#7322) 2026-07-21 06:32:24 +00:00
comment-attachment-guard-workflow.test.js ci: add suspicious comment attachment guard (#6599) 2026-07-10 09:40:54 +00:00
dev.test.js fix(review): report what the transcripts prove; build the roster in one call (#7033) 2026-07-18 00:43:57 +00:00
e2e-workflow.test.js fix(ci): keep the post-merge E2E signal on main alive (#7795) 2026-07-28 11:54:52 +00:00
generate-changelog.test.js feat(release): generate AI-assisted release notes (#6756) 2026-07-12 13:00:22 +00:00
generate-release-notes.test.js fix(scripts): slim release-note model prompts and log request timing (#7941) 2026-07-28 15:32:56 +00:00
get-release-version-python-sdk.test.js feat(sdk-python): add network timeouts to release version helper (#3833) 2026-05-05 19:25:00 +08:00
get-release-version.test.js fix(release): bump preview base past published stable (#7978) 2026-07-29 23:21:00 +00:00
install-script.test.js fix(cli): avoid updating active CLI processes (#6874) 2026-07-15 00:33:17 +00:00
issue-triage-ownership-workflow.test.js fix(ci): consolidate issue triage ownership (#7180) 2026-07-19 10:58:40 +00:00
lint.test.js revert: remove local PR verification gate (#7031) 2026-07-16 11:24:38 +00:00
main-ci-failure-issue-workflow.test.js fix(ci): keep the post-merge E2E signal on main alive (#7795) 2026-07-28 11:54:52 +00:00
no-ak-integration-ci.test.js feat(core): propagate trusted daemon invocation context (#7279) 2026-07-23 06:49:11 +00:00
package-assets.test.js fix(release): raise prepared package size limit to 96 MB (#6687) (#6691) 2026-07-11 01:01:59 +00:00
package-scripts.test.js fix(release): bump preview base past published stable (#7978) 2026-07-29 23:21:00 +00:00
pr-force-push-reminder-workflow.test.js ci(autofix): run agents on dedicated ECS runners (#6207) 2026-07-03 07:40:07 +00:00
pr-self-report-label.test.js ci: label a PR that closes an issue its own author opened (#7630) 2026-07-24 05:50:47 +00:00
qwen-autofix-workflow.test.js feat(autofix): back off scan inspection of idle candidates (#8049) 2026-07-30 04:26:43 +00:00
qwen-fleet-shepherd-workflow.test.js feat(autofix): label-driven takeover and release; fix forced-dispatch green no-op (#7165) 2026-07-19 07:35:13 +00:00
qwen-pr-review-workflow.test.js fix(review): give the review retry the remaining time budget (#7852) 2026-07-28 05:29:20 +00:00
qwen-resolve-workflow.test.js fix(ci): serialise the two workflows that push to a PR head branch (#7392) 2026-07-21 07:40:56 +00:00
qwen-triage-finalize-workflow.test.js fix(ci): rename triage status marker to avoid duplicate-guard collision (#7723) 2026-07-26 15:51:22 +00:00
qwen-triage-workflow.test.js feat(triage): make /verify evidence screenshots actually possible (#8016) 2026-07-30 05:37:42 +00:00
release-helpers.test.js refactor: extract shared release helper utilities (#3834) 2026-05-05 10:15:17 +08:00
release-sdk-workflow.test.js fix(ci): skip empty SDK release PR (#6861) 2026-07-14 13:19:42 +00:00
sandbox-command.test.js fix(scripts): avoid shell injection in sandbox command detection (#6108) 2026-07-01 16:20:40 +08:00
sdk-node-exporter-stub.test.js perf(telemetry): lazy-load the SDK and split OTLP exporter chains by protocol (#7276) 2026-07-21 07:35:30 +00:00
serve-fast-path-bundle-check.test.js perf(cli): replace comment-json settings parser (#7747) 2026-07-26 14:42:51 +00:00
start.test.js fix(review): report what the transcripts prove; build the roster in one call (#7033) 2026-07-18 00:43:57 +00:00
test-setup.ts feat(installer): add standalone archive installation (#3776) 2026-05-11 13:25:48 +08:00
update-ecs-runner-qwen-workflow.test.js fix(ci): update the Qwen binary used by ECS runners (#8000) 2026-07-29 05:24:56 +00:00
upload-aliyun-oss-assets.test.js feat(installer): add standalone hosted install and uninstall flow (#3828) 2026-05-21 11:57:10 +08:00
vitest.config.ts refactor(auth): unify provider config in core, simplify /auth as "Connect a Provider" (#4287) 2026-05-20 23:48:52 +08:00
workspaces.test.js feat(desktop): Add desktop app package with Qwen ACP SDK integration (#3778) 2026-06-11 21:57:20 +08:00