* feat(verify): sweep sibling shapes, calibrate replays, measure suggested fixes
The verify-pr skill produced a `merge-ready` report on a head that leaks
live HTML. Measured on #8147 at b0ae340b with two control arms and one
treatment arm (real qwen3.8-max-preview, isolated QWEN_HOME each, arms
differing only by this file): both control arms concluded `merge-ready`,
the treatment arm concluded `findings` and shipped a measured two-line
fix.
The decisive arm is the control replicate. It spent the same budget as
the treatment arm and built a stronger instrument — a micromark+GFM
rendering oracle, 23 hostile fixtures, 254 assertions. It built the
HTML-block-times-fence fixture and separately probed code spans, and
never crossed the two axes. Re-running that arm's own oracle on the
fixture it never built renders a live <img> and a live <a href>, so it
would have failed its own guarantee-1 assertion. The miss was which
fixtures the agent thought to construct, which is what these rules
supply.
Added to the A/B method: sweep the sibling shapes a parser-class fix
does not pin; treat untrusted text reaching a parser as a scaling
question and probe it with a timed ladder; audit a stated cost against
the repo's own accounting of the same mechanism; enumerate the unnamed
siblings of an accepted-tradeoff list. Added to shared-state probing: an
instruction in a prompt is not an invariant, and interleavings that
fabricate a result outrank ones that crash. Added to vacuity: run the
mutation in reverse (a suite green with and without a candidate fix pins
nothing along that axis), and read a test's name as a claim about its
fixture. Workflow replays now calibrate against a real production
artifact before their cells are believed, and suggested fixes ship with
collateral measured. The follow-up status enum gains `worsened`, because
a fix can move an accepted tradeoff.
Rules whose effect was not measured are marked as such in the PR
description rather than presented as validated.
* feat(verify): read reports as coverage specs, prove negatives by census
Six more rules, each taken from a hands-on verification round that found
something the skill could not have produced.
From #8037, a fallback parser for XML tool calls: a bug report enumerates
its own cases, and those names are coverage claims the fix inherits —
holding the issue's reported 1,898-character preamble fixed and varying
only the tool showed the guard declining run_shell_command, which the
issue names explicitly, so the fix covered half of what it was filed
against. The same round found one defect with a loud variant (schema
rejects a dropped argument) and a silent one (a truncated file), which
fixes the ordering: rank variants by observability, not blast radius —
the same ordering the concurrency rule already uses. And its harness
replayed the reported wire bytes without reproducing the model-side
degradation that produces them, so Not covered now asks which of the two
a reproduction actually has.
From #8005, a Goal runtime driven through a real TUI: walking the PR's
own Reviewer Test Plan step by step turned an unrunnable step into the
round's sharpest finding, because no code path could insert real user
input into an active turn and the feature's completion criterion needed
exactly that. Proving it took a census rather than a reading — 30
captured verifier payloads carried one evidence kind and never the
required one. And its mutation matrix is the reason survivors are now
gated on a positive control: two mutations survived 429/429 and 326/326,
believable only because a third, expected-caught mutation turned one test
red.
None of these six were measured with an A/B on this file; they are drawn
from rounds that found real defects. The PR description says so.
* fix(verify-pr): address review — calibration fallback, timeout cap, pinning, section placement (#8242)
* fix(scripts): pin bare words in skill assertions, not markdown emphasis markers (#8242)
* fix(verify-pr): address round-3 review — pin payload words, bind timeout, relocate misplaced bullets (#8242)
---------
Co-authored-by: verify <verify@local>
Co-authored-by: qwen-code-dev-bot <qwen-code-dev@service.alibaba.com>
Co-authored-by: Qwen Code Bot <qwen-code-bot@users.noreply.github.com>
bc382c3ff9