mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-08-25 16:44:36 +00:00
fix(autofix): stop counting idle timeouts toward the timeout cap (#9673)
* fix(autofix): stop counting idle timeouts toward the timeout cap TIMEOUT_WINDOW_CAP exists to stop a PR that is too big to finish a round inside the agent's time budget, and its remedy says exactly that: split the PR or raise the budget. A silent-sandbox (idle) timeout is a different failure entirely — the idle watchdog kills the round because the sandbox produced no output at all, so no budget increase cures it and nothing about the PR caused it. Counting a failure whose prescribed remedy is inapplicable is what parked healthy PRs: over the 14 days to 2026-08-21 idle rounds were 58 of 119 timeouts, every one of the 51 windows that tripped this cap tripped it at exactly three, and 9 of the 12 PRs then carrying autofix/needs-human had been stopped here — #8332 at 24 rounds, #8368 at 28, #8276 at 16, all still pushing rounds when they were parked. Gate the cap on budget timeouts alone. A persistently wedged sandbox stays bounded, because an idle round pushes nothing and resets no streak, so CONSECUTIVE_FAILURE_CAP still terminates it; what no longer terminates is idle rounds interleaved with real progress, where the PR is not stuck and the runner is. Idle rounds stay visible through a job-log warning, which reaches whoever owns the runners without spending a comment on someone's PR. Two consequences inside the census. The idle needle became the full emitted headline prefix rather than a bare substring, because the count is now subtracted and must be a subset of the total — a loose needle could match provider error text on the same line and drive the difference negative. And the all-idle remedy branch is gone as unreachable: the guard now fires only when budget timeouts alone reach the cap, so a counted window always holds more of them than idle ones. * fix(autofix): emit idle census warning on terminal runs (#9673) Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(autofix): pin idle-census counts and extend idle exclusion to prepare census (#9673) * test(autofix): reuse pinned IDLE_HEAD fixture in timeout-census replay (#9673) --------- Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> Co-authored-by: qwen-code-dev-bot <qwen-code-dev-bot@users.noreply.github.com>
This commit is contained in:
Shaojin Wen
GitHub
parent
7703d1c310
commit
f89335e453
3 changed files with 378 additions and 79 deletions
66
.github/workflows/qwen-autofix.md
vendored
66
.github/workflows/qwen-autofix.md
vendored
|
|
@ -124,6 +124,7 @@ task-oriented guides — what a maintainer types and what happens next — see:
|
|||
- [70. review-address · Report dry-run / failure — -c drops any partial multi-byte sequence a byte-level head -c may have split, so the…](#af-070)
|
||||
- [71. review-address · Report dry-run / failure — Bilingual companion. Repo convention is English first, Chinese in a collapsed <details>.…](#af-071)
|
||||
- [72. review-address · Report dry-run / failure — Flip the status comment out of "working" so a finished round never leaves a live-looking…](#af-072)
|
||||
- [73. review-address · Report dry-run / failure — Idle (silent-sandbox) timeouts are EXCLUDED from the cumulative timeout cap.…](#af-073)
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -2090,3 +2091,68 @@ publishing" and report a successful round as a failed one. An empty
|
|||
'stale' (prepare itself crashed) still finalises — that IS this job's
|
||||
round, and it is exactly the case that must not stay "working".
|
||||
```
|
||||
|
||||
<a id="af-073"></a>
|
||||
|
||||
### 73. review-address · Report dry-run / failure — Idle (silent-sandbox) timeouts are EXCLUDED from the cumulative timeout cap.
|
||||
|
||||
In `review-address` · `Report dry-run / failure`.
|
||||
|
||||
```text
|
||||
TIMEOUT_WINDOW_CAP exists to stop a PR that is too big to finish a
|
||||
round inside the agent's time budget; its remedy says so ("split or
|
||||
reduce the PR, or raise the agent time budget AND its step backstop").
|
||||
An idle timeout is a different failure entirely: run-agent.mjs's idle
|
||||
watchdog kills the round after QWEN_IDLE_TIMEOUT_MS (20m) because the
|
||||
sandbox produced no output at all — the four observed hangs (#8663 x2,
|
||||
#8761 r3, #8763 r4) each printed their last byte at docker container
|
||||
entry and then sat silent. Nothing about the PR caused it, and the
|
||||
breaker's own headline already told the reader that "no budget increase
|
||||
can cure" it. Counting a failure whose prescribed remedy is
|
||||
inapplicable is what parked healthy PRs.
|
||||
|
||||
Measured on 2026-08-21, over the preceding 14 days: 119 timeouts, of
|
||||
which 58 (49%) were idle. 51 windows tripped this cap, every one of
|
||||
them at exactly N=3. Of the 12 open PRs then carrying
|
||||
autofix/needs-human, 9 had been stopped here — #8332 at 24 rounds,
|
||||
#8368 at 28, #8276 at 16, all still producing pushed rounds when they
|
||||
were parked. With idle rounds counted, the fleet timeout rate was
|
||||
8.5% per round, so a window accumulated three of them in ~35 rounds by
|
||||
arithmetic alone, independent of whether the PR was stuck. Excluding
|
||||
idle drops the rate to 4.3%, which needs ~69 rounds — beyond the
|
||||
deepest window ever observed (22/100).
|
||||
|
||||
The escape hatch that makes the exclusion safe: an idle round pushes
|
||||
nothing and matches none of CONSEC_FAIL's streak-reset needles
|
||||
("Addressed the latest review feedback", "no changes needed", "AutoFix
|
||||
could not start", "updated a stale base"), so a persistently wedged
|
||||
sandbox still terminates the PR at CONSECUTIVE_FAILURE_CAP. What no
|
||||
longer terminates it is idle rounds INTERLEAVED with real progress —
|
||||
which is the intended change: that PR is not stuck, the runner is.
|
||||
|
||||
Two consequences inside the block. IDLE_N's needle became the full
|
||||
emitted headline prefix ('AutoFix ran out of time before finishing
|
||||
(idle-timeout') rather than a bare 'idle-timeout' substring: IDLE_N is
|
||||
now subtracted from TIMEOUT_N, so it MUST be a subset of it, and a
|
||||
loose needle could otherwise match provider error text that
|
||||
API_ERROR_DETAIL puts on the same first line and drive the difference
|
||||
negative. And the all-idle remedy branch is gone as unreachable: the
|
||||
guard now fires only when BUDGET_TIMEOUT_N alone reaches the cap, so a
|
||||
tripped window always holds at least TIMEOUT_WINDOW_CAP genuine budget
|
||||
timeouts — idle rounds can outnumber budget ones in it, but the budget
|
||||
remedy applies because those budget timeouts exist, not because they
|
||||
are the majority.
|
||||
|
||||
Idle rounds stay visible through a job-log ::warning:: rather than a PR
|
||||
comment — the signal belongs to whoever owns the runners, and infra
|
||||
noise should not spend a comment on someone's PR. The census and its
|
||||
warning run outside the cap's terminal guard: the all-idle shape stops
|
||||
via the consecutive breaker with that breaker's headline, and the
|
||||
terminal run's log is exactly where the wedged runner must be named.
|
||||
|
||||
The same exclusion applies to the prepare step's PRIOR_TIMEOUTS census
|
||||
(af-049): its budget warning tells the agent to narrow scope — the
|
||||
budget remedy again — and an idle round never exhausted any budget, so
|
||||
it must not steer the narrowing. Idle rounds are excluded there with
|
||||
the same needle the cap census uses.
|
||||
```
|
||||
|
|
|
|||
73
.github/workflows/qwen-autofix.yml
vendored
73
.github/workflows/qwen-autofix.yml
vendored
|
|
@ -245,7 +245,9 @@ env:
|
|||
# pushed rounds in between, so the consecutive cap never fired and the PR
|
||||
# kept walking into the same wall; #7846 the same, twice. Counted over the
|
||||
# current counting window (window-scoped like every other census), so a
|
||||
# re-arm clears it along with the round counter.
|
||||
# re-arm clears it along with the round counter. Counts BUDGET timeouts
|
||||
# only: silent-sandbox (idle) timeouts are infra, not PR size, and are
|
||||
# excluded — see qwen-autofix.md#af-073.
|
||||
TIMEOUT_WINDOW_CAP: '3'
|
||||
# Do not claim more issues when too many existing autofix PRs are still open.
|
||||
MAX_OPEN_AUTOFIX_PRS: '5'
|
||||
|
|
@ -4991,6 +4993,9 @@ jobs:
|
|||
# Time-budget exhaustions SINCE THE LAST SUCCESSFUL ROUND mean
|
||||
# the standard address-everything prompt is not converging at
|
||||
# Full rationale → qwen-autofix.md#af-049
|
||||
# Idle (silent-sandbox) timeouts are excluded like in the cap
|
||||
# census: the narrowing advice targets budget exhaustion, and an
|
||||
# infra-killed round never had any budget to exhaust (af-073).
|
||||
PRIOR_TIMEOUTS="$(jq -r --arg ab "${AUTOFIX_BOT}" --arg key "${LIVE_REARM_KEY}" '
|
||||
[ .[] | select((.user.login // "") == $ab)
|
||||
| select((.body // "") | contains("<!-- autofix-eval "))
|
||||
|
|
@ -4999,7 +5004,7 @@ jobs:
|
|||
] | sort_by(.created_at)
|
||||
| map((.body | gsub("\r"; "") | split("\n")[0]))
|
||||
| (map(test("Addressed the latest review feedback|no changes needed")) | rindex(true) // -1) as $lastok
|
||||
| [ .[($lastok + 1):][] | select(contains("AutoFix ran out of time before finishing")) ] | length' "${WORKDIR}/ic.json" 2> /dev/null || true)"
|
||||
| [ .[($lastok + 1):][] | select(contains("AutoFix ran out of time before finishing") and (contains("AutoFix ran out of time before finishing (idle-timeout") | not)) ] | length' "${WORKDIR}/ic.json" 2> /dev/null || true)"
|
||||
if [[ "${PRIOR_TIMEOUTS}" -ge 1 ]]; then
|
||||
echo
|
||||
echo '## Budget warning: previous round(s) ran out of time'
|
||||
|
|
@ -6674,9 +6679,31 @@ jobs:
|
|||
# full agent budget with nothing to show (observed on #7929:
|
||||
# three timeouts with successes in between; #7846 twice). The
|
||||
# census reuses PRIOR_HEADS, so it is window-scoped exactly
|
||||
# like the consecutive one and a re-arm clears it. Only
|
||||
# overrides a would-be RETRY: a round already terminal keeps
|
||||
# its own headline (the consecutive breaker included).
|
||||
# like the consecutive one and a re-arm clears it. Only the
|
||||
# cap gate below overrides a would-be RETRY: a round already
|
||||
# terminal keeps its own headline (the consecutive breaker
|
||||
# included). The idle census and its warning run OUTSIDE that
|
||||
# guard: the all-idle shape terminates via the consecutive
|
||||
# breaker above, and that terminal run's job log is exactly
|
||||
# where the wedged runner must be named.
|
||||
# Idle (silent-sandbox) timeouts are EXCLUDED from this cap:
|
||||
# a wedged runner/docker is not this PR being too big, an idle
|
||||
# round dies at QWEN_IDLE_TIMEOUT_MS having produced no byte
|
||||
# (a fraction of a real round), and no budget increase cures
|
||||
# one — so this breaker's remedy does not apply to them. A
|
||||
# persistently wedged sandbox stays bounded by
|
||||
# CONSECUTIVE_FAILURE_CAP, which an idle round DOES feed.
|
||||
# Full rationale → qwen-autofix.md#af-073
|
||||
IDLE_N="$(grep -c 'AutoFix ran out of time before finishing (idle-timeout' <<< "${PRIOR_HEADS}" || true)"
|
||||
if [[ "${AGENT_TIMEOUT:-}" == 'idle-timeout'* ]]; then
|
||||
IDLE_N=$(( IDLE_N + 1 ))
|
||||
fi
|
||||
# Excluding idle from the cap must not hide it. The job log is
|
||||
# the right surface: it reaches the operator without spending a
|
||||
# PR comment on infra noise.
|
||||
if [[ "${IDLE_N}" -gt 0 ]]; then
|
||||
echo "::warning::#${PR}: ${IDLE_N} silent-sandbox (idle) timeout(s) this counting window — excluded from the ${TIMEOUT_WINDOW_CAP}-timeout cap; check the sandbox image and the runner docker daemon"
|
||||
fi
|
||||
if [[ "${MARK_ROUND}" != "${MAX_ROUNDS}" ]]; then
|
||||
# Needle matches the emitted headline verbatim — first lines
|
||||
# can embed provider error text (API_ERROR_DETAIL puts up to
|
||||
|
|
@ -6686,14 +6713,12 @@ jobs:
|
|||
if [[ -n "${AGENT_TIMEOUT:-}" ]]; then
|
||||
TIMEOUT_N=$(( TIMEOUT_N + 1 ))
|
||||
fi
|
||||
# Idle (silent-sandbox) timeouts share the census — they burn
|
||||
# the same full budget — but no budget increase cures them, so
|
||||
# when the window contains any, the breaker says so.
|
||||
IDLE_N="$(grep -c 'idle-timeout' <<< "${PRIOR_HEADS}" || true)"
|
||||
if [[ "${AGENT_TIMEOUT:-}" == 'idle-timeout'* ]]; then
|
||||
IDLE_N=$(( IDLE_N + 1 ))
|
||||
fi
|
||||
if [[ "${TIMEOUT_N}" -ge "${TIMEOUT_WINDOW_CAP}" ]]; then
|
||||
# IDLE_N's needle is TIMEOUT_N's needle extended with the
|
||||
# idle cause's opening token — every line matching it also
|
||||
# matches TIMEOUT_N's, so IDLE_N can never exceed TIMEOUT_N
|
||||
# and the subtraction below can never go negative.
|
||||
BUDGET_TIMEOUT_N=$(( TIMEOUT_N - IDLE_N ))
|
||||
if [[ "${BUDGET_TIMEOUT_N}" -ge "${TIMEOUT_WINDOW_CAP}" ]]; then
|
||||
MARK_ROUND="${MAX_ROUNDS}"
|
||||
# The headline states what the census MEASURED — the
|
||||
# window's cumulative count — not "stopped after N
|
||||
|
|
@ -6701,23 +6726,19 @@ jobs:
|
|||
# failed differently (a gate rejection landing on a window
|
||||
# that already carries the cap — the exact rollout state
|
||||
# of #7929/#7846).
|
||||
# No all-idle branch here: the guard above only fires when
|
||||
# BUDGET_TIMEOUT_N alone reaches the cap, so a tripped
|
||||
# window always holds at least TIMEOUT_WINDOW_CAP genuine
|
||||
# budget timeouts — idle rounds can outnumber budget ones
|
||||
# in it, but the budget remedy is always the right one.
|
||||
IDLE_CLAUSE=''
|
||||
IDLE_CLAUSE_ZH=''
|
||||
if [[ "${IDLE_N}" -gt 0 ]]; then
|
||||
IDLE_CLAUSE=" ${IDLE_N} of those were silent-sandbox (idle) timeouts that no budget increase can cure — investigate the sandbox image and runner docker daemon for those."
|
||||
IDLE_CLAUSE_ZH="其中 ${IDLE_N} 次是静默 sandbox（idle）超时，提高预算也治不了 —— 请针对这些排查 sandbox 镜像与 runner 的 docker daemon。"
|
||||
IDLE_CLAUSE=" The window also holds ${IDLE_N} silent-sandbox (idle) timeout(s), which no budget increase can cure and which do NOT count toward this cap — investigate the sandbox image and runner docker daemon separately."
|
||||
IDLE_CLAUSE_ZH="本窗口另有 ${IDLE_N} 次静默 sandbox（idle）超时，提高预算也治不了，且不计入本上限 —— 请另行排查 sandbox 镜像与 runner 的 docker daemon。"
|
||||
fi
|
||||
# Mirror the round-level split: when EVERY counted timeout
|
||||
# was idle, the closing remedy must not prescribe the
|
||||
# budget increase the clause above just declared useless.
|
||||
REMEDY='split or reduce the PR (or raise the agent time budget AND its step backstop together)'
|
||||
REMEDY_ZH='拆分或缩减该 PR（或同时提高 agent 时间预算与其步骤兜底）'
|
||||
if [[ "${IDLE_N}" -ge "${TIMEOUT_N}" ]]; then
|
||||
REMEDY='investigate the sandbox image and runner docker daemon'
|
||||
REMEDY_ZH='排查 sandbox 镜像与 runner 的 docker daemon'
|
||||
fi
|
||||
HEADLINE="🤖 AutoFix stopped: this counting window now contains ${TIMEOUT_N} time-budget exhaustions (pushed rounds in between included; this round itself may have failed differently). That is ${TIMEOUT_N} full agent runs that pushed nothing.${IDLE_CLAUSE} A human should ${REMEDY}, then comment \`${RETRY_COMMAND}\` to re-arm. Until then future scans will skip this PR."
|
||||
HEADLINE_ZH="🤖 AutoFix 已停止：当前计数窗口内已累计 ${TIMEOUT_N} 次时间预算耗尽（含其间推送过的轮次；本轮本身可能以别的方式失败）。即 ${TIMEOUT_N} 次完整 agent 运行没有推送任何内容。${IDLE_CLAUSE_ZH}应由人工${REMEDY_ZH}，然后评论 \`${RETRY_COMMAND}\` 重新武装。在此之前，后续扫描将跳过本 PR。"
|
||||
HEADLINE="🤖 AutoFix stopped: this counting window now contains ${BUDGET_TIMEOUT_N} agent time-budget exhaustions (pushed rounds in between included; this round itself may have failed differently). That is ${BUDGET_TIMEOUT_N} full agent runs that pushed nothing.${IDLE_CLAUSE} A human should split or reduce the PR (or raise the agent time budget AND its step backstop together), then comment \`${RETRY_COMMAND}\` to re-arm. Until then future scans will skip this PR."
|
||||
HEADLINE_ZH="🤖 AutoFix 已停止：当前计数窗口内已累计 ${BUDGET_TIMEOUT_N} 次时间预算耗尽（含其间推送过的轮次；本轮本身可能以别的方式失败）。即 ${BUDGET_TIMEOUT_N} 次完整 agent 运行没有推送任何内容。${IDLE_CLAUSE_ZH}应由人工拆分或缩减该 PR（或同时提高 agent 时间预算与其步骤兜底），然后评论 \`${RETRY_COMMAND}\` 重新武装。在此之前，后续扫描将跳过本 PR。"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
|
|
|||
|
|
@ -345,6 +345,16 @@ function runDevelopIssue(dir, stub) {
|
|||
]);
|
||||
}
|
||||
|
||||
// The idle-timeout sentinel detail exactly as run-agent.mjs's template
|
||||
// emits it (with the observed 20-minute window filled in), and the retry
|
||||
// headline the report step builds from it via
|
||||
// CAUSE="ran out of time before finishing (${AGENT_TIMEOUT})".
|
||||
// Single-sourced so the composition test can tie the runner's emission to
|
||||
// these fixtures and to the workflow's classification needles.
|
||||
const IDLE_NOW =
|
||||
'idle-timeout (no output for 1200000ms — the sandbox likely hung at startup)';
|
||||
const IDLE_HEAD = `🤖 AutoFix ran out of time before finishing (${IDLE_NOW}) (attempt 2/100) — it will retry on the next scan.`;
|
||||
|
||||
describe('qwen-autofix workflow', () => {
|
||||
it('keeps ECS issue autofix limited to forced and ready-for-agent issues', () => {
|
||||
expect(workflow).toContain('autofixTier');
|
||||
|
|
@ -6792,6 +6802,44 @@ exit 1
|
|||
K,
|
||||
),
|
||||
).toBe('2');
|
||||
// Idle (silent-sandbox) rounds are EXCLUDED from this census exactly
|
||||
// like from the cap: the narrowing advice targets budget exhaustion,
|
||||
// and the idle watchdog killed the round before any budget was
|
||||
// exhausted (af-073). One pushed round plus one pure idle round must
|
||||
// count zero — pre-fix the census reported 1 and told the agent to
|
||||
// narrow scope for a wedged runner. Deleting the exclusion's jq clause
|
||||
// flips both back to counting idle rounds and must fail here.
|
||||
expect(
|
||||
runCensus(
|
||||
[
|
||||
mk(PUSH_HEADLINE, K, '2026-07-29T04:00:00Z'),
|
||||
mk(IDLE_HEAD, K, '2026-07-29T05:00:00Z'),
|
||||
],
|
||||
K,
|
||||
),
|
||||
).toBe('0');
|
||||
expect(
|
||||
runCensus(
|
||||
[
|
||||
mk(PUSH_HEADLINE, K, '2026-07-29T04:00:00Z'),
|
||||
mk(IDLE_HEAD, K, '2026-07-29T05:00:00Z'),
|
||||
mk(IDLE_HEAD, K, '2026-07-29T06:00:00Z'),
|
||||
],
|
||||
K,
|
||||
),
|
||||
).toBe('0');
|
||||
// ...but budget timeouts still count beside idle rounds, and an idle
|
||||
// round between two budget ones is not a success, so it resets nothing.
|
||||
expect(
|
||||
runCensus(
|
||||
[
|
||||
mk(TIMEOUT_HEADLINE, K, '2026-07-29T04:00:00Z'),
|
||||
mk(IDLE_HEAD, K, '2026-07-29T05:00:00Z'),
|
||||
mk(TIMEOUT_HEADLINE, K, '2026-07-29T06:00:00Z'),
|
||||
],
|
||||
K,
|
||||
),
|
||||
).toBe('2');
|
||||
// Legacy pre-takeover markers (no win= field) count under key 'none' —
|
||||
// the common real case: a PR that timed out before any re-arm.
|
||||
expect(
|
||||
|
|
@ -6831,6 +6879,21 @@ exit 1
|
|||
expect(prepareBranchAndFeedbackStep).toContain(
|
||||
'contains("AutoFix ran out of time before finishing")',
|
||||
);
|
||||
// The idle exclusion reuses the cap census's IDLE_N needle VERBATIM —
|
||||
// a divergent token would classify the same headline differently in
|
||||
// the two censuses.
|
||||
expect(prepareBranchAndFeedbackStep).toContain(
|
||||
'and (contains("AutoFix ran out of time before finishing (idle-timeout") | not)',
|
||||
);
|
||||
const prepareIdleNeedle = prepareBranchAndFeedbackStep.match(
|
||||
/and \(contains\("([^"]+)"\) \| not\)/,
|
||||
)?.[1];
|
||||
const capIdleNeedle = reviewAddressReportStep.match(
|
||||
/IDLE_N="\$\(grep -c '([^']+)'/,
|
||||
)?.[1];
|
||||
expect(prepareIdleNeedle).toBeTruthy();
|
||||
expect(capIdleNeedle).toBeTruthy();
|
||||
expect(prepareIdleNeedle).toBe(capIdleNeedle);
|
||||
expect(reviewAddressReportStep).toContain(
|
||||
'CAUSE="ran out of time before finishing (${AGENT_TIMEOUT})"',
|
||||
);
|
||||
|
|
@ -14992,13 +15055,7 @@ exit 1
|
|||
reviewAddressReportStep.match(/^\s*HEADLINE_ZH="/gm) ?? [];
|
||||
expect(headlineAssignments.length).toBeGreaterThan(0);
|
||||
expect(headlineZhAssignments).toHaveLength(headlineAssignments.length);
|
||||
for (const name of [
|
||||
'CAUSE',
|
||||
'LAST_FIX',
|
||||
'GATE_CLAUSE',
|
||||
'IDLE_CLAUSE',
|
||||
'REMEDY',
|
||||
]) {
|
||||
for (const name of ['CAUSE', 'LAST_FIX', 'GATE_CLAUSE', 'IDLE_CLAUSE']) {
|
||||
const en =
|
||||
reviewAddressReportStep.match(new RegExp(`^\\s*${name}=`, 'gm')) ?? [];
|
||||
const zh =
|
||||
|
|
@ -15037,6 +15094,9 @@ exit 1
|
|||
'轮未能推送任何内容',
|
||||
],
|
||||
['HEADLINE', 'time-budget exhaustions', '次时间预算耗尽'],
|
||||
// The cap remedy is inlined in HEADLINE/HEADLINE_ZH (the REMEDY
|
||||
// variables are gone) — its EN/ZH pairing stays pinned here.
|
||||
['HEADLINE', 'split or reduce the PR', '拆分或缩减该 PR'],
|
||||
[
|
||||
'HEADLINE',
|
||||
'deferred this item to a human under instruction',
|
||||
|
|
@ -15098,8 +15158,12 @@ exit 1
|
|||
'自身本轮之前的代码需要处理',
|
||||
],
|
||||
['IDLE_CLAUSE', 'no budget increase can cure', '提高预算也治不了'],
|
||||
['REMEDY', 'split or reduce the PR', '拆分或缩减该 PR'],
|
||||
['REMEDY', 'investigate the sandbox image', '排查 sandbox 镜像'],
|
||||
['IDLE_CLAUSE', 'do NOT count toward this cap', '不计入本上限'],
|
||||
[
|
||||
'IDLE_CLAUSE',
|
||||
'investigate the sandbox image and runner docker daemon separately',
|
||||
'请另行排查 sandbox 镜像与 runner 的 docker daemon',
|
||||
],
|
||||
]) {
|
||||
expect(
|
||||
reviewAddressReportStep,
|
||||
|
|
@ -15643,8 +15707,7 @@ exit 1
|
|||
// advice: more minutes cannot cure a sandbox that produced nothing.
|
||||
const idleCapped = run({
|
||||
OUTCOME: 'failed',
|
||||
AGENT_TIMEOUT:
|
||||
'idle-timeout (no output for 1200000ms — the sandbox likely hung at startup)',
|
||||
AGENT_TIMEOUT: IDLE_NOW,
|
||||
ROUND: '4',
|
||||
});
|
||||
expect(idleCapped).toContain('this was the last automatic attempt');
|
||||
|
|
@ -15881,7 +15944,7 @@ exit 1
|
|||
'bash',
|
||||
[
|
||||
'-c',
|
||||
`set -uo pipefail\nWORKDIR='${dir}'\nMARK_ROUND=${markRound}\nMAX_ROUNDS=100\nCONSECUTIVE_FAILURE_CAP=${cap}\nTIMEOUT_WINDOW_CAP=${timeoutCap}\nAGENT_TIMEOUT='${agentTimeout}'\nCONSEC_FAIL=0\nREPO=o/r\nPR=1\nAUTOFIX_BOT=qwen-code-dev-bot\nRETRY_COMMAND='@qwen-code /retry'\nAPI_ERROR_DETAIL='${apiErrorDetail}'\nAPI_ERROR_KIND='${apiErrorKind}'\nPREPARE_OUTCOME='${prepareOutcome}'\nSTALE_BASE_RETRY='${staleBaseRetry}'\n${window !== undefined ? `WINDOW='${window}'\n` : ''}HEADLINE=orig\n${script}\nprintf '%s|%s|%s' "$MARK_ROUND" "${'${CONSEC_FAIL}'}" "$HEADLINE"`,
|
||||
`set -uo pipefail\nWORKDIR='${dir}'\nMARK_ROUND=${markRound}\nMAX_ROUNDS=100\nCONSECUTIVE_FAILURE_CAP=${cap}\nTIMEOUT_WINDOW_CAP=${timeoutCap}\nAGENT_TIMEOUT='${agentTimeout}'\nCONSEC_FAIL=0\nREPO=o/r\nPR=1\nAUTOFIX_BOT=qwen-code-dev-bot\nRETRY_COMMAND='@qwen-code /retry'\nAPI_ERROR_DETAIL='${apiErrorDetail}'\nAPI_ERROR_KIND='${apiErrorKind}'\nPREPARE_OUTCOME='${prepareOutcome}'\nSTALE_BASE_RETRY='${staleBaseRetry}'\n${window !== undefined ? `WINDOW='${window}'\n` : ''}HEADLINE=orig\nHEADLINE_ZH=orig\n${script}\nprintf '\\n@@R@@%s|%s|%s|%s' "$MARK_ROUND" "${'${CONSEC_FAIL}'}" "$HEADLINE" "$HEADLINE_ZH"`,
|
||||
],
|
||||
{
|
||||
env: { ...process.env, PATH: `${bin}:${process.env.PATH}` },
|
||||
|
|
@ -15889,12 +15952,21 @@ exit 1
|
|||
},
|
||||
);
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
const [mark, consec, headline] = out.split('|');
|
||||
// The block echoes ::warning:: log lines (the idle-timeout census), so
|
||||
// read the result off its sentinel — job-log noise must never be
|
||||
// parsed as a field. The pre-sentinel half is the job-log surface the
|
||||
// census warning targets; return it so tests can pin it.
|
||||
const sentinelAt = out.lastIndexOf('@@R@@');
|
||||
const [mark, consec, headline, headlineZh] = out
|
||||
.slice(sentinelAt + 5)
|
||||
.split('|');
|
||||
return {
|
||||
mark,
|
||||
consec: Number(consec),
|
||||
terminal: mark === '100',
|
||||
headline,
|
||||
headlineZh,
|
||||
log: out.slice(0, sentinelAt),
|
||||
};
|
||||
};
|
||||
|
||||
|
|
@ -16004,50 +16076,112 @@ exit 1
|
|||
expect(interleaved.terminal).toBe(true);
|
||||
expect(interleaved.headline).toContain('time-budget exhaustions');
|
||||
expect(interleaved.headline).toContain('/retry');
|
||||
// Idle (silent-sandbox) timeouts share the census — each burns a full
|
||||
// budget — and when the window contains any, the breaker's advice says
|
||||
// a budget increase cannot cure them.
|
||||
const IDLE_HEAD =
|
||||
'🤖 AutoFix ran out of time before finishing (idle-timeout (no output for 1200000ms — the sandbox likely hung at startup)) (attempt 2/100) — it will retry on the next scan.';
|
||||
const idleMixed = run([IDLE_HEAD, PUSH, IDLE_HEAD, PUSH], {
|
||||
agentTimeout:
|
||||
'idle-timeout (no output for 1200000ms — the sandbox likely hung at startup)',
|
||||
// Idle (silent-sandbox) timeouts are EXCLUDED from this cap: the remedy
|
||||
// it prescribes (split the PR / raise the budget) cannot cure a runner
|
||||
// that produced no output at all, and counting them parked healthy PRs
|
||||
// (af-073). Interleaved with pushes they must never terminate — this is
|
||||
// the shape that stopped #8332 at 24 rounds and #8368 at 28 while both
|
||||
// were still pushing.
|
||||
const allIdle = run([IDLE_HEAD, PUSH, IDLE_HEAD, PUSH, IDLE_HEAD, PUSH], {
|
||||
agentTimeout: IDLE_NOW,
|
||||
});
|
||||
expect(idleMixed.terminal).toBe(true);
|
||||
expect(idleMixed.headline).toContain('time-budget exhaustions');
|
||||
expect(idleMixed.headline).toContain(
|
||||
'silent-sandbox (idle) timeouts that no budget increase can cure',
|
||||
);
|
||||
// An ALL-idle window swaps the closing remedy for the sandbox
|
||||
// investigation — mirroring the round-level split — instead of
|
||||
// prescribing the budget increase the clause above declared useless.
|
||||
expect(idleMixed.headline).toContain(
|
||||
'A human should investigate the sandbox image and runner docker daemon',
|
||||
);
|
||||
expect(idleMixed.headline).not.toContain('raise the agent time budget');
|
||||
// A MIXED window (any real budget timeout) keeps the budget remedy.
|
||||
const idleSome = run([TIMEOUT_HEAD, PUSH, IDLE_HEAD, PUSH], {
|
||||
agentTimeout:
|
||||
'idle-timeout (no output for 1200000ms — the sandbox likely hung at startup)',
|
||||
expect(allIdle.terminal).toBe(false);
|
||||
expect(allIdle.headline).toBe('orig');
|
||||
// Idle rounds do not become budget timeouts by piling up: no count of
|
||||
// them alone reaches the cap.
|
||||
expect(
|
||||
run(
|
||||
Array(timeoutCap * 3)
|
||||
.fill(IDLE_HEAD)
|
||||
.flatMap((h) => [h, PUSH]),
|
||||
{
|
||||
agentTimeout: IDLE_NOW,
|
||||
},
|
||||
).terminal,
|
||||
).toBe(false);
|
||||
// The escape hatch that makes the exclusion safe: an idle round pushes
|
||||
// nothing and matches no streak-reset needle, so a PERSISTENTLY wedged
|
||||
// sandbox still terminates — at the CONSECUTIVE cap, with its own
|
||||
// headline. Without this the exclusion would let a dead runner loop
|
||||
// forever.
|
||||
const idleStreak = run(Array(cap - 1).fill(IDLE_HEAD), {
|
||||
agentTimeout: IDLE_NOW,
|
||||
});
|
||||
expect(idleSome.terminal).toBe(true);
|
||||
expect(idleSome.headline).toContain('2 of those were silent-sandbox');
|
||||
expect(idleSome.headline).toContain('raise the agent time budget');
|
||||
// The CURRENT round's idle timeout is counted by the increment, not
|
||||
// the grep: cap-1 budget priors plus an idle current round render
|
||||
// "1 of those were silent-sandbox". Deleting the IDLE_N increment
|
||||
// suppresses the clause entirely (the grep sees no idle prior) and
|
||||
// must fail here.
|
||||
expect(idleStreak).toMatchObject({ consec: cap, terminal: true });
|
||||
expect(idleStreak.headline).toContain(
|
||||
'consecutive rounds that pushed nothing',
|
||||
);
|
||||
// ...and the TERMINAL run's job log still names the wedged runner: the
|
||||
// census warning runs outside the cap's terminal guard precisely so an
|
||||
// all-idle stop — which lands on the consecutive breaker's headline —
|
||||
// keeps its only infra signal. Moving the echo back under the guard
|
||||
// suppresses the warning here and must fail.
|
||||
expect(idleStreak.log).toContain(`::warning::#1: ${cap} silent-sandbox`);
|
||||
// A window whose BUDGET timeouts alone reach the cap still trips, and
|
||||
// the count it reports is the budget one — not the total, which would
|
||||
// re-inflate the number the exclusion just corrected.
|
||||
const mixedTrips = run(
|
||||
[TIMEOUT_HEAD, PUSH, TIMEOUT_HEAD, PUSH, IDLE_HEAD, PUSH],
|
||||
{ agentTimeout: 'timeout (3000000ms)' },
|
||||
);
|
||||
expect(mixedTrips.terminal).toBe(true);
|
||||
expect(mixedTrips.headline).toContain(
|
||||
`${timeoutCap} agent time-budget exhaustions`,
|
||||
);
|
||||
// ...and it names the idle rounds as excluded, so the operator still
|
||||
// learns the runner misbehaved on a PR stopped for an unrelated reason.
|
||||
expect(mixedTrips.headline).toContain('do NOT count toward this cap');
|
||||
expect(mixedTrips.headline).toContain('raise the agent time budget');
|
||||
// The idle clause's COUNT is pinned numerically in both languages —
|
||||
// mixedTrips holds exactly one idle round while TIMEOUT_N is
|
||||
// timeoutCap + 1, so an ${IDLE_N} → ${TIMEOUT_N} swap would inflate
|
||||
// the reported fleet problem and must fail here.
|
||||
expect(mixedTrips.headline).toContain('also holds 1 silent-sandbox');
|
||||
expect(mixedTrips.headlineZh).toContain('本窗口另有 1 次静默');
|
||||
// The ZH headline interpolates the same budget-only count — pin both
|
||||
// halves, or a ${BUDGET_TIMEOUT_N} → ${TIMEOUT_N} mutation on the ZH
|
||||
// line alone ships green while the comment's Chinese half re-inflates
|
||||
// the number the exclusion just corrected.
|
||||
expect(mixedTrips.headlineZh).toContain(`${timeoutCap} 次时间预算耗尽`);
|
||||
expect(mixedTrips.headlineZh).not.toContain(
|
||||
`${timeoutCap + 1} 次时间预算耗尽`,
|
||||
);
|
||||
// The headline interpolates BUDGET_TIMEOUT_N TWICE; the second
|
||||
// sentence ("That is … full agent runs") needs its own pins in both
|
||||
// languages, or the same swap mutant re-inflates exactly the count the
|
||||
// exclusion corrected — and calls an idle round a "full agent run".
|
||||
expect(mixedTrips.headline).toContain(
|
||||
`That is ${timeoutCap} full agent runs`,
|
||||
);
|
||||
expect(mixedTrips.headline).not.toContain(
|
||||
`That is ${timeoutCap + 1} full agent runs`,
|
||||
);
|
||||
expect(mixedTrips.headlineZh).toContain(`即 ${timeoutCap} 次完整`);
|
||||
// The idle census ::warning:: is the only observability left for
|
||||
// excluded idle timeouts — one idle round in this window warns exactly
|
||||
// once (deleting the echo, flipping -gt 0, or swapping the count each
|
||||
// fail here).
|
||||
expect(mixedTrips.log).toContain('::warning::#1: 1 silent-sandbox');
|
||||
// ...and its CAP interpolation and guidance tail are pinned too: a
|
||||
// ${TIMEOUT_WINDOW_CAP} → ${CONSECUTIVE_FAILURE_CAP} swap would
|
||||
// misstate the cap on the very channel designated the idle signal,
|
||||
// and a reworded tail would drop the operator guidance.
|
||||
expect(mixedTrips.log).toContain(
|
||||
`excluded from the ${timeoutCap}-timeout cap; check the sandbox image and the runner docker daemon`,
|
||||
);
|
||||
// One idle round is enough to hold a would-be-capped window open: cap-1
|
||||
// budget priors plus an idle current round is cap-1 budget timeouts, not
|
||||
// cap. Deleting the IDLE_N increment (or the subtraction) terminates
|
||||
// here and must fail.
|
||||
const idleCurrentOnly = run(Array(timeoutCap - 1).fill(TIMEOUT_HEAD), {
|
||||
agentTimeout:
|
||||
'idle-timeout (no output for 1200000ms — the sandbox likely hung at startup)',
|
||||
agentTimeout: IDLE_NOW,
|
||||
});
|
||||
expect(idleCurrentOnly.terminal).toBe(true);
|
||||
expect(idleCurrentOnly.headline).toContain(
|
||||
'1 of those were silent-sandbox (idle) timeouts',
|
||||
);
|
||||
// A window WITHOUT idle rounds keeps today's advice untouched.
|
||||
expect(idleCurrentOnly.terminal).toBe(false);
|
||||
expect(idleCurrentOnly.headline).toBe('orig');
|
||||
// A window WITHOUT idle rounds says nothing about the sandbox — in the
|
||||
// headline or the job log.
|
||||
expect(interleaved.headline).not.toContain('silent-sandbox');
|
||||
expect(interleaved.log).not.toContain('::warning');
|
||||
// One short of the cap keeps retrying (current round not a timeout).
|
||||
expect(run([TIMEOUT_HEAD, PUSH, TIMEOUT_HEAD])).toMatchObject({
|
||||
terminal: false,
|
||||
|
|
@ -16110,8 +16244,31 @@ exit 1
|
|||
expect(reviewAddressReportStep).toContain(
|
||||
'TIMEOUT_N="$(grep -c \'AutoFix ran out of time before finishing\' <<< "${PRIOR_HEADS}" || true)"',
|
||||
);
|
||||
// IDLE_N is SUBTRACTED from TIMEOUT_N, so its needle must be a strict
|
||||
// extension of TIMEOUT_N's — a bare 'idle-timeout' substring could match
|
||||
// provider error text that API_ERROR_DETAIL puts on the same first line
|
||||
// and drive the difference negative.
|
||||
expect(reviewAddressReportStep).toContain(
|
||||
'IDLE_N="$(grep -c \'idle-timeout\' <<< "${PRIOR_HEADS}" || true)"',
|
||||
'IDLE_N="$(grep -c \'AutoFix ran out of time before finishing (idle-timeout\' <<< "${PRIOR_HEADS}" || true)"',
|
||||
);
|
||||
// ...checked on the needles extracted from the workflow pins above,
|
||||
// not on literals — a literal-vs-literal comparison is true by
|
||||
// construction and would stay green whatever the workflow says.
|
||||
const timeoutNeedle = reviewAddressReportStep.match(
|
||||
/TIMEOUT_N="\$\(grep -c '([^']+)'/,
|
||||
)?.[1];
|
||||
const idleNeedle = reviewAddressReportStep.match(
|
||||
/IDLE_N="\$\(grep -c '([^']+)'/,
|
||||
)?.[1];
|
||||
expect(timeoutNeedle).toBeTruthy();
|
||||
expect(idleNeedle).toBeTruthy();
|
||||
expect(idleNeedle).toContain(timeoutNeedle);
|
||||
// The cap gates on the budget-only count, never the total.
|
||||
expect(reviewAddressReportStep).toContain(
|
||||
'BUDGET_TIMEOUT_N=$(( TIMEOUT_N - IDLE_N ))',
|
||||
);
|
||||
expect(reviewAddressReportStep).toContain(
|
||||
'if [[ "${BUDGET_TIMEOUT_N}" -ge "${TIMEOUT_WINDOW_CAP}" ]]; then',
|
||||
);
|
||||
// The reset detector keys on literal substrings; pin them to the actual
|
||||
// "Push and report" emit lines so a reword breaks this test, not silently
|
||||
|
|
@ -16157,6 +16314,61 @@ exit 1
|
|||
).toMatchObject({ consec: 2, terminal: false });
|
||||
});
|
||||
|
||||
it('ties the run-agent idle sentinel to the workflow classification and the replay fixture', () => {
|
||||
// The idle classification lives in three independently-pinned places:
|
||||
// run-agent.mjs's detail template (the EMITTER), the workflow's
|
||||
// current-round glob and census needles (the CONSUMERS), and this
|
||||
// file's replay fixture (the WITNESS). A format change on the emitter
|
||||
// side must break this test — not silently stop the workflow
|
||||
// classifying idle rounds while the fixture keeps replaying the old
|
||||
// shape. Extract the REAL template and check every consumer against it.
|
||||
const runner = readFileSync(autofixRunnerScriptPath, 'utf8');
|
||||
const idleDetailTemplate = runner.match(
|
||||
/result\.idleTimedOut\s*\?\s*`([^`]+)`/,
|
||||
)?.[1];
|
||||
expect(idleDetailTemplate).toBeTruthy();
|
||||
// The static prefix — emitted before any interpolation — is what the
|
||||
// workflow's current-round classification keys on.
|
||||
const detailPrefix = idleDetailTemplate.split('${')[0];
|
||||
const globTokens = [
|
||||
...reviewAddressReportStep.matchAll(
|
||||
/\[\[ "\$\{AGENT_TIMEOUT(?::-)?\}" == '([^']+)'\* \]\]/g,
|
||||
),
|
||||
].map((m) => m[1]);
|
||||
expect(globTokens.length).toBeGreaterThanOrEqual(2);
|
||||
for (const token of globTokens) {
|
||||
expect(detailPrefix.startsWith(token)).toBe(true);
|
||||
}
|
||||
// The census needles, extracted as in the breaker test: IDLE_N's must
|
||||
// be TIMEOUT_N's plus ' (' plus that same opening token.
|
||||
const timeoutNeedle = reviewAddressReportStep.match(
|
||||
/TIMEOUT_N="\$\(grep -c '([^']+)'/,
|
||||
)?.[1];
|
||||
const idleNeedle = reviewAddressReportStep.match(
|
||||
/IDLE_N="\$\(grep -c '([^']+)'/,
|
||||
)?.[1];
|
||||
expect(timeoutNeedle).toBeTruthy();
|
||||
expect(idleNeedle).toBeTruthy();
|
||||
for (const token of new Set(globTokens)) {
|
||||
expect(idleNeedle).toBe(`${timeoutNeedle} (${token}`);
|
||||
}
|
||||
// The replay fixture embeds the CAUSE-shaped headline with a concrete
|
||||
// ms value — derive it from the template so a reworded sentinel fails
|
||||
// here instead of shipping a fixture that replays a fantasy shape.
|
||||
const detail = idleDetailTemplate.replace(
|
||||
/\$\{QWEN_IDLE_TIMEOUT_MS\}/g,
|
||||
'1200000',
|
||||
);
|
||||
expect(detail).toBe(IDLE_NOW);
|
||||
const causeTemplate = reviewAddressReportStep.match(
|
||||
/CAUSE="(ran out of time before finishing \(\$\{AGENT_TIMEOUT\}\))"/,
|
||||
)?.[1];
|
||||
expect(causeTemplate).toBeTruthy();
|
||||
expect(IDLE_HEAD).toContain(
|
||||
causeTemplate.replace('${AGENT_TIMEOUT}', detail),
|
||||
);
|
||||
});
|
||||
|
||||
it('posts the review-address report wrapper lines bilingually', () => {
|
||||
// The agent's own address-summary.md / no-action.md ends with a collapsed
|
||||
// Chinese block, but these workflow-appended wrapper lines sit OUTSIDE it —
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue