perf(ci): make the autofix fleet caps operator-tunable and raise them (#8731)

The three caps that bound the autofix review loop were compiled-in
literals sized for a much smaller takeover pool, so growing the pool
meant editing the workflow, opening a PR and waiting for review every
time. The takeover pool is not static — it grew to 37 PRs by
2026-08-08, and it keeps growing.

Back all three with repository variables, keeping the literals as
fallbacks, so the loop is resized in Settings → Variables with no code
change:

  QWEN_AUTOFIX_MAX_PARALLEL              (fallback 20, was 5)
  QWEN_AUTOFIX_MAX_TARGETS_PER_SCAN      (fallback 30, was 10)
  QWEN_AUTOFIX_MAX_CANDIDATE_INSPECTIONS (fallback 60, unchanged)

Verified on a live runner that `max-parallel` accepts the expression and
schedules by it — a 6-leg matrix resolving to 3 started exactly 3 legs
and began the 4th only after a slot freed. Not assumed: an invalid
expression here makes the whole file invalid, which this repository just
paid 12.9 hours of dead review automation for.

The raised fallbacks are sized against measurements, not guesses. At 5
slots the fleet served ~14% of the takeover pool at once, reproducing at
a larger scale the 81-minute tail measured back at 3. The ecs-qwen fleet
is 84 runners, so 20 concurrent legs take under a quarter of it, and the
legs sampled that day finished in 3-28 minutes. Worst case rises to 100
runner-hours across the fleet (20 slots x the 300-minute job cap), and
per-PR head-write concurrency groups are per-PR, so this adds no push
contention.

MAX_TARGETS_PER_SCAN has to stay above max-parallel or the scan cannot
emit enough legs to fill the matrix. That relation is pinned for the
fallbacks by an existing test and stated at both definitions for the
variables, where it becomes an operator invariant.

Mutation-tested, 4 of 4 caught: fallback equal to the budget, fallback
above it, and dropping either variable back to a literal.

Co-authored-by: verify <verify@local>
This commit is contained in:
Shaojin Wen 2026-08-08 16:53:07 +08:00 committed by GitHub
parent 59b750fc4d
commit ee98f7420b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 57 additions and 14 deletions

View file

@ -164,14 +164,27 @@ env:
INFRA_FAILURE_SIGNATURES: 'lost communication with the server|No space left on device|ENOSPC|received a shutdown signal|The runner has received|Failed to initialize container|runner (was|has been) (lost|terminated)|invalid index-pack output|RPC failed'
# Upper bound on review targets emitted per scan (fan-out defense-in-depth;
# excess is logged and deferred to the next scan).
MAX_TARGETS_PER_SCAN: '10'
# TUNABLE WITHOUT A CODE CHANGE: set the repository variable to re-size the
# loop as the takeover pool grows — Settings → Variables, no PR, no deploy.
# The literal here is only the fallback when the variable is unset.
# Why 30: 37 PRs carried autofix/takeover on 2026-08-08, so the previous
# budget of 10 emitted at most 27% of the eligible set per tick and pushed
# the rest a scan further out every time. It must also stay strictly above
# the address matrix's max-parallel or the matrix can never fill, and the
# same-repo candidate pool was 51, so 30 still bounds a pathological
# backlog. RAISE BOTH TOGETHER: this must stay above QWEN_AUTOFIX_MAX_PARALLEL.
MAX_TARGETS_PER_SCAN: '${{ vars.QWEN_AUTOFIX_MAX_TARGETS_PER_SCAN || 30 }}'
# Upper bound on candidates INSPECTED per scan: idle candidates consume
# serial API calls even when they emit nothing, and takeover widens the
# candidate pool. Candidates are inspected NEWEST-first; past the budget
# the oldest tail defers — old quiet PRs are the least likely to hold new
# feedback, and a deferred PR with a live conflict is still picked up by
# the shepherd's conflict lever.
MAX_CANDIDATE_INSPECTIONS: '60'
# TUNABLE WITHOUT A CODE CHANGE, for the same reason as the two above: this
# one gates whether a PR is even LOOKED AT, so it has to grow ahead of the
# candidate pool or the oldest PRs starve. The pool was 51 same-repo open
# PRs on 2026-08-08, still under the 60 fallback.
MAX_CANDIDATE_INSPECTIONS: '${{ vars.QWEN_AUTOFIX_MAX_CANDIDATE_INSPECTIONS || 60 }}'
# Maintainer-facing engagement labels (applying labels requires GitHub
# triage+, so the permission gate is GitHub's own): TAKEOVER opts a PR —
# including a human-authored one — into the loop; SKIP opts any PR out
@ -3277,14 +3290,29 @@ jobs:
# Measured at 3, on the scan that selected 7 PRs: the legs started
# 3-at-a-time and each new one began 3-4s after a slot freed, so the
# 7th PR waited 81 minutes for a slot it could have had immediately.
# 5 halves that tail while staying a real bound — the point of the cap
# is that a backlog cannot open an unbounded number of agent runs at
# once, not the specific number. The 300-minute job cap raises the
# worst-case hold to 5 runner-hours per slot (25 across the fleet) and
# holds the per-PR head-write concurrency group for the same window.
# The cap itself does not lengthen the queueing tail above; the raised
# 120-minute budget does, for the PRs that exhaust it.
max-parallel: 5
# 5 halved that tail — the point of the cap is that a backlog cannot
# open an unbounded number of agent runs at once, not the specific
# number.
# TUNABLE WITHOUT A CODE CHANGE: set QWEN_AUTOFIX_MAX_PARALLEL in
# Settings → Variables to re-size the fleet as the takeover pool grows;
# the literal below is only the fallback when the variable is unset.
# Verified on a live runner that `max-parallel` accepts this expression
# and schedules by it — a 6-leg matrix at 3 started 3, then began the
# 4th only once a slot freed.
# Why 20: 37 PRs carried the label on 2026-08-08, so 5 slots served
# ~14% of them at a time and the tail measured at 3 simply reappeared
# at a larger scale. The ecs-qwen fleet is 84 runners, so 20 concurrent
# legs occupy under a quarter of it, and the executed legs sampled that
# day finished in 3-28 minutes.
# The 300-minute job cap puts the worst case at 5 runner-hours per slot
# (100 across the fleet at 20) and holds the per-PR head-write
# concurrency group for the same window. Different PRs never share that
# group, so raising this does not add push contention.
# RAISE BOTH TOGETHER: this must stay strictly below
# MAX_TARGETS_PER_SCAN, or the scan cannot emit enough legs to fill the
# matrix and the extra slots sit idle. A test pins that for the
# fallbacks; for the variables it is an operator invariant.
max-parallel: '${{ fromJSON(vars.QWEN_AUTOFIX_MAX_PARALLEL || 20) }}'
matrix:
target: '${{ fromJSON(needs.review-scan.outputs.targets) }}'
# Serialises every writer of THIS PR's head branch, across workflows.