The AI release-notes generator timed out on every batch in the v0.21.1
finalize run on a GitHub-hosted runner (US, Azure westus3) hitting a
remote LLM endpoint. The 60s per-request timeout was too close to the
edge: two successful requests took 54.2s and 56.7s, and every other
batch hit the 60s abort. The 12-minute total budget was consumed by
retries on those timeouts, and the circuit breaker opened after 3
consecutive failures, skipping highlights entirely.
- timeoutMs: 60s -> 180s — give the model enough headroom to generate
a full JSON response over a high-RTT cross-region connection.
- totalTimeoutMs: 12min -> 30min — ~140 PRs at 8 per batch is ~18
batches; at ~90s each that's ~27 minutes of model time.
- batchSize: 12 -> 8 — fewer entries per prompt means the model
generates less per request, returning faster and reducing the
chance of a single slow request dragging the whole batch.
- workflow timeout-minutes: 15 -> 35 — match the new 30min budget
plus a margin for git/npm setup.
No changes to retry logic, circuit breaker, or prompt shape.
* fix(scripts): harden retry classification and preserve deadline error context
Follow-up to #7535 addressing remaining review suggestions:
- isRetryableModelError: content-validation errors ('Model response did not
contain message content.') are now non-retryable. These are deterministic
failures from our own code — retrying the same prompt reproduces the same
failure. Network-level errors (no HTTP status) remain retryable.
- Deadline-expired errors now include the original error message instead of
the generic 'budget exhausted', so oncall can see the root cause.
- Added 4 tests: HTTP 429 retry, network error retry, content-validation
non-retry, deadline error preservation.
* fix(release): preserve last model error across retry budget
* test(release): cover retry budget expiry after backoff
* refactor(release): extract deadline error helper and preserve cause
Consolidate the three identical budget-exhausted throw sites into a single deadlineError() helper so they share one message format and source variable, and attach the last model error as the Error cause so its stack trace survives into CI logs. Also conform the release-notes tests added by this series to Prettier.
* fix(scripts): retry model calls and surface degraded release notes
The stable-release notes generator fell back to pull-request titles on
any model error and the finalize workflow stayed green with no visible
degradation. Four recent stable releases lost every summary batch and
the highlights call to 60s timeouts without anyone noticing.
The completer now retries each request (default 2 retries, exponential
backoff with jitter) on timeouts, network errors, 429, and 5xx, while
4xx fails immediately. After three consecutive failed summary batches a
circuit breaker stops paying the timeout for the rest and skips the
highlights call, since the model side is down rather than slow.
Fallback warnings are emitted as :⚠️: workflow commands so they
render as run annotations, and a degraded-run block is appended to
GITHUB_STEP_SUMMARY when any fallback occurred, distinguishing fully
degraded (titles only) from partially degraded runs.
Refs #7523
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* ci: retrigger after flaky fork-dispatch test failure
* fix(scripts): create the step-summary parent dir before appending
The ubuntu CI shard of this PR failed the summary test with ENOENT on
readFileSync after appendFileSync: the append reached the runner fine,
but the parent of the summary path can be missing there. Create it
recursively first, which is what the function's contract (the note must
land in the summary file) already implies.
* fix(scripts): assert the degraded summary through the fs mock
test-setup.ts no-op mocks appendFileSync for the whole scripts project, so the new test's real-file read could never see the write under test:scripts. Assert on the mock call instead.
* fix(scripts): keep summary-write failures from costing the notes and make the reset test real
appendDegradedStepSummary now runs through tryAppendDegradedStepSummary so a filesystem error on the auxiliary summary cannot kill the release before the notes are written, and the circuit-breaker recovery test's mock returns valid summary payloads so the reset path is actually exercised.
* fix(scripts): escape model-controlled text in workflow commands
Warnings can carry model output (parse errors, PR-derived fields) into
:⚠️: commands. Percent-encode %/CR/LF so a forged ::error:: line
cannot emit a second runner command, and collapse newlines in the
step-summary markdown list so a multi-line warning does not break the
list. Closes the P1 from PR review.
* fix(scripts): bound release-note AI generation time
* fix(scripts): log model request retries to stderr
Retry attempts in createOpenAiCompleter were silent, so an oncall
engineer could not distinguish slow model responses from exponential-
backoff retries in the CI log. Emit a one-line stderr log on each retry
with the attempt count, the triggering error, and the backoff delay.
Plain stderr (not :⚠️:) so transient retries that recover do not
become run annotations.
* fix(scripts): escape retry log workflow commands
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: 易良 <1204183885@qq.com>
Co-authored-by: yiliang114 <effortyiliang@gmail.com>
* feat(release): generate AI-assisted release notes
* docs(release): remove AI release notes planning docs
* fix(release): preserve complete AI release notes
* refactor(release): trim AI notes generator surface
* ci: Open autofix issues for main CI failures
* fix(release): harden AI notes review feedback
* fix(release): close AI notes review gaps
