ci: let canonical PR rerun matrices finish

Disable native Node matrix fail-fast for every openclaw/openclaw PR
attempt. Run 36804915849 attempt 2 cancelled 57 jobs after an inherited
main failure, preventing the remaining green proof needed by the
explicit prior-CI admin landing route.

Keep first-attempt monitoring, runner caps, routing, timeouts, and other
matrices unchanged. Qualify cancellation against each run's tested
workflow: retain historical expressions and accept the new expression
only for PRs in other workflow repositories. Align CI and landing docs.

Local proof: cancellation verifier 41 tests, workflow control 14 tests,
monitor 65 tests, hourly CI 22 tests, focused Node planning 1 test,
runner-cap and workflow-size guards 2 tests. The new regression failed
on the original workflow. Workflow sanity, formatting, and focused lint
passed; ci.yml is 404072 bytes under the 480000-byte budget. Codex P2
review found no actionable findings. No CI dispatch or rerun requested.
This commit is contained in:
Peter Steinberger 2026-09-30 20:31:15 -07:00
parent 2fd8e321ba
commit 55fe1b4889
No known key found for this signature in database
11 changed files with 88 additions and 47 deletions

View file

@ -270,7 +270,10 @@ These are intentionally guarded by the `ci-workflow-guards`,
API and job deadlines remain unchanged.
The aggregate preserves failure-triggered PR cancellation through the
`pr-fail-fast` cause outputs; superseded runs without a failure cause still
skip the aggregate. PR Node matrices use native fail-fast. The same-repository
skip the aggregate. Canonical PR Node matrices disable native fail-fast on
every attempt; reruns complete every leg so inherited main failures leave the
remaining admin-landing proof intact. Native fail-fast applies only to PRs in
other workflow repositories. Historical runs retain their tested policy. The same-repository
PR first-attempt monitor alone has `actions: write` and adds one 4-class registration per
eligible PR, or uses hosted Ubuntu under the outage override. Main/manual
matrices remain complete. The monitor starts after preflight, observes failures

View file

@ -267,8 +267,17 @@ ordered terminal steps, the expected shard ordinal, bounded timestamps reaching
the deadline, successful cleanup, and no other failed or cancelled step. Preserve
any unfinished receipt or canary coverage in its independent failure attribution.
For the existing Node matrix's native fail-fast (including fork PRs whose monitor
is skipped), use `cancellation.kind: "matrix-fail-fast"` and
Current `openclaw/openclaw` PR reruns do not use native matrix fail-fast: every
Node matrix leg can finish, preserving the remaining proof for inherited-red admin
landing. This also applies to fork PRs targeting `openclaw/openclaw`; the workflow
repository, not the head repository, owns this policy. The first-attempt monitor
is unchanged. Native matrix fail-fast remains enabled only for PRs running in
other repositories.
Historical runs still use their tested workflow's policy, including the former
expression that enabled native fail-fast on canonical PR reruns. For a run whose
tested workflow and attempt/repository context enable native fail-fast, use
`cancellation.kind: "matrix-fail-fast"` and
`workflowJob: "checks-node-core-test-nondist-shard"` instead of monitor `jobId`/`step`.
Its `causedBy` must name a nonempty, unique subset of independently admitted
failed roots that actually caused this matrix cancellation. Add `members`, the

View file

@ -3137,7 +3137,7 @@ jobs:
runs-on: ${{ (github.event_name == 'workflow_dispatch' && startsWith(inputs.dispatch_id, 'full-release-validation-') && vars.OPENCLAW_RELEASE_RUNNER_GROUP || '') != '' && fromJSON(format('{{"group":{0},"labels":{1}}}', toJSON((github.event_name == 'workflow_dispatch' && startsWith(inputs.dispatch_id, 'full-release-validation-') && vars.OPENCLAW_RELEASE_RUNNER_GROUP || '')), toJSON(needs.preflight.outputs.node_runner_backend == 'runson' && matrix.runner == 'runson-c8i-8xlarge' && github.run_attempt == 1 && format('runs-on={0}-{1}/family=c8i.8xlarge/cpu=32/ram=64/spot=true/retry=false/image=ubuntu24-full-x64/volume=80gb', github.run_id, matrix.check_name) || matrix.runner == 'runson-c8i-8xlarge' && 'ubuntu-24.04' || (needs.preflight.outputs.node_runner_backend == 'runson' || needs.preflight.outputs.ci_qualification == 'true') && github.run_attempt == 1 && ((matrix.runner == 'blacksmith-8vcpu-ubuntu-2404' && contains(fromJSON('["checks-node-compact-large-9","checks-node-compact-large-5"]'), matrix.check_name)) && 'blacksmith-16vcpu-ubuntu-2404' || matrix.runner || 'blacksmith-4vcpu-ubuntu-2404') || (needs.preflight.outputs.ci_qualification == 'true' && (github.run_attempt == 1 && needs.preflight.outputs.qualification_runner_backend || 'github') || vars.OPENCLAW_CI_RUNNER_BACKEND) == 'github' && 'ubuntu-24.04' || (contains(fromJSON('["hybrid","runson"]'), (needs.preflight.outputs.ci_qualification == 'true' && (github.run_attempt == 1 && needs.preflight.outputs.qualification_runner_backend || 'github') || vars.OPENCLAW_CI_RUNNER_BACKEND)) && github.run_attempt > 1) && 'ubuntu-24.04' || ((github.event_name == 'workflow_dispatch' && ((needs.preflight.outputs.node_runner_backend != 'runson' && needs.preflight.outputs.ci_qualification != 'true') || github.run_attempt != 1)) || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository && github.run_attempt > 1)) && 'ubuntu-24.04' || (github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || contains(fromJSON('["OWNER","MEMBER","COLLABORATOR","CONTRIBUTOR"]'), github.event.pull_request.author_association)) && ((matrix.runner == 'blacksmith-8vcpu-ubuntu-2404' && contains(fromJSON('["checks-node-compact-large-9","checks-node-compact-large-5"]'), matrix.check_name)) && 'blacksmith-16vcpu-ubuntu-2404' || matrix.runner || 'blacksmith-4vcpu-ubuntu-2404') || 'ubuntu-24.04')))) || (needs.preflight.outputs.node_runner_backend == 'runson' && matrix.runner == 'runson-c8i-8xlarge' && github.run_attempt == 1 && format('runs-on={0}-{1}/family=c8i.8xlarge/cpu=32/ram=64/spot=true/retry=false/image=ubuntu24-full-x64/volume=80gb', github.run_id, matrix.check_name) || matrix.runner == 'runson-c8i-8xlarge' && 'ubuntu-24.04' || (needs.preflight.outputs.node_runner_backend == 'runson' || needs.preflight.outputs.ci_qualification == 'true') && github.run_attempt == 1 && ((matrix.runner == 'blacksmith-8vcpu-ubuntu-2404' && contains(fromJSON('["checks-node-compact-large-9","checks-node-compact-large-5"]'), matrix.check_name)) && 'blacksmith-16vcpu-ubuntu-2404' || matrix.runner || 'blacksmith-4vcpu-ubuntu-2404') || (needs.preflight.outputs.ci_qualification == 'true' && (github.run_attempt == 1 && needs.preflight.outputs.qualification_runner_backend || 'github') || vars.OPENCLAW_CI_RUNNER_BACKEND) == 'github' && 'ubuntu-24.04' || (contains(fromJSON('["hybrid","runson"]'), (needs.preflight.outputs.ci_qualification == 'true' && (github.run_attempt == 1 && needs.preflight.outputs.qualification_runner_backend || 'github') || vars.OPENCLAW_CI_RUNNER_BACKEND)) && github.run_attempt > 1) && 'ubuntu-24.04' || ((github.event_name == 'workflow_dispatch' && ((needs.preflight.outputs.node_runner_backend != 'runson' && needs.preflight.outputs.ci_qualification != 'true') || github.run_attempt != 1)) || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository && github.run_attempt > 1)) && 'ubuntu-24.04' || (github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || contains(fromJSON('["OWNER","MEMBER","COLLABORATOR","CONTRIBUTOR"]'), github.event.pull_request.author_association)) && ((matrix.runner == 'blacksmith-8vcpu-ubuntu-2404' && contains(fromJSON('["checks-node-compact-large-9","checks-node-compact-large-5"]'), matrix.check_name)) && 'blacksmith-16vcpu-ubuntu-2404' || matrix.runner || 'blacksmith-4vcpu-ubuntu-2404') || 'ubuntu-24.04')) }}
timeout-minutes: ${{ matrix.timeout_minutes || 60 }}
strategy:
fail-fast: ${{ github.event_name == 'pull_request' && (github.run_attempt != 1 || github.repository != 'openclaw/openclaw') }}
fail-fast: ${{ github.event_name == 'pull_request' && github.repository != 'openclaw/openclaw' }}
max-parallel: ${{ github.event_name == 'pull_request' && github.repository == 'openclaw/openclaw' && github.event.pull_request.head.repo.full_name == github.repository && contains(fromJSON('["OWNER","MEMBER","COLLABORATOR","CONTRIBUTOR"]'), github.event.pull_request.author_association) && github.run_attempt == 1 && needs.preflight.outputs.frozen_target != 'true' && needs.preflight.outputs.runner_profile != 'github' && needs.preflight.outputs.node_runner_backend != 'runson' && contains(fromJSON('["","blacksmith","hybrid"]'), vars.OPENCLAW_CI_RUNNER_BACKEND) && 130 || 96 }}
matrix: ${{ fromJson(needs.preflight.outputs.checks_node_core_nondist_matrix) }}
steps:

View file

@ -29,8 +29,10 @@ GitHub-hosted runners.
First-attempt PR Node matrices let the scoped monitor classify failures before
cancelling eligible same-repository work. Fork monitoring is read-only. Exact
known hourly-main test and supported static failures can remain advisory when the PR leaves their
subjects unchanged and all remaining checks finish. Retries retain native matrix
fail-fast. Main and manual runs retain complete matrices. See
subjects unchanged and all remaining checks finish. Canonical PR reruns let every
Node matrix leg finish so inherited failures do not cancel the remaining proof
needed for an explicit admin landing. Native matrix fail-fast applies only to PRs
in other repositories. Main and manual runs retain complete matrices. See
[failure cancellation](/ci/pipeline#fail-fast-order).
First-hop compatibility uses a 3,200-second container budget and a 3,500-second lane

View file

@ -708,7 +708,7 @@ automation account, and SecOps-owned-path cases before declaring enforcement act
3. `build-artifacts` and the locale checks overlap with the fast Linux lanes. Control UI and native app source PRs exclude generated locale snapshots/resources; their serialized refresh workflows repair and auto-merge isolated generated PRs in the background. Source CI still blocks stale source inventories and unsafe localization calls. Generated PRs, manual CI, and release prep enforce full translated/platform-generated parity. Canonical `release/YYYY.M.PATCH` branches may include release-prep locale repairs with the other generated release output.
4. Baseline ratchets and selected Node test shards start independently after preflight. Node rows consume the manifest, not ratchet outputs. `ci-gate` still requires every selected ratchet to pass, and the PR failure monitor still cancels remaining work after a ratchet failure. Frozen targets retain their existing ratchet selection.
5. Current plans with guards run `check:coercion-helpers` there once; fast-only plans retain its standalone row. Other platform and runtime lanes fan out independently: `checks-fast-core` (including startup corpus), `checks-fast-contracts-plugins`, `checks-fast-contracts-channels`, `checks-windows`, `macos-node`, `macos-swift`, `ios-build`, the screenshot shards, and `android`.
6. For canonical-repository PRs selecting Node rows, `pr-fail-fast` watches the first attempt and classifies failures before cancelling eligible same-repository work. Fork PR monitoring is read-only and never requests cancellation; unknown failures remain blocking through normal lane results. Only that job has `actions: write`. It starts after preflight and observes failures while the installed check planner queues or runs. Clean completion combines preflight's other job counts with the planner's exact admitted check count, published by its successful `CI check job count v1: N` step. It rechecks the current PR head, auto-merge setting, and newer runs before cancellation. Retries retain native matrix fail-fast. The monitor checks out trusted base-revision scripts. It adds one 4-vCPU Blacksmith registration per eligible same-repository PR, or uses hosted Ubuntu for fork PRs and under the outage override. The hybrid hosted admission owner reserves that fork row before spending the unchanged 45-row optional-offload budget. Main, manual runs, and retries do not start it. Observation ends before the monitor's job limit; ordinary lane verification still owns the result when no failure was observed. Partial reruns ignore monitor causes and results retained from earlier attempts.
6. For canonical-repository PRs selecting Node rows, `pr-fail-fast` watches the first attempt and classifies failures before cancelling eligible same-repository work. Fork PR monitoring is read-only and never requests cancellation; unknown failures remain blocking through normal lane results. Only that job has `actions: write`. It starts after preflight and observes failures while the installed check planner queues or runs. Clean completion combines preflight's other job counts with the planner's exact admitted check count, published by its successful `CI check job count v1: N` step. It rechecks the current PR head, auto-merge setting, and newer runs before cancellation. Canonical PR reruns let every Node matrix leg finish so inherited main failures cannot cancel the remaining proof needed for an explicit admin landing. Native matrix fail-fast applies only to PRs whose workflow repository is not `openclaw/openclaw`, on any attempt. The monitor checks out trusted base-revision scripts. It adds one 4-vCPU Blacksmith registration per eligible same-repository PR, or uses hosted Ubuntu for fork PRs and under the outage override. The hybrid hosted admission owner reserves that fork row before spending the unchanged 45-row optional-offload budget. Main, manual runs, and retries do not start it. Observation ends before the monitor's job limit; ordinary lane verification still owns the result when no failure was observed. Partial reruns ignore monitor causes and results retained from earlier attempts.
7. `openclaw/ci-gate` waits for every selected lane. Preflight and security must succeed; downstream jobs may skip only when unselected by the manifest and existing event, runner, and compatibility conditions. An unexpected selected skip or any failed or canceled downstream job fails the aggregate. Failure-triggered cancellation preserves the originating job's identity and runs the gate to report failure, including a cancellation request with an uncertain response. The existing critical-path route already keeps trusted hybrid first attempts on the 4-vCPU Blacksmith class. A first-attempt same-repository failure also uses that class under the default or explicit Blacksmith profile so hosted assignment cannot consume the cancellation grace period. Retries and the GitHub outage override retain hosted aggregation. A superseded run without a recorded failure cause skips final reporting and releases its concurrency slot as before.
Bot-authored, same-repository PRs containing only generated native locale data

View file

@ -155,7 +155,11 @@ pending/skipped `openclaw/ci-gate`. An explicitly approved `pre-existing-failure
attribution instead binds the current failed attempt, effective gate check-run,
tested merge/base, unchanged failure inputs, and inspected qualification artifacts.
Every failed job and fail-fast cancellation must be accounted for; cancelled
coverage stays unrun. An independently attributed cancelled Node test,
coverage stays unrun. Current `openclaw/openclaw` PR reruns let every Node matrix
leg finish; only PRs in other workflow repositories use native matrix fail-fast.
Historical runs retain their tested workflow's cancellation policy, so the matrix
attribution route still verifies that exact expression and run context.
An independently attributed cancelled Node test,
`check-prod-types`, or real-Gateway UI root can use
`failures[].failedStep: { number, workflowJob }`, with
`checks-node-core-test-nondist-shard`, `check-shard`, or

View file

@ -88,7 +88,7 @@ export async function monitorPrFailure(options) {
throw new Error("Invalid PR cancellation context");
}
// Partial reruns reuse successful jobs; their attempt inventory is not the
// complete manifest. Native matrix fail-fast remains active on those runs.
// complete manifest. Canonical PR reruns let every matrix leg finish.
if (runAttempt !== 1) {
return "retry";
}

View file

@ -376,22 +376,27 @@ function verifyMatrixCancellation(context, cancellation, members) {
const owner = workflow?.jobs?.[workflowJob];
const failFast = owner?.strategy?.["fail-fast"];
const repository = run.repository?.full_name;
const attemptAwareFailFast =
// Historical runs retain the cancellation policy from their tested workflow.
const historicalAttemptAware =
failFast ===
"${{ github.event_name == 'pull_request' && (github.run_attempt != 1 || github.repository != 'openclaw/openclaw') }}" &&
"${{ github.event_name == 'pull_request' && (github.run_attempt != 1 || github.repository != 'openclaw/openclaw') }}";
const scopedFailFast =
(historicalAttemptAware ||
failFast ===
"${{ github.event_name == 'pull_request' && github.repository != 'openclaw/openclaw' }}") &&
positiveInteger(run.run_attempt) &&
typeof repository === "string" &&
/^[A-Za-z0-9-]+\/[A-Za-z0-9_.-]+$/u.test(repository) &&
// Actions compares strings without case; github.repository is the workflow owner, not the fork.
(run.run_attempt > 1 || repository.toLowerCase() !== "openclaw/openclaw");
((historicalAttemptAware && run.run_attempt > 1) ||
repository.toLowerCase() !== "openclaw/openclaw");
requireEvidence(
owner?.name === "${{ matrix.check_name || 'checks-node-core-test-nondist-shard' }}" &&
Array.isArray(owner.needs) &&
owner.needs.includes("preflight") &&
owner.strategy?.matrix ===
"${{ fromJson(needs.preflight.outputs.checks_node_core_nondist_matrix) }}" &&
([true, "${{ github.event_name == 'pull_request' }}"].includes(failFast) ||
attemptAwareFailFast) &&
([true, "${{ github.event_name == 'pull_request' }}"].includes(failFast) || scopedFailFast) &&
[undefined, false].includes(owner["continue-on-error"]),
"the tested workflow must enable the existing PR matrix fail-fast contract",
);

View file

@ -492,7 +492,7 @@ describe("PR failure monitor", () => {
expect(await f.monitor()).toBe("failure-cancelled");
expect(f.events).toEqual(["cause 3", "POST /actions/runs/100/cancel"]);
});
it("leaves partial reruns to native fail-fast without waiting for cached jobs", async () => {
it("skips partial reruns without cancelling or waiting for cached jobs", async () => {
const f = fixture({ jobs: [job(3)] });
expect(await f.monitor(100, 2)).toBe("retry");
expect(f.fetchMock).not.toHaveBeenCalled();

View file

@ -181,7 +181,7 @@ describe("PR failure cancellation", () => {
});
it.each(["pull_request", "push", "workflow_dispatch"] as const)(
"keeps first-attempt continuation within the canonical monitor's scope (%s)",
"keeps canonical PR matrices complete and continuation within the first-attempt monitor (%s)",
(eventName) => {
const workflow = readCiWorkflow();
const node = workflow.jobs["checks-node-core-test-nondist-shard"];
@ -189,7 +189,9 @@ describe("PR failure cancellation", () => {
for (const [repository, headRepository, runAttempt, nativeFailFast, continuation] of [
["openclaw/openclaw", "openclaw/openclaw", 1, false, "1"],
["openclaw/openclaw", "contributor/openclaw", 1, false, "1"],
["openclaw/openclaw", "openclaw/openclaw", 2, true, "0"],
["openclaw/openclaw", "openclaw/openclaw", 2, false, "0"],
["openclaw/openclaw", "contributor/openclaw", 2, false, "0"],
["openclaw/openclaw", "openclaw/openclaw", 3, false, "0"],
["fork/openclaw", "fork/openclaw", 1, true, "0"],
["fork/openclaw", "contributor/openclaw", 1, true, "0"],
["fork/openclaw", "fork/openclaw", 2, true, "0"],

View file

@ -1,9 +1,12 @@
import { expect, it } from "vitest";
import { describe, expect, it } from "vitest";
import { verifyPriorCiCancellation } from "../../scripts/pr-lib/merge-prior-ci-cancellation.mjs";
const attemptAwareFailFast =
"${{ github.event_name == 'pull_request' && (github.run_attempt != 1 || github.repository != 'openclaw/openclaw') }}";
const repositoryScopedFailFast =
"${{ github.event_name == 'pull_request' && github.repository != 'openclaw/openclaw' }}";
function qualify(run: Record<string, unknown>, failFast: string | boolean = attemptAwareFailFast) {
const failed = { id: 1, name: "failed", conclusion: "failure", steps: [] };
const cancelled = { id: 2, name: "cancelled", conclusion: "cancelled", steps: [] };
@ -47,30 +50,37 @@ function qualify(run: Record<string, unknown>, failFast: string | boolean = atte
});
}
it.each([
["same repository rerun", 2, "openclaw/openclaw", true],
["later same repository rerun", 3, "openclaw/openclaw", true],
["other repository first attempt", 1, "example/openclaw", true],
["same repository first attempt", 1, "openclaw/openclaw", false],
["case-insensitive repository", 1, "OpenClaw/OpenClaw", false],
["missing attempt", undefined, "example/openclaw", false],
["string attempt", "2", "example/openclaw", false],
["zero attempt", 0, "example/openclaw", false],
["fractional attempt", 1.5, "example/openclaw", false],
["missing repository", 2, undefined, false],
["empty repository", 2, "", false],
["malformed repository", 2, "openclaw", false],
["non-string repository", 2, 123, false],
] as const)(
"qualifies the attempt-aware matrix contract: %s",
(_name, attempt, repository, accepted) => {
describe.each([
["historical attempt-aware", attemptAwareFailFast, true],
["repository-scoped", repositoryScopedFailFast, false],
] as const)("%s matrix contract", (_contract, expression, canonicalRerunAccepted) => {
it.each([
["same repository rerun", 2, "openclaw/openclaw", canonicalRerunAccepted],
["later same repository rerun", 3, "openclaw/openclaw", canonicalRerunAccepted],
["case-insensitive rerun", 2, "OpenClaw/OpenClaw", canonicalRerunAccepted],
["other repository first attempt", 1, "example/openclaw", true],
["other repository rerun", 2, "example/openclaw", true],
["same repository first attempt", 1, "openclaw/openclaw", false],
["case-insensitive repository", 1, "OpenClaw/OpenClaw", false],
["missing attempt", undefined, "example/openclaw", false],
["string attempt", "2", "example/openclaw", false],
["zero attempt", 0, "example/openclaw", false],
["fractional attempt", 1.5, "example/openclaw", false],
["missing repository", 2, undefined, false],
["empty repository", 2, "", false],
["malformed repository", 2, "openclaw", false],
["non-string repository", 2, 123, false],
] as const)("qualifies %s", (_name, attempt, repository, accepted) => {
const invoke = () =>
qualify({
event: "pull_request",
run_attempt: attempt,
repository: { full_name: repository },
head_repository: { full_name: "contributor/fork" },
});
qualify(
{
event: "pull_request",
run_attempt: attempt,
repository: { full_name: repository },
head_repository: { full_name: "contributor/fork" },
},
expression,
);
if (accepted) {
expect(invoke()).toMatchObject({ cancelledJobIds: [2] });
} else {
@ -78,13 +88,19 @@ it.each([
"the tested workflow must enable the existing PR matrix fail-fast contract",
);
}
},
);
});
it.each([undefined, "push", "workflow_dispatch"])("refuses non-PR run context: %s", (event) => {
expect(() =>
qualify({ event, run_attempt: 2, repository: { full_name: "openclaw/openclaw" } }),
).toThrow("matrix cancellation requires the existing PR Node matrix owner");
it.each([undefined, "push", "workflow_dispatch", "schedule"])(
"refuses non-PR run context: %s",
(event) => {
expect(() =>
qualify(
{ event, run_attempt: 2, repository: { full_name: "example/openclaw" } },
expression,
),
).toThrow("matrix cancellation requires the existing PR Node matrix owner");
},
);
});
it("refuses arbitrary expressions even when their run context would enable cancellation", () => {