fix(ci): keep the failure dump when the disk is full, and pin four guard holes

Review round 1 found five holes in the guards this PR added. Four are fixed
here; the fifth is a comment that misstated its own mirror.

The state-at-failure dump wrote its only copy into the samples file — the very
file whose writability the dump exists to investigate. Under ENOSPC the
redirect failed, `2>/dev/null` and `|| true` swallowed the failure and its
status, and the step exited 0 having recorded nothing, so oncall could not tell
"dumped, nothing interesting" from "could not write". Mirror through `tee -a`,
which still emits the data to stdout when the file write fails and reports why
on a stderr the block does not swallow. An executed case occupies the target
path with a directory and asserts the dump survives in stdout while the step
stays green.

The other three are pins that did not bite:

- The shared-prelude floor stayed at 12 after the divergence exemption narrowed
  and re-admitted the collector, making the intersection 13. A floor below the
  real count lets any one shared step be deleted out of either job unnoticed.
- The serializer at the heart of this PR had no fixture. Both sides of every
  comparison flow through it, so regressing the call site to the pre-PR array
  replacer compares {} to {} and the drift loop reports green — the exact blind
  spot the recursion replaced. Pin a known nested key on the serialized OUTPUT
  rather than on the serializer alone, because the regression that matters is a
  call site swapped back to an allowlist.
- Nothing pinned the `needs` edge that makes every classifier output this suite
  reads non-empty at runtime; deleting it decouples the lane silently, dropping
  runs-on to hosted and taking the degraded empty→full profile fallback.

And the collector's comment cited `test` as its mirror for lastness while
`test`'s own collector sits four steps from the end, so the hole this PR closes
here stays open there. Correct the comment and leave that job to a follow-up
rather than widen this change into a second job.

Mutations measured red: needs edge deleted; a shared prelude step deleted; the
serializer call site regressed to the array-replacer form; tee reverted to a
bare redirect, which reds both the pin and the executed case.
This commit is contained in:
Shaojin Wen 2026-09-04 09:37:05 +08:00
parent 54cbaa0119
commit 2082e26228
3 changed files with 91 additions and 10 deletions

View file

@ -2,6 +2,7 @@ import assert from 'node:assert/strict';
import { spawnSync } from 'node:child_process';
import {
chmodSync,
mkdirSync,
mkdtempSync,
readFileSync,
rmSync,
@ -176,11 +177,14 @@ describe('ci.yml disk-pressure evidence', () => {
// The install sampler dies with that step's trap … EXIT, so the 19
// substantive steps after it write no samples; the failure-gated dump
// is what puts state-at-failure into the artifact. It must APPEND —
// a bare > would truncate the install-window samples — and must sit
// before the collector.
// a bare > would truncate the install-window samples — must sit
// before the collector, and must mirror to the job log, because the
// file it appends to is the one whose writability is under
// investigation. `tee -a` carries all three; the executed case below
// is what proves the mirroring actually survives a failed write.
const dump = lintStep('Dump disk state on failure');
assert.equal(dump.if, '${{ failure() }}');
assert.match(dump.run, />> "\$DISK_SAMPLES"/);
assert.match(dump.run, /tee -a "\$DISK_SAMPLES"/);
assert.doesNotMatch(dump.run, /[^>]> "\$DISK_SAMPLES"/);
assert.ok(
names.indexOf('Dump disk state on failure') <
@ -188,6 +192,53 @@ describe('ci.yml disk-pressure evidence', () => {
);
});
it('keeps the failure dump in the job log when the samples file is unwritable', () => {
// The dump exists to explain an ENOSPC death, so it cannot depend on the
// filesystem being writable: redirected straight into the samples file, a
// failed write loses the dump and its own diagnostic together and the step
// still exits 0, leaving oncall unable to tell "dumped, nothing
// interesting" from "could not write". Occupy the target path with a
// directory so tee's append fails the way a full disk would.
const root = mkdtempSync(join(tmpdir(), 'ci-disk-pressure-'));
mkdirSync(join(root, 'disk-pressure-samples.log'));
try {
const result = spawnSync(
'bash',
[
'-e',
'-o',
'pipefail',
'-c',
lintStep('Dump disk state on failure').run,
],
{
encoding: 'utf8',
timeout: 30_000,
env: { ...process.env, RUNNER_TEMP: root },
},
);
assert.equal(result.error, undefined);
// `|| true` keeps a failed dump from failing the job it is diagnosing.
assert.equal(
result.status,
0,
`signal: ${result.signal}\nstdout: ${result.stdout}\nstderr: ${result.stderr}`,
);
// The data survives in the job log even though the file write failed.
assert.match(result.stdout, /^DISKCONTEXT failure-dump /m);
// And tee's reason is not swallowed by the block's own 2>/dev/null,
// which redirects the brace group only.
assert.ok(
result.stderr.length > 0,
`tee's write failure reached neither the file nor the log\nstdout: ${result.stdout}`,
);
} finally {
rmSync(root, { recursive: true, force: true });
}
});
it('keeps install failure status while writing the pre-install sample', () => {
const root = mkdtempSync(join(tmpdir(), 'ci-disk-pressure-'));
const npm = join(root, 'npm');

View file

@ -1214,8 +1214,14 @@ jobs:
# State-at-failure dump for the 19 substantive steps after install: the
# install sampler is reaped by that step's `trap … EXIT`, so a death in
# ESLint or the bundle-closure build would otherwise upload an artifact
# holding only install-window samples. Appends — a bare `>` would
# truncate exactly those samples.
# holding only install-window samples. `tee -a` appends — a bare `>`
# would truncate exactly those samples — and mirrors to the job log,
# because the file it appends to is the one whose writability is under
# investigation: redirected straight into it, an ENOSPC death loses the
# dump and its own diagnostic together, leaving oncall unable to tell
# "dumped, nothing interesting" from "could not write". tee still emits
# the data to stdout when the file write fails, and reports why on the
# stderr this block does not swallow; it exits 1 there, hence `|| true`.
- name: 'Dump disk state on failure'
if: '${{ failure() }}'
run: |-
@ -1225,13 +1231,16 @@ jobs:
df -hT || true
df -i || true
grep -E 'MemTotal|MemAvailable' /proc/meminfo || true
} >> "$DISK_SAMPLES" 2>/dev/null || true
} 2>/dev/null | tee -a "$DISK_SAMPLES" || true
# Collector for the #10035 telemetry, kept as the LAST step: failure()
# is evaluated when the step is reached and never revisited, so at its
# old slot just after install it had already been passed (and skipped,
# nothing yet failed) by the time any lint/static step could die — it
# could only ever collect a prelude failure. Mirrored from `test`, whose
# collector likewise follows its heavy step. Distinct artifact name:
# could only ever collect a prelude failure. `test` is NOT mirrored here
# and does not get that treatment: its collector sits four steps from the
# end, so a failure in the packaging or upload steps that follow it is
# never collected in that job. Same loss class, deliberately left to a
# follow-up rather than widened into this change. Distinct artifact name:
# both jobs can fail in one run and upload-artifact v4+ rejects
# duplicate names.
- name: 'Upload disk-pressure samples'

View file

@ -343,6 +343,16 @@ describe('post-merge push lane', () => {
expect(String(ci.jobs.lint_and_static['runs-on'])).toBe(
'${{ fromJSON(needs.classify_pr.outputs.ubuntu_runner || \'["ubuntu-latest"]\') }}',
);
// The needs edge is what makes every needs.classify_pr.outputs.* this
// describe relies on non-empty at runtime. Drop it and the lane silently
// decouples from the classifier: runs-on falls through the `|| '…'` to
// hosted runners, skip_ci reads '' so a release-sync PR promised a no-op
// pass runs the full battery, and `Use trusted CI profile` takes its
// documented degraded empty→full fallback.
expect(
[].concat(ci.jobs.lint_and_static.needs ?? []),
'lint_and_static must consume the classifier',
).toContain('classify_pr');
});
it('lint_and_static keeps the minimal token its required-check role needs', () => {
@ -393,10 +403,21 @@ describe('post-merge push lane', () => {
);
const t = byName('test');
const l = byName('lint_and_static');
// The serializer is the guard's eyes: both sides of every comparison below
// flow through it, so a regression that drops nested fields compares `{}`
// to `{}` and the drift loop reports green — the exact array-replacer blind
// spot the recursion above replaced. Pin a known nested key on the
// serialized OUTPUT rather than on canon alone, because the regression that
// matters is a call site swapped back to an allowlist, which a canon-only
// fixture cannot see.
expect(t.get('Checkout')).toContain('"fetch-depth":1');
const shared = [...t.keys()].filter((n) => l.has(n));
// The prelude is what is duplicated; if this floor ever drops, steps
// were renamed apart and the guard is no longer guarding anything.
expect(shared.length).toBeGreaterThanOrEqual(12);
// were renamed apart and the guard is no longer guarding anything. 13 and
// not 12: the collector is deliberately in both jobs, so it is in the
// intersection, and a floor below the real count lets any one shared step
// be deleted out of either job without turning this red.
expect(shared.length).toBeGreaterThanOrEqual(13);
for (const n of shared) {
expect(l.get(n), `step "${n}" drifted between the two jobs`).toBe(
t.get(n),