qwen-code/scripts/upload-aliyun-oss-assets.js
易良 bb518cee86
ci: host PR evidence outside Git (#9985)
* ci: host PR evidence outside Git

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* ci: pin node and ossutil paths in privileged OSS uploads

The publish-verify evidence upload resolved `node` through a PATH
prefixed with $RUNNER_TEMP — the one directory PR code can write to
via the verify container's bind mount — so a planted fake `node`
there won interpreter resolution inside the step that holds
CI_BOT_PAT and the OSS config (PATH hijack). Resolve node under the
inherited clean PATH and run ossutil from a fresh job-private copy
of the sha256-verified binary installed above instead of resolving
either through $RUNNER_TEMP. Apply the same shape to
web-shell-visuals-publish.yml so the pattern cannot regress if that
job ever moves to a persistent pool.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* ci: make OSS setup best-effort and pair bucket/base-URL defaults

Mirror qwen-triage.yml in web-shell-visuals-publish.yml: mark the
ossutil install and credential-config steps continue-on-error so a
setup failure cannot abort the no-image marker-comment path (the
upload itself stays loud when images are present). In both
publishers, derive the default ALIYUN_OSS_PUBLIC_BASE_URL from
ALIYUN_OSS_BUCKET so overriding only one of the two vars cannot
post comment links that 404 against (or show stale objects from)
the other bucket. Re-record both workflow sizes in the baseline.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* ci: enrich evidence-upload warning with local preconditions

The single "Failed to upload evidence images" warning could not
distinguish its causes: the ossutil install and credential-config
steps run with continue-on-error and render green when they fail, so
a missing binary or config looks identical to a transient upload
blip or rotated keys. Probe both preconditions into the warning
(ossutil=ok/MISSING, config=ok/MISSING) so the investigator sees the
gap without trawling three steps of logs.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* test: enforce the uploader flag contract and pin the production arm

The evidence-hosting stub shifted --config away unvalidated, and every
harness run sets VERIFY_ASSETS_UPLOADER, so the production
`node "$uploader"` arm and its --config flag were never exercised:
dropping --config from the workflow kept every test green while the
real uploader exits 1 on it, silently degrading every /verify report
to text-only. Make the stub reject a missing/empty --bucket, --config
or --prefix, and pin the production invocation shape (inherited-PATH
node resolution plus all three ossutil flags). Mutation-probed: with
--config removed from the workflow both tests now fail.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* test: pin the ossutil credential lifecycle in both publishers

Nothing pinned the credential lifecycle this PR adds — the
sha256-checked ossutil install, the credential-config step, and the
if:always() cleanup — so a future edit dropping any of them (or the
always() condition) would leave every test green while the OSS key
pair persists in $RUNNER_TEMP on the persistent ecs-qwen pool. Add
shape assertions for all three steps in qwen-triage.yml and
web-shell-visuals-publish.yml, mirroring the release/desktop wiring
pins in install-script.test.js and desktop-oss-workflow.test.js.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* docs: align assets-repo guidance with the self-targeting guard

The PR-review workflow now blanks a QWEN_REVIEW_ASSETS_REPO that equals
the repository under review, but the user doc, the bundled review skill,
and the parseAssetsRepo unset-error still recommended pointing it at the
repo under review — so a maintainer following the published guidance
landed in the "not set" refusal path, and the refusal message
re-recommended the exact value the guard had just rejected. Update all
three surfaces to the new contract: a dedicated external image-host
repository (or a fork/scratch repo), with unset or self-targeting
values deliberately degrading to prose and local artifact paths.

Co-authored-by: Qwen-Coder <qwen-coder @alibabacloud.com>

* test: execute the visuals hosting block against a stub uploader

The workflow-level assertions covered the OSS upload path with shape
regexes only, so a --prefix that drifts away from RAW_BASE (every
preview URL 404s) or a dropped --config line (the uploader hard-fails
before any comment posts) both left the suite green. Extract the
hosting block and run it against a stub uploader that mirrors the real
flag contract and records the upload destination, in the same pattern
as the verify path in scripts/tests/qwen-triage-workflow.test.js: the
images arm must land the staged files under
$STUB_ROOT/$ALIYUN_OSS_BUCKET/pr-assets/web-shell-visuals/<pr>/<sha>
with the exact --bucket/--config/--prefix wiring, and RAW_BASE must
equal that layout; the no-change arm must never invoke the uploader.

Co-authored-by: Qwen-Coder <qwen-coder @alibabacloud.com>

* fix(ci): gate ossutil consumers on verified install

Prevent the persistent publish runner from executing a stale binary under RUNNER_TEMP when the best-effort install step fails. Pin both the install outcome gate and the hardened PATH invocation in focused workflow tests.

Co-authored-by: Qwen-Coder <qwen-coder @alibabacloud.com>

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(ci): grant publisher checkout access

Co-authored-by: Qwen-Coder <qwen-coder @alibabacloud.com>

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* test(ci): pin OSS publisher failure paths

Co-authored-by: Qwen-Coder <qwen-coder @alibabacloud.com>

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* test(ci): exercise production OSS dispatch

Co-authored-by: Qwen-Coder <qwen-coder @alibabacloud.com>

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(ci): isolate OSS publisher on hosted runner

* fix(ci): sync size baseline and stale runner comments

Co-authored-by: Qwen-Coder <qwen-coder @alibabacloud.com>

* fix(ci): give each visuals publish a fresh OSS prefix

The web-shell prefix was pr-assets/web-shell-visuals/<pr>/<head-sha>, so a
re-run for the same head wrote back over the object keys the already-posted
comment referenced. GitHub serves comment images through a caching proxy, so
reviewers would keep seeing the previous run's screenshots at a URL whose
bytes had changed. The Git-backed design got cache-busting free from the
per-run commit SHA; add the run id back to restore it (the verify lane
already carries <run-id>-<attempt>).

Also:

- Resolve the destination as ALIYUN_OSS_PR_ASSETS_BUCKET -> ALIYUN_OSS_BUCKET
  -> qwen-code-assets in both publishers, with the public base URL derived
  from whichever wins. Unset, this is byte-for-byte today's behaviour; set,
  it moves untrusted PR-derived evidence off the bucket that also serves
  release, desktop, and live-host downloads without a workflow change.
- Sparse-check out /package.json alongside the uploader: it is ESM in a .js
  file, and package.json declares engines >= 22.0 while neither publisher
  pins a Node, so without the "type": "module" marker the uploader only
  parses on the Node versions that infer module syntax.
- Require the credential file, not just a green install outcome, before
  dispatching the /verify upload. The configure step is continue-on-error,
  so a missing or rotated secret otherwise cost three retry backoffs per
  image before degrading to the same text-only report.
- Drop the stale "runs on the persistent ECS pool" premise from the
  clear-stale-downloads step; publish-verify is on ubuntu-latest now.

* fix(ci): retry the ossutil download in both PR publishers

The release syncs download ossutil with no retry, which is fine there: a
red run is rerun by hand. Here the consequence differs. A CDN blip in the
web-shell publisher leaves no ossutil for the upload, and since the upload
aborts the step under set -e, the PR loses its preview comment entirely
rather than degrading; in the verify publisher it silently costs the report
its evidence images. Three retries with --retry-all-errors covers the
transient class without changing any success path.

Also reword the unset-QWEN_REVIEW_ASSETS_REPO refusal. It ended with a flat
claim about the PR-review workflow blanking a self-targeting designation,
which reads as a non sequitur to the far more common reader: someone running
the CLI locally who simply never set the variable. Keep the diagnostic — a
maintainer whose repository variable IS set needs it — but scope it to CI
and say why self-targeting is discouraged in the first place.

* fix(ci): include run attempt in the web-shell OSS asset prefix

The workflow_run id is stable across re-run attempts, so a maintainer
re-run of the visuals publish overwrote the exact object keys an
already-posted comment references (camo keeps rendering attempt-1
shots). Append run_attempt to the prefix, mirroring the verify lane,
and cover same-runId attempt-1-vs-attempt-2 in the hosting harness.

Co-authored-by: Qwen-Coder <qwen-coder @alibabacloud.com>

* fix(ci): close out PR evidence hosting review findings

- The web-shell visuals prefix gains the run attempt: a re-run keeps
  the run id and only increments the attempt, so the old prefix let a
  same-head re-run write back over the object keys an already-posted
  comment references (camo keeps serving the stale screenshots).
  Mirrors the verify lane <run-id>-<attempt>; the re-run test now
  models same-run attempt 1 vs 2, not just distinct run ids.
- The QWEN_REVIEW_ASSETS_REPO self-targeting guard now trims and
  compares case-insensitively before the CLI reads the value, so
  padded or case-shifted self-references degrade like an unset one.
- Both publisher checkouts pin ref: github.sha (the default-branch
  head on their events) instead of relying on default resolution.
- The ossutil download retry budget fits the 10-minute job cap in
  both publishers (~6.1 min worst case; was ~20 min).
- Upload attempts are bounded via OSS_UPLOAD_ATTEMPT_TIMEOUT_MS so a
  stalled ossutil degrades to the text-only report / re-triggerable
  publish instead of burning the job cap; release syncs stay
  unbounded (the knob is opt-in).
- Fix the serving-model comment (the extension allowlist, not the
  magic bytes, is what pins the served Content-Type) and pin what
  earlier suites left unpinned: sparse-checkout lists, bucket-to-URL
  derivation, curl retry flags, persist-credentials, and the ossutil
  isolation lines. Both uploader stubs now require the --config file
  to exist, so config drift turns the harnesses red.
- runHostingBlock tears down its mkdtemp fixture instead of leaking
  it on every call.

# Conflicts:
#	.github/scripts/web-shell-visuals-publish.test.mjs
#	.github/workflows/web-shell-visuals-publish.yml
#	docs/design/2026-08-25-pr-evidence-oss-hosting.md

* fix(ci): gate visuals-publish Configure step on ossutil install outcome

The Install ossutil step keeps continue-on-error: true so a failed install must not block the no-image marker-comment path, but the unguarded Configure Aliyun OSS credentials step still ran and failed confusingly on the missing binary. Give the install step the install-ossutil id and gate Configure on its success outcome, matching the qwen-triage.yml twins.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

---------

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Co-authored-by: Qwen-Coder <qwen-coder @alibabacloud.com>
Co-authored-by: Shaojin Wen <shaojin.wensj@alibaba-inc.com>
2026-08-29 04:36:57 +00:00

201 lines
5.2 KiB
JavaScript

#!/usr/bin/env node
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/
import path from 'node:path';
import { spawnSync } from 'node:child_process';
import { fail, isMainModule, readOptionValue } from './release-script-utils.js';
const MAX_UPLOAD_ATTEMPTS = 3;
const INITIAL_BACKOFF_MS = 2000;
const ATTEMPT_TIMEOUT_ENV = 'OSS_UPLOAD_ATTEMPT_TIMEOUT_MS';
if (isMainModule(import.meta.url)) {
try {
main(process.argv.slice(2));
} catch (error) {
console.error(error instanceof Error ? error.message : error);
process.exitCode = 1;
}
}
function main(argv) {
const args = parseUploadArgs(argv);
if (args.help) {
printUsage();
return;
}
uploadAssets(args, { attemptTimeoutMs: resolveAttemptTimeoutMs() });
}
// A stalled ossutil (a black-hole socket that accepts but never progresses)
// must not hang the caller forever: the PR publishers run inside a
// 10-minute job cap, and one stuck upload would burn the cap and lose the
// whole report/comment. The bound is opt-in via env so the release syncs —
// which move much larger files — keep today's unbounded behaviour exactly.
function resolveAttemptTimeoutMs() {
const raw = process.env[ATTEMPT_TIMEOUT_ENV];
if (!raw) {
return 0;
}
const value = Number(raw);
if (!Number.isInteger(value) || value < 0) {
fail(
`${ATTEMPT_TIMEOUT_ENV} must be a non-negative integer of milliseconds, got ${JSON.stringify(raw)}.`,
);
}
return value;
}
function printUsage() {
console.log(`Usage: node scripts/upload-aliyun-oss-assets.js [options] ASSET...
Uploads local assets to a public Aliyun OSS prefix via ossutil.
Options:
--bucket NAME OSS bucket name.
--config PATH ossutil config path.
--prefix PREFIX Destination object prefix.
-h, --help Show this help message.
`);
}
function parseUploadArgs(argv) {
const args = {
assets: [],
bucket: '',
config: '',
help: false,
prefix: '',
};
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
if (arg === '--help' || arg === '-h') {
args.help = true;
continue;
}
if (arg === '--bucket') {
args.bucket = readOptionValue(argv, index, arg);
index += 1;
continue;
}
if (arg === '--config') {
args.config = readOptionValue(argv, index, arg);
index += 1;
continue;
}
if (arg === '--prefix') {
args.prefix = readOptionValue(argv, index, arg).replace(/\/+$/, '');
index += 1;
continue;
}
if (arg.startsWith('-')) {
fail(`Unknown option: ${arg}`);
}
args.assets.push(arg);
}
if (args.help) {
return args;
}
if (!args.bucket) {
fail('--bucket requires a value');
}
if (!args.config) {
fail('--config requires a value');
}
if (!args.prefix) {
fail('--prefix requires a value');
}
if (args.assets.length === 0) {
fail('At least one ASSET path is required');
}
return args;
}
function uploadAssets(
{ assets, bucket, config, prefix },
{
ossutilCommand = 'ossutil',
ossutilCommandArgs = [],
attemptTimeoutMs = 0,
} = {},
) {
for (const asset of assets) {
const key = `${prefix}/${path.basename(asset)}`;
uploadWithRetry(asset, bucket, key, config, {
ossutilCommand,
ossutilCommandArgs,
attemptTimeoutMs,
});
}
}
function uploadWithRetry(
asset,
bucket,
key,
config,
{ ossutilCommand, ossutilCommandArgs, attemptTimeoutMs },
) {
for (let attempt = 1; attempt <= MAX_UPLOAD_ATTEMPTS; attempt += 1) {
const result = spawnSync(
ossutilCommand,
[
...ossutilCommandArgs,
'cp',
asset,
`oss://${bucket}/${key}`,
'-c',
config,
'-f',
'--acl',
'public-read',
],
{
stdio: 'inherit',
// 0 disables the bound (the release syncs' default); a positive
// value SIGKILLs an attempt that stalls past it instead of letting
// it hang the caller out to the job cap.
...(attemptTimeoutMs > 0
? { timeout: attemptTimeoutMs, killSignal: 'SIGKILL' }
: {}),
},
);
// A timeout surfaces as error.code ETIMEDOUT and a killed child — that
// is a retryable attempt failure, not a spawn error to rethrow.
if (result.error && result.error.code !== 'ETIMEDOUT') {
throw result.error;
}
if (result.status === 0) {
return;
}
if (attempt < MAX_UPLOAD_ATTEMPTS) {
const delayMs = INITIAL_BACKOFF_MS * 2 ** (attempt - 1);
console.warn(
`Upload attempt ${attempt}/${MAX_UPLOAD_ATTEMPTS} failed for ${path.basename(asset)}${
result.error ? ' (timed out)' : ''
}, retrying in ${delayMs / 1000}s...`,
);
sleepSync(delayMs);
}
}
fail(
`ossutil failed after ${MAX_UPLOAD_ATTEMPTS} attempts while uploading ${asset}`,
);
}
// Cross-platform synchronous sleep. `spawnSync('sleep', ...)` is unavailable
// on Windows runners; Atomics.wait blocks the current thread without spawning.
function sleepSync(ms) {
Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);
}
export { parseUploadArgs, resolveAttemptTimeoutMs, uploadAssets };