qwen-code/integration-tests/globalSetup.test.ts
qwen-code-dev-bot e45ef59959
test(e2e): isolate protocol checks and ACP test state (#11286)
* fix(test): stop measuring model latency anywhere under CI (#11271)

The prompt-latency probe in qwen-serve-baseline sends 20 real prompts
through the shared OpenAI-compatible gateway and asserts p99 < 60s. One
gateway-queued prompt in twenty fails it, and each vitest retry re-issues
all 20 prompts into the same degraded window, so a single slow window
fails every attempt and turns the shard red.

Since the pool skip (#11004) the probe runs on exactly one E2E leg —
macOS shard 2/2, the only shard holding the file — and that leg failed
twice within six hours on unrelated commits (runs 34070970091 and
34088422718, both full-duration with retries consumed) while every Linux
leg, macOS shard 1/2, and the OpenTUI leg stayed green. The measured
quantity is gateway weather, not the daemon; the same argument the pool
skip made for host contention applies to the shared gateway.

Skip the probe whenever CI is set (any populated CI marker, matching the
repo's other CI checks), keeping the self-hosted disjunct so a
pool-shaped shell outside CI keeps its specific skip reason. The probe
still runs off CI on a credential, and
QWEN_BASELINE_ENABLE_PROMPT_LATENCY=1 force-runs it anywhere. The skip
reason distinguishes CI gateway contention from pool host contention.

* fix(test): tolerate transient model-serving errors in the acp plan-mode case (#11271)

The CI-wide prompt-latency skip rested on a misattribution: in both
cited runs the latency probe ran and passed, and the macOS leg went
red on cli/acp-integration.test.ts > blocks write tools in plan mode —
killed by a transient model-serving -32603 returned over ACP, with the
retry attempt then tripping ENOTEMPTY on the previous attempt's
leftover .qwen-home. Revert the CI skip, re-issue the plan-mode
session/prompt on the observed gateway -32603 shape (bounded at 3
attempts, 2s apart, inside the existing per-request timeout), and let
the TestRig.setup directory reset retry through the mid-delete refill
race the way globalSetup's teardown already does.

* test(e2e): isolate protocol checks from model service

* fix(test): surface fake-server setup failures in json-output teardown (#11271)

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(test): close ACP teardown race and tighten fake-server E2E (#11271)

- Move per-agent QWEN_HOME out of rig.testDir: the agent keeps writing
  there ~300ms after exit, racing global teardown's recursive rm
  (ENOTEMPTY). cleanup() now removes it with retries.
- Drop the plan-mode test's permissionHandler: the plan-mode guard
  short-circuits before any permission request, so it could never run
  and its comment claimed coverage the test does not have.
- Name the collected tool-call events in the plan-mode assertion so a
  wire-shape drift is distinguishable from a plan-mode let-through.
- Stub loopback NO_PROXY in json-output tests: an inherited
  HTTP(S)_PROXY otherwise tunnels the fake-server POST and the cases
  time out. Two-chunk fixture restores multi-delta accumulator coverage.
- Make the auth-mismatch case hermetic via fakeModelArgs instead of the
  ambient OPENAI_* secrets.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(test): reclaim leaked ACP qwen homes and unify scratch teardown (#11271)

---------

Co-authored-by: yiliang114 <effortyiliang@gmail.com>
Co-authored-by: 易良 <1204183885@qq.com>
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
2026-09-08 05:05:36 +00:00

217 lines
7.5 KiB
TypeScript

/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/
import { spawn } from 'node:child_process';
import { existsSync } from 'node:fs';
import {
mkdir,
mkdtemp,
readFile,
readdir,
rm,
utimes,
writeFile,
} from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { ACP_HOME_PREFIX } from './scratch-dir.js';
// The env keys globalSetup's setup() writes, saved so a case can restore the
// suite-wide values after re-importing the module and running its lifecycle.
const SETUP_ENV_KEYS = [
'INTEGRATION_TEST_FILE_DIR',
'QWEN_CODE_INTEGRATION_TEST',
'TELEMETRY_LOG_FILE',
'E2E_TEST_FILE_DIR',
'TEST_CLI_PATH',
'VERBOSE',
'KEEP_OUTPUT',
] as const;
describe('globalSetup memory-file save/restore', () => {
let qwenHome: string;
let savedEnv: Map<string, string | undefined>;
beforeEach(async () => {
qwenHome = await mkdtemp(join(tmpdir(), 'qwen-globalsetup-test-'));
savedEnv = new Map(
[...SETUP_ENV_KEYS, 'QWEN_HOME'].map((key) => [key, process.env[key]]),
);
process.env['QWEN_HOME'] = qwenHome;
// Let teardown remove the run directories this case creates.
process.env['KEEP_OUTPUT'] = 'false';
});
afterEach(async () => {
for (const [key, value] of savedEnv) {
if (value === undefined) {
delete process.env[key];
} else {
process.env[key] = value;
}
}
vi.resetModules();
await rm(qwenHome, { recursive: true, force: true });
});
// memoryFilePath is captured at module import time, so point QWEN_HOME at
// the scratch dir BEFORE a fresh import of the module.
async function loadGlobalSetup() {
vi.resetModules();
return import('./globalSetup.js');
}
it('restores the saved memory file after the run', async () => {
await writeFile(join(qwenHome, 'QWEN.md'), 'original content', 'utf-8');
const { setup, teardown } = await loadGlobalSetup();
await setup();
await writeFile(join(qwenHome, 'QWEN.md'), 'mutated by tests', 'utf-8');
await expect(teardown()).resolves.toBeUndefined();
await expect(readFile(join(qwenHome, 'QWEN.md'), 'utf-8')).resolves.toBe(
'original content',
);
});
it('does not exit an all-green run red when the restore cannot write', async () => {
// The persistent pool runners can carry a readable-but-unwritable
// QWEN.md left behind by a privileged job; before #10325 the teardown
// restore threw on it and exited every all-green E2E run on that host
// red with no failing test. Swap the file for a directory after setup()
// read it — the write then fails regardless of privilege, since root
// bypasses permission bits.
await writeFile(join(qwenHome, 'QWEN.md'), 'original content', 'utf-8');
const { setup, teardown } = await loadGlobalSetup();
await setup();
await rm(join(qwenHome, 'QWEN.md'), { force: true });
await mkdir(join(qwenHome, 'QWEN.md'));
await expect(teardown()).resolves.toBeUndefined();
});
});
describe('globalSetup hermetic qwen home', () => {
let tmpRoot: string;
let savedEnv: Map<string, string | undefined>;
beforeEach(async () => {
tmpRoot = await mkdtemp(join(tmpdir(), 'qwen-globalsetup-tmp-'));
savedEnv = new Map(
[...SETUP_ENV_KEYS, 'QWEN_HOME', 'TMPDIR'].map((key) => [
key,
process.env[key],
]),
);
// The scratch home is only created when nothing has pinned QWEN_HOME, and
// it lands in the OS temp dir — redirect that so the case owns what the
// module picks. Both are read at import time, so they must be set first.
delete process.env['QWEN_HOME'];
process.env['TMPDIR'] = tmpRoot;
process.env['KEEP_OUTPUT'] = 'false';
});
afterEach(async () => {
for (const [key, value] of savedEnv) {
if (value === undefined) {
delete process.env[key];
} else {
process.env[key] = value;
}
}
vi.resetModules();
await rm(tmpRoot, { recursive: true, force: true });
});
async function loadGlobalSetup() {
vi.resetModules();
return import('./globalSetup.js');
}
it('points the run at its own qwen home', async () => {
const { setup, teardown } = await loadGlobalSetup();
await setup();
const home = process.env['QWEN_HOME'];
expect(home?.startsWith(tmpRoot)).toBe(true);
await expect(teardown()).resolves.toBeUndefined();
expect(existsSync(home!)).toBe(false);
});
it('sweeps a stale per-agent ACP home a torn-down run left behind', async () => {
// cli/acp-integration.test.ts gives each spawned agent its own QWEN_HOME
// directly under the OS temp dir, and a worker torn down mid-test never
// runs the cleanup that removes it. The home is reclaimed only while its
// prefix stays nested under the sweeper's — ACP_HOME_PREFIX is the exact
// string the creation site builds the name from.
const leakedHome = join(tmpRoot, `${ACP_HOME_PREFIX}leaked`);
await mkdir(leakedHome, { recursive: true });
const leakedAt = new Date(Date.now() - 48 * 60 * 60 * 1000);
await utimes(leakedHome, leakedAt, leakedAt);
const { setup, teardown } = await loadGlobalSetup();
await setup();
expect(existsSync(leakedHome)).toBe(false);
await teardown();
});
it('does not exit an all-green run red when the scratch home cannot be removed', async () => {
// A CLI child that outlives its test keeps writing under `debug/`, so the
// removal walk reaches a directory that refills before the rmdir and
// throws ENOTEMPTY. That is how this cleanup first exited an all-green
// E2E run red, the same way the memory-file restore did in #10325. Stand
// in for that child with a writer that keeps the directory refilling —
// unlike a permission trick, it does not depend on not running as root.
const { setup, teardown } = await loadGlobalSetup();
await setup();
const debugDir = join(process.env['QWEN_HOME']!, 'debug');
await mkdir(debugDir, { recursive: true });
const writer = spawn(
process.execPath,
[
'-e',
// A tight loop, not a timer: the removal walk only fails when a file
// appears between its readdir and its rmdir, and a millisecond timer
// is far too slow to land inside that window. Self-limiting, so a
// child that outlives the kill below cannot spin indefinitely.
`const fs = require('node:fs');
const dir = ${JSON.stringify(debugDir)};
const stopAt = Date.now() + 30_000;
while (Date.now() < stopAt) {
try {
fs.mkdirSync(dir, { recursive: true });
fs.writeFileSync(dir + '/' + process.hrtime.bigint() + '.log', 'x');
} catch {}
}`,
],
{ stdio: 'ignore' },
);
try {
// Node takes tens of milliseconds to boot, and the removal walk finishes
// in about one — without waiting for the writer to actually be producing,
// teardown would clear an idle directory and the case would pass against
// the very bug it exists to catch.
const deadline = Date.now() + 10_000;
while ((await readdir(debugDir)).length < 50) {
if (Date.now() > deadline) {
throw new Error('writer never started refilling the debug directory');
}
await new Promise((resolve) => setTimeout(resolve, 20));
}
await expect(teardown()).resolves.toBeUndefined();
} finally {
writer.kill('SIGKILL');
}
});
});