mirror of
https://github.com/openclaw/openclaw.git
synced 2026-10-03 09:39:25 +00:00
perf(test): admit isolated Gateway readers to the existing worker pool (#146550)
* perf(test): admit isolated Gateway readers to the existing worker pool * test: align Gateway scheduling and sandbox cache expectations * test(ui): retain safe Gateway failure state in CI logs
This commit is contained in:
parent
b0752a8c55
commit
3c16ced267
7 changed files with 383 additions and 125 deletions
|
|
@ -70,9 +70,9 @@ Expanded serial large/small jobs admit 210 predicted seconds; eligible hybrid pa
|
|||
|
||||
Every selected project discovers Chromium. The first selected bundle-consuming project builds one private production bundle/preview and publishes its URL through Vitest's invocation-scoped root context; later consumers share it until invocation teardown. Standalone projects have no bundle setup or URL bridge, so standalone-only selections skip that build. Enabled manual proof capture uses the shared upload directory, including the MCP and Logs suites.
|
||||
|
||||
The dedicated real-Gateway job runs its complete selected inventory in one invocation through `test/vitest/vitest.ui-e2e-prebuilt.config.ts`. It requires a clean checkout and completed runtime, private QA, and canonical Control UI artifacts from `OPENCLAW_BUILD_PRIVATE_QA=1 pnpm build:ci-artifacts`. Source and built outputs must remain unchanged until all workers and children finish. A readiness failure stops the invocation without rebuilding or falling back to another config. MCP conformance owns a source server and runs serially first; the remaining files then share the existing two-worker limit. The invocation preview builds its own private output from the same source. This adds no CI jobs or shards. The ordinary local config keeps real-Gateway files serial, and frozen targets lacking the prebuilt config retain their original serial command.
|
||||
The dedicated real-Gateway job runs its complete selected inventory in one invocation through `test/vitest/vitest.ui-e2e-prebuilt.config.ts`. It requires a clean checkout and completed runtime, private QA, and canonical Control UI artifacts from `OPENCLAW_BUILD_PRIVATE_QA=1 pnpm build:ci-artifacts`. Source and built outputs must remain unchanged until all workers and children finish. A readiness failure stops the invocation without rebuilding or falling back to another config. Files outside the prebuilt config’s shared-reader/writer allowlist run serially first; audited fixtures with private HOME, state, ports, and cleanup then share the existing two-worker limit. The allowlist admits both invocation-preview consumers and fixtures serving canonical built UI bytes through their own prepared Gateway child. The invocation preview builds its own private output from the same source. This adds no CI jobs or shards. The ordinary local config keeps real-Gateway files serial, and frozen targets lacking the prebuilt config retain their original serial command.
|
||||
|
||||
A controlled Linux comparison covering all 14 files and 25 tests reduced invocation elapsed time from 309.374 to 202.027 seconds. This measures the test invocation, not complete CI timing or achievement of the CI latency target.
|
||||
The original two-worker rollout had a controlled Linux comparison covering its then-complete inventory of 14 files and 25 tests, reducing invocation elapsed time from 309.374 to 202.027 seconds. Those historical results do not measure later allowlist additions, complete CI timing, or achievement of the CI latency target.
|
||||
|
||||
Eligible `control-ui` rows request `blacksmith-32vcpu-ubuntu-2404`; the browser-extension row keeps the 8-vCPU request, and eligible real-Gateway jobs request the 32-class. Backend, event, contributor-trust and cache-write boundaries are unchanged, including hybrid first attempts and trusted contributor forks. In [run 33692146223](https://github.com/openclaw/openclaw/actions/runs/33692146223), the two slowest UI rows requested the 8-vCPU label but reported two CPUs; their 356/383-second test steps set the 8:20 non-Windows wall. The same run's 32-vCPU jobs reported eight CPUs. The larger request added no workers. In [run 33695337496](https://github.com/openclaw/openclaw/actions/runs/33695337496), all twelve UI rows reported eight CPUs and finished by 4:38 from workflow creation, with 102–145-second test steps. That margin supports consolidating to six rows; reduced-row timings still require native proof. Stale file weights also need the existing refit's independent-run and replacement thresholds, rather than a one-run manual adjustment.
|
||||
|
||||
|
|
|
|||
|
|
@ -242,7 +242,7 @@ Native dependency policy:
|
|||
- No provider keys required; `OPENCLAW_UI_E2E_SKIP_REAL_GATEWAY=1` excludes real-Gateway suites
|
||||
- Browser dependency must be present (`pnpm --dir ui exec playwright install chromium`)
|
||||
|
||||
The dedicated real-Gateway CI job uses `test/vitest/vitest.ui-e2e-prebuilt.config.ts` after `OPENCLAW_BUILD_PRIVATE_QA=1 pnpm build:ci-artifacts` completes in a clean checkout. Keep source and built outputs unchanged until all workers and children finish. MCP conformance runs serially first, then the other 13 files share at most two workers in the same invocation, with no extra jobs or shards. Readiness failures stop execution without rebuilding or falling back. The ordinary local config keeps real-Gateway files serial; frozen targets without the prebuilt config keep their original serial command. See [CI](/ci) for the resource policy and bounded timing evidence.
|
||||
The dedicated real-Gateway CI job uses `test/vitest/vitest.ui-e2e-prebuilt.config.ts` after `OPENCLAW_BUILD_PRIVATE_QA=1 pnpm build:ci-artifacts` completes in a clean checkout. Keep source and built outputs unchanged until all workers and children finish. Files outside the prebuilt config’s shared-reader/writer allowlist run serially first. Audited fixtures own their HOME, state, ports, and cleanup, and share at most two workers in the same invocation, with no extra jobs or shards. Readiness failures stop execution without rebuilding or falling back. The ordinary local config keeps real-Gateway files serial; frozen targets without the prebuilt config keep their original serial command. See [CI](/ci) for the resource policy and bounded timing evidence.
|
||||
|
||||
### E2E: OpenShell backend smoke
|
||||
|
||||
|
|
|
|||
|
|
@ -530,48 +530,6 @@ describe("Control UI E2E resource ownership", () => {
|
|||
expect(result.rootWorkers).toBe(workers);
|
||||
}
|
||||
expect(result.files.filter((entry) => entry.phase === 1)).toEqual([
|
||||
{
|
||||
file: "ui/src/e2e/chat-agent-avatar.real-gateway.e2e.test.ts",
|
||||
project: "ui-e2e-serial-standalone",
|
||||
phase: 1,
|
||||
workers: 1,
|
||||
fileParallelism: false,
|
||||
},
|
||||
{
|
||||
file: "ui/src/e2e/chat-composer-websearch-kill-switch.real-gateway.e2e.test.ts",
|
||||
project: "ui-e2e-serial-standalone",
|
||||
phase: 1,
|
||||
workers: 1,
|
||||
fileParallelism: false,
|
||||
},
|
||||
{
|
||||
file: "ui/src/e2e/chat-flow.catalog-bootstrap.e2e.test.ts",
|
||||
project: "ui-e2e-serial",
|
||||
phase: 1,
|
||||
workers: 1,
|
||||
fileParallelism: false,
|
||||
},
|
||||
{
|
||||
file: "ui/src/e2e/chat-stop-finished-run.real-gateway.e2e.test.ts",
|
||||
project: "ui-e2e-serial-standalone",
|
||||
phase: 1,
|
||||
workers: 1,
|
||||
fileParallelism: false,
|
||||
},
|
||||
{
|
||||
file: "ui/src/e2e/chat-thinking-metadata.real-gateway.e2e.test.ts",
|
||||
project: "ui-e2e-serial-standalone",
|
||||
phase: 1,
|
||||
workers: 1,
|
||||
fileParallelism: false,
|
||||
},
|
||||
{
|
||||
file: "ui/src/e2e/command-palette-catalog.real-gateway.e2e.test.ts",
|
||||
project: "ui-e2e-serial-standalone",
|
||||
phase: 1,
|
||||
workers: 1,
|
||||
fileParallelism: false,
|
||||
},
|
||||
{
|
||||
file: "ui/src/e2e/desktop-resize.real-gateway.e2e.test.ts",
|
||||
project: "ui-e2e-serial-standalone",
|
||||
|
|
@ -579,13 +537,6 @@ describe("Control UI E2E resource ownership", () => {
|
|||
workers: 1,
|
||||
fileParallelism: false,
|
||||
},
|
||||
{
|
||||
file: "ui/src/e2e/device-alias-rename.real-gateway.e2e.test.ts",
|
||||
project: "ui-e2e-serial",
|
||||
phase: 1,
|
||||
workers: 1,
|
||||
fileParallelism: false,
|
||||
},
|
||||
{
|
||||
file: "ui/src/e2e/device-platform-family.real-gateway.e2e.test.ts",
|
||||
project: "ui-e2e-serial-standalone",
|
||||
|
|
@ -607,27 +558,6 @@ describe("Control UI E2E resource ownership", () => {
|
|||
workers: 1,
|
||||
fileParallelism: false,
|
||||
},
|
||||
{
|
||||
file: "ui/src/e2e/model-catalog-partial-refresh.real-gateway.e2e.test.ts",
|
||||
project: "ui-e2e-serial-standalone",
|
||||
phase: 1,
|
||||
workers: 1,
|
||||
fileParallelism: false,
|
||||
},
|
||||
{
|
||||
file: "ui/src/e2e/model-picker-search.real-gateway.e2e.test.ts",
|
||||
project: "ui-e2e-serial-standalone",
|
||||
phase: 1,
|
||||
workers: 1,
|
||||
fileParallelism: false,
|
||||
},
|
||||
{
|
||||
file: "ui/src/e2e/profile-page.real-gateway.e2e.test.ts",
|
||||
project: "ui-e2e-serial",
|
||||
phase: 1,
|
||||
workers: 1,
|
||||
fileParallelism: false,
|
||||
},
|
||||
{
|
||||
file: "ui/src/e2e/quota-reset-status.real-gateway.e2e.test.ts",
|
||||
project: "ui-e2e-serial-standalone",
|
||||
|
|
@ -635,16 +565,9 @@ describe("Control UI E2E resource ownership", () => {
|
|||
workers: 1,
|
||||
fileParallelism: false,
|
||||
},
|
||||
{
|
||||
file: "ui/src/e2e/worker-initial-setup.real-gateway.e2e.test.ts",
|
||||
project: "ui-e2e-serial",
|
||||
phase: 1,
|
||||
workers: 1,
|
||||
fileParallelism: false,
|
||||
},
|
||||
]);
|
||||
const parallel = result.files.filter((entry) => entry.phase === 2);
|
||||
expect(parallel).toHaveLength(13);
|
||||
expect(parallel).toHaveLength(24);
|
||||
expect(parallel.every((entry) => entry.fileParallelism)).toBe(true);
|
||||
expect(parallel.every((entry) => entry.workers === result.rootWorkers)).toBe(true);
|
||||
for (const entry of parallel) {
|
||||
|
|
|
|||
|
|
@ -7,16 +7,29 @@ import { intersectIncludePatterns } from "./vitest.pattern-file.ts";
|
|||
import { createUiE2eVitestConfig, uiE2eRealGatewayTestFiles } from "./vitest.ui-e2e.config.ts";
|
||||
|
||||
// New real-Gateway files stay serial until their shared readers/writers are audited.
|
||||
// Listed fixtures own their HOME, state, ports, and cleanup; UI bytes are either
|
||||
// borrowed from the invocation preview or read by their prepared Gateway child.
|
||||
const parallelFiles = new Set([
|
||||
"ui/src/e2e/agent-file-lifecycle.real-gateway.e2e.test.ts",
|
||||
"ui/src/e2e/chat-agent-avatar.real-gateway.e2e.test.ts",
|
||||
"ui/src/e2e/chat-composer-websearch-kill-switch.real-gateway.e2e.test.ts",
|
||||
"ui/src/e2e/chat-flow.catalog-bootstrap.e2e.test.ts",
|
||||
"ui/src/e2e/chat-loading-performance.real-gateway.e2e.test.ts",
|
||||
"ui/src/e2e/chat-project-media.real-gateway.e2e.test.ts",
|
||||
"ui/src/e2e/chat-stop-finished-run.real-gateway.e2e.test.ts",
|
||||
"ui/src/e2e/chat-thinking-metadata.real-gateway.e2e.test.ts",
|
||||
"ui/src/e2e/chat-widget-sandbox.real-gateway.e2e.test.ts",
|
||||
"ui/src/e2e/command-palette-catalog.real-gateway.e2e.test.ts",
|
||||
"ui/src/e2e/control-ui-auth-transports.e2e.test.ts",
|
||||
"ui/src/e2e/cron-duration-save.real-gateway.e2e.test.ts",
|
||||
"ui/src/e2e/device-alias-rename.real-gateway.e2e.test.ts",
|
||||
"ui/src/e2e/logs-lifecycle.e2e.test.ts",
|
||||
"ui/src/e2e/model-catalog-partial-refresh.real-gateway.e2e.test.ts",
|
||||
"ui/src/e2e/model-picker-search.real-gateway.e2e.test.ts",
|
||||
"ui/src/e2e/profile-page.real-gateway.e2e.test.ts",
|
||||
"ui/src/e2e/session-progress-hovercard.real-gateway.e2e.test.ts",
|
||||
"ui/src/e2e/usage-sessions-owner-attribution.e2e.test.ts",
|
||||
"ui/src/e2e/worker-initial-setup.real-gateway.e2e.test.ts",
|
||||
"extensions/qa-lab/src/control-ui-media-transcript.real-gateway.e2e.test.ts",
|
||||
"extensions/qa-lab/src/session-host-command-state.real-gateway.e2e.test.ts",
|
||||
"extensions/qa-lab/src/control-ui-openclaw-delegation.real-gateway.e2e.test.ts",
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ import { spawnSync } from "node:child_process";
|
|||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { isRecord } from "@openclaw/normalization-core/record-coerce";
|
||||
import { chromium } from "playwright";
|
||||
import { describe, expect, inject, it } from "vitest";
|
||||
import type { AuthHealthSummary } from "../../../src/agents/auth-health.js";
|
||||
|
|
@ -16,6 +17,7 @@ import {
|
|||
} from "../../../test/e2e/qa-lab/runtime/quota-reset.test-support.js";
|
||||
import { createControlUiE2eArtifactDir } from "../test-helpers/control-ui-e2e-artifacts.js";
|
||||
import { waitForControlUiGatewayReady } from "../test-helpers/control-ui-e2e-readiness.js";
|
||||
import { captureControlUiE2eFailureDiagnostics } from "../test-helpers/control-ui-e2e.js";
|
||||
|
||||
type QuotaFixture = Awaited<ReturnType<typeof createQuotaResetFixture>>;
|
||||
type SavedState = {
|
||||
|
|
@ -164,6 +166,22 @@ async function captureFinalStatus(
|
|||
path: path.join(artifactDir, "provider-status.png"),
|
||||
animations: "disabled",
|
||||
});
|
||||
} catch (error) {
|
||||
const latestResponse = (method: string) => {
|
||||
const observation = observations.findLast(
|
||||
(entry) => isRecord(entry) && entry.action === "browser-rpc" && entry.method === method,
|
||||
);
|
||||
return isRecord(observation) ? observation.frame : undefined;
|
||||
};
|
||||
await captureControlUiE2eFailureDiagnostics(page, {
|
||||
error: error instanceof Error ? error : new Error("Quota final-status failure"),
|
||||
label: "quota-final-status",
|
||||
modelResponses: {
|
||||
list: latestResponse("models.list"),
|
||||
authStatus: latestResponse("models.authStatus"),
|
||||
},
|
||||
});
|
||||
throw error;
|
||||
} finally {
|
||||
await fs.writeFile(
|
||||
path.join(artifactDir, "rendered-page.json"),
|
||||
|
|
|
|||
|
|
@ -15,16 +15,121 @@ import {
|
|||
|
||||
describe("shared proof capture", () => {
|
||||
const tempDirs = useAutoCleanupTempDirTracker(afterEach);
|
||||
afterEach(() => vi.unstubAllEnvs());
|
||||
afterEach(() => {
|
||||
vi.unstubAllEnvs();
|
||||
vi.restoreAllMocks();
|
||||
document.body.replaceChildren();
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ shardIndex: "5", shardCount: "6" },
|
||||
{ shardIndex: undefined, shardCount: undefined },
|
||||
{ shardIndex: "5", shardCount: "6", failure: "none", sendLabel: "Send message" },
|
||||
{ shardIndex: undefined, shardCount: undefined, failure: "none", sendLabel: "Loading chat" },
|
||||
{
|
||||
shardIndex: undefined,
|
||||
shardCount: undefined,
|
||||
failure: "screenshot",
|
||||
sendLabel: "private-label",
|
||||
},
|
||||
{
|
||||
shardIndex: undefined,
|
||||
shardCount: undefined,
|
||||
failure: "storage",
|
||||
sendLabel: "Sending message...",
|
||||
},
|
||||
])(
|
||||
"retains each failure capture with its shard provenance ($shardIndex/$shardCount)",
|
||||
async ({ shardIndex, shardCount }) => {
|
||||
"retains safe failure state despite $failure failure ($sendLabel; $shardIndex/$shardCount)",
|
||||
async ({ shardIndex, shardCount, failure, sendLabel }) => {
|
||||
const parent = tempDirs.make("control-ui-failure-proof-");
|
||||
vi.stubEnv("OPENCLAW_UI_E2E_DIAGNOSTIC_DIR", parent);
|
||||
const diagnosticParent = failure === "storage" ? path.join(parent, "blocked") : parent;
|
||||
if (failure === "storage") {
|
||||
writeFileSync(diagnosticParent, "not a directory");
|
||||
}
|
||||
vi.stubEnv("OPENCLAW_UI_E2E_DIAGNOSTIC_DIR", diagnosticParent);
|
||||
const logs = vi.spyOn(console, "error").mockImplementation(() => {});
|
||||
vi.spyOn(performance, "getEntriesByType").mockReturnValue([]);
|
||||
const app = document.createElement("openclaw-app");
|
||||
Object.assign(app, {
|
||||
runtime: {
|
||||
context: {
|
||||
gateway: {
|
||||
snapshot: {
|
||||
phase: failure === "storage" ? "private-phase" : "connected",
|
||||
hello: { token: "private-hello" },
|
||||
},
|
||||
},
|
||||
agents: { state: { connected: true } },
|
||||
},
|
||||
},
|
||||
});
|
||||
const composer = document.createElement("div");
|
||||
composer.className = "agent-chat__composer-combobox";
|
||||
const textarea = document.createElement("textarea");
|
||||
textarea.value = failure === "storage" ? "" : "private-draft";
|
||||
textarea.disabled = failure === "storage";
|
||||
composer.append(textarea);
|
||||
const send = document.createElement("button");
|
||||
send.className = "chat-send-btn--send";
|
||||
send.setAttribute("aria-label", sendLabel);
|
||||
send.setAttribute("aria-busy", failure === "storage" ? "true" : "false");
|
||||
send.disabled = failure !== "none" || sendLabel === "Loading chat";
|
||||
const providerHead = document.createElement("div");
|
||||
providerHead.className = "model-providers__head";
|
||||
const badge = document.createElement("span");
|
||||
badge.className = "settings-status";
|
||||
badge.textContent = failure === "none" ? "Ready" : "private-badge";
|
||||
providerHead.append(badge);
|
||||
document.body.append(app, composer, send, providerHead);
|
||||
const modelResponses =
|
||||
failure === "none"
|
||||
? {}
|
||||
: {
|
||||
list: {
|
||||
ok: true,
|
||||
payload: {
|
||||
models:
|
||||
failure === "screenshot"
|
||||
? []
|
||||
: [
|
||||
{ id: "private-model", available: true },
|
||||
{ available: false, unavailableReason: "private-reason" },
|
||||
{ available: "private-availability" },
|
||||
],
|
||||
pendingProviders: [],
|
||||
providerOutcomes:
|
||||
failure === "screenshot"
|
||||
? []
|
||||
: [
|
||||
{
|
||||
provider: "private-provider",
|
||||
profileId: "private-profile",
|
||||
status: "ready",
|
||||
},
|
||||
{ status: "private-outcome" },
|
||||
],
|
||||
},
|
||||
},
|
||||
authStatus: {
|
||||
ok: true,
|
||||
payload: {
|
||||
providers:
|
||||
failure === "screenshot"
|
||||
? []
|
||||
: [
|
||||
{
|
||||
profiles: [
|
||||
{
|
||||
profileId: "private-profile",
|
||||
status: "ok",
|
||||
token: "private-token",
|
||||
},
|
||||
{ status: "expired" },
|
||||
{ status: "private-health" },
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
};
|
||||
vi.stubEnv("VITEST_SHARD_INDEX", shardIndex);
|
||||
vi.stubEnv("VITEST_SHARD_COUNT", shardCount);
|
||||
vi.stubEnv("SHARD_INDEX", shardIndex ? undefined : "unrelated-shard");
|
||||
|
|
@ -34,34 +139,113 @@ describe("shared proof capture", () => {
|
|||
writeFileSync(path.join(parent, "prior.png"), "prior-proof");
|
||||
// SAFETY: this fixture implements the Page boundary used by failure diagnostics.
|
||||
const page = {
|
||||
evaluate: async () => ({ marker: "failed-page" }),
|
||||
evaluate: async (read: () => unknown) => read(),
|
||||
isClosed: () => false,
|
||||
url: () => "http://127.0.0.1/chat",
|
||||
screenshot: async (options: { path: string }) => {
|
||||
expect(
|
||||
logs.mock.calls.some(([message]) => message === "[control-ui-e2e] failure state"),
|
||||
).toBe(true);
|
||||
if (failure === "screenshot") {
|
||||
throw new Error("private-screenshot-error");
|
||||
}
|
||||
writeFileSync(options.path, "failure-proof");
|
||||
return Buffer.from("failure-proof");
|
||||
},
|
||||
} as unknown as Page;
|
||||
for (let attempt = 0; attempt < 2; attempt += 1) {
|
||||
await captureControlUiE2eFailureDiagnostics(page, {
|
||||
error: new Error("Synthetic request timeout"),
|
||||
label: "chat.send",
|
||||
});
|
||||
const original = new Error("private-original-error");
|
||||
const failedAction = async () => {
|
||||
try {
|
||||
throw original;
|
||||
} catch (error) {
|
||||
await captureControlUiE2eFailureDiagnostics(page, {
|
||||
error: original,
|
||||
label: "chat.send",
|
||||
modelResponses,
|
||||
});
|
||||
throw error;
|
||||
}
|
||||
};
|
||||
await expect(failedAction()).rejects.toBe(original);
|
||||
}
|
||||
const directories = readdirSync(parent, { withFileTypes: true }).filter((entry) =>
|
||||
entry.isDirectory(),
|
||||
);
|
||||
expect(directories).toHaveLength(2);
|
||||
const summaries = logs.mock.calls
|
||||
.filter(([message]) => message === "[control-ui-e2e] failure state")
|
||||
.map(([, summary]) => summary);
|
||||
expect(summaries).toHaveLength(2);
|
||||
for (const summary of summaries) {
|
||||
expect(summary).toMatchObject({
|
||||
browser: {
|
||||
gatewayPhase: failure === "storage" ? "unknown" : "connected",
|
||||
connected: true,
|
||||
documentReadyState: expect.stringMatching(/^(?:loading|interactive|complete)$/u),
|
||||
providerStatuses: [
|
||||
{
|
||||
status: failure === "none" ? "Ready" : "unknown",
|
||||
length: badge.textContent.length,
|
||||
},
|
||||
],
|
||||
composer: {
|
||||
draftLength: failure === "storage" ? 0 : 13,
|
||||
nonempty: failure !== "storage",
|
||||
disabled: failure === "storage",
|
||||
send: {
|
||||
label: sendLabel === "private-label" ? "unknown" : sendLabel,
|
||||
labelLength: sendLabel.length,
|
||||
disabled: failure !== "none" || sendLabel === "Loading chat",
|
||||
busy: failure === "storage",
|
||||
},
|
||||
},
|
||||
},
|
||||
models: {
|
||||
listSeen: failure !== "none",
|
||||
listOk: failure === "none" ? null : true,
|
||||
models: failure === "none" ? null : failure === "screenshot" ? 0 : 3,
|
||||
available: failure === "none" ? null : failure === "screenshot" ? 0 : 1,
|
||||
unavailable: failure === "none" ? null : failure === "screenshot" ? 0 : 1,
|
||||
unknownAvailability: failure === "none" ? null : failure === "screenshot" ? 0 : 1,
|
||||
pendingProviders: failure === "none" ? null : 0,
|
||||
providerOutcomes:
|
||||
failure === "none"
|
||||
? null
|
||||
: {
|
||||
ready: failure === "storage" ? 1 : 0,
|
||||
"auth-rejected": 0,
|
||||
unavailable: 0,
|
||||
unknown: failure === "storage" ? 1 : 0,
|
||||
},
|
||||
authSeen: failure !== "none",
|
||||
authOk: failure === "none" ? null : true,
|
||||
profiles:
|
||||
failure === "none"
|
||||
? null
|
||||
: {
|
||||
ok: failure === "storage" ? 1 : 0,
|
||||
expiring: 0,
|
||||
expired: failure === "storage" ? 1 : 0,
|
||||
missing: 0,
|
||||
static: 0,
|
||||
unknown: failure === "storage" ? 1 : 0,
|
||||
},
|
||||
},
|
||||
});
|
||||
}
|
||||
expect(JSON.stringify(logs.mock.calls)).not.toContain("private-");
|
||||
expect(directories).toHaveLength(failure === "storage" ? 0 : 2);
|
||||
for (const directory of directories) {
|
||||
const root = path.join(parent, directory.name);
|
||||
const files = readdirSync(root);
|
||||
expect(files).toHaveLength(2);
|
||||
expect(files).toHaveLength(failure === "screenshot" ? 1 : 2);
|
||||
const reportFile = files.find((file) => file.endsWith(".json"));
|
||||
expect(reportFile).toBeDefined();
|
||||
const report = JSON.parse(readFileSync(path.join(root, reportFile!), "utf8"));
|
||||
expect(report).toMatchObject({
|
||||
label: "chat.send",
|
||||
captureErrors: [],
|
||||
captureErrors:
|
||||
failure === "screenshot" ? [expect.stringContaining("private-screenshot-error")] : [],
|
||||
ci: {
|
||||
githubJob: "checks-ui-e2e",
|
||||
runAttempt: "2",
|
||||
|
|
@ -70,8 +254,12 @@ describe("shared proof capture", () => {
|
|||
vitestShardCount: shardCount ?? null,
|
||||
},
|
||||
});
|
||||
expect(files).toContain(report.screenshot);
|
||||
expect(readFileSync(path.join(root, report.screenshot), "utf8")).toBe("failure-proof");
|
||||
if (failure === "screenshot") {
|
||||
expect(report.screenshot).toBeNull();
|
||||
} else {
|
||||
expect(files).toContain(report.screenshot);
|
||||
expect(readFileSync(path.join(root, report.screenshot), "utf8")).toBe("failure-proof");
|
||||
}
|
||||
}
|
||||
expect(readFileSync(path.join(parent, "prior.png"), "utf8")).toBe("prior-proof");
|
||||
},
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import path from "node:path";
|
|||
import { fileURLToPath } from "node:url";
|
||||
import type { HelloOk } from "@openclaw/gateway-protocol";
|
||||
import { normalizeAgentId } from "@openclaw/normalization-core/agent-id";
|
||||
import { asOptionalRecord } from "@openclaw/normalization-core/record-coerce";
|
||||
import { buildControlUiSessionPath } from "@openclaw/session-url-contract";
|
||||
import type { ConsoleMessage, Frame, Locator, Page, Request } from "playwright";
|
||||
import type { InlineConfig, Plugin, PreviewServer, ViteDevServer } from "vite";
|
||||
|
|
@ -3484,6 +3485,67 @@ function createMockGatewayControls(
|
|||
};
|
||||
}
|
||||
|
||||
type ControlUiE2eFailureDiagnosticsOptions = {
|
||||
error: Error;
|
||||
label: string;
|
||||
pageErrors?: string[];
|
||||
pageEvents?: ControlUiE2eDiagnosticEvent[];
|
||||
modelResponses?: { list?: unknown; authStatus?: unknown };
|
||||
};
|
||||
|
||||
function summarizeRecordedModelResponses(
|
||||
responses: NonNullable<ControlUiE2eFailureDiagnosticsOptions["modelResponses"]>,
|
||||
) {
|
||||
const list = asOptionalRecord(responses.list);
|
||||
const auth = asOptionalRecord(responses.authStatus);
|
||||
const catalog = asOptionalRecord(list?.payload);
|
||||
const health = asOptionalRecord(auth?.payload);
|
||||
const models = Array.isArray(catalog?.models) ? catalog.models : undefined;
|
||||
const profiles = Array.isArray(health?.providers)
|
||||
? health.providers.flatMap((provider) => {
|
||||
const record = asOptionalRecord(provider);
|
||||
return Array.isArray(record?.profiles) ? record.profiles : [];
|
||||
})
|
||||
: undefined;
|
||||
const statusCounts = (entries: unknown, allowed: string[]) => {
|
||||
if (!Array.isArray(entries)) {
|
||||
return null;
|
||||
}
|
||||
const statuses = entries.map(
|
||||
(entry) => allowed.find((status) => asOptionalRecord(entry)?.status === status) ?? "unknown",
|
||||
);
|
||||
return Object.fromEntries(
|
||||
[...allowed, "unknown"].map((status) => [
|
||||
status,
|
||||
statuses.filter((entry) => entry === status).length,
|
||||
]),
|
||||
);
|
||||
};
|
||||
return {
|
||||
listSeen: responses.list !== undefined,
|
||||
listOk: typeof list?.ok === "boolean" ? list.ok : null,
|
||||
models: models?.length ?? null,
|
||||
available:
|
||||
models?.filter((model) => asOptionalRecord(model)?.available === true).length ?? null,
|
||||
unavailable:
|
||||
models?.filter((model) => asOptionalRecord(model)?.available === false).length ?? null,
|
||||
unknownAvailability:
|
||||
models?.filter((model) => typeof asOptionalRecord(model)?.available !== "boolean").length ??
|
||||
null,
|
||||
pendingProviders: Array.isArray(catalog?.pendingProviders)
|
||||
? catalog.pendingProviders.length
|
||||
: null,
|
||||
providerOutcomes: statusCounts(catalog?.providerOutcomes, [
|
||||
"ready",
|
||||
"auth-rejected",
|
||||
"unavailable",
|
||||
]),
|
||||
authSeen: responses.authStatus !== undefined,
|
||||
authOk: typeof auth?.ok === "boolean" ? auth.ok : null,
|
||||
profiles: statusCounts(profiles, ["ok", "expiring", "expired", "missing", "static"]),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Capture a screenshot plus a browser/app-state report for a failed E2E wait.
|
||||
* Wired into mock-Gateway request timeouts automatically; boot/readiness waits
|
||||
|
|
@ -3492,20 +3554,12 @@ function createMockGatewayControls(
|
|||
*/
|
||||
export async function captureControlUiE2eFailureDiagnostics(
|
||||
page: Page,
|
||||
options: {
|
||||
error: Error;
|
||||
label: string;
|
||||
pageErrors?: string[];
|
||||
pageEvents?: ControlUiE2eDiagnosticEvent[];
|
||||
},
|
||||
options: ControlUiE2eFailureDiagnosticsOptions,
|
||||
): Promise<void> {
|
||||
try {
|
||||
await captureControlUiE2eFailureDiagnosticsUnsafe(page, options);
|
||||
} catch (captureError) {
|
||||
console.error("[control-ui-e2e] failed to capture failure diagnostics", {
|
||||
captureError,
|
||||
label: options.label,
|
||||
});
|
||||
} catch {
|
||||
console.error("[control-ui-e2e] failed to capture failure diagnostics");
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -3518,27 +3572,14 @@ async function captureControlUiE2eFailureDiagnosticsUnsafe(
|
|||
// The mock-Gateway installer keeps a per-page diagnostic ring; default to
|
||||
// it so ad-hoc test callers get console/navigation history for free.
|
||||
pageEvents = controlUiE2ePageDiagnostics.get(page) ?? [],
|
||||
}: {
|
||||
error: Error;
|
||||
label: string;
|
||||
pageErrors?: string[];
|
||||
pageEvents?: ControlUiE2eDiagnosticEvent[];
|
||||
},
|
||||
modelResponses,
|
||||
}: ControlUiE2eFailureDiagnosticsOptions,
|
||||
): Promise<void> {
|
||||
const configuredDir = process.env.OPENCLAW_UI_E2E_DIAGNOSTIC_DIR?.trim();
|
||||
const artifactDir = createControlUiE2eArtifactDir(
|
||||
"failure",
|
||||
configuredDir || path.join(resolveRepoRoot(), ".artifacts", "control-ui-e2e-timeouts", "local"),
|
||||
);
|
||||
const safeMethod = label.replaceAll(/[^a-zA-Z0-9_.-]+/gu, "-");
|
||||
const captureId = `${new Date().toISOString().replaceAll(/[:.]/gu, "-")}-${safeMethod}`;
|
||||
const screenshotName = `${captureId}.png`;
|
||||
const screenshotPath = path.join(artifactDir, screenshotName);
|
||||
const reportPath = path.join(artifactDir, `${captureId}.json`);
|
||||
const captureErrors: string[] = [];
|
||||
let browserState: unknown = null;
|
||||
let summary: unknown = { available: false };
|
||||
try {
|
||||
browserState = await page.evaluate(() => {
|
||||
const { failureSummary, ...state } = await page.evaluate(() => {
|
||||
const copy = (value: unknown): unknown => {
|
||||
try {
|
||||
return structuredClone(value) as unknown;
|
||||
|
|
@ -3603,7 +3644,64 @@ async function captureControlUiE2eFailureDiagnosticsUnsafe(
|
|||
}
|
||||
customElementCounts[name] = (customElementCounts[name] ?? 0) + 1;
|
||||
}
|
||||
const textarea = document.querySelector<HTMLTextAreaElement>(
|
||||
".agent-chat__composer-combobox textarea",
|
||||
);
|
||||
const send = document.querySelector<HTMLButtonElement>(".chat-send-btn--send");
|
||||
const sendLabel = send?.getAttribute("aria-label");
|
||||
// Submit-disabled labels can contain server errors. Only known static UI copy
|
||||
// may reach CI logs; private reports retain the existing detailed state.
|
||||
const safeValue = (value: unknown, allowed: string[]) =>
|
||||
allowed.find((entry) => entry === value) ?? "unknown";
|
||||
return {
|
||||
failureSummary: {
|
||||
gatewayPhase: safeValue(gatewaySnapshot?.phase, [
|
||||
"stopped",
|
||||
"connecting",
|
||||
"connected",
|
||||
"offline",
|
||||
"reconnecting",
|
||||
"starting",
|
||||
"reload-required",
|
||||
]),
|
||||
connected: typeof agentsState?.connected === "boolean" ? agentsState.connected : null,
|
||||
documentReadyState: safeValue(document.readyState, [
|
||||
"loading",
|
||||
"interactive",
|
||||
"complete",
|
||||
]),
|
||||
providerStatuses: [
|
||||
...document.querySelectorAll(".model-providers__head .settings-status"),
|
||||
]
|
||||
.slice(0, 8)
|
||||
.map((badge) => {
|
||||
const text = badge.textContent?.trim();
|
||||
return {
|
||||
status: safeValue(text, ["Ready", "Signed in", "Configured", "Failed"]),
|
||||
length: text?.length ?? 0,
|
||||
};
|
||||
}),
|
||||
composer: textarea
|
||||
? {
|
||||
draftLength: textarea.value.length,
|
||||
nonempty: textarea.value.length > 0,
|
||||
disabled: textarea.disabled,
|
||||
send: send
|
||||
? {
|
||||
label: safeValue(sendLabel, [
|
||||
"Send message",
|
||||
"Write a message to send.",
|
||||
"Sending message...",
|
||||
"Loading chat",
|
||||
]),
|
||||
labelLength: sendLabel?.length ?? 0,
|
||||
disabled: send.disabled,
|
||||
busy: send.getAttribute("aria-busy") === "true",
|
||||
}
|
||||
: null,
|
||||
}
|
||||
: null,
|
||||
},
|
||||
app: {
|
||||
agentSelection: copy(context?.agentSelection?.state ?? null),
|
||||
gateway: {
|
||||
|
|
@ -3662,9 +3760,27 @@ async function captureControlUiE2eFailureDiagnosticsUnsafe(
|
|||
),
|
||||
};
|
||||
});
|
||||
summary = failureSummary;
|
||||
browserState = state;
|
||||
} catch (evaluateError) {
|
||||
captureErrors.push(`page.evaluate: ${String(evaluateError)}`);
|
||||
}
|
||||
// Normal PR CI may not upload this artifact owner. Emit safe facts before any
|
||||
// capture I/O so a broken screenshot or output directory cannot hide the state.
|
||||
console.error("[control-ui-e2e] failure state", {
|
||||
browser: summary,
|
||||
models: modelResponses ? summarizeRecordedModelResponses(modelResponses) : null,
|
||||
});
|
||||
const configuredDir = process.env.OPENCLAW_UI_E2E_DIAGNOSTIC_DIR?.trim();
|
||||
const artifactDir = createControlUiE2eArtifactDir(
|
||||
"failure",
|
||||
configuredDir || path.join(resolveRepoRoot(), ".artifacts", "control-ui-e2e-timeouts", "local"),
|
||||
);
|
||||
const safeMethod = label.replaceAll(/[^a-zA-Z0-9_.-]+/gu, "-");
|
||||
const captureId = `${new Date().toISOString().replaceAll(/[:.]/gu, "-")}-${safeMethod}`;
|
||||
const screenshotName = `${captureId}.png`;
|
||||
const screenshotPath = path.join(artifactDir, screenshotName);
|
||||
const reportPath = path.join(artifactDir, `${captureId}.json`);
|
||||
let screenshotWritten = false;
|
||||
try {
|
||||
await page.screenshot({ fullPage: true, path: screenshotPath });
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue