mirror of
https://github.com/openclaw/openclaw.git
synced 2026-10-03 01:29:56 +00:00
fix: avoid false Gateway failures during cold starts (#153063)
Avoid false Gateway timeout and unreachable reports during slow cold starts. Doctor, health, and status diagnostics now reuse the existing readiness owner and one monotonic deadline, preserve observed startup progress, and leave a Gateway that is still starting alone. Remove Doctor's duplicate restart polling loop. Provider usage consumes the remaining allowance, reports Timeout before dispatch when exhausted, and cancels active usage work on expiry. Preserve explicit timeouts, target/auth selection, and real process/version/build failures. Health and channel-status JSON can return a documented non-failing starting payload; consumers that need a complete snapshot must distinguish it and rerun after startup. No new configuration, flags, dependencies, or stored-data migration. Validation: 592 cases across 37 files on isolated Linux Testbox; installed-package cold-start/down-Gateway matrix and unmodified 2026.9.5 updater-to-candidate Doctor proof; focused command/transport regressions and changed-file gates. The existing independent review is scoped-clean, and ClawSweeper reports no actionable correctness finding. Thanks to @Suidge for the report and startup measurements. Fixes #152970.
This commit is contained in:
parent
025b994448
commit
1b3f38e7e4
85 changed files with 2508 additions and 710 deletions
|
|
@ -63,7 +63,7 @@ With `--json`, every channel entry includes `label` alongside its accounts, inst
|
|||
values. Omit the option to keep each command's default or broader account scope;
|
||||
do not pass an empty shell variable to request that scope.
|
||||
|
||||
- `channels status`: `--channel <name>`, `--probe`, `--timeout <ms>` (default `10000`), `--json`
|
||||
- `channels status`: `--channel <name>`, `--probe`, `--timeout <ms>` (default `60000`), `--json`
|
||||
- `channels capabilities`: `--channel <name>`, `--agent <id>`, `--account <id>` (requires `--channel`), `--target <dest>` (requires `--channel`), `--timeout <ms>` (default `10000`, capped at `30000`), `--json`
|
||||
- `channels resolve <entries...>`: `--channel <name>`, `--account <id>`, `--agent <id>`, `--kind <auto|user|group|channel>` (default `auto`), `--json`
|
||||
- `channels logs`: `--channel <name|all>` (default `all`), `--lines <n>` (default `200`), `--json`
|
||||
|
|
@ -80,6 +80,8 @@ state plus probe results such as `works`, `probe failed`, `audit ok`, or `audit
|
|||
If the gateway is unreachable, `channels status` falls back to config-only summaries
|
||||
instead of live probe output.
|
||||
|
||||
Before probing channels, the command waits for local Gateway startup using the shared readiness budget and reports its observed phase. Startup still in progress at the deadline is a non-failing result, not an unreachable Gateway. In that case `--json` returns `{ "status": "starting", "startupPhase": "…" }`; rerun after startup to collect channel results.
|
||||
|
||||
`channels status` does not support `--deep`; use `openclaw channels status --probe` for channel checks. The separate top-level `openclaw status --deep` command provides a broader status probe.
|
||||
|
||||
## Inbound dead letters
|
||||
|
|
|
|||
|
|
@ -76,6 +76,10 @@ follows the current account's instance (`openclaw@<user>.service`) while
|
|||
preserving the shared template. Run Doctor as that account after the system
|
||||
service owner stops its instance.
|
||||
|
||||
Doctor waits for a starting local Gateway using the shared 60-second readiness budget, both on its initial check and after an approved restart. It reports the observed startup phase while waiting. A Gateway that still reports startup at the deadline produces a non-failing “still starting” result; Doctor leaves it running and does not offer another restart. Connection failure without startup evidence remains a diagnostic failure. This also applies when an installed updater invokes the candidate Doctor.
|
||||
|
||||
Plugin initialization and database startup checks can make a cold start take longer than ten seconds on a loaded or older host. Let the existing Gateway finish starting before requesting a separate restart. Remote Gateway diagnostics keep using the configured remote target.
|
||||
|
||||
## Remote Gateway recovery
|
||||
|
||||
With `gateway.mode: "remote"`, a failed Gateway health check does not trigger
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ Fetch a health snapshot from the running Gateway over WebSocket RPC (no direct c
|
|||
| Flag | Default | Description |
|
||||
| ---------------- | ------- | --------------------------------------------------------------------------------- |
|
||||
| `--json` | `false` | Print machine-readable JSON instead of text. |
|
||||
| `--timeout <ms>` | `10000` | Connection timeout in milliseconds. |
|
||||
| `--timeout <ms>` | `60000` | Connection timeout in milliseconds. |
|
||||
| `--verbose` | `false` | Forces a live probe and expands output across all configured accounts and agents. |
|
||||
| `--debug` | `false` | Alias for `--verbose`. |
|
||||
|
||||
|
|
@ -30,12 +30,14 @@ openclaw health --debug
|
|||
|
||||
## Behavior
|
||||
|
||||
- Local Gateway startup uses the shared 60-second readiness budget and reports its observed phase. An explicit `--timeout` overrides that budget. Startup still in progress at the deadline returns a non-failing result; JSON reports `{ "status": "starting", "startupPhase": "…" }` instead of a health snapshot.
|
||||
|
||||
- Without `--verbose`, the Gateway can return a cached snapshot and refresh it in the background for the next caller. A cached snapshot is fresh for up to 60 seconds and unchanged from live channel runtime state.
|
||||
- `--verbose` forces a live probe of each channel account. It also prints Gateway connection details. It expands human-readable output across all configured accounts and agents, instead of just the default agent.
|
||||
- An unconfigured or disabled preferred account does not hide probe results from other active accounts. Ordinary output still follows the default agent's account bindings. `--verbose` includes all accounts.
|
||||
- When the displayed account is disabled, health text shows `disabled` and `status --deep` marks it `OFF`, even if the account remains configured.
|
||||
- Unhealthy channel lines include the recorded startup error when available, so a stopped channel reports its failure cause alongside its state.
|
||||
- `--json` always returns the full snapshot: channels, per-account probes, plugin load state, context-engine quarantine state, model-pricing cache state, event-loop health, delivery-queue warnings, and per-agent session stores.
|
||||
- Once ready, `--json` returns the full snapshot: channels, per-account probes, plugin load state, context-engine quarantine state, model-pricing cache state, event-loop health, delivery-queue warnings, and per-agent session stores.
|
||||
- Session ages in text and JSON use the Gateway's clock.
|
||||
- Heartbeat intervals in text show the resolved cadence without rounding away milliseconds. Week units are retained for long intervals.
|
||||
- Top-level `ok: true` means the health RPC succeeded and the Gateway produced a snapshot. Queue warnings do not change it to `false`.
|
||||
|
|
|
|||
|
|
@ -31,9 +31,11 @@ openclaw status --usage --agent work
|
|||
| `--usage` | Prints normalized provider usage windows as `X% left`. |
|
||||
| `--agent <id>` | Selects the agent auth/profile scope for `--usage`. Required when an explicit multi-agent fleet has no default. |
|
||||
| `--json` | Machine-readable output. |
|
||||
| `--timeout <ms>` | Probe timeout in milliseconds (default: `10000`). |
|
||||
| `--timeout <ms>` | Probe timeout in milliseconds (default: `60000`). |
|
||||
| `--verbose` / `--debug` | Also print the raw Gateway target resolution before the report. |
|
||||
|
||||
Status starts one monotonic probe deadline when the command begins. Local readiness, Gateway status, provider usage, and deep health consume the remaining allowance. Remote targets and explicit Gateway URLs skip local readiness but keep the same deadline. Local probes report the observed startup phase while waiting. If the Gateway is still starting when the budget expires, status reports “still starting” instead of unreachable and skips deep channel probes. JSON keeps the status report and adds `gateway.readiness: "still-starting"` and `gateway.startupPhase`. An explicit `--timeout` limits that shared allowance.
|
||||
|
||||
Channels without a probe, such as WhatsApp, report lifecycle health instead.
|
||||
In the Health table, `healthy` is `OK`; degraded lifecycle states and failed
|
||||
probes remain `WARN`. A lifecycle `OK` does not mean a live probe ran.
|
||||
|
|
@ -157,6 +159,11 @@ Use `openclaw skills check --agent <id>` to inspect the missing requirements.
|
|||
|
||||
- `--usage` prints normalized provider usage windows as `X% left`.
|
||||
It also adds usage snapshots to `--all`; `--agent` keeps the same usage-only scope.
|
||||
Usage probes receive the remaining shared probe budget, capped by `--timeout` when set;
|
||||
providers that exceed that bound report `Timeout` in the usage output.
|
||||
An exhausted budget reports `Timeout` without starting provider auth or usage
|
||||
requests. Expiry cancels active usage requests and prevents late auth results
|
||||
from starting another request; completed provider snapshots remain available.
|
||||
- In an explicit multi-agent setup, `--usage` reads the auth profiles owned by
|
||||
`agents.defaults.systemAgent.agentId` by default. Pass `--agent <id>` to
|
||||
inspect another agent; without either owner, OpenClaw does not guess one
|
||||
|
|
|
|||
|
|
@ -131,10 +131,20 @@ additive fields on the existing contract; they add no hook or user setting.
|
|||
},
|
||||
fetchUsageSnapshot: async (ctx) => {
|
||||
// fetchAcmeUsage is your plugin's own vendor API call, not an SDK export.
|
||||
return await fetchAcmeUsage(ctx.token, ctx.timeoutMs);
|
||||
return await fetchAcmeUsage(ctx.token, ctx.timeoutMs, {
|
||||
fetch: ctx.fetchFn,
|
||||
signal: ctx.signal,
|
||||
});
|
||||
},
|
||||
```
|
||||
|
||||
Both usage hooks receive an optional `ctx.signal` for collection cancellation.
|
||||
`ctx.fetchFn` already combines it with request cancellation; custom transports
|
||||
must forward `ctx.signal` to their I/O. Check cancellation before starting
|
||||
additional auth work after an await. An exhausted budget invokes neither hook
|
||||
and produces a visible `Timeout` snapshot. Core retains completed siblings and
|
||||
tracks unfinished work through cleanup, including auth-owned credential refresh.
|
||||
|
||||
`resolveUsageAuth` has three outcomes. Return
|
||||
`{ token, accountId?, subscriptionType?, rateLimitTier? }` when the
|
||||
provider has a usage/billing credential (the optional fields carry
|
||||
|
|
|
|||
|
|
@ -254,6 +254,7 @@ export default defineSingleProviderPluginEntry({
|
|||
token: ctx.token,
|
||||
baseUrl: configuredBaseUrl(ctx.config),
|
||||
timeoutMs: ctx.timeoutMs,
|
||||
signal: ctx.signal,
|
||||
}),
|
||||
};
|
||||
},
|
||||
|
|
|
|||
|
|
@ -121,6 +121,7 @@ function mockFetchGuard(response: Response): MockedFunction<ClawRouterUsageFetch
|
|||
|
||||
describe("ClawRouter usage", () => {
|
||||
it("maps the managed monthly budget and usage totals", async () => {
|
||||
const signal = new AbortController().signal;
|
||||
const fetchGuard = mockFetchGuard(
|
||||
Response.json({
|
||||
budget: {
|
||||
|
|
@ -145,6 +146,7 @@ describe("ClawRouter usage", () => {
|
|||
token: "proxy-key",
|
||||
baseUrl: "https://clawrouter.example/v1",
|
||||
timeoutMs: 5000,
|
||||
signal,
|
||||
fetchGuard,
|
||||
});
|
||||
|
||||
|
|
@ -181,6 +183,7 @@ describe("ClawRouter usage", () => {
|
|||
},
|
||||
}),
|
||||
auditContext: "clawrouter.usage",
|
||||
signal,
|
||||
mode: "trusted_env_proxy",
|
||||
}),
|
||||
);
|
||||
|
|
|
|||
|
|
@ -91,6 +91,7 @@ export async function fetchClawRouterUsage(params: {
|
|||
token: string;
|
||||
baseUrl?: string;
|
||||
timeoutMs: number;
|
||||
signal?: AbortSignal;
|
||||
/** Test-only seam; production keeps the shared SSRF guard owning transport. */
|
||||
fetchGuard?: ClawRouterUsageFetchGuard;
|
||||
}): Promise<ProviderUsageSnapshot> {
|
||||
|
|
@ -109,6 +110,7 @@ export async function fetchClawRouterUsage(params: {
|
|||
},
|
||||
},
|
||||
timeoutMs: params.timeoutMs,
|
||||
signal: params.signal,
|
||||
policy: ssrfPolicyFromHttpBaseUrlAllowedHostname(rootUrl),
|
||||
auditContext: "clawrouter.usage",
|
||||
}),
|
||||
|
|
|
|||
|
|
@ -131,6 +131,7 @@ type CodexAppServerJsonClientOptions = Pick<
|
|||
sessionKey?: string;
|
||||
sessionId?: string;
|
||||
isolated?: boolean;
|
||||
signal?: AbortSignal;
|
||||
assertCurrent?: () => void;
|
||||
catalogPreview?: true;
|
||||
catalogPreviewCache?: CodexCatalogPreviewCache;
|
||||
|
|
@ -221,6 +222,7 @@ const CODEX_USAGE_DEADLINE_RESERVE_MS =
|
|||
/** Reads rate limits and best-effort account identity from one isolated app-server session. */
|
||||
export async function readCodexAppServerUsage(options: {
|
||||
timeoutMs: number;
|
||||
signal?: AbortSignal;
|
||||
agentDir?: string;
|
||||
authProfileId?: string;
|
||||
config?: Parameters<typeof resolveCodexAppServerAuthProfileIdForAgent>[0]["config"];
|
||||
|
|
@ -233,6 +235,7 @@ export async function readCodexAppServerUsage(options: {
|
|||
return await withCodexAppServerJsonClient(
|
||||
{
|
||||
timeoutMs: options.timeoutMs,
|
||||
signal: options.signal,
|
||||
timeoutMessage: "codex app-server usage read timed out",
|
||||
agentDir: options.agentDir,
|
||||
...(options.authProfileId ? { authProfileId: options.authProfileId } : {}),
|
||||
|
|
@ -307,6 +310,11 @@ export async function withCodexAppServerJsonClient<T>(
|
|||
let errorPhase: CodexControlRequestPhase | undefined;
|
||||
observeControlPhase(params.controlObservation, activePhase);
|
||||
const timeoutController = new AbortController();
|
||||
const abort = () => timeoutController.abort(params.signal?.reason);
|
||||
params.signal?.addEventListener("abort", abort, { once: true });
|
||||
if (params.signal?.aborted) {
|
||||
abort();
|
||||
}
|
||||
const deadline =
|
||||
Number.isFinite(timeoutMs) && timeoutMs > 0 ? performance.now() + timeoutMs : undefined;
|
||||
const isPastDeadline = () => deadline !== undefined && performance.now() >= deadline;
|
||||
|
|
@ -324,6 +332,7 @@ export async function withCodexAppServerJsonClient<T>(
|
|||
};
|
||||
|
||||
try {
|
||||
throwIfAbandoned();
|
||||
return await withAbortableTimeout({
|
||||
signal: timeoutController.signal,
|
||||
timeoutMs,
|
||||
|
|
@ -489,6 +498,7 @@ export async function withCodexAppServerJsonClient<T>(
|
|||
}
|
||||
throw error;
|
||||
} finally {
|
||||
params.signal?.removeEventListener("abort", abort);
|
||||
// `withTimeout` only stops awaiting. Abort the shared operation before its
|
||||
// timeout becomes observable so no delayed acquire can issue a request or retry.
|
||||
timeoutController.abort(createScopeCleanupError(timeoutMessage));
|
||||
|
|
|
|||
|
|
@ -0,0 +1,98 @@
|
|||
import { createDeferred } from "openclaw/plugin-sdk/extension-shared";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const { createIsolatedCodexAppServerClient } = vi.hoisted(() => ({
|
||||
createIsolatedCodexAppServerClient: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock("./shared-client.js", () => ({
|
||||
createIsolatedCodexAppServerClient,
|
||||
getLeasedSharedCodexAppServerClient: vi.fn(),
|
||||
isCodexAppServerStartSelectionChangedError: () => false,
|
||||
releaseLeasedSharedCodexAppServerClient: vi.fn(),
|
||||
retireSharedCodexAppServerClientIfCurrent: vi.fn(),
|
||||
}));
|
||||
|
||||
// Prepare the deferred runtime dependency before testing controlled cancellation.
|
||||
await import("./sandbox-guard.js");
|
||||
const { CodexAppServerScopedRequestRejectedError, readCodexAppServerUsage } =
|
||||
await import("./request.js");
|
||||
|
||||
describe("Codex usage cancellation", () => {
|
||||
beforeEach(() => {
|
||||
createIsolatedCodexAppServerClient.mockReset();
|
||||
vi.useFakeTimers();
|
||||
});
|
||||
afterEach(() => vi.useRealTimers());
|
||||
|
||||
it("does not acquire a usage client after its owner has aborted", async () => {
|
||||
const reason = new Error("Usage deadline exhausted");
|
||||
createIsolatedCodexAppServerClient.mockResolvedValue({
|
||||
request: vi.fn(async () => ({})),
|
||||
closeAndWait: vi.fn(async () => undefined),
|
||||
});
|
||||
|
||||
await expect(
|
||||
readCodexAppServerUsage({ timeoutMs: 3_500, signal: AbortSignal.abort(reason) }),
|
||||
).rejects.toBe(reason);
|
||||
expect(createIsolatedCodexAppServerClient).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each(["acquisition", "request"] as const)(
|
||||
"aborts usage %s when the outer deadline expires",
|
||||
async (phase) => {
|
||||
const controller = new AbortController();
|
||||
const reason = new Error("Usage deadline exhausted");
|
||||
const started = createDeferred<AbortSignal>();
|
||||
const closed = createDeferred<void>();
|
||||
const operation = createDeferred<never>();
|
||||
const untilAbort = (signal: AbortSignal) => {
|
||||
started.resolve(signal);
|
||||
signal.addEventListener("abort", () => operation.reject(signal.reason), { once: true });
|
||||
return operation.promise;
|
||||
};
|
||||
const request = vi.fn((_method: string, _params: unknown, options: { signal: AbortSignal }) =>
|
||||
untilAbort(options.signal),
|
||||
);
|
||||
const closeAndWait = vi.fn(async () => closed.resolve());
|
||||
createIsolatedCodexAppServerClient.mockImplementation(
|
||||
async ({ abandonSignal }: { abandonSignal: AbortSignal }) =>
|
||||
phase === "acquisition" ? await untilAbort(abandonSignal) : { request, closeAndWait },
|
||||
);
|
||||
|
||||
const result = readCodexAppServerUsage({ timeoutMs: 3_500, signal: controller.signal });
|
||||
const settledError = result.then(
|
||||
() => undefined,
|
||||
(error: unknown) => error,
|
||||
);
|
||||
try {
|
||||
const producerSignal = await Promise.race([
|
||||
started.promise,
|
||||
settledError.then((error) => {
|
||||
throw error instanceof Error
|
||||
? error
|
||||
: new CodexAppServerScopedRequestRejectedError(
|
||||
"Usage completed without starting its producer",
|
||||
{ cause: error },
|
||||
);
|
||||
}),
|
||||
]);
|
||||
controller.abort(reason);
|
||||
expect(producerSignal.aborted).toBe(true);
|
||||
expect(producerSignal.reason).toBe(reason);
|
||||
expect(await settledError).toBe(reason);
|
||||
expect(request).toHaveBeenCalledTimes(phase === "request" ? 1 : 0);
|
||||
if (phase === "request") {
|
||||
await closed.promise;
|
||||
expect(closeAndWait).toHaveBeenCalledExactlyOnceWith({
|
||||
exitTimeoutMs: 300,
|
||||
forceKillDelayMs: 200,
|
||||
});
|
||||
}
|
||||
} finally {
|
||||
operation.reject(new Error("Test fixture closed"));
|
||||
await Promise.allSettled([operation.promise, settledError]);
|
||||
}
|
||||
},
|
||||
);
|
||||
});
|
||||
|
|
@ -20,6 +20,7 @@ function usageContext(
|
|||
|
||||
describe("Codex app-server provider usage", () => {
|
||||
it("contributes OpenAI usage windows for the synthetic app-server credential", async () => {
|
||||
const signal = new AbortController().signal;
|
||||
const readUsage = vi.fn(async () => ({
|
||||
rateLimits: {
|
||||
rateLimitsByLimitId: {
|
||||
|
|
@ -36,7 +37,9 @@ describe("Codex app-server provider usage", () => {
|
|||
accountEmail: "codex-account@example.com",
|
||||
}));
|
||||
|
||||
await expect(fetchCodexAppServerUsageSnapshot(usageContext(), { readUsage })).resolves.toEqual({
|
||||
await expect(
|
||||
fetchCodexAppServerUsageSnapshot(usageContext({ signal }), { readUsage }),
|
||||
).resolves.toEqual({
|
||||
provider: "openai",
|
||||
displayName: "OpenAI",
|
||||
windows: [{ label: "5h", usedPercent: 9, resetAt: 1_700_003_600_000 }],
|
||||
|
|
@ -45,6 +48,7 @@ describe("Codex app-server provider usage", () => {
|
|||
});
|
||||
expect(readUsage).toHaveBeenCalledWith({
|
||||
timeoutMs: 3_500,
|
||||
signal,
|
||||
agentDir: undefined,
|
||||
config: {},
|
||||
startOptions: expect.objectContaining({
|
||||
|
|
|
|||
|
|
@ -29,6 +29,7 @@ export async function fetchCodexAppServerUsageSnapshot(
|
|||
const appServer = resolveCodexAppServerRuntimeOptions({ pluginConfig: options.pluginConfig });
|
||||
const usage = await (options.readUsage ?? readCodexAppServerUsage)({
|
||||
timeoutMs: ctx.timeoutMs,
|
||||
signal: ctx.signal,
|
||||
agentDir: ctx.agentDir,
|
||||
...(ctx.authProfileId ? { authProfileId: ctx.authProfileId } : {}),
|
||||
config: ctx.config,
|
||||
|
|
|
|||
|
|
@ -371,6 +371,7 @@ export default defineSingleProviderPluginEntry({
|
|||
baseUrl: ctx.config.models?.providers?.openrouter?.baseUrl,
|
||||
request: ctx.config.models?.providers?.openrouter?.request,
|
||||
timeoutMs: ctx.timeoutMs,
|
||||
signal: ctx.signal,
|
||||
fetchFn: ctx.fetchFn,
|
||||
}),
|
||||
};
|
||||
|
|
|
|||
|
|
@ -193,6 +193,7 @@ describe("OpenRouter usage", () => {
|
|||
])(
|
||||
"routes $name through canonical guarded transport instead of the ambient proxy wrapper",
|
||||
async ({ request, dispatcherPolicy }) => {
|
||||
const signal = new AbortController().signal;
|
||||
const ambientProxyFetch = vi.fn(async () => Response.json({ data: { usage: 1 } }));
|
||||
const canonicalRuntimeFetch = vi.fn(async () => Response.json({ data: { usage: 1 } }));
|
||||
const release = vi.fn(async () => undefined);
|
||||
|
|
@ -213,6 +214,7 @@ describe("OpenRouter usage", () => {
|
|||
baseUrl: "https://private.example.invalid/router/v1",
|
||||
request,
|
||||
timeoutMs: 1000,
|
||||
signal,
|
||||
fetchFn: ambientProxyFetch as unknown as typeof fetch,
|
||||
});
|
||||
|
||||
|
|
@ -225,6 +227,7 @@ describe("OpenRouter usage", () => {
|
|||
expect(params.fetchImpl).toBeUndefined();
|
||||
expect(params.maxRedirects).toBe(0);
|
||||
expect(params.timeoutMs).toBe(1000);
|
||||
expect(params.signal).toBe(signal);
|
||||
expect(params.policy).toEqual({ allowedOrigins: ["https://private.example.invalid"] });
|
||||
}
|
||||
} finally {
|
||||
|
|
|
|||
|
|
@ -103,6 +103,7 @@ async function fetchEndpoint(params: {
|
|||
ssrfPolicy: ReturnType<typeof resolveOpenRouterSsrfPolicy>;
|
||||
dispatcherPolicy: ReturnType<typeof resolveProviderHttpRequestConfig>["dispatcherPolicy"];
|
||||
timeoutMs: number;
|
||||
signal?: AbortSignal;
|
||||
fetchFn: typeof fetch;
|
||||
}): Promise<EndpointResult> {
|
||||
let guardedResponse: Awaited<ReturnType<typeof fetchWithSsrFGuard>>;
|
||||
|
|
@ -118,6 +119,7 @@ async function fetchEndpoint(params: {
|
|||
redirect: "error",
|
||||
},
|
||||
timeoutMs: params.timeoutMs,
|
||||
signal: params.signal,
|
||||
// The shared guard controls redirects manually; zero hops preserves fail-closed usage auth.
|
||||
maxRedirects: 0,
|
||||
policy: params.ssrfPolicy,
|
||||
|
|
@ -150,6 +152,7 @@ export async function fetchOpenRouterUsage(params: {
|
|||
baseUrl?: string;
|
||||
request?: ModelProviderConfig["request"];
|
||||
timeoutMs: number;
|
||||
signal?: AbortSignal;
|
||||
fetchFn: typeof fetch;
|
||||
}): Promise<ProviderUsageSnapshot> {
|
||||
const requestConfig = resolveProviderHttpRequestConfig({
|
||||
|
|
@ -169,6 +172,7 @@ export async function fetchOpenRouterUsage(params: {
|
|||
ssrfPolicy: resolveOpenRouterSsrfPolicy(requestConfig, params.request),
|
||||
dispatcherPolicy: requestConfig.dispatcherPolicy,
|
||||
timeoutMs: params.timeoutMs,
|
||||
signal: params.signal,
|
||||
fetchFn: params.fetchFn,
|
||||
};
|
||||
const [creditsResult, keyResult] = await Promise.all([
|
||||
|
|
|
|||
|
|
@ -207,7 +207,7 @@ export async function registerChannelsCli(
|
|||
.description("Show channel status (use openclaw status --deep for a full connection check)")
|
||||
.option("--channel <name>", `Only show one channel (${formatCliChannelOptions(["all"])})`)
|
||||
.option("--probe", "Probe channel credentials", false)
|
||||
.option("--timeout <ms>", "Timeout in ms", "10000")
|
||||
.option("--timeout <ms>", "Timeout in ms")
|
||||
.option("--json", "Output JSON", false)
|
||||
.action(async (opts) => {
|
||||
await runChannelsCommand(async () => {
|
||||
|
|
|
|||
|
|
@ -1,5 +1,7 @@
|
|||
// The real route-first channel status path must honor the shared catalog guard policy.
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import * as gatewayCall from "../gateway/call.js";
|
||||
import { DEFAULT_RESTART_HEALTH_TIMEOUT_MS } from "./daemon-cli/restart-health.constants.js";
|
||||
|
||||
const loaded = vi.hoisted(() => {
|
||||
const modules = new Set<string>();
|
||||
|
|
@ -10,7 +12,6 @@ const loaded = vi.hoisted(() => {
|
|||
},
|
||||
};
|
||||
});
|
||||
const callGatewayMock = vi.hoisted(() => vi.fn(async () => ({ channelAccounts: {} })));
|
||||
const runtime = vi.hoisted(() => ({
|
||||
error: vi.fn(),
|
||||
exit: vi.fn(),
|
||||
|
|
@ -32,10 +33,6 @@ vi.mock("../commands/channels/status.runtime.js", () => {
|
|||
};
|
||||
});
|
||||
|
||||
vi.mock("../gateway/call.js", () => ({
|
||||
callGateway: callGatewayMock,
|
||||
}));
|
||||
|
||||
vi.mock("./progress.js", () => ({
|
||||
withProgress: vi.fn(async (_opts, run: () => Promise<unknown>) => await run()),
|
||||
}));
|
||||
|
|
@ -53,12 +50,16 @@ describe("routed channels status cold imports", () => {
|
|||
// eager import during route.js's own static import graph is exactly the
|
||||
// regression evidence this test exists to keep.
|
||||
vi.clearAllMocks();
|
||||
// Keep readiness target/auth selection real; replace only the wire request.
|
||||
vi.spyOn(gatewayCall, "callGateway").mockResolvedValue({ channelAccounts: {} });
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
});
|
||||
afterEach(() => vi.restoreAllMocks());
|
||||
|
||||
it("keeps successful all-channel JSON routes cold with and without probing", async () => {
|
||||
for (const probe of [false, true]) {
|
||||
vi.clearAllMocks();
|
||||
const timeoutMs = probe ? 30000 : 10000;
|
||||
const timeoutMs = DEFAULT_RESTART_HEALTH_TIMEOUT_MS;
|
||||
const argv = [
|
||||
"node",
|
||||
"openclaw",
|
||||
|
|
@ -74,8 +75,8 @@ describe("routed channels status cold imports", () => {
|
|||
|
||||
await expect(tryRouteCli(argv)).resolves.toBe(true);
|
||||
|
||||
expect(callGatewayMock).toHaveBeenCalledOnce();
|
||||
expect(callGatewayMock).toHaveBeenCalledWith({
|
||||
expect(gatewayCall.callGateway).toHaveBeenCalledOnce();
|
||||
expect(gatewayCall.callGateway).toHaveBeenCalledWith({
|
||||
method: "channels.status",
|
||||
params: { probe, timeoutMs },
|
||||
timeoutMs,
|
||||
|
|
|
|||
197
src/cli/daemon-cli/diagnostic-readiness.test.ts
Normal file
197
src/cli/daemon-cli/diagnostic-readiness.test.ts
Normal file
|
|
@ -0,0 +1,197 @@
|
|||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import type { OpenClawConfig } from "../../config/types.openclaw.js";
|
||||
import { gatewayHealthResponse } from "../../gateway/health-response.test-support.js";
|
||||
import {
|
||||
callGateway,
|
||||
inspectPortUsage,
|
||||
monotonicClock,
|
||||
readGatewayOwnerLease,
|
||||
requestStartupProbe,
|
||||
resetRestartHealthMocks,
|
||||
resolveGatewayProbeAuthSafeWithSecretInputs,
|
||||
restoreRestartHealthMocks,
|
||||
} from "./restart-health.test-helpers.js";
|
||||
|
||||
const { readRuntime, readCommand } = vi.hoisted(() => ({
|
||||
readCommand: vi.fn(async () => ({ programArguments: ["gateway", "--port", "18789"] })),
|
||||
readRuntime: vi.fn(async () => ({ status: "stopped" })),
|
||||
}));
|
||||
vi.mock("../../daemon/service.js", () => ({
|
||||
resolveGatewayService: () => ({ readRuntime, readCommand }),
|
||||
}));
|
||||
const { waitForGatewayDiagnosticReadiness } = await import("./diagnostic-readiness.js");
|
||||
|
||||
describe("diagnostic Gateway readiness", () => {
|
||||
beforeEach(() => {
|
||||
resetRestartHealthMocks();
|
||||
inspectPortUsage.mockImplementation(async (port) => ({
|
||||
port,
|
||||
status: "free",
|
||||
listeners: [],
|
||||
hints: [],
|
||||
}));
|
||||
readRuntime.mockReset();
|
||||
readRuntime.mockResolvedValue({ status: "stopped" });
|
||||
readCommand.mockReset();
|
||||
readCommand.mockResolvedValue({ programArguments: ["gateway", "--port", "18789"] });
|
||||
vi.stubEnv("OPENCLAW_GATEWAY_URL", undefined);
|
||||
vi.stubEnv("OPENCLAW_GATEWAY_PORT", undefined);
|
||||
});
|
||||
afterEach(() => {
|
||||
restoreRestartHealthMocks();
|
||||
vi.unstubAllEnvs();
|
||||
});
|
||||
|
||||
it.each<{ envUrl?: string; url?: string; config?: OpenClawConfig }>([
|
||||
{ url: "ws://127.0.0.1:18789" },
|
||||
{ config: { gateway: { mode: "remote", remote: { url: "wss://peer.example" } } } },
|
||||
{ envUrl: "wss://peer.example" },
|
||||
])("preserves an explicit or remote target: %j", async ({ envUrl, ...options }) => {
|
||||
if (envUrl) {
|
||||
vi.stubEnv("OPENCLAW_GATEWAY_URL", envUrl);
|
||||
}
|
||||
await expect(
|
||||
waitForGatewayDiagnosticReadiness({ config: {}, ...options }),
|
||||
).resolves.toBeUndefined();
|
||||
expect(inspectPortUsage).not.toHaveBeenCalled();
|
||||
expect(callGateway).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("defers to original diagnostic authentication when no shared credential is available", async () => {
|
||||
const result = await waitForGatewayDiagnosticReadiness({
|
||||
config: { gateway: { auth: { mode: "token" } } },
|
||||
});
|
||||
expect(result).toBeUndefined();
|
||||
expect(monotonicClock.nowMs).toBe(0);
|
||||
expect(readRuntime).not.toHaveBeenCalled();
|
||||
expect(inspectPortUsage).not.toHaveBeenCalled();
|
||||
expect(callGateway).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([22_000, 61_000])(
|
||||
"charges %d ms of authentication preparation to the caller's absolute deadline",
|
||||
async (authElapsedMs) => {
|
||||
resolveGatewayProbeAuthSafeWithSecretInputs.mockImplementation(async () => {
|
||||
monotonicClock.nowMs += authElapsedMs;
|
||||
return { auth: { token: "fixture-token" } };
|
||||
});
|
||||
readGatewayOwnerLease.mockReturnValue({
|
||||
owner: "fixture-owner",
|
||||
pid: 8000,
|
||||
host: "fixture-host",
|
||||
startedAt: 1,
|
||||
port: 18789,
|
||||
mode: "foreground",
|
||||
supervisor: null,
|
||||
state: "live",
|
||||
expired: false,
|
||||
});
|
||||
|
||||
const result = await waitForGatewayDiagnosticReadiness({
|
||||
config: { gateway: { auth: { mode: "token" } } },
|
||||
timeoutMs: 60_000,
|
||||
deadlineMs: 60_000,
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({
|
||||
healthy: false,
|
||||
waitOutcome: "timeout",
|
||||
elapsedMs: Math.max(0, 60_000 - authElapsedMs),
|
||||
});
|
||||
expect(monotonicClock.nowMs).toBe(Math.max(60_000, authElapsedMs));
|
||||
expect(callGateway).not.toHaveBeenCalled();
|
||||
if (authElapsedMs >= 60_000) {
|
||||
expect(inspectPortUsage).not.toHaveBeenCalled();
|
||||
expect(result?.probeError).toBe("Gateway readiness budget exhausted.");
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it.each([20_000, 7_500])(
|
||||
"observes a foreground startup using the selected config, auth, port and %d ms budget",
|
||||
async (timeoutMs) => {
|
||||
const config: OpenClawConfig = { gateway: { port: 19091, auth: { mode: "token" } } };
|
||||
resolveGatewayProbeAuthSafeWithSecretInputs.mockResolvedValue({
|
||||
auth: { token: "fixture-token" },
|
||||
});
|
||||
readGatewayOwnerLease.mockReturnValue({
|
||||
owner: "fixture-owner",
|
||||
pid: 8000,
|
||||
host: "fixture-host",
|
||||
startedAt: 1,
|
||||
port: 19091,
|
||||
mode: "foreground",
|
||||
supervisor: null,
|
||||
state: "live",
|
||||
expired: false,
|
||||
});
|
||||
inspectPortUsage.mockImplementation(async (port) => ({
|
||||
port,
|
||||
status: "busy",
|
||||
listeners: [{ pid: 8000 }],
|
||||
hints: [],
|
||||
}));
|
||||
requestStartupProbe.mockImplementation(async () => ({
|
||||
statusCode: monotonicClock.nowMs < 20_000 ? 503 : 200,
|
||||
body: JSON.stringify(
|
||||
monotonicClock.nowMs < 20_000
|
||||
? { status: "starting", pendingReason: "plugin-convergence" }
|
||||
: { status: "started" },
|
||||
),
|
||||
}));
|
||||
callGateway.mockImplementation(gatewayHealthResponse());
|
||||
const result = await waitForGatewayDiagnosticReadiness({
|
||||
config,
|
||||
token: "fixture-token",
|
||||
timeoutMs,
|
||||
});
|
||||
expect(result).toMatchObject({
|
||||
healthy: timeoutMs === 20_000,
|
||||
waitOutcome: timeoutMs === 20_000 ? "healthy" : "still-starting",
|
||||
elapsedMs: timeoutMs,
|
||||
runtime: { status: "running", pid: 8000 },
|
||||
portUsage: { port: 19091 },
|
||||
});
|
||||
expect(readRuntime).not.toHaveBeenCalled();
|
||||
if (timeoutMs === 20_000) {
|
||||
expect(callGateway).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ config, token: "fixture-token", localPortOverride: 19091 }),
|
||||
);
|
||||
} else {
|
||||
expect(result?.startupPhase).toBe("plugin-convergence");
|
||||
expect(callGateway).not.toHaveBeenCalled();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("does not use a different installed service as the selected Gateway's process identity", async () => {
|
||||
readRuntime.mockResolvedValueOnce({ status: "running" });
|
||||
const result = await waitForGatewayDiagnosticReadiness({
|
||||
config: { gateway: { auth: { mode: "none" } } },
|
||||
localPortOverride: 19092,
|
||||
timeoutMs: 1_000,
|
||||
});
|
||||
expect(result).toMatchObject({
|
||||
healthy: false,
|
||||
waitOutcome: "timeout",
|
||||
elapsedMs: 1_000,
|
||||
runtime: { status: "unknown" },
|
||||
portUsage: { port: 19092 },
|
||||
});
|
||||
expect(readRuntime).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("bounds an actually-down Gateway with the caller's shorter deadline", async () => {
|
||||
const result = await waitForGatewayDiagnosticReadiness({
|
||||
config: { gateway: { auth: { mode: "none" } } },
|
||||
timeoutMs: 1_250,
|
||||
});
|
||||
expect(result).toMatchObject({
|
||||
healthy: false,
|
||||
waitOutcome: "timeout",
|
||||
elapsedMs: 1_250,
|
||||
portUsage: { status: "free" },
|
||||
});
|
||||
expect(callGateway).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
98
src/cli/daemon-cli/diagnostic-readiness.ts
Normal file
98
src/cli/daemon-cli/diagnostic-readiness.ts
Normal file
|
|
@ -0,0 +1,98 @@
|
|||
import { resolveConfigPath, resolveGatewayPort, resolveStateDir } from "../../config/paths.js";
|
||||
import type { OpenClawConfig } from "../../config/types.openclaw.js";
|
||||
import { mergeGatewayServiceEnv } from "../../daemon/service-env-merge.js";
|
||||
import type { GatewayServiceCommandConfig } from "../../daemon/service-types.js";
|
||||
import { resolveGatewayService } from "../../daemon/service.js";
|
||||
import { isImplicitLocalGatewayTarget } from "../../gateway/call.js";
|
||||
import { resolveGatewayProbeAuthSafeWithSecretInputs } from "../../gateway/probe-auth.js";
|
||||
import { readGatewayOwnerLease } from "../../infra/gateway-owner-lease.js";
|
||||
import { LOOPBACK_PORT_PROBE_HOSTS } from "../../infra/ports-probe.js";
|
||||
import { parseTcpPortFromArgs } from "../../infra/tcp-port.js";
|
||||
import { resolveGatewayRestartProbeContext } from "./restart-health-probe.js";
|
||||
import { DEFAULT_RESTART_HEALTH_TIMEOUT_MS } from "./restart-health.constants.js";
|
||||
import { waitForGatewayHealthyRestart, type GatewayRestartSnapshot } from "./restart-health.js";
|
||||
|
||||
/** Returns undefined when the original diagnostic path owns target or authentication handling. */
|
||||
export async function waitForGatewayDiagnosticReadiness(opts: {
|
||||
config?: OpenClawConfig;
|
||||
timeoutMs?: number;
|
||||
deadlineMs?: number;
|
||||
url?: string;
|
||||
token?: string;
|
||||
password?: string;
|
||||
ignoreEnvUrlOverride?: boolean;
|
||||
localPortOverride?: number;
|
||||
onProgress?: (phase: string) => void;
|
||||
}): Promise<GatewayRestartSnapshot | undefined> {
|
||||
if (!(await isImplicitLocalGatewayTarget(opts))) {
|
||||
return undefined;
|
||||
}
|
||||
const probeContext = opts.config
|
||||
? {
|
||||
config: opts.config,
|
||||
auth: (
|
||||
await resolveGatewayProbeAuthSafeWithSecretInputs({
|
||||
cfg: opts.config,
|
||||
mode: "local",
|
||||
explicitAuth: { token: opts.token, password: opts.password },
|
||||
})
|
||||
).auth,
|
||||
}
|
||||
: await resolveGatewayRestartProbeContext(process.env, {
|
||||
token: opts.token,
|
||||
password: opts.password,
|
||||
});
|
||||
if (
|
||||
!probeContext.auth?.token &&
|
||||
!probeContext.auth?.password &&
|
||||
probeContext.config.gateway?.auth?.mode !== "none"
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
const port = opts.localPortOverride ?? resolveGatewayPort(probeContext.config);
|
||||
const nativeService = resolveGatewayService();
|
||||
let nativeCommand: Promise<GatewayServiceCommandConfig | null> | undefined;
|
||||
return waitForGatewayHealthyRestart({
|
||||
port,
|
||||
timeoutMs: opts.timeoutMs ?? DEFAULT_RESTART_HEALTH_TIMEOUT_MS,
|
||||
deadlineMs: opts.deadlineMs,
|
||||
probeContext,
|
||||
probeHosts: LOOPBACK_PORT_PROBE_HOSTS,
|
||||
requirePluginHealth: false,
|
||||
onProgress: opts.onProgress,
|
||||
service: {
|
||||
readCommand: async () => null,
|
||||
readRuntime: async (env, options) => {
|
||||
const owner = readGatewayOwnerLease({ env, port });
|
||||
if (
|
||||
owner?.state === "live" &&
|
||||
(owner.mode === "foreground" || owner.supervisor?.kind === "external")
|
||||
) {
|
||||
return { status: "running", pid: owner.pid };
|
||||
}
|
||||
const startedAt = performance.now();
|
||||
const command = await (nativeCommand ??= nativeService
|
||||
.readCommand(env, options)
|
||||
.catch(() => null));
|
||||
const serviceEnv = mergeGatewayServiceEnv(env, command);
|
||||
const servicePort =
|
||||
parseTcpPortFromArgs(command?.programArguments) ??
|
||||
resolveGatewayPort(probeContext.config, serviceEnv);
|
||||
if (
|
||||
!command ||
|
||||
servicePort !== port ||
|
||||
resolveStateDir(serviceEnv) !== resolveStateDir(env) ||
|
||||
resolveConfigPath(serviceEnv) !== resolveConfigPath(env)
|
||||
) {
|
||||
return { status: "unknown" };
|
||||
}
|
||||
return nativeService.readRuntime(env, {
|
||||
...options,
|
||||
...(options?.timeoutMs === undefined
|
||||
? {}
|
||||
: { timeoutMs: Math.max(1, options.timeoutMs - (performance.now() - startedAt)) }),
|
||||
});
|
||||
},
|
||||
},
|
||||
});
|
||||
}
|
||||
|
|
@ -30,6 +30,39 @@ import type {
|
|||
} from "./restart-health.types.js";
|
||||
import { allListenersOwnedByRuntimePid } from "./restart-port-ownership.js";
|
||||
|
||||
const GATEWAY_RESTART_PROBE_TIMEOUT_MS = 3_000;
|
||||
|
||||
export async function readGatewayStartupPhase(params: {
|
||||
configuredProbe: ConfiguredGatewayLocalProbe;
|
||||
port: number;
|
||||
timeoutMs?: number;
|
||||
signal?: AbortSignal;
|
||||
}): Promise<string | undefined> {
|
||||
const response = await params.configuredProbe.requestHttp({
|
||||
host: "127.0.0.1",
|
||||
pathname: "/startupz",
|
||||
port: params.port,
|
||||
timeoutMs: Math.min(
|
||||
params.timeoutMs ?? GATEWAY_RESTART_PROBE_TIMEOUT_MS,
|
||||
GATEWAY_RESTART_PROBE_TIMEOUT_MS,
|
||||
),
|
||||
...(params.signal ? { signal: params.signal } : {}),
|
||||
});
|
||||
if (response?.statusCode !== 503) {
|
||||
return undefined;
|
||||
}
|
||||
try {
|
||||
const startup = asOptionalRecord(JSON.parse(response.body));
|
||||
return startup?.status === "starting" &&
|
||||
typeof startup.pendingReason === "string" &&
|
||||
startup.pendingReason.trim()
|
||||
? formatGatewayRestartProbeError(startup.pendingReason)
|
||||
: undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
export type GatewayRestartProbeAuth = {
|
||||
token?: string;
|
||||
password?: string;
|
||||
|
|
@ -76,7 +109,10 @@ export async function waitForGatewayHttpReadiness(params: {
|
|||
host: "127.0.0.1",
|
||||
pathname: "/healthz",
|
||||
port: params.port,
|
||||
timeoutMs: Math.min(remainingMs, params.probeTimeoutMs ?? 3_000),
|
||||
timeoutMs: Math.min(
|
||||
remainingMs,
|
||||
params.probeTimeoutMs ?? GATEWAY_RESTART_PROBE_TIMEOUT_MS,
|
||||
),
|
||||
...(params.signal ? { signal: params.signal } : {}),
|
||||
})
|
||||
.then((result) => result?.statusCode ?? null),
|
||||
|
|
@ -85,7 +121,10 @@ export async function waitForGatewayHttpReadiness(params: {
|
|||
host: "127.0.0.1",
|
||||
pathname: "/readyz",
|
||||
port: params.port,
|
||||
timeoutMs: Math.min(remainingMs, params.probeTimeoutMs ?? 3_000),
|
||||
timeoutMs: Math.min(
|
||||
remainingMs,
|
||||
params.probeTimeoutMs ?? GATEWAY_RESTART_PROBE_TIMEOUT_MS,
|
||||
),
|
||||
...(params.signal ? { signal: params.signal } : {}),
|
||||
})
|
||||
.then((result) => result?.statusCode ?? null),
|
||||
|
|
@ -269,7 +308,7 @@ export async function confirmGatewayReachable(params: {
|
|||
requireLocalBackendSharedAuth: authNone,
|
||||
deviceIdentity: null,
|
||||
sharedStateMode: "read-only",
|
||||
timeoutMs: params.timeoutMs ?? 3_000,
|
||||
timeoutMs: params.timeoutMs ?? GATEWAY_RESTART_PROBE_TIMEOUT_MS,
|
||||
...(params.signal ? { signal: params.signal } : {}),
|
||||
onHelloOk: (hello) => {
|
||||
result.gatewayVersion = hello.server.version;
|
||||
|
|
@ -306,6 +345,7 @@ export type GatewayRestartProbeContext = {
|
|||
|
||||
export async function resolveGatewayRestartProbeContext(
|
||||
env: NodeJS.ProcessEnv | undefined,
|
||||
explicitAuth?: GatewayRestartProbeAuth,
|
||||
): Promise<GatewayRestartProbeContext> {
|
||||
const mergedEnv: NodeJS.ProcessEnv = { ...process.env, ...env };
|
||||
const cfg = await createConfigIO({
|
||||
|
|
@ -320,6 +360,7 @@ export async function resolveGatewayRestartProbeContext(
|
|||
cfg,
|
||||
mode: "local",
|
||||
env: mergedEnv,
|
||||
explicitAuth,
|
||||
});
|
||||
return { auth: resolved.auth, config: cfg };
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2,9 +2,11 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
|||
import { gatewayHealthResponse } from "../../gateway/health-response.test-support.js";
|
||||
import {
|
||||
callGateway,
|
||||
classifyPortListener,
|
||||
inspectPortUsage,
|
||||
makeGatewayService,
|
||||
monotonicClock,
|
||||
requestStartupProbe,
|
||||
resetRestartHealthMocks,
|
||||
restoreRestartHealthMocks,
|
||||
} from "./restart-health.test-helpers.js";
|
||||
|
|
@ -16,6 +18,209 @@ describe("restart startup progress", () => {
|
|||
beforeEach(resetRestartHealthMocks);
|
||||
afterEach(restoreRestartHealthMocks);
|
||||
|
||||
it.each([20_000, 90_000])(
|
||||
"observes explicit startup phases before readiness at %d ms",
|
||||
async (readyAtMs) => {
|
||||
inspectPortUsage.mockResolvedValue({
|
||||
port: 18789,
|
||||
status: "busy",
|
||||
listeners: [{ pid: 8000 }],
|
||||
hints: [],
|
||||
});
|
||||
requestStartupProbe.mockImplementation(async () => ({
|
||||
statusCode: monotonicClock.nowMs < readyAtMs ? 503 : 200,
|
||||
body: JSON.stringify(
|
||||
monotonicClock.nowMs < readyAtMs
|
||||
? { status: "starting", pendingReason: "plugin-convergence" }
|
||||
: { status: "started" },
|
||||
),
|
||||
}));
|
||||
callGateway.mockImplementation(gatewayHealthResponse());
|
||||
const onProgress = vi.fn();
|
||||
const result = await waitForGatewayHealthyRestart({
|
||||
service: makeGatewayService({ status: "running", pid: 8000 }),
|
||||
port: 18789,
|
||||
requirePluginHealth: false,
|
||||
timeoutMs: 60_000,
|
||||
onProgress,
|
||||
});
|
||||
expect(result).toMatchObject(
|
||||
readyAtMs < 60_000
|
||||
? { healthy: true, waitOutcome: "healthy", elapsedMs: readyAtMs }
|
||||
: {
|
||||
healthy: false,
|
||||
waitOutcome: "still-starting",
|
||||
elapsedMs: 60_000,
|
||||
startupPhase: "plugin-convergence",
|
||||
},
|
||||
);
|
||||
expect(onProgress).toHaveBeenCalledWith("plugin-convergence");
|
||||
expect(callGateway).toHaveBeenCalledTimes(readyAtMs < 60_000 ? 1 : 0);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
{ statusCode: 503, body: '{"ready":false,"failing":["discord"]}' },
|
||||
{ statusCode: 503, body: '{"status":"draining","pendingReason":"shutdown"}' },
|
||||
{ statusCode: 200, body: '{"status":"starting","pendingReason":"invalid-status"}' },
|
||||
])("does not mistake another HTTP readiness outcome for startup: %j", async (response) => {
|
||||
inspectPortUsage.mockResolvedValue({
|
||||
port: 18789,
|
||||
status: "busy",
|
||||
listeners: [{ pid: 8000 }],
|
||||
hints: [],
|
||||
});
|
||||
requestStartupProbe.mockResolvedValue(response);
|
||||
callGateway.mockImplementation(gatewayHealthResponse());
|
||||
const result = await waitForGatewayHealthyRestart({
|
||||
service: makeGatewayService({ status: "running", pid: 8000 }),
|
||||
port: 18789,
|
||||
requirePluginHealth: false,
|
||||
});
|
||||
expect(result).toMatchObject({ healthy: true, waitOutcome: "healthy", elapsedMs: 0 });
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ deadlineMs: 1_500, healthy: false, waitOutcome: "timeout", elapsedMs: 1_500 },
|
||||
{ deadlineMs: 2_000, healthy: true, waitOutcome: "healthy", elapsedMs: 2_000 },
|
||||
])("keeps settling within the original $deadlineMs ms deadline", async (expected) => {
|
||||
inspectPortUsage.mockResolvedValue({
|
||||
port: 18789,
|
||||
status: "busy",
|
||||
listeners: [{ pid: 8000 }],
|
||||
hints: [],
|
||||
});
|
||||
callGateway.mockImplementation(async (options) => {
|
||||
if (monotonicClock.nowMs < 1_000) {
|
||||
throw new Error("connect ECONNREFUSED");
|
||||
}
|
||||
return gatewayHealthResponse()(options);
|
||||
});
|
||||
|
||||
const result = await waitForGatewayHealthyRestart({
|
||||
service: makeGatewayService({ status: "running", pid: 8000 }),
|
||||
port: 18789,
|
||||
requirePluginHealth: false,
|
||||
timeoutMs: 1_000,
|
||||
deadlineMs: expected.deadlineMs,
|
||||
settle: { probes: 3 },
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({
|
||||
healthy: expected.healthy,
|
||||
waitOutcome: expected.waitOutcome,
|
||||
elapsedMs: expected.elapsedMs,
|
||||
});
|
||||
expect(monotonicClock.nowMs).toBe(expected.deadlineMs);
|
||||
});
|
||||
|
||||
it("does not retain still-starting evidence across a changed process generation", async () => {
|
||||
const service = makeGatewayService({ status: "running", pid: 8000 });
|
||||
vi.mocked(service.readRuntime).mockImplementation(async () => ({
|
||||
status: "running",
|
||||
pid: monotonicClock.nowMs < 1_000 ? 8000 : 9000,
|
||||
}));
|
||||
inspectPortUsage.mockResolvedValue({
|
||||
port: 18789,
|
||||
status: "busy",
|
||||
listeners: [{ pid: 8000 }, { pid: 9000 }],
|
||||
hints: [],
|
||||
});
|
||||
requestStartupProbe.mockResolvedValue({
|
||||
statusCode: 503,
|
||||
body: '{"status":"starting","pendingReason":"plugin-convergence"}',
|
||||
});
|
||||
const result = await waitForGatewayHealthyRestart({
|
||||
service,
|
||||
port: 18789,
|
||||
timeoutMs: 2_000,
|
||||
requirePluginHealth: false,
|
||||
});
|
||||
expect(result).toMatchObject({
|
||||
healthy: false,
|
||||
waitOutcome: "generation-changed",
|
||||
elapsedMs: 2_000,
|
||||
});
|
||||
});
|
||||
|
||||
it.each([
|
||||
{
|
||||
name: "stopped service with a stale Gateway listener",
|
||||
stopped: true,
|
||||
listenerPid: 9000,
|
||||
outcome: "stale-pids",
|
||||
},
|
||||
{
|
||||
name: "running service with an unrelated listener",
|
||||
listenerPid: 9000,
|
||||
foreign: true,
|
||||
outcome: "timeout",
|
||||
},
|
||||
{
|
||||
name: "owned listener running the wrong version",
|
||||
listenerPid: 8000,
|
||||
expectedVersion: "candidate-version",
|
||||
outcome: "version-mismatch",
|
||||
},
|
||||
{
|
||||
name: "owned listener running the wrong build",
|
||||
listenerPid: 8000,
|
||||
expectedBuildId: "candidate-build",
|
||||
outcome: "build-id-mismatch",
|
||||
},
|
||||
])(
|
||||
"does not let HTTP startup hide $name",
|
||||
async ({ stopped, listenerPid, foreign, expectedVersion, expectedBuildId, outcome }) => {
|
||||
inspectPortUsage.mockResolvedValue({
|
||||
port: 18789,
|
||||
status: "busy",
|
||||
listeners: [{ pid: listenerPid }],
|
||||
hints: [],
|
||||
});
|
||||
requestStartupProbe.mockResolvedValue({
|
||||
statusCode: 503,
|
||||
body: '{"status":"starting","pendingReason":"plugin-convergence"}',
|
||||
});
|
||||
if (foreign) {
|
||||
classifyPortListener.mockReturnValue("unknown");
|
||||
}
|
||||
if (expectedVersion || expectedBuildId) {
|
||||
callGateway.mockImplementation(
|
||||
gatewayHealthResponse({
|
||||
server: { version: "previous-version", buildId: "previous-build" },
|
||||
}),
|
||||
);
|
||||
}
|
||||
const result = await waitForGatewayHealthyRestart({
|
||||
service: makeGatewayService(
|
||||
stopped ? { status: "stopped" } : { status: "running", pid: 8000 },
|
||||
),
|
||||
port: 18789,
|
||||
timeoutMs: 2_000,
|
||||
requireRunningService: true,
|
||||
requirePluginHealth: false,
|
||||
expectedVersion,
|
||||
expectedBuildId,
|
||||
});
|
||||
expect(result).toMatchObject({ healthy: false, waitOutcome: outcome });
|
||||
if (stopped) {
|
||||
expect(result.staleGatewayPids).toEqual([9000]);
|
||||
}
|
||||
if (expectedVersion) {
|
||||
expect(result.versionMismatch).toEqual({
|
||||
expected: expectedVersion,
|
||||
actual: "previous-version",
|
||||
});
|
||||
}
|
||||
if (expectedBuildId) {
|
||||
expect(result.buildIdMismatch).toEqual({
|
||||
expected: expectedBuildId,
|
||||
actual: "previous-build",
|
||||
});
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it.each([null, 1])("observes delayed child readiness with exitCode=%s", async (exitCode) => {
|
||||
const child = { pid: process.pid, exitCode, signalCode: null };
|
||||
inspectPortUsage.mockImplementation(async (port) => ({
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import {
|
|||
import type { GatewayService } from "../../daemon/service.js";
|
||||
import type { CallGatewayOptions } from "../../gateway/call.js";
|
||||
import { gatewayHealthResponse } from "../../gateway/health-response.test-support.js";
|
||||
import type { ConfiguredGatewayLocalProbe } from "../../gateway/local-http-probe.js";
|
||||
import type { GatewayLockIdentity } from "../../infra/gateway-lock.js";
|
||||
import type { PortUsage } from "../../infra/ports-types.js";
|
||||
|
||||
|
|
@ -29,6 +30,25 @@ export function gatewayResponseError(message: string): GatewayProtocolRequestErr
|
|||
retainGatewayResponsePayload(error, undefined);
|
||||
return error;
|
||||
}
|
||||
export const requestStartupProbe = vi.fn<ConfiguredGatewayLocalProbe["requestHttp"]>();
|
||||
|
||||
vi.mock("../../gateway/local-http-probe.js", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("../../gateway/local-http-probe.js")>();
|
||||
return {
|
||||
...actual,
|
||||
createConfiguredGatewayLocalProbe: (
|
||||
...args: Parameters<typeof actual.createConfiguredGatewayLocalProbe>
|
||||
) => {
|
||||
const probe = actual.createConfiguredGatewayLocalProbe(...args);
|
||||
return {
|
||||
...probe,
|
||||
requestHttp: (params: Parameters<ConfiguredGatewayLocalProbe["requestHttp"]>[0]) =>
|
||||
params.pathname === "/startupz" ? requestStartupProbe(params) : probe.requestHttp(params),
|
||||
};
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
export const createConfigIO = vi.fn();
|
||||
export const readBestEffortConfig = vi.fn(async () => ({}));
|
||||
export const resolveGatewayProbeAuthSafeWithSecretInputs = vi.fn<
|
||||
|
|
@ -237,6 +257,8 @@ export async function waitForStoppedFreeGatewayRestart(
|
|||
}
|
||||
|
||||
export function resetRestartHealthMocks() {
|
||||
requestStartupProbe.mockReset();
|
||||
requestStartupProbe.mockResolvedValue(null);
|
||||
monotonicClock.nowMs = 0;
|
||||
vi.spyOn(performance, "now").mockImplementation(() => monotonicClock.nowMs);
|
||||
inspectPortUsage.mockReset();
|
||||
|
|
|
|||
|
|
@ -22,6 +22,7 @@ import { isPidAlive } from "../../shared/pid-alive.js";
|
|||
import { sleep } from "../../utils.js";
|
||||
import {
|
||||
confirmGatewayReachable,
|
||||
readGatewayStartupPhase,
|
||||
resolveGatewayRestartProbeContext,
|
||||
type GatewayReachability,
|
||||
type GatewayRestartProbeContext,
|
||||
|
|
@ -31,7 +32,11 @@ import {
|
|||
DEFAULT_RESTART_HEALTH_DELAY_MS,
|
||||
} from "./restart-health.constants.js";
|
||||
import type { GatewayRestartSnapshot, GatewayRestartWaitOutcome } from "./restart-health.types.js";
|
||||
import { hasListenerAttributionGap, listenerOwnedByRuntimePid } from "./restart-port-ownership.js";
|
||||
import {
|
||||
allListenersOwnedByRuntimePid,
|
||||
hasListenerAttributionGap,
|
||||
listenerOwnedByRuntimePid,
|
||||
} from "./restart-port-ownership.js";
|
||||
export {
|
||||
DEFAULT_RESTART_HEALTH_ATTEMPTS,
|
||||
DEFAULT_RESTART_HEALTH_DELAY_MS,
|
||||
|
|
@ -124,6 +129,9 @@ export async function inspectGatewayRestart(params: {
|
|||
);
|
||||
let reachability: GatewayReachability | null = null;
|
||||
let probeError: string | undefined;
|
||||
let gatewayBootId: string | undefined;
|
||||
let gatewayVersion: string | null | undefined;
|
||||
let gatewayBuildId: string | null | undefined;
|
||||
let activatedPluginErrors: PluginHealthErrorSummary[] = [];
|
||||
let unavailablePlugins: GatewayReachability["unavailablePlugins"] = [];
|
||||
let channelProbeErrors: Array<{ id: string; error: string }> = [];
|
||||
|
|
@ -138,6 +146,9 @@ export async function inspectGatewayRestart(params: {
|
|||
...(params.signal ? { signal: params.signal } : {}),
|
||||
});
|
||||
probeError = reachability.probeError;
|
||||
gatewayBootId = reachability.gatewayBootId;
|
||||
gatewayVersion = reachability.gatewayVersion;
|
||||
gatewayBuildId = reachability.gatewayBuildId;
|
||||
activatedPluginErrors = reachability.activatedPluginErrors;
|
||||
unavailablePlugins = reachability.unavailablePlugins;
|
||||
channelProbeErrors = reachability.channelProbeErrors;
|
||||
|
|
@ -171,7 +182,20 @@ export async function inspectGatewayRestart(params: {
|
|||
}
|
||||
|
||||
params.signal?.throwIfAborted();
|
||||
if (portUsage.status === "busy" && runtime.status !== "running") {
|
||||
const startupPhase =
|
||||
portUsage.status === "busy" && params.configuredProbe
|
||||
? await readGatewayStartupPhase({
|
||||
configuredProbe: params.configuredProbe,
|
||||
port: params.port,
|
||||
timeoutMs: remainingTimeoutMs(),
|
||||
...(params.signal ? { signal: params.signal } : {}),
|
||||
})
|
||||
: undefined;
|
||||
if (startupPhase && (expectedVersion || expectedBuildId)) {
|
||||
// Startup cannot conceal a previous install or a definitive plugin/channel failure.
|
||||
await loadReachability();
|
||||
}
|
||||
if (!startupPhase && portUsage.status === "busy" && runtime.status !== "running") {
|
||||
const reachable = await loadReachability();
|
||||
if (reachable.reachable) {
|
||||
return finalizeGatewayRestartSnapshot(
|
||||
|
|
@ -215,23 +239,20 @@ export async function inspectGatewayRestart(params: {
|
|||
listenerOwnedByRuntimePid({ listener, runtimePid }),
|
||||
) || listenerAttributionGap
|
||||
: gatewayListeners.length > 0 || listenerAttributionGap;
|
||||
let healthy = running && ownsPort;
|
||||
let gatewayBootId: string | undefined;
|
||||
let gatewayVersion: string | null | undefined;
|
||||
let gatewayBuildId: string | null | undefined;
|
||||
let healthy = running && ownsPort && !startupPhase;
|
||||
if (requiresGatewayProbe && healthy && portUsage.status === "busy") {
|
||||
const reachable = await loadReachability();
|
||||
healthy = reachable.reachable;
|
||||
gatewayBootId = reachable.gatewayBootId;
|
||||
gatewayVersion = reachable.gatewayVersion;
|
||||
gatewayBuildId = reachable.gatewayBuildId;
|
||||
}
|
||||
if (!healthy && running && portUsage.status === "busy" && !requiresGatewayProbe) {
|
||||
if (
|
||||
!healthy &&
|
||||
!startupPhase &&
|
||||
running &&
|
||||
portUsage.status === "busy" &&
|
||||
!requiresGatewayProbe
|
||||
) {
|
||||
const reachable = await loadReachability();
|
||||
healthy = reachable.reachable;
|
||||
gatewayBootId = reachable.gatewayBootId;
|
||||
gatewayVersion = reachable.gatewayVersion;
|
||||
gatewayBuildId = reachable.gatewayBuildId;
|
||||
}
|
||||
// Read after probes: an owner can acquire the coordinator while health is unavailable.
|
||||
const owner =
|
||||
|
|
@ -262,6 +283,7 @@ export async function inspectGatewayRestart(params: {
|
|||
...(gatewayBootId ? { gatewayBootId } : {}),
|
||||
...(gatewayVersion !== undefined ? { gatewayVersion } : {}),
|
||||
...(gatewayBuildId !== undefined ? { gatewayBuildId } : {}),
|
||||
...(startupPhase ? { startupPhase } : {}),
|
||||
...(probeError ? { probeError } : {}),
|
||||
...(activatedPluginErrors.length ? { activatedPluginErrors } : {}),
|
||||
...(unavailablePlugins.length ? { unavailablePlugins } : {}),
|
||||
|
|
@ -315,6 +337,8 @@ type GatewayRestartWaitOptions = {
|
|||
attempts?: number;
|
||||
delayMs?: number;
|
||||
timeoutMs?: number;
|
||||
/** Absolute performance.now() deadline supplied by a longer diagnostic operation. */
|
||||
deadlineMs?: number;
|
||||
settle?: { probes: number };
|
||||
env?: NodeJS.ProcessEnv;
|
||||
expectedVersion?: string | null;
|
||||
|
|
@ -324,6 +348,8 @@ type GatewayRestartWaitOptions = {
|
|||
supervisorKeepsAlive?: boolean;
|
||||
isStartupMigrationActive?: typeof hasActiveStartupMigrationLease;
|
||||
probeHosts?: readonly string[];
|
||||
probeContext?: GatewayRestartProbeContext;
|
||||
onProgress?: (phase: string) => void;
|
||||
signal?: AbortSignal;
|
||||
};
|
||||
|
||||
|
|
@ -350,23 +376,49 @@ export async function waitForGatewayHealthyRestart(
|
|||
}),
|
||||
};
|
||||
const startedAtMs = performance.now();
|
||||
const remainingDeadlineMs =
|
||||
params.deadlineMs === undefined ? undefined : Math.max(0, params.deadlineMs - startedAtMs);
|
||||
const timeoutMs =
|
||||
remainingDeadlineMs === undefined
|
||||
? params.timeoutMs
|
||||
: Math.min(params.timeoutMs ?? remainingDeadlineMs, remainingDeadlineMs);
|
||||
if (remainingDeadlineMs === 0) {
|
||||
return withWaitContext(
|
||||
{
|
||||
runtime: { status: "unknown" },
|
||||
portUsage: { port: params.port, status: "unknown", listeners: [], hints: [] },
|
||||
healthy: false,
|
||||
staleGatewayPids: [],
|
||||
probeError: "Gateway readiness budget exhausted.",
|
||||
},
|
||||
"timeout",
|
||||
0,
|
||||
);
|
||||
}
|
||||
const attempts = params.attempts ?? DEFAULT_RESTART_HEALTH_ATTEMPTS;
|
||||
const delayMs = params.delayMs ?? DEFAULT_RESTART_HEALTH_DELAY_MS;
|
||||
const settleProbes = Math.max(1, params.settle?.probes ?? 1);
|
||||
const settleDurationMs = (settleProbes - 1) * delayMs;
|
||||
const settleDurationMs = Math.min(
|
||||
(settleProbes - 1) * delayMs,
|
||||
remainingDeadlineMs === undefined
|
||||
? Infinity
|
||||
: Math.max(0, remainingDeadlineMs - (params.timeoutMs ?? remainingDeadlineMs)),
|
||||
);
|
||||
// A longer update budget must not make an old heartbeat count as fresh progress.
|
||||
const progressWindowMs = attempts * delayMs;
|
||||
const standardDeadlineMs = params.timeoutMs ?? progressWindowMs;
|
||||
const standardDeadlineMs = timeoutMs ?? progressWindowMs;
|
||||
const probeTimeoutMs = () =>
|
||||
params.timeoutMs === undefined
|
||||
timeoutMs === undefined
|
||||
? undefined
|
||||
: Math.max(1, params.timeoutMs + settleDurationMs - (performance.now() - startedAtMs));
|
||||
: Math.max(1, timeoutMs + settleDurationMs - (performance.now() - startedAtMs));
|
||||
const updateInProgress = (params.env ?? process.env).OPENCLAW_UPDATE_IN_PROGRESS === "1";
|
||||
|
||||
const probeContext = await resolveGatewayRestartProbeContext(params.env).catch(() => ({
|
||||
auth: undefined,
|
||||
config: {},
|
||||
}));
|
||||
const probeContext =
|
||||
params.probeContext ??
|
||||
(await resolveGatewayRestartProbeContext(params.env).catch(() => ({
|
||||
auth: undefined,
|
||||
config: {},
|
||||
})));
|
||||
const configuredProbe = createConfiguredGatewayLocalProbe(probeContext.config);
|
||||
const probeHosts =
|
||||
params.probeHosts ??
|
||||
|
|
@ -406,22 +458,35 @@ export async function waitForGatewayHealthyRestart(
|
|||
let observedPid: number | undefined;
|
||||
let observedBootId: string | undefined;
|
||||
let generationChanged = false;
|
||||
const expiredOutcome = (elapsedMs: number, atStartupCap: boolean): GatewayRestartWaitOutcome =>
|
||||
atStartupCap &&
|
||||
!generationChanged &&
|
||||
snapshot.runtime.status === "running" &&
|
||||
(snapshot.runtime.pid !== undefined || snapshot.gatewayBootId !== undefined) &&
|
||||
!snapshot.versionMismatch &&
|
||||
!snapshot.buildIdMismatch &&
|
||||
!snapshot.channelProbeErrors?.length &&
|
||||
!(params.requirePluginHealth !== false && snapshot.activatedPluginErrors?.length) &&
|
||||
snapshot.staleGatewayPids.length === 0 &&
|
||||
startupProgressDeadlineMs > progressWindowMs &&
|
||||
elapsedMs < startupProgressDeadlineMs
|
||||
let reportedStartupPhase: string | undefined;
|
||||
let lastProgressPhase: string | undefined;
|
||||
const expiredOutcome = (elapsedMs: number, atStartupCap: boolean): GatewayRestartWaitOutcome => {
|
||||
if (generationChanged) {
|
||||
return "generation-changed";
|
||||
}
|
||||
if (
|
||||
snapshot.runtime.status !== "running" ||
|
||||
(snapshot.runtime.pid === undefined && snapshot.gatewayBootId === undefined) ||
|
||||
snapshot.versionMismatch ||
|
||||
snapshot.buildIdMismatch ||
|
||||
snapshot.channelProbeErrors?.length ||
|
||||
(params.requirePluginHealth !== false && snapshot.activatedPluginErrors?.length) ||
|
||||
snapshot.staleGatewayPids.length > 0
|
||||
) {
|
||||
return "timeout";
|
||||
}
|
||||
const ownedStartup =
|
||||
reportedStartupPhase &&
|
||||
snapshot.portUsage.status === "busy" &&
|
||||
snapshot.runtime.pid !== undefined &&
|
||||
allListenersOwnedByRuntimePid(snapshot.portUsage.listeners, snapshot.runtime.pid);
|
||||
return ownedStartup ||
|
||||
(atStartupCap &&
|
||||
startupProgressDeadlineMs > progressWindowMs &&
|
||||
elapsedMs < startupProgressDeadlineMs)
|
||||
? "still-starting"
|
||||
: generationChanged
|
||||
? "generation-changed"
|
||||
: "timeout";
|
||||
: "timeout";
|
||||
};
|
||||
|
||||
for (let attempt = 0; ; attempt += 1) {
|
||||
params.signal?.throwIfAborted();
|
||||
|
|
@ -444,7 +509,7 @@ export async function waitForGatewayHealthyRestart(
|
|||
// process earns the startup watchdog; later phases never reset its finite cap.
|
||||
updateStartupDeadlineMs ??= Math.max(standardDeadlineMs, STARTUP_MIGRATION_LEASE_TTL_MS);
|
||||
}
|
||||
const boundedDeadlineMs = params.timeoutMs ?? updateStartupDeadlineMs;
|
||||
const boundedDeadlineMs = timeoutMs ?? updateStartupDeadlineMs;
|
||||
// A managed settle streak needs a concrete process identity. Scheduled Tasks can
|
||||
// report running without exposing a PID, so Windows retains status-only proof.
|
||||
const healthy =
|
||||
|
|
@ -452,13 +517,16 @@ export async function waitForGatewayHealthyRestart(
|
|||
(!params.requireRunningService ||
|
||||
(snapshot.runtime.status === "running" &&
|
||||
(process.platform === "win32" || typeof snapshot.runtime.pid === "number")));
|
||||
snapshot.startupPhase = healthy
|
||||
? "settling healthy Gateway"
|
||||
: snapshot.runtime.status !== "running"
|
||||
? "waiting for managed service"
|
||||
: snapshot.portUsage.status === "free"
|
||||
? "waiting for Gateway listener"
|
||||
: "waiting for Gateway health and identity";
|
||||
reportedStartupPhase = snapshot.startupPhase;
|
||||
snapshot.startupPhase =
|
||||
reportedStartupPhase ??
|
||||
(healthy
|
||||
? "settling healthy Gateway"
|
||||
: snapshot.runtime.status !== "running"
|
||||
? "waiting for managed service"
|
||||
: snapshot.portUsage.status === "free"
|
||||
? "waiting for Gateway listener"
|
||||
: "waiting for Gateway health and identity");
|
||||
if (boundedDeadlineMs !== undefined && elapsedMs > boundedDeadlineMs + settleDurationMs) {
|
||||
return withWaitContext(
|
||||
{ ...snapshot, healthy: false },
|
||||
|
|
@ -570,9 +638,16 @@ export async function waitForGatewayHealthyRestart(
|
|||
elapsedMs = Math.max(0, performance.now() - startedAtMs);
|
||||
}
|
||||
|
||||
if (migrationActive) {
|
||||
if (migrationActive && !reportedStartupPhase) {
|
||||
snapshot.startupPhase = "startup migration";
|
||||
}
|
||||
if (
|
||||
(reportedStartupPhase || snapshot.runtime.status === "running") &&
|
||||
snapshot.startupPhase !== lastProgressPhase
|
||||
) {
|
||||
lastProgressPhase = snapshot.startupPhase;
|
||||
params.onProgress?.(snapshot.startupPhase);
|
||||
}
|
||||
const runtimePid = snapshot.runtime.pid;
|
||||
const ownsListener =
|
||||
snapshot.portUsage.status === "busy" &&
|
||||
|
|
@ -628,7 +703,12 @@ export async function waitForGatewayHealthyRestart(
|
|||
);
|
||||
}
|
||||
}
|
||||
await sleep(delayMs, params.signal);
|
||||
await sleep(
|
||||
boundedDeadlineMs === undefined
|
||||
? delayMs
|
||||
: Math.min(delayMs, Math.max(0, boundedDeadlineMs + settleDurationMs - elapsedMs)),
|
||||
params.signal,
|
||||
);
|
||||
snapshot = await inspectGatewayRestart({
|
||||
service,
|
||||
port: params.port,
|
||||
|
|
|
|||
|
|
@ -222,7 +222,7 @@ export function registerStatusHealthSessionsCommands(program: Command) {
|
|||
.option("--usage", "Show model provider usage/quota snapshots", false)
|
||||
.option("--agent <id>", "Agent id for --usage auth scope")
|
||||
.option("--deep", "Probe channels (WhatsApp Web + Telegram + Discord + Slack + Signal)", false)
|
||||
.option("--timeout <ms>", "Probe timeout in milliseconds", "10000")
|
||||
.option("--timeout <ms>", "Probe timeout in milliseconds")
|
||||
.option("--verbose", "Verbose logging", false)
|
||||
.option("--debug", "Alias for --verbose", false)
|
||||
.addHelpText(
|
||||
|
|
@ -267,7 +267,7 @@ export function registerStatusHealthSessionsCommands(program: Command) {
|
|||
.command("health")
|
||||
.description("Fetch health from the running gateway")
|
||||
.option("--json", "Output JSON instead of text", false)
|
||||
.option("--timeout <ms>", "Connection timeout in milliseconds", "10000")
|
||||
.option("--timeout <ms>", "Connection timeout in milliseconds")
|
||||
.option("--verbose", "Verbose logging", false)
|
||||
.option("--debug", "Alias for --verbose", false)
|
||||
.addHelpText(
|
||||
|
|
|
|||
|
|
@ -1,5 +1,11 @@
|
|||
// Channels status command-flow tests cover gateway calls, config fallback, and timeout validation.
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
vi.mock("../cli/daemon-cli/diagnostic-readiness.js", () => ({
|
||||
waitForGatewayDiagnosticReadiness: vi.fn(async () => undefined),
|
||||
}));
|
||||
import { validateChannelsStatusParams } from "../../packages/gateway-protocol/src/index.js";
|
||||
import { waitForGatewayDiagnosticReadiness } from "../cli/daemon-cli/diagnostic-readiness.js";
|
||||
import { GatewaySecretRefUnavailableError } from "../gateway/credentials.js";
|
||||
import { GatewayTransportError } from "../gateway/transport-error.js";
|
||||
import { DEFAULT_ACCOUNT_ID } from "../routing/session-key.js";
|
||||
|
|
@ -178,6 +184,7 @@ function createGatewayTransportError(message = "Gateway not reachable (ECONNREFU
|
|||
|
||||
describe("channelsStatusCommand SecretRef fallback flow", () => {
|
||||
beforeEach(() => {
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
mocks.callGateway.mockReset();
|
||||
mocks.resolveCommandConfigWithSecrets.mockReset();
|
||||
mocks.readConfigFileSnapshot.mockClear();
|
||||
|
|
@ -191,6 +198,53 @@ describe("channelsStatusCommand SecretRef fallback flow", () => {
|
|||
mocks.listChannelPlugins.mockReturnValue([createTokenOnlyPlugin()]);
|
||||
});
|
||||
|
||||
it("sends valid channel RPC parameters after fractional startup timing", async () => {
|
||||
const runtime = { log: vi.fn(), error: vi.fn(), exit: vi.fn() };
|
||||
vi.mocked(waitForGatewayDiagnosticReadiness).mockImplementationOnce(async () => {
|
||||
vi.spyOn(performance, "now").mockReturnValue(1250.25);
|
||||
return {
|
||||
healthy: true,
|
||||
waitOutcome: "healthy",
|
||||
elapsedMs: 1250.25,
|
||||
runtime: { status: "running", pid: 42 },
|
||||
portUsage: { port: 18789, status: "busy", listeners: [{ pid: 42 }], hints: [] },
|
||||
staleGatewayPids: [],
|
||||
};
|
||||
});
|
||||
mocks.callGateway.mockResolvedValueOnce({});
|
||||
|
||||
await channelsStatusCommand({ probe: true, json: true, timeout: "5000" }, runtime);
|
||||
|
||||
expect(mocks.callGateway).toHaveBeenCalledOnce();
|
||||
const request = mocks.callGateway.mock.calls[0]?.[0];
|
||||
expect(validateChannelsStatusParams(request?.params)).toBe(true);
|
||||
expect(request?.params.timeoutMs).toBe(3750);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it("reports pending startup in JSON without falling back to a Gateway error", async () => {
|
||||
const runtime = { log: vi.fn(), error: vi.fn(), exit: vi.fn() };
|
||||
vi.mocked(waitForGatewayDiagnosticReadiness).mockResolvedValueOnce({
|
||||
healthy: false,
|
||||
waitOutcome: "still-starting",
|
||||
startupPhase: "startup-sidecars",
|
||||
elapsedMs: 60_000,
|
||||
runtime: { status: "running", pid: 42 },
|
||||
portUsage: { port: 18789, status: "busy", listeners: [], hints: [] },
|
||||
staleGatewayPids: [],
|
||||
});
|
||||
await channelsStatusCommand({ probe: true, json: true }, runtime);
|
||||
expect(runtime.log.mock.calls.map(([message]) => JSON.parse(message))).toEqual([
|
||||
{ status: "starting", startupPhase: "startup-sidecars" },
|
||||
]);
|
||||
expect(runtime.error).not.toHaveBeenCalled();
|
||||
expect(mocks.callGateway).not.toHaveBeenCalled();
|
||||
expect(mocks.requireValidConfig).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("passes a channel filter to the gateway status request", async () => {
|
||||
mocks.callGateway.mockResolvedValue({
|
||||
channelAccounts: { imessage: [] },
|
||||
|
|
@ -202,8 +256,8 @@ describe("channelsStatusCommand SecretRef fallback flow", () => {
|
|||
|
||||
expect(mocks.callGateway).toHaveBeenCalledWith({
|
||||
method: "channels.status",
|
||||
params: { channel: "imsg", probe: true, timeoutMs: 30000 },
|
||||
timeoutMs: 30000,
|
||||
params: { channel: "imsg", probe: true, timeoutMs: 60000 },
|
||||
timeoutMs: 60000,
|
||||
});
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
// Implements `openclaw channels status` with gateway status and config-only fallback.
|
||||
import { redactSensitiveUrlLikeString } from "@openclaw/net-policy/redact-sensitive-url";
|
||||
import { normalizeOptionalLowercaseString } from "@openclaw/normalization-core/string-coerce";
|
||||
import { DEFAULT_RESTART_HEALTH_TIMEOUT_MS } from "../../cli/daemon-cli/restart-health.constants.js";
|
||||
import {
|
||||
formatCliFailureLines,
|
||||
isExpectedCliError,
|
||||
|
|
@ -13,6 +14,7 @@ import { isGatewaySecretRefUnavailableError } from "../../gateway/credentials.js
|
|||
import { formatErrorMessage } from "../../infra/errors.js";
|
||||
import { defaultRuntime, type RuntimeEnv, writeRuntimeJson } from "../../runtime.js";
|
||||
import { createLazyRuntimeModule } from "../../shared/lazy-runtime.js";
|
||||
import { waitForGatewayDiagnostic } from "../gateway-diagnostic-readiness.js";
|
||||
|
||||
const loadChannelsStatusRuntime = createLazyRuntimeModule(() => import("./status.runtime.js"));
|
||||
|
||||
|
|
@ -42,7 +44,7 @@ export async function channelsStatusCommand(
|
|||
normalizeOptionalLowercaseString(opts.channel) === "all"
|
||||
? { ...opts, channel: undefined }
|
||||
: opts;
|
||||
const timeoutMs = parseTimeoutMsWithFallback(opts.timeout, opts.probe ? 30_000 : 10_000, {
|
||||
const timeoutMs = parseTimeoutMsWithFallback(opts.timeout, DEFAULT_RESTART_HEALTH_TIMEOUT_MS, {
|
||||
invalidType: "error",
|
||||
});
|
||||
const statusLabel = opts.probe ? "Checking channel status (probe)…" : "Checking channel status…";
|
||||
|
|
@ -51,6 +53,10 @@ export async function channelsStatusCommand(
|
|||
runtime.log(statusLabel);
|
||||
}
|
||||
try {
|
||||
const remainingMs = await waitForGatewayDiagnostic({ timeoutMs, json: opts.json }, runtime);
|
||||
if (remainingMs === undefined) {
|
||||
return;
|
||||
}
|
||||
const payload = await withProgress(
|
||||
{
|
||||
label: statusLabel,
|
||||
|
|
@ -60,7 +66,7 @@ export async function channelsStatusCommand(
|
|||
async () => {
|
||||
const params: { channel?: string; probe: boolean; timeoutMs: number } = {
|
||||
probe: Boolean(opts.probe),
|
||||
timeoutMs,
|
||||
timeoutMs: remainingMs,
|
||||
};
|
||||
if (args.channel) {
|
||||
params.channel = args.channel;
|
||||
|
|
@ -68,7 +74,7 @@ export async function channelsStatusCommand(
|
|||
return await callGateway({
|
||||
method: "channels.status",
|
||||
params,
|
||||
timeoutMs,
|
||||
timeoutMs: remainingMs,
|
||||
});
|
||||
},
|
||||
);
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
// Doctor restart-health tests cover transient ECONNREFUSED after an approved gateway restart.
|
||||
// Doctor delegates post-restart diagnostics to the health command readiness owner.
|
||||
import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { ExitError } from "../runtime.js";
|
||||
import { createDoctorPrompter } from "./doctor-prompter.js";
|
||||
|
|
@ -145,64 +145,15 @@ describe("maybeRepairGatewayDaemon restart health", () => {
|
|||
return runtime;
|
||||
}
|
||||
|
||||
it.each([500, 25_000])(
|
||||
"waits for a Gateway that refuses connections for %i ms after restart",
|
||||
async (readyAfterMs) => {
|
||||
const startedAt = performance.now();
|
||||
healthCommand.mockImplementation(async () => {
|
||||
if (performance.now() - startedAt < readyAfterMs) {
|
||||
throw new Error("connect ECONNREFUSED 127.0.0.1:18789");
|
||||
}
|
||||
});
|
||||
|
||||
const repair = runAutoRepair();
|
||||
await vi.runAllTimersAsync();
|
||||
const runtime = await repair;
|
||||
|
||||
expect(service.restart).toHaveBeenCalledOnce();
|
||||
expect(runtime.error).not.toHaveBeenCalled();
|
||||
expect(performance.now() - startedAt).toBe(readyAfterMs);
|
||||
expect(healthCommand).toHaveBeenCalledTimes(readyAfterMs / 500 + 1);
|
||||
if (readyAfterMs === 25_000) {
|
||||
expect(note).toHaveBeenCalledWith(
|
||||
expect.stringContaining("Gateway is still starting (20 s elapsed)"),
|
||||
"Gateway",
|
||||
);
|
||||
}
|
||||
expect(note.mock.calls.some(([message]) => String(message).includes("not reachable"))).toBe(
|
||||
false,
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
it("reports an unreachable Gateway only after the full restart budget", async () => {
|
||||
healthCommand.mockRejectedValue(new Error("connect ECONNREFUSED 127.0.0.1:18789"));
|
||||
let completed = false;
|
||||
const repair = runAutoRepair().then((runtime) => {
|
||||
completed = true;
|
||||
return runtime;
|
||||
});
|
||||
|
||||
await vi.advanceTimersByTimeAsync(59_999);
|
||||
expect(completed).toBe(false);
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
const runtime = await repair;
|
||||
|
||||
expect(runtime.error).toHaveBeenCalledOnce();
|
||||
expect(runtime.error).toHaveBeenCalledWith(expect.stringContaining("Gateway not reachable"));
|
||||
expect(runtime.error).toHaveBeenCalledWith(expect.stringContaining("Doctor waited 60 s"));
|
||||
expect(runtime.error).toHaveBeenCalledWith(
|
||||
expect.stringContaining("openclaw gateway status --deep"),
|
||||
);
|
||||
expect(runtime.error).toHaveBeenCalledWith(expect.stringContaining("ECONNREFUSED"));
|
||||
expect(vi.getTimerCount()).toBe(0);
|
||||
});
|
||||
|
||||
it("returns on the first probe when the restarted Gateway is healthy", async () => {
|
||||
const startedAt = performance.now();
|
||||
const runtime = await runAutoRepair();
|
||||
|
||||
expect(healthCommand).toHaveBeenCalledOnce();
|
||||
expect(healthCommand).toHaveBeenCalledWith(
|
||||
{ json: false, config: { gateway: {} } },
|
||||
expect.any(Object),
|
||||
);
|
||||
expect(performance.now()).toBe(startedAt);
|
||||
expect(vi.getTimerCount()).toBe(0);
|
||||
expect(runtime.error).not.toHaveBeenCalled();
|
||||
|
|
|
|||
|
|
@ -1085,7 +1085,7 @@ describe("maybeRepairGatewayDaemon", () => {
|
|||
expect(healthCommand).toHaveBeenCalledOnce();
|
||||
expect(service.restart).not.toHaveBeenCalled();
|
||||
expect(note).toHaveBeenCalledWith(
|
||||
"Gateway is healthy after recent restart; skipping restart prompt.",
|
||||
"Preserving the recent Gateway restart; skipping restart prompt.",
|
||||
"Gateway",
|
||||
);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -1,10 +1,6 @@
|
|||
/** Doctor gateway daemon repair flow for service install, bootstrap, restart, and port hints. */
|
||||
import { note } from "../../packages/terminal-core/src/note.js";
|
||||
import { formatCliCommand } from "../cli/command-format.js";
|
||||
import {
|
||||
DEFAULT_RESTART_HEALTH_DELAY_MS,
|
||||
DEFAULT_RESTART_HEALTH_TIMEOUT_MS,
|
||||
} from "../cli/daemon-cli/restart-health.constants.js";
|
||||
import { resolveGatewayPort } from "../config/config.js";
|
||||
import { isDefaultInstallIdentity } from "../config/paths.js";
|
||||
import type { OpenClawConfig } from "../config/types.openclaw.js";
|
||||
|
|
@ -62,53 +58,6 @@ import { resolveGatewaySetupRuntime } from "./gateway-setup-runtime.js";
|
|||
import { formatGatewayClosedDiagnostic, formatHealthCheckFailure } from "./health-format.js";
|
||||
import { healthCommandNonExiting } from "./health.js";
|
||||
|
||||
function isTransientGatewayUnreachableError(error: unknown): boolean {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
return /\bECONNREFUSED\b|couldn't connect|connection refused/i.test(message);
|
||||
}
|
||||
|
||||
async function waitForGatewayHealthAfterRestart(runtime: RuntimeEnv): Promise<void> {
|
||||
const startedAt = performance.now();
|
||||
const deadline = startedAt + DEFAULT_RESTART_HEALTH_TIMEOUT_MS;
|
||||
let nextProgressAt = startedAt + 5_000;
|
||||
let lastError: unknown;
|
||||
while (true) {
|
||||
const remainingMs = deadline - performance.now();
|
||||
if (remainingMs <= 0) {
|
||||
break;
|
||||
}
|
||||
try {
|
||||
await healthCommandNonExiting(
|
||||
{ json: false, timeoutMs: Math.min(10_000, remainingMs) },
|
||||
runtime,
|
||||
);
|
||||
return;
|
||||
} catch (err) {
|
||||
lastError = err;
|
||||
if (err instanceof ExitError || !isTransientGatewayUnreachableError(err)) {
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
const now = performance.now();
|
||||
if (now >= deadline) {
|
||||
break;
|
||||
}
|
||||
if (now >= nextProgressAt) {
|
||||
note(
|
||||
`Gateway is still starting (${Math.round((now - startedAt) / 1000)} s elapsed). The host may be slow; waiting for restart readiness.`,
|
||||
"Gateway",
|
||||
);
|
||||
nextProgressAt = now + 5_000;
|
||||
}
|
||||
await sleep(Math.min(DEFAULT_RESTART_HEALTH_DELAY_MS, deadline - now));
|
||||
}
|
||||
const detail = lastError instanceof Error ? lastError.message : String(lastError);
|
||||
throw new Error(
|
||||
`Gateway not reachable after restart; Doctor waited ${Math.round((performance.now() - startedAt) / 1000)} s.\n${detail}\nRun ${formatCliCommand("openclaw gateway status --deep")} to diagnose.`,
|
||||
{ cause: lastError },
|
||||
);
|
||||
}
|
||||
|
||||
type LaunchAgentBootstrapDoctorOutcome =
|
||||
| { status: "skipped" }
|
||||
| { status: "not-loaded" }
|
||||
|
|
@ -536,8 +485,8 @@ export async function maybeRepairGatewayDaemon(params: {
|
|||
const recentRestart = readGatewayRestartHandoffSync(serviceEnv);
|
||||
if (recentRestart) {
|
||||
try {
|
||||
await healthCommandNonExiting({ json: false, timeoutMs: 10_000 }, params.runtime);
|
||||
note("Gateway is healthy after recent restart; skipping restart prompt.", "Gateway");
|
||||
await healthCommandNonExiting({ json: false, config: params.cfg }, params.runtime);
|
||||
note("Preserving the recent Gateway restart; skipping restart prompt.", "Gateway");
|
||||
return;
|
||||
} catch {
|
||||
// Health probe failed — fall through to the restart prompt below.
|
||||
|
|
@ -567,7 +516,7 @@ export async function maybeRepairGatewayDaemon(params: {
|
|||
return;
|
||||
}
|
||||
try {
|
||||
await waitForGatewayHealthAfterRestart(params.runtime);
|
||||
await healthCommandNonExiting({ json: false, config: params.cfg }, params.runtime);
|
||||
} catch (err) {
|
||||
// A trapped ExitError means healthCommand already printed its own
|
||||
// reachable-gateway diagnostic; re-formatting it would only add noise.
|
||||
|
|
|
|||
|
|
@ -1,11 +1,16 @@
|
|||
// Doctor gateway health tests cover gateway probe failures, auth requirements, and repair messages.
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
vi.mock("../cli/daemon-cli/diagnostic-readiness.js", () => ({
|
||||
waitForGatewayDiagnosticReadiness: vi.fn(async () => undefined),
|
||||
}));
|
||||
import { GatewayClientRequestError } from "../../packages/gateway-client/src/index.js";
|
||||
import {
|
||||
GatewayProtocolRequestTimeoutError,
|
||||
retainGatewayResponsePayload,
|
||||
} from "../../packages/gateway-client/src/protocol-request.js";
|
||||
import { createDeferred } from "../../test/helpers/promise.js";
|
||||
import { waitForGatewayDiagnosticReadiness } from "../cli/daemon-cli/diagnostic-readiness.js";
|
||||
import type { OpenClawConfig } from "../config/config.js";
|
||||
import { GatewayTransportError } from "../gateway/transport-error.js";
|
||||
import { collectChannelStatusIssues } from "../infra/channels-status-issues.js";
|
||||
|
|
@ -65,6 +70,7 @@ describe("checkGatewayHealth", () => {
|
|||
const cfg = {} as OpenClawConfig;
|
||||
|
||||
beforeEach(() => {
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
callGateway.mockReset();
|
||||
isGatewayCredentialsRequiredError.mockReset();
|
||||
isGatewayCredentialsRequiredError.mockReturnValue(false);
|
||||
|
|
@ -80,9 +86,35 @@ describe("checkGatewayHealth", () => {
|
|||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
vi.unstubAllEnvs();
|
||||
});
|
||||
|
||||
it("keeps a starting Gateway non-failing without running diagnostics or prompting recovery", async () => {
|
||||
const runtime = { log: vi.fn(), error: vi.fn(), exit: vi.fn() };
|
||||
vi.mocked(waitForGatewayDiagnosticReadiness).mockImplementationOnce(async (opts) => {
|
||||
opts.onProgress?.("startup-sidecars");
|
||||
return {
|
||||
healthy: false,
|
||||
waitOutcome: "still-starting",
|
||||
startupPhase: "startup-sidecars",
|
||||
elapsedMs: 60_000,
|
||||
runtime: { status: "running", pid: 42 },
|
||||
portUsage: { port: 18789, status: "busy", listeners: [], hints: [] },
|
||||
staleGatewayPids: [],
|
||||
};
|
||||
});
|
||||
await expect(checkGatewayHealth({ runtime, cfg })).resolves.toEqual({
|
||||
healthOk: true,
|
||||
authenticated: false,
|
||||
});
|
||||
expect(runtime.log).toHaveBeenCalledWith(
|
||||
"Gateway is still starting (phase: startup-sidecars).",
|
||||
);
|
||||
expect(runtime.error).not.toHaveBeenCalled();
|
||||
expect(callGateway).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([false, true])(
|
||||
"reports live paths when status RPC rejection is %s",
|
||||
async (rejectStatus) => {
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import { note } from "../../packages/terminal-core/src/note.js";
|
|||
import { sanitizeTerminalText } from "../../packages/terminal-core/src/safe-text.js";
|
||||
import { formatCliCommand } from "../cli/command-format.js";
|
||||
import { probeGatewayStatus } from "../cli/daemon-cli/probe.js";
|
||||
import { DEFAULT_RESTART_HEALTH_TIMEOUT_MS } from "../cli/daemon-cli/restart-health.constants.js";
|
||||
import {
|
||||
compareCliGatewayStateDirs,
|
||||
GATEWAY_SERVICE_PATHS_UNVERIFIED,
|
||||
|
|
@ -36,6 +37,7 @@ import type { RuntimeEnv } from "../runtime.js";
|
|||
import type { StatusSummary } from "../status/summary.js";
|
||||
import { VERSION } from "../version.js";
|
||||
import { projectDoctorSecretRuntimeDegradations } from "./doctor-secret-runtime-degradation.js";
|
||||
import { waitForGatewayDiagnostic } from "./gateway-diagnostic-readiness.js";
|
||||
import {
|
||||
GATEWAY_HEALTH_CREDENTIALS_REQUIRED_MESSAGE,
|
||||
GATEWAY_HEALTH_CREDENTIALS_REQUIRED_TITLE,
|
||||
|
|
@ -291,16 +293,25 @@ export async function checkGatewayHealth(params: {
|
|||
const { bindAgentToolGatewayRequest } = await import("../agents/tools/in-process-gateway.js");
|
||||
const requestGateway = bindAgentToolGatewayRequest({ hostedOnly: true });
|
||||
const timeoutMs =
|
||||
typeof params.timeoutMs === "number" && params.timeoutMs > 0 ? params.timeoutMs : 10_000;
|
||||
typeof params.timeoutMs === "number" && params.timeoutMs > 0
|
||||
? params.timeoutMs
|
||||
: DEFAULT_RESTART_HEALTH_TIMEOUT_MS;
|
||||
let healthOk = false;
|
||||
let status: StatusSummary | undefined;
|
||||
let gatewaySnapshot: GatewayHello["snapshot"] | undefined;
|
||||
try {
|
||||
const remainingMs = await waitForGatewayDiagnostic(
|
||||
{ config: params.cfg, timeoutMs },
|
||||
params.runtime,
|
||||
);
|
||||
if (remainingMs === undefined) {
|
||||
return { healthOk: true, authenticated: false };
|
||||
}
|
||||
const statusStartedAt = performance.now();
|
||||
status = await callGateway<StatusSummary>({
|
||||
method: "status",
|
||||
params: { includeChannelSummary: false },
|
||||
timeoutMs,
|
||||
timeoutMs: remainingMs,
|
||||
config: params.cfg,
|
||||
onHelloOk: ({ snapshot }: GatewayHello) => {
|
||||
gatewaySnapshot = snapshot;
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { useAutoCleanupTempDirTracker } from "../../test/helpers/temp-dir.js";
|
||||
import type { OpenClawConfig } from "../config/config.js";
|
||||
import * as gatewayCall from "../gateway/call.js";
|
||||
import {
|
||||
completeGatewayBootLifecycle,
|
||||
recordGatewayBootStart,
|
||||
|
|
@ -14,12 +15,6 @@ const mocks = vi.hoisted(() => ({
|
|||
hasActiveGatewayExecCredential: vi.fn(),
|
||||
note: vi.fn(),
|
||||
}));
|
||||
vi.mock("../gateway/call.js", () => ({
|
||||
callGateway: mocks.callGateway,
|
||||
buildGatewayConnectionDetails: () => ({ url: "ws://127.0.0.1:18789" }),
|
||||
buildGatewayProbeConnectionDetails: mocks.buildGatewayProbeConnectionDetails,
|
||||
isGatewayCredentialsRequiredError: () => false,
|
||||
}));
|
||||
vi.mock("../flows/doctor-gateway-exec-credential.js", () => ({
|
||||
hasActiveGatewayExecCredential: mocks.hasActiveGatewayExecCredential,
|
||||
}));
|
||||
|
|
@ -31,6 +26,7 @@ vi.mock("../../packages/terminal-core/src/note.js", () => ({ note: mocks.note })
|
|||
const tempDirs = useAutoCleanupTempDirTracker((cleanup) => {
|
||||
afterEach(() => {
|
||||
closeOpenClawStateDatabaseForTest();
|
||||
vi.restoreAllMocks();
|
||||
vi.unstubAllEnvs();
|
||||
cleanup();
|
||||
});
|
||||
|
|
@ -44,6 +40,10 @@ beforeEach(() => {
|
|||
mocks.buildGatewayProbeConnectionDetails
|
||||
.mockReset()
|
||||
.mockResolvedValue({ url: "ws://127.0.0.1:18789" });
|
||||
vi.spyOn(gatewayCall, "callGateway").mockImplementation(mocks.callGateway);
|
||||
vi.spyOn(gatewayCall, "buildGatewayProbeConnectionDetails").mockImplementation(
|
||||
mocks.buildGatewayProbeConnectionDetails,
|
||||
);
|
||||
mocks.hasActiveGatewayExecCredential.mockReset().mockResolvedValue(false);
|
||||
mocks.note.mockReset();
|
||||
const bootId = recordGatewayBootStart(process.env, completedAtMs - 1_000);
|
||||
|
|
@ -66,6 +66,7 @@ describe("Doctor installation replacement diagnostics", () => {
|
|||
const runtime = { log: vi.fn(), error: vi.fn(), exit: vi.fn() };
|
||||
const result = await checkGatewayHealth({ cfg: {}, runtime });
|
||||
|
||||
expect(mocks.callGateway).toHaveBeenCalledWith(expect.objectContaining({ method: "status" }));
|
||||
expect(result.healthOk).toBe(reachable);
|
||||
expect(mocks.note).toHaveBeenCalledWith(
|
||||
`Previous installation replacement (${new Date(completedAtMs).toISOString()}): ${reason}`,
|
||||
|
|
|
|||
|
|
@ -1,6 +1,8 @@
|
|||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { DEFAULT_RESTART_HEALTH_TIMEOUT_MS } from "../cli/daemon-cli/restart-health.constants.js";
|
||||
import { createCoreHealthChecks } from "../flows/doctor-core-checks.js";
|
||||
import { resolveFinalDoctorHealthContributions } from "../flows/doctor-health-contributions-final.js";
|
||||
import * as gatewayCall from "../gateway/call.js";
|
||||
import { resolveLeastPrivilegeOperatorScopesForMethod } from "../gateway/method-scopes.js";
|
||||
import { createGatewayMethodRegistry } from "../gateway/methods/registry.js";
|
||||
import type {
|
||||
|
|
@ -23,7 +25,6 @@ const {
|
|||
collectClawStateHealthFindings: vi.fn(),
|
||||
collectWhatsappResponsivenessHealthFindings: vi.fn(),
|
||||
}));
|
||||
vi.mock("../gateway/call.js", () => ({ callGateway: socketCall }));
|
||||
vi.mock("../../packages/terminal-core/src/note.js", () => ({ note }));
|
||||
vi.mock("../claws/doctor.js", () => ({ collectClawStateHealthFindings }));
|
||||
vi.mock("./doctor-whatsapp-responsiveness.js", () => ({
|
||||
|
|
@ -53,13 +54,19 @@ function contextFor(handlers: GatewayRequestHandlers): GatewayRequestContext {
|
|||
describe("Doctor hosted Gateway reads", () => {
|
||||
beforeEach(() => {
|
||||
socketCall.mockReset().mockRejectedValue(new Error("unexpected Gateway socket"));
|
||||
// Hosted routing and readiness stay real; only socket requests are replaced.
|
||||
vi.spyOn(gatewayCall, "callGateway").mockImplementation(socketCall);
|
||||
note.mockReset();
|
||||
collectClawStateHealthFindings.mockReset();
|
||||
collectWhatsappResponsivenessHealthFindings.mockReset().mockReturnValue([]);
|
||||
});
|
||||
afterEach(() => vi.unstubAllEnvs());
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
vi.unstubAllEnvs();
|
||||
});
|
||||
|
||||
it("reads exporter diagnostics locally while retaining the wire health probes", async () => {
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
socketCall.mockImplementation(async ({ method }) => {
|
||||
if (method === "status" || method === "channels.status") {
|
||||
return {};
|
||||
|
|
@ -80,10 +87,12 @@ describe("Doctor hosted Gateway reads", () => {
|
|||
],
|
||||
}),
|
||||
});
|
||||
await withPluginRuntimeGatewayContextResolver(
|
||||
() => context,
|
||||
() => checkGatewayHealth({ runtime, cfg: {} }),
|
||||
);
|
||||
await expect(
|
||||
withPluginRuntimeGatewayContextResolver(
|
||||
() => context,
|
||||
() => checkGatewayHealth({ runtime, cfg: {} }),
|
||||
),
|
||||
).resolves.toMatchObject({ healthOk: true, authenticated: true });
|
||||
expect(note).toHaveBeenCalledWith(
|
||||
"diagnostics-otel · logs · started · stdout",
|
||||
"Telemetry exporters",
|
||||
|
|
@ -92,6 +101,10 @@ describe("Doctor hosted Gateway reads", () => {
|
|||
"status",
|
||||
"channels.status",
|
||||
]);
|
||||
expect(socketCall).toHaveBeenNthCalledWith(
|
||||
1,
|
||||
expect.objectContaining({ timeoutMs: DEFAULT_RESTART_HEALTH_TIMEOUT_MS }),
|
||||
);
|
||||
});
|
||||
|
||||
it("reads cached memory readiness without opening a Gateway socket", async () => {
|
||||
|
|
|
|||
51
src/commands/gateway-diagnostic-readiness.ts
Normal file
51
src/commands/gateway-diagnostic-readiness.ts
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
import { sanitizeTerminalText } from "../../packages/terminal-core/src/safe-text.js";
|
||||
import { waitForGatewayDiagnosticReadiness } from "../cli/daemon-cli/diagnostic-readiness.js";
|
||||
import { DEFAULT_RESTART_HEALTH_TIMEOUT_MS } from "../cli/daemon-cli/restart-health.constants.js";
|
||||
import { buildGatewayProbeConnectionDetails } from "../gateway/call.js";
|
||||
import { GatewayTransportError } from "../gateway/transport-error.js";
|
||||
import { type RuntimeEnv, writeRuntimeJson } from "../runtime.js";
|
||||
|
||||
/** Keep startup progress separate from terminal diagnostic failure. */
|
||||
export async function waitForGatewayDiagnostic(
|
||||
opts: Parameters<typeof waitForGatewayDiagnosticReadiness>[0] & { json?: boolean },
|
||||
runtime: RuntimeEnv,
|
||||
): Promise<number | undefined> {
|
||||
const timeoutMs = opts.timeoutMs ?? DEFAULT_RESTART_HEALTH_TIMEOUT_MS;
|
||||
const startedAtMs = performance.now();
|
||||
const deadlineMs = Math.min(opts.deadlineMs ?? Infinity, startedAtMs + timeoutMs);
|
||||
const readiness = await waitForGatewayDiagnosticReadiness({
|
||||
...opts,
|
||||
deadlineMs,
|
||||
onProgress: (phase) => {
|
||||
if (!opts.json) {
|
||||
runtime.log(`Gateway is still starting (phase: ${sanitizeTerminalText(phase)}).`);
|
||||
}
|
||||
},
|
||||
});
|
||||
// RPC timeout fields require integer milliseconds.
|
||||
const remainingMs = Math.max(0, Math.ceil(deadlineMs - performance.now()));
|
||||
if (
|
||||
remainingMs > 0 &&
|
||||
(!readiness ||
|
||||
readiness.healthy ||
|
||||
readiness.waitOutcome === "channel-errors" ||
|
||||
readiness.waitOutcome === "plugin-errors")
|
||||
) {
|
||||
return remainingMs;
|
||||
}
|
||||
if (readiness?.waitOutcome === "still-starting") {
|
||||
if (opts.json) {
|
||||
writeRuntimeJson(runtime, {
|
||||
status: "starting",
|
||||
startupPhase: readiness.startupPhase,
|
||||
});
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
throw new GatewayTransportError({
|
||||
kind: "timeout",
|
||||
timeoutMs,
|
||||
connectionDetails: await buildGatewayProbeConnectionDetails(opts),
|
||||
message: `${!readiness || readiness.healthy ? "Gateway diagnostic budget exhausted" : "Gateway not reachable"} after waiting ${Math.round((performance.now() - startedAtMs) / 1000)} s.\n${sanitizeTerminalText(readiness?.probeError ?? readiness?.startupPhase ?? "Readiness could not be confirmed.")}\nRun openclaw gateway status --deep to diagnose.`,
|
||||
});
|
||||
}
|
||||
|
|
@ -1,8 +1,13 @@
|
|||
// Health command tests cover gateway health probes, JSON output, and status formatting.
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
vi.mock("../cli/daemon-cli/diagnostic-readiness.js", () => ({
|
||||
waitForGatewayDiagnosticReadiness: vi.fn(async () => undefined),
|
||||
}));
|
||||
import { GatewayClientRequestError } from "../../packages/gateway-client/src/index.js";
|
||||
import { retainGatewayResponsePayload } from "../../packages/gateway-client/src/protocol-request.js";
|
||||
import { stripAnsi } from "../../packages/terminal-core/src/ansi.js";
|
||||
import { waitForGatewayDiagnosticReadiness } from "../cli/daemon-cli/diagnostic-readiness.js";
|
||||
import { replaceSessionEntry } from "../config/sessions/session-accessor.js";
|
||||
import { ExitError } from "../runtime.js";
|
||||
import { withOpenClawTestState } from "../test-utils/openclaw-test-state.js";
|
||||
|
|
@ -147,6 +152,7 @@ function requireFirstGatewayRequest(): Record<string, unknown> {
|
|||
|
||||
describe("healthCommand", () => {
|
||||
beforeEach(() => {
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
vi.clearAllMocks();
|
||||
buildGatewayConnectionDetailsMock.mockReturnValue({
|
||||
message: TEST_GATEWAY_MESSAGE,
|
||||
|
|
@ -815,6 +821,69 @@ describe("healthCommand", () => {
|
|||
expect(probeGatewayStatusMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ elapsedMs: 4000, skipReadiness: false },
|
||||
{ elapsedMs: 5000, skipReadiness: false },
|
||||
{ elapsedMs: 4000, skipReadiness: true },
|
||||
{ elapsedMs: 5000, skipReadiness: true },
|
||||
])(
|
||||
"charges target/auth preparation and readiness to one budget ($elapsedMs, $skipReadiness)",
|
||||
async ({ elapsedMs, skipReadiness }) => {
|
||||
vi.mocked(waitForGatewayDiagnosticReadiness).mockImplementationOnce(async () => {
|
||||
vi.spyOn(performance, "now").mockReturnValue(elapsedMs);
|
||||
return skipReadiness
|
||||
? undefined
|
||||
: {
|
||||
healthy: true,
|
||||
waitOutcome: "healthy",
|
||||
elapsedMs: 1000,
|
||||
runtime: { status: "running", pid: 42 },
|
||||
portUsage: { port: 18789, status: "busy", listeners: [{ pid: 42 }], hints: [] },
|
||||
staleGatewayPids: [],
|
||||
};
|
||||
});
|
||||
if (elapsedMs === 5000) {
|
||||
await expect(healthCommand({ timeoutMs: 5000, config: {} }, runtime)).rejects.toThrow(
|
||||
"Gateway diagnostic budget exhausted",
|
||||
);
|
||||
expect(callGatewayMock).not.toHaveBeenCalled();
|
||||
} else {
|
||||
callGatewayMock.mockResolvedValueOnce(createHealthSummary());
|
||||
await healthCommand({ timeoutMs: 5000, config: {} }, runtime);
|
||||
expect(callGatewayMock).toHaveBeenCalledWith(expect.objectContaining({ timeoutMs: 1000 }));
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("keeps readiness exhaustion machine-readable without a second network probe", async () => {
|
||||
vi.mocked(waitForGatewayDiagnosticReadiness).mockResolvedValueOnce({
|
||||
healthy: false,
|
||||
waitOutcome: "timeout",
|
||||
elapsedMs: 5000,
|
||||
probeError: "connect ECONNREFUSED",
|
||||
runtime: { status: "stopped" },
|
||||
portUsage: { port: 18789, status: "free", listeners: [], hints: [] },
|
||||
staleGatewayPids: [],
|
||||
});
|
||||
const { formatGatewayTransportErrorJson } =
|
||||
await vi.importActual<typeof import("../gateway/call.js")>("../gateway/call.js");
|
||||
formatGatewayTransportErrorJsonMock.mockImplementation(formatGatewayTransportErrorJson);
|
||||
|
||||
await healthCommand({ json: true, timeoutMs: 5000, config: {} }, runtime);
|
||||
|
||||
expect(JSON.parse(requireFirstRuntimeLog())).toMatchObject({
|
||||
ok: false,
|
||||
error: { type: "gateway_transport_error", kind: "timeout", timeoutMs: 5000 },
|
||||
gateway: { url: TEST_GATEWAY_URL },
|
||||
});
|
||||
expect(runtime.exit).toHaveBeenCalledWith(1);
|
||||
expect(callGatewayMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("keeps credential failures machine-readable when the gateway is unreachable", async () => {
|
||||
const error = new Error("gateway health requires credentials");
|
||||
const payload = {
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import { isRich } from "../../packages/terminal-core/src/theme.js";
|
|||
import { resolveChannelDefaultAccountId } from "../channels/plugins/helpers.js";
|
||||
import { listReadOnlyChannelPluginsForConfig } from "../channels/plugins/read-only.js";
|
||||
import { probeGatewayStatus } from "../cli/daemon-cli/probe.js";
|
||||
import { DEFAULT_RESTART_HEALTH_TIMEOUT_MS } from "../cli/daemon-cli/restart-health.constants.js";
|
||||
import { withProgress } from "../cli/progress.js";
|
||||
import type { OpenClawConfig } from "../config/types.openclaw.js";
|
||||
import {
|
||||
|
|
@ -27,6 +28,7 @@ import { formatDurationCompact } from "../infra/format-time/format-duration.js";
|
|||
import { createSubsystemLogger } from "../logging/subsystem.js";
|
||||
import { buildChannelAccountBindings, resolvePreferredAccountId } from "../routing/bindings.js";
|
||||
import { ExitError, type RuntimeEnv, writeRuntimeJson } from "../runtime.js";
|
||||
import { waitForGatewayDiagnostic } from "./gateway-diagnostic-readiness.js";
|
||||
import {
|
||||
buildCredentialsRequiredHealthDiagnostic,
|
||||
buildRateLimitedHealthDiagnostic,
|
||||
|
|
@ -40,7 +42,6 @@ import { logGatewayConnectionDetails } from "./status.gateway-connection.js";
|
|||
export { formatHealthChannelLines } from "./health-format.js";
|
||||
export type { HealthSummary } from "../gateway/health/types.js";
|
||||
|
||||
const DEFAULT_TIMEOUT_MS = 10_000;
|
||||
const healthLog = createSubsystemLogger("health");
|
||||
|
||||
const debugHealth = (
|
||||
|
|
@ -96,7 +97,7 @@ export async function emitReachableGatewayAuthDiagnostic(params: {
|
|||
password: params.password,
|
||||
tlsFingerprint: details.tlsFingerprint,
|
||||
preauthHandshakeTimeoutMs: details.preauthHandshakeTimeoutMs,
|
||||
timeoutMs: params.timeoutMs ?? DEFAULT_TIMEOUT_MS,
|
||||
timeoutMs: params.timeoutMs ?? DEFAULT_RESTART_HEALTH_TIMEOUT_MS,
|
||||
config: params.config,
|
||||
json: params.json,
|
||||
});
|
||||
|
|
@ -173,6 +174,10 @@ export async function healthCommand(
|
|||
// Always query the running gateway; do not open a direct Baileys socket here.
|
||||
let summary: HealthSummary;
|
||||
try {
|
||||
const remainingMs = await waitForGatewayDiagnostic({ ...opts, config: cfg }, runtime);
|
||||
if (remainingMs === undefined) {
|
||||
return;
|
||||
}
|
||||
summary = await withProgress(
|
||||
{
|
||||
label: "Checking gateway health…",
|
||||
|
|
@ -183,7 +188,7 @@ export async function healthCommand(
|
|||
await callGateway<HealthSummary>({
|
||||
method: "health",
|
||||
params: opts.verbose ? { probe: true } : undefined,
|
||||
timeoutMs: opts.timeoutMs,
|
||||
timeoutMs: remainingMs,
|
||||
config: cfg,
|
||||
token: opts.token,
|
||||
password: opts.password,
|
||||
|
|
|
|||
|
|
@ -10,20 +10,25 @@ import {
|
|||
resolveStatusServiceSummaries,
|
||||
resolveStatusUsageSummary,
|
||||
} from "./status-runtime-shared.ts";
|
||||
import {
|
||||
resolveStatusGatewayProbeTimeoutMs,
|
||||
type StatusGatewayProbeBudget,
|
||||
} from "./status.gateway-probe-budget.js";
|
||||
import { resolveNodeOnlyGatewayInfo } from "./status.node-mode.js";
|
||||
import { collectStatusScanOverview } from "./status.scan-overview.ts";
|
||||
|
||||
/** Runs the full read-only status report and writes it to the runtime logger. */
|
||||
export async function statusAllCommand(
|
||||
runtime: RuntimeEnv,
|
||||
opts?: { timeoutMs?: number; usage?: boolean; agent?: string },
|
||||
opts: StatusGatewayProbeBudget & { usage?: boolean; agent?: string },
|
||||
): Promise<void> {
|
||||
await withProgress({ label: "Scanning status --all…", total: 11 }, async (progress) => {
|
||||
const overview = await collectStatusScanOverview({
|
||||
env: process.env,
|
||||
commandName: "status --all",
|
||||
opts: {
|
||||
timeoutMs: opts?.timeoutMs,
|
||||
timeoutMs: opts.timeoutMs,
|
||||
gatewayProbeDeadlineMs: opts.gatewayProbeDeadlineMs,
|
||||
},
|
||||
showSecrets: false,
|
||||
runtime,
|
||||
|
|
@ -42,7 +47,7 @@ export async function statusAllCommand(
|
|||
},
|
||||
});
|
||||
progress.setLabel("Checking services…");
|
||||
const [daemon, nodeService] = await resolveStatusServiceSummaries(opts?.timeoutMs);
|
||||
const [daemon, nodeService] = await resolveStatusServiceSummaries(opts.timeoutMs);
|
||||
const nodeOnlyGateway = await resolveNodeOnlyGatewayInfo({
|
||||
daemon,
|
||||
node: nodeService,
|
||||
|
|
@ -56,14 +61,16 @@ export async function statusAllCommand(
|
|||
nodeService,
|
||||
nodeOnlyGateway,
|
||||
progress,
|
||||
timeoutMs: opts?.timeoutMs,
|
||||
timeoutMs: opts.timeoutMs,
|
||||
gatewayProbeDeadlineMs: opts.gatewayProbeDeadlineMs,
|
||||
})),
|
||||
});
|
||||
|
||||
if (opts?.usage) {
|
||||
if (opts.usage) {
|
||||
const usage = await resolveStatusUsageSummary({
|
||||
config: overview.cfg,
|
||||
timeoutMs: opts.timeoutMs,
|
||||
timeoutMs: resolveStatusGatewayProbeTimeoutMs(opts),
|
||||
gatewayProbeDeadlineMs: opts.gatewayProbeDeadlineMs,
|
||||
...(opts.agent ? { agentId: opts.agent } : {}),
|
||||
});
|
||||
lines.push("", ...formatUsageReportLines(usage));
|
||||
|
|
|
|||
|
|
@ -610,6 +610,17 @@ describe("status-all diagnosis port checks", () => {
|
|||
expect(output).not.toContain("Retry: openclaw gateway stability");
|
||||
});
|
||||
|
||||
it("preserves startup phase in channel diagnosis", async () => {
|
||||
const params = createBaseParams([]);
|
||||
params.gatewayStartupPhase = "plugins";
|
||||
await appendStatusAllDiagnosis(params);
|
||||
|
||||
const output = params.lines.join("\n");
|
||||
expect(output).toContain("Channel issues skipped (gateway still starting (phase plugins))");
|
||||
expect(output).not.toContain("gateway unreachable");
|
||||
expect(output).not.toContain("Gateway health:");
|
||||
});
|
||||
|
||||
it("does not read or display stale stderr tails on Darwin", async () => {
|
||||
const originalPlatform = process.platform;
|
||||
Object.defineProperty(process, "platform", { value: "darwin" });
|
||||
|
|
|
|||
|
|
@ -158,6 +158,7 @@ export async function appendStatusAllDiagnosis(params: {
|
|||
exporterDiagnostics: StatusGatewayDiagnosticsResult | null;
|
||||
agentStatus?: AgentStatusLike;
|
||||
gatewayReachable: boolean;
|
||||
gatewayStartupPhase?: string;
|
||||
health: Awaited<ReturnType<typeof resolveStatusGatewayHealthSafe>> | null | undefined;
|
||||
nodeOnlyGateway: NodeOnlyGatewayInfo | null;
|
||||
}) {
|
||||
|
|
@ -468,6 +469,11 @@ export async function appendStatusAllDiagnosis(params: {
|
|||
`Channel issues skipped (node-only mode; query ${params.nodeOnlyGateway.gatewayTarget})`,
|
||||
"ok",
|
||||
);
|
||||
} else if (params.gatewayStartupPhase) {
|
||||
emitCheck(
|
||||
`Channel issues skipped (gateway still starting (phase ${params.gatewayStartupPhase}))`,
|
||||
"ok",
|
||||
);
|
||||
} else {
|
||||
emitCheck(
|
||||
`Channel issues skipped (gateway ${params.gatewayReachable ? "query failed" : "unreachable"})`,
|
||||
|
|
|
|||
|
|
@ -283,6 +283,27 @@ describe("status-all format", () => {
|
|||
expect(output).not.toContain("signed");
|
||||
});
|
||||
|
||||
it("reports startup phase without an unreachable error in text and JSON", () => {
|
||||
const params = {
|
||||
gatewayMode: "local" as const,
|
||||
gatewayConnection: { url: "ws://127.0.0.1:18789" },
|
||||
remoteUrlMissing: false,
|
||||
gatewayReachable: false,
|
||||
gatewayProbe: { startupPhase: "plugins", error: null },
|
||||
gatewayProbeAuth: null,
|
||||
gatewaySelf: null,
|
||||
};
|
||||
expect(getStatusOverviewRowValue("Gateway", params)).toContain(
|
||||
"still starting (phase plugins)",
|
||||
);
|
||||
expect(getStatusOverviewRowValue("Gateway", params)).not.toContain("unreachable");
|
||||
expect(buildGatewayStatusJsonPayload(params)).toMatchObject({
|
||||
readiness: "still-starting",
|
||||
startupPhase: "plugins",
|
||||
error: null,
|
||||
});
|
||||
});
|
||||
|
||||
it("builds shared gateway surface values for node and gateway views", () => {
|
||||
expect(
|
||||
buildStatusOverviewSurfaceRows({
|
||||
|
|
|
|||
|
|
@ -32,6 +32,7 @@ type StatusGatewayConnection = {
|
|||
type StatusGatewayProbe = {
|
||||
connectLatencyMs?: number | null;
|
||||
error?: string | null;
|
||||
startupPhase?: string;
|
||||
} | null;
|
||||
|
||||
type StatusGatewayProbeAuth = {
|
||||
|
|
@ -370,11 +371,13 @@ function buildGatewayStatusSummaryParts(params: {
|
|||
: targetText;
|
||||
const reachText = params.remoteUrlMissing
|
||||
? "misconfigured (remote.url missing)"
|
||||
: params.gatewayReachable
|
||||
? `reachable ${formatDurationPrecise(params.gatewayProbe?.connectLatencyMs ?? 0)}`
|
||||
: params.gatewayProbe?.error
|
||||
? `unreachable (${params.gatewayProbe.error})`
|
||||
: "unreachable";
|
||||
: params.gatewayProbe?.startupPhase
|
||||
? `still starting (phase ${params.gatewayProbe.startupPhase})`
|
||||
: params.gatewayReachable
|
||||
? `reachable ${formatDurationPrecise(params.gatewayProbe?.connectLatencyMs ?? 0)}`
|
||||
: params.gatewayProbe?.error
|
||||
? `unreachable (${params.gatewayProbe.error})`
|
||||
: "unreachable";
|
||||
const authText = params.gatewayReachable
|
||||
? `auth ${formatGatewayAuthUsed(params.gatewayProbeAuth)}`
|
||||
: "";
|
||||
|
|
@ -445,6 +448,7 @@ export function buildGatewayStatusJsonPayload(params: {
|
|||
connectLatencyMs?: number | null;
|
||||
error?: string | null;
|
||||
health?: unknown;
|
||||
startupPhase?: string;
|
||||
}
|
||||
| null
|
||||
| undefined;
|
||||
|
|
@ -457,6 +461,9 @@ export function buildGatewayStatusJsonPayload(params: {
|
|||
urlSource: params.gatewayConnection.urlSource,
|
||||
misconfigured: params.remoteUrlMissing,
|
||||
reachable: params.gatewayReachable,
|
||||
...(params.gatewayProbe?.startupPhase
|
||||
? { readiness: "still-starting", startupPhase: params.gatewayProbe.startupPhase }
|
||||
: {}),
|
||||
connectLatencyMs: params.gatewayProbe?.connectLatencyMs ?? null,
|
||||
self: params.gatewaySelf ?? null,
|
||||
error: params.gatewayProbe?.error ?? null,
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import type { RestartSentinelPayload } from "../../infra/restart-sentinel.js";
|
|||
import type { UpdateRunRecord } from "../../infra/update-run-record.js";
|
||||
import { writeSkill } from "../../skills/test-support/e2e-test-helpers.js";
|
||||
import { withEnvAsync } from "../../test-utils/env.js";
|
||||
import { createStatusGatewayProbeBudget } from "../status.gateway-probe-budget.js";
|
||||
import { baseStatusGatewaySnapshot, baseStatusOverviewSurface } from "../status.test-support.ts";
|
||||
|
||||
const mocks = vi.hoisted(() => ({
|
||||
|
|
@ -108,6 +109,7 @@ describe("buildStatusAllReportData", () => {
|
|||
eligible: 0,
|
||||
missing: 0,
|
||||
});
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
mocks.listUpdateRuns.mockReturnValue([]);
|
||||
mocks.findActiveUpdateRun.mockReturnValue(undefined);
|
||||
mocks.getUpdateRun.mockReturnValue(undefined);
|
||||
|
|
@ -116,6 +118,10 @@ describe("buildStatusAllReportData", () => {
|
|||
mocks.resolveStatusGatewayHealthSafe.mockResolvedValue(undefined);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it.each([
|
||||
"completed",
|
||||
"active",
|
||||
|
|
@ -197,6 +203,7 @@ describe("buildStatusAllReportData", () => {
|
|||
});
|
||||
}
|
||||
const report = await buildStatusAllReportData({
|
||||
...createStatusGatewayProbeBudget(),
|
||||
overview: {
|
||||
...baseStatusOverviewSurface,
|
||||
cfg: {},
|
||||
|
|
@ -264,47 +271,61 @@ describe("buildStatusAllReportData", () => {
|
|||
},
|
||||
);
|
||||
|
||||
it("collects delivery and exporter stability projections in parallel", async () => {
|
||||
await buildStatusAllReportData({
|
||||
overview: {
|
||||
cfg: {},
|
||||
gatewaySnapshot: {
|
||||
gatewayReachable: true,
|
||||
gatewayProbe: { error: null },
|
||||
gatewayCallOverrides: undefined,
|
||||
gatewayConnection: {},
|
||||
remoteUrlMissing: false,
|
||||
},
|
||||
secretDiagnostics: [],
|
||||
tailscaleMode: "off",
|
||||
tailscaleDns: null,
|
||||
agentStatus: { agents: [], defaultId: null },
|
||||
channels: { rows: [], details: [] },
|
||||
channelIssues: [],
|
||||
runtimeDegradation: { degradedSecretOwners: [], degradedPlugins: [] },
|
||||
osSummary: { label: "test" },
|
||||
} as never,
|
||||
daemon: {} as never,
|
||||
nodeService: {} as never,
|
||||
nodeOnlyGateway: null,
|
||||
progress: { setLabel: vi.fn(), tick: vi.fn() },
|
||||
});
|
||||
it.each([false, true])(
|
||||
"collects stability projections only after readiness (starting: %s)",
|
||||
async (starting) => {
|
||||
const report = await buildStatusAllReportData({
|
||||
...createStatusGatewayProbeBudget(),
|
||||
overview: {
|
||||
cfg: {},
|
||||
gatewaySnapshot: {
|
||||
gatewayReachable: !starting,
|
||||
gatewayProbe: { error: null, ...(starting ? { startupPhase: "plugins" } : {}) },
|
||||
gatewayCallOverrides: undefined,
|
||||
gatewayConnection: {},
|
||||
remoteUrlMissing: false,
|
||||
},
|
||||
secretDiagnostics: [],
|
||||
tailscaleMode: "off",
|
||||
tailscaleDns: null,
|
||||
agentStatus: { agents: [], defaultId: null },
|
||||
channels: { rows: [], details: [] },
|
||||
channelIssues: [],
|
||||
runtimeDegradation: { degradedSecretOwners: [], degradedPlugins: [] },
|
||||
osSummary: { label: "test" },
|
||||
} as never,
|
||||
daemon: {} as never,
|
||||
nodeService: {} as never,
|
||||
nodeOnlyGateway: null,
|
||||
progress: { setLabel: vi.fn(), tick: vi.fn() },
|
||||
});
|
||||
|
||||
expect(mocks.resolveStatusGatewayDiagnosticsSafe.mock.calls).toEqual([
|
||||
[
|
||||
expect.objectContaining({
|
||||
gatewayReachable: true,
|
||||
}),
|
||||
],
|
||||
[
|
||||
expect.objectContaining({
|
||||
gatewayReachable: true,
|
||||
type: "telemetry.exporter",
|
||||
}),
|
||||
],
|
||||
]);
|
||||
expect(mocks.resolveStatusSummaryFromOverview).not.toHaveBeenCalled();
|
||||
});
|
||||
if (starting) {
|
||||
expect(mocks.resolveStatusGatewayDiagnosticsSafe).not.toHaveBeenCalled();
|
||||
expect(mocks.resolveStatusGatewayHealthSafe).not.toHaveBeenCalled();
|
||||
expect(report.diagnosis.gatewayStartupPhase).toBe("plugins");
|
||||
expect(report.diagnosis.health).toBeUndefined();
|
||||
return;
|
||||
}
|
||||
|
||||
expect(mocks.resolveStatusGatewayDiagnosticsSafe.mock.calls).toEqual([
|
||||
[
|
||||
expect.objectContaining({
|
||||
gatewayReachable: true,
|
||||
gatewayProbeDeadlineMs: 60_000,
|
||||
}),
|
||||
],
|
||||
[
|
||||
expect.objectContaining({
|
||||
gatewayReachable: true,
|
||||
gatewayProbeDeadlineMs: 60_000,
|
||||
type: "telemetry.exporter",
|
||||
}),
|
||||
],
|
||||
]);
|
||||
expect(mocks.resolveStatusSummaryFromOverview).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
|
||||
it("renders the system agent's unfiltered readiness and refreshes newly installed binaries", async () => {
|
||||
const rootDir = tempDirs.make("openclaw-status-readiness-");
|
||||
|
|
@ -342,6 +363,7 @@ describe("buildStatusAllReportData", () => {
|
|||
);
|
||||
mocks.buildWorkspaceSkillReadiness.mockImplementation(actual.buildWorkspaceSkillReadiness);
|
||||
const params = {
|
||||
...createStatusGatewayProbeBudget(),
|
||||
overview: {
|
||||
cfg: {
|
||||
plugins: { enabled: false },
|
||||
|
|
@ -442,6 +464,7 @@ describe("buildStatusAllReportData", () => {
|
|||
|
||||
it("does not inspect the first workspace when an explicit fleet has no owner", async () => {
|
||||
await buildStatusAllReportData({
|
||||
...createStatusGatewayProbeBudget(),
|
||||
overview: {
|
||||
cfg: {
|
||||
agents: {
|
||||
|
|
|
|||
|
|
@ -59,6 +59,7 @@ async function resolveStatusAllLocalDiagnosis(params: {
|
|||
gatewayCallOverrides: StatusScanOverviewResult["gatewaySnapshot"]["gatewayCallOverrides"];
|
||||
nodeOnlyGateway: NodeOnlyGatewayInfo | null;
|
||||
timeoutMs?: number;
|
||||
gatewayProbeDeadlineMs: number;
|
||||
}): Promise<{
|
||||
configPath: string;
|
||||
health: StatusGatewayHealthSafe | undefined;
|
||||
|
|
@ -79,6 +80,7 @@ async function resolveStatusAllLocalDiagnosis(params: {
|
|||
channelIssues: StatusScanOverviewResult["channelIssues"];
|
||||
agentStatus: StatusScanOverviewResult["agentStatus"];
|
||||
gatewayReachable: boolean;
|
||||
gatewayStartupPhase?: string;
|
||||
health: StatusGatewayHealthSafe | undefined;
|
||||
deliveryDiagnostics: StatusGatewayDiagnosticsResult | null;
|
||||
exporterDiagnostics: StatusGatewayDiagnosticsResult | null;
|
||||
|
|
@ -90,27 +92,30 @@ async function resolveStatusAllLocalDiagnosis(params: {
|
|||
const configPath = resolveStatusAllConfigPath(snap?.path);
|
||||
const diagnosticsParams = {
|
||||
config: overview.cfg,
|
||||
gatewayProbeDeadlineMs: params.gatewayProbeDeadlineMs,
|
||||
timeoutMs: Math.min(5000, params.timeoutMs ?? 10_000),
|
||||
gatewayReachable: params.gatewayReachable,
|
||||
...(params.gatewayCallOverrides ? { callOverrides: params.gatewayCallOverrides } : {}),
|
||||
};
|
||||
|
||||
const [health, deliveryDiagnostics, exporterDiagnostics] = params.nodeOnlyGateway
|
||||
? [undefined, null, null]
|
||||
: await Promise.all([
|
||||
resolveStatusGatewayHealthSafe({
|
||||
config: overview.cfg,
|
||||
timeoutMs: Math.min(8000, params.timeoutMs ?? 10_000),
|
||||
gatewayReachable: params.gatewayReachable,
|
||||
gatewayProbeError: params.gatewayProbe?.error ?? null,
|
||||
...(params.gatewayCallOverrides ? { callOverrides: params.gatewayCallOverrides } : {}),
|
||||
}),
|
||||
resolveStatusGatewayDiagnosticsSafe(diagnosticsParams),
|
||||
resolveStatusGatewayDiagnosticsSafe({
|
||||
...diagnosticsParams,
|
||||
type: "telemetry.exporter",
|
||||
}),
|
||||
]);
|
||||
const [health, deliveryDiagnostics, exporterDiagnostics] =
|
||||
params.nodeOnlyGateway || params.gatewayProbe?.startupPhase
|
||||
? [undefined, null, null]
|
||||
: await Promise.all([
|
||||
resolveStatusGatewayHealthSafe({
|
||||
config: overview.cfg,
|
||||
gatewayProbeDeadlineMs: params.gatewayProbeDeadlineMs,
|
||||
timeoutMs: Math.min(8000, params.timeoutMs ?? 10_000),
|
||||
gatewayReachable: params.gatewayReachable,
|
||||
gatewayProbeError: params.gatewayProbe?.error ?? null,
|
||||
...(params.gatewayCallOverrides ? { callOverrides: params.gatewayCallOverrides } : {}),
|
||||
}),
|
||||
resolveStatusGatewayDiagnosticsSafe(diagnosticsParams),
|
||||
resolveStatusGatewayDiagnosticsSafe({
|
||||
...diagnosticsParams,
|
||||
type: "telemetry.exporter",
|
||||
}),
|
||||
]);
|
||||
|
||||
params.progress.setLabel("Checking local state…");
|
||||
// These probes are intentionally best-effort so status-all can still print a partial report.
|
||||
|
|
@ -181,6 +186,7 @@ async function resolveStatusAllLocalDiagnosis(params: {
|
|||
channelIssues: overview.channelIssues,
|
||||
agentStatus: overview.agentStatus,
|
||||
gatewayReachable: params.gatewayReachable,
|
||||
gatewayStartupPhase: params.gatewayProbe?.startupPhase,
|
||||
health,
|
||||
deliveryDiagnostics,
|
||||
exporterDiagnostics,
|
||||
|
|
@ -197,6 +203,7 @@ export async function buildStatusAllReportData(params: {
|
|||
nodeOnlyGateway: NodeOnlyGatewayInfo | null;
|
||||
progress: StatusAllProgress;
|
||||
timeoutMs?: number;
|
||||
gatewayProbeDeadlineMs: number;
|
||||
}) {
|
||||
const gatewaySnapshot = params.overview.gatewaySnapshot;
|
||||
const [{ configPath, health, diagnosis }, summary] = await Promise.all([
|
||||
|
|
@ -208,6 +215,7 @@ export async function buildStatusAllReportData(params: {
|
|||
gatewayCallOverrides: gatewaySnapshot.gatewayCallOverrides,
|
||||
nodeOnlyGateway: params.nodeOnlyGateway,
|
||||
timeoutMs: params.timeoutMs,
|
||||
gatewayProbeDeadlineMs: params.gatewayProbeDeadlineMs,
|
||||
}),
|
||||
params.overview.runtimeDegradation ??
|
||||
resolveStatusSummaryFromOverview({ overview: params.overview }),
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
// Status JSON command tests cover runtime invocation and structured status JSON output.
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { runStatusJsonCommand } from "./status-json-command.ts";
|
||||
import { createStatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
import { createStatusScanResultFixture } from "./status.test-support.ts";
|
||||
import { createTestRuntime } from "./test-runtime-config-helpers.js";
|
||||
|
||||
|
|
@ -20,6 +21,11 @@ vi.mock("./status-json-runtime.ts", () => ({
|
|||
describe("runStatusJsonCommand", () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it("shares the fast-json scan and output flow", async () => {
|
||||
|
|
@ -50,7 +56,13 @@ describe("runStatusJsonCommand", () => {
|
|||
const scanStatusJsonFast = vi.fn(async () => scan);
|
||||
|
||||
await runStatusJsonCommand({
|
||||
opts: { deep: true, usage: true, agent: "beta", timeoutMs: 1234, all: true },
|
||||
opts: {
|
||||
...createStatusGatewayProbeBudget(1234),
|
||||
deep: true,
|
||||
usage: true,
|
||||
agent: "beta",
|
||||
all: true,
|
||||
},
|
||||
runtime,
|
||||
scanStatusJsonFast,
|
||||
includeSecurityAudit: true,
|
||||
|
|
@ -58,10 +70,19 @@ describe("runStatusJsonCommand", () => {
|
|||
suppressHealthErrors: true,
|
||||
});
|
||||
|
||||
expect(scanStatusJsonFast).toHaveBeenCalledWith({ timeoutMs: 1234, all: true }, runtime);
|
||||
expect(scanStatusJsonFast).toHaveBeenCalledWith(
|
||||
{ timeoutMs: 1234, gatewayProbeDeadlineMs: 1234, all: true },
|
||||
runtime,
|
||||
);
|
||||
expect(mocks.resolveStatusJsonOutput).toHaveBeenCalledWith({
|
||||
scan,
|
||||
opts: { deep: true, usage: true, agent: "beta", timeoutMs: 1234, all: true },
|
||||
opts: {
|
||||
...createStatusGatewayProbeBudget(1234),
|
||||
deep: true,
|
||||
usage: true,
|
||||
agent: "beta",
|
||||
all: true,
|
||||
},
|
||||
includeSecurityAudit: true,
|
||||
includePluginCompatibility: true,
|
||||
suppressHealthErrors: true,
|
||||
|
|
@ -70,7 +91,13 @@ describe("runStatusJsonCommand", () => {
|
|||
built: true,
|
||||
input: {
|
||||
scan,
|
||||
opts: { deep: true, usage: true, agent: "beta", timeoutMs: 1234, all: true },
|
||||
opts: {
|
||||
...createStatusGatewayProbeBudget(1234),
|
||||
deep: true,
|
||||
usage: true,
|
||||
agent: "beta",
|
||||
all: true,
|
||||
},
|
||||
includeSecurityAudit: true,
|
||||
includePluginCompatibility: true,
|
||||
suppressHealthErrors: true,
|
||||
|
|
@ -84,7 +111,7 @@ describe("runStatusJsonCommand", () => {
|
|||
|
||||
await expect(
|
||||
runStatusJsonCommand({
|
||||
opts: { agent: "beta" },
|
||||
opts: { ...createStatusGatewayProbeBudget(), agent: "beta" },
|
||||
runtime,
|
||||
scanStatusJsonFast,
|
||||
includeSecurityAudit: false,
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@
|
|||
import { readUpdateRunStatus } from "../infra/update-run-status.js";
|
||||
import { type RuntimeEnv, writeRuntimeJson } from "../runtime.js";
|
||||
import { resolveStatusJsonOutput } from "./status-json-runtime.ts";
|
||||
import type { StatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
|
||||
type StatusJsonCommandOptions = {
|
||||
deep?: boolean;
|
||||
|
|
@ -22,19 +23,23 @@ export function assertStatusUsageAgentScope(opts: StatusJsonCommandOptions): voi
|
|||
|
||||
/** Runs the fast status scan, resolves optional deep fields, and writes JSON through the runtime. */
|
||||
export async function runStatusJsonCommand(params: {
|
||||
opts: StatusJsonCommandOptions;
|
||||
opts: StatusJsonCommandOptions & StatusGatewayProbeBudget;
|
||||
runtime: RuntimeEnv;
|
||||
includeSecurityAudit: boolean;
|
||||
includePluginCompatibility?: boolean;
|
||||
suppressHealthErrors?: boolean;
|
||||
scanStatusJsonFast: (
|
||||
opts: { timeoutMs?: number; all?: boolean },
|
||||
opts: StatusGatewayProbeBudget & { all?: boolean },
|
||||
runtime: RuntimeEnv,
|
||||
) => Promise<Parameters<typeof resolveStatusJsonOutput>[0]["scan"]>;
|
||||
}) {
|
||||
assertStatusUsageAgentScope(params.opts);
|
||||
const scan = await params.scanStatusJsonFast(
|
||||
{ timeoutMs: params.opts.timeoutMs, all: params.opts.all },
|
||||
{
|
||||
timeoutMs: params.opts.timeoutMs,
|
||||
gatewayProbeDeadlineMs: params.opts.gatewayProbeDeadlineMs,
|
||||
all: params.opts.all,
|
||||
},
|
||||
params.runtime,
|
||||
);
|
||||
const updateRunStatus = readUpdateRunStatus();
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
// Status JSON runtime tests cover runtime status payload construction and command dependencies.
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { resolveStatusJsonOutput } from "./status-json-runtime.ts";
|
||||
import { createStatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
import { createStatusScanResultFixture } from "./status.test-support.ts";
|
||||
|
||||
const mocks = vi.hoisted(() => ({
|
||||
|
|
@ -75,6 +76,7 @@ function requireStatusPayloadInput() {
|
|||
describe("status-json-runtime", () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
mocks.resolveStatusRuntimeSnapshot.mockResolvedValue({
|
||||
securityAudit: { summary: { critical: 1 } },
|
||||
usage: { providers: [] },
|
||||
|
|
@ -85,6 +87,10 @@ describe("status-json-runtime", () => {
|
|||
});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it("records requested local inspections as not collected for online JSON", async () => {
|
||||
const scan = createScan();
|
||||
scan.collection = {
|
||||
|
|
@ -93,7 +99,7 @@ describe("status-json-runtime", () => {
|
|||
};
|
||||
const result = await resolveStatusJsonOutput({
|
||||
scan,
|
||||
opts: {},
|
||||
opts: createStatusGatewayProbeBudget(),
|
||||
includeSecurityAudit: true,
|
||||
includePluginCompatibility: true,
|
||||
});
|
||||
|
|
@ -117,7 +123,7 @@ describe("status-json-runtime", () => {
|
|||
const scan = createScan();
|
||||
const result = await resolveStatusJsonOutput({
|
||||
scan,
|
||||
opts: { deep: true, usage: true, agent: "beta", timeoutMs: 1234 },
|
||||
opts: { ...createStatusGatewayProbeBudget(1234), deep: true, usage: true, agent: "beta" },
|
||||
includeSecurityAudit: true,
|
||||
includePluginCompatibility: true,
|
||||
});
|
||||
|
|
@ -126,6 +132,7 @@ describe("status-json-runtime", () => {
|
|||
config: { update: { channel: "stable" }, gateway: {} },
|
||||
sourceConfig: { gateway: {} },
|
||||
timeoutMs: 1234,
|
||||
gatewayProbeDeadlineMs: 1234,
|
||||
agentId: "beta",
|
||||
usage: true,
|
||||
deep: true,
|
||||
|
|
@ -177,7 +184,7 @@ describe("status-json-runtime", () => {
|
|||
const { env: _env, ...scanWithoutEnv } = createScan();
|
||||
await resolveStatusJsonOutput({
|
||||
scan: scanWithoutEnv,
|
||||
opts: { deep: false, usage: false, timeoutMs: 500 },
|
||||
opts: { ...createStatusGatewayProbeBudget(500), deep: false, usage: false },
|
||||
includeSecurityAudit: false,
|
||||
includePluginCompatibility: false,
|
||||
});
|
||||
|
|
@ -186,6 +193,7 @@ describe("status-json-runtime", () => {
|
|||
config: { update: { channel: "stable" }, gateway: {} },
|
||||
sourceConfig: { gateway: {} },
|
||||
timeoutMs: 500,
|
||||
gatewayProbeDeadlineMs: 500,
|
||||
usage: false,
|
||||
deep: false,
|
||||
gatewayReachable: true,
|
||||
|
|
@ -215,7 +223,7 @@ describe("status-json-runtime", () => {
|
|||
|
||||
await resolveStatusJsonOutput({
|
||||
scan: createScan(),
|
||||
opts: { deep: true, timeoutMs: 500 },
|
||||
opts: { ...createStatusGatewayProbeBudget(500), deep: true },
|
||||
includeSecurityAudit: false,
|
||||
suppressHealthErrors: true,
|
||||
});
|
||||
|
|
@ -228,6 +236,7 @@ describe("status-json-runtime", () => {
|
|||
config: { update: { channel: "stable" }, gateway: {} },
|
||||
sourceConfig: { gateway: {} },
|
||||
timeoutMs: 500,
|
||||
gatewayProbeDeadlineMs: 500,
|
||||
usage: undefined,
|
||||
deep: true,
|
||||
gatewayReachable: true,
|
||||
|
|
|
|||
|
|
@ -8,16 +8,16 @@ import {
|
|||
resolveStatusRuntimeSnapshot,
|
||||
resolveStatusUsageSummary,
|
||||
} from "./status-runtime-shared.ts";
|
||||
import type { StatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
import type { StatusJsonScanResult } from "./status.scan-result.ts";
|
||||
|
||||
/** Builds the status JSON object from a completed scan plus optional runtime/deep probes. */
|
||||
export async function resolveStatusJsonOutput(params: {
|
||||
scan: StatusJsonScanResult;
|
||||
opts: {
|
||||
opts: StatusGatewayProbeBudget & {
|
||||
deep?: boolean;
|
||||
usage?: boolean;
|
||||
agent?: string;
|
||||
timeoutMs?: number;
|
||||
};
|
||||
includeSecurityAudit: boolean;
|
||||
includePluginCompatibility?: boolean;
|
||||
|
|
@ -30,10 +30,15 @@ export async function resolveStatusJsonOutput(params: {
|
|||
config: scan.cfg,
|
||||
sourceConfig: scan.sourceConfig,
|
||||
timeoutMs: opts.timeoutMs,
|
||||
gatewayProbeDeadlineMs: opts.gatewayProbeDeadlineMs,
|
||||
...(opts.agent ? { agentId: opts.agent } : {}),
|
||||
usage: opts.usage,
|
||||
deep: opts.deep,
|
||||
gatewayReachable: scan.gatewayReachable,
|
||||
...(scan.gatewayProbe?.startupPhase
|
||||
? { gatewayStartupPhase: scan.gatewayProbe.startupPhase }
|
||||
: {}),
|
||||
...(scan.gatewayProbe?.error ? { gatewayProbeError: scan.gatewayProbe.error } : {}),
|
||||
includeSecurityAudit: params.includeSecurityAudit && !scan.collection,
|
||||
suppressHealthErrors: params.suppressHealthErrors,
|
||||
...(scan.collection && opts.usage
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@
|
|||
|
||||
import type { RuntimeEnv } from "../runtime.js";
|
||||
import { runStatusJsonCommand } from "./status-json-command.ts";
|
||||
import { createStatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
import { scanStatusJsonFast } from "./status.scan.fast-json.js";
|
||||
|
||||
/** Runs status JSON with the standard fast scan and all-mode security audit behavior. */
|
||||
|
|
@ -17,7 +18,7 @@ export async function statusJsonCommand(
|
|||
runtime: RuntimeEnv,
|
||||
) {
|
||||
await runStatusJsonCommand({
|
||||
opts,
|
||||
opts: { ...opts, ...createStatusGatewayProbeBudget(opts.timeoutMs) },
|
||||
runtime,
|
||||
scanStatusJsonFast,
|
||||
includeSecurityAudit: opts.all === true || opts.deep === true,
|
||||
|
|
|
|||
|
|
@ -146,6 +146,7 @@ export function buildStatusCommandOverviewRows(
|
|||
const lastHeartbeatValue = buildStatusLastHeartbeatValue({
|
||||
deep: params.opts.deep,
|
||||
gatewayReachable: params.surface.gatewayReachable,
|
||||
gatewayStartupPhase: params.surface.gatewayProbe?.startupPhase,
|
||||
lastHeartbeat: params.lastHeartbeat,
|
||||
warn: params.warn,
|
||||
muted: params.muted,
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
// Status runtime shared tests cover gateway health, runtime details, and safe status probe fallbacks.
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
resolveStatusGatewayDiagnosticsSafe,
|
||||
resolveStatusGatewayHealth,
|
||||
|
|
@ -9,6 +9,7 @@ import {
|
|||
resolveStatusServiceSummaries,
|
||||
resolveStatusUsageSummary,
|
||||
} from "./status-runtime-shared.ts";
|
||||
import { createStatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
|
||||
const mocks = vi.hoisted(() => ({
|
||||
loadProviderUsageSummary: vi.fn(),
|
||||
|
|
@ -67,6 +68,7 @@ function requireProviderUsageCall(): {
|
|||
describe("status-runtime-shared", () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
mocks.loadProviderUsageSummary.mockResolvedValue({ providers: [] });
|
||||
mocks.runSecurityAudit.mockResolvedValue({ summary: { critical: 0 }, findings: [] });
|
||||
mocks.callGateway.mockResolvedValue({ ok: true });
|
||||
|
|
@ -75,6 +77,10 @@ describe("status-runtime-shared", () => {
|
|||
mocks.resolveModelAuthLabel.mockReturnValue(undefined);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it("resolves the shared security audit payload", async () => {
|
||||
await resolveStatusSecurityAudit({
|
||||
config: { gateway: {} },
|
||||
|
|
@ -91,16 +97,24 @@ describe("status-runtime-shared", () => {
|
|||
});
|
||||
});
|
||||
|
||||
it("resolves usage summaries with the provided timeout", async () => {
|
||||
await resolveStatusUsageSummary({
|
||||
timeoutMs: 1234,
|
||||
config: { gateway: {} },
|
||||
});
|
||||
it("passes the remaining status budget through to provider usage", async () => {
|
||||
const clock = vi.spyOn(performance, "now").mockReturnValue(22_000);
|
||||
try {
|
||||
await resolveStatusRuntimeSnapshot({
|
||||
config: { gateway: {} },
|
||||
sourceConfig: { gateway: {} },
|
||||
usage: true,
|
||||
gatewayReachable: true,
|
||||
gatewayProbeDeadlineMs: 60_000,
|
||||
});
|
||||
|
||||
const usageCall = requireProviderUsageCall();
|
||||
expect(usageCall.timeoutMs).toBe(1234);
|
||||
expect(usageCall.config).toEqual({ gateway: {} });
|
||||
expect(usageCall.agentDir).toContain("main");
|
||||
const usageCall = requireProviderUsageCall();
|
||||
expect(usageCall.timeoutMs).toBe(38_000);
|
||||
expect(usageCall.config).toEqual({ gateway: {} });
|
||||
expect(usageCall.agentDir).toContain("main");
|
||||
} finally {
|
||||
clock.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it("uses the named system agent for agent-scoped usage credentials", async () => {
|
||||
|
|
@ -115,10 +129,10 @@ describe("status-runtime-shared", () => {
|
|||
},
|
||||
};
|
||||
|
||||
await resolveStatusUsageSummary({ config });
|
||||
await resolveStatusUsageSummary({ ...createStatusGatewayProbeBudget(), config });
|
||||
|
||||
expect(mocks.loadProviderUsageSummary).toHaveBeenCalledWith({
|
||||
timeoutMs: undefined,
|
||||
timeoutMs: 60_000,
|
||||
config,
|
||||
agentDir: "/tmp/status-ops-agent",
|
||||
});
|
||||
|
|
@ -127,6 +141,7 @@ describe("status-runtime-shared", () => {
|
|||
it("requires a system owner for usage credentials in an explicit multi-agent roster", async () => {
|
||||
await expect(
|
||||
resolveStatusUsageSummary({
|
||||
...createStatusGatewayProbeBudget(),
|
||||
config: {
|
||||
agents: {
|
||||
ownership: "explicit",
|
||||
|
|
@ -138,9 +153,54 @@ describe("status-runtime-shared", () => {
|
|||
expect(mocks.loadProviderUsageSummary).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("adds Codex synthetic usage for configured OpenAI Codex runtime routes without profiles", async () => {
|
||||
mocks.loadProviderUsageSummary
|
||||
.mockResolvedValueOnce({
|
||||
it.each([
|
||||
{ elapsedMs: 2000, remainingMs: 1456 },
|
||||
{ elapsedMs: 3456, remainingMs: 0 },
|
||||
])(
|
||||
"shares the usage deadline with Codex synthetic usage ($elapsedMs ms spent)",
|
||||
async ({ elapsedMs, remainingMs }) => {
|
||||
mocks.loadProviderUsageSummary
|
||||
.mockImplementationOnce(async () => {
|
||||
vi.spyOn(performance, "now").mockReturnValue(elapsedMs);
|
||||
return {
|
||||
updatedAt: 1,
|
||||
providers: [
|
||||
{
|
||||
provider: "anthropic",
|
||||
displayName: "Claude",
|
||||
windows: [],
|
||||
error: "HTTP 429",
|
||||
},
|
||||
],
|
||||
};
|
||||
})
|
||||
.mockResolvedValueOnce({
|
||||
updatedAt: 2,
|
||||
providers: [
|
||||
{
|
||||
provider: "openai",
|
||||
displayName: "OpenAI",
|
||||
windows: [{ label: "5h", usedPercent: 9 }],
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
await expect(
|
||||
resolveStatusUsageSummary({
|
||||
...createStatusGatewayProbeBudget(3456),
|
||||
config: {
|
||||
agents: {
|
||||
defaults: {
|
||||
model: { primary: "openai/gpt-5.5" },
|
||||
models: {
|
||||
"openai/gpt-5.5": { agentRuntime: { id: "codex" } },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
agentDir: "/tmp/status-agent",
|
||||
}),
|
||||
).resolves.toEqual({
|
||||
updatedAt: 1,
|
||||
providers: [
|
||||
{
|
||||
|
|
@ -149,11 +209,6 @@ describe("status-runtime-shared", () => {
|
|||
windows: [],
|
||||
error: "HTTP 429",
|
||||
},
|
||||
],
|
||||
})
|
||||
.mockResolvedValueOnce({
|
||||
updatedAt: 2,
|
||||
providers: [
|
||||
{
|
||||
provider: "openai",
|
||||
displayName: "OpenAI",
|
||||
|
|
@ -162,52 +217,21 @@ describe("status-runtime-shared", () => {
|
|||
],
|
||||
});
|
||||
|
||||
await expect(
|
||||
resolveStatusUsageSummary({
|
||||
timeoutMs: 3456,
|
||||
config: {
|
||||
agents: {
|
||||
defaults: {
|
||||
model: { primary: "openai/gpt-5.5" },
|
||||
models: {
|
||||
"openai/gpt-5.5": { agentRuntime: { id: "codex" } },
|
||||
},
|
||||
},
|
||||
expect(mocks.loadProviderUsageSummary).toHaveBeenNthCalledWith(2, {
|
||||
timeoutMs: remainingMs,
|
||||
providers: ["openai"],
|
||||
auth: [
|
||||
{
|
||||
provider: "openai",
|
||||
token: "codex-app-server",
|
||||
hookProvider: "codex",
|
||||
},
|
||||
},
|
||||
],
|
||||
config: expect.any(Object),
|
||||
agentDir: "/tmp/status-agent",
|
||||
}),
|
||||
).resolves.toEqual({
|
||||
updatedAt: 1,
|
||||
providers: [
|
||||
{
|
||||
provider: "anthropic",
|
||||
displayName: "Claude",
|
||||
windows: [],
|
||||
error: "HTTP 429",
|
||||
},
|
||||
{
|
||||
provider: "openai",
|
||||
displayName: "OpenAI",
|
||||
windows: [{ label: "5h", usedPercent: 9 }],
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
expect(mocks.loadProviderUsageSummary).toHaveBeenNthCalledWith(2, {
|
||||
timeoutMs: 3456,
|
||||
providers: ["openai"],
|
||||
auth: [
|
||||
{
|
||||
provider: "openai",
|
||||
token: "codex-app-server",
|
||||
hookProvider: "codex",
|
||||
},
|
||||
],
|
||||
config: expect.any(Object),
|
||||
agentDir: "/tmp/status-agent",
|
||||
});
|
||||
});
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("keeps existing OpenAI usage when Codex synthetic usage has no windows", async () => {
|
||||
mocks.loadProviderUsageSummary
|
||||
|
|
@ -234,7 +258,7 @@ describe("status-runtime-shared", () => {
|
|||
|
||||
await expect(
|
||||
resolveStatusUsageSummary({
|
||||
timeoutMs: 3456,
|
||||
...createStatusGatewayProbeBudget(3456),
|
||||
config: {
|
||||
agents: {
|
||||
defaults: {
|
||||
|
|
@ -261,7 +285,7 @@ describe("status-runtime-shared", () => {
|
|||
|
||||
it("does not add Codex synthetic usage for OpenAI routes pinned to OpenClaw runtime", async () => {
|
||||
await resolveStatusUsageSummary({
|
||||
timeoutMs: 3456,
|
||||
...createStatusGatewayProbeBudget(3456),
|
||||
config: {
|
||||
agents: {
|
||||
defaults: {
|
||||
|
|
@ -283,7 +307,7 @@ describe("status-runtime-shared", () => {
|
|||
mocks.resolveModelAuthLabel.mockReturnValue("api-key (openai:api)");
|
||||
|
||||
await resolveStatusUsageSummary({
|
||||
timeoutMs: 3456,
|
||||
...createStatusGatewayProbeBudget(3456),
|
||||
config: {
|
||||
agents: {
|
||||
defaults: {
|
||||
|
|
@ -310,7 +334,7 @@ describe("status-runtime-shared", () => {
|
|||
|
||||
it("resolves usage summaries with explicit agent scope", async () => {
|
||||
await resolveStatusUsageSummary({
|
||||
timeoutMs: 2345,
|
||||
...createStatusGatewayProbeBudget(2345),
|
||||
config: { gateway: {} },
|
||||
agentDir: "/tmp/status-agent",
|
||||
});
|
||||
|
|
@ -334,7 +358,7 @@ describe("status-runtime-shared", () => {
|
|||
};
|
||||
|
||||
await resolveStatusUsageSummary({
|
||||
timeoutMs: 2345,
|
||||
...createStatusGatewayProbeBudget(2345),
|
||||
config,
|
||||
agentId: "beta",
|
||||
});
|
||||
|
|
@ -349,6 +373,7 @@ describe("status-runtime-shared", () => {
|
|||
it("rejects an unknown explicit usage owner", async () => {
|
||||
await expect(
|
||||
resolveStatusUsageSummary({
|
||||
...createStatusGatewayProbeBudget(),
|
||||
config: {
|
||||
agents: {
|
||||
ownership: "explicit",
|
||||
|
|
@ -364,7 +389,7 @@ describe("status-runtime-shared", () => {
|
|||
it("resolves gateway health with the shared probe call shape", async () => {
|
||||
await resolveStatusGatewayHealth({
|
||||
config: { gateway: {} },
|
||||
timeoutMs: 5000,
|
||||
...createStatusGatewayProbeBudget(5000),
|
||||
});
|
||||
|
||||
expect(mocks.callGateway).toHaveBeenCalledWith({
|
||||
|
|
@ -378,6 +403,7 @@ describe("status-runtime-shared", () => {
|
|||
it("returns a fallback health error when the gateway is unreachable", async () => {
|
||||
await expect(
|
||||
resolveStatusGatewayHealthSafe({
|
||||
...createStatusGatewayProbeBudget(),
|
||||
config: { gateway: {} },
|
||||
gatewayReachable: false,
|
||||
gatewayProbeError: "timeout",
|
||||
|
|
@ -389,7 +415,7 @@ describe("status-runtime-shared", () => {
|
|||
it("passes gateway call overrides through the safe health path", async () => {
|
||||
await resolveStatusGatewayHealthSafe({
|
||||
config: { gateway: {} },
|
||||
timeoutMs: 4321,
|
||||
...createStatusGatewayProbeBudget(4321),
|
||||
gatewayReachable: true,
|
||||
callOverrides: {
|
||||
url: "ws://127.0.0.1:18789",
|
||||
|
|
@ -411,7 +437,7 @@ describe("status-runtime-shared", () => {
|
|||
await expect(
|
||||
resolveStatusGatewayDiagnosticsSafe({
|
||||
config: { gateway: {} },
|
||||
timeoutMs: 4321,
|
||||
...createStatusGatewayProbeBudget(4321),
|
||||
gatewayReachable: true,
|
||||
type: "telemetry.exporter",
|
||||
}),
|
||||
|
|
@ -431,7 +457,7 @@ describe("status-runtime-shared", () => {
|
|||
await expect(
|
||||
resolveStatusGatewayDiagnosticsSafe({
|
||||
config: { gateway: {} },
|
||||
timeoutMs: 4321,
|
||||
...createStatusGatewayProbeBudget(4321),
|
||||
gatewayReachable: true,
|
||||
}),
|
||||
).resolves.toEqual({
|
||||
|
|
@ -452,7 +478,7 @@ describe("status-runtime-shared", () => {
|
|||
resolveStatusRuntimeSnapshot({
|
||||
config: { gateway: {} },
|
||||
sourceConfig: { gateway: { mode: "local" } },
|
||||
timeoutMs: 1234,
|
||||
...createStatusGatewayProbeBudget(1234),
|
||||
usage: true,
|
||||
deep: true,
|
||||
gatewayReachable: true,
|
||||
|
|
@ -481,6 +507,7 @@ describe("status-runtime-shared", () => {
|
|||
const resolveUsage = vi.fn(async () => ({ updatedAt: 1, providers: [] }));
|
||||
|
||||
await resolveStatusRuntimeSnapshot({
|
||||
...createStatusGatewayProbeBudget(),
|
||||
config: { gateway: {} },
|
||||
sourceConfig: { gateway: {} },
|
||||
agentId: "beta",
|
||||
|
|
@ -492,7 +519,8 @@ describe("status-runtime-shared", () => {
|
|||
expect(resolveUsage).toHaveBeenCalledWith({
|
||||
config: { gateway: {} },
|
||||
agentId: "beta",
|
||||
timeoutMs: undefined,
|
||||
timeoutMs: 60_000,
|
||||
gatewayProbeDeadlineMs: 60_000,
|
||||
});
|
||||
});
|
||||
|
||||
|
|
@ -501,6 +529,7 @@ describe("status-runtime-shared", () => {
|
|||
|
||||
await expect(
|
||||
resolveStatusRuntimeSnapshot({
|
||||
...createStatusGatewayProbeBudget(),
|
||||
config: { gateway: {} },
|
||||
sourceConfig: { gateway: {} },
|
||||
deep: true,
|
||||
|
|
@ -513,11 +542,63 @@ describe("status-runtime-shared", () => {
|
|||
});
|
||||
});
|
||||
|
||||
it("shares the readiness deadline with deep health and skips heartbeat when it expires", async () => {
|
||||
const clock = vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
try {
|
||||
mocks.callGateway.mockImplementation(async () => {
|
||||
clock.mockReturnValue(38_000);
|
||||
return { ok: true };
|
||||
});
|
||||
const result = await resolveStatusRuntimeSnapshot({
|
||||
config: {},
|
||||
sourceConfig: {},
|
||||
deep: true,
|
||||
gatewayReachable: true,
|
||||
gatewayProbeDeadlineMs: 38_000,
|
||||
});
|
||||
|
||||
expect(result.health).toEqual({ ok: true });
|
||||
expect(result.lastHeartbeat).toBeNull();
|
||||
expect(mocks.callGateway).toHaveBeenCalledExactlyOnceWith({
|
||||
method: "health",
|
||||
params: { probe: true },
|
||||
config: {},
|
||||
timeoutMs: 38_000,
|
||||
});
|
||||
} finally {
|
||||
clock.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ gatewayStartupPhase: "plugins", health: undefined },
|
||||
{ gatewayStartupPhase: undefined, health: { error: "connection refused" } },
|
||||
])(
|
||||
"uses the completed initial probe for deep health ($gatewayStartupPhase)",
|
||||
async ({ gatewayStartupPhase, health }) => {
|
||||
const snapshot = await resolveStatusRuntimeSnapshot({
|
||||
...createStatusGatewayProbeBudget(),
|
||||
config: {},
|
||||
sourceConfig: {},
|
||||
deep: true,
|
||||
gatewayReachable: false,
|
||||
gatewayStartupPhase,
|
||||
gatewayProbeError: "connection refused",
|
||||
suppressHealthErrors: true,
|
||||
});
|
||||
|
||||
expect(snapshot.health).toEqual(health);
|
||||
expect(snapshot.lastHeartbeat).toBeNull();
|
||||
expect(mocks.callGateway).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
|
||||
it("does not suppress failed deep health probes for text status", async () => {
|
||||
mocks.callGateway.mockRejectedValueOnce(new Error("gateway health probe timed out"));
|
||||
|
||||
await expect(
|
||||
resolveStatusRuntimeSnapshot({
|
||||
...createStatusGatewayProbeBudget(),
|
||||
config: { gateway: {} },
|
||||
sourceConfig: { gateway: {} },
|
||||
deep: true,
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ import { createLazyImportLoader } from "../shared/lazy-promise.js";
|
|||
import type { HealthSummary } from "./health.js";
|
||||
import type { StatusUsageSummaryOptions } from "./status-usage.runtime.js";
|
||||
import { getDaemonStatusSummary, getNodeDaemonStatusSummary } from "./status.daemon.js";
|
||||
import { resolveStatusGatewayProbeTimeoutMs } from "./status.gateway-probe-budget.js";
|
||||
|
||||
const statusUsageModuleLoader = createLazyImportLoader(() => import("./status-usage.runtime.js"));
|
||||
const securityAuditModuleLoader = createLazyImportLoader(
|
||||
|
|
@ -44,12 +45,17 @@ export async function resolveStatusUsageSummary(params: StatusUsageSummaryOption
|
|||
export async function resolveStatusGatewayHealth(params: {
|
||||
config: OpenClawConfig;
|
||||
timeoutMs?: number;
|
||||
gatewayProbeDeadlineMs: number;
|
||||
}) {
|
||||
const { callGateway } = await gatewayCallModuleLoader.load();
|
||||
const timeoutMs = resolveStatusGatewayProbeTimeoutMs(params);
|
||||
if (timeoutMs === 0) {
|
||||
throw new Error("Gateway probe budget exhausted before health check.");
|
||||
}
|
||||
return await callGateway<HealthSummary>({
|
||||
method: "health",
|
||||
params: { probe: true },
|
||||
timeoutMs: params.timeoutMs,
|
||||
timeoutMs,
|
||||
config: params.config,
|
||||
});
|
||||
}
|
||||
|
|
@ -58,6 +64,7 @@ export async function resolveStatusGatewayHealth(params: {
|
|||
export async function resolveStatusGatewayHealthSafe(params: {
|
||||
config: OpenClawConfig;
|
||||
timeoutMs?: number;
|
||||
gatewayProbeDeadlineMs: number;
|
||||
gatewayReachable: boolean;
|
||||
gatewayProbeError?: string | null;
|
||||
callOverrides?: {
|
||||
|
|
@ -71,10 +78,14 @@ export async function resolveStatusGatewayHealthSafe(params: {
|
|||
return { error: params.gatewayProbeError ?? "gateway unreachable" };
|
||||
}
|
||||
const { callGateway } = await gatewayCallModuleLoader.load();
|
||||
const timeoutMs = resolveStatusGatewayProbeTimeoutMs(params);
|
||||
if (timeoutMs === 0) {
|
||||
return { error: "Gateway probe budget exhausted before health check." };
|
||||
}
|
||||
return await callGateway<HealthSummary>({
|
||||
method: "health",
|
||||
params: { probe: true },
|
||||
timeoutMs: params.timeoutMs,
|
||||
timeoutMs,
|
||||
config: params.config,
|
||||
...params.callOverrides,
|
||||
}).catch((err: unknown) => ({ error: String(err) }));
|
||||
|
|
@ -86,6 +97,7 @@ export type StatusGatewayDiagnosticsResult = Result<unknown, string>;
|
|||
export async function resolveStatusGatewayDiagnosticsSafe(params: {
|
||||
config: OpenClawConfig;
|
||||
timeoutMs?: number;
|
||||
gatewayProbeDeadlineMs: number;
|
||||
gatewayReachable: boolean;
|
||||
type?: string;
|
||||
callOverrides?: {
|
||||
|
|
@ -98,10 +110,14 @@ export async function resolveStatusGatewayDiagnosticsSafe(params: {
|
|||
return { ok: false, error: "gateway unreachable" };
|
||||
}
|
||||
const { callGateway } = await gatewayCallModuleLoader.load();
|
||||
const timeoutMs = resolveStatusGatewayProbeTimeoutMs(params);
|
||||
if (timeoutMs === 0) {
|
||||
return { ok: false, error: "Gateway probe budget exhausted before diagnostics." };
|
||||
}
|
||||
return await callGateway<unknown>({
|
||||
method: "diagnostics.stability",
|
||||
params: { limit: 1000, ...(params.type ? { type: params.type } : {}) },
|
||||
timeoutMs: params.timeoutMs,
|
||||
timeoutMs,
|
||||
config: params.config,
|
||||
...params.callOverrides,
|
||||
}).then<StatusGatewayDiagnosticsResult, StatusGatewayDiagnosticsResult>(
|
||||
|
|
@ -114,16 +130,21 @@ export async function resolveStatusGatewayDiagnosticsSafe(params: {
|
|||
async function resolveStatusLastHeartbeat(params: {
|
||||
config: OpenClawConfig;
|
||||
timeoutMs?: number;
|
||||
gatewayProbeDeadlineMs: number;
|
||||
gatewayReachable: boolean;
|
||||
}) {
|
||||
if (!params.gatewayReachable) {
|
||||
return null;
|
||||
}
|
||||
const { callGateway } = await gatewayCallModuleLoader.load();
|
||||
const timeoutMs = resolveStatusGatewayProbeTimeoutMs(params);
|
||||
if (timeoutMs === 0) {
|
||||
return null;
|
||||
}
|
||||
return await callGateway<HeartbeatEventPayload | null>({
|
||||
method: "last-heartbeat",
|
||||
params: {},
|
||||
timeoutMs: params.timeoutMs,
|
||||
timeoutMs,
|
||||
config: params.config,
|
||||
}).catch(() => null);
|
||||
}
|
||||
|
|
@ -153,46 +174,58 @@ type StatusSecurityAudit = Awaited<ReturnType<typeof resolveStatusSecurityAudit>
|
|||
async function resolveStatusRuntimeDetails(params: {
|
||||
config: OpenClawConfig;
|
||||
timeoutMs?: number;
|
||||
gatewayProbeDeadlineMs: number;
|
||||
agentId?: string;
|
||||
usage?: boolean;
|
||||
deep?: boolean;
|
||||
gatewayReachable: boolean;
|
||||
gatewayStartupPhase?: string;
|
||||
gatewayProbeError?: string | null;
|
||||
suppressHealthErrors?: boolean;
|
||||
resolveUsage?: (input: StatusUsageSummaryOptions) => Promise<StatusUsageSummary>;
|
||||
resolveHealth?: (input: {
|
||||
config: OpenClawConfig;
|
||||
timeoutMs?: number;
|
||||
gatewayProbeDeadlineMs: number;
|
||||
}) => Promise<StatusGatewayHealth>;
|
||||
}) {
|
||||
const resolveUsageSummary = params.resolveUsage ?? resolveStatusUsageSummary;
|
||||
const resolveGatewayHealthSummary = params.resolveHealth ?? resolveStatusGatewayHealth;
|
||||
const usage = params.usage
|
||||
? await resolveUsageSummary({
|
||||
timeoutMs: params.timeoutMs,
|
||||
timeoutMs: resolveStatusGatewayProbeTimeoutMs(params),
|
||||
gatewayProbeDeadlineMs: params.gatewayProbeDeadlineMs,
|
||||
config: params.config,
|
||||
...(params.agentId ? { agentId: params.agentId } : {}),
|
||||
})
|
||||
: undefined;
|
||||
// JSON status remains nonthrowing, but requested probe failures must stay visible.
|
||||
const health = params.deep
|
||||
? params.suppressHealthErrors
|
||||
? await resolveGatewayHealthSummary({
|
||||
config: params.config,
|
||||
timeoutMs: params.timeoutMs,
|
||||
}).catch((error: unknown) => ({ error: String(error) }))
|
||||
: await resolveGatewayHealthSummary({
|
||||
config: params.config,
|
||||
timeoutMs: params.timeoutMs,
|
||||
})
|
||||
: undefined;
|
||||
const health =
|
||||
params.deep && !params.gatewayStartupPhase
|
||||
? !params.gatewayReachable
|
||||
? { error: params.gatewayProbeError ?? "Gateway is unreachable" }
|
||||
: params.suppressHealthErrors
|
||||
? await resolveGatewayHealthSummary({
|
||||
config: params.config,
|
||||
timeoutMs: params.timeoutMs,
|
||||
gatewayProbeDeadlineMs: params.gatewayProbeDeadlineMs,
|
||||
}).catch((error: unknown) => ({ error: String(error) }))
|
||||
: await resolveGatewayHealthSummary({
|
||||
config: params.config,
|
||||
timeoutMs: params.timeoutMs,
|
||||
gatewayProbeDeadlineMs: params.gatewayProbeDeadlineMs,
|
||||
})
|
||||
: undefined;
|
||||
// Last heartbeat is a deep-only gateway call; fast status should not spend network time here.
|
||||
const lastHeartbeat = params.deep
|
||||
? await resolveStatusLastHeartbeat({
|
||||
config: params.config,
|
||||
timeoutMs: params.timeoutMs,
|
||||
gatewayReachable: params.gatewayReachable,
|
||||
})
|
||||
: null;
|
||||
const lastHeartbeat =
|
||||
params.deep && !params.gatewayStartupPhase
|
||||
? await resolveStatusLastHeartbeat({
|
||||
config: params.config,
|
||||
timeoutMs: params.timeoutMs,
|
||||
gatewayProbeDeadlineMs: params.gatewayProbeDeadlineMs,
|
||||
gatewayReachable: params.gatewayReachable,
|
||||
})
|
||||
: null;
|
||||
const [gatewayService, nodeService] = await resolveStatusServiceSummaries(params.timeoutMs);
|
||||
const result = {
|
||||
usage,
|
||||
|
|
@ -215,10 +248,13 @@ export async function resolveStatusRuntimeSnapshot(params: {
|
|||
config: OpenClawConfig;
|
||||
sourceConfig: OpenClawConfig;
|
||||
timeoutMs?: number;
|
||||
gatewayProbeDeadlineMs: number;
|
||||
agentId?: string;
|
||||
usage?: boolean;
|
||||
deep?: boolean;
|
||||
gatewayReachable: boolean;
|
||||
gatewayStartupPhase?: string;
|
||||
gatewayProbeError?: string | null;
|
||||
includeSecurityAudit?: boolean;
|
||||
suppressHealthErrors?: boolean;
|
||||
resolveSecurityAudit?: (input: {
|
||||
|
|
@ -230,6 +266,7 @@ export async function resolveStatusRuntimeSnapshot(params: {
|
|||
resolveHealth?: (input: {
|
||||
config: OpenClawConfig;
|
||||
timeoutMs?: number;
|
||||
gatewayProbeDeadlineMs: number;
|
||||
}) => Promise<StatusGatewayHealth>;
|
||||
}) {
|
||||
const securityAudit = params.includeSecurityAudit
|
||||
|
|
@ -242,10 +279,13 @@ export async function resolveStatusRuntimeSnapshot(params: {
|
|||
const runtimeDetails = await resolveStatusRuntimeDetails({
|
||||
config: params.config,
|
||||
timeoutMs: params.timeoutMs,
|
||||
gatewayProbeDeadlineMs: params.gatewayProbeDeadlineMs,
|
||||
...(params.agentId ? { agentId: params.agentId } : {}),
|
||||
usage: params.usage,
|
||||
deep: params.deep,
|
||||
gatewayReachable: params.gatewayReachable,
|
||||
gatewayStartupPhase: params.gatewayStartupPhase,
|
||||
gatewayProbeError: params.gatewayProbeError,
|
||||
suppressHealthErrors: params.suppressHealthErrors,
|
||||
resolveUsage: params.resolveUsage,
|
||||
resolveHealth: params.resolveHealth,
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ import {
|
|||
shouldUseCodexSyntheticUsageForRuntime,
|
||||
resolveUsageCredentialType,
|
||||
} from "../status/codex-synthetic-usage.js";
|
||||
import { resolveStatusGatewayProbeTimeoutMs } from "./status.gateway-probe-budget.js";
|
||||
|
||||
const providerUsageLoader = createLazyImportLoader(() => import("../infra/provider-usage.js"));
|
||||
|
||||
|
|
@ -61,6 +62,7 @@ function shouldUseConfiguredCodexSyntheticUsage(params: {
|
|||
export type StatusUsageSummaryOptions = {
|
||||
config: OpenClawConfig;
|
||||
timeoutMs?: number;
|
||||
gatewayProbeDeadlineMs: number;
|
||||
agentId?: string;
|
||||
agentDir?: string;
|
||||
};
|
||||
|
|
@ -86,7 +88,7 @@ export async function resolveStatusUsageSummary(params: StatusUsageSummaryOption
|
|||
agentDir = resolveAgentDir(params.config, resolvedAgentId);
|
||||
}
|
||||
const usage = await loadProviderUsageSummary({
|
||||
timeoutMs: params.timeoutMs,
|
||||
timeoutMs: resolveStatusGatewayProbeTimeoutMs(params),
|
||||
config: params.config,
|
||||
agentDir,
|
||||
});
|
||||
|
|
@ -100,7 +102,7 @@ export async function resolveStatusUsageSummary(params: StatusUsageSummaryOption
|
|||
return usage;
|
||||
}
|
||||
const codexUsage = await loadProviderUsageSummary({
|
||||
timeoutMs: params.timeoutMs,
|
||||
timeoutMs: resolveStatusGatewayProbeTimeoutMs(params),
|
||||
providers: ["openai"],
|
||||
auth: [buildCodexSyntheticUsageAuth()],
|
||||
config: params.config,
|
||||
|
|
|
|||
30
src/commands/status.channel-plugin.test-helpers.ts
Normal file
30
src/commands/status.channel-plugin.test-helpers.ts
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
function createChannelIssueCollector(channel: string) {
|
||||
return (accounts: Array<Record<string, unknown>>) =>
|
||||
accounts
|
||||
.filter((account) => typeof account.lastError === "string" && account.lastError)
|
||||
.map((account) => ({
|
||||
channel,
|
||||
accountId: typeof account.accountId === "string" ? account.accountId : "default",
|
||||
message: `Channel error: ${String(account.lastError)}`,
|
||||
}));
|
||||
}
|
||||
|
||||
export function createErrorChannelPlugin(params: { id: string; label: string; docsPath: string }) {
|
||||
return {
|
||||
id: params.id,
|
||||
meta: {
|
||||
id: params.id,
|
||||
label: params.label,
|
||||
selectionLabel: params.label,
|
||||
docsPath: params.docsPath,
|
||||
blurb: "mock",
|
||||
},
|
||||
config: {
|
||||
listAccountIds: () => ["default"],
|
||||
resolveAccount: () => ({}),
|
||||
},
|
||||
status: {
|
||||
collectStatusIssues: createChannelIssueCollector(params.id),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
|
@ -20,7 +20,12 @@ describe("status cold imports", () => {
|
|||
}));
|
||||
|
||||
const { resolveStatusRuntimeSnapshot } = await import("./status-runtime-shared.js");
|
||||
const params = { config: {}, sourceConfig: {}, gatewayReachable: false };
|
||||
const params = {
|
||||
config: {},
|
||||
sourceConfig: {},
|
||||
gatewayReachable: false,
|
||||
gatewayProbeDeadlineMs: performance.now() + 60_000,
|
||||
};
|
||||
const snapshot = await resolveStatusRuntimeSnapshot(params);
|
||||
|
||||
expect(snapshot).toEqual({
|
||||
|
|
|
|||
|
|
@ -76,6 +76,26 @@ describe("buildStatusCommandReportData", () => {
|
|||
);
|
||||
});
|
||||
|
||||
it("keeps pending startup guidance distinct from reachability failure", async () => {
|
||||
const params = createStatusCommandReportDataParams();
|
||||
const report = await buildStatusCommandReportData({
|
||||
...params,
|
||||
surface: {
|
||||
...params.surface,
|
||||
gatewayReachable: false,
|
||||
gatewayProbe: { startupPhase: "plugins", error: null },
|
||||
},
|
||||
health: undefined,
|
||||
lastHeartbeat: null,
|
||||
});
|
||||
|
||||
expect(
|
||||
stripAnsi(report.overviewRows.find(({ Item }) => Item === "Last heartbeat")?.Value ?? ""),
|
||||
).toBe("not checked (gateway still starting; phase plugins)");
|
||||
expect(report.footerLines.at(-1)).toBe(" Retry after startup: openclaw status --deep");
|
||||
expect(report.footerLines.join("\n")).not.toContain("Fix reachability first");
|
||||
});
|
||||
|
||||
it("builds report inputs from shared status surfaces", async () => {
|
||||
const baseParams = createStatusCommandReportDataParams();
|
||||
const result = await buildStatusCommandReportData(
|
||||
|
|
|
|||
|
|
@ -222,6 +222,7 @@ export async function buildStatusCommandReportData(params: {
|
|||
formatCliCommand,
|
||||
nodeOnlyGateway: params.surface.nodeOnlyGateway,
|
||||
gatewayReachable: params.surface.gatewayReachable,
|
||||
gatewayStartupPhase: params.surface.gatewayProbe?.startupPhase,
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -128,6 +128,7 @@ export function buildStatusHeartbeatValue(params: { summary: Pick<SummaryLike, "
|
|||
export function buildStatusLastHeartbeatValue(params: {
|
||||
deep?: boolean;
|
||||
gatewayReachable: boolean;
|
||||
gatewayStartupPhase?: string;
|
||||
lastHeartbeat: HeartbeatEventPayload | null;
|
||||
warn: (value: string) => string;
|
||||
muted: (value: string) => string;
|
||||
|
|
@ -137,6 +138,11 @@ export function buildStatusLastHeartbeatValue(params: {
|
|||
// Fast status omits the row entirely instead of implying heartbeat is missing.
|
||||
return null;
|
||||
}
|
||||
if (params.gatewayStartupPhase) {
|
||||
return params.muted(
|
||||
`not checked (gateway still starting; phase ${params.gatewayStartupPhase})`,
|
||||
);
|
||||
}
|
||||
if (!params.gatewayReachable) {
|
||||
return params.warn("unavailable");
|
||||
}
|
||||
|
|
@ -425,6 +431,7 @@ export function buildStatusFooterLines(params: {
|
|||
formatCliCommand: (value: string) => string;
|
||||
nodeOnlyGateway: unknown;
|
||||
gatewayReachable: boolean;
|
||||
gatewayStartupPhase?: string;
|
||||
}) {
|
||||
return [
|
||||
"FAQ: https://docs.openclaw.ai/faq",
|
||||
|
|
@ -435,9 +442,11 @@ export function buildStatusFooterLines(params: {
|
|||
` Need to debug live? ${params.formatCliCommand("openclaw logs --follow")}`,
|
||||
params.nodeOnlyGateway
|
||||
? ` Need node service? ${params.formatCliCommand("openclaw node status")}`
|
||||
: params.gatewayReachable
|
||||
? ` Need to test channels? ${params.formatCliCommand("openclaw status --deep")}`
|
||||
: ` Fix reachability first: ${params.formatCliCommand("openclaw gateway probe")}`,
|
||||
: params.gatewayStartupPhase
|
||||
? ` Retry after startup: ${params.formatCliCommand("openclaw status --deep")}`
|
||||
: params.gatewayReachable
|
||||
? ` Need to test channels? ${params.formatCliCommand("openclaw status --deep")}`
|
||||
: ` Fix reachability first: ${params.formatCliCommand("openclaw gateway probe")}`,
|
||||
];
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -24,6 +24,7 @@ import {
|
|||
} from "./status-runtime-shared.ts";
|
||||
import { buildStatusUpdateRows } from "./status-update-restart.ts";
|
||||
import { logGatewayConnectionDetails } from "./status.gateway-connection.ts";
|
||||
import { createStatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
|
||||
const statusScanModuleLoader = createLazyImportLoader(() => import("./status.scan.js"));
|
||||
const statusScanFastJsonModuleLoader = createLazyImportLoader(
|
||||
|
|
@ -102,6 +103,7 @@ export async function statusCommand(
|
|||
},
|
||||
runtime: RuntimeEnv,
|
||||
) {
|
||||
const probeBudget = createStatusGatewayProbeBudget(opts.timeoutMs);
|
||||
assertStatusUsageAgentScope(opts);
|
||||
for (const finding of await collectNodeRuntimeFindings()) {
|
||||
const write = opts.json ? runtime.error : runtime.log;
|
||||
|
|
@ -113,13 +115,13 @@ export async function statusCommand(
|
|||
// Human `--all` has a dedicated report path; JSON `--all` stays on the JSON schema.
|
||||
await statusAllModuleLoader
|
||||
.load()
|
||||
.then(({ statusAllCommand }) => statusAllCommand(runtime, opts));
|
||||
.then(({ statusAllCommand }) => statusAllCommand(runtime, { ...opts, ...probeBudget }));
|
||||
return;
|
||||
}
|
||||
|
||||
if (opts.json) {
|
||||
await runStatusJsonCommand({
|
||||
opts,
|
||||
opts: { ...opts, ...probeBudget },
|
||||
runtime,
|
||||
includeSecurityAudit: opts.all === true || opts.deep === true,
|
||||
includePluginCompatibility: opts.all === true,
|
||||
|
|
@ -134,7 +136,7 @@ export async function statusCommand(
|
|||
|
||||
const scan = await statusScanModuleLoader
|
||||
.load()
|
||||
.then(({ scanStatus }) => scanStatus({ timeoutMs: opts.timeoutMs, deep: opts.deep }));
|
||||
.then(({ scanStatus }) => scanStatus({ ...probeBudget, deep: opts.deep }));
|
||||
|
||||
const {
|
||||
cfg,
|
||||
|
|
@ -184,11 +186,13 @@ export async function statusCommand(
|
|||
} = await resolveStatusRuntimeSnapshot({
|
||||
config: scan.cfg,
|
||||
sourceConfig: scan.sourceConfig,
|
||||
timeoutMs: opts.timeoutMs,
|
||||
...probeBudget,
|
||||
...(opts.agent ? { agentId: opts.agent } : {}),
|
||||
usage: opts.usage,
|
||||
deep: opts.deep,
|
||||
gatewayReachable,
|
||||
...(gatewayProbe?.startupPhase ? { gatewayStartupPhase: gatewayProbe.startupPhase } : {}),
|
||||
...(gatewayProbe?.error ? { gatewayProbeError: gatewayProbe.error } : {}),
|
||||
includeSecurityAudit: opts.all === true || opts.deep === true,
|
||||
resolveSecurityAudit: async (input) =>
|
||||
await withProgress(
|
||||
|
|
|
|||
18
src/commands/status.gateway-probe-budget.ts
Normal file
18
src/commands/status.gateway-probe-budget.ts
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
import { DEFAULT_RESTART_HEALTH_TIMEOUT_MS } from "../cli/daemon-cli/restart-health.constants.js";
|
||||
|
||||
export type StatusGatewayProbeBudget = {
|
||||
timeoutMs?: number;
|
||||
gatewayProbeDeadlineMs: number;
|
||||
};
|
||||
|
||||
export function createStatusGatewayProbeBudget(timeoutMs?: number): StatusGatewayProbeBudget {
|
||||
return {
|
||||
timeoutMs,
|
||||
gatewayProbeDeadlineMs: performance.now() + (timeoutMs ?? DEFAULT_RESTART_HEALTH_TIMEOUT_MS),
|
||||
};
|
||||
}
|
||||
|
||||
export function resolveStatusGatewayProbeTimeoutMs(opts: StatusGatewayProbeBudget): number {
|
||||
const remainingMs = Math.max(0, Math.ceil(opts.gatewayProbeDeadlineMs - performance.now()));
|
||||
return opts.timeoutMs === undefined ? remainingMs : Math.min(opts.timeoutMs, remainingMs);
|
||||
}
|
||||
|
|
@ -1,7 +1,8 @@
|
|||
import { beforeEach, expect, it, vi } from "vitest";
|
||||
import { afterEach, beforeEach, expect, it, vi } from "vitest";
|
||||
import { createEmptyTaskAuditSummary } from "../tasks/task-registry.audit.shared.js";
|
||||
import { createEmptyTaskRegistrySummary } from "../tasks/task-registry.summary.js";
|
||||
import { withOpenClawTestState } from "../test-utils/openclaw-test-state.js";
|
||||
import { createStatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
import { scanStatusJsonFast } from "./status.scan.fast-json.js";
|
||||
|
||||
const mocks = vi.hoisted(() => ({
|
||||
|
|
@ -10,6 +11,7 @@ const mocks = vi.hoisted(() => ({
|
|||
localSummary: vi.fn(),
|
||||
callGateway: vi.fn(),
|
||||
probeGateway: vi.fn(),
|
||||
waitForGatewayDiagnosticReadiness: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock("../config/config.js", async (importOriginal) => {
|
||||
|
|
@ -45,9 +47,14 @@ vi.mock("./status.update.js", () => ({
|
|||
vi.mock("../gateway/probe.js", () => ({
|
||||
probeGateway: mocks.probeGateway,
|
||||
}));
|
||||
vi.mock("../cli/daemon-cli/diagnostic-readiness.js", () => ({
|
||||
waitForGatewayDiagnosticReadiness: mocks.waitForGatewayDiagnosticReadiness,
|
||||
}));
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
mocks.waitForGatewayDiagnosticReadiness.mockReset();
|
||||
mocks.fullConfigReads = 0;
|
||||
mocks.probeGateway.mockResolvedValue({
|
||||
ok: true,
|
||||
|
|
@ -63,13 +70,18 @@ beforeEach(() => {
|
|||
});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ withProjection: false, remote: false },
|
||||
{ withProjection: true, remote: false },
|
||||
{ withProjection: true, remote: true },
|
||||
{ withProjection: false, remote: false, readinessElapsedMs: 0 },
|
||||
{ withProjection: true, remote: false, readinessElapsedMs: 0 },
|
||||
{ withProjection: true, remote: true, readinessElapsedMs: 0 },
|
||||
{ withProjection: true, remote: false, readinessElapsedMs: 22_000 },
|
||||
])(
|
||||
"serves online fleet JSON without local discovery ($withProjection, remote: $remote)",
|
||||
async ({ withProjection, remote }) => {
|
||||
"serves online fleet JSON without local discovery ($withProjection, remote: $remote, startup: $readinessElapsedMs)",
|
||||
async ({ withProjection, remote, readinessElapsedMs }) => {
|
||||
await withOpenClawTestState(
|
||||
{ layout: "split", prefix: "status-gateway-projection-" },
|
||||
async (state) => {
|
||||
|
|
@ -136,10 +148,22 @@ it.each([
|
|||
sessions: { ...summary.sessions, count: 99 },
|
||||
});
|
||||
|
||||
const result = await scanStatusJsonFast(
|
||||
{},
|
||||
{ log: vi.fn(), error: vi.fn(), exit: vi.fn() },
|
||||
);
|
||||
const clock = vi.spyOn(performance, "now");
|
||||
if (readinessElapsedMs) {
|
||||
mocks.waitForGatewayDiagnosticReadiness.mockImplementationOnce(async () => {
|
||||
clock.mockReturnValue(readinessElapsedMs);
|
||||
return { healthy: true, elapsedMs: readinessElapsedMs, waitOutcome: "healthy" };
|
||||
});
|
||||
mocks.probeGateway.mockImplementationOnce(async () => {
|
||||
clock.mockReturnValue(readinessElapsedMs + 1000);
|
||||
return { ok: true, connectLatencyMs: 1, error: null, status: null, presence: [] };
|
||||
});
|
||||
}
|
||||
const result = await scanStatusJsonFast(createStatusGatewayProbeBudget(), {
|
||||
log: vi.fn(),
|
||||
error: vi.fn(),
|
||||
exit: vi.fn(),
|
||||
});
|
||||
|
||||
expect(mocks.fullConfigReads).toBe(0);
|
||||
expect(mocks.localAgents).not.toHaveBeenCalled();
|
||||
|
|
@ -157,6 +181,7 @@ it.each([
|
|||
expect.objectContaining({
|
||||
method: "status",
|
||||
params: { includeChannelSummary: false, includeCliProjection: true },
|
||||
timeoutMs: readinessElapsedMs ? 37_000 : 60_000,
|
||||
}),
|
||||
);
|
||||
},
|
||||
|
|
@ -190,13 +215,18 @@ it("keeps offline config diagnostics and local collection", async () => {
|
|||
mocks.localAgents.mockResolvedValue({ agentStatus: local, sessionStores: undefined });
|
||||
mocks.localSummary.mockResolvedValue({ sessions: { count: 4 }, heartbeat: { agents: [] } });
|
||||
|
||||
const result = await scanStatusJsonFast({}, { log: vi.fn(), error: vi.fn(), exit: vi.fn() });
|
||||
const result = await scanStatusJsonFast(createStatusGatewayProbeBudget(), {
|
||||
log: vi.fn(),
|
||||
error: vi.fn(),
|
||||
exit: vi.fn(),
|
||||
});
|
||||
|
||||
expect(mocks.fullConfigReads).toBe(1);
|
||||
expect(mocks.localAgents).toHaveBeenCalledOnce();
|
||||
expect(mocks.localSummary).toHaveBeenCalledOnce();
|
||||
expect(result.agentStatus).toEqual(local);
|
||||
expect(result.gatewayReachable).toBe(false);
|
||||
expect(mocks.waitForGatewayDiagnosticReadiness).toHaveBeenCalledOnce();
|
||||
expect(result.collection).toBeUndefined();
|
||||
expect(result.configDiagnostics?.issues).toEqual(
|
||||
expect.arrayContaining([
|
||||
|
|
|
|||
|
|
@ -39,7 +39,7 @@ vi.mock("./status.scan.shared.js", async (original) => ({
|
|||
vi.mock("./status.scan.js", () => ({ scanStatus: scanWithGitProbe }));
|
||||
vi.mock("./status.scan.fast-json.js", () => ({ scanStatusJsonFast: scanWithGitProbe }));
|
||||
|
||||
async function scanWithGitProbe(opts: { timeoutMs?: number; all?: boolean }) {
|
||||
async function scanWithGitProbe(opts: Parameters<typeof createStatusScanCoreBootstrap>[0]["opts"]) {
|
||||
const fixture = createStatusScanResultFixture({
|
||||
env: { OPENCLAW_STATE_DIR: process.env.OPENCLAW_STATE_DIR },
|
||||
});
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ import { afterEach, expect, it, vi } from "vitest";
|
|||
import { clearPluginMetadataLifecycleCaches } from "../plugins/plugin-metadata-lifecycle.js";
|
||||
import { createColdPluginFixture } from "../plugins/test-helpers/cold-plugin-fixtures.js";
|
||||
import { withOpenClawTestState } from "../test-utils/openclaw-test-state.js";
|
||||
import { createStatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
|
||||
const registryLoads = vi.hoisted(() => ({ count: 0 }));
|
||||
|
||||
|
|
@ -75,7 +76,7 @@ it("builds plugin metadata once for a status scan", async () => {
|
|||
|
||||
const overview = await collectStatusScanOverview({
|
||||
commandName: "status --json",
|
||||
opts: {},
|
||||
opts: createStatusGatewayProbeBudget(),
|
||||
showSecrets: false,
|
||||
includeChannelsData: false,
|
||||
skipUpdateCheck: true,
|
||||
|
|
|
|||
|
|
@ -1,10 +1,15 @@
|
|||
// Status scan overview tests cover overview collection and gateway/runtime summary inputs.
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { createSqliteWalHealth } from "./sqlite-wal-health.test-support.js";
|
||||
import { createStatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
import { collectStatusScanOverview } from "./status.scan-overview.ts";
|
||||
|
||||
const sqliteWal = createSqliteWalHealth();
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
const mocks = vi.hoisted(() => ({
|
||||
hasConfiguredChannelsForReadOnlyScope: vi.fn(),
|
||||
resolveCommandConfigWithSecrets: vi.fn(),
|
||||
|
|
@ -98,6 +103,7 @@ function firstChannelsTableCall(): ChannelsTableCall {
|
|||
describe("collectStatusScanOverview", () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
|
||||
mocks.hasConfiguredChannelsForReadOnlyScope.mockReturnValue(true);
|
||||
mocks.getStatusCommandSecretTargetIds.mockReturnValue([]);
|
||||
|
|
@ -169,7 +175,7 @@ describe("collectStatusScanOverview", () => {
|
|||
it("uses gateway fallback overrides for channels.status when requested", async () => {
|
||||
const result = await collectStatusScanOverview({
|
||||
commandName: "status --all",
|
||||
opts: { timeoutMs: 1234 },
|
||||
opts: createStatusGatewayProbeBudget(1234),
|
||||
showSecrets: false,
|
||||
useGatewayCallOverridesForChannelsStatus: true,
|
||||
});
|
||||
|
|
@ -201,7 +207,7 @@ describe("collectStatusScanOverview", () => {
|
|||
it("can keep channel overview on metadata-only status paths", async () => {
|
||||
const result = await collectStatusScanOverview({
|
||||
commandName: "status",
|
||||
opts: { timeoutMs: 1234 },
|
||||
opts: createStatusGatewayProbeBudget(1234),
|
||||
showSecrets: false,
|
||||
includeLiveChannelStatus: false,
|
||||
includeChannelSetupRuntimeFallback: false,
|
||||
|
|
@ -247,7 +253,7 @@ describe("collectStatusScanOverview", () => {
|
|||
});
|
||||
const result = await collectStatusScanOverview({
|
||||
commandName: "status",
|
||||
opts: {},
|
||||
opts: createStatusGatewayProbeBudget(),
|
||||
showSecrets: true,
|
||||
});
|
||||
|
||||
|
|
@ -296,7 +302,7 @@ describe("collectStatusScanOverview", () => {
|
|||
|
||||
const result = await collectStatusScanOverview({
|
||||
commandName: "status",
|
||||
opts: {},
|
||||
opts: createStatusGatewayProbeBudget(),
|
||||
showSecrets: false,
|
||||
includeChannelsData: false,
|
||||
});
|
||||
|
|
@ -340,7 +346,7 @@ describe("collectStatusScanOverview", () => {
|
|||
});
|
||||
const result = await collectStatusScanOverview({
|
||||
commandName: "status",
|
||||
opts: {},
|
||||
opts: createStatusGatewayProbeBudget(),
|
||||
showSecrets: false,
|
||||
includeChannelsData: false,
|
||||
});
|
||||
|
|
|
|||
|
|
@ -9,20 +9,21 @@ import type { collectChannelStatusIssues as collectChannelStatusIssuesFn } from
|
|||
import { resolveOsSummary } from "../infra/os-summary.js";
|
||||
import type { UpdateCheckResult } from "../infra/update-check.js";
|
||||
import { applyLoggingConfig } from "../logging/logger.js";
|
||||
import type { RuntimeEnv } from "../runtime.js";
|
||||
import { defaultRuntime, type RuntimeEnv } from "../runtime.js";
|
||||
import { createLazyImportLoader } from "../shared/lazy-promise.js";
|
||||
import type { StatusSessionStores } from "../status/session-stores.js";
|
||||
import type { StatusSummary } from "../status/summary.js";
|
||||
import type { buildChannelsTable as buildChannelsTableFn } from "./status-all/channels.js";
|
||||
import type { AgentLocalStatusesResult } from "./status.agent-local.js";
|
||||
import {
|
||||
resolveStatusGatewayProbeTimeoutMs,
|
||||
type StatusGatewayProbeBudget,
|
||||
} from "./status.gateway-probe-budget.js";
|
||||
import {
|
||||
buildColdStartStatusSummary,
|
||||
createStatusScanCoreBootstrap,
|
||||
} from "./status.scan.bootstrap-shared.js";
|
||||
import {
|
||||
resolveStatusGatewayProbeTimeoutMs,
|
||||
type GatewayProbeSnapshot,
|
||||
} from "./status.scan.shared.js";
|
||||
import type { GatewayProbeSnapshot } from "./status.scan.shared.js";
|
||||
|
||||
const statusScanDepsRuntimeModuleLoader = createLazyImportLoader(
|
||||
() => import("./status.scan.deps.runtime.js"),
|
||||
|
|
@ -54,7 +55,7 @@ async function resolveStatusChannelsStatus(params: {
|
|||
cfg: OpenClawConfig;
|
||||
configPath: string;
|
||||
gatewayReachable: boolean;
|
||||
opts: { timeoutMs?: number; all?: boolean };
|
||||
opts: StatusGatewayProbeBudget & { all?: boolean };
|
||||
gatewayCallOverrides?: GatewayProbeSnapshot["gatewayCallOverrides"];
|
||||
useGatewayCallOverrides?: boolean;
|
||||
}) {
|
||||
|
|
@ -63,18 +64,19 @@ async function resolveStatusChannelsStatus(params: {
|
|||
return null;
|
||||
}
|
||||
const { callGateway } = await gatewayCallModuleLoader.load();
|
||||
const timeoutMs = resolveStatusGatewayProbeTimeoutMs(params.opts);
|
||||
if (timeoutMs === 0) {
|
||||
return null;
|
||||
}
|
||||
return await callGateway({
|
||||
config: params.cfg,
|
||||
configPath: params.configPath,
|
||||
method: "channels.status",
|
||||
params: {
|
||||
probe: false,
|
||||
timeoutMs: Math.min(8000, params.opts.timeoutMs ?? 10_000),
|
||||
timeoutMs: Math.min(8000, timeoutMs),
|
||||
},
|
||||
timeoutMs: Math.min(
|
||||
resolveStatusGatewayProbeTimeoutMs({ all: params.opts.all }),
|
||||
params.opts.timeoutMs ?? 10_000,
|
||||
),
|
||||
timeoutMs,
|
||||
...(params.useGatewayCallOverrides === true ? (params.gatewayCallOverrides ?? {}) : {}),
|
||||
}).catch(() => null);
|
||||
}
|
||||
|
|
@ -129,7 +131,7 @@ export type StatusScanOverviewResult = {
|
|||
export async function collectStatusScanOverview(params: {
|
||||
env?: NodeJS.ProcessEnv;
|
||||
commandName: string;
|
||||
opts: { timeoutMs?: number; all?: boolean };
|
||||
opts: StatusGatewayProbeBudget & { all?: boolean };
|
||||
showSecrets: boolean;
|
||||
runtime?: RuntimeEnv;
|
||||
allowMissingConfigFastPath?: boolean;
|
||||
|
|
@ -143,7 +145,7 @@ export async function collectStatusScanOverview(params: {
|
|||
includeChannelsData?: boolean;
|
||||
includeLiveChannelStatus?: boolean;
|
||||
includeLocalStatusRpcFallback?: boolean;
|
||||
gatewayProbeTimeoutMs?: number;
|
||||
gatewaySnapshot?: GatewayProbeSnapshot;
|
||||
includeChannelSetupRuntimeFallback?: boolean;
|
||||
useGatewayCallOverridesForChannelsStatus?: boolean;
|
||||
includeAdvertisedControlUiLinks?: boolean;
|
||||
|
|
@ -183,8 +185,6 @@ export async function collectStatusScanOverview(params: {
|
|||
const configDiagnostics =
|
||||
skipMissingConfig || snapshot.valid ? null : { path: snapshot.path, issues: snapshot.issues };
|
||||
// Secret resolution precedes probing, and each request uses the scan's existing budget.
|
||||
const gatewayProbeTimeoutMs =
|
||||
params.gatewayProbeTimeoutMs ?? resolveStatusGatewayProbeTimeoutMs(params.opts);
|
||||
const { resolvedConfig: cfg, diagnostics } = skipMissingConfig
|
||||
? { resolvedConfig: loadedConfig, diagnostics: [] }
|
||||
: await measureCliCommandStartup(
|
||||
|
|
@ -200,7 +200,7 @@ export async function collectStatusScanOverview(params: {
|
|||
await commandSecretTargetsModuleLoader.load()
|
||||
).getStatusCommandSecretTargetIds(loadedConfig, env),
|
||||
mode: "read_only_status",
|
||||
gatewaySecretResolveTimeoutMs: gatewayProbeTimeoutMs,
|
||||
gatewaySecretResolveTimeoutMs: resolveStatusGatewayProbeTimeoutMs(params.opts),
|
||||
...(params.runtime ? { runtime: params.runtime } : {}),
|
||||
}),
|
||||
),
|
||||
|
|
@ -233,7 +233,16 @@ export async function collectStatusScanOverview(params: {
|
|||
fetchGitUpdate: params.fetchGitUpdate,
|
||||
includeRegistryUpdate: params.includeRegistryUpdate,
|
||||
includeLocalStatusRpcFallback: params.includeLocalStatusRpcFallback,
|
||||
gatewayProbeTimeoutMs,
|
||||
gatewaySnapshot: params.gatewaySnapshot,
|
||||
onGatewayProgress: params.progress
|
||||
? (phase) => {
|
||||
const message = `Gateway still starting (phase ${phase})`;
|
||||
params.progress?.setLabel(message);
|
||||
if (!process.stderr.isTTY) {
|
||||
(params.runtime ?? defaultRuntime).log(message);
|
||||
}
|
||||
}
|
||||
: undefined,
|
||||
getTailnetHostname: async (runner) => {
|
||||
return await statusScanDepsRuntimeModuleLoader
|
||||
.load()
|
||||
|
|
@ -283,21 +292,23 @@ export async function collectStatusScanOverview(params: {
|
|||
if (gatewaySnapshot.gatewayReachable) {
|
||||
const status =
|
||||
gatewaySnapshot.gatewayProbe?.status ??
|
||||
(await measureCliCommandStartup(
|
||||
"status.gateway-degradation",
|
||||
() =>
|
||||
gatewayCallModuleLoader.load().then(({ callGateway }) =>
|
||||
callGateway<StatusSummary>({
|
||||
config: cfg,
|
||||
configPath: snapshot.path,
|
||||
method: "status",
|
||||
params: { includeChannelSummary: false },
|
||||
timeoutMs: Math.min(5000, params.opts.timeoutMs ?? 10_000),
|
||||
...gatewaySnapshot.gatewayCallOverrides,
|
||||
}).catch(() => null),
|
||||
),
|
||||
{ config: cfg, env },
|
||||
));
|
||||
(resolveStatusGatewayProbeTimeoutMs(params.opts) > 0
|
||||
? await measureCliCommandStartup(
|
||||
"status.gateway-degradation",
|
||||
() =>
|
||||
gatewayCallModuleLoader.load().then(({ callGateway }) =>
|
||||
callGateway<StatusSummary>({
|
||||
config: cfg,
|
||||
configPath: snapshot.path,
|
||||
method: "status",
|
||||
params: { includeChannelSummary: false },
|
||||
timeoutMs: Math.min(5000, resolveStatusGatewayProbeTimeoutMs(params.opts)),
|
||||
...gatewaySnapshot.gatewayCallOverrides,
|
||||
}).catch(() => null),
|
||||
),
|
||||
{ config: cfg, env },
|
||||
)
|
||||
: null);
|
||||
runtimeDegradation = status
|
||||
? {
|
||||
degradedSecretOwners: status.degradedSecretOwners ?? [],
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import type { UpdateCheckResult } from "../infra/update-check.js";
|
|||
import { runExec } from "../process/exec.js";
|
||||
import { createEmptyTaskAuditSummary } from "../tasks/task-registry.audit.shared.js";
|
||||
import { createEmptyTaskRegistrySummary } from "../tasks/task-registry.summary.js";
|
||||
import type { StatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
import {
|
||||
buildTailscaleHttpsUrl,
|
||||
resolveGatewayProbeSnapshot,
|
||||
|
|
@ -74,13 +75,13 @@ type StatusScanCoreBootstrapParams<TAgentStatus> = {
|
|||
configPath: string;
|
||||
env: NodeJS.ProcessEnv;
|
||||
hasConfiguredChannels: boolean;
|
||||
opts: { timeoutMs?: number; all?: boolean };
|
||||
opts: StatusGatewayProbeBudget & { all?: boolean };
|
||||
skipUpdateCheck?: boolean;
|
||||
fetchGitUpdate?: boolean;
|
||||
includeRegistryUpdate?: boolean;
|
||||
includeLocalStatusRpcFallback?: boolean;
|
||||
gatewayProbeTimeoutMs?: number;
|
||||
gatewaySnapshot?: GatewayProbeSnapshot;
|
||||
onGatewayProgress?: (phase: string) => void;
|
||||
getTailnetHostname: (runner: StatusScanExecRunner) => Promise<string | null>;
|
||||
getUpdateCheckResult: (params: {
|
||||
timeoutMs: number;
|
||||
|
|
@ -135,11 +136,9 @@ export async function createStatusScanCoreBootstrap<TAgentStatus>(
|
|||
env: params.env,
|
||||
opts: {
|
||||
...params.opts,
|
||||
...(params.gatewayProbeTimeoutMs !== undefined
|
||||
? { timeoutMs: params.gatewayProbeTimeoutMs }
|
||||
: {}),
|
||||
...(skipColdStartNetworkChecks ? { skipProbe: true } : {}),
|
||||
localStatusRpcFallback: params.includeLocalStatusRpcFallback !== false,
|
||||
onProgress: params.onGatewayProgress,
|
||||
},
|
||||
}),
|
||||
{ config: params.cfg, env: params.env },
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
// Status scan fast-json tests cover scan defaults, memory config, and JSON-safe status payloads.
|
||||
import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { GENERATED_BUNDLED_CHANNEL_CONFIG_METADATA } from "../config/bundled-channel-config-metadata.generated.js";
|
||||
import { createStatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
import {
|
||||
applyStatusScanDefaults,
|
||||
createStatusMemorySearchConfig,
|
||||
|
|
@ -71,12 +72,14 @@ beforeAll(async () => {
|
|||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
configureFastJsonStatus();
|
||||
originalForceStderr = loggingStateRef.forceConsoleToStderr;
|
||||
loggingStateRef.forceConsoleToStderr = false;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
loggingStateRef.forceConsoleToStderr = originalForceStderr;
|
||||
});
|
||||
|
||||
|
|
@ -84,7 +87,7 @@ describe("scanStatusJsonFast", () => {
|
|||
it("does not preload configured channel plugins for the lean JSON path", async () => {
|
||||
mocks.hasConfiguredChannels.mockReturnValue(true);
|
||||
|
||||
await scanStatusJsonFast({}, {} as never);
|
||||
await scanStatusJsonFast(createStatusGatewayProbeBudget(), {} as never);
|
||||
|
||||
expect(mocks.getStatusCommandSecretTargetIds).toHaveBeenCalledWith(
|
||||
createStatusMemorySearchConfig(),
|
||||
|
|
@ -112,7 +115,7 @@ describe("scanStatusJsonFast", () => {
|
|||
configDiagnostics,
|
||||
});
|
||||
|
||||
const result = await scanStatusJsonFast({}, {} as never);
|
||||
const result = await scanStatusJsonFast(createStatusGatewayProbeBudget(), {} as never);
|
||||
|
||||
expect(result.configDiagnostics).toEqual(configDiagnostics);
|
||||
});
|
||||
|
|
@ -142,19 +145,22 @@ describe("scanStatusJsonFast", () => {
|
|||
} as never,
|
||||
});
|
||||
|
||||
await scanStatusJsonFast({}, {} as never);
|
||||
await scanStatusJsonFast(createStatusGatewayProbeBudget(), {} as never);
|
||||
|
||||
expect(mocks.ensurePluginRegistryLoaded).not.toHaveBeenCalled();
|
||||
expect(mocks.resolveCommandSecretRefsViaGateway).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ opts: {}, expected: 1000 },
|
||||
{ opts: {}, expected: 60_000 },
|
||||
{ opts: { timeoutMs: 1500 }, expected: 1500 },
|
||||
{ opts: { all: true }, expected: 5000 },
|
||||
{ opts: { all: true }, expected: 60_000 },
|
||||
{ opts: { all: true, timeoutMs: 700 }, expected: 700 },
|
||||
])("bounds gateway secret resolution to $expected ms for $opts", async ({ opts, expected }) => {
|
||||
await scanStatusJsonFast(opts, {} as never);
|
||||
await scanStatusJsonFast(
|
||||
{ ...opts, ...createStatusGatewayProbeBudget(opts.timeoutMs) },
|
||||
{} as never,
|
||||
);
|
||||
expect(mocks.resolveCommandSecretRefsViaGateway).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ gatewaySecretResolveTimeoutMs: expected }),
|
||||
);
|
||||
|
|
@ -163,7 +169,7 @@ describe("scanStatusJsonFast", () => {
|
|||
it("skips plugin compatibility loading even when configured channels are present", async () => {
|
||||
mocks.hasConfiguredChannels.mockReturnValue(true);
|
||||
|
||||
await scanStatusJsonFast({}, {} as never);
|
||||
await scanStatusJsonFast(createStatusGatewayProbeBudget(), {} as never);
|
||||
|
||||
expect(mocks.buildPluginCompatibilityNotices).not.toHaveBeenCalled();
|
||||
});
|
||||
|
|
@ -177,7 +183,10 @@ describe("scanStatusJsonFast", () => {
|
|||
};
|
||||
mocks.buildPluginCompatibilityNotices.mockReturnValue([notice]);
|
||||
|
||||
const result = await scanStatusJsonFast({ all: true }, {} as never);
|
||||
const result = await scanStatusJsonFast(
|
||||
{ ...createStatusGatewayProbeBudget(), all: true },
|
||||
{} as never,
|
||||
);
|
||||
|
||||
expect(mocks.buildPluginCompatibilityNotices).toHaveBeenCalledWith({
|
||||
config: createStatusMemorySearchConfig(),
|
||||
|
|
@ -188,7 +197,7 @@ describe("scanStatusJsonFast", () => {
|
|||
it("keeps default fast JSON update scans local-only", async () => {
|
||||
mocks.hasConfiguredChannels.mockReturnValue(true);
|
||||
|
||||
await scanStatusJsonFast({ timeoutMs: 1234 }, {} as never);
|
||||
await scanStatusJsonFast(createStatusGatewayProbeBudget(1234), {} as never);
|
||||
|
||||
expect(mocks.getUpdateCheckResult).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
|
|
@ -202,7 +211,7 @@ describe("scanStatusJsonFast", () => {
|
|||
it("restores registry-backed update checks and remote git fetches when --all is requested", async () => {
|
||||
mocks.hasConfiguredChannels.mockReturnValue(true);
|
||||
|
||||
await scanStatusJsonFast({ all: true }, {} as never);
|
||||
await scanStatusJsonFast({ ...createStatusGatewayProbeBudget(), all: true }, {} as never);
|
||||
|
||||
expect(mocks.getUpdateCheckResult).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
|
|
@ -217,16 +226,16 @@ describe("scanStatusJsonFast", () => {
|
|||
mocks.hasConfiguredChannels.mockReturnValue(true);
|
||||
mocks.callGateway.mockResolvedValue({ sessions: 1 });
|
||||
|
||||
await scanStatusJsonFast({}, {} as never);
|
||||
await scanStatusJsonFast(createStatusGatewayProbeBudget(), {} as never);
|
||||
|
||||
expect(mocks.probeGateway).toHaveBeenCalledWith(expect.objectContaining({ timeoutMs: 1000 }));
|
||||
expect(mocks.probeGateway).toHaveBeenCalledWith(expect.objectContaining({ timeoutMs: 60_000 }));
|
||||
expect(mocks.callGateway).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("honors explicit gateway probe timeouts on the lean JSON path", async () => {
|
||||
mocks.hasConfiguredChannels.mockReturnValue(true);
|
||||
|
||||
await scanStatusJsonFast({ timeoutMs: 5000 }, {} as never);
|
||||
await scanStatusJsonFast(createStatusGatewayProbeBudget(5000), {} as never);
|
||||
|
||||
expect(mocks.probeGateway).toHaveBeenCalledWith(expect.objectContaining({ timeoutMs: 5000 }));
|
||||
});
|
||||
|
|
@ -235,7 +244,7 @@ describe("scanStatusJsonFast", () => {
|
|||
mocks.hasConfiguredChannels.mockReturnValue(true);
|
||||
mocks.callGateway.mockResolvedValue({ sessions: 1 });
|
||||
|
||||
await scanStatusJsonFast({ all: true }, {} as never);
|
||||
await scanStatusJsonFast({ ...createStatusGatewayProbeBudget(), all: true }, {} as never);
|
||||
|
||||
expect(mocks.callGateway).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
|
|
@ -248,7 +257,7 @@ describe("scanStatusJsonFast", () => {
|
|||
it("keeps the fast JSON summary off the channel plugin summary path", async () => {
|
||||
mocks.hasConfiguredChannels.mockReturnValue(true);
|
||||
|
||||
await scanStatusJsonFast({}, {} as never);
|
||||
await scanStatusJsonFast(createStatusGatewayProbeBudget(), {} as never);
|
||||
|
||||
expect(mocks.getStatusSummary).toHaveBeenCalledOnce();
|
||||
const summaryOptions = firstCallArg(mocks.getStatusSummary, "status summary options") as {
|
||||
|
|
@ -258,7 +267,7 @@ describe("scanStatusJsonFast", () => {
|
|||
});
|
||||
|
||||
it("skips memory inspection for the lean status --json fast path", async () => {
|
||||
const result = await scanStatusJsonFast({}, {} as never);
|
||||
const result = await scanStatusJsonFast(createStatusGatewayProbeBudget(), {} as never);
|
||||
|
||||
expect(result.memory).toBeNull();
|
||||
expect(mocks.hasConfiguredChannels).not.toHaveBeenCalled();
|
||||
|
|
@ -267,7 +276,10 @@ describe("scanStatusJsonFast", () => {
|
|||
});
|
||||
|
||||
it("restores memory inspection when --all is requested", async () => {
|
||||
const result = await scanStatusJsonFast({ all: true }, {} as never);
|
||||
const result = await scanStatusJsonFast(
|
||||
{ ...createStatusGatewayProbeBudget(), all: true },
|
||||
{} as never,
|
||||
);
|
||||
|
||||
expect(result.memory).toStrictEqual({
|
||||
agentId: "main",
|
||||
|
|
@ -298,7 +310,7 @@ describe("scanStatusJsonFast", () => {
|
|||
NODE_ENV: undefined,
|
||||
},
|
||||
async () => {
|
||||
await scanStatusJsonFast({}, {} as never);
|
||||
await scanStatusJsonFast(createStatusGatewayProbeBudget(), {} as never);
|
||||
},
|
||||
);
|
||||
|
||||
|
|
@ -323,7 +335,7 @@ describe("scanStatusJsonFast", () => {
|
|||
summary: createStatusSummary({ linkChannel: { linked: false } }),
|
||||
});
|
||||
|
||||
await scanStatusJsonFast({}, {} as never);
|
||||
await scanStatusJsonFast(createStatusGatewayProbeBudget(), {} as never);
|
||||
|
||||
expect(mocks.ensurePluginRegistryLoaded).not.toHaveBeenCalled();
|
||||
expect(loggingStateRef.forceConsoleToStderr).toBe(false);
|
||||
|
|
@ -335,7 +347,7 @@ describe("scanStatusJsonFast", () => {
|
|||
url: "ws://127.0.0.1:18789",
|
||||
config: resolvedConfig,
|
||||
auth: {},
|
||||
timeoutMs: 1000,
|
||||
timeoutMs: 60_000,
|
||||
detailLevel: "presence",
|
||||
});
|
||||
expect(probeArgs.env).toBe(process.env);
|
||||
|
|
@ -356,7 +368,7 @@ describe("scanStatusJsonFast", () => {
|
|||
NODE_ENV: undefined,
|
||||
},
|
||||
async () => {
|
||||
await scanStatusJsonFast({}, {} as never);
|
||||
await scanStatusJsonFast(createStatusGatewayProbeBudget(), {} as never);
|
||||
},
|
||||
);
|
||||
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import type { OpenClawConfig } from "../config/types.js";
|
|||
import type { RuntimeEnv } from "../runtime.js";
|
||||
import { createLazyImportLoader } from "../shared/lazy-promise.js";
|
||||
import { isRecord } from "../utils.js";
|
||||
import type { StatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
import { executeStatusScanFromOverview } from "./status.scan-execute.ts";
|
||||
import { collectStatusScanOverview } from "./status.scan-overview.ts";
|
||||
import type { StatusJsonScanResult } from "./status.scan-result.ts";
|
||||
|
|
@ -73,15 +74,14 @@ function hasPotentialConfiguredChannelsForStatusJson(cfg: OpenClawConfig): boole
|
|||
|
||||
/** Runs the default fast status JSON scan. */
|
||||
export async function scanStatusJsonFast(
|
||||
opts: {
|
||||
timeoutMs?: number;
|
||||
opts: StatusGatewayProbeBudget & {
|
||||
all?: boolean;
|
||||
},
|
||||
runtime: RuntimeEnv,
|
||||
): Promise<StatusJsonScanResult> {
|
||||
const online = await (await statusGatewayModuleLoader.load()).scanStatusJsonGateway(opts);
|
||||
if (online) {
|
||||
return online;
|
||||
if (online.scan) {
|
||||
return online.scan;
|
||||
}
|
||||
const overview = await collectStatusScanOverview({
|
||||
env: process.env,
|
||||
|
|
@ -95,7 +95,7 @@ export async function scanStatusJsonFast(
|
|||
fetchGitUpdate: opts.all === true,
|
||||
includeRegistryUpdate: opts.all === true,
|
||||
includeLocalStatusRpcFallback: opts.all === true,
|
||||
gatewayProbeTimeoutMs: opts.all === true ? undefined : (opts.timeoutMs ?? 1000),
|
||||
gatewaySnapshot: online.gatewaySnapshot,
|
||||
});
|
||||
const pluginCompatibility = opts.all
|
||||
? await statusScanPluginStatusModuleLoader
|
||||
|
|
|
|||
|
|
@ -7,22 +7,27 @@ import { callGateway, isImplicitLocalGatewayTarget } from "../gateway/call.js";
|
|||
import { resolveOsSummary } from "../infra/os-summary.js";
|
||||
import { resolveMemoryPluginStatus } from "../status/memory-plugin.js";
|
||||
import type { StatusSummary } from "../status/summary.js";
|
||||
import {
|
||||
resolveStatusGatewayProbeTimeoutMs,
|
||||
type StatusGatewayProbeBudget,
|
||||
} from "./status.gateway-probe-budget.js";
|
||||
import { buildStatusScanResult, type StatusJsonScanResult } from "./status.scan-result.js";
|
||||
import {
|
||||
buildColdStartStatusSummary,
|
||||
createStatusScanCoreBootstrap,
|
||||
} from "./status.scan.bootstrap-shared.js";
|
||||
import { resolveGatewayProbeSnapshot } from "./status.scan.shared.js";
|
||||
import { resolveGatewayProbeSnapshot, type GatewayProbeSnapshot } from "./status.scan.shared.js";
|
||||
|
||||
/** The running Gateway owns fleet admission and status; local discovery is the offline fallback. */
|
||||
export async function scanStatusJsonGateway(opts: {
|
||||
timeoutMs?: number;
|
||||
all?: boolean;
|
||||
}): Promise<StatusJsonScanResult | null> {
|
||||
export async function scanStatusJsonGateway(
|
||||
opts: StatusGatewayProbeBudget & {
|
||||
all?: boolean;
|
||||
},
|
||||
): Promise<{ scan?: StatusJsonScanResult; gatewaySnapshot?: GatewayProbeSnapshot }> {
|
||||
const env = process.env;
|
||||
const configPath = resolveConfigPath(env);
|
||||
if (!existsSync(configPath)) {
|
||||
return null;
|
||||
return {};
|
||||
}
|
||||
const cfg = await measureCliCommandStartup(
|
||||
"status.connection-config",
|
||||
|
|
@ -30,38 +35,39 @@ export async function scanStatusJsonGateway(opts: {
|
|||
{ env },
|
||||
);
|
||||
if (!cfg) {
|
||||
return null;
|
||||
return {};
|
||||
}
|
||||
let projectionError = "Gateway status is unavailable.";
|
||||
const [gatewaySnapshot, status] = await Promise.all([
|
||||
resolveGatewayProbeSnapshot({
|
||||
cfg,
|
||||
configPath,
|
||||
env,
|
||||
opts: {
|
||||
timeoutMs: opts.timeoutMs ?? 1000,
|
||||
localStatusRpcFallback: false,
|
||||
},
|
||||
}),
|
||||
measureCliCommandStartup(
|
||||
"status.gateway-projection",
|
||||
() =>
|
||||
callGateway<StatusSummary>({
|
||||
config: cfg,
|
||||
configPath,
|
||||
method: "status",
|
||||
params: { includeChannelSummary: false, includeCliProjection: true },
|
||||
timeoutMs: opts.timeoutMs ?? 10_000,
|
||||
}).catch((error: unknown) => {
|
||||
projectionError = error instanceof Error ? error.message : String(error);
|
||||
return null;
|
||||
}),
|
||||
{ config: cfg, env },
|
||||
),
|
||||
]);
|
||||
if (!status && !gatewaySnapshot.gatewayProbe?.ok) {
|
||||
return null;
|
||||
const gatewaySnapshot = await resolveGatewayProbeSnapshot({
|
||||
cfg,
|
||||
configPath,
|
||||
env,
|
||||
opts,
|
||||
});
|
||||
if (!gatewaySnapshot.gatewayReachable) {
|
||||
return { gatewaySnapshot };
|
||||
}
|
||||
const status = await measureCliCommandStartup(
|
||||
"status.gateway-projection",
|
||||
() => {
|
||||
const timeoutMs = resolveStatusGatewayProbeTimeoutMs(opts);
|
||||
if (timeoutMs === 0) {
|
||||
projectionError = "Gateway probe budget exhausted before status projection.";
|
||||
return Promise.resolve(null);
|
||||
}
|
||||
return callGateway<StatusSummary>({
|
||||
config: cfg,
|
||||
configPath,
|
||||
method: "status",
|
||||
params: { includeChannelSummary: false, includeCliProjection: true },
|
||||
timeoutMs,
|
||||
}).catch((error: unknown) => {
|
||||
projectionError = error instanceof Error ? error.message : String(error);
|
||||
return null;
|
||||
});
|
||||
},
|
||||
{ config: cfg, env },
|
||||
);
|
||||
const { cliProjection, ...summary }: StatusSummary = status ?? buildColdStartStatusSummary();
|
||||
if (status) {
|
||||
gatewaySnapshot.gatewayReachable = true;
|
||||
|
|
@ -128,7 +134,7 @@ export async function scanStatusJsonGateway(opts: {
|
|||
bootstrap.updatePromise,
|
||||
]);
|
||||
const conflict = resolveGatewayAuthTokenSourceConflict({ cfg, env });
|
||||
return buildStatusScanResult({
|
||||
const scan = buildStatusScanResult({
|
||||
env,
|
||||
cfg: statusConfig,
|
||||
sourceConfig: cfg,
|
||||
|
|
@ -219,4 +225,5 @@ export async function scanStatusJsonGateway(opts: {
|
|||
],
|
||||
},
|
||||
});
|
||||
return { scan };
|
||||
}
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ import {
|
|||
} from "../gateway/minimal-gateway.test-helpers.js";
|
||||
import { racePromiseWithAbortSignal } from "../infra/abort-signal.js";
|
||||
import { createUnreachableGatewayProbe } from "./gateway-status/test-support.js";
|
||||
import { createStatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
import {
|
||||
buildTailscaleHttpsUrl,
|
||||
resolveGatewayProbeSnapshot as resolveGatewayProbeSnapshotOwner,
|
||||
|
|
@ -35,6 +36,7 @@ const resolveGatewayProbeSnapshot = (
|
|||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
cleanupTempDirs(tempDirs);
|
||||
});
|
||||
|
||||
|
|
@ -42,6 +44,7 @@ const mocks = vi.hoisted(() => ({
|
|||
buildGatewayConnectionDetailsWithResolvers: vi.fn(),
|
||||
resolveGatewayProbeTarget: vi.fn(),
|
||||
probeGateway: vi.fn(),
|
||||
waitForGatewayDiagnosticReadiness: vi.fn(),
|
||||
callGateway: vi.fn(),
|
||||
resolveGatewayProbeAuthResolution: vi.fn(),
|
||||
pickGatewaySelfPresence: vi.fn(),
|
||||
|
|
@ -109,6 +112,10 @@ vi.mock("../gateway/probe.js", () => ({
|
|||
probeGateway: mocks.probeGateway,
|
||||
}));
|
||||
|
||||
vi.mock("../cli/daemon-cli/diagnostic-readiness.js", () => ({
|
||||
waitForGatewayDiagnosticReadiness: mocks.waitForGatewayDiagnosticReadiness,
|
||||
}));
|
||||
|
||||
vi.mock("../gateway/call.js", () => ({
|
||||
callGateway: mocks.callGateway,
|
||||
}));
|
||||
|
|
@ -124,6 +131,8 @@ vi.mock("./gateway-presence.js", () => ({
|
|||
describe("resolveGatewayProbeSnapshot", () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
mocks.waitForGatewayDiagnosticReadiness.mockReset();
|
||||
mocks.buildGatewayConnectionDetailsWithResolvers.mockReturnValue({
|
||||
url: "ws://127.0.0.1:18789",
|
||||
urlSource: "local loopback",
|
||||
|
|
@ -145,7 +154,7 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
it("skips auth resolution and probe for missing remote urls by default", async () => {
|
||||
const result = await resolveGatewayProbeSnapshot({
|
||||
cfg: {},
|
||||
opts: {},
|
||||
opts: createStatusGatewayProbeBudget(),
|
||||
});
|
||||
|
||||
expect(mocks.resolveGatewayProbeAuthResolution).not.toHaveBeenCalled();
|
||||
|
|
@ -165,6 +174,83 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
});
|
||||
});
|
||||
|
||||
it.each(["healthy", "plugin-errors", "channel-errors"])(
|
||||
"waits through a 22-second cold start before probing the selected Gateway (%s)",
|
||||
async (waitOutcome) => {
|
||||
vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout", "Date"] });
|
||||
try {
|
||||
mocks.resolveGatewayProbeTarget.mockReturnValue({
|
||||
gatewayMode: "local",
|
||||
remoteUrlMissing: false,
|
||||
});
|
||||
const startedAt = Date.now();
|
||||
mocks.waitForGatewayDiagnosticReadiness.mockImplementation(async () => {
|
||||
await new Promise((resolve) => {
|
||||
setTimeout(resolve, 22_000);
|
||||
});
|
||||
vi.spyOn(performance, "now").mockReturnValue(22_000);
|
||||
return { healthy: waitOutcome === "healthy", elapsedMs: 22_000, waitOutcome };
|
||||
});
|
||||
mocks.probeGateway.mockImplementation(async () => ({
|
||||
...createUnreachableGatewayProbe("ws://127.0.0.1:18789", "timeout"),
|
||||
ok: Date.now() - startedAt >= 22_000,
|
||||
}));
|
||||
const pending = resolveGatewayProbeSnapshot({
|
||||
cfg: {},
|
||||
opts: createStatusGatewayProbeBudget(),
|
||||
});
|
||||
await vi.advanceTimersByTimeAsync(22_000);
|
||||
const result = await pending;
|
||||
|
||||
expect(result.gatewayReachable).toBe(true);
|
||||
expect(readProbeCall()).toMatchObject({
|
||||
timeoutMs: 38_000,
|
||||
auth: { token: "tok", password: "pw" },
|
||||
});
|
||||
expect(mocks.waitForGatewayDiagnosticReadiness).toHaveBeenCalledWith({
|
||||
config: {},
|
||||
timeoutMs: 60_000,
|
||||
deadlineMs: 60_000,
|
||||
onProgress: undefined,
|
||||
token: "tok",
|
||||
password: "pw",
|
||||
});
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
{ waitOutcome: "still-starting", startupPhase: "plugins", error: null },
|
||||
{ waitOutcome: "timeout", startupPhase: undefined, error: "connection refused" },
|
||||
])(
|
||||
"preserves $waitOutcome without another probe budget",
|
||||
async ({ waitOutcome, startupPhase, error }) => {
|
||||
mocks.resolveGatewayProbeTarget.mockReturnValue({
|
||||
gatewayMode: "local",
|
||||
remoteUrlMissing: false,
|
||||
});
|
||||
mocks.resolveGatewayProbeAuthResolution.mockResolvedValue({ auth: {}, warning: undefined });
|
||||
mocks.waitForGatewayDiagnosticReadiness.mockResolvedValue({
|
||||
healthy: false,
|
||||
elapsedMs: 60_000,
|
||||
waitOutcome,
|
||||
startupPhase,
|
||||
probeError: error,
|
||||
});
|
||||
const result = await resolveGatewayProbeSnapshot({
|
||||
cfg: {},
|
||||
opts: createStatusGatewayProbeBudget(),
|
||||
});
|
||||
|
||||
expect(result.gatewayProbe).toMatchObject({ ok: false, error });
|
||||
expect(result.gatewayProbe?.startupPhase).toBe(startupPhase);
|
||||
expect(mocks.probeGateway).not.toHaveBeenCalled();
|
||||
expect(mocks.callGateway).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
|
||||
it("can probe the local fallback when remote url is missing", async () => {
|
||||
mocks.probeGateway.mockResolvedValue({
|
||||
ok: true,
|
||||
|
|
@ -180,6 +266,7 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
const result = await resolveGatewayProbeSnapshot({
|
||||
cfg: {},
|
||||
opts: {
|
||||
...createStatusGatewayProbeBudget(),
|
||||
detailLevel: "full",
|
||||
probeWhenRemoteUrlMissing: true,
|
||||
resolveAuthWhenRemoteUrlMissing: true,
|
||||
|
|
@ -221,7 +308,7 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
});
|
||||
const result = await resolveGatewayProbeSnapshot({
|
||||
cfg: {},
|
||||
opts: {},
|
||||
opts: createStatusGatewayProbeBudget(),
|
||||
});
|
||||
|
||||
expect(result.gatewayProbe?.error).toBe("timeout; warn");
|
||||
|
|
@ -258,7 +345,7 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
|
||||
const result = await resolveGatewayProbeSnapshot({
|
||||
cfg: {},
|
||||
opts: {},
|
||||
opts: createStatusGatewayProbeBudget(),
|
||||
});
|
||||
|
||||
expect(result.gatewayReachable).toBe(true);
|
||||
|
|
@ -278,9 +365,7 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
|
||||
const result = await resolveGatewayProbeSnapshot({
|
||||
cfg: {},
|
||||
opts: {
|
||||
timeoutMs: 8000,
|
||||
},
|
||||
opts: createStatusGatewayProbeBudget(8000),
|
||||
});
|
||||
|
||||
const gatewayCall = readGatewayCall();
|
||||
|
|
@ -317,7 +402,7 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
|
||||
const result = await resolveGatewayProbeSnapshot({
|
||||
cfg: {},
|
||||
opts: {},
|
||||
opts: createStatusGatewayProbeBudget(),
|
||||
});
|
||||
|
||||
expect(mocks.callGateway).not.toHaveBeenCalled();
|
||||
|
|
@ -337,12 +422,12 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
|
||||
await resolveGatewayProbeSnapshot({
|
||||
cfg: {},
|
||||
opts: {},
|
||||
opts: createStatusGatewayProbeBudget(),
|
||||
});
|
||||
|
||||
const probeCall = readProbeCall();
|
||||
expect(probeCall).not.toHaveProperty("preauthHandshakeTimeoutMs");
|
||||
expect(probeCall.timeoutMs).toBe(2500);
|
||||
expect(probeCall.timeoutMs).toBe(60_000);
|
||||
const gatewayCall = readGatewayCall();
|
||||
expect(gatewayCall.config).toEqual({});
|
||||
expect(gatewayCall.timeoutMs).toBe(2000);
|
||||
|
|
@ -363,7 +448,7 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
|
||||
await resolveGatewayProbeSnapshot({
|
||||
cfg: {},
|
||||
opts: { timeoutMs },
|
||||
opts: createStatusGatewayProbeBudget(timeoutMs),
|
||||
});
|
||||
|
||||
const probeCall = readProbeCall();
|
||||
|
|
@ -373,7 +458,7 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
},
|
||||
);
|
||||
|
||||
it("enforces an explicit CLI timeout against a real local fallback status RPC", async ({
|
||||
it("gives a real local fallback status RPC only the remaining explicit CLI timeout", async ({
|
||||
signal,
|
||||
}) => {
|
||||
const gateway = new WebSocketServer({ host: "127.0.0.1", port: 0 });
|
||||
|
|
@ -381,7 +466,6 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
const address = gateway.address() as AddressInfo;
|
||||
const url = `ws://127.0.0.1:${address.port}`;
|
||||
const observedMethods: string[] = [];
|
||||
const presenceRequested = createDeferred();
|
||||
const statusRequested = createDeferred();
|
||||
gateway.on("connection", (socket) => {
|
||||
sendMinimalGatewayConnectChallenge(socket);
|
||||
|
|
@ -403,9 +487,6 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
return;
|
||||
}
|
||||
observedMethods.push(frame.method);
|
||||
if (frame.method === "system-presence") {
|
||||
presenceRequested.resolve();
|
||||
}
|
||||
if (frame.method === "status") {
|
||||
statusRequested.resolve();
|
||||
const responseTimer = setTimeout(() => {
|
||||
|
|
@ -428,10 +509,9 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
gatewayMode: "local",
|
||||
remoteUrlMissing: false,
|
||||
});
|
||||
mocks.probeGateway.mockImplementation(async (...args: unknown[]) => {
|
||||
const { probeGateway } =
|
||||
await vi.importActual<typeof import("../gateway/probe.js")>("../gateway/probe.js");
|
||||
return await probeGateway(...(args as Parameters<typeof probeGateway>));
|
||||
mocks.probeGateway.mockImplementation(async () => {
|
||||
vi.spyOn(performance, "now").mockReturnValue(150);
|
||||
return createUnreachableGatewayProbe(url, "timeout");
|
||||
});
|
||||
mocks.callGateway.mockImplementation(async (...args: unknown[]) => {
|
||||
const { callGateway } =
|
||||
|
|
@ -441,7 +521,7 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
const parsed = parseStatusRouteArgs(["node", "openclaw", "status", "--timeout", "250"]);
|
||||
expect(parsed?.timeoutMs).toBe(250);
|
||||
|
||||
const startups = Array.from({ length: 2 }, () =>
|
||||
const startups = Array.from({ length: 1 }, () =>
|
||||
createDeferred<{
|
||||
completion: ReturnType<typeof clientReadiness.startGatewayClientWhenEventLoopReady>;
|
||||
}>(),
|
||||
|
|
@ -481,21 +561,18 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
try {
|
||||
pending = resolveGatewayProbeSnapshot({
|
||||
cfg: { gateway: { auth: { mode: "token", token: "tok" } } },
|
||||
opts: { timeoutMs: parsed?.timeoutMs },
|
||||
opts: createStatusGatewayProbeBudget(parsed?.timeoutMs),
|
||||
});
|
||||
|
||||
// Drive deadline time only after real socket requests cross their boundary.
|
||||
await driveStartup(startups[0]!);
|
||||
await racePromiseWithAbortSignal(presenceRequested.promise, signal);
|
||||
await vi.advanceTimersByTimeAsync(250);
|
||||
await driveStartup(startups[1]!);
|
||||
await racePromiseWithAbortSignal(statusRequested.promise, signal);
|
||||
await vi.advanceTimersByTimeAsync(250);
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
const result = await pending;
|
||||
|
||||
expect(readProbeCall().timeoutMs).toBe(250);
|
||||
expect(readGatewayCall().timeoutMs).toBe(250);
|
||||
expect(observedMethods).toEqual(["system-presence", "status"]);
|
||||
expect(readGatewayCall().timeoutMs).toBe(100);
|
||||
expect(observedMethods).toEqual(["status"]);
|
||||
expect(result.gatewayProbe?.ok).toBe(false);
|
||||
expect(result.gatewayProbe?.error).toContain("timeout");
|
||||
} finally {
|
||||
|
|
@ -537,7 +614,7 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
|
||||
const result = await resolveGatewayProbeSnapshot({
|
||||
cfg: {},
|
||||
opts: {},
|
||||
opts: createStatusGatewayProbeBudget(),
|
||||
});
|
||||
|
||||
const gatewayCall = readGatewayCall();
|
||||
|
|
@ -563,7 +640,7 @@ describe("resolveGatewayProbeSnapshot", () => {
|
|||
|
||||
const result = await resolveGatewayProbeSnapshot({
|
||||
cfg: { gateway: { mode: "remote", remote: { url: "wss://gateway.example/ws" } } },
|
||||
opts: {},
|
||||
opts: createStatusGatewayProbeBudget(),
|
||||
});
|
||||
|
||||
expect(mocks.callGateway).not.toHaveBeenCalled();
|
||||
|
|
|
|||
|
|
@ -22,10 +22,17 @@ import type { MemoryPluginStatus } from "../status/memory-plugin.js";
|
|||
import type { StatusSummary } from "../status/summary.js";
|
||||
import { pickGatewaySelfPresence } from "./gateway-presence.js";
|
||||
import { isProbeReachable } from "./gateway-status/helpers.js";
|
||||
import {
|
||||
resolveStatusGatewayProbeTimeoutMs,
|
||||
type StatusGatewayProbeBudget,
|
||||
} from "./status.gateway-probe-budget.js";
|
||||
|
||||
const gatewayProbeModuleLoader = createLazyImportLoader(() => import("./status.gateway-probe.js"));
|
||||
const probeGatewayModuleLoader = createLazyImportLoader(() => import("../gateway/probe.js"));
|
||||
const gatewayCallModuleLoader = createLazyImportLoader(() => import("../gateway/call.js"));
|
||||
const gatewayReadinessModuleLoader = createLazyImportLoader(
|
||||
() => import("../cli/daemon-cli/diagnostic-readiness.js"),
|
||||
);
|
||||
const memoryPresenceModuleLoader = createLazyImportLoader(async () => {
|
||||
const { loadBundledPluginPublicArtifactModuleSync } =
|
||||
await import("../plugins/public-surface-loader.js");
|
||||
|
|
@ -34,13 +41,6 @@ const memoryPresenceModuleLoader = createLazyImportLoader(async () => {
|
|||
}>({ dirName: "memory-core", artifactBasename: "status-api.js" });
|
||||
});
|
||||
|
||||
export function resolveStatusGatewayProbeTimeoutMs(opts: {
|
||||
timeoutMs?: number;
|
||||
all?: boolean;
|
||||
}): number {
|
||||
return opts.timeoutMs ?? (opts.all ? 5000 : 2500);
|
||||
}
|
||||
|
||||
function loadGatewayProbeModule() {
|
||||
return gatewayProbeModuleLoader.load();
|
||||
}
|
||||
|
|
@ -133,7 +133,7 @@ async function applyLocalStatusRpcFallback(params: {
|
|||
password?: string;
|
||||
};
|
||||
timeoutMs: number;
|
||||
timeoutMsExplicit: boolean;
|
||||
gatewayProbeDeadlineMs: number;
|
||||
enabled?: boolean;
|
||||
}): Promise<GatewayProbeResult | null> {
|
||||
if (params.enabled === false) {
|
||||
|
|
@ -142,25 +142,24 @@ async function applyLocalStatusRpcFallback(params: {
|
|||
if (!shouldTryLocalStatusRpcFallback(params)) {
|
||||
return params.gatewayProbe;
|
||||
}
|
||||
// Explicit probe budgets are operator-owned; only implicit fallback defaults get a floor.
|
||||
const boundedFallbackTimeoutMs = Math.min(
|
||||
2000,
|
||||
params.timeoutMsExplicit ? params.timeoutMs : Math.max(1000, params.timeoutMs),
|
||||
);
|
||||
// The fallback uses the gateway status RPC because it can succeed after probe handshake ambiguity.
|
||||
const status = await loadGatewayCallModule()
|
||||
.then(({ callGateway }) =>
|
||||
callGateway<Partial<StatusSummary>>({
|
||||
.then(({ callGateway }) => {
|
||||
const timeoutMs = Math.min(2000, resolveStatusGatewayProbeTimeoutMs(params));
|
||||
if (timeoutMs === 0) {
|
||||
return null;
|
||||
}
|
||||
return callGateway<Partial<StatusSummary>>({
|
||||
config: params.cfg,
|
||||
configPath: params.configPath,
|
||||
method: "status",
|
||||
token: params.gatewayProbeAuth.token,
|
||||
password: params.gatewayProbeAuth.password,
|
||||
timeoutMs: boundedFallbackTimeoutMs,
|
||||
timeoutMs,
|
||||
mode: GATEWAY_CLIENT_MODES.BACKEND,
|
||||
clientName: GATEWAY_CLIENT_NAMES.GATEWAY_CLIENT,
|
||||
}),
|
||||
)
|
||||
});
|
||||
})
|
||||
.catch(() => null);
|
||||
if (!status) {
|
||||
return params.gatewayProbe;
|
||||
|
|
@ -201,8 +200,7 @@ export async function resolveGatewayProbeSnapshot(params: {
|
|||
cfg: OpenClawConfig;
|
||||
configPath: string;
|
||||
env: NodeJS.ProcessEnv;
|
||||
opts: {
|
||||
timeoutMs?: number;
|
||||
opts: StatusGatewayProbeBudget & {
|
||||
all?: boolean;
|
||||
skipProbe?: boolean;
|
||||
detailLevel?: "none" | "presence" | "full";
|
||||
|
|
@ -210,6 +208,7 @@ export async function resolveGatewayProbeSnapshot(params: {
|
|||
resolveAuthWhenRemoteUrlMissing?: boolean;
|
||||
mergeAuthWarningIntoProbeError?: boolean;
|
||||
localStatusRpcFallback?: boolean;
|
||||
onProgress?: (phase: string) => void;
|
||||
};
|
||||
}): Promise<GatewayProbeSnapshot> {
|
||||
const gatewayConnection = buildGatewayConnectionDetailsWithResolvers({
|
||||
|
|
@ -229,22 +228,64 @@ export async function resolveGatewayProbeSnapshot(params: {
|
|||
)
|
||||
: { auth: {}, warning: undefined };
|
||||
let gatewayProbeAuthWarning = gatewayProbeAuthResolution.warning;
|
||||
const timeoutMsExplicit = params.opts.timeoutMs !== undefined;
|
||||
const probeTimeoutMs = resolveStatusGatewayProbeTimeoutMs(params.opts);
|
||||
const initialGatewayProbe = shouldProbe
|
||||
? await loadProbeGatewayModule()
|
||||
.then(({ probeGateway }) =>
|
||||
probeGateway({
|
||||
url: gatewayConnection.url,
|
||||
const remainingTimeoutMs = () => resolveStatusGatewayProbeTimeoutMs(params.opts);
|
||||
const readiness =
|
||||
shouldProbe && remainingTimeoutMs() > 0
|
||||
? await gatewayReadinessModuleLoader.load().then(({ waitForGatewayDiagnosticReadiness }) =>
|
||||
waitForGatewayDiagnosticReadiness({
|
||||
config: params.cfg,
|
||||
auth: gatewayProbeAuthResolution.auth,
|
||||
env: params.env,
|
||||
timeoutMs: probeTimeoutMs,
|
||||
detailLevel: params.opts.detailLevel ?? "presence",
|
||||
timeoutMs: remainingTimeoutMs(),
|
||||
deadlineMs: params.opts.gatewayProbeDeadlineMs,
|
||||
onProgress: params.opts.onProgress,
|
||||
...gatewayProbeAuthResolution.auth,
|
||||
}),
|
||||
)
|
||||
.catch(() => null)
|
||||
: null;
|
||||
: undefined;
|
||||
const canDiagnose =
|
||||
readiness?.healthy ||
|
||||
readiness?.waitOutcome === "plugin-errors" ||
|
||||
readiness?.waitOutcome === "channel-errors";
|
||||
const unavailableProbe = (): GatewayProbeResult => ({
|
||||
ok: false,
|
||||
url: gatewayConnection.url,
|
||||
connectLatencyMs: null,
|
||||
error:
|
||||
readiness?.waitOutcome === "still-starting"
|
||||
? null
|
||||
: (readiness?.probeError ??
|
||||
(remainingTimeoutMs() === 0
|
||||
? "Gateway probe budget exhausted."
|
||||
: "Gateway is unreachable")),
|
||||
...(readiness?.waitOutcome === "still-starting"
|
||||
? { startupPhase: readiness.startupPhase ?? "startup" }
|
||||
: {}),
|
||||
auth: { role: null, scopes: [], capability: "unknown" },
|
||||
close: null,
|
||||
health: null,
|
||||
status: null,
|
||||
presence: null,
|
||||
configSnapshot: null,
|
||||
});
|
||||
const initialGatewayProbe: GatewayProbeResult | null =
|
||||
(readiness && !canDiagnose) || (shouldProbe && remainingTimeoutMs() === 0)
|
||||
? unavailableProbe()
|
||||
: shouldProbe
|
||||
? await loadProbeGatewayModule()
|
||||
.then(({ probeGateway }) => {
|
||||
const timeoutMs = remainingTimeoutMs();
|
||||
return timeoutMs === 0
|
||||
? unavailableProbe()
|
||||
: probeGateway({
|
||||
url: gatewayConnection.url,
|
||||
config: params.cfg,
|
||||
auth: gatewayProbeAuthResolution.auth,
|
||||
env: params.env,
|
||||
timeoutMs,
|
||||
detailLevel: params.opts.detailLevel ?? "presence",
|
||||
});
|
||||
})
|
||||
.catch(() => null)
|
||||
: null;
|
||||
const gatewayProbe = await applyLocalStatusRpcFallback({
|
||||
cfg: params.cfg,
|
||||
configPath: params.configPath,
|
||||
|
|
@ -252,14 +293,18 @@ export async function resolveGatewayProbeSnapshot(params: {
|
|||
gatewayUrl: gatewayConnection.url,
|
||||
gatewayProbe: initialGatewayProbe,
|
||||
gatewayProbeAuth: gatewayProbeAuthResolution.auth,
|
||||
timeoutMs: probeTimeoutMs,
|
||||
timeoutMsExplicit,
|
||||
enabled: params.opts.localStatusRpcFallback !== false,
|
||||
timeoutMs: remainingTimeoutMs(),
|
||||
gatewayProbeDeadlineMs: params.opts.gatewayProbeDeadlineMs,
|
||||
enabled:
|
||||
params.opts.localStatusRpcFallback !== false &&
|
||||
remainingTimeoutMs() > 0 &&
|
||||
(!readiness || canDiagnose),
|
||||
});
|
||||
if (
|
||||
(params.opts.mergeAuthWarningIntoProbeError ?? true) &&
|
||||
gatewayProbeAuthWarning &&
|
||||
gatewayProbe?.ok === false
|
||||
gatewayProbe?.ok === false &&
|
||||
!gatewayProbe.startupPhase
|
||||
) {
|
||||
gatewayProbe.error = gatewayProbe.error
|
||||
? `${gatewayProbe.error}; ${gatewayProbeAuthWarning}`
|
||||
|
|
|
|||
|
|
@ -293,6 +293,9 @@ export async function loadStatusScanModuleForTest(
|
|||
vi.doMock("../gateway/probe.js", () => ({
|
||||
probeGateway: mocks.probeGateway,
|
||||
}));
|
||||
vi.doMock("../cli/daemon-cli/diagnostic-readiness.js", () => ({
|
||||
waitForGatewayDiagnosticReadiness: async () => undefined,
|
||||
}));
|
||||
vi.doMock("../gateway/probe-target.js", () => ({
|
||||
resolveGatewayProbeTarget: mocks.resolveGatewayProbeTarget,
|
||||
}));
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
// Status scan tests cover fast scan defaults, memory setup, gateway probes, and status aggregation.
|
||||
import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { createStatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
import {
|
||||
applyStatusScanDefaults,
|
||||
createStatusMemorySearchConfig,
|
||||
|
|
@ -29,12 +30,14 @@ beforeAll(async () => {
|
|||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
configureScanStatus();
|
||||
originalForceStderr = loggingStateRef.forceConsoleToStderr;
|
||||
loggingStateRef.forceConsoleToStderr = false;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
loggingStateRef.forceConsoleToStderr = originalForceStderr;
|
||||
});
|
||||
|
||||
|
|
@ -132,7 +135,7 @@ describe("scanStatus", () => {
|
|||
);
|
||||
}
|
||||
|
||||
const result = await scanStatus({});
|
||||
const result = await scanStatus(createStatusGatewayProbeBudget());
|
||||
|
||||
expect(result.summary.heartbeat).toEqual({
|
||||
defaultAgentId: "main",
|
||||
|
|
@ -157,7 +160,7 @@ describe("scanStatus", () => {
|
|||
summary: createStatusSummary({ linkChannel: { linked: false } }),
|
||||
});
|
||||
|
||||
await scanStatus({});
|
||||
await scanStatus(createStatusGatewayProbeBudget());
|
||||
|
||||
expect(mocks.getStatusSummary).toHaveBeenCalledWith({
|
||||
config: resolvedConfig,
|
||||
|
|
@ -195,7 +198,7 @@ describe("scanStatus", () => {
|
|||
configSnapshot: null,
|
||||
});
|
||||
|
||||
await scanStatus({});
|
||||
await scanStatus(createStatusGatewayProbeBudget());
|
||||
|
||||
expect(
|
||||
mocks.callGateway.mock.calls.some(([call]) => {
|
||||
|
|
@ -224,10 +227,10 @@ describe("scanStatus", () => {
|
|||
it("bounds gateway secret resolution by the fast status probe budget", async () => {
|
||||
configureScanStatus();
|
||||
|
||||
await scanStatus({});
|
||||
await scanStatus(createStatusGatewayProbeBudget());
|
||||
|
||||
expect(mocks.resolveCommandSecretRefsViaGateway).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ gatewaySecretResolveTimeoutMs: 2500 }),
|
||||
expect.objectContaining({ gatewaySecretResolveTimeoutMs: 60_000 }),
|
||||
);
|
||||
});
|
||||
|
||||
|
|
@ -258,7 +261,7 @@ describe("scanStatus", () => {
|
|||
configSnapshot: null,
|
||||
});
|
||||
|
||||
await scanStatus({ deep: true, timeoutMs: 5000 });
|
||||
await scanStatus({ ...createStatusGatewayProbeBudget(5000), deep: true });
|
||||
|
||||
expect(mocks.callGateway).toHaveBeenCalledTimes(2);
|
||||
expect(
|
||||
|
|
@ -271,7 +274,7 @@ describe("scanStatus", () => {
|
|||
probe: false,
|
||||
timeoutMs: 5000,
|
||||
},
|
||||
timeoutMs: 2500,
|
||||
timeoutMs: 5000,
|
||||
});
|
||||
expect(mocks.buildChannelsTable).toHaveBeenCalledOnce();
|
||||
expect(firstBuildChannelsTableCall()).toStrictEqual([
|
||||
|
|
@ -297,7 +300,7 @@ describe("scanStatus", () => {
|
|||
gatewayProbe: false,
|
||||
});
|
||||
|
||||
await scanStatus({});
|
||||
await scanStatus(createStatusGatewayProbeBudget());
|
||||
|
||||
expect(mocks.getUpdateCheckResult).not.toHaveBeenCalled();
|
||||
expect(mocks.probeGateway).not.toHaveBeenCalled();
|
||||
|
|
@ -306,7 +309,7 @@ describe("scanStatus", () => {
|
|||
it("keeps default text status off plugin compatibility and memory scans", async () => {
|
||||
configureScanStatus({ memoryConfigured: true });
|
||||
|
||||
await scanStatus({});
|
||||
await scanStatus(createStatusGatewayProbeBudget());
|
||||
|
||||
expect(mocks.buildPluginCompatibilityNotices).not.toHaveBeenCalled();
|
||||
expect(mocks.getMemorySearchManager).not.toHaveBeenCalled();
|
||||
|
|
|
|||
|
|
@ -2,15 +2,17 @@
|
|||
// Human `status --all` and JSON status use their dedicated command paths.
|
||||
|
||||
import { withProgress } from "../cli/progress.js";
|
||||
import type { StatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
import { executeStatusScanFromOverview } from "./status.scan-execute.ts";
|
||||
import { collectStatusScanOverview } from "./status.scan-overview.ts";
|
||||
import type { StatusScanResult } from "./status.scan-result.ts";
|
||||
|
||||
/** Runs the text status scan. */
|
||||
export async function scanStatus(opts: {
|
||||
timeoutMs?: number;
|
||||
deep?: boolean;
|
||||
}): Promise<StatusScanResult> {
|
||||
export async function scanStatus(
|
||||
opts: StatusGatewayProbeBudget & {
|
||||
deep?: boolean;
|
||||
},
|
||||
): Promise<StatusScanResult> {
|
||||
return await withProgress(
|
||||
{
|
||||
label: "Scanning status…",
|
||||
|
|
|
|||
|
|
@ -237,7 +237,7 @@ describe("getStatusSummary read-only session access", () => {
|
|||
const timeline = state.path("status-timeline.jsonl");
|
||||
const scan = await withEnvAsync(
|
||||
{ OPENCLAW_DIAGNOSTICS: "timeline", OPENCLAW_DIAGNOSTICS_TIMELINE_PATH: timeline },
|
||||
() => scanStatus({ timeoutMs: 100 }),
|
||||
() => scanStatus({ timeoutMs: 100, gatewayProbeDeadlineMs: performance.now() + 100 }),
|
||||
);
|
||||
expect(scan.agentStatus.totalSessions).toBe(12);
|
||||
expect(scan.agentStatus.agents[0]?.lastUpdatedAt).toBe(12);
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import type { PluginCompatibilityNotice } from "../plugins/status.js";
|
|||
import { createCompatibilityNotice } from "../plugins/status.test-fixtures.js";
|
||||
import { createEmptyTaskRegistrySummary } from "../tasks/task-registry.summary.js";
|
||||
import { captureEnv, deleteTestEnvValue, setTestEnvValue } from "../test-utils/env.js";
|
||||
import { createErrorChannelPlugin } from "./status.channel-plugin.test-helpers.js";
|
||||
import type { StatusScanResult } from "./status.scan-result.js";
|
||||
import { createTestRuntime } from "./test-runtime-config-helpers.js";
|
||||
|
||||
|
|
@ -51,37 +52,6 @@ function createUnknownUsageSessionStore() {
|
|||
};
|
||||
}
|
||||
|
||||
function createChannelIssueCollector(channel: string) {
|
||||
return (accounts: Array<Record<string, unknown>>) =>
|
||||
accounts
|
||||
.filter((account) => typeof account.lastError === "string" && account.lastError)
|
||||
.map((account) => ({
|
||||
channel,
|
||||
accountId: typeof account.accountId === "string" ? account.accountId : "default",
|
||||
message: `Channel error: ${String(account.lastError)}`,
|
||||
}));
|
||||
}
|
||||
|
||||
function createErrorChannelPlugin(params: { id: string; label: string; docsPath: string }) {
|
||||
return {
|
||||
id: params.id,
|
||||
meta: {
|
||||
id: params.id,
|
||||
label: params.label,
|
||||
selectionLabel: params.label,
|
||||
docsPath: params.docsPath,
|
||||
blurb: "mock",
|
||||
},
|
||||
config: {
|
||||
listAccountIds: () => ["default"],
|
||||
resolveAccount: () => ({}),
|
||||
},
|
||||
status: {
|
||||
collectStatusIssues: createChannelIssueCollector(params.id),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function withUnknownUsageStore(run: () => Promise<void>) {
|
||||
mocks.loadSessionStore.mockReturnValue(createUnknownUsageSessionStore());
|
||||
await run();
|
||||
|
|
@ -920,6 +890,7 @@ vi.mock("./status.daemon.js", () => ({
|
|||
|
||||
describe("statusCommand", () => {
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
mocks.loadConfig.mockReset();
|
||||
mocks.loadConfig.mockReturnValue({ session: {} });
|
||||
mocks.loadSessionStore.mockReset();
|
||||
|
|
@ -1067,6 +1038,7 @@ describe("statusCommand", () => {
|
|||
});
|
||||
|
||||
it("scopes usage resolution to the scanned config", async () => {
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
const snapshotMock = resolveStatusRuntimeSnapshot as Mock;
|
||||
const usageMock = resolveStatusUsageSummary as Mock;
|
||||
snapshotMock.mockClear();
|
||||
|
|
@ -1078,21 +1050,29 @@ describe("statusCommand", () => {
|
|||
| {
|
||||
config: unknown;
|
||||
timeoutMs?: number;
|
||||
gatewayProbeDeadlineMs: number;
|
||||
usage?: boolean;
|
||||
resolveUsage?: (input: { config: unknown; timeoutMs?: number }) => Promise<unknown>;
|
||||
resolveUsage?: (input: {
|
||||
config: unknown;
|
||||
timeoutMs?: number;
|
||||
gatewayProbeDeadlineMs: number;
|
||||
}) => Promise<unknown>;
|
||||
}
|
||||
| undefined;
|
||||
expect(params?.usage).toBe(true);
|
||||
expect(params?.timeoutMs).toBe(1234);
|
||||
expect(params?.gatewayProbeDeadlineMs).toBe(1234);
|
||||
if (!params?.resolveUsage) {
|
||||
throw new Error("missing status usage resolver");
|
||||
}
|
||||
await params.resolveUsage({
|
||||
timeoutMs: 1234,
|
||||
gatewayProbeDeadlineMs: 1234,
|
||||
config: params.config,
|
||||
});
|
||||
expect(usageMock).toHaveBeenCalledWith({
|
||||
timeoutMs: 1234,
|
||||
gatewayProbeDeadlineMs: 1234,
|
||||
config: params?.config,
|
||||
});
|
||||
});
|
||||
|
|
@ -1157,12 +1137,17 @@ describe("statusCommand", () => {
|
|||
});
|
||||
|
||||
it("passes deep mode through to the text status scan", async () => {
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
const { scanStatus } = await import("./status.scan.js");
|
||||
vi.mocked(scanStatus).mockClear();
|
||||
|
||||
await statusCommand({ deep: true, timeoutMs: 5000 }, runtime);
|
||||
|
||||
expect(scanStatus).toHaveBeenCalledWith({ timeoutMs: 5000, deep: true });
|
||||
expect(scanStatus).toHaveBeenCalledWith({
|
||||
timeoutMs: 5000,
|
||||
gatewayProbeDeadlineMs: 5000,
|
||||
deep: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("surfaces unknown usage when totalTokens is missing", async () => {
|
||||
|
|
|
|||
|
|
@ -1,16 +1,27 @@
|
|||
import { Command } from "commander";
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { registerStatusHealthSessionsCommands } from "../cli/program/register.status-health-sessions.js";
|
||||
import type { OpenClawConfig } from "../config/types.js";
|
||||
import type { UsageSummary } from "../infra/provider-usage.types.js";
|
||||
import { defaultRuntime } from "../runtime.js";
|
||||
import { createUnreachableGatewayProbe } from "./gateway-status/test-support.js";
|
||||
import { statusJsonCommand } from "./status-json.js";
|
||||
import type { StatusUsageSummaryOptions } from "./status-usage.runtime.js";
|
||||
import { createStatusGatewayProbeBudget } from "./status.gateway-probe-budget.js";
|
||||
import type { StatusScanOverviewResult } from "./status.scan-overview.js";
|
||||
import type { StatusScanResult } from "./status.scan-result.js";
|
||||
import type { scanStatus } from "./status.scan.js";
|
||||
import { resolveGatewayProbeSnapshot } from "./status.scan.shared.js";
|
||||
import { baseStatusServices, createStatusScanResultFixture } from "./status.test-support.js";
|
||||
|
||||
const mocks = vi.hoisted(() => ({
|
||||
scan: vi.fn<() => Promise<StatusScanResult>>(),
|
||||
scan: vi.fn<(opts: Parameters<typeof scanStatus>[0]) => Promise<StatusScanResult>>(),
|
||||
usage: vi.fn<(options: StatusUsageSummaryOptions) => Promise<UsageSummary>>(),
|
||||
probe: vi.fn<typeof import("../gateway/probe.js").probeGateway>(),
|
||||
callGateway:
|
||||
vi.fn<
|
||||
(params: Parameters<typeof import("../gateway/call.js").callGateway>[0]) => Promise<unknown>
|
||||
>(),
|
||||
gatewayService: vi.fn(),
|
||||
nodeService: vi.fn(),
|
||||
runtime: { log: vi.fn(), error: vi.fn(), exit: vi.fn() },
|
||||
|
|
@ -23,9 +34,9 @@ vi.mock("../runtime.js", async (importOriginal) => ({
|
|||
vi.mock("./status.scan.js", () => ({ scanStatus: mocks.scan }));
|
||||
vi.mock("./status.scan.fast-json.js", () => ({ scanStatusJsonFast: mocks.scan }));
|
||||
vi.mock("./status.scan-overview.js", () => ({
|
||||
resolveStatusSummaryFromOverview: async () => (await mocks.scan()).summary,
|
||||
collectStatusScanOverview: async () => {
|
||||
const scan = await mocks.scan();
|
||||
resolveStatusSummaryFromOverview: async () => createStatusScanResultFixture().summary,
|
||||
collectStatusScanOverview: async ({ opts }: { opts: Parameters<typeof scanStatus>[0] }) => {
|
||||
const scan = await mocks.scan(opts);
|
||||
return {
|
||||
...scan,
|
||||
coldStart: false,
|
||||
|
|
@ -49,6 +60,14 @@ vi.mock("./status.scan-overview.js", () => ({
|
|||
vi.mock("./status-usage.runtime.js", () => ({
|
||||
resolveStatusUsageSummary: mocks.usage,
|
||||
}));
|
||||
vi.mock("../gateway/probe.js", () => ({ probeGateway: mocks.probe }));
|
||||
vi.mock("../gateway/call.js", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("../gateway/call.js")>()),
|
||||
callGateway: mocks.callGateway,
|
||||
}));
|
||||
vi.mock("./status.gateway-probe.js", () => ({
|
||||
resolveGatewayProbeAuthResolution: async () => ({ auth: { token: "fixture-token" } }),
|
||||
}));
|
||||
vi.mock("./status.daemon.js", () => ({
|
||||
getDaemonStatusSummary: mocks.gatewayService,
|
||||
getNodeDaemonStatusSummary: mocks.nodeService,
|
||||
|
|
@ -133,6 +152,7 @@ const summaries = {
|
|||
describe("status usage routing through Commander", () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
const scan = createStatusScanResultFixture({
|
||||
cfg: config,
|
||||
sourceConfig: config,
|
||||
|
|
@ -163,6 +183,80 @@ describe("status usage routing through Commander", () => {
|
|||
mocks.nodeService.mockResolvedValue(baseStatusServices.nodeService);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it.each(["text", "commander-json", "fast-json"])(
|
||||
"shares the command deadline across remote usage and deep probes (%s)",
|
||||
async (mode) => {
|
||||
const clock = vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
try {
|
||||
const scan = await mocks.scan(createStatusGatewayProbeBudget());
|
||||
const remoteConfig: OpenClawConfig = {
|
||||
...config,
|
||||
gateway: { mode: "remote", remote: { url: "wss://gateway.example.com" } },
|
||||
};
|
||||
mocks.scan.mockImplementation(async (opts) => ({
|
||||
...scan,
|
||||
cfg: remoteConfig,
|
||||
sourceConfig: remoteConfig,
|
||||
...(await resolveGatewayProbeSnapshot({
|
||||
cfg: remoteConfig,
|
||||
configPath: "/tmp/status-usage-fixture.json",
|
||||
env: {},
|
||||
opts,
|
||||
})),
|
||||
}));
|
||||
mocks.probe.mockImplementation(async () => {
|
||||
clock.mockReturnValue(22_000);
|
||||
return {
|
||||
...createUnreachableGatewayProbe("wss://gateway.example.com", "fixture"),
|
||||
ok: true,
|
||||
error: null,
|
||||
};
|
||||
});
|
||||
mocks.usage.mockImplementation(async () => {
|
||||
clock.mockReturnValue(30_000);
|
||||
return summaries.default;
|
||||
});
|
||||
mocks.callGateway.mockImplementation(async ({ method }) => {
|
||||
if (method === "health") {
|
||||
clock.mockReturnValue(35_000);
|
||||
return { ok: true, channels: {}, agents: [], ts: 1, durationMs: 0 };
|
||||
}
|
||||
return null;
|
||||
});
|
||||
if (mode === "fast-json") {
|
||||
await statusJsonCommand({ usage: true, deep: true }, defaultRuntime);
|
||||
} else {
|
||||
const program = new Command();
|
||||
registerStatusHealthSessionsCommands(program);
|
||||
await program.parseAsync(
|
||||
["status", "--usage", "--deep", ...(mode === "commander-json" ? ["--json"] : [])],
|
||||
{ from: "user" },
|
||||
);
|
||||
}
|
||||
|
||||
expect(mocks.runtime.error).not.toHaveBeenCalled();
|
||||
expect(mocks.probe).toHaveBeenCalledWith(expect.objectContaining({ timeoutMs: 60_000 }));
|
||||
expect(mocks.usage).toHaveBeenCalledExactlyOnceWith({
|
||||
config: remoteConfig,
|
||||
timeoutMs: 38_000,
|
||||
gatewayProbeDeadlineMs: 60_000,
|
||||
});
|
||||
expect(
|
||||
mocks.callGateway.mock.calls.map(([input]) => [input.method, input.timeoutMs]),
|
||||
).toEqual([
|
||||
["health", 30_000],
|
||||
["last-heartbeat", 25_000],
|
||||
]);
|
||||
} finally {
|
||||
clock.mockRestore();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
{
|
||||
name: "plain usage",
|
||||
|
|
@ -230,11 +324,50 @@ describe("status usage routing through Commander", () => {
|
|||
expect(mocks.usage).toHaveBeenCalledOnce();
|
||||
expect(mocks.usage).toHaveBeenCalledWith({
|
||||
config,
|
||||
timeoutMs: 10_000,
|
||||
timeoutMs: 60_000,
|
||||
gatewayProbeDeadlineMs: 60_000,
|
||||
...(agentId ? { agentId } : {}),
|
||||
});
|
||||
} else {
|
||||
expect(mocks.usage).not.toHaveBeenCalled();
|
||||
}
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ args: [], timeoutMs: undefined, elapsedMs: 22_000, expected: 38_000 },
|
||||
{ args: ["--all"], timeoutMs: undefined, elapsedMs: 22_000, expected: 38_000 },
|
||||
{ args: ["--json", "--all"], timeoutMs: undefined, elapsedMs: 22_000, expected: 38_000 },
|
||||
{ args: [], timeoutMs: 1234, elapsedMs: 234, expected: 1000 },
|
||||
{ args: ["--all"], timeoutMs: 1234, elapsedMs: 234, expected: 1000 },
|
||||
{ args: ["--json", "--all"], timeoutMs: 1234, elapsedMs: 234, expected: 1000 },
|
||||
])(
|
||||
"bounds usage after readiness ($args, $timeoutMs)",
|
||||
async ({ args, timeoutMs, elapsedMs, expected }) => {
|
||||
const clock = vi.spyOn(performance, "now").mockReturnValue(0);
|
||||
try {
|
||||
const scan = await mocks.scan(createStatusGatewayProbeBudget(timeoutMs));
|
||||
mocks.scan.mockImplementation(async () => {
|
||||
clock.mockReturnValue(elapsedMs);
|
||||
return scan;
|
||||
});
|
||||
mocks.usage.mockResolvedValue(summaries.default);
|
||||
const program = new Command();
|
||||
registerStatusHealthSessionsCommands(program);
|
||||
await program.parseAsync(
|
||||
["status", "--usage", ...args, ...(timeoutMs ? ["--timeout", String(timeoutMs)] : [])],
|
||||
{ from: "user" },
|
||||
);
|
||||
|
||||
expect(mocks.runtime.error).not.toHaveBeenCalled();
|
||||
expect(mocks.runtime.exit).not.toHaveBeenCalled();
|
||||
expect(mocks.usage).toHaveBeenCalledExactlyOnceWith({
|
||||
config,
|
||||
timeoutMs: expected,
|
||||
gatewayProbeDeadlineMs: timeoutMs ?? 60_000,
|
||||
});
|
||||
} finally {
|
||||
clock.mockRestore();
|
||||
}
|
||||
},
|
||||
);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -436,11 +436,9 @@ async function runGatewayHealthChecks(ctx: DoctorHealthFlowContext): Promise<voi
|
|||
}
|
||||
const { checkGatewayHealth, probeGatewayMemoryStatus } =
|
||||
await import("../commands/doctor-gateway-health.js");
|
||||
const timeoutMs = ctx.options.nonInteractive === true ? 3000 : 10_000;
|
||||
const { healthOk, authenticated, status } = await checkGatewayHealth({
|
||||
runtime: ctx.runtime,
|
||||
cfg: ctx.cfg,
|
||||
timeoutMs,
|
||||
});
|
||||
ctx.gatewayHealthSkipped = false;
|
||||
ctx.healthOk = healthOk;
|
||||
|
|
@ -449,7 +447,6 @@ async function runGatewayHealthChecks(ctx: DoctorHealthFlowContext): Promise<voi
|
|||
ctx.gatewayMemoryProbe = authenticated
|
||||
? await probeGatewayMemoryStatus({
|
||||
cfg: ctx.cfg,
|
||||
timeoutMs,
|
||||
})
|
||||
: { checked: false, ready: false, skipped: healthOk };
|
||||
}
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ type GatewayLocalProbeTarget = {
|
|||
export type ConfiguredGatewayLocalProbe = {
|
||||
requestHttp(params: {
|
||||
host: string;
|
||||
pathname: "/healthz" | "/readyz";
|
||||
pathname: "/healthz" | "/readyz" | "/startupz";
|
||||
port: number;
|
||||
timeoutMs: number;
|
||||
signal?: AbortSignal;
|
||||
|
|
@ -37,7 +37,7 @@ export function normalizeGatewayHttpProbeHost(host: string): string {
|
|||
|
||||
export async function requestGatewayLocalHttpProbe(params: {
|
||||
host: string;
|
||||
pathname: "/healthz" | "/readyz";
|
||||
pathname: "/healthz" | "/readyz" | "/startupz";
|
||||
port: number;
|
||||
timeoutMs: number;
|
||||
tlsFingerprints?: readonly string[];
|
||||
|
|
|
|||
|
|
@ -69,6 +69,7 @@ export type GatewayProbeServerSummary = {
|
|||
|
||||
export type GatewayProbeResult = {
|
||||
ok: boolean;
|
||||
startupPhase?: string;
|
||||
/** Set only after a Gateway hello or a correlated protocol response. */
|
||||
gatewayReached?: true;
|
||||
url: string;
|
||||
|
|
|
|||
|
|
@ -42,6 +42,7 @@ export type ProviderAuth = {
|
|||
type AuthStore = ReturnType<typeof ensureAuthProfileStore>;
|
||||
|
||||
type UsageAuthState = {
|
||||
signal?: AbortSignal;
|
||||
cfg: OpenClawConfig;
|
||||
env: NodeJS.ProcessEnv;
|
||||
agentDir?: string;
|
||||
|
|
@ -64,6 +65,7 @@ function resolveProviderApiKeyFromConfig(params: {
|
|||
providerIds: string[];
|
||||
envDirect?: Array<string | undefined>;
|
||||
}): string | undefined {
|
||||
params.state.signal?.throwIfAborted();
|
||||
const envDirect = params.envDirect?.map(normalizeSecretInput).find(Boolean);
|
||||
if (envDirect) {
|
||||
return envDirect;
|
||||
|
|
@ -219,7 +221,9 @@ async function resolveProviderApiKeyCandidatesFromConfigAndStore(params: {
|
|||
profileId,
|
||||
agentDir: params.state.agentDir,
|
||||
});
|
||||
params.state.signal?.throwIfAborted();
|
||||
} catch {
|
||||
params.state.signal?.throwIfAborted();
|
||||
// Preserve the remaining credential candidates when one SecretRef fails.
|
||||
continue;
|
||||
}
|
||||
|
|
@ -299,6 +303,7 @@ async function resolveOAuthToken(params: {
|
|||
provider: string;
|
||||
excludeProfileIds?: string[];
|
||||
}): Promise<ProviderAuth | null> {
|
||||
params.state.signal?.throwIfAborted();
|
||||
if (!params.state.allowAuthProfileStore) {
|
||||
return null;
|
||||
}
|
||||
|
|
@ -328,6 +333,7 @@ async function resolveOAuthToken(params: {
|
|||
profileId,
|
||||
agentDir: params.state.agentDir,
|
||||
});
|
||||
params.state.signal?.throwIfAborted();
|
||||
if (!resolved) {
|
||||
continue;
|
||||
}
|
||||
|
|
@ -352,6 +358,7 @@ async function resolveOAuthToken(params: {
|
|||
...(cred.email ? { email: cred.email } : {}),
|
||||
};
|
||||
} catch {
|
||||
params.state.signal?.throwIfAborted();
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
|
|
@ -368,6 +375,7 @@ async function resolveProviderUsageAuthViaPlugin(params: {
|
|||
config: params.state.cfg,
|
||||
env: params.state.env,
|
||||
context: {
|
||||
signal: params.state.signal,
|
||||
config: params.state.cfg,
|
||||
agentDir: params.state.agentDir,
|
||||
env: params.state.env,
|
||||
|
|
@ -486,6 +494,7 @@ function hasAuthProfileCredentialSource(params: {
|
|||
}
|
||||
|
||||
export async function resolveProviderAuths(params: {
|
||||
signal?: AbortSignal;
|
||||
providers: UsageProviderId[];
|
||||
auth?: ProviderAuth[];
|
||||
getStore?: () => AuthStore;
|
||||
|
|
@ -495,11 +504,13 @@ export async function resolveProviderAuths(params: {
|
|||
env?: NodeJS.ProcessEnv;
|
||||
onError?: (provider: UsageProviderId, error: unknown) => void;
|
||||
}): Promise<ProviderAuth[]> {
|
||||
params.signal?.throwIfAborted();
|
||||
if (params.auth) {
|
||||
return params.auth;
|
||||
}
|
||||
|
||||
const stateBase = {
|
||||
signal: params.signal,
|
||||
cfg: params.config ?? getRuntimeConfig(),
|
||||
env: params.env ?? process.env,
|
||||
agentDir: params.agentDir,
|
||||
|
|
@ -522,6 +533,7 @@ export async function resolveProviderAuths(params: {
|
|||
const auths: ProviderAuth[] = [];
|
||||
|
||||
for (const provider of params.providers) {
|
||||
params.signal?.throwIfAborted();
|
||||
try {
|
||||
const directCredentialState = { ...stateBase, allowAuthProfileStore: false };
|
||||
const credentialProviderIds = resolveUsageCredentialProviderIds({
|
||||
|
|
@ -561,6 +573,7 @@ export async function resolveProviderAuths(params: {
|
|||
state,
|
||||
provider,
|
||||
});
|
||||
params.signal?.throwIfAborted();
|
||||
if (pluginAuth.auth) {
|
||||
auths.push(pluginAuth.auth);
|
||||
continue;
|
||||
|
|
@ -573,10 +586,12 @@ export async function resolveProviderAuths(params: {
|
|||
state,
|
||||
provider,
|
||||
});
|
||||
params.signal?.throwIfAborted();
|
||||
if (fallbackAuth) {
|
||||
auths.push(fallbackAuth);
|
||||
}
|
||||
} catch (error) {
|
||||
params.signal?.throwIfAborted();
|
||||
if (!params.onError) {
|
||||
throw error;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
// Covers provider usage summary loading across auth and plugin paths.
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import type { ProviderResolveUsageAuthContext } from "../plugins/types.js";
|
||||
import { AsyncWorkScope } from "../shared/async-work-scope.js";
|
||||
import { createDeferredCore } from "../shared/deferred.js";
|
||||
import { createProviderUsageFetch, makeResponse } from "../test-utils/provider-usage-fetch.js";
|
||||
|
|
@ -27,6 +28,185 @@ describe("provider-usage.load", () => {
|
|||
vi.restoreAllMocks();
|
||||
resetProviderUsageSnapshotWithPluginMock();
|
||||
});
|
||||
afterEach(() => vi.useRealTimers());
|
||||
|
||||
it.each([false, true])(
|
||||
"returns Timeout without auth or fetch dispatch for an exhausted budget (synthetic: %s)",
|
||||
async (synthetic) => {
|
||||
vi.useFakeTimers();
|
||||
const provider = synthetic ? "openai" : "anthropic";
|
||||
const fetch = vi.fn(async () => new Response("{}"));
|
||||
resolveProviderUsageAuthWithPluginMock.mockResolvedValue({ token: "fixture-token" });
|
||||
resolveProviderUsageSnapshotWithPluginMock.mockImplementation(async ({ context }) => {
|
||||
await context.fetchFn("https://usage.example.test");
|
||||
return { provider, displayName: provider, windows: [] };
|
||||
});
|
||||
|
||||
const pending = loadProviderUsageSummary({
|
||||
providers: [provider],
|
||||
...(synthetic
|
||||
? { auth: [{ provider, token: "codex-app-server", hookProvider: "codex" }] }
|
||||
: {}),
|
||||
config: {},
|
||||
env: { ANTHROPIC_API_KEY: "fixture-token" },
|
||||
now: usageNow,
|
||||
timeoutMs: 0,
|
||||
fetch,
|
||||
});
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
expect(await pending).toEqual({
|
||||
updatedAt: usageNow,
|
||||
providers: [
|
||||
{ provider, displayName: synthetic ? "OpenAI" : "Claude", windows: [], error: "Timeout" },
|
||||
],
|
||||
});
|
||||
expect(resolveProviderUsageAuthWithPluginMock).not.toHaveBeenCalled();
|
||||
expect(resolveProviderUsageSnapshotWithPluginMock).not.toHaveBeenCalled();
|
||||
expect(fetch).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
["init", false],
|
||||
["request", false],
|
||||
["init", true],
|
||||
["request", true],
|
||||
] as const)(
|
||||
"cancels the usage fetch (%s signal, caller abort: %s)",
|
||||
async (signalSource, callerAbort) => {
|
||||
vi.useFakeTimers();
|
||||
const scope = new AsyncWorkScope();
|
||||
const started = createDeferredCore();
|
||||
const response = createDeferredCore<Response>();
|
||||
const caller = new AbortController();
|
||||
const aborted = vi.fn();
|
||||
let fetchSignal: AbortSignal | null | undefined;
|
||||
const fetch = vi.fn<typeof globalThis.fetch>(async (input, init) => {
|
||||
fetchSignal = init?.signal ?? (input instanceof Request ? input.signal : undefined);
|
||||
fetchSignal?.addEventListener(
|
||||
"abort",
|
||||
() => {
|
||||
aborted();
|
||||
response.reject(fetchSignal?.reason);
|
||||
},
|
||||
{ once: true },
|
||||
);
|
||||
started.resolve();
|
||||
return response.promise;
|
||||
});
|
||||
resolveProviderUsageSnapshotWithPluginMock.mockImplementation(async ({ context }) => {
|
||||
await (signalSource === "init"
|
||||
? context.fetchFn("https://usage.example.test", { signal: caller.signal })
|
||||
: context.fetchFn(new Request("https://usage.example.test", { signal: caller.signal })));
|
||||
return { provider: "anthropic", displayName: "Claude", windows: [] };
|
||||
});
|
||||
const pending = scope.track(() =>
|
||||
loadProviderUsageSummary({
|
||||
auth: [{ provider: "anthropic", token: "fixture-token" }],
|
||||
config: {},
|
||||
env: {},
|
||||
timeoutMs: 1,
|
||||
fetch,
|
||||
}),
|
||||
);
|
||||
try {
|
||||
await started.promise;
|
||||
if (callerAbort) {
|
||||
caller.abort(new Error("Caller cancelled"));
|
||||
}
|
||||
await vi.advanceTimersByTimeAsync(callerAbort ? 0 : 1);
|
||||
expect(fetchSignal?.aborted).toBe(true);
|
||||
expect(aborted).toHaveBeenCalledOnce();
|
||||
expect(caller.signal.aborted).toBe(callerAbort);
|
||||
expect((await pending).providers).toEqual([
|
||||
{
|
||||
provider: "anthropic",
|
||||
displayName: "Claude",
|
||||
windows: [],
|
||||
error: callerAbort ? "Caller cancelled" : "Timeout",
|
||||
},
|
||||
]);
|
||||
await scope.drain();
|
||||
expect(scope.hasPendingWork).toBe(false);
|
||||
} finally {
|
||||
response.resolve(new Response("{}"));
|
||||
await pending;
|
||||
await scope.drain();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["key", "candidates", "oauth"])(
|
||||
"rejects a retained %s auth helper after the usage deadline",
|
||||
async (helper) => {
|
||||
vi.useFakeTimers();
|
||||
const scope = new AsyncWorkScope();
|
||||
const captured = createDeferredCore<ProviderResolveUsageAuthContext>();
|
||||
const release = createDeferredCore();
|
||||
resolveProviderUsageAuthWithPluginMock.mockImplementation(async ({ context }) => {
|
||||
captured.resolve(context);
|
||||
await release.promise;
|
||||
return { handled: true };
|
||||
});
|
||||
const pending = scope.track(() =>
|
||||
loadProviderUsageSummary({
|
||||
providers: ["anthropic"],
|
||||
config: {},
|
||||
env: { ANTHROPIC_API_KEY: "fixture-token" },
|
||||
timeoutMs: 1,
|
||||
}),
|
||||
);
|
||||
try {
|
||||
const context = await captured.promise;
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
expect((await pending).providers[0]?.error).toBe("Timeout");
|
||||
expect(context.signal?.aborted).toBe(true);
|
||||
await expect(
|
||||
Promise.resolve().then(async () => {
|
||||
if (helper === "key") {
|
||||
context.resolveApiKeyFromConfigAndStore();
|
||||
} else if (helper === "candidates") {
|
||||
await context.resolveApiKeyCandidatesFromConfigAndStore?.();
|
||||
} else {
|
||||
await context.resolveOAuthToken();
|
||||
}
|
||||
}),
|
||||
).rejects.toBe(context.signal?.reason);
|
||||
expect(resolveProviderUsageSnapshotWithPluginMock).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
release.resolve();
|
||||
await pending;
|
||||
await scope.drain();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("does not dispatch usage after auth completes beyond the deadline", async () => {
|
||||
vi.useFakeTimers();
|
||||
const scope = new AsyncWorkScope();
|
||||
const auth = createDeferredCore<{ token: string }>();
|
||||
resolveProviderUsageAuthWithPluginMock.mockReturnValue(auth.promise);
|
||||
const pending = scope.track(() =>
|
||||
loadProviderUsageSummary({
|
||||
providers: ["anthropic"],
|
||||
config: {},
|
||||
env: { ANTHROPIC_API_KEY: "fixture-token" },
|
||||
timeoutMs: 1,
|
||||
}),
|
||||
);
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
expect((await pending).providers[0]?.error).toBe("Timeout");
|
||||
expect(resolveProviderUsageAuthWithPluginMock).toHaveBeenCalledOnce();
|
||||
auth.resolve({ token: "fixture-token" });
|
||||
await scope.drain();
|
||||
expect(resolveProviderUsageSnapshotWithPluginMock).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
auth.resolve({ token: "fixture-token" });
|
||||
await pending;
|
||||
await scope.drain();
|
||||
}
|
||||
});
|
||||
|
||||
it("loads snapshots for copilot gemini codex and Xiaomi providers", async () => {
|
||||
resolveProviderUsageSnapshotWithPluginMock.mockImplementation(
|
||||
|
|
|
|||
|
|
@ -23,22 +23,6 @@ import type {
|
|||
UsageSummary,
|
||||
} from "./provider-usage.types.js";
|
||||
|
||||
// Built-in fallback intentionally reports unsupported until a plugin supplies usage behavior.
|
||||
async function fetchProviderUsageSnapshotFallback(params: {
|
||||
auth: ProviderAuth;
|
||||
timeoutMs: number;
|
||||
fetchFn: typeof fetch;
|
||||
}): Promise<ProviderUsageSnapshot> {
|
||||
void params.timeoutMs;
|
||||
void params.fetchFn;
|
||||
return {
|
||||
provider: params.auth.provider,
|
||||
displayName: providerUsageLabel(params.auth.provider) ?? params.auth.provider,
|
||||
windows: [],
|
||||
error: "Unsupported provider",
|
||||
};
|
||||
}
|
||||
|
||||
type UsageSummaryOptions = {
|
||||
now?: number;
|
||||
timeoutMs?: number;
|
||||
|
|
@ -59,6 +43,7 @@ async function fetchProviderUsageSnapshot(params: {
|
|||
agentDir?: string;
|
||||
workspaceDir?: string;
|
||||
timeoutMs: number;
|
||||
signal: AbortSignal;
|
||||
fetchFn: typeof fetch;
|
||||
}): Promise<ProviderUsageSnapshot> {
|
||||
const pluginSnapshot = await resolveProviderUsageSnapshotWithPlugin({
|
||||
|
|
@ -79,17 +64,18 @@ async function fetchProviderUsageSnapshot(params: {
|
|||
rateLimitTier: params.auth.rateLimitTier,
|
||||
email: params.auth.email,
|
||||
timeoutMs: params.timeoutMs,
|
||||
signal: params.signal,
|
||||
fetchFn: params.fetchFn,
|
||||
},
|
||||
});
|
||||
if (pluginSnapshot) {
|
||||
return pluginSnapshot;
|
||||
}
|
||||
return await fetchProviderUsageSnapshotFallback({
|
||||
auth: params.auth,
|
||||
timeoutMs: params.timeoutMs,
|
||||
fetchFn: params.fetchFn,
|
||||
});
|
||||
return (
|
||||
pluginSnapshot ?? {
|
||||
provider: params.auth.provider,
|
||||
displayName: providerUsageLabel(params.auth.provider) ?? params.auth.provider,
|
||||
windows: [],
|
||||
error: "Unsupported provider",
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
/** Loads usage snapshots from configured provider auth and plugin-backed usage hooks. */
|
||||
|
|
@ -100,13 +86,6 @@ export async function loadProviderUsageSummary(
|
|||
const timeoutMs = opts.timeoutMs ?? PROVIDER_USAGE_TIMEOUT_MS;
|
||||
const config = opts.config ?? getRuntimeConfig();
|
||||
const env = opts.env ?? process.env;
|
||||
const fetchFn = opts.fetch
|
||||
? resolveFetch(opts.fetch)
|
||||
: (resolveProxyFetchFromEnv(env) ?? resolveFetch());
|
||||
if (!fetchFn) {
|
||||
throw new Error("fetch is not available");
|
||||
}
|
||||
|
||||
const descriptors: ProviderUsagePluginDescriptor[] = opts.providers
|
||||
? opts.providers.map((provider) => ({
|
||||
provider,
|
||||
|
|
@ -132,46 +111,71 @@ export async function loadProviderUsageSummary(
|
|||
windows: [],
|
||||
error,
|
||||
});
|
||||
if (timeoutMs <= 0) {
|
||||
return {
|
||||
updatedAt: now,
|
||||
providers: descriptors.map(({ provider }) => failureSnapshot(provider, "Timeout")),
|
||||
};
|
||||
}
|
||||
const fetchFn = opts.fetch
|
||||
? resolveFetch(opts.fetch)
|
||||
: (resolveProxyFetchFromEnv(env) ?? resolveFetch());
|
||||
if (!fetchFn) {
|
||||
throw new Error("fetch is not available");
|
||||
}
|
||||
let authStore = opts.authStore;
|
||||
const getAuthStore = () =>
|
||||
(authStore ??= ensureAuthProfileStore(opts.agentDir, { allowKeychainPrompt: false }));
|
||||
const tasks = descriptors.map(({ provider }) => {
|
||||
// The response deadline does not end the auth/fetch producer's lifetime.
|
||||
return raceUsageTimeout(
|
||||
trackAsyncWork(async () => {
|
||||
let authError: unknown;
|
||||
const auth =
|
||||
opts.auth?.find((candidate) => candidate.provider === provider) ??
|
||||
(
|
||||
await resolveProviderAuths({
|
||||
providers: [provider],
|
||||
agentDir: opts.agentDir,
|
||||
config,
|
||||
env,
|
||||
getStore: getAuthStore,
|
||||
store: opts.authStore,
|
||||
onError: (_provider, error) => {
|
||||
authError = error;
|
||||
},
|
||||
})
|
||||
)[0];
|
||||
if (authError) {
|
||||
const message = formatErrorMessage(authError);
|
||||
return failureSnapshot(provider, message.trim() || "Auth failed");
|
||||
}
|
||||
if (!auth) {
|
||||
return undefined;
|
||||
}
|
||||
return await fetchProviderUsageSnapshot({
|
||||
auth,
|
||||
config,
|
||||
env,
|
||||
agentDir: opts.agentDir,
|
||||
workspaceDir: opts.workspaceDir,
|
||||
timeoutMs,
|
||||
fetchFn,
|
||||
});
|
||||
}),
|
||||
(signal) =>
|
||||
trackAsyncWork(async () => {
|
||||
let authError: unknown;
|
||||
const auth =
|
||||
opts.auth?.find((candidate) => candidate.provider === provider) ??
|
||||
(
|
||||
await resolveProviderAuths({
|
||||
providers: [provider],
|
||||
agentDir: opts.agentDir,
|
||||
config,
|
||||
env,
|
||||
signal,
|
||||
getStore: getAuthStore,
|
||||
store: opts.authStore,
|
||||
onError: (_provider, error) => {
|
||||
authError = error;
|
||||
},
|
||||
})
|
||||
)[0];
|
||||
signal.throwIfAborted();
|
||||
if (authError) {
|
||||
const message = formatErrorMessage(authError);
|
||||
return failureSnapshot(provider, message.trim() || "Auth failed");
|
||||
}
|
||||
if (!auth) {
|
||||
return undefined;
|
||||
}
|
||||
return await fetchProviderUsageSnapshot({
|
||||
auth,
|
||||
config,
|
||||
env,
|
||||
agentDir: opts.agentDir,
|
||||
workspaceDir: opts.workspaceDir,
|
||||
timeoutMs,
|
||||
signal,
|
||||
fetchFn: (input, init) => {
|
||||
signal.throwIfAborted();
|
||||
const callerSignal =
|
||||
init?.signal === undefined && input instanceof Request
|
||||
? input.signal
|
||||
: init?.signal;
|
||||
return fetchFn(input, {
|
||||
...init,
|
||||
signal: callerSignal ? AbortSignal.any([signal, callerSignal]) : signal,
|
||||
});
|
||||
},
|
||||
});
|
||||
}),
|
||||
timeoutMs,
|
||||
failureSnapshot(provider, "Timeout"),
|
||||
).catch((error: unknown) => {
|
||||
|
|
|
|||
|
|
@ -57,10 +57,10 @@ describe("provider-usage.shared", () => {
|
|||
},
|
||||
])("$name", async ({ promise, expected, error }) => {
|
||||
if (error) {
|
||||
await expect(raceUsageTimeout(promise(), 100, "fallback")).rejects.toThrow(error);
|
||||
await expect(raceUsageTimeout(promise, 100, "fallback")).rejects.toThrow(error);
|
||||
return;
|
||||
}
|
||||
await expect(raceUsageTimeout(promise(), 100, "fallback")).resolves.toBe(expected);
|
||||
await expect(raceUsageTimeout(promise, 100, "fallback")).resolves.toBe(expected);
|
||||
});
|
||||
|
||||
it("returns fallback when timeout wins", async () => {
|
||||
|
|
@ -68,7 +68,7 @@ describe("provider-usage.shared", () => {
|
|||
const late = new Promise<string>((resolve) => {
|
||||
setTimeout(() => resolve("late"), 50);
|
||||
});
|
||||
const result = raceUsageTimeout(late, 1, "fallback");
|
||||
const result = raceUsageTimeout(() => late, 1, "fallback");
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
await expect(result).resolves.toBe("fallback");
|
||||
});
|
||||
|
|
@ -78,7 +78,7 @@ describe("provider-usage.shared", () => {
|
|||
const setTimeoutSpy = vi.spyOn(globalThis, "setTimeout");
|
||||
|
||||
const result = raceUsageTimeout(
|
||||
new Promise<string>(() => {}),
|
||||
() => new Promise<string>(() => {}),
|
||||
Number.MAX_SAFE_INTEGER,
|
||||
"fallback",
|
||||
);
|
||||
|
|
@ -92,7 +92,7 @@ describe("provider-usage.shared", () => {
|
|||
it("clears the timeout after successful work", async () => {
|
||||
const clearTimeoutSpy = vi.spyOn(globalThis, "clearTimeout");
|
||||
|
||||
await expect(raceUsageTimeout(Promise.resolve("ok"), 100, "fallback")).resolves.toBe("ok");
|
||||
await expect(raceUsageTimeout(async () => "ok", 100, "fallback")).resolves.toBe("ok");
|
||||
|
||||
expect(clearTimeoutSpy).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -78,20 +78,24 @@ export const ignoredErrors = new Set([
|
|||
export const clampPercent = (value: number) =>
|
||||
Math.max(0, Math.min(100, Number.isFinite(value) ? value : 0));
|
||||
|
||||
/** Resolves a promise with a fallback when usage collection exceeds the timeout. */
|
||||
/** Aborts usage collection and returns a fallback when its deadline expires. */
|
||||
export const raceUsageTimeout = async <T>(
|
||||
work: Promise<T>,
|
||||
work: (signal: AbortSignal) => Promise<T>,
|
||||
ms: number,
|
||||
fallback: T,
|
||||
): Promise<T> => {
|
||||
let timeout: NodeJS.Timeout | undefined;
|
||||
const controller = new AbortController();
|
||||
const timeoutMs = resolveTimerTimeoutMs(ms, 1);
|
||||
try {
|
||||
return await Promise.race([
|
||||
work,
|
||||
new Promise<T>((resolve) => {
|
||||
timeout = setTimeout(() => resolve(fallback), timeoutMs);
|
||||
timeout = setTimeout(() => {
|
||||
resolve(fallback);
|
||||
controller.abort(new DOMException("Usage collection timed out", "TimeoutError"));
|
||||
}, timeoutMs);
|
||||
}),
|
||||
work(controller.signal),
|
||||
]);
|
||||
} finally {
|
||||
if (timeout) {
|
||||
|
|
|
|||
|
|
@ -150,6 +150,8 @@ export type ProviderPreparedRuntimeAuth = {
|
|||
* token blob, read a legacy credential file, or pick between aliases).
|
||||
*/
|
||||
export type ProviderResolveUsageAuthContext = {
|
||||
/** Cancel provider-owned work when the usage collection deadline expires. */
|
||||
signal?: AbortSignal;
|
||||
config: OpenClawConfig;
|
||||
agentDir?: string;
|
||||
workspaceDir?: string;
|
||||
|
|
@ -203,6 +205,8 @@ export type ProviderResolvedUsageAuth = ProviderUsageAuthToken | { handled: true
|
|||
* owns the provider-specific HTTP request + response normalization.
|
||||
*/
|
||||
export type ProviderFetchUsageSnapshotContext = {
|
||||
/** Custom transports must preserve this signal; fetchFn already includes it. */
|
||||
signal?: AbortSignal;
|
||||
config: OpenClawConfig;
|
||||
agentDir?: string;
|
||||
workspaceDir?: string;
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue