openclaw/scripts/lib/openclaw-e2e-instance.sh
Peter Steinberger 54abbae585
fix: refresh stale Gateway shutdown budgets during updates and Doctor (#153636)
* fix: refresh stale Gateway stop policies before maintenance

Main recognized historical systemd timeouts, but maintenance could stop the
resident before repair, and Doctor and already-current or no-restart updates
could preserve stale computed policy. A published 2026.9.5 resident also retains
its startup shutdown budget after the unit changes.

Refresh owned policy through the existing definition-mutation and backup owners,
confirm daemon reload, preserve operator drop-ins, and retain restore/reload/input
retirement ordering. Share maintenance between update and Doctor, and warn when
a non-stopping refresh still has a short effective manager timeout.

Publish process-owned shutdown budgets and lifecycle write-custody facts. Reuse
the suspension owner and existing update deadline: stop when idle, warn and stop
at the deadline for ordinary work or unknown custody, and refuse only current
reported write custody with its exact owner phase. Reread native policy when the
new Gateway accepts shutdown without resetting its elapsed budget or watchdog.

Thanks @ezimerman for the installed-unit and shutdown evidence.
Fixes #153153. Refs #150898, #152879, #153017.

* fix: preserve the resident shutdown budget in Gateway status

The kernel request-context adapter copied host lifecycle control methods but
dropped the recorded shutdown-budget getter. Real Linux package proof observed
a 325-second startup budget while status omitted it, forcing maintenance onto
the legacy unknown-budget path.

Forward the live getter through the existing adapter without changing request
authority or adding another budget owner. Add a real-kernel registered-status
regression for short and adequate budgets; the corrected test fails on the
original adapter and passes with the forwarding line.

Refs #153153.

* test: control the Gateway shutdown clock consistently

Restore the explicit node:perf_hooks performance import for the run-loop regressions that retain their own fake-clock assertions. They must control the same monotonic clock as the production shutdown-budget owner. Keep the deadline assertions, timers, and production behavior unchanged.

* fix: preserve Doctor legacy reads during Gateway preflight

The stale-Gateway probe opened the canonical owner-lease database without
Doctor's existing legacy-catalog admission. With a built install and a busy
Gateway port, that read created WAL/SHM files and rejected a supported repair
before maintenance could stop the Gateway.

Carry the existing read admission through restart inspection to the lease owner.
Keep ordinary restart validation unchanged and preserve canonical artifacts.
Use synthetic port, build, and reachability facts in the regression fixture while
retaining real lease reads and byte-preservation assertions.

Refs #153153.

* refactor: keep Gateway maintenance owners within line limits

The L903 stop-policy repair exceeded the existing line-growth gate after
composition with current restart and service identity handling. Move request
upgrade policy into its existing request owner and service revalidation into
one sibling implementation without changing their behavior.

Preserve original request admission time and Doctor's statically primed
maintenance facade across package replacement. The bounded drain, recorded
custody-only refusal, warning policy, and native backup/reload ordering remain
unchanged. No options, schema changes, suppressions, or baseline growth.

Validation: 398 focused tests, core typecheck, line-growth ratchet, and fresh
Codex P1 review passed. Full changed checks continue on the frozen source.

* test: retain backup custody coverage through the archive walker

The rebase incorporated the maintained archive walker, but the L903 custody
regression still referenced the retired tar-create mock. Use the existing
walker mock bound to the real backup command without changing assertions or
production code.

Validation: 132 backup, migration, cron, coordinator, and suspension tests
passed. Fresh Codex P1 review is clean. Full changed checks continue.

* fix: preserve maintenance lifecycle and wrapper contracts

Doctor fixtures advertised a running native service without the matching
resident identity, effective policy, or lifecycle readiness response. Supply
those facts through the same RPC and native-query boundaries used by the real
maintenance owner, including fresh readiness without a resident budget.

Keep exact suspension assertions current with the additive custody category.
Install the model-acquisition fixture's manager after normal PATH setup so
startup and shutdown observe the same policy under the original deadlines.
Include the custody owner in the canonical PR-wrapper source inventory.

Move the unchanged Doctor inspection assertion and shared fixtures into their
existing policy/support owners to preserve the line-growth ratchet. Do not
change the bounded-deferral, warning, or reported-write-custody refusal policy.

* fix: preserve Stop ownership through shutdown budget refresh

An asynchronous systemd budget read could resume after Stop captured a
foreground updater and re-arm the hard-exit worker. Keep watchdog admission
with the run loop's current successor owner, and represent an absent cleanup
deadline with no process-cleanup budget.

Preserve foreground no-op service ownership and the full native allowance
after verified parking. Keep one fake clock for boundary tests, prepare
fixture operations before held scripts, and join cancelled fixture work
before the next case. Move unchanged budget cases into their support owner.

Retain the ruling that unknown custody cannot block an update and only
reported write custody may refuse maintenance. Preserve all existing test
assertions and limits. The full Linux changed gate, 790 focused Linux tests,
and a fresh P1 review passed; local host limitations are recorded in the PR.

* refactor(doctor): extract update-run admission from doctor-maintenance

* fix: preserve native policy and write custody during maintenance

* fix: keep shutdown integration within source and type gates

* fix(test): own survivor model endpoint before baseline setup

---------

Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
2026-09-21 12:31:05 +08:00

611 lines
21 KiB
Bash

#!/usr/bin/env bash
# Shared in-container lifecycle helpers for Docker/Bash E2E lanes.
openclaw_e2e_eval_test_state_from_b64() {
local encoded="${1:?missing OpenClaw test-state script}"
local decoded
if ! decoded="$(printf '%s' "$encoded" | base64 -d)"; then
echo "Invalid OpenClaw test-state base64 payload" >&2
return 1
fi
if [ -z "${decoded//[[:space:]]/}" ]; then
echo "OpenClaw test-state base64 payload decoded to an empty script" >&2
return 1
fi
eval "$decoded"
}
openclaw_e2e_read_positive_int_env() {
local name="${1:?missing environment variable name}"
local fallback="${2:?missing fallback value}"
local value="${!name-}"
if [ -z "${!name+x}" ]; then
value="$fallback"
fi
if [[ ! "$value" =~ ^[0-9]+$ ]] || (( 10#$value < 1 )); then
echo "invalid $name: $value" >&2
return 2
fi
printf '%s\n' "$value"
}
openclaw_e2e_read_nonnegative_int_env() {
local name="${1:?missing environment variable name}"
local fallback="${2:?missing fallback value}"
local value="${!name-}"
if [ -z "${!name+x}" ]; then
value="$fallback"
fi
if [[ ! "$value" =~ ^[0-9]+$ ]]; then
echo "invalid $name: $value" >&2
return 2
fi
printf '%s\n' "$value"
}
openclaw_e2e_resolve_entrypoint() {
local entry
for entry in dist/index.mjs dist/index.js; do
[ -f "$entry" ] && { printf '%s\n' "$entry"; return 0; }
done
echo "OpenClaw entrypoint not found under dist/" >&2
return 1
}
openclaw_e2e_run_script_entrypoint() {
local stem="${1:?missing OpenClaw E2E script stem}"
shift
if [ -f "$stem.mts" ]; then
tsx "$stem.mts" "$@"
return
fi
if [ -f "$stem.mjs" ]; then
node "$stem.mjs" "$@"
return
fi
echo "OpenClaw E2E script entrypoint not found: $stem.{mts,mjs}" >&2
return 1
}
openclaw_e2e_package_root() {
local prefix="${1:-}"
if [ -n "$prefix" ]; then
printf '%s/lib/node_modules/openclaw\n' "$prefix"
return 0
fi
printf '%s/openclaw\n' "$(npm root -g)"
}
openclaw_e2e_package_entrypoint() {
local root="${1:?missing package root}"
local entry
for entry in "$root/dist/index.mjs" "$root/dist/index.js"; do
[ -f "$entry" ] && { printf '%s\n' "$entry"; return 0; }
done
echo "OpenClaw package entrypoint not found under $root/dist/" >&2
return 1
}
openclaw_e2e_maybe_timeout() {
local timeout_value="$1"
shift
if [ -z "$timeout_value" ] || [ "$timeout_value" = "0" ]; then
"$@"
return
fi
local timeout_bin=""
if command -v timeout >/dev/null 2>&1; then
timeout_bin="timeout"
elif command -v gtimeout >/dev/null 2>&1; then
timeout_bin="gtimeout"
fi
if [ -z "$timeout_bin" ]; then
if command -v node >/dev/null 2>&1; then
echo "timeout command not found; using Node watchdog for OpenClaw E2E command timeout $timeout_value" >&2
if [[ "$1" != */* ]]; then
local resolved_command
resolved_command="$(command -v "$1" 2>/dev/null || true)"
if [ -n "$resolved_command" ]; then
set -- "$resolved_command" "${@:2}"
fi
fi
# Keep stdin attached to the child instead of consuming it as watchdog source.
node --input-type=module -e '
const [, timeoutValue, command, ...args] = process.argv;
const parseTimeoutMs = (value) => {
const match = /^([0-9]+(?:\.[0-9]+)?)(ms|s|m|h)?$/u.exec(String(value ?? "").trim());
if (!match) {
throw new Error(`unsupported timeout value: ${value}`);
}
const amount = Number(match[1]);
const unit = match[2] ?? "s";
const multiplier = unit === "ms" ? 1 : unit === "s" ? 1_000 : unit === "m" ? 60_000 : 3_600_000;
return Math.max(1, Math.ceil(amount * multiplier));
};
if (!command) {
console.error("missing command for Node watchdog");
process.exit(1);
}
const { spawn } = await import("node:child_process");
let timeoutMs;
try {
timeoutMs = parseTimeoutMs(timeoutValue);
} catch (error) {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
}
const child = spawn(command, args, {
detached: process.platform !== "win32",
env: process.env,
stdio: "inherit",
});
let timedOut = false;
let parentSignal = null;
let parentSignalTimer = null;
const signalExitCodes = new Map([
["SIGHUP", 129],
["SIGINT", 130],
["SIGTERM", 143],
]);
const killGraceMs = Number.parseInt(
process.env.OPENCLAW_E2E_TIMEOUT_KILL_GRACE_MS || "30000",
10,
);
const killTarget = process.platform === "win32" ? child.pid : -child.pid;
const killChild = (signal) => {
if (!child.pid) {
return;
}
try {
process.kill(killTarget, signal);
} catch {
try {
child.kill(signal);
} catch {}
}
};
const timer = setTimeout(() => {
timedOut = true;
console.error(`OpenClaw E2E command timed out after ${timeoutValue}`);
killChild("SIGTERM");
setTimeout(() => killChild("SIGKILL"), killGraceMs).unref();
}, timeoutMs);
const forwardSignal = (signal) => {
if (parentSignal) {
killChild("SIGKILL");
process.exit(signalExitCodes.get(signal) ?? 1);
}
parentSignal = signal;
clearTimeout(timer);
killChild(signal);
parentSignalTimer = setTimeout(() => {
killChild("SIGKILL");
process.exit(signalExitCodes.get(signal) ?? 1);
}, killGraceMs);
parentSignalTimer.unref();
};
process.once("SIGINT", forwardSignal);
process.once("SIGTERM", forwardSignal);
process.once("SIGHUP", forwardSignal);
child.on("close", (code, signal) => {
clearTimeout(timer);
if (parentSignalTimer) {
clearTimeout(parentSignalTimer);
}
if (timedOut) {
process.exit(124);
}
if (parentSignal) {
process.exit(signalExitCodes.get(parentSignal) ?? 1);
}
if (code !== null) {
process.exit(code);
}
if (signal) {
process.kill(process.pid, signal);
}
process.exit(1);
});
child.on("error", (error) => {
clearTimeout(timer);
console.error(error.message);
process.exit(127);
});
' -- "$timeout_value" "$@"
return
fi
echo "timeout command not found and Node is unavailable; cannot bound OpenClaw E2E command after $timeout_value" >&2
return 127
fi
if "$timeout_bin" --kill-after=1s 1s true >/dev/null 2>&1; then
"$timeout_bin" --kill-after=30s "$timeout_value" "$@"
else
"$timeout_bin" "$timeout_value" "$@"
fi
}
openclaw_e2e_print_log() {
local path="$1"
local max_bytes max_lines redactor_module
max_bytes="$(openclaw_e2e_read_nonnegative_int_env OPENCLAW_E2E_LOG_TAIL_BYTES 262144)" || return $?
max_lines="$(openclaw_e2e_read_nonnegative_int_env OPENCLAW_E2E_LOG_TAIL_LINES 120)" || return $?
[ -f "$path" ] || return 0
echo "--- $path ---"
redactor_module="${OPENCLAW_E2E_REDACTOR_MODULE:-$(openclaw_e2e_package_root)/dist/plugin-sdk/logging-core.js}"
[ -f "$redactor_module" ] || redactor_module="$PWD/dist/plugin-sdk/logging-core.js"
if [ ! -f "$redactor_module" ]; then
echo "[failure log omitted: canonical redactor unavailable]"
return 0
fi
if ! { tail -c "$max_bytes" "$path" 2>/dev/null | tail -n "$max_lines" || tail -n "$max_lines" "$path" || true; } | \
node --input-type=module -e '
import { text } from "node:stream/consumers";
import { pathToFileURL } from "node:url";
const { redactSensitiveText } = await import(pathToFileURL(process.argv[1]).href);
process.stdout.write(redactSensitiveText(await text(process.stdin), { mode: "tools" }));
' "$redactor_module"; then
echo "[failure log omitted: canonical redaction failed]"
fi
}
openclaw_e2e_enable_failure_diagnostics() {
# Keep diagnostics outside command log redirections, including inherited traps.
# Scenarios reserve fd 4 and provide dump_debug_logs before enabling this handler.
exec 4>&2
set -E
trap 'status=$?; dump_debug_logs "$status" "$BASH_COMMAND" >&4 2>&4; exit "$status"' ERR
}
openclaw_e2e_install_package() {
local log_file="$1"
local label="${2:-mounted OpenClaw package}"
local prefix="${3:-}"
local package_tgz="${OPENCLAW_CURRENT_PACKAGE_TGZ:?missing OPENCLAW_CURRENT_PACKAGE_TGZ}"
local timeout_value="${OPENCLAW_E2E_NPM_INSTALL_TIMEOUT:-600s}"
local args=(-g)
if [ -n "$prefix" ]; then
args+=("--prefix" "$prefix")
fi
echo "Installing $label..."
if openclaw_e2e_maybe_timeout "$timeout_value" npm install "${args[@]}" "$package_tgz" --no-fund --no-audit >"$log_file" 2>&1; then
echo "Installed $label."
return 0
else
local install_status=$?
if [ "$install_status" -eq 124 ] || [ "$install_status" -eq 137 ]; then
echo "npm install timed out after $timeout_value for $label" >&2
fi
echo "npm install failed for $label" >&2
if [ -f "$log_file" ]; then
openclaw_e2e_print_log "$log_file" >&2
fi
return "$install_status"
fi
}
openclaw_e2e_find_dep_package() {
local dep_path="$1"
shift
find "$@" -path "*/node_modules/$dep_path/package.json" -print -quit 2>/dev/null || true
}
openclaw_e2e_assert_dep_absent() {
local dep_path="$1"
shift
if [ -n "$(openclaw_e2e_find_dep_package "$dep_path" "$@")" ]; then
echo "$dep_path should not be installed" >&2
find "$@" -path "*/node_modules/$dep_path/package.json" -print 2>/dev/null >&2 || true
exit 1
fi
}
openclaw_e2e_assert_dep_present() {
local dep_path="$1"
shift
if [ -n "$(openclaw_e2e_find_dep_package "$dep_path" "$@")" ]; then
return 0
fi
echo "$dep_path was not installed on demand" >&2
find "$@" -maxdepth 6 -type d -name node_modules -print 2>/dev/null >&2 || true
exit 1
}
openclaw_e2e_write_state_env() {
local target="${1:-/tmp/openclaw-test-state-env}"
{
printf 'export HOME=%q\n' "$HOME"
printf 'export OPENCLAW_HOME=%q\n' "$OPENCLAW_HOME"
printf 'export OPENCLAW_STATE_DIR=%q\n' "$OPENCLAW_STATE_DIR"
printf 'export OPENCLAW_CONFIG_PATH=%q\n' "$OPENCLAW_CONFIG_PATH"
printf 'export OPENCLAW_AGENT_DIR=%q\n' "${OPENCLAW_AGENT_DIR-}"
} >"$target"
}
openclaw_e2e_install_trash_shim() {
local shim_dir="${OPENCLAW_E2E_BIN_DIR:-}"
if [ -z "$shim_dir" ]; then
if [ -n "${OPENCLAW_STATE_DIR:-}" ]; then
shim_dir="$OPENCLAW_STATE_DIR/e2e-bin"
else
shim_dir="$(mktemp -d "${TMPDIR:-/tmp}/openclaw-bin.XXXXXX")"
fi
OPENCLAW_E2E_BIN_DIR="$shim_dir"
export OPENCLAW_E2E_BIN_DIR
fi
case ":$PATH:" in
*":$shim_dir:"*) ;;
*) export PATH="$shim_dir:$PATH" ;;
esac
mkdir -p "$shim_dir"
cat >"$shim_dir/trash" <<'TRASH'
#!/usr/bin/env bash
set -euo pipefail
trash_dir="$HOME/.Trash"
mkdir -p "$trash_dir"
for target in "$@"; do
[ -e "$target" ] || continue
base="$(basename "$target")"
dest="$trash_dir/$base"
[ -e "$dest" ] && dest="$trash_dir/${base}-$(date +%s)-$$"
mv "$target" "$dest"
done
TRASH
chmod +x "$shim_dir/trash"
}
openclaw_e2e_run_script_with_pty() {
local command="$1"
local log_path="$2"
local timeout_value="${OPENCLAW_E2E_COMMAND_TIMEOUT:-300s}"
if script --version >/dev/null 2>&1; then
openclaw_e2e_maybe_timeout "$timeout_value" script -q -f -c "$command" "$log_path"
elif node -e 'import("@lydell/node-pty")' >/dev/null 2>&1; then
openclaw_e2e_maybe_timeout "$timeout_value" node scripts/e2e/lib/run-with-pty.mjs "$log_path" /bin/bash -lc "$command"
else
openclaw_e2e_maybe_timeout "$timeout_value" script -q -F "$log_path" /bin/bash -lc "$command"
fi
}
openclaw_e2e_start_tracked_process() {
local log_path="${1:?missing OpenClaw E2E process log path}"
shift
if command -v setsid >/dev/null 2>&1; then
setsid "$@" >"$log_path" 2>&1 &
printf '%s\n' "$!"
return
fi
node --input-type=module - "$log_path" "$@" <<'NODE'
import { closeSync, openSync } from "node:fs";
import { spawn } from "node:child_process";
const [logPath, command, ...args] = process.argv.slice(2);
if (!command) {
console.error("missing command for OpenClaw E2E tracked process");
process.exit(1);
}
const logFd = openSync(logPath, "a");
const child = spawn(command, args, {
detached: process.platform !== "win32",
env: process.env,
stdio: ["ignore", logFd, logFd],
});
closeSync(logFd);
child.unref();
console.log(child.pid);
NODE
}
openclaw_e2e_signal_process() {
local pid="${1:-}" signal="${2:-TERM}"
[ -n "$pid" ] || return 0
if kill -0 -- "-$pid" >/dev/null 2>&1; then
kill "-$signal" -- "-$pid" >/dev/null 2>&1 || true
return 0
fi
kill "-$signal" "$pid" >/dev/null 2>&1 || true
}
openclaw_e2e_process_alive() {
local pid="${1:-}"
[ -n "$pid" ] || return 1
kill -0 "$pid" >/dev/null 2>&1 || kill -0 -- "-$pid" >/dev/null 2>&1
}
openclaw_e2e_stop_process() {
local pid="${1:-}" _
[ -n "$pid" ] || return 0
openclaw_e2e_signal_process "$pid" TERM
for _ in $(seq 1 40); do
! openclaw_e2e_process_alive "$pid" && { wait "$pid" >/dev/null 2>&1 || true; return 0; }
sleep 0.25
done
openclaw_e2e_signal_process "$pid" KILL
wait "$pid" >/dev/null 2>&1 || true
}
openclaw_e2e_terminate_gateways() {
openclaw_e2e_stop_process "${1:-}"
}
openclaw_e2e_start_mock_openai() { openclaw_e2e_start_tracked_process "$2" env "MOCK_PORT=$1" node scripts/e2e/mock-openai-server.mjs; }
openclaw_e2e_wait_mock_openai() {
# Port 0 uses the supplied process/log receipt and prints its allocated port.
local port="$1" attempts="${2:-80}" timeout_ms="${3:-400}" _
local base_url="${4:-}" pid="${5:-}" log="${6:-}" ready_port="$port" line
if [ "$port" = "0" ] && { [ -z "$pid" ] || [ -z "$log" ] || [ -n "$base_url" ]; }; then
echo "Allocated mock OpenAI readiness requires its process and log, without a base URL override." >&2
return 2
fi
for _ in $(seq 1 "$attempts"); do
if [ "$port" = "0" ]; then
# Only the actual listener publishes this complete line. An unrelated
# server's /health response cannot attest to ownership of the mock port.
openclaw_e2e_process_alive "$pid" || break
ready_port=""
while IFS= read -r line; do
if [[ "$line" =~ ^mock-openai\ listening\ on\ ([1-9][0-9]{0,4})$ ]] && [ "${BASH_REMATCH[1]}" -le 65535 ]; then
ready_port="${BASH_REMATCH[1]}"
break
fi
done < <(head -c 65536 "$log" 2>/dev/null)
fi
if [ -n "$ready_port" ] && openclaw_e2e_probe_http "${base_url:-http://127.0.0.1:${ready_port}}/health" ok "$timeout_ms"; then
if [ "$port" = "0" ]; then
openclaw_e2e_process_alive "$pid" || break
printf '%s\n' "$ready_port"
fi
return 0
fi
sleep 0.1
done
if [ "$port" = "0" ]; then
openclaw_e2e_print_log "$log" >&2
echo "Mock OpenAI did not publish a live listener." >&2
return 1
fi
openclaw_e2e_probe_http "${base_url:-http://127.0.0.1:${port}}/health" ok "$timeout_ms"
}
openclaw_e2e_start_gateway() { openclaw_e2e_start_tracked_process "$3" node "$1" gateway --port "$2" --bind loopback --allow-unconfigured; }
openclaw_e2e_exec_gateway() { exec node "$1" gateway --port "$2" --bind "${3:-loopback}" --allow-unconfigured >"$4" 2>&1; }
openclaw_e2e_gateway_log_port_from_text() {
sed -nE 's/.*(127\.0\.0\.1|localhost):([0-9]+).*/\2/p' | tail -n 1
}
openclaw_e2e_wait_gateway_ready() {
local pid="$1" log="$2" attempts="${3:-300}" ready_port="${4:-}" readiness_mode="${5:-strict}" _ saw_ready_log=false
local ready_scan_offset=0 ready_scan_carry="" ready_scan_carry_chars=256
local ready_log_pattern='\[gateway\] ready'
# Published baselines logged their listener before the modern ready marker existed.
if [ "$readiness_mode" = "legacy-ready-log-ok" ]; then
ready_log_pattern='\[gateway\] (ready|listening on)'
fi
for _ in $(seq 1 "$attempts"); do
! kill -0 "$pid" >/dev/null 2>&1 && {
echo "Gateway exited before becoming ready"
wait "$pid" || true
tail -n 120 "$log" 2>/dev/null || true
return 1
}
if [ "$saw_ready_log" != "true" ] && [ -f "$log" ]; then
local log_bytes="0"
log_bytes="$(wc -c <"$log" 2>/dev/null || echo 0)"
log_bytes="${log_bytes//[[:space:]]/}"
if ! [[ "$log_bytes" =~ ^[0-9]+$ ]]; then
log_bytes="0"
fi
if [ "$log_bytes" -lt "$ready_scan_offset" ]; then
ready_scan_offset=0
ready_scan_carry=""
fi
if [ "$log_bytes" -gt "$ready_scan_offset" ]; then
local new_log_text scan_text
new_log_text="$(tail -c +"$((ready_scan_offset + 1))" "$log" 2>/dev/null || true)"
ready_scan_offset="$log_bytes"
scan_text="${ready_scan_carry}${new_log_text}"
if [ "${#scan_text}" -gt "$ready_scan_carry_chars" ]; then
ready_scan_carry="${scan_text: -$ready_scan_carry_chars}"
else
ready_scan_carry="$scan_text"
fi
local ready_log_lines
ready_log_lines="$(printf "%s" "$scan_text" | grep -E "$ready_log_pattern" || true)"
if [ -n "$ready_log_lines" ]; then
saw_ready_log=true
[ -n "$ready_port" ] || ready_port="$(printf "%s" "$ready_log_lines" | openclaw_e2e_gateway_log_port_from_text)"
fi
fi
fi
if [ "$saw_ready_log" = "true" ]; then
[ -n "$ready_port" ] || ready_port="${OPENCLAW_E2E_GATEWAY_READY_PORT:-18789}"
if [ "$readiness_mode" = "legacy-ready-log-ok" ]; then
openclaw_e2e_probe_tcp 127.0.0.1 "$ready_port" 400 && return 0
else
openclaw_e2e_probe_http "http://127.0.0.1:${ready_port}/readyz" ok 400 && return 0
fi
fi
sleep 0.25
done
if [ "$saw_ready_log" = "true" ]; then
if [ "$readiness_mode" = "legacy-ready-log-ok" ]; then
echo "Gateway startup log was found, but TCP listener probe never succeeded"
else
echo "Gateway log reported ready, but /readyz probe never succeeded"
fi
else
echo "Gateway did not become ready"
fi
tail -n 120 "$log" 2>/dev/null || true
return 1
}
openclaw_e2e_probe_tcp() {
node --input-type=module -e '
import net from "node:net";
const socket = net.createConnection({ host: process.argv[1], port: Number(process.argv[2]) });
const timeout = setTimeout(() => { socket.destroy(); process.exit(1); }, Number(process.argv[3] ?? 400));
socket.on("connect", () => { clearTimeout(timeout); socket.end(); process.exit(0); });
socket.on("error", () => { clearTimeout(timeout); process.exit(1); });
' "$1" "$2" "${3:-400}"
}
openclaw_e2e_probe_http() {
node --input-type=module -e '
const expected = process.argv[2] ?? "ok";
const timeoutMs = Number(process.argv[3] ?? 400);
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), timeoutMs);
let exitCode = 1;
let response;
try {
response = await fetch(process.argv[1], { signal: controller.signal });
const passed = expected === "ok" ? response.ok : response.status === Number(expected);
exitCode = passed ? 0 : 1;
} catch {
exitCode = 1;
} finally {
clearTimeout(timer);
await response?.body?.cancel?.().catch(() => undefined);
}
process.exit(exitCode);
' "$1" "${2:-ok}" "${3:-400}"
}
openclaw_e2e_probe_http_status() {
openclaw_e2e_probe_http "$1" "${2:-200}" "${3:-400}"
}
openclaw_e2e_assert_file() { [ -f "$1" ] || { echo "Missing file: $1"; exit 1; }; }
openclaw_e2e_assert_dir() { [ -d "$1" ] || { echo "Missing dir: $1"; exit 1; }; }
openclaw_e2e_assert_log_not_contains() {
! grep -q "$2" "$1" || { echo "Unexpected log output: $2"; exit 1; }
}
openclaw_e2e_run_logged() {
local label="$1" log_root="${OPENCLAW_E2E_LOG_DIR:-${TMPDIR:-/tmp}}" log_path safe_label
shift
safe_label="${label//[^A-Za-z0-9_.-]/-}"
[ -n "$safe_label" ] || safe_label="command"
mkdir -p "$log_root"
log_path="$(mktemp "$log_root/openclaw-${safe_label}.XXXXXX.log")"
OPENCLAW_E2E_LAST_LOG_PATH="$log_path"
export OPENCLAW_E2E_LAST_LOG_PATH
openclaw_e2e_run_command "$@" >"$log_path" 2>&1 || { openclaw_e2e_print_log "$log_path"; exit 1; }
}
openclaw_e2e_run_command() {
local timeout_value="${OPENCLAW_E2E_COMMAND_TIMEOUT:-300s}"
openclaw_e2e_maybe_timeout "$timeout_value" "$@"
}
openclaw_e2e_fixture_plugin_command() {
local runner=()
while [[ "${1:-}" != "--" && "$#" -gt 0 ]]; do
runner+=("$1")
shift
done
shift
local help consent
# Preserve the successful candidate help result for evidence checks after mutation.
# Clear it first so a failed replacement probe cannot expose stale capabilities.
OPENCLAW_E2E_LAST_FIXTURE_PLUGIN_CAPABILITY_CONSENT_SUPPORTED=
# Use the caller's bounded runner for both calls; never cache across package replacement.
help="$("${runner[@]}" "$1" "$2" --help)" || {
local probe_status=$?
printf '%s\nPlugin fixture help probe failed with status %s: %s %s\n' "$help" "$probe_status" "$1" "$2" >&2
return "$probe_status"
}
consent="$(printf '%s' "$help" | node scripts/e2e/lib/package-compat.mjs fixture-consent)" || return $?
if [[ -n "$consent" ]]; then
OPENCLAW_E2E_LAST_FIXTURE_PLUGIN_CAPABILITY_CONSENT_SUPPORTED=1
set -- "$@" "$consent"
else
OPENCLAW_E2E_LAST_FIXTURE_PLUGIN_CAPABILITY_CONSENT_SUPPORTED=0
fi
"${runner[@]}" "$@"
}
openclaw_e2e_enable_openclaw_cli_timeout() {
OPENCLAW_E2E_CLI_BIN="$(type -P openclaw)"
if [ -z "$OPENCLAW_E2E_CLI_BIN" ]; then
echo "OpenClaw CLI binary not found on PATH" >&2
return 1
fi
export OPENCLAW_E2E_CLI_BIN
openclaw() {
openclaw_e2e_run_command "$OPENCLAW_E2E_CLI_BIN" "$@"
}
}
openclaw_e2e_dump_logs() {
local path
for path in "$@"; do
openclaw_e2e_print_log "$path"
done
}