mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-16 20:33:56 +00:00
7bf8ade2db
* Studio: make research browser harnesses enforce their claims * Have the browser harnesses own their dev server for PR #8736 The new CI step backgrounds `npm run dev`, so $! is the npm wrapper and the EXIT trap leaves the vite node child alive holding the port and the step's stdout pipe. Reproduced: after the trap fires the port still answers 200. The ANSI smoke already solved this, so its start_vite / stop_process / drain_process_output move into _playwright_robust.py and all three harnesses use them. The CI step is two plain python3 calls now, and each file runs standalone the way its own docstring says. Readiness checks content, not status. Vite's SPA fallback answers 200 with index.html for anything missing, so a status check passes on a deleted smoke page. Confirmed locally: /smoke-DOESNOTEXIST.html returns 200. The report phase keeps a real, hit-tested click. A synthetic element.click() lands even with `body { pointer-events: none }` stranded, which is the freeze under test, so clicks_registered had stopped covering it. Measured both ways against a stranded layer: synthetic +1, real +0 and not actionable. The stall probe stays alongside it as main_thread_stall_ms, since nothing here reads an input timestamp, and its budget goes to 1000ms: 500 left only 1.2x against 342-416ms measured on a loaded host, and 1000 still fails ten times the report size (1518ms). Also: chat wall time and rAF count come from one page evaluation so they bracket the same interval; smoke-ansi-main.tsx joins the typechecked entries; the ANSI default port moves off the contended 8000; the job timeout goes to 20 minutes now that two browser smokes sit inside it; and the contract test pins the new verdicts plus the self-hosting rule. Verified: all three harnesses exit 0 standalone with no leftover vite and every port closed; typecheck 0; 2389 frontend tests; contract tests pass and fail when the new guards are removed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden the shared dev-server lifecycle for PR #8736 Simulated the failure modes of the self-hosting change in an isolated venv. Five were real; all five are fixed here, each with a test that fails when the fix is reverted. 1. A busy port was measured instead of refused. Under --strictPort our vite exits, and the readiness poll then talks to whoever else holds the port. Worst case is a squatter serving a page that happens to contain the entry string, which content matching alone accepts. start_vite now refuses an occupied port up front. 2. A dead server cost the full 120s timeout, three times per CI run. Readiness now takes the process and gives up the moment it exits, surfacing vite's own last lines. Measured: 120s to 0.0s. 3. SIGTERM orphaned the server. `finally` covers exceptions and SIGINT but not SIGTERM, which is what a CI cancel sends, so the exact leak this work is about survived a cancelled job. SIGTERM and SIGHUP now tear down registered servers, chaining to any previous handler, and atexit covers the rest. 4. Teardown could raise over the failure that called it. stop_process runs from a `finally`, and a child outliving SIGKILL turned a clean harness failure into a TimeoutExpired traceback with the real error lost. 5. An exported-but-empty SMOKE_BASE_URL counted as external, so no server started and the harness drove "" as its base URL. Empty now means unset. tests/studio/test_playwright_server_lifecycle.py drives both platform branches by injecting os.name, so the Windows path (CREATE_NEW_PROCESS_GROUP, taskkill /T then /T /F) is checked on every run rather than on someone's machine. It needs no browser and no npm, so CI runs it before the Chromium install. Verified: all three harnesses pass self-hosting and in the pre-existing SMOKE_BASE_URL mode, where an external server is correctly left running; ANSI smoke passes on chromium, firefox and webkit; no leftover processes and every port closed; 22 simulations and 23 in-repo tests pass; all four mutants of the fixes above are caught. * Run the lifecycle tests after the playwright install They import the harnesses, which import playwright, so the browserless step I put before the install could never have worked on a clean runner. Caught by staging CI, reproduced locally in a pytest-only venv (same 5 failures), and confirmed fixed in that same venv once playwright is present: 23 passed. pytest moves into the existing install line rather than getting its own pip call. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments added for PR #8736 Comment-only pass: collapses the duplicated SMOKE_BASE_URL notes, the teardown and readiness asides, and the click rationale in the contract test. Intent unchanged, 12 fewer lines. Pre-existing comments from #8633 are left alone. AST-verified comments only (comment_tools.py check: 4/4 OK), 56 tests pass. * Skip the harness-import tests when playwright is absent Cross-platform staging CI on macos-14 failed 5 of 23: the two tests that reload a harness module pull in playwright.sync_api, and that runner does not ship it. Several workflows run tests/studio without installing playwright, so the file broke collection there rather than only in Frontend CI. pytest.importorskip is the convention already used in this directory (test_cached_model_path_selection.py, test_pdf_qa_recipe_contract.py). The other 18 touch stdlib-only helpers and are unaffected. Verified in a pytest-only venv: 18 passed, 5 skipped, where it was 5 failed. With playwright present all 17 still run rather than skipping. * Honour SMOKE_BASE_URL in the ANSI smoke It started its own server unconditionally, so pointing it at an external one still spawned a second vite, and with the new occupied-port check it now raises before the external page is ever tested. Reproduced against a server on the default 5203: RuntimeError: 127.0.0.1:5203 is already serving. The other two harnesses already derive OWNS_SERVER from SMOKE_BASE_URL; this brings the third into line. Self-hosting is unchanged, since OWNS_SERVER is true there. Verified both ways: external mode exits 0 and leaves that server running; self-hosting still starts and stops its own. * Let the POSIX teardown tests run on Windows Cross-platform staging on windows-latest failed 3 of 23: AttributeError: <module 'os' (frozen)> has no attribute 'killpg' os.killpg is POSIX-only, so monkeypatch.setattr had no attribute to replace and raised during setup. The tests that exist to prove the Windows branch is covered were themselves the ones Windows could not run. raising = False lets them install the stub on either platform; the assertions are unchanged, and os.name is already injected so the POSIX branch is what they exercise. Verified with os.killpg deleted from the interpreter to mimic the runner: 17 passed, same as on POSIX. * Make the Windows paths actually work in the harness and its tests Two Windows-only breaks, both confirmed against the Python docs before fixing. signal.SIGKILL is Unix-only, so the tests that force os.name to posix to exercise the POSIX teardown could not name it on a Windows interpreter, and raised after the fake process timed out. raising=False on os.killpg was not enough. A posix_branch fixture now stubs both, and the assertion compares against the same portable constant. start_vite ran a bare npm. On Windows npm is the batch file npm.cmd, and CreateProcess cannot execute a .cmd with shell=False, so the self-hosting default this PR documents would have failed with FileNotFoundError before vite started. shutil.which honours PATHEXT and resolves npm.cmd there, and returns /usr/bin/npm here. Verified with os.killpg and signal.SIGKILL both deleted from the interpreter: 17 passed, same as POSIX. The ANSI smoke still runs end to end on Linux through shutil.which. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
707 lines
27 KiB
Python
707 lines
27 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""Shared CI-runner workarounds for the Unsloth Playwright tests (Chromium flags,
|
|
view-transition killer, page recovery, post-action response wait). Imported
|
|
directly by the standalone scripts; does NOT depend on pytest.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import atexit
|
|
import json
|
|
import os
|
|
import shutil
|
|
import signal
|
|
import socket
|
|
import subprocess
|
|
import sys
|
|
import threading
|
|
import time
|
|
import urllib.error
|
|
import urllib.request
|
|
from collections import deque
|
|
from pathlib import Path
|
|
from typing import Any, Callable
|
|
|
|
FRONTEND = Path(__file__).resolve().parents[2] / "studio" / "frontend"
|
|
_LIVE_SERVERS: list[subprocess.Popen[str]] = []
|
|
_PREV_HANDLERS: dict[int, Any] = {}
|
|
|
|
# Chromium launch args.
|
|
# Throttling flags stop Chromium deprioritising CPU/timers when it thinks the
|
|
# headless window is backgrounded (run 25586583024 stalled inference + render).
|
|
# TranslateUI strips a pointer-intercepting popup; ipc-flooding-protection off
|
|
# lets rapid clicks through during the slider sweep.
|
|
# No `--single-process`. It was darwin-only, for a pipeTransport.js JSON-RPC crash,
|
|
# and it caps Chromium at exactly ONE BrowserContext: opening a second one kills the
|
|
# browser with SIGTRAP, and the next new_page() raises "Target page, context or
|
|
# browser has been closed". Closing a context does it too, even with another still
|
|
# open. That is what made "Update banner layout regression" red on every macos-14 run
|
|
# (it needs a context per viewport), and it is why playwright_chat_ui.py had to keep
|
|
# every step inside one context. Measured with chromium-headless-shell 151: with the
|
|
# flag, a second context dies immediately; without it, 12 open/close cycles pass.
|
|
_BASE_CHROMIUM_ARGS = (
|
|
"--disable-dev-shm-usage",
|
|
"--no-sandbox",
|
|
"--disable-gpu",
|
|
"--disable-background-timer-throttling",
|
|
"--disable-renderer-backgrounding",
|
|
"--disable-backgrounding-occluded-windows",
|
|
"--disable-features=TranslateUI",
|
|
"--disable-ipc-flooding-protection",
|
|
)
|
|
|
|
|
|
def chromium_launch_args(platform: str | None = None) -> list[str]:
|
|
"""Chromium launch args. Same on every platform; `platform` is accepted so
|
|
callers that pass one keep working."""
|
|
del platform
|
|
return list(_BASE_CHROMIUM_ARGS)
|
|
|
|
|
|
# Init script injected into every Playwright context.
|
|
# CSS view-transitions render a full-window pseudo-element that intercepts
|
|
# pointer events after each theme/route swap, so Playwright reports
|
|
# `<html> intercepts pointer events` on the next click. Killing the
|
|
# pseudo-elements + shimming startViewTransition synchronously fixes both.
|
|
# Idempotent and safe to install on every page.
|
|
_VIEW_TRANSITION_KILLER_JS = """
|
|
(function () {
|
|
try {
|
|
const css = `
|
|
::view-transition,
|
|
::view-transition-group(*),
|
|
::view-transition-image-pair(*),
|
|
::view-transition-old(*),
|
|
::view-transition-new(*) {
|
|
display: none !important;
|
|
animation: none !important;
|
|
opacity: 0 !important;
|
|
}
|
|
html, body { pointer-events: auto !important; }
|
|
`;
|
|
const style = document.createElement("style");
|
|
style.id = "playwright-no-view-transition";
|
|
style.textContent = css;
|
|
(document.head || document.documentElement).appendChild(style);
|
|
if (typeof document.startViewTransition === "function") {
|
|
document.startViewTransition = function (cb) {
|
|
try { if (cb) cb(); } catch (e) {}
|
|
return {
|
|
ready: Promise.resolve(),
|
|
finished: Promise.resolve(),
|
|
updateCallbackDone: Promise.resolve(),
|
|
skipTransition: () => {},
|
|
};
|
|
};
|
|
}
|
|
} catch (e) { /* noop */ }
|
|
})();
|
|
"""
|
|
|
|
|
|
def install_view_transition_killer(ctx: Any) -> None:
|
|
"""Inject the CSS view-transition killer into every page in `ctx`."""
|
|
ctx.add_init_script(_VIEW_TRANSITION_KILLER_JS)
|
|
|
|
|
|
# Server health pre-flight.
|
|
# On the macos-14 free runner /api/health can return 200 while /api/auth still
|
|
# 503s (auth DB mid-migration); this in-script probe catches that gap before a
|
|
# 60s change-password timeout.
|
|
|
|
|
|
# The smoke pages are dev-server-only, so each harness owns its server. A backgrounded
|
|
# `npm run dev &` puts the npm WRAPPER in $!, and killing that orphans the node child
|
|
# holding the port and stdout. Hence the process group, stdout drain and SIGKILL escalation.
|
|
|
|
|
|
def drain_process_output(proc: subprocess.Popen[str], sink: deque[str] | None = None) -> None:
|
|
"""Consume vite's output so its pipe cannot fill and wedge; keep the tail for errors."""
|
|
if proc.stdout is not None:
|
|
for line in proc.stdout:
|
|
if sink is not None:
|
|
sink.append(line.rstrip())
|
|
|
|
|
|
def _port_is_taken(port: int, host: str) -> bool:
|
|
with socket.socket() as probe:
|
|
probe.settimeout(1.0)
|
|
return probe.connect_ex((host, port)) == 0
|
|
|
|
|
|
def _stop_live_servers() -> None:
|
|
while _LIVE_SERVERS:
|
|
stop_process(_LIVE_SERVERS.pop())
|
|
|
|
|
|
def _handle_fatal_signal(signum, frame) -> None:
|
|
_stop_live_servers()
|
|
previous = _PREV_HANDLERS.get(signum, signal.SIG_DFL)
|
|
if callable(previous):
|
|
previous(signum, frame)
|
|
return
|
|
signal.signal(signum, signal.SIG_DFL)
|
|
os.kill(os.getpid(), signum)
|
|
|
|
|
|
def _arm_teardown_signals() -> None:
|
|
"""`finally` covers exceptions and SIGINT but not SIGTERM, and a CI cancel sends SIGTERM.
|
|
Without this the server outlives the harness, which is the whole thing being fixed."""
|
|
if _PREV_HANDLERS or os.name == "nt":
|
|
return
|
|
for signum in (signal.SIGTERM, signal.SIGHUP):
|
|
try:
|
|
_PREV_HANDLERS[signum] = signal.signal(signum, _handle_fatal_signal)
|
|
except (ValueError, OSError):
|
|
_PREV_HANDLERS.clear() # not the main thread; leave signals alone
|
|
return
|
|
|
|
|
|
def start_vite(port: int, *, host: str = "127.0.0.1") -> subprocess.Popen[str]:
|
|
"""Start `vite dev` on `port` in its own process group, with stdout drained.
|
|
|
|
Refuses an occupied port. --strictPort would make vite exit anyway, and then the
|
|
readiness poll would be talking to whatever else is listening, not to us.
|
|
"""
|
|
if _port_is_taken(port, host):
|
|
raise RuntimeError(
|
|
f"{host}:{port} is already serving. Stop it, or move this harness with SMOKE_PORT."
|
|
)
|
|
process_group = (
|
|
{"creationflags": subprocess.CREATE_NEW_PROCESS_GROUP}
|
|
if os.name == "nt"
|
|
else {"start_new_session": True}
|
|
)
|
|
# shutil.which honours PATHEXT, so this resolves npm.cmd on Windows. CreateProcess cannot
|
|
# run a .cmd directly, so a bare "npm" is a FileNotFoundError there.
|
|
npm = shutil.which("npm") or "npm"
|
|
proc = subprocess.Popen(
|
|
[npm, "run", "dev", "--", "--host", host, "--port", str(port), "--strictPort"],
|
|
cwd = FRONTEND,
|
|
stdout = subprocess.PIPE,
|
|
stderr = subprocess.STDOUT,
|
|
text = True,
|
|
**process_group,
|
|
)
|
|
tail: deque[str] = deque(maxlen = 20)
|
|
proc.vite_tail = tail # type: ignore[attr-defined]
|
|
threading.Thread(target = drain_process_output, args = (proc, tail), daemon = True).start()
|
|
_LIVE_SERVERS.append(proc)
|
|
_arm_teardown_signals()
|
|
atexit.register(_stop_live_servers)
|
|
return proc
|
|
|
|
|
|
def stop_process(proc: subprocess.Popen[str]) -> None:
|
|
"""SIGTERM the process group, escalating to SIGKILL if it does not go."""
|
|
if proc in _LIVE_SERVERS:
|
|
_LIVE_SERVERS.remove(proc)
|
|
if proc.poll() is not None:
|
|
return
|
|
|
|
if os.name == "nt":
|
|
subprocess.run(
|
|
["taskkill", "/PID", str(proc.pid), "/T"],
|
|
check = False,
|
|
stdout = subprocess.DEVNULL,
|
|
stderr = subprocess.DEVNULL,
|
|
)
|
|
else:
|
|
try:
|
|
os.killpg(proc.pid, signal.SIGTERM)
|
|
except ProcessLookupError:
|
|
return
|
|
|
|
try:
|
|
proc.wait(timeout = 10)
|
|
except subprocess.TimeoutExpired:
|
|
if os.name == "nt":
|
|
subprocess.run(
|
|
["taskkill", "/PID", str(proc.pid), "/T", "/F"],
|
|
check = False,
|
|
stdout = subprocess.DEVNULL,
|
|
stderr = subprocess.DEVNULL,
|
|
)
|
|
else:
|
|
try:
|
|
os.killpg(proc.pid, signal.SIGKILL)
|
|
except ProcessLookupError:
|
|
pass
|
|
# Called from a `finally`: never raise over the failure that sent us here.
|
|
try:
|
|
proc.wait(timeout = 10)
|
|
except subprocess.TimeoutExpired:
|
|
pass
|
|
|
|
|
|
def wait_for_smoke_page(
|
|
url: str,
|
|
entry: str,
|
|
*,
|
|
proc: subprocess.Popen[str] | None = None,
|
|
timeout_s: float = 120.0,
|
|
info: Callable[[str], None] | None = None,
|
|
) -> None:
|
|
"""Block until `url` serves a page that really references `entry`.
|
|
|
|
Vite's SPA fallback answers 200 with index.html for any path it cannot resolve, so a
|
|
deleted smoke page still looks healthy. Match the module specifier, not the status.
|
|
"""
|
|
deadline = time.monotonic() + timeout_s
|
|
last = "no response"
|
|
while time.monotonic() < deadline:
|
|
# Ours died (busy port, missing node_modules): stop instead of polling out the timeout.
|
|
if proc is not None and proc.poll() is not None:
|
|
tail = "\n".join(getattr(proc, "vite_tail", []))
|
|
raise RuntimeError(
|
|
f"vite exited with code {proc.returncode} before serving {url}\n{tail}"
|
|
)
|
|
try:
|
|
with urllib.request.urlopen(url, timeout = 3.0) as r:
|
|
body = r.read().decode("utf-8", errors = "replace")
|
|
if r.status == 200 and entry in body:
|
|
if info is not None:
|
|
info(f"{url} ready (serves {entry})")
|
|
return
|
|
last = (
|
|
f"status={r.status}, {entry} "
|
|
f"{'present' if entry in body else 'MISSING (SPA fallback?)'}"
|
|
)
|
|
except Exception as exc:
|
|
last = f"{type(exc).__name__}: {exc}"
|
|
time.sleep(0.5)
|
|
raise RuntimeError(f"vite did not serve {url} referencing {entry} within {timeout_s}s ({last})")
|
|
|
|
|
|
def _http_get_status_and_body(url: str, timeout: float) -> tuple[int, dict | None]:
|
|
try:
|
|
with urllib.request.urlopen(url, timeout = timeout) as r:
|
|
try:
|
|
body = json.loads(r.read().decode("utf-8", errors = "replace"))
|
|
except Exception:
|
|
body = None
|
|
return r.status, body
|
|
except urllib.error.HTTPError as exc:
|
|
return exc.code, None
|
|
except Exception:
|
|
return -1, None
|
|
|
|
|
|
def wait_for_health(
|
|
base_url: str,
|
|
*,
|
|
timeout: float = 30.0,
|
|
info: Callable[[str], None] | None = None,
|
|
) -> bool:
|
|
"""Poll {base_url}/api/health until status==200; True on success, False on
|
|
timeout, never raises. Diagnostic only (the workflow's wait is authoritative)."""
|
|
deadline = time.monotonic() + timeout
|
|
last_status: int | None = None
|
|
last_body: dict | None = None
|
|
while time.monotonic() < deadline:
|
|
status, body = _http_get_status_and_body(
|
|
f"{base_url}/api/health",
|
|
timeout = 3.0,
|
|
)
|
|
last_status, last_body = status, body
|
|
# Accept any 200 -- different Unsloth builds report status differently.
|
|
if status == 200:
|
|
if info is not None:
|
|
info(f"health pre-flight OK: status=200, body keys={list((body or {}).keys())}")
|
|
return True
|
|
time.sleep(0.5)
|
|
if info is not None:
|
|
info(
|
|
f"health pre-flight TIMED OUT after {timeout}s; "
|
|
f"last_status={last_status}, last_body={last_body!r}"
|
|
)
|
|
return False
|
|
|
|
|
|
# Page recovery: if the page died mid-test, open a fresh one in the same context
|
|
# (localStorage auth survives); otherwise leave it alone. Optionally re-navigates.
|
|
|
|
|
|
def recover_or_replace_page(
|
|
page: Any,
|
|
ctx: Any,
|
|
*,
|
|
default_timeout_ms: int = 60_000,
|
|
goto_url: str | None = None,
|
|
settle_networkidle: bool = True,
|
|
info: Callable[[str], None] | None = None,
|
|
) -> Any:
|
|
"""Return a usable page, replacing `page` if closed; optionally navigate to
|
|
`goto_url`. Recovery errors are logged and swallowed for the caller to retry."""
|
|
try:
|
|
if page.is_closed():
|
|
page = ctx.new_page()
|
|
page.set_default_timeout(default_timeout_ms)
|
|
except Exception as exc:
|
|
if info is not None:
|
|
info(f"recovery: page.is_closed() check failed: {exc!r}")
|
|
if goto_url is not None:
|
|
try:
|
|
page.goto(goto_url, wait_until = "domcontentloaded", timeout = default_timeout_ms)
|
|
if settle_networkidle:
|
|
try:
|
|
page.wait_for_load_state("networkidle", timeout = 30_000)
|
|
except Exception:
|
|
pass
|
|
except Exception as exc:
|
|
if info is not None:
|
|
info(f"recovery: page.goto({goto_url!r}) failed: {exc!r}")
|
|
return page
|
|
|
|
|
|
# ─────────────────────────────────────────────────────────────────────
|
|
# POST-and-wait: surface server errors immediately, fall back cleanly.
|
|
# ─────────────────────────────────────────────────────────────────────
|
|
|
|
|
|
def click_and_wait_for_response(
|
|
page: Any,
|
|
*,
|
|
url_substr: str,
|
|
method: str = "POST",
|
|
do_click: Callable[[], None],
|
|
timeout_ms: int = 30_000,
|
|
info: Callable[[str], None] | None = None,
|
|
) -> tuple[int | None, Exception | None]:
|
|
"""Click + wait for the matching XHR/fetch response; (status, None) on success
|
|
or (None, exception) on capture failure. Falls back to a fire-and-forget click
|
|
so the outer retry loop runs. Callers check `status >= 400`."""
|
|
try:
|
|
with page.expect_response(
|
|
lambda r: url_substr in r.url and r.request.method == method,
|
|
timeout = timeout_ms,
|
|
) as resp_info:
|
|
do_click()
|
|
resp = resp_info.value
|
|
return resp.status, None
|
|
except Exception as exc:
|
|
if info is not None:
|
|
info(
|
|
f"click_and_wait_for_response({url_substr!r}, {method}) failed: "
|
|
f"{type(exc).__name__}: {str(exc)[:150]}; falling back to fire-and-forget click"
|
|
)
|
|
try:
|
|
do_click()
|
|
except Exception:
|
|
pass
|
|
return None, exc
|
|
|
|
|
|
# Console-error / page-error filtering.
|
|
# - BENIGN_PAGE_ERROR_PATTERNS: CI-infra JS errors with no user-visible effect;
|
|
# the page-error gate must not count these.
|
|
# - BENIGN_CONSOLE_ERROR_PATTERNS: same-cause console.error events, used only to
|
|
# filter noise from diagnostic dumps (tests don't gate on console.error).
|
|
|
|
BENIGN_PAGE_ERROR_PATTERNS: tuple[str, ...] = (
|
|
"Request failed (422)",
|
|
"Failed to fetch",
|
|
"NetworkError",
|
|
"Load failed",
|
|
"At least one non-system message is required",
|
|
"An internal error occurred",
|
|
)
|
|
|
|
BENIGN_CONSOLE_ERROR_PATTERNS: tuple[str, ...] = (
|
|
# macos-14 buffer exhaustion; the test catches the underlying request
|
|
# failure via expect_response and retries.
|
|
"net::ERR_NO_BUFFER_SPACE",
|
|
# Intentional fetch aborts (unmount, route change) log a console.error.
|
|
"AbortError",
|
|
"The user aborted a request",
|
|
# Lazy chunk no longer needed because the user navigated away mid-load.
|
|
"Loading chunk",
|
|
# Also a benign page-error; here for the diagnostic dump path.
|
|
"Failed to fetch",
|
|
)
|
|
|
|
|
|
def is_benign_page_error(msg: str) -> bool:
|
|
return any(p in msg for p in BENIGN_PAGE_ERROR_PATTERNS)
|
|
|
|
|
|
def is_benign_console_error(msg: str) -> bool:
|
|
return any(p in msg for p in BENIGN_CONSOLE_ERROR_PATTERNS)
|
|
|
|
|
|
# ─────────────────────────────────────────────────────────────────────
|
|
# Diagnostic dump.
|
|
# ─────────────────────────────────────────────────────────────────────
|
|
|
|
|
|
def dump_diagnostics(
|
|
page: Any,
|
|
art_dir: Path | str,
|
|
name: str,
|
|
*,
|
|
info: Callable[[str], None] | None = None,
|
|
extra: dict | None = None,
|
|
) -> None:
|
|
"""Write a screenshot + JSON sidecar (URL/title/body/storage) under art_dir.
|
|
Diagnostic only, never raises; both best-effort."""
|
|
art = Path(art_dir)
|
|
try:
|
|
art.mkdir(parents = True, exist_ok = True)
|
|
except Exception:
|
|
pass
|
|
try:
|
|
page.screenshot(
|
|
path = str(art / f"{name}.png"),
|
|
full_page = True,
|
|
timeout = 90_000,
|
|
animations = "disabled",
|
|
)
|
|
except Exception as exc:
|
|
if info is not None:
|
|
info(f"diagnostics: screenshot {name} failed: {exc}")
|
|
payload: dict[str, Any] = {"name": name, "ts": time.time()}
|
|
try:
|
|
payload["url"] = page.url
|
|
except Exception:
|
|
payload["url"] = "<page closed>"
|
|
try:
|
|
payload["title"] = page.title()
|
|
except Exception:
|
|
pass
|
|
try:
|
|
payload["body_excerpt"] = page.evaluate(
|
|
"""() => (document.body && document.body.innerText || '').slice(0, 800)""",
|
|
)
|
|
except Exception:
|
|
pass
|
|
try:
|
|
payload["local_storage_keys"] = page.evaluate(
|
|
"""() => Object.keys(localStorage)""",
|
|
)
|
|
except Exception:
|
|
pass
|
|
if extra:
|
|
payload["extra"] = extra
|
|
try:
|
|
(art / f"{name}.json").write_text(
|
|
json.dumps(payload, indent = 2, default = str),
|
|
encoding = "utf-8",
|
|
)
|
|
except Exception as exc:
|
|
if info is not None:
|
|
info(f"diagnostics: json sidecar {name} failed: {exc}")
|
|
|
|
|
|
# Markers for the transient Playwright error raised when a navigation, reload, or
|
|
# auth refresh destroys the JS execution context while an evaluate is in flight.
|
|
# Stored lowercase and matched against a lowercased message: Playwright varies the
|
|
# casing across versions ("Frame was detached" vs "frame was detached"), so a
|
|
# case-sensitive check would miss the very races this is meant to catch.
|
|
_CONTEXT_LOST_MARKERS = (
|
|
"execution context was destroyed",
|
|
"context with specified id",
|
|
"frame was detached",
|
|
"target closed",
|
|
"target page, context or browser has been closed",
|
|
"execution context is not available",
|
|
)
|
|
|
|
# HTTP methods whose replay is side-effect-free, so an evaluate_fetch hit by a
|
|
# mid-call context loss may safely re-run. Mutating methods are excluded by
|
|
# default (see evaluate_fetch) to avoid double-applying an already-sent request.
|
|
_IDEMPOTENT_METHODS = frozenset({"GET", "HEAD", "OPTIONS"})
|
|
|
|
|
|
# Robust page/locator.evaluate.
|
|
# A navigation mid-evaluate destroys the execution context and raises at the Python
|
|
# level (not a JS result), which would crash the script. Retry that transient class
|
|
# within a small budget, settling the page first; non-transient or persistent errors
|
|
# still propagate.
|
|
def robust_evaluate(
|
|
target: Any,
|
|
expression: str,
|
|
arg: Any = None,
|
|
*,
|
|
retries: int = 2,
|
|
backoff_ms: int = 250,
|
|
) -> Any:
|
|
"""`target.evaluate(expression, arg)` for a Page or Locator, retried when a
|
|
concurrent navigation destroys the execution context. Re-raises on a
|
|
non-transient error or after the final attempt."""
|
|
page = target if hasattr(target, "wait_for_load_state") else getattr(target, "page", None)
|
|
attempts = max(1, int(retries) + 1)
|
|
for attempt in range(attempts):
|
|
try:
|
|
return target.evaluate(expression, arg)
|
|
except Exception as exc:
|
|
exc_msg = str(exc).lower()
|
|
transient = any(s in exc_msg for s in _CONTEXT_LOST_MARKERS)
|
|
if not transient or attempt == attempts - 1:
|
|
raise
|
|
try:
|
|
sys.stderr.write(
|
|
f"[robust_evaluate] execution context lost "
|
|
f"({attempt + 1}/{attempts}); settling + retrying\n"
|
|
)
|
|
sys.stderr.flush()
|
|
except Exception:
|
|
pass
|
|
if page is not None:
|
|
try:
|
|
page.wait_for_load_state("domcontentloaded", timeout = 10_000)
|
|
except Exception:
|
|
pass
|
|
time.sleep((backoff_ms * (2**attempt)) / 1000.0)
|
|
|
|
|
|
# Bounded in-page fetch.
|
|
# `page.evaluate(...)` has no `timeout=`, so a stuck fetch hangs the script until
|
|
# the runner timeout (run 25696797934 / PR #5387 burned 27+ min). evaluate_fetch
|
|
# wraps the fetch in an AbortController.signal so the JS side always resolves --
|
|
# real response, or synthetic `{status: 0, error: "AbortError..."}` after timeout_ms.
|
|
# It also retries the evaluate itself when a navigation destroys the execution
|
|
# context mid-call (a transient Playwright race, not a real fetch failure).
|
|
def evaluate_fetch(
|
|
page: Any,
|
|
url: str,
|
|
*,
|
|
method: str = "GET",
|
|
headers: dict[str, str] | None = None,
|
|
body: Any = None,
|
|
timeout_ms: int = 20_000,
|
|
transport_retries: int = 2,
|
|
transport_backoff_ms: int = 250,
|
|
retry_on_context_loss: bool | None = None,
|
|
) -> dict[str, Any]:
|
|
"""Run `fetch(url, opts)` in the page with an AbortSignal deadline; returns
|
|
`{"status", "body", "error"}` (status==0 + AbortError on timeout). Treat
|
|
status==0 or non-None error as transport failure. `body` may be str (verbatim)
|
|
or dict/list (JSON-encoded); pass headers explicitly for Content-Type/Auth.
|
|
|
|
`retry_on_context_loss` controls whether a navigation that destroys the JS
|
|
context mid-call replays the in-page fetch. The request may have already
|
|
reached the backend before the context died, so replaying a mutating call is
|
|
unsafe: a spent single-use POST /api/auth/refresh comes back 401, and a
|
|
duplicate POST /api/inference/load that lands while the first is still in
|
|
`loading_models` is rejected (the backend returns False -> 500) even though
|
|
the original load succeeds. Default (None) therefore retries only idempotent
|
|
reads (GET/HEAD/OPTIONS) and never replays a mutating method; pass an explicit
|
|
bool to override per call. Context loss on a non-retried call propagates."""
|
|
body_arg: str | None
|
|
if body is None:
|
|
body_arg = None
|
|
elif isinstance(body, (str, bytes)):
|
|
body_arg = body if isinstance(body, str) else body.decode("utf-8")
|
|
else:
|
|
body_arg = json.dumps(body)
|
|
js = """
|
|
async ({url, method, headers, body, timeoutMs}) => {
|
|
const ctrl = new AbortController();
|
|
const t = setTimeout(() => ctrl.abort(), timeoutMs);
|
|
try {
|
|
const opts = {method: method, headers: headers, signal: ctrl.signal};
|
|
if (body !== null) opts.body = body;
|
|
const r = await fetch(url, opts);
|
|
clearTimeout(t);
|
|
let parsed;
|
|
try {
|
|
parsed = await r.json();
|
|
} catch (_e) {
|
|
try {
|
|
parsed = await r.text();
|
|
} catch (_e2) {
|
|
parsed = null;
|
|
}
|
|
}
|
|
return {status: r.status, body: parsed, error: null};
|
|
} catch (e) {
|
|
clearTimeout(t);
|
|
return {status: 0, body: null, error: String(e)};
|
|
}
|
|
}
|
|
"""
|
|
payload = {
|
|
"url": url,
|
|
"method": method,
|
|
"headers": headers or {},
|
|
"body": body_arg,
|
|
"timeoutMs": int(timeout_ms),
|
|
}
|
|
# Retry transport failures only: status != 0 (real HTTP) and AbortError
|
|
# (caller's deadline) propagate; status==0 (stale-keepalive / "Failed to
|
|
# fetch" after auth rotation) retries after backoff to evict the dead socket.
|
|
last: dict[str, Any] | None = None
|
|
attempts = max(1, int(transport_retries) + 1)
|
|
# Replay the in-page evaluate on a context loss only for idempotent reads;
|
|
# mutating methods (POST/PUT/PATCH/DELETE) may have already hit the backend,
|
|
# so retrying would re-send them (see docstring). Honor an explicit override.
|
|
if retry_on_context_loss is None:
|
|
retry_on_context_loss = method.upper() in _IDEMPOTENT_METHODS
|
|
ctx_retries = 2 if retry_on_context_loss else 0
|
|
for attempt in range(attempts):
|
|
# robust_evaluate retries the evaluate when a navigation destroys the
|
|
# execution context mid-call; the loop here retries transport failures.
|
|
result = robust_evaluate(
|
|
page, js, payload, retries = ctx_retries, backoff_ms = transport_backoff_ms
|
|
)
|
|
last = result
|
|
try:
|
|
status = int(result.get("status") or 0)
|
|
except (TypeError, ValueError):
|
|
status = 0
|
|
if status != 0:
|
|
return result
|
|
err = str(result.get("error") or "")
|
|
if "AbortError" in err:
|
|
return result
|
|
if attempt < attempts - 1:
|
|
wait_ms = transport_backoff_ms * (2**attempt)
|
|
try:
|
|
sys.stderr.write(
|
|
f"[evaluate_fetch] {method} {url}: transport failure "
|
|
f"({attempt + 1}/{attempts}, err={err!r}); "
|
|
f"retrying in {wait_ms}ms\n"
|
|
)
|
|
sys.stderr.flush()
|
|
except Exception:
|
|
pass
|
|
time.sleep(wait_ms / 1000.0)
|
|
return last or {"status": 0, "body": None, "error": "no attempt made"}
|
|
|
|
|
|
# Wall-clock watchdog.
|
|
# A browser wedge (CPU-pinned JS, silent renderer crash, asyncio deadlock) can
|
|
# still hang the script. A daemon Timer calls os._exit(2) after deadline_s; exit
|
|
# code 2 lets the workflow's `set -e` propagate. Pick deadline_s above the
|
|
# slowest healthy run (macos-14 cold cache ~7-9 min) but under the 30-min cap.
|
|
def install_wall_clock_watchdog(
|
|
deadline_s: float,
|
|
*,
|
|
label: str = "playwright",
|
|
info: Callable[[str], None] | None = None,
|
|
) -> threading.Timer:
|
|
"""Start a daemon Timer that hard-exits the process at `deadline_s`; returned
|
|
so the caller can `.cancel()` on clean exit (daemonised, dies with process)."""
|
|
|
|
def _kaboom() -> None:
|
|
msg = (
|
|
f"[{label}] WATCHDOG: hit {deadline_s:.0f}s wall-clock "
|
|
f"deadline; forcing exit(2). The script wedged somewhere "
|
|
f"the per-action timeouts could not bound. Inspect the "
|
|
f"most recent step printed above to localise."
|
|
)
|
|
try:
|
|
sys.stderr.write(msg + "\n")
|
|
sys.stderr.flush()
|
|
except Exception:
|
|
pass
|
|
os._exit(2)
|
|
|
|
timer = threading.Timer(deadline_s, _kaboom)
|
|
timer.daemon = True
|
|
timer.start()
|
|
if info is not None:
|
|
info(f"watchdog armed: hard-exit at {deadline_s:.0f}s")
|
|
return timer
|