vrr/unsloth
2
0
Fork 0
mirror of https://github.com/unslothai/unsloth.git synced 2026-08-24 00:04:14 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 30
unsloth/tests/studio/playwright_settings_tabs.py
Daniel Han 1c3dde199b
Drain the sampling debounce long enough for the node CI actually runs (#9332) 
* Stop Frontend CI installing Chromium's system libraries through apt every run

`playwright install --with-deps chromium` runs its own `apt-get update`
inside itself, so it bypassed everything CI has learned about apt: the shared
retry helper's 20s transfer cap, APT_ACQUIRE_RETRIES: '0', and the archive
cache. The job failed 3 of 8 runs on main.

Job 96072994354 (main, 2026-08-19): 9 packages, 21.1 MB, and
`fonts-wqy-zenhei [7472 kB]` alone took 5m50s off azure.archive.ubuntu.com.
Both 420s attempts died mid-download. That is the same mirror and the same
package that took the webkit shards down in #9289. Attempt 2 logged
"Need to get 8833 kB/21.1 MB", so apt does resume partials across attempts and
still could not finish.

Split the way studio-ui-smoke.yml splits it: download the engine, launch it to
find out whether the system libraries are actually missing, and run
`install-deps` only if they are. ubuntu-latest is a browser-testing image and
usually ships them, so the common path now runs no apt at all.

The browser and apt-archive cache keys are deliberately identical to the
chromium-only shards in studio-ui-smoke.yml (engine token `c`): same image,
same Playwright version, same single engine, so the entry is shared rather than
duplicated against a budget measured at 99.3% full.

The step's authorised worst case doubles with the second helper call, to
2 x (2 x 420s + 125s) = 1930s, so its timeout goes 17m -> 33m and stays under
the job's 40m. Both guarded calls are skipped on the common path.

Guard: tests/studio/test_playwright_install_avoids_with_deps.py fails the build
if `--with-deps` returns to any workflow, and is wired into
workflow-trigger-lint, the only job with no paths filter.

* Drain the sampling debounce long enough for the node CI actually runs

Frontend CI has been red on main since #9055, not intermittently: eight
consecutive main runs failed at `Unit tests`, every one on node v22.23.2.

The three suites from #9055 wait for a debounced write with a fixed drain --
three rounds of tick(1000) plus six setImmediate turns -- and then assert. Three
rounds is enough on node 24, which is what a dev box happens to have, and is not
enough on node 22, which `setup-node: 22` resolves to. The same chain drains far
fewer continuations per round there, so the write had not landed when the
assertion ran.

Reproduced by downloading node 22.23.2 and running the suites against both.
Measured on the compat suite:

  rounds   3    10    30    60
  failing  7     5     1     0

The compat suite reported it as a missing value (expected 1.37, actual
undefined). The simulation suite reported it as an ORDERING violation -- "chat A
temperature: owed 0.6, shows 1.37" -- because a scenario whose write has not
landed looks exactly like one that wrote the wrong value, which is why this read
as a feature bug rather than a slow test.

There were three copies of the drain: one per suite plus the shared
thread-sampling-world helper the simulations run through. Fixing only the two
suites left the simulations red, since their scenarios drain through the helper.

The bound is 200, generous rather than tuned to the observed 60, and settle()
now takes an optional `until` predicate: it returns as soon as the caller can
see the work, and throws naming itself if the condition never holds, so the next
slow runtime reports "settle gave up" instead of an assertion on a missing value.

Verified: node 22.23.2 compat 16/16 and simulation 18/18 (both were failing);
node 24 full frontend suite 4060/4060. Only the simulation suite imports the
helper, and no fixed three-round drain remains in tests/.

* Raise the scenario drain to 600: 200 was marginal, and Windows needed more

At 200 the simulations were 0 failing on one run and 1 on the next on the same
machine, and the Windows runner -- slower again -- still had 2 of 120 orderings
short. 600 is 0 failing across three consecutive runs, at 107s against 52s.

Also records the adaptive version that was tried and is wrong here, so it is not
tried again: the rows only change WHEN the write lands, so 'rows have stopped
changing' is precisely the pending state being waited through. Quiescence on
that observable stops early by construction and scored 4 failures where the
fixed bound scored 0.

* Stop the settings smoke asserting a tab count a new page invalidates

Unskipping the browser smokes surfaced this immediately: the blocked-panel run
failed with 'blocking the data panel took the dialog down' while its own report
said dialog: True. Nothing had taken the dialog down. The check was

    if not state["dialog"] or state["nav"] != 12:

and the keyboard-shortcuts page had made the nav 13. A stale constant, reading
as an error-handling regression.

The nav size is now read before the panel is blocked and compared against
itself, which is the invariant that was meant: blocking a panel must not
collapse the dialog, whatever size the dialog is.

The same drift had a quieter half. The smoke's TABS list still had twelve
entries, so keyboard-shortcuts had no browser coverage at all and the smoke
stayed green without it. It is added here, and
tests/studio/test_settings_smoke_covers_every_tab.py pins both directions
against settings-dialog.tsx so the next page cannot go uncovered silently. It
also checks the workflow's PW_CHUNK_FAIL names a tab that exists -- that value
lives in studio-frontend-ci.yml, not in the smoke, and a rename would leave the
run blocking nothing while still reporting PASS.

Wired into workflow-trigger-lint, the only job with no paths filter, because it
reads a workflow. Mutation-tested both ways: dropping the tab from TABS and
restoring the literal count each turn it red.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Drain the sampling suites on the loader too, and fail loudly when it gives up

The fixed round count in the previous commit was still a guess, and the Windows
job proved it: the SAME commit at 600 rounds passed one run and failed the next
with "A1: 2 violation(s) across 120 orderings", reporting stale reads as lost
edits. Raising the number again was not the answer.

The pending work has a second half nothing was waiting on. The store's
thread-scoped write ends in `await import("../utils/chat-history-storage")`
(chat-runtime-store.ts:1326 and :1754), and these suites register() a resolver
hook, which routes that import through the hooks thread. Three repeat imports of
an already-loaded module:

  v24.14.0  no hook 1, 1, 1 turns    hook registered 1, 1, 1
  v22.23.2  no hook 1, 1, 1 turns    hook registered 6, 3, 35

That is the whole green-locally / red-on-CI split, and it is why a loaded Windows
runner fails what the same commit passed an hour earlier: the pending work is a
message to another thread, so its cost is scheduling latency, not instructions.
No round count is correct for that.

Counting the mocked timers alone does not cover it either, which is worth
recording since it is the obvious next idea. With the counter installed and 25
consecutive quiet rounds per drain, 150 macrotask turns of nothing, v22.23.2
still lost 7 orderings across A1 and A3, every one a write that had not landed.

So drain on both observables. tests/helpers/mock-timer-drain.ts wraps the MOCKED
setTimeout with a counter, giving an exact count of timers scheduled and not yet
fired or cleared, and each round also issues its own import and waits for it, so
the wait scales with the loader instead of guessing at it. The drain returns when
no timer is outstanding and three consecutive rounds neither scheduled nor fired
one. With the probe, three quiet rounds is green on v22 and v24 alike.

The generous bound stays as a BACKSTOP that THROWS and names what was still
outstanding, which is the change that matters most here. An under-drain used to
be indistinguishable from the store losing an edit, so it sent the investigation
into the store for two rounds. Now it says so:

  settle: drain exhausted after 2 rounds, with no timer pending but work still
  scheduling or firing within the last 3 rounds. Nothing read after this point is
  trustworthy: a queued write has not landed, so the store still shows the
  PREVIOUS value, which reads as a wrong value rather than a missing one. Fix the
  work or raise the backstop; do not read this as the store losing an edit.

It is also much faster, because it stops when the work is done rather than after
600 rounds regardless. A typical drain now takes 4 rounds; instrumented over 840
drains in A1 the maximum was 4.

  simulation  v24 101.7s -> 30s      v22.23.2 106.5s -> 29s
  compat      v24 0.54s -> 0.46s     v22.23.2 0.61s -> 0.52s

The three copies of the drain shape, one in the world helper and one in each test
file, are now one helper. The dead end already recorded is kept next to the new
measurements: quiescence on the store ROWS is still wrong, because the rows only
change WHEN the write lands.

Verified: both suites green twice each on v24.14.0 and on v22.23.2, the version
setup-node 22 resolves to. Full frontend suite 4080 passed, 0 failed. The
exhaustion throw was confirmed by forcing the backstop to 2, which produces the
message above and no wrong-value violation. A deliberately broken
sanitizeThreadScopedSettings still produces a real ordering violation on both
runtimes, so the drain is not exiting early. Test files only.

* Read the workflow and the dialog as UTF-8, not as the platform default

test_source_read_encoding caught four read_text() calls this PR added with no
encoding. It is right and they are a real defect: the guard reads
settings-dialog.tsx, playwright_settings_tabs.py and studio-frontend-ci.yml, and
on a Windows runner Path.read_text() uses the ANSI code page, so any non-ASCII
byte in any of them raises UnicodeDecodeError. The whole point of this guard is
that a settings page can be added without anyone noticing; a guard that cannot
be collected on Windows fails the same way.

Repo tests (CPU) was otherwise clean: 1 failed, 8781 passed.

* Tell a pwsh crash apart from install.ps1 losing its exit code

test_the_pwsh_filter_keeps_the_log_clean_and_the_exit_code_intact went red on a
hosted ubuntu runner with completely empty stdout and pwsh's own banner: "An
error has occurred that was not properly handled. Additional information is shown
below. The PowerShell process will exit." The interpreter died; the script never
ran. Workflow trigger lint is green on the last 8 main runs and this PR touches
no PowerShell, so it is the runner, not the repo.

Read as an ordinary failure it says "$LASTEXITCODE did not survive the added
pipeline stages, so a failing install.ps1 would leave its step green" -- an
accusation against the installer, raised by a run that produced no evidence
either way. That is the same shape as the drain in this PR: an environment
shortfall wearing the costume of a product regression.

_run_pwsh retries only that case, and the distinction is what keeps it honest:
a run that reaches the `RC=` line is returned on the first attempt whatever the
value, so a genuine regression can never be retried into green. Only a run with
no RC= AND the crash banner is retried, because it carries no verdict to
preserve. If both attempts crash it fails with a message naming the interpreter,
not install.ps1.

17 passed. Exercised _run_pwsh against a script that prints the banner and no
RC=: it raises, so the branch is not vacuous.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-08-19 23:55:01 -07:00

532 lines
22 KiB
Python
Raw Permalink Blame History

# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""Settings dialog behaviour harness: every tab renders, search jumps land, deep-open works.
Drives smoke-settings.html (the real SettingsDialog and store, no backend), so a static-import
tree and a React.lazy one are directly comparable. Emits a JSON report to $PW_OUT for a
field-by-field diff.
PW_ENGINE=chromium PW_PORT=5399 PW_OUT=out.json python tests/studio/playwright_settings_tabs.py
PW_CHUNK_DELAY_MS delays every tab module response, widening the window a lazy panel is
in-flight for; PW_CHUNK_FAIL=<tab> aborts one tab's module outright.
"""
import json
import os
import re
import sys
import time
from pathlib import Path
from playwright.sync_api import sync_playwright
sys.path.insert(0, str(Path(__file__).resolve().parent))
from _playwright_robust import ( # noqa: E402
chromium_launch_args,
start_vite,
stop_process,
click_forced,
)
TABS = [
"general",
"profile",
"appearance",
"resources",
"chat",
"voice",
"connections",
"data",
"api-keys",
"agents",
"keyboard-shortcuts",
"debugging",
"about",
]
ENGINE = os.environ.get("PW_ENGINE", "chromium")
PORT = int(os.environ.get("PW_PORT", "5399"))
OUT = Path(os.environ.get("PW_OUT", "logs/settings_tabs_report.json"))
CHUNK_DELAY_MS = int(os.environ.get("PW_CHUNK_DELAY_MS", "0"))
CHUNK_FAIL = os.environ.get("PW_CHUNK_FAIL", "")
SETTLE_MS = 600
SETTLE_TIMEOUT_S = 25.0
# The panel scroller inside the dialog, the element `mainScrollRef` points at.
PANEL = 'div[role="dialog"] main div.hover-scrollbar'
# How long the Data module is held, and how far into that hold the deep-open is abandoned.
DEEP_OPEN_HOLD_MS = 2500
DEEP_OPEN_ABANDON_MS = 300
# The Data panel's own module, however the server spells it (vite appends ?t=, ?v=).
DATA_MODULE = re.compile(r"/data-tab(\.tsx)?(\?|$)")
report: dict = {
"engine": ENGINE,
"chunk_delay_ms": CHUNK_DELAY_MS,
"chunk_fail": CHUNK_FAIL,
"steps": [],
"tabs": {},
"failures": [],
}
def log(msg: str) -> None:
print(f"[settings-tabs] {msg}", flush = True)
def fail(msg: str) -> None:
log(f"FAIL {msg}")
report["failures"].append(msg)
SNAPSHOT_JS = """(sel) => {
const root = document.querySelector(sel);
if (!root) return { present: false, sig: 'none' };
const labels = [...root.querySelectorAll('[data-settings-label]')]
.map((el) => el.dataset.settingsLabel);
const text = (root.innerText || '').trim();
return {
present: true,
labels,
elements: root.querySelectorAll('*').length,
textLength: text.length,
// Identity of what is on screen, insensitive to live values (versions, sizes).
sig: labels.join('|') + '#' + root.querySelectorAll('*').length,
};
}"""
def snapshot(page) -> dict:
return page.evaluate(SNAPSHOT_JS, PANEL)
def settle(page, timeout_s: float = SETTLE_TIMEOUT_S) -> dict:
"""Poll until the panel signature holds still for SETTLE_MS, then return it."""
deadline = time.time() + timeout_s
last = snapshot(page)
stable_since = time.time()
while time.time() < deadline:
page.wait_for_timeout(50)
now = snapshot(page)
if now.get("sig") != last.get("sig"):
last = now
stable_since = time.time()
continue
if (time.time() - stable_since) * 1000 >= SETTLE_MS:
return now
last["settle_timeout"] = True
return last
def settle_panel(page, timeout_s: float = SETTLE_TIMEOUT_S) -> dict:
"""Settle, but do not accept a placeholder as the answer.
The panel renders from a deferred value, so the outgoing content stays on screen until
the incoming one is ready, and under load that hand-off can hold still past SETTLE_MS.
"""
deadline = time.time() + timeout_s
latest = settle(page, timeout_s = timeout_s)
while latest.get("elements", 0) < 5 and time.time() < deadline:
page.wait_for_timeout(100)
latest = settle(page, timeout_s = max(1.0, deadline - time.time()))
return latest
def click_tab_and_observe(page, tab: str) -> dict:
"""Click a tab; record how long the panel keeps the old content and whether it blanks."""
before = snapshot(page)
click_forced(page.locator(f'[data-testid="settings-tab-{tab}"]'), timeout = 15000)
started = time.time()
changed_ms = None
blank_frames = 0
frames = 0
deadline = started + SETTLE_TIMEOUT_S
while time.time() < deadline:
now = snapshot(page)
frames += 1
if now.get("present") and now.get("elements", 0) < 5:
blank_frames += 1
if changed_ms is None and now.get("sig") != before.get("sig"):
changed_ms = round((time.time() - started) * 1000)
break
page.wait_for_timeout(20)
final = settle_panel(page)
return {
"before_sig": before.get("sig"),
"changed_ms": changed_ms,
"blank_frames": blank_frames,
"frames_sampled": frames,
"settled": final,
}
def require_harness(page) -> None:
"""Fail with the cause rather than a selector timeout if the page has moved on.
Vite dev proxies /api to 127.0.0.1:8888. With a real Studio listening there and no
token, those calls come back 401 and the app's auth handling navigates, which takes
the harness's window with it. Every later step then times out on a dialog that cannot
exist. This is not the dialog's doing: it happens on main too, where the harness is
unmounted before the first open. So say so.
"""
if not page.evaluate("() => !!window.__settingsSmoke"):
raise RuntimeError(
"the harness page is gone (window.__settingsSmoke undefined). This smoke page "
f"is served by vite on {PORT} and proxies /api to 127.0.0.1:8888; a Studio "
"listening there answers 401 and the app navigates away. Run this without a "
"backend on 8888."
)
def open_dialog(page, tab: str | None = None) -> None:
require_harness(page)
page.evaluate("(t) => window.__settingsSmoke.open(t || undefined)", tab)
page.wait_for_selector('div[role="dialog"]', timeout = 15000)
# Only subpages put a back button in the panel header, so this is locale-independent.
ON_SUBPAGE_JS = """() => ({
subpage: !!document.querySelector('div[role="dialog"] main header button'),
elements: (document.querySelector('div[role="dialog"] main div.hover-scrollbar')
|| { querySelectorAll: () => [] }).querySelectorAll('*').length,
})"""
# Deep-open the archived chats, then walk away partway through the hold. Both halves run in
# the page, because a route handler that sleeps blocks this script too.
ABANDON_DEEP_OPEN_JS = """(delay) => {
window.__abandonedAt = null;
window.__settingsSmoke.openArchived('chats');
setTimeout(() => {
const panel = document.querySelector('div[role="dialog"] main div.hover-scrollbar');
window.__abandonedAt = {
elements: panel ? panel.querySelectorAll('*').length : null,
subpage: !!document.querySelector('div[role="dialog"] main header button'),
};
window.__settingsSmoke.close();
}, delay);
}"""
def run_abandoned_deep_open(page) -> None:
"""A deep-open the panel never mounted for must not outlive the navigation.
`openArchivedChats` sets `archivedRequested`, and DataTab is the only thing that clears
it. Now that the panel is fetched on first view, closing the dialog while that fetch is
in flight leaves the request set with nothing to consume it, and the next ordinary visit
to Data opens an archive listing nobody asked for. Runs before anything else opens the
dialog, so the Data module is still cold and the hold is what decides when it arrives.
"""
# Matched on the Data module alone rather than on everything. The sleep below runs on
# the driver thread, so a handler that sees every request makes the whole page's module
# load queue behind it, and the panel that arrives next has to be rendered by a main
# thread that got the lot at once. On a busy two-core runner that pushed the reopen
# below past its timeout even though nothing was wrong with the dialog.
def hold_data(route):
time.sleep(DEEP_OPEN_HOLD_MS / 1000)
return route.continue_()
page.route(DATA_MODULE, hold_data)
try:
page.evaluate(ABANDON_DEEP_OPEN_JS, DEEP_OPEN_ABANDON_MS)
page.wait_for_timeout(DEEP_OPEN_HOLD_MS + 1500)
finally:
page.unroute(DATA_MODULE, hold_data)
abandoned = page.evaluate("() => window.__abandonedAt")
open_dialog(page, "data")
settled = settle_panel(page)
landed = page.evaluate(ON_SUBPAGE_JS)
report["abandoned_deep_open"] = {
"abandoned_at": abandoned,
"landed": landed,
"state": page.evaluate("() => window.__settingsSmoke.state()"),
"settled_sig": settled.get("sig"),
}
log(f"abandoned deep-open: at close {abandoned}, later visit {landed}")
if not abandoned or abandoned.get("subpage"):
fail(f"deep-open abandon never happened mid-load, so nothing was tested ({abandoned})")
elif landed["elements"] < 5:
fail(f"after an abandoned deep-open, Data did not render at all ({landed})")
elif landed["subpage"]:
fail("an abandoned archive deep-open reopened the archive on the next visit to Data")
else:
log("an abandoned deep-open leaves the next visit to Data on the main page")
page.evaluate("() => window.__settingsSmoke.close()")
page.wait_for_timeout(200)
# Dropping the abandoned ones must not drop the honoured ones.
page.evaluate("() => window.__settingsSmoke.openArchived('chats')")
page.wait_for_selector('div[role="dialog"]', timeout = 15000)
settle_panel(page)
honoured = page.evaluate(ON_SUBPAGE_JS)
report["abandoned_deep_open"]["honoured"] = honoured
if not honoured["subpage"]:
fail(f"a deep-open the panel did reach no longer opens the archive ({honoured})")
else:
log("a deep-open the panel reaches still opens the archive")
page.evaluate("() => window.__settingsSmoke.close()")
page.wait_for_timeout(200)
report["steps"].append("abandoned-deep-open")
def run_chunk_fail(page) -> None:
"""One panel's module is blocked. The dialog must survive it, and so must the app."""
open_dialog(page)
settle_panel(page)
click_forced(page.locator('[data-testid="settings-tab-general"]'), timeout = 15000)
settle_panel(page)
# Read the nav size instead of hardcoding it. The invariant is that blocking a
# panel does not collapse the dialog, not that the dialog has any particular
# number of tabs -- and the hardcoded 12 outlived its truth: the
# keyboard-shortcuts page made it 13 and this smoke started failing with
# "took the dialog down" while reporting dialog: True, which reads like an
# error-handling regression and was a stale constant.
nav_before = page.evaluate(
"() => document.querySelectorAll('[data-testid^=\"settings-tab-\"]').length"
)
if nav_before < 2:
fail(f"the settings nav was already empty before blocking anything ({nav_before})")
click_forced(page.locator(f'[data-testid="settings-tab-{CHUNK_FAIL}"]'), timeout = 15000)
page.wait_for_timeout(3000)
state = page.evaluate(
"""() => ({
dialog: !!document.querySelector('div[role="dialog"]'),
nav: document.querySelectorAll('[data-testid^="settings-tab-"]').length,
harness: !!document.querySelector('[data-testid="harness-root"]'),
boundary: !!document.querySelector('[data-testid="harness-error-boundary"]'),
panelText: (document.querySelector('div[role="dialog"] main div.hover-scrollbar')
|| {}).innerText || null,
bodyText: (document.body.innerText || '').trim().length,
})"""
)
report["chunk_fail_state"] = state
log(f"after blocking {CHUNK_FAIL}: {state}")
# The idle prefetch pulls every panel, so a blocked one must not surface as a rejection.
errors = page.evaluate("() => window.__settingsSmoke.errors()")
report["chunk_fail_window_errors"] = errors
unhandled = [
e for e in errors if "dynamically imported module" in e or "Importing a module" in e
]
if unhandled:
fail(f"blocking the {CHUNK_FAIL} panel left an unhandled rejection: {unhandled}")
else:
log("no unhandled rejection from the idle prefetch")
if state["boundary"]:
fail(
f"blocking the {CHUNK_FAIL} panel unmounted the app: the throw reached the "
"harness root boundary, and the real app has none"
)
if not state["dialog"] or state["nav"] != nav_before:
fail(
f"blocking the {CHUNK_FAIL} panel took the dialog down "
f"(nav was {nav_before} before, {state})"
)
else:
log("the dialog and its twelve nav entries survived")
# Another tab must still work.
click_forced(page.locator('[data-testid="settings-tab-about"]'), timeout = 15000)
after = settle_panel(page)
report["chunk_fail_recovery"] = after
if not after.get("present") or after.get("elements", 0) < 5:
fail(f"after a failed panel, another tab no longer renders ({after})")
else:
log("another tab still renders after the failure")
def run(page) -> None:
if CHUNK_FAIL:
run_chunk_fail(page)
return
# --- 1. a deep-open walked away from mid-load is not replayed later --------------
run_abandoned_deep_open(page)
# --- 2. every tab renders when selected -----------------------------------------
open_dialog(page)
settle_panel(page)
# Start off the persisted tab, so the first iteration is a real switch.
click_forced(page.locator('[data-testid="settings-tab-about"]'), timeout = 15000)
settle_panel(page)
for tab in TABS:
obs = click_tab_and_observe(page, tab)
report["tabs"][tab] = obs
snap = obs["settled"]
if not snap.get("present"):
fail(f"tab {tab}: panel container missing")
elif snap.get("elements", 0) < 5:
fail(f"tab {tab}: panel settled empty ({snap})")
elif snap.get("sig") == obs["before_sig"]:
fail(f"tab {tab}: panel never changed from the previous tab ({snap['sig']})")
else:
log(
f"tab {tab}: changed after {obs['changed_ms']}ms, "
f"{snap['elements']} elements, {len(snap['labels'])} labels, "
f"blank frames {obs['blank_frames']}/{obs['frames_sampled']}"
)
report["steps"].append("all-tabs-render")
# --- 3. close/reopen, and deep-open straight to a tab ----------------------------
page.evaluate("() => window.__settingsSmoke.close()")
page.wait_for_timeout(300)
for tab in ("voice", "api-keys", "data", "about", "connections"):
open_dialog(page, tab)
state = page.evaluate("() => window.__settingsSmoke.state()")
snap = settle_panel(page)
report.setdefault("deep_open", {})[tab] = {"state": state, "settled": snap}
if state["activeTab"] != tab:
fail(f"deep-open {tab}: store activeTab is {state['activeTab']}")
if not snap.get("present") or snap.get("elements", 0) < 5:
fail(f"deep-open {tab}: panel did not render ({snap})")
elif snap["sig"] != report["tabs"][tab]["settled"]["sig"]:
fail(
f"deep-open {tab}: panel differs from the same tab reached by clicking "
f"({snap['sig']} vs {report['tabs'][tab]['settled']['sig']})"
)
else:
log(f"deep-open {tab}: matches the clicked-to panel")
page.evaluate("() => window.__settingsSmoke.close()")
page.wait_for_timeout(200)
report["steps"].append("deep-open")
# --- 4. search, then jump to a result and confirm the scroll target flashed ------
# A real setting well down a long panel, so a jump that never happens shows in scrollTop.
target_tab = "general"
target_label = report["tabs"][target_tab]["settled"]["labels"][-1]
query = target_label.split()[0]
open_dialog(page, "about")
settle_panel(page)
search = page.locator('div[role="dialog"] aside input').first
search.fill(query)
page.wait_for_timeout(400)
entries = page.locator('div[role="dialog"] aside button:not([data-testid])')
texts = [entries.nth(i).inner_text().strip() for i in range(entries.count())]
report["search"] = {"query": query, "target_label": target_label, "result_texts": texts}
log(f"search '{query}' -> {texts}")
index = next((i for i, txt in enumerate(texts) if txt == target_label), None)
if index is None:
fail(f"search '{query}': '{target_label}' not among results {texts}")
else:
click_forced(entries.nth(index))
flashed = None
deadline = time.time() + 15
while time.time() < deadline:
state = page.evaluate(
"""(sel) => {
const hit = document.querySelector('.settings-search-hit');
const root = document.querySelector(sel);
return {
hit: hit ? (hit.dataset.settingsLabel || hit.tagName) : null,
scrollTop: root ? root.scrollTop : null,
};
}""",
PANEL,
)
if state["hit"]:
flashed = state
break
page.wait_for_timeout(30)
report["search"]["flashed"] = flashed
report["search"]["settled"] = settle_panel(page)
if not flashed:
fail(
f"search jump to '{target_label}': never flashed "
f"(panel {report['search']['settled'].get('sig')})"
)
elif flashed["hit"] != target_label:
fail(f"search jump: flashed '{flashed['hit']}', expected '{target_label}'")
else:
log(f"search jump to '{target_label}': flashed, scrollTop {flashed['scrollTop']}")
report["steps"].append("search-jump")
report["page_errors"] = page.evaluate("() => window.__settingsSmoke.errors()")
def write_report() -> None:
OUT.parent.mkdir(parents = True, exist_ok = True)
OUT.write_text(json.dumps(report, indent = 2), encoding = "utf-8")
log(f"report -> {OUT}")
def main() -> int:
vite = start_vite(PORT)
try:
url = f"http://127.0.0.1:{PORT}/smoke-settings.html"
with sync_playwright() as pw:
launcher = getattr(pw, ENGINE)
kwargs: dict = {"headless": True}
if ENGINE == "chromium":
kwargs["args"] = chromium_launch_args()
browser = launcher.launch(**kwargs)
ctx = browser.new_context(
viewport = {"width": 1440, "height": 900},
reduced_motion = "reduce",
)
page = ctx.new_page()
if CHUNK_DELAY_MS or CHUNK_FAIL:
def handle(route):
path = route.request.url
if "/tabs/" not in path:
return route.continue_()
if CHUNK_FAIL and f"/{CHUNK_FAIL}-tab" in path:
return route.abort("failed")
if CHUNK_DELAY_MS:
time.sleep(CHUNK_DELAY_MS / 1000)
return route.continue_()
page.route("**/*", handle)
console: list[str] = []
page.on("console", lambda m: console.append(f"{m.type}: {m.text}"))
page.on("pageerror", lambda e: console.append(f"pageerror: {e}"))
for attempt in range(40):
try:
page.goto(url, wait_until = "domcontentloaded", timeout = 30000)
break
except Exception:
if attempt == 39:
raise
time.sleep(2)
page.wait_for_function("() => !!window.__settingsSmoke", timeout = 120000)
# Vite dev re-optimizes deps on first sight and full-reloads; settle, then
# reload once for a stable dep graph.
page.wait_for_timeout(4000)
page.reload(wait_until = "domcontentloaded")
page.wait_for_function("() => !!window.__settingsSmoke", timeout = 120000)
page.wait_for_timeout(1500)
try:
run(page)
finally:
report["console"] = [
c
for c in console
if ("error" in c.lower() or "pageerror" in c.lower())
and "502" not in c
and "403" not in c
][:40]
boundary = page.locator('[data-testid="harness-error-boundary"]')
report["error_boundary"] = boundary.inner_text() if boundary.count() else None
if report["error_boundary"] and not CHUNK_FAIL:
fail(f"error boundary tripped: {report['error_boundary']}")
browser.close()
except Exception as exc:
# A cold-start dev server can fail to serve a module before anything is under
# test. Report that rather than dying with no record.
report["aborted"] = f"{type(exc).__name__}: {exc}"
fail(f"harness aborted: {report['aborted'].splitlines()[0]}")
write_report()
raise
finally:
stop_process(vite)
write_report()
if report["failures"]:
log(f"{len(report['failures'])} FAILURES")
for f in report["failures"]:
log(f" - {f}")
return 1
log("PASS")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Reference in a new issue View git blame Copy permalink