vrr/unsloth
2
0
Fork 0
mirror of https://github.com/unslothai/unsloth.git synced 2026-08-21 06:43:53 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 28

Drain the sampling debounce long enough for the node CI actually runs (#9332)

Browse source 
* Stop Frontend CI installing Chromium's system libraries through apt every run

`playwright install --with-deps chromium` runs its own `apt-get update`
inside itself, so it bypassed everything CI has learned about apt: the shared
retry helper's 20s transfer cap, APT_ACQUIRE_RETRIES: '0', and the archive
cache. The job failed 3 of 8 runs on main.

Job 96072994354 (main, 2026-08-19): 9 packages, 21.1 MB, and
`fonts-wqy-zenhei [7472 kB]` alone took 5m50s off azure.archive.ubuntu.com.
Both 420s attempts died mid-download. That is the same mirror and the same
package that took the webkit shards down in #9289. Attempt 2 logged
"Need to get 8833 kB/21.1 MB", so apt does resume partials across attempts and
still could not finish.

Split the way studio-ui-smoke.yml splits it: download the engine, launch it to
find out whether the system libraries are actually missing, and run
`install-deps` only if they are. ubuntu-latest is a browser-testing image and
usually ships them, so the common path now runs no apt at all.

The browser and apt-archive cache keys are deliberately identical to the
chromium-only shards in studio-ui-smoke.yml (engine token `c`): same image,
same Playwright version, same single engine, so the entry is shared rather than
duplicated against a budget measured at 99.3% full.

The step's authorised worst case doubles with the second helper call, to
2 x (2 x 420s + 125s) = 1930s, so its timeout goes 17m -> 33m and stays under
the job's 40m. Both guarded calls are skipped on the common path.

Guard: tests/studio/test_playwright_install_avoids_with_deps.py fails the build
if `--with-deps` returns to any workflow, and is wired into
workflow-trigger-lint, the only job with no paths filter.

* Drain the sampling debounce long enough for the node CI actually runs

Frontend CI has been red on main since #9055, not intermittently: eight
consecutive main runs failed at `Unit tests`, every one on node v22.23.2.

The three suites from #9055 wait for a debounced write with a fixed drain --
three rounds of tick(1000) plus six setImmediate turns -- and then assert. Three
rounds is enough on node 24, which is what a dev box happens to have, and is not
enough on node 22, which `setup-node: 22` resolves to. The same chain drains far
fewer continuations per round there, so the write had not landed when the
assertion ran.

Reproduced by downloading node 22.23.2 and running the suites against both.
Measured on the compat suite:

  rounds   3    10    30    60
  failing  7     5     1     0

The compat suite reported it as a missing value (expected 1.37, actual
undefined). The simulation suite reported it as an ORDERING violation -- "chat A
temperature: owed 0.6, shows 1.37" -- because a scenario whose write has not
landed looks exactly like one that wrote the wrong value, which is why this read
as a feature bug rather than a slow test.

There were three copies of the drain: one per suite plus the shared
thread-sampling-world helper the simulations run through. Fixing only the two
suites left the simulations red, since their scenarios drain through the helper.

The bound is 200, generous rather than tuned to the observed 60, and settle()
now takes an optional `until` predicate: it returns as soon as the caller can
see the work, and throws naming itself if the condition never holds, so the next
slow runtime reports "settle gave up" instead of an assertion on a missing value.

Verified: node 22.23.2 compat 16/16 and simulation 18/18 (both were failing);
node 24 full frontend suite 4060/4060. Only the simulation suite imports the
helper, and no fixed three-round drain remains in tests/.

* Raise the scenario drain to 600: 200 was marginal, and Windows needed more

At 200 the simulations were 0 failing on one run and 1 on the next on the same
machine, and the Windows runner -- slower again -- still had 2 of 120 orderings
short. 600 is 0 failing across three consecutive runs, at 107s against 52s.

Also records the adaptive version that was tried and is wrong here, so it is not
tried again: the rows only change WHEN the write lands, so 'rows have stopped
changing' is precisely the pending state being waited through. Quiescence on
that observable stops early by construction and scored 4 failures where the
fixed bound scored 0.

* Stop the settings smoke asserting a tab count a new page invalidates

Unskipping the browser smokes surfaced this immediately: the blocked-panel run
failed with 'blocking the data panel took the dialog down' while its own report
said dialog: True. Nothing had taken the dialog down. The check was

    if not state["dialog"] or state["nav"] != 12:

and the keyboard-shortcuts page had made the nav 13. A stale constant, reading
as an error-handling regression.

The nav size is now read before the panel is blocked and compared against
itself, which is the invariant that was meant: blocking a panel must not
collapse the dialog, whatever size the dialog is.

The same drift had a quieter half. The smoke's TABS list still had twelve
entries, so keyboard-shortcuts had no browser coverage at all and the smoke
stayed green without it. It is added here, and
tests/studio/test_settings_smoke_covers_every_tab.py pins both directions
against settings-dialog.tsx so the next page cannot go uncovered silently. It
also checks the workflow's PW_CHUNK_FAIL names a tab that exists -- that value
lives in studio-frontend-ci.yml, not in the smoke, and a rename would leave the
run blocking nothing while still reporting PASS.

Wired into workflow-trigger-lint, the only job with no paths filter, because it
reads a workflow. Mutation-tested both ways: dropping the tab from TABS and
restoring the literal count each turn it red.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Drain the sampling suites on the loader too, and fail loudly when it gives up

The fixed round count in the previous commit was still a guess, and the Windows
job proved it: the SAME commit at 600 rounds passed one run and failed the next
with "A1: 2 violation(s) across 120 orderings", reporting stale reads as lost
edits. Raising the number again was not the answer.

The pending work has a second half nothing was waiting on. The store's
thread-scoped write ends in `await import("../utils/chat-history-storage")`
(chat-runtime-store.ts:1326 and :1754), and these suites register() a resolver
hook, which routes that import through the hooks thread. Three repeat imports of
an already-loaded module:

  v24.14.0  no hook 1, 1, 1 turns    hook registered 1, 1, 1
  v22.23.2  no hook 1, 1, 1 turns    hook registered 6, 3, 35

That is the whole green-locally / red-on-CI split, and it is why a loaded Windows
runner fails what the same commit passed an hour earlier: the pending work is a
message to another thread, so its cost is scheduling latency, not instructions.
No round count is correct for that.

Counting the mocked timers alone does not cover it either, which is worth
recording since it is the obvious next idea. With the counter installed and 25
consecutive quiet rounds per drain, 150 macrotask turns of nothing, v22.23.2
still lost 7 orderings across A1 and A3, every one a write that had not landed.

So drain on both observables. tests/helpers/mock-timer-drain.ts wraps the MOCKED
setTimeout with a counter, giving an exact count of timers scheduled and not yet
fired or cleared, and each round also issues its own import and waits for it, so
the wait scales with the loader instead of guessing at it. The drain returns when
no timer is outstanding and three consecutive rounds neither scheduled nor fired
one. With the probe, three quiet rounds is green on v22 and v24 alike.

The generous bound stays as a BACKSTOP that THROWS and names what was still
outstanding, which is the change that matters most here. An under-drain used to
be indistinguishable from the store losing an edit, so it sent the investigation
into the store for two rounds. Now it says so:

  settle: drain exhausted after 2 rounds, with no timer pending but work still
  scheduling or firing within the last 3 rounds. Nothing read after this point is
  trustworthy: a queued write has not landed, so the store still shows the
  PREVIOUS value, which reads as a wrong value rather than a missing one. Fix the
  work or raise the backstop; do not read this as the store losing an edit.

It is also much faster, because it stops when the work is done rather than after
600 rounds regardless. A typical drain now takes 4 rounds; instrumented over 840
drains in A1 the maximum was 4.

  simulation  v24 101.7s -> 30s      v22.23.2 106.5s -> 29s
  compat      v24 0.54s -> 0.46s     v22.23.2 0.61s -> 0.52s

The three copies of the drain shape, one in the world helper and one in each test
file, are now one helper. The dead end already recorded is kept next to the new
measurements: quiescence on the store ROWS is still wrong, because the rows only
change WHEN the write lands.

Verified: both suites green twice each on v24.14.0 and on v22.23.2, the version
setup-node 22 resolves to. Full frontend suite 4080 passed, 0 failed. The
exhaustion throw was confirmed by forcing the backstop to 2, which produces the
message above and no wrong-value violation. A deliberately broken
sanitizeThreadScopedSettings still produces a real ordering violation on both
runtimes, so the drain is not exiting early. Test files only.

* Read the workflow and the dialog as UTF-8, not as the platform default

test_source_read_encoding caught four read_text() calls this PR added with no
encoding. It is right and they are a real defect: the guard reads
settings-dialog.tsx, playwright_settings_tabs.py and studio-frontend-ci.yml, and
on a Windows runner Path.read_text() uses the ANSI code page, so any non-ASCII
byte in any of them raises UnicodeDecodeError. The whole point of this guard is
that a settings page can be added without anyone noticing; a guard that cannot
be collected on Windows fails the same way.

Repo tests (CPU) was otherwise clean: 1 failed, 8781 passed.

* Tell a pwsh crash apart from install.ps1 losing its exit code

test_the_pwsh_filter_keeps_the_log_clean_and_the_exit_code_intact went red on a
hosted ubuntu runner with completely empty stdout and pwsh's own banner: "An
error has occurred that was not properly handled. Additional information is shown
below. The PowerShell process will exit." The interpreter died; the script never
ran. Workflow trigger lint is green on the last 8 main runs and this PR touches
no PowerShell, so it is the runner, not the repo.

Read as an ordinary failure it says "$LASTEXITCODE did not survive the added
pipeline stages, so a failing install.ps1 would leave its step green" -- an
accusation against the installer, raised by a run that produced no evidence
either way. That is the same shape as the drain in this PR: an environment
shortfall wearing the costume of a product regression.

_run_pwsh retries only that case, and the distinction is what keeps it honest:
a run that reaches the `RC=` line is returned on the first attempt whatever the
value, so a genuine regression can never be retried into green. Only a run with
no RC= AND the crash banner is retried, because it carries no verdict to
preserve. If both attempts crash it fails with a message naming the interpreter,
not install.ps1.

17 passed. Exercised _run_pwsh against a script that prints the banner and no
RC=: it raises, so the branch is not vacuous.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Daniel Han 2026-08-19 23:55:01 -07:00 • committed by GitHub
parent 149d29752f
commit 1c3dde199b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
10 changed files with 785 additions and 65 deletions
Download patch file Download diff file Expand all files Collapse all files

163
.github/workflows/studio-frontend-ci.yml vendored
View file

@ -56,8 +56,10 @@ jobs:
# Two browser smokes (~50s) plus the Chromium install now sit inside this.
# 20 was not a budget, it was the bound on the Chromium install: the step had
# none of its own, so the job timeout was what eventually stopped it, and it
# stopped everything after it too. That step is bounded at 16 now, so this can
# be sized for the work plus that worst case.
# stopped everything after it too. That step carries its own bound now -- 33m,
# covering the two guarded helper calls it authorises -- so this is sized for
# the work plus that worst case, and stays above it so the STEP timeout is what
# fires first and names itself.
timeout-minutes: 40
defaults:
run:
@ -201,22 +203,159 @@ jobs:
# Smokes go last: every step carries an implicit success(), so running them ahead of
# the build gates let one red smoke skip the build and all three bundle assertions.
# Costs nothing here, since each smoke starts its own vite server and reads no dist/.
# `--with-deps` shells out to apt, so this is an apt step wearing a different
# name and it fails the same way. On 2026-08-19 it sat here for 16m38s, the
# job hit `timeout-minutes: 20`, and the run was reported as "cancelled" with
# the browser smokes and the lifecycle tests below simply skipped. Bounded at
# 2 x 420s because the browser download is genuinely large; a third attempt
# would not fit the budget and the first two already cover a mirror hiccup.
#
# The install below was `playwright install --with-deps chromium`. That flag
# shells out to apt, so it was an apt step wearing a different name and it
# failed the same way: it once sat here for 16m38s, the job hit its
# `timeout-minutes`, and the run was reported as "cancelled" with the browser
# smokes and the lifecycle tests below simply skipped. It was still failing on
# 2026-08-19 (job 96072994354): 9 packages, 21.1 MB, `fonts-wqy-zenhei
# [7472 kB]` alone taking 5m50s off azure.archive.ubuntu.com, both 420s
# attempts dying mid-download -- the same mirror and the same package that took
# the webkit shards down in #9289.
#
# So it is split the way studio-ui-smoke.yml splits it: engine download, then a
# launch probe, then apt ONLY if the probe says the libraries are missing. The
# 2 x 420s budget stays: it is the engine download that is genuinely large, and
# a third attempt would not fit the step timeout.
- name: Pin the Playwright version so the browser cache has a key
id: pw
working-directory: ${{ github.workspace }}
run: |
python3 -m pip install 'playwright>=1.45,<2' pytest
echo "version=$(python3 -c 'from importlib.metadata import version; print(version("playwright"))')" >> "$GITHUB_OUTPUT"
# Keys deliberately IDENTICAL to the chromium-only shards in
# studio-ui-smoke.yml (engine token `c`): same runner image, same Playwright
# version, same single engine, so the same entry is correct for both and this
# job starts warm off whatever main saved there. Diverging the key here would
# cost a second copy of the same bytes against a budget measured at 99.3% full.
- name: Restore the Playwright browser cache
id: pw-cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
continue-on-error: true
with:
path: ~/.cache/ms-playwright
key: ms-playwright-${{ runner.os }}-${{ steps.pw.outputs.version }}-c-v2
- name: Restore the apt archive cache (chromium's .deb set)
id: apt-cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
continue-on-error: true
with:
path: ${{ github.workspace }}/.apt-archives
key: apt-archives-${{ runner.os }}-${{ env.ImageOS }}-${{ env.ImageVersion }}-c-v1
- name: Install Chromium for browser smokes
working-directory: ${{ github.workspace }}
timeout-minutes: 17
# Two helper calls now, not one -- the engine download and the apt
# transaction are separated -- so the authorised worst case doubles:
# 2 x (2 x 420s + 125s lock wait) = 1930s. 17m used to cover one call and
# would now cut the last attempt off mid-download, which
# test_the_retry_budget_fits_inside_the_step_timeout fails the build for.
#
# Both calls are guarded, so the common path spends none of this: the engine
# download is skipped on a browser-cache hit, and apt is skipped entirely
# when the probe says the libraries are already there. This budget is only
# reachable when the cache misses AND the image is missing libraries AND the
# mirror is degraded -- which is the run that used to fail outright.
timeout-minutes: 33
env:
RETRY_ATTEMPTS: '2'
RETRY_ATTEMPT_TIMEOUT: '420'
# `--with-deps` used to run apt unconditionally here, and apt's own retries
# multiplied every stalled transfer inside the attempt budget. Observed on
# main 2026-08-19 (job 96072994354): 9 packages, 21.1 MB, and
# `fonts-wqy-zenhei [7472 kB]` alone took 5m50s off
# azure.archive.ubuntu.com -- both 420s attempts died mid-download and the
# job failed. Same mirror and the same package that took the webkit shards
# down in #9289.
APT_ACQUIRE_RETRIES: '0'
run: |
python3 -m pip install 'playwright>=1.45,<2' pytest
bash .github/scripts/retry-with-apt-lock.sh \
python3 -m playwright install --with-deps chromium
# The engine first, and WITHOUT --with-deps. That flag makes playwright run
# its own `apt-get update` internally, which is the one apt call this repo
# cannot restructure -- so it is not used. A CDN download and an apt
# transaction are two different failures and they are separated here, the
# same way studio-ui-smoke.yml separates them.
if [ "${{ steps.pw-cache.outputs.cache-hit }}" != "true" ]; then
bash .github/scripts/retry-with-apt-lock.sh \
python3 -m playwright install chromium
fi
# Then ask whether the system libraries are actually missing instead of
# assuming they are. ubuntu-latest is a browser-testing image and ships
# nearly all of them; when it does, this skips an apt update and a
# transaction that would install nothing. Launching the engine is the
# honest test of that -- it is what the smokes below are about to do.
probe() {
python3 - "$@" <<'PY'
import sys
from playwright.sync_api import sync_playwright
missing = []
with sync_playwright() as p:
for name in sys.argv[1:]:
try:
browser = getattr(p, name).launch()
browser.close()
except Exception as exc:
missing.append(f"{name}: {type(exc).__name__}")
if missing:
print("engines that will not start: " + "; ".join(missing))
sys.exit(1)
print("chromium launches; system libraries are present")
PY
}
if probe chromium; then
echo "::notice::skipped playwright install-deps; the runner image already has the libraries"
else
echo "system libraries are missing, installing them"
# Hand apt last run's .debs before it goes looking for them. apt checks
# each file against its index and re-fetches only what does not match, so
# a stale cache costs a download rather than a wrong install.
if [ -d "${{ github.workspace }}/.apt-archives" ]; then
sudo cp "${{ github.workspace }}"/.apt-archives/*.deb /var/cache/apt/archives/ 2>/dev/null || true
echo "seeded $(ls "${{ github.workspace }}"/.apt-archives/*.deb 2>/dev/null | wc -l) cached .deb files"
fi
bash .github/scripts/retry-with-apt-lock.sh \
python3 -m playwright install-deps chromium
# Harvest for next time: apt keeps what it installed in the archive dir
# until something runs `apt-get clean`, so this is exactly what it used.
mkdir -p "${{ github.workspace }}/.apt-archives"
sudo cp /var/cache/apt/archives/*.deb "${{ github.workspace }}/.apt-archives/" 2>/dev/null || true
sudo chown -R "$(id -u):$(id -g)" "${{ github.workspace }}/.apt-archives" || true
# Fail loudly rather than proceeding into smokes that cannot launch a
# browser: without this the real error surfaces later as an opaque
# per-test timeout in whichever smoke happens to run first.
probe chromium
fi
# Both saves are main-only, the rule every cache in this repo follows: a
# PR-scoped entry can only be restored by re-runs of that same PR while still
# counting against the shared budget, evicting the copy every PR can read.
#
# And NOT under always(). These keys are immutable, so a save that runs after a
# failed install would store a half-downloaded engine or a partial .deb set
# under the key every later run reads, and no later run could replace it. The
# implicit success() is what makes the payload trustworthy;
# test_cache_budget_discipline.py fails the build if either save reaches for
# always() again.
- name: Save the Playwright browser cache
if: github.ref == 'refs/heads/main' && steps.pw-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
continue-on-error: true
with:
path: ~/.cache/ms-playwright
key: ${{ steps.pw-cache.outputs.cache-primary-key }}
- name: Save the apt archive cache
if: github.ref == 'refs/heads/main' && steps.apt-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
continue-on-error: true
with:
path: ${{ github.workspace }}/.apt-archives
key: ${{ steps.apt-cache.outputs.cache-primary-key }}
# Browserless, but imports the harnesses (hence playwright), so it sits after the
# install. Covers the Windows teardown branch no runner here executes.

2
.github/workflows/workflow-trigger-lint.yml vendored
View file

@ -128,6 +128,8 @@ jobs:
tests/studio/test_install_phase_timing.py \
tests/studio/test_mac_bundled_job_phases.py \
tests/studio/test_mac_host_offload_optin.py \
tests/studio/test_settings_smoke_covers_every_tab.py \
tests/studio/test_playwright_install_avoids_with_deps.py \
tests/studio/test_macos_slots_per_commit.py \
tests/studio/test_main_runs_survive_merge_bursts.py \
tests/studio/test_pester_bootstrap_hardening.py \

269
studio/frontend/tests/helpers/mock-timer-drain.ts Normal file
View file

@ -0,0 +1,269 @@
// SPDX-License-Identifier: AGPL-3.0-only
// Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
// Waiting out the debounced writes the chat store schedules, on the store's own pending
// work rather than on a round count.
//
// The shape every caller needs is the same: advance the mocked clock, then let the promise
// continuations that firing released run, and repeat -- the writes these suites assert on
// sit behind a 400ms debounce whose callback then awaits, so one tick is never enough, and
// a timer scheduled from inside a continuation only fires on the NEXT tick.
//
// A FIXED round count was what this used to be, and it is a guess that fails in one
// direction silently. Measured on v22.23.2, the version `setup-node: 22` resolved to, the
// sibling compat suite scored 7 failures at 3 rounds, 5 at 10, 1 at 30 and 0 at 60; the
// simulation, which runs many more steps per scenario, still lost 2 of 120 orderings at
// 200 and needed 600 (107s against 52s) -- and even 600 was only load-dependently enough,
// since the same commit at 600 passed one Windows CI run and failed the next. The failure
// mode is what makes this expensive: a drain that returns with a write still queued leaves
// the caller reading a stale value, which the simulation reports as an ORDERING violation
// ("chat A minP: owed 0, shows 0.01"), indistinguishable from the store genuinely losing
// the edit. Hours go into the store before anyone suspects the drain.
//
// So drain on the real observables instead. There are two, because the work has two halves:
//
// 1. The debounce. `t.mock.timers.enable` replaces globalThis.setTimeout with its own;
// wrapping THAT (and clearTimeout) with a counter gives an exact count of timers
// scheduled and not yet fired or cleared, which is the pending state itself rather than
// a proxy for it. Stop only when the count is zero AND a further QUIET_ROUNDS rounds
// neither scheduled a timer nor ran one, since a promise continuation is free to
// schedule the next debounce a turn after the last one fired.
// 2. The module loader. The write ends in an `await import()` (see probeModuleLoader),
// which these suites route through a registered resolver hook and therefore through
// the hooks thread. Nothing counts that, and on v22.23.2 it takes 3 to 35 macrotask
// turns where v24 takes 1 -- measured, three repeat imports of an already-loaded
// module. That is the whole green-locally / red-on-CI split, and quiet rounds alone do
// not cover it: at 25 quiet rounds per drain (150 turns of nothing) v22 still lost 7
// orderings across A1 and A3, all of them writes that had not landed. Each round
// issues its own import and waits for it, so the wait scales with the loader instead
// of guessing at it. With that, 3 quiet rounds is green on v22 and v24 alike.
//
// The generous bound stays, as a BACKSTOP that THROWS and names what was still outstanding.
// It is not the exit path any more, so it costs nothing to leave high, and a suite that
// hits it is told the drain gave up rather than handed a stale read to misattribute.
//
// Not to be retried: quiescence on the store ROWS. The rows only change WHEN the write
// lands, so "rows have stopped changing" is precisely the pending state being waited
// through, and it stops early by construction -- it scored 4 failures where the fixed bound
// scored 0. Timers and the loader are the opposite kind of observable: they exist while the
// work is outstanding, and are gone when it is done.
import type { TestContext } from "node:test";
type TimerHandle = ReturnType<typeof setTimeout>;
type SetTimeoutFn = typeof globalThis.setTimeout;
type ClearTimeoutFn = typeof globalThis.clearTimeout;
/** How far each round advances the mocked clock. Longer than any debounce here, short
* enough that a chain of sequential debounces still fires one per round. */
const TICK_MS = 1000;
/** Macrotask turns per round. Each one drains the whole microtask queue behind it. */
const MICROTASK_TURNS = 6;
/** Consecutive rounds of no timer scheduled, none fired and none outstanding. Rounds, not
* turns: a round also waits out the loader, which is the part that varies by runtime. */
const QUIET_ROUNDS = 3;
/** Backstop only. Reaching it is a failure, not the normal exit. */
const MAX_ROUNDS = 600;
/** Set on the wrapper so a second enable() in the same test is not double-wrapped, and so
* a drain can tell it is looking at a counted setTimeout rather than a raw mocked one.
* `enable()` installs a fresh mocked setTimeout each test, which drops this and makes the
* next install re-wrap with fresh counts. */
const COUNTED = Symbol("unsloth.mockTimerDrain.counted");
interface TimerCounter {
/** Scheduled and not yet fired or cleared. */
outstanding: number;
/** Every schedule, fire and clear since install: the "did anything happen" signal. */
activity: number;
/** Live handles, so clearTimeout knows whether it is cancelling real pending work. */
entries: Map<TimerHandle, { done: boolean; handle?: TimerHandle }>;
}
let counter: TimerCounter | null = null;
function countedSetTimeout(): (SetTimeoutFn & { [COUNTED]?: true }) | null {
const current = globalThis.setTimeout as SetTimeoutFn & { [COUNTED]?: true };
return current[COUNTED] === true ? current : null;
}
/** Copy the own symbols (util.promisify.custom above all) onto the wrapper, so code that
* reaches for them still finds them on the function the tests installed. */
function inheritSymbols(from: object, to: object): void {
for (const key of Object.getOwnPropertySymbols(from)) {
const descriptor = Object.getOwnPropertyDescriptor(from, key);
if (descriptor !== undefined) Object.defineProperty(to, key, descriptor);
}
}
/**
* Enable node's mocked setTimeout for this test and count what the code under test
* schedules on it. Call this instead of `t.mock.timers.enable` -- the counter has to be
* installed over the MOCKED setTimeout, and before any product code runs, or the timers
* scheduled in between are invisible and a drain reports quiet while they are pending.
*
* Returns the tick the drain wants, for callers that would rather not repeat it.
*/
export function enableCountedTimers(t: TestContext): (ms: number) => void {
t.mock.timers.enable({ apis: ["setTimeout"] });
if (countedSetTimeout() === null) {
const mockedSetTimeout = globalThis.setTimeout as SetTimeoutFn;
const mockedClearTimeout = globalThis.clearTimeout as ClearTimeoutFn;
const state: TimerCounter = {
outstanding: 0,
activity: 0,
entries: new Map(),
};
counter = state;
const wrappedSetTimeout = ((
callback: (...args: unknown[]) => void,
ms?: number,
...args: unknown[]
) => {
// The handle lives on the entry because the callback closes over it before
// mockedSetTimeout has returned it.
const entry: { done: boolean; handle?: TimerHandle } = { done: false };
state.outstanding += 1;
state.activity += 1;
const run = (...callbackArgs: unknown[]): void => {
// Guard the arithmetic rather than the call: a mocked timer only fires from
// tick(), but clearTimeout may have retired this entry first.
if (!entry.done) {
entry.done = true;
state.outstanding -= 1;
state.activity += 1;
if (entry.handle !== undefined) state.entries.delete(entry.handle);
}
callback(...callbackArgs);
};
entry.handle = mockedSetTimeout(
run as never,
ms as never,
...(args as never[]),
) as TimerHandle;
if (!entry.done) state.entries.set(entry.handle, entry);
return entry.handle;
}) as SetTimeoutFn & { [COUNTED]?: true };
inheritSymbols(mockedSetTimeout, wrappedSetTimeout);
wrappedSetTimeout[COUNTED] = true;
const wrappedClearTimeout = ((handle?: TimerHandle) => {
if (handle !== undefined) {
const entry = state.entries.get(handle);
if (entry !== undefined && !entry.done) {
entry.done = true;
state.outstanding -= 1;
state.activity += 1;
}
state.entries.delete(handle);
}
return mockedClearTimeout(handle as never);
}) as ClearTimeoutFn;
inheritSymbols(mockedClearTimeout, wrappedClearTimeout);
globalThis.setTimeout = wrappedSetTimeout;
globalThis.clearTimeout = wrappedClearTimeout;
}
return (ms: number) => t.mock.timers.tick(ms);
}
/** Which turn of a round waits out the module loader; see probeModuleLoader. */
const LOADER_PROBE_AFTER_TURN = 2;
/**
* Wait for the module loader to come back, so that a dynamic import the code under test
* issued earlier in this round has settled by the time the round is judged quiet.
*
* The store's thread-scoped write ends in `await import("../utils/chat-history-storage")`,
* and these suites `register()` a resolver hook, which puts that import on the hooks
* thread. That round trip is the second kind of pending work here, and unlike a debounce it
* has no timer to count: on v24 it settles in ONE macrotask turn, but measured on v22.23.2
* with a hook registered, three repeat imports of an already-loaded module settled in 6, 3
* and 35 turns. That is the whole node-24-green / node-22-red split, and the reason a
* Windows runner under load fails what the same commit passed an hour earlier: the pending
* work is a message to another thread, so its cost is scheduling latency, not instructions.
*
* Issuing our own import is what turns that into something to wait ON. The hooks thread
* serves one request at a time and replies over the same port, so a reply to a request
* issued after the store's arrives after it: when this resolves, the store's import has
* resolved too and its continuation is at most a turn behind, which the rest of the round
* covers. Re-importing this module rather than some marker file keeps it honest -- it is
* certainly loaded, and it goes through the same registered hooks the store's import does.
*/
function probeModuleLoader(): Promise<unknown> {
return import(import.meta.url);
}
/** Debounce timers scheduled and not yet fired or cleared. */
export function pendingTimerCount(): number {
return countedSetTimeout() === null || counter === null
? 0
: counter.outstanding;
}
export interface DrainOptions {
/** An extra condition the caller needs true before the drain may return. Quiescence is
* still required with it: a condition that flips mid-chain must not cut the rest off. */
until?: () => boolean;
/** Named in the exhaustion message, so a failure says which wait gave up. */
label?: string;
/** Backstop override. For proving the throw fires; not for tuning a wait. */
maxRounds?: number;
}
/**
* Advance the mocked clock until the code under test has no scheduled timer left and has
* stopped scheduling new ones, then return. Throws rather than returning early if the
* backstop runs out, because everything a caller reads afterwards would otherwise be a
* stale value wearing the costume of a wrong one.
*/
export async function drainMockedTimers(
tick: (ms: number) => void,
options: DrainOptions = {},
): Promise<void> {
const { until, label = "drain", maxRounds = MAX_ROUNDS } = options;
const state = counter;
if (countedSetTimeout() === null || state === null) {
throw new Error(
`${label}: the timer counter is not installed, so there is nothing to drain ON. ` +
"Enable the mocked clock with enableCountedTimers(t) rather than " +
"t.mock.timers.enable, and do it before the code under test schedules anything.",
);
}
let quiet = 0;
for (let round = 0; round < maxRounds; round += 1) {
const activityBefore = state.activity;
tick(TICK_MS);
for (let turn = 0; turn < MICROTASK_TURNS; turn += 1) {
await new Promise((resolve) => setImmediate(resolve));
// Half way through the round, once the timer callbacks fired by the tick have had a
// turn to issue their own import, wait out the loader with them.
if (turn === LOADER_PROBE_AFTER_TURN) await probeModuleLoader();
}
quiet =
state.outstanding === 0 && state.activity === activityBefore
? quiet + 1
: 0;
if (quiet >= QUIET_ROUNDS && (until === undefined || until())) return;
}
const pending = state.outstanding;
if (pending > 0 || quiet < QUIET_ROUNDS) {
throw new Error(
`${label}: drain exhausted after ${maxRounds} rounds, ` +
(pending > 0
? `with ${pending} timer(s) still pending`
: `with no timer pending but work still scheduling or firing within the ` +
`last ${QUIET_ROUNDS} rounds`) +
". Nothing read after this point is trustworthy: a queued write has not landed, " +
"so the store still shows the PREVIOUS value, which reads as a wrong value " +
"rather than a missing one. Fix the work or raise the backstop; do not read this " +
"as the store losing an edit.",
);
}
throw new Error(
`${label}: drain exhausted after ${maxRounds} rounds. The timers all settled, but ` +
"the caller's condition never held, so the work either never ran or is not the " +
"work this was waiting for. This is not the assertion below failing on a wrong value.",
);
}

19
studio/frontend/tests/helpers/thread-sampling-world.ts
View file

@ -9,6 +9,7 @@
// instance through the "?scenario=" query thread-sampling-resolver.mjs understands.
// Register that resolver and install the localStorage fake before importing this.
import { drainMockedTimers } from "./mock-timer-drain.ts";
import { threadRows } from "./store-stubs/chat-history-storage.ts";
import { settingsHttp } from "./store-stubs/settings-http.ts";
@ -136,14 +137,18 @@ interface StoreModule {
beginThreadScopedPairing: (threadId: string) => void;
}
/** Let the debounced writers and their promise chains finish. */
/** Let the debounced writers and their promise chains finish.
*
* On the store's own pending timers and on the module loader, not on a round count: a scenario whose write has not
* landed yet looks exactly like one that wrote the wrong value, so an under-drain here
* surfaces as an ORDERING violation ("chat A temperature: owed 0.6, shows 1.37") and sends
* the reader into the store instead of into the wait. drainMockedTimers throws when it
* gives up, which is the whole point of it; see tests/helpers/mock-timer-drain.ts for the
* measurements that killed the fixed count and for the observable NOT to drain on.
*
* The caller must have enabled the clock with enableCountedTimers(t). */
async function drain(tick: (ms: number) => void): Promise<void> {
for (let round = 0; round < 3; round += 1) {
tick(1000);
for (let i = 0; i < 6; i += 1) {
await new Promise((resolve) => setImmediate(resolve));
}
}
await drainMockedTimers(tick, { label: "runScenario drain" });
}
/**

42
studio/frontend/tests/thread-scoped-sampling-compat.test.ts
View file

@ -18,6 +18,10 @@ import { register } from "node:module";
import test from "node:test";
import { installLocalStorageFake } from "./helpers/kit.ts";
import {
drainMockedTimers,
enableCountedTimers,
} from "./helpers/mock-timer-drain.ts";
const { store: localStorageFake } = installLocalStorageFake();
localStorageFake.set("unsloth_chat_settings_imported_to_studio_db", "true");
@ -121,13 +125,15 @@ async function world(rows: Record<string, Record<string, unknown>> = {}) {
};
}
async function settle(tick: (ms: number) => void): Promise<void> {
for (let round = 0; round < 3; round += 1) {
tick(1000);
for (let i = 0; i < 6; i += 1) {
await new Promise((resolve) => setImmediate(resolve));
}
}
// Wait out the debounced write each case asserts on. The wait is on the store's own
// outstanding timers and on the module loader, not on a round count: three rounds passed on
// a dev box's node 24 and failed on the node 22 CI pins, and every count picked since has
// been a guess that fails silently in one direction. See tests/helpers/mock-timer-drain.ts.
async function settle(
tick: (ms: number) => void,
until?: () => boolean,
): Promise<void> {
await drainMockedTimers(tick, { until, label: "settle" });
}
function assertUsable(sampling: Record<string, unknown>, where: string): void {
@ -150,7 +156,7 @@ function assertUsable(sampling: Record<string, unknown>, where: string): void {
// ---------------------------------------------------------------------------
test("C1: a legacy row opens on the installation sampling, and nothing is zeroed", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await world({ L: { ...LEGACY_ROW } });
await settle((ms) => t.mock.timers.tick(ms));
@ -174,7 +180,7 @@ test("C1: a legacy row opens on the installation sampling, and nothing is zeroed
// legacy chat stored no sampling, so it follows the installation defaults and a model load
// moves those. Nothing chosen is lost, but the second visit shows different numbers.
test("C1b: a legacy chat follows the installation defaults, which a model load moves", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await world({ L: { ...LEGACY_ROW } });
await settle((ms) => t.mock.timers.tick(ms));
w.open("L");
@ -210,7 +216,7 @@ test("C1b: a legacy chat follows the installation defaults, which a model load m
});
test("C1c: a legacy chat that the user then edits pins the WHOLE set, not just the edit", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await world({ L: { ...LEGACY_ROW } });
await settle((ms) => t.mock.timers.tick(ms));
w.open("L");
@ -247,7 +253,7 @@ test("C1c: a legacy chat that the user then edits pins the WHOLE set, not just t
// ---------------------------------------------------------------------------
test("C2: an empty, null or absent snapshot opens on the installation settings", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
for (const snapshot of [null, {}, undefined]) {
const w = await world();
await settle((ms) => t.mock.timers.tick(ms));
@ -377,7 +383,7 @@ test("C3c: the sanitizer never throws, whatever it is handed", () => {
});
test("C3d: a NaN or Infinity in a row cannot reach the store", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await world({
N: {
...LEGACY_ROW,
@ -398,7 +404,7 @@ test("C3d: a NaN or Infinity in a row cannot reach the store", async (t) => {
// 1.2 was brought back to 1.0), but the load path applies a recommendation to the live
// params unclamped, so a custom model_defaults yaml still reaches this.
test("C3e: an out-of-range recommendation never reaches a chat that pinned that key", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await world();
await settle((ms) => t.mock.timers.tick(ms));
w.open("A");
@ -427,7 +433,7 @@ test("C3e: an out-of-range recommendation never reaches a chat that pinned that
});
test("C3f: an out-of-range recommendation taken with no chat open cannot be pinned", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await world();
await settle((ms) => t.mock.timers.tick(ms));
@ -487,7 +493,7 @@ test("C4: the sanitizer keeps every falsy and negative value", () => {
});
test("C4b: a falsy edit round-trips capture -> persist -> restore", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
for (const topK of [-1, 0]) {
const edge = { ...FALSY_EDGE, topK };
const w = await world();
@ -521,7 +527,7 @@ test("C4b: a falsy edit round-trips capture -> persist -> restore", async (t) =>
});
test("C4c: a falsy pinned value survives a model load and a model switch", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await world();
await settle((ms) => t.mock.timers.tick(ms));
w.open("A");
@ -560,7 +566,7 @@ test("C4c: a falsy pinned value survives a model load and a model switch", async
});
test("C4d: an empty system prompt is a choice, not an absent one", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await world();
await settle((ms) => t.mock.timers.tick(ms));
// A is given a prompt, B is deliberately cleared
@ -592,7 +598,7 @@ test("C4d: an empty system prompt is a choice, not an absent one", async (t) =>
// ---------------------------------------------------------------------------
test("C5: a one-megabyte system prompt is neither truncated nor fatal", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const huge = "x".repeat(1024 * 1024);
assert.equal(huge.length, 1_048_576);

57
studio/frontend/tests/thread-scoped-sampling-simulation.test.ts
View file

@ -24,6 +24,10 @@ import { register } from "node:module";
import test from "node:test";
import { installLocalStorageFake } from "./helpers/kit.ts";
import {
drainMockedTimers,
enableCountedTimers,
} from "./helpers/mock-timer-drain.ts";
const { store: localStorageFake, fireWindowEvent } = installLocalStorageFake();
// Skip the legacy import path: it would look for settings this test never wrote.
@ -101,7 +105,7 @@ async function sweep(
// ---------------------------------------------------------------------------
test("A1: every ordering of edit / load / Think / model switch / reopen", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const orderings = permutations<Op>([
"editTemp",
"loadQwen",
@ -114,7 +118,7 @@ test("A1: every ordering of edit / load / Think / model switch / reopen", async
});
test("A2: every ordering of prompt edit / post-load defaults / external / unload / second chat", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const orderings = permutations<Op>([
"editPrompt",
"qwenPostLoad",
@ -129,7 +133,7 @@ test("A2: every ordering of prompt edit / post-load defaults / external / unload
});
test("A3: both chats, both edits, both Think positions, in every order", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const orderings = permutations<Op>([
"editTemp",
"editPrompt",
@ -145,7 +149,7 @@ test("A3: both chats, both edits, both Think positions, in every order", async (
});
test("A4: every four-step interleaving over the wider alphabet", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const alphabet: Op[] = [
"editTemp",
"editPrompt",
@ -163,7 +167,7 @@ test("A4: every four-step interleaving over the wider alphabet", async (t) => {
});
test("A5: hydration interleaved -- nothing leaks, whatever the order", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
// Without the shadow model: a chat opened before the server answered follows this
// browser's cache, so what it is "owed" is not yet decided. The leak and usability
// invariants still hold, and they are the ones that matter here.
@ -179,7 +183,7 @@ test("A5: hydration interleaved -- nothing leaks, whatever the order", async (t)
});
test("A6: hand-picked long sequences", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const long: Op[][] = [
// the reported gap: two chats, a model in between, back to the first
[
@ -286,7 +290,7 @@ test("A6: hand-picked long sequences", async (t) => {
// ---------------------------------------------------------------------------
test("the pinned values really are stored on the chat's own row", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
await runScenario(
["hydrate", "openA", "editTemp", "editPrompt", "openB", "reopenA"],
(ms) => t.mock.timers.tick(ms),
@ -301,7 +305,7 @@ test("the pinned values really are stored on the chat's own row", async (t) => {
});
test("I3: an edit with no chat open reaches the installation-wide settings", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
await runScenario(["hydrate", "editTemp", "editPrompt"], (ms) =>
t.mock.timers.tick(ms),
);
@ -311,7 +315,7 @@ test("I3: an edit with no chat open reaches the installation-wide settings", asy
});
test("I4: a model's recommendation reaches the installation and the model's memory", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
await runScenario(
["hydrate", "openA", "editTemp", "editPrompt", "loadQwen", "switchLlama"],
(ms) => t.mock.timers.tick(ms),
@ -375,17 +379,20 @@ async function raceWorld() {
};
}
async function settle(tick: (ms: number) => void): Promise<void> {
for (let round = 0; round < 3; round += 1) {
tick(1000);
for (let i = 0; i < 6; i += 1) {
await new Promise((resolve) => setImmediate(resolve));
}
}
// Wait out the debounced write each race test asserts on. The wait is on the store's own
// outstanding timers and on the module loader, not on a round count, so a slower runtime
// takes more rounds instead of silently returning short and turning a stale read into a
// wrong-value failure; see tests/helpers/mock-timer-drain.ts for why a count was never the
// right bound.
async function settle(
tick: (ms: number) => void,
until?: () => boolean,
): Promise<void> {
await drainMockedTimers(tick, { until, label: "settle" });
}
test("B1: a slider moved while /api/chat/settings is still in flight survives it", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await raceWorld();
settingsHttp.hold();
const hydrating = w.store().hydratePersistedSettings();
@ -403,7 +410,7 @@ test("B1: a slider moved while /api/chat/settings is still in flight survives it
});
test("B2: an edit made while the chat's own read is out is stored on that chat", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await raceWorld();
await w.store().hydratePersistedSettings();
await settle((ms) => t.mock.timers.tick(ms));
@ -433,7 +440,7 @@ test("B2: an edit made while the chat's own read is out is stored on that chat",
});
test("B3: leaving mid-read sends the held edit to its own chat, not the next one", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await raceWorld();
await w.store().hydratePersistedSettings();
await settle((ms) => t.mock.timers.tick(ms));
@ -470,7 +477,7 @@ test("B3: leaving mid-read sends the held edit to its own chat, not the next one
});
test("B4: a model load landing during the pairing window does not take the chat's edit", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await raceWorld();
await w.store().hydratePersistedSettings();
await settle((ms) => t.mock.timers.tick(ms));
@ -503,7 +510,7 @@ test("B4: a model load landing during the pairing window does not take the chat'
});
test("B4b: the held sampling edit that survives a load is the LAST one made", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await raceWorld();
await w.store().hydratePersistedSettings();
await settle((ms) => t.mock.timers.tick(ms));
@ -522,7 +529,7 @@ test("B4b: the held sampling edit that survives a load is the LAST one made", as
});
test("B4c: a falsy held edit is not treated as no edit at all", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await raceWorld();
await w.store().hydratePersistedSettings();
await settle((ms) => t.mock.timers.tick(ms));
@ -556,7 +563,7 @@ test("B4c: a falsy held edit is not treated as no edit at all", async (t) => {
});
test("B5: two rapid model switches with a pinned chat open", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await raceWorld();
await w.store().hydratePersistedSettings();
await settle((ms) => t.mock.timers.tick(ms));
@ -591,7 +598,7 @@ test("B5: two rapid model switches with a pinned chat open", async (t) => {
});
test("B6: a thread switch while a load is in flight keeps each chat's own values", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await raceWorld();
await w.store().hydratePersistedSettings();
await settle((ms) => t.mock.timers.tick(ms));
@ -619,7 +626,7 @@ test("B6: a thread switch while a load is in flight keeps each chat's own values
});
test("B7: a tab closing with a held edit beacons it to the chat it was made in", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
enableCountedTimers(t);
const w = await raceWorld();
await w.store().hydratePersistedSettings();
await settle((ms) => t.mock.timers.tick(ms));

19
tests/studio/playwright_settings_tabs.py
View file

@ -41,6 +41,7 @@ TABS = [
"data",
"api-keys",
"agents",
"keyboard-shortcuts",
"debugging",
"about",
]
@ -275,6 +276,17 @@ def run_chunk_fail(page) -> None:
settle_panel(page)
click_forced(page.locator('[data-testid="settings-tab-general"]'), timeout = 15000)
settle_panel(page)
# Read the nav size instead of hardcoding it. The invariant is that blocking a
# panel does not collapse the dialog, not that the dialog has any particular
# number of tabs -- and the hardcoded 12 outlived its truth: the
# keyboard-shortcuts page made it 13 and this smoke started failing with
# "took the dialog down" while reporting dialog: True, which reads like an
# error-handling regression and was a stale constant.
nav_before = page.evaluate(
"() => document.querySelectorAll('[data-testid^=\"settings-tab-\"]').length"
)
if nav_before < 2:
fail(f"the settings nav was already empty before blocking anything ({nav_before})")
click_forced(page.locator(f'[data-testid="settings-tab-{CHUNK_FAIL}"]'), timeout = 15000)
page.wait_for_timeout(3000)
state = page.evaluate(
@ -305,8 +317,11 @@ def run_chunk_fail(page) -> None:
f"blocking the {CHUNK_FAIL} panel unmounted the app: the throw reached the "
"harness root boundary, and the real app has none"
)
if not state["dialog"] or state["nav"] != 12:
fail(f"blocking the {CHUNK_FAIL} panel took the dialog down ({state})")
if not state["dialog"] or state["nav"] != nav_before:
fail(
f"blocking the {CHUNK_FAIL} panel took the dialog down "
f"(nav was {nav_before} before, {state})"
)
else:
log("the dialog and its twelve nav entries survived")
# Another tab must still work.

34
tests/studio/test_install_phase_timing.py
View file

@ -368,6 +368,38 @@ for _candidate in ("pwsh", "powershell"):
continue
# The banner pwsh prints when the interpreter itself dies rather than the script failing.
# Seen on a hosted ubuntu runner mid-run, with completely empty stdout: no "RC=" at all,
# not a wrong one. Read as a normal failure it accuses the pipeline of losing
# $LASTEXITCODE, which is a claim about install.ps1, so a runner hiccup arrives looking
# like a product regression. Distinguish the two.
_PWSH_CRASHED = "The PowerShell process will exit"
def _run_pwsh(script: str, attempts: int = 2):
"""Run `script` under pwsh, retrying only an interpreter crash.
A crash yields no verdict either way, so retrying it is not papering over a failure:
there is nothing to paper over yet. A run that reaches `RC=` is returned as-is on the
first attempt, whatever the value, so a real regression is never retried into green.
"""
proc = None
for _ in range(attempts):
proc = subprocess.run(
[PWSH, "-NoProfile", "-Command", script], capture_output = True, text = True
)
if "RC=" in proc.stdout:
return proc
if _PWSH_CRASHED not in (proc.stdout + proc.stderr):
return proc
raise AssertionError(
f"pwsh itself terminated abnormally on all {attempts} attempts and never reached "
f"the `RC=` line, so this run says nothing about whether $LASTEXITCODE survives "
f"the pipeline. That is the interpreter dying, not install.ps1 losing its exit "
f"code.\nstdout: {proc.stdout!r}\nstderr: {proc.stderr!r}"
)
@pytest.mark.skipif(PWSH is None, reason = "no PowerShell on this platform")
def test_the_pwsh_filter_keeps_the_log_clean_and_the_exit_code_intact(tmp_path):
"""Same two claims for the Windows dialect, which is where the 291s actually is.
@ -387,7 +419,7 @@ def test_the_pwsh_filter_keeps_the_log_clean_and_the_exit_code_intact(tmp_path):
Write-Output "RC=$LASTEXITCODE"
"""
)
proc = subprocess.run([PWSH, "-NoProfile", "-Command", script], capture_output = True, text = True)
proc = _run_pwsh(script)
assert "RC=7" in proc.stdout, (
f"$LASTEXITCODE did not survive the added pipeline stages, so a failing "
f"install.ps1 would leave its step green:\n{proc.stdout}\n{proc.stderr}"

123
tests/studio/test_playwright_install_avoids_with_deps.py Normal file
View file

@ -0,0 +1,123 @@
# Unsloth Zoo - Utilities for Unsloth
# Copyright 2023-present Daniel Han-Chen, Michael Han-Chen & the Unsloth team. All rights reserved.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
"""`playwright install --with-deps` must not come back.
The flag looks like a convenience and is actually an unbounded apt step wearing
a different name: playwright runs its own `apt-get update` inside it, which is
the one apt call in this repo that cannot be restructured to try the image's
lists first. Everything CI has learned about apt -- the shared retry helper, the
20s transfer cap, `APT_ACQUIRE_RETRIES: '0'`, the archive cache -- applies to
the `install-deps` subcommand and is bypassed entirely when the work happens
inside `install --with-deps`.
Both failures that motivated this were the same shape and the same package:
studio-ui-smoke.yml webkit shards, 181 packages / 102 MB
studio-frontend-ci.yml chromium, 9 packages / 21.1 MB, and
`fonts-wqy-zenhei [7472 kB]` alone took 5m50s off azure.archive.ubuntu.com
The supported shape is: download the engine, launch it to find out whether the
system libraries are actually missing, and run `install-deps` only if they are.
That is what this guard pins -- not the comments describing it, which drift.
"""
from __future__ import annotations
import re
from pathlib import Path
import pytest
import yaml
WORKFLOWS = Path(__file__).resolve().parents[2] / ".github" / "workflows"
# `playwright install --with-deps`, however the flag is spelled or ordered, and
# whether invoked as `playwright`, `python -m playwright` or `python3 -m playwright`.
# Deliberately NOT anchored on `chromium`/`webkit`: the engine list is irrelevant
# to the defect, which is that apt runs at all.
_WITH_DEPS = re.compile(r"playwright\s+install\b[^\n]*--with-deps")
def _run_steps(path: Path):
"""(job name, step name, run body) for every step that runs a shell body."""
doc = yaml.safe_load(path.read_text()) or {}
for job_name, job in (doc.get("jobs") or {}).items():
if not isinstance(job, dict):
continue
for step in job.get("steps") or []:
if not isinstance(step, dict):
continue
run = step.get("run")
if isinstance(run, str):
yield job_name, step.get("name", "<unnamed>"), run
def _workflow_files() -> list[Path]:
return sorted(WORKFLOWS.glob("*.yml"))
@pytest.mark.parametrize("path", _workflow_files(), ids = lambda p: p.name)
def test_no_workflow_installs_playwright_browsers_with_deps(path: Path) -> None:
offenders = [
f"{path.name}::{job}::{step}"
for job, step, run in _run_steps(path)
if _WITH_DEPS.search(run)
]
assert not offenders, (
"`playwright install --with-deps` runs apt-get update inside itself, "
"bypassing the shared retry helper, the 20s transfer cap and "
"APT_ACQUIRE_RETRIES: '0'. Download the engine, probe by launching it, "
"and call `playwright install-deps` only when the probe fails -- see the "
"`Install Chromium for browser smokes` step in studio-frontend-ci.yml. "
f"Offending steps: {offenders}"
)
def test_the_guard_can_see_the_pattern_it_forbids() -> None:
"""A regex typo would make every assertion above vacuously pass."""
for body in (
"python3 -m playwright install --with-deps chromium",
"python -m playwright install --with-deps chromium firefox webkit",
"playwright install --with-deps",
"playwright install chromium --with-deps",
):
assert _WITH_DEPS.search(body), body
def test_the_guard_does_not_fire_on_the_supported_shape() -> None:
"""The split form, and the unrelated `--with-deps` on scan_packages.py."""
for body in (
"python3 -m playwright install chromium",
"python -m playwright install-deps chromium",
"python scripts/scan_packages.py --with-deps requirements.txt",
):
assert not _WITH_DEPS.search(body), body
def test_at_least_one_workflow_installs_playwright_browsers() -> None:
"""Pins that the parametrisation is actually looking at something."""
installers = [
f"{path.name}::{step}"
for path in _workflow_files()
for _job, step, run in _run_steps(path)
if re.search(r"playwright\s+install\b", run)
]
assert installers, (
"no workflow step runs `playwright install`; either the suite moved or "
"this guard is no longer reading the workflows it thinks it is"
)

122
tests/studio/test_settings_smoke_covers_every_tab.py Normal file
View file

@ -0,0 +1,122 @@
# Unsloth Zoo - Utilities for Unsloth
# Copyright 2023-present Daniel Han-Chen, Michael Han-Chen & the Unsloth team. All rights reserved.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
"""The settings smoke must drive every tab the dialog actually has.
`playwright_settings_tabs.py` keeps its own list of tab ids. A settings page
added to `settings-dialog.tsx` without touching that list ships untested, and
nothing says so: the smoke keeps passing, because it still drives the twelve it
knows about.
That is not hypothetical. The keyboard-shortcuts page took the dialog to
thirteen tabs while the smoke's list stayed at twelve, so the new page had no
browser coverage at all. The same drift also broke the smoke's `nav != 12`
assertion, which then failed as "blocking the data panel took the dialog down"
while reporting `dialog: True` -- a stale constant reading like an
error-handling regression.
Both halves are pinned here: the lists agree, and the smoke no longer hardcodes
a nav size that any new page invalidates.
"""
from __future__ import annotations
import re
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
SMOKE = ROOT / "tests" / "studio" / "playwright_settings_tabs.py"
DIALOG = ROOT / "studio" / "frontend" / "src" / "features" / "settings" / "settings-dialog.tsx"
def _smoke_tabs() -> list[str]:
"""The TABS list literal the smoke drives."""
text = SMOKE.read_text(encoding = "utf-8")
block = re.search(r"^TABS = \[(.*?)^\]", text, re.S | re.M)
assert block, "could not find the TABS list in playwright_settings_tabs.py"
return re.findall(r'"([a-z0-9-]+)"', block.group(1))
def _dialog_tabs() -> list[str]:
"""The tab ids the dialog renders, in declaration order."""
text = DIALOG.read_text(encoding = "utf-8")
# Scope to the tab array first: `id:` appears on other objects in this file, and
# a bare scan would invent tabs. Both spellings live in here -- the one-line
# `{ id: "general", labelKey: ... }` and the multi-line form -- so the id match
# deliberately does NOT anchor to the start of a line.
block = re.search(r"SETTINGS_TABS[^=]*=\s*\[(.*?)^\]", text, re.S | re.M)
region = block.group(1) if block else text
return re.findall(r'\bid:\s*"([a-z0-9-]+)"', region)
def test_the_smoke_drives_every_tab_the_dialog_defines() -> None:
smoke, dialog = set(_smoke_tabs()), set(_dialog_tabs())
missing = sorted(dialog - smoke)
assert not missing, (
f"settings-dialog.tsx defines tabs the settings smoke never opens: {missing}. "
"Add them to TABS in tests/studio/playwright_settings_tabs.py, or the page "
"ships with no browser coverage and the smoke still goes green."
)
def test_the_smoke_does_not_drive_a_tab_that_no_longer_exists() -> None:
smoke, dialog = set(_smoke_tabs()), set(_dialog_tabs())
stale = sorted(smoke - dialog)
assert not stale, (
f"the settings smoke drives tabs the dialog no longer defines: {stale}. "
"It will fail looking for a selector that cannot appear."
)
def test_the_chunk_fail_tab_is_one_the_dialog_has() -> None:
"""The blocked-panel run names its tab in the workflow, not in the smoke.
`CHUNK_FAIL` defaults to the empty string, so the tab comes from
`PW_CHUNK_FAIL` in studio-frontend-ci.yml. Renaming that tab would leave the
run blocking nothing at all, and the smoke would still report PASS on a
panel it never broke.
"""
workflow = (ROOT / ".github" / "workflows" / "studio-frontend-ci.yml").read_text(
encoding = "utf-8"
)
targets = re.findall(r"PW_CHUNK_FAIL:\s*([a-z0-9-]+)", workflow)
assert targets, "no PW_CHUNK_FAIL in studio-frontend-ci.yml; the blocked-panel run is not wired"
dialog = set(_dialog_tabs())
unknown = sorted(t for t in set(targets) if t not in dialog)
assert not unknown, (
f"studio-frontend-ci.yml blocks settings tabs the dialog does not define: "
f"{unknown}. The run would block nothing and assert against an unbroken panel."
)
def test_the_nav_assertion_is_not_hardcoded() -> None:
"""A literal count here is what made a new settings page look like a regression."""
text = SMOKE.read_text(encoding = "utf-8")
assert 'state["nav"] != nav_before' in text, (
"the blocked-panel check must compare the nav against the size measured "
"before blocking, not a literal. A hardcoded count fails the moment a "
"settings page is added, and reports it as the dialog being taken down."
)
assert not re.search(
r'state\["nav"\]\s*!=\s*\d+', text
), "found a hardcoded nav count in the blocked-panel check"
def test_the_guard_reads_both_files() -> None:
"""Neither list may be empty, or every assertion above passes vacuously."""
assert len(_smoke_tabs()) >= 5, _smoke_tabs()
assert len(_dialog_tabs()) >= 5, _dialog_tabs()