mirror of
https://github.com/unslothai/unsloth.git
synced 2026-08-23 07:44:06 +00:00
58403dd001
* Studio: measure where a heavy thread stalls, as a curve over thread content Users report Studio and Desktop going sluggish after long generations with code cells and text. That is a statement about content volume, so the new harness varies characters of thread content rather than message count, and the fixture carries the mix the report names: prose, large code fences, tool calls with collapsible output, code-execution result panes, HTML and canvas artifacts, and inline images. The primary metrics are DOM-observable and wall-clock, because Unsloth Desktop is a Tauri webview and not Chromium. PerformanceObserver accepts type longtask on WebKit 26.5 and Firefox 153 without throwing and then never fires, so support is read from supportedEntryTypes; CDP counters are recorded alongside and labelled Chromium-only. * Settle the highlighter between repetitions and tolerate constant engine chatter Re-opening the thread throws away every highlighted fence, so repetitions 2 and 3 were measuring a thread that was still building itself: on Chromium at 300K the scroll gesture read 667ms on the first repetition and 1100ms on the two after it, and the difference was the re-highlighting. Firefox 153 emits exactly two scroll-anchoring notices per run at every size. A warning count that grows with the thread still fails; a constant one does not, or the harness could never report a Gecko number. * Measure time to settle from the start of the action, and stop crediting floored metrics Measured from the end of the gesture, time to settle reads ~50ms at every size on every engine, because the answer is then three frames, which is the minimum the loop can return. From the start of the action it is what a user waits. A count that goes 0 to 4 has answered the question and counts as discriminating. A floored timing that is zero or negative at the smallest size has not: it says the action resolves inside one frame there, which is a metric with no room to move. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Record a crashed cell instead of losing the whole matrix A WebKit page that ran out of memory at 300K on a loaded machine took eight good measurements down with it. The cell is now recorded as crashed, the run continues, and the verdict still fails on it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Say precisely what the Long Tasks observer does off Chromium * Say how to bound a wedged engine, having measured that nothing in-process can On a macos-14 runner Chromium finished all three sizes in 90 seconds and then Playwright's WebKit wedged at the smallest size and never came back, which cost the whole matrix. page.evaluate and browser.new_page have no timeout, and SIGALRM does not help: the sync API blocks the main thread inside a greenlet, so the exception lands in the driver and the caller never resumes. The process boundary is the only bound that works, so the docstring says to drive one engine per invocation under an external timeout. * macOS runners have no timeout(1), so give the portable bound instead * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Load the crypto polyfill on the heavy-thread smoke page Same defect as the thread-weight page: the smoke page this branch adds was missing <script src="/crypto-boot.js"></script>, so crypto-uuid-boot.test.ts fails with "smoke-heavy-thread.html must load /crypto-boot.js". It matters more here than on a normal smoke page. This harness is what the perf numbers are measured on, and a page that lacks the polyfill differs from production in exactly the kind of way that makes a measurement mean something other than it claims. Verified: the named assertion fails before the change, and all four tests in the file pass after it. * Measure what the labels say in the heavy-thread harness Every repetition deleted a message and nothing put it back, so repetitions 2 and 3 ran on a smaller thread than the census recorded before the loop. An instrumented run at 25K read 20, 19, 18 messages at the start of each repetition; the smoke page now exposes restore() and it reads 20, 20, 20. One cycle is 20 messages against 10 content kinds, so at 25K those deletions were taking a whole kind each time. The re-open window closed on three calm frames, which held 7 rAF samples for an action taking up to 1.4s, and the leftover highlighting was absorbed by the untimed gate at the top of the next repetition. It now settles on no long frame and no new highlighted token for a grace period, and reports the time of the last activity so the grace is not added to both ends of every ratio. The token probe is polled rather than read per frame because it is a document-wide query whose cost would otherwise grow with the signal. The recorder decided ownership from a shared running flag, so a callback scheduled by the previous action ran once more under the next one and both loops appended to the same array. It carries a generation token now. Both settle() calls in the menu script compare a MutationObserver flag before the observer microtask has run, so each waits out a full double rAF. The growth axis carries a count of those floors instead of a flag, and the menu total carries two. median() dropped None, so a repetition where the menu never opened was averaged away and the null checks downstream never saw it; it now returns None if any repetition did, and a key that was null throughout stays present. A scroll, jump or re-open that never settled is a harness failure rather than an axis reading not recorded. Expanding the tool panes after waiting for the highlighter mounted two fresh unhighlighted fences per cycle whose work landed in the keystroke window, the next thing timed. Seeding and repetitions share one build_fixture() now. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Run the measurement integrity tests in CI The 22 tests added with the harness fixes were registered in no workflow, so they ran nowhere. A test that cannot fail is not a guard, and these are the ones that keep a timed-out repetition from being published as a median of three. * Stub the fork count with a real zero, and only that endpoint getForkCount returns data.count and the badge's guard is count <= 0. An empty object body makes that undefined, and undefined <= 0 is false, so the badge rendered on every assistant message with a title reading undefined forks from this message. Measured at 25000 chars: 10 badges and 4031 DOM nodes before, 0 badges and 3981 after. That is DOM in proportion to thread size, added to the axis this harness exists to measure. The stub also matched every url containing /api/, so any other request a measured interaction made was resolved locally before Playwright emitted it and measure_cell's listener never counted it. The harness could report zero stray API requests while fanning out, which is the thing it claims to catch. It matches the fork count endpoint alone now. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Answer the requests the harness provokes from an allowlist Narrowing the fetch stub to the fork-count endpoint was right, and it broke the run: the delete action persists through syncExportedRepositoryToBackend whenever remoteId is truthy, which the synthetic __LOCALID_ id always is, so three requests reached the wire inside the delete measurement and the stray check failed every size. CI showed it as HARNESS-BROKEN, 5 requests, at both 25000 and 100000 chars. The stub is an explicit allowlist now rather than either a blanket match or a single endpoint. Each entry answers a request the harness itself provokes, with the body that endpoint really returns, so no round trip lands in a timed region, and anything unlisted still goes to the network and trips the counter. Narrowing it is also what made two requests visible that the blanket match had been hiding since the beginning: re-opening a thread asks for the project list and the knowledge bases. Those are app fan-out rather than fixture upkeep, so they are stubbed to keep the network out of the reopen window but counted and printed as stubbed api requests. Answering a request must not delete it from the record. The fork-count guard now pins the body to its own allowlist entry rather than scanning the whole file, since another entry legitimately answers with an empty object and a file-wide check would fail on it while saying nothing about fork counts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add seedCompactTail and gapMetrics for the viewport gap measurement #9058 measures the empty band below the last mounted row and needs two things this harness did not expose. Landing them here rather than having that branch duplicate 767 lines of an unmerged harness. seedCompactTail(targetChars, tailMessages) builds the heavy fixture with the same buildThread call seed() makes, then appends N one-word messages, so the first mount commit lands entirely on compact rows. That is the worst case for a fixed size initial window. Census parity is the point: seedCompactTail (25000, 16) reports 36 messages against seed(25000)'s 20, a tail of exactly 16, with every other count unchanged. gapMetrics() reports the band below the last row, measured against the viewport's bottom edge rather than scrollHeight, so the viewport's own bottom spacer counts as the gap it always was and the caller subtracts spacerHeight to get the part the mount window owns. Computed any other way the numbers stop being comparable across sizes. Both degenerate returns are kept: no viewport gives ok 0 alone, and a viewport with no rows gives ok 0 with mountedRows and clientHeight. The contract test is separate from the rest because what it protects is on another branch: #9058's probe preflights for these exact names and keys, so a rename here breaks a probe nothing in this repo would otherwise exercise. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gate the tool expand on a count that a closed thread fails The wait after expandTools() read collapsibleOutputs, which is the Radix collapsible CONTENT ELEMENT. Radix keeps that element in the tree for its collapse animation, so it is present whether the card is open or shut. Measured at 25000 chars before any expandTools() call: collapsibleOutputs 2 of the 2 expected while codeExecutionPanes was 0, and after expanding, collapsibleOutputs was still 2. The gate could not fail, so the ordering that build_fixture exists to guarantee, expand and then wait for the highlighter, was never enforced. It reads codeExecutionPanes now, the pane's own pre, which is 0 collapsed and 22 expanded at 300000 chars on all three engines. The comment on collapsibleOutputs claimed Radix mounts the content only while open; that is measurably false and it now says what the count really is, with a warning not to gate on it. Also records what the timed windows scan, since two review items asked. Each window scans a fixed number of times rather than once per frame, and the cost was measured rather than argued: the re-open makes two messageCount passes, 0.4ms of a 2292ms re-open at 300000 chars and 0.0ms of 363ms at 25000, and the menu scans four times for 2.7ms of 3208ms against 0.3ms of 375ms. The share is 0.017 and 0.08 percent, the same at both ends of the axis. * Run the gap contract in frontend CI I added test_heavy_thread_gap_contract.py and registered it in neither the path filter nor the pytest step, so the guard that keeps #9058's probe working ran nowhere. That is the third time this round something was added that could not fail, and this one was mine, two commits after fixing the same class twice. In both lists now. * Count the reopen paint floor instead of declaring it zero growth() subtracts one ~33ms double-rAF vsync floor per wait a metric is clocked across, and the count was a hand-declared integer in GROWTH_AXES. reopen ms declared 0. Reopening is driven by a React state update, so the count check immediately after openThread() always still sees the unmounted tree and the loop always pays at least one __nextPaint() before it can observe the rebuilt messages. That is the same floor already subtracted from jump painted ms and delete ms. Leaving it at 0 kept a full floor of constant baseline in both ends of the ratio, which drags the ratio towards 1 and can report a real reopen curve as flat, worst when the smallest fixture rebuilds near the vsync floor. Rather than change the constant and leave the next one to be declared by hand, REOPEN_JS now counts the waits it actually pays and returns them, and floor_declaration_problems compares the declaration against the count for every engine and size, from harness_failures, so a mismatch fails the run instead of being published. FLOOR_COUNTERS is keyed on the exact axis name, since a prefix would let a later reopen settle ms axis be checked against this one's declaration. A cell that reports no count is a failure rather than a skip; leaving it silent is how the check would quietly stop checking. The clean-cell fixture in the integrity tests gained the field, and it caught the change: with the axis back at 0 the existing clean-cell test goes red too. Eight assertions added, each made to fail on its own targeted broken tree before being kept: the axis back at 0, the checker short-circuited, a missing count skipped rather than reported, the crashed-cell guard removed, the checker unwired from harness_failures, the counter removed from the loop, and mismatches accepted. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not report an action that did not run as an unverified paint floor Self-review of the floor check. An action that never ran carries no paintWaits, so the new check reported its floor as unverified on top of the failure harness_failures already raises for the action itself, with the reason. Two failures for one cause, and the second buries the first. Skipped only when the action explicitly reports ran false. An action that DID run and still has no count is still a failure, since that is the case where the subtraction is genuinely unverified, and there is a test on that side too so the skip cannot be widened into a blanket exemption. Both proven red: removing the skip, and widening it to always. * Measure the wall floor per window, and stop filing exceptions as chatter Two things. The generated wall ms axes declared zero double-rAF waits for every action, while the explicit axes declared theirs by hand. MENU_JS opens the recorder before opening the menu and closes it after closing it, so it crosses the same two waits menu open+close ms correctly declares, and menu wall ms was subtracting none of them. That leaves roughly two vsync floors of constant baseline in both ends of the ratio, which compresses it towards 1 and can label a primary metric flat on a small or fast fixture. Rather than hand-declare a second set of numbers, the recorder now counts the waits each window is clocked across and reports them, and the wall axes read that count from the row. growth() accepts a callable floor for this. Waits taken outside a recorder window, ACTION_SETUPS among them, are excluded by construction rather than by remembering to exclude them, and begin() zeroes the counter so a window cannot inherit the waits of the ones before it. Separately, console.error and uncaught pageerror went into the same list as Firefox's two scroll-anchoring notices and were tolerated by the same > 4 allowance. Engine chatter is the engine describing itself; an application exception is not that, and a single one inside a measured interaction means the interaction did not do what the row says. Severity is preserved now, the allowance applies to warnings only, and any error in seeding or in the measured actions fails the run. The counts and the first message are printed either way. Ten assertions added and each proven red on its own broken tree. Two did not go red on the first attempt: the severity assertion pinned an exact one-line expression and stayed green when the predicate was moved to its own line, and the counter-reset break did not apply at all because its needle was mis-indented. Both are fixed and both now fail as they should. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Require a meaningful rise before a zero-based counter counts as an answer No ratio can be formed against zero, so DISCRIMINATION_RATIO never applied to the counter axes at all and large > small was the entire test. The CI workflow runs one repetition on Chromium, so there is no median to smooth a stray dropped frame, and harness_failures accepts any ONE discriminating axis: 0 missed frames at 25K and 1 at 100K was marked as discriminating and could carry the whole liveness verdict while every latency axis was flat or broken. ZERO_BASED_MIN_RISE is what such a counter has to reach. 5 because these axes are dropped frames and long tasks: at twelve times the content a real curve produces them in quantity, while one or two is what an unloaded machine produces on its own. Absolute rather than a ratio because there is nothing to divide by, and the reason string now distinguishes a counter that rose too little from one that never moved, so a reader can see which happened. Five assertions, each proven red on its own broken tree: back to a bare large > small, the threshold lowered to 1, and every zero-based counter rejected outright. The last of those covers the control, since a check that rejects every counter would leave the harness unable to report a live run at all. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Say which axes are counts instead of inferring it from the paint floor The zero branch keyed on floored, which only identifies a timing that had a paint floor subtracted. An unfloored timing does not have one: longest stall ms and worst frame ms read zero at the smallest size whenever the action resolves before the recorder produces a sample, and were then judged as dropped-frame counters, so a noisy 5ms at the largest size read as a rise of 5 and discriminated. harness_failures accepts any single discriminating axis, so that stray millisecond could carry a run in which every valid latency curve was flat. COUNTER_AXES states which axes are counts. Only frames over 33ms is one; everything else is milliseconds. A timing that reads zero at the smallest size is now reported as having no rise to measure rather than being given a counter's credit. Three assertions, each proven red on its own broken tree: timings judged as counters again, no axis classified as a counter at all, and a timing axis classified as a count. The middle one matters because emptying the set would silently turn every counter into a timing and remove the only zero-based axis the liveness verdict has. One correction: the first version of the set assertion required a counter's name not to end in ms, which is wrong, since the counter axis is called frames over 33ms and does. It names the set exactly now, which is the point of classifying it explicitly. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Put the floor COUNT in the report, not the thing that computes it Making the wall floor a callable put the lambda itself into the growth report. main() attaches that report to results and json.dumps it, so a complete run raised Object of type function is not JSON serializable after every measurement had already been taken, including the new CI smoke. That is my regression from the previous commit and it broke every full run. resolve_floor returns an int, growth uses it, and the report stores the resolved count at each end of the ratio rather than a boolean. A boolean would serialise and say nothing; the counts let a reader check the subtraction instead of trusting it. The reason no test caught this is that none of them serialised the report, so three assertions now do, and one of them covers the callable-floor axis specifically since that is the case that broke. Both proven red: the callable put back into the report, and a boolean marker in place of the count. * Keep fractional paint floors, and hold counters to the noise floor at any baseline Two follow-ons from the same review. resolve_floor cast to int. summarise takes a median across repetitions, so a run whose repetitions paid 1 and 2 waits reports 1.5, and truncating that left half a vsync floor in the wall axis. The documented two-repetition configurations are precisely the ones that produce halves. The median is kept as a float, which serialises fine. The noise floor only applied when a counter started at exactly zero. A dropped-frame count going 1 to 2 is a ratio of 2.0, cleared DISCRIMINATION_RATIO, and since harness_failures accepts any single discriminating axis, one incidental frame could carry the CI smoke while every latency axis was flat. A ratio on a counter is only meaningful once there are enough events for it to be about the content rather than about one frame either way, so the floor now applies whatever the baseline, and the reason string says which of the two rules rejected the axis. The floor stays a count of events and is NOT applied to timings, which would silently reject real latency curves that happen to sit at low absolute values. That has its own test. Five assertions, each proven red on its own broken tree: the floor truncated again, the noise floor skipped for nonzero baselines, the noise floor applied to timings as well, and every counter treated as noise, which is what covers the control. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Subtract the whole window's floors from the axes that span it quiet() and quietUntilIdle() return the elapsed time since this.startedAt, not the time they themselves took, and gestureMs is computed from startedAt as well. All three therefore span the entire recorder window and contain every double-rAF wait in it, and all three declared zero. For the scroll that is twenty vsync floors left in both ends of the ratio, which compresses it hard enough to report a real size-dependent regression as flat. scroll gesture ms, scroll settle ms and jump settle ms now take the measured paint_waits. Counted at runtime rather than declared, because the twenty come from a loop: the literal nextPaint count in the source is one, so any number written in here would have been wrong the same way the zero was. Deliberately NOT applied to everything. jump painted ms starts at a mark taken after begin() and spans one wait while the jump's window holds two, and MENU_JS awaits no paint at all, so its window count is zero while its two floors are real, coming from settle() reading the pre-MutationObserver state on entry for open and again for close. Giving either the window count would subtract a floor the number never contained, or drop one that it did. The rule is that an axis measured from startedAt takes the measured count and an axis measured from a later mark keeps a declared one. Seven assertions, each proven red on its own broken tree: each of the three axes back to zero, jump painted given the whole-window floor, and menu given a window count of zero. That last break also turned two PRE-EXISTING menu tests red, which independently confirms the menu axis really does carry both of its floors. There is an end-to-end case too: with the floors left in, a 16x scroll curve reads as 1.86x. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>
12 lines
287 B
HTML
12 lines
287 B
HTML
<!doctype html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="UTF-8" />
|
|
<script src="/crypto-boot.js"></script>
|
|
<title>heavy thread smoke</title>
|
|
</head>
|
|
<body>
|
|
<div id="root"></div>
|
|
<script type="module" src="/smoke-heavy-thread-main.tsx"></script>
|
|
</body>
|
|
</html>
|