a2ed9928bb
* docs: Design the OpenTUI submit and exit parity batch Six gaps where the ink renderer has a mechanism the OpenTUI port carried only the shadow of: bare quit tokens, the missing client shutdown signal on exit, mid-turn steering that skips the vision bridge and @-expansion, an e2e leg that cannot hold a stream open, and the noflicker scenario's outcome being judged globally. * fix(cli): Let bare quit tokens reach the OpenTUI dispatcher Typing `exit`, `quit`, `:q` or `:wq` in the OpenTUI renderer sent them to the model as prose, and mid-turn they queued behind the stream instead of stopping it. ink rewrites that family to `/quit` ahead of its message queue; the port had the exemption but not the rewrite, so the gate let a quit through only in the one spelling that already looked like a command. * fix(cli): Stop background work before an OpenTUI session exits Every OpenTUI exit drained the cleanup chain without first telling the client that shutdown was imminent, so memory extract, dream and skill review could spawn new agent work during the exit window — the one thing that keeps a process from finishing. ink makes that signal on its quit path; here all three exits share one drain, so the signal belongs to the drain. * fix(cli): Resolve OpenTUI mid-turn steering like an idle prompt Text the composer queues while a turn runs reached the model raw: no `@path` expansion, no read deadline, and no prompt-side vision bridge, so an image could ride as `inlineData` to a model that cannot read it while ink converts it and discloses the egress. Both hops now resolve a steered message the way an idle submission does — expansion under ink's 10 s read timeout, then the bridge — and a turn that dies mid-resolution hands its texts back to the queue instead of losing them. The bridge can also pick a full-turn vision model mid-turn; the rest of the turn stays on it, including the model named in the mapper's own notices, so the override is read per boundary rather than once per turn. Part of U-21 and U-25 in #8662. * test(cli): Pin the exit check between queued command dispatches Round four of #10831 noted that the deferred-command drain checks the shared exit latch twice but only the pre-loop check had a test: deleting the per-iteration one left the suite green. This adds the crossing that check exists for — the exit begins while an earlier command is still awaiting its outcome — by flipping the latch inside the first dispatch and requiring the second never to run. Part of U-23 in #8662. * fix(scripts): Require the offline parity gate's base to fail The offline no-flicker scenario runs a fixture emitter that injects clears and unbalanced DEC 2026 as its base side, yet the runner accepted both-pass as a passing outcome. A base fixture that stopped emitting its defect would have kept the gate green while proving nothing at all. Scenarios can now declare their base a defect fixture with `expectBaseFailure`, which turns both-pass into a failure — set on the offline scenario only, since the credentials scenario compares against ink and the self-test override path asserts both-pass deliberately. A gate failure names its own cause in the report and on the console line. Part of U-15 in #8662. * test(integration): Hold the fake model stream open mid-turn Every mid-turn behaviour in the submit path is invisible to a test that cannot say "the CLI is streaming right now": the fake server wrote each SSE chunk synchronously, and awaiting its handler holds a turn only before the first byte. FakeOpenAIResponse gains holdAfterChunks + holdUntil, so a test can stop the stream after a delta it can see on screen and release it on purpose. close() tears held connections down, which is pinned because "the run cannot hang on my instrument" is exactly the claim an instrument like this must not make untested. Four cases use it, one file for both renderers: /quit and a bare `exit` typed into a live stream, a slash command held back mid-turn, and a steered @file mention. The last two assert on captured request bodies rather than screen text, because OpenTUI repaints by diffing cells and renders no slash command output at all. Closes U-24 in #8662. * docs: Record the batch-8 measurement boundary and adjacent gaps The design doc said each new case was checked red-then-green before its own fix commit. What was done instead is mutation probes, and two of the four leave the OpenTUI leg green — so replace the claim with the measured table and its cause: a submission that skips the mid-turn gate is steered into the same turn and still reaches the model only after the held turn ends, and process teardown closes the socket however the turn ends. Both halves are pinned at unit level, which is a different claim. Register the three gaps this batch found and deliberately did not fix: steered text has no transcript echo (U-26), neither hop warns about an unsupported image format (U-27), and OpenTUI keeps slash command output in a history no view renders (U-28) — the last is why the first version of the deferral case failed for a reason unrelated to the gate under test. * docs: Correct the mid-turn requeue rule in the batch-8 design The design doc said a timed-out mid-turn read hands its text back to the queue. The code requeues on an abort only — a timeout or a declined read is dropped with a warning, which is what ink does and what the steering test pins — so the sentence described a behaviour the port deliberately does not have. * docs: Measure both of the batch-8 drain's exit-latch checks Deleting the in-loop check reddens exactly the test written for it, and deleting the pre-loop check reddens nothing at unit level. The pair is what makes the documented redundancy a measurement instead of an argument, so both now sit in the coverage boundary alongside the two mutations the E2E leg cannot see. --------- Co-authored-by: Shaojin Wen <shaojin.wensj@alibaba-inc.com> |
||
|---|---|---|
| .. | ||
| fixtures | ||
| lib | ||
| test | ||
| accept-noflicker.sh | ||
| component-parity.sh | ||
| README.md | ||
| runner.mjs | ||
| self-test.mjs | ||
TUI parity harness
Deterministic, dependency-light harness that compares a base CLI capture
(current Ink behavior) against a fixed CLI capture (OpenTUI behavior) with
machine-checkable metrics and reviewable artifacts. Node built-ins plus the
repository's existing @lydell/node-pty capability for PTY allocation.
Runner contract
node scripts/tui-parity/runner.mjs \
--scenario <file|dir> ... \
--out <dir> \
[--base '<command>'] [--fixed '<command>']
--scenariois a*.scenario.jsonfile or a directory of them.--outis the artifact directory. Exit codes: 0 pass, 1 threshold/capture failure, 2 usage or validation error.--base/--fixedoverride the scenario commands (single scenario only). This is the real-CLI contract: point base at the Ink build and fixed at the OpenTUI build with the same scenario parameters. Overrides always capture through a native PTY. Relative.js/.mjs/.cjsscript paths afternoderesolve against the repository root. The final resolved argv (after any override) is re-validated before anything is captured: declaredcompareParamsmust be present and equal on both sides, and terminal-size flags must match the capture geometry. A divergent override aborts the run with a parameter-binding error instead of executing under a stale "validated" claim.- Base and fixed always share the scenario's terminal size, timeout, stdin
input, environment, and thresholds. Scenarios that attach per-side
parameters are rejected.
compareParamsnames argv flags whose values are extracted, normalised (--flag valueand--flag=valueare equivalent), and required to be present and equal on both sides; terminal-size flags (--rows/--lines/--columns/--cols) must additionally matchterminal.rows/terminal.columns. Any mismatch fails validation before anything runs.
Native TTY capture is enforced
Plain argv commands are captured through a native PTY allocated by the
harness (@lydell/node-pty, rows/columns and TERM from the scenario). The
child therefore runs with a real TTY, and each side's summary.json records
the evidence (capture.tty: mode, backend, allocated, rows, columns,
handshake). If no PTY backend is available, direct capture is refused
with native TTY capture refused: ...; the harness never silently falls back
to non-TTY pipes for a real CLI. Two explicit waivers exist, both recorded in
artifacts and reports:
"pty": "fixture"— the command is a deterministic fixture that needs no TTY (the harness's own emitters). Captured over pipes and clearly marked as waived; real-CLI captures may not use this mode."pty": "wrapped"— the command already runs inside a caller-supplied PTY wrapper (expect, tmux,script, orlib/tty-handshake.mjs). The harness generates a per-run nonce inTUI_PARITY_PTY_NONCEand verifies that the captured stream contains the matchingESC ] 697 ; tty-handshake ; <nonce> BELmarker.lib/tty-handshake.mjsemits that marker only when its stdout really is a TTY, and refuses otherwise. A missing handshake fails the side.
TUI_PARITY_NO_PTY=1 forces the no-backend path in tests.
Scenario schema
{
"id": "kebab-case-id",
"description": "what is being compared",
"terminal": { "rows": 12, "columns": 40 },
"timeoutMs": 10000,
"input": [{ "data": "\r", "delayMs": 0 }],
"env": { "KEY": "value" },
"compareParams": ["--frames", "--rows"],
"commands": {
"base": ["argv", "captured through a native PTY"],
"fixed": { "argv": ["argv..."], "pty": "fixture" }
},
"thresholds": { "maxFullScreenClears": 0 },
"expectBaseFailure": false,
"proves": "what a passing run proves",
"doesNotProve": "what it cannot prove"
}
commands.<side> is either a plain argv array (native PTY capture) or an
object with argv and optional pty ("fixture" or "wrapped"). Any other
per-side key is rejected.
expectBaseFailure (optional boolean, default false) declares that the base
side is a defect fixture rather than a clean reference. Such a scenario passes
only when the base side fails: a both-pass there means the fixture emitted no
defect, so the comparison would report success while proving nothing.
Thresholds (all optional, evaluated per side): maxFullScreenClears,
maxPartialScreenErases, maxLineErases, maxDuplicateEvents,
maxDec2026Unbalanced (integers >= 0), and requireSync,
requireEventMarkers, requireExitCodeZero (booleans; the exit-code
requirement defaults to true; spawn failures, timeouts, PTY refusals, and
unverified handshakes always fail a side).
Metrics
Counted from each side's raw stdout:
- Full-screen clear opcodes:
ESC[2J,ESC[3J, RIS (ESC c). - Line erases:
ESC[Kin modes 0/1/2; partial screen erasesESC[Jmodes 0/1 tracked separately. - DEC 2026 synchronised-output begin/end (
ESC[?2026h/ESC[?2026l) plus an unbalanced count for stray ends and unclosed begins.requireSyncaccepts a stream only if at least one begin exists, pairs balance, and — when live-output event markers are measured — every marker occurrence was emitted inside an active DEC 2026 interval. Coverage is measured per event, not inferred from counts: empty begin/end pairs that wrap no events and markers emitted outside every interval failrequireSynceven when the begin count equals the unique event count. - Live-output event markers using OSC
697;<id>;<seq>(BEL or ST terminated). A marker whose(id, seq)was already seen is a duplicate. Each marker occurrence is additionally classified as covered (inside an active DEC 2026 interval) or unwrapped (outside every interval), and both counts are reported. Without markers the duplicate count is reported as not measurable. - Process exit code, signal, timeout, and duration.
Timeout handling tracks and clears every pending input timer, and kills the capture process tree where supported (SIGTERM to the process group, SIGKILL after a 1s grace). The kill escalation survives the main child closing: the SIGKILL step is not cancelled when the captured command exits, and after it fires the harness re-probes the process group until it is confirmed gone (bounded). A descendant that ignores SIGTERM and detaches stdio therefore cannot outlive the capture, and a capture cannot linger on delayed input or orphaned descendants.
proves and doesNotProve are mandatory scenario fields and are copied
verbatim into every report, so each scenario states what its result does and
does not demonstrate.
Artifacts
Per run under --out:
run-summary.json
<scenario-id>/
base|fixed/
raw.ansi exact captured stdout
stderr.txt captured stderr (empty for PTY captures: a real
terminal merges stderr into the PTY stream)
screen.txt final screen from the built-in terminal model
summary.json capture metadata (incl. tty evidence), metrics,
verdict, reasons
comparison.json side-by-side metrics, deltas, outcome
report.md reviewer-facing report
Outcomes: base-fails-fixed-passes (evidence that the fix removes the
defect), both-pass (no base-side defect exhibited), fixed-fails
(threshold violation), capture-error (spawn failure, refusal, or timeout).
The run passes only on the first two — and for a scenario with
expectBaseFailure, only on base-fails-fixed-passes.
Fixtures and tests
fixtures/emitters/tui-emitter.mjsis a deterministic emitter: identical flags produce identical bytes, and its flags inject the exact defects the metrics count (--clears-per-frame,--scrollback-clears,--dups,--sync,--hang-ms,--exit-code).--tree-hangspawns an idle grandchild and hangs, to exercise process-tree kill on timeout.--stubborn-hangspawns a grandchild that ignores SIGTERM and detaches its stdio, then hangs; it exercises kill escalation that must survive the main child closing.fixtures/wrappers/pty-launcher.mjswraps a command in a PTY, standing in for external wrappers in handshake tests.fixtures/scenarios/stream-redraw.scenario.jsonreproduces a failing base and passing fixed side without launching the real CLI.
node --test scripts/tui-parity/test # unit + end-to-end tests
node scripts/tui-parity/self-test.mjs # full pipeline self-check
Tests cover the PTY refusal and the native/wrapped happy paths, divergent
compareParams (including --rows/--columns), per-side parameter
rejection, and a bounded regression test where input.delayMs exceeds
timeoutMs. They also lock in the three adversarial counterexamples as
regressions: empty DEC 2026 pairs plus unwrapped event markers can never
satisfy requireSync; a descendant that ignores SIGTERM and detaches stdio
is still reaped after the main child closes (fixture and native PTY); and a
--base/--fixed override that diverges from compareParams or the capture
geometry aborts before anything runs. Fixture results are evidence about the
harness and its metrics, not about the products. Real-CLI parity claims
require running the same harness with real base/fixed commands through the
native PTY contract.
Normalization limits
The built-in terminal model supports printable text, CR/LF/BS/TAB, cursor
movement, ED/EL/IL/DL/ICH/DCH/ECH, SGR (ignored), alternate screen
(47/1047/1049), and RIS. Scroll regions, double-width characters, and
wide-grapheme measurement are not modeled; screen.txt is a review aid, and
the machine-checkable part of the harness is the metric counts. PTY captures
apply the terminal line discipline (e.g. LF echoed as CRLF); metrics count
opcodes, not whitespace, so this does not change threshold semantics.