docs: describe the Codex half of the parallel cold parse

CHANGELOG.md

@@ -3,7 +3,8 @@
 ## Unreleased
 
 ### Changed
-- **A large cold Claude parse now runs across worker threads.** Reading, decoding and line-parsing a session JSONL is per-file work that never touches anything shared, so it moves onto `worker_threads`; each worker ships its parsed turns back as a JSON string and the parent installs them in the exact order the serial loop would. Everything with cross-file state — the streaming-message dedup, canonical project paths, spawn links, PR correlation, progress saves — stays on the main thread, and a file whose message ids were already claimed by an earlier file (or whose worker failed) is simply re-parsed in-process, so the session cache and every payload are identical either way. On a 6 GB corpus a cold `status` drops from 27.5s to 14.8s with peak RSS up 2.27 GB → 2.52 GB. Threads only engage for a genuinely large cold parse: never with fewer than 200 pending whole-file re-parses, under 200 MB behind them, 2 or fewer cores, or under 4 GB of available memory — so warm and incremental runs are untouched and spawn nothing. Otherwise the count is `min(cores - 1, min(0.25 × available, 2 GB) / 256 MB, pendingFiles / 50)`, where available is `process.availableMemory()` (cgroup-aware in containers) rather than free memory, which on macOS reports free pages and would switch the feature on and off between runs. `CODEBURN_PARSE_WORKERS=0` forces the serial parse and `CODEBURN_PARSE_WORKERS=N` forces N (capped at the core count), both bypassing every gate; `CODEBURN_VERBOSE=1` prints the resolved count and why. Only Claude sessions are parallelized so far.
+- **Codex rollouts parse across worker threads too, and the workload gate now takes bytes or files.** Codex is the bigger half of a real cold parse — a 4 GB rollout corpus against 1.8 GB of Claude sessions — and it was still decoding one file at a time. A whole-file rollout decode now runs on the same pool, against an empty dedup set, and comes back with the calls, the dedup keys it claimed, and the codex-cache entry it would have written; the parent installs all three in the serial loop's order, so `codex-results.json` and every payload come out byte-identical to a serial run. Cross-file state stays where it was: a forked rollout replaying its parent's token_count history collides on the parent's keys and is re-parsed in-process, and no worker ever touches the cache module's per-directory state. Files the Codex cache can serve exactly or resume into from a byte offset never reach a worker — they read a few KB and the resume state belongs to the parent. The workload gate is now files OR bytes rather than both (200 whole-file parses or 200 MB behind them), and the count takes `max(pendingFiles / 50, pendingBytes / 200 MB)`, so a corpus of a few hundred huge rollouts parallelizes instead of falling back to one thread. The decision is per provider, and at most one pool is alive at a time.
+- **A large cold Claude parse now runs across worker threads.** Reading, decoding and line-parsing a session JSONL is per-file work that never touches anything shared, so it moves onto `worker_threads`; each worker ships its parsed turns back as a JSON string and the parent installs them in the exact order the serial loop would. Everything with cross-file state — the streaming-message dedup, canonical project paths, spawn links, PR correlation, progress saves — stays on the main thread, and a file whose message ids were already claimed by an earlier file (or whose worker failed) is simply re-parsed in-process, so the session cache and every payload are identical either way. On a 6 GB corpus a cold `status` drops from 27.5s to 14.8s with peak RSS up 2.27 GB → 2.52 GB. Threads only engage for a genuinely large cold parse: never with fewer than 200 pending whole-file re-parses and under 200 MB behind them, 2 or fewer cores, or under 4 GB of available memory — so warm and incremental runs are untouched and spawn nothing. Otherwise the count is `min(cores - 1, min(0.25 × available, 2 GB) / 256 MB, pendingFiles / 50)`, where available is `process.availableMemory()` (cgroup-aware in containers) rather than free memory, which on macOS reports free pages and would switch the feature on and off between runs. `CODEBURN_PARSE_WORKERS=0` forces the serial parse and `CODEBURN_PARSE_WORKERS=N` forces N (capped at the core count), both bypassing every gate; `CODEBURN_VERBOSE=1` prints the resolved count and why.
 - **A warm launch rewrites only the month that changed, and a ranged query reads only the months it can report on.** Per-provider shards still meant one appended session republished that provider's entire history — 95 MB for Claude on a 6 GB corpus. Each provider's shard is now split again by the UTC month of the cached session's FIRST turn, a bucket that never moves as a session grows, so an append rewrites one month. Every shard records the newest month it holds, which lets `--period today/week` skip the shards that cannot contribute a turn to the range; the skipped months stay on disk untouched across the save, and providers whose cache is the only surviving record (durable) or whose parse fingerprint moved are always read in full. Remaining shards are read concurrently. Existing v8 and v7 caches are re-laid-out losslessly on first load and the old layout removed once the new one is published: nothing re-parses.
 - **A warm launch rewrites only the provider that changed.** The session cache was a single blob, so any provider appending a few KB republished the whole thing — 147 MB of stringify + fsync on a 6 GB corpus, ~18% of a warm run. It is now a version-suffixed directory holding one shard per provider plus a small envelope, written per provider and published by a single envelope rename. An existing v7 cache is re-laid-out losslessly on first load and the old file removed once the new layout is on disk: nothing re-parses. One unreadable shard now costs that provider a re-parse instead of discarding every provider's history, and partial saves during a cold parse are triggered every 2000 files rather than every 5 seconds, so a slow cold parse no longer rewrites the growing cache on a wall clock.
 - **An appended Codex rollout parses only its tail.** Rollout files are append-only and the active ones run to hundreds of MB, but the Codex result cache keyed on mtime + size alone, so any growth re-read the file from byte 0. Each entry now records a restart point at the last task boundary — byte offset plus the state the single-pass decode carries across it — and a grown file with the same inode resumes there, producing output identical to a full re-parse. An entry without a usable restart point simply re-parses in full once and gains one.

docs/architecture.md

@@ -71,31 +71,43 @@ output formatter (Ink TUI, JSON, or menubar-json)
 
 ### Parallel Cold Parse
 
-A cold Claude parse spends most of its time on work that is per-file and pure:
-reading a session JSONL, decoding it, and turning each line into a journal entry.
-`src/parse-workers.ts` moves that onto `worker_threads` when the pending workload
-is big enough to pay for them. Each worker runs `parseClaudeFileFull` against an
-empty dedup set and ships the result back as a JSON string; the parent installs
-results in the same order the serial loop would, and everything with cross-file
-state (the `seenMsgIds` dedup, canonical project paths, spawn links, PR
-correlation) stays on the main thread. A file whose message ids were already
-claimed by an earlier file, or whose worker failed, is re-parsed in-process — so
-the output is identical to the serial path either way. Only whole-file re-parses
-go off-thread; the append/incremental path is untouched. Workers are created at
-the start of a qualifying parse and terminated when it ends, so the resident
-`serve` child never accumulates threads.
+A cold parse spends most of its time on work that is per-file and pure: reading a
+session JSONL or a Codex rollout, decoding it, and turning each line into a
+journal entry. `src/parse-workers.ts` moves that onto `worker_threads` when the
+pending workload is big enough to pay for them. Each worker runs the same
+per-file function the serial path runs — `parseClaudeFileFull` for a Claude
+session, `parseCodexFileFull` for a Codex rollout — against an empty dedup set,
+and ships the result back as a JSON string together with every dedup key it
+claimed. The parent installs results in the same order the serial loop would, and
+everything with cross-file state (the dedup sets, canonical project paths, spawn
+links, PR correlation, the Codex result cache) stays on the main thread. A file
+whose keys were already claimed by an earlier file, or whose worker failed, is
+re-parsed in-process — so the output is identical to the serial path either way.
+That overlap check is what makes a forked Codex rollout safe: it replays its
+parent's token_count history under the parent's key namespace, collides, and is
+re-parsed against the real dedup set.
+
+A Codex worker never touches `src/codex-cache.ts`: it returns the cache entry it
+would have written and the parent writes it, in install order, so
+`flushCodexCache` publishes exactly what a serial parse would. Only whole-file
+parses go off-thread; the append/incremental paths (a Claude append, a Codex
+byte-offset resume) are untouched and stay in-process. The decision is made per
+provider — the Claude scan and the provider loop run one after the other, so at
+most one pool is alive — and the pool is terminated when its scan ends, so the
+resident `serve` child never accumulates threads.
 
 The pool is off by default for anything that is not a large cold parse:
 
 | Gate | Serial when |
 |---|---|
-| Pending files | fewer than 200 whole-file re-parses |
-| Pending bytes | under 200 MB behind those files |
+| Pending workload | fewer than 200 whole-file parses AND under 200 MB behind them (either one on its own qualifies) |
 | Cores | `availableParallelism() <= 2` |
 | Memory | under 4 GB available |
 
 Otherwise the worker count is
-`min(cores - 1, min(0.25 * available, 2 GB) / 256 MB, pendingFiles / 50)`.
+`min(cores - 1, min(0.25 * available, 2 GB) / 256 MB, max(pendingFiles / 50, pendingBytes / 200 MB))`.
+Files and bytes each earn threads on their own, so a few hundred multi-hundred-MB
+Codex rollouts parallelize as well as a few thousand small Claude transcripts.
 
 "Available" is `process.availableMemory()`, falling back to `os.totalmem()`. It is
 deliberately not `os.freemem()`: on macOS that counts free pages rather than