Three fixes from review, all measured on this box.
The workload gate was files OR bytes. The files arm is wrong: 250 pending files
holding 117 KB between them spawned 5 threads and ran ~5% SLOWER than serial,
and a file count only starts paying for itself around 400. Gate on bytes alone;
the count still takes max(files / 50, bytes / 200 MB), so a few hundred huge
rollouts keep their threads.
The flat 256 MB per-worker memory budget was contradicted by the Codex workload:
a 260 MB rollout peaks near 430 MB in its worker, linearly across the pool. It is
now derived per parse as clamp(256 MB, 2 x average pending file + 128 MB, 1 GB),
which leaves a corpus of small Claude transcripts where it was and stops
over-subscribing on rollouts. The parent's buffer of up to pool.size finished
results is part of that peak and is named in the comment.
The worker/file pairing at both install sites was positional, guarded only by
position (Claude) or a path membership check (Codex). Each worker now echoes its
path and the parent asserts it, outside the per-file try: a misalignment would
install one session's turns under another's path -- a wrong number nobody would
ever notice -- so it fails the run rather than being swallowed as a parse
failure. On the Claude side that meant hoisting the whole worker-result block
above the try, which is safe because an append never consumes a result in either
its shortcut or its straddled-fallthrough case.
Codex is the bigger half of a real cold parse (4 GB of rollouts against 1.8 GB
of Claude sessions) and was still decoding one file at a time.
A whole-file rollout decode now runs on the #1008 pool. parseCodexFileFull is
the serial decode with the codex cache switched off: no hit lookup, and the
entry it would have written comes back to the parent instead. The worker runs it
against an EMPTY dedup set and returns the calls, the keys it claimed, and that
entry; the parent installs all three in the serial loop's order, so an empty key
intersection is the proof that a serial parse would have dropped nothing either.
On overlap the whole file is discarded and re-parsed in-process -- which is what
makes a forked rollout safe, since it replays its parent's token_count history
under the parent's key namespace and collides outright.
Nothing cross-file moves off the main thread: the dedup set, canonical project
paths and the codex cache's per-directory state all stay in the parent, and a
file the cache can serve exactly or resume into from a byte offset never reaches
a worker. The decision is per provider -- the Claude scan and the provider loop
run one after the other, so at most one pool is alive -- and the pool is
terminated when its scan ends.
The workload gate is now files OR bytes rather than both, and the count takes
max(files / 50, bytes / 200 MB): a corpus of a few hundred multi-hundred-MB
rollouts is as parallelisable as a few thousand small transcripts, and would
otherwise have earned one thread or none.
os.freemem() reports free pages on macOS, not available memory: on an idle
128 GB machine it reads a few hundred MB, so the 2 GB gate switched the worker
pool on and off between runs on the platform the desktop app ships to. The gate
and the budget now use process.availableMemory() (cgroup/rlimit-aware in a
container), falling back to os.totalmem(): serial under 4 GB available, budget
min(0.25 * available, 2 GB). An 8 GB box earns 8 threads, a 4 GB box none.
The verbose line now carries every decision input — cores, available GB, pending
files and bytes — on both the gate and the go path, so one support log explains
itself.
Reading, decoding and line-parsing a Claude session JSONL is per-file work that
touches nothing shared, so it moves onto worker_threads for a large cold parse.
parseClaudeFileFull() is the extracted unit both sides run; a worker runs it
against an empty dedup set and returns the result as a JSON string, and the
parent installs results in the order the serial loop would. Everything with
cross-file state stays on the main thread, and a file whose message ids were
already claimed (or whose worker failed) re-parses in-process, so the output is
identical to the serial path.
Thread count is decided per parse: never with <=2 cores, under 2 GB free memory,
fewer than 200 pending whole-file re-parses or under 200 MB behind them, so warm
and incremental runs spawn nothing. CODEBURN_PARSE_WORKERS overrides it. The pool
is terminated when the parse ends, so the resident serve child accumulates no
threads.
