fix(parser): gate parse workers on available memory, not free memory

os.freemem() reports free pages on macOS, not available memory: on an idle
128 GB machine it reads a few hundred MB, so the 2 GB gate switched the worker
pool on and off between runs on the platform the desktop app ships to. The gate
and the budget now use process.availableMemory() (cgroup/rlimit-aware in a
container), falling back to os.totalmem(): serial under 4 GB available, budget
min(0.25 * available, 2 GB). An 8 GB box earns 8 threads, a 4 GB box none.

The verbose line now carries every decision input — cores, available GB, pending
files and bytes — on both the gate and the go path, so one support log explains
itself.

CHANGELOG.md

@@ -3,7 +3,7 @@
 ## Unreleased
 
 ### Changed
-- **A large cold Claude parse now runs across worker threads.** Reading, decoding and line-parsing a session JSONL is per-file work that never touches anything shared, so it moves onto `worker_threads`; each worker ships its parsed turns back as a JSON string and the parent installs them in the exact order the serial loop would. Everything with cross-file state — the streaming-message dedup, canonical project paths, spawn links, PR correlation, progress saves — stays on the main thread, and a file whose message ids were already claimed by an earlier file (or whose worker failed) is simply re-parsed in-process, so the session cache and every payload are identical either way. On a 6 GB corpus a cold `status` drops from 27.5s to 14.8s with peak RSS up 2.27 GB → 2.52 GB. Threads only engage for a genuinely large cold parse: never with 2 or fewer cores, under 2 GB free memory, fewer than 200 pending whole-file re-parses or under 200 MB behind them — so warm and incremental runs are untouched and spawn nothing. Otherwise the count is `min(cores - 1, min(0.4 × freemem, 2 GB) / 256 MB, pendingFiles / 50)`. `CODEBURN_PARSE_WORKERS=0` forces the serial parse and `CODEBURN_PARSE_WORKERS=N` forces N (capped at the core count), both bypassing every gate; `CODEBURN_VERBOSE=1` prints the resolved count and why. Only Claude sessions are parallelized so far.
+- **A large cold Claude parse now runs across worker threads.** Reading, decoding and line-parsing a session JSONL is per-file work that never touches anything shared, so it moves onto `worker_threads`; each worker ships its parsed turns back as a JSON string and the parent installs them in the exact order the serial loop would. Everything with cross-file state — the streaming-message dedup, canonical project paths, spawn links, PR correlation, progress saves — stays on the main thread, and a file whose message ids were already claimed by an earlier file (or whose worker failed) is simply re-parsed in-process, so the session cache and every payload are identical either way. On a 6 GB corpus a cold `status` drops from 27.5s to 14.8s with peak RSS up 2.27 GB → 2.52 GB. Threads only engage for a genuinely large cold parse: never with fewer than 200 pending whole-file re-parses, under 200 MB behind them, 2 or fewer cores, or under 4 GB of available memory — so warm and incremental runs are untouched and spawn nothing. Otherwise the count is `min(cores - 1, min(0.25 × available, 2 GB) / 256 MB, pendingFiles / 50)`, where available is `process.availableMemory()` (cgroup-aware in containers) rather than free memory, which on macOS reports free pages and would switch the feature on and off between runs. `CODEBURN_PARSE_WORKERS=0` forces the serial parse and `CODEBURN_PARSE_WORKERS=N` forces N (capped at the core count), both bypassing every gate; `CODEBURN_VERBOSE=1` prints the resolved count and why. Only Claude sessions are parallelized so far.
 - **A warm launch rewrites only the month that changed, and a ranged query reads only the months it can report on.** Per-provider shards still meant one appended session republished that provider's entire history — 95 MB for Claude on a 6 GB corpus. Each provider's shard is now split again by the UTC month of the cached session's FIRST turn, a bucket that never moves as a session grows, so an append rewrites one month. Every shard records the newest month it holds, which lets `--period today/week` skip the shards that cannot contribute a turn to the range; the skipped months stay on disk untouched across the save, and providers whose cache is the only surviving record (durable) or whose parse fingerprint moved are always read in full. Remaining shards are read concurrently. Existing v8 and v7 caches are re-laid-out losslessly on first load and the old layout removed once the new one is published: nothing re-parses.
 - **A warm launch rewrites only the provider that changed.** The session cache was a single blob, so any provider appending a few KB republished the whole thing — 147 MB of stringify + fsync on a 6 GB corpus, ~18% of a warm run. It is now a version-suffixed directory holding one shard per provider plus a small envelope, written per provider and published by a single envelope rename. An existing v7 cache is re-laid-out losslessly on first load and the old file removed once the new layout is on disk: nothing re-parses. One unreadable shard now costs that provider a re-parse instead of discarding every provider's history, and partial saves during a cold parse are triggered every 2000 files rather than every 5 seconds, so a slow cold parse no longer rewrites the growing cache on a wall clock.
 - **An appended Codex rollout parses only its tail.** Rollout files are append-only and the active ones run to hundreds of MB, but the Codex result cache keyed on mtime + size alone, so any growth re-read the file from byte 0. Each entry now records a restart point at the last task boundary — byte offset plus the state the single-pass decode carries across it — and a grown file with the same inode resumes there, producing output identical to a full re-parse. An entry without a usable restart point simply re-parses in full once and gains one.

docs/architecture.md

@@ -89,13 +89,18 @@ The pool is off by default for anything that is not a large cold parse:
 
 | Gate | Serial when |
 |---|---|
-| Cores | `availableParallelism() <= 2` |
-| Free memory | `os.freemem() < 2 GB` |
 | Pending files | fewer than 200 whole-file re-parses |
 | Pending bytes | under 200 MB behind those files |
+| Cores | `availableParallelism() <= 2` |
+| Memory | under 4 GB available |
 
 Otherwise the worker count is
-`min(cores - 1, min(0.4 * freemem, 2 GB) / 256 MB, pendingFiles / 50)`.
+`min(cores - 1, min(0.25 * available, 2 GB) / 256 MB, pendingFiles / 50)`.
+
+"Available" is `process.availableMemory()`, falling back to `os.totalmem()`. It is
+deliberately not `os.freemem()`: on macOS that counts free pages rather than
+available memory and reads as a few hundred MB on an idle 128 GB machine, so a
+gate built on it switches the feature on and off between runs.
 
 `CODEBURN_PARSE_WORKERS` overrides the decision and skips every gate above:
 `0` forces the serial parse, `N` forces N workers (capped at the core count).

src/parse-workers.ts

@@ -1,4 +1,4 @@
-import { availableParallelism, freemem } from 'os'
+import { availableParallelism, totalmem } from 'os'
 import { Worker } from 'worker_threads'
 import { snapshotPricingState } from './models.js'
 import type { ClaudeFileParse } from './parser.js'
@@ -8,6 +8,7 @@ import type { ClaudeFileParse } from './parser.js'
 // budget is what turns a parallel parse into a swapping one.
 const PER_WORKER_RSS_BYTES = 256 * 1024 * 1024
 const MEMORY_BUDGET_CAP_BYTES = 2 * 1024 * 1024 * 1024
+const MIN_AVAILABLE_BYTES = 4 * 1024 * 1024 * 1024
 const MIN_FILES_PER_WORKER = 50
 // Below these, a parse is warm/incremental and the thread startup + result
 // transfer costs more than the parallelism buys.
@@ -16,10 +17,18 @@ const MIN_PENDING_BYTES = 200 * 1024 * 1024
 
 export type ParseWorkerDecision = { workers: number; reason: string }
 
-export type SystemCapacity = { cores: number; freeBytes: number }
+export type SystemCapacity = { cores: number; availableBytes: number }
 
+// `process.availableMemory()` respects a container's cgroup / rlimit, which is the
+// case this gate exists for; where it is absent it falls back to total RAM. Free
+// memory is deliberately NOT used: on macOS `os.freemem()` counts free pages, not
+// available memory, and reads as a few hundred MB on an idle 128 GB machine — a
+// gate built on it turns the feature on and off at random.
 function currentSystemCapacity(): SystemCapacity {
-  return { cores: availableParallelism(), freeBytes: freemem() }
+  return {
+    cores: availableParallelism(),
+    availableBytes: typeof process.availableMemory === 'function' ? process.availableMemory() : totalmem(),
+  }
 }
 
 /// Decide how many parse worker threads a pending workload earns. Returning 0
@@ -30,28 +39,32 @@ export function decideParseWorkers(
   sys: SystemCapacity = currentSystemCapacity(),
   env: NodeJS.ProcessEnv = process.env,
 ): ParseWorkerDecision {
+  // Every reason carries the full decision input, so a support log line explains
+  // itself without a second run.
+  const inputs = `${sys.cores} cores, ${Math.round(sys.availableBytes / 1e9 * 10) / 10} GB available, ${pending.files} pending files / ${Math.round(pending.bytes / 1e6)} MB`
+
   const override = env['CODEBURN_PARSE_WORKERS']
   if (override !== undefined && override !== '') {
     const n = Number(override)
     if (!Number.isFinite(n) || n < 0) return { workers: 0, reason: `invalid CODEBURN_PARSE_WORKERS=${override}` }
     const capped = Math.min(Math.floor(n), sys.cores)
-    return { workers: capped, reason: capped === 0 ? 'forced serial (CODEBURN_PARSE_WORKERS=0)' : `forced by CODEBURN_PARSE_WORKERS=${override}` }
+    return { workers: capped, reason: `${capped === 0 ? 'forced serial' : 'forced'} by CODEBURN_PARSE_WORKERS=${override}; ${inputs}` }
   }
 
   // Workload gates first, so a warm run's log line says "warm", not whatever the
   // machine happened to look like at that moment.
-  if (pending.files < MIN_PENDING_FILES) return { workers: 0, reason: `${pending.files} pending files below ${MIN_PENDING_FILES}` }
-  if (pending.bytes < MIN_PENDING_BYTES) return { workers: 0, reason: `${Math.round(pending.bytes / 1e6)} MB pending below ${Math.round(MIN_PENDING_BYTES / 1e6)} MB` }
-  if (sys.cores <= 2) return { workers: 0, reason: `only ${sys.cores} cores available` }
-  if (sys.freeBytes < 2 * 1024 * 1024 * 1024) return { workers: 0, reason: `only ${Math.round(sys.freeBytes / 1e6)} MB free memory` }
+  if (pending.files < MIN_PENDING_FILES) return { workers: 0, reason: `below ${MIN_PENDING_FILES} pending files; ${inputs}` }
+  if (pending.bytes < MIN_PENDING_BYTES) return { workers: 0, reason: `below ${Math.round(MIN_PENDING_BYTES / 1e6)} MB pending; ${inputs}` }
+  if (sys.cores <= 2) return { workers: 0, reason: `too few cores; ${inputs}` }
+  if (sys.availableBytes < MIN_AVAILABLE_BYTES) return { workers: 0, reason: `below ${Math.round(MIN_AVAILABLE_BYTES / 1e9)} GB available memory; ${inputs}` }
 
-  const memoryBudget = Math.min(0.4 * sys.freeBytes, MEMORY_BUDGET_CAP_BYTES)
+  const memoryBudget = Math.min(0.25 * sys.availableBytes, MEMORY_BUDGET_CAP_BYTES)
   const workers = Math.min(
     sys.cores - 1,
     Math.floor(memoryBudget / PER_WORKER_RSS_BYTES),
     Math.floor(pending.files / MIN_FILES_PER_WORKER),
   )
-  return { workers, reason: `${sys.cores} cores, ${Math.round(sys.freeBytes / 1e9 * 10) / 10} GB free, ${pending.files} pending files` }
+  return { workers, reason: inputs }
 }
 
 // In dist the entry is the bundled sibling of this module and a worker can load

tests/parse-workers.test.ts

@@ -13,25 +13,30 @@ import { clearSessionCache, parseAllSessions, parseClaudeFileFull } from '../src
 // that spawn real threads.
 vi.setConfig({ testTimeout: 60_000 })
 
-const BIG_SYSTEM = { cores: 16, freeBytes: 32 * 1024 ** 3 }
+const BIG_SYSTEM = { cores: 16, availableBytes: 32 * 1024 ** 3 }
 const BIG_PENDING = { files: 5000, bytes: 6 * 1024 ** 3 }
 const NO_ENV = {} as NodeJS.ProcessEnv
 
 describe('decideParseWorkers', () => {
   it('scales with cores, memory budget and pending file count', () => {
-    // 15 (cores-1) vs 8 (2 GB budget / 256 MB) vs 100 (5000/50) -> memory wins
+    // 15 (cores-1) vs 8 (2 GB budget / 256 MB) vs 100 (5000/50) -> memory cap wins
     expect(decideParseWorkers(BIG_PENDING, BIG_SYSTEM, NO_ENV).workers).toBe(8)
     // Fewer cores than the memory budget allows -> cores-1 wins
-    expect(decideParseWorkers(BIG_PENDING, { cores: 6, freeBytes: 32 * 1024 ** 3 }, NO_ENV).workers).toBe(5)
+    expect(decideParseWorkers(BIG_PENDING, { cores: 6, availableBytes: 32 * 1024 ** 3 }, NO_ENV).workers).toBe(5)
+    // 8 GB reaches the same cap as 32 GB: a quarter of it is the 2 GB budget
+    expect(decideParseWorkers(BIG_PENDING, { cores: 16, availableBytes: 8 * 1024 ** 3 }, NO_ENV).workers).toBe(8)
+    // Under that, the quarter-of-available budget is the binding constraint
+    expect(decideParseWorkers(BIG_PENDING, { cores: 16, availableBytes: 6 * 1024 ** 3 }, NO_ENV).workers).toBe(6)
     // The smallest machine that clears every gate still only earns 2 threads
-    expect(decideParseWorkers({ files: 200, bytes: 300 * 1024 ** 2 }, { cores: 3, freeBytes: 2 * 1024 ** 3 }, NO_ENV).workers).toBe(2)
+    expect(decideParseWorkers({ files: 200, bytes: 300 * 1024 ** 2 }, { cores: 3, availableBytes: 4 * 1024 ** 3 }, NO_ENV).workers).toBe(2)
     // Few enough files that MIN_FILES_PER_WORKER is the binding constraint
     expect(decideParseWorkers({ files: 300, bytes: 6 * 1024 ** 3 }, BIG_SYSTEM, NO_ENV).workers).toBe(6)
   })
 
   it('stays serial on low-spec machines and on warm/small parses', () => {
-    expect(decideParseWorkers(BIG_PENDING, { cores: 2, freeBytes: 32 * 1024 ** 3 }, NO_ENV).workers).toBe(0)
-    expect(decideParseWorkers(BIG_PENDING, { cores: 16, freeBytes: 1024 ** 3 }, NO_ENV).workers).toBe(0)
+    expect(decideParseWorkers(BIG_PENDING, { cores: 2, availableBytes: 32 * 1024 ** 3 }, NO_ENV).workers).toBe(0)
+    // A 4 GB box: availableMemory() always reads a little under the nominal size
+    expect(decideParseWorkers(BIG_PENDING, { cores: 16, availableBytes: 3.9 * 1024 ** 3 }, NO_ENV).workers).toBe(0)
     // Warm/incremental: a handful of appended files
     expect(decideParseWorkers({ files: 12, bytes: 6 * 1024 ** 3 }, BIG_SYSTEM, NO_ENV).workers).toBe(0)
     // Many files but almost no bytes behind them
@@ -42,12 +47,24 @@ describe('decideParseWorkers', () => {
     expect(decideParseWorkers(BIG_PENDING, BIG_SYSTEM, { CODEBURN_PARSE_WORKERS: '0' }).workers).toBe(0)
     expect(decideParseWorkers(BIG_PENDING, BIG_SYSTEM, { CODEBURN_PARSE_WORKERS: '4' }).workers).toBe(4)
     // Capped by the core count
-    expect(decideParseWorkers(BIG_PENDING, { cores: 4, freeBytes: 32 * 1024 ** 3 }, { CODEBURN_PARSE_WORKERS: '32' }).workers).toBe(4)
+    expect(decideParseWorkers(BIG_PENDING, { cores: 4, availableBytes: 32 * 1024 ** 3 }, { CODEBURN_PARSE_WORKERS: '32' }).workers).toBe(4)
     // A tiny fixture corpus still gets threads when forced — that is what makes
     // the determinism test below able to exercise them at all.
     expect(decideParseWorkers({ files: 3, bytes: 1000 }, BIG_SYSTEM, { CODEBURN_PARSE_WORKERS: '3' }).workers).toBe(3)
     expect(decideParseWorkers(BIG_PENDING, BIG_SYSTEM, { CODEBURN_PARSE_WORKERS: 'nonsense' }).workers).toBe(0)
   })
+
+  it('reports the decision inputs in every reason, gate or not', () => {
+    for (const d of [
+      decideParseWorkers(BIG_PENDING, BIG_SYSTEM, NO_ENV),
+      decideParseWorkers({ files: 12, bytes: 1000 }, BIG_SYSTEM, NO_ENV),
+      decideParseWorkers(BIG_PENDING, BIG_SYSTEM, { CODEBURN_PARSE_WORKERS: '2' }),
+    ]) {
+      expect(d.reason).toContain('16 cores')
+      expect(d.reason).toContain('GB available')
+      expect(d.reason).toContain('pending files')
+    }
+  })
 })
 
 function sessionLines(project: number, session: string, turns: number): string {