diff --git a/CHANGELOG.md b/CHANGELOG.md index 477c7f4f..9242c070 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,23 @@ ## Unreleased +### Changed +- **Codex rollouts parse across worker threads too, and the workload gate now takes bytes or files.** Codex is the bigger half of a real cold parse — a 4 GB rollout corpus against 1.8 GB of Claude sessions — and it was still decoding one file at a time. A whole-file rollout decode now runs on the same pool, against an empty dedup set, and comes back with the calls, the dedup keys it claimed, and the codex-cache entry it would have written; the parent installs all three in the serial loop's order, so `codex-results.json` and every payload come out byte-identical to a serial run. Cross-file state stays where it was: a forked rollout replaying its parent's token_count history collides on the parent's keys and is re-parsed in-process, and no worker ever touches the cache module's per-directory state. Files the Codex cache can serve exactly or resume into from a byte offset never reach a worker — they read a few KB and the resume state belongs to the parent. The workload gate is now pending BYTES alone (200 MB), not file count: 250 pending files holding under a megabyte between them spawned threads that made the run ~5% slower, while a few hundred huge rollouts were being turned away. The count takes `max(pendingFiles / 50, pendingBytes / 200 MB)`, and the per-thread memory budget is derived per parse as `clamp(256 MB, 2 × average pending file + 128 MB, 1 GB)` rather than a flat 256 MB — a 260 MB rollout peaks near 430 MB in its worker and scales linearly with the pool, so the flat figure over-subscribed exactly the workload this adds. The decision is per provider, and at most one pool is alive at a time. +- **A large cold Claude parse now runs across worker threads.** Reading, decoding and line-parsing a session JSONL is per-file work that never touches anything shared, so it moves onto `worker_threads`; each worker ships its parsed turns back as a JSON string and the parent installs them in the exact order the serial loop would. Everything with cross-file state — the streaming-message dedup, canonical project paths, spawn links, PR correlation, progress saves — stays on the main thread, and a file whose message ids were already claimed by an earlier file (or whose worker failed) is simply re-parsed in-process, so the session cache and every payload are identical either way. On a 6 GB corpus a cold `status` drops from 27.5s to 14.8s with peak RSS up 2.27 GB → 2.52 GB. Threads only engage for a genuinely large cold parse: never with under 200 MB behind the pending whole-file re-parses, 2 or fewer cores, or under 4 GB of available memory — so warm and incremental runs are untouched and spawn nothing. Otherwise the count is `min(cores - 1, min(0.25 × available, 2 GB) / 256 MB, pendingFiles / 50)`, where available is `process.availableMemory()` (cgroup-aware in containers) rather than free memory, which on macOS reports free pages and would switch the feature on and off between runs. `CODEBURN_PARSE_WORKERS=0` forces the serial parse and `CODEBURN_PARSE_WORKERS=N` forces N (capped at the core count), both bypassing every gate; `CODEBURN_VERBOSE=1` prints the resolved count and why. +- **A warm launch rewrites only the month that changed, and a ranged query reads only the months it can report on.** Per-provider shards still meant one appended session republished that provider's entire history — 95 MB for Claude on a 6 GB corpus. Each provider's shard is now split again by the UTC month of the cached session's FIRST turn, a bucket that never moves as a session grows, so an append rewrites one month. Every shard records the newest month it holds, which lets `--period today/week` skip the shards that cannot contribute a turn to the range; the skipped months stay on disk untouched across the save, and providers whose cache is the only surviving record (durable) or whose parse fingerprint moved are always read in full. Remaining shards are read concurrently. Existing v8 and v7 caches are re-laid-out losslessly on first load and the old layout removed once the new one is published: nothing re-parses. +- **A warm launch rewrites only the provider that changed.** The session cache was a single blob, so any provider appending a few KB republished the whole thing — 147 MB of stringify + fsync on a 6 GB corpus, ~18% of a warm run. It is now a version-suffixed directory holding one shard per provider plus a small envelope, written per provider and published by a single envelope rename. An existing v7 cache is re-laid-out losslessly on first load and the old file removed once the new layout is on disk: nothing re-parses. One unreadable shard now costs that provider a re-parse instead of discarding every provider's history, and partial saves during a cold parse are triggered every 2000 files rather than every 5 seconds, so a slow cold parse no longer rewrites the growing cache on a wall clock. +- **An appended Codex rollout parses only its tail.** Rollout files are append-only and the active ones run to hundreds of MB, but the Codex result cache keyed on mtime + size alone, so any growth re-read the file from byte 0. Each entry now records a restart point at the last task boundary — byte offset plus the state the single-pass decode carries across it — and a grown file with the same inode resumes there, producing output identical to a full re-parse. An entry without a usable restart point simply re-parses in full once and gains one. +- **A date-ranged report classifies only the turns it keeps.** Every cached turn went through the turn classifier — category, retries, edit detection, and a full reconstruction of its API calls — before the date slice discarded most of them, so a week view paid to classify all of history to keep a few percent of it. The keep/drop decision is now taken on the raw cached turn and only the survivors are classified, still from their complete call list, with the branch and pull-request carries still walking the full ordered turn list. Output is byte-identical. +- **One rule for every cache file.** `CODEBURN_CACHE_DIR` when set, otherwise `~/.cache/codeburn`. `XDG_CACHE_HOME` is no longer consulted; the sync ledger, the only file that ever honored it, is merged into the canonical location on first read and the legacy copy is retired, so nothing is re-uploaded after the move. (#972) + +### Fixed (Desktop & Menubar) +- **The resident `codeburn serve` child.** The first real panel request is also the cache warm-up, so startup never runs an artificial warm-up query beside a duplicate one-shot child; each served command carries its own read-only option allowlist, and anything outside it falls back to a normal spawn; the child exits when its stdin closes, so it can never outlive the app. Requests whose response exceeds the 16 MiB frame limit still replace the child, but that deliberate kill no longer spends the resident's unexpected-death budget. (#972) + +### Fixed +- **Bash command splitting was quadratic on long whitespace-heavy commands.** The separator regex retried its leading `\s*` from every offset; matching the separator alone and widening over whitespace by hand makes cold parse ~24% and warm ~40% faster on large corpora, output unchanged. +- **Cold parse no longer retains full message bodies through cached previews.** `flatSlice` skipped its Buffer round-trip for strings already within the bound, but provider adapters pre-truncate user-message previews with `.slice(0, 500)` before the cache-site call — those pre-sliced views are still V8 SlicedStrings pinning their large parent, so the retention that OOM'd cold parses of large histories survived. The round-trip now always runs. +- **Kiro sessions carry the real `projectPath`** (CLI meta.cwd, v2 `workspacePaths[0]`, workspace sessions' `workspaceDirectory`), so git-repo attribution can resolve them; previously they were attribution-blind. Bumps the kiro parse version, so the first run after upgrade re-parses kiro history once, and kiro sessions in linked git worktrees now group under the main repo. + ## 0.9.20 - 2026-08-10 ### Added diff --git a/app/electron/cli.test.ts b/app/electron/cli.test.ts index e95bc93d..0fb11504 100644 --- a/app/electron/cli.test.ts +++ b/app/electron/cli.test.ts @@ -1,10 +1,10 @@ // @vitest-environment node -import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' +import { describe, it, expect, beforeEach, afterEach } from 'vitest' import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync, chmodSync } from 'node:fs' import { tmpdir } from 'node:os' import { dirname, join, isAbsolute, relative, win32, posix } from 'node:path' -import { spawnCli, spawnCliAction, spawnEnvFor, spawnSpecFor, killAll, CliError, nodeManagerDirs, notFoundStage, resolveCodeburnPath, resolveTarget } from './cli' +import { spawnCli, spawnCliAction, spawnEnvFor, spawnSpecFor, startServe, killAll, shutdownAll, CliError, nodeManagerDirs, notFoundStage, resolveCodeburnPath, resolveTarget } from './cli' let dir: string const originalBin = process.env.CODEBURN_BIN @@ -23,6 +23,61 @@ function fakeBin(name: string, body: string): string { return p } +function readMaybe(path: string): string { + try { return readFileSync(path, 'utf8') } catch { return '' } +} + +async function waitFor(condition: () => boolean, timeoutMs = 3_000): Promise { + const deadline = Date.now() + timeoutMs + while (!condition()) { + if (Date.now() > deadline) throw new Error('waitFor timed out') + await new Promise(resolve => setTimeout(resolve, 10)) + } +} + +/** A protocol-faithful fake CLI whose serve child accepts requests before its + * delayed ready frame. Files expose process starts and heavy request executions + * without relying on timing or private ServeClient internals. */ +function fakeResidentBin(): { + startsFile: string + heavyFile: string + oneShotsFile: string + actionsFile: string + serveEnvFile: string +} { + const startsFile = join(dir, 'serve-starts') + const heavyFile = join(dir, 'heavy-requests') + const oneShotsFile = join(dir, 'one-shot-reads') + const actionsFile = join(dir, 'actions') + const serveEnvFile = join(dir, 'serve-progress-env') + fakeBin( + 'resident.js', + `const fs = require('node:fs'); const readline = require('node:readline'); + const command = process.argv[2]; + if (command === 'serve') { + fs.appendFileSync(${JSON.stringify(startsFile)}, 's'); + const generation = fs.readFileSync(${JSON.stringify(startsFile)}, 'utf8').length; + fs.writeFileSync(${JSON.stringify(serveEnvFile)}, process.env.CODEBURN_PROGRESS || ''); + const rl = readline.createInterface({ input: process.stdin }); + rl.on('line', line => { + const request = JSON.parse(line); + fs.appendFileSync(${JSON.stringify(heavyFile)}, 'h'); + const progress = 'CODEBURN_PROGRESS ' + JSON.stringify({ kind: 'provider', provider: 'claude', state: 'start', generation }) + '\\n'; + process.stdout.write(JSON.stringify({ id: request.id, progress }) + '\\n'); + process.stdout.write(JSON.stringify({ id: request.id, ok: true, output: JSON.stringify({ via: 'serve', generation, args: request.args }) }) + '\\n'); + }); + setTimeout(() => process.stdout.write(JSON.stringify({ ready: true, pid: process.pid }) + '\\n'), 100); + } else if (command === 'currency') { + fs.appendFileSync(${JSON.stringify(actionsFile)}, 'a'); + process.stdout.write('currency updated'); + } else { + fs.appendFileSync(${JSON.stringify(oneShotsFile)}, 'o'); + process.stdout.write(JSON.stringify({ via: 'spawn', command })); + }`, + ) + return { startsFile, heavyFile, oneShotsFile, actionsFile, serveEnvFile } +} + /** Writes the repo CLI under this test's isolated dev-root override. */ function fakeDevRepoCli(): string { const repoRoot = join(dir, 'dev-repo') @@ -39,6 +94,7 @@ beforeEach(() => { }) afterEach(() => { + killAll() if (originalBin === undefined) delete process.env.CODEBURN_BIN else process.env.CODEBURN_BIN = originalBin if (originalPathDirs === undefined) delete process.env.CODEBURN_PATH_DIRS @@ -357,19 +413,19 @@ describe('spawnCli coalescing (read-only)', () => { expect(readFileSync(countFile, 'utf8')).toBe('x') // exactly one spawn }) - it('spawns again once the 5s result cache has expired', async () => { - vi.useFakeTimers({ toFake: ['Date'] }) - try { - const countFile = join(dir, 'spawns') - fakeBin('counter-ttl.js', `require('fs').appendFileSync(${JSON.stringify(countFile)},'x'); process.stdout.write(JSON.stringify({ok:1}))`) - vi.setSystemTime(0) - await spawnCli(['status']) - vi.setSystemTime(6_000) - await spawnCli(['status']) - expect(readFileSync(countFile, 'utf8')).toBe('xx') // cache expired → new spawn - } finally { - vi.useRealTimers() - } + it('reflects an external config change on the next same-argv read', async () => { + const configFile = join(dir, 'external-config') + const countFile = join(dir, 'spawns') + writeFileSync(configFile, 'before') + fakeBin( + 'external-config.js', + `const fs = require('node:fs'); fs.appendFileSync(${JSON.stringify(countFile)}, 'x'); process.stdout.write(JSON.stringify({ value: fs.readFileSync(${JSON.stringify(configFile)}, 'utf8') }))`, + ) + + await expect(spawnCli(['model-alias', '--list'])).resolves.toEqual({ value: 'before' }) + writeFileSync(configFile, 'after') + await expect(spawnCli(['model-alias', '--list'])).resolves.toEqual({ value: 'after' }) + expect(readFileSync(countFile, 'utf8')).toBe('xx') }) it('never coalesces config-mutating action calls', async () => { @@ -379,14 +435,433 @@ describe('spawnCli coalescing (read-only)', () => { expect(readFileSync(countFile, 'utf8')).toBe('xx') // two independent spawns }) - it('flushes the read cache when an action completes, so post-action refetches are fresh', async () => { + it('runs a fresh read after a config-mutating action', async () => { const countFile = join(dir, 'spawns') fakeBin('mixed.js', `require('fs').appendFileSync(${JSON.stringify(countFile)},'x'); process.stdout.write(JSON.stringify({ok:1}))`) - await spawnCli(['model-alias', '--list']) // primes the 5s cache - await spawnCliAction(['model-alias', 'a', 'b']) // config change → cache flush - await spawnCli(['model-alias', '--list']) // must NOT serve the pre-action cache + await spawnCli(['model-alias', '--list']) + await spawnCliAction(['model-alias', 'a', 'b']) + await spawnCli(['model-alias', '--list']) expect(readFileSync(countFile, 'utf8')).toBe('xxx') }) + + it('fences old in-flight reads across a mutation without deleting the new flight', async () => { + const configFile = join(dir, 'generation-config') + const startsFile = join(dir, 'generation-read-starts') + const releaseDir = join(dir, 'generation-release') + mkdirSync(releaseDir) + writeFileSync(configFile, 'old') + fakeBin( + 'generation-fence.js', + `const fs = require('node:fs'); const path = require('node:path'); + if (process.argv[3] === '--list') { + const value = fs.readFileSync(${JSON.stringify(configFile)}, 'utf8'); + fs.appendFileSync(${JSON.stringify(startsFile)}, 'r'); + const generation = fs.readFileSync(${JSON.stringify(startsFile)}, 'utf8').length; + const release = path.join(${JSON.stringify(releaseDir)}, String(generation)); + const timer = setInterval(() => { + if (!fs.existsSync(release)) return; + clearInterval(timer); + process.stdout.write(JSON.stringify({ value, generation })); + }, 5); + } else { + fs.writeFileSync(${JSON.stringify(configFile)}, 'new'); + process.stdout.write('updated'); + }`, + ) + + const oldRead = spawnCli(['model-alias', '--list']) + await waitFor(() => readMaybe(startsFile) === 'r') + await expect(spawnCliAction(['model-alias', 'alias', 'model'])) + .resolves.toMatchObject({ ok: true }) + + const newRead = spawnCli(['model-alias', '--list']) + await waitFor(() => readMaybe(startsFile) === 'rr') + writeFileSync(join(releaseDir, '1'), '') + await expect(oldRead).resolves.toEqual({ value: 'old', generation: 1 }) + + // Settling the superseded flight must not remove the current generation's + // entry: this identical call still shares read #2 instead of spawning #3. + const coalescedNewRead = spawnCli(['model-alias', '--list']) + await new Promise(resolve => setTimeout(resolve, 100)) + expect(readMaybe(startsFile)).toBe('rr') + + writeFileSync(join(releaseDir, '2'), '') + await expect(Promise.all([newRead, coalescedNewRead])).resolves.toEqual([ + { value: 'new', generation: 2 }, + { value: 'new', generation: 2 }, + ]) + }) +}) + +describe('resident serve single-flight', () => { + it('startServe is idempotent and creates only one resident child', async () => { + const files = fakeResidentBin() + startServe() + startServe() + + const result = await spawnCli(['status', '--double-start'], { timeoutMs: 5_000 }) as { generation: number } + + expect(result.generation).toBe(1) + expect(readMaybe(files.startsFile)).toBe('s') + expect(readMaybe(files.heavyFile)).toBe('h') + }) + + it('lazily starts a new resident after an unexpected death and one-shot fallback', async () => { + const startsFile = join(dir, 'serve-starts') + const oneShotsFile = join(dir, 'one-shot-reads') + fakeBin( + 'dies-once-resident.js', + `const fs = require('node:fs'); const readline = require('node:readline'); + const command = process.argv[2]; + if (command === 'serve') { + fs.appendFileSync(${JSON.stringify(startsFile)}, 's'); + const generation = fs.readFileSync(${JSON.stringify(startsFile)}, 'utf8').length; + const rl = readline.createInterface({ input: process.stdin }); + rl.once('line', line => { + const request = JSON.parse(line); + if (generation === 1) process.exit(1); + process.stdout.write(JSON.stringify({ id: request.id, ok: true, output: JSON.stringify({ via: 'serve', generation }) }) + '\\n'); + }); + } else { + fs.appendFileSync(${JSON.stringify(oneShotsFile)}, 'o'); + process.stdout.write(JSON.stringify({ via: 'spawn' })); + }`, + ) + startServe() + + await expect(spawnCli(['status', '--first'], { timeoutMs: 5_000 })) + .resolves.toEqual({ via: 'spawn' }) + await expect(spawnCli(['models', '--second'], { timeoutMs: 5_000 })) + .resolves.toEqual({ via: 'serve', generation: 2 }) + + expect(readMaybe(startsFile)).toBe('ss') + expect(readMaybe(oneShotsFile)).toBe('o') + }) + + it('gives the first resident status request the power-user cold timeout floor', async () => { + fakeBin( + 'slow-cold-resident.js', + `const readline = require('node:readline'); + if (process.argv[2] === 'serve') { + const rl = readline.createInterface({ input: process.stdin }); + rl.on('line', line => { + const request = JSON.parse(line); + setTimeout(() => process.stdout.write(JSON.stringify({ id: request.id, ok: true, output: JSON.stringify({ via: 'serve' }) }) + '\\n'), 80); + }); + } else { + process.stdout.write(JSON.stringify({ via: 'spawn' })); + }`, + ) + startServe() + + await expect(spawnCli(['status', '--cold-floor'], { timeoutMs: 20 })) + .resolves.toEqual({ via: 'serve' }) + }) + + it('starts a queued resident timeout only after the request ahead settles', async () => { + fakeBin( + 'serial-resident.js', + `const readline = require('node:readline'); + if (process.argv[2] === 'serve') { + const rl = readline.createInterface({ input: process.stdin }); + (async () => { + for await (const line of rl) { + const request = JSON.parse(line); + if (request.args.includes('--slow')) await new Promise(resolve => setTimeout(resolve, 400)); + process.stdout.write(JSON.stringify({ id: request.id, ok: true, output: JSON.stringify({ via: 'serve', args: request.args }) }) + '\\n'); + } + })(); + } else { + process.stdout.write(JSON.stringify({ via: 'spawn' })); + }`, + ) + startServe() + await expect(spawnCli(['status', '--warm'], { timeoutMs: 5_000 })) + .resolves.toMatchObject({ via: 'serve' }) + + const slow = spawnCli(['sessions', '--slow'], { timeoutMs: 1_000 }) + const queued = spawnCli(['models', '--queued'], { timeoutMs: 200 }) + const [slowResult, queuedResult] = await Promise.all([slow, queued]) + + expect(slowResult).toMatchObject({ via: 'serve' }) + expect(queuedResult).toMatchObject({ via: 'serve' }) + }) + + it('uses the first real request as the only heavy execution, even before ready', async () => { + const files = fakeResidentBin() + startServe() + + const result = await spawnCli(['status', '--format', 'menubar-json'], { + timeoutMs: 5_000, + extraEnv: { CODEBURN_PROGRESS: '1' }, + }) as { via: string; generation: number } + + expect(result).toMatchObject({ via: 'serve', generation: 1 }) + expect(readMaybe(files.startsFile)).toBe('s') + expect(readMaybe(files.heavyFile)).toBe('h') + expect(readMaybe(files.oneShotsFile)).toBe('') + expect(readMaybe(files.serveEnvFile)).toBe('1') + }) + + it('forwards serve progress frames through the read onStderr callback', async () => { + fakeResidentBin() + startServe() + const chunks: string[] = [] + + await spawnCli(['status'], { + timeoutMs: 5_000, + extraEnv: { CODEBURN_PROGRESS: '1' }, + onStderr: chunk => { chunks.push(chunk) }, + }) + + expect(chunks.join('')).toBe('CODEBURN_PROGRESS {"kind":"provider","provider":"claude","state":"start","generation":1}\n') + }) + + it('rejects and terminates a resident that emits an oversized valid JSON frame', async () => { + const startsFile = join(dir, 'oversized-frame-starts') + const oneShotsFile = join(dir, 'oversized-frame-one-shots') + fakeBin( + 'oversized-frame-resident.js', + `const fs = require('node:fs'); const readline = require('node:readline'); + if (process.argv[2] === 'serve') { + fs.appendFileSync(${JSON.stringify(startsFile)}, 's'); + const generation = fs.readFileSync(${JSON.stringify(startsFile)}, 'utf8').length; + const rl = readline.createInterface({ input: process.stdin }); + rl.once('line', line => { + const request = JSON.parse(line); + const output = generation === 1 + ? JSON.stringify({ value: 'x'.repeat(16 * 1024 * 1024 + 1024) }) + : JSON.stringify({ generation }); + process.stdout.write(JSON.stringify({ id: request.id, ok: true, output }) + '\\n'); + }); + setInterval(() => {}, 1000); + } else { + fs.appendFileSync(${JSON.stringify(oneShotsFile)}, 'o'); + process.stdout.write('{}'); + }`, + ) + startServe() + + await expect(spawnCli(['status', '--oversized-frame'], { timeoutMs: 5_000 })) + .rejects.toMatchObject({ kind: 'too-large' } satisfies Partial) + await expect(spawnCli(['status', '--after-oversized-frame'], { timeoutMs: 5_000 })) + .resolves.toEqual({ generation: 2 }) + expect(readMaybe(startsFile)).toBe('ss') + expect(readMaybe(oneShotsFile)).toBe('') + }) + + it('keeps serve enabled after more overflows than the resident death budget', async () => { + const startsFile = join(dir, 'overflow-budget-starts') + const oneShotsFile = join(dir, 'overflow-budget-one-shots') + fakeBin( + 'always-oversized-resident.js', + `const fs = require('node:fs'); const readline = require('node:readline'); + if (process.argv[2] === 'serve') { + fs.appendFileSync(${JSON.stringify(startsFile)}, 's'); + const rl = readline.createInterface({ input: process.stdin }); + rl.once('line', line => { + const request = JSON.parse(line); + const output = JSON.stringify({ value: 'x'.repeat(16 * 1024 * 1024 + 1024) }); + process.stdout.write(JSON.stringify({ id: request.id, ok: true, output }) + '\\n'); + }); + setInterval(() => {}, 1000); + } else { + fs.appendFileSync(${JSON.stringify(oneShotsFile)}, 'o'); + process.stdout.write('{}'); + }`, + ) + startServe() + + for (let attempt = 0; attempt < 4; attempt += 1) { + await expect(spawnCli(['status', '--overflow', String(attempt)], { timeoutMs: 5_000 })) + .rejects.toMatchObject({ kind: 'too-large' } satisfies Partial) + } + + // An overflow kill is deliberate, so it never spends the unexpected-death + // budget: the fourth request still reaches a resident, not a one-shot. + expect(readMaybe(startsFile)).toBe('ssss') + expect(readMaybe(oneShotsFile)).toBe('') + }) + + it('rejects and terminates a resident whose protocol line never terminates', async () => { + const startsFile = join(dir, 'unterminated-line-starts') + const oneShotsFile = join(dir, 'unterminated-line-one-shots') + fakeBin( + 'unterminated-line-resident.js', + `const fs = require('node:fs'); const readline = require('node:readline'); + if (process.argv[2] === 'serve') { + fs.appendFileSync(${JSON.stringify(startsFile)}, 's'); + const rl = readline.createInterface({ input: process.stdin }); + rl.once('line', () => process.stdout.write('x'.repeat(16 * 1024 * 1024 + 1024))); + setInterval(() => {}, 1000); + } else { + fs.appendFileSync(${JSON.stringify(oneShotsFile)}, 'o'); + process.stdout.write('{}'); + }`, + ) + startServe() + + await expect(spawnCli(['status', '--unterminated-line'], { timeoutMs: 5_000 })) + .rejects.toMatchObject({ kind: 'too-large' } satisfies Partial) + expect(readMaybe(startsFile)).toBe('s') + expect(readMaybe(oneShotsFile)).toBe('') + }) + + it('keeps requests with any non-progress env override on the one-shot path', async () => { + const files = fakeResidentBin() + startServe() + + const result = await spawnCli(['status'], { + timeoutMs: 5_000, + extraEnv: { CODEBURN_PROGRESS: '1', CODEBURN_TEST_MODE: 'isolated' }, + }) as { via: string } + + expect(result.via).toBe('spawn') + expect(readMaybe(files.heavyFile)).toBe('') + expect(readMaybe(files.oneShotsFile)).toBe('o') + }) + + it('treats empty and undefined-only env overrides as serve-compatible', async () => { + const files = fakeResidentBin() + startServe() + + const empty = await spawnCli(['status', '--empty-env'], { + timeoutMs: 5_000, + extraEnv: {}, + }) as { via: string } + const undefinedOnly = await spawnCli(['models', '--undefined-env'], { + timeoutMs: 5_000, + extraEnv: { CODEBURN_PROGRESS: undefined }, + }) as { via: string } + + expect(empty.via).toBe('serve') + expect(undefinedOnly.via).toBe('serve') + expect(readMaybe(files.heavyFile)).toBe('hh') + expect(readMaybe(files.oneShotsFile)).toBe('') + }) + + it('restarts the resident child after a successful config mutation', async () => { + const files = fakeResidentBin() + startServe() + + const before = await spawnCli(['status'], { timeoutMs: 5_000 }) as { generation: number } + const action = await spawnCliAction(['currency', 'EUR'], { timeoutMs: 5_000 }) + const after = await spawnCli(['status'], { timeoutMs: 5_000 }) as { generation: number } + + expect(action).toMatchObject({ ok: true, stdout: 'currency updated', code: 0 }) + expect(before.generation).toBe(1) + expect(after.generation).toBe(2) + expect(readMaybe(files.startsFile)).toBe('ss') + expect(readMaybe(files.heavyFile)).toBe('hh') + expect(readMaybe(files.actionsFile)).toBe('a') + }) + + it('preserves the unexpected-death budget across mutation restarts', async () => { + const startsFile = join(dir, 'serve-starts') + const oneShotsFile = join(dir, 'one-shot-reads') + fakeBin( + 'crashing-resident.js', + `const fs = require('node:fs'); const readline = require('node:readline'); + const command = process.argv[2]; + if (command === 'serve') { + fs.appendFileSync(${JSON.stringify(startsFile)}, 's'); + const rl = readline.createInterface({ input: process.stdin }); + rl.once('line', () => process.exit(1)); + } else if (command === 'currency') { + process.stdout.write('currency updated'); + } else { + fs.appendFileSync(${JSON.stringify(oneShotsFile)}, 'o'); + process.stdout.write(JSON.stringify({ via: 'spawn' })); + }`, + ) + startServe() + + for (let attempt = 0; attempt < 3; attempt += 1) { + await expect(spawnCli(['status', '--attempt', String(attempt)], { timeoutMs: 5_000 })) + .resolves.toEqual({ via: 'spawn' }) + await expect(spawnCliAction(['currency', attempt % 2 === 0 ? 'EUR' : 'USD'], { timeoutMs: 5_000 })) + .resolves.toMatchObject({ ok: true }) + } + + // A mutation may replace a healthy child, but it must not erase real crash + // history and resurrect serve after the third unexpected death. + expect(readMaybe(startsFile)).toBe('sss') + await expect(spawnCli(['status', '--after-budget'], { timeoutMs: 5_000 })) + .resolves.toEqual({ via: 'spawn' }) + expect(readMaybe(startsFile)).toBe('sss') + expect(readMaybe(oneShotsFile)).toBe('oooo') + }) + + it('stops lazy crash recovery after three consecutive resident deaths', async () => { + const startsFile = join(dir, 'serve-starts') + const oneShotsFile = join(dir, 'one-shot-reads') + fakeBin( + 'always-crashing-resident.js', + `const fs = require('node:fs'); const readline = require('node:readline'); + if (process.argv[2] === 'serve') { + fs.appendFileSync(${JSON.stringify(startsFile)}, 's'); + const rl = readline.createInterface({ input: process.stdin }); + rl.once('line', () => process.exit(1)); + } else { + fs.appendFileSync(${JSON.stringify(oneShotsFile)}, 'o'); + process.stdout.write(JSON.stringify({ via: 'spawn' })); + }`, + ) + startServe() + + for (let attempt = 0; attempt < 4; attempt += 1) { + await expect(spawnCli(['status', '--lazy-crash', String(attempt)], { timeoutMs: 5_000 })) + .resolves.toEqual({ via: 'spawn' }) + } + + expect(readMaybe(startsFile)).toBe('sss') + expect(readMaybe(oneShotsFile)).toBe('oooo') + }) + + it('does not spawn a one-shot fallback after killAll destroys serve', async () => { + const requestSeenFile = join(dir, 'request-seen') + const oneShotsFile = join(dir, 'one-shot-reads') + fakeBin( + 'shutdown-resident.js', + `const fs = require('node:fs'); const readline = require('node:readline'); + if (process.argv[2] === 'serve') { + const rl = readline.createInterface({ input: process.stdin }); + rl.once('line', () => { fs.writeFileSync(${JSON.stringify(requestSeenFile)}, '1'); }); + } else { + fs.appendFileSync(${JSON.stringify(oneShotsFile)}, 'o'); + process.stdout.write('{}'); + }`, + ) + startServe() + const pending = spawnCli(['status', '--shutdown'], { timeoutMs: 60_000 }) + for (let attempt = 0; attempt < 400 && !readMaybe(requestSeenFile); attempt += 1) { + await new Promise(resolve => setTimeout(resolve, 5)) + } + const requestSeen = readMaybe(requestSeenFile) + killAll() + + expect(requestSeen).toBe('1') + await expect(pending).rejects.toMatchObject({ kind: 'nonzero' }) + await new Promise(resolve => setTimeout(resolve, 25)) + expect(readMaybe(oneShotsFile)).toBe('') + }) + + it('keeps the warm resident child after a successful export', async () => { + const files = fakeResidentBin() + startServe() + + const before = await spawnCli(['status'], { timeoutMs: 5_000 }) as { generation: number } + const action = await spawnCliAction(['export', '-f', 'json', '-o', join(dir, 'usage.json')], { timeoutMs: 5_000 }) + // A different panel query proves which resident generation handled the + // next served read without relying on same-request coalescing. + const after = await spawnCli(['models', '--format', 'json'], { timeoutMs: 5_000 }) as { generation: number } + + expect(action.ok).toBe(true) + expect(before.generation).toBe(1) + expect(after.generation).toBe(1) + expect(readMaybe(files.startsFile)).toBe('s') + expect(readMaybe(files.heavyFile)).toBe('hh') + }) }) describe('killAll', () => { @@ -398,6 +873,43 @@ describe('killAll', () => { killAll() await expect(pending).rejects.toMatchObject({ kind: 'nonzero' }) }) + + it('terminal shutdown rejects new read and action races without spawning', async () => { + const startsFile = join(dir, 'starts') + fakeBin( + 'shutdown-guard.js', + `require('node:fs').appendFileSync(${JSON.stringify(startsFile)}, 'x'); process.stdout.write('{}')`, + ) + + shutdownAll() + startServe() + + await expect(spawnCli(['status', '--after-shutdown'])) + .rejects.toMatchObject({ kind: 'nonzero' }) + await expect(spawnCliAction(['currency', 'EUR'])) + .resolves.toMatchObject({ ok: false, code: null }) + expect(readMaybe(startsFile)).toBe('') + }) + + it('terminal shutdown cancels read and action slots admitted before their spawn microtask', async () => { + const startsFile = join(dir, 'starts-after-admission') + fakeBin( + 'shutdown-after-admission.js', + `require('node:fs').appendFileSync(${JSON.stringify(startsFile)}, process.argv[2] + '\\n'); if (process.argv[2] === 'status') process.stdout.write('{}'); else process.stdout.write('updated')`, + ) + + // Both calls synchronously acquire the two free scheduler slots. Their + // actual spawn resumes in a microtask, which is exactly the before-quit race. + const read = spawnCli(['status', '--admitted']) + const action = spawnCliAction(['currency', 'EUR']) + shutdownAll() + + await Promise.all([ + expect(read).rejects.toMatchObject({ kind: 'nonzero' }), + expect(action).resolves.toMatchObject({ ok: false, code: null }), + ]) + expect(readMaybe(startsFile)).toBe('') + }) }) describe('spawnCli concurrency scheduler', () => { @@ -525,6 +1037,52 @@ describe('spawnCli concurrency scheduler', () => { await delay(50) expect(startedList(startedFile)).not.toContain('sessions') // never spawned }) + + it('limits six simultaneous resident-failure fallbacks to two one-shot children', async () => { + const startedFile = join(dir, 'fallback-started') + const activeDir = join(dir, 'fallback-active'); mkdirSync(activeDir) + const activeCountsFile = join(dir, 'fallback-active-counts') + const releaseDir = join(dir, 'fallback-release'); mkdirSync(releaseDir) + fakeBin( + 'failing-resident-with-blocked-fallbacks.js', + `const fs = require('node:fs'); const path = require('node:path'); const readline = require('node:readline'); + if (process.argv[2] === 'serve') { + const rl = readline.createInterface({ input: process.stdin }); + rl.on('line', line => { + const request = JSON.parse(line); + process.stdout.write(JSON.stringify({ id: request.id, ok: false, error: 'resident failed' }) + '\\n'); + }); + } else { + const id = process.argv[3]; + fs.appendFileSync(${JSON.stringify(startedFile)}, id + '\\n'); + const activeFile = path.join(${JSON.stringify(activeDir)}, String(process.pid)); + fs.writeFileSync(activeFile, ''); + fs.appendFileSync(${JSON.stringify(activeCountsFile)}, fs.readdirSync(${JSON.stringify(activeDir)}).length + '\\n'); + const releaseFile = path.join(${JSON.stringify(releaseDir)}, id); + const timer = setInterval(() => { + if (!fs.existsSync(releaseFile)) return; + clearInterval(timer); + fs.unlinkSync(activeFile); + process.stdout.write(JSON.stringify({ via: 'spawn', id })); + }, 5); + }`, + ) + startServe() + + const requests = Array.from({ length: 6 }, (_, index) => + spawnCli(['status', `fallback-${index}`], { timeoutMs: 5_000 }), + ) + await waitUntil(() => startedList(startedFile).length >= 2) + await delay(150) + const admittedBeforeRelease = startedList(startedFile) + + for (let index = 0; index < 6; index += 1) release(releaseDir, `fallback-${index}`) + await Promise.all(requests) + + const activeCounts = startedList(activeCountsFile).map(Number) + expect(admittedBeforeRelease).toHaveLength(2) + expect(Math.max(...activeCounts)).toBeLessThanOrEqual(2) + }) }) describe('spawnCliAction', () => { diff --git a/app/electron/cli.ts b/app/electron/cli.ts index ad63a8ec..2d449a43 100644 --- a/app/electron/cli.ts +++ b/app/electron/cli.ts @@ -54,11 +54,12 @@ export class CliError extends Error { } const DEFAULT_TIMEOUT_MS = 45_000 +// The first status query may hydrate a power-user cache from scratch. Every +// resident request admitted before that succeeds shares this floor so a later +// short request cannot kill the child while it waits behind the cold scan. +export const DESKTOP_COLD_TIMEOUT_MS = 10 * 60_000 // A runaway CLI (or a compromised binary) must not exhaust main-process memory. const MAX_OUTPUT_BYTES = 16 * 1024 * 1024 -// Same-cadence pollers fire near-identical read spawns; share one child and hold -// its result briefly so six overview hooks don't launch six processes at once. -const COALESCE_TTL_MS = 5_000 // A cold-cache CLI spawn costs seconds at ~120% CPU; letting every poll + // prefetch launch at once saturates the machine. Cap how many children run // concurrently — the rest queue and drain as slots free (interactive first). @@ -67,7 +68,10 @@ const MAX_CONCURRENT_CLI = 2 // Every live child so `before-quit` can reap them (Electron does not on macOS). const activeChildren = new Set() const readInflight = new Map>() -const readCache = new Map() +// Successful mutations advance the epoch before their promise resolves. A read +// begun against older config may still settle for its original caller, but can +// never be reused by the post-mutation refetch or delete that newer flight. +let readGeneration = 0 // Concurrency scheduler. `running` counts spawned (not queued) children; waiters // hold the slot-grant resolver for a queued spawn. Two queues so interactive @@ -76,6 +80,7 @@ type SlotWaiter = { resolve: () => void; reject: (err: unknown) => void } let running = 0 const interactiveQueue: SlotWaiter[] = [] const backgroundQueue: SlotWaiter[] = [] +let shuttingDown = false /** Grant free slots to queued waiters, interactive first, up to the cap. */ function pumpSlots(): void { @@ -101,9 +106,8 @@ function releaseSlot(): void { pumpSlots() } -/** SIGKILL every in-flight child and cancel anything still queued for a slot. - * Wired to Electron's `before-quit`. */ -export function killAll(): void { +/** Reap every child and cancel anything still queued for a slot. */ +function reapAll(): void { serveClient?.destroy() serveClient = null for (const child of activeChildren) child.kill('SIGKILL') @@ -117,6 +121,19 @@ export function killAll(): void { for (const waiter of waiting) waiter.reject(new CliError('nonzero', 'codeburn cancelled')) } +/** Test/dev cleanup that permits a later fresh start in this same process. */ +export function killAll(): void { + shuttingDown = false + reapAll() +} + +/** Terminal app shutdown: reap current work and reject any IPC race that arrives + * while Electron is still flushing telemetry before the final quit pass. */ +export function shutdownAll(): void { + shuttingDown = true + reapAll() +} + // Homebrew + common Node version managers, mirroring mac/CodeburnCLI.swift so a // GUI-launched app (minimal PATH) still finds a globally-installed `codeburn`. export function nodeManagerDirs(): string[] { @@ -378,6 +395,25 @@ function runCli(spec: SpawnSpec, cmdLabel: string, timeoutMs: number, onStderr?: }) } +/** Run a one-shot read under the global child cap. A slot grant resumes on a + * microtask, so terminal shutdown must be checked again immediately before the + * synchronous spawn call. */ +async function runScheduledCli( + spec: SpawnSpec, + cmdLabel: string, + timeoutMs: number, + priority: SpawnPriority, + onStderr?: (chunk: string) => void, +): Promise { + await acquireSlot(priority) + try { + if (shuttingDown) throw new CliError('nonzero', 'codeburn is shutting down') + return await runCli(spec, cmdLabel, timeoutMs, onStderr) + } finally { + releaseSlot() + } +} + /** * Spawn `codeburn ` with plain argv (never a shell), collect stdout, and * decode it as JSON. Rejects with a structured {@link CliError}: @@ -387,8 +423,9 @@ function runCli(spec: SpawnSpec, cmdLabel: string, timeoutMs: number, onStderr?: * timeout the process was killed after `timeoutMs` * too-large stdout+stderr exceeded {@link MAX_OUTPUT_BYTES} * - * Read-only, so concurrent identical calls share one child and a 5s result cache - * absorbs same-cadence pollers. Never use this for config-mutating commands. + * Read-only, so concurrent identical calls share one child. Settled results are + * never cached here because config can also change outside the desktop app. + * Never use this for config-mutating commands. */ // ── Resident serve child ──────────────────────────────────────────────── // The heavy read queries (one per panel) each pay seconds of CLI startup on @@ -397,75 +434,141 @@ function runCli(spec: SpawnSpec, cmdLabel: string, timeoutMs: number, onStderr?: // stdio and the cache stays parsed in the child. Routing rules keep this // strictly an optimization: // - only SERVE_ROUTED commands (the app's JSON panel queries) are eligible; -// - requests route through serve only once the child is READY AND WARM, so -// the cold-start path keeps its spawn (with its stderr progress events); +// - the first real panel request is also the cache warm-up, so startup never +// runs an artificial warm-up query beside a duplicate one-shot child; +// - progress frames from serve are forwarded through the same onStderr hook +// used by a one-shot cold start; // - any serve failure falls back to a normal spawn for that call; // - three child deaths permanently disable serve for this app run. const SERVE_ROUTED = new Set(['status', 'models', 'sessions', 'compare', 'yield', 'spend', 'optimize', 'audit']) -const SERVE_REQUEST_TIMEOUT_MS = 60_000 const SERVE_MAX_RESTARTS = 3 class ServeClient { private child: ReturnType | null = null - private pending = new Map void; reject: (e: Error) => void; timer: NodeJS.Timeout }>() + private pending = new Map void + reject: (e: Error) => void + timer: NodeJS.Timeout + warmsServe: boolean + decodedBytes: number + onStderr?: (chunk: string) => void + }>() private nextId = 1 - private ready = false - private warm = false private deaths = 0 private buffer = '' + private bufferBytes = 0 + private warmed = false + private destroyed = false + private requestTail: Promise = Promise.resolve() constructor(private readonly spec: SpawnSpec) {} - isWarmAndReady(): boolean { return this.ready && this.warm && this.child !== null } + isRunning(): boolean { return this.child !== null } disabled(): boolean { return this.deaths >= SERVE_MAX_RESTARTS } + isDestroyed(): boolean { return this.destroyed } start(): void { - if (this.child || this.disabled()) return + if (this.child || this.disabled() || this.destroyed) return const child = spawn(this.spec.bin, [...this.spec.args], { shell: false, stdio: ['pipe', 'pipe', 'ignore'], env: this.spec.env }) this.child = child child.stdout!.setEncoding('utf8') - child.stdout!.on('data', (chunk: string) => this.onData(chunk)) - const onGone = () => this.onDeath() + child.stdout!.on('data', (chunk: string) => { + // A replaced child's stream can drain after its exit callback. Never let + // those stale bytes repopulate the shared line buffer for the new child. + if (this.child === child) this.onData(child, chunk) + }) + const onGone = () => this.onDeath(child) child.on('exit', onGone) child.on('error', onGone) - // Background warm-up: one cheap query makes the child parse the session - // cache once; every later panel fetch reuses the in-memory copy. - void this.request(['status', '--format', 'menubar-json', '--period', 'today'], SERVE_REQUEST_TIMEOUT_MS) - .then(() => { this.warm = true }) - .catch(() => { /* warm-up failure just leaves routing on the spawn path */ }) } - private onData(chunk: string): void { + private onData(child: ReturnType, chunk: string): void { this.buffer += chunk + this.bufferBytes += Buffer.byteLength(chunk) let idx: number while ((idx = this.buffer.indexOf('\n')) >= 0) { - const line = this.buffer.slice(0, idx).trim() + const rawLine = this.buffer.slice(0, idx) this.buffer = this.buffer.slice(idx + 1) + const rawLineBytes = Buffer.byteLength(rawLine) + this.bufferBytes = Math.max(0, this.bufferBytes - rawLineBytes - 1) + if (rawLineBytes > MAX_OUTPUT_BYTES) { + this.terminateForOverflow(child) + return + } + const line = rawLine.trim() if (!line) continue - let msg: { id?: number; ready?: boolean; ok?: boolean; refused?: boolean; output?: string; error?: string } + let msg: { id?: number; ready?: boolean; progress?: string; ok?: boolean; refused?: boolean; output?: string; error?: string } try { msg = JSON.parse(line) } catch { continue } - if (msg.ready) { this.ready = true; continue } + if (msg.ready) continue if (typeof msg.id !== 'number') continue const waiter = this.pending.get(msg.id) if (!waiter) continue + if (typeof msg.progress === 'string') { + if (!this.consumeDecodedOutput(child, waiter, msg.progress)) return + if (waiter.onStderr) { + try { waiter.onStderr(msg.progress) } catch { /* progress consumers never own the request */ } + } + continue + } + const terminalOutput = typeof msg.output === 'string' ? msg.output : typeof msg.error === 'string' ? msg.error : '' + if (!this.consumeDecodedOutput(child, waiter, terminalOutput)) return this.pending.delete(msg.id) clearTimeout(waiter.timer) if (msg.ok && typeof msg.output === 'string') { + if (waiter.warmsServe) this.warmed = true try { waiter.resolve(JSON.parse(msg.output)) } catch { waiter.reject(new CliError('bad-json', 'codeburn produced output that was not valid JSON')) } } else { waiter.reject(new CliError('nonzero', msg.error ?? 'serve request failed')) } } + // Complete lines are bounded above before parsing. Bound the partial frame + // too, otherwise a child that never emits '\n' can grow this buffer forever. + if (this.bufferBytes > MAX_OUTPUT_BYTES) this.terminateForOverflow(child) } - private onDeath(): void { - const child = this.child + private consumeDecodedOutput( + child: ReturnType, + waiter: { decodedBytes: number }, + output: string, + ): boolean { + waiter.decodedBytes += Buffer.byteLength(output) + if (waiter.decodedBytes <= MAX_OUTPUT_BYTES) return true + this.terminateForOverflow(child) + return false + } + + private terminateForOverflow(child: ReturnType): void { + if (this.child !== child) return + const error = new CliError('too-large', `codeburn serve produced more than ${MAX_OUTPUT_BYTES} bytes`) + // Detach synchronously before SIGKILL. A new request may start the next + // generation immediately; the old child's eventual exit must not reject it. this.child = null - this.ready = false - this.warm = false - this.deaths += 1 - if (child) activeChildren.delete(child as never) + this.buffer = '' + this.bufferBytes = 0 + this.warmed = false + // Deliberate termination, not a crash: it must not spend the unexpected-death + // budget, or three oversized payloads would disable serve for the app run. + activeChildren.delete(child as never) + for (const [, waiter] of this.pending) { + clearTimeout(waiter.timer) + waiter.reject(error) + } + this.pending.clear() + child.kill('SIGKILL') + } + + private onDeath(child: ReturnType, countsTowardBudget = true): void { + // Both `error` and `exit` can fire for one child, and destroy() performs the + // same cleanup synchronously. Only the currently-owned child may transition + // this client or reject its pending requests. + if (this.child !== child) return + this.child = null + this.buffer = '' + this.bufferBytes = 0 + this.warmed = false + if (countsTowardBudget) this.deaths += 1 + activeChildren.delete(child as never) for (const [, waiter] of this.pending) { clearTimeout(waiter.timer) waiter.reject(new CliError('nonzero', 'codeburn serve exited')) @@ -473,10 +576,32 @@ class ServeClient { this.pending.clear() } - request(args: string[], timeoutMs: number): Promise { + restartAfterMutation(): void { + const child = this.child + if (child) { + // This is an intentional replacement, not a crash. Detach first so the + // later exit event cannot consume the unexpected-death budget. + this.onDeath(child, false) + child.kill('SIGKILL') + } + this.start() + } + + request(args: string[], timeoutMs: number, onStderr?: (chunk: string) => void): Promise { + // The stdio server is deliberately serial. Mirror that contract client-side + // so queued calls do not start their timers while a cold request is still + // hydrating the cache in front of them. + const run = () => this.requestNow(args, timeoutMs, onStderr) + const result = this.requestTail.then(run, run) + this.requestTail = result.then(() => undefined, () => undefined) + return result + } + + private requestNow(args: string[], timeoutMs: number, onStderr?: (chunk: string) => void): Promise { const child = this.child if (!child?.stdin) return Promise.reject(new CliError('nonzero', 'serve not running')) const id = this.nextId++ + const effectiveTimeoutMs = this.warmed ? timeoutMs : Math.max(timeoutMs, DESKTOP_COLD_TIMEOUT_MS) return new Promise((resolve, reject) => { const timer = setTimeout(() => { // A hung request would block the serialized queue behind it; kill the @@ -484,8 +609,15 @@ class ServeClient { this.pending.delete(id) reject(new CliError('timeout', 'codeburn serve timed out')) child.kill('SIGKILL') - }, timeoutMs) - this.pending.set(id, { resolve, reject, timer }) + }, effectiveTimeoutMs) + this.pending.set(id, { + resolve, + reject, + timer, + warmsServe: args[0] === 'status', + decodedBytes: 0, + ...(onStderr ? { onStderr } : {}), + }) child.stdin!.write(JSON.stringify({ id, args }) + '\n', (err) => { if (err) { this.pending.delete(id) @@ -497,68 +629,113 @@ class ServeClient { } destroy(): void { + this.destroyed = true this.deaths = SERVE_MAX_RESTARTS - this.child?.kill('SIGKILL') - this.onDeath() + const child = this.child + if (!child) return + this.onDeath(child, false) + child.kill('SIGKILL') } } let serveClient: ServeClient | null = null -/** Start the resident serve child and its warm-up query. Called once from app - * startup (never from the spawn path, so unit tests of the scheduler and the - * cold-start flow are byte-identical without it). Safe to call repeatedly. */ -export function startServeWarmup(): void { +/** Start the resident serve child without issuing a query. The first real panel + * request is accepted immediately (even before the ready frame) and performs + * the one cold-cache hydration while streaming progress back to the splash. */ +export function startServe(): void { + if (shuttingDown) return const target = resolveTarget() if (!target) return if (serveClient?.disabled()) return - if (!serveClient) serveClient = new ServeClient(spawnSpecFor(target, ['serve', '--stdio'])) + if (!serveClient) { + const spec = spawnSpecFor(target, ['serve', '--stdio']) + spec.env = { ...spec.env, CODEBURN_PROGRESS: '1' } + serveClient = new ServeClient(spec) + } serveClient.start() } +function restartServeAfterMutation(): void { + // CLI-only consumers never started serve, so do not create a surprise daemon + // for them. In Electron, replace the resident child immediately so its parser + // and output memos cannot survive a successful config mutation. Reusing the + // client preserves its app-lifetime budget of unexpected child deaths. + if (!serveClient) return + serveClient.restartAfterMutation() +} + +function actionInvalidatesServe(args: string[]): boolean { + // Export only writes the caller-selected artifact. Every other current + // Electron action changes config or device state, and future actions restart + // by default until they are explicitly proven state-preserving. + return args[0] !== 'export' +} + +function isServeCompatibleEnv(extraEnv?: NodeJS.ProcessEnv): boolean { + if (!extraEnv) return true + const entries = Object.entries(extraEnv).filter(([, value]) => value !== undefined) + if (entries.length === 0) return true + return entries.length === 1 && entries[0]![0] === 'CODEBURN_PROGRESS' && entries[0]![1] === '1' +} + export function spawnCli( args: string[], opts: { timeoutMs?: number; onStderr?: (chunk: string) => void; extraEnv?: NodeJS.ProcessEnv; priority?: SpawnPriority } = {}, ): Promise { + if (shuttingDown) return Promise.reject(new CliError('nonzero', 'codeburn is shutting down')) const target = resolveTarget() if (!target) return Promise.reject(new CliError('not-found', 'codeburn CLI not found', notFoundStage())) const spec = spawnSpecFor(target, args) if (opts.extraEnv) spec.env = { ...spec.env, ...opts.extraEnv } - const key = JSON.stringify([spec.bin, ...spec.args]) - const cached = readCache.get(key) - if (cached && Date.now() - cached.at < COALESCE_TTL_MS) return Promise.resolve(cached.value) + const generation = readGeneration + const key = JSON.stringify([generation, spec.bin, ...spec.args]) const existing = readInflight.get(key) // A same-cadence re-poll during a slow cold warmup coalesces onto the one // in-flight child (which already carries onStderr); no second cold parse. - // Coalesce/cache hits settle here, BEFORE queueing, so they never hold a slot. + // Coalesced calls settle here, BEFORE queueing, so they never hold a slot. if (existing) return existing - // Serve fast-path: warm resident child answers the panel query without a - // spawn. The child is started once at app startup (startServeWarmup); until - // it is warm, every call keeps the plain spawn path. - if (SERVE_ROUTED.has(args[0] ?? '') && !opts.extraEnv) { + const priority = opts.priority ?? 'interactive' + + // Serve fast-path: the child is started once at app startup. It accepts the + // first real query before its ready frame, making that request the single + // cache warm-up. CODEBURN_PROGRESS is compatible because startServe sets it + // on the resident child; any other per-call env needs an isolated one-shot. + if (SERVE_ROUTED.has(args[0] ?? '') && isServeCompatibleEnv(opts.extraEnv)) { const serve = serveClient - if (serve?.isWarmAndReady()) { - const flight = serve.request(args, opts.timeoutMs ?? DEFAULT_TIMEOUT_MS) - .catch(() => runCli(spec, args[0] ?? '', opts.timeoutMs ?? DEFAULT_TIMEOUT_MS, opts.onStderr)) - .then(value => { readCache.set(key, { at: Date.now(), value }); return value }) + // Recover lazily from an unexpected child death. start() is synchronous and + // idempotent, and the client's lifetime death budget prevents an endlessly + // crashing binary from being respawned on every poll. + if (serve && !serve.isRunning() && !serve.disabled()) serve.start() + if (serve?.isRunning()) { + const flight = serve.request(args, opts.timeoutMs ?? DEFAULT_TIMEOUT_MS, opts.onStderr) + .catch(err => { + // App shutdown is terminal: never turn rejected resident requests + // into brand-new one-shot children after killAll() has reaped them. + if (serve.isDestroyed() || (err instanceof CliError && err.kind === 'too-large')) throw err + return runScheduledCli( + spec, + args[0] ?? '', + opts.timeoutMs ?? DEFAULT_TIMEOUT_MS, + priority, + opts.onStderr, + ) + }) .finally(() => { readInflight.delete(key) }) readInflight.set(key, flight) return flight } } - const priority = opts.priority ?? 'interactive' - const flight = (async () => { - await acquireSlot(priority) - try { - return await runCli(spec, args[0] ?? '', opts.timeoutMs ?? DEFAULT_TIMEOUT_MS, opts.onStderr) - } finally { - releaseSlot() - } - })() - .then(value => { readCache.set(key, { at: Date.now(), value }); return value }) + const flight = runScheduledCli( + spec, + args[0] ?? '', + opts.timeoutMs ?? DEFAULT_TIMEOUT_MS, + priority, + opts.onStderr, + ) .finally(() => { readInflight.delete(key) }) readInflight.set(key, flight) return flight @@ -568,6 +745,7 @@ export function spawnCli( * Mutations count as interactive, so they take a run slot ahead of any queued * background warm — a Settings save is never stuck behind speculative prefetch. */ export function spawnCliAction(args: string[], opts: { timeoutMs?: number } = {}): Promise { + if (shuttingDown) return Promise.resolve({ ok: false, stdout: '', stderr: 'codeburn is shutting down', code: null }) const timeoutMs = opts.timeoutMs ?? DEFAULT_TIMEOUT_MS const target = resolveTarget() if (!target) return Promise.resolve({ ok: false, stdout: '', stderr: 'codeburn CLI not found', code: null }) @@ -580,6 +758,7 @@ export function spawnCliAction(args: string[], opts: { timeoutMs?: number } = {} return { ok: false, stdout: '', stderr: 'codeburn cancelled', code: null } } try { + if (shuttingDown) return { ok: false, stdout: '', stderr: 'codeburn is shutting down', code: null } return await runAction(spec, args, timeoutMs) } finally { releaseSlot() @@ -600,9 +779,13 @@ function runAction(spec: SpawnSpec, args: string[], timeoutMs: number): Promise< settled = true clearTimeout(timer) activeChildren.delete(child) - // The action may have changed config the read cache still reflects; a - // Settings refetch fires immediately after, so serve it fresh data. - readCache.clear() + if (result.ok && actionInvalidatesServe(args)) { + // Fence coalescing before the action promise resolves. An immediate + // same-argv refetch belongs to the new config generation even while an + // older read is still running. + readGeneration += 1 + restartServeAfterMutation() + } resolve(result) } diff --git a/app/electron/main.ts b/app/electron/main.ts index 7d2c2bb4..d3676474 100644 --- a/app/electron/main.ts +++ b/app/electron/main.ts @@ -1,7 +1,7 @@ import { app, BrowserWindow, dialog, ipcMain, Menu, nativeTheme, shell, type MenuItemConstructorOptions } from 'electron' import path from 'node:path' -import { CliError, killAll, resolveCodeburnPath, spawnCli, spawnCliAction, startServeWarmup, type ActionResult, type SpawnPriority } from './cli' +import { CliError, DESKTOP_COLD_TIMEOUT_MS, resolveCodeburnPath, shutdownAll, spawnCli, spawnCliAction, startServe, type ActionResult, type SpawnPriority } from './cli' import { getQuota, sanitizeError } from './quota' import { Telemetry } from './telemetry' import { createUpdateChecker, type UpdateChecker, type UpdateStatus } from './updates' @@ -77,7 +77,7 @@ export type Envelope = { ok: true; value: T } | { ok: false; error: // slowness. Give the first (cold) overview a long window; revert to the default // once it succeeds. Sections gate their own first poll on this one resolving so // the cold hydration runs ONCE, not once per section in parallel. -const WARMUP_TIMEOUT_MS = 10 * 60_000 +const WARMUP_TIMEOUT_MS = DESKTOP_COLD_TIMEOUT_MS // Wire marker for CLI scan-progress lines (src/parser.ts: PROGRESS_LINE_PREFIX). const PROGRESS_LINE_PREFIX = 'CODEBURN_PROGRESS ' // IPC channel carrying cold-start scan-progress events to the splash. @@ -564,15 +564,15 @@ function bootstrap(): void { app.on('before-quit', createBeforeQuitHandler({ getTelemetry: () => telemetryInstance, - killAll, + killAll: shutdownAll, quit: () => app.quit(), })) void app.whenReady().then(() => { - // Start the resident serve child early so its warm-up (one cache parse) - // finishes during the first panels' cold spawns; every fetch after that - // answers from the warm child in milliseconds. - startServeWarmup() + // Start the resident child early, but issue no artificial warm-up query: + // the first real overview request is the single cache hydration and streams + // its progress through serve. Every later panel reuses that parsed cache. + startServe() // Consent-gated anonymous telemetry (desktop only). Nothing transmits until // the onboarding consent screen is completed and the toggle is on; EU/EEA/ // UK/CH installs default the toggle off. Dev builds never send. diff --git a/app/scripts/stage-cli.mjs b/app/scripts/stage-cli.mjs index 69085678..2d4381f9 100644 --- a/app/scripts/stage-cli.mjs +++ b/app/scripts/stage-cli.mjs @@ -11,6 +11,7 @@ // build/cli/package.json (root package.json: {version}, type:module) // build/cli/dist/cli.js (Node-version-guard launcher → ./main.js) // build/cli/dist/main.js (the bundle) +// build/cli/dist/parse-worker.js (the parse worker thread's own entry) // build/cli/node_modules/ (production dependency closure) // // The production closure is copied out of the already-installed root @@ -29,7 +30,7 @@ const dist = join(root, 'dist') const rootModules = join(root, 'node_modules') const stage = join(appDir, 'build', 'cli') -for (const f of ['cli.js', 'main.js']) { +for (const f of ['cli.js', 'main.js', 'parse-worker.js']) { if (!existsSync(join(dist, f))) { throw new Error(`stage-cli: ${join(dist, f)} is missing — build the root CLI first`) } @@ -41,6 +42,10 @@ mkdirSync(join(stage, 'dist'), { recursive: true }) copyFileSync(join(root, 'package.json'), join(stage, 'package.json')) copyFileSync(join(dist, 'cli.js'), join(stage, 'dist', 'cli.js')) copyFileSync(join(dist, 'main.js'), join(stage, 'dist', 'main.js')) +// The cold-parse worker pool resolves this as a sibling of the bundle it runs +// from, so it has to be staged alongside main.js or a packaged app silently +// loses every parse thread. +copyFileSync(join(dist, 'parse-worker.js'), join(stage, 'dist', 'parse-worker.js')) // Desktop-app launch shim (the app spawns this, not cli.js). The packaged app // runs the CLI with Electron's own binary as Node (ELECTRON_RUN_AS_NODE=1). diff --git a/docs/architecture.md b/docs/architecture.md index 088e46f8..2f7277b1 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -69,6 +69,67 @@ output formatter (Ink TUI, JSON, or menubar-json) `src/parser.ts` is the central aggregator. Public exports: `parseAllSessions`, `filterProjectsByName`, `extractMcpInventory`. It owns the dedup `Set` (`seenKeys`) that is passed into every provider parser so a turn that surfaces in two providers (Claude logs vs. Cursor mirror, for instance) is counted once. +### Parallel Cold Parse + +A cold parse spends most of its time on work that is per-file and pure: reading a +session JSONL or a Codex rollout, decoding it, and turning each line into a +journal entry. `src/parse-workers.ts` moves that onto `worker_threads` when the +pending workload is big enough to pay for them. Each worker runs the same +per-file function the serial path runs — `parseClaudeFileFull` for a Claude +session, `parseCodexFileFull` for a Codex rollout — against an empty dedup set, +and ships the result back as a JSON string together with every dedup key it +claimed. The parent installs results in the same order the serial loop would, and +everything with cross-file state (the dedup sets, canonical project paths, spawn +links, PR correlation, the Codex result cache) stays on the main thread. A file +whose keys were already claimed by an earlier file, or whose worker failed, is +re-parsed in-process — so the output is identical to the serial path either way. +That overlap check is what makes a forked Codex rollout safe: it replays its +parent's token_count history under the parent's key namespace, collides, and is +re-parsed against the real dedup set. + +A Codex worker never touches `src/codex-cache.ts`: it returns the cache entry it +would have written and the parent writes it, in install order, so +`flushCodexCache` publishes exactly what a serial parse would. Only whole-file +parses go off-thread; the append/incremental paths (a Claude append, a Codex +byte-offset resume) are untouched and stay in-process. The decision is made per +provider — the Claude scan and the provider loop run one after the other, so at +most one pool is alive — and the pool is terminated when its scan ends, so the +resident `serve` child never accumulates threads. + +The pool is off by default for anything that is not a large cold parse: + +| Gate | Serial when | +|---|---| +| Pending bytes | under 200 MB behind the pending whole-file parses | +| Cores | `availableParallelism() <= 2` | +| Memory | under 4 GB available | + +Otherwise the worker count is +`min(cores - 1, min(0.25 * available, 2 GB) / perWorker, max(pendingFiles / 50, pendingBytes / 200 MB))`. +Files and bytes each earn threads on their own, so a few hundred multi-hundred-MB +Codex rollouts parallelize as well as a few thousand small Claude transcripts. The +gate is bytes only, deliberately: 250 pending files holding under a megabyte +between them spawn threads that make the run ~5% slower, and a file count only +starts paying for itself around 400. + +`perWorker` is the per-thread memory budget, derived per parse as +`clamp(256 MB, 2 x (pendingBytes / pendingFiles) + 128 MB, 1 GB)`. A flat figure +was wrong in both directions: small Claude transcripts peak well under 256 MB, +while a 260 MB Codex rollout peaks near 430 MB in its worker and scales linearly +with the pool. The budget also covers the parent, which buffers up to `pool.size` +finished results while it installs one. + +"Available" is `process.availableMemory()`, falling back to `os.totalmem()`. It is +deliberately not `os.freemem()`: on macOS that counts free pages rather than +available memory and reads as a few hundred MB on an idle 128 GB machine, so a +gate built on it switches the feature on and off between runs. On Linux outside a +memory-limited cgroup, `availableMemory()` reports free memory and can still +under-report on a busy host — which fails safe, to fewer threads or none. + +`CODEBURN_PARSE_WORKERS` overrides the decision and skips every gate above: +`0` forces the serial parse, `N` forces N workers (capped at the core count). +`CODEBURN_VERBOSE=1` prints the resolved worker count and the reason for it. + ### Cache Layers Three caches under `~/.cache/codeburn/` (override with `CODEBURN_CACHE_DIR`): diff --git a/docs/providers/kiro.md b/docs/providers/kiro.md index 0252e901..d6e10b32 100644 --- a/docs/providers/kiro.md +++ b/docs/providers/kiro.md @@ -59,6 +59,7 @@ The stores are disjoint (v2 sessions use `sess_`-prefixed IDs in a separate dire - Token counts are estimated via char count (`CHARS_PER_TOKEN = 4`). - **Credits are the cost source; tokens stay estimated.** Kiro bills in credits ($20/mo for 1,000; overage $0.04/credit). CLI (`metering_usage`), v1 executions (`usageSummary[].usage`), and v2 (`usage_summary.promptTurnSummaries[].usage`) turns record real credits, converted to USD at `USD_PER_KIRO_CREDIT = 0.04` (the public overage rate — the same never-understate approach as Codebuff). Turns without credit data fall back to token-estimated cost (`costIsEstimated: true`); legacy `.chat` and workspace-session records carry no usage data, so they are always token-estimated. Note: an earlier CLI implementation summed credit values directly as dollars, overstating cost 25×. Token *counts* remain char-estimated everywhere (input undercounts: only visible transcript text is seen, not the full resent context; v2's `session_metadata.contextUsage.usagePercentage` × context window is a better input proxy if ever needed). v2 does keep the real `modelId`, so unlike the v1 execution-file path it is not mislabeled `kiro-auto`. - **Cost is frozen at parse time.** Kiro is on the `costUSD` pass-through allowlist in `providerCallToCachedCall` (alongside mistral-vibe, devin, hermes, …), so its credit-based cost survives the session cache instead of being re-priced from estimated tokens — token re-pricing understated/overstated real kiro spend by up to 16× per model. The tradeoff, shared with all allowlisted providers: `codeburn price-override` and `model-alias` do not affect kiro dollar amounts (token *counts* are unaffected). Historical caches from before this change re-parse via the `CACHE_VERSION` bump to 5. +- **`projectPath` for git attribution.** The parser now records the session's working directory as `projectPath` (CLI `meta.cwd`, v2 `workspacePaths[0]`, workspace sessions' `workspaceDirectory`), which sync attribution needs to resolve the git repo. The `project-path-v1` parse-version bump re-parses cached kiro history once; sessions in linked git worktrees now group under the main repo. ## When fixing a bug here diff --git a/mac/Sources/CodeBurnMenubar/CodeBurnApp.swift b/mac/Sources/CodeBurnMenubar/CodeBurnApp.swift index e6366baa..63144f76 100644 --- a/mac/Sources/CodeBurnMenubar/CodeBurnApp.swift +++ b/mac/Sources/CodeBurnMenubar/CodeBurnApp.swift @@ -127,9 +127,8 @@ final class AppDelegate: NSObject, NSApplicationDelegate, NSPopoverDelegate, NSM // interaction (popover open, wake) refreshes immediately. restorePersistedCurrency() - // Resident serve child: payload fetches answer from a warm CLI once - // its warm-up completes; until then (and on any failure) fetches keep - // the spawn path. See ServeConnection. + // Start the resident CLI early without an artificial query. The first + // real status refresh becomes its only cold warm-up. See ServeConnection. Task { await ServeConnection.shared.ensureStarted() } // #868 experiment: restore only the activation half of the #147 fix. // Packaged builds ship LSUIElement=true, so the policy is .accessory diff --git a/mac/Sources/CodeBurnMenubar/CurrencyState.swift b/mac/Sources/CodeBurnMenubar/CurrencyState.swift index def6cf32..1c9f3d12 100644 --- a/mac/Sources/CodeBurnMenubar/CurrencyState.swift +++ b/mac/Sources/CodeBurnMenubar/CurrencyState.swift @@ -77,11 +77,7 @@ actor FXRateCache { private var loaded = false private var cacheFilePath: String { - let base = FileManager.default.urls(for: .cachesDirectory, in: .userDomainMask)[0] - return base - .appendingPathComponent("codeburn-mac", isDirectory: true) - .appendingPathComponent("fx-rates.json") - .path + return (CodeBurnCacheDirectory.resolve() as NSString).appendingPathComponent("fx-rates.json") } private func loadIfNeeded() { diff --git a/mac/Sources/CodeBurnMenubar/Data/CodeBurnCacheDirectory.swift b/mac/Sources/CodeBurnMenubar/Data/CodeBurnCacheDirectory.swift new file mode 100644 index 00000000..d5e31b82 --- /dev/null +++ b/mac/Sources/CodeBurnMenubar/Data/CodeBurnCacheDirectory.swift @@ -0,0 +1,18 @@ +import Foundation + +/// Resolves the on-disk directory shared by the CLI, desktop app and menubar. +enum CodeBurnCacheDirectory { + static func resolve( + environment: [String: String] = ProcessInfo.processInfo.environment, + homeDirectory: URL = FileManager.default.homeDirectoryForCurrentUser + ) -> String { + if let override = environment["CODEBURN_CACHE_DIR"], + !override.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty { + return override + } + return homeDirectory + .appendingPathComponent(".cache", isDirectory: true) + .appendingPathComponent("codeburn", isDirectory: true) + .path + } +} diff --git a/mac/Sources/CodeBurnMenubar/Data/DataClient.swift b/mac/Sources/CodeBurnMenubar/Data/DataClient.swift index cafe0449..73159eed 100644 --- a/mac/Sources/CodeBurnMenubar/Data/DataClient.swift +++ b/mac/Sources/CodeBurnMenubar/Data/DataClient.swift @@ -123,21 +123,68 @@ struct DataClient { subcommand: [String], qualityOfService: QualityOfService = .userInitiated ) async throws -> ProcessResult { - // Serve fast path: a warm resident `codeburn serve` child answers the - // status payload without a spawn (no node boot, no session-cache - // reload). Any serve failure falls back to the spawn path below, so - // this is strictly an optimization; it also takes no spawn slot. + try await runCLI( + subcommand: subcommand, + serveRequest: { args in + try await ServeConnection.shared.request(args: args) + }, + spawnFallback: { + await spawnLimiter.acquire() + defer { Task { await spawnLimiter.release() } } + let process = CodeburnCLI.makeProcess( + subcommand: subcommand, + qualityOfService: qualityOfService + ) + return try await runProcess( + process, + timeoutSeconds: spawnTimeoutSeconds, + label: subcommand.joined(separator: " ") + ) + } + ) + } + + /// Internal seam for behavior-shaped lifecycle tests. Production supplies + /// the shared resident and globally limited one-shot closures above. + static func runCLI( + subcommand: [String], + serveRequest: ([String]) async throws -> Data, + spawnFallback: () async throws -> ProcessResult + ) async throws -> ProcessResult { + // Serve path: the first real status payload warms the resident child, + // then later payloads reuse it (no node boot or session-cache reload). + // Transport/protocol failures fall back to the spawn path below, so + // the resident remains an optimization. Resource-policy failures stay + // terminal and cannot bypass the resident output ceiling. if ServeConnection.isEligible(subcommand) { - if let stdout = try? await ServeConnection.shared.requestIfWarm(args: subcommand) { + do { + let stdout = try await serveRequest(subcommand) return ProcessResult(stdout: stdout, stderr: "", exitCode: 0) + } catch let error as CancellationError { + // Cancellation is control flow from the refresh owner. Starting + // a fallback process here would turn cancelled work into a new + // expensive cold parse and delay task teardown. + throw error + } catch { + if let terminalError = terminalServeError(error) { + throw terminalError + } + // Resident serve is only an optimization. Protocol, child, and + // timeout failures retain the established one-shot fallback, + // unless a sibling teardown raced this task's cancellation. + try Task.checkCancellation() } } - await spawnLimiter.acquire() - defer { Task { await spawnLimiter.release() } } - let process = CodeburnCLI.makeProcess(subcommand: subcommand, qualityOfService: qualityOfService) - return try await runProcess(process, - timeoutSeconds: spawnTimeoutSeconds, - label: subcommand.joined(separator: " ")) + return try await spawnFallback() + } + + /// Some resident failures are terminal resource-policy decisions, not + /// transport failures. Retrying those through the one-shot path would redo + /// the cold scan and could bypass the resident's stricter output ceiling. + static func terminalServeError(_ error: Error) -> DataClientError? { + guard let failure = error as? ServeConnection.ServeRequestFailed, + failure.reason == .outputTooLarge else { return nil } + return .outputTooLarge } /// Runs an already-configured process to completion, draining its output and diff --git a/mac/Sources/CodeBurnMenubar/Data/MenubarStatusCache.swift b/mac/Sources/CodeBurnMenubar/Data/MenubarStatusCache.swift index ef5f217a..72a8dfb7 100644 --- a/mac/Sources/CodeBurnMenubar/Data/MenubarStatusCache.swift +++ b/mac/Sources/CodeBurnMenubar/Data/MenubarStatusCache.swift @@ -9,8 +9,9 @@ struct MenubarStatusCache { /// Default location under `~/.cache/codeburn/`. static func standard() -> MenubarStatusCache { - let home = FileManager.default.homeDirectoryForCurrentUser.path - return MenubarStatusCache(statusPath: "\(home)/.cache/codeburn/menubar-status.json") + let cacheDir = CodeBurnCacheDirectory.resolve() + let path = (cacheDir as NSString).appendingPathComponent("menubar-status.json") + return MenubarStatusCache(statusPath: path) } struct BadgeRead { diff --git a/mac/Sources/CodeBurnMenubar/Data/ServeConnection.swift b/mac/Sources/CodeBurnMenubar/Data/ServeConnection.swift index 0eb13578..4d1f879c 100644 --- a/mac/Sources/CodeBurnMenubar/Data/ServeConnection.swift +++ b/mac/Sources/CodeBurnMenubar/Data/ServeConnection.swift @@ -1,3 +1,4 @@ +import Darwin import Foundation /// A resident `codeburn serve --stdio` child, held so payload fetches skip the @@ -6,53 +7,118 @@ import Foundation /// replies are `{id, ok, output}`. Mirrors the desktop app's client contract: /// /// - Only `status` payload queries route here; anything else spawns as before. -/// - Requests route through serve only once the child is READY and WARM (one -/// completed query), so cold start behaves exactly as today. -/// - Any failure falls back to the spawn path for that call; three child -/// deaths disable serve for this app run. +/// - The first real status request is also the warm-up. It may be written +/// before the child announces READY; the pipe buffers it until serve reads +/// stdin, avoiding a second one-shot process that parses the same cache. +/// - Transport/protocol failures fall back to the spawn path for that call; +/// resource-policy failures remain terminal. Three child deaths disable +/// serve for this app run. /// - The child's stdin closing (app quit, even SIGKILL) ends the server loop /// on the CLI side, so no orphan survives the menubar. actor ServeConnection { static let shared = ServeConnection() + typealias ProcessFactory = ([String], QualityOfService) -> Process + typealias TimeoutSleep = @Sendable (UInt64) async throws -> Void + + private struct QueuedRequest { + let token: Int + let args: [String] + let continuation: CheckedContinuation + } + + private struct ActiveRequest { + let token: Int + let id: Int + let args: [String] + let child: Process + } + private var process: Process? private var stdinHandle: FileHandle? private var nextId = 1 + private var nextRequestToken = 1 + private var queuedRequests: [QueuedRequest] = [] + private var activeRequest: ActiveRequest? private var pending: [Int: CheckedContinuation] = [:] - private var ready = false - private var warm = false + private var requestTimeouts: [Int: Task] = [:] + private var timeoutOwners: [Int: Process] = [:] + private var responseBytes: [Int: Int] = [:] private var deaths = 0 private var buffer = Data() + private var receivedTerminalResponse = false + private var outputTasks: [ObjectIdentifier: Task] = [:] + private var terminationTasks: [ObjectIdentifier: Task] = [:] + private let makeProcess: ProcessFactory + private let timeoutSleep: TimeoutSleep + private let terminationGraceSleep: TimeoutSleep + private let responseLimitBytes: Int private static let maxDeaths = 3 - private static let requestTimeoutSeconds: UInt64 = 60 + static let maxResponseBytes = 16 * 1024 * 1024 + private static let stdoutReadChunkBytes = 64 * 1024 + private static let terminationGraceNanoseconds: UInt64 = 1_000_000_000 + private static let coldRequestTimeoutNanoseconds: UInt64 = 10 * 60 * 1_000_000_000 + private static let warmRequestTimeoutNanoseconds: UInt64 = 60 * 1_000_000_000 struct ServeUnavailable: Error {} - struct ServeRequestFailed: Error { let message: String } + enum FailureReason: Sendable, Equatable { + case generic + case outputTooLarge + } + struct ServeRequestFailed: Error, Sendable { + let message: String + let reason: FailureReason + + init(message: String, reason: FailureReason = .generic) { + self.message = message + self.reason = reason + } + } + + init( + makeProcess: @escaping ProcessFactory = CodeburnCLI.makeProcess, + timeoutSleep: @escaping TimeoutSleep = { nanoseconds in + try await Task.sleep(nanoseconds: nanoseconds) + }, + terminationGraceSleep: @escaping TimeoutSleep = { nanoseconds in + try await Task.sleep(nanoseconds: nanoseconds) + }, + responseLimitBytes: Int = ServeConnection.maxResponseBytes + ) { + self.makeProcess = makeProcess + self.timeoutSleep = timeoutSleep + self.terminationGraceSleep = terminationGraceSleep + precondition(responseLimitBytes > 0) + self.responseLimitBytes = responseLimitBytes + } static func isEligible(_ subcommand: [String]) -> Bool { subcommand.first == "status" } - /// Kick the child off (idempotent). Called from app startup; fetches keep - /// spawning until the warm-up completes. + /// Kick the child off (idempotent). Called from app startup and again by + /// the first request in case the startup task has not run yet. func ensureStarted() { guard process == nil, deaths < Self.maxDeaths else { return } - let child = CodeburnCLI.makeProcess(subcommand: ["serve", "--stdio"], qualityOfService: .utility) + // This single resident serves both background and user-visible status + // requests. Its cold hydration replaces the old interactive one-shot, + // so keep the child at the same user-initiated QoS as visible fetches. + let child = makeProcess(["serve", "--stdio"], .userInitiated) let stdinPipe = Pipe() + let stdinWriter = stdinPipe.fileHandleForWriting + // Suppress SIGPIPE only for this connection's write end. A process-wide + // SIG_IGN leaks into unrelated libraries and children; F_SETNOSIGPIPE + // keeps a closed child stdin on the normal throwable EPIPE path. + guard Darwin.fcntl(stdinWriter.fileDescriptor, F_SETNOSIGPIPE, 1) == 0 else { + deaths = Self.maxDeaths + return + } let stdoutPipe = Pipe() + let stdoutReader = stdoutPipe.fileHandleForReading child.standardInput = stdinPipe child.standardOutput = stdoutPipe child.standardError = FileHandle.nullDevice - stdoutPipe.fileHandleForReading.readabilityHandler = { handle in - let data = handle.availableData - guard !data.isEmpty else { return } - Task { await ServeConnection.shared.consume(data) } - } - child.terminationHandler = { _ in - stdoutPipe.fileHandleForReading.readabilityHandler = nil - Task { await ServeConnection.shared.childDied() } - } do { try child.run() } catch { @@ -60,112 +126,404 @@ actor ServeConnection { return } process = child - stdinHandle = stdinPipe.fileHandleForWriting - Task { - // Warm-up: one cheap query makes the child parse the session cache - // once; every later payload answers from the warm in-memory copy. - _ = try? await self.send(args: ["status", "--format", "menubar-json", "--period", "today", "--no-optimize"]) - await self.markWarm() + stdinHandle = stdinWriter + let generation = ObjectIdentifier(child) + // One blocking reader owns this generation's stdout. It never reads a + // second bounded chunk until the actor has consumed the first, giving + // the 16 MiB protocol limit real backpressure instead of accumulating + // an unbounded callback/AsyncStream backlog. EOF is observed only after + // the pipe's final bytes, so child death cannot overtake a split reply. + outputTasks[generation] = Task.detached { [weak self] in + var bytes = [UInt8](repeating: 0, count: Self.stdoutReadChunkBytes) + while !Task.isCancelled { + let count = Darwin.read(stdoutReader.fileDescriptor, &bytes, bytes.count) + if count > 0 { + guard let self else { break } + await self.consume(Data(bytes[0.. Data { - guard ready, warm, process != nil else { throw ServeUnavailable() } - return try await send(args: args) + /// Send the first real payload through the resident child. A request does + /// not need to wait for the READY frame: stdin is safe to write as soon as + /// Process.run() succeeds, and serve serializes it after initialization. + func request(args: [String]) async throws -> Data { + try Task.checkCancellation() + ensureStarted() + guard process != nil else { throw ServeUnavailable() } + let token = nextRequestToken + nextRequestToken += 1 + let response = try await withTaskCancellationHandler { + try await withCheckedThrowingContinuation { continuation in + queuedRequests.append(QueuedRequest( + token: token, + args: args, + continuation: continuation + )) + startNextRequestIfPossible() + } + } onCancel: { + Task { await self.cancelRequest(token: token) } + } + try Task.checkCancellation() + return response } func shutdown() { deaths = Self.maxDeaths process?.terminate() - failAllPending() + for task in terminationTasks.values { task.cancel() } + terminationTasks.removeAll() + cancelAllTimeouts() + failAllRequests() process = nil stdinHandle = nil + buffer = Data() + receivedTerminalResponse = false } // MARK: - internals - private func markWarm() { - if process != nil { warm = true } - } + private func startNextRequestIfPossible() { + guard activeRequest == nil, !queuedRequests.isEmpty else { return } + ensureStarted() + guard let stdinHandle, let child = process else { + failQueuedRequests(error: ServeUnavailable()) + return + } + // A Process can report not-running just before its termination callback + // reaches the ordered event stream. Keep the request queued for that + // event instead of writing to a generation which is already exiting. + guard child.isRunning else { return } - private func send(args: [String]) async throws -> Data { - guard let stdinHandle, let child = process else { throw ServeUnavailable() } + let request = queuedRequests.removeFirst() let id = nextId nextId += 1 - let request: [String: Any] = ["id": id, "args": args] - let line = try JSONSerialization.data(withJSONObject: request) - return try await withThrowingTaskGroup(of: Data.self) { group in - group.addTask { - try await withCheckedThrowingContinuation { (continuation: CheckedContinuation) in - Task { await self.registerPending(id: id, continuation: continuation) } - do { - try stdinHandle.write(contentsOf: line + Data("\n".utf8)) - } catch { - Task { await self.rejectPending(id: id, error: ServeRequestFailed(message: "stdin write failed")) } - } - } - } - group.addTask { - try await Task.sleep(nanoseconds: Self.requestTimeoutSeconds * 1_000_000_000) - // A hung request would block the serialized queue behind it: - // kill the child so everything falls back to spawns. - await self.rejectPending(id: id, error: ServeRequestFailed(message: "serve timeout")) - child.terminate() - throw ServeRequestFailed(message: "serve timeout") - } - let result = try await group.next()! - group.cancelAll() - return result + let line: Data + do { + line = try JSONSerialization.data(withJSONObject: ["id": id, "args": request.args]) + } catch { + request.continuation.resume(throwing: error) + startNextRequestIfPossible() + return + } + + // The previous response can resume its caller just before EOF reaches + // this actor. Avoid admitting a successor to an already-reaped child; + // the reader's ordered EOF path will start it on a replacement. + guard child.isRunning else { + queuedRequests.insert(request, at: 0) + outputStreamEnded(for: child) + return + } + + // Select and arm the timeout only when this request becomes the sole + // protocol request in flight. A queued request must not spend its own + // budget while its predecessor is still hydrating or draining. + let timeoutNanoseconds = receivedTerminalResponse + ? Self.warmRequestTimeoutNanoseconds + : Self.coldRequestTimeoutNanoseconds + activeRequest = ActiveRequest( + token: request.token, + id: id, + args: request.args, + child: child + ) + pending[id] = request.continuation + responseBytes[id] = 0 + do { + try stdinHandle.write(contentsOf: line + Data("\n".utf8)) + armTimeout(id: id, child: child, nanoseconds: timeoutNanoseconds) + } catch { + // The previous terminal frame can resume its caller just before + // EOF detaches that generation. Preserve this never-admitted + // request and retry it on the replacement instead of surfacing a + // transient EPIPE to the UI. + pending.removeValue(forKey: id) + responseBytes.removeValue(forKey: id) + activeRequest = nil + queuedRequests.insert(request, at: 0) + outputStreamEnded(for: child) } } - private func registerPending(id: Int, continuation: CheckedContinuation) { - pending[id] = continuation + private func cancelRequest(token: Int) { + if let index = queuedRequests.firstIndex(where: { $0.token == token }) { + let request = queuedRequests.remove(at: index) + request.continuation.resume(throwing: CancellationError()) + return + } + guard let activeRequest, activeRequest.token == token, + let continuation = pending.removeValue(forKey: activeRequest.id) else { return } + continuation.resume(throwing: CancellationError()) + // Caller cancellation abandons only this response. The serialized serve + // child may still be doing the expensive first hydration, and killing it + // here lets tab switches and UI watchdogs restart that work indefinitely. + // Its independent request timeout remains armed: a command that never + // returns is still reaped, so it cannot wedge every later serialized call. } - private func rejectPending(id: Int, error: Error) { + private func armTimeout(id: Int, child: Process, nanoseconds: UInt64) { + let sleep = timeoutSleep + timeoutOwners[id] = child + requestTimeouts[id] = Task.detached { [weak self] in + do { + try await sleep(nanoseconds) + } catch { + return + } + await self?.requestTimedOut(id: id) + } + } + + private func requestTimedOut(id: Int) { + guard let child = timeoutOwners.removeValue(forKey: id) else { return } + requestTimeouts.removeValue(forKey: id) + responseBytes.removeValue(forKey: id) if let continuation = pending.removeValue(forKey: id) { - continuation.resume(throwing: error) + continuation.resume(throwing: ServeRequestFailed(message: "serve timeout")) } - } - - private func consume(_ data: Data) { - buffer.append(data) - while let newline = buffer.firstIndex(of: UInt8(ascii: "\n")) { - let lineData = buffer.subdata(in: buffer.startIndex.. Bool { + guard let current = responseBytes[id], + count <= responseLimitBytes - current else { + outputOverflowed(child) + return false + } + responseBytes[id] = current + count + return true + } + + private func outputOverflowed(_ child: Process) { + guard process === child else { return } + // Detach this exact generation before terminating it. Its eventual exit + // and any already-scheduled stdout callbacks are then stale and cannot + // consume a second death or corrupt a replacement generation. + process = nil + stdinHandle = nil + buffer = Data() + receivedTerminalResponse = false + deaths += 1 + cancelTimeouts(ownedBy: child) + failAllRequests(error: ServeRequestFailed( + message: "serve output exceeded \(responseLimitBytes) bytes", + reason: .outputTooLarge + )) + if child.isRunning { child.terminate() } + } + + private func childDied(_ child: Process) { + guard process === child else { return } + process = nil + stdinHandle = nil + buffer.removeAll() + receivedTerminalResponse = false + deaths += 1 + cancelTimeouts(ownedBy: child) + if let activeRequest, activeRequest.child === child { + if let continuation = pending.removeValue(forKey: activeRequest.id) { + // Only read-only status requests enter this connection. If a + // generation exits after admission but before its terminal + // reply, retain the waiter and retry on the replacement rather + // than racing it into a one-shot fallback. A timed-out or + // cancelled waiter is already absent and is never retried. + queuedRequests.insert(QueuedRequest( + token: activeRequest.token, + args: activeRequest.args, + continuation: continuation + ), at: 0) + } + self.activeRequest = nil + } + // Requests which were never written survive an ordinary child crash. + // They begin on a replacement only after this ordered death event. + startNextRequestIfPossible() + } + + private func failAllRequests( + error: Error = ServeRequestFailed(message: "serve exited") + ) { for (_, continuation) in pending { - continuation.resume(throwing: ServeRequestFailed(message: "serve exited")) + continuation.resume(throwing: error) } pending.removeAll() + activeRequest = nil + failQueuedRequests(error: error) + } + + private func failQueuedRequests(error: Error) { + let requests = queuedRequests + queuedRequests.removeAll() + for request in requests { + request.continuation.resume(throwing: error) + } } } diff --git a/mac/Sources/CodeBurnMenubar/Data/SubscriptionSnapshotStore.swift b/mac/Sources/CodeBurnMenubar/Data/SubscriptionSnapshotStore.swift index 9357ee94..2c99dd06 100644 --- a/mac/Sources/CodeBurnMenubar/Data/SubscriptionSnapshotStore.swift +++ b/mac/Sources/CodeBurnMenubar/Data/SubscriptionSnapshotStore.swift @@ -14,13 +14,8 @@ struct SubscriptionSnapshot: Codable, Sendable { private let snapshotFilename = "subscription-snapshots.json" private let pruneOlderThanSeconds: TimeInterval = 30 * 24 * 3600 -private func snapshotsCacheDir() -> String { - return ProcessInfo.processInfo.environment["CODEBURN_CACHE_DIR"] - ?? (NSHomeDirectory() as NSString).appendingPathComponent(".cache/codeburn") -} - private func snapshotsPath() -> String { - return (snapshotsCacheDir() as NSString).appendingPathComponent(snapshotFilename) + return (CodeBurnCacheDirectory.resolve() as NSString).appendingPathComponent(snapshotFilename) } private actor SnapshotLock { diff --git a/mac/Sources/CodeBurnMenubar/Security/SafeFile.swift b/mac/Sources/CodeBurnMenubar/Security/SafeFile.swift index 3d6bda57..3b3dea29 100644 --- a/mac/Sources/CodeBurnMenubar/Security/SafeFile.swift +++ b/mac/Sources/CodeBurnMenubar/Security/SafeFile.swift @@ -2,7 +2,7 @@ import Foundation /// Symlink-safe file I/O with atomic writes and optional cross-process flock. /// -/// Every cache file we touch (`~/Library/Caches/codeburn-mac/fx-rates.json`, +/// Every cache file we touch (`~/.cache/codeburn/fx-rates.json`, /// `~/.cache/codeburn/subscription-snapshots.json`, `~/.config/codeburn/config.json`) is a /// legitimate target for a local-symlink attack: if an attacker plants a symlink from one of /// those paths to, say, `~/.ssh/config`, a naive `Data.write(to:)` blindly follows the link and diff --git a/mac/Tests/CodeBurnMenubarTests/CodeBurnCacheDirectoryTests.swift b/mac/Tests/CodeBurnMenubarTests/CodeBurnCacheDirectoryTests.swift new file mode 100644 index 00000000..cb217ed5 --- /dev/null +++ b/mac/Tests/CodeBurnMenubarTests/CodeBurnCacheDirectoryTests.swift @@ -0,0 +1,36 @@ +import Foundation +import Testing +@testable import CodeBurnMenubar + +@Suite("CodeBurnCacheDirectory") +struct CodeBurnCacheDirectoryTests { + @Test("honors CODEBURN_CACHE_DIR override") + func honorsOverride() { + let resolved = CodeBurnCacheDirectory.resolve( + environment: ["CODEBURN_CACHE_DIR": "/tmp/codeburn-shared-cache"], + homeDirectory: URL(fileURLWithPath: "/Users/test") + ) + + #expect(resolved == "/tmp/codeburn-shared-cache") + } + + @Test("falls back to the user's standard cache directory") + func fallsBackToStandardDirectory() { + let resolved = CodeBurnCacheDirectory.resolve( + environment: [:], + homeDirectory: URL(fileURLWithPath: "/Users/test", isDirectory: true) + ) + + #expect(resolved == "/Users/test/.cache/codeburn") + } + + @Test("ignores an empty cache override") + func ignoresEmptyOverride() { + let resolved = CodeBurnCacheDirectory.resolve( + environment: ["CODEBURN_CACHE_DIR": " \n"], + homeDirectory: URL(fileURLWithPath: "/Users/test", isDirectory: true) + ) + + #expect(resolved == "/Users/test/.cache/codeburn") + } +} diff --git a/mac/Tests/CodeBurnMenubarTests/ServeConnectionTests.swift b/mac/Tests/CodeBurnMenubarTests/ServeConnectionTests.swift new file mode 100644 index 00000000..a5c7588e --- /dev/null +++ b/mac/Tests/CodeBurnMenubarTests/ServeConnectionTests.swift @@ -0,0 +1,1161 @@ +import Darwin +import Foundation +import Testing +@testable import CodeBurnMenubar + +private let ignoredSIGPIPEHandlerBits = unsafeBitCast(SIG_IGN, to: UInt.self) +private let coldTimeoutNanoseconds: UInt64 = 10 * 60 * 1_000_000_000 +private let warmTimeoutNanoseconds: UInt64 = 60 * 1_000_000_000 +private let terminationGraceNanoseconds: UInt64 = 1_000_000_000 + +private func currentSIGPIPEHandlerBits() -> UInt { + var action = sigaction() + _ = sigaction(SIGPIPE, nil, &action) + return unsafeBitCast(action.__sigaction_u.__sa_handler, to: UInt.self) +} + +private actor TimeoutRecorder { + private var values: [UInt64] = [] + + func recordAndSleep(_ nanoseconds: UInt64) async throws { + values.append(nanoseconds) + // Cold timers stay pending until the fake child replies and the + // connection cancels them. The warm timer returns immediately to exercise + // the timeout path without a real one-minute wait. + if nanoseconds == warmTimeoutNanoseconds { return } + try await Task.sleep(nanoseconds: 5 * 1_000_000_000) + } + + func recordAndWait(_ nanoseconds: UInt64) async throws { + values.append(nanoseconds) + // This recorder verifies timeout selection without firing the timeout. + // The response must deterministically win, then cancel this sleeper. + try await Task.sleep(nanoseconds: 5 * 1_000_000_000) + } + + func snapshot() -> [UInt64] { values } +} + +private actor FallbackRecorder { + private var calls = 0 + + func record() { calls += 1 } + func snapshot() -> Int { calls } +} + +/// A cancellation-aware timeout clock that tests can advance explicitly. This +/// keeps the regression independent of the production ten-minute cold budget. +private actor ManualTimeoutClock { + private struct Waiter { + let nanoseconds: UInt64 + let continuation: CheckedContinuation + } + + private var nextToken = 0 + private var waiters: [Int: Waiter] = [:] + private var recorded: [UInt64] = [] + + func sleep(_ nanoseconds: UInt64) async throws { + let token = nextToken + nextToken += 1 + recorded.append(nanoseconds) + try await withTaskCancellationHandler { + try await withCheckedThrowingContinuation { (continuation: CheckedContinuation) in + if Task.isCancelled { + continuation.resume(throwing: CancellationError()) + } else { + waiters[token] = Waiter(nanoseconds: nanoseconds, continuation: continuation) + } + } + } onCancel: { + Task { await self.cancel(token) } + } + } + + func snapshot() -> [UInt64] { + waiters.keys.sorted().compactMap { waiters[$0]?.nanoseconds } + } + + func history() -> [UInt64] { recorded } + + func fireOldest() { + guard let token = waiters.keys.min(), let waiter = waiters.removeValue(forKey: token) else { return } + waiter.continuation.resume() + } + + private func cancel(_ token: Int) { + guard let waiter = waiters.removeValue(forKey: token) else { return } + waiter.continuation.resume(throwing: CancellationError()) + } +} + +private final class QualityOfServiceRecorder: @unchecked Sendable { + private let lock = NSLock() + private var values: [QualityOfService] = [] + + func record(_ value: QualityOfService) { + lock.lock() + values.append(value) + lock.unlock() + } + + func snapshot() -> [QualityOfService] { + lock.lock() + defer { lock.unlock() } + return values + } +} + +private final class ProcessQueue: @unchecked Sendable { + private let lock = NSLock() + private var processes: [Process] + + init(_ processes: [Process]) { + self.processes = processes + } + + func take(qualityOfService: QualityOfService) -> Process { + lock.lock() + let child = processes.removeFirst() + lock.unlock() + child.qualityOfService = qualityOfService + return child + } + + var remainingCount: Int { + lock.lock() + defer { lock.unlock() } + return processes.count + } +} + +@Suite("ServeConnection", .serialized) +struct ServeConnectionTests { + @Test("the resident child starts at user-initiated QoS") + func residentChildUsesInteractiveQoS() async { + let recorder = QualityOfServiceRecorder() + let connection = ServeConnection { _, qualityOfService in + recorder.record(qualityOfService) + let child = Process() + child.executableURL = URL(fileURLWithPath: "/bin/sh") + child.arguments = ["-c", "while IFS= read -r line; do :; done"] + child.qualityOfService = qualityOfService + return child + } + + await connection.ensureStarted() + + #expect(recorder.snapshot() == [.userInitiated]) + await connection.shutdown() + } + + @Test("cancelling a hung request returns promptly") + func cancellationUnblocksPendingContinuation() async throws { + let dir = NSTemporaryDirectory() + "serve-connection-cancel-test-" + UUID().uuidString + try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(atPath: dir) } + let requestMarker = dir + "/request-read" + + let connection = ServeConnection { _, qualityOfService in + let child = Process() + child.executableURL = URL(fileURLWithPath: "/bin/sh") + child.arguments = ["-c", "IFS= read -r line; : > \"$1\"; sleep 1", "serve-fixture", requestMarker] + child.qualityOfService = qualityOfService + return child + } + + let request = Task { + try await connection.request(args: ["status", "--format", "menubar-json"]) + } + for _ in 0..<200 where !FileManager.default.fileExists(atPath: requestMarker) { + try await Task.sleep(nanoseconds: 10_000_000) + } + #expect(FileManager.default.fileExists(atPath: requestMarker)) + + let clock = ContinuousClock() + let started = clock.now + request.cancel() + do { + _ = try await request.value + #expect(Bool(false), "cancelled request unexpectedly succeeded") + } catch { + #expect(error is CancellationError) + } + let elapsed = started.duration(to: clock.now) + #expect(elapsed < .milliseconds(500)) + await connection.shutdown() + } + + @Test("a request queued during cancelled hydration completes on the same child") + func cancellationKeepsQueuedRequestOnResidentChild() async throws { + let dir = NSTemporaryDirectory() + "serve-connection-cancel-overlap-test-" + UUID().uuidString + try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(atPath: dir) } + let pidsFile = dir + "/pids" + let eventsFile = dir + "/events" + let releaseMarker = dir + "/release-first" + let recorder = TimeoutRecorder() + + let connection = ServeConnection( + makeProcess: { _, qualityOfService in + let child = Process() + child.executableURL = URL(fileURLWithPath: "/bin/sh") + child.arguments = ["-c", """ + printf '%s\n' "$$" >> "$1" + IFS= read -r first + first_id=$(printf '%s' "$first" | sed -E 's/.*"id":([0-9]+).*/\\1/') + printf 'first-read\n' >> "$2" + while [ ! -f "$3" ]; do sleep 0.01; done + printf '{"id":%s,"ok":true,"output":"late-%s"}\n' "$first_id" "$first_id" + printf 'late-first\n' >> "$2" + IFS= read -r second + second_id=$(printf '%s' "$second" | sed -E 's/.*"id":([0-9]+).*/\\1/') + printf 'second-read\n' >> "$2" + printf '{"id":%s,"ok":true,"output":"live-%s"}\n' "$second_id" "$second_id" + printf 'second-replied\n' >> "$2" + """, "serve-fixture", pidsFile, eventsFile, releaseMarker] + child.qualityOfService = qualityOfService + return child + }, + timeoutSleep: { nanoseconds in + try await recorder.recordAndWait(nanoseconds) + } + ) + + let first = Task { + try await connection.request(args: ["status", "--request", "first"]) + } + for _ in 0..<200 { + let events = (try? String(contentsOfFile: eventsFile, encoding: .utf8)) ?? "" + if events.contains("first-read\n") { break } + try await Task.sleep(nanoseconds: 10_000_000) + } + #expect(try String(contentsOfFile: eventsFile, encoding: .utf8) == "first-read\n") + + first.cancel() + do { + _ = try await first.value + #expect(Bool(false), "cancelled request unexpectedly succeeded") + } catch { + #expect(error is CancellationError) + } + + // Submit the next request while the child is still blocked hydrating + // the cancelled first one. It stays client-side queued: neither its + // stdin line nor its own timeout may begin yet. + let second = Task { + try await connection.request(args: ["status", "--request", "second"]) + } + try await Task.sleep(nanoseconds: 100_000_000) + #expect(await recorder.snapshot() == [coldTimeoutNanoseconds]) + #expect(try String(contentsOfFile: eventsFile, encoding: .utf8) == "first-read\n") + + _ = FileManager.default.createFile(atPath: releaseMarker, contents: Data()) + let secondPayload = try await second.value + + #expect(String(decoding: secondPayload, as: UTF8.self) == "live-2") + #expect(await recorder.snapshot() == [coldTimeoutNanoseconds, warmTimeoutNanoseconds]) + let pids = try String(contentsOfFile: pidsFile, encoding: .utf8) + .split(separator: "\n") + #expect(pids.count == 1) + let events = try String(contentsOfFile: eventsFile, encoding: .utf8) + .split(separator: "\n") + #expect(events == ["first-read", "late-first", "second-read", "second-replied"]) + await connection.shutdown() + } + + @Test("a cancelled never-returning request retains a timeout owner and cannot wedge later work") + func cancelledHungRequestIsEventuallyReaped() async throws { + let dir = NSTemporaryDirectory() + "serve-connection-cancel-timeout-test-" + UUID().uuidString + try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(atPath: dir) } + let firstReadMarker = dir + "/first-read" + let pidsFile = dir + "/pids" + let clock = ManualTimeoutClock() + let graceClock = ManualTimeoutClock() + + let stuckChild = Process() + stuckChild.executableURL = URL(fileURLWithPath: "/bin/sh") + stuckChild.arguments = ["-c", """ + trap '' TERM + printf '%s\n' "$$" >> "$1" + IFS= read -r line + : > "$2" + while :; do :; done + """, "serve-fixture", pidsFile, firstReadMarker] + + let replacement = Process() + replacement.executableURL = URL(fileURLWithPath: "/bin/sh") + replacement.arguments = ["-c", """ + printf '%s\n' "$$" >> "$1" + while IFS= read -r line; do + id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/') + printf '{"id":%s,"ok":true,"output":"replacement-%s"}\n' "$id" "$id" + done + """, "serve-fixture", pidsFile] + + let children = ProcessQueue([stuckChild, replacement]) + let connection = ServeConnection( + makeProcess: { _, qualityOfService in + children.take(qualityOfService: qualityOfService) + }, + timeoutSleep: { nanoseconds in + try await clock.sleep(nanoseconds) + }, + terminationGraceSleep: { nanoseconds in + try await graceClock.sleep(nanoseconds) + } + ) + defer { + if stuckChild.isRunning { _ = Darwin.kill(stuckChild.processIdentifier, SIGKILL) } + } + + let abandoned = Task { + try await connection.request(args: ["status", "--request", "stuck"]) + } + for _ in 0..<200 where !FileManager.default.fileExists(atPath: firstReadMarker) { + try await Task.sleep(nanoseconds: 10_000_000) + } + #expect(FileManager.default.fileExists(atPath: firstReadMarker)) + #expect(await clock.snapshot() == [coldTimeoutNanoseconds]) + + abandoned.cancel() + do { + _ = try await abandoned.value + #expect(Bool(false), "cancelled request unexpectedly succeeded") + } catch { + #expect(error is CancellationError) + } + + // The caller is gone, but the independently-owned cold timeout must + // remain armed. This assertion is the red-before regression: the old + // task-group race cancelled the only timeout along with the caller. + #expect(await clock.snapshot() == [coldTimeoutNanoseconds]) + let successor = Task { + try await connection.request(args: ["status", "--request", "after-cancel"]) + } + try await Task.sleep(nanoseconds: 100_000_000) + #expect(await clock.snapshot() == [coldTimeoutNanoseconds]) + #expect(children.remainingCount == 1) + + await clock.fireOldest() + for _ in 0..<200 where children.remainingCount > 0 { + try await Task.sleep(nanoseconds: 10_000_000) + } + let replacementStartedBeforeOldEOF = children.remainingCount == 0 + #expect(replacementStartedBeforeOldEOF) + // Keep the red-before run finite: the old implementation waits for EOF + // forever because this fixture deliberately ignores SIGTERM. + if !replacementStartedBeforeOldEOF { + _ = Darwin.kill(stuckChild.processIdentifier, SIGKILL) + for _ in 0..<200 where children.remainingCount > 0 { + try await Task.sleep(nanoseconds: 10_000_000) + } + } + + // The retired child ignores SIGTERM, yet its stale stdout remains open. + // The queued successor must already run on a replacement; it cannot wait + // for either old-generation EOF or the force-kill grace period. + let payload = try await successor.value + #expect(String(decoding: payload, as: UTF8.self) == "replacement-2") + #expect(await clock.snapshot().isEmpty) + #expect(await clock.history() == [coldTimeoutNanoseconds, coldTimeoutNanoseconds]) + #expect(await graceClock.snapshot() == [terminationGraceNanoseconds]) + #expect(stuckChild.isRunning) + #expect(try String(contentsOfFile: pidsFile, encoding: .utf8).split(separator: "\n").count == 2) + + await graceClock.fireOldest() + for _ in 0..<200 where stuckChild.isRunning { + try await Task.sleep(nanoseconds: 10_000_000) + } + #expect(!stuckChild.isRunning) + #expect(stuckChild.terminationReason == .uncaughtSignal) + #expect(stuckChild.terminationStatus == SIGKILL) + await connection.shutdown() + } + + @Test("shutdown during the termination grace force-kills the SIGTERM-ignoring generation") + func shutdownDuringGraceKillsStubbornChild() async throws { + let dir = NSTemporaryDirectory() + "serve-connection-shutdown-grace-test-" + UUID().uuidString + try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(atPath: dir) } + let firstReadMarker = dir + "/first-read" + let clock = ManualTimeoutClock() + let graceClock = ManualTimeoutClock() + + let stuckChild = Process() + stuckChild.executableURL = URL(fileURLWithPath: "/bin/sh") + stuckChild.arguments = ["-c", """ + trap '' TERM + IFS= read -r line + : > "$1" + while :; do :; done + """, "serve-fixture", firstReadMarker] + + let children = ProcessQueue([stuckChild]) + let connection = ServeConnection( + makeProcess: { _, qualityOfService in + children.take(qualityOfService: qualityOfService) + }, + timeoutSleep: { nanoseconds in + try await clock.sleep(nanoseconds) + }, + terminationGraceSleep: { nanoseconds in + try await graceClock.sleep(nanoseconds) + } + ) + defer { + if stuckChild.isRunning { _ = Darwin.kill(stuckChild.processIdentifier, SIGKILL) } + } + + let request = Task { + try await connection.request(args: ["status", "--request", "stuck"]) + } + for _ in 0..<200 where !FileManager.default.fileExists(atPath: firstReadMarker) { + try await Task.sleep(nanoseconds: 10_000_000) + } + #expect(FileManager.default.fileExists(atPath: firstReadMarker)) + #expect(await clock.snapshot() == [coldTimeoutNanoseconds]) + + // Time out the request: the generation is retired and SIGTERM'd, and the + // SIGKILL escalation parks on the injected grace clock. + await clock.fireOldest() + do { + _ = try await request.value + #expect(Bool(false), "timed-out request unexpectedly succeeded") + } catch let error as ServeConnection.ServeRequestFailed { + #expect(error.message == "serve timeout") + } + for _ in 0..<200 where await graceClock.snapshot().isEmpty { + try await Task.sleep(nanoseconds: 10_000_000) + } + #expect(await graceClock.snapshot() == [terminationGraceNanoseconds]) + #expect(stuckChild.isRunning) // SIGTERM ignored; escalation still pending + + // Shutdown must not merely cancel the escalation. The retired generation + // is already detached from `process`, so nothing else will reap it; the + // grace task's cancellation path has to SIGKILL it or it outlives the app. + await connection.shutdown() + for _ in 0..<200 where stuckChild.isRunning { + try await Task.sleep(nanoseconds: 10_000_000) + } + #expect(!stuckChild.isRunning) + #expect(stuckChild.terminationReason == .uncaughtSignal) + #expect(stuckChild.terminationStatus == SIGKILL) + } + + @Test("timed-out generations consume one death each and stop at the resident budget") + func timeoutDeathBudgetIsExact() async throws { + let dir = NSTemporaryDirectory() + "serve-connection-timeout-budget-test-" + UUID().uuidString + try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(atPath: dir) } + let readsFile = dir + "/reads" + let clock = ManualTimeoutClock() + let processes = (0..<3).map { _ in + let child = Process() + child.executableURL = URL(fileURLWithPath: "/bin/sh") + child.arguments = ["-c", """ + trap '' TERM + IFS= read -r line + printf r >> "$1" + while :; do :; done + """, "serve-fixture", readsFile] + return child + } + let children = ProcessQueue(processes) + let connection = ServeConnection( + makeProcess: { _, qualityOfService in + children.take(qualityOfService: qualityOfService) + }, + timeoutSleep: { nanoseconds in + try await clock.sleep(nanoseconds) + }, + terminationGraceSleep: { _ in } + ) + defer { + for child in processes where child.isRunning { + _ = Darwin.kill(child.processIdentifier, SIGKILL) + } + } + + for attempt in 0..<3 { + let request = Task { + try await connection.request(args: ["status", "--attempt", String(attempt)]) + } + for _ in 0..<200 { + let reads = (try? String(contentsOfFile: readsFile, encoding: .utf8).count) ?? 0 + if reads == attempt + 1, await clock.snapshot().count == 1 { break } + try await Task.sleep(nanoseconds: 10_000_000) + } + #expect((try? String(contentsOfFile: readsFile, encoding: .utf8).count) == attempt + 1) + await clock.fireOldest() + do { + _ = try await request.value + Issue.record("timeout \(attempt) unexpectedly succeeded") + } catch let error as ServeConnection.ServeRequestFailed { + #expect(error.message == "serve timeout") + } + } + + #expect(children.remainingCount == 0) + do { + _ = try await connection.request(args: ["status", "--after-budget"]) + Issue.record("resident restarted after three timed-out generations") + } catch { + #expect(error is ServeConnection.ServeUnavailable) + } + for _ in 0..<200 where processes.contains(where: \.isRunning) { + try await Task.sleep(nanoseconds: 10_000_000) + } + #expect(processes.allSatisfy { !$0.isRunning }) + #expect(processes.allSatisfy { + $0.terminationReason == .uncaughtSignal && $0.terminationStatus == SIGKILL + }) + await connection.shutdown() + } + + @Test("external cancellations keep one child and safely discard late replies") + func cancellationsKeepResidentChildAlive() async throws { + let dir = NSTemporaryDirectory() + "serve-connection-cancel-reuse-test-" + UUID().uuidString + try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(atPath: dir) } + let pidsFile = dir + "/pids" + let requestsFile = dir + "/requests" + let lateRepliesFile = dir + "/late-replies" + + let connection = ServeConnection { _, qualityOfService in + let child = Process() + child.executableURL = URL(fileURLWithPath: "/bin/sh") + child.arguments = ["-c", """ + printf '%s\n' "$$" >> "$1" + while IFS= read -r line; do + printf r >> "$2" + id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/') + if [ "$id" -le 3 ]; then + sleep 0.05 + printf '{"id":%s,"ok":true,"output":"late-%s"}\n' "$id" "$id" + printf l >> "$3" + else + printf '{"id":%s,"ok":true,"output":"live-%s"}\n' "$id" "$id" + fi + done + """, "serve-fixture", pidsFile, requestsFile, lateRepliesFile] + child.qualityOfService = qualityOfService + return child + } + + for attempt in 0..<3 { + let request = Task { + try await connection.request(args: ["status", "--attempt", String(attempt)]) + } + for _ in 0..<200 { + let reads = (try? String(contentsOfFile: requestsFile, encoding: .utf8).count) ?? 0 + if reads >= attempt + 1 { break } + try await Task.sleep(nanoseconds: 10_000_000) + } + request.cancel() + do { + _ = try await request.value + #expect(Bool(false), "cancelled request unexpectedly succeeded") + } catch { + #expect(error is CancellationError) + } + + // The fake child deliberately emits the now-orphaned response after + // cancellation. It must be ignored without double-resuming anything, + // and the same resident child must remain available for the next id. + for _ in 0..<200 { + let replies = (try? String(contentsOfFile: lateRepliesFile, encoding: .utf8).count) ?? 0 + if replies >= attempt + 1 { break } + try await Task.sleep(nanoseconds: 10_000_000) + } + let replies = (try? String(contentsOfFile: lateRepliesFile, encoding: .utf8).count) ?? 0 + #expect(replies == attempt + 1) + } + + let finalPayload = try await connection.request(args: ["status", "--attempt", "final"]) + #expect(String(decoding: finalPayload, as: UTF8.self) == "live-4") + let pids = try String(contentsOfFile: pidsFile, encoding: .utf8) + .split(separator: "\n") + #expect(pids.count == 1) + #expect(try String(contentsOfFile: requestsFile, encoding: .utf8) == "rrrr") + #expect(try String(contentsOfFile: lateRepliesFile, encoding: .utf8) == "lll") + await connection.shutdown() + } + + @Test("cancelling a queued request never writes it or arms its timeout") + func queuedCancellationNeverReachesChild() async throws { + let dir = NSTemporaryDirectory() + "serve-connection-queued-cancel-test-" + UUID().uuidString + try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(atPath: dir) } + let requestsFile = dir + "/requests" + let releaseMarker = dir + "/release" + let recorder = TimeoutRecorder() + + let connection = ServeConnection( + makeProcess: { _, qualityOfService in + let child = Process() + child.executableURL = URL(fileURLWithPath: "/bin/sh") + child.arguments = ["-c", """ + count=0 + while IFS= read -r line; do + count=$((count + 1)) + printf '%s\n' "$line" >> "$1" + if [ "$count" -eq 1 ]; then + while [ ! -f "$2" ]; do sleep 0.01; done + fi + id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/') + printf '{"id":%s,"ok":true,"output":"served-%s"}\n' "$id" "$id" + done + """, "serve-fixture", requestsFile, releaseMarker] + child.qualityOfService = qualityOfService + return child + }, + timeoutSleep: { nanoseconds in + try await recorder.recordAndWait(nanoseconds) + } + ) + + let first = Task { try await connection.request(args: ["status", "first"]) } + for _ in 0..<200 where !(FileManager.default.fileExists(atPath: requestsFile)) { + try await Task.sleep(nanoseconds: 10_000_000) + } + let cancelled = Task { try await connection.request(args: ["status", "cancelled"]) } + let third = Task { try await connection.request(args: ["status", "third"]) } + try await Task.sleep(nanoseconds: 100_000_000) + cancelled.cancel() + do { + _ = try await cancelled.value + Issue.record("queued cancellation unexpectedly succeeded") + } catch { + #expect(error is CancellationError) + } + #expect(await recorder.snapshot() == [coldTimeoutNanoseconds]) + + _ = FileManager.default.createFile(atPath: releaseMarker, contents: Data()) + #expect(String(decoding: try await first.value, as: UTF8.self) == "served-1") + #expect(String(decoding: try await third.value, as: UTF8.self) == "served-2") + let requests = try String(contentsOfFile: requestsFile, encoding: .utf8) + #expect(requests.contains("first")) + #expect(requests.contains("third")) + #expect(!requests.contains("cancelled")) + #expect(await recorder.snapshot() == [coldTimeoutNanoseconds, warmTimeoutNanoseconds]) + await connection.shutdown() + } + + @Test("shutdown fails the active request and every client-side queued request") + func shutdownDrainsClientQueue() async throws { + let dir = NSTemporaryDirectory() + "serve-connection-shutdown-queue-test-" + UUID().uuidString + try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(atPath: dir) } + let requestMarker = dir + "/request-read" + let recorder = TimeoutRecorder() + let connection = ServeConnection( + makeProcess: { _, qualityOfService in + let child = Process() + child.executableURL = URL(fileURLWithPath: "/bin/sh") + child.arguments = ["-c", "IFS= read -r line; : > \"$1\"; sleep 5", "serve-fixture", requestMarker] + child.qualityOfService = qualityOfService + return child + }, + timeoutSleep: { nanoseconds in + try await recorder.recordAndWait(nanoseconds) + } + ) + + let active = Task { try await connection.request(args: ["status", "active"]) } + for _ in 0..<200 where !FileManager.default.fileExists(atPath: requestMarker) { + try await Task.sleep(nanoseconds: 10_000_000) + } + let queued = Task { try await connection.request(args: ["status", "queued"]) } + try await Task.sleep(nanoseconds: 100_000_000) + #expect(await recorder.snapshot() == [coldTimeoutNanoseconds]) + await connection.shutdown() + + for request in [active, queued] { + do { + _ = try await request.value + Issue.record("shutdown request unexpectedly succeeded") + } catch { + #expect(error is ServeConnection.ServeRequestFailed) + } + } + } + + @Test("late stdout from a replaced child cannot corrupt or warm its replacement") + func staleGenerationStdoutIsDiscarded() async throws { + let oldChild = Process() + oldChild.executableURL = URL(fileURLWithPath: "/bin/sh") + oldChild.arguments = ["-c", "IFS= read -r line; sleep 0.1; exit 1"] + + let newChild = Process() + newChild.executableURL = URL(fileURLWithPath: "/bin/sh") + newChild.arguments = ["-c", """ + while IFS= read -r line; do + id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/') + printf '{"id":%s,"ok":true,"output":"new-%s"}\n' "$id" "$id" + done + """] + + let children = ProcessQueue([oldChild, newChild]) + let recorder = TimeoutRecorder() + let connection = ServeConnection( + makeProcess: { _, qualityOfService in + children.take(qualityOfService: qualityOfService) + }, + timeoutSleep: { nanoseconds in + try await recorder.recordAndWait(nanoseconds) + } + ) + + // The admitted read survives the old generation's crash and retries + // on the replacement. Request id 1 belonged to the old child; the + // replacement receives id 2. + let retried = try await connection.request(args: ["status", "--generation", "old"]) + #expect(String(decoding: retried, as: UTF8.self) == "new-2") + + // Model both harmful trailing shapes after the replacement owns the + // connection: a complete terminal would incorrectly select the warm + // timeout, while a fragment would corrupt the replacement's first line. + await connection.consume( + Data("{\"id\":1,\"ok\":true,\"output\":\"late-old\"}\n".utf8), + from: oldChild + ) + await connection.consume(Data("{\"id\":1".utf8), from: oldChild) + + let payload = try await connection.request(args: ["status", "--generation", "new"]) + + #expect(String(decoding: payload, as: UTF8.self) == "new-3") + #expect(await recorder.snapshot() == [ + coldTimeoutNanoseconds, + coldTimeoutNanoseconds, + warmTimeoutNanoseconds, + ]) + #expect(children.remainingCount == 0) + await connection.shutdown() + } + + @Test("queued requests arm their warm timeout only after cold hydration finishes") + func coldAndWarmTimeoutSelection() async throws { + let dir = NSTemporaryDirectory() + "serve-connection-timeout-test-" + UUID().uuidString + try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(atPath: dir) } + let releaseMarker = dir + "/release-cold-responses" + let recorder = TimeoutRecorder() + + let connection = ServeConnection( + makeProcess: { _, qualityOfService in + let child = Process() + child.executableURL = URL(fileURLWithPath: "/bin/sh") + child.arguments = ["-c", """ + IFS= read -r first + while [ ! -f "$1" ]; do sleep 0.01; done + for slot in first second third; do + if [ "$slot" = first ]; then line="$first"; else IFS= read -r line; fi + id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/') + printf '{"id":%s,"ok":true,"output":"served"}\\n' "$id" + done + """, "serve-fixture", releaseMarker] + child.qualityOfService = qualityOfService + return child + }, + timeoutSleep: { nanoseconds in + try await recorder.recordAndWait(nanoseconds) + } + ) + + let first = Task { try await connection.request(args: ["status", "--request", "one"]) } + let second = Task { try await connection.request(args: ["status", "--request", "two"]) } + try await Task.sleep(nanoseconds: 100_000_000) + #expect(await recorder.snapshot() == [coldTimeoutNanoseconds]) + + _ = FileManager.default.createFile(atPath: releaseMarker, contents: Data()) + let firstPayload = try await first.value + let secondPayload = try await second.value + #expect(String(decoding: firstPayload, as: UTF8.self) == "served") + #expect(String(decoding: secondPayload, as: UTF8.self) == "served") + + let thirdPayload = try await connection.request(args: ["status", "--request", "three"]) + #expect(String(decoding: thirdPayload, as: UTF8.self) == "served") + let allSelections = await recorder.snapshot() + #expect(allSelections == [ + coldTimeoutNanoseconds, + warmTimeoutNanoseconds, + warmTimeoutNanoseconds, + ]) + await connection.shutdown() + } + + @Test("a failed terminal response does not mark the resident child warm") + func failedTerminalResponseKeepsColdTimeout() async throws { + let recorder = TimeoutRecorder() + let connection = ServeConnection( + makeProcess: { _, qualityOfService in + let child = Process() + child.executableURL = URL(fileURLWithPath: "/bin/sh") + child.arguments = ["-c", """ + count=0 + while IFS= read -r line; do + count=$((count + 1)) + id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/') + if [ "$count" -eq 1 ]; then + printf '{"id":%s,"ok":false,"error":"cold failure"}\\n' "$id" + else + printf '{"id":%s,"ok":true,"output":"served-%s"}\\n' "$id" "$count" + fi + done + """] + child.qualityOfService = qualityOfService + return child + }, + timeoutSleep: { nanoseconds in + try await recorder.recordAndWait(nanoseconds) + } + ) + + do { + _ = try await connection.request(args: ["status", "--request", "failed"]) + #expect(Bool(false), "failed response unexpectedly succeeded") + } catch { + #expect(error is ServeConnection.ServeRequestFailed) + } + + let second = try await connection.request(args: ["status", "--request", "cold-success"]) + let third = try await connection.request(args: ["status", "--request", "warm-success"]) + + #expect(String(decoding: second, as: UTF8.self) == "served-2") + #expect(String(decoding: third, as: UTF8.self) == "served-3") + #expect(await recorder.snapshot() == [ + coldTimeoutNanoseconds, + coldTimeoutNanoseconds, + warmTimeoutNanoseconds, + ]) + await connection.shutdown() + } + + @Test("an actual stdout flood is bounded and the next generation stays healthy") + func oversizedFrameTerminatesOnlyItsGeneration() async throws { + let oldChild = Process() + oldChild.executableURL = URL(fileURLWithPath: "/bin/sh") + oldChild.arguments = ["-c", """ + IFS= read -r line + dd if=/dev/zero bs=1024 count=1 2>/dev/null | tr '\\0' x + sleep 5 + """] + + let replacement = Process() + replacement.executableURL = URL(fileURLWithPath: "/bin/sh") + replacement.arguments = ["-c", """ + IFS= read -r line + id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/') + printf '{"id":%s,"ok":true,"output":"replacement"}\\n' "$id" + """] + + let children = ProcessQueue([oldChild, replacement]) + let recorder = TimeoutRecorder() + let connection = ServeConnection( + makeProcess: { _, qualityOfService in + children.take(qualityOfService: qualityOfService) + }, + timeoutSleep: { nanoseconds in + try await recorder.recordAndWait(nanoseconds) + }, + responseLimitBytes: 128 + ) + + do { + _ = try await connection.request(args: ["status", "--oversized"]) + #expect(Bool(false), "oversized resident frame unexpectedly succeeded") + } catch let error as ServeConnection.ServeRequestFailed { + #expect(error.reason == .outputTooLarge) + } + + await connection.ensureStarted() + let payload = try await connection.request(args: ["status", "--replacement"]) + #expect(String(decoding: payload, as: UTF8.self) == "replacement") + #expect(children.remainingCount == 0) + await connection.shutdown() + } + + @Test("an unterminated frame and cumulative progress cannot bypass the resident limit") + func partialAndCumulativeFramesAreBounded() async throws { + for mode in ["partial", "progress"] { + let child = Process() + child.executableURL = URL(fileURLWithPath: "/bin/sh") + child.arguments = ["-c", "IFS= read -r line; sleep 5"] + let recorder = TimeoutRecorder() + let connection = ServeConnection( + makeProcess: { _, qualityOfService in + child.qualityOfService = qualityOfService + return child + }, + timeoutSleep: { nanoseconds in + try await recorder.recordAndWait(nanoseconds) + }, + responseLimitBytes: 128 + ) + let request = Task { try await connection.request(args: ["status", "--mode", mode]) } + for _ in 0..<200 { + if await recorder.snapshot().count == 1 { break } + try await Task.sleep(nanoseconds: 10_000_000) + } + + if mode == "partial" { + await connection.consume(Data(repeating: UInt8(ascii: "x"), count: 129), from: child) + } else { + let progress = String(repeating: "p", count: 70) + let frame = Data("{\"id\":1,\"progress\":\"\(progress)\"}\n".utf8) + #expect(frame.count < 128) + await connection.consume(frame, from: child) + await connection.consume(frame, from: child) + } + + do { + _ = try await request.value + #expect(Bool(false), "\(mode) overflow unexpectedly succeeded") + } catch let error as ServeConnection.ServeRequestFailed { + #expect(error.reason == .outputTooLarge) + } + await connection.shutdown() + } + } + + @Test("a cancelled request keeps its cumulative progress bound until the child finishes") + func cancelledRequestStillBoundsOrphanProgress() async throws { + let oldChild = Process() + oldChild.executableURL = URL(fileURLWithPath: "/bin/sh") + oldChild.arguments = ["-c", "IFS= read -r line; sleep 5"] + + let replacement = Process() + replacement.executableURL = URL(fileURLWithPath: "/bin/sh") + replacement.arguments = ["-c", """ + IFS= read -r line + id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/') + printf '{"id":%s,"ok":true,"output":"healthy"}\\n' "$id" + """] + + let children = ProcessQueue([oldChild, replacement]) + let recorder = TimeoutRecorder() + let connection = ServeConnection( + makeProcess: { _, qualityOfService in + children.take(qualityOfService: qualityOfService) + }, + timeoutSleep: { nanoseconds in + try await recorder.recordAndWait(nanoseconds) + }, + responseLimitBytes: 128 + ) + + let abandoned = Task { try await connection.request(args: ["status", "--abandoned"]) } + for _ in 0..<200 { + if await recorder.snapshot().count == 1 { break } + try await Task.sleep(nanoseconds: 10_000_000) + } + abandoned.cancel() + do { + _ = try await abandoned.value + Issue.record("cancelled request unexpectedly succeeded") + } catch { + #expect(error is CancellationError) + } + + let progress = String(repeating: "p", count: 70) + let frame = Data("{\"id\":1,\"progress\":\"\(progress)\"}\n".utf8) + await connection.consume(frame, from: oldChild) + await connection.consume(frame, from: oldChild) + + await connection.ensureStarted() + #expect(children.remainingCount == 0) + let payload = try await connection.request(args: ["status", "--replacement"]) + #expect(String(decoding: payload, as: UTF8.self) == "healthy") + await connection.shutdown() + } + + @Test("each overflow consumes exactly one resident death") + func overflowDeathBudgetIsExact() async throws { + let processes = (0..<3).map { _ in + let child = Process() + child.executableURL = URL(fileURLWithPath: "/bin/sh") + child.arguments = ["-c", "IFS= read -r line; sleep 5"] + return child + } + let children = ProcessQueue(processes) + let recorder = TimeoutRecorder() + let connection = ServeConnection( + makeProcess: { _, qualityOfService in + children.take(qualityOfService: qualityOfService) + }, + timeoutSleep: { nanoseconds in + try await recorder.recordAndWait(nanoseconds) + }, + responseLimitBytes: 64 + ) + + for attempt in 0..<3 { + let request = Task { try await connection.request(args: ["status", "--attempt", "\(attempt)"]) } + for _ in 0..<200 { + if await recorder.snapshot().count == attempt + 1 { break } + try await Task.sleep(nanoseconds: 10_000_000) + } + await connection.consume(Data(repeating: UInt8(ascii: "x"), count: 65), from: processes[attempt]) + do { + _ = try await request.value + Issue.record("overflow \(attempt) unexpectedly succeeded") + } catch let error as ServeConnection.ServeRequestFailed { + #expect(error.reason == .outputTooLarge) + } + } + + #expect(children.remainingCount == 0) + do { + _ = try await connection.request(args: ["status", "--after-budget"]) + Issue.record("resident restarted after exhausting its death budget") + } catch { + #expect(error is ServeConnection.ServeUnavailable) + } + await connection.shutdown() + } + + @Test("output overflow is not eligible for a one-shot fallback") + func outputOverflowIsTerminalForDataClient() async { + let overflow = ServeConnection.ServeRequestFailed( + message: "too large", + reason: .outputTooLarge + ) + let fallback = FallbackRecorder() + do { + _ = try await DataClient.runCLI( + subcommand: ["status", "--format", "menubar-json"], + serveRequest: { _ in throw overflow }, + spawnFallback: { + await fallback.record() + return DataClient.ProcessResult(stdout: Data(), stderr: "", exitCode: 0) + } + ) + Issue.record("output overflow unexpectedly fell back or succeeded") + } catch DataClientError.outputTooLarge { + // Expected: the one-shot closure must remain untouched. + } catch { + Issue.record("unexpected terminal error: \(error)") + } + #expect(await fallback.snapshot() == 0) + + let ordinary = ServeConnection.ServeRequestFailed(message: "serve exited") + do { + let result = try await DataClient.runCLI( + subcommand: ["status", "--format", "menubar-json"], + serveRequest: { _ in throw ordinary }, + spawnFallback: { + await fallback.record() + return DataClient.ProcessResult(stdout: Data("fallback".utf8), stderr: "", exitCode: 0) + } + ) + #expect(String(decoding: result.stdout, as: UTF8.self) == "fallback") + } catch { + Issue.record("ordinary serve failure did not use fallback: \(error)") + } + #expect(await fallback.snapshot() == 1) + } + + @Test("the first real request is the only cold-start query") + func firstRequestIsTheWarmup() async throws { + let dir = NSTemporaryDirectory() + "serve-connection-test-" + UUID().uuidString + try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(atPath: dir) } + let requestLog = dir + "/requests.log" + + let connection = ServeConnection { _, qualityOfService in + let child = Process() + child.executableURL = URL(fileURLWithPath: "/bin/sh") + child.arguments = ["-c", """ + while IFS= read -r line; do + printf 'request\\n' >> "$1" + id=$(printf '%s' "$line" | sed -E 's/.*\"id\":([0-9]+).*/\\1/') + printf '{\"id\":%s,\"progress\":\"scanning\"}\\n' "$id" + printf '{\"id\":%s,\"ok\":true,\"output\":\"served\"}\\n' "$id" + # Emit READY after the terminal response. The client must + # register and complete the first real request without it. + printf '{\"ready\":true,\"pid\":1}\\n' + done + """, "serve-fixture", requestLog] + child.qualityOfService = qualityOfService + return child + } + + await connection.ensureStarted() + let payload = try await connection.request(args: ["status", "--format", "menubar-json"]) + + #expect(String(decoding: payload, as: UTF8.self) == "served") + let requests = try String(contentsOfFile: requestLog, encoding: .utf8) + .split(separator: "\n") + #expect(requests.count == 1) + await connection.shutdown() + } + + @Test("split terminal bytes are drained before child death and the next generation stays clean") + func finalStdoutDrainPrecedesTermination() async throws { + let first = Process() + first.executableURL = URL(fileURLWithPath: "/bin/sh") + first.arguments = ["-c", """ + IFS= read -r line + id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/') + printf '{"id":%s,"ok":true,' "$id" + printf '"output":"final-drain"}\n' + """] + + let replacement = Process() + replacement.executableURL = URL(fileURLWithPath: "/bin/sh") + replacement.arguments = ["-c", """ + while IFS= read -r line; do + id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/') + printf '{"id":%s,"ok":true,"output":"replacement"}\n' "$id" + done + """] + + let children = ProcessQueue([first, replacement]) + let connection = ServeConnection { _, qualityOfService in + children.take(qualityOfService: qualityOfService) + } + + let drained = try await connection.request(args: ["status", "drain"]) + #expect(String(decoding: drained, as: UTF8.self) == "final-drain") + let next = try await connection.request(args: ["status", "next"]) + #expect(String(decoding: next, as: UTF8.self) == "replacement") + #expect(children.remainingCount == 0) + await connection.shutdown() + } + + @Test("a child that closes stdin fails the request without terminating the app") + func closedChildStdinDoesNotRaiseSIGPIPE() async throws { + let dir = NSTemporaryDirectory() + "serve-connection-sigpipe-test-" + UUID().uuidString + try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(atPath: dir) } + let closedMarker = dir + "/stdin-closed" + let sigpipeHandlerBefore = currentSIGPIPEHandlerBits() + + let connection = ServeConnection { _, qualityOfService in + let child = Process() + child.executableURL = URL(fileURLWithPath: "/bin/sh") + child.arguments = ["-c", "exec 0<&-; : > \"$1\"; sleep 2", "serve-fixture", closedMarker] + child.qualityOfService = qualityOfService + return child + } + + await connection.ensureStarted() + #expect(currentSIGPIPEHandlerBits() == sigpipeHandlerBefore) + #expect(currentSIGPIPEHandlerBits() != ignoredSIGPIPEHandlerBits) + for _ in 0..<200 where !FileManager.default.fileExists(atPath: closedMarker) { + try await Task.sleep(nanoseconds: 10_000_000) + } + #expect(FileManager.default.fileExists(atPath: closedMarker)) + + var requestFailed = false + do { + _ = try await connection.request(args: ["status", "--format", "menubar-json"]) + } catch { + requestFailed = true + } + #expect(requestFailed) + await connection.shutdown() + } +} diff --git a/package.json b/package.json index 0137c62b..7d1b7e8d 100644 --- a/package.json +++ b/package.json @@ -8,7 +8,8 @@ "codeburn": "dist/cli.js" }, "files": [ - "dist" + "dist", + "!dist/parse-worker.js.map" ], "scripts": { "bundle-litellm": "node scripts/bundle-litellm.mjs", diff --git a/src/antigravity-statusline.ts b/src/antigravity-statusline.ts index 15f49093..27b24067 100644 --- a/src/antigravity-statusline.ts +++ b/src/antigravity-statusline.ts @@ -3,6 +3,7 @@ import { randomBytes } from 'crypto' import { dirname, join } from 'path' import { homedir } from 'os' +import { getCodeburnCacheDir } from './cache-dir.js' import { recordAntigravityStatusLinePayload, snapshotAntigravityStatusLinePayload, @@ -54,12 +55,8 @@ function settingsPath(): string { ?? join(homedir(), '.gemini', 'antigravity-cli', 'settings.json') } -function codeburnCacheDir(): string { - return process.env['CODEBURN_CACHE_DIR'] ?? join(homedir(), '.cache', 'codeburn') -} - function previousStatusLinePath(): string { - return join(codeburnCacheDir(), 'antigravity-statusline-previous.json') + return join(getCodeburnCacheDir(), 'antigravity-statusline-previous.json') } async function readSettings(): Promise { diff --git a/src/bash-utils.ts b/src/bash-utils.ts index b6388a95..55fc50aa 100644 --- a/src/bash-utils.ts +++ b/src/bash-utils.ts @@ -1,6 +1,8 @@ import { basename } from 'path' import stripAnsi from 'strip-ansi' +const WHITESPACE = /\s/ + function stripQuotedStrings(command: string): string { return command.replace(/"[^"]*"|'[^']*'/g, match => ' '.repeat(match.length)) } @@ -19,12 +21,22 @@ export function extractBashCommands(rawCommand: string): string[] { const command = stripAnsi(rawCommand) const stripped = stripQuotedStrings(command) - const separatorRegex = /\s*(?:&&|;|\|)\s*/g + // Match the separator alone, then widen over surrounding whitespace by hand. + // /\s*(?:&&|;|\|)\s*/ retried its leading \s* from every offset, quadratic on + // long whitespace-heavy commands. Widening is required (not cosmetic): stripQuotedStrings + // blanks quoted text, and segments are sliced from the original string. + const separatorRegex = /(?:&&|;|\|)/g const separators: Array<{ start: number; end: number }> = [] let match: RegExpExecArray | null while ((match = separatorRegex.exec(stripped)) !== null) { - separators.push({ start: match.index, end: match.index + match[0].length }) + let start = match.index + while (start > 0 && WHITESPACE.test(stripped[start - 1]!)) start-- + let end = match.index + match[0].length + while (end < stripped.length && WHITESPACE.test(stripped[end]!)) end++ + const prevEnd = separators[separators.length - 1]?.end ?? 0 + separators.push({ start: Math.max(start, prevEnd), end }) + separatorRegex.lastIndex = end } const ranges: Array<[number, number]> = [] @@ -93,7 +105,7 @@ const GIT_READ_SUBCOMMANDS = new Set([ export function isReadShapedBashCommand(rawCommand: string): boolean { if (!rawCommand || !rawCommand.trim()) return false const stripped = stripQuotedStrings(stripAnsi(rawCommand)) - const segments = stripped.split(/\s*(?:&&|;|\|)\s*/) + const segments = stripped.split(/(?:&&|;|\|)/) let sawCommand = false for (const segment of segments) { const trimmed = segment.trim() diff --git a/src/cache-dir.ts b/src/cache-dir.ts new file mode 100644 index 00000000..a202be05 --- /dev/null +++ b/src/cache-dir.ts @@ -0,0 +1,13 @@ +import { homedir } from 'os' +import { join } from 'path' + +/** + * Resolve CodeBurn's shared cache directory at call time. + * + * Reading the environment on every call matters for embedded consumers and + * tests that change CODEBURN_CACHE_DIR after importing the CLI modules. + */ +export function getCodeburnCacheDir(): string { + const override = process.env['CODEBURN_CACHE_DIR'] + return override?.trim() ? override : join(homedir(), '.cache', 'codeburn') +} diff --git a/src/cache-refresh-lock.ts b/src/cache-refresh-lock.ts index 58faf281..f467886f 100644 --- a/src/cache-refresh-lock.ts +++ b/src/cache-refresh-lock.ts @@ -1,9 +1,10 @@ import { createHash, randomBytes } from 'crypto' import { existsSync } from 'fs' import { mkdir, open, readFile, stat, unlink, utimes, writeFile } from 'fs/promises' -import { homedir } from 'os' import { join } from 'path' +import { getCodeburnCacheDir } from './cache-dir.js' + const LOCK_FILE = 'session-refresh.lock' const TAKEOVER_FILE = `${LOCK_FILE}.takeover` const DEFAULT_HEARTBEAT_MS = 10_000 @@ -46,10 +47,6 @@ const defaultClock: RefreshLockClock = { wallNow: () => Date.now(), } -function defaultCacheDir(): string { - return process.env['CODEBURN_CACHE_DIR'] ?? join(homedir(), '.cache', 'codeburn') -} - function delay(ms: number): Promise { return new Promise(resolve => { setTimeout(resolve, ms) }) } @@ -197,7 +194,7 @@ export async function acquireCacheRefreshLock(options: RefreshLockOptions = {}): leaveSingleFlight() } - const cacheDir = options.cacheDir ?? defaultCacheDir() + const cacheDir = options.cacheDir ?? getCodeburnCacheDir() const clock = options.clock ?? defaultClock const heartbeatMs = options.heartbeatMs ?? DEFAULT_HEARTBEAT_MS const staleMs = options.staleMs ?? DEFAULT_STALE_MS diff --git a/src/codex-cache.ts b/src/codex-cache.ts index 6146e8e9..ca3f306a 100644 --- a/src/codex-cache.ts +++ b/src/codex-cache.ts @@ -1,9 +1,10 @@ import { readFile, mkdir, stat, open, rename, unlink } from 'fs/promises' import { existsSync } from 'fs' import { randomBytes } from 'crypto' -import { join } from 'path' -import { homedir } from 'os' +import { join, resolve } from 'path' +import { AsyncLocalStorage } from 'node:async_hooks' +import { getCodeburnCacheDir } from './cache-dir.js' import type { ParsedProviderCall } from './providers/types.js' // v4: attribute MCP calls emitted as event_msg/mcp_tool_call_end (issue #478). @@ -14,45 +15,85 @@ import type { ParsedProviderCall } from './providers/types.js' // v6/v7: rich-session-capture — per-call locAdded/locRemoved/editFailed from // patch_apply_end. Sessions cached under v5 lack these fields; re-parse to add. // v8: persist native MCP timing and compact invocation attribution. +// Deliberately NOT bumped for the resume fields (dev/ino + resumeOffset/ +// resumeState): they are additive and absence-safe in both directions, so a +// bump would only throw away a warm multi-hundred-MB cache to gain nothing. An +// entry without them simply re-parses in full once and gains them. const CODEX_CACHE_VERSION = 8 const CACHE_FILE = 'codex-results.json' -type FileFingerprint = { mtimeMs: number; sizeBytes: number } +export type CodexFileFingerprint = { dev: number; ino: number; mtimeMs: number; sizeBytes: number } +type FileFingerprint = CodexFileFingerprint type FileEntry = { + // Absent on entries written before the resume support landed. + dev?: number + ino?: number mtimeMs: number sizeBytes: number project: string calls: ParsedProviderCall[] + /** Byte offset of a complete-line boundary the parser can restart from. */ + resumeOffset?: number + /** Opaque parser state captured at `resumeOffset` (shape owned by the Codex parser). */ + resumeState?: unknown + /** How many of `calls` were decoded before `resumeOffset`. */ + resumeCallCount?: number } +/** An exact fingerprint match, or an append the parser can resume into. */ +export type CodexCacheHit = + | { kind: 'exact'; calls: ParsedProviderCall[] } + | { kind: 'resume'; calls: ParsedProviderCall[]; offset: number; state: unknown; callCount: number } + type ResultCache = { version: number files: Record } -function getCacheDir(): string { - return process.env['CODEBURN_CACHE_DIR'] ?? join(homedir(), '.cache', 'codeburn') +const cacheDirContext = new AsyncLocalStorage() + +function currentCacheDir(): string { + return cacheDirContext.getStore() ?? resolve(getCodeburnCacheDir()) } -function getCachePath(): string { - return join(getCacheDir(), CACHE_FILE) +// A parse can cross many async boundaries before the Codex provider publishes +// its incremental cache. Embedded hosts are allowed to change the process env +// between calls, so pin the call-time directory for the whole transaction +// instead of re-reading CODEBURN_CACHE_DIR at each cache operation. +export function withCodexCacheDirectory(cacheDir: string, operation: () => T): T { + return cacheDirContext.run(resolve(cacheDir), operation) } -let memCache: ResultCache | null = null +function getCachePath(cacheDir: string): string { + return join(cacheDir, CACHE_FILE) +} -async function loadCache(): Promise { - if (memCache) return memCache +// Embedded consumers can change CODEBURN_CACHE_DIR without reloading this +// module. Keep each directory's in-memory state separate so a warm cache (or an +// unflushed update) from A can never be read from or written into B. +const memCaches = new Map() + +// Dropped by the resident RSS guard. Every write is published by +// flushCodexCache() in the parse's finally, so the next load re-reads disk. +export function clearCodexMemCaches(): void { + memCaches.clear() +} + +async function loadCache(cacheDir: string): Promise { + const inMemory = memCaches.get(cacheDir) + if (inMemory) return inMemory try { - const raw = await readFile(getCachePath(), 'utf-8') + const raw = await readFile(getCachePath(cacheDir), 'utf-8') const cache = JSON.parse(raw) as ResultCache if (cache.version === CODEX_CACHE_VERSION && cache.files && typeof cache.files === 'object') { - memCache = cache + memCaches.set(cacheDir, cache) return cache } } catch {} - memCache = { version: CODEX_CACHE_VERSION, files: {} } - return memCache + const empty = { version: CODEX_CACHE_VERSION, files: {} } + memCaches.set(cacheDir, empty) + return empty } function getEntry(cache: ResultCache, filePath: string, fp: FileFingerprint): FileEntry | null { @@ -64,14 +105,51 @@ function getEntry(cache: ResultCache, filePath: string, fp: FileFingerprint): Fi return null } +// A grown file is only assumed to be an APPEND if the recorded boundary still +// falls right after a newline. A same-inode rewrite (truncate + refill, or an +// in-place edit) that happens to end up larger would otherwise resume into the +// middle of an unrelated line. Reading one byte is cheaper than being wrong. +async function endsLineAt(filePath: string, offset: number): Promise { + if (offset === 0) return true + try { + const handle = await open(filePath, 'r') + try { + const buf = Buffer.alloc(1) + const { bytesRead } = await handle.read(buf, 0, 1, offset - 1) + return bytesRead === 1 && buf[0] === 0x0a + } finally { + await handle.close() + } + } catch { + return false + } +} + export async function readCachedCodexResults( filePath: string, -): Promise { +): Promise { try { const s = await stat(filePath) - const cache = await loadCache() - const entry = getEntry(cache, filePath, { mtimeMs: s.mtimeMs, sizeBytes: s.size }) - return entry?.calls ?? null + const cache = await loadCache(currentCacheDir()) + const fp = { dev: s.dev, ino: s.ino, mtimeMs: s.mtimeMs, sizeBytes: s.size } + const entry = getEntry(cache, filePath, fp) + if (entry) return { kind: 'exact', calls: entry.calls } + // Rollouts are append-only: the same inode, grown past a boundary we + // recorded, can be picked up from that boundary instead of re-read whole. + const stale = cache.files[filePath] + if ( + stale + && stale.dev === fp.dev + && stale.ino === fp.ino + && stale.resumeOffset !== undefined + && stale.resumeState !== undefined + && stale.resumeCallCount !== undefined + && fp.sizeBytes > stale.sizeBytes + && stale.resumeOffset <= fp.sizeBytes + && await endsLineAt(filePath, stale.resumeOffset) + ) { + return { kind: 'resume', calls: stale.calls, offset: stale.resumeOffset, state: stale.resumeState, callCount: stale.resumeCallCount } + } } catch {} return null } @@ -81,8 +159,8 @@ export async function getCachedCodexProject( ): Promise { try { const s = await stat(filePath) - const cache = await loadCache() - const entry = getEntry(cache, filePath, { mtimeMs: s.mtimeMs, sizeBytes: s.size }) + const cache = await loadCache(currentCacheDir()) + const entry = getEntry(cache, filePath, { dev: s.dev, ino: s.ino, mtimeMs: s.mtimeMs, sizeBytes: s.size }) return entry?.project ?? null } catch {} return null @@ -93,7 +171,7 @@ export async function fingerprintFile( ): Promise { try { const s = await stat(filePath) - return { mtimeMs: s.mtimeMs, sizeBytes: s.size } + return { dev: s.dev, ino: s.ino, mtimeMs: s.mtimeMs, sizeBytes: s.size } } catch { return null } @@ -104,19 +182,25 @@ export async function writeCachedCodexResults( project: string, calls: ParsedProviderCall[], fingerprint: FileFingerprint, + resume?: { offset: number; state: unknown; callCount: number }, ): Promise { try { - const cache = await loadCache() + const cache = await loadCache(currentCacheDir()) cache.files[filePath] = { + dev: fingerprint.dev, + ino: fingerprint.ino, mtimeMs: fingerprint.mtimeMs, sizeBytes: fingerprint.sizeBytes, project, calls, + ...(resume ? { resumeOffset: resume.offset, resumeState: resume.state, resumeCallCount: resume.callCount } : {}), } } catch {} } export async function flushCodexCache(): Promise { + const cacheDir = currentCacheDir() + const memCache = memCaches.get(cacheDir) if (!memCache) return try { // Evict entries for files that no longer exist on disk @@ -129,9 +213,8 @@ export async function flushCodexCache(): Promise { } } - const dir = getCacheDir() - if (!existsSync(dir)) await mkdir(dir, { recursive: true }) - const finalPath = getCachePath() + if (!existsSync(cacheDir)) await mkdir(cacheDir, { recursive: true }) + const finalPath = getCachePath(cacheDir) const tempPath = `${finalPath}.${randomBytes(8).toString('hex')}.tmp` const payload = JSON.stringify(memCache) const handle = await open(tempPath, 'w', 0o600) diff --git a/src/content-utils.ts b/src/content-utils.ts index 5ca31ae9..0c41afb6 100644 --- a/src/content-utils.ts +++ b/src/content-utils.ts @@ -24,3 +24,35 @@ export function normalizeContentBlocks { @@ -111,7 +104,7 @@ async function loadCachedRate(code: string): Promise { } async function cacheRate(code: string, rate: number): Promise { - await mkdir(getCacheDir(), { recursive: true }) + await mkdir(getCodeburnCacheDir(), { recursive: true }) await writeFile(getRateCachePath(), JSON.stringify({ timestamp: Date.now(), code, rate })) } @@ -138,7 +131,13 @@ async function getExchangeRate(code: string): Promise { export async function loadCurrency(): Promise { const config = await readConfig() - if (!config.currency) return + if (!config.currency) { + // A long-lived `serve` process may previously have loaded a non-USD + // currency. Removing the config entry is the USD reset contract, so reset + // the module state as well as letting the output memo invalidate. + active = USD + return + } const code = config.currency.code.toUpperCase() const rate = await getExchangeRate(code) diff --git a/src/cursor-cache.ts b/src/cursor-cache.ts index 28a7820f..d48ca33c 100644 --- a/src/cursor-cache.ts +++ b/src/cursor-cache.ts @@ -1,8 +1,8 @@ import { readFile, writeFile, mkdir, rename, stat, unlink } from 'fs/promises' import { join } from 'path' -import { homedir } from 'os' import { randomBytes } from 'crypto' +import { getCodeburnCacheDir } from './cache-dir.js' import type { ParsedProviderCall } from './providers/types.js' // Bumped to 3 for the workspace-aware breakdown change: the cursor parser @@ -31,12 +31,8 @@ type ResultCache = { const CACHE_FILE = 'cursor-results.json' -function getCacheDir(): string { - return join(homedir(), '.cache', 'codeburn') -} - function getCachePath(): string { - return join(getCacheDir(), CACHE_FILE) + return join(getCodeburnCacheDir(), CACHE_FILE) } async function getDbFingerprint(dbPath: string): Promise<{ mtimeMs: number; size: number } | null> { @@ -86,7 +82,7 @@ export async function writeCachedResults( const fp = await getDbFingerprint(dbPath) if (!fp) return - const dir = getCacheDir() + const dir = getCodeburnCacheDir() await mkdir(dir, { recursive: true }).catch(() => {}) const cache: ResultCache = { version: CURSOR_CACHE_VERSION, diff --git a/src/daily-cache.ts b/src/daily-cache.ts index 7abb445c..76e787f0 100644 --- a/src/daily-cache.ts +++ b/src/daily-cache.ts @@ -1,8 +1,9 @@ import { randomBytes } from 'crypto' import { existsSync } from 'fs' import { mkdir, open, readdir, readFile, rename, stat, unlink } from 'fs/promises' -import { homedir } from 'os' import { join } from 'path' + +import { getCodeburnCacheDir } from './cache-dir.js' import type { DateRange, ProjectSummary } from './types.js' // Bumped to 17: copilot CLI sessions were misclassified as VS Code transcripts @@ -176,10 +177,6 @@ export type DailyCache = { watermarkTrusted?: boolean } -function getCacheDir(): string { - return process.env['CODEBURN_CACHE_DIR'] ?? join(homedir(), '.cache', 'codeburn') -} - /** IANA name of the current local timezone (respects the TZ env var). Days are * bucketed by local midnight, so this tags the cache for TZ-change invalidation. */ export function currentTzKey(): string { @@ -187,7 +184,7 @@ export function currentTzKey(): string { } function getCachePath(): string { - return join(getCacheDir(), DAILY_CACHE_FILENAME) + return join(getCodeburnCacheDir(), DAILY_CACHE_FILENAME) } /** Absolute path of the active (version-suffixed) daily cache file. */ @@ -379,7 +376,7 @@ function isAdoptableCache(parsed: unknown): parsed is AdoptableCache { /// bump lossless: the new version starts from the union of everything every /// previous version ever recorded, then re-derives what sources still support. async function adoptOlderDailyCaches(): Promise { - const dir = getCacheDir() + const dir = getCodeburnCacheDir() let names: string[] = [] try { names = await readdir(dir) @@ -449,7 +446,7 @@ async function adoptOlderDailyCaches(): Promise { } export async function saveDailyCache(cache: DailyCache): Promise { - const dir = getCacheDir() + const dir = getCodeburnCacheDir() if (!existsSync(dir)) await mkdir(dir, { recursive: true }) const finalPath = getCachePath() const tempPath = `${finalPath}.${randomBytes(8).toString('hex')}.tmp` diff --git a/src/models.ts b/src/models.ts index f1d9ad0b..bf4ed448 100644 --- a/src/models.ts +++ b/src/models.ts @@ -1,8 +1,9 @@ import { readFile, writeFile, mkdir } from 'fs/promises' import { join } from 'path' -import { homedir } from 'os' -import snapshotData from './data/litellm-snapshot.json' -import fallbackData from './data/pricing-fallback.json' + +import { getCodeburnCacheDir } from './cache-dir.js' +import snapshotData from './data/litellm-snapshot.json' with { type: 'json' } +import fallbackData from './data/pricing-fallback.json' with { type: 'json' } import { fetchWithTimeout } from './fetch-utils.js' export type ModelCosts = { @@ -143,13 +144,8 @@ function getLowercasePricingIndex(): Map { return lowercasePricingIndex } -function getCacheDir(): string { - if (process.env['CODEBURN_CACHE_DIR']) return process.env['CODEBURN_CACHE_DIR'] - return join(homedir(), '.cache', 'codeburn') -} - function getCachePath(): string { - return join(getCacheDir(), 'litellm-pricing.json') + return join(getCodeburnCacheDir(), 'litellm-pricing.json') } /// Clamp a per-token rate to a sane non-negative value. Defense in depth @@ -202,7 +198,7 @@ async function fetchAndCachePricing(): Promise> { if (stripped !== name && !pricing.has(stripped)) pricing.set(stripped, costs) } - await mkdir(getCacheDir(), { recursive: true }) + await mkdir(getCodeburnCacheDir(), { recursive: true }) await writeFile(getCachePath(), JSON.stringify({ timestamp: Date.now(), data: Object.fromEntries(pricing), @@ -999,3 +995,33 @@ export function getShortModelName(model: string): string { } return canonical } + +// Pricing is process-global state assembled at CLI startup from the cached +// LiteLLM snapshot plus user config. A parse worker thread starts with none of +// it, and re-running loadPricing() there would mean N more disk reads (or, on a +// cold pricing cache, N network fetches). Ship the resolved state across +// instead, so every thread prices a call exactly as the main thread would. +export type PricingSnapshot = { + pricing: Map + aliases: Record + priceOverrides: Record + localModelSavings: Record +} + +export function snapshotPricingState(): PricingSnapshot { + return { + pricing: pricingCache, + aliases: userAliases, + priceOverrides: userPriceOverridesConfig, + localModelSavings: userLocalModelSavings, + } +} + +export function restorePricingState(snapshot: PricingSnapshot): void { + pricingCache = snapshot.pricing + sortedPricingKeys = null + lowercasePricingIndex = null + setModelAliases(snapshot.aliases) + setPriceOverrides(snapshot.priceOverrides) + setLocalModelSavings(snapshot.localModelSavings) +} diff --git a/src/parse-worker.ts b/src/parse-worker.ts new file mode 100644 index 00000000..df3d9cbf --- /dev/null +++ b/src/parse-worker.ts @@ -0,0 +1,34 @@ +import { parentPort, workerData } from 'worker_threads' +import { restorePricingState, type PricingSnapshot } from './models.js' +import type { ParseJob } from './parse-workers.js' +import { parseClaudeFileFull } from './parser.js' +import { parseCodexFileFull } from './providers/codex.js' + +const port = parentPort +if (!port) throw new Error('parse-worker must be started as a worker thread') + +restorePricingState((workerData as { pricing: PricingSnapshot }).pricing) + +// The parsed turns go back as a JSON string rather than as a live object graph: +// structured-cloning a whole corpus of turns costs more than the parallel parse +// saves, while a string is a single copy the parent re-parses at memcpy speed. +// `msgIds` / `keys` is every dedup key this file claimed; the parent uses it to +// prove no earlier file already owned one before installing the result. A Codex +// job also carries back the cache entry it would have written, because the cache +// module's per-directory state belongs to the parent, not to a thread. +port.on('message', (msg: ParseJob) => { + void (async () => { + try { + const seen = new Set() + if (msg.kind === 'codex') { + const parsed = await parseCodexFileFull(msg.source, seen) + port.postMessage({ json: JSON.stringify({ ...parsed, keys: [...seen], path: msg.source.path }) }) + return + } + const parsed = await parseClaudeFileFull(msg.filePath, seen) + port.postMessage({ json: parsed === null ? null : JSON.stringify({ ...parsed, msgIds: [...seen], path: msg.filePath }) }) + } catch (err) { + port.postMessage({ error: err instanceof Error ? err.message : String(err) }) + } + })() +}) diff --git a/src/parse-workers.ts b/src/parse-workers.ts new file mode 100644 index 00000000..25639687 --- /dev/null +++ b/src/parse-workers.ts @@ -0,0 +1,245 @@ +import { availableParallelism, totalmem } from 'os' +import { Worker } from 'worker_threads' +import { snapshotPricingState } from './models.js' +import type { ClaudeFileParse } from './parser.js' +import type { SessionSource } from './providers/types.js' + +// A worker holds one file's entries plus its serialized result, and the parent +// buffers up to `pool.size` finished results while it installs one — both are in +// this budget. A flat 256 MB was measured wrong on Codex: a 260 MB rollout peaks +// near 430 MB per worker and scales linearly with the pool. So derive it from the +// average pending file instead, floored at the small-transcript figure and capped +// at 1 GB. Going over the budget is what turns a parallel parse into a swapping one. +const MIN_PER_WORKER_RSS_BYTES = 256 * 1024 * 1024 +const MAX_PER_WORKER_RSS_BYTES = 1024 * 1024 * 1024 +const PER_WORKER_RSS_OVERHEAD_BYTES = 128 * 1024 * 1024 +const MEMORY_BUDGET_CAP_BYTES = 2 * 1024 * 1024 * 1024 +const MIN_AVAILABLE_BYTES = 4 * 1024 * 1024 * 1024 +const MIN_FILES_PER_WORKER = 50 +const MIN_BYTES_PER_WORKER = 200 * 1024 * 1024 +// Below this a parse is warm/incremental and the thread startup + result transfer +// costs more than the parallelism buys. Bytes, not file count: 250 pending files +// holding under a megabyte between them spawn threads that make the run ~5% +// SLOWER, and the file count only starts paying for itself around 400. +const MIN_PENDING_BYTES = 200 * 1024 * 1024 + +export type ParseWorkerDecision = { workers: number; reason: string } + +export type SystemCapacity = { cores: number; availableBytes: number } + +// `process.availableMemory()` respects a container's cgroup / rlimit, which is the +// case this gate exists for; where it is absent it falls back to total RAM. Free +// memory is deliberately NOT used: on macOS `os.freemem()` counts free pages, not +// available memory, and reads as a few hundred MB on an idle 128 GB machine — a +// gate built on it turns the feature on and off at random. +function currentSystemCapacity(): SystemCapacity { + return { + cores: availableParallelism(), + availableBytes: typeof process.availableMemory === 'function' ? process.availableMemory() : totalmem(), + } +} + +/// Decide how many parse worker threads a pending workload earns. Returning 0 +/// means "parse serially" — the only behaviour before this existed, and still +/// the behaviour for every warm run, every small corpus, and every low-spec box. +export function decideParseWorkers( + pending: { files: number; bytes: number }, + sys: SystemCapacity = currentSystemCapacity(), + env: NodeJS.ProcessEnv = process.env, +): ParseWorkerDecision { + // Every reason carries the full decision input, so a support log line explains + // itself without a second run. + const inputs = `${sys.cores} cores, ${Math.round(sys.availableBytes / 1e9 * 10) / 10} GB available, ${pending.files} pending files / ${Math.round(pending.bytes / 1e6)} MB` + + const override = env['CODEBURN_PARSE_WORKERS'] + if (override !== undefined && override !== '') { + const n = Number(override) + if (!Number.isFinite(n) || n < 0) return { workers: 0, reason: `invalid CODEBURN_PARSE_WORKERS=${override}` } + const capped = Math.min(Math.floor(n), sys.cores) + return { workers: capped, reason: `${capped === 0 ? 'forced serial' : 'forced'} by CODEBURN_PARSE_WORKERS=${override}; ${inputs}` } + } + + // Workload gates first, so a warm run's log line says "warm", not whatever the + // machine happened to look like at that moment. + if (pending.bytes < MIN_PENDING_BYTES) return { workers: 0, reason: `below ${Math.round(MIN_PENDING_BYTES / 1e6)} MB pending; ${inputs}` } + if (sys.cores <= 2) return { workers: 0, reason: `too few cores; ${inputs}` } + if (sys.availableBytes < MIN_AVAILABLE_BYTES) return { workers: 0, reason: `below ${Math.round(MIN_AVAILABLE_BYTES / 1e9)} GB available memory; ${inputs}` } + + const memoryBudget = Math.min(0.25 * sys.availableBytes, MEMORY_BUDGET_CAP_BYTES) + const perWorker = Math.min( + MAX_PER_WORKER_RSS_BYTES, + Math.max(MIN_PER_WORKER_RSS_BYTES, 2 * (pending.bytes / Math.max(1, pending.files)) + PER_WORKER_RSS_OVERHEAD_BYTES), + ) + // Files and bytes each earn threads on their own: a few hundred huge rollouts + // are as parallelisable as a few thousand small transcripts, and gating the + // count on files alone would hand a 6 GB / 60-file workload a single thread. + const workers = Math.min( + sys.cores - 1, + Math.floor(memoryBudget / perWorker), + Math.max( + Math.floor(pending.files / MIN_FILES_PER_WORKER), + Math.floor(pending.bytes / MIN_BYTES_PER_WORKER), + ), + ) + return { workers, reason: inputs } +} + +// In dist the entry is the bundled sibling of this module and a worker can load +// it directly. Running from source (tsx, vitest) the entry is TypeScript, and a +// worker thread inherits none of the parent's loader hooks — so register tsx's +// inside the thread before importing. tsx is a devDependency, which is exactly +// the only situation where the entry can be a .ts file at all. +function workerBootstrap(entryUrl: string): { source: string | URL; eval: boolean } { + if (!entryUrl.endsWith('.ts')) return { source: new URL(entryUrl), eval: false } + return { + eval: true, + // Chained, not awaited: the eval scope is CommonJS, and a top-level await of + // the entry re-enters it as a require(esm) cycle. + source: ` +process.noDeprecation = true +import('tsx/esm/api').then(tsx => { tsx.register(); return import(${JSON.stringify(entryUrl)}) }) +`, + } +} + +function workerEntryUrl(): string { + const ext = import.meta.url.endsWith('.ts') ? '.ts' : '.js' + return new URL(`./parse-worker${ext}`, import.meta.url).href +} + +/// One whole-file parse for a worker to run. Both kinds carry exactly what the +/// serial per-file parse takes, so the worker can run that same function. +export type ParseJob = + | { kind: 'claude'; filePath: string } + | { kind: 'codex'; source: SessionSource } + +export type ParseWorkerResult = + | { ok: true; parsed: T | null } + | { ok: false; error: string } + +export type ClaudeWorkerParse = ClaudeFileParse & { msgIds: string[]; path: string } + +type Task = { job: ParseJob; resolve: (r: ParseWorkerResult) => void } + +type WorkerMessage = { json?: string | null; error?: string } + +export class ParseWorkerPool { + private readonly workers: Worker[] = [] + private readonly idle: Worker[] = [] + private readonly inflight = new Map() + private readonly queue: Task[] = [] + private closed = false + + constructor(size: number) { + const boot = workerBootstrap(workerEntryUrl()) + const workerData = { pricing: snapshotPricingState() } + try { + for (let i = 0; i < size; i++) { + const worker = new Worker(boot.source, { eval: boot.eval, workerData }) + worker.on('message', (msg: WorkerMessage) => this.settle(worker, msg)) + worker.on('error', (err: Error) => this.settle(worker, { error: err.message }, true)) + worker.on('exit', () => this.drop(worker)) + this.workers.push(worker) + this.idle.push(worker) + } + } catch (err) { + for (const w of this.workers) void w.terminate() + this.workers.length = 0 + this.idle.length = 0 + throw err + } + } + + get size(): number { + return this.workers.length + } + + /// Parse one file off-thread. Never rejects: a worker-side failure (or a dead + /// pool) comes back as `ok: false` so the caller can fall back to an in-process + /// parse and never lose a file to a crashed thread. + submit(job: ParseJob): Promise> { + return new Promise>((resolve) => { + if (this.closed || this.workers.length === 0) { + resolve({ ok: false, error: 'parse worker pool unavailable' }) + return + } + this.queue.push({ job, resolve: resolve as (r: ParseWorkerResult) => void }) + this.pump() + }) + } + + async close(): Promise { + this.closed = true + const pending = [...this.queue] + this.queue.length = 0 + for (const task of pending) task.resolve({ ok: false, error: 'parse worker pool closed' }) + await Promise.all(this.workers.map(w => w.terminate())) + this.workers.length = 0 + this.idle.length = 0 + this.inflight.clear() + } + + private pump(): void { + while (this.queue.length > 0 && this.idle.length > 0) { + const worker = this.idle.pop()! + const task = this.queue.shift()! + this.inflight.set(worker, task) + worker.postMessage(task.job) + } + } + + private settle(worker: Worker, msg: WorkerMessage, fatal = false): void { + const task = this.inflight.get(worker) + this.inflight.delete(worker) + if (task) { + if (msg.error !== undefined) task.resolve({ ok: false, error: msg.error }) + else task.resolve({ ok: true, parsed: msg.json == null ? null : JSON.parse(msg.json) }) + } + if (fatal) return + if (!this.closed) { + this.idle.push(worker) + this.pump() + } + } + + // A thread that died takes its queue slot with it; the remaining files are + // handed back for a serial parse rather than being lost. + private drop(worker: Worker): void { + const i = this.workers.indexOf(worker) + if (i >= 0) this.workers.splice(i, 1) + const j = this.idle.indexOf(worker) + if (j >= 0) this.idle.splice(j, 1) + const task = this.inflight.get(worker) + if (task) { + this.inflight.delete(worker) + task.resolve({ ok: false, error: 'parse worker exited' }) + } + if (this.workers.length === 0) { + const pending = [...this.queue] + this.queue.length = 0 + for (const t of pending) t.resolve({ ok: false, error: 'all parse workers exited' }) + } + } +} + +/// Yield results for `jobs` in the SAME order they were given, no matter which +/// worker finishes first. Keeps exactly `pool.size` files in flight, so at most +/// that many parsed results are buffered while the caller installs one. +export async function* parseFilesInOrder( + pool: ParseWorkerPool, + jobs: readonly ParseJob[], +): AsyncGenerator, void, void> { + const inflight: Array>> = [] + let next = 0 + const fill = (): void => { + while (inflight.length < Math.max(1, pool.size) && next < jobs.length) { + inflight.push(pool.submit(jobs[next++]!)) + } + } + fill() + for (let i = 0; i < jobs.length; i++) { + const result = await inflight.shift()! + fill() + yield result + } +} diff --git a/src/parser.ts b/src/parser.ts index 712295b0..7eec184c 100644 --- a/src/parser.ts +++ b/src/parser.ts @@ -4,12 +4,13 @@ import { basename, dirname, join, resolve, sep } from 'path' import { readSessionLines } from './fs-utils.js' import { calculateCost, calculateLocalModelSavings, getShortModelName, isProxiedPath, getProxyPathsConfigHash, getModelAliasesConfigHash, getPriceOverridesConfigHash, getLocalModelSavingsConfigHash } from './models.js' import { resolveSubagentAttribution, sessionIdentity } from './sessions-report.js' -import { normalizeContentBlocks } from './content-utils.js' +import { normalizeContentBlocks, flatSlice, flatString } from './content-utils.js' import { discoverAllSessions, getProvider } from './providers/index.js' -import { flushCodexCache } from './codex-cache.js' +import { flushCodexCache, readCachedCodexResults, withCodexCacheDirectory, writeCachedCodexResults } from './codex-cache.js' import { antigravityCascadeIdFromPath, flushAntigravityCache, shouldReparseAntigravitySource } from './providers/antigravity.js' -import { getDesktopSessionsDirs } from './providers/claude.js' +import { getClaudeConfigDirs, getDesktopSessionsDirs } from './providers/claude.js' import { isSqliteBusyError } from './sqlite.js' +import { getCodeburnCacheDir } from './cache-dir.js' import { type CachedCall, type CachedFile, @@ -22,11 +23,16 @@ import { DURABLE_PROVIDER_NAMES, fingerprintFile, isCacheComplete, + isCacheDirty, loadCache, + markCacheDirty, + monthScopeForRange, reconcileFile, saveCache, } from './session-cache.js' import { acquireCacheRefreshLock, type RefreshLockHandle } from './cache-refresh-lock.js' +import { decideParseWorkers, parseFilesInOrder, ParseWorkerPool, type ClaudeWorkerParse, type ParseJob } from './parse-workers.js' +import type { CodexFullParse } from './providers/codex.js' import { dateKey } from './day-aggregator.js' import type { ParsedProviderCall, SessionSource } from './providers/types.js' import type { @@ -91,7 +97,27 @@ function isCoworkSession(cwd: string, filePath: string): boolean { }) } +// Memoizes resolveCanonicalProjectPath: every ParsedProviderCall with a +// projectPath pays the .git-marker directory walk (one lstat per ancestor +// level), and a session's calls all share one cwd — without this cache a +// cold parse re-walks the same few directories thousands of times +// (measured ~+5% cold-parse time for a large kiro store). Filesystem facts +// can go stale in a long-lived process (a dir converted to a worktree +// mid-run), so the cache is cleared with the session cache. +// Stores the Promise, not the resolved value: callers within the same +// Promise.all batch would otherwise all miss the cache and each re-walk the +// filesystem before the first walk's result lands. +const canonicalPathCache = new Map>() + async function resolveCanonicalProjectPath(cwd: string): Promise<{ path: string; isWorktree: boolean }> { + const cached = canonicalPathCache.get(cwd) + if (cached) return cached + const result = resolveCanonicalProjectPathUncached(cwd) + canonicalPathCache.set(cwd, result) + return result +} + +async function resolveCanonicalProjectPathUncached(cwd: string): Promise<{ path: string; isWorktree: boolean }> { const trimmed = cwd.trim() if (!trimmed) return { path: cwd, isWorktree: false } @@ -1261,7 +1287,7 @@ export function collectToolResultMeta(entry: JournalEntry, map: Map)['aiTitle'] - if (typeof t === 'string' && t.trim()) meta.title = t.trim().slice(0, 200) + if (typeof t === 'string' && t.trim()) meta.title = flatString(t.trim().slice(0, 200)) } else if (entry.type === 'pr-link') { const url = (entry as Record)['prUrl'] if (typeof url === 'string' && url && !meta.prLinks.includes(url)) meta.prLinks.push(url) @@ -1900,7 +1926,7 @@ export async function readAgentType(filePath: string): Promise !f.append).map(f => f.filePath) + const pendingBytes = changedFiles.reduce((n, f) => f.append ? n : n + f.info.fp.sizeBytes, 0) + const decision = decideParseWorkers({ files: fullReparsePaths.length, bytes: pendingBytes }) + if (process.env['CODEBURN_VERBOSE'] === '1') { + process.stderr.write(`codeburn: claude parse workers=${decision.workers} (${decision.reason})\n`) + } + // A pool that cannot even start (worker entry missing from an odd packaging, + // thread limit reached) must degrade to the serial parse, not fail the run. + let pool: ParseWorkerPool | null = null + if (decision.workers > 0) { try { - if (append) { - // Append-only growth: parse ONLY the bytes past the cached resume offset - // and merge with the cached turns, rather than re-reading the file from 0. - // On a studio machine where live agents constantly append to session - // JSONL, this is the dominant warm-run cost. The merged result is - // byte-for-byte identical to a full re-parse (see mergeBoundaryCalls). - const tracker = { lastCompleteLineOffset: append.readFromOffset } - const toolResultMeta = new Map() - const sessionMeta = emptySessionMeta() - const newEntries = await parseClaudeEntries(filePath, tracker, append.readFromOffset, { toolResultMeta, sessionMeta }) - const cached = append.cached - - // Straddle guard: a streamed assistant message id that first appeared in - // the committed prefix can be restated inside the appended region - // (image-heavy turns stream one id across several records over seconds). - // The appended region is grouped before this file's cached keys join - // seenMsgIds, so the restated id would count twice; suppressing it - // instead would freeze the stale first emission. Neither matches a full - // re-parse, so on any id overlap the shortcut is abandoned and the file - // re-parses from byte 0 (rare: ~0.3% of real files). - const cachedIds = new Set(cached.turns.flatMap(t => t.calls.map(c => c.deduplicationKey))) - const straddles = newEntries !== null && newEntries.some(e => { - const id = getMessageId(e) - return id !== null && cachedIds.has(id) - }) - if (!straddles) { - const newTurns = newEntries - ? parsedTurnsToCachedTurns(groupIntoTurns(dedupeStreamingMessageIds(newEntries), seenMsgIds, toolResultMeta)) - : [] - - const mergedTurns: CachedTurn[] = cached.turns.map(t => ({ ...t, calls: [...t.calls] })) - if (newTurns.length > 0) { - let startIdx = 0 - // A first new turn with no leading user message is a continuation of - // the last cached turn — merge its calls in (a full re-parse would put - // them in that same turn), then append the remaining new turns. - if (!newTurns[0]!.userMessage.trim() && mergedTurns.length > 0) { - const last = mergedTurns[mergedTurns.length - 1]! - last.calls = mergeBoundaryCalls(last.calls, newTurns[0]!.calls) - // A PR referenced in the appended continuation belongs to this same - // turn: union its refs in so the shortcut matches a full re-parse. - const refs = Array.from(new Set([...(last.prRefs ?? []), ...(newTurns[0]!.prRefs ?? [])])).sort() - if (refs.length > 0) last.prRefs = refs - // A subagent spawned in the appended continuation belongs to this - // same turn: union its spawn ids in for the same reason. - const spawnIds = Array.from(new Set([...(last.spawnToolUseIds ?? []), ...(newTurns[0]!.spawnToolUseIds ?? [])])) - if (spawnIds.length > 0) last.spawnToolUseIds = spawnIds - startIdx = 1 - } - for (let i = startIdx; i < newTurns.length; i++) mergedTurns.push(newTurns[i]!) - } - - // The cached region's dedup keys were not added to seenMsgIds (only - // unchanged files pre-seed it), so add them now — a full re-parse would - // have, and later files dedup cross-file against them. - for (const t of cached.turns) for (const c of t.calls) seenMsgIds.add(c.deduplicationKey) - - // First-cwd wins, and the first cwd lives in the cached region whenever - // one was resolved there; only re-derive if the cached region had none. - let canonicalCwd = cached.canonicalCwd - let canonicalProjectName = cached.canonicalProjectName - let workingDirectory = cached.workingDirectory - if (canonicalCwd === undefined && newEntries) { - const cwd = extractCanonicalCwd(newEntries) - workingDirectory = workingDirectory ?? cwd - const canonical = (cwd && !isCoworkSession(cwd, filePath)) ? await resolveCanonicalProjectPath(cwd) : undefined - canonicalCwd = canonical?.path - canonicalProjectName = canonical?.isWorktree ? projectNameFromPath(canonical.path, info.dirName) : undefined - } - - // Inventory is a sorted set union; cached (older entries) ∪ new = full. - const mcpInventory = newEntries - ? Array.from(new Set([...cached.mcpInventory, ...extractMcpInventory(newEntries)])).sort() - : cached.mcpInventory - - // Session meta merges across the append boundary: title is last-wins - // (prefer the newly-parsed tail), PR links union, isSidechain is sticky. - // parentSessionId is sticky (cached-first, it is the earliest region); - // agentSpawnLinks union (cached-first, first-seen spawn id per agent wins). - const mergedTitle = sessionMeta.title ?? cached.title - const mergedPrLinks = Array.from(new Set([...(cached.prLinks ?? []), ...sessionMeta.prLinks])) - const mergedSidechain = cached.isSidechain === true || sessionMeta.isSidechain - const mergedParentSessionId = cached.parentSessionId ?? sessionMeta.parentSessionId - const mergedSpawnLinks = { ...sessionMeta.agentSpawnLinks, ...cached.agentSpawnLinks } - const mergedAmbiguousIds = Array.from(new Set([...(cached.ambiguousSpawnAgentIds ?? []), ...sessionMeta.ambiguousSpawnAgentIds])) - - section.files[filePath] = { - fingerprint: info.fp, - lastCompleteLineOffset: tracker.lastCompleteLineOffset, - canonicalCwd, - ...(workingDirectory ? { workingDirectory } : {}), - canonicalProjectName, - mcpInventory, - turns: mergedTurns, - agentType: cached.agentType, - ...(mergedTitle ? { title: mergedTitle } : {}), - ...(mergedPrLinks.length > 0 ? { prLinks: mergedPrLinks } : {}), - ...(mergedSidechain ? { isSidechain: true } : {}), - ...(mergedParentSessionId ? { parentSessionId: mergedParentSessionId } : {}), - ...(Object.keys(mergedSpawnLinks).length > 0 ? { agentSpawnLinks: mergedSpawnLinks } : {}), - ...(mergedAmbiguousIds.length > 0 ? { ambiguousSpawnAgentIds: mergedAmbiguousIds } : {}), - } - ;(diskCache as { _dirty?: boolean })._dirty = true - filesDone++ - await parseProgress.tick(filesDone) - if (filesDone % 50 === 0 || filesDone === progressTotal) { - emitScanProgress({ kind: 'tick', provider: 'claude', done: filesDone, total: progressTotal }) - } - if (onFileParsed) await onFileParsed() - continue - } - // Straddled: fall through to the full re-parse below. - } - - const tracker = { lastCompleteLineOffset: 0 } - const toolResultMeta = new Map() - const sessionMeta = emptySessionMeta() - const entries = await parseClaudeEntries(filePath, tracker, undefined, { toolResultMeta, sessionMeta }) - if (!entries) { filesDone++; await parseProgress.tick(filesDone); continue } - - const turns = groupIntoTurns(dedupeStreamingMessageIds(entries), seenMsgIds, toolResultMeta) - const cwd = extractCanonicalCwd(entries) - const canonical = (cwd && !isCoworkSession(cwd, filePath)) ? await resolveCanonicalProjectPath(cwd) : undefined - section.files[filePath] = { - fingerprint: info.fp, - lastCompleteLineOffset: tracker.lastCompleteLineOffset, - canonicalCwd: canonical?.path, - ...(cwd ? { workingDirectory: cwd } : {}), - canonicalProjectName: canonical?.isWorktree ? projectNameFromPath(canonical.path, info.dirName) : undefined, - mcpInventory: extractMcpInventory(entries), - turns: parsedTurnsToCachedTurns(turns), - agentType: await readAgentType(filePath), - ...(sessionMeta.title ? { title: sessionMeta.title } : {}), - ...(sessionMeta.prLinks.length > 0 ? { prLinks: sessionMeta.prLinks } : {}), - ...(sessionMeta.isSidechain ? { isSidechain: true } : {}), - ...(sessionMeta.parentSessionId ? { parentSessionId: sessionMeta.parentSessionId } : {}), - ...(Object.keys(sessionMeta.agentSpawnLinks).length > 0 ? { agentSpawnLinks: sessionMeta.agentSpawnLinks } : {}), - ...(sessionMeta.ambiguousSpawnAgentIds.length > 0 ? { ambiguousSpawnAgentIds: sessionMeta.ambiguousSpawnAgentIds } : {}), - } - ;(diskCache as { _dirty?: boolean })._dirty = true + pool = new ParseWorkerPool(decision.workers) } catch (err) { - // A single malformed Claude session file must not abort the whole run — that - // would empty the daily-cache backfill and wipe the trend/history (issue #441, - // same isolation the provider path already has). Record a failure marker keyed - // by the current fingerprint so it isn't re-read and re-thrown every run; it - // re-parses only if the file changes. - section.files[filePath] = { fingerprint: info.fp, mcpInventory: [], turns: [], failed: true } - ;(diskCache as { _dirty?: boolean })._dirty = true - warnProviderParseFailure('claude', filePath, err) + process.stderr.write(`codeburn: parse workers unavailable, parsing serially (${err instanceof Error ? err.message : String(err)})\n`) } - filesDone++ - await parseProgress.tick(filesDone) - // Machine-readable tick for the app splash (throttled to ~every 50 files so - // a large cold run doesn't flood stderr), plus a partial-progress save. - if (filesDone % 50 === 0 || filesDone === progressTotal) { - emitScanProgress({ kind: 'tick', provider: 'claude', done: filesDone, total: progressTotal }) + } + const offThread = pool + ? parseFilesInOrder(pool, fullReparsePaths.map(filePath => ({ kind: 'claude', filePath }))) + : null + // Files whose worker result had to be thrown away because an earlier file had + // already claimed one of its message ids. Expected to stay near zero; a large + // count means the corpus is full of resumed sessions and the pool is doing + // double work. + let workerDiscards = 0 + + const installClaudeFile = async (filePath: string, info: FileInfo, parsed: ClaudeFileParse): Promise => { + const cwd = parsed.workingDirectory + const canonical = (cwd && !isCoworkSession(cwd, filePath)) ? await resolveCanonicalProjectPath(cwd) : undefined + section.files[filePath] = { + fingerprint: info.fp, + lastCompleteLineOffset: parsed.lastCompleteLineOffset, + canonicalCwd: canonical?.path, + ...(cwd ? { workingDirectory: cwd } : {}), + canonicalProjectName: canonical?.isWorktree ? projectNameFromPath(canonical.path, info.dirName) : undefined, + mcpInventory: parsed.mcpInventory, + turns: parsed.turns, + agentType: parsed.agentType, + ...(parsed.title ? { title: parsed.title } : {}), + ...(parsed.prLinks?.length ? { prLinks: parsed.prLinks } : {}), + ...(parsed.isSidechain ? { isSidechain: true } : {}), + ...(parsed.parentSessionId ? { parentSessionId: parsed.parentSessionId } : {}), + ...(Object.keys(parsed.agentSpawnLinks ?? {}).length > 0 ? { agentSpawnLinks: parsed.agentSpawnLinks } : {}), + ...(parsed.ambiguousSpawnAgentIds?.length ? { ambiguousSpawnAgentIds: parsed.ambiguousSpawnAgentIds } : {}), } - if (onFileParsed) await onFileParsed() + markCacheDirty(diskCache, 'claude', filePath) + } + + try { + for (const { filePath, info, append } of changedFiles) { + // Marked here, not after the re-parse: an unreadable file `continue`s out + // below, and the deletion would otherwise live only in memory. + delete section.files[filePath] + markCacheDirty(diskCache, 'claude', filePath) + + // Off-thread results arrive in this order (parseFilesInOrder), so the Nth + // full re-parse here is the Nth yielded result — appends never consume one, + // in either the shortcut or the straddled-fallthrough case. A worker parses + // against an EMPTY dedup set, so an EMPTY id intersection is the proof that a + // serial parse would have dropped nothing either — that, and only that, makes + // the result installable. On any overlap the WHOLE file is discarded and + // re-parsed in-process. Never patch the overlapping turns out of a worker + // result instead: a drop is not local to its own turn, because + // parsedTurnsToCachedTurns delta-encodes gitBranch across turns, so removing + // one turn changes whether a LATER turn carries a gitBranch key. + // Deliberately OUTSIDE the per-file try below: the pairing is positional, and + // a misalignment would install one session's turns under another's path — a + // wrong number nobody would ever notice, so it fails the run instead of being + // caught as a parse failure. + let parsed: ClaudeFileParse | null | undefined + if (offThread && !append) { + const result = (await offThread.next()).value + if (result?.ok && result.parsed) { + if (result.parsed.path !== filePath) { + throw new Error(`claude parse worker result out of order: got ${result.parsed.path}, expected ${filePath}`) + } + if (result.parsed.msgIds.some(id => seenMsgIds.has(id))) { + workerDiscards++ + parsed = undefined + } else { + for (const id of result.parsed.msgIds) seenMsgIds.add(id) + parsed = result.parsed + } + } else if (result?.ok) { + parsed = null + } + } + + try { + if (append) { + // Append-only growth: parse ONLY the bytes past the cached resume offset + // and merge with the cached turns, rather than re-reading the file from 0. + // On a studio machine where live agents constantly append to session + // JSONL, this is the dominant warm-run cost. The merged result is + // byte-for-byte identical to a full re-parse (see mergeBoundaryCalls). + const tracker = { lastCompleteLineOffset: append.readFromOffset } + const toolResultMeta = new Map() + const sessionMeta = emptySessionMeta() + const newEntries = await parseClaudeEntries(filePath, tracker, append.readFromOffset, { toolResultMeta, sessionMeta }) + const cached = append.cached + + // Straddle guard: a streamed assistant message id that first appeared in + // the committed prefix can be restated inside the appended region + // (image-heavy turns stream one id across several records over seconds). + // The appended region is grouped before this file's cached keys join + // seenMsgIds, so the restated id would count twice; suppressing it + // instead would freeze the stale first emission. Neither matches a full + // re-parse, so on any id overlap the shortcut is abandoned and the file + // re-parses from byte 0 (rare: ~0.3% of real files). + const cachedIds = new Set(cached.turns.flatMap(t => t.calls.map(c => c.deduplicationKey))) + const straddles = newEntries !== null && newEntries.some(e => { + const id = getMessageId(e) + return id !== null && cachedIds.has(id) + }) + if (!straddles) { + const newTurns = newEntries + ? parsedTurnsToCachedTurns(groupIntoTurns(dedupeStreamingMessageIds(newEntries), seenMsgIds, toolResultMeta)) + : [] + + const mergedTurns: CachedTurn[] = cached.turns.map(t => ({ ...t, calls: [...t.calls] })) + if (newTurns.length > 0) { + let startIdx = 0 + // A first new turn with no leading user message is a continuation of + // the last cached turn — merge its calls in (a full re-parse would put + // them in that same turn), then append the remaining new turns. + if (!newTurns[0]!.userMessage.trim() && mergedTurns.length > 0) { + const last = mergedTurns[mergedTurns.length - 1]! + last.calls = mergeBoundaryCalls(last.calls, newTurns[0]!.calls) + // A PR referenced in the appended continuation belongs to this same + // turn: union its refs in so the shortcut matches a full re-parse. + const refs = Array.from(new Set([...(last.prRefs ?? []), ...(newTurns[0]!.prRefs ?? [])])).sort() + if (refs.length > 0) last.prRefs = refs + // A subagent spawned in the appended continuation belongs to this + // same turn: union its spawn ids in for the same reason. + const spawnIds = Array.from(new Set([...(last.spawnToolUseIds ?? []), ...(newTurns[0]!.spawnToolUseIds ?? [])])) + if (spawnIds.length > 0) last.spawnToolUseIds = spawnIds + startIdx = 1 + } + for (let i = startIdx; i < newTurns.length; i++) mergedTurns.push(newTurns[i]!) + } + + // The cached region's dedup keys were not added to seenMsgIds (only + // unchanged files pre-seed it), so add them now — a full re-parse would + // have, and later files dedup cross-file against them. + for (const t of cached.turns) for (const c of t.calls) seenMsgIds.add(c.deduplicationKey) + + // First-cwd wins, and the first cwd lives in the cached region whenever + // one was resolved there; only re-derive if the cached region had none. + let canonicalCwd = cached.canonicalCwd + let canonicalProjectName = cached.canonicalProjectName + let workingDirectory = cached.workingDirectory + if (canonicalCwd === undefined && newEntries) { + const cwd = extractCanonicalCwd(newEntries) + workingDirectory = workingDirectory ?? cwd + const canonical = (cwd && !isCoworkSession(cwd, filePath)) ? await resolveCanonicalProjectPath(cwd) : undefined + canonicalCwd = canonical?.path + canonicalProjectName = canonical?.isWorktree ? projectNameFromPath(canonical.path, info.dirName) : undefined + } + + // Inventory is a sorted set union; cached (older entries) ∪ new = full. + const mcpInventory = newEntries + ? Array.from(new Set([...cached.mcpInventory, ...extractMcpInventory(newEntries)])).sort() + : cached.mcpInventory + + // Session meta merges across the append boundary: title is last-wins + // (prefer the newly-parsed tail), PR links union, isSidechain is sticky. + // parentSessionId is sticky (cached-first, it is the earliest region); + // agentSpawnLinks union (cached-first, first-seen spawn id per agent wins). + const mergedTitle = sessionMeta.title ?? cached.title + const mergedPrLinks = Array.from(new Set([...(cached.prLinks ?? []), ...sessionMeta.prLinks])) + const mergedSidechain = cached.isSidechain === true || sessionMeta.isSidechain + const mergedParentSessionId = cached.parentSessionId ?? sessionMeta.parentSessionId + const mergedSpawnLinks = { ...sessionMeta.agentSpawnLinks, ...cached.agentSpawnLinks } + const mergedAmbiguousIds = Array.from(new Set([...(cached.ambiguousSpawnAgentIds ?? []), ...sessionMeta.ambiguousSpawnAgentIds])) + + section.files[filePath] = { + fingerprint: info.fp, + lastCompleteLineOffset: tracker.lastCompleteLineOffset, + canonicalCwd, + ...(workingDirectory ? { workingDirectory } : {}), + canonicalProjectName, + mcpInventory, + turns: mergedTurns, + agentType: cached.agentType, + ...(mergedTitle ? { title: mergedTitle } : {}), + ...(mergedPrLinks.length > 0 ? { prLinks: mergedPrLinks } : {}), + ...(mergedSidechain ? { isSidechain: true } : {}), + ...(mergedParentSessionId ? { parentSessionId: mergedParentSessionId } : {}), + ...(Object.keys(mergedSpawnLinks).length > 0 ? { agentSpawnLinks: mergedSpawnLinks } : {}), + ...(mergedAmbiguousIds.length > 0 ? { ambiguousSpawnAgentIds: mergedAmbiguousIds } : {}), + } + markCacheDirty(diskCache, 'claude', filePath) + filesDone++ + await parseProgress.tick(filesDone) + if (filesDone % 50 === 0 || filesDone === progressTotal) { + emitScanProgress({ kind: 'tick', provider: 'claude', done: filesDone, total: progressTotal }) + } + if (onFileParsed) await onFileParsed() + continue + } + // Straddled: fall through to the full re-parse below. + } + + if (parsed === undefined) parsed = await parseClaudeFileFull(filePath, seenMsgIds) + if (!parsed) { filesDone++; await parseProgress.tick(filesDone); continue } + + await installClaudeFile(filePath, info, parsed) + } catch (err) { + // A single malformed Claude session file must not abort the whole run — that + // would empty the daily-cache backfill and wipe the trend/history (issue #441, + // same isolation the provider path already has). Record a failure marker keyed + // by the current fingerprint so it isn't re-read and re-thrown every run; it + // re-parses only if the file changes. + section.files[filePath] = { fingerprint: info.fp, mcpInventory: [], turns: [], failed: true } + markCacheDirty(diskCache, 'claude', filePath) + warnProviderParseFailure('claude', filePath, err) + } + filesDone++ + await parseProgress.tick(filesDone) + // Machine-readable tick for the app splash (throttled to ~every 50 files so + // a large cold run doesn't flood stderr), plus a partial-progress save. + if (filesDone % 50 === 0 || filesDone === progressTotal) { + emitScanProgress({ kind: 'tick', provider: 'claude', done: filesDone, total: progressTotal }) + } + if (onFileParsed) await onFileParsed() + } + } finally { + await pool?.close() + } + if (pool && process.env['CODEBURN_VERBOSE'] === '1') { + process.stderr.write(`codeburn: claude parse workers done, ${workerDiscards}/${fullReparsePaths.length} results re-parsed in-process on id overlap\n`) } parseProgress.finish() @@ -2160,7 +2257,7 @@ async function scanProjectDirs( // but they carry attributable PR spend (surfaced above as a legacy split). if (section.files[cachedPath]?.prLinks?.length) continue delete section.files[cachedPath] - ;(diskCache as { _dirty?: boolean })._dirty = true + markCacheDirty(diskCache, 'claude', cachedPath) } } @@ -2187,7 +2284,13 @@ async function scanProjectDirs( let carriedPrRefs: string[] | undefined let prRefsAtRangeStart: string[] | undefined let frozePrRefs = !dateRange - let classifiedTurns = cachedFile.turns.map(turn => { + // The keep/drop decision is taken on the RAW turn, before classifying it: + // `cachedTurnToClassified` maps `calls` 1:1 onto `assistantCalls`, so a turn + // with no call in range is dropped whole by the slicer below and classifying + // it is pure waste (on a week view that is nearly all of history). The + // carries above still run over the FULL ordered turn list. + const classifiedTurns: ClassifiedTurn[] = [] + for (const turn of cachedFile.turns) { if (turn.gitBranch) carriedBranch = turn.gitBranch if (dateRange && !frozePrRefs) { const firstTs = turn.calls[0]?.timestamp @@ -2197,10 +2300,16 @@ async function scanProjectDirs( } } if (turn.prRefs?.length) carriedPrRefs = turn.prRefs - return cachedTurnToClassified(turn, carriedBranch) - }) - // Captured from the FULL turn list, before the date slice below can drop the - // turn a branch was first seen on. Lets the by-branch report keep this + if (dateRange && !callsInRange(turn.calls, dateRange)) continue + const classified = cachedTurnToClassified(turn, carriedBranch) + // Slice rather than drop: a turn spanning local midnight would otherwise + // lose every call that lands in the requested day (issue #852). Only + // `assistantCalls`/`timestamp` are touched — see classifiedTurnSlicedToRange. + const sliced = dateRange ? classifiedTurnSlicedToRange(classified, dateRange) : classified + if (sliced) classifiedTurns.push(sliced) + } + // Captured from the FULL turn list, which the date slice above can strip of + // the turn a branch was first seen on. Lets the by-branch report keep this // session's in-range unbranched spend as `null` instead of discarding it. const everHadBranch = carriedBranch !== undefined @@ -2210,16 +2319,6 @@ async function scanProjectDirs( // sessions that both spawned subagents and referenced a PR. const spawnPrSets = cachedFile.prLinks?.length ? buildSpawnPrSets(cachedFile.turns) : {} - if (dateRange) { - // Slice rather than drop: a turn spanning local midnight would otherwise - // lose every call that lands in the requested day (issue #852). Only - // `assistantCalls`/`timestamp` are touched — see classifiedTurnSlicedToRange. - classifiedTurns = classifiedTurns.flatMap(turn => { - const sliced = classifiedTurnSlicedToRange(turn, dateRange) - return sliced ? [sliced] : [] - }) - } - // A PR-linked parent that spawned subagents is kept even when its OWN turns all // fall out of range, as a 0-cost fold ANCHOR: an in-range child (an async agent // that outlived the parent's last in-range turn) still needs the parent's @@ -2453,7 +2552,7 @@ function parsedTurnToCachedTurn(turn: ParsedTurn): CachedTurn { return { timestamp: turn.timestamp, sessionId: turn.sessionId, - userMessage: turn.userMessage.slice(0, 2000), + userMessage: flatSlice(turn.userMessage, 2000), calls: turn.assistantCalls.map(apiCallToCachedCall), // Stored per-turn directly (already sorted/deduped in groupIntoTurns), unlike // gitBranch's change-detection dedup, so each turn's refs are self-contained. @@ -2484,7 +2583,7 @@ function providerCallToCachedTurn(call: ParsedProviderCall): CachedTurn { return { timestamp: call.timestamp, sessionId: call.sessionId, - userMessage: call.userMessage.slice(0, 2000), + userMessage: flatSlice(call.userMessage, 2000), calls: [providerCallToCachedCall(call)], ...(prRefs.length ? { prRefs } : {}), } @@ -2507,7 +2606,7 @@ function providerCallsToCachedTurns(calls: ParsedProviderCall[]): CachedTurn[] { turn = { timestamp: call.timestamp, sessionId: call.sessionId, - userMessage: call.userMessage.slice(0, 2000), + userMessage: flatSlice(call.userMessage, 2000), calls: [], ...(prRefs.length ? { prRefs } : {}), } @@ -2647,6 +2746,54 @@ async function parseClaudeEntries( return entries } +// Everything a cold Claude re-parse does for ONE file: read + decode + line-parse +// the JSONL, group it into turns, shape it for the cache. Depends on nothing +// process-wide except `seenMsgIds`, so a worker thread can run it against a fresh +// empty set and the parent can install the result verbatim once it has confirmed +// none of those ids were already claimed by an earlier file. Canonical-path +// resolution deliberately stays with the caller: it walks the filesystem behind a +// process-global memo. +export type ClaudeFileParse = { + lastCompleteLineOffset: number + workingDirectory?: string + mcpInventory: string[] + turns: CachedTurn[] + agentType?: string + title?: string + prLinks?: string[] + isSidechain?: boolean + parentSessionId?: string + agentSpawnLinks?: Record + ambiguousSpawnAgentIds?: string[] +} + +export async function parseClaudeFileFull( + filePath: string, + seenMsgIds: Set, +): Promise { + const tracker = { lastCompleteLineOffset: 0 } + const toolResultMeta = new Map() + const sessionMeta = emptySessionMeta() + const entries = await parseClaudeEntries(filePath, tracker, undefined, { toolResultMeta, sessionMeta }) + if (!entries) return null + + const turns = groupIntoTurns(dedupeStreamingMessageIds(entries), seenMsgIds, toolResultMeta) + const cwd = extractCanonicalCwd(entries) + return { + lastCompleteLineOffset: tracker.lastCompleteLineOffset, + ...(cwd ? { workingDirectory: cwd } : {}), + mcpInventory: extractMcpInventory(entries), + turns: parsedTurnsToCachedTurns(turns), + agentType: await readAgentType(filePath), + ...(sessionMeta.title ? { title: sessionMeta.title } : {}), + ...(sessionMeta.prLinks.length > 0 ? { prLinks: sessionMeta.prLinks } : {}), + ...(sessionMeta.isSidechain ? { isSidechain: true } : {}), + ...(sessionMeta.parentSessionId ? { parentSessionId: sessionMeta.parentSessionId } : {}), + ...(Object.keys(sessionMeta.agentSpawnLinks).length > 0 ? { agentSpawnLinks: sessionMeta.agentSpawnLinks } : {}), + ...(sessionMeta.ambiguousSpawnAgentIds.length > 0 ? { ambiguousSpawnAgentIds: sessionMeta.ambiguousSpawnAgentIds } : {}), + } +} + function getOrCreateProviderSection(cache: SessionCache, provider: string): ProviderSection { const envFp = computeEnvFingerprint(provider) const existing = cache.providers[provider] @@ -2665,6 +2812,7 @@ function getOrCreateProviderSection(cache: SessionCache, provider: string): Prov } } cache.providers[provider] = section + markCacheDirty(cache, provider) return section } @@ -2782,10 +2930,15 @@ export function emitScanProgress(event: ScanProgressEvent): void { try { process.stderr.write(`${PROGRESS_LINE_PREFIX}${JSON.stringify(event)}\n`) } catch { /* stderr closed */ } } -// Minimum spacing between partial-progress saves during a cold parse. Low enough +// Files parsed between partial-progress saves during a cold parse. Low enough // that an interrupted long run loses little work, high enough that repeated -// full-cache writes never dominate a fast warm run. -const PROGRESS_SAVE_THROTTLE_MS = 5000 +// cache writes never dominate the parse. +const PROGRESS_SAVE_FILE_INTERVAL = 2000 +// Only the claude scan reports per file; every other provider calls saveProgress +// once, at its own boundary. Without a time floor the counter would never reach +// the interval during a long non-claude phase and progress saves would simply +// stop happening there. +const PROGRESS_SAVE_MAX_INTERVAL_MS = 30_000 export function createScanProgress(label: string, total: number) { const show = !interactiveScanUI && total > 20 && process.stderr.isTTY === true @@ -2832,8 +2985,8 @@ function turnSlicedToRange(turn: CachedTurn, dateRange: DateRange): CachedTurn | return { ...turn, calls: inRangeCalls, timestamp: inRangeCalls[0]!.timestamp } } -// Same slice, applied post-classification (scanProjectDirs classifies every -// turn from its FULL call list up front, before date filtering — see the +// Same slice, applied post-classification (scanProjectDirs classifies each +// surviving turn from its FULL call list, before date filtering — see the // carriedBranch/carriedPrRefs comments in scanProjectDirs — so this only // trims `assistantCalls` and re-anchors `timestamp`; `category`/`subCategory`/ // `retries`/`hasEdits` stay exactly as classified from the complete turn. @@ -2869,6 +3022,10 @@ async function parseProviderSources( ): Promise { const provider = await getProvider(providerName) if (!provider) return [] + // The environment is a call-time input. Capture Antigravity's cache target + // for this whole parse transaction so a host changing CODEBURN_CACHE_DIR + // before the final flush cannot redirect A's dirty state into (or past) B. + const antigravityCacheDir = providerName === 'antigravity' ? getCodeburnCacheDir() : undefined const section = getOrCreateProviderSection(diskCache, providerName) const allDiscoveredFiles = new Set() @@ -2933,6 +3090,45 @@ async function parseProviderSources( } } + // Codex rollouts are the bulk of a cold parse (multi-GB against Claude's + // hundreds of MB), so whole-file decodes go to worker threads. A file the + // codex cache can serve exactly, or resume into from a byte offset, stays + // in-process: it reads a few KB, and it is the codex cache's own per-directory + // state that a thread must never own. The eligible list is built with the same + // filters (and in the same order) the parse loop applies, so the Nth result + // parseFilesInOrder yields is the Nth file that reaches the worker branch. + // The decision is per provider rather than pooled across Claude+Codex because + // the two scans run one after the other — at most one pool is ever alive — and + // a per-provider count is what the verbose line can honestly report. + const workerJobs: ParseJob[] = [] + const workerPaths = new Set() + let workerDiscards = 0 + let pendingBytes = 0 + if (providerName === 'codex' && !readOnly) { + for (const { source, fp } of changedSources) { + if (dateRange && fp.mtimeMs < dateRange.start.getTime()) continue + if (await readCachedCodexResults(source.path)) continue + workerJobs.push({ kind: 'codex', source }) + workerPaths.add(source.path) + pendingBytes += fp.sizeBytes + } + } + const decision = workerJobs.length > 0 + ? decideParseWorkers({ files: workerJobs.length, bytes: pendingBytes }) + : { workers: 0, reason: 'no full parses pending' } + if (providerName === 'codex' && !readOnly && process.env['CODEBURN_VERBOSE'] === '1') { + process.stderr.write(`codeburn: codex parse workers=${decision.workers} (${decision.reason})\n`) + } + let pool: ParseWorkerPool | null = null + if (decision.workers > 0) { + try { + pool = new ParseWorkerPool(decision.workers) + } catch (err) { + process.stderr.write(`codeburn: parse workers unavailable, parsing serially (${err instanceof Error ? err.message : String(err)})\n`) + } + } + const offThread = pool ? parseFilesInOrder(pool, workerJobs) : null + // Parse changed files, update cache let didParse = false // Track which paths have already been cleared this pass so that subsequent @@ -2952,15 +3148,47 @@ async function parseProviderSources( // that pruned-away data is preserved for monotonic monthly totals. if (!provider.durableSources && !clearedPaths.has(source.path)) { delete section.files[source.path] + markCacheDirty(diskCache, providerName, source.path) clearedPaths.add(source.path) } - const parser = provider.createSessionParser(source, parserDedup, dateRange) + // Off-thread results arrive in this order, so the Nth eligible file here is + // the Nth yielded result. A worker decodes against an EMPTY dedup set, so an + // EMPTY key intersection is the proof that a serial parse would have dropped + // nothing either — that, and only that, makes the result installable. On any + // overlap (a forked rollout replaying its parent's token_count history is + // exactly this) the WHOLE file is discarded and re-parsed in-process against + // the real dedup set. Deliberately OUTSIDE the per-file try below: the + // pairing is positional, and a misalignment would install one rollout's + // calls under another's path — a wrong number nobody would ever notice, so + // it fails the run instead of being caught as a parse failure. + let providerCalls: ParsedProviderCall[] | undefined + if (offThread && workerPaths.has(source.path)) { + const result = (await offThread.next()).value + if (result?.ok && result.parsed) { + if (result.parsed.path !== source.path) { + throw new Error(`codex parse worker result out of order: got ${result.parsed.path}, expected ${source.path}`) + } + if (result.parsed.keys.some(k => parserDedup.has(k))) { + workerDiscards++ + } else { + for (const k of result.parsed.keys) parserDedup.add(k) + providerCalls = result.parsed.calls + // The worker never touches the codex cache; publish its entry here, + // in install order, so flushCodexCache writes what serial would. + const write = result.parsed.write + if (write) await writeCachedCodexResults(source.path, write.project, providerCalls, write.fingerprint, write.resume) + } + } + } try { - const providerCalls: ParsedProviderCall[] = [] - for await (const call of parser.parse()) { - providerCalls.push(call) + if (!providerCalls) { + const parser = provider.createSessionParser(source, parserDedup, dateRange) + providerCalls = [] + for await (const call of parser.parse()) { + providerCalls.push(call) + } } const canonicalCalls = await Promise.all(providerCalls.map(canonicalizeProviderCallProject)) const turns = providerCallsToCachedTurns(canonicalCalls) @@ -2997,7 +3225,7 @@ async function parseProviderSources( } } didParse = true - ;(diskCache as { _dirty?: boolean })._dirty = true + markCacheDirty(diskCache, providerName, source.path) } catch (err) { if (isSqliteBusyError(err)) { warnProviderReadFailureOnce(providerName, err) @@ -3010,16 +3238,20 @@ async function parseProviderSources( // on every refresh; it re-parses only if it changes. Empty turns => no // usage contributed. section.files[source.path] = { fingerprint: fp, mcpInventory: [], turns: [], failed: true } - ;(diskCache as { _dirty?: boolean })._dirty = true + markCacheDirty(diskCache, providerName, source.path) warnProviderParseFailure(providerName, source.path, err) continue } } } finally { + await pool?.close() + if (pool && process.env['CODEBURN_VERBOSE'] === '1') { + process.stderr.write(`codeburn: codex parse workers done, ${workerDiscards}/${workerJobs.length} results re-parsed in-process on id overlap\n`) + } if (didParse && providerName === 'codex') await flushCodexCache() if (didParse && providerName === 'antigravity') { const liveIds = new Set(sources.map(s => antigravityCascadeIdFromPath(s.path))) - await flushAntigravityCache(liveIds) + await flushAntigravityCache(liveIds, antigravityCacheDir) } } @@ -3027,14 +3259,14 @@ async function parseProviderSources( // parseAllSessions can fast-check without a getProvider() round-trip. if (!readOnly && provider.durableSources && !section.durable) { section.durable = true - ;(diskCache as { _dirty?: boolean })._dirty = true + markCacheDirty(diskCache, providerName) } if (!readOnly && sources.length > 0 && !provider.durableSources) { for (const cachedPath of Object.keys(section.files)) { if (!allDiscoveredFiles.has(cachedPath)) { delete section.files[cachedPath] - ;(diskCache as { _dirty?: boolean })._dirty = true + markCacheDirty(diskCache, providerName, cachedPath) } } } @@ -3051,7 +3283,7 @@ async function parseProviderSources( .reduce((max, ts) => Math.max(max, ts), 0) if (newestTs > 0 && newestTs < cutoffMs) { delete section.files[cachedPath] - ;(diskCache as { _dirty?: boolean })._dirty = true + markCacheDirty(diskCache, providerName, cachedPath) } } } @@ -3190,7 +3422,15 @@ async function parseProviderSources( const CACHE_TTL_MS = 180_000 const MAX_CACHE_ENTRIES = 10 -const sessionCache = new Map() +type SessionCacheEntry = { + data: ProjectSummary[] + createdAt: number + validatedFrom: number + startMs?: number + endMs?: number + sig?: string +} +const sessionCache = new Map() // Burst reuse for a resident process (codeburn serve). Every payload command // anchors its range end at its own `new Date()`, so two panel fetches issued @@ -3207,15 +3447,16 @@ function parseBurstWindowMs(): number { // A resident process (codeburn serve) can install a validator that answers // "has any watched session root changed since this timestamp?" — typically -// backed by fs.watch over every provider's probeRoots(). While the validator -// reports clean, a previous parse stays reusable well past the burst window, -// bounded by a hard cap so a missed filesystem event self-heals instead of -// pinning stale data forever. Null (the default everywhere but serve) keeps -// reuse strictly inside the burst window. -let parseReuseValidator: ((sinceTs: number) => boolean) | null = null +// backed by fs.watch over every provider's probeRoots(). Clean extends reuse +// to the hard cap, dirty rejects every memo, and unknown (watcher coverage is +// unavailable or began too late) falls back to the ordinary exact TTL / short +// burst rather than disabling caching. Null keeps those ordinary semantics. +export type ParseReuseValidation = 'clean' | 'dirty' | 'unknown' +type ParseReuseValidator = (sinceTs: number) => ParseReuseValidation +let parseReuseValidator: ParseReuseValidator | null = null const VALIDATED_REUSE_CAP_MS = 5 * 60 * 1000 -export function setParseReuseValidator(validator: ((sinceTs: number) => boolean) | null): void { +export function setParseReuseValidator(validator: ParseReuseValidator | null): void { parseReuseValidator = validator } @@ -3227,9 +3468,13 @@ function burstReuse(dateRange: DateRange, sig: string): ProjectSummary[] | null const endMs = dateRange.end.getTime() for (const entry of sessionCache.values()) { if (entry.sig !== sig || entry.startMs !== startMs || entry.endMs === undefined) continue - const age = now - entry.ts + const validation = parseReuseValidator?.(entry.validatedFrom) ?? 'unknown' + // A dirty event during the producing parse must not be hidden even by the + // short burst. Unknown coverage, however, retains that bounded fallback. + if (validation === 'dirty') continue + const age = now - entry.createdAt const insideBurst = age <= windowMs - const validatedClean = parseReuseValidator !== null && age <= VALIDATED_REUSE_CAP_MS && parseReuseValidator(entry.ts) + const validatedClean = validation === 'clean' && age <= VALIDATED_REUSE_CAP_MS if (!insideBurst && !validatedClean) continue if (endMs < entry.endMs || endMs - entry.endMs > Math.max(windowMs, validatedClean ? VALIDATED_REUSE_CAP_MS : 0)) continue return filterProjectsByDateRange(entry.data, dateRange) @@ -3237,34 +3482,36 @@ function burstReuse(dateRange: DateRange, sig: string): ProjectSummary[] | null return null } -function cacheKey(dateRange?: DateRange, providerFilter?: string): string { +function cacheKey(dateRange: DateRange | undefined, providerFilter: string | undefined, claudeDiscoveryRoots: readonly string[]): string { const s = dateRange ? `${dateRange.start.getTime()}:${dateRange.end.getTime()}` : 'none' - // Include the Claude config-dir env so a config change in a long-lived - // process (menubar / GNOME extension / test workers) does not return - // stale data keyed under a previous configuration. - const claudeEnv = (process.env['CLAUDE_CONFIG_DIRS'] ?? '') + '|' + (process.env['CLAUDE_CONFIG_DIR'] ?? '') + // Key on the effective roots, not only their env inputs: GUI consumers can + // change config.json claudeConfigDirs while a resident serve process stays + // alive. Normalized roots also collapse syntactically different inputs that + // discover the same directories. + const claudeRoots = JSON.stringify(claudeDiscoveryRoots) // Proxy attribution (totalProxiedCostUSD) is computed live from proxyPaths and // then cached, so the key must change when that config changes. // Pricing-affecting config participates so a memoized parse (exact-key or // burst-reused in a resident serve process) can never present costs priced // under aliases/overrides/savings the user has since changed. - return `${s}:${providerFilter ?? 'all'}:${claudeEnv}:${getProxyPathsConfigHash()}:${getModelAliasesConfigHash()}:${getPriceOverridesConfigHash()}:${getLocalModelSavingsConfigHash()}` + return `${s}:${providerFilter ?? 'all'}:${claudeRoots}:${getProxyPathsConfigHash()}:${getModelAliasesConfigHash()}:${getPriceOverridesConfigHash()}:${getLocalModelSavingsConfigHash()}` } export function clearSessionCache(): void { sessionCache.clear() + canonicalPathCache.clear() } -function cachePut(key: string, data: ProjectSummary[]) { +function cachePut(key: string, data: ProjectSummary[], parseStartedAt: number) { const now = Date.now() for (const [k, v] of sessionCache) { - if (now - v.ts > CACHE_TTL_MS) sessionCache.delete(k) + if (now - v.createdAt > CACHE_TTL_MS) sessionCache.delete(k) } if (sessionCache.size >= MAX_CACHE_ENTRIES) { - const oldest = [...sessionCache.entries()].sort((a, b) => a[1].ts - b[1].ts)[0] + const oldest = [...sessionCache.entries()].sort((a, b) => a[1].createdAt - b[1].createdAt)[0] if (oldest) sessionCache.delete(oldest[0]) } - sessionCache.set(key, { data, ts: now, ...(putMeta ?? {}) }) + sessionCache.set(key, { data, createdAt: now, validatedFrom: parseStartedAt, ...(putMeta ?? {}) }) putMeta = null } @@ -3707,19 +3954,50 @@ export function isSessionHydrationComplete(): boolean { // chart (gapStart = lastComputedDate + 1 never looks back at them). let readOnlyServedStale = false -export async function parseAllSessions(dateRange?: DateRange, providerFilter?: string): Promise { - const key = cacheKey(dateRange, providerFilter) +export function parseAllSessions(dateRange?: DateRange, providerFilter?: string): Promise { + // Capture synchronously, before the first await. AsyncLocalStorage keeps all + // Codex cache reads, dirty writes, and the final flush on this call-time + // directory even if an embedding host changes the process env mid-parse. + const codexCacheDir = getCodeburnCacheDir() + return withCodexCacheDirectory(codexCacheDir, () => parseAllSessionsInCacheScope(dateRange, providerFilter)) +} + +async function parseAllSessionsInCacheScope(dateRange?: DateRange, providerFilter?: string): Promise { + // Anchor freshness before any config, cache, or session input is read. A + // watched-root event that lands while this parse is in flight must remain + // newer than the resulting memo instead of being blessed retroactively. + const parseStartedAt = Date.now() + const claudeDiscoveryRoots = await getClaudeConfigDirs() + const key = cacheKey(dateRange, providerFilter, claudeDiscoveryRoots) const cached = sessionCache.get(key) - if (cached && Date.now() - cached.ts < CACHE_TTL_MS) return cached.data + if (cached) { + const age = Date.now() - cached.createdAt + const validation = parseReuseValidator?.(cached.validatedFrom) ?? 'unknown' + if ( + validation !== 'dirty' + && (age < CACHE_TTL_MS || (validation === 'clean' && age <= VALIDATED_REUSE_CAP_MS)) + ) return cached.data + } // The signature is the key minus the range: what must match for a burst // reuse (provider, config env, proxy hash) regardless of the now-anchor. - const burstSig = cacheKey(undefined, providerFilter) + const burstSig = cacheKey(undefined, providerFilter, claudeDiscoveryRoots) if (dateRange) { const reused = burstReuse(dateRange, burstSig) if (reused) return reused } - let diskCache = await loadCache() + // Load only the month shards a query over `dateRange` can possibly report + // on. Sessions whose every turn falls outside the range are dropped from the + // report anyway, so skipping their shards changes nothing except the bytes + // read — and a save writes only dirty months, leaving the skipped ones on + // disk untouched (see saveCache). Cross-file dedup is weakened, not broken: + // the pre-seed of `seenMsgIds` / `seenKeys` only covers loaded files, so a key + // that a skipped file also holds is no longer suppressed. Totals are + // unaffected (a suppressed duplicate contributes nothing either way), but for + // a proxied key emitted under two providers the attribution can land on a + // different provider than a full load would pick. + const loadScope = dateRange ? monthScopeForRange(dateRange.start, dateRange.end) : undefined + let diskCache = await loadCache(loadScope) await cleanupOrphanedTempFiles() // Cold-hydration coordination (advisory, cross-process). Engages whenever the @@ -3732,10 +4010,10 @@ export async function parseAllSessions(dateRange?: DateRange, providerFilter?: s // doubt it proceeds unlocked. if (!isCacheComplete(diskCache)) { const hydration = await beginColdHydration(true) - if (hydration.waited) diskCache = await loadCache() + if (hydration.waited) diskCache = await loadCache(loadScope) const isCold = !isCacheComplete(diskCache) try { - return await runParse(key, diskCache, dateRange, providerFilter, { isCold }) + return await runParse(key, diskCache, dateRange, providerFilter, { isCold, burstSig, parseStartedAt }) } finally { await hydration.release() } @@ -3747,20 +4025,20 @@ export async function parseAllSessions(dateRange?: DateRange, providerFilter?: s const priorSnapshot = diskCache const refresh = await acquireCacheRefreshLock() if (refresh.outcome === 'timed-out' || refresh.outcome === 'unavailable') { - return runParse(key, priorSnapshot, dateRange, providerFilter, { readOnly: true }) + return runParse(key, priorSnapshot, dateRange, providerFilter, { readOnly: true, burstSig, parseStartedAt }) } if (refresh.outcome === 'completed-by-other') { - return runParse(key, await loadCache(), dateRange, providerFilter, { readOnly: true }) + return runParse(key, await loadCache(loadScope), dateRange, providerFilter, { readOnly: true, burstSig, parseStartedAt }) } try { // Reload only after ownership is canonical; this closes the lost-update // window between the pre-gate read and the holder's completed publication. - diskCache = await loadCache() - return await runParse(key, diskCache, dateRange, providerFilter, { refreshLock: refresh.handle }) + diskCache = await loadCache(loadScope) + return await runParse(key, diskCache, dateRange, providerFilter, { refreshLock: refresh.handle, burstSig, parseStartedAt }) } catch (err) { if (!(err instanceof RefreshFenceLostError) && !(err instanceof RefreshPublicationUnavailableError)) throw err - return runParse(key, await loadCache(), dateRange, providerFilter, { readOnly: true }) + return runParse(key, await loadCache(loadScope), dateRange, providerFilter, { readOnly: true, burstSig, parseStartedAt }) } finally { await refresh.handle.release() } @@ -3773,14 +4051,16 @@ type RunParseOptions = { isCold?: boolean readOnly?: boolean refreshLock?: RefreshLockHandle + burstSig: string + parseStartedAt: number } async function runParse( key: string, diskCache: SessionCache, - dateRange?: DateRange, - providerFilter?: string, - options: RunParseOptions = {}, + dateRange: DateRange | undefined, + providerFilter: string | undefined, + options: RunParseOptions, ): Promise { const { isCold = false, readOnly = false, refreshLock } = options readOnlyServedStale = false @@ -3798,15 +4078,22 @@ async function runParse( providerGroups.set(source.provider, existing) } - // Cold-run robustness: persist partial progress during a long parse (throttled) - // so a run interrupted before the single end-of-parse save still leaves a warm - // cache behind. saveCache is atomic (temp + rename) and clears `_dirty`, so this + // Cold-run robustness: persist partial progress during a long parse so a run + // interrupted before the single end-of-parse save still leaves a warm cache + // behind. Triggered by files parsed rather than elapsed time: the cost of a + // save scales with the corpus, not the clock, so a wall-clock throttle made a + // slow cold parse rewrite the whole (growing) cache every few seconds. At this + // interval a ~18k-file cold parse saves under a dozen times. saveCache is + // atomic (temp + rename) and writes only the dirty provider shards, so this // never races the final save below. + let filesSinceSave = 0 let lastSaveAt = Date.now() const saveProgress = async (): Promise => { if (!isCold || readOnly) return - if (!(diskCache as { _dirty?: boolean })._dirty) return - if (Date.now() - lastSaveAt < PROGRESS_SAVE_THROTTLE_MS) return + if (!isCacheDirty(diskCache)) return + filesSinceSave++ + if (filesSinceSave < PROGRESS_SAVE_FILE_INTERVAL && Date.now() - lastSaveAt < PROGRESS_SAVE_MAX_INTERVAL_MS) return + filesSinceSave = 0 lastSaveAt = Date.now() try { await saveCache(diskCache) } catch { /* best-effort partial save */ } } @@ -3890,7 +4177,7 @@ async function runParse( // partial saves keep `complete: false` and the next launch resumes cold. const wasComplete = isCacheComplete(diskCache) if (!readOnly && !wasComplete) diskCache.complete = true - if (!readOnly && ((diskCache as { _dirty?: boolean })._dirty || !wasComplete)) { + if (!readOnly && (isCacheDirty(diskCache) || !wasComplete)) { try { const published = await saveCache(diskCache, refreshLock?.verifyStillOwner) if (!published) throw new RefreshFenceLostError() @@ -3942,7 +4229,7 @@ async function runParse( const result = Array.from(mergedMap.values()).sort((a, b) => b.totalCostUSD - a.totalCostUSD) correlateCrossProviderPrSessions(result) - if (dateRange) setCachePutMeta({ startMs: dateRange.start.getTime(), endMs: dateRange.end.getTime(), sig: cacheKey(undefined, providerFilter) }) - cachePut(key, result) + if (dateRange) setCachePutMeta({ startMs: dateRange.start.getTime(), endMs: dateRange.end.getTime(), sig: options.burstSig }) + cachePut(key, result, options.parseStartedAt) return result } diff --git a/src/providers/antigravity.ts b/src/providers/antigravity.ts index 31449567..fad0f39d 100644 --- a/src/providers/antigravity.ts +++ b/src/providers/antigravity.ts @@ -1,11 +1,12 @@ import { readdir, readFile, mkdir, stat, open, rename, unlink } from 'fs/promises' import { execFile } from 'child_process' import { randomBytes } from 'crypto' -import { basename, join } from 'path' +import { basename, join, resolve } from 'path' import { homedir } from 'os' import { fileURLToPath } from 'url' import https from 'https' +import { getCodeburnCacheDir } from '../cache-dir.js' import { calculateCost } from '../models.js' import { isSqliteAvailable, isSqliteBusyError, openDatabase } from '../sqlite.js' import type { ProbeRoot, Provider, SessionSource, SessionParser, ParsedProviderCall } from './types.js' @@ -161,8 +162,17 @@ type AntigravityGenMetadataRow = { const cachedServers = new Map() const cachedModelMaps = new Map() -let memCache: AntigravityCache | null = null -let cacheDirty = false +type AntigravityCacheState = { cache: AntigravityCache; dirty: boolean } +const cacheStates = new Map() + +// Dropped by the resident RSS guard. A dirty state holds cascades not yet on +// disk, so it stays resident until its own flush publishes it. +export function clearAntigravityCacheStates(): void { + for (const [dir, state] of cacheStates) { + if (!state.dirty) cacheStates.delete(dir) + } +} + let httpsAgent: https.Agent | undefined const protoTextDecoder = new TextDecoder('utf-8', { fatal: false }) @@ -175,16 +185,16 @@ function getAgent(): https.Agent { return httpsAgent } -function getCacheDir(): string { - return process.env['CODEBURN_CACHE_DIR'] ?? join(homedir(), '.cache', 'codeburn') +function currentCacheDir(): string { + return resolve(getCodeburnCacheDir()) } -function getCachePath(): string { - return join(getCacheDir(), 'antigravity-results.json') +function getCachePath(cacheDir: string): string { + return join(cacheDir, 'antigravity-results.json') } export function getAntigravityStatusLineEventsPath(): string { - return join(getCacheDir(), 'antigravity-statusline.jsonl') + return join(getCodeburnCacheDir(), 'antigravity-statusline.jsonl') } function execFileText(command: string, args: string[], timeout = 3000): Promise { @@ -323,22 +333,30 @@ export function extractAntigravityGeneratorMetadata(resp: unknown): GeneratorMet return Array.isArray(metadata) ? metadata : [] } -async function loadCache(): Promise { - if (memCache) return memCache +async function loadCache(cacheDir: string): Promise { + const inMemory = cacheStates.get(cacheDir) + if (inMemory) return inMemory try { - const raw = await readFile(getCachePath(), 'utf-8') + const raw = await readFile(getCachePath(cacheDir), 'utf-8') const cache = JSON.parse(raw) as AntigravityCache if (cache.version === CACHE_VERSION && cache.cascades && typeof cache.cascades === 'object') { - memCache = cache - return cache + const state = { cache, dirty: false } + cacheStates.set(cacheDir, state) + return state } } catch { /* no cache or invalid */ } - memCache = { version: CACHE_VERSION, cascades: {} } - return memCache + const state: AntigravityCacheState = { + cache: { version: CACHE_VERSION, cascades: {} }, + dirty: false, + } + cacheStates.set(cacheDir, state) + return state } -async function flushCache(liveCascadeIds?: Set): Promise { - if (!memCache) return +async function flushCache(liveCascadeIds?: Set, cacheDir = currentCacheDir()): Promise { + const state = cacheStates.get(cacheDir) + if (!state) return + const memCache = state.cache // If the caller supplied liveCascadeIds, we must run the eviction step // even when no cascade was added or updated this run; otherwise deleted // .pb files would persist in the cache forever once it stops getting @@ -348,16 +366,14 @@ async function flushCache(liveCascadeIds?: Set): Promise { for (const id of Object.keys(memCache.cascades)) { if (!liveCascadeIds.has(id)) { delete memCache.cascades[id] - cacheDirty = true + state.dirty = true } } } - if (!cacheDirty) return + if (!state.dirty) return try { - - const dir = getCacheDir() - await mkdir(dir, { recursive: true }) - const finalPath = getCachePath() + await mkdir(cacheDir, { recursive: true }) + const finalPath = getCachePath(cacheDir) const tempPath = `${finalPath}.${randomBytes(8).toString('hex')}.tmp` const handle = await open(tempPath, 'w', 0o600) try { @@ -371,7 +387,7 @@ async function flushCache(liveCascadeIds?: Set): Promise { } catch { try { await unlink(tempPath) } catch { /* cleanup */ } } - cacheDirty = false + state.dirty = false } catch { /* best-effort */ } } @@ -1009,7 +1025,7 @@ export async function recordAntigravityStatusLinePayload(input: unknown): Promis if (!event) return false const path = getAntigravityStatusLineEventsPath() - await mkdir(getCacheDir(), { recursive: true, mode: 0o700 }) + await mkdir(getCodeburnCacheDir(), { recursive: true, mode: 0o700 }) const fd = await open(path, 'a', 0o600) try { await fd.appendFile(`${JSON.stringify(event)}\n`, { encoding: 'utf-8' }) @@ -1173,7 +1189,9 @@ export async function snapshotAntigravityStatusLinePayload(input: unknown): Prom const s = await stat(source.path).catch(() => null) if (!s) return false - const cache = await loadCache() + const cacheDir = currentCacheDir() + const state = await loadCache(cacheDir) + const cache = state.cache const cached = cache.cascades[cascadeId] if (cached && cached.mtimeMs === s.mtimeMs && cached.sizeBytes === s.size && cached.calls.length > 0) { return true @@ -1195,8 +1213,8 @@ export async function snapshotAntigravityStatusLinePayload(input: unknown): Prom sizeBytes: s.size, calls: snapshotCalls, } - cacheDirty = true - await flushCache() + state.dirty = true + await flushCache(undefined, cacheDir) return cache.cascades[cascadeId]!.calls.length > 0 } catch { return false @@ -1300,7 +1318,8 @@ function createParser(source: SessionSource, seenKeys: Set): SessionPars } const cascadeId = antigravityCascadeIdFromPath(source.path) - const cache = await loadCache() + const state = await loadCache(currentCacheDir()) + const cache = state.cache const s = await stat(source.path).catch(() => null) if (!s) return @@ -1331,7 +1350,7 @@ function createParser(source: SessionSource, seenKeys: Set): SessionPars sizeBytes: s.size, calls: sqliteResults, } - cacheDirty = true + state.dirty = true for (const call of sqliteResults) { if (seenKeys.has(call.deduplicationKey)) continue @@ -1384,7 +1403,7 @@ function createParser(source: SessionSource, seenKeys: Set): SessionPars sizeBytes: s.size, calls: results, } - cacheDirty = true + state.dirty = true for (const call of results) { if (seenKeys.has(call.deduplicationKey)) continue @@ -1445,8 +1464,8 @@ export function createAntigravityProvider(): Provider { } } -export async function flushAntigravityCache(liveCascadeIds?: Set): Promise { - await flushCache(liveCascadeIds) +export async function flushAntigravityCache(liveCascadeIds?: Set, cacheDir?: string): Promise { + await flushCache(liveCascadeIds, cacheDir ? resolve(cacheDir) : currentCacheDir()) } export const antigravity = createAntigravityProvider() diff --git a/src/providers/codex.ts b/src/providers/codex.ts index 7191c6fe..3328a3ef 100644 --- a/src/providers/codex.ts +++ b/src/providers/codex.ts @@ -6,7 +6,7 @@ import { homedir } from 'os' import { readSessionLines } from '../fs-utils.js' import { calculateCost } from '../models.js' -import { readCachedCodexResults, writeCachedCodexResults, getCachedCodexProject, fingerprintFile } from '../codex-cache.js' +import { readCachedCodexResults, writeCachedCodexResults, getCachedCodexProject, fingerprintFile, type CodexFileFingerprint } from '../codex-cache.js' import { normalizeContentBlocks } from '../content-utils.js' import { estimateTokensFromChars } from '../token-estimate.js' import type { ToolCall } from '../types.js' @@ -569,52 +569,131 @@ function resolveModel(info: CodexEntry['payload'], sessionModel?: string): strin return firstModelString(info?.model, info?.info?.model, info?.info?.model_name, sessionModel) ?? 'gpt-5' } -function createParser(source: SessionSource, seenKeys: Set): SessionParser { +// Everything the single-pass decode carries across a `task_started` boundary. +// A rollout is append-only, so recording this at such a boundary (where the +// previous task has been flushed to `results` and the per-task accumulators are +// empty) lets a later run restart there and produce byte-identical output +// instead of re-reading the whole file. Every field the loop below reads from a +// PRIOR line must appear here, or a resumed decode silently diverges. +type CodexResumeState = { + sessionModel?: string + sessionId: string + sessionCwd?: string + forkedFromId: string + forkCutoff: string + prevCumulativeTotal: number | null + prevInput: number + prevCached: number + prevOutput: number + prevReasoning: number + pendingTools: string[] + pendingToolSequence: ToolCall[][] + pendingUserMessage: string + pendingOutputChars: number + pendingLocAdded: number + pendingLocRemoved: number + pendingEditFailed: number + estCounter: number + turnCounter: number + currentTurnId: string + taskStartedAt?: number +} + +// The state comes back off our own JSON cache; a truncated or hand-edited file +// must fall back to a full re-parse rather than decode against nonsense. +function isResumeState(value: unknown): value is CodexResumeState { + if (!value || typeof value !== 'object') return false + const v = value as Record + return typeof v['sessionId'] === 'string' + && typeof v['forkedFromId'] === 'string' + && typeof v['forkCutoff'] === 'string' + && (v['prevCumulativeTotal'] === null || typeof v['prevCumulativeTotal'] === 'number') + && typeof v['prevInput'] === 'number' + && typeof v['prevCached'] === 'number' + && typeof v['prevOutput'] === 'number' + && typeof v['prevReasoning'] === 'number' + && Array.isArray(v['pendingTools']) + && Array.isArray(v['pendingToolSequence']) + && typeof v['pendingUserMessage'] === 'string' + && typeof v['pendingOutputChars'] === 'number' + && typeof v['pendingLocAdded'] === 'number' + && typeof v['pendingLocRemoved'] === 'number' + && typeof v['pendingEditFailed'] === 'number' + && typeof v['estCounter'] === 'number' + && typeof v['turnCounter'] === 'number' + && typeof v['currentTurnId'] === 'string' +} + +/** What the serial path would have written to the codex cache for one file. */ +export type CodexCacheWrite = { + project: string + fingerprint: CodexFileFingerprint + resume?: { offset: number; state: unknown; callCount: number } +} + +// When `capture` is passed the parse is a whole-file decode that never touches +// the codex cache: no hit lookup (so no resume), and the entry it would have +// written comes back through `capture` for the caller to install. That is what +// lets a worker thread run this exact decode without owning the cache module's +// per-directory state. +function createParser(source: SessionSource, seenKeys: Set, capture?: { write?: CodexCacheWrite }): SessionParser { return { async *parse(): AsyncGenerator { - const cached = await readCachedCodexResults(source.path) - if (cached) { - for (const call of cached) { + const hit = capture ? null : await readCachedCodexResults(source.path) + if (hit?.kind === 'exact') { + for (const call of hit.calls) { if (seenKeys.has(call.deduplicationKey)) continue seenKeys.add(call.deduplicationKey) yield call } return } + const resume = hit && isResumeState(hit.state) + ? { offset: hit.offset, state: hit.state, calls: hit.calls.slice(0, hit.callCount) } + : null const fp = await fingerprintFile(source.path) if (!fp) return - let sessionModel: string | undefined - let sessionId = '' - let sessionCwd: string | undefined - let forkedFromId = '' - let forkCutoff = '' + let sessionModel: string | undefined = resume?.state.sessionModel + let sessionId = resume?.state.sessionId ?? '' + let sessionCwd: string | undefined = resume?.state.sessionCwd + let forkedFromId = resume?.state.forkedFromId ?? '' + let forkCutoff = resume?.state.forkCutoff ?? '' // Null sentinel rather than `0` so the FIRST event is never confused // with a duplicate. A session that only emits last_token_usage (no // total_token_usage) reports cumulativeTotal=0 on every event; with a // 0-initialized prev, the first event would have matched and been // dropped. Once we've observed any event, we record its cumulative // total and dedup on equality regardless of whether it is zero. - let prevCumulativeTotal: number | null = null - let prevInput = 0 - let prevCached = 0 - let prevOutput = 0 - let prevReasoning = 0 - let pendingTools: string[] = [] - let pendingToolSequence: ToolCall[][] = [] - let pendingUserMessage = '' - let pendingOutputChars = 0 + let prevCumulativeTotal: number | null = resume?.state.prevCumulativeTotal ?? null + let prevInput = resume?.state.prevInput ?? 0 + let prevCached = resume?.state.prevCached ?? 0 + let prevOutput = resume?.state.prevOutput ?? 0 + let prevReasoning = resume?.state.prevReasoning ?? 0 + let pendingTools: string[] = resume ? [...resume.state.pendingTools] : [] + let pendingToolSequence: ToolCall[][] = resume ? [...resume.state.pendingToolSequence] : [] + let pendingUserMessage = resume?.state.pendingUserMessage ?? '' + let pendingOutputChars = resume?.state.pendingOutputChars ?? 0 // Rich-session-capture: edit LOC deltas and failed-patch count accumulated // across a turn's patch_apply_end events, flushed onto the turn's call. - let pendingLocAdded = 0 - let pendingLocRemoved = 0 - let pendingEditFailed = 0 - let estCounter = 0 - let turnCounter = 0 - let currentTurnId = `${sessionId}:t0` + let pendingLocAdded = resume?.state.pendingLocAdded ?? 0 + let pendingLocRemoved = resume?.state.pendingLocRemoved ?? 0 + let pendingEditFailed = resume?.state.pendingEditFailed ?? 0 + let estCounter = resume?.state.estCounter ?? 0 + let turnCounter = resume?.state.turnCounter ?? 0 + let currentTurnId = resume?.state.currentTurnId ?? `${sessionId}:t0` let sawAnyLine = false const results: ParsedProviderCall[] = [] + // Calls already decoded before the resume boundary. They pass through the + // same cross-provider dedup a full decode would have applied to them. + if (resume) { + for (const call of resume.calls) { + if (seenKeys.has(call.deduplicationKey)) continue + seenKeys.add(call.deduplicationKey) + results.push(call) + } + } // Calls decoded since the last task_started, held back so task_complete can // stamp active/toolWait timing before they are appended to results. Emitting // a task only once its timing is known keeps single-pass and split/resume @@ -623,15 +702,25 @@ function createParser(source: SessionSource, seenKeys: Set): SessionPars let pendingTaskCalls: ParsedProviderCall[] = [] let taskGeneratedTokens = 0 let taskToolIntervals: Array<[number, number]> = [] - let taskStartedAt: number | undefined + let taskStartedAt: number | undefined = resume?.state.taskStartedAt const openToolStarts = new Map() + // Resume point for the NEXT run, refreshed at every task boundary. + const tracker = { lastCompleteLineOffset: resume?.offset ?? 0 } + let resumeOffset = resume?.offset ?? 0 + let resumeState: CodexResumeState | null = resume?.state ?? null + let resumeCallCount = results.length + // Stream the session file line by line. Heavy Codex sessions can exceed // 250 MB on disk; reading the entire file into a string would either hit // the readSessionFile cap or push V8 toward its 512 MB string limit // after split('\n'). readSessionLines streams raw buffers and hands // huge lines to the compact parser without full string conversion. - for await (const rawLine of readSessionLines(source.path, undefined, { largeLineAsBuffer: true })) { + for await (const rawLine of readSessionLines(source.path, undefined, { + largeLineAsBuffer: true, + byteOffsetTracker: tracker, + ...(resume ? { startByteOffset: resume.offset } : {}), + })) { sawAnyLine = true const entry = parseCodexLine(rawLine) if (!entry) continue @@ -684,6 +773,33 @@ function createParser(source: SessionSource, seenKeys: Set): SessionPars const startedAt = entry.timestamp ? Date.parse(entry.timestamp) : NaN taskStartedAt = Number.isFinite(startedAt) ? startedAt : undefined openToolStarts.clear() + // Everything decoded so far is now in `results` and the per-task + // accumulators are empty: a clean restart point for an appended tail. + resumeOffset = tracker.lastCompleteLineOffset + resumeCallCount = results.length + resumeState = { + ...(sessionModel !== undefined ? { sessionModel } : {}), + sessionId, + ...(sessionCwd !== undefined ? { sessionCwd } : {}), + forkedFromId, + forkCutoff, + prevCumulativeTotal, + prevInput, + prevCached, + prevOutput, + prevReasoning, + pendingTools: [...pendingTools], + pendingToolSequence: [...pendingToolSequence], + pendingUserMessage, + pendingOutputChars, + pendingLocAdded, + pendingLocRemoved, + pendingEditFailed, + estCounter, + turnCounter, + currentTurnId, + ...(taskStartedAt !== undefined ? { taskStartedAt } : {}), + } continue } @@ -997,13 +1113,23 @@ function createParser(source: SessionSource, seenKeys: Set): SessionPars // If the stream yielded nothing the file was unreadable, oversized, or // empty. Skip cache write so a transient failure can't pin an empty - // result set against a fingerprint that would otherwise be re-parsed. - if (!sawAnyLine) return + // result set against a fingerprint that would otherwise be re-parsed. On a + // resume the earlier calls are still valid output, so serve them - but + // still leave the cache entry alone. + if (!sawAnyLine) { + if (resume) for (const call of results) yield call + return + } // Flush the final task, which has no following task_started to trigger it. results.push(...pendingTaskCalls) - await writeCachedCodexResults(source.path, source.project, results, fp) + const resumeWrite = resumeState ? { offset: resumeOffset, state: resumeState, callCount: resumeCallCount } : undefined + if (capture) { + capture.write = { project: source.project, fingerprint: fp, ...(resumeWrite ? { resume: resumeWrite } : {}) } + } else { + await writeCachedCodexResults(source.path, source.project, results, fp, resumeWrite) + } for (const call of results) { yield call @@ -1012,6 +1138,20 @@ function createParser(source: SessionSource, seenKeys: Set): SessionPars } } +export type CodexFullParse = { calls: ParsedProviderCall[]; write?: CodexCacheWrite } + +/// Decode one rollout end to end, exactly as the serial path does for a file +/// with no cache entry, without reading or writing the codex cache. `seenKeys` +/// is the dedup set the decode runs against — pass an empty one off-thread and +/// let the caller prove no earlier file claimed any of the keys before +/// installing the result. +export async function parseCodexFileFull(source: SessionSource, seenKeys: Set): Promise { + const capture: { write?: CodexCacheWrite } = {} + const calls: ParsedProviderCall[] = [] + for await (const call of createParser(source, seenKeys, capture).parse()) calls.push(call) + return { calls, ...(capture.write ? { write: capture.write } : {}) } +} + export function createCodexProvider(codexDir?: string): Provider { const dir = getCodexDir(codexDir) diff --git a/src/providers/kiro.ts b/src/providers/kiro.ts index 6723f723..4e0308d9 100644 --- a/src/providers/kiro.ts +++ b/src/providers/kiro.ts @@ -5,6 +5,7 @@ import { basename, dirname, extname, join } from 'path' import { homedir } from 'os' import { readSessionFile } from '../fs-utils.js' +import { flatSlice, flatString } from '../content-utils.js' import { calculateCost } from '../models.js' import { estimateTokensFromChars } from '../token-estimate.js' import type { ToolCall } from '../types.js' @@ -98,7 +99,10 @@ function extractToolNames(content: string): string[] { let match while ((match = regex.exec(content)) !== null) { const name = match[1]!.trim() - tools.push(toolNameMap[name] ?? name) + // flatString: regex match groups are V8 SlicedStrings that retain the + // ENTIRE subject string — storing them in the session cache would pin + // every scanned assistant-content buffer. Mapped names are flat literals. + tools.push(toolNameMap[name] ?? flatString(name)) } return tools } @@ -217,7 +221,7 @@ function parseChatFile(data: KiroChatFile, sessionId: string, project: string, s if (msg.role === 'human') { if (msg.content.startsWith('')) continue inputChars += msg.content.length - pendingUserMessage = msg.content.slice(0, 500) + pendingUserMessage = flatSlice(msg.content, 500) } if (msg.role === 'bot') { const msgTools = extractToolNames(msg.content) @@ -296,7 +300,7 @@ function parseModernExecution(data: KiroModernExecution, sourcePath: string, see if (directInput) { inputChars += directInput.length - pendingUserMessage = directInput.slice(0, 500) + pendingUserMessage = flatSlice(directInput, 500) } if (directOutput) { @@ -328,7 +332,7 @@ function parseModernExecution(data: KiroModernExecution, sourcePath: string, see if (role === 'human' || role === 'user') { if (!text) continue inputChars += text.length - pendingUserMessage = text.slice(0, 500) + pendingUserMessage = flatSlice(text, 500) } else if (role === 'bot' || role === 'assistant' || role === 'ai' || role === 'model') { if (text) outputChars += text.length if (text || tools.length > 0) hasOutputActivity = true @@ -506,6 +510,7 @@ function parseCliSession(meta: KiroCliSessionMeta, entries: KiroCliEntry[], seen userMessage: pendingUserMessage, sessionId, project, + ...(meta.cwd ? { projectPath: meta.cwd } : {}), }) turnIndex++ } @@ -526,7 +531,7 @@ function parseCliSession(meta: KiroCliSessionMeta, entries: KiroCliEntry[], seen for (const item of content) { const rec = asRecord(item) if (rec && rec['kind'] === 'text' && typeof rec['data'] === 'string') { - pendingUserMessage = (rec['data'] as string).slice(0, 500) + pendingUserMessage = flatSlice(rec['data'] as string, 500) inputChars += (rec['data'] as string).length } } @@ -605,7 +610,7 @@ async function parseWorkspaceSession(record: Record, source: Se const text = extractText(msg['content']) if (role === 'user' && text) { inputChars += text.length - pendingUserMessage = text.slice(0, 500) + pendingUserMessage = flatSlice(text, 500) } else if (role === 'assistant' && !execBacked && text && text !== 'On it.') { // An item carrying an executionId is execution-backed: its content is // counted from the execution file, so counting it here would double-count. @@ -662,6 +667,9 @@ async function parseWorkspaceSession(record: Record, source: Se deduplicationKey: dedupKey, userMessage: pendingUserMessage, sessionId, + ...(typeof record['workspaceDirectory'] === 'string' && record['workspaceDirectory'] + ? { projectPath: record['workspaceDirectory'] as string } + : {}), }) return results @@ -774,6 +782,7 @@ async function parseV2Session(source: SessionSource, seenKeys: Set): Pro userMessage: turnUserMessage, sessionId, project: source.project, + ...(meta.workspacePaths?.[0] ? { projectPath: meta.workspacePaths[0] } : {}), }) } } @@ -794,7 +803,7 @@ async function parseV2Session(source: SessionSource, seenKeys: Set): Pro // for the upcoming turn_start. if (inTurn) flushTurn() const text = typeof payload['content'] === 'string' ? payload['content'] as string : extractText(payload['content']) - pendingUserMessage = text.slice(0, 500) + pendingUserMessage = flatSlice(text, 500) pendingUserChars = text.length } else if (type === 'turn_start') { if (inTurn) flushTurn() diff --git a/src/serve.ts b/src/serve.ts index dc80e391..20ec0a6c 100644 --- a/src/serve.ts +++ b/src/serve.ts @@ -1,8 +1,11 @@ import { watch, type FSWatcher } from 'fs' -import { stat } from 'fs/promises' +import { readFile, stat } from 'fs/promises' +import { createHash } from 'crypto' import { createInterface } from 'readline' import type { Command } from 'commander' +import { getConfigFilePath } from './config.js' +import type { ParseReuseValidation } from './parser.js' // --------------------------------------------------------------------------- // codeburn serve --stdio: a resident query server for the desktop app. @@ -28,16 +31,77 @@ import type { Command } from 'commander' // already guards between processes. // --------------------------------------------------------------------------- -// First-token allowlist of the app's heavy read queries. Deliberately absent: -// every config mutation (currency, model-alias set, budget, price-override, -// proxy-path, plan), export (writes files), share/devices (network + pairing -// state), menubar/web/mcp/guard/sync/act (process management or writes). // Past this resident-set size the serve loop drops its in-memory memos and // re-parses on the next request. 3GB leaves generous room for the largest // observed corpora while bounding a pathological one. const SERVE_MAX_RSS_BYTES = 3 * 1024 * 1024 * 1024 -const SERVE_COMMANDS = new Set(['status', 'overview', 'models', 'sessions', 'compare', 'yield', 'spend', 'optimize', 'audit']) +type OutputMemoEntry = { + createdAt: number + validatedFrom: number + output: string + configFingerprint: string +} + +// Kept as a small seam so the ordering contract can be tested without relying +// on filesystem watcher scheduling: an event arriving while a parse is in +// flight must be newer than the memo produced by that parse. +export function createOutputMemoEntry( + parseStartedAt: number, + parseCompletedAt: number, + output: string, + configFingerprint: string, +): OutputMemoEntry { + return { createdAt: parseCompletedAt, validatedFrom: parseStartedAt, output, configFingerprint } +} + +type ServeOptionKind = 'flag' | 'value' + +// This is intentionally a positive, command-specific option schema rather +// than a shared denylist. If a command later gains a write-capable option it +// remains a normal one-shot CLI action until it is explicitly reviewed here. +// The entries mirror the Commander definitions in main.ts. In particular, +// optimize omits its apply-only surface (--apply, --yes, --dry-run, --only). +const SERVE_OPTIONS: Readonly>>> = { + status: { + '--format': 'value', '--scope': 'value', '--provider': 'value', '--project': 'value', + '--exclude': 'value', '--period': 'value', '--day': 'value', '--from': 'value', + '--to': 'value', '--days': 'value', '--no-optimize': 'flag', '--no-timeline': 'flag', + '--claude-config-source': 'value', + }, + overview: { + '-p': 'value', '--period': 'value', '--from': 'value', '--to': 'value', + '--provider': 'value', '--project': 'value', '--exclude': 'value', '--no-color': 'flag', + }, + models: { + '-p': 'value', '--period': 'value', '--from': 'value', '--to': 'value', + '--provider': 'value', '--task': 'value', '--by-task': 'flag', '--by-agent': 'flag', + '--top': 'value', '--min-cost': 'value', '--no-totals': 'flag', '--format': 'value', + }, + sessions: { + '-p': 'value', '--period': 'value', '--from': 'value', '--to': 'value', + '--provider': 'value', '--format': 'value', '--by-pr': 'flag', '--no-pager': 'flag', + }, + compare: { + '-p': 'value', '--period': 'value', '--provider': 'value', '--format': 'value', + '--model-a': 'value', '--model-b': 'value', + }, + yield: { + '-p': 'value', '--period': 'value', '--provider': 'value', '--format': 'value', + }, + spend: { + '-p': 'value', '--period': 'value', '--from': 'value', '--to': 'value', + '--provider': 'value', '--format': 'value', + }, + optimize: { + '-p': 'value', '--period': 'value', '--from': 'value', '--to': 'value', + '--provider': 'value', '--format': 'value', '--json': 'flag', + }, + audit: { + '-p': 'value', '--period': 'value', '--from': 'value', '--to': 'value', + '--provider': 'value', '--format': 'value', + }, +} type ServeRequest = { id: string | number; args: string[] } @@ -50,30 +114,73 @@ function isServeRequest(value: unknown): value is ServeRequest { function allowed(args: string[]): boolean { const first = args[0] - if (!first || !SERVE_COMMANDS.has(first)) return false - // No request may smuggle a second positional that turns a read into - // something else; the allowed commands take flags only. - return args.slice(1).every((a, i, all) => a.startsWith('-') || (i > 0 && all[i - 1]!.startsWith('--'))) + if (!first) return false + const options = SERVE_OPTIONS[first] + if (!options) return false + + // Served commands have no positional arguments. Long options may use the + // standard --name=value form; otherwise every value must immediately + // follow an option declared as value-bearing in that command's schema. + for (let i = 1; i < args.length; i++) { + const token = args[i]! + const separator = token.startsWith('--') ? token.indexOf('=') : -1 + const option = separator >= 0 ? token.slice(0, separator) : token + const inlineValue = separator >= 0 + const kind = options[option] + if (!kind) return false + if (kind === 'flag') { + if (inlineValue) return false + continue + } + if (inlineValue) continue + const value = args[++i] + if (value === undefined || value.startsWith('-')) return false + } + return true } class ExitSignal extends Error { constructor(public readonly code: number) { super(`exit ${code}`) } } -/// Run one argv through a fresh program, capturing everything the command -/// writes to stdout. process.exit inside a handler is converted to a thrown -/// ExitSignal so a failing request can never take the server down. -async function runCaptured(buildProgram: () => Command, args: string[]): Promise<{ output: string; code: number }> { +function chunkToString(chunk: unknown, encoding: unknown): string { + if (typeof chunk === 'string') return chunk + if (chunk instanceof Uint8Array) { + return Buffer.from(chunk).toString(typeof encoding === 'string' ? encoding as BufferEncoding : 'utf8') + } + return String(chunk) +} + +function finishWrite(rest: unknown[]): void { + const callback = rest[rest.length - 1] + if (typeof callback === 'function') (callback as () => void)() +} + +/// Run one argv through a fresh program, capturing command stdout for the +/// final response and forwarding command stderr as progress. process.exit +/// inside a handler is converted to a thrown ExitSignal so a failing request +/// can never take the server down. +async function runCaptured( + buildProgram: () => Command, + args: string[], + onProgress: (progress: string) => void, +): Promise<{ output: string; code: number }> { const chunks: string[] = [] const originalWrite = process.stdout.write.bind(process.stdout) + const originalErrorWrite = process.stderr.write.bind(process.stderr) const originalExit = process.exit.bind(process) process.stdout.write = ((chunk: unknown, ...rest: unknown[]) => { - chunks.push(typeof chunk === 'string' ? chunk : String(chunk)) - const last = rest[rest.length - 1] - if (typeof last === 'function') (last as () => void)() + chunks.push(chunkToString(chunk, rest[0])) + finishWrite(rest) return true }) as typeof process.stdout.write + process.stderr.write = ((chunk: unknown, ...rest: unknown[]) => { + const progress = chunkToString(chunk, rest[0]) + if (progress) onProgress(progress) + finishWrite(rest) + return true + }) as typeof process.stderr.write process.exit = ((code?: number) => { throw new ExitSignal(code ?? 0) }) as typeof process.exit try { @@ -86,19 +193,57 @@ async function runCaptured(buildProgram: () => Command, args: string[]): Promise throw err } finally { process.stdout.write = originalWrite + process.stderr.write = originalErrorWrite process.exit = originalExit } } +/// A cheap per-request fingerprint for the configuration that affects query +/// rendering and aggregation. Hashing the small config file tracks effective +/// content rather than filesystem churn: a byte-identical rewrite keeps the +/// memo hot, while any real change invalidates immediately. A missing config +/// is a stable state; every other read failure fails closed (no memo reuse). +async function getConfigFingerprint(): Promise { + const path = getConfigFilePath() + try { + const content = await readFile(path) + const digest = createHash('sha256').update(content).digest('hex') + return `${path}\u0000sha256:${digest}` + } catch (err) { + if ((err as NodeJS.ErrnoException).code === 'ENOENT') return `${path}\u0000missing` + return null + } +} + /// Watch every provider's probe roots (the same paths codeburn doctor reports /// as "where discovery looks") so the parse-reuse validator can answer "did /// any session data change since T?" without a stat sweep. macOS fs.watch -/// rides FSEvents and supports recursive directory watches; a root that fails -/// to watch is simply not covered, which only shortens reuse (the burst -/// window and the hard cap still apply), never staleness. -async function startRootWatchers(): Promise<{ startedAt: number; lastEventAt: () => number; close: () => void }> { +/// rides FSEvents and supports recursive directory watches. A probe failure or +/// a watch failure for an existing root disables event-driven reuse for this +/// generation; a root absent at setup is rechecked by the parser's hard cap. +type RootWatcherState = { + startedAt: number + lastEventAt: () => number + healthy: () => boolean + close: () => void +} + +export function classifyRootReuse( + sinceTs: number, + state: { startedAt: number; lastEventAt: number; healthy: boolean }, +): ParseReuseValidation { + // A known event is conclusive even if watcher coverage degraded afterward. + // Unknown means only that no dirty evidence exists and cleanliness cannot be + // established for the whole interval. + if (state.lastEventAt >= sinceTs) return 'dirty' + if (!state.healthy || sinceTs < state.startedAt) return 'unknown' + return 'clean' +} + +async function startRootWatchers(): Promise { let lastEventAt = 0 - const startedAt = Date.now() + let healthy = true + let closed = false const watchers: FSWatcher[] = [] try { const { getAllProviders } = await import('./providers/index.js') @@ -108,21 +253,54 @@ async function startRootWatchers(): Promise<{ startedAt: number; lastEventAt: () if (!provider.probeRoots) continue try { for (const root of await provider.probeRoots()) roots.add(root.path) - } catch { /* a failing probe just goes unwatched */ } + } catch { + // An unknown probe result could hide an existing input root, so no + // global all-roots-quiet claim is safe for this watcher generation. + healthy = false + } } for (const root of roots) { + let info: Awaited> + try { + info = await stat(root) + } catch (err) { + // An absent discovery root contains no sessions at arm time. If it is + // created later there is no child watcher to see that creation, so the + // parser's hard reuse cap remains the eventual revalidation backstop. + // Other stat failures mean an existing input could be uncovered. + if ((err as NodeJS.ErrnoException).code !== 'ENOENT') healthy = false + continue + } try { - const info = await stat(root) const watcher = watch(root, { recursive: info.isDirectory() }, () => { lastEventAt = Date.now() }) - watcher.on('error', () => { /* dropped watcher = shorter reuse, never staleness */ }) + watcher.on('error', () => { healthy = false }) watchers.push(watcher) - } catch { /* nonexistent root: nothing to watch */ } + } catch { + // stat proved this input exists, so failing to arm it invalidates the + // global quiet predicate even when other roots remain watched. + healthy = false + } } - } catch { /* watcherless serve still works via the burst window */ } + } catch { + // Discovery itself failed. Existing watchers are still closed normally, + // but they cannot validate reuse for an incomplete root set. + healthy = false + } + if (watchers.length === 0) return null + + // Coverage begins only after at least one watcher has been successfully + // armed. A parse performed while provider probing/stat/watch setup was in + // flight must not be blessed retroactively as watched. + const startedAt = Date.now() return { startedAt, lastEventAt: () => lastEventAt, - close: () => { for (const w of watchers) w.close() }, + healthy: () => healthy && !closed, + close: () => { + if (closed) return + closed = true + for (const w of watchers) w.close() + }, } } @@ -136,27 +314,49 @@ export async function runStdioServe(buildProgram: () => Command): Promise // parse stays valid past the burst window (capped in parser.ts, so a missed // filesystem event self-heals within minutes). This is what turns a warm // no-change fetch into a no-op instead of a stat sweep. - let rootsQuietSince: ((sinceTs: number) => boolean) | null = null - void startRootWatchers().then(async (w) => { + let rootReuseValidation: ((sinceTs: number) => ParseReuseValidation) | null = null + // Mutable object properties keep cleanup visible to TypeScript even though + // setup assigns them from an asynchronous continuation. + const watcherLifecycle: { + state: RootWatcherState | null + resetValidator: (() => void) | null + } = { state: null, resetValidator: null } + const watcherSetup = startRootWatchers().then(async (w) => { + watcherLifecycle.state = w + if (!w) return const { setParseReuseValidator } = await import('./parser.js') // Clean means: the watchers were already armed when the parse happened, // and no filesystem event has landed since. lastEventAt of 0 is a quiet // system (clean for anything parsed after arming), not an unknown. - const quiet = (sinceTs: number): boolean => sinceTs >= w.startedAt && w.lastEventAt() < sinceTs - rootsQuietSince = quiet - setParseReuseValidator(quiet) - }).catch(() => { /* watcherless serve still works via the burst window */ }) + const validate = (sinceTs: number): ParseReuseValidation => classifyRootReuse(sinceTs, { + startedAt: w.startedAt, + lastEventAt: w.lastEventAt(), + healthy: w.healthy(), + }) + rootReuseValidation = validate + setParseReuseValidator(validate) + watcherLifecycle.resetValidator = () => setParseReuseValidator(null) + }).catch(() => { + watcherLifecycle.state?.close() + watcherLifecycle.state = null + }) // Output-level memo: an identical panel query while the roots are quiet // returns the previous stdout verbatim - the aggregation work is skipped - // too, not just the parse. Invalidation is the same event-or-cap rule the - // parse reuse uses. + // too, not just the parse. Session data uses the same event-or-cap rule as + // parse reuse; config.json is fingerprinted on every request because it can + // change rendering without touching a provider root. const OUTPUT_MEMO_CAP_MS = 5 * 60 * 1000 - const outputMemo = new Map() + const outputMemo = new Map() + let observedConfigFingerprint: string | null | undefined if (process.stdin.isTTY) { process.stderr.write('codeburn serve speaks JSON over stdio and exists for the desktop app to hold warm.\nNothing interactive happens here; press Ctrl+C to exit.\n') } - const write = (value: unknown): void => { process.stdout.write(JSON.stringify(value) + '\n') } + // Keep the protocol transport anchored to the real stdout. runCaptured() + // temporarily replaces process.stdout.write to collect command output; a + // dynamic lookup here would swallow progress frames into the final payload. + const protocolWrite = process.stdout.write.bind(process.stdout) + const write = (value: unknown): void => { protocolWrite(JSON.stringify(value) + '\n') } write({ ready: true, pid: process.pid }) // Strict serialization: each request chains on the previous one. @@ -183,18 +383,39 @@ export async function runStdioServe(buildProgram: () => Command): Promise write({ id: request.id, ok: false, refused: true, error: 'command not served' }) return } + const configFingerprint = await getConfigFingerprint() + if (observedConfigFingerprint !== undefined && configFingerprint !== observedConfigFingerprint) { + outputMemo.clear() + } + observedConfigFingerprint = configFingerprint + // A permission or transient read failure must shorten reuse, never make + // an old result look current. + if (configFingerprint === null) outputMemo.clear() + const memoKey = request.args.join('\u0000') const memoHit = outputMemo.get(memoKey) - if (memoHit && Date.now() - memoHit.at < OUTPUT_MEMO_CAP_MS && rootsQuietSince?.(memoHit.at)) { + if ( + configFingerprint !== null + && memoHit?.configFingerprint === configFingerprint + && Date.now() - memoHit.createdAt < OUTPUT_MEMO_CAP_MS + && rootReuseValidation?.(memoHit.validatedFrom) === 'clean' + ) { write({ id: request.id, ok: true, output: memoHit.output }) return } try { - const { output, code } = await runCaptured(buildProgram, request.args) + const parseStartedAt = Date.now() + const { output, code } = await runCaptured( + buildProgram, + request.args, + progress => write({ id: request.id, progress }), + ) if (code === 0) { - outputMemo.set(memoKey, { at: Date.now(), output }) + if (configFingerprint !== null) { + outputMemo.set(memoKey, createOutputMemoEntry(parseStartedAt, Date.now(), output, configFingerprint)) + } if (outputMemo.size > 32) { - const oldest = [...outputMemo.entries()].sort((a, b) => a[1].at - b[1].at)[0] + const oldest = [...outputMemo.entries()].sort((a, b) => a[1].createdAt - b[1].createdAt)[0] if (oldest) outputMemo.delete(oldest[0]) } write({ id: request.id, ok: true, output }) @@ -212,16 +433,35 @@ export async function runStdioServe(buildProgram: () => Command): Promise if (process.memoryUsage().rss > SERVE_MAX_RSS_BYTES) { const { clearSessionCache } = await import('./parser.js') const { clearLoadCacheMemo } = await import('./session-cache.js') + const { clearCodexMemCaches } = await import('./codex-cache.js') + const { clearAntigravityCacheStates } = await import('./providers/antigravity.js') clearSessionCache() clearLoadCacheMemo() + clearCodexMemCaches() + clearAntigravityCacheStates() if (typeof globalThis.gc === 'function') globalThis.gc() } }) }) - // The app owns this process: stdin closing means the app is gone. - await new Promise((resolve) => { - rl.on('close', resolve) - process.stdin.on('end', resolve) + // The app owns this process: stdin closing (or failing) means the app is + // gone. Always release FSEvents handles and the module-global validator; + // otherwise an existing Claude root keeps a naturally closed child alive. + const transportClosed = new Promise((resolve) => { + rl.once('close', resolve) + process.stdin.once('end', resolve) + process.stdin.once('error', resolve) }) + try { + await transportClosed + } finally { + rl.close() + await watcherSetup + rootReuseValidation = null + try { + watcherLifecycle.resetValidator?.() + } finally { + watcherLifecycle.state?.close() + } + } } diff --git a/src/session-cache.ts b/src/session-cache.ts index 242551d1..6ded7829 100644 --- a/src/session-cache.ts +++ b/src/session-cache.ts @@ -1,9 +1,9 @@ -import { readFile, stat, open, rename, unlink, readdir, mkdir } from 'fs/promises' +import { readFile, stat, open, rename, unlink, readdir, mkdir, rm } from 'fs/promises' import { existsSync, readFileSync, unlinkSync } from 'fs' import { createHash, randomBytes } from 'crypto' import { join } from 'path' -import { homedir } from 'os' +import { getCodeburnCacheDir } from './cache-dir.js' import type { ToolCall } from './types.js' // ── Types ────────────────────────────────────────────────────────────── @@ -157,19 +157,37 @@ export type SessionCache = { // INVARIANT: a version bump must extend `PRIOR_CACHE_VERSIONS` (the adoption path // below) to EVERY prior version that can still exist on disk, or expired-PR // history from the immediately preceding build silently vanishes. -export const CACHE_VERSION = 7 +// v8: on-disk layout only - the single blob became a directory of per-provider +// shards plus a small envelope, so a launch that only touched one provider +// rewrites just that provider's file. The turn shape is unchanged, so a v7 file +// migrates losslessly (migrateSingleFileCache) rather than re-parsing. +// v9: on-disk layout only - a provider's shard split further by the UTC month of +// each cached file, so one appended session rewrites one month instead of the +// provider's whole (100MB-scale) history, and a ranged query loads only the +// months it can possibly report on. Turn shape unchanged, so v8 and v7 both +// migrate losslessly. +export const CACHE_VERSION = 9 -// The cache filename is version-suffixed so different binaries (e.g. an old -// launchd menubar on a prior release and a newer desktop app) each own a -// distinct file and can never clobber each other's incompatible schema. Bumping -// CACHE_VERSION automatically mints a fresh filename, superseding the migration -// dance the legacy unversioned file used to need. -const CACHE_FILE = `session-cache.v${CACHE_VERSION}.json` +// The cache directory is version-suffixed for the same reason the file used to +// be: different binaries (an old launchd menubar, a newer desktop app) each own +// a distinct layout and can never clobber each other's incompatible schema. +const CACHE_DIR_NAME = `session-cache.v${CACHE_VERSION}` +// The v8 shard directory, read once by the lossless v8 -> v9 re-layout. +const PRIOR_SHARD_DIR_NAME = 'session-cache.v8' +// Written LAST on every save: it names the shard file of every provider-month, so +// the rename that publishes it is the single point at which a save becomes visible. +const ENVELOPE_FILE = 'envelope.json' // The pre-versioning filename. Never written or deleted anymore — old binaries // still own it. On first load we adopt-copy it once (see loadCache) when the // versioned file is absent and the legacy file's version matches ours. const LEGACY_CACHE_FILE = 'session-cache.json' const TEMP_FILE_MAX_AGE_MS = 5 * 60 * 1000 +// A shard the published envelope does not name is either superseded garbage or +// a CONCURRENT writer's shard that its envelope has not published yet. The +// second case is why this guard is an order of magnitude above the temp-file +// one: sweeping a live save's shard out from under it would publish an envelope +// naming a file that no longer exists. No save takes an hour. +const UNREFERENCED_SHARD_MAX_AGE_MS = 60 * 60 * 1000 // Env vars that change what a provider discovers or how its sessions parse. // computeEnvFingerprint hashes exactly these to decide when a provider's cache @@ -271,7 +289,12 @@ export const PROVIDER_PARSE_VERSIONS: Record = { hermes: 'reasoning-output-accounting-v1-est-cost', 'lingtai-tui': 'token-ledger-registry-activity-v3', 'ibm-bob': 'worktree-project-grouping-v1', - kiro: 'ide-parsing-v1-est-cost', + // project-path-v1: the parser now records the session's full working + // directory as projectPath (CLI meta.cwd, v2 workspacePaths[0], workspace + // sessions' workspaceDirectory), which sync attribution needs to resolve + // the git repo. Cached entries from before the bump lack projectPath and + // would serve attribution-blind sessions forever without a re-parse. + kiro: 'ide-parsing-v1-est-cost-project-path-v1', opencode: 'session-model-v1', quickdesk: 'emf-sqlite-v2-est-cost', kimicode: 'wire-usage-v1-est-cost', @@ -281,23 +304,142 @@ export const PROVIDER_PARSE_VERSIONS: Record = { antigravity: 'worktree-project-grouping-v5', } -// ── Cache Dir ────────────────────────────────────────────────────────── - -function getCacheDir(): string { - return process.env['CODEBURN_CACHE_DIR'] ?? join(homedir(), '.cache', 'codeburn') -} - -function getCachePath(): string { - return join(getCacheDir(), CACHE_FILE) -} - function getLegacyCachePath(): string { - return join(getCacheDir(), LEGACY_CACHE_FILE) + return join(getCodeburnCacheDir(), LEGACY_CACHE_FILE) } -/** Absolute path of the active (version-suffixed) session cache file. */ -export function sessionCachePath(): string { - return getCachePath() +/** Absolute path of the active (version-suffixed) session cache directory. */ +export function sessionCacheDir(): string { + return join(getCodeburnCacheDir(), CACHE_DIR_NAME) +} + +// `until` is the UTC month of the newest turn any file in the shard holds. The +// shard's own key is the month of the OLDEST (a file is bucketed by its first +// turn), so the pair bounds every turn the shard can contribute and a ranged +// load can skip the shard outright when the two do not overlap the query. +type ShardRef = { name: string; until: string } +type EnvelopeProvider = { + envFingerprint: string + durable?: boolean + /** month (`YYYY-MM`, or `0000-00` for turn-less files) -> shard */ + shards: Record +} +type CacheEnvelope = { + version: number + complete?: boolean + nonce: string + providers: Record +} + +// Files with no turns (failure markers, empty sessions) have no month to bucket +// by. They live in one always-loaded bucket, which is also what makes the only +// possible re-bucketing safe: a file leaves this bucket the first time it gains +// a turn, and the bucket it leaves is guaranteed to be in memory. +const UNDATED_BUCKET = '0000-00' +// Sentinel inside `dirtyBuckets`: every bucket of the provider is dirty. +const ALL_BUCKETS = '*' + +function monthKey(timestamp: string | undefined): string | null { + if (!timestamp) return null + const ms = Date.parse(timestamp) + if (Number.isNaN(ms)) return null + const d = new Date(ms) + return `${d.getUTCFullYear()}-${String(d.getUTCMonth() + 1).padStart(2, '0')}` +} + +/** The UTC month span a cached file covers: `bucket` is its OLDEST turn's month + * (the shard it lives in), `until` its NEWEST (how far forward the shard can + * contribute). Both scan every turn rather than reading turns[0]/turns[-1]: + * several providers emit turns out of chronological order (cursor composers by + * ROWID, goose/crush/copilot by a DESC ordering), and a `until < bucket` span + * is empty, which makes the shard unreachable at EVERY scope. */ +export function cacheFileSpan(file: CachedFile): { bucket: string; until: string } { + let bucket: string | null = null + let until: string | null = null + for (const turn of file.turns) { + const month = monthKey(turn.timestamp) + if (month === null) continue + if (bucket === null || month < bucket) bucket = month + if (until === null || month > until) until = month + } + return bucket === null ? { bucket: UNDATED_BUCKET, until: UNDATED_BUCKET } : { bucket, until: until! } +} + +/** The shard bucket a cached file belongs to. Derived from the file's own turns, + * so an APPEND never moves it: appending can only extend `until`. */ +export function cacheBucketMonth(file: CachedFile): string { + return cacheFileSpan(file).bucket +} + +// Save bookkeeping, held beside the cache rather than on it so it never lands in +// a shard's JSON or in a caller's deep-equality. +type CacheState = { + dirty: boolean + /** provider -> dirty months (or `ALL_BUCKETS`). */ + dirtyBuckets: Map> + /** provider -> the shard refs the last load/save published. */ + shards: Map> + /** provider -> months held in memory; `null` when the whole provider loaded. */ + loaded: Map | null> + /** provider -> the envFingerprint the published envelope recorded. */ + fingerprints: Map + /** `provider\0path` -> the bucket the entry was loaded/saved under, so a + * delete or a re-bucketing can dirty the bucket it is leaving. */ + bucketOf: Map + /** The load scope this cache was read under, for the cross-request memo. */ + scope: string +} +const cacheStates = new WeakMap() + +function stateOf(cache: SessionCache): CacheState { + let state = cacheStates.get(cache) + if (!state) { + state = { + dirty: false, + dirtyBuckets: new Map(), + shards: new Map(), + loaded: new Map(), + fingerprints: new Map(), + bucketOf: new Map(), + scope: 'all', + } + cacheStates.set(cache, state) + } + return state +} + +function markBucketDirty(state: CacheState, provider: string, bucket: string): void { + state.dirty = true + let buckets = state.dirtyBuckets.get(provider) + if (!buckets) { buckets = new Set(); state.dirtyBuckets.set(provider, buckets) } + buckets.add(bucket) +} + +function isBucketDirty(state: CacheState, provider: string, bucket: string): boolean { + const buckets = state.dirtyBuckets.get(provider) + return buckets !== undefined && (buckets.has(ALL_BUCKETS) || buckets.has(bucket)) +} + +/** Record that `provider`'s section changed, so the next save rewrites the + * affected shards. Pass `filePath` whenever the change is scoped to one cached + * file — both the bucket it was last saved in and the bucket it is in now are + * marked, so a delete, a rewrite and a re-bucketing are all covered whichever + * order the caller mutates and marks in. Omitting it dirties every bucket. */ +export function markCacheDirty(cache: SessionCache, provider: string, filePath?: string): void { + const state = stateOf(cache) + if (filePath === undefined) { markBucketDirty(state, provider, ALL_BUCKETS); return } + const prior = state.bucketOf.get(`${provider}\0${filePath}`) + if (prior !== undefined) markBucketDirty(state, provider, prior) + const file = cache.providers[provider]?.files[filePath] + if (file) markBucketDirty(state, provider, cacheBucketMonth(file)) + // A path with neither a prior bucket nor a live entry (deleted before this + // process ever saw it) still has to move `dirty`, or the save is skipped. + state.dirty = true +} + +/** True when any provider section changed since the last save. */ +export function isCacheDirty(cache: SessionCache): boolean { + return stateOf(cache).dirty } // ── Env Fingerprint ──────────────────────────────────────────────────── @@ -442,6 +584,12 @@ function validateCachedFile(f: unknown): f is CachedFile { && (o['turns'] as unknown[]).every(validateTurn) } +// A shard's payload: the provider's `files` map, restricted to one month. +function validateFiles(v: unknown): v is Record { + if (!v || typeof v !== 'object' || Array.isArray(v)) return false + return Object.values(v as Record).every(validateCachedFile) +} + function validateProviderSection(s: unknown): s is ProviderSection { if (!s || typeof s !== 'object') return false const o = s as Record @@ -450,10 +598,11 @@ function validateProviderSection(s: unknown): s is ProviderSection { return Object.values(o['files'] as Record).every(validateCachedFile) } -function validateCache(raw: unknown): raw is SessionCache { +// Full validation of a single-file (pre-v8) cache blob at `version`. +function validateCache(raw: unknown, version: number): raw is SessionCache { if (!raw || typeof raw !== 'object') return false const o = raw as Record - if (o['version'] !== CACHE_VERSION) return false + if (o['version'] !== version) return false if (!o['providers'] || typeof o['providers'] !== 'object' || Array.isArray(o['providers'])) return false return Object.values(o['providers'] as Record).every(validateProviderSection) } @@ -466,7 +615,7 @@ function validateCache(raw: unknown): raw is SessionCache { // CACHE_VERSION bump MUST extend this list to every prior version that can still // exist on disk, or that history silently vanishes. (v5 was missed on the 5->6 // bump; v6 on the 6->7 bump; both are listed here.) -const PRIOR_CACHE_VERSIONS = [6, 5] as const +const PRIOR_CACHE_VERSIONS = [7, 6, 5] as const function priorCacheFile(version: number): string { return `session-cache.v${version}.json` @@ -492,7 +641,7 @@ function isCacheEnvelope(raw: unknown, version: number): raw is { version: numbe // sources. The daily cache (durable cost history) is not touched. async function adoptPriorCache(version: number): Promise { try { - const raw = await readFile(join(getCacheDir(), priorCacheFile(version)), 'utf-8') + const raw = await readFile(join(getCodeburnCacheDir(), priorCacheFile(version)), 'utf-8') const parsed = JSON.parse(raw) if (!isCacheEnvelope(parsed, version)) return null const migrated: SessionCache = { version: CACHE_VERSION, providers: {}, complete: false } @@ -542,54 +691,239 @@ async function adoptNewestPriorCache(): Promise { return merged } -// In-process memo of the parsed cache, keyed by the file identity that last -// produced it. On a 100MB+ corpus the JSON.parse of the session cache is -// seconds of work per load; a resident process (codeburn serve) pays it once -// and revalidates with a stat() per request. A rewrite by ANOTHER process -// moves mtime/size and forces a reload, so cross-process freshness is -// preserved; saveCache updates the memo write-through so the object handed -// out stays the canonical one after a refresh. -let cacheMemo: { path: string; mtimeMs: number; size: number; cache: SessionCache } | null = null +// In-process memo of the parsed cache, keyed by the envelope nonce that last +// produced it. On a 100MB+ corpus the JSON.parse of the shards is seconds of +// work per load; a resident process (codeburn serve) pays it once and +// revalidates by re-reading the (tiny) envelope per request. A save by ANOTHER +// process mints a new nonce and forces a reload, so cross-process freshness is +// preserved; saveCache updates the memo write-through so the object handed out +// stays the canonical one after a refresh. +let cacheMemo: { dir: string; nonce: string; scope: string; cache: SessionCache } | null = null export function clearLoadCacheMemo(): void { cacheMemo = null } -export async function loadCache(): Promise { - const path = getCachePath() +/** Months (UTC `YYYY-MM`, inclusive) a query can possibly report on. The load + * widens this by one month BELOW `fromMonth` and none above (see + * shardInScope): every cross-range carry in the report reads BACKWARDS from the + * first in-range turn, never forwards, so there is nothing above the range to + * reach for. */ +export type CacheLoadScope = { fromMonth: string; toMonth: string } + +export function monthScopeForRange(start: Date, end: Date): CacheLoadScope { + return { fromMonth: monthKey(start.toISOString())!, toMonth: monthKey(end.toISOString())! } +} + +function previousMonth(month: string): string { + const [y, m] = month.split('-').map(Number) as [number, number] + return m === 1 ? `${y - 1}-12` : `${y}-${String(m - 1).padStart(2, '0')}` +} + +// A shard is in scope when its [bucket .. until] span overlaps the query. One +// extra month of slack BELOW the range (and none above — every carry reads +// backwards) covers the cross-range carries that read turns from before the +// window: the pre-range PR set / git branch a session carries into its first +// in-range turn (both resolved from the same file, so they only need the file +// loaded at all), and the out-of-range subagent-spawn ANCHOR whose in-range +// child folds into it. LIMITATION: an anchor whose last turn is two or more +// months before its child's is not loaded, so that child attributes without the +// parent's PR set. One month of slack is the deliberate ceiling; widening it +// gives back the read savings the scope exists for. +// The undated bucket has no span and is always loaded. +function shardInScope(bucket: string, until: string, scope: CacheLoadScope): boolean { + if (bucket === UNDATED_BUCKET) return true + return bucket <= scope.toMonth && until >= previousMonth(scope.fromMonth) +} + +function isShardRef(v: unknown): v is ShardRef { + if (!v || typeof v !== 'object') return false + const o = v as Record + return typeof o['name'] === 'string' && typeof o['until'] === 'string' +} + +function isEnvelope(raw: unknown): raw is CacheEnvelope { + if (!raw || typeof raw !== 'object') return false + const o = raw as Record + if (o['version'] !== CACHE_VERSION || typeof o['nonce'] !== 'string') return false + const providers = o['providers'] + if (!providers || typeof providers !== 'object' || Array.isArray(providers)) return false + return Object.values(providers as Record).every(p => { + if (!p || typeof p !== 'object') return false + const e = p as Record + if (typeof e['envFingerprint'] !== 'string') return false + if (!e['shards'] || typeof e['shards'] !== 'object' || Array.isArray(e['shards'])) return false + return Object.values(e['shards'] as Record).every(isShardRef) + }) +} + +async function readEnvelope(dir: string): Promise { try { - const info = await stat(path) - if (cacheMemo && cacheMemo.path === path && cacheMemo.mtimeMs === info.mtimeMs && cacheMemo.size === info.size) { - return cacheMemo.cache - } - const raw = await readFile(path, 'utf-8') - const parsed = JSON.parse(raw) - if (!validateCache(parsed)) return afterMissingVersionedCache() - cacheMemo = { path, mtimeMs: info.mtimeMs, size: info.size, cache: parsed } - return parsed + const parsed = JSON.parse(await readFile(join(dir, ENVELOPE_FILE), 'utf-8')) + return isEnvelope(parsed) ? parsed : null } catch { - return afterMissingVersionedCache() + return null } } -// The current versioned file is absent/unreadable. Prefer adopting the newest -// prior versioned file's expired-source PR orphans (v6 before v5); failing that, -// fall back to the legacy unversioned file. Either way the versioned file is -// minted on the next save. -async function afterMissingVersionedCache(): Promise { +// A shard that is missing or malformed costs exactly the provider-months it +// held, not the provider and never the whole cache: those files re-parse while +// every other month keeps serving. +async function loadShard(path: string): Promise | null> { + try { + const parsed = JSON.parse(await readFile(path, 'utf-8')) + return validateFiles(parsed) ? parsed : null + } catch { + return null + } +} + +/** + * Read the cache. With a `scope`, only the shards whose months can contribute a + * turn to that range are read — everything else stays on disk and is carried + * across the next save untouched (see saveCache). Durable providers and any + * provider whose recorded fingerprint no longer matches are always read in + * full: the first because its cache is the only surviving record of pruned + * usage, the second because a fingerprint change discards the whole section and + * must see every entry it is discarding. + */ +export async function loadCache(scope?: CacheLoadScope): Promise { + const dir = sessionCacheDir() + const envelope = await readEnvelope(dir) + if (!envelope) return afterMissingShardCache() + const scopeKey = scope ? `${scope.fromMonth}..${scope.toMonth}` : 'all' + if (cacheMemo && cacheMemo.dir === dir && cacheMemo.nonce === envelope.nonce + && (cacheMemo.scope === 'all' || cacheMemo.scope === scopeKey)) return cacheMemo.cache + + const cache: SessionCache = { version: CACHE_VERSION, providers: {}, complete: envelope.complete === true } + const state = stateOf(cache) + const reads: Promise[] = [] + for (const [provider, meta] of Object.entries(envelope.providers)) { + const section: ProviderSection = { + envFingerprint: meta.envFingerprint, + files: {}, + ...(meta.durable ? { durable: true } : {}), + } + // Recorded even when every shard is skipped or unreadable: the section is + // what tells the next save which provider these carried-forward shard refs + // belong to, and what stops the reconcile from re-parsing under a + // fingerprint the envelope already agrees with. + cache.providers[provider] = section + const full = !scope || meta.durable === true || meta.envFingerprint !== computeEnvFingerprint(provider) + const loaded: Set | null = full ? null : new Set() + // Shards are read concurrently but merged in envelope order, so the result + // never depends on which read finished first. A path that somehow ended up + // in two shards resolves to the FRESHEST fingerprint and dirties both + // buckets, so the next save prunes the loser instead of letting it linger. + const pending: { bucket: string; files: Promise | null> }[] = [] + for (const [bucket, ref] of Object.entries(meta.shards)) { + if (loaded && !shardInScope(bucket, ref.until, scope!)) continue + loaded?.add(bucket) + pending.push({ bucket, files: loadShard(join(dir, ref.name)) }) + } + reads.push((async () => { + for (const { bucket, files: read } of pending) { + const files = await read + // Unreadable: the bucket counts as loaded-and-empty and is marked + // dirty, so the re-parsed files replace it instead of the stale shard + // being carried forward forever. + if (!files) { markBucketDirty(state, provider, bucket); continue } + for (const [path, file] of Object.entries(files)) { + const key = `${provider}\0${path}` + const seenIn = state.bucketOf.get(key) + if (seenIn !== undefined) { + markBucketDirty(state, provider, seenIn) + markBucketDirty(state, provider, bucket) + if (section.files[path]!.fingerprint.mtimeMs >= file.fingerprint.mtimeMs) continue + } + state.bucketOf.set(key, bucket) + section.files[path] = file + } + } + })()) + state.loaded.set(provider, loaded) + state.shards.set(provider, meta.shards) + state.fingerprints.set(provider, meta.envFingerprint) + } + await Promise.all(reads) + state.scope = scopeKey + cacheMemo = { dir, nonce: envelope.nonce, scope: scopeKey, cache } + return cache +} + +// The shard directory is absent/unreadable. Prefer a LOSSLESS re-layout of the +// newest prior layout that is present (v8 provider shards, then the v7 single +// file — both hold the current turn shape, so nothing re-parses); failing that, +// adopt the prior versions' expired-source PR orphans, then the legacy +// unversioned file. Either way the shard directory is minted on the next save. +async function afterMissingShardCache(): Promise { + const relaid = await migrateProviderShardCache() ?? await migrateSingleFileCache() + if (relaid) return relaid const prior = await adoptNewestPriorCache() if (prior) return prior - // validateCache requires version === CACHE_VERSION, so a different-version - // legacy file is ignored (left intact). We copy it into the versioned file once - // via saveCache; the legacy file is never modified. + // validateCache requires the version to match, so a different-version legacy + // file is ignored (left intact). We copy it into the shard layout once via + // saveCache; the legacy file is never modified. return adoptLegacyCache() } +// One-time, lossless re-layout of the v8 per-provider shard directory: v9 +// changed the on-disk LAYOUT only, so every entry moves across verbatim (just +// re-bucketed by month in memory) and nothing re-parses. The v8 directory is +// removed only once the v9 save has published. +async function migrateProviderShardCache(): Promise { + const dir = join(getCodeburnCacheDir(), PRIOR_SHARD_DIR_NAME) + let envelope: { complete?: boolean; shards: Record } + try { + const parsed = JSON.parse(await readFile(join(dir, ENVELOPE_FILE), 'utf-8')) as Record + if (parsed['version'] !== 8 || !parsed['shards'] || typeof parsed['shards'] !== 'object') return null + envelope = parsed as { complete?: boolean; shards: Record } + } catch { + return null + } + const cache: SessionCache = { version: CACHE_VERSION, providers: {}, complete: envelope.complete === true } + await Promise.all(Object.entries(envelope.shards).map(async ([provider, name]) => { + try { + const parsed = JSON.parse(await readFile(join(dir, name), 'utf-8')) + if (validateProviderSection(parsed)) cache.providers[provider] = parsed + } catch { /* one unreadable v8 shard costs that provider, as it already did */ } + })) + return publishRelaidCache(cache, () => rm(dir, { recursive: true, force: true })) +} + +// One-time, lossless re-layout of the v7 single-file cache. v7 never wrote a +// shard directory, so it is migrated straight to v9 without minting a v8 in +// between. +async function migrateSingleFileCache(): Promise { + const v7Path = join(getCodeburnCacheDir(), priorCacheFile(7)) + let parsed: unknown + try { + parsed = JSON.parse(await readFile(v7Path, 'utf-8')) + } catch { + return null + } + if (!validateCache(parsed, 7)) return null + return publishRelaidCache( + { version: CACHE_VERSION, providers: parsed.providers, complete: parsed.complete === true }, + () => unlink(v7Path), + ) +} + +// Every section is marked dirty so the save writes each month's shard; the old +// layout is retired only once that save has published. +async function publishRelaidCache(cache: SessionCache, retire: () => Promise): Promise { + for (const provider of Object.keys(cache.providers)) markCacheDirty(cache, provider) + const published = await saveCache(cache).catch(() => false) + if (published) await retryCacheFileMutation(async () => { await retire() }) + return cache +} + async function adoptLegacyCache(): Promise { try { const raw = await readFile(getLegacyCachePath(), 'utf-8') const parsed = JSON.parse(raw) - if (!validateCache(parsed)) return emptyCache() + if (!validateCache(parsed, CACHE_VERSION)) return emptyCache() + for (const provider of Object.keys(parsed.providers)) markCacheDirty(parsed, provider) await saveCache(parsed).catch(() => {}) return parsed } catch { @@ -597,15 +931,19 @@ async function adoptLegacyCache(): Promise { } } -export async function saveCache(cache: SessionCache, verifyStillOwner?: () => Promise): Promise { - const dir = getCacheDir() - if (!existsSync(dir)) await mkdir(dir, { recursive: true }) +// Shard filenames carry a fresh nonce on every write, so a save never overwrites +// the file the currently-published envelope points at: readers keep seeing a +// consistent set until the envelope rename publishes the new one, and a writer +// that loses the ownership fence leaves the canonical shards untouched. +function shardFileName(provider: string, bucket: string): string { + return `${provider.replace(/[^A-Za-z0-9_-]/g, '_')}.${bucket}.${randomBytes(8).toString('hex')}.json` +} - const finalPath = getCachePath() +// The temp name carries a nonce: two processes writing the SAME final path +// (the envelope, every save) would otherwise share one temp file and interleave +// their writes into a torn or foreign payload. +async function writeFileAtomic(finalPath: string, payload: string): Promise { const tempPath = `${finalPath}.${randomBytes(8).toString('hex')}.tmp` - delete (cache as { _dirty?: boolean })._dirty - const payload = JSON.stringify(cache) - const handle = await open(tempPath, 'w', 0o600) try { await handle.writeFile(payload, { encoding: 'utf-8' }) @@ -613,44 +951,275 @@ export async function saveCache(cache: SessionCache, verifyStillOwner?: () => Pr } finally { await handle.close() } - try { - // The warm refresh transaction passes an ownership fence. It must be the - // final operation before publication so a displaced writer cannot replace - // the canonical cache with its stale snapshot. - if (verifyStillOwner && !await verifyStillOwner()) { - await retryCacheFileMutation(() => unlink(tempPath)) - return false - } - let renamed = false for (let attempt = 0; attempt < 3; attempt++) { try { await rename(tempPath, finalPath) - renamed = true - break + return } catch (err) { const code = (err as NodeJS.ErrnoException).code if ((code !== 'EPERM' && code !== 'EBUSY') || attempt === 2) throw err await new Promise(resolve => { setTimeout(resolve, 10 * (attempt + 1)) }) } } - if (!renamed) throw new Error('session cache rename failed') - // Write-through: the object just published IS the freshest state; capture - // the post-rename file identity so the next loadCache in this process - // reuses it instead of re-parsing what it just wrote. - try { - const info = await stat(finalPath) - cacheMemo = { path: finalPath, mtimeMs: info.mtimeMs, size: info.size, cache } - } catch { - cacheMemo = null - } - return true } catch (err) { await retryCacheFileMutation(() => unlink(tempPath)) throw err } } +function bucketFiles(section: ProviderSection): { groups: Map>; until: Map } { + const groups = new Map>() + const until = new Map() + for (const [path, file] of Object.entries(section.files)) { + const span = cacheFileSpan(file) + let group = groups.get(span.bucket) + if (!group) { group = {}; groups.set(span.bucket, group) } + group[path] = file + const seen = until.get(span.bucket) + if (seen === undefined || span.until > seen) until.set(span.bucket, span.until) + } + return { groups, until } +} + +function untilMonth(files: Record): string { + let until = UNDATED_BUCKET + for (const file of Object.values(files)) { + const month = cacheFileSpan(file).until + if (month > until) until = month + } + return until +} + +// What a save has decided about one provider, carried across the ownership +// fence so every shard READ that a save needs happens as late as possible (see +// the phase-two comment in saveCache). +type ProviderPlan = { + section: ProviderSection + groups: Map> + loaded: Set | null + priorRefs: Record + reset: boolean + /** Paths that may ALSO still sit in a shard this run never loaded. */ + moved: Set + /** Buckets whose payload has to be merged with the published shard first. */ + deferred: string[] + /** bucket -> the shard name the merge was built from, for the retry below. */ + mergedFrom: Map + refs: Record +} + +export async function saveCache(cache: SessionCache, verifyStillOwner?: () => Promise): Promise { + const dir = sessionCacheDir() + if (!existsSync(dir)) await mkdir(dir, { recursive: true, mode: 0o700 }) + + const state = stateOf(cache) + const written = new Set() + const plans = new Map() + + const writeShard = async (provider: string, bucket: string, files: Record): Promise => { + const name = shardFileName(provider, bucket) + await writeFileAtomic(join(dir, name), JSON.stringify(files)) + written.add(name) + return { name, until: untilMonth(files) } + } + + // Overlay this run's entries for `bucket` onto the published shard `from`, + // minus any path that has since moved to another month. + const mergeShard = async (provider: string, plan: ProviderPlan, bucket: string, from: string | undefined): Promise => { + const files = plan.groups.get(bucket)! + const onDisk = from ? await loadShard(join(dir, from)) : null + if (!onDisk) return writeShard(provider, bucket, files) + for (const path of plan.moved) delete onDisk[path] + return writeShard(provider, bucket, { ...onDisk, ...files }) + } + + try { + // ── Phase one: everything that can be written from memory alone ────── + for (const [provider, section] of Object.entries(cache.providers)) { + const priorRefs = state.shards.get(provider) ?? {} + const loaded = state.loaded.get(provider) ?? null + // A fingerprint change discards the section outright (see + // getOrCreateProviderSection), so the months it did not load must be + // dropped rather than carried — they hold entries under the old + // fingerprint. loadCache never scopes such a provider, so `loaded` is + // null here in practice; the guard is what makes that safe to rely on. + const priorFingerprint = state.fingerprints.get(provider) + const reset = priorFingerprint !== undefined && priorFingerprint !== section.envFingerprint + const { groups } = bucketFiles(section) + const plan: ProviderPlan = { section, groups, loaded, priorRefs, reset, moved: new Set(), deferred: [], mergedFrom: new Map(), refs: {} } + plans.set(provider, plan) + + // An entry whose bucket this run never loaded may ALSO still exist, under + // an older month, in a shard we are about to carry across verbatim — a + // re-parse that shifted the file's oldest turn, or (the common #441 path) + // a parse failure that left a turn-less marker with no month at all. Left + // alone, the path would live in two shards at once and a later load could + // resolve to the stale copy. Both cases are rare, so the prune they + // trigger below reads shards it otherwise would not. + if (loaded) { + for (const [path, file] of Object.entries(section.files)) { + if (state.bucketOf.has(`${provider}\0${path}`)) continue + const bucket = cacheFileSpan(file).bucket + if (!loaded.has(bucket) || bucket === UNDATED_BUCKET) plan.moved.add(path) + } + } + + for (const [bucket, files] of groups) { + const prior = priorRefs[bucket] + // `priorRefs` is this process's snapshot from its last load or save. + // ANOTHER process may have republished that shard since, unlinking the + // file we are about to name — so reuse is conditional on the file still + // being there, and a vanished one is rewritten from memory. + if (prior && !isBucketDirty(state, provider, bucket) && existsSync(join(dir, prior.name))) { + plan.refs[bucket] = prior + continue + } + // Dirty but never loaded: memory holds only the entries this run wrote + // into the bucket, so the published shard's other entries have to be + // merged back in or the save would drop them. Deferred to phase two so + // the read happens against the CURRENT shard, not a stale name. + if (loaded && !loaded.has(bucket) && prior) { plan.deferred.push(bucket); continue } + plan.refs[bucket] = await writeShard(provider, bucket, files) + } + } + + // The warm refresh transaction passes an ownership fence. It must be the + // final operation before publication so a displaced writer cannot replace + // the canonical cache with its stale snapshot. Shards written above are + // unreferenced until the envelope names them, so a lost fence publishes + // nothing. + if (verifyStillOwner && !await verifyStillOwner()) { + for (const name of written) await retryCacheFileMutation(() => unlink(join(dir, name))) + return false + } + + // ── Phase two: everything that has to read the published shards ────── + // Re-read the envelope first. Between our load and now, another process may + // have republished any month we are carrying or merging into; adopting its + // CURRENT name is what keeps a carried orphan (an expired transcript's PR + // spend, unrecoverable by any re-parse) from being dropped just because the + // name we remembered was retired. It also shrinks the read-modify-write + // window for a merge down to the publish itself. That window is not zero: + // two processes merging into the same unloaded month can still interleave, + // and the loser's entries are re-derived on the next parse rather than lost + // for good — a full lock here would cost every save the contention. + const live = await readEnvelope(dir) + for (const [provider, plan] of plans) { + const liveShards = live?.providers[provider]?.shards ?? {} + const currentName = (bucket: string): string | undefined => { + const name = liveShards[bucket]?.name ?? plan.priorRefs[bucket]?.name + return name && existsSync(join(dir, name)) ? name : undefined + } + + for (const bucket of plan.deferred) { + plan.refs[bucket] = await mergeShard(provider, plan, bucket, currentName(bucket)) + plan.mergedFrom.set(bucket, currentName(bucket)) + } + + // Months this run never loaded keep their published shard. This is the + // invariant that makes a scoped load safe to save from. A month another + // process published while we held a partial view is adopted for the same + // reason: dropping it would delete history we never even saw. + if (!plan.loaded || plan.reset) continue + const carried = new Set([...Object.keys(plan.priorRefs), ...Object.keys(liveShards)]) + for (const bucket of carried) { + if (plan.refs[bucket] || plan.groups.has(bucket) || plan.loaded.has(bucket)) continue + const name = currentName(bucket) + if (!name) continue + const ref = { name, until: (liveShards[bucket] ?? plan.priorRefs[bucket])!.until } + if (plan.moved.size === 0) { plan.refs[bucket] = ref; continue } + // A path that moved into another month must not survive here too. + const onDisk = await loadShard(join(dir, name)) + if (!onDisk || !Object.keys(onDisk).some(p => plan.moved.has(p))) { plan.refs[bucket] = ref; continue } + for (const path of plan.moved) delete onDisk[path] + if (Object.keys(onDisk).length > 0) plan.refs[bucket] = await writeShard(provider, bucket, onDisk) + } + } + + // One optimistic retry: if another process republished a month we merged + // into while we were reading it, our shard was built on a superseded + // pre-image and would drop that process's entries. Redoing the merge from + // the current shard narrows the read-modify-write window from a shard read + // down to the envelope publish below. It does not close it — a save that + // loses the remaining race has its entries re-derived by the next parse + // (the reconcile sees no cache entry and re-reads the file), never silently + // dropped for good. A lock here would tax every save for a rare interleave. + const settled = await readEnvelope(dir) + for (const [provider, plan] of plans) { + for (const [bucket, mergedFrom] of plan.mergedFrom) { + const now = settled?.providers[provider]?.shards[bucket]?.name + if (!now || now === mergedFrom || !existsSync(join(dir, now))) continue + plan.refs[bucket] = await mergeShard(provider, plan, bucket, now) + } + } + + // Last look before publishing: a concurrent save may have unlinked a shard + // in the moment since. An envelope must never name a file that is already + // gone — that reads back as a corrupt month and drops its history. + const providers: Record = {} + for (const [provider, plan] of plans) { + const shards: Record = {} + for (const [bucket, ref] of Object.entries(plan.refs)) { + if (written.has(ref.name) || existsSync(join(dir, ref.name))) { shards[bucket] = ref; continue } + const files = plan.groups.get(bucket) + // A carried month whose file vanished and whose content was never in + // memory cannot be rewritten; dropping the reference is the only honest + // option, and the sweep retires the name. + if (files) shards[bucket] = await writeShard(provider, bucket, files) + } + providers[provider] = { + envFingerprint: plan.section.envFingerprint, + ...(plan.section.durable ? { durable: true } : {}), + shards, + } + } + + const envelope: CacheEnvelope = { + version: CACHE_VERSION, + complete: cache.complete === true, + nonce: randomBytes(8).toString('hex'), + providers, + } + await writeFileAtomic(join(dir, ENVELOPE_FILE), JSON.stringify(envelope)) + + // Shards the new envelope no longer references are garbage; a reader that + // already opened one keeps reading it, and any failure here is swept later + // by cleanupOrphanedTempFiles. + const retired: string[] = [] + for (const [provider, priorRefs] of state.shards) { + const kept = providers[provider]?.shards ?? {} + for (const [bucket, ref] of Object.entries(priorRefs)) { + if (kept[bucket]?.name !== ref.name) retired.push(ref.name) + } + } + + state.dirty = false + state.dirtyBuckets.clear() + state.shards.clear() + state.fingerprints.clear() + state.bucketOf.clear() + // `loaded` deliberately survives: a merged-and-rewritten shard is complete + // on disk but still partial in memory, so the next save has to merge again. + for (const [provider, meta] of Object.entries(providers)) { + state.shards.set(provider, meta.shards) + state.fingerprints.set(provider, meta.envFingerprint) + for (const [path, file] of Object.entries(cache.providers[provider]!.files)) { + state.bucketOf.set(`${provider}\0${path}`, cacheFileSpan(file).bucket) + } + } + // Write-through: the object just published IS the freshest state, so the + // next loadCache in this process reuses it instead of re-parsing. Its scope + // is whatever was loaded, not `all` — a save never widens what is in memory. + cacheMemo = { dir, nonce: envelope.nonce, scope: state.scope, cache } + for (const name of retired) await retryCacheFileMutation(() => unlink(join(dir, name))) + return true + } catch (err) { + for (const name of written) await retryCacheFileMutation(() => unlink(join(dir, name))) + throw err + } +} + async function retryCacheFileMutation(operation: () => Promise): Promise { for (let attempt = 0; attempt < 3; attempt++) { try { @@ -801,26 +1370,58 @@ export function mergeCallByDedupKey( // ── Temp Cleanup ─────────────────────────────────────────────────────── +async function unlinkIfOlderThan(path: string, maxAgeMs: number, now: number): Promise { + try { + const s = await stat(path) + if (now - s.mtimeMs > maxAgeMs) await unlink(path) + } catch {} +} + +// Sweeps our own shard directory: interrupted temp writes, plus shards the +// published envelope no longer references. Also retires the single-file layout's +// leftover temps in the parent directory, which nothing writes anymore. export async function cleanupOrphanedTempFiles(): Promise { - const dir = getCacheDir() + const now = Date.now() + const parent = getCodeburnCacheDir() + + // `session-cache.v.json..tmp` from a pre-v8 binary interrupted + // mid-write. Age-guarded, so an old binary's in-flight write is left alone. + try { + for (const entry of await readdir(parent)) { + if (!/^session-cache\.v\d+\.json\..*\.tmp$/.test(entry)) continue + await unlinkIfOlderThan(join(parent, entry), TEMP_FILE_MAX_AGE_MS, now) + } + } catch {} + + const dir = sessionCacheDir() if (!existsSync(dir)) return - try { - const entries = await readdir(dir) - const now = Date.now() + const referenced = new Set([ENVELOPE_FILE]) + const envelope = await readEnvelope(dir) + if (envelope) { + for (const meta of Object.values(envelope.providers)) { + for (const ref of Object.values(meta.shards)) referenced.add(ref.name) + } + // A published v9 envelope means the re-layout completed. Its retirement of + // the old layout is a separate, unsynchronised step, so a crash in between + // leaves 100MB+ of superseded cache behind forever. Age-guarded for the + // same reason the shard sweep is: an OLD binary may still be writing there. + await unlinkIfOlderThan(join(getCodeburnCacheDir(), priorCacheFile(7)), UNREFERENCED_SHARD_MAX_AGE_MS, now) + const v8Dir = join(getCodeburnCacheDir(), PRIOR_SHARD_DIR_NAME) + try { + const s = await stat(join(v8Dir, ENVELOPE_FILE)) + if (now - s.mtimeMs > UNREFERENCED_SHARD_MAX_AGE_MS) await rm(v8Dir, { recursive: true, force: true }) + } catch {} + } - // Only our own (versioned) temp files. Legacy `session-cache.json.*.tmp` - // temps belong to old binaries mid-write and must not be touched. - const prefix = `${CACHE_FILE}.` - for (const entry of entries) { - if (!entry.startsWith(prefix) || !entry.endsWith('.tmp')) continue - try { - const fullPath = join(dir, entry) - const s = await stat(fullPath) - if (now - s.mtimeMs > TEMP_FILE_MAX_AGE_MS) { - await unlink(fullPath) - } - } catch {} + try { + for (const entry of await readdir(dir)) { + if (entry.endsWith('.tmp')) { + await unlinkIfOlderThan(join(dir, entry), TEMP_FILE_MAX_AGE_MS, now) + continue + } + if (!envelope || referenced.has(entry)) continue + await unlinkIfOlderThan(join(dir, entry), UNREFERENCED_SHARD_MAX_AGE_MS, now) } } catch {} } @@ -846,7 +1447,7 @@ export type HydrationHandle = { waited: boolean; release: () => Promise } const NOOP_HANDLE: HydrationHandle = { waited: false, release: async () => {} } function lockPath(): string { - return join(getCacheDir(), HYDRATION_LOCK_FILE) + return join(getCodeburnCacheDir(), HYDRATION_LOCK_FILE) } // Our own pid never counts as a foreign holder: a same-process lock is either @@ -869,7 +1470,7 @@ async function readLockRecord(): Promise { async function writeOurLock(): Promise { try { - const dir = getCacheDir() + const dir = getCodeburnCacheDir() if (!existsSync(dir)) await mkdir(dir, { recursive: true }) const handle = await open(lockPath(), 'wx', 0o600) try { await handle.writeFile(JSON.stringify({ pid: process.pid, at: Date.now() }), { encoding: 'utf-8' }) } diff --git a/src/sync/ledger.ts b/src/sync/ledger.ts index 99ef8560..8e81a63d 100644 --- a/src/sync/ledger.ts +++ b/src/sync/ledger.ts @@ -6,8 +6,8 @@ */ import { existsSync, mkdirSync, readFileSync, writeFileSync, unlinkSync, renameSync } from 'fs' -import { join } from 'path' -import { homedir } from 'os' +import { join, resolve } from 'path' +import { getCodeburnCacheDir } from '../cache-dir.js' export interface LedgerEntry { key: string // deduplicationKey @@ -16,34 +16,69 @@ export interface LedgerEntry { const SIX_MONTHS_MS = 180 * 24 * 60 * 60 * 1000 -function cacheDir(): string { - // Honor XDG_CACHE_HOME — the ledger is reconstructible state, not config - const xdg = process.env.XDG_CACHE_HOME - const base = xdg && xdg.trim() ? xdg : join(homedir(), '.cache') - return join(base, 'codeburn') +function ledgerCacheDir(): string { + return getCodeburnCacheDir() } function ledgerPath(): string { - return join(cacheDir(), 'sync-ledger.json') + return join(ledgerCacheDir(), 'sync-ledger.json') } -export function readLedger(): LedgerEntry[] { - const path = ledgerPath() - if (!existsSync(path)) return [] +// Before the shared cache resolver existed, sync alone wrote beneath +// XDG_CACHE_HOME. Treat that location as a one-time migration source only; +// CODEBURN_CACHE_DIR (when non-empty) is authoritative and must never import +// from an unrelated XDG tree. +function legacyXdgLedgerPath(): string | null { + if (process.env.CODEBURN_CACHE_DIR?.trim()) return null + const xdg = process.env.XDG_CACHE_HOME + if (!xdg?.trim()) return null + const legacy = join(xdg, 'codeburn', 'sync-ledger.json') + return resolve(legacy) === resolve(ledgerPath()) ? null : legacy +} + +function readLedgerFile(path: string): LedgerEntry[] | null { try { - const raw = readFileSync(path, 'utf-8') - const entries = JSON.parse(raw) as unknown - if (!Array.isArray(entries)) return [] + const entries = JSON.parse(readFileSync(path, 'utf-8')) as unknown + if (!Array.isArray(entries)) return null return entries.filter( (e): e is LedgerEntry => typeof e === 'object' && e !== null && typeof e.key === 'string' ) } catch { - return [] + return null } } +export function readLedger(): LedgerEntry[] { + const path = ledgerPath() + const legacyPath = legacyXdgLedgerPath() + const canonicalEntries = existsSync(path) ? readLedgerFile(path) : null + if (!legacyPath || !existsSync(legacyPath)) return canonicalEntries ?? [] + const legacyEntries = readLedgerFile(legacyPath) + if (!legacyEntries) return canonicalEntries ?? [] + + // Canonical wins for duplicate keys, but retain every key that exists only + // in the historical ledger so an upgrade cannot re-upload old calls. + const merged = [...(canonicalEntries ?? [])] + const keys = new Set(merged.map(entry => entry.key)) + for (const entry of legacyEntries) { + if (keys.has(entry.key)) continue + keys.add(entry.key) + merged.push(entry) + } + + // Publish the canonical copy before retiring the legacy source. If the + // write fails, keep and return the old ledger so deduplication still works. + try { + writeLedger(merged) + try { unlinkSync(legacyPath) } catch { /* canonical copy already wins */ } + } catch { + return merged + } + return merged +} + export function writeLedger(entries: LedgerEntry[]): void { - const dir = cacheDir() + const dir = ledgerCacheDir() mkdirSync(dir, { recursive: true }) // Atomic write: a crash mid-write must not corrupt the ledger — a corrupt // ledger reads as empty and the next push re-sends the whole window. @@ -77,11 +112,34 @@ export function ledgerKeySet(): Set { return new Set(readLedger().map(e => e.key)) } -/** Clear the ledger (for sync reset). Returns the number of entries removed. */ -export function clearLedger(): number { - const path = ledgerPath() - if (!existsSync(path)) return 0 - const count = readLedger().length - unlinkSync(path) - return count +function isMissingFileError(error: unknown): boolean { + return typeof error === 'object' && error !== null && 'code' in error && error.code === 'ENOENT' +} + +/** Clear every eligible ledger (for sync reset). Returns the number of unique + * entries removed. This deliberately bypasses readLedger(): reset must delete + * canonical and legacy files independently, never migrate one into the other. */ +export function clearLedger(): number { + const canonicalPath = ledgerPath() + const legacyPath = legacyXdgLedgerPath() + const targets = [canonicalPath, ...(legacyPath ? [legacyPath] : [])].map(path => ({ + path, + entries: readLedgerFile(path) ?? [], + })) + const removedKeys = new Set() + let deletionError: unknown + + // Attempt every target even if one unlink fails. A retry then has only the + // actual remainder to remove, while ENOENT is the idempotent success case. + for (const target of targets) { + try { + unlinkSync(target.path) + for (const entry of target.entries) removedKeys.add(entry.key) + } catch (error) { + if (!isMissingFileError(error) && deletionError === undefined) deletionError = error + } + } + + if (deletionError !== undefined) throw deletionError + return removedKeys.size } diff --git a/tests/bash-commands.test.ts b/tests/bash-commands.test.ts index 949d50ff..69f5e870 100644 --- a/tests/bash-commands.test.ts +++ b/tests/bash-commands.test.ts @@ -1,4 +1,6 @@ import { describe, it, expect } from 'vitest' +import { basename } from 'path' +import stripAnsi from 'strip-ansi' import { extractBashCommands, isReadShapedBashCommand } from '../src/bash-utils.js' import { BASH_TOOLS } from '../src/classifier.js' @@ -118,6 +120,157 @@ describe('BASH_TOOLS', () => { it('rejects unknown tools', () => { expect(BASH_TOOLS.has('Read')).toBe(false) }) }) +// Regression coverage for the quadratic -> linear separator-matching rewrite. +// The old regex (/\s*(?:&&|;|\|)\s*/g, and the equivalent split form) is kept +// here verbatim as a reference so new/old output can be diffed on tricky inputs. +describe('separator regex fix: parity with pre-fix implementation', () => { + function stripQuotedStringsRef(command: string): string { + return command.replace(/"[^"]*"|'[^']*'/g, match => ' '.repeat(match.length)) + } + + const COMMAND_PREFIXES_REF = new Set([ + 'sudo', 'doas', + 'npx', 'bunx', + 'time', + 'nice', 'nohup', 'stdbuf', + 'rtk', + ]) + + const READ_ONLY_BASH_REF = new Set([ + 'rg', 'grep', 'egrep', 'fgrep', 'ag', + 'cat', 'head', 'tail', 'less', 'more', + 'ls', 'find', 'fd', 'tree', + 'wc', 'stat', 'file', 'du', 'df', + 'which', 'type', 'pwd', 'printenv', 'env', + 'readlink', 'realpath', 'basename', 'dirname', + 'jq', 'diff', + ]) + + const GIT_READ_SUBCOMMANDS_REF = new Set([ + 'log', 'diff', 'status', 'show', 'blame', 'grep', + 'shortlog', 'describe', 'rev-parse', 'ls-files', + ]) + + function extractBashCommandsOld(rawCommand: string): string[] { + if (!rawCommand || !rawCommand.trim()) return [] + + const command = stripAnsi(rawCommand) + const stripped = stripQuotedStringsRef(command) + + const separatorRegex = /\s*(?:&&|;|\|)\s*/g + const separators: Array<{ start: number; end: number }> = [] + let match: RegExpExecArray | null + + while ((match = separatorRegex.exec(stripped)) !== null) { + separators.push({ start: match.index, end: match.index + match[0].length }) + } + + const ranges: Array<[number, number]> = [] + let cursor = 0 + for (const sep of separators) { + ranges.push([cursor, sep.start]) + cursor = sep.end + } + ranges.push([cursor, command.length]) + + const commands: string[] = [] + for (const [start, end] of ranges) { + const segment = command.slice(start, end).trim() + if (!segment) continue + + const tokens = segment.split(/\s+/) + let i = 0 + while (i < tokens.length) { + if (/^\w+=/.test(tokens[i]!)) { i++; continue } + const next = tokens[i + 1] + if ( + next !== undefined && + COMMAND_PREFIXES_REF.has(basename(tokens[i]!)) && + !next.startsWith('-') && + !/["']/.test(next) + ) { i++; continue } + break + } + const base = i < tokens.length ? basename(tokens[i]!) : '' + + if (base && base !== 'cd' && base !== 'true' && base !== 'false') { + commands.push(base) + } + } + + return commands + } + + function isReadShapedBashCommandOld(rawCommand: string): boolean { + if (!rawCommand || !rawCommand.trim()) return false + const stripped = stripQuotedStringsRef(stripAnsi(rawCommand)) + const segments = stripped.split(/\s*(?:&&|;|\|)\s*/) + let sawCommand = false + for (const segment of segments) { + const trimmed = segment.trim() + if (!trimmed) continue + const tokens = trimmed.split(/\s+/) + let i = 0 + while (i < tokens.length && (/^\w+=/.test(tokens[i]!) || COMMAND_PREFIXES_REF.has(basename(tokens[i]!)))) i++ + const base = i < tokens.length ? basename(tokens[i]!) : '' + if (!base) continue + sawCommand = true + if (base === 'git') { + const sub = tokens[i + 1] + if (!sub || !GIT_READ_SUBCOMMANDS_REF.has(sub)) return false + continue + } + if (!READ_ONLY_BASH_REF.has(base)) return false + } + return sawCommand + } + + function buildWhitespaceHeavyCommand(): string { + const parts: string[] = [] + for (let i = 0; i < 10; i++) parts.push('git' + ' '.repeat(2000) + 'status') + return parts.join(' && ') + } + + const TRICKY_INPUTS: string[] = [ + 'echo "a && b" && ls', + "foo 'x;y';bar", + 'cat < { + for (const input of TRICKY_INPUTS) { + expect(extractBashCommands(input)).toEqual(extractBashCommandsOld(input)) + } + }) + + it('isReadShapedBashCommand matches the old implementation across tricky separator inputs', () => { + for (const input of TRICKY_INPUTS) { + expect(isReadShapedBashCommand(input)).toBe(isReadShapedBashCommandOld(input)) + } + }) + + it('runs the whitespace-heavy command in well under 50ms (old form is quadratic)', () => { + const big = buildWhitespaceHeavyCommand() + const t0 = Date.now() + extractBashCommands(big) + expect(Date.now() - t0).toBeLessThan(50) + }) +}) + describe('isReadShapedBashCommand (#941)', () => { it('accepts single read commands and read-only git subcommands', () => { expect(isReadShapedBashCommand('rg -n "x" src/')).toBe(true) diff --git a/tests/cache-dir.test.ts b/tests/cache-dir.test.ts new file mode 100644 index 00000000..eaef6c74 --- /dev/null +++ b/tests/cache-dir.test.ts @@ -0,0 +1,25 @@ +import { afterEach, describe, expect, it } from 'vitest' +import { join } from 'path' +import { homedir } from 'os' +import { getCodeburnCacheDir } from '../src/cache-dir.js' + +describe('getCodeburnCacheDir', () => { + const original = process.env['CODEBURN_CACHE_DIR'] + + afterEach(() => { + if (original === undefined) delete process.env['CODEBURN_CACHE_DIR'] + else process.env['CODEBURN_CACHE_DIR'] = original + }) + + it('resolves an explicit override at call time', () => { + process.env['CODEBURN_CACHE_DIR'] = '/tmp/codeburn-one' + expect(getCodeburnCacheDir()).toBe('/tmp/codeburn-one') + process.env['CODEBURN_CACHE_DIR'] = '/tmp/codeburn-two' + expect(getCodeburnCacheDir()).toBe('/tmp/codeburn-two') + }) + + it.each(['', ' ', '\n\t'])('treats a blank override as absent (%j)', value => { + process.env['CODEBURN_CACHE_DIR'] = value + expect(getCodeburnCacheDir()).toBe(join(homedir(), '.cache', 'codeburn')) + }) +}) diff --git a/tests/cache-directory-switch.test.ts b/tests/cache-directory-switch.test.ts new file mode 100644 index 00000000..175cd828 --- /dev/null +++ b/tests/cache-directory-switch.test.ts @@ -0,0 +1,277 @@ +import { mkdir, mkdtemp, readFile, rm, stat, writeFile } from 'fs/promises' +import { existsSync } from 'fs' +import { tmpdir } from 'os' +import { join } from 'path' + +import { afterEach, beforeEach, describe, expect, it } from 'vitest' + +import { + clearCodexMemCaches, + fingerprintFile, + flushCodexCache, + readCachedCodexResults, + writeCachedCodexResults, +} from '../src/codex-cache.js' +import { + clearAntigravityCacheStates, + createAntigravityProvider, + flushAntigravityCache, +} from '../src/providers/antigravity.js' +import type { ParsedProviderCall } from '../src/providers/types.js' + +const originalCacheDir = process.env['CODEBURN_CACHE_DIR'] +const originalHome = process.env['HOME'] +const originalCodexHome = process.env['CODEX_HOME'] +let root: string + +function call(provider: string, marker: string): ParsedProviderCall { + return { + provider, + model: marker, + inputTokens: 1, + outputTokens: 1, + cacheCreationInputTokens: 0, + cacheReadInputTokens: 0, + cachedInputTokens: 0, + reasoningTokens: 0, + webSearchRequests: 0, + costUSD: 0, + tools: [], + bashCommands: [], + timestamp: '2026-08-12T00:00:00.000Z', + speed: 'standard', + deduplicationKey: `${provider}:${marker}`, + userMessage: '', + sessionId: marker, + } +} + +async function seedAntigravityCache( + cacheDir: string, + sourcePath: string, + marker: string, +): Promise { + const sourceStat = await stat(sourcePath) + await mkdir(cacheDir, { recursive: true }) + await writeFile(join(cacheDir, 'antigravity-results.json'), JSON.stringify({ + version: 5, + cascades: { + shared: { + mtimeMs: sourceStat.mtimeMs, + sizeBytes: sourceStat.size, + calls: [call('antigravity', marker)], + }, + }, + })) +} + +async function readAntigravityModel(sourcePath: string): Promise { + const parser = createAntigravityProvider().createSessionParser({ + path: sourcePath, + project: 'fixture', + provider: 'antigravity', + }, new Set()) + for await (const parsed of parser.parse()) return parsed.model + return undefined +} + +beforeEach(async () => { + root = await mkdtemp(join(tmpdir(), 'codeburn-cache-switch-')) +}) + +afterEach(async () => { + if (originalCacheDir === undefined) delete process.env['CODEBURN_CACHE_DIR'] + else process.env['CODEBURN_CACHE_DIR'] = originalCacheDir + if (originalHome === undefined) delete process.env['HOME'] + else process.env['HOME'] = originalHome + if (originalCodexHome === undefined) delete process.env['CODEX_HOME'] + else process.env['CODEX_HOME'] = originalCodexHome + await rm(root, { recursive: true, force: true }) +}) + +describe('call-time CODEBURN_CACHE_DIR isolation', () => { + it('keeps Codex reads and writes keyed by the active cache directory', async () => { + const sourcePath = join(root, 'rollout.jsonl') + const cacheA = join(root, 'cache-a') + const cacheB = join(root, 'cache-b') + await writeFile(sourcePath, '{}\n') + const fingerprint = await fingerprintFile(sourcePath) + expect(fingerprint).not.toBeNull() + + process.env['CODEBURN_CACHE_DIR'] = cacheA + await writeCachedCodexResults(sourcePath, 'project-a', [call('codex', 'from-a')], fingerprint!) + await flushCodexCache() + + process.env['CODEBURN_CACHE_DIR'] = cacheB + expect(await readCachedCodexResults(sourcePath)).toBeNull() + await writeCachedCodexResults(sourcePath, 'project-b', [call('codex', 'from-b')], fingerprint!) + await flushCodexCache() + + const diskB = JSON.parse(await readFile(join(cacheB, 'codex-results.json'), 'utf8')) + expect(diskB.files[sourcePath].calls.map((entry: ParsedProviderCall) => entry.model)).toEqual(['from-b']) + + process.env['CODEBURN_CACHE_DIR'] = cacheA + expect((await readCachedCodexResults(sourcePath))?.calls.map(entry => entry.model)).toEqual(['from-a']) + }) + + it('does not flush dirty Codex state from A into B', async () => { + const sourceA = join(root, 'a.jsonl') + const sourceB = join(root, 'b.jsonl') + const cacheA = join(root, 'cache-a-dirty') + const cacheB = join(root, 'cache-b-dirty') + await writeFile(sourceA, 'a\n') + await writeFile(sourceB, 'b\n') + + process.env['CODEBURN_CACHE_DIR'] = cacheA + await writeCachedCodexResults(sourceA, 'project-a', [call('codex', 'dirty-a')], (await fingerprintFile(sourceA))!) + + process.env['CODEBURN_CACHE_DIR'] = cacheB + await writeCachedCodexResults(sourceB, 'project-b', [call('codex', 'dirty-b')], (await fingerprintFile(sourceB))!) + await flushCodexCache() + + const diskB = JSON.parse(await readFile(join(cacheB, 'codex-results.json'), 'utf8')) + expect(Object.keys(diskB.files)).toEqual([sourceB]) + + process.env['CODEBURN_CACHE_DIR'] = cacheA + await flushCodexCache() + const diskA = JSON.parse(await readFile(join(cacheA, 'codex-results.json'), 'utf8')) + expect(Object.keys(diskA.files)).toEqual([sourceA]) + }) + + it('pins Codex reads, dirty writes, and flushes to the parse call-time directory', async () => { + const home = join(root, 'parse-home') + const codexHome = join(root, 'parse-codex-home') + const sessionDir = join(codexHome, 'sessions', '2026', '08', '12') + const cacheA = join(root, 'parse-cache-a') + const cacheB = join(root, 'parse-cache-b') + await mkdir(sessionDir, { recursive: true }) + await mkdir(home, { recursive: true }) + const sourcePath = join(sessionDir, 'rollout-cache-dir-switch.jsonl') + await writeFile(sourcePath, [ + JSON.stringify({ + type: 'session_meta', + timestamp: '2026-08-12T10:00:00.000Z', + payload: { + cwd: '/Users/test/cache-dir-transaction', + originator: 'codex-cli', + session_id: 'cache-dir-transaction', + model: 'gpt-5.3-codex', + }, + }), + JSON.stringify({ + type: 'event_msg', + timestamp: '2026-08-12T10:01:00.000Z', + payload: { + type: 'token_count', + info: { + model: 'gpt-5.3-codex', + last_token_usage: { + input_tokens: 10, + cached_input_tokens: 0, + output_tokens: 5, + reasoning_output_tokens: 0, + total_tokens: 15, + }, + total_token_usage: { + input_tokens: 10, + cached_input_tokens: 0, + output_tokens: 5, + reasoning_output_tokens: 0, + total_tokens: 15, + }, + }, + }, + }), + ].join('\n') + '\n') + + process.env['HOME'] = home + process.env['CODEX_HOME'] = codexHome + process.env['CODEBURN_CACHE_DIR'] = cacheA + const { clearSessionCache, parseAllSessions } = await import('../src/parser.js') + clearSessionCache() + + // parseAllSessions reaches its first await before any Codex cache access. + // Switching the host env immediately after invocation deterministically + // exercises every later read/write/flush under the captured A transaction. + const parsing = parseAllSessions(undefined, 'codex') + process.env['CODEBURN_CACHE_DIR'] = cacheB + const projects = await parsing + + expect(projects.some(project => project.sessions.some(session => + session.turns.some(turn => turn.assistantCalls.some(entry => entry.provider === 'codex')) + ))).toBe(true) + expect(existsSync(join(cacheA, 'codex-results.json'))).toBe(true) + expect(existsSync(join(cacheB, 'codex-results.json'))).toBe(false) + const diskA = JSON.parse(await readFile(join(cacheA, 'codex-results.json'), 'utf8')) + expect(diskA.files[sourcePath].calls).toHaveLength(1) + clearSessionCache() + }) + + it('loads Antigravity cache entries from the active directory after A to B', async () => { + const sourcePath = join(root, 'shared.pb') + const cacheA = join(root, 'agy-cache-a') + const cacheB = join(root, 'agy-cache-b') + await writeFile(sourcePath, 'fixture') + await seedAntigravityCache(cacheA, sourcePath, 'from-a') + await seedAntigravityCache(cacheB, sourcePath, 'from-b') + + process.env['CODEBURN_CACHE_DIR'] = cacheA + expect(await readAntigravityModel(sourcePath)).toBe('from-a') + + process.env['CODEBURN_CACHE_DIR'] = cacheB + expect(await readAntigravityModel(sourcePath)).toBe('from-b') + }) + + it('does not flush dirty Antigravity state from A into B', async () => { + const sourcePath = join(root, 'shared.pb') + const cacheA = join(root, 'agy-cache-a-dirty') + const cacheB = join(root, 'agy-cache-b-dirty') + await writeFile(sourcePath, 'fixture') + await seedAntigravityCache(cacheA, sourcePath, 'from-a') + await seedAntigravityCache(cacheB, sourcePath, 'from-b') + + process.env['CODEBURN_CACHE_DIR'] = cacheA + expect(await readAntigravityModel(sourcePath)).toBe('from-a') + + // The provider parse transaction captures A. Even if the host changes its + // call-time env before the deferred flush, eviction/publication stays on A. + process.env['CODEBURN_CACHE_DIR'] = cacheB + await flushAntigravityCache(new Set(), cacheA) + + expect(existsSync(join(cacheB, 'antigravity-results.json'))).toBe(true) + const diskB = JSON.parse(await readFile(join(cacheB, 'antigravity-results.json'), 'utf8')) + expect(diskB.cascades.shared.calls[0].model).toBe('from-b') + const diskA = JSON.parse(await readFile(join(cacheA, 'antigravity-results.json'), 'utf8')) + expect(diskA.cascades).toEqual({}) + }) + + it('drops clean per-directory memos when the resident RSS guard clears them', async () => { + const codexSource = join(root, 'guard.jsonl') + const antigravitySource = join(root, 'shared.pb') + const cacheDir = join(root, 'guard-cache') + await writeFile(codexSource, '{}\n') + await writeFile(antigravitySource, 'fixture') + await seedAntigravityCache(cacheDir, antigravitySource, 'before') + + process.env['CODEBURN_CACHE_DIR'] = cacheDir + await writeCachedCodexResults(codexSource, 'project', [call('codex', 'before')], (await fingerprintFile(codexSource))!) + await flushCodexCache() + expect(await readAntigravityModel(antigravitySource)).toBe('before') + + // Another process republishes both cache files. Without the clear, the + // resident keeps serving its warm copies. + await seedAntigravityCache(cacheDir, antigravitySource, 'after') + const codexDisk = JSON.parse(await readFile(join(cacheDir, 'codex-results.json'), 'utf8')) + codexDisk.files[codexSource].calls[0].model = 'after' + await writeFile(join(cacheDir, 'codex-results.json'), JSON.stringify(codexDisk)) + + expect((await readCachedCodexResults(codexSource))?.calls.map(entry => entry.model)).toEqual(['before']) + expect(await readAntigravityModel(antigravitySource)).toBe('before') + + clearCodexMemCaches() + clearAntigravityCacheStates() + + expect((await readCachedCodexResults(codexSource))?.calls.map(entry => entry.model)).toEqual(['after']) + expect(await readAntigravityModel(antigravitySource)).toBe('after') + }) +}) diff --git a/tests/cache-persistence-coldstart.test.ts b/tests/cache-persistence-coldstart.test.ts index c18b677f..7ab1d893 100644 --- a/tests/cache-persistence-coldstart.test.ts +++ b/tests/cache-persistence-coldstart.test.ts @@ -4,7 +4,8 @@ import { join } from 'path' import { tmpdir } from 'os' import { parseAllSessions, clearSessionCache } from '../src/parser.js' -import { CACHE_VERSION, sessionCachePath } from '../src/session-cache.js' +import { CACHE_VERSION } from '../src/session-cache.js' +import { readCacheOnDisk } from './fixtures/session-cache-io.js' let tmpDir: string let cacheDir: string @@ -48,7 +49,7 @@ describe('cold-start cache persistence', () => { const projects = await parseAllSessions() expect(projects.length).toBeGreaterThan(0) - const raw = JSON.parse(await readFile(sessionCachePath(), 'utf-8')) + const raw = await readCacheOnDisk() expect(raw.version).toBe(CACHE_VERSION) const claudeFiles = Object.keys(raw.providers?.claude?.files ?? {}) expect(claudeFiles.length).toBeGreaterThan(0) diff --git a/tests/cache-refresh-lock.test.ts b/tests/cache-refresh-lock.test.ts index cdabaeec..b59c6834 100644 --- a/tests/cache-refresh-lock.test.ts +++ b/tests/cache-refresh-lock.test.ts @@ -8,7 +8,7 @@ import { type RefreshLockClock, } from '../src/cache-refresh-lock.js' import { clearSessionCache, parseAllSessions } from '../src/parser.js' -import { emptyCache, loadCache, saveCache, sessionCachePath } from '../src/session-cache.js' +import { emptyCache, loadCache, saveCache, sessionCacheDir } from '../src/session-cache.js' const dirs: string[] = [] @@ -202,7 +202,7 @@ describe('warm session-cache refresh lock', () => { await result.handle.release() expect(JSON.parse(await readFile(lockPath(dir), 'utf-8')).token).toBe('successor') - expect(sessionCachePath()).toContain(dir) + expect(sessionCacheDir()).toContain(dir) }) // retry shields environmental fd/CPU starvation in a saturated full-suite diff --git a/tests/codex-cache-invalidation.test.ts b/tests/codex-cache-invalidation.test.ts index e4e53111..35b80f7f 100644 --- a/tests/codex-cache-invalidation.test.ts +++ b/tests/codex-cache-invalidation.test.ts @@ -13,7 +13,7 @@ import { createHash } from 'crypto' import { join } from 'path' import { clearSessionCache, parseAllSessions } from '../src/parser.js' -import { sessionCachePath } from '../src/session-cache.js' +import { readCacheOnDisk, writeCacheOnDisk } from './fixtures/session-cache-io.js' const testRoot = vi.hoisted(() => { const root = `${process.env['TMPDIR'] || '/tmp'}/codex-stale-repro-${process.pid}-${Date.now()}` @@ -75,8 +75,7 @@ describe('codex parser change invalidates stale session-cache (#478/#513)', () = // release: pre-fix envFingerprint, unchanged file fingerprint, cached // turns lack the mcp__ tool. Also reset codex-results.json to v4 so the // provider (if it runs at all) must genuinely re-parse. - const cachePath = sessionCachePath() - const cache = JSON.parse(await readFile(cachePath, 'utf8')) + const cache = await readCacheOnDisk() as any cache.providers.codex.envFingerprint = preFixFingerprint() for (const f of Object.values(cache.providers.codex.files) as any[]) { for (const turn of f.turns) { @@ -88,7 +87,7 @@ describe('codex parser change invalidates stale session-cache (#478/#513)', () = } } } - await writeFile(cachePath, JSON.stringify(cache)) + await writeCacheOnDisk(cache) const codexCachePath = join(CACHE_DIR, 'codex-results.json') const codexCache = JSON.parse(await readFile(codexCachePath, 'utf8')) codexCache.version = 4 diff --git a/tests/fixtures/cache-refresh-worker.ts b/tests/fixtures/cache-refresh-worker.ts index a2807df1..1a24c0c4 100644 --- a/tests/fixtures/cache-refresh-worker.ts +++ b/tests/fixtures/cache-refresh-worker.ts @@ -3,7 +3,7 @@ import { mkdir, readFile, writeFile } from 'fs/promises' import { join } from 'path' import { acquireCacheRefreshLock } from '../../src/cache-refresh-lock.js' -import { loadCache, saveCache } from '../../src/session-cache.js' +import { loadCache, markCacheDirty, saveCache } from '../../src/session-cache.js' const [cacheDir, barrierDir, id, sourcePath, bypass = 'false'] = process.argv.slice(2) if (!cacheDir || !barrierDir || !id || !sourcePath) throw new Error('missing worker argument') @@ -33,7 +33,7 @@ try { mcpInventory: [], turns: [], } - ;(cache as { _dirty?: boolean })._dirty = true + markCacheDirty(cache, 'regression') await writeFile(join(barrierDir, `${id}.parsed`), '') await waitFor(`${id}.save`) const published = await saveCache(cache, refresh?.handle.verifyStillOwner) diff --git a/tests/fixtures/session-cache-io.ts b/tests/fixtures/session-cache-io.ts new file mode 100644 index 00000000..445a769b --- /dev/null +++ b/tests/fixtures/session-cache-io.ts @@ -0,0 +1,35 @@ +// Test-side IO for the sharded session cache: the on-disk form is a directory +// (envelope + one shard per provider), so tests read and write it through the +// real load/save path instead of touching a single JSON file. +import { readFile, readdir } from 'fs/promises' +import { join } from 'path' + +import { + clearLoadCacheMemo, + loadCache, + markCacheDirty, + saveCache, + sessionCacheDir, + type SessionCache, +} from '../../src/session-cache.js' + +/** The cache exactly as it is on disk, bypassing the in-process memo. */ +export async function readCacheOnDisk(): Promise { + clearLoadCacheMemo() + return loadCache() +} + +/** Publish `cache`, rewriting every provider's shard. */ +export async function writeCacheOnDisk(cache: SessionCache): Promise { + for (const provider of Object.keys(cache.providers)) markCacheDirty(cache, provider) + await saveCache(cache) + clearLoadCacheMemo() +} + +/** Byte-level snapshot of the whole cache directory (names + contents). */ +export async function cacheDirSnapshot(): Promise { + const dir = sessionCacheDir() + const names = (await readdir(dir)).sort() + const parts = await Promise.all(names.map(async name => `${name}:${await readFile(join(dir, name), 'utf-8')}`)) + return parts.join('\n') +} diff --git a/tests/flat-slice.test.ts b/tests/flat-slice.test.ts new file mode 100644 index 00000000..8e997ba4 --- /dev/null +++ b/tests/flat-slice.test.ts @@ -0,0 +1,117 @@ +/** + * Tests for flatSlice — the SlicedString-retention fix. + * + * Background: `String.prototype.slice` returns a V8 SlicedString that + * retains its entire parent string. Storing short slices of large session + * strings (100KB+ agent prompts) in the long-lived session cache pinned + * gigabytes of parent buffers during cold parses, OOMing the default heap + * (issue observed at ~5.5GB peak for 3.2GB of kiro session files; ~300MB + * after flattening). + */ + +import { describe, it, expect } from 'vitest' + +import { flatSlice, flatString } from '../src/content-utils.js' + +describe('flatSlice', () => { + it('returns the prefix for strings over the bound', () => { + const big = 'x'.repeat(10_000) + const out = flatSlice(big, 500) + expect(out.length).toBe(500) + expect(out).toBe(big.slice(0, 500)) + }) + + it('returns the string itself when within the bound', () => { + const small = 'hello world' + expect(flatSlice(small, 500)).toBe(small) + }) + + it('handles multi-byte characters without corruption', () => { + // Emoji + CJK near the boundary — Buffer round-trip must not produce + // invalid UTF-8 replacement chars for chars fully inside the slice. + const s = '🐾'.repeat(300) // each emoji is 2 UTF-16 code units + const out = flatSlice(s, 500) + expect(out).toBe(s.slice(0, 500)) + }) + + it('preserves a lone surrogate at a mid-pair cut', () => { + // A cut landing between the high and low surrogate of a pair leaves a + // lone surrogate. utf16le round-trips code units byte-for-byte, so the + // lone surrogate survives intact (unlike utf-8, which would replace it + // with U+FFFD). + const s = 'ab' + '🐾'.repeat(300) // odd offset puts every emoji across even boundaries + const out = flatSlice(s, 501) // cuts mid-pair + expect(out.length).toBe(501) + expect(out.slice(0, 500)).toBe(s.slice(0, 500)) // content before the cut intact + expect(out.charCodeAt(500)).toBe(s.charCodeAt(500)) // lone surrogate preserved + }) + + it('does not retain the parent of an already-sliced view', () => { + // The bug this early-return removal fixes: provider adapters pre-truncate + // with .slice(0, 500) before the cache-site flatSlice call, so a naive + // "already within bound" early return would skip flattening and leave + // the SlicedString pinning its 100KB parent. + const before = process.memoryUsage().heapUsed + const kept: string[] = [] + for (let i = 0; i < 1000; i++) { + const parent = (i % 10).toString().repeat(100_000) + i + const preSliced = parent.slice(0, 500) + kept.push(flatSlice(preSliced, 2000)) + } + if (typeof global.gc === 'function') global.gc() + const after = process.memoryUsage().heapUsed + const growthMB = (after - before) / 1048576 + expect(kept.length).toBe(1000) + expect(growthMB).toBeLessThan(50) + }) + + it('does not retain the parent string (heap growth stays bounded)', () => { + // Property test for the retention fix: keep 1000 short prefixes of + // 1000 distinct 100KB strings. With plain .slice() each prefix pins its + // 100KB parent (~200MB in UTF-16 total). With flatSlice, retained data + // is ~1000 × 500 chars ≈ 1MB. Assert heap growth is far below the + // retention scenario. Threshold is generous (50MB) to be CI-safe while + // still failing decisively if retention returns (>190MB). When the test + // runner exposes gc (vitest under --expose-gc), force a collection so + // transient parent garbage doesn't inflate the measurement. + const before = process.memoryUsage().heapUsed + const kept: string[] = [] + for (let i = 0; i < 1000; i++) { + // Distinct content so V8 cannot intern/share the parents. + const parent = (i % 10).toString().repeat(100_000) + kept.push(flatSlice(parent + i, 500)) + } + if (typeof global.gc === 'function') global.gc() + const after = process.memoryUsage().heapUsed + const growthMB = (after - before) / 1048576 + expect(kept.length).toBe(1000) + expect(growthMB).toBeLessThan(50) + }) +}) + +describe('flatString', () => { + it('returns an equal string for any input', () => { + expect(flatString('')).toBe('') + expect(flatString('hello')).toBe('hello') + expect(flatString('🐾 multi-byte ✓')).toBe('🐾 multi-byte ✓') + }) + + it('does not retain the parent of a regex match group', () => { + // match[1] is a SlicedString retaining the entire subject. flatString + // must break that link: keep 1000 short match groups of distinct 100KB + // subjects and assert bounded heap growth (same thresholds as the + // flatSlice retention test). + const before = process.memoryUsage().heapUsed + const kept: string[] = [] + for (let i = 0; i < 1000; i++) { + const subject = `tool_${i}` + (i % 10).toString().repeat(100_000) + const m = /([^<]+)<\/name>/.exec(subject) + kept.push(flatString(m![1]!)) + } + if (typeof global.gc === 'function') global.gc() + const after = process.memoryUsage().heapUsed + const growthMB = (after - before) / 1048576 + expect(kept.length).toBe(1000) + expect(growthMB).toBeLessThan(50) + }) +}) diff --git a/tests/hydration-interrupt-convergence.test.ts b/tests/hydration-interrupt-convergence.test.ts index 7ad68a09..e3cb761e 100644 --- a/tests/hydration-interrupt-convergence.test.ts +++ b/tests/hydration-interrupt-convergence.test.ts @@ -6,7 +6,7 @@ import { afterEach, beforeAll, beforeEach, describe, expect, it } from 'vitest' import { loadPricing, setLocalModelSavings, setModelAliases } from '../src/models.js' import { buildMenubarPayloadForRange } from '../src/usage-aggregator.js' import { clearSessionCache } from '../src/parser.js' -import { sessionCachePath } from '../src/session-cache.js' +import { readCacheOnDisk, writeCacheOnDisk } from './fixtures/session-cache-io.js' import { dailyCachePath } from '../src/daily-cache.js' import type { DateRange } from '../src/types.js' @@ -91,9 +91,9 @@ describe('interrupted hydration converges to the uninterrupted result', () => { // (a) Session cache: present but NOT marked complete — an interrupted cold // start's throttled partial save. - const sessionRaw = JSON.parse(await readFile(sessionCachePath(), 'utf-8')) + const sessionRaw = await readCacheOnDisk() sessionRaw.complete = false - await writeFile(sessionCachePath(), JSON.stringify(sessionRaw), 'utf-8') + await writeCacheOnDisk(sessionRaw) // (b) Daily cache: frozen with the older days dropped but `lastComputedDate` // advanced to yesterday and NO completeness marker — the exact freeze that @@ -118,7 +118,7 @@ describe('interrupted hydration converges to the uninterrupted result', () => { expect(healed.current.calls).toBe(reference.current.calls) // And the on-disk markers are now durably complete, so the next launch is warm. - expect(JSON.parse(await readFile(sessionCachePath(), 'utf-8')).complete).toBe(true) + expect((await readCacheOnDisk()).complete).toBe(true) expect(JSON.parse(await readFile(dailyCachePath(), 'utf-8')).complete).toBe(true) }) }) diff --git a/tests/kiro-cache-invalidation.test.ts b/tests/kiro-cache-invalidation.test.ts index 0046dd7a..93bc9c1b 100644 --- a/tests/kiro-cache-invalidation.test.ts +++ b/tests/kiro-cache-invalidation.test.ts @@ -25,9 +25,9 @@ import { CACHE_VERSION, computeEnvFingerprint, fingerprintFile, - sessionCachePath, type SessionCache, } from '../src/session-cache.js' +import { writeCacheOnDisk } from './fixtures/session-cache-io.js' // The kiro provider singleton captures homedir() when its module is first // imported, so HOME must point at the test root before ../src/parser.js is @@ -99,7 +99,7 @@ async function seedCache(execPath: string, envFingerprint: string): Promise { + const root = `${process.env['TMPDIR'] || '/tmp'}/kiro-projpath-${process.pid}-${Date.now()}` + process.env['HOME'] = `${root}/home` + process.env['USERPROFILE'] = `${root}/home` + return root +}) + +const HOME = join(testRoot, 'home') +const CACHE_DIR = join(testRoot, 'cache') +const KIRO_SESSIONS = join(HOME, '.kiro', 'sessions') +const CLI_DIR = join(KIRO_SESSIONS, 'cli') + +const CLI_CWD = '/local/home/testuser/workplace/my-project' +const V2_WORKSPACE = '/local/home/testuser/workplace/ide-project' + +beforeEach(() => { + process.env['HOME'] = HOME + process.env['USERPROFILE'] = HOME + process.env['CODEBURN_CACHE_DIR'] = CACHE_DIR + delete process.env['KIRO_HOME'] + clearSessionCache() +}) + +afterAll(async () => { + await rm(testRoot, { recursive: true, force: true }) +}) + +/** Write a minimal kiro CLI session: .jsonl entries + companion .json meta. */ +async function seedCliSession(id: string, cwd: string): Promise { + await mkdir(CLI_DIR, { recursive: true }) + const jsonlPath = join(CLI_DIR, `${id}.jsonl`) + const entries = [ + { kind: 'Prompt', data: { content: [{ kind: 'text', data: 'add a feature' }] } }, + { kind: 'AssistantMessage', data: { content: [{ kind: 'text', data: 'Done — added the feature and tests.' }] } }, + ] + await writeFile(jsonlPath, entries.map(e => JSON.stringify(e)).join('\n')) + await writeFile(join(CLI_DIR, `${id}.json`), JSON.stringify({ + session_id: id, + cwd, + created_at: '2026-08-01T10:00:00Z', + updated_at: '2026-08-01T10:05:00Z', + session_state: { + rts_model_state: { model_info: { model_id: 'auto' } }, + conversation_metadata: { + user_turn_metadatas: [ + { end_timestamp: '2026-08-01T10:05:00Z', metering_usage: [] }, + ], + }, + }, + })) + return jsonlPath +} + +/** Write a minimal v2 IDE session: sessions//sess_/{session.json,messages.jsonl}. */ +async function seedV2Session(id: string, workspacePath: string): Promise { + const sessDir = join(KIRO_SESSIONS, 'f'.repeat(32), `sess_${id}`) + await mkdir(sessDir, { recursive: true }) + await writeFile(join(sessDir, 'session.json'), JSON.stringify({ + id, + modelId: 'auto', + workspacePaths: [workspacePath], + createdAt: '2026-08-01T11:00:00Z', + })) + const events = [ + { timestamp: '2026-08-01T11:00:00Z', payload: { type: 'user', content: 'fix the bug' } }, + { timestamp: '2026-08-01T11:00:01Z', payload: { type: 'turn_start', executionId: 'x1' } }, + { timestamp: '2026-08-01T11:00:05Z', payload: { type: 'assistant', content: 'Fixed the bug in handler.ts by checking null first.' } }, + { timestamp: '2026-08-01T11:00:06Z', payload: { type: 'turn_end', executionId: 'x1' } }, + ] + await writeFile(join(sessDir, 'messages.jsonl'), events.map(e => JSON.stringify(e)).join('\n')) +} + +function kiroAgentDir(): string { + if (process.platform === 'darwin') { + return join(HOME, 'Library', 'Application Support', 'Kiro', 'User', 'globalStorage', 'kiro.kiroagent') + } + if (process.platform === 'win32') { + return join(HOME, 'AppData', 'Roaming', 'Kiro', 'User', 'globalStorage', 'kiro.kiroagent') + } + return join(HOME, '.config', 'Kiro', 'User', 'globalStorage', 'kiro.kiroagent') +} + +/** Write a minimal IDE workspace-session: + * /workspace-sessions//.json */ +async function seedWorkspaceSession(id: string, workspaceDirectory: string): Promise { + const encoded = Buffer.from(workspaceDirectory, 'utf-8').toString('base64').replace(/=/g, '_') + const dir = join(kiroAgentDir(), 'workspace-sessions', encoded) + await mkdir(dir, { recursive: true }) + await writeFile(join(dir, `${id}.json`), JSON.stringify({ + sessionId: id, + selectedModel: 'auto', + workspaceDirectory, + history: [ + { message: { role: 'user', content: 'refactor the config loader' } }, + { message: { role: 'assistant', content: 'Refactored the loader into three small functions with tests.' } }, + ], + })) +} + +async function kiroCalls() { + const projects = await parseAllSessions(undefined, 'kiro') + return projects.flatMap(p => p.sessions.map(s => ({ project: p.project, projectPath: p.projectPath, session: s }))) +} + +describe('kiro projectPath emission', () => { + it('CLI session: projectPath is the full meta.cwd, project the basename', async () => { + await seedCliSession('cli-001', CLI_CWD) + const rows = await kiroCalls() + const row = rows.find(r => r.project === 'my-project') + expect(row).toBeDefined() + expect(row!.projectPath).toBe(CLI_CWD) + }) + + it('v2 IDE session: projectPath is workspacePaths[0]', async () => { + await seedV2Session('v2-001', V2_WORKSPACE) + const rows = await kiroCalls() + const row = rows.find(r => r.project === 'ide-project') + expect(row).toBeDefined() + expect(row!.projectPath).toBe(V2_WORKSPACE) + }) + + it('workspace session: projectPath is workspaceDirectory', async () => { + const WS_DIR = '/local/home/testuser/workplace/ws-project' + await seedWorkspaceSession('ws-001', WS_DIR) + const rows = await kiroCalls() + const row = rows.find(r => r.project === 'ws-project') + expect(row).toBeDefined() + expect(row!.projectPath).toBe(WS_DIR) + }) +}) + +describe('kiro projectPath cache invalidation (project-path-v1 bump)', () => { + // The fingerprint a cache written by the PREVIOUS release carries: same env + // vars, but the parser version before the project-path-v1 bump. + function preBumpFingerprint(): string { + const parts = [`KIRO_HOME=${process.env['KIRO_HOME'] ?? ''}`, 'parser=ide-parsing-v1-est-cost'] + return createHash('sha256').update(parts.join('\0')).digest('hex').slice(0, 16) + } + + it('the bump changed the env fingerprint', () => { + expect(computeEnvFingerprint('kiro')).not.toBe(preBumpFingerprint()) + }) + + it('a pre-bump cache entry (no projectPath) is re-parsed and gains projectPath', async () => { + const jsonlPath = await seedCliSession('cli-002', CLI_CWD) + + // Seed a cache exactly as the pre-bump release would have left it: + // correct file fingerprint, pre-bump env fingerprint, turns WITHOUT + // projectPath on the cached calls. + const fp = await fingerprintFile(jsonlPath) + if (!fp) throw new Error('failed to fingerprint seeded session file') + const cache: SessionCache = { + version: CACHE_VERSION, + providers: { + kiro: { + envFingerprint: preBumpFingerprint(), + files: { + [jsonlPath]: { fingerprint: fp, mcpInventory: [], turns: [] }, + }, + }, + }, + } + await mkdir(CACHE_DIR, { recursive: true }) + await writeCacheOnDisk(cache) + clearSessionCache() + + const rows = await kiroCalls() + const row = rows.find(r => r.project === 'my-project') + expect(row).toBeDefined() + expect(row!.projectPath).toBe(CLI_CWD) + }) +}) diff --git a/tests/parse-workers.test.ts b/tests/parse-workers.test.ts new file mode 100644 index 00000000..4185be59 --- /dev/null +++ b/tests/parse-workers.test.ts @@ -0,0 +1,484 @@ +import { spawnSync } from 'node:child_process' +import { appendFile, mkdir, mkdtemp, readFile, readdir, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { createHash } from 'node:crypto' + +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' + +import { decideParseWorkers, ParseWorkerPool, parseFilesInOrder, type ClaudeWorkerParse } from '../src/parse-workers.js' +import { clearSessionCache, parseAllSessions, parseClaudeFileFull } from '../src/parser.js' +import { parseCodexFileFull, type CodexFullParse } from '../src/providers/codex.js' +import type { SessionSource } from '../src/providers/types.js' + +// Two full cold CLI parses of a multi-hundred-file corpus, plus in-process parses +// that spawn real threads. +vi.setConfig({ testTimeout: 60_000 }) + +const BIG_SYSTEM = { cores: 16, availableBytes: 32 * 1024 ** 3 } +const BIG_PENDING = { files: 5000, bytes: 6 * 1024 ** 3 } +const NO_ENV = {} as NodeJS.ProcessEnv + +describe('decideParseWorkers', () => { + it('scales with cores, memory budget and pending file count', () => { + // 15 (cores-1) vs 8 (2 GB budget / 256 MB) vs 100 (5000/50) -> memory cap wins + expect(decideParseWorkers(BIG_PENDING, BIG_SYSTEM, NO_ENV).workers).toBe(8) + // Fewer cores than the memory budget allows -> cores-1 wins + expect(decideParseWorkers(BIG_PENDING, { cores: 6, availableBytes: 32 * 1024 ** 3 }, NO_ENV).workers).toBe(5) + // 8 GB reaches the same cap as 32 GB: a quarter of it is the 2 GB budget + expect(decideParseWorkers(BIG_PENDING, { cores: 16, availableBytes: 8 * 1024 ** 3 }, NO_ENV).workers).toBe(8) + // Under that, the quarter-of-available budget is the binding constraint + expect(decideParseWorkers(BIG_PENDING, { cores: 16, availableBytes: 6 * 1024 ** 3 }, NO_ENV).workers).toBe(6) + // The smallest machine that clears every gate still only earns 2 threads + expect(decideParseWorkers({ files: 200, bytes: 300 * 1024 ** 2 }, { cores: 3, availableBytes: 4 * 1024 ** 3 }, NO_ENV).workers).toBe(2) + // Big average file: the per-worker budget scales with it (2 x 260 MB + 128 MB), + // so the same 2 GB buys 3 threads instead of the 8 a flat 256 MB would. + expect(decideParseWorkers({ files: 250, bytes: 65 * 1024 ** 3 }, BIG_SYSTEM, NO_ENV).workers).toBe(3) + // 300 files earn 6, but the 6 GB behind them earn 30 — bytes win, then the + // memory budget caps it. A Codex corpus is exactly this shape. + expect(decideParseWorkers({ files: 300, bytes: 6 * 1024 ** 3 }, BIG_SYSTEM, NO_ENV).workers).toBe(8) + // Few enough files AND bytes that MIN_FILES_PER_WORKER is the binding constraint + expect(decideParseWorkers({ files: 300, bytes: 700 * 1024 ** 2 }, BIG_SYSTEM, NO_ENV).workers).toBe(6) + }) + + it('stays serial on low-spec machines and on warm/small parses', () => { + expect(decideParseWorkers(BIG_PENDING, { cores: 2, availableBytes: 32 * 1024 ** 3 }, NO_ENV).workers).toBe(0) + // A 4 GB box: availableMemory() always reads a little under the nominal size + expect(decideParseWorkers(BIG_PENDING, { cores: 16, availableBytes: 3.9 * 1024 ** 3 }, NO_ENV).workers).toBe(0) + // Warm/incremental: the byte gate is not reached + expect(decideParseWorkers({ files: 12, bytes: 10 * 1024 ** 2 }, BIG_SYSTEM, NO_ENV).workers).toBe(0) + }) + + it('gates on bytes alone, so a thin corpus never spawns threads it cannot pay for', () => { + // 250 files holding under a megabyte between them: threads made this ~5% slower + expect(decideParseWorkers({ files: 250, bytes: 917 * 1024 }, BIG_SYSTEM, NO_ENV).workers).toBe(0) + expect(decideParseWorkers({ files: 5000, bytes: 10 * 1024 ** 2 }, BIG_SYSTEM, NO_ENV).workers).toBe(0) + expect(decideParseWorkers({ files: 250, bytes: 917 * 1024 }, BIG_SYSTEM, NO_ENV).reason) + .toContain('below 210 MB pending') + // 150 rollouts over the byte gate: far under any file-count threshold, and the + // biggest workload there is + expect(decideParseWorkers({ files: 150, bytes: 4 * 1024 ** 3 }, BIG_SYSTEM, NO_ENV).workers).toBe(8) + }) + + it('honours CODEBURN_PARSE_WORKERS, which also bypasses the auto gates', () => { + expect(decideParseWorkers(BIG_PENDING, BIG_SYSTEM, { CODEBURN_PARSE_WORKERS: '0' }).workers).toBe(0) + expect(decideParseWorkers(BIG_PENDING, BIG_SYSTEM, { CODEBURN_PARSE_WORKERS: '4' }).workers).toBe(4) + // Capped by the core count + expect(decideParseWorkers(BIG_PENDING, { cores: 4, availableBytes: 32 * 1024 ** 3 }, { CODEBURN_PARSE_WORKERS: '32' }).workers).toBe(4) + // A tiny fixture corpus still gets threads when forced — that is what makes + // the determinism test below able to exercise them at all. + expect(decideParseWorkers({ files: 3, bytes: 1000 }, BIG_SYSTEM, { CODEBURN_PARSE_WORKERS: '3' }).workers).toBe(3) + expect(decideParseWorkers(BIG_PENDING, BIG_SYSTEM, { CODEBURN_PARSE_WORKERS: 'nonsense' }).workers).toBe(0) + }) + + it('reports the decision inputs in every reason, gate or not', () => { + for (const d of [ + decideParseWorkers(BIG_PENDING, BIG_SYSTEM, NO_ENV), + decideParseWorkers({ files: 12, bytes: 1000 }, BIG_SYSTEM, NO_ENV), + decideParseWorkers(BIG_PENDING, BIG_SYSTEM, { CODEBURN_PARSE_WORKERS: '2' }), + ]) { + expect(d.reason).toContain('16 cores') + expect(d.reason).toContain('GB available') + expect(d.reason).toContain('pending files') + } + }) +}) + +type Turn = { id: string; t: number } + +function sessionLines(project: string, session: string, turns: Turn[]): string { + const lines: string[] = [] + for (const { id, t } of turns) { + const ts = new Date(Date.UTC(2026, 4, 4 + (t % 5), 9, t % 60, 0)).toISOString() + const gitBranch = t % 3 === 0 ? 'main' : 'feature' + lines.push(JSON.stringify({ + type: 'user', sessionId: session, timestamp: ts, cwd: `/tmp/proj${project}`, gitBranch, + message: { role: 'user', content: `task ${t} in ${project}` }, + })) + lines.push(JSON.stringify({ + type: 'assistant', sessionId: session, timestamp: ts, cwd: `/tmp/proj${project}`, gitBranch, + message: { + id, type: 'message', role: 'assistant', model: 'claude-sonnet-4-5', + content: [ + { type: 'text', text: 'x'.repeat(200) }, + { type: 'tool_use', id: `tu-${id}`, name: 'Edit', input: { file_path: '/tmp/x', old_string: 'a', new_string: 'b' } }, + ], + usage: { input_tokens: 400 + t, output_tokens: 40 + t, cache_read_input_tokens: 9 }, + }, + })) + } + return lines.join('\n') + '\n' +} + +const range = (n: number, from = 0): number[] => Array.from({ length: n }, (_, i) => i + from) + +async function writeCorpus(claudeDir: string, projects: number, filesPerProject: number): Promise { + const written: string[] = [] + for (let p = 0; p < projects; p++) { + const dir = join(claudeDir, 'projects', `-tmp-proj${p}`) + await mkdir(dir, { recursive: true }) + for (let f = 0; f < filesPerProject; f++) { + const session = `${p}${f}`.padStart(8, '0') + '-aaaa-bbbb-cccc-000000000000' + const path = join(dir, `${session}.jsonl`) + await writeFile(path, sessionLines(String(p), session, range(12).map(t => ({ id: `msg-${p}-${session}-${t}`, t })))) + written.push(path) + } + } + return written +} + +/// A resumed Claude session: the new transcript restates the original's assistant +/// messages verbatim (same message ids) before adding its own. Cross-file dedup +/// means whichever file is installed FIRST keeps those turns and the other loses +/// them, so this fixture is only stable if worker results are installed in the +/// serial order — and it is the only fixture that drives the discard/re-parse path, +/// since a worker parses against an empty dedup set and cannot see the overlap. +async function writeResumedPair(claudeDir: string, tag: string, originalName: string, resumedName: string): Promise { + const dir = join(claudeDir, 'projects', `-tmp-${tag}`) + await mkdir(dir, { recursive: true }) + const shared = range(6).map(t => ({ id: `${tag}-m${t}`, t })) + await writeFile(join(dir, `${originalName}.jsonl`), sessionLines(tag, originalName, shared)) + await writeFile( + join(dir, `${resumedName}.jsonl`), + sessionLines(tag, resumedName, [...shared, ...range(4, 6).map(t => ({ id: `${tag}-n${t}`, t }))]), + ) +} + +type CodexTask = { n: number; at: string } + +/// One Codex rollout: session_meta plus a complete task cycle per entry. A +/// token_count dedup key is namespaced by the FORK PARENT when there is one and +/// keyed on the cumulative token breakdown, so a fork restating a parent's tasks +/// emits exactly the parent's keys — the cross-file overlap that only the +/// install-order check can resolve. The replayed tasks are timestamped well past +/// `metaTs + 5s` on purpose: inside that window the parser drops replays outright +/// and the dedup path would never be reached. +function codexRollout(sessionId: string, cwd: string, tasks: CodexTask[], forkedFrom?: string, metaTs = '2026-05-04T09:00:00.000Z'): string { + const lines = [JSON.stringify({ + type: 'session_meta', + timestamp: metaTs, + payload: { + cwd, originator: 'codex-cli', session_id: sessionId, model: 'gpt-5.3-codex', + ...(forkedFrom ? { forked_from_id: forkedFrom } : {}), + }, + })] + lines.push(...codexTaskLines(tasks)) + return lines.join('\n') + '\n' +} + +function codexTaskLines(tasks: CodexTask[]): string[] { + const lines: string[] = [] + for (const { n, at } of tasks) { + const ts = (s: number) => new Date(Date.parse(at) + s * 1000).toISOString() + lines.push( + JSON.stringify({ type: 'event_msg', timestamp: ts(0), payload: { type: 'task_started' } }), + JSON.stringify({ type: 'response_item', timestamp: ts(1), payload: { type: 'message', role: 'user', content: [{ type: 'input_text', text: `codex task ${n}` }] } }), + JSON.stringify({ type: 'response_item', timestamp: ts(2), payload: { type: 'function_call', name: 'shell', call_id: `c${n}`, arguments: JSON.stringify({ command: `ls ${n}` }) } }), + JSON.stringify({ type: 'response_item', timestamp: ts(3), payload: { type: 'function_call_output', call_id: `c${n}` } }), + JSON.stringify({ type: 'event_msg', timestamp: ts(4), payload: { type: 'patch_apply_end', success: true, changes: { [`/tmp/cx/f${n}.ts`]: { unified_diff: '@@ -1 +1,2 @@\n-old\n+new\n+extra\n' } } } }), + JSON.stringify({ type: 'response_item', timestamp: ts(5), payload: { type: 'message', role: 'assistant', content: [{ type: 'output_text', text: 'y'.repeat(120) }] } }), + JSON.stringify({ + type: 'event_msg', timestamp: ts(6), + payload: { + type: 'token_count', + info: { + last_token_usage: { input_tokens: 100, cached_input_tokens: 20, output_tokens: 50, reasoning_output_tokens: 10, total_tokens: 180 }, + total_token_usage: { input_tokens: 100 * n, cached_input_tokens: 20 * n, output_tokens: 50 * n, reasoning_output_tokens: 10 * n, total_tokens: 160 * n }, + }, + }, + }), + JSON.stringify({ type: 'event_msg', timestamp: ts(7), payload: { type: 'task_complete', duration_ms: 4000 } }), + ) + } + return lines +} + +async function writeCodexRollout(codexHome: string, day: string, name: string, body: string): Promise { + const dir = join(codexHome, 'sessions', '2026', '05', day) + await mkdir(dir, { recursive: true }) + const path = join(dir, `rollout-${name}.jsonl`) + await writeFile(path, body) + return path +} + +/// A parent rollout and a fork that replays its tasks before adding its own. +/// `parentFirst` flips which file is created first, since discovery follows +/// directory order: the shared keys must land on whichever file the SERIAL loop +/// reaches first, in either order. +async function writeForkedCodexPair(codexHome: string, day: string, tag: string, parentFirst: boolean): Promise { + const shared = [1, 2, 3].map(n => ({ n, at: `2026-05-04T09:${String(10 + n).padStart(2, '0')}:00.000Z` })) + const parent = codexRollout(`${tag}-parent`, `/tmp/cx${tag}`, shared) + const fork = codexRollout( + `${tag}-fork`, + `/tmp/cx${tag}`, + [...shared.map(t => ({ ...t, at: `2026-05-04T10:${String(10 + t.n).padStart(2, '0')}:00.000Z` })), { n: 4, at: '2026-05-04T10:30:00.000Z' }], + `${tag}-parent`, + ) + const order: Array<[string, string]> = parentFirst + ? [[`${tag}-a-parent`, parent], [`${tag}-b-fork`, fork]] + : [[`${tag}-a-fork`, fork], [`${tag}-b-parent`, parent]] + for (const [name, body] of order) await writeCodexRollout(codexHome, day, name, body) +} + +/// Cache shard file names carry a random nonce, so compare bodies keyed by +/// `.` instead of by file name. +async function shardBodies(cacheDir: string): Promise> { + const dir = join(cacheDir, 'session-cache.v9') + const out: Record = {} + for (const name of (await readdir(dir).catch(() => []))) { + if (name === 'envelope.json' || !name.endsWith('.json')) continue + const key = name.split('.').slice(0, 2).join('.') + out[key] = createHash('sha256').update(await readFile(join(dir, name))).digest('hex') + } + return out +} + +/// The Codex incremental cache is a single JSON file; both runs read the same +/// rollouts, so it must come out identical byte for byte. +async function codexResults(cacheDir: string): Promise { + return readFile(join(cacheDir, 'codex-results.json'), 'utf-8').catch(() => null) +} + +function runCli(args: string[], home: string, extraEnv: Record) { + return spawnSync(process.execPath, ['--import', 'tsx', 'src/cli.ts', ...args], { + cwd: process.cwd(), + env: { + ...process.env, + CLAUDE_CONFIG_DIR: join(home, '.claude'), + CODEX_HOME: join(home, '.codex'), + CODEBURN_CACHE_DIR: join(home, '.cache', 'codeburn'), + HOME: home, + TZ: 'UTC', + ...extraEnv, + }, + encoding: 'utf-8', + timeout: 60_000, + }) +} + +function stripVolatile(payload: unknown): unknown { + if (Array.isArray(payload)) return payload.map(stripVolatile) + if (payload && typeof payload === 'object') { + return Object.fromEntries( + Object.entries(payload as Record) + .filter(([k]) => !k.toLowerCase().startsWith('generated')) + .map(([k, v]) => [k, stripVolatile(v)]), + ) + } + if (typeof payload === 'number') return Math.round(payload * 1e9) / 1e9 + return payload +} + +describe('parallel cold parse', () => { + let home: string + + beforeEach(async () => { + home = await mkdtemp(join(tmpdir(), 'cb-cold-')) + }) + + afterEach(async () => { + await rm(home, { recursive: true, force: true }) + }) + + /// Both runs read the SAME corpus, so the absolute paths embedded in the cache + /// shards match and the bodies can be compared byte for byte. + async function bothWays(extraParallelEnv: Record = {}) { + const serialCache = join(home, 'cache-serial') + const parallelCache = join(home, 'cache-parallel') + const args = ['status', '--format', 'menubar-json'] + const serial = runCli(args, home, { CODEBURN_PARSE_WORKERS: '0', CODEBURN_CACHE_DIR: serialCache }) + const parallel = runCli(args, home, { CODEBURN_PARSE_WORKERS: '3', CODEBURN_CACHE_DIR: parallelCache, ...extraParallelEnv }) + + expect(serial.status, serial.stderr).toBe(0) + expect(parallel.status, parallel.stderr).toBe(0) + expect(stripVolatile(JSON.parse(parallel.stdout))).toEqual(stripVolatile(JSON.parse(serial.stdout))) + + const serialShards = await shardBodies(serialCache) + expect(Object.keys(serialShards).length).toBeGreaterThan(0) + expect(await shardBodies(parallelCache)).toEqual(serialShards) + // Byte-compared, not deep-equalled: the Codex cache is written entry by entry + // in install order, so the key order is itself a claim about that order. + expect(await codexResults(parallelCache)).toEqual(await codexResults(serialCache)) + return parallel + } + + // The whole point of the feature: threads may only ever be a speed change. + it('produces an identical payload and byte-identical cache shards with and without workers', async () => { + await writeCorpus(join(home, '.claude'), 4, 12) + await bothWays() + }) + + // Resumed sessions in both filename orders: the restating file sorts after the + // original in one project and before it in the other, so install order decides + // which file keeps the shared turns either way. Out-of-order installation, or + // any attempt to patch overlapping turns out of a worker result instead of + // discarding the whole file, changes the answer. + it("matches the serial parse when files restate each other's message ids", async () => { + const claude = join(home, '.claude') + await writeResumedPair(claude, 'fwd', '00000000-aaaa-bbbb-cccc-000000000000', '99999999-aaaa-bbbb-cccc-000000000000') + await writeResumedPair(claude, 'rev', '99999999-dddd-bbbb-cccc-000000000000', '00000000-dddd-bbbb-cccc-000000000000') + + const parallel = await bothWays({ CODEBURN_VERBOSE: '1' }) + + // Pin that the discard path actually ran rather than passing by luck. + const overlaps = [...parallel.stderr.matchAll(/(\d+)\/\d+ results re-parsed in-process on id overlap/g)] + .reduce((n, m) => n + Number(m[1]), 0) + expect(overlaps).toBeGreaterThan(0) + }) + + // Codex is the bigger half of a real cold parse, and its cross-file dedup is + // stronger than Claude's: a forked rollout replays its parent's token_count + // history under the PARENT's key namespace, so two files claim the same keys + // outright. Both fork orders are present, so install order decides which file + // keeps the shared tasks either way. + it('matches the serial parse for a mixed Claude + Codex corpus with forked rollouts', async () => { + await writeCorpus(join(home, '.claude'), 2, 6) + const codex = join(home, '.codex') + await writeForkedCodexPair(codex, '04', 'fwd', true) + await writeForkedCodexPair(codex, '05', 'rev', false) + for (const n of range(4)) { + await writeCodexRollout(codex, '06', `plain-${n}`, codexRollout(`plain-${n}`, `/tmp/cx${n}`, [1, 2].map(t => ({ n: t, at: `2026-05-06T0${n}:${t}0:00.000Z` })))) + } + + const parallel = await bothWays({ CODEBURN_VERBOSE: '1' }) + + expect(parallel.stderr).toContain('codeburn: codex parse workers=3') + // Pin that the codex-cache comparison in bothWays was not vacuous. + expect(await codexResults(join(home, 'cache-parallel'))).toContain('rollout-') + // Pin that the codex discard path actually ran rather than passing by luck. + const codexDiscards = [...parallel.stderr.matchAll(/codex parse workers done, (\d+)\/\d+ results/g)] + .reduce((n, m) => n + Number(m[1]), 0) + expect(codexDiscards).toBeGreaterThan(0) + }) + + // Workers only ever run WHOLE-file decodes. A rollout that grew by a few KB is + // resumed from its last task boundary in-process: a thread hop would cost more + // than it saves, and the resume state lives in the parent's codex cache. + it('never hands a resumable rollout to a worker', async () => { + const codex = join(home, '.codex') + const path = await writeCodexRollout(codex, '04', 'grow', codexRollout('grow', '/tmp/cxg', [1, 2, 3].map(n => ({ n, at: `2026-05-04T09:${n}0:00.000Z` })))) + const cache = join(home, 'cache-inc') + const args = ['status', '--format', 'menubar-json'] + + const cold = runCli(args, home, { CODEBURN_PARSE_WORKERS: '3', CODEBURN_CACHE_DIR: cache, CODEBURN_VERBOSE: '1' }) + expect(cold.status, cold.stderr).toBe(0) + expect(cold.stderr).toContain('codeburn: codex parse workers=3') + + await appendFile(path, codexTaskLines([{ n: 4, at: '2026-05-04T09:40:00.000Z' }]).join('\n') + '\n') + const warm = runCli(args, home, { CODEBURN_PARSE_WORKERS: '3', CODEBURN_CACHE_DIR: cache, CODEBURN_VERBOSE: '1' }) + expect(warm.status, warm.stderr).toBe(0) + expect(warm.stderr).toContain('codeburn: codex parse workers=0 (no full parses pending)') + }) +}) + +describe('ParseWorkerPool', () => { + let home: string + let files: string[] + let codexPath: string + let codexSource: SessionSource + + beforeEach(async () => { + clearSessionCache() + home = await mkdtemp(join(tmpdir(), 'cb-pool-')) + files = await writeCorpus(join(home, '.claude'), 2, 4) + codexPath = await writeCodexRollout( + join(home, '.codex'), '04', 'pool', + codexRollout('pool-1', '/tmp/cx', [1, 2].map(n => ({ n, at: `2026-05-04T09:${n}0:00.000Z` }))), + ) + codexSource = { provider: 'codex', path: codexPath, project: 'tmp-cx' } + process.env['CLAUDE_CONFIG_DIR'] = join(home, '.claude') + // Isolated so a parse in this process can never walk the developer's own + // ~/.codex, and so the pool the Codex path opens is covered by the leak check. + process.env['CODEX_HOME'] = join(home, '.codex') + process.env['CODEBURN_CACHE_DIR'] = join(home, '.cache', 'codeburn') + }) + + afterEach(async () => { + clearSessionCache() + delete process.env['CODEBURN_PARSE_WORKERS'] + delete process.env['CODEX_HOME'] + await rm(home, { recursive: true, force: true }) + }) + + function liveWorkers(): number { + return process.getActiveResourcesInfo().filter(r => r === 'Worker').length + } + + it('returns results in submission order and terminates every thread on close', async () => { + const before = liveWorkers() + const pool = new ParseWorkerPool(3) + const results = [] + for await (const r of parseFilesInOrder(pool, files.map(filePath => ({ kind: 'claude' as const, filePath })))) results.push(r) + await pool.close() + + expect(results).toHaveLength(files.length) + for (const r of results) expect(r.ok).toBe(true) + // Each fixture session's first turn names its own project, which pins the + // yielded order to the submitted order rather than to completion order. + const projects = results.map(r => (r.ok && r.parsed ? r.parsed.turns[0]?.userMessage : undefined)) + expect(projects).toEqual(files.map((_, i) => `task 0 in ${Math.floor(i / 4)}`)) + expect(liveWorkers()).toBe(before) + }) + + // A worker that cannot answer must hand the file back, never drop it: the + // caller's fallback is an in-process parse, and it has to land on the same + // result the worker would have produced. + it('reports failures instead of throwing, and the serial fallback matches', async () => { + const pool = new ParseWorkerPool(1) + const fromWorker = await pool.submit({ kind: 'claude', filePath: files[0]! }) + await pool.close() + + const afterClose = await pool.submit({ kind: 'claude', filePath: files[1]! }) + expect(afterClose.ok).toBe(false) + + const serial = await parseClaudeFileFull(files[0]!, new Set()) + expect(fromWorker.ok).toBe(true) + if (!fromWorker.ok || !fromWorker.parsed) throw new Error('expected a parsed result') + const { msgIds, path, ...worker } = fromWorker.parsed + expect(msgIds.length).toBeGreaterThan(0) + // Echoed back so the parent can assert the positional worker/file pairing. + expect(path).toBe(files[0]) + expect(worker).toEqual(JSON.parse(JSON.stringify(serial))) + }) + + + // Same contract for a Codex rollout: the off-thread decode is the serial + // decode, including the cache entry the parent has to install, and a worker + // that cannot answer hands the file back for an in-process parse. + it('decodes a codex rollout off-thread exactly as the serial path does', async () => { + const before = liveWorkers() + const pool = new ParseWorkerPool(1) + const fromWorker = await pool.submit({ kind: 'codex', source: codexSource }) + await pool.close() + expect(liveWorkers()).toBe(before) + + const afterClose = await pool.submit({ kind: 'codex', source: codexSource }) + expect(afterClose.ok).toBe(false) + + const seen = new Set() + const serial = await parseCodexFileFull(codexSource, seen) + if (!fromWorker.ok || !fromWorker.parsed) throw new Error('expected a parsed result') + const { keys, path, ...worker } = fromWorker.parsed + expect(keys.length).toBeGreaterThan(0) + expect(new Set(keys)).toEqual(seen) + // Echoed back so the parent can assert the positional worker/file pairing. + expect(path).toBe(codexPath) + expect(worker).toEqual(JSON.parse(JSON.stringify(serial))) + // The decode itself must never have touched the codex cache file. + expect(await codexResults(join(home, '.cache', 'codeburn'))).toBeNull() + }) + + // The resident `serve` child parses over and over in one process; a thread + // that outlives its parse would accumulate across requests. + it('leaves no live worker behind after back-to-back parses', async () => { + const before = liveWorkers() + process.env['CODEBURN_PARSE_WORKERS'] = '2' + + await parseAllSessions() + expect(liveWorkers()).toBe(before) + + clearSessionCache() + await parseAllSessions() + expect(liveWorkers()).toBe(before) + }) +}) diff --git a/tests/parser-antigravity-timestamp.test.ts b/tests/parser-antigravity-timestamp.test.ts index dbe11550..e394af5e 100644 --- a/tests/parser-antigravity-timestamp.test.ts +++ b/tests/parser-antigravity-timestamp.test.ts @@ -7,7 +7,7 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest' import { getDateRange } from '../src/cli-date.js' import { clearSessionCache, parseAllSessions } from '../src/parser.js' -import { sessionCachePath } from '../src/session-cache.js' +import { readCacheOnDisk } from './fixtures/session-cache-io.js' import { isSqliteAvailable } from '../src/sqlite.js' import type { DateRange } from '../src/types.js' @@ -40,9 +40,7 @@ function createGenMetadataDb(dbPath: string, fixture: Fixture): void { } async function cachedAntigravityTurns(cacheDir: string, dbPath: string): Promise> { - const saved = JSON.parse(await readFile(sessionCachePath(), 'utf-8')) as { - providers: Record }> }> - } + const saved = await readCacheOnDisk() return saved.providers['antigravity']?.files[dbPath]?.turns ?? [] } diff --git a/tests/parser-cache-refresh-timeout.test.ts b/tests/parser-cache-refresh-timeout.test.ts index 8883c5a7..53e53e0e 100644 --- a/tests/parser-cache-refresh-timeout.test.ts +++ b/tests/parser-cache-refresh-timeout.test.ts @@ -8,7 +8,7 @@ vi.mock('../src/cache-refresh-lock.js', () => ({ })) import { clearSessionCache, isSessionHydrationComplete, parseAllSessions } from '../src/parser.js' -import { sessionCachePath } from '../src/session-cache.js' +import { cacheDirSnapshot } from './fixtures/session-cache-io.js' let root: string let sessionPath: string @@ -52,12 +52,12 @@ describe('parseAllSessions warm refresh timeout', () => { it('serves the prior complete snapshot and leaves the holder cache untouched', async () => { await writeSession(50) expect(output(await parseAllSessions(undefined, 'claude'))).toBe(50) - const before = await readFile(sessionCachePath(), 'utf-8') + const before = await cacheDirSnapshot() await writeSession(5000) clearSessionCache() expect(output(await parseAllSessions(undefined, 'claude'))).toBe(50) - expect(await readFile(sessionCachePath(), 'utf-8')).toBe(before) + expect(await cacheDirSnapshot()).toBe(before) }) // The snapshot a timed-out refresh serves is only as good as what has changed diff --git a/tests/parser-classify-after-slice.test.ts b/tests/parser-classify-after-slice.test.ts new file mode 100644 index 00000000..be48e0e7 --- /dev/null +++ b/tests/parser-classify-after-slice.test.ts @@ -0,0 +1,111 @@ +import { afterEach, beforeEach, expect, it } from 'vitest' +import { mkdtemp, mkdir, writeFile, rm } from 'fs/promises' +import { join } from 'path' +import { tmpdir } from 'os' + +import { parseAllSessions, filterProjectsByDateRange, clearSessionCache } from '../src/parser.js' +import { loadPricing } from '../src/models.js' +import type { ClassifiedTurn, DateRange } from '../src/types.js' + +// scanProjectDirs decides the date slice on the RAW cached turn and classifies +// only survivors. The classification itself must still see each surviving +// turn's COMPLETE call list, and the branch/PR carries must still run over the +// full ordered turn list — so this fixture puts the branch anchor and the PR +// reference before the range, and straddles the range start with a turn whose +// only Edit lands on the out-of-range side. + +const SESSION = '22222222-2222-4222-8222-222222222222' +const CWD = '/tmp/slice-proj' +const BRANCH = 'feat/carry' +const PR = 'https://github.com/o/r/pull/42' +const RANGE: DateRange = { + start: new Date('2026-07-20T00:00:00.000Z'), + end: new Date('2026-07-20T23:59:59.999Z'), +} + +let tmpDir: string + +beforeEach(async () => { + clearSessionCache() + tmpDir = await mkdtemp(join(tmpdir(), 'slice-')) + process.env['CLAUDE_CONFIG_DIR'] = join(tmpDir, 'claude') + process.env['CODEBURN_CACHE_DIR'] = join(tmpDir, 'cache') +}) + +afterEach(async () => { + clearSessionCache() + delete process.env['CLAUDE_CONFIG_DIR'] + delete process.env['CODEBURN_CACHE_DIR'] + await rm(tmpDir, { recursive: true, force: true }) +}) + +function user(ts: string, content: string): string { + return JSON.stringify({ type: 'user', sessionId: SESSION, timestamp: ts, cwd: CWD, gitBranch: BRANCH, message: { role: 'user', content } }) +} + +function assistant(ts: string, id: string, tools: string[]): string { + return JSON.stringify({ + type: 'assistant', sessionId: SESSION, timestamp: ts, cwd: CWD, gitBranch: BRANCH, + message: { + id, type: 'message', role: 'assistant', model: 'claude-sonnet-4-5', + content: tools.map((name, i) => ({ type: 'tool_use', id: `${id}_${i}`, name, input: {} })), + usage: { input_tokens: 100, output_tokens: 50 }, + }, + }) +} + +async function writeTranscript(): Promise { + const projDir = join(tmpDir, 'claude', 'projects', 'slice-proj') + await mkdir(projDir, { recursive: true }) + await writeFile(join(projDir, `${SESSION}.jsonl`), [ + // Before the range: the only turn carrying the branch (the cache elides an + // unchanged branch on later turns) and the only PR reference. + user('2026-07-19T09:00:00.000Z', `please finish ${PR}`), + assistant('2026-07-19T09:00:05.000Z', 'm1', ['Read']), + // Straddles the range start: the Edit is on the out-of-range call. + user('2026-07-19T23:50:00.000Z', 'keep going overnight'), + assistant('2026-07-19T23:50:10.000Z', 'm2', ['Edit']), + assistant('2026-07-20T00:10:00.000Z', 'm3', ['Read']), + // Fully inside the range. + user('2026-07-20T10:00:00.000Z', 'what changed?'), + assistant('2026-07-20T10:00:05.000Z', 'm4', ['Read']), + ].join('\n') + '\n', 'utf-8') +} + +function shape(turn: ClassifiedTurn): unknown { + return { + timestamp: turn.timestamp, + category: turn.category, + subCategory: turn.subCategory, + retries: turn.retries, + hasEdits: turn.hasEdits, + gitBranch: turn.gitBranch, + prRefs: turn.prRefs, + calls: turn.assistantCalls.map(c => c.timestamp), + } +} + +it('slices before classifying without changing carried branch, PR, or turn classification', async () => { + await loadPricing() + await writeTranscript() + + const sliced = await parseAllSessions(RANGE, 'claude') + // Reference: the old order — classify every turn from the full history, then + // apply the same range slice afterwards. + clearSessionCache() + const reference = filterProjectsByDateRange(await parseAllSessions(undefined, 'claude'), RANGE) + + const session = sliced[0]!.sessions[0]! + expect(session.turns.map(shape)).toEqual(reference[0]!.sessions[0]!.turns.map(shape)) + + // The branch anchor and the PR reference both live before the range. + expect(session.everHadBranch).toBe(true) + expect(session.turns.every(t => t.gitBranch === BRANCH)).toBe(true) + expect(session.prRefsAtRangeStart).toEqual([PR]) + + // The straddling turn kept only its in-range call, but was classified from + // the complete call list — the Edit it dropped still counts. + const straddled = session.turns[0]! + expect(straddled.assistantCalls.map(c => c.timestamp)).toEqual(['2026-07-20T00:10:00.000Z']) + expect(straddled.hasEdits).toBe(true) +}) diff --git a/tests/parser-gemini-cache.test.ts b/tests/parser-gemini-cache.test.ts index b5510274..d2f8c7e1 100644 --- a/tests/parser-gemini-cache.test.ts +++ b/tests/parser-gemini-cache.test.ts @@ -5,7 +5,8 @@ import { join } from 'path' import { afterEach, beforeEach, describe, expect, it } from 'vitest' import { clearSessionCache, parseAllSessions } from '../src/parser.js' -import { CACHE_VERSION, computeEnvFingerprint, sessionCachePath } from '../src/session-cache.js' +import { CACHE_VERSION, computeEnvFingerprint, type SessionCache } from '../src/session-cache.js' +import { readCacheOnDisk, writeCacheOnDisk } from './fixtures/session-cache-io.js' import type { DateRange } from '../src/types.js' let home: string @@ -54,7 +55,7 @@ describe('Gemini session cache migration', () => { })) const fileStat = await stat(sessionPath) - await writeFile(sessionCachePath(), JSON.stringify({ + await writeCacheOnDisk({ version: CACHE_VERSION, providers: { gemini: { @@ -97,7 +98,7 @@ describe('Gemini session cache migration', () => { }, }, }, - })) + } as SessionCache) const range: DateRange = { start: new Date('2026-05-16T00:00:00.000Z'), @@ -117,7 +118,7 @@ describe('Gemini session cache migration', () => { 'gemini:gemini-session-1:g2', ]) - const savedCache = JSON.parse(await readFile(sessionCachePath(), 'utf-8')) + const savedCache = await readCacheOnDisk() as any const savedKeys = savedCache.providers.gemini.files[sessionPath].turns.flatMap((turn: { calls: Array<{ deduplicationKey: string }> }) => turn.calls.map(call => call.deduplicationKey), ) diff --git a/tests/parser-hydration-lock.test.ts b/tests/parser-hydration-lock.test.ts index 79e14983..5a41adb8 100644 --- a/tests/parser-hydration-lock.test.ts +++ b/tests/parser-hydration-lock.test.ts @@ -6,12 +6,13 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest' import { existsSync } from 'fs' -import { mkdir, mkdtemp, rm, unlink, writeFile, readFile } from 'fs/promises' +import { mkdir, mkdtemp, rm, unlink, writeFile } from 'fs/promises' import { tmpdir } from 'os' import { join } from 'path' import { clearSessionCache, parseAllSessions } from '../src/parser.js' -import { sessionCachePath } from '../src/session-cache.js' +import { sessionCacheDir } from '../src/session-cache.js' +import { readCacheOnDisk, writeCacheOnDisk } from './fixtures/session-cache-io.js' let tmpHome: string let cacheDir: string @@ -71,17 +72,16 @@ describe('parseAllSessions hydration lock', () => { await writeClaudeSession(50) expect(totalOutput(await parseAllSessions(undefined, 'claude'))).toBe(50) - const warm = JSON.parse(await readFile(sessionCachePath(), 'utf-8')) - for (const section of Object.values(warm.providers) as Array<{ files: Record }> }> }>) { + const tampered = await readCacheOnDisk() + for (const section of Object.values(tampered.providers)) { for (const file of Object.values(section.files)) { for (const turn of file.turns) for (const call of turn.calls) call.usage.outputTokens = 999 } } - const tampered = JSON.stringify(warm) // Go cold: remove the versioned cache and drop the in-memory cache so the // next parse genuinely cold-starts and consults the lock. - await unlink(sessionCachePath()) + await rm(sessionCacheDir(), { recursive: true }) clearSessionCache() // A fresh lock held by another live process (pid 1 is always alive and is @@ -97,7 +97,7 @@ describe('parseAllSessions hydration lock', () => { // The "first process" finishes: it leaves the warm (tampered) cache behind // and releases the lock. The waiter wakes, reloads, and serves the cache. - await writeFile(sessionCachePath(), tampered) + await writeCacheOnDisk(tampered) await unlink(lockPath()) const result = await promise @@ -119,8 +119,9 @@ describe('parseAllSessions hydration lock', () => { expect(totalOutput(result)).toBe(50) // Lock released in the finally. expect(existsSync(lockPath())).toBe(false) - // The parse warmed the versioned cache. - expect(existsSync(sessionCachePath())).toBe(true) + // The parse warmed the versioned cache: the envelope is what publishes it, + // so the directory merely existing proves nothing. + expect(existsSync(join(sessionCacheDir(), 'envelope.json'))).toBe(true) }) it('ignores a fresh lock whose pid is dead', async () => { diff --git a/tests/parser-incremental-append.test.ts b/tests/parser-incremental-append.test.ts index 3c192146..364ac663 100644 --- a/tests/parser-incremental-append.test.ts +++ b/tests/parser-incremental-append.test.ts @@ -20,7 +20,7 @@ vi.mock('../src/fs-utils.js', async (importOriginal) => { }) import { parseAllSessions, clearSessionCache } from '../src/parser.js' -import { sessionCachePath } from '../src/session-cache.js' +import { readCacheOnDisk, writeCacheOnDisk } from './fixtures/session-cache-io.js' import type { ProjectSummary } from '../src/types.js' let tmpDir: string @@ -129,8 +129,8 @@ describe('incremental append parsing', () => { await writeFile(sessionPath, baseLines().join('\n') + '\n') await parseWith(warmCache) - const cachedOffset: number = JSON.parse(await readFile(sessionCachePath(), 'utf-8')) - .providers.claude.files[sessionPath].lastCompleteLineOffset + const cachedOffset = (await readCacheOnDisk()) + .providers['claude']!.files[sessionPath]!.lastCompleteLineOffset! expect(cachedOffset).toBeGreaterThan(0) // 2) append a new complete turn plus a torn (invalid JSON, no newline) tail. @@ -317,10 +317,9 @@ describe('incremental append parsing', () => { await parseWith(warmCache) // Corrupt the persisted offset to point far beyond the file, then grow it. - const cachePath = sessionCachePath() - const cache = JSON.parse(await readFile(cachePath, 'utf-8')) - cache.providers.claude.files[sessionPath].lastCompleteLineOffset = 10_000_000 - await writeFile(cachePath, JSON.stringify(cache)) + const cache = await readCacheOnDisk() + cache.providers['claude']!.files[sessionPath]!.lastCompleteLineOffset = 10_000_000 + await writeCacheOnDisk(cache) await appendFile(sessionPath, userLine('2026-05-01T13:00:00.000Z', 'grow the file') + '\n' + diff --git a/tests/parser.test.ts b/tests/parser.test.ts index 4bd0c5c2..98c45376 100644 --- a/tests/parser.test.ts +++ b/tests/parser.test.ts @@ -14,7 +14,8 @@ import { createRequire } from 'node:module' import { isSqliteAvailable } from '../src/sqlite.js' import { clearSessionCache, parseAllSessions, setParseReuseValidator } from '../src/parser.js' -import { loadCache, saveCache, sessionCachePath } from '../src/session-cache.js' +import { loadCache, saveCache } from '../src/session-cache.js' +import { readCacheOnDisk, writeCacheOnDisk } from './fixtures/session-cache-io.js' import type { SessionSource, SessionParser, ParsedProviderCall } from '../src/providers/types.js' // ── Synthetic provider state ─────────────────────────────────────────────── @@ -23,6 +24,7 @@ import type { SessionSource, SessionParser, ParsedProviderCall } from '../src/pr let _synthSources: SessionSource[] = [] let _synthDurable = false let _synthYields: ParsedProviderCall[] = [] +let _synthOnParse: (() => void | Promise) | null = null vi.mock('../src/providers/index.js', async (importOriginal) => { type Mod = typeof import('../src/providers/index.js') @@ -52,6 +54,7 @@ vi.mock('../src/providers/index.js', async (importOriginal) => { createSessionParser(_s: SessionSource, _k: Set): SessionParser { return { async *parse(): AsyncGenerator { + await _synthOnParse?.() for (const call of _synthYields) { // Respect seenKeys so that when multiple sources share the same // dedup key, only the first source yields it (mirrors real parsers). @@ -190,13 +193,16 @@ beforeEach(async () => { _synthSources = [] _synthDurable = false _synthYields = [] + _synthOnParse = null }) afterEach(async () => { clearSessionCache() + setParseReuseValidator(null) vi.unstubAllEnvs() _synthSources = [] + _synthOnParse = null await rm(tmpHome, { recursive: true, force: true }) await rm(tmpCache, { recursive: true, force: true }) @@ -444,12 +450,10 @@ describe('(f) durable orphans survive a parse-version bump', () => { // Simulate the fingerprint a PREVIOUS release computed (any mismatching // value takes the same code path as a real parse-version bump). - const { readFile, writeFile: writeFileFs } = await import('fs/promises') - const cachePath = sessionCachePath() - const disk = JSON.parse(await readFile(cachePath, 'utf-8')) as { providers: Record } + const disk = await readCacheOnDisk() expect(disk.providers['copilot']).toBeDefined() disk.providers['copilot']!.envFingerprint = '0000000000000000' - await writeFileFs(cachePath, JSON.stringify(disk), 'utf-8') + await writeCacheOnDisk(disk) // First parse after the "upgrade": the orphan must still be counted and // must survive in the rewritten cache, not be erased with the section. @@ -728,6 +732,75 @@ describe('(q) parse burst reuse (CODEBURN_PARSE_BURST_MS)', () => { }) describe('(r) validated parse reuse (setParseReuseValidator)', () => { + it('falls back to the exact TTL when watcher coverage is unknown, but rejects dirty', async () => { + vi.stubEnv('CODEBURN_PARSE_BURST_MS', '0') + clearSessionCache() + const start = new Date(Date.now() - 60 * 60 * 1000) + const end = new Date() + const ts = new Date(Date.now() - 10 * 60 * 1000).toISOString() + const synthFile = join(tmpHome, 'synth-unknown-exact.txt') + await writeFile(synthFile, 'first input') + _synthSources = [{ path: synthFile, project: 'p', provider: 'test-synthetic' }] + _synthYields = [{ + provider: 'test-synthetic', model: 'synth-model', + inputTokens: 1, outputTokens: 5, cacheCreationInputTokens: 0, cacheReadInputTokens: 0, + cachedInputTokens: 0, reasoningTokens: 0, webSearchRequests: 0, + costUSD: 0, costIsEstimated: false, tools: [], bashCommands: [], skills: [], + timestamp: ts, speed: 'standard', deduplicationKey: 'synth-unknown-exact-1', userMessage: 'hi', sessionId: 'sue-1', + }] as never + + expect(totalOutput(await parseAllSessions({ start, end }, 'test-synthetic'))).toBe(5) + _synthYields = [..._synthYields, { + ...( _synthYields[0] as object ), deduplicationKey: 'synth-unknown-exact-2', outputTokens: 7, + }] as never + await writeFile(synthFile, 'second input with changed fingerprint') + + // An unhealthy/pre-arm watcher cannot extend freshness, but it must retain + // the normal exact-key TTL instead of forcing a full rescan every request. + setParseReuseValidator(() => 'unknown') + expect(totalOutput(await parseAllSessions({ start, end }, 'test-synthetic'))).toBe(5) + + // The same entry must be rejected immediately once a real change is known. + setParseReuseValidator(() => 'dirty') + expect(totalOutput(await parseAllSessions({ start, end }, 'test-synthetic'))).toBe(12) + }) + + it('falls back to the short burst when watcher coverage is unknown, but dirty wins inside it', async () => { + vi.stubEnv('CODEBURN_PARSE_BURST_MS', '10000') + clearSessionCache() + const start = new Date(Date.now() - 60 * 60 * 1000) + const firstEnd = new Date() + const ts = new Date(Date.now() - 10 * 60 * 1000).toISOString() + const synthFile = join(tmpHome, 'synth-unknown-burst.txt') + await writeFile(synthFile, 'first input') + _synthSources = [{ path: synthFile, project: 'p', provider: 'test-synthetic' }] + _synthYields = [{ + provider: 'test-synthetic', model: 'synth-model', + inputTokens: 1, outputTokens: 5, cacheCreationInputTokens: 0, cacheReadInputTokens: 0, + cachedInputTokens: 0, reasoningTokens: 0, webSearchRequests: 0, + costUSD: 0, costIsEstimated: false, tools: [], bashCommands: [], skills: [], + timestamp: ts, speed: 'standard', deduplicationKey: 'synth-unknown-burst-1', userMessage: 'hi', sessionId: 'sub-1', + }] as never + + expect(totalOutput(await parseAllSessions({ start, end: firstEnd }, 'test-synthetic'))).toBe(5) + _synthYields = [..._synthYields, { + ...( _synthYields[0] as object ), deduplicationKey: 'synth-unknown-burst-2', outputTokens: 7, + }] as never + await writeFile(synthFile, 'second input with changed fingerprint') + + setParseReuseValidator(() => 'unknown') + expect(totalOutput(await parseAllSessions( + { start, end: new Date(firstEnd.getTime() + 100) }, + 'test-synthetic', + ))).toBe(5) + + setParseReuseValidator(() => 'dirty') + expect(totalOutput(await parseAllSessions( + { start, end: new Date(firstEnd.getTime() + 200) }, + 'test-synthetic', + ))).toBe(12) + }) + it('reuses past the burst window while the validator reports quiet, never when dirty', async () => { vi.stubEnv('CODEBURN_PARSE_BURST_MS', '1') clearSessionCache() @@ -750,14 +823,14 @@ describe('(r) validated parse reuse (setParseReuseValidator)', () => { // 1ms burst window has certainly elapsed; with a quiet validator the // previous parse is still served (world changed, result must not). await new Promise(r => setTimeout(r, 5)) - setParseReuseValidator(() => true) + setParseReuseValidator(() => 'clean') _synthYields = [..._synthYields, { ...( _synthYields[0] as object ), deduplicationKey: 'synth-val-2', outputTokens: 7 }] as never await writeFile(synthFile, 'placeholder v2') const second = await parseAllSessions({ start, end: new Date(Date.now() + 500) }, 'test-synthetic') expect(totalOutput(second)).toBe(5) // A dirty validator ends the reuse: fresh parse sees the new call. - setParseReuseValidator(() => false) + setParseReuseValidator(() => 'dirty') const third = await parseAllSessions({ start, end: new Date(Date.now() + 1000) }, 'test-synthetic') expect(totalOutput(third)).toBe(12) @@ -766,4 +839,85 @@ describe('(r) validated parse reuse (setParseReuseValidator)', () => { _synthSources = [] _synthYields = [] }) + + it('rejects an exact-key memo when a root event arrived during its parse', async () => { + clearSessionCache() + const start = new Date(Date.now() - 60 * 60 * 1000) + const end = new Date() + const ts = new Date(Date.now() - 10 * 60 * 1000).toISOString() + const synthFile = join(tmpHome, 'synth-exact-event-during-parse.txt') + await writeFile(synthFile, 'first input') + _synthSources = [{ path: synthFile, project: 'p', provider: 'test-synthetic' }] + _synthYields = [{ + provider: 'test-synthetic', model: 'synth-model', + inputTokens: 1, outputTokens: 5, cacheCreationInputTokens: 0, cacheReadInputTokens: 0, + cachedInputTokens: 0, reasoningTokens: 0, webSearchRequests: 0, + costUSD: 0, costIsEstimated: false, tools: [], bashCommands: [], skills: [], + timestamp: ts, speed: 'standard', deduplicationKey: 'synth-exact-event-1', userMessage: 'hi', sessionId: 'see-1', + }] as never + + let rootEventAt = 0 + setParseReuseValidator(sinceTs => rootEventAt === 0 || rootEventAt < sinceTs ? 'clean' : 'dirty') + _synthOnParse = async () => { + await new Promise(resolve => setTimeout(resolve, 10)) + rootEventAt = Date.now() + await new Promise(resolve => setTimeout(resolve, 10)) + } + const first = await parseAllSessions({ start, end }, 'test-synthetic') + expect(totalOutput(first)).toBe(5) + _synthOnParse = null + + _synthYields = [..._synthYields, { + ...( _synthYields[0] as object ), deduplicationKey: 'synth-exact-event-2', outputTokens: 7, + }] as never + await writeFile(synthFile, 'second input') + const second = await parseAllSessions({ start, end }, 'test-synthetic') + expect(totalOutput(second)).toBe(12) + }) + + it('does not bless a root event that arrived while the cached parse was running', async () => { + vi.stubEnv('CODEBURN_PARSE_BURST_MS', '1') + clearSessionCache() + const start = new Date(Date.now() - 60 * 60 * 1000) + const firstEnd = new Date() + const ts = new Date(Date.now() - 10 * 60 * 1000).toISOString() + const synthFile = join(tmpHome, 'synth-event-during-parse.txt') + await writeFile(synthFile, 'first input') + _synthSources = [{ path: synthFile, project: 'p', provider: 'test-synthetic' }] + _synthYields = [{ + provider: 'test-synthetic', model: 'synth-model', + inputTokens: 1, outputTokens: 5, cacheCreationInputTokens: 0, cacheReadInputTokens: 0, + cachedInputTokens: 0, reasoningTokens: 0, webSearchRequests: 0, + costUSD: 0, costIsEstimated: false, tools: [], bashCommands: [], skills: [], + timestamp: ts, speed: 'standard', deduplicationKey: 'synth-event-1', userMessage: 'hi', sessionId: 'se-1', + }] as never + + let rootEventAt = 0 + _synthOnParse = async () => { + // Bracket the controlled event so it is strictly after parse start and + // strictly before completion, independent of same-millisecond clocks. + await new Promise(resolve => setTimeout(resolve, 10)) + rootEventAt = Date.now() + await new Promise(resolve => setTimeout(resolve, 10)) + } + const first = await parseAllSessions({ start, end: firstEnd }, 'test-synthetic') + expect(totalOutput(first)).toBe(5) + expect(rootEventAt).toBeGreaterThan(0) + _synthOnParse = null + + // Outside the 1ms burst, old code validated against cachePut completion + // and reused stale output because the in-parse event appeared older. The + // parse-start timestamp makes the validator reject reuse and rescan. + await new Promise(resolve => setTimeout(resolve, 5)) + setParseReuseValidator(sinceTs => rootEventAt < sinceTs ? 'clean' : 'dirty') + _synthYields = [..._synthYields, { + ...( _synthYields[0] as object ), deduplicationKey: 'synth-event-2', outputTokens: 7, + }] as never + await writeFile(synthFile, 'second input') + const second = await parseAllSessions( + { start, end: new Date(firstEnd.getTime() + 500) }, + 'test-synthetic', + ) + expect(totalOutput(second)).toBe(12) + }) }) diff --git a/tests/providers/claude-config-dirs.test.ts b/tests/providers/claude-config-dirs.test.ts index 571469b1..899fb335 100644 --- a/tests/providers/claude-config-dirs.test.ts +++ b/tests/providers/claude-config-dirs.test.ts @@ -390,6 +390,37 @@ describe('claude provider — config.json claudeConfigDirs (menubar-driven)', () expect(paths).toContain(join(personal, 'projects', '-Users-you-app')) }) + it('invalidates the exact parse memo when config.json adds a Claude discovery root', async () => { + const work = await makeConfigDir('claude-work', []) + const personal = await makeConfigDir('claude-personal', []) + const slug = '-Users-you-shared-app' + const cwd = '/Users/you/shared-app' + await writeSession(work, slug, 'sess-work', [ + summaryLine('sess-work', cwd), + userLine('u1', 'sess-work', cwd, 'hi from work'), + assistantLine('a1', 'u1', 'sess-work', cwd), + ]) + await writeSession(personal, slug, 'sess-personal', [ + summaryLine('sess-personal', cwd), + userLine('u2', 'sess-personal', cwd, 'hi from personal'), + assistantLine('a2', 'u2', 'sess-personal', cwd), + ]) + + await writeConfigJson([work]) + const first = await parseAllSessions(undefined, 'claude') + expect(first.flatMap(project => project.sessions).map(session => session.sessionId)).toEqual(['sess-work']) + + // Same argv/date range and unchanged env: only the effective roots sourced + // from config.json differ. A resident process must not return the exact-key + // memo populated by the first call. + await writeConfigJson([work, personal]) + const second = await parseAllSessions(undefined, 'claude') + expect(second.flatMap(project => project.sessions).map(session => session.sessionId).sort()).toEqual([ + 'sess-personal', + 'sess-work', + ]) + }) + it('lets env CLAUDE_CONFIG_DIRS override config.json', async () => { const fromEnv = await makeConfigDir('claude-env', ['-Users-you-app']) const fromFile = await makeConfigDir('claude-file', ['-Users-you-app']) diff --git a/tests/providers/codewhale.test.ts b/tests/providers/codewhale.test.ts index bdf9fd3a..0b6a29da 100644 --- a/tests/providers/codewhale.test.ts +++ b/tests/providers/codewhale.test.ts @@ -4,7 +4,7 @@ import { tmpdir } from 'os' import { join } from 'path' import { clearSessionCache, parseAllSessions } from '../../src/parser.js' -import { sessionCachePath } from '../../src/session-cache.js' +import { readCacheOnDisk } from '../fixtures/session-cache-io.js' import { MAX_SESSION_FILE_BYTES } from '../../src/fs-utils.js' import { codewhale, createCodeWhaleProvider } from '../../src/providers/codewhale.js' import type { ParsedProviderCall } from '../../src/providers/types.js' @@ -275,10 +275,8 @@ describe('codewhale provider', () => { expect(first[0]!.totalCostUSD).toBeCloseTo(0.75) expect(second[0]!.totalCostUSD).toBeCloseTo(0.75) - const cache = JSON.parse(await readFile(sessionCachePath(), 'utf-8')) as { - providers: { codewhale: { envFingerprint: string } } - } - expect(cache.providers.codewhale.envFingerprint).toMatch(/^[a-f0-9]{16}$/) + const cache = await readCacheOnDisk() + expect(cache.providers['codewhale']!.envFingerprint).toMatch(/^[a-f0-9]{16}$/) }) it('exposes canonical model and tool display names', () => { diff --git a/tests/providers/codex-resume.test.ts b/tests/providers/codex-resume.test.ts new file mode 100644 index 00000000..e63e41f8 --- /dev/null +++ b/tests/providers/codex-resume.test.ts @@ -0,0 +1,287 @@ +// Codex rollouts are append-only and the active ones are huge, so a run that +// re-read a grown session from byte 0 paid for the whole file to pick up a few +// KB. The parser now restarts from the last task boundary it recorded. What has +// to hold: the resumed decode is byte-identical to a full re-parse, and it +// really does start at an offset rather than quietly re-reading everything. +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' +import { appendFile, mkdtemp, mkdir, rm, writeFile } from 'fs/promises' +import { join } from 'path' +import { tmpdir } from 'os' + +const readLineCalls: Array<{ filePath: string; startByteOffset?: number }> = [] +vi.mock('../../src/fs-utils.js', async (importOriginal) => { + const actual = await importOriginal() + return { + ...actual, + readSessionLines: (filePath: string, skip?: unknown, options?: { startByteOffset?: number }) => { + readLineCalls.push({ filePath, startByteOffset: options?.startByteOffset }) + // eslint-disable-next-line @typescript-eslint/no-explicit-any + return (actual.readSessionLines as any)(filePath, skip, options) + }, + } +}) + +import { clearCodexMemCaches, flushCodexCache, withCodexCacheDirectory } from '../../src/codex-cache.js' +import { createCodexProvider } from '../../src/providers/codex.js' +import type { ParsedProviderCall } from '../../src/providers/types.js' + +let tmpDir: string +let sessionPath: string + +beforeEach(async () => { + tmpDir = await mkdtemp(join(tmpdir(), 'codex-resume-')) + readLineCalls.length = 0 +}) + +afterEach(async () => { + await rm(tmpDir, { recursive: true, force: true }) +}) + +function meta(): string { + return JSON.stringify({ + type: 'session_meta', + timestamp: '2026-04-14T10:00:00Z', + payload: { cwd: '/Users/test/proj', originator: 'codex-cli', session_id: 'sess-1', model: 'gpt-5.3-codex' }, + }) +} + +// One complete task: user turn, tools, an edit, an MCP call, usage, completion. +function task(n: number, cumulative: { input: number; cached: number; output: number; reasoning: number }): string[] { + const at = (s: number) => `2026-04-14T10:${String(n).padStart(2, '0')}:${String(s).padStart(2, '0')}Z` + return [ + JSON.stringify({ type: 'event_msg', timestamp: at(0), payload: { type: 'task_started' } }), + JSON.stringify({ + type: 'response_item', timestamp: at(1), + payload: { type: 'message', role: 'user', content: [{ type: 'input_text', text: `task ${n}` }] }, + }), + JSON.stringify({ + type: 'response_item', timestamp: at(2), + payload: { type: 'function_call', name: 'shell', call_id: `c${n}`, arguments: JSON.stringify({ command: `ls ${n}` }) }, + }), + JSON.stringify({ + type: 'response_item', timestamp: at(3), + payload: { type: 'function_call_output', call_id: `c${n}` }, + }), + JSON.stringify({ + type: 'event_msg', timestamp: at(4), + payload: { + type: 'patch_apply_end', success: n % 2 === 0, + changes: { [`/Users/test/proj/f${n}.ts`]: { unified_diff: '@@ -1 +1,2 @@\n-old\n+new\n+extra\n' } }, + }, + }), + JSON.stringify({ + type: 'event_msg', timestamp: at(5), + payload: { type: 'mcp_tool_call_end', call_id: `m${n}`, invocation: { server: 'github', tool: 'list' }, duration_ms: 120, result: { Ok: {} } }, + }), + JSON.stringify({ + type: 'response_item', timestamp: at(6), + payload: { type: 'message', role: 'assistant', content: [{ type: 'output_text', text: 'x'.repeat(40) }] }, + }), + JSON.stringify({ + type: 'event_msg', timestamp: at(7), + payload: { + type: 'token_count', + info: { + last_token_usage: { input_tokens: 100, cached_input_tokens: 20, output_tokens: 50, reasoning_output_tokens: 10, total_tokens: 180 }, + total_token_usage: { + input_tokens: cumulative.input, cached_input_tokens: cumulative.cached, + output_tokens: cumulative.output, reasoning_output_tokens: cumulative.reasoning, + total_tokens: cumulative.input + cumulative.output + cumulative.reasoning, + }, + }, + }, + }), + JSON.stringify({ type: 'event_msg', timestamp: at(8), payload: { type: 'task_complete', duration_ms: 5000 } }), + ] +} + +function tasks(from: number, to: number): string[] { + const lines: string[] = [] + for (let n = from; n <= to; n++) { + lines.push(...task(n, { input: 100 * n, cached: 20 * n, output: 50 * n, reasoning: 10 * n })) + } + return lines +} + +async function writeRollout(lines: string[]): Promise { + const dir = join(tmpDir, 'sessions', '2026', '04', '14') + await mkdir(dir, { recursive: true }) + const path = join(dir, 'rollout-sess-1.jsonl') + await writeFile(path, lines.join('\n') + '\n') + return path +} + +async function parse(cacheDir: string, codexDir = tmpDir): Promise { + clearCodexMemCaches() + return withCodexCacheDirectory(cacheDir, async () => { + const provider = createCodexProvider(codexDir) + const sources = await provider.discoverSessions() + const seenKeys = new Set() + const calls: ParsedProviderCall[] = [] + for (const source of sources) { + for await (const call of provider.createSessionParser!(source, seenKeys).parse()) calls.push(call) + } + await flushCodexCache() + clearCodexMemCaches() + return calls + }) +} + +describe('codex incremental resume', () => { + it('resumes at a task boundary and matches a full re-parse exactly', async () => { + const warmCache = join(tmpDir, 'cache-warm') + const coldCache = join(tmpDir, 'cache-cold') + + sessionPath = await writeRollout([meta(), ...tasks(1, 3)]) + const first = await parse(warmCache) + expect(first.length).toBe(3) + + await appendFile(sessionPath, tasks(4, 6).join('\n') + '\n') + + readLineCalls.length = 0 + const resumed = await parse(warmCache) + const resumeReads = readLineCalls.filter(c => c.filePath === sessionPath) + // The parse re-entered the file at a boundary rather than at byte 0. + expect(resumeReads.some(c => (c.startByteOffset ?? 0) > 0)).toBe(true) + expect(resumeReads.every(c => (c.startByteOffset ?? 0) > 0)).toBe(true) + + // Byte-for-byte agreement with a decode that never saw a cache. + const full = await parse(coldCache) + expect(resumed.length).toBe(6) + expect(JSON.stringify(resumed)).toBe(JSON.stringify(full)) + }) + + it('stays exact across successive appends, resuming from a resumed state', async () => { + const warmCache = join(tmpDir, 'cache-warm') + + sessionPath = await writeRollout([meta(), ...tasks(1, 2)]) + await parse(warmCache) + await appendFile(sessionPath, tasks(3, 4).join('\n') + '\n') + await parse(warmCache) + // A tail with no task boundary at all: the next run restarts from the same + // boundary and re-decodes the open task. + await appendFile(sessionPath, tasks(5, 5).slice(1).join('\n') + '\n') + const resumed = await parse(warmCache) + + const full = await parse(join(tmpDir, 'cache-cold')) + expect(JSON.stringify(resumed)).toBe(JSON.stringify(full)) + }) + + it('serves an unchanged file from the cache without reading it', async () => { + const cacheDir = join(tmpDir, 'cache') + sessionPath = await writeRollout([meta(), ...tasks(1, 2)]) + const first = await parse(cacheDir) + + readLineCalls.length = 0 + const second = await parse(cacheDir) + expect(readLineCalls.filter(c => c.filePath === sessionPath)).toHaveLength(0) + expect(JSON.stringify(second)).toBe(JSON.stringify(first)) + }) + + it('falls back to a full re-parse when the stored resume state is unusable', async () => { + const cacheDir = join(tmpDir, 'cache') + sessionPath = await writeRollout([meta(), ...tasks(1, 2)]) + await parse(cacheDir) + + const cachePath = join(cacheDir, 'codex-results.json') + const { readFile } = await import('fs/promises') + const raw = JSON.parse(await readFile(cachePath, 'utf-8')) + raw.files[sessionPath].resumeState = { garbage: true } + await writeFile(cachePath, JSON.stringify(raw)) + const { clearCodexMemCaches } = await import('../../src/codex-cache.js') + clearCodexMemCaches() + + await appendFile(sessionPath, tasks(3, 3).join('\n') + '\n') + readLineCalls.length = 0 + const resumed = await parse(cacheDir) + expect(readLineCalls.filter(c => c.filePath === sessionPath).every(c => (c.startByteOffset ?? 0) === 0)).toBe(true) + + const full = await parse(join(tmpDir, 'cache-cold')) + expect(JSON.stringify(resumed)).toBe(JSON.stringify(full)) + }) +}) + +// The resume snapshot has to carry EVERY field the decode reads from an earlier +// line. Missing one shows up only when the split lands between the line that +// sets it and the line that reads it — so split at every line boundary and +// require the resumed decode to equal a full one each time. +const J = (o: unknown) => JSON.stringify(o) +const ts = (n: number, s: number) => `2026-04-14T${String(10 + Math.floor(n / 60)).padStart(2, '0')}:${String(n % 60).padStart(2, '0')}:${String(s).padStart(2, '0')}Z` + +function richTask(n: number, opts: { tokens?: false | 'empty'; reasoning?: number; model?: string; noComplete?: boolean } = {}): string[] { + const lines: string[] = [J({ type: 'event_msg', timestamp: ts(n, 0), payload: { type: 'task_started' } })] + if (opts.model) lines.push(J({ type: 'turn_context', timestamp: ts(n, 0), payload: { model: opts.model } })) + lines.push( + J({ type: 'response_item', timestamp: ts(n, 1), payload: { type: 'message', role: 'user', content: [{ type: 'input_text', text: `please do task ${n} with some length` }] } }), + J({ type: 'response_item', timestamp: ts(n, 2), payload: { type: 'function_call', name: 'shell', call_id: `c${n}`, arguments: J({ command: `ls ${n}` }) } }), + J({ type: 'response_item', timestamp: ts(n, 3), payload: { type: 'function_call_output', call_id: `c${n}` } }), + J({ type: 'event_msg', timestamp: ts(n, 4), payload: { type: 'patch_apply_end', success: n % 3 !== 0, changes: { [`/Users/test/proj/f${n}.ts`]: { unified_diff: '@@ -1 +1,2 @@\n-old\n+new\n+extra\n' } } } }), + J({ type: 'event_msg', timestamp: ts(n, 5), payload: { type: 'mcp_tool_call_end', call_id: `m${n}`, invocation: { server: 'github', tool: 'list' }, duration_ms: 120, result: { Ok: {} } } }), + J({ type: 'response_item', timestamp: ts(n, 6), payload: { type: 'message', role: 'assistant', content: [{ type: 'output_text', text: 'y'.repeat(120) }] } }), + ) + if (opts.tokens === 'empty') { + // No `info`: the estimated-usage path, which advances estCounter. + lines.push(J({ type: 'event_msg', timestamp: ts(n, 7), payload: { type: 'token_count' } })) + } else if (opts.tokens !== false) { + const c = { input: 100 * n, cached: 20 * n, output: 50 * n, reasoning: (opts.reasoning ?? 10) * n } + lines.push(J({ type: 'event_msg', timestamp: ts(n, 7), payload: { type: 'token_count', info: { + last_token_usage: { input_tokens: 100, cached_input_tokens: 20, output_tokens: 50, reasoning_output_tokens: opts.reasoning ?? 10, total_tokens: 180 }, + total_token_usage: { input_tokens: c.input, cached_input_tokens: c.cached, output_tokens: c.output, reasoning_output_tokens: c.reasoning, total_tokens: c.input + c.output + c.reasoning }, + } } })) + } + if (!opts.noComplete) lines.push(J({ type: 'event_msg', timestamp: ts(n, 8), payload: { type: 'task_complete', duration_ms: 5000 } })) + return lines +} + +const RICH_LINES = [ + J({ type: 'session_meta', timestamp: ts(0, 0), payload: { cwd: '/Users/test/proj', originator: 'codex-cli', session_id: 'sess-1', model: 'gpt-5.3-codex' } }), + ...richTask(1), + ...richTask(2, { tokens: 'empty' }), + ...richTask(7, { tokens: 'empty' }), + ...richTask(3, { model: 'gpt-5.3-codex-mini' }), + ...richTask(4, { reasoning: 33 }), + ...richTask(5, { noComplete: true }), + ...richTask(6), +] + +const FORK_LINES = [ + J({ type: 'session_meta', timestamp: ts(0, 0), payload: { cwd: '/Users/test/proj', originator: 'codex-cli', session_id: 'sess-2', forked_from_id: 'sess-1', model: 'gpt-5.3-codex' } }), + // Parent history replayed inside the 5s fork cutoff: must stay skipped across a split. + ...richTask(0).map(l => l.replace(/2026-04-14T10:00:0\d/g, '2026-04-14T10:00:01')), + ...richTask(11), + ...richTask(12, { tokens: 'empty' }), +] + +describe('codex resume differential', () => { + async function rollout(lines: string[]): Promise<{ codexDir: string; path: string }> { + const codexDir = await mkdtemp(join(tmpdir(), 'codex-split-')) + const dir = join(codexDir, 'sessions', '2026', '04', '14') + await mkdir(dir, { recursive: true }) + const path = join(dir, 'rollout-sess-1.jsonl') + await writeFile(path, lines.join('\n') + '\n') + return { codexDir, path } + } + + async function assertEverySplitMatches(lines: string[]): Promise { + const base = await rollout(lines) + const full = await parse(await mkdtemp(join(tmpdir(), 'codex-c-')), base.codexDir) + expect(full.length).toBeGreaterThan(0) + + for (let k = 1; k < lines.length; k++) { + const split = await rollout(lines.slice(0, k)) + const cacheDir = await mkdtemp(join(tmpdir(), 'codex-c-')) + await parse(cacheDir, split.codexDir) + await appendFile(split.path, lines.slice(k).join('\n') + '\n') + const resumed = await parse(cacheDir, split.codexDir) + expect(JSON.stringify(resumed), `split after line ${k}: ${lines[k - 1]!.slice(0, 80)}`).toBe(JSON.stringify(full)) + } + } + + it('matches a full re-parse at every line boundary of a rich session', async () => { + await assertEverySplitMatches(RICH_LINES) + }, 60_000) + + it('matches a full re-parse at every line boundary of a forked session', async () => { + await assertEverySplitMatches(FORK_LINES) + }, 60_000) +}) diff --git a/tests/providers/cursor.test.ts b/tests/providers/cursor.test.ts index 61151a6f..34c28d88 100644 --- a/tests/providers/cursor.test.ts +++ b/tests/providers/cursor.test.ts @@ -1,5 +1,5 @@ import { describe, it, expect, beforeEach, afterEach } from 'vitest' -import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises' +import { mkdtemp, mkdir, readFile, writeFile, rm } from 'node:fs/promises' import { join } from 'node:path' import { tmpdir } from 'node:os' import { createRequire } from 'node:module' @@ -87,6 +87,37 @@ describe('cursor cache', () => { const result = await readCachedResults('/nonexistent/path.db', new Date(0).toISOString()) expect(result).toBeNull() }) + + it('honors CODEBURN_CACHE_DIR at call time', async () => { + const root = await mkdtemp(join(tmpdir(), 'cursor-cache-override-')) + const previousCacheDir = process.env['CODEBURN_CACHE_DIR'] + const dbPath = join(root, 'state.vscdb') + const firstCacheDir = join(root, 'cache-a') + const secondCacheDir = join(root, 'cache-b') + const firstFloor = '2026-01-01T00:00:00.000Z' + const secondFloor = '2026-02-01T00:00:00.000Z' + await writeFile(dbPath, 'cursor-db-fixture') + + try { + const { writeCachedResults } = await import('../../src/cursor-cache.js') + process.env['CODEBURN_CACHE_DIR'] = firstCacheDir + await writeCachedResults(dbPath, [], firstFloor) + + process.env['CODEBURN_CACHE_DIR'] = secondCacheDir + await writeCachedResults(dbPath, [], secondFloor) + + const firstPath = join(firstCacheDir, 'cursor-results.json') + const secondPath = join(secondCacheDir, 'cursor-results.json') + const first = JSON.parse(await readFile(firstPath, 'utf-8')) as { lookbackFloor: string } + const second = JSON.parse(await readFile(secondPath, 'utf-8')) as { lookbackFloor: string } + expect(first.lookbackFloor).toBe(firstFloor) + expect(second.lookbackFloor).toBe(secondFloor) + } finally { + if (previousCacheDir === undefined) delete process.env['CODEBURN_CACHE_DIR'] + else process.env['CODEBURN_CACHE_DIR'] = previousCacheDir + await rm(root, { recursive: true, force: true }) + } + }) }) // Regression: Cursor renamed the per-workspace composer list key from diff --git a/tests/serve-stdio.test.ts b/tests/serve-stdio.test.ts index 2f0af775..66a26145 100644 --- a/tests/serve-stdio.test.ts +++ b/tests/serve-stdio.test.ts @@ -1,6 +1,31 @@ import { describe, it, expect, beforeAll, afterAll } from 'vitest' import { spawn, type ChildProcess } from 'child_process' +import { mkdir, readFile, writeFile } from 'fs/promises' import { join } from 'path' +import { classifyRootReuse, createOutputMemoEntry } from '../src/serve.js' + +it('timestamps a completed output memo before parsing begins', () => { + const parseStartedAt = 100 + const rootEventDuringParseAt = 150 + const parseCompletedAt = 200 + const memo = createOutputMemoEntry(parseStartedAt, parseCompletedAt, 'output', 'config') + const rootsQuietSince = (sinceTs: number): boolean => rootEventDuringParseAt < sinceTs + + // The old completion timestamp incorrectly made the in-parse event look + // older than the memo. The start timestamp keeps it visible to validation. + expect(rootsQuietSince(parseCompletedAt)).toBe(true) + expect(memo.createdAt).toBe(parseCompletedAt) + expect(memo.validatedFrom).toBe(parseStartedAt) + expect(rootsQuietSince(memo.validatedFrom)).toBe(false) +}) + +it('classifies watcher gaps as unknown without confusing them with dirty roots', () => { + expect(classifyRootReuse(100, { startedAt: 50, lastEventAt: 0, healthy: false })).toBe('unknown') + expect(classifyRootReuse(100, { startedAt: 150, lastEventAt: 0, healthy: true })).toBe('unknown') + expect(classifyRootReuse(100, { startedAt: 50, lastEventAt: 100, healthy: false })).toBe('dirty') + expect(classifyRootReuse(100, { startedAt: 50, lastEventAt: 100, healthy: true })).toBe('dirty') + expect(classifyRootReuse(100, { startedAt: 50, lastEventAt: 99, healthy: true })).toBe('clean') +}) // End-to-end protocol test for `codeburn serve --stdio` (the desktop app's // resident query server). Runs the real entry through tsx against the @@ -10,6 +35,8 @@ describe('codeburn serve --stdio', () => { let child: ChildProcess let buffer = '' const waiters = new Map) => void>() + const progressFrames = new Map>>() + let configPath = '' let readyResolve: () => void const ready = new Promise(resolve => { readyResolve = resolve }) @@ -25,6 +52,23 @@ describe('codeburn serve --stdio', () => { } beforeAll(async () => { + const home = process.env['HOME']! + configPath = join(home, '.config', 'codeburn', 'config.json') + await mkdir(join(home, '.config', 'codeburn'), { recursive: true }) + // Give the resident process one real provider root to arm. With no + // successfully armed roots, event-driven reuse correctly stays disabled. + await mkdir(join(home, '.claude', 'projects'), { recursive: true }) + await writeFile(configPath, JSON.stringify({ currency: { code: 'USD' } }), 'utf8') + + // Keep the EUR half of the config-freshness regression fully offline. + const cacheDir = join(home, '.cache', 'codeburn') + await mkdir(cacheDir, { recursive: true }) + await writeFile(join(cacheDir, 'exchange-rate.json'), JSON.stringify({ + timestamp: Date.now(), + code: 'EUR', + rate: 0.9, + }), 'utf8') + child = spawn(process.execPath, ['--import', 'tsx', join(__dirname, '..', 'src', 'cli.ts'), 'serve', '--stdio'], { stdio: ['pipe', 'pipe', 'ignore'], env: { ...process.env }, @@ -40,6 +84,13 @@ describe('codeburn serve --stdio', () => { let msg: Record try { msg = JSON.parse(line) } catch { continue } if (msg['ready']) { readyResolve(); continue } + if (typeof msg['progress'] === 'string' && !('ok' in msg)) { + const id = msg['id'] as number + const frames = progressFrames.get(id) ?? [] + frames.push(msg) + progressFrames.set(id, frames) + continue + } const waiter = waiters.get(msg['id'] as number) if (waiter) { waiters.delete(msg['id'] as number); waiter(msg) } } @@ -82,9 +133,218 @@ describe('codeburn serve --stdio', () => { expect(res['refused']).toBe(true) }) + it('refuses every optimize apply-only option without touching shell config or the action journal', async () => { + const home = process.env['HOME']! + const zshrc = join(home, '.zshrc') + const journal = join(home, '.config', 'codeburn', 'actions', 'journal.jsonl') + await writeFile(zshrc, '# user-owned\n', 'utf8') + + // `optimize` is the only served command whose Commander definition also + // has mutation-capable options. The full request below used to execute a + // shell-config action inside the resident process. + const applied = await request(300, [ + 'optimize', '--apply', '--yes', '--only', 'bash-output-cap', '--period', 'today', + ]) + expect(applied).toMatchObject({ ok: false, refused: true }) + + // Keep the allowlist categorical: apply-only modifiers are not useful to + // a read query and must not become resident options on their own either. + for (const [id, args] of [ + [301, ['optimize', '--yes']], + [302, ['optimize', '--dry-run']], + [303, ['optimize', '--only', 'bash-output-cap']], + ] as const) { + expect(await request(id, [...args])).toMatchObject({ ok: false, refused: true }) + } + + expect(await readFile(zshrc, 'utf8')).toBe('# user-owned\n') + await expect(readFile(journal, 'utf8')).rejects.toMatchObject({ code: 'ENOENT' }) + }, 60_000) + + it('accepts the reviewed read-only option surface for every served command', async () => { + const commands: Array<[number, string[]]> = [ + [310, ['status', '--format', 'json', '--period', 'today']], + [311, ['overview', '--period', 'today', '--no-color']], + [312, ['models', '--format', 'json', '--period', 'today', '--no-totals']], + [313, ['sessions', '--format', 'json', '--period', 'today', '--no-pager']], + [314, ['compare', '--format', 'json', '--period', 'today']], + [315, ['yield', '--format', 'json', '--period', 'today']], + [316, ['spend', '--format', 'flow-json', '--period', 'today']], + [317, ['optimize', '--format', 'json', '--period', 'today']], + [318, ['audit', '--format', 'json', '--period', 'today']], + ] + for (const [id, args] of commands) { + expect(await request(id, args)).toMatchObject({ ok: true }) + } + }, 60_000) + it('survives a malformed request line and keeps serving', async () => { sendRaw('this is not json') const res = await request(6, ['status', '--format', 'menubar-json', '--period', 'today']) expect(res['ok']).toBe(true) }, 60_000) + + it('streams captured command stderr as protocol progress frames', async () => { + const res = await request(7, ['status', '--provider', 'definitely-not-a-real-provider']) + expect(res['ok']).toBe(false) + + const frames = progressFrames.get(7) ?? [] + expect(frames.length).toBeGreaterThan(0) + expect(frames.every(frame => Object.keys(frame).sort().join(',') === 'id,progress')).toBe(true) + expect(frames.map(frame => frame['progress']).join('')).toContain('unknown provider') + }, 60_000) + + it('discovers a newly configured Claude root on identical resident argv', async () => { + const home = process.env['HOME']! + const rootA = join(home, 'claude-root-a') + const rootB = join(home, 'claude-root-b') + const slug = '-Users-test-shared-project' + const cwd = '/Users/test/shared-project' + + const writeClaudeSession = async (root: string, sessionId: string, marker: string): Promise => { + const projectDir = join(root, 'projects', slug) + await mkdir(projectDir, { recursive: true }) + const lines = [ + { + type: 'summary', summary: marker, leafUuid: `leaf-${marker}`, sessionId, cwd, + timestamp: '2026-08-12T10:00:00.000Z', + }, + { + type: 'user', uuid: `user-${marker}`, sessionId, cwd, + timestamp: '2026-08-12T10:00:01.000Z', message: { role: 'user', content: marker }, + }, + { + type: 'assistant', uuid: `assistant-${marker}`, parentUuid: `user-${marker}`, sessionId, cwd, + timestamp: '2026-08-12T10:00:02.000Z', + message: { + id: `msg-${marker}`, type: 'message', role: 'assistant', model: 'claude-sonnet-4-6', + content: [{ type: 'text', text: 'reply' }], usage: { input_tokens: 100, output_tokens: 50 }, + }, + }, + ] + await writeFile(join(projectDir, `${sessionId}.jsonl`), lines.map(line => JSON.stringify(line)).join('\n')) + } + + await writeClaudeSession(rootA, 'resident-session-a', 'a') + await writeClaudeSession(rootB, 'resident-session-b', 'b') + const args = ['sessions', '--period', 'lifetime', '--provider', 'claude', '--format', 'json', '--no-pager'] + + await writeFile(configPath, JSON.stringify({ claudeConfigDirs: [rootA] }), 'utf8') + const first = await request(200, args) + expect(first['ok']).toBe(true) + expect((JSON.parse(first['output'] as string) as Array<{ sessionId: string }>).map(row => row.sessionId)).toEqual([ + 'resident-session-a', + ]) + + // Same command in the same process; only config.json adds root B. + await writeFile(configPath, JSON.stringify({ claudeConfigDirs: [rootA, rootB] }), 'utf8') + const second = await request(201, args) + expect(second['ok']).toBe(true) + expect((JSON.parse(second['output'] as string) as Array<{ sessionId: string }>).map(row => row.sessionId).sort()).toEqual([ + 'resident-session-a', + 'resident-session-b', + ]) + + // Keep the following currency-freshness regression self-contained. + await writeFile(configPath, JSON.stringify({ currency: { code: 'USD' } }), 'utf8') + }, 60_000) + + it('invalidates identical-argv output memo immediately when config.json changes', async () => { + const args = ['status', '--format', 'menubar-json', '--period', 'week', '--no-optimize', '--no-timeline'] + const usdConfig = JSON.stringify({ currency: { code: 'USD' } }) + await writeFile(configPath, usdConfig, 'utf8') + + let previous = await request(8, args) + expect(previous['ok']).toBe(true) + expect((JSON.parse(previous['output'] as string) as { currency: { code: string } }).currency.code).toBe('USD') + + // Prove this argv is actually hitting the output memo before testing its + // invalidation. The root watchers arm asynchronously at serve startup, so + // allow a few requests until two byte-identical generated payloads arrive. + let memoized: Record | null = null + for (let id = 9; id < 110; id++) { + await new Promise(resolve => setTimeout(resolve, 20)) + const next = await request(id, args) + if (next['output'] === previous['output']) { + memoized = next + break + } + previous = next + } + expect(memoized).not.toBeNull() + + // A byte-identical rewrite changes filesystem metadata but not effective + // configuration. The memo must survive it and return the exact generated + // payload, including the original volatile `generated` timestamp. + await new Promise(resolve => setTimeout(resolve, 20)) + await writeFile(configPath, usdConfig, 'utf8') + const sameBytes = await request(110, args) + expect(sameBytes['ok']).toBe(true) + expect(sameBytes['output']).toBe(memoized!['output']) + // `generated` is minted per render, so an unchanged stamp is the proof + // that a memo hit returns the stored string instead of re-rendering. + const stamp = (res: Record): string => + (JSON.parse(res['output'] as string) as { generated: string }).generated + expect(stamp(sameBytes)).toBe(stamp(memoized!)) + + // Same byte length as USD: a size-only fingerprint would miss this. + await writeFile(configPath, JSON.stringify({ currency: { code: 'EUR' } }), 'utf8') + const fresh = await request(111, args) + expect(fresh['ok']).toBe(true) + expect((JSON.parse(fresh['output'] as string) as { currency: { code: string } }).currency.code).toBe('EUR') + expect(fresh['output']).not.toBe(memoized!['output']) + + // Removing the configured currency is the USD reset contract. The serve + // process must reset its module-level currency state as well as invalidate + // the output memo, otherwise a long-lived child keeps rendering EUR. + await writeFile(configPath, '{}', 'utf8') + const reset = await request(112, args) + expect(reset['ok']).toBe(true) + expect((JSON.parse(reset['output'] as string) as { + currency: { code: string; rate: number } + }).currency).toMatchObject({ code: 'USD', rate: 1 }) + }, 60_000) + + it('exits on natural stdin EOF after arming a watcher for an existing Claude root', async () => { + const claudeRoot = join(process.env['HOME']!, 'claude-eof-root') + await mkdir(join(claudeRoot, 'projects'), { recursive: true }) + + const eofChild = spawn(process.execPath, ['--import', 'tsx', join(__dirname, '..', 'src', 'cli.ts'), 'serve', '--stdio'], { + stdio: ['pipe', 'pipe', 'ignore'], + env: { ...process.env, CLAUDE_CONFIG_DIR: claudeRoot }, + }) + let stdout = '' + const becameReady = new Promise((resolve, reject) => { + eofChild.once('error', reject) + eofChild.stdout!.setEncoding('utf8') + eofChild.stdout!.on('data', (chunk: string) => { + stdout += chunk + if (stdout.split('\n').some(line => { + try { return (JSON.parse(line) as { ready?: boolean }).ready === true } catch { return false } + })) resolve() + }) + eofChild.once('exit', (code, signal) => reject(new Error(`serve exited before ready: ${code ?? signal}`))) + }) + const exited = new Promise(resolve => eofChild.once('exit', () => resolve(true))) + + let naturalExit = false + try { + await becameReady + // READY is intentionally emitted before provider probing; give the real + // watcher setup time to finish so the regression exercises its handle. + await new Promise(resolve => setTimeout(resolve, 500)) + eofChild.stdin!.end() + naturalExit = await Promise.race([ + exited, + new Promise(resolve => setTimeout(() => resolve(false), 2_000)), + ]) + } finally { + if (!naturalExit) { + eofChild.kill('SIGKILL') + await exited + } + } + + expect(naturalExit).toBe(true) + }, 10_000) }) diff --git a/tests/session-cache-rich-capture.test.ts b/tests/session-cache-rich-capture.test.ts index 11301308..3ed609d7 100644 --- a/tests/session-cache-rich-capture.test.ts +++ b/tests/session-cache-rich-capture.test.ts @@ -12,8 +12,8 @@ import { emptyCache, loadCache, saveCache, - sessionCachePath, } from '../src/session-cache.js' +import { writeCacheOnDisk } from './fixtures/session-cache-io.js' const TMP_DIR = join(tmpdir(), `codeburn-rich-cache-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`) @@ -113,7 +113,7 @@ describe('session cache round-trip for rich-capture fields', () => { }, } if (!existsSync(TMP_DIR)) await mkdir(TMP_DIR, { recursive: true }) - await writeFile(sessionCachePath(), JSON.stringify(oldCache), 'utf-8') + await writeCacheOnDisk(oldCache) const loaded = await loadCache() const call = loaded.providers['claude']!.files['/x/old.jsonl']!.turns[0]!.calls[0]! diff --git a/tests/session-cache-shards.test.ts b/tests/session-cache-shards.test.ts new file mode 100644 index 00000000..ff53c9b6 --- /dev/null +++ b/tests/session-cache-shards.test.ts @@ -0,0 +1,802 @@ +// Provider x month shard layout (CACHE_VERSION 9): the on-disk cache is a +// directory holding one envelope plus one shard per provider-month. What matters +// here is that the move off the older layouts loses nothing, that a file's +// bucket never moves when the session is appended to, that a save rewrites only +// the months that changed (including when the load was scoped to a subset of +// them), and that one unreadable shard costs exactly one month. +import { afterEach, beforeEach, describe, expect, it } from 'vitest' +import { mkdir, readFile, readdir, rm, stat, utimes, writeFile } from 'fs/promises' +import { existsSync } from 'fs' +import { tmpdir } from 'os' +import { join } from 'path' + +import { + CACHE_VERSION, + cacheBucketMonth, + cacheFileSpan, + computeEnvFingerprint, + cleanupOrphanedTempFiles, + clearLoadCacheMemo, + loadCache, + markCacheDirty, + monthScopeForRange, + saveCache, + sessionCacheDir, + type CachedFile, + type SessionCache, +} from '../src/session-cache.js' + +let TMP_DIR: string + +beforeEach(async () => { + TMP_DIR = join(tmpdir(), `codeburn-shard-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`) + process.env['CODEBURN_CACHE_DIR'] = TMP_DIR + await mkdir(TMP_DIR, { recursive: true }) + clearLoadCacheMemo() +}) + +afterEach(async () => { + if (existsSync(TMP_DIR)) await rm(TMP_DIR, { recursive: true }) +}) + +function turnAt(timestamp: string, key = 'msg-1'): CachedFile['turns'][number] { + const base = cachedFile().turns[0]! + return { ...base, timestamp, calls: [{ ...base.calls[0]!, timestamp, deduplicationKey: key }] } +} + +function fileSpanning(first: string, last?: string): CachedFile { + return cachedFile({ turns: last ? [turnAt(first, 'a'), turnAt(last, 'b')] : [turnAt(first, 'a')] }) +} + +function cachedFile(overrides: Partial = {}): CachedFile { + return { + fingerprint: { dev: 1, ino: 2, mtimeMs: 3, sizeBytes: 4 }, + lastCompleteLineOffset: 128, + mcpInventory: ['mcp__github__list'], + turns: [{ + timestamp: '2026-05-15T10:00:00Z', + sessionId: 'sess-1', + userMessage: 'do the thing', + calls: [{ + provider: 'claude', + model: 'claude-sonnet-4-20250514', + usage: { + inputTokens: 1000, + outputTokens: 500, + cacheCreationInputTokens: 0, + cacheReadInputTokens: 0, + cachedInputTokens: 0, + reasoningTokens: 0, + webSearchRequests: 0, + cacheCreationOneHourTokens: 0, + }, + costUSD: 0.01, + speed: 'standard', + timestamp: '2026-05-15T10:00:00Z', + tools: ['Read'], + bashCommands: [], + skills: [], + subagentTypes: [], + deduplicationKey: 'msg-1', + }], + }], + ...overrides, + } +} + +function v7Cache(): SessionCache { + return { + version: 7, + complete: true, + providers: { + claude: { + envFingerprint: 'claude-fp', + files: { + '/live/a.jsonl': cachedFile(), + '/live/b.jsonl': cachedFile({ turns: [] }), + // An orphaned PR-linked entry: its transcript is gone and can never + // re-parse, so the migration has to carry it across verbatim. + '/gone/pruned.jsonl': cachedFile({ prLinks: ['https://github.com/o/r/pull/1'] }), + }, + }, + codex: { + envFingerprint: 'codex-fp', + durable: true, + files: { '/live/rollout.jsonl': cachedFile() }, + }, + }, + } +} + +async function shardNames(): Promise { + return (await readdir(sessionCacheDir())).sort() +} + +async function envelope(): Promise<{ providers: Record }> }> { + return JSON.parse(await readFile(join(sessionCacheDir(), 'envelope.json'), 'utf-8')) +} + +/** name -> bytes, for every shard on disk. */ +async function shardBytes(): Promise> { + const dir = sessionCacheDir() + const out = new Map() + for (const name of await shardNames()) out.set(name, await readFile(join(dir, name), 'utf-8')) + return out +} + +describe('v7 -> shard migration', () => { + it('is lossless: every entry survives, shards replace the v7 file, reload matches', async () => { + const v7 = v7Cache() + const v7Path = join(TMP_DIR, 'session-cache.v7.json') + await writeFile(v7Path, JSON.stringify(v7)) + + const loaded = await loadCache() + // Same content, re-stamped at the current version. + expect(loaded).toEqual({ ...v7, version: CACHE_VERSION }) + + // Shards on disk, v7 blob removed. + expect(existsSync(v7Path)).toBe(false) + const names = await shardNames() + expect(names).toContain('envelope.json') + // claude's three entries split by month: two dated 2026-05, one turn-less. + expect(Object.keys((await envelope()).providers['claude']!.shards).sort()).toEqual(['0000-00', '2026-05']) + expect(Object.keys((await envelope()).providers['codex']!.shards)).toEqual(['2026-05']) + + // A second load reads only the shards and produces the same cache. + clearLoadCacheMemo() + expect(await loadCache()).toEqual(loaded) + }) + + it('leaves a corrupt v7 file alone and starts fresh', async () => { + await writeFile(join(TMP_DIR, 'session-cache.v7.json'), '{broken') + const loaded = await loadCache() + expect(loaded.providers).toEqual({}) + expect(existsSync(join(TMP_DIR, 'session-cache.v7.json'))).toBe(true) + }) +}) + +describe('per-provider dirty tracking', () => { + it('rewrites only the provider that changed', async () => { + await writeFile(join(TMP_DIR, 'session-cache.v7.json'), JSON.stringify(v7Cache())) + const cache = await loadCache() + + const dir = sessionCacheDir() + const before = new Map() + for (const name of await shardNames()) before.set(name, await readFile(join(dir, name), 'utf-8')) + + cache.providers['codex']!.files['/live/rollout.jsonl'] = cachedFile({ mcpInventory: ['changed'] }) + markCacheDirty(cache, 'codex') + await saveCache(cache) + + const after = await shardNames() + const claudeShard = [...before.keys()].find(n => n.startsWith('claude.'))! + // The untouched provider keeps its exact file, byte for byte. + expect(after).toContain(claudeShard) + expect(await readFile(join(dir, claudeShard), 'utf-8')).toBe(before.get(claudeShard)) + // The changed provider is republished under a new name; the old one is gone. + const codexBefore = [...before.keys()].find(n => n.startsWith('codex.'))! + const codexAfter = after.find(n => n.startsWith('codex.'))! + expect(codexAfter).not.toBe(codexBefore) + expect(after).not.toContain(codexBefore) + + clearLoadCacheMemo() + const reloaded = await loadCache() + expect(reloaded.providers['codex']!.files['/live/rollout.jsonl']!.mcpInventory).toEqual(['changed']) + expect(reloaded.providers['claude']).toEqual(cache.providers['claude']) + }) +}) + +describe('corrupt shard isolation', () => { + it('drops only the unreadable month, keeping every other month and provider', async () => { + const cache: SessionCache = { + version: CACHE_VERSION, + complete: true, + providers: { + claude: { + envFingerprint: 'claude-fp', + files: { '/live/may.jsonl': fileSpanning('2026-05-15T10:00:00Z'), '/live/jun.jsonl': fileSpanning('2026-06-15T10:00:00Z') }, + }, + codex: { envFingerprint: 'codex-fp', files: { '/live/r.jsonl': fileSpanning('2026-05-15T10:00:00Z') } }, + }, + } + markCacheDirty(cache, 'claude') + markCacheDirty(cache, 'codex') + await saveCache(cache) + + const dir = sessionCacheDir() + await writeFile(join(dir, (await envelope()).providers['claude']!.shards['2026-05']!.name), '{"/x":{"turns":') + + clearLoadCacheMemo() + const reloaded = await loadCache() + expect(Object.keys(reloaded.providers['claude']!.files)).toEqual(['/live/jun.jsonl']) + expect(reloaded.providers['codex']).toEqual(cache.providers['codex']) + + // Self-heals: the unreadable month is republished from whatever re-parses + // into it rather than being carried forward corrupt forever. + reloaded.providers['claude']!.files['/live/may.jsonl'] = fileSpanning('2026-05-15T10:00:00Z') + markCacheDirty(reloaded, 'claude', '/live/may.jsonl') + await saveCache(reloaded) + clearLoadCacheMemo() + expect(Object.keys((await loadCache()).providers['claude']!.files).sort()).toEqual(['/live/jun.jsonl', '/live/may.jsonl']) + }) +}) + +describe('cleanupOrphanedTempFiles', () => { + it('sweeps stale shard temps and unreferenced shards, keeping the live ones', async () => { + await saveCache({ version: CACHE_VERSION, complete: true, providers: { + claude: { envFingerprint: 'fp', files: { '/a.jsonl': cachedFile() } }, + } }) + const dir = sessionCacheDir() + const live = (await shardNames()).find(n => n.startsWith('claude.'))! + + const backdate = async (path: string, minutes: number) => { + const at = new Date(Date.now() - minutes * 60 * 1000) + await utimes(path, at, at) + } + + const oldTemp = join(dir, 'claude.deadbeef.json.tmp') + await writeFile(oldTemp, 'partial') + await backdate(oldTemp, 10) + const orphanShard = join(dir, 'codex.deadbeef.json') + await writeFile(orphanShard, '{}') + await backdate(orphanShard, 90) + // Unreferenced but fresh: this is what a CONCURRENT save's shard looks like + // before its envelope lands, so the sweep must leave it alone. + const inFlightShard = join(dir, 'codex.c0ffee00.json') + await writeFile(inFlightShard, '{}') + const recentTemp = join(dir, 'claude.feedface.json.tmp') + await writeFile(recentTemp, 'in flight') + // The live shard is far older than the temp cutoff; being referenced is what + // protects it, not its age. + await backdate(join(dir, live), 120) + + await cleanupOrphanedTempFiles() + + expect(existsSync(oldTemp)).toBe(false) + expect(existsSync(orphanShard)).toBe(false) + expect(existsSync(inFlightShard)).toBe(true) + expect(existsSync(recentTemp)).toBe(true) + expect(existsSync(join(dir, live))).toBe(true) + expect(existsSync(join(dir, 'envelope.json'))).toBe(true) + }) + + it('retires the pre-v8 single-file layout temps left in the parent directory', async () => { + await saveCache({ version: CACHE_VERSION, complete: true, providers: {} }) + const legacyTemp = join(TMP_DIR, 'session-cache.v7.json.abc123.tmp') + await writeFile(legacyTemp, 'orphan from an older build') + const at = new Date(Date.now() - 10 * 60 * 1000) + await utimes(legacyTemp, at, at) + const freshLegacyTemp = join(TMP_DIR, 'session-cache.v7.json.def456.tmp') + await writeFile(freshLegacyTemp, 'an old binary mid-write') + + await cleanupOrphanedTempFiles() + + expect(existsSync(legacyTemp)).toBe(false) + expect(existsSync(freshLegacyTemp)).toBe(true) + }) +}) + +// Two live processes share one cache directory routinely: a one-shot CLI beside +// the resident serve child, or two menubar polls. Neither may publish an +// envelope naming a file that is not there — that reads back as a corrupt +// provider and silently drops its history. +describe('concurrent writers', () => { + function seed(provider: string, tag: string, files: number): SessionCache { + const cache: SessionCache = { + version: CACHE_VERSION, + complete: true, + providers: { + [provider]: { + envFingerprint: tag, + files: Object.fromEntries( + Array.from({ length: files }, (_, i) => [`/f/${provider}/${i}.jsonl`, cachedFile()]), + ), + }, + }, + } + markCacheDirty(cache, provider) + return cache + } + + async function assertReferentialIntegrity(expected: string[]): Promise { + const dir = sessionCacheDir() + for (const meta of Object.values((await envelope()).providers)) { + for (const ref of Object.values(meta.shards)) { + expect(existsSync(join(dir, ref.name)), `envelope names a missing shard: ${ref.name}`).toBe(true) + } + } + clearLoadCacheMemo() + const loaded = await loadCache() + expect(Object.keys(loaded.providers).length).toBeGreaterThan(0) + for (const provider of expected) expect(loaded.providers[provider]).toBeDefined() + } + + it('never publishes a dangling envelope when two saves race', async () => { + for (let round = 0; round < 15; round++) { + await Promise.allSettled([ + saveCache(seed('claude', `a${round}`, 40)), + saveCache(seed('codex', `b${round}`, 40)), + ]) + // Whichever won, the published set has to be internally consistent and + // hold at least the provider that got there last. + await assertReferentialIntegrity([]) + } + }) + + it('a stale writer rewrites a shard another process retired instead of orphaning it', async () => { + // Seed: claude holds an expired-source PR orphan no re-parse can recover. + const initial: SessionCache = { + version: CACHE_VERSION, + complete: true, + providers: { + claude: { envFingerprint: 'fp', files: { '/gone/pruned.jsonl': cachedFile({ prLinks: ['https://github.com/o/r/pull/1'] }) } }, + codex: { envFingerprint: 'fp', durable: true, files: { '/live/r.jsonl': cachedFile() } }, + }, + } + markCacheDirty(initial, 'claude') + markCacheDirty(initial, 'codex') + await saveCache(initial) + + // Process B loads now, recording claude's current shard name. + clearLoadCacheMemo() + const b = await loadCache() + + // Process A independently touches ONLY claude and republishes, retiring the + // shard file B is still holding a name for. + clearLoadCacheMemo() + const a = await loadCache() + a.providers['claude']!.files['/live/new.jsonl'] = cachedFile() + markCacheDirty(a, 'claude') + await saveCache(a) + + // B now saves an unrelated codex change. + b.providers['codex']!.files['/live/r2.jsonl'] = cachedFile() + markCacheDirty(b, 'codex') + await saveCache(b) + + await assertReferentialIntegrity(['claude', 'codex']) + clearLoadCacheMemo() + const final = await loadCache() + expect(final.providers['claude']!.files['/gone/pruned.jsonl']).toBeDefined() + expect(final.providers['codex']!.files['/live/r2.jsonl']).toBeDefined() + }) +}) + +describe('month buckets', () => { + it('keeps a file in its first-turn month when the session is appended to', async () => { + const cache: SessionCache = { + version: CACHE_VERSION, + complete: true, + providers: { claude: { envFingerprint: 'fp', files: { '/live/long.jsonl': fileSpanning('2026-05-15T10:00:00Z') } } }, + } + markCacheDirty(cache, 'claude') + await saveCache(cache) + expect(Object.keys((await envelope()).providers['claude']!.shards)).toEqual(['2026-05']) + + // Two months of appends later the bucket is unchanged; only `until` moves, + // which is what lets a ranged load still find this session. + const appended = fileSpanning('2026-05-15T10:00:00Z', '2026-07-02T10:00:00Z') + expect(cacheBucketMonth(appended)).toBe('2026-05') + cache.providers['claude']!.files['/live/long.jsonl'] = appended + markCacheDirty(cache, 'claude', '/live/long.jsonl') + await saveCache(cache) + const shards = (await envelope()).providers['claude']!.shards + expect(Object.keys(shards)).toEqual(['2026-05']) + expect(shards['2026-05']!.until).toBe('2026-07') + + // ...and a July query still loads it, despite the May bucket key. + clearLoadCacheMemo() + const scoped = await loadCache(monthScopeForRange(new Date('2026-07-01T00:00:00Z'), new Date('2026-07-31T23:59:59Z'))) + expect(scoped.providers['claude']!.files['/live/long.jsonl']).toBeDefined() + }) + + it('rewrites only the month that changed', async () => { + const cache: SessionCache = { + version: CACHE_VERSION, + complete: true, + providers: { + claude: { + envFingerprint: 'fp', + files: { + '/live/mar.jsonl': fileSpanning('2026-03-10T10:00:00Z'), + '/live/apr.jsonl': fileSpanning('2026-04-10T10:00:00Z'), + '/live/may.jsonl': fileSpanning('2026-05-10T10:00:00Z'), + }, + }, + }, + } + markCacheDirty(cache, 'claude') + await saveCache(cache) + const before = await shardBytes() + const untouched = [ + (await envelope()).providers['claude']!.shards['2026-03']!.name, + (await envelope()).providers['claude']!.shards['2026-04']!.name, + ] + + cache.providers['claude']!.files['/live/may.jsonl'] = cachedFile({ turns: [turnAt('2026-05-10T10:00:00Z', 'a')], mcpInventory: ['changed'] }) + markCacheDirty(cache, 'claude', '/live/may.jsonl') + await saveCache(cache) + + const after = await shardBytes() + for (const name of untouched) expect(after.get(name)).toBe(before.get(name)) + expect(after.has((await envelope()).providers['claude']!.shards['2026-05']!.name)).toBe(true) + }) + + it('dirties the month a deleted file was in', async () => { + const cache: SessionCache = { + version: CACHE_VERSION, + complete: true, + providers: { + claude: { + envFingerprint: 'fp', + files: { '/live/mar.jsonl': fileSpanning('2026-03-10T10:00:00Z'), '/live/mar2.jsonl': fileSpanning('2026-03-11T10:00:00Z') }, + }, + }, + } + markCacheDirty(cache, 'claude') + await saveCache(cache) + + delete cache.providers['claude']!.files['/live/mar2.jsonl'] + markCacheDirty(cache, 'claude', '/live/mar2.jsonl') + await saveCache(cache) + + clearLoadCacheMemo() + expect(Object.keys((await loadCache()).providers['claude']!.files)).toEqual(['/live/mar.jsonl']) + }) +}) + +describe('scoped load', () => { + async function seedThreeMonths(): Promise { + const cache: SessionCache = { + version: CACHE_VERSION, + complete: true, + providers: { + claude: { + envFingerprint: computeEnvFingerprint('claude'), + files: { + '/live/mar.jsonl': fileSpanning('2026-03-10T10:00:00Z'), + '/live/apr.jsonl': fileSpanning('2026-04-10T10:00:00Z'), + '/live/jun.jsonl': fileSpanning('2026-06-10T10:00:00Z'), + }, + }, + }, + } + markCacheDirty(cache, 'claude') + await saveCache(cache) + } + + const juneScope = monthScopeForRange(new Date('2026-06-01T00:00:00Z'), new Date('2026-06-30T23:59:59Z')) + + it('reads only the months the range can report on, plus one of slack', async () => { + await seedThreeMonths() + clearLoadCacheMemo() + const scoped = await loadCache(juneScope) + // June is in range; May would be the slack month (absent here); March and + // April cannot contribute a June turn and stay on disk. + expect(Object.keys(scoped.providers['claude']!.files)).toEqual(['/live/jun.jsonl']) + }) + + it('save from a scoped load leaves the unloaded months byte-identical', async () => { + await seedThreeMonths() + const before = await shardBytes() + const kept = [ + (await envelope()).providers['claude']!.shards['2026-03']!.name, + (await envelope()).providers['claude']!.shards['2026-04']!.name, + ] + + clearLoadCacheMemo() + const scoped = await loadCache(juneScope) + scoped.providers['claude']!.files['/live/jun2.jsonl'] = fileSpanning('2026-06-20T10:00:00Z') + markCacheDirty(scoped, 'claude', '/live/jun2.jsonl') + await saveCache(scoped) + + const after = await shardBytes() + for (const name of kept) expect(after.get(name), `unloaded month rewritten: ${name}`).toBe(before.get(name)) + + clearLoadCacheMemo() + const full = await loadCache() + expect(Object.keys(full.providers['claude']!.files).sort()) + .toEqual(['/live/apr.jsonl', '/live/jun.jsonl', '/live/jun2.jsonl', '/live/mar.jsonl']) + }) + + it('merges rather than replaces when a re-parse lands in an unloaded month', async () => { + await seedThreeMonths() + clearLoadCacheMemo() + const scoped = await loadCache(juneScope) + // A March session that was never loaded is re-parsed (its mtime moved) and + // written straight back into the March bucket. + scoped.providers['claude']!.files['/live/mar.jsonl'] = cachedFile({ turns: [turnAt('2026-03-10T10:00:00Z', 'z')], mcpInventory: ['reparsed'] }) + markCacheDirty(scoped, 'claude', '/live/mar.jsonl') + await saveCache(scoped) + + clearLoadCacheMemo() + const full = await loadCache() + expect(full.providers['claude']!.files['/live/mar.jsonl']!.mcpInventory).toEqual(['reparsed']) + expect(Object.keys(full.providers['claude']!.files).sort()) + .toEqual(['/live/apr.jsonl', '/live/jun.jsonl', '/live/mar.jsonl']) + }) + + it('never scopes a provider whose fingerprint moved, or a durable one', async () => { + const cache: SessionCache = { + version: CACHE_VERSION, + complete: true, + providers: { + claude: { envFingerprint: 'stale-fp', files: { '/live/mar.jsonl': fileSpanning('2026-03-10T10:00:00Z') } }, + copilot: { envFingerprint: computeEnvFingerprint('copilot'), durable: true, files: { '/live/otel.db': fileSpanning('2026-03-10T10:00:00Z') } }, + }, + } + markCacheDirty(cache, 'claude') + markCacheDirty(cache, 'copilot') + await saveCache(cache) + + clearLoadCacheMemo() + const scoped = await loadCache(juneScope) + // Both would be skipped on month alone; both are read in full anyway, so the + // fingerprint reset and the durable orphan carry-forward see every entry. + expect(scoped.providers['claude']!.files['/live/mar.jsonl']).toBeDefined() + expect(scoped.providers['copilot']!.files['/live/otel.db']).toBeDefined() + }) +}) + +describe('v8 -> v9 migration', () => { + it('re-buckets the v8 provider shards losslessly and retires the v8 directory', async () => { + const v8Dir = join(TMP_DIR, 'session-cache.v8') + await mkdir(v8Dir, { recursive: true }) + const section = { + envFingerprint: 'claude-fp', + files: { + '/live/mar.jsonl': fileSpanning('2026-03-10T10:00:00Z'), + '/live/jun.jsonl': fileSpanning('2026-06-10T10:00:00Z'), + '/gone/pruned.jsonl': cachedFile({ prLinks: ['https://github.com/o/r/pull/1'] }), + }, + } + await writeFile(join(v8Dir, 'claude.abc.json'), JSON.stringify(section)) + await writeFile(join(v8Dir, 'envelope.json'), JSON.stringify({ + version: 8, complete: true, nonce: 'n', shards: { claude: 'claude.abc.json' }, + })) + + const loaded = await loadCache() + expect(loaded.providers['claude']).toEqual(section) + expect(loaded.complete).toBe(true) + expect(existsSync(v8Dir)).toBe(false) + expect(Object.keys((await envelope()).providers['claude']!.shards).sort()).toEqual(['2026-03', '2026-05', '2026-06']) + + clearLoadCacheMemo() + expect(await loadCache()).toEqual(loaded) + }) +}) + +// Several providers emit turns in a non-chronological order (cursor composers by +// ROWID, goose / crush / copilot by a DESC ordering). Reading the span off +// turns[0]/turns[-1] then gives `until < bucket` — an empty span, unreachable at +// every scope. +describe('out-of-order turns', () => { + const outOfOrder = () => cachedFile({ turns: [turnAt('2026-08-10T10:00:00Z', 'a'), turnAt('2026-03-04T10:00:00Z', 'b')] }) + + it('spans oldest to newest whatever order the turns arrive in', () => { + const span = cacheFileSpan(outOfOrder()) + expect(span).toEqual({ bucket: '2026-03', until: '2026-08' }) + expect(cacheBucketMonth(outOfOrder())).toBe('2026-03') + }) + + it('stays reachable at the scope of either end', async () => { + const cache: SessionCache = { + version: CACHE_VERSION, + complete: true, + providers: { claude: { envFingerprint: computeEnvFingerprint('claude'), files: { '/live/desc.jsonl': outOfOrder() } } }, + } + markCacheDirty(cache, 'claude') + await saveCache(cache) + expect((await envelope()).providers['claude']!.shards['2026-03']!.until).toBe('2026-08') + + for (const [from, to] of [['2026-03-01', '2026-03-31'], ['2026-08-01', '2026-08-31']] as const) { + clearLoadCacheMemo() + const scoped = await loadCache(monthScopeForRange(new Date(`${from}T00:00:00Z`), new Date(`${to}T23:59:59Z`))) + expect(scoped.providers['claude']!.files['/live/desc.jsonl'], `unreachable at ${from}`).toBeDefined() + } + }) +}) + +// A path must never end up in two shards at once: on a later load the two copies +// race and the stale one can win, and nothing sweeps it because the envelope +// names both. +describe('re-bucketing out of an unloaded month', () => { + const juneScope = monthScopeForRange(new Date('2026-06-01T00:00:00Z'), new Date('2026-06-30T23:59:59Z')) + + async function seed(): Promise { + const cache: SessionCache = { + version: CACHE_VERSION, + complete: true, + providers: { + claude: { + envFingerprint: computeEnvFingerprint('claude'), + files: { + '/live/moving.jsonl': fileSpanning('2026-01-10T10:00:00Z'), + '/live/stay.jsonl': fileSpanning('2026-01-11T10:00:00Z'), + '/live/jun.jsonl': fileSpanning('2026-06-10T10:00:00Z'), + }, + }, + }, + } + markCacheDirty(cache, 'claude') + await saveCache(cache) + } + + /** Every shard's view of `path`, so a duplicate is visible directly. */ + async function copiesOf(path: string): Promise { + const dir = sessionCacheDir() + const found: string[] = [] + for (const [bucket, ref] of Object.entries((await envelope()).providers['claude']!.shards)) { + const files = JSON.parse(await readFile(join(dir, ref.name), 'utf-8')) + if (files[path]) found.push(bucket) + } + return found.sort() + } + + it('(a) drops the old copy when a re-parse moves the file to another month', async () => { + await seed() + clearLoadCacheMemo() + const scoped = await loadCache(juneScope) + expect(scoped.providers['claude']!.files['/live/moving.jsonl']).toBeUndefined() + // Re-parsed from byte 0 after a rewrite: its oldest turn is now in May. + scoped.providers['claude']!.files['/live/moving.jsonl'] = cachedFile({ turns: [turnAt('2026-05-02T10:00:00Z', 'new')], mcpInventory: ['reparsed'] }) + markCacheDirty(scoped, 'claude', '/live/moving.jsonl') + await saveCache(scoped) + + expect(await copiesOf('/live/moving.jsonl')).toEqual(['2026-05']) + clearLoadCacheMemo() + const full = await loadCache() + expect(full.providers['claude']!.files['/live/moving.jsonl']!.mcpInventory).toEqual(['reparsed']) + expect(full.providers['claude']!.files['/live/stay.jsonl']).toBeDefined() + }) + + it('(b) drops the old copy when a parse failure leaves a turn-less marker', async () => { + await seed() + clearLoadCacheMemo() + const scoped = await loadCache(juneScope) + // The #441 path: the file threw, so only a failure marker is cached. + scoped.providers['claude']!.files['/live/moving.jsonl'] = { fingerprint: { dev: 1, ino: 2, mtimeMs: 9, sizeBytes: 4 }, mcpInventory: [], turns: [], failed: true } + markCacheDirty(scoped, 'claude', '/live/moving.jsonl') + await saveCache(scoped) + + expect(await copiesOf('/live/moving.jsonl')).toEqual(['0000-00']) + clearLoadCacheMemo() + const full = await loadCache() + expect(full.providers['claude']!.files['/live/moving.jsonl']!.failed).toBe(true) + expect(full.providers['claude']!.files['/live/stay.jsonl']).toBeDefined() + }) + + it('resolves a duplicate to the freshest copy and prunes it on the next save', async () => { + await seed() + // Forge the split state directly: the same path in two shards. + const dir = sessionCacheDir() + const env = await envelope() + const janName = env.providers['claude']!.shards['2026-01']!.name + const junName = env.providers['claude']!.shards['2026-06']!.name + const jun = JSON.parse(await readFile(join(dir, junName), 'utf-8')) + jun['/live/moving.jsonl'] = cachedFile({ turns: [turnAt('2026-06-02T10:00:00Z', 'fresh')], fingerprint: { dev: 1, ino: 2, mtimeMs: 999, sizeBytes: 4 }, mcpInventory: ['fresh'] }) + await writeFile(join(dir, junName), JSON.stringify(jun)) + + clearLoadCacheMemo() + const full = await loadCache() + // Newest fingerprint wins, whichever shard finished reading first. + expect(full.providers['claude']!.files['/live/moving.jsonl']!.mcpInventory).toEqual(['fresh']) + await saveCache(full) + expect(await copiesOf('/live/moving.jsonl')).toEqual(['2026-06']) + expect(existsSync(join(dir, janName))).toBe(false) + }) +}) + +describe('carried months under a concurrent writer', () => { + it('adopts the current shard rather than dropping an orphan-bearing month', async () => { + const orphan = cachedFile({ turns: [turnAt('2026-03-10T10:00:00Z', 'm')], prLinks: ['https://github.com/o/r/pull/1'] }) + const initial: SessionCache = { + version: CACHE_VERSION, + complete: true, + providers: { + claude: { + envFingerprint: computeEnvFingerprint('claude'), + files: { '/gone/mar.jsonl': orphan, '/live/jun.jsonl': fileSpanning('2026-06-10T10:00:00Z') }, + }, + }, + } + markCacheDirty(initial, 'claude') + await saveCache(initial) + + // Process B loads June-scoped: March is carried, by the name it saw. + clearLoadCacheMemo() + const b = await loadCache(monthScopeForRange(new Date('2026-06-01T00:00:00Z'), new Date('2026-06-30T23:59:59Z'))) + + // Process A independently republishes March, retiring the file B remembers. + clearLoadCacheMemo() + const a = await loadCache(monthScopeForRange(new Date('2026-03-01T00:00:00Z'), new Date('2026-03-31T23:59:59Z'))) + a.providers['claude']!.files['/gone/mar2.jsonl'] = cachedFile({ turns: [turnAt('2026-03-12T10:00:00Z', 'n')], prLinks: ['https://github.com/o/r/pull/2'] }) + markCacheDirty(a, 'claude', '/gone/mar2.jsonl') + await saveCache(a) + + // B saves its own unrelated June change. + b.providers['claude']!.files['/live/jun2.jsonl'] = fileSpanning('2026-06-20T10:00:00Z') + markCacheDirty(b, 'claude', '/live/jun2.jsonl') + await saveCache(b) + + clearLoadCacheMemo() + const final = await loadCache() + // March survived under A's name, with both orphans; June has both files. + expect(Object.keys(final.providers['claude']!.files).sort()) + .toEqual(['/gone/mar.jsonl', '/gone/mar2.jsonl', '/live/jun.jsonl', '/live/jun2.jsonl']) + for (const ref of Object.values((await envelope()).providers['claude']!.shards)) { + expect(existsSync(join(sessionCacheDir(), ref.name))).toBe(true) + } + }) + + // Two saves merging into the SAME unloaded month are a read-modify-write with + // no lock between them. In the product they are serialised by the warm refresh + // lock; this covers what survives when they are not. The optimistic retry in + // saveCache narrows the window to the envelope publish, and a loser's entries + // are re-derived by the next parse (the reconcile finds no cache entry and + // re-reads the file) rather than being lost for good. + it('two scoped saves merging into the same unloaded month keep the envelope sound', async () => { + const base: SessionCache = { + version: CACHE_VERSION, + complete: true, + providers: { + claude: { + envFingerprint: computeEnvFingerprint('claude'), + files: { '/live/mar.jsonl': fileSpanning('2026-03-10T10:00:00Z'), '/live/jun.jsonl': fileSpanning('2026-06-10T10:00:00Z') }, + }, + }, + } + markCacheDirty(base, 'claude') + await saveCache(base) + const juneScope = monthScopeForRange(new Date('2026-06-01T00:00:00Z'), new Date('2026-06-30T23:59:59Z')) + + clearLoadCacheMemo() + const p1 = await loadCache(juneScope) + clearLoadCacheMemo() + const p2 = await loadCache(juneScope) + // Both re-parse a different March session neither of them loaded. + p1.providers['claude']!.files['/live/mar-a.jsonl'] = fileSpanning('2026-03-20T10:00:00Z') + markCacheDirty(p1, 'claude', '/live/mar-a.jsonl') + p2.providers['claude']!.files['/live/mar-b.jsonl'] = fileSpanning('2026-03-21T10:00:00Z') + markCacheDirty(p2, 'claude', '/live/mar-b.jsonl') + await Promise.allSettled([saveCache(p1), saveCache(p2)]) + + clearLoadCacheMemo() + const final = await loadCache() + // The envelope is internally consistent and the pre-existing March session + // survived; a read-modify-write loser is re-derived by the next parse. + for (const ref of Object.values((await envelope()).providers['claude']!.shards)) { + expect(existsSync(join(sessionCacheDir(), ref.name))).toBe(true) + } + expect(final.providers['claude']!.files['/live/mar.jsonl']).toBeDefined() + expect(final.providers['claude']!.files['/live/jun.jsonl']).toBeDefined() + const landed = ['/live/mar-a.jsonl', '/live/mar-b.jsonl'].filter(p => final.providers['claude']!.files[p]) + expect(landed.length).toBeGreaterThanOrEqual(1) + }) +}) + +describe('retiring an orphaned prior layout', () => { + it('sweeps a v8 directory and a v7 file left behind by an interrupted re-layout', async () => { + await saveCache({ version: CACHE_VERSION, complete: true, providers: { + claude: { envFingerprint: 'fp', files: { '/a.jsonl': fileSpanning('2026-05-10T10:00:00Z') } }, + } }) + const v8Dir = join(TMP_DIR, 'session-cache.v8') + await mkdir(v8Dir, { recursive: true }) + await writeFile(join(v8Dir, 'envelope.json'), JSON.stringify({ version: 8, nonce: 'n', shards: {} })) + await writeFile(join(v8Dir, 'claude.abc.json'), '{}') + const v7 = join(TMP_DIR, 'session-cache.v7.json') + await writeFile(v7, '{}') + + // Fresh: an in-flight write by an older binary must be left alone. + await cleanupOrphanedTempFiles() + expect(existsSync(v8Dir)).toBe(true) + expect(existsSync(v7)).toBe(true) + + const old = new Date(Date.now() - 90 * 60 * 1000) + await utimes(join(v8Dir, 'envelope.json'), old, old) + await utimes(v7, old, old) + await cleanupOrphanedTempFiles() + expect(existsSync(v8Dir)).toBe(false) + expect(existsSync(v7)).toBe(false) + }) +}) diff --git a/tests/session-cache.test.ts b/tests/session-cache.test.ts index de8df9c0..718c0c77 100644 --- a/tests/session-cache.test.ts +++ b/tests/session-cache.test.ts @@ -21,11 +21,12 @@ import { mergeCallByDedupKey, reconcileFile, saveCache, - sessionCachePath, + sessionCacheDir, } from '../src/session-cache.js' +import { readCacheOnDisk, writeCacheOnDisk } from './fixtures/session-cache-io.js' -// Version-suffixed filename (e.g. session-cache.v5.json) the cache now writes to. -const CACHE_FILE = () => basename(sessionCachePath()) +// Version-suffixed directory (e.g. session-cache.v8) the cache now writes to. +const CACHE_DIR = () => basename(sessionCacheDir()) const TMP_DIR = join(tmpdir(), `codeburn-scache-test-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`) @@ -185,8 +186,7 @@ describe('loadCache / saveCache', () => { it('atomic write does not leave partial file on error', async () => { await saveCache(emptyCache()) - const raw = await readFile(sessionCachePath(), 'utf-8') - expect(JSON.parse(raw)).toEqual(emptyCache()) + expect(await readCacheOnDisk()).toEqual(emptyCache()) }) }) @@ -196,14 +196,15 @@ describe('versioned cache file + legacy adoption', () => { function validCache(): SessionCache { return { version: CACHE_VERSION, + complete: false, providers: { claude: { envFingerprint: 'abc123', files: { '/path/to/session.jsonl': makeCachedFile() } } }, } } - it('writes and reads the version-suffixed file, never the legacy name', async () => { - expect(basename(sessionCachePath())).toBe(`session-cache.v${CACHE_VERSION}.json`) + it('writes and reads the version-suffixed directory, never the legacy name', async () => { + expect(basename(sessionCacheDir())).toBe(`session-cache.v${CACHE_VERSION}`) await saveCache(validCache()) - expect(existsSync(sessionCachePath())).toBe(true) + expect(existsSync(sessionCacheDir())).toBe(true) expect(existsSync(join(TMP_DIR, 'session-cache.json'))).toBe(false) expect(await loadCache()).toEqual(validCache()) }) @@ -213,9 +214,9 @@ describe('versioned cache file + legacy adoption', () => { const legacy = join(TMP_DIR, 'session-cache.json') await writeFile(legacy, JSON.stringify(validCache())) - // Versioned file absent → adopt-copy from legacy on first load. + // Versioned directory absent → adopt-copy from legacy on first load. expect(await loadCache()).toEqual(validCache()) - expect(existsSync(sessionCachePath())).toBe(true) + expect(existsSync(sessionCacheDir())).toBe(true) // Legacy left intact (not deleted, not rewritten). expect(existsSync(legacy)).toBe(true) expect(JSON.parse(await readFile(legacy, 'utf-8'))).toEqual(validCache()) @@ -224,7 +225,7 @@ describe('versioned cache file + legacy adoption', () => { // file exists. const mutated: SessionCache = { version: CACHE_VERSION, providers: { codex: { envFingerprint: 'zzz', files: {} } } } await writeFile(legacy, JSON.stringify(mutated)) - expect(await loadCache()).toEqual(validCache()) + expect(await readCacheOnDisk()).toEqual(validCache()) }) it('ignores a different-version legacy file and never touches it', async () => { @@ -234,8 +235,8 @@ describe('versioned cache file + legacy adoption', () => { await writeFile(legacy, JSON.stringify(stale)) expect((await loadCache()).providers).toEqual({}) - // No versioned file adopted; legacy left byte-intact. - expect(existsSync(sessionCachePath())).toBe(false) + // No versioned directory adopted; legacy left byte-intact. + expect(existsSync(sessionCacheDir())).toBe(false) expect(JSON.parse(await readFile(legacy, 'utf-8'))).toEqual(stale) }) @@ -246,9 +247,9 @@ describe('versioned cache file + legacy adoption', () => { await writeFile(legacy, legacyContent) await saveCache(validCache()) - // The versioned file holds the new data; the legacy file is byte-untouched. + // The shards hold the new data; the legacy file is byte-untouched. expect(await readFile(legacy, 'utf-8')).toBe(legacyContent) - expect(JSON.parse(await readFile(sessionCachePath(), 'utf-8'))).toEqual(validCache()) + expect(await readCacheOnDisk()).toEqual(validCache()) }) }) @@ -820,7 +821,7 @@ describe('loadCache validation', () => { } } }, } await writeRawCache(cache) - expect((await loadCache())).toEqual(cache) + expect(await loadCache()).toEqual(cache) }) it('accepts a fully valid cache with all fields populated', async () => { @@ -845,7 +846,8 @@ describe('cleanupOrphanedTempFiles', () => { it('removes .tmp files older than 5 minutes', async () => { await mkdir(TMP_DIR, { recursive: true }) - const oldTmp = join(TMP_DIR, `${CACHE_FILE()}.abc123.tmp`) + await mkdir(join(TMP_DIR, CACHE_DIR()), { recursive: true }) + const oldTmp = join(TMP_DIR, CACHE_DIR(), 'claude.abc123.json.tmp') await writeFile(oldTmp, 'stale') const { utimes } = await import('fs/promises') const oldTime = new Date(Date.now() - 10 * 60 * 1000) @@ -858,7 +860,8 @@ describe('cleanupOrphanedTempFiles', () => { it('preserves recent .tmp files', async () => { await mkdir(TMP_DIR, { recursive: true }) - const recentTmp = join(TMP_DIR, `${CACHE_FILE()}.def456.tmp`) + await mkdir(join(TMP_DIR, CACHE_DIR()), { recursive: true }) + const recentTmp = join(TMP_DIR, CACHE_DIR(), 'claude.def456.json.tmp') await writeFile(recentTmp, 'recent') await cleanupOrphanedTempFiles() diff --git a/tests/sync-ledger-otlp.test.ts b/tests/sync-ledger-otlp.test.ts index 0b32f80d..b7c72ca6 100644 --- a/tests/sync-ledger-otlp.test.ts +++ b/tests/sync-ledger-otlp.test.ts @@ -2,7 +2,7 @@ * Unit tests for sync ledger and OTLP payload builder. */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' import { mkdtemp, rm } from 'fs/promises' import { join } from 'path' import { tmpdir } from 'os' @@ -233,17 +233,21 @@ describe('batchCalls', () => { describe('ledger', () => { let tmpDir: string const originalHome = process.env.HOME + const originalCacheDir = process.env.CODEBURN_CACHE_DIR + const originalXdgCacheDir = process.env.XDG_CACHE_HOME beforeEach(async () => { tmpDir = await mkdtemp(join(tmpdir(), 'codeburn-ledger-')) process.env.HOME = tmpDir - // env-isolation.ts redirects XDG_CACHE_HOME to a per-worker sandbox shared - // across tests — the ledger honors XDG, so point it at the per-test dir. - process.env.XDG_CACHE_HOME = join(tmpDir, '.cache') + process.env.CODEBURN_CACHE_DIR = join(tmpDir, '.cache', 'codeburn') }) afterEach(async () => { process.env.HOME = originalHome + if (originalCacheDir === undefined) delete process.env.CODEBURN_CACHE_DIR + else process.env.CODEBURN_CACHE_DIR = originalCacheDir + if (originalXdgCacheDir === undefined) delete process.env.XDG_CACHE_HOME + else process.env.XDG_CACHE_HOME = originalXdgCacheDir await rm(tmpDir, { recursive: true, force: true }) }) @@ -308,6 +312,91 @@ describe('ledger', () => { expect(clearLedger()).toBe(0) }) + it('clearLedger removes coexisting canonical and eligible legacy ledgers without adopting either', async () => { + const { clearLedger } = await import('../src/sync/ledger.js') + const { existsSync, mkdirSync, writeFileSync } = await import('fs') + const canonicalDir = join(tmpDir, '.cache', 'codeburn') + const xdgDir = join(tmpDir, 'xdg-clear-coexisting') + const legacyDir = join(xdgDir, 'codeburn') + + delete process.env.CODEBURN_CACHE_DIR + process.env.XDG_CACHE_HOME = xdgDir + mkdirSync(canonicalDir, { recursive: true }) + mkdirSync(legacyDir, { recursive: true }) + writeFileSync(join(canonicalDir, 'sync-ledger.json'), JSON.stringify([ + { key: 'canonical', ts: '2026-07-02T00:00:00Z' }, + { key: 'duplicate', ts: '2026-07-03T00:00:00Z' }, + ])) + writeFileSync(join(legacyDir, 'sync-ledger.json'), JSON.stringify([ + { key: 'legacy', ts: '2026-07-01T00:00:00Z' }, + { key: 'duplicate', ts: '2025-01-01T00:00:00Z' }, + ])) + + expect(clearLedger()).toBe(3) + expect(existsSync(join(canonicalDir, 'sync-ledger.json'))).toBe(false) + expect(existsSync(join(legacyDir, 'sync-ledger.json'))).toBe(false) + }) + + it('clearLedger attempts both targets, reports a real unlink failure, and can retry the remainder', async () => { + const fs = await import('fs') + const canonicalDir = join(tmpDir, '.cache', 'codeburn') + const canonicalPath = join(canonicalDir, 'sync-ledger.json') + const xdgDir = join(tmpDir, 'xdg-clear-retry') + const legacyDir = join(xdgDir, 'codeburn') + const legacyPath = join(legacyDir, 'sync-ledger.json') + + delete process.env.CODEBURN_CACHE_DIR + process.env.XDG_CACHE_HOME = xdgDir + fs.mkdirSync(canonicalDir, { recursive: true }) + fs.mkdirSync(legacyDir, { recursive: true }) + fs.writeFileSync(canonicalPath, JSON.stringify([ + { key: 'canonical', ts: '2026-07-02T00:00:00Z' }, + ])) + fs.writeFileSync(legacyPath, JSON.stringify([ + { key: 'legacy', ts: '2026-07-01T00:00:00Z' }, + ])) + + const attempts: string[] = [] + let failCanonicalOnce = true + vi.doMock('fs', async () => { + const actual = await vi.importActual('fs') + return { + ...actual, + unlinkSync: (path: fs.PathLike) => { + const value = String(path) + attempts.push(value) + if (value === canonicalPath && failCanonicalOnce) { + failCanonicalOnce = false + throw Object.assign(new Error('injected canonical unlink failure'), { code: 'EACCES' }) + } + return actual.unlinkSync(path) + }, + } + }) + vi.resetModules() + + try { + const { clearLedger } = await import('../src/sync/ledger.js') + expect(() => clearLedger()).toThrow('injected canonical unlink failure') + expect(attempts).toContain(canonicalPath) + expect(attempts).toContain(legacyPath) + expect(fs.existsSync(canonicalPath)).toBe(true) + expect(fs.existsSync(legacyPath)).toBe(false) + expect(JSON.parse(fs.readFileSync(canonicalPath, 'utf8'))).toEqual([ + { key: 'canonical', ts: '2026-07-02T00:00:00Z' }, + ]) + + // The successful legacy deletion is not replayed or migrated. Retrying + // removes only the canonical remainder; its missing peer is ENOENT-safe. + expect(clearLedger()).toBe(1) + expect(fs.existsSync(canonicalPath)).toBe(false) + expect(fs.existsSync(legacyPath)).toBe(false) + } finally { + vi.doUnmock('fs') + vi.resetModules() + } + }) + it('corrupt ledger file reads as empty (crash-safe recovery)', async () => { const { readLedger } = await import('../src/sync/ledger.js') const { mkdirSync, writeFileSync } = await import('fs') @@ -328,21 +417,168 @@ describe('ledger', () => { expect(existsSync(join(dir, 'sync-ledger.json.tmp'))).toBe(false) }) - it('honors XDG_CACHE_HOME when set', async () => { + it('honors CODEBURN_CACHE_DIR at call time', async () => { const { writeLedger, readLedger } = await import('../src/sync/ledger.js') const { existsSync } = await import('fs') const { join } = await import('path') - const xdgDir = join(process.env.HOME!, 'xdg-cache') - const original = process.env.XDG_CACHE_HOME + const firstDir = join(tmpDir, 'cache-a') + const secondDir = join(tmpDir, 'cache-b') + + process.env.CODEBURN_CACHE_DIR = firstDir + writeLedger([{ key: 'first', ts: '2026-07-01T00:00:00Z' }]) + process.env.CODEBURN_CACHE_DIR = secondDir + writeLedger([{ key: 'second', ts: '2026-07-02T00:00:00Z' }]) + + expect(existsSync(join(firstDir, 'sync-ledger.json'))).toBe(true) + expect(existsSync(join(secondDir, 'sync-ledger.json'))).toBe(true) + expect(readLedger().map(e => e.key)).toEqual(['second']) + + process.env.CODEBURN_CACHE_DIR = firstDir + expect(readLedger().map(e => e.key)).toEqual(['first']) + }) + + it('uses the shared default when both overrides are explicitly absent', async () => { + const { writeLedger } = await import('../src/sync/ledger.js') + const { existsSync } = await import('fs') + const { join } = await import('path') + + delete process.env.CODEBURN_CACHE_DIR + delete process.env.XDG_CACHE_HOME + writeLedger([{ key: 'default', ts: '2026-07-01T00:00:00Z' }]) + + expect(existsSync(join(tmpDir, '.cache', 'codeburn', 'sync-ledger.json'))).toBe(true) + }) + + it('adopts an XDG-only legacy ledger into the canonical default and writes there thereafter', async () => { + const { appendToLedger, readLedger } = await import('../src/sync/ledger.js') + const { existsSync, mkdirSync, readFileSync, writeFileSync } = await import('fs') + const { join } = await import('path') + const xdgDir = join(tmpDir, 'xdg-cache') + const legacyDir = join(xdgDir, 'codeburn') + const canonicalDir = join(tmpDir, '.cache', 'codeburn') + + delete process.env.CODEBURN_CACHE_DIR process.env.XDG_CACHE_HOME = xdgDir - try { - writeLedger([{ key: 'xdg-entry', ts: '2026-07-01T00:00:00Z' }]) - expect(existsSync(join(xdgDir, 'codeburn', 'sync-ledger.json'))).toBe(true) - expect(readLedger().map(e => e.key)).toEqual(['xdg-entry']) - } finally { - if (original === undefined) delete process.env.XDG_CACHE_HOME - else process.env.XDG_CACHE_HOME = original - } + mkdirSync(legacyDir, { recursive: true }) + writeFileSync(join(legacyDir, 'sync-ledger.json'), JSON.stringify([ + { key: 'legacy', ts: '2026-07-01T00:00:00Z' }, + ])) + + expect(readLedger().map(entry => entry.key)).toEqual(['legacy']) + expect(existsSync(join(canonicalDir, 'sync-ledger.json'))).toBe(true) + expect(existsSync(join(legacyDir, 'sync-ledger.json'))).toBe(false) + + appendToLedger([{ key: 'canonical', ts: '2026-07-02T00:00:00Z' }]) + expect(JSON.parse(readFileSync(join(canonicalDir, 'sync-ledger.json'), 'utf8')).map((entry: { key: string }) => entry.key)).toEqual([ + 'legacy', + 'canonical', + ]) + expect(existsSync(join(legacyDir, 'sync-ledger.json'))).toBe(false) + }) + + it('does not adopt a legacy XDG ledger when CODEBURN_CACHE_DIR is explicitly set', async () => { + const { readLedger, writeLedger } = await import('../src/sync/ledger.js') + const { existsSync, mkdirSync, writeFileSync } = await import('fs') + const explicitDir = join(tmpDir, 'explicit-cache-precedence') + const xdgDir = join(tmpDir, 'xdg-cache-precedence') + const legacyDir = join(xdgDir, 'codeburn') + + process.env.CODEBURN_CACHE_DIR = explicitDir + process.env.XDG_CACHE_HOME = xdgDir + mkdirSync(legacyDir, { recursive: true }) + writeFileSync(join(legacyDir, 'sync-ledger.json'), JSON.stringify([ + { key: 'legacy', ts: '2026-07-01T00:00:00Z' }, + ])) + + expect(readLedger()).toEqual([]) + writeLedger([{ key: 'explicit', ts: '2026-07-02T00:00:00Z' }]) + + expect(readLedger().map(entry => entry.key)).toEqual(['explicit']) + expect(existsSync(join(explicitDir, 'sync-ledger.json'))).toBe(true) + expect(existsSync(join(legacyDir, 'sync-ledger.json'))).toBe(true) + }) + + it('merges an XDG legacy ledger into an existing canonical ledger once', async () => { + const { readLedger } = await import('../src/sync/ledger.js') + const { existsSync, mkdirSync, writeFileSync } = await import('fs') + const canonicalDir = join(tmpDir, '.cache', 'codeburn') + const xdgDir = join(tmpDir, 'xdg-cache-merge') + const legacyDir = join(xdgDir, 'codeburn') + + delete process.env.CODEBURN_CACHE_DIR + process.env.XDG_CACHE_HOME = xdgDir + mkdirSync(canonicalDir, { recursive: true }) + mkdirSync(legacyDir, { recursive: true }) + writeFileSync(join(canonicalDir, 'sync-ledger.json'), JSON.stringify([ + { key: 'canonical', ts: '2026-07-02T00:00:00Z' }, + { key: 'duplicate', ts: '2026-07-03T00:00:00Z' }, + ])) + writeFileSync(join(legacyDir, 'sync-ledger.json'), JSON.stringify([ + { key: 'legacy', ts: '2026-07-01T00:00:00Z' }, + { key: 'duplicate', ts: '2025-01-01T00:00:00Z' }, + ])) + + expect(readLedger()).toEqual([ + { key: 'canonical', ts: '2026-07-02T00:00:00Z' }, + { key: 'duplicate', ts: '2026-07-03T00:00:00Z' }, + { key: 'legacy', ts: '2026-07-01T00:00:00Z' }, + ]) + expect(existsSync(join(legacyDir, 'sync-ledger.json'))).toBe(false) + // A later read is canonical-only and stable; XDG is no longer active. + expect(readLedger().map(entry => entry.key)).toEqual(['canonical', 'duplicate', 'legacy']) + }) + + it('prefers non-empty CODEBURN_CACHE_DIR over XDG_CACHE_HOME', async () => { + const { writeLedger } = await import('../src/sync/ledger.js') + const { existsSync } = await import('fs') + const { join } = await import('path') + const explicitDir = join(tmpDir, 'explicit-cache') + const xdgDir = join(tmpDir, 'xdg-cache') + + process.env.CODEBURN_CACHE_DIR = explicitDir + process.env.XDG_CACHE_HOME = xdgDir + writeLedger([{ key: 'explicit', ts: '2026-07-01T00:00:00Z' }]) + + expect(existsSync(join(explicitDir, 'sync-ledger.json'))).toBe(true) + expect(existsSync(join(xdgDir, 'codeburn', 'sync-ledger.json'))).toBe(false) + }) + + it.each(['', ' '])('ignores empty CODEBURN_CACHE_DIR %j and writes to the canonical default', async explicit => { + const { writeLedger } = await import('../src/sync/ledger.js') + const { existsSync } = await import('fs') + const { join } = await import('path') + const xdgDir = join(tmpDir, `xdg-cache-${explicit.length}`) + + process.env.CODEBURN_CACHE_DIR = explicit + process.env.XDG_CACHE_HOME = xdgDir + writeLedger([{ key: 'xdg-fallback', ts: '2026-07-01T00:00:00Z' }]) + + expect(existsSync(join(tmpDir, '.cache', 'codeburn', 'sync-ledger.json'))).toBe(true) + expect(existsSync(join(xdgDir, 'codeburn', 'sync-ledger.json'))).toBe(false) + }) + + it.each(['', ' '])('ignores empty XDG_CACHE_HOME %j and uses the shared default', async xdg => { + const { writeLedger } = await import('../src/sync/ledger.js') + const { existsSync } = await import('fs') + const { join } = await import('path') + + delete process.env.CODEBURN_CACHE_DIR + process.env.XDG_CACHE_HOME = xdg + writeLedger([{ key: 'default-fallback', ts: '2026-07-01T00:00:00Z' }]) + + expect(existsSync(join(tmpDir, '.cache', 'codeburn', 'sync-ledger.json'))).toBe(true) + }) + + it.each(['', ' '])('uses the shared default when both overrides are empty and CODEBURN is %j', async explicit => { + const { writeLedger } = await import('../src/sync/ledger.js') + const { existsSync } = await import('fs') + const { join } = await import('path') + + process.env.CODEBURN_CACHE_DIR = explicit + process.env.XDG_CACHE_HOME = ' ' + writeLedger([{ key: 'default-fallback', ts: '2026-07-01T00:00:00Z' }]) + + expect(existsSync(join(tmpDir, '.cache', 'codeburn', 'sync-ledger.json'))).toBe(true) }) }) diff --git a/tsup.config.ts b/tsup.config.ts index c20e55c9..94f7ccec 100644 --- a/tsup.config.ts +++ b/tsup.config.ts @@ -1,7 +1,7 @@ import { defineConfig } from 'tsup' export default defineConfig({ - entry: ['src/main.ts'], + entry: ['src/main.ts', 'src/parse-worker.ts'], format: ['esm'], target: 'node20', outDir: 'dist',