The comment at the install site claimed only that an overlapping worker result
'is discarded'. State why the empty-set result is installable at all — an empty
id intersection is proof a serial parse would have dropped nothing — and why the
tempting shortcut is wrong: parsedTurnsToCachedTurns delta-encodes gitBranch
across turns, so dropping one turn changes whether a LATER turn carries a
gitBranch key. Overlap discards the whole file, never individual turns.
Tests: the end-to-end determinism check now runs both parses over the SAME
corpus, so cache shard BODIES are compared byte for byte instead of just their
keys, and a new resumed-session fixture (a transcript restating another file's
message ids, in both filename orders) makes install order decide the answer.
Verified by mutation: removing the discard guard fails it, and yielding worker
results out of order fails it.
CODEBURN_VERBOSE now reports how many worker results were re-parsed in-process
on id overlap, which is what the new test asserts on. The worker bundle's source
map is excluded from the published package (-1.8 MB).
os.freemem() reports free pages on macOS, not available memory: on an idle
128 GB machine it reads a few hundred MB, so the 2 GB gate switched the worker
pool on and off between runs on the platform the desktop app ships to. The gate
and the budget now use process.availableMemory() (cgroup/rlimit-aware in a
container), falling back to os.totalmem(): serial under 4 GB available, budget
min(0.25 * available, 2 GB). An 8 GB box earns 8 threads, a 4 GB box none.
The verbose line now carries every decision input — cores, available GB, pending
files and bytes — on both the gate and the go path, so one support log explains
itself.
Pins the gates that keep threads off low-spec machines and warm runs, that a
forced worker count bypasses them, that results come back in submission order,
that a dead pool reports failure instead of throwing (and the serial fallback
lands on the same result), that no thread outlives a parse across back-to-back
parses, and that a cold CLI parse with and without workers produces the same
payload and the same cache shards.
A file's shard span was read off turns[0]/turns[-1], but several providers
emit turns non-chronologically (cursor by ROWID, goose/crush/copilot by a
DESC ordering). That produced until < bucket - an empty span, so the shard
was unreachable at every scope and its sessions re-parsed every run.
cacheFileSpan now takes the min and max month over all turns.
An entry re-bucketing out of a month the run never loaded (a re-parse that
moved its oldest turn, or the #441 failure marker that has no turns at all)
left the old copy in the carried shard, so one path lived in two shards and
a later load could resolve to the stale one. A save now prunes those paths
from the shards it carries, and a load merges shards in envelope order,
resolving any duplicate to the freshest fingerprint and dirtying both
buckets so the next save retires the loser.
A carried month whose shard another writer had republished was dropped from
the envelope outright, losing expired-transcript PR orphans no re-parse can
recover. The envelope is now re-read just before publishing and the current
shard name adopted; a ref is dropped only when that envelope lacks it too.
The same re-read moves every merge read after the ownership fence and gives
the merge one optimistic retry, so the read-modify-write window shrinks to
the publish itself.
Also: retire an orphaned v8 directory / v7 file left by an interrupted
re-layout, age-guarded, once a v9 envelope is published.
A memo hit must not re-render. The generated stamp is minted per render,
so asserting it is unchanged across the hit states that contract directly
instead of leaving it implied by byte equality.
scanProjectDirs ran every cached turn through cachedTurnToClassified —
per-call reconstruction plus the turn classifier's category / retries /
edit regexes — and only then applied the date slice, so a week view paid
to classify all of history to keep a few percent of it.
The keep/drop decision now runs on the raw CachedTurn (calls map 1:1 onto
assistantCalls, so callsInRange sees the same survivors as the classified
slicer) and only survivors are classified, still from their complete call
list. The branch and PR-set carries still walk the full ordered turn list,
and buildSpawnPrSets still reads the pre-slice turns.
Real corpus, warm one-shot status --format menubar-json --no-optimize,
identical payloads: today 1561ms -> 1318ms, week 1560ms -> 1410ms,
month 1788ms -> 1675ms.
Two live processes share one cache directory routinely (a one-shot CLI
beside the resident serve child, two menubar polls), and the shard layout
had two ways to lose data there.
- The atomic write used a FIXED temp name, so two writers publishing the
envelope — every save does — shared one `envelope.json.tmp` and
interleaved into a torn payload, or one deleted the shards the other's
envelope named. 39 of 40 rounds ended in a total cache loss. The temp
name carries a nonce again, as it did before the shard layout.
- A save reused a shard filename from its own load snapshot without
checking the file was still there. Another process republishing that
provider unlinks the old shard, so the stale writer published an
envelope naming a deleted file — read back as a corrupt provider and
dropped whole, including PR-linked orphans no re-parse can recover. A
reused shard is now existence-checked, and re-verified once more
immediately before the envelope is published; a vanished one is
rewritten from memory.
Also:
- Progress saves take a 30s floor beside the file counter. Only the
claude scan reports per file; every other provider calls saveProgress
once at its own boundary, so the counter alone never fired there.
- The unreferenced-shard sweep waits an hour (temps still 5 minutes): an
unreferenced shard may belong to a concurrent save whose envelope has
not landed yet.
- The sweep also retires the pre-v8 single-file temps in the parent
directory, which nothing writes anymore.
- The shard directory is created 0o700.
- The claude and provider paths mark the cache dirty where they DELETE a
stale entry, not only where they replace it: an unreadable file skips
the replace, and the deletion would otherwise live only in memory.
- Codex only treats a grown file as an append when the recorded boundary
still lands just after a newline, so a same-inode rewrite that happens
to end up larger re-parses instead of resuming mid-line.
Codex rollout files are append-only and the active ones run to hundreds
of MB, but any growth re-read the file from byte 0 because the cache
keyed only on mtime+size. The parser now records a restart point at every
task_started boundary — the byte offset plus the state the single-pass
decode carries across it — and a grown file with the same dev/ino picks
up from there.
The boundary sits at the task_started line itself, so the task it opens
is re-decoded from the tail; the entry stores how many calls were decoded
before that point so the resumed run starts from exactly those and cannot
double-count the open task. An unusable or absent snapshot falls back to
a full re-parse.
CODEX_CACHE_VERSION is deliberately not bumped: the new fields are
additive and absence-safe both ways, so a bump would discard a warm
cache for nothing.
A warm launch rewrote the entire session cache whenever any provider
appended a few KB: on a 6 GB corpus that is a 155 MB stringify + fsync
every run. The on-disk cache is now a version-suffixed directory holding
one shard per provider plus a small envelope, and a save rewrites only
the providers marked dirty.
- Dirtiness is tracked per provider (markCacheDirty) instead of one
global flag, so an appended Claude session no longer republishes
Codex, Copilot and the rest.
- Shards carry a nonce in their filename and the envelope is renamed
last, so a save is published at a single point: readers never see a
half-updated set, and a writer that loses the refresh ownership fence
leaves the canonical shards untouched.
- A shard that fails validation is treated as an absent provider rather
than rejecting the whole cache, so one malformed turn costs one
provider's re-parse instead of every provider's history.
- v7 migrates losslessly: the blob is re-laid-out into shards and
removed only once that save publishes. Nothing re-parses.
- Cold-parse progress saves now trigger every N files parsed rather than
every 5s, so a slow cold parse no longer rewrites the growing cache on
a wall clock.
The separator regex retried its leading \s* from every offset, which is
quadratic on the long whitespace-heavy commands agents emit; that one regex
was ~30% of a warm status run on a multi-GB corpus. Match the separator
alone and widen over whitespace by hand. Output is unchanged (differential
check over 42k real commands).
Clear the per-directory Codex and Antigravity memo maps in the resident RSS
guard; document the single cache-dir rule (XDG_CACHE_HOME no longer
consulted, ledger migrated); stop output-overflow terminations from spending
the resident's unexpected-death budget.
flatSlice returned strings within the bound unchanged, but provider adapters
pre-truncate with .slice(0, 500) before the cache site, so those views still
pinned their parent buffers. Always flatten; the round-trip is ~150ns per turn.
Use utf16le so lone surrogates survive the copy.
Cache the canonical-path Promise instead of the resolved value so calls in one
Promise.all batch share a single walk.
Document the one-time kiro re-parse and worktree regrouping.
The kiro provider was reading the full working directory from session
metadata (meta.cwd for CLI sessions, workspacePaths[0] for v2 IDE
sessions, workspaceDirectory for workspace sessions) but discarding it
via basename(), keeping only the leaf name for display. This meant
computeAttributionRecords could never resolve kiro sessions to a git
repo, so `codeburn sync push --attribution` produced 0 facts for all
kiro-originated sessions.
Now passes the full path as projectPath on emitted ParsedProviderCalls,
which buildRepoGroups uses to resolve git identity and correlate
commits with sessions via timestamp windows. The path stays local:
only the normalized origin remote egresses in attribution spans.
Behavior changes beyond attribution:
- kiro calls now flow through canonicalizeProviderCallProject, so kiro
sessions in LINKED GIT WORKTREES canonicalize to the main repository:
their report project name changes from the worktree dir name to the
main repo name (consistent with claude/codex behavior).
- workingDirectory is now populated on kiro calls.
- PROVIDER_PARSE_VERSIONS.kiro bumped (project-path-v1): cached entries
predate projectPath and are served without re-invoking the parser, so
without the bump this fix silently no-ops for every warm cache. The
bump forces a one-time cold kiro re-parse on upgrade.
ORDERING: this commit must land WITH (or after) the preceding
SlicedString OOM fix. The forced cold re-parse it triggers is exactly
the workload that OOM'd before that fix on multi-GB kiro stores.
Perf: per-call canonicalization added a measured +5% to cold parse
(.git-marker lstat walk per call). resolveCanonicalProjectPath is now
memoized on cwd (cleared with the session cache), removing the
redundant walks for all providers.
Tests: projectPath emission fixtures for all three session formats
(CLI, v2 IDE, workspace-session), fingerprint-change assertion, and a
regression test seeding a pre-bump cache entry and proving the re-parse
recovers projectPath.
AI-Origin: human
String.prototype.slice returns a V8 SlicedString: a view that retains a
reference to its ENTIRE parent string. The parsers store short previews
of message text (userMessage.slice(0, 500/2000)) in the long-lived
session cache. Session files routinely carry 100KB+ strings (agent-
injected system prompts, tool results), so every cached preview pinned
its full parent buffer for the life of the process.
Measured on 3.2GB of kiro CLI session files (6,659 files, largest 40MB):
cold parse, default heap, before: 4.33GB peak -> OOM crash
cold parse, 8GB heap, before: 5.67GB peak (kiro provider alone)
after kiro flatSlice: 0.91GB peak
after parser.ts cache sites too: 0.64GB peak
original failing command (cold,
default heap, all providers): 0.89GB peak -> completes
Warm runs were always fine (~0.29GB) because the cache's JSON round-trip
flattens the strings on load — which made this bug appear intermittent:
it only fired on a cold or invalidated cache.
Fix: flatSlice() in content-utils.ts forces a flat copy via Buffer
round-trip. Applied at the six kiro userMessage capture sites and the
three shared cache-building sites in parser.ts (protects all providers).
Regression test asserts the no-retention property via bounded heap
growth over 1000 large-parent slices.
AI-Origin: human
cli-json-daily, spend-flow and cli-emitters spawn the real CLI per test
and blow the 5s default under full parallel suite load while passing in
isolation - the flake set #948 documented on unmodified main, observed
again locally (cli-json-daily) and in CI (cli-emitters on a green PR).
Same file-level remedy the CLI menubar suite already uses; the default
stays 5s for everything else.
The cwd evidence rule attributed ANY session sharing a checkout with a
PR-linked session, with no time bound - so a repo whose only captured PR
link was pasted once became a black hole: 129 of 131 sessions and a
month of unrelated work (~$7.4K direct, $11.2K displayed) attributed to
one PR, observed live on the desktop Pull requests tab.
Cwd anchors now carry the evidence sessions' own activity window (union
across evidence for the same PR set), and only sessions overlapping that
window plus a 6h pad inherit the PR. The rule's charter is 'a tool
session launched around PR work in this checkout', which is inherently a
same-working-stretch claim; the design's own philosophy (timestamps
narrow, never create) now applies to this rule too.
On the real corpus the row corrected to $450.87 / 21 sessions across the
PR's actual two-day working stretch. Regression pins both directions:
nearby same-cwd session inherits, weeks-later one never does; multiple
evidence sessions widen the window.
The remaining warm-serve cost was the per-request discovery sweep
(stat-ing thousands of session files) plus re-aggregation, even when
nothing on disk had changed. Serve now watches every provider's
probeRoots() via fs.watch (FSEvents-backed recursive watches on macOS)
and injects a quiet-since validator into the parser: while the watched
roots are quiet, a previous parse stays reusable past the burst window,
and an output-level memo returns identical panel queries verbatim - so a
fetch with no data changes skips the sweep AND the aggregation.
Safety rails, in order: a parse is validated-reusable only if the
watchers were armed before it ran; any filesystem event ends reuse
instantly; a 5-minute hard cap self-heals a missed event; a root that
fails to watch just goes uncovered (shorter reuse, never staleness);
outside serve the validator is never installed and behavior is
byte-identical. During an active AI session the session roots fire
constantly, so reuse correctly stays inside the 10s burst window - the
extended reuse serves the idle-browsing case it was built for.
The one watched path inside the cache dir is antigravity's statusline
file specifically, so serve's own cache writes never self-invalidate.
Adversarial review of the serve design surfaced three weaknesses a
one-shot CLI never had, because it never lived long enough:
- Pricing-affecting config (model aliases, price overrides, local-model
savings) now participates in the parse memo key. Config reloads fresh
per request (the preAction hook), but a memoized or burst-reused parse
embedded costs priced under the OLD config; the widened key makes any
such change an automatic memo miss. New alias-hash helper + tests.
- Memory guard: past 3GB RSS the serve loop drops its in-memory memos
(session cache + parse entries) and the next request re-parses once.
The child never exits for this, so the client's death budget is
untouched.
- codeburn serve typed in an interactive terminal now explains itself on
stderr instead of hanging silently on stdin.
Every CLI spawn on a large corpus pays seconds of fixed cost before any
query work: node boot, a 100MB+ session-cache JSON.parse, the discovery +
fingerprint sweep, and serve-time classification. The desktop app spawns
one CLI per panel fetch, so it pays that cost per panel.
codeburn serve --stdio is the same CLI kept warm: the app holds one child,
sends {id, args} per line, and gets the command's stdout back. Three layers
make it fast, each disabled outside serve so one-shot runs stay byte-exact:
- loadCache memo (session-cache.ts): the parsed cache object is reused
while a stat() shows the file unchanged; saveCache updates it
write-through. A rewrite by another process still forces a fresh read.
- burst reuse (parser.ts, CODEBURN_PARSE_BURST_MS, serve sets 10s): panel
bursts anchor their range ends at their own new Date(), so the exact-key
memo never hits in real traffic; within the window a re-anchored range is
served by trimming the previous parse instead of re-running discovery.
- fresh commander program per request (main.ts buildProgram factory),
because commander option state is sticky across parses.
The server allows only the app's read queries (status/overview/models/
sessions/compare/yield/spend/optimize/audit), refuses everything else
(client falls back to a spawn), serializes requests, and converts
process.exit into a caught signal. The app starts the child once at
startup; requests route through it only when warm, cold-start keeps the
spawn path with its progress events, any serve failure falls back to a
spawn, and three child deaths disable serve for the app run.
Measured on a real 17B-token corpus: panel fetches drop from ~7.4s per
spawn to 5-900ms warm (sessions/spend 5ms, status 898ms). One-shot CLI
output verified byte-identical against the pre-branch baseline.
Improve dashboard refresh coordination, scrolling, responsive layout, and data-density behavior, including the Windows-safe resize correction validated on the final head.
Add doctor probeRoots coverage for the remaining fixed-location providers while keeping discovery and diagnostics on the same shared root-resolution logic.
Round 2 of the independent review proved five things by mutation: it broke the
behavior and the tests stayed green. Every one is now pinned.
The most important invariant in this change was the least guarded. Copilot must
have NO entry in PROVIDER_ENV_VARS - declaring any of its nine reads moves its
fingerprint and re-opens the durable history-loss path - but only one of the
nine was covered, so declaring any of the other eight passed the whole suite.
Now the absence of the entry is asserted directly, and all nine vars are
table-tested for fingerprint stability.
Doctor stops blaming parse-only overrides for a failed discovery.
CODEBURN_CURSOR_MAX_BUBBLES caps how many bubbles Cursor parses and
KIMI_MODEL_NAME renames an attributed model; neither relocates anything, so
"NOTHING FOUND (override CODEBURN_CURSOR_MAX_BUBBLES set...)" pointed the user
at the wrong thing. Both join NON_DISCOVERY_ENV_VARS, which exists for exactly
this, and both still appear in Details - only the verdict's blame line changes.
The secret-redaction and ambient-suppression tests are table-driven over both
names each covers, since removing either second name (VERCEL_OIDC_TOKEN,
LOCALAPPDATA) previously leaked or surfaced it with every test still passing.
The changelog no longer claims a one-time re-parse for the Vercel gateway: it
is a network provider re-fetched on every writable run, so its declaration is a
read-only-path correction, not a migration. Fourteen file-backed providers
migrate once.
Five findings from a cross-model review of the previous two commits, each
verified on the code before acting:
Copilot is no longer declared. Declaring anything for it changes its
fingerprint, and getOrCreateProviderSection keeps only cached entries whose
source path is gone - but OTel discovery returns one source per DB file
(copilot.ts:1935) and that DB keeps existing, so the entry would be dropped and
re-parsed, destroying conversations Copilot has since pruned from the DB that
only the cache still holds. Trading a staleness bug for a data-loss bug is a
bad trade; copilot waits for the durable carry-forward to merge instead of
drop, and its reads are allowlisted with that reason.
The Vercel gateway credentials ARE declared, reversing the previous commit's
reasoning, which was wrong: servedSources is seeded with every discovered
source (parser.ts:2875) before the network branch, and the network re-fetch
(parser.ts:2888) only runs when !readOnly, so a read-only refresh serves the
cached report and an undeclared credential keeps reporting the previous
account's usage after a swap. Doctor redacts credential values so a key can
never reach terminal output or the JSON report.
AMBIENT_ENV_VARS narrows to APPDATA and LOCALAPPDATA. Windows sets those for
every process so they carry no intent, but the XDG vars are opt-in and do:
suppressing them made doctor answer a deliberately relocated XDG_DATA_HOME with
"tool likely not installed", which is worse than the noise it avoided.
The guard's allowlist is keyed by file and var, not var alone - a var
allowlisted for one file silenced every other file's undeclared read of it.
Cursor drops its stale XDG_DATA_HOME declaration, which it never reads; its
fingerprint already changes here, so this costs no extra migration.
cursor-agent keeps its equally stale one, since removing it would force a
re-parse to fix nothing.
The nine undeclared overrides in #920 all slipped through the same way: the
declaration lives in one file and the read in another, and nothing tied them
together. Add the static guard the issue asked for - every process.env read in
src/providers is either declared in PROVIDER_ENV_VARS for the provider(s) that
file serves, or allowlisted with a reason. It resolves bracket literals, dot
access and `process.env[CONST]` indirection (open-design's ENV_DIR), and fails
loudly on any read it cannot resolve to a name rather than skipping it, since a
silently skipped read is how this class of defect survives. A read-bearing
provider file missing from the file-to-provider map fails too, so a new
provider cannot join without being mapped. A second assertion catches a
PROVIDER_ENV_VARS key that is not a registered provider name, which declares
nothing and fails just as silently.
Plus the direct regression: each of the nine reported (provider, var) pairs
must move the fingerprint, with codex/CODEX_HOME as the control the issue used,
and the round trip asserted so the hash stays a pure function of the
environment.
Reconcile with #914 (which independently fixed the same date-sensitive
fixtures): keep this branch's equivalent date fixes for parser.test.ts and
cli-durable-totals.test.ts, and drop the global vitest retry #914 added now
that this branch fixes the flakes at the root (fs.rm retries, longer timeouts,
serial cache-lock CI step). The targeted cache-lock local retries stay.
Hermes, Cursor, OpenCode and copilot OTel sources all live in SQLite
databases that their agents keep open in WAL mode for the life of the
process. Committed writes park in <db>-wal until a checkpoint, so the
main file's stat can sit hours or days behind the newest committed data.
fingerprintFile only statted the main file, which broke two ways:
- The date-range mtime pre-filter in parseProviderSources read the stale
mtime as "nothing in range" and skipped the source entirely. Every
Hermes session committed after the last checkpoint vanished from
reports: the today-parse skipped the db (mtime < local midnight) while
the backfill only keeps days through yesterday. Exactly the "17
sessions in the DB, 14 reported, the 3 from today missing" report in
issue #913.
- reconcileFile saw an unchanged fingerprint between checkpoints and kept
serving stale cached turns for sessions that had since grown.
Fold the -wal sibling into the fingerprint: newest mtime wins and sizes
add, so both WAL growth and a checkpoint (db grows, wal truncates) move
the fingerprint. -shm is deliberately ignored (it mutates on reads).
Bare SQLite paths get the fold only when the extension says database, so
JSONL transcript fingerprints (offset-based append detection) are
untouched.
Refs #913
parser.test.ts (a)/(f): createJsonlSession stamped events at a fixed 2026-05-01
that aged past the 90-day retention window, pruning to zero; date them relative
to now. cli-durable-totals: seedLiveTodaySession stamped noon, which is in the
future on a pre-noon run so the provider-scoped today slice (ends at now)
dropped it while the all path (ends at range end) kept it; seed a past-today
time. cache-refresh-lock and other integration tests starve under a saturated
parallel run and fail closed; add a small global retry and raise the two most
load-sensitive lock tests. Test-only; no production code changed.
