The Optimize scan does real fs I/O (readdir/stat) that only resolves on a
real event-loop turn, but the wait loop counted 20 vi.advanceTimersByTimeAsync
hops under full fake timers, which flush fake timers + microtasks but never
give real I/O a chance to complete. Under load that read a stale
"Scanning Today..." frame. Scope fake timers to just what the 60s
auto-refresh interval needs, leave setImmediate/Date real, and wait on a
real wall-clock deadline instead of a fixed hop count.
* fix(optimize): scope transcript-derived findings to the selected provider
scanSessions() always ran discoverAllSessions('claude'), so every finding it
feeds was computed from Claude transcripts regardless of --provider, while the
header (sessions, calls, cost) came from the already-filtered projects. Under
--provider codex the two described different providers, and the Claude-derived
numbers read as the selected provider's.
Skip the scan when the filter excludes Claude, and skip the detectors it feeds
rather than handing them an empty scan: emptiness reads as "never invoked", so
an empty scan turned every skill, agent and command into a reported ghost.
Findings derived from projects (MCP tool coverage, capability reliability,
low-worth sessions, context bloat, outliers, model recommendations) already
filter correctly and still run.
The result cache key now carries the provider, since the provider decides
whether the scan runs at all.
* fix(optimize): thread the provider through the apply, aggregator and TUI scans
The previous commit fixed one of four scanAndDetect callers. The other three
carry a provider filter and dropped it:
- act/optimize-apply.ts: `optimize --apply` branches in main.ts before the
code that threads it, so a Codex-scoped run planned applies off Claude
findings. This is the worst of the three because `unused-skills` is
appliable and its plan moves directories out of ~/.claude/skills.
- usage-aggregator.ts: AggregateOpts.provider was already honoured for the
usage half but not for the optimize half, so the menubar, desktop and web
surfaces carried the same mismatch.
- dashboard.tsx: `p` cycles activeProvider and `o` opens optimize off the
same state, so the TUI could show Claude findings under a Codex view.
activeProvider joins the callback deps; reloadData already clears
optimizeResult on a provider switch, so no extra invalidation is needed.
Covered by a dry-run test on the apply path: under `provider: 'codex'` a fake
home holding an uninvoked skill must plan nothing, and under 'claude' the same
fixture must still plan the archive.
Security audit follow-ups on the DeepSeek Harness provider.
- **Decompression bomb.** Every zstd frame was decoded with no output bound, so
a 16 KB crafted log expanded to ~916 MB of RSS (a 65 KB one declares 2 GB).
Each frame now decodes under a 64 MB per-call cap, and the caps chain into a
running per-file budget of MAX_SESSION_FILE_BYTES: a frame is given only the
bytes the file has left, so node throws ERR_BUFFER_TOO_LARGE without
allocating past the cap. The throw propagates out of the existing skip path,
which discards the whole file rather than counting the frames read before the
bomb, so a crafted tail cannot poison a partial total. The discovery header
read takes the same per-frame cap. Measured on a 65 KB / 2 GB bomb: 916 MB
-> 67 MB peak, zero calls emitted, one notice.
Lines are still materialized eagerly; the byte budget bounds that, and making
the read lazy would change readEventLines' contract for no further bound.
- **Usage type confusion.** Token fields were read with `?? 0` and never
type-checked, so a string or array inputTokens flowed into the global totals
and the persisted cache, where `0 + [1, 2]` becomes "01,2". They now go
through numberOrZero (copilot.ts semantics: finite, positive, else 0).
All-zero calls are still skipped.
- **Snap over-scope.** The personal-files read entry is `$HOME/.dsh/sessions`
rather than all of `$HOME/.dsh`; the provider reads nothing else.
- **Third-party notice.** scanZstdFrames is transcribed from
@deepseek-ai/dsh-session-persistence-jsonl. The published npm package is
BSD-3-Clause (Copyright (c) 2026, DeepSeek) while the monorepo source
declares MIT for the same package; THIRD_PARTY_NOTICES.md reproduces the
stricter of the two and ships via package.json `files`.
zstd landed in node:zlib in 22.15; the package floor and CI pin are 22.13, so
the runtime already degrades with a notice there. The tests compressed
fixtures at runtime and failed outright. Fixture writes now fall back to
plain jsonl below 22.15 so parsing semantics still run, container-specific
tests skip, and the Tests workflow runs on both 22.13.0 and latest 22.x.
Reviewed src/providers/dsh.ts against deepseek-harness @ 99f6f02f and fixed
what the format says but the parser did not:
- A forked session's log replays its parent's events verbatim, and codeburn
parses the parent's own log as its own session, so every inherited call was
billed twice. The header's parentSession + seedLength mark that prefix;
events with seq < seedLength are now skipped.
- The model now comes from the reporting assistant/message's own
message.source, which is what actually served the step. request/header only
describes the request DSH was about to make, and is the fallback.
- user/message also carries agent-injected context (runtime snapshots, skill
bodies) under source.kind 'plugin'; only a typed prompt becomes the preview,
and it is bounded to 500 chars like every other provider rather than holding
a whole injected system prompt per turn.
- A log stamped with a session format version other than 0 is skipped with a
notice. The format is pinned at 0 upstream with no compatibility implied, so
reading a bumped format under today's assumptions would report confident
wrong numbers.
- Timestamps go through the seconds-vs-milliseconds guard and fall back to the
header createdAt, so a call can no longer carry an empty timestamp and land
in the undated cache shard.
- The compressed read buffers the whole log to scan its frames, so it now takes
the same oversize guard readSessionFile applies to the uncompressed variant.
- The zstd-unavailable notice fired once per session log; each distinct notice
is now emitted once.
- Emit workingDirectory beside projectPath, as codex does.
Tests add the upstream examples/acp-agent snapshot as a fixture, covering the
real record shapes: packed reasoning-chunks/tool-call-chunks storage rows, a
plugin-injected user/message beside the typed one, and both the streamed usage
chunk and the final assistant/message usage for the same step. Plus the same
snapshot re-encoded as multi-frame zstd with a torn tail (identical output), a
forked session, an unsupported format version, and unparsable lines.
Three fixes from review, all measured on this box.
The workload gate was files OR bytes. The files arm is wrong: 250 pending files
holding 117 KB between them spawned 5 threads and ran ~5% SLOWER than serial,
and a file count only starts paying for itself around 400. Gate on bytes alone;
the count still takes max(files / 50, bytes / 200 MB), so a few hundred huge
rollouts keep their threads.
The flat 256 MB per-worker memory budget was contradicted by the Codex workload:
a 260 MB rollout peaks near 430 MB in its worker, linearly across the pool. It is
now derived per parse as clamp(256 MB, 2 x average pending file + 128 MB, 1 GB),
which leaves a corpus of small Claude transcripts where it was and stops
over-subscribing on rollouts. The parent's buffer of up to pool.size finished
results is part of that peak and is named in the comment.
The worker/file pairing at both install sites was positional, guarded only by
position (Claude) or a path membership check (Codex). Each worker now echoes its
path and the parent asserts it, outside the per-file try: a misalignment would
install one session's turns under another's path -- a wrong number nobody would
ever notice -- so it fails the run rather than being swallowed as a parse
failure. On the Claude side that meant hoisting the whole worker-result block
above the try, which is safe because an append never consumes a result in either
its shortcut or its straddled-fallthrough case.
A mixed Claude + Codex corpus with forked rollouts in both creation orders,
asserting an identical payload, byte-identical cache shards and a byte-identical
codex-results.json between CODEBURN_PARSE_WORKERS=0 and =3, with the codex
discard count pinned above zero so the overlap path is really exercised.
Plus: a resumable rollout never reaching a worker (the decision line reports no
full parses pending after an append), the off-thread decode matching
parseCodexFileFull exactly including the cache entry it hands back, no cache file
written by the decode itself, no leaked threads, and the files-OR-bytes gate.
The comment at the install site claimed only that an overlapping worker result
'is discarded'. State why the empty-set result is installable at all — an empty
id intersection is proof a serial parse would have dropped nothing — and why the
tempting shortcut is wrong: parsedTurnsToCachedTurns delta-encodes gitBranch
across turns, so dropping one turn changes whether a LATER turn carries a
gitBranch key. Overlap discards the whole file, never individual turns.
Tests: the end-to-end determinism check now runs both parses over the SAME
corpus, so cache shard BODIES are compared byte for byte instead of just their
keys, and a new resumed-session fixture (a transcript restating another file's
message ids, in both filename orders) makes install order decide the answer.
Verified by mutation: removing the discard guard fails it, and yielding worker
results out of order fails it.
CODEBURN_VERBOSE now reports how many worker results were re-parsed in-process
on id overlap, which is what the new test asserts on. The worker bundle's source
map is excluded from the published package (-1.8 MB).
os.freemem() reports free pages on macOS, not available memory: on an idle
128 GB machine it reads a few hundred MB, so the 2 GB gate switched the worker
pool on and off between runs on the platform the desktop app ships to. The gate
and the budget now use process.availableMemory() (cgroup/rlimit-aware in a
container), falling back to os.totalmem(): serial under 4 GB available, budget
min(0.25 * available, 2 GB). An 8 GB box earns 8 threads, a 4 GB box none.
The verbose line now carries every decision input — cores, available GB, pending
files and bytes — on both the gate and the go path, so one support log explains
itself.
Pins the gates that keep threads off low-spec machines and warm runs, that a
forced worker count bypasses them, that results come back in submission order,
that a dead pool reports failure instead of throwing (and the serial fallback
lands on the same result), that no thread outlives a parse across back-to-back
parses, and that a cold CLI parse with and without workers produces the same
payload and the same cache shards.
A file's shard span was read off turns[0]/turns[-1], but several providers
emit turns non-chronologically (cursor by ROWID, goose/crush/copilot by a
DESC ordering). That produced until < bucket - an empty span, so the shard
was unreachable at every scope and its sessions re-parsed every run.
cacheFileSpan now takes the min and max month over all turns.
An entry re-bucketing out of a month the run never loaded (a re-parse that
moved its oldest turn, or the #441 failure marker that has no turns at all)
left the old copy in the carried shard, so one path lived in two shards and
a later load could resolve to the stale one. A save now prunes those paths
from the shards it carries, and a load merges shards in envelope order,
resolving any duplicate to the freshest fingerprint and dirtying both
buckets so the next save retires the loser.
A carried month whose shard another writer had republished was dropped from
the envelope outright, losing expired-transcript PR orphans no re-parse can
recover. The envelope is now re-read just before publishing and the current
shard name adopted; a ref is dropped only when that envelope lacks it too.
The same re-read moves every merge read after the ownership fence and gives
the merge one optimistic retry, so the read-modify-write window shrinks to
the publish itself.
Also: retire an orphaned v8 directory / v7 file left by an interrupted
re-layout, age-guarded, once a v9 envelope is published.
A memo hit must not re-render. The generated stamp is minted per render,
so asserting it is unchanged across the hit states that contract directly
instead of leaving it implied by byte equality.
scanProjectDirs ran every cached turn through cachedTurnToClassified —
per-call reconstruction plus the turn classifier's category / retries /
edit regexes — and only then applied the date slice, so a week view paid
to classify all of history to keep a few percent of it.
The keep/drop decision now runs on the raw CachedTurn (calls map 1:1 onto
assistantCalls, so callsInRange sees the same survivors as the classified
slicer) and only survivors are classified, still from their complete call
list. The branch and PR-set carries still walk the full ordered turn list,
and buildSpawnPrSets still reads the pre-slice turns.
Real corpus, warm one-shot status --format menubar-json --no-optimize,
identical payloads: today 1561ms -> 1318ms, week 1560ms -> 1410ms,
month 1788ms -> 1675ms.
Two live processes share one cache directory routinely (a one-shot CLI
beside the resident serve child, two menubar polls), and the shard layout
had two ways to lose data there.
- The atomic write used a FIXED temp name, so two writers publishing the
envelope — every save does — shared one `envelope.json.tmp` and
interleaved into a torn payload, or one deleted the shards the other's
envelope named. 39 of 40 rounds ended in a total cache loss. The temp
name carries a nonce again, as it did before the shard layout.
- A save reused a shard filename from its own load snapshot without
checking the file was still there. Another process republishing that
provider unlinks the old shard, so the stale writer published an
envelope naming a deleted file — read back as a corrupt provider and
dropped whole, including PR-linked orphans no re-parse can recover. A
reused shard is now existence-checked, and re-verified once more
immediately before the envelope is published; a vanished one is
rewritten from memory.
Also:
- Progress saves take a 30s floor beside the file counter. Only the
claude scan reports per file; every other provider calls saveProgress
once at its own boundary, so the counter alone never fired there.
- The unreferenced-shard sweep waits an hour (temps still 5 minutes): an
unreferenced shard may belong to a concurrent save whose envelope has
not landed yet.
- The sweep also retires the pre-v8 single-file temps in the parent
directory, which nothing writes anymore.
- The shard directory is created 0o700.
- The claude and provider paths mark the cache dirty where they DELETE a
stale entry, not only where they replace it: an unreadable file skips
the replace, and the deletion would otherwise live only in memory.
- Codex only treats a grown file as an append when the recorded boundary
still lands just after a newline, so a same-inode rewrite that happens
to end up larger re-parses instead of resuming mid-line.
Codex rollout files are append-only and the active ones run to hundreds
of MB, but any growth re-read the file from byte 0 because the cache
keyed only on mtime+size. The parser now records a restart point at every
task_started boundary — the byte offset plus the state the single-pass
decode carries across it — and a grown file with the same dev/ino picks
up from there.
The boundary sits at the task_started line itself, so the task it opens
is re-decoded from the tail; the entry stores how many calls were decoded
before that point so the resumed run starts from exactly those and cannot
double-count the open task. An unusable or absent snapshot falls back to
a full re-parse.
CODEX_CACHE_VERSION is deliberately not bumped: the new fields are
additive and absence-safe both ways, so a bump would discard a warm
cache for nothing.
A warm launch rewrote the entire session cache whenever any provider
appended a few KB: on a 6 GB corpus that is a 155 MB stringify + fsync
every run. The on-disk cache is now a version-suffixed directory holding
one shard per provider plus a small envelope, and a save rewrites only
the providers marked dirty.
- Dirtiness is tracked per provider (markCacheDirty) instead of one
global flag, so an appended Claude session no longer republishes
Codex, Copilot and the rest.
- Shards carry a nonce in their filename and the envelope is renamed
last, so a save is published at a single point: readers never see a
half-updated set, and a writer that loses the refresh ownership fence
leaves the canonical shards untouched.
- A shard that fails validation is treated as an absent provider rather
than rejecting the whole cache, so one malformed turn costs one
provider's re-parse instead of every provider's history.
- v7 migrates losslessly: the blob is re-laid-out into shards and
removed only once that save publishes. Nothing re-parses.
- Cold-parse progress saves now trigger every N files parsed rather than
every 5s, so a slow cold parse no longer rewrites the growing cache on
a wall clock.
The separator regex retried its leading \s* from every offset, which is
quadratic on the long whitespace-heavy commands agents emit; that one regex
was ~30% of a warm status run on a multi-GB corpus. Match the separator
alone and widen over whitespace by hand. Output is unchanged (differential
check over 42k real commands).
Clear the per-directory Codex and Antigravity memo maps in the resident RSS
guard; document the single cache-dir rule (XDG_CACHE_HOME no longer
consulted, ledger migrated); stop output-overflow terminations from spending
the resident's unexpected-death budget.
flatSlice returned strings within the bound unchanged, but provider adapters
pre-truncate with .slice(0, 500) before the cache site, so those views still
pinned their parent buffers. Always flatten; the round-trip is ~150ns per turn.
Use utf16le so lone surrogates survive the copy.
Cache the canonical-path Promise instead of the resolved value so calls in one
Promise.all batch share a single walk.
Document the one-time kiro re-parse and worktree regrouping.
Reads DSH sessions from $DSH_HOME/sessions (default ~/.dsh/sessions):
one directory per session holding session.jsonl.zstd (or an
uncompressed session.jsonl when compression=none).
The .zstd log is a concatenation of independent zstd frames (one per
appended event batch), which node:zlib's one-shot API cannot decode
whole; the provider ports the frame-boundary scan from the official
@deepseek-ai/dsh-session-persistence-jsonl package and decompresses
frame by frame. zstd needs Node >= 22.15; older runtimes get a notice
and DSH data is skipped.
Usage follows dsh-token-meter semantics: an assistant/message usage
report is the final value for its (turn, step) and replaces the
earlier assistant/chunk sample instead of double counting. Models come
from the most recent request/header config; reasoning tokens are
billed at the output rate. One parsed call per (turn, step), dedup key
dsh:<sessionId>:<turn>:<step>.
The kiro provider was reading the full working directory from session
metadata (meta.cwd for CLI sessions, workspacePaths[0] for v2 IDE
sessions, workspaceDirectory for workspace sessions) but discarding it
via basename(), keeping only the leaf name for display. This meant
computeAttributionRecords could never resolve kiro sessions to a git
repo, so `codeburn sync push --attribution` produced 0 facts for all
kiro-originated sessions.
Now passes the full path as projectPath on emitted ParsedProviderCalls,
which buildRepoGroups uses to resolve git identity and correlate
commits with sessions via timestamp windows. The path stays local:
only the normalized origin remote egresses in attribution spans.
Behavior changes beyond attribution:
- kiro calls now flow through canonicalizeProviderCallProject, so kiro
sessions in LINKED GIT WORKTREES canonicalize to the main repository:
their report project name changes from the worktree dir name to the
main repo name (consistent with claude/codex behavior).
- workingDirectory is now populated on kiro calls.
- PROVIDER_PARSE_VERSIONS.kiro bumped (project-path-v1): cached entries
predate projectPath and are served without re-invoking the parser, so
without the bump this fix silently no-ops for every warm cache. The
bump forces a one-time cold kiro re-parse on upgrade.
ORDERING: this commit must land WITH (or after) the preceding
SlicedString OOM fix. The forced cold re-parse it triggers is exactly
the workload that OOM'd before that fix on multi-GB kiro stores.
Perf: per-call canonicalization added a measured +5% to cold parse
(.git-marker lstat walk per call). resolveCanonicalProjectPath is now
memoized on cwd (cleared with the session cache), removing the
redundant walks for all providers.
Tests: projectPath emission fixtures for all three session formats
(CLI, v2 IDE, workspace-session), fingerprint-change assertion, and a
regression test seeding a pre-bump cache entry and proving the re-parse
recovers projectPath.
AI-Origin: human
String.prototype.slice returns a V8 SlicedString: a view that retains a
reference to its ENTIRE parent string. The parsers store short previews
of message text (userMessage.slice(0, 500/2000)) in the long-lived
session cache. Session files routinely carry 100KB+ strings (agent-
injected system prompts, tool results), so every cached preview pinned
its full parent buffer for the life of the process.
Measured on 3.2GB of kiro CLI session files (6,659 files, largest 40MB):
cold parse, default heap, before: 4.33GB peak -> OOM crash
cold parse, 8GB heap, before: 5.67GB peak (kiro provider alone)
after kiro flatSlice: 0.91GB peak
after parser.ts cache sites too: 0.64GB peak
original failing command (cold,
default heap, all providers): 0.89GB peak -> completes
Warm runs were always fine (~0.29GB) because the cache's JSON round-trip
flattens the strings on load — which made this bug appear intermittent:
it only fired on a cold or invalidated cache.
Fix: flatSlice() in content-utils.ts forces a flat copy via Buffer
round-trip. Applied at the six kiro userMessage capture sites and the
three shared cache-building sites in parser.ts (protects all providers).
Regression test asserts the no-retention property via bounded heap
growth over 1000 large-parent slices.
AI-Origin: human
cli-json-daily, spend-flow and cli-emitters spawn the real CLI per test
and blow the 5s default under full parallel suite load while passing in
isolation - the flake set #948 documented on unmodified main, observed
again locally (cli-json-daily) and in CI (cli-emitters on a green PR).
Same file-level remedy the CLI menubar suite already uses; the default
stays 5s for everything else.
The cwd evidence rule attributed ANY session sharing a checkout with a
PR-linked session, with no time bound - so a repo whose only captured PR
link was pasted once became a black hole: 129 of 131 sessions and a
month of unrelated work (~$7.4K direct, $11.2K displayed) attributed to
one PR, observed live on the desktop Pull requests tab.
Cwd anchors now carry the evidence sessions' own activity window (union
across evidence for the same PR set), and only sessions overlapping that
window plus a 6h pad inherit the PR. The rule's charter is 'a tool
session launched around PR work in this checkout', which is inherently a
same-working-stretch claim; the design's own philosophy (timestamps
narrow, never create) now applies to this rule too.
On the real corpus the row corrected to $450.87 / 21 sessions across the
PR's actual two-day working stretch. Regression pins both directions:
nearby same-cwd session inherits, weeks-later one never does; multiple
evidence sessions widen the window.
The remaining warm-serve cost was the per-request discovery sweep
(stat-ing thousands of session files) plus re-aggregation, even when
nothing on disk had changed. Serve now watches every provider's
probeRoots() via fs.watch (FSEvents-backed recursive watches on macOS)
and injects a quiet-since validator into the parser: while the watched
roots are quiet, a previous parse stays reusable past the burst window,
and an output-level memo returns identical panel queries verbatim - so a
fetch with no data changes skips the sweep AND the aggregation.
Safety rails, in order: a parse is validated-reusable only if the
watchers were armed before it ran; any filesystem event ends reuse
instantly; a 5-minute hard cap self-heals a missed event; a root that
fails to watch just goes uncovered (shorter reuse, never staleness);
outside serve the validator is never installed and behavior is
byte-identical. During an active AI session the session roots fire
constantly, so reuse correctly stays inside the 10s burst window - the
extended reuse serves the idle-browsing case it was built for.
The one watched path inside the cache dir is antigravity's statusline
file specifically, so serve's own cache writes never self-invalidate.
Adversarial review of the serve design surfaced three weaknesses a
one-shot CLI never had, because it never lived long enough:
- Pricing-affecting config (model aliases, price overrides, local-model
savings) now participates in the parse memo key. Config reloads fresh
per request (the preAction hook), but a memoized or burst-reused parse
embedded costs priced under the OLD config; the widened key makes any
such change an automatic memo miss. New alias-hash helper + tests.
- Memory guard: past 3GB RSS the serve loop drops its in-memory memos
(session cache + parse entries) and the next request re-parses once.
The child never exits for this, so the client's death budget is
untouched.
- codeburn serve typed in an interactive terminal now explains itself on
stderr instead of hanging silently on stdin.
Every CLI spawn on a large corpus pays seconds of fixed cost before any
query work: node boot, a 100MB+ session-cache JSON.parse, the discovery +
fingerprint sweep, and serve-time classification. The desktop app spawns
one CLI per panel fetch, so it pays that cost per panel.
codeburn serve --stdio is the same CLI kept warm: the app holds one child,
sends {id, args} per line, and gets the command's stdout back. Three layers
make it fast, each disabled outside serve so one-shot runs stay byte-exact:
- loadCache memo (session-cache.ts): the parsed cache object is reused
while a stat() shows the file unchanged; saveCache updates it
write-through. A rewrite by another process still forces a fresh read.
- burst reuse (parser.ts, CODEBURN_PARSE_BURST_MS, serve sets 10s): panel
bursts anchor their range ends at their own new Date(), so the exact-key
memo never hits in real traffic; within the window a re-anchored range is
served by trimming the previous parse instead of re-running discovery.
- fresh commander program per request (main.ts buildProgram factory),
because commander option state is sticky across parses.
The server allows only the app's read queries (status/overview/models/
sessions/compare/yield/spend/optimize/audit), refuses everything else
(client falls back to a spawn), serializes requests, and converts
process.exit into a caught signal. The app starts the child once at
startup; requests route through it only when warm, cold-start keeps the
spawn path with its progress events, any serve failure falls back to a
spawn, and three child deaths disable serve for the app run.
Measured on a real 17B-token corpus: panel fetches drop from ~7.4s per
spawn to 5-900ms warm (sessions/spend 5ms, status 898ms). One-shot CLI
output verified byte-identical against the pre-branch baseline.
