Every dated command re-walks and re-stats every provider tree before it can
decide what the cache already covers. That sweep was strictly serial at every
level -- one readdir, one stat, one state.json read at a time, and one provider
after another -- so on a 21k-file / 9-provider corpus it owned most of a warm
run's wall clock with the machine idle waiting on the kernel.
Measured per warm `today` pass on that corpus before this change: 859 ms in
discoverAllSessions (kimicode 467, codex 143, claude 135, grok 77) and 727 ms in
the Claude project walk + fingerprint pass.
- fs-utils: mapWithConcurrency + FS_SCAN_CONCURRENCY, one bounded, order-
preserving helper for the whole sweep.
- providers/index: run provider discovery concurrently, concatenated in
registry order.
- claude/codex/grok/kimicode: walk each level with the level fanned out,
re-concatenated in readdir order before anything reconciles.
- parser: the Claude dir walk and both fingerprint passes (scanProjectDirs and
parseProviderSources) fan out, then reconcile serially in discovery order,
which is what changedFiles ordering and the seenMsgIds pre-seed depend on.
- parser: collectJsonlFiles reads entries with their types, so a plain file no
longer costs a wasted subagents/ probe.
- codex-cache: share one in-flight load between concurrent readers. The memo is
only populated after the read + parse resolves, so concurrent discovery had
every caller re-reading and re-parsing the same (here 59 MB) file.
No reconciliation logic changed: the same fingerprints reach the same cache
comparison in the same order. Warm `today` 5.30s -> 2.99s, cold 34.7s -> 31.1s
(medians of 5 / 2, isolated HOME + cache). today/report -p month/models/sessions
JSON, warm and cold, per-provider and combined, are byte-identical apart from
the run's own `generated` timestamp.
Closes#1104
## What & why
The JetBrains Copilot plugin (IntelliJ, PyCharm, RubyMine, …) stores its
chat/agent sessions under `~/.config/github-copilot/<ide>/<kind>/<storeId>/` —
a location none of the existing Copilot sources (CLI JSONL, VS Code chat
sessions/transcripts, OTel SQLite) read. As a result all JetBrains Copilot
usage was silently uncounted in every CodeBurn report. This adds a reader for
that store so those sessions are discovered, priced, and attributed to the
right project.
## How it works
- **Reader.** The store's session content is a Nitrite `.db` — an H2 MVStore of
Java-serialized documents. It is scanned as `latin1` for byte-offset
stability: no Java deserializer, no new dependency, and it is not SQLite so
`node:sqlite` is not involved.
- **Reply text.** Assistant replies live in nested-escaped
`{"__first__":{"type":"Subgraph"…}}` blobs. The text is recovered by
unescaping one level at a time and, at the depth where the Markdown record's
`data` field is a well-formed one-level-escaped JSON document, reading it
structurally — so a reply containing its own quotes is never truncated or
duplicated (which would otherwise inflate the estimate).
- **Tokens/cost.** The store records no token counts, so output tokens are
estimated from the reply text (`CHARS_PER_TOKEN = 4`, re-decoded
latin1→utf8 so multibyte replies count by codepoint) and every call is marked
`costIsEstimated`. Failed generations (error status, no reply) are billed $0.
- **Sessions.** One `.db` holds many chat tabs; turns are grouped back to their
conversation GUID so the UI shows one session per tab, deduped by reply
content per conversation.
- **Project attribution**, most authoritative first:
1. the plugin-recorded `projectName` field (JetBrains Copilot 1.12+), joined
across kind dirs by store id — the billable turns live in
`chat-agent-sessions`, but the label is usually written into the sibling
`chat-sessions`/`chat-edit-sessions` store. Read length-delimited and
re-decoded latin1→utf8 so non-ASCII repo names round-trip.
2. the `.git` repo root of a referenced `file://` path.
3. a generic `copilot-jetbrains` bucket when neither signal exists.
The conversation title is a chat-thread name, not a project, so it is kept
out of the project field and surfaced as the session label instead.
Override the JetBrains github-copilot root with
`CODEBURN_COPILOT_JETBRAINS_DIR`.
## Docs
- `docs/providers/copilot.md` — full JetBrains section (store layout, latin1
scan, reply extraction, projectName precedence + cross-kind join).
- `docs/providers/README.md` — Copilot storage updated to note the Nitrite .db.
## How to verify
- `npm test -- copilot` and `npx tsc --noEmit` (fixtures reproduce the real
nested-escaped .db framing, including quote- and multibyte-bearing replies).
- End to end against a real install:
`CODEBURN_CACHE_DIR=$(mktemp -d) node dist/cli.js status --provider copilot \
--period all --format menubar-json`
— JetBrains sessions appear By-Project under their real repo names.
- Set `CODEBURN_COPILOT_JETBRAINS_DIR` to a fixture root to parse a controlled
store without touching the real config dir.
The 2GB stream cap silently dropped whole Codex session files over the limit, so a live 2.4GB session (about 308M tokens of real gpt-5.5 spend) was excluded and reported as zero. Raise MAX_STREAM_SESSION_FILE_BYTES to 4GB (the line-by-line reader is bounded-memory and handles multi-GB files), and surface any remaining oversize skip via an always-on notice instead of a verbose-gated warning so a dropped session is never silent. Adds a maxBytes option for testability and tests covering the cap boundary.
Cache normalized turns/calls to ~/.cache/codeburn/session-cache.json so
the CLI skips re-parsing unchanged JSONL files on subsequent runs.
File reconciliation uses dev+ino+mtime+size fingerprinting; cost,
classification, and summaries are recomputed at query time. Atomic
writes via temp+fsync+rename, deep structural validation on load,
per-provider env fingerprinting, and best-effort save so cache failures
never break the CLI. ~6x speedup on warm cache.
Three-layer fix for V8 heap exhaustion when parsing heavy session data:
1. Buffer-based readSessionLines (fs-utils.ts): Replace readline with raw
Buffer streaming using Buffer.indexOf(0x0a). Eliminates ConsString trees
that caused OOM when regex-flattening 100MB+ lines. Two-state machine
(ACCUMULATING/SCANNING) skips old lines at ~2KB cost instead of 200MB.
2. Large-line streaming parser (parser.ts): Hand-written JSON scanner for
lines >32KB extracts only cost/token/tool fields without JSON.parse,
avoiding full object graph allocation. Dual string/Buffer paths.
3. Dashboard memory management (dashboard.tsx): Disable auto-refresh for
heavy periods (30d/month/all), clear old dataset before reload via
nextTick to allow GC, prevent overlapping reloads with mutex, lazy
optimize scanning on keypress instead of useEffect.
Also fixes three race conditions in dashboard reload deduplication:
- Early return after nextTick bypassing finally block (permanent mutex lock)
- A->B->A period switching dropping final reload (stale pending)
- Stale pendingReloadRef not cleared when in-flight matches request
Heavy Codex users hit MAX_SESSION_FILE_BYTES (128 MB) on long-running
sessions. The file is read in full via readSessionFile and then split on
'\n', so even bumping the cap eventually runs into V8's 512 MB string
limit (split doubles the high-water mark).
readSessionLines is a streaming generator that already exists in
fs-utils for exactly this case but only readFirstLine was using it.
Switch the Codex provider to consume it and let the cap apply only when
streaming would still be unreasonable.
Changes:
- src/fs-utils.ts: introduce MAX_STREAM_SESSION_FILE_BYTES (2 GB) and
apply it in readSessionLines instead of the full-read cap. Keep
MAX_SESSION_FILE_BYTES for readSessionFile / readSessionFileSync
consumers that materialize the whole file.
- src/providers/codex.ts: replace `readSessionFile -> split('\n')` with
`for await (... of readSessionLines)`. Add sawAnyLine guard so a
failed/empty stream skips cache write, preserving the previous
early-return behavior.
Empirical impact on a real account with one 247 MB rollout: 7-day totals
went from 4,536 calls / €358.69 / 20.1M input tokens to 6,111 calls /
€550.67 / 37.3M input tokens. The previously-skipped session is now
included; no other behavior changes.
Refs #204
- Remove bidirectional fuzzy match in getModelCosts that could return
wrong pricing when a short canonical name prefix-matched a longer key
- Use explicit undefined check in parseLiteLLMEntry so free models with
zero cost are not silently dropped from the LiteLLM pricing database
- Destroy read stream in finally block of readSessionLines to prevent
file descriptor leaks when the generator is abandoned early
- Extend CSV injection escaping to cover tab and carriage-return prefixes
- Add optional chaining fallback for empty periods in exportCsv/exportJson
- Add regression tests for all fixes (models, export, fs-utils)
Adds readSessionFile / readSessionFileSync / readSessionLines with a
128 MB hard cap and 8 MB streaming threshold. Verbose mode
(CODEBURN_VERBOSE=1) logs skipped and failed reads to stderr.
Prepares the MEDIUM-1 migration of all provider read paths.
