* fix(copilot): bill CLI input and cache tokens from session.shutdown
Copilot CLI sessions (~/.copilot/session-state/*/events.jsonl) recorded
only outputTokens, because the parser read assistant.message events and
ignored the session.shutdown rollup. That rollup is the sole on-disk
source of a CLI session's input, cache-read and cache-write tokens, so
cost was underreported (in the reported case 35k+ cache-read tokens were
dropped entirely).
Handle session.shutdown in the CLI (non-transcript) JSONL path and emit
one supplementary call per model from modelMetrics.<model>.usage. The
per-turn assistant.message events keep owning output tokens, tools and
turn structure; the shutdown call contributes only the input and cache
tokens they lack, with output excluded, so no dimension double-counts and
the combined cost equals the full CLI-measured cost. usage.inputTokens is
cache-inclusive (input + cache_read + cache_write), so the cache
components are subtracted to recover the uncached input calculateCost
expects. These are real counts written by the CLI, so costIsEstimated is
false. Sessions without session.shutdown (crashed or still running) keep
the exact previous output-only behavior, and the VS Code, JetBrains and
OTel paths are untouched.
Bump the copilot entry in PROVIDER_PARSE_VERSIONS so already-cached CLI
sessions re-parse and pick up the corrected tokens and cost.
* fix(copilot): never emit an empty-timestamp shutdown call
A session.shutdown without its own timestamp and without sessionStartTime
produced a call with an empty timestamp, which the date-range filters in
parser.ts drop silently, erasing exactly the tokens this feature bills.
Fall back to the last stamped event in the session.
* fix(cache): durable orphans survive provider fingerprint changes
Bumping a durable provider's PROVIDER_PARSE_VERSIONS changed its env
fingerprint, and getOrCreateProviderSection replaced the whole section on
mismatch. For copilot OTel orphans the discarded cache entry is the only
remaining record of pruned spans, so the first post-upgrade run silently
and permanently erased month-to-date history. Carry forward exactly the
entries whose source no longer exists; present sources still drop and
re-parse under the new fingerprint, which is the point of a bump.
Regression test forges a pre-upgrade fingerprint on a pruned-source
orphan and asserts it stays counted and cached across the transition.
Maintainer follow-up:
- Derive JetBrains dedup keys from the reply content (sha256 prefix plus a
per-hash occurrence counter) instead of the blob's scan position. Copilot
is a durable provider: cached turns are never deleted and a re-parse
appends any unseen key, while MVStore compaction can rewrite the store
with blobs in a different byte order. With positional keys, a rewrite
that moves a new blob ahead of an old one hands the new turn the old
key (skipped as seen) and re-emits the old turn under a fresh index,
double-billing it. Covered by a regression test that fails on the
positional scheme.
- Add CODEBURN_COPILOT_JETBRAINS_DIR to the env-isolation cleared list so
a developer's real JetBrains store never bleeds into fixture tests.
JetBrains Copilot plugin ≤1.5.x (e.g. 1.5.59-243) stores all session turns
inside ONE large binary-framed outer Nitrite document, rather than the
per-turn {"__first__":{"type":"Subgraph",...}} blobs introduced in later
plugins (≥1.12.x, e.g. 1.12.1-251).
In the old format each assistant turn is a UUID-keyed Value entry whose
value field contains a JSON-string-escaped AgentRound record:
{"<uuid>":{"type":"Value","value":"{\"type\":\"AgentRound\",
\"data\":\"{...reply...}\"}"}, ...}
The extractResponseText depth-unescape loop already handles this one extra
level of escaping; the only gap was that extractJetBrainsDbTurns never fed
it the outer document — it only scanned for __first__/Subgraph blobs, which
the old plugin never writes.
Add a fallback that activates when the Subgraph scan produces zero turns but
'AgentRound' text is present in the raw file (old-format signal). It locates
the binary-framed outer document (UUID-keyed Value entry, hex matched
case-insensitively so an uppercase UUID does not fall through to $0), extracts
it with matchJsonObject, and passes it to extractResponseText. Because the outer
document holds every turn in one blob, this emits ONE session-level call per
document (all rounds' replies joined): cost/tokens are correct, only the
per-turn call-count granularity is coarser — an accepted tradeoff for legacy
data. MVStore keeps two identical collection copies; seenReplies dedupes them.
The fallback is guarded by turns.length === 0 so new-format sessions (whose
Subgraph scan succeeds) are completely unaffected and never double-counted.
Tests: old-format doc with multiple AgentRound rounds → 1 call whose token
count equals the two non-empty replies joined (the empty tool-call round is
excluded); an uppercase-UUID variant (fails without the case-insensitive
match); and a guard that new-format Subgraph turns are not double-counted.
docs/providers/copilot.md documents the old format and the one-call-per-session
limitation.
JetBrains Copilot has two turn shapes in the Nitrite .db:
- ask mode — the reply is a `Markdown` record's `text`;
- agent / plan mode (e.g. PyCharm agent sessions, `/plan …`) — the reply is the
`reply` field of an `AgentRound` record, and the `Markdown` record instead
holds the USER's prompt.
extractResponseText only read Markdown, so agent-mode turns yielded no reply
text: they were discovered (session/turn counts showed up) but priced at $0
because output tokens came out zero. On this machine that silently
under-counted a PyCharm session ($0 → $0.35) and several IntelliJ agent turns.
Determine the mode by the PRESENCE of an `AgentRound` record and read only that
record's `reply` (collecting every non-empty round in a multi-round blob).
Crucially, an agent blob whose reply is empty — a failed turn or a pure
tool-call round — does NOT fall back to the Markdown record, so a user prompt
is never mistaken for the assistant's output; such turns bill $0 as before.
Ask-mode blobs (no AgentRound) keep reading Markdown. Plan mode's sidecar
records — Thinking, PendingChanges (proposed diff, under `content`), AskQuestion,
Notification, SubTurn, and file-read `text` results — are never read as output.
Verified across all local stores: the two reply shapes never coexist in one
blob, so the split is unambiguous.
Tests: agent-mode reply extraction (ignoring the prompt Markdown), pure
tool-call rounds → $0, multi-round collection, and a failed agent turn → $0.
docs/providers/copilot.md documents both turn shapes and the ignored sidecar
records.
## What & why
The JetBrains Copilot plugin (IntelliJ, PyCharm, RubyMine, …) stores its
chat/agent sessions under `~/.config/github-copilot/<ide>/<kind>/<storeId>/` —
a location none of the existing Copilot sources (CLI JSONL, VS Code chat
sessions/transcripts, OTel SQLite) read. As a result all JetBrains Copilot
usage was silently uncounted in every CodeBurn report. This adds a reader for
that store so those sessions are discovered, priced, and attributed to the
right project.
## How it works
- **Reader.** The store's session content is a Nitrite `.db` — an H2 MVStore of
Java-serialized documents. It is scanned as `latin1` for byte-offset
stability: no Java deserializer, no new dependency, and it is not SQLite so
`node:sqlite` is not involved.
- **Reply text.** Assistant replies live in nested-escaped
`{"__first__":{"type":"Subgraph"…}}` blobs. The text is recovered by
unescaping one level at a time and, at the depth where the Markdown record's
`data` field is a well-formed one-level-escaped JSON document, reading it
structurally — so a reply containing its own quotes is never truncated or
duplicated (which would otherwise inflate the estimate).
- **Tokens/cost.** The store records no token counts, so output tokens are
estimated from the reply text (`CHARS_PER_TOKEN = 4`, re-decoded
latin1→utf8 so multibyte replies count by codepoint) and every call is marked
`costIsEstimated`. Failed generations (error status, no reply) are billed $0.
- **Sessions.** One `.db` holds many chat tabs; turns are grouped back to their
conversation GUID so the UI shows one session per tab, deduped by reply
content per conversation.
- **Project attribution**, most authoritative first:
1. the plugin-recorded `projectName` field (JetBrains Copilot 1.12+), joined
across kind dirs by store id — the billable turns live in
`chat-agent-sessions`, but the label is usually written into the sibling
`chat-sessions`/`chat-edit-sessions` store. Read length-delimited and
re-decoded latin1→utf8 so non-ASCII repo names round-trip.
2. the `.git` repo root of a referenced `file://` path.
3. a generic `copilot-jetbrains` bucket when neither signal exists.
The conversation title is a chat-thread name, not a project, so it is kept
out of the project field and surfaced as the session label instead.
Override the JetBrains github-copilot root with
`CODEBURN_COPILOT_JETBRAINS_DIR`.
## Docs
- `docs/providers/copilot.md` — full JetBrains section (store layout, latin1
scan, reply extraction, projectName precedence + cross-kind join).
- `docs/providers/README.md` — Copilot storage updated to note the Nitrite .db.
## How to verify
- `npm test -- copilot` and `npx tsc --noEmit` (fixtures reproduce the real
nested-escaped .db framing, including quote- and multibyte-bearing replies).
- End to end against a real install:
`CODEBURN_CACHE_DIR=$(mktemp -d) node dist/cli.js status --provider copilot \
--period all --format menubar-json`
— JetBrains sessions appear By-Project under their real repo names.
- Set `CODEBURN_COPILOT_JETBRAINS_DIR` to a fixture root to parse a controlled
store without touching the real config dir.
VS Code GitHub Copilot Chat users with no OTel agent-traces.db and no
~/.copilot/session-state/ saw $0.00 cost: codeburn never read VS Code's
core chat persistence, the only on-disk source carrying their token counts.
Add a fourth Copilot source that reads the VS Code chat delta-journals at
workspaceStorage/<hash>/chatSessions/*.jsonl and
globalStorage/emptyWindowChatSessions/*.jsonl. The files are a kind:0
snapshot / kind:1 path-set / kind:2 array-append journal; we replay it
(prototype-pollution-safe: __proto__/prototype/constructor segments are
rejected and containers use Object.create(null)) and read each request's
result.metadata.promptTokens / outputTokens (falling back to
completionTokens) and resolvedModel for pricing.
Dedup so users with multiple sources are not double-counted: prefer OTel
(skip chatSessions discovery when an OTel source is present), and skip a
workspace's legacy GitHub.copilot-chat/transcripts when that workspace has
chatSessions. New env overrides CODEBURN_COPILOT_GLOBAL_STORAGE_DIR and the
existing CODEBURN_COPILOT_WS_STORAGE_DIR keep discovery testable.
Tests cover the reporter's real request shape (promptTokens 32543 /
outputTokens 60 -> non-zero cost), empty sessions, emptyWindow discovery,
append-then-edit replay, requestId dedup, prototype-pollution paths, the
transcript skip, and the OTel-prefer skip.
Follow-up to #433. Adds tests for the JetBrains format: discovery of a jb
chat.jsonl, parsing into a call with inferred model/tools/user message, the
isJetBrainsFormat routing guard (legacy user.message files must not route to
the JB parser), and the id-less dedup fallback.
Also fixes inferJBProjectFromContent: it split homedir() on the platform
separator but the recorded tool path on a fixed '/', so project inference
always fell back to the raw session id on Windows. Split both on either
separator.
Two passes of validators across CLI accuracy, dashboard UX, menubar Swift,
performance, security, and end-to-end smoke tests on real session data.
Data-correctness fixes:
- parseLocalDate rejects month/day overflow. JS Date silently rolled
Feb 31 to Mar 3, so --from 2026-02-31 --to 2026-03-15 quietly dropped
sessions on Feb 28 - Mar 2. Now throws "Invalid date" with a clear
reason. Leap-day case covered (2024-02-29 valid, 2025-02-29 rejected).
- CSV/JSON exports use the active currency's natural decimal places. The
previous round2 helper produced ¥412.37 in CSV while the dashboard
rendered ¥412 — finance teams comparing the two surfaces saw a
discrepancy. New roundForActiveCurrency consults Intl.NumberFormat for
the right precision (0 for JPY/KRW/CLP, 2 for USD/EUR, etc).
- Copilot toolRequests is Array.isArray-guarded in both modern and legacy
event branches. Previously a corrupt session with toolRequests=null or
a string aborted the whole file's parse loop and silently dropped every
legitimate call after it.
- Codex token_count dedup uses a null sentinel for prevCumulativeTotal so
the first event is never confused with a duplicate. Sessions that emit
only last_token_usage (no total_token_usage) report cumulativeTotal=0
on every event; with the previous 0-initialized prev, the first event
matched the dedup guard and was dropped.
- LiteLLM pricing values are clamped to [0, 1] per token via safePerTokenRate.
Defense in depth against a tampered upstream JSON shipping negative or
absurdly large per-token costs that would otherwise propagate into all
cost totals.
Performance:
- Cursor SQLite parse no longer pegs at minutes on multi-GB DBs. Two
changes: per-conversation user-message buffer uses an index pointer
instead of Array.shift() (which was O(n) per call); and a real ROWID
cutoff via subquery limits the scan to the most recent 250k bubbles
with a stderr warning so power users get a partial report rather than
a stalled CLI.
- Spawned codeburn CLI subprocesses are terminated when the calling Task
is cancelled. Without this, rapid period/provider tab clicks in the
menubar cancelled the Task but left the subprocess running to
completion, piling up zombie processes.
UX:
- Dashboard period switch flips to loading and clears projects
synchronously before reloadData runs, eliminating the frame where the
new period label rendered over the old period's projects.
- Optimize findings tab paginates 3-at-a-time with j/k scroll. With 4
new detectors plus 7 originals, 8-10 findings * 6 lines was scrolling
the StatusBar off the alt buffer top.
- Custom --from/--to ranges hide the period tab strip and disable the
1-5 / arrow keys so a stray period press no longer abandons the user's
explicit range. A "Custom range: X to Y" banner replaces the tab strip.
- OpenCode storage-format warning is per-table-set, rate-limited to once
per process, and points the user at OpenCode's migration step or the
issue tracker. The previous all-or-nothing check fired the generic
"format not recognized" string for any schema mismatch.
Menubar / OAuth:
- Both Claude and Codex bootstrap (Reconnect button) now honour the
usageBlockedUntil 429 backoff that refreshIfBootstrapped respects.
Spamming Reconnect during sustained rate-limit windows previously
hammered the upstream endpoint on every click.
- Codex Retry-After HTTP header is parsed (delta-seconds plus IMF-fixdate
fallback) so we don't over-back-off when ChatGPT tells us a shorter
window than our 5-minute floor.
- Both credential cache files are written via SafeFile.write
(O_CREAT | O_EXCL | O_NOFOLLOW with explicit 0600) so there is no race
window where the temp file briefly exists at default umask, and a
symlink at the destination cannot redirect the write. Reads now route
through SafeFile.read with a 64 KiB cap, closing the symlink-follow gap
on Data(contentsOf:).
CI signal:
- TypeScript strict typecheck (tsc --noEmit) is now zero errors. The
six errors in src/providers/copilot.ts came from a discriminated-union
catch-all branch whose `data: Record<string, unknown>` shape TS picked
over the specific event branches when narrowing on `type`. Removed the
catch-all; runtime falls through unknown event types via the existing
if/else chain.
Tests added: 16 new (now 555 total)
- date-range-filter: month/day/year overflow rejection, leap-day correctness
- currency-rounding: convertCost no-rounding contract, roundForActiveCurrency
for USD/JPY/KRW/EUR
- providers/copilot: malformed toolRequests does not abort the parse
- providers/cursor-bubble-dedup: re-parse after token mutation does not
double-count, single parse yields one call per bubble
- providers/codex: first event with cumulativeTotal=0 not dropped,
consecutive zero-cumulative duplicates still deduped
The Copilot provider only looked in ~/.copilot/session-state/ which is
from an older CLI tool. VS Code Copilot agent stores transcripts in
~/Library/Application Support/Code/User/workspaceStorage/*/GitHub.copilot-chat/transcripts/.
The new transcript format has no outputTokens or model_change events,
so tokens are estimated from content length and the model is inferred
from tool call ID prefixes. Both legacy and VS Code paths are now
scanned in parallel.
Fixes#161
- init currentModel to '' and skip assistant messages before first
session.model_change to avoid silent misattribution
- add comment documenting why inputTokens is always 0
- fix delete_file tool mapping ('Edit' -> 'Delete')
- add schema doc comment to ToolRequest optional fields
- remove catch-all from CopilotEvent union for proper TS narrowing
- add tests: pre-model-change skip, workspace.yaml quote/comment strip,
longest-prefix model display name match
