mirror of
https://github.com/AgentSeal/codeburn.git
synced 2026-08-29 18:33:01 +00:00
a5a0ddaeac
The Copilot CLI and the GitHub Copilot desktop app both write ~/.copilot/session-store.db unconditionally; its assistant_usage_events table holds one row per API request. Until now input/cache tokens for these surfaces came only from the session.shutdown rollups in events.jsonl, which are written only on clean shutdown (a crash loses the whole leg's input/cache accounting) and lump each session leg into one per-model total. The rollup also RESETS its counters at in-session compaction (traced on a clean single-process 107-request session whose sole rollup covered exactly its five post-compaction requests), so even cleanly-closed long sessions were truncated; on a long-history machine the store recovered ~35% of real Copilot spend lost to crashes and compaction resets. The DB rows are per-request, crash-proof, and carry real timestamps. The store's input_tokens is cache-INCLUSIVE (input + cache_read + cache_write), the same convention as the shutdown rollups — verified against each row's token_details_json and by reconciling per-session sums against the CLI's own footers and rollups across two machines (1,380+ rows, 8 models, CLI 1.0.70–1.0.79, schema_version 6): every divergence was a rollup gap. Emitted calls mirror the shutdown-call contract: input/cache/reasoning only, output 0 — per-turn output stays owned by the events.jsonl assistant.message calls. Rollup-vs-store precedence is RECONCILED at serve time, per (session, model), and only there. Both representations always parse and cache; parseProviderSources aggregates the cached calls and, wherever store rows exist for a (session, model), drops the rollup calls and serves the rows plus per-leg RESIDUAL calls: each rollup leg subtracts only the rows in its own interval — rows commit strictly before their leg's shutdown line, so a leg at time T covers exactly the rows in (previous leg's T, T] — and any remainder (per token component, floored at zero) serves once at that leg's own timestamp. A store missing requests a leg covered — adopted mid-session, rows pruned before ever being read — therefore still serves that tail exactly once ON THAT LEG'S DAY, a crash-tail row the rollup never saw can never cancel it, and a complete store serves pure per-request granularity with every residual retired to zero. The decision reads only cached contents, never discovery: deleting or resetting the store changes nothing served, so finalized daily history can never flip on an absence epoch; cached rows of a deleted store remain the record until the 90-day orphan age-out (which exempts still-discovered paths). The serve set is the one coherent snapshot — nothing a writer does between discovery and a parse can change what one pass sees — and read-time precedence heals persisted duplication (stale epochs, runtimes without node:sqlite, restored files) instead of preserving it, following the buildDurablePeriod pattern. Store rows and rollups carry supplementary accounting weight. A rollup (or its residual) is aggregate accounting, never a request: zero api-call/model-call/turn weight, tokens and cost fully retained. A store row is one real request, but when it pairs with a served per-turn call it is supplementary too; rows pair with same-model per-turn calls by timestamp adjacency (monotone matching, tight 2-minute window — the two are written at the same completion moment, and a wide window would let a crash-only row pair against a neighbor whose own row is missing), computed once over the FULL serve set so a date-range boundary that separates a row from its call cannot double the request across adjacent day queries. Only the unpaired rows — store-only requests, exactly where crash-lost requests sit — count. Supplementary-only turns fold into the nearest behavioral turn within 30 minutes; with no behavioral turn to fold into they stay separate weightless turns, each on its own day, with apiCalls 0 — and the session emission gate admits usage-bearing zero-call sessions. The weight propagates into the daily cache: aggregateProjectsIntoDays applies the same rule to every calls counter and category-turn count it seals, so v19 history and live summaries can never disagree about what was a request. A changed source whose read defers on the busy shape (locked, EACCES, corrupt mid-replace — discovery still emits the source; only true absence or a schema mismatch reads as absent) now marks session hydration incomplete, so the daily backfill holds its watermark instead of finalizing a day the deferred rows never reached; an unchanged unreadable store defers nothing. The verdict travels with its result — the 180s memo and the serve burst-reuse restore the hydration verdict their cached data was parsed under, so a memoized partial parse cannot inherit a later parse's complete — and a discovered source whose FINGERPRINT cannot be read (EACCES on a present file) defers instead of silently skipping, while a genuinely deleted file stays a silent skip. Copilot reasoning tokens are no longer double-billed at the report layer: they are a subset of the output the per-turn calls already price, and copilot joins claude in the reasoning-inside-output case of the query-time cost recompute. Store dedup keys are content-discriminated — copilot-store:<sid>:<rowId>:<fnv1a64(created_at|tokens|model)> — because AUTOINCREMENT prevents id reuse only within one database lifetime: a same-path DB reset reusing row ids now mints new keys instead of the durable union swallowing the new usage, while a byte-identical re-insert still collapses (64-bit: 32-bit FNV collisions between plausible token tuples are constructible). Every call of a session serves under one project label resolved at serve time — the session-state-derived label when the serve set knows it, else the store rows' own — so neither rows cached before events.jsonl existed nor an events.jsonl orphaned by a session-state prune can split the session across two grouping keys. CODEBURN_COPILOT_SESSION_STORE_DB is read but deliberately NOT fingerprinted, per the #927 ruling (any copilot fingerprint change drops cached entries whose path still exists, destroying pruned history only the cache holds); the read is allowlisted in the #927 guard, and serve-time reconciliation makes repointing safe without a fingerprint — the new store's rows parse on sight and the old path's entries persist as durable orphans. The copilot parse version appends session-store-v2 and the daily cache bumps v17 → v19: per-day attribution, call counts and costs all change against pre-store builds. 19, not 18: an earlier pushed head of this PR already claimed v18 under different accounting, and the carry-forward would adopt those days as finalized without re-deriving them. Verified by A/B on snapshots of two real stores, a live SIGKILL crash test (row present, no rollup, tokens recovered exactly), live resumes whose warm-cache deltas matched new rows to the token, upgrade-healing at 4,800-session scale, and serve-level regressions pinning every maintainer finding from six review rounds: the rows-then-shutdown race, stale-cache healing, age-out exemption, absence-epoch identity, progressive row landing with residual retirement, behavioral weight across all four pinned scenarios, the hydration fence, project unification in both directions, the same-path reset, mixed coverage (crash tail vs covered-leg gap), multi-leg residual day attribution, range-invariant pairing, memo-scoped hydration verdicts, and the fingerprint-failure fence. |
||
|---|---|---|
| .. | ||
| fixtures | ||
| providers | ||
| security | ||
| setup | ||
| sharing | ||
| act-journal.test.ts | ||
| act-model-defaults.test.ts | ||
| act-report.test.ts | ||
| act-undo.test.ts | ||
| antigravity-statusline.test.ts | ||
| audit-report.test.ts | ||
| bash-commands.test.ts | ||
| blob-to-text.test.ts | ||
| budget.test.ts | ||
| cache-dir.test.ts | ||
| cache-directory-switch.test.ts | ||
| cache-persistence-coldstart.test.ts | ||
| cache-refresh-lock-corrupt-body.test.ts | ||
| cache-refresh-lock-process.test.ts | ||
| cache-refresh-lock.test.ts | ||
| classifier.test.ts | ||
| cli-budget.test.ts | ||
| cli-codex-tps.test.ts | ||
| cli-date.test.ts | ||
| cli-deepseek-v4-pricing.test.ts | ||
| cli-devin-model-variants.test.ts | ||
| cli-durable-totals.test.ts | ||
| cli-emitters.test.ts | ||
| cli-export-date-range.test.ts | ||
| cli-json-daily.test.ts | ||
| cli-model-savings.test.ts | ||
| cli-plan.test.ts | ||
| cli-price-override.test.ts | ||
| cli-provider-validation.test.ts | ||
| cli-proxy-path.test.ts | ||
| cli-refresh-help.test.ts | ||
| cli-settings-json.test.ts | ||
| cli-status-menubar.test.ts | ||
| codex-cache-invalidation.test.ts | ||
| codex-credits.test.ts | ||
| codex-rich-capture.test.ts | ||
| codex-throughput.test.ts | ||
| compare-stats.test.ts | ||
| content-utils.test.ts | ||
| context-budget-home.test.ts | ||
| context-tree-api-prefix.test.ts | ||
| context-tui-title.test.ts | ||
| cross-provider-pr-correlation.test.ts | ||
| currency-rounding.test.ts | ||
| daily-cache-carry-forward.test.ts | ||
| daily-cache-degraded-completeness.test.ts | ||
| daily-cache-tz-dedup.test.ts | ||
| daily-cache.test.ts | ||
| dashboard.test.ts | ||
| date-range-filter.test.ts | ||
| day-aggregator-savings.test.ts | ||
| day-aggregator.test.ts | ||
| defer-plans.test.ts | ||
| devices-json.test.ts | ||
| doctor.test.ts | ||
| export.test.ts | ||
| fetch-utils.test.ts | ||
| flat-slice.test.ts | ||
| format-carried-cost-note.test.ts | ||
| format-mark-estimated.test.ts | ||
| format-status-bar.test.ts | ||
| fs-utils.test.ts | ||
| granular-history.test.ts | ||
| guard-hooks.test.ts | ||
| guard-install.test.ts | ||
| hydration-interrupt-convergence.test.ts | ||
| ink-win.test.ts | ||
| kiro-cache-invalidation.test.ts | ||
| kiro-projectpath.test.ts | ||
| local-model-savings.test.ts | ||
| mcp-coverage.test.ts | ||
| mcp-deferral.test.ts | ||
| mcp-redact.test.ts | ||
| mcp-server.test.ts | ||
| mcp-tables.test.ts | ||
| menubar-carried-headline.test.ts | ||
| menubar-installer.test.ts | ||
| menubar-json.test.ts | ||
| menubar-report-parity.test.ts | ||
| menubar-savings.test.ts | ||
| minimax.test.ts | ||
| model-breakdown.test.ts | ||
| model-efficiency.test.ts | ||
| models-hoist.test.ts | ||
| models-report.test.ts | ||
| models.test.ts | ||
| optimize-apply.test.ts | ||
| optimize-fs.test.ts | ||
| optimize-zero-edit-savings.test.ts | ||
| optimize.test.ts | ||
| otel-cache-aggregation.test.ts | ||
| overview.test.ts | ||
| parse-workers.test.ts | ||
| parser-advisor-usage.test.ts | ||
| parser-antigravity-timestamp.test.ts | ||
| parser-cache-refresh-timeout.test.ts | ||
| parser-classify-after-slice.test.ts | ||
| parser-claude-cwd.test.ts | ||
| parser-compact-entry.test.ts | ||
| parser-estimated-cost.test.ts | ||
| parser-filter.test.ts | ||
| parser-gemini-cache.test.ts | ||
| parser-hydration-lock.test.ts | ||
| parser-incremental-append.test.ts | ||
| parser-large-json-scanner-parity.test.ts | ||
| parser-large-json-scanner.test.ts | ||
| parser-large-session.test.ts | ||
| parser-local-savings.test.ts | ||
| parser-mcp-inventory.test.ts | ||
| parser-number-helpers.test.ts | ||
| parser-proxy-codex-only.test.ts | ||
| parser-proxy-merge.test.ts | ||
| parser-proxy-pricing.test.ts | ||
| parser-rich-capture.test.ts | ||
| parser-single-pass-scanner.test.ts | ||
| parser-skip-line.test.ts | ||
| parser-subagent-collection.test.ts | ||
| parser-subagent-range.test.ts | ||
| parser.test.ts | ||
| plan-usage.test.ts | ||
| plans.test.ts | ||
| pricing-fallback-data.test.ts | ||
| project-filter-durable-totals.test.ts | ||
| provider-env-declarations.test.ts | ||
| provider-probe-roots-tier2.test.ts | ||
| provider-probe-roots.test.ts | ||
| provider-registry.test.ts | ||
| provider-turn-grouping.test.ts | ||
| quota.test.ts | ||
| scan-progress.test.ts | ||
| serve-stdio.test.ts | ||
| session-cache-rich-capture.test.ts | ||
| session-cache-shards.test.ts | ||
| session-cache-v5-adoption.test.ts | ||
| session-cache.test.ts | ||
| sessions-by-branch.test.ts | ||
| sessions-by-pr.test.ts | ||
| sessions-report.test.ts | ||
| sessions-subagent-fold.test.ts | ||
| spend-flow.test.ts | ||
| sync-attribution-cli.test.ts | ||
| sync-attribution.test.ts | ||
| sync-e2e.test.ts | ||
| sync-headless-e2e.test.ts | ||
| sync-infra-e2e.test.ts | ||
| sync-ledger-otlp.test.ts | ||
| sync-push.test.ts | ||
| sync.test.ts | ||
| token-estimate.test.ts | ||
| usage-aggregator-savings.test.ts | ||
| usage-aggregator.test.ts | ||
| web-dashboard.test.ts | ||
| wheel-scroll.test.ts | ||
| workflow-insights.test.ts | ||
| workflow-panel.test.ts | ||
| yield-overlap-attribution.test.ts | ||
| yield-repo-grouping.test.ts | ||
| yield.test.ts | ||