vrr/codeburn
2
0
Fork 0
mirror of https://github.com/AgentSeal/codeburn.git synced 2026-08-26 17:03:31 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 5
codeburn/tests/plan-usage.test.ts
Matthew Kelch a5a0ddaeac
feat(copilot): read per-request input/cache from session-store.db 
The Copilot CLI and the GitHub Copilot desktop app both write
~/.copilot/session-store.db unconditionally; its assistant_usage_events
table holds one row per API request. Until now input/cache tokens for
these surfaces came only from the session.shutdown rollups in
events.jsonl, which are written only on clean shutdown (a crash loses
the whole leg's input/cache accounting) and lump each session leg into
one per-model total. The rollup also RESETS its counters at in-session
compaction (traced on a clean single-process 107-request session whose
sole rollup covered exactly its five post-compaction requests), so even
cleanly-closed long sessions were truncated; on a long-history machine
the store recovered ~35% of real Copilot spend lost to crashes and
compaction resets. The DB rows are per-request, crash-proof, and carry
real timestamps.

The store's input_tokens is cache-INCLUSIVE (input + cache_read +
cache_write), the same convention as the shutdown rollups — verified
against each row's token_details_json and by reconciling per-session
sums against the CLI's own footers and rollups across two machines
(1,380+ rows, 8 models, CLI 1.0.70–1.0.79, schema_version 6): every
divergence was a rollup gap. Emitted calls mirror the shutdown-call
contract: input/cache/reasoning only, output 0 — per-turn output stays
owned by the events.jsonl assistant.message calls.

Rollup-vs-store precedence is RECONCILED at serve time, per
(session, model), and only there. Both representations always parse and
cache; parseProviderSources aggregates the cached calls and, wherever
store rows exist for a (session, model), drops the rollup calls and
serves the rows plus per-leg RESIDUAL calls: each rollup leg subtracts
only the rows in its own interval — rows commit strictly before their
leg's shutdown line, so a leg at time T covers exactly the rows in
(previous leg's T, T] — and any remainder (per token component, floored
at zero) serves once at that leg's own timestamp. A store missing
requests a leg covered — adopted mid-session, rows pruned before ever
being read — therefore still serves that tail exactly once ON THAT
LEG'S DAY, a crash-tail row the rollup never saw can never cancel it,
and a complete store serves pure per-request granularity with every
residual retired to zero. The decision reads only cached contents, never discovery:
deleting or resetting the store changes nothing served, so finalized
daily history can never flip on an absence epoch; cached rows of a
deleted store remain the record until the 90-day orphan age-out (which
exempts still-discovered paths). The serve set is the one coherent
snapshot — nothing a writer does between discovery and a parse can
change what one pass sees — and read-time precedence heals persisted
duplication (stale epochs, runtimes without node:sqlite, restored
files) instead of preserving it, following the buildDurablePeriod
pattern.

Store rows and rollups carry supplementary accounting weight. A rollup
(or its residual) is aggregate accounting, never a request: zero
api-call/model-call/turn weight, tokens and cost fully retained. A
store row is one real request, but when it pairs with a served per-turn
call it is supplementary too; rows pair with same-model per-turn calls
by timestamp adjacency (monotone matching, tight 2-minute window — the
two are written at the same completion moment, and a wide window would
let a crash-only row pair against a neighbor whose own row is missing),
computed once over the FULL serve set so a date-range boundary that
separates a row from its call cannot double the request across adjacent
day queries. Only the unpaired rows — store-only requests, exactly
where crash-lost requests sit — count. Supplementary-only turns fold
into the nearest behavioral turn within 30 minutes; with no behavioral
turn to fold into they stay separate weightless turns, each on its own
day, with apiCalls 0 — and the session emission gate admits
usage-bearing zero-call sessions. The weight
propagates into the daily cache: aggregateProjectsIntoDays applies the
same rule to every calls counter and category-turn count it seals, so
v19 history and live summaries can never disagree about what was a
request.

A changed source whose read defers on the busy shape (locked, EACCES,
corrupt mid-replace — discovery still emits the source; only true
absence or a schema mismatch reads as absent) now marks session
hydration incomplete, so the daily backfill holds its watermark instead
of finalizing a day the deferred rows never reached; an unchanged
unreadable store defers nothing. The verdict travels with its result —
the 180s memo and the serve burst-reuse restore the hydration verdict
their cached data was parsed under, so a memoized partial parse cannot
inherit a later parse's complete — and a discovered source whose
FINGERPRINT cannot be read (EACCES on a present file) defers instead of
silently skipping, while a genuinely deleted file stays a silent skip. Copilot reasoning tokens are no longer
double-billed at the report layer: they are a subset of the output the
per-turn calls already price, and copilot joins claude in the
reasoning-inside-output case of the query-time cost recompute.

Store dedup keys are content-discriminated —
copilot-store:<sid>:<rowId>:<fnv1a64(created_at|tokens|model)> —
because AUTOINCREMENT prevents id reuse only within one database
lifetime: a same-path DB reset reusing row ids now mints new keys
instead of the durable union swallowing the new usage, while a
byte-identical re-insert still collapses (64-bit: 32-bit FNV
collisions between plausible token tuples are constructible). Every
call of a session serves under one project label resolved at serve
time — the session-state-derived label when the serve set knows it,
else the store rows' own — so neither rows cached before events.jsonl
existed nor an events.jsonl orphaned by a session-state prune can
split the session across two grouping keys.

CODEBURN_COPILOT_SESSION_STORE_DB is read but deliberately NOT
fingerprinted, per the #927 ruling (any copilot fingerprint change
drops cached entries whose path still exists, destroying pruned history
only the cache holds); the read is allowlisted in the #927 guard, and
serve-time reconciliation makes repointing safe without a fingerprint —
the new store's rows parse on sight and the old path's entries persist
as durable orphans. The copilot parse version appends session-store-v2
and the daily cache bumps v17 → v19: per-day attribution, call counts
and costs all change against pre-store builds. 19, not 18: an earlier
pushed head of this PR already claimed v18 under different accounting,
and the carry-forward would adopt those days as finalized without
re-deriving them.

Verified by A/B on snapshots of two real stores, a live SIGKILL crash
test (row present, no rollup, tokens recovered exactly), live resumes
whose warm-cache deltas matched new rows to the token, upgrade-healing
at 4,800-session scale, and serve-level regressions pinning every
maintainer finding from six review rounds: the rows-then-shutdown race,
stale-cache healing, age-out exemption, absence-epoch identity,
progressive row landing with residual retirement, behavioral weight
across all four pinned scenarios, the hydration fence, project
unification in both directions, the same-path reset, mixed
coverage (crash tail vs covered-leg gap), multi-leg residual day
attribution, range-invariant pairing, memo-scoped hydration verdicts,
and the fingerprint-failure fence.
2026-08-17 17:52:01 -04:00

412 lines
13 KiB
TypeScript
Raw Blame History

import { mkdtemp, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { savePlan, type Plan } from '../src/config.js'
import { activePlansFromMap, computePeriodFromResetDay, getPlanScopedProjects, getPlanUsage, getPlanUsageFromProjects, getPlanUsages } from '../src/plan-usage.js'
import type { ProjectSummary } from '../src/types.js'
const { parseAllSessionsMock } = vi.hoisted(() => ({
parseAllSessionsMock: vi.fn(),
}))
vi.mock('../src/parser.js', () => ({
parseAllSessions: parseAllSessionsMock,
}))
describe('computePeriodFromResetDay', () => {
it('uses current month when today is on/after reset day', () => {
const { periodStart, periodEnd } = computePeriodFromResetDay(1, new Date('2026-04-17T10:00:00.000Z'))
expect(periodStart.getFullYear()).toBe(2026)
expect(periodStart.getMonth()).toBe(3)
expect(periodStart.getDate()).toBe(1)
expect(periodEnd.getMonth()).toBe(4)
expect(periodEnd.getDate()).toBe(1)
})
it('uses previous month when today is before reset day', () => {
const { periodStart, periodEnd } = computePeriodFromResetDay(15, new Date('2026-04-03T10:00:00.000Z'))
expect(periodStart.getMonth()).toBe(2)
expect(periodStart.getDate()).toBe(15)
expect(periodEnd.getMonth()).toBe(3)
expect(periodEnd.getDate()).toBe(15)
})
it('clamps reset day into 1..28', () => {
const { periodStart } = computePeriodFromResetDay(99, new Date('2026-04-27T10:00:00.000Z'))
expect(periodStart.getDate()).toBe(28)
})
})
describe('getPlanScopedProjects supplementary accounting', () => {
const plan: Plan = { id: 'custom', monthlyUsd: 100, provider: 'all', resetDay: 1, setAt: '2026-08-01T00:00:00.000Z' }
const today = new Date('2026-08-10T12:00:00.000Z')
function copilotCall(costUSD: number, timestamp: string, supplementaryAccounting: boolean) {
return {
provider: 'copilot',
model: 'claude-sonnet-4-5',
usage: {
inputTokens: supplementaryAccounting ? 40 : 0,
outputTokens: 0,
cacheCreationInputTokens: 0,
cacheReadInputTokens: 0,
cachedInputTokens: 0,
reasoningTokens: 0,
webSearchRequests: 0,
},
costUSD,
tools: [],
mcpTools: [],
skills: [],
hasAgentSpawn: false,
hasPlanMode: false,
speed: 'standard',
timestamp,
bashCommands: [],
deduplicationKey: `copilot-${timestamp}`,
supplementaryAccounting,
}
}
it('weighs calls behaviorally and keeps a cost-bearing zero-call session', () => {
const scoped = getPlanScopedProjects(plan, [
{
project: 'codeburn',
projectPath: '/tmp/codeburn',
totalCostUSD: 1.75,
totalApiCalls: 2,
sessions: [
{
// One real request served alongside its paired store row.
turns: [{
timestamp: '2026-08-05T12:00:00.000Z',
assistantCalls: [
copilotCall(1.0, '2026-08-05T12:00:00.000Z', false),
copilotCall(0.5, '2026-08-05T12:00:05.000Z', true),
],
}],
},
{
// Rollup-only session: real spend, zero behavioral requests.
turns: [{
timestamp: '2026-08-06T12:00:00.000Z',
assistantCalls: [copilotCall(0.25, '2026-08-06T12:00:00.000Z', true)],
}],
},
],
},
] as ProjectSummary[], today)
expect(scoped).toHaveLength(1)
expect(scoped[0]!.sessions.map(session => session.apiCalls)).toEqual([1, 0])
expect(scoped[0]!.sessions.map(session => session.totalCostUSD)).toEqual([1.5, 0.25])
expect(scoped[0]!.totalApiCalls).toBe(1)
expect(scoped[0]!.totalCostUSD).toBeCloseTo(1.75, 10)
expect(getPlanUsageFromProjects(plan, scoped, today).spentApiEquivalentUsd).toBeCloseTo(1.75, 10)
})
})
describe('getPlanUsage', () => {
beforeEach(() => {
parseAllSessionsMock.mockReset()
})
it('passes provider filter from plan and computes status', async () => {
parseAllSessionsMock.mockResolvedValue([
{
totalCostUSD: 160,
sessions: [],
},
])
const usage = await getPlanUsage({
id: 'claude-max',
monthlyUsd: 200,
provider: 'claude',
resetDay: 1,
setAt: '2026-04-01T00:00:00.000Z',
}, new Date('2026-04-10T10:00:00.000Z'))
expect(parseAllSessionsMock).toHaveBeenCalledWith(
expect.objectContaining({ start: expect.any(Date), end: expect.any(Date) }),
'claude',
)
expect(usage.spentApiEquivalentUsd).toBe(160)
expect(usage.percentUsed).toBe(80)
expect(usage.status).toBe('near')
})
it('projects using median daily spend (not mean)', async () => {
const dailyCosts = [1, 100, 1, 100, 1, 100, 1]
const turns = dailyCosts.map((cost, idx) => ({
timestamp: `2026-04-${String(idx + 1).padStart(2, '0')}T12:00:00.000Z`,
assistantCalls: [{ costUSD: cost }],
}))
parseAllSessionsMock.mockResolvedValue([
{
totalCostUSD: dailyCosts.reduce((sum, value) => sum + value, 0),
sessions: [{ turns }],
},
])
const usage = await getPlanUsage({
id: 'custom',
monthlyUsd: 500,
provider: 'all',
resetDay: 1,
setAt: '2026-04-01T00:00:00.000Z',
}, new Date('2026-04-07T12:00:00.000Z'))
// Median(1,100,1,100,1,100,1) = 1, so remaining 23 days adds 23.
expect(Math.round(usage.projectedMonthUsd)).toBe(327)
expect(parseAllSessionsMock).toHaveBeenCalledWith(
expect.objectContaining({ start: expect.any(Date), end: expect.any(Date) }),
'all',
)
})
it('computes plan usage from pre-fetched projects', () => {
const usage = getPlanUsageFromProjects({
id: 'custom',
monthlyUsd: 100,
provider: 'all',
resetDay: 1,
setAt: '2026-04-01T00:00:00.000Z',
}, [
{
totalCostUSD: 40,
sessions: [
{
turns: [
{ timestamp: '2026-04-02T12:00:00.000Z', assistantCalls: [{ costUSD: 20 }] },
{ timestamp: '2026-04-03T12:00:00.000Z', assistantCalls: [{ costUSD: 20 }] },
],
},
],
},
], new Date('2026-04-10T10:00:00.000Z'))
expect(usage.spentApiEquivalentUsd).toBe(40)
expect(usage.budgetUsd).toBe(100)
expect(usage.status).toBe('under')
})
it('projects month-end spend from API call timestamps', () => {
const usage = getPlanUsageFromProjects({
id: 'custom',
monthlyUsd: 100,
provider: 'all',
resetDay: 1,
setAt: '2026-04-01T00:00:00.000Z',
}, [
{
project: 'codeburn',
projectPath: '/tmp/codeburn',
totalCostUSD: 10,
totalApiCalls: 1,
sessions: [
{
turns: [
{
timestamp: '2026-03-31T23:59:00.000Z',
assistantCalls: [{ costUSD: 10, timestamp: '2026-04-01T10:00:00.000Z' }],
},
],
},
],
},
] as ProjectSummary[], new Date('2026-04-01T12:00:00.000Z'))
expect(Math.round(usage.projectedMonthUsd)).toBe(300)
})
it('returns active plans in provider display order', () => {
const plans = activePlansFromMap({
codex: {
id: 'custom',
monthlyUsd: 200,
provider: 'codex',
resetDay: 1,
setAt: '2026-04-01T00:00:00.000Z',
},
claude: {
id: 'claude-max',
monthlyUsd: 200,
provider: 'claude',
resetDay: 1,
setAt: '2026-04-01T00:00:00.000Z',
},
cursor: {
id: 'none',
monthlyUsd: 0,
provider: 'cursor',
resetDay: 1,
setAt: '2026-04-01T00:00:00.000Z',
},
})
expect(plans.map(plan => plan.provider)).toEqual(['claude', 'codex'])
})
it('keeps the provider-specific parser filter for one active plan', async () => {
const dir = await mkdtemp(join(tmpdir(), 'codeburn-plan-usage-test-'))
process.env['HOME'] = dir
try {
await savePlan({
id: 'claude-max',
monthlyUsd: 200,
provider: 'claude',
resetDay: 1,
setAt: '2026-04-01T00:00:00.000Z',
})
parseAllSessionsMock.mockResolvedValue([
{
project: 'codeburn',
projectPath: '/tmp/codeburn',
totalCostUSD: 80,
totalApiCalls: 1,
sessions: [],
},
] satisfies ProjectSummary[])
const usages = await getPlanUsages(new Date('2026-04-10T12:00:00.000Z'))
expect(parseAllSessionsMock).toHaveBeenCalledTimes(1)
expect(parseAllSessionsMock).toHaveBeenCalledWith(
expect.objectContaining({ start: expect.any(Date), end: expect.any(Date) }),
'claude',
)
expect(usages).toHaveLength(1)
expect(usages[0]?.spentApiEquivalentUsd).toBe(80)
} finally {
await rm(dir, { recursive: true, force: true })
}
})
it('computes multiple active plan usages from one all-provider parse', async () => {
const dir = await mkdtemp(join(tmpdir(), 'codeburn-plan-usage-test-'))
process.env['HOME'] = dir
try {
await savePlan({
id: 'claude-max',
monthlyUsd: 200,
provider: 'claude',
resetDay: 1,
setAt: '2026-04-01T00:00:00.000Z',
})
await savePlan({
id: 'custom',
monthlyUsd: 100,
provider: 'codex',
resetDay: 1,
setAt: '2026-04-01T00:00:00.000Z',
})
parseAllSessionsMock.mockResolvedValue([
{
project: 'codeburn',
projectPath: '/tmp/codeburn',
totalCostUSD: 150,
totalApiCalls: 2,
sessions: [
{
sessionId: 'session-1',
project: 'codeburn',
firstTimestamp: '2026-04-03T10:00:00.000Z',
lastTimestamp: '2026-04-03T11:00:00.000Z',
totalCostUSD: 150,
totalInputTokens: 0,
totalOutputTokens: 0,
totalCacheReadTokens: 0,
totalCacheWriteTokens: 0,
apiCalls: 2,
modelBreakdown: {},
toolBreakdown: {},
mcpBreakdown: {},
bashBreakdown: {},
categoryBreakdown: {},
skillBreakdown: {},
turns: [
{
userMessage: 'work',
timestamp: '2026-04-03T10:00:00.000Z',
sessionId: 'session-1',
category: 'coding',
retries: 0,
hasEdits: true,
assistantCalls: [
{
provider: 'claude',
model: 'claude-opus-4-7',
usage: {
inputTokens: 0,
outputTokens: 0,
cacheCreationInputTokens: 0,
cacheReadInputTokens: 0,
cachedInputTokens: 0,
reasoningTokens: 0,
webSearchRequests: 0,
},
costUSD: 100,
tools: [],
mcpTools: [],
skills: [],
hasAgentSpawn: false,
hasPlanMode: false,
speed: 'standard',
timestamp: '2026-04-03T10:00:00.000Z',
bashCommands: [],
deduplicationKey: 'claude-1',
},
{
provider: 'codex',
model: 'gpt-5.5',
usage: {
inputTokens: 0,
outputTokens: 0,
cacheCreationInputTokens: 0,
cacheReadInputTokens: 0,
cachedInputTokens: 0,
reasoningTokens: 0,
webSearchRequests: 0,
},
costUSD: 50,
tools: [],
mcpTools: [],
skills: [],
hasAgentSpawn: false,
hasPlanMode: false,
speed: 'standard',
timestamp: '2026-04-03T11:00:00.000Z',
bashCommands: [],
deduplicationKey: 'codex-1',
},
],
},
],
},
],
},
] satisfies ProjectSummary[])
const usages = await getPlanUsages(new Date('2026-04-10T12:00:00.000Z'))
expect(parseAllSessionsMock).toHaveBeenCalledTimes(1)
expect(parseAllSessionsMock).toHaveBeenCalledWith(
expect.objectContaining({ start: expect.any(Date), end: expect.any(Date) }),
'all',
)
expect(usages.map(usage => usage.plan.provider)).toEqual(['claude', 'codex'])
expect(usages.map(usage => usage.spentApiEquivalentUsd)).toEqual([100, 50])
} finally {
await rm(dir, { recursive: true, force: true })
}
})
})
Reference in a new issue View git blame Copy permalink