diff --git a/CHANGELOG.md b/CHANGELOG.md index c82301ab..a5a9dc2a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,7 @@ ### Fixed - **A date-ranged run no longer republishes the month shards it never read.** A scoped load leaves an out-of-range month on disk, so the files it holds have no visible cache entry and the reconcile re-parses them — re-deriving the entry the shard already stores. That re-parse marked the unloaded month dirty, and the save merged and republished it under a fresh nonce name on every single run, byte-identical content and all, so a repeated `codeburn status --format json` churned old months (on a real corpus: claude/2026-03, cursor/2026-02 and warp/2026-03 renamed every run) and left the retired shards for the sweeper. A merge into an unloaded month that neither adds, changes nor removes an entry now keeps the published shard, so unchanged months keep their names and their bytes. (#1032) +- **An upgrade no longer loses history for days whose transcripts have only PARTLY aged out.** The never-lose contract carried a cached (day, provider) slice forward only when the re-derivation found NOTHING for it, but transcripts expire per FILE rather than per day: on a day whose sources are mostly gone, a handful of turns from surviving later files still bucket onto it, so the fresh slice came back non-empty but truncated and REPLACED the full cached one. On a real cache upgrading from the last shipped daily-cache version, 2026-07-16 fell from $1,685.17 / 12,530 calls to $385.44 / 560 calls, and 13 days lost $2,765.75, 19,209 calls and 520 sessions in total. A fresh slice now replaces a settled baseline slice only when it carries at least as many CALLS - the same or more evidence; fewer calls means the source set demonstrably lost data, and the baseline is kept whole. The comparison is on calls alone: cost and tokens are re-priced accounting on the same evidence, which is exactly what a legitimate re-derivation changes (the Grok accounting fix keeps its per-day calls and is unaffected), and session counts drift down by a few on days whose sources are entirely intact. Days inside a 7-day settle window stay authoritative - their session files are still on disk, so a shrink there is a real change rather than expiry. The trade-off is deliberate and matches the direction this cache has always chosen: a future fix that legitimately REDUCES calls on a settled day keeps the older, higher value until that day is re-derived at an equal or greater call count. The timezone-change re-derive gets the exact form of the same rule - what the fresh parse can no longer explain under the old bucketing is added on top of the fresh slice instead of being dropped - and the cross-file adoption union is unchanged, where the newer schema still wins per (day, provider). - **The session chart legend now leads with a visible session disambiguator and title instead of the project path.** Every series in a monorepo shared the same project prefix, so the only thing separating them was a truncated hex fragment — and per-application cost attribution is the main reason to open that chart. `SessionSummary.title` is already parsed and already rendered in the Context tab; the legend now puts the short session id first, prefers the title, and falls back to the previous project-based label when a session never produced one. Titles come from transcripts, so they are stripped of ANSI and control characters and capped before they reach either the legend or the tooltip. (#997) - **Context-bloat detection now counts reasoning tokens as generated output.** `detectContextBloat` divided context by `totalOutputTokens` alone, but reasoning is stored beside output rather than inside it, so for every reasoning-bearing provider the detector saw a fraction of the tokens actually generated and invented findings - a session whose real ratio was 20:1, under the 25:1 threshold, was reported as 133:1 and "high impact". It now uses the same `output + reasoning` sum the reports use, which corrects grok, codex, kiro, hermes, qwen and cursor-agent alike. - **The unpriced-models warning in the dashboard is now readable at every terminal width.** It lived in a fixed-width panel with an inline model list and a fix command, so it clipped mid-name at 80 columns and clipped *earlier* at 200, where the three-column layout narrows each panel - neither the affected models nor a runnable command survived. The panel line is now a pointer, `! N unpriced: codeburn models --unpriced` (shortened to `! N: codeburn models --unpriced` below 45 columns of panel), and the model list moves to that command's plain output, which is full width, copyable, and lists every model rather than the first two. The command's hint no longer reads as an unconditional instruction to alias: a subscription or flat-rate model is correctly $0, and mapping it onto another model's per-token rate would invent spend that was never billed. Provider-supplied model IDs are now stripped of terminal control characters in every human-readable report rather than only on the unpriced path, and `--unpriced` shows raw IDs instead of friendly names because `model-alias` keys on the raw ID. (#969) diff --git a/src/daily-cache.ts b/src/daily-cache.ts index 91dc9b10..1eb19ce6 100644 --- a/src/daily-cache.ts +++ b/src/daily-cache.ts @@ -15,7 +15,11 @@ import type { DateRange, ProjectSummary } from './types.js' // not just Grok. That pass reads the warm session cache (CACHE_VERSION is // unchanged and only PROVIDER_PARSE_VERSIONS.grok moved), so it costs seconds // rather than a full re-parse, and adoptOlderDailyCaches keeps the superseded -// file as the baseline for days no source can still re-derive. +// file as the baseline for days no source can still re-derive. Days whose +// sources only PARTLY survive are held by the partial-survival guard in +// mergeDayEntries, which is what makes a global re-derive safe to force: on a +// real 108-day cache the 17 -> 19 pass moves Grok cost and tokens and nothing +// else - the day-by-day call counts come back identical. // // The shipped predecessor is v17; v18 was an unreleased draft of this change // and only exists in pre-release checkouts. 19 clears both. @@ -856,6 +860,52 @@ function hasPositiveDayContent(day: DailyEntry): boolean { return false } +/// PARTIAL SURVIVAL (the v14 never-lose contract, extended past all-or-nothing). +/// v14 protected a (date, provider) slice only when the fresh derivation found +/// NOTHING there. But transcripts age out per FILE, not per day: Claude Code +/// deletes them after ~30 days, and turn-anchored bucketing means a handful of +/// turns from surviving later files still land on a mostly-aged-out day. The +/// fresh slice is then non-empty but truncated, and replacing the baseline with +/// it silently deletes the rest (measured on a real cache upgrading 17 -> 19: +/// 2026-07-16 fell from $1,685.17 / 12,530 calls to $385.44 / 560 calls). +/// +/// So a fresh slice replaces a settled baseline slice only when it carries at +/// least as many CALLS — the same or more evidence. Fewer calls means the +/// source set demonstrably lost data, and the baseline is kept whole. +/// +/// Why calls and not sessions: session counts shrink routinely on days whose +/// sources are entirely intact (a session's turns re-attribute to a neighbouring +/// day), measured at 1-5 sessions on recent days whose call counts were +/// identical across the re-derivation. A sessions test would freeze stale slices +/// on healthy days. Why not cost/tokens: those are re-priced accounting on the +/// same evidence — exactly what a legitimate re-derivation changes (#1015 Grok +/// keeps its per-day calls and raises cost, and is unaffected by this guard). +/// +/// Why no source-set test instead: a day entry records no source files, counts +/// or fingerprints, and the session cache is keyed by file rather than by day, +/// so "were this day's sources all present?" cannot be answered from the cache. +/// The calls comparison is the available proxy. +/// +/// TRADE-OFF: a future fix that legitimately REDUCES calls on a settled day +/// (deduplication) is blocked, and that day keeps the older, higher value until +/// its slice is re-derived under an equal-or-greater call count. That is the +/// "estimate high, never lose" direction v14 already chose over silent loss. +/// +/// Recent days stay authoritative: within the settle window their session files +/// are still on disk, so a shrink there is a real change (the user deleted a +/// transcript), not aged-out sources. Seven days is far inside the ~30-day +/// retention floor of the shortest-lived source we know of, so any shrink older +/// than that is source loss with overwhelming likelihood. +const SETTLE_DAYS = 7 + +function settleCutoffDate(now: Date): string { + return toDateString(new Date(now.getFullYear(), now.getMonth(), now.getDate() - SETTLE_DAYS)) +} + +function isPartialSurvival(date: string, baseline: ProviderDaySlice, fresh: ProviderDaySlice, settleCutoff: string): boolean { + return date < settleCutoff && fresh.calls < baseline.calls +} + /// Index `freshUnderOldTz` (the same parse re-aggregated under the cache's OLD /// tzKey) by date then provider, so the merge can subtract exactly what the /// fresh parse still explains under the old bucketing. @@ -886,21 +936,34 @@ function buildTzSubtraction(days: DailyEntry[]): ReadonlyMap>, + /// Set ONLY by the complete-parse re-derive, where `primary` is a fresh + /// derivation from live sources and `secondary` is the cache baseline: a + /// primary slice with fewer calls there means sources aged out, so the + /// baseline wins on settled days (`isPartialSurvival`). The adoption union + /// leaves it off - both sides are cache generations there and the newer + /// schema deliberately wins per (date, provider). + guardPartialSurvival = false, ): DailyEntry[] { const byDate = new Map() + const settleCutoff = settleCutoffDate(new Date()) for (const day of primary) byDate.set(day.date, structuredClone(day)) for (const day of secondary) { const existing = byDate.get(day.date) @@ -927,7 +990,6 @@ export function mergeDayEntries( // day) still carry a real session count — worth preserving. if (!hasSliceData(slice) && !(slice.sessions ?? 0)) continue const existingSlice = Object.hasOwn(existing.providers, provider) ? existing.providers[provider] : undefined - if (existingSlice && hasSliceData(existingSlice)) continue let toAdd = slice let residual = false if (subtract) { @@ -943,6 +1005,13 @@ export function mergeDayEntries( residual = true } } + if (existingSlice && hasSliceData(existingSlice) && !residual) { + if (!guardPartialSurvival || !isPartialSurvival(day.date, slice, existingSlice, settleCutoff)) continue + // The baseline holds more evidence than the sources can still produce: + // swap the fresh slice back out for it (inverse of addSliceIntoDay, so + // the day's totals and nested maps stay reconciled with its slices). + subtractSliceFromDay(existing, provider, existingSlice) + } addSliceIntoDay(existing, provider, toAdd, residual) if (markSecondaryCarried) existing.carried = true } @@ -1105,7 +1174,7 @@ export async function ensureCacheHydrated( tzSubtraction = buildTzSubtraction(aggregateDaysInTz(wideProjects, c.tzKey)) } const merged = parseWasComplete - ? mergeDayEntries(freshDays, baseline, true, tzSubtraction) + ? mergeDayEntries(freshDays, baseline, true, tzSubtraction, true) : mergeDayEntries(baseline, freshDays, false) c = { version: DAILY_CACHE_VERSION, diff --git a/tests/daily-cache-carry-forward.test.ts b/tests/daily-cache-carry-forward.test.ts index a3056b80..78cfeab8 100644 --- a/tests/daily-cache-carry-forward.test.ts +++ b/tests/daily-cache-carry-forward.test.ts @@ -712,3 +712,102 @@ describe('adoption union across older cache files', () => { expect(hydrated.complete).toBe(true) }) }) + +describe('partial survival: a truncated fresh slice cannot delete a settled baseline', () => { + // The real 0.9.20 -> next upgrade loss: transcripts age out per FILE, so a + // mostly-forgotten day still gets a handful of turns from surviving later + // files. The fresh slice is non-empty but truncated, and before this guard it + // replaced the baseline outright ($1,685.17 / 12,530 calls -> $385.44 / 560). + const settled = daysAgoStr(33) + + it('keeps the baseline slice when the fresh derivation lost calls on a settled day', () => { + const fresh = day(settled, { claude: slice(385.44, 560, { sessions: 0, inputTokens: 10 }) }) + const baseline = day(settled, { claude: slice(1685.17, 12530, { sessions: 214, inputTokens: 500 }) }) + const merged = mergeDayEntries([fresh], [baseline], true, undefined, true) + const m = merged[0]! + expect(m.providers['claude']).toMatchObject({ cost: 1685.17, calls: 12530, sessions: 214 }) + // Day totals track the swap - they must equal the kept slice, not the sum. + expect(m.cost).toBeCloseTo(1685.17, 5) + expect(m.calls).toBe(12530) + expect(m.sessions).toBe(214) + expect(m.inputTokens).toBe(500) + expect(m.carried).toBe(true) + }) + + it('a truncated slice cannot take other providers down with it', () => { + const fresh = day(settled, { claude: slice(10, 5), codex: slice(7, 3) }) + const baseline = day(settled, { claude: slice(100, 50) }) + const m = mergeDayEntries([fresh], [baseline], true, undefined, true)[0]! + expect(m.providers['claude']).toMatchObject({ cost: 100, calls: 50 }) + expect(m.providers['codex']).toMatchObject({ cost: 7, calls: 3 }) + expect(m.cost).toBeCloseTo(107, 5) + expect(m.calls).toBe(53) + }) + + it('the fresh slice wins on equal calls at a different cost (the Grok re-pricing)', () => { + const fresh = day(settled, { grok: slice(11.89, 21, { sessions: 21, outputTokens: 900 }) }) + const baseline = day(settled, { grok: slice(3.29, 21, { sessions: 21, outputTokens: 200 }) }) + const m = mergeDayEntries([fresh], [baseline], true, undefined, true)[0]! + expect(m.providers['grok']).toMatchObject({ cost: 11.89, calls: 21, outputTokens: 900 }) + expect(m.cost).toBeCloseTo(11.89, 5) + expect(m.carried).toBeUndefined() + }) + + it('a fresh slice with FEWER sessions but equal calls still wins (sessions drift on healthy days)', () => { + const fresh = day(settled, { claude: slice(1021.11, 3295, { sessions: 71 }) }) + const baseline = day(settled, { claude: slice(1021.11, 3295, { sessions: 72 }) }) + const m = mergeDayEntries([fresh], [baseline], true, undefined, true)[0]! + expect(m.providers['claude']!.sessions).toBe(71) + }) + + it('recent days stay authoritative: a shrink inside the settle window is honored', () => { + const recent = daysAgoStr(2) + const fresh = day(recent, { claude: slice(20, 4) }) + const baseline = day(recent, { claude: slice(90, 40) }) + const m = mergeDayEntries([fresh], [baseline], true, undefined, true)[0]! + expect(m.providers['claude']).toMatchObject({ cost: 20, calls: 4 }) + expect(m.cost).toBe(20) + }) + + it('the adoption union is unguarded: the newer schema still wins per (date, provider)', () => { + const newer = day(settled, { claude: slice(50, 5) }) + const older = day(settled, { claude: slice(100, 10) }) + const m = mergeDayEntries([newer], [older], true)[0]! + expect(m.providers['claude']).toMatchObject({ cost: 50, calls: 5 }) + }) + + it('end-to-end: a version-bump re-derive whose transcripts aged out keeps the full day', async () => { + const cache: DailyCache = { + version: DAILY_CACHE_VERSION, + savingsConfigHash: 'cfg-A', + tzKey: currentTzKey(), + lastComputedDate: daysAgoStr(1), + days: [day(settled, { claude: slice(1685.17, 12530, { sessions: 214 }) })], + complete: false, // what a version bump / adoption leaves behind + } + await saveDailyCache(cache) + const truncated = [day(settled, { claude: slice(385.44, 560, { sessions: 0 }) })] + const out = await ensureCacheHydrated(noSessions, () => truncated, 'cfg-A') + const kept = out.days.find(d => d.date === settled)! + expect(kept.providers['claude']).toMatchObject({ cost: 1685.17, calls: 12530 }) + expect(kept.cost).toBeCloseTo(1685.17, 5) + // A --period all payload sums the kept slices, not the truncated ones. + const period = buildPeriodDataFromDays(out.days, 'all') + expect(period.cost).toBeCloseTo(1685.17, 5) + expect(period.calls).toBe(12530) + }) + + it('a fully sourceless day is still carried whole (v14 behavior, unchanged)', async () => { + const cache: DailyCache = { + version: DAILY_CACHE_VERSION, + savingsConfigHash: 'cfg-A', + tzKey: currentTzKey(), + lastComputedDate: daysAgoStr(1), + days: [day(settled, { claude: slice(230.06, 400) })], + complete: false, + } + await saveDailyCache(cache) + const out = await ensureCacheHydrated(noSessions, () => [], 'cfg-A') + expect(out.days[0]).toMatchObject({ date: settled, cost: 230.06, calls: 400, carried: true }) + }) +})