docs(grok): state the mixed-session drop and the global daily re-derivation plainly

The daily-cache re-derivation test seeded v18, a version that only ever
existed as an unreleased draft of this change. Seed the shipped v17 so the
test models the 17 -> 19 upgrade path users actually hit, and rename it: the
bump re-derives every day for every provider, not just Grok, because the
daily cache has no per-provider invalidation. The Grok day stays as the
fixture since Grok is what the bump exists to correct.

The changelog entry now says outright that Grok totals change materially on
upgrade (150K -> 96.3M cache-read tokens on a 568-session corpus), that a
turn without a turn_completed record inside an otherwise-covered session is
dropped rather than estimated, and that the one-time daily re-derivation
reads the warm session cache and keeps the superseded file. The
context-bloat denominator fix moves to Fixed and names the providers it
corrects.

docs/providers/grok.md gets the same undercount warning in the token model
and a matching entry under Quirks.

CHANGELOG.md

@@ -16,7 +16,7 @@
 - **DeepSeek Harness (`dsh`) is now a supported provider.** Reads DeepSeek's open-source agent harness from `~/.dsh/sessions` (`DSH_HOME` relocates the root), both the default zstd logs and the uncompressed `session.jsonl` variant. A `.zstd` log is a concatenation of independent zstd frames, one per write batch, so it is decoded frame by frame behind a structural frame scan and a torn trailing frame from a crashed writer is ignored rather than failing the file (needs Node 22.15+ for `zlib` zstd; below that dsh is skipped with a notice instead of counted as $0). One call per `(turn, step)`, with the step's final `assistant/message` usage superseding the streamed `assistant/chunk` sample of the same call rather than adding to it, the model taken from the message that served the step, and reasoning tokens billed at the output rate. DSH records tokens but no cost, so calls are priced from the shared tables. The events a forked session replays from its parent are skipped, since codeburn already counts the parent's own log. The session format is pinned at version 0 upstream with no compatibility implied, so a log stamped with any other version is skipped with a notice instead of read under today's assumptions.
 
 ### Changed
-- **Grok Build now uses the CLI's authoritative completed-turn usage.** `turn_completed.usage` records are deduplicated by prompt and emitted as one session-level call from verified top-level totals; `modelUsage` only selects a priced attribution id, so multi-model rate attribution remains deliberately out of scope. Reasoning is clamped to reported output before the exclusive output/reasoning split, and the old context-size heuristic remains for sessions without usable top-level usage. In mixed sessions, uncovered pre-upgrade turns are dropped and the row is marked estimated rather than claiming full provider coverage; already-cached Grok sessions re-parse once. (#998)
+- **Grok Build now reads the CLI's own completed-turn usage instead of estimating it.** Usage comes from the `turn_completed.usage` records Grok CLI already writes into `updates.jsonl` (`inputTokens`, `outputTokens`, `cachedReadTokens`, `cacheCreationTokens`, `reasoningTokens`), deduplicated by `prompt_id` and emitted as one session-level call from the top-level totals. The previous parser reconstructed an estimate from the running `_meta.totalTokens` context counter, so **existing Grok totals will change materially on upgrade** - on one real 568-session corpus cache-read went from 150K to 96.3M tokens, total tokens from 20.0M to 113.9M, and cost from $36.98 to $56.79. Cache read and cache creation are subsets of input and reasoning is a subset of output, so reasoning is clamped to the record's reported output and split back out to match this repo's exclusive-reasoning contract. `modelUsage` only selects a priced attribution id; multi-model rate attribution stays out of scope, so one session is priced at one model's rate. `costUsdTicks` is ignored because its scale is undocumented. Sessions with no usable record - older CLI versions - keep the old context-curve heuristic and stay flagged estimated. **In a session that has at least one `turn_completed` record, turns without one are not counted at all** (their tokens are dropped rather than estimated), and the session is marked estimated instead of claiming full provider coverage. Cached Grok sessions re-parse once. The daily cache re-derives once on first run after upgrade: this is a global re-derivation of every day and every provider, since the daily cache has no per-provider invalidation, but it reads the warm session cache rather than re-parsing transcripts, so it costs seconds (~3s on the corpus above), and the superseded cache file is retained on disk as the baseline for days no source can still re-derive. (#998)
 - **Codex rollouts parse across worker threads too, and the workload gate now takes bytes or files.** Codex is the bigger half of a real cold parse — a 4 GB rollout corpus against 1.8 GB of Claude sessions — and it was still decoding one file at a time. A whole-file rollout decode now runs on the same pool, against an empty dedup set, and comes back with the calls, the dedup keys it claimed, and the codex-cache entry it would have written; the parent installs all three in the serial loop's order, so `codex-results.json` and every payload come out byte-identical to a serial run. Cross-file state stays where it was: a forked rollout replaying its parent's token_count history collides on the parent's keys and is re-parsed in-process, and no worker ever touches the cache module's per-directory state. Files the Codex cache can serve exactly or resume into from a byte offset never reach a worker — they read a few KB and the resume state belongs to the parent. The workload gate is now pending BYTES alone (200 MB), not file count: 250 pending files holding under a megabyte between them spawned threads that made the run ~5% slower, while a few hundred huge rollouts were being turned away. The count takes `max(pendingFiles / 50, pendingBytes / 200 MB)`, and the per-thread memory budget is derived per parse as `clamp(256 MB, 2 × average pending file + 128 MB, 1 GB)` rather than a flat 256 MB — a 260 MB rollout peaks near 430 MB in its worker and scales linearly with the pool, so the flat figure over-subscribed exactly the workload this adds. The decision is per provider, and at most one pool is alive at a time.
 - **A large cold Claude parse now runs across worker threads.** Reading, decoding and line-parsing a session JSONL is per-file work that never touches anything shared, so it moves onto `worker_threads`; each worker ships its parsed turns back as a JSON string and the parent installs them in the exact order the serial loop would. Everything with cross-file state — the streaming-message dedup, canonical project paths, spawn links, PR correlation, progress saves — stays on the main thread, and a file whose message ids were already claimed by an earlier file (or whose worker failed) is simply re-parsed in-process, so the session cache and every payload are identical either way. On a 6 GB corpus a cold `status` drops from 27.5s to 14.8s with peak RSS up 2.27 GB → 2.52 GB. Threads only engage for a genuinely large cold parse: never with under 200 MB behind the pending whole-file re-parses, 2 or fewer cores, or under 4 GB of available memory — so warm and incremental runs are untouched and spawn nothing. Otherwise the count is `min(cores - 1, min(0.25 × available, 2 GB) / 256 MB, pendingFiles / 50)`, where available is `process.availableMemory()` (cgroup-aware in containers) rather than free memory, which on macOS reports free pages and would switch the feature on and off between runs. `CODEBURN_PARSE_WORKERS=0` forces the serial parse and `CODEBURN_PARSE_WORKERS=N` forces N (capped at the core count), both bypassing every gate; `CODEBURN_VERBOSE=1` prints the resolved count and why.
 - **A warm launch rewrites only the month that changed, and a ranged query reads only the months it can report on.** Per-provider shards still meant one appended session republished that provider's entire history — 95 MB for Claude on a 6 GB corpus. Each provider's shard is now split again by the UTC month of the cached session's FIRST turn, a bucket that never moves as a session grows, so an append rewrites one month. Every shard records the newest month it holds, which lets `--period today/week` skip the shards that cannot contribute a turn to the range; the skipped months stay on disk untouched across the save, and providers whose cache is the only surviving record (durable) or whose parse fingerprint moved are always read in full. Remaining shards are read concurrently. Existing v8 and v7 caches are re-laid-out losslessly on first load and the old layout removed once the new one is published: nothing re-parses.
@@ -30,6 +30,7 @@
 - **The resident `codeburn serve` child.** The first real panel request is also the cache warm-up, so startup never runs an artificial warm-up query beside a duplicate one-shot child; each served command carries its own read-only option allowlist, and anything outside it falls back to a normal spawn; the child exits when its stdin closes, so it can never outlive the app. Requests whose response exceeds the 16 MiB frame limit still replace the child, but that deliberate kill no longer spends the resident's unexpected-death budget. (#972)
 
 ### Fixed
+- **Context-bloat detection now counts reasoning tokens as generated output.** `detectContextBloat` divided context by `totalOutputTokens` alone, but reasoning is stored beside output rather than inside it, so for every reasoning-bearing provider the detector saw a fraction of the tokens actually generated and invented findings - a session whose real ratio was 20:1, under the 25:1 threshold, was reported as 133:1 and "high impact". It now uses the same `output + reasoning` sum the reports use, which corrects grok, codex, kiro, hermes, qwen and cursor-agent alike.
 - **The unpriced-models warning in the dashboard is now readable at every terminal width.** It lived in a fixed-width panel with an inline model list and a fix command, so it clipped mid-name at 80 columns and clipped *earlier* at 200, where the three-column layout narrows each panel - neither the affected models nor a runnable command survived. The panel line is now a pointer, `! N unpriced: codeburn models --unpriced` (shortened to `! N: codeburn models --unpriced` below 45 columns of panel), and the model list moves to that command's plain output, which is full width, copyable, and lists every model rather than the first two. The command's hint no longer reads as an unconditional instruction to alias: a subscription or flat-rate model is correctly $0, and mapping it onto another model's per-token rate would invent spend that was never billed. Provider-supplied model IDs are now stripped of terminal control characters in every human-readable report rather than only on the unpriced path, and `--unpriced` shows raw IDs instead of friendly names because `model-alias` keys on the raw ID. (#969)
 - **`codeburn models --unpriced --top N` returned nothing for a `--top N` smaller than the number of priced models.** `--top` is applied inside `aggregateModels`, before the unpriced filter, on rows sorted cost-first — and unpriced rows are $0 on both, so they sorted last and the slice removed exactly the rows the flag exists to show. A user with unpriced models was told they had none. The slice now runs after the filter — and after ranking, because unpriced rows tie at $0 on both keys, so slicing them in aggregate order kept whichever models happened to appear earliest in the transcript rather than the largest. The order now matches the one the unpriced-models warning shows. (#969)
 - **Old durable sources remain visible while they still exist.** The 90-day session-cache age-out now applies only after a durable source disappears from discovery, so an unchanged older Copilot source keeps reporting usage and reuses its persisted fingerprint instead of being reparsed and immediately discarded. (#987) On long-lived machines this makes previously dropped history reappear, so lifetime totals can jump once after upgrading.

docs/providers/grok.md

@@ -21,7 +21,7 @@ JSON + JSONL. `summary.json` holds the session id, cwd, timestamps, and `current
 
 **Estimated fallback.** Older sessions without any valid `turn_completed.usage` record use the running context fill (`totalTokens` per chunk) and the existing compaction-aware per-turn curve. That path remains flagged `costIsEstimated`; a completed record is never blended with the heuristic.
 
-If a session spans a CLI upgrade and has both completed records and streamed turns without a matching record, the all-or-nothing authoritative path keeps the recorded totals, drops the uncovered turns, and marks the emitted row `costIsEstimated: true`. A later parse can fill an open turn once it writes a record; pre-upgrade turns that never do are dropped.
+**Mixed sessions undercount.** The choice between the two paths is per session, not per turn. If a session has at least one usable `turn_completed` record, the whole session is billed from the summed records and any turn WITHOUT a record contributes nothing at all - its tokens are dropped, not estimated, so such a session reads low. The row is marked `costIsEstimated: true` rather than claiming full provider coverage. This is deliberate: blending the heuristic into real records would reintroduce the roughly 5x output over-count this parser exists to remove. It happens when a session straddles a CLI upgrade or a run dies before writing its last record; an open turn is filled by a later parse once it writes one, pre-upgrade turns never are. Measured on a 568-session corpus, 1 turn out of 566 was uncovered.
 
 ## Pricing
 
@@ -38,6 +38,7 @@ Per `grok:<session-dir>:<updated_at>:<id>`.
 ## Quirks
 
 - **Two token paths.** Completed turns carry provider usage; sessions from older CLI versions have only the context curve and therefore remain estimates (likely an upper bound, since re-sent context is cached server-side and not exposed in those files).
+- **A turn with no `turn_completed` record is dropped inside an otherwise-covered session** (see Token model). The session still reports, marked estimated, but reads low by those turns.
 - **No bash-command capture.** Tool names come from `signals.toolsUsed`; per-command bash text is not extracted, so `bashCommands` is empty.
 - **Whole-session timestamp.** Spend is attributed to `updated_at`, since the context curve is cumulative.
 - **Subscription vs API.** Grok Build runs via either a metered xAI API account (tiered) or a SuperGrok subscription; the session files do not record which.

src/daily-cache.ts

@@ -8,10 +8,17 @@ import type { DateRange, ProjectSummary } from './types.js'
 
 // Bumped to 19: Grok authoritative usage now keeps one session-level rollup
 // from top-level totals, clamps reasoning to reported output, and labels mixed
-// authoritative/heuristic coverage. Days already finalized at v18 contain the
-// per-model split from the previous draft, so raising MIN_SUPPORTED_VERSION
-// forces a one-time re-derivation; v14 carry-forward keeps source-less history
-// intact.
+// authoritative/heuristic coverage. Every day finalized under the previous
+// accounting carries the old Grok totals, and the daily cache has no
+// per-provider invalidation, so raising MIN_SUPPORTED_VERSION is the only
+// lever: it forces a one-time re-derivation of ALL days, for every provider,
+// not just Grok. That pass reads the warm session cache (CACHE_VERSION is
+// unchanged and only PROVIDER_PARSE_VERSIONS.grok moved), so it costs seconds
+// rather than a full re-parse, and adoptOlderDailyCaches keeps the superseded
+// file as the baseline for days no source can still re-derive.
+//
+// The shipped predecessor is v17; v18 was an unreleased draft of this change
+// and only exists in pre-release checkouts. 19 clears both.
 //
 // Bumped to 17: copilot CLI sessions were misclassified as VS Code transcripts
 // (#944), so days finalized at v16 or earlier carry output-only copilot costs —

tests/daily-cache-version-rederivation.test.ts

@@ -10,8 +10,12 @@ import {
   type DailyEntry,
 } from '../src/daily-cache.js'
 
-const PRE_FIX_DAILY_VERSION = 18
-const cacheRoot = join(tmpdir(), `codeburn-grok-daily-${process.pid}-${Date.now()}`)
+// The last SHIPPED daily-cache version before the Grok accounting change, so
+// this models the real 17 -> 19 upgrade path users hit. (v18 existed only as an
+// unreleased draft.) Anything below MIN_SUPPORTED_VERSION is untrusted, which
+// is what makes the re-derivation global rather than Grok-scoped.
+const PRE_FIX_DAILY_VERSION = 17
+const cacheRoot = join(tmpdir(), `codeburn-daily-rederive-${process.pid}-${Date.now()}`)
 
 function day(date: string, cost: number): DailyEntry {
   return {
@@ -63,8 +67,12 @@ afterEach(async () => {
   await rm(cacheRoot, { recursive: true, force: true })
 })
 
-describe('Grok daily-cache accounting rederivation', () => {
-  it('re-derives a v18 Grok day while preserving the old cache as the baseline', async () => {
+// Raising MIN_SUPPORTED_VERSION re-derives EVERY day from EVERY provider, not
+// only Grok - the daily cache has no per-provider invalidation. A Grok day is
+// used here because Grok is the provider whose totals the bump exists to
+// correct; the mechanism under test is version-wide.
+describe('daily-cache re-derivation on a DAILY_CACHE_VERSION bump', () => {
+  it('re-derives a day from a below-minimum v17 cache while preserving the old file', async () => {
     const date = toDateString(new Date(Date.now() - 7 * 24 * 60 * 60 * 1000))
     const yesterday = toDateString(new Date(Date.now() - 24 * 60 * 60 * 1000))
     const oldPath = join(cacheRoot, `daily-cache.v${PRE_FIX_DAILY_VERSION}.json`)