From 231e7cc0d4645ce1044ad5f1ec4c5e1a53e6fd3b Mon Sep 17 00:00:00 2001 From: AgentSeal Date: Thu, 9 Jul 2026 21:04:56 +0200 Subject: [PATCH 1/4] feat(report): surface unpriced models across overview, JSON, dashboard, and MCP Models missing from the pricing tables silently contributed $0 to every total with no indication anywhere (the warning was gated behind CODEBURN_VERBOSE). Add render-time detection that flags aggregated model rows with usage but $0 cost whose pricing lookup fails right now, and surface it in overview, report/today/month JSON (unpricedModels), the TUI By Model panel, and the MCP get_usage summary. Render-time detection covers cached sessions (cost is computed at parse time) and heals as soon as pricing data, an alias, or a price override arrives. Rows with real cost are never flagged: aggregation keys rows by display name, which the pricing lookup misses; $0 display-name rows are reverse-resolved to their raw id before flagging so sessions cached before their model's pricing landed don't get a misleading alias hint. Local models and model-savings mappings stay excluded: $0 is correct for them. Fixes #638 --- README.md | 88 +++++++++++++++++++++++++++++++++++++++++ src/dashboard.tsx | 13 +++++- src/main.ts | 12 +++++- src/mcp/tables.ts | 4 ++ src/menubar-json.ts | 8 ++++ src/models.ts | 65 ++++++++++++++++++++++++++++++ src/overview.ts | 13 +++++- src/usage-aggregator.ts | 11 ++++-- tests/models.test.ts | 71 +++++++++++++++++++++++++++++++++ tests/overview.test.ts | 33 ++++++++++++++++ 10 files changed, 311 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 3fa2f1d8..8cab6b46 100644 --- a/README.md +++ b/README.md @@ -34,8 +34,11 @@ Everything runs locally. No wrapper, no proxy, no API keys, nothing leaves your

Quick start · Find waste · + Apply fixes · + Guard · Compare models · Track what shipped · + MCP · Supported tools · Commands · Features · @@ -66,6 +69,8 @@ Also runs via `bunx codeburn` or `dx codeburn`, or `brew install codeburn` on ma codeburn menubar ``` +On Linux, a GNOME Shell extension gives the same panel view; see [Linux (GNOME)](#linux-gnome). + Requires **Node.js 22.13+** and at least one supported tool with session data on disk. For Cursor and OpenCode, `better-sqlite3` installs automatically. ## Your month at a glance @@ -131,6 +136,39 @@ Each finding shows the estimated token and dollar savings plus a ready-to-paste You can also open it inline from the dashboard: press `o` when a finding count appears in the status bar, `b` to return. +## Apply fixes, undo anytime + +```bash +codeburn optimize --apply # review and apply fixes interactively +codeburn optimize --apply --dry-run # print the plan, change nothing +codeburn optimize --apply --yes # apply every appliable fix without prompting +codeburn act list # every change CodeBurn has made +codeburn act undo --last # roll the most recent change back +codeburn act report # realized vs estimated savings +``` + +`codeburn optimize` finds the waste; `--apply` fixes the config-class findings for you: settings values, environment variables, archiving unused agents and skills. Every change is backed up and journaled before it lands. `codeburn act list` shows the history and `codeburn act undo ` restores the original files (it refuses if the files changed since being applied, unless you pass `--force`). + +The loop closes on honesty: once an applied fix is at least 3 days old, `codeburn act report` compares its estimated savings against what your sessions actually did, and later `codeburn optimize` runs show that realized figure in the header. Estimates get checked against reality, not just claimed. + +## Guard your budget + +```bash +codeburn guard install # hooks into this project's .claude/settings.json +codeburn guard install --global # or into ~/.claude/settings.json +codeburn guard status # caps, install locations, flagged projects +codeburn guard uninstall # removes cleanly, leaves your own hooks alone +``` + +Guard installs opt-in hooks into Claude Code that watch session cost while you work: + +- **Soft cap** (default $5): a one-time in-session warning when a session passes it. +- **Hard cap** (default $15): stops the session; `codeburn guard allow` lifts it for that session only. +- **Checkpoint** (default $3): if a session ends past this with no edits and no commits, a nudge suggests starting fresh with a named deliverable. +- **Session openers**: projects where optimize found waste get a one-line flag at session start. + +Caps are edited in `~/.config/codeburn/guard.json` (set a value to `null` to disable it). Add `--statusline` to show session cost in the Claude Code status line. Installs go through the same journal as everything else, so `codeburn act undo` removes them too. Hooks fail open: a broken guard never blocks a session. + ## Compare models ```bash @@ -237,6 +275,33 @@ defaults write org.agentseal.codeburn-menubar CodeBurnMenubarCompact -bool true Relaunch the app to apply. To revert: `defaults delete org.agentseal.codeburn-menubar CodeBurnMenubarCompact`. +### Linux (GNOME) + +Linux gets the same ambient view through a GNOME Shell extension (GNOME 45+): spend in the top panel, period switcher, compact mode, and daily budget alerts. It lives in [`gnome/`](gnome/): + +```bash +git clone https://github.com/getagentseal/codeburn && cd codeburn/gnome +./install.sh +gnome-extensions enable codeburn@codeburn.dev +``` + +See [gnome/README.md](gnome/README.md) for settings and development notes. On Windows, `codeburn web` is the always-on view for now. + +## CodeBurn in your agent (MCP) + +```bash +claude mcp add codeburn -- npx -y codeburn mcp +``` + +`codeburn mcp` runs a local MCP server over stdio, so Claude Code, Cursor, or any MCP client can ask "where did my tokens go this week?" or "how do I spend less?" mid-conversation. It exposes two tools: + +| Tool | What it returns | +|------|-----------------| +| `get_usage` | Spend and usage with breakdowns by tool, model, project, and task (fast) | +| `get_savings` | Cost reductions: waste findings, retry tax, routing waste (slower, deeper analysis) | + +Everything is read from local disk, same as the CLI. Project names are pseudonymized by default; the agent only sees real names if it asks with `include_project_names: true`. For other MCP clients, configure a stdio server with command `npx` and args `-y codeburn mcp`. + ## Supported tools CodeBurn auto-detects which AI tools you use. Each logo links to its provider doc. @@ -336,6 +401,19 @@ Run `codeburn` for the dashboard, or use a subcommand below. Most commands also | `codeburn yield` | Productive vs reverted/abandoned spend, correlated against git | | `codeburn yield -p 30days` | Yield analysis for the last 30 days | +**Fix & control** + +| Command | What it does | +|---------|--------------| +| `codeburn optimize --apply` | Interactively apply config-class fixes (`--yes`, `--dry-run`, `--only `) | +| `codeburn act list` | Every change CodeBurn has applied, newest first | +| `codeburn act undo ` | Roll a change back (`--last` for the most recent, `--force` if files drifted) | +| `codeburn act report` | Realized vs estimated savings for applied fixes | +| `codeburn guard install` | Budget-cap hooks for Claude Code (`--global`, `--statusline`) | +| `codeburn guard status` | Show caps, install locations, and flagged projects | +| `codeburn guard allow` | Lift the hard cap for the current session | +| `codeburn mcp` | MCP server (stdio) exposing usage and savings to AI agents | + **Models** | Command | What it does | @@ -428,6 +506,16 @@ codeburn model-alias --remove "my-proxy-model" # remove alias Aliases are stored in `~/.config/codeburn/config.json` and applied at runtime before pricing lookup. The target name can be anything in the [LiteLLM model list](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json) or a canonical name from the fallback table (e.g. `claude-sonnet-4-6`, `claude-opus-4-5`, `gpt-4o`). Built-in aliases ship for known proxy model name variants. User-configured aliases take precedence over built-ins. +### Local Models, Custom Prices, and Proxies + +```bash +codeburn price-override my-model --input 0.27 --output 1.10 # USD per 1M tokens +codeburn model-savings "llama3.1:8b" gpt-4o # local model, counted as savings +codeburn proxy-path ~/work/copilot-repo # subscription-covered project +``` + +`price-override` sets exact rates for any model (input, output, cache read, cache creation), useful for private deployments or models LiteLLM prices wrong. `model-savings` maps a free local model to a paid baseline: the local calls stay $0, and the dashboard shows what the same tokens would have cost on the baseline. `proxy-path` marks a project routed through a subscription-backed proxy (e.g. Claude Code over GitHub Copilot), so its API-rate cost is reported as subscription-covered and your net out-of-pocket stays honest. All three support `--list` and `--remove`. + ### Filtering ```bash diff --git a/src/dashboard.tsx b/src/dashboard.tsx index 62156173..064be43f 100644 --- a/src/dashboard.tsx +++ b/src/dashboard.tsx @@ -6,7 +6,7 @@ import { CATEGORY_LABELS, type DateRange, type ProjectSummary, type TaskCategory import { formatCost, formatTokens } from './format.js' import { aggregateModelEfficiency } from './model-efficiency.js' import { parseAllSessions, filterProjectsByName } from './parser.js' -import { loadPricing } from './models.js' +import { findUnpricedModels, loadPricing } from './models.js' import { getAllProviders } from './providers/index.js' import { scanAndDetect, type WasteFinding, type WasteAction, type OptimizeResult } from './optimize.js' import { estimateContextBudget, type ContextBudget } from './context-budget.js' @@ -358,6 +358,12 @@ function ModelBreakdown({ projects, pw, bw }: { projects: ProjectSummary[]; pw: } const sorted = Object.entries(modelTotals).sort(([, a], [, b]) => b.costUSD - a.costUSD) const maxCost = sorted[0]?.[1]?.costUSD ?? 0 + const unpriced = findUnpricedModels(Object.entries(modelTotals).map(([model, d]) => ({ + model, + calls: d.calls, + cost: d.costUSD, + tokens: d.freshInput + d.cacheRead + d.cacheWrite, + }))) return ( @@ -381,6 +387,11 @@ function ModelBreakdown({ projects, pw, bw }: { projects: ProjectSummary[]; pw: ) })} + {unpriced.length > 0 && ( + + {`! ${unpriced.length} model${unpriced.length === 1 ? '' : 's'} unpriced at $0 (${unpriced.slice(0, 2).map(u => u.model).join(', ')}${unpriced.length > 2 ? ', ...' : ''}) - fix: codeburn model-alias`} + + )} ) } diff --git a/src/main.ts b/src/main.ts index cc992670..a159408f 100644 --- a/src/main.ts +++ b/src/main.ts @@ -2,7 +2,7 @@ import { isAbsolute } from 'path' import { Command } from 'commander' import { installMenubarApp } from './menubar-installer.js' import { exportCsv, exportJson, type PeriodExport } from './export.js' -import { loadPricing, setModelAliases, setPriceOverrides, setLocalModelSavings, setProxyPaths, normalizeProxyPath } from './models.js' +import { findUnpricedModels, loadPricing, setModelAliases, setPriceOverrides, setLocalModelSavings, setProxyPaths, normalizeProxyPath } from './models.js' import { parseAllSessions, filterProjectsByName, filterProjectsByDateRange, clearSessionCache } from './parser.js' import { allProviderNames } from './providers/index.js' import { convertCost } from './currency.js' @@ -460,6 +460,16 @@ function buildJsonReport(projects: ProjectSummary[], period: string, periodKey: daily, projects: projectList, models, + // Models with recorded usage that resolve to no pricing data right now + // (#638). Their calls contribute $0 to every cost figure above, so + // consumers can tell "cheap" from "uncounted". Empty when all models + // priced. Fix entries via `codeburn model-alias` or `price-override`. + unpricedModels: findUnpricedModels(Object.entries(modelMap).map(([model, d]) => ({ + model, + calls: d.calls, + cost: d.cost, + tokens: d.inputTokens + d.outputTokens + d.cacheReadTokens + d.cacheWriteTokens, + }))), activities, tools: sortedMap(toolMap), mcpServers: sortedMap(mcpMap), diff --git a/src/mcp/tables.ts b/src/mcp/tables.ts index cfcb9957..6ce27bec 100644 --- a/src/mcp/tables.ts +++ b/src/mcp/tables.ts @@ -15,9 +15,13 @@ const oneShot = (r: number | null) => (r === null ? 'n/a' : pct(r * 100)) export function renderSummaryTable(p: MenubarPayload): string { const c = p.current + const unpriced = c.unpricedModels ?? [] return [ `**${c.label}** — ${formatCost(c.cost)} · ${c.calls} calls · ${c.sessions} sessions`, `cache hit ${pct(c.cacheHitPercent)} · one-shot ${oneShot(c.oneShotRate)} · in ${formatTokens(c.inputTokens)} / out ${formatTokens(c.outputTokens)}`, + ...(unpriced.length > 0 + ? [`⚠ ${unpriced.length} model${unpriced.length === 1 ? '' : 's'} unpriced, counted at $0: ${unpriced.map(u => `${u.model} (${u.calls} calls)`).join(', ')}. Cost above understates real spend; fix with \`codeburn model-alias\` or \`codeburn price-override\`.`] + : []), '', '_Top models_', mdTable(['Model', 'Cost', 'Calls'], c.topModels.slice(0, 5).map(m => [m.name, formatCost(m.cost), String(m.calls)])), diff --git a/src/menubar-json.ts b/src/menubar-json.ts index f337f9c9..1da1e56b 100644 --- a/src/menubar-json.ts +++ b/src/menubar-json.ts @@ -21,6 +21,10 @@ export type PeriodData = { codexCredits?: number categories: Array<{ name: string; cost: number; savingsUSD: number; turns: number; editTurns: number; oneShotTurns: number }> models: Array<{ name: string; cost: number; savingsUSD: number; calls: number }> + /// Models with usage in the period whose pricing lookup fails against the + /// current tables (#638): their calls contribute $0 to `cost`. Optional so + /// PeriodData producers that predate the field keep compiling. + unpricedModels?: Array<{ model: string; calls: number; tokens: number }> projects?: Array<{ name: string; cost: number; savingsUSD: number; sessions: number; sessionDetails?: Array<{ cost: number; savingsUSD: number; calls: number; inputTokens: number; outputTokens: number; date: string; models: Array<{ name: string; cost: number; savingsUSD: number }> }> }> modelEfficiency?: Array<{ name: string; costPerEdit: number | null; oneShotRate: number | null }> topSessions?: Array<{ project: string; cost: number; savingsUSD: number; calls: number; date: string }> @@ -140,6 +144,9 @@ export type MenubarPayload = { savingsBaselineModel: string calls: number }> + /// See PeriodData.unpricedModels: usage priced at $0 for lack of pricing + /// data. Empty when every model in the period resolved a price. + unpricedModels: Array<{ model: string; calls: number; tokens: number }> /// Local-model savings rollup, distinct from the routing-waste / /// optimize savings concepts which describe hypothetical optimization /// opportunities. This block tracks counterfactual spend that was @@ -368,6 +375,7 @@ export function buildMenubarPayload( codexCredits: current.codexCredits ?? 0, topActivities: buildTopActivities(current.categories), topModels: buildTopModels(current.models), + unpricedModels: current.unpricedModels ?? [], localModelSavings: breakdowns?.localModelSavings ?? { totalUSD: 0, calls: 0, byModel: [], byProvider: [] }, providers: buildProviders(providers), topProjects: buildTopProjects(current.projects ?? []), diff --git a/src/models.ts b/src/models.ts index 7586fcd6..3438507e 100644 --- a/src/models.ts +++ b/src/models.ts @@ -679,6 +679,47 @@ function looksLikeLocalModel(name: string): boolean { return false } +export interface UnpricedModelUsage { + model: string + calls: number + tokens: number +} + +// Render-time unpriced detection (#638): flag aggregated model rows that carry +// usage but $0 cost AND whose pricing lookup fails against the CURRENT tables. +// Cost is computed at parse time and cached, so a parse-time registry would +// miss cached sessions; a render-time check covers both and heals itself the +// moment pricing data, an alias, or a price override arrives. +// +// Both conditions are required because aggregation keys rows by DISPLAY name +// (parser.ts keys modelBreakdown via getShortModelName), which the pricing +// lookup does not resolve: a priced model like "Opus 4.8" fails the lookup but +// has real cost, so cost > 0 rows are never flagged. For $0 rows the display +// name is reverse-resolved before flagging, so sessions cached before their +// model's pricing landed don't produce a misleading "add an alias" hint. +// Conservative by design: a display key merging priced and unpriced raw ids +// carries cost > 0 and is not flagged. Local-looking models and models with a +// local-savings mapping are excluded because $0 is their correct cost. +export function findUnpricedModels( + rows: Iterable<{ model: string; calls: number; cost: number; tokens?: number }>, +): UnpricedModelUsage[] { + const out: UnpricedModelUsage[] = [] + for (const row of rows) { + const { model } = row + const tokens = row.tokens ?? 0 + if (!model || model === '') continue + if (row.calls <= 0 && tokens <= 0) continue + if (row.cost > 0) continue + if (looksLikeLocalModel(model)) continue + if (getLocalSavingsBaseline(model)) continue + if (getModelCosts(model)) continue + const rawId = reverseDisplayName(model) + if (rawId && (getModelCosts(rawId) || getLocalSavingsBaseline(rawId) || looksLikeLocalModel(rawId))) continue + out.push({ model, calls: row.calls, tokens }) + } + return out.sort((a, b) => (b.tokens - a.tokens) || (b.calls - a.calls) || a.model.localeCompare(b.model)) +} + function shouldWarnAboutUnknownModel(name: string): boolean { if (!name || name === '') return false if (warnedUnknownModels.has(name)) return false @@ -833,6 +874,30 @@ const SHORT_NAMES: Record = { const SORTED_SHORT_NAMES: [string, string][] = Object.entries(SHORT_NAMES) .sort((a, b) => b[0].length - a[0].length) +// Display name -> raw model id, for aggregation surfaces keyed by +// getShortModelName output (see findUnpricedModels). First table entry wins; +// derived Claude names ("Opus 4.8") are handled by pattern below. +let reverseDisplayNamesCache: Map | null = null +function reverseDisplayName(display: string): string | null { + if (!reverseDisplayNamesCache) { + reverseDisplayNamesCache = new Map() + for (const [id, name] of [...Object.entries(SHORT_NAMES), ...Object.entries(autoModelNames)]) { + if (!reverseDisplayNamesCache.has(name)) reverseDisplayNamesCache.set(name, id) + } + } + const direct = reverseDisplayNamesCache.get(display) + if (direct) return direct + // Inverse of deriveClaudeShortName: "Opus 4.8" -> claude-opus-4-8. + const claude = /^(Opus|Sonnet|Haiku) (\d+)(?:\.(\d+))?$/.exec(display) + if (claude) { + const family = claude[1]!.toLowerCase() + return claude[3] !== undefined + ? `claude-${family}-${claude[2]}-${claude[3]}` + : `claude-${family}-${claude[2]}` + } + return null +} + // Anthropic's id scheme is `claude--[-]`, so every new // version is derivable — no hand-maintained entry per release. (Legacy 3.x ids // put the family last, e.g. `claude-3-5-sonnet`, and stay in SHORT_NAMES.) diff --git a/src/overview.ts b/src/overview.ts index a08ac198..074d8b01 100644 --- a/src/overview.ts +++ b/src/overview.ts @@ -4,7 +4,7 @@ import { homedir } from 'os' import { CATEGORY_LABELS, type ProjectSummary, type TaskCategory } from './types.js' import { formatCost as baseCost } from './currency.js' -import { getShortModelName } from './models.js' +import { findUnpricedModels, getShortModelName } from './models.js' import { dateKey } from './day-aggregator.js' // Display-only helpers. The shared formatters omit thousands separators and @@ -160,6 +160,17 @@ export function renderOverview( out.push(kv('Calls', formatCount(calls) + c.dim(' sessions ') + formatCount(sessions))) out.push(kv('Cache hit', `${cacheHit.toFixed(1)}%`)) if (savings > 0) out.push(kv('Savings', formatCost(savings) + c.dim(' (local models)'))) + const unpriced = findUnpricedModels( + [...byModel.entries()].map(([model, d]) => ({ model, calls: d.calls, cost: d.cost, tokens: d.tokens })), + ) + if (unpriced.length > 0) { + const shown = unpriced.slice(0, 3) + .map((u) => `${u.model} (${formatTokens(u.tokens)} tok)`) + .join(', ') + const more = unpriced.length > 3 ? ` +${unpriced.length - 3} more` : '' + out.push(kv('Unpriced', c.yellow(`${unpriced.length} model${unpriced.length === 1 ? '' : 's'} at $0: `) + shown + more)) + out.push(kv('', c.dim('Fix: codeburn model-alias "" '))) + } out.push('') // Tokens breakdown: input / output / cache in (written) / cache out (read) diff --git a/src/usage-aggregator.ts b/src/usage-aggregator.ts index 42240122..6fceb7af 100644 --- a/src/usage-aggregator.ts +++ b/src/usage-aggregator.ts @@ -2,7 +2,7 @@ import { homedir } from 'node:os' import { CATEGORY_LABELS, type ProjectSummary, type TaskCategory, type DateRange } from './types.js' import { type PeriodData, type ProviderCost, type BreakdownArrays, type MenubarPayload, buildMenubarPayload } from './menubar-json.js' import { parseAllSessions, filterProjectsByName, filterProjectsByDays } from './parser.js' -import { getLocalModelSavingsConfigHash, getPriceOverridesConfigHash, getShortModelName } from './models.js' +import { findUnpricedModels, getLocalModelSavingsConfigHash, getPriceOverridesConfigHash, getShortModelName } from './models.js' import { getAllProviders } from './providers/index.js' import { aggregateProjectsIntoDays, buildPeriodDataFromDays } from './day-aggregator.js' import { aggregateModelEfficiency } from './model-efficiency.js' @@ -13,7 +13,7 @@ import { getDaysInRange, ensureCacheHydrated, loadDailyCache, emptyCache, BACKFI export function buildPeriodData(label: string, projects: ProjectSummary[]): PeriodData { const sessions = projects.flatMap(p => p.sessions) const catTotals: Record = {} - const modelTotals: Record = {} + const modelTotals: Record = {} let inputTokens = 0, outputTokens = 0, cacheReadTokens = 0, cacheWriteTokens = 0 for (const sess of sessions) { @@ -30,10 +30,11 @@ export function buildPeriodData(label: string, projects: ProjectSummary[]): Peri catTotals[cat].oneShotTurns += d.oneShotTurns } for (const [model, d] of Object.entries(sess.modelBreakdown)) { - if (!modelTotals[model]) modelTotals[model] = { calls: 0, cost: 0, savingsUSD: 0 } + if (!modelTotals[model]) modelTotals[model] = { calls: 0, cost: 0, savingsUSD: 0, tokens: 0 } modelTotals[model].calls += d.calls modelTotals[model].cost += d.costUSD modelTotals[model].savingsUSD += d.savingsUSD + modelTotals[model].tokens += d.tokens.inputTokens + d.tokens.outputTokens + d.tokens.cacheReadInputTokens + d.tokens.cacheCreationInputTokens } } @@ -49,7 +50,9 @@ export function buildPeriodData(label: string, projects: ProjectSummary[]): Peri .map(([cat, d]) => ({ name: CATEGORY_LABELS[cat as TaskCategory] ?? cat, ...d })), models: Object.entries(modelTotals) .sort(([, a], [, b]) => b.cost - a.cost) - .map(([name, d]) => ({ name, ...d })), + .map(([name, d]) => ({ name, calls: d.calls, cost: d.cost, savingsUSD: d.savingsUSD })), + unpricedModels: findUnpricedModels(Object.entries(modelTotals) + .map(([model, d]) => ({ model, calls: d.calls, cost: d.cost, tokens: d.tokens }))), } } diff --git a/tests/models.test.ts b/tests/models.test.ts index 4ce651f5..4e81fb79 100644 --- a/tests/models.test.ts +++ b/tests/models.test.ts @@ -4,6 +4,7 @@ import { join } from 'path' import { describe, it, expect, beforeAll, afterEach } from 'vitest' import { + findUnpricedModels, getModelCosts, getShortModelName, calculateCost, @@ -675,3 +676,73 @@ describe('observed provider model aliases', () => { expect(calculateCost('qwen3-max-2026-01-23', 1_000_000, 1_000_000, 0, 0, 0)).toBe(0) }) }) + +describe('findUnpricedModels', () => { + it('flags an unknown paid-looking model with $0 cost and skips priced ones', () => { + const rows = [ + { model: 'claude-opus-4-6', calls: 10, cost: 2.5, tokens: 5000 }, + { model: 'zz-mystery-paid-model-999', calls: 3, cost: 0, tokens: 1200 }, + ] + const unpriced = findUnpricedModels(rows) + expect(unpriced).toEqual([{ model: 'zz-mystery-paid-model-999', calls: 3, tokens: 1200 }]) + }) + + it('never flags a row that carries real cost, even when the lookup misses', () => { + // Aggregation keys rows by display name; the lookup misses but the row was + // priced at parse time, so it must not be reported as unpriced. + const unpriced = findUnpricedModels([ + { model: 'Opus 4.8', calls: 100, cost: 42.5, tokens: 1_000_000 }, + { model: 'zz-unknown-but-priced-elsewhere', calls: 5, cost: 0.01, tokens: 500 }, + ]) + expect(unpriced).toEqual([]) + }) + + it('reverse-resolves display names before flagging $0 rows', () => { + // Sessions cached before a model's pricing landed carry $0 cost under a + // display-name key. The raw id prices today, so no "add an alias" hint. + const unpriced = findUnpricedModels([ + { model: 'Fable 5', calls: 94, cost: 0, tokens: 12_000_000 }, + { model: 'Opus 4.8', calls: 33, cost: 0, tokens: 1_600_000 }, + { model: 'GPT-4o', calls: 7, cost: 0, tokens: 900 }, + ]) + expect(unpriced).toEqual([]) + }) + + it('skips synthetic, empty, local-looking, and zero-usage rows', () => { + const unpriced = findUnpricedModels([ + { model: '', calls: 5, cost: 0, tokens: 100 }, + { model: '', calls: 5, cost: 0, tokens: 100 }, + { model: 'llama3.1:8b', calls: 5, cost: 0, tokens: 100 }, + { model: 'zz-quantized-model-bf16', calls: 5, cost: 0, tokens: 100 }, + { model: 'zz-no-usage-model', calls: 0, cost: 0, tokens: 0 }, + ]) + expect(unpriced).toEqual([]) + }) + + it('heals when the user configures an alias or a price override', () => { + const model = 'zz-proxy-renamed-model-x1' + expect(findUnpricedModels([{ model, calls: 1, cost: 0, tokens: 10 }])).toHaveLength(1) + + setModelAliases({ [model]: 'claude-opus-4-6' }) + expect(findUnpricedModels([{ model, calls: 1, cost: 0, tokens: 10 }])).toEqual([]) + setModelAliases({}) + + setPriceOverrides({ [model]: { input: 1, output: 2 } }) + expect(findUnpricedModels([{ model, calls: 1, cost: 0, tokens: 10 }])).toEqual([]) + }) + + it('skips models mapped via model-savings (intentionally $0)', () => { + const model = 'zz-my-local-runner' + expect(findUnpricedModels([{ model, calls: 1, cost: 0, tokens: 10 }])).toHaveLength(1) + setLocalModelSavings({ [model]: 'gpt-4o' }) + expect(findUnpricedModels([{ model, calls: 1, cost: 0, tokens: 10 }])).toEqual([]) + }) + + it('sorts by tokens, then calls', () => { + const unpriced = findUnpricedModels([ + { model: 'zz-small', calls: 9, cost: 0, tokens: 10 }, + { model: 'zz-big', calls: 1, cost: 0, tokens: 9999 }, + ]) + expect(unpriced.map(u => u.model)).toEqual(['zz-big', 'zz-small']) + }) +}) diff --git a/tests/overview.test.ts b/tests/overview.test.ts index 5f2dd759..89d6d758 100644 --- a/tests/overview.test.ts +++ b/tests/overview.test.ts @@ -116,3 +116,36 @@ describe('renderOverview', () => { expect(out).not.toContain(' OS ') }) }) + +describe('renderOverview unpriced models', () => { + it('warns when a model with usage has no pricing data', () => { + const out = renderOverview([makeProject({ + project: 'mystery', + projectPath: '/Users/test/mystery', + cost: 0, + calls: 4, + model: 'zz-mystery-paid-model-999', + provider: 'claude', + tokens: { input: 1000, output: 200, cacheR: 0, cacheW: 0 }, + })], { label: 'June 2026', color: false }) + + expect(out).toContain('Unpriced') + expect(out).toContain('1 model at $0') + expect(out).toContain('zz-mystery-paid-model-999') + expect(out).toContain('codeburn model-alias') + }) + + it('stays silent when every model is priced', () => { + const out = renderOverview([makeProject({ + project: 'priced', + projectPath: '/Users/test/priced', + cost: 5, + calls: 2, + model: 'claude-opus-4-8', + provider: 'claude', + tokens: { input: 100, output: 50, cacheR: 0, cacheW: 0 }, + })], { label: 'June 2026', color: false }) + + expect(out).not.toContain('Unpriced') + }) +}) From e6162f6b50061096a28d21f93d795c36cde7dea8 Mon Sep 17 00:00:00 2001 From: AgentSeal Date: Thu, 9 Jul 2026 21:17:47 +0200 Subject: [PATCH 2/4] fix(report): address unpriced-detector review findings Adversarial review found two false-negative paths and two polish items: - Drop the reverse-display-name suppression. Droid prices the lowercased display name and gets $0, so a $0 row keyed "Sonnet 4.6" is real uncounted spend; suppressing it because claude-sonnet-4-6 prices today hid exactly the failure this feature exists to expose. - Treat zero-rate pricing hits as unpriced. LiteLLM ships [0,0] stubs for models it lists without a price; a stub hit means unknown price, not free. Explicit zero-rate user overrides still count as free. - Make MenubarPayload.current.unpricedModels optional for source compatibility with payload producers that predate the field. - Replace localeCompare with a byte comparison so tied rows sort identically on every host locale. Refs #638 --- src/menubar-json.ts | 5 +-- src/models.ts | 76 +++++++++++++++++++++----------------------- tests/models.test.ts | 50 +++++++++++++++++++++++++---- 3 files changed, 83 insertions(+), 48 deletions(-) diff --git a/src/menubar-json.ts b/src/menubar-json.ts index 1da1e56b..b4ff9191 100644 --- a/src/menubar-json.ts +++ b/src/menubar-json.ts @@ -145,8 +145,9 @@ export type MenubarPayload = { calls: number }> /// See PeriodData.unpricedModels: usage priced at $0 for lack of pricing - /// data. Empty when every model in the period resolved a price. - unpricedModels: Array<{ model: string; calls: number; tokens: number }> + /// data. Empty when every model in the period resolved a price. Optional + /// so payload producers that predate the field stay source-compatible. + unpricedModels?: Array<{ model: string; calls: number; tokens: number }> /// Local-model savings rollup, distinct from the routing-waste / /// optimize savings concepts which describe hypothetical optimization /// opportunities. This block tracks counterfactual spend that was diff --git a/src/models.ts b/src/models.ts index 3438507e..9b444173 100644 --- a/src/models.ts +++ b/src/models.ts @@ -685,21 +685,42 @@ export interface UnpricedModelUsage { tokens: number } +function hasBillableRate(costs: ModelCosts): boolean { + return costs.inputCostPerToken > 0 + || costs.outputCostPerToken > 0 + || costs.cacheWriteCostPerToken > 0 + || costs.cacheReadCostPerToken > 0 +} + +// The user-override lookup from getModelCosts, in isolation. Lets the unpriced +// detector distinguish "explicitly declared free by the user" (a zero-rate +// override) from a zero-rate LiteLLM stub, which means "listed but unknown +// price" and must still be flagged. +function userPriceOverrideFor(model: string): ModelCosts | null { + const withPrefix = model.replace(/@.*$/, '').replace(/-\d{8}$/, '') + const canonicalName = getCanonicalName(model) + const canonical = resolveAlias(canonicalName) + return getPriceOverrideExact(model, withPrefix, canonicalName, canonical) + ?? getPriceOverridePrefix(canonical) + ?? getPriceOverrideCaseInsensitive(canonical, withPrefix) +} + // Render-time unpriced detection (#638): flag aggregated model rows that carry -// usage but $0 cost AND whose pricing lookup fails against the CURRENT tables. -// Cost is computed at parse time and cached, so a parse-time registry would -// miss cached sessions; a render-time check covers both and heals itself the +// usage but $0 cost AND whose pricing lookup yields no billable rate right +// now. Cost is computed at parse time and cached, so a parse-time registry +// would miss cached sessions; a render-time check covers both and heals the // moment pricing data, an alias, or a price override arrives. // -// Both conditions are required because aggregation keys rows by DISPLAY name +// Rows with cost > 0 are never flagged: aggregation keys rows by DISPLAY name // (parser.ts keys modelBreakdown via getShortModelName), which the pricing -// lookup does not resolve: a priced model like "Opus 4.8" fails the lookup but -// has real cost, so cost > 0 rows are never flagged. For $0 rows the display -// name is reverse-resolved before flagging, so sessions cached before their -// model's pricing landed don't produce a misleading "add an alias" hint. -// Conservative by design: a display key merging priced and unpriced raw ids -// carries cost > 0 and is not flagged. Local-looking models and models with a -// local-savings mapping are excluded because $0 is their correct cost. +// lookup misses, so a priced model like "Opus 4.8" would otherwise false-flag. +// $0 display-name rows ARE flagged even when the raw id would price today: +// those tokens really did enter the report at $0 (a provider priced a +// transformed name, or the session was cached before its model's pricing +// landed). Conservative by design: a display key merging priced and unpriced +// raw ids carries cost > 0 and is not flagged. Local-looking models and +// models with a local-savings mapping are excluded because $0 is their +// correct cost, as are zero-rate USER overrides (explicitly declared free). export function findUnpricedModels( rows: Iterable<{ model: string; calls: number; cost: number; tokens?: number }>, ): UnpricedModelUsage[] { @@ -712,12 +733,13 @@ export function findUnpricedModels( if (row.cost > 0) continue if (looksLikeLocalModel(model)) continue if (getLocalSavingsBaseline(model)) continue - if (getModelCosts(model)) continue - const rawId = reverseDisplayName(model) - if (rawId && (getModelCosts(rawId) || getLocalSavingsBaseline(rawId) || looksLikeLocalModel(rawId))) continue + const costs = getModelCosts(model) + if (costs && hasBillableRate(costs)) continue + if (costs && userPriceOverrideFor(model)) continue out.push({ model, calls: row.calls, tokens }) } - return out.sort((a, b) => (b.tokens - a.tokens) || (b.calls - a.calls) || a.model.localeCompare(b.model)) + return out.sort((a, b) => (b.tokens - a.tokens) || (b.calls - a.calls) + || (a.model < b.model ? -1 : a.model > b.model ? 1 : 0)) } function shouldWarnAboutUnknownModel(name: string): boolean { @@ -874,30 +896,6 @@ const SHORT_NAMES: Record = { const SORTED_SHORT_NAMES: [string, string][] = Object.entries(SHORT_NAMES) .sort((a, b) => b[0].length - a[0].length) -// Display name -> raw model id, for aggregation surfaces keyed by -// getShortModelName output (see findUnpricedModels). First table entry wins; -// derived Claude names ("Opus 4.8") are handled by pattern below. -let reverseDisplayNamesCache: Map | null = null -function reverseDisplayName(display: string): string | null { - if (!reverseDisplayNamesCache) { - reverseDisplayNamesCache = new Map() - for (const [id, name] of [...Object.entries(SHORT_NAMES), ...Object.entries(autoModelNames)]) { - if (!reverseDisplayNamesCache.has(name)) reverseDisplayNamesCache.set(name, id) - } - } - const direct = reverseDisplayNamesCache.get(display) - if (direct) return direct - // Inverse of deriveClaudeShortName: "Opus 4.8" -> claude-opus-4-8. - const claude = /^(Opus|Sonnet|Haiku) (\d+)(?:\.(\d+))?$/.exec(display) - if (claude) { - const family = claude[1]!.toLowerCase() - return claude[3] !== undefined - ? `claude-${family}-${claude[2]}-${claude[3]}` - : `claude-${family}-${claude[2]}` - } - return null -} - // Anthropic's id scheme is `claude--[-]`, so every new // version is derivable — no hand-maintained entry per release. (Legacy 3.x ids // put the family last, e.g. `claude-3-5-sonnet`, and stay in SHORT_NAMES.) diff --git a/tests/models.test.ts b/tests/models.test.ts index 4e81fb79..b0eebff4 100644 --- a/tests/models.test.ts +++ b/tests/models.test.ts @@ -697,15 +697,51 @@ describe('findUnpricedModels', () => { expect(unpriced).toEqual([]) }) - it('reverse-resolves display names before flagging $0 rows', () => { - // Sessions cached before a model's pricing landed carry $0 cost under a - // display-name key. The raw id prices today, so no "add an alias" hint. + it('flags $0 display-name rows even when the raw id would price today', () => { + // Droid prices the lowercased display name ("claude sonnet 4.6" -> no + // pricing -> $0) and the parser keys the row by display name. Those + // tokens really entered the report at $0, so the row must be flagged + // even though claude-sonnet-4-6 itself is priced. const unpriced = findUnpricedModels([ - { model: 'Fable 5', calls: 94, cost: 0, tokens: 12_000_000 }, - { model: 'Opus 4.8', calls: 33, cost: 0, tokens: 1_600_000 }, - { model: 'GPT-4o', calls: 7, cost: 0, tokens: 900 }, + { model: 'Sonnet 4.6', calls: 12, cost: 0, tokens: 500_000 }, ]) - expect(unpriced).toEqual([]) + expect(unpriced).toEqual([{ model: 'Sonnet 4.6', calls: 12, tokens: 500_000 }]) + }) + + it('flags zero-rate pricing stubs but not explicit zero-rate user overrides', async () => { + // LiteLLM ships [0,0] stubs for models it lists but has no price for; + // a stub hit means "unknown price", not "free". + const cacheRoot = await mkdtemp(join(tmpdir(), 'codeburn-pricing-cache-')) + try { + process.env['CODEBURN_CACHE_DIR'] = cacheRoot + await writeFile(join(cacheRoot, 'litellm-pricing.json'), JSON.stringify({ + timestamp: Date.now(), + data: { + 'zz-zero-stub-model': { + inputCostPerToken: 0, + outputCostPerToken: 0, + cacheWriteCostPerToken: 0, + cacheReadCostPerToken: 0, + webSearchCostPerRequest: 0, + fastMultiplier: 1, + }, + }, + }), 'utf-8') + await loadPricing() + + expect(getModelCosts('zz-zero-stub-model')).not.toBeNull() + const rows = [{ model: 'zz-zero-stub-model', calls: 3, cost: 0, tokens: 1100 }] + expect(findUnpricedModels(rows)).toHaveLength(1) + + // An explicit user override at zero rates means "this model is free". + setPriceOverrides({ 'zz-zero-stub-model': { input: 0, output: 0 } }) + expect(findUnpricedModels(rows)).toEqual([]) + } finally { + delete process.env['CODEBURN_CACHE_DIR'] + await rm(cacheRoot, { recursive: true, force: true }) + setPriceOverrides({}) + await loadPricing() + } }) it('skips synthetic, empty, local-looking, and zero-usage rows', () => { From bf6eee00f3aeb006c3126f209450bf813e46ffec Mon Sep 17 00:00:00 2001 From: AgentSeal Date: Thu, 9 Jul 2026 21:22:23 +0200 Subject: [PATCH 3/4] fix(report): only exact zero-rate overrides suppress the unpriced flag getModelCosts resolves exact overrides before any table hit, so an exact zero-rate override provably priced the model and means free. Prefix and case-insensitive overrides resolve after table hits: a zero-rate stub shadowed by one still reports $0 from the stub, so it stays flagged. Refs #638 --- src/models.ts | 18 ++++++++++-------- tests/models.test.ts | 6 ++++++ 2 files changed, 16 insertions(+), 8 deletions(-) diff --git a/src/models.ts b/src/models.ts index 9b444173..b314ae92 100644 --- a/src/models.ts +++ b/src/models.ts @@ -692,17 +692,19 @@ function hasBillableRate(costs: ModelCosts): boolean { || costs.cacheReadCostPerToken > 0 } -// The user-override lookup from getModelCosts, in isolation. Lets the unpriced -// detector distinguish "explicitly declared free by the user" (a zero-rate -// override) from a zero-rate LiteLLM stub, which means "listed but unknown -// price" and must still be flagged. -function userPriceOverrideFor(model: string): ModelCosts | null { +// Exact-override lookup with the same key derivation getModelCosts uses. Lets +// the unpriced detector distinguish "explicitly declared free by the user" (a +// zero-rate override) from a zero-rate LiteLLM stub, which means "listed but +// unknown price" and must still be flagged. Only the EXACT override form is +// consulted: getModelCosts checks it before any table hit, so when one exists +// it is provably what priced the model. Prefix and case-insensitive overrides +// resolve AFTER table hits and so cannot prove the $0 was intentional; a +// zero-rate stub shadowed by one still gets flagged (the honest direction). +function exactPriceOverrideFor(model: string): ModelCosts | null { const withPrefix = model.replace(/@.*$/, '').replace(/-\d{8}$/, '') const canonicalName = getCanonicalName(model) const canonical = resolveAlias(canonicalName) return getPriceOverrideExact(model, withPrefix, canonicalName, canonical) - ?? getPriceOverridePrefix(canonical) - ?? getPriceOverrideCaseInsensitive(canonical, withPrefix) } // Render-time unpriced detection (#638): flag aggregated model rows that carry @@ -735,7 +737,7 @@ export function findUnpricedModels( if (getLocalSavingsBaseline(model)) continue const costs = getModelCosts(model) if (costs && hasBillableRate(costs)) continue - if (costs && userPriceOverrideFor(model)) continue + if (costs && exactPriceOverrideFor(model)) continue out.push({ model, calls: row.calls, tokens }) } return out.sort((a, b) => (b.tokens - a.tokens) || (b.calls - a.calls) diff --git a/tests/models.test.ts b/tests/models.test.ts index b0eebff4..1f703857 100644 --- a/tests/models.test.ts +++ b/tests/models.test.ts @@ -736,6 +736,12 @@ describe('findUnpricedModels', () => { // An explicit user override at zero rates means "this model is free". setPriceOverrides({ 'zz-zero-stub-model': { input: 0, output: 0 } }) expect(findUnpricedModels(rows)).toEqual([]) + + // A prefix override cannot prove intent: getModelCosts resolves table + // hits before prefix overrides, so the $0 came from the stub, not the + // user. Still flagged. + setPriceOverrides({ 'zz-zero-stub': { input: 0, output: 0 } }) + expect(findUnpricedModels(rows)).toHaveLength(1) } finally { delete process.env['CODEBURN_CACHE_DIR'] await rm(cacheRoot, { recursive: true, force: true }) From 4018a0ba9937732151abeba05dddac61f6a17f14 Mon Sep 17 00:00:00 2001 From: AgentSeal Date: Thu, 9 Jul 2026 21:27:33 +0200 Subject: [PATCH 4/4] fix(tui): keep the model-alias hint visible when the unpriced footer truncates At 80 columns the By Model footer truncated mid-command. The flagged models are already visible as $0 rows in the table above, so put the fix command before the names and let the names truncate instead. Refs #638 --- src/dashboard.tsx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/dashboard.tsx b/src/dashboard.tsx index 064be43f..a79c9c22 100644 --- a/src/dashboard.tsx +++ b/src/dashboard.tsx @@ -389,7 +389,7 @@ function ModelBreakdown({ projects, pw, bw }: { projects: ProjectSummary[]; pw: })} {unpriced.length > 0 && ( - {`! ${unpriced.length} model${unpriced.length === 1 ? '' : 's'} unpriced at $0 (${unpriced.slice(0, 2).map(u => u.model).join(', ')}${unpriced.length > 2 ? ', ...' : ''}) - fix: codeburn model-alias`} + {`! ${unpriced.length} model${unpriced.length === 1 ? '' : 's'} unpriced at $0, fix: codeburn model-alias (${unpriced.slice(0, 2).map(u => u.model).join(', ')}${unpriced.length > 2 ? ', ...' : ''})`} )}