mirror of
https://github.com/AgentSeal/codeburn.git
synced 2026-08-20 22:14:36 +00:00
fix(grok): read the CLI's own completed-turn usage instead of estimating it
Grok CLI writes a turn_completed update carrying a full usage object -- inputTokens, outputTokens, cachedReadTokens, cacheCreationTokens, reasoningTokens -- into the same updates.jsonl the parser already reads. We ignored it and reconstructed an estimate from _meta.totalTokens, a running context-size counter that rides on unrelated events, with a total < prevTotal * 0.5 reset as the turn boundary. On the cache-heavy session reported in #998 that reconstruction captured about 1.4% of the real cache-read volume and roughly 6% of the day's tokens, while over-counting output about fivefold. cacheCreationInputTokens and reasoningTokens were hardcoded to zero regardless of what the session held. The parser now reads turn_completed.usage, keyed by the record's snake_case prompt_id so a re-emitted turn cannot double count, and sums across turns. Two decompositions matter, both derivable from the reported numbers: totalTokens equals inputTokens + outputTokens exactly, so cachedReadTokens and cacheCreationTokens are subsets of input and are subtracted out per record before pricing, matching the cache-exclusive convention codex and copilot already use; and reasoningTokens is a subset of output. That second one needs care, because the repo contract is the opposite of Grok's: ParsedProviderCall.reasoningTokens is exclusive of outputTokens everywhere, and every consumer sums the two -- tests/providers/kiro.test.ts says so outright. So reasoning is clamped to the reported output and output is emitted without it, and the downstream sum reconstructs Grok's number. Without the clamp a record with reasoning > output produced a negative output and left the pipeline pricing reasoning instead. Multi-model attribution is deliberately out of scope. modelUsage only selects a priced attribution id; a session that used two models is priced at one rate. Splitting per model was tried and dropped: chooseAuthoritativeModel's priced-id fallback exists to avoid a truthful-but-$0 row when modelUsage names an id this checkout cannot price, and per-model pricing loses it -- the reporter's own session collapsed from $1.20 to near zero the moment a second id appeared. When no valid completed record exists -- older Grok CLI versions -- the old heuristic still runs, unchanged. The decision is taken from the deduplicated records rather than latched per line, so a superseded or all-zero record cannot flip a session off the heuristic and drop it. A session only partly covered by turn_completed records keeps costIsEstimated: true rather than presenting itself as fully provider-measured. costUsdTicks is deliberately not read. Its scale is undocumented, and guessing it would fabricate spend. Bumps the grok parse version, and DAILY_CACHE_VERSION with MIN_SUPPORTED_VERSION together, since the daily cache serves every day before today and retains ten years. Moving them in lockstep is what keeps the carry-forward lossless: the filename is version-suffixed, so the old file stays on disk and is adopted for days no source can still re-derive. Separately, detectContextBloat divided by outputTokens alone. Reasoning is stored beside output for every reasoning-bearing provider, so the detector saw a fraction of the generated tokens and invented high-impact findings -- a session whose provider-reported ratio is 20:1, below the 25:1 threshold, was reported as 133:1 with 710K tokens of claimed savings. It now uses the same output + reasoning sum the reports use, which fixes codex, kiro, hermes, qwen and cursor-agent too. Reported in #998.
This commit is contained in:
parent
7e421b14bb
commit
72aad9e012
18 changed files with 1020 additions and 292 deletions
|
|
@ -3,6 +3,7 @@
|
|||
## Unreleased
|
||||
|
||||
### Changed
|
||||
- **Grok Build now uses the CLI's authoritative completed-turn usage.** `turn_completed.usage` records are deduplicated by prompt and emitted as one session-level call from verified top-level totals; `modelUsage` only selects a priced attribution id, so multi-model rate attribution remains deliberately out of scope. Reasoning is clamped to reported output before the exclusive output/reasoning split, and the old context-size heuristic remains for sessions without usable top-level usage. In mixed sessions, uncovered pre-upgrade turns are dropped and the row is marked estimated rather than claiming full provider coverage; already-cached Grok sessions re-parse once. (#998)
|
||||
- **Codex rollouts parse across worker threads too, and the workload gate now takes bytes or files.** Codex is the bigger half of a real cold parse — a 4 GB rollout corpus against 1.8 GB of Claude sessions — and it was still decoding one file at a time. A whole-file rollout decode now runs on the same pool, against an empty dedup set, and comes back with the calls, the dedup keys it claimed, and the codex-cache entry it would have written; the parent installs all three in the serial loop's order, so `codex-results.json` and every payload come out byte-identical to a serial run. Cross-file state stays where it was: a forked rollout replaying its parent's token_count history collides on the parent's keys and is re-parsed in-process, and no worker ever touches the cache module's per-directory state. Files the Codex cache can serve exactly or resume into from a byte offset never reach a worker — they read a few KB and the resume state belongs to the parent. The workload gate is now pending BYTES alone (200 MB), not file count: 250 pending files holding under a megabyte between them spawned threads that made the run ~5% slower, while a few hundred huge rollouts were being turned away. The count takes `max(pendingFiles / 50, pendingBytes / 200 MB)`, and the per-thread memory budget is derived per parse as `clamp(256 MB, 2 × average pending file + 128 MB, 1 GB)` rather than a flat 256 MB — a 260 MB rollout peaks near 430 MB in its worker and scales linearly with the pool, so the flat figure over-subscribed exactly the workload this adds. The decision is per provider, and at most one pool is alive at a time.
|
||||
- **A large cold Claude parse now runs across worker threads.** Reading, decoding and line-parsing a session JSONL is per-file work that never touches anything shared, so it moves onto `worker_threads`; each worker ships its parsed turns back as a JSON string and the parent installs them in the exact order the serial loop would. Everything with cross-file state — the streaming-message dedup, canonical project paths, spawn links, PR correlation, progress saves — stays on the main thread, and a file whose message ids were already claimed by an earlier file (or whose worker failed) is simply re-parsed in-process, so the session cache and every payload are identical either way. On a 6 GB corpus a cold `status` drops from 27.5s to 14.8s with peak RSS up 2.27 GB → 2.52 GB. Threads only engage for a genuinely large cold parse: never with under 200 MB behind the pending whole-file re-parses, 2 or fewer cores, or under 4 GB of available memory — so warm and incremental runs are untouched and spawn nothing. Otherwise the count is `min(cores - 1, min(0.25 × available, 2 GB) / 256 MB, pendingFiles / 50)`, where available is `process.availableMemory()` (cgroup-aware in containers) rather than free memory, which on macOS reports free pages and would switch the feature on and off between runs. `CODEBURN_PARSE_WORKERS=0` forces the serial parse and `CODEBURN_PARSE_WORKERS=N` forces N (capped at the core count), both bypassing every gate; `CODEBURN_VERBOSE=1` prints the resolved count and why.
|
||||
- **A warm launch rewrites only the month that changed, and a ranged query reads only the months it can report on.** Per-provider shards still meant one appended session republished that provider's entire history — 95 MB for Claude on a 6 GB corpus. Each provider's shard is now split again by the UTC month of the cached session's FIRST turn, a bucket that never moves as a session grows, so an append rewrites one month. Every shard records the newest month it holds, which lets `--period today/week` skip the shards that cannot contribute a turn to the range; the skipped months stay on disk untouched across the save, and providers whose cache is the only surviving record (durable) or whose parse fingerprint moved are always read in full. Remaining shards are read concurrently. Existing v8 and v7 caches are re-laid-out losslessly on first load and the old layout removed once the new one is published: nothing re-parses.
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ Grok Build, xAI's coding CLI. Sessions use the `grok-build` model by default.
|
|||
|
||||
- **Source:** `src/providers/grok.ts`
|
||||
- **Loading:** eager (`src/providers/index.ts`)
|
||||
- **Test:** `tests/providers/grok.test.ts`
|
||||
- **Test:** `tests/grok-parser-pipeline.test.ts`, `tests/providers/grok.test.ts`
|
||||
|
||||
## Where it reads from
|
||||
|
||||
|
|
@ -13,19 +13,23 @@ Grok Build, xAI's coding CLI. Sessions use the `grok-build` model by default.
|
|||
|
||||
## Storage format
|
||||
|
||||
JSON + JSONL. `summary.json` holds the session id, cwd, timestamps, and `current_model_id`. `signals.json` holds `modelsUsed`, `toolsUsed`, and `contextTokensUsed`. `updates.jsonl` is the ACP log: each streamed chunk carries `params._meta.totalTokens` (running context size) and `params._meta.promptId` (one per turn).
|
||||
JSON + JSONL. `summary.json` holds the session id, cwd, timestamps, and `current_model_id`. `signals.json` holds `modelsUsed`, `toolsUsed`, and `contextTokensUsed`. `updates.jsonl` is the ACP log: streamed chunks carry `params._meta.totalTokens` (running context size) and `params._meta.promptId` (one per turn); newer CLI versions also append `params.update.sessionUpdate: "turn_completed"` with snake_case `prompt_id` and a provider-recorded `usage` object.
|
||||
|
||||
## Token model
|
||||
|
||||
**Estimated.** Grok does not log billable input/output tokens. It only records the running context fill (`totalTokens` per chunk, and `contextTokensUsed` in signals). The parser reconstructs a rough estimate from the per-turn `totalTokens` curve: input is the context entering each turn, output is the context growth during it. The result is flagged `costIsEstimated` and re-priced with `calculateCost`.
|
||||
**Authoritative when available.** A `turn_completed` record reports the whole input side in `inputTokens` and the whole output side in `outputTokens`. `cachedReadTokens` is treated as a subset of input; `cacheCreationTokens` is treated as another input subset by analogy because the record exposes no separate fresh-input field, with any per-record violation clamped locally. The top-level totals are the accounting basis. `modelUsage` is retained only as a model-attribution signal; multi-model attribution is deliberately out of scope, so one session uses one selected model's rate. `reasoningTokens` is clamped to that record's reported output before the parser emits exclusive output plus reasoning, preserving the reported total through the cache pipeline. These counts are provider-recorded, so `costIsEstimated` is false for fully covered sessions while CodeBurn applies its own pricing table. `costUsdTicks` is ignored because its scale is undocumented.
|
||||
|
||||
**Estimated fallback.** Older sessions without any valid `turn_completed.usage` record use the running context fill (`totalTokens` per chunk) and the existing compaction-aware per-turn curve. That path remains flagged `costIsEstimated`; a completed record is never blended with the heuristic.
|
||||
|
||||
If a session spans a CLI upgrade and has both completed records and streamed turns without a matching record, the all-or-nothing authoritative path keeps the recorded totals, drops the uncovered turns, and marks the emitted row `costIsEstimated: true`. A later parse can fill an open turn once it writes a record; pre-upgrade turns that never do are dropped.
|
||||
|
||||
## Pricing
|
||||
|
||||
`grok-build` is aliased to `grok-build-0.1` in `src/models.ts`, so it prices off the bundled LiteLLM fallback. Note that xAI's published API rate and the LiteLLM fallback figure differ, so treat the cost as an estimate and verify against your xAI usage console.
|
||||
`grok-build` is aliased to `grok-build-0.1` in `src/models.ts`, so it prices off the bundled LiteLLM fallback. If `usage.modelUsage` contains a model id that CodeBurn can price, that id is preferred; when the real id is not priced yet, the existing summary/signals model is retained so a known alias does not become a $0 row. This is a single attribution choice for the session, not a per-model accounting split; multi-model rate attribution is a follow-up. CodeBurn still does not use Grok's undocumented `costUsdTicks`.
|
||||
|
||||
## Caching
|
||||
|
||||
None.
|
||||
Authoritative records expose cache-read and cache-creation token counts. The legacy estimate has no cache-creation signal and keeps its inferred cache-read count.
|
||||
|
||||
## Deduplication
|
||||
|
||||
|
|
@ -33,7 +37,7 @@ Per `grok:<session-dir>:<updated_at>:<id>`.
|
|||
|
||||
## Quirks
|
||||
|
||||
- **No cache or output/tool-token split.** Only context fill is available, so cache fields are `0` and the cost is an estimate (likely an upper bound, since re-sent context is cached server-side and not exposed in the session files).
|
||||
- **Two token paths.** Completed turns carry provider usage; sessions from older CLI versions have only the context curve and therefore remain estimates (likely an upper bound, since re-sent context is cached server-side and not exposed in those files).
|
||||
- **No bash-command capture.** Tool names come from `signals.toolsUsed`; per-command bash text is not extracted, so `bashCommands` is empty.
|
||||
- **Whole-session timestamp.** Spend is attributed to `updated_at`, since the context curve is cumulative.
|
||||
- **Subscription vs API.** Grok Build runs via either a metered xAI API account (tiered) or a SuperGrok subscription; the session files do not record which.
|
||||
|
|
@ -41,5 +45,5 @@ Per `grok:<session-dir>:<updated_at>:<id>`.
|
|||
## When fixing a bug here
|
||||
|
||||
1. Discovery: check the `sessions/<cwd>/<uuid>/` walk and the `GROK_HOME` resolution.
|
||||
2. Token estimate: see `estimateTokens` (groups `updates.jsonl` by `promptId`).
|
||||
2. Token accounting: see `parseUpdates` (deduplicates `turn_completed` by snake_case `prompt_id`, then falls back to grouping streamed chunks by camelCase `_meta.promptId`).
|
||||
3. Add a fixture-format session under `tests/providers/grok.test.ts`; do not mock the filesystem.
|
||||
|
|
|
|||
|
|
@ -13,10 +13,6 @@ export type ApplyOptions = {
|
|||
yes?: boolean
|
||||
dryRun?: boolean
|
||||
only?: string
|
||||
// Mirrors `optimize --provider`. The scan below only reads Claude
|
||||
// transcripts, and this path does not just report findings, it plans and
|
||||
// applies them - a Codex-scoped run must never offer to edit ~/.claude.
|
||||
provider?: string
|
||||
actionsDir?: string
|
||||
ctx?: PlanContext
|
||||
// Test seams: crafted findings skip the session scan; streams default to
|
||||
|
|
@ -107,7 +103,7 @@ export async function runOptimizeApply(
|
|||
let costRate = opts.costRate ?? 0
|
||||
if (!findings) {
|
||||
errout.write(chalk.dim(' Analyzing your sessions...\n'))
|
||||
const scanned = await scanAndDetect(projects, dateRange, opts.provider)
|
||||
const scanned = await scanAndDetect(projects, dateRange)
|
||||
findings = scanned.findings
|
||||
costRate = scanned.costRate
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,6 +6,13 @@ import { join } from 'path'
|
|||
import { getCodeburnCacheDir } from './cache-dir.js'
|
||||
import type { DateRange, ProjectSummary } from './types.js'
|
||||
|
||||
// Bumped to 19: Grok authoritative usage now keeps one session-level rollup
|
||||
// from top-level totals, clamps reasoning to reported output, and labels mixed
|
||||
// authoritative/heuristic coverage. Days already finalized at v18 contain the
|
||||
// per-model split from the previous draft, so raising MIN_SUPPORTED_VERSION
|
||||
// forces a one-time re-derivation; v14 carry-forward keeps source-less history
|
||||
// intact.
|
||||
//
|
||||
// Bumped to 17: copilot CLI sessions were misclassified as VS Code transcripts
|
||||
// (#944), so days finalized at v16 or earlier carry output-only copilot costs —
|
||||
// the session.shutdown rollup's input/cache tokens were dropped. Raising
|
||||
|
|
@ -74,8 +81,8 @@ import type { DateRange, ProjectSummary } from './types.js'
|
|||
// that older binaries skipped. v8 added local-model savings to the daily
|
||||
// rollup; the `savingsConfigHash` field is invalidated separately when the
|
||||
// user changes their `localModelSavings` mapping.
|
||||
export const DAILY_CACHE_VERSION = 17
|
||||
const MIN_SUPPORTED_VERSION = 17
|
||||
export const DAILY_CACHE_VERSION = 19
|
||||
const MIN_SUPPORTED_VERSION = 19
|
||||
// Version-suffixed so different binaries each own a distinct file and never
|
||||
// clobber an incompatible schema. Bumping the version mints a fresh filename;
|
||||
// adoptOlderDailyCaches then unions days out of every previous file (including
|
||||
|
|
|
|||
|
|
@ -1460,14 +1460,14 @@ export function InteractiveDashboard({ initialProjects, initialDailyHistoryProje
|
|||
const generation = reloadGenerationRef.current
|
||||
setOptimizeLoading(true)
|
||||
try {
|
||||
const result = await scanAndDetect(projects, currentRange(), activeProvider)
|
||||
const result = await scanAndDetect(projects, currentRange())
|
||||
if (reloadGenerationRef.current === generation) setOptimizeResult(result)
|
||||
} catch (error) {
|
||||
console.error(error)
|
||||
} finally {
|
||||
if (reloadGenerationRef.current === generation) setOptimizeLoading(false)
|
||||
}
|
||||
}, [optimizeAvailable, projects, currentRange, optimizeLoading, optimizeResult, activeProvider])
|
||||
}, [optimizeAvailable, projects, currentRange, optimizeLoading, optimizeResult])
|
||||
|
||||
useEffect(() => {
|
||||
const refreshIntervalMs = getRefreshIntervalMs(refreshSeconds ?? 0)
|
||||
|
|
|
|||
19
src/main.ts
19
src/main.ts
|
|
@ -1831,7 +1831,7 @@ program
|
|||
const projects = await parseAllSessions(range, opts.provider)
|
||||
if (opts.apply) {
|
||||
const { runOptimizeApply } = await import('./act/optimize-apply.js')
|
||||
await runOptimizeApply(projects, range, { yes: opts.yes, dryRun: opts.dryRun, only: opts.only, provider: opts.provider })
|
||||
await runOptimizeApply(projects, range, { yes: opts.yes, dryRun: opts.dryRun, only: opts.only })
|
||||
return
|
||||
}
|
||||
assertFormat(format, ['text', 'json'], 'optimize')
|
||||
|
|
@ -1849,9 +1849,9 @@ program
|
|||
appliedHeader = buildOptimizeAppliedHeader(applied) ?? undefined
|
||||
previouslyApplied = applied.appliedByFinding
|
||||
} catch { /* the header is optional; never block the findings */ }
|
||||
await runOptimize(projects, label, range, { format, appliedHeader, previouslyApplied, provider: opts.provider })
|
||||
await runOptimize(projects, label, range, { format, appliedHeader, previouslyApplied })
|
||||
} else {
|
||||
await runOptimize(projects, label, range, { format, provider: opts.provider })
|
||||
await runOptimize(projects, label, range, { format })
|
||||
}
|
||||
})
|
||||
|
||||
|
|
@ -2075,7 +2075,6 @@ program
|
|||
.option('--by-agent', 'One row per (provider, model, agent) instead of one row per (provider, model). Claude subagent transcripts only; other providers and main sessions bucket under "main"')
|
||||
.option('--top <n>', 'Show only the top N rows', (v: string) => parseInt(v, 10))
|
||||
.option('--min-cost <usd>', 'Hide rows below this cost threshold', (v: string) => parseFloat(v))
|
||||
.option('--unpriced', 'Show only models with usage that currently price at $0')
|
||||
.option('--no-totals', 'Suppress the footer totals row')
|
||||
.option('--format <format>', 'Output format: table, markdown, json, csv', 'table')
|
||||
.action(async (opts) => {
|
||||
|
|
@ -2100,21 +2099,13 @@ program
|
|||
}
|
||||
|
||||
const projects = await parseAllSessions(range, opts.provider)
|
||||
let rows = await aggregateModels(projects, {
|
||||
const rows = await aggregateModels(projects, {
|
||||
byTask: !!opts.byTask,
|
||||
byAgent: !!opts.byAgent,
|
||||
taskFilter: opts.task,
|
||||
topN: typeof opts.top === 'number' && Number.isFinite(opts.top) ? opts.top : undefined,
|
||||
minCost: typeof opts.minCost === 'number' && Number.isFinite(opts.minCost) ? opts.minCost : (opts.unpriced ? 0 : 0.01),
|
||||
minCost: typeof opts.minCost === 'number' && Number.isFinite(opts.minCost) ? opts.minCost : 0.01,
|
||||
})
|
||||
if (opts.unpriced) {
|
||||
rows = rows.filter(row => findUnpricedModels([{
|
||||
model: row.model,
|
||||
calls: row.calls,
|
||||
cost: row.costUSD,
|
||||
tokens: row.totalTokens,
|
||||
}]).length > 0)
|
||||
}
|
||||
|
||||
const fmt = (opts.format ?? 'table').toLowerCase()
|
||||
if (rows.length === 0 && (fmt === 'table' || fmt === 'markdown')) {
|
||||
|
|
|
|||
|
|
@ -307,7 +307,7 @@ const BUILTIN_ALIASES: Record<string, string> = {
|
|||
// reports that quote literal slugs (e.g. forum.cursor.com/t/154933).
|
||||
'claude-4-sonnet': 'claude-sonnet-4',
|
||||
'claude-4-sonnet-1m': 'claude-sonnet-4',
|
||||
'claude-4-sonnet-thinking': 'claude-sonnet-4',
|
||||
'claude-4-sonnet-thinking': 'claude-sonnet-4-5',
|
||||
'claude-4.5-sonnet': 'claude-sonnet-4-5',
|
||||
'claude-4.5-sonnet-thinking': 'claude-sonnet-4-5',
|
||||
'claude-4.6-sonnet': 'claude-sonnet-4-6',
|
||||
|
|
|
|||
|
|
@ -551,18 +551,7 @@ export async function scanJsonlFile(
|
|||
return { calls, cwds, apiCalls, userMessages }
|
||||
}
|
||||
|
||||
// The session scan reads Claude Code transcripts only, so a `--provider` that
|
||||
// excludes Claude leaves nothing for it to do. Callers must also skip the
|
||||
// detectors it feeds (see `claudeOnly` in scanAndDetect) — the empty scan
|
||||
// returned here is an absence of measurement, not a measurement of absence.
|
||||
export function providerCoversClaude(provider?: string): boolean {
|
||||
return !provider || provider === 'all' || provider === 'claude'
|
||||
}
|
||||
|
||||
async function scanSessions(dateRange?: DateRange, provider?: string): Promise<ScanData> {
|
||||
if (!providerCoversClaude(provider)) {
|
||||
return { toolCalls: [], projectCwds: new Set(), apiCalls: [], userMessages: [] }
|
||||
}
|
||||
async function scanSessions(dateRange?: DateRange): Promise<ScanData> {
|
||||
const sources = await discoverAllSessions('claude')
|
||||
const allCalls: ToolCall[] = []
|
||||
const allCwds = new Set<string>()
|
||||
|
|
@ -2711,7 +2700,9 @@ export function findContextBloatCandidates(projects: ProjectSummary[]): ContextB
|
|||
|
||||
for (const session of sessions) {
|
||||
const inputTokens = sessionEffectiveContextTokens(session)
|
||||
const outputTokens = session.totalOutputTokens
|
||||
// Reasoning is stored separately from ordinary output, but both are
|
||||
// generated tokens for this detector. Reports already use their sum.
|
||||
const outputTokens = session.totalOutputTokens + session.totalReasoningTokens
|
||||
const ratio = inputTokens / Math.max(outputTokens, 1)
|
||||
const currentMs = new Date(session.firstTimestamp).getTime()
|
||||
const gapMs = previousTimestampMs !== null ? currentMs - previousTimestampMs : null
|
||||
|
|
@ -2988,7 +2979,7 @@ export function computeInputCostRate(projects: ProjectSummary[]): number {
|
|||
type CacheEntry = { data: OptimizeResult; ts: number }
|
||||
const resultCache = new Map<string, CacheEntry>()
|
||||
|
||||
export function cacheKey(projects: ProjectSummary[], dateRange: DateRange | undefined, provider?: string): string {
|
||||
export function cacheKey(projects: ProjectSummary[], dateRange: DateRange | undefined): string {
|
||||
const dr = dateRange ? `${dateRange.start.getTime()}-${dateRange.end.getTime()}` : 'all'
|
||||
// Fingerprint enough of the dataset that two materially different inputs
|
||||
// cannot collide onto one cached OptimizeResult. Project count + api-call
|
||||
|
|
@ -3005,27 +2996,23 @@ export function cacheKey(projects: ProjectSummary[], dateRange: DateRange | unde
|
|||
}
|
||||
// Costs scaled to whole micro-dollars so float jitter cannot thrash the key.
|
||||
const fingerprint = `${projects.length}:${calls}:${Math.round(cost * 1e6)}:${Math.round(savings * 1e6)}:${Math.round(proxied * 1e6)}`
|
||||
// The provider decides whether the Claude session scan runs at all, so two
|
||||
// filters that happen to share a project fingerprint must not share a result.
|
||||
return `${provider ?? 'all'}:${dr}:${fingerprint}`
|
||||
return `${dr}:${fingerprint}`
|
||||
}
|
||||
|
||||
export async function scanAndDetect(
|
||||
projects: ProjectSummary[],
|
||||
dateRange?: DateRange,
|
||||
provider?: string,
|
||||
): Promise<OptimizeResult> {
|
||||
if (projects.length === 0) {
|
||||
return { findings: [], costRate: 0, healthScore: 100, healthGrade: 'A', modelRecommendations: [] }
|
||||
}
|
||||
|
||||
const key = cacheKey(projects, dateRange, provider)
|
||||
const key = cacheKey(projects, dateRange)
|
||||
const cached = resultCache.get(key)
|
||||
if (cached && Date.now() - cached.ts < RESULT_CACHE_TTL_MS) return cached.data
|
||||
|
||||
const costRate = computeInputCostRate(projects)
|
||||
const scanCoversClaude = providerCoversClaude(provider)
|
||||
const { toolCalls, projectCwds, apiCalls, userMessages } = await scanSessions(dateRange, provider)
|
||||
const { toolCalls, projectCwds, apiCalls, userMessages } = await scanSessions(dateRange)
|
||||
const mcpCoverage = aggregateMcpCoverage(projects)
|
||||
|
||||
const findings: WasteFinding[] = []
|
||||
|
|
@ -3040,44 +3027,35 @@ export async function scanAndDetect(
|
|||
)
|
||||
const firstSessionIds = findYoungProjectFirstSessionIds(projects)
|
||||
const outlierExclusions = new Set([...lowWorthSessionIds, ...contextBloatVisibleIds, ...firstSessionIds])
|
||||
// Detectors fed by the session scan or by `~/.claude` config only mean
|
||||
// anything when the run covers Claude. Under a different `--provider` they
|
||||
// must be skipped rather than handed an empty scan: emptiness reads as
|
||||
// "never invoked", so every skill, agent and command would be reported as
|
||||
// unused when it was simply not measured.
|
||||
const claudeOnly = (detect: () => WasteFinding | null): (() => WasteFinding | null) =>
|
||||
scanCoversClaude ? detect : () => null
|
||||
const syncDetectors: Array<() => WasteFinding | null> = [
|
||||
claudeOnly(() => detectCacheBloat(apiCalls, projects, dateRange)),
|
||||
claudeOnly(() => detectLowReadEditRatio(toolCalls)),
|
||||
claudeOnly(() => detectJunkReads(toolCalls, dateRange)),
|
||||
claudeOnly(() => detectDuplicateReads(toolCalls, dateRange)),
|
||||
claudeOnly(() => detectUnusedMcp(toolCalls, projects, projectCwds, mcpCoverage)),
|
||||
() => detectCacheBloat(apiCalls, projects, dateRange),
|
||||
() => detectLowReadEditRatio(toolCalls),
|
||||
() => detectJunkReads(toolCalls, dateRange),
|
||||
() => detectDuplicateReads(toolCalls, dateRange),
|
||||
() => detectUnusedMcp(toolCalls, projects, projectCwds, mcpCoverage),
|
||||
() => detectMcpToolCoverage(projects, mcpCoverage),
|
||||
() => detectMcpProfileAdvisor(projects, mcpCoverage),
|
||||
// mcp-deferral-gaps family (#614): detection only, no apply plans yet.
|
||||
claudeOnly(() => detectMcpDeferralOff(toolCalls, projects, projectCwds, apiCalls)),
|
||||
claudeOnly(() => detectMcpAlwaysLoadHygiene(projects, projectCwds, apiCalls, mcpCoverage)),
|
||||
claudeOnly(() => detectMcpDeferThreshold(projects, projectCwds)),
|
||||
() => detectMcpDeferralOff(toolCalls, projects, projectCwds, apiCalls),
|
||||
() => detectMcpAlwaysLoadHygiene(projects, projectCwds, apiCalls, mcpCoverage),
|
||||
() => detectMcpDeferThreshold(projects, projectCwds),
|
||||
() => detectCapabilityReliability(projects),
|
||||
() => detectLowWorthSessions(projects),
|
||||
() => detectContextBloat(projects, lowWorthSessionIds),
|
||||
() => detectSessionOutliers(projects, outlierExclusions),
|
||||
claudeOnly(() => detectBloatedClaudeMd(projectCwds)),
|
||||
claudeOnly(() => detectBashBloat()),
|
||||
() => detectBloatedClaudeMd(projectCwds),
|
||||
() => detectBashBloat(),
|
||||
]
|
||||
for (const detect of syncDetectors) {
|
||||
const finding = detect()
|
||||
if (finding) findings.push(finding)
|
||||
}
|
||||
|
||||
const ghostResults = scanCoversClaude
|
||||
? await Promise.all([
|
||||
detectGhostAgents(toolCalls),
|
||||
detectGhostSkills(toolCalls),
|
||||
detectGhostCommands(userMessages),
|
||||
])
|
||||
: []
|
||||
const ghostResults = await Promise.all([
|
||||
detectGhostAgents(toolCalls),
|
||||
detectGhostSkills(toolCalls),
|
||||
detectGhostCommands(userMessages),
|
||||
])
|
||||
for (const f of ghostResults) if (f) findings.push(f)
|
||||
|
||||
findings.sort((a, b) => urgencyScore(b) - urgencyScore(a))
|
||||
|
|
@ -3305,7 +3283,7 @@ export async function runOptimize(
|
|||
projects: ProjectSummary[],
|
||||
periodLabel: string,
|
||||
dateRange?: DateRange,
|
||||
opts: { format?: 'text' | 'json'; appliedHeader?: string; previouslyApplied?: Record<string, string>; provider?: string } = {},
|
||||
opts: { format?: 'text' | 'json'; appliedHeader?: string; previouslyApplied?: Record<string, string> } = {},
|
||||
): Promise<void> {
|
||||
const format = opts.format ?? 'text'
|
||||
if (projects.length === 0 && format === 'text') {
|
||||
|
|
@ -3317,7 +3295,7 @@ export async function runOptimize(
|
|||
process.stderr.write(chalk.dim(' Analyzing your sessions...\n'))
|
||||
}
|
||||
|
||||
const result = await scanAndDetect(projects, dateRange, opts.provider)
|
||||
const result = await scanAndDetect(projects, dateRange)
|
||||
const { findings, costRate, healthScore, healthGrade } = result
|
||||
const sessions = projects.flatMap(p => p.sessions)
|
||||
const periodCost = projects.reduce((s, p) => s + p.totalCostUSD, 0)
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ import { basename, dirname, join } from 'path'
|
|||
import { homedir } from 'os'
|
||||
|
||||
import { readSessionFile } from '../fs-utils.js'
|
||||
import { calculateCost, getShortModelName } from '../models.js'
|
||||
import { calculateCost, getModelCosts, getShortModelName } from '../models.js'
|
||||
import { extractBashCommands } from '../bash-utils.js'
|
||||
import type { ProbeRoot, Provider, SessionSource, SessionParser, ParsedProviderCall } from './types.js'
|
||||
|
||||
|
|
@ -12,14 +12,15 @@ import type { ProbeRoot, Provider, SessionSource, SessionParser, ParsedProviderC
|
|||
// or ~/.grok. Each session dir holds summary.json, signals.json, and the ACP
|
||||
// log updates.jsonl.
|
||||
//
|
||||
// Grok does NOT record billable input/output tokens. signals.json carries
|
||||
// `contextTokensUsed` (current context fill) and updates.jsonl carries a running
|
||||
// `_meta.totalTokens` per streamed chunk; there is no per-call input/output
|
||||
// split. We reconstruct an ESTIMATE from the per-turn totalTokens curve. Agentic
|
||||
// turns re-send the growing context every call, and that re-sent context is
|
||||
// cached server-side, so we bill the unique context (summed per compaction segment) as fresh input,
|
||||
// the re-sent remainder as cache reads, and the per-turn growth as output. Cost
|
||||
// is flagged estimated; grok-build is priced via its grok-build-0.1 alias.
|
||||
// Newer Grok CLI versions append a `turn_completed` update with provider-recorded
|
||||
// input/output/cache/reasoning usage. That record is authoritative: cached reads
|
||||
// are part of input, and reasoning is a subset of output. Cache creation is
|
||||
// treated as another input subset by analogy because the record exposes no
|
||||
// separate fresh-input field; any per-record violation is clamped before pricing.
|
||||
// Older sessions only carry
|
||||
// `signals.json.contextTokensUsed` and the running `_meta.totalTokens` curve; for
|
||||
// those we retain the old compaction-aware estimate and mark its cost estimated.
|
||||
// `costUsdTicks` is deliberately ignored because its scale is not documented.
|
||||
|
||||
const toolNameMap: Record<string, string> = {
|
||||
bash: 'Bash',
|
||||
|
|
@ -80,28 +81,136 @@ function safeDecode(name: string): string {
|
|||
}
|
||||
}
|
||||
|
||||
// updates.jsonl is one ACP JSON-RPC notification per line; streamed chunks carry
|
||||
// params._meta.{totalTokens, promptId}. totalTokens is the running context size,
|
||||
// so grouping by promptId (one per turn) gives each turn's first/last value.
|
||||
// updates.jsonl is one ACP JSON-RPC notification per line. Streamed chunks carry
|
||||
// params._meta.{totalTokens, promptId}; completed turns carry snake_case
|
||||
// params.update.{prompt_id, usage}.
|
||||
type GrokUpdate = {
|
||||
params?: {
|
||||
_meta?: { totalTokens?: number; promptId?: string }
|
||||
update?: { sessionUpdate?: string; title?: string; rawInput?: { command?: unknown; subagent_type?: unknown } }
|
||||
_meta?: { totalTokens?: unknown; promptId?: unknown }
|
||||
update?: {
|
||||
sessionUpdate?: unknown
|
||||
prompt_id?: unknown
|
||||
usage?: unknown
|
||||
title?: unknown
|
||||
rawInput?: { command?: unknown; subagent_type?: unknown }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Single pass over updates.jsonl: per-turn totalTokens for the cost estimate,
|
||||
// plus the real tool calls (each tool_call's title -> a tool, and
|
||||
// run_terminal_command's rawInput.command -> shell commands).
|
||||
function parseUpdates(updates: string): {
|
||||
type GrokUsageValues = {
|
||||
inputTokens: number
|
||||
outputTokens: number
|
||||
cacheReadTokens: number
|
||||
cacheCreationTokens: number
|
||||
reasoningTokens: number
|
||||
}
|
||||
|
||||
type GrokAuthoritativeUsage = GrokUsageValues & {
|
||||
modelUsage: Map<string, GrokUsageValues>
|
||||
}
|
||||
|
||||
type GrokTokenTotals = {
|
||||
input: number
|
||||
cacheRead: number
|
||||
output: number
|
||||
cacheCreation: number
|
||||
reasoning: number
|
||||
}
|
||||
|
||||
function emptyTokenTotals(): GrokTokenTotals {
|
||||
return { input: 0, cacheRead: 0, output: 0, cacheCreation: 0, reasoning: 0 }
|
||||
}
|
||||
|
||||
const authoritativeTokenFields = [
|
||||
'inputTokens',
|
||||
'outputTokens',
|
||||
'cachedReadTokens',
|
||||
'cacheCreationTokens',
|
||||
'reasoningTokens',
|
||||
] as const
|
||||
|
||||
function isRecord(value: unknown): value is Record<string, unknown> {
|
||||
return typeof value === 'object' && value !== null && !Array.isArray(value)
|
||||
}
|
||||
|
||||
// JSONL is third-party input. Keep the check local to this provider so bad
|
||||
// usage fields become absent rather than leaking NaN, negative tokens, or a
|
||||
// throwing arithmetic operation into the session aggregate.
|
||||
function finiteNonNegative(value: unknown): number | undefined {
|
||||
if (typeof value !== 'number' || !Number.isFinite(value) || value < 0) return undefined
|
||||
// Token counts this large are not meaningful in a session and can overflow
|
||||
// when summed or priced. Capping preserves the non-negative finite invariant.
|
||||
return Math.min(value, Number.MAX_SAFE_INTEGER)
|
||||
}
|
||||
|
||||
function addTokenCounts(left: number, right: number): number {
|
||||
return Math.min(Number.MAX_SAFE_INTEGER, left + right)
|
||||
}
|
||||
|
||||
function readUsageNumber(usage: Record<string, unknown>, field: string): number | undefined {
|
||||
return finiteNonNegative(usage[field])
|
||||
}
|
||||
|
||||
function readModelUsage(usage: Record<string, unknown>): Map<string, GrokUsageValues> {
|
||||
const modelUsage = usage['modelUsage']
|
||||
const result = new Map<string, GrokUsageValues>()
|
||||
if (!isRecord(modelUsage)) return result
|
||||
|
||||
for (const [modelId, rawModelUsage] of Object.entries(modelUsage)) {
|
||||
if (!modelId || !isRecord(rawModelUsage)) continue
|
||||
const values = authoritativeTokenFields.map((field) => finiteNonNegative(rawModelUsage[field]))
|
||||
if (!values.some((value) => value !== undefined)) continue
|
||||
result.set(modelId, {
|
||||
inputTokens: values[0] ?? 0,
|
||||
outputTokens: values[1] ?? 0,
|
||||
cacheReadTokens: values[2] ?? 0,
|
||||
cacheCreationTokens: values[3] ?? 0,
|
||||
reasoningTokens: values[4] ?? 0,
|
||||
})
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
function parseAuthoritativeUsage(raw: unknown): GrokAuthoritativeUsage | null {
|
||||
if (!isRecord(raw)) return null
|
||||
|
||||
const values = authoritativeTokenFields.map((field) => readUsageNumber(raw, field))
|
||||
const modelUsage = readModelUsage(raw)
|
||||
return {
|
||||
inputTokens: values[0] ?? 0,
|
||||
outputTokens: values[1] ?? 0,
|
||||
cacheReadTokens: values[2] ?? 0,
|
||||
cacheCreationTokens: values[3] ?? 0,
|
||||
reasoningTokens: values[4] ?? 0,
|
||||
modelUsage,
|
||||
}
|
||||
}
|
||||
|
||||
function chooseAuthoritativeModel(modelIds: string[], existingModel: string): string {
|
||||
// modelUsage is the best attribution signal, but it may contain a newer
|
||||
// provider id that this checkout cannot price yet (for example
|
||||
// `grok-4.6-build`). Prefer an actual model id when it prices; otherwise keep
|
||||
// the existing summary/signals id when that one prices, avoiding a truthful
|
||||
// but $0 row. If neither prices, retain the actual model id for attribution.
|
||||
const pricedActualModel = modelIds.find((modelId) => getModelCosts(modelId) !== null)
|
||||
if (pricedActualModel) return pricedActualModel
|
||||
if (getModelCosts(existingModel) !== null) return existingModel
|
||||
return modelIds[0] ?? existingModel
|
||||
}
|
||||
|
||||
// Single pass over updates.jsonl: retain the old per-turn totalTokens estimate,
|
||||
// the deduplicated authoritative turn records, and the real tool calls.
|
||||
function parseUpdates(updates: string): {
|
||||
usage: GrokTokenTotals
|
||||
modelIds: string[]
|
||||
authoritative: boolean
|
||||
hasUncompletedTurn: boolean
|
||||
tools: string[]
|
||||
bashCommands: string[]
|
||||
subagentTypes: string[]
|
||||
} {
|
||||
const turns = new Map<string, { first: number; last: number }>()
|
||||
const completedUsages = new Map<string, GrokAuthoritativeUsage>()
|
||||
const tools: string[] = []
|
||||
const bashCommands: string[] = []
|
||||
const subagentTypes: string[] = []
|
||||
|
|
@ -111,6 +220,7 @@ function parseUpdates(updates: string): {
|
|||
let prevTotal = -1
|
||||
let segmentPeak = 0
|
||||
let inputFresh = 0
|
||||
let completedWithoutPromptId = 0
|
||||
|
||||
for (const line of updates.split('\n')) {
|
||||
if (!line.trim()) continue
|
||||
|
|
@ -122,16 +232,16 @@ function parseUpdates(updates: string): {
|
|||
}
|
||||
if (!params) continue
|
||||
|
||||
const total = params._meta?.totalTokens
|
||||
if (typeof total === 'number') {
|
||||
const total = finiteNonNegative(params._meta?.totalTokens)
|
||||
if (total !== undefined) {
|
||||
if (prevTotal >= 0 && total < prevTotal * 0.5) {
|
||||
inputFresh += segmentPeak // close the segment a compaction just ended
|
||||
inputFresh = addTokenCounts(inputFresh, segmentPeak) // close the segment a compaction just ended
|
||||
segmentPeak = 0
|
||||
}
|
||||
if (total > segmentPeak) segmentPeak = total
|
||||
prevTotal = total
|
||||
|
||||
const promptId = params._meta?.promptId
|
||||
const promptId = typeof params._meta?.promptId === 'string' ? params._meta.promptId : undefined
|
||||
if (promptId) {
|
||||
const turn = turns.get(promptId)
|
||||
if (!turn) turns.set(promptId, { first: total, last: total })
|
||||
|
|
@ -140,6 +250,18 @@ function parseUpdates(updates: string): {
|
|||
}
|
||||
|
||||
const update = params.update
|
||||
if (update?.sessionUpdate === 'turn_completed') {
|
||||
const usage = parseAuthoritativeUsage(update.usage)
|
||||
if (usage) {
|
||||
const promptId = typeof update.prompt_id === 'string' && update.prompt_id.length > 0
|
||||
? update.prompt_id
|
||||
: `turn_completed:${completedWithoutPromptId++}`
|
||||
// Re-emitted turn_completed notifications are cumulative updates for
|
||||
// the same turn. Last write wins so they cannot double count.
|
||||
completedUsages.set(promptId, usage)
|
||||
}
|
||||
}
|
||||
|
||||
if (update?.sessionUpdate === 'tool_call' && typeof update.title === 'string') {
|
||||
tools.push(toolNameMap[update.title] ?? update.title)
|
||||
if (update.title === 'run_terminal_command' && typeof update.rawInput?.command === 'string') {
|
||||
|
|
@ -151,17 +273,101 @@ function parseUpdates(updates: string): {
|
|||
}
|
||||
}
|
||||
|
||||
inputFresh += segmentPeak // close the final segment
|
||||
inputFresh = addTokenCounts(inputFresh, segmentPeak) // close the final segment
|
||||
let sumFirst = 0
|
||||
let output = 0
|
||||
for (const { first, last } of turns.values()) {
|
||||
sumFirst += first
|
||||
output += Math.max(0, last - first)
|
||||
sumFirst = addTokenCounts(sumFirst, first)
|
||||
output = addTokenCounts(output, Math.max(0, last - first))
|
||||
}
|
||||
// Fresh input (summed segment peaks) is billed once; the rest of the per-turn
|
||||
// re-sends are cache reads (Grok caches them, even though it reports nothing).
|
||||
const cacheRead = Math.max(0, sumFirst - inputFresh)
|
||||
return { input: inputFresh, cacheRead, output, tools, bashCommands, subagentTypes }
|
||||
const estimated = {
|
||||
input: inputFresh,
|
||||
cacheRead: Math.max(0, sumFirst - inputFresh),
|
||||
output,
|
||||
}
|
||||
|
||||
const usageTotals = emptyTokenTotals()
|
||||
const modelIds: string[] = []
|
||||
const seenModelIds = new Set<string>()
|
||||
for (const usage of completedUsages.values()) {
|
||||
addUsageToTotals(usageTotals, usage)
|
||||
for (const modelId of usage.modelUsage.keys()) {
|
||||
if (seenModelIds.has(modelId)) continue
|
||||
seenModelIds.add(modelId)
|
||||
modelIds.push(modelId)
|
||||
}
|
||||
}
|
||||
|
||||
// Decide from the final, prompt-deduplicated records. A positive modelUsage
|
||||
// entry is attribution metadata only; it is not a substitute for the
|
||||
// top-level accounting fields. This also prevents a superseded positive
|
||||
// record from suppressing the streaming fallback.
|
||||
const hasPositiveCompletedUsage = [...completedUsages.values()].some(hasPositiveTopLevelUsage)
|
||||
if (!hasPositiveCompletedUsage || !hasPositiveTotals(usageTotals)) {
|
||||
// Older Grok CLI versions have no completed usage record. Keep the
|
||||
// heuristic only here; blending it into a real record would reintroduce the
|
||||
// large over-count this parser is fixing. A record that only has modelUsage,
|
||||
// or whose final deduplicated values are empty, is treated the same way.
|
||||
return {
|
||||
usage: { input: estimated.input, cacheRead: estimated.cacheRead, output: estimated.output, cacheCreation: 0, reasoning: 0 },
|
||||
modelIds: [],
|
||||
authoritative: false,
|
||||
hasUncompletedTurn: false,
|
||||
tools,
|
||||
bashCommands,
|
||||
subagentTypes,
|
||||
}
|
||||
}
|
||||
|
||||
const hasUncompletedTurn = [...turns.keys()].some(promptId => !completedUsages.has(promptId))
|
||||
|
||||
// calculateCost follows the cache-exclusive input convention used by the
|
||||
// other real-usage providers. Each record is decomposed before its totals
|
||||
// are added, so an inconsistent record cannot consume another record's fresh
|
||||
// input budget.
|
||||
return {
|
||||
usage: usageTotals,
|
||||
modelIds,
|
||||
authoritative: true,
|
||||
hasUncompletedTurn,
|
||||
tools,
|
||||
bashCommands,
|
||||
subagentTypes,
|
||||
}
|
||||
}
|
||||
|
||||
function hasPositiveTopLevelUsage(usage: GrokAuthoritativeUsage): boolean {
|
||||
return usage.inputTokens > 0
|
||||
|| usage.outputTokens > 0
|
||||
|| usage.cacheReadTokens > 0
|
||||
|| usage.cacheCreationTokens > 0
|
||||
|| usage.reasoningTokens > 0
|
||||
}
|
||||
|
||||
function addUsageToTotals(totals: GrokTokenTotals, usage: GrokUsageValues): void {
|
||||
// `cacheCreationTokens` is treated as an input subset by analogy. Clamp the
|
||||
// exclusive portion per record before summing the session. Reasoning is
|
||||
// reported inside outputTokens by Grok, so clamp it to that same record's
|
||||
// output before the session totals are accumulated.
|
||||
const reasoningTokens = Math.min(usage.reasoningTokens, usage.outputTokens)
|
||||
totals.input = addTokenCounts(
|
||||
totals.input,
|
||||
Math.max(0, usage.inputTokens - usage.cacheReadTokens - usage.cacheCreationTokens),
|
||||
)
|
||||
totals.cacheRead = addTokenCounts(totals.cacheRead, usage.cacheReadTokens)
|
||||
totals.output = addTokenCounts(totals.output, usage.outputTokens)
|
||||
totals.cacheCreation = addTokenCounts(totals.cacheCreation, usage.cacheCreationTokens)
|
||||
totals.reasoning = addTokenCounts(totals.reasoning, reasoningTokens)
|
||||
}
|
||||
|
||||
function hasPositiveTotals(totals: GrokTokenTotals): boolean {
|
||||
return totals.input > 0
|
||||
|| totals.cacheRead > 0
|
||||
|| totals.output > 0
|
||||
|| totals.cacheCreation > 0
|
||||
|| totals.reasoning > 0
|
||||
}
|
||||
|
||||
function createParser(source: SessionSource, seenKeys: Set<string>): SessionParser {
|
||||
|
|
@ -172,37 +378,64 @@ function createParser(source: SessionSource, seenKeys: Set<string>): SessionPars
|
|||
const updates = await readSessionFile(source.path)
|
||||
if (!summary || updates === null) return
|
||||
|
||||
const { input, cacheRead, output, tools, bashCommands, subagentTypes } = parseUpdates(updates)
|
||||
if (input === 0 && output === 0) return
|
||||
|
||||
const signals = await readJson<GrokSignals>(join(dir, 'signals.json'))
|
||||
const model =
|
||||
const existingModel =
|
||||
summary.current_model_id ?? signals?.primaryModelId ?? signals?.modelsUsed?.[0] ?? 'grok-build'
|
||||
const parsed = parseUpdates(updates)
|
||||
if (!hasPositiveTotals(parsed.usage)) return
|
||||
|
||||
const timestamp = summary.updated_at ?? summary.last_active_at ?? summary.created_at ?? ''
|
||||
const sessionId = summary.info?.id ?? basename(dir)
|
||||
|
||||
const dedupKey = `${source.provider}:${dir}:${timestamp}:${sessionId}`
|
||||
if (seenKeys.has(dedupKey)) return
|
||||
seenKeys.add(dedupKey)
|
||||
// Multi-model attribution is deliberately out of scope: modelUsage may
|
||||
// help choose a priced attribution id, but top-level totals remain the
|
||||
// accounting source and one session uses one model's rate.
|
||||
const model = parsed.authoritative ? chooseAuthoritativeModel(parsed.modelIds, existingModel) : existingModel
|
||||
|
||||
const baseDedupKey = `${source.provider}:${dir}:${timestamp}:${sessionId}`
|
||||
if (seenKeys.has(baseDedupKey)) return
|
||||
seenKeys.add(baseDedupKey)
|
||||
|
||||
// `addUsageToTotals` clamps reasoning per authoritative record before
|
||||
// summing, so the aggregate preserves this identity as well.
|
||||
const reasoningTokens = parsed.usage.reasoning
|
||||
yield {
|
||||
provider: source.provider,
|
||||
model,
|
||||
inputTokens: input,
|
||||
outputTokens: output,
|
||||
cacheCreationInputTokens: 0,
|
||||
cacheReadInputTokens: cacheRead,
|
||||
cachedInputTokens: cacheRead,
|
||||
reasoningTokens: 0,
|
||||
inputTokens: parsed.usage.input,
|
||||
// Grok reports reasoning INSIDE outputTokens, but the repo contract is
|
||||
// the opposite: ParsedProviderCall.reasoningTokens is exclusive of
|
||||
// outputTokens, and every consumer sums the two (parser.ts's
|
||||
// cachedCallToApiCall for cost, modelBreakdown for tokens, and the
|
||||
// models/audit reports). tests/providers/kiro.test.ts states it
|
||||
// outright. So split it here rather than special-casing grok in five
|
||||
// downstream places: subtracting reasoning makes `output + reasoning`
|
||||
// reconstruct exactly the number Grok reported.
|
||||
outputTokens: parsed.usage.output - reasoningTokens,
|
||||
cacheCreationInputTokens: parsed.usage.cacheCreation,
|
||||
cacheReadInputTokens: parsed.usage.cacheRead,
|
||||
cachedInputTokens: parsed.usage.cacheRead,
|
||||
reasoningTokens,
|
||||
webSearchRequests: 0,
|
||||
costUSD: calculateCost(model, input, output, 0, cacheRead, 0),
|
||||
costIsEstimated: true,
|
||||
tools,
|
||||
bashCommands,
|
||||
subagentTypes,
|
||||
// Authoritative token counts are measured even though CodeBurn applies
|
||||
// its own pricing table; only the legacy context-curve path is an
|
||||
// estimate. The full provider output is priced once here, which is
|
||||
// what the downstream `output + reasoning` recompute reproduces.
|
||||
costUSD: calculateCost(
|
||||
model,
|
||||
parsed.usage.input,
|
||||
parsed.usage.output,
|
||||
parsed.usage.cacheCreation,
|
||||
parsed.usage.cacheRead,
|
||||
0,
|
||||
),
|
||||
costIsEstimated: !parsed.authoritative || parsed.hasUncompletedTurn,
|
||||
tools: parsed.tools,
|
||||
bashCommands: parsed.bashCommands,
|
||||
subagentTypes: parsed.subagentTypes,
|
||||
timestamp,
|
||||
speed: 'standard',
|
||||
deduplicationKey: dedupKey,
|
||||
deduplicationKey: baseDedupKey,
|
||||
userMessage: summary.session_summary ?? summary.generated_title ?? '',
|
||||
sessionId,
|
||||
project: source.project,
|
||||
|
|
|
|||
|
|
@ -283,7 +283,10 @@ export const PROVIDER_PARSE_VERSIONS: Record<string, string> = {
|
|||
// transcripts (both carry producer 'copilot-agent'), skipping the shutdown
|
||||
// input/cache rollup; this bump re-parses them so the missing tokens land.
|
||||
copilot: 'cli-shutdown-cost-v1-skills-source-provenance-v1',
|
||||
grok: 'estimated-cost-v1',
|
||||
// authoritative-usage-v4: persist one Grok session call from top-level
|
||||
// authoritative totals, use modelUsage only for priced attribution, clamp
|
||||
// reasoning per record, and label mixed sessions estimated.
|
||||
grok: 'authoritative-usage-v4',
|
||||
hermes: 'reasoning-output-accounting-v1-est-cost',
|
||||
'lingtai-tui': 'token-ledger-registry-activity-v3',
|
||||
'ibm-bob': 'worktree-project-grouping-v1',
|
||||
|
|
|
|||
|
|
@ -935,7 +935,7 @@ export async function buildMenubarPayloadForRange(periodInfo: PeriodInfo, opts:
|
|||
}
|
||||
})()
|
||||
|
||||
const optimize = opts.optimize === false ? null : await scanAndDetect(scanProjects, scanRange, opts.provider)
|
||||
const optimize = opts.optimize === false ? null : await scanAndDetect(scanProjects, scanRange)
|
||||
const granularRange = opts.daysSelection?.range ?? scanRange
|
||||
const granularHistory = opts.timeline === false ? undefined : buildGranularHistory(scanProjects, granularRange)
|
||||
return buildMenubarPayload(currentData, providers, optimize, dailyHistory, retryTax, routingWaste, breakdowns, claudeConfigs, granularHistory)
|
||||
|
|
|
|||
100
tests/daily-cache-grok-rederivation.test.ts
Normal file
100
tests/daily-cache-grok-rederivation.test.ts
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { mkdir, readFile, rm, writeFile } from 'fs/promises'
|
||||
import { join } from 'path'
|
||||
import { tmpdir } from 'os'
|
||||
|
||||
import {
|
||||
currentTzKey,
|
||||
ensureCacheHydrated,
|
||||
toDateString,
|
||||
type DailyEntry,
|
||||
} from '../src/daily-cache.js'
|
||||
|
||||
const PRE_FIX_DAILY_VERSION = 18
|
||||
const cacheRoot = join(tmpdir(), `codeburn-grok-daily-${process.pid}-${Date.now()}`)
|
||||
|
||||
function day(date: string, cost: number): DailyEntry {
|
||||
return {
|
||||
date,
|
||||
cost,
|
||||
savingsUSD: 0,
|
||||
calls: 1,
|
||||
sessions: 1,
|
||||
inputTokens: 100,
|
||||
outputTokens: 20,
|
||||
cacheReadTokens: 30,
|
||||
cacheWriteTokens: 0,
|
||||
editTurns: 0,
|
||||
oneShotTurns: 0,
|
||||
models: {
|
||||
'Grok Build': {
|
||||
calls: 1,
|
||||
cost,
|
||||
savingsUSD: 0,
|
||||
inputTokens: 100,
|
||||
outputTokens: 20,
|
||||
cacheReadTokens: 30,
|
||||
cacheWriteTokens: 0,
|
||||
},
|
||||
},
|
||||
categories: {},
|
||||
providers: {
|
||||
grok: {
|
||||
calls: 1,
|
||||
cost,
|
||||
savingsUSD: 0,
|
||||
sessions: 1,
|
||||
inputTokens: 100,
|
||||
outputTokens: 20,
|
||||
cacheReadTokens: 30,
|
||||
cacheWriteTokens: 0,
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
process.env['CODEBURN_CACHE_DIR'] = cacheRoot
|
||||
await rm(cacheRoot, { recursive: true, force: true })
|
||||
await mkdir(cacheRoot, { recursive: true })
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await rm(cacheRoot, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
describe('Grok daily-cache accounting rederivation', () => {
|
||||
it('re-derives a v18 Grok day while preserving the old cache as the baseline', async () => {
|
||||
const date = toDateString(new Date(Date.now() - 7 * 24 * 60 * 60 * 1000))
|
||||
const yesterday = toDateString(new Date(Date.now() - 24 * 60 * 60 * 1000))
|
||||
const oldPath = join(cacheRoot, `daily-cache.v${PRE_FIX_DAILY_VERSION}.json`)
|
||||
const oldCache = {
|
||||
version: PRE_FIX_DAILY_VERSION,
|
||||
savingsConfigHash: 'cfg',
|
||||
tzKey: currentTzKey(),
|
||||
lastComputedDate: yesterday,
|
||||
days: [day(date, 99)],
|
||||
complete: true,
|
||||
watermarkTrusted: true,
|
||||
}
|
||||
await writeFile(oldPath, JSON.stringify(oldCache))
|
||||
|
||||
let parseCount = 0
|
||||
const corrected = day(date, 2)
|
||||
const hydrated = await ensureCacheHydrated(
|
||||
async () => {
|
||||
parseCount++
|
||||
return []
|
||||
},
|
||||
() => [corrected],
|
||||
'cfg',
|
||||
() => true,
|
||||
)
|
||||
|
||||
const refreshedDay = hydrated.days.find(entry => entry.date === date)
|
||||
expect(parseCount).toBe(1)
|
||||
expect(refreshedDay?.providers.grok?.cost).toBe(2)
|
||||
expect(refreshedDay?.cost).toBe(2)
|
||||
expect(JSON.parse(await readFile(oldPath, 'utf8'))).toEqual(oldCache)
|
||||
})
|
||||
})
|
||||
266
tests/grok-parser-pipeline.test.ts
Normal file
266
tests/grok-parser-pipeline.test.ts
Normal file
|
|
@ -0,0 +1,266 @@
|
|||
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
|
||||
import { mkdir, rm, writeFile } from 'fs/promises'
|
||||
import { join } from 'path'
|
||||
|
||||
import { calculateCost } from '../src/models.js'
|
||||
import { clearSessionCache, parseAllSessions } from '../src/parser.js'
|
||||
|
||||
// The exported Grok provider resolves GROK_HOME when its singleton is created,
|
||||
// before the test body runs. Set the root during module hoisting, then re-assert
|
||||
// the call-time cache/env values in beforeEach after env-isolation runs.
|
||||
const testRoot = vi.hoisted(() => {
|
||||
const root = `${process.env['TMPDIR'] || '/tmp'}/grok-pipeline-${process.pid}-${Date.now()}`
|
||||
process.env['GROK_HOME'] = `${root}/grok`
|
||||
return root
|
||||
})
|
||||
|
||||
const GROK_HOME = join(testRoot, 'grok')
|
||||
const CACHE_DIR = join(testRoot, 'cache')
|
||||
|
||||
type UsageOptions = {
|
||||
input: number
|
||||
output: number
|
||||
cacheRead?: number
|
||||
cacheCreation?: number
|
||||
reasoning?: number
|
||||
model?: string
|
||||
modelUsage?: Record<string, Record<string, unknown>>
|
||||
}
|
||||
|
||||
type StreamingTurn = {
|
||||
promptId: string
|
||||
totals: number[]
|
||||
}
|
||||
|
||||
type CompletedTurn = {
|
||||
promptId?: string
|
||||
usage: Record<string, unknown>
|
||||
}
|
||||
|
||||
function usage(opts: UsageOptions): Record<string, unknown> {
|
||||
const cacheRead = opts.cacheRead ?? 0
|
||||
const cacheCreation = opts.cacheCreation ?? 0
|
||||
const reasoning = opts.reasoning ?? 0
|
||||
const model = opts.model ?? 'grok-build'
|
||||
const singleModel = {
|
||||
inputTokens: opts.input,
|
||||
outputTokens: opts.output,
|
||||
cachedReadTokens: cacheRead,
|
||||
cacheCreationTokens: cacheCreation,
|
||||
reasoningTokens: reasoning,
|
||||
}
|
||||
return {
|
||||
inputTokens: opts.input,
|
||||
outputTokens: opts.output,
|
||||
cachedReadTokens: cacheRead,
|
||||
cacheCreationTokens: cacheCreation,
|
||||
reasoningTokens: reasoning,
|
||||
modelUsage: opts.modelUsage ?? { [model]: singleModel },
|
||||
}
|
||||
}
|
||||
|
||||
async function writeSession(
|
||||
record: Record<string, unknown>,
|
||||
uuid = '019edf9c-0000-7000-8000-000000000101',
|
||||
options: { turns?: StreamingTurn[]; completedTurns?: CompletedTurn[] } = {},
|
||||
): Promise<void> {
|
||||
const cwd = '/Users/test/grok-pipeline'
|
||||
const dir = join(GROK_HOME, 'sessions', '%2FUsers%2Ftest%2Fgrok-pipeline', uuid)
|
||||
await mkdir(dir, { recursive: true })
|
||||
await writeFile(join(dir, 'summary.json'), JSON.stringify({
|
||||
info: { id: uuid, cwd },
|
||||
created_at: '2026-08-17T09:00:00.000Z',
|
||||
updated_at: '2026-08-17T09:05:00.000Z',
|
||||
current_model_id: 'grok-build',
|
||||
session_summary: 'pipeline regression',
|
||||
}))
|
||||
await writeFile(join(dir, 'signals.json'), JSON.stringify({
|
||||
primaryModelId: 'grok-build',
|
||||
modelsUsed: ['grok-build'],
|
||||
}))
|
||||
const completedTurns = options.completedTurns ?? [{ promptId: 'pipeline-turn', usage: record }]
|
||||
const lines: Record<string, unknown>[] = []
|
||||
for (const turn of options.turns ?? []) {
|
||||
for (const totalTokens of turn.totals) {
|
||||
lines.push({
|
||||
method: 'session/update',
|
||||
params: {
|
||||
sessionId: uuid,
|
||||
_meta: { eventId: `stream-${turn.promptId}-${totalTokens}`, totalTokens, promptId: turn.promptId },
|
||||
},
|
||||
})
|
||||
}
|
||||
}
|
||||
for (const [index, completed] of completedTurns.entries()) {
|
||||
lines.push({
|
||||
method: 'session/update',
|
||||
params: {
|
||||
sessionId: uuid,
|
||||
update: {
|
||||
sessionUpdate: 'turn_completed',
|
||||
...(completed.promptId !== undefined ? { prompt_id: completed.promptId } : {}),
|
||||
usage: completed.usage,
|
||||
},
|
||||
_meta: { eventId: `completed-${index}` },
|
||||
},
|
||||
})
|
||||
}
|
||||
await writeFile(join(dir, 'updates.jsonl'), lines.map(line => JSON.stringify(line)).join('\n') + '\n')
|
||||
}
|
||||
|
||||
async function parseGrokSessions() {
|
||||
const projects = await parseAllSessions(undefined, 'grok')
|
||||
return projects.flatMap(project => project.sessions)
|
||||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
clearSessionCache()
|
||||
await rm(testRoot, { recursive: true, force: true })
|
||||
process.env['GROK_HOME'] = GROK_HOME
|
||||
process.env['CODEBURN_CACHE_DIR'] = CACHE_DIR
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
clearSessionCache()
|
||||
await rm(testRoot, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
describe('Grok parser through the session-cache pipeline', () => {
|
||||
it('keeps reasoning inside output pricing on both cold and warm parses', async () => {
|
||||
await writeSession(usage({ input: 1000, output: 200, cacheRead: 500, cacheCreation: 100, reasoning: 150 }))
|
||||
|
||||
const cold = (await parseGrokSessions())[0]!
|
||||
const coldCall = cold.turns[0]!.assistantCalls[0]!
|
||||
clearSessionCache()
|
||||
const warm = (await parseGrokSessions())[0]!
|
||||
const warmCall = warm.turns[0]!.assistantCalls[0]!
|
||||
const expected = calculateCost('grok-build', 400, 200, 100, 500, 0)
|
||||
|
||||
expect(cold.apiCalls).toBe(1)
|
||||
expect(coldCall.costUSD).toBeCloseTo(expected, 12)
|
||||
expect(warm.apiCalls).toBe(1)
|
||||
expect(warmCall.costUSD).toBeCloseTo(expected, 12)
|
||||
expect(warmCall.costUSD).not.toBeCloseTo(calculateCost('grok-build', 400, 350, 100, 500, 0), 12)
|
||||
|
||||
// Cost is only half of it: `models` and the audit report sum
|
||||
// outputTokens + reasoningTokens for the token column. Emitting the
|
||||
// provider's cache-inclusive output verbatim inflated that column by the
|
||||
// reasoning tokens even once the cost was right, so pin the split and the
|
||||
// sum the reports actually render.
|
||||
const breakdown = Object.values(cold.modelBreakdown)[0]!
|
||||
expect(breakdown.tokens.outputTokens).toBe(50) // 200 reported - 150 reasoning
|
||||
expect(breakdown.tokens.reasoningTokens).toBe(150)
|
||||
expect(breakdown.tokens.outputTokens + breakdown.tokens.reasoningTokens).toBe(200)
|
||||
})
|
||||
|
||||
it('keeps one session call and uses top-level totals for a multi-model record', async () => {
|
||||
await writeSession(usage({
|
||||
input: 3000,
|
||||
output: 300,
|
||||
cacheRead: 600,
|
||||
cacheCreation: 100,
|
||||
reasoning: 30,
|
||||
modelUsage: {
|
||||
'grok-4.6-build': {
|
||||
inputTokens: 2000,
|
||||
outputTokens: 200,
|
||||
cachedReadTokens: 500,
|
||||
cacheCreationTokens: 100,
|
||||
reasoningTokens: 20,
|
||||
},
|
||||
'grok-latest': {
|
||||
inputTokens: 1000,
|
||||
outputTokens: 100,
|
||||
cachedReadTokens: 100,
|
||||
cacheCreationTokens: 0,
|
||||
reasoningTokens: 10,
|
||||
},
|
||||
},
|
||||
}), '019edf9c-0000-7000-8000-000000000102')
|
||||
|
||||
const cold = (await parseGrokSessions())[0]!
|
||||
const coldCalls = cold.turns.flatMap(turn => turn.assistantCalls)
|
||||
const expected = calculateCost('grok-latest', 2300, 300, 100, 600, 0)
|
||||
|
||||
expect(cold.turns).toHaveLength(1)
|
||||
expect(cold.apiCalls).toBe(1)
|
||||
expect(coldCalls.map(call => call.model)).toEqual(['grok-latest'])
|
||||
expect(coldCalls.map(call => call.usage.inputTokens)).toEqual([2300])
|
||||
expect(coldCalls[0]!.usage.outputTokens + coldCalls[0]!.usage.reasoningTokens).toBe(300)
|
||||
expect(cold.totalCostUSD).toBeCloseTo(expected, 12)
|
||||
|
||||
clearSessionCache()
|
||||
const warm = (await parseGrokSessions())[0]!
|
||||
expect(warm.turns).toHaveLength(1)
|
||||
expect(warm.apiCalls).toBe(1)
|
||||
expect(warm.totalCostUSD).toBeCloseTo(expected, 12)
|
||||
})
|
||||
|
||||
it('falls back to the streaming estimate when usage exists only under modelUsage', async () => {
|
||||
await writeSession({
|
||||
modelUsage: {
|
||||
'grok-4.6-build': {
|
||||
inputTokens: 1000,
|
||||
outputTokens: 100,
|
||||
},
|
||||
},
|
||||
}, '019edf9c-0000-7000-8000-000000000103', {
|
||||
turns: [{ promptId: 'legacy-turn', totals: [1000, 1200] }],
|
||||
})
|
||||
|
||||
const sessions = await parseGrokSessions()
|
||||
expect(sessions).toHaveLength(1)
|
||||
const call = sessions[0]!.turns[0]!.assistantCalls[0]!
|
||||
expect(call.isEstimated).toBe(true)
|
||||
expect(call.usage.outputTokens).toBe(200)
|
||||
})
|
||||
|
||||
it('uses the final deduplicated record when deciding whether to estimate', async () => {
|
||||
await writeSession({}, '019edf9c-0000-7000-8000-000000000104', {
|
||||
turns: [{ promptId: 'superseded-turn', totals: [1000, 1200] }],
|
||||
completedTurns: [
|
||||
{ promptId: 'superseded-turn', usage: usage({ input: 1000, output: 100 }) },
|
||||
{ promptId: 'superseded-turn', usage: {} },
|
||||
],
|
||||
})
|
||||
|
||||
const sessions = await parseGrokSessions()
|
||||
expect(sessions).toHaveLength(1)
|
||||
const call = sessions[0]!.turns[0]!.assistantCalls[0]!
|
||||
expect(call.isEstimated).toBe(true)
|
||||
expect(call.usage.outputTokens).toBe(200)
|
||||
})
|
||||
|
||||
it('marks a mixed authoritative session estimated when a streamed turn has no record', async () => {
|
||||
await writeSession(usage({ input: 1000, output: 100 }), '019edf9c-0000-7000-8000-000000000105', {
|
||||
turns: [
|
||||
{ promptId: 'pre-upgrade-turn', totals: [1000, 1400] },
|
||||
{ promptId: 'authoritative-turn', totals: [1400, 1600] },
|
||||
],
|
||||
completedTurns: [{ promptId: 'authoritative-turn', usage: usage({ input: 800, output: 80 }) }],
|
||||
})
|
||||
|
||||
const sessions = await parseGrokSessions()
|
||||
expect(sessions).toHaveLength(1)
|
||||
const call = sessions[0]!.turns[0]!.assistantCalls[0]!
|
||||
expect(call.isEstimated).toBe(true)
|
||||
expect(call.usage.inputTokens).toBe(800)
|
||||
expect(call.usage.outputTokens + call.usage.reasoningTokens).toBe(80)
|
||||
})
|
||||
|
||||
it('clamps reasoning to reported output before the real pipeline prices the call', async () => {
|
||||
await writeSession(usage({ input: 1000, output: 100, cacheRead: 500, cacheCreation: 100, reasoning: 250 }), '019edf9c-0000-7000-8000-000000000106')
|
||||
|
||||
const sessions = await parseGrokSessions()
|
||||
const call = sessions[0]!.turns[0]!.assistantCalls[0]!
|
||||
const expected = calculateCost('grok-build', 400, 100, 100, 500, 0)
|
||||
expect(call.usage.outputTokens).toBe(0)
|
||||
expect(call.usage.reasoningTokens).toBe(100)
|
||||
expect(call.usage.outputTokens + call.usage.reasoningTokens).toBe(100)
|
||||
expect(call.costUSD).toBeCloseTo(expected, 12)
|
||||
|
||||
clearSessionCache()
|
||||
const warmCall = (await parseGrokSessions())[0]!.turns[0]!.assistantCalls[0]!
|
||||
expect(warmCall.costUSD).toBeCloseTo(expected, 12)
|
||||
})
|
||||
})
|
||||
|
|
@ -1,9 +1,6 @@
|
|||
import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { spawnSync } from 'node:child_process'
|
||||
|
||||
import { describe, it, expect, vi } from 'vitest'
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import chalk from 'chalk'
|
||||
import stripAnsi from 'strip-ansi'
|
||||
|
||||
|
|
@ -716,66 +713,6 @@ describe('renderCsv', () => {
|
|||
})
|
||||
|
||||
describe('models CLI breakdown flags', () => {
|
||||
vi.setConfig({ testTimeout: 30_000 })
|
||||
|
||||
it('filters the models report to unpriced rows', async () => {
|
||||
const home = await mkdtemp(join(tmpdir(), 'codeburn-models-unpriced-'))
|
||||
try {
|
||||
const projectDir = join(home, '.claude', 'projects', 'models-unpriced')
|
||||
await mkdir(projectDir, { recursive: true })
|
||||
await writeFile(join(projectDir, 'session.jsonl'), [
|
||||
JSON.stringify({
|
||||
type: 'user',
|
||||
sessionId: 'models-unpriced-session',
|
||||
timestamp: '2026-05-09T00:00:00.000Z',
|
||||
cwd: '/tmp/models-unpriced',
|
||||
message: { role: 'user', content: 'Use one priced and one unpriced model.' },
|
||||
}),
|
||||
JSON.stringify({
|
||||
type: 'assistant',
|
||||
sessionId: 'models-unpriced-session',
|
||||
timestamp: '2026-05-09T00:01:00.000Z',
|
||||
cwd: '/tmp/models-unpriced',
|
||||
message: {
|
||||
id: 'priced',
|
||||
type: 'message',
|
||||
role: 'assistant',
|
||||
model: 'claude-sonnet-4-6',
|
||||
content: [{ type: 'text', text: 'priced' }],
|
||||
usage: { input_tokens: 1000, output_tokens: 100, cache_read_input_tokens: 0, cache_creation_input_tokens: 0 },
|
||||
},
|
||||
}),
|
||||
JSON.stringify({
|
||||
type: 'assistant',
|
||||
sessionId: 'models-unpriced-session',
|
||||
timestamp: '2026-05-09T00:02:00.000Z',
|
||||
cwd: '/tmp/models-unpriced',
|
||||
message: {
|
||||
id: 'unpriced',
|
||||
type: 'message',
|
||||
role: 'assistant',
|
||||
model: 'zz-unpriced-frontier-model',
|
||||
content: [{ type: 'text', text: 'unpriced' }],
|
||||
usage: { input_tokens: 2000, output_tokens: 200, cache_read_input_tokens: 0, cache_creation_input_tokens: 0 },
|
||||
},
|
||||
}),
|
||||
].join('\n') + '\n')
|
||||
|
||||
const res = spawnSync(
|
||||
process.execPath,
|
||||
['--import', 'tsx', 'src/cli.ts', 'models', '--unpriced', '--from', '2026-05-09', '--to', '2026-05-09', '--provider', 'claude', '--format', 'json'],
|
||||
{ cwd: process.cwd(), env: { ...process.env, HOME: home, CLAUDE_CONFIG_DIR: join(home, '.claude'), CODEBURN_CACHE_DIR: join(home, '.cache', 'codeburn'), TZ: 'UTC' }, encoding: 'utf-8', timeout: 30_000 },
|
||||
)
|
||||
|
||||
expect(res.status, `stdout: ${res.stdout}\nstderr: ${res.stderr}`).toBe(0)
|
||||
const rows = JSON.parse(res.stdout) as Array<{ model: string; calls: number }>
|
||||
expect(rows.map(row => row.model)).toEqual(['zz-unpriced-frontier-model'])
|
||||
expect(rows[0]?.calls).toBe(1)
|
||||
} finally {
|
||||
await rm(home, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects --by-task and --by-agent together with a clear error and exit 1', () => {
|
||||
const res = spawnSync(
|
||||
process.execPath,
|
||||
|
|
|
|||
|
|
@ -505,7 +505,7 @@ describe('Cursor model variants resolve to pricing', () => {
|
|||
// Sonnet family
|
||||
['claude-4-sonnet', 'claude-sonnet-4'],
|
||||
['claude-4-sonnet-1m', 'claude-sonnet-4'],
|
||||
['claude-4-sonnet-thinking', 'claude-sonnet-4'],
|
||||
['claude-4-sonnet-thinking', 'claude-sonnet-4-5'],
|
||||
['claude-4.5-sonnet', 'claude-sonnet-4-5'],
|
||||
['claude-4.5-sonnet-thinking', 'claude-sonnet-4-5'],
|
||||
['claude-4.6-sonnet', 'claude-sonnet-4-6'],
|
||||
|
|
@ -558,14 +558,6 @@ describe('Cursor model variants resolve to pricing', () => {
|
|||
expect(costs!.outputCostPerToken).toBe(expected!.outputCostPerToken)
|
||||
})
|
||||
}
|
||||
|
||||
// Regression for #912: Cursor's unversioned `claude-4-sonnet-thinking`
|
||||
// slug is the thinking variant of Sonnet 4, not Sonnet 4.5. The two models
|
||||
// currently share a price, so the display name pins the canonical identity
|
||||
// independently of today's pricing coincidence.
|
||||
it('keeps claude-4-sonnet-thinking in the Sonnet 4 model family', () => {
|
||||
expect(getShortModelName('claude-4-sonnet-thinking')).toBe('Sonnet 4')
|
||||
})
|
||||
})
|
||||
|
||||
describe('Cursor house model pricing', () => {
|
||||
|
|
|
|||
|
|
@ -1,5 +1,4 @@
|
|||
import { describe, it, expect, afterAll, afterEach, beforeEach, vi } from 'vitest'
|
||||
import { Writable } from 'node:stream'
|
||||
import { describe, it, expect, afterAll, beforeEach, vi } from 'vitest'
|
||||
import { mkdtempSync, rmSync, mkdirSync, writeFileSync, utimesSync } from 'fs'
|
||||
import { tmpdir } from 'os'
|
||||
import { join } from 'path'
|
||||
|
|
@ -30,8 +29,6 @@ import {
|
|||
estimateContextBudget,
|
||||
discoverProjectCwd,
|
||||
} from '../src/context-budget.js'
|
||||
import type { ProjectSummary } from '../src/types.js'
|
||||
import { runOptimizeApply } from '../src/act/optimize-apply.js'
|
||||
|
||||
// ============================================================================
|
||||
// Helpers for filesystem fixtures
|
||||
|
|
@ -374,94 +371,6 @@ describe('scanAndDetect', () => {
|
|||
expect(result.healthGrade).toBe('A')
|
||||
expect(result.costRate).toBe(0)
|
||||
})
|
||||
|
||||
// The session scan only ever reads Claude Code transcripts, so under a
|
||||
// non-Claude --provider it used to report Claude-derived findings beside a
|
||||
// header scoped to the other provider - e.g. `optimize --provider codex`
|
||||
// printing a read/edit ratio counted from Claude sessions.
|
||||
describe('provider scoping', () => {
|
||||
// These fixtures live in the shared fake home, so they have to come back
|
||||
// out: later suites in this file assert on an otherwise empty ~/.claude.
|
||||
const CLAUDE_DIR = join(FAKE_HOME_FOR_MOCK, '.claude')
|
||||
afterEach(() => {
|
||||
for (const sub of ['projects', 'skills']) {
|
||||
rmSync(join(CLAUDE_DIR, sub), { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
function claudeSessionWithEditHeavyTurns(): void {
|
||||
const projectDir = join(CLAUDE_DIR, 'projects', 'provider-scope')
|
||||
mkdirSync(projectDir, { recursive: true })
|
||||
const now = new Date().toISOString()
|
||||
const entry = (name: string, file: string) => JSON.stringify({
|
||||
type: 'assistant', timestamp: now,
|
||||
message: { content: [{ type: 'tool_use', name, input: { file_path: file } }] },
|
||||
})
|
||||
const lines = [entry('Read', '/src/a.ts')]
|
||||
for (let i = 0; i < 12; i++) lines.push(entry('Edit', `/src/f${i}.ts`))
|
||||
writeFileSync(join(projectDir, 'session.jsonl'), lines.join('\n'))
|
||||
}
|
||||
|
||||
// scanAndDetect memoises on (provider, range, project fingerprint) for 60s,
|
||||
// and the cache is module-level, so tests that differ only in what is on
|
||||
// disk would serve each other's results. `seed` moves the fingerprint so
|
||||
// each case scans for real.
|
||||
function projectFixture(seed: number): ProjectSummary {
|
||||
return {
|
||||
project: 'provider-scope',
|
||||
projectPath: '/tmp/provider-scope',
|
||||
sessions: [],
|
||||
totalCostUSD: 1,
|
||||
totalApiCalls: 13 + seed,
|
||||
} as unknown as ProjectSummary
|
||||
}
|
||||
|
||||
it('reports transcript-derived findings when scoped to claude', async () => {
|
||||
claudeSessionWithEditHeavyTurns()
|
||||
const result = await scanAndDetect([projectFixture(1)], undefined, 'claude')
|
||||
expect(result.findings.map(f => f.id)).toContain('read-edit-ratio')
|
||||
})
|
||||
|
||||
it('omits transcript-derived findings when scoped to another provider', async () => {
|
||||
claudeSessionWithEditHeavyTurns()
|
||||
mkdirSync(join(CLAUDE_DIR, 'skills', 'never-invoked'), { recursive: true })
|
||||
writeFileSync(join(CLAUDE_DIR, 'skills', 'never-invoked', 'SKILL.md'), '# skill\n')
|
||||
|
||||
const result = await scanAndDetect([projectFixture(2)], undefined, 'codex')
|
||||
const ids = result.findings.map(f => f.id)
|
||||
|
||||
expect(ids).not.toContain('read-edit-ratio')
|
||||
// An unmeasured skill must not be reported as an unused one: the scan
|
||||
// returns nothing under this filter, which is not evidence of disuse.
|
||||
expect(ids).not.toContain('unused-skills')
|
||||
})
|
||||
|
||||
// The apply path reaches scanAndDetect through its own entry point, so it
|
||||
// needs its own guard: `unused-skills` is appliable, and its plan moves
|
||||
// directories out of ~/.claude/skills. Reporting a Codex-labelled finding
|
||||
// is a wrong number; offering to archive every skill off one is a wrong
|
||||
// number with side effects.
|
||||
async function applyDryRun(provider: string): Promise<string> {
|
||||
const chunks: string[] = []
|
||||
const output = new Writable({ write(c, _e, cb) { chunks.push(String(c)); cb() } })
|
||||
const errorOutput = new Writable({ write(_c, _e, cb) { cb() } })
|
||||
await runOptimizeApply([projectFixture(3)], undefined, { provider, dryRun: true, output, errorOutput })
|
||||
return chunks.join('')
|
||||
}
|
||||
|
||||
it('plans no applies from Claude findings when scoped to another provider', async () => {
|
||||
claudeSessionWithEditHeavyTurns()
|
||||
mkdirSync(join(CLAUDE_DIR, 'skills', 'never-invoked'), { recursive: true })
|
||||
writeFileSync(join(CLAUDE_DIR, 'skills', 'never-invoked', 'SKILL.md'), '# skill\n')
|
||||
|
||||
const codex = await applyDryRun('codex')
|
||||
expect(codex).toContain('No appliable config-class fixes')
|
||||
expect(codex).not.toContain('never-invoked')
|
||||
|
||||
const claude = await applyDryRun('claude')
|
||||
expect(claude).toContain('never-invoked')
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
// ============================================================================
|
||||
|
|
|
|||
|
|
@ -52,6 +52,7 @@ function projectWithSessions(costs: number[], project = 'app'): ProjectSummary {
|
|||
totalCostUSD: cost,
|
||||
totalInputTokens: tokens,
|
||||
totalOutputTokens: tokens,
|
||||
totalReasoningTokens: 0,
|
||||
totalCacheReadTokens: 0,
|
||||
totalCacheWriteTokens: 0,
|
||||
apiCalls: 1,
|
||||
|
|
@ -105,6 +106,7 @@ function contextSession(
|
|||
totalCostUSD: 1,
|
||||
totalInputTokens: 0,
|
||||
totalOutputTokens: 0,
|
||||
totalReasoningTokens: 0,
|
||||
totalCacheReadTokens: 0,
|
||||
totalCacheWriteTokens: 0,
|
||||
apiCalls: 1,
|
||||
|
|
@ -365,6 +367,18 @@ describe('detectContextBloat', () => {
|
|||
expect(detectContextBloat([project])).toBeNull()
|
||||
})
|
||||
|
||||
it('counts reasoning with output when measuring context pressure', () => {
|
||||
const project = projectWithContextSessions([
|
||||
contextSession(0, {
|
||||
totalInputTokens: 100_000,
|
||||
totalOutputTokens: 3_500,
|
||||
totalReasoningTokens: 2_000,
|
||||
}),
|
||||
])
|
||||
|
||||
expect(detectContextBloat([project])).toBeNull()
|
||||
})
|
||||
|
||||
it('discounts cache reads when estimating context pressure', () => {
|
||||
const project = projectWithContextSessions([
|
||||
contextSession(0, {
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ import { join } from 'path'
|
|||
import { tmpdir } from 'os'
|
||||
|
||||
import { createGrokProvider } from '../../src/providers/grok.js'
|
||||
import { calculateCost } from '../../src/models.js'
|
||||
import type { ParsedProviderCall } from '../../src/providers/types.js'
|
||||
|
||||
let tmpDir: string
|
||||
|
|
@ -24,6 +25,7 @@ async function writeSession(opts: {
|
|||
cwd?: string
|
||||
model?: string
|
||||
turns?: Array<{ promptId: string; totals: number[] }>
|
||||
completedTurns?: Array<{ promptId?: string; usage: unknown }>
|
||||
toolCalls?: Array<{ title: string; rawInput: Record<string, unknown> }>
|
||||
toolsUsed?: string[]
|
||||
} = {}) {
|
||||
|
|
@ -72,6 +74,21 @@ async function writeSession(opts: {
|
|||
}))
|
||||
}
|
||||
}
|
||||
for (const completed of opts.completedTurns ?? []) {
|
||||
lines.push(JSON.stringify({
|
||||
timestamp: 1786724773,
|
||||
method: '_x.ai/session/update',
|
||||
params: {
|
||||
sessionId: uuid,
|
||||
update: {
|
||||
sessionUpdate: 'turn_completed',
|
||||
...(completed.promptId === undefined ? {} : { prompt_id: completed.promptId }),
|
||||
usage: completed.usage,
|
||||
},
|
||||
_meta: { eventId: 'event-1', agentTimestampMs: 1786724773589 },
|
||||
},
|
||||
}))
|
||||
}
|
||||
for (const tc of opts.toolCalls ?? [
|
||||
{ title: 'read_file', rawInput: { target_directory: '.' } },
|
||||
{ title: 'grep', rawInput: { pattern: 'x' } },
|
||||
|
|
@ -89,6 +106,47 @@ async function writeSession(opts: {
|
|||
return { dir, uuid }
|
||||
}
|
||||
|
||||
function authoritativeUsage(opts: {
|
||||
input?: number
|
||||
output?: number
|
||||
cacheRead?: number
|
||||
cacheCreation?: number
|
||||
reasoning?: number
|
||||
model?: string
|
||||
modelUsage?: Record<string, Record<string, unknown>>
|
||||
} = {}): Record<string, unknown> {
|
||||
const input = opts.input ?? 1000
|
||||
const output = opts.output ?? 100
|
||||
const cacheRead = opts.cacheRead ?? 0
|
||||
const cacheCreation = opts.cacheCreation ?? 0
|
||||
const reasoning = opts.reasoning ?? 0
|
||||
const model = opts.model ?? 'grok-4.6-build'
|
||||
const singleModelUsage = {
|
||||
inputTokens: input,
|
||||
outputTokens: output,
|
||||
totalTokens: input + output,
|
||||
cachedReadTokens: cacheRead,
|
||||
cacheCreationTokens: cacheCreation,
|
||||
reasoningTokens: reasoning,
|
||||
modelCalls: 1,
|
||||
apiDurationMs: 1000,
|
||||
costUsdTicks: 125117780000,
|
||||
}
|
||||
return {
|
||||
inputTokens: input,
|
||||
outputTokens: output,
|
||||
totalTokens: input + output,
|
||||
cachedReadTokens: cacheRead,
|
||||
cacheCreationTokens: cacheCreation,
|
||||
reasoningTokens: reasoning,
|
||||
modelCalls: 1,
|
||||
apiDurationMs: 1000,
|
||||
costUsdTicks: 125117780000,
|
||||
modelUsage: opts.modelUsage ?? { [model]: singleModelUsage },
|
||||
numTurns: 1,
|
||||
}
|
||||
}
|
||||
|
||||
describe('grok provider - discovery', () => {
|
||||
it('discovers each session dir and derives project from cwd', async () => {
|
||||
await writeSession({ cwd: '/Users/test/myproject' })
|
||||
|
|
@ -123,7 +181,7 @@ describe('grok provider - parsing', () => {
|
|||
return calls
|
||||
}
|
||||
|
||||
it('emits one estimated call per session from the totalTokens curve', async () => {
|
||||
it('emits one estimated call per session from the totalTokens fallback curve', async () => {
|
||||
await writeSession()
|
||||
const calls = await parse()
|
||||
expect(calls).toHaveLength(1)
|
||||
|
|
@ -144,6 +202,245 @@ describe('grok provider - parsing', () => {
|
|||
expect(call.deduplicationKey).toContain('grok:')
|
||||
})
|
||||
|
||||
it('uses one turn_completed usage record as authoritative and splits cache subsets from input', async () => {
|
||||
await writeSession({
|
||||
turns: [],
|
||||
completedTurns: [{
|
||||
promptId: 'real-prompt-1',
|
||||
usage: authoritativeUsage({
|
||||
input: 12851663,
|
||||
output: 36633,
|
||||
cacheRead: 12092032,
|
||||
cacheCreation: 0,
|
||||
reasoning: 29077,
|
||||
}),
|
||||
}],
|
||||
})
|
||||
|
||||
const calls = await parse()
|
||||
expect(calls).toHaveLength(1)
|
||||
const call = calls[0]!
|
||||
expect(call.model).toBe('grok-build')
|
||||
expect(call.inputTokens).toBe(759631) // 12851663 - 12092032 - 0
|
||||
expect(call.cacheReadInputTokens).toBe(12092032)
|
||||
expect(call.cacheCreationInputTokens).toBe(0)
|
||||
// Grok reports reasoning inside outputTokens; the repo contract wants them
|
||||
// split, so output is emitted exclusive of reasoning and the two sum back
|
||||
// to the 36633 the record reported.
|
||||
expect(call.outputTokens).toBe(7556) // 36633 - 29077
|
||||
expect(call.reasoningTokens).toBe(29077)
|
||||
expect(call.outputTokens + call.reasoningTokens).toBe(36633)
|
||||
expect(call.costIsEstimated).toBe(false)
|
||||
expect(call.costUSD).toBe(calculateCost('grok-build', 759631, 36633, 0, 12092032, 0))
|
||||
})
|
||||
|
||||
it('sums distinct turn_completed prompt ids exactly once each', async () => {
|
||||
await writeSession({
|
||||
turns: [],
|
||||
completedTurns: [
|
||||
{ promptId: 'p1', usage: authoritativeUsage({ input: 1000, output: 100, cacheRead: 600, cacheCreation: 50, reasoning: 10 }) },
|
||||
{ promptId: 'p2', usage: authoritativeUsage({ input: 2000, output: 200, cacheRead: 1000, cacheCreation: 100, reasoning: 20 }) },
|
||||
],
|
||||
})
|
||||
|
||||
const [call] = await parse()
|
||||
expect(call).toMatchObject({
|
||||
inputTokens: 1250,
|
||||
cacheReadInputTokens: 1600,
|
||||
cacheCreationInputTokens: 150,
|
||||
outputTokens: 270, // 300 reported - 30 reasoning
|
||||
reasoningTokens: 30,
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps one authoritative call and uses top-level totals for multi-model usage', async () => {
|
||||
await writeSession({
|
||||
turns: [],
|
||||
completedTurns: [{
|
||||
promptId: 'multi-model',
|
||||
usage: authoritativeUsage({
|
||||
input: 3000,
|
||||
output: 300,
|
||||
cacheRead: 600,
|
||||
cacheCreation: 100,
|
||||
reasoning: 30,
|
||||
modelUsage: {
|
||||
'grok-build-0.1': {
|
||||
inputTokens: 1000,
|
||||
outputTokens: 100,
|
||||
cachedReadTokens: 100,
|
||||
cacheCreationTokens: 0,
|
||||
reasoningTokens: 10,
|
||||
},
|
||||
'grok-latest': {
|
||||
inputTokens: 2000,
|
||||
outputTokens: 200,
|
||||
cachedReadTokens: 500,
|
||||
cacheCreationTokens: 100,
|
||||
reasoningTokens: 20,
|
||||
},
|
||||
},
|
||||
}),
|
||||
}],
|
||||
})
|
||||
|
||||
const calls = await parse()
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]).toMatchObject({
|
||||
model: 'grok-build-0.1',
|
||||
inputTokens: 2300,
|
||||
outputTokens: 270,
|
||||
reasoningTokens: 30,
|
||||
costUSD: calculateCost('grok-build-0.1', 2300, 300, 100, 600, 0),
|
||||
})
|
||||
expect(calls[0]!.outputTokens + calls[0]!.reasoningTokens).toBe(300)
|
||||
expect(calls[0]!.turnId).toBeUndefined()
|
||||
})
|
||||
|
||||
it('uses the last turn_completed record for a duplicate prompt id', async () => {
|
||||
await writeSession({
|
||||
turns: [],
|
||||
completedTurns: [
|
||||
{ promptId: 'same-prompt', usage: authoritativeUsage({ input: 500, output: 50, cacheRead: 100, reasoning: 5 }) },
|
||||
{ promptId: 'same-prompt', usage: authoritativeUsage({ input: 800, output: 80, cacheRead: 200, cacheCreation: 25, reasoning: 8 }) },
|
||||
],
|
||||
})
|
||||
|
||||
const [call] = await parse()
|
||||
expect(call).toMatchObject({
|
||||
inputTokens: 575,
|
||||
cacheReadInputTokens: 200,
|
||||
cacheCreationInputTokens: 25,
|
||||
outputTokens: 72, // 80 reported - 8 reasoning
|
||||
reasoningTokens: 8,
|
||||
})
|
||||
})
|
||||
|
||||
it('uses unique fallback keys when completed records omit prompt_id', async () => {
|
||||
await writeSession({
|
||||
turns: [],
|
||||
completedTurns: [
|
||||
{ usage: authoritativeUsage({ input: 100, output: 10 }) },
|
||||
{ usage: authoritativeUsage({ input: 200, output: 20 }) },
|
||||
],
|
||||
})
|
||||
|
||||
const [call] = await parse()
|
||||
expect(call).toMatchObject({ inputTokens: 300, outputTokens: 30, reasoningTokens: 0 })
|
||||
})
|
||||
|
||||
it('ignores a still-streaming turn but marks mixed coverage estimated', async () => {
|
||||
await writeSession({
|
||||
turns: [{ promptId: 'still-streaming', totals: [10000, 15000] }],
|
||||
completedTurns: [{ promptId: 'completed', usage: authoritativeUsage({ input: 900, output: 90, cacheRead: 300, reasoning: 20 }) }],
|
||||
})
|
||||
|
||||
const [call] = await parse()
|
||||
expect(call).toMatchObject({
|
||||
inputTokens: 600,
|
||||
cacheReadInputTokens: 300,
|
||||
outputTokens: 70, // 90 reported - 20 reasoning
|
||||
reasoningTokens: 20,
|
||||
costIsEstimated: true,
|
||||
})
|
||||
})
|
||||
|
||||
it('treats malformed authoritative fields as absent without throwing or corrupting totals', async () => {
|
||||
await writeSession({
|
||||
turns: [],
|
||||
completedTurns: [{
|
||||
promptId: 'malformed',
|
||||
usage: {
|
||||
inputTokens: -1,
|
||||
outputTokens: 4,
|
||||
totalTokens: 'not-a-number',
|
||||
cachedReadTokens: Number.NaN,
|
||||
cacheCreationTokens: 'not-a-number',
|
||||
reasoningTokens: -2,
|
||||
modelUsage: {},
|
||||
},
|
||||
}],
|
||||
})
|
||||
|
||||
const [call] = await parse()
|
||||
expect(call).toBeDefined()
|
||||
expect(call!.inputTokens).toBe(0)
|
||||
expect(call!.outputTokens).toBe(4)
|
||||
expect(call!.cacheReadInputTokens).toBe(0)
|
||||
expect(call!.cacheCreationInputTokens).toBe(0)
|
||||
expect(call!.reasoningTokens).toBe(0)
|
||||
expect(Number.isFinite(call!.costUSD)).toBe(true)
|
||||
expect(call!.costUSD).toBeGreaterThanOrEqual(0)
|
||||
})
|
||||
|
||||
it('keeps the heuristic when a completed record reports all-zero usage', async () => {
|
||||
await writeSession({
|
||||
turns: [
|
||||
{ promptId: 'streaming-1', totals: [20000, 25000] },
|
||||
{ promptId: 'streaming-2', totals: [30000, 35000] },
|
||||
],
|
||||
completedTurns: [{
|
||||
promptId: 'zero-usage',
|
||||
usage: authoritativeUsage({ input: 0, output: 0, cacheRead: 0, cacheCreation: 0, reasoning: 0 }),
|
||||
}],
|
||||
})
|
||||
|
||||
const [call] = await parse()
|
||||
expect(call).toBeDefined()
|
||||
expect(call!.inputTokens).toBe(35000)
|
||||
expect(call!.cacheReadInputTokens).toBe(15000)
|
||||
expect(call!.outputTokens).toBe(10000)
|
||||
expect(call!.costIsEstimated).toBe(true)
|
||||
})
|
||||
|
||||
it('clamps cache-exclusive input per completed record before summing', async () => {
|
||||
await writeSession({
|
||||
turns: [],
|
||||
completedTurns: [
|
||||
{ promptId: 'inconsistent', usage: authoritativeUsage({ input: 100, output: 10, cacheRead: 80, cacheCreation: 50 }) },
|
||||
{ promptId: 'consistent', usage: authoritativeUsage({ input: 100, output: 20 }) },
|
||||
],
|
||||
})
|
||||
|
||||
const [call] = await parse()
|
||||
expect(call).toMatchObject({
|
||||
inputTokens: 100,
|
||||
cacheReadInputTokens: 80,
|
||||
cacheCreationInputTokens: 50,
|
||||
outputTokens: 30,
|
||||
})
|
||||
})
|
||||
|
||||
it('does not add reasoning tokens on top of provider-reported output for cost', async () => {
|
||||
await writeSession({
|
||||
turns: [],
|
||||
model: 'grok-build',
|
||||
completedTurns: [{
|
||||
promptId: 'reasoning-subset',
|
||||
usage: authoritativeUsage({
|
||||
input: 1000,
|
||||
output: 200,
|
||||
cacheRead: 500,
|
||||
cacheCreation: 100,
|
||||
reasoning: 150,
|
||||
model: 'grok-build',
|
||||
}),
|
||||
}],
|
||||
})
|
||||
|
||||
const [call] = await parse()
|
||||
expect(call).toBeDefined()
|
||||
expect(call!.inputTokens).toBe(400)
|
||||
// Output is emitted exclusive of reasoning, and the two sum to the 200 the
|
||||
// record reported. The cost prices that full 200 once - the downstream
|
||||
// `outputTokens + reasoningTokens` recompute lands on the same number.
|
||||
expect(call!.outputTokens).toBe(50) // 200 - 150
|
||||
expect(call!.reasoningTokens).toBe(150)
|
||||
expect(call!.outputTokens + call!.reasoningTokens).toBe(200)
|
||||
expect(call!.costUSD).toBe(calculateCost('grok-build', 400, 200, 100, 500, 0))
|
||||
expect(call!.costUSD).not.toBe(calculateCost('grok-build', 400, 350, 100, 500, 0))
|
||||
})
|
||||
|
||||
it('skips a session with no token growth', async () => {
|
||||
await writeSession({ turns: [{ promptId: 'p1', totals: [0, 0] }] })
|
||||
expect(await parse()).toHaveLength(0)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue