From 4f79036a2269bb43f95f736ca8c44bc60b0cc9d6 Mon Sep 17 00:00:00 2001 From: Shaojin Wen Date: Wed, 5 Aug 2026 10:18:06 +0800 Subject: [PATCH] feat(review): a cost ledger from the records already on disk (#8471) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(review): a cost ledger from the records already on disk "0.21.3 was fine, 0.21.4 got slow" was settled only by replaying a whole review under a telemetry exporter and hand-aggregating the output — hours of forensics to find a repair round that had silently doubled a run (measured: a +93/-48 PR at high effort cost 523 model calls and 37.8M input tokens, 9.7M of them redelivering prompts the agents had already acted on). The usage data was on disk the whole time: every chat and subagent transcript event carries usageMetadata. qwen review cost-ledger --plan aggregates those records — the same files check-coverage trusts for delivery, found via the same environment-exported location, floored at the plan's mtime so a review started an hour into a session does not bill that hour — into per-stream totals: the main loop and each agent, with input / cached / output / thinking counts and wall time. Step 8 pastes the printed block into the saved report, so the next slowness question is a diff of two archives instead of an excavation. Informational by construction: an incomputable ledger prints why and exits 0. Validated against the measured run above — 521 calls, 37.7M input (93% cached), 849k output, 107 min — matching the telemetry-side aggregation, minus the two side-query calls that are not the review's. * test: register cost-ledger in the subcommand registry and its demand message * fix(review): honest cost-ledger output and a safe archive write (#8471) Address the review of the cost ledger: report output tokens once (thinking is a subset of candidates, not a sibling), keep the --out write inside the exit-0 contract and mkdir its parent, name a missing plan as the plan, read each transcript once, compare timestamps as instants, fold relaunched agents into marked rows, and archive the full ledger next to the Step 8 report. * fix(review): cost ledger — honest failures, validated plan, shared records (#8471) Address the second review round: a missing or faulted chat transcript now says "cost-ledger unavailable" instead of rendering agents-only totals as the whole cost (the plan proves the main loop ran), and a subagent dir that fails listing with anything but ENOENT does the same. The --plan file is validated as a plan report before its mtime alone sets the billing window. Output derives from totalTokenCount − promptTokenCount when present, correct under both usage conventions. Chunk agents label "chunk N" via the shared CHUNK_RE instead of the malformed "agent chunk N of M"; the transcript listing is one helper shared with the coverage gate; glued JSONL records are recovered via parseLineTolerant; totals reuse the rows' accumulator; folded (×N) rows rank by combined total; stale agent files are skipped by mtime without being opened. Rendered block gains pluralization, "agent runs: N", and a B tier; SKILL.md states the ledger's bounded window. * fix(review): close the cost-ledger audit — refusals, labels, pinned math Address the remaining review threads on the cost ledger: - Refuse agents-only totals when the chat file exists but holds no above-floor records: a degraded recorder leaves the file present and empty while agents run — the same infrastructure fact as an unreadable transcript, and exactly the output the missing-file refusal exists to prevent. - Read the agent label from the first user record, not a raw 64KB head slice: a fork's agent_bootstrap record precedes the launch prompt, quotes other agents' identity lines, and can outgrow any fixed window. - Distinguish parallel invariant agents by their owned file, so per-file runs stop folding into a phantom (xN) relaunch row. - Coerce negative provider counts to zero: the agent path records usage uncoerced, and summed negatives rendered >100% cached shares. - Accept degraded diff-less Step 1 reports: validate diffLines + chunks, the pair every plan report carries, instead of check-coverage's stricter contract that refused them. - Pin every branch the second round proved unobservable: the exit code on all handler paths, total - prompt under both usage conventions, per-agent fault tolerance, the wall-minutes conversion, array-shaped usage, the mtime pre-filter and the event-level floor, human() rounding, per-condition plan validation, sort order against a lexical readdir, the zero-event skip, the --out per-stream archive contract, error messages naming their paths, truncation membership and folded-run counting, and the assistant-type filter. Every new assertion was mutation-probed: each mutant the review named now turns the suite red. * review: pipeline stages keep their own ledger rows The (×N) fold keyed on the label alone, and three legitimate multi-launch shapes shared one: a reverse-audit chunk auditor is launched with the same 'chunk N of M' identity as the Step 3B territory finder (five audit rounds folded into the finder's row — one agent where six pipeline stages ran), and repeat rounds of the findings roles carry their round OUTSIDE the backticks (every round folded as a phantom relaunch). labelOf now reads the stage from the audit brief's record key in the launch (audit chunk N (round K)) and the round from the identity LINE — never the whole launch, whose folded findings can quote a budget disclosure's own '(round N)' — so rounds are rows and only true relaunches and same-round verify shards fold. The (×N) comment now says what the marker means: N runs under one label. * fix(cli): annotate cost-ledger test helper to restore strict build (#8471) * fix(cli): anchor cost-ledger labels and harden broken-usage defenses (#8471) --------- Co-authored-by: verify Co-authored-by: qwen-code-dev-bot --- packages/cli/src/commands/review.test.ts | 1 + packages/cli/src/commands/review.ts | 4 +- .../src/commands/review/cost-ledger.test.ts | 1314 +++++++++++++++++ .../cli/src/commands/review/cost-ledger.ts | 555 +++++++ .../cli/src/commands/review/lib/coverage.ts | 2 +- .../commands/review/lib/transcripts.test.ts | 33 + .../src/commands/review/lib/transcripts.ts | 43 +- packages/cli/src/utils/stdioHelpers.test.ts | 24 +- packages/cli/src/utils/stdioHelpers.ts | 15 + .../core/src/skills/bundled/review/SKILL.md | 1 + 10 files changed, 1982 insertions(+), 10 deletions(-) create mode 100644 packages/cli/src/commands/review/cost-ledger.test.ts create mode 100644 packages/cli/src/commands/review/cost-ledger.ts diff --git a/packages/cli/src/commands/review.test.ts b/packages/cli/src/commands/review.test.ts index 8b560f5047..f7129ac177 100644 --- a/packages/cli/src/commands/review.test.ts +++ b/packages/cli/src/commands/review.test.ts @@ -57,6 +57,7 @@ describe('reviewCommand', () => { 'script-lint', 'resolve-anchors', 'check-coverage', + 'cost-ledger', 'presubmit', 'test-efficacy', 'test-plan', diff --git a/packages/cli/src/commands/review.ts b/packages/cli/src/commands/review.ts index cb70d08d02..00331ec239 100644 --- a/packages/cli/src/commands/review.ts +++ b/packages/cli/src/commands/review.ts @@ -34,6 +34,7 @@ import { submitCommand } from './review/submit.js'; import { testEfficacyCommand } from './review/test-efficacy.js'; import { testPlanCommand } from './review/test-plan.js'; import { cleanupCommand } from './review/cleanup.js'; +import { costLedgerCommand } from './review/cost-ledger.js'; import { runCommand } from './review/run.js'; import { saveArtifactCommand } from './review/save-artifact.js'; @@ -61,6 +62,7 @@ export const reviewCommand: CommandModule = { .command(scriptLintCommand) .command(resolveAnchorsCommand) .command(checkCoverageCommand) + .command(costLedgerCommand) .command(presubmitCommand) .command(testEfficacyCommand) .command(testPlanCommand) @@ -72,7 +74,7 @@ export const reviewCommand: CommandModule = { .command(cleanupCommand) .demandCommand( 1, - 'Specify a subcommand: run, parse-args, fetch-pr, capture-local, plan-diff, pr-context, comment-status, load-rules, agent-prompt, build-test, base-tree, test-delta, drive, mock-provider, extract-step, script-lint, resolve-anchors, check-coverage, presubmit, test-efficacy, test-plan, findings, publish-assets, compose-review, save-artifact, submit, or cleanup.', + 'Specify a subcommand: run, parse-args, fetch-pr, capture-local, plan-diff, pr-context, comment-status, load-rules, agent-prompt, build-test, base-tree, test-delta, drive, mock-provider, extract-step, script-lint, resolve-anchors, check-coverage, cost-ledger, presubmit, test-efficacy, test-plan, findings, publish-assets, compose-review, save-artifact, submit, or cleanup.', ) .version(false), handler: () => { diff --git a/packages/cli/src/commands/review/cost-ledger.test.ts b/packages/cli/src/commands/review/cost-ledger.test.ts new file mode 100644 index 0000000000..10fd8bc407 --- /dev/null +++ b/packages/cli/src/commands/review/cost-ledger.test.ts @@ -0,0 +1,1314 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, it, expect, afterEach, vi } from 'vitest'; +import { + existsSync, + mkdtempSync, + mkdirSync, + readFileSync, + rmSync, + utimesSync, + writeFileSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { + computeLedger, + renderLedger, + costLedgerCommand, +} from './cost-ledger.js'; + +const SESSION = 'S-ledger'; + +function event( + timestamp: string, + usage: { + input?: number; + cached?: number; + output?: number; + thoughts?: number; + }, + extra: Record = {}, +): string { + return JSON.stringify({ + type: 'assistant', + timestamp, + usageMetadata: { + promptTokenCount: usage.input ?? 0, + cachedContentTokenCount: usage.cached ?? 0, + candidatesTokenCount: usage.output ?? 0, + thoughtsTokenCount: usage.thoughts ?? 0, + // This helper models ONE usage convention: total = prompt + candidates, + // with thinking a subset of candidates. The disjoint convention + // (thoughts a sibling of candidates, total = prompt + candidates + + // thoughts) is real too — the both-conventions derivation test writes + // those records raw, where the two formulas actually diverge. + totalTokenCount: (usage.input ?? 0) + (usage.output ?? 0), + }, + ...extra, + }); +} + +function userRecord(text: string): string { + return JSON.stringify({ + type: 'user', + timestamp: '2026-08-03T10:06:00Z', + message: { role: 'user', parts: [{ text }] }, + }); +} + +describe('cost-ledger — the spend, from the records already on disk', () => { + const dirs: string[] = []; + afterEach(() => { + for (const d of dirs.splice(0)) rmSync(d, { recursive: true, force: true }); + }); + + function fixture(): { + plan: string; + env: NodeJS.ProcessEnv; + project: string; + } { + const project = mkdtempSync(join(tmpdir(), 'ledger-')); + dirs.push(project); + mkdirSync(join(project, 'chats'), { recursive: true }); + mkdirSync(join(project, 'subagents', SESSION), { recursive: true }); + // Recording on, no above-floor records yet. Tests that need calls + // overwrite this; tests for a missing chat file remove it; tests for + // the empty-window refusal use it as-is. + writeFileSync(join(project, 'chats', `${SESSION}.jsonl`), ''); + const plan = join(project, 'plan.json'); + // A real plan, shape-wise: the ledger validates it before trusting its + // mtime as the billing floor. + writeFileSync( + plan, + JSON.stringify({ + diffPathAbsolute: join(project, 'diff.txt'), + diffLines: 10, + chunks: [{ id: 1, startLine: 1, endLine: 10 }], + }), + ); + // The review "started" at 10:00; the plan's mtime is the billing floor. + const start = new Date('2026-08-03T10:00:00Z'); + utimesSync(plan, start, start); + return { + plan, + project, + env: { + QWEN_CODE_PROJECT_DIR: project, + QWEN_CODE_SESSION_ID: SESSION, + } as NodeJS.ProcessEnv, + }; + } + + function chatFile(project: string): string { + return join(project, 'chats', `${SESSION}.jsonl`); + } + + /** + * One real main-loop call. Agents-only records with no above-floor + * main-loop call are refused as an unreadable chat transcript, so every + * agent-focused fixture writes the main call its agents' launch implies. + */ + function writeMainCall(project: string): void { + writeFileSync( + chatFile(project), + event('2026-08-03T10:01:00Z', { input: 500, output: 50 }), + ); + } + + it('aggregates the main loop and each agent, newest records only', () => { + const { plan, env, project } = fixture(); + writeFileSync( + chatFile(project), + [ + // Before the plan: the session's earlier, unrelated conversation. + event('2026-08-03T09:00:00Z', { input: 500_000, output: 9_000 }), + event('2026-08-03T10:01:00Z', { + input: 100_000, + cached: 90_000, + output: 1_000, + thoughts: 200, + }), + event('2026-08-03T10:05:00Z', { + input: 110_000, + cached: 105_000, + output: 2_000, + }), + // Non-assistant and usage-less lines are not model calls. + JSON.stringify({ type: 'user', timestamp: '2026-08-03T10:02:00Z' }), + JSON.stringify({ + type: 'assistant', + timestamp: '2026-08-03T10:03:00Z', + }), + // Nor is a non-assistant record CARRYING usage — a `}{`-glued + // corruption fragment can leave a stray usageMetadata attached to + // the wrong half. The type filter, not the usage check, excludes it. + JSON.stringify({ + type: 'user', + timestamp: '2026-08-03T10:04:00Z', + usageMetadata: { + promptTokenCount: 7_777, + candidatesTokenCount: 7, + totalTokenCount: 7_784, + }, + }), + ].join('\n'), + ); + writeFileSync( + join(project, 'subagents', SESSION, 'agent-general-purpose-a1.jsonl'), + [ + userRecord('You are review agent `2` — Agent 2: Security.'), + event('2026-08-03T10:06:30Z', { + input: 33_000, + output: 400, + thoughts: 100, + }), + event('2026-08-03T10:08:00Z', { + input: 40_000, + cached: 33_000, + output: 600, + }), + ].join('\n'), + ); + + const ledger = computeLedger(plan, env); + + expect(ledger.totals.calls).toBe(4); + expect(ledger.totals.inputTokens).toBe(283_000); + expect(ledger.totals.cachedTokens).toBe(228_000); + expect(ledger.totals.outputTokens).toBe(4_000); + expect(ledger.totals.thoughtsTokens).toBe(300); + // 10:01:00 → 10:08:00. + expect(ledger.totals.wallSeconds).toBe(420); + + expect(ledger.main?.calls).toBe(2); + expect(ledger.main?.inputTokens).toBe(210_000); + + expect(ledger.agents).toHaveLength(1); + expect(ledger.agents[0].label).toBe('agent 2'); + expect(ledger.agents[0].inputTokens).toBe(73_000); + + // The block is archived verbatim, so pin the per-stream values too — not + // just the totals line — and the seconds→minutes wall conversion, whose + // only other non-zero fixture asserts the raw field. + const text = renderLedger(ledger); + expect(text).toContain('7 min wall'); + expect(text).toContain('main loop: 2 calls · 210k in · 3k out'); + expect(text).toContain('agent 2: 2 calls · 73k in · 1k out'); + }); + + it('derives output as total − prompt under both usage conventions', () => { + const { plan, env, project } = fixture(); + writeFileSync( + chatFile(project), + [ + // The disjoint convention: thoughts a sibling of candidates, so + // total = prompt + candidates + thoughts. Output must be + // total − prompt = 500 — bare candidates would drop the thinking. + JSON.stringify({ + type: 'assistant', + timestamp: '2026-08-03T10:01:00Z', + usageMetadata: { + promptTokenCount: 1_000, + candidatesTokenCount: 400, + thoughtsTokenCount: 100, + totalTokenCount: 1_500, + }, + }), + // No total reported at all: candidates is the only number there is, + // and the subset convention keeps it correct. + JSON.stringify({ + type: 'assistant', + timestamp: '2026-08-03T10:02:00Z', + usageMetadata: { + promptTokenCount: 1_000, + candidatesTokenCount: 400, + }, + }), + ].join('\n'), + ); + const ledger = computeLedger(plan, env); + // 500 (total − prompt) + 400 (candidates fallback). + expect(ledger.main?.outputTokens).toBe(900); + expect(ledger.main?.thoughtsTokens).toBe(100); + }); + + it('coerces broken provider counts instead of corrupting totals', () => { + const { plan, env, project } = fixture(); + writeFileSync( + chatFile(project), + [ + event('2026-08-03T10:01:00Z', { + input: 1_000, + cached: 500, + output: 100, + }), + // A broken OpenAI-compat proxy: the agent path records provider + // usage uncoerced, so negative counts reach the ledger. Summed, they + // would render >100% cached shares and negative rows in the archive. + JSON.stringify({ + type: 'assistant', + timestamp: '2026-08-03T10:02:00Z', + usageMetadata: { + promptTokenCount: -5_000, + cachedContentTokenCount: -4_000, + candidatesTokenCount: -1_000, + totalTokenCount: -6_000, + }, + }), + // Positive but INVERTED: cached above prompt passes the ≥ 0 check + // and would render a 500% share. It is clamped to its own prompt, + // so the rendered share and the archived JSON both stay ≤ 100%. + JSON.stringify({ + type: 'assistant', + timestamp: '2026-08-03T10:03:00Z', + usageMetadata: { + promptTokenCount: 100, + cachedContentTokenCount: 5_000, + candidatesTokenCount: 50, + totalTokenCount: 150, + }, + }), + // Mixed-sign: prompt negative, total positive. Deriving output as + // total − coerced-prompt would bill the call's whole total as + // output; it falls back to candidates instead. + JSON.stringify({ + type: 'assistant', + timestamp: '2026-08-03T10:04:00Z', + usageMetadata: { + promptTokenCount: -5_000, + candidatesTokenCount: 100, + totalTokenCount: 6_000, + }, + }), + ].join('\n'), + ); + const ledger = computeLedger(plan, env); + expect(ledger.totals.calls).toBe(4); + // 1_000 + 0 + 100 + 0. + expect(ledger.totals.inputTokens).toBe(1_100); + // 500 + 0 + min(5_000, 100) + 0. + expect(ledger.totals.cachedTokens).toBe(600); + // 100 + 0 + (150 − 100) + 100 (candidates fallback). + expect(ledger.totals.outputTokens).toBe(250); + // The clamp keeps the archived share sane: 600 / 1_100 → 55%, never + // the 500% the raw counts would render. + expect(renderLedger(ledger)).toContain('(55% cached)'); + }); + + it('renders a one-line summary a reader can act on', () => { + const { plan, env, project } = fixture(); + writeFileSync( + chatFile(project), + event('2026-08-03T10:01:00Z', { + input: 1_200_000, + cached: 600_000, + output: 10_000, + thoughts: 5_000, + }), + ); + const text = renderLedger(computeLedger(plan, env)); + expect(text).toContain('1 model call'); + expect(text).toContain('1.2M input (50% cached)'); + // Thinking is a subset of output: the total reports output once, with the + // thinking inside it — never output + thinking. + expect(text).toContain('10k output (5k thinking)'); + expect(text).toContain('main loop: 1 call · 1.2M in · 10k out'); + }); + + it('refuses an empty window even when no agents ran', () => { + const { plan, env } = fixture(); + // The recorder pre-creates the chat file and degrades permanently on a + // failed first append, while a live review always holds at least one + // above-floor main-loop record — the plan itself is a main-loop write. + // A zero window is therefore the degraded-recorder fact, with or + // without agents — never a cheaper "empty review". + expect(() => computeLedger(plan, env)).toThrow( + /no main-loop usage records at or after the plan/, + ); + }); + + it('throws TranscriptsUnavailable through to the caller when the env is bare', () => { + const { plan } = fixture(); + expect(() => computeLedger(plan, {} as NodeJS.ProcessEnv)).toThrow( + /QWEN_CODE_PROJECT_DIR/, + ); + }); + + it('names a missing plan as the plan, not the usage records', () => { + const { env } = fixture(); + expect(() => computeLedger('/nonexistent/plan.json', env)).toThrow( + /could not read the plan report/, + ); + }); + + it('rejects an existing file that is not the plan report', () => { + const { env, project } = fixture(); + // The mtime alone defines the billing window; a wrong-but-existing file + // (the findings JSON, the report) must say so, not silently move the + // floor. + const notPlan = join(project, 'findings.json'); + writeFileSync(notPlan, '{}'); + expect(() => computeLedger(notPlan, env)).toThrow( + /not a review plan report/, + ); + writeFileSync(notPlan, 'not json at all'); + expect(() => computeLedger(notPlan, env)).toThrow( + /not a review plan report/, + ); + // Each validated field independently: a fixture per condition, each + // satisfying the other, so relaxing either one alone turns a test red. + writeFileSync(notPlan, JSON.stringify({ chunks: [{ id: 1 }] })); + expect(() => computeLedger(notPlan, env)).toThrow( + /not a review plan report/, + ); + writeFileSync(notPlan, JSON.stringify({ diffLines: 42 })); + expect(() => computeLedger(notPlan, env)).toThrow( + /not a review plan report/, + ); + }); + + it('accepts a degraded diff-less plan — its mtime is still the floor', () => { + const { plan, env, project } = fixture(); + // fetch-pr writes diffPathAbsolute: null with chunks: [] on two real + // paths (unresolvable merge base, the tiling fallback). That file IS the + // Step 1 plan report, and the ledger needs only its mtime. + writeFileSync( + plan, + JSON.stringify({ diffPathAbsolute: null, diffLines: 0, chunks: [] }), + ); + const start = new Date('2026-08-03T10:00:00Z'); + utimesSync(plan, start, start); + writeMainCall(project); + expect(computeLedger(plan, env).totals.calls).toBe(1); + }); + + it('refuses to render agents-only totals when the chat transcript is missing', () => { + const { plan, env, project } = fixture(); + // Agents ran and left records; the chat file never existed (chat + // recording off). The plan proves the main loop made calls, so the + // agents-only sum is not the review's cost — say the ledger is + // unavailable instead of printing it as the total. + rmSync(chatFile(project)); + writeFileSync( + join(project, 'subagents', SESSION, 'agent-role-1.jsonl'), + [ + userRecord('You are review agent `1` — dimension 1.'), + event('2026-08-03T10:06:30Z', { input: 1_000, output: 100 }), + ].join('\n'), + ); + expect(() => computeLedger(plan, env)).toThrow( + /could not read the chat transcript/, + ); + }); + + it('refuses agents-only totals when the chat file exists but is empty', () => { + const { plan, env, project } = fixture(); + // The recorder pre-creates the chat file; a failed first append degrades + // it permanently, leaving the file present with zero records while + // agents run. That must not slip past the missing-file refusal and + // present agents-only totals as the review's whole cost. + writeFileSync( + join(project, 'subagents', SESSION, 'agent-role-1.jsonl'), + [ + userRecord('You are review agent `1` — dimension 1.'), + event('2026-08-03T10:06:30Z', { input: 1_000, output: 100 }), + ].join('\n'), + ); + expect(() => computeLedger(plan, env)).toThrow( + /could not read the chat transcript/, + ); + // Same refusal when every main-loop record predates the plan: an + // above-floor agent with no above-floor main call is the same state. + writeFileSync( + chatFile(project), + event('2026-08-03T09:00:00Z', { input: 1_000, output: 100 }), + ); + expect(() => computeLedger(plan, env)).toThrow( + /no main-loop usage records at or after the plan/, + ); + }); + + it('loses only the unreadable agent transcript, not the whole ledger', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + // EISDIR on read: a listed transcript that cannot be opened. One corrupt + // agent record must cost that agent's row, never the ledger. + mkdirSync(join(project, 'subagents', SESSION, 'agent-bad.jsonl')); + writeFileSync( + join(project, 'subagents', SESSION, 'agent-role-ok.jsonl'), + [ + userRecord('You are review agent `ok` — dimension.'), + event('2026-08-03T10:06:30Z', { input: 1_000, output: 100 }), + ].join('\n'), + ); + const ledger = computeLedger(plan, env); + expect(ledger.agents).toHaveLength(1); + expect(ledger.agents[0].label).toBe('agent ok'); + expect(ledger.agents[0].inputTokens).toBe(1_000); + }); + + it('ignores the harness sidecar files beside the transcripts', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + writeFileSync( + join(project, 'subagents', SESSION, 'agent-role-1.jsonl'), + [ + userRecord('You are review agent `1` — dimension 1.'), + event('2026-08-03T10:06:30Z', { input: 1_000, output: 100 }), + ].join('\n'), + ); + // The harness writes siblings per agent (agent-transcript.ts): a + // `.meta.json` that carries an `agentId`, and a transient + // `.jsonl.stream`. Both readers of this dir share one filter + // (listAgentTranscriptFiles); admitted here, the meta's `agentId` would + // surface as a phantom row and break (×N) folding. + writeFileSync( + join(project, 'subagents', SESSION, 'agent-role-1.meta.json'), + JSON.stringify({ + agentId: 'role-1', + agentType: 'general-purpose', + description: 'dimension 1', + parentSessionId: SESSION, + parentAgentId: null, + createdAt: '2026-08-03T10:06:00.000Z', + status: 'completed', + }), + ); + writeFileSync( + join(project, 'subagents', SESSION, 'agent-role-1.jsonl.stream'), + 'streaming text, not jsonl records', + ); + const ledger = computeLedger(plan, env); + expect(ledger.agents).toHaveLength(1); + expect(ledger.agents[0].id).toBe('role-1'); + expect(renderLedger(ledger)).not.toContain('(×2)'); + }); + + it('surfaces a fault reading the chat file, not a zero ledger', () => { + const { plan, env, project } = fixture(); + rmSync(chatFile(project), { recursive: true, force: true }); + mkdirSync(chatFile(project)); // EISDIR where the file should be + expect(() => computeLedger(plan, env)).toThrow( + /could not read the chat transcript/, + ); + }); + + it('surfaces an unreadable subagent directory instead of main-loop-only totals', () => { + const { plan, env, project } = fixture(); + writeFileSync( + chatFile(project), + [event('2026-08-03T10:01:00Z', { input: 1_000, output: 100 })].join('\n'), + ); + rmSync(join(project, 'subagents', SESSION), { + recursive: true, + force: true, + }); + // ENOTDIR where the directory should be: not ENOENT, so not "no agents". + writeFileSync(join(project, 'subagents', SESSION), 'in the way'); + expect(() => computeLedger(plan, env)).toThrow( + /could not list the subagent transcripts/, + ); + }); + + it('orders mixed-precision timestamps by time, not by string', () => { + const { plan, env, project } = fixture(); + writeFileSync( + chatFile(project), + [ + // Lexically "…:00.500Z" < "…:00Z" ('.' < 'Z'), but it is the later + // instant. String comparison would swap first and last. + event('2026-08-03T10:01:00Z', { input: 1_000, output: 100 }), + event('2026-08-03T10:01:00.500Z', { input: 1_000, output: 100 }), + ].join('\n'), + ); + const ledger = computeLedger(plan, env); + expect(ledger.main?.firstAt).toBe('2026-08-03T10:01:00Z'); + expect(ledger.main?.lastAt).toBe('2026-08-03T10:01:00.500Z'); + }); + + it('rounds 999.5k up to 1.0M, not 1000k', () => { + const { plan, env, project } = fixture(); + writeFileSync( + chatFile(project), + event('2026-08-03T10:01:00Z', { input: 999_500, output: 100 }), + ); + expect(renderLedger(computeLedger(plan, env))).toContain('1.0M input'); + }); + + it('renders billions with a B tier, not 1500.0M', () => { + const { plan, env, project } = fixture(); + writeFileSync( + chatFile(project), + event('2026-08-03T10:01:00Z', { input: 1_500_000_000, output: 100 }), + ); + expect(renderLedger(computeLedger(plan, env))).toContain('1.5B input'); + }); + + it('rounds fractional tiers to the nearest unit, not down', () => { + const { plan, env, project } = fixture(); + writeFileSync( + chatFile(project), + event('2026-08-03T10:01:00Z', { + input: 45_600, + cached: 5_700, + output: 100, + }), + ); + // 45_600 → "46k" (the docstring's own example, not "45k"), and + // 5_700 / 45_600 = 12.5% → "13% cached". + expect(renderLedger(computeLedger(plan, env))).toContain( + '46k input (13% cached)', + ); + }); + + it('recovers usage records glued onto one line by an interrupted append', () => { + const { plan, env, project } = fixture(); + const a = event('2026-08-03T10:01:00Z', { input: 1_000, output: 100 }); + const b = event('2026-08-03T10:02:00Z', { input: 2_000, output: 200 }); + // No newline between them: the documented corruption shape of these + // incrementally flushed files. A bare JSON.parse drops both records. + writeFileSync(chatFile(project), `${a}${b}\n`); + const ledger = computeLedger(plan, env); + expect(ledger.totals.calls).toBe(2); + expect(ledger.totals.inputTokens).toBe(3_000); + }); + + it('skips null-shaped lines instead of losing the whole ledger', () => { + const { plan, env, project } = fixture(); + writeFileSync( + chatFile(project), + [ + // JSON.parse('null') succeeds, so the parse guard alone would not + // catch it; "usageMetadata": null passes an === undefined guard. + 'null', + JSON.stringify({ + type: 'assistant', + timestamp: '2026-08-03T10:01:00Z', + usageMetadata: null, + }), + // Arrays are objects to typeof: without its own guard this would + // count as a phantom zero-token call. + JSON.stringify({ + type: 'assistant', + timestamp: '2026-08-03T10:01:30Z', + usageMetadata: [], + }), + event('2026-08-03T10:02:00Z', { input: 1_000, output: 100 }), + ].join('\n'), + ); + const ledger = computeLedger(plan, env); + expect(ledger.totals.calls).toBe(1); + expect(ledger.totals.inputTokens).toBe(1_000); + }); + + it('labels a chunk agent from its identity line, not as "agent chunk …"', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + writeFileSync( + join(project, 'subagents', SESSION, 'agent-chunk-c3.jsonl'), + [ + userRecord( + 'You are review agent `chunk 3 of 5` — the territory agent for ' + + 'lines 1-100 of the diff.', + ), + event('2026-08-03T10:06:30Z', { input: 1_000, output: 100 }), + ].join('\n'), + ); + const ledger = computeLedger(plan, env); + expect(ledger.agents[0].label).toBe('chunk 3'); + }); + + it('labels a free-text chunk mention by the file id, not the quoted chunk', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + // No identity line at the head: an older harness, or an agent this + // review never launched. Its free text names a chunk, but parsing free + // text for labels folds it into the real chunk-3 agent's row as a + // phantom (×2) relaunch. The file's own id is the one label it owns. + writeFileSync( + join(project, 'subagents', SESSION, 'agent-general-purpose-b7.jsonl'), + [ + userRecord('Task: reviewing chunk 3 of 5 for this PR.'), + event('2026-08-03T10:06:30Z', { input: 1_000, output: 100 }), + ].join('\n'), + ); + const ledger = computeLedger(plan, env); + expect(ledger.agents[0].label).toBe('general-purpose-b7'); + }); + + it('falls back to the file id when the prompt names neither role nor chunk', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + writeFileSync( + join(project, 'subagents', SESSION, 'agent-general-purpose-z0.jsonl'), + [ + userRecord('Do something useful.'), + event('2026-08-03T10:06:30Z', { input: 1_000, output: 100 }), + ].join('\n'), + ); + const ledger = computeLedger(plan, env); + expect(ledger.agents[0].label).toBe('general-purpose-z0'); + }); + + it('distinguishes parallel invariant agents by full path, not basename', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + // Step 3B launches invariant-a once PER heavy file, and a monorepo + // routinely holds same-basename files in different packages. Folding by + // bare role — or by basename — renders one (×2) row, the marker + // reserved for relaunches, and erases the per-file breakdown the + // distinguisher exists to keep. + for (const [file, owned] of [ + ['agent-invariant-x1.jsonl', 'packages/cli/src/config/storage.ts'], + ['agent-invariant-x2.jsonl', 'packages/core/src/config/storage.ts'], + ] as const) { + writeFileSync( + join(project, 'subagents', SESSION, file), + [ + userRecord( + 'You are review agent `invariant-a` — the hot-path audit.' + + ` Your file: \`${owned}\`.`, + ), + event('2026-08-03T10:06:30Z', { input: 1_000, output: 100 }), + ].join('\n'), + ); + } + const text = renderLedger(computeLedger(plan, env)); + expect(text).toContain( + 'agent invariant-a (packages/cli/src/config/storage.ts):', + ); + expect(text).toContain( + 'agent invariant-a (packages/core/src/config/storage.ts):', + ); + expect(text).not.toContain('(×2)'); + }); + + it('labels a fork agent from its launch prompt, not the bootstrap before it', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + writeFileSync( + join(project, 'subagents', SESSION, 'agent-fork-f1.jsonl'), + [ + // A fork's first record is the inherited conversation — an + // agent_bootstrap system record that quotes another agent's identity + // line and can outgrow any fixed head slice. The launch prompt is the + // first USER record, after it. + JSON.stringify({ + type: 'system', + subtype: 'agent_bootstrap', + timestamp: '2026-08-03T10:05:00Z', + systemPayload: { + kind: 'fork', + history: [ + { + text: + 'Earlier turn: You are review agent `chunk 1 of 5` — ' + + `the territory agent. ${'x'.repeat(70_000)}`, + }, + ], + }, + }), + userRecord('You are review agent `verify` — the reverse audit.'), + event('2026-08-03T10:06:30Z', { input: 1_000, output: 100 }), + ].join('\n'), + ); + const ledger = computeLedger(plan, env); + expect(ledger.agents[0].label).toBe('agent verify'); + }); + + it('orders agents by input, biggest first, regardless of file order', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + // The small agent gets the lexically EARLIER name and is written FIRST: + // a write-order filesystem and a lexical readdir both list [small, big], + // so only the sort can produce [big, small]. + writeFileSync( + join(project, 'subagents', SESSION, 'agent-role-a.jsonl'), + [ + userRecord('You are review agent `small` — dimension.'), + event('2026-08-03T10:06:30Z', { input: 1_000, output: 100 }), + ].join('\n'), + ); + writeFileSync( + join(project, 'subagents', SESSION, 'agent-role-z.jsonl'), + [ + userRecord('You are review agent `big` — dimension.'), + event('2026-08-03T10:06:30Z', { input: 900_000, output: 100 }), + ].join('\n'), + ); + const ledger = computeLedger(plan, env); + expect(ledger.agents.map((a) => a.label)).toEqual([ + 'agent big', + 'agent small', + ]); + }); + + it('skips agent files whose mtime predates the plan without opening them', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + // An earlier review in the same session: the dir is session-scoped and + // never pruned, so its files are still listed — the mtime pre-filter is + // what keeps them out. + const stale = join(project, 'subagents', SESSION, 'agent-role-old.jsonl'); + writeFileSync( + stale, + [ + userRecord('You are review agent `old` — an earlier review.'), + event('2026-08-03T10:06:30Z', { input: 9_000, output: 900 }), + ].join('\n'), + ); + const before = new Date('2026-08-03T09:30:00Z'); + utimesSync(stale, before, before); + const ledger = computeLedger(plan, env); + expect(ledger.agents).toEqual([]); + expect(ledger.totals.inputTokens).toBe(500); + }); + + it('floors agent events by timestamp even when the file itself is fresh', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + // A first-review agent still appending when the second review's plan + // lands: the file's mtime crosses the new floor (the pre-filter keeps + // it), so the event-level floor is the only thing keeping the earlier + // review's spend out of this ledger. + const straddling = join( + project, + 'subagents', + SESSION, + 'agent-role-first.jsonl', + ); + writeFileSync( + straddling, + [ + userRecord('You are review agent `first` — the earlier review.'), + event('2026-08-03T09:00:00Z', { input: 9_000, output: 900 }), + ].join('\n'), + ); + const after = new Date('2026-08-03T10:30:00Z'); + utimesSync(straddling, after, after); + const ledger = computeLedger(plan, env); + expect(ledger.agents).toEqual([]); + expect(ledger.totals.inputTokens).toBe(500); + }); + + it('skips an agent killed before its first response, with no phantom row', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + // Only the launch prompt, no usage event: the harness wrote the record, + // the agent never got a model response. + writeFileSync( + join(project, 'subagents', SESSION, 'agent-role-dead.jsonl'), + userRecord('You are review agent `dead` — killed at launch.'), + ); + writeFileSync( + join(project, 'subagents', SESSION, 'agent-role-live.jsonl'), + [ + userRecord('You are review agent `live` — dimension.'), + event('2026-08-03T10:06:30Z', { input: 1_000, output: 100 }), + ].join('\n'), + ); + const ledger = computeLedger(plan, env); + expect(ledger.agents).toHaveLength(1); + expect(ledger.agents[0].label).toBe('agent live'); + // The phantom must not inflate the run count the fold math feeds on. + expect(renderLedger(ledger)).toContain('agent runs: 1'); + }); + + it('folds a relaunched agent into one (×N) row', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + for (const [file, input] of [ + ['agent-general-purpose-a1.jsonl', 10_000], + ['agent-general-purpose-d9.jsonl', 12_000], + ] as const) { + writeFileSync( + join(project, 'subagents', SESSION, file), + [ + userRecord('You are review agent `2` — Agent 2: Security.'), + event('2026-08-03T10:06:30Z', { input, output: 100 }), + ].join('\n'), + ); + } + const text = renderLedger(computeLedger(plan, env)); + // The doubled run reads as one marked row, not two rows named alike. + expect(text).toContain('agent 2 (×2)'); + expect(text).toContain('22k in'); + expect(text).toContain('agent runs: 2'); + }); + + it('ranks a folded (×N) row by its combined total, not its first member', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + // Two relaunches at 5.0M each must outrank a solo 6.0M agent once + // folded, or the doubled run this ledger exists to surface is truncated + // away by the half that sorted lower. + for (const file of ['agent-role-p1.jsonl', 'agent-role-p2.jsonl']) { + writeFileSync( + join(project, 'subagents', SESSION, file), + [ + userRecord('You are review agent `2` — Agent 2: Security.'), + event('2026-08-03T10:06:30Z', { input: 5_000_000, output: 100 }), + ].join('\n'), + ); + } + writeFileSync( + join(project, 'subagents', SESSION, 'agent-role-solo.jsonl'), + [ + userRecord('You are review agent `3` — Agent 3: Tests.'), + event('2026-08-03T10:06:30Z', { input: 6_000_000, output: 100 }), + ].join('\n'), + ); + const text = renderLedger(computeLedger(plan, env)); + expect(text).toContain('agent 2 (×2): 2 calls · 10.0M in · 200 out'); + expect(text.indexOf('agent 2 (×2)')).toBeLessThan( + text.indexOf('agent 3: 1 call'), + ); + }); + + it('truncates the agent block past eight rows, keeping the biggest', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + for (let i = 1; i <= 9; i++) { + writeFileSync( + join(project, 'subagents', SESSION, `agent-role-${i}.jsonl`), + [ + userRecord(`You are review agent \`${i}\` — dimension ${i}.`), + event('2026-08-03T10:06:30Z', { input: 10_000 + i, output: 100 }), + ].join('\n'), + ); + } + const text = renderLedger(computeLedger(plan, env)); + expect(text).toContain('agent runs: 9'); + // Membership, not just the footnote: the cut keeps the top spenders and + // truncates the smallest — never the other way around. + expect(text).toContain('agent 9:'); + expect(text).not.toContain('agent 1:'); + expect(text).toContain('…and 1 more agent · 10k in combined'); + }); + + it('renders exactly eight agent rows with no truncation footnote', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + // The common full-roster shape (5 chunk + 3 dimension agents) lands + // exactly on the cut: no "…and 0 more agents" nonsense line. + for (let i = 1; i <= 8; i++) { + writeFileSync( + join(project, 'subagents', SESSION, `agent-role-${i}.jsonl`), + [ + userRecord(`You are review agent \`${i}\` — dimension ${i}.`), + event('2026-08-03T10:06:30Z', { input: 10_000 + i, output: 100 }), + ].join('\n'), + ); + } + const text = renderLedger(computeLedger(plan, env)); + expect(text).toContain('agent runs: 8'); + expect(text).toContain('agent 8:'); + expect(text).not.toContain('…and'); + }); + + it('counts folded runs below the cut as agents, not rows', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + // Eight solo rows big enough to keep the folded pair below the cut: the + // footnote must count its RUNS (2), not its rows (1) — a repair-round + // doubling hidden under the cut is exactly what the ledger exists to + // surface. + for (let i = 1; i <= 8; i++) { + writeFileSync( + join(project, 'subagents', SESSION, `agent-role-${i}.jsonl`), + [ + userRecord(`You are review agent \`${i}\` — dimension ${i}.`), + event('2026-08-03T10:06:30Z', { input: 100_000 + i, output: 100 }), + ].join('\n'), + ); + } + for (const file of ['agent-role-d1.jsonl', 'agent-role-d2.jsonl']) { + writeFileSync( + join(project, 'subagents', SESSION, file), + [ + userRecord('You are review agent `dup` — relaunched dimension.'), + event('2026-08-03T10:06:30Z', { input: 1_000, output: 100 }), + ].join('\n'), + ); + } + const text = renderLedger(computeLedger(plan, env)); + expect(text).toContain('agent runs: 10'); + expect(text).toContain('…and 2 more agents · 2k in combined'); + }); + it('keeps a reverse-audit chunk auditor apart from the territory finder', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + writeFileSync( + join(project, 'subagents', SESSION, 'agent-t3.jsonl'), + [ + userRecord( + 'You are review agent `chunk 3 of 5` — the territory agent.\n' + + 'read_file(file_path="/abs/diff.txt", offset=100, limit=50)', + ), + event('2026-08-03T10:07:00Z', { input: 40_000, output: 500 }), + ].join('\n'), + ); + writeFileSync( + join(project, 'subagents', SESSION, 'agent-ra3.jsonl'), + [ + userRecord( + 'You are review agent `chunk 3 of 5` — the territory agent.\n' + + 'read_file(file_path="/p/plan-prompts/reverse-audit--chunk-3--round-2--ab12cd.brief.md")', + ), + event('2026-08-03T10:08:00Z', { input: 30_000, output: 400 }), + ].join('\n'), + ); + + const text = renderLedger(computeLedger(plan, env)); + expect(text).toContain('chunk 3:'); + expect(text).toContain('audit chunk 3 (round 2):'); + expect(text).not.toContain('(×2)'); + }); + + it('labels an auditor from its own brief line, not a quoted audit path', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + // From round 2 on, an auditor's launch carries the folded findings ABOVE + // its own brief line, and those findings quote earlier rounds' brief + // paths — bare and in read_file shape alike. The label must come from + // the agent's OWN brief line, the last brief-shaped read_file in the + // launch: the folds always sit above it. + writeFileSync( + join(project, 'subagents', SESSION, 'agent-ra4.jsonl'), + [ + userRecord( + 'You are review agent `chunk 5 of 5` — the territory agent.\n' + + '\n' + + '## Already confirmed — do not re-report these\n' + + '\n' + + '- misattributed spend: reverse-audit--chunk-3--round-1--ab12cd.brief.md\n' + + '- quoted launch block:\n' + + ' read_file(file_path="/p/plan-prompts/reverse-audit--chunk-3--round-1--ab12cd.brief.md")\n' + + '\n' + + '**Your brief is a file. Read it first.**\n' + + '\n' + + 'read_file(file_path="/p/plan-prompts/reverse-audit--chunk-5--round-2--ef56ab.brief.md")', + ), + event('2026-08-03T10:08:00Z', { input: 30_000, output: 400 }), + ].join('\n'), + ); + + const text = renderLedger(computeLedger(plan, env)); + expect(text).toContain('audit chunk 5 (round 2):'); + expect(text).not.toContain('audit chunk 3'); + }); + + it('separates rounds by label while shards of one round still fold', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + const put = (file: string, identity: string, input: number) => + writeFileSync( + join(project, 'subagents', SESSION, file), + [ + userRecord(identity), + event('2026-08-03T10:07:00Z', { input, output: 100 }), + ].join('\n'), + ); + put( + 'agent-ra1.jsonl', + 'You are review agent `reverse-audit` — Reverse audit (round 1).', + 20_000, + ); + put( + 'agent-ra2.jsonl', + 'You are review agent `reverse-audit` — Reverse audit (round 2).', + 21_000, + ); + put( + 'agent-v1.jsonl', + 'You are review agent `verify` — Verification (round 2).', + 9_000, + ); + put( + 'agent-v2.jsonl', + 'You are review agent `verify` — Verification (round 2).', + 8_000, + ); + + const text = renderLedger(computeLedger(plan, env)); + expect(text).toContain('agent reverse-audit (round 1):'); + expect(text).toContain('agent reverse-audit (round 2):'); + expect(text).toContain('agent verify (round 2) (×2):'); + }); + + it('reads the round from the identity line, never from folded findings', () => { + const { plan, env, project } = fixture(); + writeMainCall(project); + writeFileSync( + join(project, 'subagents', SESSION, 'agent-v0.jsonl'), + [ + userRecord( + 'You are review agent `verify` — Verification.\n' + + '- quoted ledger row: agent verify (round 4): 1 call · 7k in · 80 out\n', + ), + event('2026-08-03T10:07:00Z', { input: 7_000, output: 80 }), + ].join('\n'), + ); + + const text = renderLedger(computeLedger(plan, env)); + expect(text).toContain('agent verify:'); + expect(text).not.toContain('agent verify (round 4)'); + }); +}); + +describe('cost-ledger command boundary — informational, never a failure', () => { + const dirs: string[] = []; + const savedEnv: Record = {}; + + function setEnv(env: NodeJS.ProcessEnv): void { + for (const k of ['QWEN_CODE_PROJECT_DIR', 'QWEN_CODE_SESSION_ID']) { + if (!(k in savedEnv)) savedEnv[k] = process.env[k]; + if (env[k] === undefined) delete process.env[k]; + else process.env[k] = env[k]; + } + } + + afterEach(() => { + process.exitCode = undefined; + for (const [k, v] of Object.entries(savedEnv)) { + if (v === undefined) delete process.env[k]; + else process.env[k] = v; + } + for (const d of dirs.splice(0)) rmSync(d, { recursive: true, force: true }); + }); + + function fixture(): { plan: string; project: string } { + const project = mkdtempSync(join(tmpdir(), 'ledger-cmd-')); + dirs.push(project); + mkdirSync(join(project, 'chats'), { recursive: true }); + writeFileSync(join(project, 'chats', `${SESSION}.jsonl`), ''); + const plan = join(project, 'plan.json'); + writeFileSync( + plan, + JSON.stringify({ + diffPathAbsolute: join(project, 'diff.txt'), + diffLines: 10, + chunks: [{ id: 1, startLine: 1, endLine: 10 }], + }), + ); + return { plan, project }; + } + + /** Drive the real yargs handler, as `qwen review cost-ledger` does. */ + function run(args: Record): { + stdout: string; + stderr: string; + } { + const stdout: string[] = []; + const stderr: string[] = []; + const outSpy = vi + .spyOn(process.stdout, 'write') + .mockImplementation((chunk) => { + stdout.push(chunk.toString()); + return true; + }); + const errSpy = vi + .spyOn(process.stderr, 'write') + .mockImplementation((chunk) => { + stderr.push(chunk.toString()); + return true; + }); + try { + (costLedgerCommand.handler as (a: unknown) => void)(args); + } finally { + outSpy.mockRestore(); + errSpy.mockRestore(); + } + // The describe's whole contract, asserted on every path: the ledger is + // informational, and a review must never fail on its own accounting — + // "exits 0" in each title is this line, not a hope. + expect(process.exitCode ?? 0).toBe(0); + return { stdout: stdout.join(''), stderr: stderr.join('') }; + } + + it('exits 0 with a reason when the ledger cannot be computed', () => { + const { plan } = fixture(); + setEnv({} as NodeJS.ProcessEnv); + const { stderr } = run({ plan }); + expect(stderr).toContain('cost-ledger unavailable'); + expect(stderr).toContain('QWEN_CODE_PROJECT_DIR'); + }); + + it('exits 0 and names the plan when the plan is missing', () => { + setEnv({ + QWEN_CODE_PROJECT_DIR: '/tmp', + QWEN_CODE_SESSION_ID: SESSION, + } as NodeJS.ProcessEnv); + const { stderr } = run({ plan: '/nonexistent/plan.json' }); + expect(stderr).toContain('cost-ledger unavailable'); + // The path, contiguous with OUR message: the relayed line is all a + // maintainer gets in headless CI, and the errno text happening to carry + // the path must not stand in for the message naming it. + expect(stderr).toContain( + 'could not read the plan report /nonexistent/plan.json', + ); + }); + + it('exits 0 and names a missing chat transcript instead of printing agents-only totals', () => { + const { plan, project } = fixture(); + rmSync(join(project, 'chats', `${SESSION}.jsonl`)); + setEnv({ + QWEN_CODE_PROJECT_DIR: project, + QWEN_CODE_SESSION_ID: SESSION, + } as NodeJS.ProcessEnv); + const { stdout, stderr } = run({ plan }); + expect(stderr).toContain('cost-ledger unavailable'); + // Contiguous, so the errno text carrying the path cannot mask a message + // that stopped naming it. + expect(stderr).toContain( + `could not read the chat transcript ${join(project, 'chats', `${SESSION}.jsonl`)}`, + ); + expect(stdout).not.toContain('Cost ledger:'); + }); + + it('exits 0 when the terminal writes throw — the reader went away', () => { + const { plan, project } = fixture(); + const start = new Date('2026-08-03T10:00:00Z'); + utimesSync(plan, start, start); + setEnv({ + QWEN_CODE_PROJECT_DIR: project, + QWEN_CODE_SESSION_ID: SESSION, + } as NodeJS.ProcessEnv); + // A pipe whose reader left (`qwen … | head`): the write throws. All + // three terminal writes — the unavailable warning, the could-not-write + // warning, and the ledger block — must absorb it; the review must never + // fail on its own accounting, including the accounting of a reader that + // left. + const pipeGone = (): boolean => { + throw Object.assign(new Error('write EPIPE'), { code: 'EPIPE' }); + }; + const runWithBrokenPipes = (args: Record): void => { + const outSpy = vi + .spyOn(process.stdout, 'write') + .mockImplementation(pipeGone); + const errSpy = vi + .spyOn(process.stderr, 'write') + .mockImplementation(pipeGone); + try { + (costLedgerCommand.handler as (a: unknown) => void)(args); + } finally { + outSpy.mockRestore(); + errSpy.mockRestore(); + } + expect(process.exitCode ?? 0).toBe(0); + }; + // The unavailable-warning arm: no above-floor records. + runWithBrokenPipes({ plan }); + // The ledger-block arm: a real spend. + writeFileSync( + join(project, 'chats', `${SESSION}.jsonl`), + JSON.stringify({ + type: 'assistant', + timestamp: '2026-08-03T10:01:00Z', + usageMetadata: { + promptTokenCount: 1_000, + candidatesTokenCount: 100, + totalTokenCount: 1_100, + }, + }), + ); + runWithBrokenPipes({ plan }); + }); + + it('writes --out into a directory it creates, with every stream kept', () => { + const { plan, project } = fixture(); + // A real spend, so the archive contract is observable: SKILL.md promises + // the --out JSON keeps every stream — totals alone cannot answer the + // "which agent doubled" question the archive exists for. + const start = new Date('2026-08-03T10:00:00Z'); + utimesSync(plan, start, start); + writeFileSync( + join(project, 'chats', `${SESSION}.jsonl`), + JSON.stringify({ + type: 'assistant', + timestamp: '2026-08-03T10:01:00Z', + usageMetadata: { + promptTokenCount: 1_000, + candidatesTokenCount: 100, + totalTokenCount: 1_100, + }, + }), + ); + mkdirSync(join(project, 'subagents', SESSION), { recursive: true }); + writeFileSync( + join(project, 'subagents', SESSION, 'agent-role-1.jsonl'), + [ + JSON.stringify({ + type: 'user', + timestamp: '2026-08-03T10:06:00Z', + message: { + role: 'user', + parts: [{ text: 'You are review agent `1` — dimension 1.' }], + }, + }), + JSON.stringify({ + type: 'assistant', + timestamp: '2026-08-03T10:06:30Z', + usageMetadata: { + promptTokenCount: 2_000, + candidatesTokenCount: 200, + totalTokenCount: 2_200, + }, + }), + ].join('\n'), + ); + setEnv({ + QWEN_CODE_PROJECT_DIR: project, + QWEN_CODE_SESSION_ID: SESSION, + } as NodeJS.ProcessEnv); + const out = join(project, 'archive', 'nested', 'ledger.json'); + const { stdout } = run({ plan, out }); + expect(stdout).toContain('Cost ledger:'); + const written = JSON.parse(readFileSync(out, 'utf8')) as { + totals: { calls: number }; + main: { id: string } | null; + agents: Array<{ id: string; label: string; inputTokens: number }>; + }; + expect(written.totals.calls).toBe(2); + expect(written.main?.id).toBe('main'); + expect(written.agents).toHaveLength(1); + expect(written.agents[0]).toMatchObject({ + id: 'role-1', + label: 'agent 1', + inputTokens: 2_000, + }); + }); + + it('degrades a failed --out write to a warning and still exits 0', () => { + const { plan, project } = fixture(); + const start = new Date('2026-08-03T10:00:00Z'); + utimesSync(plan, start, start); + writeFileSync( + join(project, 'chats', `${SESSION}.jsonl`), + JSON.stringify({ + type: 'assistant', + timestamp: '2026-08-03T10:01:00Z', + usageMetadata: { + promptTokenCount: 1_000, + candidatesTokenCount: 100, + totalTokenCount: 1_100, + }, + }), + ); + setEnv({ + QWEN_CODE_PROJECT_DIR: project, + QWEN_CODE_SESSION_ID: SESSION, + } as NodeJS.ProcessEnv); + const blocked = join(project, 'blocked'); + writeFileSync(blocked, 'a file where the archive directory would go'); + const { stdout, stderr } = run({ plan, out: join(blocked, 'ledger.json') }); + expect(stderr).toContain('could not write'); + expect(stdout).toContain('Cost ledger:'); + expect(existsSync(join(blocked, 'ledger.json'))).toBe(false); + }); +}); diff --git a/packages/cli/src/commands/review/cost-ledger.ts b/packages/cli/src/commands/review/cost-ledger.ts new file mode 100644 index 0000000000..2d9222f59b --- /dev/null +++ b/packages/cli/src/commands/review/cost-ledger.ts @@ -0,0 +1,555 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +// `qwen review cost-ledger`: what this review actually cost, from the +// harness's own usage records. +// +// The number exists because it kept having to be excavated. A maintainer's +// "0.21.3 was fine, 0.21.4 got slow" was settled only by replaying a whole +// review under a telemetry exporter and hand-aggregating half a million +// telemetry lines — hours of forensics for a question one printed table +// answers. The same excavation found the money: a small +93/-48 PR at high +// effort cost 523 model calls and 37.8M input tokens, 9.7M of them a repair +// round redelivering prompts the agents had already acted on. Nobody chose +// that spend; nobody could see it either. +// +// The data was on disk the whole time: every chat and subagent transcript +// event carries `usageMetadata` (prompt / candidates / thoughts / cached +// counts). This subcommand aggregates those records — the same records +// `check-coverage` trusts for delivery, read from the same +// environment-exported location — into per-stream totals. It is +// **informational**: a ledger that cannot be computed prints why and exits 0, +// because a review must never fail on its own accounting. +// +// A "model call" is an assistant record carrying `usageMetadata`; a turn +// whose provider returned no usage is invisible, so call counts are a floor, +// not an exact API-call tally. + +import type { CommandModule } from 'yargs'; +import { mkdirSync, readFileSync, statSync, writeFileSync } from 'node:fs'; +import { dirname, join, resolve } from 'node:path'; +import { parseLineTolerant } from '@qwen-code/qwen-code-core'; +import { + writeStdoutLineSafe, + writeStderrLineSafe, +} from '../../utils/stdioHelpers.js'; +import { + transcriptPaths, + listAgentTranscriptFiles, + TranscriptsUnavailableError, + textOf, +} from './lib/transcripts.js'; +import { CHUNK_RE } from './lib/coverage.js'; + +interface CostLedgerArgs { + plan: string; + out?: string; +} + +interface StreamCost { + /** `main` for the orchestrator session, else the agent file's id. */ + id: string; + /** Human label: the role parsed from the launch prompt when one is found. */ + label: string; + calls: number; + inputTokens: number; + cachedTokens: number; + outputTokens: number; + thoughtsTokens: number; + firstAt: string | null; + lastAt: string | null; +} + +interface Ledger { + totals: Omit & { wallSeconds: number }; + main: StreamCost | null; + agents: StreamCost[]; +} + +interface UsageEvent { + timestampMs: number; + timestamp: string; + input: number; + cached: number; + output: number; + thoughts: number; +} + +/** + * One read of a transcript: its usage-bearing assistant events, floor-filtered, + * plus the launch prompt the label comes from. + * + * The launch prompt is the first `user` record's text — the same anchor + * `parseTranscript` in lib/transcripts.ts uses — never a raw byte slice of the + * file: a fork agent's transcript opens with an `agent_bootstrap` system + * record carrying the entire inherited conversation, which can quote other + * agents' identity lines and outgrow any fixed head window. + * + * Read failures throw; the caller decides what they mean — for the chat file, + * "the ledger cannot be computed"; for one agent file, "that agent is lost". + */ +function readUsage( + file: string, + floorMs: number, +): { events: UsageEvent[]; launch: string } { + const raw = readFileSync(file, 'utf8'); + const events: UsageEvent[] = []; + let launch = ''; + for (const line of raw.split('\n')) { + if (line.trim() === '') continue; + // parseLineTolerant recovers the `}{`-glued records an interrupted append + // leaves behind — the documented corruption shape of these incrementally + // flushed files — and drops non-object lines, which a bare JSON.parse + // parses happily (`null`, `42`) only to trip the property reads below. + for (const rec of parseLineTolerant>(line, file)) { + if (launch === '' && rec['type'] === 'user') { + launch = textOf(rec); + } + if (rec['type'] !== 'assistant') continue; + const usage = rec['usageMetadata']; + if (usage === null || typeof usage !== 'object') continue; + if (Array.isArray(usage)) continue; + const u = usage as Record; + const ts = rec['timestamp']; + if (typeof ts !== 'string') continue; + const tsMs = Date.parse(ts); + // The chat file spans the whole session, not the review: a `/review` + // launched an hour into a working session would otherwise bill that hour's + // conversation to the review. The plan's own mtime marks the review start + // — the same floor `check-coverage` applies to transcripts. + if (!Number.isFinite(tsMs) || tsMs < floorMs) continue; + // Finite ≥ 0, else null: the main loop coerces broken-proxy usage + // (negative or NaN counts) before recording, but the agent path records + // raw provider usage, and each consumer below picks its own fallback + // for a count that did not survive the provider. + const rawCount = (k: string): number | null => { + const v = u[k]; + return typeof v === 'number' && Number.isFinite(v) && v >= 0 ? v : null; + }; + const rawPrompt = rawCount('promptTokenCount'); + const rawTotal = rawCount('totalTokenCount'); + const prompt = rawPrompt ?? 0; + const candidates = rawCount('candidatesTokenCount') ?? 0; + events.push({ + timestampMs: tsMs, + timestamp: ts, + input: prompt, + // Cached is part of the prompt it is reported with; a broken proxy + // can report the pair inverted, and the inversion would otherwise + // land in BOTH the rendered share and the archived --out JSON. + cached: Math.min(rawCount('cachedContentTokenCount') ?? 0, prompt), + // `total − prompt` is the output including thinking under BOTH usage + // conventions — reasoning inside candidates, and thoughts disjoint + // from them — the same derivation tokenEstimation uses. Candidates + // alone is the fallback when the provider reported no total; it stays + // correct for the providers this CLI converts, which clamp thoughts + // inside candidates. Derive it only when BOTH operands survived the + // provider intact: a mixed-sign record coerces prompt to 0 while + // keeping a positive total, and the subtraction would bill the + // call's whole total as output. + output: + rawPrompt !== null && rawTotal !== null && rawTotal > rawPrompt + ? rawTotal - rawPrompt + : candidates, + thoughts: rawCount('thoughtsTokenCount') ?? 0, + }); + } + } + return { events, launch }; +} + +/** + * An auditor's own brief line, exactly as `buildRoleLaunchPrompt` prints it. + * A bare `reverse-audit--chunk-N--round-M--` path mentioned anywhere is + * NOT this: folded findings quote other rounds' brief paths routinely. + */ +const AUDIT_BRIEF_RE = + /read_file\(file_path="[^"]*reverse-audit--chunk-(\d+)--round-(\d+)--[0-9a-f][^"]*\.brief\.md"\)/g; + +/** A role label out of the launch prompt, else the fallback. */ +function labelOf(launch: string, fallback: string): string { + // Every identity-based parse stays on the identity LINE, and nothing runs + // at all without one at the head. The folded findings below it can quote + // budget disclosures' "(round N)", other agents' `Your file:` lines, ledger + // rows, and `You are review agent` lines — a whole-launch match would hand + // the quoted label to the agent carrying the quote. A prompt with no + // identity line is an older harness's, or an agent this review never + // launched (the session's transcript dir also holds nested subagents): its + // free text can name anything, so keep the one label it owns — the file id. + const nl = launch.indexOf('\n'); + const identity = nl === -1 ? launch : launch.slice(0, nl); + if (!identity.startsWith('You are review agent `')) return fallback; + // A reverse-audit chunk auditor shares its launch shape with the territory + // finder; only its brief path carries the stage and the round — without + // it, five audit rounds fold into one row and the ledger reports one agent + // where six pipeline stages ran. Match the agent's OWN brief line, never a + // quoted mention: the folds sit ABOVE the agent's own brief line + // (foldFindings folds them there), so the last brief-shaped read_file in + // the launch is the agent's own. + let auditChunk: RegExpExecArray | null = null; + for (const m of launch.matchAll(AUDIT_BRIEF_RE)) auditChunk = m; + if (auditChunk) { + return `audit chunk ${auditChunk[1]} (round ${auditChunk[2]})`; + } + const role = /^You are review agent `([^`]+)`/.exec(identity); + if (!role) return fallback; + const round = /\(round (\d+)\)/.exec(identity); + const chunk = CHUNK_RE.exec(role[1]); + // A chunk role is `chunk N of M`; prefixing it with "agent" would read as + // a malformed role, so resolve it through the same regex coverage uses. + if (chunk) return `chunk ${chunk[1]}`; + if (round) { + // Shards of one verify round carry the same label and fold; distinct + // rounds — verify and reverse-audit alike — are distinct rows. + return `agent ${role[1]} (round ${round[1]})`; + } + // An invariant role launches once PER heavy file. The role alone would + // fold those parallel runs into one (×N) row — the marker reserved for + // relaunches — and lose the per-file breakdown. The identity line names + // the owned file; the FULL path is the distinguisher, because a monorepo + // routinely holds same-basename files in different packages. + const file = /Your file: `([^`]+)`/.exec(identity); + if (file) return `agent ${role[1]} (${file[1]})`; + return `agent ${role[1]}`; +} + +function foldEvents( + id: string, + label: string, + events: UsageEvent[], +): StreamCost { + const s: StreamCost = { + id, + label, + calls: 0, + inputTokens: 0, + cachedTokens: 0, + outputTokens: 0, + thoughtsTokens: 0, + firstAt: null, + lastAt: null, + }; + let firstMs = Number.POSITIVE_INFINITY; + let lastMs = Number.NEGATIVE_INFINITY; + for (const e of events) { + s.calls += 1; + s.inputTokens += e.input; + s.cachedTokens += e.cached; + s.outputTokens += e.output; + s.thoughtsTokens += e.thoughts; + if (e.timestampMs < firstMs) { + firstMs = e.timestampMs; + s.firstAt = e.timestamp; + } + if (e.timestampMs > lastMs) { + lastMs = e.timestampMs; + s.lastAt = e.timestamp; + } + } + return s; +} + +/** 12_345_678 → "12.3M"; 45_600 → "46k"; 890 → "890". */ +function human(n: number): string { + // 999_500 rounds to 1000k; from there up, render in M so it reads "1.0M". + // The same boundary at the B tier keeps 1.5e9 from reading "1500.0M". + if (n >= 999_500_000) return `${(n / 1_000_000_000).toFixed(1)}B`; + if (n >= 999_500) return `${(n / 1_000_000).toFixed(1)}M`; + if (n >= 1_000) return `${Math.round(n / 1_000)}k`; + return String(n); +} + +/** "1 call", "2 calls" — the rendered block is archived verbatim. */ +const plural = (n: number, word: string): string => + `${n} ${word}${n === 1 ? '' : 's'}`; + +/** + * The plan's mtime is the billing floor. Validate the file IS a Step 1 plan + * report before trusting that mtime: a wrong-but-existing file (the findings + * JSON, the report written minutes earlier) would move the floor silently in + * either direction. The shape checked is the pair every Step 1 report carries + * — `diffLines` and `chunks` — not `check-coverage`'s stricter contract: a + * degraded capture (unresolvable merge base, the tiling fallback) writes + * `diffPathAbsolute: null` with `chunks: []`, and that report's mtime is + * still exactly the floor this ledger needs. + */ +function planFloorMs(planPath: string): number { + let raw: string; + let floorMs: number; + try { + raw = readFileSync(planPath, 'utf8'); + floorMs = statSync(planPath).mtimeMs; + } catch (err) { + throw new Error( + `could not read the plan report ${planPath}: ${(err as Error).message}`, + ); + } + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch { + parsed = null; + } + const plan = + parsed !== null && typeof parsed === 'object' + ? (parsed as Record) + : undefined; + if ( + typeof plan?.['diffLines'] !== 'number' || + !Array.isArray(plan?.['chunks']) + ) { + throw new Error(`not a review plan report: ${planPath}`); + } + return floorMs; +} + +export function computeLedger( + planPath: string, + env: NodeJS.ProcessEnv = process.env, +): Ledger { + const floorMs = planFloorMs(planPath); + const { projectDir, sessionId, dir } = transcriptPaths(env); + + const chatFile = join(projectDir, 'chats', `${sessionId}.jsonl`); + let mainEvents: UsageEvent[]; + try { + mainEvents = readUsage(chatFile, floorMs).events; + } catch (err) { + // The plan's existence proves the main loop ran: a missing or unreadable + // chat file is an infrastructure fact (chat recording off, or a fault), + // not a verdict that the loop made no calls. Agents-only totals would + // read as the review's whole cost, so say the ledger cannot be computed. + throw new Error( + `could not read the chat transcript ${chatFile}: ` + + `${(err as Error).message}`, + ); + } + const main = + mainEvents.length > 0 ? foldEvents('main', 'main loop', mainEvents) : null; + + let files: string[]; + try { + files = listAgentTranscriptFiles(dir); + } catch (err) { + if ((err as NodeJS.ErrnoException).code === 'ENOENT') { + // No subagent dir is a real state (a low-effort review runs no agents); + // the ledger reports what exists. + files = []; + } else { + // EACCES/EIO/ENOTDIR are not "no agents": main-loop-only totals would + // read as the complete ledger. + throw new Error( + `could not list the subagent transcripts at ${dir}: ` + + `${(err as Error).message}`, + ); + } + } + + const agents: StreamCost[] = []; + const agentEvents: UsageEvent[] = []; + for (const f of files) { + const full = join(dir, f); + let mtimeMs: number; + try { + mtimeMs = statSync(full).mtimeMs; + } catch { + continue; // Gone between listing and stat. + } + // The transcript dir is session-scoped and never pruned: files from + // earlier reviews this session predate the floor, and a file whose last + // write predates it cannot hold an above-floor record — the same + // membership test `readTranscripts` applies. Skip it without opening. + if (mtimeMs < floorMs) continue; + let read: { events: UsageEvent[]; launch: string }; + try { + read = readUsage(full, floorMs); + } catch { + continue; // This agent's record is lost; the rest still count. + } + if (read.events.length === 0) continue; + const id = f.replace(/^agent-/, '').replace(/\.jsonl$/, ''); + agents.push(foldEvents(id, labelOf(read.launch, id), read.events)); + agentEvents.push(...read.events); + } + agents.sort((a, b) => b.inputTokens - a.inputTokens); + + // A present-but-empty window is not a lighter version of a missing one. + // The recorder pre-creates the chat file and degrades permanently if its + // first append fails, so "exists, yet no above-floor records" — with or + // without agents — is the same infrastructure fact as "unreadable": a live + // review holds at least one above-floor main-loop record, because the plan + // itself is a main-loop write. Rendering a zero ledger from this shape + // would be diffed against real numbers — exactly the fabrication the + // refusal above names. + if (mainEvents.length === 0) { + throw new Error( + `could not read the chat transcript ${chatFile}: no main-loop usage ` + + 'records at or after the plan', + ); + } + + // The same events the per-stream rows fold, folded once more — one + // accumulator, so a new usage counter cannot land in the rows and miss the + // headline. + const totals = foldEvents('totals', 'totals', [ + ...mainEvents, + ...agentEvents, + ]); + const wallSeconds = + totals.firstAt !== null && totals.lastAt !== null + ? Math.max( + 0, + Math.round( + (Date.parse(totals.lastAt) - Date.parse(totals.firstAt)) / 1000, + ), + ) + : 0; + + const { id: _i, label: _l, ...totalsRest } = totals; + return { totals: { ...totalsRest, wallSeconds }, main, agents }; +} + +/** The printed block: one summary line, the main loop, the top consumers. */ +export function renderLedger(ledger: Ledger): string { + const t = ledger.totals; + const cachedPct = + t.inputTokens > 0 ? Math.round((t.cachedTokens / t.inputTokens) * 100) : 0; + const lines: string[] = []; + lines.push( + `Cost ledger: ${plural(t.calls, 'model call')} · ` + + `${human(t.inputTokens)} input (${cachedPct}% cached) · ` + + `${human(t.outputTokens)} output (${human(t.thoughtsTokens)} thinking) · ` + + `${Math.round(t.wallSeconds / 60)} min wall`, + ); + if (ledger.main !== null) { + const m = ledger.main; + lines.push( + ` main loop: ${plural(m.calls, 'call')} · ${human(m.inputTokens)} in · ` + + `${human(m.outputTokens)} out`, + ); + } + if (ledger.agents.length > 0) { + // Equal labels fold into one row marked (×N): a relaunched agent keeps + // its label, and verify shards deliberately share one — the marker reads + // "N runs under this label", and the repair round this ledger exists to + // surface becomes visible, not merely present. Rounds and audit chunks + // do NOT share labels (labelOf carries their stage and round), so a + // five-round audit is five rows, not a phantom ×5 relaunch. + const rows: Array<{ + label: string; + calls: number; + inputTokens: number; + outputTokens: number; + count: number; + }> = []; + for (const a of ledger.agents) { + const row = rows.find((r) => r.label === a.label); + if (row) { + row.calls += a.calls; + row.inputTokens += a.inputTokens; + row.outputTokens += a.outputTokens; + row.count += 1; + } else { + rows.push({ + label: a.label, + calls: a.calls, + inputTokens: a.inputTokens, + outputTokens: a.outputTokens, + count: 1, + }); + } + } + // Rank by the folded total, not the first member's share: a doubled run + // must not be truncated away by the half that sorted lower. + rows.sort((a, b) => b.inputTokens - a.inputTokens); + lines.push(` agent runs: ${ledger.agents.length}`); + for (const r of rows.slice(0, 8)) { + const label = r.count > 1 ? `${r.label} (×${r.count})` : r.label; + lines.push( + ` ${label}: ${plural(r.calls, 'call')} · ` + + `${human(r.inputTokens)} in · ${human(r.outputTokens)} out`, + ); + } + if (rows.length > 8) { + const rest = rows.slice(8); + const restIn = rest.reduce((n, r) => n + r.inputTokens, 0); + const restAgents = rest.reduce((n, r) => n + r.count, 0); + lines.push( + ` …and ${plural(restAgents, 'more agent')} · ` + + `${human(restIn)} in combined`, + ); + } + } + return lines.join('\n'); +} + +function runCostLedger(args: CostLedgerArgs): void { + // EPIPE arrives two ways when the reader goes away (`qwen … | head`, a + // daemon's closed redirect): a sync throw out of the write, and an async + // 'error' event on the pipe. The safe writers catch the first; destroy the + // stream on the second — the convention nonInteractiveCli uses — and + // detach both listeners on exit. A review must never fail on its own + // accounting, including the accounting of a reader that left. + const stdoutErrorHandler = (err: NodeJS.ErrnoException): void => { + if (err.code === 'EPIPE') process.stdout.destroy(); + }; + const stderrErrorHandler = (err: NodeJS.ErrnoException): void => { + if (err.code === 'EPIPE') process.stderr.destroy(); + }; + process.stdout.on('error', stdoutErrorHandler); + process.stderr.on('error', stderrErrorHandler); + try { + let ledger: Ledger; + try { + ledger = computeLedger(args.plan, process.env); + } catch (err) { + // Informational, always: a review must never fail on its own accounting. + const why = + err instanceof TranscriptsUnavailableError + ? err.message + : (err as Error).message; + writeStderrLineSafe(`cost-ledger unavailable — ${why}`); + return; + } + if (args.out !== undefined && args.out.length > 0) { + // A failed archive write degrades to a warning: the ledger was computed, + // and the exit code must stay 0 either way. + try { + mkdirSync(dirname(resolve(args.out)), { recursive: true }); + writeFileSync(args.out, JSON.stringify(ledger, null, 2)); + } catch (err) { + writeStderrLineSafe( + `cost-ledger: could not write ${args.out} — ${(err as Error).message}`, + ); + } + } + writeStdoutLineSafe(renderLedger(ledger)); + } finally { + process.stdout.removeListener('error', stdoutErrorHandler); + process.stderr.removeListener('error', stderrErrorHandler); + } +} + +export const costLedgerCommand: CommandModule = { + command: 'cost-ledger', + describe: + "Aggregate this review's model-call cost from the harness's usage records", + builder: (yargs) => + yargs + .option('plan', { + type: 'string', + demandOption: true, + describe: + 'The plan report from Step 1 — its mtime marks the review start', + }) + .option('out', { + type: 'string', + describe: 'Also write the full ledger as JSON to this path', + }), + handler: (args) => { + runCostLedger(args as unknown as CostLedgerArgs); + }, +}; diff --git a/packages/cli/src/commands/review/lib/coverage.ts b/packages/cli/src/commands/review/lib/coverage.ts index 778aa12d32..1bee5ddddf 100644 --- a/packages/cli/src/commands/review/lib/coverage.ts +++ b/packages/cli/src/commands/review/lib/coverage.ts @@ -235,7 +235,7 @@ function readPlan(path: string): { plan: Plan; mtimeMs: number } { } /** `chunk 13 of 25` — written into the prompt by `agent-prompt`, in code. */ -const CHUNK_RE = /\bchunk\s+(\d+)\s+of\s+\d+\b/i; +export const CHUNK_RE = /\bchunk\s+(\d+)\s+of\s+\d+\b/i; /** The chunk this agent owns, when it was launched to own one. */ function assignedChunk(rec: AgentRecord): number | null { diff --git a/packages/cli/src/commands/review/lib/transcripts.test.ts b/packages/cli/src/commands/review/lib/transcripts.test.ts index 9b563f06b7..adb92140bb 100644 --- a/packages/cli/src/commands/review/lib/transcripts.test.ts +++ b/packages/cli/src/commands/review/lib/transcripts.test.ts @@ -67,6 +67,39 @@ describe('readTranscripts — defensive parsing', () => { expect(readTranscripts(undefined, ENV)).toEqual([]); }); + it('skips the harness sidecar files beside a transcript', () => { + // The harness writes sibling files per agent into this dir — + // agent-transcript.ts writes `agent-.meta.json` via writeAgentMeta, + // and a meta carries an `agentId` key. Admitted by the filter, it would + // parse to a phantom zero-tool-call AgentRecord: it is the + // `.endsWith('.jsonl')` filter, not parseTranscript, that keeps it out. + file( + 'agent-a1.jsonl', + JSON.stringify({ + agentId: 'a1', + agentName: 'general-purpose', + type: 'user', + message: { role: 'user', parts: [{ text: 'chunk 1 of 1' }] }, + }) + '\n', + ); + file( + 'agent-a1.meta.json', + JSON.stringify({ + agentId: 'a1', + agentType: 'general-purpose', + description: 'dimension 1', + parentSessionId: 'S1', + parentAgentId: null, + createdAt: '2026-08-03T10:06:00.000Z', + status: 'completed', + }), + ); + file('agent-a1.jsonl.stream', 'streaming text, not jsonl records'); + const recs = readTranscripts(undefined, ENV); + expect(recs).toHaveLength(1); + expect(recs[0].agentId).toBe('a1'); + }); + it('skips an empty transcript file', () => { file('agent-empty.jsonl', ''); expect(readTranscripts(undefined, ENV)).toEqual([]); diff --git a/packages/cli/src/commands/review/lib/transcripts.ts b/packages/cli/src/commands/review/lib/transcripts.ts index ae3bb57531..08274dbd0e 100644 --- a/packages/cli/src/commands/review/lib/transcripts.ts +++ b/packages/cli/src/commands/review/lib/transcripts.ts @@ -87,16 +87,22 @@ export interface AgentRecord { export class TranscriptsUnavailableError extends Error {} /** - * Where this session's subagent transcripts live. + * The environment this module reads, validated once and returned together. * * Both halves come from the environment the CLI exported, never from an argument: * a path the model can choose is a path the model can point somewhere flattering. * `QWEN_CODE_PROJECT_DIR` exists because the project dir is keyed on the session's * *launch* cwd, and this subcommand may well be running inside a PR worktree the * skill `cd`-ed into — recomputing it from `process.cwd()` yields a directory that - * never existed. + * never existed. Callers that need both halves (the chat file lives beside the + * subagent dir) take them here rather than re-reading the env after `transcriptDir` + * validated it. */ -export function transcriptDir(env: NodeJS.ProcessEnv = process.env): string { +export function transcriptPaths(env: NodeJS.ProcessEnv = process.env): { + projectDir: string; + sessionId: string; + dir: string; +} { const projectDir = env['QWEN_CODE_PROJECT_DIR']?.trim(); const sessionId = env['QWEN_CODE_SESSION_ID']?.trim(); if (!projectDir || !sessionId) { @@ -105,11 +111,20 @@ export function transcriptDir(env: NodeJS.ProcessEnv = process.env): string { "this run cannot find the harness's record of what its agents did", ); } - return join(projectDir, 'subagents', sessionId); + return { + projectDir, + sessionId, + dir: join(projectDir, 'subagents', sessionId), + }; +} + +/** Where this session's subagent transcripts live. */ +export function transcriptDir(env: NodeJS.ProcessEnv = process.env): string { + return transcriptPaths(env).dir; } /** Text out of a record's message parts. */ -function textOf(rec: Record): string { +export function textOf(rec: Record): string { const msg = rec['message'] as { parts?: unknown } | undefined; const parts = Array.isArray(msg?.parts) ? msg.parts : []; return parts @@ -306,6 +321,21 @@ function parseTranscript(file: string, diffPath?: string): AgentRecord | null { }; } +/** + * The session's subagent transcript files, one listing every reader shares. + * + * The coverage gate and the cost ledger both claim to read "the same records", + * and the harness writes sibling file kinds per agent (`.meta.json`, + * `.jsonl.stream`) with a generalized `-.jsonl` namespace planned — + * so the definition of "which files are transcripts" lives here, once, not in + * each reader's own filter. Throws on any readdir failure; what the caller + * does with that (name the fault, or treat an absent dir as "no agents") is + * its decision. + */ +export function listAgentTranscriptFiles(dir: string): string[] { + return readdirSync(dir).filter((name) => name.endsWith('.jsonl')); +} + /** * Every subagent this session launched, as the harness recorded it. * @@ -323,7 +353,7 @@ export function readTranscripts( const dir = transcriptDir(env); let names: string[]; try { - names = readdirSync(dir); + names = listAgentTranscriptFiles(dir); } catch (err) { // No directory at all is an *infrastructure* fact, not a verdict about the // agents. Conflating the two would let a read-only HOME or a full disk read @@ -337,7 +367,6 @@ export function readTranscripts( const out: AgentRecord[] = []; for (const name of names) { - if (!name.endsWith('.jsonl')) continue; const rec = parseTranscript(join(dir, name), diffPath); if (!rec) continue; if (since !== undefined && rec.mtimeMs < since) continue; diff --git a/packages/cli/src/utils/stdioHelpers.test.ts b/packages/cli/src/utils/stdioHelpers.test.ts index db30c6c583..cab69e1074 100644 --- a/packages/cli/src/utils/stdioHelpers.test.ts +++ b/packages/cli/src/utils/stdioHelpers.test.ts @@ -5,7 +5,11 @@ */ import { afterEach, describe, expect, it, vi } from 'vitest'; -import { writeStderrLine, writeStderrLineSafe } from './stdioHelpers.js'; +import { + writeStderrLine, + writeStderrLineSafe, + writeStdoutLineSafe, +} from './stdioHelpers.js'; afterEach(() => vi.restoreAllMocks()); @@ -50,3 +54,21 @@ describe('writeStderrLineSafe', () => { expect(() => writeStderrLineSafe('boom')).not.toThrow(); }); }); + +describe('writeStdoutLineSafe', () => { + it('writes with a trailing newline when stdout is healthy', () => { + const write = vi.spyOn(process.stdout, 'write').mockReturnValue(true); + + writeStdoutLineSafe('hello'); + + expect(write).toHaveBeenCalledWith('hello\n'); + }); + + it('swallows EPIPE instead of taking the caller down with it', () => { + vi.spyOn(process.stdout, 'write').mockImplementation(() => { + throw Object.assign(new Error('write EPIPE'), { code: 'EPIPE' }); + }); + + expect(() => writeStdoutLineSafe('boom')).not.toThrow(); + }); +}); diff --git a/packages/cli/src/utils/stdioHelpers.ts b/packages/cli/src/utils/stdioHelpers.ts index d0c88ca8a4..587837945b 100644 --- a/packages/cli/src/utils/stdioHelpers.ts +++ b/packages/cli/src/utils/stdioHelpers.ts @@ -32,6 +32,21 @@ export const writeStderrLine = (message: string): void => { process.stderr.write(message.endsWith('\n') ? message : `${message}\n`); }; +/** + * `writeStdoutLine` that cannot throw. + * + * Same contract as `writeStderrLineSafe`: use it where the write is + * incidental to the work in hand — an informational block whose reader + * going away (`qwen … | head`) must not fail the command. + */ +export const writeStdoutLineSafe = (message: string): void => { + try { + writeStdoutLine(message); + } catch { + // stdout is gone. Whatever this line had to say, its reader left. + } +}; + /** * `writeStderrLine` that cannot throw. * diff --git a/packages/core/src/skills/bundled/review/SKILL.md b/packages/core/src/skills/bundled/review/SKILL.md index c548f29346..fe1e9fe827 100644 --- a/packages/core/src/skills/bundled/review/SKILL.md +++ b/packages/core/src/skills/bundled/review/SKILL.md @@ -1112,6 +1112,7 @@ Report content should include: - All findings with verification status. Read them out of the findings artifact `qwen review findings` wrote (`.qwen/tmp/qwen-review-{target}-findings.json`) rather than re-typing them from the terminal — a third transcription of the same list is a third chance for a severity to drift, which has happened inside a single review. - **Per-finding outcomes, when Step 6B ran** — `fixed` / `skipped` / `no_change_needed`, with the reason for every `skipped`. The artifact already carries them; a `--fix` run whose archive does not say which findings were applied is a report that reads as if all of them were. - Verdict (high and medium effort — a low quick pass claims none; a medium verdict never exceeds Comment, since it runs no reverse audit — see Step 5) +- **The cost ledger — run it, do not compute it.** `"${QWEN_CODE_CLI:-qwen}" review cost-ledger --plan --out .qwen/reviews/-cost-ledger.json` aggregates the model calls the harness recorded for this review — the main loop and each agent, with input / cached / output / thinking token counts and wall time — from the harness's own usage records, the same records the coverage gate trusts. The window is bounded: it starts at the plan's mtime, and the ledger runs at this step, so the pre-plan bootstrap turns and the composition after this snapshot are not captured, and side queries such as chat compression leave no usage records to capture at all. Paste its printed block into the report verbatim, and relay the first line in the terminal summary. The printed block lists only the eight biggest agents; the `--out` JSON keeps every one, so the diffable record survives in full (worktree mode: resolve `--out` against the main project directory, like the report itself). If it prints `cost-ledger unavailable`, note that instead — it is informational and never blocks a review. Why it is in the archive: a "this version got slower" report is unanswerable from memory, and the one time it was answered properly took hours of telemetry forensics to find a repair round that had silently doubled a run. The ledger makes the next such question a diff of two saved reports. **The report's verdict is not yours to type.** `compose-review` printed the exact `Verdict:` line in Step 6 and persisted the same line as `verdictLine` inside `.qwen/tmp/qwen-review-{target}-composed.json` — copy either, verbatim. Do not reconstruct it from `event` + `cappedBy`: a presubmit downgrade also depends on fields that pair does not carry, and a rebuilt line can differ from the computed one. (And not `$(jq …)`: a `jq` binary is not guaranteed on the host, and a substitution that fails leaves the archived verdict blank or literal — worse than absent, because it looks written.)