mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-08-06 23:35:34 +00:00
feat(review): a cost ledger from the records already on disk (#8471)
* feat(review): a cost ledger from the records already on disk "0.21.3 was fine, 0.21.4 got slow" was settled only by replaying a whole review under a telemetry exporter and hand-aggregating the output — hours of forensics to find a repair round that had silently doubled a run (measured: a +93/-48 PR at high effort cost 523 model calls and 37.8M input tokens, 9.7M of them redelivering prompts the agents had already acted on). The usage data was on disk the whole time: every chat and subagent transcript event carries usageMetadata. qwen review cost-ledger --plan <plan> aggregates those records — the same files check-coverage trusts for delivery, found via the same environment-exported location, floored at the plan's mtime so a review started an hour into a session does not bill that hour — into per-stream totals: the main loop and each agent, with input / cached / output / thinking counts and wall time. Step 8 pastes the printed block into the saved report, so the next slowness question is a diff of two archives instead of an excavation. Informational by construction: an incomputable ledger prints why and exits 0. Validated against the measured run above — 521 calls, 37.7M input (93% cached), 849k output, 107 min — matching the telemetry-side aggregation, minus the two side-query calls that are not the review's. * test: register cost-ledger in the subcommand registry and its demand message * fix(review): honest cost-ledger output and a safe archive write (#8471) Address the review of the cost ledger: report output tokens once (thinking is a subset of candidates, not a sibling), keep the --out write inside the exit-0 contract and mkdir its parent, name a missing plan as the plan, read each transcript once, compare timestamps as instants, fold relaunched agents into marked rows, and archive the full ledger next to the Step 8 report. * fix(review): cost ledger — honest failures, validated plan, shared records (#8471) Address the second review round: a missing or faulted chat transcript now says "cost-ledger unavailable" instead of rendering agents-only totals as the whole cost (the plan proves the main loop ran), and a subagent dir that fails listing with anything but ENOENT does the same. The --plan file is validated as a plan report before its mtime alone sets the billing window. Output derives from totalTokenCount − promptTokenCount when present, correct under both usage conventions. Chunk agents label "chunk N" via the shared CHUNK_RE instead of the malformed "agent chunk N of M"; the transcript listing is one helper shared with the coverage gate; glued JSONL records are recovered via parseLineTolerant; totals reuse the rows' accumulator; folded (×N) rows rank by combined total; stale agent files are skipped by mtime without being opened. Rendered block gains pluralization, "agent runs: N", and a B tier; SKILL.md states the ledger's bounded window. * fix(review): close the cost-ledger audit — refusals, labels, pinned math Address the remaining review threads on the cost ledger: - Refuse agents-only totals when the chat file exists but holds no above-floor records: a degraded recorder leaves the file present and empty while agents run — the same infrastructure fact as an unreadable transcript, and exactly the output the missing-file refusal exists to prevent. - Read the agent label from the first user record, not a raw 64KB head slice: a fork's agent_bootstrap record precedes the launch prompt, quotes other agents' identity lines, and can outgrow any fixed window. - Distinguish parallel invariant agents by their owned file, so per-file runs stop folding into a phantom (xN) relaunch row. - Coerce negative provider counts to zero: the agent path records usage uncoerced, and summed negatives rendered >100% cached shares. - Accept degraded diff-less Step 1 reports: validate diffLines + chunks, the pair every plan report carries, instead of check-coverage's stricter contract that refused them. - Pin every branch the second round proved unobservable: the exit code on all handler paths, total - prompt under both usage conventions, per-agent fault tolerance, the wall-minutes conversion, array-shaped usage, the mtime pre-filter and the event-level floor, human() rounding, per-condition plan validation, sort order against a lexical readdir, the zero-event skip, the --out per-stream archive contract, error messages naming their paths, truncation membership and folded-run counting, and the assistant-type filter. Every new assertion was mutation-probed: each mutant the review named now turns the suite red. * review: pipeline stages keep their own ledger rows The (×N) fold keyed on the label alone, and three legitimate multi-launch shapes shared one: a reverse-audit chunk auditor is launched with the same 'chunk N of M' identity as the Step 3B territory finder (five audit rounds folded into the finder's row — one agent where six pipeline stages ran), and repeat rounds of the findings roles carry their round OUTSIDE the backticks (every round folded as a phantom relaunch). labelOf now reads the stage from the audit brief's record key in the launch (audit chunk N (round K)) and the round from the identity LINE — never the whole launch, whose folded findings can quote a budget disclosure's own '(round N)' — so rounds are rows and only true relaunches and same-round verify shards fold. The (×N) comment now says what the marker means: N runs under one label. * fix(cli): annotate cost-ledger test helper to restore strict build (#8471) * fix(cli): anchor cost-ledger labels and harden broken-usage defenses (#8471) --------- Co-authored-by: verify <verify@local> Co-authored-by: qwen-code-dev-bot <qwen-code-dev@service.alibaba.com>
This commit is contained in:
Shaojin Wen
GitHub
parent
32e2741577
commit
4f79036a22
10 changed files with 1982 additions and 10 deletions
|
|
@ -57,6 +57,7 @@ describe('reviewCommand', () => {
|
|||
'script-lint',
|
||||
'resolve-anchors',
|
||||
'check-coverage',
|
||||
'cost-ledger',
|
||||
'presubmit',
|
||||
'test-efficacy',
|
||||
'test-plan',
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ import { submitCommand } from './review/submit.js';
|
|||
import { testEfficacyCommand } from './review/test-efficacy.js';
|
||||
import { testPlanCommand } from './review/test-plan.js';
|
||||
import { cleanupCommand } from './review/cleanup.js';
|
||||
import { costLedgerCommand } from './review/cost-ledger.js';
|
||||
import { runCommand } from './review/run.js';
|
||||
import { saveArtifactCommand } from './review/save-artifact.js';
|
||||
|
||||
|
|
@ -61,6 +62,7 @@ export const reviewCommand: CommandModule = {
|
|||
.command(scriptLintCommand)
|
||||
.command(resolveAnchorsCommand)
|
||||
.command(checkCoverageCommand)
|
||||
.command(costLedgerCommand)
|
||||
.command(presubmitCommand)
|
||||
.command(testEfficacyCommand)
|
||||
.command(testPlanCommand)
|
||||
|
|
@ -72,7 +74,7 @@ export const reviewCommand: CommandModule = {
|
|||
.command(cleanupCommand)
|
||||
.demandCommand(
|
||||
1,
|
||||
'Specify a subcommand: run, parse-args, fetch-pr, capture-local, plan-diff, pr-context, comment-status, load-rules, agent-prompt, build-test, base-tree, test-delta, drive, mock-provider, extract-step, script-lint, resolve-anchors, check-coverage, presubmit, test-efficacy, test-plan, findings, publish-assets, compose-review, save-artifact, submit, or cleanup.',
|
||||
'Specify a subcommand: run, parse-args, fetch-pr, capture-local, plan-diff, pr-context, comment-status, load-rules, agent-prompt, build-test, base-tree, test-delta, drive, mock-provider, extract-step, script-lint, resolve-anchors, check-coverage, cost-ledger, presubmit, test-efficacy, test-plan, findings, publish-assets, compose-review, save-artifact, submit, or cleanup.',
|
||||
)
|
||||
.version(false),
|
||||
handler: () => {
|
||||
|
|
|
|||
1314
packages/cli/src/commands/review/cost-ledger.test.ts
Normal file
1314
packages/cli/src/commands/review/cost-ledger.test.ts
Normal file
File diff suppressed because it is too large
Load diff
555
packages/cli/src/commands/review/cost-ledger.ts
Normal file
555
packages/cli/src/commands/review/cost-ledger.ts
Normal file
|
|
@ -0,0 +1,555 @@
|
|||
/**
|
||||
* @license
|
||||
* Copyright 2026 Qwen Team
|
||||
* SPDX-License-Identifier: Apache-2.0
|
||||
*/
|
||||
|
||||
// `qwen review cost-ledger`: what this review actually cost, from the
|
||||
// harness's own usage records.
|
||||
//
|
||||
// The number exists because it kept having to be excavated. A maintainer's
|
||||
// "0.21.3 was fine, 0.21.4 got slow" was settled only by replaying a whole
|
||||
// review under a telemetry exporter and hand-aggregating half a million
|
||||
// telemetry lines — hours of forensics for a question one printed table
|
||||
// answers. The same excavation found the money: a small +93/-48 PR at high
|
||||
// effort cost 523 model calls and 37.8M input tokens, 9.7M of them a repair
|
||||
// round redelivering prompts the agents had already acted on. Nobody chose
|
||||
// that spend; nobody could see it either.
|
||||
//
|
||||
// The data was on disk the whole time: every chat and subagent transcript
|
||||
// event carries `usageMetadata` (prompt / candidates / thoughts / cached
|
||||
// counts). This subcommand aggregates those records — the same records
|
||||
// `check-coverage` trusts for delivery, read from the same
|
||||
// environment-exported location — into per-stream totals. It is
|
||||
// **informational**: a ledger that cannot be computed prints why and exits 0,
|
||||
// because a review must never fail on its own accounting.
|
||||
//
|
||||
// A "model call" is an assistant record carrying `usageMetadata`; a turn
|
||||
// whose provider returned no usage is invisible, so call counts are a floor,
|
||||
// not an exact API-call tally.
|
||||
|
||||
import type { CommandModule } from 'yargs';
|
||||
import { mkdirSync, readFileSync, statSync, writeFileSync } from 'node:fs';
|
||||
import { dirname, join, resolve } from 'node:path';
|
||||
import { parseLineTolerant } from '@qwen-code/qwen-code-core';
|
||||
import {
|
||||
writeStdoutLineSafe,
|
||||
writeStderrLineSafe,
|
||||
} from '../../utils/stdioHelpers.js';
|
||||
import {
|
||||
transcriptPaths,
|
||||
listAgentTranscriptFiles,
|
||||
TranscriptsUnavailableError,
|
||||
textOf,
|
||||
} from './lib/transcripts.js';
|
||||
import { CHUNK_RE } from './lib/coverage.js';
|
||||
|
||||
interface CostLedgerArgs {
|
||||
plan: string;
|
||||
out?: string;
|
||||
}
|
||||
|
||||
interface StreamCost {
|
||||
/** `main` for the orchestrator session, else the agent file's id. */
|
||||
id: string;
|
||||
/** Human label: the role parsed from the launch prompt when one is found. */
|
||||
label: string;
|
||||
calls: number;
|
||||
inputTokens: number;
|
||||
cachedTokens: number;
|
||||
outputTokens: number;
|
||||
thoughtsTokens: number;
|
||||
firstAt: string | null;
|
||||
lastAt: string | null;
|
||||
}
|
||||
|
||||
interface Ledger {
|
||||
totals: Omit<StreamCost, 'id' | 'label'> & { wallSeconds: number };
|
||||
main: StreamCost | null;
|
||||
agents: StreamCost[];
|
||||
}
|
||||
|
||||
interface UsageEvent {
|
||||
timestampMs: number;
|
||||
timestamp: string;
|
||||
input: number;
|
||||
cached: number;
|
||||
output: number;
|
||||
thoughts: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* One read of a transcript: its usage-bearing assistant events, floor-filtered,
|
||||
* plus the launch prompt the label comes from.
|
||||
*
|
||||
* The launch prompt is the first `user` record's text — the same anchor
|
||||
* `parseTranscript` in lib/transcripts.ts uses — never a raw byte slice of the
|
||||
* file: a fork agent's transcript opens with an `agent_bootstrap` system
|
||||
* record carrying the entire inherited conversation, which can quote other
|
||||
* agents' identity lines and outgrow any fixed head window.
|
||||
*
|
||||
* Read failures throw; the caller decides what they mean — for the chat file,
|
||||
* "the ledger cannot be computed"; for one agent file, "that agent is lost".
|
||||
*/
|
||||
function readUsage(
|
||||
file: string,
|
||||
floorMs: number,
|
||||
): { events: UsageEvent[]; launch: string } {
|
||||
const raw = readFileSync(file, 'utf8');
|
||||
const events: UsageEvent[] = [];
|
||||
let launch = '';
|
||||
for (const line of raw.split('\n')) {
|
||||
if (line.trim() === '') continue;
|
||||
// parseLineTolerant recovers the `}{`-glued records an interrupted append
|
||||
// leaves behind — the documented corruption shape of these incrementally
|
||||
// flushed files — and drops non-object lines, which a bare JSON.parse
|
||||
// parses happily (`null`, `42`) only to trip the property reads below.
|
||||
for (const rec of parseLineTolerant<Record<string, unknown>>(line, file)) {
|
||||
if (launch === '' && rec['type'] === 'user') {
|
||||
launch = textOf(rec);
|
||||
}
|
||||
if (rec['type'] !== 'assistant') continue;
|
||||
const usage = rec['usageMetadata'];
|
||||
if (usage === null || typeof usage !== 'object') continue;
|
||||
if (Array.isArray(usage)) continue;
|
||||
const u = usage as Record<string, unknown>;
|
||||
const ts = rec['timestamp'];
|
||||
if (typeof ts !== 'string') continue;
|
||||
const tsMs = Date.parse(ts);
|
||||
// The chat file spans the whole session, not the review: a `/review`
|
||||
// launched an hour into a working session would otherwise bill that hour's
|
||||
// conversation to the review. The plan's own mtime marks the review start
|
||||
// — the same floor `check-coverage` applies to transcripts.
|
||||
if (!Number.isFinite(tsMs) || tsMs < floorMs) continue;
|
||||
// Finite ≥ 0, else null: the main loop coerces broken-proxy usage
|
||||
// (negative or NaN counts) before recording, but the agent path records
|
||||
// raw provider usage, and each consumer below picks its own fallback
|
||||
// for a count that did not survive the provider.
|
||||
const rawCount = (k: string): number | null => {
|
||||
const v = u[k];
|
||||
return typeof v === 'number' && Number.isFinite(v) && v >= 0 ? v : null;
|
||||
};
|
||||
const rawPrompt = rawCount('promptTokenCount');
|
||||
const rawTotal = rawCount('totalTokenCount');
|
||||
const prompt = rawPrompt ?? 0;
|
||||
const candidates = rawCount('candidatesTokenCount') ?? 0;
|
||||
events.push({
|
||||
timestampMs: tsMs,
|
||||
timestamp: ts,
|
||||
input: prompt,
|
||||
// Cached is part of the prompt it is reported with; a broken proxy
|
||||
// can report the pair inverted, and the inversion would otherwise
|
||||
// land in BOTH the rendered share and the archived --out JSON.
|
||||
cached: Math.min(rawCount('cachedContentTokenCount') ?? 0, prompt),
|
||||
// `total − prompt` is the output including thinking under BOTH usage
|
||||
// conventions — reasoning inside candidates, and thoughts disjoint
|
||||
// from them — the same derivation tokenEstimation uses. Candidates
|
||||
// alone is the fallback when the provider reported no total; it stays
|
||||
// correct for the providers this CLI converts, which clamp thoughts
|
||||
// inside candidates. Derive it only when BOTH operands survived the
|
||||
// provider intact: a mixed-sign record coerces prompt to 0 while
|
||||
// keeping a positive total, and the subtraction would bill the
|
||||
// call's whole total as output.
|
||||
output:
|
||||
rawPrompt !== null && rawTotal !== null && rawTotal > rawPrompt
|
||||
? rawTotal - rawPrompt
|
||||
: candidates,
|
||||
thoughts: rawCount('thoughtsTokenCount') ?? 0,
|
||||
});
|
||||
}
|
||||
}
|
||||
return { events, launch };
|
||||
}
|
||||
|
||||
/**
|
||||
* An auditor's own brief line, exactly as `buildRoleLaunchPrompt` prints it.
|
||||
* A bare `reverse-audit--chunk-N--round-M--<hex>` path mentioned anywhere is
|
||||
* NOT this: folded findings quote other rounds' brief paths routinely.
|
||||
*/
|
||||
const AUDIT_BRIEF_RE =
|
||||
/read_file\(file_path="[^"]*reverse-audit--chunk-(\d+)--round-(\d+)--[0-9a-f][^"]*\.brief\.md"\)/g;
|
||||
|
||||
/** A role label out of the launch prompt, else the fallback. */
|
||||
function labelOf(launch: string, fallback: string): string {
|
||||
// Every identity-based parse stays on the identity LINE, and nothing runs
|
||||
// at all without one at the head. The folded findings below it can quote
|
||||
// budget disclosures' "(round N)", other agents' `Your file:` lines, ledger
|
||||
// rows, and `You are review agent` lines — a whole-launch match would hand
|
||||
// the quoted label to the agent carrying the quote. A prompt with no
|
||||
// identity line is an older harness's, or an agent this review never
|
||||
// launched (the session's transcript dir also holds nested subagents): its
|
||||
// free text can name anything, so keep the one label it owns — the file id.
|
||||
const nl = launch.indexOf('\n');
|
||||
const identity = nl === -1 ? launch : launch.slice(0, nl);
|
||||
if (!identity.startsWith('You are review agent `')) return fallback;
|
||||
// A reverse-audit chunk auditor shares its launch shape with the territory
|
||||
// finder; only its brief path carries the stage and the round — without
|
||||
// it, five audit rounds fold into one row and the ledger reports one agent
|
||||
// where six pipeline stages ran. Match the agent's OWN brief line, never a
|
||||
// quoted mention: the folds sit ABOVE the agent's own brief line
|
||||
// (foldFindings folds them there), so the last brief-shaped read_file in
|
||||
// the launch is the agent's own.
|
||||
let auditChunk: RegExpExecArray | null = null;
|
||||
for (const m of launch.matchAll(AUDIT_BRIEF_RE)) auditChunk = m;
|
||||
if (auditChunk) {
|
||||
return `audit chunk ${auditChunk[1]} (round ${auditChunk[2]})`;
|
||||
}
|
||||
const role = /^You are review agent `([^`]+)`/.exec(identity);
|
||||
if (!role) return fallback;
|
||||
const round = /\(round (\d+)\)/.exec(identity);
|
||||
const chunk = CHUNK_RE.exec(role[1]);
|
||||
// A chunk role is `chunk N of M`; prefixing it with "agent" would read as
|
||||
// a malformed role, so resolve it through the same regex coverage uses.
|
||||
if (chunk) return `chunk ${chunk[1]}`;
|
||||
if (round) {
|
||||
// Shards of one verify round carry the same label and fold; distinct
|
||||
// rounds — verify and reverse-audit alike — are distinct rows.
|
||||
return `agent ${role[1]} (round ${round[1]})`;
|
||||
}
|
||||
// An invariant role launches once PER heavy file. The role alone would
|
||||
// fold those parallel runs into one (×N) row — the marker reserved for
|
||||
// relaunches — and lose the per-file breakdown. The identity line names
|
||||
// the owned file; the FULL path is the distinguisher, because a monorepo
|
||||
// routinely holds same-basename files in different packages.
|
||||
const file = /Your file: `([^`]+)`/.exec(identity);
|
||||
if (file) return `agent ${role[1]} (${file[1]})`;
|
||||
return `agent ${role[1]}`;
|
||||
}
|
||||
|
||||
function foldEvents(
|
||||
id: string,
|
||||
label: string,
|
||||
events: UsageEvent[],
|
||||
): StreamCost {
|
||||
const s: StreamCost = {
|
||||
id,
|
||||
label,
|
||||
calls: 0,
|
||||
inputTokens: 0,
|
||||
cachedTokens: 0,
|
||||
outputTokens: 0,
|
||||
thoughtsTokens: 0,
|
||||
firstAt: null,
|
||||
lastAt: null,
|
||||
};
|
||||
let firstMs = Number.POSITIVE_INFINITY;
|
||||
let lastMs = Number.NEGATIVE_INFINITY;
|
||||
for (const e of events) {
|
||||
s.calls += 1;
|
||||
s.inputTokens += e.input;
|
||||
s.cachedTokens += e.cached;
|
||||
s.outputTokens += e.output;
|
||||
s.thoughtsTokens += e.thoughts;
|
||||
if (e.timestampMs < firstMs) {
|
||||
firstMs = e.timestampMs;
|
||||
s.firstAt = e.timestamp;
|
||||
}
|
||||
if (e.timestampMs > lastMs) {
|
||||
lastMs = e.timestampMs;
|
||||
s.lastAt = e.timestamp;
|
||||
}
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
/** 12_345_678 → "12.3M"; 45_600 → "46k"; 890 → "890". */
|
||||
function human(n: number): string {
|
||||
// 999_500 rounds to 1000k; from there up, render in M so it reads "1.0M".
|
||||
// The same boundary at the B tier keeps 1.5e9 from reading "1500.0M".
|
||||
if (n >= 999_500_000) return `${(n / 1_000_000_000).toFixed(1)}B`;
|
||||
if (n >= 999_500) return `${(n / 1_000_000).toFixed(1)}M`;
|
||||
if (n >= 1_000) return `${Math.round(n / 1_000)}k`;
|
||||
return String(n);
|
||||
}
|
||||
|
||||
/** "1 call", "2 calls" — the rendered block is archived verbatim. */
|
||||
const plural = (n: number, word: string): string =>
|
||||
`${n} ${word}${n === 1 ? '' : 's'}`;
|
||||
|
||||
/**
|
||||
* The plan's mtime is the billing floor. Validate the file IS a Step 1 plan
|
||||
* report before trusting that mtime: a wrong-but-existing file (the findings
|
||||
* JSON, the report written minutes earlier) would move the floor silently in
|
||||
* either direction. The shape checked is the pair every Step 1 report carries
|
||||
* — `diffLines` and `chunks` — not `check-coverage`'s stricter contract: a
|
||||
* degraded capture (unresolvable merge base, the tiling fallback) writes
|
||||
* `diffPathAbsolute: null` with `chunks: []`, and that report's mtime is
|
||||
* still exactly the floor this ledger needs.
|
||||
*/
|
||||
function planFloorMs(planPath: string): number {
|
||||
let raw: string;
|
||||
let floorMs: number;
|
||||
try {
|
||||
raw = readFileSync(planPath, 'utf8');
|
||||
floorMs = statSync(planPath).mtimeMs;
|
||||
} catch (err) {
|
||||
throw new Error(
|
||||
`could not read the plan report ${planPath}: ${(err as Error).message}`,
|
||||
);
|
||||
}
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(raw);
|
||||
} catch {
|
||||
parsed = null;
|
||||
}
|
||||
const plan =
|
||||
parsed !== null && typeof parsed === 'object'
|
||||
? (parsed as Record<string, unknown>)
|
||||
: undefined;
|
||||
if (
|
||||
typeof plan?.['diffLines'] !== 'number' ||
|
||||
!Array.isArray(plan?.['chunks'])
|
||||
) {
|
||||
throw new Error(`not a review plan report: ${planPath}`);
|
||||
}
|
||||
return floorMs;
|
||||
}
|
||||
|
||||
export function computeLedger(
|
||||
planPath: string,
|
||||
env: NodeJS.ProcessEnv = process.env,
|
||||
): Ledger {
|
||||
const floorMs = planFloorMs(planPath);
|
||||
const { projectDir, sessionId, dir } = transcriptPaths(env);
|
||||
|
||||
const chatFile = join(projectDir, 'chats', `${sessionId}.jsonl`);
|
||||
let mainEvents: UsageEvent[];
|
||||
try {
|
||||
mainEvents = readUsage(chatFile, floorMs).events;
|
||||
} catch (err) {
|
||||
// The plan's existence proves the main loop ran: a missing or unreadable
|
||||
// chat file is an infrastructure fact (chat recording off, or a fault),
|
||||
// not a verdict that the loop made no calls. Agents-only totals would
|
||||
// read as the review's whole cost, so say the ledger cannot be computed.
|
||||
throw new Error(
|
||||
`could not read the chat transcript ${chatFile}: ` +
|
||||
`${(err as Error).message}`,
|
||||
);
|
||||
}
|
||||
const main =
|
||||
mainEvents.length > 0 ? foldEvents('main', 'main loop', mainEvents) : null;
|
||||
|
||||
let files: string[];
|
||||
try {
|
||||
files = listAgentTranscriptFiles(dir);
|
||||
} catch (err) {
|
||||
if ((err as NodeJS.ErrnoException).code === 'ENOENT') {
|
||||
// No subagent dir is a real state (a low-effort review runs no agents);
|
||||
// the ledger reports what exists.
|
||||
files = [];
|
||||
} else {
|
||||
// EACCES/EIO/ENOTDIR are not "no agents": main-loop-only totals would
|
||||
// read as the complete ledger.
|
||||
throw new Error(
|
||||
`could not list the subagent transcripts at ${dir}: ` +
|
||||
`${(err as Error).message}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const agents: StreamCost[] = [];
|
||||
const agentEvents: UsageEvent[] = [];
|
||||
for (const f of files) {
|
||||
const full = join(dir, f);
|
||||
let mtimeMs: number;
|
||||
try {
|
||||
mtimeMs = statSync(full).mtimeMs;
|
||||
} catch {
|
||||
continue; // Gone between listing and stat.
|
||||
}
|
||||
// The transcript dir is session-scoped and never pruned: files from
|
||||
// earlier reviews this session predate the floor, and a file whose last
|
||||
// write predates it cannot hold an above-floor record — the same
|
||||
// membership test `readTranscripts` applies. Skip it without opening.
|
||||
if (mtimeMs < floorMs) continue;
|
||||
let read: { events: UsageEvent[]; launch: string };
|
||||
try {
|
||||
read = readUsage(full, floorMs);
|
||||
} catch {
|
||||
continue; // This agent's record is lost; the rest still count.
|
||||
}
|
||||
if (read.events.length === 0) continue;
|
||||
const id = f.replace(/^agent-/, '').replace(/\.jsonl$/, '');
|
||||
agents.push(foldEvents(id, labelOf(read.launch, id), read.events));
|
||||
agentEvents.push(...read.events);
|
||||
}
|
||||
agents.sort((a, b) => b.inputTokens - a.inputTokens);
|
||||
|
||||
// A present-but-empty window is not a lighter version of a missing one.
|
||||
// The recorder pre-creates the chat file and degrades permanently if its
|
||||
// first append fails, so "exists, yet no above-floor records" — with or
|
||||
// without agents — is the same infrastructure fact as "unreadable": a live
|
||||
// review holds at least one above-floor main-loop record, because the plan
|
||||
// itself is a main-loop write. Rendering a zero ledger from this shape
|
||||
// would be diffed against real numbers — exactly the fabrication the
|
||||
// refusal above names.
|
||||
if (mainEvents.length === 0) {
|
||||
throw new Error(
|
||||
`could not read the chat transcript ${chatFile}: no main-loop usage ` +
|
||||
'records at or after the plan',
|
||||
);
|
||||
}
|
||||
|
||||
// The same events the per-stream rows fold, folded once more — one
|
||||
// accumulator, so a new usage counter cannot land in the rows and miss the
|
||||
// headline.
|
||||
const totals = foldEvents('totals', 'totals', [
|
||||
...mainEvents,
|
||||
...agentEvents,
|
||||
]);
|
||||
const wallSeconds =
|
||||
totals.firstAt !== null && totals.lastAt !== null
|
||||
? Math.max(
|
||||
0,
|
||||
Math.round(
|
||||
(Date.parse(totals.lastAt) - Date.parse(totals.firstAt)) / 1000,
|
||||
),
|
||||
)
|
||||
: 0;
|
||||
|
||||
const { id: _i, label: _l, ...totalsRest } = totals;
|
||||
return { totals: { ...totalsRest, wallSeconds }, main, agents };
|
||||
}
|
||||
|
||||
/** The printed block: one summary line, the main loop, the top consumers. */
|
||||
export function renderLedger(ledger: Ledger): string {
|
||||
const t = ledger.totals;
|
||||
const cachedPct =
|
||||
t.inputTokens > 0 ? Math.round((t.cachedTokens / t.inputTokens) * 100) : 0;
|
||||
const lines: string[] = [];
|
||||
lines.push(
|
||||
`Cost ledger: ${plural(t.calls, 'model call')} · ` +
|
||||
`${human(t.inputTokens)} input (${cachedPct}% cached) · ` +
|
||||
`${human(t.outputTokens)} output (${human(t.thoughtsTokens)} thinking) · ` +
|
||||
`${Math.round(t.wallSeconds / 60)} min wall`,
|
||||
);
|
||||
if (ledger.main !== null) {
|
||||
const m = ledger.main;
|
||||
lines.push(
|
||||
` main loop: ${plural(m.calls, 'call')} · ${human(m.inputTokens)} in · ` +
|
||||
`${human(m.outputTokens)} out`,
|
||||
);
|
||||
}
|
||||
if (ledger.agents.length > 0) {
|
||||
// Equal labels fold into one row marked (×N): a relaunched agent keeps
|
||||
// its label, and verify shards deliberately share one — the marker reads
|
||||
// "N runs under this label", and the repair round this ledger exists to
|
||||
// surface becomes visible, not merely present. Rounds and audit chunks
|
||||
// do NOT share labels (labelOf carries their stage and round), so a
|
||||
// five-round audit is five rows, not a phantom ×5 relaunch.
|
||||
const rows: Array<{
|
||||
label: string;
|
||||
calls: number;
|
||||
inputTokens: number;
|
||||
outputTokens: number;
|
||||
count: number;
|
||||
}> = [];
|
||||
for (const a of ledger.agents) {
|
||||
const row = rows.find((r) => r.label === a.label);
|
||||
if (row) {
|
||||
row.calls += a.calls;
|
||||
row.inputTokens += a.inputTokens;
|
||||
row.outputTokens += a.outputTokens;
|
||||
row.count += 1;
|
||||
} else {
|
||||
rows.push({
|
||||
label: a.label,
|
||||
calls: a.calls,
|
||||
inputTokens: a.inputTokens,
|
||||
outputTokens: a.outputTokens,
|
||||
count: 1,
|
||||
});
|
||||
}
|
||||
}
|
||||
// Rank by the folded total, not the first member's share: a doubled run
|
||||
// must not be truncated away by the half that sorted lower.
|
||||
rows.sort((a, b) => b.inputTokens - a.inputTokens);
|
||||
lines.push(` agent runs: ${ledger.agents.length}`);
|
||||
for (const r of rows.slice(0, 8)) {
|
||||
const label = r.count > 1 ? `${r.label} (×${r.count})` : r.label;
|
||||
lines.push(
|
||||
` ${label}: ${plural(r.calls, 'call')} · ` +
|
||||
`${human(r.inputTokens)} in · ${human(r.outputTokens)} out`,
|
||||
);
|
||||
}
|
||||
if (rows.length > 8) {
|
||||
const rest = rows.slice(8);
|
||||
const restIn = rest.reduce((n, r) => n + r.inputTokens, 0);
|
||||
const restAgents = rest.reduce((n, r) => n + r.count, 0);
|
||||
lines.push(
|
||||
` …and ${plural(restAgents, 'more agent')} · ` +
|
||||
`${human(restIn)} in combined`,
|
||||
);
|
||||
}
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
function runCostLedger(args: CostLedgerArgs): void {
|
||||
// EPIPE arrives two ways when the reader goes away (`qwen … | head`, a
|
||||
// daemon's closed redirect): a sync throw out of the write, and an async
|
||||
// 'error' event on the pipe. The safe writers catch the first; destroy the
|
||||
// stream on the second — the convention nonInteractiveCli uses — and
|
||||
// detach both listeners on exit. A review must never fail on its own
|
||||
// accounting, including the accounting of a reader that left.
|
||||
const stdoutErrorHandler = (err: NodeJS.ErrnoException): void => {
|
||||
if (err.code === 'EPIPE') process.stdout.destroy();
|
||||
};
|
||||
const stderrErrorHandler = (err: NodeJS.ErrnoException): void => {
|
||||
if (err.code === 'EPIPE') process.stderr.destroy();
|
||||
};
|
||||
process.stdout.on('error', stdoutErrorHandler);
|
||||
process.stderr.on('error', stderrErrorHandler);
|
||||
try {
|
||||
let ledger: Ledger;
|
||||
try {
|
||||
ledger = computeLedger(args.plan, process.env);
|
||||
} catch (err) {
|
||||
// Informational, always: a review must never fail on its own accounting.
|
||||
const why =
|
||||
err instanceof TranscriptsUnavailableError
|
||||
? err.message
|
||||
: (err as Error).message;
|
||||
writeStderrLineSafe(`cost-ledger unavailable — ${why}`);
|
||||
return;
|
||||
}
|
||||
if (args.out !== undefined && args.out.length > 0) {
|
||||
// A failed archive write degrades to a warning: the ledger was computed,
|
||||
// and the exit code must stay 0 either way.
|
||||
try {
|
||||
mkdirSync(dirname(resolve(args.out)), { recursive: true });
|
||||
writeFileSync(args.out, JSON.stringify(ledger, null, 2));
|
||||
} catch (err) {
|
||||
writeStderrLineSafe(
|
||||
`cost-ledger: could not write ${args.out} — ${(err as Error).message}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
writeStdoutLineSafe(renderLedger(ledger));
|
||||
} finally {
|
||||
process.stdout.removeListener('error', stdoutErrorHandler);
|
||||
process.stderr.removeListener('error', stderrErrorHandler);
|
||||
}
|
||||
}
|
||||
|
||||
export const costLedgerCommand: CommandModule = {
|
||||
command: 'cost-ledger',
|
||||
describe:
|
||||
"Aggregate this review's model-call cost from the harness's usage records",
|
||||
builder: (yargs) =>
|
||||
yargs
|
||||
.option('plan', {
|
||||
type: 'string',
|
||||
demandOption: true,
|
||||
describe:
|
||||
'The plan report from Step 1 — its mtime marks the review start',
|
||||
})
|
||||
.option('out', {
|
||||
type: 'string',
|
||||
describe: 'Also write the full ledger as JSON to this path',
|
||||
}),
|
||||
handler: (args) => {
|
||||
runCostLedger(args as unknown as CostLedgerArgs);
|
||||
},
|
||||
};
|
||||
|
|
@ -235,7 +235,7 @@ function readPlan(path: string): { plan: Plan; mtimeMs: number } {
|
|||
}
|
||||
|
||||
/** `chunk 13 of 25` — written into the prompt by `agent-prompt`, in code. */
|
||||
const CHUNK_RE = /\bchunk\s+(\d+)\s+of\s+\d+\b/i;
|
||||
export const CHUNK_RE = /\bchunk\s+(\d+)\s+of\s+\d+\b/i;
|
||||
|
||||
/** The chunk this agent owns, when it was launched to own one. */
|
||||
function assignedChunk(rec: AgentRecord): number | null {
|
||||
|
|
|
|||
|
|
@ -67,6 +67,39 @@ describe('readTranscripts — defensive parsing', () => {
|
|||
expect(readTranscripts(undefined, ENV)).toEqual([]);
|
||||
});
|
||||
|
||||
it('skips the harness sidecar files beside a transcript', () => {
|
||||
// The harness writes sibling files per agent into this dir —
|
||||
// agent-transcript.ts writes `agent-<id>.meta.json` via writeAgentMeta,
|
||||
// and a meta carries an `agentId` key. Admitted by the filter, it would
|
||||
// parse to a phantom zero-tool-call AgentRecord: it is the
|
||||
// `.endsWith('.jsonl')` filter, not parseTranscript, that keeps it out.
|
||||
file(
|
||||
'agent-a1.jsonl',
|
||||
JSON.stringify({
|
||||
agentId: 'a1',
|
||||
agentName: 'general-purpose',
|
||||
type: 'user',
|
||||
message: { role: 'user', parts: [{ text: 'chunk 1 of 1' }] },
|
||||
}) + '\n',
|
||||
);
|
||||
file(
|
||||
'agent-a1.meta.json',
|
||||
JSON.stringify({
|
||||
agentId: 'a1',
|
||||
agentType: 'general-purpose',
|
||||
description: 'dimension 1',
|
||||
parentSessionId: 'S1',
|
||||
parentAgentId: null,
|
||||
createdAt: '2026-08-03T10:06:00.000Z',
|
||||
status: 'completed',
|
||||
}),
|
||||
);
|
||||
file('agent-a1.jsonl.stream', 'streaming text, not jsonl records');
|
||||
const recs = readTranscripts(undefined, ENV);
|
||||
expect(recs).toHaveLength(1);
|
||||
expect(recs[0].agentId).toBe('a1');
|
||||
});
|
||||
|
||||
it('skips an empty transcript file', () => {
|
||||
file('agent-empty.jsonl', '');
|
||||
expect(readTranscripts(undefined, ENV)).toEqual([]);
|
||||
|
|
|
|||
|
|
@ -87,16 +87,22 @@ export interface AgentRecord {
|
|||
export class TranscriptsUnavailableError extends Error {}
|
||||
|
||||
/**
|
||||
* Where this session's subagent transcripts live.
|
||||
* The environment this module reads, validated once and returned together.
|
||||
*
|
||||
* Both halves come from the environment the CLI exported, never from an argument:
|
||||
* a path the model can choose is a path the model can point somewhere flattering.
|
||||
* `QWEN_CODE_PROJECT_DIR` exists because the project dir is keyed on the session's
|
||||
* *launch* cwd, and this subcommand may well be running inside a PR worktree the
|
||||
* skill `cd`-ed into — recomputing it from `process.cwd()` yields a directory that
|
||||
* never existed.
|
||||
* never existed. Callers that need both halves (the chat file lives beside the
|
||||
* subagent dir) take them here rather than re-reading the env after `transcriptDir`
|
||||
* validated it.
|
||||
*/
|
||||
export function transcriptDir(env: NodeJS.ProcessEnv = process.env): string {
|
||||
export function transcriptPaths(env: NodeJS.ProcessEnv = process.env): {
|
||||
projectDir: string;
|
||||
sessionId: string;
|
||||
dir: string;
|
||||
} {
|
||||
const projectDir = env['QWEN_CODE_PROJECT_DIR']?.trim();
|
||||
const sessionId = env['QWEN_CODE_SESSION_ID']?.trim();
|
||||
if (!projectDir || !sessionId) {
|
||||
|
|
@ -105,11 +111,20 @@ export function transcriptDir(env: NodeJS.ProcessEnv = process.env): string {
|
|||
"this run cannot find the harness's record of what its agents did",
|
||||
);
|
||||
}
|
||||
return join(projectDir, 'subagents', sessionId);
|
||||
return {
|
||||
projectDir,
|
||||
sessionId,
|
||||
dir: join(projectDir, 'subagents', sessionId),
|
||||
};
|
||||
}
|
||||
|
||||
/** Where this session's subagent transcripts live. */
|
||||
export function transcriptDir(env: NodeJS.ProcessEnv = process.env): string {
|
||||
return transcriptPaths(env).dir;
|
||||
}
|
||||
|
||||
/** Text out of a record's message parts. */
|
||||
function textOf(rec: Record<string, unknown>): string {
|
||||
export function textOf(rec: Record<string, unknown>): string {
|
||||
const msg = rec['message'] as { parts?: unknown } | undefined;
|
||||
const parts = Array.isArray(msg?.parts) ? msg.parts : [];
|
||||
return parts
|
||||
|
|
@ -306,6 +321,21 @@ function parseTranscript(file: string, diffPath?: string): AgentRecord | null {
|
|||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The session's subagent transcript files, one listing every reader shares.
|
||||
*
|
||||
* The coverage gate and the cost ledger both claim to read "the same records",
|
||||
* and the harness writes sibling file kinds per agent (`.meta.json`,
|
||||
* `.jsonl.stream`) with a generalized `<kind>-<id>.jsonl` namespace planned —
|
||||
* so the definition of "which files are transcripts" lives here, once, not in
|
||||
* each reader's own filter. Throws on any readdir failure; what the caller
|
||||
* does with that (name the fault, or treat an absent dir as "no agents") is
|
||||
* its decision.
|
||||
*/
|
||||
export function listAgentTranscriptFiles(dir: string): string[] {
|
||||
return readdirSync(dir).filter((name) => name.endsWith('.jsonl'));
|
||||
}
|
||||
|
||||
/**
|
||||
* Every subagent this session launched, as the harness recorded it.
|
||||
*
|
||||
|
|
@ -323,7 +353,7 @@ export function readTranscripts(
|
|||
const dir = transcriptDir(env);
|
||||
let names: string[];
|
||||
try {
|
||||
names = readdirSync(dir);
|
||||
names = listAgentTranscriptFiles(dir);
|
||||
} catch (err) {
|
||||
// No directory at all is an *infrastructure* fact, not a verdict about the
|
||||
// agents. Conflating the two would let a read-only HOME or a full disk read
|
||||
|
|
@ -337,7 +367,6 @@ export function readTranscripts(
|
|||
|
||||
const out: AgentRecord[] = [];
|
||||
for (const name of names) {
|
||||
if (!name.endsWith('.jsonl')) continue;
|
||||
const rec = parseTranscript(join(dir, name), diffPath);
|
||||
if (!rec) continue;
|
||||
if (since !== undefined && rec.mtimeMs < since) continue;
|
||||
|
|
|
|||
|
|
@ -5,7 +5,11 @@
|
|||
*/
|
||||
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
import { writeStderrLine, writeStderrLineSafe } from './stdioHelpers.js';
|
||||
import {
|
||||
writeStderrLine,
|
||||
writeStderrLineSafe,
|
||||
writeStdoutLineSafe,
|
||||
} from './stdioHelpers.js';
|
||||
|
||||
afterEach(() => vi.restoreAllMocks());
|
||||
|
||||
|
|
@ -50,3 +54,21 @@ describe('writeStderrLineSafe', () => {
|
|||
expect(() => writeStderrLineSafe('boom')).not.toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe('writeStdoutLineSafe', () => {
|
||||
it('writes with a trailing newline when stdout is healthy', () => {
|
||||
const write = vi.spyOn(process.stdout, 'write').mockReturnValue(true);
|
||||
|
||||
writeStdoutLineSafe('hello');
|
||||
|
||||
expect(write).toHaveBeenCalledWith('hello\n');
|
||||
});
|
||||
|
||||
it('swallows EPIPE instead of taking the caller down with it', () => {
|
||||
vi.spyOn(process.stdout, 'write').mockImplementation(() => {
|
||||
throw Object.assign(new Error('write EPIPE'), { code: 'EPIPE' });
|
||||
});
|
||||
|
||||
expect(() => writeStdoutLineSafe('boom')).not.toThrow();
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -32,6 +32,21 @@ export const writeStderrLine = (message: string): void => {
|
|||
process.stderr.write(message.endsWith('\n') ? message : `${message}\n`);
|
||||
};
|
||||
|
||||
/**
|
||||
* `writeStdoutLine` that cannot throw.
|
||||
*
|
||||
* Same contract as `writeStderrLineSafe`: use it where the write is
|
||||
* incidental to the work in hand — an informational block whose reader
|
||||
* going away (`qwen … | head`) must not fail the command.
|
||||
*/
|
||||
export const writeStdoutLineSafe = (message: string): void => {
|
||||
try {
|
||||
writeStdoutLine(message);
|
||||
} catch {
|
||||
// stdout is gone. Whatever this line had to say, its reader left.
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* `writeStderrLine` that cannot throw.
|
||||
*
|
||||
|
|
|
|||
|
|
@ -1112,6 +1112,7 @@ Report content should include:
|
|||
- All findings with verification status. Read them out of the findings artifact `qwen review findings` wrote (`.qwen/tmp/qwen-review-{target}-findings.json`) rather than re-typing them from the terminal — a third transcription of the same list is a third chance for a severity to drift, which has happened inside a single review.
|
||||
- **Per-finding outcomes, when Step 6B ran** — `fixed` / `skipped` / `no_change_needed`, with the reason for every `skipped`. The artifact already carries them; a `--fix` run whose archive does not say which findings were applied is a report that reads as if all of them were.
|
||||
- Verdict (high and medium effort — a low quick pass claims none; a medium verdict never exceeds Comment, since it runs no reverse audit — see Step 5)
|
||||
- **The cost ledger — run it, do not compute it.** `"${QWEN_CODE_CLI:-qwen}" review cost-ledger --plan <the plan report from Step 1> --out .qwen/reviews/<report>-cost-ledger.json` aggregates the model calls the harness recorded for this review — the main loop and each agent, with input / cached / output / thinking token counts and wall time — from the harness's own usage records, the same records the coverage gate trusts. The window is bounded: it starts at the plan's mtime, and the ledger runs at this step, so the pre-plan bootstrap turns and the composition after this snapshot are not captured, and side queries such as chat compression leave no usage records to capture at all. Paste its printed block into the report verbatim, and relay the first line in the terminal summary. The printed block lists only the eight biggest agents; the `--out` JSON keeps every one, so the diffable record survives in full (worktree mode: resolve `--out` against the main project directory, like the report itself). If it prints `cost-ledger unavailable`, note that instead — it is informational and never blocks a review. Why it is in the archive: a "this version got slower" report is unanswerable from memory, and the one time it was answered properly took hours of telemetry forensics to find a repair round that had silently doubled a run. The ledger makes the next such question a diff of two saved reports.
|
||||
|
||||
**The report's verdict is not yours to type.** `compose-review` printed the exact `Verdict:` line in Step 6 and persisted the same line as `verdictLine` inside `.qwen/tmp/qwen-review-{target}-composed.json` — copy either, verbatim. Do not reconstruct it from `event` + `cappedBy`: a presubmit downgrade also depends on fields that pair does not carry, and a rebuilt line can differ from the computed one. (And not `$(jq …)`: a `jq` binary is not guaranteed on the host, and a substitution that fails leaves the archived verdict blank or literal — worse than absent, because it looks written.)
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue