mirror of
https://github.com/AgentSeal/codeburn.git
synced 2026-08-25 08:22:42 +00:00
fix(codex): attribute MCP + skills from the exec custom tool and item model
The `mcp-cli call <server> <tool>` matcher from #656 only ran on `function_call` arguments. Codex records the same exec two other ways: the custom-tool transport carries the shell tool as a `custom_tool_call` whose payload is an `input` program (no `arguments` at all), and the item model repeats a finished command as `event_msg`/`item_completed` with a `CommandExecution` item holding an argv `command`. Both reach the Bash counter and neither reached the matcher, so a CLI-wrapped MCP call stayed missing from the MCP breakdown (#478). All three shapes now feed one classification pipeline. The same pipeline learns skills: Codex has no skill tool, so a skill load is a shell read of the skill's SKILL.md, which landed entirely under Bash. A read counts only when the command segment starts with a file-reading binary and the path ends in `<name>/SKILL.md`; the skill is `<name>`, matching pi's key (#588) and the Claude parser's vocabulary. A grep near a SKILL.md stays Bash. Attribution only: the item-model path adds no tool of its own, and a command carried by both shapes is attributed once (per-name claim ledger). Cost, tokens, calls and daily rollups cannot move. CODEX_CACHE_VERSION 13 -> 14 and the codex parse version both move: both layers store tools/toolSequence/skills verbatim, and on a warm cache the fix is otherwise invisible (verified by reverting the bumps: skills stay empty). DAILY_CACHE_VERSION is deliberately unchanged - the daily cache holds no tool/mcp/skill breakdown, and turn category cannot change because every added attribution rides on a call that already has Bash.
This commit is contained in:
parent
a9b7670f2f
commit
1c2ac6e8a4
5 changed files with 198 additions and 15 deletions
|
|
@ -38,6 +38,7 @@
|
|||
- **The resident `codeburn serve` child.** The first real panel request is also the cache warm-up, so startup never runs an artificial warm-up query beside a duplicate one-shot child; each served command carries its own read-only option allowlist, and anything outside it falls back to a normal spawn; the child exits when its stdin closes, so it can never outlive the app. Requests whose response exceeds the 16 MiB frame limit still replace the child, but that deliberate kill no longer spends the resident's unexpected-death budget. (#972)
|
||||
|
||||
### Fixed
|
||||
- **Codex MCP and skill usage is attributed from every shape Codex records a shell command in.** `mcp-cli call <server> <tool>` was only recognized when the command arrived as `function_call` arguments (#656). Codex has two other shapes for the same exec: its custom-tool transport records the shell tool as a `custom_tool_call` whose payload is an `input` program rather than `arguments`, and its item model repeats a finished command as `event_msg`/`item_completed` carrying a `CommandExecution` item with an argv `command`. Both reached the Bash counter and neither reached the matcher, so a CLI-wrapped MCP call stayed absent from the MCP breakdown exactly as before the fix. All three shapes now feed one classification pipeline. The same pipeline learns skills: Codex has no skill tool, so loading one is a shell read of the skill's `SKILL.md`, and those reads landed entirely under Bash with the Skills dimension empty. A read counts as a skill load only when the command segment starts with a file-reading binary (`cat`/`bat`/`sed`/`head`/`tail`/`less`/`more`) and the path it reads ends in `<name>/SKILL.md`; the skill is `<name>`, the same key `pi` derives for a native skill read (#588) and the same vocabulary the Claude parser records from the `Skill` tool. A `grep`/`rg`/`ls` that merely mentions a `SKILL.md` is a search near the file, not a skill load, and stays plain Bash. This is attribution only — no call, token or cost figure moves, and a command carried by both a response item and an item-model item is attributed once. On a 1,397-rollout corpus: Skills went from empty to 7 skills over 35 turns (55 attributions), Bash was unchanged at 42,170, and cost, calls, tokens, sessions, daily, models and projects came back identical. Cached Codex sessions re-parse once (`CODEX_CACHE_VERSION` 13 → 14 and the codex parse version both move; without them the fix is invisible on a warm cache). Thanks @chr-evensen. (#478)
|
||||
- **`gpt-5.6-codex` and `gpt-5.6-codex-max` now have their own pricing rows.** Neither id is in LiteLLM yet, and both were missing from the bundled snapshot — flagged during #1075 verification on a real corpus (285 sessions, 5,446 calls). `getModelCosts` already resolved both through the `gpt-5.6` prefix fallback, so live pricing was already correct once a session priced fresh; every prior Codex-suffixed id LiteLLM does carry bills identically to its bare-model sibling of the same generation (`gpt-5-codex` == `gpt-5`, `gpt-5.1-codex` == `gpt-5.1-codex-max` == `gpt-5.1`, `gpt-5.2-codex` == `gpt-5.2`, `gpt-5.3-codex` == `gpt-5.3`), which is the evidence both new rows mirror rather than inventing a rate. The gap that does not self-heal is the daily cache: it has no per-provider invalidation, so a day finalized while either id had no billable rate keeps that $0 forever. Raising `MIN_SUPPORTED_VERSION` (v23 -> v24) forces the one-time re-derivation, a lossless no-op for days already correct. (#1077)
|
||||
- **Mixed-version installs no longer thrash the Codex / Cursor / Antigravity result caches.** Daily and session caches already own a version-suffixed file so an old desktop binary and a newer CLI cannot clobber each other. The three per-provider result caches still used one unsuffixed filename with an internal version field, so a v10 and a v11 binary rewrote the same `codex-results.json` (and the Cursor / Antigravity siblings) on every run and each re-parsed its whole corpus. They now write `*-results.v<n>.json` the same way the daily cache does. The unsuffixed file is left for older binaries; a matching-version copy is adopted once and never overwritten. (#1082)
|
||||
- **Codex spend no longer counts reasoning tokens twice, and cache writes are priced only where OpenAI actually charges for them.** OpenAI bills reasoning tokens as *part of* `output_tokens`, not on top of it — on a 1,396-rollout corpus all 134,316 events carrying a total satisfy `input + output == total` — but CodeBurn added `reasoning_output_tokens` to output when pricing a Codex call and again in the models, audit and per-model displays. Every Codex number was therefore too high: on that corpus **cost by $166.03 (3.5%)** and **displayed Output tokens by 34.6%** ($4,713.12 -> $4,547.09; 22.6M -> 16.8M output tokens). The raw `reasoningTokens` figure is unchanged and still reported on its own; only the double-count is gone. Both places that price a Codex call — the parser and the cache-rehydration re-price — now go through one shared `billableOutputTokens` helper, so a cold run and a warm run can never disagree. Separately, Codex's `cache_write_input_tokens` (new in codex PR #33454) was never read and cache-creation tokens were hardcoded to 0; they are now carved out of the uncached-input bucket and clamped so they can never exceed it. That carve-out happens **only on models whose pricing source publishes a real cache-write rate** — gpt-5.6 and its terra/sol/luna variants charge 1.25x input for a cache write, everything before it charges nothing extra — because CodeBurn fabricates a 1.25x rate when a source omits one, and charging that would have invented a surcharge on gpt-5.5, gpt-5.4, gpt-5.3-codex and gpt-5. On models without an explicit rate the tokens stay in the plain input bucket and the price is unchanged to the cent. The field is new enough that today's impact is $0 on that corpus. Codex sessions re-parse once and the daily cache re-derives once off the warm session cache (a global re-derivation of every day and every provider, since it has no per-provider invalidation); no other provider's numbers move. Days whose Codex transcripts have since aged out are held by the same never-lose guard #1040 relies on: a re-derivation that finds fewer calls than the settled baseline keeps the older, pre-fix (double-counted) total rather than truncating it, so those days do not pick up the repricing until their sources are re-derived with equal or greater evidence. Long-context pricing tiers from the same report are tracked separately in #1076 and the missing `gpt-5.6-codex` snapshot rows in #1077. Thanks @chr-evensen. (#1075)
|
||||
|
|
|
|||
|
|
@ -34,7 +34,13 @@ import type { ParsedProviderCall } from './providers/types.js'
|
|||
// re-parse. Not 12: v12 is claimed by feat/core-extraction's own port of this
|
||||
// throughput feature (PR #1086), so reusing it would let two incompatible
|
||||
// schemas share a filename.
|
||||
export const CODEX_CACHE_VERSION = 13
|
||||
// v14: MCP + Skill attribution for the shapes the classic `function_call` path
|
||||
// never reached (#478) - the `exec` custom tool's `input` program and the item
|
||||
// model's `item_completed`/`CommandExecution` item - plus SKILL.md reads landing
|
||||
// in `skills`. This file stores each call's `tools`/`toolSequence`/`skills`
|
||||
// verbatim (they are passed through on read, never re-derived), so v13 entries
|
||||
// keep the old, MCP- and skill-less attribution until they re-parse.
|
||||
export const CODEX_CACHE_VERSION = 14
|
||||
export const CODEX_LEGACY_CACHE_FILE = 'codex-results.json'
|
||||
export function codexCacheFileName(version = CODEX_CACHE_VERSION): string {
|
||||
return `codex-results.v${version}.json`
|
||||
|
|
|
|||
|
|
@ -67,11 +67,7 @@ const toolNameMap: Record<string, string> = {
|
|||
// false-positive, an accepted tradeoff for the common case. \s+ and the token
|
||||
// class don't overlap, so there is no catastrophic backtracking.
|
||||
const MCP_CLI_CALL = /(?<![\w.-])mcp-cli(?:\s+(?!call\b)[^\s;|&]+)*\s+call\s+(\S+)\s+(\S+)/
|
||||
function mcpToolFromShellCommand(command: unknown): string | null {
|
||||
const text = typeof command === 'string'
|
||||
? command
|
||||
: Array.isArray(command) ? command.filter(x => typeof x === 'string').join(' ') : ''
|
||||
if (!text) return null
|
||||
function mcpToolFromShellCommand(text: string): string | null {
|
||||
const m = MCP_CLI_CALL.exec(text)
|
||||
if (!m) return null
|
||||
const server = m[1]!.replace(/['"]/g, '')
|
||||
|
|
@ -80,6 +76,37 @@ function mcpToolFromShellCommand(command: unknown): string | null {
|
|||
return `mcp__${server}__${tool}`
|
||||
}
|
||||
|
||||
// Codex has no dedicated skill tool: loading a skill is an ordinary shell exec
|
||||
// that reads the skill's `SKILL.md`, so the usage landed entirely under Bash and
|
||||
// the Skills dimension stayed empty (#478). Recognize only a *read* of a
|
||||
// SKILL.md, deliberately narrowly:
|
||||
// - the command segment must start with a file-reading binary (cat/sed/head/
|
||||
// tail/less/more/bat) — a `grep`/`rg`/`ls` that merely mentions a SKILL.md
|
||||
// is a search near the file, not a skill load, and stays plain Bash;
|
||||
// - the path it reads must END in `<name>/SKILL.md`, so `<name>` is the skill.
|
||||
// The name is the parent directory, the same key `src/providers/pi.ts` derives
|
||||
// for a native skill read (#588) and the same vocabulary the Claude parser
|
||||
// records from the `Skill` tool, so the Skills breakdown stays cross-provider
|
||||
// coherent. Substring matching, so a command that merely quotes the phrase can
|
||||
// false-positive — the same accepted tradeoff as MCP_CLI_CALL above.
|
||||
const SKILL_MD_READ = /(?<![\w.-])(?:cat|bat|sed|head|tail|less|more)\b[^;|&]*?[\s'"]([^\s;|&'"]*[\\/]([^\\/;|&'"]+)[\\/]SKILL\.md)\b/
|
||||
function skillFromShellCommand(text: string): string | null {
|
||||
const m = SKILL_MD_READ.exec(text)
|
||||
const name = m?.[2]?.trim()
|
||||
return name ? name : null
|
||||
}
|
||||
|
||||
/// Flatten every shape Codex records a shell command in to one string:
|
||||
/// `function_call` arguments carry it as a plain string, the item model's
|
||||
/// `CommandExecution` item as an argv array (`["/bin/zsh","-lc","..."]`), and
|
||||
/// the `exec` custom tool as the JS program that calls `tools.exec_command`.
|
||||
/// One text, one classification pipeline (#478).
|
||||
function shellCommandText(command: unknown): string {
|
||||
if (typeof command === 'string') return command
|
||||
if (Array.isArray(command)) return command.filter(x => typeof x === 'string').join(' ')
|
||||
return ''
|
||||
}
|
||||
|
||||
// Count added/removed lines from a Codex `patch_apply_end` change's
|
||||
// `unified_diff`. A leading '+' is an added line and '-' a removed line; the
|
||||
// '+++'/'---' file headers and '@@' hunk headers are excluded. Numbers only —
|
||||
|
|
@ -604,6 +631,9 @@ type CodexResumeState = {
|
|||
prevReasoning: number
|
||||
pendingTools: string[]
|
||||
pendingToolSequence: ToolCall[][]
|
||||
/// Optional so a resume state written before skills attribution existed still
|
||||
/// decodes (isResumeState does not require it); absent reads as "no skills".
|
||||
pendingSkills?: string[]
|
||||
pendingUserMessage: string
|
||||
pendingOutputChars: number
|
||||
pendingLocAdded: number
|
||||
|
|
@ -697,6 +727,46 @@ function createParser(source: SessionSource, seenKeys: Set<string>, capture?: {
|
|||
let prevReasoning = resume?.state.prevReasoning ?? 0
|
||||
let pendingTools: string[] = resume ? [...resume.state.pendingTools] : []
|
||||
let pendingToolSequence: ToolCall[][] = resume ? [...resume.state.pendingToolSequence] : []
|
||||
let pendingSkills: string[] = resume?.state.pendingSkills ? [...resume.state.pendingSkills] : []
|
||||
// Attribution names already emitted from a `response_item` tool call.
|
||||
// Codex's item model repeats a finished shell command as
|
||||
// `item_completed`/`CommandExecution`; a rollout that carries BOTH shapes
|
||||
// must attribute the call once, and one that carries ONLY the item must
|
||||
// still attribute it. Counting per derived name (not per command text,
|
||||
// which the two shapes spell differently - argv array vs plain string)
|
||||
// makes repeats of the same call line up one-for-one. The response item is
|
||||
// always written first (the model requests the call, the item completes
|
||||
// it), so the item side only ever cancels an attribution, never adds a
|
||||
// duplicate.
|
||||
const itemDedup = new Map<string, number>()
|
||||
|
||||
/// The single classification pipeline every Codex shell-command shape
|
||||
/// feeds (#478): `function_call` arguments, the `exec` custom tool's JS
|
||||
/// `input`, and the item model's `CommandExecution` item. It adds MCP and
|
||||
/// Skill attribution only - the exec itself is counted as Bash by its
|
||||
/// response item, and `viaItem` calls never add a tool of their own, so
|
||||
/// call/token/cost totals cannot move.
|
||||
const attributeShellCommand = (text: string, viaItem: boolean): void => {
|
||||
if (!text) return
|
||||
const claim = (name: string): boolean => {
|
||||
const pending = itemDedup.get(name) ?? 0
|
||||
if (!viaItem) { itemDedup.set(name, pending + 1); return true }
|
||||
if (pending === 0) return true
|
||||
itemDedup.set(name, pending - 1)
|
||||
return false
|
||||
}
|
||||
const mcpTool = mcpToolFromShellCommand(text)
|
||||
if (mcpTool && claim(mcpTool)) {
|
||||
pendingTools.push(mcpTool)
|
||||
pendingToolSequence.push([{ tool: mcpTool }])
|
||||
}
|
||||
const skill = skillFromShellCommand(text)
|
||||
if (skill && claim(`skill:${skill}`)) {
|
||||
pendingSkills.push(skill)
|
||||
pendingTools.push('Skill')
|
||||
pendingToolSequence.push([{ tool: 'Skill', file: skill }])
|
||||
}
|
||||
}
|
||||
let pendingUserMessage = resume?.state.pendingUserMessage ?? ''
|
||||
let pendingOutputChars = resume?.state.pendingOutputChars ?? 0
|
||||
// Rich-session-capture: edit LOC deltas and failed-patch count accumulated
|
||||
|
|
@ -801,6 +871,7 @@ function createParser(source: SessionSource, seenKeys: Set<string>, capture?: {
|
|||
entry.payload?.type === 'custom_tool_call' ||
|
||||
entry.payload?.type === 'custom_tool_call_output' ||
|
||||
entry.payload?.type === 'mcp_tool_call_end' ||
|
||||
entry.payload?.type === 'item_completed' ||
|
||||
entry.payload?.type === 'patch_apply_end'
|
||||
)) continue
|
||||
|
||||
|
|
@ -833,6 +904,7 @@ function createParser(source: SessionSource, seenKeys: Set<string>, capture?: {
|
|||
prevReasoning,
|
||||
pendingTools: [...pendingTools],
|
||||
pendingToolSequence: [...pendingToolSequence],
|
||||
pendingSkills: [...pendingSkills],
|
||||
pendingUserMessage,
|
||||
pendingOutputChars,
|
||||
pendingLocAdded,
|
||||
|
|
@ -861,13 +933,21 @@ function createParser(source: SessionSource, seenKeys: Set<string>, capture?: {
|
|||
if (typeof fp === 'string') call.file = fp
|
||||
const cmd = args['command'] ?? args['cmd']
|
||||
if (typeof cmd === 'string') call.command = cmd
|
||||
// Attribute a CLI-wrapped MCP call (e.g. `mcp-cli call server tool`)
|
||||
// to the MCP breakdown too; the exec still counts as Bash above.
|
||||
const mcpTool = mcpToolFromShellCommand(cmd)
|
||||
if (mcpTool) {
|
||||
pendingTools.push(mcpTool)
|
||||
pendingToolSequence.push([{ tool: mcpTool }])
|
||||
}
|
||||
attributeShellCommand(shellCommandText(cmd), false)
|
||||
}
|
||||
// A `custom_tool_call` has no `arguments`: the shell tool's payload is
|
||||
// the `input` program that calls `tools.exec_command({cmd: ...})`. The
|
||||
// MCP/skill matchers never saw it, so a CLI-wrapped MCP call or a
|
||||
// SKILL.md read made through Codex's custom-tool transport counted only
|
||||
// as Bash (#478). Restricted to the shell tool: `apply_patch` is a
|
||||
// custom tool too and its `input` is file content, where a documented
|
||||
// command would read as a real invocation. `call.command` is
|
||||
// deliberately NOT set from it either - that field feeds the retry
|
||||
// heuristic's read-shaped-command test, and a JS program is not a
|
||||
// shell command.
|
||||
if (mapped === 'Bash') {
|
||||
const input = (entry.payload as Record<string, unknown>)['input']
|
||||
if (typeof input === 'string') attributeShellCommand(input, false)
|
||||
}
|
||||
const callId = entry.payload.call_id
|
||||
const started = entry.timestamp ? Date.parse(entry.timestamp) : NaN
|
||||
|
|
@ -965,6 +1045,21 @@ function createParser(source: SessionSource, seenKeys: Set<string>, capture?: {
|
|||
continue
|
||||
}
|
||||
|
||||
// Codex's item model records a finished shell command as
|
||||
// `event_msg`/`item_completed` carrying a `CommandExecution` item whose
|
||||
// `command` is the argv array (`["/bin/zsh","-lc","..."]`) - a shape the
|
||||
// `function_call`-only tool path never reached, so a CLI-wrapped MCP call
|
||||
// or a SKILL.md read made under it stayed invisible (#478). Attribution
|
||||
// only: the exec's own Bash count comes from its response item, exactly
|
||||
// as `exec_command_end` is left alone for the classic transport.
|
||||
if (entry.type === 'event_msg' && entry.payload?.type === 'item_completed') {
|
||||
const item = (entry.payload as Record<string, unknown>)['item'] as Record<string, unknown> | undefined
|
||||
if (item && item['type'] === 'CommandExecution') {
|
||||
attributeShellCommand(shellCommandText(item['command']), true)
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
if (entry.type === 'response_item' && entry.payload?.type === 'message' && entry.payload?.role === 'user') {
|
||||
const texts = normalizeContentBlocks(entry.payload.content)
|
||||
.filter(c => c.type === 'input_text')
|
||||
|
|
@ -1001,7 +1096,7 @@ function createParser(source: SessionSource, seenKeys: Set<string>, capture?: {
|
|||
const timestamp = entry.timestamp ?? ''
|
||||
const dedupKey = `codex:${sessionId}:${timestamp}:est${estCounter++}`
|
||||
|
||||
if (seenKeys.has(dedupKey)) { pendingTools = []; pendingToolSequence = []; pendingUserMessage = ''; pendingOutputChars = 0; pendingLocAdded = 0; pendingLocRemoved = 0; pendingEditFailed = 0; continue }
|
||||
if (seenKeys.has(dedupKey)) { pendingTools = []; pendingToolSequence = []; pendingSkills = []; pendingUserMessage = ''; pendingOutputChars = 0; pendingLocAdded = 0; pendingLocRemoved = 0; pendingEditFailed = 0; continue }
|
||||
seenKeys.add(dedupKey)
|
||||
|
||||
const costUSD = calculateCost(model, estInput, estOutput, 0, 0, 0)
|
||||
|
|
@ -1025,6 +1120,7 @@ function createParser(source: SessionSource, seenKeys: Set<string>, capture?: {
|
|||
deduplicationKey: dedupKey,
|
||||
turnId: currentTurnId,
|
||||
toolSequence: pendingToolSequence.length > 0 ? pendingToolSequence : undefined,
|
||||
...(pendingSkills.length > 0 ? { skills: pendingSkills } : {}),
|
||||
userMessage: pendingUserMessage,
|
||||
sessionId,
|
||||
...(sessionCwd ? { projectPath: sessionCwd, workingDirectory: sessionCwd } : {}),
|
||||
|
|
@ -1036,6 +1132,7 @@ function createParser(source: SessionSource, seenKeys: Set<string>, capture?: {
|
|||
|
||||
pendingTools = []
|
||||
pendingToolSequence = []
|
||||
pendingSkills = []
|
||||
pendingUserMessage = ''
|
||||
pendingOutputChars = 0
|
||||
pendingLocAdded = 0
|
||||
|
|
@ -1170,6 +1267,7 @@ function createParser(source: SessionSource, seenKeys: Set<string>, capture?: {
|
|||
deduplicationKey: dedupKey,
|
||||
turnId: currentTurnId,
|
||||
toolSequence: pendingToolSequence.length > 0 ? pendingToolSequence : undefined,
|
||||
...(pendingSkills.length > 0 ? { skills: pendingSkills } : {}),
|
||||
userMessage: pendingUserMessage,
|
||||
sessionId,
|
||||
...(sessionCwd ? { projectPath: sessionCwd, workingDirectory: sessionCwd } : {}),
|
||||
|
|
@ -1181,6 +1279,7 @@ function createParser(source: SessionSource, seenKeys: Set<string>, capture?: {
|
|||
|
||||
pendingTools = []
|
||||
pendingToolSequence = []
|
||||
pendingSkills = []
|
||||
pendingUserMessage = ''
|
||||
pendingOutputChars = 0
|
||||
pendingLocAdded = 0
|
||||
|
|
|
|||
|
|
@ -290,7 +290,11 @@ export const PROVIDER_PARSE_VERSIONS: Record<string, string> = {
|
|||
// same double-count codex-pricing-v1 removed from cost. Cached entries store
|
||||
// activeGeneratedTokens/activeDurationMs/toolWaitMs verbatim (cachedCallToApiCall
|
||||
// passes them through without recomputing), so this does NOT self-heal either.
|
||||
codex: 'mcp-attribution-v5-est-cost-active-timing-mcp-wait-rich-capture-v1-cross-provider-pr-v1-session-meta-model-v1-session-meta-fields-v1-codex-pricing-v1-codex-tps-v1',
|
||||
// codex-mcp-skills-v1 (#478): CLI-wrapped MCP calls and SKILL.md reads made
|
||||
// through the `exec` custom tool or the item model's `CommandExecution` item
|
||||
// were counted as Bash only. Cached sessions store tools/toolSequence/skills
|
||||
// verbatim, so they must re-parse to gain the attribution.
|
||||
codex: 'mcp-attribution-v5-est-cost-active-timing-mcp-wait-rich-capture-v1-cross-provider-pr-v1-session-meta-model-v1-session-meta-fields-v1-codex-pricing-v1-codex-tps-v1-codex-mcp-skills-v1',
|
||||
cursor: 'composer-anchored-crediting-v1-est-cost',
|
||||
'cursor-agent': 'workspaceless-transcript-v1',
|
||||
// source-provenance-v1 (#944): CLI sessions were misread as VS Code
|
||||
|
|
|
|||
|
|
@ -1030,6 +1030,79 @@ describe('codex provider - JSONL parsing', () => {
|
|||
])
|
||||
})
|
||||
|
||||
// #478 follow-up: the shapes the `function_call` path never reached. Fixtures
|
||||
// are synthesized from the real shapes seen in Codex rollouts (a `custom_tool_call`
|
||||
// whose payload is an `input` JS program, and an item-model `item_completed`
|
||||
// carrying a `CommandExecution` item with an argv `command`); no real session
|
||||
// content is used.
|
||||
it('attributes MCP + Skill usage from the exec custom tool and the item model', async () => {
|
||||
const customExec = (input: string, callId: string) => JSON.stringify({
|
||||
type: 'response_item',
|
||||
timestamp: '2026-04-14T10:00:30Z',
|
||||
payload: { type: 'custom_tool_call', call_id: callId, name: 'exec', input },
|
||||
})
|
||||
const commandExecutionItem = (command: string[]) => JSON.stringify({
|
||||
type: 'event_msg',
|
||||
timestamp: '2026-04-14T10:00:40Z',
|
||||
payload: { type: 'item_completed', item: { type: 'CommandExecution', command, exit_code: 0 } },
|
||||
})
|
||||
const filePath = await writeSession(tmpDir, '2026-04-14', 'rollout-exec-items.jsonl', [
|
||||
sessionMeta({ session_id: 'sess-exec-items', model: 'gpt-5.5' }),
|
||||
userMessage('use the MCP CLI and load a skill'),
|
||||
// custom-tool transport: MCP call and a skill read, both inside the JS program.
|
||||
customExec('const r = await tools.exec_command({cmd:"mcp-cli call github get_issue \'{}\'"}); text(r.output);', 'c1'),
|
||||
customExec('const r = await tools.exec_command({cmd:"sed -n \'1,200p\' /Users/x/.codex/skills/control-in-app-browser/SKILL.md"}); text(r.output);', 'c2'),
|
||||
// Negatives: a lookup subcommand, and a grep that merely mentions a SKILL.md.
|
||||
customExec('const r = await tools.exec_command({cmd:"mcp-cli info github"}); text(r.output);', 'c3'),
|
||||
customExec('const r = await tools.exec_command({cmd:"grep -rn TODO /Users/x/.codex/skills/deploy/SKILL.md"}); text(r.output);', 'c4'),
|
||||
// item model, no matching response item: must attribute on its own.
|
||||
commandExecutionItem(['/bin/zsh', '-lc', "mcp-cli call optimizely-cms-mcp help '{}'"]),
|
||||
commandExecutionItem(['/bin/zsh', '-lc', 'cat /Users/x/.codex/skills/graphify/SKILL.md']),
|
||||
commandExecutionItem(['/bin/zsh', '-lc', 'ls -la']),
|
||||
tokenCount({ timestamp: '2026-04-14T10:01:00Z', last: { input: 300, output: 100 }, total: { total: 400 } }),
|
||||
])
|
||||
|
||||
const provider = createCodexProvider(tmpDir)
|
||||
const source = { path: filePath, project: 'test', provider: 'codex' }
|
||||
const calls: ParsedProviderCall[] = []
|
||||
for await (const call of provider.createSessionParser(source, new Set()).parse()) calls.push(call)
|
||||
|
||||
expect(calls).toHaveLength(1)
|
||||
const tools = calls[0]!.tools
|
||||
// Attribution only: the four custom-tool execs stay Bash, and the item-model
|
||||
// entries add no tool of their own, so the Bash count is unchanged at 4.
|
||||
expect(tools.filter(t => t === 'Bash')).toHaveLength(4)
|
||||
expect(tools.filter(t => t.startsWith('mcp__')).sort()).toEqual([
|
||||
'mcp__github__get_issue',
|
||||
'mcp__optimizely-cms-mcp__help',
|
||||
])
|
||||
expect(calls[0]!.skills?.slice().sort()).toEqual(['control-in-app-browser', 'graphify'])
|
||||
expect(tools.filter(t => t === 'Skill')).toHaveLength(2)
|
||||
})
|
||||
|
||||
it('counts a command carried by BOTH the response item and the item model once', async () => {
|
||||
const cmd = "mcp-cli call github get_issue '{}'"
|
||||
const filePath = await writeSession(tmpDir, '2026-04-14', 'rollout-both-shapes.jsonl', [
|
||||
sessionMeta({ session_id: 'sess-both-shapes', model: 'gpt-5.5' }),
|
||||
userMessage('call it twice'),
|
||||
JSON.stringify({ type: 'response_item', timestamp: '2026-04-14T10:00:30Z', payload: { type: 'function_call', name: 'exec_command', arguments: JSON.stringify({ cmd }) } }),
|
||||
JSON.stringify({ type: 'event_msg', timestamp: '2026-04-14T10:00:31Z', payload: { type: 'item_completed', item: { type: 'CommandExecution', command: ['/bin/zsh', '-lc', cmd] } } }),
|
||||
JSON.stringify({ type: 'response_item', timestamp: '2026-04-14T10:00:32Z', payload: { type: 'function_call', name: 'exec_command', arguments: JSON.stringify({ cmd }) } }),
|
||||
JSON.stringify({ type: 'event_msg', timestamp: '2026-04-14T10:00:33Z', payload: { type: 'item_completed', item: { type: 'CommandExecution', command: ['/bin/zsh', '-lc', cmd] } } }),
|
||||
tokenCount({ timestamp: '2026-04-14T10:01:00Z', last: { input: 300, output: 100 }, total: { total: 400 } }),
|
||||
])
|
||||
|
||||
const provider = createCodexProvider(tmpDir)
|
||||
const source = { path: filePath, project: 'test', provider: 'codex' }
|
||||
const calls: ParsedProviderCall[] = []
|
||||
for await (const call of provider.createSessionParser(source, new Set()).parse()) calls.push(call)
|
||||
|
||||
expect(calls).toHaveLength(1)
|
||||
// Two execs, two MCP attributions - not four.
|
||||
expect(calls[0]!.tools.filter(t => t === 'Bash')).toHaveLength(2)
|
||||
expect(calls[0]!.tools.filter(t => t === 'mcp__github__get_issue')).toHaveLength(2)
|
||||
})
|
||||
|
||||
it('normalizes Codex subagent tool calls to Agent', async () => {
|
||||
const filePath = await writeSession(tmpDir, '2026-04-14', 'rollout-agent.jsonl', [
|
||||
sessionMeta({ session_id: 'sess-agent', model: 'gpt-5.5' }),
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue