mirror of
https://github.com/AgentSeal/codeburn.git
synced 2026-11-22 12:59:06 +00:00
* fix(optimize): count slash commands named with their leading slash as used
A slash command shows up in the transcript as <command-name>/review</command-name>.
commandNames kept the slash, so "/review" never matched the defined command
"review": every command the user actually ran was reported unused and the apply
plan archived it. The leading slash is now stripped.
commandNames produces stored facts, so FACTS_VERSION goes to 2 and the
optimize facts store rebuilds once on the next run.
* fix(optimize): name MCP servers the way Claude Code namespaces their tools
Claude Code turns every character outside letters, digits, "_" and "-" into
"_" when it builds tool names, so a server "docs.local" is called as
mcp__docs_local__*. Config names were only mapped ":" -> "_", so such a server
was reported as never called and offered for removal, counted twice by the
deferral check, and shown with 0 calls by the alwaysLoad check. One helper,
mcpNamespace, now applies the full rule in the config loader, the local server
names and the remove plan's key lookup.
* fix(optimize): charge a never-called MCP server's schema once
A configured server that was never called is in unused-mcp, and the same
schema was charged again by mcp-deferral-off (unpinned) or
mcp-alwaysload-hygiene (pinned), so potential savings and the share of spend
summed it twice. The servers unused-mcp reports are now passed to both checks
and left out there.
* fix(act): recommend a default model only for Claude Code projects
The model-default plan writes "model" into <project>/.claude/settings.json,
which only Claude Code reads. A Codex project got a gpt model recommended and
apply-model wrote it there, so Claude Code in that project started on a model
it does not know. recommendModelDefault now returns nothing for a non-Claude
provider.
* fix(optimize): read the bash output cap from the settings env block first
Claude Code takes BASH_MAX_OUTPUT_LENGTH from the env block of its settings
files. The check only read shell profiles (lines with export) and the process
env, so a cap of 10000 set in settings.json still reported the 30K default and
the apply plan appended an export to ~/.zshrc that the settings value
overrides. The check now uses the same lookup as the deferral checks: settings
env from the most specific scope, then shell profiles.
* fix(act): plan each optimize fix right before applying it
optimize --apply built every plan up front, each carrying the hash of the
config file as first read. When two fixes edit the same file (remove one MCP
server and unpin another in .mcp.json, or two edits to ~/.claude.json), the
first apply changed the file and the second was refused as "changed since the
plan was built", exit code 1. Each plan is now rebuilt from the file on disk
just before it runs; the baseline captured for it is kept, and the stale-plan
check still guards edits made while the plan is being applied.
* fix(optimize): fast mode on background agents saves dollars, not tokens
Switching subagent calls from fast to standard speed bills the same tokens at
a lower rate. The finding claimed every output token of those calls as tokens
saved, which inflated the token headline and pushed it up the urgency ranking.
tokensSaved is now 0; the dollar figure is unchanged and still carries it.
* fix(optimize): charge an unused MCP server only to Claude sessions
unused-mcp reads servers from Claude Code config, but multiplied their schema
by every session of every provider in the period, so under --provider all a
handful of Claude sessions next to hundreds of Codex or Cursor sessions
produced a large token and dollar figure. It now counts and prices only Claude
sessions, as the deferral checks already do.
* fix(optimize): stop reading MCP servers from ~/.claude/settings.json
Claude Code reads MCP servers from ~/.claude.json and .mcp.json, not from the
mcpServers key of ~/.claude/settings.json or settings.local.json. The MCP
checks read both user settings files, while the remove plan never edits them,
so a server listed only there showed as an unused server with savings under
"Fix now" and apply skipped it as "not found in editable config". The loader
now leaves those two files out; its home directory parameter, and the one on
the alwaysLoad check that only fed it, go with them.
* fix(act): match MCP servers in the report by their namespace
The report's baseline keeps config names such as "docs.local", while sessions
name the server as Claude Code does in tool names ("docs_local"). A removed
server that came back never matched, so act report counted savings instead of
reading it as reverted. sessionLoadsAny now compares through mcpNamespace.
* fix(optimize): keep every config name that maps to one MCP namespace
Claude Code turns "a.b" and "a:b" into the same tool namespace "a_b", so the
MCP config loader keyed both under one entry and kept only the newest name.
unused-mcp then named one of them, and the remove plan left the other config
entry in place. The entry now keeps every config name; unused-mcp and the
alwaysLoad check pass each one to their plans.
* fix(act): read a "__" server name whole from report inventory entries
Inventory entries are mcp__<server>__<tool>, and the report took the server
as the second "__" segment, so a server named "a__b" was read as "a" and a
re-added "a__b" never showed as reverted. The tool is now cut off from the
right, leaving the full server name.
* fix(act): resolve MCP namespaces against the known servers
mcp__a__b__c can be server "a__b" with tool "c" or server "a" with tool
"b__c", and both fixed rules (first segment, or cut from the right) pick the
wrong server for some names. A tool name is now split at the longest server
namespace found in the MCP config (plus the servers being measured in act
report), falling back to the first segment only when none matches. This
applies to act report's inventory reads and to the call names unused-mcp and
mcp-deferral-off count.
* fix(act): replan only for files an earlier fix in this run changed
Rebuilding every plan right before it ran took a fresh hash of the current
file, so under --apply --yes a config file edited outside codeburn between
selection and apply was rewritten anyway, where the stale-plan guard used to
refuse it. The run now records the hash of every file it writes. A plan is
rebuilt only when one of its files differs from the plan solely because an
earlier fix in this run wrote it; any other change keeps the original plan,
and runAction rejects it with "changed since the plan was built".
* docs(optimize): note why an MCP tool name with two known prefixes is ambiguous
124 lines
5.7 KiB
TypeScript
124 lines
5.7 KiB
TypeScript
import { describe, expect, it } from 'vitest'
|
|
|
|
import { detectFastModeBackground, detectLongContextPremium, findingClass } from '../src/optimize.js'
|
|
import { calculateCost } from '../src/models.js'
|
|
import type { ParsedApiCall, ProjectSummary, SessionSummary } from '../src/types.js'
|
|
|
|
type Tokens = { input?: number; output?: number; cacheRead?: number }
|
|
|
|
function apiCall(model: string, speed: ParsedApiCall['speed'], t: Tokens, i: number): ParsedApiCall {
|
|
const input = t.input ?? 0
|
|
const output = t.output ?? 0
|
|
const cacheRead = t.cacheRead ?? 0
|
|
return {
|
|
provider: 'claude',
|
|
model,
|
|
usage: {
|
|
inputTokens: input,
|
|
outputTokens: output,
|
|
cacheCreationInputTokens: 0,
|
|
cacheReadInputTokens: cacheRead,
|
|
cachedInputTokens: 0,
|
|
reasoningTokens: 0,
|
|
webSearchRequests: 0,
|
|
},
|
|
costUSD: calculateCost(model, input, output, 0, cacheRead, 0, speed, 0, 'claude'),
|
|
tools: [],
|
|
mcpTools: [],
|
|
skills: [],
|
|
subagentTypes: [],
|
|
hasAgentSpawn: false,
|
|
hasPlanMode: false,
|
|
speed,
|
|
timestamp: `2026-10-01T10:${String(i).padStart(2, '0')}:00Z`,
|
|
bashCommands: [],
|
|
deduplicationKey: `call-${i}`,
|
|
}
|
|
}
|
|
|
|
function session(id: string, calls: ParsedApiCall[], isSidechain = false): SessionSummary {
|
|
return {
|
|
sessionId: id,
|
|
project: 'app',
|
|
isSidechain,
|
|
firstTimestamp: '2026-10-01T10:00:00Z',
|
|
lastTimestamp: '2026-10-01T11:00:00Z',
|
|
totalCostUSD: calls.reduce((s, c) => s + c.costUSD, 0),
|
|
apiCalls: calls.length,
|
|
turns: calls.map(c => ({ userMessage: 'go', assistantCalls: [c], timestamp: c.timestamp, sessionId: id, category: 'coding', retries: 0, hasEdits: false })),
|
|
} as unknown as SessionSummary
|
|
}
|
|
|
|
function projects(...sessions: SessionSummary[]): ProjectSummary[] {
|
|
return [{ project: 'app', projectPath: '/tmp/app', sessions } as unknown as ProjectSummary]
|
|
}
|
|
|
|
describe('detectFastModeBackground', () => {
|
|
it('reports the fast premium of subagent calls', () => {
|
|
// opus-4-8 fast is 2x: 100k output x $25/M = $2.50 extra.
|
|
const finding = detectFastModeBackground(projects(session('agent-1', [apiCall('claude-opus-4-8', 'fast', { output: 100_000 }, 0)], true)))
|
|
expect(finding).not.toBeNull()
|
|
expect(finding!.id).toBe('fast-mode-background')
|
|
expect(findingClass(finding!)).toBe('nudge')
|
|
expect(finding!.savingsUSD).toBeCloseTo(2.5, 10)
|
|
expect(finding!.tokensSaved).toBe(0)
|
|
expect(finding!.impact).toBe('low')
|
|
expect(finding!.title).toContain('$2.50')
|
|
expect(finding!.explanation).toContain('1 fast-mode call ')
|
|
expect(finding!.explanation).toContain('Opus 4.8')
|
|
})
|
|
|
|
it('rates a premium above $20 high', () => {
|
|
const calls = Array.from({ length: 10 }, (_, i) => apiCall('claude-opus-4-8', 'fast', { output: 100_000 }, i))
|
|
expect(detectFastModeBackground(projects(session('agent-1', calls, true)))!.impact).toBe('high')
|
|
})
|
|
|
|
it('ignores main-thread fast calls and premiums under $1', () => {
|
|
expect(detectFastModeBackground(projects(session('main', [apiCall('claude-opus-4-8', 'fast', { output: 100_000 }, 0)])))).toBeNull()
|
|
expect(detectFastModeBackground(projects(session('agent-1', [apiCall('claude-opus-4-8', 'fast', { output: 800 }, 0)], true)))).toBeNull()
|
|
})
|
|
|
|
it('ignores a model without a fast multiplier', () => {
|
|
expect(detectFastModeBackground(projects(session('agent-1', [apiCall('claude-opus-4-6', 'fast', { output: 100_000 }, 0)], true)))).toBeNull()
|
|
})
|
|
})
|
|
|
|
describe('detectLongContextPremium', () => {
|
|
// sonnet-4-5 above 200k: 10k input + 290k cache read + 1k output costs
|
|
// $0.06 + $0.174 + $0.0225 = $0.2565 against $0.03 + $0.087 + $0.015 = $0.132.
|
|
const over = (i: number) => apiCall('claude-sonnet-4-5', 'standard', { input: 10_000, cacheRead: 290_000, output: 1_000 }, i)
|
|
|
|
it('reports calls priced above the tier', () => {
|
|
const finding = detectLongContextPremium(projects(
|
|
session('s1', Array.from({ length: 10 }, (_, i) => over(i))),
|
|
session('s2', [apiCall('claude-sonnet-4-5', 'standard', { input: 150_000, output: 1_000 }, 20)]),
|
|
))
|
|
expect(finding).not.toBeNull()
|
|
expect(finding!.id).toBe('long-context-premium')
|
|
expect(findingClass(finding!)).toBe('keep')
|
|
expect(finding!.apply).toBeUndefined()
|
|
expect(finding!.savingsUSD).toBeCloseTo(10 * 0.1245, 10)
|
|
expect(finding!.tokensSaved).toBe(10 * 100_000)
|
|
expect(finding!.impact).toBe('low')
|
|
expect(finding!.explanation).toContain('10 of 11 calls (90.9%) in 1 session ')
|
|
expect(finding!.explanation).toContain('Sonnet 4.5 above 200.0K: 10 calls')
|
|
expect(finding!.fix.type === 'paste' && finding!.fix.text).toContain('200.0K on Sonnet 4.5')
|
|
})
|
|
|
|
it('prices only the tier on a fast call, leaving the fast premium to its own finding', () => {
|
|
// gpt-5.5 above 272k at standard: $0.10 + $0.29 + $0.045 = $0.435 against $0.05 + $0.145 + $0.03 = $0.225.
|
|
const fast = Array.from({ length: 10 }, (_, i) => apiCall('gpt-5.5', 'fast', { input: 10_000, cacheRead: 290_000, output: 1_000 }, i))
|
|
expect(detectLongContextPremium(projects(session('s1', fast)))!.savingsUSD).toBeCloseTo(10 * 0.21, 10)
|
|
})
|
|
|
|
it('ignores calls under the threshold and premiums under $1', () => {
|
|
expect(detectLongContextPremium(projects(session('s1', Array.from({ length: 10 }, (_, i) =>
|
|
apiCall('claude-sonnet-4-5', 'standard', { input: 10_000, cacheRead: 190_000, output: 1_000 }, i)))))).toBeNull()
|
|
expect(detectLongContextPremium(projects(session('s1', [over(0)])))).toBeNull()
|
|
})
|
|
|
|
it('ignores a model without a long-context tier', () => {
|
|
expect(detectLongContextPremium(projects(session('s1', Array.from({ length: 10 }, (_, i) =>
|
|
apiCall('claude-sonnet-4-6', 'standard', { input: 10_000, cacheRead: 290_000, output: 1_000 }, i)))))).toBeNull()
|
|
})
|
|
})
|