mirror of
https://github.com/AgentSeal/codeburn.git
synced 2026-08-23 15:34:19 +00:00
fix(kiro): estimate chat-file input tokens from every human turn, and invalidate cached history
decodeKiroChatFile estimated input tokens from pendingUserMessage - the last human turn sliced to 500 chars - while output summed every bot char, so any multi-turn session or final prompt over 500 chars under-reported input tokens (and therefore costUSD) severalfold. The estimate is now the sum of every human turn's full character count, with the 500-char cap kept for the display userMessage only. That closes most of the gap but does NOT reach parity with the modern-execution, CLI-session and V2 arms: those count tool and system records as input - their code comments state tool results are fed back to the model - while the chat arm still counts only human records. Tool content demonstrably exists in the format (the G2 fixture carries a tool record), so the chat arm still under-reports; it just under-reports far less than before. The blast radius is the chat-file arm alone: the IDE-file dispatcher routes any record carrying a chat array plus metadata to decodeKiroChatFile, so this is chat-shaped Kiro IDE files, not every Kiro prompt. The identity-preamble exclusion now trims leading whitespace before its startsWith match. Pre-fix, a near miss (a leading newline, a BOM, a wrapper) was nearly harmless, because the preamble only mattered if it happened to be the last human turn; post-fix, every unmatched system-injected human record adds its FULL length to input, and preambles are large - a missed match is a silent multi-thousand-token inflation on every affected session. Leading whitespace tolerance is cheap (a genuine prompt never starts with whitespace plus an identity tag) and the failure asymmetry favours exclusion: a false negative inflates tokens, a false positive only skips a preamble. A renamed preamble remains a residual risk, noted in the near-miss regression test. Cached history is affected, which is what a user actually sees. session-cache serves unchanged files without invoking the provider parser, so bump kiro's PROVIDER_PARSE_VERSIONS fingerprint (ide-parsing-v1 -> v2) to force one re-parse of every already-cached kiro session; without it the pre-fix token and cost numbers would be served forever. The daily rollup ALSO needs invalidating for this fix to be fully visible: days finalized before the fix keep their pre-fix kiro cost in the daily cache, and ensureCacheHydrated re-derives them only on a version bump, a savings-config change, a timezone change, or an incomplete cache — the session-cache re-parse alone leaves finalized day totals untouched. So this commit bumps BOTH layers: the session-cache PROVIDER_PARSE_VERSIONS fingerprint above forces the one re-parse of every already-cached kiro session, and DAILY_CACHE_VERSION (15 -> 17, MIN_SUPPORTED_VERSION raised with it; 16 is skipped because main already claimed it for the codex structural-discovery fix, and claiming 16 here would load a main-built v16 cache as current and complete, so the invalidation would never fire) forces the daily rollup's one-time re-derivation, so finalized day totals are rebuilt under the corrected estimate. The re-derive reaches every day whose kiro chat files still exist; sourceless days carry forward with their pre-fix totals under the v14 NEVER-LOSE rule (a carry-forward, not a refresh — nothing can reconstruct them once the files are gone). Update the G2 parity golden: A1 was pinned at 125 tokens for a 3000-char prompt (the 500-char slice / 4); the corrected value is 750 (3000 / 4), with a comment marking 125 as a pre-fix value so it is not restored. 3000, 2400 and 1000+1000 are all exact multiples of four, so add G2b pinning the estimator's rounding with an odd length (3001 chars -> 751 tokens; round and floor would both give 750). Add a money-path regression test (2400-char prompt -> 600 tokens, userMessage still 500-capped for display), a multi-turn accumulation test (an identical resubmitted prompt counts again - a real second model input; identity messages stay excluded), a near-miss identity test (leading-newline and BOM preambles stay excluded), extend the kiro cache-invalidation test to pin the v1 -> v2 fingerprint bump, and add a daily-cache regression test seeding a complete pre-fix v15 cache (unchanged savings hash and timezone, so nothing but the version bump can invalidate it) and proving the bump forces the re-derive that lands the corrected kiro cost — while the v15 file is never rewritten.
This commit is contained in:
ozymandiashh
parent
c49fa23590
commit
bc5a85df3e
9 changed files with 375 additions and 11 deletions
|
|
@ -4,6 +4,7 @@
|
|||
|
||||
### Fixed
|
||||
- Claude Desktop and Cowork sessions are discovered for Windows Microsoft Store (MSIX) installs. (#611)
|
||||
- **Kiro chat sessions no longer under-report input.** The token estimate for chat-file sessions (legacy `.chat` and chat-shaped IDE files) now counts every human turn's full text instead of only the last 500 characters, so multi-turn sessions and long prompts stop under-reporting input (and cost) severalfold. The estimate still counts human turns only — tool and system content, and resent context, are not included — and the summary still shows the last prompt, capped at 500 chars. Already-cached sessions re-parse once so the corrected numbers replace the old ones, and the daily cache bumps to v17 with this fix (v16 was skipped — main already claimed it for the codex structural-discovery fix), so finalized day totals re-derive too — for days whose kiro chat files still exist: since the v14 NEVER-LOSE carry-forward, a bump re-derives what sources survive and carries every sourceless (day, provider) slice forward with its old values, so a day whose kiro chat files have since been deleted keeps the pre-fix cost after the bump (minor in practice, since kiro IDE sources usually persist, but a carry-forward, not a refresh). (#909)
|
||||
|
||||
## 0.9.19 - 2026-07-20
|
||||
|
||||
|
|
|
|||
|
|
@ -57,7 +57,7 @@ The stores are disjoint (v2 sessions use `sess_`-prefixed IDs in a separate dire
|
|||
- **Model ID normalization.** Kiro stores models like `claude-1.2`; the parser rewrites the dot to a hyphen so they match `claude-1-2` in the pricing snapshot. Add new versions here when Kiro ships them.
|
||||
- **Tool name extraction accepts text and structured calls.** Kiro can embed tool calls inside message text as `<tool_use><name>...</name>` or expose structured `toolCalls` / `tool_calls` / `tools` entries.
|
||||
- Token counts are estimated via char count (`CHARS_PER_TOKEN = 4`).
|
||||
- **Credits are the cost source; tokens stay estimated.** Kiro bills in credits ($20/mo for 1,000; overage $0.04/credit). CLI (`metering_usage`), v1 executions (`usageSummary[].usage`), and v2 (`usage_summary.promptTurnSummaries[].usage`) turns record real credits, converted to USD at `USD_PER_KIRO_CREDIT = 0.04` (the public overage rate — the same never-understate approach as Codebuff). Turns without credit data fall back to token-estimated cost (`costIsEstimated: true`); legacy `.chat` and workspace-session records carry no usage data, so they are always token-estimated. Note: an earlier CLI implementation summed credit values directly as dollars, overstating cost 25×. Token *counts* remain char-estimated everywhere (input undercounts: only visible transcript text is seen, not the full resent context; v2's `session_metadata.contextUsage.usagePercentage` × context window is a better input proxy if ever needed). v2 does keep the real `modelId`, so unlike the v1 execution-file path it is not mislabeled `kiro-auto`.
|
||||
- **Credits are the cost source; tokens stay estimated.** Kiro bills in credits ($20/mo for 1,000; overage $0.04/credit). CLI (`metering_usage`), v1 executions (`usageSummary[].usage`), and v2 (`usage_summary.promptTurnSummaries[].usage`) turns record real credits, converted to USD at `USD_PER_KIRO_CREDIT = 0.04` (the public overage rate — the same never-understate approach as Codebuff). Turns without credit data fall back to token-estimated cost (`costIsEstimated: true`); legacy `.chat` and workspace-session records carry no usage data, so they are always token-estimated. Note: an earlier CLI implementation summed credit values directly as dollars, overstating cost 25×. Token *counts* remain char-estimated everywhere. The chat-file arm now sums every human turn's full text (identity preamble excluded) instead of only the last 500-char slice, so its input no longer undercounts for long or multi-turn chats; it still skips `tool`/`system` records, which the modern-execution, CLI, and v2 arms count as input (tool results are fed back to the model), and no arm sees the full resent context. v2's `session_metadata.contextUsage.usagePercentage` × context window is a better input proxy if ever needed. v2 does keep the real `modelId`, so unlike the v1 execution-file path it is not mislabeled `kiro-auto`.
|
||||
- **Cost is frozen at parse time.** Kiro is on the `costUSD` pass-through allowlist in `providerCallToCachedCall` (alongside mistral-vibe, devin, hermes, …), so its credit-based cost survives the session cache instead of being re-priced from estimated tokens — token re-pricing understated/overstated real kiro spend by up to 16× per model. The tradeoff, shared with all allowlisted providers: `codeburn price-override` and `model-alias` do not affect kiro dollar amounts (token *counts* are unaffected). Historical caches from before this change re-parse via the `CACHE_VERSION` bump to 5.
|
||||
|
||||
## When fixing a bug here
|
||||
|
|
|
|||
|
|
@ -5,7 +5,23 @@ import { homedir } from 'os'
|
|||
import { join } from 'path'
|
||||
import type { DateRange, ProjectSummary } from './types.js'
|
||||
|
||||
// Bumped to 15: per-project daily rollups. Days and provider slices now carry
|
||||
// Bumped to 17 — 16 is skipped on purpose: main already spent it on the codex
|
||||
// structural-discovery fix (eece4cf), so a v16 cache in the wild is a
|
||||
// main-owned cache meaning the codex fix, not this one. Claiming 16 here too
|
||||
// would let a user who has ever run a main build load its v16 cache as
|
||||
// CURRENT and COMPLETE, and this invalidation would never fire. 17 is the
|
||||
// first version that also means the kiro chat-file fix.
|
||||
//
|
||||
// kiro chat-file input tokens are now estimated from every human turn's full
|
||||
// text instead of the last 500 chars (#909, this PR), so a v15 rollup
|
||||
// finalized by the pre-fix binary carries kiro costs off by up to
|
||||
// severalfold. Nothing downstream can notice on its own: `usage-aggregator`
|
||||
// serves every day before today from this cache, and retention is ten years,
|
||||
// so an upgrading user with a warm complete cache would keep the stale
|
||||
// pre-fix kiro day totals forever while freshly reparsed sessions disagreed
|
||||
// with them. Raising MIN_SUPPORTED_VERSION forces the one-time re-derivation.
|
||||
//
|
||||
// v15: per-project daily rollups. Days and provider slices now carry
|
||||
// a `projects` breakdown (cost/calls/savings/sessions per project) so project
|
||||
// history outlives the session files, like models and categories already do.
|
||||
// This bump is the first to ride the v14 carry-forward: the old cache is
|
||||
|
|
@ -57,8 +73,8 @@ import type { DateRange, ProjectSummary } from './types.js'
|
|||
// that older binaries skipped. v8 added local-model savings to the daily
|
||||
// rollup; the `savingsConfigHash` field is invalidated separately when the
|
||||
// user changes their `localModelSavings` mapping.
|
||||
export const DAILY_CACHE_VERSION = 15
|
||||
const MIN_SUPPORTED_VERSION = 15
|
||||
export const DAILY_CACHE_VERSION = 17
|
||||
const MIN_SUPPORTED_VERSION = 17
|
||||
// Version-suffixed so different binaries each own a distinct file and never
|
||||
// clobber an incompatible schema. Bumping the version mints a fresh filename;
|
||||
// adoptOlderDailyCaches then unions days out of every previous file (including
|
||||
|
|
|
|||
|
|
@ -221,7 +221,11 @@ export const PROVIDER_PARSE_VERSIONS: Record<string, string> = {
|
|||
hermes: 'reasoning-output-accounting-v1-est-cost',
|
||||
'lingtai-tui': 'token-ledger-registry-activity-v3',
|
||||
'ibm-bob': 'worktree-project-grouping-v1',
|
||||
kiro: 'ide-parsing-v1-est-cost',
|
||||
// v2: chat-file input tokens now estimate from the FULL prompt (sum of every
|
||||
// human turn), not the last 500-char slice — so costUSD changed for chat-arm
|
||||
// sessions. Bump so already-cached kiro sessions re-parse once and the
|
||||
// corrected tokens/cost land instead of being served pre-fix forever.
|
||||
kiro: 'ide-parsing-v2-est-cost',
|
||||
quickdesk: 'emf-sqlite-v2-est-cost',
|
||||
kimicode: 'wire-usage-v1-est-cost',
|
||||
'kilo-code': 'worktree-project-grouping-v1',
|
||||
|
|
|
|||
|
|
@ -466,3 +466,79 @@ describe('ensureCacheHydrated: timezone invalidation', () => {
|
|||
expect(preserved.days[0]!.date).toBe(twoDaysAgoStr)
|
||||
})
|
||||
})
|
||||
|
||||
// A complete v15 cache is trusted as-is (unchanged savings hash, matching tz,
|
||||
// complete marker set) — so without the v17 bump, an upgrading user keeps the
|
||||
// pre-fix kiro day totals forever while freshly reparsed sessions disagree
|
||||
// with them. The bump mints a fresh filename, adoption marks the result
|
||||
// incomplete, and the next hydration re-derives the window; the corrected kiro
|
||||
// estimate (every human turn's full text, #909) wins wherever the sources
|
||||
// survive, and sourceless days carry forward under the v14 NEVER-LOSE rule.
|
||||
describe('ensureCacheHydrated: schema version invalidation (#909)', () => {
|
||||
function kiroDay(date: string, cost: number, calls: number, inputTokens: number): DailyEntry {
|
||||
return {
|
||||
date,
|
||||
cost,
|
||||
savingsUSD: 0,
|
||||
calls,
|
||||
sessions: 1,
|
||||
inputTokens,
|
||||
outputTokens: 200,
|
||||
cacheReadTokens: 0,
|
||||
cacheWriteTokens: 0,
|
||||
editTurns: 0,
|
||||
oneShotTurns: 0,
|
||||
models: {
|
||||
'claude-haiku-4-5': { calls, cost, savingsUSD: 0, inputTokens, outputTokens: 200, cacheReadTokens: 0, cacheWriteTokens: 0 },
|
||||
},
|
||||
categories: {},
|
||||
providers: {
|
||||
kiro: { cost, calls, savingsUSD: 0, sessions: 1, inputTokens, outputTokens: 200 },
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
it('re-derives a warm complete v15 cache instead of serving its pre-fix kiro totals', async () => {
|
||||
vi.useFakeTimers()
|
||||
vi.setSystemTime(new Date('2026-06-12T12:00:00.000Z'))
|
||||
|
||||
const { writeFile, mkdir } = await import('fs/promises')
|
||||
await mkdir(TMP_CACHE_ROOT, { recursive: true })
|
||||
// A cache exactly as a pre-fix release left it: current schema at the
|
||||
// time, finalized off a complete parse, watermark at yesterday, matching
|
||||
// tz and savings hash. Nothing but the version bump can invalidate it.
|
||||
const v15 = {
|
||||
version: 15,
|
||||
savingsConfigHash: '',
|
||||
tzKey: currentTzKey(),
|
||||
lastComputedDate: '2026-06-11',
|
||||
days: [kiroDay('2026-06-11', 4.55, 1, 500)],
|
||||
complete: true,
|
||||
}
|
||||
await writeFile(join(TMP_CACHE_ROOT, 'daily-cache.v15.json'), JSON.stringify(v15), 'utf-8')
|
||||
|
||||
let parseCalls = 0
|
||||
const hydrated = await ensureCacheHydrated(
|
||||
async () => {
|
||||
parseCalls += 1
|
||||
return []
|
||||
},
|
||||
// The corrected kiro accounting: 750 input tokens (3000 chars / 4)
|
||||
// instead of the 500-char-slice 125, i.e. the full-prompt estimate.
|
||||
() => [kiroDay('2026-06-11', 18.2, 1, 750)],
|
||||
)
|
||||
|
||||
// The whole point: the window is re-parsed rather than served frozen.
|
||||
expect(parseCalls).toBe(1)
|
||||
// ...and the fresh derivation wins over the stale v15 day, at the
|
||||
// provider-slice level where the pre-fix kiro cost actually lived.
|
||||
const day = hydrated.days.find(d => d.date === '2026-06-11')
|
||||
expect(day?.cost).toBe(18.2)
|
||||
expect(day?.providers['kiro']?.cost).toBe(18.2)
|
||||
expect(day?.providers['kiro']?.inputTokens).toBe(750)
|
||||
expect(hydrated.version).toBe(DAILY_CACHE_VERSION)
|
||||
expect(hydrated.complete).toBe(true)
|
||||
// The v15 file is never rewritten or deleted — old binaries still own it.
|
||||
expect(JSON.parse(await readFile(join(TMP_CACHE_ROOT, 'daily-cache.v15.json'), 'utf-8')).version).toBe(15)
|
||||
})
|
||||
})
|
||||
|
|
|
|||
|
|
@ -57,11 +57,18 @@ function kiroAgentDir(): string {
|
|||
|
||||
// What computeEnvFingerprint('kiro') returned before kiro had an entry in
|
||||
// PROVIDER_PARSE_VERSIONS: no env vars, no parser version, i.e. a hash of
|
||||
// zero parts. This is the fingerprint sitting in every pre-fix cache.
|
||||
// zero parts. This is the fingerprint sitting in every pre-registration cache.
|
||||
function preFixFingerprint(): string {
|
||||
return createHash('sha256').update([].join('\0')).digest('hex').slice(0, 16)
|
||||
}
|
||||
|
||||
// What computeEnvFingerprint('kiro') returned under the FIRST parser version
|
||||
// ('ide-parsing-v1-est-cost'): the fingerprint sitting in every cache written
|
||||
// by the release that shipped the 500-char-slice input-token estimate.
|
||||
function v1Fingerprint(): string {
|
||||
return createHash('sha256').update('parser=ide-parsing-v1-est-cost').digest('hex').slice(0, 16)
|
||||
}
|
||||
|
||||
// Writes one IDE execution file in the context.messages[].entries format that
|
||||
// the pre-fix parser turned into 0 turns, and returns its path.
|
||||
async function seedExecutionFile(): Promise<string> {
|
||||
|
|
@ -84,6 +91,33 @@ async function seedExecutionFile(): Promise<string> {
|
|||
return path
|
||||
}
|
||||
|
||||
// Writes one chat-shaped IDE file (chat array + metadata) with a 3000-char
|
||||
// human prompt, and returns its path. Routes to decodeKiroChatFile.
|
||||
async function seedChatFile(): Promise<string> {
|
||||
const dir = join(kiroAgentDir(), 'c'.repeat(32))
|
||||
await mkdir(dir, { recursive: true })
|
||||
const path = join(dir, 'chat-stale-001.chat')
|
||||
await writeFile(path, JSON.stringify({
|
||||
executionId: 'exec-chat-stale-001',
|
||||
actionId: 'act',
|
||||
chat: [
|
||||
{ role: 'human', content: '<identity>\nYou are Kiro.\n</identity>' },
|
||||
{ role: 'bot', content: 'I will follow these instructions.' },
|
||||
{ role: 'human', content: 'x'.repeat(3000) },
|
||||
{ role: 'bot', content: 'short' },
|
||||
],
|
||||
metadata: {
|
||||
modelId: 'claude-haiku-4-5',
|
||||
modelProvider: 'qdev',
|
||||
workflow: 'act',
|
||||
workflowId: 'wf-chat-stale-001',
|
||||
startTime: 1780000000000,
|
||||
endTime: 1780000001000,
|
||||
},
|
||||
}))
|
||||
return path
|
||||
}
|
||||
|
||||
async function seedCache(execPath: string, envFingerprint: string): Promise<void> {
|
||||
const fp = await fingerprintFile(execPath)
|
||||
if (!fp) throw new Error('failed to fingerprint seeded execution file')
|
||||
|
|
@ -102,6 +136,60 @@ async function seedCache(execPath: string, envFingerprint: string): Promise<void
|
|||
await writeFile(sessionCachePath(), JSON.stringify(cache))
|
||||
}
|
||||
|
||||
// Seeds the same chat file as seedChatFile, but cached under the given
|
||||
// fingerprint with a turn carrying the PRE-FIX token estimate (125 = the
|
||||
// 500-char slice of the 3000-char prompt). If the cache is served as-is, the
|
||||
// stale estimate survives; a fingerprint bump discards the section and the
|
||||
// corrected math (3000 / 4 = 750) lands on re-parse.
|
||||
async function seedChatCache(envFingerprint: string): Promise<string> {
|
||||
const path = await seedChatFile()
|
||||
const fp = await fingerprintFile(path)
|
||||
if (!fp) throw new Error('failed to fingerprint seeded chat file')
|
||||
const cache: SessionCache = {
|
||||
version: CACHE_VERSION,
|
||||
providers: {
|
||||
kiro: {
|
||||
envFingerprint,
|
||||
files: {
|
||||
[path]: {
|
||||
fingerprint: fp,
|
||||
mcpInventory: [],
|
||||
turns: [{
|
||||
timestamp: '2026-05-27T00:00:00.000Z',
|
||||
sessionId: 'wf-chat-stale-001',
|
||||
userMessage: 'x'.repeat(500),
|
||||
calls: [{
|
||||
provider: 'kiro',
|
||||
model: 'claude-haiku-4-5',
|
||||
usage: {
|
||||
inputTokens: 125,
|
||||
outputTokens: 2,
|
||||
cacheCreationInputTokens: 0,
|
||||
cacheReadInputTokens: 0,
|
||||
cachedInputTokens: 0,
|
||||
reasoningTokens: 0,
|
||||
webSearchRequests: 0,
|
||||
cacheCreationOneHourTokens: 0,
|
||||
},
|
||||
speed: 'standard',
|
||||
timestamp: '2026-05-27T00:00:00.000Z',
|
||||
tools: [],
|
||||
bashCommands: [],
|
||||
skills: [],
|
||||
subagentTypes: [],
|
||||
deduplicationKey: 'kiro:wf-chat-stale-001:exec-chat-stale-001',
|
||||
}],
|
||||
}],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
await mkdir(CACHE_DIR, { recursive: true })
|
||||
await writeFile(sessionCachePath(), JSON.stringify(cache))
|
||||
return path
|
||||
}
|
||||
|
||||
async function parseKiroCalls() {
|
||||
const projects = await parseAllSessions(undefined, 'kiro')
|
||||
return projects
|
||||
|
|
@ -127,6 +215,26 @@ describe('Kiro session cache invalidation', () => {
|
|||
expect(computeEnvFingerprint('kiro')).not.toBe(preFixFingerprint())
|
||||
})
|
||||
|
||||
it('bumps the kiro parser version again for the full-prompt input-token fix', () => {
|
||||
// The v1 fingerprint is what the release that shipped the 500-char-slice
|
||||
// estimate wrote into every user cache. It must NOT match the current one,
|
||||
// or cached chat files would keep the pre-fix 125-token estimate forever.
|
||||
expect(computeEnvFingerprint('kiro')).not.toBe(v1Fingerprint())
|
||||
})
|
||||
|
||||
it('control: a chat cache at the CURRENT fingerprint is honored (stale 125 stays)', async () => {
|
||||
await seedChatCache(computeEnvFingerprint('kiro'))
|
||||
|
||||
const calls = await parseKiroCalls()
|
||||
|
||||
// The seeded cache is structurally valid and trusted: the unchanged chat
|
||||
// file is not re-parsed, so the pre-fix 125-token estimate survives
|
||||
// verbatim. This proves the seed is real (not silently ignored) — and that
|
||||
// WITHOUT a fingerprint bump, the stale estimate would be served forever.
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]!.usage.inputTokens).toBe(125)
|
||||
})
|
||||
|
||||
it('control: a zero-turn cache entry at the current fingerprint is honored', async () => {
|
||||
const execPath = await seedExecutionFile()
|
||||
await seedCache(execPath, computeEnvFingerprint('kiro'))
|
||||
|
|
@ -140,6 +248,18 @@ describe('Kiro session cache invalidation', () => {
|
|||
expect(calls).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('regression: a v1 cache fingerprint forces a re-parse that lands the corrected tokens', async () => {
|
||||
await seedChatCache(v1Fingerprint())
|
||||
|
||||
const calls = await parseKiroCalls()
|
||||
|
||||
// The v1 fingerprint no longer matches, the stale section is discarded,
|
||||
// the unchanged chat file re-parses, and the full-prompt estimate (3000
|
||||
// chars / 4 = 750) replaces the cached 125 — the money-path correction.
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]!.usage.inputTokens).toBe(750)
|
||||
})
|
||||
|
||||
it('regression: a pre-fix cache fingerprint forces a re-parse that recovers the calls', async () => {
|
||||
const execPath = await seedExecutionFile()
|
||||
await seedCache(execPath, preFixFingerprint())
|
||||
|
|
|
|||
|
|
@ -340,11 +340,11 @@ describe('kiro golden pins (raw calls, unmodified provider)', () => {
|
|||
expect(blankCalls[0]!.deduplicationKey).toBe('kiro:wf-g1b2:')
|
||||
})
|
||||
|
||||
it('G2 — A1 input tokens derived from truncated prompt; A2 from full prompt', async () => {
|
||||
it('G2 — A1 input tokens from the full prompt (500 cap is display-only); A2 modern arm from full prompt', async () => {
|
||||
const wsHashA1 = 'a'.repeat(32)
|
||||
const wsDirA1 = join(tmpDir, 'g2a1', wsHashA1)
|
||||
await mkdir(wsDirA1, { recursive: true })
|
||||
const chatPath = join(wsDirA1, 'trunc.chat')
|
||||
const chatPath = join(wsDirA1, 'long.chat')
|
||||
await writeFile(chatPath, makeChatFile({
|
||||
executionId: 'exec-g2a1',
|
||||
workflowId: 'wf-g2a1',
|
||||
|
|
@ -353,7 +353,13 @@ describe('kiro golden pins (raw calls, unmodified provider)', () => {
|
|||
}))
|
||||
const a1 = await parseSource({ path: chatPath, project: 'p', provider: 'kiro' })
|
||||
expect(a1).toHaveLength(1)
|
||||
expect(a1[0]!.inputTokens).toBe(125)
|
||||
// 3000 fixture chars / 4 per token = 750. The prior pin (125) encoded the
|
||||
// pre-fix behaviour: input tokens were estimated from the last human turn
|
||||
// sliced to 500 chars (500 / 4 = 125), under-reporting cost for any longer
|
||||
// prompt. That bug was fixed by porting upstream 6c4645a ('fix(kiro):
|
||||
// estimate input tokens from the full prompt, not a 500-char slice');
|
||||
// do not restore 125.
|
||||
expect(a1[0]!.inputTokens).toBe(750)
|
||||
expect(a1[0]!.userMessage.length).toBe(500)
|
||||
|
||||
const wsHashA2 = 'b'.repeat(32)
|
||||
|
|
@ -371,6 +377,28 @@ describe('kiro golden pins (raw calls, unmodified provider)', () => {
|
|||
expect(a2[0]!.inputTokens).toBe(500)
|
||||
})
|
||||
|
||||
it('G2b — A1 rounding is pinned: an odd length forces ceil, not round/floor', async () => {
|
||||
// 3000 (G2), 2400 (money-path regression) and 1000+1000 (multi-turn
|
||||
// accumulation) are all exact multiples of four, so they cannot
|
||||
// distinguish ceil from round or floor in estimateTokensFromChars.
|
||||
// 3001 chars / 4 = 750.25: ceil gives 751, round and floor both give
|
||||
// 750 — only the ceil pin passes here.
|
||||
const wsHash = 'd'.repeat(32)
|
||||
const wsDir = join(tmpDir, 'g2b', wsHash)
|
||||
await mkdir(wsDir, { recursive: true })
|
||||
const chatPath = join(wsDir, 'odd.chat')
|
||||
await writeFile(chatPath, makeChatFile({
|
||||
executionId: 'exec-g2b',
|
||||
workflowId: 'wf-g2b',
|
||||
userPrompt: 'x'.repeat(3001),
|
||||
botResponses: ['short'],
|
||||
}))
|
||||
const calls = await parseSource({ path: chatPath, project: 'p', provider: 'kiro' })
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]!.inputTokens).toBe(751)
|
||||
expect(calls[0]!.userMessage.length).toBe(500)
|
||||
})
|
||||
|
||||
it('G3 — A1 single-tool toolSequence is present but undefined', async () => {
|
||||
const wsHash = 'c'.repeat(32)
|
||||
const wsDir = join(tmpDir, 'g3', wsHash)
|
||||
|
|
|
|||
|
|
@ -220,6 +220,104 @@ describe('kiro provider - chat file parsing', () => {
|
|||
expect(calls[0]!.outputTokens).toBe(109)
|
||||
})
|
||||
|
||||
it('estimates input tokens from the full prompt, not a 500-char slice (money-path)', async () => {
|
||||
// Regression: decodeKiroChatFile estimated input tokens from
|
||||
// pendingUserMessage (the last human turn sliced to 500 chars) while
|
||||
// output summed every bot char, so a long prompt undercounted input
|
||||
// tokens - and cost - severalfold.
|
||||
const wsHash = 'n'.repeat(32)
|
||||
const wsDir = join(tmpDir, wsHash)
|
||||
await mkdir(wsDir, { recursive: true })
|
||||
const chatPath = join(wsDir, 'long.chat')
|
||||
const prompt = 'x'.repeat(2400) // 2400 chars / 4 = 600 tokens; a 500-slice would give 125
|
||||
await writeFile(chatPath, makeChatFile({ userPrompt: prompt, botResponses: ['ok'] }))
|
||||
|
||||
const source = { path: chatPath, project: 'test', provider: 'kiro' }
|
||||
const calls: ParsedProviderCall[] = []
|
||||
for await (const call of kiro.createSessionParser(source, new Set()).parse()) calls.push(priceProviderCall(call))
|
||||
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]!.inputTokens).toBe(600)
|
||||
// userMessage stays capped for display; only the token estimate uses the full length.
|
||||
expect(calls[0]!.userMessage.length).toBe(500)
|
||||
})
|
||||
|
||||
it('accumulates EVERY human turn - including an identical resubmit - and skips identity messages', async () => {
|
||||
// A resubmitted prompt is a real second model input (the model consumed
|
||||
// its tokens twice), so it must be counted - the same semantics every
|
||||
// other kiro arm uses (modern-execution, ws-session, cli, v2 all sum
|
||||
// every human/user message). The one non-input human message, the
|
||||
// `<identity>` system preamble, stays excluded.
|
||||
const wsHash = 'o'.repeat(32)
|
||||
const wsDir = join(tmpDir, wsHash)
|
||||
await mkdir(wsDir, { recursive: true })
|
||||
const chatPath = join(wsDir, 'multi.chat')
|
||||
const prompt = 'y'.repeat(1000) // 1000 chars / 4 = 250 tokens
|
||||
const chat = [
|
||||
{ role: 'human', content: '<identity>\nYou are Kiro.\n</identity>' }, // excluded
|
||||
{ role: 'bot', content: 'I will follow these instructions.' },
|
||||
{ role: 'human', content: prompt }, // 250
|
||||
{ role: 'bot', content: 'first answer' },
|
||||
{ role: 'human', content: prompt }, // resubmit - identical text, counted again: +250
|
||||
{ role: 'bot', content: 'second answer' },
|
||||
]
|
||||
await writeFile(chatPath, JSON.stringify({
|
||||
executionId: 'exec-multi-001',
|
||||
actionId: 'act',
|
||||
chat,
|
||||
metadata: { modelId: 'claude-haiku-4-5', modelProvider: 'qdev', workflow: 'act', workflowId: 'wf-multi-001', startTime: 1777333000000, endTime: 1777333010000 },
|
||||
}))
|
||||
|
||||
const source = { path: chatPath, project: 'test', provider: 'kiro' }
|
||||
const calls: ParsedProviderCall[] = []
|
||||
for await (const call of kiro.createSessionParser(source, new Set()).parse()) calls.push(priceProviderCall(call))
|
||||
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]!.inputTokens).toBe(500) // 250 + 250, identity not counted
|
||||
// Display shows the LAST human turn, capped at 500.
|
||||
expect(calls[0]!.userMessage).toBe(prompt.slice(0, 500))
|
||||
})
|
||||
|
||||
it('excludes a near-miss identity preamble (leading newline / BOM) instead of counting it as input', async () => {
|
||||
// The <identity> preamble is a large system-injected human record. With
|
||||
// the full-length accumulator, a near-miss match - a leading newline or
|
||||
// BOM before the tag - would silently add the preamble's whole length to
|
||||
// input tokens on every affected session. decodeKiroChatFile trims
|
||||
// leading whitespace before the startsWith match; pin that here so a
|
||||
// regression cannot re-inflate tokens. (A renamed preamble remains a
|
||||
// residual risk: no prefix match can catch it, and it is out of scope.)
|
||||
const wsHash = 'r'.repeat(32)
|
||||
const wsDir = join(tmpDir, wsHash)
|
||||
await mkdir(wsDir, { recursive: true })
|
||||
const chatPath = join(wsDir, 'near-miss.chat')
|
||||
const prompt = 'z'.repeat(1000) // 1000 chars / 4 = 250 tokens
|
||||
const chat = [
|
||||
{ role: 'human', content: '\n<identity>\nYou are Kiro.\n</identity>' }, // leading newline - still excluded
|
||||
{ role: 'bot', content: 'I will follow these instructions.' },
|
||||
{ role: 'human', content: '\uFEFF<identity>\nYou are Kiro.\n</identity>' }, // BOM - still excluded
|
||||
{ role: 'bot', content: 'I will follow these instructions too.' },
|
||||
{ role: 'human', content: prompt },
|
||||
{ role: 'bot', content: 'answer' },
|
||||
]
|
||||
await writeFile(chatPath, JSON.stringify({
|
||||
executionId: 'exec-near-001',
|
||||
actionId: 'act',
|
||||
chat,
|
||||
metadata: { modelId: 'claude-haiku-4-5', modelProvider: 'qdev', workflow: 'act', workflowId: 'wf-near-001', startTime: 1777333000000, endTime: 1777333010000 },
|
||||
}))
|
||||
|
||||
const source = { path: chatPath, project: 'test', provider: 'kiro' }
|
||||
const calls: ParsedProviderCall[] = []
|
||||
for await (const call of kiro.createSessionParser(source, new Set()).parse()) calls.push(priceProviderCall(call))
|
||||
|
||||
expect(calls).toHaveLength(1)
|
||||
// Only the real prompt counts: 250. If either near-miss preamble were
|
||||
// treated as input, tokens would jump by both preamble lengths.
|
||||
expect(calls[0]!.inputTokens).toBe(250)
|
||||
// Display shows the last human turn, capped at 500.
|
||||
expect(calls[0]!.userMessage).toBe(prompt.slice(0, 500))
|
||||
})
|
||||
|
||||
it('normalizes dot-versioned model IDs to dashes', async () => {
|
||||
const wsHash = 'h'.repeat(32)
|
||||
const wsDir = join(tmpDir, wsHash)
|
||||
|
|
|
|||
|
|
@ -186,12 +186,33 @@ export function decodeKiroChatFile(input: {
|
|||
if (modelId === 'auto' || !modelId) modelId = 'kiro-auto'
|
||||
|
||||
let pendingUserMessage = ''
|
||||
// Accumulate every human turn's full length for the input-token estimate,
|
||||
// mirroring the modern-execution path's inputChars accumulator. The prior
|
||||
// code estimated input tokens from pendingUserMessage.length alone - the
|
||||
// LAST human turn truncated to 500 chars - so a multi-turn session, or any
|
||||
// final prompt over 500 chars, undercounted input tokens (and therefore
|
||||
// costUSD) severalfold, while output correctly summed all bot chars. Note
|
||||
// the chat arm still counts only human records: tool and system records
|
||||
// stay excluded, unlike the modern-execution, CLI and v2 arms (whose
|
||||
// comments state tool results are fed back to the model as input), so this
|
||||
// arm still under-reports - just far less than before.
|
||||
let inputChars = 0
|
||||
const allTools: string[] = []
|
||||
const toolSequence: KiroToolCall[][] = []
|
||||
|
||||
for (const msg of chat) {
|
||||
if (msg.role === 'human') {
|
||||
if (msg.content.startsWith('<identity>')) continue
|
||||
// The <identity> system preamble is the one non-input human record.
|
||||
// Trim leading whitespace before matching: preambles are large, and
|
||||
// with the full-length accumulator below a missed match (a leading
|
||||
// newline or BOM before the tag) would silently add the preamble's
|
||||
// whole length to input tokens on every affected session. Tolerating
|
||||
// leading whitespace costs nothing real - a genuine prompt never
|
||||
// starts with whitespace plus an identity tag - and the failure
|
||||
// asymmetry favours exclusion: a false negative inflates tokens by
|
||||
// the preamble length, a false positive only skips a preamble.
|
||||
if (msg.content.trimStart().startsWith('<identity>')) continue
|
||||
inputChars += msg.content.length
|
||||
pendingUserMessage = msg.content.slice(0, 500)
|
||||
}
|
||||
if (msg.role === 'bot') {
|
||||
|
|
@ -210,7 +231,7 @@ export function decodeKiroChatFile(input: {
|
|||
if (seen.has(dedupKey)) return { calls, diagnostics: [] }
|
||||
|
||||
const outputTokens = estimateTokensFromChars(totalOutputChars)
|
||||
const inputTokens = estimateTokensFromChars(pendingUserMessage.length)
|
||||
const inputTokens = estimateTokensFromChars(inputChars)
|
||||
const tsDate = parseKiroTimestamp(metadata.startTime)
|
||||
if (!tsDate) return { calls, diagnostics: [] }
|
||||
const timestamp = tsDate.toISOString()
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue