mirror of
https://github.com/AgentSeal/codeburn.git
synced 2026-10-21 05:23:19 +00:00
Fix interleaved DSH accounting and preserve inclusive output
This commit is contained in:
xz
parent
a2f1fe2007
commit
cd2e0dd43a
8 changed files with 115 additions and 36 deletions
|
|
@ -72,18 +72,20 @@ None at the provider level; the log file is the cached source path and the norma
|
|||
|
||||
- **DSH is a developer preview.** The parser explicitly supports released formats v0-v3 and checks both the canonical generation filename and header. A future version is skipped with a notice; **a version bump upstream still requires a semantic reader update, not just relaxing the check.**
|
||||
- **The JSONL backend only.** DSH also ships an opt-in SQLite persistence backend (`@deepseek-ai/dsh-session-persistence-sqlite`); it is not the default and is not read.
|
||||
- **DSH records tokens, never dollars.** `usage` is `{ inputTokens, outputTokens, cacheReadTokens?, cacheWriteTokens?, reasoningTokens? }` with no cost field, so every call is priced from the shared tables. The buckets are disjoint on input; `reasoningTokens` is informational detail already included in `outputTokens`, so CodeBurn subtracts it from ordinary output and retains it as a separate reasoning bucket for shared pricing and display. pi-ai routes do not persist separate reasoning detail. Complete valid usage keeps `costIsEstimated` false; incomplete or inconsistent usage is reported with a notice and marked estimated. An attempt without usage is omitted with a notice rather than represented as an exact zero.
|
||||
- **DSH records tokens, never dollars.** `usage` is `{ inputTokens, outputTokens, cacheReadTokens?, cacheWriteTokens?, reasoningTokens? }` with no cost field, so every call is priced from the shared tables. The buckets are disjoint on input; `reasoningTokens` is informational detail already included in `outputTokens`, as documented in the [DSH TokenUsage contract](https://github.com/deepseek-ai/deepseek-harness/blob/c291e7961a515f6d7af9304e7fd1d257929aef26/docs/subsystems/llm-streaming.md#tokenusage). CodeBurn preserves raw output and applies the shared inclusive-output rule to pricing, cached reads, and display. pi-ai routes do not persist separate reasoning detail. Complete valid usage keeps `costIsEstimated` false; incomplete or inconsistent usage is reported with a notice and marked estimated. An attempt without usage is omitted with a notice rather than represented as an exact zero.
|
||||
- **`assistant/message` usage wins over the `assistant/chunk` sample** for the same `(turn, step)` — the two are adjacent reports of one API call, not two calls. A late chunk never overwrites a final report, so the two are never summed.
|
||||
- **The model comes from the message, not the request.** `data.message.source.model` is what actually served the step; the current `request/context` model is the fallback, followed by `request/header`. A changed header model clears the previous context fallback. The `provider` field there (`deepseek-official`) is the upstream LLM route, not the tool — the codeburn provider name is always `dsh`.
|
||||
- **A forked session's log replays its parent's events.** v0/v1 use header `seedLength`; v2/v3 use the last `session/end-seed` marker carrying `{ inherited: true }`. CodeBurn excludes the inherited prefix to avoid billing the parent's calls twice.
|
||||
- **A forked session's log replays its parent's events.** v0/v1 use header `seedLength` only when `parentSession` is present, preserving the legacy non-fork behavior; v2/v3 use the last `session/end-seed` marker carrying `{ inherited: true }`. CodeBurn excludes the inherited prefix to avoid billing the parent's calls twice.
|
||||
- **`user/message` also carries agent-injected context** (runtime snapshots, skill bodies, file-change notices) under `source.kind: 'plugin'`. Only `kind: 'user'` messages become the preview.
|
||||
- **Delta chunks are packed.** Runs of streamed deltas are stored as `text-chunks` / `reasoning-chunks` / `tool-call-chunks` storage rows rather than one event per line. They carry no usage and no tool identity the `tool/call` event lacks, so they are ignored — as is any event type the parser does not know.
|
||||
- **A torn final zstd frame is ignored.** A crashed writer leaves an incomplete trailing frame; the complete frames before it parse normally. A structurally corrupt file is skipped whole with a notice rather than throwing.
|
||||
|
||||
## When fixing a bug here
|
||||
|
||||
`v3-retry.jsonl` also covers a failed `assistant/attempt`, scheduled retry, and successful settlement with exact `totalTokens`. The same official strict restore and reducer yield input 110, output 24, cache read 33, and cache write 7 (174 total tokens).
|
||||
|
||||
1. Reproduce with a minimal session dir: `sessions/--proj--/<id>/session.jsonl` (uncompressed is easiest to hand-write).
|
||||
2. `tests/fixtures/dsh/bash-tool-turn.jsonl` is the upstream `examples/acp-agent/tests/snapshots/bash-tool-turn/session.jsonl` snapshot with its template placeholders filled in — refresh it from the DSH repo when the format moves.
|
||||
3. Run `tests/providers/dsh.test.ts`.
|
||||
4. `.zstd` fixtures must compress **each batch separately**; one `zstdCompressSync` over the whole file is a single-frame layout DSH never writes.
|
||||
5. `tests/fixtures/dsh/v0.jsonl` through `v3.jsonl` are minimal synthetic, sanitized format fixtures. Each was restored with DSH's official `sessionFormatCatalog` (`recovery: 'strict', validation: 'current'`) and folded through `tokenUsageProjectionDefinition` at DSH commit `2ad76263a53497763575b5ad4ca5acf67ae00987`. All four yield uncached input 100, full output 20, cache read 30, and cache write 5; the provider tests assert those buckets, including the output/reasoning split. v0/v1 cite their top-level chunk via `sourceEventSeqs`; v2/v3 carry the embedded stream.
|
||||
5. `tests/fixtures/dsh/v0.jsonl` through `v3.jsonl` are minimal synthetic, sanitized format fixtures. Each was restored with DSH's official `sessionFormatCatalog` (`recovery: 'strict', validation: 'current'`) and folded through `tokenUsageProjectionDefinition` at DSH commit `c291e7961a515f6d7af9304e7fd1d257929aef26`. All four yield uncached input 100, full output 20, cache read 30, and cache write 5; the provider tests assert those buckets, including inclusive output and informational reasoning detail. v0/v1 cite their top-level chunk via `sourceEventSeqs`; v2/v3 carry the embedded stream.
|
||||
|
|
|
|||
|
|
@ -33,7 +33,9 @@ export type ModelCosts = {
|
|||
/// and nothing else, and its supplementary store-row/shutdown calls carry
|
||||
/// reasoningTokens with outputTokens 0 while the per-turn assistant.message call
|
||||
/// bills the full output, so adding reasoning on top bills it twice.
|
||||
const REASONING_INCLUDED_IN_OUTPUT = new Set(['claude', 'codex', 'copilot'])
|
||||
/// DSH TokenUsage includes reasoning in output too; see the pinned contract:
|
||||
/// https://github.com/deepseek-ai/deepseek-harness/blob/c291e7961a515f6d7af9304e7fd1d257929aef26/docs/subsystems/llm-streaming.md#tokenusage
|
||||
const REASONING_INCLUDED_IN_OUTPUT = new Set(['claude', 'codex', 'copilot', 'dsh'])
|
||||
|
||||
/// Output tokens to bill and display for one call. Single source of truth so
|
||||
/// the pricing sites and the display sums can never disagree about whether a
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ import { homedir } from 'os'
|
|||
import zlib from 'zlib'
|
||||
|
||||
import { MAX_SESSION_FILE_BYTES, readSessionFile, readSessionLines } from '../fs-utils.js'
|
||||
import { calculateCost, getShortModelName } from '../models.js'
|
||||
import { billableOutputTokens, calculateCost, getShortModelName } from '../models.js'
|
||||
import { extractBashCommands } from '../bash-utils.js'
|
||||
import type { ProbeRoot, Provider, SessionSource, SessionParser, ParsedProviderCall } from './types.js'
|
||||
|
||||
|
|
@ -486,7 +486,8 @@ function createParser(source: SessionSource, seenKeys: Set<string>): SessionPars
|
|||
// Events a forked session inherited from its parent. They are a verbatim
|
||||
// copy of the parent's log, which codeburn parses as its own session, so
|
||||
// counting them here would bill the same calls twice.
|
||||
let inheritedCut = formatVersion <= 1 && typeof header.seedLength === 'number'
|
||||
let inheritedCut = formatVersion <= 1 && typeof header.parentSession === 'string' && header.parentSession
|
||||
&& typeof header.seedLength === 'number'
|
||||
? header.seedLength - 1
|
||||
: -1
|
||||
if (formatVersion >= 2) {
|
||||
|
|
@ -505,7 +506,7 @@ function createParser(source: SessionSource, seenKeys: Set<string>): SessionPars
|
|||
}
|
||||
const userMessageByTurn = new Map<number, string>()
|
||||
const buckets = new Map<string, StepBucket>()
|
||||
const projectionState: { last: { key: string; index: number } | null } = { last: null }
|
||||
const activeAttempts = new Map<string, number>()
|
||||
|
||||
for (const event of events) {
|
||||
if (event.type === 'session') {
|
||||
|
|
@ -541,7 +542,7 @@ function createParser(source: SessionSource, seenKeys: Set<string>): SessionPars
|
|||
if (event.type === 'llm/retry-started') {
|
||||
const turn = event.data?.turn ?? currentTurn
|
||||
const step = event.data?.step ?? 0
|
||||
if (projectionState.last?.key === `${turn}:${step}`) projectionState.last = null
|
||||
activeAttempts.delete(`${turn}:${step}`)
|
||||
continue
|
||||
}
|
||||
|
||||
|
|
@ -607,7 +608,7 @@ function createParser(source: SessionSource, seenKeys: Set<string>): SessionPars
|
|||
}
|
||||
const key = `${turn}:${step}`
|
||||
if (!usage) {
|
||||
if (projectionState.last?.key !== key) {
|
||||
if (!activeAttempts.has(key)) {
|
||||
notice(`codeburn: DSH session contains an attempt without usage; totals may be incomplete: ${source.path}\n`)
|
||||
}
|
||||
continue
|
||||
|
|
@ -618,17 +619,16 @@ function createParser(source: SessionSource, seenKeys: Set<string>): SessionPars
|
|||
buckets.set(key, bucket)
|
||||
}
|
||||
const observation = { usage, time: event.time, model: reportedModel, final: isFinal }
|
||||
// This is the official tokenUsage projection's replacement slot: a
|
||||
// settlement for the same (turn, step) replaces the immediately prior
|
||||
// sample, while llm/retry-started closes the slot so the next attempt
|
||||
// is additive.
|
||||
if (projectionState.last?.key === key) {
|
||||
if (!bucket.observations[projectionState.last.index]?.final || isFinal) {
|
||||
bucket.observations[projectionState.last.index] = observation
|
||||
// A different step cannot close this step's replacement slot. Only
|
||||
// its own retry-started event makes the next observation additive.
|
||||
const activeIndex = activeAttempts.get(key)
|
||||
if (activeIndex !== undefined) {
|
||||
if (!bucket.observations[activeIndex]?.final || isFinal) {
|
||||
bucket.observations[activeIndex] = observation
|
||||
}
|
||||
} else {
|
||||
bucket.observations.push(observation)
|
||||
projectionState.last = { key, index: bucket.observations.length - 1 }
|
||||
activeAttempts.set(key, bucket.observations.length - 1)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -659,16 +659,16 @@ function createParser(source: SessionSource, seenKeys: Set<string>): SessionPars
|
|||
seenKeys.add(dedupKey)
|
||||
|
||||
// DSH TokenUsage defines reasoning as informational detail already
|
||||
// included in outputTokens. Split the persisted buckets so shared
|
||||
// pricing and display can add ordinary output and reasoning once.
|
||||
const costUSD = calculateCost(observation.model, input, output, cacheWrite, cacheRead, 0)
|
||||
// included in outputTokens. Preserve raw output and use the same
|
||||
// shared rule as cache rehydration and display (#1075).
|
||||
const costUSD = calculateCost(observation.model, input, billableOutputTokens('dsh', output, reasoning), cacheWrite, cacheRead, 0)
|
||||
const [turn] = key.split(':').map(Number)
|
||||
|
||||
yield {
|
||||
provider: 'dsh',
|
||||
model: observation.model,
|
||||
inputTokens: input,
|
||||
outputTokens: output - reasoning,
|
||||
outputTokens: output,
|
||||
cacheCreationInputTokens: cacheWrite,
|
||||
cacheReadInputTokens: cacheRead,
|
||||
cachedInputTokens: cacheRead,
|
||||
|
|
|
|||
|
|
@ -371,7 +371,7 @@ export const PROVIDER_PARSE_VERSIONS: Record<string, string> = {
|
|||
grok: 'authoritative-usage-v4',
|
||||
// v0-v3 generations, embedded attempt streams, retry accounting, and the
|
||||
// version-specific inherited-prefix rules all change cached DSH calls.
|
||||
dsh: 'session-formats-v0-v3-attempts-v4',
|
||||
dsh: 'session-formats-v0-v3-attempts-v5',
|
||||
// cost-provenance-v3: preserve Hermes included/estimated/actual status and
|
||||
// rebuild the provider section alongside the v3 lifetime ledger. The parse
|
||||
// bump is required with the ledger bump: seeding a new ledger from a section
|
||||
|
|
|
|||
|
|
@ -105,6 +105,12 @@ describe('#1115 billableOutputTokens on report/optimize totals', () => {
|
|||
expect(sessionBillableOutputTokens(makeSession('codex', 10, 3))).toBe(10)
|
||||
})
|
||||
|
||||
it('inclusive dsh: shared session and daily totals include reasoning exactly once', () => {
|
||||
expect(billableOutputTokens('dsh', 10, 3)).toBe(10)
|
||||
expect(sessionBillableOutputTokens(makeSession('dsh', 10, 3))).toBe(10)
|
||||
expect(aggregateProjectsIntoDays([makeProject(makeSession('dsh', 10, 3))])[0]!.outputTokens).toBe(10)
|
||||
})
|
||||
|
||||
it('day-aggregator uses billable output per call', () => {
|
||||
const grokDays = aggregateProjectsIntoDays([makeProject(makeSession('grok', 10, 3))])
|
||||
expect(grokDays[0]!.outputTokens).toBe(13)
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ const testRoot = vi.hoisted(() => {
|
|||
})
|
||||
|
||||
import { clearSessionCache, parseAllSessions } from '../src/parser.js'
|
||||
import { calculateCost } from '../src/models.js'
|
||||
import { billableOutputTokens, calculateCost } from '../src/models.js'
|
||||
import { aggregateSessions } from '../src/sessions-report.js'
|
||||
import { aggregateModelStats } from '../src/compare-stats.js'
|
||||
import { aggregateProjectsIntoDays } from '../src/day-aggregator.js'
|
||||
|
|
@ -21,7 +21,7 @@ import {
|
|||
fingerprintFile,
|
||||
type SessionCache,
|
||||
} from '../src/session-cache.js'
|
||||
import { writeCacheOnDisk } from './fixtures/session-cache-io.js'
|
||||
import { readCacheOnDisk, writeCacheOnDisk } from './fixtures/session-cache-io.js'
|
||||
|
||||
const DSH_HOME = join(testRoot, 'dsh')
|
||||
const CACHE_DIR = join(testRoot, 'cache')
|
||||
|
|
@ -98,6 +98,25 @@ afterAll(async () => {
|
|||
})
|
||||
|
||||
describe('DSH multi-generation session cache', () => {
|
||||
it('reparses v4 caches that stored exclusive output before the shared inclusive rule', async () => {
|
||||
const path = await writeLog(3, [])
|
||||
await writeFile(path, [header(3), message(0, 1, 100, 3)].join('\n') + '\n')
|
||||
await calls()
|
||||
const cache = await readCacheOnDisk()
|
||||
const section = cache.providers.dsh!
|
||||
section.envFingerprint = createHash('sha256').update([
|
||||
`DSH_HOME=${DSH_HOME}`, 'parser=session-formats-v0-v3-attempts-v4',
|
||||
].join('\0')).digest('hex').slice(0, 16)
|
||||
for (const turn of section.files[path]!.turns) {
|
||||
for (const call of turn.calls) call.usage.outputTokens = 2
|
||||
}
|
||||
await writeCacheOnDisk(cache)
|
||||
clearSessionCache()
|
||||
const corrected = await calls()
|
||||
expect(corrected[0]!.usage).toMatchObject({ outputTokens: 5, reasoningTokens: 3 })
|
||||
expect(corrected[0]!.costUSD).toBeCloseTo(calculateCost('gpt-5.4', 100, 5, 0, 0, 0), 12)
|
||||
})
|
||||
|
||||
it('counts separate legacy and current sessions while counting migrated history only once, cold and warm', async () => {
|
||||
const versions = [0, 1, 2, 3] as const
|
||||
for (const version of versions) {
|
||||
|
|
@ -125,7 +144,7 @@ describe('DSH multi-generation session cache', () => {
|
|||
const parsed = sessions.flatMap(session => session.turns).flatMap(turn => turn.assistantCalls)
|
||||
expect(parsed).toHaveLength(5)
|
||||
expect(parsed.reduce((sum, call) => sum + call.usage.inputTokens, 0)).toBe(500)
|
||||
expect(parsed.reduce((sum, call) => sum + call.usage.outputTokens + call.usage.reasoningTokens, 0)).toBe(100)
|
||||
expect(parsed.reduce((sum, call) => sum + billableOutputTokens('dsh', call.usage.outputTokens, call.usage.reasoningTokens), 0)).toBe(100)
|
||||
expect(parsed.reduce((sum, call) => sum + call.usage.cacheReadInputTokens, 0)).toBe(150)
|
||||
expect(parsed.reduce((sum, call) => sum + call.usage.cacheCreationInputTokens, 0)).toBe(25)
|
||||
}
|
||||
|
|
@ -194,7 +213,7 @@ describe('DSH multi-generation session cache', () => {
|
|||
expect(await calls()).toEqual([])
|
||||
})
|
||||
|
||||
it('preserves disjoint reasoning buckets and shared totals through cold and hot caches', async () => {
|
||||
it('preserves inclusive reasoning buckets and shared totals through cold and hot caches', async () => {
|
||||
const path = await writeLog(3, [])
|
||||
await writeFile(path, [header(3), message(0, 1, 100, 3)].join('\n') + '\n')
|
||||
const expectedCost = calculateCost('gpt-5.4', 100, 5, 0, 0, 0)
|
||||
|
|
@ -205,7 +224,7 @@ describe('DSH multi-generation session cache', () => {
|
|||
const parsedCalls = projects.flatMap(project => project.sessions)
|
||||
.flatMap(session => session.turns).flatMap(turn => turn.assistantCalls)
|
||||
expect(parsedCalls).toHaveLength(1)
|
||||
expect(parsedCalls[0]!.usage).toMatchObject({ inputTokens: 100, outputTokens: 2, reasoningTokens: 3 })
|
||||
expect(parsedCalls[0]!.usage).toMatchObject({ inputTokens: 100, outputTokens: 5, reasoningTokens: 3 })
|
||||
expect(parsedCalls[0]!.costUSD).toBeCloseTo(expectedCost, 12)
|
||||
expect(aggregateSessions(projects)).toEqual([expect.objectContaining({ outputTokens: 5, cost: expectedCost })])
|
||||
expect(aggregateModelStats(projects)).toEqual([expect.objectContaining({ outputTokens: 5, cost: expectedCost })])
|
||||
|
|
|
|||
11
tests/fixtures/dsh/v3-retry.jsonl
vendored
Normal file
11
tests/fixtures/dsh/v3-retry.jsonl
vendored
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
{"type":"session","version":3,"id":"fixture-retry","createdAt":1786707340000,"cwd":"/fixture/project","delegationDepth":0,"isSeeded":false}
|
||||
{"type":"turn/start","data":{"turn":1},"seq":0,"time":1786707340000}
|
||||
{"type":"step/start","data":{"turn":1,"step":1},"seq":1,"time":1786707340001}
|
||||
{"type":"request/header","data":{"header":{"config":{"provider":"deepseek-official","model":"deepseek-v4-flash"}},"reason":"initial"},"seq":2,"time":1786707340002}
|
||||
{"type":"request/context","data":{"provider":"deepseek-official","model":"deepseek-v4-flash"},"seq":3,"time":1786707340003}
|
||||
{"type":"assistant/attempt","data":{"turn":1,"step":1,"stream":[{"type":"chunk","time":1786707340000,"chunk":{"type":"usage","usage":{"inputTokens":10,"outputTokens":4,"cacheReadTokens":3,"cacheWriteTokens":2,"reasoningTokens":2,"totalTokens":19}}}]},"seq":4,"time":1786707340004}
|
||||
{"type":"llm/retry","time":1786707340005,"data":{"retryId":"fixture-retry-1","turn":1,"step":1,"provider":"deepseek-official","mode":"normal","policyKey":"default","retry":1,"maxRetries":1,"delayMs":0,"failure":{"code":"SERVER","message":"Fixture failure"}},"seq":5}
|
||||
{"type":"llm/retry-started","data":{"turn":1,"step":1,"retryId":"fixture-retry-1","retry":1},"seq":6,"time":1786707340006}
|
||||
{"type":"assistant/message","surfaceOp":"append","data":{"turn":1,"step":1,"message":{"id":"fixture-success","role":"assistant","content":[{"type":"text","text":"Done"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash"}},"usage":{"inputTokens":100,"outputTokens":20,"cacheReadTokens":30,"cacheWriteTokens":5,"reasoningTokens":8,"totalTokens":155},"stream":[{"type":"chunk","time":1786707340000,"chunk":{"type":"usage","usage":{"inputTokens":100,"outputTokens":20,"cacheReadTokens":30,"cacheWriteTokens":5,"reasoningTokens":8,"totalTokens":155}}}]},"seq":7,"time":1786707340007}
|
||||
{"type":"step/end","data":{"turn":1,"step":1},"seq":8,"time":1786707340008}
|
||||
{"type":"turn/end","data":{"turn":1,"reason":{"kind":"completed"}},"seq":9,"time":1786707340009}
|
||||
|
|
@ -142,7 +142,7 @@ async function writeVersionedSession(
|
|||
}
|
||||
|
||||
// Released layouts checked against deepseek-ai/deepseek-harness at
|
||||
// 2ad76263a53497763575b5ad4ca5acf67ae00987:
|
||||
// c291e7961a515f6d7af9304e7fd1d257929aef26:
|
||||
// packages/session/session-format-v0-to-v1/src/codec.ts (v1),
|
||||
// packages/session/session-format-v1-to-v2/src/codec.ts (v2), and
|
||||
// packages/session/session-format-v2-to-v3/src/codec.ts (v3).
|
||||
|
|
@ -302,6 +302,45 @@ describe('dsh provider - session discovery', () => {
|
|||
})
|
||||
|
||||
describe('dsh provider - parsing', () => {
|
||||
it('counts failed and successful attempts from the official-validated retry fixture', async () => {
|
||||
const content = await readFile(join(import.meta.dirname, '../fixtures/dsh/v3-retry.jsonl'), 'utf8')
|
||||
const path = await writeVersionedSession(3, '--fixture--', 'fixture-retry', content.trim().split('\n'))
|
||||
const calls = await parseAll(createDshProvider(tmpDir), path)
|
||||
expect(calls.map(call => [call.inputTokens, call.outputTokens, call.reasoningTokens])).toEqual([[10, 4, 2], [100, 20, 8]])
|
||||
expect(calls.every(call => call.model === 'deepseek-v4-flash')).toBe(true)
|
||||
expect(calls.reduce((sum, call) => sum + call.inputTokens + billableOutputTokens('dsh', call.outputTokens, call.reasoningTokens)
|
||||
+ call.cacheReadInputTokens + call.cacheCreationInputTokens, 0)).toBe(174)
|
||||
})
|
||||
|
||||
it.each(([0, 1, 2, 3] as const).flatMap(version => [false, true].map(retry => ({ version, retry }))))(
|
||||
'replaces non-adjacent settlements per step in v$version (retry=$retry)', async ({ version, retry }) => {
|
||||
const sample = (step: number, input: number) => version <= 1
|
||||
? chunkUsage(1, step, { inputTokens: input, outputTokens: 10 }, 1786707340000)
|
||||
: JSON.stringify({ type: 'assistant/attempt', seq: step, time: 1786707340000,
|
||||
data: { turn: 1, step, stream: embeddedUsage({ inputTokens: input, outputTokens: 10 }, 1786707340000) } })
|
||||
const rows = [version === 0 ? sessionHeader() : versionedHeader(version),
|
||||
sample(1, 100), sample(2, 200),
|
||||
assistantMessage(1, 1, { inputTokens: 100, outputTokens: 10 }, 1786707340100),
|
||||
...(retry ? [JSON.stringify({ type: 'llm/retry-started', seq: 8, time: 1786707340150,
|
||||
data: { turn: 1, step: 1, retryId: 'interleaved-retry', retry: 1 } }),
|
||||
assistantMessage(1, 1, { inputTokens: 150, outputTokens: 10 }, 1786707340180)] : []),
|
||||
assistantMessage(1, 2, { inputTokens: 200, outputTokens: 10 }, 1786707340200)]
|
||||
const path = version === 0 ? await writePlainSession('--fixture--', 'interleaved', rows)
|
||||
: await writeVersionedSession(version, '--fixture--', 'interleaved', rows)
|
||||
const calls = await parseAll(createDshProvider(tmpDir), path)
|
||||
expect(calls.map(call => call.inputTokens)).toEqual(retry ? [100, 150, 200] : [100, 200])
|
||||
})
|
||||
|
||||
it.each([0, 1] as const)('retains non-fork v%s usage when only seedLength is present', async version => {
|
||||
const rows = [JSON.stringify({ type: 'session', version, id: 'non-fork', seedLength: 4,
|
||||
createdAt: 1786707340000, cwd: '/fixture/project', delegationDepth: 0 }),
|
||||
...[1, 2, 3, 4].map((step, seq) => JSON.stringify({ type: 'assistant/message', seq,
|
||||
time: 1786707340000, data: { turn: 1, step, usage: { inputTokens: 100, outputTokens: 10 } } }))]
|
||||
const path = version === 0 ? await writePlainSession('--fixture--', 'non-fork', rows)
|
||||
: await writeVersionedSession(version, '--fixture--', 'non-fork', rows)
|
||||
expect((await parseAll(createDshProvider(tmpDir), path)).map(call => call.inputTokens)).toEqual([100, 100, 100, 100])
|
||||
})
|
||||
|
||||
it.each([0, 1, 2, 3] as const)('matches official DSH totals for the sanitized v%s fixture', async version => {
|
||||
const content = await readFile(join(import.meta.dirname, `../fixtures/dsh/v${version}.jsonl`), 'utf8')
|
||||
const lines = content.trim().split('\n')
|
||||
|
|
@ -310,7 +349,7 @@ describe('dsh provider - parsing', () => {
|
|||
: await writeVersionedSession(version, '--fixture--', 'fixture-session', lines)
|
||||
const calls = await parseAll(createDshProvider(tmpDir), path)
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]).toMatchObject({ inputTokens: 100, outputTokens: 12, reasoningTokens: 8, cacheReadInputTokens: 30, cacheCreationInputTokens: 5 })
|
||||
expect(calls[0]).toMatchObject({ inputTokens: 100, outputTokens: 20, reasoningTokens: 8, cacheReadInputTokens: 30, cacheCreationInputTokens: 5 })
|
||||
expect(billableOutputTokens('dsh', calls[0]!.outputTokens, calls[0]!.reasoningTokens)).toBe(20)
|
||||
})
|
||||
it('does not reuse the previous request context after a model change', async () => {
|
||||
|
|
@ -354,7 +393,7 @@ describe('dsh provider - parsing', () => {
|
|||
])
|
||||
const calls = await parseAll(createDshProvider(tmpDir), filePath)
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]).toMatchObject({ inputTokens: 10, outputTokens: 0, costIsEstimated: true })
|
||||
expect(calls[0]).toMatchObject({ inputTokens: 10, outputTokens: 'outputTokens' in usage ? usage.outputTokens : 0, costIsEstimated: true })
|
||||
})
|
||||
|
||||
it.each([1, 2, 3] as const)('uses only the usage layout belonging to format v%s', async (version) => {
|
||||
|
|
@ -584,8 +623,8 @@ describe('dsh provider - parsing', () => {
|
|||
const calls = await parseAll(createDshProvider(tmpDir), filePath)
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]!.inputTokens).toBe(14981)
|
||||
// The final output includes reasoning; expose only ordinary output here.
|
||||
expect(calls[0]!.outputTokens).toBe(47)
|
||||
// Preserve raw output, including reasoning, for the shared billing rule.
|
||||
expect(calls[0]!.outputTokens).toBe(656)
|
||||
expect(calls[0]!.reasoningTokens).toBe(609)
|
||||
expect(calls[0]!.timestamp).toBe(new Date(1786707340050).toISOString())
|
||||
})
|
||||
|
|
@ -628,7 +667,7 @@ describe('dsh provider - parsing', () => {
|
|||
expect(calls.map(c => c.model)).toEqual(['deepseek-v4-pro', 'deepseek-v4-pro', 'deepseek-v4-flash'])
|
||||
})
|
||||
|
||||
it('splits inclusive reasoning into disjoint output buckets at the output rate', async () => {
|
||||
it('preserves inclusive reasoning and bills output once through the shared rule', async () => {
|
||||
const filePath = await writeZstdSession('--C-Users-test-myproject--', 'session-reason', [
|
||||
[sessionHeader({ id: 'session-reason' })],
|
||||
[requestHeader('deepseek-v4-pro')],
|
||||
|
|
@ -637,7 +676,7 @@ describe('dsh provider - parsing', () => {
|
|||
|
||||
const calls = await parseAll(createDshProvider(tmpDir), filePath)
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]).toMatchObject({ outputTokens: 60, reasoningTokens: 40 })
|
||||
expect(calls[0]).toMatchObject({ outputTokens: 100, reasoningTokens: 40 })
|
||||
expect(billableOutputTokens('dsh', calls[0]!.outputTokens, calls[0]!.reasoningTokens)).toBe(100)
|
||||
// DSH's TokenUsage reasoningTokens is informational detail already
|
||||
// included in outputTokens, so the output bucket is billed exactly once.
|
||||
|
|
@ -791,7 +830,7 @@ describe('dsh provider - real log fidelity', () => {
|
|||
expect(calls.map(c => c.model)).toEqual(['deepseek-v4-flash', 'deepseek-v4-flash'])
|
||||
expect(calls[0]).toMatchObject({
|
||||
inputTokens: 2877,
|
||||
outputTokens: 72,
|
||||
outputTokens: 90,
|
||||
cacheReadInputTokens: 0,
|
||||
reasoningTokens: 18,
|
||||
sessionId: 'e128dda9-ed11-4868-8266-0ef90d03c3d6',
|
||||
|
|
@ -799,7 +838,7 @@ describe('dsh provider - real log fidelity', () => {
|
|||
projectPath: '/home/u/proj',
|
||||
workingDirectory: '/home/u/proj',
|
||||
})
|
||||
expect(calls[1]).toMatchObject({ inputTokens: 168, outputTokens: 3, cacheReadInputTokens: 2816, reasoningTokens: 22 })
|
||||
expect(calls[1]).toMatchObject({ inputTokens: 168, outputTokens: 25, cacheReadInputTokens: 2816, reasoningTokens: 22 })
|
||||
// Reasoning is a subset of output, so it must not be added a second time.
|
||||
expect(calls[0]!.costUSD).toBe(calculateCost('deepseek-v4-flash', 2877, 90, 0, 0, 0))
|
||||
expect(calls[0]!.costUSD).toBeGreaterThan(0)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue