Base moved to DAILY_CACHE_VERSION 23; this PR's discovery fix now takes 24
to force its own one-time re-derivation on top of the codex structural-
discovery bump. CHANGELOG keeps both entries.
Base has moved since this branch opened: main shipped 17 in v0.9.20 and
now sits at 20, with 21 (#946) and 22 (#1056) claimed on the main-side
pipeline. Bumping to only 16 would let a main-built cache pass the
version check unchanged, so the widened Codex discovery re-derivation
this PR depends on would never fire for those users. Take 23 to stay
above every value a real cache file can carry on either line of
history, and update the test literal and MIN_SUPPORTED_VERSION to
match.
Also documents, next to CODEX_CACHE_VERSION, why that constant is
deliberately not bumped alongside it: the guarded record shapes were
measured at 0 occurrences across 136k real events, so forcing a full
re-parse of multi-GB rollout corpora for them is a bad trade. The
daily-cache bump alone already propagates the discovery widening.
Two pre-existing CLI-suite failures blocked #923's new cli CI job on Linux:
- tests/models.test.ts DeepSeek v4 pricing (x3): loadPricing() fetched the
live LiteLLM table during tests, and live data wins over the bundled
snapshot, so the DeepSeek v4 assertions went red when upstream repriced.
Ports main's fix (4866332b): CODEBURN_PRICING_SNAPSHOT_ONLY skips the fetch
and prices purely off the bundled snapshot; env-isolation.ts sets it for
the whole suite.
- tests/parser-proxy-pricing.test.ts case-insensitive isProxiedPath: the
branch's isProxiedPath/normalizeProxyPath is byte-identical to main's
(case folding is deliberately darwin/win32-only, per the function's own
comment). The test hardcoded the macOS-only expectation, so it fails by
design on Linux CI. Main hit and fixed the same thing (bcf11552); this
ports that fix verbatim: assert the platform-correct behavior instead of
a hardcoded `true`.
Three discovery paths silently dropped sessions. Each fix carries a regression
test that fails against the current code.
**pi/omp** inspected only the first physical line of each transcript. When Oh
My Pi started writing a `type: "title"` slot line ahead of the session header,
every OMP session became invisible. Discovery now scans a bounded twenty
leading lines for the session record, skipping blank and malformed lines rather
than giving up; the bound caps how many LINES are inspected (twenty parse
attempts per file), so a message-only or pathological transcript costs at most
twenty line reads instead of a scan to EOF. It does not cap bytes: the
streaming reader buffers one physical line whole, so a single oversized line
is bounded only by the stream reader's cap, not by this bound. Discovery also
validates the session record's `cwd` before `basename`, falling back to the
project directory for a malformed non-string value — a `cwd: 42` record no
longer crashes discovery.
**cline** scanned only the VS Code stable globalStorage root, so sessions
written under Code - Insiders or VSCodium were never found. All variant roots
are scanned now, matching the roo-code and kilo-code siblings, plus Cline's
home-data root, deduplicated by task id with the newest copy winning.
**opencode and kilo-code** had a session-level fallback query selecting
`model_id` — a column neither schema has; both store `model` as JSON text. On
any real database the query threw, the fallback returned null, and the
zero-yield session rollup never emitted a call. Usage for interrupted or
user-only sessions was simply missing, with no error surfaced. The query now
reads the real column and reports `providerID/id`, which the display-name and
pricing paths already canonicalize. A kilo-code mirror of the opencode
zero-yield fallback test pins the fix on the kilo path end to end.
**Versioning.** The per-provider parse fingerprints (session-cache.ts) are
bumped so already-cached sessions re-parse once and pick up the working
fallback; they do not collide with the sibling ports. The shared
DAILY_CACHE_VERSION/MIN_SUPPORTED_VERSION is bumped 16 -> 17: a warm
complete daily cache skips re-hydration, and the ten-year retention window
would otherwise keep serving the pre-fix opencode/kilo/pi/omp zeroes forever,
exactly where this fix matters most. The bump forces the one-time re-derive;
a regression test seeds the pre-fix v16 complete cache and proves the
recovered usage lands. v16 itself is skipped: main already spent it on the
codex structural-discovery fix (eece4cf), so claiming 16 here would load a
main-built v16 cache as current and complete and the invalidation would
never fire.
One caveat: pi/omp discovery now uses the streaming reader with its 4 GB cap
while parse still caps at 128 MB, so a file between those sizes is listed at
discovery and skipped at parse. No usage delta, but the asymmetry is real.
Ports two upstream fixes this branch never received (eece4cf, 4ff3497).
**Discovery gated on a client identity string.** `isValidCodexSession`
required `payload.originator` to start with "codex". But `originator` is a
free-form client identity, not a format marker: anything driving
`codex app-server` writes structurally identical rollouts with its own value —
"t3code_desktop", "JetBrains.IntelliJ IDEA", whatever ships next. Every
third-party frontend was silently dropped, and each one needed a new allowlist
entry (#626, #873). Validation is now structural.
Be clear about what that gate was and was not. It was never a security
boundary — anyone able to write into the sessions directory could write
`"originator":"codex-cli"` and pass it. It was accidental integrity
protection, and removing it widens what gets ingested from those directories
to any well-formed `session_meta` line. The trust boundary is unchanged: write
access to the Codex home, which is itself configurable via `CODEX_HOME`. A
crafted rollout can inflate cost or impersonate a project path, exactly as it
could before by spelling the originator correctly.
**Non-string fields on the parse path.** A garbage `timestamp` threw
RangeError out of `toISOString()` and zeroed the session. Guarding only that
one would have been the smaller half of the problem: the timestamps that reach
emitted calls were unguarded too, and a numeric one produces `NaN-NaN-NaN` day
buckets that the daily cache then keeps for ten years — silent, and persistent.
Token counts could go NaN and slip past a `=== 0` check into reported cost.
`session_id` and `forked_from_id` could coerce an object into a dedup key.
All of those are now guarded; the fields still read through a raw cast are
listed nowhere, because there are none left on this path.
**The cache is bumped to 16.** Rollouts rejected before they were ever parsed
now contribute usage, and nothing downstream can notice: the aggregator serves
every day before today from this cache, with ten-year retention, so an
upgrading user would keep pre-fix history forever while today disagreed with it.
Reviewers split on this bump and both arguments are worth having. For it: cache
versions are per-branch lineage, and landing unbumped leaves a user on this
branch with stale history that merging main cannot repair retroactively.
Against: most users are unaffected and pay a full re-derivation for nothing.
The number stays 16, and it means the same thing it means on main: main's 16
was set by eece4cf, the structural-discovery fix this PR ports — same change,
same bump, no collision to resolve. The sibling PRs in this batch that need
their own invalidation are moving to 17 instead of colliding on 16.
detectJunkReads took its count from core's detector and re-derived its display
and trend from the host's own JUNK_DIRS regex. The two lists disagreed: core
classifies `vendor`, `site-packages`, `out` and `target` as junk; the host's
did not.
Core was already counting reads under those segments — the count is not what
was wrong. The host's derivation was. In a repo whose junk reads all live under
one of them — Go and PHP vendor, Python site-packages, Rust target, Java or
Next out — the host loop matched nothing, so `recentJunkReads` stayed at zero.
Where the window also had recent activity, `computeTrend` read that as fixed
and returned 'resolved', and the finding was dropped. Whole ecosystems never
saw it. In mixed repos it survived but rendered incoherently: the explanation
quoted core's total while the directory list and the CLAUDE.md suggestion came
from the narrower host counts, so the numbers did not add up and the suggested
directories omitted the one actually causing the waste.
Core now exports `junkSegmentOf`, which returns the exact segment that made a
path junk. The host deletes JUNK_DIRS and JUNK_PATTERN outright and asks core
in both loops — junk-reads and duplicate-reads, which had the same split. The
host still names the directory for the payload, because a class alone cannot
render `vendor/ (5x)`; it just no longer decides what junk means.
Inside core, the precedence rule (dependency > build > vcs) lived in the order
of three loops, and adding junkSegmentOf duplicated them. Both functions now
consult one private helper, so the two cannot drift — a second copy of one rule
is what caused this bug in the first place.
Two display changes fall out. A path under two junk segments of the same table
now names the first in path order rather than the first in array order, so
`/x/build/dist/y` reports `build` where it used to report `dist`; the count is
unchanged. Terminal junk directories and Windows-style paths are now matched,
which the old slash-delimited regex missed — those align the host with what
core was already counting.
One robustness note: the old `JUNK_PATTERN.test()` coerced a truthy non-string
`file_path`, while `junkSegmentOf` would throw on one. The type is narrowed at
the point of use so the new path cannot throw where the old one could not.
COMPLETES Phase 8 of the @codeburn/core extraction: 36/36 provider identities
(including qwen) now decode in core.
The row -> call mapping moves to packages/core/src/providers/vercel-gateway/
verbatim: day/model/cost defaults, the all-zero skip BEFORE the dedup key is
burned, the `vercel-gateway:<day>:<model>` key with add-after-skip semantics,
`${day}T12:00:00.000Z` timestamp synthesis ('' for a missing day), and the
`${day}:${model}` session id. The decoder is pure over supplied rows: no fs,
env, clock, or network.
Everything network stays host-side and byte-identical: the authenticated
/v1/report fetch, the AI_GATEWAY_API_KEY / VERCEL_OIDC_TOKEN reads, both stderr
warnings, discovery, `network: true`, and the gate that yields nothing when the
scan has no date range.
Adapter shape adjudication: the draft used createBridgedProvider and needed two
contortions to fit it — a Symbol-keyed dateRange stashed on the shared
SessionSource, and `project` packed into the records payload then unpacked by a
decode wrapper (the bridge passes neither the date range nor the source to the
mapping step). The Symbol injection also mutated the caller's discovered source
object, an observable behavior change. Rejected both; this provider now uses a
plain bespoke adapter like antigravity/kiro, which the bridge's own header
already documents as the escape hatch for providers it was not built to cover.
No change to bridge.ts, parser.ts, pricing-pass.ts, or session-cache.ts.
Emitted key shape is unchanged and gated by key-set assertions: `costUSD` is
present and there is NO `costBasis` key, so parser.ts keeps passing the
gateway's own dollar figure through instead of repricing it.
Validator fixes on top of the draft:
- Dropped the Symbol side-channel and the packed-payload decode wrapper.
- Corrected a false claim in the core observations header: it said the raw `day`
was never emitted, but `day` is spliced verbatim into the timestamp, into
startedAt/endedAt, and into the dedup key. The envelope's date-time constraint
is what actually bounds it; a new smuggling arm pins that a hostile `day`
fails envelope validation.
- Added golden arms for both stderr warnings, for the no-date-range arm making
no fetch at all, and for the provider not mutating the discovered source.
Verification: the strengthened goldens were run against the pre-migration
provider restored in place and pass identically (13/13). Swapping the all-zero
skip and the dedup burn fails tests at both the core and CLI layers.
Core 509 tests, CLI 2470, root 2470 — all green.
Move all five kiro parse arms (legacy .chat, v1 modern execution,
workspace-session, CLI .jsonl, v2 IDE event log) into
@codeburn/core/providers/kiro as pure decode. The host keeps discovery,
every file read, the companion-file reads (CLI .json, v2 session.json),
the workspace-session mtime stat, project attribution, model display
names, and all pricing.
Behaviour is byte-identical to the pre-move provider; the four dedup-key
namespaces (kiro:, kiro:ws-session:, kiro-cli:, kiro-v2:) and the five
per-arm ParsedProviderCall key sets are unchanged.
Preserved verbatim, with tests that discriminate against plausible
refactors:
- A4's asymmetric turnIndex: a zero-output turn does NOT consume a
user_turn_metadatas slot, while the dedup-hit and bad-timestamp skip
arms DO. Every later turn's timestamp and metered credits depend on
this. All three arms are mutation-tested.
- A1's toolSequence key stays present-with-undefined for single-entry
sequences, gated by an Object.keys() assertion (toEqual cannot see it).
- A1's input tokens still derive from the already-truncated 500-char
prompt, unlike every other arm.
- A5's dedup fallback stays `execId || String(calls.length)`, evaluated
at flush time.
- The three-way workspace-session prepare/finish split keeps the mtime
stat behind both content gates rather than hoisting it.
Credits seam: core emits `credits: number` and never prices. The host
multiplies by USD_PER_KIRO_CREDIT and builds costUSD/costBasis, matching
the codebuff precedent.
Validator fixes on top of the migration:
- A1's dedup key regained the raw `data.executionId` field; a fallback to
basename(path) had been introduced, changing keys for chat files with a
missing, empty, or non-string executionId.
- The content-smuggling non-vacuousness guard was vacuous per vector (an
aggregate re-decode compared against itself); it now asserts per-vector
call counts, verified by breaking two fixtures.
- Restored a mangled comment on the load-bearing v2-root derivation and
several explanatory comments dropped during the move.
- Added goldens G1b (raw executionId), G4b (exact credit products) and
G6b (turnIndex advances across a bad-timestamp turn), plus a core test
for the same; all reproduce against the pre-move provider.
PROVIDER_PARSE_VERSIONS['kiro'] and CACHE_VERSION are unchanged. The
companion-file fingerprint blind spot (session-cache.ts) and the dead
project parameter on the old parseChatFile are left as-is.
This completes the Phase 8 tail.
Cursor is the bucket-D stateful multi-store provider: it keeps its bespoke
adapter (no createBridgedProvider) and the host retains every store read.
Tagged-record composition over five query families. The host issues all five
queries in the load-bearing order [1] composerData meta, [2] bubble COUNT,
[3] bubbles (paged or since), [4] agentKv, [5] user messages, preserving the
per-query degradation semantics: [1]/[4]/[5] degrade to empty, [2] to total=0,
and a [3] failure early-returns zero calls while still writing the (empty)
cache and skipping [4]/[5]. decodeCursor() receives the four row sets plus the
host-supplied agentKv timestamp and performs the whole stitched pass — the
pre-pass scan, the agentKv fold, the user-message queue, and all three emit
arms (per-bubble, per-conversation input, stream-only).
The 19-key emitted call shape is a cache-compatibility contract. CURSOR_CACHE_VERSION
stays 6 and PROVIDER_PARSE_VERSIONS['cursor'] is unchanged, so existing
cursor-results.json files on disk are replayed through the new path; adding,
dropping, or undefined-ing any key would poison them. The goldens gate the key
set with Object.keys() on every arm, which is what catches a key present with an
undefined value — toEqual cannot see one, and JSON.stringify drops exactly those
on the way into the cache.
Pricing, bash base-name extraction, display names, project attribution, SQL,
paging, env reads and clock reads all stay host-side; toProviderCall is the
single mapper that re-adds costBasis 'estimated', costIsEstimated true and the
resolved pricing model. No costUSD is ever emitted.
Composer-id / dedup-key envelope finding: composer ids and request ids flow into
sessionId and into the envelope's dedupKeys by design, exactly like every other
provider's machine identifiers, so the content-smuggling block deliberately does
not plant a secret there. Hashing dedup keys uniformly is a schema-wide
follow-up, not a cursor-local change.
Validator fixes on top of the migration:
- relocated the content-smuggling block from cursor-decode.test.ts into
content-smuggling.test.ts, matching all 25 other providers, and added a probe
proving each planted secret actually reaches the field it guards
- G11 now gates the undefined-key cache contract it claimed to (toEqual passed
under an injected extra undefined key; toStrictEqual plus key-set assertions
catch it)
- restored the CODEBURN_CURSOR_MAX_BUBBLES override in the golden's afterEach so
G12 cannot leak the scan budget into cursor-large-db-cap.test.ts
- restored the comment rationales that must travel with their code verbatim
(pos-cursor queue performance note, parseComposerIdFromKey CR/LF signature,
agentKv stream and pending-flush notes, dedup-key history, costIsEstimated)
The 12 goldens were cross-checked against the pre-migration provider by
restoring it in place: they pass identically on both sides. H2 (queue pop before
skip) and H4 (arm B dedup-key burn before the timestamp check) were mutation
tested and are killed by G8 and C6 respectively.
Moves every record-parsing arm into @codeburn/core/providers/antigravity:
the protobuf wire reader, the gen_metadata row decode, the RPC
generatorMetadata decode, the statusline JSONL run-collapse/delta decode,
and model canonicalization. All five emit arms (statusline, cache-hit,
sqlite, RPC, RPC-failure fallback) plus the snapshot write-path keep their
host-side control flow byte-for-byte.
Cache integrity: CACHE_VERSION stays 5 and antigravity-results.json keeps
its shape. The cache is not re-derivable — the RPC-failure fallback arm
replays cached.calls for cascades whose language server is gone — so no
bump, and the cache-write-before-seenKeys-filter ordering is preserved in
both the sqlite and RPC arms.
Pricing stays host-side: normalizePricingModel / PRICING_ALIASES remain in
the CLI, and every arm now emits through one shared toProviderCall carrying
costBasis 'estimated' plus pricingModel, and no costUSD.
parseStatusLinePayload's wall clock becomes an injected `at` parameter; the
host passes new Date().toISOString() at each call site, once per payload.
Validator fixes on top of the migration:
- content-smuggling: the hostile statusline record used the hook-payload
shape, so the decoder dropped it and the cwd/session_id assertions proved
nothing; corrected to the recorded-event shape, with a call-count guard.
- goldens: G14 maps through the exported host toProviderCall instead of a
private copy, and a new G15 pins the emit loop's turnIndex-before-seenKeys
and previousSnapshotUsage-before-skip ordering, which no golden could
previously detect.
- core: removed an unused type import and a dead CANONICAL_TOOL_NAME const;
restored the statusline reasoningTokens rationale comment.
Moves the Mistral Vibe rich decode into @codeburn/core behind the bridge:
discovery, file I/O and pricing stay CLI-side, the pure record decode
(including the even allocation of the session dollar figure and the session
token totals across assistant messages) moves byte-exact into core.
Seam adjudication: the draft brief said the `session_cost > 0` gate should
move into core. It cannot — that gate is the first arm of a three-arm cost
resolution whose last arm consults the generic price table, which is
forbidden in core. The whole resolution therefore stays host-side in
`calculateSessionCost` and core receives a pre-resolved `sessionCost`;
core keeps only the pure allocation arithmetic (allocateInteger /
allocateCost, allocationIndex / assistantOrdinal), unchanged in rounding,
remainder distribution and float operation order. The move is
observationally equivalent: the original resolved the cost once per session
during parse, with no per-call state.
Validator fixes on the draft:
- core dropped the `metadata.session_id || basename(sessionDir)` fallback,
emitting sessionId '' (and colliding dedup keys `mistral-vibe::<msg>`)
for sessions whose meta.json omits session_id. The host now passes
`sessionIdFallback`.
- `toProviderCall` set `turnId: undefined` unconditionally, adding a key
the original omits entirely on the no-assistant session-level arm.
- the moved comment block described a seam that no longer exists; adjusted
minimally to describe where the resolution actually lives.
Parity was verified arm-by-arm against the pre-migration decode checked out
in place over a 28-session adversarial fixture matrix (strict equality plus
per-call key-presence), and the adversarial allocation arms (7 over 3,
1 over 3, terminal zero-cost, idx-N keys, basename fallback, float-op order)
are promoted into the bridge and core decode tests with values captured
from the original.
Unifies the three-file OpenCode decode family — session-message.ts,
sqlite-session-parser.ts, and opencode-file-parser.ts — into a single core
module, packages/core/src/providers/opencode-session, exposing a two-arm
tagged envelope (kind: 'sqlite' | 'file') over one shared assistant-turn
builder. The SQLite driver, both WITH RECURSIVE session_tree queries,
blobToText, discovery, and extractBashCommands all stay CLI-side.
session-message.ts shrinks to the discovery-side sanitize helper;
sqlite-session-parser.ts and opencode-file-parser.ts keep their SQL and
directory walks and expose readRecords adapters. No decode logic remains
CLI-side in any of the three.
kilo-code is now fully converted: both arms — cline task dirs via the batch-S1
core vscode-cline module and SQLite via the new module — run through core
decodes from one bridged provider, and the thin createClineParser adapter S1
retained is deleted.
The CODEBURN_VERBOSE zero-yield notice is reconstructed host-side by
zed-style decode wrappers, byte-identical for both the OpenCode and KiloCode
labels, with parseFailCount/roleSkipCount derived from the decode's
malformed-json / unknown-shape diagnostics. The session-level SQLite fallback
is pre-fetched onto the envelope instead of queried lazily — output-neutral,
changing only I/O volume — and keeps its distinct key shape, emitting no
skills/subagentTypes keys.
Validator fixes on top of the migration:
- opencode's file arm emitted a spurious SQLite verbose stderr line, reporting
a previous SQLite source's session id and message count, because the
readRecords -> decode count handoff was never cleared when switching arms.
The handoff is now reset on the file path. Pinned by a new regression test.
- The S8 and F9 dedup goldens did not actually pin add-after-build: the SQLite
fixture used two different message ids, and both file fixtures were degenerate
(the CLI one wrote the same filename twice and silently overwrote; the core
one shared a text part across both messages, so the "skipped" message built
successfully). All four now fail if seenKeys.add is hoisted above the build.
- Added key-presence gates to the goldens, since toEqual accepts a
present-but-undefined key: the session-level fallback must omit
skills/subagentTypes, a message-arm call per arm must carry them, and
fallbackCostUSD must be present for cost 0 and absent for an absent cost.
Goldens were captured pre-migration and re-verified against the original tree
by restoring 38172892's sources in place; all 36 pass on both sides.
Pre-existing issues moved verbatim and left alone: the bare-Record tool map
with a truthy hit check in normalizeToolName (prototype leak on names like
'constructor'), opencode.ts's duplicated display-only tool map, and the dead
inner role guard in the SQLite arm. Every new lookup introduced by the
unification is a Map.
Cuts the decode shared by the vscode-cline family out of the CLI and into
@codeburn/core/providers/vscode-cline. This is the first shared-module batch:
one core decoder now backs four provider identities, with the provider name
sourced from context.providerId and the per-consumer fallback model passed as
a decode knob.
- cline, ibm-bob and roo-code become createBridgedProvider specs. Discovery,
file I/O and pricing stay CLI-side; vscode-cline-parser.ts keeps the
discovery helpers and gains readClineRecords (I/O adapter) plus
toClineProviderCall (rich -> ParsedProviderCall, where cost re-enters).
- kilo-code.ts is a zero-line diff. Its SQLite arm does not move until batch
S2, so createClineParser is retained but reimplemented as a thin adapter
over the core decode — I/O and mapping only, no decode logic CLI-side.
- Behavior is preserved verbatim, including the dedup burn-through: the key
is added to seenKeys before the zero-token skip, so a skipped entry burns
its key. Also preserved: the unguarded new Date(entry.ts), the cost != null
(not !== undefined) measured-cost test, and project/projectPath written
unconditionally so key presence is unchanged.
Validator pass over the migration:
- Added a permanent key-presence gate to the shared-bridge goldens. toEqual
cannot distinguish an absent key from a present-but-undefined one, which
hid exactly this class of regression in earlier batches; the goldens now
assert costUSD is absent when estimated and that project/projectPath are
always present.
- Turned the kilo-code .db: dispatch case from a negative-only assertion into
a positive one: it now captures and asserts the SQLite arm's open-failure
line, proving dispatch reached that arm rather than merely yielding nothing
(and keeping the line out of test output).
Parity was checked against the pre-migration decoder as oracle over 25 task
fixtures per provider, comparing values, key presence, key order, JSON
serialization and the resulting seenKeys sets across two dedup passes — exact
match for all four consumers and for the retained adapter.
Bridged via a tagged envelope record across the four parse paths
(jsonl / chatsession / jetbrains / otel): readRecords shapes one envelope
per source kind and a single decodeCopilot dispatches on envelope.kind.
The sqlite driver, the three OTel queries, blob reads, git-root probing,
extractBashCommands / extractOtelBashCommands, and the sanctioned
shutdown-site calculateCost residual all stay host-side. The residual
keeps its exact argument order (model, input, 0, cacheWrite, cacheRead, 0),
emits costUSD + costIsEstimated: false with no costBasis key, and carries
its explanatory comment unchanged.
Validator fixes over the draft:
- collectJetBrainsRepoDirCandidates used a [\|"] character class instead
of the original's (?:\\|") alternation, so a file:// path containing a
pipe was truncated, its dir never resolved, and the JetBrains project
fell back to the generic bucket. Restored the alternation.
- the otel arm dropped the subagentTypes key when undefined; the original
always emits it. Restored so key presence matches byte for byte.
- otel chat-span attributes were looked up with a linear Array.find per
chat span; replaced with a per-span-id map built in the same pass.
Verified by a differential harness running the pre-migration provider
against the migrated one over 27 scenarios (all eight goldens plus
edge cases), comparing calls, seenKeys, and key presence. The harness is
not committed; the scenarios it proved are now goldens G9-G17.
Move the pure record decode for five sqlite-backed providers into
@codeburn/core, keeping discovery, sqlite I/O, pricing, and bash
base-name extraction CLI-side behind the dual-registry bridge.
sqlite variant: the host opens the database, runs the SQL, textualizes
BLOBs (warp stylized_command, quickdesk content/tool_names), probes
sqlite_master table variants (quickdesk), and re-throws SQLITE_BUSY so a
transient lock on a live database retries instead of being cached as an
empty result (hermes). The core decoders receive already-normalized rows.
Validator fixes over the drafted migration (original behavior is the
authority in every case):
- warp: restored the `run_command` tool-name check as an identity
comparison. The draft had rewritten it as a bare Record lookup, so
'constructor'/'toString'/'__proto__' resolved to inherited Object
members instead of passing through. Added a regression test.
- warp: restored `safeNumber` to the host's semantics (positive finite
numbers only). The draft's rewrite accepted numeric strings and
negative values, changing token budgets and dominant-model selection
for untrusted conversation_data.
- warp: `estimateTokensFromChars` now mirrors the host helper exactly.
- devin: model DISPLAY names move back CLI-side. The draft shipped a
7-line regex standing in for the host's `getShortModelName`, which
mis-rendered every non-Claude model id. The decoder now emits the raw
generation_model / model_name ids and the CLI formats them.
- devin: dropped the added `costBasis: 'measured'`; the pre-migration
call carried `costUSD` with no basis marker. Golden corrected.
- quickdesk: fixed an inverted source-variant predicate that routed
`sourceId: 'metrics'` on a sessions.db path (and `sourceId:
'sessions-db'` on a metrics path) to the wrong parser.
- cursor-agent: the "unrecognized transcript format" warning is now
re-emitted from the real decode result via a decode wrapper. The draft
re-derived `recognized` host-side as `hasUser && hasAssistant`, which
silently dropped the warning whenever a jsonl transcript had both roles
but produced no turns.
- cursor-agent: restored the per-conversation summary cache, so the
attribution database is opened once per conversation id, not per source.
- cursor-agent/devin: the session id is derived once, host-side, and
passed through, removing a second derivation that could disagree with
the host's on Windows-style paths.
- hermes: `browser_vision` maps to Vision, matching the original table.
Goldens re-verified by running each bridge fixture through the
pre-migration provider implementation: all five reproduce field-for-field.
Category B (sqlite) variant of the bridge migration: the sqlite driver and every
SQL query stay CLI-side. Each provider's `readRecords` opens the database, runs
the same queries as before, and hands the resulting rows (blob and all) to a
pure core decoder; `toProviderCall` maps the rich, cost-free decode back onto
ParsedProviderCall, where cost re-enters via the parser.ts pricing pass.
Per provider:
- crush: session row + dominant-model query -> one combined record. Crush stores
cost in dollars, so a row with cost > 0 carries `measuredCostUSD` (costBasis
'measured'); a zero-cost row falls back to token estimation, arm order intact.
- zcode: model_usage + tool_usage row sets -> one composite record. Each turn's
tools still attach to the first non-skipped usage row of that turn only.
- zed: threads rows handed over compressed; zstd decompression, JSON parsing and
per-request/cumulative-remainder accounting are pure. The Node >= 22.15 zstd
capability check stays host-side.
- forge: conversation row handed over with `context` still serialized; JSON
parsing and per-message decode are pure. Bash base-name extraction (and its
strip-ansi dependency) stays CLI-side over the decoder's raw command strings.
- goose: session + assistant tool-message + first-user-message rows, BLOB
columns pre-converted to text host-side, bundled into one composite record.
Validator fixes (original behavior is the authority):
- forge: the draft replaced the pre-migration `mapToolName` switch with an
object-literal lookup. Tool names come straight from conversation JSON, so
names colliding with Object.prototype members ("constructor", "toString",
"__proto__", "hasOwnProperty") resolved to inherited Functions / the prototype
object and were pushed into `tools` as non-strings instead of falling through
to the identity default. Restored the switch and pinned the arm in the fixture.
- zed: the draft routed the "skipped N unreadable Zed threads" notice into
record diagnostics, which the bridge discards, silently dropping a warning the
pre-migration decode printed. Re-emitted host-side from the diagnostics count
and pinned with a stderr assertion.
- Fixture coverage extended for the arms that were regression-blind: forge's
prototype-named tool calls, zed's aggregate stderr line, and goose's
single-turn `toolSequence` omission plus the unparseable-timestamp fallback.
Parity was verified independently of the bridge tests with a git-show harness
that runs the same fixtures through the pre-migration provider files and asserts
field-for-field equality, including the extra arms above.
All five providers wired through createBridgedProvider with decode in
packages/core only. Deep validation corrected the initial drafts: zerostack
dedup-key/sessionId/project regressions, droid and mux crude bash-extraction
reimplementations (host now runs the real extractBashCommands on raw
strings), lingtai-tui dedup-key source, open-design dropped project. Goldens
are byte-identical toEqual pins anchored to the pre-migration decode.
Move the pure record decode for grok, kimi, and codewhale into
@codeburn/core and rewire the CLI providers through createBridgedProvider,
keeping discovery, file I/O, config resolution, and pricing host-side.
Validation fixes (legacy-parity, caught arm-by-arm against the pre-lift
decode):
- grok/codewhale: the host toProviderCall wrongly wrapped bashCommands in a
Set. The legacy decode deduped nothing (flat push of extracted base
commands), so per-command breakdown counts drifted. Drop the Set.
- kimi: the core decode dropped the legacy per-turn tool Set, so repeated
tools in a turn no longer collapsed. Restore [...new Set(tools)].
- codewhale: remove an unused codeWhaleToolNameMap import.
Parity fixtures now exercise the dedup arms (repeated tools/commands) and the
goldens are pinned to the ORIGINAL in-CLI decode output, captured by running
the pre-migration providers over the same fixtures.
Redesign the three token-waste detectors as pure @codeburn/core detectors that
consume an ObservationEnvelope of fingerprinted resource refs (never raw paths):
- Envelope schema 0.2.0: CallObservation gains optional resourceReads /
resourceEdits (ResourceRef = {resourceId 16-hex, resourceClass}). claude/codex
toObservations fingerprint toolSequence file paths into these; raw paths never
cross the boundary. Keep observation-0.1.0.json frozen; emit 0.2.0.
- Host privacy key (D1): sync keystore under the codeburn config dir, random
32-byte key generated on first use, stable across runs, never emitted.
- core/detectors: junk-reads, duplicate-reads, context-bloat — pure, zero
fs/env, each emitting Finding[] with confidence(basis), machine-readable
evidence, algorithmVersion.
- optimize.ts delegates the three to core, mapping Finding -> WasteFinding;
display strings, fix payloads and trend stay host-derived.
Numbers parity: all optimize tests pass unchanged; frozen-corpus optimize JSON
(3552 sessions) is byte-identical PRE vs POST.
Carve the Codex decoder into @codeburn/core/providers/codex as a pure function
over supplied rollout records with EXPLICIT, JSON-serializable state
(CodexDecodeState): the running cumulative token counters, fork-replay cutoff,
mid-turn accumulators, id/turn bookkeeping, and the cross-file dedup memory
(seenKeys). Decoding a corpus in one pass equals decoding it in any number of
passes when the state threads between them (the resume invariant).
Cost leaves the decoder: the rich calls are cost-free and the CLI prices them
through the established estimated-cost seam (costBasis: 'estimated'), retiring
both Phase-0 residual calculateCost sites. The CLI codex provider is now
discovery + file streaming + cache I/O + pricing.
codex-results cache bumped to v8 (one deliberate bump): it persists the decoder
end-state blob + a byte offset + host-priced calls, so a grown rollout resumes
from the stored state and decodes only the appended bytes. Lossless — rollout
files are durable, so the one-time re-derive rebuilds identical data.
Adds a codex path to the content-smuggling guardrail and the import-smoke
subpath set. Byte-parity verified against a frozen 826-file corpus (codex-only
and all-providers, rel_tol 1e-9); perf within ±1.5% cold / ±0.5% warm.
Move the pure Claude JSONL decode out of packages/cli/src/parser.ts into
packages/core/src/providers/claude/: the large-line buffer scanner, line
parsing, compaction, per-call/turn extraction, rich-capture meta collectors,
turn grouping, and streaming dedup. These are pure over supplied records/lines
(no fs/env/clock/pricing) and carry content in-memory only at this host-facing
layer.
Add the minimizing transform toObservations(richDecode, {privacyKey,...}) that
maps the rich decode into the strict Phase-2 observation envelope: only
fingerprints, enums, numbers, timestamps, dedup keys, and canonical tool names
cross the boundary. Extend the content-smuggling guardrail with a real claude
decode -> toObservations case planting secrets in every captured free-text
field and asserting the serialized envelope surfaces none.
The CLI parser is now the adapter: it keeps all I/O, discovery, incremental
append, cache shapes, classification, and host-side pricing. parseApiCall /
parseAdvisorCalls / groupIntoTurns become thin wrappers that price the cost-free
core decode (calculateCost + local-model savings) and split raw bash commands
(strip-ansi stays out of the zod-only core), then map into ParsedApiCall /
ParsedTurn. Shared record + tool-vocab definitions move to core; the CLI
re-exports them so every existing import path is unchanged.
No parse-version bump; CachedCall/CachedTurn shapes and values byte-identical.
Core runtime deps stay zod-only.
