fix: thread partial-progress saves into non-Claude provider phases

scanProjectDirs (Claude) has long taken a throttled saveProgress
callback so a killed cold parse resumes from a warm cache;
parseProviderSources (every other provider) only persisted at the
whole-provider boundary. A run killed mid-way through a large single-
provider phase (a multi-GB codex corpus is the common case) discarded
everything parsed since the last provider finished, restarting that
whole phase from zero.

parseProviderSources now accepts the same onFileParsed callback,
invoked once per source right after that source's cache entry lands,
outside the per-file try/catch, mirroring scanProjectDirs' placement
and throttle. A file's fingerprint is only ever set after it fully
parses, so a mid-file kill can never leave a half-parsed file looking
complete on resume.

Verified against a real ~2GB Claude + ~4GB codex corpus (frozen
snapshot): kill -9 mid-codex-phase leaves a genuine partial cache
(no codex-results flush, complete: false), a resume only reparses the
unfinished remainder, and the final report is byte-identical to an
uninterrupted cold parse.

CHANGELOG.md

@@ -18,6 +18,7 @@
 - **DeepSeek Harness (`dsh`) is now a supported provider.** Reads DeepSeek's open-source agent harness from `~/.dsh/sessions` (`DSH_HOME` relocates the root), both the default zstd logs and the uncompressed `session.jsonl` variant. A `.zstd` log is a concatenation of independent zstd frames, one per write batch, so it is decoded frame by frame behind a structural frame scan and a torn trailing frame from a crashed writer is ignored rather than failing the file (needs Node 22.15+ for `zlib` zstd; below that dsh is skipped with a notice instead of counted as $0). One call per `(turn, step)`, with the step's final `assistant/message` usage superseding the streamed `assistant/chunk` sample of the same call rather than adding to it, the model taken from the message that served the step, and reasoning tokens billed at the output rate. DSH records tokens but no cost, so calls are priced from the shared tables. The events a forked session replays from its parent are skipped, since codeburn already counts the parent's own log. The session format is pinned at version 0 upstream with no compatibility implied, so a log stamped with any other version is skipped with a notice instead of read under today's assumptions.
 
 ### Changed
+- **A kill mid-way through a non-Claude provider phase no longer restarts that whole phase.** `scanProjectDirs` (Claude) has long taken a throttled `saveProgress` callback so a killed cold parse resumes from a warm cache; `parseProviderSources` (every other provider — codex, cursor, gemini, and the rest) did not, and only persisted at the whole-provider boundary. On a large single-provider corpus (a multi-GB codex history is the common case) an app-timeout SIGKILL, crash, or force-quit during that phase discarded everything parsed since the last provider finished, forcing the entire phase to re-parse from zero on the next run. `parseProviderSources` now takes the same callback, invoked once per source right after that source's cache entry lands (mirroring `scanProjectDirs`' placement, outside the per-file try/catch), on the same file-count/wall-clock throttle. A file only ever gets a fingerprint once it has fully parsed, so a mid-file kill can never leave a half-parsed file's entry looking complete on resume.
 - **Routed model ids price as the model they wrap, and an unknown vendor prefix no longer prices by blind stripping.** Token-plan and gateway spellings of the same model (`omniroute:`, `cp/`, `cline-pass/`, `cline-free/`, `cmd/`, `antigravity/`) are peeled and the remaining id is priced, so a Cline Pass or OmniRoute session shows a `~` estimate instead of $0. In exchange, `provider/model` is no longer treated as authority on its own: the leading segment is stripped only when it is a namespace the bundled pricing catalog itself uses (`anthropic/`, `openai/`, `google/`, `x-ai/`, `qwen/`, `moonshotai/`, `nousresearch/`, `xiaomi/`, `z-ai/`, and every other vendor prefix in the LiteLLM snapshot), one of the routing wrappers above, or one of the client-side spellings `kimi/`, `mimo/`, `zhipu/`, `litellm_proxy/` and `openai_like/`. Anything else stays unpriced and is reported as unpriced rather than inheriting the price of a same-named cloud row, and local-runner prefixes (`ollama/`, `lmstudio/`, `hosted_vllm/`, `local/`) are excluded on purpose so an unlisted local tag can never invent cloud spend. A user price override for the bare id wins over the catalog row a routed spelling would otherwise hit.
 - **SQLite providers now survive read-only database parents.** A read-only SQLite open is not read-only on disk: on a WAL database SQLite must create `<db>-shm` and `<db>-wal` in the database's own directory, so a source on read-only media, under restrictive permissions, or inside a Flatpak/snap confinement failed with `attempt to write a readonly database` (or `unable to open database file` when a `-wal` was present without its `-shm`), and both discovery sites swallowed it — the provider read as "not installed" rather than as an error. That covers cursor, cursor-agent, opencode, goose, warp, kilo-code, zerostack and the copilot agent-traces database. The direct open stays the fast path and is byte-identical when it succeeds. When it fails for want of sidecars: a database with no WAL frames to lose is opened in place with `immutable=1`, which costs nothing and cannot go stale; a database with a non-empty `-wal` is copied with its `-wal` into the CodeBurn cache and read there, so its un-checkpointed rows are never silently dropped. The copy costs one database's worth of disk and is taken once per change — it is keyed by the main-plus-WAL fingerprint, published under a fingerprint-stamped name so a refresh never overwrites a copy another process is reading, and superseded copies are evicted once a day has passed without a read, keeping at most one predecessor. If the cache itself cannot be written, the database is skipped with a notice naming it and the reason rather than in silence. The original provider database is never opened writable or modified.
 - **Grok Build now reads the CLI's own completed-turn usage instead of estimating it.** Usage comes from the `turn_completed.usage` records Grok CLI already writes into `updates.jsonl` (`inputTokens`, `outputTokens`, `cachedReadTokens`, `cacheCreationTokens`, `reasoningTokens`), deduplicated by `prompt_id` and emitted as one session-level call from the top-level totals. The previous parser reconstructed an estimate from the running `_meta.totalTokens` context counter, so **existing Grok totals will change materially on upgrade** - on one real 568-session corpus cache-read went from 150K to 96.3M tokens, total tokens from 20.0M to 113.9M, and cost from $36.98 to $56.79. Cache read and cache creation are subsets of input and reasoning is a subset of output, so reasoning is clamped to the record's reported output and split back out to match this repo's exclusive-reasoning contract. `modelUsage` only selects a priced attribution id; multi-model rate attribution stays out of scope, so one session is priced at one model's rate. `costUsdTicks` is ignored because its scale is undocumented. Sessions with no usable record - older CLI versions - keep the old context-curve heuristic and stay flagged estimated. **In a session that has at least one `turn_completed` record, turns without one are not counted at all** (their tokens are dropped rather than estimated), and the session is marked estimated instead of claiming full provider coverage. Cached Grok sessions re-parse once. The daily cache re-derives once on first run after upgrade: this is a global re-derivation of every day and every provider, since the daily cache has no per-provider invalidation, but it reads the warm session cache rather than re-parsing transcripts, so it costs seconds (~3s on the corpus above), and the superseded cache file is retained on disk as the baseline for days no source can still re-derive. (#998)

src/parser.ts

@@ -3020,12 +3020,19 @@ function classifiedTurnSlicedToDays(turn: ClassifiedTurn, days: Set<string>): Cl
   return { ...turn, assistantCalls: inRangeCalls, timestamp: inRangeCalls[0]!.timestamp }
 }
 
-async function parseProviderSources(
+export async function parseProviderSources(
   providerName: string,
   sources: SessionSource[],
   seenKeys: Set<string>,
   diskCache: SessionCache,
   dateRange?: DateRange,
+  // Cold-run robustness: called after each source's cache entry lands (mirrors
+  // scanProjectDirs' onFileParsed) so a throttled caller can persist partial
+  // progress mid-provider. Without this a run killed during a large single-
+  // provider phase (e.g. a multi-GB codex corpus) restarted that whole phase
+  // from zero, even though Claude's phase already survived the same kill via
+  // scanProjectDirs.
+  onFileParsed?: () => Promise<void>,
   readOnly = false,
 ): Promise<ProjectSummary[]> {
   const provider = await getProvider(providerName)
@@ -3251,8 +3258,11 @@ async function parseProviderSources(
         section.files[source.path] = { fingerprint: fp, mcpInventory: [], turns: [], failed: true }
         markCacheDirty(diskCache, providerName, source.path)
         warnProviderParseFailure(providerName, source.path, err)
-        continue
       }
+      // Outside the try/catch (matches scanProjectDirs' onFileParsed placement)
+      // so a throttled caller only ever observes this source's cache entry once
+      // it is fully installed above — success or recorded failure, never mid-write.
+      if (onFileParsed) await onFileParsed()
     }
   } finally {
     await pool?.close()
@@ -4152,7 +4162,7 @@ async function runParse(
   for (const [providerName, sources] of providerGroups) {
     emitScanProgress({ kind: 'provider', provider: providerName, state: 'start' })
     try {
-      const projects = await parseProviderSources(providerName, sources, seenKeys, diskCache, dateRange, readOnly)
+      const projects = await parseProviderSources(providerName, sources, seenKeys, diskCache, dateRange, saveProgress, readOnly)
       emitScanProgress({ kind: 'provider', provider: providerName, state: 'done', files: sources.length })
       otherProjects.push(...projects)
     } catch (err) {
@@ -4181,7 +4191,7 @@ async function runParse(
     // constant — both checks are O(1) and avoid a getProvider() dynamic-import
     // round-trip for every unprocessed provider in the disk cache.
     if (!section.durable && !DURABLE_PROVIDER_NAMES.has(providerName)) continue
-    const projects = await parseProviderSources(providerName, [], seenKeys, diskCache, dateRange, readOnly)
+    const projects = await parseProviderSources(providerName, [], seenKeys, diskCache, dateRange, saveProgress, readOnly)
     otherProjects.push(...projects)
   }
 

tests/provider-phase-partial-save.test.ts

@@ -0,0 +1,124 @@
+// Regression test for the codex/non-Claude cold-parse restart-loop: a run
+// killed mid-provider (app timeout SIGKILL, crash, force-quit) used to lose
+// the ENTIRE provider phase, because scanProjectDirs received a saveProgress
+// callback for throttled partial-progress persistence but parseProviderSources
+// did not — non-Claude providers only persisted at the whole-provider
+// boundary. This exercises the callback parseProviderSources now accepts,
+// mirroring scanProjectDirs' onFileParsed.
+//
+// Uses the kiro provider as the multi-file fixture (simple JSON parser, no
+// worker-pool indirection) so the test can call parseProviderSources directly
+// and count exactly how many files get (re)parsed on each run.
+
+import { describe, it, expect, beforeEach, afterAll, vi } from 'vitest'
+import { mkdir, rm, writeFile } from 'fs/promises'
+import { join } from 'path'
+
+import { parseProviderSources } from '../src/parser.js'
+import { getProvider } from '../src/providers/index.js'
+import { CACHE_VERSION, type SessionCache } from '../src/session-cache.js'
+
+// The kiro provider singleton captures homedir() when its module is first
+// imported, so HOME must point at the test root before ../src/parser.js is
+// evaluated (same constraint as tests/kiro-cache-invalidation.test.ts).
+const testRoot = vi.hoisted(() => {
+  const root = `${process.env['TMPDIR'] || '/tmp'}/kiro-partial-save-${process.pid}-${Date.now()}`
+  process.env['HOME'] = `${root}/home`
+  process.env['USERPROFILE'] = `${root}/home`
+  return root
+})
+const HOME = join(testRoot, 'home')
+
+function kiroAgentDir(): string {
+  if (process.platform === 'darwin') {
+    return join(HOME, 'Library', 'Application Support', 'Kiro', 'User', 'globalStorage', 'kiro.kiroagent')
+  }
+  if (process.platform === 'win32') {
+    return join(HOME, 'AppData', 'Roaming', 'Kiro', 'User', 'globalStorage', 'kiro.kiroagent')
+  }
+  return join(HOME, '.config', 'Kiro', 'User', 'globalStorage', 'kiro.kiroagent')
+}
+
+// Writes `n` independent execution files (same shape as the existing Kiro
+// fixture in kiro-cache-invalidation.test.ts), each a self-contained "file"
+// parseProviderSources processes and progress-saves individually.
+async function seedExecutionFiles(n: number): Promise<void> {
+  const dir = join(kiroAgentDir(), 'a'.repeat(32), 'b'.repeat(32))
+  await mkdir(dir, { recursive: true })
+  for (let i = 0; i < n; i++) {
+    await writeFile(join(dir, `exec-${i}`), JSON.stringify({
+      executionId: `exec-${i}`,
+      workflowType: 'chat-agent',
+      status: 'succeed',
+      startTime: 1780000000000 + i,
+      chatSessionId: `session-${i}`,
+      context: {
+        messages: [
+          { role: 'human', entries: [{ type: 'text', text: `Question number ${i}` }] },
+          { role: 'bot', entries: [{ type: 'text', text: `Answer number ${i}, with enough content to yield a non-zero token estimate.` }] },
+        ],
+      },
+    }))
+  }
+}
+
+function callCount(projects: Awaited<ReturnType<typeof parseProviderSources>>): number {
+  return projects.flatMap(p => p.sessions).flatMap(s => s.turns).flatMap(t => t.assistantCalls).length
+}
+
+beforeEach(async () => {
+  await rm(testRoot, { recursive: true, force: true })
+})
+afterAll(async () => {
+  await rm(testRoot, { recursive: true, force: true })
+})
+
+describe('parseProviderSources partial-progress threading', () => {
+  it('a kill right after a mid-provider save resumes without re-parsing already-completed files', async () => {
+    await seedExecutionFiles(4)
+    const provider = await getProvider('kiro')
+    if (!provider) throw new Error('kiro provider unavailable in this environment')
+    const sources = await provider.discoverSessions()
+    expect(sources).toHaveLength(4)
+
+    // Reference: an uninterrupted cold parse of all 4 files.
+    const reference = await parseProviderSources(
+      'kiro', sources, new Set(), { version: CACHE_VERSION, providers: {} }, undefined, undefined, false,
+    )
+    expect(callCount(reference)).toBe(4)
+
+    // Simulated kill: the process dies the instant the 2nd file's cache entry
+    // is durable — `snapshot` is exactly what a throttled disk save would have
+    // persisted at that moment. Throwing from the callback (which fires AFTER
+    // the file's entry lands, outside the per-file try/catch) propagates out of
+    // parseProviderSources uncaught, the same as a real interruption: nothing
+    // after this point ever reaches disk.
+    const KILLED = new Error('simulated kill mid-provider')
+    const killDiskCache: SessionCache = { version: CACHE_VERSION, providers: {} }
+    let killCalls = 0
+    let snapshot: SessionCache | undefined
+    const onFileParsed = async () => {
+      killCalls++
+      if (killCalls === 2) {
+        snapshot = structuredClone(killDiskCache)
+        throw KILLED
+      }
+    }
+    await expect(
+      parseProviderSources('kiro', sources, new Set(), killDiskCache, undefined, onFileParsed, false),
+    ).rejects.toBe(KILLED)
+    expect(killCalls).toBe(2)
+    expect(Object.keys(snapshot?.providers['kiro']?.files ?? {})).toHaveLength(2)
+
+    // Resume: re-run against exactly what the kill left behind. Only the 2
+    // files the kill never reached should get (re)parsed.
+    let resumeParses = 0
+    const resumed = await parseProviderSources(
+      'kiro', sources, new Set(), snapshot!, undefined, async () => { resumeParses++ }, false,
+    )
+    expect(resumeParses).toBe(2)
+
+    // Convergence: the resumed run's totals match the never-interrupted reference.
+    expect(callCount(resumed)).toBe(callCount(reference))
+  })
+})