mirror of
https://github.com/AgentSeal/codeburn.git
synced 2026-10-19 18:23:41 +00:00
perf: store the session cache in day pieces (#1550)
* Store the session cache in day pieces with a per-provider index Layout v10: a provider's files live in pieces by the UTC day of their newest call, each with a key file, and an index row per file (piece, fingerprint, call span, v9 month) read on every load. A ranged load reads the index, the key files and only the members it reports on; a save rewrites only the pieces it touched. Rows keep v9's full-load order and month scope, so orphans, the first-paint snapshot and dedup see what they saw on v9. The v9 month shards re-lay out once, streaming, and are removed after v10 publishes. * Skip cached transcripts outside a narrow query's months when unchanged v9 could not see them, so every narrow query re-parsed each older Claude transcript it discovered and republished the cache. The index knows their fingerprint; an unchanged one needs nothing, a changed one still re-parses as if uncached. * Read piece members in batches through one buffer, and full loads concurrently * Write piece key files without a sync; rebuild lost keys from the piece * Changelog: day pieces * Seed dedup with the keys of unchanged transcripts outside the query's months before a whole-file parse * Upgrade-path check: verify the v10 envelope, indexes, pieces and key files
This commit is contained in:
Resham Joshi
GitHub
parent
3f14d95118
commit
e5dff62fa1
14 changed files with 1397 additions and 800 deletions
|
|
@ -7,6 +7,7 @@
|
|||
- **Menu bar apps 0.9.14 to 0.9.24 load Today again with a 0.9.25 CLI.** Homebrew and npm update the CLI on their own, and 0.9.25 left `avgCostPerSession` out of a project row whose session count is a lower bound, a key those apps require, so the popover showed "Couldn't load Today". The payload always carries it again, with `sessionCountBasis` still telling 0.9.25 and later not to show a partial average. A new test checks the payload against the keys every released menu bar app requires. Reported in #1541, thanks @AleksandrLiadov.
|
||||
- **Update in the menu bar app restarts it into the new version.** The app runs `codeburn menubar --force` as its own child, and `pkill`/`pgrep` skip their own ancestors unless told otherwise, so the running app was never stopped: the bundle was replaced and `open` re-activated the old copy. The CLI now finds the app among its ancestors, swaps the bundle, exits, and a detached helper stops the old copy and opens the new one. Run from a terminal, it stops the app first as before. Because the fix is in the CLI, older menu bar apps update correctly once the CLI is current. Reported in #1541, thanks @AleksandrLiadov.
|
||||
- **OpenCode 2.x: upgraded history is no longer lost and v2 turns price like 1.x ones.** OpenCode 2 copies 1.x sessions into new tables on first start, but the copy drops compaction summary and task-tool turns, skips messages it cannot decode, and can stop partway. CodeBurn read only the copy for any migrated session, so on one real upgraded database more than half the sessions and three quarters of the all-time OpenCode cost disappeared. A session present in both generations now reads its legacy turns first and adds only the turns written after the upgrade, so pre-upgrade figures match 1.x exactly. v2 turns also carried a `provider/model` id that failed pricing for ids like `opencode/claude-haiku-4-5` or `github-copilot/gpt-5` so they showed as $0 in reports; they now use the bare id 1.x used. OpenCode sessions re-parse once. Users on CodeBurn 0.9.24 or older see no OpenCode 2 usage at all and need 0.9.25 or later. Reported by a user.
|
||||
- **The session cache is stored by day, so a today view reads only today.** Each provider keeps an index of every cached session and one file per day of activity. A narrow range reads the index, the duplicate-check keys and only the sessions it reports on, and saving an appended session rewrites one day instead of a month. Older Claude transcripts are no longer re-parsed on every narrow query. The old cache converts once on first run, a session at a time. On a heavy corpus `overview -p today` drops from about 3.1 s and 690 MB to 0.8 s and 560 MB; on a normal one from 0.54 s to 0.41 s. Figures are unchanged.
|
||||
- **`overview`, `status` and the menu bar load only the part of the session cache a date range can use.** Cached sessions outside the range stay on disk and are held in memory as small stubs that still take part in duplicate checks and clean-up, so figures are unchanged. Shards are written one session per line, so a month past 512 MB is saved instead of silently re-parsed on every run, and Codex project names come from a small index instead of the whole results cache. On a heavy corpus `overview -p today` drops from about 3.4 GB to 1.2 GB and runs under a 1 GB heap where it used to crash; on a normal one it drops from about 970 MB to 280 MB and gets faster. Closes #1504.
|
||||
- **An expired Claude Code login is recovered from the Keychain, and `codeburn quota` and the desktop app say so when it cannot be.** Both providers read the OAuth credential from `~/.claude/.credentials.json` and fall back to the macOS Keychain, which is where Claude Code keeps it on a normal Mac. After a 401 the retry re-read the file whatever the credential had come from, so a Keychain user got `Temporarily unavailable.` until Claude Code was run again, and the desktop Plans card kept showing the last percentage it had fetched. A 401 now re-reads the store the credential came from, which picks up a login Claude Code has since renewed. When nothing newer is there and the credential's `expiresAt` has passed, the row reports a terminal failure with a line telling you to run Claude Code once. The desktop app still reads the Keychain only on Check now or a forced refresh, and a background poll keeps that reconnect prompt instead of falling back to Check now. A 401 on a credential still within its life stays transient with its backoff. (#1516, thanks @ozymandiashh)
|
||||
- Export daily rows now bucket each call on its own day, like the history and the summary, and no longer drop a turn that has no timestamp.
|
||||
|
|
|
|||
|
|
@ -32,7 +32,7 @@ const WORK = process.env['UPGRADE_PATH_WORK'] || join(tmpdir(), 'codeburn upgrad
|
|||
// else these two are the knobs to move, and the assertions below will say so.
|
||||
const OLD_SESSION_CACHE = 'session-cache.v7.json'
|
||||
const OLD_DAILY_CACHE = 'daily-cache.v17.json'
|
||||
const NEW_SESSION_CACHE_DIR = 'session-cache.v9'
|
||||
const NEW_SESSION_CACHE_DIR = 'session-cache.v10'
|
||||
|
||||
const HOME = join(WORK, 'user home')
|
||||
const PAYLOADS = join(WORK, 'payloads')
|
||||
|
|
@ -136,24 +136,34 @@ function capture(bin, cacheDir, outDir, extra = {}) {
|
|||
// product, so two invocations legitimately differ there.
|
||||
const stripGenerated = obj => JSON.parse(JSON.stringify(obj, (k, v) => (k.startsWith('generated') || k === 'liveSessions' ? undefined : v)))
|
||||
|
||||
// Shards carry real stat data and are published under a random filename, so
|
||||
// "identical" means identical after normalizing both away.
|
||||
// Pieces, their key files and the indexes carry real stat data and are
|
||||
// published under a random filename, so "identical" means identical after
|
||||
// normalizing both away. An index row is
|
||||
// [path, day, month, until, dev, ino, mtimeMs, size, first, last, flags, offset, length].
|
||||
// Rows are compared by path, not position: a date-ranged run reorders them
|
||||
// exactly as v9 reordered its month shards, and v9's check ignored order too.
|
||||
function shardSnapshot(cacheDir) {
|
||||
const dir = join(cacheDir, NEW_SESSION_CACHE_DIR)
|
||||
if (!existsSync(dir)) return null
|
||||
const out = {}
|
||||
for (const name of readdirSync(dir).sort()) {
|
||||
if (name === 'envelope.json') continue
|
||||
if (name === 'envelope.json' || name.endsWith('.tmp')) continue
|
||||
const body = JSON.parse(readFileSync(join(dir, name), 'utf8'))
|
||||
for (const entry of Object.values(body)) {
|
||||
if (entry && typeof entry === 'object' && entry.fingerprint) {
|
||||
delete entry.fingerprint.dev
|
||||
delete entry.fingerprint.ino
|
||||
delete entry.fingerprint.mtimeMs
|
||||
if (name.startsWith('index.')) {
|
||||
body.pieces = Object.keys(body.pieces).sort()
|
||||
for (const row of body.rows) row.splice(4, 3)
|
||||
body.rows.sort((a, b) => (a[0] < b[0] ? -1 : a[0] > b[0] ? 1 : 0))
|
||||
} else {
|
||||
for (const entry of Object.values(body)) {
|
||||
if (entry && typeof entry === 'object' && entry.fingerprint) {
|
||||
delete entry.fingerprint.dev
|
||||
delete entry.fingerprint.ino
|
||||
delete entry.fingerprint.mtimeMs
|
||||
}
|
||||
}
|
||||
}
|
||||
// Key the bucket off the shard's provider.month prefix, dropping the nonce.
|
||||
out[name.replace(/\.[0-9a-f]{16}\.json$/, '')] = sortDeep(body)
|
||||
// Key by provider.day (or index.provider), dropping the nonce.
|
||||
out[name.replace(/\.[0-9a-f]{16}(\.keys)?\.json$/, '$1')] = sortDeep(body)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
|
@ -211,9 +221,26 @@ check(!existsSync(join(upgradeCache, OLD_SESSION_CACHE)), `${OLD_SESSION_CACHE}
|
|||
check(existsSync(join(upgradeCache, NEW_SESSION_CACHE_DIR)), `${NEW_SESSION_CACHE_DIR}/ present`)
|
||||
check(existsSync(join(upgradeCache, NEW_SESSION_CACHE_DIR, 'envelope.json')), `${NEW_SESSION_CACHE_DIR}/envelope.json present`)
|
||||
const envelope = JSON.parse(readFileSync(join(upgradeCache, NEW_SESSION_CACHE_DIR, 'envelope.json'), 'utf8'))
|
||||
check(envelope.version === 9 && Object.keys(envelope.providers ?? {}).length > 0,
|
||||
check(envelope.version === 10 && Object.keys(envelope.providers ?? {}).length > 0,
|
||||
`envelope at version ${envelope.version} with ${Object.keys(envelope.providers ?? {}).length} providers`)
|
||||
check(readdirSync(join(upgradeCache, NEW_SESSION_CACHE_DIR)).some(n => n !== 'envelope.json'), 'shards published alongside the envelope')
|
||||
// Every provider names an index, every index names its pieces, and every piece
|
||||
// has its key file; together they hold one member per index row.
|
||||
{
|
||||
const dir = join(upgradeCache, NEW_SESSION_CACHE_DIR)
|
||||
let rows = 0, members = 0, missing = []
|
||||
for (const [provider, meta] of Object.entries(envelope.providers ?? {})) {
|
||||
if (!meta.index || !existsSync(join(dir, meta.index))) { missing.push(`${provider} index`); continue }
|
||||
const index = JSON.parse(readFileSync(join(dir, meta.index), 'utf8'))
|
||||
rows += index.rows.length
|
||||
for (const piece of Object.values(index.pieces)) {
|
||||
const keys = piece.replace(/\.json$/, '.keys.json')
|
||||
if (!existsSync(join(dir, piece)) || !existsSync(join(dir, keys))) { missing.push(piece); continue }
|
||||
members += Object.keys(JSON.parse(readFileSync(join(dir, piece), 'utf8'))).length
|
||||
}
|
||||
}
|
||||
check(missing.length === 0 && rows > 0, `indexes, pieces and key files published alongside the envelope${missing.length ? ` (missing: ${missing.join(', ')})` : ''}`)
|
||||
check(rows === members, `every index row has its member: ${rows} rows, ${members} members`)
|
||||
}
|
||||
const newDailyCache = newestDailyCacheAfter(upgradeCache, OLD_DAILY_CACHE)
|
||||
check(newDailyCache !== null, `${newDailyCache ?? 'current daily cache'} re-derived`)
|
||||
check(existsSync(join(upgradeCache, OLD_DAILY_CACHE)), `${OLD_DAILY_CACHE} kept as the carry-forward baseline`)
|
||||
|
|
@ -301,7 +328,7 @@ const readExport = dir => stripGenerated(JSON.parse(readFileSync(join(PAYLOADS,
|
|||
check(JSON.stringify(readExport('workers-0')) === JSON.stringify(readExport('workers-3')),
|
||||
'export payload identical with and without workers')
|
||||
check(JSON.stringify(shardSnapshot(serialCache)) === JSON.stringify(shardSnapshot(parallelCache)),
|
||||
'shard bodies identical with and without workers (fingerprint stat data and shard nonces normalized)')
|
||||
'piece, key file and index bodies identical with and without workers (fingerprint stat data and nonces normalized)')
|
||||
|
||||
// A forced pool that never actually spawned would make the check above vacuous.
|
||||
const verbose = run(newBin.cmd, [...newBin.args, 'status', '--format', 'json', '--period', 'all'], {
|
||||
|
|
@ -329,7 +356,7 @@ if (pending.length === 0) skip('no "parse workers=" line on a warm run; cannot c
|
|||
else check(pending.every(l => /0 pending files|no full parses pending/.test(l)),
|
||||
`nothing re-parsed on the warm run: ${pending.map(l => l.replace(/^codeburn: /, '').trim()).join(' | ')}`)
|
||||
|
||||
check(JSON.stringify(shardSnapshot(upgradeCache)) === JSON.stringify(beforeBodies), 'warm run left every shard body unchanged')
|
||||
check(JSON.stringify(shardSnapshot(upgradeCache)) === JSON.stringify(beforeBodies), 'warm run left every piece, key file and index unchanged')
|
||||
check(JSON.stringify(stripGenerated(warm.menubar)) === JSON.stringify(stripGenerated(upgraded.menubar)),
|
||||
'warm run reports the same payload as the run that migrated the cache')
|
||||
|
||||
|
|
|
|||
|
|
@ -29,6 +29,7 @@ import {
|
|||
type SessionCache,
|
||||
beginColdHydration,
|
||||
cacheEntriesInLoadOrder,
|
||||
cacheHiddenFingerprint,
|
||||
cacheStubs,
|
||||
cleanupOrphanedTempFiles,
|
||||
computeEnvFingerprint,
|
||||
|
|
@ -42,6 +43,7 @@ import {
|
|||
isCacheDirty,
|
||||
isCacheStub,
|
||||
loadCache,
|
||||
loadCacheHiddenKeys,
|
||||
loadCacheStubs,
|
||||
markCacheDirty,
|
||||
markProviderComplete,
|
||||
|
|
@ -2062,6 +2064,7 @@ async function scanProjectDirs(
|
|||
}))
|
||||
}
|
||||
const servedStubs: string[][][] = []
|
||||
const hiddenUnchanged: string[] = []
|
||||
|
||||
for (const [i, { filePath, dirName, source }] of discovered.entries()) {
|
||||
allDiscoveredFiles.add(filePath)
|
||||
|
|
@ -2071,7 +2074,13 @@ async function scanProjectDirs(
|
|||
const cached = section.files[filePath]
|
||||
const stub = cached ? undefined : stubs?.get(filePath)
|
||||
const action = reconcileFile(fp, cached ?? stub)
|
||||
if (!readOnly && deferToBackgroundFill(filePath, fp, cached ?? stub)) {
|
||||
// Cached, but in no month this range reports on, and unchanged: nothing to
|
||||
// parse or serve. A changed one re-parses as if uncached, as it always did.
|
||||
const hidden = cached || stub || readOnly ? undefined : cacheHiddenFingerprint(diskCache, 'claude', filePath)
|
||||
if (hidden && reconcileFile(fp, { fingerprint: hidden }).action === 'unchanged') {
|
||||
hiddenUnchanged.push(filePath)
|
||||
continue
|
||||
} else if (!readOnly && deferToBackgroundFill(filePath, fp, cached ?? stub)) {
|
||||
continue
|
||||
} else if (stub && (readOnly || action.action === 'unchanged')) {
|
||||
if (readOnly && action.action !== 'unchanged') readOnlyServedStale = true
|
||||
|
|
@ -2134,6 +2143,11 @@ async function scanProjectDirs(
|
|||
}
|
||||
}
|
||||
for (const keys of servedStubs) for (const turn of keys) for (const key of turn) seenMsgIds.add(key)
|
||||
// Only a whole-file parse can replay an older transcript's message ids (a
|
||||
// fork or resume); an append's new region never does, so it reads no keys.
|
||||
if (hiddenUnchanged.length > 0 && changedFiles.some(f => !f.append)) {
|
||||
for (const keys of await loadCacheHiddenKeys(diskCache, 'claude', hiddenUnchanged)) for (const turn of keys) for (const key of turn) seenMsgIds.add(key)
|
||||
}
|
||||
|
||||
const parseProgress = createScanProgress('parsing changed claude sessions', changedFiles.length)
|
||||
const progressTotal = changedFiles.length
|
||||
|
|
|
|||
1529
src/session-cache.ts
1529
src/session-cache.ts
File diff suppressed because it is too large
Load diff
|
|
@ -113,16 +113,17 @@ describe('bounded load through the provider parse', () => {
|
|||
expect(JSON.stringify(boundedSnapshot)).toBe(JSON.stringify(await snapshot('all')))
|
||||
})
|
||||
|
||||
it('evicts a deleted transcript that was only a stub from its shard on disk', async () => {
|
||||
it('evicts a deleted transcript that was only a stub from its piece on disk', async () => {
|
||||
const doomed = rollout('04-10', 'aadoomed', '2099-04-10T10:00:00.000Z', 2)
|
||||
const kept = rollout('04-12', 'cckept', '2099-04-12T10:00:00.000Z', 2)
|
||||
const kept = rollout('04-10', 'cckept', '2099-04-10T12:00:00.000Z', 2)
|
||||
rollout('05-01', 'bbtoday', '2099-05-01T10:00:00.000Z', 2)
|
||||
await parseAllSessions(undefined, 'codex')
|
||||
const aprilShard = async (): Promise<string> => {
|
||||
const aprilPiece = async (): Promise<string> => {
|
||||
const envelope = JSON.parse(await readFile(join(sessionCacheDir(), 'envelope.json'), 'utf-8'))
|
||||
return readFile(join(sessionCacheDir(), envelope.providers.codex.shards['2099-04'].name), 'utf-8')
|
||||
const index = JSON.parse(await readFile(join(sessionCacheDir(), envelope.providers.codex.index), 'utf-8'))
|
||||
return readFile(join(sessionCacheDir(), index.pieces['2099-04-10']), 'utf-8')
|
||||
}
|
||||
const before = await aprilShard()
|
||||
const before = await aprilPiece()
|
||||
const line = (text: string, path: string) => text.split('\n').find(l => l.startsWith(JSON.stringify(path) + ':'))?.replace(/,$/, '')
|
||||
expect(line(before, doomed)).toBeDefined()
|
||||
|
||||
|
|
@ -132,7 +133,7 @@ describe('bounded load through the provider parse', () => {
|
|||
|
||||
rmSync(doomed)
|
||||
await reparse(may1)
|
||||
const after = await aprilShard()
|
||||
const after = await aprilPiece()
|
||||
expect(after).not.toContain(doomed)
|
||||
expect(line(after, kept)).toBe(line(before, kept))
|
||||
expect(Object.keys(JSON.parse(after))).toEqual([kept])
|
||||
|
|
|
|||
|
|
@ -78,7 +78,7 @@ describe.skipIf(!isSqliteAvailable())('cachedCallToApiCall threads nanoAiu', ()
|
|||
})
|
||||
|
||||
it('serves nanoAiu on a cold parse and a warm cache read without a CACHE_VERSION bump', async () => {
|
||||
expect(CACHE_VERSION).toBe(9)
|
||||
expect(CACHE_VERSION).toBe(10)
|
||||
|
||||
clearSessionCache()
|
||||
const cold = await parseAllSessions(undefined, 'copilot')
|
||||
|
|
@ -88,6 +88,6 @@ describe.skipIf(!isSqliteAvailable())('cachedCallToApiCall threads nanoAiu', ()
|
|||
clearSessionCache()
|
||||
const warm = await parseAllSessions(undefined, 'copilot')
|
||||
expect(collectNanoAiu(warm)).toEqual([1_500_000_000])
|
||||
expect(CACHE_VERSION).toBe(9)
|
||||
expect(CACHE_VERSION).toBe(10)
|
||||
})
|
||||
})
|
||||
|
|
|
|||
18
tests/fixtures/session-cache-io.ts
vendored
18
tests/fixtures/session-cache-io.ts
vendored
|
|
@ -10,6 +10,7 @@ import {
|
|||
markCacheDirty,
|
||||
saveCache,
|
||||
sessionCacheDir,
|
||||
type CachedFile,
|
||||
type SessionCache,
|
||||
} from '../../src/session-cache.js'
|
||||
|
||||
|
|
@ -33,3 +34,20 @@ export async function cacheDirSnapshot(): Promise<string> {
|
|||
const parts = await Promise.all(names.map(async name => `${name}:${await readFile(join(dir, name), 'utf-8')}`))
|
||||
return parts.join('\n')
|
||||
}
|
||||
|
||||
/** The published pieces of `provider` (UTC day -> file name), straight off disk. */
|
||||
export async function publishedPieces(provider: string): Promise<Record<string, string>> {
|
||||
const dir = sessionCacheDir()
|
||||
const envelope = JSON.parse(await readFile(join(dir, 'envelope.json'), 'utf-8')) as { providers: Record<string, { index: string }> }
|
||||
const index = envelope.providers[provider]?.index
|
||||
if (!index) return {}
|
||||
return (JSON.parse(await readFile(join(dir, index), 'utf-8')) as { pieces: Record<string, string> }).pieces
|
||||
}
|
||||
|
||||
/** Every member the published pieces of `provider` hold, straight off disk. */
|
||||
export async function publishedMembers(provider: string): Promise<Record<string, CachedFile>> {
|
||||
const dir = sessionCacheDir()
|
||||
const files: Record<string, CachedFile> = {}
|
||||
for (const name of Object.values(await publishedPieces(provider))) Object.assign(files, JSON.parse(await readFile(join(dir, name), 'utf-8')))
|
||||
return files
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
// watcher proved untouched, and coalesces shard publication. Both are freshness
|
||||
// trades, so what is pinned here is when they are NOT taken.
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import { mkdtemp, mkdir, readFile, writeFile, rm, rename } from 'fs/promises'
|
||||
import { mkdtemp, mkdir, writeFile, rm, rename } from 'fs/promises'
|
||||
import { join } from 'path'
|
||||
import { tmpdir } from 'os'
|
||||
|
||||
|
|
@ -14,7 +14,8 @@ import {
|
|||
setShardPublishCoalescing,
|
||||
setSweepWatchSource,
|
||||
} from '../src/parser.js'
|
||||
import { clearLoadCacheMemo, fingerprintFileCount, loadCache, markCacheDirty, saveCache, sessionCacheDir } from '../src/session-cache.js'
|
||||
import { clearLoadCacheMemo, fingerprintFileCount, loadCache, markCacheDirty, saveCache } from '../src/session-cache.js'
|
||||
import { publishedMembers } from './fixtures/session-cache-io.js'
|
||||
|
||||
let tmpDir: string
|
||||
|
||||
|
|
@ -210,15 +211,7 @@ describe('incremental discovery sweep', () => {
|
|||
describe('coalesced shard publication', () => {
|
||||
// Read the published shards straight off disk: the usual test reader drops the
|
||||
// load memo, and dropping it is exactly what retires a held publish.
|
||||
const publishedFiles = async (provider = 'claude'): Promise<Record<string, { turns: unknown[] }>> => {
|
||||
const dir = sessionCacheDir()
|
||||
const envelope = JSON.parse(await readFile(join(dir, 'envelope.json'), 'utf-8')) as
|
||||
{ providers: Record<string, { shards: Record<string, { name: string }> }> }
|
||||
const names = Object.values(envelope.providers[provider]?.shards ?? {}).map(s => s.name)
|
||||
const files: Record<string, { turns: unknown[] }> = {}
|
||||
for (const name of names) Object.assign(files, JSON.parse(await readFile(join(dir, name), 'utf-8')))
|
||||
return files
|
||||
}
|
||||
const publishedFiles = (provider = 'claude'): Promise<Record<string, { turns: unknown[] }>> => publishedMembers(provider)
|
||||
const claudeFiles = async (): Promise<string[]> => Object.keys(await publishedFiles())
|
||||
|
||||
it('publishes the first time, holds the next, and flushes on shutdown', async () => {
|
||||
|
|
|
|||
|
|
@ -220,14 +220,15 @@ async function writeForkedCodexPair(codexHome: string, day: string, tag: string,
|
|||
for (const [name, body] of order) await writeCodexRollout(codexHome, day, name, body)
|
||||
}
|
||||
|
||||
/// Cache shard file names carry a random nonce, so compare bodies keyed by
|
||||
/// `<provider>.<month>` instead of by file name.
|
||||
/// Cache piece file names carry a random nonce, so compare bodies keyed by
|
||||
/// `<provider>.<day>` instead of by file name. An index names pieces by their
|
||||
/// nonce, so it is compared through the pieces it names.
|
||||
async function shardBodies(cacheDir: string): Promise<Record<string, string>> {
|
||||
const dir = join(cacheDir, 'session-cache.v9')
|
||||
const dir = join(cacheDir, 'session-cache.v10')
|
||||
const out: Record<string, string> = {}
|
||||
for (const name of (await readdir(dir).catch(() => []))) {
|
||||
if (name === 'envelope.json' || !name.endsWith('.json')) continue
|
||||
const key = name.split('.').slice(0, 2).join('.')
|
||||
if (name === 'envelope.json' || name.startsWith('index.') || !name.endsWith('.json')) continue
|
||||
const key = name.split('.').slice(0, 2).join('.') + (name.endsWith('.keys.json') ? '.keys' : '')
|
||||
out[key] = createHash('sha256').update(await readFile(join(dir, name))).digest('hex')
|
||||
}
|
||||
return out
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
// discovered source stays, flagged or not
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
|
||||
import { mkdtemp, mkdir, writeFile, rm, unlink } from 'fs/promises'
|
||||
import { mkdtemp, mkdir, readFile, writeFile, rm, unlink } from 'fs/promises'
|
||||
import { tmpdir } from 'os'
|
||||
import { join } from 'path'
|
||||
import { createRequire } from 'node:module'
|
||||
|
|
@ -18,8 +18,8 @@ import { calculateCost } from '../src/models.js'
|
|||
import { aggregateProjectsIntoDays } from '../src/day-aggregator.js'
|
||||
import { DAILY_CACHE_VERSION, currentTzKey, ensureCacheHydrated, saveDailyCache } from '../src/daily-cache.js'
|
||||
import { clearSessionCache, isSessionHydrationComplete, parseAllSessions, setParseReuseValidator } from '../src/parser.js'
|
||||
import { CACHE_VERSION, clearLoadCacheMemo, computeEnvFingerprint, loadCache, PROVIDER_PARSE_VERSIONS, saveCache } from '../src/session-cache.js'
|
||||
import { cacheDirSnapshot, readCacheOnDisk, writeCacheOnDisk } from './fixtures/session-cache-io.js'
|
||||
import { CACHE_VERSION, clearLoadCacheMemo, computeEnvFingerprint, loadCache, PROVIDER_PARSE_VERSIONS, saveCache, sessionCacheDir } from '../src/session-cache.js'
|
||||
import { publishedPieces, readCacheOnDisk, writeCacheOnDisk } from './fixtures/session-cache-io.js'
|
||||
import type { SessionSource, SessionParser, ParsedProviderCall } from '../src/providers/types.js'
|
||||
import { setHome } from './setup/home.js'
|
||||
|
||||
|
|
@ -3555,20 +3555,19 @@ describe.skipIf(!isSqliteAvailable())('(sc) month-sharded cache integration for
|
|||
// retainWhilePresent: >90d rows serve and the entry survives its save.
|
||||
expect(sumInput(await parseAllSessions(undefined, 'copilot'))).toBe(200)
|
||||
|
||||
// Appending an OLDER row moves the file's oldest-turn month, which
|
||||
// re-buckets the shard. The path must live in exactly ONE shard file —
|
||||
// a stale copy in the old bucket would resurrect dropped data on a
|
||||
// partial load. cacheDirSnapshot exposes raw shard bytes; the merged
|
||||
// Appending an OLDER row moves the file's oldest-turn month. The path must
|
||||
// live in exactly ONE piece file — a stale copy would resurrect dropped
|
||||
// data on a partial load. The raw piece bytes show it; the merged
|
||||
// loadCache view (which collapses duplicates) cannot see this.
|
||||
insertStoreRow(dbPath, 'sess-ret', 150, 0, 0, at(130))
|
||||
clearSessionCache()
|
||||
clearLoadCacheMemo()
|
||||
expect(sumInput(await parseAllSessions(undefined, 'copilot'))).toBe(350)
|
||||
|
||||
const snapshot = await cacheDirSnapshot()
|
||||
const shardSections = snapshot.split('\n').filter(s => s.length > 0 && !s.startsWith('envelope.json:'))
|
||||
// The shards are JSON, so a Windows path appears with its separators escaped.
|
||||
const holding = shardSections.filter(s => s.includes(JSON.stringify(dbPath).slice(1, -1)))
|
||||
const pieces = Object.values(await publishedPieces('copilot'))
|
||||
const bodies = await Promise.all(pieces.map(name => readFile(join(sessionCacheDir(), name), 'utf-8')))
|
||||
// The pieces are JSON, so a Windows path appears with its separators escaped.
|
||||
const holding = bodies.filter(s => s.includes(JSON.stringify(dbPath).slice(1, -1)))
|
||||
expect(holding).toHaveLength(1)
|
||||
const disk = await readCacheOnDisk()
|
||||
expect(disk.providers['copilot']!.files[dbPath]!.turns.flatMap(t => t.calls)).toHaveLength(3)
|
||||
|
|
|
|||
|
|
@ -130,7 +130,7 @@ describe('vercel-gateway end-to-end (parseAllSessions network path)', () => {
|
|||
// halves of that round-trip — the total above is the read half, the stored
|
||||
// record is the write half.
|
||||
const { readdirSync, readFileSync } = await import('node:fs')
|
||||
const cacheRoot = join(cacheDir, 'session-cache.v9')
|
||||
const cacheRoot = join(cacheDir, 'session-cache.v10')
|
||||
const stored = readdirSync(cacheRoot)
|
||||
.map(f => readFileSync(join(cacheRoot, f), 'utf-8'))
|
||||
.join('')
|
||||
|
|
|
|||
|
|
@ -1,8 +1,7 @@
|
|||
// A ranged load holds the cached files its range cannot report on as stubs.
|
||||
// These pin what that must never cost: a save rewrites a bucket holding stubs
|
||||
// without losing or rebuilding their members, a single-line shard from an
|
||||
// older build converts on the next save, and a wider request in the same
|
||||
// process loads the members it now needs.
|
||||
// These pin what that must never cost: a save leaves the pieces of stubs
|
||||
// alone, a single-line v9 shard re-lays out into pieces, and a wider request
|
||||
// in the same process loads the members it now needs.
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
|
||||
import { mkdir, readFile, rm, writeFile } from 'fs/promises'
|
||||
import { existsSync } from 'fs'
|
||||
|
|
@ -70,22 +69,22 @@ async function publish(files: Record<string, CachedFile>): Promise<void> {
|
|||
clearLoadCacheMemo()
|
||||
}
|
||||
|
||||
async function mayShard(): Promise<{ name: string; text: string }> {
|
||||
const envelope = JSON.parse(await readFile(join(sessionCacheDir(), 'envelope.json'), 'utf-8'))
|
||||
const name = envelope.providers.claude.shards['2026-05'].name as string
|
||||
return { name, text: await readFile(join(sessionCacheDir(), name), 'utf-8') }
|
||||
}
|
||||
|
||||
function memberLine(text: string, path: string): string | undefined {
|
||||
return text.split('\n').find(line => line.startsWith(JSON.stringify(path) + ':'))?.replace(/,$/, '')
|
||||
async function pieceOf(day: string): Promise<{ name: string; text: string }> {
|
||||
const dir = sessionCacheDir()
|
||||
const envelope = JSON.parse(await readFile(join(dir, 'envelope.json'), 'utf-8'))
|
||||
const index = JSON.parse(await readFile(join(dir, envelope.providers.claude.index), 'utf-8'))
|
||||
const name = index.pieces[`2026-05-${day}`] as string
|
||||
return { name, text: await readFile(join(dir, name), 'utf-8') }
|
||||
}
|
||||
|
||||
describe('bounded ranged load', () => {
|
||||
it('rewrites a bucket holding stubs by copying their lines, in full-load order', async () => {
|
||||
it('rewrites only the pieces that changed, and keeps full-load order', async () => {
|
||||
await publish({ '/p/x.jsonl': fileAt('20', 'x'), '/p/y.jsonl': fileAt('02', 'y'), '/p/w.jsonl': fileAt('03', 'w') })
|
||||
const before = await mayShard()
|
||||
expect(before.text.startsWith('{\n')).toBe(true)
|
||||
expect(JSON.parse(before.text)).toEqual({ '/p/x.jsonl': fileAt('20', 'x'), '/p/y.jsonl': fileAt('02', 'y'), '/p/w.jsonl': fileAt('03', 'w') })
|
||||
const y = await pieceOf('02')
|
||||
const w = await pieceOf('03')
|
||||
const x = await pieceOf('20')
|
||||
expect(x.text.startsWith('{\n')).toBe(true)
|
||||
expect(JSON.parse(x.text)).toEqual({ '/p/x.jsonl': fileAt('20', 'x') })
|
||||
|
||||
const cache = await loadCache(day20)
|
||||
const section = cache.providers['claude']!
|
||||
|
|
@ -100,53 +99,46 @@ describe('bounded ranged load', () => {
|
|||
markCacheDirty(cache, 'claude', '/p/z.jsonl')
|
||||
expect(await saveCache(cache)).toBe(true)
|
||||
|
||||
const after = await mayShard()
|
||||
expect(after.name).not.toBe(before.name)
|
||||
expect(memberLine(after.text, '/p/y.jsonl')).toBe(memberLine(before.text, '/p/y.jsonl'))
|
||||
expect(memberLine(after.text, '/p/w.jsonl')).toBe(memberLine(before.text, '/p/w.jsonl'))
|
||||
const parsed = JSON.parse(after.text) as Record<string, CachedFile>
|
||||
// A full load would hold the re-parsed file after the loaded ones.
|
||||
expect(Object.keys(parsed)).toEqual(['/p/y.jsonl', '/p/w.jsonl', '/p/x.jsonl', '/p/z.jsonl'])
|
||||
expect(parsed['/p/x.jsonl']).toEqual(fileAt('20', 'x', 1))
|
||||
expect(parsed['/p/y.jsonl']).toEqual(fileAt('02', 'y'))
|
||||
// The stubs' pieces are not touched at all.
|
||||
expect(await pieceOf('02')).toEqual(y)
|
||||
expect(await pieceOf('03')).toEqual(w)
|
||||
const after = await pieceOf('20')
|
||||
expect(after.name).not.toBe(x.name)
|
||||
expect(JSON.parse(after.text)).toEqual({ '/p/x.jsonl': fileAt('20', 'x', 1) })
|
||||
|
||||
// Dirty but unchanged: the published shard is kept.
|
||||
// Dirty but unchanged: the published piece is kept.
|
||||
markCacheDirty(cache, 'claude', '/p/x.jsonl')
|
||||
expect(await saveCache(cache)).toBe(true)
|
||||
expect((await mayShard()).name).toBe(after.name)
|
||||
expect((await pieceOf('20')).name).toBe(after.name)
|
||||
|
||||
// A full load would hold the re-parsed file after the loaded ones.
|
||||
clearLoadCacheMemo()
|
||||
const full = await loadCache()
|
||||
expect(full.providers['claude']!.files).toEqual(parsed)
|
||||
expect(Object.keys(full.providers['claude']!.files)).toEqual(['/p/y.jsonl', '/p/w.jsonl', '/p/x.jsonl', '/p/z.jsonl'])
|
||||
expect(full.providers['claude']!.files['/p/x.jsonl']).toEqual(fileAt('20', 'x', 1))
|
||||
})
|
||||
|
||||
it('converts a single-line shard from an older build on the next save and still loads stubs from it', async () => {
|
||||
it('re-lays a single-line v9 shard out and still loads stubs from its pieces', async () => {
|
||||
const files = { '/p/x.jsonl': fileAt('20', 'x'), '/p/y.jsonl': fileAt('02', 'y') }
|
||||
const dir = sessionCacheDir()
|
||||
await mkdir(dir, { recursive: true })
|
||||
await writeFile(join(dir, 'claude.2026-05.0123456789abcdef.json'), JSON.stringify(files))
|
||||
await writeFile(join(dir, 'envelope.json'), JSON.stringify({
|
||||
version: CACHE_VERSION, complete: true, nonce: 'legacy',
|
||||
const v9 = join(TMP_DIR, 'session-cache.v9')
|
||||
await mkdir(v9, { recursive: true })
|
||||
await writeFile(join(v9, 'claude.2026-05.0123456789abcdef.json'), JSON.stringify(files))
|
||||
await writeFile(join(v9, 'envelope.json'), JSON.stringify({
|
||||
version: 9, complete: true, nonce: 'legacy',
|
||||
providers: { claude: { envFingerprint: computeEnvFingerprint('claude'), complete: true, shards: { '2026-05': { name: 'claude.2026-05.0123456789abcdef.json', until: '2026-05' } } } },
|
||||
}))
|
||||
|
||||
const cache = await loadCache(day20)
|
||||
expect(existsSync(v9)).toBe(false)
|
||||
expect(Object.keys(cache.providers['claude']!.files)).toEqual(['/p/x.jsonl'])
|
||||
expect([...cacheStubs(cache, 'claude')!.keys()]).toEqual(['/p/y.jsonl'])
|
||||
// Any save rewrites it, even one whose only change is elsewhere.
|
||||
cache.providers['codex'] = { envFingerprint: computeEnvFingerprint('codex'), files: { '/c/r.jsonl': fileAt('20', 'r') } }
|
||||
markCacheDirty(cache, 'codex', '/c/r.jsonl')
|
||||
expect(await saveCache(cache)).toBe(true)
|
||||
expect(JSON.parse((await pieceOf('02')).text)).toEqual({ '/p/y.jsonl': fileAt('02', 'y') })
|
||||
|
||||
const after = await mayShard()
|
||||
expect(after.name).not.toBe('claude.2026-05.0123456789abcdef.json')
|
||||
expect(after.text.startsWith('{\n')).toBe(true)
|
||||
expect(JSON.parse(after.text)).toEqual(files)
|
||||
|
||||
// A wider request in this process loads the stub from its new line.
|
||||
// A wider request in this process loads the stub from its piece.
|
||||
const wider = await loadCache(monthScopeForRange(new Date('2026-05-01T00:00:00.000Z'), new Date('2026-05-20T23:59:59.999Z')))
|
||||
expect(wider).toBe(cache)
|
||||
expect(cacheStubs(cache, 'claude')!.size).toBe(0)
|
||||
expect(cache.providers['claude']!.files['/p/y.jsonl']).toEqual(fileAt('02', 'y'))
|
||||
expect(cache.providers['claude']!.files).toEqual(files)
|
||||
})
|
||||
|
||||
it('reads the same members as a full load for every range', async () => {
|
||||
|
|
|
|||
267
tests/session-cache-day-pieces.test.ts
Normal file
267
tests/session-cache-day-pieces.test.ts
Normal file
|
|
@ -0,0 +1,267 @@
|
|||
// Day pieces (CACHE_VERSION 10): a ranged load reads the index, the key files,
|
||||
// and only the members the range reports on; an append that crosses a day
|
||||
// moves the file between pieces; v9 month shards re-lay out losslessly; and a
|
||||
// narrow query no longer re-parses cached transcripts outside its months.
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { mkdir, readFile, readdir, rm, writeFile } from 'fs/promises'
|
||||
import { existsSync } from 'fs'
|
||||
import { tmpdir } from 'os'
|
||||
import { join } from 'path'
|
||||
|
||||
import {
|
||||
CACHE_VERSION,
|
||||
cacheStubs,
|
||||
clearLoadCacheMemo,
|
||||
computeEnvFingerprint,
|
||||
loadCache,
|
||||
loadCacheStubs,
|
||||
markCacheDirty,
|
||||
monthScopeForRange,
|
||||
saveCache,
|
||||
sessionCacheDir,
|
||||
type CachedFile,
|
||||
type SessionCache,
|
||||
} from '../src/session-cache.js'
|
||||
import { clearSessionCache, parseAllSessions } from '../src/parser.js'
|
||||
|
||||
let TMP_DIR: string
|
||||
|
||||
beforeEach(async () => {
|
||||
TMP_DIR = join(tmpdir(), `codeburn-pieces-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`)
|
||||
process.env['CODEBURN_CACHE_DIR'] = TMP_DIR
|
||||
await mkdir(TMP_DIR, { recursive: true })
|
||||
clearLoadCacheMemo()
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
clearLoadCacheMemo()
|
||||
if (existsSync(TMP_DIR)) await rm(TMP_DIR, { recursive: true })
|
||||
})
|
||||
|
||||
function fileAt(days: string[], key: string): CachedFile {
|
||||
return {
|
||||
fingerprint: { dev: 1, ino: key.length, mtimeMs: 3, sizeBytes: 4 },
|
||||
lastCompleteLineOffset: 9,
|
||||
mcpInventory: [],
|
||||
turns: days.map((day, i) => {
|
||||
const timestamp = `2026-05-${day}T10:00:00.000Z`
|
||||
return {
|
||||
timestamp,
|
||||
sessionId: key,
|
||||
userMessage: 'go',
|
||||
calls: [{
|
||||
provider: 'claude',
|
||||
model: 'claude-sonnet-4-20250514',
|
||||
usage: { inputTokens: 10, outputTokens: 5, cacheCreationInputTokens: 0, cacheReadInputTokens: 0, cachedInputTokens: 0, reasoningTokens: 0, webSearchRequests: 0, cacheCreationOneHourTokens: 0 },
|
||||
costUSD: 0.01,
|
||||
speed: 'standard',
|
||||
timestamp,
|
||||
tools: [],
|
||||
bashCommands: [],
|
||||
skills: [],
|
||||
subagentTypes: [],
|
||||
deduplicationKey: `${key}-${i}`,
|
||||
}],
|
||||
}
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
const may = (day: string) => monthScopeForRange(new Date(`2026-05-${day}T00:00:00.000Z`), new Date(`2026-05-${day}T23:59:59.999Z`))
|
||||
|
||||
async function publish(files: Record<string, CachedFile>): Promise<void> {
|
||||
const cache: SessionCache = { version: CACHE_VERSION, complete: true, providers: { claude: { envFingerprint: computeEnvFingerprint('claude'), files } } }
|
||||
markCacheDirty(cache, 'claude')
|
||||
expect(await saveCache(cache)).toBe(true)
|
||||
clearLoadCacheMemo()
|
||||
}
|
||||
|
||||
async function pieces(): Promise<Record<string, string>> {
|
||||
const dir = sessionCacheDir()
|
||||
const envelope = JSON.parse(await readFile(join(dir, 'envelope.json'), 'utf-8'))
|
||||
return JSON.parse(await readFile(join(dir, envelope.providers.claude.index), 'utf-8')).pieces
|
||||
}
|
||||
|
||||
describe('piece selection', () => {
|
||||
it('reads members only from the pieces the range reports on; the rest come from key files', async () => {
|
||||
const files = { '/p/a.jsonl': fileAt(['01'], 'a'), '/p/b.jsonl': fileAt(['05', '06'], 'b'), '/p/c.jsonl': fileAt(['20'], 'c') }
|
||||
await publish(files)
|
||||
const named = await pieces()
|
||||
expect(Object.keys(named).sort()).toEqual(['2026-05-01', '2026-05-06', '2026-05-20'])
|
||||
// Wreck the members of every piece but the 20th's: a load that read them
|
||||
// would lose those files.
|
||||
for (const day of ['2026-05-01', '2026-05-06']) await writeFile(join(sessionCacheDir(), named[day]!), 'not json')
|
||||
|
||||
const cache = await loadCache(may('20'))
|
||||
expect(cache.providers['claude']!.files).toEqual({ '/p/c.jsonl': files['/p/c.jsonl'] })
|
||||
const stubs = cacheStubs(cache, 'claude')!
|
||||
expect([...stubs.keys()]).toEqual(['/p/a.jsonl', '/p/b.jsonl'])
|
||||
expect(stubs.get('/p/b.jsonl')!.keys).toEqual([['b-0'], ['b-1']])
|
||||
expect(stubs.get('/p/b.jsonl')!.fingerprint).toEqual(files['/p/b.jsonl'].fingerprint)
|
||||
})
|
||||
|
||||
it('rebuilds stub keys from the piece when its key file is gone (key files are not synced)', async () => {
|
||||
const files = { '/p/a.jsonl': fileAt(['01'], 'a'), '/p/c.jsonl': fileAt(['20'], 'c') }
|
||||
await publish(files)
|
||||
await rm(join(sessionCacheDir(), (await pieces())['2026-05-01']!.replace(/\.json$/, '.keys.json')))
|
||||
const cache = await loadCache(may('20'))
|
||||
expect(cacheStubs(cache, 'claude')!.get('/p/a.jsonl')!.keys).toEqual([['a-0']])
|
||||
})
|
||||
})
|
||||
|
||||
describe('an append that crosses a day', () => {
|
||||
it('moves the file to its new piece and rewrites the old one without it', async () => {
|
||||
await publish({ '/p/y.jsonl': fileAt(['02'], 'y'), '/p/v.jsonl': fileAt(['02'], 'v'), '/p/x.jsonl': fileAt(['20'], 'x') })
|
||||
const before = await pieces()
|
||||
const oldPiece = await readFile(join(sessionCacheDir(), before['2026-05-02']!), 'utf-8')
|
||||
|
||||
const cache = await loadCache(may('20'))
|
||||
await loadCacheStubs(cache, 'claude', ['/p/y.jsonl'])
|
||||
cache.providers['claude']!.files['/p/y.jsonl'] = fileAt(['02', '20'], 'y')
|
||||
markCacheDirty(cache, 'claude', '/p/y.jsonl')
|
||||
expect(await saveCache(cache)).toBe(true)
|
||||
|
||||
const after = await pieces()
|
||||
const dir = sessionCacheDir()
|
||||
const day02 = await readFile(join(dir, after['2026-05-02']!), 'utf-8')
|
||||
const day20 = JSON.parse(await readFile(join(dir, after['2026-05-20']!), 'utf-8'))
|
||||
expect(after['2026-05-02']).not.toBe(before['2026-05-02'])
|
||||
expect(existsSync(join(dir, before['2026-05-02']!))).toBe(false)
|
||||
expect(Object.keys(JSON.parse(day02))).toEqual(['/p/v.jsonl'])
|
||||
// The neighbour's line is copied, byte for byte.
|
||||
const line = (text: string) => text.split('\n').find(l => l.startsWith('"/p/v.jsonl":'))!.replace(/,$/, '')
|
||||
expect(line(day02)).toBe(line(oldPiece))
|
||||
expect(Object.keys(day20).sort()).toEqual(['/p/x.jsonl', '/p/y.jsonl'])
|
||||
|
||||
clearLoadCacheMemo()
|
||||
const full = await loadCache()
|
||||
// The rewritten entry follows the loaded ones, where a re-parse puts it on v9.
|
||||
expect(Object.keys(full.providers['claude']!.files)).toEqual(['/p/v.jsonl', '/p/x.jsonl', '/p/y.jsonl'])
|
||||
expect(full.providers['claude']!.files['/p/y.jsonl']).toEqual(fileAt(['02', '20'], 'y'))
|
||||
})
|
||||
})
|
||||
|
||||
describe('v9 -> v10 migration', () => {
|
||||
const v9 = () => join(TMP_DIR, 'session-cache.v9')
|
||||
|
||||
async function writeV9(): Promise<Record<string, CachedFile>> {
|
||||
await mkdir(v9(), { recursive: true })
|
||||
const may = { '/p/b.jsonl': fileAt(['09'], 'b'), '/p/a.jsonl': fileAt(['01', '21'], 'a') }
|
||||
const undated: Record<string, CachedFile> = { '/p/failed.jsonl': { fingerprint: { dev: 1, ino: 9, mtimeMs: 3, sizeBytes: 4 }, mcpInventory: [], turns: [], failed: true } }
|
||||
// The month shard as stage 1 wrote it (one member per line), the undated
|
||||
// one as main wrote it (one line).
|
||||
const lines = Object.entries(may).map(([k, v]) => `${JSON.stringify(k)}:${JSON.stringify(v)}`)
|
||||
await writeFile(join(v9(), 'claude.2026-05.aa.json'), `{\n${lines.join(',\n')}\n}`)
|
||||
await writeFile(join(v9(), 'claude.0000-00.bb.json'), JSON.stringify(undated))
|
||||
await writeFile(join(v9(), 'envelope.json'), JSON.stringify({ version: 9, complete: true, nonce: 'n', providers: {
|
||||
claude: { envFingerprint: computeEnvFingerprint('claude'), complete: true, completeFrom: 5, shards: {
|
||||
'0000-00': { name: 'claude.0000-00.bb.json', until: '0000-00' },
|
||||
'2026-05': { name: 'claude.2026-05.aa.json', until: '2026-05' },
|
||||
} },
|
||||
} }))
|
||||
return { ...undated, ...may }
|
||||
}
|
||||
|
||||
it('is lossless, keeps v9 load order, and retires v9 once v10 is published', async () => {
|
||||
const expected = await writeV9()
|
||||
const cache = await loadCache()
|
||||
expect(existsSync(v9())).toBe(false)
|
||||
expect(cache.complete).toBe(true)
|
||||
expect(cache.providers['claude']!.complete).toBe(true)
|
||||
expect(cache.providers['claude']!.completeFrom).toBe(5)
|
||||
expect(Object.keys(cache.providers['claude']!.files)).toEqual(['/p/failed.jsonl', '/p/b.jsonl', '/p/a.jsonl'])
|
||||
expect(cache.providers['claude']!.files).toEqual(expected)
|
||||
expect(Object.keys(await pieces()).sort()).toEqual(['0000-00-00', '2026-05-09', '2026-05-21'])
|
||||
// Members moved verbatim.
|
||||
const a = await readFile(join(sessionCacheDir(), (await pieces())['2026-05-21']!), 'utf-8')
|
||||
expect(a).toBe(`{\n${JSON.stringify('/p/a.jsonl')}:${JSON.stringify(expected['/p/a.jsonl'])}\n}`)
|
||||
})
|
||||
|
||||
it('leaves v9 in place when the v10 directory cannot be written', async () => {
|
||||
await writeV9()
|
||||
await writeFile(join(TMP_DIR, `session-cache.v${CACHE_VERSION}`), 'in the way')
|
||||
await loadCache().catch(() => null)
|
||||
expect(existsSync(join(v9(), 'envelope.json'))).toBe(true)
|
||||
expect(await readdir(v9())).toHaveLength(3)
|
||||
})
|
||||
})
|
||||
|
||||
describe('a narrow query and an older transcript', () => {
|
||||
const home = () => join(TMP_DIR, 'home')
|
||||
const saved = { dir: process.env['CLAUDE_CONFIG_DIR'], desktop: process.env['CODEBURN_DESKTOP_SESSIONS_DIR'] }
|
||||
|
||||
beforeEach(() => {
|
||||
process.env['CLAUDE_CONFIG_DIR'] = home()
|
||||
process.env['CODEBURN_DESKTOP_SESSIONS_DIR'] = join(TMP_DIR, 'no-desktop')
|
||||
clearSessionCache()
|
||||
})
|
||||
afterEach(() => {
|
||||
if (saved.dir === undefined) delete process.env['CLAUDE_CONFIG_DIR']; else process.env['CLAUDE_CONFIG_DIR'] = saved.dir
|
||||
if (saved.desktop === undefined) delete process.env['CODEBURN_DESKTOP_SESSIONS_DIR']; else process.env['CODEBURN_DESKTOP_SESSIONS_DIR'] = saved.desktop
|
||||
delete process.env['CODEBURN_PROGRESS']
|
||||
clearSessionCache()
|
||||
})
|
||||
|
||||
async function session(name: string, iso: string): Promise<void> {
|
||||
const dir = join(home(), 'projects', 'proj')
|
||||
await mkdir(dir, { recursive: true })
|
||||
await writeFile(join(dir, `${name}.jsonl`), [
|
||||
{ type: 'user', uuid: `u-${name}`, sessionId: name, timestamp: iso, cwd: '/tmp/proj', message: { role: 'user', content: 'go' } },
|
||||
{ type: 'assistant', uuid: `a-${name}`, sessionId: name, timestamp: iso, cwd: '/tmp/proj',
|
||||
message: { id: `msg-${name}`, type: 'message', role: 'assistant', model: 'claude-sonnet-4-5', content: [], usage: { input_tokens: 100, output_tokens: 50 } } },
|
||||
].map(line => JSON.stringify(line)).join('\n') + '\n')
|
||||
}
|
||||
|
||||
it('does not re-parse a cached June file on a September day, and writes nothing', async () => {
|
||||
await session('june', '2099-06-10T10:00:00Z')
|
||||
await session('today', '2099-09-23T10:00:00Z')
|
||||
await parseAllSessions()
|
||||
const day = { start: new Date('2099-09-23T00:00:00.000Z'), end: new Date('2099-09-23T23:59:59.999Z') }
|
||||
const narrow = async (): Promise<number[]> => {
|
||||
clearSessionCache()
|
||||
clearLoadCacheMemo()
|
||||
const totals: number[] = []
|
||||
process.env['CODEBURN_PROGRESS'] = '1'
|
||||
const spy = vi.spyOn(process.stderr, 'write').mockImplementation(((chunk: unknown) => {
|
||||
const m = /"kind":"tick","provider":"claude","done":0,"total":(\d+)/.exec(String(chunk))
|
||||
if (m) totals.push(Number(m[1]))
|
||||
return true
|
||||
}) as typeof process.stderr.write)
|
||||
try {
|
||||
const projects = await parseAllSessions(day, 'claude')
|
||||
expect(projects.reduce((n, p) => n + p.totalApiCalls, 0)).toBe(1)
|
||||
} finally {
|
||||
spy.mockRestore()
|
||||
delete process.env['CODEBURN_PROGRESS']
|
||||
}
|
||||
return totals
|
||||
}
|
||||
await narrow()
|
||||
const envelope = await readFile(join(sessionCacheDir(), 'envelope.json'), 'utf-8')
|
||||
expect(await narrow()).toEqual([0])
|
||||
expect(await readFile(join(sessionCacheDir(), 'envelope.json'), 'utf-8')).toBe(envelope)
|
||||
})
|
||||
|
||||
it('lets the cached June file keep the calls a fork made today replays', async () => {
|
||||
await session('june', '2099-06-10T10:00:00Z')
|
||||
await parseAllSessions()
|
||||
// A fork of the June session: its message id restated under today's date, then a new call.
|
||||
const dir = join(home(), 'projects', 'proj')
|
||||
await writeFile(join(dir, 'fork.jsonl'), [
|
||||
{ type: 'user', uuid: 'u-fork-0', sessionId: 'fork', timestamp: '2099-09-23T10:00:00Z', cwd: '/tmp/proj', message: { role: 'user', content: 'go' } },
|
||||
{ type: 'assistant', uuid: 'a-fork-0', sessionId: 'fork', timestamp: '2099-09-23T10:00:00Z', cwd: '/tmp/proj',
|
||||
message: { id: 'msg-june', type: 'message', role: 'assistant', model: 'claude-sonnet-4-5', content: [], usage: { input_tokens: 100, output_tokens: 50 } } },
|
||||
{ type: 'user', uuid: 'u-fork-1', sessionId: 'fork', timestamp: '2099-09-23T10:01:00Z', cwd: '/tmp/proj', message: { role: 'user', content: 'more' } },
|
||||
{ type: 'assistant', uuid: 'a-fork-1', sessionId: 'fork', timestamp: '2099-09-23T10:01:00Z', cwd: '/tmp/proj',
|
||||
message: { id: 'msg-fork', type: 'message', role: 'assistant', model: 'claude-sonnet-4-5', content: [], usage: { input_tokens: 100, output_tokens: 50 } } },
|
||||
].map(line => JSON.stringify(line)).join('\n') + '\n')
|
||||
const calls = async (range?: { start: Date; end: Date }): Promise<number> => {
|
||||
clearSessionCache()
|
||||
clearLoadCacheMemo()
|
||||
return (await parseAllSessions(range, 'claude')).reduce((n, p) => n + p.totalApiCalls, 0)
|
||||
}
|
||||
expect(await calls({ start: new Date('2099-09-23T00:00:00.000Z'), end: new Date('2099-09-23T23:59:59.999Z') })).toBe(1)
|
||||
expect(await calls({ start: new Date('2099-06-01T00:00:00.000Z'), end: new Date('2099-06-30T23:59:59.999Z') })).toBe(1)
|
||||
expect(await calls()).toBe(2)
|
||||
})
|
||||
})
|
||||
|
|
@ -1,9 +1,10 @@
|
|||
// Provider x month shard layout (CACHE_VERSION 9): the on-disk cache is a
|
||||
// directory holding one envelope plus one shard per provider-month. What matters
|
||||
// here is that the move off the older layouts loses nothing, that a file's
|
||||
// bucket never moves when the session is appended to, that a save rewrites only
|
||||
// the months that changed (including when the load was scoped to a subset of
|
||||
// them), and that one unreadable shard costs exactly one month.
|
||||
// Provider x day piece layout (CACHE_VERSION 10): the on-disk cache is a
|
||||
// directory holding one envelope, one index per provider and one piece (plus
|
||||
// its key file) per provider-day. What matters here is that the move off the
|
||||
// older layouts loses nothing, that a save rewrites only the pieces that
|
||||
// changed (including when the load was scoped to a subset of the months), that
|
||||
// a file keeps its v9 month for scoping whichever piece it is in, and that one
|
||||
// unreadable piece costs exactly the files it held.
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
|
||||
import { mkdir, readFile, readdir, rm, stat, utimes, writeFile } from 'fs/promises'
|
||||
import { existsSync } from 'fs'
|
||||
|
|
@ -116,10 +117,18 @@ async function shardNames(): Promise<string[]> {
|
|||
return (await readdir(sessionCacheDir())).sort()
|
||||
}
|
||||
|
||||
async function envelope(): Promise<{ providers: Record<string, { shards: Record<string, { name: string; until: string }> }> }> {
|
||||
async function envelope(): Promise<{ providers: Record<string, { index: string }> }> {
|
||||
return JSON.parse(await readFile(join(sessionCacheDir(), 'envelope.json'), 'utf-8'))
|
||||
}
|
||||
|
||||
async function index(provider: string): Promise<{ pieces: Record<string, string>; rows: unknown[][] }> {
|
||||
return JSON.parse(await readFile(join(sessionCacheDir(), (await envelope()).providers[provider]!.index), 'utf-8'))
|
||||
}
|
||||
|
||||
async function pieces(provider: string): Promise<Record<string, string>> {
|
||||
return (await index(provider)).pieces
|
||||
}
|
||||
|
||||
/** name -> bytes, for every shard on disk. */
|
||||
async function shardBytes(): Promise<Map<string, string>> {
|
||||
const dir = sessionCacheDir()
|
||||
|
|
@ -142,9 +151,9 @@ describe('v7 -> shard migration', () => {
|
|||
expect(existsSync(v7Path)).toBe(false)
|
||||
const names = await shardNames()
|
||||
expect(names).toContain('envelope.json')
|
||||
// claude's three entries split by month: two dated 2026-05, one turn-less.
|
||||
expect(Object.keys((await envelope()).providers['claude']!.shards).sort()).toEqual(['0000-00', '2026-05'])
|
||||
expect(Object.keys((await envelope()).providers['codex']!.shards)).toEqual(['2026-05'])
|
||||
// claude's three entries split by day: two called on 2026-05-15, one turn-less.
|
||||
expect(Object.keys(await pieces('claude')).sort()).toEqual(['0000-00-00', '2026-05-15'])
|
||||
expect(Object.keys(await pieces('codex'))).toEqual(['2026-05-15'])
|
||||
|
||||
// A second load reads only the shards and produces the same cache.
|
||||
clearLoadCacheMemo()
|
||||
|
|
@ -191,7 +200,7 @@ describe('per-provider dirty tracking', () => {
|
|||
})
|
||||
|
||||
describe('corrupt shard isolation', () => {
|
||||
it('drops only the unreadable month, keeping every other month and provider', async () => {
|
||||
it('drops only the unreadable piece, keeping every other piece and provider', async () => {
|
||||
const cache: SessionCache = {
|
||||
version: CACHE_VERSION,
|
||||
complete: true,
|
||||
|
|
@ -208,14 +217,14 @@ describe('corrupt shard isolation', () => {
|
|||
await saveCache(cache)
|
||||
|
||||
const dir = sessionCacheDir()
|
||||
await writeFile(join(dir, (await envelope()).providers['claude']!.shards['2026-05']!.name), '{"/x":{"turns":')
|
||||
await writeFile(join(dir, (await pieces('claude'))['2026-05-15']!), '{"/x":{"turns":')
|
||||
|
||||
clearLoadCacheMemo()
|
||||
const reloaded = await loadCache()
|
||||
expect(Object.keys(reloaded.providers['claude']!.files)).toEqual(['/live/jun.jsonl'])
|
||||
expect(reloaded.providers['codex']).toEqual(cache.providers['codex'])
|
||||
|
||||
// Self-heals: the unreadable month is republished from whatever re-parses
|
||||
// Self-heals: the unreadable piece is republished from whatever re-parses
|
||||
// into it rather than being carried forward corrupt forever.
|
||||
reloaded.providers['claude']!.files['/live/may.jsonl'] = fileSpanning('2026-05-15T10:00:00Z')
|
||||
markCacheDirty(reloaded, 'claude', '/live/may.jsonl')
|
||||
|
|
@ -304,9 +313,10 @@ describe('concurrent writers', () => {
|
|||
|
||||
async function assertReferentialIntegrity(expected: string[]): Promise<void> {
|
||||
const dir = sessionCacheDir()
|
||||
for (const meta of Object.values((await envelope()).providers)) {
|
||||
for (const ref of Object.values(meta.shards)) {
|
||||
expect(existsSync(join(dir, ref.name)), `envelope names a missing shard: ${ref.name}`).toBe(true)
|
||||
for (const [provider, meta] of Object.entries((await envelope()).providers)) {
|
||||
expect(existsSync(join(dir, meta.index)), `envelope names a missing index: ${meta.index}`).toBe(true)
|
||||
for (const name of Object.values(await pieces(provider))) {
|
||||
expect(existsSync(join(dir, name)), `index names a missing piece: ${name}`).toBe(true)
|
||||
}
|
||||
}
|
||||
clearLoadCacheMemo()
|
||||
|
|
@ -366,8 +376,8 @@ describe('concurrent writers', () => {
|
|||
})
|
||||
})
|
||||
|
||||
describe('month buckets', () => {
|
||||
it('keeps a file in its first-turn month when the session is appended to', async () => {
|
||||
describe('day pieces', () => {
|
||||
it('moves an appended file to the day of its newest call and keeps its first-turn month', async () => {
|
||||
const cache: SessionCache = {
|
||||
version: CACHE_VERSION,
|
||||
complete: true,
|
||||
|
|
@ -375,26 +385,27 @@ describe('month buckets', () => {
|
|||
}
|
||||
markCacheDirty(cache, 'claude')
|
||||
await saveCache(cache)
|
||||
expect(Object.keys((await envelope()).providers['claude']!.shards)).toEqual(['2026-05'])
|
||||
expect(Object.keys(await pieces('claude'))).toEqual(['2026-05-15'])
|
||||
|
||||
// Two months of appends later the bucket is unchanged; only `until` moves,
|
||||
// which is what lets a ranged load still find this session.
|
||||
// Two months of appends later the file lives in July's piece, but its v9
|
||||
// month is unchanged; only `until` moves, which is what lets a ranged load
|
||||
// still find this session.
|
||||
const appended = fileSpanning('2026-05-15T10:00:00Z', '2026-07-02T10:00:00Z')
|
||||
expect(cacheBucketMonth(appended)).toBe('2026-05')
|
||||
cache.providers['claude']!.files['/live/long.jsonl'] = appended
|
||||
markCacheDirty(cache, 'claude', '/live/long.jsonl')
|
||||
await saveCache(cache)
|
||||
const shards = (await envelope()).providers['claude']!.shards
|
||||
expect(Object.keys(shards)).toEqual(['2026-05'])
|
||||
expect(shards['2026-05']!.until).toBe('2026-07')
|
||||
expect(Object.keys(await pieces('claude'))).toEqual(['2026-07-02'])
|
||||
const [row] = (await index('claude')).rows
|
||||
expect(row!.slice(0, 4)).toEqual(['/live/long.jsonl', '2026-07-02', '2026-05', '2026-07'])
|
||||
|
||||
// ...and a July query still loads it, despite the May bucket key.
|
||||
// ...and a July query still loads it.
|
||||
clearLoadCacheMemo()
|
||||
const scoped = await loadCache(monthScopeForRange(new Date('2026-07-01T00:00:00Z'), new Date('2026-07-31T23:59:59Z')))
|
||||
expect(scoped.providers['claude']!.files['/live/long.jsonl']).toBeDefined()
|
||||
})
|
||||
|
||||
it('rewrites only the month that changed', async () => {
|
||||
it('rewrites only the day that changed', async () => {
|
||||
const cache: SessionCache = {
|
||||
version: CACHE_VERSION,
|
||||
complete: true,
|
||||
|
|
@ -412,10 +423,7 @@ describe('month buckets', () => {
|
|||
markCacheDirty(cache, 'claude')
|
||||
await saveCache(cache)
|
||||
const before = await shardBytes()
|
||||
const untouched = [
|
||||
(await envelope()).providers['claude']!.shards['2026-03']!.name,
|
||||
(await envelope()).providers['claude']!.shards['2026-04']!.name,
|
||||
]
|
||||
const untouched = [(await pieces('claude'))['2026-03-10']!, (await pieces('claude'))['2026-04-10']!]
|
||||
|
||||
cache.providers['claude']!.files['/live/may.jsonl'] = cachedFile({ turns: [turnAt('2026-05-10T10:00:00Z', 'a')], mcpInventory: ['changed'] })
|
||||
markCacheDirty(cache, 'claude', '/live/may.jsonl')
|
||||
|
|
@ -423,10 +431,10 @@ describe('month buckets', () => {
|
|||
|
||||
const after = await shardBytes()
|
||||
for (const name of untouched) expect(after.get(name)).toBe(before.get(name))
|
||||
expect(after.has((await envelope()).providers['claude']!.shards['2026-05']!.name)).toBe(true)
|
||||
expect(after.has((await pieces('claude'))['2026-05-10']!)).toBe(true)
|
||||
})
|
||||
|
||||
it('dirties the month a deleted file was in', async () => {
|
||||
it('dirties the piece a deleted file was in', async () => {
|
||||
const cache: SessionCache = {
|
||||
version: CACHE_VERSION,
|
||||
complete: true,
|
||||
|
|
@ -483,10 +491,7 @@ describe('scoped load', () => {
|
|||
it('save from a scoped load leaves the unloaded months byte-identical', async () => {
|
||||
await seedThreeMonths()
|
||||
const before = await shardBytes()
|
||||
const kept = [
|
||||
(await envelope()).providers['claude']!.shards['2026-03']!.name,
|
||||
(await envelope()).providers['claude']!.shards['2026-04']!.name,
|
||||
]
|
||||
const kept = [(await pieces('claude'))['2026-03-10']!, (await pieces('claude'))['2026-04-10']!]
|
||||
|
||||
clearLoadCacheMemo()
|
||||
const scoped = await loadCache(juneScope)
|
||||
|
|
@ -503,12 +508,12 @@ describe('scoped load', () => {
|
|||
.toEqual(['/live/apr.jsonl', '/live/jun.jsonl', '/live/jun2.jsonl', '/live/mar.jsonl'])
|
||||
})
|
||||
|
||||
it('keeps the unloaded month\'s shard name when a re-parse re-derives the same entry', async () => {
|
||||
it('keeps the unloaded month\'s piece name when a re-parse re-derives the same entry', async () => {
|
||||
await seedThreeMonths()
|
||||
// The March entry is invisible to a June-scoped run, so the reconcile
|
||||
// re-parses that file and writes the identical entry straight back. Nothing
|
||||
// changed, so the March shard must keep its name run after run (#1032).
|
||||
const nameOf = async (): Promise<string> => (await envelope()).providers['claude']!.shards['2026-03']!.name
|
||||
// A changed March file is re-parsed from a June-scoped run and can write
|
||||
// the identical entry straight back. Nothing changed, so the March piece
|
||||
// must keep its name run after run (#1032).
|
||||
const nameOf = async (): Promise<string> => (await pieces('claude'))['2026-03-10']!
|
||||
const before = await nameOf()
|
||||
for (let run = 0; run < 2; run++) {
|
||||
clearLoadCacheMemo()
|
||||
|
|
@ -582,8 +587,8 @@ describe('scoped load', () => {
|
|||
})
|
||||
})
|
||||
|
||||
describe('v8 -> v9 migration', () => {
|
||||
it('re-buckets the v8 provider shards losslessly and retires the v8 directory', async () => {
|
||||
describe('v8 migration', () => {
|
||||
it('re-lays the v8 provider shards out losslessly and retires the v8 directory', async () => {
|
||||
const v8Dir = join(TMP_DIR, 'session-cache.v8')
|
||||
await mkdir(v8Dir, { recursive: true })
|
||||
const section = {
|
||||
|
|
@ -603,7 +608,7 @@ describe('v8 -> v9 migration', () => {
|
|||
expect(loaded.providers['claude']).toEqual(section)
|
||||
expect(loaded.complete).toBe(true)
|
||||
expect(existsSync(v8Dir)).toBe(false)
|
||||
expect(Object.keys((await envelope()).providers['claude']!.shards).sort()).toEqual(['2026-03', '2026-05', '2026-06'])
|
||||
expect(Object.keys(await pieces('claude')).sort()).toEqual(['2026-03-10', '2026-05-15', '2026-06-10'])
|
||||
|
||||
clearLoadCacheMemo()
|
||||
expect(await loadCache()).toEqual(loaded)
|
||||
|
|
@ -631,7 +636,7 @@ describe('out-of-order turns', () => {
|
|||
}
|
||||
markCacheDirty(cache, 'claude')
|
||||
await saveCache(cache)
|
||||
expect((await envelope()).providers['claude']!.shards['2026-03']!.until).toBe('2026-08')
|
||||
expect((await index('claude')).rows[0]!.slice(1, 4)).toEqual(['2026-08-10', '2026-03', '2026-08'])
|
||||
|
||||
for (const [from, to] of [['2026-03-01', '2026-03-31'], ['2026-08-01', '2026-08-31']] as const) {
|
||||
clearLoadCacheMemo()
|
||||
|
|
@ -641,9 +646,8 @@ describe('out-of-order turns', () => {
|
|||
})
|
||||
})
|
||||
|
||||
// A path must never end up in two shards at once: on a later load the two copies
|
||||
// race and the stale one can win, and nothing sweeps it because the envelope
|
||||
// names both.
|
||||
// A path must never end up in two pieces at once: a stale copy could be read
|
||||
// back, and nothing would sweep it.
|
||||
describe('re-bucketing out of an unloaded month', () => {
|
||||
const juneScope = monthScopeForRange(new Date('2026-06-01T00:00:00Z'), new Date('2026-06-30T23:59:59Z'))
|
||||
|
||||
|
|
@ -666,13 +670,13 @@ describe('re-bucketing out of an unloaded month', () => {
|
|||
await saveCache(cache)
|
||||
}
|
||||
|
||||
/** Every shard's view of `path`, so a duplicate is visible directly. */
|
||||
/** Every piece's view of `path`, so a duplicate is visible directly. */
|
||||
async function copiesOf(path: string): Promise<string[]> {
|
||||
const dir = sessionCacheDir()
|
||||
const found: string[] = []
|
||||
for (const [bucket, ref] of Object.entries((await envelope()).providers['claude']!.shards)) {
|
||||
const files = JSON.parse(await readFile(join(dir, ref.name), 'utf-8'))
|
||||
if (files[path]) found.push(bucket)
|
||||
for (const [day, name] of Object.entries(await pieces('claude'))) {
|
||||
const files = JSON.parse(await readFile(join(dir, name), 'utf-8'))
|
||||
if (files[path]) found.push(day)
|
||||
}
|
||||
return found.sort()
|
||||
}
|
||||
|
|
@ -687,7 +691,7 @@ describe('re-bucketing out of an unloaded month', () => {
|
|||
markCacheDirty(scoped, 'claude', '/live/moving.jsonl')
|
||||
await saveCache(scoped)
|
||||
|
||||
expect(await copiesOf('/live/moving.jsonl')).toEqual(['2026-05'])
|
||||
expect(await copiesOf('/live/moving.jsonl')).toEqual(['2026-05-02'])
|
||||
clearLoadCacheMemo()
|
||||
const full = await loadCache()
|
||||
expect(full.providers['claude']!.files['/live/moving.jsonl']!.mcpInventory).toEqual(['reparsed'])
|
||||
|
|
@ -703,31 +707,30 @@ describe('re-bucketing out of an unloaded month', () => {
|
|||
markCacheDirty(scoped, 'claude', '/live/moving.jsonl')
|
||||
await saveCache(scoped)
|
||||
|
||||
expect(await copiesOf('/live/moving.jsonl')).toEqual(['0000-00'])
|
||||
expect(await copiesOf('/live/moving.jsonl')).toEqual(['0000-00-00'])
|
||||
clearLoadCacheMemo()
|
||||
const full = await loadCache()
|
||||
expect(full.providers['claude']!.files['/live/moving.jsonl']!.failed).toBe(true)
|
||||
expect(full.providers['claude']!.files['/live/stay.jsonl']).toBeDefined()
|
||||
})
|
||||
|
||||
it('resolves a duplicate to the freshest copy and prunes it on the next save', async () => {
|
||||
await seed()
|
||||
// Forge the split state directly: the same path in two shards.
|
||||
const dir = sessionCacheDir()
|
||||
const env = await envelope()
|
||||
const janName = env.providers['claude']!.shards['2026-01']!.name
|
||||
const junName = env.providers['claude']!.shards['2026-06']!.name
|
||||
const jun = JSON.parse(await readFile(join(dir, junName), 'utf-8'))
|
||||
jun['/live/moving.jsonl'] = cachedFile({ turns: [turnAt('2026-06-02T10:00:00Z', 'fresh')], fingerprint: { dev: 1, ino: 2, mtimeMs: 999, sizeBytes: 4 }, mcpInventory: ['fresh'] })
|
||||
await writeFile(join(dir, junName), JSON.stringify(jun))
|
||||
it('resolves a v9 duplicate to the freshest copy when it re-lays the shards out', async () => {
|
||||
const v9 = join(TMP_DIR, 'session-cache.v9')
|
||||
await mkdir(v9, { recursive: true })
|
||||
const stale = fileSpanning('2026-01-10T10:00:00Z')
|
||||
const fresh = cachedFile({ turns: [turnAt('2026-06-02T10:00:00Z', 'fresh')], fingerprint: { dev: 1, ino: 2, mtimeMs: 999, sizeBytes: 4 }, mcpInventory: ['fresh'] })
|
||||
await writeFile(join(v9, 'claude.2026-01.a.json'), JSON.stringify({ '/live/moving.jsonl': stale, '/live/stay.jsonl': fileSpanning('2026-01-11T10:00:00Z') }))
|
||||
await writeFile(join(v9, 'claude.2026-06.b.json'), JSON.stringify({ '/live/moving.jsonl': fresh }))
|
||||
await writeFile(join(v9, 'envelope.json'), JSON.stringify({ version: 9, complete: true, nonce: 'n', providers: {
|
||||
claude: { envFingerprint: computeEnvFingerprint('claude'), shards: { '2026-01': { name: 'claude.2026-01.a.json', until: '2026-01' }, '2026-06': { name: 'claude.2026-06.b.json', until: '2026-06' } } },
|
||||
} }))
|
||||
|
||||
clearLoadCacheMemo()
|
||||
const full = await loadCache()
|
||||
// Newest fingerprint wins, whichever shard finished reading first.
|
||||
expect(full.providers['claude']!.files['/live/moving.jsonl']!.mcpInventory).toEqual(['fresh'])
|
||||
await saveCache(full)
|
||||
expect(await copiesOf('/live/moving.jsonl')).toEqual(['2026-06'])
|
||||
expect(existsSync(join(dir, janName))).toBe(false)
|
||||
// Newest fingerprint wins, at the first copy's place in load order.
|
||||
expect(Object.keys(full.providers['claude']!.files)).toEqual(['/live/moving.jsonl', '/live/stay.jsonl'])
|
||||
expect(full.providers['claude']!.files['/live/moving.jsonl']).toEqual(fresh)
|
||||
expect(await copiesOf('/live/moving.jsonl')).toEqual(['2026-06-02'])
|
||||
expect(existsSync(v9)).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
|
|
@ -765,11 +768,11 @@ describe('carried months under a concurrent writer', () => {
|
|||
|
||||
clearLoadCacheMemo()
|
||||
const final = await loadCache()
|
||||
// March survived under A's name, with both orphans; June has both files.
|
||||
// March survived with both orphans; June has both files.
|
||||
expect(Object.keys(final.providers['claude']!.files).sort())
|
||||
.toEqual(['/gone/mar.jsonl', '/gone/mar2.jsonl', '/live/jun.jsonl', '/live/jun2.jsonl'])
|
||||
for (const ref of Object.values((await envelope()).providers['claude']!.shards)) {
|
||||
expect(existsSync(join(sessionCacheDir(), ref.name))).toBe(true)
|
||||
for (const name of Object.values(await pieces('claude'))) {
|
||||
expect(existsSync(join(sessionCacheDir(), name))).toBe(true)
|
||||
}
|
||||
})
|
||||
|
||||
|
|
@ -809,8 +812,8 @@ describe('carried months under a concurrent writer', () => {
|
|||
const final = await loadCache()
|
||||
// The envelope is internally consistent and the pre-existing March session
|
||||
// survived; a read-modify-write loser is re-derived by the next parse.
|
||||
for (const ref of Object.values((await envelope()).providers['claude']!.shards)) {
|
||||
expect(existsSync(join(sessionCacheDir(), ref.name))).toBe(true)
|
||||
for (const name of Object.values(await pieces('claude'))) {
|
||||
expect(existsSync(join(sessionCacheDir(), name))).toBe(true)
|
||||
}
|
||||
expect(final.providers['claude']!.files['/live/mar.jsonl']).toBeDefined()
|
||||
expect(final.providers['claude']!.files['/live/jun.jsonl']).toBeDefined()
|
||||
|
|
@ -820,7 +823,7 @@ describe('carried months under a concurrent writer', () => {
|
|||
})
|
||||
|
||||
describe('retiring an orphaned prior layout', () => {
|
||||
it('sweeps a v8 directory and a v7 file left behind by an interrupted re-layout', async () => {
|
||||
it('sweeps v9 and v8 directories and a v7 file left behind by an interrupted re-layout', async () => {
|
||||
await saveCache({ version: CACHE_VERSION, complete: true, providers: {
|
||||
claude: { envFingerprint: 'fp', files: { '/a.jsonl': fileSpanning('2026-05-10T10:00:00Z') } },
|
||||
} })
|
||||
|
|
@ -830,17 +833,23 @@ describe('retiring an orphaned prior layout', () => {
|
|||
await writeFile(join(v8Dir, 'claude.abc.json'), '{}')
|
||||
const v7 = join(TMP_DIR, 'session-cache.v7.json')
|
||||
await writeFile(v7, '{}')
|
||||
const v9Dir = join(TMP_DIR, 'session-cache.v9')
|
||||
await mkdir(v9Dir, { recursive: true })
|
||||
await writeFile(join(v9Dir, 'envelope.json'), JSON.stringify({ version: 9, nonce: 'n', providers: {} }))
|
||||
|
||||
// Fresh: an in-flight write by an older binary must be left alone.
|
||||
await cleanupOrphanedTempFiles()
|
||||
expect(existsSync(v8Dir)).toBe(true)
|
||||
expect(existsSync(v9Dir)).toBe(true)
|
||||
expect(existsSync(v7)).toBe(true)
|
||||
|
||||
const old = new Date(Date.now() - 90 * 60 * 1000)
|
||||
await utimes(join(v8Dir, 'envelope.json'), old, old)
|
||||
await utimes(join(v9Dir, 'envelope.json'), old, old)
|
||||
await utimes(v7, old, old)
|
||||
await cleanupOrphanedTempFiles()
|
||||
expect(existsSync(v8Dir)).toBe(false)
|
||||
expect(existsSync(v9Dir)).toBe(false)
|
||||
expect(existsSync(v7)).toBe(false)
|
||||
})
|
||||
})
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue