Merge pull request #1017 from ozymandiashh/fix/sqlite-readonly-parent

fix(sqlite): survive a read-only database parent instead of reporting no sessions
This commit is contained in:
Resham Joshi 2026-08-18 11:48:51 -07:00 committed by GitHub
commit 2766224bf5
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 679 additions and 7 deletions

View file

@ -16,6 +16,7 @@
- **DeepSeek Harness (`dsh`) is now a supported provider.** Reads DeepSeek's open-source agent harness from `~/.dsh/sessions` (`DSH_HOME` relocates the root), both the default zstd logs and the uncompressed `session.jsonl` variant. A `.zstd` log is a concatenation of independent zstd frames, one per write batch, so it is decoded frame by frame behind a structural frame scan and a torn trailing frame from a crashed writer is ignored rather than failing the file (needs Node 22.15+ for `zlib` zstd; below that dsh is skipped with a notice instead of counted as $0). One call per `(turn, step)`, with the step's final `assistant/message` usage superseding the streamed `assistant/chunk` sample of the same call rather than adding to it, the model taken from the message that served the step, and reasoning tokens billed at the output rate. DSH records tokens but no cost, so calls are priced from the shared tables. The events a forked session replays from its parent are skipped, since codeburn already counts the parent's own log. The session format is pinned at version 0 upstream with no compatibility implied, so a log stamped with any other version is skipped with a notice instead of read under today's assumptions.
### Changed
- **SQLite providers now survive read-only database parents.** A read-only SQLite open is not read-only on disk: on a WAL database SQLite must create `<db>-shm` and `<db>-wal` in the database's own directory, so a source on read-only media, under restrictive permissions, or inside a Flatpak/snap confinement failed with `attempt to write a readonly database` (or `unable to open database file` when a `-wal` was present without its `-shm`), and both discovery sites swallowed it — the provider read as "not installed" rather than as an error. That covers cursor, cursor-agent, opencode, goose, warp, kilo-code, zerostack and the copilot agent-traces database. The direct open stays the fast path and is byte-identical when it succeeds. When it fails for want of sidecars: a database with no WAL frames to lose is opened in place with `immutable=1`, which costs nothing and cannot go stale; a database with a non-empty `-wal` is copied with its `-wal` into the CodeBurn cache and read there, so its un-checkpointed rows are never silently dropped. The copy costs one database's worth of disk and is taken once per change — it is keyed by the main-plus-WAL fingerprint, published under a fingerprint-stamped name so a refresh never overwrites a copy another process is reading, and superseded copies are evicted once a day has passed without a read, keeping at most one predecessor. If the cache itself cannot be written, the database is skipped with a notice naming it and the reason rather than in silence. The original provider database is never opened writable or modified.
- **Grok Build now reads the CLI's own completed-turn usage instead of estimating it.** Usage comes from the `turn_completed.usage` records Grok CLI already writes into `updates.jsonl` (`inputTokens`, `outputTokens`, `cachedReadTokens`, `cacheCreationTokens`, `reasoningTokens`), deduplicated by `prompt_id` and emitted as one session-level call from the top-level totals. The previous parser reconstructed an estimate from the running `_meta.totalTokens` context counter, so **existing Grok totals will change materially on upgrade** - on one real 568-session corpus cache-read went from 150K to 96.3M tokens, total tokens from 20.0M to 113.9M, and cost from $36.98 to $56.79. Cache read and cache creation are subsets of input and reasoning is a subset of output, so reasoning is clamped to the record's reported output and split back out to match this repo's exclusive-reasoning contract. `modelUsage` only selects a priced attribution id; multi-model rate attribution stays out of scope, so one session is priced at one model's rate. `costUsdTicks` is ignored because its scale is undocumented. Sessions with no usable record - older CLI versions - keep the old context-curve heuristic and stay flagged estimated. **In a session that has at least one `turn_completed` record, turns without one are not counted at all** (their tokens are dropped rather than estimated), and the session is marked estimated instead of claiming full provider coverage. Cached Grok sessions re-parse once. The daily cache re-derives once on first run after upgrade: this is a global re-derivation of every day and every provider, since the daily cache has no per-provider invalidation, but it reads the warm session cache rather than re-parsing transcripts, so it costs seconds (~3s on the corpus above), and the superseded cache file is retained on disk as the baseline for days no source can still re-derive. (#998)
- **Codex rollouts parse across worker threads too, and the workload gate now takes bytes or files.** Codex is the bigger half of a real cold parse — a 4 GB rollout corpus against 1.8 GB of Claude sessions — and it was still decoding one file at a time. A whole-file rollout decode now runs on the same pool, against an empty dedup set, and comes back with the calls, the dedup keys it claimed, and the codex-cache entry it would have written; the parent installs all three in the serial loop's order, so `codex-results.json` and every payload come out byte-identical to a serial run. Cross-file state stays where it was: a forked rollout replaying its parent's token_count history collides on the parent's keys and is re-parsed in-process, and no worker ever touches the cache module's per-directory state. Files the Codex cache can serve exactly or resume into from a byte offset never reach a worker — they read a few KB and the resume state belongs to the parent. The workload gate is now pending BYTES alone (200 MB), not file count: 250 pending files holding under a megabyte between them spawned threads that made the run ~5% slower, while a few hundred huge rollouts were being turned away. The count takes `max(pendingFiles / 50, pendingBytes / 200 MB)`, and the per-thread memory budget is derived per parse as `clamp(256 MB, 2 × average pending file + 128 MB, 1 GB)` rather than a flat 256 MB — a 260 MB rollout peaks near 430 MB in its worker and scales linearly with the pool, so the flat figure over-subscribed exactly the workload this adds. The decision is per provider, and at most one pool is alive at a time.
- **A large cold Claude parse now runs across worker threads.** Reading, decoding and line-parsing a session JSONL is per-file work that never touches anything shared, so it moves onto `worker_threads`; each worker ships its parsed turns back as a JSON string and the parent installs them in the exact order the serial loop would. Everything with cross-file state — the streaming-message dedup, canonical project paths, spawn links, PR correlation, progress saves — stays on the main thread, and a file whose message ids were already claimed by an earlier file (or whose worker failed) is simply re-parsed in-process, so the session cache and every payload are identical either way. On a 6 GB corpus a cold `status` drops from 27.5s to 14.8s with peak RSS up 2.27 GB → 2.52 GB. Threads only engage for a genuinely large cold parse: never with under 200 MB behind the pending whole-file re-parses, 2 or fewer cores, or under 4 GB of available memory — so warm and incremental runs are untouched and spawn nothing. Otherwise the count is `min(cores - 1, min(0.25 × available, 2 GB) / 256 MB, pendingFiles / 50)`, where available is `process.availableMemory()` (cgroup-aware in containers) rather than free memory, which on macOS reports free pages and would switch the feature on and off between runs. `CODEBURN_PARSE_WORKERS=0` forces the serial parse and `CODEBURN_PARSE_WORKERS=N` forces N (capped at the core count), both bypassing every gate; `CODEBURN_VERBOSE=1` prints the resolved count and why.

View file

@ -5,7 +5,16 @@ import { homedir } from 'os'
import { calculateCost } from '../models.js'
import { extractBashCommands } from '../bash-utils.js'
import { readCachedResults, writeCachedResults } from '../cursor-cache.js'
import { isSqliteAvailable, isSqliteBusyError, getSqliteLoadError, openDatabase, blobToText, type SqliteDatabase } from '../sqlite.js'
import {
isSqliteAvailable,
isSqliteBusyError,
getSqliteLoadError,
openDatabase,
blobToText,
isSqliteReadonlyError,
warnSqliteReadonlyOnce,
type SqliteDatabase,
} from '../sqlite.js'
import { estimateTokensFromChars } from '../token-estimate.js'
import type { DateRange } from '../types.js'
import type { ProbeRoot, Provider, SessionSource, SessionParser, ParsedProviderCall } from './types.js'
@ -188,7 +197,8 @@ function loadWorkspaceMap(workspaceStorageDir: string): WorkspaceMapping {
let db: SqliteDatabase
try {
db = openDatabase(wsDbPath)
} catch {
} catch (err) {
if (isSqliteReadonlyError(err)) warnSqliteReadonlyOnce(wsDbPath)
continue
}
try {

View file

@ -2,7 +2,16 @@ import { readdir } from 'fs/promises'
import { join } from 'path'
import { calculateCost } from '../models.js'
import { isSqliteAvailable, getSqliteLoadError, openDatabase, blobToText, isSqliteBusyError, type SqliteDatabase } from '../sqlite.js'
import {
isSqliteAvailable,
getSqliteLoadError,
openDatabase,
blobToText,
isSqliteBusyError,
isSqliteReadonlyError,
warnSqliteReadonlyOnce,
type SqliteDatabase,
} from '../sqlite.js'
import { buildAssistantCall, parseTimestamp, sanitize, type MessageData, type PartData } from './session-message.js'
import type {
SessionSource,
@ -310,7 +319,8 @@ export async function discoverSqliteSessions(
let db: SqliteDatabase
try {
db = openDatabase(dbPath)
} catch {
} catch (err) {
if (isSqliteReadonlyError(err)) warnSqliteReadonlyOnce(dbPath)
continue
}

View file

@ -1,4 +1,10 @@
import { createRequire } from 'node:module'
import { copyFileSync, existsSync, mkdirSync, readdirSync, renameSync, statSync, unlinkSync, utimesSync } from 'node:fs'
import { createHash, randomBytes } from 'node:crypto'
import { join } from 'node:path'
import { pathToFileURL } from 'node:url'
import { getCodeburnCacheDir } from './cache-dir.js'
/// Thin SQLite read-only wrapper over Node's built-in `node:sqlite` module (stable in
/// Node 24, experimental in Node 22 / 23). Replaces the earlier `better-sqlite3` binding
@ -14,12 +20,14 @@ export type SqliteDatabase = {
close(): void
}
type DatabaseSyncCtor = new (path: string, options?: { readOnly?: boolean }) => {
type DatabaseSyncInstance = {
prepare(sql: string): { all(...params: unknown[]): Row[] }
exec?(sql: string): void
close(): void
}
type DatabaseSyncCtor = new (path: string, options?: { readOnly?: boolean }) => DatabaseSyncInstance
let DatabaseSync: DatabaseSyncCtor | null = null
let loadAttempted = false
let loadError: string | null = null
@ -116,12 +124,304 @@ export function isSqliteBusyError(err: unknown): boolean {
)
}
/// SQLite reports SQLITE_READONLY_DIRECTORY as ERR_SQLITE_ERROR with an extended
/// result code on the Node 22 builds CodeBurn supports. Keep the base-code check
/// so this also covers SQLITE_READONLY and its other extended variants, while
/// leaving ENOENT/SQLITE_CANTOPEN distinguishable to callers.
export function isSqliteReadonlyError(err: unknown): boolean {
const e = err as { code?: unknown; errcode?: unknown; errstr?: unknown; message?: unknown } | null
const code = typeof e?.code === 'string' ? e.code : ''
const errcode = typeof e?.errcode === 'number' ? e.errcode : null
const message = [
typeof e?.message === 'string' ? e.message : '',
typeof e?.errstr === 'string' ? e.errstr : '',
].join(' ')
return (
(errcode !== null && (errcode & 0xff) === 8) ||
/SQLITE_READONLY|attempt to write a readonly database|readonly database|read-only database/i.test(`${code} ${message}`)
)
}
/// A read-only parent reports SQLITE_READONLY_DIRECTORY when it must create the
/// sidecars from scratch, but SQLITE_CANTOPEN when a `-wal` is present and the
/// `-shm` it needs to index it is not. openReadonlyCache re-throws the original
/// error when the database itself is missing, which is the other CANTOPEN.
function isSqliteSidecarError(err: unknown): boolean {
if (isSqliteReadonlyError(err)) return true
const errcode = (err as { errcode?: unknown } | null)?.errcode
return typeof errcode === 'number' && (errcode & 0xff) === 14
}
let uriFilenamesSupported: boolean | null = null
/// node:sqlite only enables SQLITE_OPEN_URI from Node 22.15 on (measured: 22.13
/// and 22.14 fail, 22.15 and later work). Below that a `file:...` location is
/// taken literally and fails as CANTOPEN, so the immutable open is not attempted
/// there. The probe is an in-memory URI rather than a version comparison: it
/// answers the question directly and touches no filesystem either way.
export function sqliteSupportsUriFilenames(): boolean {
if (uriFilenamesSupported !== null) return uriFilenamesSupported
uriFilenamesSupported = false
const Driver = loadDriver() ? DatabaseSync : null
if (Driver !== null) {
try {
new Driver('file:codeburn-uri-probe?mode=memory', { readOnly: true }).close()
uriFilenamesSupported = true
} catch {
// An older build: locations are plain paths, and the copy fallback covers
// exactly the case the immutable open would have.
}
}
return uriFilenamesSupported
}
type DatabaseFingerprint = {
dev: number
ino: number
mtimeMs: number
sizeBytes: number
walBytes: number
}
/// A superseded copy is dropped once it has gone this long without being used.
/// The delay is what keeps a concurrent reader of the previous copy from having
/// its file yanked out from under it.
const CACHE_ENTRY_MAX_AGE_MS = 24 * 60 * 60 * 1000
const warnedDatabases = new Set<string>()
/// One notice per source path per run: a provider may discover many sessions
/// from the same database, and the first notice already says what happened.
function warnSqliteOnce(path: string, message: string): void {
if (warnedDatabases.has(path)) return
warnedDatabases.add(path)
process.stderr.write(message)
}
/// A read-only SQLite connection can still need sidecar files.
export function warnSqliteReadonlyOnce(path: string): void {
warnSqliteOnce(
path,
`codeburn: SQLite database ${path} is in a read-only directory and needs sidecar files; using a cache copy when necessary. ` +
'The original database is not modified.\n',
)
}
function errorCode(err: unknown): string | undefined {
if (typeof err !== 'object' || err === null || !('code' in err)) return undefined
const code = err.code
return typeof code === 'string' ? code : undefined
}
function describeError(err: unknown): string {
return err instanceof Error ? err.message : String(err)
}
/// This deliberately mirrors fingerprintSqliteFile/fingerprintFile in
/// session-cache.ts. openDatabase is synchronous, so the fallback uses the
/// synchronous fs APIs only after the direct open has already failed; the
/// ordinary successful open remains probe-free.
function fingerprintDatabase(path: string): DatabaseFingerprint {
const main = statSync(path)
let wal: ReturnType<typeof statSync> | null = null
try {
wal = statSync(path + '-wal')
} catch (err) {
if (errorCode(err) !== 'ENOENT') throw err
}
return {
dev: main.dev,
ino: main.ino,
mtimeMs: wal ? Math.max(main.mtimeMs, wal.mtimeMs) : main.mtimeMs,
sizeBytes: main.size + (wal?.size ?? 0),
walBytes: wal?.size ?? 0,
}
}
function sameFingerprint(a: DatabaseFingerprint, b: DatabaseFingerprint): boolean {
return (
a.dev === b.dev &&
a.ino === b.ino &&
a.mtimeMs === b.mtimeMs &&
a.sizeBytes === b.sizeBytes &&
a.walBytes === b.walBytes
)
}
function unlinkQuietly(path: string): void {
try {
unlinkSync(path)
} catch {
// Already gone, or still held open by another CodeBurn on Windows. Either
// way the next run's eviction pass gets another chance at it.
}
}
function copyOptionalFile(sourcePath: string, destinationPath: string): boolean {
try {
copyFileSync(sourcePath, destinationPath)
return true
} catch (err) {
if (errorCode(err) === 'ENOENT') return false
throw err
}
}
function sourceKeyOf(sourcePath: string): string {
return createHash('sha256').update(sourcePath, 'utf8').digest('hex').slice(0, 32)
}
/// The copy is named after the source it came from AND the fingerprint it was
/// taken at, so a refresh publishes a new file rather than overwriting one that
/// another process may still have open.
function cacheEntryName(sourceKey: string, fingerprint: DatabaseFingerprint): string {
const parts = `${fingerprint.dev}:${fingerprint.ino}:${fingerprint.mtimeMs}:${fingerprint.sizeBytes}:${fingerprint.walBytes}`
return `${sourceKey}.${createHash('sha256').update(parts).digest('hex').slice(0, 16)}.db`
}
function dropCopy(cacheDir: string, name: string): void {
unlinkQuietly(join(cacheDir, name))
unlinkQuietly(join(cacheDir, name + '-wal'))
unlinkQuietly(join(cacheDir, name + '-shm'))
}
/// Superseded copies are cleaned up here rather than by overwriting them: keep
/// the one in use plus at most one predecessor, and drop anything untouched for
/// a day, which is also what a source path that no longer exists looks like.
/// Reuse touches the copy, so its mtime is last-use rather than copy time.
function evictSupersededCopies(cacheDir: string, sourceKey: string, keepName: string): void {
let names: string[]
try {
names = readdirSync(cacheDir)
} catch {
return
}
const now = Date.now()
const superseded: { name: string, mtimeMs: number }[] = []
for (const name of names) {
if (!name.endsWith('.db') || name === keepName) continue
let mtimeMs: number
try {
mtimeMs = statSync(join(cacheDir, name)).mtimeMs
} catch {
continue
}
if (name.startsWith(`${sourceKey}.`)) superseded.push({ name, mtimeMs })
else if (now - mtimeMs > CACHE_ENTRY_MAX_AGE_MS) dropCopy(cacheDir, name)
}
superseded.sort((a, b) => b.mtimeMs - a.mtimeMs)
for (const [index, entry] of superseded.entries()) {
if (index > 0 || now - entry.mtimeMs > CACHE_ENTRY_MAX_AGE_MS) dropCopy(cacheDir, entry.name)
}
}
/// A concurrent CodeBurn may have published the same copy first. The name is
/// the fingerprint, so the content is identical by construction and losing that
/// race is not an error.
function publish(tempPath: string, finalPath: string): void {
try {
renameSync(tempPath, finalPath)
} catch (err) {
if (!existsSync(finalPath)) throw err
}
}
function readOnlyCachePath(sourcePath: string, fingerprint: DatabaseFingerprint): string {
const cacheDir = join(getCodeburnCacheDir(), 'sqlite-ro')
mkdirSync(cacheDir, { recursive: true, mode: 0o700 })
const sourceKey = sourceKeyOf(sourcePath)
const name = cacheEntryName(sourceKey, fingerprint)
const cachePath = join(cacheDir, name)
if (existsSync(cachePath)) {
const now = new Date()
try {
utimesSync(cachePath, now, now)
} catch {
// mtime is only the eviction clock; a copy we cannot touch still reads.
}
evictSupersededCopies(cacheDir, sourceKey, name)
return cachePath
}
const tempBase = `${cachePath}.tmp-${process.pid}-${randomBytes(8).toString('hex')}`
const tempWal = tempBase + '-wal'
try {
copyFileSync(sourcePath, tempBase)
const copiedWal = copyOptionalFile(sourcePath + '-wal', tempWal)
// Do not publish a cache made from a moving database. A live WAL writer will
// normally make the direct open succeed once its sidecars exist; this check
// covers the narrow race where the source changes during the copy fallback.
if (!sameFingerprint(fingerprintDatabase(sourcePath), fingerprint)) {
throw new Error('SQLite database changed while preparing its read-only cache copy')
}
// The -wal goes first: a reader that can see the database must never find it
// without the sidecar holding its most recent rows.
if (copiedWal) publish(tempWal, cachePath + '-wal')
publish(tempBase, cachePath)
evictSupersededCopies(cacheDir, sourceKey, name)
return cachePath
} finally {
unlinkQuietly(tempBase)
unlinkQuietly(tempWal)
}
}
function openReadonlyCache(path: string, originalError: unknown): DatabaseSyncInstance {
const Driver = DatabaseSync
if (Driver === null) throw new Error(getSqliteLoadError())
let fingerprint: DatabaseFingerprint
try {
fingerprint = fingerprintDatabase(path)
} catch {
// Preserve the original SQLite error when the source disappeared or became
// inaccessible between the failed query and the fallback probe.
throw originalError
}
// An absent or empty -wal holds no frames, so there is nothing to go stale and
// nothing worth copying: immutable lets SQLite skip the -shm it cannot create
// and read the source in place.
if (fingerprint.walBytes === 0 && sqliteSupportsUriFilenames()) {
try {
return new Driver(`${pathToFileURL(path).href}?immutable=1`, { readOnly: true })
} catch {
// Understood but refused: the copy covers it.
}
}
let cachedPath: string
try {
cachedPath = readOnlyCachePath(path, fingerprint)
} catch (err) {
warnSqliteOnce(
path,
`codeburn: SQLite database ${path} is in a read-only directory and its cache copy could not be written ` +
`(${describeError(err)}); skipping this database.\n`,
)
throw originalError
}
return new Driver(cachedPath, { readOnly: true })
}
export function openDatabase(path: string): SqliteDatabase {
if (!loadDriver() || DatabaseSync === null) {
throw new Error(getSqliteLoadError())
}
const db = new DatabaseSync(path, { readOnly: true })
let db: DatabaseSyncInstance
let fallbackUsed = false
try {
db = new DatabaseSync(path, { readOnly: true })
} catch (err) {
if (!isSqliteSidecarError(err)) throw err
fallbackUsed = true
db = openReadonlyCache(path, err)
warnSqliteReadonlyOnce(path)
}
try {
db.exec?.('PRAGMA busy_timeout = 1000')
} catch {
@ -130,7 +430,26 @@ export function openDatabase(path: string): SqliteDatabase {
return {
query<T extends Row = Row>(sql: string, params: unknown[] = []): T[] {
return db.prepare(sql).all(...params) as T[]
try {
return db.prepare(sql).all(...params) as T[]
} catch (err) {
if (!isSqliteSidecarError(err)) throw err
if (fallbackUsed) throw err
fallbackUsed = true
try {
db.close()
} catch {
// The failed connection may already have been closed by node:sqlite.
}
db = openReadonlyCache(path, err)
warnSqliteReadonlyOnce(path)
try {
db.exec?.('PRAGMA busy_timeout = 1000')
} catch {
// Best effort, matching the direct-open path above.
}
return db.prepare(sql).all(...params) as T[]
}
},
close() {
db.close()

View file

@ -0,0 +1,332 @@
import { chmodSync, copyFileSync, existsSync, mkdirSync, readdirSync, statSync, utimesSync, writeFileSync } from 'node:fs'
import { mkdtemp, rm } from 'node:fs/promises'
import { createRequire } from 'node:module'
import { join } from 'node:path'
import { tmpdir } from 'node:os'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import {
isSqliteReadonlyError,
openDatabase,
sqliteSupportsUriFilenames,
} from '../src/sqlite.js'
import {
discoverSqliteSessions,
type SqliteProviderConfig,
} from '../src/providers/sqlite-session-parser.js'
const requireForTest = createRequire(import.meta.url)
type NativeDatabase = {
exec(sql: string): void
prepare(sql: string): { run(...params: unknown[]): void; all(...params: unknown[]): unknown[] }
close(): void
}
type NativeDatabaseCtor = new (path: string) => NativeDatabase
const { DatabaseSync: NativeDatabase } = requireForTest('node:sqlite') as {
DatabaseSync: NativeDatabaseCtor
}
let sourceRoot: string
let cacheRoot: string
let previousCacheDir: string | undefined
const openWriters: NativeDatabase[] = []
beforeEach(async () => {
sourceRoot = await mkdtemp(join(tmpdir(), 'codeburn-sqlite-source-'))
cacheRoot = await mkdtemp(join(tmpdir(), 'codeburn-sqlite-cache-'))
previousCacheDir = process.env['CODEBURN_CACHE_DIR']
process.env['CODEBURN_CACHE_DIR'] = cacheRoot
})
afterEach(async () => {
chmodSync(sourceRoot, 0o755)
chmodSync(cacheRoot, 0o755)
for (const writer of openWriters.splice(0)) writer.close()
await rm(sourceRoot, { recursive: true, force: true })
await rm(cacheRoot, { recursive: true, force: true })
if (previousCacheDir === undefined) delete process.env['CODEBURN_CACHE_DIR']
else process.env['CODEBURN_CACHE_DIR'] = previousCacheDir
})
function createClosedWalDatabase(dbPath: string): void {
const db = new NativeDatabase(dbPath)
db.exec('PRAGMA journal_mode=WAL')
db.exec('CREATE TABLE values_table (c INTEGER)')
db.prepare('INSERT INTO values_table (c) VALUES (?)').run(1)
db.close()
}
function createOpenWalDatabase(dbPath: string): NativeDatabase {
const db = new NativeDatabase(dbPath)
db.exec('PRAGMA journal_mode=WAL')
db.exec('CREATE TABLE values_table (c INTEGER)')
db.prepare('INSERT INTO values_table (c) VALUES (?)').run(1)
expect(existsSync(dbPath + '-wal')).toBe(true)
expect(existsSync(dbPath + '-shm')).toBe(true)
openWriters.push(db)
return db
}
/// A database plus a non-empty -wal and no -shm: what a snapshot, an rsync or an
/// unclean unmount of a live source leaves behind. The second row exists only in
/// the -wal, so dropping it would be silent data loss rather than an error.
function writeUncheckpointedWalDatabase(dbPath: string): void {
const originDir = join(sourceRoot, `origin-${readdirSync(sourceRoot).length}`)
mkdirSync(originDir)
const originPath = join(originDir, 'state.vscdb')
const writer = new NativeDatabase(originPath)
writer.exec('PRAGMA journal_mode=WAL')
writer.exec('CREATE TABLE values_table (c INTEGER)')
writer.prepare('INSERT INTO values_table (c) VALUES (?)').run(1)
writer.exec('PRAGMA wal_checkpoint(TRUNCATE)')
writer.exec('PRAGMA wal_autocheckpoint=0')
writer.prepare('INSERT INTO values_table (c) VALUES (?)').run(2)
copyFileSync(originPath, dbPath)
copyFileSync(originPath + '-wal', dbPath + '-wal')
writer.close()
expect(statSync(dbPath + '-wal').size).toBeGreaterThan(0)
expect(existsSync(dbPath + '-shm')).toBe(false)
}
function createDiscoveryDatabase(dbPath: string): void {
const db = new NativeDatabase(dbPath)
db.exec('PRAGMA journal_mode=WAL')
db.exec(`
CREATE TABLE session (
id TEXT PRIMARY KEY,
directory TEXT,
title TEXT,
time_created INTEGER,
parent_id TEXT,
time_archived INTEGER
)
`)
db.exec('CREATE TABLE message (id TEXT PRIMARY KEY, session_id TEXT, time_created INTEGER, data BLOB)')
db.exec('CREATE TABLE part (id INTEGER PRIMARY KEY, message_id TEXT, session_id TEXT, data BLOB)')
db.prepare(
'INSERT INTO session (id, directory, title, time_created, parent_id, time_archived) VALUES (?, ?, ?, ?, ?, ?)',
).run('session-1', '/tmp/project', 'Read-only fixture', Date.now(), null, null)
db.close()
}
function makeSourceParentReadOnly(skip: (reason?: string) => void): boolean {
chmodSync(sourceRoot, 0o555)
const mode = statSync(sourceRoot).mode & 0o777
if ((mode & 0o222) !== 0) {
skip(`SKIP: chmod 0555 did not make the fixture parent non-writable (mode ${mode.toString(8)})`)
return false
}
return true
}
function makeSourceParentWritable(): void {
chmodSync(sourceRoot, 0o755)
}
function cachedDatabaseFiles(): string[] {
try {
return readdirSync(join(cacheRoot, 'sqlite-ro')).filter(name => name.endsWith('.db'))
} catch {
return []
}
}
function readValue(dbPath: string): number {
const db = openDatabase(dbPath)
try {
const rows = db.query<{ c: number }>('SELECT c FROM values_table')
return rows[0]?.c ?? -1
} finally {
db.close()
}
}
describe('SQLite read-only parent fallback', () => {
it('keeps the existing writable-parent open behaviour', () => {
const dbPath = join(sourceRoot, 'state.vscdb')
createClosedWalDatabase(dbPath)
expect(existsSync(dbPath + '-wal')).toBe(false)
expect(existsSync(dbPath + '-shm')).toBe(false)
expect(readValue(dbPath)).toBe(1)
expect(existsSync(dbPath + '-wal')).toBe(true)
expect(existsSync(dbPath + '-shm')).toBe(true)
expect(cachedDatabaseFiles()).toEqual([])
})
it('reads a read-only parent with no -wal, in place where it can and by copy where it cannot', ({ skip }) => {
const dbPath = join(sourceRoot, 'state.vscdb')
createClosedWalDatabase(dbPath)
expect(existsSync(dbPath + '-wal')).toBe(false)
expect(existsSync(dbPath + '-shm')).toBe(false)
if (!makeSourceParentReadOnly(skip)) return
expect(readValue(dbPath)).toBe(1)
expect(existsSync(dbPath + '-wal')).toBe(false)
expect(existsSync(dbPath + '-shm')).toBe(false)
// No WAL frames exist, so there is nothing to go stale and nothing worth
// copying: immutable reads the source in place. node:sqlite only honours
// URI filenames on newer builds, and on the 22.13 floor the copy stands in.
expect(cachedDatabaseFiles()).toHaveLength(sqliteSupportsUriFilenames() ? 0 : 1)
})
it('opens directly when a read-only parent already has WAL sidecars', ({ skip }) => {
const dbPath = join(sourceRoot, 'state.vscdb')
createOpenWalDatabase(dbPath)
if (!makeSourceParentReadOnly(skip)) return
expect(readValue(dbPath)).toBe(1)
expect(cachedDatabaseFiles()).toEqual([])
})
it('reads un-checkpointed WAL rows when the parent is read-only and the -shm is absent', ({ skip }) => {
// SQLite reports a -wal without its -shm as SQLITE_CANTOPEN, not
// SQLITE_READONLY, and the un-checkpointed row lives only in the -wal.
const dbPath = join(sourceRoot, 'state.vscdb')
writeUncheckpointedWalDatabase(dbPath)
if (!makeSourceParentReadOnly(skip)) return
const db = openDatabase(dbPath)
try {
expect(db.query<{ c: number }>('SELECT c FROM values_table ORDER BY c')).toEqual([{ c: 1 }, { c: 2 }])
} finally {
db.close()
}
expect(existsSync(dbPath + '-shm')).toBe(false)
expect(cachedDatabaseFiles()).toHaveLength(1)
})
it('reuses an unchanged fallback copy instead of copying the database again', ({ skip }) => {
const dbPath = join(sourceRoot, 'state.vscdb')
writeUncheckpointedWalDatabase(dbPath)
if (!makeSourceParentReadOnly(skip)) return
expect(readValue(dbPath)).toBe(1)
const first = cachedDatabaseFiles()
expect(first).toHaveLength(1)
const cachedPath = join(cacheRoot, 'sqlite-ro', first[0]!)
const firstIno = statSync(cachedPath).ino
expect(readValue(dbPath)).toBe(1)
expect(cachedDatabaseFiles()).toEqual(first)
expect(statSync(cachedPath).ino).toBe(firstIno)
})
it('publishes a refreshed copy beside the old one and keeps at most one predecessor', ({ skip }) => {
const dbPath = join(sourceRoot, 'state.vscdb')
writeUncheckpointedWalDatabase(dbPath)
if (!makeSourceParentReadOnly(skip)) return
expect(readValue(dbPath)).toBe(1)
const [first] = cachedDatabaseFiles()
const firstIno = statSync(join(cacheRoot, 'sqlite-ro', first!)).ino
// A changed source must not overwrite the copy a concurrent reader may still
// have open: Windows cannot unlink it, and the name is the fingerprint.
makeSourceParentWritable()
writeUncheckpointedWalDatabase(dbPath)
if (!makeSourceParentReadOnly(skip)) return
expect(readValue(dbPath)).toBe(1)
const second = cachedDatabaseFiles()
expect(second).toHaveLength(2)
expect(second).toContain(first)
expect(statSync(join(cacheRoot, 'sqlite-ro', first!)).ino).toBe(firstIno)
makeSourceParentWritable()
writeUncheckpointedWalDatabase(dbPath)
if (!makeSourceParentReadOnly(skip)) return
expect(readValue(dbPath)).toBe(1)
const third = cachedDatabaseFiles()
expect(third).toHaveLength(2)
expect(third).not.toContain(first)
})
it('evicts a copy left untouched for a day, including one whose source is gone', ({ skip }) => {
const dbPath = join(sourceRoot, 'state.vscdb')
writeUncheckpointedWalDatabase(dbPath)
if (!makeSourceParentReadOnly(skip)) return
expect(readValue(dbPath)).toBe(1)
const cacheDir = join(cacheRoot, 'sqlite-ro')
const predecessor = join(cacheDir, cachedDatabaseFiles()[0]!)
// A copy of a database that no longer exists is simply one nothing touches.
const orphan = join(cacheDir, `${'0'.repeat(32)}.deadbeefdeadbeef.db`)
writeFileSync(orphan, 'orphan')
const aDayAndAnHourAgo = new Date(Date.now() - 25 * 60 * 60 * 1000)
utimesSync(orphan, aDayAndAnHourAgo, aDayAndAnHourAgo)
makeSourceParentWritable()
writeUncheckpointedWalDatabase(dbPath)
if (!makeSourceParentReadOnly(skip)) return
expect(readValue(dbPath)).toBe(1)
expect(existsSync(orphan)).toBe(false)
expect(existsSync(predecessor)).toBe(true)
// A day without a read and the superseded copy goes too.
utimesSync(predecessor, aDayAndAnHourAgo, aDayAndAnHourAgo)
expect(readValue(dbPath)).toBe(1)
expect(existsSync(predecessor)).toBe(false)
expect(cachedDatabaseFiles()).toHaveLength(1)
})
it('says so instead of going quiet when the cache copy cannot be written', ({ skip }) => {
const dbPath = join(sourceRoot, 'state.vscdb')
writeUncheckpointedWalDatabase(dbPath)
if (!makeSourceParentReadOnly(skip)) return
chmodSync(cacheRoot, 0o555)
const stderr = vi.spyOn(process.stderr, 'write').mockReturnValue(true)
try {
expect(() => readValue(dbPath)).toThrow()
const notices = stderr.mock.calls.filter(([chunk]) => String(chunk).includes('cache copy could not be written'))
expect(notices).toHaveLength(1)
expect(String(notices[0]?.[0])).toContain(dbPath)
} finally {
stderr.mockRestore()
chmodSync(cacheRoot, 0o755)
}
})
it('keeps a genuinely missing database distinguishable from SQLITE_READONLY', () => {
let thrown: unknown
try {
openDatabase(join(sourceRoot, 'missing.vscdb'))
} catch (err) {
thrown = err
}
expect(thrown).toBeDefined()
expect(isSqliteReadonlyError(thrown)).toBe(false)
expect(thrown).toMatchObject({ errcode: 14, message: 'unable to open database file' })
})
it('surfaces one read-only notice in SQLite discovery and still finds the session', async ({ skip }) => {
const dbPath = join(sourceRoot, 'state.db')
createDiscoveryDatabase(dbPath)
if (!makeSourceParentReadOnly(skip)) return
const config: SqliteProviderConfig = {
providerName: 'opencode',
displayName: 'OpenCode',
dbDir: sourceRoot,
dbFilePrefix: 'state',
}
const stderr = vi.spyOn(process.stderr, 'write').mockReturnValue(true)
try {
const sessions = await discoverSqliteSessions(config)
expect(sessions).toHaveLength(1)
expect(sessions[0]?.path).toBe(`${dbPath}:session-1`)
expect(stderr.mock.calls.filter(([chunk]) => String(chunk).includes('read-only directory'))).toHaveLength(1)
await discoverSqliteSessions(config)
expect(stderr.mock.calls.filter(([chunk]) => String(chunk).includes('read-only directory'))).toHaveLength(1)
} finally {
stderr.mockRestore()
}
})
})