mirror of
https://github.com/AgentSeal/codeburn.git
synced 2026-08-21 14:34:32 +00:00
Merge pull request #1017 from ozymandiashh/fix/sqlite-readonly-parent
fix(sqlite): survive a read-only database parent instead of reporting no sessions
This commit is contained in:
commit
2766224bf5
5 changed files with 679 additions and 7 deletions
|
|
@ -16,6 +16,7 @@
|
|||
- **DeepSeek Harness (`dsh`) is now a supported provider.** Reads DeepSeek's open-source agent harness from `~/.dsh/sessions` (`DSH_HOME` relocates the root), both the default zstd logs and the uncompressed `session.jsonl` variant. A `.zstd` log is a concatenation of independent zstd frames, one per write batch, so it is decoded frame by frame behind a structural frame scan and a torn trailing frame from a crashed writer is ignored rather than failing the file (needs Node 22.15+ for `zlib` zstd; below that dsh is skipped with a notice instead of counted as $0). One call per `(turn, step)`, with the step's final `assistant/message` usage superseding the streamed `assistant/chunk` sample of the same call rather than adding to it, the model taken from the message that served the step, and reasoning tokens billed at the output rate. DSH records tokens but no cost, so calls are priced from the shared tables. The events a forked session replays from its parent are skipped, since codeburn already counts the parent's own log. The session format is pinned at version 0 upstream with no compatibility implied, so a log stamped with any other version is skipped with a notice instead of read under today's assumptions.
|
||||
|
||||
### Changed
|
||||
- **SQLite providers now survive read-only database parents.** A read-only SQLite open is not read-only on disk: on a WAL database SQLite must create `<db>-shm` and `<db>-wal` in the database's own directory, so a source on read-only media, under restrictive permissions, or inside a Flatpak/snap confinement failed with `attempt to write a readonly database` (or `unable to open database file` when a `-wal` was present without its `-shm`), and both discovery sites swallowed it — the provider read as "not installed" rather than as an error. That covers cursor, cursor-agent, opencode, goose, warp, kilo-code, zerostack and the copilot agent-traces database. The direct open stays the fast path and is byte-identical when it succeeds. When it fails for want of sidecars: a database with no WAL frames to lose is opened in place with `immutable=1`, which costs nothing and cannot go stale; a database with a non-empty `-wal` is copied with its `-wal` into the CodeBurn cache and read there, so its un-checkpointed rows are never silently dropped. The copy costs one database's worth of disk and is taken once per change — it is keyed by the main-plus-WAL fingerprint, published under a fingerprint-stamped name so a refresh never overwrites a copy another process is reading, and superseded copies are evicted once a day has passed without a read, keeping at most one predecessor. If the cache itself cannot be written, the database is skipped with a notice naming it and the reason rather than in silence. The original provider database is never opened writable or modified.
|
||||
- **Grok Build now reads the CLI's own completed-turn usage instead of estimating it.** Usage comes from the `turn_completed.usage` records Grok CLI already writes into `updates.jsonl` (`inputTokens`, `outputTokens`, `cachedReadTokens`, `cacheCreationTokens`, `reasoningTokens`), deduplicated by `prompt_id` and emitted as one session-level call from the top-level totals. The previous parser reconstructed an estimate from the running `_meta.totalTokens` context counter, so **existing Grok totals will change materially on upgrade** - on one real 568-session corpus cache-read went from 150K to 96.3M tokens, total tokens from 20.0M to 113.9M, and cost from $36.98 to $56.79. Cache read and cache creation are subsets of input and reasoning is a subset of output, so reasoning is clamped to the record's reported output and split back out to match this repo's exclusive-reasoning contract. `modelUsage` only selects a priced attribution id; multi-model rate attribution stays out of scope, so one session is priced at one model's rate. `costUsdTicks` is ignored because its scale is undocumented. Sessions with no usable record - older CLI versions - keep the old context-curve heuristic and stay flagged estimated. **In a session that has at least one `turn_completed` record, turns without one are not counted at all** (their tokens are dropped rather than estimated), and the session is marked estimated instead of claiming full provider coverage. Cached Grok sessions re-parse once. The daily cache re-derives once on first run after upgrade: this is a global re-derivation of every day and every provider, since the daily cache has no per-provider invalidation, but it reads the warm session cache rather than re-parsing transcripts, so it costs seconds (~3s on the corpus above), and the superseded cache file is retained on disk as the baseline for days no source can still re-derive. (#998)
|
||||
- **Codex rollouts parse across worker threads too, and the workload gate now takes bytes or files.** Codex is the bigger half of a real cold parse — a 4 GB rollout corpus against 1.8 GB of Claude sessions — and it was still decoding one file at a time. A whole-file rollout decode now runs on the same pool, against an empty dedup set, and comes back with the calls, the dedup keys it claimed, and the codex-cache entry it would have written; the parent installs all three in the serial loop's order, so `codex-results.json` and every payload come out byte-identical to a serial run. Cross-file state stays where it was: a forked rollout replaying its parent's token_count history collides on the parent's keys and is re-parsed in-process, and no worker ever touches the cache module's per-directory state. Files the Codex cache can serve exactly or resume into from a byte offset never reach a worker — they read a few KB and the resume state belongs to the parent. The workload gate is now pending BYTES alone (200 MB), not file count: 250 pending files holding under a megabyte between them spawned threads that made the run ~5% slower, while a few hundred huge rollouts were being turned away. The count takes `max(pendingFiles / 50, pendingBytes / 200 MB)`, and the per-thread memory budget is derived per parse as `clamp(256 MB, 2 × average pending file + 128 MB, 1 GB)` rather than a flat 256 MB — a 260 MB rollout peaks near 430 MB in its worker and scales linearly with the pool, so the flat figure over-subscribed exactly the workload this adds. The decision is per provider, and at most one pool is alive at a time.
|
||||
- **A large cold Claude parse now runs across worker threads.** Reading, decoding and line-parsing a session JSONL is per-file work that never touches anything shared, so it moves onto `worker_threads`; each worker ships its parsed turns back as a JSON string and the parent installs them in the exact order the serial loop would. Everything with cross-file state — the streaming-message dedup, canonical project paths, spawn links, PR correlation, progress saves — stays on the main thread, and a file whose message ids were already claimed by an earlier file (or whose worker failed) is simply re-parsed in-process, so the session cache and every payload are identical either way. On a 6 GB corpus a cold `status` drops from 27.5s to 14.8s with peak RSS up 2.27 GB → 2.52 GB. Threads only engage for a genuinely large cold parse: never with under 200 MB behind the pending whole-file re-parses, 2 or fewer cores, or under 4 GB of available memory — so warm and incremental runs are untouched and spawn nothing. Otherwise the count is `min(cores - 1, min(0.25 × available, 2 GB) / 256 MB, pendingFiles / 50)`, where available is `process.availableMemory()` (cgroup-aware in containers) rather than free memory, which on macOS reports free pages and would switch the feature on and off between runs. `CODEBURN_PARSE_WORKERS=0` forces the serial parse and `CODEBURN_PARSE_WORKERS=N` forces N (capped at the core count), both bypassing every gate; `CODEBURN_VERBOSE=1` prints the resolved count and why.
|
||||
|
|
|
|||
|
|
@ -5,7 +5,16 @@ import { homedir } from 'os'
|
|||
import { calculateCost } from '../models.js'
|
||||
import { extractBashCommands } from '../bash-utils.js'
|
||||
import { readCachedResults, writeCachedResults } from '../cursor-cache.js'
|
||||
import { isSqliteAvailable, isSqliteBusyError, getSqliteLoadError, openDatabase, blobToText, type SqliteDatabase } from '../sqlite.js'
|
||||
import {
|
||||
isSqliteAvailable,
|
||||
isSqliteBusyError,
|
||||
getSqliteLoadError,
|
||||
openDatabase,
|
||||
blobToText,
|
||||
isSqliteReadonlyError,
|
||||
warnSqliteReadonlyOnce,
|
||||
type SqliteDatabase,
|
||||
} from '../sqlite.js'
|
||||
import { estimateTokensFromChars } from '../token-estimate.js'
|
||||
import type { DateRange } from '../types.js'
|
||||
import type { ProbeRoot, Provider, SessionSource, SessionParser, ParsedProviderCall } from './types.js'
|
||||
|
|
@ -188,7 +197,8 @@ function loadWorkspaceMap(workspaceStorageDir: string): WorkspaceMapping {
|
|||
let db: SqliteDatabase
|
||||
try {
|
||||
db = openDatabase(wsDbPath)
|
||||
} catch {
|
||||
} catch (err) {
|
||||
if (isSqliteReadonlyError(err)) warnSqliteReadonlyOnce(wsDbPath)
|
||||
continue
|
||||
}
|
||||
try {
|
||||
|
|
|
|||
|
|
@ -2,7 +2,16 @@ import { readdir } from 'fs/promises'
|
|||
import { join } from 'path'
|
||||
|
||||
import { calculateCost } from '../models.js'
|
||||
import { isSqliteAvailable, getSqliteLoadError, openDatabase, blobToText, isSqliteBusyError, type SqliteDatabase } from '../sqlite.js'
|
||||
import {
|
||||
isSqliteAvailable,
|
||||
getSqliteLoadError,
|
||||
openDatabase,
|
||||
blobToText,
|
||||
isSqliteBusyError,
|
||||
isSqliteReadonlyError,
|
||||
warnSqliteReadonlyOnce,
|
||||
type SqliteDatabase,
|
||||
} from '../sqlite.js'
|
||||
import { buildAssistantCall, parseTimestamp, sanitize, type MessageData, type PartData } from './session-message.js'
|
||||
import type {
|
||||
SessionSource,
|
||||
|
|
@ -310,7 +319,8 @@ export async function discoverSqliteSessions(
|
|||
let db: SqliteDatabase
|
||||
try {
|
||||
db = openDatabase(dbPath)
|
||||
} catch {
|
||||
} catch (err) {
|
||||
if (isSqliteReadonlyError(err)) warnSqliteReadonlyOnce(dbPath)
|
||||
continue
|
||||
}
|
||||
|
||||
|
|
|
|||
325
src/sqlite.ts
325
src/sqlite.ts
|
|
@ -1,4 +1,10 @@
|
|||
import { createRequire } from 'node:module'
|
||||
import { copyFileSync, existsSync, mkdirSync, readdirSync, renameSync, statSync, unlinkSync, utimesSync } from 'node:fs'
|
||||
import { createHash, randomBytes } from 'node:crypto'
|
||||
import { join } from 'node:path'
|
||||
import { pathToFileURL } from 'node:url'
|
||||
|
||||
import { getCodeburnCacheDir } from './cache-dir.js'
|
||||
|
||||
/// Thin SQLite read-only wrapper over Node's built-in `node:sqlite` module (stable in
|
||||
/// Node 24, experimental in Node 22 / 23). Replaces the earlier `better-sqlite3` binding
|
||||
|
|
@ -14,12 +20,14 @@ export type SqliteDatabase = {
|
|||
close(): void
|
||||
}
|
||||
|
||||
type DatabaseSyncCtor = new (path: string, options?: { readOnly?: boolean }) => {
|
||||
type DatabaseSyncInstance = {
|
||||
prepare(sql: string): { all(...params: unknown[]): Row[] }
|
||||
exec?(sql: string): void
|
||||
close(): void
|
||||
}
|
||||
|
||||
type DatabaseSyncCtor = new (path: string, options?: { readOnly?: boolean }) => DatabaseSyncInstance
|
||||
|
||||
let DatabaseSync: DatabaseSyncCtor | null = null
|
||||
let loadAttempted = false
|
||||
let loadError: string | null = null
|
||||
|
|
@ -116,12 +124,304 @@ export function isSqliteBusyError(err: unknown): boolean {
|
|||
)
|
||||
}
|
||||
|
||||
/// SQLite reports SQLITE_READONLY_DIRECTORY as ERR_SQLITE_ERROR with an extended
|
||||
/// result code on the Node 22 builds CodeBurn supports. Keep the base-code check
|
||||
/// so this also covers SQLITE_READONLY and its other extended variants, while
|
||||
/// leaving ENOENT/SQLITE_CANTOPEN distinguishable to callers.
|
||||
export function isSqliteReadonlyError(err: unknown): boolean {
|
||||
const e = err as { code?: unknown; errcode?: unknown; errstr?: unknown; message?: unknown } | null
|
||||
const code = typeof e?.code === 'string' ? e.code : ''
|
||||
const errcode = typeof e?.errcode === 'number' ? e.errcode : null
|
||||
const message = [
|
||||
typeof e?.message === 'string' ? e.message : '',
|
||||
typeof e?.errstr === 'string' ? e.errstr : '',
|
||||
].join(' ')
|
||||
|
||||
return (
|
||||
(errcode !== null && (errcode & 0xff) === 8) ||
|
||||
/SQLITE_READONLY|attempt to write a readonly database|readonly database|read-only database/i.test(`${code} ${message}`)
|
||||
)
|
||||
}
|
||||
|
||||
/// A read-only parent reports SQLITE_READONLY_DIRECTORY when it must create the
|
||||
/// sidecars from scratch, but SQLITE_CANTOPEN when a `-wal` is present and the
|
||||
/// `-shm` it needs to index it is not. openReadonlyCache re-throws the original
|
||||
/// error when the database itself is missing, which is the other CANTOPEN.
|
||||
function isSqliteSidecarError(err: unknown): boolean {
|
||||
if (isSqliteReadonlyError(err)) return true
|
||||
const errcode = (err as { errcode?: unknown } | null)?.errcode
|
||||
return typeof errcode === 'number' && (errcode & 0xff) === 14
|
||||
}
|
||||
|
||||
let uriFilenamesSupported: boolean | null = null
|
||||
|
||||
/// node:sqlite only enables SQLITE_OPEN_URI from Node 22.15 on (measured: 22.13
|
||||
/// and 22.14 fail, 22.15 and later work). Below that a `file:...` location is
|
||||
/// taken literally and fails as CANTOPEN, so the immutable open is not attempted
|
||||
/// there. The probe is an in-memory URI rather than a version comparison: it
|
||||
/// answers the question directly and touches no filesystem either way.
|
||||
export function sqliteSupportsUriFilenames(): boolean {
|
||||
if (uriFilenamesSupported !== null) return uriFilenamesSupported
|
||||
uriFilenamesSupported = false
|
||||
const Driver = loadDriver() ? DatabaseSync : null
|
||||
if (Driver !== null) {
|
||||
try {
|
||||
new Driver('file:codeburn-uri-probe?mode=memory', { readOnly: true }).close()
|
||||
uriFilenamesSupported = true
|
||||
} catch {
|
||||
// An older build: locations are plain paths, and the copy fallback covers
|
||||
// exactly the case the immutable open would have.
|
||||
}
|
||||
}
|
||||
return uriFilenamesSupported
|
||||
}
|
||||
|
||||
type DatabaseFingerprint = {
|
||||
dev: number
|
||||
ino: number
|
||||
mtimeMs: number
|
||||
sizeBytes: number
|
||||
walBytes: number
|
||||
}
|
||||
|
||||
/// A superseded copy is dropped once it has gone this long without being used.
|
||||
/// The delay is what keeps a concurrent reader of the previous copy from having
|
||||
/// its file yanked out from under it.
|
||||
const CACHE_ENTRY_MAX_AGE_MS = 24 * 60 * 60 * 1000
|
||||
const warnedDatabases = new Set<string>()
|
||||
|
||||
/// One notice per source path per run: a provider may discover many sessions
|
||||
/// from the same database, and the first notice already says what happened.
|
||||
function warnSqliteOnce(path: string, message: string): void {
|
||||
if (warnedDatabases.has(path)) return
|
||||
warnedDatabases.add(path)
|
||||
process.stderr.write(message)
|
||||
}
|
||||
|
||||
/// A read-only SQLite connection can still need sidecar files.
|
||||
export function warnSqliteReadonlyOnce(path: string): void {
|
||||
warnSqliteOnce(
|
||||
path,
|
||||
`codeburn: SQLite database ${path} is in a read-only directory and needs sidecar files; using a cache copy when necessary. ` +
|
||||
'The original database is not modified.\n',
|
||||
)
|
||||
}
|
||||
|
||||
function errorCode(err: unknown): string | undefined {
|
||||
if (typeof err !== 'object' || err === null || !('code' in err)) return undefined
|
||||
const code = err.code
|
||||
return typeof code === 'string' ? code : undefined
|
||||
}
|
||||
|
||||
function describeError(err: unknown): string {
|
||||
return err instanceof Error ? err.message : String(err)
|
||||
}
|
||||
|
||||
/// This deliberately mirrors fingerprintSqliteFile/fingerprintFile in
|
||||
/// session-cache.ts. openDatabase is synchronous, so the fallback uses the
|
||||
/// synchronous fs APIs only after the direct open has already failed; the
|
||||
/// ordinary successful open remains probe-free.
|
||||
function fingerprintDatabase(path: string): DatabaseFingerprint {
|
||||
const main = statSync(path)
|
||||
let wal: ReturnType<typeof statSync> | null = null
|
||||
try {
|
||||
wal = statSync(path + '-wal')
|
||||
} catch (err) {
|
||||
if (errorCode(err) !== 'ENOENT') throw err
|
||||
}
|
||||
return {
|
||||
dev: main.dev,
|
||||
ino: main.ino,
|
||||
mtimeMs: wal ? Math.max(main.mtimeMs, wal.mtimeMs) : main.mtimeMs,
|
||||
sizeBytes: main.size + (wal?.size ?? 0),
|
||||
walBytes: wal?.size ?? 0,
|
||||
}
|
||||
}
|
||||
|
||||
function sameFingerprint(a: DatabaseFingerprint, b: DatabaseFingerprint): boolean {
|
||||
return (
|
||||
a.dev === b.dev &&
|
||||
a.ino === b.ino &&
|
||||
a.mtimeMs === b.mtimeMs &&
|
||||
a.sizeBytes === b.sizeBytes &&
|
||||
a.walBytes === b.walBytes
|
||||
)
|
||||
}
|
||||
|
||||
function unlinkQuietly(path: string): void {
|
||||
try {
|
||||
unlinkSync(path)
|
||||
} catch {
|
||||
// Already gone, or still held open by another CodeBurn on Windows. Either
|
||||
// way the next run's eviction pass gets another chance at it.
|
||||
}
|
||||
}
|
||||
|
||||
function copyOptionalFile(sourcePath: string, destinationPath: string): boolean {
|
||||
try {
|
||||
copyFileSync(sourcePath, destinationPath)
|
||||
return true
|
||||
} catch (err) {
|
||||
if (errorCode(err) === 'ENOENT') return false
|
||||
throw err
|
||||
}
|
||||
}
|
||||
|
||||
function sourceKeyOf(sourcePath: string): string {
|
||||
return createHash('sha256').update(sourcePath, 'utf8').digest('hex').slice(0, 32)
|
||||
}
|
||||
|
||||
/// The copy is named after the source it came from AND the fingerprint it was
|
||||
/// taken at, so a refresh publishes a new file rather than overwriting one that
|
||||
/// another process may still have open.
|
||||
function cacheEntryName(sourceKey: string, fingerprint: DatabaseFingerprint): string {
|
||||
const parts = `${fingerprint.dev}:${fingerprint.ino}:${fingerprint.mtimeMs}:${fingerprint.sizeBytes}:${fingerprint.walBytes}`
|
||||
return `${sourceKey}.${createHash('sha256').update(parts).digest('hex').slice(0, 16)}.db`
|
||||
}
|
||||
|
||||
function dropCopy(cacheDir: string, name: string): void {
|
||||
unlinkQuietly(join(cacheDir, name))
|
||||
unlinkQuietly(join(cacheDir, name + '-wal'))
|
||||
unlinkQuietly(join(cacheDir, name + '-shm'))
|
||||
}
|
||||
|
||||
/// Superseded copies are cleaned up here rather than by overwriting them: keep
|
||||
/// the one in use plus at most one predecessor, and drop anything untouched for
|
||||
/// a day, which is also what a source path that no longer exists looks like.
|
||||
/// Reuse touches the copy, so its mtime is last-use rather than copy time.
|
||||
function evictSupersededCopies(cacheDir: string, sourceKey: string, keepName: string): void {
|
||||
let names: string[]
|
||||
try {
|
||||
names = readdirSync(cacheDir)
|
||||
} catch {
|
||||
return
|
||||
}
|
||||
const now = Date.now()
|
||||
const superseded: { name: string, mtimeMs: number }[] = []
|
||||
for (const name of names) {
|
||||
if (!name.endsWith('.db') || name === keepName) continue
|
||||
let mtimeMs: number
|
||||
try {
|
||||
mtimeMs = statSync(join(cacheDir, name)).mtimeMs
|
||||
} catch {
|
||||
continue
|
||||
}
|
||||
if (name.startsWith(`${sourceKey}.`)) superseded.push({ name, mtimeMs })
|
||||
else if (now - mtimeMs > CACHE_ENTRY_MAX_AGE_MS) dropCopy(cacheDir, name)
|
||||
}
|
||||
superseded.sort((a, b) => b.mtimeMs - a.mtimeMs)
|
||||
for (const [index, entry] of superseded.entries()) {
|
||||
if (index > 0 || now - entry.mtimeMs > CACHE_ENTRY_MAX_AGE_MS) dropCopy(cacheDir, entry.name)
|
||||
}
|
||||
}
|
||||
|
||||
/// A concurrent CodeBurn may have published the same copy first. The name is
|
||||
/// the fingerprint, so the content is identical by construction and losing that
|
||||
/// race is not an error.
|
||||
function publish(tempPath: string, finalPath: string): void {
|
||||
try {
|
||||
renameSync(tempPath, finalPath)
|
||||
} catch (err) {
|
||||
if (!existsSync(finalPath)) throw err
|
||||
}
|
||||
}
|
||||
|
||||
function readOnlyCachePath(sourcePath: string, fingerprint: DatabaseFingerprint): string {
|
||||
const cacheDir = join(getCodeburnCacheDir(), 'sqlite-ro')
|
||||
mkdirSync(cacheDir, { recursive: true, mode: 0o700 })
|
||||
|
||||
const sourceKey = sourceKeyOf(sourcePath)
|
||||
const name = cacheEntryName(sourceKey, fingerprint)
|
||||
const cachePath = join(cacheDir, name)
|
||||
if (existsSync(cachePath)) {
|
||||
const now = new Date()
|
||||
try {
|
||||
utimesSync(cachePath, now, now)
|
||||
} catch {
|
||||
// mtime is only the eviction clock; a copy we cannot touch still reads.
|
||||
}
|
||||
evictSupersededCopies(cacheDir, sourceKey, name)
|
||||
return cachePath
|
||||
}
|
||||
|
||||
const tempBase = `${cachePath}.tmp-${process.pid}-${randomBytes(8).toString('hex')}`
|
||||
const tempWal = tempBase + '-wal'
|
||||
try {
|
||||
copyFileSync(sourcePath, tempBase)
|
||||
const copiedWal = copyOptionalFile(sourcePath + '-wal', tempWal)
|
||||
|
||||
// Do not publish a cache made from a moving database. A live WAL writer will
|
||||
// normally make the direct open succeed once its sidecars exist; this check
|
||||
// covers the narrow race where the source changes during the copy fallback.
|
||||
if (!sameFingerprint(fingerprintDatabase(sourcePath), fingerprint)) {
|
||||
throw new Error('SQLite database changed while preparing its read-only cache copy')
|
||||
}
|
||||
|
||||
// The -wal goes first: a reader that can see the database must never find it
|
||||
// without the sidecar holding its most recent rows.
|
||||
if (copiedWal) publish(tempWal, cachePath + '-wal')
|
||||
publish(tempBase, cachePath)
|
||||
evictSupersededCopies(cacheDir, sourceKey, name)
|
||||
return cachePath
|
||||
} finally {
|
||||
unlinkQuietly(tempBase)
|
||||
unlinkQuietly(tempWal)
|
||||
}
|
||||
}
|
||||
|
||||
function openReadonlyCache(path: string, originalError: unknown): DatabaseSyncInstance {
|
||||
const Driver = DatabaseSync
|
||||
if (Driver === null) throw new Error(getSqliteLoadError())
|
||||
|
||||
let fingerprint: DatabaseFingerprint
|
||||
try {
|
||||
fingerprint = fingerprintDatabase(path)
|
||||
} catch {
|
||||
// Preserve the original SQLite error when the source disappeared or became
|
||||
// inaccessible between the failed query and the fallback probe.
|
||||
throw originalError
|
||||
}
|
||||
|
||||
// An absent or empty -wal holds no frames, so there is nothing to go stale and
|
||||
// nothing worth copying: immutable lets SQLite skip the -shm it cannot create
|
||||
// and read the source in place.
|
||||
if (fingerprint.walBytes === 0 && sqliteSupportsUriFilenames()) {
|
||||
try {
|
||||
return new Driver(`${pathToFileURL(path).href}?immutable=1`, { readOnly: true })
|
||||
} catch {
|
||||
// Understood but refused: the copy covers it.
|
||||
}
|
||||
}
|
||||
|
||||
let cachedPath: string
|
||||
try {
|
||||
cachedPath = readOnlyCachePath(path, fingerprint)
|
||||
} catch (err) {
|
||||
warnSqliteOnce(
|
||||
path,
|
||||
`codeburn: SQLite database ${path} is in a read-only directory and its cache copy could not be written ` +
|
||||
`(${describeError(err)}); skipping this database.\n`,
|
||||
)
|
||||
throw originalError
|
||||
}
|
||||
return new Driver(cachedPath, { readOnly: true })
|
||||
}
|
||||
|
||||
export function openDatabase(path: string): SqliteDatabase {
|
||||
if (!loadDriver() || DatabaseSync === null) {
|
||||
throw new Error(getSqliteLoadError())
|
||||
}
|
||||
|
||||
const db = new DatabaseSync(path, { readOnly: true })
|
||||
let db: DatabaseSyncInstance
|
||||
let fallbackUsed = false
|
||||
try {
|
||||
db = new DatabaseSync(path, { readOnly: true })
|
||||
} catch (err) {
|
||||
if (!isSqliteSidecarError(err)) throw err
|
||||
fallbackUsed = true
|
||||
db = openReadonlyCache(path, err)
|
||||
warnSqliteReadonlyOnce(path)
|
||||
}
|
||||
try {
|
||||
db.exec?.('PRAGMA busy_timeout = 1000')
|
||||
} catch {
|
||||
|
|
@ -130,7 +430,26 @@ export function openDatabase(path: string): SqliteDatabase {
|
|||
|
||||
return {
|
||||
query<T extends Row = Row>(sql: string, params: unknown[] = []): T[] {
|
||||
return db.prepare(sql).all(...params) as T[]
|
||||
try {
|
||||
return db.prepare(sql).all(...params) as T[]
|
||||
} catch (err) {
|
||||
if (!isSqliteSidecarError(err)) throw err
|
||||
if (fallbackUsed) throw err
|
||||
fallbackUsed = true
|
||||
try {
|
||||
db.close()
|
||||
} catch {
|
||||
// The failed connection may already have been closed by node:sqlite.
|
||||
}
|
||||
db = openReadonlyCache(path, err)
|
||||
warnSqliteReadonlyOnce(path)
|
||||
try {
|
||||
db.exec?.('PRAGMA busy_timeout = 1000')
|
||||
} catch {
|
||||
// Best effort, matching the direct-open path above.
|
||||
}
|
||||
return db.prepare(sql).all(...params) as T[]
|
||||
}
|
||||
},
|
||||
close() {
|
||||
db.close()
|
||||
|
|
|
|||
332
tests/sqlite-readonly-parent.test.ts
Normal file
332
tests/sqlite-readonly-parent.test.ts
Normal file
|
|
@ -0,0 +1,332 @@
|
|||
import { chmodSync, copyFileSync, existsSync, mkdirSync, readdirSync, statSync, utimesSync, writeFileSync } from 'node:fs'
|
||||
import { mkdtemp, rm } from 'node:fs/promises'
|
||||
import { createRequire } from 'node:module'
|
||||
import { join } from 'node:path'
|
||||
import { tmpdir } from 'node:os'
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
import {
|
||||
isSqliteReadonlyError,
|
||||
openDatabase,
|
||||
sqliteSupportsUriFilenames,
|
||||
} from '../src/sqlite.js'
|
||||
import {
|
||||
discoverSqliteSessions,
|
||||
type SqliteProviderConfig,
|
||||
} from '../src/providers/sqlite-session-parser.js'
|
||||
|
||||
const requireForTest = createRequire(import.meta.url)
|
||||
|
||||
type NativeDatabase = {
|
||||
exec(sql: string): void
|
||||
prepare(sql: string): { run(...params: unknown[]): void; all(...params: unknown[]): unknown[] }
|
||||
close(): void
|
||||
}
|
||||
|
||||
type NativeDatabaseCtor = new (path: string) => NativeDatabase
|
||||
|
||||
const { DatabaseSync: NativeDatabase } = requireForTest('node:sqlite') as {
|
||||
DatabaseSync: NativeDatabaseCtor
|
||||
}
|
||||
|
||||
let sourceRoot: string
|
||||
let cacheRoot: string
|
||||
let previousCacheDir: string | undefined
|
||||
const openWriters: NativeDatabase[] = []
|
||||
|
||||
beforeEach(async () => {
|
||||
sourceRoot = await mkdtemp(join(tmpdir(), 'codeburn-sqlite-source-'))
|
||||
cacheRoot = await mkdtemp(join(tmpdir(), 'codeburn-sqlite-cache-'))
|
||||
previousCacheDir = process.env['CODEBURN_CACHE_DIR']
|
||||
process.env['CODEBURN_CACHE_DIR'] = cacheRoot
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
chmodSync(sourceRoot, 0o755)
|
||||
chmodSync(cacheRoot, 0o755)
|
||||
for (const writer of openWriters.splice(0)) writer.close()
|
||||
await rm(sourceRoot, { recursive: true, force: true })
|
||||
await rm(cacheRoot, { recursive: true, force: true })
|
||||
if (previousCacheDir === undefined) delete process.env['CODEBURN_CACHE_DIR']
|
||||
else process.env['CODEBURN_CACHE_DIR'] = previousCacheDir
|
||||
})
|
||||
|
||||
function createClosedWalDatabase(dbPath: string): void {
|
||||
const db = new NativeDatabase(dbPath)
|
||||
db.exec('PRAGMA journal_mode=WAL')
|
||||
db.exec('CREATE TABLE values_table (c INTEGER)')
|
||||
db.prepare('INSERT INTO values_table (c) VALUES (?)').run(1)
|
||||
db.close()
|
||||
}
|
||||
|
||||
function createOpenWalDatabase(dbPath: string): NativeDatabase {
|
||||
const db = new NativeDatabase(dbPath)
|
||||
db.exec('PRAGMA journal_mode=WAL')
|
||||
db.exec('CREATE TABLE values_table (c INTEGER)')
|
||||
db.prepare('INSERT INTO values_table (c) VALUES (?)').run(1)
|
||||
expect(existsSync(dbPath + '-wal')).toBe(true)
|
||||
expect(existsSync(dbPath + '-shm')).toBe(true)
|
||||
openWriters.push(db)
|
||||
return db
|
||||
}
|
||||
|
||||
/// A database plus a non-empty -wal and no -shm: what a snapshot, an rsync or an
|
||||
/// unclean unmount of a live source leaves behind. The second row exists only in
|
||||
/// the -wal, so dropping it would be silent data loss rather than an error.
|
||||
function writeUncheckpointedWalDatabase(dbPath: string): void {
|
||||
const originDir = join(sourceRoot, `origin-${readdirSync(sourceRoot).length}`)
|
||||
mkdirSync(originDir)
|
||||
const originPath = join(originDir, 'state.vscdb')
|
||||
const writer = new NativeDatabase(originPath)
|
||||
writer.exec('PRAGMA journal_mode=WAL')
|
||||
writer.exec('CREATE TABLE values_table (c INTEGER)')
|
||||
writer.prepare('INSERT INTO values_table (c) VALUES (?)').run(1)
|
||||
writer.exec('PRAGMA wal_checkpoint(TRUNCATE)')
|
||||
writer.exec('PRAGMA wal_autocheckpoint=0')
|
||||
writer.prepare('INSERT INTO values_table (c) VALUES (?)').run(2)
|
||||
copyFileSync(originPath, dbPath)
|
||||
copyFileSync(originPath + '-wal', dbPath + '-wal')
|
||||
writer.close()
|
||||
expect(statSync(dbPath + '-wal').size).toBeGreaterThan(0)
|
||||
expect(existsSync(dbPath + '-shm')).toBe(false)
|
||||
}
|
||||
|
||||
function createDiscoveryDatabase(dbPath: string): void {
|
||||
const db = new NativeDatabase(dbPath)
|
||||
db.exec('PRAGMA journal_mode=WAL')
|
||||
db.exec(`
|
||||
CREATE TABLE session (
|
||||
id TEXT PRIMARY KEY,
|
||||
directory TEXT,
|
||||
title TEXT,
|
||||
time_created INTEGER,
|
||||
parent_id TEXT,
|
||||
time_archived INTEGER
|
||||
)
|
||||
`)
|
||||
db.exec('CREATE TABLE message (id TEXT PRIMARY KEY, session_id TEXT, time_created INTEGER, data BLOB)')
|
||||
db.exec('CREATE TABLE part (id INTEGER PRIMARY KEY, message_id TEXT, session_id TEXT, data BLOB)')
|
||||
db.prepare(
|
||||
'INSERT INTO session (id, directory, title, time_created, parent_id, time_archived) VALUES (?, ?, ?, ?, ?, ?)',
|
||||
).run('session-1', '/tmp/project', 'Read-only fixture', Date.now(), null, null)
|
||||
db.close()
|
||||
}
|
||||
|
||||
function makeSourceParentReadOnly(skip: (reason?: string) => void): boolean {
|
||||
chmodSync(sourceRoot, 0o555)
|
||||
const mode = statSync(sourceRoot).mode & 0o777
|
||||
if ((mode & 0o222) !== 0) {
|
||||
skip(`SKIP: chmod 0555 did not make the fixture parent non-writable (mode ${mode.toString(8)})`)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
function makeSourceParentWritable(): void {
|
||||
chmodSync(sourceRoot, 0o755)
|
||||
}
|
||||
|
||||
function cachedDatabaseFiles(): string[] {
|
||||
try {
|
||||
return readdirSync(join(cacheRoot, 'sqlite-ro')).filter(name => name.endsWith('.db'))
|
||||
} catch {
|
||||
return []
|
||||
}
|
||||
}
|
||||
|
||||
function readValue(dbPath: string): number {
|
||||
const db = openDatabase(dbPath)
|
||||
try {
|
||||
const rows = db.query<{ c: number }>('SELECT c FROM values_table')
|
||||
return rows[0]?.c ?? -1
|
||||
} finally {
|
||||
db.close()
|
||||
}
|
||||
}
|
||||
|
||||
describe('SQLite read-only parent fallback', () => {
|
||||
it('keeps the existing writable-parent open behaviour', () => {
|
||||
const dbPath = join(sourceRoot, 'state.vscdb')
|
||||
createClosedWalDatabase(dbPath)
|
||||
expect(existsSync(dbPath + '-wal')).toBe(false)
|
||||
expect(existsSync(dbPath + '-shm')).toBe(false)
|
||||
|
||||
expect(readValue(dbPath)).toBe(1)
|
||||
|
||||
expect(existsSync(dbPath + '-wal')).toBe(true)
|
||||
expect(existsSync(dbPath + '-shm')).toBe(true)
|
||||
expect(cachedDatabaseFiles()).toEqual([])
|
||||
})
|
||||
|
||||
it('reads a read-only parent with no -wal, in place where it can and by copy where it cannot', ({ skip }) => {
|
||||
const dbPath = join(sourceRoot, 'state.vscdb')
|
||||
createClosedWalDatabase(dbPath)
|
||||
expect(existsSync(dbPath + '-wal')).toBe(false)
|
||||
expect(existsSync(dbPath + '-shm')).toBe(false)
|
||||
if (!makeSourceParentReadOnly(skip)) return
|
||||
|
||||
expect(readValue(dbPath)).toBe(1)
|
||||
|
||||
expect(existsSync(dbPath + '-wal')).toBe(false)
|
||||
expect(existsSync(dbPath + '-shm')).toBe(false)
|
||||
// No WAL frames exist, so there is nothing to go stale and nothing worth
|
||||
// copying: immutable reads the source in place. node:sqlite only honours
|
||||
// URI filenames on newer builds, and on the 22.13 floor the copy stands in.
|
||||
expect(cachedDatabaseFiles()).toHaveLength(sqliteSupportsUriFilenames() ? 0 : 1)
|
||||
})
|
||||
|
||||
it('opens directly when a read-only parent already has WAL sidecars', ({ skip }) => {
|
||||
const dbPath = join(sourceRoot, 'state.vscdb')
|
||||
createOpenWalDatabase(dbPath)
|
||||
if (!makeSourceParentReadOnly(skip)) return
|
||||
|
||||
expect(readValue(dbPath)).toBe(1)
|
||||
expect(cachedDatabaseFiles()).toEqual([])
|
||||
})
|
||||
|
||||
it('reads un-checkpointed WAL rows when the parent is read-only and the -shm is absent', ({ skip }) => {
|
||||
// SQLite reports a -wal without its -shm as SQLITE_CANTOPEN, not
|
||||
// SQLITE_READONLY, and the un-checkpointed row lives only in the -wal.
|
||||
const dbPath = join(sourceRoot, 'state.vscdb')
|
||||
writeUncheckpointedWalDatabase(dbPath)
|
||||
if (!makeSourceParentReadOnly(skip)) return
|
||||
|
||||
const db = openDatabase(dbPath)
|
||||
try {
|
||||
expect(db.query<{ c: number }>('SELECT c FROM values_table ORDER BY c')).toEqual([{ c: 1 }, { c: 2 }])
|
||||
} finally {
|
||||
db.close()
|
||||
}
|
||||
expect(existsSync(dbPath + '-shm')).toBe(false)
|
||||
expect(cachedDatabaseFiles()).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('reuses an unchanged fallback copy instead of copying the database again', ({ skip }) => {
|
||||
const dbPath = join(sourceRoot, 'state.vscdb')
|
||||
writeUncheckpointedWalDatabase(dbPath)
|
||||
if (!makeSourceParentReadOnly(skip)) return
|
||||
|
||||
expect(readValue(dbPath)).toBe(1)
|
||||
const first = cachedDatabaseFiles()
|
||||
expect(first).toHaveLength(1)
|
||||
const cachedPath = join(cacheRoot, 'sqlite-ro', first[0]!)
|
||||
const firstIno = statSync(cachedPath).ino
|
||||
expect(readValue(dbPath)).toBe(1)
|
||||
expect(cachedDatabaseFiles()).toEqual(first)
|
||||
expect(statSync(cachedPath).ino).toBe(firstIno)
|
||||
})
|
||||
|
||||
it('publishes a refreshed copy beside the old one and keeps at most one predecessor', ({ skip }) => {
|
||||
const dbPath = join(sourceRoot, 'state.vscdb')
|
||||
writeUncheckpointedWalDatabase(dbPath)
|
||||
if (!makeSourceParentReadOnly(skip)) return
|
||||
expect(readValue(dbPath)).toBe(1)
|
||||
const [first] = cachedDatabaseFiles()
|
||||
const firstIno = statSync(join(cacheRoot, 'sqlite-ro', first!)).ino
|
||||
|
||||
// A changed source must not overwrite the copy a concurrent reader may still
|
||||
// have open: Windows cannot unlink it, and the name is the fingerprint.
|
||||
makeSourceParentWritable()
|
||||
writeUncheckpointedWalDatabase(dbPath)
|
||||
if (!makeSourceParentReadOnly(skip)) return
|
||||
expect(readValue(dbPath)).toBe(1)
|
||||
const second = cachedDatabaseFiles()
|
||||
expect(second).toHaveLength(2)
|
||||
expect(second).toContain(first)
|
||||
expect(statSync(join(cacheRoot, 'sqlite-ro', first!)).ino).toBe(firstIno)
|
||||
|
||||
makeSourceParentWritable()
|
||||
writeUncheckpointedWalDatabase(dbPath)
|
||||
if (!makeSourceParentReadOnly(skip)) return
|
||||
expect(readValue(dbPath)).toBe(1)
|
||||
const third = cachedDatabaseFiles()
|
||||
expect(third).toHaveLength(2)
|
||||
expect(third).not.toContain(first)
|
||||
})
|
||||
|
||||
it('evicts a copy left untouched for a day, including one whose source is gone', ({ skip }) => {
|
||||
const dbPath = join(sourceRoot, 'state.vscdb')
|
||||
writeUncheckpointedWalDatabase(dbPath)
|
||||
if (!makeSourceParentReadOnly(skip)) return
|
||||
expect(readValue(dbPath)).toBe(1)
|
||||
const cacheDir = join(cacheRoot, 'sqlite-ro')
|
||||
const predecessor = join(cacheDir, cachedDatabaseFiles()[0]!)
|
||||
|
||||
// A copy of a database that no longer exists is simply one nothing touches.
|
||||
const orphan = join(cacheDir, `${'0'.repeat(32)}.deadbeefdeadbeef.db`)
|
||||
writeFileSync(orphan, 'orphan')
|
||||
const aDayAndAnHourAgo = new Date(Date.now() - 25 * 60 * 60 * 1000)
|
||||
utimesSync(orphan, aDayAndAnHourAgo, aDayAndAnHourAgo)
|
||||
|
||||
makeSourceParentWritable()
|
||||
writeUncheckpointedWalDatabase(dbPath)
|
||||
if (!makeSourceParentReadOnly(skip)) return
|
||||
expect(readValue(dbPath)).toBe(1)
|
||||
expect(existsSync(orphan)).toBe(false)
|
||||
expect(existsSync(predecessor)).toBe(true)
|
||||
|
||||
// A day without a read and the superseded copy goes too.
|
||||
utimesSync(predecessor, aDayAndAnHourAgo, aDayAndAnHourAgo)
|
||||
expect(readValue(dbPath)).toBe(1)
|
||||
|
||||
expect(existsSync(predecessor)).toBe(false)
|
||||
expect(cachedDatabaseFiles()).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('says so instead of going quiet when the cache copy cannot be written', ({ skip }) => {
|
||||
const dbPath = join(sourceRoot, 'state.vscdb')
|
||||
writeUncheckpointedWalDatabase(dbPath)
|
||||
if (!makeSourceParentReadOnly(skip)) return
|
||||
chmodSync(cacheRoot, 0o555)
|
||||
|
||||
const stderr = vi.spyOn(process.stderr, 'write').mockReturnValue(true)
|
||||
try {
|
||||
expect(() => readValue(dbPath)).toThrow()
|
||||
const notices = stderr.mock.calls.filter(([chunk]) => String(chunk).includes('cache copy could not be written'))
|
||||
expect(notices).toHaveLength(1)
|
||||
expect(String(notices[0]?.[0])).toContain(dbPath)
|
||||
} finally {
|
||||
stderr.mockRestore()
|
||||
chmodSync(cacheRoot, 0o755)
|
||||
}
|
||||
})
|
||||
|
||||
it('keeps a genuinely missing database distinguishable from SQLITE_READONLY', () => {
|
||||
let thrown: unknown
|
||||
try {
|
||||
openDatabase(join(sourceRoot, 'missing.vscdb'))
|
||||
} catch (err) {
|
||||
thrown = err
|
||||
}
|
||||
|
||||
expect(thrown).toBeDefined()
|
||||
expect(isSqliteReadonlyError(thrown)).toBe(false)
|
||||
expect(thrown).toMatchObject({ errcode: 14, message: 'unable to open database file' })
|
||||
})
|
||||
|
||||
it('surfaces one read-only notice in SQLite discovery and still finds the session', async ({ skip }) => {
|
||||
const dbPath = join(sourceRoot, 'state.db')
|
||||
createDiscoveryDatabase(dbPath)
|
||||
if (!makeSourceParentReadOnly(skip)) return
|
||||
|
||||
const config: SqliteProviderConfig = {
|
||||
providerName: 'opencode',
|
||||
displayName: 'OpenCode',
|
||||
dbDir: sourceRoot,
|
||||
dbFilePrefix: 'state',
|
||||
}
|
||||
const stderr = vi.spyOn(process.stderr, 'write').mockReturnValue(true)
|
||||
try {
|
||||
const sessions = await discoverSqliteSessions(config)
|
||||
expect(sessions).toHaveLength(1)
|
||||
expect(sessions[0]?.path).toBe(`${dbPath}:session-1`)
|
||||
expect(stderr.mock.calls.filter(([chunk]) => String(chunk).includes('read-only directory'))).toHaveLength(1)
|
||||
|
||||
await discoverSqliteSessions(config)
|
||||
expect(stderr.mock.calls.filter(([chunk]) => String(chunk).includes('read-only directory'))).toHaveLength(1)
|
||||
} finally {
|
||||
stderr.mockRestore()
|
||||
}
|
||||
})
|
||||
})
|
||||
Loading…
Add table
Add a link
Reference in a new issue