mirror of
https://github.com/AgentSeal/codeburn.git
synced 2026-08-29 10:22:42 +00:00
Merge pull request #1120 from getagentseal/fix/1117-menubar-livelock
fix(menubar): no-output watchdog, lock signal cleanup, dead-pid takeover, serve orphan reaping
This commit is contained in:
commit
84c4a973a0
14 changed files with 756 additions and 58 deletions
|
|
@ -37,6 +37,7 @@
|
|||
- **The snap asks for the log directories it reads, not each tool's whole home.** The first Snap Store submission declared a `personal-files` read of every AI tool's root — `$HOME/.claude`, `$HOME/.codex`, `$HOME/.cursor` and the rest — and that interface is recursive, so it granted read of every credential file those roots hold. Each entry now names the subdirectory the provider actually opens (`.claude/projects`, `.codex/sessions`, `.cline/data`, `.vibe/logs/session`, `.dsh/sessions`, `.kiro/sessions`, `.quickwork/{profiles.json,sessions,metrics}`, `.config/Claude/local-agent-mode-sessions`, `.config/Open Design/{runs,data/runs,namespaces}`), two are single files (`.forge/.forge.db`, `.zcode/cli/db/db.sqlite`), and the editor entries name only the extension folders holding transcripts instead of the editor's whole configuration. Five providers that were missing entirely and would have shown no data are declared — opencode, crush, goose, kilo, kimi-code — and four roots stay roots only because the file the provider opens sits directly in them (`.config/github-copilot`, `.local/share/{opencode,crush,kilo}`). One credential file is now requested openly rather than implicitly: `.claude/.credentials.json`, read-only, for the live plan gauge. Codex's equivalent would need write access to the Codex CLI's own `auth.json` to rotate the token, so neither it nor a Codex root is declared and the Codex live gauge is disabled under `$SNAP`; Codex usage and cost are unaffected, they come from the session rollouts. Two consequences inside the snap: `.lingtai` is dropped, because its per-agent log directory needs a wildcard the interface has no form for, and `optimize`, `context-budget` and `act` no longer see the user-scope `~/.claude/settings.json`, `agents/`, `skills/` and `commands/` — project-scope copies still work through the `home` plug. Nothing outside the snap changes.
|
||||
|
||||
### Fixed (Desktop & Menubar)
|
||||
- **The menubar stops killing its own cold cache rebuild.** On a large corpus after the 0.9.20 cache-version bump, the menubar never completed a single fetch: every child was killed at exactly 45 seconds, and each kill left the `session-refresh.lock` behind for the next one to wait out. Four things were wrong and all four are fixed. The 45-second cap was a TOTAL-runtime kill, so it is now the same no-output watchdog the desktop app got in #1096 — spawns and the resident `codeburn serve` alike set `CODEBURN_PROGRESS=1`, the window restarts on every byte of stdout or stderr (the CLI heartbeats every 10 seconds while parsing, and now also while WAITING on the refresh lock, which was the silent stretch that mattered), and only a genuinely mute child is killed, with the same 10-minute cold floor until the first payload lands and the same 15-minute absolute ceiling behind it. Kills are SIGTERM first and SIGKILL only after a 5-second grace, and the refresh lock now arms the same signal cleanup the hydration lock has, so a killed holder unlinks its own lock instead of leaving one. A lock that was abandoned anyway is recovered immediately rather than after 90 seconds: a waiter takes over a lock whose recorded holder pid is gone, or whose heartbeat has frozen, and the waiter's own budget is now derived from the stale window so it can never again expire before the gate it is waiting for opens — a live holder, whose heartbeat keeps the mtime fresh and whose pid answers, is never taken from. The resident child's fixed 60-second warm request cap becomes a silence window that each progress frame restarts, and a spent restart budget is a five-minute cooldown rather than leaving the resident dead for the rest of the app run. Serve orphans get three new backstops: the app closes its end of a retired child's stdin (dropping the handle was not enough — the pipe stayed alive inside the Process), reaps every child it started synchronously at quit before the async shutdown can be skipped, and records the child's pid and command line so a serve orphaned by a crash is reaped on the next launch. On the CLI side a serve child's final exit no longer runs through the `process.exit` an in-flight request has monkeypatched, and its post-drain cleanup is bounded like the drain. (#1117)
|
||||
- **A long panel query is no longer killed for being slow.** The desktop app capped every read at 45 seconds of TOTAL runtime, so on a slow machine `optimize`, `yield`, `models`, `sessions`, `spend`, `audit`, `act report` and `plan` were SIGKILLed mid-parse and the panel painted a red "timed out" that a 60-second poll then reproduced forever. That cap is now a no-output watchdog: the window restarts on every byte the child writes, so only a genuinely silent child times out, with a 15-minute absolute ceiling still catching a livelocked one. Silence now means stopped rather than slow, because the parse itself heartbeats: every read spawn sets `CODEBURN_PROGRESS=1`, and under it a running parse emits a keepalive line every 10 seconds — the stretch that mattered was a cold parse's inter-provider cache save, measured at 31.6 seconds of total silence on a large corpus, which the old scan-progress stream did not cover at all. A served request heartbeats for its whole duration, parse and the aggregation and serialization after it alike; a one-shot spawn heartbeats through the parse, and the roughly 8-second tail that follows it stays well inside the window on its own. Resident `codeburn serve` requests reset their window on each frame of their own response. Alongside it: the cold-cache floor now covers EVERY read while the first hydration is still running, not just the overview, so a section that starts polling the moment the app is ready is not killed waiting behind that parse; a read that times out while the hydration is still going keeps the indexing splash instead of painting an error panel, bounded so that an install which can never hydrate still surfaces a real error once the cold window has elapsed rather than sitting behind the splash forever; and a read killed for timing out, or a resident child replaced by a settings mutation, is sent SIGTERM first and SIGKILL only after a 5-second grace, so the child can unlink its own cache refresh lock rather than leave it for the next parse's stale-pid takeover (quit stays a hard kill, its flush budget being shorter than the grace). A hydration that genuinely needs more than 15 minutes is still ended by the absolute ceiling, but it no longer starts from nothing next time: the partial cache saved along the way means successive polls converge instead of each repeating the whole scan. The app also records the resident child's pid and full command line, and reaps a serve orphaned by a previous crash on the next launch — on Windows too, where there is no `ps` and no stdin-close recourse after a crash — signalling only when that pid still runs that exact command. Separately, `codeburn serve` drains an in-flight request before exiting on stdin close, bounded at 45 seconds so a wedged request cannot turn the child into the orphan the drain prevents.
|
||||
- **The menubar's copies of your Claude and Codex credentials move out of Application Support and into the login Keychain.** Connecting a provider used to leave the copied OAuth material in `~/Library/Application Support/CodeBurn/*-credentials.v1.json`, written world-readable (0644) because macOS ignores `.completeFileProtection` outside iOS. The copy now lives in a CodeBurn-owned login-Keychain item, and the first read after upgrading migrates the old file: it is reopened with `O_NOFOLLOW`, refused if it is a symlink or not owned by you, repaired to 0600 before a single secret byte is read, written to the Keychain, read back and compared, and only then unlinked — a failed or unverified write leaves the (now 0600) file in place so a retry can still find it, and the next read retries the cleanup. Where both a Keychain item and an old file exist, the one that expires later wins before anything is removed, so an item left behind by a much older build cannot displace a fresher token. Claude's entry no longer stores a refresh token at all — the CLI owns that grant and the menubar never spends it — and any refresh token in a historical blob is dropped on read. Disconnect only reports success once the material is actually gone; if the delete fails it says so and leaves the provider connected so you can retry. Keychain reads are non-interactive and are skipped outright while the login Keychain is locked, so a background quota refresh can never raise an unlock panel. (#1037)
|
||||
- **First launch no longer asks to control System Events.** The macOS menubar registered its login item by driving System Events over AppleScript, which made macOS put up an Automation consent dialog the first time the app ran. It now registers itself through `SMAppService.mainApp`, an in-process call that needs no Automation grant; there is no AppleScript fallback, so a failure logs and leaves the login item unset rather than bringing the prompt back. The same `codeburn.loginItemRegistered` guard still limits this to the first launch, so a login item you removed by hand stays removed. (#1026)
|
||||
|
|
|
|||
|
|
@ -84,6 +84,10 @@ final class AppDelegate: NSObject, NSApplicationDelegate, NSPopoverDelegate, NSM
|
|||
private var refreshLoopHeartbeatAt: Date = .distantPast
|
||||
|
||||
func applicationWillTerminate(_ notification: Notification) {
|
||||
// Synchronously, before the actor hop: the app can exit before a
|
||||
// detached Task is ever scheduled, and a serve child that outlives us
|
||||
// is the orphan in #1117. shutdown() still runs for the tidy case.
|
||||
ServeChildRegistry.shared.reapAll()
|
||||
Task { await ServeConnection.shared.shutdown() }
|
||||
if let monitor = rightClickMonitor {
|
||||
NSEvent.removeMonitor(monitor)
|
||||
|
|
|
|||
98
mac/Sources/CodeBurnMenubar/Data/CLIWatchdog.swift
Normal file
98
mac/Sources/CodeBurnMenubar/Data/CLIWatchdog.swift
Normal file
|
|
@ -0,0 +1,98 @@
|
|||
import Foundation
|
||||
|
||||
/// Swift port of the desktop app's no-output watchdog (`app/electron/cli.ts`, #1096).
|
||||
///
|
||||
/// The CLI timeout is a NO-OUTPUT window, not a total-runtime cap: every byte the
|
||||
/// child writes on stdout or stderr restarts it. Spawns set `CODEBURN_PROGRESS=1`,
|
||||
/// under which the parser heartbeats every 10s for the whole duration of a parse
|
||||
/// (`PROGRESS_KEEPALIVE_MS` in `src/parser.ts`), so a legitimately slow cold parse
|
||||
/// on a large corpus is never killed while a genuinely wedged child still is.
|
||||
///
|
||||
/// The pure `verdict` exists so the arithmetic is testable without spawning a
|
||||
/// child and waiting minutes for a real ceiling.
|
||||
enum CLIWatchdog {
|
||||
/// Silence a live child cannot produce: 4.5x the CLI's 10s keepalive cadence.
|
||||
/// Matches `DEFAULT_TIMEOUT_MS` in app/electron/cli.ts.
|
||||
static let silenceSeconds: Double = 45
|
||||
/// Until this app has seen one successful payload the on-disk cache may be
|
||||
/// empty, and a full hydration has genuinely silent stretches before the
|
||||
/// first keepalive can be armed. Matches `DESKTOP_COLD_TIMEOUT_MS`. Finite
|
||||
/// on purpose: a child that never emits a byte still dies here, it just gets
|
||||
/// the cold budget to prove itself first.
|
||||
static let coldSilenceSeconds: Double = 10 * 60
|
||||
/// Backstop for the watchdog: a livelocked child that chatters forever
|
||||
/// without finishing is still reaped. Matches `MAX_RUNTIME_MS`.
|
||||
static let ceilingSeconds: Double = 15 * 60
|
||||
/// SIGTERM is catchable, so the CLI unlinks its own refresh lock before dying
|
||||
/// (`armSignalCleanup` in src/session-cache.ts and src/cache-refresh-lock.ts).
|
||||
/// Only a child that ignores it gets SIGKILL. Matches `KILL_GRACE_MS`.
|
||||
static let killGraceSeconds: Double = 5
|
||||
|
||||
enum Verdict: Equatable {
|
||||
case wait
|
||||
case silent
|
||||
case ceiling
|
||||
}
|
||||
|
||||
/// `now`, `startedAt` and `lastOutputAt` are seconds on one monotonic scale.
|
||||
static func verdict(now: Double,
|
||||
startedAt: Double,
|
||||
lastOutputAt: Double,
|
||||
silenceSeconds: Double) -> Verdict {
|
||||
if now - lastOutputAt >= silenceSeconds { return .silent }
|
||||
if now - startedAt >= ceilingSeconds { return .ceiling }
|
||||
return .wait
|
||||
}
|
||||
|
||||
/// The silence window for a request: the cold floor applies until this app
|
||||
/// has completed one payload, exactly as the electron client floors every
|
||||
/// request admitted before its resident child is warm.
|
||||
static func silenceWindow(warm: Bool) -> Double {
|
||||
warm ? silenceSeconds : max(silenceSeconds, coldSilenceSeconds)
|
||||
}
|
||||
|
||||
/// Progress heartbeats share stderr with real diagnostics, and every read
|
||||
/// spawn now enables them, so they must never become the error message.
|
||||
static func withoutProgressLines(_ stderr: String) -> String {
|
||||
stderr
|
||||
.split(separator: "\n", omittingEmptySubsequences: false)
|
||||
.filter { !$0.hasPrefix(progressLinePrefix) }
|
||||
.joined(separator: "\n")
|
||||
.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
}
|
||||
|
||||
/// Wire marker for CLI scan-progress lines (`PROGRESS_LINE_PREFIX`, src/parser.ts).
|
||||
static let progressLinePrefix = "CODEBURN_PROGRESS "
|
||||
|
||||
/// Environment for a spawn that should heartbeat through the watchdog.
|
||||
static func withProgressHeartbeat(_ environment: [String: String]?) -> [String: String] {
|
||||
var env = environment ?? ProcessInfo.processInfo.environment
|
||||
env["CODEBURN_PROGRESS"] = "1"
|
||||
return env
|
||||
}
|
||||
}
|
||||
|
||||
/// Monotonic last-output marker shared by a process's two drain tasks and its
|
||||
/// watchdog timer. `uptimeNanoseconds` is unaffected by wall-clock jumps.
|
||||
final class OutputActivity: @unchecked Sendable {
|
||||
private let lock = NSLock()
|
||||
private var lastOutput: Double
|
||||
|
||||
init() { lastOutput = OutputActivity.now() }
|
||||
|
||||
static func now() -> Double {
|
||||
Double(DispatchTime.now().uptimeNanoseconds) / 1_000_000_000
|
||||
}
|
||||
|
||||
func touch() {
|
||||
lock.lock()
|
||||
lastOutput = OutputActivity.now()
|
||||
lock.unlock()
|
||||
}
|
||||
|
||||
var lastOutputAt: Double {
|
||||
lock.lock()
|
||||
defer { lock.unlock() }
|
||||
return lastOutput
|
||||
}
|
||||
}
|
||||
|
|
@ -6,9 +6,16 @@ import Foundation
|
|||
/// Pipe file descriptors pinned forever.
|
||||
private let maxPayloadBytes = 20 * 1024 * 1024
|
||||
private let maxStderrBytes = 256 * 1024
|
||||
private let spawnTimeoutSeconds: UInt64 = 45
|
||||
private let maxConcurrentSpawns = 6
|
||||
|
||||
/// Whether this app run has ever completed a payload fetch. Until it has, the
|
||||
/// on-disk cache may be cold and a spawn gets the long floor; afterwards the
|
||||
/// ordinary silence window applies. Mirrors the resident client's `warmed`.
|
||||
actor SpawnWarmth {
|
||||
private(set) var warm = false
|
||||
func markWarm() { warm = true }
|
||||
}
|
||||
|
||||
enum DataClientError: Error {
|
||||
case spawn(String)
|
||||
case nonZeroExit(code: Int32, stderr: String)
|
||||
|
|
@ -62,7 +69,9 @@ struct DataClient {
|
|||
throw DataClientError.nonZeroExit(code: result.exitCode, stderr: result.stderr)
|
||||
}
|
||||
do {
|
||||
return try JSONDecoder().decode(MenubarPayload.self, from: result.stdout)
|
||||
let payload = try JSONDecoder().decode(MenubarPayload.self, from: result.stdout)
|
||||
await warmth.markWarm()
|
||||
return payload
|
||||
} catch {
|
||||
let snippet = String(decoding: result.stdout.prefix(2048), as: UTF8.self)
|
||||
throw DataClientError.decode(CLIDecodeFailure(
|
||||
|
|
@ -119,6 +128,9 @@ struct DataClient {
|
|||
/// dozens of node processes at once.
|
||||
private static let spawnLimiter = AsyncSemaphore(maxConcurrentSpawns)
|
||||
|
||||
/// Cold floor gate for the one-shot path. See `SpawnWarmth`.
|
||||
static let warmth = SpawnWarmth()
|
||||
|
||||
private static func runCLI(
|
||||
subcommand: [String],
|
||||
qualityOfService: QualityOfService = .userInitiated
|
||||
|
|
@ -135,9 +147,13 @@ struct DataClient {
|
|||
subcommand: subcommand,
|
||||
qualityOfService: qualityOfService
|
||||
)
|
||||
// Heartbeats for the no-output watchdog: a multi-minute parse
|
||||
// writes progress lines to stderr instead of going silent.
|
||||
process.environment = CLIWatchdog.withProgressHeartbeat(process.environment)
|
||||
let silenceSeconds = CLIWatchdog.silenceWindow(warm: await warmth.warm)
|
||||
return try await runProcess(
|
||||
process,
|
||||
timeoutSeconds: spawnTimeoutSeconds,
|
||||
timeoutSeconds: silenceSeconds,
|
||||
label: subcommand.joined(separator: " ")
|
||||
)
|
||||
}
|
||||
|
|
@ -188,7 +204,10 @@ struct DataClient {
|
|||
}
|
||||
|
||||
/// Runs an already-configured process to completion, draining its output and
|
||||
/// enforcing a hard timeout.
|
||||
/// enforcing the no-output watchdog: `timeoutSeconds` bounds SILENCE, not
|
||||
/// total runtime, and restarts on every byte the child produces. See
|
||||
/// `CLIWatchdog` for why (and for the absolute ceiling that still bounds a
|
||||
/// child which chatters forever).
|
||||
///
|
||||
/// CRITICAL: nothing here may block a worker thread waiting for the process.
|
||||
/// `process.waitUntilExit()` is a blocking syscall. An earlier fix moved it
|
||||
|
|
@ -200,7 +219,7 @@ struct DataClient {
|
|||
/// `process.terminationHandler`, which fires on a Foundation-managed queue and
|
||||
/// blocks nothing, so the timeout always has a free thread to fire on.
|
||||
static func runProcess(_ process: Process,
|
||||
timeoutSeconds: UInt64,
|
||||
timeoutSeconds: Double,
|
||||
label: String) async throws -> ProcessResult {
|
||||
let outPipe = Pipe()
|
||||
let errPipe = Pipe()
|
||||
|
|
@ -218,14 +237,36 @@ struct DataClient {
|
|||
throw DataClientError.spawn(error.localizedDescription)
|
||||
}
|
||||
|
||||
// Silence watchdog, not a runtime cap: `activity` is touched by both
|
||||
// drains, so the window restarts on every byte (progress keepalives
|
||||
// included) and only a genuinely mute child is killed.
|
||||
let activity = OutputActivity()
|
||||
let startedAt = OutputActivity.now()
|
||||
let timeoutTimer = DispatchSource.makeTimerSource(queue: DispatchQueue.global(qos: .utility))
|
||||
timeoutTimer.schedule(deadline: .now() + .seconds(Int(timeoutSeconds)))
|
||||
timeoutTimer.schedule(deadline: .now() + .seconds(1), repeating: .seconds(1))
|
||||
timeoutTimer.setEventHandler {
|
||||
if process.isRunning {
|
||||
NSLog("CodeBurn: CLI subprocess timed out after %llus for %@ — terminating",
|
||||
guard process.isRunning else { return }
|
||||
let verdict = CLIWatchdog.verdict(
|
||||
now: OutputActivity.now(),
|
||||
startedAt: startedAt,
|
||||
lastOutputAt: activity.lastOutputAt,
|
||||
silenceSeconds: timeoutSeconds
|
||||
)
|
||||
switch verdict {
|
||||
case .wait:
|
||||
return
|
||||
case .silent:
|
||||
NSLog("CodeBurn: CLI subprocess produced no output for %.0fs for %@ — terminating",
|
||||
timeoutSeconds, label)
|
||||
terminateWithEscalation(process)
|
||||
case .ceiling:
|
||||
NSLog("CodeBurn: CLI subprocess exceeded %.0fs for %@ — terminating",
|
||||
CLIWatchdog.ceilingSeconds, label)
|
||||
}
|
||||
// One verdict per process: the escalation owns the kill from here,
|
||||
// and a repeating tick would re-send SIGTERM every second for the
|
||||
// whole grace. Cancelling from inside the handler is supported.
|
||||
timeoutTimer.cancel()
|
||||
terminateWithEscalation(process)
|
||||
}
|
||||
timeoutTimer.resume()
|
||||
defer { timeoutTimer.cancel() }
|
||||
|
|
@ -233,8 +274,8 @@ struct DataClient {
|
|||
let outHandle = outPipe.fileHandleForReading
|
||||
let errHandle = errPipe.fileHandleForReading
|
||||
let (out, err) = await withTaskCancellationHandler {
|
||||
async let stdoutData = drain(outHandle, limit: maxPayloadBytes)
|
||||
async let stderrData = drain(errHandle, limit: maxStderrBytes)
|
||||
async let stdoutData = drain(outHandle, limit: maxPayloadBytes, activity: activity)
|
||||
async let stderrData = drain(errHandle, limit: maxStderrBytes, activity: activity)
|
||||
return await (stdoutData, stderrData)
|
||||
} onCancel: {
|
||||
terminateWithEscalation(process)
|
||||
|
|
@ -249,20 +290,23 @@ struct DataClient {
|
|||
throw DataClientError.outputTooLarge
|
||||
}
|
||||
|
||||
let stderrString = String(data: err, encoding: .utf8) ?? ""
|
||||
let stderrString = CLIWatchdog.withoutProgressLines(String(data: err, encoding: .utf8) ?? "")
|
||||
return ProcessResult(stdout: out, stderr: stderrString, exitCode: process.terminationStatus)
|
||||
}
|
||||
|
||||
/// Ask a child to exit, then insist. The grace exists so the CLI's signal
|
||||
/// cleanup can unlink the cache refresh lock it may be holding before it
|
||||
/// dies; only a child that ignores SIGTERM is SIGKILLed.
|
||||
private static func terminateWithEscalation(_ process: Process) {
|
||||
guard process.isRunning else { return }
|
||||
process.terminate()
|
||||
let pid = process.processIdentifier
|
||||
DispatchQueue.global(qos: .utility).asyncAfter(deadline: .now() + 0.5) {
|
||||
DispatchQueue.global(qos: .utility).asyncAfter(deadline: .now() + CLIWatchdog.killGraceSeconds) {
|
||||
if process.isRunning { kill(pid, SIGKILL) }
|
||||
}
|
||||
}
|
||||
|
||||
private static func drain(_ handle: FileHandle, limit: Int) async -> Data {
|
||||
private static func drain(_ handle: FileHandle, limit: Int, activity: OutputActivity) async -> Data {
|
||||
let fd = handle.fileDescriptor
|
||||
let flags = Darwin.fcntl(fd, F_GETFL)
|
||||
if flags >= 0 {
|
||||
|
|
@ -280,6 +324,7 @@ struct DataClient {
|
|||
Darwin.read(fd, ptr.baseAddress!, toRead)
|
||||
}
|
||||
if n > 0 {
|
||||
activity.touch()
|
||||
buffer.append(contentsOf: chunk.prefix(n))
|
||||
} else if n == 0 {
|
||||
break
|
||||
|
|
|
|||
|
|
@ -36,12 +36,20 @@ actor ServeConnection {
|
|||
|
||||
private var process: Process?
|
||||
private var stdinHandle: FileHandle?
|
||||
/// Permanent refusal (explicit shutdown, or a binary that cannot be spawned
|
||||
/// at all). Distinct from a spent death budget, which recovers with time.
|
||||
private var disabled = false
|
||||
private var lastDeathAt: Date?
|
||||
private var nextId = 1
|
||||
private var nextRequestToken = 1
|
||||
private var queuedRequests: [QueuedRequest] = []
|
||||
private var activeRequest: ActiveRequest?
|
||||
private var pending: [Int: CheckedContinuation<Data, Error>] = [:]
|
||||
private var requestTimeouts: [Int: Task<Void, Never>] = [:]
|
||||
private var requestCeilings: [Int: Task<Void, Never>] = [:]
|
||||
/// Silence window of the in-flight request, so any frame carrying its id can
|
||||
/// re-arm the same budget. See `CLIWatchdog`.
|
||||
private var requestSilence: [Int: UInt64] = [:]
|
||||
private var timeoutOwners: [Int: Process] = [:]
|
||||
private var responseBytes: [Int: Int] = [:]
|
||||
private var deaths = 0
|
||||
|
|
@ -57,9 +65,18 @@ actor ServeConnection {
|
|||
private static let maxDeaths = 3
|
||||
static let maxResponseBytes = 16 * 1024 * 1024
|
||||
private static let stdoutReadChunkBytes = 64 * 1024
|
||||
private static let terminationGraceNanoseconds: UInt64 = 1_000_000_000
|
||||
private static let coldRequestTimeoutNanoseconds: UInt64 = 10 * 60 * 1_000_000_000
|
||||
private static let warmRequestTimeoutNanoseconds: UInt64 = 60 * 1_000_000_000
|
||||
private static let terminationGraceNanoseconds = nanoseconds(CLIWatchdog.killGraceSeconds)
|
||||
/// Absolute backstop per request: a child that heartbeats forever without
|
||||
/// ever answering is still reaped.
|
||||
private static let requestCeilingNanoseconds = nanoseconds(CLIWatchdog.ceilingSeconds)
|
||||
/// The death budget is spent by a transient cause as easily as a permanent
|
||||
/// one (a cold-cache pile-up spends all three in a minute). Let a later tick
|
||||
/// try again instead of leaving the resident dead for the whole app run.
|
||||
private static let deathBudgetResetSeconds: TimeInterval = 300
|
||||
|
||||
private static func nanoseconds(_ seconds: Double) -> UInt64 {
|
||||
UInt64(seconds * 1_000_000_000)
|
||||
}
|
||||
|
||||
struct ServeUnavailable: Error {}
|
||||
enum FailureReason: Sendable, Equatable {
|
||||
|
|
@ -99,19 +116,37 @@ actor ServeConnection {
|
|||
|
||||
/// Kick the child off (idempotent). Called from app startup and again by
|
||||
/// the first request in case the startup task has not run yet.
|
||||
///
|
||||
/// Re-entry: a spent death budget is a cooldown, not a life sentence. The
|
||||
/// budget's job is to stop a crash loop from spawning a child per tick; once
|
||||
/// `deathBudgetResetSeconds` have passed with no new death, the next tick
|
||||
/// gets a fresh budget instead of leaving every fetch on the one-shot path
|
||||
/// for the rest of the app run.
|
||||
func ensureStarted() {
|
||||
guard process == nil, deaths < Self.maxDeaths else { return }
|
||||
guard !disabled, process == nil else { return }
|
||||
if deaths >= Self.maxDeaths {
|
||||
guard let lastDeathAt,
|
||||
Date().timeIntervalSince(lastDeathAt) >= Self.deathBudgetResetSeconds else { return }
|
||||
deaths = 0
|
||||
}
|
||||
// Before adding a child, clear the one a crashed previous run (or an
|
||||
// earlier generation of this one) left behind. Idempotent: the recorded
|
||||
// pid is only signalled if it is still that exact serve command.
|
||||
ServeOrphanReaper.reap()
|
||||
// This single resident serves both background and user-visible status
|
||||
// requests. Its cold hydration replaces the old interactive one-shot,
|
||||
// so keep the child at the same user-initiated QoS as visible fetches.
|
||||
let child = makeProcess(["serve", "--stdio"], .userInitiated)
|
||||
// Progress frames are what re-arm this connection's no-output watchdog
|
||||
// during a cold hydration; without them serve is silent for minutes.
|
||||
child.environment = CLIWatchdog.withProgressHeartbeat(child.environment)
|
||||
let stdinPipe = Pipe()
|
||||
let stdinWriter = stdinPipe.fileHandleForWriting
|
||||
// Suppress SIGPIPE only for this connection's write end. A process-wide
|
||||
// SIG_IGN leaks into unrelated libraries and children; F_SETNOSIGPIPE
|
||||
// keeps a closed child stdin on the normal throwable EPIPE path.
|
||||
guard Darwin.fcntl(stdinWriter.fileDescriptor, F_SETNOSIGPIPE, 1) == 0 else {
|
||||
deaths = Self.maxDeaths
|
||||
disabled = true
|
||||
return
|
||||
}
|
||||
let stdoutPipe = Pipe()
|
||||
|
|
@ -122,11 +157,18 @@ actor ServeConnection {
|
|||
do {
|
||||
try child.run()
|
||||
} catch {
|
||||
deaths = Self.maxDeaths // spawn path can't produce the binary either better than makeProcess did
|
||||
disabled = true // spawn path can't produce the binary either better than makeProcess did
|
||||
return
|
||||
}
|
||||
process = child
|
||||
stdinHandle = stdinWriter
|
||||
ServeChildRegistry.shared.add(child)
|
||||
// Record the argv WITHOUT the `/usr/bin/env --` prefix: that is the part
|
||||
// exec rewrites away before `ps` can see it. See serveCommandMatches.
|
||||
ServeOrphanReaper.record(
|
||||
pid: child.processIdentifier,
|
||||
command: (child.arguments ?? []).drop(while: { $0 == "--" }).joined(separator: " ")
|
||||
)
|
||||
let generation = ObjectIdentifier(child)
|
||||
// One blocking reader owns this generation's stdout. It never reads a
|
||||
// second bounded chunk until the actor has consumed the first, giving
|
||||
|
|
@ -177,16 +219,24 @@ actor ServeConnection {
|
|||
}
|
||||
|
||||
func shutdown() {
|
||||
disabled = true
|
||||
deaths = Self.maxDeaths
|
||||
process?.terminate()
|
||||
for task in terminationTasks.values { task.cancel() }
|
||||
terminationTasks.removeAll()
|
||||
cancelAllTimeouts()
|
||||
failAllRequests()
|
||||
try? stdinHandle?.close()
|
||||
process = nil
|
||||
stdinHandle = nil
|
||||
buffer = Data()
|
||||
receivedTerminalResponse = false
|
||||
// Deliberately harder than every other kill path, and covering RETIRED
|
||||
// generations too: quit has no budget to wait a SIGTERM grace out, and
|
||||
// the grace task dies with the app, so a child that defers SIGTERM (a
|
||||
// node process mid-synchronous-parse does) would outlive us. A refresh
|
||||
// lock left behind by SIGKILL self-heals through the next parse's
|
||||
// dead-pid takeover (src/cache-refresh-lock.ts).
|
||||
ServeChildRegistry.shared.reapAll()
|
||||
}
|
||||
|
||||
// MARK: - internals
|
||||
|
|
@ -226,10 +276,9 @@ actor ServeConnection {
|
|||
|
||||
// Select and arm the timeout only when this request becomes the sole
|
||||
// protocol request in flight. A queued request must not spend its own
|
||||
// budget while its predecessor is still hydrating or draining.
|
||||
let timeoutNanoseconds = receivedTerminalResponse
|
||||
? Self.warmRequestTimeoutNanoseconds
|
||||
: Self.coldRequestTimeoutNanoseconds
|
||||
// budget while its predecessor is still hydrating or draining. The
|
||||
// budget bounds SILENCE: every frame carrying this id restarts it.
|
||||
let timeoutNanoseconds = Self.nanoseconds(CLIWatchdog.silenceWindow(warm: receivedTerminalResponse))
|
||||
activeRequest = ActiveRequest(
|
||||
token: request.token,
|
||||
id: id,
|
||||
|
|
@ -271,8 +320,26 @@ actor ServeConnection {
|
|||
}
|
||||
|
||||
private func armTimeout(id: Int, child: Process, nanoseconds: UInt64) {
|
||||
let sleep = timeoutSleep
|
||||
timeoutOwners[id] = child
|
||||
requestSilence[id] = nanoseconds
|
||||
armSilenceTimer(id: id, nanoseconds: nanoseconds)
|
||||
// Deliberately a real sleep, not `timeoutSleep`: the ceiling is a
|
||||
// 15-minute backstop behind the injected silence budget, and routing it
|
||||
// through the same seam would make every test's timeout ledger carry it.
|
||||
// Its arithmetic is covered by CLIWatchdog.verdict.
|
||||
requestCeilings[id] = Task.detached { [weak self] in
|
||||
do {
|
||||
try await Task<Never, Never>.sleep(nanoseconds: Self.requestCeilingNanoseconds)
|
||||
} catch {
|
||||
return
|
||||
}
|
||||
await self?.requestTimedOut(id: id)
|
||||
}
|
||||
}
|
||||
|
||||
private func armSilenceTimer(id: Int, nanoseconds: UInt64) {
|
||||
let sleep = timeoutSleep
|
||||
requestTimeouts[id]?.cancel()
|
||||
requestTimeouts[id] = Task.detached { [weak self] in
|
||||
do {
|
||||
try await sleep(nanoseconds)
|
||||
|
|
@ -283,6 +350,14 @@ actor ServeConnection {
|
|||
}
|
||||
}
|
||||
|
||||
/// Any frame carrying this request's id is proof of life: restart its
|
||||
/// watchdog so a long cold parse that heartbeats progress is never killed
|
||||
/// mid-flight. The absolute ceiling is untouched.
|
||||
private func touchRequest(id: Int) {
|
||||
guard let nanoseconds = requestSilence[id] else { return }
|
||||
armSilenceTimer(id: id, nanoseconds: nanoseconds)
|
||||
}
|
||||
|
||||
private func requestTimedOut(id: Int) {
|
||||
guard let child = timeoutOwners.removeValue(forKey: id) else { return }
|
||||
requestTimeouts.removeValue(forKey: id)
|
||||
|
|
@ -302,11 +377,7 @@ actor ServeConnection {
|
|||
// reach EOF (for example, a stuck child can ignore SIGTERM or a
|
||||
// descendant can retain the pipe), so waiting for the reader would also
|
||||
// spend every queued caller's timeout before it can even be admitted.
|
||||
process = nil
|
||||
stdinHandle = nil
|
||||
buffer = Data()
|
||||
receivedTerminalResponse = false
|
||||
deaths += 1
|
||||
retireCurrentGeneration()
|
||||
if activeRequest?.id == id { activeRequest = nil }
|
||||
cancelTimeouts(ownedBy: child)
|
||||
terminateTimedOutChild(child)
|
||||
|
|
@ -319,6 +390,8 @@ actor ServeConnection {
|
|||
private func cancelTimeout(id: Int) {
|
||||
timeoutOwners.removeValue(forKey: id)
|
||||
requestTimeouts.removeValue(forKey: id)?.cancel()
|
||||
requestCeilings.removeValue(forKey: id)?.cancel()
|
||||
requestSilence.removeValue(forKey: id)
|
||||
responseBytes.removeValue(forKey: id)
|
||||
}
|
||||
|
||||
|
|
@ -330,14 +403,34 @@ actor ServeConnection {
|
|||
private func cancelAllTimeouts() {
|
||||
for task in requestTimeouts.values { task.cancel() }
|
||||
requestTimeouts.removeAll()
|
||||
for task in requestCeilings.values { task.cancel() }
|
||||
requestCeilings.removeAll()
|
||||
requestSilence.removeAll()
|
||||
timeoutOwners.removeAll()
|
||||
responseBytes.removeAll()
|
||||
}
|
||||
|
||||
/// Detach the current generation. Closing our end of its stdin is the app's
|
||||
/// half of "stdin closing ends the server loop": dropping the FileHandle is
|
||||
/// NOT enough, because the Pipe stays alive inside `child.standardInput` for
|
||||
/// as long as this generation's reader or termination task holds the
|
||||
/// Process, so the write end stays open and the retired child never sees
|
||||
/// EOF. That is how a retired-but-alive serve child becomes an orphan.
|
||||
private func retireCurrentGeneration() {
|
||||
try? stdinHandle?.close()
|
||||
process = nil
|
||||
stdinHandle = nil
|
||||
buffer = Data()
|
||||
receivedTerminalResponse = false
|
||||
deaths += 1
|
||||
lastDeathAt = Date()
|
||||
}
|
||||
|
||||
private func outputStreamFinished(for child: Process) {
|
||||
outputTasks.removeValue(forKey: ObjectIdentifier(child))
|
||||
if !child.isRunning {
|
||||
terminationTasks.removeValue(forKey: ObjectIdentifier(child))?.cancel()
|
||||
ServeChildRegistry.shared.remove(child)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -363,17 +456,23 @@ actor ServeConnection {
|
|||
|
||||
private func forceKillAfterGrace(_ child: Process) {
|
||||
terminationTasks.removeValue(forKey: ObjectIdentifier(child))
|
||||
guard child.isRunning else { return }
|
||||
guard child.isRunning else {
|
||||
ServeChildRegistry.shared.remove(child)
|
||||
return
|
||||
}
|
||||
_ = Darwin.kill(child.processIdentifier, SIGKILL)
|
||||
ServeChildRegistry.shared.remove(child)
|
||||
}
|
||||
|
||||
private func outputStreamEnded(for child: Process) {
|
||||
guard process === child else { return }
|
||||
// EOF/read failure is a transport death even if the process has not
|
||||
// reaped yet. Terminate that exact generation so a child which closed
|
||||
// stdout cannot survive after the actor starts its replacement.
|
||||
if child.isRunning { child.terminate() }
|
||||
// stdout cannot survive after the actor starts its replacement, and
|
||||
// escalate: SIGTERM alone is deferred by a node child that is inside a
|
||||
// synchronous parse when it arrives.
|
||||
childDied(child)
|
||||
terminateTimedOutChild(child)
|
||||
}
|
||||
|
||||
// Internal so the generation guard can be exercised deterministically by
|
||||
|
|
@ -420,6 +519,10 @@ actor ServeConnection {
|
|||
// Menubar has no progress UI, but must leave the request pending
|
||||
// until the terminal response arrives if such a frame is emitted.
|
||||
if let progress = object["progress"] as? String {
|
||||
// Proof of life for a request still being worked on: restart its
|
||||
// silence window. A terminal frame does not need this - it
|
||||
// cancels the timers outright a few lines down.
|
||||
touchRequest(id: id)
|
||||
guard accountResponseBytes(Data(progress.utf8).count, id: id, child: child) else { return }
|
||||
return
|
||||
}
|
||||
|
|
@ -467,26 +570,18 @@ actor ServeConnection {
|
|||
// Detach this exact generation before terminating it. Its eventual exit
|
||||
// and any already-scheduled stdout callbacks are then stale and cannot
|
||||
// consume a second death or corrupt a replacement generation.
|
||||
process = nil
|
||||
stdinHandle = nil
|
||||
buffer = Data()
|
||||
receivedTerminalResponse = false
|
||||
deaths += 1
|
||||
retireCurrentGeneration()
|
||||
cancelTimeouts(ownedBy: child)
|
||||
failAllRequests(error: ServeRequestFailed(
|
||||
message: "serve output exceeded \(responseLimitBytes) bytes",
|
||||
reason: .outputTooLarge
|
||||
))
|
||||
if child.isRunning { child.terminate() }
|
||||
terminateTimedOutChild(child)
|
||||
}
|
||||
|
||||
private func childDied(_ child: Process) {
|
||||
guard process === child else { return }
|
||||
process = nil
|
||||
stdinHandle = nil
|
||||
buffer.removeAll()
|
||||
receivedTerminalResponse = false
|
||||
deaths += 1
|
||||
retireCurrentGeneration()
|
||||
cancelTimeouts(ownedBy: child)
|
||||
if let activeRequest, activeRequest.child === child {
|
||||
if let continuation = pending.removeValue(forKey: activeRequest.id) {
|
||||
|
|
@ -527,3 +622,116 @@ actor ServeConnection {
|
|||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Best-effort reap of a serve child orphaned by a previous run: a crash leaves
|
||||
/// no one to close the child's stdin and no `applicationWillTerminate` to reap
|
||||
/// it. Port of `reapOrphanServe` in app/electron/cli.ts (#1096).
|
||||
enum ServeOrphanReaper {
|
||||
static func pidFileURL() -> URL {
|
||||
URL(fileURLWithPath: CodeBurnCacheDirectory.resolve())
|
||||
.appendingPathComponent("menubar-serve.pid", isDirectory: false)
|
||||
}
|
||||
|
||||
static func record(pid: pid_t, command: String) {
|
||||
let url = pidFileURL()
|
||||
try? FileManager.default.createDirectory(
|
||||
at: url.deletingLastPathComponent(),
|
||||
withIntermediateDirectories: true
|
||||
)
|
||||
let body = ["pid": Int(pid), "cmd": command] as [String: Any]
|
||||
guard let data = try? JSONSerialization.data(withJSONObject: body) else { return }
|
||||
try? data.write(to: url) // reaping is best-effort
|
||||
}
|
||||
|
||||
/// Reads the pid recorded by the previous run and — because pids are
|
||||
/// recycled — signals it only after `ps` confirms the process is still that
|
||||
/// exact serve child. SIGTERM, never SIGKILL: the orphan may be holding the
|
||||
/// cache refresh lock and can release it on the way out.
|
||||
static func reap() {
|
||||
let url = pidFileURL()
|
||||
defer { try? FileManager.default.removeItem(at: url) }
|
||||
guard let data = try? Data(contentsOf: url),
|
||||
let record = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
|
||||
let pid = record["pid"] as? Int,
|
||||
let cmd = record["cmd"] as? String,
|
||||
pid > 1, pid != Int(ProcessInfo.processInfo.processIdentifier),
|
||||
!cmd.isEmpty,
|
||||
serveCommandMatches(recorded: cmd, observed: commandLine(of: pid_t(pid)))
|
||||
else { return }
|
||||
_ = Darwin.kill(pid_t(pid), SIGTERM)
|
||||
}
|
||||
|
||||
/// Suffix match on the full recorded argv, not a keyword sniff: any looser
|
||||
/// test signals whatever unrelated process inherited the pid.
|
||||
///
|
||||
/// Electron can compare exactly because it spawns its own binary. We spawn
|
||||
/// through `/usr/bin/env`, and the CLI is a shebang script, so by the time
|
||||
/// `ps` sees the process the argv has been rewritten twice: `env -- <cli>
|
||||
/// serve --stdio` becomes `node <cli> serve --stdio`. The interpreter
|
||||
/// prefix is the only part that varies, so the recorded `<cli> serve
|
||||
/// --stdio` has to match as a suffix rather than in full.
|
||||
static func serveCommandMatches(recorded: String, observed: String?) -> Bool {
|
||||
guard let observed else { return false }
|
||||
let normalize = { (value: String) -> String in
|
||||
value.split(whereSeparator: { $0 == " " || $0 == "\t" || $0 == "\n" || $0 == "\"" })
|
||||
.joined(separator: " ")
|
||||
}
|
||||
let wanted = normalize(recorded)
|
||||
return !wanted.isEmpty && normalize(observed).hasSuffix(wanted)
|
||||
}
|
||||
|
||||
/// `-ww` defeats ps's width truncation; the pid is an integer before it is
|
||||
/// passed, and the call never goes through a shell.
|
||||
private static func commandLine(of pid: pid_t) -> String? {
|
||||
let process = Process()
|
||||
process.executableURL = URL(fileURLWithPath: "/bin/ps")
|
||||
process.arguments = ["-ww", "-o", "command=", "-p", String(pid)]
|
||||
let pipe = Pipe()
|
||||
process.standardOutput = pipe
|
||||
process.standardError = FileHandle.nullDevice
|
||||
guard (try? process.run()) != nil else { return nil }
|
||||
// readToEnd returns at EOF, which is this short-lived child exiting; no
|
||||
// waitUntilExit, which would park the calling thread (see DataClient).
|
||||
let data = (try? pipe.fileHandleForReading.readToEnd()) ?? nil
|
||||
let text = String(data: data ?? Data(), encoding: .utf8)?
|
||||
.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
|
||||
return text.isEmpty ? nil : text
|
||||
}
|
||||
}
|
||||
|
||||
/// Every serve child this app run has started and not yet seen exit, readable
|
||||
/// WITHOUT an actor hop.
|
||||
///
|
||||
/// `applicationWillTerminate` runs `Task { await shutdown() }`, and the app can
|
||||
/// exit before that task is ever scheduled — so the actor's own reaping is not
|
||||
/// reachable on the path that matters most. This registry is, and a Process that
|
||||
/// still reports `isRunning` has not been waited on, so its pid is still ours
|
||||
/// and cannot have been recycled under us.
|
||||
final class ServeChildRegistry: @unchecked Sendable {
|
||||
static let shared = ServeChildRegistry()
|
||||
|
||||
private let lock = NSLock()
|
||||
private var children: [ObjectIdentifier: Process] = [:]
|
||||
|
||||
func add(_ child: Process) {
|
||||
lock.lock()
|
||||
children[ObjectIdentifier(child)] = child
|
||||
lock.unlock()
|
||||
}
|
||||
|
||||
func remove(_ child: Process) {
|
||||
lock.lock()
|
||||
children.removeValue(forKey: ObjectIdentifier(child))
|
||||
lock.unlock()
|
||||
}
|
||||
|
||||
func reapAll() {
|
||||
lock.lock()
|
||||
let all = Array(children.values)
|
||||
children.removeAll()
|
||||
lock.unlock()
|
||||
for child in all where child.isRunning {
|
||||
_ = Darwin.kill(child.processIdentifier, SIGKILL)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
79
mac/Tests/CodeBurnMenubarTests/CLIWatchdogTests.swift
Normal file
79
mac/Tests/CodeBurnMenubarTests/CLIWatchdogTests.swift
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
import Foundation
|
||||
import Testing
|
||||
@testable import CodeBurnMenubar
|
||||
|
||||
@Suite("CLI watchdog")
|
||||
struct CLIWatchdogTests {
|
||||
@Test("silence is measured from the last output, not from the start")
|
||||
func silenceIsMeasuredFromLastOutput() {
|
||||
// Ten minutes in, still chattering every second: not silent.
|
||||
let verdict = CLIWatchdog.verdict(now: 600, startedAt: 0, lastOutputAt: 599, silenceSeconds: 45)
|
||||
#expect(verdict == .wait)
|
||||
}
|
||||
|
||||
@Test("a mute child is killed after the silence window")
|
||||
func muteChildIsKilled() {
|
||||
#expect(CLIWatchdog.verdict(now: 44, startedAt: 0, lastOutputAt: 0, silenceSeconds: 45) == .wait)
|
||||
#expect(CLIWatchdog.verdict(now: 45, startedAt: 0, lastOutputAt: 0, silenceSeconds: 45) == .silent)
|
||||
}
|
||||
|
||||
/// The cold floor must not become a licence to live forever: a child that
|
||||
/// never emits a byte is still killed, it just gets the cold budget first.
|
||||
@Test("the cold floor still ends in a kill for a child that never speaks")
|
||||
func coldFloorStillEndsInAKill() {
|
||||
let window = CLIWatchdog.silenceWindow(warm: false)
|
||||
#expect(window == CLIWatchdog.coldSilenceSeconds)
|
||||
#expect(CLIWatchdog.verdict(now: window - 1, startedAt: 0, lastOutputAt: 0, silenceSeconds: window) == .wait)
|
||||
#expect(CLIWatchdog.verdict(now: window, startedAt: 0, lastOutputAt: 0, silenceSeconds: window) == .silent)
|
||||
}
|
||||
|
||||
@Test("a child that chatters forever is still reaped at the ceiling")
|
||||
func chatteringChildHitsTheCeiling() {
|
||||
let now = CLIWatchdog.ceilingSeconds + 1
|
||||
#expect(CLIWatchdog.verdict(now: now, startedAt: 0, lastOutputAt: now - 1, silenceSeconds: 45) == .ceiling)
|
||||
}
|
||||
|
||||
@Test("a warm request uses the short silence window")
|
||||
func warmRequestUsesShortWindow() {
|
||||
#expect(CLIWatchdog.silenceWindow(warm: true) == CLIWatchdog.silenceSeconds)
|
||||
#expect(CLIWatchdog.silenceSeconds < CLIWatchdog.coldSilenceSeconds)
|
||||
#expect(CLIWatchdog.coldSilenceSeconds < CLIWatchdog.ceilingSeconds)
|
||||
}
|
||||
|
||||
@Test("progress heartbeats never become the error message")
|
||||
func progressLinesAreStrippedFromStderr() {
|
||||
let stderr = """
|
||||
CODEBURN_PROGRESS {"kind":"keepalive"}
|
||||
warn: something real
|
||||
CODEBURN_PROGRESS {"kind":"tick"}
|
||||
"""
|
||||
#expect(CLIWatchdog.withoutProgressLines(stderr) == "warn: something real")
|
||||
#expect(CLIWatchdog.withoutProgressLines("CODEBURN_PROGRESS {\"kind\":\"keepalive\"}") == "")
|
||||
}
|
||||
|
||||
@Test("spawn env enables the CLI keepalive the watchdog listens for")
|
||||
func spawnEnvEnablesKeepalive() {
|
||||
let env = CLIWatchdog.withProgressHeartbeat(["PATH": "/usr/bin"])
|
||||
#expect(env["CODEBURN_PROGRESS"] == "1")
|
||||
#expect(env["PATH"] == "/usr/bin")
|
||||
}
|
||||
|
||||
@Test("orphan reap only signals the exact recorded serve command")
|
||||
func orphanReapMatchesExactCommand() {
|
||||
// A shebang exec rewrites argv[0], so the recorded argv matches as a suffix.
|
||||
#expect(ServeOrphanReaper.serveCommandMatches(
|
||||
recorded: "/opt/homebrew/bin/codeburn serve --stdio",
|
||||
observed: "node /opt/homebrew/bin/codeburn serve --stdio"
|
||||
))
|
||||
// A recycled pid running something else is never signalled.
|
||||
#expect(!ServeOrphanReaper.serveCommandMatches(
|
||||
recorded: "/opt/homebrew/bin/codeburn serve --stdio",
|
||||
observed: "node /opt/homebrew/bin/codeburn status"
|
||||
))
|
||||
#expect(!ServeOrphanReaper.serveCommandMatches(
|
||||
recorded: "/opt/homebrew/bin/codeburn serve --stdio",
|
||||
observed: nil
|
||||
))
|
||||
#expect(!ServeOrphanReaper.serveCommandMatches(recorded: "", observed: "anything"))
|
||||
}
|
||||
}
|
||||
|
|
@ -195,6 +195,33 @@ struct DataClientProcessTests {
|
|||
"every concurrent process should exit 0 via the terminationHandler wait path")
|
||||
}
|
||||
|
||||
/// #1117: the window bounds SILENCE, not runtime. A child that keeps talking
|
||||
/// past its window runs to completion; the old fixed timeout killed it.
|
||||
@Test("a chattering child outlives its window", .timeLimit(.minutes(1)))
|
||||
func chatteringChildOutlivesItsWindow() async throws {
|
||||
let process = Process()
|
||||
process.executableURL = URL(fileURLWithPath: "/bin/sh")
|
||||
process.arguments = ["-c", "for i in 1 2 3 4 5 6; do echo tick; sleep 0.4; done"]
|
||||
let result = try await DataClient.runProcess(process, timeoutSeconds: 1, label: "chatty")
|
||||
#expect(result.exitCode == 0, "a child emitting output every 0.4s must survive a 1s silence window")
|
||||
}
|
||||
|
||||
/// The CLI's keepalive goes to STDERR, so stderr has to re-arm the watchdog
|
||||
/// too - and must not then turn up as the error message.
|
||||
@Test("stderr keepalives keep a stdout-silent child alive and stay out of the error text",
|
||||
.timeLimit(.minutes(1)))
|
||||
func stderrKeepalivesCountAsOutput() async throws {
|
||||
let process = Process()
|
||||
process.executableURL = URL(fileURLWithPath: "/bin/sh")
|
||||
process.arguments = [
|
||||
"-c",
|
||||
#"for i in 1 2 3 4 5 6; do printf 'CODEBURN_PROGRESS {"kind":"keepalive"}\n' >&2; sleep 0.4; done; exit 3"#,
|
||||
]
|
||||
let result = try await DataClient.runProcess(process, timeoutSeconds: 1, label: "keepalive")
|
||||
#expect(result.exitCode == 3, "stderr keepalives must restart the silence window")
|
||||
#expect(result.stderr == "", "progress lines must never become the error message")
|
||||
}
|
||||
|
||||
/// The async semaphore never lets more than its count run concurrently.
|
||||
@Test("async semaphore caps concurrency")
|
||||
func asyncSemaphoreCapsConcurrency() async {
|
||||
|
|
|
|||
|
|
@ -5,8 +5,8 @@ import Testing
|
|||
|
||||
private let ignoredSIGPIPEHandlerBits = unsafeBitCast(SIG_IGN, to: UInt.self)
|
||||
private let coldTimeoutNanoseconds: UInt64 = 10 * 60 * 1_000_000_000
|
||||
private let warmTimeoutNanoseconds: UInt64 = 60 * 1_000_000_000
|
||||
private let terminationGraceNanoseconds: UInt64 = 1_000_000_000
|
||||
private let warmTimeoutNanoseconds: UInt64 = 45 * 1_000_000_000
|
||||
private let terminationGraceNanoseconds: UInt64 = 5 * 1_000_000_000
|
||||
|
||||
private func currentSIGPIPEHandlerBits() -> UInt {
|
||||
var action = sigaction()
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
import { createHash, randomBytes } from 'crypto'
|
||||
import { existsSync } from 'fs'
|
||||
import { existsSync, readFileSync, unlinkSync } from 'fs'
|
||||
import { mkdir, open, readFile, stat, unlink, utimes, writeFile } from 'fs/promises'
|
||||
import { join } from 'path'
|
||||
|
||||
|
|
@ -9,7 +9,13 @@ const LOCK_FILE = 'session-refresh.lock'
|
|||
const TAKEOVER_FILE = `${LOCK_FILE}.takeover`
|
||||
const DEFAULT_HEARTBEAT_MS = 10_000
|
||||
const DEFAULT_STALE_MS = 90_000
|
||||
const DEFAULT_WAIT_MS = 30_000
|
||||
// A waiter that gives up before the stale gate opens can NEVER recover an
|
||||
// abandoned lock, so an abandoned lock livelocks every later process: each one
|
||||
// burns its whole wait, times out, serves read-only, and the leftover survives
|
||||
// (#1117). The default is therefore derived from staleMs rather than fixed, so
|
||||
// the two can never drift back out of order. The common abandoned case does not
|
||||
// wait this long anyway - a dead holder's pid is detected on the first poll.
|
||||
const DEFAULT_WAIT_MARGIN_MS = 30_000
|
||||
const DEFAULT_POLL_MS = 100
|
||||
const WINDOWS_RETRIES = 3
|
||||
|
||||
|
|
@ -51,6 +57,51 @@ function delay(ms: number): Promise<void> {
|
|||
return new Promise(resolve => { setTimeout(resolve, ms) })
|
||||
}
|
||||
|
||||
// Mirrors session-cache.ts's hydrating.lock probe. Our own pid never counts as a
|
||||
// foreign holder. EPERM means the pid exists but belongs to another user - still
|
||||
// alive. Windows supports signal 0 as an existence test the same way.
|
||||
//
|
||||
// A false "alive" (the holder died and an unrelated process inherited its pid)
|
||||
// only delays recovery to the age gate. A false "dead" would be the dangerous
|
||||
// direction, and needs the lock file to have been written by a process on
|
||||
// another host - i.e. a cache dir on a network share, which nothing supports.
|
||||
function pidLooksAlive(pid: number): boolean {
|
||||
if (!Number.isInteger(pid) || pid <= 0 || pid === process.pid) return false
|
||||
try { process.kill(pid, 0); return true }
|
||||
catch (err) { return (err as NodeJS.ErrnoException).code === 'EPERM' }
|
||||
}
|
||||
|
||||
// The path of the lock this process currently owns, for the signal handler.
|
||||
// Single-flight guarantees at most one owned lock per process at a time.
|
||||
let ownedLockPath: string | null = null
|
||||
|
||||
// Synchronous variant for the signal path: a handler can't await, so read +
|
||||
// unlink synchronously. Only unlinks a lock we actually own.
|
||||
function removeOurLockSync(): void {
|
||||
if (!ownedLockPath) return
|
||||
try {
|
||||
const parsed = JSON.parse(readFileSync(ownedLockPath, 'utf-8')) as Partial<LockRecord>
|
||||
if (parsed?.pid === process.pid) unlinkSync(ownedLockPath)
|
||||
} catch { /* best-effort; nothing to clean or already gone */ }
|
||||
}
|
||||
|
||||
// Arm once, only while we hold the lock: on a catchable termination (Ctrl-C, or
|
||||
// the menubar/desktop watchdog's SIGTERM) clean our lock before dying so a
|
||||
// killed refresh leaves no leftover. SIGKILL can't be caught, so that path still
|
||||
// relies on the waiter's dead-pid takeover. process.once + re-raise preserves
|
||||
// the default exit and any other listener. Mirrors session-cache.ts.
|
||||
let signalCleanupArmed = false
|
||||
function armSignalCleanup(): void {
|
||||
if (signalCleanupArmed) return
|
||||
signalCleanupArmed = true
|
||||
for (const sig of ['SIGINT', 'SIGTERM'] as const) {
|
||||
process.once(sig, () => {
|
||||
removeOurLockSync()
|
||||
process.kill(process.pid, sig)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
function isBusyError(err: unknown): boolean {
|
||||
const code = (err as NodeJS.ErrnoException | undefined)?.code
|
||||
return code === 'EPERM' || code === 'EBUSY'
|
||||
|
|
@ -198,7 +249,7 @@ export async function acquireCacheRefreshLock(options: RefreshLockOptions = {}):
|
|||
const clock = options.clock ?? defaultClock
|
||||
const heartbeatMs = options.heartbeatMs ?? DEFAULT_HEARTBEAT_MS
|
||||
const staleMs = options.staleMs ?? DEFAULT_STALE_MS
|
||||
const waitMs = options.waitMs ?? DEFAULT_WAIT_MS
|
||||
const waitMs = options.waitMs ?? staleMs + DEFAULT_WAIT_MARGIN_MS
|
||||
const pollMs = options.pollMs ?? DEFAULT_POLL_MS
|
||||
const sleep = options.sleep ?? delay
|
||||
const lockPath = join(cacheDir, LOCK_FILE)
|
||||
|
|
@ -219,6 +270,24 @@ export async function acquireCacheRefreshLock(options: RefreshLockOptions = {}):
|
|||
return next
|
||||
}
|
||||
|
||||
// An abandoned lock is one whose heartbeat has FROZEN - mtime stops advancing
|
||||
// the moment the holder dies or is displaced, and a live holder rewrites it
|
||||
// every heartbeatMs - OR whose recorded holder pid is gone. The pid check is
|
||||
// what turns a SIGKILLed holder from a full staleMs stall into a one-poll
|
||||
// recovery; the age check still covers every holder we cannot probe (a corrupt
|
||||
// body, a future version's record shape, a holder on another host).
|
||||
//
|
||||
// A LIVE holder is never abandoned by either clause, which is the whole safety
|
||||
// argument: its heartbeat keeps the mtime inside staleMs and its pid answers
|
||||
// signal 0. Our own pid is treated as alive so a leaked in-process handle is
|
||||
// never stolen from either. Callers re-observe under the takeover guard and
|
||||
// require the bytes to be unchanged before acting on this.
|
||||
const abandoned = (observation: Observation): boolean => {
|
||||
if (Math.max(0, clock.wallNow() - observation.mtimeMs) > staleMs) return true
|
||||
const pid = observation.record?.pid
|
||||
return pid !== undefined && pid !== process.pid && !pidLooksAlive(pid)
|
||||
}
|
||||
|
||||
const acquireTakeoverGuard = async (): Promise<'created' | 'exists' | 'unavailable'> => {
|
||||
const created = await createExclusive(takeoverPath, body())
|
||||
if (created !== 'exists') return created
|
||||
|
|
@ -226,7 +295,10 @@ export async function acquireCacheRefreshLock(options: RefreshLockOptions = {}):
|
|||
if (staleGuard === 'missing') return createExclusive(takeoverPath, body())
|
||||
if (staleGuard === 'changing') return 'exists'
|
||||
if (staleGuard === 'unavailable') return 'unavailable'
|
||||
if (Math.max(0, clock.wallNow() - staleGuard.mtimeMs) <= staleMs) return 'exists'
|
||||
// Same abandonment test as the primary lock: a holder killed while it held
|
||||
// the guard would otherwise block every takeover for a full staleMs and
|
||||
// defeat the dead-pid fast path on the lock itself.
|
||||
if (!abandoned(staleGuard)) return 'exists'
|
||||
const reverified = await observe(takeoverPath)
|
||||
if (reverified === 'missing') return createExclusive(takeoverPath, body())
|
||||
if (reverified === 'changing') return 'exists'
|
||||
|
|
@ -273,6 +345,8 @@ export async function acquireCacheRefreshLock(options: RefreshLockOptions = {}):
|
|||
const makeHandle = (): RefreshLockHandle => {
|
||||
let released = false
|
||||
let heartbeatRunning = false
|
||||
ownedLockPath = lockPath
|
||||
armSignalCleanup()
|
||||
const heartbeat = setInterval(() => {
|
||||
void serializeOwnerOp(async () => {
|
||||
if (released || heartbeatRunning) return
|
||||
|
|
@ -316,6 +390,7 @@ export async function acquireCacheRefreshLock(options: RefreshLockOptions = {}):
|
|||
release: async () => {
|
||||
if (released) return
|
||||
released = true
|
||||
ownedLockPath = null
|
||||
clearInterval(heartbeat)
|
||||
while (heartbeatRunning) await sleep(1)
|
||||
await removeIfOwned()
|
||||
|
|
@ -340,7 +415,7 @@ export async function acquireCacheRefreshLock(options: RefreshLockOptions = {}):
|
|||
if (current === 'unavailable') return { outcome: 'unavailable' }
|
||||
if (current === 'changing') return null
|
||||
if (current === 'missing' || !sameObservation(stale, current)) return null
|
||||
if (Math.max(0, clock.wallNow() - current.mtimeMs) <= staleMs) return null
|
||||
if (!abandoned(current)) return null
|
||||
if (!await retryWindowsMutation(() => unlink(lockPath), sleep)) return { outcome: 'unavailable' }
|
||||
// Publish the successor while the takeover guard is still canonical.
|
||||
// Otherwise a waiter can observe neither file and misclassify the narrow
|
||||
|
|
@ -390,8 +465,7 @@ export async function acquireCacheRefreshLock(options: RefreshLockOptions = {}):
|
|||
// and left alone, because it may belong to a live owner whose heartbeat
|
||||
// will repair it. Worst case we time out and serve the prior snapshot
|
||||
// read-only for one staleMs window instead of freezing forever.
|
||||
const age = Math.max(0, clock.wallNow() - observation.mtimeMs)
|
||||
if (age > staleMs) {
|
||||
if (abandoned(observation)) {
|
||||
const takeover = await tryTakeover(observation)
|
||||
if (takeover) {
|
||||
if (takeover.outcome !== 'acquired') leave()
|
||||
|
|
|
|||
|
|
@ -2450,11 +2450,16 @@ return program
|
|||
|
||||
if (process.argv[2] === 'serve') {
|
||||
const { runStdioServe } = await import('./serve.js')
|
||||
// Bind the REAL exit before serving. runCaptured() replaces process.exit with
|
||||
// a throw for the duration of a request, and a request still in flight when
|
||||
// the drain bound expires never restores it - so the exit below would throw
|
||||
// instead of exiting, which is exactly the orphan this line prevents.
|
||||
const hardExit = process.exit.bind(process)
|
||||
await runStdioServe(buildProgram)
|
||||
// stdin closed, so the owning app is gone. Exit outright: any handle that
|
||||
// outlives the transport (a watcher, a pending timer) would otherwise leave
|
||||
// this child running as an orphan for as long as the machine is up.
|
||||
process.exit(0)
|
||||
hardExit(0)
|
||||
} else {
|
||||
buildProgram().parse()
|
||||
}
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ import {
|
|||
saveCache,
|
||||
sourcePathStatCandidates,
|
||||
} from './session-cache.js'
|
||||
import { acquireCacheRefreshLock, type RefreshLockHandle } from './cache-refresh-lock.js'
|
||||
import { acquireCacheRefreshLock, type RefreshLockHandle, type RefreshLockOutcome } from './cache-refresh-lock.js'
|
||||
import { decideParseWorkers, parseFilesInOrder, ParseWorkerPool, type ClaudeWorkerParse, type ParseJob } from './parse-workers.js'
|
||||
import type { CodexFullParse } from './providers/codex.js'
|
||||
import { dateKey } from './day-aggregator.js'
|
||||
|
|
@ -4942,7 +4942,18 @@ async function parseAllSessionsInCacheScope(dateRange?: DateRange, providerFilte
|
|||
// Keep the snapshot loaded before acquisition: timeout/unavailable paths serve
|
||||
// exactly this complete snapshot and never mutate or invalidate the holder.
|
||||
const priorSnapshot = diskCache
|
||||
const refresh = await acquireCacheRefreshLock()
|
||||
// Heartbeat the WAIT too, not just the parse behind it. This is the one place
|
||||
// a healthy process is deliberately idle for a long stretch, and the desktop
|
||||
// and menubar watchdogs read silence as a dead child - which is how a waiter
|
||||
// blocked on an abandoned lock got killed at 45s and minted the next stale
|
||||
// lock (#1117). runParse arms its own keepalive; this covers the gap before it.
|
||||
startProgressKeepalive()
|
||||
let refresh: RefreshLockOutcome
|
||||
try {
|
||||
refresh = await acquireCacheRefreshLock()
|
||||
} finally {
|
||||
stopProgressKeepalive()
|
||||
}
|
||||
if (refresh.outcome === 'timed-out' || refresh.outcome === 'unavailable') {
|
||||
return runParse(key, priorSnapshot, dateRange, providerFilter, { readOnly: true, burstSig, parseStartedAt })
|
||||
}
|
||||
|
|
|
|||
10
src/serve.ts
10
src/serve.ts
|
|
@ -485,7 +485,15 @@ export async function runStdioServe(buildProgram: () => Command): Promise<void>
|
|||
let drainTimer: ReturnType<typeof setTimeout> | undefined
|
||||
await Promise.race([queue, new Promise<void>(resolve => { drainTimer = setTimeout(resolve, drainMs) })])
|
||||
clearTimeout(drainTimer)
|
||||
await watcherSetup
|
||||
// Bounded and non-fatal for the same reason the drain is: the app is gone,
|
||||
// and cleanup that hangs (watcher discovery on a stalled mount) or throws
|
||||
// must not stop this child from reaching its exit.
|
||||
let watcherTimer: ReturnType<typeof setTimeout> | undefined
|
||||
await Promise.race([
|
||||
watcherSetup.catch(() => undefined),
|
||||
new Promise<void>(resolve => { watcherTimer = setTimeout(resolve, drainMs) }),
|
||||
])
|
||||
clearTimeout(watcherTimer)
|
||||
rootReuseValidation = null
|
||||
try {
|
||||
watcherLifecycle.resetValidator?.()
|
||||
|
|
|
|||
|
|
@ -130,6 +130,35 @@ describe('warm refresh child-process regression', () => {
|
|||
expect(Object.keys((await loadCache()).providers['regression']?.files ?? {})).toEqual([source])
|
||||
})
|
||||
|
||||
// #1117: the menubar/desktop watchdog kills a CLI child with SIGTERM first,
|
||||
// precisely so the holder can unlink its own lock instead of leaving one for
|
||||
// every later spawn to wait out.
|
||||
it('unlinks its own lock when the holder is SIGTERMed mid-transaction', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'cb-refresh-sigterm-'))
|
||||
roots.push(root)
|
||||
const cacheDir = join(root, 'cache')
|
||||
const barriers = join(root, 'barriers')
|
||||
await mkdir(cacheDir, { recursive: true })
|
||||
await mkdir(barriers, { recursive: true })
|
||||
process.env['CODEBURN_CACHE_DIR'] = cacheDir
|
||||
const initial = emptyCache()
|
||||
initial.complete = true
|
||||
await saveCache(initial)
|
||||
const source = join(root, 'changed.json')
|
||||
await writeFile(source, JSON.stringify({ output: 505 }))
|
||||
|
||||
const holder = worker(cacheDir, barriers, 'a', source)
|
||||
// The worker blocks on its save barrier while holding the lock.
|
||||
await waitFor(join(barriers, 'a.parsed'))
|
||||
const lock = join(cacheDir, 'session-refresh.lock')
|
||||
expect(existsSync(lock)).toBe(true)
|
||||
|
||||
const exited = new Promise<void>(resolve => { holder.once('exit', () => resolve()) })
|
||||
holder.kill('SIGTERM')
|
||||
await exited
|
||||
expect(existsSync(lock)).toBe(false)
|
||||
})
|
||||
|
||||
it('serializes disjoint parsed updates so the later publication cannot drop the first', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'cb-refresh-process-'))
|
||||
roots.push(root)
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import { spawn } from 'child_process'
|
||||
import { chmod, mkdir, mkdtemp, readFile, rm, stat, unlink, utimes, writeFile } from 'fs/promises'
|
||||
import { tmpdir } from 'os'
|
||||
import { join } from 'path'
|
||||
|
|
@ -32,6 +33,15 @@ function fakeClock(start = 1_000): RefreshLockClock & { advance: (ms: number) =>
|
|||
}
|
||||
}
|
||||
|
||||
/** The pid of a process that has definitively exited, for the dead-holder gate. */
|
||||
async function exitedPid(): Promise<number> {
|
||||
const child = spawn(process.execPath, ['-e', ''], { stdio: 'ignore' })
|
||||
const pid = child.pid
|
||||
if (!pid) throw new Error('no pid')
|
||||
await new Promise<void>(resolve => { child.once('exit', () => resolve()) })
|
||||
return pid
|
||||
}
|
||||
|
||||
afterEach(async () => {
|
||||
delete process.env['CODEBURN_CACHE_DIR']
|
||||
await Promise.all(dirs.splice(0).map(dir => rm(dir, { recursive: true, force: true })))
|
||||
|
|
@ -134,6 +144,105 @@ describe('warm session-cache refresh lock', () => {
|
|||
}
|
||||
})
|
||||
|
||||
// #1117: the waiter budget now outlives the stale gate by design, so the
|
||||
// live-holder guarantee has to hold for a wait many times longer than staleMs,
|
||||
// not just for one shorter than it.
|
||||
it('never takes over a heartbeating owner however far the wait outlives staleMs', async () => {
|
||||
const dir = await tempDir()
|
||||
const path = lockPath(dir)
|
||||
const clock = fakeClock()
|
||||
await writeFile(path, JSON.stringify({ pid: 1, token: 'holder', at: clock.wallNow() }))
|
||||
const beat = async (): Promise<void> => {
|
||||
const now = new Date(clock.wallNow())
|
||||
await utimes(path, now, now)
|
||||
}
|
||||
await beat()
|
||||
|
||||
const result = await acquireCacheRefreshLock({
|
||||
cacheDir: dir,
|
||||
clock,
|
||||
staleMs: 500,
|
||||
waitMs: 5_000,
|
||||
pollMs: 1,
|
||||
// pid 1 is alive and the holder keeps pace with the clock, so neither
|
||||
// abandonment clause ever opens even though the waiter spends ten stale
|
||||
// windows in the loop.
|
||||
sleep: async () => { clock.advance(100); await beat() },
|
||||
})
|
||||
expect(result).toEqual({ outcome: 'timed-out' })
|
||||
expect(JSON.parse(await readFile(path, 'utf-8')).token).toBe('holder')
|
||||
})
|
||||
|
||||
it('takes over a lock whose holder pid is gone without waiting out the stale window', async () => {
|
||||
const dir = await tempDir()
|
||||
const path = lockPath(dir)
|
||||
// Wall clock frozen so the age gate can NEVER open; monotonic still runs so
|
||||
// the waiter budget expires instead of spinning. Only the dead pid can
|
||||
// explain a takeover here.
|
||||
let monotonic = 1_000
|
||||
const clock: RefreshLockClock = { wallNow: () => 1_000, monotonicNow: () => monotonic }
|
||||
const gone = await exitedPid()
|
||||
await writeFile(path, JSON.stringify({ pid: gone, token: 'killed', at: clock.wallNow() }))
|
||||
const now = new Date(clock.wallNow())
|
||||
await utimes(path, now, now)
|
||||
|
||||
const result = await acquireCacheRefreshLock({
|
||||
cacheDir: dir,
|
||||
clock,
|
||||
waitMs: 5_000,
|
||||
pollMs: 1,
|
||||
sleep: async ms => { monotonic += ms },
|
||||
})
|
||||
expect(result.outcome).toBe('acquired')
|
||||
if (result.outcome !== 'acquired') return
|
||||
expect(JSON.parse(await readFile(path, 'utf-8'))).toMatchObject({ pid: process.pid, token: result.handle.token })
|
||||
await result.handle.release()
|
||||
})
|
||||
|
||||
it('takes over a lock whose heartbeat froze even though its holder pid is alive', async () => {
|
||||
const dir = await tempDir()
|
||||
const path = lockPath(dir)
|
||||
// pid 1 always answers signal 0, so only the frozen mtime can explain it.
|
||||
await writeFile(path, JSON.stringify({ pid: 1, token: 'frozen', at: 1 }))
|
||||
await utimes(path, new Date(1), new Date(1))
|
||||
const clock = fakeClock(1_000_000)
|
||||
|
||||
const result = await acquireCacheRefreshLock({
|
||||
cacheDir: dir,
|
||||
clock,
|
||||
staleMs: 90_000,
|
||||
waitMs: 5_000,
|
||||
pollMs: 1,
|
||||
sleep: async ms => { clock.advance(ms) },
|
||||
})
|
||||
expect(result.outcome).toBe('acquired')
|
||||
if (result.outcome !== 'acquired') return
|
||||
await result.handle.release()
|
||||
})
|
||||
|
||||
it('defaults the waiter budget past the stale gate', async () => {
|
||||
const dir = await tempDir()
|
||||
const path = lockPath(dir)
|
||||
const clock = fakeClock()
|
||||
await writeFile(path, JSON.stringify({ pid: 1, token: 'frozen', at: clock.wallNow() }))
|
||||
const now = new Date(clock.wallNow())
|
||||
await utimes(path, now, now)
|
||||
|
||||
// No waitMs override: the default must be long enough that a lock which goes
|
||||
// stale mid-wait is still recovered by THIS waiter rather than timing out
|
||||
// and leaving the leftover for the next process to trip over (#1117).
|
||||
const result = await acquireCacheRefreshLock({
|
||||
cacheDir: dir,
|
||||
clock,
|
||||
staleMs: 90_000,
|
||||
pollMs: 1,
|
||||
sleep: async () => { clock.advance(1_000) },
|
||||
})
|
||||
expect(result.outcome).toBe('acquired')
|
||||
if (result.outcome !== 'acquired') return
|
||||
await result.handle.release()
|
||||
})
|
||||
|
||||
it('heartbeats its own lock body and mtime with the injected clock', async () => {
|
||||
const dir = await tempDir()
|
||||
const clock = fakeClock(10_000)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue