ouroboros/docs/PERSISTENCE.md

# PERSISTENCE.md — durable data-plane inventory

Every durable entity under the runtime data root (`DATA_DIR`, default
`~/Ouroboros/data/`), with its schema, migration path, retention and reset
semantics (what deleting the entity while the server is stopped does).
These contracts are local to each entity; there is no generic persistence
framework.

`tests/test_persistence_inventory.py` scans data-path constructors in
`ouroboros/`, `supervisor/`, `server.py` and `launcher.py` and requires every
scanned data-relative path to be covered by a row here (count-anchored both ways).

## Shared idioms (the vocabulary the rows use)

- **Stamp** — the opt-in `_schema_version` key from
  `ouroboros/contracts/schema_versions.py` (ABI-2): missing/invalid reads as
  legacy 0; existing files are not retrofitted. Some pre-ABI-2 entities carry
  their own spelling (`schema_version`, `schema`, `state_version`) — noted per
  row; new stamps use the shared key.
- **GC retention** — `ouroboros/retention.py`: one owner knob
  `OUROBOROS_GC_RETENTION_DAYS` (default 7, clamped 1–365; legacy per-subsystem
  keys migrate). Governs subagent worktrees, headless/task drives, task trees,
  service logs, consumed schedule receipts,
  confirmed capability probes, delegate recovery/supervision sweeps, code_intel
  reconcile-failed prunes, and agent media. Memory journals retain full new rows independently of this knob.
- **Rotation** — `supervisor/state.py::rotate_jsonl_log_if_needed`: >800 KB →
  atomic rename to `archive/<prefix>_<ts>.jsonl` under the append lock.
  Applied on the supervisor tick to `chat.jsonl`, `progress.jsonl`,
  `events.jsonl`, `tools.jsonl`, `supervisor.jsonl`,
  `task_reflections.jsonl`. Chain readers enumerate
  `archive/<stem>_*.jsonl` name-sorted (chronological by construction);
  `utils.jsonl_chain_handles` is the rotation-race-safe traversal
  (open-live-first + inode dedup).
- **`archive/` is durable history, never GC'd** (BIBLE P1): readers backfill
  from rotated segments; no retention sweep touches `archive/` and none may be
  added.
- **Atomic writes** — `atomic_write_json`/`atomic_write_text` (tmp+rename) and
  `update_json_locked` (sidecar `<file>.lock`); JSONL appends go through
  `append_jsonl` (O_APPEND + sidecar lock) unless noted.

## 1. Root files

| Path | Writer | Format | schema_version | Retention | Reset |
|---|---|---|---|---|---|
| `settings.json` | `ouroboros/config.py` save_settings (lock `settings.json.lock`, integrity guard) | JSON env-key map | none — not needed: migration is per-key inside `load_settings` (legacy keys migrate on read); external sha256 pin `OUROBOROS_SETTINGS_SHA256` makes it immutable when set | fixed-size overwrite | recreated from defaults; ALL owner secrets/modes lost — never delete casually |
| `.ouroboros_isolated_benchmark` | EXTERNAL writer — the benchmark launchers (`devtools/benchmarks/evolve_smoke.py`, `devtools/benchmarks/editbench/run_editbench.py`, `devtools/benchmarks/cybergym/cybergym_server.py`) stamp it into their throwaway data root; the runtime only READS it (`supervisor/state.py` rotate suppression, `ouroboros/agent_startup_checks.py`) | one-line marker text | none — not needed: presence IS the fact | write-once per benchmark drive; never GC'd | the drive stops declaring itself synthetic: JSONL rotation and the benchmark-only startup carve-outs switch back to live-root behaviour on a throwaway root |
| `settings.json.lock`, `*.lock` sidecars, `locks/**` | `ouroboros/platform_layer.py` lock family (+ `supervisor/state.py`, `supervisor/update_merge.py`, `ouroboros/skill_lifecycle_queue.py`) | O_EXCL lockfiles (unlinked on release) or flock files (persistent); plus one transient kernel-lock capability probe file per lock directory (`.kernel-lock-probe.<pid>.<ns>`, created, locked and unlinked once per process by `kernel_file_locks_enforced`) | none — not needed | self-healing (mtime/pid staleness) — except a lock whose owner died and whose pid was REUSED by a live process: it reads as alive whoever owns the pid (`kill(0)` succeeds for a same-uid impostor and answers EPERM for another user's — both mean alive), is never reclaimed by age while the impostor lives, and needs a hand repair meanwhile | zero durable state; deleting while stopped is a no-op |

## 2. `state/` — singletons (fixed-size overwrite)

| Path | Writer | schema_version | Retention | Reset |
|---|---|---|---|---|
| `state/state.json`, `state/state.last_good.json` | `supervisor/state.py` (lock `locks/state.lock`) | `_schema_version: 1` (ABI-2, stamp-on-write) | overwrite, totals-only cost projection; legacy keys popped on load | falls back to last_good, then mints defaults; loses owner binding, spend counters, session id |
| `state/queue_snapshot.json` | `supervisor/queue_snapshot.py` | `_schema_version: 1` (ABI-2) | overwrite; self-expires (max_age 900 s) | absent = 0 restored; queued-unstarted tasks silently dropped |
| `state/direct_roots.json` | `supervisor/direct_roots.py` — the main loop's off-lock projection of live direct-chat turns, written beside the snapshot on every tick (`publish_direct_roots`, `server.py`); queue init hands the previous process's rows to snapshot restore, which fences them, and empties the file in the same step (`take_direct_roots`, `supervisor/queue.py`) so a dead process's turns never outlive it | none — not needed: the whole file is re-derived every tick, and an unreadable one reads as no direct roots | overwrite; the rows are whatever the actor registry holds this tick, and a turn mid-admission (its actor lock only ever TRIED) is skipped behind ONE aggregate `incomplete` fact rather than blocking the loop | absent/empty = no live direct-chat roots are known: a task's peer roster (`ouroboros/peer_roster.py`) lists only the pooled roots from `state/queue_snapshot.json` until the next tick rewrites it |
| `state/advisory_review.json` | `ouroboros/review_state.py` (lock `locks/advisory_review.lock`) | own pre-ABI-2 spelling `state_version: 3` (+duplicate `schema_version`) — kept; `_schema_version` deliberately avoids this key | bounded on write: runs 10, attempts 50, debts 50; `open_obligations` coalesced | recreated empty; recorded blocking obligations forgiven, commit gate demands fresh review (fail-closed) |
| `state/scheduled_tasks.json` | `supervisor/queue_schedules.py` | `schema_version: 1` — authored at the write seam (`_write_scheduled_tasks`); legacy files gain it on their next write | consumed `once` receipts age out past GC retention on the scheduler tick (`prune_consumed_once_records`); 2 MB WARN = prune broken or live set huge | owner cron/once schedules lost; skill-manifest schedules resync automatically |
| `state/terminal_deliveries.json` | `supervisor/terminal_delivery.py` | own `schema_version: 2` | bounded: delivered 512, pending 64, replays 5 | dedupe + owed-outbox lost: possible double- or never-delivery of one buffered terminal answer |
| `state/cancel_intents.json` | `ouroboros/cancel_intents.py` | own `schema_version: 1` | self-draining (settled rows leave) | in-flight cancels lost: cancelled-unsettled task revives as pending; forensics survive in supervisor.jsonl |
| `state/update_letter.json` | `ouroboros/update_letter.py` (`refresh_after_check` — the ONE writer, synchronous inside a FETCHING update check: boot and the Updates panel's check button) | own record shape keyed by the update range (`base_sha`, `target_sha`, channel, target ref) — no `_schema_version` | overwrite per check; the letter is never deleted after the update lands (it outlives its range; `project_letter` reads `applied`/`other` by SHA equality + the recorded `target_in_head` fact) | absent = no letter: the Updates panel and the agent's Runtime context (`official_update_projection`) show the check's status alone; the next FETCHING check rewrites it |
| `state/capability_evidence.json` | `ouroboros/capability_evidence.py` | none — accepted (self-healing cache; TTLs on read) | expired probe keys drop at the write seam: failed/unprobeable past their read TTL, confirmed past GC retention (blip-keep evidence survives inside retention); owner acks never expire | recreated; every reviewer window resolves `unknown` (sends are sized by the full-window assumption and re-probed), owner acks must be re-given; no review authority depends on it |
| `state/evolution_campaign.json` | `supervisor/evolution_lifecycle.py` (CAS under state.lock) | own `schema_version: 1` (campaign) / 2 (active_transaction); the active transaction carries the pre-commit `commit_intent` (reviewed tree + parents) written BEFORE `git commit`, which boot recovery turns into the `commit_receipt` a crash never wrote | bounded histories (50) | in-flight self-modification transaction unabsorbable; anti-repeat fingerprints lost |
| `state/evolution_metrics_cache.json` | `ouroboros/utils.py` | own `schema: 1` (strictly validated) | one point per git tag, no prune — accepted (derived cache) | pure cache; recomputed from git |
| `state/projects.json`, `state/project_task_bindings.json` | `ouroboros/projects_registry.py` (sidecar locks) | `_schema_version: 2` / `1` (ABI-2) | never age-pruned (owner curates); deletes are durable tombstones | tombstones live here: losing it can resurrect deleted project rooms (marker unlink mitigates) |
| `state/ui_preferences.json` | `ouroboros/gateway/ui_preferences.py` | none — not needed (defaults contract + strict unknown-key rejection; retired keys dropped on write) | bounded (widget_order 200, cursors 1000) | cosmetic; defaults restored |
| `state/server_port` | `ouroboros/server_entrypoint.py` | none — not needed (1-line transport fact) | overwrite; pre-start unlink | readers fall back to default port |
| `state/server_port.bindings.json` | `ouroboros/server_process.py` (`record_service_binding`, `clear_service_binding`, shared locked update) | no schema version; exact service keys and measured process identity validated on read | at most main, host_service and local_model; overwrite per observed binding, exact-identity removal on close | current endpoint facts are lost; browser falls back to existing port/custody facts and refuses known endpoints with unconfirmed identity |
| `state/server_process.json` | `launcher.py` | none — PID/path-bound process facts; optional `server_host_source` distinguishes parent env from saved Settings, absent on old launchers means unknown | one-shot, self-deleting | stray-server reap skipped once; bind-source disclosure becomes unknown, never inferred from value equality |
| `state/auth_secret.key` | `ouroboros/server_auth.py` | none — not needed | write-once | regenerated; one forced owner re-login (accepted) |
| `state/worker_pids.json` | `supervisor/worker_pool_lifecycle.py` | none — not needed (legacy reap path; SSOT is process_ledger) | overwrite ≤ pool size | orphan reap falls to the custody reaper |
| `state/extension_companions.json` | `ouroboros/extension_companion.py` | none — not needed (runtime snapshot) | bounded by live companions | launcher loses force-kill map; leaked child processes possible |
| `state/pending_restart_verify.json` | `supervisor/evolution_lifecycle.py` (`write_pending_restart_marker`, the one writer — the supervisor's evolution restart and the agent's `restart` tool in `ouroboros/tools/control_runtime.py` both go through it) | none — not needed (one-shot marker, consumed by rename to `.claimed.<pid>.json`) | one-shot | evolution restart refused fail-closed without its receipt |
| `state/advisory_overrides.json` | `ouroboros/tools/review.py`, `ouroboros/tools/claude_advisory_review.py` | none — not needed (visibility counter; each bypass also durably in events.jsonl) | bounded (recent 10) | bypass counter lost; no gate reads it |
| `state/scope_delivery_migration.json` | `ouroboros/tools/review_admission.py` (`disclose_scope_delivery_migration`, atomic write) | none — not needed: presence IS the fact (the notice was given) | write-once per install | the one-time `review_scope_delivery_migrated` notice — a stored bare api scope row now runs a retrieving inspection episode — is emitted again on the next review; no review behaviour depends on the file |
| `state/reviewer_slot_last_execution.json` | `ouroboros/reviewer_slot_config.py` | none — not needed (disclosure, never enforcement) | capped 64, oldest evicted | UI disclosure line lost until next review |
| `state/usage_import_watermark.json` | `ouroboros/usage_legacy_import.py` | none — completion boolean is the contract | write-once | legacy import re-runs, fingerprint-deduped against the ledger (safe) |
| `state/presence_bindings.json` | `ouroboros/presence_bindings.py` (`update_json_locked`, strict existing dict) | own `schema_version: 1` — a mismatched version or a non-dict `bindings` map is a typed refusal, never a silent migration | overwrite; bounded by the owner-created links | every owner link from a transport room to a behavior skill is lost; presence stops routing those rooms until the links are re-created |
| `state/presence_turn_gate/last-<sha256>.json` | `ouroboros/presence_runner.py` (`atomic_write_json`, one file per conversation key, written under the conversation lock at the end of an executed turn) | flat JSON: `conversation_key`, `task_id`, `outcome`, `message`, `transport_sends`, `work_ref`, `finished_at`, `delivery`; a rebuildable projection, task results and delivery receipts stay canonical | overwrite per executed turn; a cached replay writes only to rebuild a pointer that never landed or names an older turn; the write is best effort (a failure logs a warning, the next replay retries); no age GC (one small file per conversation) | the next turn of that conversation loses the previous-turn fact until the following executed turn, or a replay of the last settled authored turn, rewrites it |
| `state/request_wire_compatibility.json` | `ouroboros/request_wire_contract.py` (`update_json_locked`) | own `schema_version: 1`; any other version reads as an empty store | per-profile actions expire on read at `REQUEST_WIRE_TTL_SEC` (14 days) | learned provider-wire adaptations are forgotten; the next send re-learns them from a fresh provider refusal (one extra failed attempt per route) |
| `state/claudexor_rotation_provisioning.json` | `ouroboros/claudexor_daemon.py` `_record_rotation_receipt` (atomic text) | none — not needed: a disclosure receipt, never read back for behaviour | last reconcile that actually patched wins | the receipt is lost; provisioning itself is idempotent — the next ensure re-reads the daemon settings and re-POSTs only when the values are absent |
| `state/delegate_terminal_refresh_cursor.json` | `ouroboros/delegate_terminal.py` `refresh_recently_settled_terminals` (`atomic_write_json`, best-effort) | none — accepted: a byte offset over the rotated custody chain plus a deferred map, self-grounding (an offset past the chain resets to 0) | overwrite; `deferred` capped at 500 tasks | the cursor re-grounds at 0 and the one-time historical pass repeats, paced by the 5 MB per-tick scan cap; no terminal evidence is lost |
| `state/extension_generation.json` | `ouroboros/extension_reconcile_queue.py` `publish_extension_generation` (write-if-changed) | own `schema_version: 1` | overwrite (one live-extension fingerprint) | an absent marker is NO evidence of divergence (fail-closed): running workers keep their spawn-time extension set until the next publish |
| `state/post_task_evolution_request.json`, `state/post_task_evolution_counter.json` | `ouroboros/post_task_evolution.py` (atomic publish; the supervisor polls and consumes) | request: own `schema_version: 1`; counter: none — not needed (a single `n`) | request is one-shot (consumed by `apply_pending_request`, deleted by the owner-stop sites); counter overwrites | a pending promotion signal is dropped (no evolution task queued) and the every-N cadence restarts its count; the durable re-arm backstop stays the `evolution_owner_stopped` state flag, not this file |
| `state/subagent_last_delegation.json` | `ouroboros/subagent_history.py` `record_last_delegation` (locked atomic, best-effort, canonical data plane; task finalization and common session custody producers) | none — additive disclosure; old single-row receipts remain readable, missing/corrupt data is unknown | latest dated observation per actor, bounded by `MAX_CONFIGURED_SUBAGENTS`, plus compatible latest receipt; undated occurrence stays unknown and replay does not refresh age | helper history disappears until a new observation; admission is unchanged and full forensics remain in custody/attempt records |
| `state/panic_stop.flag`, `state/owner_restart_no_resume.flag` | `ouroboros/server_control.py`, `server.py` (atomic pair) | none — not needed (one-shot flags, consumed on boot) | consumed | absence is the default; next boot auto-resumes |

## 3. `state/` — append-only ledgers

| Path | Writer | schema/record marker | Retention | Reset |
|---|---|---|---|---|
| `state/usage_attempts.jsonl` (+ `state/usage_attempts.quarantine.jsonl`, `state/usage_attempts.lock`) | `ouroboros/usage_ledger.py` single chokepoint (own O_APPEND+fsync under named lock — NOT append_jsonl); `ouroboros/usage_compaction.py` rewrites it whole (verified candidate, atomic swap) under the same lock | no `_schema_version`; its own validated contract: dense `seq`, `kind` discriminator, `state` machine, per-row `candidate_measurement_kind`, attribution `physical_attempt_v1`; compacted files lead with a stamped `usage_baseline` header + `usage_baseline_group` rows — accepted | bounded by compaction (config `USAGE_LEDGER_COMPACT_BYTES`, 8 MB): terminal non-review attempt chains fold into the baseline block, raw segment archived first (fsync'd) — see docs/USAGE_COMPACTION.md; 20 MB WARN can reflect broken compaction, a large unfoldable residue, or a refused/skipped pass: the name-tier refusal emits `usage_ledger_compaction_refused` once per process per data root; a policy abort (`_Abort`) emits `usage_ledger_compaction_skipped` once per process per (data root, reason); the two snapshot-race exits before archive/swap only log warnings, without a typed event; torn tails — and a charge a compaction swap erased inside its last syscall, read back from the old inode (POSIX) — quarantined, never GC'd | monetary history destroyed, `seq` restarts, budget fences read $0; watermark survives so legacy import will NOT re-run — deleting the ledger alone is unrecoverable; a ledger reset beside `archive/usage_ledger/` makes history queries raise `generation newer` while an unreferenced segment represents a newer generation and is not a byte-prefix of the live file; after fresh compactions reach the surviving archive generations, those old unreferenced segments are skipped and their attempt IDs remain absent — see [history readers](USAGE_COMPACTION.md#10-history-readers-model-send-reconciliation-and-audits); keep or reset the ledger and archive together |
| `state/skill_review_root_tasks.jsonl` (+ `state/skill_review_root_tasks.gaps.jsonl`) | `ouroboros/skill_review_history.py` (`_append_root_task_projection_once`: one compact row per root task appended when its skill review lands; a row the projection could not attribute goes to the `.gaps.jsonl` ledger via `_record_root_task_projection_gap`) | rows carry `usage_attribution_schema: physical_attempt_v1`; no version key — accepted (derived index over the per-skill `review_history.jsonl`, P7) | unbounded append; reads are BOUNDED (`skill_readiness.py`: 1 MiB / 512 records tail, a truncated or gapped read is disclosed as `projection_incomplete`) and the startup hot-store check warns past `SKILL_REVIEW_ROOT_TASKS_WARN_BYTES` | delete with `state/`; the index is not rebuilt — readiness and the acceptance packet read `projection_incomplete` until new rows accrue (the per-skill histories keep the truth) |
| `state/process_ledger.jsonl` | `ouroboros/process_custody.py` (spawn chokepoint) | none — downgrade-safe field split (`start_time`/`start_time_boot`) is the versioning device — accepted | self-compacting: reapers rewrite survivors-only | prior-generation supervised processes permanently orphaned (fingerprint index lost) |
| `state/evolution_checkpoints.jsonl` | `ouroboros/evolution_checkpoints.py` | `schema_version: 1` on every row (+`kind` on outcome rows) | unbounded append; read bounded (last 200) — accepted (structured solve-capability history is the product) | absorbed/abandoned objectives can be re-proposed (BUG3 regression); the outcome row is DERIVABLE from the resolved campaign transaction, so a row lost to a crash between the two writes is replayed at boot (`source: boot_backfill`) |
| `state/consciousness_observations.jsonl` | none — RETIRED with the observation inbox; nothing writes or reads it | n/a | the alarm clock MOVES a file left by an older version once, unread, to `archive/consciousness_observations.jsonl` (a name collision gets a `_<boot ts>` suffix) and never looks again | nothing — the rows are already historical |

## 4. `state/` — directories

| Path | Writer | schema_version | Retention | Reset |
|---|---|---|---|---|
| `state/extra-ca-bundle/*.pem` | `ouroboros/net_transport.py` (`extra_ca_bundle`; tmp file + atomic replace) | none — derived (certifi followed by the owner's `OUROBOROS_EXTRA_CA_BUNDLE` PEM), each file named by the merged bytes' digest | a changed owner file writes a new file; siblings older than a day are pruned | recreated on the next client construction; nothing beyond that process's TLS trust depends on it |
| `state/skills/<name>/` owner state (`review.json`, `review_job.json`, `grants.json`, `enabled.json`, `deps.json`, `self_authored.json`, `owner_attestation.json`, `accepted_rebuttals.json`, `health.json`, `uninstalled.json`, provenance sidecars, `auto_repair.json`, `presence_profile_state.json`) | `ouroboros/skill_loader.py`, `skill_review_runner.py`, `skill_owner_attestation.py`, `skill_review_cycles.py`, `skill_uninstall_state.py`, `extension_health.py`, `marketplace/*`, `ouroboros/gateway/marketplace.py`; allowlist SSOT `contracts/skill_payload_policy.py` | `deps.json`/`self_authored.json`/provenance: `schema_version: 1`; `review.json`/`enabled.json`/`grants.json`/`review_job.json`/`owner_attestation.json`/`accepted_rebuttals.json`: `_schema_version: 1` (ABI-2, stamp-on-write — readers keep legacy-0 tolerance, unstamped files never retrofitted); verdict/grant staleness stays pinned by `content_hash` | no age GC; hub uninstalls write an `uninstalled.json` tombstone and the startup sweep clears the dead state BY that mark — `grants.json` survives as owner authority, a reinstall self-heals the tombstone; the gateway's local delete removes the whole state dir | absent state = disabled + pending review + grants revoked (fail-closed); `owner_attestation` absence invalidates its verdict |
| `state/skills/<name>/review_history.jsonl` + `review_dispatch/` (legacy `review_dispatch.json`) | `ouroboros/skill_review_history.py` | rows carry `usage_attribution_schema: physical_attempt_v1`; no version key — accepted (derived-counter SSOT, P7) | history unbounded per skill — accepted with BOUNDED reads: every reader windows the 4 MB tail (`find_history_job_bounded` idiom); lifecycle terminal rows persist their ordinals so counters stay exact inside the window (a group aged past it restarts low — under-counts, never over-blocks); per-skill archive rotation declined (no per-skill archive plane; disclosed) | review-cycle ceiling resets to zero; paid dispatches become free again |
| `state/delegate_project_retirements/<sha256[:24]>.lock` | `ouroboros/delegate_custody_usage.py` (`project_retirement_lock`: exclusive file lock around one project's settlement/retirement decision; stale after 120 s, owner-aware) | none — not needed (lock file, no payload) | one file per project ever settled; reclaimed as stale by the next holder | delete freely; a live holder re-creates its lock |
| `state/skills/<name>/` transport dirs: `extension_calls/`, `__extension_imports/` | `ouroboros/extension_process_runner.py`, `extension_import_staging.py` | none — not needed (per-call transport files, staged import trees) | per-call files consumed; import leaves reaped owner-dead+grace | transient; recreated per call |
| `state/delegate_recovery/`, `state/delegate_recovery_transactions/` (+`active.json`), `state/delegate_supervision/` | `ouroboros/delegate_recovery.py`, `delegate_supervision.py` (+ startup sweep `delegate_state_sweep.py`) | own `schema: 1` on supervision/transactions; recovery rows fingerprinted, unversioned | terminal+age startup sweep: terminal-status recovery rows (`vetoed`/`adopted`), unreferenced transactions and settled-task supervision files past GC retention; live/resumable rows, `active.json`, no-result tasks and unreadables kept fail-closed; unreadable custody log skips the sweep | interrupted delegated runs cancelled instead of adopted; duplicate wake replay; planned handoffs vetoed |
| `state/delegate_actor_claims/*.lock`, `state/.payload_delegation_claim.lock` | `ouroboros/delegate_custody.py`, `delegate_start_claims.py` | none — locks | unlinked on release | no durable state |
| `state/code_intel/<root-sha>/inventory.json` | `ouroboros/code_intelligence.py` | own `schema_version: 2` (older/malformed rebuilt silently) | per-repo rewrite in place; stale roots age-pruned at startup by `inventory.json` mtime past GC retention (pure cache) | pure derived cache; one full re-index |
| `state/extension_reconcile/` (+`failed/`) | `ouroboros/extension_reconcile_queue.py` | none — not needed (one-shot markers) | consumed by server loop; after 5 attempts moved to `failed/`, where markers age-prune past GC retention (the failure fact stays durable in events.jsonl) | pending worker→server reconciles lost; re-toggle heals |
| `state/workspace_executor_processes/` | `ouroboros/workspace_executor.py` | own `schema_version: 1` + owner tag | unlink on stop; stale rows filtered at read (pid/cmd-sha) | service processes survive unreaped |
| `state/acceptance_fence_acks/<token>.<req>.json` | `supervisor/events_worker_reports.py` (writer, one file per pooled-worker request), `ouroboros/agent.py` (the requester reads and unlinks only its own `req`; direct turns fence in-process and write none) | none — transport ack | inline GC on write (255 newest / 3600 s); a late ack nobody waits for ages out here | the waiting worker re-sends a transition once (a read never), then records a typed `supervisor_ack_unavailable` unknown — never read as a refusal |
| `state/headless_tasks/<id>/` (child data drives) | `ouroboros/headless.py` | child `state/state.json`: `schema_version: 1` | GC-retention prune at startup (terminal + age; skips artifacts-not-terminal / refs-unpromoted) | in-flight child drives and unpromoted child refs lost |
| `state/cx/` (managed Claudexor runtime) | `ouroboros/claudexor_runtime.py` | meta files versioned (`_NODE_META_SCHEMA_VERSION: 2`, pin 1) | deliberate keep-all (rollback selects older pins); staging/displaced temporaries reaped | re-downloaded on demand; nothing durable lost |
| `state/betterleaks/` (managed runtime + cache) | `ouroboros/betterleaks_runtime.py` | own manifests (`schema_version: 1`) | keep-all archive cache — accepted (managed runtime) | re-downloaded on demand |
| `state/pycache/`, `state/python-userbase/` | `launcher.py`, `ouroboros/packaged_cli.py`, `launcher_bootstrap.py` (PYTHONPYCACHEPREFIX / PYTHONUSERBASE) | none — caches | no GC; `python-userbase` is a documented manual-recovery hazard (outranks bundle site-packages) | `pycache`: always safe (recompile); `python-userbase`: deletes REAL user-installed deps |
| `state/review_continuations/<task>.json` (+ `state/review_continuations/archived/**`, `state/review_continuations/corrupt/**` with its `.txt` reason notes) | `ouroboros/task_continuation.py` (atomic write; `rename` to `archived/` on retire, `replace_atomic` to `corrupt/` on quarantine) | none — accepted: a typed dataclass with an ownership check (a `task_id` mismatch raises) and malformed JSON quarantined, never migrated | live payload per blocked task, cleared on resume; a settled, un-resumed continuation is retired to `archived/` past `RETIRE_SETTLED_CONTINUATION_AFTER_DAYS` (7) so history stops riding into every new prompt; `archived/` and `corrupt/` unbounded — accepted (small typed payloads; the quarantine IS the corruption evidence) | blocked tasks lose their resume pointer: findings, obligations and the commit intent must be re-derived by re-running review |
| `state/skills/<name>/auth_token.json` | `ouroboros/extension_plugin_api.py` `mint_skill_token` (atomic write, chmod 0600) | none — not needed: the token is bound to `content_hash`, which IS its staleness contract | one token per skill, rotated on a content-hash change; a transient hash failure reuses or fails closed, never rotates | the next mint issues a new token, so a companion still holding the old one is de-authorized until respawn — do not delete while companions run |
| `state/skills/<name>/repair_admission.json` | `ouroboros/skill_repair_admission.py` `record_repair_admission` (atomic; the newest admission owns the record) | own `schema_version: 1` | one record per skill, superseded by the next admitted repair | the repair CAS has nothing to check against and its writes are refused — fail-closed by design, since this record exists to remove exactly that fail-open |
| `state/skills/<name>/jobs/<job>/` (`assets/`, `output/`, `tmp/`) | `ouroboros/extension_plugin_api.py` `skill_job_dir` (creates the three children on request; the extension owns the contents) | none — not needed: a workspace, not a record | unbounded: nothing sweeps it, and the gateway's local skill delete is the only path that removes it with the state dir | in-flight extension jobs lose their assets and output; the next call recreates the tree |
| `state/skills/<name>/chat_id_counter.json` | `ouroboros/gateway/host_service.py` `allocate_internal_chat_id` (atomic, under the in-process counter lock) | none — not needed: `range_name` plus last/next id | one file per skill; ids descend from `A2A_CHAT_ID_MAX` | allocation restarts at the top of the range, so a fresh A2A room can reuse a chat id already present in history |
| `state/project_source_locks/` | none in this tree — orphan plane seen in live layouts (removed feature leftover) | none | n/a | harmless; nothing reads or recreates it |

## 5. `logs/`

| Path | Writer | Record marker | Retention | Reset |
|---|---|---|---|---|
| `logs/chat.jsonl` | `supervisor/message_bus.py` (+presence, project summaries) | `direction` + optional `type`; no version — accepted (projection replayed by chain-aware readers) | rotated 800 KB → `archive/chat_*.jsonl`; archive chain WARN at 100 MB | newest generation lost; consolidation cursor reports gap (recoverable) |
| `logs/progress.jsonl` | `supervisor/message_bus.py` (+plan review) | `type: send_message`, `is_progress` | rotated 800 KB → `archive/progress_*.jsonl`; 8 MB WARN = rotation broken | current segment lost; readers archive-chain-aware |
| `logs/events.jsonl` | ~60 modules via `append_jsonl` (+`delegate_custody.emit`) | universal `type` discriminator — accepted (per-type payloads owned by emitters) | rotated 800 KB → `archive/events_*.jsonl`; custody readers (replay, fault tail-scan, `complete_custody_rows`, settled-terminal chain cursor, legacy-usage import, swarm rollup, worker-boot verify) are chain-aware; delegated start rows reference their complete raw replay envelope in `observability/blobs/**` (blob before event), with inline legacy rows still readable; `task_received` omits only an equal metadata mirror of its top-level task contract; 8 MB live WARN = rotation broken; 100 MB chain WARN = replay degradation | delegated-run custody destroyed (chain incl. archive segments): open runs invisible/unreapable; lineage, citations, legacy-usage source lost |
| `logs/tools.jsonl` | `ouroboros/loop_tool_execution.py` (+budget-drive mirror) | `type: tool_call` + task lineage (`root_task_id`/`delegation_role` from `resolve_task_lineage`, a direct root is its own root), untruncated args | rotated 800 KB → `archive/tools_*.jsonl`; tail readers (api_logs_tail, task_events) archive-backfill; 8 MB WARN = rotation broken | untruncated tool record + `result_ref` pointers lost |
| `logs/supervisor.jsonl` | supervisor family, `process_custody`, gateway control, server shutdown | `type` (+secondary `event_type`) | rotated 800 KB → `archive/supervisor_*.jsonl` + 8 MB tripwire; tail readers (`memory.read_jsonl_tail`, api_logs_tail) archive-backfill | reap receipts, rescue disclosures, shutdown causes lost |
| `logs/task_reflections.jsonl` | `ouroboros/reflection.py` (+ project-scoped copy under `projects/<id>/logs/`) | full rows unversioned; pointer rows `type: project_reflection_pointer` | rotated 800 KB → `archive/task_reflections_*.jsonl` + 8 MB tripwire; tail-20 read archive-backfills; project-scoped copies follow project retention (never age-pruned) | inter-task memory-carry signal lost |
| `logs/containment_faults.jsonl` | `ouroboros/delegate_custody.py` (mirrored to events.jsonl) | `type` ∈ CONTAINMENT_FAULT/RESOLVED joined on run_id | unbounded BY DESIGN — read whole so an open fault never ages out — accepted | health invariant degrades to the 4 MB events tail scan (the regression this file fixed) |
| `logs/chat_annotations.jsonl` | `ouroboros/project_dialogue.py` (`append_jsonl` under the shared sidecar append lock) | `type: chat_annotation` keyed by `(client_message_id, routing_token)`; latest row per key wins (an owner message keeps one row per routing act; task-authored acts ride synthetic `agent-steer:<token>` ids) | self-compacting at 800 KB under the append lock: the latest row of every message still present in the chat chain (live `chat.jsonl` + the 3 newest `archive/chat_*.jsonl`) is rewritten, the rest is DROPPED — presentation state, so it is not rotated into `archive/` | annotation cards fall back to their plain chat rows; the one named exception (#198) also loses the durable picker decision-card token, so a pending manual routing choice must be made again |
| `logs/tasks/task_<id>.txt` | `ouroboros/utils.py` log sanitization (`write_text`, best-effort) | raw oversized task text, no envelope; the log row keeps `text_full_path` | unbounded: no retention sweep names `logs/tasks/` | the truncated text in the log row stays; only the spilled full text of oversized task prompts is lost |
| `logs/agent_stdout.log` | `launcher.py` pipe-copy thread | unstructured text | bounded ~8 MB (2 MB × `.1..3` backups, rotated by the copy thread) | pre-logging crash output lost; nothing parses it |
| `logs/server.log` (+`.1..3`), `logs/launcher.log` | stdlib `RotatingFileHandler` (`server.py`, `launcher.py`) with secret-redacting filter; a spawn/forkserver worker importing `server.py` as `__mp_main__` gets a stream handler only (one rotator per file; `OUROBOROS_WORKER_START_METHOD=fork` inherits the parent's handlers unchanged) | text | bounded ~8 MB (2 MB × 4) — the model citizen | stdlib log history lost; nothing parses it |

## 6. `memory/` (Ouroboros cognition — operator read-only)

| Path | Writer | Marker | Retention | Reset |
|---|---|---|---|---|
| `memory/identity.md` | `ouroboros/tools/control_runtime.py` (full overwrite, ≥50 chars, shrink notice) | none | unbounded document — accepted (identity is the product) | reseeded default; identity lost (journal keeps history) |
| `memory/scratchpad.md` + `scratchpad_blocks.json` | `ouroboros/memory.py` (derived, regenerated from blocks under lock) | none | bounded: 10 blocks, eviction journaled first (fail-closed) | regenerated; evicted history in journal |
| `memory/WORLD.md` | `ouroboros/world_profiler.py` (write-once) | none | fixed | regenerates on restart — deletion IS the refresh mechanism |
| `memory/registry.md`, `memory/deep_review.md` | `ouroboros/tools/memory_tools.py` (section RMW), `ouroboros/agent.py` (overwrite) | none | unbounded / last-wins — accepted | recreated lazily |
| `memory/dialogue_blocks.json` + `dialogue_meta.json` | `ouroboros/consolidator.py`, `memory_nomination_receipts.py` (locked atomic) | `pending_knowledge_nominations` source-entry IDs; legacy `last_unpublished_nominations` preserved; `era_retry` (source hash + effective Light dispatch binding as key + observed route of a not-shorter era) | blocks reduced by era compression (ordinary pass: over 10 blocks, the oldest run of up to 4 SUMMARY blocks before the newest; pressure pass: every complete run, uncapped; an era is never re-compressed — calendar-era recompression is deferred, not implemented — so eras accumulate unbounded); unresolved nomination index unbounded; no tool-level resolver yet, later success never retires old debt | blocks: compressed biography irreproducible; meta: cursor and unpublished-obligation evidence lost |
| `memory/dialogue_summary.md` | none — legacy read-only (reader in context.py) | none | frozen | legacy artifact; nothing writes it |
| `memory/knowledge/**` (topic .md + `index-full.md` + `patterns.md`) | `ouroboros/tools/knowledge.py`, `consolidator.py` (index rebuild), `reflection.py` (patterns CAS rewrite) | none | topic files unbounded — accepted (curated by consolidation); backlog topic merge-only fail-closed | recreated lazily; knowledge lost |
| `memory/*_journal.jsonl`, `memory/knowledge_history.jsonl`, `memory/knowledge/patterns_history.jsonl` | `ouroboros/memory.py`, `tools/control_runtime.py`, `tools/knowledge.py`, `reflection.py` — every append through the `append_jsonl` sidecar-lock seam | scratchpad journal: `type` rows; others unversioned full-text snapshots; `knowledge_history.jsonl` `source_capture` rows carry the host stamp `writer`/`route`/`writer_input_ref`/`old_chars`/`new_chars` (rows older than the stamp read `unknown`), an automatic anchored edit also its authored `edits` (with `basis`) and, only when supplied, its `summary`; historical digested rows retain `content_digested: true` | complete new old+new snapshots are retained indefinitely; `memory_journal_compaction.py` is a read-only compatibility entry point, not a source rewriter; existing digest-only rows cannot be restored; the `memory_journal_observation` startup event gives byte sizes (or missing/unreadable) for the three named journals; scratchpad keeps its eviction journal | deleting the journals loses undo/provenance; eviction/rewrite paths fail closed when journal append fails; historically digested content remains irrecoverable |
| `memory/owner_mailbox/<task>.jsonl` + `.acks.jsonl` | `ouroboros/owner_mailbox.py` (append-only; revocation appends, reader resolves) | `kind` discriminator | lifecycle-bounded: unlinked, under the per-task mail lock appends and acknowledgements share, only once the SETTLED canonical row with post-work closed holds every exact unread row AND a verified canonical projection of every input-bearing row's attachments (`unread_mailbox.rows`/`.inputs`, `task_custody.settle_task_mailbox`; the loop thread's seam carries no inputs, the off-loop drive-custody pass and the drive settlement do); a torn read, no result, open post-work, a non-terminal row or an uncarried input keeps it | undelivered owner directives + restart-surviving hurry latch lost; acks lost ⇒ re-delivery |

## 7. Skills payloads, tasks, uploads, projects, services

| Path | Writer | Marker | Retention | Reset |
|---|---|---|---|---|
| `skills/{native,clawhub,ouroboroshub,external}/<name>/**` (+`.staging/`, `.ouroboros_env/` with `cache/ tmp/ home/`) | `ouroboros/marketplace/*`, `launcher_bootstrap.py` seed, agent self-authoring | provenance sidecars `schema_version: 1`; env `fingerprint.json: 1` | no age GC (payloads are installed software); staging rmtree'd per install, crash orphans recognized by name fragments; package caches live with the skill | bucket recreated empty; native seeds NOT re-seeded (deletion intent preserved) except post-bootstrap set; orphaned `state/skills/` rows keep grants + tombstone after the startup sweep |
| `state/skills/<name>/dependency_cache/` | `marketplace/isolated_deps.py` via existing installer subprocesses and verified resource downloads | downloaded resources keyed by sha256; package-manager cache formats | reused across payload/environment replacement; follows existing skill-state cleanup | resources are verified/downloaded again and package caches rebuilt |
| `state/skills/<name>/go/`, `state/skills/<name>/go-cache/` | Go compiler launched by `tools/skill_exec.py:_run_go_skill`, with GOPATH/GOCACHE bound to skill_state_dir | Go-owned cache formats | reused across script runs; follows existing skill-state cleanup | compiler recreates caches; skill payload and review remain unchanged |
| `task_results/<id>.json` | `ouroboros/task_results.py` (locked merge) | `_schema_version: 1` (ABI-2); unstamped/future/malformed → quarantine, no conversion — with one carve-out: the boot latch migration re-stamps unstamped rows still at `cancel_requested` in place (same status, one typed `task_result_cancel_latch_admitted` event) so a wedged task still reaches its `cancelled` terminal | UNBOUNDED — one file per task forever, no GC; lifecycle authority is retained deliberately | lifecycle authority lost; drive prunes degrade to age-only; strict authority reads break |
| `task_results/quarantine/` | `ouroboros/task_result_schema.py` (same-dir rename); `presence_runner.py` reads the retained id before a transport retry | quarantined bytes unchanged; a Presence retry with quarantined authority refuses regeneration | NEVER GC'd (pinned); recovery is manual owner re-stamp | quarantined evidence lost; a previous Presence attempt could be mistaken for a new event |
| `task_results/<id>.custody.lock`, `task_results/<id>.mail.lock`, `state/custody_staging/<id>-<token>/`, `state/custody_trash/<id>-<token>/` | `ouroboros/task_custody.py` (the custody lock every canonical-store publisher takes — copy-back, ref retry, host artifact finalization, mailbox input carry and drive settlement; the mail lock mailbox appends, acknowledgements, mailbox cleanup and the settlement's final check take; private staging copied and verified before publication, then placed create-only; a settled drive moved under the supervisor's queue interlock plus both locks, then deleted) | none — transient | locks released per phase; staging deleted by its settlement; leftovers older than an hour removed by the off-loop drive-custody pass | none: a lost staging copy or trash entry is re-prepared or was already fully in canonical custody |
| `task_results/artifacts/<id>/**` (+`verification_receipts.jsonl`, `.directory.*.tmp`, `.directory.*.json.tmp`), `task_results/artifact_versions/` | `ouroboros/artifacts.py`, `headless.py`, `outcome_receipt_store.py` | artifact and complete directory manifests `schema_version: 1`; scratch manifest 2 | artifact versions bounded (5 per name); artifacts live with their result; directory capture stages ZIP and manifest beside the result, removing owned temporaries on caught failures | deliverable bytes lost; results keep dangling manifests |
| `task_drives/<id>/**` (+`tmp_scripts/`) | `ouroboros/headless.py`, `tools/tool_context.py`, `tools/shell.py` | child stamps as above | settled off the loop thread by the drive-custody pass (terminal + age, default 7 d; `task_custody.settle_child_drive`); the `data/tmp_scripts` fallback's hard-kill orphans are swept at startup (when no script can be live), the whole-tree walk for orphaned atomic temps by the first reconcile pass of the generation (`sweep_stale_temp_files`) | scratch lost; canonical artifacts survive |
| `task_trees/<root>/blackboard.jsonl` | `ouroboros/task_tree_ledger.py` | rows unversioned; snapshot digest `schema_version: 1` | GC-retention prune at startup (root terminal + age) | swarm coordination facts lost for live trees |
| `state/subagent_worktrees.json` (registry; checkouts live OUTSIDE data root) | `ouroboros/subagent_worktrees.py` | none — malformed → typed refusal (absent = empty is the designed asymmetry) — accepted; file_baseline retains copied binary input identities | prune_orphans (age + missing checkout; skips delegated_exec) + custody-cross-checked snapshot prune (fail-closed on unreadable custody) | permanent leak of checkouts + pinned refs (nothing else names them) |
| `task_results/artifacts/<task>/source_handles/context_checkpoints` (`focus_source_<reader>-<sha256>.md`) | `ouroboros/tools/project_journal.py` (`_retain_focus_source`, through `artifacts.store_actor_source_bytes`) on the canonical root; read by `task_finalization.focus_source_projection` (digest glob for a historical selector) | native `task_source` handle on `focus.source_handle` (size + sha256, verified on read) | write-once, digest-named; lives with the task's artifacts (no separate prune) | a roster's `retained_source` reads `source_unavailable`; the focus text itself survives on the task result |
| `task_results/artifacts/<task>/delegated_runs/<run>/engine-files-manifest.json`, `task_results/artifacts/<task>/delegated_runs/<run>/*-*` | `ouroboros/delegate_directory.py`, existing streamed task-artifact owner | exact engine manifest SHA and per-file before/after digests | canonical result artifacts retained; engine keeps unapplied copy results under its existing disposition/retention owner | loss of complete binary results, baseline references and evidence after execution copy cleanup |
| `uploads/**` (+`screenshots/`, `views/`, atomic-copy `.tmp` files) | `ouroboros/gateway/files.py` through `artifacts.copy_artifact_file` for chat uploads; `tools/browser.py`, `tools/vision.py`, `server_owner_routing.py` | raw owner bytes; chat ingestion measures size and SHA256 while copying | owner attachments in the `uploads/` root: NO retention, owner-explicit delete only; chat upload no longer applies the former 50 MiB cap, while Files-browser and downstream transport limits retain their own contracts; agent-generated `screenshots/`/`views/` age out past GC retention at startup | chat attachments dangle (readers skip missing); staged task copies survive |
| `services/<task>/*.log` | `ouroboros/workspace_executor.py`, `tools/services.py` | none — raw text; archived content becomes observability blob with events.jsonl receipt | GC-retention prune at startup (archive-then-unlink); per-task terminal archive; oversize logs retained live | live tails lost; archived blobs survive |
| `projects/<id>/**` (knowledge, journal, workpad, reflections, `.project.json` marker) | `ouroboros/project_facts.py`, `projects_registry.py` | marker unversioned; registry stamped (§2) | NEVER age-pruned (owner curates; delete = durable tombstone) | per-project memory lost; registry row survives, room reappears empty |
| `projects/<id>/knowledge_history.jsonl` | `ouroboros/knowledge.py` through the shared knowledge write lock | append rows retain source topic, revision, operation facts and the host stamp; the former `knowledge_journal.jsonl` size telemetry beside it (global and project) is no longer written — an existing file is inert | follows the owning project shelf; retained with the project until explicit deletion | history/provenance lost while authored project notes remain |
| `archive/**` (rotated segments, `rescue/`, `usage_import/`, `managed_repo/`) | rotation + `supervisor/git_ops_rescue.py`, `usage_legacy_import.py`, `launcher_bootstrap.py` | segments inherit source shape; usage_import carries sha256 sidecar | UNBOUNDED BY DESIGN — durable history, never GC'd (P1) — accepted | memory horizon truncated; rescue copies of uncommitted work destroyed |
| `archive/usage_ledger/segment_*.jsonl` | `ouroboros/usage_compaction.py` (exact pre-compaction ledger bytes, written + fsync'd BEFORE the live swap) | each segment is a whole valid ledger generation; hash-pinned by the live `usage_baseline` header (`source_sha256`), chained recursively through each segment's own leading header | UNBOUNDED BY DESIGN — the folded monetary history, never GC'd (P1); read via `archived_attempt_ids` (tamper-evident, per-attempt joins for the model-send reverse sweep) | folded per-attempt monetary history unrecoverable; live aggregates (baseline block) survive, but seal/attempt joins for folded ids break — the mirror of the ledger row: deleting the ARCHIVE alone under a stamped ledger is a typed chain break on every history question, deleting the LEDGER alone raises `generation newer` for surviving newer, non-prefix segments; once fresh compactions reach those generations, old unreferenced segments are skipped and their attempt IDs are absent ([history readers](USAGE_COMPACTION.md#10-history-readers-model-send-reconciliation-and-audits)); reset both together |
| `observability/{calls,blobs,salvaged}/**` | `ouroboros/observability.py` (private 0700/0600, CAS gzip); model-send records beside the call manifests: `model_send_seal` block + write-once `<attempt>.model_send_violation.json` typed facts (`ouroboros/model_send_seal.py`) | call manifests `schema_version: 1` + custody/redaction honesty markers; blob refs sha-verified on read; `model_send_seal.seal_version: 1` with the `canonical_json_v1` basis string | preserved indefinitely BY CONTRACT (the startup census counts, never deletes); the inert `OUROBOROS_OBSERVABILITY_RETENTION_DAYS` knob is RETIRED (key in `RETIRED_SETTING_KEYS`) | every recorded `result_ref`/`manifest_ref` dangles (strict readers raise); pending delegated request bodies become unknown, while custody identity still blocks replacement and protects snapshots; replay refuses without the recorded body; salvaged outputs unrecoverable; a lost seal on a seam-dispatched attempt surfaces as a typed `unlogged_attempt` fact at the next startup sweep |
| `claudexor/**` | EXTERNAL writer — the claudexord daemon (Ouroboros only mkdirs, appends `daemon.log`, writes `ouroboros-owned.json` marker) | marker unversioned | daemon-owned; grows unbounded under our root — disclosed external plane | owner harness logins/profiles lost (fresh device-auth required) |
| `playwright-browsers/` | `ouroboros/tools/browser.py` (vendor install) | none — vendor tree | no GC — accepted (vendor cache) | re-downloaded on next browser use |
| `cache/pip`, `cache/uv` | `ouroboros/test_environment.py` | package-manager formats; TEST ROOT ONLY, not live data | disposable verification-root lifetime | recreated by explicit dependency setup |

## Reset ladder (summary)

Always safe (pure caches, recreated): `state/pycache`, `state/code_intel`,
`state/evolution_metrics_cache.json`, `playwright-browsers/`, `state/cx`,
`state/betterleaks`, lock files, `state/server_port`.
Safe with bounded cost: `WORLD.md` (regenerates),
`state/usage_import_watermark.json` (safe re-import),
`ui_preferences.json`, `auth_secret.key` (one re-login).
Fail-closed losses (system stays correct, work/authority is forgone):
skill state dirs, `advisory_review.json`, `capability_evidence.json`,
`pending_restart_verify.json`.
Dangerous (authority/history destruction): `settings.json`,
`state/usage_attempts.jsonl`, `task_results/**`, `logs/events.jsonl`,
`memory/**` (including `dialogue_meta.json`: deletion erases cursor and pending
nomination obligations; re-consolidation cannot reconstruct the old IDs),
`archive/**`, `observability/**`, `state/subagent_worktrees.json`
(leak), `claudexor/**`, `state/python-userbase` (real deps).