ouroboros/devtools/benchmarks
Ouroboros ae53779ccd release: Ouroboros v6.26.0-rc.1 — systemic hardening: immune system, custody, memory integrity, native multimodal chat
Owner-approved big release (plan: большой_релиз_v6.26.0). Workstreams:

- WS-A lint gate: ruff F-rules step in CI quick-test; fixed the real F823
  NameError class (supervisor/events.py utc_now_iso shadowing); make lint/health.
- WS-B memory integrity: atomic dialogue_blocks with corrupt-quarantine
  (.corrupt-<ts>.bak + memory_store_corrupt event); honest scratchpad journal
  (block_append_failed, corrupt storage renders as corruption not "(empty)");
  full Pattern Register window (16K cap vs 3K cut); merge-aware scratchpad
  consolidation under the sidecar lock; backlog writes through locked helpers;
  chat omission notes ("[N older unconsolidated messages omitted]").
- WS-C provider SSOT: provider registry (prefixes/credentials/resolution) in
  provider_models.py replacing 5 duplicated knowledge sites; credential-aware
  consolidation model; pricing empty-fetch retry; direct-route correctness
  (o-series max_completion_tokens, reasoning_effort, Anthropic error bodies,
  per-request timeouts on cached clients); deep/plan review budgets reserve
  output headroom inside the 1M window (min(SSOT, window − output − margin)).
- WS-D races/state: utils.update_json_locked (locked RMW, loud TimeoutError);
  task_results merge under per-file lock (cancel-latch safe); update_state
  migration for owner binding, evolution counters, budget_line, post-task
  activation; queue-lock coverage (enqueue, timeouts, worker health, task_done,
  snapshot); visible supervisor death (supervisor_error + owner notice + ghost
  consciousness/chat-agent cleanup on re-init); chat.jsonl rotation via
  os.replace under the append lock; unbounded _outbox removed; host-service
  port probe + actual-port panic sweep; skill lifecycle lane deadline
  (OUROBOROS_SKILL_LIFECYCLE_TIMEOUT_SEC, dedupe-leak-safe); consilium
  force-plan structured flag (plan_review_aggregate from the FULL result);
  atomic writers (update-intent, repo-manifest, metrics cache, post-task pair);
  review_state.save_state raises on lock timeout (advisory ledger honesty);
  apply_pending_request no-ops while a campaign is active;
  api_update_apply kills workers only AFTER update validation, with respawn
  on aborted checkout and "interrupted" terminal status.
- WS-E immune hardening: triad anti-refusal coverage contract (empty array
  needs NO_FINDINGS sentinel or bare-[] body; refusal prose with [] is
  parse_failure and never enters quorum); BIBLE P3 "Owner-chosen enforcement,
  loud advisory" bound + CHECKLISTS sync; every advisory pass-through of a
  blocking signal writes review_advisory_override + persistent
  advisory_overrides (surfaced by review_status); ToolEntry.mutates_worktree
  with dispatcher-level before/after worktree diff invalidation (covers error
  paths; read-only runs no longer invalidate; redundant manual calls removed);
  scope review fails closed without its checklist; advisory final write is a
  locked re-read merge; synthesis severity defaults to critical with WARNING
  fallbacks; find removed from SAFE_SHELL_COMMANDS; anti-thrashing state
  survives advisory criticals; preflight env scrubs secret-class variables;
  .git/index.lock age gate in worker startup checks.
- WS-F security: file-browser symlink containment on the RESOLVED path across
  all endpoints (out-of-root symlink targets are listed but inert; tests
  rewritten to the new contract deliberately); HMAC-signed session cookies
  (server-side persisted key, 30-day TTL, Secure on TLS) replacing the
  permanent password-derived cookie; password-class settings mask to a
  constant placeholder; conservative SSRF guard for the MAIN agent (link-local
  /cloud-metadata only, LAN stays reachable, per-request route re-validation);
  ClawHub zip-slip hardening (":"/backslash segments rejected + post-join
  containment) and lazy no-proxy OuroborosHub opener; single-execution
  signature dispatch for extension handlers (no TypeError re-run after side
  effects); onboarding postMessage origin checks; SHA256-pinned
  python-standalone download with pipefail; payload-resident dependency
  fingerprints only corroborate durable deps.json; skill payload re-hash
  immediately before spawn (TOCTOU narrowing).
- WS-G process custody: ouroboros/process_custody.py — spawn_supervised
  chokepoint + durable data/state/process_ledger.jsonl (pid, pgid,
  fingerprint{start_time, cmd_sha256}, purpose, scope task|session|daemon,
  owner_task, session_id); platform_layer.process_start_time primitive;
  startup + periodic reaper killing ONLY strict-fingerprint matches from dead
  generations/tasks (never by command-line class); migrations: services
  (the orphan hole), workspace executor + local model + extension companions
  (ledger write-through), worker_pids (write-through; legacy path retained);
  parent lifelines (ppid watchdog, group-suicide only as group leader) in
  worker_main, extension runner, Claude readonly child; conformance test
  pinning the Popen allowlist; ARCHITECTURE/DEVELOPMENT/CHECKLISTS entries.
- WS-H native multimodal chat: supports_vision capability map (static
  prefixes + OpenRouter /models input_modalities overlay); web chat uploads
  ride the WS frame as structured attachments (additive ChatInbound field)
  and image uploads become NATIVE image blocks via the existing Path B;
  browser screenshots inject natively for vision models via the multipart
  user-merge (tool result stays a string; file persisted under
  data/uploads/screenshots for re-view); K=3 newest-image eviction with
  caption placeholders carrying the vlm_query re-view path; image-aware token
  estimates (fixed ~1.1K-token equivalent instead of base64 length, fixing
  permanent emergency-compaction wedges); compaction renders images as
  captions (no base64 into the summarizer); GigaChat/local lanes emit explicit
  "[image omitted: model has no vision]"; internal _caption/_source_path
  metadata stripped from provider payloads.
- WS-I housekeeping (partial): SETTLED_STATUSES SSOT (+ cycle-safe mirror pin);
  owner_inject.py renamed to owner_mailbox.py; version-neutral envelope
  wording; files.py import-block cleanup. Remaining WS-I/WS-J/WS-K items are
  deferred with the owner's context-budget priority on review+release.

Review notes: triad+scope ran via scripts/run_external_review.py on the core
pack across 3 rounds to convergence (scope responded=PASS each round; round-2
criticals fixed: update_apply kill-order + respawn, lifecycle dedupe leak on
lane timeout, OUROBOROS_MAX_ROUNDS hot-reload + docs, toggle_evolution
NameError, async Anthropic timeout forwarding, supports_vision local check,
budget-update lock visibility, ChatInbound additive attachment contract,
file-browser doc sync). The FULL combined diff exceeds every triad model's
context window (~1.59M tokens > 1.05M) — reviewed in packs; remaining
cross-pack findings were verified as slicing artifacts. Adversarial critics
(GPT; Gemini/Opus rounds) ran on the working tree. Deliberate tradeoffs:
metadata-based eviction captions (no light-LLM call in the hot path);
worker ledger records use live-cmdline fingerprints with a synthetic-arg
fallback only where the OS offers no cmdline.
2026-06-10 16:08:53 +03:00
..
common fix(self-evolution): harden rc.3 server-driven evolution path 2026-06-09 04:24:31 +03:00
osworld feat(devtools): add auditable benchmark ledgers 2026-06-07 09:44:20 +03:00
programbench feat(devtools): add auditable benchmark ledgers 2026-06-07 09:44:20 +03:00
swe_bench feat(devtools): add auditable benchmark ledgers 2026-06-07 09:44:20 +03:00
swe_bench_pro release: Ouroboros v6.26.0-rc.1 — systemic hardening: immune system, custody, memory integrity, native multimodal chat 2026-06-10 16:08:53 +03:00
terminal_bench feat(devtools): add auditable benchmark ledgers 2026-06-07 09:44:20 +03:00
__init__.py feat(devtools-benchmarks): add official benchmark harnesses and workspace executor 2026-06-06 12:03:30 +03:00
evolve_smoke.py fix(self-evolution): harden rc.3 server-driven evolution path 2026-06-09 04:24:31 +03:00
README.md feat(devtools): add auditable benchmark ledgers 2026-06-07 09:44:20 +03:00

Benchmark Devtools

This directory contains thin adapters around official benchmark harnesses. The adapters prepare Ouroboros tasks, capture artifacts, and preserve traces; they do not replace official scoring.

Supported surfaces:

  • ProgramBench: official programbench eval/info and cleanroom submission layout.
  • Terminal-Bench: Harbor installed-agent adapter.
  • SWE-bench and SWE-bench Verified: standard predictions JSONL for the official SWE-bench harness.
  • SWE-bench Pro: official Pro eval wrapper and patch-capture methodology. The prediction/capture path is ledgered; the grader wrapper stays official-output-only.
  • OSWorld: stop-closed skeleton and logs normalizer only.

Shared sidecar schemas:

  • Run manifests record non-secret provenance: requested task ids where the benchmark runner exposes them before execution, requested counts/selection slots for deterministic first-N runs such as Terminal-Bench, exact argv, official command shape, output paths, model slots, source commit, dirty-state counts, and hashes. Defaults are adapter-specific (run_manifest.json, <predictions>.run_manifest.json, or osworld_preflight.run_manifest.json).
  • Result ledgers are denominator-preserving Ouroboros JSONL files. They record every requested instance, including setup failures, timeouts, and empty patches, even when the official benchmark prediction/submission format only accepts successful rows. Defaults are adapter-specific (result_index.jsonl, <predictions>.ledger.jsonl, or osworld_preflight.ledger.jsonl).

These sidecars are audit artifacts, not replacement scoring. Official benchmark harnesses and official result files remain the scoring authority.