ouroboros/devtools/benchmarks/swe_bench
Ouroboros f72d6be25f release 6.100.0: delegated runs execute in private snapshots — capture, disposition, and GC carry one honest truth (sprint phase C)
Phase C of the poltergeist delegation sprint, squash-landed onto the v6.99.0
nanny release. A mutating delegated run never edits the shared tree: the host
provisions a private execution snapshot of the authority target (baseline built
with the sensitive-veto decided before anything is hashed, pinned by a
refs/ouroboros/delegated/ ref), records the durable binding {execution_root,
baseline_sha, target_root, authority_source} on the custody rows BEFORE the
POST, and replays it byte-identically on explicit retry — pending-invocation
recovery carries the full binding so the startup GC (settled && patch_disposed)
never deletes the snapshot holding the child's only work. Terminal
reconciliation captures a settled run's diff through one drive-rooted capture
core, eagerly only at PROVEN terminality; an absent or unreadable close
captures nothing, capture-at-disposition is the retry point (C1-R2), a capture
failing there is the typed INTEGRATE_DELEGATED_CAPTURE_FAILED refusal for both
decisions, and patch_captured means a usable artifact exists (C1-R3 — a failed
manifest never mints the row and reject re-checks before releasing the
snapshot). Nothing lands without the explicit integrate_delegated_patch
apply/reject flow (proven baseline drift, NUL-safe touched paths, cleanup
following the durable disposition row, the protected-path gate scoped to the
Ouroboros body), and open obligations surface via undisposed_patches. Beside
it: SSOT cost projection (accounted_upper_bound_usd beside deprecated
cost_usd, $0-fabrication fixes, honest web cost presentation),
delegated_runs_failed on the evidence receipt, notification chat routing,
byte-accurate argv/env budgeting with --prompt-file transport, and hash-bound
skill repair.

Merge resolution UNIONS phase B's delegate module split (delegate_shared stays
the one refusal/emit/ownership author; delegate_integration imports it),
delegate_answer beside integrate_delegated_patch on every child surface, the
instructions-bearing idempotency digest, and BR2's additive terminal_detail
carry in cancel_and_verify with C's absent-branch no-capture semantics. Size
gates cleared by extraction, no grandfathering: delegate_evidence.py
(task_execution_evidence out of delegate_custody) and synthesis_cost_text.py
(synthesis cost renderers out of agent_task_pipeline), both re-exported
same-object; _delegate_start hands its started payload to one extracted
author.

The formal six-lane exact-SHA gate plus two verified fix rounds (CR1: the
startup GC fails closed over an unreadable custody log, the capture artifact
is readable across a split-drive boundary through a narrow rebind, a durable
apply intent precedes any tree mutation so a crash replay can never record a
false rejection, and delegate_start tells the snapshot truth; CR2: the
AMBIGUOUS apply state gained an explicit owner-acknowledged exit through the
normal disposition guards, a lost prune-skip row is escalated loudly, and a
failed verdict write can no longer strand a pending intent) are squashed into
this landing.

Co-authored-by: Ouroboros <311266734+ouroboros-agent@users.noreply.github.com>
2026-08-12 17:54:10 +03:00
..
__init__.py feat(devtools-benchmarks): add official benchmark harnesses and workspace executor 2026-06-06 12:03:30 +03:00
presets.py feat(devtools-benchmarks): add official benchmark harnesses and workspace executor 2026-06-06 12:03:30 +03:00
README.md feat(devtools): add auditable benchmark ledgers 2026-06-07 09:44:20 +03:00
swebench_predictions.py release 6.100.0: delegated runs execute in private snapshots — capture, disposition, and GC carry one honest truth (sprint phase C) 2026-08-12 17:54:10 +03:00

SWE-bench Devtools

These helpers generate official SWE-bench prediction JSONL files for a set of already-prepared local task checkouts.

They do not download datasets, reset repositories, or score benchmark results. Evaluation remains the official SWE-bench harness:

python -m swebench.harness.run_evaluation \
  --dataset_name princeton-nlp/SWE-bench_Verified \
  --predictions_path /path/to/predictions.jsonl \
  --max_workers 1 \
  --run_id ouroboros

Prediction rows produced by swebench_predictions.py contain only:

{"instance_id": "...", "model_name_or_path": "...", "model_patch": "..."}

Supported preset aliases in presets.py:

  • full -> princeton-nlp/SWE-bench
  • lite -> princeton-nlp/SWE-bench_Lite
  • verified -> princeton-nlp/SWE-bench_Verified

Input rows must provide instance_id, a clean git workspace_root or --workspaces-root, and problem_statement or prompt. If base_commit is present, the helper refuses to run unless the checkout HEAD matches it.

The helper also writes sidecar artifacts next to the predictions path unless explicit paths are supplied:

  • <predictions>.ledger.jsonl records every requested instance, including invalid input, dirty workspaces, timeouts, non-zero Ouroboros exits, and empty patches.
  • <predictions>.errors.jsonl contains failure rows for operator debugging.
  • <predictions>.run_manifest.json records provenance and official eval command shape.

The official predictions JSONL intentionally omits failed/empty-patch rows because the SWE-bench harness expects only prediction records. The ledger is the denominator-preserving Ouroboros audit artifact.