e879ffc675
Every claim a benchmark run makes about itself is now either enforced before money is spent or refused with a durable typed record. Admission is the outer boundary, and that is a tested property, not a convention: `admit_benchmark_run()` builds the manifest, WRITES it, and only then lets the seed gate enforce, raising `BenchmarkAdmissionRefused` with the refusal already on disk. A `finalize_run_manifest()` context manager records a typed outcome, a typed refusal/error and the REAL exit status on every exit path, including an escaping exception (an integer `SystemExit.code` is preserved). All seven migrated launchers route through both seams; none calls the builder directly; and an `ast` guard in the seam meta-test fails if any denylisted operation — filesystem assertions, docker, subprocess, state dumps, writers — precedes admission. The guard was verified to bite: against the pre-fix tree it reports four violations, two of which no review round had reached. Refusals are one family. `BenchmarkAdmissionRefused`, `RuntimeAttestationRefused` and `SeedShapeRefused` are all `RuntimeError` subclasses carrying a typed reason, deliberately not `SystemExit` — a `SystemExit`-raising assertion had made a refusal handler dead code, because `SystemExit` is not an `Exception` subclass. A behavioural parity test computes what `raise SystemExit(main())` hands the OS for each launcher and asserts it EQUALS the `exit_code` the manifest records, so a recorded status can no longer diverge from reality. `runtime_attestation()` records both facts about a live server — the HTTP `runtime_version` from the frozen `/api/health` contract and the local HEAD/VERSION of its checkout — and requires the contracted field: a bare `version` key now refuses with the non-overridable `runtime_version_absent`, so an unrelated HTTP server can no longer attest. `OBO_ALLOW_EVOLVED_VOLUME` waives only `runtime_skew`, in the shell as well as in Python, and commit availability is decided before skew so a missing commit cannot be mislabelled. Lineage is a descent check (`merge-base --is-ancestor`), never equality, so an evolution run legitimately moving HEAD forward is not corruption. `CAMPAIGN_FATAL_PROVENANCE_REASONS` is a single shared authority consumed by both SWE-Pro drivers, which now stop the schedule immediately — before any volume archival — instead of one driver burning every task. Grading gains a third state: `pass|fail|ungraded` with typed reasons beside the UNCHANGED headline formula, plus `grade_summary.json`, and the probe orchestrator reads that artifact instead of scraping log tokens and refuses to publish a grade that a failed or stale grader produced. ProgramBench's ledger is append-only at both the run root and the instance directory, skip rows included. `common/manifests.py::write_json` is atomic and byte-identical, importing `ouroboros.utils` lazily so the module stays stdlib-only for the container-side harbor agent, and `openrouter_key_remaining()` reads the authoritative `limit_remaining`. Egress isolation was built in this phase and is NOT part of this release: it is off by default per owner decision, it produced a finding in five separate review rounds, and round 10's was a direct recurrence of round 5's, so it was extracted and deferred with its diagnosis preserved. Solve containers run the same open network as before v6.75.0. The measured evidence stays in METHODOLOGY: the official SWE-bench Pro harness does not regulate the solve container's network at all (`--block_network` applies to the EVAL container), and 31 of 287 attempted instances are musl/Alpine, 84% of which previously produced a non-empty patch. GAIA, CL-Bench, Terminal-Bench and OSWorld keep their pre-v6.75.0 behaviour here and migrate in their own phases. No `ouroboros/` runtime behaviour changes. Reviewed through eleven rounds of the production commit gate (advisory, triad `claude-fable-5` + `gpt-5.6-sol` + `gemini-3.5-flash` at high effort, scope `claude-fable-5`): 24 findings, 22 fixed and 2 refuted with the evidence recorded in the changelog. Per-phase tags are deliberately not created — a single release tag lands with the campaign synthesis, an owner-approved deviation from the tag-per-commit norm. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
8.4 KiB
ProgramBench
Ouroboros adapter for official ProgramBench
cleanroom execution. Scoring always goes through the official programbench CLI.
Modes
End-to-end (run_programbench_e2e.py)
Full harness: pull task_cleanroom_v6 image, seed a host workspace, start a no-network
Docker backend, submit an Ouroboros task with executor_ref + protected_artifacts,
wait for completion, write submission.tar.gz, optionally run programbench eval.
cd <workspace>/repo
source .venv/bin/activate
export OUROBOROS_REPO_DIR=<workspace>/repo
export OUROBOROS_DATA_DIR=<workspace>/data
export OUROBOROS_BENCH_RUNS_ROOT=<workspace>/bench_runs
# Model slots: the server reads OUROBOROS_DATA_DIR/settings.json. For a pinned
# ProgramBench profile merge devtools/benchmarks/programbench/settings_base.json
# into data/settings.json before starting the server (secrets are blanked in the
# template; add the real provider key yourself, never commit it).
#
# settings_base decisions:
# - OUROBOROS_TASK_REVIEW_MODE=required so adaptive multi-pass improvement after
# task_acceptance_review actually runs on headless bench tasks.
# - OUROBOROS_RUNTIME_MODE=pro (container bench), OUROBOROS_MAX_WORKERS=4,
# OUROBOROS_SAFETY_MODE=light.
# - The solve task itself disables claude_code_edit (house rule: benches measure
# the single-model Ouroboros harness, not an external coding agent).
# - Model slots pin openai/gpt-5.5: the id must exist in the OpenRouter catalog
# (there is no openai/gpt-5.5-mini there — it 400s on every task).
# Terminal 1 — source server (not the desktop app; the sandboxed desktop cannot
# see bench_runs/ workspaces)
python server.py --host 127.0.0.1 --port 8770
# Terminal 2 — smoke (3 easy tasks)
python devtools/benchmarks/programbench/run_programbench_e2e.py \
--difficulty easy \
--slice 0:3 \
--solve-model openai/gpt-5.5 \
--eval \
--ouroboros-url http://127.0.0.1:8770
Useful flags:
--instance-id <id>— single task--solve-model <id>— expected OUROBOROS_MODEL; validated against--settings-path--dry-run— docker +ouroboros_task_body.jsononly (no solve)--skip-pull— reuse already-pulled images--redo-existing— rerun even whensubmission.tar.gzexists (clears the task checkpoint)--timeout-sec 21600— per-task wall clock (default; matches official 6h budget)--cpus 4 --memory 16g— container resources (lower than mini-swe baseline for Mac)
Prepare-only (run_programbench.py)
Writes ouroboros_task_body.json and packages an existing workspace when you already
have a running cleanroom container. Does not submit to the gateway.
Export (export_programbench_submissions.py)
Copies submission.tar.gz files into a minimal eval-compatible tree for targeted
programbench eval runs.
Budget and pacing
The task body carries metadata.budget_profile (see schemas.programbench_budget_profile);
POST /api/tasks normalizes it additively into task_contract.budget_profile
(there is deliberately no top-level task_contract field on the gateway):
improvement_policy=until_deadline,max_improvement_passes=6(the explicit cap remains authoritative under every policy),reserve_finalization_pct=15(0–100 pct),stall_rounds_threshold=12.- The 6h official budget flows through the body's
timeout_sec(gateway →deadline_at); round caps come from settings (OUROBOROS_MAX_ROUNDS).
Reliability (lessons from the 0/5 debug run)
- Model-id preflight: direct-provider routes reject legacy
provider/modelids (they needprovider::model). The runner validates every configured model slot via the runtime'smigrate_model_valuebefore any task burns and fails fast with the exact remediation. With an OpenRouter key,provider/modelis canonical and left untouched. - Checkpoints:
submit_and_waitpersists the latest task result to<instance>/ouroboros_task_checkpoint.jsonatomically on every poll; a restarted harness re-attaches to the recorded task_id instead of re-submitting, so a crash or client timeout does not discard hours of in-flight agent work. - Status inference: terminal detection reads the payload's explicit
statusagainst the settled set (completed/failed/cancelled/rejected_duplicate) and infra failures are classified fromreason_code/outcome_axes.execution(harbor-adapter pattern) — never heuristics over error text.
Invariants
- Use official
programbenchCLI for evaluation and summaries. - Use
task_cleanroomtask images; do not score locally. - Tool execution for the benchmark workspace runs in a no-network Docker backend.
- Reference binaries are declared through
resource_policy.protected_artifacts: execute is allowed; byte reads, copy/hash/static introspection/tracing/debugging are denied. - Submission artifact is
<run>/<instance_id>/submission.tar.gz. - Sidecars (
run_manifest.json,result_index.jsonl) are audit artifacts only. result_index.jsonlis APPEND-ONLY (v6.75.0). EVERY row — processed and skipped alike — is appended the moment it exists, at the run root AND in the instance dir, so a resumed instance's own history contains the skip event that explains the gap. Before this the run root ledger was one whole-file rewrite at the very end, so a crash discarded every completed instance and a resume silently replaced the previous run's history. Consequences for readers: a resumed or re-done instance appears MORE THAN ONCE, so dedup byinstance_idand take the LAST row; the file is run history, not a set. The denominator still comes from the requested selection (run_manifest.json→requested_task_ids), never from the line count.run_manifest.jsonis written FIRST (v6.75.0), right after selection and before the first instance — it carries the shared seed-provenance gate (benchmark_run_manifest(require_clean=True), escape--allow-dirty-seed) plus the runtime attestation of the server named by--ouroboros-url(the HTTPruntime_versionand the local HEAD of--repo-dir; a skew hard-stops unlessOBO_ALLOW_EVOLVED_VOLUME=1). It is written to disk as soon as it is built, so a refusal after admission still leaves a durable record of what was refused, and it is rewritten at the end with the eval status and the completed/failed counts. Previously it was written only after every instance AND the official eval, i.e. after all the spend, so the gate could not stop an unreproducible run.
Mac notes
ProgramBench images are linux/amd64 only. On Apple Silicon use Docker Desktop or
Colima (colima start --arch x86_64 --cpu 4 --memory 16) with
DOCKER_HOST=unix://$HOME/.colima/default/docker.sock if needed. The e2e runner
auto-sets DOCKER_HOST when a Colima socket is present.
Use the source Ouroboros server on port 8770 for benchmarks. The packaged
desktop app on 8765 is sandboxed and cannot access bench_runs/ workspaces.
Official programbench eval needs Python 3.11+ (typing.Self). The adapter
auto-picks python3.12/python3.11 with programbench installed, or set
PROGRAMBENCH_PYTHON=/path/to/python3.11. Eval defaults to --docker-cpus 4
(override with PROGRAMBENCH_DOCKER_CPUS).
Run dirs default under bench_runs/; set OUROBOROS_BENCH_RUNS_ROOT to redirect
(the test suite pins it to a temp dir so runs never leak into a developer's
bench_runs/).
Full-run gate (gate-20, v6.56.0) — operator methodology
This is an operator procedure, not an automatic runner feature: the
ProgramBench runner has no built-in gate, and the steps below are enforced by
the operator (and the campaign watcher) around a normal run, not by
run_programbench_e2e.py itself.
Begin a full 200-task run with a 20-task calibration gate: the 10 smoke tasks plus 10 unseen tasks. Score PAIRED per-task against the official leaderboard's per-task results (matched by instance id), not aggregate-vs-aggregate. Pass criteria: paired mean on these 20 >= the reference harness's paired mean on the same 20, AND not worse than the Codex baseline on the shared subset. On PASS, continue to the full 200; on FAIL, the operator stops the run and reports (on an overloaded host, pause instead of failing — attach load data — and re-run when contention clears). Record the gate result, task list, and per-task pairs alongside the run so a published 200-task number can always be traced back through its gate.