Two provenance fixes in the same class as the release itself, plus the
round-11/12 review findings that land in the same files.
FIX A — an isolated benchmark container carries only the providers the run
declared. `build_isolated_settings` copied EVERY provider credential present
in the live settings file regardless of which providers the run's model slots
declared, so a run pinned to OpenRouter still received direct ANTHROPIC /
OPENAI / Cloud.ru / GigaChat keys. Two consequences: a routing fallback could
spend outside the declared bucket while the manifest said otherwise, and the
reachable provider set was a function of whatever happened to be in the live
file at launch — a pinned seed that pins the code but not the environment is
not reproducible. Provider credentials are now gated on the run's declared
slots, derived from the routing SSOT (`provider_models.PROVIDER_PREFIXES` /
`provider_for_model`, the same registry `llm._resolve_remote_target` routes
on), and travel in whole groups so a key never arrives without the endpoint or
auth fields it is useless without. Owner/control secrets were never copied and
still are not. Ambiguity fails OPEN (carry a spare, disclose it) — never
closed. `benchmark_run_manifest` now records `provider_credentials`: which
credentials the container actually received, by fingerprint, never by value.
FIX B — a task the cost rail truncated says so. `usage_accounting.reserve_-
attempt` refuses on a worst-case reservation bound that reached a $6.00 rail at
$0.45 of actual spend in the v6.81.0 OSWorld smoke, stopping two of three tasks
at 13 and 22 rounds; the artefacts published `status=completed`,
`reason_code=official_evaluate` and the string `budget_exhausted` appeared
nowhere. `task_result_row` gains an always-present `runtime_outcome` projected
by the new shared `runtime_terminal_disclosure`, and every writer holding a
runtime task result now publishes it: OSWorld cu_bridge, SWE-bench and
ProgramBench success rows (the failure rows already did), the Terminal-Bench
in-container summary and disclosure ledger, the GAIA solver, the harness-bench
wrapper, and the CL-Bench per-question writer. Reward, `official_eval_status`
and adapter-stage `status` are untouched: disclosure ADDED, fact not
subtracted. TB's ledger gains a `cost_truncated` bucket — `genuine` asserts a
fair shot, which a rail-truncated trial did not get.
Review findings folded in:
- `_amend_manifest` emitted `output_paths.task_outcome` unconditionally, so the
finalized attempt manifest kept pointing at an outcome whose write failed.
The previous round fixed the ledger row and left the manifest lying; both
sides now follow the same rule.
- `ADAPTER_PATCH_MARKERS` keyed two of three detections on bare env-var names,
which the unpatched adapter may mention in a comment or a `-e` passthrough
list. That false positive OVERSTATES enforcement. Markers are now
patch-unique tokens and the uniqueness requirement is recorded beside them.
(One marker legitimately covers all three env knobs: they arrive in one loop
in one hunk.)
- CLB fidelity overstated enforcement on the DEFAULT `--path standard`:
`_docker_launcher.submit()` hardcodes `disabled_tools: []` and never imports
the patched bridge module, so the evidence is now entrypoint-specific.
- `runtime_attested` renamed to `runtime_attestation_available`: it is a tree
probe, and a definition existing is not evidence that it ran.
- README version badge alt text said 6.80.0 while the URL said 6.81.0, which
`version_carrier_desyncs` flags and the advisory preflight blocks on.
Bug-pinning tests inverted, and said so in the test docstrings:
- `test_dry_run_claims_attestation_only_when_the_patch_is_in_the_execution_-
clone` asserted `runtime_attested is True` for a DRY RUN against a clone that
merely contained an `_attest_runtime` definition — it demanded the false
positive as the contract.
- the CLB fidelity fixtures wrote bare env-var names as "the patch", which is
precisely the marker weakness above.
Docs corrected where the code falsified them: OSWorld METHODOLOGY §7.4 claimed
`OUROBOROS_MAX_ROUNDS` plus the timeout were the only per-task caps (the USD
rail binds first), §6 now says scoring reads `official_eval_status` /
`details.outcome_status` rather than filtering on `status == "completed"` and
that `output_paths.task_outcome` may be absent; CLB METHODOLOGY §3 documents
when an exported runtime mode overrides the adapter's hard-set `advanced`, and
§6 distinguishes the `--runner-path` adapter checkout from the
`--ouroboros-clone` execution seed and points at the field the code actually
writes.
Known ordering debt noted in place for the v6.82 backlog, deliberately not
restructured here: `_auto_sync_release_metadata_if_needed` runs ~87 lines after
the `_release_metadata_preflight` gate it would satisfy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six reviewed phases land as one release.
Admission is the outer boundary: every migrated launcher records a manifest before
it can touch the filesystem, and finalizes a typed outcome on every path — success,
refusal, crash, and the real exit status. A structural audit enforces that boundary
across all fourteen launchers, together with confinement computed from the active
checkout and a single manifest publisher, judging by effect rather than by callee
name and failing closed on any write form it cannot resolve.
Harness exit codes are no longer trusted as run status: inspect returns zero for an
eval that raised and harbor returns zero for a job whose trials all errored, so the
launchers now read the harness's own artefact and keep "the harness failed", "it
scored nothing" and "it scored honest zeros" distinguishable.
The acceptance dialogue reconciles receipts through one typed identity that is an
equivalence by construction, so a passing check can no longer clear a red it never
addressed. Prompt caching is normalized at every send site and cached calls stop
under-reporting their input. The owner's context mode becomes explicit and
fail-closed, with one enforcement point for every writer of a disk-authored setting.
Deliberate limits are disclosed in each bench's METHODOLOGY.md rather than implied
by silence. Isolated benchmark egress and the multi-lane script generator are
deferred to a later release with restoration patches and carry-forward notes.
Every claim a benchmark run makes about itself is now either enforced before
money is spent or refused with a durable typed record.
Admission is the outer boundary, and that is a tested property, not a
convention: `admit_benchmark_run()` builds the manifest, WRITES it, and only
then lets the seed gate enforce, raising `BenchmarkAdmissionRefused` with the
refusal already on disk. A `finalize_run_manifest()` context manager records a
typed outcome, a typed refusal/error and the REAL exit status on every exit
path, including an escaping exception (an integer `SystemExit.code` is
preserved). All seven migrated launchers route through both seams; none calls
the builder directly; and an `ast` guard in the seam meta-test fails if any
denylisted operation — filesystem assertions, docker, subprocess, state dumps,
writers — precedes admission. The guard was verified to bite: against the
pre-fix tree it reports four violations, two of which no review round had
reached.
Refusals are one family. `BenchmarkAdmissionRefused`, `RuntimeAttestationRefused`
and `SeedShapeRefused` are all `RuntimeError` subclasses carrying a typed
reason, deliberately not `SystemExit` — a `SystemExit`-raising assertion had
made a refusal handler dead code, because `SystemExit` is not an `Exception`
subclass. A behavioural parity test computes what `raise SystemExit(main())`
hands the OS for each launcher and asserts it EQUALS the `exit_code` the
manifest records, so a recorded status can no longer diverge from reality.
`runtime_attestation()` records both facts about a live server — the HTTP
`runtime_version` from the frozen `/api/health` contract and the local
HEAD/VERSION of its checkout — and requires the contracted field: a bare
`version` key now refuses with the non-overridable `runtime_version_absent`, so
an unrelated HTTP server can no longer attest. `OBO_ALLOW_EVOLVED_VOLUME` waives
only `runtime_skew`, in the shell as well as in Python, and commit availability
is decided before skew so a missing commit cannot be mislabelled. Lineage is a
descent check (`merge-base --is-ancestor`), never equality, so an evolution run
legitimately moving HEAD forward is not corruption.
`CAMPAIGN_FATAL_PROVENANCE_REASONS` is a single shared authority consumed by
both SWE-Pro drivers, which now stop the schedule immediately — before any
volume archival — instead of one driver burning every task.
Grading gains a third state: `pass|fail|ungraded` with typed reasons beside the
UNCHANGED headline formula, plus `grade_summary.json`, and the probe
orchestrator reads that artifact instead of scraping log tokens and refuses to
publish a grade that a failed or stale grader produced. ProgramBench's ledger is
append-only at both the run root and the instance directory, skip rows included.
`common/manifests.py::write_json` is atomic and byte-identical, importing
`ouroboros.utils` lazily so the module stays stdlib-only for the container-side
harbor agent, and `openrouter_key_remaining()` reads the authoritative
`limit_remaining`.
Egress isolation was built in this phase and is NOT part of this release: it is
off by default per owner decision, it produced a finding in five separate review
rounds, and round 10's was a direct recurrence of round 5's, so it was extracted
and deferred with its diagnosis preserved. Solve containers run the same open
network as before v6.75.0. The measured evidence stays in METHODOLOGY: the
official SWE-bench Pro harness does not regulate the solve container's network
at all (`--block_network` applies to the EVAL container), and 31 of 287 attempted
instances are musl/Alpine, 84% of which previously produced a non-empty patch.
GAIA, CL-Bench, Terminal-Bench and OSWorld keep their pre-v6.75.0 behaviour here
and migrate in their own phases. No `ouroboros/` runtime behaviour changes.
Reviewed through eleven rounds of the production commit gate (advisory, triad
`claude-fable-5` + `gpt-5.6-sol` + `gemini-3.5-flash` at high effort, scope
`claude-fable-5`): 24 findings, 22 fixed and 2 refuted with the evidence recorded
in the changelog. Per-phase tags are deliberately not created — a single release
tag lands with the campaign synthesis, an owner-approved deviation from the
tag-per-commit norm.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Mirrors the CLB no-swarm change (36ab50c): each bench adapter now withholds
schedule_subagent so a single-agent submittable run has zero delegation.
TB _disabled_tools(), PB schemas.py task-body, OSWorld _effective_disabled_tools().
Scoring/tasks untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause (PB cmatsuoka__figlet smoke): the agent committed a compiling state, then broke the tree with one last uncommitted edit as the acceptance improvement loop hit its pass cap; the harness ships the LIVE tree, so the verified commit protected nothing — all salvage machinery guards the answer TEXT only. Prompt-only fix (P5) over existing channels (P7): the acceptance rails line marks the last admitted improvement pass (passes_done+1 >= cap, cap>0) as FINAL and, for workspace deliveries (canonical is_workspace_mode() with an attribute fallback), EVERY workspace improvement capsule carries the tree directive — a deadline or cost rail can end the loop between capsules (triad r1) — keeping the tree at a VERIFIED state (rebuild, verify, and commit if the task calls for a commit; revert unverified edits); the 10% deadline flush AND the cost wrap-up note gain one shared commit-NEUTRAL tree sentence (acting self_worktree subagents cannot commit; a moved HEAD fails patch capture closed), byte-identical for non-workspace tasks; the ProgramBench instruction now states the exporter's true submission contract (SOURCE tarball from the CURRENT tree state; .git, root binaries, .ouroboros/ and named build/cache noise excluded at any depth; run ./compile.sh one final time), replacing the false fresh-checkout framing. Tests pin the pacing/rails seam, the workspace gating on both axes, byte-identical non-workspace texts, the real tarball contract, and the instruction truth phrases. P9 carriers bumped to 6.74.4; changelog row added. Disclosed residual (this is a MITIGATION of the incident class, not a closure): a forced tool-less exit — deadline grace or budget stop crossed inside one long round with no pacing note or capsule in the terminal stretch — can still ship an unverified last edit; the structural verification-freshness seam is a filed follow-up pending an owner decision (triad r1-r3, codex full-access review concur).
A: verdict-visible improvement capsule (verdict+tier+real blocker via one panel_reason reducer, open obligation ids, money/time/rounds/passes rails line, three real moves); reviewer-authored obligation identity (disposition_kind new|re_raise validated against the host catalog, fail-closed to new, per-pass dedup, reuse-immutable) with surviving rebuttals (previous_disposition/previous_reason/reopened_count); typed dialogue_status (continue_actionable|unreachable_here|stable_disagreement) reduced over ALL contract-valid actors with the panel quorum — termination is reviewer-authored or a real rail, never a host counter.
B: two cache-marked review-prompt segments (governance + task-stable contract), slot label off byte 0, breakpoint cap asserted on the final payload.
C: GAIA sandbox attachment staging (prompt-declared /shared_files channel, exact shared-root lookup, per-file provenance, typed per-sample infra error); CLB acceptance-claims operator patch (receipt-bound claims, knowledge nudge in the steer slot, bounded cost-finality wait); SWE-Pro cumulative shard budget (auto_run derives per_task x scheduled, run_pro seeds from cumulative spend); CLI/PB bounded task_cost_finalized waits for completed/degraded only.
D: light-mode shell guard resolves cwd before judging repo targets (resource-root label false-block fixed); post-task cost publish uses try_get_bridge.
E: generative surface-duty in commit/plan review checklists; all P9 carriers bumped.
Reviewed: 2 adversarial rounds + production triad+scope (PASSED) + codex final (GO); smoke 5x5 on all five benches verified the dialogue/staging/budget mechanics live.
Generalizable runtime + harness improvements distilled from the 35-FAIL SWE-bench Pro
autopsy, the ProgramBench improvement spec, and the TB2.1 forensics — one minor release,
no benchmark overfit.
Runtime (A/C):
- Scratch is now idempotent/adoptable: an existing untracked in-cwd file can be declared
scratch (sha recorded at declaration, re-declaration idempotent); tracked files and
directories stay blocked. The undeclared-output guard is stat-verified (real filesystem
writes, not string-shape) and, for run_script, audits on EVERY exit path — a write-then-
fail body no longer smuggles an undeclared user_files deliverable.
- Cost axis: budget_profile.cost_hard_stop_pct (0..100; None→50% historical; 0 = NO in-task
cost stop, deadline/rounds/global-gate only; malformed → 50%, never a silent 0). Latched
cost milestones (50/25/10% + wrap-up) replace the old [INFO] nudge. CLI --task-metadata-json
merges into body.metadata with host-owned keys (delegation_role/source) unforgeable.
Bench harness (D/PB/TB):
- SWE-Pro e1v2: /app is the default active workspace (via user_files root), empty memory
default, budget metadata (until_deadline, cost_hard_stop_pct=0) threaded through; bench1
fixes (shard-safe auto_run, k=1, timeline) ported into the repo; the image-load lock and
utility-image use are cross-platform (platform_layer, preflight-pulled, fail-safe restore).
- Workspace acting-integration allowlist (+integrate/compare_subagent_patches) with a
fail-closed self_worktree-under-external-workspace integration guard.
- Protected-artifact policy round-2: structural (operation-identity) false-positive
exceptions for differential-testing harnesses — vcs_diff/stat classifier, spawn-argv=argv[0]
execute, interpreter script/module operand screening, writer-target segmentation — while
read/copy/hash/introspection of the black-box reference stay blocked (incl. alias-separated
reads); pinned by a new permanent policy test.
- ProgramBench instruction v2, until_deadline profile, acceptance claims. TB ffmpeg resolver
chain (platform_layer/imageio-ffmpeg/PATH) + agent-prefix install + cv2 fallback hint.
- 4 verify-guidance clauses in prompts/SYSTEM.md (probe the consumer interface, exercise every
provided input, robust-under-ambiguity, never read hidden graders).
Docs/versions: 6.56.0 across VERSION/pyproject/web/api_types/README/ARCHITECTURE; SWE-Pro +
TB METHODOLOGY (workspace/budget/memory/retry-gate disclosure), PB gate-20 operator method,
pricing sonnet-5, DEVELOPMENT scratch contract.
Reviewed via operator triad (openai/gpt-5.5 + google/gemini-3.5-flash + anthropic/claude-fable-5)
+ scope (anthropic/claude-fable-5) to convergence: scope clean; residual triad items are a
verified reviewer false-positive and the plan-accepted single-parameter budget-threading
tradeoff on a grandfathered core-loop function (advisory-bypass, documented).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Commit 3/3 of the bench post-mortem sprint (plan smooth-skipping-charm;
commits 1-2 = bf8bde7 v6.54.3, ef786bb v6.54.4). Minor bump per plan.
Shared bench-template scaffold defaults, disclosed in the benchmarks index
and per-bench METHODOLOGY files: OUROBOROS_MAX_WORKERS=4 (same-model
decomposition slots within one task, never best-of-N; TB 2->4, GAIA
--max-workers default 1->4 with explicit 1 as the strict-baseline ablation
and the quality-profile silent 1->5 bump removed, SWE-pro 5->4),
OUROBOROS_SAFETY_MODE=light (disposable jails; deterministic guards stay),
RUNTIME_MODE=pro for container benches (GAIA deliberately stays light: its
solver runs unisolated against a live repo), and claude_code_edit disabled
in every bench solve task (single-model harness measurement). TB raises
_DEADLINE_SAFETY_SEC 30->105 from measured finalization overhead, mirrors
the registry _WEB_TOOLS denylist (youtube_transcript drift fixed, pinned by
a sync test), and the README pins why --all-model keeps review single-model.
ProgramBench e2e runner ported from the colleague's tree, adapted to our
APIs: metadata.budget_profile onto the v6.54.4 contract (footer keys
dropped, reserve pct unit-fixed 0.15->15), solve-model preflight through
migrate_model_value (fail-fast on direct-route legacy ids), atomic
per-instance checkpoints with honored reattach (client poll timeout leaves
the executor alive; the next run skips seed/start and reattaches), explicit
payload-status terminal detection, denominator-preserving ledgers
(skipped_existing rows; skipped = successful for exit code), source-only
submissions (both root binaries excluded by name), idempotent workspace
normalization shared by the prepare-only flow.
New continual_learning/ launcher wraps the EXTERNAL CL-Bench runner
(nothing vendored; fails loudly with obtain instructions), strict-sequential
guard (parallel opt-in is exclusively-stateless-bridge only), standard-path
pointer rows keep the ledger denominator, fidelity gaps of the pinned
external adapter recorded and warned. OSWorld aligned to the official 2.0
protocol (pinned OSWorld-V2@c261cb57, 500-step default, show_result.py
layout, official traj rows), bridge-level final_answer capture with
VM-state-only prompting, preflight verifies the target server's effective
scaffold settings via /api/settings (env cannot configure a --url server)
and refuses the live desktop URL without an explicit flag.
LifelongAgentBench documented as blocked (its 100% run was a no-gold-oracle
artifact). Carriers synced to 6.55.0. Review: 6 triad+scope rounds to a
fully clean verdict (16 confirmed criticals fixed, each with a regression
test); gemini/opus clean in all rounds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New SWE-bench-Pro E1v2 harness: Docker-isolated per-task runs with optional
post-task self-evolution, an autonomous multi-task runner with last-good
snapshot/restore, and offline grading. Secrets reach the container only via a
name-only env opt-in (never argv); a refused opt-in / infra skip is surfaced
through the timeline so the runner stops or skips instead of counting a
non-executed task as legitimate; --cadence off disables evolution via the
documented contract; predictions carry the leaderboard model_name_or_path.
English-only operator strings. Removes the superseded evolve_pro path.
Add denominator-preserving benchmark manifests and result ledgers across the devtools adapters, tighten Terminal-Bench/Harbor and OSWorld fail-closed preflights, and harden workspace patch artifacts for unborn, invalid, and acting-subagent git baselines.\n\nVerification: focused benchmark/headless/acting suites, isolated full pytest, adversarial review rounds, and real Ouroboros triad/scope review artifacts under bench_runs/devtools_sync_20260607_010949/reviews.
