4 commits

Author	SHA1	Message	Date
Anton Razzhigaev	a5bdf5e010	v6.81.0: benchmark artefacts must not carry or claim what did not happen ... Two provenance fixes in the same class as the release itself, plus the round-11/12 review findings that land in the same files. FIX A — an isolated benchmark container carries only the providers the run declared. `build_isolated_settings` copied EVERY provider credential present in the live settings file regardless of which providers the run's model slots declared, so a run pinned to OpenRouter still received direct ANTHROPIC / OPENAI / Cloud.ru / GigaChat keys. Two consequences: a routing fallback could spend outside the declared bucket while the manifest said otherwise, and the reachable provider set was a function of whatever happened to be in the live file at launch — a pinned seed that pins the code but not the environment is not reproducible. Provider credentials are now gated on the run's declared slots, derived from the routing SSOT (`provider_models.PROVIDER_PREFIXES` / `provider_for_model`, the same registry `llm._resolve_remote_target` routes on), and travel in whole groups so a key never arrives without the endpoint or auth fields it is useless without. Owner/control secrets were never copied and still are not. Ambiguity fails OPEN (carry a spare, disclose it) — never closed. `benchmark_run_manifest` now records `provider_credentials`: which credentials the container actually received, by fingerprint, never by value. FIX B — a task the cost rail truncated says so. `usage_accounting.reserve_- attempt` refuses on a worst-case reservation bound that reached a $6.00 rail at $0.45 of actual spend in the v6.81.0 OSWorld smoke, stopping two of three tasks at 13 and 22 rounds; the artefacts published `status=completed`, `reason_code=official_evaluate` and the string `budget_exhausted` appeared nowhere. `task_result_row` gains an always-present `runtime_outcome` projected by the new shared `runtime_terminal_disclosure`, and every writer holding a runtime task result now publishes it: OSWorld cu_bridge, SWE-bench and ProgramBench success rows (the failure rows already did), the Terminal-Bench in-container summary and disclosure ledger, the GAIA solver, the harness-bench wrapper, and the CL-Bench per-question writer. Reward, `official_eval_status` and adapter-stage `status` are untouched: disclosure ADDED, fact not subtracted. TB's ledger gains a `cost_truncated` bucket — `genuine` asserts a fair shot, which a rail-truncated trial did not get. Review findings folded in: - `_amend_manifest` emitted `output_paths.task_outcome` unconditionally, so the finalized attempt manifest kept pointing at an outcome whose write failed. The previous round fixed the ledger row and left the manifest lying; both sides now follow the same rule. - `ADAPTER_PATCH_MARKERS` keyed two of three detections on bare env-var names, which the unpatched adapter may mention in a comment or a `-e` passthrough list. That false positive OVERSTATES enforcement. Markers are now patch-unique tokens and the uniqueness requirement is recorded beside them. (One marker legitimately covers all three env knobs: they arrive in one loop in one hunk.) - CLB fidelity overstated enforcement on the DEFAULT `--path standard`: `_docker_launcher.submit()` hardcodes `disabled_tools: []` and never imports the patched bridge module, so the evidence is now entrypoint-specific. - `runtime_attested` renamed to `runtime_attestation_available`: it is a tree probe, and a definition existing is not evidence that it ran. - README version badge alt text said 6.80.0 while the URL said 6.81.0, which `version_carrier_desyncs` flags and the advisory preflight blocks on. Bug-pinning tests inverted, and said so in the test docstrings: - `test_dry_run_claims_attestation_only_when_the_patch_is_in_the_execution_- clone` asserted `runtime_attested is True` for a DRY RUN against a clone that merely contained an `_attest_runtime` definition — it demanded the false positive as the contract. - the CLB fidelity fixtures wrote bare env-var names as "the patch", which is precisely the marker weakness above. Docs corrected where the code falsified them: OSWorld METHODOLOGY §7.4 claimed `OUROBOROS_MAX_ROUNDS` plus the timeout were the only per-task caps (the USD rail binds first), §6 now says scoring reads `official_eval_status` / `details.outcome_status` rather than filtering on `status == "completed"` and that `output_paths.task_outcome` may be absent; CLB METHODOLOGY §3 documents when an exported runtime mode overrides the adapter's hard-set `advanced`, and §6 distinguishes the `--runner-path` adapter checkout from the `--ouroboros-clone` execution seed and points at the field the code actually writes. Known ordering debt noted in place for the v6.82 backlog, deliberately not restructured here: `_auto_sync_release_metadata_if_needed` runs ~87 lines after the `_release_metadata_preflight` gate it would satisfy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-26 05:04:14 +00:00
Anton Razzhigaev	f05bf94362	feat(v6.81.0): benchmark provenance becomes a gate, and the artefacts stop lying ... Six reviewed phases land as one release. Admission is the outer boundary: every migrated launcher records a manifest before it can touch the filesystem, and finalizes a typed outcome on every path — success, refusal, crash, and the real exit status. A structural audit enforces that boundary across all fourteen launchers, together with confinement computed from the active checkout and a single manifest publisher, judging by effect rather than by callee name and failing closed on any write form it cannot resolve. Harness exit codes are no longer trusted as run status: inspect returns zero for an eval that raised and harbor returns zero for a job whose trials all errored, so the launchers now read the harness's own artefact and keep "the harness failed", "it scored nothing" and "it scored honest zeros" distinguishable. The acceptance dialogue reconciles receipts through one typed identity that is an equivalence by construction, so a passing check can no longer clear a red it never addressed. Prompt caching is normalized at every send site and cached calls stop under-reporting their input. The owner's context mode becomes explicit and fail-closed, with one enforcement point for every writer of a disk-authored setting. Deliberate limits are disclosed in each bench's METHODOLOGY.md rather than implied by silence. Isolated benchmark egress and the multi-lane script generator are deferred to a later release with restoration patches and carry-forward notes.	2026-07-26 03:40:44 +00:00
Anton Razzhigaev	e879ffc675	fix(v6.75.0): benchmark run provenance becomes a gate, not a report ... Every claim a benchmark run makes about itself is now either enforced before money is spent or refused with a durable typed record. Admission is the outer boundary, and that is a tested property, not a convention: `admit_benchmark_run()` builds the manifest, WRITES it, and only then lets the seed gate enforce, raising `BenchmarkAdmissionRefused` with the refusal already on disk. A `finalize_run_manifest()` context manager records a typed outcome, a typed refusal/error and the REAL exit status on every exit path, including an escaping exception (an integer `SystemExit.code` is preserved). All seven migrated launchers route through both seams; none calls the builder directly; and an `ast` guard in the seam meta-test fails if any denylisted operation — filesystem assertions, docker, subprocess, state dumps, writers — precedes admission. The guard was verified to bite: against the pre-fix tree it reports four violations, two of which no review round had reached. Refusals are one family. `BenchmarkAdmissionRefused`, `RuntimeAttestationRefused` and `SeedShapeRefused` are all `RuntimeError` subclasses carrying a typed reason, deliberately not `SystemExit` — a `SystemExit`-raising assertion had made a refusal handler dead code, because `SystemExit` is not an `Exception` subclass. A behavioural parity test computes what `raise SystemExit(main())` hands the OS for each launcher and asserts it EQUALS the `exit_code` the manifest records, so a recorded status can no longer diverge from reality. `runtime_attestation()` records both facts about a live server — the HTTP `runtime_version` from the frozen `/api/health` contract and the local HEAD/VERSION of its checkout — and requires the contracted field: a bare `version` key now refuses with the non-overridable `runtime_version_absent`, so an unrelated HTTP server can no longer attest. `OBO_ALLOW_EVOLVED_VOLUME` waives only `runtime_skew`, in the shell as well as in Python, and commit availability is decided before skew so a missing commit cannot be mislabelled. Lineage is a descent check (`merge-base --is-ancestor`), never equality, so an evolution run legitimately moving HEAD forward is not corruption. `CAMPAIGN_FATAL_PROVENANCE_REASONS` is a single shared authority consumed by both SWE-Pro drivers, which now stop the schedule immediately — before any volume archival — instead of one driver burning every task. Grading gains a third state: `pass|fail|ungraded` with typed reasons beside the UNCHANGED headline formula, plus `grade_summary.json`, and the probe orchestrator reads that artifact instead of scraping log tokens and refuses to publish a grade that a failed or stale grader produced. ProgramBench's ledger is append-only at both the run root and the instance directory, skip rows included. `common/manifests.py::write_json` is atomic and byte-identical, importing `ouroboros.utils` lazily so the module stays stdlib-only for the container-side harbor agent, and `openrouter_key_remaining()` reads the authoritative `limit_remaining`. Egress isolation was built in this phase and is NOT part of this release: it is off by default per owner decision, it produced a finding in five separate review rounds, and round 10's was a direct recurrence of round 5's, so it was extracted and deferred with its diagnosis preserved. Solve containers run the same open network as before v6.75.0. The measured evidence stays in METHODOLOGY: the official SWE-bench Pro harness does not regulate the solve container's network at all (`--block_network` applies to the EVAL container), and 31 of 287 attempted instances are musl/Alpine, 84% of which previously produced a non-empty patch. GAIA, CL-Bench, Terminal-Bench and OSWorld keep their pre-v6.75.0 behaviour here and migrate in their own phases. No `ouroboros/` runtime behaviour changes. Reviewed through eleven rounds of the production commit gate (advisory, triad `claude-fable-5` + `gpt-5.6-sol` + `gemini-3.5-flash` at high effort, scope `claude-fable-5`): 24 findings, 22 fixed and 2 refuted with the evidence recorded in the changelog. Per-phase tags are deliberately not created — a single release tag lands with the campaign synthesis, an owner-approved deviation from the tag-per-commit norm. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-25 18:24:25 +00:00
Ouroboros	0a091cc9d1	feat: v6.55.0 bench devtools alignment — scaffold defaults, ProgramBench e2e, CLB launcher, OSWorld 2.0 ... Commit 3/3 of the bench post-mortem sprint (plan smooth-skipping-charm; commits 1-2 = bf8bde7 v6.54.3, ef786bb v6.54.4). Minor bump per plan. Shared bench-template scaffold defaults, disclosed in the benchmarks index and per-bench METHODOLOGY files: OUROBOROS_MAX_WORKERS=4 (same-model decomposition slots within one task, never best-of-N; TB 2->4, GAIA --max-workers default 1->4 with explicit 1 as the strict-baseline ablation and the quality-profile silent 1->5 bump removed, SWE-pro 5->4), OUROBOROS_SAFETY_MODE=light (disposable jails; deterministic guards stay), RUNTIME_MODE=pro for container benches (GAIA deliberately stays light: its solver runs unisolated against a live repo), and claude_code_edit disabled in every bench solve task (single-model harness measurement). TB raises _DEADLINE_SAFETY_SEC 30->105 from measured finalization overhead, mirrors the registry _WEB_TOOLS denylist (youtube_transcript drift fixed, pinned by a sync test), and the README pins why --all-model keeps review single-model. ProgramBench e2e runner ported from the colleague's tree, adapted to our APIs: metadata.budget_profile onto the v6.54.4 contract (footer keys dropped, reserve pct unit-fixed 0.15->15), solve-model preflight through migrate_model_value (fail-fast on direct-route legacy ids), atomic per-instance checkpoints with honored reattach (client poll timeout leaves the executor alive; the next run skips seed/start and reattaches), explicit payload-status terminal detection, denominator-preserving ledgers (skipped_existing rows; skipped = successful for exit code), source-only submissions (both root binaries excluded by name), idempotent workspace normalization shared by the prepare-only flow. New continual_learning/ launcher wraps the EXTERNAL CL-Bench runner (nothing vendored; fails loudly with obtain instructions), strict-sequential guard (parallel opt-in is exclusively-stateless-bridge only), standard-path pointer rows keep the ledger denominator, fidelity gaps of the pinned external adapter recorded and warned. OSWorld aligned to the official 2.0 protocol (pinned OSWorld-V2@c261cb57, 500-step default, show_result.py layout, official traj rows), bridge-level final_answer capture with VM-state-only prompting, preflight verifies the target server's effective scaffold settings via /api/settings (env cannot configure a --url server) and refuses the live desktop URL without an explicit flag. LifelongAgentBench documented as blocked (its 100% run was a no-gold-oracle artifact). Carriers synced to 6.55.0. Review: 6 triad+scope rounds to a fully clean verdict (16 confirmed criticals fixed, each with a regression test); gemini/opus clean in all rounds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-03 22:49:01 +03:00