The first cut passed the custom provider as five -c flags on every codex exec. They
share argv with the prompt, and the benchmark's longest questions overflowed the OS
limit, killing a whole task with an unrecoverable OSError. Patch and README now record
the config.toml form that the ablation actually runs.
Records the two adapter changes the luna ablation runs on, so a fresh checkout can
reproduce it: the bridge's format-repair branch no longer raises NameError on the path
it exists for, and the Codex adapter can be pointed at a custom provider so a
harness-vs-harness comparison does not silently become a provider comparison.
Patch file only — no runtime code changes. The adapter itself lives in the benchmark
checkout, per the convention this directory documents.
Regenerated from the final 2-commit ouroboros-submission branch
(3ea30ef system + a691cf3 artifacts on upstream 5f8c50eb): includes
engine-task cost harvest contract (cost_final/cost_usd_with_children,
one-shot hot path, finalize on last observe), CLBENCH_SHIM_BIND,
docker --user mapping, run registration in DEFAULT_RUN_NAMES and
leaderboard SYSTEM_DEFS, and the rewritten run-specific METHODOLOGY.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lessons from the 2026-07-26 CLB campaign, written down so they cannot repeat:
- continual_learning/METHODOLOGY.md §10-§12: the cohort metric changed
upstream (clipped nats ratio -> signed bits); the fix shipped as re-scored
reference artifacts (two reward copies per artifact) and as analyzer commit
5f8c50eb (colleague's PR pgasawa#9, merged 2026-07-19) — scoring with the
pinned analyzer mixes scales and fabricates a phantom top-1. The operator
bridge drops --run-index on 5/6 domains (ctor-only injection, no
prepare_run call): bridge multi-seed = fixed-order replicates; empirical
prompt-hash check is now a required pre-flight. Submission requires a full
default run-all (5 permuted rollouts + stateless baseline), a public
implementation link, and >=4 seeds for a strict beats-top-1 claim.
- continual_learning/RUNBOOK.md: operational lessons — score only at/after
5f8c50eb; OpenRouter spendable = min(key limit remaining, account credits);
live key rotation via runner_state settings.json; secrets hygiene for
runner_state snapshots; interrupted stateful rollouts are write-offs.
- benchmarks/README.md: mandatory four-point Upstream-Drift &
Protocol-Fidelity Pre-Flight for EVERY benchmark before expensive runs
(upstream drift incl. artifact-only re-scores; empirical protocol fidelity
by artifact diffing; submission requirements read before the run;
reconciling pinned scorers against the public leaderboard).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CRITICAL (advisory, v6.81.0). The disclosure field added this release to stop false
capability claims was making one. RUNTIME_TRUNCATION_REASON_CODES was hand-copied next
to the check and listed four codes the runtime has never emitted (max_rounds_exceeded,
task_timeout, context_exhausted, rate_limited) while omitting the two it actually uses
for the round cap and the loop-local deadline (round_limit at loop.py:3128 via
_handle_round_limit, deadline_local at loop.py:3220). A round-capped or deadline-stopped
task therefore published an affirmative `truncated: false` and run_tb.py filed it under
genuine_failure_count -- "the agent got a fair shot and got it wrong" about a trial cut
off mid-attempt. The comment above the set even named
ouroboros.outcomes.BEST_EFFORT_REASON_CODES as the SSOT while contradicting it.
Fixed as the CLASS: the set is now DERIVED from BEST_EFFORT_REASON_CODES. All six of its
codes are also "an auditor must not read this as a capability result" -- forced
finalization means the attempt was cut short by a rail rather than ended by the agent --
so it is taken whole, with no subtraction. One explicit additive delta, llm_api_error
(loop_llm_call.py:630): not a best-effort code, but the same class for an auditor as
provider_unavailable, and adapters without a separate infra channel would otherwise
publish `truncated: false` for a transport death. Every kept code is grepped to an
emitting line in ouroboros/; nothing that cannot be located there survives.
Three copies of one vocabulary become one derivation plus one pinned mirror:
* harbor_installed_agent.py now INTERPOLATES the set into the container runner template
(it is generated source, so the literal cannot go stale);
* the CL-Bench operator patch genuinely cannot import it -- that module lives in an
external clone reached only through a call-time sys.path insert -- so its mirror stays,
but a test now parses the vocabulary out of the .patch file and fails on divergence.
The patch is regenerated in place with recomputed hunk counts and verified to apply
cleanly to a pristine adapter checkout (@549998d).
Drift guard (modelled on test_credential_groups_cover_every_routable_provider): every
literal reason_code in ouroboros/ must have a recorded decision in _TRUNCATION_DECISIONS,
with the emitting line and the reasoning, and the truncating subset must equal the
published set. A code added to the runtime tomorrow fails the suite instead of silently
defaulting to an affirmative `truncated: false`.
Prose the code contradicted, now true and verified: run_tb.py's trial comment and the
'cost_truncated' bucket docstring, the OSWorld METHODOLOGY disclosure paragraph, and this
test module's FIX B header. run_tb's _provider_reasons also shed rate_limited and
provider_error -- inert, never emitted, same hand-written-vocabulary defect.
MAJOR (advisory). continual_learning/METHODOLOGY.md still documented
`extra.runtime_attested: false` and attributed the patch probe to the execution clone;
a5bdf5e renamed the field to runtime_attestation_available and corrected the target to
the --runner-path ADAPTER checkout, but only in section 6. A test asserts the old field's
absence, so the document pointed at a key the manifest provably no longer carries. Swept
the tree for runtime_attested and "execution clone": no other stale occurrence.
Verification: ruff --select F clean; node --test web/tests clean; audit_all_launchers()
and version_carrier_desyncs() empty; serial CI lane green. The parallel CI lane is red on
exactly one unrelated, pre-existing cross-test-pollution failure
(test_deep_self_review.py::TestIsReviewAvailable::test_openai) reproduced identically on
a5bdf5e and passing in isolation. VERSION stays 6.81.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two provenance fixes in the same class as the release itself, plus the
round-11/12 review findings that land in the same files.
FIX A — an isolated benchmark container carries only the providers the run
declared. `build_isolated_settings` copied EVERY provider credential present
in the live settings file regardless of which providers the run's model slots
declared, so a run pinned to OpenRouter still received direct ANTHROPIC /
OPENAI / Cloud.ru / GigaChat keys. Two consequences: a routing fallback could
spend outside the declared bucket while the manifest said otherwise, and the
reachable provider set was a function of whatever happened to be in the live
file at launch — a pinned seed that pins the code but not the environment is
not reproducible. Provider credentials are now gated on the run's declared
slots, derived from the routing SSOT (`provider_models.PROVIDER_PREFIXES` /
`provider_for_model`, the same registry `llm._resolve_remote_target` routes
on), and travel in whole groups so a key never arrives without the endpoint or
auth fields it is useless without. Owner/control secrets were never copied and
still are not. Ambiguity fails OPEN (carry a spare, disclose it) — never
closed. `benchmark_run_manifest` now records `provider_credentials`: which
credentials the container actually received, by fingerprint, never by value.
FIX B — a task the cost rail truncated says so. `usage_accounting.reserve_-
attempt` refuses on a worst-case reservation bound that reached a $6.00 rail at
$0.45 of actual spend in the v6.81.0 OSWorld smoke, stopping two of three tasks
at 13 and 22 rounds; the artefacts published `status=completed`,
`reason_code=official_evaluate` and the string `budget_exhausted` appeared
nowhere. `task_result_row` gains an always-present `runtime_outcome` projected
by the new shared `runtime_terminal_disclosure`, and every writer holding a
runtime task result now publishes it: OSWorld cu_bridge, SWE-bench and
ProgramBench success rows (the failure rows already did), the Terminal-Bench
in-container summary and disclosure ledger, the GAIA solver, the harness-bench
wrapper, and the CL-Bench per-question writer. Reward, `official_eval_status`
and adapter-stage `status` are untouched: disclosure ADDED, fact not
subtracted. TB's ledger gains a `cost_truncated` bucket — `genuine` asserts a
fair shot, which a rail-truncated trial did not get.
Review findings folded in:
- `_amend_manifest` emitted `output_paths.task_outcome` unconditionally, so the
finalized attempt manifest kept pointing at an outcome whose write failed.
The previous round fixed the ledger row and left the manifest lying; both
sides now follow the same rule.
- `ADAPTER_PATCH_MARKERS` keyed two of three detections on bare env-var names,
which the unpatched adapter may mention in a comment or a `-e` passthrough
list. That false positive OVERSTATES enforcement. Markers are now
patch-unique tokens and the uniqueness requirement is recorded beside them.
(One marker legitimately covers all three env knobs: they arrive in one loop
in one hunk.)
- CLB fidelity overstated enforcement on the DEFAULT `--path standard`:
`_docker_launcher.submit()` hardcodes `disabled_tools: []` and never imports
the patched bridge module, so the evidence is now entrypoint-specific.
- `runtime_attested` renamed to `runtime_attestation_available`: it is a tree
probe, and a definition existing is not evidence that it ran.
- README version badge alt text said 6.80.0 while the URL said 6.81.0, which
`version_carrier_desyncs` flags and the advisory preflight blocks on.
Bug-pinning tests inverted, and said so in the test docstrings:
- `test_dry_run_claims_attestation_only_when_the_patch_is_in_the_execution_-
clone` asserted `runtime_attested is True` for a DRY RUN against a clone that
merely contained an `_attest_runtime` definition — it demanded the false
positive as the contract.
- the CLB fidelity fixtures wrote bare env-var names as "the patch", which is
precisely the marker weakness above.
Docs corrected where the code falsified them: OSWorld METHODOLOGY §7.4 claimed
`OUROBOROS_MAX_ROUNDS` plus the timeout were the only per-task caps (the USD
rail binds first), §6 now says scoring reads `official_eval_status` /
`details.outcome_status` rather than filtering on `status == "completed"` and
that `output_paths.task_outcome` may be absent; CLB METHODOLOGY §3 documents
when an exported runtime mode overrides the adapter's hard-set `advanced`, and
§6 distinguishes the `--runner-path` adapter checkout from the
`--ouroboros-clone` execution seed and points at the field the code actually
writes.
Known ordering debt noted in place for the v6.82 backlog, deliberately not
restructured here: `_auto_sync_release_metadata_if_needed` runs ~87 lines after
the `_release_metadata_preflight` gate it would satisfy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four findings from the interference review of the assembled release and the
CL-Bench pre-tag smoke. All four are the same defect: a record derived from
INTENT — a flag, a constant, a pre-failure status — rather than from the fact
it describes.
1. osworld cu_bridge: the ledger row no longer asserts an artefact that was
never written. Making each publication destination independent stopped one
dead record from erasing an obtained score, but it also made the row
reachable when its target was not: `output_paths.task_outcome` was emitted
unconditionally, with the pre-failure status and without the collected
`publication_errors`. The row is now built at append time from the
destinations actually reached — the pointer only when that write succeeded,
`status` degraded to `partially_published`, the run's own status kept in
`details.outcome_status`, and the errors carried along. `official_eval_status`
and `details.reward` are untouched: the evaluation really did complete, and
demoting them would re-create the score-erasing bug.
2. CL-Bench: `extra.runtime_attestation_path` claimed the docker attestation
hook whenever `--docker` was passed, though the hook arrives only with a
tracked operator patch applied out of band per run. A docker run on an
unpatched clone was unattested while its manifest said otherwise. The new
`adapter_patch_probe` reads the execution clone for each patch's marker;
an absent attestation is recorded as absent, with a warning.
3. CL-Bench: `_fidelity_report` described the PINNED adapter, so on a patched
clone it announced a gap the patches had closed — claiming safety `full`,
advisory enforcement and no `claude_code_edit` exclusion for a run really
executing `light`/`blocking` with all nine tools disabled. Knobs now land
under `enforced_via_operator_patch` or `declared_only_pinned_adapter_gap`
according to the probe. `OUROBOROS_RUNTIME_MODE` had the same shape and is
derived too. METHODOLOGY.md section 6 is corrected to match.
4. `test_every_settings_writer_routes_through_the_shared_prologue` keyed its
writers dict on `str(path)` and compared against forward-slash literals, so
on Windows every `exempt` lookup and both final assertions would break —
the fourth Windows portability defect of this campaign. Keys are now
`as_posix()`. A sweep of the campaign's other added tests for the same
shape (dicts keyed by `str(path)`, hardcoded 'a/b.py' compared to a path)
found no second instance: the remaining hits compare `path.name`, already
call `as_posix()`, or feed assertion messages only.
The previous form of the CL-Bench attestation test pinned finding 2 — it
demanded the false claim — and is replaced by a pair asserting both
directions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six reviewed phases land as one release.
Admission is the outer boundary: every migrated launcher records a manifest before
it can touch the filesystem, and finalizes a typed outcome on every path — success,
refusal, crash, and the real exit status. A structural audit enforces that boundary
across all fourteen launchers, together with confinement computed from the active
checkout and a single manifest publisher, judging by effect rather than by callee
name and failing closed on any write form it cannot resolve.
Harness exit codes are no longer trusted as run status: inspect returns zero for an
eval that raised and harbor returns zero for a job whose trials all errored, so the
launchers now read the harness's own artefact and keep "the harness failed", "it
scored nothing" and "it scored honest zeros" distinguishable.
The acceptance dialogue reconciles receipts through one typed identity that is an
equivalence by construction, so a passing check can no longer clear a red it never
addressed. Prompt caching is normalized at every send site and cached calls stop
under-reporting their input. The owner's context mode becomes explicit and
fail-closed, with one enforcement point for every writer of a disk-authored setting.
Deliberate limits are disclosed in each bench's METHODOLOGY.md rather than implied
by silence. Isolated benchmark egress and the multi-lane script generator are
deferred to a later release with restoration patches and carry-forward notes.
Every claim a benchmark run makes about itself is now either enforced before
money is spent or refused with a durable typed record.
Admission is the outer boundary, and that is a tested property, not a
convention: `admit_benchmark_run()` builds the manifest, WRITES it, and only
then lets the seed gate enforce, raising `BenchmarkAdmissionRefused` with the
refusal already on disk. A `finalize_run_manifest()` context manager records a
typed outcome, a typed refusal/error and the REAL exit status on every exit
path, including an escaping exception (an integer `SystemExit.code` is
preserved). All seven migrated launchers route through both seams; none calls
the builder directly; and an `ast` guard in the seam meta-test fails if any
denylisted operation — filesystem assertions, docker, subprocess, state dumps,
writers — precedes admission. The guard was verified to bite: against the
pre-fix tree it reports four violations, two of which no review round had
reached.
Refusals are one family. `BenchmarkAdmissionRefused`, `RuntimeAttestationRefused`
and `SeedShapeRefused` are all `RuntimeError` subclasses carrying a typed
reason, deliberately not `SystemExit` — a `SystemExit`-raising assertion had
made a refusal handler dead code, because `SystemExit` is not an `Exception`
subclass. A behavioural parity test computes what `raise SystemExit(main())`
hands the OS for each launcher and asserts it EQUALS the `exit_code` the
manifest records, so a recorded status can no longer diverge from reality.
`runtime_attestation()` records both facts about a live server — the HTTP
`runtime_version` from the frozen `/api/health` contract and the local
HEAD/VERSION of its checkout — and requires the contracted field: a bare
`version` key now refuses with the non-overridable `runtime_version_absent`, so
an unrelated HTTP server can no longer attest. `OBO_ALLOW_EVOLVED_VOLUME` waives
only `runtime_skew`, in the shell as well as in Python, and commit availability
is decided before skew so a missing commit cannot be mislabelled. Lineage is a
descent check (`merge-base --is-ancestor`), never equality, so an evolution run
legitimately moving HEAD forward is not corruption.
`CAMPAIGN_FATAL_PROVENANCE_REASONS` is a single shared authority consumed by
both SWE-Pro drivers, which now stop the schedule immediately — before any
volume archival — instead of one driver burning every task.
Grading gains a third state: `pass|fail|ungraded` with typed reasons beside the
UNCHANGED headline formula, plus `grade_summary.json`, and the probe
orchestrator reads that artifact instead of scraping log tokens and refuses to
publish a grade that a failed or stale grader produced. ProgramBench's ledger is
append-only at both the run root and the instance directory, skip rows included.
`common/manifests.py::write_json` is atomic and byte-identical, importing
`ouroboros.utils` lazily so the module stays stdlib-only for the container-side
harbor agent, and `openrouter_key_remaining()` reads the authoritative
`limit_remaining`.
Egress isolation was built in this phase and is NOT part of this release: it is
off by default per owner decision, it produced a finding in five separate review
rounds, and round 10's was a direct recurrence of round 5's, so it was extracted
and deferred with its diagnosis preserved. Solve containers run the same open
network as before v6.75.0. The measured evidence stays in METHODOLOGY: the
official SWE-bench Pro harness does not regulate the solve container's network
at all (`--block_network` applies to the EVAL container), and 31 of 287 attempted
instances are musl/Alpine, 84% of which previously produced a non-empty patch.
GAIA, CL-Bench, Terminal-Bench and OSWorld keep their pre-v6.75.0 behaviour here
and migrate in their own phases. No `ouroboros/` runtime behaviour changes.
Reviewed through eleven rounds of the production commit gate (advisory, triad
`claude-fable-5` + `gpt-5.6-sol` + `gemini-3.5-flash` at high effort, scope
`claude-fable-5`): 24 findings, 22 fixed and 2 refuted with the evidence recorded
in the changelog. Per-phase tags are deliberately not created — a single release
tag lands with the campaign synthesis, an owner-approved deviation from the
tag-per-commit norm.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
render_run_settings unconditionally rewrote OUROBOROS_REVIEW_MODELS to the
triple-slot parity roster, defeating a campaign's declared single-reviewer
config even after the env-forward fix. Template declarations now win; the
triple roster remains the default when the template stays silent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_overrides() hardcoded CC-parity defaults (runtime=advanced, triple reviewer
list, uniform effort) and ignored the declared campaign config — 'declared vs
applied' drift on every docker-path run. run_clb.py now forwards runtime mode,
reviewer list, split review efforts, context mode and workers from the settings
template; the new operator patch makes _overrides() honor those env exports.
Also fixes CLBENCH_SOLVE_DISABLED_TOOLS being join()ed char-by-char when the
template declares it as a string.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run_clb.py._sanitized_child_env now forwards OUROBOROS_TASK_REVIEW_MODE and
OUROBOROS_MAX_SUBAGENT_DEPTH (previously silently default auto / depth 2), and
the CLB bridge adapter honors CLBENCH_SOLVE_DISABLED_TOOLS (operator patch
clb_disabled_tools_env.v6745.patch) — together they make a submittable
review-required / no-swarm CLB run configurable from the launcher.
Scoring/tasks/grader untouched; only our launcher + our systems/ouroboros plugin.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(benchmarks/continual_learning): v6.71.1 validated baseline, ops runbook, refreshed adapter pins
- Add RUNBOOK.md: field-tested at-scale recipe from the 2026-07-20 full
6-domain campaign (validated config incl. max_workers=3 with the Docker-VM
memory-sizing formula, mandatory clone-sweeper/container-reaper daemons,
smoke -> one-seed -> 5-seed submission flow per the official submitting
docs, known loss classes and their closures).
- METHODOLOGY.md: add section 9 with the v6.71.1 full-suite 1-seed results
(6-domain normalized average +0.259 vs published top-1 +0.196 and
Claude Code +0.185) and the review-mode ablation verdict (pin 1 pass);
update the honest-limits section accordingly.
- README.md: point at RUNBOOK.md, refresh the adapter pin (3ec3761: network
outage hold, format-repair round, review-passes override) and the handoff
bundle name, replace the stale internal worker-pool guidance (4 -> 3, with
the OOM rationale).
* docs(benchmarks/continual_learning): reflect the clbench_remote -> remote_work skill rename
* docs(benchmarks/continual_learning): review fixes — canonical domain names, max_workers and adapter-pin qualifiers, #9 status
Triad+scope review fixes on the maintainer rerun (base v6.74.3):
- METHODOLOGY §9 results table: use the bench's canonical task names —
blind_spectrum_monitoring (bsm) and sales_prediction — instead of the
non-canonical black_scholes (bsm) / sales_analytics, so rows map onto
the §1 domain list.
- METHODOLOGY §4: OUROBOROS_MAX_WORKERS disclosure now separates the
2026-07-01 reference-run value (4) from the at-scale value validated by
the v6.71.1 campaign (3, Docker-VM OOM rationale in RUNBOOK.md),
removing the contradiction with README/RUNBOOK.
- METHODOLOGY §3: qualify the 56764d6 adapter pin as the §8 reference-run
pin next to the v6.71.1 campaign pin 3ec3761.
- RUNBOOK: pgasawa/continual-learning-bench#9 merged 2026-07-19 — state
the mixed-metric condition in the past and point at scoring with a
leaderboard checkout that includes the fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Anton Razzhigaev <razzant@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A: verdict-visible improvement capsule (verdict+tier+real blocker via one panel_reason reducer, open obligation ids, money/time/rounds/passes rails line, three real moves); reviewer-authored obligation identity (disposition_kind new|re_raise validated against the host catalog, fail-closed to new, per-pass dedup, reuse-immutable) with surviving rebuttals (previous_disposition/previous_reason/reopened_count); typed dialogue_status (continue_actionable|unreachable_here|stable_disagreement) reduced over ALL contract-valid actors with the panel quorum — termination is reviewer-authored or a real rail, never a host counter.
B: two cache-marked review-prompt segments (governance + task-stable contract), slot label off byte 0, breakpoint cap asserted on the final payload.
C: GAIA sandbox attachment staging (prompt-declared /shared_files channel, exact shared-root lookup, per-file provenance, typed per-sample infra error); CLB acceptance-claims operator patch (receipt-bound claims, knowledge nudge in the steer slot, bounded cost-finality wait); SWE-Pro cumulative shard budget (auto_run derives per_task x scheduled, run_pro seeds from cumulative spend); CLI/PB bounded task_cost_finalized waits for completed/degraded only.
D: light-mode shell guard resolves cwd before judging repo targets (resource-root label false-block fixed); post-task cost publish uses try_get_bridge.
E: generative surface-duty in commit/plan review checklists; all P9 carriers bumped.
Reviewed: 2 adversarial rounds + production triad+scope (PASSED) + codex final (GO); smoke 5x5 on all five benches verified the dialogue/staging/budget mechanics live.
Found while extending the campaign to codebase_adaptation: the pinned bridge
passes run_index to every task ctor but not every task accepts it; the runner
venv needs mini-swe-agent for the codebase domain and an explicit pip when
built by uv. Operator-run, docs-only, owner's devtools-without-review
allowance.
Three incompatibilities found while launching the v6.56.0 bench campaign:
the safety-mode owner-guard refuses the adapter's env-side full->light
lowering at isolated-server boot (uvicorn rc=3); host.docker.internal via
host-gateway cannot reach a host-loopback step-shim on Linux (agent actions
silently lost, queries=0/reward=None); a rootful daemon writes root-owned
files into the bind-mounted data root (host bridge PermissionError). Ships
the unified diffs (applied to the external checkout, not vendored here) and
documents the clbench_remote seeding constraint under the v6.56.0 native
allowlist. Operator-run: bench harness docs only, no runtime changes;
committed under the owner's explicit devtools-without-review allowance.
Commit 3/3 of the bench post-mortem sprint (plan smooth-skipping-charm;
commits 1-2 = bf8bde7 v6.54.3, ef786bb v6.54.4). Minor bump per plan.
Shared bench-template scaffold defaults, disclosed in the benchmarks index
and per-bench METHODOLOGY files: OUROBOROS_MAX_WORKERS=4 (same-model
decomposition slots within one task, never best-of-N; TB 2->4, GAIA
--max-workers default 1->4 with explicit 1 as the strict-baseline ablation
and the quality-profile silent 1->5 bump removed, SWE-pro 5->4),
OUROBOROS_SAFETY_MODE=light (disposable jails; deterministic guards stay),
RUNTIME_MODE=pro for container benches (GAIA deliberately stays light: its
solver runs unisolated against a live repo), and claude_code_edit disabled
in every bench solve task (single-model harness measurement). TB raises
_DEADLINE_SAFETY_SEC 30->105 from measured finalization overhead, mirrors
the registry _WEB_TOOLS denylist (youtube_transcript drift fixed, pinned by
a sync test), and the README pins why --all-model keeps review single-model.
ProgramBench e2e runner ported from the colleague's tree, adapted to our
APIs: metadata.budget_profile onto the v6.54.4 contract (footer keys
dropped, reserve pct unit-fixed 0.15->15), solve-model preflight through
migrate_model_value (fail-fast on direct-route legacy ids), atomic
per-instance checkpoints with honored reattach (client poll timeout leaves
the executor alive; the next run skips seed/start and reattaches), explicit
payload-status terminal detection, denominator-preserving ledgers
(skipped_existing rows; skipped = successful for exit code), source-only
submissions (both root binaries excluded by name), idempotent workspace
normalization shared by the prepare-only flow.
New continual_learning/ launcher wraps the EXTERNAL CL-Bench runner
(nothing vendored; fails loudly with obtain instructions), strict-sequential
guard (parallel opt-in is exclusively-stateless-bridge only), standard-path
pointer rows keep the ledger denominator, fidelity gaps of the pinned
external adapter recorded and warned. OSWorld aligned to the official 2.0
protocol (pinned OSWorld-V2@c261cb57, 500-step default, show_result.py
layout, official traj rows), bridge-level final_answer capture with
VM-state-only prompting, preflight verifies the target server's effective
scaffold settings via /api/settings (env cannot configure a --url server)
and refuses the live desktop URL without an explicit flag.
LifelongAgentBench documented as blocked (its 100% run was a no-gold-oracle
artifact). Carriers synced to 6.55.0. Review: 6 triad+scope rounds to a
fully clean verdict (16 confirmed criticals fixed, each with a regression
test); gemini/opus clean in all rounds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
