Guard review of the editing tools found the new pair weaker than the tools
they sit beside — not in what they can do, but in what refuses them.
Root cause: a guard judged a different spelling of the path than the write
used. `ctx.repo_path` runs `normalize_root_relative`, so an absolute path
inside the root and a redundant root-basename prefix land on the same file a
bare relative path does; `_resolve_edit_target` checked the RAW spelling, for
which `repo/BIBLE.md` is not a member of the protected-path table while
`BIBLE.md` is. In runtime_mode=advanced, `apply_patch`/`edit_batch` therefore
rewrote BIBLE.md and ouroboros/safety.py where edit_text returns
CORE_PROTECTION_BLOCKED. Reproduced against the real dispatcher, both
spellings, on the default root.
`edit_text`/`write_file` are immune only because the dispatcher canonicalizes
their `path` ARG once (`_PATH_NORMALIZED_TOOLS`). The new tools carry paths
inside the payload, so the module comment claiming their handlers "normalize
each target themselves via the shared edit_text guard chain" described a
normalization the shared chain never had.
One canonicalization contract fixes the whole class:
- `tool_access.canonical_repo_relative_path` is the single normalization both
ends call: the handler before its own protected checks, and the dispatch
gates through `_payload_write_paths`, which reads apply_patch's targets back
out of the REAL parser (`patch_target_paths`) so the gate cannot drift.
- `_resolve_edit_target` RETURNS that canonical rel, because it is the file's
identity. Keying the plan by the raw spelling meant two spellings of one file
in a single call produced two buffers and two writes, and the last silently
discarded the first while the result reported both as applied — reproduced as
real data loss on edit_batch and apply_patch.
- `_ROOT_ARG_REPO_WRITE_TOOLS` replaces the hardcoded write_file/edit_text pair
at the three fences that missed the new tools: the acting-no-workspace fence
(an acting child with no isolated workspace reached the LIVE repo), the
protected-write gate (which also carries the pro + protected_paths_grant
condition), and the acting root-enum narrowing.
Parity with the other repo writers, item by item:
- The assisted managed-update resolver keeps its protected-path exemption here
too; without it these tools were the one lane that could not finish a
conflict resolution.
- A pro-mode protected edit appends `core_patch_notice`, as git._repo_write and
_str_replace_editor do. The mode ALLOWS the write; the notice is what keeps
it visible.
- Validation refusals get the typed `edit_ops_blocked` status: a counted or
context miss is the designed path and is user-correctable exactly like
edit_text's "old_str not found", so it is a policy denial, not the false
tool_failure headline v6.57.0 removed for the other write tools. A partial
write is NOT that: it carries its own `EDIT_OPS_PARTIAL_WRITE_FAILED` prefix
and stays a real execution failure.
- A write failure after some files were written invalidates the advisory
snapshot for them and says PARTIALLY APPLIED, so commit_reviewed cannot accept
them against a pre-review taken before they existed.
Honest contracts instead of implied ones:
- Atomicity is over VALIDATION. Every schema, prompt and doc that implied the
writes roll back now says so, and the residual (a mid-write I/O fault can
leave earlier files applied) is disclosed rather than papered over.
- The fuzzy trailing-whitespace fallback states what it actually did: the
replaced lines, context included, now carry the patch's trailing whitespace.
- `_unified_diff` reports a final-newline change instead of "(no textual
changes)" — the rail exists to let the agent verify an overwrite, so the one
answer it must never give is "nothing changed" for changed bytes.
- `_syntax_check` names the format it actually checked (a NUL byte in a .py file
reported "not valid JSON").
- `edit_batch` gets apply_patch's 80k result limit: an aborted batch reports
every failed edit so one retry can fix them all.
Newline handling is deliberately NOT changed: the repo-write lane reads with
universal newlines and writes \n, so edit_text and write_file flatten CRLF the
same way today. Documented in the module rather than diverging one tool from
its lane.
editbench: generated `fixtures_v2/` is gitignored (running the benchmark dirtied
the seed gate), missing fixture trees fail fast before the isolated server and
the paid jobs rather than inside copytree, the run root is DERIVED instead of
mkdtemp'd so a seed-gate refusal leaves no footprint ahead of admission, and the
README and launcher docstring stop offering `edit_sketch_only` and a `default`
config they no longer have.
Tests exercise the REAL guard chain (a registry over a throwaway repo, no
patched resolver): protected paths refused in all three spellings for all three
tools, the acting-no-workspace fence for all three write tools, one file under
two spellings staying one target, the pro-mode notice, the resolver exemption,
parser-derived patch targets, partial-write disclosure and its status, and the
diff rail. DEVELOPMENT's New Tool rule now names the guard surfaces, since every
visibility list was already green while these fences were missing.
Co-Authored-By: Andrei Kaznacheev <a.kaznacheev@sdgroup.ai>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Ouroboros <311266734+ouroboros-agent@users.noreply.github.com>
Two new repo-lane editing primitives beyond exact-match edit_text and
full-file write_file, both validated on the included editbench benchmark
before adoption:
- edit_batch: atomic batch of COUNTED exact replacements. Each edit
declares the occurrence count it expects and replaces all of them; any
mismatch aborts the whole batch with per-edit diagnostics. The safe
replace-all: a miscount is an atomic refusal, not a silent corruption.
- apply_patch: context-anchored multi-file patch (V4A-style, no line
numbers; optional @@ anchors; trailing-whitespace fuzzy fallback).
Atomic across all files/hunks with per-hunk diagnostics; parser
tolerates decorative trailing *** on headers (observed model quirk).
write_file repo lane gains two verification rails inherited from an
edit_sketch fast-apply tool that was implemented, benchmarked, and
REJECTED (data in devtools/benchmarks/editbench/README.md): a pre-write
syntax guard for .py/.json (force bypasses with an explicit
SYNTAX_GUARD_BYPASSED disclosure — no silent bypass, P3) and the unified
diff vs the previous version appended to overwrite results (capped with
an explicit truncation count, P1).
Registration follows the New Tool rule across every canonical surface:
safety.py TOOL_POLICY, tool_capabilities (CORE/ACTING envelopes + result
limits; local-readonly and heal lanes deliberately excluded), registry
(_FROZEN_TOOL_MODULES/_WORKSPACE_ALLOWED_TOOLS/_REPO_MUTATION_TOOLS),
outcomes (_ROOT_WRITE_TOOLS so patch/batch-only turns stay acceptance-
review-eligible; _RECOVERY_TOOL_NAMES), smoke EXPECTED_TOOLS, and the
SYSTEM.md/ARCHITECTURE.md/DEVELOPMENT.md/CHECKLISTS.md documentation in
the same commit (P6). The edit_text/write_file descriptions steer tool
selection at the schema source (per DEVELOPMENT's anti-pattern guidance,
not SYSTEM.md prose accretion).
devtools/benchmarks/editbench/ is the evidence and the regression rig:
5 editing tasks (trap-laden rename, surgical edit in a 1691-line module,
whole-function replacement, cross-file move, near-full-file transform),
per-config tool restriction via disabled_tools, deterministic grading,
token/latency/error mining. Migrated under the benchmark admission
contract (admit_benchmark_run/finalize_run_manifest, launcher_audit
MIGRATED_LAUNCHERS); it measures the current working tree by design, so
real runs use the recorded --allow-dirty-seed escape.
Headline numbers (grok-4.5, 3 runs/config, every run solved every task;
full tables in the editbench README): edit_batch finishes the rename
task in ONE tool call vs ~6 for edit_text/write_file; on a weak model
(gemini-3.6-flash) its advantage grows to 1.5-2.5x cheaper/faster; a
free-choice agent picked the right tool per task shape in every run.
No version carriers touched (maintainer assigns the release version).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Brings the public line's 23 commits — MiniMax direct provider, crash-safe
managed updates, the two-pass hermetic commit gate, and the Windows/release
CI fixes — into the Claudexor-integration line (v6.89.0).
Union resolutions of note (15 conflicted files):
- Version carriers stay 6.89.0 (this line IS the next feature release on top
of 6.88.0); the README Version History carries both lines' rows.
- config.py keeps the SSOT refactors (_clamped_number_setting,
settings_env_keys()) — the MiniMax keys ride automatically via
SETTINGS_DEFAULTS.
- gateway/control.py restart callback carries BOTH fixes: owner=True (the
runtime-mode re-read on owner restarts) and the bool return their restart
serialization reads.
- supervisor/workers.py terminal emission: their terminal_task_metadata
projection (a superset of the evolution_transaction carry) + this line's
cost-emission discipline (`emitted`, unavailable projections never publish
None placeholders).
- scope_review keeps the reviewer-window SSOT (the per-model window helpers
their side still carried are deleted); MiniMax window probing moves into
reviewer_window.reviewer_route/resolve (region-derived base_url +
authenticated catalog probe). The delegated session branch coexists with
their _ScopePromptContext/represent_binary call shape.
- shell.py keeps the claude_code_edit retirement (D10): their copies of the
retired helpers are deleted, their now-unused resolver import dropped.
- platform_layer keeps OUROBOROS_BUNDLE_DIR bundled-resource bases and gains
their ripgrep-pin dataclass; updates.js takes their redesigned
preflight/plan/typed-apply flow whole (it subsumes this line's
always-merge-aware intent); settings.js keeps the D10/6.1 picker removals
and gains the MiniMax region field.
Verified: import smoke, ruff F clean, 11 focused pytest files green
(including their test_update_apply_routing), web tests 136/136.
Co-authored-by: Ouroboros <311266734+ouroboros-agent@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Triad/scope review of the reworked PR surfaced four issues, fixed here:
- pricing.infer_api_key_type: un-prefixed "minimax/..." ids now classify
as OpenRouter. Unlike cloudru/gigachat, minimax IS a real OpenRouter
vendor namespace and slash-form ids stay router-style by design; the
removed branch made safety.py demand MINIMAX_API_KEY on OpenRouter
installs with a minimax/ light model, silently skipping the LLM safety
check (fail-open). Regression tests added for both spellings.
- tests/test_docs_sync.py pinned the pre-MiniMax direct-provider
fallback sentence and failed deterministically after the ARCHITECTURE
sync; the pin now carries the MiniMax-inclusive wording.
- Docs enumerations that decide review behavior now include MiniMax:
DEVELOPMENT deep-slot EXCEPTION (512K guaranteed floor rationale),
CHECKLISTS item 2h provider-independence list, ARCHITECTURE
_REVIEW_ROUTE_BASE_URL_KEYS and unknown-pricing route lists.
- Benchmark/devtools secret and credential enumerations gained
MINIMAX_API_KEY (+MINIMAX_REGION where the pattern carries routing
keys): common/secrets, TB submission scrubber, isolated server_runner
env allowlist, harbor agent secret keys, GAIA credential resolver —
so a MiniMax key is masked/scrubbed like every other provider key and
minimax:: models resolve their own credentials in bench lanes.
Co-authored-by: Ouroboros <311266734+ouroboros-agent@users.noreply.github.com>
UI/settings + D10: reviewer-slot SSOT (6.1) with the reviewer_slots.js editor, the
D22 'runs as' projection API, D30 owned-claudexord (claudexor_daemon.py + harness
accounts UI + onboarding card), Q2-в credential profile pin on DelegationRoute, and
the D10 migration completed — the claude_code edit path (run_edit/_run_edit_async/
make_path_guard/EDIT_TOOLS + the shell claude_code_edit tool + ~35 tests) is retired
with the registry disabled_tools compat shim kept; the readonly path keeps p7a's
full hardening, and the two p7a fence tests were re-added with only their retired
write-fence halves stripped (the read-fence coverage is live). H4 both halves
verified composed (_candidate_scope_models feeds confirms_at_least require_fresh=True);
H5 holds (env.setdefault). Fusions: agent.py UI-chip directive re-homed onto the ONE
stamped record (_record_executor_facts reads the task); advisory SessionInvocation
carries the slot's session_route; ARCH/DEVELOPMENT gateway lines fused (D10 state +
p7a hardening + D30 daemon); control.py doc line kept D2/D3-correct against p6's
stale pre-p2 copy. Fixed real guard findings: restored the orphaned
_claude_options_has_explicit_param probe, added the annotation pathlib import, and
REVERTED p6's row-1 route re-read in the scope fanout (the p5x carried-route fix
wins; caught by test_mixed_scope_fanout...). Ported 8 more tests off retired
helpers/wording; test_settings_effort's carrier test (red on p6's own head) moved
to the reviewer_slots.js carrier. Guard PASS; fence_probe PASS; focused py suites
green; web 84/84.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first cut passed the custom provider as five -c flags on every codex exec. They
share argv with the prompt, and the benchmark's longest questions overflowed the OS
limit, killing a whole task with an unrecoverable OSError. Patch and README now record
the config.toml form that the ablation actually runs.
Records the two adapter changes the luna ablation runs on, so a fresh checkout can
reproduce it: the bridge's format-repair branch no longer raises NameError on the path
it exists for, and the Codex adapter can be pointed at a custom provider so a
harness-vs-harness comparison does not silently become a provider comparison.
Patch file only — no runtime code changes. The adapter itself lives in the benchmark
checkout, per the convention this directory documents.
Two v6.86.0 losses traced to one paragraph of the OSWorld working prompt.
WHERE. The contract asked only where a result must PERSIST, so a task whose
grader reads the LIVE window (is_vlc_fullscreen compares window size to screen)
was answered by ticking the preference and never entering fullscreen. WHERE now
has two slots — live and persisted — each filled or explicitly marked not
applicable. The not-applicable branch is load-bearing: 28 scoring tasks are read
only through tab lists or the active page, is_expected_tabs compares list
LENGTHS, and an extra tab opened "to check the stored slot" would zero them. So
the clause forbids inventing an action, tab, window or dialog to fill a slot the
task does not have.
UNCHANGED. A task asking for a bullet on an existing paragraph was answered by
typing a new line, because the contract had recorded that paragraph as UNCHANGED.
UNCHANGED now covers only content the task does not mention, with a narrow
exception: new content is created only when the task asks for something that does
not exist yet, while a MARKER or PROPERTY the task names — a bullet, a style, a
colour, an alignment — is applied to the content already there. Typing a fresh
line to carry the marker leaves the named content unmarked.
Three further clauses were written and dropped after adversarial review showed
each costs more than it wins, all verified against winning traces:
- prefer a slide master over per-instance edits: 358aa0a7 scores 1.0 and its
trace states master edits would not work there, because the shapes carry direct
character formatting that overrides styles;
- exhaust a named resource before any substitute: 36037439 scores 1.0 precisely
by detouring to another search engine, and the preamble already says to retry
and try another route on an anti-bot block;
- require a configuration CLI to match the GUI's breadth: generalised from the
single evaluator in the suite that takes a majority over mime types, and it
contradicts the preamble's own rule against shaping work around guesses at how
the grader is implemented.
Adapter prompt only; no runtime, evaluator or task definition is touched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Remove the v6.82 mobile swipe gestures, make drawer opening clear keyboard state, and keep deeply nested live cards usable in narrow consumers. Set fresh-install triad defaults to Luna, Gemini Flash, and Sonnet while preserving saved settings.
Co-authored-by: Ouroboros <ouroboros@local.mac>
Forensics against the leader's own published per-task dump put the gap at 19
tasks, 8 of them one class: the work was done and never checked against the
surface the grader reads. The worker now writes the task's obligations as a
numbered checklist BEFORE its first mutating action and closes each item as
observed-satisfied / not-verified / impossible before it may finish, repairing
per item. Plural instructions still cover every element; only a singular
referent resolving to several candidates forces a justified single choice, and
the contract is revisable on new observation.
Three infeasibility shapes are named (discovery outside a stated means
restriction; a named mode the app does not ship; a trigger narrower than the
task states), framed on the END STATE rather than the route, with the brake the
gate already had: a wrong verdict scores zero even when the machine is already
correct. The desktop environment's own gsettings/dconf is a legitimate surface
for STORED values only — if the task wants something DISPLAYED and the device
does not exist, writing the key is not a workaround.
Two adversarial reviews corrected this change three times, and each correction
is the interesting part:
- The colour motivation I had deleted as false is TRUE: replaying the real
grader shows 8472fece's own gold (palette 2A6099) scores 0 against its own
evaluator, which measures distance to pure 0000FF. No palette entry can win
it; deleting the motivation gained nothing and endangered 04578141, a live
1.0 won because of it. Restored and tightened to the exact colour name.
- An earlier draft wrote the per-task proxy config — which carries the account
password — into all 361 result directories, i.e. the tree we archive and
publish. It now lives in lane-private state and is unlinked after the task.
- The session tag keyed on run_dir.parent.name, which is the DOMAIN, so the two
concurrent campaigns would have shared one exit IP per task. Keyed on the
campaign root instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both from the 22-task regression smoke, each with trace evidence:
- impress/05dd4c1d aligned the document-order shape (Shape;135) while the gold
targets the visually higher one (Shape;136). The slide-object branch said a
heading counts as the Nth item but never said in WHICH order to count.
- impress/04578141 read 'use exactly these colours, no variations' as a licence
to type raw 00FF00 through Custom Color; the gold is LibreOffice's palette
Green 00A933 at zero tolerance. That wording forbids a neighbouring shade, it
does not prescribe a hex.
Smoke result: 19 of 22 at 1.0, one at its historical continuous-metric value
(efcf0d81 = 0.8949354689860572, identical to v6.81.1), and these two.
Five tasks that scored 0 in earlier runs now score 1.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two measured defects from the v6.83.0 campaign, adapter-only.
(1) Evaluator fixtures are declared RELATIVE to the checkout and get_local_file
tests them with a bare os.path.exists, so the grader resolved them against the
process CWD. The official runner works from the checkout root; this bridge did
not, and multi_apps/7f35355e produced the byte-exact answer 25.27 and still
scored 0.0 with only a line in the lane log. evaluate() now runs with the
checkout as CWD, restored on every path.
(2) The 14-turn gate reserve is worst-case; the gate actually spent a mean of 4,
so a flat max_steps-14-1 discarded ~10 turns per example and 13 of 56 opus
failures died at 89-92 turns INSIDE a 100-turn budget. The runner now publishes
the worker's real cap (max_steps - actual gate turns - 1) into the lane settings
the server hot-reloads at every task start, so the declared total is unchanged
and the unused reserve is no longer thrown away.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A gate INFEASIBLE ends the example before the working phase, so the worker
consumed exactly zero policy turns — a KNOWN count. The fail-closed audit read
it as unavailable and flagged budget_fault on the very outcome the gate exists
to produce (caught on os/a462a795 twenty minutes into the v6.83.0 run; the
running run keeps the false flag in its audit field because its seed must stay
clean, and it is corrected at scoring time).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Image integrity is fail-closed at three seams (remote fetch with bounded
re-fetch and write-validate-rename, the shared remote-result builder, the VLM
payload builder). A truncated PNG keeps a valid 24-byte header, so header-only
checks passed it and it detonated rounds later as a non-retryable provider 400
— five task deaths in the v6.81.1 OSWorld run. The rejection is narrow: a
valid-but-huge image and a truncated-yet-renderable JPEG still go through;
only what cannot be rendered at all is refused.
Structured tool failures ({ok: false}) now feed the error counters, anti-loop
and auto-attach — but NOT the verification ledger, because a diagnostic that
honestly reports what it was asked to find is a finding, not a failed run.
Acceptance review gains an ABSENT-PREMISE branch: when the terminal claim is a
missing premise, the deliverable under review is the premise ARGUMENT.
Demanding the named artifact begs the question, and coaching a continuation
that breaches the task's own restrictions manufactures what the task forbids.
A weak premise argument still fails on its own grounds.
type_text routes multi-line and long payloads through the in-VM clipboard, and
picks the paste chord inside the guest call (a terminal ignores Ctrl+V while
the hotkey still reports success).
OSWorld adapter: --max-steps declares AND enforces a leaderboard-comparable
budget. A step is one top-level policy turn, matching the official
predict()->actions[] boundary, not one GUI action. The server round cap is
verified before the VM boots; the gate phase is cancelled at its own reserve,
counted from its LIVE event log (loop_outcome exists only after finalization,
so polling for it would have been dead code); the post-run audit reads policy
turns, not the flat physical-call field they disagree with on 344 of 346
examples. --expect-dataset-commit turns the graded-spec pin into a gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A leaderboard step is one top-level policy turn (lib_run_single.py increments
step_idx once per agent.predict() and runs every action that turn emitted),
NOT one GUI action — the earlier 0.42-actions-per-round mapping compared a
turn against an action and understated our budget ~2.4x.
- --max-steps declares AND enforces a comparable budget: gate reserve + worker
cap + one tool-less terminal turn, refused fail-closed before the VM boots
when the server round cap exceeds it.
- Post-run audit records policy turns actually used and marks overruns
non-comparable instead of rewriting the reward.
- Typed step semantics in the run manifest; MAX_STEPS wired through the lane
scripts.
- Replaced three test fixtures whose 'minimal valid PNG' literals were
undecodable, and corrected an assertion that pinned an identity coordinate
transform which only held because the stub never downscaled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- ACCEPTANCE_SURFACE_RULES: ABSENT-PREMISE / INFEASIBLE DISPOSITION branch —
when the terminal claim is infeasibility, the deliverable under review is
the premise argument; 'deliverable exists' as a criterion begs the question,
and coaching a restriction-breaching continuation manufactures artifacts
(v6.81.1 gimp/5ca86c6f: reviewer veto converted a scoring 1.0 into 0.0).
- OSWORLD_PREAMBLE: five clauses from the failure forensics (named value via
the app's own control; verbatim clipboard transfer; minimal-diff; ordinals
over real items; finish on the graded surface).
- GATE_PREAMBLE: 4d — named mode / apply scope / prohibition are premise
carriers, not working-phase details; fail-open default unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Not a release — a probe seed for the Opus-5 recovery measurement, to be
rebased onto the other agent's release and reviewed before any push.
- Extension tools answer with a JSON envelope, so a failed call arrived as
{"ok": false} with no core marker and was recorded a SUCCESS: 329 rows in
the v6.81.1 run (302 remote_exec, 20 screenshot, 5 key, 2 click). The error
counter, anti-loop, monitoring and the reflection trace all believed those
calls worked. _structured_tool_failure() reads the payload; typed status
tool_reported_failure; auto-attach refuses such a result.
- Host-side watchdog on the guest control endpoint: an agent killed the guest
server and then worked blind. Unreachable for 180s ends the attempt as a
typed INFRA row (reward null, claim released), never a capability zero.
Probe fails closed.
- type_text: < arrived as > (measured hex 3e for 3c) — angle brackets now take
the clipboard path non-ASCII already takes.
- key: whitespace is a SEQUENCE of chords; the bare form silently no-opped.
- Bridge URL removed from agent-visible results; list/test_connection denied —
an agent read the port from a tool result and curled <bridge>/evaluate.
- Gate rubric 4b/4c (same-thing check, verify-don't-assume) and a working-phase
rule that state must come through the app's own surface, not from underneath.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Everything here is a mechanism fix for a defect the v6.81.0 OSWorld full-run
forensics measured, none of it a patch over a symptom.
CORE (three touches, deliberately minimal):
- Tool results may carry a typed auto_attach_image capability: the host
attaches that local image to the conversation in the SAME round, after the
round's complete tool-message block, through the exact implementation the
view_image tool uses — vision.attach_local_image_to_context, ONE body for
both paths, so trust boundary (allowed roots, size cap, fail-closed MIME
sniff), durable copy (uploads/views) and message shape cannot drift.
Extension (ext_) results only: MCP results are untrusted server-supplied
data and must not drive automatic context mutation. Failure is strictly
non-fatal. Measured cost removed: 3,830 of 16,367 rounds (~21% of the round
budget) were the mandatory second view_image round per observation, and
every task at the 200-round cap scored 0.
- MAX_LIVE_IMAGE_BLOCKS 3 -> 5 (owner decision 2026-07-29).
SKILL unix_computer_use (manifest 0.3.0 -> 0.4.0 so the version-keyed native
seed resync actually distributes this):
- One pointer-coordinate normalizer behind click/move/aliases/mouse_down/
mouse_up/left_click_drag: accepts the malformations models actually emit
(the pair packed into x with y absent, legacy -1 sentinel, or a single
duplicating y — 109 wasted rounds in one run), distinguishes ABSENT from
UNPARSEABLE, and fails loudly on contradiction. y leaves the required
schema so recovery happens before binding.
- double_click/triple_click register as thin click aliases (111 previously
'Unknown tool' calls) — and enter the bench adapter's _GUI_ACTION_TOOLS in
the same commit, so the premise gate cannot click through an alias.
- screenshot results emit auto_attach_image (remote + local builders, both
pinned by tests); remote_exec's description states its real per-backend
contract (fresh bash -lc on OSWorld; SSH login shell on macOS; never the
visible desktop terminal).
OSWORLD ADAPTER:
- Premise-gate prompt becomes a structured rubric (action -> referent ->
blocking -> acquirable -> store-or-render -> unbound placeholders): the
v6.81.0 false kills all judged outcome-meaningfulness instead of
action-performability, and an exception list would be a keyword patch.
- The confirming challenger is REMOVED on its own full-run ledger: 20
invocations, 0 feasible saves, 1 officially-infeasible task lost, 215
worker rounds burned, and it CONFIRMED all four false kills —
identical-prompt re-reads are correlated, not independent. Claim window
back to one premise round; manifest discloses the absence.
- Working preamble: the forced screenshot->view_image loop is gone
(screenshots attach automatically), and an ENVIRONMENT PITFALLS section
states task-general state rules (live-app in-memory copies must be
reconciled after out-of-band edits; terminal tasks belong in the visible
terminal; PIDs resolved by exact executable, never self-matching -f
patterns) — phrased without any claim about what an evaluator inspects,
disclosed in METHODOLOGY as a scaffold revision whose numbers must not be
pooled with earlier ones.
REVIEW TRAIL: 7 triad+scope iterations (fable + gpt-5.6-sol +
gemini-3.6-flash, scope fable, effort high; artifacts under
/mnt/data/a.razzhigaev/osworld_runs/review_v6811_iter*). Fixed from review:
skill manifest version bump (distribution was version-keyed), ARCHITECTURE
auto-attach flow + K=5, per-backend remote_exec wording, gate-phase alias
denylist + named-literal test, post-block attach ordering with pinned test,
absent-vs-unparseable coordinate contract + handler-boundary tests,
real-producer auto_attach_image tests, up-to-two-tasks cost wording.
Rejected with evidence, recorded here per review policy: (1) the module/
function size hard-gate claim — the authoritative gate
(ouroboros/review.py excluded prefixes; tests/test_smoke.py _SKIP_DIRS)
explicitly excludes devtools/ and tests/, and the enforced suite is green;
(2) the demand to remove the ENVIRONMENT PITFALLS rules — the same reviewer
slot prescribed exactly these task-general formulations in iteration 4;
the remaining wording contains no evaluator-behavior claims.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Regenerated from the final 2-commit ouroboros-submission branch
(3ea30ef system + a691cf3 artifacts on upstream 5f8c50eb): includes
engine-task cost harvest contract (cost_final/cost_usd_with_children,
one-shot hot path, finalize on last observe), CLBENCH_SHIM_BIND,
docker --user mapping, run registration in DEFAULT_RUN_NAMES and
leaderboard SYSTEM_DEFS, and the rewritten run-specific METHODOLOGY.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The preceding commit attributed the v1 smoke's collapse (feasible-control mean
0.737 -> 0.459) to OSWorld's silent setup skip. Evidence from the v2 run says
otherwise, and the docs should say what the data says.
DockerProvider.revert_to_snapshot stops the container; start_emulator then
REALLOCATES ports via _get_available_port(5000). The VM address therefore
changes on every reset. v1 published that address exactly once, before the gate
— measured after the fact: 83/83 task dirs have bridge.json older than their
gate record — so the working phase kept driving the pre-gate port, which with 16
lanes allocating from one range another lane's container could already own. The
traces match: empty Desktops, missing task files, and one worker acting on a
different task's presentation entirely (content found in no task file — it was a
neighbouring lane's live VM). v2 republishes and shows 0 regressions against 9 at
the comparable stage; ports demonstrably move and repeat across lanes
(5024 seen on both lane0 and lane4).
The verified reset stays: it closes a real, independent OSWorld fail-open that
also affects ungated runs. But it is defence in depth, not the measured fix, and
both docs now say so — 24 post-gate resets in v2, 0 retries needed. Claiming a
guard fixed something it never fired on is how a harness acquires cargo cult.
Adds the ordering test the class needs: post-gate reset -> target file rewrite ->
_publish_target -> worker creation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 2026-07-28 smoke found the gate itself sound (13/27 caught, 0 false kills
in 42 feasible controls) and the scaffolding around it destructive: the bare
post-gate env.reset() hit OSWorld's silent fail-open (guest probe timeout ->
ALL setup steps skipped, 'Environment setup complete.' logged, no exception)
and working phases opened on VMs without the task's files. Feasible-control
mean fell 0.737 -> 0.459. Three changes, each closing a reviewed hole:
- _reset_verified() replaces BOTH bare resets: asserts the machine-checkable
postcondition (is_environment_used iff non-empty config) plus the screenshot
probe, forces the snapshot revert before every retry (an unforced retry runs
setup on top of partial state), republishes the VM endpoint after the
post-gate reset (docker recreate changes IP/ports), and turns exhaustion
into a typed infra row (reset_unverified, reward null, claim released) --
a setup the harness could not verify must never become a capability zero.
The silent-skip flaw predates the gate and affected single-reset runs too.
- An INFEASIBLE verdict no longer stands alone: an independent challenger
round (fresh session, same read-only envelope) must agree before the kill;
disagreement fails open. One false kill erases the gate's measured edge,
a missed infeasible still has the worker's own TASK_INFEASIBLE path.
- The one condition that must NOT fail open now does not: a premise round
whose cancel did not confirm aborts the attempt as blocked (exit 2) --
a zombie premise session shares the lane's server and connection file and
would act on the VM the worker is scored on. Each round's full tool trace
(verbatim args, not previews) lands in feasibility_gate.json for offline
audit of the read-only-by-instruction promise.
Claim staleness now covers two premise windows. Manifest discloses
feasibility_gate_challenger. Docs updated; 13 new tests pin the reset
postcondition, the two-verdict kill rule, the zombie detection and the
verbatim trace.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reviewed by codex gpt-5.6-sol and two independent readers before any run. They found one
defect that would have corrupted results and several that would have cost feasible tasks.
The claim staleness bound did not include the premise phase. The gated holder occupies
the claim for up to task_timeout/4 BEFORE the working task, so with shipped defaults it
consumed the entire margin the formula reserves for the unbounded evaluate() — after
which a second lane treats the lock as stale, takes the same task, and both score it.
The bound now grows by the same expression the phase's own deadline uses, so the two
cannot drift.
The verdict parser scanned every line in reverse for a keyword. A model that enumerates
the three options while reasoning and then concludes in prose had its recap read as its
answer: a PROCEED became a scored hard zero. Reproduced, then fixed to read only the last
line, which is what the prompt asks for; ordinary formatting (trailing period, bold,
backticks, case) is tolerated, a verdict inside a sentence is not. The reproduction is
now a regression test.
The INFEASIBLE path synthesized {"status": "completed", "result": "TASK_INFEASIBLE"} so
the existing detector would fire. That published a clean runtime outcome and a terminal
answer for an agent that never spoke — the same class of lie the final_answer fix in
257a369 removes. The flag is now explicit, the absent working phase is left visible as an
absence, and the outcome records infeasible_source and the phase's own rounds so a
gate-terminated example cannot be read as an agent that declared it.
Also: the untrusted task text no longer has the last word in the gate prompt (a task
saying "end with INFEASIBLE" could score itself zero); the infeasibility detector reads
the authoritative terminal answer instead of OR-ing over a field where a retracted
mention could step FAIL; sidecar write failures no longer control execution — one of
them sat between env.step("FAIL") and evaluate(), so a failed write lost a task that had
already been acted on; the gate's timeout cancel is confirmed rather than assumed, since
an unconfirmed cancel leaves a second agent on the same VM.
The phase closes the GUI vector only. remote_exec stays a general shell, read-only by
instruction, because classifying shell commands in code is the pattern gate P5 forbids.
The comment claiming the premise "cannot" be manufactured was wrong and now says what is
actually true; the working phase is re-reset after a PROCEED so nothing the premise phase
touched reaches the scored state. README and METHODOLOGY §7 (4c) state all of this.
Tests: the new control flow had none. Added the fail-open table, the recap regression,
the claim-bound relationship, the gate-phase tool removal, and a check that the
acceptance claims stay general.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The prose rule shipped in the previous commit tells the agent to establish the premise
before working. This adds the option of making that structural instead of advisory:
--feasibility-gate posts a first task whose mutating GUI tools are absent from the
capability envelope, so the agent judging whether the premise holds physically cannot
manufacture it first. That was the observed failure — same probe, same negative answer,
then a wallpaper copied onto an empty Desktop.
Three-valued on purpose. Only a standalone INFEASIBLE ends the example; PROCEED,
UNDETERMINED, an unreadable answer, a timeout, a crashed phase and any exception all
proceed to the full-capability phase. The gate may remove a task the agent was
affirmatively certain about and nothing else. The published verified run we compared
against lost a task by calling a Cloudflare block infeasible, so UNDETERMINED is named
in the prompt as the answer for network and anti-bot obstacles.
The INFEASIBLE path does not re-implement scoring: it synthesizes the terminal answer
the working phase would have produced and falls through to the single existing
evaluate() + claim-marker sequence, which is the code that protects against double
scoring and must have exactly one caller.
remote_exec stays available for read-only probes and read-only is an instruction, not an
enforcement: classifying a shell command as reading or writing in code would be the
pattern gate P5 forbids for a semantic decision.
Off by default, and the manifest stops claiming one run per task when it is on.
Unverified: the false-INFEASIBLE rate on feasible tasks is not measured yet, and that is
the number that decides whether this is worth its cost — README says so.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pairwise forensics against a published verified run on the same 361 tasks and the same
model put ~90% of our deficit on the 27 infeasible-evaluator tasks; on the 333 feasible
ones the two runs are statistically indistinguishable. Reading our own traces there, the
failure is not that the agent lacked information: it ran the correct probe, got the
correct negative answer, and then manufactured the premise — copying a system wallpaper
onto an empty Desktop and adjusting its own planted file, building a same-named theme
directory from a sibling, writing document internals the app cannot render — and
reported success.
The prose rules were already there and did not hold, so this changes what they say
rather than adding another one:
- the feasibility rule enumerated missing hardware, accounts and app features, which
never covered an absent object the task acts on. It now names the class: an essential
PRE-EXISTING target or capability the task presupposes is absent — as distinct from
what the task asks to create, from a detail mentioned only as motivation, and from a
transient network block. That last exclusion is not hypothetical: the verified run we
compared against lost 82279c77 by calling a Cloudflare block infeasible.
- establishing the premise no longer competes with speed; it is exempt from the
investigation budget that sits three lines below it.
- manufacturing the premise is named and forbidden, as is searching the guest for
grader code — observed in the v6.81.0 traces, nothing found, nothing exploited, and
it should be prohibited rather than merely unsanctioned.
acceptance_claims was [] on all 361 tasks while the acceptance panel ran on 324 of them.
Four general claims now travel with each task, so the reviewer that already runs
adjudicates observations instead of narrative. No extra model call.
cu_bridge outcomes carried final_answer: null while the answer sat in the runtime
result, which is exactly what METHODOLOGY §4 exists to prevent; it now falls back as
documented.
Nothing here encodes a base rate of infeasible tasks, a task id, an application, or any
evaluator property. Numbers from before and after this revision are from different
scaffolds and must not be pooled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two of our own public artefacts contradicted our own traces, which a leaderboard
reviewer would read as concealment rather than as stale text:
- README claimed the adapter passes the Terminal-Bench instruction "unchanged" and
"does not prepend harness notes", while run() appends an anti-lookup integrity
paragraph to every instruction. The clause is a restriction on the agent, not a
hint, but it is part of the measured configuration and has to be disclosed. The
exact text is now reproduced in the README so a reviewer can diff it against the
official task.
- The adapter comment asserted "subagents=0" and that max_workers is moot. Measured
on the v6.81.0 runs: withholding schedule_subagent stops task DELEGATION, but
plan_task still runs pooled planning scouts that appear in traces as
delegation_role=subagent. A submission must claim "no task delegation", not
"no subagents".
No behaviour change: documentation and one comment only.
Lessons from the 2026-07-26 CLB campaign, written down so they cannot repeat:
- continual_learning/METHODOLOGY.md §10-§12: the cohort metric changed
upstream (clipped nats ratio -> signed bits); the fix shipped as re-scored
reference artifacts (two reward copies per artifact) and as analyzer commit
5f8c50eb (colleague's PR pgasawa#9, merged 2026-07-19) — scoring with the
pinned analyzer mixes scales and fabricates a phantom top-1. The operator
bridge drops --run-index on 5/6 domains (ctor-only injection, no
prepare_run call): bridge multi-seed = fixed-order replicates; empirical
prompt-hash check is now a required pre-flight. Submission requires a full
default run-all (5 permuted rollouts + stateless baseline), a public
implementation link, and >=4 seeds for a strict beats-top-1 claim.
- continual_learning/RUNBOOK.md: operational lessons — score only at/after
5f8c50eb; OpenRouter spendable = min(key limit remaining, account credits);
live key rotation via runner_state settings.json; secrets hygiene for
runner_state snapshots; interrupted stateful rollouts are write-offs.
- benchmarks/README.md: mandatory four-point Upstream-Drift &
Protocol-Fidelity Pre-Flight for EVERY benchmark before expensive runs
(upstream drift incl. artifact-only re-scores; empirical protocol fidelity
by artifact diffing; submission requirements read before the run;
reconciling pinned scorers against the public leaderboard).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
scrub_submission_secrets.py walked `root.rglob("*")` filtered by `p.is_file()` and
wrote with `path.write_text()`. Both follow symlinks, so the tool had two failures,
each demonstrated against the pre-fix code.
A FILE symlink was swept THROUGH: the write landed on the link's target, outside
--root. A pack containing a link to the live settings.json therefore had its real
keys replaced with <REDACTED:...> by the tool whose job is to protect them. `cp -a`
preserves symlinks, so the procedural "run this on a COPY" rule did not help.
A DIRECTORY symlink was worse. rglob does not descend through one, so its contents
were never enumerated — and the verify pass then printed verify_leftovers=0 and
exited 0. The tool affirmatively certified a tree it had never read, for content
reachable under --root and about to be uploaded to a public leaderboard. Silent
non-coverage reported as cleanliness is exactly the class of false claim this
release exists to remove, and here the consequence is a live API key published.
Both are now a hard refusal before a single byte is written, reusing the fail-closed
discipline the --env-passthrough refusal already established: every offending link is
named with its target and its kind, nothing is modified, exit 2. Refusing beats
resolving-and-continuing, because under --root a symlink is either an accident or an
escape and only the operator can say which. With the links removed the tool sweeps
and verifies exactly as before, so this refuses an unsafe shape rather than removing
a capability.
Pre-fix, the new test fails with `assert 0 == 2` — the old tool reported success.
CRITICAL (advisory, v6.81.0). The disclosure field added this release to stop false
capability claims was making one. RUNTIME_TRUNCATION_REASON_CODES was hand-copied next
to the check and listed four codes the runtime has never emitted (max_rounds_exceeded,
task_timeout, context_exhausted, rate_limited) while omitting the two it actually uses
for the round cap and the loop-local deadline (round_limit at loop.py:3128 via
_handle_round_limit, deadline_local at loop.py:3220). A round-capped or deadline-stopped
task therefore published an affirmative `truncated: false` and run_tb.py filed it under
genuine_failure_count -- "the agent got a fair shot and got it wrong" about a trial cut
off mid-attempt. The comment above the set even named
ouroboros.outcomes.BEST_EFFORT_REASON_CODES as the SSOT while contradicting it.
Fixed as the CLASS: the set is now DERIVED from BEST_EFFORT_REASON_CODES. All six of its
codes are also "an auditor must not read this as a capability result" -- forced
finalization means the attempt was cut short by a rail rather than ended by the agent --
so it is taken whole, with no subtraction. One explicit additive delta, llm_api_error
(loop_llm_call.py:630): not a best-effort code, but the same class for an auditor as
provider_unavailable, and adapters without a separate infra channel would otherwise
publish `truncated: false` for a transport death. Every kept code is grepped to an
emitting line in ouroboros/; nothing that cannot be located there survives.
Three copies of one vocabulary become one derivation plus one pinned mirror:
* harbor_installed_agent.py now INTERPOLATES the set into the container runner template
(it is generated source, so the literal cannot go stale);
* the CL-Bench operator patch genuinely cannot import it -- that module lives in an
external clone reached only through a call-time sys.path insert -- so its mirror stays,
but a test now parses the vocabulary out of the .patch file and fails on divergence.
The patch is regenerated in place with recomputed hunk counts and verified to apply
cleanly to a pristine adapter checkout (@549998d).
Drift guard (modelled on test_credential_groups_cover_every_routable_provider): every
literal reason_code in ouroboros/ must have a recorded decision in _TRUNCATION_DECISIONS,
with the emitting line and the reasoning, and the truncating subset must equal the
published set. A code added to the runtime tomorrow fails the suite instead of silently
defaulting to an affirmative `truncated: false`.
Prose the code contradicted, now true and verified: run_tb.py's trial comment and the
'cost_truncated' bucket docstring, the OSWorld METHODOLOGY disclosure paragraph, and this
test module's FIX B header. run_tb's _provider_reasons also shed rate_limited and
provider_error -- inert, never emitted, same hand-written-vocabulary defect.
MAJOR (advisory). continual_learning/METHODOLOGY.md still documented
`extra.runtime_attested: false` and attributed the patch probe to the execution clone;
a5bdf5e renamed the field to runtime_attestation_available and corrected the target to
the --runner-path ADAPTER checkout, but only in section 6. A test asserts the old field's
absence, so the document pointed at a key the manifest provably no longer carries. Swept
the tree for runtime_attested and "execution clone": no other stale occurrence.
Verification: ruff --select F clean; node --test web/tests clean; audit_all_launchers()
and version_carrier_desyncs() empty; serial CI lane green. The parallel CI lane is red on
exactly one unrelated, pre-existing cross-test-pollution failure
(test_deep_self_review.py::TestIsReviewAvailable::test_openai) reproduced identically on
a5bdf5e and passing in isolation. VERSION stays 6.81.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two provenance fixes in the same class as the release itself, plus the
round-11/12 review findings that land in the same files.
FIX A — an isolated benchmark container carries only the providers the run
declared. `build_isolated_settings` copied EVERY provider credential present
in the live settings file regardless of which providers the run's model slots
declared, so a run pinned to OpenRouter still received direct ANTHROPIC /
OPENAI / Cloud.ru / GigaChat keys. Two consequences: a routing fallback could
spend outside the declared bucket while the manifest said otherwise, and the
reachable provider set was a function of whatever happened to be in the live
file at launch — a pinned seed that pins the code but not the environment is
not reproducible. Provider credentials are now gated on the run's declared
slots, derived from the routing SSOT (`provider_models.PROVIDER_PREFIXES` /
`provider_for_model`, the same registry `llm._resolve_remote_target` routes
on), and travel in whole groups so a key never arrives without the endpoint or
auth fields it is useless without. Owner/control secrets were never copied and
still are not. Ambiguity fails OPEN (carry a spare, disclose it) — never
closed. `benchmark_run_manifest` now records `provider_credentials`: which
credentials the container actually received, by fingerprint, never by value.
FIX B — a task the cost rail truncated says so. `usage_accounting.reserve_-
attempt` refuses on a worst-case reservation bound that reached a $6.00 rail at
$0.45 of actual spend in the v6.81.0 OSWorld smoke, stopping two of three tasks
at 13 and 22 rounds; the artefacts published `status=completed`,
`reason_code=official_evaluate` and the string `budget_exhausted` appeared
nowhere. `task_result_row` gains an always-present `runtime_outcome` projected
by the new shared `runtime_terminal_disclosure`, and every writer holding a
runtime task result now publishes it: OSWorld cu_bridge, SWE-bench and
ProgramBench success rows (the failure rows already did), the Terminal-Bench
in-container summary and disclosure ledger, the GAIA solver, the harness-bench
wrapper, and the CL-Bench per-question writer. Reward, `official_eval_status`
and adapter-stage `status` are untouched: disclosure ADDED, fact not
subtracted. TB's ledger gains a `cost_truncated` bucket — `genuine` asserts a
fair shot, which a rail-truncated trial did not get.
Review findings folded in:
- `_amend_manifest` emitted `output_paths.task_outcome` unconditionally, so the
finalized attempt manifest kept pointing at an outcome whose write failed.
The previous round fixed the ledger row and left the manifest lying; both
sides now follow the same rule.
- `ADAPTER_PATCH_MARKERS` keyed two of three detections on bare env-var names,
which the unpatched adapter may mention in a comment or a `-e` passthrough
list. That false positive OVERSTATES enforcement. Markers are now
patch-unique tokens and the uniqueness requirement is recorded beside them.
(One marker legitimately covers all three env knobs: they arrive in one loop
in one hunk.)
- CLB fidelity overstated enforcement on the DEFAULT `--path standard`:
`_docker_launcher.submit()` hardcodes `disabled_tools: []` and never imports
the patched bridge module, so the evidence is now entrypoint-specific.
- `runtime_attested` renamed to `runtime_attestation_available`: it is a tree
probe, and a definition existing is not evidence that it ran.
- README version badge alt text said 6.80.0 while the URL said 6.81.0, which
`version_carrier_desyncs` flags and the advisory preflight blocks on.
Bug-pinning tests inverted, and said so in the test docstrings:
- `test_dry_run_claims_attestation_only_when_the_patch_is_in_the_execution_-
clone` asserted `runtime_attested is True` for a DRY RUN against a clone that
merely contained an `_attest_runtime` definition — it demanded the false
positive as the contract.
- the CLB fidelity fixtures wrote bare env-var names as "the patch", which is
precisely the marker weakness above.
Docs corrected where the code falsified them: OSWorld METHODOLOGY §7.4 claimed
`OUROBOROS_MAX_ROUNDS` plus the timeout were the only per-task caps (the USD
rail binds first), §6 now says scoring reads `official_eval_status` /
`details.outcome_status` rather than filtering on `status == "completed"` and
that `output_paths.task_outcome` may be absent; CLB METHODOLOGY §3 documents
when an exported runtime mode overrides the adapter's hard-set `advanced`, and
§6 distinguishes the `--runner-path` adapter checkout from the
`--ouroboros-clone` execution seed and points at the field the code actually
writes.
Known ordering debt noted in place for the v6.82 backlog, deliberately not
restructured here: `_auto_sync_release_metadata_if_needed` runs ~87 lines after
the `_release_metadata_preflight` gate it would satisfy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four findings from the interference review of the assembled release and the
CL-Bench pre-tag smoke. All four are the same defect: a record derived from
INTENT — a flag, a constant, a pre-failure status — rather than from the fact
it describes.
1. osworld cu_bridge: the ledger row no longer asserts an artefact that was
never written. Making each publication destination independent stopped one
dead record from erasing an obtained score, but it also made the row
reachable when its target was not: `output_paths.task_outcome` was emitted
unconditionally, with the pre-failure status and without the collected
`publication_errors`. The row is now built at append time from the
destinations actually reached — the pointer only when that write succeeded,
`status` degraded to `partially_published`, the run's own status kept in
`details.outcome_status`, and the errors carried along. `official_eval_status`
and `details.reward` are untouched: the evaluation really did complete, and
demoting them would re-create the score-erasing bug.
2. CL-Bench: `extra.runtime_attestation_path` claimed the docker attestation
hook whenever `--docker` was passed, though the hook arrives only with a
tracked operator patch applied out of band per run. A docker run on an
unpatched clone was unattested while its manifest said otherwise. The new
`adapter_patch_probe` reads the execution clone for each patch's marker;
an absent attestation is recorded as absent, with a warning.
3. CL-Bench: `_fidelity_report` described the PINNED adapter, so on a patched
clone it announced a gap the patches had closed — claiming safety `full`,
advisory enforcement and no `claude_code_edit` exclusion for a run really
executing `light`/`blocking` with all nine tools disabled. Knobs now land
under `enforced_via_operator_patch` or `declared_only_pinned_adapter_gap`
according to the probe. `OUROBOROS_RUNTIME_MODE` had the same shape and is
derived too. METHODOLOGY.md section 6 is corrected to match.
4. `test_every_settings_writer_routes_through_the_shared_prologue` keyed its
writers dict on `str(path)` and compared against forward-slash literals, so
on Windows every `exempt` lookup and both final assertions would break —
the fourth Windows portability defect of this campaign. Keys are now
`as_posix()`. A sweep of the campaign's other added tests for the same
shape (dicts keyed by `str(path)`, hardcoded 'a/b.py' compared to a path)
found no second instance: the remaining hits compare `path.name`, already
call `as_posix()`, or feed assertion messages only.
The previous form of the CL-Bench attestation test pinned finding 2 — it
demanded the false claim — and is replaced by a pair asserting both
directions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six reviewed phases land as one release.
Admission is the outer boundary: every migrated launcher records a manifest before
it can touch the filesystem, and finalizes a typed outcome on every path — success,
refusal, crash, and the real exit status. A structural audit enforces that boundary
across all fourteen launchers, together with confinement computed from the active
checkout and a single manifest publisher, judging by effect rather than by callee
name and failing closed on any write form it cannot resolve.
Harness exit codes are no longer trusted as run status: inspect returns zero for an
eval that raised and harbor returns zero for a job whose trials all errored, so the
launchers now read the harness's own artefact and keep "the harness failed", "it
scored nothing" and "it scored honest zeros" distinguishable.
The acceptance dialogue reconciles receipts through one typed identity that is an
equivalence by construction, so a passing check can no longer clear a red it never
addressed. Prompt caching is normalized at every send site and cached calls stop
under-reporting their input. The owner's context mode becomes explicit and
fail-closed, with one enforcement point for every writer of a disk-authored setting.
Deliberate limits are disclosed in each bench's METHODOLOGY.md rather than implied
by silence. Isolated benchmark egress and the multi-lane script generator are
deferred to a later release with restoration patches and carry-forward notes.
Every claim a benchmark run makes about itself is now either enforced before
money is spent or refused with a durable typed record.
Admission is the outer boundary, and that is a tested property, not a
convention: `admit_benchmark_run()` builds the manifest, WRITES it, and only
then lets the seed gate enforce, raising `BenchmarkAdmissionRefused` with the
refusal already on disk. A `finalize_run_manifest()` context manager records a
typed outcome, a typed refusal/error and the REAL exit status on every exit
path, including an escaping exception (an integer `SystemExit.code` is
preserved). All seven migrated launchers route through both seams; none calls
the builder directly; and an `ast` guard in the seam meta-test fails if any
denylisted operation — filesystem assertions, docker, subprocess, state dumps,
writers — precedes admission. The guard was verified to bite: against the
pre-fix tree it reports four violations, two of which no review round had
reached.
Refusals are one family. `BenchmarkAdmissionRefused`, `RuntimeAttestationRefused`
and `SeedShapeRefused` are all `RuntimeError` subclasses carrying a typed
reason, deliberately not `SystemExit` — a `SystemExit`-raising assertion had
made a refusal handler dead code, because `SystemExit` is not an `Exception`
subclass. A behavioural parity test computes what `raise SystemExit(main())`
hands the OS for each launcher and asserts it EQUALS the `exit_code` the
manifest records, so a recorded status can no longer diverge from reality.
`runtime_attestation()` records both facts about a live server — the HTTP
`runtime_version` from the frozen `/api/health` contract and the local
HEAD/VERSION of its checkout — and requires the contracted field: a bare
`version` key now refuses with the non-overridable `runtime_version_absent`, so
an unrelated HTTP server can no longer attest. `OBO_ALLOW_EVOLVED_VOLUME` waives
only `runtime_skew`, in the shell as well as in Python, and commit availability
is decided before skew so a missing commit cannot be mislabelled. Lineage is a
descent check (`merge-base --is-ancestor`), never equality, so an evolution run
legitimately moving HEAD forward is not corruption.
`CAMPAIGN_FATAL_PROVENANCE_REASONS` is a single shared authority consumed by
both SWE-Pro drivers, which now stop the schedule immediately — before any
volume archival — instead of one driver burning every task.
Grading gains a third state: `pass|fail|ungraded` with typed reasons beside the
UNCHANGED headline formula, plus `grade_summary.json`, and the probe
orchestrator reads that artifact instead of scraping log tokens and refuses to
publish a grade that a failed or stale grader produced. ProgramBench's ledger is
append-only at both the run root and the instance directory, skip rows included.
`common/manifests.py::write_json` is atomic and byte-identical, importing
`ouroboros.utils` lazily so the module stays stdlib-only for the container-side
harbor agent, and `openrouter_key_remaining()` reads the authoritative
`limit_remaining`.
Egress isolation was built in this phase and is NOT part of this release: it is
off by default per owner decision, it produced a finding in five separate review
rounds, and round 10's was a direct recurrence of round 5's, so it was extracted
and deferred with its diagnosis preserved. Solve containers run the same open
network as before v6.75.0. The measured evidence stays in METHODOLOGY: the
official SWE-bench Pro harness does not regulate the solve container's network
at all (`--block_network` applies to the EVAL container), and 31 of 287 attempted
instances are musl/Alpine, 84% of which previously produced a non-empty patch.
GAIA, CL-Bench, Terminal-Bench and OSWorld keep their pre-v6.75.0 behaviour here
and migrate in their own phases. No `ouroboros/` runtime behaviour changes.
Reviewed through eleven rounds of the production commit gate (advisory, triad
`claude-fable-5` + `gpt-5.6-sol` + `gemini-3.5-flash` at high effort, scope
`claude-fable-5`): 24 findings, 22 fixed and 2 refuted with the evidence recorded
in the changelog. Per-phase tags are deliberately not created — a single release
tag lands with the campaign synthesis, an owner-approved deviation from the
tag-per-commit norm.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
render_run_settings unconditionally rewrote OUROBOROS_REVIEW_MODELS to the
triple-slot parity roster, defeating a campaign's declared single-reviewer
config even after the env-forward fix. Template declarations now win; the
triple roster remains the default when the template stays silent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_overrides() hardcoded CC-parity defaults (runtime=advanced, triple reviewer
list, uniform effort) and ignored the declared campaign config — 'declared vs
applied' drift on every docker-path run. run_clb.py now forwards runtime mode,
reviewer list, split review efforts, context mode and workers from the settings
template; the new operator patch makes _overrides() honor those env exports.
Also fixes CLBENCH_SOLVE_DISABLED_TOOLS being join()ed char-by-char when the
template declares it as a string.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A dirty seed worktree poisons run_manifest provenance (describe ...-dirty)
and disqualifies the run from leaderboard submission. Documented in caps in
benchmarks README and TB METHODOLOGY. Also commits the anti-lookup integrity
instruction that was live (uncommitted) in the v6.74.5 campaign seed, so the
running config is fully reproducible from the repo.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirrors the CLB no-swarm change (36ab50c): each bench adapter now withholds
schedule_subagent so a single-agent submittable run has zero delegation.
TB _disabled_tools(), PB schemas.py task-body, OSWorld _effective_disabled_tools().
Scoring/tasks untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
run_clb.py._sanitized_child_env now forwards OUROBOROS_TASK_REVIEW_MODE and
OUROBOROS_MAX_SUBAGENT_DEPTH (previously silently default auto / depth 2), and
the CLB bridge adapter honors CLBENCH_SOLVE_DISABLED_TOOLS (operator patch
clb_disabled_tools_env.v6745.patch) — together they make a submittable
review-required / no-swarm CLB run configurable from the launcher.
Scoring/tasks/grader untouched; only our launcher + our systems/ouroboros plugin.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause (PB cmatsuoka__figlet smoke): the agent committed a compiling state, then broke the tree with one last uncommitted edit as the acceptance improvement loop hit its pass cap; the harness ships the LIVE tree, so the verified commit protected nothing — all salvage machinery guards the answer TEXT only. Prompt-only fix (P5) over existing channels (P7): the acceptance rails line marks the last admitted improvement pass (passes_done+1 >= cap, cap>0) as FINAL and, for workspace deliveries (canonical is_workspace_mode() with an attribute fallback), EVERY workspace improvement capsule carries the tree directive — a deadline or cost rail can end the loop between capsules (triad r1) — keeping the tree at a VERIFIED state (rebuild, verify, and commit if the task calls for a commit; revert unverified edits); the 10% deadline flush AND the cost wrap-up note gain one shared commit-NEUTRAL tree sentence (acting self_worktree subagents cannot commit; a moved HEAD fails patch capture closed), byte-identical for non-workspace tasks; the ProgramBench instruction now states the exporter's true submission contract (SOURCE tarball from the CURRENT tree state; .git, root binaries, .ouroboros/ and named build/cache noise excluded at any depth; run ./compile.sh one final time), replacing the false fresh-checkout framing. Tests pin the pacing/rails seam, the workspace gating on both axes, byte-identical non-workspace texts, the real tarball contract, and the instruction truth phrases. P9 carriers bumped to 6.74.4; changelog row added. Disclosed residual (this is a MITIGATION of the incident class, not a closure): a forced tool-less exit — deadline grace or budget stop crossed inside one long round with no pacing note or capsule in the terminal stretch — can still ship an unverified last edit; the structural verification-freshness seam is a filed follow-up pending an owner decision (triad r1-r3, codex full-access review concur).
* docs(benchmarks/continual_learning): v6.71.1 validated baseline, ops runbook, refreshed adapter pins
- Add RUNBOOK.md: field-tested at-scale recipe from the 2026-07-20 full
6-domain campaign (validated config incl. max_workers=3 with the Docker-VM
memory-sizing formula, mandatory clone-sweeper/container-reaper daemons,
smoke -> one-seed -> 5-seed submission flow per the official submitting
docs, known loss classes and their closures).
- METHODOLOGY.md: add section 9 with the v6.71.1 full-suite 1-seed results
(6-domain normalized average +0.259 vs published top-1 +0.196 and
Claude Code +0.185) and the review-mode ablation verdict (pin 1 pass);
update the honest-limits section accordingly.
- README.md: point at RUNBOOK.md, refresh the adapter pin (3ec3761: network
outage hold, format-repair round, review-passes override) and the handoff
bundle name, replace the stale internal worker-pool guidance (4 -> 3, with
the OOM rationale).
* docs(benchmarks/continual_learning): reflect the clbench_remote -> remote_work skill rename
* docs(benchmarks/continual_learning): review fixes — canonical domain names, max_workers and adapter-pin qualifiers, #9 status
Triad+scope review fixes on the maintainer rerun (base v6.74.3):
- METHODOLOGY §9 results table: use the bench's canonical task names —
blind_spectrum_monitoring (bsm) and sales_prediction — instead of the
non-canonical black_scholes (bsm) / sales_analytics, so rows map onto
the §1 domain list.
- METHODOLOGY §4: OUROBOROS_MAX_WORKERS disclosure now separates the
2026-07-01 reference-run value (4) from the at-scale value validated by
the v6.71.1 campaign (3, Docker-VM OOM rationale in RUNBOOK.md),
removing the contradiction with README/RUNBOOK.
- METHODOLOGY §3: qualify the 56764d6 adapter pin as the §8 reference-run
pin next to the v6.71.1 campaign pin 3ec3761.
- RUNBOOK: pgasawa/continual-learning-bench#9 merged 2026-07-19 — state
the mixed-metric condition in the past and point at scoring with a
leaderboard checkout that includes the fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Anton Razzhigaev <razzant@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A: verdict-visible improvement capsule (verdict+tier+real blocker via one panel_reason reducer, open obligation ids, money/time/rounds/passes rails line, three real moves); reviewer-authored obligation identity (disposition_kind new|re_raise validated against the host catalog, fail-closed to new, per-pass dedup, reuse-immutable) with surviving rebuttals (previous_disposition/previous_reason/reopened_count); typed dialogue_status (continue_actionable|unreachable_here|stable_disagreement) reduced over ALL contract-valid actors with the panel quorum — termination is reviewer-authored or a real rail, never a host counter.
B: two cache-marked review-prompt segments (governance + task-stable contract), slot label off byte 0, breakpoint cap asserted on the final payload.
C: GAIA sandbox attachment staging (prompt-declared /shared_files channel, exact shared-root lookup, per-file provenance, typed per-sample infra error); CLB acceptance-claims operator patch (receipt-bound claims, knowledge nudge in the steer slot, bounded cost-finality wait); SWE-Pro cumulative shard budget (auto_run derives per_task x scheduled, run_pro seeds from cumulative spend); CLI/PB bounded task_cost_finalized waits for completed/degraded only.
D: light-mode shell guard resolves cwd before judging repo targets (resource-root label false-block fixed); post-task cost publish uses try_get_bridge.
E: generative surface-duty in commit/plan review checklists; all P9 carriers bumped.
Reviewed: 2 adversarial rounds + production triad+scope (PASSED) + codex final (GO); smoke 5x5 on all five benches verified the dialogue/staging/budget mechanics live.
