TB2.1 leaderboard static analysis matches config.agents[].name against the
agent name trials report; bare --agent-import-path records name=null, which
can never match (terminal-bench-2-1#121) and is unfixable after upload.
- run_tb.py: generate agent_job_config.json (name = adapter class name() +
import_path + kwargs) and launch harbor with -c instead of bare agent
flags; OUROBOROS_EFFORT_TASK at launch is recorded as the declared
reasoning_effort of the submission key.
- harbor_installed_agent.py: accept the reasoning_effort kwarg and forward
it into the container as OUROBOROS_EFFORT_TASK (declared == effective).
- README/METHODOLOGY: replace the removed 'harbor leaderboard submit' CLI
flow with the verified GitHub-PR flow (terminal-bench-2-1 lb submit,
promotion, /judge, /apply), document the named-config requirement,
trajectory_path pitfalls and the TUS upload fix pointer.
- tests: assert the new -c job-config shape (name/import_path/kwargs/effort).
The v6.62.0/#64 net-permission change removes unix_computer_use from the native
auto-enable class, and the server loads enabled extensions only at startup
(reload_all). A server started BEFORE the runner's _enable_skill write will not
hot-reload it, so the skill's ext_* tools come back "Not found" and the agent
declares the task infeasible. Documented in the cu_bridge README: seed the skill
enabled before server start (the parallel orchestrator already does — fresh
isolated server per task after seeding), or restart after enabling.
Found running a 3-task OSWorld smoke (all reward 1.0 once the skill was enabled
before server start). Docs-only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add remote backends to unix computer use
* fix(unix_computer_use): pass skill review + runner/test cleanup
- screenshots returned in place from skill_job_dir (drop data/uploads writes) — path confinement
- confine target_file reads to the skill state dir (no arbitrary path read)
- add _ssh_scp_source (scp -P + single host:path token), fixing ssh_macos screenshot
- declare runtime: python3 in SKILL.md (executable extension)
- drop unused --model arg from run_cu_bridge_agent (was misleading provenance)
- update remote-screenshot test to assert confinement (OS-agnostic, no /uploads)
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(computer-use): actually deliver screenshots to the model
- view_image: allow reading skill job/state outputs (state/skills/*) so
screenshots the unix_computer_use skill produces are view_image-able
- plugin._downscale: use in-process PIL as the primary resizer so the
screenshot is ALWAYS downscaled to the WXGA cap (1280x720) with a correct
image->input coord_transform, instead of silently no-oping on hosts without
sips/ImageMagick (which left a full-res image + identity transform while
view_image re-downscaled independently -> clicks landed ~1.2x off-target)
- run_cu_bridge_agent: drop the fastdtw/easyocr import stubs that made vlc/OCR
evaluators raise (the real deps are installed)
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(osworld): give the agent the VM sudo password in the prompt
OSWorld ships user/password = user/password (client_password) and its docs say
the agent needs it for sudo tasks (create users, start services, install pkgs).
The bridge created the env with the default password but never told the agent,
so sudo-dependent tasks (e.g. create-SSH-user) failed after the model guessed
wrong passwords. Inject env.client_password into the task prompt.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(osworld): enable DataImpulse proxy pool for proxy-flagged tasks
OSWorld tags ~52 web/multi_apps tasks with "proxy": true and expects the env to
route the VM through a residential proxy (DataImpulse) to avoid bot/geo blocks.
Wire enable_proxy into the CU bridge: resolve the proxy config from
<osworld_root>/evaluation_examples/settings/proxy/dataimpulse.json (or
PROXY_CONFIG_FILE), export it before importing desktop_env (setup.py loads the
pool at import), and pass enable_proxy accordingly. Enabled only when the config
exists so non-proxy runs and the other 317 tasks are unaffected. Creds file is
git-ignored, not committed.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(osworld bench): propagate task-review knobs + verify-final-state preamble
- server_runner: allow OUROBOROS_TASK_REVIEW_MODE / OUROBOROS_REVIEW_ENFORCEMENT
through the isolated-settings allowlist so settings.json review policy is honored.
- run_cu_bridge_agent preamble: before the verdict, verify the actual observed
final VM state satisfies the task (generic, no task-specific hints).
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(v6.62.0): unix_computer_use remote backends + OSWorld cu_bridge finalization
Finalizes PR #64 for merge into the ouroboros branch (origin/ouroboros merged
in; base retargeted from main to ouroboros). Confirmed fixes on top of kazzand's
remote-backend feature, converged over five triad+scope review rounds.
skills/unix_computer_use/plugin.py (remote osworld_http/ssh_macos backends):
- scroll x100 multiplier removed (X11 pyautogui is 1 event per detent, 1:1).
- honest ok: _osworld_result_ok() requires HTTP 200 AND guest returncode 0/
status success (the in-VM server returns 200 even on nonzero exit).
- unicode type_text pastes non-ASCII via the in-VM clipboard (typewrite fallback).
- keys: delete/backspace press BackSpace; fwd-delete presses Delete.
- fail-closed connection selection (verified complete): _is_remote() is name-based
so EVERY input tool routes through the guarded remote path; _active_connection()
marks any non-local connection that is missing/disabled/unknown-backend as
disabled and all tools refuse ("unusable") before any local dispatch — a
remote-aimed click can never hit the host desktop (behavioral test proves local
dispatch is never reached). Atomic registry writes.
- remote screenshot: 20 MB download cap + non-PNG rejection.
devtools/benchmarks/osworld/run_cu_bridge_agent.py:
- infeasible detection reads only final-answer fields.
- ax_tree disabled by default (--allow-a11y); guards for live-server + live-data-dir.
- dataset pin + budget counters; --settings-path defaults to <data-dir>/settings.json
so the max_rounds disclosure reflects the isolated server; _publish_target writes
the registry atomically.
- untrusted-task host lockdown: rather than a fragile denylist, the task is
submitted with a COMPUTED denylist = all core tools minus a small allowlist
(skill discovery/enable, view_image, read-only inspection). Every host
execution/mutation/VCS/GitHub/service/self-mod/chat surface — present or future
— is blocked by construction; the VM is driven only through the skill's ext_*
tools.
- honest manifest: official_actions=false + official_reset_evaluate=true +
action_channel="guest_execute_not_env_step" (matches METHODOLOGY §7).
Core: ouroboros/tools/vision.py root widening kept (uploads + state/skills; same
trust boundary as read_file; rejection message + tool descriptions name the new
root; also fixes the LOCAL vision loop); ouroboros/review.py bumps
MAX_TOTAL_FUNCTIONS 3775->3805, grandfathers skills/unix_computer_use/plugin.py
(repo-relative key), and adds a shared module_is_grandfathered() matcher used by
BOTH the smoke gate and compute_complexity_metrics (codebase_health) so the two
consumers can't diverge and a future skill's plugin.py is not silently exempted.
Version 6.62.0 across all carriers; docs/ARCHITECTURE.md (skill remote backends +
cu_bridge runner), docs/DEVELOPMENT.md (grandfather prose), SKILL.md, osworld
README + METHODOLOGY.md §7 updated.
Tests: 3 stale assertions updated; new remote scroll/ok/unicode/backspace/fail-
closed(missing+disabled+unknown-backend + behavioral proof)/screenshot-cap/real-
PIL-downscale/atomic-registry/settings-default/allowlist-complement/grandfather-
matcher tests; tests/test_osworld_cu_bridge.py.
Accepted advisory tradeoffs (review dissent, non-blocking, documented):
- add_connection keeps 10 keyword params (first-class agent tool schema).
- ssh_macos type_text of repr escape sequences types literally: pre-existing
quirk on a SECONDARY backend; osworld_http (the OSWorld path) is unaffected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Ouroboros <ouroboros@local.mac>
Co-authored-by: a.kaznacheev <a.kaznacheev@sdgroup.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Anton Razzhigaev <razzant@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
4.1 answer_protocol (owner quiz 16b, option C+B): the FINAL ANSWER doctrine
leaves the global prompt. New additive task_contract field answer_protocol
("" | "final_answer_line"; normalize_answer_protocol + answer_protocol_active
SSOT gate in contracts/task_contract.py, exported via contracts __init__),
propagated from /api/tasks (answer_protocol= body field, TaskCreateRequest +
api_types.js mirrors) and CLI --task-metadata-json, inherited by subagents
through the parent-contract spread. When declared, context.py injects the
protocol instruction (with the opt-in CANDIDATES ambiguity block) into the
task's runtime context; the P2 marker nudge and the pacing salvage phrases
(cost wrap-up, 10% deadline flush, intrinsic pacing) activate through the ONE
gate. Without it, ordinary chat/self tasks never see marker prompting: the
SYSTEM.md marker rule + CANDIDATES section are REMOVED; the latch + extractor
+ typed final_answer stay unconditional; final_answer_missing_sentinel keys on
the typed payload (latch-recovered answers are not "missing"); the no-op
nudge keys on expected_output semantics with marker wording only under the
protocol. GAIA's solver declares the field (--task-metadata-json); TB/SWE-Pro/
PB deliberately do NOT (their deliverables are container state / patches /
code, not an extracted line). The web UI renders a FINAL ANSWER line as a
labelled "Answer" chip (presentation only; stored text untouched).
4.2 blocking widening (S1-lite, owner quiz 18b): under required+blocking,
_collect_acceptance_obligations widens from critical-only to critical+HIGH
contributing findings with a concrete recommendation WHEN the aggregate
verdict itself is failing (signal FAIL or worst tier blocked_with_evidence) —
the PB case where reviewers converged on a concrete "misses X" at high
severity yet the task finalized clean. PASS (incl. PASS-with-dissent) keeps
the critical-only bar. The dead verdict_is_advisory request-policy key is
removed from both ReviewRequest sites — enforcement semantics live solely in
OUROBOROS_REVIEW_ENFORCEMENT. SWE-Pro settings_base flips to blocking (PB/TB
adapters already default blocking+required).
4.3 generalized verification: expected_match="bytes_equal" in
verify_and_record compares artifact_paths=[a, b] BYTE-FOR-BYTE after the
check, on the same surface as the check (executor cmp in-container, host
chunked read otherwise), recording a bounded hexdump of the first divergence
in the receipt (golden files, migration parity). schedule_subagent documents
the INDEPENDENT VERIFIER pattern (read-only memory_mode=empty child fed only
the deliverable + acceptance criteria, not the builder's own probes). The
acceptance checklist asks the SCOPE-CUT question explicitly (a silent or
unjustified narrowing is a high-severity finding; under blocking it becomes
an obligation). Effort-ceiling learning now requires the rejection text to
implicate an effort carrier (a generic parameter rejection cannot teach a
phantom ceiling).
The README claimed submissions were CLOSED pending a new process (verified
2026-06-30 against the HF PR flow). That is stale: since 2026-06 Harbor Hub
accepts TB2.1 submissions via `harbor auth login` -> `harbor upload` ->
`harbor leaderboard submit` (harbor >= 0.17), and the tbench.ai docs page
still saying "coming soon" is itself stale. This misdirection already caused
two false "submissions are closed" conclusions.
- README: replace the CLOSED status + HF-PR framing with the verified harbor
CLI flow; document the hard requirements (ATIF trajectory per passing
trial, trajectories before first upload, jobs become public -> mandatory
secret scrub, headless auth options, multi-job submissions, content-hash
task-version checks, dynamic LLM validation); bump harbor 0.13.1 -> 0.17.1.
- METHODOLOGY: add "Leaderboard submission" section (trajectory provenance,
scrub policy, which runs may be submitted).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Harbor Hub static validation requires an ATIF trajectory
(agent/trajectory.json) for every passing trial, and submitted jobs become
public. Add:
- atif.py: stdlib-only ATIF-v1.7 builder mapping recorded trial logs
(tools.jsonl, progress.jsonl, chat.jsonl, events.jsonl, run summary) to an
honest step-by-step trajectory; shared by the adapter and the converter.
- build_atif_trajectories.py: backfill converter for existing job dirs, with
optional harbor schema validation (--validate).
- scrub_submission_secrets.py: pre-upload secret scrubber for submission
copies (structural settings blanking + literal value sweep + zero-leftover
verify pass); required because container-secret runs persist live keys in
every trial's ouroboros-data/settings.json.
- harbor_installed_agent.py: emit /logs/agent/trajectory.json in-container at
the end of each trial (best-effort, covers normal finish and teardown).
- test_atif.py: structural invariants the harbor validator enforces.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found while extending the campaign to codebase_adaptation: the pinned bridge
passes run_index to every task ctor but not every task accepts it; the runner
venv needs mini-swe-agent for the codebase domain and an explicit pip when
built by uv. Operator-run, docs-only, owner's devtools-without-review
allowance.
Three incompatibilities found while launching the v6.56.0 bench campaign:
the safety-mode owner-guard refuses the adapter's env-side full->light
lowering at isolated-server boot (uvicorn rc=3); host.docker.internal via
host-gateway cannot reach a host-loopback step-shim on Linux (agent actions
silently lost, queries=0/reward=None); a rootful daemon writes root-owned
files into the bind-mounted data root (host bridge PermissionError). Ships
the unified diffs (applied to the external checkout, not vendored here) and
documents the clbench_remote seeding constraint under the v6.56.0 native
allowlist. Operator-run: bench harness docs only, no runtime changes;
committed under the owner's explicit devtools-without-review allowance.
bwrap_isolate.py carried an unused `import sys` from before v6.56.0 (the base
commit's quick-test was already red on `ruff check --select F`). Remove it so the
v6.56.0 release CI (which runs the same F-check) is green. No behavior change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Generalizable runtime + harness improvements distilled from the 35-FAIL SWE-bench Pro
autopsy, the ProgramBench improvement spec, and the TB2.1 forensics — one minor release,
no benchmark overfit.
Runtime (A/C):
- Scratch is now idempotent/adoptable: an existing untracked in-cwd file can be declared
scratch (sha recorded at declaration, re-declaration idempotent); tracked files and
directories stay blocked. The undeclared-output guard is stat-verified (real filesystem
writes, not string-shape) and, for run_script, audits on EVERY exit path — a write-then-
fail body no longer smuggles an undeclared user_files deliverable.
- Cost axis: budget_profile.cost_hard_stop_pct (0..100; None→50% historical; 0 = NO in-task
cost stop, deadline/rounds/global-gate only; malformed → 50%, never a silent 0). Latched
cost milestones (50/25/10% + wrap-up) replace the old [INFO] nudge. CLI --task-metadata-json
merges into body.metadata with host-owned keys (delegation_role/source) unforgeable.
Bench harness (D/PB/TB):
- SWE-Pro e1v2: /app is the default active workspace (via user_files root), empty memory
default, budget metadata (until_deadline, cost_hard_stop_pct=0) threaded through; bench1
fixes (shard-safe auto_run, k=1, timeline) ported into the repo; the image-load lock and
utility-image use are cross-platform (platform_layer, preflight-pulled, fail-safe restore).
- Workspace acting-integration allowlist (+integrate/compare_subagent_patches) with a
fail-closed self_worktree-under-external-workspace integration guard.
- Protected-artifact policy round-2: structural (operation-identity) false-positive
exceptions for differential-testing harnesses — vcs_diff/stat classifier, spawn-argv=argv[0]
execute, interpreter script/module operand screening, writer-target segmentation — while
read/copy/hash/introspection of the black-box reference stay blocked (incl. alias-separated
reads); pinned by a new permanent policy test.
- ProgramBench instruction v2, until_deadline profile, acceptance claims. TB ffmpeg resolver
chain (platform_layer/imageio-ffmpeg/PATH) + agent-prefix install + cv2 fallback hint.
- 4 verify-guidance clauses in prompts/SYSTEM.md (probe the consumer interface, exercise every
provided input, robust-under-ambiguity, never read hidden graders).
Docs/versions: 6.56.0 across VERSION/pyproject/web/api_types/README/ARCHITECTURE; SWE-Pro +
TB METHODOLOGY (workspace/budget/memory/retry-gate disclosure), PB gate-20 operator method,
pricing sonnet-5, DEVELOPMENT scratch contract.
Reviewed via operator triad (openai/gpt-5.5 + google/gemini-3.5-flash + anthropic/claude-fable-5)
+ scope (anthropic/claude-fable-5) to convergence: scope clean; residual triad items are a
verified reviewer false-positive and the plan-accepted single-parameter budget-threading
tradeoff on a grandfathered core-loop function (advisory-bypass, documented).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
run_harness could only run a fresh --limit N; resuming a killed codex/claude_code/
hermes row from where it stopped required re-running completed tasks. Add
--sample-id (comma-separated ids, overrides --limit) so a row resumes only its
remaining tasks — methodology-neutral (same agent/model/task, only sample
selection), mirroring the flag run_gaia already exposes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The GAIA answer key is cached on the host (metadata.parquet Final answer column +
per-task .jsonld under ~/.cache/inspect_evals and the HF dataset cache), and every
solver runs its agent as a host subprocess with shell access — so the agent can read
the answers off disk (observed: a Codex row ran `find … gaia_dataset … jq
<sample>.jsonld`; the leakage audit caught it). This is a filesystem sibling of the
web-lookup cheat.
Wrap every solver subprocess (Ouroboros + codex/claude/hermes, symmetrically) in
`bwrap --dev-bind / /` with the answer-cache directories masked by empty tmpfs,
leaving the rest of the FS, the network (web search), and CLI config intact. The
inspect scorer runs in the main process outside the wrapper, so scoring is unaffected.
New bwrap_isolate.py (default-on; GAIA_BWRAP_ISOLATE=0 to disable; fails loudly if
bwrap is missing rather than running silently unprotected). Disclosed in METHODOLOGY.
Verified: answer cache invisible inside the wrap; codex/claude solve end-to-end;
the Ouroboros dedicated server binds loopback and runs inside the namespace.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
codex-cli >=0.142 defaults to a WebSocket transport to /v1/responses that does not
carry OPENAI_API_KEY for API-key (non-ChatGPT-login) auth, so a service-account key
401s with "Missing bearer or basic authentication in header". The codex GAIA row
could not run at all. Define a custom HTTP provider (wire_api=responses) hitting the
SAME direct-OpenAI endpoint and model — transport-only, no endpoint alignment. Gated
by GAIA_CODEX_HTTP_PROVIDER (default on; set 0 if a future codex build fixes WS auth).
Disclosed in METHODOLOGY per-harness coverage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2026-07-04 GAIA runs caught agents looking up the published answer key
(googling "GAIA benchmark ... answer", downloading metadata.jsonl from mirrors)
and running the weak strict_ddgs retriever instead of native web search — the
latter mislabeled as parity with the native-searching Codex/Claude Code targets.
- Anti-lookup prompt: SSOT GAIA_ANTI_LEAK_INSTRUCTION appended by all four
solvers, worded to avoid the benchmark name / "FINAL ANSWER" marker / the
leak-query regex so it neither primes the model nor self-flags in traces.
- Full trace capture for audit parity: Claude Code -> --output-format
stream-json --verbose (per-sample claude_code_trace.jsonl); Codex -> --json
(codex_trace.jsonl). Both were black boxes before.
- Observability: persist usage.web_search_sources on llm_usage rows in
events.jsonl so the audit can see native (server-side) web-search citations,
which never appear in tools.jsonl (behavior-neutral).
- Audit v2 (audit_leakage.py + new SSOT leak_targets.py): strip prompt
boilerplate before scanning traces (no self-flag); catch quoted "FINAL ANSWER"
queries and dash-named GitHub mirror paths; STRONG gold-verbatim only when the
gold answer came FROM a leak source (plain gold in a normal page is weak);
scan events.jsonl native citations by root_task_id; parse CC/codex JSONL
traces; fix WEB_TOOLS (+skill_exec/start_service, -nonexistent fetch_url/bash).
- Scoring: score_gaia.py --leakage-audit reports leakage_adjusted_accuracy
(STRONG-flagged samples counted incorrect) alongside the authoritative score.
- Docs: METHODOLOGY (quality_openrouter_web is the publishable default; strict_ddgs
is a handicap, not parity; pre-registered zeroing rule; per-harness coverage),
README_harness_compare (who-searches table), ARCHITECTURE (three web-access
mechanisms and their model/observability consequences).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The TB harbor adapter hardcoded OUROBOROS_SAFETY_MODE=light and only exposed
review_enforcement as an agent-kwarg that run_tb never passed. To support
alternate scaffold configs (e.g. a gemini-3.5-flash run with a diverse review
triad, advisory enforcement, and safety off):
- harbor_installed_agent.py: safety_mode is now a validated agent-kwarg
(full|light|off, default light — unchanged behavior); forwarded to the
container as OUROBOROS_SAFETY_MODE instead of the hardcoded 'light'.
- run_tb.py: new --review-enforcement {blocking,advisory} (default blocking)
and --safety-mode {full,light,off} (default light) flags, both threaded into
the harbor agent-kwargs. Defaults preserve the prior run behavior exactly.
Custom review triads (distinct reviewer models) + effort are set via the
already-forwarded OUROBOROS_REVIEW_MODELS / OUROBOROS_EFFORT_* env (no --all-model,
which would collapse review to a monoculture). Dry-run verified the harbor
command carries safety_mode=off / review_enforcement=advisory / gemini model.
ruff -F clean; both files compile.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add NousResearch Hermes Agent as a GAIA baseline harness (sibling of
codex/claude_code), for the codex↔ouroboros↔hermes comparison:
- hermes_solver.py: drives `hermes chat -q PROMPT --verbose -m <model>
--provider <provider>` (default OpenRouter gpt-5.5, effort=high from
~/.hermes/config.yaml). chat -q --verbose (not -z) is used deliberately so the
full tool trace (web_search/browser) reaches stdout and is auditable; the
unclipped verbose stream is saved to <sample>/hermes_trace.txt. Web tools stay
enabled for parity with the web-using baselines.
- run_harness.py: register "hermes" harness (GAIA_HERMES_MODEL).
- audit_leakage.py: scan <run>/run_root/samples/*/hermes_trace.txt so the
deterministic leakage layer sees Hermes's real web activity (inspect messages
carry only the final answer for CLI harnesses).
Validated on live smoke: answers extract correctly from the verbose stream;
audit sees web activity (with_web_activity>0), flags none, and the visited-host
sanity shows only wikipedia/openrouter (no huggingface/GAIA). ruff -F clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GAIA validation answers are public on HuggingFace, so web-capable rows need a
post-hoc leakage audit rather than egress sandboxing (parity with web-using
baselines Codex/Claude Code; HAL arXiv 2510.11977 sanctions LLM-judge log
inspection as the alternative to isolation).
- audit_leakage.py: two-layer, diagnostic-only (never mutates scores).
Layer 1 deterministic: flags samples that REQUESTED a GAIA-answer-source URL
(huggingface/hf/datasets-server hosts or a GAIA dataset-file path), ran a
benchmark-hunting web_search query, or RETRIEVED the distinctive gold verbatim;
leak URLs appearing only in returned results are a weak signal for the judge,
not a hard flag (the bare token 'gaia' recurs in legitimate content). Layer 2:
HAL-style LLM judge over each sample's web activity. Handles both harness
shapes (ouroboros tools.jsonl; codex/claude_code/null inspect messages).
- METHODOLOGY.md: leakage-audit protocol, per-harness coverage asymmetry
(Claude Code --output-format json hides intra-CLI tool calls -> transcript+judge
only), and the Hermes cost-reduced k=1 disclosure (TB2.1 not leaderboard-valid).
- README_harness_compare.md: integrity caveat points to the audit instead of
prescribing egress isolation.
Validated on live smoke runs (ouroboros + null + claude_code shapes); ruff -F
clean; test_devtools_benchmarks green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Commit 3/3 of the bench post-mortem sprint (plan smooth-skipping-charm;
commits 1-2 = bf8bde7 v6.54.3, ef786bb v6.54.4). Minor bump per plan.
Shared bench-template scaffold defaults, disclosed in the benchmarks index
and per-bench METHODOLOGY files: OUROBOROS_MAX_WORKERS=4 (same-model
decomposition slots within one task, never best-of-N; TB 2->4, GAIA
--max-workers default 1->4 with explicit 1 as the strict-baseline ablation
and the quality-profile silent 1->5 bump removed, SWE-pro 5->4),
OUROBOROS_SAFETY_MODE=light (disposable jails; deterministic guards stay),
RUNTIME_MODE=pro for container benches (GAIA deliberately stays light: its
solver runs unisolated against a live repo), and claude_code_edit disabled
in every bench solve task (single-model harness measurement). TB raises
_DEADLINE_SAFETY_SEC 30->105 from measured finalization overhead, mirrors
the registry _WEB_TOOLS denylist (youtube_transcript drift fixed, pinned by
a sync test), and the README pins why --all-model keeps review single-model.
ProgramBench e2e runner ported from the colleague's tree, adapted to our
APIs: metadata.budget_profile onto the v6.54.4 contract (footer keys
dropped, reserve pct unit-fixed 0.15->15), solve-model preflight through
migrate_model_value (fail-fast on direct-route legacy ids), atomic
per-instance checkpoints with honored reattach (client poll timeout leaves
the executor alive; the next run skips seed/start and reattaches), explicit
payload-status terminal detection, denominator-preserving ledgers
(skipped_existing rows; skipped = successful for exit code), source-only
submissions (both root binaries excluded by name), idempotent workspace
normalization shared by the prepare-only flow.
New continual_learning/ launcher wraps the EXTERNAL CL-Bench runner
(nothing vendored; fails loudly with obtain instructions), strict-sequential
guard (parallel opt-in is exclusively-stateless-bridge only), standard-path
pointer rows keep the ledger denominator, fidelity gaps of the pinned
external adapter recorded and warned. OSWorld aligned to the official 2.0
protocol (pinned OSWorld-V2@c261cb57, 500-step default, show_result.py
layout, official traj rows), bridge-level final_answer capture with
VM-state-only prompting, preflight verifies the target server's effective
scaffold settings via /api/settings (env cannot configure a --url server)
and refuses the live desktop URL without an explicit flag.
LifelongAgentBench documented as blocked (its 100% run was a no-gold-oracle
artifact). Carriers synced to 6.55.0. Review: 6 triad+scope rounds to a
fully clean verdict (16 confirmed criticals fixed, each with a regression
test); gemini/opus clean in all rounds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port the pre-task self-evolution capability from a.kaznacheev's bench
workspace (developed against v6.45.5) onto the current harness:
- run_pro.py: --pretask-evolution / --pretask-evolution-wait-max /
--solve-step-budget / --no-hard-timeouts flags; per-task evolution
objective built from public task inputs only; pretask_evolution object
in result/timeline rows (present only when the flag is on)
- entrypoint_pro.sh: flag-gated evolve start -> wait (absorbed /
no_promotion / degraded / cycle_completed) -> evolve stop -> import
health-gate with rollback to pre-evolution HEAD on failure
- README: e1v2 'Pre-task evolution (optional, default OFF)' section and
a pointer in the swe_bench_pro README
The baseline path is unchanged when the flag is absent: no new env vars,
mounts, files, or result keys (verified by diff review; optional-attr
access uses the established getattr pattern). Harness tests pass
(test_swe_pro_e1v2 / test_devtools_benchmarks / test_v647_megacommit).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Triangulated against the OFFICIAL harbor 0.13.1 leaderboard validator
(static_validation.py), the HF submission README, 10 real accepted
submission configs, and tbench integrity/timeouts posts:
- Add "Leaderboard Validity Rules (verified)" table + primary-source URLs:
k>=5; task timeout_multiplier==1.0; agent-setup & environment-build
timeout multipliers MUST be null (validator rejects any value; 10/10
accepted submissions null); no resource overrides; web access ALLOWED
(judge-policed, only solution-lookup / TB-site access banned);
pre-built/pinned images (force_build=false) standard (8/10 submissions);
submissions currently CLOSED.
- Fix the misleading "use --environment-build-timeout-multiplier 4 /
--agent-setup-timeout-multiplier 4" advice: now flagged
LEADERBOARD-DISQUALIFYING; the faithful fix for heavy/slow builds is
pre-built images, not a multiplier.
- Add cost reality (k=5 ~$ per trial; verify config before a full run) and
hard-won gotchas (resume vs no-resume, install-timeout, pause artifacts).
Doc-only; no behavior change; no version bump.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run_pro.py / orchestrate_probe.py: forward OPENAI_API_KEY into the task container
(name-only, value via env) and accept either OPENROUTER_API_KEY or OPENAI_API_KEY,
so a --solve-model openai::gpt-5.5 run routes direct to api.openai.com (dodging the
OpenRouter gpt-5.5 429 storm). Additive; 0-impact when OPENAI_API_KEY is unset
(OpenRouter path unchanged). Secret opt-in gate preserved.
METHODOLOGY.md: add authoritative section 0 (Official SWE-bench Pro contract) —
solver input vs evaluator-only metadata, the patch-based resolved-iff rule
(reset->apply->before_repo_set_cmd->run_script->parser; resolved iff
FAIL_TO_PASS u PASS_TO_PASS subset of passed), the 1865-task / 41-repo split
(731 public), local-vs-official separation, issue-#93 + network integrity notes,
and a raw Pass@1 reporting rule. Sources cited.
Devtools-only; no runtime change -> no version bump (P9 exception, owner-authorized).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two GENERAL, leak-free, advisory-only verification-scaffolding improvements from the SWE-bench
Pro forensics. NO new hard block; additive (no scratch / no masking => byte-identical).
Fix#1 — run_command/run_script gain scratch=[...]: a sanctioned channel for a throwaway in-repo
verification file the agent writes, runs, and deletes (e.g. an in-package test that MUST live in
the repo to compile). Declared scratch is exempt from the undeclared-output guard, never registered
as an artifact, confined to the cwd, honored ONLY when the cwd is a git worktree AND the path is
NEW + git-untracked (so it cannot mask a real or pre-existing edit), and EXCLUDED from the workspace
patch via a {abs_path: sha256} manifest (headless) — a file is dropped ONLY while its content still
matches, so a later real file at the same path is never lost. Fingerprints are recorded on every
exit path incl. timeout. The ARTIFACT_AUDIT_GAP strips scratch from the change-set (a separate
real deliverable is still audited). The scratch manifest is excluded from collect_task_artifact_records.
Fix#2 — verify_and_record gains a FLAG-ONLY exit-masking sensor: a shell-string check whose
pipeline can launder the real exit code (operator-aware shlex: '| tail/grep', '|| true', '>/dev/null',
incl. no-space forms) records check_exit_masking on the receipt (status UNCHANGED), projected into
the verification ledger + the acceptance reviewer summary, plus a one-shot ADVISORY finalize nudge —
so a PASS over a possibly-laundered green is reconsidered. P5: flag-only telemetry, decides nothing.
Reviewed via the real triad+scope pipeline (gpt-5.5 + gemini-3.5-flash + opus-4.8 triad, gpt-5.5
scope) to convergence over 5 rounds (6 real findings fixed: manifest staleness, no-space operators,
timeout leak, Windows-portable tests, <=8 params, audit-gap scope). Tests: tests/test_v652_scratch_and_masking.py
(serial, cross-platform). Docs (ARCHITECTURE/DEVELOPMENT/README/bench prompt) + version carriers synced.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reframe the in-repo tests as possibly INCOMPLETE (missing cases), not wrong: derive
required behavior from <requirements>/<interface> and cover spec-implied edge/error cases
the checked-out tests do not exercise; add a step-4 spec-coverage self-check and an
anti-dismissal clause (a genuine failure the spec agrees with is a real bug in the change).
Research-backed (codex gpt-5.5 + multi-angle web): ImpossibleBench shows a blanket
'tests may lie' framing backfires on hard multi-file tasks (raises spec-violating
shortcuts 54%->66%), so the wording stays 'incomplete, not wrong'; the proven ~3x lever
(SWE-bench Pro requirements+interface augmentation) is already injected.
Bench-only devtools prompt; no runtime/contract change; no version bump (owner-authorized).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
v6.51.0's full-test matrix failed (all 3 OS) on 3 pre-existing tests not updated for
the v6.51.0 changes; NO runtime regression:
- test_review_substrate_v2: the two acceptance-diff tests patched collect_turn_diff with
a 1-arg lambda and asserted the agent diff at a top-level key. The v6.51.0 tool path
routes through build_task_acceptance_evidence (collect_turn_diff now takes
include_recent_commit; the agent diff is demoted under the redacted agent_supplied
block). Tests updated to the new layout — intent (host-owned diff, agent value
demoted-not-promoted) preserved.
- test_swe_pro_e1v2 runtime-mode passthrough broke because the consolidated run_pro reads
args.volume_suffix; run_instance now reads it defensively via getattr.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Runtime:
- verify_and_record: recover a stringified-argv `check` via a shared SSOT
(shell_parse.recover_stringified_argv / normalize_check_argv) used by BOTH
run_command and the shell guard (the guard inspects EXACTLY what executes);
a string check runs via a NON-login `sh -c` so it inherits the bootstrapped
PATH (ends the `[go,: not found` exit-127 and the login-shell `go: not found`).
- loop: a one-shot ADVISORY red-verification finalize nudge when the latest
host-attested verify receipt is unreconciled-RED
(outcomes.latest_unreconciled_failed_receipt; `declared` does NOT reconcile).
- task-acceptance reviewer is process-aware (review_evidence.build_task_acceptance_evidence):
full redacted normalized contract + first-class verification_summary +
bounded/redacted tool-call trajectory + leak-safe artifact manifest +
provenance tags, under a deterministic disclosed-truncation budget; critiques
HOW the task was solved — advisory, NO new blocking semantics.
Bench/devtools:
- prompt_baseline.txt: local-green-is-not-the-oracle disclosure + interface-exactness
+ run-the-real-test (anti-grep-mask), scaffold disclosure only (no answer/test leak).
- METHODOLOGY.md: Benchmark Legitimacy & Anti-Cheat section (scaffold allowed;
git-history-mining / gold-test-seeding forbidden, with verified canonical sources).
- run_pro: host zstd image cache (OBO_SWEPRO_IMG_CACHE, opt-in/atomic/fail-soft);
orchestrate_probe consolidation (explicit container-secret opt-in, ensure_outside_repo,
platform_layer timeout group-kill + per-worker container reap).
Reviewed: 5-round real triad+scope convergence (gpt-5.5 scope + triad) + adversarial
subagents (2 rounds) + codex code-review + multi-model plan review. Adversarial
false-completion, leak-safety, and guard==execution tests included.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The codex_solver shelled `codex exec` with no reasoning-effort override, so
runs used codex's own config.toml default (xhigh). For an apples-to-apples
cross-harness comparison with Ouroboros, add GAIA_CODEX_EFFORT (default "high")
and inject `-c model_reasoning_effort=<effort>` into the exec command; record
the chosen effort in the result dict and sample metadata. Mirrors the existing
GAIA_CLAUDE_EFFORT knob in claude_code_solver. README documents the knob.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three accuracy fixes for the Claude Code GAIA solver so its score is not
artificially deflated and is comparable to the other harnesses:
- _extract_final_answer: when the CLI emits a verbose meta-wrapper without the
FINAL ANSWER marker ("the answer remains **Fred**", "already computed: 17.056"),
recover the bare answer via fallback patterns + strip markdown/punctuation,
instead of handing the prose line to the quasi-exact-match scorer.
- --effort knob (GAIA_CLAUDE_EFFORT, default high): pin Claude Code's reasoning
effort to match the other configs (was running at the CLI default, unpinned).
- capture total_cost_usd + usage from the claude -p json envelope into metadata
for accurate per-run cost accounting.
Devtools-only; no runtime change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run_gaia.py gains --sample-id (comma-separated ids) to replay a chosen subset of tasks, mirrored into the Inspect argv. _write_manifest now records the EXACT selected ids via a shared _requested_task_ids() helper when --sample-id is set (else the limit-derived level list), so a sample-id rerun no longer writes a manifest claiming the first N level tasks — keeping the audit/denominator contract honest. Focused test covers the manifest selector and the argv lockstep.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Four streams in one reviewed commit (owner decision).
Reliability (verify-before-done, FR3):
- New core `verify_and_record` tool: the host runs the agent's declared check
through the full run_command machinery (`_PROCESS_COMMAND_TOOLS` shell guards,
owner-restore, light-mode repo-write block, executor backend, env scrub) and
writes a durable host-attested receipt; receipts feed the verification ledger.
- Binary objective-axis flags `receipt_absent` / `expected_output_ungrounded` (M2
zero-grounding) surface an unverified-but-clean turn WITHOUT downgrading solved
(anti-oscillation). One-shot verify-before-done nudge; deadline-flush salvage
prompt (M4); cosmetic reason_code invariant (M6).
Capability:
- `query_code(root=user_files)` read-only structured intelligence over an external
target (e.g. a benchmark /app) with search_code-shape guards + bounded
symlink-safe structural walks.
- External-workspace host commands scrub the Ouroboros repo from PYTHONPATH so a
target's `import web/server` cannot shadow-import the agent.
- Effect-based artifact-audit nudge; vcs_diff workspace-mode regression test.
- Deep COOPERATIVE acting subagents build in ONE host-minted shared git tree, with
a depth-aware reservation so a wide first level cannot starve depth.
UX + bench:
- Skill->hub publish eligibility is one SSOT predicate (`skill_publish_eligibility`)
the gateway serializes (`submit_hub`) and the Skills card renders, ending the
clean-only desync (advisory-only warnings now enable Submit).
- SWE-bench Pro grader: raw Pass@1 headline + separate diagnostic
CONTAMINATION_AUDIT.md (gold never shown to the solver).
CORE_PATCH_NOTICE: protected files (ouroboros/tools/registry.py, ouroboros/safety.py,
prompts/SYSTEM.md, prompts/SAFETY.md) touched under the constitutional flag for the new
verify_and_record core tool + its safety policy + the R4 interface-discipline line.
Reviewed: 2 adversarial subagent rounds (8 findings) + 6 triad+scope rounds +
multi-model plan-review + codex gpt-5.5; all agreed findings fixed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an inspect_ai rig to measure agent HARNESSES on GAIA with the official scorer,
isolating scaffold from model. New solvers wrap external CLIs as inspect solvers:
claude_code_solver (claude -p, key-injected), codex_solver (codex exec), and
null_solver (zero-capability integrity probe). run_harness.py is the reproducible
launcher; daemonize.py detaches multi-hour runs; run_gaia.py gains --subset
(e.g. 2023_all) for all-levels. README documents setup, the model-lock bridge,
and the network-isolation / pass@1 integrity caveats. Devtools-only; no runtime change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Profile-driven E1v2 adapter (shared common/model_slots.py single-model pinning +
profiles/*.json) plus three resilience/validity fixes for the SWE-bench Pro harness.
devtools only; no runtime change and no version bump, consistent with prior
devtools-only commits (e.g. 7f0b35d, 9eae6e9).
- Crash/teardown resilience: run_pro writes the timeline/predictions row BEFORE the
post-solve teardown (volume dump + next image pull), times its docker
cache-load/inspect ops (reaping the zstd child on any failure), and RESUME-skips a
task whose non-empty patch.diff already exists (no docker, via resume_result()).
auto_run gains --task-wall-timeout (9000s): it launches run_pro in its own process
group (platform_layer.subprocess_new_group_kwargs) and on overrun kills the whole
tree (platform_layer.kill_process_tree), reaps it, and removes the named obopro-* /
obopro-dump-* containers, then continues — a colima teardown stall no longer hangs
the run or triggers a needless re-pull/re-solve.
- musl install-in-image transport: Alpine/musl images with no prebuilt
oboros-env-musl volume install Ouroboros into the task image at container start
(venv from mounted source, pip install with a graceful tree-sitter fallback),
removing the musl-skip class.
- Gold-history strip (SWE-bench Pro issue #93, OPEN/unpatched): strip_gold_history.sh
removes future git history from each task image before the agent starts (warn-only)
so the gold solution cannot be recovered via git.
Runtime mode now flows from the settings profile (forwarded via env only when
--runtime-mode is explicit). Adds focused tests (resume no-docker, timeout
kill+cleanup+LEGIT, runtime-mode passthrough, strip keeps base / drops gold) and
documents the behavior in the README and the ARCHITECTURE devtools section.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The v6.46.0 GAIA adapter could not start its dedicated benchmark server while the
desktop app was running: (1) it bound the default Host Service port 8767, which the
live app holds, crashing startup ("port 8767 busy"); (2) settings_base.json ships
EMPTY provider-key placeholders, which apply_settings_to_env pops from os.environ,
erasing the real keys _sanitized_host_env preserved ("No supported provider configured").
Fix (devtools-only, no runtime-core change):
- _free_port(): auto-pick distinct free ports for BOTH the main server (GAIA_OUROBOROS_URL
/ OUROBOROS_SERVER_PORT) and the Host Service (OUROBOROS_HOST_SERVICE_PORT), so the bench
server coexists with the desktop app and parallel configs never collide on a port.
- _resolve_provider_keys(): inject the REAL keys for the configured models + the pinned web
backend (from os.environ, then data/settings.json) into the rendered settings so the
server detects the provider; keys land only in the isolated, gitignored run dir. 'auto'
web pulls all keys (cascade); a pinned backend only its own; OPENAI_BASE_URL is dropped
for an 'openai' web pin (official web_search needs an empty base_url).
Verified out-of-the-box: plain run_gaia.py started the dedicated server cleanly (free Host
Service port, Supervisor ready, key injected) alongside a running desktop app + two live
L1 runs. New focused test (test_gaia_render_injects_keys_and_free_host_service_port) + all
90 devtools tests green; ruff F clean. devtools-only follow-up to v6.46.0 (no version bump,
matching the devtools-commit precedent).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A GAIA validation audit surfaced harness artifacts that masked Ouroboros's real
capability. This lands the GENERALIZABLE fixes (helping all users, not fitting the
benchmark), a Max-mode generative context-window probe, and a reviewed GAIA adapter.
Runtime:
- Budget exhaustion TERMINALIZES undispatched PENDING tasks (`fail_tasks`,
reason_code=budget_exhausted + task_done event, canonical budget_drive_root,
cancel-requested -> CANCELLED) instead of a silent never-dispatched freeze.
- `search_code` gains a shared wall-clock deadline (rg walk + batch loop + Python
fallback, one budget) and a DISTINCT deadline-truncation signal so a timed-out
search never reads as an authoritative "no matches" (OUROBOROS_SEARCH_CODE_WALL_SEC).
- `web_search` backend pin (OUROBOROS_WEBSEARCH_BACKEND: auto|ddgs|openai|openrouter|
anthropic; openai/ddgs are TRUE pins, no cascade); `ddgs` is now a dependency with
429 backoff so a fixed-model run can force pure retrieval.
- `OUROBOROS_OR_PROVIDER` (resilience/repro presets) routes OpenRouter providers for
rate-limit resilience without losing prompt-cache warmth (gap-merged; never overrides
the require_parameters / reasoning-continuity pins), behind a supports_message_cache_control predicate.
Answer integrity: a tool-count-stamped latch recovers a produced FINAL ANSWER whose
marker a later (un-grounded) round drops, while new tool work invalidates a stale latch
(Q7 no-downgrade); acceptance review on the ADVISORY task surface stops over-failing a
grounded answer (surface-keyed: majority-not-single-FAIL, no solved->DEGRADED on empty
coach). The P3 commit/scope immune gate is untouched.
Security: `user_files` honors OUROBOROS_USER_FILES_ROOT (jailed scratch home; '~'/'~user'
and unnamed deliverables stay IN the jail); the authoritative observability blob is
redacted by default (env-only OUROBOROS_OBSERVABILITY_KEEP_RAW opt-in) with an allowlist
secret matcher that spares cognitive content (sha/commit/uuid/route_fp/model/answer).
Capability: a generative context-window probe (public LLMClient.probe_oversized_context
seam) empirically confirms a route's >=1M window from a FREE over-window reject — only on
the explicit Max toggle/save, never auto-confirming a possibly-paid 200, fail-closed,
bypasses a stale lazy cache; plus a cloud.ru route base_url fix.
Bench: the GAIA adapter (devtools) stages attachments robustly, injects the FINAL ANSWER
protocol, crash-isolates per-sample timeouts (+ server-side ceiling so a hung sample
cannot orphan the shared queue), and exposes per-config knobs (vision slot, review models,
web backend, budget, epochs) with cross-provider credential preservation.
Version carriers bumped 6.45.5 -> 6.46.0; MAX_TOTAL_FUNCTIONS 3520 -> 3524 (reviewed
single-purpose helpers); ARCHITECTURE/README updated. Reviewed to convergence (adversarial
subagents + codex gpt-5.5 + real triad+scope) with a green full suite.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add build_env_volume.sh to build the read-only oboros-env conda volume from this
repo's requirements.txt (dependencies only; Ouroboros source is seeded into
/obo-repo and imported via PYTHONPATH), so e1v2/run_pro.py no longer depends on an
externally hand-built environment volume. Document the env build, the single-task
smoke flow, and the local-eval gotchas (Colima DOCKER_HOST for the official
evaluator's docker SDK; rely on per-instance ours_output.json since the official
aggregator KeyErrors on FAIL_TO_PASS casing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Keep GAIA run roots valid when their own per-run settings/data env is active, and retry the transient supervisor startup window so the first sample does not flake before producing a structured answer.
WIP checkpoint (local, not pushed) — to be reviewed-merged into repo/ after the re-run.
Fixes two TB2.1 methodology bugs that depressed gpt-5.5's measured score:
1. Network: the adapter set allowed_resources={web:false,network:false} on
disable_agent_web, which blocked `git clone` even though TB2.1 tasks need it
(every task.toml allow_internet=true; verifier clones in-container). Replace
with a declarative `disabled_tools` tool-policy: withhold the agent's own
web_search/browse/browser/analyze_screenshot/vlm_query tools (anti-reward-hack,
== original _WEB_TOOLS set) while leaving shell egress (git/pip/curl) intact.
- task_contract.disabled_tools (additive frozen-ABI field) + normalize_disabled_tools
- registry hides disabled tools from schemas()/core_only/get_schema_by_name/
available_tools (+ dynamic extension/MCP) and blocks them at execute
- /api/tasks + TaskCreateRequest (+ api_types.js mirror) thread the field
- propagates to subagents via the parent-contract spread
- view_image intentionally kept (local image-to-model, outside _WEB_TOOLS)
2. Deadline: Harbor's per-task wall-clock cap was never plumbed into Ouroboros
(deadline_at=""), so pacing/graceful-finalize were dormant and the agent was
hard-killed mid-work with empty results.
- D1: adapter resolves the per-task [agent].timeout_sec from cached task.toml,
subtracts elapsed install/server time + 30s safety, passes timeout_sec ->
deadline_at so the agent paces and self-finalizes a partial before the kill
- D2: surface inherited deadline_at from task_contract into task_metadata so
spawned subagents also pace/finalize
- _capture_current_task_summary records captured_after_cancellation only on a
real interruption (not a terminal infra_failed exit), so the disclosure
ledger no longer misreads a terminal provider_unavailable as a cancellation
Plus the run_tb disclosure-ledger honest taxonomy (pass/provider_infra/cancelled/
genuine) and tests (test_disabled_tools_policy.py, test_run_tb_methodology regression,
adapter terminal-nonzero-exit test) + docs/ARCHITECTURE.md ABI rows.
Reviewed: codex (gpt-5.5) + 7 rounds triad+scope to convergence; only the
version-bump touchpoint remains, intentionally deferred to the eventual merge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
THREE changes from the TB2.1 trace-deepdive follow-up, plus all real fixes from the 3-way review
gauntlet (Ouroboros triad+scope, codex gpt-5.5, adversarial multimodel).
#1 (devtools/run_tb.apply_all_model): --all-model now defaults to ONE reviewer at low effort
(3 identical = monoculture, no diversity); configurable via --review-slots / --review-effort.
Bench env only; no config default change.
#2 (ouroboros/tools/shell._resolve_effective_timeout): SSOT hygiene. OUROBOROS_TOOL_TIMEOUT_SEC=600
was silently dropped to the in-code 360 (the `!= default_setting` skip). Rewritten as one
normalized pipeline (override>env>settings>config>fallback -> ceiling -> deadline-clamp -> >=1);
env/settings no longer bypass the clamp. + diagnostic TOOL_TIMEOUT message (start_service /
timeout_sec= / best-effort-before-deadline; no command-keyword classifier). RELEASE NOTE: ex-360
installs now get 600. (0 TOOL_TIMEOUT events in the TB run -- hygiene, not the benchmark fix.)
#3 (ouroboros/tools/vision.view_image + prompts/SYSTEM.md): new tool injecting a LOCAL image NATIVELY
into the active model's context (reusing _inject_native_screenshot + supports_vision + K=3
eviction). LOCAL PATHS ONLY (no URL/base64); reuses vlm_query's EXACT trust checks via shared
_load_local_image_payload. Registered OUTSIDE _WEB_TOOLS so allowed_resources.web=false no longer
blocks local vision. Resource class: local_file_to_model.
GAUNTLET FIXES (all verified real; rejected items were decided tradeoffs / cosmetic / pre-existing):
- CRITICAL (triad+adversarial): view_image was missing from the tool-visibility allowlists, so it was
INVISIBLE in workspace_mode (the benchmark!) and to subagents. Added to
registry._WORKSPACE_ALLOWED_TOOLS + tool_capabilities.{CORE,LOCAL_READONLY_SUBAGENT,ACTING_SUBAGENT}
(matching vlm_query/analyze_screenshot). (_EPHEMERAL_ALLOWED_TOOLS intentionally NOT added — vlm_query
isn't there either.)
- CRITICAL (triad): view_image had no safety.TOOL_POLICY entry (breaks the coverage invariant test).
Added "view_image": POLICY_SKIP (matches vlm_query/analyze_screenshot; a local-read tool, no shell/net).
- codex+adversarial: the K=3 image-eviction placeholder pointed re-view at vlm_query (web-blocked under
web=false); changed to view_image (loop.py).
- codex: view_image saved re-view copies with a doubled extension (chart.png.png); use stem + actual
(downscaled) mime ext.
- BLOCKER (codex+adversarial): stale tests updated to the new behavior — test_run_tb_methodology.py
(1 reviewer + low + a review_slots=3 case), test_shell_run_shell.py (360->600 SSOT), and
test_multimodal_chat.py (view_image re-view placeholder).
- adversarial: the new timeout tests now stub load_settings() for determinism.
- triad: documented view_image in docs/ARCHITECTURE.md (vision-tools paragraph).
Declined (evidence-based): shared native-injection helper extraction (advisory; eviction is already
centralized in _evict_stale_image_blocks so no drift; ~8-line duplication; no-overengineering);
rename _RUN_SHELL_DEFAULT_TIMEOUT_SEC (cosmetic); vlm_query message wording (cosmetic, no test breaks);
symlink-deny (confirmed safe: resolve()-then-contain). Version bump (6.40.1) happens at MERGE.
Tests: 275 focused tests green (vision, multimodal, shell, safety-policy invariant, tool_capabilities,
devtools, iteration-2).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
