mirror of
https://github.com/razzant/ouroboros.git
synced 2026-08-10 11:03:16 +00:00
24991292e8
WIP checkpoint (local, not pushed) — to be reviewed-merged into repo/ after the re-run.
Fixes two TB2.1 methodology bugs that depressed gpt-5.5's measured score:
1. Network: the adapter set allowed_resources={web:false,network:false} on
disable_agent_web, which blocked `git clone` even though TB2.1 tasks need it
(every task.toml allow_internet=true; verifier clones in-container). Replace
with a declarative `disabled_tools` tool-policy: withhold the agent's own
web_search/browse/browser/analyze_screenshot/vlm_query tools (anti-reward-hack,
== original _WEB_TOOLS set) while leaving shell egress (git/pip/curl) intact.
- task_contract.disabled_tools (additive frozen-ABI field) + normalize_disabled_tools
- registry hides disabled tools from schemas()/core_only/get_schema_by_name/
available_tools (+ dynamic extension/MCP) and blocks them at execute
- /api/tasks + TaskCreateRequest (+ api_types.js mirror) thread the field
- propagates to subagents via the parent-contract spread
- view_image intentionally kept (local image-to-model, outside _WEB_TOOLS)
2. Deadline: Harbor's per-task wall-clock cap was never plumbed into Ouroboros
(deadline_at=""), so pacing/graceful-finalize were dormant and the agent was
hard-killed mid-work with empty results.
- D1: adapter resolves the per-task [agent].timeout_sec from cached task.toml,
subtracts elapsed install/server time + 30s safety, passes timeout_sec ->
deadline_at so the agent paces and self-finalizes a partial before the kill
- D2: surface inherited deadline_at from task_contract into task_metadata so
spawned subagents also pace/finalize
- _capture_current_task_summary records captured_after_cancellation only on a
real interruption (not a terminal infra_failed exit), so the disclosure
ledger no longer misreads a terminal provider_unavailable as a cancellation
Plus the run_tb disclosure-ledger honest taxonomy (pass/provider_infra/cancelled/
genuine) and tests (test_disabled_tools_policy.py, test_run_tb_methodology regression,
adapter terminal-nonzero-exit test) + docs/ARCHITECTURE.md ABI rows.
Reviewed: codex (gpt-5.5) + 7 rounds triad+scope to convergence; only the
version-bump touchpoint remains, intentionally deferred to the eventual merge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| common | ||
| harness_bench_fast | ||
| osworld | ||
| programbench | ||
| swe_bench | ||
| swe_bench_pro | ||
| terminal_bench | ||
| __init__.py | ||
| evolve_smoke.py | ||
| README.md | ||
Ouroboros Benchmark Devtools
This directory contains tracked operator tooling for reproducible benchmark work. These files are reviewed when touched, but are not imported by the runtime core and are not packaged as app runtime code.
Integrations
terminal_bench/— Harbor installed-agent adapter for Terminal-Bench 2.1. Userun_tb.pyfor leaderboard-shaped k-trial runs and submission layout; userun_harbor_smoke.pyfor small local smoke runs.osworld/— OSWorld logs tooling plusrun_step_agent.py, an official env.step-shaped runner that passes VM screenshots as native image attachments to Ouroboros.swe_bench_pro/— SWE-bench Pro patch capture/grading. Frozen prepared repos usepro_predictions.py; persistent evolutionary runs usee1v2/run_pro.py/e1v2/auto_run.py.swe_bench/— standard SWE-bench prediction helpers.programbench/— ProgramBench cleanroom runner.harness_bench_fast/— Ouroboros CLI wrapper and methodology notes for the publicai-forever/harness-bench-fastrunner.common/— shared manifests, result ledgers, safe run roots, secret hygiene, and official command builders.
Output Roots
Write generated run artifacts under an explicit benchmark output root outside
repo/ and outside live runtime data/, typically
/Users/anton/Ouroboros/bench_runs/.... Tests must set
OUROBOROS_BENCH_RUNS_ROOT to a temporary directory so local test runs do not
pollute real benchmark bundles.
Shared Sidecar Schemas
- Run manifests record non-secret provenance: requested task ids where the
benchmark runner exposes them before execution, requested counts/selection
slots for deterministic first-N runs such as Terminal-Bench, exact argv,
official command shape, output paths, model slots, source commit, dirty-state
counts, and hashes. Defaults are adapter-specific (
run_manifest.json,<predictions>.run_manifest.json, orosworld_preflight.run_manifest.json). - Result ledgers are denominator-preserving Ouroboros JSONL files. They record
every requested instance, including setup failures, timeouts, and empty
patches, even when the official benchmark prediction/submission format only
accepts successful rows. Defaults are adapter-specific (
result_index.jsonl,<predictions>.ledger.jsonl, orosworld_preflight.ledger.jsonl).
These sidecars are audit artifacts, not replacement scoring. Official benchmark harnesses and official result files remain the scoring authority.
Methodology Rule
Benchmark changes must be general-purpose harness improvements first. Do not add task-specific answers, hidden verifier knowledge, or resource/timeout overrides that violate a benchmark's official submission rules.