mirror of
https://github.com/razzant/ouroboros.git
synced 2026-08-06 09:09:48 +00:00
6ff64927c4
Add denominator-preserving benchmark manifests and result ledgers across the devtools adapters, tighten Terminal-Bench/Harbor and OSWorld fail-closed preflights, and harden workspace patch artifacts for unborn, invalid, and acting-subagent git baselines.\n\nVerification: focused benchmark/headless/acting suites, isolated full pytest, adversarial review rounds, and real Ouroboros triad/scope review artifacts under bench_runs/devtools_sync_20260607_010949/reviews. |
||
|---|---|---|
| .. | ||
| common | ||
| osworld | ||
| programbench | ||
| swe_bench | ||
| swe_bench_pro | ||
| terminal_bench | ||
| __init__.py | ||
| README.md | ||
Benchmark Devtools
This directory contains thin adapters around official benchmark harnesses. The adapters prepare Ouroboros tasks, capture artifacts, and preserve traces; they do not replace official scoring.
Supported surfaces:
- ProgramBench: official
programbench eval/infoand cleanroom submission layout. - Terminal-Bench: Harbor installed-agent adapter.
- SWE-bench and SWE-bench Verified: standard predictions JSONL for the official SWE-bench harness.
- SWE-bench Pro: official Pro eval wrapper and patch-capture methodology. The prediction/capture path is ledgered; the grader wrapper stays official-output-only.
- OSWorld: stop-closed skeleton and logs normalizer only.
Shared sidecar schemas:
- Run manifests record non-secret provenance: requested task ids where the
benchmark runner exposes them before execution, requested counts/selection
slots for deterministic first-N runs such as Terminal-Bench, exact argv,
official command shape, output paths, model slots, source commit, dirty-state
counts, and hashes. Defaults are adapter-specific (
run_manifest.json,<predictions>.run_manifest.json, orosworld_preflight.run_manifest.json). - Result ledgers are denominator-preserving Ouroboros JSONL files. They record
every requested instance, including setup failures, timeouts, and empty
patches, even when the official benchmark prediction/submission format only
accepts successful rows. Defaults are adapter-specific (
result_index.jsonl,<predictions>.ledger.jsonl, orosworld_preflight.ledger.jsonl).
These sidecars are audit artifacts, not replacement scoring. Official benchmark harnesses and official result files remain the scoring authority.