vrr/ouroboros
2
0
Fork 0
mirror of https://github.com/razzant/ouroboros.git synced 2026-08-07 09:33:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9
ouroboros/devtools/benchmarks
History
Exact
Ouroboros 41c8c0fc7a devtools(gaia): pin codex reasoning effort via GAIA_CODEX_EFFORT 
The codex_solver shelled `codex exec` with no reasoning-effort override, so
runs used codex's own config.toml default (xhigh). For an apples-to-apples
cross-harness comparison with Ouroboros, add GAIA_CODEX_EFFORT (default "high")
and inject `-c model_reasoning_effort=<effort>` into the exec command; record
the chosen effort in the result dict and sample metadata. Mirrors the existing
GAIA_CLAUDE_EFFORT knob in claude_code_solver. README documents the knob.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 20:59:58 +03:00
..
common devtools(swe-bench-pro): adapter hardening — crash-resilience + musl install-in-image + gold-history strip 2026-06-25 18:40:52 +03:00
gaia devtools(gaia): pin codex reasoning effort via GAIA_CODEX_EFFORT 2026-06-27 20:59:58 +03:00
harness_bench_fast devtools(bench): upgrade harness-bench-fast adapter + rebase slot keys to v6.39 HEAVY/FALLBACKS 2026-06-20 23:44:06 +03:00
osworld feat(core): v6.40.0 — LLM-first project naming, per-model self-DoS guard, soft join-ledger, orchestrator read-only roots, turn-into-project ordering, deferrals 1-5 2026-06-21 18:32:37 +03:00
programbench swe-pro: e1v2 evolution harness (replaces evolve_pro) 2026-06-11 16:10:31 +03:00
swe_bench feat(devtools): add auditable benchmark ledgers 2026-06-07 09:44:20 +03:00
swe_bench_pro feat(v6.50.0): add subagent reconciliation layer and harden SWE-Pro adapter 2026-06-27 08:04:57 +03:00
terminal_bench devtools(tb): faithful-network tool-policy + agent deadline awareness (TB2.1 methodology fix) 2026-06-23 22:09:02 +03:00
__init__.py feat(devtools-benchmarks): add official benchmark harnesses and workspace executor 2026-06-06 12:03:30 +03:00
evolve_smoke.py swe-pro: e1v2 evolution harness (replaces evolve_pro) 2026-06-11 16:10:31 +03:00
README.md swe-pro: e1v2 evolution harness (replaces evolve_pro) 2026-06-11 16:10:31 +03:00

README.md

Ouroboros Benchmark Devtools

This directory contains tracked operator tooling for reproducible benchmark work. These files are reviewed when touched, but are not imported by the runtime core and are not packaged as app runtime code.

Integrations

  • terminal_bench/ — Harbor installed-agent adapter for Terminal-Bench 2.1. Use run_tb.py for leaderboard-shaped k-trial runs and submission layout; use run_harbor_smoke.py for small local smoke runs.
  • osworld/ — OSWorld logs tooling plus run_step_agent.py, an official env.step-shaped runner that passes VM screenshots as native image attachments to Ouroboros.
  • swe_bench_pro/ — SWE-bench Pro patch capture/grading. Frozen prepared repos use pro_predictions.py; persistent evolutionary runs use e1v2/run_pro.py / e1v2/auto_run.py.
  • swe_bench/ — standard SWE-bench prediction helpers.
  • programbench/ — ProgramBench cleanroom runner.
  • harness_bench_fast/ — Ouroboros CLI wrapper and methodology notes for the public ai-forever/harness-bench-fast runner.
  • common/ — shared manifests, result ledgers, safe run roots, secret hygiene, and official command builders.

Output Roots

Write generated run artifacts under an explicit benchmark output root outside repo/ and outside live runtime data/, typically /Users/anton/Ouroboros/bench_runs/.... Tests must set OUROBOROS_BENCH_RUNS_ROOT to a temporary directory so local test runs do not pollute real benchmark bundles.

Shared Sidecar Schemas

  • Run manifests record non-secret provenance: requested task ids where the benchmark runner exposes them before execution, requested counts/selection slots for deterministic first-N runs such as Terminal-Bench, exact argv, official command shape, output paths, model slots, source commit, dirty-state counts, and hashes. Defaults are adapter-specific (run_manifest.json, <predictions>.run_manifest.json, or osworld_preflight.run_manifest.json).
  • Result ledgers are denominator-preserving Ouroboros JSONL files. They record every requested instance, including setup failures, timeouts, and empty patches, even when the official benchmark prediction/submission format only accepts successful rows. Defaults are adapter-specific (result_index.jsonl, <predictions>.ledger.jsonl, or osworld_preflight.ledger.jsonl).

These sidecars are audit artifacts, not replacement scoring. Official benchmark harnesses and official result files remain the scoring authority.

Methodology Rule

Benchmark changes must be general-purpose harness improvements first. Do not add task-specific answers, hidden verifier knowledge, or resource/timeout overrides that violate a benchmark's official submission rules.