ouroboros/devtools/benchmarks/common
Anton Razzhigaev 7b2f68fd04 devtools: pin bench review-cycle ceiling; docs: contract-fingerprint scope note
Owner decisions Q6-A and Q3-A (2026-08-21).

Q6-A: the shared OUROBOROS_REVIEW_MAX_CYCLES cap (shipped default "2") also
bounds task-acceptance panels (passes = cycles - 1); past bench campaigns ran
3-26 paid acceptance panels per task, so the shipped default would silently
change bench methodology comparability. Pin the ceiling off in every bench
scaffold:
- add "OUROBOROS_REVIEW_MAX_CYCLES": "unlimited" (the canonical unlimited
  sentinel per ouroboros/review_cycles.py) to all bench settings templates:
  continual_learning, gaia, osworld, programbench settings_base.json and the
  four swe_bench_pro e1v2 templates (settings_base, _run_settings.example,
  settings_sonnet46_probe, profiles/light_subagents_gpt55);
- terminal_bench has no JSON template: pin the same value in the
  harbor_installed_agent container env beside the other review knobs;
- forward the key through _ISO_SETTINGS_ALLOW_EXACT in common/server_runner.py
  so a live-settings pin reaches isolated bench servers like the other
  non-secret review-policy knobs.

Also align swe_bench_pro/e1v2/settings_base.json to
OUROBOROS_REVIEW_ENFORCEMENT=advisory: METHODOLOGY.md documents that E1v2
templates deliberately stay advisory (the in-bench commit_reviewed evolution
lane is structurally uncommittable under blocking with the no-version-bump
steer); "blocking" in settings_base was a disclosed discrepancy. Pure JSON
forbids an inline comment, so the rationale lives here and in METHODOLOGY.md.

Q3-A: document in docs/DEVELOPMENT.md the deliberate review-contract
fingerprint scope — roster, routes, enforcement, resolved efforts, prompt
constants; governance-document contents (BIBLE/CHECKLISTS/ARCHITECTURE) stay
outside it, so editing them neither lapses recorded verdicts nor frees
replays; the accepted trade-off is that an old verdict can replay under
amended governance text.

No scoring/grading/task code touched; version carriers unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 04:46:26 +00:00
..
__init__.py feat(devtools-benchmarks): add official benchmark harnesses and workspace executor 2026-06-06 12:03:30 +03:00
launcher_audit.py feat: add apply_patch and edit_batch editing tools with editbench evidence 2026-08-06 00:03:01 +03:00
manifests.py Fix benchmark actor provenance 2026-08-20 02:32:32 +03:00
model_slots.py Bind Harbor smoke to its effective Light actor 2026-08-20 04:52:26 +03:00
official_commands.py feat: v6.55.0 bench devtools alignment — scaffold defaults, ProgramBench e2e, CLB launcher, OSWorld 2.0 2026-07-03 22:49:01 +03:00
result_index.py Review-fix batch (Sol delta review on 50e824fe..d18d7ebf): owner-stop deadline honesty 2026-08-15 19:19:57 +03:00
run_roots.py feat(v6.81.0): benchmark provenance becomes a gate, and the artefacts stop lying 2026-07-26 03:40:44 +00:00
secrets.py fix(minimax): review findings — slash-form routing, docs pins, secret enums 2026-08-04 16:09:57 +03:00
server_runner.py devtools: pin bench review-cycle ceiling; docs: contract-fingerprint scope note 2026-08-21 04:46:26 +00:00
subprocesses.py feat(devtools-benchmarks): add official benchmark harnesses and workspace executor 2026-06-06 12:03:30 +03:00