mirror of
https://github.com/razzant/ouroboros.git
synced 2026-08-31 02:24:51 +00:00
952e210cc0
Rename the worker model slots to a role model where only OUROBOROS_MODEL carries a real default and every other worker slot is empty->Main: - OUROBOROS_MODEL_CODE -> OUROBOROS_MODEL_HEAVY (+USE_LOCAL_CODE -> USE_LOCAL_HEAVY), default now EMPTY -> falls back to Main; OUROBOROS_MODEL_LIGHT default EMPTY -> Main. - New OUROBOROS_MODEL_FALLBACKS comma-separated cross-model chain replacing the singular OUROBOROS_MODEL_FALLBACK. SSOT config.parse_fallback_chain (FALLBACKS, then legacy singular env; no SETTINGS_DEFAULTS injection so an explicit-empty slot means "no fallback") + config.migrate_legacy_slot_keys (used by load_settings AND build_colab_settings). 429-aware resilience: - New ouroboros/fallback_cooldown.py: per-process 429-aware cooldown (default-on, fail-soft, passive timestamp heal) for transiently-failed models (429/5xx/overloaded). - loop._run_cross_model_fallback_chain: deadline-aware walk of the fallback chain skipping cooled-down models, per-candidate total attempt cap, cross-family reasoning sanitize; the bench (FALLBACKS==Main) dedupes to an empty chain -> no-op. Body-error 429 (rate_limit) cools the model down regardless of finish_reason; the primary's same-model transient-retry budget is preserved (cooldown+fallback is the second layer). Subagent lane role model: - Lane "code" -> "heavy" (no legacy alias; durable-data ingress tolerates an unknown stored lane defensively). `auto` routes a mutating first-level child (write_surface OR may_mutate, via normalize_bool) to Heavy and a read-only child to Light; an explicit main/heavy lane is honored only at/below OUROBOROS_SUBAGENT_CAPABILITY_DEPTH_LIMIT (advanced config knob), deeper descendants fall to Light with a visible downgrade note (P1, not a silent horizon cut). Carriers: config, subagents, tools/control, provider_models, server_runtime, settings_setup_contract, tools/vision, llm, pricing, colab_bootstrap, loop, loop_llm_call, safety, tools/claude_advisory_review, prompts/SYSTEM.md, docs/ARCHITECTURE.md, docs/DEVELOPMENT.md, web settings/onboarding, devtools/benchmarks manifest + tests. Version bump is deferred to the final release (authorized P9 deferral, precedent v6.38.0). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| common | ||
| harness_bench_fast | ||
| osworld | ||
| programbench | ||
| swe_bench | ||
| swe_bench_pro | ||
| terminal_bench | ||
| __init__.py | ||
| evolve_smoke.py | ||
| README.md | ||
Ouroboros Benchmark Devtools
This directory contains tracked operator tooling for reproducible benchmark work. These files are reviewed when touched, but are not imported by the runtime core and are not packaged as app runtime code.
Integrations
terminal_bench/— Harbor installed-agent adapter for Terminal-Bench 2.1. Userun_tb.pyfor leaderboard-shaped k-trial runs and submission layout; userun_harbor_smoke.pyfor small local smoke runs.osworld/— OSWorld logs tooling plusrun_step_agent.py, an official env.step-shaped runner that passes VM screenshots as native image attachments to Ouroboros.swe_bench_pro/— SWE-bench Pro patch capture/grading. Frozen prepared repos usepro_predictions.py; persistent evolutionary runs usee1v2/run_pro.py/e1v2/auto_run.py.swe_bench/— standard SWE-bench prediction helpers.programbench/— ProgramBench cleanroom runner.harness_bench_fast/— Ouroboros CLI wrapper and methodology notes for the publicai-forever/harness-bench-fastrunner.common/— shared manifests, result ledgers, safe run roots, secret hygiene, and official command builders.
Output Roots
Write generated run artifacts under an explicit benchmark output root outside
repo/ and outside live runtime data/, typically
/Users/anton/Ouroboros/bench_runs/.... Tests must set
OUROBOROS_BENCH_RUNS_ROOT to a temporary directory so local test runs do not
pollute real benchmark bundles.
Shared Sidecar Schemas
- Run manifests record non-secret provenance: requested task ids where the
benchmark runner exposes them before execution, requested counts/selection
slots for deterministic first-N runs such as Terminal-Bench, exact argv,
official command shape, output paths, model slots, source commit, dirty-state
counts, and hashes. Defaults are adapter-specific (
run_manifest.json,<predictions>.run_manifest.json, orosworld_preflight.run_manifest.json). - Result ledgers are denominator-preserving Ouroboros JSONL files. They record
every requested instance, including setup failures, timeouts, and empty
patches, even when the official benchmark prediction/submission format only
accepts successful rows. Defaults are adapter-specific (
result_index.jsonl,<predictions>.ledger.jsonl, orosworld_preflight.ledger.jsonl).
These sidecars are audit artifacts, not replacement scoring. Official benchmark harnesses and official result files remain the scoring authority.
Methodology Rule
Benchmark changes must be general-purpose harness improvements first. Do not add task-specific answers, hidden verifier knowledge, or resource/timeout overrides that violate a benchmark's official submission rules.