mirror of
https://github.com/razzant/ouroboros.git
synced 2026-08-04 16:19:50 +00:00
c36e9cd4cf
A GAIA validation audit surfaced harness artifacts that masked Ouroboros's real capability. This lands the GENERALIZABLE fixes (helping all users, not fitting the benchmark), a Max-mode generative context-window probe, and a reviewed GAIA adapter. Runtime: - Budget exhaustion TERMINALIZES undispatched PENDING tasks (`fail_tasks`, reason_code=budget_exhausted + task_done event, canonical budget_drive_root, cancel-requested -> CANCELLED) instead of a silent never-dispatched freeze. - `search_code` gains a shared wall-clock deadline (rg walk + batch loop + Python fallback, one budget) and a DISTINCT deadline-truncation signal so a timed-out search never reads as an authoritative "no matches" (OUROBOROS_SEARCH_CODE_WALL_SEC). - `web_search` backend pin (OUROBOROS_WEBSEARCH_BACKEND: auto|ddgs|openai|openrouter| anthropic; openai/ddgs are TRUE pins, no cascade); `ddgs` is now a dependency with 429 backoff so a fixed-model run can force pure retrieval. - `OUROBOROS_OR_PROVIDER` (resilience/repro presets) routes OpenRouter providers for rate-limit resilience without losing prompt-cache warmth (gap-merged; never overrides the require_parameters / reasoning-continuity pins), behind a supports_message_cache_control predicate. Answer integrity: a tool-count-stamped latch recovers a produced FINAL ANSWER whose marker a later (un-grounded) round drops, while new tool work invalidates a stale latch (Q7 no-downgrade); acceptance review on the ADVISORY task surface stops over-failing a grounded answer (surface-keyed: majority-not-single-FAIL, no solved->DEGRADED on empty coach). The P3 commit/scope immune gate is untouched. Security: `user_files` honors OUROBOROS_USER_FILES_ROOT (jailed scratch home; '~'/'~user' and unnamed deliverables stay IN the jail); the authoritative observability blob is redacted by default (env-only OUROBOROS_OBSERVABILITY_KEEP_RAW opt-in) with an allowlist secret matcher that spares cognitive content (sha/commit/uuid/route_fp/model/answer). Capability: a generative context-window probe (public LLMClient.probe_oversized_context seam) empirically confirms a route's >=1M window from a FREE over-window reject — only on the explicit Max toggle/save, never auto-confirming a possibly-paid 200, fail-closed, bypasses a stale lazy cache; plus a cloud.ru route base_url fix. Bench: the GAIA adapter (devtools) stages attachments robustly, injects the FINAL ANSWER protocol, crash-isolates per-sample timeouts (+ server-side ceiling so a hung sample cannot orphan the shared queue), and exposes per-config knobs (vision slot, review models, web backend, budget, epochs) with cross-provider credential preservation. Version carriers bumped 6.45.5 -> 6.46.0; MAX_TOTAL_FUNCTIONS 3520 -> 3524 (reviewed single-purpose helpers); ARCHITECTURE/README updated. Reviewed to convergence (adversarial subagents + codex gpt-5.5 + real triad+scope) with a green full suite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| common | ||
| gaia | ||
| harness_bench_fast | ||
| osworld | ||
| programbench | ||
| swe_bench | ||
| swe_bench_pro | ||
| terminal_bench | ||
| __init__.py | ||
| evolve_smoke.py | ||
| README.md | ||
Ouroboros Benchmark Devtools
This directory contains tracked operator tooling for reproducible benchmark work. These files are reviewed when touched, but are not imported by the runtime core and are not packaged as app runtime code.
Integrations
terminal_bench/— Harbor installed-agent adapter for Terminal-Bench 2.1. Userun_tb.pyfor leaderboard-shaped k-trial runs and submission layout; userun_harbor_smoke.pyfor small local smoke runs.osworld/— OSWorld logs tooling plusrun_step_agent.py, an official env.step-shaped runner that passes VM screenshots as native image attachments to Ouroboros.swe_bench_pro/— SWE-bench Pro patch capture/grading. Frozen prepared repos usepro_predictions.py; persistent evolutionary runs usee1v2/run_pro.py/e1v2/auto_run.py.swe_bench/— standard SWE-bench prediction helpers.programbench/— ProgramBench cleanroom runner.harness_bench_fast/— Ouroboros CLI wrapper and methodology notes for the publicai-forever/harness-bench-fastrunner.common/— shared manifests, result ledgers, safe run roots, secret hygiene, and official command builders.
Output Roots
Write generated run artifacts under an explicit benchmark output root outside
repo/ and outside live runtime data/, typically
/Users/anton/Ouroboros/bench_runs/.... Tests must set
OUROBOROS_BENCH_RUNS_ROOT to a temporary directory so local test runs do not
pollute real benchmark bundles.
Shared Sidecar Schemas
- Run manifests record non-secret provenance: requested task ids where the
benchmark runner exposes them before execution, requested counts/selection
slots for deterministic first-N runs such as Terminal-Bench, exact argv,
official command shape, output paths, model slots, source commit, dirty-state
counts, and hashes. Defaults are adapter-specific (
run_manifest.json,<predictions>.run_manifest.json, orosworld_preflight.run_manifest.json). - Result ledgers are denominator-preserving Ouroboros JSONL files. They record
every requested instance, including setup failures, timeouts, and empty
patches, even when the official benchmark prediction/submission format only
accepts successful rows. Defaults are adapter-specific (
result_index.jsonl,<predictions>.ledger.jsonl, orosworld_preflight.ledger.jsonl).
These sidecars are audit artifacts, not replacement scoring. Official benchmark harnesses and official result files remain the scoring authority.
Methodology Rule
Benchmark changes must be general-purpose harness improvements first. Do not add task-specific answers, hidden verifier knowledge, or resource/timeout overrides that violate a benchmark's official submission rules.