mirror of
https://github.com/razzant/ouroboros.git
synced 2026-08-06 09:09:48 +00:00
eb33abd491
Phase 1 Blocks 4+6 of the combined release plan (owner-approved single commit): Block 4 — unix_computer_use (renamed from computer_use; Windows stays a future separate skill per P7): - Coordinate normalization: screenshots downscale to fit WXGA and persist the exact image->input transform; input tools consume the last screenshot's image space (raw=true bypass); honest approx flags for multi-display unions (incl. deceptively clean sub-1.0 scales) and TCC-denied logical-size failures. - Wayland: session detection, grim capture, ydotool pointer (mask codes 0x40|btn / 0x80|btn), wtype typing; key/hold_key are honestly unsupported on Wayland (ydotool key takes raw keycodes only — silent no-op success is refused). - New actions: left_click_drag, mouse_down/mouse_up, triple_click, hold_key (macOS pure-modifier-only honesty), cursor_position, wait. - X11 keys: f1-f16 aliases, super/meta modifiers, original-case preservation for case-sensitive keysyms in key AND hold_key; '--' option terminator in type_text. - ax_tree: macOS set-of-marks of the frontmost window (numbered role/title/center marks, INPUT-space coordinates) with honest degradation; Linux degrades to window list. - Native launcher-seed trust (named, hash-pinned, audited exception — CHECKLISTS §Skills, plan-approved): the launcher stamps review.json status=clean (repo_commit_gate/native_seed) at its three payload-write moments; verdict hash-pinned AFTER seeding and provenance-BOUND at load (no .seed-origin or unverifiable provenance => pending); hash exemption covers ONLY the top-level marker of native-bucket payloads; one-shot legacy-hash migration cannot whitewash edits; zero-grant skills (tool/subprocess surface only) auto-enable only when no explicit owner choice exists (a disable survives version resync); OUROBOROS_TRUST_NATIVE_SEEDED_SKILLS default-on opt-out. Block 6 — bench adapters (devtools): - run_tb.py: methodology-allowed setup/build timeout multipliers (4x); per-task multiplier stays 1.0. - Single-model metric hardening: the container fallback is pinned to the effective main model in every reachable configuration (kwarg -> forwarded host model -> packaged default) so SETTINGS_DEFAULTS cannot resurrect a different fallback model inside the settings-less container; host fallback is never forwarded. - max_workers default 2 (plan_task scouts); defensive AgentContext deadline probe (no-op today, harbor-docs-verified). Also: SYSTEM.md attempt-cap paragraph (plan-approved surface, v6.30 catch-up doc-sync). Upgrade note: legacy computer_use payloads remain as reclassified non-launcher skills until manually deleted. Gauntlet: adversarial pass 1 (GPT+Gemini, 5 blockers fixed) + pass 2 (Fable, SAFE TO COMMIT, 3 minors applied); triad+scope rounds 1-5 with scope responded every round; round 5 clean (0 criticals). |
||
|---|---|---|
| .. | ||
| common | ||
| harness_bench_fast | ||
| osworld | ||
| programbench | ||
| swe_bench | ||
| swe_bench_pro | ||
| terminal_bench | ||
| __init__.py | ||
| evolve_smoke.py | ||
| README.md | ||
Ouroboros Benchmark Devtools
This directory contains tracked operator tooling for reproducible benchmark work. These files are reviewed when touched, but are not imported by the runtime core and are not packaged as app runtime code.
Integrations
terminal_bench/— Harbor installed-agent adapter for Terminal-Bench 2.1. Userun_tb.pyfor leaderboard-shaped k-trial runs and submission layout; userun_harbor_smoke.pyfor small local smoke runs.osworld/— OSWorld logs tooling plusrun_step_agent.py, an official env.step-shaped runner that passes VM screenshots as native image attachments to Ouroboros.swe_bench_pro/— SWE-bench Pro patch capture/grading. Frozen prepared repos usepro_predictions.py; persistent evolutionary runs usee1v2/run_pro.py/e1v2/auto_run.py.swe_bench/— standard SWE-bench prediction helpers.programbench/— ProgramBench cleanroom runner.harness_bench_fast/— Ouroboros CLI wrapper and methodology notes for the publicai-forever/harness-bench-fastrunner.common/— shared manifests, result ledgers, safe run roots, secret hygiene, and official command builders.
Output Roots
Write generated run artifacts under an explicit benchmark output root outside
repo/ and outside live runtime data/, typically
/Users/anton/Ouroboros/bench_runs/.... Tests must set
OUROBOROS_BENCH_RUNS_ROOT to a temporary directory so local test runs do not
pollute real benchmark bundles.
Shared Sidecar Schemas
- Run manifests record non-secret provenance: requested task ids where the
benchmark runner exposes them before execution, requested counts/selection
slots for deterministic first-N runs such as Terminal-Bench, exact argv,
official command shape, output paths, model slots, source commit, dirty-state
counts, and hashes. Defaults are adapter-specific (
run_manifest.json,<predictions>.run_manifest.json, orosworld_preflight.run_manifest.json). - Result ledgers are denominator-preserving Ouroboros JSONL files. They record
every requested instance, including setup failures, timeouts, and empty
patches, even when the official benchmark prediction/submission format only
accepts successful rows. Defaults are adapter-specific (
result_index.jsonl,<predictions>.ledger.jsonl, orosworld_preflight.ledger.jsonl).
These sidecars are audit artifacts, not replacement scoring. Official benchmark harnesses and official result files remain the scoring authority.
Methodology Rule
Benchmark changes must be general-purpose harness improvements first. Do not add task-specific answers, hidden verifier knowledge, or resource/timeout overrides that violate a benchmark's official submission rules.