mirror of
https://github.com/razzant/ouroboros.git
synced 2026-08-05 00:29:46 +00:00
a5bdf5e010
Two provenance fixes in the same class as the release itself, plus the round-11/12 review findings that land in the same files. FIX A — an isolated benchmark container carries only the providers the run declared. `build_isolated_settings` copied EVERY provider credential present in the live settings file regardless of which providers the run's model slots declared, so a run pinned to OpenRouter still received direct ANTHROPIC / OPENAI / Cloud.ru / GigaChat keys. Two consequences: a routing fallback could spend outside the declared bucket while the manifest said otherwise, and the reachable provider set was a function of whatever happened to be in the live file at launch — a pinned seed that pins the code but not the environment is not reproducible. Provider credentials are now gated on the run's declared slots, derived from the routing SSOT (`provider_models.PROVIDER_PREFIXES` / `provider_for_model`, the same registry `llm._resolve_remote_target` routes on), and travel in whole groups so a key never arrives without the endpoint or auth fields it is useless without. Owner/control secrets were never copied and still are not. Ambiguity fails OPEN (carry a spare, disclose it) — never closed. `benchmark_run_manifest` now records `provider_credentials`: which credentials the container actually received, by fingerprint, never by value. FIX B — a task the cost rail truncated says so. `usage_accounting.reserve_- attempt` refuses on a worst-case reservation bound that reached a $6.00 rail at $0.45 of actual spend in the v6.81.0 OSWorld smoke, stopping two of three tasks at 13 and 22 rounds; the artefacts published `status=completed`, `reason_code=official_evaluate` and the string `budget_exhausted` appeared nowhere. `task_result_row` gains an always-present `runtime_outcome` projected by the new shared `runtime_terminal_disclosure`, and every writer holding a runtime task result now publishes it: OSWorld cu_bridge, SWE-bench and ProgramBench success rows (the failure rows already did), the Terminal-Bench in-container summary and disclosure ledger, the GAIA solver, the harness-bench wrapper, and the CL-Bench per-question writer. Reward, `official_eval_status` and adapter-stage `status` are untouched: disclosure ADDED, fact not subtracted. TB's ledger gains a `cost_truncated` bucket — `genuine` asserts a fair shot, which a rail-truncated trial did not get. Review findings folded in: - `_amend_manifest` emitted `output_paths.task_outcome` unconditionally, so the finalized attempt manifest kept pointing at an outcome whose write failed. The previous round fixed the ledger row and left the manifest lying; both sides now follow the same rule. - `ADAPTER_PATCH_MARKERS` keyed two of three detections on bare env-var names, which the unpatched adapter may mention in a comment or a `-e` passthrough list. That false positive OVERSTATES enforcement. Markers are now patch-unique tokens and the uniqueness requirement is recorded beside them. (One marker legitimately covers all three env knobs: they arrive in one loop in one hunk.) - CLB fidelity overstated enforcement on the DEFAULT `--path standard`: `_docker_launcher.submit()` hardcodes `disabled_tools: []` and never imports the patched bridge module, so the evidence is now entrypoint-specific. - `runtime_attested` renamed to `runtime_attestation_available`: it is a tree probe, and a definition existing is not evidence that it ran. - README version badge alt text said 6.80.0 while the URL said 6.81.0, which `version_carrier_desyncs` flags and the advisory preflight blocks on. Bug-pinning tests inverted, and said so in the test docstrings: - `test_dry_run_claims_attestation_only_when_the_patch_is_in_the_execution_- clone` asserted `runtime_attested is True` for a DRY RUN against a clone that merely contained an `_attest_runtime` definition — it demanded the false positive as the contract. - the CLB fidelity fixtures wrote bare env-var names as "the patch", which is precisely the marker weakness above. Docs corrected where the code falsified them: OSWorld METHODOLOGY §7.4 claimed `OUROBOROS_MAX_ROUNDS` plus the timeout were the only per-task caps (the USD rail binds first), §6 now says scoring reads `official_eval_status` / `details.outcome_status` rather than filtering on `status == "completed"` and that `output_paths.task_outcome` may be absent; CLB METHODOLOGY §3 documents when an exported runtime mode overrides the adapter's hard-set `advanced`, and §6 distinguishes the `--runner-path` adapter checkout from the `--ouroboros-clone` execution seed and points at the field the code actually writes. Known ordering debt noted in place for the v6.82 backlog, deliberately not restructured here: `_auto_sync_release_metadata_if_needed` runs ~87 lines after the `_release_metadata_preflight` gate it would satisfy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| presets.py | ||
| README.md | ||
| swebench_predictions.py | ||
SWE-bench Devtools
These helpers generate official SWE-bench prediction JSONL files for a set of already-prepared local task checkouts.
They do not download datasets, reset repositories, or score benchmark results. Evaluation remains the official SWE-bench harness:
python -m swebench.harness.run_evaluation \
--dataset_name princeton-nlp/SWE-bench_Verified \
--predictions_path /path/to/predictions.jsonl \
--max_workers 1 \
--run_id ouroboros
Prediction rows produced by swebench_predictions.py contain only:
{"instance_id": "...", "model_name_or_path": "...", "model_patch": "..."}
Supported preset aliases in presets.py:
full->princeton-nlp/SWE-benchlite->princeton-nlp/SWE-bench_Liteverified->princeton-nlp/SWE-bench_Verified
Input rows must provide instance_id, a clean git workspace_root or
--workspaces-root, and problem_statement or prompt. If base_commit is
present, the helper refuses to run unless the checkout HEAD matches it.
The helper also writes sidecar artifacts next to the predictions path unless explicit paths are supplied:
<predictions>.ledger.jsonlrecords every requested instance, including invalid input, dirty workspaces, timeouts, non-zero Ouroboros exits, and empty patches.<predictions>.errors.jsonlcontains failure rows for operator debugging.<predictions>.run_manifest.jsonrecords provenance and official eval command shape.
The official predictions JSONL intentionally omits failed/empty-patch rows because the SWE-bench harness expects only prediction records. The ledger is the denominator-preserving Ouroboros audit artifact.