mirror of
https://github.com/razzant/ouroboros.git
synced 2026-08-04 16:19:50 +00:00
v6.81.0: benchmark artefacts must not carry or claim what did not happen
Two provenance fixes in the same class as the release itself, plus the round-11/12 review findings that land in the same files. FIX A — an isolated benchmark container carries only the providers the run declared. `build_isolated_settings` copied EVERY provider credential present in the live settings file regardless of which providers the run's model slots declared, so a run pinned to OpenRouter still received direct ANTHROPIC / OPENAI / Cloud.ru / GigaChat keys. Two consequences: a routing fallback could spend outside the declared bucket while the manifest said otherwise, and the reachable provider set was a function of whatever happened to be in the live file at launch — a pinned seed that pins the code but not the environment is not reproducible. Provider credentials are now gated on the run's declared slots, derived from the routing SSOT (`provider_models.PROVIDER_PREFIXES` / `provider_for_model`, the same registry `llm._resolve_remote_target` routes on), and travel in whole groups so a key never arrives without the endpoint or auth fields it is useless without. Owner/control secrets were never copied and still are not. Ambiguity fails OPEN (carry a spare, disclose it) — never closed. `benchmark_run_manifest` now records `provider_credentials`: which credentials the container actually received, by fingerprint, never by value. FIX B — a task the cost rail truncated says so. `usage_accounting.reserve_- attempt` refuses on a worst-case reservation bound that reached a $6.00 rail at $0.45 of actual spend in the v6.81.0 OSWorld smoke, stopping two of three tasks at 13 and 22 rounds; the artefacts published `status=completed`, `reason_code=official_evaluate` and the string `budget_exhausted` appeared nowhere. `task_result_row` gains an always-present `runtime_outcome` projected by the new shared `runtime_terminal_disclosure`, and every writer holding a runtime task result now publishes it: OSWorld cu_bridge, SWE-bench and ProgramBench success rows (the failure rows already did), the Terminal-Bench in-container summary and disclosure ledger, the GAIA solver, the harness-bench wrapper, and the CL-Bench per-question writer. Reward, `official_eval_status` and adapter-stage `status` are untouched: disclosure ADDED, fact not subtracted. TB's ledger gains a `cost_truncated` bucket — `genuine` asserts a fair shot, which a rail-truncated trial did not get. Review findings folded in: - `_amend_manifest` emitted `output_paths.task_outcome` unconditionally, so the finalized attempt manifest kept pointing at an outcome whose write failed. The previous round fixed the ledger row and left the manifest lying; both sides now follow the same rule. - `ADAPTER_PATCH_MARKERS` keyed two of three detections on bare env-var names, which the unpatched adapter may mention in a comment or a `-e` passthrough list. That false positive OVERSTATES enforcement. Markers are now patch-unique tokens and the uniqueness requirement is recorded beside them. (One marker legitimately covers all three env knobs: they arrive in one loop in one hunk.) - CLB fidelity overstated enforcement on the DEFAULT `--path standard`: `_docker_launcher.submit()` hardcodes `disabled_tools: []` and never imports the patched bridge module, so the evidence is now entrypoint-specific. - `runtime_attested` renamed to `runtime_attestation_available`: it is a tree probe, and a definition existing is not evidence that it ran. - README version badge alt text said 6.80.0 while the URL said 6.81.0, which `version_carrier_desyncs` flags and the advisory preflight blocks on. Bug-pinning tests inverted, and said so in the test docstrings: - `test_dry_run_claims_attestation_only_when_the_patch_is_in_the_execution_- clone` asserted `runtime_attested is True` for a DRY RUN against a clone that merely contained an `_attest_runtime` definition — it demanded the false positive as the contract. - the CLB fidelity fixtures wrote bare env-var names as "the patch", which is precisely the marker weakness above. Docs corrected where the code falsified them: OSWorld METHODOLOGY §7.4 claimed `OUROBOROS_MAX_ROUNDS` plus the timeout were the only per-task caps (the USD rail binds first), §6 now says scoring reads `official_eval_status` / `details.outcome_status` rather than filtering on `status == "completed"` and that `output_paths.task_outcome` may be absent; CLB METHODOLOGY §3 documents when an exported runtime mode overrides the adapter's hard-set `advanced`, and §6 distinguishes the `--runner-path` adapter checkout from the `--ouroboros-clone` execution seed and points at the field the code actually writes. Known ordering debt noted in place for the v6.82 backlog, deliberately not restructured here: `_auto_sync_release_metadata_if_needed` runs ~87 lines after the `_release_metadata_preflight` gate it would satisfy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Anton Razzhigaev
parent
e33b834c38
commit
a5bdf5e010
23 changed files with 1077 additions and 119 deletions
|
|
@ -9,7 +9,7 @@
|
|||
[](https://github.com/razzant/ouroboros/releases)
|
||||
[](https://github.com/razzant/ouroboros/releases)
|
||||
[](https://github.com/razzant/OuroborosHub)
|
||||
[](VERSION)
|
||||
[](VERSION)
|
||||
|
||||
Ouroboros is an open-source, general-purpose AI agent whose identity, durable memory, and history continue across tasks and restarts. It works on external projects, coordinates a live swarm of specialist agents, and can rewrite the implementation it runs on, including its code, architecture, prompts, tools, and dependencies. Reflection can also change how it understands itself without severing that continuity.
|
||||
|
||||
|
|
|
|||
|
|
@ -395,6 +395,30 @@ def model_slot_snapshot(settings_path: pathlib.Path | None = None, *,
|
|||
return slots
|
||||
|
||||
|
||||
def provider_credential_disclosure(settings_path: pathlib.Path | None = None) -> dict[str, Any]:
|
||||
"""Record WHICH provider credentials the described server can reach — by fingerprint.
|
||||
|
||||
``model_slots`` already says which models a run declared; this says which providers it
|
||||
could actually have spent on, which is not the same fact and is the one a routing
|
||||
fallback can falsify. Reads the settings FILE only: an isolated benchmark server loads
|
||||
its provider credentials from the sanitized settings.json (see
|
||||
``server_runner.build_isolated_settings``), not from the launcher's own environment.
|
||||
An absent/unreadable settings path yields ``{"available": False}`` — a stated gap, never
|
||||
a silently empty grant list."""
|
||||
from devtools.benchmarks.common.secrets import isolated_credential_grants
|
||||
|
||||
path = pathlib.Path(settings_path) if settings_path else None
|
||||
if not path or not path.exists():
|
||||
return {"available": False, "reason": "settings_path_absent"}
|
||||
try:
|
||||
loaded = json.loads(path.read_text(encoding="utf-8"))
|
||||
except Exception:
|
||||
return {"available": False, "reason": "settings_unreadable"}
|
||||
if not isinstance(loaded, dict):
|
||||
return {"available": False, "reason": "settings_not_an_object"}
|
||||
return {"available": True, **isolated_credential_grants(loaded)}
|
||||
|
||||
|
||||
class BenchmarkAdmissionRefused(RuntimeError):
|
||||
"""A refused admission gate that CARRIES the manifest describing what it refused.
|
||||
|
||||
|
|
@ -576,6 +600,7 @@ def benchmark_run_manifest(
|
|||
"isolated_data_root": str(meta.get("isolated_data_root") or ""),
|
||||
"output_paths": meta.get("output_paths") or {},
|
||||
"model_slots": model_slot_snapshot(meta_settings_path),
|
||||
"provider_credentials": provider_credential_disclosure(meta_settings_path),
|
||||
"source": source,
|
||||
"seed_gate": gate,
|
||||
"extra": meta.get("extra") or {},
|
||||
|
|
|
|||
|
|
@ -14,6 +14,57 @@ def append_result_index(run_dir: pathlib.Path, row: dict[str, Any]) -> None:
|
|||
fh.write(json.dumps(row, ensure_ascii=False) + "\n")
|
||||
|
||||
|
||||
# Reason codes on which the RUNTIME stopped a task for a reason that is not "the task is
|
||||
# finished". A truncated run and an honest failure are otherwise indistinguishable in a
|
||||
# benchmark artefact, which is how an aggregator records `2/3` with no indication that a
|
||||
# third of the run was cost-truncated. NOT an exhaustive failure taxonomy — only the class
|
||||
# an auditor must never mistake for a capability result. SSOT for the vocabulary:
|
||||
# `ouroboros.outcomes.BEST_EFFORT_REASON_CODES` + `loop._handle_budget_exceeded`.
|
||||
RUNTIME_TRUNCATION_REASON_CODES = frozenset({
|
||||
"budget_exhausted", "max_rounds_exceeded", "task_timeout", "cancelled",
|
||||
"context_exhausted", "provider_unavailable", "llm_api_error", "rate_limited",
|
||||
})
|
||||
|
||||
|
||||
def runtime_terminal_disclosure(task_result: Any) -> dict[str, Any]:
|
||||
"""Project the RUNTIME's OWN terminal reason out of a task-result payload.
|
||||
|
||||
``GET /api/tasks/<id>`` and the CLI's ``--result-json-out`` both carry ``reason_code``,
|
||||
``outcome_axes`` and ``loop_outcome.resource_limit`` (``ouroboros.outcomes``,
|
||||
``task_results.write_task_result``); adapters historically read them only on their OWN
|
||||
failure branch and hard-coded an adapter-stage literal on the success branch. This makes
|
||||
the runtime reason a first-class, always-present field.
|
||||
|
||||
``{"available": False}`` when the writer genuinely has no runtime result — a STATED gap.
|
||||
It never invents a reason, never touches reward, and never demotes an eval status: the
|
||||
evaluation really did run, so this is disclosure ADDED, not fact subtracted."""
|
||||
if not isinstance(task_result, dict) or not task_result:
|
||||
return {"available": False}
|
||||
loop_outcome = task_result.get("loop_outcome")
|
||||
loop_outcome = loop_outcome if isinstance(loop_outcome, dict) else {}
|
||||
resource_limit = task_result.get("resource_limit")
|
||||
if not isinstance(resource_limit, dict):
|
||||
candidate = loop_outcome.get("resource_limit")
|
||||
resource_limit = candidate if isinstance(candidate, dict) else {}
|
||||
axes = task_result.get("outcome_axes")
|
||||
axes = axes if isinstance(axes, dict) else {}
|
||||
execution = axes.get("execution") if isinstance(axes.get("execution"), dict) else {}
|
||||
reason_code = str(task_result.get("reason_code") or "")
|
||||
return {
|
||||
"available": True,
|
||||
"status": str(task_result.get("status") or ""),
|
||||
"reason_code": reason_code,
|
||||
"truncated": reason_code in RUNTIME_TRUNCATION_REASON_CODES,
|
||||
"degraded": bool(task_result.get("degraded") or loop_outcome.get("degraded")),
|
||||
"degraded_reason": str(
|
||||
task_result.get("degraded_reason") or loop_outcome.get("degraded_reason") or ""
|
||||
),
|
||||
"execution_status": str(execution.get("status") or ""),
|
||||
"execution_reason_code": str(execution.get("reason_code") or ""),
|
||||
"resource_limit": resource_limit,
|
||||
}
|
||||
|
||||
|
||||
def task_result_row(
|
||||
*,
|
||||
benchmark: str,
|
||||
|
|
@ -22,7 +73,11 @@ def task_result_row(
|
|||
metadata: dict[str, Any] | None = None,
|
||||
**overrides: Any,
|
||||
) -> dict[str, Any]:
|
||||
"""Create a denominator-preserving per-task result row."""
|
||||
"""Create a denominator-preserving per-task result row.
|
||||
|
||||
Pass ``runtime_result=<task result payload>`` (metadata or keyword) wherever the adapter
|
||||
holds one: ``runtime_outcome`` then discloses why the RUNTIME stopped, independently of
|
||||
the adapter-stage ``reason_code`` this row's ``status`` describes."""
|
||||
meta = dict(metadata or {})
|
||||
for key, value in overrides.items():
|
||||
if value is not None:
|
||||
|
|
@ -34,6 +89,7 @@ def task_result_row(
|
|||
"instance_id": str(instance_id),
|
||||
"status": status,
|
||||
"reason_code": str(meta.get("reason_code") or ""),
|
||||
"runtime_outcome": runtime_terminal_disclosure(meta.get("runtime_result")),
|
||||
"prediction_written": bool(meta.get("prediction_written")),
|
||||
"official_eval_status": str(meta.get("official_eval_status") or "not_run"),
|
||||
"output_paths": meta.get("output_paths") or {},
|
||||
|
|
|
|||
|
|
@ -2,9 +2,11 @@
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
from typing import Any, Iterable
|
||||
|
||||
|
||||
SECRET_KEYS = (
|
||||
|
|
@ -39,5 +41,61 @@ def load_secret_env(path: pathlib.Path | None = None) -> dict[str, str]:
|
|||
return values
|
||||
|
||||
|
||||
def redacted_env_summary(env: dict[str, str]) -> dict[str, bool]:
|
||||
return {key: bool(env.get(key)) for key in SECRET_KEYS}
|
||||
def redacted_env_summary(env: dict[str, str], keys: Iterable[str] | None = None) -> dict[str, bool]:
|
||||
return {key: bool(env.get(key)) for key in (SECRET_KEYS if keys is None else keys)}
|
||||
|
||||
|
||||
def credential_fingerprint(value: Any) -> str:
|
||||
"""Stable, non-reversible identity for a credential value — NEVER the value itself.
|
||||
|
||||
A truncated SHA-256 over a high-entropy API key is not brute-forceable, and it is what
|
||||
lets an auditor answer the only question that matters across two runs: was this the SAME
|
||||
key? Empty/absent values fingerprint to the empty string, not to the hash of ""."""
|
||||
text = str(value or "")
|
||||
if not text:
|
||||
return ""
|
||||
return "sha256:" + hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
|
||||
def isolated_credential_grants(cfg: dict) -> dict:
|
||||
"""Describe, by FINGERPRINT and never by value, which provider credentials a benchmark
|
||||
settings mapping actually carries — and which its declared model slots called for.
|
||||
|
||||
``planned_keys`` is the derivation (``ouroboros.provider_models.provider_credential_plan``,
|
||||
which reads the same prefix->provider registry ``llm._resolve_remote_target`` routes on);
|
||||
``granted`` is the truth about the file. The two are reported separately on purpose: a
|
||||
caller may hand an explicit credential override the slots did not ask for, and an auditor
|
||||
must SEE that rather than infer it. Prevention without evidence is half a fix."""
|
||||
from ouroboros.provider_models import ALL_PROVIDER_CREDENTIAL_KEYS, provider_credential_plan
|
||||
|
||||
settings = cfg or {}
|
||||
plan = provider_credential_plan(settings)
|
||||
present = {
|
||||
key: settings.get(key)
|
||||
for key in sorted(ALL_PROVIDER_CREDENTIAL_KEYS)
|
||||
if str(settings.get(key) or "").strip()
|
||||
}
|
||||
return {
|
||||
"schema": "ouroboros.benchmark.provider_credentials.v1",
|
||||
"declared_model_slots": plan["declared_model_slots"],
|
||||
"providers": plan["providers"],
|
||||
"planned_keys": plan["planned_keys"],
|
||||
"fail_open": plan["fail_open"],
|
||||
"granted": credential_disclosure(present, sorted(present)),
|
||||
}
|
||||
|
||||
|
||||
def credential_disclosure(env: dict[str, Any], keys: Iterable[str] | None = None) -> dict[str, dict[str, Any]]:
|
||||
"""Fingerprint-level extension of ``redacted_env_summary``: ``{key: {present, fingerprint}}``.
|
||||
|
||||
Same mechanism, one rung more auditable — a bare ``True`` cannot distinguish "the run
|
||||
reached the declared bucket" from "the run reached some other key that happened to be in
|
||||
the live settings file"."""
|
||||
names = sorted(env) if keys is None else list(keys)
|
||||
return {
|
||||
str(key): {
|
||||
"present": bool(str(env.get(key) or "").strip()),
|
||||
"fingerprint": credential_fingerprint(env.get(key)),
|
||||
}
|
||||
for key in names
|
||||
}
|
||||
|
|
|
|||
|
|
@ -31,6 +31,8 @@ if __package__ in {None, ""}:
|
|||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[3]))
|
||||
|
||||
from devtools.benchmarks.common.manifests import runtime_attestation
|
||||
from devtools.benchmarks.common.secrets import isolated_credential_grants # noqa: F401 (re-export)
|
||||
from ouroboros.provider_models import ALL_PROVIDER_CREDENTIAL_KEYS, provider_credential_plan
|
||||
from ouroboros.platform_layer import (
|
||||
kill_pid_tree,
|
||||
subprocess_new_group_kwargs,
|
||||
|
|
@ -63,16 +65,14 @@ STALE_INHERITED_ENV_KEYS = (
|
|||
# credentials/endpoints, model slots, effort, and budget. Owner/control secrets and knobs
|
||||
# (GITHUB_TOKEN, OUROBOROS_NETWORK_PASSWORD, transport/skill secrets, owner chat ids, etc.)
|
||||
# are NEVER copied — the isolated data root is readable by untrusted benchmark tasks.
|
||||
# Model-slot / effort / local-model / gigachat-provider key families (all are model or
|
||||
# provider config — safe to copy). GIGACHAT_* covers its credentials+endpoint+scope.
|
||||
_ISO_SETTINGS_ALLOW_PREFIX = ("OUROBOROS_MODEL", "OUROBOROS_EFFORT", "LOCAL_MODEL_", "GIGACHAT_")
|
||||
# EXPLICIT provider creds/endpoints + review/budget keys. Deliberately NOT a `*_API_KEY`
|
||||
# pattern: a custom skill secret could be named `<x>_API_KEY` and must NOT be copied.
|
||||
# Model-slot / effort / local-model key families (all are model config — safe to copy).
|
||||
# NOTE the absent `GIGACHAT_` prefix: every provider credential family, GigaChat's included,
|
||||
# is gated on the run's DECLARED slots below instead of riding an unconditional prefix.
|
||||
_ISO_SETTINGS_ALLOW_PREFIX = ("OUROBOROS_MODEL", "OUROBOROS_EFFORT", "LOCAL_MODEL_")
|
||||
# NON-credential review/budget/model keys. Provider credentials are deliberately NOT here —
|
||||
# see _grant_provider_credentials. Deliberately NOT a `*_API_KEY` pattern either: a custom
|
||||
# skill secret could be named `<x>_API_KEY` and must NOT be copied.
|
||||
_ISO_SETTINGS_ALLOW_EXACT = frozenset({
|
||||
"OPENROUTER_API_KEY", "OPENAI_API_KEY", "OPENAI_BASE_URL",
|
||||
"OPENAI_COMPATIBLE_API_KEY", "OPENAI_COMPATIBLE_BASE_URL",
|
||||
"CLOUDRU_FOUNDATION_MODELS_API_KEY", "CLOUDRU_FOUNDATION_MODELS_BASE_URL",
|
||||
"ANTHROPIC_API_KEY",
|
||||
"OUROBOROS_WEBSEARCH_MODEL", "OUROBOROS_REVIEW_MODELS",
|
||||
"OUROBOROS_SCOPE_REVIEW_MODELS", "OUROBOROS_SCOPE_REVIEW_MODEL",
|
||||
# Review policy knobs (non-secret): must propagate so settings.json's task-acceptance
|
||||
|
|
@ -110,17 +110,38 @@ def _is_secret_env_key(key: str) -> bool:
|
|||
|
||||
|
||||
def build_isolated_settings(live_cfg: dict, **overrides) -> dict:
|
||||
"""Build an isolated benchmark settings.json from live settings, copying ONLY the
|
||||
EXPLICIT provider/model/budget allowlist above (never owner/control secrets like
|
||||
GITHUB_TOKEN / OUROBOROS_NETWORK_PASSWORD / transport / skill secrets / owner knobs),
|
||||
then applying explicit isolated overrides. The isolated data root is reachable by
|
||||
untrusted benchmark tasks, so this is the hermetic 'provider keys + model slots only' seed."""
|
||||
"""Build an isolated benchmark settings.json from live settings: copy the non-credential
|
||||
model/effort/budget/review allowlist above, apply the explicit isolated overrides, and
|
||||
then grant ONLY the provider credentials the resulting run's DECLARED model slots need.
|
||||
|
||||
Owner/control secrets (GITHUB_TOKEN, OUROBOROS_NETWORK_PASSWORD, transport/skill secrets,
|
||||
owner knobs) were never copied and still are not. What changes here is narrower and was
|
||||
the real defect: the copied provider set used to be a function of whatever happened to be
|
||||
in the live settings file at launch, so a run pinned to OpenRouter still received direct
|
||||
ANTHROPIC_API_KEY / OPENAI_API_KEY / Cloud.ru / GigaChat credentials. A routing fallback
|
||||
could then spend outside the declared bucket while the manifest said otherwise, and two
|
||||
nominally identical runs could reach different providers invisibly — a pinned seed that
|
||||
pins the code but not the environment is not reproducible.
|
||||
|
||||
Credentials travel in whole GROUPS (``PROVIDER_CREDENTIAL_GROUPS``), so a key never
|
||||
arrives without the endpoint/auth fields it is useless without (GigaChat
|
||||
CREDENTIALS+PASSWORD+endpoint+scope, Cloud.ru key+base_url). An explicit override always
|
||||
wins over the derived grant. Use ``isolated_credential_grants`` on the RESULT to record
|
||||
what was granted."""
|
||||
out: dict = {}
|
||||
for key, value in (live_cfg or {}).items():
|
||||
ks = str(key)
|
||||
if ks in ALL_PROVIDER_CREDENTIAL_KEYS:
|
||||
continue # gated below on the declared slots, never copied wholesale
|
||||
if ks in _ISO_SETTINGS_ALLOW_EXACT or ks.startswith(_ISO_SETTINGS_ALLOW_PREFIX):
|
||||
out[ks] = value
|
||||
out.update(overrides)
|
||||
for key in provider_credential_plan(out)["planned_keys"]:
|
||||
if key in (overrides or {}):
|
||||
continue
|
||||
value = (live_cfg or {}).get(key)
|
||||
if value not in (None, ""):
|
||||
out[key] = value
|
||||
return out
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -76,7 +76,7 @@ the agent reaches data only through counted QUERY actions).
|
|||
|
||||
| Knob | Value | Why |
|
||||
|---|---|---|
|
||||
| `OUROBOROS_RUNTIME_MODE` | `advanced` | what the adapter hard-sets for the isolated server (full live-agent runtime in a sandboxed throwaway clone). The template must keep `advanced`. |
|
||||
| `OUROBOROS_RUNTIME_MODE` | `advanced` by default; **overridable on a patched adapter checkout** | the pinned adapter hard-sets `advanced` for the isolated server (full live-agent runtime in a sandboxed throwaway clone), and the template must keep `advanced`. **Exception (v6.74.5+):** when `clb_env_campaign_overrides.v6745.patch` is applied to the `--runner-path` adapter checkout, its `_overrides()` loop lets an exported `OUROBOROS_RUNTIME_MODE` WIN over the hard-set `advanced` — the same single hunk that forwards `OUROBOROS_REVIEW_ENFORCEMENT` and `OUROBOROS_SAFETY_MODE`, so all three arrive together or not at all. `run_clb.py` forwards the template's value, so a template selecting e.g. `pro` really runs `pro` on a patched checkout. `fidelity.enforced_via_runner_interface.OUROBOROS_RUNTIME_MODE` states which of the two happened for the run at hand; do not assume `advanced` from this table. |
|
||||
| memory | `memory_mode="shared"` on every solve task; ONE persistent server per stateful rollout; fresh server per stateless instance | memory persistence across the strictly sequential stream IS the measured quantity |
|
||||
| continuity | conversation reset at question boundary; within-question resume | memory-based continual learning, deliberately NOT whole-rollout ICL (divergence from the Claude Code reference, disclosed) |
|
||||
| evolution | OFF for the headline (`stateful_noevo`) | the CC reference has no self-modification; evolution is a separately-labeled condition (`stateful_evo`, `--evolution`) |
|
||||
|
|
@ -235,36 +235,62 @@ channel the pinned external adapter actually honors: `--system-params`
|
|||
(model, `max_workers`, evolution, resume, timeouts) and child env
|
||||
(`OUROBOROS_EFFORT_TASK`, `OUROBOROS_OR_PROVIDER`, `OUROBOROS_TOTAL_BUDGET`).
|
||||
|
||||
**Two different checkouts, do not confuse them.** `--ouroboros-clone` is the
|
||||
EXECUTION SEED (the Ouroboros checkout the adapter boots its agent servers
|
||||
from; the seed-provenance gate binds to it). `--runner-path` is the external
|
||||
continual-learning-bench ADAPTER checkout, and `<runner>/src/systems/ouroboros`
|
||||
is where the operator patches below are applied and probed. Patching the seed
|
||||
instead of the adapter checkout does nothing — the report will honestly say
|
||||
"unpatched".
|
||||
|
||||
Three knobs have **no forward channel in the pinned adapter (56764d6)** and
|
||||
depend on tracked operator patches being applied in the EXECUTION CLONE:
|
||||
depend on tracked operator patches being applied in the ADAPTER CHECKOUT:
|
||||
|
||||
- `OUROBOROS_SAFETY_MODE=light` — effective on the host engine path (env
|
||||
inheritance); the docker engine forwards only an explicit `-e` list.
|
||||
Forwarded by `clb_env_campaign_overrides.v6745.patch`, which lets an
|
||||
exported value win over the engine's parity defaults in `_overrides`.
|
||||
- `OUROBOROS_REVIEW_ENFORCEMENT=blocking` — same channel, same gap, same
|
||||
patch.
|
||||
patch, **same hunk** (one `for _k in (...)` loop, which is why a single
|
||||
marker legitimately covers all three env knobs including
|
||||
`OUROBOROS_RUNTIME_MODE`).
|
||||
- `CLBENCH_SOLVE_DISABLED_TOOLS` (incl. `claude_code_edit`) — the pinned
|
||||
bridge hardcodes `DISABLED_TOOLS = []`. Read from env by
|
||||
`clb_disabled_tools_env.v6745.patch`, so every declared tool reaches the
|
||||
task contract.
|
||||
task contract. **ENTRYPOINT-SPECIFIC:** that patch touches
|
||||
`run_clbench_bridge_agent.py`, which ONLY `--path bridge` executes. On the
|
||||
DEFAULT `--path standard` the run goes `run_benchmark.py` → `system.py` →
|
||||
`_docker_launcher.submit()`, which hardcodes `"disabled_tools": []` and never
|
||||
reads the env list — so on the standard path this knob is a declared-only gap
|
||||
even against a fully patched checkout, and the report says exactly that.
|
||||
|
||||
**Whether these are in force is a fact about the clone, not about the flags.**
|
||||
The launcher probes the execution clone for each patch's marker
|
||||
**Whether these are in force is a fact about the checkout, not about the
|
||||
flags.** The launcher probes the adapter checkout for each patch's markers
|
||||
(`adapter_patch_probe`) and records the verdict per knob: enforced knobs land
|
||||
under `fidelity.enforced_via_operator_patch`, unenforced ones under
|
||||
`fidelity.declared_only_pinned_adapter_gap` with a stderr warning, and the raw
|
||||
probe under `fidelity.adapter_operator_patches`. The same probe decides
|
||||
`extra.runtime_attestation_path` / `extra.runtime_attested`: on the docker path
|
||||
the attestation exists only if `clb_docker_runtime_attestation.v6746.patch` is
|
||||
applied, and a run without it is recorded as UNATTESTED rather than claiming a
|
||||
check that never ran.
|
||||
`fidelity.declared_only_pinned_adapter_gap` with a stderr warning.
|
||||
`fidelity.adapter_operator_patches` holds the applied-bool map only; the FULL
|
||||
probe — including `adapter_path` (which tree was read) and `evidence` — is
|
||||
written to `extra.adapter_operator_patches`. The markers are patch-unique
|
||||
tokens (a comment tag, a `def` line, the exact expression the patch
|
||||
introduces), never bare env-var names, because the pinned adapter may mention
|
||||
an env name without the patch being applied and the resulting false positive
|
||||
would OVERSTATE enforcement.
|
||||
|
||||
On an **unpatched** clone, docker-path runs execute with safety `full`,
|
||||
advisory review enforcement, and without the `claude_code_edit` exclusion; any
|
||||
published number must say so. On a **patched** clone all three are applied, and
|
||||
the manifest says that instead — reading the gap off the flags understated runs
|
||||
that were in fact stricter than declared.
|
||||
The same probe decides `extra.runtime_attestation_path` /
|
||||
`extra.runtime_attestation_available`: on the docker path the attestation hook
|
||||
exists only if `clb_docker_runtime_attestation.v6746.patch` is applied, and a
|
||||
run without it is recorded as UNATTESTED rather than claiming a check that
|
||||
never ran. The field is named `..._available` on purpose — it is a TREE probe,
|
||||
so it says the hook is present in the checkout that will execute, not that
|
||||
attestation has already run.
|
||||
|
||||
On an **unpatched** adapter checkout, docker-path runs execute with safety
|
||||
`full`, advisory review enforcement, and without the `claude_code_edit`
|
||||
exclusion; any published number must say so. On a **patched** checkout the two
|
||||
env knobs are applied, and the manifest says that instead — reading the gap off
|
||||
the flags understated runs that were in fact stricter than declared. The
|
||||
disabled-tools knob additionally requires `--path bridge` (above).
|
||||
|
||||
## 7. Honest limits
|
||||
|
||||
|
|
|
|||
|
|
@ -155,3 +155,26 @@ the external checkout root (a clean clone at `549998d` plus the patches above).
|
|||
`runtime_attestation` lands. On an older clone the patch fails closed with an
|
||||
explicit ImportError message naming the requirement, so do not apply it to a
|
||||
pre-v6.75.0 bench clone.
|
||||
|
||||
## Addendum (v6.81.0, 2026-07-26)
|
||||
|
||||
11. `clb_multi_instance_outcomes.v6746.patch` was **regenerated in place** (no new
|
||||
file, same apply order) to publish the RUNTIME's own terminal reason in each
|
||||
`q###/task_outcome.json`. `_write_instance_outcome` gains a keyword-only
|
||||
`runtime_result` and a `runtime_outcome` column projected by a new local
|
||||
`_runtime_terminal_disclosure` helper; the agent-turn call site passes `final`,
|
||||
the co-resolved call site passes nothing (it had no turn, so the row honestly
|
||||
reads `{"available": false}`). WHY: `ouroboros_status` alone cannot carry it —
|
||||
a question the per-task USD reservation rail truncated reports status
|
||||
`failed` exactly like a genuinely wrong answer, so an aggregator could not tell
|
||||
a cost-truncated question from a capability failure. Reward, success and
|
||||
`cost_usd` are untouched; this ADDS disclosure.
|
||||
The helper is a deliberate LOCAL mirror of
|
||||
`devtools.benchmarks.common.result_index.runtime_terminal_disclosure` (the SSOT
|
||||
for the vocabulary) for the same reason `_atomic_write_json` is local: this
|
||||
module lives in the external checkout and only reaches the Ouroboros clone
|
||||
through `_launcher`'s call-time `sys.path` insert. Keep the two in sync.
|
||||
Launcher side (`run_clb.py`, our own code, no patch file):
|
||||
`collect_results` carries `runtime_outcome` into `results.json` /
|
||||
`result_index.jsonl`, defaulting to `{"available": false}` for rows written
|
||||
before this patch — a stated gap, never a silent absence.
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
import time
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
@@ -439,6 +440,85 @@
|
||||
@@ -439,6 +440,122 @@
|
||||
return _poll_terminal(base, task_id, args.task_timeout_sec)
|
||||
|
||||
|
||||
|
|
@ -48,12 +48,18 @@
|
|||
+
|
||||
+def _write_instance_outcome(run_dir: Path, domain: str, instance_index: int,
|
||||
+ outcome: Optional[dict], *, ouroboros_status: str,
|
||||
+ cost_usd) -> None:
|
||||
+ cost_usd, runtime_result: Optional[dict] = None) -> None:
|
||||
+ """Write ONE q###/task_outcome.json row (same schema run_clb.collect_results reads).
|
||||
+
|
||||
+ ``ouroboros_status`` is the honest provenance of the row: a real agent-turn row
|
||||
+ carries the Ouroboros task status, a co-resolved row carries
|
||||
+ ``auto_resolved_no_agent_turn``.
|
||||
+
|
||||
+ ``runtime_result`` is the RUNTIME's own terminal payload, from which
|
||||
+ ``runtime_outcome`` is projected. ``ouroboros_status`` alone cannot carry it: a task
|
||||
+ the per-task USD reservation rail truncated reports status "failed" exactly like an
|
||||
+ honest failure, so an aggregator could not tell a cost-truncated question from a
|
||||
+ genuinely wrong one. Reward and success are untouched — this ADDS disclosure.
|
||||
+ """
|
||||
+ _atomic_write_json(run_dir / "task_outcome.json", {
|
||||
+ "domain": domain, "instance_index": instance_index,
|
||||
|
|
@ -61,9 +67,40 @@
|
|||
+ "success": (outcome or {}).get("success"),
|
||||
+ "ouroboros_status": ouroboros_status,
|
||||
+ "cost_usd": cost_usd,
|
||||
+ "runtime_outcome": _runtime_terminal_disclosure(runtime_result),
|
||||
+ })
|
||||
+
|
||||
+
|
||||
+def _runtime_terminal_disclosure(task_result: Optional[dict]) -> dict:
|
||||
+ """Local mirror of devtools.benchmarks.common.result_index.runtime_terminal_disclosure.
|
||||
+
|
||||
+ A LOCAL seam, deliberately: this module lives in the external CL-Bench checkout and
|
||||
+ only reaches the Ouroboros clone through `_launcher`'s call-time sys.path insert, so a
|
||||
+ module-level import of the shared helper would break the standard entrypoint. Keep the
|
||||
+ two in sync — the shared helper is the SSOT for the vocabulary.
|
||||
+ """
|
||||
+ if not isinstance(task_result, dict) or not task_result:
|
||||
+ return {"available": False}
|
||||
+ loop_outcome = task_result.get("loop_outcome")
|
||||
+ loop_outcome = loop_outcome if isinstance(loop_outcome, dict) else {}
|
||||
+ resource_limit = task_result.get("resource_limit")
|
||||
+ if not isinstance(resource_limit, dict):
|
||||
+ candidate = loop_outcome.get("resource_limit")
|
||||
+ resource_limit = candidate if isinstance(candidate, dict) else {}
|
||||
+ reason_code = str(task_result.get("reason_code") or "")
|
||||
+ return {
|
||||
+ "available": True,
|
||||
+ "status": str(task_result.get("status") or ""),
|
||||
+ "reason_code": reason_code,
|
||||
+ "truncated": reason_code in (
|
||||
+ "budget_exhausted", "max_rounds_exceeded", "task_timeout",
|
||||
+ "cancelled", "context_exhausted", "provider_unavailable",
|
||||
+ "llm_api_error", "rate_limited"),
|
||||
+ "degraded": bool(task_result.get("degraded") or loop_outcome.get("degraded")),
|
||||
+ "resource_limit": resource_limit,
|
||||
+ }
|
||||
+
|
||||
+
|
||||
+def _refresh_instance_reward(run_dir: Path, outcome: dict) -> bool:
|
||||
+ """Fill in a reward that only materialised later, KEEPING the row's provenance.
|
||||
+
|
||||
|
|
@ -94,7 +131,7 @@
|
|||
def run_stateful(args, out_root: Path, *, evolution: bool) -> list:
|
||||
"""STATEFUL ROLLOUT (CC-comparable): ONE persistent Ouroboros server + ONE CONTINUOUS task across all
|
||||
instances (the schema_drift migration fires mid-sequence, as CC's rollout experiences it). One agent
|
||||
@@ -471,21 +551,61 @@
|
||||
@@ -471,21 +588,62 @@
|
||||
final = _submit_and_poll(eng, prompt, args)
|
||||
(rd / "ouroboros_task_final.json").write_text(json.dumps(final, indent=2), encoding="utf-8")
|
||||
oc = _api(url, "GET", "/_outcome", timeout=30)
|
||||
|
|
@ -122,7 +159,8 @@
|
|||
- }, indent=2), encoding="utf-8")
|
||||
+ _write_instance_outcome(rd, args.domain, idx, this,
|
||||
+ ouroboros_status=str(final.get("status") or ""),
|
||||
+ cost_usd=final.get("cost_usd"))
|
||||
+ cost_usd=final.get("cost_usd"),
|
||||
+ runtime_result=final)
|
||||
+ # Co-resolved instances the agent never had a turn on: recorded honestly
|
||||
+ # with ouroboros_status="auto_resolved_no_agent_turn" and NO agent-turn
|
||||
+ # artefacts (no prompt.txt / ouroboros_task_final.json / absorb.json),
|
||||
|
|
|
|||
|
|
@ -156,38 +156,70 @@ def check_runner(runner: pathlib.Path) -> dict:
|
|||
|
||||
|
||||
# Tracked operator patches whose EFFECT this launcher's provenance block describes, each keyed
|
||||
# to a marker that only the applied patch puts into the execution clone. See
|
||||
# operator_patches/README.md; a patch renamed there must be renamed here in the same commit.
|
||||
ADAPTER_PATCH_MARKERS: dict[str, tuple[str, str]] = {
|
||||
"clb_docker_runtime_attestation.v6746": ("_docker_launcher.py", "_attest_runtime"),
|
||||
"clb_env_campaign_overrides.v6745": ("_docker_launcher.py", "OUROBOROS_SAFETY_MODE"),
|
||||
"clb_disabled_tools_env.v6745": ("run_clbench_bridge_agent.py", "CLBENCH_SOLVE_DISABLED_TOOLS"),
|
||||
# to markers that only the applied patch puts into the ADAPTER checkout (`--runner-path`, the
|
||||
# external continual-learning-bench tree) — NOT the `--ouroboros-clone` execution seed the
|
||||
# seed gate binds to; the two are different trees and an operator who patches the wrong one
|
||||
# gets an honest "unpatched" report. See operator_patches/README.md; a patch renamed there
|
||||
# must be renamed here in the same commit.
|
||||
#
|
||||
# MARKER-UNIQUENESS REQUIREMENT — do not weaken this. Every token below must be something ONLY
|
||||
# the applied patch writes: its own comment tag, a `def` line, the exact expression it
|
||||
# introduces. A bare env-var NAME is NOT such a token. The pinned adapter may legitimately
|
||||
# mention `OUROBOROS_SAFETY_MODE` or `CLBENCH_SOLVE_DISABLED_TOOLS` in a comment or a `-e`
|
||||
# passthrough list without the patch being applied, and the resulting false positive errs in
|
||||
# the DANGEROUS direction: the manifest reports `enforced_via_operator_patch` for a run that
|
||||
# actually executed unenforced. Every token in the tuple must match, never just one.
|
||||
ADAPTER_PATCH_MARKERS: dict[str, tuple[str, tuple[str, ...]]] = {
|
||||
"clb_docker_runtime_attestation.v6746": ("_docker_launcher.py", (
|
||||
"def _attest_runtime(", "self.runtime_attestation")),
|
||||
"clb_env_campaign_overrides.v6745": ("_docker_launcher.py", (
|
||||
"Operator env overrides (campaign knobs)", "_v = os.environ.get(_k)")),
|
||||
"clb_disabled_tools_env.v6745": ("run_clbench_bridge_agent.py", (
|
||||
"Operator patch 2026-07-23: honor CLBENCH_SOLVE_DISABLED_TOOLS",
|
||||
'_os.environ.get("CLBENCH_SOLVE_DISABLED_TOOLS"')),
|
||||
}
|
||||
|
||||
# Which adapter module each entrypoint actually executes. The bridge entrypoint runs
|
||||
# `run_clbench_bridge_agent.py`; `--path standard` (the DEFAULT) goes through
|
||||
# `run_benchmark.py` -> `system.py` -> `_docker_launcher.submit()`, which hardcodes
|
||||
# `"disabled_tools": []`. A marker in a module the chosen entrypoint never imports says
|
||||
# nothing about the run.
|
||||
_DISABLED_TOOLS_FORWARDING_ENTRYPOINTS = ("bridge",)
|
||||
|
||||
|
||||
def adapter_patch_probe(runner: pathlib.Path) -> dict:
|
||||
"""Which tracked operator patches are ACTUALLY present in the execution clone.
|
||||
"""Which tracked operator patches are ACTUALLY present in the adapter checkout.
|
||||
|
||||
The provenance and fidelity blocks used to describe the patched adapter from CONSTANTS —
|
||||
"``--docker`` was passed, therefore the attestation hook ran; the adapter is the pinned
|
||||
commit, therefore these three knobs are dropped". Both statements are about a tree, and
|
||||
the tree is the thing that was never consulted: the clone is patched (or not) by an
|
||||
operator, out of band, per run. The attestation claim was false on an unpatched clone and
|
||||
the fidelity gap was false on a patched one — the same defect this campaign already fixed
|
||||
where a manifest named a model that never ran.
|
||||
"``--docker`` was passed, therefore the attestation hook is there; the adapter is the
|
||||
pinned commit, therefore these three knobs are dropped". Both statements are about a tree,
|
||||
and the tree is the thing that was never consulted: the adapter subtree is patched (or
|
||||
not) by an operator, out of band, per run.
|
||||
|
||||
So the record is DERIVED, by looking for a marker the applied patch leaves behind. An
|
||||
So the record is DERIVED, by looking for markers the applied patch leaves behind. An
|
||||
unreadable or absent adapter file yields ``False``: absent provenance recorded honestly is
|
||||
fine, a claimed one that did not happen is not.
|
||||
|
||||
NOTE THE SUBJECT. This probes the ``--runner-path`` ADAPTER subtree (the external
|
||||
continual-learning-bench checkout, ``<runner>/src/systems/ouroboros``), NOT the
|
||||
``--ouroboros-clone`` execution seed the seed gate binds to. They are different trees;
|
||||
``adapter_path`` in the returned probe names the one that was read.
|
||||
|
||||
THIS PROBE IS STATIC by design (owner-approved goal: a tree probe). It answers "is the
|
||||
patch in this tree", never "did the effect happen at runtime" — which is why the field it
|
||||
feeds is named ``runtime_attestation_available``, not ``runtime_attested``.
|
||||
"""
|
||||
adapter = pathlib.Path(runner).expanduser().resolve(strict=False) / ADAPTER_REL
|
||||
probe: dict = {"adapter_path": str(adapter), "patches": {}}
|
||||
for name, (filename, marker) in ADAPTER_PATCH_MARKERS.items():
|
||||
try:
|
||||
applied = marker in (adapter / filename).read_text(encoding="utf-8")
|
||||
except OSError:
|
||||
applied = False
|
||||
probe["patches"][name] = applied
|
||||
probe: dict = {"adapter_path": str(adapter), "evidence": "static_source_scan", "patches": {}}
|
||||
sources: dict[str, str] = {}
|
||||
for name, (filename, markers) in ADAPTER_PATCH_MARKERS.items():
|
||||
if filename not in sources:
|
||||
try:
|
||||
sources[filename] = (adapter / filename).read_text(encoding="utf-8")
|
||||
except OSError:
|
||||
sources[filename] = ""
|
||||
text = sources[filename]
|
||||
probe["patches"][name] = bool(text) and all(marker in text for marker in markers)
|
||||
return probe
|
||||
|
||||
|
||||
|
|
@ -240,33 +272,39 @@ def _fidelity_report(settings: dict, args: argparse.Namespace,
|
|||
"""
|
||||
disabled = list(settings.get("CLBENCH_SOLVE_DISABLED_TOOLS") or [])
|
||||
applied = dict((patch_probe or {}).get("patches") or {})
|
||||
# ONE marker for the three env knobs is CORRECT, not a shortcut: the campaign-override
|
||||
# patch adds OUROBOROS_RUNTIME_MODE, OUROBOROS_REVIEW_ENFORCEMENT and OUROBOROS_SAFETY_MODE
|
||||
# in a single loop in a single hunk, so they arrive together or not at all.
|
||||
# The host engine boots a real `IsolatedServer`, which inherits the launcher's env, so the
|
||||
# two env knobs are enforced there whether or not the docker-only patch is applied.
|
||||
# env knobs are enforced there whether or not the docker-only patch is applied.
|
||||
env_knobs_forwarded = (not args.docker) or bool(applied.get("clb_env_campaign_overrides.v6745"))
|
||||
tools_forwarded = bool(applied.get("clb_disabled_tools_env.v6745"))
|
||||
# ENTRYPOINT-SPECIFIC. The patch lives in `run_clbench_bridge_agent.py`, which ONLY the
|
||||
# bridge entrypoint executes. On the DEFAULT `--path standard` the run goes
|
||||
# run_benchmark.py -> system.py -> `_docker_launcher.submit()`, which hardcodes
|
||||
# `"disabled_tools": []`; `system.py` never mentions the list at all. Keying purely on the
|
||||
# marker therefore filed the knob as ENFORCED on the default path against a patched clone
|
||||
# and claimed every declared tool — claude_code_edit included — was excluded from the task
|
||||
# contract, for a run whose executed path excluded nothing. Overstating enforcement is the
|
||||
# dangerous direction, and it is exactly the class this fix exists to remove.
|
||||
tools_forwarded = (bool(applied.get("clb_disabled_tools_env.v6745"))
|
||||
and args.path in _DISABLED_TOOLS_FORWARDING_ENTRYPOINTS)
|
||||
enforced: dict = {}
|
||||
gap: dict = {}
|
||||
(enforced if env_knobs_forwarded else gap)["OUROBOROS_SAFETY_MODE"] = {
|
||||
"declared": settings.get("OUROBOROS_SAFETY_MODE"),
|
||||
"status": ("exported in env and applied in-container by operator patch "
|
||||
_env_status = ("exported in env and applied in-container by operator patch "
|
||||
"clb_env_campaign_overrides.v6745 (docker engine `_overrides`)"
|
||||
if args.docker else
|
||||
"exported in env — effective on the HOST engine path "
|
||||
"(IsolatedServer inherits env)")
|
||||
if env_knobs_forwarded else
|
||||
("exported in env, but the docker engine forwards only an explicit -e list and DROPS "
|
||||
"it: operator patch clb_env_campaign_overrides.v6745 is NOT applied in this clone"),
|
||||
"(IsolatedServer inherits env)") if env_knobs_forwarded else (
|
||||
"exported in env, but the docker engine forwards only an explicit -e list and DROPS "
|
||||
"it: operator patch clb_env_campaign_overrides.v6745 is NOT applied in this adapter "
|
||||
"checkout")
|
||||
(enforced if env_knobs_forwarded else gap)["OUROBOROS_SAFETY_MODE"] = {
|
||||
"declared": settings.get("OUROBOROS_SAFETY_MODE"),
|
||||
"status": _env_status,
|
||||
}
|
||||
(enforced if env_knobs_forwarded else gap)["OUROBOROS_REVIEW_ENFORCEMENT"] = {
|
||||
"declared": settings.get("OUROBOROS_REVIEW_ENFORCEMENT"),
|
||||
"status": ("exported in env and applied in-container by operator patch "
|
||||
"clb_env_campaign_overrides.v6745 (docker engine `_overrides`)"
|
||||
if args.docker else
|
||||
"exported in env — effective on the HOST engine path "
|
||||
"(IsolatedServer inherits env)")
|
||||
if env_knobs_forwarded else
|
||||
("exported in env, but the docker engine forwards only an explicit -e list and DROPS "
|
||||
"it: operator patch clb_env_campaign_overrides.v6745 is NOT applied in this clone"),
|
||||
"status": _env_status,
|
||||
}
|
||||
(enforced if tools_forwarded else gap)["CLBENCH_SOLVE_DISABLED_TOOLS"] = {
|
||||
"declared": disabled,
|
||||
|
|
@ -275,12 +313,21 @@ def _fidelity_report(settings: dict, args: argparse.Namespace,
|
|||
"declared tool — including claude_code_edit — is excluded from the task "
|
||||
"contract")
|
||||
if tools_forwarded else
|
||||
("exported in env as a comma list; operator patch clb_disabled_tools_env.v6745 is NOT "
|
||||
"applied in this clone, so the pinned adapter submits tasks WITHOUT disabled_tools "
|
||||
"(no claude_code_edit exclusion)"),
|
||||
((f"exported in env as a comma list, but this run uses --path {args.path} "
|
||||
"(run_benchmark.py -> system.py -> _docker_launcher.submit, which hardcodes "
|
||||
"disabled_tools=[]); operator patch clb_disabled_tools_env.v6745 patches "
|
||||
"run_clbench_bridge_agent.py, which that entrypoint never executes")
|
||||
if applied.get("clb_disabled_tools_env.v6745") else
|
||||
("exported in env as a comma list; operator patch clb_disabled_tools_env.v6745 is NOT "
|
||||
"applied in this clone, so the pinned adapter submits tasks WITHOUT disabled_tools "
|
||||
"(no claude_code_edit exclusion)")),
|
||||
}
|
||||
return {
|
||||
# STATIC availability of each tracked patch in the `--runner-path` ADAPTER checkout —
|
||||
# NOT a claim that any of them ran. The full probe (adapter_path, evidence) is recorded
|
||||
# separately at `extra.adapter_operator_patches`; METHODOLOGY §6 names both.
|
||||
"adapter_operator_patches": applied,
|
||||
"adapter_operator_patches_evidence": "static_source_scan",
|
||||
"enforced_via_operator_patch": enforced,
|
||||
"declared_only_pinned_adapter_gap": gap,
|
||||
"enforced_via_runner_interface": {
|
||||
|
|
@ -466,6 +513,7 @@ def collect_results(run_dir: pathlib.Path) -> dict:
|
|||
"reward": None,
|
||||
"success": None,
|
||||
"ouroboros_status": "external_runner_sidecar_only",
|
||||
"runtime_outcome": {"available": False},
|
||||
"cost_usd": None,
|
||||
})
|
||||
for cond_name in sorted(set(found_condition_names) | set(expected_conditions)):
|
||||
|
|
@ -502,6 +550,14 @@ def collect_results(run_dir: pathlib.Path) -> dict:
|
|||
"success": row.get("success"),
|
||||
"ouroboros_status": str(row.get("ouroboros_status") or ""),
|
||||
"cost_usd": row.get("cost_usd"),
|
||||
# The RUNTIME's own terminal reason, written by operator patch
|
||||
# clb_multi_instance_outcomes.v6746. `ouroboros_status` cannot carry
|
||||
# it: a question the per-task USD rail truncated says "failed" exactly
|
||||
# like a genuinely wrong answer. Rows written before that patch carry
|
||||
# no such key, so the gap is STATED rather than assumed absent.
|
||||
"runtime_outcome": (row.get("runtime_outcome")
|
||||
if isinstance(row.get("runtime_outcome"), dict)
|
||||
else {"available": False}),
|
||||
})
|
||||
for expected in expected_ids:
|
||||
domain, _, qid = expected.partition(":")
|
||||
|
|
@ -516,6 +572,7 @@ def collect_results(run_dir: pathlib.Path) -> dict:
|
|||
"success": None,
|
||||
"ouroboros_status": "missing_outcome",
|
||||
"cost_usd": None,
|
||||
"runtime_outcome": {"available": False},
|
||||
})
|
||||
scored = [r for r in rewards.values() if r is not None]
|
||||
conditions[cond_name] = {
|
||||
|
|
@ -713,7 +770,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
else "launcher_checkout_dry_run"),
|
||||
"execution_clone": str(execution_clone) if execution_clone is not None else "",
|
||||
# `runtime_attestation_path` is NOT claimed here. On the docker path it is a
|
||||
# fact about the execution clone (is the operator patch applied?), and a
|
||||
# fact about the ADAPTER checkout (is the operator patch applied?), and a
|
||||
# filesystem probe may not run before `admit_benchmark_run`. It is derived
|
||||
# post-admission and attached to the retained manifest, next to `fidelity`.
|
||||
"report_grade": "local_low_seed" if (args.runs < 5 or args.path == "bridge") else "leaderboard_shape",
|
||||
|
|
@ -782,30 +839,36 @@ def main(argv: list[str] | None = None) -> int:
|
|||
"planned_invocations": plans,
|
||||
"settings_template": str(pathlib.Path(args.settings).expanduser()),
|
||||
"settings_derived": str(rendered_path)}
|
||||
# Where the runtime attestation for THIS run comes from — derived, never assumed. The
|
||||
# host engine is attested inside `IsolatedServer._wait_ready()`. The docker engine is a
|
||||
# thin stand-in that never calls it, so its attestation exists only if the tracked
|
||||
# operator patch is applied in the clone that will execute (README.md item 10). Naming
|
||||
# the patch because `--docker` was passed asserted a check that had not run.
|
||||
attested = bool(patch_probe["patches"]["clb_docker_runtime_attestation.v6746"])
|
||||
# NAME THE FACT YOU HAVE. This is a TREE probe (owner-approved goal): it establishes
|
||||
# that the attestation hook is PRESENT in the adapter checkout the run will execute,
|
||||
# which is not the same claim as "attestation ran" — a dry run establishes it without a
|
||||
# container ever starting. The field is therefore `runtime_attestation_available`, and
|
||||
# the enforcement itself still lives where it belongs: the hook fails the run closed on
|
||||
# skew / unreachable runtime / unknown commit, before any paid task.
|
||||
hook_available = bool(patch_probe["patches"]["clb_docker_runtime_attestation.v6746"])
|
||||
manifest["extra"].update({
|
||||
"solve_model": _litellm_model(args.model),
|
||||
"fidelity": fidelity,
|
||||
"adapter_operator_patches": patch_probe,
|
||||
"runtime_attestation_path": (
|
||||
("docker engine: operator_patch clb_docker_runtime_attestation.v6746 "
|
||||
"(DockerOuroborosEngine._attest_runtime, post-health, pre-solve)" if attested else
|
||||
"(DockerOuroborosEngine._attest_runtime, post-health, pre-solve)"
|
||||
if hook_available else
|
||||
"NONE: the docker engine never calls IsolatedServer._wait_ready() and operator "
|
||||
"patch clb_docker_runtime_attestation.v6746 is NOT applied in the execution "
|
||||
"clone — THIS RUN IS UNATTESTED")
|
||||
"patch clb_docker_runtime_attestation.v6746 is NOT applied in the "
|
||||
"--runner-path ADAPTER checkout (not the --ouroboros-clone seed) — "
|
||||
"THIS RUN IS UNATTESTED")
|
||||
if args.docker else
|
||||
"host engine: IsolatedServer._wait_ready()"
|
||||
),
|
||||
"runtime_attested": (not args.docker) or attested,
|
||||
"runtime_attestation_available": (not args.docker) or hook_available,
|
||||
"runtime_attestation_evidence": (
|
||||
"static_source_scan of the --runner-path adapter checkout" if args.docker
|
||||
else "in-tree IsolatedServer._wait_ready (no operator patch involved)"),
|
||||
"provider_env_present": redacted_env_summary(child_env),
|
||||
})
|
||||
if args.docker and not attested:
|
||||
print("[clb] WARNING: docker path with NO runtime attestation — operator patch "
|
||||
if args.docker and not hook_available:
|
||||
print("[clb] WARNING: docker path with NO runtime attestation hook — operator patch "
|
||||
"clb_docker_runtime_attestation.v6746 is not applied in "
|
||||
f"{patch_probe['adapter_path']}; the run's provenance records this honestly "
|
||||
"rather than claiming a check that did not run.", file=sys.stderr)
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ from typing import Any
|
|||
if str(pathlib.Path(__file__).resolve().parents[4]) not in sys.path:
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[4]))
|
||||
|
||||
from devtools.benchmarks.common.result_index import runtime_terminal_disclosure
|
||||
from devtools.benchmarks.common.run_roots import ensure_outside_repo, run_root
|
||||
from devtools.benchmarks.gaia.inspect_solver import (
|
||||
GAIA_ANTI_LEAK_INSTRUCTION,
|
||||
|
|
@ -150,6 +151,10 @@ def run_ouroboros(prompt: str, sample_id: str = "sample", attachments: list[path
|
|||
"final_answer": str(answer or "").strip(),
|
||||
"returncode": proc.returncode,
|
||||
"result_json": str(result_json),
|
||||
# The full runtime task result is already on disk here and was read for the answer
|
||||
# alone; the terminal reason travelled no further than a path pointer, so a sample
|
||||
# the runtime truncated scored as an ordinary wrong answer in the Inspect log.
|
||||
"runtime_outcome": runtime_terminal_disclosure(payload),
|
||||
"stderr_tail": proc.stderr[-4000:],
|
||||
}
|
||||
|
||||
|
|
@ -476,6 +481,7 @@ def ouroboros_solver():
|
|||
if not hasattr(state, "metadata") or getattr(state, "metadata") is None:
|
||||
state.metadata = {}
|
||||
state.metadata["ouroboros_result_json"] = result.get("result_json", "")
|
||||
state.metadata["ouroboros_runtime_outcome"] = result.get("runtime_outcome") or {"available": False}
|
||||
if not hasattr(state, "output") or getattr(state, "output") is None:
|
||||
state.output = SimpleNamespace(completion="")
|
||||
state.output.completion = result["final_answer"]
|
||||
|
|
|
|||
|
|
@ -25,6 +25,13 @@ from typing import Any
|
|||
|
||||
|
||||
DEFAULT_REPO = pathlib.Path(__file__).resolve().parents[3]
|
||||
# Invoked as a SCRIPT by `harness_bench run-cli`, so the repo root is not implicitly on the
|
||||
# path the way it is for the package-imported adapters.
|
||||
if str(DEFAULT_REPO) not in sys.path:
|
||||
sys.path.insert(0, str(DEFAULT_REPO))
|
||||
|
||||
from devtools.benchmarks.common.result_index import runtime_terminal_disclosure # noqa: E402
|
||||
|
||||
DEFAULT_DATA = DEFAULT_REPO.parent / "data"
|
||||
DEFAULT_SETTINGS = DEFAULT_DATA / "settings.json"
|
||||
DEFAULT_OUROBOROS_BIN = DEFAULT_REPO.parent / ".venv" / "bin" / "ouroboros"
|
||||
|
|
@ -50,6 +57,14 @@ def _task_id_from_workspace(workspace: pathlib.Path) -> str:
|
|||
return name
|
||||
|
||||
|
||||
def _read_json(path: pathlib.Path) -> Any:
|
||||
"""Best-effort read; an absent/corrupt result file must not fail the wrapper."""
|
||||
try:
|
||||
return json.loads(pathlib.Path(path).read_text(encoding="utf-8"))
|
||||
except Exception: # noqa: BLE001 - disclosure is best-effort, the run's exit code is not
|
||||
return None
|
||||
|
||||
|
||||
def _write_json(path: pathlib.Path, payload: Any) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
|
|
@ -218,6 +233,11 @@ def main(argv: list[str] | None = None) -> int:
|
|||
"stdout_chars": len(completed.stdout or ""),
|
||||
"stderr_chars": len(completed.stderr or ""),
|
||||
"result_json": str(result_json),
|
||||
# The runtime task result is written to `result_json` by `--result-json-out`, but
|
||||
# the summary only ever POINTED at it, so a task the runtime truncated (per-task
|
||||
# USD rail, round cap) was indistinguishable from an honest verifier failure in
|
||||
# every artefact a reader of this summary sees.
|
||||
"runtime_outcome": runtime_terminal_disclosure(_read_json(result_json)),
|
||||
},
|
||||
)
|
||||
return int(completed.returncode)
|
||||
|
|
|
|||
|
|
@ -161,6 +161,29 @@ final-state-only numbers; say so. Preflight-blocked and adapter-error tasks
|
|||
stay in the denominator via `result_index.jsonl` (`status=blocked` /
|
||||
`adapter_error`).
|
||||
|
||||
**Do NOT score by filtering `result_index.jsonl` rows on `status == "completed"`.**
|
||||
`status` is a claim about the RECORD, not about the evaluation:
|
||||
|
||||
- `status="partially_published"` means the score was obtained and the official
|
||||
evaluation really ran, but at least one destination (the outcome sidecar, the
|
||||
manifest amendment, the ledger) could not be written. The status the run
|
||||
reached is preserved verbatim in `details.outcome_status`, and
|
||||
`details.publication_errors` lists the gap. Filtering on `status` silently
|
||||
drops these SCORED rows.
|
||||
- `output_paths.task_outcome` MAY BE ABSENT. The pointer is emitted only when
|
||||
that write actually succeeded — a row naming a path that does not exist is
|
||||
worse than one naming none, because a reader cannot distinguish it from a
|
||||
file deleted later. The finalized attempt manifest follows the same rule.
|
||||
- `runtime_outcome` (see §7.4) carries the RUNTIME's own terminal reason,
|
||||
independently of the adapter-stage `status`/`reason_code`. A cost-truncated
|
||||
task publishes `status="completed"`, `reason_code="official_evaluate"` and
|
||||
`runtime_outcome.reason_code="budget_exhausted"`, `truncated=true` — all
|
||||
three are true at once and the artefact must be read that way.
|
||||
|
||||
Score from `details.reward` / `official_eval_status` (which are deliberately
|
||||
never demoted by a publication failure), and use `status` only to judge how
|
||||
complete the record is.
|
||||
|
||||
A refused runtime attestation keeps its EVIDENCE: all three launchers catch
|
||||
`RuntimeAttestationRefused` and persist the record it carries (the exact typed
|
||||
reason plus `runtime_version`, `repo_head`, `repo_version`) under
|
||||
|
|
@ -210,14 +233,33 @@ leaderboard run without the disclosures below.
|
|||
iff the last recorded action is `FAIL`, so this matches the official
|
||||
semantics; a `FAIL` on a feasible task scores 0. Detection reads only the
|
||||
terminal answer (never intermediate reasoning) to avoid spurious flips.
|
||||
4. **Budget is rounds + wall-clock, NOT leaderboard steps.** There is no
|
||||
per-task step cap; the budget is the bench server's `OUROBOROS_MAX_ROUNDS`
|
||||
(default 200) plus `--task_timeout_sec`. A leaderboard "step" is one model
|
||||
turn (which may batch several pyautogui actions); an Ouroboros round is not
|
||||
step-equivalent. `task_outcome.json` records `budget_counters`
|
||||
(`llm_rounds`, `screenshots`, `gui_action_calls`, `remote_exec_calls`) and
|
||||
`max_rounds_effective`; report these alongside any score. The current
|
||||
Verified leaderboard standard is 100 steps.
|
||||
4. **Budget is rounds + wall-clock + a per-task USD rail, NOT leaderboard
|
||||
steps.** There is no per-task STEP cap. There are THREE caps, and the one
|
||||
that binds in practice is the third:
|
||||
- the bench server's `OUROBOROS_MAX_ROUNDS` (default 200);
|
||||
- `--task_timeout_sec` wall clock;
|
||||
- **the runtime's per-task USD reservation rail**
|
||||
(`OUROBOROS_PER_TASK_COST_USD`, enforced by
|
||||
`usage_accounting.reserve_attempt`: it refuses when
|
||||
`root_accounted + reservation_upper_bound > root_limit_usd`). The bound is
|
||||
a WORST-CASE estimate that grows with multimodal context, so on a
|
||||
screenshot-heavy OSWorld task it reaches the rail far below actual spend —
|
||||
in the v6.81.0 smoke it tripped a $6.00 rail at **$0.45 of actual spend**,
|
||||
stopping tasks at 13 and 22 rounds while `max_rounds_effective` reported
|
||||
200. An earlier version of this item claimed there was no other per-task
|
||||
cap; that was false, and it is exactly the kind of claim this file exists
|
||||
to prevent.
|
||||
|
||||
A leaderboard "step" is one model turn (which may batch several pyautogui
|
||||
actions); an Ouroboros round is not step-equivalent. `task_outcome.json`
|
||||
records `budget_counters` (`llm_rounds`, `screenshots`, `gui_action_calls`,
|
||||
`remote_exec_calls`) and `max_rounds_effective`; report these alongside any
|
||||
score. **A task the USD rail truncated is NOT a capability failure.** Read
|
||||
`runtime_outcome` in `task_outcome.json` / `result_index.jsonl`: it carries
|
||||
the runtime's own `reason_code` (`budget_exhausted`), a `truncated` flag and
|
||||
the `resource_limit` block. Report truncated tasks separately from honest
|
||||
failures; `max_rounds_effective` alone will not tell you they happened. The
|
||||
current Verified leaderboard standard is 100 steps.
|
||||
5. **Observation modality.** Screenshot-only by DEFAULT: `ax_tree` is disabled
|
||||
per task unless `--allow-a11y`. A run with `--allow-a11y` must be reported as
|
||||
"Additional a11y tree used: Yes" (the leaderboard separates Screenshot /
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ import os
|
|||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
|
@ -49,7 +49,11 @@ from devtools.benchmarks.common.manifests import (
|
|||
runtime_attestation,
|
||||
write_json,
|
||||
)
|
||||
from devtools.benchmarks.common.result_index import append_result_index, task_result_row
|
||||
from devtools.benchmarks.common.result_index import (
|
||||
append_result_index,
|
||||
runtime_terminal_disclosure,
|
||||
task_result_row,
|
||||
)
|
||||
from devtools.benchmarks.common.run_roots import assert_outside_repo, timestamp_run_id
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
|
|
@ -602,6 +606,14 @@ class CuBridgeRun:
|
|||
# True once THIS attempt holds the task claim (or when no claim dir is configured). Only an
|
||||
# owner writes the artefacts under `run_dir` that are shared between attempts.
|
||||
owns_task: bool = False
|
||||
# The RUNTIME's own terminal task result (`GET /api/tasks/<id>`), stashed the moment the
|
||||
# poll ends so EVERY outcome path below discloses why Ouroboros stopped — not just the
|
||||
# coarse `ouroboros_status`. Two of three tasks in the v6.81.0 OSWorld smoke were
|
||||
# terminated by the per-task USD reservation rail (`reason_code=budget_exhausted`) and the
|
||||
# artefact published `status=completed, reason_code=official_evaluate`, so an aggregator
|
||||
# recorded 2/3 with no way to tell a cost-truncated run from an honest failure. Lives on
|
||||
# the run record rather than as a parameter so no outcome path can forget it.
|
||||
runtime_result: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
|
|
@ -638,10 +650,18 @@ def _write_cu_outcome(run: CuBridgeRun, reward: float | None, status: str, reaso
|
|||
and DISCLOSED (`publication_errors`, and a best-effort rewrite of the sidecars that carry
|
||||
it) instead of cancelling the destinations that would have succeeded.
|
||||
"""
|
||||
# `status`/`reason_code` here are the ADAPTER's stage vocabulary ("completed",
|
||||
# "official_evaluate"). `runtime_outcome` is a SEPARATE fact: why the Ouroboros runtime
|
||||
# itself stopped. They disagree exactly when it matters — a task the per-task USD rail
|
||||
# truncated still evaluates, so the adapter honestly reports `completed`/`official_evaluate`
|
||||
# while the runtime reports `budget_exhausted`. Publishing only the former made a truncated
|
||||
# run indistinguishable from an honest failure. Reward and `official_eval_status` are
|
||||
# untouched: this ADDS disclosure, it does not subtract fact.
|
||||
outcome = {
|
||||
"ok": status == "completed",
|
||||
"task_id": run.example_id, "domain": run.domain, "reward": reward,
|
||||
"status": status, "reason_code": reason, "error": error,
|
||||
"runtime_outcome": runtime_terminal_disclosure(run.runtime_result),
|
||||
"result_dir": str(run.run_dir), "attempt_dir": str(run.attempt_dir),
|
||||
"claim_owner": bool(run.owns_task), **(extra or {}),
|
||||
}
|
||||
|
|
@ -658,12 +678,23 @@ def _write_cu_outcome(run: CuBridgeRun, reward: float | None, status: str, reaso
|
|||
file=sys.stderr, flush=True)
|
||||
|
||||
def _amend_manifest() -> None:
|
||||
"""Amend the ADMITTED manifest — WITHOUT a pointer to an outcome that was not written.
|
||||
|
||||
Same rule as `_ledger_row`, and it has to be applied on BOTH sides: a pointer naming a
|
||||
path that does not exist is worse than no pointer, because a reader cannot tell it from
|
||||
a file deleted later. Fixing only the ledger row left the finalized attempt manifest
|
||||
still naming the missing file. `attempt_outcome` is published immediately before this,
|
||||
so `failed_destinations` is already authoritative here.
|
||||
"""
|
||||
from devtools.benchmarks.osworld.run_step_agent import amend_task_manifest
|
||||
output_paths: dict[str, str] = {"attempt_dir": str(run.attempt_dir)}
|
||||
if "attempt_outcome" not in failed_destinations:
|
||||
output_paths["task_outcome"] = str(run.attempt_dir / "task_outcome.json")
|
||||
run.base_manifest.update(amend_task_manifest(
|
||||
run.base_manifest,
|
||||
output_paths={"task_outcome": str(run.attempt_dir / "task_outcome.json"),
|
||||
"attempt_dir": str(run.attempt_dir)},
|
||||
output_paths=output_paths,
|
||||
extra={"attempt_dir": str(run.attempt_dir), "claim_owner": bool(run.owns_task),
|
||||
"runtime_outcome": runtime_terminal_disclosure(run.runtime_result),
|
||||
**(extra or {})},
|
||||
))
|
||||
|
||||
|
|
@ -722,6 +753,7 @@ def _write_cu_outcome(run: CuBridgeRun, reward: float | None, status: str, reaso
|
|||
benchmark="osworld", instance_id=run.example_id,
|
||||
status="partially_published" if partial else status,
|
||||
reason_code=reason,
|
||||
runtime_result=run.runtime_result,
|
||||
official_eval_status="completed" if reward is not None else "not_run",
|
||||
output_paths=output_paths,
|
||||
error=error, details={"domain": run.domain, "reward": reward,
|
||||
|
|
@ -993,6 +1025,11 @@ def _run_cu_bridge(args: argparse.Namespace, final: dict[str, Any], run: CuBridg
|
|||
break
|
||||
time.sleep(8)
|
||||
(run_dir / "ouroboros_task_final.json").write_text(json.dumps(latest, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
# Hand the RUNTIME's own terminal reason to every outcome path below (including the
|
||||
# adapter_error ones). Set here, once, rather than threaded as a parameter: the poll is
|
||||
# the only place it exists, and an outcome path that forgets it publishes an artefact in
|
||||
# which a cost-truncated run is indistinguishable from an honest failure.
|
||||
run.runtime_result = dict(latest)
|
||||
|
||||
infeasible_declared = _final_answer_declares_infeasible(latest)
|
||||
fail_info: dict[str, Any] = {}
|
||||
|
|
|
|||
|
|
@ -354,6 +354,7 @@ def _process_instance(instance: dict[str, Any], cfg: InstanceRunConfig) -> dict[
|
|||
instance_id=instance_id,
|
||||
status="failed",
|
||||
reason_code=str(task_result.get("reason_code") or "") or "task_not_completed",
|
||||
runtime_result=task_result,
|
||||
prediction_written=False,
|
||||
official_eval_status="not_run",
|
||||
output_paths=output_paths,
|
||||
|
|
@ -373,6 +374,11 @@ def _process_instance(instance: dict[str, Any], cfg: InstanceRunConfig) -> dict[
|
|||
instance_id=instance_id,
|
||||
status="completed",
|
||||
reason_code="submission_prepared",
|
||||
# The non-completed branch above already publishes the runtime `reason_code`; the
|
||||
# success branch dropped it, so a submission prepared by a cost-truncated run read
|
||||
# exactly like one from a run that finished. `infra_failed` collapses the runtime's
|
||||
# terminal reason to one bool and cannot carry `budget_exhausted` at all.
|
||||
runtime_result=task_result,
|
||||
prediction_written=True,
|
||||
official_eval_status="not_run",
|
||||
output_paths=output_paths,
|
||||
|
|
|
|||
|
|
@ -364,9 +364,16 @@ def _run_prediction_rows(
|
|||
instance_id=instance_id,
|
||||
status="completed",
|
||||
reason_code="patch_generated",
|
||||
# A patch produced under `budget_exhausted` used to be indistinguishable
|
||||
# from a clean completion: the failure branches above already publish the
|
||||
# runtime `reason_code`, the SUCCESS branch dropped it for a hard-coded
|
||||
# adapter-stage literal. The patch is still real and still scored — this
|
||||
# only says how the runtime stopped producing it.
|
||||
runtime_result=task_result,
|
||||
prediction_written=True,
|
||||
official_eval_status="pending",
|
||||
output_paths={"prediction_jsonl": str(output_path)},
|
||||
output_paths={"prediction_jsonl": str(output_path),
|
||||
"task_result": str(result_json_path)},
|
||||
details={"patch_bytes": len(result.stdout.encode("utf-8", errors="replace"))},
|
||||
)
|
||||
)
|
||||
|
|
|
|||
|
|
@ -838,6 +838,18 @@ PY
|
|||
or str(execution.get("status") or "") == "infra_failed"
|
||||
or str(execution.get("reason_code") or "") == "llm_api_error"
|
||||
)
|
||||
# The runtime can stop a task for a reason that is NOT "the task is finished" --
|
||||
# the per-task USD reservation rail (budget_exhausted), a round cap, a context
|
||||
# cap. That is neither infra_failed nor a fair-shot wrong answer, and without it
|
||||
# a cost-truncated trial is indistinguishable from an honest failure downstream.
|
||||
loop_outcome = latest.get("loop_outcome") if isinstance(latest.get("loop_outcome"), dict) else {{}}
|
||||
resource_limit = latest.get("resource_limit")
|
||||
if not isinstance(resource_limit, dict):
|
||||
resource_limit = loop_outcome.get("resource_limit")
|
||||
truncated = reason_code in (
|
||||
"budget_exhausted", "max_rounds_exceeded", "task_timeout",
|
||||
"cancelled", "context_exhausted",
|
||||
)
|
||||
summary = {{
|
||||
"return_code": 2 if infra_failed else 0,
|
||||
"task_status_code": 0 if status == "completed" else 1,
|
||||
|
|
@ -846,6 +858,12 @@ PY
|
|||
"status": status,
|
||||
"reason_code": reason_code,
|
||||
"infra_failed": infra_failed,
|
||||
"truncated": truncated,
|
||||
"resource_limit": resource_limit if isinstance(resource_limit, dict) else {{}},
|
||||
"degraded": bool(latest.get("degraded") or loop_outcome.get("degraded")),
|
||||
"degraded_reason": str(
|
||||
latest.get("degraded_reason") or loop_outcome.get("degraded_reason") or ""
|
||||
),
|
||||
"cost_usd": latest.get("cost_usd"),
|
||||
"prompt_tokens": latest.get("prompt_tokens"),
|
||||
"completion_tokens": latest.get("completion_tokens"),
|
||||
|
|
|
|||
|
|
@ -654,6 +654,11 @@ def write_disclosure_ledger(*, jobs_dir: pathlib.Path, out_path: pathlib.Path, r
|
|||
"exception_type": exception_type,
|
||||
"reason_code": adapter_summary.get("reason_code"),
|
||||
"infra_failed": bool(adapter_summary.get("infra_failed")),
|
||||
# The runtime's own cost/round/context truncation, forwarded by the adapter
|
||||
# summary. Without it a trial the per-task USD rail stopped is counted as a
|
||||
# fair-shot wrong answer, which is a claim about capability that never happened.
|
||||
"truncated": bool(adapter_summary.get("truncated")),
|
||||
"resource_limit": adapter_summary.get("resource_limit") or {},
|
||||
"captured_after_cancellation": captured_after_cancellation,
|
||||
"cost_usd": agent_result.get("cost_usd"),
|
||||
"turns": agent_meta.get("turns"),
|
||||
|
|
@ -695,6 +700,11 @@ def write_disclosure_ledger(*, jobs_dir: pathlib.Path, out_path: pathlib.Path, r
|
|||
post-cancellation network teardown (captured_after_cancellation) -- the
|
||||
harness cut egress at the finish line, the agent's own finalization LLM call
|
||||
died on DNS, and the real outcome was masked.
|
||||
'cost_truncated' the RUNTIME stopped the trial on its own resource rail (per-task USD
|
||||
reservation / round cap / context cap) rather than on an answer. Not
|
||||
provider_infra (nothing was unavailable) and emphatically not 'genuine':
|
||||
'genuine' asserts a FAIR SHOT, and a trial cut off at $0.45 of actual
|
||||
spend by a worst-case reservation bound did not get one.
|
||||
'genuine' reward 0 having reached a real terminal (final_message / tool_failure / ...)
|
||||
-- a fair-shot wrong answer. NOTE: captured_after_cancellation is NOT used to
|
||||
route here; it is set broadly on teardown, including on trials that DID emit
|
||||
|
|
@ -710,6 +720,10 @@ def write_disclosure_ledger(*, jobs_dir: pathlib.Path, out_path: pathlib.Path, r
|
|||
return "provider_infra" # setup timeout / nonzero exit / transport: real infra fault.
|
||||
if t.get("infra_failed"):
|
||||
return "provider_infra"
|
||||
if t.get("truncated") and t.get("reason_code") not in _provider_reasons:
|
||||
# Runtime resource rail. Ordered AFTER the infra checks (an infra fault that also
|
||||
# tripped a rail is still an infra fault) and BEFORE 'genuine'.
|
||||
return "cost_truncated"
|
||||
if t.get("reason_code") in _provider_reasons:
|
||||
# Provider reason WITH the teardown flag == masked harness cancellation, not a real
|
||||
# provider fault; WITHOUT it == a genuine mid-run provider/network death.
|
||||
|
|
@ -742,10 +756,14 @@ def write_disclosure_ledger(*, jobs_dir: pathlib.Path, out_path: pathlib.Path, r
|
|||
"api_rate_limit_error_count": int(exception_histogram.get("ApiRateLimitError", 0)),
|
||||
"provider_or_infra_failure_count": int(provider_or_infra_failures),
|
||||
"wall_clock_cancellation_count": int(categories["cancelled"]),
|
||||
"cost_truncated_count": int(categories["cost_truncated"]),
|
||||
"genuine_failure_count": int(categories["genuine"]),
|
||||
"exception_note": (
|
||||
"Honest taxonomy: every reward-0 trial is exactly one of provider_or_infra_failure / "
|
||||
"wall_clock_cancellation / genuine_failure (reward-1 trials are 'pass'). agent_timeout_count "
|
||||
"wall_clock_cancellation / cost_truncated / genuine_failure (reward-1 trials are 'pass'). "
|
||||
"cost_truncated is the RUNTIME's own resource rail (per-task USD reservation, round cap, "
|
||||
"context cap): nothing was unavailable and no answer was reached, so it is neither infra "
|
||||
"nor a fair-shot wrong answer. agent_timeout_count "
|
||||
"is the Harbor-named subset. A provider reason_code (e.g. provider_unavailable) only counts as "
|
||||
"provider_or_infra when it was NOT a post-cancellation teardown artifact: the harness cuts "
|
||||
"container egress at the finish line, so an agent's own finalization/summary LLM calls die on "
|
||||
|
|
@ -764,6 +782,7 @@ def write_disclosure_ledger(*, jobs_dir: pathlib.Path, out_path: pathlib.Path, r
|
|||
f"{ledger['agent_timeout_count']} AgentTimeoutError, "
|
||||
f"{ledger['provider_or_infra_failure_count']} provider/infra, "
|
||||
f"{ledger['wall_clock_cancellation_count']} wall-clock-cancelled, "
|
||||
f"{ledger['cost_truncated_count']} cost-truncated, "
|
||||
f"{ledger['genuine_failure_count']} genuine failures -> {out_path}"
|
||||
)
|
||||
return ledger
|
||||
|
|
|
|||
|
|
@ -172,11 +172,13 @@ def test_report_grade_k5_not_valid_is_local():
|
|||
|
||||
# --- disclosure ledger ----------------------------------------------------------
|
||||
|
||||
def _write_trial(d: pathlib.Path, task: str, reward, exc=None, reason=None):
|
||||
def _write_trial(d: pathlib.Path, task: str, reward, exc=None, reason=None, truncated=False):
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
meta = {"turns": 3}
|
||||
if reason is not None:
|
||||
meta["summary"] = {"reason_code": reason, "infra_failed": False}
|
||||
meta["summary"] = {"reason_code": reason, "infra_failed": False, "truncated": truncated,
|
||||
"resource_limit": ({"status": "resource_limited", "scope": "root"}
|
||||
if truncated else {})}
|
||||
(d / "result.json").write_text(json.dumps({
|
||||
"task_name": task, "trial_name": d.name,
|
||||
"verifier_result": {"rewards": {"reward": reward}},
|
||||
|
|
@ -209,6 +211,29 @@ def test_disclosure_ledger_provider_unavailable_vs_cancellation(tmp_path):
|
|||
assert led["genuine_failure_count"] == 0
|
||||
|
||||
|
||||
def test_disclosure_ledger_separates_cost_truncation_from_a_fair_shot_wrong_answer(tmp_path):
|
||||
"""A trial the RUNTIME's own resource rail stopped is not a `genuine` failure.
|
||||
|
||||
`genuine` asserts a FAIR SHOT — reward 0 having reached a real terminal. A trial cut off
|
||||
by the per-task USD reservation bound (a worst-case estimate that reached the rail at
|
||||
$0.45 of actual spend in the v6.81.0 OSWorld smoke) never got one, and counting it as a
|
||||
wrong answer overstates the failure as a capability result. It is not provider/infra
|
||||
either: nothing was unavailable.
|
||||
"""
|
||||
jobs = tmp_path / "job"
|
||||
_write_trial(jobs / "b1", "alpha", 0.0, reason="budget_exhausted", truncated=True)
|
||||
_write_trial(jobs / "b2", "beta", 0.0) # a real fair-shot wrong answer
|
||||
led = run_tb.write_disclosure_ledger(jobs_dir=jobs, out_path=tmp_path / "led.json", run_meta={})
|
||||
assert led["cost_truncated_count"] == 1
|
||||
assert led["genuine_failure_count"] == 1 # b2 only
|
||||
assert led["provider_or_infra_failure_count"] == 0
|
||||
assert led["wall_clock_cancellation_count"] == 0
|
||||
truncated_row = next(t for t in led["trials"] if t["task_name"] == "alpha")
|
||||
assert truncated_row["truncated"] is True
|
||||
assert truncated_row["resource_limit"]["scope"] == "root"
|
||||
assert "cost_truncated" in led["exception_note"]
|
||||
|
||||
|
||||
def test_disclosure_ledger_counts(tmp_path):
|
||||
jobs = tmp_path / "job"
|
||||
_write_trial(jobs / "t1", "alpha", 1.0)
|
||||
|
|
|
|||
|
|
@ -36,6 +36,48 @@ MODEL_PROVIDER_CREDENTIAL_KEYS: frozenset[str] = frozenset({
|
|||
"GIGACHAT_PASSWORD",
|
||||
})
|
||||
|
||||
# EVERY env/settings key ``llm.LLM._resolve_remote_target`` reads for a provider, GROUPED so
|
||||
# a credential and the fields it is useless without travel together or not at all (GigaChat
|
||||
# needs CREDENTIALS *or* USER+PASSWORD plus its endpoint/scope; Cloud.ru's key is meaningless
|
||||
# against the wrong base_url; the openai-compatible lane legitimately falls back to the legacy
|
||||
# OPENAI_* pair). Deriving a per-run credential set from anything but this table is guessing:
|
||||
# `anthropic/claude-sonnet-4.6` is an OPENROUTER model id, only `anthropic::…` is direct.
|
||||
PROVIDER_CREDENTIAL_GROUPS: dict[str, tuple[str, ...]] = {
|
||||
"openrouter": ("OPENROUTER_API_KEY",),
|
||||
"openai": ("OPENAI_API_KEY",),
|
||||
"anthropic": ("ANTHROPIC_API_KEY",),
|
||||
"cloudru": ("CLOUDRU_FOUNDATION_MODELS_API_KEY", "CLOUDRU_FOUNDATION_MODELS_BASE_URL"),
|
||||
"gigachat": (
|
||||
"GIGACHAT_CREDENTIALS", "GIGACHAT_PASSWORD", "GIGACHAT_USER",
|
||||
"GIGACHAT_BASE_URL", "GIGACHAT_SCOPE", "GIGACHAT_VERIFY_SSL_CERTS",
|
||||
),
|
||||
"openai-compatible": (
|
||||
"OPENAI_COMPATIBLE_API_KEY", "OPENAI_COMPATIBLE_BASE_URL",
|
||||
"OPENAI_API_KEY", "OPENAI_BASE_URL",
|
||||
),
|
||||
"local": (),
|
||||
}
|
||||
|
||||
# Settings keys that hold a ROUTED model identity (prefix -> provider via provider_for_model).
|
||||
# Superset of the live slots; a key absent from settings still declares whatever
|
||||
# ``config.SETTINGS_DEFAULTS`` will hand the runtime, which is why declared_model_settings()
|
||||
# fills the defaults in rather than treating "unset" as "unused".
|
||||
MODEL_SETTING_KEYS: tuple[str, ...] = (
|
||||
"OUROBOROS_MODEL", "OUROBOROS_MODEL_HEAVY", "OUROBOROS_MODEL_LIGHT",
|
||||
"OUROBOROS_MODEL_VISION", "OUROBOROS_MODEL_CONSCIOUSNESS",
|
||||
"OUROBOROS_MODEL_FALLBACKS", "OUROBOROS_MODEL_FALLBACK",
|
||||
"OUROBOROS_MODEL_DEEP_SELF_REVIEW", "OUROBOROS_WEBSEARCH_MODEL",
|
||||
"OUROBOROS_REVIEW_MODELS", "OUROBOROS_SCOPE_REVIEW_MODELS",
|
||||
"OUROBOROS_SCOPE_REVIEW_MODEL",
|
||||
)
|
||||
|
||||
# Settings keys whose value is a Claude Agent SDK / Claude Code model NAME (``opus[1m]``),
|
||||
# NOT a routed model identity: they carry no provider prefix, so provider_for_model would
|
||||
# mis-route them to OpenRouter. Their transport is the Anthropic SDK subprocess, which
|
||||
# authenticates with ANTHROPIC_API_KEY (tools/shell.py claude_code_edit,
|
||||
# tools/claude_advisory_review.py), so a non-empty value DECLARES the anthropic provider.
|
||||
CLAUDE_SDK_MODEL_SETTING_KEYS: tuple[str, ...] = ("CLAUDE_CODE_MODEL", "CLAUDE_AGENT_SDK_MODEL")
|
||||
|
||||
|
||||
def provider_for_model(model: str) -> str:
|
||||
"""Return the execution provider for a model id (``local`` for local lanes)."""
|
||||
|
|
@ -99,6 +141,85 @@ def resolve_credentialed_model(default_model: str) -> str:
|
|||
return default_model
|
||||
|
||||
|
||||
def declared_model_settings(settings: dict) -> dict[str, str]:
|
||||
"""Return the model slots a settings mapping DECLARES, with runtime defaults filled in.
|
||||
|
||||
An absent or empty slot is not "unused": the server falls back to
|
||||
``config.SETTINGS_DEFAULTS`` for it, so the default's provider is genuinely reachable and
|
||||
must be declared. Lazy config import (config imports this module)."""
|
||||
from ouroboros.config import SETTINGS_DEFAULTS
|
||||
|
||||
declared: dict[str, str] = {}
|
||||
for key in (*MODEL_SETTING_KEYS, *CLAUDE_SDK_MODEL_SETTING_KEYS):
|
||||
value = str((settings or {}).get(key) or "").strip()
|
||||
if not value:
|
||||
value = str(SETTINGS_DEFAULTS.get(key) or "").strip()
|
||||
if value:
|
||||
declared[key] = value
|
||||
return declared
|
||||
|
||||
|
||||
def providers_for_declared_models(declared: dict) -> dict[str, list[str]]:
|
||||
"""Map ``{settings key: model string}`` to ``{provider: sorted model strings}``.
|
||||
|
||||
Comma chains (fallbacks, review triads) are expanded; the Claude-SDK slots resolve to
|
||||
``anthropic`` by transport rather than by prefix."""
|
||||
found: dict[str, set] = {}
|
||||
for key, raw in (declared or {}).items():
|
||||
text = str(raw or "").strip()
|
||||
if not text:
|
||||
continue
|
||||
if str(key) in CLAUDE_SDK_MODEL_SETTING_KEYS:
|
||||
found.setdefault("anthropic", set()).add(text)
|
||||
continue
|
||||
for part in text.split(","):
|
||||
model = part.strip()
|
||||
if model:
|
||||
found.setdefault(provider_for_model(model), set()).add(model)
|
||||
return {provider: sorted(models) for provider, models in sorted(found.items())}
|
||||
|
||||
|
||||
def credential_keys_for_providers(providers) -> tuple[str, ...]:
|
||||
"""Return the ordered, de-duplicated credential keys a provider set needs."""
|
||||
keys: list[str] = []
|
||||
for provider in providers:
|
||||
group = PROVIDER_CREDENTIAL_GROUPS.get(str(provider))
|
||||
if group is None:
|
||||
# Unknown provider: fail OPEN with its primary key rather than silently
|
||||
# handing a run no credential at all.
|
||||
group = tuple(filter(None, (PROVIDER_ENV_KEYS.get(str(provider), ""),)))
|
||||
for key in group:
|
||||
if key not in keys:
|
||||
keys.append(key)
|
||||
return tuple(keys)
|
||||
|
||||
|
||||
ALL_PROVIDER_CREDENTIAL_KEYS: frozenset[str] = frozenset(
|
||||
key for group in PROVIDER_CREDENTIAL_GROUPS.values() for key in group
|
||||
)
|
||||
|
||||
|
||||
def provider_credential_plan(settings: dict) -> dict:
|
||||
"""Derive WHICH provider credentials a settings mapping's declared models actually need.
|
||||
|
||||
Returns ``{declared_model_slots, providers, planned_keys, fail_open}``. ``fail_open`` is
|
||||
the disclosed escape hatch: when nothing resolves (a settings mapping with no model slot
|
||||
at all) the plan is the FULL credential universe, because a benchmark that dies on a
|
||||
missing key at hour six is worse than one that carries a spare."""
|
||||
declared = declared_model_settings(settings)
|
||||
providers = providers_for_declared_models(declared)
|
||||
planned = credential_keys_for_providers(providers)
|
||||
fail_open = not planned
|
||||
if fail_open:
|
||||
planned = tuple(sorted(ALL_PROVIDER_CREDENTIAL_KEYS))
|
||||
return {
|
||||
"declared_model_slots": declared,
|
||||
"providers": providers,
|
||||
"planned_keys": sorted(planned),
|
||||
"fail_open": fail_open,
|
||||
}
|
||||
|
||||
|
||||
OPENAI_DIRECT_DEFAULTS = {
|
||||
"main": "openai::gpt-5.5",
|
||||
"heavy": "openai::gpt-5.5",
|
||||
|
|
|
|||
|
|
@ -1324,6 +1324,10 @@ def _handle_advisory_pre_review(
|
|||
repo_dir = pathlib.Path(ctx.repo_dir)
|
||||
drive_root = pathlib.Path(ctx.drive_root)
|
||||
|
||||
# KNOWN ORDERING DEBT (v6.82 backlog, deliberately NOT restructured here): this self-repair
|
||||
# runs ~87 lines AFTER `_release_metadata_preflight`, the gate it exists to satisfy, so with
|
||||
# respect to that gate it is dead code — a desynced version carrier still blocks. Left in
|
||||
# place because reordering runtime review machinery is out of scope for a provenance commit.
|
||||
auto_synced_paths = _auto_sync_release_metadata_if_needed(ctx, repo_dir, drive_root, paths)
|
||||
if paths is not None and auto_synced_paths:
|
||||
paths = sorted({str(p) for p in list(paths) + auto_synced_paths if str(p).strip()})
|
||||
|
|
|
|||
237
tests/test_benchmark_provenance_v6810.py
Normal file
237
tests/test_benchmark_provenance_v6810.py
Normal file
|
|
@ -0,0 +1,237 @@
|
|||
"""Provenance contracts for benchmark artefacts (v6.81.0).
|
||||
|
||||
Two claims a benchmark artefact must never make falsely:
|
||||
|
||||
* FIX A — a container carries only the provider credentials the run DECLARED, and the
|
||||
manifest discloses which ones it got (by fingerprint, never by value);
|
||||
* FIX B — a task the RUNTIME stopped for a reason other than finishing (the per-task USD
|
||||
reservation rail, a round cap) says so in the artefact, instead of being indistinguishable
|
||||
from an honest failure.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from devtools.benchmarks.common.manifests import provider_credential_disclosure
|
||||
from devtools.benchmarks.common.result_index import (
|
||||
runtime_terminal_disclosure,
|
||||
task_result_row,
|
||||
)
|
||||
from devtools.benchmarks.common.secrets import (
|
||||
credential_disclosure,
|
||||
isolated_credential_grants,
|
||||
)
|
||||
from devtools.benchmarks.common.server_runner import build_isolated_settings
|
||||
from ouroboros.provider_models import (
|
||||
PROVIDER_CREDENTIAL_GROUPS,
|
||||
PROVIDER_PREFIXES,
|
||||
credential_keys_for_providers,
|
||||
provider_credential_plan,
|
||||
)
|
||||
|
||||
# A live settings file carrying EVERY provider credential the owner has configured. This is
|
||||
# the realistic shape: the owner's file accumulates keys over time, and which of them a
|
||||
# benchmark container could reach used to be a function of that accumulation.
|
||||
_LIVE = {
|
||||
"OUROBOROS_MODEL": "anthropic/claude-sonnet-5",
|
||||
"OUROBOROS_MODEL_HEAVY": "claude-opus-4.8",
|
||||
"OUROBOROS_MODEL_LIGHT": "anthropic/claude-sonnet-4.6",
|
||||
"OUROBOROS_MODEL_FALLBACKS": "openai/gpt-5.5",
|
||||
"OUROBOROS_REVIEW_MODELS": "anthropic/claude-fable-5,openai/gpt-5.6-sol",
|
||||
"OPENROUTER_API_KEY": "or-value",
|
||||
"OPENAI_API_KEY": "oa-value",
|
||||
"OPENAI_BASE_URL": "https://compat.example/v1",
|
||||
"ANTHROPIC_API_KEY": "an-value",
|
||||
"CLOUDRU_FOUNDATION_MODELS_API_KEY": "cr-value",
|
||||
"CLOUDRU_FOUNDATION_MODELS_BASE_URL": "https://cloudru.example/v1",
|
||||
"OPENAI_COMPATIBLE_API_KEY": "compat-value",
|
||||
"GIGACHAT_CREDENTIALS": "gc-value",
|
||||
"GIGACHAT_PASSWORD": "gp-value",
|
||||
"GITHUB_TOKEN": "gh-value",
|
||||
"OUROBOROS_NETWORK_PASSWORD": "np-value",
|
||||
"TELEGRAM_BOT_TOKEN": "tg-value",
|
||||
"TOTAL_BUDGET": 100.0,
|
||||
}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- FIX A
|
||||
|
||||
|
||||
def test_isolated_settings_grant_only_the_declared_providers_credentials():
|
||||
"""A run pinned to OpenRouter must not receive the owner's DIRECT provider keys.
|
||||
|
||||
Owner/control secrets were already excluded and still are. The defect was narrower: every
|
||||
provider credential in the live file was copied regardless of which providers the run
|
||||
declared, so a routing fallback could spend outside the declared bucket while the manifest
|
||||
said otherwise — and the reachable provider set was a function of whatever happened to be
|
||||
in the live file at launch, which makes two nominally identical runs differ invisibly.
|
||||
"""
|
||||
out = build_isolated_settings(_LIVE, OUROBOROS_RUNTIME_MODE="advanced")
|
||||
|
||||
assert out["OPENROUTER_API_KEY"] == "or-value" # every declared slot routes here
|
||||
assert out["ANTHROPIC_API_KEY"] == "an-value" # CLAUDE_CODE_MODEL's SDK transport
|
||||
for never in ("OPENAI_API_KEY", "OPENAI_BASE_URL", "OPENAI_COMPATIBLE_API_KEY",
|
||||
"CLOUDRU_FOUNDATION_MODELS_API_KEY", "CLOUDRU_FOUNDATION_MODELS_BASE_URL",
|
||||
"GIGACHAT_CREDENTIALS", "GIGACHAT_PASSWORD"):
|
||||
assert never not in out, f"{never} was not declared by any model slot"
|
||||
# Unchanged: owner/control and transport secrets were never copied and must stay out.
|
||||
for owner_secret in ("GITHUB_TOKEN", "OUROBOROS_NETWORK_PASSWORD", "TELEGRAM_BOT_TOKEN"):
|
||||
assert owner_secret not in out
|
||||
|
||||
|
||||
def test_declaring_a_direct_provider_slot_grants_exactly_that_provider():
|
||||
"""The mirror: a run that DOES declare a direct lane must still be able to authenticate.
|
||||
|
||||
Fail-closed in the wrong direction is worse than a spare key — a benchmark that dies on a
|
||||
missing credential at hour six burns the whole schedule.
|
||||
"""
|
||||
cloudru = build_isolated_settings(_LIVE, OUROBOROS_MODEL="cloudru::zai-org/GLM-4.7")
|
||||
assert cloudru["CLOUDRU_FOUNDATION_MODELS_API_KEY"] == "cr-value"
|
||||
assert "GIGACHAT_CREDENTIALS" not in cloudru
|
||||
|
||||
compat = build_isolated_settings(_LIVE, OUROBOROS_MODEL="openai-compatible::local-llm")
|
||||
assert compat["OPENAI_COMPATIBLE_API_KEY"] == "compat-value"
|
||||
# The openai-compatible lane legitimately falls back to the legacy OPENAI_* pair.
|
||||
assert compat["OPENAI_API_KEY"] == "oa-value"
|
||||
assert compat["OPENAI_BASE_URL"] == "https://compat.example/v1"
|
||||
|
||||
|
||||
def test_paired_credentials_travel_together_or_not_at_all():
|
||||
"""GigaChat needs CREDENTIALS *or* USER+PASSWORD plus its endpoint; Cloud.ru needs its
|
||||
base_url. A key without the fields it is useless without is a broken grant, and the
|
||||
`GIGACHAT_` blanket prefix used to smuggle exactly half of one in unconditionally."""
|
||||
from devtools.benchmarks.common.server_runner import _ISO_SETTINGS_ALLOW_PREFIX
|
||||
|
||||
assert "GIGACHAT_" not in _ISO_SETTINGS_ALLOW_PREFIX, \
|
||||
"the GigaChat family must be gated on the declared slots, not copied by prefix"
|
||||
|
||||
giga = build_isolated_settings(_LIVE, OUROBOROS_MODEL="gigachat::GigaChat-3-Ultra")
|
||||
assert giga["GIGACHAT_CREDENTIALS"] == "gc-value"
|
||||
assert giga["GIGACHAT_PASSWORD"] == "gp-value"
|
||||
|
||||
without = build_isolated_settings(_LIVE)
|
||||
assert "GIGACHAT_CREDENTIALS" not in without and "GIGACHAT_PASSWORD" not in without
|
||||
|
||||
|
||||
def test_credential_groups_cover_every_routable_provider():
|
||||
"""Drift guard. `provider_for_model` can only return a provider from PROVIDER_PREFIXES;
|
||||
a new one without a credential group would silently grant nothing."""
|
||||
for _prefix, provider in PROVIDER_PREFIXES:
|
||||
assert provider in PROVIDER_CREDENTIAL_GROUPS, provider
|
||||
assert credential_keys_for_providers(["openrouter"]) == ("OPENROUTER_API_KEY",)
|
||||
|
||||
|
||||
def test_a_settings_mapping_with_no_slots_fails_OPEN_and_discloses_it(monkeypatch):
|
||||
"""No resolvable slot at all must not mean "no credentials" — that kills a run outright.
|
||||
|
||||
Ambiguity resolves toward carrying a spare, never toward removing one, and the escape is
|
||||
taken OPENLY: `fail_open` rides in the record so an auditor is not left reading a full
|
||||
credential list as if the slots had asked for it.
|
||||
"""
|
||||
import ouroboros.provider_models as pm
|
||||
|
||||
# Realistic case: SETTINGS_DEFAULTS fill the empty slots, so this still resolves narrowly.
|
||||
plan = provider_credential_plan({"OUROBOROS_MODEL": "", "CLAUDE_CODE_MODEL": ""})
|
||||
assert plan["fail_open"] is False and plan["planned_keys"]
|
||||
|
||||
# Degenerate case: nothing resolvable at all.
|
||||
monkeypatch.setattr(pm, "declared_model_settings", lambda _settings: {})
|
||||
degenerate = provider_credential_plan({})
|
||||
assert degenerate["fail_open"] is True
|
||||
assert degenerate["planned_keys"] == sorted(pm.ALL_PROVIDER_CREDENTIAL_KEYS)
|
||||
|
||||
|
||||
def test_manifest_discloses_granted_credentials_by_fingerprint_never_by_value(tmp_path):
|
||||
"""Prevention without evidence is half a fix: the artefact must let an auditor see what
|
||||
the run could reach — and must never carry the value itself."""
|
||||
settings_path = tmp_path / "settings.json"
|
||||
out = build_isolated_settings(_LIVE)
|
||||
settings_path.write_text(json.dumps(out), encoding="utf-8")
|
||||
|
||||
disclosure = provider_credential_disclosure(settings_path)
|
||||
assert disclosure["available"] is True
|
||||
assert sorted(disclosure["granted"]) == ["ANTHROPIC_API_KEY", "OPENROUTER_API_KEY"]
|
||||
assert disclosure["granted"]["OPENROUTER_API_KEY"]["present"] is True
|
||||
assert disclosure["granted"]["OPENROUTER_API_KEY"]["fingerprint"].startswith("sha256:")
|
||||
assert disclosure["fail_open"] is False
|
||||
assert "openrouter" in disclosure["providers"]
|
||||
|
||||
blob = json.dumps(disclosure)
|
||||
for value in ("or-value", "an-value", "oa-value", "gh-value"):
|
||||
assert value not in blob, "a disclosure must never carry a credential value"
|
||||
|
||||
# The same key fingerprints identically across runs — that IS the audit question.
|
||||
assert (credential_disclosure({"OPENROUTER_API_KEY": "or-value"})["OPENROUTER_API_KEY"]
|
||||
== disclosure["granted"]["OPENROUTER_API_KEY"])
|
||||
# An absent settings path is a STATED gap, never a silently empty grant list.
|
||||
assert provider_credential_disclosure(tmp_path / "nope.json") == {
|
||||
"available": False, "reason": "settings_path_absent"}
|
||||
|
||||
|
||||
def test_isolated_credential_grants_reports_the_file_not_the_intent():
|
||||
"""`planned_keys` is the derivation; `granted` is the truth about the file. An explicit
|
||||
override the slots never asked for must be VISIBLE, not inferred away."""
|
||||
out = build_isolated_settings(_LIVE, ANTHROPIC_API_KEY="", OPENAI_API_KEY="forced")
|
||||
grants = isolated_credential_grants(out)
|
||||
assert "OPENAI_API_KEY" not in grants["planned_keys"]
|
||||
assert grants["granted"]["OPENAI_API_KEY"]["present"] is True
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- FIX B
|
||||
|
||||
# The shape `GET /api/tasks/<id>` returns for a task the per-task USD reservation rail
|
||||
# stopped: `usage_accounting.reserve_attempt` refuses, `loop._handle_budget_exceeded` stamps
|
||||
# the reason and the resource-limit block, `task_results.write_task_result` persists both.
|
||||
_BUDGET_TRUNCATED = {
|
||||
"status": "failed",
|
||||
"reason_code": "budget_exhausted",
|
||||
"total_rounds": 13,
|
||||
"loop_outcome": {
|
||||
"reason_code": "budget_exhausted",
|
||||
"resource_limit": {"status": "resource_limited", "scope": "root",
|
||||
"resume_policy": "increase_or_reset_budget_then_retry"},
|
||||
},
|
||||
"outcome_axes": {"execution": {"status": "failed", "reason_code": "budget_exhausted"}},
|
||||
}
|
||||
|
||||
|
||||
def test_runtime_terminal_disclosure_names_a_cost_truncated_run():
|
||||
disclosed = runtime_terminal_disclosure(_BUDGET_TRUNCATED)
|
||||
assert disclosed["available"] is True
|
||||
assert disclosed["reason_code"] == "budget_exhausted"
|
||||
assert disclosed["truncated"] is True
|
||||
assert disclosed["resource_limit"]["scope"] == "root"
|
||||
assert disclosed["execution_reason_code"] == "budget_exhausted"
|
||||
|
||||
|
||||
def test_runtime_terminal_disclosure_states_the_gap_instead_of_inventing_one():
|
||||
"""A writer with no runtime result must say so — never a fabricated reason, never a
|
||||
silent absence a reader would mistake for "nothing to report"."""
|
||||
assert runtime_terminal_disclosure(None) == {"available": False}
|
||||
assert runtime_terminal_disclosure({}) == {"available": False}
|
||||
ok = runtime_terminal_disclosure({"status": "completed", "reason_code": "final_answer"})
|
||||
assert ok["available"] is True and ok["truncated"] is False
|
||||
|
||||
|
||||
def test_task_result_row_publishes_the_runtime_reason_alongside_the_adapter_stage():
|
||||
"""The two vocabularies are independent facts and BOTH must reach the ledger.
|
||||
|
||||
An adapter honestly reports `completed`/`official_evaluate` — the evaluation really did
|
||||
run — while the runtime reports `budget_exhausted`. Publishing only the former is how an
|
||||
aggregator records 2/3 with no indication that a third of the run was cost-truncated.
|
||||
"""
|
||||
row = task_result_row(
|
||||
benchmark="osworld", instance_id="chrome/abc", status="completed",
|
||||
reason_code="official_evaluate", official_eval_status="completed",
|
||||
runtime_result=_BUDGET_TRUNCATED, details={"reward": 0.0},
|
||||
)
|
||||
assert row["status"] == "completed" # unchanged: not demoted
|
||||
assert row["official_eval_status"] == "completed" # unchanged: the eval DID run
|
||||
assert row["reason_code"] == "official_evaluate" # unchanged: adapter stage
|
||||
assert row["runtime_outcome"]["reason_code"] == "budget_exhausted"
|
||||
assert row["runtime_outcome"]["truncated"] is True
|
||||
|
||||
# Every row carries the field, so an auditor never has to guess whether it was omitted
|
||||
# because nothing happened or because the writer forgot.
|
||||
assert task_result_row(benchmark="gaia", instance_id="x",
|
||||
status="failed")["runtime_outcome"] == {"available": False}
|
||||
|
|
@ -370,22 +370,29 @@ def test_dry_run_records_seed_escape_and_attestation_path(tmp_path):
|
|||
# run is unattested. Naming the patch because `--docker` was passed asserted a provenance
|
||||
# check that never ran — the same defect as a manifest naming a model that never ran. The
|
||||
# previous form of this assertion pinned that bug: it demanded the false claim.
|
||||
assert extra["runtime_attested"] is False
|
||||
assert extra["runtime_attestation_available"] is False
|
||||
assert "UNATTESTED" in extra["runtime_attestation_path"]
|
||||
assert extra["adapter_operator_patches"]["patches"][
|
||||
"clb_docker_runtime_attestation.v6746"] is False
|
||||
|
||||
|
||||
def test_dry_run_claims_attestation_only_when_the_patch_is_in_the_execution_clone(tmp_path):
|
||||
"""The mirror: a clone that REALLY carries the hook is recorded as attested.
|
||||
def test_attestation_field_claims_availability_not_that_the_check_ran(tmp_path):
|
||||
"""The mirror — but it names the fact the probe actually has.
|
||||
|
||||
Both directions matter. Deriving the field from the tree is only correct if it still
|
||||
reports the patched path when the patch is there — otherwise the fix would trade a false
|
||||
claim for a false denial, and an auditor would discount an attested run.
|
||||
Both directions matter: deriving the field from the tree is only correct if it still
|
||||
reports the patched path when the patch is there, otherwise the fix trades a false claim
|
||||
for a false denial. What this test no longer does is call it `runtime_attested`.
|
||||
|
||||
INVERTED BUG-PINNING TEST. The previous form asserted `extra["runtime_attested"] is True`
|
||||
for a DRY RUN against a clone that merely contained an `_attest_runtime` DEFINITION — no
|
||||
container ever started, so nothing was attested. It encoded a false positive as the
|
||||
contract. The probe is a tree probe (owner-approved goal) and the field is now
|
||||
`runtime_attestation_available`, which is exactly what a source scan can establish.
|
||||
"""
|
||||
runner = _fake_runner(tmp_path)
|
||||
(runner / "src" / "systems" / "ouroboros" / "_docker_launcher.py").write_text(
|
||||
"class DockerOuroborosEngine:\n"
|
||||
" def __init__(self):\n self.runtime_attestation: dict = {}\n"
|
||||
" def _attest_runtime(self, clone):\n pass\n",
|
||||
encoding="utf-8")
|
||||
clone = _fake_clone(tmp_path)
|
||||
|
|
@ -398,11 +405,87 @@ def test_dry_run_claims_attestation_only_when_the_patch_is_in_the_execution_clon
|
|||
assert rc == 0
|
||||
extra = json.loads((Path(json.loads(buf.getvalue())["run_root"]) / "run_manifest.json")
|
||||
.read_text(encoding="utf-8"))["extra"]
|
||||
assert extra["runtime_attested"] is True
|
||||
assert extra["runtime_attestation_available"] is True
|
||||
assert "runtime_attested" not in extra, \
|
||||
"a dry run attested nothing; the manifest must not carry a field claiming it did"
|
||||
assert extra["runtime_attestation_evidence"] == (
|
||||
"static_source_scan of the --runner-path adapter checkout")
|
||||
assert "clb_docker_runtime_attestation" in extra["runtime_attestation_path"]
|
||||
assert "UNATTESTED" not in extra["runtime_attestation_path"]
|
||||
|
||||
|
||||
def test_patch_probe_ignores_bare_env_name_mentions(tmp_path):
|
||||
"""A marker must be PATCH-UNIQUE, not a bare env-var name.
|
||||
|
||||
The pinned adapter may legitimately name `OUROBOROS_SAFETY_MODE` or
|
||||
`CLBENCH_SOLVE_DISABLED_TOOLS` in a comment or a docker `-e` passthrough list without the
|
||||
patch being applied. Keying detection on the bare name made the probe false-positive, and
|
||||
the direction is the dangerous one: the manifest would file the knob under
|
||||
`enforced_via_operator_patch` for a run that executed unenforced.
|
||||
"""
|
||||
runner = _fake_runner(tmp_path)
|
||||
adapter = runner / "src" / "systems" / "ouroboros"
|
||||
(adapter / "_docker_launcher.py").write_text(
|
||||
"# forwards nothing; the -e list below is just a passthrough\n"
|
||||
'PASSTHROUGH = ["OUROBOROS_SAFETY_MODE", "OUROBOROS_RUNTIME_MODE"]\n'
|
||||
"# see also _attest_runtime in a newer adapter\n",
|
||||
encoding="utf-8")
|
||||
(adapter / "run_clbench_bridge_agent.py").write_text(
|
||||
"# CLBENCH_SOLVE_DISABLED_TOOLS is honored by an operator patch we did not apply\n"
|
||||
"DISABLED_TOOLS: list[str] = []\n",
|
||||
encoding="utf-8")
|
||||
probe = run_clb.adapter_patch_probe(runner)
|
||||
assert probe["patches"] == {
|
||||
"clb_docker_runtime_attestation.v6746": False,
|
||||
"clb_env_campaign_overrides.v6745": False,
|
||||
"clb_disabled_tools_env.v6745": False,
|
||||
}
|
||||
assert probe["evidence"] == "static_source_scan"
|
||||
|
||||
|
||||
def test_disabled_tools_gap_is_entrypoint_specific(tmp_path):
|
||||
"""`--path standard` never executes the patched bridge module, so the knob is a GAP there.
|
||||
|
||||
Verified against the live adapter: on the standard path the run goes run_benchmark.py ->
|
||||
system.py -> `_docker_launcher.submit()`, which hardcodes `"disabled_tools": []`; only
|
||||
`run_clbench_bridge_agent.py` (bridge) and `_live_bridge.py` pass DISABLED_TOOLS. Keying
|
||||
the verdict purely on the marker filed the knob as ENFORCED on the DEFAULT path against a
|
||||
patched checkout and claimed claude_code_edit was excluded from a task contract that
|
||||
excluded nothing.
|
||||
"""
|
||||
runner = _fake_runner(tmp_path)
|
||||
adapter = runner / "src" / "systems" / "ouroboros"
|
||||
clone = _fake_clone(tmp_path)
|
||||
# A genuinely patched bridge module (patch-unique tokens, as the operator's patch writes).
|
||||
(adapter / "run_clbench_bridge_agent.py").write_text(
|
||||
"import os as _os\n"
|
||||
"# Operator patch 2026-07-23: honor CLBENCH_SOLVE_DISABLED_TOOLS from env\n"
|
||||
'DISABLED_TOOLS: list[str] = [t.strip() for t in '
|
||||
'_os.environ.get("CLBENCH_SOLVE_DISABLED_TOOLS", "").split(",") if t.strip()]\n',
|
||||
encoding="utf-8")
|
||||
|
||||
def _fidelity_for(path: str) -> dict:
|
||||
buf = io.StringIO()
|
||||
with contextlib.redirect_stdout(buf):
|
||||
rc = run_clb.main([
|
||||
"--runner-path", str(runner), "--ouroboros-clone", str(clone),
|
||||
"--path", path, "--allow-dirty-seed", "--dry-run",
|
||||
])
|
||||
assert rc == 0
|
||||
return json.loads(buf.getvalue())["fidelity"]
|
||||
|
||||
bridge = _fidelity_for("bridge")
|
||||
assert "CLBENCH_SOLVE_DISABLED_TOOLS" in bridge["enforced_via_operator_patch"]
|
||||
|
||||
standard = _fidelity_for("standard")
|
||||
assert "CLBENCH_SOLVE_DISABLED_TOOLS" not in standard["enforced_via_operator_patch"], \
|
||||
"the standard entrypoint never imports the patched bridge module"
|
||||
gap = standard["declared_only_pinned_adapter_gap"]["CLBENCH_SOLVE_DISABLED_TOOLS"]
|
||||
assert "standard" in gap["status"] and "disabled_tools=[]" in gap["status"]
|
||||
# The patch IS present in the checkout — the probe must keep saying so, honestly.
|
||||
assert standard["adapter_operator_patches"]["clb_disabled_tools_env.v6745"] is True
|
||||
|
||||
|
||||
def test_fidelity_follows_the_execution_clone_not_the_pinned_commit_constant(tmp_path):
|
||||
"""The three "declared-only" knobs are enforced iff their operator patch is in the clone.
|
||||
|
||||
|
|
@ -432,11 +515,22 @@ def test_fidelity_follows_the_execution_clone_not_the_pinned_commit_constant(tmp
|
|||
assert knob in unpatched["declared_only_pinned_adapter_gap"]
|
||||
assert knob not in unpatched["enforced_via_operator_patch"]
|
||||
|
||||
# Now apply the two forwarding patches' markers, as the operator does before a real run.
|
||||
# Now apply the two forwarding patches, as the operator does before a real run. The
|
||||
# fixtures carry the patches' OWN tokens (comment tag + the exact expression they
|
||||
# introduce), not bare env-var names — see
|
||||
# test_patch_probe_ignores_bare_env_name_mentions for why the difference is load-bearing.
|
||||
(adapter / "_docker_launcher.py").write_text(
|
||||
"ov['OUROBOROS_SAFETY_MODE'] = os.environ['OUROBOROS_SAFETY_MODE']\n", encoding="utf-8")
|
||||
" # Operator env overrides (campaign knobs). Parity defaults stay authoritative\n"
|
||||
' for _k in ("OUROBOROS_RUNTIME_MODE", "OUROBOROS_REVIEW_ENFORCEMENT",\n'
|
||||
' "OUROBOROS_SAFETY_MODE"):\n'
|
||||
" _v = os.environ.get(_k)\n"
|
||||
" if _v:\n ov[_k] = _v\n",
|
||||
encoding="utf-8")
|
||||
(adapter / "run_clbench_bridge_agent.py").write_text(
|
||||
"DISABLED_TOOLS = os.environ.get('CLBENCH_SOLVE_DISABLED_TOOLS', '').split(',')\n",
|
||||
"import os as _os\n"
|
||||
"# Operator patch 2026-07-23: honor CLBENCH_SOLVE_DISABLED_TOOLS from env\n"
|
||||
'DISABLED_TOOLS: list[str] = [t.strip() for t in '
|
||||
'_os.environ.get("CLBENCH_SOLVE_DISABLED_TOOLS", "").split(",") if t.strip()]\n',
|
||||
encoding="utf-8")
|
||||
|
||||
patched = _fidelity_of()
|
||||
|
|
|
|||
|
|
@ -1451,3 +1451,15 @@ def test_cu_bridge_ledger_row_never_points_at_an_outcome_that_was_not_written(
|
|||
assert row["details"]["reward"] == 1.0
|
||||
assert any("attempt_outcome" in e for e in row["details"]["publication_errors"]), \
|
||||
"the row must carry the collected publication errors"
|
||||
# BOTH SIDES of the same rule. The previous round fixed the ledger row and left the
|
||||
# manifest lying: `_amend_manifest` still added `output_paths.task_outcome`
|
||||
# unconditionally, so the finalized attempt manifest kept naming the missing file. A
|
||||
# pointer is a pointer wherever it is written.
|
||||
attempt_manifests = sorted(
|
||||
(results / "chrome" / "abc" / "attempts").glob("*/task_run_manifest.json"))
|
||||
assert attempt_manifests, "the attempt manifest must still be finalized"
|
||||
manifest = json.loads(attempt_manifests[-1].read_text(encoding="utf-8"))
|
||||
assert "task_outcome" not in (manifest.get("output_paths") or {}), \
|
||||
"the manifest must not point at an artefact whose write failed either"
|
||||
assert (manifest.get("output_paths") or {}).get("attempt_dir"), \
|
||||
"...while the pointer that IS valid survives"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue