feat(devtools-benchmarks): add official benchmark harnesses and workspace executor

This commit is contained in:
Ouroboros 2026-06-06 12:03:30 +03:00
parent 3a0500e43e
commit d863197cf6
76 changed files with 7979 additions and 589 deletions

View file

@ -31,6 +31,7 @@ on:
- 'build_windows.ps1'
- 'Dockerfile'
- 'scripts/**'
- 'devtools/**'
- 'packaging/**'
- 'VERSION'
- 'README.md'

View file

@ -7,7 +7,7 @@
[![Linux](https://img.shields.io/badge/Linux-x86__64-orange.svg)](https://github.com/razzant/ouroboros/releases)
[![Windows](https://img.shields.io/badge/Windows-x64-blue.svg)](https://github.com/razzant/ouroboros/releases)
[![OuroborosHub](https://img.shields.io/badge/OuroborosHub-skills%20marketplace-8A2BE2.svg)](https://github.com/razzant/OuroborosHub)
[![Version 6.18.1](https://img.shields.io/badge/version-6.18.1-green.svg)](VERSION)
[![Version 6.19.0-rc.1](https://img.shields.io/badge/version-6.19.0--rc.1-green.svg)](VERSION)
A self-modifying AI agent that writes its own code, rewrites its own mind, and evolves autonomously. Born February 16, 2026.
@ -158,8 +158,11 @@ the task id immediately.
`schedule add/list/remove` manages queue-backed scheduled tasks through the same
gateway and supervisor queue; schedules use standard 5-field cron, host-local
timezone by default, and a single catch-up run after downtime.
Benchmark helpers under `scripts/` expect clean, local, per-instance checkouts;
they do not reset or commit target repositories.
Benchmark helpers live under `devtools/benchmarks/`. They are tracked
operator tooling, reviewed when touched, and kept out of runtime imports. They
prepare official benchmark inputs/runs for ProgramBench, Terminal-Bench/Harbor,
SWE-bench, SWE-bench Pro, and OSWorld logs inspection without replacing
official scoring harnesses.
You can also override the bind address and port:
@ -517,13 +520,13 @@ the contribution guide only routes to those sources.
| Version | Date | Description |
|---------|------|-------------|
| 6.19.0-rc.1 | 2026-06-06 | **feat(devtools-benchmarks): official benchmark devtools and executor-backed workspaces.** Adds tracked `devtools/benchmarks/` tooling for ProgramBench, Terminal-Bench/Harbor, SWE-bench, SWE-bench Pro, and OSWorld logs-only inspection while keeping official harnesses as the scoring source of truth. External workspace process/service tools can now route through a host-owned `executor_ref` backend, including Docker `network=none` enforcement for benchmark cleanrooms, and broad review Atlas packs keep unrelated devtools as `excluded_dir` coverage-manifest entries while fully reviewing touched devtools files. |
| 6.18.1 | 2026-06-06 | **fix(release): keep macOS signing compatible with mobile browser checks.** macOS packages now bundle Chromium headless shell only and use the managed Playwright cache for first-use WebKit downloads, avoiding the nested `.framework`/`.xpc`/`.tbd` payload that breaks hardened-runtime codesigning inside a PyInstaller app. Linux, Docker, and Windows packages still bundle Chromium and WebKit; `engine=webkit` plus iPhone device descriptors remains the expected mobile/iOS verification path. |
| 6.18.0 | 2026-06-05 | **feat(frontend-browser): mobile-grade browser checks and UI polish.** Browser tools now support explicit `engine=chromium|webkit` and Playwright device descriptors, with packaged Chromium/WebKit install paths and CI smoke coverage for both engines. The chat composer uses a responsive glass toolbar: desktop controls stay inside the frosted field, while mobile lifts Consilium and Low/Max above the textarea and keeps Send inside. Chat-wide drag/drop stages attachments until Send; nested subagent child cards stay visible but collapsed by default with role-first headings; widget cards can be reordered via a persisted owner-local UI preference endpoint. |
| 6.17.0 | 2026-06-04 | **release: deep core capability envelope and typed task outcomes.** Task results move to a `task_contract` + `outcome_axes` + `verification_ledger` contract: objective success comes only from `task_acceptance_review` and otherwise stays `not_evaluated`, while lifecycle, execution health, artifact state, review, and objective are separate axes. Main/direct/evolution tasks start with the full enabled capability envelope, workspace parents can delegate through local-readonly subagents, and those children inherit workspace/deadline/resource context while staying locally readonly and allowing enabled external tools by owner policy. Headless/API/CLI tasks get absolute deadlines, finalization grace, explicit patch states (`ready_with_changes`, `ready_no_changes`, `missing`, `failed`), public result projection without new `result_status`, and task-aware docs layout. Evolution campaigns now count absorbed cycles only after a reviewed self-mod commit survives restart verification. |
| 6.16.0 | 2026-06-04 | **feat(extensions): reconcile worker-enabled companions in the server.** Worker-side extension enables/disables now write durable per-request markers under `state/extension_reconcile/`, and the server lifespan runs a lightweight pickup task that reconciles the server registry, starts any registered-but-missing companion processes, stops companions for disabled skills, and removes processed markers. This closes the companion-only gap left by per-process extension registries: agent `toggle_skill` and post-review auto-enable remain fire-and-forget, but the server now catches up without waiting for a route hit or restart. |
| 6.15.0 | 2026-06-04 | **feat(extensions): full out-of-process extension parity + durable health diff.** A new execution-mode capability matrix in `contracts/plugin_api.py` (PluginAPI 1.3) is the single source of truth for what an out-of-process (isolated-dep/native) extension may call, surfaced to `register(api)` through `get_runtime_info()` so a skill negotiates instead of crashing mid-registration. `on_unload` now runs at child teardown; `send_ws_message` relays through a new Host Service loopback bridge (`POST /ui/ws-message`, identity derived from the skill token, host-side namespacing); and `register_companion_process` becomes a cataloged, host-spawned and supervised surface so isolated-dep skills can run long-lived background work (a per-call child only blocks `subscribe_event` and `register_supervised_task`, which a companion replaces). Enabling an extension now verifies it via the real out-of-process catalog/register dry-run and reverts `enabled` on failure — on every enable path (UI toggle, agent tool, and post-review auto-enable) — so a skill is never left enabled-but-broken. A per-skill `health.json` records live→broken regressions across restarts and surfaces them through the health invariants, the startup verification, and the Installed UI. The chat composer replaces the Send dropdown with a Low\|Max segmented control and a one-shot Consilium pill. |
| 6.14.0 | 2026-06-03 | **feat(provider): add GigaChat as a first-class provider.** Adds Sber GigaChat (`gigachat::` model values) routed through the native `gigachat` library instead of the OpenAI SDK, since GigaChat is not OpenAI-compatible: OpenAI tools map to GigaChat functions with a single `function_call` per turn, the system message stays first while later system-reminders are demoted to user, and function results are JSON-wrapped; `reasoning_effort` is omitted on the GigaChat path so replies are not consumed by hidden reasoning. A GigaChat-only install is self-sufficient: model slots and the review and scope reviewer slots all resolve to `gigachat::` models like the Cloud.ru direct-provider pattern (an owner without a 1M-context reviewer can opt into the audited degraded advisory scope review), and this single-isolated-provider expectation is documented as an invariant in DEVELOPMENT.md and the review checklist. Sber tariff-based static pricing backs cost tracking (GigaChat-2 family at published rates; the GigaChat-3 flagship approximated at the Max tier), and the `gigachat` dependency is pinned below 0.3. |
Older releases are preserved in Git tags and GitHub releases. The 6.0.0 through 6.9.0-rc.2 rows, the 5.2.0 through 5.33.0-rc.6 rows, and former `4.0.0` rows are rolled off to respect the P9 changelog cap; their full bodies remain at their git tags.
Older releases are preserved in Git tags and GitHub releases. The 6.0.0 through 6.14.0 rows, the 5.2.0 through 5.33.0-rc.6 rows, and former `4.0.0` rows are rolled off to respect the P9 changelog cap; their full bodies remain at their git tags.
---

View file

@ -1 +1 @@
6.18.1
6.19.0-rc.1

13
devtools/README.md Normal file
View file

@ -0,0 +1,13 @@
# Ouroboros Devtools
`devtools/` contains operator-side and benchmark support code that should be
versioned with Ouroboros without becoming part of the runtime core.
Rules:
- Generated logs, datasets, run outputs, Docker layers, and secrets do not live
here.
- Default benchmark outputs go under `/Users/anton/Ouroboros/bench_runs/`.
- Runtime modules must not import `devtools`.
- This is not an immune-system bypass: touched files are reviewed normally.
- Promote code out of `devtools` only through a separate reviewed runtime plan.

4
devtools/__init__.py Normal file
View file

@ -0,0 +1,4 @@
"""Operator/devtools namespace.
Code here is tracked and reviewed when touched, but it is not runtime core.
"""

View file

@ -0,0 +1,15 @@
# Benchmark Devtools
This directory contains thin adapters around official benchmark harnesses. The
adapters prepare Ouroboros tasks, capture artifacts, and preserve traces; they
do not replace official scoring.
Supported surfaces:
- ProgramBench: official `programbench eval/info` and cleanroom submission
layout.
- Terminal-Bench: Harbor installed-agent adapter.
- SWE-bench and SWE-bench Verified: standard predictions JSONL for the official
SWE-bench harness.
- SWE-bench Pro: official Pro eval wrapper and patch-capture methodology.
- OSWorld: stop-closed skeleton and logs normalizer only.

View file

@ -0,0 +1 @@
"""Benchmark support helpers for official external harnesses."""

View file

@ -0,0 +1 @@
"""Shared benchmark helper utilities."""

View file

@ -0,0 +1,27 @@
"""Benchmark manifest helpers."""
from __future__ import annotations
import json
import pathlib
from typing import Any
def read_json(path: pathlib.Path) -> Any:
return json.loads(path.read_text(encoding="utf-8"))
def write_json(path: pathlib.Path, payload: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
def read_jsonl(path: pathlib.Path) -> list[dict[str, Any]]:
rows: list[dict[str, Any]] = []
for line in path.read_text(encoding="utf-8").splitlines():
if not line.strip():
continue
loaded = json.loads(line)
if isinstance(loaded, dict):
rows.append(loaded)
return rows

View file

@ -0,0 +1,29 @@
"""Declarative official benchmark command builders."""
from __future__ import annotations
import pathlib
def programbench_eval_cmd(run_root: pathlib.Path) -> list[str]:
return ["programbench", "eval", str(run_root)]
def programbench_info_cmd(run_root: pathlib.Path) -> list[str]:
return ["programbench", "info", str(run_root)]
def swebench_eval_cmd(dataset_name: str, predictions_path: pathlib.Path, run_id: str, workers: int = 1) -> list[str]:
return [
"python",
"-m",
"swebench.harness.run_evaluation",
"--dataset_name",
dataset_name,
"--predictions_path",
str(predictions_path),
"--max_workers",
str(int(workers)),
"--run_id",
run_id,
]

View file

@ -0,0 +1,13 @@
"""Result indexing utilities shared by benchmark adapters."""
from __future__ import annotations
import json
import pathlib
from typing import Any
def append_result_index(run_dir: pathlib.Path, row: dict[str, Any]) -> None:
run_dir.mkdir(parents=True, exist_ok=True)
with (run_dir / "result_index.jsonl").open("a", encoding="utf-8") as fh:
fh.write(json.dumps(row, ensure_ascii=False) + "\n")

View file

@ -0,0 +1,69 @@
"""Run-root helpers for benchmark devtools."""
from __future__ import annotations
import os
import pathlib
import re
import time
DEFAULT_BENCH_RUNS_ROOT = pathlib.Path("/Users/anton/Ouroboros/bench_runs")
_SAFE_BENCHMARK_ID_RE = re.compile(r"^[A-Za-z0-9_.-]+$")
def timestamp_run_id(prefix: str) -> str:
safe = "".join(ch if ch.isalnum() or ch in ("-", "_") else "-" for ch in prefix).strip("-_")
return f"{safe or 'run'}_{time.strftime('%Y%m%d_%H%M%S')}"
def run_root(benchmark: str, run_id: str = "") -> pathlib.Path:
root = pathlib.Path(os.environ.get("OUROBOROS_BENCH_RUNS_ROOT") or DEFAULT_BENCH_RUNS_ROOT)
bench = "".join(ch if ch.isalnum() or ch in ("-", "_") else "-" for ch in benchmark).strip("-_")
rid = run_id or timestamp_run_id(bench)
return (root / bench / rid).resolve(strict=False)
def ensure_outside_repo(path: pathlib.Path, repo_dir: pathlib.Path) -> pathlib.Path:
resolved = pathlib.Path(path).expanduser().resolve(strict=False)
repo = pathlib.Path(repo_dir).resolve(strict=False)
try:
resolved.relative_to(repo)
except ValueError:
resolved.mkdir(parents=True, exist_ok=True)
return resolved
raise ValueError(f"benchmark run output must not be under repo/: {resolved}")
def safe_benchmark_id(value: str, *, field: str = "instance_id") -> str:
text = str(value or "").strip()
if (
not text
or text in {".", ".."}
or "/" in text
or "\\" in text
or pathlib.PurePath(text).is_absolute()
or not _SAFE_BENCHMARK_ID_RE.fullmatch(text)
):
raise ValueError(f"{field} must be a single safe path component")
return text
def safe_join_under(root: pathlib.Path, *parts: str) -> pathlib.Path:
base = pathlib.Path(root).expanduser().resolve(strict=False)
resolved = base.joinpath(*[str(part or "") for part in parts]).resolve(strict=False)
try:
resolved.relative_to(base)
except ValueError as exc:
raise ValueError(f"benchmark output path escapes run root: {resolved}") from exc
return resolved
def ensure_file_output_outside_repo(path: pathlib.Path, repo_dir: pathlib.Path) -> pathlib.Path:
resolved = pathlib.Path(path).expanduser().resolve(strict=False)
ensure_outside_repo(resolved.parent, repo_dir)
return resolved
def repo_root_from_devtools() -> pathlib.Path:
return pathlib.Path(__file__).resolve().parents[3]

View file

@ -0,0 +1,43 @@
"""Secret-loading helpers that never print credential values."""
from __future__ import annotations
import json
import os
import pathlib
SECRET_KEYS = (
"OPENROUTER_API_KEY",
"OPENAI_API_KEY",
"ANTHROPIC_API_KEY",
"GITHUB_TOKEN",
)
def settings_path(default_home: pathlib.Path | None = None) -> pathlib.Path:
home = default_home or pathlib.Path("/Users/anton/Ouroboros")
return pathlib.Path(os.environ.get("OUROBOROS_SETTINGS_PATH") or home / "data" / "settings.json")
def load_secret_env(path: pathlib.Path | None = None) -> dict[str, str]:
values: dict[str, str] = {}
for key in SECRET_KEYS:
value = os.environ.get(key)
if value:
values[key] = value
settings_file = path or settings_path()
try:
loaded = json.loads(settings_file.read_text(encoding="utf-8"))
except Exception:
loaded = {}
if isinstance(loaded, dict):
for key in SECRET_KEYS:
value = loaded.get(key)
if value and key not in values:
values[key] = str(value)
return values
def redacted_env_summary(env: dict[str, str]) -> dict[str, bool]:
return {key: bool(env.get(key)) for key in SECRET_KEYS}

View file

@ -0,0 +1,46 @@
"""Small subprocess helpers for benchmark adapters."""
from __future__ import annotations
import json
import pathlib
import subprocess
from dataclasses import dataclass
@dataclass(frozen=True)
class CommandResult:
cmd: list[str]
returncode: int
stdout: str
stderr: str
def run_logged(cmd: list[str], *, cwd: pathlib.Path | None = None, timeout: int | None = None) -> CommandResult:
proc = subprocess.run(
[str(part) for part in cmd],
cwd=str(cwd) if cwd else None,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
timeout=timeout,
)
return CommandResult([str(part) for part in cmd], proc.returncode, proc.stdout or "", proc.stderr or "")
def write_command_result(path: pathlib.Path, result: CommandResult) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(
json.dumps(
{
"cmd": result.cmd,
"returncode": result.returncode,
"stdout_chars": len(result.stdout),
"stderr_chars": len(result.stderr),
},
ensure_ascii=False,
indent=2,
)
+ "\n",
encoding="utf-8",
)

View file

@ -0,0 +1,26 @@
# OSWorld Devtools
This directory is intentionally a stop-closed skeleton for v6.19.0-rc.1.
Local material currently available in this workspace is a logs-only bundle:
```text
/Users/anton/Ouroboros/bench_logs/osworld_sample60_seed20260603_opus47_logs_only
```
That bundle is useful for trace inspection, but it is not enough to claim
official OSWorld reproducibility. Official OSWorld local evaluation requires a
runnable OSWorld checkout, desktop/control infrastructure, and an agent adapter
inside the official runner. Public verified leaderboard claims require the
official verification path.
Files:
- `normalize_logs.py` indexes logs-only bundles for analysis.
- `schemas.py` validates the known logs-only JSON layout.
- `osworld_adapter_skeleton.py` refuses to run unless the official environment,
live Ouroboros server, computer-use payload, and output-root isolation are all
present.
No reward calculation, VM reset flow, or leaderboard scoring is implemented in
this release.

View file

@ -0,0 +1 @@
"""OSWorld log normalization and fail-closed adapter skeleton."""

View file

@ -0,0 +1,62 @@
#!/usr/bin/env python3
"""Normalize an OSWorld logs-only bundle into an inspectable index."""
from __future__ import annotations
import argparse
import json
from pathlib import Path
from typing import Any
if __package__ in {None, ""}:
import sys
sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
from devtools.benchmarks.common.run_roots import ensure_file_output_outside_repo, repo_root_from_devtools
from devtools.benchmarks.osworld.schemas import read_traj_jsonl, validate_bundle
def normalize_bundle(root: Path) -> dict[str, Any]:
bundle = validate_bundle(root)
base = Path(bundle["root"])
traj_paths = sorted(base.glob("**/traj.jsonl"))
traces = []
for path in traj_paths:
rows = read_traj_jsonl(path)
traces.append(
{
"path": str(path.relative_to(base)),
"events": len(rows),
"first_type": str(rows[0].get("type") or rows[0].get("event") or "") if rows else "",
"last_type": str(rows[-1].get("type") or rows[-1].get("event") or "") if rows else "",
}
)
return {
"bundle_root": str(base),
"summary": bundle["summary"],
"sample_manifest": bundle["sample_manifest"],
"trace_manifest": bundle["trace_manifest"],
"traj_count": len(traces),
"traces": traces,
}
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("bundle_root")
parser.add_argument("--output", default="")
args = parser.parse_args()
normalized = normalize_bundle(Path(args.bundle_root))
text = json.dumps(normalized, ensure_ascii=False, indent=2) + "\n"
if args.output:
out = ensure_file_output_outside_repo(Path(args.output), repo_root_from_devtools())
out.write_text(text, encoding="utf-8")
else:
print(text, end="")
return 0
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -0,0 +1,93 @@
#!/usr/bin/env python3
"""Fail-closed OSWorld adapter skeleton.
This file intentionally does not implement OSWorld scoring. It verifies that a
runnable official OSWorld environment and Ouroboros computer-use surface exist
before a future adapter is allowed to proceed.
"""
from __future__ import annotations
import argparse
import json
import urllib.request
from pathlib import Path
from typing import Any
def _http_json(url: str, timeout: float = 5.0) -> dict[str, Any]:
with urllib.request.urlopen(url, timeout=timeout) as resp:
raw = resp.read().decode("utf-8", errors="replace")
return json.loads(raw) if raw.strip().startswith("{") else {"raw": raw, "status": getattr(resp, "status", None)}
def _outside(path: Path, forbidden: list[Path]) -> bool:
resolved = path.expanduser().resolve(strict=False)
for root in forbidden:
try:
resolved.relative_to(root.expanduser().resolve(strict=False))
return False
except ValueError:
continue
return True
def preflight(
*,
osworld_root: Path,
ouroboros_url: str,
osworld_server_url: str,
computer_use_payload: Path,
output_root: Path,
repo_root: Path,
data_root: Path,
) -> dict[str, Any]:
failures: list[str] = []
if not osworld_root.is_dir():
failures.append(f"official OSWorld checkout not found: {osworld_root}")
if not (osworld_root / "run.py").exists() and not (osworld_root / "evaluation_examples").exists():
failures.append(f"OSWorld checkout shape is not recognized: {osworld_root}")
if not computer_use_payload.exists():
failures.append(f"computer_use payload is missing: {computer_use_payload}")
if not _outside(output_root, [repo_root, data_root]):
failures.append(f"output root must be outside repo and runtime data: {output_root}")
try:
_http_json(ouroboros_url.rstrip("/") + "/api/state")
except Exception as exc:
failures.append(f"Ouroboros server is not reachable: {type(exc).__name__}: {exc}")
try:
urllib.request.urlopen(osworld_server_url.rstrip("/") + "/", timeout=5).read(1)
except Exception as exc:
failures.append(f"OSWorld desktop/control server is not reachable: {type(exc).__name__}: {exc}")
return {"ok": not failures, "failures": failures}
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--osworld-root", required=True)
parser.add_argument("--ouroboros-url", default="http://127.0.0.1:8765")
parser.add_argument("--osworld-server-url", required=True)
parser.add_argument("--computer-use-payload", required=True)
parser.add_argument("--output-root", required=True)
parser.add_argument("--repo-root", default="/Users/anton/Ouroboros/repo")
parser.add_argument("--data-root", default="/Users/anton/Ouroboros/data")
args = parser.parse_args()
result = preflight(
osworld_root=Path(args.osworld_root).expanduser(),
ouroboros_url=args.ouroboros_url,
osworld_server_url=args.osworld_server_url,
computer_use_payload=Path(args.computer_use_payload).expanduser(),
output_root=Path(args.output_root).expanduser(),
repo_root=Path(args.repo_root).expanduser(),
data_root=Path(args.data_root).expanduser(),
)
print(json.dumps(result, ensure_ascii=False, indent=2))
if not result["ok"]:
return 2
print("OSWorld runnable adapter is not implemented in this release; preflight passed only.")
return 3
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -0,0 +1,54 @@
"""Validation helpers for OSWorld logs-only bundles."""
from __future__ import annotations
import json
from pathlib import Path
from typing import Any
REQUIRED_BUNDLE_FILES = (
"SUMMARY.json",
"sample_manifest.json",
)
def load_json(path: Path) -> dict[str, Any]:
data = json.loads(path.read_text(encoding="utf-8"))
if not isinstance(data, dict):
raise ValueError(f"{path} must contain a JSON object")
return data
def validate_bundle(root: Path) -> dict[str, Any]:
base = Path(root).expanduser().resolve(strict=False)
missing = [name for name in REQUIRED_BUNDLE_FILES if not (base / name).is_file()]
if missing:
raise ValueError(f"OSWorld logs-only bundle is missing: {', '.join(missing)}")
summary = load_json(base / "SUMMARY.json")
sample_manifest = load_json(base / "sample_manifest.json")
root_trace_manifest = base / "trace_manifest.json"
if root_trace_manifest.is_file():
trace_manifest: dict[str, Any] = load_json(root_trace_manifest)
else:
nested = sorted(path.relative_to(base).as_posix() for path in base.glob("**/traces/trace_manifest.json"))
if not nested:
raise ValueError("OSWorld logs-only bundle has no trace_manifest.json files")
trace_manifest = {"trace_manifest_paths": nested}
return {
"root": str(base),
"summary": summary,
"sample_manifest": sample_manifest,
"trace_manifest": trace_manifest,
}
def read_traj_jsonl(path: Path) -> list[dict[str, Any]]:
rows: list[dict[str, Any]] = []
for raw in Path(path).read_text(encoding="utf-8", errors="replace").splitlines():
if not raw.strip():
continue
item = json.loads(raw)
if isinstance(item, dict):
rows.append(item)
return rows

View file

@ -0,0 +1,15 @@
# ProgramBench
This adapter prepares Ouroboros workspace tasks for official ProgramBench
cleanroom execution.
Invariants:
- Use official `programbench` CLI for evaluation and summaries.
- Use `task_cleanroom` task images; do not score locally.
- Tool execution for the benchmark workspace runs in a no-network Docker
backend.
- Reference binaries are declared through
`resource_policy.protected_artifacts`: execute is allowed, byte reads,
copy/hash/static introspection/tracing/debugging are denied.
- Submission artifact is `<run>/<instance_id>/submission.tar.gz`.

View file

@ -0,0 +1 @@
"""ProgramBench adapter helpers."""

View file

@ -0,0 +1,173 @@
"""ProgramBench cleanroom adapter primitives."""
from __future__ import annotations
import pathlib
import json
import subprocess
import tarfile
from typing import Any
from devtools.benchmarks.common.manifests import write_json
from devtools.benchmarks.common.official_commands import programbench_eval_cmd, programbench_info_cmd
from devtools.benchmarks.programbench.schemas import task_body
def docker_executor_ref(
*,
container_name: str,
workspace_host_path: pathlib.Path,
workspace_backend_path: str = "/workspace",
) -> dict[str, Any]:
return {
"type": "docker_exec",
"id": container_name,
"container_name": container_name,
"network": "none",
"workspace_host_path": str(pathlib.Path(workspace_host_path).resolve(strict=False)),
"workspace_backend_path": workspace_backend_path,
}
def build_ouroboros_task_body(
*,
instruction: str,
workspace_host_path: pathlib.Path,
container_name: str,
protected_backend_paths: list[str] | None = None,
task_id: str = "",
) -> dict[str, Any]:
ensure_git_workspace(workspace_host_path)
protected = protected_backend_paths or ["/workspace/executable", "executable"]
return task_body(
description=instruction,
workspace_root=str(pathlib.Path(workspace_host_path).resolve(strict=False)),
executor_ref=docker_executor_ref(container_name=container_name, workspace_host_path=workspace_host_path),
protected_paths=protected,
task_id=task_id,
)
def preflight_cleanroom_container(container_name: str) -> dict[str, Any]:
proc = subprocess.run(
["docker", "inspect", container_name],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
timeout=15,
)
if proc.returncode != 0:
raise RuntimeError(f"docker inspect failed for {container_name}: {proc.stderr.strip()}")
data = json.loads(proc.stdout or "[]")
if not isinstance(data, list) or not data:
raise RuntimeError(f"docker inspect returned no container data for {container_name}")
info = data[0]
config = info.get("Config") if isinstance(info, dict) else {}
host_config = info.get("HostConfig") if isinstance(info, dict) else {}
image = str((config or {}).get("Image") or (info or {}).get("Image") or "")
network = str((host_config or {}).get("NetworkMode") or "")
if "task_cleanroom" not in image:
raise RuntimeError(f"ProgramBench container must use a task_cleanroom image, got {image!r}")
if network != "none":
raise RuntimeError(f"ProgramBench inference container must use Docker NetworkMode=none, got {network!r}")
return {"image": image, "network": network}
def ensure_git_workspace(workspace_root: pathlib.Path) -> None:
root = pathlib.Path(workspace_root).resolve(strict=False)
probe = subprocess.run(["git", "rev-parse", "--show-toplevel"], cwd=root, capture_output=True, text=True, timeout=10)
if probe.returncode == 0 and pathlib.Path((probe.stdout or "").strip()).resolve(strict=False) == root:
return
subprocess.run(["git", "init"], cwd=root, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, timeout=30)
subprocess.run(["git", "config", "user.email", "ouroboros-bench@example.invalid"], cwd=root, check=True, timeout=10)
subprocess.run(["git", "config", "user.name", "Ouroboros Bench"], cwd=root, check=True, timeout=10)
def create_submission_tarball(
workspace_root: pathlib.Path,
out_path: pathlib.Path,
*,
protected_paths: list[str] | None = None,
workspace_backend_path: str = "/workspace",
) -> pathlib.Path:
root = pathlib.Path(workspace_root).resolve(strict=False)
protected = _protected_submission_paths(root, protected_paths or [], workspace_backend_path=workspace_backend_path)
out_path.parent.mkdir(parents=True, exist_ok=True)
with tarfile.open(out_path, "w:gz") as tar:
for path in sorted(root.rglob("*")):
rel = path.relative_to(root)
if _skip_submission_path(rel):
continue
resolved = path.resolve(strict=False)
if any(_path_matches(resolved, protected_path) for protected_path in protected):
continue
tar.add(path, arcname=rel.as_posix(), recursive=False)
return out_path
def _protected_submission_paths(root: pathlib.Path, protected_paths: list[str], *, workspace_backend_path: str) -> list[pathlib.Path]:
protected: list[pathlib.Path] = []
backend_prefix = str(workspace_backend_path or "/workspace").rstrip("/")
for raw in protected_paths:
text = str(raw or "").strip()
if not text:
continue
if text == backend_prefix:
protected.append(root)
continue
if text.startswith(backend_prefix + "/"):
rel = text[len(backend_prefix) + 1:]
protected.append((root / rel).resolve(strict=False))
continue
candidate = pathlib.Path(text)
if candidate.is_absolute():
continue
protected.append((root / candidate).resolve(strict=False))
return list(dict.fromkeys(protected))
def _path_matches(candidate: pathlib.Path, protected: pathlib.Path) -> bool:
if candidate == protected:
return True
try:
candidate.relative_to(protected)
return True
except ValueError:
return False
def _skip_submission_path(rel: pathlib.PurePath) -> bool:
parts = set(rel.parts)
return bool(parts & {
".git",
".ouroboros",
"__pycache__",
".pytest_cache",
".ruff_cache",
".mypy_cache",
"node_modules",
"build",
"dist",
"htmlcov",
}) or rel.name in {".DS_Store", ".coverage", "coverage.xml"} or rel.suffix in {".pyc", ".pyo", ".log", ".tmp"}
def run_official_eval(run_root: pathlib.Path) -> dict[str, Any]:
eval_proc = subprocess.run(programbench_eval_cmd(run_root), capture_output=True, text=True)
info_proc = subprocess.run(programbench_info_cmd(run_root), capture_output=True, text=True)
result = {
"eval": {
"cmd": programbench_eval_cmd(run_root),
"returncode": eval_proc.returncode,
"stdout": eval_proc.stdout,
"stderr": eval_proc.stderr,
},
"info": {
"cmd": programbench_info_cmd(run_root),
"returncode": info_proc.returncode,
"stdout": info_proc.stdout,
"stderr": info_proc.stderr,
},
}
write_json(pathlib.Path(run_root) / "programbench_eval_result.json", result)
return result

View file

@ -0,0 +1,64 @@
#!/usr/bin/env python3
"""ProgramBench adapter entrypoint.
This script intentionally stops before reinventing ProgramBench orchestration.
It prepares task bodies/submissions for official cleanroom runs and delegates
evaluation to the official `programbench` CLI.
"""
from __future__ import annotations
import argparse
import pathlib
import sys
if __package__ in {None, ""}:
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[3]))
from devtools.benchmarks.common.manifests import write_json
from devtools.benchmarks.common.run_roots import ensure_outside_repo, run_root, safe_join_under
from devtools.benchmarks.programbench.programbench_adapter import (
build_ouroboros_task_body,
create_submission_tarball,
preflight_cleanroom_container,
run_official_eval,
)
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--repo-dir", default=str(pathlib.Path(__file__).resolve().parents[3]))
parser.add_argument("--workspace", required=True)
parser.add_argument("--instruction-file", required=True)
parser.add_argument("--container-name", required=True)
parser.add_argument("--instance-id", required=True)
parser.add_argument("--run-id", default="")
parser.add_argument("--protected-path", action="append", default=[], help="protected reference path inside cleanroom; repeatable")
parser.add_argument("--eval", action="store_true", help="run official programbench eval/info after writing submission")
args = parser.parse_args()
out_root = ensure_outside_repo(run_root("programbench", args.run_id), pathlib.Path(args.repo_dir))
instance_dir = safe_join_under(out_root, args.instance_id)
preflight = preflight_cleanroom_container(args.container_name)
protected_paths = args.protected_path or ["/workspace/executable", "executable"]
body = build_ouroboros_task_body(
instruction=pathlib.Path(args.instruction_file).read_text(encoding="utf-8"),
workspace_host_path=pathlib.Path(args.workspace),
container_name=args.container_name,
protected_backend_paths=protected_paths,
)
body.setdefault("metadata", {})["cleanroom_preflight"] = preflight
write_json(instance_dir / "ouroboros_task_body.json", body)
create_submission_tarball(
pathlib.Path(args.workspace),
instance_dir / "submission.tar.gz",
protected_paths=protected_paths,
)
if args.eval:
run_official_eval(out_root)
print(instance_dir)
return 0
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -0,0 +1,52 @@
"""ProgramBench adapter schemas."""
from __future__ import annotations
from typing import Any
def protected_reference_policy(paths: list[str]) -> dict[str, Any]:
clean = [str(path) for path in paths if str(path or "").strip()]
return {
"protected_artifacts": [
{
"id": "programbench_reference",
"role": "black_box_reference",
"paths": clean,
"allow": ["execute"],
"deny": [
"read_bytes",
"copy",
"hash",
"static_introspection",
"dynamic_trace",
"debug",
],
}
]
}
def task_body(
*,
description: str,
workspace_root: str,
executor_ref: dict[str, Any],
protected_paths: list[str],
task_id: str = "",
) -> dict[str, Any]:
body: dict[str, Any] = {
"description": description,
"workspace_root": workspace_root,
"workspace_mode": "external",
"memory_mode": "empty",
"allowed_resources": {"web": False, "network": False, "internet": False},
"resource_policy": protected_reference_policy(protected_paths),
"executor_ref": executor_ref,
"actor_id": "programbench",
"source": "programbench",
"metadata": {"source": "programbench"},
}
if task_id:
body["task_id"] = task_id
return body

View file

@ -0,0 +1,31 @@
# SWE-bench Devtools
These helpers generate official SWE-bench prediction JSONL files for a set of
already-prepared local task checkouts.
They do not download datasets, reset repositories, or score benchmark results.
Evaluation remains the official SWE-bench harness:
```bash
python -m swebench.harness.run_evaluation \
--dataset_name princeton-nlp/SWE-bench_Verified \
--predictions_path /path/to/predictions.jsonl \
--max_workers 1 \
--run_id ouroboros
```
Prediction rows produced by `swebench_predictions.py` contain only:
```json
{"instance_id": "...", "model_name_or_path": "...", "model_patch": "..."}
```
Supported preset aliases in `presets.py`:
- `full` -> `princeton-nlp/SWE-bench`
- `lite` -> `princeton-nlp/SWE-bench_Lite`
- `verified` -> `princeton-nlp/SWE-bench_Verified`
Input rows must provide `instance_id`, a clean git `workspace_root` or
`--workspaces-root`, and `problem_statement` or `prompt`. If `base_commit` is
present, the helper refuses to run unless the checkout HEAD matches it.

View file

@ -0,0 +1 @@
"""SWE-bench prediction helpers."""

View file

@ -0,0 +1,29 @@
"""Official SWE-bench dataset preset names used by devtools."""
from __future__ import annotations
PRESETS: dict[str, str] = {
"full": "princeton-nlp/SWE-bench",
"lite": "princeton-nlp/SWE-bench_Lite",
"verified": "princeton-nlp/SWE-bench_Verified",
}
ALIASES: dict[str, str] = {
"swe-bench": PRESETS["full"],
"swebench": PRESETS["full"],
"swe-bench-lite": PRESETS["lite"],
"swebench-lite": PRESETS["lite"],
"swe-bench-verified": PRESETS["verified"],
"swebench-verified": PRESETS["verified"],
"SWE-bench/SWE-bench_Verified": PRESETS["verified"],
}
def resolve_preset(name: str) -> str:
key = str(name or "").strip()
if key in PRESETS:
return PRESETS[key]
if key in ALIASES:
return ALIASES[key]
return key

View file

@ -0,0 +1,324 @@
#!/usr/bin/env python3
"""Generate SWE-bench predictions JSONL with Ouroboros.
This helper prepares the official prediction artifact only. Evaluation remains
the responsibility of ``swebench.harness.run_evaluation``.
"""
from __future__ import annotations
import argparse
import json
import shlex
import subprocess
import sys
import tempfile
from pathlib import Path
from typing import Any
if __package__ in {None, ""}:
sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
from devtools.benchmarks.common.run_roots import (
ensure_file_output_outside_repo,
ensure_outside_repo,
repo_root_from_devtools,
safe_benchmark_id,
)
from devtools.benchmarks.common.official_commands import swebench_eval_cmd
from devtools.benchmarks.swe_bench.presets import resolve_preset
from ouroboros.config import get_finalization_grace_sec
REPO_ROOT = repo_root_from_devtools()
def _records(path: Path) -> list[dict[str, Any]]:
rows: list[dict[str, Any]] = []
for raw in path.read_text(encoding="utf-8").splitlines():
if not raw.strip():
continue
item = json.loads(raw)
if isinstance(item, dict):
rows.append(item)
return rows
def _resolve_workspace(item: dict[str, Any], workspaces_root: str) -> str:
workspace = str(item.get("workspace_root") or "").strip()
if workspace or not workspaces_root:
return workspace
root = Path(workspaces_root).expanduser()
instance_id = str(item.get("instance_id") or "")
repo = str(item.get("repo") or "").strip()
candidates = [root / instance_id]
if repo:
candidates.extend([root / repo.replace("/", "__"), root / repo.split("/")[-1]])
for candidate in candidates:
if candidate.is_dir():
return str(candidate)
return ""
def _git_stdout(args: list[str], cwd: Path) -> subprocess.CompletedProcess[str]:
return subprocess.run(args, cwd=cwd, capture_output=True, text=True, timeout=10)
def _record_error(errors: list[dict[str, Any]], row: dict[str, Any], continue_on_error: bool) -> None:
if not continue_on_error:
raise RuntimeError(str(row.get("error") or row))
errors.append(row)
def _write_logs(logs_dir: str, instance_id: str, stdout: str, stderr: str, summary: dict[str, Any]) -> None:
if not logs_dir:
return
log_dir = Path(logs_dir).expanduser() / safe_benchmark_id(instance_id)
log_dir.mkdir(parents=True, exist_ok=True)
(log_dir / "ouroboros.stdout").write_text(stdout, encoding="utf-8")
(log_dir / "ouroboros.stderr").write_text(stderr, encoding="utf-8")
(log_dir / "ouroboros-agent-result.json").write_text(
json.dumps(summary, ensure_ascii=False, indent=2),
encoding="utf-8",
)
def _build_ouroboros_cmd(args: argparse.Namespace, workspace: Path, result_json_path: Path, prompt: str) -> list[str]:
cli_prefix = shlex.split(args.cli) if args.cli else [sys.executable, "-m", "ouroboros.cli"]
return [
*cli_prefix,
"run",
"--workspace",
str(workspace),
"--memory-mode",
"empty",
"--timeout",
str(int(args.timeout)),
"--patch",
"--result-json-out",
str(result_json_path),
prompt,
]
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--input", required=True, help="JSONL instances")
parser.add_argument("--output", required=True, help="SWE-bench predictions JSONL")
parser.add_argument("--model-name", default="ouroboros-cli")
parser.add_argument("--cli", default="", help="optional Ouroboros CLI command prefix")
parser.add_argument("--timeout", type=int, default=7200, help="per-instance Ouroboros timeout seconds")
parser.add_argument("--continue-on-error", action="store_true")
parser.add_argument("--errors-output", default="")
parser.add_argument("--logs-dir", default="")
parser.add_argument("--workspaces-root", default="")
parser.add_argument("--print-eval-command", default="", help="optional preset/dataset for official eval command")
args = parser.parse_args()
output_path = ensure_file_output_outside_repo(Path(args.output), REPO_ROOT)
errors_output_path = (
ensure_file_output_outside_repo(Path(args.errors_output), REPO_ROOT)
if args.errors_output
else Path(str(output_path) + ".errors.jsonl")
)
logs_dir = str(ensure_outside_repo(Path(args.logs_dir), REPO_ROOT)) if args.logs_dir else ""
predictions: list[dict[str, str]] = []
errors: list[dict[str, Any]] = []
for item in _records(Path(args.input)):
instance_id = str(item.get("instance_id") or "")
try:
safe_instance_id = safe_benchmark_id(instance_id)
except ValueError as exc:
_record_error(
errors,
{
"instance_id": instance_id,
"error": str(exc),
"reason_code": "invalid_instance_id",
},
args.continue_on_error,
)
continue
workspace = _resolve_workspace(item, args.workspaces_root)
prompt = str(item.get("problem_statement") or item.get("prompt") or "")
if not instance_id or not workspace or not prompt:
_record_error(
errors,
{
"instance_id": instance_id,
"error": "each row must include instance_id, workspace_root or --workspaces-root, and problem_statement/prompt",
"reason_code": "invalid_instance",
},
args.continue_on_error,
)
continue
workspace_path = Path(workspace).expanduser().resolve(strict=False)
if not workspace_path.is_dir():
_record_error(
errors,
{
"instance_id": instance_id,
"error": f"workspace_root is not a directory for {instance_id}: {workspace}",
"reason_code": "invalid_workspace",
},
args.continue_on_error,
)
continue
head = _git_stdout(["git", "rev-parse", "HEAD"], workspace_path)
if head.returncode != 0:
_record_error(
errors,
{
"instance_id": instance_id,
"error": f"workspace_root is not a git checkout for {instance_id}: {workspace_path}",
"reason_code": "not_git_checkout",
},
args.continue_on_error,
)
continue
base_commit = str(item.get("base_commit") or "").strip()
if base_commit and head.stdout.strip() != base_commit:
_record_error(
errors,
{
"instance_id": instance_id,
"error": f"workspace HEAD for {instance_id} is {head.stdout.strip()}, expected base_commit {base_commit}",
"reason_code": "wrong_base_commit",
},
args.continue_on_error,
)
continue
status = _git_stdout(["git", "status", "--porcelain=v1", "--untracked-files=all"], workspace_path)
if status.returncode != 0 or status.stdout.strip():
_record_error(
errors,
{
"instance_id": instance_id,
"error": f"workspace must be clean before SWE-bench run for {instance_id}",
"reason_code": "dirty_workspace",
},
args.continue_on_error,
)
continue
if logs_dir:
result_json_path = Path(logs_dir) / safe_instance_id / "task_result.json"
else:
result_json_path = Path(tempfile.gettempdir()) / f"ouroboros_swebench_{safe_instance_id}.task_result.json"
result_json_path.parent.mkdir(parents=True, exist_ok=True)
cmd = _build_ouroboros_cmd(args, workspace_path, result_json_path, prompt)
try:
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=int(args.timeout) + get_finalization_grace_sec() + 60,
)
except subprocess.TimeoutExpired as exc:
stdout = exc.stdout if isinstance(exc.stdout, str) else (exc.stdout or b"").decode("utf-8", errors="replace")
stderr = exc.stderr if isinstance(exc.stderr, str) else (exc.stderr or b"").decode("utf-8", errors="replace")
_write_logs(
logs_dir,
instance_id,
stdout,
stderr,
{
"instance_id": instance_id,
"returncode": 124,
"stdout_chars": len(stdout),
"stderr_chars": len(stderr),
"timeout_sec": int(args.timeout),
"failure_mode": "timeout",
},
)
_record_error(
errors,
{
"instance_id": instance_id,
"returncode": 124,
"error": f"ouroboros run timed out after {int(args.timeout)}s",
"timeout": True,
},
args.continue_on_error,
)
continue
task_result: dict[str, Any] = {}
if result_json_path.exists():
try:
loaded = json.loads(result_json_path.read_text(encoding="utf-8"))
if isinstance(loaded, dict):
task_result = loaded
except Exception:
task_result = {}
summary = {
"instance_id": instance_id,
"returncode": result.returncode,
"stdout_chars": len(result.stdout or ""),
"stderr_chars": len(result.stderr or ""),
"patch_empty": not bool((result.stdout or "").strip()),
"timeout_sec": int(args.timeout),
"outcome_axes": task_result.get("outcome_axes"),
"reason_code": task_result.get("reason_code"),
"artifact_bundle": task_result.get("artifact_bundle"),
}
_write_logs(logs_dir, instance_id, result.stdout or "", result.stderr or "", summary)
if result.returncode != 0:
details = (result.stderr or result.stdout or "").strip()
if len(details) > 4000:
details = details[:4000] + "\n...[truncated]"
_record_error(
errors,
{
"instance_id": instance_id,
"returncode": result.returncode,
"error": details or f"ouroboros run exited {result.returncode}",
"outcome_axes": task_result.get("outcome_axes"),
"reason_code": task_result.get("reason_code"),
"artifact_bundle": task_result.get("artifact_bundle"),
"trace_refs": task_result.get("trace_refs"),
},
args.continue_on_error,
)
continue
if not (result.stdout or "").strip():
_record_error(
errors,
{
"instance_id": instance_id,
"returncode": 0,
"error": "ouroboros run produced no patch",
"outcome_axes": task_result.get("outcome_axes"),
"reason_code": task_result.get("reason_code") or "no_patch",
"artifact_bundle": task_result.get("artifact_bundle"),
"trace_refs": task_result.get("trace_refs"),
},
args.continue_on_error,
)
continue
predictions.append(
{
"instance_id": instance_id,
"model_name_or_path": args.model_name,
"model_patch": result.stdout,
}
)
output_path.write_text(
"\n".join(json.dumps(row, ensure_ascii=False) for row in predictions) + ("\n" if predictions else ""),
encoding="utf-8",
)
if errors:
errors_output_path.write_text(
"\n".join(json.dumps(row, ensure_ascii=False) for row in errors) + "\n",
encoding="utf-8",
)
if args.print_eval_command:
print(" ".join(shlex.quote(part) for part in swebench_eval_cmd(resolve_preset(args.print_eval_command), output_path, "ouroboros", 1)))
return 0
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -0,0 +1,126 @@
# SWE-bench Pro Methodology Notes
These notes summarize the portable lessons from prior Ouroboros CLI runs on
SWE-bench Pro (`scaleapi/SWE-bench_Pro-os`, dataset `ScaleAI/SWE-bench_Pro`,
images `jefzda/sweap-images:{dockerhub_tag}`, task repositories under `/app`).
They are not a replacement driver or scorer. They document how to prepare
prediction patches and how to inspect official Pro evaluator outputs without
repeating the same failure modes.
Included files:
- `capture_patch.sh`: standalone `model_patch` capture for a task repository.
- `grade_pro.py`: wrapper that runs the official Pro eval and prints a
diagnostic, non-leaderboard summary of official per-instance outputs.
## 1. Capturing `model_patch`
Patch capture determines what the official evaluator sees, and it is the most
common source of false failures.
- Capture like the reference SWE-agent/mini-swe-agent scaffold:
`git add -A && git diff --cached <base_commit>`. A plain
`git diff <base>` loses new untracked source files, and several real Pro
fixes add files.
- Write the captured diff to an explicit path outside the Ouroboros repository,
normally under `/Users/anton/Ouroboros/bench_runs/`. The helper rejects
repo-internal output paths so benchmark artifacts cannot dirty `devtools/`.
- Remove environment artifacts that `git add -A` can capture. The
`JUNK_RE` pattern in `capture_patch.sh` intentionally covers runtime dumps,
caches, dependency folders, build outputs, coverage output, and similar
generated files. Do not copy broad SWE-agent defaults such as
`*.cfg`, `*.toml`, `setup.py`, or `*.lock`: Pro fixes can legitimately touch
configuration and lock files.
- Remove binary blobs. `git diff --cached --numstat <base>` prints
`-\t-\t<file>` for binary files. Build verification can leave compiled
binaries in the repository; those can inflate a tiny source patch into a huge
binary patch. Text additions such as `.go`, `.ts`, and `.py` files remain.
- In workspace mode, capture from the real task repository, usually `/app`, not
from Ouroboros's internal repository. Verify that `git -C /app status` shows
the intended modifications after the solve.
- Agent-created scratch files are the agent's responsibility, not a reason to
over-filter patches. The helper filters environment artifacts and binary
blobs, not arbitrary source-like files left by the agent.
## 2. Official Pro Eval And Diagnostic Summary
Run the official evaluator:
```bash
python swe_bench_pro_eval.py \
--use_local_docker \
--docker_platform linux/amd64 \
--dockerhub_username jefzda \
--scripts_dir run_scripts \
--raw_sample_path <SWE-bench_Pro-os>/helper_code/sweap_eval_full_v2.jsonl \
--patch_path patches.json
```
`grade_pro.py` wraps this command and then reads official per-instance
`{prefix}_output.json` files to print a diagnostic table. That table is not a
leaderboard result and is not a replacement scorer.
Important details:
- The Pro raw sample uses uppercase `FAIL_TO_PASS` and `PASS_TO_PASS` fields.
Some Hugging Face-derived rows use lowercase names; handle both when
inspecting diagnostics.
- If the official progress-bar accuracy aggregator fails or prints a misleading
zero, inspect per-instance output files directly. The official evaluator
output remains the source of truth; the local diagnostic only helps debug.
- Pro tamper protection restores test files from the fix commit after applying
the agent patch. Agent edits to test files do not count as passing fixes.
## 3. Streaming Or Evolutionary Runs
Stateful runs introduce failure classes that frozen baseline runs do not have.
- Budget ledgers can accidentally carry over between tasks. Per-task caps should
reset per task while learned state/code can carry forward as intended.
- Count API errors by structured event type, not by substring occurrences inside
nested provider messages. Separate transient transport failures from
context-overflow recovery.
- Workspace mode often needs `memory_mode=forked`; shared memory can be
forbidden with an external workspace. Verify that canonical parent reflections
still grow across tasks.
- If task N has an infrastructure failure, restore state to the snapshot after
the last clean task and rerun the suffix. Keep per-task snapshots of runtime
data and source state.
## 4. Container And Environment Pitfalls
- glibc runtimes mounted into Alpine/musl task images may not run. Use a
compatible runtime build or glibc-based images when available.
- Readiness checks need wall-clock limits. Dependency installation under
emulation can block `/api/state` for several minutes, and not every image has
`curl`; a Python readiness probe from the agent environment is often more
portable.
- Do not wait for heartbeat files such as `state/queue_snapshot.json` to become
quiet. Watch durable outputs such as task reflections or task result files.
- On macOS bind mounts, host-side files can lag behind container writes. For
live monitoring, read files inside the container with `docker exec`.
## 5. Debugging Checklist
1. Is the patch size reasonable? Huge patches often mean binary blobs; zero-byte
patches often mean the wrong workspace was captured.
2. Inspect the `*.status.txt` emitted by `capture_patch.sh`.
3. Check raw sample field casing for `FAIL_TO_PASS` and `PASS_TO_PASS`.
4. Compare per-instance `{prefix}_output.json` files to see exactly which tests
are missing.
5. Confirm that the agent did not rely on test-file edits.
6. Classify API errors by event type and failure class.
7. In stateful runs, check startup budget state before blaming solve quality.

View file

@ -0,0 +1,19 @@
# SWE-bench Pro Devtools
SWE-bench Pro is kept separate from standard SWE-bench because the colleague
materials target the `SWE-bench_Pro-os` evaluator and a Pro-specific patch JSON
handoff.
Files:
- `METHODOLOGY.md` documents the capture and grading assumptions.
- `capture_patch.sh` captures a task-repository patch with untracked text files,
filters environment junk, drops binary blobs, and requires an explicit output
path outside the Ouroboros repo.
- `pro_predictions.py` creates Ouroboros-style prediction JSONL by running
`capture_patch.sh` for prepared task repositories.
- `grade_pro.py` invokes the official Pro evaluator when `--skip-run` is not
supplied, then aggregates official per-instance outputs.
The aggregation in `grade_pro.py` is not replacement scoring. The official Pro
eval output remains the source of truth.

View file

@ -0,0 +1 @@
"""SWE-bench Pro helper scripts."""

View file

@ -0,0 +1,70 @@
#!/usr/bin/env bash
# Capture a SWE-bench Pro model_patch from the task repository.
#
# Patch capture determines what the official evaluator sees. A plain
# `git diff BASE` misses new untracked files, while an unfiltered `git add -A`
# captures runtime junk such as Redis dumps, node_modules, and compiled
# binaries. This helper follows the SWE-agent/mini-swe-agent reference shape,
# then removes environment artifacts and binary blobs.
#
# Usage:
# ./capture_patch.sh <REPO_DIR> <BASE_COMMIT> <OUT.diff>
#
# The agent is expected to have already edited <REPO_DIR>. <BASE_COMMIT> is the
# task base commit from the dataset.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd -P)"
WORK="${1:?usage: capture_patch.sh <REPO_DIR> <BASE_COMMIT> <OUT.diff>}"
BASE="${2:?base_commit is required}"
OUT="${3:?output path is required and must be outside the Ouroboros repo}"
OUT_ABS="$(python3 - "$OUT" <<'PY'
import pathlib
import sys
print(pathlib.Path(sys.argv[1]).expanduser().resolve(strict=False))
PY
)"
case "$OUT_ABS" in
"$REPO_ROOT"|"$REPO_ROOT"/*)
echo "output path must be outside the Ouroboros repo: $OUT_ABS" >&2
exit 2
;;
esac
OUT_DIR_ABS="$(dirname "$OUT_ABS")"
mkdir -p "$OUT_DIR_ABS"
STATUS_OUT="${OUT_ABS%.diff}.status.txt"
git -C "$WORK" rev-parse --verify "$BASE^{commit}" >/dev/null
cleanup() {
git -C "$WORK" reset -q >/dev/null 2>&1 || true
}
trap cleanup EXIT
# (1) Include newly created source files. Several real Pro fixes add files, and
# a clean `git diff BASE` would omit them.
git -C "$WORK" add -A
# Keep a status snapshot for mismatch debugging: M=modified, A=added,
# ??=untracked.
git -C "$WORK" status --porcelain >"$STATUS_OUT"
# (2) Drop environment artifacts. These patterns were chosen to avoid broad
# SWE-agent defaults such as *.cfg/*.toml/setup.py/*.lock, which can remove real
# Pro fixes.
JUNK_RE='appendonlydir|\.rdb$|\.aof$|\.manifest$|\.log$|\.tmp$|\.pid$|\.sock$|(^|/)node_modules/|__pycache__|\.pyc$|\.pyo$|\.pytest_cache|\.ruff_cache|\.mypy_cache|/\.cache/|(^|/)dist/|(^|/)build/|\.DS_Store|(^|/)\.coverage$|coverage\.xml$|/htmlcov/'
while IFS= read -r f; do
git -C "$WORK" reset -q -- "$f" 2>/dev/null
done < <(git -C "$WORK" diff --cached --name-only "$BASE" | grep -E "$JUNK_RE" || true)
# (3) Drop binary blobs. `git diff --cached --numstat` prints
# "-\t-\t<file>" for binary files. Text source additions remain included.
git -C "$WORK" diff --cached --numstat "$BASE" | awk -F'\t' '$1=="-" && $2=="-" {print $3}' | while IFS= read -r f; do
[ -n "$f" ] && git -C "$WORK" reset -q -- "$f" 2>/dev/null
done
# (4) Emit final model_patch and restore the index without touching the working
# tree.
git -C "$WORK" diff --cached --binary "$BASE" >"$OUT_ABS"
echo "patch -> $OUT_ABS ($(wc -c <"$OUT_ABS" 2>/dev/null || echo 0)B, files: $(grep -cE '^diff --git' "$OUT_ABS" 2>/dev/null || echo 0))" >&2

View file

@ -0,0 +1,135 @@
#!/usr/bin/env python3
"""Run official SWE-bench Pro eval, then print a diagnostic summary.
This wrapper does not replace Pro scoring. The official ``swe_bench_pro_eval.py``
output remains the source of truth. The post-run table only helps inspect
per-instance output files.
python pro/grade_pro.py --predictions runs/pro_smoke/predictions.jsonl --workers 4
"""
from __future__ import annotations
import argparse, ast, csv, json, os, pathlib, subprocess, sys
if __package__ in {None, ""}:
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[3]))
from devtools.benchmarks.common.run_roots import ensure_outside_repo, repo_root_from_devtools, run_root
PRO = pathlib.Path(__file__).resolve().parent
DEFAULT_RUN_ROOT = run_root("swe_bench_pro")
EVAL_REPO = DEFAULT_RUN_ROOT / "SWE-bench_Pro-os"
CSV_DEFAULT = PRO / "task_order_pro_70.csv"
def load_predictions(path: pathlib.Path) -> list[dict]:
return [json.loads(l) for l in path.read_text().splitlines() if l.strip()]
def colleague_verdicts(csv_path: pathlib.Path) -> dict:
with csv_path.open() as f:
return {r["instance_id"]: r["verdict"] for r in csv.DictReader(f)}
def raw_sample_index(raw_sample: pathlib.Path) -> dict:
"""Return raw official sample rows by instance id for diagnostic inspection."""
idx = {}
for l in raw_sample.read_text(errors="replace").splitlines():
if not l.strip():
continue
try:
d = json.loads(l)
except Exception:
continue
idx[d.get("instance_id", "")] = d
return idx
def as_set(v) -> set:
if isinstance(v, str):
try:
return set(json.loads(v))
except Exception:
try:
return set(ast.literal_eval(v))
except Exception:
return set()
return set(v or [])
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--predictions", default=str(DEFAULT_RUN_ROOT / "predictions.jsonl"))
ap.add_argument("--out-dir", default=str(DEFAULT_RUN_ROOT / "pro_eval"))
ap.add_argument("--eval-repo", default=str(EVAL_REPO), help="external checkout of scaleapi/SWE-bench_Pro-os")
ap.add_argument("--prefix", default="ours")
ap.add_argument("--workers", type=int, default=4)
ap.add_argument("--platform", default="linux/amd64")
ap.add_argument("--skip-run", action="store_true", help="aggregate an existing out-dir without running official eval")
ap.add_argument("--csv", default="", help="optional CSV with instance_id,verdict columns for baseline comparison")
args = ap.parse_args()
eval_repo = pathlib.Path(args.eval_repo).expanduser().resolve()
raw_sample = eval_repo / "helper_code" / "sweap_eval_full_v2.jsonl"
if not eval_repo.is_dir():
print(f"error: eval harness clone is missing: {eval_repo}\n git clone --depth 1 https://github.com/scaleapi/SWE-bench_Pro-os.git {eval_repo}", file=sys.stderr)
return 2
if not raw_sample.is_file():
print(f"error: raw_sample is missing: {raw_sample}", file=sys.stderr); return 2
preds = load_predictions(pathlib.Path(args.predictions).expanduser())
preds = [p for p in preds if (p.get("model_patch") or "").strip()]
out_dir = ensure_outside_repo(pathlib.Path(args.out_dir), repo_root_from_devtools())
patches = [{"instance_id": p["instance_id"], "patch": p["model_patch"], "prefix": args.prefix} for p in preds]
patches_path = out_dir / "patches.json"
patches_path.write_text(json.dumps(patches), encoding="utf-8")
print(f"[swe-pro] wrote {len(patches)} patches to {patches_path}", file=sys.stderr)
if not args.skip_run:
cmd = [sys.executable, "swe_bench_pro_eval.py",
"--raw_sample_path", str(raw_sample),
"--patch_path", str(patches_path),
"--output_dir", str(out_dir),
"--scripts_dir", "run_scripts",
"--num_workers", str(args.workers),
"--dockerhub_username", "jefzda",
"--use_local_docker", "--docker_platform", args.platform]
print("[swe-pro] running official eval:\n " + " ".join(cmd), file=sys.stderr)
r = subprocess.run(cmd, cwd=str(eval_repo))
if r.returncode != 0:
print(f"[swe-pro] official eval returned {r.returncode}; summarizing available output", file=sys.stderr)
rs = raw_sample_index(raw_sample)
csv_path = pathlib.Path(args.csv).expanduser() if args.csv else CSV_DEFAULT
verd = colleague_verdicts(csv_path) if csv_path.is_file() else {}
print("\n[diagnostic] Non-leaderboard summary derived from official per-instance outputs.")
print(f"{'instance':52} {'diagnostic':18} {'baseline':10} {'tests P/missing/total'}")
n_res = 0
for p in patches:
iid = p["instance_id"]
od = out_dir / iid / f"{args.prefix}_output.json"
if not od.is_file():
od2 = list((out_dir).glob(f"**/{args.prefix}_output.json"))
od = next((x for x in od2 if iid in str(x)), od)
status = "NO_OUTPUT"; ntp = "-"
if od.is_file():
try:
out = json.loads(od.read_text())
tests = out.get("tests") or []
passed = {t["name"] for t in tests if t.get("status") == "PASSED"}
rsd = rs.get(iid, {})
f2p = as_set(rsd.get("FAIL_TO_PASS") or rsd.get("fail_to_pass"))
p2p = as_set(rsd.get("PASS_TO_PASS") or rsd.get("pass_to_pass"))
need = f2p | p2p
resolved = bool(need) and need <= passed
n_res += int(resolved)
status = "DIAGNOSTIC_PASS" if resolved else "DIAGNOSTIC_FAIL"
ntp = f"{len(passed)}/{len(need - passed)}/{len(need)}"
except Exception as e:
status = f"PARSE_ERR:{e}"[:18]
print(f"{iid[:52]:52} {status:18} {verd.get(iid,'?'):10} {ntp}")
print(f"\n[diagnostic] pass-like {n_res}/{len(patches)}; official Pro eval output remains source of truth.")
return 0
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -0,0 +1,80 @@
#!/usr/bin/env python3
"""Capture SWE-bench Pro prediction patches from prepared task repositories."""
from __future__ import annotations
import argparse
import json
import subprocess
from pathlib import Path
from typing import Any
if __package__ in {None, ""}:
import sys
sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
from devtools.benchmarks.common.run_roots import ensure_file_output_outside_repo, ensure_outside_repo, run_root, safe_benchmark_id
CAPTURE = Path(__file__).resolve().parent / "capture_patch.sh"
REPO_ROOT = Path(__file__).resolve().parents[3]
def _rows(path: Path) -> list[dict[str, Any]]:
return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line.strip()]
def _capture_patch(repo_dir: Path, base_commit: str, out_path: Path) -> str:
out_path.parent.mkdir(parents=True, exist_ok=True)
proc = subprocess.run(
["bash", str(CAPTURE), str(repo_dir), base_commit, str(out_path)],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
timeout=120,
)
if proc.returncode != 0:
raise RuntimeError(f"capture_patch.sh failed for {repo_dir}: {proc.stderr or proc.stdout}")
patch = out_path.read_text(encoding="utf-8", errors="replace")
if not patch.strip():
raise RuntimeError(f"capture_patch.sh produced an empty patch for {repo_dir}")
return patch
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--input", required=True, help="JSONL rows with instance_id, repo_dir/workspace_root, base_commit")
parser.add_argument("--output", required=True, help="prediction JSONL")
parser.add_argument("--patch-dir", default="", help="optional directory for captured .diff files")
parser.add_argument("--model-name", default="ouroboros-pro")
args = parser.parse_args()
output = ensure_file_output_outside_repo(Path(args.output), REPO_ROOT)
patch_dir = Path(args.patch_dir).expanduser() if args.patch_dir else run_root("swe_bench_pro") / "patches"
ensure_outside_repo(patch_dir, REPO_ROOT)
predictions: list[dict[str, str]] = []
for item in _rows(Path(args.input).expanduser()):
instance_id = str(item.get("instance_id") or "").strip()
safe_instance_id = safe_benchmark_id(instance_id)
repo_dir = Path(str(item.get("repo_dir") or item.get("workspace_root") or "")).expanduser()
base_commit = str(item.get("base_commit") or "").strip()
if not instance_id or not repo_dir.is_dir() or not base_commit:
raise RuntimeError("each row must include instance_id, repo_dir/workspace_root, and base_commit")
patch = _capture_patch(repo_dir, base_commit, patch_dir / f"{safe_instance_id}.diff")
predictions.append(
{
"instance_id": instance_id,
"model_name_or_path": args.model_name,
"model_patch": patch,
}
)
output.write_text(
"\n".join(json.dumps(row, ensure_ascii=False) for row in predictions) + ("\n" if predictions else ""),
encoding="utf-8",
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -0,0 +1,364 @@
# Ouroboros Terminal-Bench / Harbor Installed Adapter
## Short Summary
The current adapter runs **full Ouroboros inside each Terminal-Bench task
container**.
High-level flow:
1. Harbor creates the official Terminal-Bench task container.
2. The adapter uploads the current local Ouroboros `repo/` source into that
container at `/opt/ouroboros-src`.
3. The adapter creates an isolated venv at `/opt/ouroboros-venv`.
4. The adapter installs Ouroboros from the uploaded source.
5. The adapter starts an in-container Ouroboros server/supervisor on
`127.0.0.1:8765`.
6. The adapter submits the official Terminal-Bench instruction as an external
workspace task, with `/app` or `/workspace` as the workspace root.
7. Ouroboros solves the task using its normal runtime/tools.
8. Harbor runs the official verifier.
This is intentionally **not** the old host-side terminal bridge. Ouroboros is
not asked to return one shell command per turn. It runs as normal inside the
task container.
## Why Installed Mode
The earlier adapter kept Ouroboros on the host and translated task state into a
JSON command loop. That made traces look artificially weak: Ouroboros saw a
terminal snapshot and had to return one shell command at a time.
The installed adapter evaluates Ouroboros more directly:
- each trial gets a fresh Ouroboros runtime;
- each trial gets a fresh `/logs/agent/ouroboros-data` data directory;
- the task workspace is passed as `workspace_root`;
- Ouroboros uses normal workspace tools and shell tools internally;
- Harbor still owns the task container and verifier.
## What Is Copied Into The Container
The adapter copies the current local source tree:
```text
/Users/anton/Ouroboros/repo -> /opt/ouroboros-src
```
It deliberately excludes local runtime/state noise:
```text
.git
.venv
data
data_evaluated
__pycache__
.pytest_cache
.ruff_cache
build
dist
node_modules
```
So the benchmark container gets current code, but not the operator's main
Ouroboros memory, logs, task results, or chat history.
## Runtime State In The Container
Each trial uses:
```text
OUROBOROS_REPO_DIR=/opt/ouroboros-src
OUROBOROS_DATA_DIR=/logs/agent/ouroboros-data
OUROBOROS_SETTINGS_PATH=/logs/agent/ouroboros-data/settings.json
OUROBOROS_RUNTIME_MODE=pro
OUROBOROS_REVIEW_ENFORCEMENT=advisory
OUROBOROS_WORKER_START_METHOD=spawn
```
This means:
```text
1 benchmark task = 1 fresh in-container Ouroboros = 1 unique ouroboros-data folder
```
The host `/Users/anton/Ouroboros/data` is not copied into the container.
## Provider Secret Boundary
Installed-container mode does not inject long-lived provider credentials into
Terminal-Bench task containers by default. If host settings or environment
contain provider keys, the adapter fails closed with a clear error instead of
starting a container that can expose those keys to in-container shell tools.
The intended durable solution is a reviewed host-mediated LLM bridge with scoped
task credentials. For trusted local smoke runs only, an operator may set:
```bash
OUROBOROS_BENCH_ALLOW_CONTAINER_SECRETS=1
```
Do not use that opt-in for publishable benchmark runs unless the task container,
logs, and output root are under operator control and the risk is explicitly
accepted.
## Task Instruction Integrity
The adapter now passes the official Terminal-Bench instruction unchanged:
```python
"description": instruction
```
It does not prepend harness notes or task-specific hints.
The only technical wrapper is the API request metadata and `workspace_root`.
## Workspace Resolution
Most Terminal-Bench tasks use `/app`.
Some images use `/workspace`. The adapter resolves this before starting the
Ouroboros task:
1. use `/app` if it exists;
2. otherwise use `/workspace` if it exists;
3. otherwise create `/app`.
The selected path is passed as external workspace root.
## Lifecycle
Harbor calls:
```python
await agent.setup(environment)
await agent.run(instruction, environment, context)
```
`setup()` is inherited from Harbor's `BaseInstalledAgent`; it calls our
`install()`.
`install()`:
1. uploads clean Ouroboros source into `/opt/ouroboros-src`;
2. installs system basics (`git`, `curl`, `bash`, Python/venv support);
3. if the system Python is older than 3.10, installs Python 3.12 with `uv`;
4. creates `/opt/ouroboros-venv`;
5. installs requirements and editable Ouroboros.
`run()`:
1. uploads the task instruction to `/logs/agent/instruction.txt`;
2. checks configured provider/network reachability;
3. resolves `/app` vs `/workspace`;
4. ensures the workspace is a git worktree root;
5. starts in-container Ouroboros server;
6. creates an Ouroboros task through `/api/tasks`;
7. polls `/api/tasks/<task_id>` until a final status;
8. saves task result and trace files;
9. shuts down the in-container server and cancels the task if needed.
## Why Direct API Polling
The adapter originally used:
```bash
ouroboros run --jsonl ...
```
That was fragile because the CLI stream could hang or get cancelled while the
internal task already had a final state.
The current adapter uses direct API lifecycle:
```text
POST /api/tasks
GET /api/tasks/<task_id>
POST /api/tasks/<task_id>/cancel
```
This gives the adapter a task id immediately, lets it capture task state on
timeout/cancellation, and avoids depending on an SSE/CLI stream.
## Timeout Semantics
The adapter does **not** set an internal task timeout by default:
```python
task_timeout_sec = None
```
That means Harbor controls agent execution timeout from the task config:
```text
task.toml [agent].timeout_sec
```
Setup and environment timeouts are separate:
- environment build/start: Harbor environment timeout;
- agent setup: Harbor setup timeout;
- agent execution: task `[agent].timeout_sec`;
- verifier: task `[verifier].timeout_sec`.
For heavy Docker builds, use:
```bash
--environment-build-timeout-multiplier 4
```
For installed Ouroboros setup, use:
```bash
--agent-setup-timeout-multiplier 4
```
## Common Commands
### Terminal-Bench 2.1 smoke
```bash
PYTHONPATH=/Users/anton/Ouroboros/repo \
harbor run \
--dataset terminal-bench/terminal-bench-2-1 \
--include-task-name terminal-bench/regex-log \
--agent-import-path devtools.benchmarks.terminal_bench.harbor_installed_agent:OuroborosTerminalBenchAgent \
--model ouroboros-gpt-5.5-tb21-smoke \
--agent-kwarg ouroboros_model=openai/gpt-5.5 \
--agent-kwarg install_timeout_sec=1200 \
--agent-kwarg server_start_timeout_sec=240 \
--agent-setup-timeout-multiplier 4 \
--n-concurrent 1 \
--n-tasks 1 \
--yes \
--force-build
```
### Full cached Terminal-Bench 2.0-style dataset
```bash
PYTHONPATH=/Users/anton/Ouroboros/repo \
harbor run \
--path /Users/anton/Ouroboros/data/harbor_local_datasets/terminal_bench_full_cached_89 \
--agent-import-path devtools.benchmarks.terminal_bench.harbor_installed_agent:OuroborosTerminalBenchAgent \
--model ouroboros-gpt-5.5-full \
--agent-kwarg ouroboros_model=openai/gpt-5.5 \
--agent-kwarg install_timeout_sec=1200 \
--agent-kwarg server_start_timeout_sec=240 \
--agent-setup-timeout-multiplier 4 \
--n-concurrent 1 \
--yes \
--force-build
```
### Full Terminal-Bench 2.1
```bash
PYTHONPATH=/Users/anton/Ouroboros/repo \
harbor run \
--dataset terminal-bench/terminal-bench-2-1 \
--agent-import-path devtools.benchmarks.terminal_bench.harbor_installed_agent:OuroborosTerminalBenchAgent \
--model ouroboros-gpt-5.5-tb21-full \
--agent-kwarg ouroboros_model=openai/gpt-5.5 \
--agent-kwarg install_timeout_sec=1200 \
--agent-kwarg server_start_timeout_sec=240 \
--agent-setup-timeout-multiplier 4 \
--environment-build-timeout-multiplier 4 \
--n-concurrent 1 \
--yes \
--force-build
```
## Model Selection
Harbor's `--model` is metadata for the Harbor result.
The actual Ouroboros model is passed via:
```bash
--agent-kwarg ouroboros_model=<provider/model>
```
Examples:
```bash
--agent-kwarg ouroboros_model=openai/gpt-5.5
--agent-kwarg ouroboros_model=google/gemini-3.5-flash
--agent-kwarg ouroboros_model=anthropic/claude-opus-4-7
```
The adapter sets:
```text
OUROBOROS_MODEL
OUROBOROS_MODEL_CODE
OUROBOROS_MODEL_LIGHT
```
to that value inside the container.
## Trace Locations
For each Harbor trial:
```text
<trial>/agent/ouroboros-data/
```
contains the fresh in-container Ouroboros data directory.
Useful files:
```text
<trial>/agent/ouroboros-data/logs/events.jsonl
<trial>/agent/ouroboros-data/logs/progress.jsonl
<trial>/agent/ouroboros-data/logs/supervisor.jsonl
<trial>/agent/ouroboros-data/state/headless_tasks/<task_id>/data/logs/tools.jsonl
<trial>/agent/ouroboros-task-result.json
<trial>/agent/ouroboros-run.jsonl
<trial>/agent/ouroboros-run-summary.json
<trial>/verifier/test-stdout.txt
<trial>/verifier/reward.txt
```
Heavy files usually come from:
```text
<trial>/agent/ouroboros-data/task_results/artifacts/<task_id>/workspace.patch
<trial>/agent/ouroboros-data/task_results/artifacts/<task_id>/workspace_patch.json
```
Those can be omitted when creating logs-only bundles.
## Known Infrastructure Notes
- Old task images with Python 3.9 require adapter-installed Python 3.12 via
`uv`; this is handled automatically.
- Some task Docker builds need more than 600 seconds; use
`--environment-build-timeout-multiplier`.
- Some tasks still hit Harbor `AgentTimeoutError`; verifier can still produce a
reward if the workspace has enough final state.
- `RuntimeError` from the adapter should not be used for ordinary Ouroboros
`status=failed`; the adapter records task status and returns control so Harbor
can run the verifier.
## Files To Share With Developers
Minimum:
```text
repo/devtools/benchmarks/terminal_bench/harbor_installed_agent.py
```
Recommended:
```text
repo/devtools/benchmarks/terminal_bench/README.md
```
Useful example result:
```text
data/harbor_jobs/ouroboros_v650_tb21_smoke_gpt55/2026-05-29__00-39-23/result.json
```

View file

@ -0,0 +1 @@
"""Terminal-Bench Harbor adapter."""

View file

@ -0,0 +1,729 @@
"""Harbor installed-agent entrypoint for evaluating full Ouroboros in Terminal-Bench.
This adapter intentionally does not translate Ouroboros decisions into shell
commands. Harbor starts a task container, this class installs Ouroboros inside
that container, starts the normal Ouroboros server/supervisor, and submits the
Terminal-Bench instruction as an external workspace task rooted at ``/app``.
"""
from __future__ import annotations
import json
import logging
import os
import shlex
import shutil
import tempfile
import textwrap
import time
from pathlib import Path
from typing import Any
try: # Harbor is an optional benchmark dependency.
from harbor.agents.installed.base import BaseInstalledAgent
from harbor.environments.base import BaseEnvironment
from harbor.models.agent.context import AgentContext
except Exception: # pragma: no cover - exercised when Harbor is absent.
BaseInstalledAgent = object # type: ignore[assignment]
BaseEnvironment = Any # type: ignore[assignment]
AgentContext = Any # type: ignore[assignment]
_CONTAINER_SRC = "/opt/ouroboros-src"
_CONTAINER_VENV = "/opt/ouroboros-venv"
_CONTAINER_DATA = "/logs/agent/ouroboros-data"
_CONTAINER_WORKSPACE = "/app"
_SERVER_URL = "http://127.0.0.1:8765"
_CONTAINER_SECRET_OPT_IN = "OUROBOROS_BENCH_ALLOW_CONTAINER_SECRETS"
_SECRET_ENV_KEYS = frozenset({
"ANTHROPIC_API_KEY",
"CLOUDRU_FOUNDATION_MODELS_API_KEY",
"GIGACHAT_CREDENTIALS",
"GIGACHAT_PASSWORD",
"GIGACHAT_USER",
"OPENAI_API_KEY",
"OPENAI_COMPATIBLE_API_KEY",
"OPENROUTER_API_KEY",
})
log = logging.getLogger(__name__)
def _workspace_root() -> Path:
return Path(__file__).resolve().parents[4]
def _default_host_settings_path() -> Path:
return Path(os.environ.get("OUROBOROS_SETTINGS_PATH") or _workspace_root() / "data" / "settings.json")
def _json_load(path: Path) -> dict[str, Any]:
try:
return json.loads(path.read_text(encoding="utf-8"))
except Exception:
return {}
def _repo_root() -> Path:
return Path(__file__).resolve().parents[3]
def _copy_clean_source(source: Path, target: Path) -> None:
excluded_dirs = {
".git",
".mypy_cache",
".pytest_cache",
".ruff_cache",
".venv",
"__pycache__",
"build",
"data",
"data_evaluated",
"dist",
"node-standalone",
"node_modules",
"python-standalone",
"venv",
}
excluded_suffixes = {".key", ".pem", ".pfx", ".p12", ".pyc", ".pyo"}
excluded_names = {
".DS_Store",
".env",
".env.dev",
".env.development",
".env.example",
".env.local",
".env.production",
".env.staging",
".env.test",
".git-credentials",
".netrc",
".npmrc",
".pypirc",
".release_notes.md",
"credentials.json",
"aws-credentials.json",
"gcp-service-account.json",
"id_ed25519",
"id_rsa",
"repo.bundle",
"repo_bundle_manifest.json",
"service-account.json",
"secrets.ini",
"secrets.json",
"secrets.toml",
"secrets.yaml",
}
def ignore(_: str, names: list[str]) -> set[str]:
ignored: set[str] = set()
for name in names:
if name in excluded_dirs or name in excluded_names:
ignored.add(name)
continue
if any(name.endswith(suffix) for suffix in excluded_suffixes):
ignored.add(name)
return ignored
shutil.copytree(source, target, ignore=ignore, symlinks=True)
class OuroborosTerminalBenchAgent(BaseInstalledAgent):
"""Install and run full Ouroboros inside the Terminal-Bench task container."""
SUPPORTS_WINDOWS = False
def __init__(
self,
logs_dir: Path,
model_name: str = "ouroboros-inside",
*args: Any,
**kwargs: Any,
) -> None:
workspace_dir = str(kwargs.pop("workspace_dir", _CONTAINER_WORKSPACE))
host_settings_path = str(kwargs.pop("host_settings_path", ""))
install_timeout_sec = int(kwargs.pop("install_timeout_sec", 900))
server_start_timeout_sec = int(kwargs.pop("server_start_timeout_sec", 180))
task_timeout_sec = kwargs.pop("task_timeout_sec", None)
max_workers = int(kwargs.pop("max_workers", 1))
runtime_mode = str(kwargs.pop("runtime_mode", "pro"))
review_enforcement = str(kwargs.pop("review_enforcement", "advisory"))
ouroboros_model = str(kwargs.pop("ouroboros_model", ""))
try:
super().__init__(*args, logs_dir=logs_dir, model_name=model_name, **kwargs)
except TypeError:
super().__init__()
self.logs_dir = Path(logs_dir)
self.model_name = model_name
self.workspace_dir = workspace_dir
self.host_settings_path = Path(
host_settings_path
or os.environ.get("OUROBOROS_SETTINGS_PATH")
or _default_host_settings_path()
).expanduser()
self.install_timeout_sec = int(install_timeout_sec)
self.server_start_timeout_sec = int(server_start_timeout_sec)
self.task_timeout_sec = (
int(task_timeout_sec)
if task_timeout_sec is not None and int(task_timeout_sec) > 0
else None
)
self.max_workers = int(max_workers)
self.runtime_mode = runtime_mode
self.review_enforcement = review_enforcement
self.ouroboros_model = ouroboros_model
self._run_summary: dict[str, Any] = {}
@staticmethod
def name() -> str:
return "Ouroboros Installed"
def version(self) -> str | None:
return "0.2.0"
def _host_settings(self) -> dict[str, Any]:
return _json_load(self.host_settings_path)
def _container_secret_injection_allowed(self, settings: dict[str, Any]) -> bool:
value = os.environ.get(_CONTAINER_SECRET_OPT_IN)
if value is None:
value = settings.get(_CONTAINER_SECRET_OPT_IN)
return str(value or "").strip().lower() in {"1", "true", "yes", "allow"}
def _available_host_secret_keys(self, settings: dict[str, Any]) -> list[str]:
keys: list[str] = []
for key in sorted(_SECRET_ENV_KEYS):
if str(os.environ.get(key) or settings.get(key) or "").strip():
keys.append(key)
return keys
def _enforce_container_secret_policy(self, env: dict[str, str]) -> None:
settings = self._host_settings()
blocked = self._available_host_secret_keys(settings)
if blocked and not self._container_secret_injection_allowed(settings):
names = ", ".join(blocked)
raise RuntimeError(
"Terminal-Bench installed-container mode refuses to inject long-lived provider "
f"credentials into task containers by default ({names}). Use a host-mediated LLM "
"bridge when available, or set OUROBOROS_BENCH_ALLOW_CONTAINER_SECRETS=1 only for "
"trusted local smoke runs where the task container and logs are under operator control."
)
if not blocked and not any(key in env for key in _SECRET_ENV_KEYS):
return
def _container_env(self) -> dict[str, str]:
settings = self._host_settings()
allow_secrets = self._container_secret_injection_allowed(settings)
keys = [
"OPENAI_BASE_URL",
"OPENAI_COMPATIBLE_BASE_URL",
"CLOUDRU_FOUNDATION_MODELS_BASE_URL",
"GIGACHAT_SCOPE",
"GIGACHAT_BASE_URL",
"GIGACHAT_VERIFY_SSL_CERTS",
"GIGACHAT_PROFANITY_CHECK",
"OUROBOROS_MODEL",
"OUROBOROS_MODEL_CODE",
"OUROBOROS_MODEL_LIGHT",
"OUROBOROS_MODEL_FALLBACK",
"OUROBOROS_WEBSEARCH_MODEL",
"OUROBOROS_SCOPE_REVIEW_MODEL",
"OUROBOROS_EFFORT_TASK",
"OUROBOROS_RETURN_REASONING",
"TOTAL_BUDGET",
"OUROBOROS_PER_TASK_COST_USD",
"OUROBOROS_SOFT_TIMEOUT_SEC",
"OUROBOROS_HARD_TIMEOUT_SEC",
"OUROBOROS_TOOL_TIMEOUT_SEC",
]
if allow_secrets:
keys.extend(sorted(_SECRET_ENV_KEYS))
env: dict[str, str] = {}
for key in keys:
value = os.environ.get(key)
if value is None:
value = settings.get(key)
if value not in (None, ""):
env[key] = str(value)
if self.ouroboros_model:
env["OUROBOROS_MODEL"] = self.ouroboros_model
env["OUROBOROS_MODEL_CODE"] = self.ouroboros_model
env["OUROBOROS_MODEL_LIGHT"] = self.ouroboros_model
env.update(
{
"OUROBOROS_REPO_DIR": _CONTAINER_SRC,
"OUROBOROS_DATA_DIR": _CONTAINER_DATA,
"OUROBOROS_SETTINGS_PATH": f"{_CONTAINER_DATA}/settings.json",
"OUROBOROS_PID_FILE": "/logs/agent/ouroboros.pid",
"OUROBOROS_PORT_FILE": f"{_CONTAINER_DATA}/state/server_port",
"OUROBOROS_SERVER_HOST": "127.0.0.1",
"OUROBOROS_SERVER_PORT": "8765",
"OUROBOROS_WORKER_START_METHOD": "spawn",
"OUROBOROS_RUNTIME_MODE": self.runtime_mode,
"OUROBOROS_REVIEW_ENFORCEMENT": self.review_enforcement,
"OUROBOROS_MAX_WORKERS": str(self.max_workers),
"PYTHONUNBUFFERED": "1",
}
)
return env
async def _append_log(self, environment: BaseEnvironment, message: str) -> None:
safe = json.dumps(message)
await environment.exec(
command=f"mkdir -p /logs/agent && python3 - <<'PY'\n"
"from pathlib import Path\n"
f"Path('/logs/agent/ouroboros-install.log').open('a', encoding='utf-8').write({safe} + '\\n')\n"
"PY",
user="root",
)
async def _upload_source(self, environment: BaseEnvironment) -> None:
source = _repo_root()
with tempfile.TemporaryDirectory(prefix="ouroboros-tb-src-") as tmp:
clean = Path(tmp) / "repo"
_copy_clean_source(source, clean)
await environment.exec(
command=f"rm -rf {_CONTAINER_SRC} && mkdir -p {_CONTAINER_SRC}",
user="root",
)
await environment.upload_dir(clean, _CONTAINER_SRC)
async def install(self, environment: BaseEnvironment) -> None:
started = time.monotonic()
await self._append_log(environment, "install: starting source upload")
await self._upload_source(environment)
await self._append_log(environment, "install: source uploaded")
install_cmd = textwrap.dedent(
f"""
set -euo pipefail
mkdir -p /logs/agent {_CONTAINER_DATA}/logs {_CONTAINER_DATA}/state
{{
echo "install: installing system dependencies"
if command -v apt-get >/dev/null 2>&1; then
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get install -y --no-install-recommends git curl bash ca-certificates procps python3 python3-venv python3-pip
elif command -v apk >/dev/null 2>&1; then
apk add --no-cache git curl bash ca-certificates procps python3 py3-pip py3-virtualenv
elif command -v yum >/dev/null 2>&1; then
yum install -y git curl bash ca-certificates procps python3 python3-pip
else
echo "install: no known package manager; assuming required tools already exist"
fi
PYTHON_BIN="$(command -v python3 || command -v python)"
PY_OK="$("$PYTHON_BIN" - <<'PY'
import sys
print(1 if sys.version_info >= (3, 10) else 0)
PY
)"
if [ "$PY_OK" != "1" ]; then
echo "install: system Python is too old; installing Python 3.12 with uv"
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
uv python install 3.12
PYTHON_BIN="$(uv python find 3.12)"
fi
echo "install: using $PYTHON_BIN"
"$PYTHON_BIN" -m venv {_CONTAINER_VENV} || {{
"$PYTHON_BIN" -m pip install --break-system-packages --user virtualenv || "$PYTHON_BIN" -m pip install --user virtualenv
"$PYTHON_BIN" -m virtualenv {_CONTAINER_VENV}
}}
. {_CONTAINER_VENV}/bin/activate
python -m pip install --upgrade pip setuptools wheel
python -m pip install -r {_CONTAINER_SRC}/requirements.txt
python -m pip install -e {_CONTAINER_SRC} --no-deps
chmod -R a+rX {_CONTAINER_SRC} {_CONTAINER_VENV} /logs/agent
{_CONTAINER_VENV}/bin/python -c 'import importlib.metadata; print("ouroboros", importlib.metadata.version("ouroboros"))'
echo "install: complete"
}} 2>&1 | tee -a /logs/agent/ouroboros-install.log
"""
).strip()
await self.exec_as_root(
environment,
command=install_cmd,
timeout_sec=self.install_timeout_sec,
)
elapsed = time.monotonic() - started
await self._append_log(environment, f"install: elapsed_sec={elapsed:.1f}")
async def _ensure_workspace_git_root(self, environment: BaseEnvironment) -> None:
workspace_dir = shlex.quote(self.workspace_dir)
command = textwrap.dedent(
f"""
set -euo pipefail
workspace_dir={workspace_dir}
cd "$workspace_dir"
if git rev-parse --show-toplevel >/tmp/ouroboros-git-root 2>/dev/null; then
root="$(cat /tmp/ouroboros-git-root)"
if [ "$root" != "$workspace_dir" ]; then
echo "workspace git root is $root, expected $workspace_dir" >&2
exit 2
fi
else
git init
git config user.email ouroboros-bench@example.invalid
git config user.name "Ouroboros Bench"
fi
"""
).strip()
result = await environment.exec(command=command, cwd=self.workspace_dir, timeout_sec=60)
if result.return_code != 0:
raise RuntimeError(
f"failed to prepare {self.workspace_dir} as git workspace: "
f"stdout={result.stdout!r} stderr={result.stderr!r}"
)
async def _resolve_workspace_dir(self, environment: BaseEnvironment) -> None:
"""Use /app when present, but support tasks whose Dockerfile uses /workspace."""
requested = self.workspace_dir
quoted_requested = shlex.quote(requested)
result = await environment.exec(command=f"test -d {quoted_requested}", timeout_sec=10)
if result.return_code == 0:
return
if requested == _CONTAINER_WORKSPACE:
fallback = await environment.exec(command="test -d /workspace", timeout_sec=10)
if fallback.return_code == 0:
self.workspace_dir = "/workspace"
await self._append_log(environment, "workspace: /app missing, using /workspace")
return
create = await environment.exec(command=f"mkdir -p {quoted_requested}", user="root", timeout_sec=10)
if create.return_code != 0:
raise RuntimeError(
f"failed to create workspace {requested}: stdout={create.stdout!r} stderr={create.stderr!r}"
)
await self._append_log(environment, f"workspace: created {requested}")
async def _start_server(self, environment: BaseEnvironment, env: dict[str, str]) -> None:
start_cmd = textwrap.dedent(
f"""
set -euo pipefail
mkdir -p {_CONTAINER_DATA}/logs {_CONTAINER_DATA}/state /logs/agent
rm -f /logs/agent/ouroboros.pid
cd {_CONTAINER_SRC}
nohup {_CONTAINER_VENV}/bin/python server.py --host 127.0.0.1 --port 8765 \
> /logs/agent/ouroboros-server.stdout.log \
2> /logs/agent/ouroboros-server.stderr.log &
echo "$!" > /logs/agent/ouroboros.pid
"""
).strip()
result = await environment.exec(command=start_cmd, env=env, timeout_sec=30)
if result.return_code != 0:
raise RuntimeError(f"failed to start Ouroboros server: {result.stdout}\n{result.stderr}")
wait_cmd = textwrap.dedent(
f"""
{_CONTAINER_VENV}/bin/python - <<'PY'
import json
import pathlib
import sys
import time
import urllib.request
deadline = time.time() + {self.server_start_timeout_sec}
last_error = ""
while time.time() < deadline:
try:
with urllib.request.urlopen("{_SERVER_URL}/api/state", timeout=5) as resp:
data = json.loads(resp.read().decode("utf-8"))
pathlib.Path("/logs/agent/ouroboros-state.json").write_text(
json.dumps(data, ensure_ascii=False, indent=2),
encoding="utf-8",
)
if data.get("supervisor_ready"):
print(json.dumps({{"ready": True, "state": data}}, ensure_ascii=False))
sys.exit(0)
last_error = "server responded but supervisor_ready=false"
except Exception as exc:
last_error = repr(exc)
time.sleep(2)
print(json.dumps({{"ready": False, "error": last_error}}, ensure_ascii=False))
sys.exit(1)
PY
"""
).strip()
result = await environment.exec(command=wait_cmd, env=env, timeout_sec=self.server_start_timeout_sec + 20)
if result.return_code != 0:
raise RuntimeError(f"Ouroboros server did not become ready: {result.stdout}\n{result.stderr}")
async def _network_preflight(self, environment: BaseEnvironment, env: dict[str, str]) -> None:
provider_url = ""
provider_name = ""
if env.get("OPENROUTER_API_KEY"):
provider_url = "https://openrouter.ai/api/v1/models"
provider_name = "openrouter"
elif env.get("OPENAI_API_KEY"):
provider_url = "https://api.openai.com/v1/models"
provider_name = "openai"
elif env.get("ANTHROPIC_API_KEY"):
provider_url = "https://api.anthropic.com/v1/models"
provider_name = "anthropic"
elif env.get("CLOUDRU_FOUNDATION_MODELS_API_KEY"):
provider_url = (env.get("CLOUDRU_FOUNDATION_MODELS_BASE_URL") or "https://foundation-models.api.cloud.ru/v1").rstrip("/") + "/models"
provider_name = "cloudru"
elif env.get("GIGACHAT_CREDENTIALS") or (env.get("GIGACHAT_USER") and env.get("GIGACHAT_PASSWORD")):
provider_url = (env.get("GIGACHAT_BASE_URL") or "https://gigachat.devices.sberbank.ru/api/v1").rstrip("/") + "/models"
provider_name = "gigachat"
if not provider_url:
(self.logs_dir / "network-preflight.txt").write_text(
"provider preflight skipped: no provider API key was injected; "
"Ouroboros runtime will surface provider configuration errors.\n",
encoding="utf-8",
)
return
command = textwrap.dedent(
f"""
python3 - <<'PY'
import sys
import urllib.error
import urllib.request
req = urllib.request.Request({provider_url!r}, method="GET")
try:
with urllib.request.urlopen(req, timeout=10) as resp:
print({provider_name!r} + "_preflight_status", resp.status)
sys.exit(0 if 200 <= resp.status < 500 else 1)
except urllib.error.HTTPError as exc:
print({provider_name!r} + "_preflight_status", exc.code)
sys.exit(0 if 200 <= exc.code < 500 else 1)
except Exception as exc:
print({provider_name!r} + "_preflight_error", type(exc).__name__)
sys.exit(1)
PY
"""
).strip()
result = await environment.exec(command=command, timeout_sec=20)
(self.logs_dir / "network-preflight.txt").write_text(
f"stdout:\n{result.stdout or ''}\nstderr:\n{result.stderr or ''}\nreturn_code={result.return_code}\n",
encoding="utf-8",
)
if result.return_code != 0:
raise RuntimeError(f"container cannot reach configured provider endpoint ({provider_name})")
async def _run_ouroboros_task(self, environment: BaseEnvironment, env: dict[str, str]) -> dict[str, Any]:
workspace_root = json.dumps(self.workspace_dir)
runner = textwrap.dedent(
f"""
import json
import os
import pathlib
import sys
import time
import urllib.parse
import urllib.request
instruction = pathlib.Path("/logs/agent/instruction.txt").read_text(encoding="utf-8")
started = time.time()
run_log = pathlib.Path("/logs/agent/ouroboros-run.jsonl")
stderr_log = pathlib.Path("/logs/agent/ouroboros-run.stderr.log")
task_id_path = pathlib.Path("/logs/agent/ouroboros-current-task-id.txt")
def api(method, path, body=None, timeout=30):
data = None
headers = {{"Accept": "application/json"}}
if body is not None:
data = json.dumps(body, ensure_ascii=False).encode("utf-8")
headers["Content-Type"] = "application/json"
req = urllib.request.Request("{_SERVER_URL}" + path, data=data, headers=headers, method=method)
with urllib.request.urlopen(req, timeout=timeout) as resp:
raw = resp.read().decode("utf-8", errors="replace")
return json.loads(raw) if raw.strip() else {{}}
def emit(event):
with run_log.open("a", encoding="utf-8") as f:
f.write(json.dumps(event, ensure_ascii=False) + "\\n")
created = api("POST", "/api/tasks", {{
"description": instruction,
"workspace_root": {workspace_root},
"workspace_mode": "external",
"memory_mode": "empty",
"actor_id": "harbor-terminal-bench",
"source": "terminal-bench",
"metadata": {{"source": "terminal-bench", "delegation_role": "root"}},
}})
task_id = str(created.get("task_id") or "")
if not task_id:
stderr_log.write_text(f"task creation did not return task_id: {{created!r}}\\n", encoding="utf-8")
print(json.dumps({{"return_code": 1, "elapsed_sec": round(time.time() - started, 3), "status": "create_failed"}}))
sys.exit(1)
task_id_path.write_text(task_id, encoding="utf-8")
emit({{"type": "task_created", "task_id": task_id, "data": created}})
latest = {{}}
seen_events = set()
final_statuses = {{"completed", "failed", "cancelled", "rejected_duplicate"}}
while True:
result = api("GET", "/api/tasks/" + urllib.parse.quote(task_id), timeout=30)
for event in result.get("events") or []:
key = (str(event.get("type") or ""), str(event.get("ts") or event.get("seq") or ""))
if key in seen_events:
continue
seen_events.add(key)
emit({{"type": "task_event", "task_id": task_id, "data": event}})
status = str(result.get("status") or "")
if status in final_statuses:
latest = result
pathlib.Path("/logs/agent/ouroboros-task-result.json").write_text(
json.dumps(latest, ensure_ascii=False, indent=2),
encoding="utf-8",
)
emit({{"type": "final", "task_id": task_id, "result": latest}})
break
time.sleep(2)
status = str(latest.get("status") or "")
summary = {{
"return_code": 0,
"task_status_code": 0 if status == "completed" else 1,
"elapsed_sec": round(time.time() - started, 3),
"task_id": latest.get("task_id") or latest.get("id"),
"status": status,
"cost_usd": latest.get("cost_usd"),
"prompt_tokens": latest.get("prompt_tokens"),
"completion_tokens": latest.get("completion_tokens"),
"total_rounds": latest.get("total_rounds"),
}}
pathlib.Path("/logs/agent/ouroboros-run-summary.json").write_text(
json.dumps(summary, ensure_ascii=False, indent=2),
encoding="utf-8",
)
print(json.dumps(summary, ensure_ascii=False))
sys.exit(0)
"""
).strip()
command = "cat > /tmp/run_ouroboros_task.py <<'PY'\n" + runner + "\nPY\n" + (
f"{_CONTAINER_VENV}/bin/python /tmp/run_ouroboros_task.py"
)
result = await environment.exec(
command=command,
env=env,
cwd=self.workspace_dir,
timeout_sec=(self.task_timeout_sec + 60 if self.task_timeout_sec is not None else None),
)
if result.return_code != 0:
raise RuntimeError(f"Ouroboros task runner failed: {result.stdout}\n{result.stderr}")
try:
return json.loads((result.stdout or "").strip().splitlines()[-1])
except Exception:
return {"raw_stdout": result.stdout or "", "raw_stderr": result.stderr or ""}
async def _stop_server(self, environment: BaseEnvironment) -> None:
await environment.exec(
command=(
"if [ -s /logs/agent/ouroboros-current-task-id.txt ]; then "
"TASK_ID=$(cat /logs/agent/ouroboros-current-task-id.txt); "
"export TASK_ID; "
f"{_CONTAINER_VENV}/bin/python - <<'PY' || true\n"
"import os, urllib.parse, urllib.request\n"
"task_id = os.environ.get('TASK_ID', '')\n"
"if task_id:\n"
f" urllib.request.urlopen(urllib.request.Request('{_SERVER_URL}/api/tasks/' + urllib.parse.quote(task_id) + '/cancel', data=b'{{}}', method='POST'), timeout=5).read()\n"
"PY\n"
"fi; "
"if [ -f /logs/agent/ouroboros.pid ]; then "
"kill $(cat /logs/agent/ouroboros.pid) 2>/dev/null || true; "
"fi; "
"pkill -TERM -f '/opt/ouroboros-src|/opt/ouroboros-venv/bin/ouroboros' 2>/dev/null || true"
),
timeout_sec=10,
)
async def _capture_current_task_summary(self, environment: BaseEnvironment) -> None:
"""Persist best-effort task state if Harbor cancels agent.run mid-exec."""
command = textwrap.dedent(
f"""
if [ ! -s /logs/agent/ouroboros-current-task-id.txt ]; then
exit 0
fi
{_CONTAINER_VENV}/bin/python - <<'PY'
import json
import pathlib
import time
import urllib.parse
import urllib.request
task_id = pathlib.Path("/logs/agent/ouroboros-current-task-id.txt").read_text(encoding="utf-8").strip()
if not task_id:
raise SystemExit(0)
try:
with urllib.request.urlopen("{_SERVER_URL}/api/tasks/" + urllib.parse.quote(task_id), timeout=10) as resp:
latest = json.loads(resp.read().decode("utf-8", errors="replace"))
except Exception as exc:
pathlib.Path("/logs/agent/ouroboros-run.stderr.log").open("a", encoding="utf-8").write(
"best-effort task summary failed: " + repr(exc) + "\\n"
)
raise SystemExit(0)
pathlib.Path("/logs/agent/ouroboros-task-result.json").write_text(
json.dumps(latest, ensure_ascii=False, indent=2),
encoding="utf-8",
)
status = str(latest.get("status") or "")
summary = {{
"return_code": 0,
"task_status_code": 0 if status == "completed" else 1,
"elapsed_sec": None,
"task_id": latest.get("task_id") or latest.get("id") or task_id,
"status": status,
"cost_usd": latest.get("cost_usd"),
"prompt_tokens": latest.get("prompt_tokens"),
"completion_tokens": latest.get("completion_tokens"),
"total_rounds": latest.get("total_rounds"),
"captured_after_cancellation": True,
"captured_at": time.time(),
}}
pathlib.Path("/logs/agent/ouroboros-run-summary.json").write_text(
json.dumps(summary, ensure_ascii=False, indent=2),
encoding="utf-8",
)
with pathlib.Path("/logs/agent/ouroboros-run.jsonl").open("a", encoding="utf-8") as f:
f.write(json.dumps({{"type": "final", "task_id": task_id, "result": latest, "captured_after_cancellation": True}}, ensure_ascii=False) + "\\n")
PY
"""
).strip()
await environment.exec(command=command, timeout_sec=30)
async def run(self, instruction: str, environment: BaseEnvironment, context: AgentContext) -> None:
self.logs_dir.mkdir(parents=True, exist_ok=True)
(self.logs_dir / "instruction.txt").write_text(instruction, encoding="utf-8")
await environment.upload_file(self.logs_dir / "instruction.txt", "/logs/agent/instruction.txt")
env = self._container_env()
try:
self._enforce_container_secret_policy(env)
await self._network_preflight(environment, env)
await self._resolve_workspace_dir(environment)
await self._ensure_workspace_git_root(environment)
await self._start_server(environment, env)
self._run_summary = await self._run_ouroboros_task(environment, env)
finally:
try:
await self._capture_current_task_summary(environment)
except Exception as exc:
(getattr(self, "logger", None) or log).warning("Failed to capture in-container Ouroboros task summary: %s", exc)
try:
await self._stop_server(environment)
except Exception as exc:
(getattr(self, "logger", None) or log).warning("Failed to stop in-container Ouroboros cleanly: %s", exc)
cost = self._run_summary.get("cost_usd")
prompt_tokens = self._run_summary.get("prompt_tokens")
completion_tokens = self._run_summary.get("completion_tokens")
context.cost_usd = float(cost) if cost is not None else None
context.n_input_tokens = int(prompt_tokens) if prompt_tokens is not None else None
context.n_output_tokens = int(completion_tokens) if completion_tokens is not None else None
context.metadata = {
"adapter_mode": "installed_ouroboros",
"workspace_dir": self.workspace_dir,
"runtime_mode": self.runtime_mode,
"review_enforcement": self.review_enforcement,
"summary": self._run_summary,
}
InstalledOuroborosTerminalBenchAgent = OuroborosTerminalBenchAgent
__all__ = ["OuroborosTerminalBenchAgent", "InstalledOuroborosTerminalBenchAgent"]

View file

@ -0,0 +1,97 @@
#!/usr/bin/env python3
"""Build a Harbor command for a one-task Terminal-Bench smoke run."""
from __future__ import annotations
import argparse
import json
import pathlib
import shlex
import subprocess
import sys
if __package__ in {None, ""}:
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[3]))
from devtools.benchmarks.common.run_roots import ensure_outside_repo, repo_root_from_devtools, run_root as default_run_root
AGENT_IMPORT = "devtools.benchmarks.terminal_bench.harbor_installed_agent:OuroborosTerminalBenchAgent"
def harbor_command(
*,
task_name: str,
model: str,
run_root: pathlib.Path,
dataset: str = "terminal-bench/terminal-bench-2-1",
harbor_bin: str = "harbor",
execute: bool = False,
) -> list[str]:
cmd = [
harbor_bin,
"run",
"--dataset",
dataset,
"--include-task-name",
task_name,
"--agent-import-path",
AGENT_IMPORT,
"--model",
f"ouroboros-{model.replace('/', '-')}",
"--agent-kwarg",
f"ouroboros_model={model}",
"--agent-kwarg",
"install_timeout_sec=1200",
"--agent-kwarg",
"server_start_timeout_sec=240",
"--agent-setup-timeout-multiplier",
"4",
"--environment-build-timeout-multiplier",
"4",
"--n-concurrent",
"1",
"--n-tasks",
"1",
"--output-dir",
str(run_root),
"--yes",
]
if execute:
cmd.append("--force-build")
return cmd
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--task", default="terminal-bench/regex-log")
parser.add_argument("--model", default="openai/gpt-5.5")
parser.add_argument("--dataset", default="terminal-bench/terminal-bench-2-1")
parser.add_argument("--harbor-bin", default="harbor")
parser.add_argument("--run-root", default="")
parser.add_argument("--execute", action="store_true")
args = parser.parse_args()
run_root = ensure_outside_repo(
pathlib.Path(args.run_root).expanduser() if args.run_root else default_run_root("terminal_bench"),
repo_root_from_devtools(),
)
cmd = harbor_command(
task_name=args.task,
model=args.model,
run_root=run_root,
dataset=args.dataset,
harbor_bin=args.harbor_bin,
execute=args.execute,
)
manifest = {"run_root": str(run_root), "cmd": cmd, "agent_import": AGENT_IMPORT}
run_root.mkdir(parents=True, exist_ok=True)
(run_root / "harbor_command.json").write_text(json.dumps(manifest, indent=2), encoding="utf-8")
print(shlex.join(cmd))
if not args.execute:
return 0
return subprocess.run(cmd, cwd=pathlib.Path(__file__).resolve().parents[3]).returncode
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -1,4 +1,4 @@
# Ouroboros v6.18.1 — Architecture & Reference
# Ouroboros v6.19.0-rc.1 — Architecture & Reference
This file is NOT a changelog. Version history lives in README.md, git tags, and commit log.
@ -106,6 +106,7 @@ server.py (Starlette+uvicorn) ← HTTP + WebSocket on configurable host:port (de
├── git_shell_policy.py ← Structural git argv classifiers for shell safety guards
├── protected_artifacts.py ← Task-contract protected artifact policy helpers for execute-only black-box references
├── shell_parse.py ← Shared shell argv/inline-command parser helpers used by guardrails without importing the tools package
├── workspace_executor.py ← Host-owned local/docker_exec workspace process backend, path mapping, executor traces, and executor service lifecycle
├── tool_capabilities.py ← SSOT for tool sets (core, parallel-safe, truncation, browser)
├── tool_access.py ← Tool API v2 policy matrix: ToolProfile × ResourceRoot × Operation
├── tool_policy.py ← Tool access policy and gating (imports from tool_capabilities)
@ -175,8 +176,7 @@ build_windows.ps1 ← Windows build (PyInstaller → .zip)
scripts/build_repo_bundle.py ← Builds `repo.bundle` + `repo_bundle_manifest.json` for packaged releases
scripts/run_external_review.py ← v5.1.2 dev-loop tool: invokes `ouroboros.tools.parallel_review.run_parallel_review` from outside the runtime against `git diff --cached`. Reads `~/Ouroboros/data/settings.json` for `OPENROUTER_API_KEY` / `OUROBOROS_REVIEW_MODELS` / `OUROBOROS_SCOPE_REVIEW_MODELS`, builds a minimal `ToolContext`, prints FULL raw triad+scope output (no truncation). Used to dry-run the same review pipeline `commit_reviewed` triggers before any actual commit. Output: stdout (and optional `--output PATH`). Not part of the runtime gate; review-exempt dev tool.
scripts/cleanup_test_pollution.py ← Dry-run-first cleanup utility for local test-pollution artifacts: known test skill state dirs, stale `__extension_imports`, and accidental `MagicMock`-named repo-root files. Use `--apply` only after inspecting planned removals.
scripts/swebench_cli_agent.py ← Helper that turns local checkout-backed SWE-bench rows into prediction JSONL via the CLI/headless task API.
scripts/terminal_bench_cli_agent.py ← Minimal Terminal-Bench BaseAgent bridge that delegates task solving through `ouroboros run` when the task workspace is mounted on the gateway host.
devtools/benchmarks/ ← Tracked operator benchmark tooling (ProgramBench, Terminal-Bench/Harbor, SWE-bench, SWE-bench Pro, OSWorld logs skeleton). It is reviewed when touched, is manifest-accounted by Atlas, is not imported by runtime core, and is not packaged as runtime app code.
packaging/cli/ ← Packaged CLI shell/cmd wrappers and user-local installer launchers copied into desktop artifacts
Dockerfile ← Docker image (web UI runtime)
```
@ -273,13 +273,30 @@ The CLI downloads patch artifacts through the task artifact endpoint, waits for
artifact finalization in `--patch` / `--patch-out` mode, and fails nonzero when
the patch is missing, empty, or failed. `--no-stream` suppresses live progress
but still waits; `--detach` is the explicit create-and-return mode.
Benchmark helper scripts likewise require clean per-instance local checkouts;
they do not reset or commit benchmark workspaces.
Benchmark devtools under `devtools/benchmarks/` require clean per-instance local
checkouts or official benchmark containers; they do not commit target
repositories. Broad scope/plan/deep-review packs list unrelated `devtools/`
files in the Atlas manifest without inlining every benchmark harness, while
touched `devtools/` files are fully included in triad/scope review. This is a
context-management rule, not an immune-system escape hatch.
Workspace mode is a tool-routing and blast-radius guard, not an OS sandbox.
Like OpenClaw's host workspace mode, absolute host paths are not a hard security
boundary unless a future Docker/SSH/remote sandbox is added around tool
execution. Do not grow ad-hoc shell parsing to approximate that sandbox.
boundary unless a Docker/SSH/remote backend is added around tool execution.
When task metadata contains a host-owned `executor_ref`, `run_command`,
`run_script`, and service tools route process execution through the declared
backend (`local` or `docker_exec`) only when the requested cwd is covered by an
executor path mapping. Unmapped task-drive, artifact-store, and user-files cwd
paths remain local host execution roots. File tools continue to operate on the
shared host workspace. `executor_ref.network=none` is enforced by the backend
transport for mapped backend executions, for example by requiring Docker
`NetworkMode=none`; LLM provider traffic remains outside the benchmark tool
environment. Executor-backed
foreground commands and services are also written to durable
`data/state/workspace_executor_processes/` records so server-side panic and
emergency cleanup can stop local process groups or Docker-side pidfile/service
processes even if the worker that started them has died. Do not grow ad-hoc
shell parsing to approximate that sandbox.
Project-local dependency installs are ordinary workspace work. In
`runtime_mode=pro`, system/global dependency installs may be attempted through
`run_command` and the safety supervisor when needed by the external workspace;
@ -445,6 +462,7 @@ finalization states.
│ │ ├── extension_companions.json ← Runtime snapshot for live extension companion processes
│ │ ├── extension_reconcile/ ← Worker-written extension reconcile markers consumed by the server lifespan pickup task
│ │ ├── review_continuations/ ← Per-task blocked-review continuation payloads (+ quarantined corrupt files under `corrupt/`)
│ │ ├── workspace_executor_processes/ ← Durable local/docker executor foreground/service cleanup records for panic/shutdown recovery
│ │ └── skills/ ← Phase 3 external-skill state plane (sibling of advisory_review.json, not shared)
│ │ └── <skill_name>/
│ │ ├── enabled.json ← {"enabled": bool, "updated_at": iso_ts}
@ -1271,8 +1289,11 @@ The panic sequence (in `server.py:_execute_panic_stop()`):
5. kill_all_tracked_subprocesses() ← os.killpg(SIGKILL) every tracked
│ foreground subprocess process group
│ (shell commands and ALL their children)
6. kill_workers(force=True) ← SIGTERM+SIGKILL all multiprocessing workers
7. os._exit(99) ← immediate hard exit, kills daemon threads
6. kill_all_foreground(data_dir) ← stop durable executor-backed foreground
│ local/docker processes
7. kill_all_services(data_dir) ← stop service and executor-service groups
8. kill_workers(force=True) ← SIGTERM+SIGKILL all multiprocessing workers
9. os._exit(99) ← immediate hard exit, kills daemon threads
```
Launcher handles exit code 99:
@ -1300,17 +1321,21 @@ use `start_new_session=True` via `_tracked_subprocess_run()` in
`ouroboros/tools/services.py::_start_service`, which starts each service with
`subprocess_new_group_kwargs()` and records it in the `_SERVICES` registry.
Both paths create a separate process group for each subprocess and its children.
Executor-backed workspace processes additionally record local pids, Docker
pidfiles, and service pids under `data/state/workspace_executor_processes/` so
panic can clean them up from the server process after worker death.
On panic or timeout, the entire process tree is killed via
`os.killpg(pgid, SIGKILL)` — no orphans possible, even for deeply nested
foreground shell/script/service subprocess trees.
Panic/emergency paths call `kill_all_tracked_subprocesses()` and
`kill_all_services()` without log finalization so emergency stop remains fast;
normal lifespan shutdown may pass a drive root to `kill_all_services(drive_root)`
to archive server-process service logs before removing live log files. Services
started inside worker tasks normally finalize in `loop.py` task cleanup; forced
worker termination kills the worker process tree and archives remaining task
service logs best-effort from `data/services/<task_id>/`.
`kill_all_foreground(data_dir)` plus `kill_all_services(data_dir)` without log
finalization so emergency stop remains fast; normal lifespan shutdown may pass a
drive root to `kill_all_services(drive_root)` to archive server-process service
logs before removing live log files. Services started inside worker tasks
normally finalize in `loop.py` task cleanup; forced worker termination kills the
worker process tree and archives remaining task service logs best-effort from
`data/services/<task_id>/`.
Active subprocesses are tracked in a thread-safe global set and cleaned up
automatically on completion or via `kill_all_tracked_subprocesses()` on panic.
@ -1409,7 +1434,7 @@ via `tests/test_contracts.py`.
|----------|------|-------------|
| `ToolContextProtocol` — workspace/task-aware minimum every tool handler relies on (attributes: `repo_dir`, `drive_root`, `budget_drive_root`, `pending_events`, `emit_progress_fn`, `current_chat_id`, `task_id`, `task_metadata`, `task_contract`, `workspace_root`, `workspace_mode`; methods: `repo_path`, `drive_path`, `drive_logs`, `active_repo_dir`, `is_workspace_mode`) | `ouroboros/contracts/tool_context.py` | `ouroboros.tools.registry.ToolContext` must satisfy it (duck-typed check + AST field/method parity) |
| `ToolEntryProtocol` + `GetToolsProtocol` — the tool-module ABI | `ouroboros/contracts/tool_abi.py` | Every entry returned by `ToolRegistry._entries` must satisfy `ToolEntryProtocol` |
| `api_v1` WS/HTTP envelopes — inbound: `ChatInbound`, `CommandInbound`; outbound WS: `ChatOutbound`, `PhotoOutbound`, `VideoOutbound`, `TypingOutbound`, `LogOutbound`, `ExtensionLifecycleOutbound`; HTTP: `HealthResponse`, `StateResponse` (Phase 2 adds `runtime_mode: str` and `skills_repo_configured: bool`; v5.11.0 adds `github_token_configured: bool`; v6.13.0 adds `context_mode: str`), `EvolutionStateSnapshot`, `SettingsNetworkMeta`, `SettingsMeta` (`custom_secret_keys` + setup contract metadata) | `ouroboros/gateway/contracts.py` | AST scans of `supervisor/message_bus.py` chat/media envelopes, `gateway/state.py::api_state`, `gateway/state.py::api_health`, `gateway/settings.py::_build_network_meta`, and `gateway/ws.py::ws_endpoint` inbound dispatch assert no un-declared keys leak out; `tests/test_contracts.py::test_state_response_declares_phase2_runtime_mode_keys` explicitly pins the Phase 2 fields and later additive state keys |
| `api_v1` WS/HTTP envelopes — inbound: `ChatInbound`, `CommandInbound`; outbound WS: `ChatOutbound`, `PhotoOutbound`, `VideoOutbound`, `TypingOutbound`, `LogOutbound`, `ExtensionLifecycleOutbound`; HTTP: `HealthResponse`, `StateResponse` (Phase 2 adds `runtime_mode: str` and `skills_repo_configured: bool`; v5.11.0 adds `github_token_configured: bool`; v6.13.0 adds `context_mode: str`), `TaskCreateRequest` + `ExecutorRef` for host-owned executor-backed external workspace tasks, `TaskCreateResponse`, `EvolutionStateSnapshot`, `SettingsNetworkMeta`, `SettingsMeta` (`custom_secret_keys` + setup contract metadata) | `ouroboros/gateway/contracts.py` | AST scans of `supervisor/message_bus.py` chat/media envelopes, `gateway/state.py::api_state`, `gateway/state.py::api_health`, `gateway/settings.py::_build_network_meta`, and `gateway/ws.py::ws_endpoint` inbound dispatch assert no un-declared keys leak out; `tests/test_contracts.py::test_state_response_declares_runtime_and_capability_keys` explicitly pins runtime/capability state keys, and `tests/test_contracts.py::test_task_create_request_declares_executor_ref_contract` pins the executor request surface |
| `chat_id_policy` — SSOT for A2A/synthetic chat-id filtering across message bus, history, memory, and consolidation | `ouroboros/contracts/chat_id_policy.py` | `tests/test_chat_id_policy.py` pins boundaries and human/transport positive ids |
| `task_contract` — canonical host-draft task objective/output/constraint/resource/deadline/workspace/lineage contract helpers (`build_task_contract`, `attach_task_contract`, `normalize_allowed_resources`, `normalize_resource_policy`) | `ouroboros/contracts/task_contract.py` | `tests/test_contracts.py::test_public_api_is_stable` pins the public helper names; task/outcome tests pin resource and resource-policy normalization plus contract propagation |
| `PluginAPI` (Phase 4, v1.3) + `ExtensionRegistrationError` + `FORBIDDEN_EXTENSION_SETTINGS` + `VALID_EXTENSION_PERMISSIONS` + `VALID_EXTENSION_ROUTE_METHODS` — the surface every `type: extension` skill's `plugin.py::register(api)` binds against (`register_tool`, `register_route`, `register_ws_handler`, `register_ui_tab`, `register_settings_section`, `register_supervised_task`, `register_companion_process`, `subscribe_event`, `get_skill_token`, `send_ws_message`, `on_unload`, `log`, `get_settings`, `get_state_dir`, `skill_job_dir`, `get_runtime_info`). `skill_job_dir(job_id)` creates isolated `jobs/<sanitized_id>-<hash>/{assets,output,tmp}` state folders so generation skills do not overwrite their own assets across jobs. `VALID_EXTENSION_PERMISSIONS` includes host-mediated permissions (`companion_process`, `supervised_task`, `subscribe_event`, `inject_chat`) that require review/owner grants as documented in CHECKLISTS.md. The `ExecutionMode` capability matrix (`MATRIX_CAPABILITIES` / `OUT_OF_PROCESS_UNAVAILABLE_CAPABILITIES` / `capability_available` / `available_capabilities`) is the SSOT for which side-effect surfaces an out-of-process child may use and is pinned by the contract test. | `ouroboros/contracts/plugin_api.py` | `tests/test_contracts.py::test_plugin_api_surface_is_frozen` pins the frozen method set; `tests/test_contracts.py::test_extension_route_methods_contract_matches_server_dispatch` pins the route-methods tuple; `tests/test_extension_loader.py::test_plugin_api_impl_matches_protocol` asserts the concrete `PluginAPIImpl` structurally satisfies the runtime-checkable Protocol |

View file

@ -111,6 +111,7 @@ or Intent/Scope checklists are.
| 13 | Changing LLM output-token budgets? | Grep the whole repo for `max_tokens`, `max_completion_tokens`, `_MAX_TOKENS`, and `max_toks`. Keep `docs/ARCHITECTURE.md` §LLM output token budgets and `tests/test_max_tokens_constants.py` in sync so main-loop, VLM, summaries, compaction, skill publish, and consciousness floors cannot drift independently. |
| 14 | Changing extension loader/dispatch or isolated deps? | Native-risk extension imports and tool/route/WS handlers must stay out-of-process. Add or run regression tests where a native-risk plugin aborts during import and the host survives, plus tool/route child-dispatch tests. Do not "fix" failures by importing native-risk plugin code in `server.py`. |
| 15 | Changing `supervisor/git_ops.py`, `launcher.py`, `server.py`, `ouroboros/tools/review_helpers.py`, `ouroboros/tools/git.py`, tests, or evolution scheduling/checkpoint code? | Prove two invariants before review spend: (1) pytest/preflight cannot mutate the live repo or live `data/` (`OUROBOROS_DATA_DIR` / `OUROBOROS_SETTINGS_PATH` must be isolated, and `OUROBOROS_MANAGED_BY_LAUNCHER` must not leak into test subprocesses); (2) autonomous restart/reset cannot erase active evolution work — it must either land a reviewed local commit or preserve a rescue/transaction recovery pointer and pause/stop the campaign. |
| 16 | Changing `devtools/benchmarks/`? | Confirm it preserves official benchmark boundaries: no replacement scoring, no benchmark-specific prompt/routing hacks, no generated benchmark outputs under `repo/`, no secrets printed or committed, and no runtime-core imports from `devtools/`. Touched `devtools` files are reviewable executable operator code, even though unrelated `devtools` files use Atlas `excluded_dir` coverage-manifest entries and stay compact in broad packs. |
Rule: read before write. Never reconstruct `VERSION`, `pyproject.toml`
`version`, or the README badge from memory — one stale reconstruction creates
@ -138,7 +139,7 @@ Used by `commit_reviewed` for all changes to the Ouroboros repository.
| # | item | what to check | severity when FAIL |
|---|------|---------------|--------------------|
| 1 | bible_compliance | Does the diff violate any BIBLE.md principle? | critical |
| 2 | development_compliance | Does it follow DEVELOPMENT.md patterns? Check explicitly: (a) naming conventions (snake_case modules/vars, PascalCase classes, UPPER_SNAKE_CASE constants); (b) entity type rules — Gateway classes contain ONLY transport, no business logic; Tool functions are thin wrappers; (c) module-size target stays near one context window (~1000 lines) with a hard fail above 1600 lines for non-grandfathered modules, method-size target stays under 150 lines with a hard fail above 300 lines, codebase-wide total Python function/method count stays under the smoke hard gate defined by `ouroboros/review.py::MAX_TOTAL_FUNCTIONS` (the literal value evolves with the codebase — consult the constant rather than hardcoding the number), and functions keep `<= 8` params; (d) no gratuitous abstract layers (P7 Minimalism); (e) new LLM calls go through the shared `LLMClient`/`llm.py` layer, not ad-hoc HTTP clients; (f) cognitive artifacts (identity.md, scratchpad, task reflections, review outputs) must NOT use hardcoded `[:N]` truncation — explicit omission notes required; (g) new `get_tools()` exports follow the ToolEntry pattern in registry.py; (h) provider independence — no change may make a core capability (agent loop, multi-model commit review, scope review, or memory/context flows) silently require a second provider or OpenRouter specifically, and every supported single direct provider (local, OpenAI, Anthropic, Cloud.ru, GigaChat) must keep its model AND review/scope slots self-fillable (see DEVELOPMENT.md "Provider Independence"). | critical |
| 2 | development_compliance | Does it follow DEVELOPMENT.md patterns? Check explicitly: (a) naming conventions (snake_case modules/vars, PascalCase classes, UPPER_SNAKE_CASE constants); (b) entity type rules — Gateway classes contain ONLY transport, no business logic; Tool functions are thin wrappers; (c) module-size target stays near one context window (~1000 lines) with a hard fail above 1600 lines for non-grandfathered modules, method-size target stays under 150 lines with a hard fail above 300 lines, runtime-code total Python function/method count stays under the smoke hard gate defined by `ouroboros/review.py::MAX_TOTAL_FUNCTIONS` (the literal value evolves with the codebase — consult the constant rather than hardcoding the number; `devtools/` is excluded from this health gate but reviewed when touched), and functions keep `<= 8` params; (d) no gratuitous abstract layers (P7 Minimalism); (e) new LLM calls go through the shared `LLMClient`/`llm.py` layer, not ad-hoc HTTP clients; (f) cognitive artifacts (identity.md, scratchpad, task reflections, review outputs) must NOT use hardcoded `[:N]` truncation — explicit omission notes required; (g) new `get_tools()` exports follow the ToolEntry pattern in registry.py; (h) provider independence — no change may make a core capability (agent loop, multi-model commit review, scope review, or memory/context flows) silently require a second provider or OpenRouter specifically, and every supported single direct provider (local, OpenAI, Anthropic, Cloud.ru, GigaChat) must keep its model AND review/scope slots self-fillable (see DEVELOPMENT.md "Provider Independence"). | critical |
| 3 | secrets_check | Are secrets, API keys, .env files, credentials present in the diff? | critical |
| 4 | code_quality | Careful code review: bugs, logic errors, crashes, regressions, race conditions, resource leaks? | critical |
| 5 | security_issues | Security vulnerabilities: injection, path traversal, secret leakage, unsafe operations? | critical |

View file

@ -168,7 +168,7 @@ Derived from P7 (Minimalism): entire codebase fits in one context window.
- Module hard gate: 1600 lines for non-grandfathered modules in `tests/test_smoke.py`. Grandfathered (`GRANDFATHERED_OVERSIZED_MODULES` in `ouroboros/review.py`): `llm.py`, `claude_advisory_review.py`, `review_state.py`, `server.py`, temporary v5.7.1 debt `git.py`, and temporary v6.15/v6.16 debt `extension_loader.py` (OOP extension parity plus worker->server companion reconcile crossed the gate; the registry-coupled `PluginAPIImpl`/loader split is the deferred follow-up) — split deferred until each surface stabilises, with `git.py` expected to pay down in the next tools pass.
- Method target: <150 lines. Crossing that line is a decomposition signal, not an automatic failure by itself.
- Method hard gate: 300 lines in `tests/test_smoke.py`.
- Codebase-wide function-count hard gate: enforced by `tests/test_smoke.py` against the value defined in `ouroboros/review.py::MAX_TOTAL_FUNCTIONS` (single source of truth — bump the constant when adding a feature with an explicit comment justifying the increase).
- Runtime-code function-count hard gate: enforced by `tests/test_smoke.py` against the value defined in `ouroboros/review.py::MAX_TOTAL_FUNCTIONS` (single source of truth — bump the constant when adding a feature with an explicit comment justifying the increase). Tracked `devtools/` operator code is excluded from this runtime health gate, but touched `devtools/` files are still fully reviewed.
- Function parameters: <8.
- Net complexity growth per cycle approaches zero.
- If a feature is not used in the current cycle — it is premature.
@ -382,12 +382,19 @@ Before every commit, verify the following:
#### Task Contract Resource Policy
- When a task contract declares `resource_policy.protected_artifacts`, enforce it as a typed affordance policy in every runtime mode: execute-only black-box references may be run, but byte reads, copy/hash/static introspection, tracing, and debugging against declared paths are blocked. Do not add benchmark-specific command gates.
#### Devtools And Benchmark Tooling
- `devtools/` is tracked operator code, not runtime core. It may contain benchmark harness adapters, smoke runners, and reproducibility helpers that should be versioned with Ouroboros, but runtime modules under `ouroboros/`, `server.py`, web modules, and build scripts must not import it.
- `devtools/` is not included in the Python runtime package discovery; it is repository-side operator tooling, not an installed dependency of the Ouroboros app.
- `devtools/` is not an immune-system bypass. If a commit touches `devtools/`, triad/scope reviewers inspect those touched files fully. Unrelated `devtools/` files use the Atlas `excluded_dir` disposition and stay coverage-manifest-only in broad packs so benchmark harness code does not drown normal core reviews.
- Benchmark adapters must preserve official task instructions, official scoring/evaluation commands, and official artifact formats. They may build predictions, launch official runners, normalize logs, or aggregate official outputs, but must not implement benchmark-specific prompt hacks, routing hacks, or replacement scoring.
- Generated benchmark runs, datasets, container outputs, logs, predictions, and submissions belong under `/Users/anton/Ouroboros/bench_runs/` or another explicit output root outside `repo/`, never under `devtools/`.
#### Light Mode External Deliverables
- `runtime_mode=light` is a self-modification boundary, not an OS sandbox. User-visible deliverables are allowed when they are outside the Ouroboros repo/control-plane.
- Preferred flow: `task_drive` for scratch, `artifact_store` for canonical deliverables, and `user_files` for the owner's visible copy (for example `Desktop/report.html`). `write_file(root=user_files)` and declared process `outputs` must register/copy canonical task artifacts. Rewrites of the same user-visible source keep the previous canonical artifact in non-manifest history with last-5 retention; history is for recovery, not a second deliverable list.
- `run_command`/`run_script`/`start_service` may use cwd under `active_workspace`, task-scoped `task_drive`, task-scoped `artifact_store`, and external `user_files` where the active profile permits it. In light direct tasks, omitted `run_script.cwd` defaults to task scratch instead of the Ouroboros repo; long-running services in light must use an explicit external/task/artifact cwd. Declared service `outputs` are copied into the task artifact store when the service stops.
- In external workspace mode, light-mode self-repo dirty checks snapshot the system repo, not the active workspace. Workspace file/build artifacts belong to workspace patch/artifact finalization; workspace HEAD/ref changes are still blocked by the separate workspace git-ref guard.
- `claude_code_edit` remains a first-class high-capability coding tool for substantial external artifacts; do not remove, hide, or downgrade it when refactoring Tool API names. It may run under external user/task/artifact cwd in direct light tasks, and under active workspace/task/artifact cwd in workspace tasks, while Ouroboros repo/control-plane cwd stays on the reviewed self-modification path. Use `outputs=[...]` when it creates deliverables that must be audited.
- `claude_code_edit` remains a first-class high-capability coding tool for substantial external artifacts; do not remove, hide, or downgrade it when refactoring Tool API names. It may run under external user/task/artifact cwd in direct light tasks, and under active workspace/task/artifact cwd in workspace tasks, while Ouroboros repo/control-plane cwd stays on the reviewed self-modification path. In docker executor-backed external workspaces, mapped active workspace cwd is blocked until a reviewed backend-safe Claude Code path exists; unmapped task/artifact/user cwd remains available where the active profile permits it. Use `outputs=[...]` when it creates deliverables that must be audited.
- Do not recommend `runtime_data/uploads`, skill payloads, or owner state directories as generic artifact transport.
#### Live Subagent Task Constraints

View file

@ -227,6 +227,7 @@ class OuroborosAgent:
"task_group_id",
"task_group",
"subagent_envelope",
"executor_ref",
):
if task.get(key) not in (None, ""):
task_metadata[key] = task.get(key)
@ -244,6 +245,7 @@ class OuroborosAgent:
memory_mode=str(task.get("memory_mode") or ""),
budget_drive_root=str(task.get("budget_drive_root") or ""),
task_metadata=task_metadata,
executor_ref=task_metadata.get("executor_ref") if isinstance(task_metadata.get("executor_ref"), dict) else {},
pending_events=self._pending_events,
current_chat_id=self._current_chat_id,
current_task_type=self._current_task_type,

View file

@ -14,7 +14,7 @@ import pathlib
import re
import subprocess
from dataclasses import asdict, dataclass, field
from typing import Dict, List
from typing import Dict, Iterable, List
from ouroboros.utils import atomic_write_json, utc_now_iso
@ -249,8 +249,35 @@ def _is_sensitive_inventory_path(rel_path: str) -> bool:
return suffix in _SENSITIVE_EXTENSIONS and bool(_SENSITIVE_NAME_RE.search(name))
def build_code_inventory(repo_root: pathlib.Path, *, drive_root: pathlib.Path | None = None, persist: bool = True) -> CodeInventory:
def _is_excluded_inventory_path(path: pathlib.Path, excluded_paths: list[pathlib.Path]) -> bool:
try:
resolved = pathlib.Path(path).resolve(strict=False)
except Exception:
return False
for excluded in excluded_paths:
if resolved == excluded:
return True
try:
if excluded.is_dir():
resolved.relative_to(excluded)
return True
except Exception:
continue
return False
def build_code_inventory(
repo_root: pathlib.Path,
*,
drive_root: pathlib.Path | None = None,
persist: bool = True,
exclude_paths: Iterable[pathlib.Path] | None = None,
) -> CodeInventory:
root = pathlib.Path(repo_root).resolve(strict=False)
excluded_paths = [
pathlib.Path(path).expanduser().resolve(strict=False)
for path in (exclude_paths or [])
]
files = []
for path in _tracked_files(root):
try:
@ -259,6 +286,8 @@ def build_code_inventory(repo_root: pathlib.Path, *, drive_root: pathlib.Path |
rel_parts = path.parts
if any(part in _SKIP_DIRS for part in rel_parts):
continue
if _is_excluded_inventory_path(path, excluded_paths):
continue
if path.is_file():
files.append(_file_fact(root, path))
coverage: Dict[str, int] = {}

View file

@ -9,9 +9,9 @@ from __future__ import annotations
from typing import Any, Dict, Optional
try: # Python 3.11+
from typing import Literal, NotRequired, TypedDict # type: ignore[attr-defined]
from typing import Literal, NotRequired, Required, TypedDict # type: ignore[attr-defined]
except ImportError: # pragma: no cover - CI supports Python 3.10.
from typing_extensions import Literal, NotRequired, TypedDict # type: ignore[assignment]
from typing_extensions import Literal, NotRequired, Required, TypedDict # type: ignore[assignment]
class ChatInbound(TypedDict):
@ -413,6 +413,46 @@ class ChatHistoryResponse(TypedDict, total=False):
error: str
class ExecutorRef(TypedDict, total=False):
type: Required[Literal["local", "docker_exec"]]
id: NotRequired[str]
network: NotRequired[Literal["host", "none"]]
workspace_host_path: NotRequired[str]
workspace_backend_path: NotRequired[str]
# Required at runtime when type == "docker_exec".
container_name: NotRequired[str]
path_mappings: NotRequired[list[Dict[str, str]]]
class _TaskCreateRequestRequired(TypedDict):
description: str
class TaskCreateRequest(_TaskCreateRequestRequired, total=False):
task_id: str
type: str
chat_id: int
depth: int
session_id: str
workspace_root: str
workspace_mode: str
memory_mode: str
attachments: list[Dict[str, Any]]
allowed_resources: Dict[str, Any]
resource_policy: Dict[str, Any]
executor_ref: ExecutorRef
deadline_at: str
timeout_sec: float
timeout: float
context: str
expected_output: str
constraints: str
context_requires_self_body_docs: bool
actor_id: str
source: str
metadata: Dict[str, Any]
class TaskCreateResponse(TypedDict, total=False):
ok: bool
task_id: str
@ -587,6 +627,8 @@ __all__ = [
"ModelCatalogResponse",
"FileBrowserListResponse",
"ChatHistoryResponse",
"ExecutorRef",
"TaskCreateRequest",
"TaskCreateResponse",
"TaskListResponse",
"TaskEvent",

View file

@ -40,13 +40,14 @@ from ouroboros.task_status import (
find_child_tasks,
load_effective_task_result,
)
from ouroboros.tool_access import paths_overlap_casefold
from ouroboros.tool_access import path_is_relative_to, paths_overlap_casefold
from ouroboros.utils import iter_jsonl_objects, utc_now_iso
from ouroboros.workspace_preflight import (
collect_workspace_preflight,
render_workspace_preflight_summary,
summarize_workspace_preflight,
)
from ouroboros.workspace_executor import normalize_executor_ref
_LOG_SOURCES = (
@ -72,6 +73,8 @@ _RESERVED_METADATA_KEYS = frozenset({
"task_contract",
"allowed_resources",
"deadline_at",
"executor_ref",
"workspace_executor",
})
@ -102,7 +105,7 @@ async def api_tasks_create(request: Request) -> JSONResponse:
body = await request_json_or(request, {})
if not isinstance(body, dict):
return json_error("request body must be a JSON object", 400)
description = str(body.get("description") or body.get("text") or body.get("prompt") or "").strip()
description = str(body.get("description") or "").strip()
if not description:
return json_error("description is required", 400)
@ -161,6 +164,37 @@ async def api_tasks_create(request: Request) -> JSONResponse:
resource_policy = normalize_resource_policy(body.get("resource_policy") or raw_metadata.get("resource_policy") or {})
if resource_policy:
metadata["resource_policy"] = resource_policy
if "executor_ref" in raw_metadata or "workspace_executor" in raw_metadata:
return json_error("metadata.executor_ref/workspace_executor is reserved; pass executor_ref as a top-level task field", 400)
if "executor_ref" in body:
raw_executor_ref = body.get("executor_ref")
if not isinstance(raw_executor_ref, dict) or not raw_executor_ref:
return json_error("executor_ref must be a JSON object", 400)
if workspace_root is None:
return json_error("executor_ref requires an external workspace_root", 400)
try:
normalized_executor = normalize_executor_ref(raw_executor_ref)
except ValueError as exc:
return json_error(str(exc), 400)
if normalized_executor is not None:
for mapping in normalized_executor.mappings:
for protected_root, label in ((repo_dir, "Ouroboros system repo"), (drive_root, "Ouroboros data drive")):
if paths_overlap_casefold(mapping.host_path, protected_root):
return json_error(f"executor_ref mapping must not overlap the {label}", 400)
if not any(path_is_relative_to(workspace_root, mapping.host_path) for mapping in normalized_executor.mappings):
return json_error("executor_ref mappings must cover workspace_root", 400)
metadata["executor_ref"] = {
"type": normalized_executor.kind,
"id": normalized_executor.executor_id,
"network": normalized_executor.network,
"workspace_host_path": str(normalized_executor.mappings[0].host_path),
"workspace_backend_path": normalized_executor.mappings[0].backend_path,
"container_name": normalized_executor.container_name,
"path_mappings": [
{"host_path": str(mapping.host_path), "backend_path": mapping.backend_path}
for mapping in normalized_executor.mappings
],
}
try:
deadline_at = _normalize_deadline_at(body.get("deadline_at") or raw_metadata.get("deadline_at") or "")
except ValueError as exc:

View file

@ -13,6 +13,8 @@ from ouroboros.shell_parse import (
unwrap_env_argv,
)
from ouroboros.tool_access import resolve_shell_cwd
from ouroboros.tools.shell_guards import writer_target_tokens
from ouroboros.workspace_executor import executor_ref_from_ctx, map_backend_path, map_host_path
_DEFAULT_DENIED_OPERATIONS = frozenset({
"read_bytes",
@ -21,10 +23,13 @@ _DEFAULT_DENIED_OPERATIONS = frozenset({
"static_introspection",
"dynamic_trace",
"debug",
"write",
"delete",
})
_SHELLS = frozenset({"bash", "sh", "zsh"})
_SHELLS = frozenset({"bash", "cmd", "powershell", "pwsh", "sh", "zsh"})
_HIGH_RISK_INTERPRETERS = frozenset({
"bash", "sh", "zsh", "python", "python3", "node", "ruby", "perl", "php",
"bash", "sh", "zsh", "python", "python3", "pythonw", "pypy", "pypy3",
"node", "ruby", "perl", "php",
})
_SHELL_COMMAND_OPERATIONS = {
"cat": "read_bytes",
@ -32,6 +37,17 @@ _SHELL_COMMAND_OPERATIONS = {
"tail": "read_bytes",
"less": "read_bytes",
"more": "read_bytes",
"grep": "static_introspection",
"egrep": "static_introspection",
"fgrep": "static_introspection",
"rg": "static_introspection",
"ripgrep": "static_introspection",
"ag": "static_introspection",
"ack": "static_introspection",
"sed": "static_introspection",
"awk": "static_introspection",
"diff": "static_introspection",
"cmp": "static_introspection",
"file": "static_introspection",
"strings": "static_introspection",
"hexdump": "static_introspection",
@ -41,7 +57,36 @@ _SHELL_COMMAND_OPERATIONS = {
"nm": "static_introspection",
"otool": "static_introspection",
"cp": "copy",
"copy": "copy",
"dd": "copy",
"rsync": "copy",
"tar": "copy",
"zip": "copy",
"type": "read_bytes",
"xcopy": "copy",
"robocopy": "copy",
"certutil": "hash",
"get-content": "read_bytes",
"gc": "read_bytes",
"select-string": "static_introspection",
"copy-item": "copy",
"get-filehash": "hash",
"del": "delete",
"erase": "delete",
"mv": "delete",
"move": "delete",
"rd": "delete",
"ren": "delete",
"rename": "delete",
"rename-item": "delete",
"remove-item": "delete",
"ri": "delete",
"rm": "delete",
"rmdir": "delete",
"unlink": "delete",
"shred": "delete",
"tee": "write",
"truncate": "write",
"sha256sum": "hash",
"shasum": "hash",
"md5sum": "hash",
@ -51,6 +96,31 @@ _SHELL_COMMAND_OPERATIONS = {
"gdb": "debug",
"lldb": "debug",
}
_CMD_INLINE_SWITCHES = frozenset({"/c", "/k"})
_POWERSHELL_INLINE_SWITCHES = frozenset({"-c", "-command", "/c"})
_POWERSHELL_ENCODED_SWITCHES = frozenset({"-encodedcommand", "-enc", "-e"})
_GIT_STATIC_INTROSPECTION_SUBCOMMANDS = frozenset({
"blame",
"annotate",
"cat-file",
"diff",
"grep",
"show",
})
_GIT_PATCH_LOG_FLAGS = frozenset({"-p", "-u", "--patch", "--patch-with-stat", "--stat-with-summary"})
_GIT_GLOBAL_OPTIONS_WITH_VALUE = frozenset({
"-C",
"-c",
"--config-env",
"--exec-path",
"--git-dir",
"--namespace",
"--super-prefix",
"--work-tree",
})
_DIRECTORY_TARGET_OPERATIONS = frozenset({"copy", "delete", "read_bytes", "static_introspection", "write"})
_SHELL_GLOB_CHARS = frozenset("*?[")
_FIND_EXPRESSION_MARKERS = frozenset({"!", "(", ")"})
def _task_contract(ctx: Any) -> Dict[str, Any]:
@ -104,10 +174,58 @@ def _resolve_policy_path(ctx: Any, raw_path: str) -> pathlib.Path | None:
return (roots[0] / path).resolve(strict=False)
def _backend_spellings_for_host_path(ctx: Any, path: pathlib.Path) -> set[str]:
try:
executor = executor_ref_from_ctx(ctx)
if executor is None:
return set()
backend = map_host_path(executor, pathlib.Path(path))
except Exception:
return set()
return {backend, backend.rstrip("/")}
def _policy_backend_spellings(ctx: Any, raw_path: str, resolved: pathlib.Path | None) -> set[str]:
spellings: set[str] = set()
text = str(raw_path or "").strip()
if text:
spellings.add(slash_normalize_path_text(text).rstrip("/"))
if resolved is not None:
spellings.update(_backend_spellings_for_host_path(ctx, resolved))
return {item for item in spellings if item}
def _backend_cwd_relative_spellings(ctx: Any, work_dir: pathlib.Path, spellings: set[str]) -> set[str]:
try:
executor = executor_ref_from_ctx(ctx)
if executor is None:
return set()
backend_cwd = map_host_path(executor, pathlib.Path(work_dir)).rstrip("/")
except Exception:
return set()
relative: set[str] = set()
for spelling in spellings:
normalized = slash_normalize_path_text(spelling).rstrip("/")
if normalized.startswith(backend_cwd + "/"):
rel = normalized[len(backend_cwd) + 1:]
if rel:
relative.add(rel)
return relative
def protected_artifact_paths(ctx: Any) -> List[pathlib.Path]:
paths: List[pathlib.Path] = []
for record in _artifact_records(ctx):
for raw_path in record.get("paths") or []:
text = str(raw_path)
try:
executor_ref = executor_ref_from_ctx(ctx)
if executor_ref is not None and text.strip().startswith("/"):
mapped = map_backend_path(executor_ref, text)
if mapped not in paths:
paths.append(mapped)
except Exception:
pass
resolved = _resolve_policy_path(ctx, str(raw_path))
if resolved is not None and resolved not in paths:
paths.append(resolved)
@ -116,8 +234,8 @@ def protected_artifact_paths(ctx: Any) -> List[pathlib.Path]:
def _operation_denied(record: Dict[str, Any], operation: str) -> bool:
allow = {str(item).strip() for item in (record.get("allow") or []) if str(item).strip()}
if operation in allow:
return False
if allow:
return operation not in allow
deny = {str(item).strip() for item in (record.get("deny") or []) if str(item).strip()}
if deny:
return operation in deny
@ -141,13 +259,43 @@ def _matches(candidate: pathlib.Path, protected_path: pathlib.Path) -> bool:
return False
def _backend_spelling_matches(candidate: pathlib.Path, protected_spellings: set[str]) -> bool:
try:
raw = str(candidate)
except Exception:
return False
normalized = slash_normalize_path_text(raw).rstrip("/")
if not normalized:
return False
variants = {normalized}
if not normalized.startswith("/"):
variants.add(f"/{normalized}")
for protected in protected_spellings:
if not protected:
continue
protected_norm = slash_normalize_path_text(protected).rstrip("/")
protected_variants = {protected_norm}
if not protected_norm.startswith("/"):
protected_variants.add(f"/{protected_norm}")
if variants & protected_variants:
return True
return False
def block_reason_for_path(ctx: Any, target: pathlib.Path, operation: str) -> str:
for record in _artifact_records(ctx):
if not _operation_denied(record, operation):
continue
for raw_path in record.get("paths") or []:
protected_path = _resolve_policy_path(ctx, str(raw_path))
if protected_path is not None and _matches(pathlib.Path(target), protected_path):
protected_spellings = _policy_backend_spellings(ctx, str(raw_path), protected_path)
target_backend_spellings = _backend_spellings_for_host_path(ctx, pathlib.Path(target))
if (
protected_path is not None
and _matches(pathlib.Path(target), protected_path)
or _backend_spelling_matches(pathlib.Path(target), protected_spellings)
or bool(protected_spellings & target_backend_spellings)
):
artifact_id = str(record.get("id") or pathlib.Path(str(raw_path)).name or "protected artifact")
return (
"⚠️ RESOURCE_POLICY_BLOCKED: task_contract.resource_policy protects "
@ -164,6 +312,232 @@ def any_protected_target(ctx: Any, candidates: Iterable[pathlib.Path], operation
return ""
def _directory_contains_protected_target(ctx: Any, candidates: Iterable[pathlib.Path], operation: str) -> str:
for candidate in candidates:
try:
candidate_resolved = pathlib.Path(candidate).expanduser().resolve(strict=False)
except (OSError, TypeError, ValueError):
continue
if not candidate_resolved.is_dir():
continue
for record in _artifact_records(ctx):
if not _operation_denied(record, operation):
continue
for raw_path in record.get("paths") or []:
protected_paths: list[pathlib.Path] = []
protected_path = _resolve_policy_path(ctx, str(raw_path))
if protected_path is not None:
protected_paths.append(pathlib.Path(protected_path))
try:
executor_ref = executor_ref_from_ctx(ctx)
if executor_ref is not None and str(raw_path).strip().startswith("/"):
protected_paths.append(map_backend_path(executor_ref, str(raw_path)))
except Exception:
pass
for candidate_protected in protected_paths:
try:
candidate_protected.resolve(strict=False).relative_to(candidate_resolved)
except ValueError:
continue
except Exception:
continue
return block_reason_for_path(ctx, candidate_protected, operation)
return ""
def _resolve_candidate_path(ctx: Any, work_dir: pathlib.Path, text: str) -> pathlib.Path | None:
try:
path = pathlib.Path(text).expanduser()
if path.is_absolute():
try:
executor_ref = executor_ref_from_ctx(ctx)
return map_backend_path(executor_ref, text) if executor_ref is not None else path.resolve(strict=False)
except Exception:
return path.resolve(strict=False)
return (pathlib.Path(work_dir) / path).resolve(strict=False)
except (OSError, TypeError, ValueError):
return None
def _contains_shell_glob(text: str) -> bool:
return any(char in str(text or "") for char in _SHELL_GLOB_CHARS)
def _glob_base_candidate(ctx: Any, work_dir: pathlib.Path, text: str) -> pathlib.Path | None:
normalized = str(text or "").replace("\\", "/")
first_glob = min((idx for idx, char in enumerate(normalized) if char in _SHELL_GLOB_CHARS), default=-1)
if first_glob < 0:
return None
prefix = normalized[:first_glob]
if "/" in prefix:
base_text = prefix.rsplit("/", 1)[0] or "/"
else:
base_text = "."
return _resolve_candidate_path(ctx, work_dir, base_text)
def _inline_shell_command(argv: list[str], shell_name: str) -> str:
if shell_name in {"bash", "sh", "zsh"}:
return shell_command_string(argv)
switches = _CMD_INLINE_SWITCHES if shell_name == "cmd" else _POWERSHELL_INLINE_SWITCHES
for idx, arg in enumerate(argv[1:], start=1):
if str(arg or "").strip().lower() in switches:
return " ".join(str(part) for part in argv[idx + 1:])
return ""
def _uses_powershell_encoded_command(argv: list[str], shell_name: str) -> bool:
if shell_name not in {"powershell", "pwsh"}:
return False
return any(str(arg or "").strip().lower() in _POWERSHELL_ENCODED_SWITCHES for arg in argv[1:])
def _looks_like_versioned_python_interpreter(name: str) -> bool:
for prefix in ("python", "pypy"):
suffix = name.removeprefix(prefix)
if suffix == name or not suffix:
continue
suffix = suffix.removesuffix("m")
parts = suffix.split(".")
if parts and all(part.isdigit() for part in parts):
return True
return False
def _is_high_risk_interpreter(name: str) -> bool:
return name in _HIGH_RISK_INTERPRETERS or _looks_like_versioned_python_interpreter(name)
def _git_subcommand_index(argv: list[str]) -> int | None:
idx = 1
while idx < len(argv):
token = str(argv[idx] or "")
if token == "--":
idx += 1
continue
if token == "-C" or token in _GIT_GLOBAL_OPTIONS_WITH_VALUE:
idx += 2
continue
if any(token.startswith(option + "=") for option in _GIT_GLOBAL_OPTIONS_WITH_VALUE):
idx += 1
continue
if token.startswith("-"):
idx += 1
continue
return idx
return None
def _git_static_introspection_operation(argv: list[str]) -> str | None:
subcmd_idx = _git_subcommand_index(argv)
if subcmd_idx is None:
return None
subcmd = pathlib.PurePath(argv[subcmd_idx]).name.lower().removesuffix(".exe")
if subcmd in _GIT_STATIC_INTROSPECTION_SUBCOMMANDS:
return "static_introspection"
if subcmd == "log" and any(str(token or "") in _GIT_PATCH_LOG_FLAGS for token in argv[subcmd_idx + 1:]):
return "static_introspection"
return None
def _find_operation(argv: list[str]) -> str:
args = [str(token or "") for token in argv[1:]]
if "-delete" in args:
return "delete"
for idx, token in enumerate(args):
if token not in {"-exec", "-execdir"} or idx + 1 >= len(args):
continue
executable = pathlib.PurePath(args[idx + 1]).name.lower().removesuffix(".exe")
operation = _SHELL_COMMAND_OPERATIONS.get(executable)
if operation:
return operation
if _is_high_risk_interpreter(executable):
return "read_bytes"
return "static_introspection"
return "static_introspection"
def _find_has_explicit_start_path(argv: list[str]) -> bool:
for token in (str(item or "") for item in argv[1:]):
if not token:
continue
if token == "--":
continue
if token in _FIND_EXPRESSION_MARKERS or token.startswith("-"):
return False
return True
return False
def _git_work_dir(ctx: Any, argv: list[str], initial_work_dir: pathlib.Path) -> pathlib.Path:
work_dir = pathlib.Path(initial_work_dir)
idx = 1
while idx < len(argv):
token = str(argv[idx] or "")
if token == "--":
idx += 1
continue
if token == "-C" and idx + 1 < len(argv):
resolved = _resolve_candidate_path(ctx, work_dir, str(argv[idx + 1] or ""))
if resolved is not None:
work_dir = resolved
idx += 2
continue
if token.startswith("-C") and len(token) > 2:
resolved = _resolve_candidate_path(ctx, work_dir, token[2:])
if resolved is not None:
work_dir = resolved
idx += 1
continue
if token in _GIT_GLOBAL_OPTIONS_WITH_VALUE:
idx += 2
continue
if any(token.startswith(option + "=") for option in _GIT_GLOBAL_OPTIONS_WITH_VALUE):
idx += 1
continue
if token.startswith("-"):
idx += 1
continue
break
return work_dir
def _git_candidate_tokens(argv: list[str]) -> list[str]:
subcmd_idx = _git_subcommand_index(argv)
if subcmd_idx is None:
return []
tokens: list[str] = []
rest = argv[subcmd_idx + 1:]
for token in rest:
text = str(token or "")
if not text or text == "--":
continue
if text.startswith("-") and not pathlib.Path(text).is_absolute():
continue
tokens.append(text)
if ":" not in text:
continue
# Git object syntax such as HEAD:path/to/file or :path/to/file can
# read the protected bytes without naming a filesystem path directly.
if len(text) >= 2 and text[1] == ":" and text[0].isalpha():
continue
rev_path = text.split(":", 1)[1].lstrip("./")
if rev_path:
tokens.append(rev_path)
return tokens
def _git_static_introspection_is_path_limited(work_dir: pathlib.Path, candidates: list[pathlib.Path]) -> bool:
for candidate in candidates:
try:
resolved = pathlib.Path(candidate).resolve(strict=False)
except Exception:
continue
if resolved.exists():
return True
return False
def shell_block_reason(ctx: Any, raw_cmd: Any, *, cwd: str = "", default_cwd: pathlib.Path | None = None) -> str:
protected_paths = protected_artifact_paths(ctx)
if not protected_paths:
@ -178,12 +552,24 @@ def shell_block_reason(ctx: Any, raw_cmd: Any, *, cwd: str = "", default_cwd: pa
if not argv:
return ""
first = pathlib.PurePath(argv[0]).name.lower().removesuffix(".exe")
if _uses_powershell_encoded_command(argv, first):
return (
"⚠️ RESOURCE_POLICY_BLOCKED: task_contract.resource_policy protects "
"black-box artifacts; PowerShell EncodedCommand is not allowed while "
"protected artifacts are declared."
)
if first in _SHELLS:
inline = shell_command_string(argv)
inline = _inline_shell_command(argv, first)
if inline:
return shell_block_reason(ctx, inline, cwd=cwd, default_cwd=default_cwd)
operation = _SHELL_COMMAND_OPERATIONS.get(first)
high_risk = first in _HIGH_RISK_INTERPRETERS
operation = (
_git_static_introspection_operation(argv)
if first == "git"
else _find_operation(argv)
if first == "find"
else _SHELL_COMMAND_OPERATIONS.get(first)
)
high_risk = _is_high_risk_interpreter(first)
try:
work_dir, _cwd_root, _allowed = resolve_shell_cwd(ctx, cwd)
except Exception:
@ -197,8 +583,17 @@ def shell_block_reason(ctx: Any, raw_cmd: Any, *, cwd: str = "", default_cwd: pa
for protected in protected_paths:
if first_target == pathlib.Path(protected).resolve(strict=False):
return block_reason_for_path(ctx, first_target, "execute")
if first == "git":
work_dir = _git_work_dir(ctx, argv, pathlib.Path(work_dir))
candidate_tokens = [*env_values, *_git_candidate_tokens(argv)]
else:
candidate_tokens = [*env_values, *argv[1:]]
if first == "find" and not _find_has_explicit_start_path(argv):
candidate_tokens.append(".")
candidates: list[pathlib.Path] = []
for raw in [*env_values, *argv[1:]]:
write_target_texts = list(writer_target_tokens(argv))
candidate_tokens.extend(write_target_texts)
for raw in candidate_tokens:
text = str(raw or "")
if not text or text in {"|", "&&", "||", ";"}:
continue
@ -208,13 +603,30 @@ def shell_block_reason(ctx: Any, raw_cmd: Any, *, cwd: str = "", default_cwd: pa
continue
if text.startswith("-") and not pathlib.Path(text).is_absolute():
continue
try:
path = pathlib.Path(text).expanduser()
candidates.append(path.resolve(strict=False) if path.is_absolute() else (pathlib.Path(work_dir) / path).resolve(strict=False))
except (OSError, ValueError):
if _contains_shell_glob(text):
glob_base = _glob_base_candidate(ctx, pathlib.Path(work_dir), text)
if glob_base is not None:
candidates.append(glob_base)
continue
candidate = _resolve_candidate_path(ctx, pathlib.Path(work_dir), text)
if candidate is not None:
candidates.append(candidate)
if first == "git" and operation == "static_introspection" and not _git_static_introspection_is_path_limited(pathlib.Path(work_dir), candidates):
candidates.append(pathlib.Path(work_dir).resolve(strict=False))
if write_target_texts:
write_block = any_protected_target(ctx, candidates, "write")
if write_block:
return write_block
write_dir_block = _directory_contains_protected_target(ctx, candidates, "write")
if write_dir_block:
return write_dir_block
if operation:
return any_protected_target(ctx, candidates, operation)
direct_block = any_protected_target(ctx, candidates, operation)
if direct_block:
return direct_block
if operation in _DIRECTORY_TARGET_OPERATIONS:
return _directory_contains_protected_target(ctx, candidates, operation)
return ""
if not high_risk:
return ""
default_block = any_protected_target(ctx, candidates, "read_bytes")
@ -222,9 +634,17 @@ def shell_block_reason(ctx: Any, raw_cmd: Any, *, cwd: str = "", default_cwd: pa
return default_block
tail_text = " ".join(str(part or "") for part in [*env_values, *argv[1:]])
tail_text_posix = slash_normalize_path_text(tail_text)
records = _artifact_records(ctx)
for protected in protected_paths:
protected = pathlib.Path(protected).resolve(strict=False)
needles = {str(protected), protected.as_posix(), slash_normalize_path_text(protected)}
for record in records:
for raw_path in record.get("paths") or []:
protected_path = _resolve_policy_path(ctx, str(raw_path))
if protected_path is not None and _matches(protected, protected_path):
backend_spellings = _policy_backend_spellings(ctx, str(raw_path), protected_path)
needles.update(backend_spellings)
needles.update(_backend_cwd_relative_spellings(ctx, pathlib.Path(work_dir), backend_spellings))
try:
rel = protected.relative_to(pathlib.Path(work_dir).resolve(strict=False))
if str(rel) not in {"", "."}:

View file

@ -16,6 +16,7 @@ _HEALTH_SKIP_DIR_PREFIXES = (
".mypy_cache/",
"node_modules/",
".venv/",
"devtools/",
"tests/",
)
TARGET_MODULE_LINES = 1000
@ -91,7 +92,14 @@ MAX_FUNCTION_LINES = 300
# verified smoke result; pay it down after the recovery release lands.
# Recovery completion restores strict scope-review contract parsing after the
# atlas/context-limit incident; keep the cap at the verified smoke result.
MAX_TOTAL_FUNCTIONS = 2768
# v6.19.0-rc.1 adds the workspace executor bridge as a first-class runtime
# boundary for container-backed benchmark workspaces, including service
# process-group lifecycle helpers, shared devtools output-root guards, and
# strict cleanup/protected-artifact review follow-ups. Review cycle 2 added
# durable executor panic-cleanup/backend-path helpers. Keep deliberate headroom
# for small safety/review helpers so minor fixes do not churn this gate; pay down
# with a focused simplification pass after the prerelease lands.
MAX_TOTAL_FUNCTIONS = 3000
# Grandfathered modules are accepted debt until their surfaces stabilize/split.
GRANDFATHERED_OVERSIZED_MODULES = {
"llm.py",

View file

@ -83,10 +83,17 @@ def execute_panic_stop(
except Exception:
pass
try:
from ouroboros.workspace_executor import kill_all_foreground
kill_all_foreground(data_dir, wait=False)
except Exception:
pass
try:
from ouroboros.tools.services import kill_all_services
kill_all_services(wait=False)
kill_all_services(data_dir, wait=False)
except Exception:
pass

View file

@ -236,6 +236,15 @@ def _schedule_task(
or metadata.get("allowed_resources")
or {}
)
executor_ref = {}
executor_accessor = getattr(ctx, "workspace_executor_ref", None)
if callable(executor_accessor):
try:
candidate = executor_accessor()
if isinstance(candidate, dict) and candidate:
executor_ref = dict(candidate)
except Exception:
executor_ref = {}
lane_slots = expand_subagent_lane_slots(requested_model_lane, depth=new_depth)
if not lane_slots:
return "⚠️ SUBTASK_STATUS_ERROR: no subagent lane slots resolved; subagent was not scheduled."
@ -350,6 +359,9 @@ def _schedule_task(
evt["workspace_root"] = workspace_root
if workspace_mode:
evt["workspace_mode"] = workspace_mode
if executor_ref:
evt["executor_ref"] = executor_ref
evt["metadata"] = {**(evt.get("metadata") if isinstance(evt.get("metadata"), dict) else {}), "executor_ref": executor_ref}
if context:
evt["context"] = context
if parent_task_id:
@ -372,6 +384,7 @@ def _schedule_task(
context=context,
workspace_root=workspace_root,
workspace_mode=workspace_mode,
executor_ref=executor_ref,
allowed_resources=allowed_resources,
task_contract=child_contract,
chat_id=current_chat_id or None,

View file

@ -136,6 +136,17 @@ def is_skill_control_plane_path(target: pathlib.Path, data_root: pathlib.Path) -
return _policy_is_skill_control_plane_path(target, data_root)
def _is_workspace_executor_control_state_path(target: pathlib.Path, data_root: pathlib.Path) -> bool:
try:
rel_parts = pathlib.Path(target).resolve(strict=False).relative_to(
pathlib.Path(data_root).resolve(strict=False)
).parts
except (OSError, ValueError):
return False
lowered = [str(part).casefold() for part in rel_parts]
return "state" in lowered and "workspace_executor_processes" in lowered
def _list_dir(root: pathlib.Path, rel: str, max_entries: int = 500) -> List[str]:
target = (root / safe_relpath(rel)).resolve()
if not target.exists():
@ -557,6 +568,7 @@ def _data_write(
target_path = pathlib.Path(p)
settings_path = pathlib.Path(_cfg.SETTINGS_PATH)
data_root = pathlib.Path(_cfg.DATA_DIR).resolve(strict=False)
ctx_data_root = pathlib.Path(ctx.drive_root).resolve(strict=False)
if task_constraint and task_constraint.mode == "skill_repair" and task_constraint.payload_root:
lexical_target = pathlib.Path(p).resolve(strict=False)
else:
@ -596,6 +608,17 @@ def _data_write(
"SKILL.openclaw.md, .seed-origin) are owner/review controlled. "
"Edit the payload's user-authored files instead and rerun skill_review."
)
if (
_is_workspace_executor_control_state_path(lexical_target, ctx_data_root)
or _is_workspace_executor_control_state_path(target_path, ctx_data_root)
or _is_workspace_executor_control_state_path(lexical_target, data_root)
or _is_workspace_executor_control_state_path(target_path, data_root)
):
return (
"⚠️ DATA_WRITE_BLOCKED: workspace executor process records are "
"owner/runtime control-plane state. Use process/service lifecycle "
"tools instead of writing state/workspace_executor_processes directly."
)
matches = False
try:
if target_path.exists() and settings_path.exists():
@ -721,6 +744,46 @@ def _join_write_results(results: List[str]) -> str:
return rendered
def _protected_artifact_write_block(
ctx: ToolContext,
root: str,
paths: List[str],
*,
bucket: str = "",
skill_name: str = "",
prefix: str,
) -> str:
for rel_path in paths:
if not str(rel_path or "").strip():
continue
try:
target = resolve_resource_path(ctx, root=root, path=str(rel_path), bucket=bucket, skill_name=skill_name)
except Exception:
continue
block_reason = block_reason_for_path(ctx, target, "write")
if block_reason:
return f"⚠️ {prefix}: protected artifact path blocked: {block_reason}"
return ""
def _protected_artifact_list_block(
ctx: ToolContext,
root: str,
path: str,
*,
bucket: str = "",
skill_name: str = "",
) -> str:
try:
target = resolve_resource_path(ctx, root=root, path=path, bucket=bucket, skill_name=skill_name)
except Exception:
return ""
direct_block = block_reason_for_path(ctx, target, "static_introspection")
if direct_block:
return direct_block
return ""
def _read_file(
ctx: ToolContext,
path: str,
@ -798,6 +861,9 @@ def _list_files(
normalized, block = _access_or_block(ctx, root, "list")
if block:
return block
protected_list_block = _protected_artifact_list_block(ctx, normalized, path, bucket=bucket, skill_name=skill_name)
if protected_list_block:
return protected_list_block
if normalized == "active_workspace":
return _repo_list(ctx, dir=path, max_entries=max_entries)
if normalized == "runtime_data":
@ -846,6 +912,20 @@ def _write_file(
return "⚠️ WRITE_FILE_BLOCKED: root=system_repo writes require the active workspace to be the system repo."
except Exception as exc:
return f"⚠️ WRITE_FILE_BLOCKED: could not validate system_repo root: {type(exc).__name__}: {exc}"
write_paths = [path]
for item in files or []:
if isinstance(item, dict):
write_paths.append(str(item.get("path") or ""))
protected_block = _protected_artifact_write_block(
ctx,
normalized,
write_paths,
bucket=bucket,
skill_name=skill_name,
prefix="WRITE_FILE_BLOCKED",
)
if protected_block:
return protected_block
if normalized in {"active_workspace", "system_repo"}:
from ouroboros.tools.git import _repo_write
@ -947,6 +1027,16 @@ def _edit_text(
return "⚠️ EDIT_TEXT_BLOCKED: root=system_repo edits require the active workspace to be the system repo."
except Exception as exc:
return f"⚠️ EDIT_TEXT_BLOCKED: could not validate system_repo root: {type(exc).__name__}: {exc}"
protected_block = _protected_artifact_write_block(
ctx,
normalized,
[path],
bucket=bucket,
skill_name=skill_name,
prefix="EDIT_TEXT_BLOCKED",
)
if protected_block:
return protected_block
if normalized in {"active_workspace", "system_repo"}:
from ouroboros.tools.git import _str_replace_editor
@ -975,6 +1065,14 @@ def _edit_text(
)
try:
target = resolve_resource_path(ctx, root=normalized, path=path, bucket=bucket, skill_name=skill_name)
if normalized == "runtime_data":
data_root = pathlib.Path(ctx.drive_root).resolve(strict=False)
if _is_workspace_executor_control_state_path(target, data_root):
return (
"⚠️ EDIT_TEXT_BLOCKED: workspace executor process records are "
"owner/runtime control-plane state. Use process/service lifecycle "
"tools instead of editing state/workspace_executor_processes directly."
)
if normalized == "artifact_store":
block_reason = artifact_store_path_block_reason(target)
if block_reason:
@ -1161,6 +1259,12 @@ def _code_search(ctx: ToolContext, query: str, path: str = ".",
return f"⚠️ SEARCH_ERROR: {type(exc).__name__}: {exc}"
if not search_root.exists():
return f"⚠️ SEARCH_ERROR: path not found: {display_search_path}"
protected_root_block = block_reason_for_path(ctx, search_root, "static_introspection")
if protected_root_block:
return protected_root_block
protected_root_read_block = block_reason_for_path(ctx, search_root, "read_bytes")
if protected_root_read_block and search_root.is_file():
return protected_root_read_block
subagent_readonly = _is_local_readonly_subagent(ctx)
if subagent_readonly:
block_msg = _local_readonly_resource_block(ctx, normalized, search_root, root_path, action="SEARCH")
@ -1177,6 +1281,7 @@ def _code_search(ctx: ToolContext, query: str, path: str = ".",
matches: List[str] = []
files_searched = 0
protected_omitted = 0
truncated = False
for dirpath, dirnames, filenames in os.walk(str(search_root)):
@ -1203,6 +1308,9 @@ def _code_search(ctx: ToolContext, query: str, path: str = ".",
continue
if normalized == "user_files" and user_files_path_block_reason(ctx, fp):
continue
if block_reason_for_path(ctx, fp, "read_bytes"):
protected_omitted += 1
continue
if _is_search_skippable(fp):
continue
@ -1227,11 +1335,14 @@ def _code_search(ctx: ToolContext, query: str, path: str = ".",
break
if not matches:
return f"No matches found for {'regex' if regex else 'literal'} `{query}` in {display_search_path} ({files_searched} files searched)."
suffix = f" {protected_omitted} protected artifact file(s) omitted." if protected_omitted else ""
return f"No matches found for {'regex' if regex else 'literal'} `{query}` in {display_search_path} ({files_searched} files searched).{suffix}"
header = f"Found {len(matches)} match{'es' if len(matches) != 1 else ''} in {display_search_path} ({files_searched} files searched)"
if truncated:
header += f" — truncated at {max_results} results"
if protected_omitted:
header += f"{protected_omitted} protected artifact file(s) omitted"
return header + "\n\n" + "\n".join(matches)
_SKIP_DIRS = frozenset({
@ -1261,14 +1372,29 @@ def _extract_python_symbols(file_path: pathlib.Path) -> Tuple[List[str], List[st
def _codebase_digest(ctx: ToolContext) -> str:
"""Generate a compact file/symbol digest for the codebase."""
from ouroboros.code_intelligence import build_code_inventory, render_codebase_digest
from ouroboros.protected_artifacts import block_reason_for_path, protected_artifact_paths
repo_root = active_repo_dir_for(ctx)
protected_paths = protected_artifact_paths(ctx)
inventory = build_code_inventory(
active_repo_dir_for(ctx),
repo_root,
drive_root=pathlib.Path(ctx.drive_root),
persist=not _is_local_readonly_subagent(ctx),
persist=not _is_local_readonly_subagent(ctx) and not protected_paths,
exclude_paths=protected_paths,
)
if protected_paths:
inventory.files = [
file for file in inventory.files
if not (
block_reason_for_path(ctx, repo_root / file.path, "hash")
or block_reason_for_path(ctx, repo_root / file.path, "static_introspection")
)
]
coverage: dict[str, int] = {}
for file in inventory.files:
coverage[file.disposition] = coverage.get(file.disposition, 0) + 1
inventory.coverage = coverage
if _is_local_readonly_subagent(ctx):
repo_root = active_repo_dir_for(ctx)
inventory.files = [
file for file in inventory.files
if not _is_subagent_secret_repo_target(repo_root / file.path, repo_root)

View file

@ -1278,6 +1278,7 @@ def _git_diff(
max_chars: int = 0,
) -> str:
try:
repo_dir = active_repo_dir_for(ctx)
cmd = ["git", "diff"]
if staged:
cmd.append("--staged")
@ -1287,7 +1288,12 @@ def _git_diff(
cmd.append("--stat")
if str(path or "").strip():
cmd.extend(["--", safe_relpath(str(path))])
return _limit_git_output(run_cmd(cmd, cwd=active_repo_dir_for(ctx)), max_chars)
from ouroboros.protected_artifacts import shell_block_reason as protected_artifact_shell_block_reason
protected_block = protected_artifact_shell_block_reason(ctx, cmd, cwd=str(repo_dir), default_cwd=repo_dir)
if protected_block:
return protected_block
return _limit_git_output(run_cmd(cmd, cwd=repo_dir), max_chars)
except Exception as e:
return f"⚠️ GIT_ERROR: {_sanitize_git_error(str(e))}"

View file

@ -46,6 +46,7 @@ from ouroboros.tools.shell_guards import (
shell_has_write_indicator,
runtime_data_write_targets,
shell_writer_targets_protected,
workspace_executor_state_write_block,
writer_target_tokens,
)
from ouroboros.artifacts import task_artifact_dir_path, task_id_for_artifacts
@ -102,6 +103,21 @@ def system_repo_dir_for(ctx: Any) -> pathlib.Path:
return pathlib.Path(getattr(ctx, "system_repo_dir", None) or getattr(ctx, "repo_dir"))
def _executor_backend_candidate_allowed(ctx: Any, candidate: str, allowed_roots: List[pathlib.Path]) -> bool:
try:
from ouroboros.workspace_executor import executor_ref_from_ctx as _executor_ref_from_ctx
from ouroboros.workspace_executor import map_backend_path as _executor_map_backend_path
executor_ref = _executor_ref_from_ctx(ctx)
if executor_ref is None:
return False
resolved = _executor_map_backend_path(executor_ref, candidate)
return any(resolved.is_relative_to(root) for root in allowed_roots)
except Exception:
return False
def _detect_runtime_mode_elevation(text_lower: str) -> bool:
"""Detect shell/script attempts to change ``OUROBOROS_RUNTIME_MODE``."""
has_save = "save_settings" in text_lower
@ -478,6 +494,7 @@ class ToolContext:
memory_mode: str = ""
budget_drive_root: str = ""
task_metadata: Dict[str, Any] = field(default_factory=dict)
executor_ref: Dict[str, Any] = field(default_factory=dict)
pending_events: List[Dict[str, Any]] = field(default_factory=list)
current_chat_id: Optional[int] = None
current_task_type: Optional[str] = None
@ -553,6 +570,13 @@ class ToolContext:
def task_drive_root(self) -> pathlib.Path:
return (pathlib.Path(self.drive_root).resolve(strict=False) / "task_drives" / task_id_for_artifacts(self)).resolve(strict=False)
def workspace_executor_ref(self) -> Dict[str, Any]:
if isinstance(self.executor_ref, dict) and self.executor_ref:
return dict(self.executor_ref)
if isinstance(self.task_metadata, dict) and isinstance(self.task_metadata.get("executor_ref"), dict):
return dict(self.task_metadata["executor_ref"])
return {}
@dataclass
class ToolEntry:
@ -877,16 +901,19 @@ class ToolRegistry:
argv_executable = pathlib.PurePath(argv_for_write[0]).name.lower().removesuffix(".exe") if argv_for_write else ""
write_target_argvs = [argv_for_write] if argv_for_write else []
if argv_executable in {"sh", "bash", "zsh"}:
inline_cmd = shell_command_string(argv_for_write)
inline_cmd = next((str(argv_for_write[idx + 1] or "") for idx, token in enumerate(argv_for_write[1:], start=1) if str(token or "") in {"-c", "--command"} and idx + 1 < len(argv_for_write)), "")
if not inline_cmd:
inline_cmd = shell_command_string(argv_for_write)
inline_argv = strip_leading_env_assignments(unwrap_env_argv(shell_argv(inline_cmd)))
if inline_argv:
write_target_argvs.append(inline_argv)
explicit_write_targets = list(dict.fromkeys(str(token) for target_argv in write_target_argvs for token in writer_target_tokens(target_argv) if str(token or "").strip()))
executable_path_tokens = {str(target_argv[0]) for target_argv in write_target_argvs if target_argv}
writeish = shell_has_write_indicator(raw_cmd) or (bool(argv_for_write) and argv_executable in LIGHT_SHELL_WRITER_COMMANDS) or bool(explicit_write_targets)
protected_artifact_block = protected_artifact_shell_block_reason(self._ctx, raw_cmd, cwd=str(args.get("cwd") or ""), default_cwd=active_repo_dir_for(self._ctx))
if protected_artifact_block:
if protected_artifact_block := protected_artifact_shell_block_reason(self._ctx, raw_cmd, cwd=str(args.get("cwd") or ""), default_cwd=active_repo_dir_for(self._ctx)):
return protected_artifact_block
if writeish and (executor_state_block := workspace_executor_state_write_block(raw_cmd, drive_root=pathlib.Path(self._ctx.drive_root), cwd=str(args.get("cwd") or ""), default_cwd=active_repo_dir_for(self._ctx))):
return executor_state_block
if workspace_mode and writeish:
active_root_declared = active_repo_dir_for(self._ctx)
active_root = active_root_declared.resolve(strict=False)
@ -959,6 +986,8 @@ class ToolRegistry:
if candidate == "/dev/null":
continue
if is_absolute_path_text(candidate):
if _executor_backend_candidate_allowed(self._ctx, candidate, [*allowed_relative_roots, *allowed_data_roots]):
continue
if not re.match(r"^[A-Za-z]:[\\/]", candidate) and not candidate.startswith("\\\\"):
try:
resolved = pathlib.Path(candidate).resolve(strict=False)
@ -1017,15 +1046,9 @@ class ToolRegistry:
# Elevation pattern: blocked in all modes.
if _detect_runtime_mode_elevation(cmd_lower):
return (
"⚠️ ELEVATION_BLOCKED: shell command pattern looks like an OUROBOROS_RUNTIME_MODE elevation attempt "
"(mentions ``save_settings`` together with ``OUROBOROS_RUNTIME_MODE``, or invokes ``ouroboros.config.save_settings`` directly). Runtime mode is owner-controlled — change it by stopping the agent and editing settings.json directly, then restart."
)
return "⚠️ ELEVATION_BLOCKED: shell command pattern looks like an OUROBOROS_RUNTIME_MODE elevation attempt (mentions ``save_settings`` together with ``OUROBOROS_RUNTIME_MODE``, or invokes ``ouroboros.config.save_settings`` directly). Runtime mode is owner-controlled — change it by stopping the agent and editing settings.json directly, then restart."
if _detect_context_mode_self_lowering(cmd_lower):
return (
"⚠️ CONTEXT_MODE_SELF_LOWERING_BLOCKED: shell command pattern looks like an attempt to lower OUROBOROS_CONTEXT_MODE to low "
"through settings.json or /api/owner/context-mode. Context mode is owner-controlled — ask the owner to change the Low/Max toggle or edit settings while the agent is stopped."
)
return "⚠️ CONTEXT_MODE_SELF_LOWERING_BLOCKED: shell command pattern looks like an attempt to lower OUROBOROS_CONTEXT_MODE to low through settings.json or /api/owner/context-mode. Context mode is owner-controlled — ask the owner to change the Low/Max toggle or edit settings while the agent is stopped."
if _mentions_skill_owner_state(cmd_lower):
return (
"⚠️ SKILL_STATE_WRITE_BLOCKED: skill review, enablement, "

View file

@ -344,7 +344,7 @@ def _build_file_facts(
facts.disposition = "excluded_test"
facts.reason = "wider tests excluded by atlas policy"
return facts
if _skip_by_dir(rel) and not force_include and not (include_tests and rel.startswith("tests/")):
if _skip_by_dir(rel) and rel not in anchors and not force_include and not (include_tests and rel.startswith("tests/")):
facts.disposition = "excluded_dir"
facts.reason = "excluded non-agent-logic directory"
return facts

View file

@ -198,6 +198,9 @@ _FULL_REPO_BINARY_EXTENSIONS = frozenset({
})
_FULL_REPO_SKIP_DIR_PREFIXES = (
".cursor/", ".github/", ".vscode/", ".idea/", "assets/",
# Operator/devtools sources are tracked and reviewed when touched, but are
# not core runtime context for unrelated broad scope packs.
"devtools/",
# Full pack excludes tests; touched tests are still sent separately.
"tests/",
)

View file

@ -24,6 +24,14 @@ from ouroboros.platform_layer import (
from ouroboros.tools.registry import ToolContext, ToolEntry
from ouroboros.tool_access import resolve_shell_cwd
from ouroboros.utils import append_jsonl, utc_now_iso
from ouroboros.workspace_executor import executor_ref_from_ctx
from ouroboros.workspace_executor import kill_all_services as executor_kill_all_services
from ouroboros.workspace_executor import map_host_path as executor_map_host_path
from ouroboros.workspace_executor import service_logs as executor_service_logs
from ouroboros.workspace_executor import service_status as executor_service_status
from ouroboros.workspace_executor import start_service as executor_start_service
from ouroboros.workspace_executor import stop_service as executor_stop_service
from ouroboros.workspace_executor import stop_task_services as executor_stop_task_services
@dataclass
@ -56,6 +64,17 @@ def _service_key(ctx: ToolContext, name: str) -> str:
return f"{task_id}:{name}"
def _executor_can_run_cwd(ctx: ToolContext, workdir: pathlib.Path) -> bool:
executor_ref = executor_ref_from_ctx(ctx)
if executor_ref is None:
return False
try:
executor_map_host_path(executor_ref, pathlib.Path(workdir).resolve(strict=False))
return True
except Exception:
return False
def _tail(path: pathlib.Path, chars: int) -> str:
if not path.exists():
return ""
@ -328,6 +347,14 @@ def _start_service(
workdir = pathlib.Path(workdir).resolve(strict=False)
except Exception as exc:
return f"⚠️ SERVICE_CWD_ERROR: {type(exc).__name__}: {exc}"
try:
from ouroboros.protected_artifacts import shell_block_reason
protected_block = shell_block_reason(ctx, cmd, cwd=str(workdir), default_cwd=workdir)
if protected_block:
return protected_block
except Exception:
pass
declared_outputs = [str(item) for item in (outputs or []) if str(item or "").strip()]
try:
from ouroboros.tools.shell import _snapshot_declared_outputs
@ -335,6 +362,21 @@ def _start_service(
before_outputs = _snapshot_declared_outputs(ctx, declared_outputs, workdir, cwd_root=cwd_root)
except Exception:
before_outputs = {}
if _executor_can_run_cwd(ctx, workdir):
try:
payload = executor_start_service(
ctx,
name=service_name,
cmd=[str(part) for part in cmd],
host_cwd=workdir,
cwd_root=cwd_root,
readiness=dict(readiness or {}),
outputs=declared_outputs,
before_outputs=before_outputs,
)
return json.dumps(payload, ensure_ascii=False, indent=2)
except Exception as exc:
return f"⚠️ SERVICE_START_ERROR: executor backend failed: {type(exc).__name__}: {exc}"
task_id = str(getattr(ctx, "task_id", "") or "manual")
log_dir = pathlib.Path(ctx.drive_root) / "services" / task_id
log_dir.mkdir(parents=True, exist_ok=True)
@ -423,9 +465,14 @@ def _service_status(ctx: ToolContext, name: str = "service") -> str:
key = _service_key(ctx, service_name)
with _LOCK:
record = _SERVICES.get(key)
if not record:
return f"⚠️ SERVICE_NOT_FOUND: {name}"
return json.dumps(_status_payload(record), ensure_ascii=False, indent=2)
if record:
return json.dumps(_status_payload(record), ensure_ascii=False, indent=2)
if executor_ref_from_ctx(ctx) is not None:
payload = executor_service_status(ctx, service_name)
if payload is None:
return f"⚠️ SERVICE_NOT_FOUND: {name}"
return json.dumps(payload, ensure_ascii=False, indent=2)
return f"⚠️ SERVICE_NOT_FOUND: {name}"
def _service_logs(ctx: ToolContext, name: str = "service", tail: int = 8000) -> str:
@ -435,32 +482,41 @@ def _service_logs(ctx: ToolContext, name: str = "service", tail: int = 8000) ->
key = _service_key(ctx, service_name)
with _LOCK:
record = _SERVICES.get(key)
if not record:
return f"⚠️ SERVICE_NOT_FOUND: {name}"
try:
tail_chars = int(tail or 8000)
except (TypeError, ValueError):
return "⚠️ TOOL_ARG_ERROR (service_logs): tail must be an integer."
tail_chars = min(max(1, tail_chars), _MAX_SERVICE_LOG_TAIL_CHARS)
text = str(redact_projection(_tail(record.log_path, tail_chars)).value)
ref = {}
omitted_reason = ""
try:
size = record.log_path.stat().st_size if record.log_path.exists() else 0
if size <= _MAX_SERVICE_LOG_BLOB_BYTES:
full = record.log_path.read_text(encoding="utf-8", errors="replace") if record.log_path.exists() else ""
ref = write_blob(pathlib.Path(ctx.drive_root), full, kind="txt")
else:
omitted_reason = f"log exceeds {_MAX_SERVICE_LOG_BLOB_BYTES} byte blob cap"
except Exception:
if record:
try:
tail_chars = int(tail or 8000)
except (TypeError, ValueError):
return "⚠️ TOOL_ARG_ERROR (service_logs): tail must be an integer."
tail_chars = min(max(1, tail_chars), _MAX_SERVICE_LOG_TAIL_CHARS)
text = str(redact_projection(_tail(record.log_path, tail_chars)).value)
ref = {}
return json.dumps({
"service_id": record.service_id,
"name": record.name,
"tail": text,
"full_log_ref": ref,
"full_log_omitted": omitted_reason,
}, ensure_ascii=False, indent=2)
omitted_reason = ""
try:
size = record.log_path.stat().st_size if record.log_path.exists() else 0
if size <= _MAX_SERVICE_LOG_BLOB_BYTES:
full = record.log_path.read_text(encoding="utf-8", errors="replace") if record.log_path.exists() else ""
ref = write_blob(pathlib.Path(ctx.drive_root), full, kind="txt")
else:
omitted_reason = f"log exceeds {_MAX_SERVICE_LOG_BLOB_BYTES} byte blob cap"
except Exception:
ref = {}
return json.dumps({
"service_id": record.service_id,
"name": record.name,
"tail": text,
"full_log_ref": ref,
"full_log_omitted": omitted_reason,
}, ensure_ascii=False, indent=2)
if executor_ref_from_ctx(ctx) is not None:
try:
tail_chars = int(tail or 8000)
except (TypeError, ValueError):
return "⚠️ TOOL_ARG_ERROR (service_logs): tail must be an integer."
payload = executor_service_logs(ctx, service_name, tail_chars)
if payload is None:
return f"⚠️ SERVICE_NOT_FOUND: {name}"
return json.dumps(payload, ensure_ascii=False, indent=2)
return f"⚠️ SERVICE_NOT_FOUND: {name}"
def _stop_service(ctx: ToolContext, name: str = "service") -> str:
@ -470,40 +526,70 @@ def _stop_service(ctx: ToolContext, name: str = "service") -> str:
key = _service_key(ctx, service_name)
with _LOCK:
record = _SERVICES.pop(key, None)
if not record:
return f"⚠️ SERVICE_NOT_FOUND: {name}"
_stop_record(record)
payload = _status_payload(record)
payload["log_finalization"] = _finalize_service_log(ctx, record)
artifact_note = ""
artifact_failed = False
if record.outputs:
try:
from ouroboros.tools.shell import _register_process_outputs
if record:
_stop_record(record)
payload = _status_payload(record)
payload["log_finalization"] = _finalize_service_log(ctx, record)
artifact_note = ""
artifact_failed = False
if record.outputs:
try:
from ouroboros.tools.shell import _register_process_outputs
artifact_note, artifact_failed = _register_process_outputs(
ctx,
record.outputs,
pathlib.Path(record.cwd),
cwd_root=record.cwd_root,
before_outputs=record.before_outputs,
artifact_note, artifact_failed = _register_process_outputs(
ctx,
record.outputs,
pathlib.Path(record.cwd),
cwd_root=record.cwd_root,
before_outputs=record.before_outputs,
)
except Exception as exc:
artifact_note = f"\n\n⚠️ ARTIFACT_OUTPUT_ERROR:\n- service output finalization failed: {type(exc).__name__}: {exc}"
artifact_failed = True
elif record.cwd_root == "user_files":
payload["artifact_audit_gap"] = (
"⚠️ ARTIFACT_AUDIT_GAP: service ran in user_files cwd without outputs=[...]. "
"If it created a deliverable, rerun/register the file with outputs or "
"write_file(root=artifact_store) before claiming it."
)
except Exception as exc:
artifact_note = f"\n\n⚠️ ARTIFACT_OUTPUT_ERROR:\n- service output finalization failed: {type(exc).__name__}: {exc}"
artifact_failed = True
elif record.cwd_root == "user_files":
payload["artifact_audit_gap"] = (
"⚠️ ARTIFACT_AUDIT_GAP: service ran in user_files cwd without outputs=[...]. "
"If it created a deliverable, rerun/register the file with outputs or "
"write_file(root=artifact_store) before claiming it."
)
if artifact_note:
payload["artifact_outputs"] = artifact_note.strip()
payload["artifact_output_failed"] = bool(artifact_failed)
rendered = json.dumps(payload, ensure_ascii=False, indent=2)
if artifact_failed:
return "⚠️ ARTIFACT_OUTPUT_ERROR (stop_service): declared service outputs were not finalized.\n\n" + rendered
return rendered
if artifact_note:
payload["artifact_outputs"] = artifact_note.strip()
payload["artifact_output_failed"] = bool(artifact_failed)
rendered = json.dumps(payload, ensure_ascii=False, indent=2)
if artifact_failed:
return "⚠️ ARTIFACT_OUTPUT_ERROR (stop_service): declared service outputs were not finalized.\n\n" + rendered
return rendered
if executor_ref_from_ctx(ctx) is not None:
payload = executor_stop_service(ctx, service_name)
if payload is None:
return f"⚠️ SERVICE_NOT_FOUND: {name}"
if payload.get("stop_failed"):
return "⚠️ SERVICE_STOP_ERROR (stop_service): executor backend did not confirm service termination.\n\n" + json.dumps(payload, ensure_ascii=False, indent=2)
artifact_note = ""
artifact_failed = False
before_outputs = payload.pop("_before_outputs", {})
if payload.get("outputs"):
try:
from ouroboros.tools.shell import _register_process_outputs
artifact_note, artifact_failed = _register_process_outputs(
ctx,
[str(item) for item in (payload.get("outputs") or [])],
pathlib.Path(str(payload.get("host_cwd") or ".")),
cwd_root=str(payload.get("cwd_root") or ""),
before_outputs=before_outputs if isinstance(before_outputs, dict) else None,
)
except Exception as exc:
artifact_note = f"\n\n⚠️ ARTIFACT_OUTPUT_ERROR:\n- executor service output finalization failed: {type(exc).__name__}: {exc}"
artifact_failed = True
if artifact_note:
payload["artifact_outputs"] = artifact_note.strip()
payload["artifact_output_failed"] = bool(artifact_failed)
rendered = json.dumps(payload, ensure_ascii=False, indent=2)
if artifact_failed:
return "⚠️ ARTIFACT_OUTPUT_ERROR (stop_service): declared executor service outputs were not finalized.\n\n" + rendered
return rendered
return f"⚠️ SERVICE_NOT_FOUND: {name}"
def stop_task_services(ctx: ToolContext) -> List[Dict[str, Any]]:
@ -524,6 +610,10 @@ def stop_task_services(ctx: ToolContext) -> List[Dict[str, Any]]:
stopped.append(payload)
except Exception:
pass
try:
stopped.extend(executor_stop_task_services(ctx))
except Exception:
pass
return stopped
@ -544,6 +634,10 @@ def kill_all_services(
if wait and drive_root is not None:
payload["log_finalization"] = _finalize_service_log_for_drive(pathlib.Path(drive_root), record)
stopped.append(payload)
try:
stopped.extend(executor_kill_all_services(drive_root, wait=wait))
except Exception:
pass
if wait and drive_root is not None and stopped:
def _compact(payload: Dict[str, Any]) -> Dict[str, Any]:
item = dict(payload)

View file

@ -51,6 +51,10 @@ from ouroboros.contracts.skill_payload_policy import (
decide_payload_short_form,
resolve_skill_payload_target,
)
from ouroboros.workspace_executor import execute as executor_execute
from ouroboros.workspace_executor import executor_ref_from_ctx
from ouroboros.workspace_executor import map_backend_path as executor_map_backend_path
from ouroboros.workspace_executor import map_host_path as executor_map_host_path
log = logging.getLogger(__name__)
@ -167,6 +171,15 @@ def _allowed_output_roots(ctx: ToolContext, work_dir: pathlib.Path, cwd_root: st
def _protected_output_source_reason(ctx: ToolContext, source: pathlib.Path, label: str, changed_paths: set[str]) -> str:
"""Return a block reason for protected/control-plane output sources."""
try:
from ouroboros.protected_artifacts import block_reason_for_path
protected_artifact_reason = block_reason_for_path(ctx, source, "copy")
if protected_artifact_reason:
return protected_artifact_reason
except Exception:
pass
name_lower = source.name.lower()
if (
source.name.startswith(".")
@ -214,7 +227,13 @@ def _resolve_declared_output(
if not text:
return None, "empty output path"
raw = pathlib.Path(text).expanduser()
if raw.is_absolute() or text.startswith("~"):
executor_ref = executor_ref_from_ctx(ctx)
if executor_ref is not None and raw.is_absolute() and not text.startswith("~"):
try:
source = executor_map_backend_path(executor_ref, text)
except ValueError:
source = raw.resolve(strict=False)
elif raw.is_absolute() or text.startswith("~"):
source = raw.resolve(strict=False)
else:
source = (pathlib.Path(work_dir) / safe_relpath(text)).resolve(strict=False)
@ -318,6 +337,17 @@ def _register_process_outputs(
return "\n\n" + prefix + ":\n" + "\n".join(f"- {note}" for note in notes), failed
def _executor_can_run_cwd(ctx: ToolContext, work_dir: pathlib.Path) -> bool:
executor_ref = executor_ref_from_ctx(ctx)
if executor_ref is None:
return False
try:
executor_map_host_path(executor_ref, pathlib.Path(work_dir).resolve(strict=False))
return True
except Exception:
return False
def _resolve_git_root(path: pathlib.Path) -> pathlib.Path | None:
try:
from ouroboros.review_state import discover_repo_root
@ -783,18 +813,25 @@ def _run_shell(ctx: ToolContext, cmd, cwd: str = "", outputs: List[str] | None =
timeout_sec = _resolve_effective_timeout(_RUN_SHELL_DEFAULT_TIMEOUT_SEC)
bootstrap_process_path()
try:
res = _tracked_subprocess_run(
cmd, cwd=str(work_dir),
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
text=True, timeout=timeout_sec,
)
if _executor_can_run_cwd(ctx, pathlib.Path(work_dir)):
res = executor_execute(ctx, cmd, pathlib.Path(work_dir), timeout_sec)
else:
res = _tracked_subprocess_run(
cmd, cwd=str(work_dir),
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
text=True, timeout=timeout_sec,
)
if res.returncode != 0:
executor_note = ""
if getattr(res, "backend_trace", None):
executor_note = "\n\nEXECUTOR_TRACE:\n" + json.dumps(res.backend_trace, ensure_ascii=False, indent=2)
if _is_search_no_match(res):
return autocorrect_note + (
f"{_describe_returncode(res.returncode, cwd=work_dir)} (no matches)\n"
f"{_format_process_output(res.stdout or '', '')}"
f"{executor_note}"
)
return autocorrect_note + f"⚠️ SHELL_EXIT_ERROR: command exited with {_describe_returncode(res.returncode, cwd=work_dir)}.\n\n{_format_process_output(res.stdout or '', res.stderr or '')}"
return autocorrect_note + f"⚠️ SHELL_EXIT_ERROR: command exited with {_describe_returncode(res.returncode, cwd=work_dir)}.\n\n{_format_process_output(res.stdout or '', res.stderr or '')}{executor_note}"
after_changed = _status_snapshot(repo_root)
if after_changed != before_changed:
_invalidate_advisory(
@ -837,7 +874,10 @@ def _run_shell(ctx: ToolContext, cmd, cwd: str = "", outputs: List[str] | None =
+ f"{_format_process_output(res.stdout or '', res.stderr or '')}"
+ artifact_note
)
return autocorrect_note + f"{_describe_returncode(0, cwd=work_dir)}\n{_format_process_output(res.stdout or '', res.stderr or '')}{artifact_note}{audit_note}"
executor_note = ""
if getattr(res, "backend_trace", None):
executor_note = "\n\nEXECUTOR_TRACE:\n" + json.dumps(res.backend_trace, ensure_ascii=False, indent=2)
return autocorrect_note + f"{_describe_returncode(0, cwd=work_dir)}\n{_format_process_output(res.stdout or '', res.stderr or '')}{artifact_note}{audit_note}{executor_note}"
except subprocess.TimeoutExpired:
return (
f"⚠️ TOOL_TIMEOUT (run_command): command exceeded {timeout_sec}s. "
@ -924,6 +964,18 @@ def _control_restore_note(restored: list[str]) -> str:
)
def _claude_code_executor_block_reason(ctx: ToolContext, work_dir_path: pathlib.Path) -> str:
executor_ref = executor_ref_from_ctx(ctx)
if executor_ref is None or executor_ref.kind != "docker_exec" or not _executor_can_run_cwd(ctx, work_dir_path):
return ""
return (
"⚠️ CLAUDE_CODE_EDIT_BLOCKED: docker executor-backed workspaces route "
"process execution through a backend, but claude_code_edit edits host "
"paths directly. Use read/write/edit tools or run_command inside the "
"mapped workspace until a reviewed backend-safe Claude Code path exists."
)
def _claude_code_edit(ctx: ToolContext, prompt: str, cwd: str = "", budget: float = 5.0, validate: bool = False, bucket: str = "", skill_name: str = "", outputs: List[str] | None = None) -> str:
"""Delegate SDK edits with cwd and protected-path safety hooks."""
from ouroboros.tools.git import _acquire_git_lock, _release_git_lock
@ -994,7 +1046,7 @@ def _claude_code_edit(ctx: ToolContext, prompt: str, cwd: str = "", budget: floa
allowed_roots = []
allowed_text = ", ".join(f"{name}={pathlib.Path(root).resolve(strict=False)}" for name, root in allowed_roots)
return f"⚠️ CLAUDE_CODE_ERROR: cwd escapes allowed workspace edit roots. {e}. allowed_roots: {allowed_text}. Use the active workspace, task_drive, or artifact_store for workspace tasks."
if cwd_root not in {"active_workspace", "task_drive", "artifact_store"}:
if cwd_root not in {"active_workspace", "task_drive", "artifact_store", "user_files"}:
return "⚠️ CLAUDE_CODE_ERROR: cwd root is unavailable for workspace task edits."
work_dir_root = cwd_root
else:
@ -1021,6 +1073,9 @@ def _claude_code_edit(ctx: ToolContext, prompt: str, cwd: str = "", budget: floa
return f"⚠️ CLAUDE_CODE_ERROR: cwd not found or not a directory: {cwd}"
work_dir = str(candidate)
work_dir_path = pathlib.Path(work_dir).resolve()
executor_block = _claude_code_executor_block_reason(ctx, work_dir_path)
if executor_block:
return executor_block
before_outputs = _snapshot_declared_outputs(ctx, outputs, work_dir_path, cwd_root=work_dir_root)
skill_control_snapshots = {}
sidecar_root = pathlib.Path(skill_payload_root).resolve() if skill_payload_root is not None else None
@ -1242,9 +1297,20 @@ def _run_script(
+ ". Re-run with outputs=[...] or write the canonical deliverable via root=artifact_store."
)
try:
root = pathlib.Path(ctx.task_drive_root()) / "tmp_scripts"
workdir, _cwd_root, _allowed = resolve_shell_cwd(ctx, cwd)
resolved_workdir = pathlib.Path(workdir).resolve(strict=False)
except Exception:
root = pathlib.Path(ctx.drive_root) / "tmp_scripts"
if executor_ref_from_ctx(ctx) is not None:
return f"⚠️ RUN_SCRIPT_BLOCKED: executor-backed run_script could not resolve mapped cwd {cwd!r}."
resolved_workdir = pathlib.Path("")
executor_active = _executor_can_run_cwd(ctx, resolved_workdir) if str(resolved_workdir) else False
if executor_active:
root = resolved_workdir / ".ouroboros" / "tmp_scripts"
else:
try:
root = pathlib.Path(ctx.task_drive_root()) / "tmp_scripts"
except Exception:
root = pathlib.Path(ctx.drive_root) / "tmp_scripts"
root.mkdir(parents=True, exist_ok=True)
suffix = ".py" if "python" in pathlib.PurePath(interp).name else ".sh"
script_path = root / f"script_{uuid.uuid4().hex}{suffix}"
@ -1253,7 +1319,16 @@ def _run_script(
os.chmod(script_path, 0o600)
except OSError:
pass
argv = [interp, str(script_path), *[str(item) for item in (args or [])]]
script_arg = str(script_path)
if executor_active:
executor = executor_ref_from_ctx(ctx)
if executor is not None and executor.kind != "local":
try:
script_arg = executor_map_host_path(executor, script_path)
except Exception as exc:
script_path.unlink(missing_ok=True)
return f"⚠️ RUN_SCRIPT_BLOCKED: executor-backed run_script could not map temp script path: {type(exc).__name__}: {exc}"
argv = [interp, script_arg, *[str(item) for item in (args or [])]]
effective_cwd = str(cwd or "")
if (
not effective_cwd.strip()
@ -1261,7 +1336,16 @@ def _run_script(
and not bool(getattr(ctx, "is_workspace_mode", lambda: False)())
):
effective_cwd = str(pathlib.Path(ctx.task_drive_root()).resolve(strict=False))
result = _run_shell(ctx, argv, cwd=effective_cwd, outputs=outputs)
try:
result = _run_shell(ctx, argv, cwd=effective_cwd, outputs=outputs)
finally:
if executor_active:
try:
script_path.unlink(missing_ok=True)
script_path.parent.rmdir()
script_path.parent.parent.rmdir()
except OSError:
pass
if str(result).lstrip().startswith("⚠️"):
return f"{result}\n# script_path={script_path}"
return f"# script_path={script_path}\n{result}"
@ -1323,7 +1407,10 @@ def get_tools() -> List[ToolEntry]:
"description": (
"Working directory under the active repo/workspace, task_drive, artifact_store, "
"an external absolute/~/ user_files path, or an explicit "
"data/skills/<bucket>/<skill> payload path for skill repair."
"data/skills/<bucket>/<skill> payload path for skill repair. "
"For docker executor-backed external workspaces, mapped active_workspace cwd is "
"blocked until a backend-safe Claude Code path exists; unmapped task_drive, "
"artifact_store, and user_files cwd remain valid where runtime mode permits."
),
},
"budget": {"type": "number", "default": 5.0},

View file

@ -27,6 +27,7 @@ SHELL_WRITE_INDICATORS = (
"mv ", "cp ", "chmod ", "chown ", "unlink ", "delete", "trash",
"rsync ", "write_text", "open(", ".write(", ".writelines(",
"os.remove(", "os.unlink(", "os.mkdir(", "os.makedirs(", "sort -o",
"writefilesync", "appendfilesync", "createwritestream",
)
_SAFE_STDIO_REDIRECT_TOKENS = frozenset({
">/dev/null",
@ -327,6 +328,46 @@ def shell_writer_targets_protected(raw_cmd: Any) -> bool:
return bool(target_text and any(cf in target_text for cf in PROTECTED_RUNTIME_PATHS_LOWER))
def _workspace_executor_state_target(path: pathlib.Path, drive_root: pathlib.Path) -> bool:
try:
rel_parts = pathlib.Path(path).resolve(strict=False).relative_to(
pathlib.Path(drive_root).resolve(strict=False)
).parts
except (OSError, ValueError):
return False
lowered = [str(part).casefold() for part in rel_parts]
return "state" in lowered and "workspace_executor_processes" in lowered
def workspace_executor_state_write_block(
raw_cmd: Any,
*,
drive_root: pathlib.Path,
cwd: str = "",
default_cwd: pathlib.Path | None = None,
) -> str:
try:
drive = pathlib.Path(drive_root).resolve(strict=False)
work_dir = pathlib.Path(cwd).expanduser() if str(cwd or "").strip() else pathlib.Path(default_cwd or ".")
if not work_dir.is_absolute():
work_dir = pathlib.Path(default_cwd or ".") / work_dir
work_dir = work_dir.resolve(strict=False)
except Exception:
return ""
targets = [
target for target in runtime_data_write_targets(raw_cmd, drive_root=drive, work_dir=work_dir, allowed_roots=[])
if _workspace_executor_state_target(pathlib.Path(target), drive)
]
if not targets:
return ""
return (
"⚠️ WORKSPACE_EXECUTOR_STATE_WRITE_BLOCKED: workspace executor process records "
"are owner/runtime control-plane state. Use process/service lifecycle tools "
"instead of shell-writing state/workspace_executor_processes. Paths: "
+ ", ".join(targets[:5])
)
def light_shell_repo_mutation(
raw_cmd: Any,
*,

File diff suppressed because it is too large Load diff

View file

@ -350,7 +350,7 @@ Use `web_search` when external API/library/model behavior may be stale or versio
- New files or intentional full rewrites: `write_file` (shrink guard applies) → `commit_reviewed`.
- Coordinated/multi-file/non-obvious edits: plan the data flow, apply focused `edit_text`/`write_file` calls, inspect diff → `commit_reviewed`.
- For non-trivial, headless, workspace, or effectful work, state success criteria early and call `plan_task` before major design/build/edit work unless it is explicitly unnecessary; choose its `context_level` yourself (`minimal`, `localized`, `broad`, or `constitutional`) based on the actual risk and scope. If you skip `plan_task`, say why in the reasoning trace or final summary.
- For substantial external code artifacts, `claude_code_edit` may work in an external `user_files`, `task_drive`, or `artifact_store` cwd in direct tasks; workspace tasks use the active workspace plus task/artifact roots. This is a first-class coding path, not a shell workaround. Pass `outputs=[...]` for generated deliverables so they are copied into the task artifact store. Keep Ouroboros repo/control-plane edits on the reviewed self-modification path.
- For substantial external code artifacts, `claude_code_edit` may work in an external `user_files`, `task_drive`, or `artifact_store` cwd in direct tasks; workspace tasks use the active workspace plus task/artifact roots. In docker executor-backed external workspaces, mapped active workspace cwd is blocked until a reviewed backend-safe Claude Code path exists; unmapped `task_drive`, `artifact_store`, and `user_files` cwd remain valid where the active profile permits them. This is a first-class coding path, not a shell workaround. Pass `outputs=[...]` for generated deliverables so they are copied into the task artifact store. Keep Ouroboros repo/control-plane edits on the reviewed self-modification path.
- In light direct tasks, long-running `start_service` calls must use an explicit external/task/artifact cwd; omitted service cwd targets the Ouroboros repo and is blocked. Pass service `outputs=[...]` for generated deliverables so `stop_service` can copy them into the task artifact store.
- Before saying work is done, reopen or otherwise verify the changed deliverable/artifact through the most authoritative available surface; if verification is blocked or incomplete, say that explicitly.
- For shared-state or multi-pass logic, write the data flow/invariants before editing.

View file

@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "ouroboros"
version = "6.18.1"
version = "6.19.0rc1"
description = "Self-creating AI agent with constitution, background consciousness, and persistent identity"
readme = "README.md"
license = {text = "MIT"}
@ -96,7 +96,6 @@ web = [
"providers/*.png",
"providers/*.ico",
]
[tool.ruff]
line-length = 120
target-version = "py310"

View file

@ -1,231 +0,0 @@
#!/usr/bin/env python3
"""Minimal SWE-bench prediction helper backed by ``ouroboros run``.
Input is a JSONL file whose rows include ``instance_id``, ``workspace_root``,
and an instruction field (``problem_statement`` or ``prompt``). Output is a
SWE-bench-compatible predictions JSONL.
"""
from __future__ import annotations
import argparse
import json
import shlex
import subprocess
import sys
import tempfile
from pathlib import Path
from typing import Any
from ouroboros.config import get_finalization_grace_sec
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--input", required=True, help="JSONL instances")
parser.add_argument("--output", required=True, help="predictions JSONL")
parser.add_argument("--model-name", default="ouroboros-cli")
parser.add_argument("--cli", default="", help="optional Ouroboros CLI command prefix, e.g. 'ouroboros'")
parser.add_argument("--timeout", type=int, default=7200, help="per-instance Ouroboros CLI timeout seconds")
parser.add_argument("--continue-on-error", action="store_true", help="continue after failed instances and write errors JSONL")
parser.add_argument("--errors-output", default="", help="errors JSONL path; defaults to <output>.errors.jsonl when continuing")
parser.add_argument("--logs-dir", default="", help="optional directory for per-instance stdout/stderr logs")
parser.add_argument(
"--workspaces-root",
default="",
help="optional directory containing per-instance or repo-name local checkouts",
)
args = parser.parse_args()
rows = []
errors = []
def record_error(row: dict) -> bool:
if not args.continue_on_error:
raise RuntimeError(str(row.get("error") or row))
errors.append(row)
return True
for raw in Path(args.input).read_text(encoding="utf-8").splitlines():
if not raw.strip():
continue
item: Any = json.loads(raw)
if not isinstance(item, dict):
continue
instance_id = str(item.get("instance_id") or "")
workspace = str(item.get("workspace_root") or "").strip()
if not workspace and args.workspaces_root:
root = Path(args.workspaces_root).expanduser()
repo = str(item.get("repo") or "").strip()
candidates = [root / instance_id]
if repo:
candidates.extend([root / repo.replace("/", "__"), root / repo.split("/")[-1]])
for candidate in candidates:
if candidate.is_dir():
workspace = str(candidate)
break
prompt = str(item.get("problem_statement") or item.get("prompt") or "")
if not instance_id or not workspace or not prompt:
record_error({
"instance_id": instance_id,
"error": "each row must include instance_id, workspace_root or --workspaces-root, and problem_statement/prompt",
"reason_code": "invalid_instance",
})
continue
workspace_path = Path(workspace).expanduser().resolve(strict=False)
if not workspace_path.is_dir():
record_error({
"instance_id": instance_id,
"error": f"workspace_root is not a directory for {instance_id}: {workspace}",
"reason_code": "invalid_workspace",
})
continue
head = subprocess.run(["git", "rev-parse", "HEAD"], cwd=workspace_path, capture_output=True, text=True, timeout=10)
if head.returncode != 0:
record_error({
"instance_id": instance_id,
"error": f"workspace_root is not a git checkout for {instance_id}: {workspace_path}",
"reason_code": "not_git_checkout",
})
continue
base_commit = str(item.get("base_commit") or "").strip()
if base_commit and head.stdout.strip() != base_commit:
record_error({
"instance_id": instance_id,
"error": f"workspace HEAD for {instance_id} is {head.stdout.strip()}, expected base_commit {base_commit}",
"reason_code": "wrong_base_commit",
})
continue
status = subprocess.run(
["git", "status", "--porcelain=v1", "--untracked-files=all"],
cwd=workspace_path,
capture_output=True,
text=True,
timeout=10,
)
if status.returncode != 0 or status.stdout.strip():
record_error({
"instance_id": instance_id,
"error": f"workspace must be clean before SWE-bench run for {instance_id}",
"reason_code": "dirty_workspace",
})
continue
cli_prefix = shlex.split(args.cli) if args.cli else [sys.executable, "-m", "ouroboros.cli"]
if args.logs_dir:
result_json_path = Path(args.logs_dir).expanduser() / instance_id / "task_result.json"
else:
result_json_path = Path(tempfile.gettempdir()) / f"ouroboros_swebench_{instance_id}.task_result.json"
result_json_path.parent.mkdir(parents=True, exist_ok=True)
cmd = [
*cli_prefix,
"run",
"--workspace",
str(workspace_path),
"--memory-mode",
"empty",
"--timeout",
str(int(args.timeout)),
"--patch",
"--result-json-out",
str(result_json_path),
prompt,
]
try:
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=int(args.timeout) + get_finalization_grace_sec() + 60,
)
except subprocess.TimeoutExpired as exc:
stdout = exc.stdout if isinstance(exc.stdout, str) else (exc.stdout or b"").decode("utf-8", errors="replace")
stderr = exc.stderr if isinstance(exc.stderr, str) else (exc.stderr or b"").decode("utf-8", errors="replace")
if args.logs_dir:
log_dir = Path(args.logs_dir).expanduser() / instance_id
log_dir.mkdir(parents=True, exist_ok=True)
(log_dir / "ouroboros.stdout").write_text(stdout, encoding="utf-8")
(log_dir / "ouroboros.stderr").write_text(stderr, encoding="utf-8")
error_row = {
"instance_id": instance_id,
"returncode": 124,
"error": f"ouroboros run timed out after {int(args.timeout)}s",
"timeout": True,
}
if not args.continue_on_error:
raise RuntimeError(error_row["error"]) from exc
errors.append(error_row)
continue
task_result: dict[str, Any] = {}
if result_json_path.exists():
try:
loaded = json.loads(result_json_path.read_text(encoding="utf-8"))
if isinstance(loaded, dict):
task_result = loaded
except Exception:
task_result = {}
if args.logs_dir:
log_dir = Path(args.logs_dir).expanduser() / instance_id
log_dir.mkdir(parents=True, exist_ok=True)
(log_dir / "ouroboros.stdout").write_text(result.stdout, encoding="utf-8")
(log_dir / "ouroboros.stderr").write_text(result.stderr, encoding="utf-8")
(log_dir / "ouroboros-agent-result.json").write_text(json.dumps({
"instance_id": instance_id,
"returncode": result.returncode,
"stdout_chars": len(result.stdout or ""),
"stderr_chars": len(result.stderr or ""),
"patch_empty": not bool((result.stdout or "").strip()),
"timeout_sec": int(args.timeout),
"outcome_axes": task_result.get("outcome_axes"),
"reason_code": task_result.get("reason_code"),
"artifact_bundle": task_result.get("artifact_bundle"),
}, ensure_ascii=False, indent=2), encoding="utf-8")
if result.returncode != 0:
details = (result.stderr or result.stdout or "").strip()
if len(details) > 4000:
details = details[:4000] + "\n...[truncated]"
error_row = {
"instance_id": instance_id,
"returncode": result.returncode,
"error": details or f"ouroboros run exited {result.returncode}",
"outcome_axes": task_result.get("outcome_axes"),
"reason_code": task_result.get("reason_code"),
"artifact_bundle": task_result.get("artifact_bundle"),
"trace_refs": task_result.get("trace_refs"),
}
if not args.continue_on_error:
raise RuntimeError(error_row["error"])
errors.append(error_row)
continue
if not (result.stdout or "").strip():
error_row = {
"instance_id": instance_id,
"returncode": 0,
"error": "ouroboros run produced no patch",
"outcome_axes": task_result.get("outcome_axes"),
"reason_code": task_result.get("reason_code") or "no_patch",
"artifact_bundle": task_result.get("artifact_bundle"),
"trace_refs": task_result.get("trace_refs"),
}
if not args.continue_on_error:
raise RuntimeError(error_row["error"])
errors.append(error_row)
continue
rows.append({
"instance_id": instance_id,
"model_name_or_path": args.model_name,
"model_patch": result.stdout,
})
Path(args.output).write_text(
"\n".join(json.dumps(row, ensure_ascii=False) for row in rows) + ("\n" if rows else ""),
encoding="utf-8",
)
if errors:
error_path = Path(args.errors_output).expanduser() if args.errors_output else Path(str(args.output) + ".errors.jsonl")
error_path.write_text(
"\n".join(json.dumps(row, ensure_ascii=False) for row in errors) + "\n",
encoding="utf-8",
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -1,186 +0,0 @@
"""Terminal-Bench custom agent bridge for Ouroboros CLI."""
from __future__ import annotations
import os
import json
import shlex
import subprocess
import sys
from pathlib import Path
from typing import Any
try: # Terminal-Bench is an optional benchmark dependency.
from terminal_bench.agents.base_agent import AgentResult, BaseAgent
from terminal_bench.agents.failure_mode import FailureMode
except Exception: # pragma: no cover - exercised only when tbench is installed.
AgentResult = None # type: ignore[assignment]
BaseAgent = object # type: ignore[assignment]
FailureMode = None # type: ignore[assignment]
def _git_dirty_reason(workspace: Path) -> str:
"""Return a failure reason for dirty git workspaces; allow non-git mounts."""
probe = subprocess.run(
["git", "rev-parse", "--is-inside-work-tree"],
cwd=workspace,
capture_output=True,
text=True,
)
if probe.returncode != 0 or probe.stdout.strip().lower() != "true":
return ""
status = subprocess.run(
["git", "status", "--porcelain"],
cwd=workspace,
capture_output=True,
text=True,
)
if status.returncode != 0:
return "git_status_failed"
if status.stdout.strip():
return "dirty_git_workspace"
return ""
class OuroborosTerminalBenchAgent(BaseAgent): # type: ignore[misc, valid-type]
"""Bridge Terminal-Bench to a mounted workspace served by Ouroboros CLI."""
def __init__(
self,
workspace_root: str = "",
model_name: str = "ouroboros-cli",
timeout_sec: int = 7200,
cli: str = "",
**kwargs: Any,
) -> None:
try:
super().__init__(**kwargs)
except TypeError:
super().__init__()
self.workspace_root = workspace_root or os.environ.get("OUROBOROS_TBENCH_WORKSPACE_ROOT", "")
self.model_name = model_name
self.timeout_sec = int(timeout_sec)
self.cli = cli or os.environ.get("OUROBOROS_CLI", "")
@staticmethod
def name() -> str:
return "Ouroboros CLI"
def perform_task(self, task_description: str, session: Any, logging_dir: Path | None = None) -> Any:
workspace = Path(self.workspace_root).expanduser().resolve(strict=False) if self.workspace_root else None
if workspace is None or not workspace.is_dir():
if AgentResult is None or FailureMode is None:
return {"success": False, "output": "workspace_root must point to the mounted Terminal-Bench task workspace"}
return AgentResult(failure_mode=FailureMode.UNKNOWN_AGENT_ERROR)
dirty_reason = _git_dirty_reason(workspace)
if dirty_reason:
summary = {
"cmd": [],
"workspace_root": str(workspace),
"returncode": None,
"stdout_chars": 0,
"stderr_chars": 0,
"timeout_sec": self.timeout_sec,
"failure_mode": dirty_reason,
}
if logging_dir is not None:
Path(logging_dir).mkdir(parents=True, exist_ok=True)
(Path(logging_dir) / "ouroboros-agent-result.json").write_text(
json.dumps(summary, ensure_ascii=False, indent=2),
encoding="utf-8",
)
if AgentResult is None or FailureMode is None:
return {"success": False, "output": f"refusing dirty git workspace: {dirty_reason}"}
return AgentResult(failure_mode=FailureMode.UNKNOWN_AGENT_ERROR)
prompt = self._render_instruction(task_description) if hasattr(self, "_render_instruction") else task_description
cli_prefix = shlex.split(self.cli) if self.cli else [sys.executable, "-m", "ouroboros.cli"]
cmd = [
*cli_prefix,
"run",
"--workspace",
str(workspace),
"--memory-mode",
"empty",
"--timeout",
str(self.timeout_sec),
prompt,
]
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=self.timeout_sec + 60)
final = result.stdout.strip()
summary = {
"cmd": cmd,
"workspace_root": str(workspace),
"returncode": result.returncode,
"stdout_chars": len(result.stdout or ""),
"stderr_chars": len(result.stderr or ""),
"final_text_empty": not bool(final),
"timeout_sec": self.timeout_sec,
"failure_mode": "",
}
if logging_dir is not None:
Path(logging_dir).mkdir(parents=True, exist_ok=True)
(Path(logging_dir) / "ouroboros.stdout").write_text(result.stdout, encoding="utf-8")
(Path(logging_dir) / "ouroboros.stderr").write_text(result.stderr, encoding="utf-8")
if result.returncode != 0:
summary["failure_mode"] = "non_zero_exit"
if logging_dir is not None:
(Path(logging_dir) / "ouroboros-agent-result.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8")
if AgentResult is None or FailureMode is None:
return {"success": False, "output": result.stderr or result.stdout or f"exit {result.returncode}"}
return AgentResult(failure_mode=FailureMode.UNKNOWN_AGENT_ERROR)
if not final:
summary["failure_mode"] = "empty_final_text"
if logging_dir is not None:
(Path(logging_dir) / "ouroboros-agent-result.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8")
if AgentResult is None or FailureMode is None:
return {"success": False, "output": "ouroboros run produced empty final text"}
return AgentResult(failure_mode=FailureMode.UNKNOWN_AGENT_ERROR)
if final:
try:
submit_command = f"/submit {shlex.quote(final)}"
if hasattr(session, "send_keys"):
session.send_keys([submit_command, "Enter"], block=True)
elif hasattr(session, "send_command"):
session.send_command(submit_command)
elif hasattr(session, "run"):
session.run(submit_command)
else:
raise RuntimeError("Terminal-Bench session has no supported submit method")
except Exception as exc:
summary["failure_mode"] = "submit_failed"
summary["submit_error"] = f"{type(exc).__name__}: {exc}"
if logging_dir is not None:
(Path(logging_dir) / "ouroboros-agent-result.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8")
if AgentResult is None or FailureMode is None:
return {"success": False, "output": summary["submit_error"]}
return AgentResult(failure_mode=FailureMode.UNKNOWN_AGENT_ERROR)
summary["failure_mode"] = "none"
if logging_dir is not None:
(Path(logging_dir) / "ouroboros-agent-result.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8")
if AgentResult is None or FailureMode is None:
return {"success": True, "output": final}
return AgentResult(failure_mode=FailureMode.NONE)
except subprocess.TimeoutExpired as exc:
stdout = (exc.stdout or "").decode("utf-8", errors="replace") if isinstance(exc.stdout, bytes) else str(exc.stdout or "")
stderr = (exc.stderr or "").decode("utf-8", errors="replace") if isinstance(exc.stderr, bytes) else str(exc.stderr or "")
if logging_dir is not None:
Path(logging_dir).mkdir(parents=True, exist_ok=True)
(Path(logging_dir) / "ouroboros.stdout").write_text(stdout, encoding="utf-8")
(Path(logging_dir) / "ouroboros.stderr").write_text(stderr, encoding="utf-8")
(Path(logging_dir) / "ouroboros-agent-result.json").write_text(json.dumps({
"cmd": cmd,
"workspace_root": str(workspace),
"returncode": 124,
"stdout_chars": len(stdout),
"stderr_chars": len(stderr),
"timeout_sec": self.timeout_sec,
"failure_mode": "timeout",
}, ensure_ascii=False, indent=2), encoding="utf-8")
if AgentResult is None or FailureMode is None:
return {"success": False, "output": f"ouroboros cli timed out after {self.timeout_sec}s", "timeout": True}
return AgentResult(failure_mode=FailureMode.AGENT_TIMEOUT)
__all__ = ["OuroborosTerminalBenchAgent"]

View file

@ -992,6 +992,11 @@ async def lifespan(app):
kill_all_tracked_subprocesses()
except Exception:
pass
try:
from ouroboros.workspace_executor import kill_all_foreground
kill_all_foreground(lifespan_drive_root)
except Exception:
pass
try:
from ouroboros.tools.services import kill_all_services
kill_all_services(lifespan_drive_root)
@ -1056,9 +1061,14 @@ def _emergency_process_cleanup(*, port_sweep: bool = True) -> None:
kill_all_tracked_subprocesses()
except Exception:
pass
try:
from ouroboros.workspace_executor import kill_all_foreground
kill_all_foreground(DATA_DIR, wait=False)
except Exception:
pass
try:
from ouroboros.tools.services import kill_all_services
kill_all_services(wait=False)
kill_all_services(DATA_DIR, wait=False)
except Exception:
pass
try:

View file

@ -889,6 +889,7 @@ def test_ci_branch_filters_include_packaging_assets():
workflow = _ci_workflow()
assert "- 'packaging/**'" in workflow
assert "- 'devtools/**'" in workflow
def test_ci_release_prerelease_flag_uses_preflight_output():

View file

@ -978,3 +978,56 @@ def test_state_response_declares_runtime_and_capability_keys():
f"StateResponse lost the runtime/capability key {required!r}; "
"ARCHITECTURE.md §11.3 contract is out of sync."
)
def test_task_create_request_declares_executor_ref_contract():
"""TaskCreateRequest pins executor_ref as the gateway-owned backend contract."""
from ouroboros.gateway.contracts import ExecutorRef, TaskCreateRequest
from ouroboros.workspace_executor import normalize_executor_ref
request_keys = set(TaskCreateRequest.__annotations__.keys())
for required in (
"description",
"task_id",
"type",
"chat_id",
"depth",
"session_id",
"workspace_root",
"workspace_mode",
"memory_mode",
"attachments",
"allowed_resources",
"resource_policy",
"executor_ref",
"deadline_at",
"timeout_sec",
"timeout",
"context",
"expected_output",
"constraints",
"context_requires_self_body_docs",
"actor_id",
"source",
"metadata",
):
assert required in request_keys
assert TaskCreateRequest.__required_keys__ == frozenset({"description"})
executor_keys = set(ExecutorRef.__annotations__.keys())
for required in ("type", "workspace_host_path", "workspace_backend_path", "network", "container_name", "path_mappings"):
assert required in executor_keys
assert normalize_executor_ref(
{
"type": "local",
"path_mappings": [{"host_path": tempfile.gettempdir(), "backend_path": "/workspace"}],
}
)
with pytest.raises(ValueError, match="requires container_name"):
normalize_executor_ref(
{
"type": "docker_exec",
"workspace_host_path": tempfile.gettempdir(),
"workspace_backend_path": "/workspace",
}
)

View file

@ -0,0 +1,671 @@
from __future__ import annotations
import asyncio
import contextlib
import io
import importlib.util
import json
import shlex
import shutil
import subprocess
import sys
import tarfile
import urllib.error
import urllib.request
from pathlib import Path
from types import SimpleNamespace
import pytest
from devtools.benchmarks.common.official_commands import programbench_eval_cmd, swebench_eval_cmd
from devtools.benchmarks.osworld.normalize_logs import normalize_bundle
from devtools.benchmarks.programbench.programbench_adapter import (
build_ouroboros_task_body,
create_submission_tarball,
preflight_cleanroom_container,
)
from devtools.benchmarks.swe_bench.presets import resolve_preset
REPO_ROOT = Path(__file__).resolve().parents[1]
_BASH_CAPTURE_AVAILABLE = sys.platform != "win32" and shutil.which("bash") is not None
def _git_repo(path: Path) -> str:
path.mkdir(parents=True, exist_ok=True)
subprocess.run(["git", "init"], cwd=path, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
subprocess.run(["git", "config", "user.email", "test@example.invalid"], cwd=path, check=True)
subprocess.run(["git", "config", "user.name", "Test"], cwd=path, check=True)
(path / "app.py").write_text("print('base')\n", encoding="utf-8")
subprocess.run(["git", "add", "app.py"], cwd=path, check=True)
subprocess.run(["git", "commit", "-m", "base"], cwd=path, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
return subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=path, text=True).strip()
def test_runtime_core_does_not_import_devtools():
runtime_paths = [REPO_ROOT / "ouroboros", REPO_ROOT / "server.py"]
offenders: list[str] = []
for root in runtime_paths:
files = [root] if root.is_file() else sorted(root.rglob("*.py"))
for path in files:
text = path.read_text(encoding="utf-8", errors="ignore")
if "import devtools" in text or "from devtools" in text:
offenders.append(str(path.relative_to(REPO_ROOT)))
assert not offenders
def test_official_command_builders_do_not_replace_scoring():
assert programbench_eval_cmd(Path("/runs/pb")) == ["programbench", "eval", "/runs/pb"]
assert swebench_eval_cmd("princeton-nlp/SWE-bench_Verified", Path("/runs/predictions.jsonl"), "ouroboros", 2) == [
"python",
"-m",
"swebench.harness.run_evaluation",
"--dataset_name",
"princeton-nlp/SWE-bench_Verified",
"--predictions_path",
"/runs/predictions.jsonl",
"--max_workers",
"2",
"--run_id",
"ouroboros",
]
def test_pyproject_does_not_package_devtools_runtime_assets():
pyproject = (REPO_ROOT / "pyproject.toml").read_text(encoding="utf-8")
assert '"devtools*"' not in pyproject
assert "devtools = [" not in pyproject
assert '"benchmarks/**/*.sh"' not in pyproject
assert '"benchmarks/**/*.md"' not in pyproject
def test_executable_devtools_entrypoints_support_direct_help():
scripts = [
"devtools/benchmarks/programbench/run_programbench.py",
"devtools/benchmarks/terminal_bench/run_harbor_smoke.py",
"devtools/benchmarks/swe_bench/swebench_predictions.py",
"devtools/benchmarks/swe_bench_pro/grade_pro.py",
"devtools/benchmarks/swe_bench_pro/pro_predictions.py",
"devtools/benchmarks/osworld/normalize_logs.py",
"devtools/benchmarks/osworld/osworld_adapter_skeleton.py",
]
for rel in scripts:
proc = subprocess.run(
[sys.executable, str(REPO_ROOT / rel), "--help"],
cwd=REPO_ROOT,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
timeout=20,
)
assert proc.returncode == 0, f"{rel} failed:\nSTDOUT:\n{proc.stdout}\nSTDERR:\n{proc.stderr}"
assert "usage:" in proc.stdout.lower()
def test_programbench_task_body_sets_executor_and_protected_policy(tmp_path):
workspace = tmp_path / "workspace"
_git_repo(workspace)
body = build_ouroboros_task_body(
instruction="solve",
workspace_host_path=workspace,
container_name="pb-cleanroom",
protected_backend_paths=["/workspace/executable"],
)
assert body["allowed_resources"] == {"web": False, "network": False, "internet": False}
assert body["actor_id"] == "programbench"
assert body["source"] == "programbench"
assert "actor_id" not in body["metadata"]
assert body["executor_ref"]["type"] == "docker_exec"
assert body["executor_ref"]["network"] == "none"
protected = body["resource_policy"]["protected_artifacts"][0]
assert protected["role"] == "black_box_reference"
assert protected["allow"] == ["execute"]
assert {"read_bytes", "hash", "static_introspection", "dynamic_trace", "debug"} <= set(protected["deny"])
def test_programbench_git_workspace_does_not_commit_protected_reference(tmp_path):
workspace = tmp_path / "workspace"
workspace.mkdir()
(workspace / "executable").write_text("protected-bytes\n", encoding="utf-8")
build_ouroboros_task_body(
instruction="solve",
workspace_host_path=workspace,
container_name="pb-cleanroom",
protected_backend_paths=["/workspace/executable"],
)
head = subprocess.run(["git", "rev-parse", "--verify", "HEAD"], cwd=workspace, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
show = subprocess.run(["git", "show", "HEAD:executable"], cwd=workspace, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
assert head.returncode != 0
assert show.returncode != 0
def test_programbench_submission_tarball_excludes_repo_noise(tmp_path):
workspace = tmp_path / "workspace"
(workspace / ".git").mkdir(parents=True)
(workspace / ".git" / "HEAD").write_text("ref\n", encoding="utf-8")
(workspace / ".ouroboros").mkdir()
(workspace / ".ouroboros" / "trace.json").write_text("{}\n", encoding="utf-8")
(workspace / "node_modules" / "pkg").mkdir(parents=True)
(workspace / "node_modules" / "pkg" / "index.js").write_text("junk\n", encoding="utf-8")
(workspace / "build").mkdir()
(workspace / "build" / "out.o").write_text("junk\n", encoding="utf-8")
(workspace / "dist").mkdir()
(workspace / "dist" / "bundle.js").write_text("junk\n", encoding="utf-8")
(workspace / "executable").write_text("protected\n", encoding="utf-8")
(workspace / "solution.py").write_text("print('ok')\n", encoding="utf-8")
tar_path = create_submission_tarball(
workspace,
tmp_path / "submission.tar.gz",
protected_paths=["/workspace/executable", "executable"],
)
with tarfile.open(tar_path, "r:gz") as tar:
names = set(tar.getnames())
assert "solution.py" in names
assert ".git/HEAD" not in names
assert ".ouroboros/trace.json" not in names
assert "node_modules/pkg/index.js" not in names
assert "build/out.o" not in names
assert "dist/bundle.js" not in names
assert "executable" not in names
def test_programbench_instance_path_stays_under_run_root(tmp_path):
from devtools.benchmarks.common.run_roots import safe_join_under
root = tmp_path / "programbench-run"
assert safe_join_under(root, "cheat/cheat") == root.resolve(strict=False) / "cheat" / "cheat"
with pytest.raises(ValueError, match="escapes run root"):
safe_join_under(root, "../escape")
with pytest.raises(ValueError, match="escapes run root"):
safe_join_under(root, "/tmp/escape")
def test_programbench_cleanroom_preflight_requires_task_cleanroom_and_no_network(monkeypatch):
calls = []
def fake_run(cmd, **kwargs):
calls.append(cmd)
return subprocess.CompletedProcess(
cmd,
0,
stdout=json.dumps([
{
"Config": {"Image": "ghcr.io/facebookresearch/programbench/foo:task_cleanroom"},
"HostConfig": {"NetworkMode": "none"},
}
]),
stderr="",
)
import devtools.benchmarks.programbench.programbench_adapter as adapter
monkeypatch.setattr(adapter.subprocess, "run", fake_run)
assert preflight_cleanroom_container("pb") == {
"image": "ghcr.io/facebookresearch/programbench/foo:task_cleanroom",
"network": "none",
}
assert calls[0][:2] == ["docker", "inspect"]
def test_swe_verified_preset_uses_official_dataset_name():
assert resolve_preset("verified") == "princeton-nlp/SWE-bench_Verified"
assert resolve_preset("SWE-bench/SWE-bench_Verified") == "princeton-nlp/SWE-bench_Verified"
def test_terminal_bench_harbor_adapter_is_optional_import():
spec = importlib.util.spec_from_file_location(
"tb_harbor_adapter",
REPO_ROOT / "devtools" / "benchmarks" / "terminal_bench" / "harbor_installed_agent.py",
)
assert spec and spec.loader
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
assert module.OuroborosTerminalBenchAgent.name() == "Ouroboros Installed"
def test_terminal_bench_adapter_does_not_commit_target_workspace():
adapter = (REPO_ROOT / "devtools" / "benchmarks" / "terminal_bench" / "harbor_installed_agent.py").read_text(encoding="utf-8")
assert "git add -A" not in adapter
assert "git commit --allow-empty" not in adapter
def test_terminal_bench_source_copy_excludes_secret_shaped_files(tmp_path):
import devtools.benchmarks.terminal_bench.harbor_installed_agent as tb_agent
source = tmp_path / "source"
target = tmp_path / "target"
source.mkdir()
(source / "module.py").write_text("print('ok')\n", encoding="utf-8")
secret_names = (
".env",
".env.example",
".git-credentials",
".netrc",
".npmrc",
".pypirc",
"aws-credentials.json",
"credentials.json",
"gcp-service-account.json",
"id_rsa",
"repo.bundle",
"repo_bundle_manifest.json",
"secrets.json",
"service-account.json",
)
for name in secret_names:
(source / name).write_text("secret\n", encoding="utf-8")
(source / "cert.pem").write_text("secret\n", encoding="utf-8")
(source / "python-standalone").mkdir()
(source / "python-standalone" / "python").write_text("binary\n", encoding="utf-8")
tb_agent._copy_clean_source(source, target)
assert (target / "module.py").exists()
for name in (*secret_names, "cert.pem", "python-standalone"):
assert not (target / name).exists()
def test_terminal_bench_network_preflight_uses_configured_provider(tmp_path, monkeypatch):
import devtools.benchmarks.terminal_bench.harbor_installed_agent as tb_agent
def fake_urlopen(req, timeout=0):
raise urllib.error.HTTPError(req.full_url, 401, "Unauthorized", hdrs=None, fp=None)
monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen)
class Env:
def __init__(self) -> None:
self.command = ""
async def exec(self, *, command, timeout_sec=None, env=None, cwd=None):
self.command = command
script = command.split("python3 - <<'PY'\n", 1)[1].rsplit("\nPY", 1)[0]
stdout = io.StringIO()
code = 0
try:
with contextlib.redirect_stdout(stdout):
exec(script, {})
except SystemExit as exc:
code = int(exc.code or 0)
return SimpleNamespace(return_code=code, stdout=stdout.getvalue(), stderr="")
from types import SimpleNamespace
env = Env()
agent = tb_agent.OuroborosTerminalBenchAgent(logs_dir=tmp_path)
asyncio.run(agent._network_preflight(env, {"OPENAI_API_KEY": "sk-test"}))
assert "api.openai.com" in env.command
assert "openrouter.ai" not in env.command
assert "urllib.error.HTTPError" in env.command
assert "openai_preflight_status 401" in (tmp_path / "network-preflight.txt").read_text(encoding="utf-8")
def test_terminal_bench_adapter_forwards_gigachat_and_preflights_direct_provider(tmp_path, monkeypatch):
import devtools.benchmarks.terminal_bench.harbor_installed_agent as tb_agent
monkeypatch.setenv("OUROBOROS_BENCH_ALLOW_CONTAINER_SECRETS", "1")
monkeypatch.setenv("GIGACHAT_CREDENTIALS", "gigachat-test-credentials")
monkeypatch.setenv("GIGACHAT_BASE_URL", "https://gigachat.example.invalid/api/v1")
class Env:
def __init__(self) -> None:
self.command = ""
async def exec(self, *, command, timeout_sec=None, env=None, cwd=None):
self.command = command
script = command.split("python3 - <<'PY'\n", 1)[1].rsplit("\nPY", 1)[0]
stdout = io.StringIO()
code = 0
try:
with contextlib.redirect_stdout(stdout):
exec(script, {})
except SystemExit as exc:
code = int(exc.code or 0)
return SimpleNamespace(return_code=code, stdout=stdout.getvalue(), stderr="")
def fake_urlopen(req, timeout=0):
raise urllib.error.HTTPError(req.full_url, 401, "Unauthorized", hdrs=None, fp=None)
monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen)
agent = tb_agent.OuroborosTerminalBenchAgent(logs_dir=tmp_path)
injected = agent._container_env()
env = Env()
asyncio.run(agent._network_preflight(env, injected))
assert injected["GIGACHAT_CREDENTIALS"] == "gigachat-test-credentials"
assert "gigachat.example.invalid/api/v1/models" in env.command
assert "gigachat_preflight_status 401" in (tmp_path / "network-preflight.txt").read_text(encoding="utf-8")
def test_terminal_bench_adapter_refuses_container_secret_injection_by_default(tmp_path, monkeypatch):
import devtools.benchmarks.terminal_bench.harbor_installed_agent as tb_agent
monkeypatch.delenv("OUROBOROS_BENCH_ALLOW_CONTAINER_SECRETS", raising=False)
monkeypatch.setenv("OPENROUTER_API_KEY", "sk-test-container-secret")
agent = tb_agent.OuroborosTerminalBenchAgent(logs_dir=tmp_path)
injected = agent._container_env()
assert "OPENROUTER_API_KEY" not in injected
with pytest.raises(RuntimeError, match="refuses to inject long-lived provider credentials"):
agent._enforce_container_secret_policy(injected)
def test_terminal_bench_task_body_uses_top_level_actor_id():
adapter = (REPO_ROOT / "devtools" / "benchmarks" / "terminal_bench" / "harbor_installed_agent.py").read_text(encoding="utf-8")
assert '"actor_id": "harbor-terminal-bench"' in adapter
assert '"metadata": {{"source": "terminal-bench", "delegation_role": "root"}}' in adapter
assert '"metadata": {{"actor_id": "harbor-terminal-bench"' not in adapter
@pytest.mark.skipif(not _BASH_CAPTURE_AVAILABLE, reason="capture_patch.sh is a POSIX shell helper; Python wrappers are covered separately")
def test_swe_pro_capture_keeps_untracked_text_and_drops_binary(tmp_path):
repo = tmp_path / "repo"
base = _git_repo(repo)
(repo / "new_file.py").write_text("print('new')\n", encoding="utf-8")
(repo / "binary.bin").write_bytes(b"\x00\x01\x02\x03")
(repo / "build").mkdir()
(repo / "build" / "out.txt").write_text("junk\n", encoding="utf-8")
(repo / "dist").mkdir()
(repo / "dist" / "out.txt").write_text("junk\n", encoding="utf-8")
(repo / "app.py").write_text("print('changed')\n", encoding="utf-8")
capture = REPO_ROOT / "devtools" / "benchmarks" / "swe_bench_pro" / "capture_patch.sh"
out = tmp_path / "patch.diff"
subprocess.run(["bash", str(capture), str(repo), base, str(out)], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
patch = out.read_text(encoding="utf-8")
assert "new_file.py" in patch
assert "app.py" in patch
assert "binary.bin" not in patch
assert "build/out.txt" not in patch
assert "dist/out.txt" not in patch
@pytest.mark.skipif(not _BASH_CAPTURE_AVAILABLE, reason="capture_patch.sh is a POSIX shell helper; Python wrappers are covered separately")
def test_swe_pro_capture_requires_valid_base_and_external_output(tmp_path):
repo = tmp_path / "repo"
base = _git_repo(repo)
(repo / "app.py").write_text("print('changed')\n", encoding="utf-8")
capture = REPO_ROOT / "devtools" / "benchmarks" / "swe_bench_pro" / "capture_patch.sh"
missing_output = subprocess.run(["bash", str(capture), str(repo), base], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
bad_base = subprocess.run(
["bash", str(capture), str(repo), "not-a-commit", str(tmp_path / "bad.diff")],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
internal_output = REPO_ROOT / "devtools" / "should-not-write.diff"
internal_dir = REPO_ROOT / "_test_rejected_capture_output_dir"
nested_internal_output = internal_dir / "out.diff"
shutil.rmtree(internal_dir, ignore_errors=True)
try:
repo_internal = subprocess.run(
["bash", str(capture), str(repo), base, str(internal_output)],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
nested_repo_internal = subprocess.run(
["bash", str(capture), str(repo), base, str(nested_internal_output)],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
finally:
internal_output.unlink(missing_ok=True)
shutil.rmtree(internal_dir, ignore_errors=True)
assert missing_output.returncode != 0
assert bad_base.returncode != 0
assert repo_internal.returncode != 0
assert "outside the Ouroboros repo" in repo_internal.stderr
assert nested_repo_internal.returncode != 0
assert "outside the Ouroboros repo" in nested_repo_internal.stderr
assert not internal_dir.exists()
def test_swe_pro_grade_runs_official_eval_with_raw_sample(tmp_path, monkeypatch):
import devtools.benchmarks.swe_bench_pro.grade_pro as grade_pro
eval_repo = tmp_path / "SWE-bench_Pro-os"
helper = eval_repo / "helper_code"
helper.mkdir(parents=True)
raw_sample = helper / "sweap_eval_full_v2.jsonl"
raw_sample.write_text(json.dumps({"instance_id": "x", "FAIL_TO_PASS": [], "PASS_TO_PASS": []}) + "\n", encoding="utf-8")
predictions = tmp_path / "predictions.jsonl"
predictions.write_text(json.dumps({"instance_id": "x", "model_patch": "diff --git a/a b/a\n", "model_name_or_path": "m"}) + "\n", encoding="utf-8")
captured: dict[str, object] = {}
def fake_run(cmd, **kwargs):
captured["cmd"] = list(cmd)
captured["cwd"] = kwargs.get("cwd")
return subprocess.CompletedProcess(cmd, 0, stdout="", stderr="")
monkeypatch.setattr(grade_pro.subprocess, "run", fake_run)
monkeypatch.setattr(
sys,
"argv",
[
"grade_pro.py",
"--predictions",
str(predictions),
"--out-dir",
str(tmp_path / "out"),
"--eval-repo",
str(eval_repo),
],
)
assert grade_pro.main() == 0
assert "--raw_sample_path" in captured["cmd"]
assert str(raw_sample) in captured["cmd"]
assert captured["cwd"] == str(eval_repo)
def test_swe_pro_grade_rejects_repo_internal_output(tmp_path, monkeypatch):
import devtools.benchmarks.swe_bench_pro.grade_pro as grade_pro
eval_repo = tmp_path / "SWE-bench_Pro-os"
helper = eval_repo / "helper_code"
helper.mkdir(parents=True)
raw_sample = helper / "sweap_eval_full_v2.jsonl"
raw_sample.write_text(json.dumps({"instance_id": "x", "FAIL_TO_PASS": [], "PASS_TO_PASS": []}) + "\n", encoding="utf-8")
predictions = tmp_path / "predictions.jsonl"
predictions.write_text(json.dumps({"instance_id": "x", "model_patch": "diff --git a/a b/a\n", "model_name_or_path": "m"}) + "\n", encoding="utf-8")
internal_out = REPO_ROOT / "_test_rejected_grade_output_dir"
shutil.rmtree(internal_out, ignore_errors=True)
monkeypatch.setattr(
sys,
"argv",
[
"grade_pro.py",
"--predictions",
str(predictions),
"--out-dir",
str(internal_out),
"--eval-repo",
str(eval_repo),
"--skip-run",
],
)
try:
with pytest.raises(ValueError, match="under repo"):
grade_pro.main()
assert not internal_out.exists()
finally:
shutil.rmtree(internal_out, ignore_errors=True)
def test_swe_pro_prediction_capture_rejects_empty_patch(tmp_path, monkeypatch):
import devtools.benchmarks.swe_bench_pro.pro_predictions as pro_predictions
repo = tmp_path / "repo"
repo.mkdir()
out = tmp_path / "empty.diff"
def fake_run(cmd, **kwargs):
out.write_text("", encoding="utf-8")
return subprocess.CompletedProcess(cmd, 0, stdout="", stderr="")
monkeypatch.setattr(pro_predictions.subprocess, "run", fake_run)
with pytest.raises(RuntimeError, match="empty patch"):
pro_predictions._capture_patch(repo, "HEAD", out)
def test_swe_predictions_rejects_unsafe_instance_id_before_logs_escape(tmp_path, monkeypatch):
import devtools.benchmarks.swe_bench.swebench_predictions as swe_predictions
input_jsonl = tmp_path / "instances.jsonl"
output_jsonl = tmp_path / "predictions.jsonl"
logs_dir = tmp_path / "logs"
input_jsonl.write_text(
json.dumps({"instance_id": "../escape", "workspace_root": "/missing", "problem_statement": "fix"}) + "\n",
encoding="utf-8",
)
monkeypatch.setattr(
sys,
"argv",
[
"swebench_predictions.py",
"--input",
str(input_jsonl),
"--output",
str(output_jsonl),
"--logs-dir",
str(logs_dir),
"--continue-on-error",
],
)
assert swe_predictions.main() == 0
errors = json.loads((tmp_path / "predictions.jsonl.errors.jsonl").read_text(encoding="utf-8").splitlines()[0])
assert errors["reason_code"] == "invalid_instance_id"
assert not (tmp_path / "escape").exists()
def test_swe_pro_predictions_rejects_unsafe_instance_id_before_patch_path(tmp_path, monkeypatch):
import devtools.benchmarks.swe_bench_pro.pro_predictions as pro_predictions
repo = tmp_path / "repo"
repo.mkdir()
input_jsonl = tmp_path / "instances.jsonl"
output_jsonl = tmp_path / "predictions.jsonl"
patch_dir = tmp_path / "patches"
input_jsonl.write_text(
json.dumps({"instance_id": "../escape", "repo_dir": str(repo), "base_commit": "HEAD"}) + "\n",
encoding="utf-8",
)
monkeypatch.setattr(pro_predictions, "_capture_patch", lambda *a, **k: pytest.fail("unsafe id should fail before capture"))
monkeypatch.setattr(
sys,
"argv",
[
"pro_predictions.py",
"--input",
str(input_jsonl),
"--output",
str(output_jsonl),
"--patch-dir",
str(patch_dir),
],
)
with pytest.raises(ValueError, match="single safe path component"):
pro_predictions.main()
assert not (tmp_path / "escape").exists()
def test_benchmark_output_helpers_reject_repo_internal_outputs(tmp_path, monkeypatch):
import devtools.benchmarks.swe_bench.swebench_predictions as swe_predictions
import devtools.benchmarks.terminal_bench.run_harbor_smoke as harbor_smoke
input_jsonl = tmp_path / "instances.jsonl"
input_jsonl.write_text("", encoding="utf-8")
monkeypatch.setattr(sys, "argv", ["swebench_predictions.py", "--input", str(input_jsonl), "--output", str(REPO_ROOT / "devtools" / "bad.jsonl")])
with pytest.raises(ValueError, match="benchmark run output must not be under repo"):
swe_predictions.main()
monkeypatch.setattr(sys, "argv", ["run_harbor_smoke.py", "--run-root", str(REPO_ROOT / "devtools" / "bad_run")])
with pytest.raises(ValueError, match="benchmark run output must not be under repo"):
harbor_smoke.main()
def test_osworld_logs_only_normalizer(tmp_path):
bundle = tmp_path / "osworld_logs"
(bundle / "sample1").mkdir(parents=True)
(bundle / "SUMMARY.json").write_text(json.dumps({"count": 1}), encoding="utf-8")
(bundle / "sample_manifest.json").write_text(json.dumps({"samples": ["sample1"]}), encoding="utf-8")
(bundle / "trace_manifest.json").write_text(json.dumps({"traces": ["sample1/traj.jsonl"]}), encoding="utf-8")
(bundle / "sample1" / "traj.jsonl").write_text(
json.dumps({"type": "start"}) + "\n" + json.dumps({"type": "end"}) + "\n",
encoding="utf-8",
)
normalized = normalize_bundle(bundle)
assert normalized["traj_count"] == 1
assert normalized["traces"][0]["events"] == 2
assert normalized["traces"][0]["last_type"] == "end"
def test_osworld_logs_only_normalizer_accepts_nested_trace_manifests(tmp_path):
bundle = tmp_path / "osworld_logs"
sample = bundle / "chrome" / "sample1"
(sample / "traces").mkdir(parents=True)
(bundle / "SUMMARY.json").write_text(json.dumps({"count": 1}), encoding="utf-8")
(bundle / "sample_manifest.json").write_text(json.dumps({"samples": ["sample1"]}), encoding="utf-8")
(sample / "traces" / "trace_manifest.json").write_text(json.dumps({"trace": "sample1"}), encoding="utf-8")
(sample / "traj.jsonl").write_text(json.dumps({"event": "done"}) + "\n", encoding="utf-8")
normalized = normalize_bundle(bundle)
assert normalized["trace_manifest"]["trace_manifest_paths"] == ["chrome/sample1/traces/trace_manifest.json"]
assert normalized["traj_count"] == 1
def test_terminal_bench_adapter_quotes_hostile_workspace_dir(tmp_path):
from devtools.benchmarks.terminal_bench.harbor_installed_agent import OuroborosTerminalBenchAgent
class FakeResult:
return_code = 0
stdout = '{"return_code": 0}\n'
stderr = ""
class FakeEnvironment:
def __init__(self):
self.calls = []
async def exec(self, **kwargs):
self.calls.append(kwargs)
return FakeResult()
hostile = "/tmp/ws'; touch /tmp/pwn; echo '"
agent = OuroborosTerminalBenchAgent(logs_dir=tmp_path, workspace_dir=hostile)
environment = FakeEnvironment()
asyncio.run(agent._resolve_workspace_dir(environment))
asyncio.run(agent._ensure_workspace_git_root(environment))
summary = asyncio.run(agent._run_ouroboros_task(environment, {}))
assert summary["return_code"] == 0
quoted = shlex.quote(hostile)
assert environment.calls[0]["command"] == f"test -d {quoted}"
git_command = environment.calls[1]["command"]
assert f"workspace_dir={quoted}" in git_command
assert "cd \"$workspace_dir\"" in git_command
runner_command = environment.calls[-1]["command"]
runner = runner_command.split("cat > /tmp/run_ouroboros_task.py <<'PY'\n", 1)[1].split("\nPY\n", 1)[0]
assert f'"workspace_root": {json.dumps(hostile)}' in runner
compile(runner, "run_ouroboros_task.py", "exec")

View file

@ -145,6 +145,20 @@ def test_task_api_enqueue_workspace_creates_child_drive(tmp_path, monkeypatch):
assert "target workspace, not the Ouroboros system repo" in captured[0]["text"]
def test_api_tasks_create_requires_description_not_legacy_aliases(monkeypatch):
captured = []
monkeypatch.setattr("supervisor.queue.enqueue_task", lambda task: captured.append(task) or task)
app = Starlette(routes=[Route("/api/tasks", endpoint=api_tasks_create, methods=["POST"])])
client = TestClient(app)
for payload in ({"text": "legacy task"}, {"prompt": "legacy task"}, {"description": ""}):
response = client.post("/api/tasks", json=payload)
assert response.status_code == 400, (payload, response.text)
assert "description is required" in response.json().get("error", "")
assert captured == []
def test_api_tasks_create_rejects_internal_task_types(tmp_path, monkeypatch):
repo = tmp_path / "repo"
repo.mkdir()
@ -1820,6 +1834,9 @@ def test_cli_run_actor_id_is_sent_as_gateway_root_field(monkeypatch, capsys):
assert cli.main(["run", "--detach", "--timeout", "7", "--actor-id", "operator-1", "hello"]) == 0
assert captured["method"] == "POST"
assert captured["path"] == "/api/tasks"
assert captured["body"]["description"] == "hello"
assert "text" not in captured["body"]
assert "prompt" not in captured["body"]
assert captured["body"]["actor_id"] == "operator-1"
assert captured["body"]["timeout_sec"] == 7.0
assert captured["body"]["source"] == "cli"
@ -1879,8 +1896,14 @@ def test_cli_wait_task_caps_poll_request_by_timeout(monkeypatch):
def test_swebench_helper_records_cli_timeout_with_continue(tmp_path, monkeypatch):
script_path = pathlib.Path(__file__).resolve().parent.parent / "scripts" / "swebench_cli_agent.py"
spec = importlib.util.spec_from_file_location("swebench_cli_agent_test", script_path)
script_path = (
pathlib.Path(__file__).resolve().parent.parent
/ "devtools"
/ "benchmarks"
/ "swe_bench"
/ "swebench_predictions.py"
)
spec = importlib.util.spec_from_file_location("swebench_predictions_test", script_path)
assert spec and spec.loader
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
@ -1911,7 +1934,7 @@ def test_swebench_helper_records_cli_timeout_with_continue(tmp_path, monkeypatch
sys,
"argv",
[
"swebench_cli_agent.py",
"swebench_predictions.py",
"--input",
str(rows_path),
"--output",
@ -1932,29 +1955,21 @@ def test_swebench_helper_records_cli_timeout_with_continue(tmp_path, monkeypatch
assert (logs_dir / "inst1" / "ouroboros.stderr").read_text(encoding="utf-8") == "partial-err"
def test_terminal_bench_helper_refuses_dirty_git_workspace(tmp_path):
script_path = pathlib.Path(__file__).resolve().parent.parent / "scripts" / "terminal_bench_cli_agent.py"
spec = importlib.util.spec_from_file_location("terminal_bench_cli_agent_test", script_path)
def test_terminal_bench_harbor_adapter_imports_without_harbor():
script_path = (
pathlib.Path(__file__).resolve().parent.parent
/ "devtools"
/ "benchmarks"
/ "terminal_bench"
/ "harbor_installed_agent.py"
)
spec = importlib.util.spec_from_file_location("terminal_bench_harbor_adapter_test", script_path)
assert spec and spec.loader
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
workspace = tmp_path / "workspace"
_init_repo_with_file(workspace)
(workspace / "tracked.txt").write_text("dirty\n", encoding="utf-8")
logs_dir = tmp_path / "logs"
agent = module.OuroborosTerminalBenchAgent(
workspace_root=str(workspace),
cli=f"{sys.executable} -c 'raise SystemExit(99)'",
)
result = agent.perform_task("fix", SimpleNamespace(), logging_dir=logs_dir)
if isinstance(result, dict):
assert result["success"] is False
assert "dirty_git_workspace" in result["output"]
summary = json.loads((logs_dir / "ouroboros-agent-result.json").read_text(encoding="utf-8"))
assert summary["failure_mode"] == "dirty_git_workspace"
assert module.OuroborosTerminalBenchAgent.name() == "Ouroboros Installed"
assert module._repo_root() == pathlib.Path(__file__).resolve().parent.parent
def test_queue_restore_accepts_headless_chat_zero(tmp_path, monkeypatch):

View file

@ -267,9 +267,9 @@ def test_advisory_pre_review_timeout_1200():
def test_full_repo_pack_excludes_junk_dirs():
"""build_full_repo_pack must skip non-agent-logic directories (assets/, tests/)."""
"""build_full_repo_pack must skip broad non-core directories."""
from ouroboros.tools.review_helpers import _FULL_REPO_SKIP_DIR_PREFIXES
for prefix in ("assets/", "tests/"):
for prefix in ("assets/", "tests/", "devtools/"):
assert prefix in _FULL_REPO_SKIP_DIR_PREFIXES, f"{prefix} not in skip list"

View file

@ -139,6 +139,42 @@ def test_atlas_force_includes_protected_workflow_even_under_skipped_github_dir(t
assert "asset text" not in pack.text
def test_atlas_devtools_manifest_only_unless_touched(tmp_path):
_write(tmp_path / "devtools" / "benchmarks" / "programbench" / "run.py", "VALUE = 'devtools full text'\n")
_write(tmp_path / "ouroboros" / "core.py", "print('core')\n")
pack = compile_review_context_atlas(
ReviewContextAtlasRequest(
repo_dir=tmp_path,
tracked_paths=("devtools/benchmarks/programbench/run.py", "ouroboros/core.py"),
anchors=("ouroboros/core.py",),
fixed_prompt_tokens=100,
target_total_tokens=20_000,
hard_total_tokens=25_000,
)
)
coverage = _coverage(pack)
assert coverage["devtools/benchmarks/programbench/run.py"]["disposition"] == "excluded_dir"
assert "devtools full text" not in pack.text
assert coverage["ouroboros/core.py"]["disposition"] == "full"
touched = compile_review_context_atlas(
ReviewContextAtlasRequest(
repo_dir=tmp_path,
tracked_paths=("devtools/benchmarks/programbench/run.py",),
anchors=("devtools/benchmarks/programbench/run.py",),
fixed_prompt_tokens=100,
target_total_tokens=20_000,
hard_total_tokens=25_000,
)
)
touched_coverage = _coverage(touched)
assert touched_coverage["devtools/benchmarks/programbench/run.py"]["disposition"] == "full"
assert "devtools full text" in touched.text
def test_atlas_marks_sensitive_binary_oversized_and_vendored_files(tmp_path):
_write(tmp_path / ".env.example", "TOKEN=secret\n")
(tmp_path / "image.png").write_bytes(b"\x89PNG\r\n\x00")

View file

@ -1,6 +1,18 @@
from types import SimpleNamespace
def test_lifespan_shutdown_kills_executor_foreground_before_services():
import inspect
import server
source = inspect.getsource(server.lifespan)
shell_idx = source.index("kill_all_tracked_subprocesses()")
foreground_idx = source.index("kill_all_foreground(lifespan_drive_root)")
service_idx = source.index("kill_all_services(lifespan_drive_root)")
assert shell_idx < foreground_idx < service_idx
def test_shutdown_task_cleanup_args_never_reports_crash_storm():
"""Graceful shutdown (requested restart or external signal) must finalize a
running task as cancelled/interrupted, never as a worker crash storm."""
@ -87,10 +99,12 @@ def test_main_graceful_restart_cleanup_avoids_port_sweep(monkeypatch):
def test_emergency_cleanup_kills_services_without_log_finalization(monkeypatch):
import server
foreground_calls = []
service_calls = []
worker_calls = []
monkeypatch.setattr("ouroboros.tools.shell.kill_all_tracked_subprocesses", lambda: None)
monkeypatch.setattr("ouroboros.workspace_executor.kill_all_foreground", lambda *a, **k: foreground_calls.append((a, k)))
monkeypatch.setattr("ouroboros.tools.services.kill_all_services", lambda *a, **k: service_calls.append((a, k)))
monkeypatch.setattr("supervisor.workers.kill_workers", lambda **kw: worker_calls.append(kw))
monkeypatch.setattr("multiprocessing.active_children", lambda: [])
@ -101,7 +115,8 @@ def test_emergency_cleanup_kills_services_without_log_finalization(monkeypatch):
server._emergency_process_cleanup(port_sweep=False)
assert service_calls == [((), {"wait": False})]
assert foreground_calls == [((server.DATA_DIR,), {"wait": False})]
assert service_calls == [((server.DATA_DIR,), {"wait": False})]
assert worker_calls == [{"force": True, "archive_service_logs": False}]
@ -113,6 +128,7 @@ def test_emergency_cleanup_during_restart_marks_tasks_cancelled(monkeypatch):
worker_calls = []
monkeypatch.setattr("ouroboros.tools.shell.kill_all_tracked_subprocesses", lambda: None)
monkeypatch.setattr("ouroboros.workspace_executor.kill_all_foreground", lambda *a, **k: None)
monkeypatch.setattr("ouroboros.tools.services.kill_all_services", lambda *a, **k: None)
monkeypatch.setattr("supervisor.workers.kill_workers", lambda **kw: worker_calls.append(kw))
monkeypatch.setattr("multiprocessing.active_children", lambda: [])
@ -137,6 +153,7 @@ def test_emergency_cleanup_during_restart_marks_tasks_cancelled(monkeypatch):
def test_panic_stop_kills_services_without_log_finalization(monkeypatch, tmp_path):
from ouroboros import server_control
foreground_calls = []
service_calls = []
worker_calls = []
@ -144,6 +161,7 @@ def test_panic_stop_kills_services_without_log_finalization(monkeypatch, tmp_pat
pass
monkeypatch.setattr("ouroboros.tools.shell.kill_all_tracked_subprocesses", lambda: None)
monkeypatch.setattr("ouroboros.workspace_executor.kill_all_foreground", lambda *a, **k: foreground_calls.append((a, k)))
monkeypatch.setattr("ouroboros.tools.services.kill_all_services", lambda *a, **k: service_calls.append((a, k)))
monkeypatch.setattr("ouroboros.local_model.get_manager", lambda: SimpleNamespace(stop_server=lambda: None))
monkeypatch.setattr("supervisor.state.load_state", lambda: {})
@ -165,5 +183,6 @@ def test_panic_stop_kills_services_without_log_finalization(monkeypatch, tmp_pat
except ExitCalled:
pass
assert service_calls == [((), {"wait": False})]
assert foreground_calls == [((tmp_path,), {"wait": False})]
assert service_calls == [((tmp_path,), {"wait": False})]
assert worker_calls == [{"force": True, "archive_service_logs": False}]

View file

@ -440,7 +440,7 @@ def test_no_bare_except_pass():
# ── AST-based function size check ───────────────────────────────
_SKIP_DIRS = {'.git', '__pycache__', 'tests', 'python-standalone', 'build', 'dist',
'venv', '.venv', 'node_modules', 'assets', '.pytest_cache'}
'venv', '.venv', 'node_modules', 'assets', 'devtools', '.pytest_cache'}
def _get_function_sizes():

View file

@ -9,7 +9,9 @@ Verifies:
- search_code tool works
"""
import inspect
import json
import os
import base64
import pathlib
import re
import sys
@ -569,6 +571,9 @@ def test_protected_black_box_artifact_policy_blocks_introspection(tmp_path, monk
direct_cmd = [str(protected)]
protected.write_text("#!/bin/sh\nprintf 'reference\\n'\n", encoding="utf-8")
generated.write_text("#!/bin/sh\nprintf 'generated\\n'\n", encoding="utf-8")
protected_dir = repo / "protected_dir"
protected_dir.mkdir()
(protected_dir / "secret.txt").write_text("secret\n", encoding="utf-8")
protected.chmod(0o755)
generated.chmod(0o755)
task_contract = build_task_contract({
@ -580,6 +585,12 @@ def test_protected_black_box_artifact_policy_blocks_introspection(tmp_path, monk
"paths": [str(protected)],
"allow": ["execute"],
"deny": ["read_bytes", "copy", "hash", "static_introspection", "dynamic_trace", "debug"],
},
{
"id": "reference-dir",
"role": "black_box_reference",
"paths": [str(protected_dir)],
"allow": ["execute"],
}
]
}
@ -597,6 +608,78 @@ def test_protected_black_box_artifact_policy_blocks_introspection(tmp_path, monk
assert "RESOURCE_POLICY_BLOCKED" not in direct
assert "reference" in direct
assert "RESOURCE_POLICY_BLOCKED" in registry.execute("read_file", {"path": protected.name})
protected_content = protected.read_text(encoding="utf-8")
write_attempt = registry.execute("write_file", {"path": protected.name, "content": "tamper\n"})
assert "RESOURCE_POLICY_BLOCKED" in write_attempt
assert protected.read_text(encoding="utf-8") == protected_content
edit_attempt = registry.execute("edit_text", {"path": protected.name, "old_str": "reference", "new_str": "tamper"})
assert "RESOURCE_POLICY_BLOCKED" in edit_attempt
assert protected.read_text(encoding="utf-8") == protected_content
shell_write_attempt = registry.execute(
"run_command",
{"cmd": ["sh", "-c", f"printf tamper > {protected.name}"], "cwd": str(repo)},
)
assert "RESOURCE_POLICY_BLOCKED" in shell_write_attempt
assert protected.read_text(encoding="utf-8") == protected_content
shell_delete_attempt = registry.execute(
"run_command",
{"cmd": ["rm", protected.name], "cwd": str(repo)},
)
assert "RESOURCE_POLICY_BLOCKED" in shell_delete_attempt
assert protected.exists()
recursive_delete_attempt = registry.execute(
"run_command",
{"cmd": ["rm", "-rf", "."], "cwd": str(repo)},
)
assert "RESOURCE_POLICY_BLOCKED" in recursive_delete_attempt
assert protected.exists()
glob_delete_attempt = registry.execute(
"run_command",
{"cmd": ["sh", "-c", "rm -rf *"], "cwd": str(repo)},
)
assert "RESOURCE_POLICY_BLOCKED" in glob_delete_attempt
assert protected.exists()
glob_read_attempt = registry.execute(
"run_command",
{"cmd": ["sh", "-c", "cat *"], "cwd": str(repo)},
)
assert "RESOURCE_POLICY_BLOCKED" in glob_read_attempt
find_exec_read = registry.execute(
"run_command",
{"cmd": ["find", ".", "-type", "f", "-exec", "cat", "{}", "+"], "cwd": str(repo)},
)
assert "RESOURCE_POLICY_BLOCKED" in find_exec_read
find_delete = registry.execute(
"run_command",
{"cmd": ["find", ".", "-delete"], "cwd": str(repo)},
)
assert "RESOURCE_POLICY_BLOCKED" in find_delete
assert protected.exists()
pathless_find_exec_read = registry.execute(
"run_command",
{"cmd": ["find", "-type", "f", "-exec", "cat", "{}", "+"], "cwd": str(repo)},
)
assert "RESOURCE_POLICY_BLOCKED" in pathless_find_exec_read
pathless_find_delete = registry.execute(
"run_command",
{"cmd": ["find", "-delete"], "cwd": str(repo)},
)
assert "RESOURCE_POLICY_BLOCKED" in pathless_find_delete
assert protected.exists()
safe_interpreter = registry.execute(
"run_command",
{
"cmd": [
sys.executable,
"-c",
"print(1)",
],
"cwd": str(repo),
},
)
assert "RESOURCE_POLICY_BLOCKED" not in safe_interpreter
assert "1" in safe_interpreter
assert "RESOURCE_POLICY_BLOCKED" in registry.execute("list_files", {"path": protected_dir.name})
interpreter_read = registry.execute(
"run_command",
{
@ -620,6 +703,18 @@ def test_protected_black_box_artifact_policy_blocks_introspection(tmp_path, monk
},
)
assert "RESOURCE_POLICY_BLOCKED" in relative_interpreter_read
versioned_interpreter_read = registry.execute(
"run_command",
{
"cmd": [
"python3.12",
"-c",
f"from pathlib import Path; print(Path({protected.name!r}).read_bytes())",
],
"cwd": str(repo),
},
)
assert "RESOURCE_POLICY_BLOCKED" in versioned_interpreter_read
constructed_path_read = registry.execute(
"run_command",
{
@ -663,6 +758,88 @@ def test_protected_black_box_artifact_policy_blocks_introspection(tmp_path, monk
assert "RESOURCE_POLICY_BLOCKED" in env_assignment_read
shell_script_read = registry.execute("run_command", {"cmd": ["sh", str(protected)]})
assert "RESOURCE_POLICY_BLOCKED" in shell_script_read
for cmd in (
["cmd.exe", "/c", "type", protected.name],
["cmd.exe", "/c", "copy", protected.name, str(repo / "copy.cmd")],
["cmd.exe", "/c", "xcopy", protected.name, str(repo / "copy-dir")],
["powershell.exe", "-Command", "Get-Content", protected.name],
["powershell.exe", "-Command", "Select-String", "reference", protected.name],
["powershell.exe", "-Command", "Copy-Item", protected.name, str(repo / "copy.ps1")],
["pwsh", "-Command", "Get-FileHash", protected.name],
["cmd.exe", "/c", "certutil", "-hashfile", protected.name],
):
result = registry.execute("run_command", {"cmd": cmd, "cwd": str(repo)})
assert "RESOURCE_POLICY_BLOCKED" in result, cmd
encoded_read = base64.b64encode(f"Get-Content {protected.name}".encode("utf-16le")).decode("ascii")
for cmd in (
["powershell.exe", "-EncodedCommand", encoded_read],
["pwsh", "-enc", encoded_read],
):
result = registry.execute("run_command", {"cmd": cmd, "cwd": str(repo)})
assert "RESOURCE_POLICY_BLOCKED" in result, cmd
search_direct = registry.execute("search_code", {"query": "reference", "path": protected.name})
assert "RESOURCE_POLICY_BLOCKED" in search_direct
search_protected_dir = registry.execute("search_code", {"query": "secret", "path": protected_dir.name})
assert "RESOURCE_POLICY_BLOCKED" in search_protected_dir
grep_read = registry.execute("run_command", {"cmd": ["grep", "reference", str(protected)]})
assert "RESOURCE_POLICY_BLOCKED" in grep_read
grep_recursive = registry.execute("run_command", {"cmd": ["grep", "-R", "reference", "."], "cwd": str(repo)})
assert "RESOURCE_POLICY_BLOCKED" in grep_recursive
rg_read = registry.execute("run_command", {"cmd": ["rg", "reference", str(protected)]})
assert "RESOURCE_POLICY_BLOCKED" in rg_read
rg_recursive = registry.execute("run_command", {"cmd": ["rg", "reference", "."], "cwd": str(repo)})
assert "RESOURCE_POLICY_BLOCKED" in rg_recursive
copy_recursive = registry.execute("run_command", {"cmd": ["cp", "-R", ".", str(repo / "copy")], "cwd": str(repo)})
assert "RESOURCE_POLICY_BLOCKED" in copy_recursive
for cmd in (
["git", "diff", "--", protected.name],
["git", "diff"],
["git", "show", f"HEAD:{protected.name}"],
["git", "show", "HEAD"],
["git", "grep", "reference", "--", protected.name],
["git", "grep", "reference"],
["git", "cat-file", "-p", f"HEAD:{protected.name}"],
["git", "log", "-p", "--", protected.name],
["git", "log", "-p"],
):
result = registry.execute("run_command", {"cmd": cmd, "cwd": str(repo)})
assert "RESOURCE_POLICY_BLOCKED" in result, cmd
assert "RESOURCE_POLICY_BLOCKED" in registry.execute("vcs_diff", {"path": protected.name})
assert "RESOURCE_POLICY_BLOCKED" in registry.execute("vcs_diff", {})
import ouroboros.code_intelligence as code_intelligence
original_file_fact = code_intelligence._file_fact
def guarded_file_fact(repo_root, path):
assert pathlib.Path(path).resolve(strict=False) != protected.resolve(strict=False)
return original_file_fact(repo_root, path)
monkeypatch.setattr(code_intelligence, "_file_fact", guarded_file_fact)
digest = registry.execute("codebase_digest", {})
assert protected.name not in digest
assert generated.name in digest
run_output_export = registry.execute("run_command", {"cmd": direct_cmd, "outputs": [protected.name], "cwd": str(repo)})
assert "ARTIFACT_OUTPUT_ERROR" in run_output_export
assert "RESOURCE_POLICY_BLOCKED" in run_output_export
script_output_export = registry.execute(
"run_script",
{"interpreter": "python3", "script": "print('ok')", "outputs": [protected.name], "cwd": str(repo)},
)
assert "RESOURCE_POLICY_BLOCKED" in script_output_export
service_cmd = ["cmd.exe", "/c", "ping", "127.0.0.1", "-n", "30"] if os.name == "nt" else ["sleep", "30"]
service_start = registry.execute(
"start_service",
{
"name": "protected-output",
"cmd": service_cmd,
"cwd": str(repo),
"outputs": [protected.name],
},
)
assert "protected-output" in service_start
service_stop = registry.execute("stop_service", {"name": "protected-output"})
assert "ARTIFACT_OUTPUT_ERROR" in service_stop
assert "RESOURCE_POLICY_BLOCKED" in service_stop
for cmd in (
["strings", str(protected)],
["objdump", "-d", str(protected)],
@ -673,14 +850,145 @@ def test_protected_black_box_artifact_policy_blocks_introspection(tmp_path, monk
["lldb", str(protected)],
["cp", str(protected), str(repo / "copy.sh")],
["dd", f"if={protected}", f"of={repo / 'copy2.sh'}"],
["tar", "-czf", str(repo / "out.tgz"), protected.name],
["tar", "-czf", str(repo / "tree.tgz"), "."],
["zip", str(repo / "out.zip"), protected.name],
["rsync", protected.name, str(repo / "copy.sh")],
):
result = registry.execute("run_command", {"cmd": cmd})
result = registry.execute("run_command", {"cmd": cmd, "cwd": str(repo)})
assert "RESOURCE_POLICY_BLOCKED" in result, cmd
generated_result = registry.execute("run_command", {"cmd": ["strings", str(generated)]})
assert "RESOURCE_POLICY_BLOCKED" not in generated_result
def test_protected_black_box_recursive_policy_maps_executor_backend_paths(tmp_path, monkeypatch):
from ouroboros.contracts.task_contract import build_task_contract
from ouroboros.tools.registry import ToolContext, ToolRegistry
system_repo = tmp_path / "system"
workspace = tmp_path / "workspace"
data = tmp_path / "data"
for path in (system_repo, workspace, data):
path.mkdir(parents=True, exist_ok=True)
protected = workspace / "executable"
protected.write_text("reference bytes\n", encoding="utf-8")
task_contract = build_task_contract({
"resource_policy": {
"protected_artifacts": [
{
"id": "reference",
"role": "black_box_reference",
"paths": ["/workspace/executable"],
"allow": ["execute"],
"deny": ["read_bytes", "copy", "hash", "static_introspection", "dynamic_trace", "debug"],
}
]
}
})
registry = ToolRegistry(repo_dir=system_repo, drive_root=data)
registry.set_context(
ToolContext(
repo_dir=system_repo,
drive_root=data,
workspace_root=workspace,
workspace_mode="external",
task_contract=task_contract,
task_metadata={"task_contract": task_contract},
executor_ref={
"type": "docker_exec",
"id": "pb-container",
"container_name": "pb-container",
"network": "none",
"workspace_host_path": str(workspace),
"workspace_backend_path": "/workspace",
},
)
)
monkeypatch.setattr("ouroboros.safety.check_safety", lambda *a, **k: (True, ""))
grep_recursive = registry.execute("run_command", {"cmd": ["grep", "-R", "reference", "."], "cwd": str(workspace)})
copy_recursive = registry.execute("run_command", {"cmd": ["cp", "-R", ".", str(workspace / "copy")], "cwd": str(workspace)})
import ouroboros.code_intelligence as code_intelligence
original_file_fact = code_intelligence._file_fact
def guarded_file_fact(repo_root, path):
assert pathlib.Path(path).resolve(strict=False) != protected.resolve(strict=False)
return original_file_fact(repo_root, path)
monkeypatch.setattr(code_intelligence, "_file_fact", guarded_file_fact)
digest = registry.execute("codebase_digest", {})
assert "RESOURCE_POLICY_BLOCKED" in grep_recursive
assert "RESOURCE_POLICY_BLOCKED" in copy_recursive
assert "executable" not in digest
def test_schedule_subagent_inherits_workspace_executor_ref(tmp_path, monkeypatch):
from ouroboros.contracts.task_contract import build_task_contract
from ouroboros.tools.registry import ToolContext, ToolRegistry
system_repo = tmp_path / "system"
workspace = tmp_path / "workspace"
data = tmp_path / "data"
for path in (system_repo, workspace, data):
path.mkdir(parents=True)
task_contract = build_task_contract({
"resource_policy": {
"protected_artifacts": [
{
"id": "reference",
"role": "black_box_reference",
"paths": ["/workspace/executable"],
"allow": ["execute"],
}
]
}
})
executor_ref = {
"type": "docker_exec",
"id": "pb-container",
"container_name": "pb-container",
"network": "none",
"workspace_host_path": str(workspace),
"workspace_backend_path": "/workspace",
}
registry = ToolRegistry(repo_dir=system_repo, drive_root=data)
ctx = ToolContext(
repo_dir=system_repo,
drive_root=data,
workspace_root=workspace,
workspace_mode="external",
task_id="parent-task",
task_contract=task_contract,
task_metadata={"task_contract": task_contract},
executor_ref=executor_ref,
)
registry.set_context(ctx)
monkeypatch.setenv("OUROBOROS_MAX_SUBAGENT_DEPTH", "4")
result = registry.execute(
"schedule_subagent",
{
"objective": "Inspect the workspace contract.",
"expected_output": "A concise report.",
"role": "auditor",
"model_lane": "light",
},
)
assert "Subagent request queued" in result
assert ctx.pending_events
event = ctx.pending_events[0]
assert event["executor_ref"] == executor_ref
assert event["metadata"]["executor_ref"] == executor_ref
child_id = event["task_id"]
persisted = json.loads((data / "task_results" / f"{child_id}.json").read_text(encoding="utf-8"))
assert persisted["executor_ref"] == executor_ref
assert persisted["task_contract"]["resource_policy"]["protected_artifacts"][0]["paths"] == ["/workspace/executable"]
def test_capability_omission_manifest_surfaces_extension_discovery_failure(tmp_path, monkeypatch):
from ouroboros import extension_loader
from ouroboros.tools import tool_discovery
@ -772,6 +1080,84 @@ def test_local_readonly_subagent_data_read_denies_secret_files(tmp_path):
assert "{}" in readable
def test_runtime_data_write_blocks_workspace_executor_control_state(tmp_path, monkeypatch):
from ouroboros.tools.registry import ToolContext, ToolRegistry
repo = tmp_path / "repo"
data = tmp_path / "data"
repo.mkdir()
data.mkdir()
state_dir = data / "state" / "workspace_executor_processes"
state_dir.mkdir(parents=True)
existing = state_dir / "foreground-forged.json"
existing.write_text("original", encoding="utf-8")
registry = ToolRegistry(repo_dir=repo, drive_root=data)
registry.set_context(ToolContext(repo_dir=repo, drive_root=data))
monkeypatch.setattr("ouroboros.safety.check_safety", lambda *a, **k: (True, ""))
direct_write = registry.execute(
"write_file",
{
"root": "runtime_data",
"path": "state/workspace_executor_processes/foreground-forged.json",
"content": "{}",
},
)
assert "DATA_WRITE_BLOCKED" in direct_write
assert existing.read_text(encoding="utf-8") == "original"
nested_write = registry.execute(
"write_file",
{
"root": "runtime_data",
"path": "state/headless_tasks/child/data/state/workspace_executor_processes/foreground-forged.json",
"content": "{}",
},
)
assert "DATA_WRITE_BLOCKED" in nested_write
edit = registry.execute(
"edit_text",
{
"root": "runtime_data",
"path": "state/workspace_executor_processes/foreground-forged.json",
"old_str": "original",
"new_str": "tampered",
},
)
assert "EDIT_TEXT_BLOCKED" in edit
assert existing.read_text(encoding="utf-8") == "original"
shell_write = registry.execute(
"run_command",
{
"cmd": [
sys.executable,
"-c",
(
"from pathlib import Path; "
f"Path(r'{existing}').write_text('{{\"owner\":\"ouroboros_workspace_executor\"}}')"
),
],
},
)
assert "WORKSPACE_EXECUTOR_STATE_WRITE_BLOCKED" in shell_write
assert existing.read_text(encoding="utf-8") == "original"
node_eval_write = registry.execute(
"run_command",
{
"cmd": [
"node",
"-e",
f"require('fs').writeFileSync({str(existing)!r}, '{{}}')",
],
},
)
assert "WORKSPACE_EXECUTOR_STATE_WRITE_BLOCKED" in node_eval_write
assert existing.read_text(encoding="utf-8") == "original"
def test_local_readonly_subagent_repo_read_denies_secret_files(tmp_path):
from ouroboros.contracts.task_constraint import TaskConstraint
from ouroboros.tools.registry import ToolContext, ToolRegistry

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,214 @@
from __future__ import annotations
import json
import subprocess
def _write_docker_records(state_dir, *, foreground_pidfile="/tmp/ouroboros-exec-test.pid", service_pid="12345"):
state_dir.mkdir(parents=True)
(state_dir / "foreground-docker.json").write_text(
json.dumps(
{
"id": "foreground-docker",
"schema_version": 1,
"owner": "ouroboros_workspace_executor",
"record_type": "foreground",
"executor_type": "docker_exec",
"executor_id": "docker",
"host_pid": 0,
"container_name": "bench",
"backend_pidfile": foreground_pidfile,
}
),
encoding="utf-8",
)
(state_dir / "service-docker.json").write_text(
json.dumps(
{
"id": "service-docker",
"schema_version": 1,
"owner": "ouroboros_workspace_executor",
"record_type": "service",
"service_id": "task:svc",
"task_id": "task",
"name": "svc",
"executor_type": "docker_exec",
"executor_id": "docker",
"container_name": "bench",
"backend_pid": service_pid,
}
),
encoding="utf-8",
)
def _install_live_docker_service(workspace_executor, tmp_path):
executor = workspace_executor.ExecutorRef(
kind="docker_exec",
executor_id="docker",
network="none",
mappings=(),
container_name="bench",
)
with workspace_executor._STATE_LOCK:
workspace_executor._SERVICES.clear()
workspace_executor._SERVICES["task:live"] = workspace_executor._ExecutorService(
service_id="task:live",
task_id="task",
name="live",
executor=executor,
cmd=["sleep", "30"],
host_cwd=tmp_path,
backend_cwd="/workspace",
cwd_root="active_workspace",
outputs=[],
before_outputs={},
backend_pid="67890",
)
def test_executor_panic_cleanup_wait_false_uses_bounded_docker_stop(tmp_path, monkeypatch):
import ouroboros.workspace_executor as workspace_executor
data = tmp_path / "data"
state_dir = data / "state" / "workspace_executor_processes"
_write_docker_records(state_dir)
_install_live_docker_service(workspace_executor, tmp_path)
docker_run_calls: list[list[str]] = []
def fake_docker_wait(cmd, **kwargs):
docker_run_calls.append([str(part) for part in cmd])
return subprocess.CompletedProcess(cmd, 0, stdout="", stderr="")
class FakePopen:
def __init__(self, cmd, **kwargs):
raise AssertionError("panic Docker cleanup must not spawn untracked helpers")
monkeypatch.setattr(workspace_executor.subprocess, "run", fake_docker_wait)
monkeypatch.setattr(workspace_executor.subprocess, "Popen", FakePopen)
killed_foreground = workspace_executor.kill_all_foreground(data, wait=False)
killed_services = workspace_executor.kill_all_services(data, wait=False)
assert len(docker_run_calls) == 3
assert all(call[:2] == ["docker", "exec"] for call in docker_run_calls)
assert any(item.get("executor_type") == "docker_exec" for item in killed_foreground)
assert any(item.get("state") == "stopped" for item in killed_services)
assert all(item.get("cleanup_dispatched") is True for item in killed_foreground + killed_services)
assert not list(state_dir.glob("*.json"))
with workspace_executor._STATE_LOCK:
assert "task:live" not in workspace_executor._SERVICES
def test_docker_executor_confirmed_cleanup_failure_preserves_records(tmp_path, monkeypatch):
import ouroboros.workspace_executor as workspace_executor
data = tmp_path / "data"
state_dir = data / "state" / "workspace_executor_processes"
_write_docker_records(state_dir)
_install_live_docker_service(workspace_executor, tmp_path)
def fake_failed_docker_wait(cmd, **kwargs):
return subprocess.CompletedProcess(cmd, 1, stdout="", stderr="permission denied")
monkeypatch.setattr(workspace_executor.subprocess, "run", fake_failed_docker_wait)
killed_foreground = workspace_executor.kill_all_foreground(data, wait=True)
killed_services = workspace_executor.kill_all_services(data, wait=True)
assert any(item.get("cleanup_dispatched") is False for item in killed_foreground)
assert any(item.get("state") == "cleanup_pending" for item in killed_foreground + killed_services)
assert {path.name for path in state_dir.glob("*.json")} == {"foreground-docker.json", "service-docker.json"}
with workspace_executor._STATE_LOCK:
assert "task:live" in workspace_executor._SERVICES
workspace_executor._SERVICES.clear()
def test_executor_cleanup_ignores_unowned_forged_process_records(tmp_path, monkeypatch):
import ouroboros.workspace_executor as workspace_executor
data = tmp_path / "data"
state_dir = data / "state" / "workspace_executor_processes"
state_dir.mkdir(parents=True)
(state_dir / "foreground-forged.json").write_text(
json.dumps(
{
"id": "foreground-forged",
"record_type": "foreground",
"executor_type": "local",
"host_pid": 1,
}
),
encoding="utf-8",
)
monkeypatch.setattr(
workspace_executor,
"_kill_host_pid",
lambda _pid: (_ for _ in ()).throw(AssertionError("forged record should be ignored")),
)
assert workspace_executor.kill_all_foreground(data, wait=False) == []
def test_executor_cleanup_ignores_owner_shaped_forged_host_pid_records(tmp_path, monkeypatch):
import ouroboros.workspace_executor as workspace_executor
data = tmp_path / "data"
state_dir = data / "state" / "workspace_executor_processes"
state_dir.mkdir(parents=True)
(state_dir / "foreground-forged.json").write_text(
json.dumps(
{
"id": "foreground-forged",
"schema_version": 1,
"owner": "ouroboros_workspace_executor",
"record_type": "foreground",
"executor_type": "local",
"host_pid": 1,
}
),
encoding="utf-8",
)
monkeypatch.setattr(
workspace_executor,
"_kill_host_pid",
lambda _pid: (_ for _ in ()).throw(AssertionError("owner-shaped forged record should be ignored")),
)
assert workspace_executor.kill_all_foreground(data, wait=False) == []
def test_executor_cleanup_ignores_pidless_docker_service_records(tmp_path, monkeypatch):
import ouroboros.workspace_executor as workspace_executor
data = tmp_path / "data"
state_dir = data / "state" / "workspace_executor_processes"
state_dir.mkdir(parents=True)
(state_dir / "service-docker.json").write_text(
json.dumps(
{
"id": "service-docker",
"schema_version": 1,
"owner": "ouroboros_workspace_executor",
"record_type": "service",
"service_id": "task:svc",
"task_id": "task",
"name": "svc",
"executor_type": "docker_exec",
"executor_id": "docker",
"container_name": "bench",
}
),
encoding="utf-8",
)
monkeypatch.setattr(
workspace_executor.subprocess,
"run",
lambda *a, **k: (_ for _ in ()).throw(AssertionError("pidless docker service record should be ignored")),
)
assert workspace_executor.kill_all_services(data, wait=False) == []

View file

@ -196,6 +196,44 @@
* @property {Object=} grants
*/
/**
* @typedef {Object} ExecutorRef
* @property {"local"|"docker_exec"} type
* @property {string=} id
* @property {"host"|"none"=} network
* @property {string=} workspace_host_path
* @property {string=} workspace_backend_path
* @property {string=} container_name Required when type is "docker_exec".
* @property {Object[]=} path_mappings
*/
/**
* @typedef {Object} TaskCreateRequest
* @property {string} description
* @property {string=} task_id
* @property {string=} type
* @property {number=} chat_id
* @property {number=} depth
* @property {string=} session_id
* @property {string=} workspace_root
* @property {"external"|"self"=} workspace_mode
* @property {"forked"|"empty"|"shared"=} memory_mode
* @property {Object[]=} attachments
* @property {Object=} allowed_resources
* @property {Object=} resource_policy
* @property {ExecutorRef=} executor_ref
* @property {string=} deadline_at
* @property {number=} timeout_sec
* @property {number=} timeout
* @property {string=} context
* @property {string=} expected_output
* @property {string=} constraints
* @property {boolean=} context_requires_self_body_docs
* @property {string=} actor_id Top-level task actor/provenance id; metadata.actor_id is reserved.
* @property {string=} source Top-level task source/provenance label.
* @property {Object=} metadata Arbitrary task metadata; executor_ref/workspace_executor keys are reserved.
*/
/**
* @typedef {Object} TaskCreateResponse
* @property {boolean} ok
@ -269,4 +307,4 @@
* @property {boolean=} ok
*/
export const GATEWAY_CONTRACT_VERSION = '6.18.1';
export const GATEWAY_CONTRACT_VERSION = '6.19.0-rc.1';

View file

@ -1,6 +1,6 @@
{
"name": "ouroboros-web",
"version": "6.18.1",
"version": "6.19.0-rc.1",
"private": true,
"type": "module",
"description": "Ouroboros browser UI package boundary",