mirror of
https://github.com/razzant/ouroboros.git
synced 2026-08-05 00:29:46 +00:00
feat(devtools-benchmarks): add official benchmark harnesses and workspace executor
This commit is contained in:
parent
3a0500e43e
commit
d863197cf6
76 changed files with 7979 additions and 589 deletions
1
.github/workflows/ci.yml
vendored
1
.github/workflows/ci.yml
vendored
|
|
@ -31,6 +31,7 @@ on:
|
|||
- 'build_windows.ps1'
|
||||
- 'Dockerfile'
|
||||
- 'scripts/**'
|
||||
- 'devtools/**'
|
||||
- 'packaging/**'
|
||||
- 'VERSION'
|
||||
- 'README.md'
|
||||
|
|
|
|||
13
README.md
13
README.md
|
|
@ -7,7 +7,7 @@
|
|||
[](https://github.com/razzant/ouroboros/releases)
|
||||
[](https://github.com/razzant/ouroboros/releases)
|
||||
[](https://github.com/razzant/OuroborosHub)
|
||||
[](VERSION)
|
||||
[](VERSION)
|
||||
|
||||
A self-modifying AI agent that writes its own code, rewrites its own mind, and evolves autonomously. Born February 16, 2026.
|
||||
|
||||
|
|
@ -158,8 +158,11 @@ the task id immediately.
|
|||
`schedule add/list/remove` manages queue-backed scheduled tasks through the same
|
||||
gateway and supervisor queue; schedules use standard 5-field cron, host-local
|
||||
timezone by default, and a single catch-up run after downtime.
|
||||
Benchmark helpers under `scripts/` expect clean, local, per-instance checkouts;
|
||||
they do not reset or commit target repositories.
|
||||
Benchmark helpers live under `devtools/benchmarks/`. They are tracked
|
||||
operator tooling, reviewed when touched, and kept out of runtime imports. They
|
||||
prepare official benchmark inputs/runs for ProgramBench, Terminal-Bench/Harbor,
|
||||
SWE-bench, SWE-bench Pro, and OSWorld logs inspection without replacing
|
||||
official scoring harnesses.
|
||||
|
||||
You can also override the bind address and port:
|
||||
|
||||
|
|
@ -517,13 +520,13 @@ the contribution guide only routes to those sources.
|
|||
|
||||
| Version | Date | Description |
|
||||
|---------|------|-------------|
|
||||
| 6.19.0-rc.1 | 2026-06-06 | **feat(devtools-benchmarks): official benchmark devtools and executor-backed workspaces.** Adds tracked `devtools/benchmarks/` tooling for ProgramBench, Terminal-Bench/Harbor, SWE-bench, SWE-bench Pro, and OSWorld logs-only inspection while keeping official harnesses as the scoring source of truth. External workspace process/service tools can now route through a host-owned `executor_ref` backend, including Docker `network=none` enforcement for benchmark cleanrooms, and broad review Atlas packs keep unrelated devtools as `excluded_dir` coverage-manifest entries while fully reviewing touched devtools files. |
|
||||
| 6.18.1 | 2026-06-06 | **fix(release): keep macOS signing compatible with mobile browser checks.** macOS packages now bundle Chromium headless shell only and use the managed Playwright cache for first-use WebKit downloads, avoiding the nested `.framework`/`.xpc`/`.tbd` payload that breaks hardened-runtime codesigning inside a PyInstaller app. Linux, Docker, and Windows packages still bundle Chromium and WebKit; `engine=webkit` plus iPhone device descriptors remains the expected mobile/iOS verification path. |
|
||||
| 6.18.0 | 2026-06-05 | **feat(frontend-browser): mobile-grade browser checks and UI polish.** Browser tools now support explicit `engine=chromium|webkit` and Playwright device descriptors, with packaged Chromium/WebKit install paths and CI smoke coverage for both engines. The chat composer uses a responsive glass toolbar: desktop controls stay inside the frosted field, while mobile lifts Consilium and Low/Max above the textarea and keeps Send inside. Chat-wide drag/drop stages attachments until Send; nested subagent child cards stay visible but collapsed by default with role-first headings; widget cards can be reordered via a persisted owner-local UI preference endpoint. |
|
||||
| 6.17.0 | 2026-06-04 | **release: deep core capability envelope and typed task outcomes.** Task results move to a `task_contract` + `outcome_axes` + `verification_ledger` contract: objective success comes only from `task_acceptance_review` and otherwise stays `not_evaluated`, while lifecycle, execution health, artifact state, review, and objective are separate axes. Main/direct/evolution tasks start with the full enabled capability envelope, workspace parents can delegate through local-readonly subagents, and those children inherit workspace/deadline/resource context while staying locally readonly and allowing enabled external tools by owner policy. Headless/API/CLI tasks get absolute deadlines, finalization grace, explicit patch states (`ready_with_changes`, `ready_no_changes`, `missing`, `failed`), public result projection without new `result_status`, and task-aware docs layout. Evolution campaigns now count absorbed cycles only after a reviewed self-mod commit survives restart verification. |
|
||||
| 6.16.0 | 2026-06-04 | **feat(extensions): reconcile worker-enabled companions in the server.** Worker-side extension enables/disables now write durable per-request markers under `state/extension_reconcile/`, and the server lifespan runs a lightweight pickup task that reconciles the server registry, starts any registered-but-missing companion processes, stops companions for disabled skills, and removes processed markers. This closes the companion-only gap left by per-process extension registries: agent `toggle_skill` and post-review auto-enable remain fire-and-forget, but the server now catches up without waiting for a route hit or restart. |
|
||||
| 6.15.0 | 2026-06-04 | **feat(extensions): full out-of-process extension parity + durable health diff.** A new execution-mode capability matrix in `contracts/plugin_api.py` (PluginAPI 1.3) is the single source of truth for what an out-of-process (isolated-dep/native) extension may call, surfaced to `register(api)` through `get_runtime_info()` so a skill negotiates instead of crashing mid-registration. `on_unload` now runs at child teardown; `send_ws_message` relays through a new Host Service loopback bridge (`POST /ui/ws-message`, identity derived from the skill token, host-side namespacing); and `register_companion_process` becomes a cataloged, host-spawned and supervised surface so isolated-dep skills can run long-lived background work (a per-call child only blocks `subscribe_event` and `register_supervised_task`, which a companion replaces). Enabling an extension now verifies it via the real out-of-process catalog/register dry-run and reverts `enabled` on failure — on every enable path (UI toggle, agent tool, and post-review auto-enable) — so a skill is never left enabled-but-broken. A per-skill `health.json` records live→broken regressions across restarts and surfaces them through the health invariants, the startup verification, and the Installed UI. The chat composer replaces the Send dropdown with a Low\|Max segmented control and a one-shot Consilium pill. |
|
||||
| 6.14.0 | 2026-06-03 | **feat(provider): add GigaChat as a first-class provider.** Adds Sber GigaChat (`gigachat::` model values) routed through the native `gigachat` library instead of the OpenAI SDK, since GigaChat is not OpenAI-compatible: OpenAI tools map to GigaChat functions with a single `function_call` per turn, the system message stays first while later system-reminders are demoted to user, and function results are JSON-wrapped; `reasoning_effort` is omitted on the GigaChat path so replies are not consumed by hidden reasoning. A GigaChat-only install is self-sufficient: model slots and the review and scope reviewer slots all resolve to `gigachat::` models like the Cloud.ru direct-provider pattern (an owner without a 1M-context reviewer can opt into the audited degraded advisory scope review), and this single-isolated-provider expectation is documented as an invariant in DEVELOPMENT.md and the review checklist. Sber tariff-based static pricing backs cost tracking (GigaChat-2 family at published rates; the GigaChat-3 flagship approximated at the Max tier), and the `gigachat` dependency is pinned below 0.3. |
|
||||
Older releases are preserved in Git tags and GitHub releases. The 6.0.0 through 6.9.0-rc.2 rows, the 5.2.0 through 5.33.0-rc.6 rows, and former `4.0.0` rows are rolled off to respect the P9 changelog cap; their full bodies remain at their git tags.
|
||||
Older releases are preserved in Git tags and GitHub releases. The 6.0.0 through 6.14.0 rows, the 5.2.0 through 5.33.0-rc.6 rows, and former `4.0.0` rows are rolled off to respect the P9 changelog cap; their full bodies remain at their git tags.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
2
VERSION
2
VERSION
|
|
@ -1 +1 @@
|
|||
6.18.1
|
||||
6.19.0-rc.1
|
||||
|
|
|
|||
13
devtools/README.md
Normal file
13
devtools/README.md
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
# Ouroboros Devtools
|
||||
|
||||
`devtools/` contains operator-side and benchmark support code that should be
|
||||
versioned with Ouroboros without becoming part of the runtime core.
|
||||
|
||||
Rules:
|
||||
|
||||
- Generated logs, datasets, run outputs, Docker layers, and secrets do not live
|
||||
here.
|
||||
- Default benchmark outputs go under `/Users/anton/Ouroboros/bench_runs/`.
|
||||
- Runtime modules must not import `devtools`.
|
||||
- This is not an immune-system bypass: touched files are reviewed normally.
|
||||
- Promote code out of `devtools` only through a separate reviewed runtime plan.
|
||||
4
devtools/__init__.py
Normal file
4
devtools/__init__.py
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
"""Operator/devtools namespace.
|
||||
|
||||
Code here is tracked and reviewed when touched, but it is not runtime core.
|
||||
"""
|
||||
15
devtools/benchmarks/README.md
Normal file
15
devtools/benchmarks/README.md
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
# Benchmark Devtools
|
||||
|
||||
This directory contains thin adapters around official benchmark harnesses. The
|
||||
adapters prepare Ouroboros tasks, capture artifacts, and preserve traces; they
|
||||
do not replace official scoring.
|
||||
|
||||
Supported surfaces:
|
||||
|
||||
- ProgramBench: official `programbench eval/info` and cleanroom submission
|
||||
layout.
|
||||
- Terminal-Bench: Harbor installed-agent adapter.
|
||||
- SWE-bench and SWE-bench Verified: standard predictions JSONL for the official
|
||||
SWE-bench harness.
|
||||
- SWE-bench Pro: official Pro eval wrapper and patch-capture methodology.
|
||||
- OSWorld: stop-closed skeleton and logs normalizer only.
|
||||
1
devtools/benchmarks/__init__.py
Normal file
1
devtools/benchmarks/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
|||
"""Benchmark support helpers for official external harnesses."""
|
||||
1
devtools/benchmarks/common/__init__.py
Normal file
1
devtools/benchmarks/common/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
|||
"""Shared benchmark helper utilities."""
|
||||
27
devtools/benchmarks/common/manifests.py
Normal file
27
devtools/benchmarks/common/manifests.py
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
"""Benchmark manifest helpers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import pathlib
|
||||
from typing import Any
|
||||
|
||||
|
||||
def read_json(path: pathlib.Path) -> Any:
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def write_json(path: pathlib.Path, payload: Any) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def read_jsonl(path: pathlib.Path) -> list[dict[str, Any]]:
|
||||
rows: list[dict[str, Any]] = []
|
||||
for line in path.read_text(encoding="utf-8").splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
loaded = json.loads(line)
|
||||
if isinstance(loaded, dict):
|
||||
rows.append(loaded)
|
||||
return rows
|
||||
29
devtools/benchmarks/common/official_commands.py
Normal file
29
devtools/benchmarks/common/official_commands.py
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
"""Declarative official benchmark command builders."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pathlib
|
||||
|
||||
|
||||
def programbench_eval_cmd(run_root: pathlib.Path) -> list[str]:
|
||||
return ["programbench", "eval", str(run_root)]
|
||||
|
||||
|
||||
def programbench_info_cmd(run_root: pathlib.Path) -> list[str]:
|
||||
return ["programbench", "info", str(run_root)]
|
||||
|
||||
|
||||
def swebench_eval_cmd(dataset_name: str, predictions_path: pathlib.Path, run_id: str, workers: int = 1) -> list[str]:
|
||||
return [
|
||||
"python",
|
||||
"-m",
|
||||
"swebench.harness.run_evaluation",
|
||||
"--dataset_name",
|
||||
dataset_name,
|
||||
"--predictions_path",
|
||||
str(predictions_path),
|
||||
"--max_workers",
|
||||
str(int(workers)),
|
||||
"--run_id",
|
||||
run_id,
|
||||
]
|
||||
13
devtools/benchmarks/common/result_index.py
Normal file
13
devtools/benchmarks/common/result_index.py
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
"""Result indexing utilities shared by benchmark adapters."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import pathlib
|
||||
from typing import Any
|
||||
|
||||
|
||||
def append_result_index(run_dir: pathlib.Path, row: dict[str, Any]) -> None:
|
||||
run_dir.mkdir(parents=True, exist_ok=True)
|
||||
with (run_dir / "result_index.jsonl").open("a", encoding="utf-8") as fh:
|
||||
fh.write(json.dumps(row, ensure_ascii=False) + "\n")
|
||||
69
devtools/benchmarks/common/run_roots.py
Normal file
69
devtools/benchmarks/common/run_roots.py
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
"""Run-root helpers for benchmark devtools."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import time
|
||||
|
||||
|
||||
DEFAULT_BENCH_RUNS_ROOT = pathlib.Path("/Users/anton/Ouroboros/bench_runs")
|
||||
_SAFE_BENCHMARK_ID_RE = re.compile(r"^[A-Za-z0-9_.-]+$")
|
||||
|
||||
|
||||
def timestamp_run_id(prefix: str) -> str:
|
||||
safe = "".join(ch if ch.isalnum() or ch in ("-", "_") else "-" for ch in prefix).strip("-_")
|
||||
return f"{safe or 'run'}_{time.strftime('%Y%m%d_%H%M%S')}"
|
||||
|
||||
|
||||
def run_root(benchmark: str, run_id: str = "") -> pathlib.Path:
|
||||
root = pathlib.Path(os.environ.get("OUROBOROS_BENCH_RUNS_ROOT") or DEFAULT_BENCH_RUNS_ROOT)
|
||||
bench = "".join(ch if ch.isalnum() or ch in ("-", "_") else "-" for ch in benchmark).strip("-_")
|
||||
rid = run_id or timestamp_run_id(bench)
|
||||
return (root / bench / rid).resolve(strict=False)
|
||||
|
||||
|
||||
def ensure_outside_repo(path: pathlib.Path, repo_dir: pathlib.Path) -> pathlib.Path:
|
||||
resolved = pathlib.Path(path).expanduser().resolve(strict=False)
|
||||
repo = pathlib.Path(repo_dir).resolve(strict=False)
|
||||
try:
|
||||
resolved.relative_to(repo)
|
||||
except ValueError:
|
||||
resolved.mkdir(parents=True, exist_ok=True)
|
||||
return resolved
|
||||
raise ValueError(f"benchmark run output must not be under repo/: {resolved}")
|
||||
|
||||
|
||||
def safe_benchmark_id(value: str, *, field: str = "instance_id") -> str:
|
||||
text = str(value or "").strip()
|
||||
if (
|
||||
not text
|
||||
or text in {".", ".."}
|
||||
or "/" in text
|
||||
or "\\" in text
|
||||
or pathlib.PurePath(text).is_absolute()
|
||||
or not _SAFE_BENCHMARK_ID_RE.fullmatch(text)
|
||||
):
|
||||
raise ValueError(f"{field} must be a single safe path component")
|
||||
return text
|
||||
|
||||
|
||||
def safe_join_under(root: pathlib.Path, *parts: str) -> pathlib.Path:
|
||||
base = pathlib.Path(root).expanduser().resolve(strict=False)
|
||||
resolved = base.joinpath(*[str(part or "") for part in parts]).resolve(strict=False)
|
||||
try:
|
||||
resolved.relative_to(base)
|
||||
except ValueError as exc:
|
||||
raise ValueError(f"benchmark output path escapes run root: {resolved}") from exc
|
||||
return resolved
|
||||
|
||||
|
||||
def ensure_file_output_outside_repo(path: pathlib.Path, repo_dir: pathlib.Path) -> pathlib.Path:
|
||||
resolved = pathlib.Path(path).expanduser().resolve(strict=False)
|
||||
ensure_outside_repo(resolved.parent, repo_dir)
|
||||
return resolved
|
||||
|
||||
|
||||
def repo_root_from_devtools() -> pathlib.Path:
|
||||
return pathlib.Path(__file__).resolve().parents[3]
|
||||
43
devtools/benchmarks/common/secrets.py
Normal file
43
devtools/benchmarks/common/secrets.py
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
"""Secret-loading helpers that never print credential values."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
|
||||
|
||||
SECRET_KEYS = (
|
||||
"OPENROUTER_API_KEY",
|
||||
"OPENAI_API_KEY",
|
||||
"ANTHROPIC_API_KEY",
|
||||
"GITHUB_TOKEN",
|
||||
)
|
||||
|
||||
|
||||
def settings_path(default_home: pathlib.Path | None = None) -> pathlib.Path:
|
||||
home = default_home or pathlib.Path("/Users/anton/Ouroboros")
|
||||
return pathlib.Path(os.environ.get("OUROBOROS_SETTINGS_PATH") or home / "data" / "settings.json")
|
||||
|
||||
|
||||
def load_secret_env(path: pathlib.Path | None = None) -> dict[str, str]:
|
||||
values: dict[str, str] = {}
|
||||
for key in SECRET_KEYS:
|
||||
value = os.environ.get(key)
|
||||
if value:
|
||||
values[key] = value
|
||||
settings_file = path or settings_path()
|
||||
try:
|
||||
loaded = json.loads(settings_file.read_text(encoding="utf-8"))
|
||||
except Exception:
|
||||
loaded = {}
|
||||
if isinstance(loaded, dict):
|
||||
for key in SECRET_KEYS:
|
||||
value = loaded.get(key)
|
||||
if value and key not in values:
|
||||
values[key] = str(value)
|
||||
return values
|
||||
|
||||
|
||||
def redacted_env_summary(env: dict[str, str]) -> dict[str, bool]:
|
||||
return {key: bool(env.get(key)) for key in SECRET_KEYS}
|
||||
46
devtools/benchmarks/common/subprocesses.py
Normal file
46
devtools/benchmarks/common/subprocesses.py
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
"""Small subprocess helpers for benchmark adapters."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import pathlib
|
||||
import subprocess
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CommandResult:
|
||||
cmd: list[str]
|
||||
returncode: int
|
||||
stdout: str
|
||||
stderr: str
|
||||
|
||||
|
||||
def run_logged(cmd: list[str], *, cwd: pathlib.Path | None = None, timeout: int | None = None) -> CommandResult:
|
||||
proc = subprocess.run(
|
||||
[str(part) for part in cmd],
|
||||
cwd=str(cwd) if cwd else None,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
timeout=timeout,
|
||||
)
|
||||
return CommandResult([str(part) for part in cmd], proc.returncode, proc.stdout or "", proc.stderr or "")
|
||||
|
||||
|
||||
def write_command_result(path: pathlib.Path, result: CommandResult) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"cmd": result.cmd,
|
||||
"returncode": result.returncode,
|
||||
"stdout_chars": len(result.stdout),
|
||||
"stderr_chars": len(result.stderr),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
26
devtools/benchmarks/osworld/README.md
Normal file
26
devtools/benchmarks/osworld/README.md
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
# OSWorld Devtools
|
||||
|
||||
This directory is intentionally a stop-closed skeleton for v6.19.0-rc.1.
|
||||
|
||||
Local material currently available in this workspace is a logs-only bundle:
|
||||
|
||||
```text
|
||||
/Users/anton/Ouroboros/bench_logs/osworld_sample60_seed20260603_opus47_logs_only
|
||||
```
|
||||
|
||||
That bundle is useful for trace inspection, but it is not enough to claim
|
||||
official OSWorld reproducibility. Official OSWorld local evaluation requires a
|
||||
runnable OSWorld checkout, desktop/control infrastructure, and an agent adapter
|
||||
inside the official runner. Public verified leaderboard claims require the
|
||||
official verification path.
|
||||
|
||||
Files:
|
||||
|
||||
- `normalize_logs.py` indexes logs-only bundles for analysis.
|
||||
- `schemas.py` validates the known logs-only JSON layout.
|
||||
- `osworld_adapter_skeleton.py` refuses to run unless the official environment,
|
||||
live Ouroboros server, computer-use payload, and output-root isolation are all
|
||||
present.
|
||||
|
||||
No reward calculation, VM reset flow, or leaderboard scoring is implemented in
|
||||
this release.
|
||||
1
devtools/benchmarks/osworld/__init__.py
Normal file
1
devtools/benchmarks/osworld/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
|||
"""OSWorld log normalization and fail-closed adapter skeleton."""
|
||||
62
devtools/benchmarks/osworld/normalize_logs.py
Normal file
62
devtools/benchmarks/osworld/normalize_logs.py
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Normalize an OSWorld logs-only bundle into an inspectable index."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
if __package__ in {None, ""}:
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
|
||||
|
||||
from devtools.benchmarks.common.run_roots import ensure_file_output_outside_repo, repo_root_from_devtools
|
||||
from devtools.benchmarks.osworld.schemas import read_traj_jsonl, validate_bundle
|
||||
|
||||
|
||||
def normalize_bundle(root: Path) -> dict[str, Any]:
|
||||
bundle = validate_bundle(root)
|
||||
base = Path(bundle["root"])
|
||||
traj_paths = sorted(base.glob("**/traj.jsonl"))
|
||||
traces = []
|
||||
for path in traj_paths:
|
||||
rows = read_traj_jsonl(path)
|
||||
traces.append(
|
||||
{
|
||||
"path": str(path.relative_to(base)),
|
||||
"events": len(rows),
|
||||
"first_type": str(rows[0].get("type") or rows[0].get("event") or "") if rows else "",
|
||||
"last_type": str(rows[-1].get("type") or rows[-1].get("event") or "") if rows else "",
|
||||
}
|
||||
)
|
||||
return {
|
||||
"bundle_root": str(base),
|
||||
"summary": bundle["summary"],
|
||||
"sample_manifest": bundle["sample_manifest"],
|
||||
"trace_manifest": bundle["trace_manifest"],
|
||||
"traj_count": len(traces),
|
||||
"traces": traces,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("bundle_root")
|
||||
parser.add_argument("--output", default="")
|
||||
args = parser.parse_args()
|
||||
|
||||
normalized = normalize_bundle(Path(args.bundle_root))
|
||||
text = json.dumps(normalized, ensure_ascii=False, indent=2) + "\n"
|
||||
if args.output:
|
||||
out = ensure_file_output_outside_repo(Path(args.output), repo_root_from_devtools())
|
||||
out.write_text(text, encoding="utf-8")
|
||||
else:
|
||||
print(text, end="")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
93
devtools/benchmarks/osworld/osworld_adapter_skeleton.py
Normal file
93
devtools/benchmarks/osworld/osworld_adapter_skeleton.py
Normal file
|
|
@ -0,0 +1,93 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Fail-closed OSWorld adapter skeleton.
|
||||
|
||||
This file intentionally does not implement OSWorld scoring. It verifies that a
|
||||
runnable official OSWorld environment and Ouroboros computer-use surface exist
|
||||
before a future adapter is allowed to proceed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
def _http_json(url: str, timeout: float = 5.0) -> dict[str, Any]:
|
||||
with urllib.request.urlopen(url, timeout=timeout) as resp:
|
||||
raw = resp.read().decode("utf-8", errors="replace")
|
||||
return json.loads(raw) if raw.strip().startswith("{") else {"raw": raw, "status": getattr(resp, "status", None)}
|
||||
|
||||
|
||||
def _outside(path: Path, forbidden: list[Path]) -> bool:
|
||||
resolved = path.expanduser().resolve(strict=False)
|
||||
for root in forbidden:
|
||||
try:
|
||||
resolved.relative_to(root.expanduser().resolve(strict=False))
|
||||
return False
|
||||
except ValueError:
|
||||
continue
|
||||
return True
|
||||
|
||||
|
||||
def preflight(
|
||||
*,
|
||||
osworld_root: Path,
|
||||
ouroboros_url: str,
|
||||
osworld_server_url: str,
|
||||
computer_use_payload: Path,
|
||||
output_root: Path,
|
||||
repo_root: Path,
|
||||
data_root: Path,
|
||||
) -> dict[str, Any]:
|
||||
failures: list[str] = []
|
||||
if not osworld_root.is_dir():
|
||||
failures.append(f"official OSWorld checkout not found: {osworld_root}")
|
||||
if not (osworld_root / "run.py").exists() and not (osworld_root / "evaluation_examples").exists():
|
||||
failures.append(f"OSWorld checkout shape is not recognized: {osworld_root}")
|
||||
if not computer_use_payload.exists():
|
||||
failures.append(f"computer_use payload is missing: {computer_use_payload}")
|
||||
if not _outside(output_root, [repo_root, data_root]):
|
||||
failures.append(f"output root must be outside repo and runtime data: {output_root}")
|
||||
try:
|
||||
_http_json(ouroboros_url.rstrip("/") + "/api/state")
|
||||
except Exception as exc:
|
||||
failures.append(f"Ouroboros server is not reachable: {type(exc).__name__}: {exc}")
|
||||
try:
|
||||
urllib.request.urlopen(osworld_server_url.rstrip("/") + "/", timeout=5).read(1)
|
||||
except Exception as exc:
|
||||
failures.append(f"OSWorld desktop/control server is not reachable: {type(exc).__name__}: {exc}")
|
||||
return {"ok": not failures, "failures": failures}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--osworld-root", required=True)
|
||||
parser.add_argument("--ouroboros-url", default="http://127.0.0.1:8765")
|
||||
parser.add_argument("--osworld-server-url", required=True)
|
||||
parser.add_argument("--computer-use-payload", required=True)
|
||||
parser.add_argument("--output-root", required=True)
|
||||
parser.add_argument("--repo-root", default="/Users/anton/Ouroboros/repo")
|
||||
parser.add_argument("--data-root", default="/Users/anton/Ouroboros/data")
|
||||
args = parser.parse_args()
|
||||
|
||||
result = preflight(
|
||||
osworld_root=Path(args.osworld_root).expanduser(),
|
||||
ouroboros_url=args.ouroboros_url,
|
||||
osworld_server_url=args.osworld_server_url,
|
||||
computer_use_payload=Path(args.computer_use_payload).expanduser(),
|
||||
output_root=Path(args.output_root).expanduser(),
|
||||
repo_root=Path(args.repo_root).expanduser(),
|
||||
data_root=Path(args.data_root).expanduser(),
|
||||
)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
if not result["ok"]:
|
||||
return 2
|
||||
print("OSWorld runnable adapter is not implemented in this release; preflight passed only.")
|
||||
return 3
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
54
devtools/benchmarks/osworld/schemas.py
Normal file
54
devtools/benchmarks/osworld/schemas.py
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
"""Validation helpers for OSWorld logs-only bundles."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
REQUIRED_BUNDLE_FILES = (
|
||||
"SUMMARY.json",
|
||||
"sample_manifest.json",
|
||||
)
|
||||
|
||||
|
||||
def load_json(path: Path) -> dict[str, Any]:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError(f"{path} must contain a JSON object")
|
||||
return data
|
||||
|
||||
|
||||
def validate_bundle(root: Path) -> dict[str, Any]:
|
||||
base = Path(root).expanduser().resolve(strict=False)
|
||||
missing = [name for name in REQUIRED_BUNDLE_FILES if not (base / name).is_file()]
|
||||
if missing:
|
||||
raise ValueError(f"OSWorld logs-only bundle is missing: {', '.join(missing)}")
|
||||
summary = load_json(base / "SUMMARY.json")
|
||||
sample_manifest = load_json(base / "sample_manifest.json")
|
||||
root_trace_manifest = base / "trace_manifest.json"
|
||||
if root_trace_manifest.is_file():
|
||||
trace_manifest: dict[str, Any] = load_json(root_trace_manifest)
|
||||
else:
|
||||
nested = sorted(path.relative_to(base).as_posix() for path in base.glob("**/traces/trace_manifest.json"))
|
||||
if not nested:
|
||||
raise ValueError("OSWorld logs-only bundle has no trace_manifest.json files")
|
||||
trace_manifest = {"trace_manifest_paths": nested}
|
||||
return {
|
||||
"root": str(base),
|
||||
"summary": summary,
|
||||
"sample_manifest": sample_manifest,
|
||||
"trace_manifest": trace_manifest,
|
||||
}
|
||||
|
||||
|
||||
def read_traj_jsonl(path: Path) -> list[dict[str, Any]]:
|
||||
rows: list[dict[str, Any]] = []
|
||||
for raw in Path(path).read_text(encoding="utf-8", errors="replace").splitlines():
|
||||
if not raw.strip():
|
||||
continue
|
||||
item = json.loads(raw)
|
||||
if isinstance(item, dict):
|
||||
rows.append(item)
|
||||
return rows
|
||||
15
devtools/benchmarks/programbench/README.md
Normal file
15
devtools/benchmarks/programbench/README.md
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
# ProgramBench
|
||||
|
||||
This adapter prepares Ouroboros workspace tasks for official ProgramBench
|
||||
cleanroom execution.
|
||||
|
||||
Invariants:
|
||||
|
||||
- Use official `programbench` CLI for evaluation and summaries.
|
||||
- Use `task_cleanroom` task images; do not score locally.
|
||||
- Tool execution for the benchmark workspace runs in a no-network Docker
|
||||
backend.
|
||||
- Reference binaries are declared through
|
||||
`resource_policy.protected_artifacts`: execute is allowed, byte reads,
|
||||
copy/hash/static introspection/tracing/debugging are denied.
|
||||
- Submission artifact is `<run>/<instance_id>/submission.tar.gz`.
|
||||
1
devtools/benchmarks/programbench/__init__.py
Normal file
1
devtools/benchmarks/programbench/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
|||
"""ProgramBench adapter helpers."""
|
||||
173
devtools/benchmarks/programbench/programbench_adapter.py
Normal file
173
devtools/benchmarks/programbench/programbench_adapter.py
Normal file
|
|
@ -0,0 +1,173 @@
|
|||
"""ProgramBench cleanroom adapter primitives."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pathlib
|
||||
import json
|
||||
import subprocess
|
||||
import tarfile
|
||||
from typing import Any
|
||||
|
||||
from devtools.benchmarks.common.manifests import write_json
|
||||
from devtools.benchmarks.common.official_commands import programbench_eval_cmd, programbench_info_cmd
|
||||
from devtools.benchmarks.programbench.schemas import task_body
|
||||
|
||||
|
||||
def docker_executor_ref(
|
||||
*,
|
||||
container_name: str,
|
||||
workspace_host_path: pathlib.Path,
|
||||
workspace_backend_path: str = "/workspace",
|
||||
) -> dict[str, Any]:
|
||||
return {
|
||||
"type": "docker_exec",
|
||||
"id": container_name,
|
||||
"container_name": container_name,
|
||||
"network": "none",
|
||||
"workspace_host_path": str(pathlib.Path(workspace_host_path).resolve(strict=False)),
|
||||
"workspace_backend_path": workspace_backend_path,
|
||||
}
|
||||
|
||||
|
||||
def build_ouroboros_task_body(
|
||||
*,
|
||||
instruction: str,
|
||||
workspace_host_path: pathlib.Path,
|
||||
container_name: str,
|
||||
protected_backend_paths: list[str] | None = None,
|
||||
task_id: str = "",
|
||||
) -> dict[str, Any]:
|
||||
ensure_git_workspace(workspace_host_path)
|
||||
protected = protected_backend_paths or ["/workspace/executable", "executable"]
|
||||
return task_body(
|
||||
description=instruction,
|
||||
workspace_root=str(pathlib.Path(workspace_host_path).resolve(strict=False)),
|
||||
executor_ref=docker_executor_ref(container_name=container_name, workspace_host_path=workspace_host_path),
|
||||
protected_paths=protected,
|
||||
task_id=task_id,
|
||||
)
|
||||
|
||||
|
||||
def preflight_cleanroom_container(container_name: str) -> dict[str, Any]:
|
||||
proc = subprocess.run(
|
||||
["docker", "inspect", container_name],
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
timeout=15,
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
raise RuntimeError(f"docker inspect failed for {container_name}: {proc.stderr.strip()}")
|
||||
data = json.loads(proc.stdout or "[]")
|
||||
if not isinstance(data, list) or not data:
|
||||
raise RuntimeError(f"docker inspect returned no container data for {container_name}")
|
||||
info = data[0]
|
||||
config = info.get("Config") if isinstance(info, dict) else {}
|
||||
host_config = info.get("HostConfig") if isinstance(info, dict) else {}
|
||||
image = str((config or {}).get("Image") or (info or {}).get("Image") or "")
|
||||
network = str((host_config or {}).get("NetworkMode") or "")
|
||||
if "task_cleanroom" not in image:
|
||||
raise RuntimeError(f"ProgramBench container must use a task_cleanroom image, got {image!r}")
|
||||
if network != "none":
|
||||
raise RuntimeError(f"ProgramBench inference container must use Docker NetworkMode=none, got {network!r}")
|
||||
return {"image": image, "network": network}
|
||||
|
||||
|
||||
def ensure_git_workspace(workspace_root: pathlib.Path) -> None:
|
||||
root = pathlib.Path(workspace_root).resolve(strict=False)
|
||||
probe = subprocess.run(["git", "rev-parse", "--show-toplevel"], cwd=root, capture_output=True, text=True, timeout=10)
|
||||
if probe.returncode == 0 and pathlib.Path((probe.stdout or "").strip()).resolve(strict=False) == root:
|
||||
return
|
||||
subprocess.run(["git", "init"], cwd=root, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, timeout=30)
|
||||
subprocess.run(["git", "config", "user.email", "ouroboros-bench@example.invalid"], cwd=root, check=True, timeout=10)
|
||||
subprocess.run(["git", "config", "user.name", "Ouroboros Bench"], cwd=root, check=True, timeout=10)
|
||||
|
||||
|
||||
def create_submission_tarball(
|
||||
workspace_root: pathlib.Path,
|
||||
out_path: pathlib.Path,
|
||||
*,
|
||||
protected_paths: list[str] | None = None,
|
||||
workspace_backend_path: str = "/workspace",
|
||||
) -> pathlib.Path:
|
||||
root = pathlib.Path(workspace_root).resolve(strict=False)
|
||||
protected = _protected_submission_paths(root, protected_paths or [], workspace_backend_path=workspace_backend_path)
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with tarfile.open(out_path, "w:gz") as tar:
|
||||
for path in sorted(root.rglob("*")):
|
||||
rel = path.relative_to(root)
|
||||
if _skip_submission_path(rel):
|
||||
continue
|
||||
resolved = path.resolve(strict=False)
|
||||
if any(_path_matches(resolved, protected_path) for protected_path in protected):
|
||||
continue
|
||||
tar.add(path, arcname=rel.as_posix(), recursive=False)
|
||||
return out_path
|
||||
|
||||
|
||||
def _protected_submission_paths(root: pathlib.Path, protected_paths: list[str], *, workspace_backend_path: str) -> list[pathlib.Path]:
|
||||
protected: list[pathlib.Path] = []
|
||||
backend_prefix = str(workspace_backend_path or "/workspace").rstrip("/")
|
||||
for raw in protected_paths:
|
||||
text = str(raw or "").strip()
|
||||
if not text:
|
||||
continue
|
||||
if text == backend_prefix:
|
||||
protected.append(root)
|
||||
continue
|
||||
if text.startswith(backend_prefix + "/"):
|
||||
rel = text[len(backend_prefix) + 1:]
|
||||
protected.append((root / rel).resolve(strict=False))
|
||||
continue
|
||||
candidate = pathlib.Path(text)
|
||||
if candidate.is_absolute():
|
||||
continue
|
||||
protected.append((root / candidate).resolve(strict=False))
|
||||
return list(dict.fromkeys(protected))
|
||||
|
||||
|
||||
def _path_matches(candidate: pathlib.Path, protected: pathlib.Path) -> bool:
|
||||
if candidate == protected:
|
||||
return True
|
||||
try:
|
||||
candidate.relative_to(protected)
|
||||
return True
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
def _skip_submission_path(rel: pathlib.PurePath) -> bool:
|
||||
parts = set(rel.parts)
|
||||
return bool(parts & {
|
||||
".git",
|
||||
".ouroboros",
|
||||
"__pycache__",
|
||||
".pytest_cache",
|
||||
".ruff_cache",
|
||||
".mypy_cache",
|
||||
"node_modules",
|
||||
"build",
|
||||
"dist",
|
||||
"htmlcov",
|
||||
}) or rel.name in {".DS_Store", ".coverage", "coverage.xml"} or rel.suffix in {".pyc", ".pyo", ".log", ".tmp"}
|
||||
|
||||
|
||||
def run_official_eval(run_root: pathlib.Path) -> dict[str, Any]:
|
||||
eval_proc = subprocess.run(programbench_eval_cmd(run_root), capture_output=True, text=True)
|
||||
info_proc = subprocess.run(programbench_info_cmd(run_root), capture_output=True, text=True)
|
||||
result = {
|
||||
"eval": {
|
||||
"cmd": programbench_eval_cmd(run_root),
|
||||
"returncode": eval_proc.returncode,
|
||||
"stdout": eval_proc.stdout,
|
||||
"stderr": eval_proc.stderr,
|
||||
},
|
||||
"info": {
|
||||
"cmd": programbench_info_cmd(run_root),
|
||||
"returncode": info_proc.returncode,
|
||||
"stdout": info_proc.stdout,
|
||||
"stderr": info_proc.stderr,
|
||||
},
|
||||
}
|
||||
write_json(pathlib.Path(run_root) / "programbench_eval_result.json", result)
|
||||
return result
|
||||
64
devtools/benchmarks/programbench/run_programbench.py
Normal file
64
devtools/benchmarks/programbench/run_programbench.py
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
#!/usr/bin/env python3
|
||||
"""ProgramBench adapter entrypoint.
|
||||
|
||||
This script intentionally stops before reinventing ProgramBench orchestration.
|
||||
It prepares task bodies/submissions for official cleanroom runs and delegates
|
||||
evaluation to the official `programbench` CLI.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
if __package__ in {None, ""}:
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[3]))
|
||||
|
||||
from devtools.benchmarks.common.manifests import write_json
|
||||
from devtools.benchmarks.common.run_roots import ensure_outside_repo, run_root, safe_join_under
|
||||
from devtools.benchmarks.programbench.programbench_adapter import (
|
||||
build_ouroboros_task_body,
|
||||
create_submission_tarball,
|
||||
preflight_cleanroom_container,
|
||||
run_official_eval,
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--repo-dir", default=str(pathlib.Path(__file__).resolve().parents[3]))
|
||||
parser.add_argument("--workspace", required=True)
|
||||
parser.add_argument("--instruction-file", required=True)
|
||||
parser.add_argument("--container-name", required=True)
|
||||
parser.add_argument("--instance-id", required=True)
|
||||
parser.add_argument("--run-id", default="")
|
||||
parser.add_argument("--protected-path", action="append", default=[], help="protected reference path inside cleanroom; repeatable")
|
||||
parser.add_argument("--eval", action="store_true", help="run official programbench eval/info after writing submission")
|
||||
args = parser.parse_args()
|
||||
|
||||
out_root = ensure_outside_repo(run_root("programbench", args.run_id), pathlib.Path(args.repo_dir))
|
||||
instance_dir = safe_join_under(out_root, args.instance_id)
|
||||
preflight = preflight_cleanroom_container(args.container_name)
|
||||
protected_paths = args.protected_path or ["/workspace/executable", "executable"]
|
||||
body = build_ouroboros_task_body(
|
||||
instruction=pathlib.Path(args.instruction_file).read_text(encoding="utf-8"),
|
||||
workspace_host_path=pathlib.Path(args.workspace),
|
||||
container_name=args.container_name,
|
||||
protected_backend_paths=protected_paths,
|
||||
)
|
||||
body.setdefault("metadata", {})["cleanroom_preflight"] = preflight
|
||||
write_json(instance_dir / "ouroboros_task_body.json", body)
|
||||
create_submission_tarball(
|
||||
pathlib.Path(args.workspace),
|
||||
instance_dir / "submission.tar.gz",
|
||||
protected_paths=protected_paths,
|
||||
)
|
||||
if args.eval:
|
||||
run_official_eval(out_root)
|
||||
print(instance_dir)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
52
devtools/benchmarks/programbench/schemas.py
Normal file
52
devtools/benchmarks/programbench/schemas.py
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
"""ProgramBench adapter schemas."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
def protected_reference_policy(paths: list[str]) -> dict[str, Any]:
|
||||
clean = [str(path) for path in paths if str(path or "").strip()]
|
||||
return {
|
||||
"protected_artifacts": [
|
||||
{
|
||||
"id": "programbench_reference",
|
||||
"role": "black_box_reference",
|
||||
"paths": clean,
|
||||
"allow": ["execute"],
|
||||
"deny": [
|
||||
"read_bytes",
|
||||
"copy",
|
||||
"hash",
|
||||
"static_introspection",
|
||||
"dynamic_trace",
|
||||
"debug",
|
||||
],
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
def task_body(
|
||||
*,
|
||||
description: str,
|
||||
workspace_root: str,
|
||||
executor_ref: dict[str, Any],
|
||||
protected_paths: list[str],
|
||||
task_id: str = "",
|
||||
) -> dict[str, Any]:
|
||||
body: dict[str, Any] = {
|
||||
"description": description,
|
||||
"workspace_root": workspace_root,
|
||||
"workspace_mode": "external",
|
||||
"memory_mode": "empty",
|
||||
"allowed_resources": {"web": False, "network": False, "internet": False},
|
||||
"resource_policy": protected_reference_policy(protected_paths),
|
||||
"executor_ref": executor_ref,
|
||||
"actor_id": "programbench",
|
||||
"source": "programbench",
|
||||
"metadata": {"source": "programbench"},
|
||||
}
|
||||
if task_id:
|
||||
body["task_id"] = task_id
|
||||
return body
|
||||
31
devtools/benchmarks/swe_bench/README.md
Normal file
31
devtools/benchmarks/swe_bench/README.md
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
# SWE-bench Devtools
|
||||
|
||||
These helpers generate official SWE-bench prediction JSONL files for a set of
|
||||
already-prepared local task checkouts.
|
||||
|
||||
They do not download datasets, reset repositories, or score benchmark results.
|
||||
Evaluation remains the official SWE-bench harness:
|
||||
|
||||
```bash
|
||||
python -m swebench.harness.run_evaluation \
|
||||
--dataset_name princeton-nlp/SWE-bench_Verified \
|
||||
--predictions_path /path/to/predictions.jsonl \
|
||||
--max_workers 1 \
|
||||
--run_id ouroboros
|
||||
```
|
||||
|
||||
Prediction rows produced by `swebench_predictions.py` contain only:
|
||||
|
||||
```json
|
||||
{"instance_id": "...", "model_name_or_path": "...", "model_patch": "..."}
|
||||
```
|
||||
|
||||
Supported preset aliases in `presets.py`:
|
||||
|
||||
- `full` -> `princeton-nlp/SWE-bench`
|
||||
- `lite` -> `princeton-nlp/SWE-bench_Lite`
|
||||
- `verified` -> `princeton-nlp/SWE-bench_Verified`
|
||||
|
||||
Input rows must provide `instance_id`, a clean git `workspace_root` or
|
||||
`--workspaces-root`, and `problem_statement` or `prompt`. If `base_commit` is
|
||||
present, the helper refuses to run unless the checkout HEAD matches it.
|
||||
1
devtools/benchmarks/swe_bench/__init__.py
Normal file
1
devtools/benchmarks/swe_bench/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
|||
"""SWE-bench prediction helpers."""
|
||||
29
devtools/benchmarks/swe_bench/presets.py
Normal file
29
devtools/benchmarks/swe_bench/presets.py
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
"""Official SWE-bench dataset preset names used by devtools."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
PRESETS: dict[str, str] = {
|
||||
"full": "princeton-nlp/SWE-bench",
|
||||
"lite": "princeton-nlp/SWE-bench_Lite",
|
||||
"verified": "princeton-nlp/SWE-bench_Verified",
|
||||
}
|
||||
|
||||
ALIASES: dict[str, str] = {
|
||||
"swe-bench": PRESETS["full"],
|
||||
"swebench": PRESETS["full"],
|
||||
"swe-bench-lite": PRESETS["lite"],
|
||||
"swebench-lite": PRESETS["lite"],
|
||||
"swe-bench-verified": PRESETS["verified"],
|
||||
"swebench-verified": PRESETS["verified"],
|
||||
"SWE-bench/SWE-bench_Verified": PRESETS["verified"],
|
||||
}
|
||||
|
||||
|
||||
def resolve_preset(name: str) -> str:
|
||||
key = str(name or "").strip()
|
||||
if key in PRESETS:
|
||||
return PRESETS[key]
|
||||
if key in ALIASES:
|
||||
return ALIASES[key]
|
||||
return key
|
||||
324
devtools/benchmarks/swe_bench/swebench_predictions.py
Normal file
324
devtools/benchmarks/swe_bench/swebench_predictions.py
Normal file
|
|
@ -0,0 +1,324 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Generate SWE-bench predictions JSONL with Ouroboros.
|
||||
|
||||
This helper prepares the official prediction artifact only. Evaluation remains
|
||||
the responsibility of ``swebench.harness.run_evaluation``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
if __package__ in {None, ""}:
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
|
||||
|
||||
from devtools.benchmarks.common.run_roots import (
|
||||
ensure_file_output_outside_repo,
|
||||
ensure_outside_repo,
|
||||
repo_root_from_devtools,
|
||||
safe_benchmark_id,
|
||||
)
|
||||
from devtools.benchmarks.common.official_commands import swebench_eval_cmd
|
||||
from devtools.benchmarks.swe_bench.presets import resolve_preset
|
||||
from ouroboros.config import get_finalization_grace_sec
|
||||
|
||||
|
||||
REPO_ROOT = repo_root_from_devtools()
|
||||
|
||||
|
||||
def _records(path: Path) -> list[dict[str, Any]]:
|
||||
rows: list[dict[str, Any]] = []
|
||||
for raw in path.read_text(encoding="utf-8").splitlines():
|
||||
if not raw.strip():
|
||||
continue
|
||||
item = json.loads(raw)
|
||||
if isinstance(item, dict):
|
||||
rows.append(item)
|
||||
return rows
|
||||
|
||||
|
||||
def _resolve_workspace(item: dict[str, Any], workspaces_root: str) -> str:
|
||||
workspace = str(item.get("workspace_root") or "").strip()
|
||||
if workspace or not workspaces_root:
|
||||
return workspace
|
||||
root = Path(workspaces_root).expanduser()
|
||||
instance_id = str(item.get("instance_id") or "")
|
||||
repo = str(item.get("repo") or "").strip()
|
||||
candidates = [root / instance_id]
|
||||
if repo:
|
||||
candidates.extend([root / repo.replace("/", "__"), root / repo.split("/")[-1]])
|
||||
for candidate in candidates:
|
||||
if candidate.is_dir():
|
||||
return str(candidate)
|
||||
return ""
|
||||
|
||||
|
||||
def _git_stdout(args: list[str], cwd: Path) -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.run(args, cwd=cwd, capture_output=True, text=True, timeout=10)
|
||||
|
||||
|
||||
def _record_error(errors: list[dict[str, Any]], row: dict[str, Any], continue_on_error: bool) -> None:
|
||||
if not continue_on_error:
|
||||
raise RuntimeError(str(row.get("error") or row))
|
||||
errors.append(row)
|
||||
|
||||
|
||||
def _write_logs(logs_dir: str, instance_id: str, stdout: str, stderr: str, summary: dict[str, Any]) -> None:
|
||||
if not logs_dir:
|
||||
return
|
||||
log_dir = Path(logs_dir).expanduser() / safe_benchmark_id(instance_id)
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
(log_dir / "ouroboros.stdout").write_text(stdout, encoding="utf-8")
|
||||
(log_dir / "ouroboros.stderr").write_text(stderr, encoding="utf-8")
|
||||
(log_dir / "ouroboros-agent-result.json").write_text(
|
||||
json.dumps(summary, ensure_ascii=False, indent=2),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def _build_ouroboros_cmd(args: argparse.Namespace, workspace: Path, result_json_path: Path, prompt: str) -> list[str]:
|
||||
cli_prefix = shlex.split(args.cli) if args.cli else [sys.executable, "-m", "ouroboros.cli"]
|
||||
return [
|
||||
*cli_prefix,
|
||||
"run",
|
||||
"--workspace",
|
||||
str(workspace),
|
||||
"--memory-mode",
|
||||
"empty",
|
||||
"--timeout",
|
||||
str(int(args.timeout)),
|
||||
"--patch",
|
||||
"--result-json-out",
|
||||
str(result_json_path),
|
||||
prompt,
|
||||
]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--input", required=True, help="JSONL instances")
|
||||
parser.add_argument("--output", required=True, help="SWE-bench predictions JSONL")
|
||||
parser.add_argument("--model-name", default="ouroboros-cli")
|
||||
parser.add_argument("--cli", default="", help="optional Ouroboros CLI command prefix")
|
||||
parser.add_argument("--timeout", type=int, default=7200, help="per-instance Ouroboros timeout seconds")
|
||||
parser.add_argument("--continue-on-error", action="store_true")
|
||||
parser.add_argument("--errors-output", default="")
|
||||
parser.add_argument("--logs-dir", default="")
|
||||
parser.add_argument("--workspaces-root", default="")
|
||||
parser.add_argument("--print-eval-command", default="", help="optional preset/dataset for official eval command")
|
||||
args = parser.parse_args()
|
||||
|
||||
output_path = ensure_file_output_outside_repo(Path(args.output), REPO_ROOT)
|
||||
errors_output_path = (
|
||||
ensure_file_output_outside_repo(Path(args.errors_output), REPO_ROOT)
|
||||
if args.errors_output
|
||||
else Path(str(output_path) + ".errors.jsonl")
|
||||
)
|
||||
logs_dir = str(ensure_outside_repo(Path(args.logs_dir), REPO_ROOT)) if args.logs_dir else ""
|
||||
|
||||
predictions: list[dict[str, str]] = []
|
||||
errors: list[dict[str, Any]] = []
|
||||
for item in _records(Path(args.input)):
|
||||
instance_id = str(item.get("instance_id") or "")
|
||||
try:
|
||||
safe_instance_id = safe_benchmark_id(instance_id)
|
||||
except ValueError as exc:
|
||||
_record_error(
|
||||
errors,
|
||||
{
|
||||
"instance_id": instance_id,
|
||||
"error": str(exc),
|
||||
"reason_code": "invalid_instance_id",
|
||||
},
|
||||
args.continue_on_error,
|
||||
)
|
||||
continue
|
||||
workspace = _resolve_workspace(item, args.workspaces_root)
|
||||
prompt = str(item.get("problem_statement") or item.get("prompt") or "")
|
||||
if not instance_id or not workspace or not prompt:
|
||||
_record_error(
|
||||
errors,
|
||||
{
|
||||
"instance_id": instance_id,
|
||||
"error": "each row must include instance_id, workspace_root or --workspaces-root, and problem_statement/prompt",
|
||||
"reason_code": "invalid_instance",
|
||||
},
|
||||
args.continue_on_error,
|
||||
)
|
||||
continue
|
||||
|
||||
workspace_path = Path(workspace).expanduser().resolve(strict=False)
|
||||
if not workspace_path.is_dir():
|
||||
_record_error(
|
||||
errors,
|
||||
{
|
||||
"instance_id": instance_id,
|
||||
"error": f"workspace_root is not a directory for {instance_id}: {workspace}",
|
||||
"reason_code": "invalid_workspace",
|
||||
},
|
||||
args.continue_on_error,
|
||||
)
|
||||
continue
|
||||
head = _git_stdout(["git", "rev-parse", "HEAD"], workspace_path)
|
||||
if head.returncode != 0:
|
||||
_record_error(
|
||||
errors,
|
||||
{
|
||||
"instance_id": instance_id,
|
||||
"error": f"workspace_root is not a git checkout for {instance_id}: {workspace_path}",
|
||||
"reason_code": "not_git_checkout",
|
||||
},
|
||||
args.continue_on_error,
|
||||
)
|
||||
continue
|
||||
base_commit = str(item.get("base_commit") or "").strip()
|
||||
if base_commit and head.stdout.strip() != base_commit:
|
||||
_record_error(
|
||||
errors,
|
||||
{
|
||||
"instance_id": instance_id,
|
||||
"error": f"workspace HEAD for {instance_id} is {head.stdout.strip()}, expected base_commit {base_commit}",
|
||||
"reason_code": "wrong_base_commit",
|
||||
},
|
||||
args.continue_on_error,
|
||||
)
|
||||
continue
|
||||
status = _git_stdout(["git", "status", "--porcelain=v1", "--untracked-files=all"], workspace_path)
|
||||
if status.returncode != 0 or status.stdout.strip():
|
||||
_record_error(
|
||||
errors,
|
||||
{
|
||||
"instance_id": instance_id,
|
||||
"error": f"workspace must be clean before SWE-bench run for {instance_id}",
|
||||
"reason_code": "dirty_workspace",
|
||||
},
|
||||
args.continue_on_error,
|
||||
)
|
||||
continue
|
||||
|
||||
if logs_dir:
|
||||
result_json_path = Path(logs_dir) / safe_instance_id / "task_result.json"
|
||||
else:
|
||||
result_json_path = Path(tempfile.gettempdir()) / f"ouroboros_swebench_{safe_instance_id}.task_result.json"
|
||||
result_json_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
cmd = _build_ouroboros_cmd(args, workspace_path, result_json_path, prompt)
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=int(args.timeout) + get_finalization_grace_sec() + 60,
|
||||
)
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
stdout = exc.stdout if isinstance(exc.stdout, str) else (exc.stdout or b"").decode("utf-8", errors="replace")
|
||||
stderr = exc.stderr if isinstance(exc.stderr, str) else (exc.stderr or b"").decode("utf-8", errors="replace")
|
||||
_write_logs(
|
||||
logs_dir,
|
||||
instance_id,
|
||||
stdout,
|
||||
stderr,
|
||||
{
|
||||
"instance_id": instance_id,
|
||||
"returncode": 124,
|
||||
"stdout_chars": len(stdout),
|
||||
"stderr_chars": len(stderr),
|
||||
"timeout_sec": int(args.timeout),
|
||||
"failure_mode": "timeout",
|
||||
},
|
||||
)
|
||||
_record_error(
|
||||
errors,
|
||||
{
|
||||
"instance_id": instance_id,
|
||||
"returncode": 124,
|
||||
"error": f"ouroboros run timed out after {int(args.timeout)}s",
|
||||
"timeout": True,
|
||||
},
|
||||
args.continue_on_error,
|
||||
)
|
||||
continue
|
||||
|
||||
task_result: dict[str, Any] = {}
|
||||
if result_json_path.exists():
|
||||
try:
|
||||
loaded = json.loads(result_json_path.read_text(encoding="utf-8"))
|
||||
if isinstance(loaded, dict):
|
||||
task_result = loaded
|
||||
except Exception:
|
||||
task_result = {}
|
||||
summary = {
|
||||
"instance_id": instance_id,
|
||||
"returncode": result.returncode,
|
||||
"stdout_chars": len(result.stdout or ""),
|
||||
"stderr_chars": len(result.stderr or ""),
|
||||
"patch_empty": not bool((result.stdout or "").strip()),
|
||||
"timeout_sec": int(args.timeout),
|
||||
"outcome_axes": task_result.get("outcome_axes"),
|
||||
"reason_code": task_result.get("reason_code"),
|
||||
"artifact_bundle": task_result.get("artifact_bundle"),
|
||||
}
|
||||
_write_logs(logs_dir, instance_id, result.stdout or "", result.stderr or "", summary)
|
||||
if result.returncode != 0:
|
||||
details = (result.stderr or result.stdout or "").strip()
|
||||
if len(details) > 4000:
|
||||
details = details[:4000] + "\n...[truncated]"
|
||||
_record_error(
|
||||
errors,
|
||||
{
|
||||
"instance_id": instance_id,
|
||||
"returncode": result.returncode,
|
||||
"error": details or f"ouroboros run exited {result.returncode}",
|
||||
"outcome_axes": task_result.get("outcome_axes"),
|
||||
"reason_code": task_result.get("reason_code"),
|
||||
"artifact_bundle": task_result.get("artifact_bundle"),
|
||||
"trace_refs": task_result.get("trace_refs"),
|
||||
},
|
||||
args.continue_on_error,
|
||||
)
|
||||
continue
|
||||
if not (result.stdout or "").strip():
|
||||
_record_error(
|
||||
errors,
|
||||
{
|
||||
"instance_id": instance_id,
|
||||
"returncode": 0,
|
||||
"error": "ouroboros run produced no patch",
|
||||
"outcome_axes": task_result.get("outcome_axes"),
|
||||
"reason_code": task_result.get("reason_code") or "no_patch",
|
||||
"artifact_bundle": task_result.get("artifact_bundle"),
|
||||
"trace_refs": task_result.get("trace_refs"),
|
||||
},
|
||||
args.continue_on_error,
|
||||
)
|
||||
continue
|
||||
predictions.append(
|
||||
{
|
||||
"instance_id": instance_id,
|
||||
"model_name_or_path": args.model_name,
|
||||
"model_patch": result.stdout,
|
||||
}
|
||||
)
|
||||
|
||||
output_path.write_text(
|
||||
"\n".join(json.dumps(row, ensure_ascii=False) for row in predictions) + ("\n" if predictions else ""),
|
||||
encoding="utf-8",
|
||||
)
|
||||
if errors:
|
||||
errors_output_path.write_text(
|
||||
"\n".join(json.dumps(row, ensure_ascii=False) for row in errors) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
if args.print_eval_command:
|
||||
print(" ".join(shlex.quote(part) for part in swebench_eval_cmd(resolve_preset(args.print_eval_command), output_path, "ouroboros", 1)))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
126
devtools/benchmarks/swe_bench_pro/METHODOLOGY.md
Normal file
126
devtools/benchmarks/swe_bench_pro/METHODOLOGY.md
Normal file
|
|
@ -0,0 +1,126 @@
|
|||
# SWE-bench Pro Methodology Notes
|
||||
|
||||
These notes summarize the portable lessons from prior Ouroboros CLI runs on
|
||||
SWE-bench Pro (`scaleapi/SWE-bench_Pro-os`, dataset `ScaleAI/SWE-bench_Pro`,
|
||||
images `jefzda/sweap-images:{dockerhub_tag}`, task repositories under `/app`).
|
||||
They are not a replacement driver or scorer. They document how to prepare
|
||||
prediction patches and how to inspect official Pro evaluator outputs without
|
||||
repeating the same failure modes.
|
||||
|
||||
Included files:
|
||||
|
||||
- `capture_patch.sh`: standalone `model_patch` capture for a task repository.
|
||||
- `grade_pro.py`: wrapper that runs the official Pro eval and prints a
|
||||
diagnostic, non-leaderboard summary of official per-instance outputs.
|
||||
|
||||
## 1. Capturing `model_patch`
|
||||
|
||||
Patch capture determines what the official evaluator sees, and it is the most
|
||||
common source of false failures.
|
||||
|
||||
- Capture like the reference SWE-agent/mini-swe-agent scaffold:
|
||||
`git add -A && git diff --cached <base_commit>`. A plain
|
||||
`git diff <base>` loses new untracked source files, and several real Pro
|
||||
fixes add files.
|
||||
|
||||
- Write the captured diff to an explicit path outside the Ouroboros repository,
|
||||
normally under `/Users/anton/Ouroboros/bench_runs/`. The helper rejects
|
||||
repo-internal output paths so benchmark artifacts cannot dirty `devtools/`.
|
||||
|
||||
- Remove environment artifacts that `git add -A` can capture. The
|
||||
`JUNK_RE` pattern in `capture_patch.sh` intentionally covers runtime dumps,
|
||||
caches, dependency folders, build outputs, coverage output, and similar
|
||||
generated files. Do not copy broad SWE-agent defaults such as
|
||||
`*.cfg`, `*.toml`, `setup.py`, or `*.lock`: Pro fixes can legitimately touch
|
||||
configuration and lock files.
|
||||
|
||||
- Remove binary blobs. `git diff --cached --numstat <base>` prints
|
||||
`-\t-\t<file>` for binary files. Build verification can leave compiled
|
||||
binaries in the repository; those can inflate a tiny source patch into a huge
|
||||
binary patch. Text additions such as `.go`, `.ts`, and `.py` files remain.
|
||||
|
||||
- In workspace mode, capture from the real task repository, usually `/app`, not
|
||||
from Ouroboros's internal repository. Verify that `git -C /app status` shows
|
||||
the intended modifications after the solve.
|
||||
|
||||
- Agent-created scratch files are the agent's responsibility, not a reason to
|
||||
over-filter patches. The helper filters environment artifacts and binary
|
||||
blobs, not arbitrary source-like files left by the agent.
|
||||
|
||||
## 2. Official Pro Eval And Diagnostic Summary
|
||||
|
||||
Run the official evaluator:
|
||||
|
||||
```bash
|
||||
python swe_bench_pro_eval.py \
|
||||
--use_local_docker \
|
||||
--docker_platform linux/amd64 \
|
||||
--dockerhub_username jefzda \
|
||||
--scripts_dir run_scripts \
|
||||
--raw_sample_path <SWE-bench_Pro-os>/helper_code/sweap_eval_full_v2.jsonl \
|
||||
--patch_path patches.json
|
||||
```
|
||||
|
||||
`grade_pro.py` wraps this command and then reads official per-instance
|
||||
`{prefix}_output.json` files to print a diagnostic table. That table is not a
|
||||
leaderboard result and is not a replacement scorer.
|
||||
|
||||
Important details:
|
||||
|
||||
- The Pro raw sample uses uppercase `FAIL_TO_PASS` and `PASS_TO_PASS` fields.
|
||||
Some Hugging Face-derived rows use lowercase names; handle both when
|
||||
inspecting diagnostics.
|
||||
|
||||
- If the official progress-bar accuracy aggregator fails or prints a misleading
|
||||
zero, inspect per-instance output files directly. The official evaluator
|
||||
output remains the source of truth; the local diagnostic only helps debug.
|
||||
|
||||
- Pro tamper protection restores test files from the fix commit after applying
|
||||
the agent patch. Agent edits to test files do not count as passing fixes.
|
||||
|
||||
## 3. Streaming Or Evolutionary Runs
|
||||
|
||||
Stateful runs introduce failure classes that frozen baseline runs do not have.
|
||||
|
||||
- Budget ledgers can accidentally carry over between tasks. Per-task caps should
|
||||
reset per task while learned state/code can carry forward as intended.
|
||||
|
||||
- Count API errors by structured event type, not by substring occurrences inside
|
||||
nested provider messages. Separate transient transport failures from
|
||||
context-overflow recovery.
|
||||
|
||||
- Workspace mode often needs `memory_mode=forked`; shared memory can be
|
||||
forbidden with an external workspace. Verify that canonical parent reflections
|
||||
still grow across tasks.
|
||||
|
||||
- If task N has an infrastructure failure, restore state to the snapshot after
|
||||
the last clean task and rerun the suffix. Keep per-task snapshots of runtime
|
||||
data and source state.
|
||||
|
||||
## 4. Container And Environment Pitfalls
|
||||
|
||||
- glibc runtimes mounted into Alpine/musl task images may not run. Use a
|
||||
compatible runtime build or glibc-based images when available.
|
||||
|
||||
- Readiness checks need wall-clock limits. Dependency installation under
|
||||
emulation can block `/api/state` for several minutes, and not every image has
|
||||
`curl`; a Python readiness probe from the agent environment is often more
|
||||
portable.
|
||||
|
||||
- Do not wait for heartbeat files such as `state/queue_snapshot.json` to become
|
||||
quiet. Watch durable outputs such as task reflections or task result files.
|
||||
|
||||
- On macOS bind mounts, host-side files can lag behind container writes. For
|
||||
live monitoring, read files inside the container with `docker exec`.
|
||||
|
||||
## 5. Debugging Checklist
|
||||
|
||||
1. Is the patch size reasonable? Huge patches often mean binary blobs; zero-byte
|
||||
patches often mean the wrong workspace was captured.
|
||||
2. Inspect the `*.status.txt` emitted by `capture_patch.sh`.
|
||||
3. Check raw sample field casing for `FAIL_TO_PASS` and `PASS_TO_PASS`.
|
||||
4. Compare per-instance `{prefix}_output.json` files to see exactly which tests
|
||||
are missing.
|
||||
5. Confirm that the agent did not rely on test-file edits.
|
||||
6. Classify API errors by event type and failure class.
|
||||
7. In stateful runs, check startup budget state before blaming solve quality.
|
||||
19
devtools/benchmarks/swe_bench_pro/README.md
Normal file
19
devtools/benchmarks/swe_bench_pro/README.md
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
# SWE-bench Pro Devtools
|
||||
|
||||
SWE-bench Pro is kept separate from standard SWE-bench because the colleague
|
||||
materials target the `SWE-bench_Pro-os` evaluator and a Pro-specific patch JSON
|
||||
handoff.
|
||||
|
||||
Files:
|
||||
|
||||
- `METHODOLOGY.md` documents the capture and grading assumptions.
|
||||
- `capture_patch.sh` captures a task-repository patch with untracked text files,
|
||||
filters environment junk, drops binary blobs, and requires an explicit output
|
||||
path outside the Ouroboros repo.
|
||||
- `pro_predictions.py` creates Ouroboros-style prediction JSONL by running
|
||||
`capture_patch.sh` for prepared task repositories.
|
||||
- `grade_pro.py` invokes the official Pro evaluator when `--skip-run` is not
|
||||
supplied, then aggregates official per-instance outputs.
|
||||
|
||||
The aggregation in `grade_pro.py` is not replacement scoring. The official Pro
|
||||
eval output remains the source of truth.
|
||||
1
devtools/benchmarks/swe_bench_pro/__init__.py
Normal file
1
devtools/benchmarks/swe_bench_pro/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
|||
"""SWE-bench Pro helper scripts."""
|
||||
70
devtools/benchmarks/swe_bench_pro/capture_patch.sh
Executable file
70
devtools/benchmarks/swe_bench_pro/capture_patch.sh
Executable file
|
|
@ -0,0 +1,70 @@
|
|||
#!/usr/bin/env bash
|
||||
# Capture a SWE-bench Pro model_patch from the task repository.
|
||||
#
|
||||
# Patch capture determines what the official evaluator sees. A plain
|
||||
# `git diff BASE` misses new untracked files, while an unfiltered `git add -A`
|
||||
# captures runtime junk such as Redis dumps, node_modules, and compiled
|
||||
# binaries. This helper follows the SWE-agent/mini-swe-agent reference shape,
|
||||
# then removes environment artifacts and binary blobs.
|
||||
#
|
||||
# Usage:
|
||||
# ./capture_patch.sh <REPO_DIR> <BASE_COMMIT> <OUT.diff>
|
||||
#
|
||||
# The agent is expected to have already edited <REPO_DIR>. <BASE_COMMIT> is the
|
||||
# task base commit from the dataset.
|
||||
set -euo pipefail
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd -P)"
|
||||
WORK="${1:?usage: capture_patch.sh <REPO_DIR> <BASE_COMMIT> <OUT.diff>}"
|
||||
BASE="${2:?base_commit is required}"
|
||||
OUT="${3:?output path is required and must be outside the Ouroboros repo}"
|
||||
OUT_ABS="$(python3 - "$OUT" <<'PY'
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
print(pathlib.Path(sys.argv[1]).expanduser().resolve(strict=False))
|
||||
PY
|
||||
)"
|
||||
case "$OUT_ABS" in
|
||||
"$REPO_ROOT"|"$REPO_ROOT"/*)
|
||||
echo "output path must be outside the Ouroboros repo: $OUT_ABS" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
OUT_DIR_ABS="$(dirname "$OUT_ABS")"
|
||||
mkdir -p "$OUT_DIR_ABS"
|
||||
STATUS_OUT="${OUT_ABS%.diff}.status.txt"
|
||||
|
||||
git -C "$WORK" rev-parse --verify "$BASE^{commit}" >/dev/null
|
||||
cleanup() {
|
||||
git -C "$WORK" reset -q >/dev/null 2>&1 || true
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
# (1) Include newly created source files. Several real Pro fixes add files, and
|
||||
# a clean `git diff BASE` would omit them.
|
||||
git -C "$WORK" add -A
|
||||
|
||||
# Keep a status snapshot for mismatch debugging: M=modified, A=added,
|
||||
# ??=untracked.
|
||||
git -C "$WORK" status --porcelain >"$STATUS_OUT"
|
||||
|
||||
# (2) Drop environment artifacts. These patterns were chosen to avoid broad
|
||||
# SWE-agent defaults such as *.cfg/*.toml/setup.py/*.lock, which can remove real
|
||||
# Pro fixes.
|
||||
JUNK_RE='appendonlydir|\.rdb$|\.aof$|\.manifest$|\.log$|\.tmp$|\.pid$|\.sock$|(^|/)node_modules/|__pycache__|\.pyc$|\.pyo$|\.pytest_cache|\.ruff_cache|\.mypy_cache|/\.cache/|(^|/)dist/|(^|/)build/|\.DS_Store|(^|/)\.coverage$|coverage\.xml$|/htmlcov/'
|
||||
while IFS= read -r f; do
|
||||
git -C "$WORK" reset -q -- "$f" 2>/dev/null
|
||||
done < <(git -C "$WORK" diff --cached --name-only "$BASE" | grep -E "$JUNK_RE" || true)
|
||||
|
||||
# (3) Drop binary blobs. `git diff --cached --numstat` prints
|
||||
# "-\t-\t<file>" for binary files. Text source additions remain included.
|
||||
git -C "$WORK" diff --cached --numstat "$BASE" | awk -F'\t' '$1=="-" && $2=="-" {print $3}' | while IFS= read -r f; do
|
||||
[ -n "$f" ] && git -C "$WORK" reset -q -- "$f" 2>/dev/null
|
||||
done
|
||||
|
||||
# (4) Emit final model_patch and restore the index without touching the working
|
||||
# tree.
|
||||
git -C "$WORK" diff --cached --binary "$BASE" >"$OUT_ABS"
|
||||
|
||||
echo "patch -> $OUT_ABS ($(wc -c <"$OUT_ABS" 2>/dev/null || echo 0)B, files: $(grep -cE '^diff --git' "$OUT_ABS" 2>/dev/null || echo 0))" >&2
|
||||
135
devtools/benchmarks/swe_bench_pro/grade_pro.py
Normal file
135
devtools/benchmarks/swe_bench_pro/grade_pro.py
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Run official SWE-bench Pro eval, then print a diagnostic summary.
|
||||
|
||||
This wrapper does not replace Pro scoring. The official ``swe_bench_pro_eval.py``
|
||||
output remains the source of truth. The post-run table only helps inspect
|
||||
per-instance output files.
|
||||
|
||||
python pro/grade_pro.py --predictions runs/pro_smoke/predictions.jsonl --workers 4
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse, ast, csv, json, os, pathlib, subprocess, sys
|
||||
|
||||
if __package__ in {None, ""}:
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[3]))
|
||||
|
||||
from devtools.benchmarks.common.run_roots import ensure_outside_repo, repo_root_from_devtools, run_root
|
||||
|
||||
PRO = pathlib.Path(__file__).resolve().parent
|
||||
DEFAULT_RUN_ROOT = run_root("swe_bench_pro")
|
||||
EVAL_REPO = DEFAULT_RUN_ROOT / "SWE-bench_Pro-os"
|
||||
CSV_DEFAULT = PRO / "task_order_pro_70.csv"
|
||||
|
||||
|
||||
def load_predictions(path: pathlib.Path) -> list[dict]:
|
||||
return [json.loads(l) for l in path.read_text().splitlines() if l.strip()]
|
||||
|
||||
|
||||
def colleague_verdicts(csv_path: pathlib.Path) -> dict:
|
||||
with csv_path.open() as f:
|
||||
return {r["instance_id"]: r["verdict"] for r in csv.DictReader(f)}
|
||||
|
||||
|
||||
def raw_sample_index(raw_sample: pathlib.Path) -> dict:
|
||||
"""Return raw official sample rows by instance id for diagnostic inspection."""
|
||||
idx = {}
|
||||
for l in raw_sample.read_text(errors="replace").splitlines():
|
||||
if not l.strip():
|
||||
continue
|
||||
try:
|
||||
d = json.loads(l)
|
||||
except Exception:
|
||||
continue
|
||||
idx[d.get("instance_id", "")] = d
|
||||
return idx
|
||||
|
||||
|
||||
def as_set(v) -> set:
|
||||
if isinstance(v, str):
|
||||
try:
|
||||
return set(json.loads(v))
|
||||
except Exception:
|
||||
try:
|
||||
return set(ast.literal_eval(v))
|
||||
except Exception:
|
||||
return set()
|
||||
return set(v or [])
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--predictions", default=str(DEFAULT_RUN_ROOT / "predictions.jsonl"))
|
||||
ap.add_argument("--out-dir", default=str(DEFAULT_RUN_ROOT / "pro_eval"))
|
||||
ap.add_argument("--eval-repo", default=str(EVAL_REPO), help="external checkout of scaleapi/SWE-bench_Pro-os")
|
||||
ap.add_argument("--prefix", default="ours")
|
||||
ap.add_argument("--workers", type=int, default=4)
|
||||
ap.add_argument("--platform", default="linux/amd64")
|
||||
ap.add_argument("--skip-run", action="store_true", help="aggregate an existing out-dir without running official eval")
|
||||
ap.add_argument("--csv", default="", help="optional CSV with instance_id,verdict columns for baseline comparison")
|
||||
args = ap.parse_args()
|
||||
|
||||
eval_repo = pathlib.Path(args.eval_repo).expanduser().resolve()
|
||||
raw_sample = eval_repo / "helper_code" / "sweap_eval_full_v2.jsonl"
|
||||
if not eval_repo.is_dir():
|
||||
print(f"error: eval harness clone is missing: {eval_repo}\n git clone --depth 1 https://github.com/scaleapi/SWE-bench_Pro-os.git {eval_repo}", file=sys.stderr)
|
||||
return 2
|
||||
if not raw_sample.is_file():
|
||||
print(f"error: raw_sample is missing: {raw_sample}", file=sys.stderr); return 2
|
||||
|
||||
preds = load_predictions(pathlib.Path(args.predictions).expanduser())
|
||||
preds = [p for p in preds if (p.get("model_patch") or "").strip()]
|
||||
out_dir = ensure_outside_repo(pathlib.Path(args.out_dir), repo_root_from_devtools())
|
||||
patches = [{"instance_id": p["instance_id"], "patch": p["model_patch"], "prefix": args.prefix} for p in preds]
|
||||
patches_path = out_dir / "patches.json"
|
||||
patches_path.write_text(json.dumps(patches), encoding="utf-8")
|
||||
print(f"[swe-pro] wrote {len(patches)} patches to {patches_path}", file=sys.stderr)
|
||||
|
||||
if not args.skip_run:
|
||||
cmd = [sys.executable, "swe_bench_pro_eval.py",
|
||||
"--raw_sample_path", str(raw_sample),
|
||||
"--patch_path", str(patches_path),
|
||||
"--output_dir", str(out_dir),
|
||||
"--scripts_dir", "run_scripts",
|
||||
"--num_workers", str(args.workers),
|
||||
"--dockerhub_username", "jefzda",
|
||||
"--use_local_docker", "--docker_platform", args.platform]
|
||||
print("[swe-pro] running official eval:\n " + " ".join(cmd), file=sys.stderr)
|
||||
r = subprocess.run(cmd, cwd=str(eval_repo))
|
||||
if r.returncode != 0:
|
||||
print(f"[swe-pro] official eval returned {r.returncode}; summarizing available output", file=sys.stderr)
|
||||
|
||||
rs = raw_sample_index(raw_sample)
|
||||
csv_path = pathlib.Path(args.csv).expanduser() if args.csv else CSV_DEFAULT
|
||||
verd = colleague_verdicts(csv_path) if csv_path.is_file() else {}
|
||||
print("\n[diagnostic] Non-leaderboard summary derived from official per-instance outputs.")
|
||||
print(f"{'instance':52} {'diagnostic':18} {'baseline':10} {'tests P/missing/total'}")
|
||||
n_res = 0
|
||||
for p in patches:
|
||||
iid = p["instance_id"]
|
||||
od = out_dir / iid / f"{args.prefix}_output.json"
|
||||
if not od.is_file():
|
||||
od2 = list((out_dir).glob(f"**/{args.prefix}_output.json"))
|
||||
od = next((x for x in od2 if iid in str(x)), od)
|
||||
status = "NO_OUTPUT"; ntp = "-"
|
||||
if od.is_file():
|
||||
try:
|
||||
out = json.loads(od.read_text())
|
||||
tests = out.get("tests") or []
|
||||
passed = {t["name"] for t in tests if t.get("status") == "PASSED"}
|
||||
rsd = rs.get(iid, {})
|
||||
f2p = as_set(rsd.get("FAIL_TO_PASS") or rsd.get("fail_to_pass"))
|
||||
p2p = as_set(rsd.get("PASS_TO_PASS") or rsd.get("pass_to_pass"))
|
||||
need = f2p | p2p
|
||||
resolved = bool(need) and need <= passed
|
||||
n_res += int(resolved)
|
||||
status = "DIAGNOSTIC_PASS" if resolved else "DIAGNOSTIC_FAIL"
|
||||
ntp = f"{len(passed)}/{len(need - passed)}/{len(need)}"
|
||||
except Exception as e:
|
||||
status = f"PARSE_ERR:{e}"[:18]
|
||||
print(f"{iid[:52]:52} {status:18} {verd.get(iid,'?'):10} {ntp}")
|
||||
print(f"\n[diagnostic] pass-like {n_res}/{len(patches)}; official Pro eval output remains source of truth.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
80
devtools/benchmarks/swe_bench_pro/pro_predictions.py
Normal file
80
devtools/benchmarks/swe_bench_pro/pro_predictions.py
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Capture SWE-bench Pro prediction patches from prepared task repositories."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
if __package__ in {None, ""}:
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
|
||||
|
||||
from devtools.benchmarks.common.run_roots import ensure_file_output_outside_repo, ensure_outside_repo, run_root, safe_benchmark_id
|
||||
|
||||
|
||||
CAPTURE = Path(__file__).resolve().parent / "capture_patch.sh"
|
||||
REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
|
||||
|
||||
def _rows(path: Path) -> list[dict[str, Any]]:
|
||||
return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line.strip()]
|
||||
|
||||
|
||||
def _capture_patch(repo_dir: Path, base_commit: str, out_path: Path) -> str:
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
proc = subprocess.run(
|
||||
["bash", str(CAPTURE), str(repo_dir), base_commit, str(out_path)],
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
timeout=120,
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
raise RuntimeError(f"capture_patch.sh failed for {repo_dir}: {proc.stderr or proc.stdout}")
|
||||
patch = out_path.read_text(encoding="utf-8", errors="replace")
|
||||
if not patch.strip():
|
||||
raise RuntimeError(f"capture_patch.sh produced an empty patch for {repo_dir}")
|
||||
return patch
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--input", required=True, help="JSONL rows with instance_id, repo_dir/workspace_root, base_commit")
|
||||
parser.add_argument("--output", required=True, help="prediction JSONL")
|
||||
parser.add_argument("--patch-dir", default="", help="optional directory for captured .diff files")
|
||||
parser.add_argument("--model-name", default="ouroboros-pro")
|
||||
args = parser.parse_args()
|
||||
|
||||
output = ensure_file_output_outside_repo(Path(args.output), REPO_ROOT)
|
||||
patch_dir = Path(args.patch_dir).expanduser() if args.patch_dir else run_root("swe_bench_pro") / "patches"
|
||||
ensure_outside_repo(patch_dir, REPO_ROOT)
|
||||
predictions: list[dict[str, str]] = []
|
||||
for item in _rows(Path(args.input).expanduser()):
|
||||
instance_id = str(item.get("instance_id") or "").strip()
|
||||
safe_instance_id = safe_benchmark_id(instance_id)
|
||||
repo_dir = Path(str(item.get("repo_dir") or item.get("workspace_root") or "")).expanduser()
|
||||
base_commit = str(item.get("base_commit") or "").strip()
|
||||
if not instance_id or not repo_dir.is_dir() or not base_commit:
|
||||
raise RuntimeError("each row must include instance_id, repo_dir/workspace_root, and base_commit")
|
||||
patch = _capture_patch(repo_dir, base_commit, patch_dir / f"{safe_instance_id}.diff")
|
||||
predictions.append(
|
||||
{
|
||||
"instance_id": instance_id,
|
||||
"model_name_or_path": args.model_name,
|
||||
"model_patch": patch,
|
||||
}
|
||||
)
|
||||
output.write_text(
|
||||
"\n".join(json.dumps(row, ensure_ascii=False) for row in predictions) + ("\n" if predictions else ""),
|
||||
encoding="utf-8",
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
364
devtools/benchmarks/terminal_bench/README.md
Normal file
364
devtools/benchmarks/terminal_bench/README.md
Normal file
|
|
@ -0,0 +1,364 @@
|
|||
# Ouroboros Terminal-Bench / Harbor Installed Adapter
|
||||
|
||||
## Short Summary
|
||||
|
||||
The current adapter runs **full Ouroboros inside each Terminal-Bench task
|
||||
container**.
|
||||
|
||||
High-level flow:
|
||||
|
||||
1. Harbor creates the official Terminal-Bench task container.
|
||||
2. The adapter uploads the current local Ouroboros `repo/` source into that
|
||||
container at `/opt/ouroboros-src`.
|
||||
3. The adapter creates an isolated venv at `/opt/ouroboros-venv`.
|
||||
4. The adapter installs Ouroboros from the uploaded source.
|
||||
5. The adapter starts an in-container Ouroboros server/supervisor on
|
||||
`127.0.0.1:8765`.
|
||||
6. The adapter submits the official Terminal-Bench instruction as an external
|
||||
workspace task, with `/app` or `/workspace` as the workspace root.
|
||||
7. Ouroboros solves the task using its normal runtime/tools.
|
||||
8. Harbor runs the official verifier.
|
||||
|
||||
This is intentionally **not** the old host-side terminal bridge. Ouroboros is
|
||||
not asked to return one shell command per turn. It runs as normal inside the
|
||||
task container.
|
||||
|
||||
## Why Installed Mode
|
||||
|
||||
The earlier adapter kept Ouroboros on the host and translated task state into a
|
||||
JSON command loop. That made traces look artificially weak: Ouroboros saw a
|
||||
terminal snapshot and had to return one shell command at a time.
|
||||
|
||||
The installed adapter evaluates Ouroboros more directly:
|
||||
|
||||
- each trial gets a fresh Ouroboros runtime;
|
||||
- each trial gets a fresh `/logs/agent/ouroboros-data` data directory;
|
||||
- the task workspace is passed as `workspace_root`;
|
||||
- Ouroboros uses normal workspace tools and shell tools internally;
|
||||
- Harbor still owns the task container and verifier.
|
||||
|
||||
## What Is Copied Into The Container
|
||||
|
||||
The adapter copies the current local source tree:
|
||||
|
||||
```text
|
||||
/Users/anton/Ouroboros/repo -> /opt/ouroboros-src
|
||||
```
|
||||
|
||||
It deliberately excludes local runtime/state noise:
|
||||
|
||||
```text
|
||||
.git
|
||||
.venv
|
||||
data
|
||||
data_evaluated
|
||||
__pycache__
|
||||
.pytest_cache
|
||||
.ruff_cache
|
||||
build
|
||||
dist
|
||||
node_modules
|
||||
```
|
||||
|
||||
So the benchmark container gets current code, but not the operator's main
|
||||
Ouroboros memory, logs, task results, or chat history.
|
||||
|
||||
## Runtime State In The Container
|
||||
|
||||
Each trial uses:
|
||||
|
||||
```text
|
||||
OUROBOROS_REPO_DIR=/opt/ouroboros-src
|
||||
OUROBOROS_DATA_DIR=/logs/agent/ouroboros-data
|
||||
OUROBOROS_SETTINGS_PATH=/logs/agent/ouroboros-data/settings.json
|
||||
OUROBOROS_RUNTIME_MODE=pro
|
||||
OUROBOROS_REVIEW_ENFORCEMENT=advisory
|
||||
OUROBOROS_WORKER_START_METHOD=spawn
|
||||
```
|
||||
|
||||
This means:
|
||||
|
||||
```text
|
||||
1 benchmark task = 1 fresh in-container Ouroboros = 1 unique ouroboros-data folder
|
||||
```
|
||||
|
||||
The host `/Users/anton/Ouroboros/data` is not copied into the container.
|
||||
|
||||
## Provider Secret Boundary
|
||||
|
||||
Installed-container mode does not inject long-lived provider credentials into
|
||||
Terminal-Bench task containers by default. If host settings or environment
|
||||
contain provider keys, the adapter fails closed with a clear error instead of
|
||||
starting a container that can expose those keys to in-container shell tools.
|
||||
|
||||
The intended durable solution is a reviewed host-mediated LLM bridge with scoped
|
||||
task credentials. For trusted local smoke runs only, an operator may set:
|
||||
|
||||
```bash
|
||||
OUROBOROS_BENCH_ALLOW_CONTAINER_SECRETS=1
|
||||
```
|
||||
|
||||
Do not use that opt-in for publishable benchmark runs unless the task container,
|
||||
logs, and output root are under operator control and the risk is explicitly
|
||||
accepted.
|
||||
|
||||
## Task Instruction Integrity
|
||||
|
||||
The adapter now passes the official Terminal-Bench instruction unchanged:
|
||||
|
||||
```python
|
||||
"description": instruction
|
||||
```
|
||||
|
||||
It does not prepend harness notes or task-specific hints.
|
||||
|
||||
The only technical wrapper is the API request metadata and `workspace_root`.
|
||||
|
||||
## Workspace Resolution
|
||||
|
||||
Most Terminal-Bench tasks use `/app`.
|
||||
|
||||
Some images use `/workspace`. The adapter resolves this before starting the
|
||||
Ouroboros task:
|
||||
|
||||
1. use `/app` if it exists;
|
||||
2. otherwise use `/workspace` if it exists;
|
||||
3. otherwise create `/app`.
|
||||
|
||||
The selected path is passed as external workspace root.
|
||||
|
||||
## Lifecycle
|
||||
|
||||
Harbor calls:
|
||||
|
||||
```python
|
||||
await agent.setup(environment)
|
||||
await agent.run(instruction, environment, context)
|
||||
```
|
||||
|
||||
`setup()` is inherited from Harbor's `BaseInstalledAgent`; it calls our
|
||||
`install()`.
|
||||
|
||||
`install()`:
|
||||
|
||||
1. uploads clean Ouroboros source into `/opt/ouroboros-src`;
|
||||
2. installs system basics (`git`, `curl`, `bash`, Python/venv support);
|
||||
3. if the system Python is older than 3.10, installs Python 3.12 with `uv`;
|
||||
4. creates `/opt/ouroboros-venv`;
|
||||
5. installs requirements and editable Ouroboros.
|
||||
|
||||
`run()`:
|
||||
|
||||
1. uploads the task instruction to `/logs/agent/instruction.txt`;
|
||||
2. checks configured provider/network reachability;
|
||||
3. resolves `/app` vs `/workspace`;
|
||||
4. ensures the workspace is a git worktree root;
|
||||
5. starts in-container Ouroboros server;
|
||||
6. creates an Ouroboros task through `/api/tasks`;
|
||||
7. polls `/api/tasks/<task_id>` until a final status;
|
||||
8. saves task result and trace files;
|
||||
9. shuts down the in-container server and cancels the task if needed.
|
||||
|
||||
## Why Direct API Polling
|
||||
|
||||
The adapter originally used:
|
||||
|
||||
```bash
|
||||
ouroboros run --jsonl ...
|
||||
```
|
||||
|
||||
That was fragile because the CLI stream could hang or get cancelled while the
|
||||
internal task already had a final state.
|
||||
|
||||
The current adapter uses direct API lifecycle:
|
||||
|
||||
```text
|
||||
POST /api/tasks
|
||||
GET /api/tasks/<task_id>
|
||||
POST /api/tasks/<task_id>/cancel
|
||||
```
|
||||
|
||||
This gives the adapter a task id immediately, lets it capture task state on
|
||||
timeout/cancellation, and avoids depending on an SSE/CLI stream.
|
||||
|
||||
## Timeout Semantics
|
||||
|
||||
The adapter does **not** set an internal task timeout by default:
|
||||
|
||||
```python
|
||||
task_timeout_sec = None
|
||||
```
|
||||
|
||||
That means Harbor controls agent execution timeout from the task config:
|
||||
|
||||
```text
|
||||
task.toml [agent].timeout_sec
|
||||
```
|
||||
|
||||
Setup and environment timeouts are separate:
|
||||
|
||||
- environment build/start: Harbor environment timeout;
|
||||
- agent setup: Harbor setup timeout;
|
||||
- agent execution: task `[agent].timeout_sec`;
|
||||
- verifier: task `[verifier].timeout_sec`.
|
||||
|
||||
For heavy Docker builds, use:
|
||||
|
||||
```bash
|
||||
--environment-build-timeout-multiplier 4
|
||||
```
|
||||
|
||||
For installed Ouroboros setup, use:
|
||||
|
||||
```bash
|
||||
--agent-setup-timeout-multiplier 4
|
||||
```
|
||||
|
||||
## Common Commands
|
||||
|
||||
### Terminal-Bench 2.1 smoke
|
||||
|
||||
```bash
|
||||
PYTHONPATH=/Users/anton/Ouroboros/repo \
|
||||
harbor run \
|
||||
--dataset terminal-bench/terminal-bench-2-1 \
|
||||
--include-task-name terminal-bench/regex-log \
|
||||
--agent-import-path devtools.benchmarks.terminal_bench.harbor_installed_agent:OuroborosTerminalBenchAgent \
|
||||
--model ouroboros-gpt-5.5-tb21-smoke \
|
||||
--agent-kwarg ouroboros_model=openai/gpt-5.5 \
|
||||
--agent-kwarg install_timeout_sec=1200 \
|
||||
--agent-kwarg server_start_timeout_sec=240 \
|
||||
--agent-setup-timeout-multiplier 4 \
|
||||
--n-concurrent 1 \
|
||||
--n-tasks 1 \
|
||||
--yes \
|
||||
--force-build
|
||||
```
|
||||
|
||||
### Full cached Terminal-Bench 2.0-style dataset
|
||||
|
||||
```bash
|
||||
PYTHONPATH=/Users/anton/Ouroboros/repo \
|
||||
harbor run \
|
||||
--path /Users/anton/Ouroboros/data/harbor_local_datasets/terminal_bench_full_cached_89 \
|
||||
--agent-import-path devtools.benchmarks.terminal_bench.harbor_installed_agent:OuroborosTerminalBenchAgent \
|
||||
--model ouroboros-gpt-5.5-full \
|
||||
--agent-kwarg ouroboros_model=openai/gpt-5.5 \
|
||||
--agent-kwarg install_timeout_sec=1200 \
|
||||
--agent-kwarg server_start_timeout_sec=240 \
|
||||
--agent-setup-timeout-multiplier 4 \
|
||||
--n-concurrent 1 \
|
||||
--yes \
|
||||
--force-build
|
||||
```
|
||||
|
||||
### Full Terminal-Bench 2.1
|
||||
|
||||
```bash
|
||||
PYTHONPATH=/Users/anton/Ouroboros/repo \
|
||||
harbor run \
|
||||
--dataset terminal-bench/terminal-bench-2-1 \
|
||||
--agent-import-path devtools.benchmarks.terminal_bench.harbor_installed_agent:OuroborosTerminalBenchAgent \
|
||||
--model ouroboros-gpt-5.5-tb21-full \
|
||||
--agent-kwarg ouroboros_model=openai/gpt-5.5 \
|
||||
--agent-kwarg install_timeout_sec=1200 \
|
||||
--agent-kwarg server_start_timeout_sec=240 \
|
||||
--agent-setup-timeout-multiplier 4 \
|
||||
--environment-build-timeout-multiplier 4 \
|
||||
--n-concurrent 1 \
|
||||
--yes \
|
||||
--force-build
|
||||
```
|
||||
|
||||
## Model Selection
|
||||
|
||||
Harbor's `--model` is metadata for the Harbor result.
|
||||
|
||||
The actual Ouroboros model is passed via:
|
||||
|
||||
```bash
|
||||
--agent-kwarg ouroboros_model=<provider/model>
|
||||
```
|
||||
|
||||
Examples:
|
||||
|
||||
```bash
|
||||
--agent-kwarg ouroboros_model=openai/gpt-5.5
|
||||
--agent-kwarg ouroboros_model=google/gemini-3.5-flash
|
||||
--agent-kwarg ouroboros_model=anthropic/claude-opus-4-7
|
||||
```
|
||||
|
||||
The adapter sets:
|
||||
|
||||
```text
|
||||
OUROBOROS_MODEL
|
||||
OUROBOROS_MODEL_CODE
|
||||
OUROBOROS_MODEL_LIGHT
|
||||
```
|
||||
|
||||
to that value inside the container.
|
||||
|
||||
## Trace Locations
|
||||
|
||||
For each Harbor trial:
|
||||
|
||||
```text
|
||||
<trial>/agent/ouroboros-data/
|
||||
```
|
||||
|
||||
contains the fresh in-container Ouroboros data directory.
|
||||
|
||||
Useful files:
|
||||
|
||||
```text
|
||||
<trial>/agent/ouroboros-data/logs/events.jsonl
|
||||
<trial>/agent/ouroboros-data/logs/progress.jsonl
|
||||
<trial>/agent/ouroboros-data/logs/supervisor.jsonl
|
||||
<trial>/agent/ouroboros-data/state/headless_tasks/<task_id>/data/logs/tools.jsonl
|
||||
<trial>/agent/ouroboros-task-result.json
|
||||
<trial>/agent/ouroboros-run.jsonl
|
||||
<trial>/agent/ouroboros-run-summary.json
|
||||
<trial>/verifier/test-stdout.txt
|
||||
<trial>/verifier/reward.txt
|
||||
```
|
||||
|
||||
Heavy files usually come from:
|
||||
|
||||
```text
|
||||
<trial>/agent/ouroboros-data/task_results/artifacts/<task_id>/workspace.patch
|
||||
<trial>/agent/ouroboros-data/task_results/artifacts/<task_id>/workspace_patch.json
|
||||
```
|
||||
|
||||
Those can be omitted when creating logs-only bundles.
|
||||
|
||||
## Known Infrastructure Notes
|
||||
|
||||
- Old task images with Python 3.9 require adapter-installed Python 3.12 via
|
||||
`uv`; this is handled automatically.
|
||||
- Some task Docker builds need more than 600 seconds; use
|
||||
`--environment-build-timeout-multiplier`.
|
||||
- Some tasks still hit Harbor `AgentTimeoutError`; verifier can still produce a
|
||||
reward if the workspace has enough final state.
|
||||
- `RuntimeError` from the adapter should not be used for ordinary Ouroboros
|
||||
`status=failed`; the adapter records task status and returns control so Harbor
|
||||
can run the verifier.
|
||||
|
||||
## Files To Share With Developers
|
||||
|
||||
Minimum:
|
||||
|
||||
```text
|
||||
repo/devtools/benchmarks/terminal_bench/harbor_installed_agent.py
|
||||
```
|
||||
|
||||
Recommended:
|
||||
|
||||
```text
|
||||
repo/devtools/benchmarks/terminal_bench/README.md
|
||||
```
|
||||
|
||||
Useful example result:
|
||||
|
||||
```text
|
||||
data/harbor_jobs/ouroboros_v650_tb21_smoke_gpt55/2026-05-29__00-39-23/result.json
|
||||
```
|
||||
1
devtools/benchmarks/terminal_bench/__init__.py
Normal file
1
devtools/benchmarks/terminal_bench/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
|||
"""Terminal-Bench Harbor adapter."""
|
||||
729
devtools/benchmarks/terminal_bench/harbor_installed_agent.py
Normal file
729
devtools/benchmarks/terminal_bench/harbor_installed_agent.py
Normal file
|
|
@ -0,0 +1,729 @@
|
|||
"""Harbor installed-agent entrypoint for evaluating full Ouroboros in Terminal-Bench.
|
||||
|
||||
This adapter intentionally does not translate Ouroboros decisions into shell
|
||||
commands. Harbor starts a task container, this class installs Ouroboros inside
|
||||
that container, starts the normal Ouroboros server/supervisor, and submits the
|
||||
Terminal-Bench instruction as an external workspace task rooted at ``/app``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import shlex
|
||||
import shutil
|
||||
import tempfile
|
||||
import textwrap
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
try: # Harbor is an optional benchmark dependency.
|
||||
from harbor.agents.installed.base import BaseInstalledAgent
|
||||
from harbor.environments.base import BaseEnvironment
|
||||
from harbor.models.agent.context import AgentContext
|
||||
except Exception: # pragma: no cover - exercised when Harbor is absent.
|
||||
BaseInstalledAgent = object # type: ignore[assignment]
|
||||
BaseEnvironment = Any # type: ignore[assignment]
|
||||
AgentContext = Any # type: ignore[assignment]
|
||||
|
||||
|
||||
_CONTAINER_SRC = "/opt/ouroboros-src"
|
||||
_CONTAINER_VENV = "/opt/ouroboros-venv"
|
||||
_CONTAINER_DATA = "/logs/agent/ouroboros-data"
|
||||
_CONTAINER_WORKSPACE = "/app"
|
||||
_SERVER_URL = "http://127.0.0.1:8765"
|
||||
_CONTAINER_SECRET_OPT_IN = "OUROBOROS_BENCH_ALLOW_CONTAINER_SECRETS"
|
||||
_SECRET_ENV_KEYS = frozenset({
|
||||
"ANTHROPIC_API_KEY",
|
||||
"CLOUDRU_FOUNDATION_MODELS_API_KEY",
|
||||
"GIGACHAT_CREDENTIALS",
|
||||
"GIGACHAT_PASSWORD",
|
||||
"GIGACHAT_USER",
|
||||
"OPENAI_API_KEY",
|
||||
"OPENAI_COMPATIBLE_API_KEY",
|
||||
"OPENROUTER_API_KEY",
|
||||
})
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _workspace_root() -> Path:
|
||||
return Path(__file__).resolve().parents[4]
|
||||
|
||||
|
||||
def _default_host_settings_path() -> Path:
|
||||
return Path(os.environ.get("OUROBOROS_SETTINGS_PATH") or _workspace_root() / "data" / "settings.json")
|
||||
|
||||
|
||||
def _json_load(path: Path) -> dict[str, Any]:
|
||||
try:
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def _repo_root() -> Path:
|
||||
return Path(__file__).resolve().parents[3]
|
||||
|
||||
|
||||
def _copy_clean_source(source: Path, target: Path) -> None:
|
||||
excluded_dirs = {
|
||||
".git",
|
||||
".mypy_cache",
|
||||
".pytest_cache",
|
||||
".ruff_cache",
|
||||
".venv",
|
||||
"__pycache__",
|
||||
"build",
|
||||
"data",
|
||||
"data_evaluated",
|
||||
"dist",
|
||||
"node-standalone",
|
||||
"node_modules",
|
||||
"python-standalone",
|
||||
"venv",
|
||||
}
|
||||
excluded_suffixes = {".key", ".pem", ".pfx", ".p12", ".pyc", ".pyo"}
|
||||
excluded_names = {
|
||||
".DS_Store",
|
||||
".env",
|
||||
".env.dev",
|
||||
".env.development",
|
||||
".env.example",
|
||||
".env.local",
|
||||
".env.production",
|
||||
".env.staging",
|
||||
".env.test",
|
||||
".git-credentials",
|
||||
".netrc",
|
||||
".npmrc",
|
||||
".pypirc",
|
||||
".release_notes.md",
|
||||
"credentials.json",
|
||||
"aws-credentials.json",
|
||||
"gcp-service-account.json",
|
||||
"id_ed25519",
|
||||
"id_rsa",
|
||||
"repo.bundle",
|
||||
"repo_bundle_manifest.json",
|
||||
"service-account.json",
|
||||
"secrets.ini",
|
||||
"secrets.json",
|
||||
"secrets.toml",
|
||||
"secrets.yaml",
|
||||
}
|
||||
|
||||
def ignore(_: str, names: list[str]) -> set[str]:
|
||||
ignored: set[str] = set()
|
||||
for name in names:
|
||||
if name in excluded_dirs or name in excluded_names:
|
||||
ignored.add(name)
|
||||
continue
|
||||
if any(name.endswith(suffix) for suffix in excluded_suffixes):
|
||||
ignored.add(name)
|
||||
return ignored
|
||||
|
||||
shutil.copytree(source, target, ignore=ignore, symlinks=True)
|
||||
|
||||
|
||||
class OuroborosTerminalBenchAgent(BaseInstalledAgent):
|
||||
"""Install and run full Ouroboros inside the Terminal-Bench task container."""
|
||||
|
||||
SUPPORTS_WINDOWS = False
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
logs_dir: Path,
|
||||
model_name: str = "ouroboros-inside",
|
||||
*args: Any,
|
||||
**kwargs: Any,
|
||||
) -> None:
|
||||
workspace_dir = str(kwargs.pop("workspace_dir", _CONTAINER_WORKSPACE))
|
||||
host_settings_path = str(kwargs.pop("host_settings_path", ""))
|
||||
install_timeout_sec = int(kwargs.pop("install_timeout_sec", 900))
|
||||
server_start_timeout_sec = int(kwargs.pop("server_start_timeout_sec", 180))
|
||||
task_timeout_sec = kwargs.pop("task_timeout_sec", None)
|
||||
max_workers = int(kwargs.pop("max_workers", 1))
|
||||
runtime_mode = str(kwargs.pop("runtime_mode", "pro"))
|
||||
review_enforcement = str(kwargs.pop("review_enforcement", "advisory"))
|
||||
ouroboros_model = str(kwargs.pop("ouroboros_model", ""))
|
||||
try:
|
||||
super().__init__(*args, logs_dir=logs_dir, model_name=model_name, **kwargs)
|
||||
except TypeError:
|
||||
super().__init__()
|
||||
self.logs_dir = Path(logs_dir)
|
||||
self.model_name = model_name
|
||||
self.workspace_dir = workspace_dir
|
||||
self.host_settings_path = Path(
|
||||
host_settings_path
|
||||
or os.environ.get("OUROBOROS_SETTINGS_PATH")
|
||||
or _default_host_settings_path()
|
||||
).expanduser()
|
||||
self.install_timeout_sec = int(install_timeout_sec)
|
||||
self.server_start_timeout_sec = int(server_start_timeout_sec)
|
||||
self.task_timeout_sec = (
|
||||
int(task_timeout_sec)
|
||||
if task_timeout_sec is not None and int(task_timeout_sec) > 0
|
||||
else None
|
||||
)
|
||||
self.max_workers = int(max_workers)
|
||||
self.runtime_mode = runtime_mode
|
||||
self.review_enforcement = review_enforcement
|
||||
self.ouroboros_model = ouroboros_model
|
||||
self._run_summary: dict[str, Any] = {}
|
||||
|
||||
@staticmethod
|
||||
def name() -> str:
|
||||
return "Ouroboros Installed"
|
||||
|
||||
def version(self) -> str | None:
|
||||
return "0.2.0"
|
||||
|
||||
def _host_settings(self) -> dict[str, Any]:
|
||||
return _json_load(self.host_settings_path)
|
||||
|
||||
def _container_secret_injection_allowed(self, settings: dict[str, Any]) -> bool:
|
||||
value = os.environ.get(_CONTAINER_SECRET_OPT_IN)
|
||||
if value is None:
|
||||
value = settings.get(_CONTAINER_SECRET_OPT_IN)
|
||||
return str(value or "").strip().lower() in {"1", "true", "yes", "allow"}
|
||||
|
||||
def _available_host_secret_keys(self, settings: dict[str, Any]) -> list[str]:
|
||||
keys: list[str] = []
|
||||
for key in sorted(_SECRET_ENV_KEYS):
|
||||
if str(os.environ.get(key) or settings.get(key) or "").strip():
|
||||
keys.append(key)
|
||||
return keys
|
||||
|
||||
def _enforce_container_secret_policy(self, env: dict[str, str]) -> None:
|
||||
settings = self._host_settings()
|
||||
blocked = self._available_host_secret_keys(settings)
|
||||
if blocked and not self._container_secret_injection_allowed(settings):
|
||||
names = ", ".join(blocked)
|
||||
raise RuntimeError(
|
||||
"Terminal-Bench installed-container mode refuses to inject long-lived provider "
|
||||
f"credentials into task containers by default ({names}). Use a host-mediated LLM "
|
||||
"bridge when available, or set OUROBOROS_BENCH_ALLOW_CONTAINER_SECRETS=1 only for "
|
||||
"trusted local smoke runs where the task container and logs are under operator control."
|
||||
)
|
||||
if not blocked and not any(key in env for key in _SECRET_ENV_KEYS):
|
||||
return
|
||||
|
||||
def _container_env(self) -> dict[str, str]:
|
||||
settings = self._host_settings()
|
||||
allow_secrets = self._container_secret_injection_allowed(settings)
|
||||
keys = [
|
||||
"OPENAI_BASE_URL",
|
||||
"OPENAI_COMPATIBLE_BASE_URL",
|
||||
"CLOUDRU_FOUNDATION_MODELS_BASE_URL",
|
||||
"GIGACHAT_SCOPE",
|
||||
"GIGACHAT_BASE_URL",
|
||||
"GIGACHAT_VERIFY_SSL_CERTS",
|
||||
"GIGACHAT_PROFANITY_CHECK",
|
||||
"OUROBOROS_MODEL",
|
||||
"OUROBOROS_MODEL_CODE",
|
||||
"OUROBOROS_MODEL_LIGHT",
|
||||
"OUROBOROS_MODEL_FALLBACK",
|
||||
"OUROBOROS_WEBSEARCH_MODEL",
|
||||
"OUROBOROS_SCOPE_REVIEW_MODEL",
|
||||
"OUROBOROS_EFFORT_TASK",
|
||||
"OUROBOROS_RETURN_REASONING",
|
||||
"TOTAL_BUDGET",
|
||||
"OUROBOROS_PER_TASK_COST_USD",
|
||||
"OUROBOROS_SOFT_TIMEOUT_SEC",
|
||||
"OUROBOROS_HARD_TIMEOUT_SEC",
|
||||
"OUROBOROS_TOOL_TIMEOUT_SEC",
|
||||
]
|
||||
if allow_secrets:
|
||||
keys.extend(sorted(_SECRET_ENV_KEYS))
|
||||
env: dict[str, str] = {}
|
||||
for key in keys:
|
||||
value = os.environ.get(key)
|
||||
if value is None:
|
||||
value = settings.get(key)
|
||||
if value not in (None, ""):
|
||||
env[key] = str(value)
|
||||
|
||||
if self.ouroboros_model:
|
||||
env["OUROBOROS_MODEL"] = self.ouroboros_model
|
||||
env["OUROBOROS_MODEL_CODE"] = self.ouroboros_model
|
||||
env["OUROBOROS_MODEL_LIGHT"] = self.ouroboros_model
|
||||
|
||||
env.update(
|
||||
{
|
||||
"OUROBOROS_REPO_DIR": _CONTAINER_SRC,
|
||||
"OUROBOROS_DATA_DIR": _CONTAINER_DATA,
|
||||
"OUROBOROS_SETTINGS_PATH": f"{_CONTAINER_DATA}/settings.json",
|
||||
"OUROBOROS_PID_FILE": "/logs/agent/ouroboros.pid",
|
||||
"OUROBOROS_PORT_FILE": f"{_CONTAINER_DATA}/state/server_port",
|
||||
"OUROBOROS_SERVER_HOST": "127.0.0.1",
|
||||
"OUROBOROS_SERVER_PORT": "8765",
|
||||
"OUROBOROS_WORKER_START_METHOD": "spawn",
|
||||
"OUROBOROS_RUNTIME_MODE": self.runtime_mode,
|
||||
"OUROBOROS_REVIEW_ENFORCEMENT": self.review_enforcement,
|
||||
"OUROBOROS_MAX_WORKERS": str(self.max_workers),
|
||||
"PYTHONUNBUFFERED": "1",
|
||||
}
|
||||
)
|
||||
return env
|
||||
|
||||
async def _append_log(self, environment: BaseEnvironment, message: str) -> None:
|
||||
safe = json.dumps(message)
|
||||
await environment.exec(
|
||||
command=f"mkdir -p /logs/agent && python3 - <<'PY'\n"
|
||||
"from pathlib import Path\n"
|
||||
f"Path('/logs/agent/ouroboros-install.log').open('a', encoding='utf-8').write({safe} + '\\n')\n"
|
||||
"PY",
|
||||
user="root",
|
||||
)
|
||||
|
||||
async def _upload_source(self, environment: BaseEnvironment) -> None:
|
||||
source = _repo_root()
|
||||
with tempfile.TemporaryDirectory(prefix="ouroboros-tb-src-") as tmp:
|
||||
clean = Path(tmp) / "repo"
|
||||
_copy_clean_source(source, clean)
|
||||
await environment.exec(
|
||||
command=f"rm -rf {_CONTAINER_SRC} && mkdir -p {_CONTAINER_SRC}",
|
||||
user="root",
|
||||
)
|
||||
await environment.upload_dir(clean, _CONTAINER_SRC)
|
||||
|
||||
async def install(self, environment: BaseEnvironment) -> None:
|
||||
started = time.monotonic()
|
||||
await self._append_log(environment, "install: starting source upload")
|
||||
await self._upload_source(environment)
|
||||
await self._append_log(environment, "install: source uploaded")
|
||||
|
||||
install_cmd = textwrap.dedent(
|
||||
f"""
|
||||
set -euo pipefail
|
||||
mkdir -p /logs/agent {_CONTAINER_DATA}/logs {_CONTAINER_DATA}/state
|
||||
{{
|
||||
echo "install: installing system dependencies"
|
||||
if command -v apt-get >/dev/null 2>&1; then
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends git curl bash ca-certificates procps python3 python3-venv python3-pip
|
||||
elif command -v apk >/dev/null 2>&1; then
|
||||
apk add --no-cache git curl bash ca-certificates procps python3 py3-pip py3-virtualenv
|
||||
elif command -v yum >/dev/null 2>&1; then
|
||||
yum install -y git curl bash ca-certificates procps python3 python3-pip
|
||||
else
|
||||
echo "install: no known package manager; assuming required tools already exist"
|
||||
fi
|
||||
|
||||
PYTHON_BIN="$(command -v python3 || command -v python)"
|
||||
PY_OK="$("$PYTHON_BIN" - <<'PY'
|
||||
import sys
|
||||
print(1 if sys.version_info >= (3, 10) else 0)
|
||||
PY
|
||||
)"
|
||||
if [ "$PY_OK" != "1" ]; then
|
||||
echo "install: system Python is too old; installing Python 3.12 with uv"
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
uv python install 3.12
|
||||
PYTHON_BIN="$(uv python find 3.12)"
|
||||
fi
|
||||
echo "install: using $PYTHON_BIN"
|
||||
"$PYTHON_BIN" -m venv {_CONTAINER_VENV} || {{
|
||||
"$PYTHON_BIN" -m pip install --break-system-packages --user virtualenv || "$PYTHON_BIN" -m pip install --user virtualenv
|
||||
"$PYTHON_BIN" -m virtualenv {_CONTAINER_VENV}
|
||||
}}
|
||||
|
||||
. {_CONTAINER_VENV}/bin/activate
|
||||
python -m pip install --upgrade pip setuptools wheel
|
||||
python -m pip install -r {_CONTAINER_SRC}/requirements.txt
|
||||
python -m pip install -e {_CONTAINER_SRC} --no-deps
|
||||
chmod -R a+rX {_CONTAINER_SRC} {_CONTAINER_VENV} /logs/agent
|
||||
{_CONTAINER_VENV}/bin/python -c 'import importlib.metadata; print("ouroboros", importlib.metadata.version("ouroboros"))'
|
||||
echo "install: complete"
|
||||
}} 2>&1 | tee -a /logs/agent/ouroboros-install.log
|
||||
"""
|
||||
).strip()
|
||||
await self.exec_as_root(
|
||||
environment,
|
||||
command=install_cmd,
|
||||
timeout_sec=self.install_timeout_sec,
|
||||
)
|
||||
elapsed = time.monotonic() - started
|
||||
await self._append_log(environment, f"install: elapsed_sec={elapsed:.1f}")
|
||||
|
||||
async def _ensure_workspace_git_root(self, environment: BaseEnvironment) -> None:
|
||||
workspace_dir = shlex.quote(self.workspace_dir)
|
||||
command = textwrap.dedent(
|
||||
f"""
|
||||
set -euo pipefail
|
||||
workspace_dir={workspace_dir}
|
||||
cd "$workspace_dir"
|
||||
if git rev-parse --show-toplevel >/tmp/ouroboros-git-root 2>/dev/null; then
|
||||
root="$(cat /tmp/ouroboros-git-root)"
|
||||
if [ "$root" != "$workspace_dir" ]; then
|
||||
echo "workspace git root is $root, expected $workspace_dir" >&2
|
||||
exit 2
|
||||
fi
|
||||
else
|
||||
git init
|
||||
git config user.email ouroboros-bench@example.invalid
|
||||
git config user.name "Ouroboros Bench"
|
||||
fi
|
||||
"""
|
||||
).strip()
|
||||
result = await environment.exec(command=command, cwd=self.workspace_dir, timeout_sec=60)
|
||||
if result.return_code != 0:
|
||||
raise RuntimeError(
|
||||
f"failed to prepare {self.workspace_dir} as git workspace: "
|
||||
f"stdout={result.stdout!r} stderr={result.stderr!r}"
|
||||
)
|
||||
|
||||
async def _resolve_workspace_dir(self, environment: BaseEnvironment) -> None:
|
||||
"""Use /app when present, but support tasks whose Dockerfile uses /workspace."""
|
||||
requested = self.workspace_dir
|
||||
quoted_requested = shlex.quote(requested)
|
||||
result = await environment.exec(command=f"test -d {quoted_requested}", timeout_sec=10)
|
||||
if result.return_code == 0:
|
||||
return
|
||||
if requested == _CONTAINER_WORKSPACE:
|
||||
fallback = await environment.exec(command="test -d /workspace", timeout_sec=10)
|
||||
if fallback.return_code == 0:
|
||||
self.workspace_dir = "/workspace"
|
||||
await self._append_log(environment, "workspace: /app missing, using /workspace")
|
||||
return
|
||||
create = await environment.exec(command=f"mkdir -p {quoted_requested}", user="root", timeout_sec=10)
|
||||
if create.return_code != 0:
|
||||
raise RuntimeError(
|
||||
f"failed to create workspace {requested}: stdout={create.stdout!r} stderr={create.stderr!r}"
|
||||
)
|
||||
await self._append_log(environment, f"workspace: created {requested}")
|
||||
|
||||
async def _start_server(self, environment: BaseEnvironment, env: dict[str, str]) -> None:
|
||||
start_cmd = textwrap.dedent(
|
||||
f"""
|
||||
set -euo pipefail
|
||||
mkdir -p {_CONTAINER_DATA}/logs {_CONTAINER_DATA}/state /logs/agent
|
||||
rm -f /logs/agent/ouroboros.pid
|
||||
cd {_CONTAINER_SRC}
|
||||
nohup {_CONTAINER_VENV}/bin/python server.py --host 127.0.0.1 --port 8765 \
|
||||
> /logs/agent/ouroboros-server.stdout.log \
|
||||
2> /logs/agent/ouroboros-server.stderr.log &
|
||||
echo "$!" > /logs/agent/ouroboros.pid
|
||||
"""
|
||||
).strip()
|
||||
result = await environment.exec(command=start_cmd, env=env, timeout_sec=30)
|
||||
if result.return_code != 0:
|
||||
raise RuntimeError(f"failed to start Ouroboros server: {result.stdout}\n{result.stderr}")
|
||||
|
||||
wait_cmd = textwrap.dedent(
|
||||
f"""
|
||||
{_CONTAINER_VENV}/bin/python - <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
deadline = time.time() + {self.server_start_timeout_sec}
|
||||
last_error = ""
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
with urllib.request.urlopen("{_SERVER_URL}/api/state", timeout=5) as resp:
|
||||
data = json.loads(resp.read().decode("utf-8"))
|
||||
pathlib.Path("/logs/agent/ouroboros-state.json").write_text(
|
||||
json.dumps(data, ensure_ascii=False, indent=2),
|
||||
encoding="utf-8",
|
||||
)
|
||||
if data.get("supervisor_ready"):
|
||||
print(json.dumps({{"ready": True, "state": data}}, ensure_ascii=False))
|
||||
sys.exit(0)
|
||||
last_error = "server responded but supervisor_ready=false"
|
||||
except Exception as exc:
|
||||
last_error = repr(exc)
|
||||
time.sleep(2)
|
||||
print(json.dumps({{"ready": False, "error": last_error}}, ensure_ascii=False))
|
||||
sys.exit(1)
|
||||
PY
|
||||
"""
|
||||
).strip()
|
||||
result = await environment.exec(command=wait_cmd, env=env, timeout_sec=self.server_start_timeout_sec + 20)
|
||||
if result.return_code != 0:
|
||||
raise RuntimeError(f"Ouroboros server did not become ready: {result.stdout}\n{result.stderr}")
|
||||
|
||||
async def _network_preflight(self, environment: BaseEnvironment, env: dict[str, str]) -> None:
|
||||
provider_url = ""
|
||||
provider_name = ""
|
||||
if env.get("OPENROUTER_API_KEY"):
|
||||
provider_url = "https://openrouter.ai/api/v1/models"
|
||||
provider_name = "openrouter"
|
||||
elif env.get("OPENAI_API_KEY"):
|
||||
provider_url = "https://api.openai.com/v1/models"
|
||||
provider_name = "openai"
|
||||
elif env.get("ANTHROPIC_API_KEY"):
|
||||
provider_url = "https://api.anthropic.com/v1/models"
|
||||
provider_name = "anthropic"
|
||||
elif env.get("CLOUDRU_FOUNDATION_MODELS_API_KEY"):
|
||||
provider_url = (env.get("CLOUDRU_FOUNDATION_MODELS_BASE_URL") or "https://foundation-models.api.cloud.ru/v1").rstrip("/") + "/models"
|
||||
provider_name = "cloudru"
|
||||
elif env.get("GIGACHAT_CREDENTIALS") or (env.get("GIGACHAT_USER") and env.get("GIGACHAT_PASSWORD")):
|
||||
provider_url = (env.get("GIGACHAT_BASE_URL") or "https://gigachat.devices.sberbank.ru/api/v1").rstrip("/") + "/models"
|
||||
provider_name = "gigachat"
|
||||
if not provider_url:
|
||||
(self.logs_dir / "network-preflight.txt").write_text(
|
||||
"provider preflight skipped: no provider API key was injected; "
|
||||
"Ouroboros runtime will surface provider configuration errors.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return
|
||||
command = textwrap.dedent(
|
||||
f"""
|
||||
python3 - <<'PY'
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
req = urllib.request.Request({provider_url!r}, method="GET")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=10) as resp:
|
||||
print({provider_name!r} + "_preflight_status", resp.status)
|
||||
sys.exit(0 if 200 <= resp.status < 500 else 1)
|
||||
except urllib.error.HTTPError as exc:
|
||||
print({provider_name!r} + "_preflight_status", exc.code)
|
||||
sys.exit(0 if 200 <= exc.code < 500 else 1)
|
||||
except Exception as exc:
|
||||
print({provider_name!r} + "_preflight_error", type(exc).__name__)
|
||||
sys.exit(1)
|
||||
PY
|
||||
"""
|
||||
).strip()
|
||||
result = await environment.exec(command=command, timeout_sec=20)
|
||||
(self.logs_dir / "network-preflight.txt").write_text(
|
||||
f"stdout:\n{result.stdout or ''}\nstderr:\n{result.stderr or ''}\nreturn_code={result.return_code}\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
if result.return_code != 0:
|
||||
raise RuntimeError(f"container cannot reach configured provider endpoint ({provider_name})")
|
||||
|
||||
async def _run_ouroboros_task(self, environment: BaseEnvironment, env: dict[str, str]) -> dict[str, Any]:
|
||||
workspace_root = json.dumps(self.workspace_dir)
|
||||
runner = textwrap.dedent(
|
||||
f"""
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
import time
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
instruction = pathlib.Path("/logs/agent/instruction.txt").read_text(encoding="utf-8")
|
||||
started = time.time()
|
||||
run_log = pathlib.Path("/logs/agent/ouroboros-run.jsonl")
|
||||
stderr_log = pathlib.Path("/logs/agent/ouroboros-run.stderr.log")
|
||||
task_id_path = pathlib.Path("/logs/agent/ouroboros-current-task-id.txt")
|
||||
|
||||
def api(method, path, body=None, timeout=30):
|
||||
data = None
|
||||
headers = {{"Accept": "application/json"}}
|
||||
if body is not None:
|
||||
data = json.dumps(body, ensure_ascii=False).encode("utf-8")
|
||||
headers["Content-Type"] = "application/json"
|
||||
req = urllib.request.Request("{_SERVER_URL}" + path, data=data, headers=headers, method=method)
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
raw = resp.read().decode("utf-8", errors="replace")
|
||||
return json.loads(raw) if raw.strip() else {{}}
|
||||
|
||||
def emit(event):
|
||||
with run_log.open("a", encoding="utf-8") as f:
|
||||
f.write(json.dumps(event, ensure_ascii=False) + "\\n")
|
||||
|
||||
created = api("POST", "/api/tasks", {{
|
||||
"description": instruction,
|
||||
"workspace_root": {workspace_root},
|
||||
"workspace_mode": "external",
|
||||
"memory_mode": "empty",
|
||||
"actor_id": "harbor-terminal-bench",
|
||||
"source": "terminal-bench",
|
||||
"metadata": {{"source": "terminal-bench", "delegation_role": "root"}},
|
||||
}})
|
||||
task_id = str(created.get("task_id") or "")
|
||||
if not task_id:
|
||||
stderr_log.write_text(f"task creation did not return task_id: {{created!r}}\\n", encoding="utf-8")
|
||||
print(json.dumps({{"return_code": 1, "elapsed_sec": round(time.time() - started, 3), "status": "create_failed"}}))
|
||||
sys.exit(1)
|
||||
task_id_path.write_text(task_id, encoding="utf-8")
|
||||
emit({{"type": "task_created", "task_id": task_id, "data": created}})
|
||||
|
||||
latest = {{}}
|
||||
seen_events = set()
|
||||
final_statuses = {{"completed", "failed", "cancelled", "rejected_duplicate"}}
|
||||
while True:
|
||||
result = api("GET", "/api/tasks/" + urllib.parse.quote(task_id), timeout=30)
|
||||
for event in result.get("events") or []:
|
||||
key = (str(event.get("type") or ""), str(event.get("ts") or event.get("seq") or ""))
|
||||
if key in seen_events:
|
||||
continue
|
||||
seen_events.add(key)
|
||||
emit({{"type": "task_event", "task_id": task_id, "data": event}})
|
||||
status = str(result.get("status") or "")
|
||||
if status in final_statuses:
|
||||
latest = result
|
||||
pathlib.Path("/logs/agent/ouroboros-task-result.json").write_text(
|
||||
json.dumps(latest, ensure_ascii=False, indent=2),
|
||||
encoding="utf-8",
|
||||
)
|
||||
emit({{"type": "final", "task_id": task_id, "result": latest}})
|
||||
break
|
||||
time.sleep(2)
|
||||
|
||||
status = str(latest.get("status") or "")
|
||||
summary = {{
|
||||
"return_code": 0,
|
||||
"task_status_code": 0 if status == "completed" else 1,
|
||||
"elapsed_sec": round(time.time() - started, 3),
|
||||
"task_id": latest.get("task_id") or latest.get("id"),
|
||||
"status": status,
|
||||
"cost_usd": latest.get("cost_usd"),
|
||||
"prompt_tokens": latest.get("prompt_tokens"),
|
||||
"completion_tokens": latest.get("completion_tokens"),
|
||||
"total_rounds": latest.get("total_rounds"),
|
||||
}}
|
||||
pathlib.Path("/logs/agent/ouroboros-run-summary.json").write_text(
|
||||
json.dumps(summary, ensure_ascii=False, indent=2),
|
||||
encoding="utf-8",
|
||||
)
|
||||
print(json.dumps(summary, ensure_ascii=False))
|
||||
sys.exit(0)
|
||||
"""
|
||||
).strip()
|
||||
command = "cat > /tmp/run_ouroboros_task.py <<'PY'\n" + runner + "\nPY\n" + (
|
||||
f"{_CONTAINER_VENV}/bin/python /tmp/run_ouroboros_task.py"
|
||||
)
|
||||
result = await environment.exec(
|
||||
command=command,
|
||||
env=env,
|
||||
cwd=self.workspace_dir,
|
||||
timeout_sec=(self.task_timeout_sec + 60 if self.task_timeout_sec is not None else None),
|
||||
)
|
||||
if result.return_code != 0:
|
||||
raise RuntimeError(f"Ouroboros task runner failed: {result.stdout}\n{result.stderr}")
|
||||
try:
|
||||
return json.loads((result.stdout or "").strip().splitlines()[-1])
|
||||
except Exception:
|
||||
return {"raw_stdout": result.stdout or "", "raw_stderr": result.stderr or ""}
|
||||
|
||||
async def _stop_server(self, environment: BaseEnvironment) -> None:
|
||||
await environment.exec(
|
||||
command=(
|
||||
"if [ -s /logs/agent/ouroboros-current-task-id.txt ]; then "
|
||||
"TASK_ID=$(cat /logs/agent/ouroboros-current-task-id.txt); "
|
||||
"export TASK_ID; "
|
||||
f"{_CONTAINER_VENV}/bin/python - <<'PY' || true\n"
|
||||
"import os, urllib.parse, urllib.request\n"
|
||||
"task_id = os.environ.get('TASK_ID', '')\n"
|
||||
"if task_id:\n"
|
||||
f" urllib.request.urlopen(urllib.request.Request('{_SERVER_URL}/api/tasks/' + urllib.parse.quote(task_id) + '/cancel', data=b'{{}}', method='POST'), timeout=5).read()\n"
|
||||
"PY\n"
|
||||
"fi; "
|
||||
"if [ -f /logs/agent/ouroboros.pid ]; then "
|
||||
"kill $(cat /logs/agent/ouroboros.pid) 2>/dev/null || true; "
|
||||
"fi; "
|
||||
"pkill -TERM -f '/opt/ouroboros-src|/opt/ouroboros-venv/bin/ouroboros' 2>/dev/null || true"
|
||||
),
|
||||
timeout_sec=10,
|
||||
)
|
||||
|
||||
async def _capture_current_task_summary(self, environment: BaseEnvironment) -> None:
|
||||
"""Persist best-effort task state if Harbor cancels agent.run mid-exec."""
|
||||
command = textwrap.dedent(
|
||||
f"""
|
||||
if [ ! -s /logs/agent/ouroboros-current-task-id.txt ]; then
|
||||
exit 0
|
||||
fi
|
||||
{_CONTAINER_VENV}/bin/python - <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import time
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
task_id = pathlib.Path("/logs/agent/ouroboros-current-task-id.txt").read_text(encoding="utf-8").strip()
|
||||
if not task_id:
|
||||
raise SystemExit(0)
|
||||
try:
|
||||
with urllib.request.urlopen("{_SERVER_URL}/api/tasks/" + urllib.parse.quote(task_id), timeout=10) as resp:
|
||||
latest = json.loads(resp.read().decode("utf-8", errors="replace"))
|
||||
except Exception as exc:
|
||||
pathlib.Path("/logs/agent/ouroboros-run.stderr.log").open("a", encoding="utf-8").write(
|
||||
"best-effort task summary failed: " + repr(exc) + "\\n"
|
||||
)
|
||||
raise SystemExit(0)
|
||||
|
||||
pathlib.Path("/logs/agent/ouroboros-task-result.json").write_text(
|
||||
json.dumps(latest, ensure_ascii=False, indent=2),
|
||||
encoding="utf-8",
|
||||
)
|
||||
status = str(latest.get("status") or "")
|
||||
summary = {{
|
||||
"return_code": 0,
|
||||
"task_status_code": 0 if status == "completed" else 1,
|
||||
"elapsed_sec": None,
|
||||
"task_id": latest.get("task_id") or latest.get("id") or task_id,
|
||||
"status": status,
|
||||
"cost_usd": latest.get("cost_usd"),
|
||||
"prompt_tokens": latest.get("prompt_tokens"),
|
||||
"completion_tokens": latest.get("completion_tokens"),
|
||||
"total_rounds": latest.get("total_rounds"),
|
||||
"captured_after_cancellation": True,
|
||||
"captured_at": time.time(),
|
||||
}}
|
||||
pathlib.Path("/logs/agent/ouroboros-run-summary.json").write_text(
|
||||
json.dumps(summary, ensure_ascii=False, indent=2),
|
||||
encoding="utf-8",
|
||||
)
|
||||
with pathlib.Path("/logs/agent/ouroboros-run.jsonl").open("a", encoding="utf-8") as f:
|
||||
f.write(json.dumps({{"type": "final", "task_id": task_id, "result": latest, "captured_after_cancellation": True}}, ensure_ascii=False) + "\\n")
|
||||
PY
|
||||
"""
|
||||
).strip()
|
||||
await environment.exec(command=command, timeout_sec=30)
|
||||
|
||||
async def run(self, instruction: str, environment: BaseEnvironment, context: AgentContext) -> None:
|
||||
self.logs_dir.mkdir(parents=True, exist_ok=True)
|
||||
(self.logs_dir / "instruction.txt").write_text(instruction, encoding="utf-8")
|
||||
await environment.upload_file(self.logs_dir / "instruction.txt", "/logs/agent/instruction.txt")
|
||||
|
||||
env = self._container_env()
|
||||
try:
|
||||
self._enforce_container_secret_policy(env)
|
||||
await self._network_preflight(environment, env)
|
||||
await self._resolve_workspace_dir(environment)
|
||||
await self._ensure_workspace_git_root(environment)
|
||||
await self._start_server(environment, env)
|
||||
self._run_summary = await self._run_ouroboros_task(environment, env)
|
||||
finally:
|
||||
try:
|
||||
await self._capture_current_task_summary(environment)
|
||||
except Exception as exc:
|
||||
(getattr(self, "logger", None) or log).warning("Failed to capture in-container Ouroboros task summary: %s", exc)
|
||||
try:
|
||||
await self._stop_server(environment)
|
||||
except Exception as exc:
|
||||
(getattr(self, "logger", None) or log).warning("Failed to stop in-container Ouroboros cleanly: %s", exc)
|
||||
|
||||
cost = self._run_summary.get("cost_usd")
|
||||
prompt_tokens = self._run_summary.get("prompt_tokens")
|
||||
completion_tokens = self._run_summary.get("completion_tokens")
|
||||
context.cost_usd = float(cost) if cost is not None else None
|
||||
context.n_input_tokens = int(prompt_tokens) if prompt_tokens is not None else None
|
||||
context.n_output_tokens = int(completion_tokens) if completion_tokens is not None else None
|
||||
context.metadata = {
|
||||
"adapter_mode": "installed_ouroboros",
|
||||
"workspace_dir": self.workspace_dir,
|
||||
"runtime_mode": self.runtime_mode,
|
||||
"review_enforcement": self.review_enforcement,
|
||||
"summary": self._run_summary,
|
||||
}
|
||||
|
||||
|
||||
InstalledOuroborosTerminalBenchAgent = OuroborosTerminalBenchAgent
|
||||
|
||||
__all__ = ["OuroborosTerminalBenchAgent", "InstalledOuroborosTerminalBenchAgent"]
|
||||
97
devtools/benchmarks/terminal_bench/run_harbor_smoke.py
Normal file
97
devtools/benchmarks/terminal_bench/run_harbor_smoke.py
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Build a Harbor command for a one-task Terminal-Bench smoke run."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import pathlib
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
if __package__ in {None, ""}:
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[3]))
|
||||
|
||||
from devtools.benchmarks.common.run_roots import ensure_outside_repo, repo_root_from_devtools, run_root as default_run_root
|
||||
|
||||
|
||||
AGENT_IMPORT = "devtools.benchmarks.terminal_bench.harbor_installed_agent:OuroborosTerminalBenchAgent"
|
||||
|
||||
|
||||
def harbor_command(
|
||||
*,
|
||||
task_name: str,
|
||||
model: str,
|
||||
run_root: pathlib.Path,
|
||||
dataset: str = "terminal-bench/terminal-bench-2-1",
|
||||
harbor_bin: str = "harbor",
|
||||
execute: bool = False,
|
||||
) -> list[str]:
|
||||
cmd = [
|
||||
harbor_bin,
|
||||
"run",
|
||||
"--dataset",
|
||||
dataset,
|
||||
"--include-task-name",
|
||||
task_name,
|
||||
"--agent-import-path",
|
||||
AGENT_IMPORT,
|
||||
"--model",
|
||||
f"ouroboros-{model.replace('/', '-')}",
|
||||
"--agent-kwarg",
|
||||
f"ouroboros_model={model}",
|
||||
"--agent-kwarg",
|
||||
"install_timeout_sec=1200",
|
||||
"--agent-kwarg",
|
||||
"server_start_timeout_sec=240",
|
||||
"--agent-setup-timeout-multiplier",
|
||||
"4",
|
||||
"--environment-build-timeout-multiplier",
|
||||
"4",
|
||||
"--n-concurrent",
|
||||
"1",
|
||||
"--n-tasks",
|
||||
"1",
|
||||
"--output-dir",
|
||||
str(run_root),
|
||||
"--yes",
|
||||
]
|
||||
if execute:
|
||||
cmd.append("--force-build")
|
||||
return cmd
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--task", default="terminal-bench/regex-log")
|
||||
parser.add_argument("--model", default="openai/gpt-5.5")
|
||||
parser.add_argument("--dataset", default="terminal-bench/terminal-bench-2-1")
|
||||
parser.add_argument("--harbor-bin", default="harbor")
|
||||
parser.add_argument("--run-root", default="")
|
||||
parser.add_argument("--execute", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
run_root = ensure_outside_repo(
|
||||
pathlib.Path(args.run_root).expanduser() if args.run_root else default_run_root("terminal_bench"),
|
||||
repo_root_from_devtools(),
|
||||
)
|
||||
cmd = harbor_command(
|
||||
task_name=args.task,
|
||||
model=args.model,
|
||||
run_root=run_root,
|
||||
dataset=args.dataset,
|
||||
harbor_bin=args.harbor_bin,
|
||||
execute=args.execute,
|
||||
)
|
||||
manifest = {"run_root": str(run_root), "cmd": cmd, "agent_import": AGENT_IMPORT}
|
||||
run_root.mkdir(parents=True, exist_ok=True)
|
||||
(run_root / "harbor_command.json").write_text(json.dumps(manifest, indent=2), encoding="utf-8")
|
||||
print(shlex.join(cmd))
|
||||
if not args.execute:
|
||||
return 0
|
||||
return subprocess.run(cmd, cwd=pathlib.Path(__file__).resolve().parents[3]).returncode
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
# Ouroboros v6.18.1 — Architecture & Reference
|
||||
# Ouroboros v6.19.0-rc.1 — Architecture & Reference
|
||||
|
||||
This file is NOT a changelog. Version history lives in README.md, git tags, and commit log.
|
||||
|
||||
|
|
@ -106,6 +106,7 @@ server.py (Starlette+uvicorn) ← HTTP + WebSocket on configurable host:port (de
|
|||
├── git_shell_policy.py ← Structural git argv classifiers for shell safety guards
|
||||
├── protected_artifacts.py ← Task-contract protected artifact policy helpers for execute-only black-box references
|
||||
├── shell_parse.py ← Shared shell argv/inline-command parser helpers used by guardrails without importing the tools package
|
||||
├── workspace_executor.py ← Host-owned local/docker_exec workspace process backend, path mapping, executor traces, and executor service lifecycle
|
||||
├── tool_capabilities.py ← SSOT for tool sets (core, parallel-safe, truncation, browser)
|
||||
├── tool_access.py ← Tool API v2 policy matrix: ToolProfile × ResourceRoot × Operation
|
||||
├── tool_policy.py ← Tool access policy and gating (imports from tool_capabilities)
|
||||
|
|
@ -175,8 +176,7 @@ build_windows.ps1 ← Windows build (PyInstaller → .zip)
|
|||
scripts/build_repo_bundle.py ← Builds `repo.bundle` + `repo_bundle_manifest.json` for packaged releases
|
||||
scripts/run_external_review.py ← v5.1.2 dev-loop tool: invokes `ouroboros.tools.parallel_review.run_parallel_review` from outside the runtime against `git diff --cached`. Reads `~/Ouroboros/data/settings.json` for `OPENROUTER_API_KEY` / `OUROBOROS_REVIEW_MODELS` / `OUROBOROS_SCOPE_REVIEW_MODELS`, builds a minimal `ToolContext`, prints FULL raw triad+scope output (no truncation). Used to dry-run the same review pipeline `commit_reviewed` triggers before any actual commit. Output: stdout (and optional `--output PATH`). Not part of the runtime gate; review-exempt dev tool.
|
||||
scripts/cleanup_test_pollution.py ← Dry-run-first cleanup utility for local test-pollution artifacts: known test skill state dirs, stale `__extension_imports`, and accidental `MagicMock`-named repo-root files. Use `--apply` only after inspecting planned removals.
|
||||
scripts/swebench_cli_agent.py ← Helper that turns local checkout-backed SWE-bench rows into prediction JSONL via the CLI/headless task API.
|
||||
scripts/terminal_bench_cli_agent.py ← Minimal Terminal-Bench BaseAgent bridge that delegates task solving through `ouroboros run` when the task workspace is mounted on the gateway host.
|
||||
devtools/benchmarks/ ← Tracked operator benchmark tooling (ProgramBench, Terminal-Bench/Harbor, SWE-bench, SWE-bench Pro, OSWorld logs skeleton). It is reviewed when touched, is manifest-accounted by Atlas, is not imported by runtime core, and is not packaged as runtime app code.
|
||||
packaging/cli/ ← Packaged CLI shell/cmd wrappers and user-local installer launchers copied into desktop artifacts
|
||||
Dockerfile ← Docker image (web UI runtime)
|
||||
```
|
||||
|
|
@ -273,13 +273,30 @@ The CLI downloads patch artifacts through the task artifact endpoint, waits for
|
|||
artifact finalization in `--patch` / `--patch-out` mode, and fails nonzero when
|
||||
the patch is missing, empty, or failed. `--no-stream` suppresses live progress
|
||||
but still waits; `--detach` is the explicit create-and-return mode.
|
||||
Benchmark helper scripts likewise require clean per-instance local checkouts;
|
||||
they do not reset or commit benchmark workspaces.
|
||||
Benchmark devtools under `devtools/benchmarks/` require clean per-instance local
|
||||
checkouts or official benchmark containers; they do not commit target
|
||||
repositories. Broad scope/plan/deep-review packs list unrelated `devtools/`
|
||||
files in the Atlas manifest without inlining every benchmark harness, while
|
||||
touched `devtools/` files are fully included in triad/scope review. This is a
|
||||
context-management rule, not an immune-system escape hatch.
|
||||
|
||||
Workspace mode is a tool-routing and blast-radius guard, not an OS sandbox.
|
||||
Like OpenClaw's host workspace mode, absolute host paths are not a hard security
|
||||
boundary unless a future Docker/SSH/remote sandbox is added around tool
|
||||
execution. Do not grow ad-hoc shell parsing to approximate that sandbox.
|
||||
boundary unless a Docker/SSH/remote backend is added around tool execution.
|
||||
When task metadata contains a host-owned `executor_ref`, `run_command`,
|
||||
`run_script`, and service tools route process execution through the declared
|
||||
backend (`local` or `docker_exec`) only when the requested cwd is covered by an
|
||||
executor path mapping. Unmapped task-drive, artifact-store, and user-files cwd
|
||||
paths remain local host execution roots. File tools continue to operate on the
|
||||
shared host workspace. `executor_ref.network=none` is enforced by the backend
|
||||
transport for mapped backend executions, for example by requiring Docker
|
||||
`NetworkMode=none`; LLM provider traffic remains outside the benchmark tool
|
||||
environment. Executor-backed
|
||||
foreground commands and services are also written to durable
|
||||
`data/state/workspace_executor_processes/` records so server-side panic and
|
||||
emergency cleanup can stop local process groups or Docker-side pidfile/service
|
||||
processes even if the worker that started them has died. Do not grow ad-hoc
|
||||
shell parsing to approximate that sandbox.
|
||||
Project-local dependency installs are ordinary workspace work. In
|
||||
`runtime_mode=pro`, system/global dependency installs may be attempted through
|
||||
`run_command` and the safety supervisor when needed by the external workspace;
|
||||
|
|
@ -445,6 +462,7 @@ finalization states.
|
|||
│ │ ├── extension_companions.json ← Runtime snapshot for live extension companion processes
|
||||
│ │ ├── extension_reconcile/ ← Worker-written extension reconcile markers consumed by the server lifespan pickup task
|
||||
│ │ ├── review_continuations/ ← Per-task blocked-review continuation payloads (+ quarantined corrupt files under `corrupt/`)
|
||||
│ │ ├── workspace_executor_processes/ ← Durable local/docker executor foreground/service cleanup records for panic/shutdown recovery
|
||||
│ │ └── skills/ ← Phase 3 external-skill state plane (sibling of advisory_review.json, not shared)
|
||||
│ │ └── <skill_name>/
|
||||
│ │ ├── enabled.json ← {"enabled": bool, "updated_at": iso_ts}
|
||||
|
|
@ -1271,8 +1289,11 @@ The panic sequence (in `server.py:_execute_panic_stop()`):
|
|||
5. kill_all_tracked_subprocesses() ← os.killpg(SIGKILL) every tracked
|
||||
│ foreground subprocess process group
|
||||
│ (shell commands and ALL their children)
|
||||
6. kill_workers(force=True) ← SIGTERM+SIGKILL all multiprocessing workers
|
||||
7. os._exit(99) ← immediate hard exit, kills daemon threads
|
||||
6. kill_all_foreground(data_dir) ← stop durable executor-backed foreground
|
||||
│ local/docker processes
|
||||
7. kill_all_services(data_dir) ← stop service and executor-service groups
|
||||
8. kill_workers(force=True) ← SIGTERM+SIGKILL all multiprocessing workers
|
||||
9. os._exit(99) ← immediate hard exit, kills daemon threads
|
||||
```
|
||||
|
||||
Launcher handles exit code 99:
|
||||
|
|
@ -1300,17 +1321,21 @@ use `start_new_session=True` via `_tracked_subprocess_run()` in
|
|||
`ouroboros/tools/services.py::_start_service`, which starts each service with
|
||||
`subprocess_new_group_kwargs()` and records it in the `_SERVICES` registry.
|
||||
Both paths create a separate process group for each subprocess and its children.
|
||||
Executor-backed workspace processes additionally record local pids, Docker
|
||||
pidfiles, and service pids under `data/state/workspace_executor_processes/` so
|
||||
panic can clean them up from the server process after worker death.
|
||||
|
||||
On panic or timeout, the entire process tree is killed via
|
||||
`os.killpg(pgid, SIGKILL)` — no orphans possible, even for deeply nested
|
||||
foreground shell/script/service subprocess trees.
|
||||
Panic/emergency paths call `kill_all_tracked_subprocesses()` and
|
||||
`kill_all_services()` without log finalization so emergency stop remains fast;
|
||||
normal lifespan shutdown may pass a drive root to `kill_all_services(drive_root)`
|
||||
to archive server-process service logs before removing live log files. Services
|
||||
started inside worker tasks normally finalize in `loop.py` task cleanup; forced
|
||||
worker termination kills the worker process tree and archives remaining task
|
||||
service logs best-effort from `data/services/<task_id>/`.
|
||||
`kill_all_foreground(data_dir)` plus `kill_all_services(data_dir)` without log
|
||||
finalization so emergency stop remains fast; normal lifespan shutdown may pass a
|
||||
drive root to `kill_all_services(drive_root)` to archive server-process service
|
||||
logs before removing live log files. Services started inside worker tasks
|
||||
normally finalize in `loop.py` task cleanup; forced worker termination kills the
|
||||
worker process tree and archives remaining task service logs best-effort from
|
||||
`data/services/<task_id>/`.
|
||||
|
||||
Active subprocesses are tracked in a thread-safe global set and cleaned up
|
||||
automatically on completion or via `kill_all_tracked_subprocesses()` on panic.
|
||||
|
|
@ -1409,7 +1434,7 @@ via `tests/test_contracts.py`.
|
|||
|----------|------|-------------|
|
||||
| `ToolContextProtocol` — workspace/task-aware minimum every tool handler relies on (attributes: `repo_dir`, `drive_root`, `budget_drive_root`, `pending_events`, `emit_progress_fn`, `current_chat_id`, `task_id`, `task_metadata`, `task_contract`, `workspace_root`, `workspace_mode`; methods: `repo_path`, `drive_path`, `drive_logs`, `active_repo_dir`, `is_workspace_mode`) | `ouroboros/contracts/tool_context.py` | `ouroboros.tools.registry.ToolContext` must satisfy it (duck-typed check + AST field/method parity) |
|
||||
| `ToolEntryProtocol` + `GetToolsProtocol` — the tool-module ABI | `ouroboros/contracts/tool_abi.py` | Every entry returned by `ToolRegistry._entries` must satisfy `ToolEntryProtocol` |
|
||||
| `api_v1` WS/HTTP envelopes — inbound: `ChatInbound`, `CommandInbound`; outbound WS: `ChatOutbound`, `PhotoOutbound`, `VideoOutbound`, `TypingOutbound`, `LogOutbound`, `ExtensionLifecycleOutbound`; HTTP: `HealthResponse`, `StateResponse` (Phase 2 adds `runtime_mode: str` and `skills_repo_configured: bool`; v5.11.0 adds `github_token_configured: bool`; v6.13.0 adds `context_mode: str`), `EvolutionStateSnapshot`, `SettingsNetworkMeta`, `SettingsMeta` (`custom_secret_keys` + setup contract metadata) | `ouroboros/gateway/contracts.py` | AST scans of `supervisor/message_bus.py` chat/media envelopes, `gateway/state.py::api_state`, `gateway/state.py::api_health`, `gateway/settings.py::_build_network_meta`, and `gateway/ws.py::ws_endpoint` inbound dispatch assert no un-declared keys leak out; `tests/test_contracts.py::test_state_response_declares_phase2_runtime_mode_keys` explicitly pins the Phase 2 fields and later additive state keys |
|
||||
| `api_v1` WS/HTTP envelopes — inbound: `ChatInbound`, `CommandInbound`; outbound WS: `ChatOutbound`, `PhotoOutbound`, `VideoOutbound`, `TypingOutbound`, `LogOutbound`, `ExtensionLifecycleOutbound`; HTTP: `HealthResponse`, `StateResponse` (Phase 2 adds `runtime_mode: str` and `skills_repo_configured: bool`; v5.11.0 adds `github_token_configured: bool`; v6.13.0 adds `context_mode: str`), `TaskCreateRequest` + `ExecutorRef` for host-owned executor-backed external workspace tasks, `TaskCreateResponse`, `EvolutionStateSnapshot`, `SettingsNetworkMeta`, `SettingsMeta` (`custom_secret_keys` + setup contract metadata) | `ouroboros/gateway/contracts.py` | AST scans of `supervisor/message_bus.py` chat/media envelopes, `gateway/state.py::api_state`, `gateway/state.py::api_health`, `gateway/settings.py::_build_network_meta`, and `gateway/ws.py::ws_endpoint` inbound dispatch assert no un-declared keys leak out; `tests/test_contracts.py::test_state_response_declares_runtime_and_capability_keys` explicitly pins runtime/capability state keys, and `tests/test_contracts.py::test_task_create_request_declares_executor_ref_contract` pins the executor request surface |
|
||||
| `chat_id_policy` — SSOT for A2A/synthetic chat-id filtering across message bus, history, memory, and consolidation | `ouroboros/contracts/chat_id_policy.py` | `tests/test_chat_id_policy.py` pins boundaries and human/transport positive ids |
|
||||
| `task_contract` — canonical host-draft task objective/output/constraint/resource/deadline/workspace/lineage contract helpers (`build_task_contract`, `attach_task_contract`, `normalize_allowed_resources`, `normalize_resource_policy`) | `ouroboros/contracts/task_contract.py` | `tests/test_contracts.py::test_public_api_is_stable` pins the public helper names; task/outcome tests pin resource and resource-policy normalization plus contract propagation |
|
||||
| `PluginAPI` (Phase 4, v1.3) + `ExtensionRegistrationError` + `FORBIDDEN_EXTENSION_SETTINGS` + `VALID_EXTENSION_PERMISSIONS` + `VALID_EXTENSION_ROUTE_METHODS` — the surface every `type: extension` skill's `plugin.py::register(api)` binds against (`register_tool`, `register_route`, `register_ws_handler`, `register_ui_tab`, `register_settings_section`, `register_supervised_task`, `register_companion_process`, `subscribe_event`, `get_skill_token`, `send_ws_message`, `on_unload`, `log`, `get_settings`, `get_state_dir`, `skill_job_dir`, `get_runtime_info`). `skill_job_dir(job_id)` creates isolated `jobs/<sanitized_id>-<hash>/{assets,output,tmp}` state folders so generation skills do not overwrite their own assets across jobs. `VALID_EXTENSION_PERMISSIONS` includes host-mediated permissions (`companion_process`, `supervised_task`, `subscribe_event`, `inject_chat`) that require review/owner grants as documented in CHECKLISTS.md. The `ExecutionMode` capability matrix (`MATRIX_CAPABILITIES` / `OUT_OF_PROCESS_UNAVAILABLE_CAPABILITIES` / `capability_available` / `available_capabilities`) is the SSOT for which side-effect surfaces an out-of-process child may use and is pinned by the contract test. | `ouroboros/contracts/plugin_api.py` | `tests/test_contracts.py::test_plugin_api_surface_is_frozen` pins the frozen method set; `tests/test_contracts.py::test_extension_route_methods_contract_matches_server_dispatch` pins the route-methods tuple; `tests/test_extension_loader.py::test_plugin_api_impl_matches_protocol` asserts the concrete `PluginAPIImpl` structurally satisfies the runtime-checkable Protocol |
|
||||
|
|
|
|||
|
|
@ -111,6 +111,7 @@ or Intent/Scope checklists are.
|
|||
| 13 | Changing LLM output-token budgets? | Grep the whole repo for `max_tokens`, `max_completion_tokens`, `_MAX_TOKENS`, and `max_toks`. Keep `docs/ARCHITECTURE.md` §LLM output token budgets and `tests/test_max_tokens_constants.py` in sync so main-loop, VLM, summaries, compaction, skill publish, and consciousness floors cannot drift independently. |
|
||||
| 14 | Changing extension loader/dispatch or isolated deps? | Native-risk extension imports and tool/route/WS handlers must stay out-of-process. Add or run regression tests where a native-risk plugin aborts during import and the host survives, plus tool/route child-dispatch tests. Do not "fix" failures by importing native-risk plugin code in `server.py`. |
|
||||
| 15 | Changing `supervisor/git_ops.py`, `launcher.py`, `server.py`, `ouroboros/tools/review_helpers.py`, `ouroboros/tools/git.py`, tests, or evolution scheduling/checkpoint code? | Prove two invariants before review spend: (1) pytest/preflight cannot mutate the live repo or live `data/` (`OUROBOROS_DATA_DIR` / `OUROBOROS_SETTINGS_PATH` must be isolated, and `OUROBOROS_MANAGED_BY_LAUNCHER` must not leak into test subprocesses); (2) autonomous restart/reset cannot erase active evolution work — it must either land a reviewed local commit or preserve a rescue/transaction recovery pointer and pause/stop the campaign. |
|
||||
| 16 | Changing `devtools/benchmarks/`? | Confirm it preserves official benchmark boundaries: no replacement scoring, no benchmark-specific prompt/routing hacks, no generated benchmark outputs under `repo/`, no secrets printed or committed, and no runtime-core imports from `devtools/`. Touched `devtools` files are reviewable executable operator code, even though unrelated `devtools` files use Atlas `excluded_dir` coverage-manifest entries and stay compact in broad packs. |
|
||||
|
||||
Rule: read before write. Never reconstruct `VERSION`, `pyproject.toml`
|
||||
`version`, or the README badge from memory — one stale reconstruction creates
|
||||
|
|
@ -138,7 +139,7 @@ Used by `commit_reviewed` for all changes to the Ouroboros repository.
|
|||
| # | item | what to check | severity when FAIL |
|
||||
|---|------|---------------|--------------------|
|
||||
| 1 | bible_compliance | Does the diff violate any BIBLE.md principle? | critical |
|
||||
| 2 | development_compliance | Does it follow DEVELOPMENT.md patterns? Check explicitly: (a) naming conventions (snake_case modules/vars, PascalCase classes, UPPER_SNAKE_CASE constants); (b) entity type rules — Gateway classes contain ONLY transport, no business logic; Tool functions are thin wrappers; (c) module-size target stays near one context window (~1000 lines) with a hard fail above 1600 lines for non-grandfathered modules, method-size target stays under 150 lines with a hard fail above 300 lines, codebase-wide total Python function/method count stays under the smoke hard gate defined by `ouroboros/review.py::MAX_TOTAL_FUNCTIONS` (the literal value evolves with the codebase — consult the constant rather than hardcoding the number), and functions keep `<= 8` params; (d) no gratuitous abstract layers (P7 Minimalism); (e) new LLM calls go through the shared `LLMClient`/`llm.py` layer, not ad-hoc HTTP clients; (f) cognitive artifacts (identity.md, scratchpad, task reflections, review outputs) must NOT use hardcoded `[:N]` truncation — explicit omission notes required; (g) new `get_tools()` exports follow the ToolEntry pattern in registry.py; (h) provider independence — no change may make a core capability (agent loop, multi-model commit review, scope review, or memory/context flows) silently require a second provider or OpenRouter specifically, and every supported single direct provider (local, OpenAI, Anthropic, Cloud.ru, GigaChat) must keep its model AND review/scope slots self-fillable (see DEVELOPMENT.md "Provider Independence"). | critical |
|
||||
| 2 | development_compliance | Does it follow DEVELOPMENT.md patterns? Check explicitly: (a) naming conventions (snake_case modules/vars, PascalCase classes, UPPER_SNAKE_CASE constants); (b) entity type rules — Gateway classes contain ONLY transport, no business logic; Tool functions are thin wrappers; (c) module-size target stays near one context window (~1000 lines) with a hard fail above 1600 lines for non-grandfathered modules, method-size target stays under 150 lines with a hard fail above 300 lines, runtime-code total Python function/method count stays under the smoke hard gate defined by `ouroboros/review.py::MAX_TOTAL_FUNCTIONS` (the literal value evolves with the codebase — consult the constant rather than hardcoding the number; `devtools/` is excluded from this health gate but reviewed when touched), and functions keep `<= 8` params; (d) no gratuitous abstract layers (P7 Minimalism); (e) new LLM calls go through the shared `LLMClient`/`llm.py` layer, not ad-hoc HTTP clients; (f) cognitive artifacts (identity.md, scratchpad, task reflections, review outputs) must NOT use hardcoded `[:N]` truncation — explicit omission notes required; (g) new `get_tools()` exports follow the ToolEntry pattern in registry.py; (h) provider independence — no change may make a core capability (agent loop, multi-model commit review, scope review, or memory/context flows) silently require a second provider or OpenRouter specifically, and every supported single direct provider (local, OpenAI, Anthropic, Cloud.ru, GigaChat) must keep its model AND review/scope slots self-fillable (see DEVELOPMENT.md "Provider Independence"). | critical |
|
||||
| 3 | secrets_check | Are secrets, API keys, .env files, credentials present in the diff? | critical |
|
||||
| 4 | code_quality | Careful code review: bugs, logic errors, crashes, regressions, race conditions, resource leaks? | critical |
|
||||
| 5 | security_issues | Security vulnerabilities: injection, path traversal, secret leakage, unsafe operations? | critical |
|
||||
|
|
|
|||
|
|
@ -168,7 +168,7 @@ Derived from P7 (Minimalism): entire codebase fits in one context window.
|
|||
- Module hard gate: 1600 lines for non-grandfathered modules in `tests/test_smoke.py`. Grandfathered (`GRANDFATHERED_OVERSIZED_MODULES` in `ouroboros/review.py`): `llm.py`, `claude_advisory_review.py`, `review_state.py`, `server.py`, temporary v5.7.1 debt `git.py`, and temporary v6.15/v6.16 debt `extension_loader.py` (OOP extension parity plus worker->server companion reconcile crossed the gate; the registry-coupled `PluginAPIImpl`/loader split is the deferred follow-up) — split deferred until each surface stabilises, with `git.py` expected to pay down in the next tools pass.
|
||||
- Method target: <150 lines. Crossing that line is a decomposition signal, not an automatic failure by itself.
|
||||
- Method hard gate: 300 lines in `tests/test_smoke.py`.
|
||||
- Codebase-wide function-count hard gate: enforced by `tests/test_smoke.py` against the value defined in `ouroboros/review.py::MAX_TOTAL_FUNCTIONS` (single source of truth — bump the constant when adding a feature with an explicit comment justifying the increase).
|
||||
- Runtime-code function-count hard gate: enforced by `tests/test_smoke.py` against the value defined in `ouroboros/review.py::MAX_TOTAL_FUNCTIONS` (single source of truth — bump the constant when adding a feature with an explicit comment justifying the increase). Tracked `devtools/` operator code is excluded from this runtime health gate, but touched `devtools/` files are still fully reviewed.
|
||||
- Function parameters: <8.
|
||||
- Net complexity growth per cycle approaches zero.
|
||||
- If a feature is not used in the current cycle — it is premature.
|
||||
|
|
@ -382,12 +382,19 @@ Before every commit, verify the following:
|
|||
#### Task Contract Resource Policy
|
||||
- When a task contract declares `resource_policy.protected_artifacts`, enforce it as a typed affordance policy in every runtime mode: execute-only black-box references may be run, but byte reads, copy/hash/static introspection, tracing, and debugging against declared paths are blocked. Do not add benchmark-specific command gates.
|
||||
|
||||
#### Devtools And Benchmark Tooling
|
||||
- `devtools/` is tracked operator code, not runtime core. It may contain benchmark harness adapters, smoke runners, and reproducibility helpers that should be versioned with Ouroboros, but runtime modules under `ouroboros/`, `server.py`, web modules, and build scripts must not import it.
|
||||
- `devtools/` is not included in the Python runtime package discovery; it is repository-side operator tooling, not an installed dependency of the Ouroboros app.
|
||||
- `devtools/` is not an immune-system bypass. If a commit touches `devtools/`, triad/scope reviewers inspect those touched files fully. Unrelated `devtools/` files use the Atlas `excluded_dir` disposition and stay coverage-manifest-only in broad packs so benchmark harness code does not drown normal core reviews.
|
||||
- Benchmark adapters must preserve official task instructions, official scoring/evaluation commands, and official artifact formats. They may build predictions, launch official runners, normalize logs, or aggregate official outputs, but must not implement benchmark-specific prompt hacks, routing hacks, or replacement scoring.
|
||||
- Generated benchmark runs, datasets, container outputs, logs, predictions, and submissions belong under `/Users/anton/Ouroboros/bench_runs/` or another explicit output root outside `repo/`, never under `devtools/`.
|
||||
|
||||
#### Light Mode External Deliverables
|
||||
- `runtime_mode=light` is a self-modification boundary, not an OS sandbox. User-visible deliverables are allowed when they are outside the Ouroboros repo/control-plane.
|
||||
- Preferred flow: `task_drive` for scratch, `artifact_store` for canonical deliverables, and `user_files` for the owner's visible copy (for example `Desktop/report.html`). `write_file(root=user_files)` and declared process `outputs` must register/copy canonical task artifacts. Rewrites of the same user-visible source keep the previous canonical artifact in non-manifest history with last-5 retention; history is for recovery, not a second deliverable list.
|
||||
- `run_command`/`run_script`/`start_service` may use cwd under `active_workspace`, task-scoped `task_drive`, task-scoped `artifact_store`, and external `user_files` where the active profile permits it. In light direct tasks, omitted `run_script.cwd` defaults to task scratch instead of the Ouroboros repo; long-running services in light must use an explicit external/task/artifact cwd. Declared service `outputs` are copied into the task artifact store when the service stops.
|
||||
- In external workspace mode, light-mode self-repo dirty checks snapshot the system repo, not the active workspace. Workspace file/build artifacts belong to workspace patch/artifact finalization; workspace HEAD/ref changes are still blocked by the separate workspace git-ref guard.
|
||||
- `claude_code_edit` remains a first-class high-capability coding tool for substantial external artifacts; do not remove, hide, or downgrade it when refactoring Tool API names. It may run under external user/task/artifact cwd in direct light tasks, and under active workspace/task/artifact cwd in workspace tasks, while Ouroboros repo/control-plane cwd stays on the reviewed self-modification path. Use `outputs=[...]` when it creates deliverables that must be audited.
|
||||
- `claude_code_edit` remains a first-class high-capability coding tool for substantial external artifacts; do not remove, hide, or downgrade it when refactoring Tool API names. It may run under external user/task/artifact cwd in direct light tasks, and under active workspace/task/artifact cwd in workspace tasks, while Ouroboros repo/control-plane cwd stays on the reviewed self-modification path. In docker executor-backed external workspaces, mapped active workspace cwd is blocked until a reviewed backend-safe Claude Code path exists; unmapped task/artifact/user cwd remains available where the active profile permits it. Use `outputs=[...]` when it creates deliverables that must be audited.
|
||||
- Do not recommend `runtime_data/uploads`, skill payloads, or owner state directories as generic artifact transport.
|
||||
|
||||
#### Live Subagent Task Constraints
|
||||
|
|
|
|||
|
|
@ -227,6 +227,7 @@ class OuroborosAgent:
|
|||
"task_group_id",
|
||||
"task_group",
|
||||
"subagent_envelope",
|
||||
"executor_ref",
|
||||
):
|
||||
if task.get(key) not in (None, ""):
|
||||
task_metadata[key] = task.get(key)
|
||||
|
|
@ -244,6 +245,7 @@ class OuroborosAgent:
|
|||
memory_mode=str(task.get("memory_mode") or ""),
|
||||
budget_drive_root=str(task.get("budget_drive_root") or ""),
|
||||
task_metadata=task_metadata,
|
||||
executor_ref=task_metadata.get("executor_ref") if isinstance(task_metadata.get("executor_ref"), dict) else {},
|
||||
pending_events=self._pending_events,
|
||||
current_chat_id=self._current_chat_id,
|
||||
current_task_type=self._current_task_type,
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ import pathlib
|
|||
import re
|
||||
import subprocess
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from typing import Dict, List
|
||||
from typing import Dict, Iterable, List
|
||||
|
||||
from ouroboros.utils import atomic_write_json, utc_now_iso
|
||||
|
||||
|
|
@ -249,8 +249,35 @@ def _is_sensitive_inventory_path(rel_path: str) -> bool:
|
|||
return suffix in _SENSITIVE_EXTENSIONS and bool(_SENSITIVE_NAME_RE.search(name))
|
||||
|
||||
|
||||
def build_code_inventory(repo_root: pathlib.Path, *, drive_root: pathlib.Path | None = None, persist: bool = True) -> CodeInventory:
|
||||
def _is_excluded_inventory_path(path: pathlib.Path, excluded_paths: list[pathlib.Path]) -> bool:
|
||||
try:
|
||||
resolved = pathlib.Path(path).resolve(strict=False)
|
||||
except Exception:
|
||||
return False
|
||||
for excluded in excluded_paths:
|
||||
if resolved == excluded:
|
||||
return True
|
||||
try:
|
||||
if excluded.is_dir():
|
||||
resolved.relative_to(excluded)
|
||||
return True
|
||||
except Exception:
|
||||
continue
|
||||
return False
|
||||
|
||||
|
||||
def build_code_inventory(
|
||||
repo_root: pathlib.Path,
|
||||
*,
|
||||
drive_root: pathlib.Path | None = None,
|
||||
persist: bool = True,
|
||||
exclude_paths: Iterable[pathlib.Path] | None = None,
|
||||
) -> CodeInventory:
|
||||
root = pathlib.Path(repo_root).resolve(strict=False)
|
||||
excluded_paths = [
|
||||
pathlib.Path(path).expanduser().resolve(strict=False)
|
||||
for path in (exclude_paths or [])
|
||||
]
|
||||
files = []
|
||||
for path in _tracked_files(root):
|
||||
try:
|
||||
|
|
@ -259,6 +286,8 @@ def build_code_inventory(repo_root: pathlib.Path, *, drive_root: pathlib.Path |
|
|||
rel_parts = path.parts
|
||||
if any(part in _SKIP_DIRS for part in rel_parts):
|
||||
continue
|
||||
if _is_excluded_inventory_path(path, excluded_paths):
|
||||
continue
|
||||
if path.is_file():
|
||||
files.append(_file_fact(root, path))
|
||||
coverage: Dict[str, int] = {}
|
||||
|
|
|
|||
|
|
@ -9,9 +9,9 @@ from __future__ import annotations
|
|||
from typing import Any, Dict, Optional
|
||||
|
||||
try: # Python 3.11+
|
||||
from typing import Literal, NotRequired, TypedDict # type: ignore[attr-defined]
|
||||
from typing import Literal, NotRequired, Required, TypedDict # type: ignore[attr-defined]
|
||||
except ImportError: # pragma: no cover - CI supports Python 3.10.
|
||||
from typing_extensions import Literal, NotRequired, TypedDict # type: ignore[assignment]
|
||||
from typing_extensions import Literal, NotRequired, Required, TypedDict # type: ignore[assignment]
|
||||
|
||||
|
||||
class ChatInbound(TypedDict):
|
||||
|
|
@ -413,6 +413,46 @@ class ChatHistoryResponse(TypedDict, total=False):
|
|||
error: str
|
||||
|
||||
|
||||
class ExecutorRef(TypedDict, total=False):
|
||||
type: Required[Literal["local", "docker_exec"]]
|
||||
id: NotRequired[str]
|
||||
network: NotRequired[Literal["host", "none"]]
|
||||
workspace_host_path: NotRequired[str]
|
||||
workspace_backend_path: NotRequired[str]
|
||||
# Required at runtime when type == "docker_exec".
|
||||
container_name: NotRequired[str]
|
||||
path_mappings: NotRequired[list[Dict[str, str]]]
|
||||
|
||||
|
||||
class _TaskCreateRequestRequired(TypedDict):
|
||||
description: str
|
||||
|
||||
|
||||
class TaskCreateRequest(_TaskCreateRequestRequired, total=False):
|
||||
task_id: str
|
||||
type: str
|
||||
chat_id: int
|
||||
depth: int
|
||||
session_id: str
|
||||
workspace_root: str
|
||||
workspace_mode: str
|
||||
memory_mode: str
|
||||
attachments: list[Dict[str, Any]]
|
||||
allowed_resources: Dict[str, Any]
|
||||
resource_policy: Dict[str, Any]
|
||||
executor_ref: ExecutorRef
|
||||
deadline_at: str
|
||||
timeout_sec: float
|
||||
timeout: float
|
||||
context: str
|
||||
expected_output: str
|
||||
constraints: str
|
||||
context_requires_self_body_docs: bool
|
||||
actor_id: str
|
||||
source: str
|
||||
metadata: Dict[str, Any]
|
||||
|
||||
|
||||
class TaskCreateResponse(TypedDict, total=False):
|
||||
ok: bool
|
||||
task_id: str
|
||||
|
|
@ -587,6 +627,8 @@ __all__ = [
|
|||
"ModelCatalogResponse",
|
||||
"FileBrowserListResponse",
|
||||
"ChatHistoryResponse",
|
||||
"ExecutorRef",
|
||||
"TaskCreateRequest",
|
||||
"TaskCreateResponse",
|
||||
"TaskListResponse",
|
||||
"TaskEvent",
|
||||
|
|
|
|||
|
|
@ -40,13 +40,14 @@ from ouroboros.task_status import (
|
|||
find_child_tasks,
|
||||
load_effective_task_result,
|
||||
)
|
||||
from ouroboros.tool_access import paths_overlap_casefold
|
||||
from ouroboros.tool_access import path_is_relative_to, paths_overlap_casefold
|
||||
from ouroboros.utils import iter_jsonl_objects, utc_now_iso
|
||||
from ouroboros.workspace_preflight import (
|
||||
collect_workspace_preflight,
|
||||
render_workspace_preflight_summary,
|
||||
summarize_workspace_preflight,
|
||||
)
|
||||
from ouroboros.workspace_executor import normalize_executor_ref
|
||||
|
||||
|
||||
_LOG_SOURCES = (
|
||||
|
|
@ -72,6 +73,8 @@ _RESERVED_METADATA_KEYS = frozenset({
|
|||
"task_contract",
|
||||
"allowed_resources",
|
||||
"deadline_at",
|
||||
"executor_ref",
|
||||
"workspace_executor",
|
||||
})
|
||||
|
||||
|
||||
|
|
@ -102,7 +105,7 @@ async def api_tasks_create(request: Request) -> JSONResponse:
|
|||
body = await request_json_or(request, {})
|
||||
if not isinstance(body, dict):
|
||||
return json_error("request body must be a JSON object", 400)
|
||||
description = str(body.get("description") or body.get("text") or body.get("prompt") or "").strip()
|
||||
description = str(body.get("description") or "").strip()
|
||||
if not description:
|
||||
return json_error("description is required", 400)
|
||||
|
||||
|
|
@ -161,6 +164,37 @@ async def api_tasks_create(request: Request) -> JSONResponse:
|
|||
resource_policy = normalize_resource_policy(body.get("resource_policy") or raw_metadata.get("resource_policy") or {})
|
||||
if resource_policy:
|
||||
metadata["resource_policy"] = resource_policy
|
||||
if "executor_ref" in raw_metadata or "workspace_executor" in raw_metadata:
|
||||
return json_error("metadata.executor_ref/workspace_executor is reserved; pass executor_ref as a top-level task field", 400)
|
||||
if "executor_ref" in body:
|
||||
raw_executor_ref = body.get("executor_ref")
|
||||
if not isinstance(raw_executor_ref, dict) or not raw_executor_ref:
|
||||
return json_error("executor_ref must be a JSON object", 400)
|
||||
if workspace_root is None:
|
||||
return json_error("executor_ref requires an external workspace_root", 400)
|
||||
try:
|
||||
normalized_executor = normalize_executor_ref(raw_executor_ref)
|
||||
except ValueError as exc:
|
||||
return json_error(str(exc), 400)
|
||||
if normalized_executor is not None:
|
||||
for mapping in normalized_executor.mappings:
|
||||
for protected_root, label in ((repo_dir, "Ouroboros system repo"), (drive_root, "Ouroboros data drive")):
|
||||
if paths_overlap_casefold(mapping.host_path, protected_root):
|
||||
return json_error(f"executor_ref mapping must not overlap the {label}", 400)
|
||||
if not any(path_is_relative_to(workspace_root, mapping.host_path) for mapping in normalized_executor.mappings):
|
||||
return json_error("executor_ref mappings must cover workspace_root", 400)
|
||||
metadata["executor_ref"] = {
|
||||
"type": normalized_executor.kind,
|
||||
"id": normalized_executor.executor_id,
|
||||
"network": normalized_executor.network,
|
||||
"workspace_host_path": str(normalized_executor.mappings[0].host_path),
|
||||
"workspace_backend_path": normalized_executor.mappings[0].backend_path,
|
||||
"container_name": normalized_executor.container_name,
|
||||
"path_mappings": [
|
||||
{"host_path": str(mapping.host_path), "backend_path": mapping.backend_path}
|
||||
for mapping in normalized_executor.mappings
|
||||
],
|
||||
}
|
||||
try:
|
||||
deadline_at = _normalize_deadline_at(body.get("deadline_at") or raw_metadata.get("deadline_at") or "")
|
||||
except ValueError as exc:
|
||||
|
|
|
|||
|
|
@ -13,6 +13,8 @@ from ouroboros.shell_parse import (
|
|||
unwrap_env_argv,
|
||||
)
|
||||
from ouroboros.tool_access import resolve_shell_cwd
|
||||
from ouroboros.tools.shell_guards import writer_target_tokens
|
||||
from ouroboros.workspace_executor import executor_ref_from_ctx, map_backend_path, map_host_path
|
||||
|
||||
_DEFAULT_DENIED_OPERATIONS = frozenset({
|
||||
"read_bytes",
|
||||
|
|
@ -21,10 +23,13 @@ _DEFAULT_DENIED_OPERATIONS = frozenset({
|
|||
"static_introspection",
|
||||
"dynamic_trace",
|
||||
"debug",
|
||||
"write",
|
||||
"delete",
|
||||
})
|
||||
_SHELLS = frozenset({"bash", "sh", "zsh"})
|
||||
_SHELLS = frozenset({"bash", "cmd", "powershell", "pwsh", "sh", "zsh"})
|
||||
_HIGH_RISK_INTERPRETERS = frozenset({
|
||||
"bash", "sh", "zsh", "python", "python3", "node", "ruby", "perl", "php",
|
||||
"bash", "sh", "zsh", "python", "python3", "pythonw", "pypy", "pypy3",
|
||||
"node", "ruby", "perl", "php",
|
||||
})
|
||||
_SHELL_COMMAND_OPERATIONS = {
|
||||
"cat": "read_bytes",
|
||||
|
|
@ -32,6 +37,17 @@ _SHELL_COMMAND_OPERATIONS = {
|
|||
"tail": "read_bytes",
|
||||
"less": "read_bytes",
|
||||
"more": "read_bytes",
|
||||
"grep": "static_introspection",
|
||||
"egrep": "static_introspection",
|
||||
"fgrep": "static_introspection",
|
||||
"rg": "static_introspection",
|
||||
"ripgrep": "static_introspection",
|
||||
"ag": "static_introspection",
|
||||
"ack": "static_introspection",
|
||||
"sed": "static_introspection",
|
||||
"awk": "static_introspection",
|
||||
"diff": "static_introspection",
|
||||
"cmp": "static_introspection",
|
||||
"file": "static_introspection",
|
||||
"strings": "static_introspection",
|
||||
"hexdump": "static_introspection",
|
||||
|
|
@ -41,7 +57,36 @@ _SHELL_COMMAND_OPERATIONS = {
|
|||
"nm": "static_introspection",
|
||||
"otool": "static_introspection",
|
||||
"cp": "copy",
|
||||
"copy": "copy",
|
||||
"dd": "copy",
|
||||
"rsync": "copy",
|
||||
"tar": "copy",
|
||||
"zip": "copy",
|
||||
"type": "read_bytes",
|
||||
"xcopy": "copy",
|
||||
"robocopy": "copy",
|
||||
"certutil": "hash",
|
||||
"get-content": "read_bytes",
|
||||
"gc": "read_bytes",
|
||||
"select-string": "static_introspection",
|
||||
"copy-item": "copy",
|
||||
"get-filehash": "hash",
|
||||
"del": "delete",
|
||||
"erase": "delete",
|
||||
"mv": "delete",
|
||||
"move": "delete",
|
||||
"rd": "delete",
|
||||
"ren": "delete",
|
||||
"rename": "delete",
|
||||
"rename-item": "delete",
|
||||
"remove-item": "delete",
|
||||
"ri": "delete",
|
||||
"rm": "delete",
|
||||
"rmdir": "delete",
|
||||
"unlink": "delete",
|
||||
"shred": "delete",
|
||||
"tee": "write",
|
||||
"truncate": "write",
|
||||
"sha256sum": "hash",
|
||||
"shasum": "hash",
|
||||
"md5sum": "hash",
|
||||
|
|
@ -51,6 +96,31 @@ _SHELL_COMMAND_OPERATIONS = {
|
|||
"gdb": "debug",
|
||||
"lldb": "debug",
|
||||
}
|
||||
_CMD_INLINE_SWITCHES = frozenset({"/c", "/k"})
|
||||
_POWERSHELL_INLINE_SWITCHES = frozenset({"-c", "-command", "/c"})
|
||||
_POWERSHELL_ENCODED_SWITCHES = frozenset({"-encodedcommand", "-enc", "-e"})
|
||||
_GIT_STATIC_INTROSPECTION_SUBCOMMANDS = frozenset({
|
||||
"blame",
|
||||
"annotate",
|
||||
"cat-file",
|
||||
"diff",
|
||||
"grep",
|
||||
"show",
|
||||
})
|
||||
_GIT_PATCH_LOG_FLAGS = frozenset({"-p", "-u", "--patch", "--patch-with-stat", "--stat-with-summary"})
|
||||
_GIT_GLOBAL_OPTIONS_WITH_VALUE = frozenset({
|
||||
"-C",
|
||||
"-c",
|
||||
"--config-env",
|
||||
"--exec-path",
|
||||
"--git-dir",
|
||||
"--namespace",
|
||||
"--super-prefix",
|
||||
"--work-tree",
|
||||
})
|
||||
_DIRECTORY_TARGET_OPERATIONS = frozenset({"copy", "delete", "read_bytes", "static_introspection", "write"})
|
||||
_SHELL_GLOB_CHARS = frozenset("*?[")
|
||||
_FIND_EXPRESSION_MARKERS = frozenset({"!", "(", ")"})
|
||||
|
||||
|
||||
def _task_contract(ctx: Any) -> Dict[str, Any]:
|
||||
|
|
@ -104,10 +174,58 @@ def _resolve_policy_path(ctx: Any, raw_path: str) -> pathlib.Path | None:
|
|||
return (roots[0] / path).resolve(strict=False)
|
||||
|
||||
|
||||
def _backend_spellings_for_host_path(ctx: Any, path: pathlib.Path) -> set[str]:
|
||||
try:
|
||||
executor = executor_ref_from_ctx(ctx)
|
||||
if executor is None:
|
||||
return set()
|
||||
backend = map_host_path(executor, pathlib.Path(path))
|
||||
except Exception:
|
||||
return set()
|
||||
return {backend, backend.rstrip("/")}
|
||||
|
||||
|
||||
def _policy_backend_spellings(ctx: Any, raw_path: str, resolved: pathlib.Path | None) -> set[str]:
|
||||
spellings: set[str] = set()
|
||||
text = str(raw_path or "").strip()
|
||||
if text:
|
||||
spellings.add(slash_normalize_path_text(text).rstrip("/"))
|
||||
if resolved is not None:
|
||||
spellings.update(_backend_spellings_for_host_path(ctx, resolved))
|
||||
return {item for item in spellings if item}
|
||||
|
||||
|
||||
def _backend_cwd_relative_spellings(ctx: Any, work_dir: pathlib.Path, spellings: set[str]) -> set[str]:
|
||||
try:
|
||||
executor = executor_ref_from_ctx(ctx)
|
||||
if executor is None:
|
||||
return set()
|
||||
backend_cwd = map_host_path(executor, pathlib.Path(work_dir)).rstrip("/")
|
||||
except Exception:
|
||||
return set()
|
||||
relative: set[str] = set()
|
||||
for spelling in spellings:
|
||||
normalized = slash_normalize_path_text(spelling).rstrip("/")
|
||||
if normalized.startswith(backend_cwd + "/"):
|
||||
rel = normalized[len(backend_cwd) + 1:]
|
||||
if rel:
|
||||
relative.add(rel)
|
||||
return relative
|
||||
|
||||
|
||||
def protected_artifact_paths(ctx: Any) -> List[pathlib.Path]:
|
||||
paths: List[pathlib.Path] = []
|
||||
for record in _artifact_records(ctx):
|
||||
for raw_path in record.get("paths") or []:
|
||||
text = str(raw_path)
|
||||
try:
|
||||
executor_ref = executor_ref_from_ctx(ctx)
|
||||
if executor_ref is not None and text.strip().startswith("/"):
|
||||
mapped = map_backend_path(executor_ref, text)
|
||||
if mapped not in paths:
|
||||
paths.append(mapped)
|
||||
except Exception:
|
||||
pass
|
||||
resolved = _resolve_policy_path(ctx, str(raw_path))
|
||||
if resolved is not None and resolved not in paths:
|
||||
paths.append(resolved)
|
||||
|
|
@ -116,8 +234,8 @@ def protected_artifact_paths(ctx: Any) -> List[pathlib.Path]:
|
|||
|
||||
def _operation_denied(record: Dict[str, Any], operation: str) -> bool:
|
||||
allow = {str(item).strip() for item in (record.get("allow") or []) if str(item).strip()}
|
||||
if operation in allow:
|
||||
return False
|
||||
if allow:
|
||||
return operation not in allow
|
||||
deny = {str(item).strip() for item in (record.get("deny") or []) if str(item).strip()}
|
||||
if deny:
|
||||
return operation in deny
|
||||
|
|
@ -141,13 +259,43 @@ def _matches(candidate: pathlib.Path, protected_path: pathlib.Path) -> bool:
|
|||
return False
|
||||
|
||||
|
||||
def _backend_spelling_matches(candidate: pathlib.Path, protected_spellings: set[str]) -> bool:
|
||||
try:
|
||||
raw = str(candidate)
|
||||
except Exception:
|
||||
return False
|
||||
normalized = slash_normalize_path_text(raw).rstrip("/")
|
||||
if not normalized:
|
||||
return False
|
||||
variants = {normalized}
|
||||
if not normalized.startswith("/"):
|
||||
variants.add(f"/{normalized}")
|
||||
for protected in protected_spellings:
|
||||
if not protected:
|
||||
continue
|
||||
protected_norm = slash_normalize_path_text(protected).rstrip("/")
|
||||
protected_variants = {protected_norm}
|
||||
if not protected_norm.startswith("/"):
|
||||
protected_variants.add(f"/{protected_norm}")
|
||||
if variants & protected_variants:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def block_reason_for_path(ctx: Any, target: pathlib.Path, operation: str) -> str:
|
||||
for record in _artifact_records(ctx):
|
||||
if not _operation_denied(record, operation):
|
||||
continue
|
||||
for raw_path in record.get("paths") or []:
|
||||
protected_path = _resolve_policy_path(ctx, str(raw_path))
|
||||
if protected_path is not None and _matches(pathlib.Path(target), protected_path):
|
||||
protected_spellings = _policy_backend_spellings(ctx, str(raw_path), protected_path)
|
||||
target_backend_spellings = _backend_spellings_for_host_path(ctx, pathlib.Path(target))
|
||||
if (
|
||||
protected_path is not None
|
||||
and _matches(pathlib.Path(target), protected_path)
|
||||
or _backend_spelling_matches(pathlib.Path(target), protected_spellings)
|
||||
or bool(protected_spellings & target_backend_spellings)
|
||||
):
|
||||
artifact_id = str(record.get("id") or pathlib.Path(str(raw_path)).name or "protected artifact")
|
||||
return (
|
||||
"⚠️ RESOURCE_POLICY_BLOCKED: task_contract.resource_policy protects "
|
||||
|
|
@ -164,6 +312,232 @@ def any_protected_target(ctx: Any, candidates: Iterable[pathlib.Path], operation
|
|||
return ""
|
||||
|
||||
|
||||
def _directory_contains_protected_target(ctx: Any, candidates: Iterable[pathlib.Path], operation: str) -> str:
|
||||
for candidate in candidates:
|
||||
try:
|
||||
candidate_resolved = pathlib.Path(candidate).expanduser().resolve(strict=False)
|
||||
except (OSError, TypeError, ValueError):
|
||||
continue
|
||||
if not candidate_resolved.is_dir():
|
||||
continue
|
||||
for record in _artifact_records(ctx):
|
||||
if not _operation_denied(record, operation):
|
||||
continue
|
||||
for raw_path in record.get("paths") or []:
|
||||
protected_paths: list[pathlib.Path] = []
|
||||
protected_path = _resolve_policy_path(ctx, str(raw_path))
|
||||
if protected_path is not None:
|
||||
protected_paths.append(pathlib.Path(protected_path))
|
||||
try:
|
||||
executor_ref = executor_ref_from_ctx(ctx)
|
||||
if executor_ref is not None and str(raw_path).strip().startswith("/"):
|
||||
protected_paths.append(map_backend_path(executor_ref, str(raw_path)))
|
||||
except Exception:
|
||||
pass
|
||||
for candidate_protected in protected_paths:
|
||||
try:
|
||||
candidate_protected.resolve(strict=False).relative_to(candidate_resolved)
|
||||
except ValueError:
|
||||
continue
|
||||
except Exception:
|
||||
continue
|
||||
return block_reason_for_path(ctx, candidate_protected, operation)
|
||||
return ""
|
||||
|
||||
|
||||
def _resolve_candidate_path(ctx: Any, work_dir: pathlib.Path, text: str) -> pathlib.Path | None:
|
||||
try:
|
||||
path = pathlib.Path(text).expanduser()
|
||||
if path.is_absolute():
|
||||
try:
|
||||
executor_ref = executor_ref_from_ctx(ctx)
|
||||
return map_backend_path(executor_ref, text) if executor_ref is not None else path.resolve(strict=False)
|
||||
except Exception:
|
||||
return path.resolve(strict=False)
|
||||
return (pathlib.Path(work_dir) / path).resolve(strict=False)
|
||||
except (OSError, TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _contains_shell_glob(text: str) -> bool:
|
||||
return any(char in str(text or "") for char in _SHELL_GLOB_CHARS)
|
||||
|
||||
|
||||
def _glob_base_candidate(ctx: Any, work_dir: pathlib.Path, text: str) -> pathlib.Path | None:
|
||||
normalized = str(text or "").replace("\\", "/")
|
||||
first_glob = min((idx for idx, char in enumerate(normalized) if char in _SHELL_GLOB_CHARS), default=-1)
|
||||
if first_glob < 0:
|
||||
return None
|
||||
prefix = normalized[:first_glob]
|
||||
if "/" in prefix:
|
||||
base_text = prefix.rsplit("/", 1)[0] or "/"
|
||||
else:
|
||||
base_text = "."
|
||||
return _resolve_candidate_path(ctx, work_dir, base_text)
|
||||
|
||||
|
||||
def _inline_shell_command(argv: list[str], shell_name: str) -> str:
|
||||
if shell_name in {"bash", "sh", "zsh"}:
|
||||
return shell_command_string(argv)
|
||||
switches = _CMD_INLINE_SWITCHES if shell_name == "cmd" else _POWERSHELL_INLINE_SWITCHES
|
||||
for idx, arg in enumerate(argv[1:], start=1):
|
||||
if str(arg or "").strip().lower() in switches:
|
||||
return " ".join(str(part) for part in argv[idx + 1:])
|
||||
return ""
|
||||
|
||||
|
||||
def _uses_powershell_encoded_command(argv: list[str], shell_name: str) -> bool:
|
||||
if shell_name not in {"powershell", "pwsh"}:
|
||||
return False
|
||||
return any(str(arg or "").strip().lower() in _POWERSHELL_ENCODED_SWITCHES for arg in argv[1:])
|
||||
|
||||
|
||||
def _looks_like_versioned_python_interpreter(name: str) -> bool:
|
||||
for prefix in ("python", "pypy"):
|
||||
suffix = name.removeprefix(prefix)
|
||||
if suffix == name or not suffix:
|
||||
continue
|
||||
suffix = suffix.removesuffix("m")
|
||||
parts = suffix.split(".")
|
||||
if parts and all(part.isdigit() for part in parts):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _is_high_risk_interpreter(name: str) -> bool:
|
||||
return name in _HIGH_RISK_INTERPRETERS or _looks_like_versioned_python_interpreter(name)
|
||||
|
||||
|
||||
def _git_subcommand_index(argv: list[str]) -> int | None:
|
||||
idx = 1
|
||||
while idx < len(argv):
|
||||
token = str(argv[idx] or "")
|
||||
if token == "--":
|
||||
idx += 1
|
||||
continue
|
||||
if token == "-C" or token in _GIT_GLOBAL_OPTIONS_WITH_VALUE:
|
||||
idx += 2
|
||||
continue
|
||||
if any(token.startswith(option + "=") for option in _GIT_GLOBAL_OPTIONS_WITH_VALUE):
|
||||
idx += 1
|
||||
continue
|
||||
if token.startswith("-"):
|
||||
idx += 1
|
||||
continue
|
||||
return idx
|
||||
return None
|
||||
|
||||
|
||||
def _git_static_introspection_operation(argv: list[str]) -> str | None:
|
||||
subcmd_idx = _git_subcommand_index(argv)
|
||||
if subcmd_idx is None:
|
||||
return None
|
||||
subcmd = pathlib.PurePath(argv[subcmd_idx]).name.lower().removesuffix(".exe")
|
||||
if subcmd in _GIT_STATIC_INTROSPECTION_SUBCOMMANDS:
|
||||
return "static_introspection"
|
||||
if subcmd == "log" and any(str(token or "") in _GIT_PATCH_LOG_FLAGS for token in argv[subcmd_idx + 1:]):
|
||||
return "static_introspection"
|
||||
return None
|
||||
|
||||
|
||||
def _find_operation(argv: list[str]) -> str:
|
||||
args = [str(token or "") for token in argv[1:]]
|
||||
if "-delete" in args:
|
||||
return "delete"
|
||||
for idx, token in enumerate(args):
|
||||
if token not in {"-exec", "-execdir"} or idx + 1 >= len(args):
|
||||
continue
|
||||
executable = pathlib.PurePath(args[idx + 1]).name.lower().removesuffix(".exe")
|
||||
operation = _SHELL_COMMAND_OPERATIONS.get(executable)
|
||||
if operation:
|
||||
return operation
|
||||
if _is_high_risk_interpreter(executable):
|
||||
return "read_bytes"
|
||||
return "static_introspection"
|
||||
return "static_introspection"
|
||||
|
||||
|
||||
def _find_has_explicit_start_path(argv: list[str]) -> bool:
|
||||
for token in (str(item or "") for item in argv[1:]):
|
||||
if not token:
|
||||
continue
|
||||
if token == "--":
|
||||
continue
|
||||
if token in _FIND_EXPRESSION_MARKERS or token.startswith("-"):
|
||||
return False
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _git_work_dir(ctx: Any, argv: list[str], initial_work_dir: pathlib.Path) -> pathlib.Path:
|
||||
work_dir = pathlib.Path(initial_work_dir)
|
||||
idx = 1
|
||||
while idx < len(argv):
|
||||
token = str(argv[idx] or "")
|
||||
if token == "--":
|
||||
idx += 1
|
||||
continue
|
||||
if token == "-C" and idx + 1 < len(argv):
|
||||
resolved = _resolve_candidate_path(ctx, work_dir, str(argv[idx + 1] or ""))
|
||||
if resolved is not None:
|
||||
work_dir = resolved
|
||||
idx += 2
|
||||
continue
|
||||
if token.startswith("-C") and len(token) > 2:
|
||||
resolved = _resolve_candidate_path(ctx, work_dir, token[2:])
|
||||
if resolved is not None:
|
||||
work_dir = resolved
|
||||
idx += 1
|
||||
continue
|
||||
if token in _GIT_GLOBAL_OPTIONS_WITH_VALUE:
|
||||
idx += 2
|
||||
continue
|
||||
if any(token.startswith(option + "=") for option in _GIT_GLOBAL_OPTIONS_WITH_VALUE):
|
||||
idx += 1
|
||||
continue
|
||||
if token.startswith("-"):
|
||||
idx += 1
|
||||
continue
|
||||
break
|
||||
return work_dir
|
||||
|
||||
|
||||
def _git_candidate_tokens(argv: list[str]) -> list[str]:
|
||||
subcmd_idx = _git_subcommand_index(argv)
|
||||
if subcmd_idx is None:
|
||||
return []
|
||||
tokens: list[str] = []
|
||||
rest = argv[subcmd_idx + 1:]
|
||||
for token in rest:
|
||||
text = str(token or "")
|
||||
if not text or text == "--":
|
||||
continue
|
||||
if text.startswith("-") and not pathlib.Path(text).is_absolute():
|
||||
continue
|
||||
tokens.append(text)
|
||||
if ":" not in text:
|
||||
continue
|
||||
# Git object syntax such as HEAD:path/to/file or :path/to/file can
|
||||
# read the protected bytes without naming a filesystem path directly.
|
||||
if len(text) >= 2 and text[1] == ":" and text[0].isalpha():
|
||||
continue
|
||||
rev_path = text.split(":", 1)[1].lstrip("./")
|
||||
if rev_path:
|
||||
tokens.append(rev_path)
|
||||
return tokens
|
||||
|
||||
|
||||
def _git_static_introspection_is_path_limited(work_dir: pathlib.Path, candidates: list[pathlib.Path]) -> bool:
|
||||
for candidate in candidates:
|
||||
try:
|
||||
resolved = pathlib.Path(candidate).resolve(strict=False)
|
||||
except Exception:
|
||||
continue
|
||||
if resolved.exists():
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def shell_block_reason(ctx: Any, raw_cmd: Any, *, cwd: str = "", default_cwd: pathlib.Path | None = None) -> str:
|
||||
protected_paths = protected_artifact_paths(ctx)
|
||||
if not protected_paths:
|
||||
|
|
@ -178,12 +552,24 @@ def shell_block_reason(ctx: Any, raw_cmd: Any, *, cwd: str = "", default_cwd: pa
|
|||
if not argv:
|
||||
return ""
|
||||
first = pathlib.PurePath(argv[0]).name.lower().removesuffix(".exe")
|
||||
if _uses_powershell_encoded_command(argv, first):
|
||||
return (
|
||||
"⚠️ RESOURCE_POLICY_BLOCKED: task_contract.resource_policy protects "
|
||||
"black-box artifacts; PowerShell EncodedCommand is not allowed while "
|
||||
"protected artifacts are declared."
|
||||
)
|
||||
if first in _SHELLS:
|
||||
inline = shell_command_string(argv)
|
||||
inline = _inline_shell_command(argv, first)
|
||||
if inline:
|
||||
return shell_block_reason(ctx, inline, cwd=cwd, default_cwd=default_cwd)
|
||||
operation = _SHELL_COMMAND_OPERATIONS.get(first)
|
||||
high_risk = first in _HIGH_RISK_INTERPRETERS
|
||||
operation = (
|
||||
_git_static_introspection_operation(argv)
|
||||
if first == "git"
|
||||
else _find_operation(argv)
|
||||
if first == "find"
|
||||
else _SHELL_COMMAND_OPERATIONS.get(first)
|
||||
)
|
||||
high_risk = _is_high_risk_interpreter(first)
|
||||
try:
|
||||
work_dir, _cwd_root, _allowed = resolve_shell_cwd(ctx, cwd)
|
||||
except Exception:
|
||||
|
|
@ -197,8 +583,17 @@ def shell_block_reason(ctx: Any, raw_cmd: Any, *, cwd: str = "", default_cwd: pa
|
|||
for protected in protected_paths:
|
||||
if first_target == pathlib.Path(protected).resolve(strict=False):
|
||||
return block_reason_for_path(ctx, first_target, "execute")
|
||||
if first == "git":
|
||||
work_dir = _git_work_dir(ctx, argv, pathlib.Path(work_dir))
|
||||
candidate_tokens = [*env_values, *_git_candidate_tokens(argv)]
|
||||
else:
|
||||
candidate_tokens = [*env_values, *argv[1:]]
|
||||
if first == "find" and not _find_has_explicit_start_path(argv):
|
||||
candidate_tokens.append(".")
|
||||
candidates: list[pathlib.Path] = []
|
||||
for raw in [*env_values, *argv[1:]]:
|
||||
write_target_texts = list(writer_target_tokens(argv))
|
||||
candidate_tokens.extend(write_target_texts)
|
||||
for raw in candidate_tokens:
|
||||
text = str(raw or "")
|
||||
if not text or text in {"|", "&&", "||", ";"}:
|
||||
continue
|
||||
|
|
@ -208,13 +603,30 @@ def shell_block_reason(ctx: Any, raw_cmd: Any, *, cwd: str = "", default_cwd: pa
|
|||
continue
|
||||
if text.startswith("-") and not pathlib.Path(text).is_absolute():
|
||||
continue
|
||||
try:
|
||||
path = pathlib.Path(text).expanduser()
|
||||
candidates.append(path.resolve(strict=False) if path.is_absolute() else (pathlib.Path(work_dir) / path).resolve(strict=False))
|
||||
except (OSError, ValueError):
|
||||
if _contains_shell_glob(text):
|
||||
glob_base = _glob_base_candidate(ctx, pathlib.Path(work_dir), text)
|
||||
if glob_base is not None:
|
||||
candidates.append(glob_base)
|
||||
continue
|
||||
candidate = _resolve_candidate_path(ctx, pathlib.Path(work_dir), text)
|
||||
if candidate is not None:
|
||||
candidates.append(candidate)
|
||||
if first == "git" and operation == "static_introspection" and not _git_static_introspection_is_path_limited(pathlib.Path(work_dir), candidates):
|
||||
candidates.append(pathlib.Path(work_dir).resolve(strict=False))
|
||||
if write_target_texts:
|
||||
write_block = any_protected_target(ctx, candidates, "write")
|
||||
if write_block:
|
||||
return write_block
|
||||
write_dir_block = _directory_contains_protected_target(ctx, candidates, "write")
|
||||
if write_dir_block:
|
||||
return write_dir_block
|
||||
if operation:
|
||||
return any_protected_target(ctx, candidates, operation)
|
||||
direct_block = any_protected_target(ctx, candidates, operation)
|
||||
if direct_block:
|
||||
return direct_block
|
||||
if operation in _DIRECTORY_TARGET_OPERATIONS:
|
||||
return _directory_contains_protected_target(ctx, candidates, operation)
|
||||
return ""
|
||||
if not high_risk:
|
||||
return ""
|
||||
default_block = any_protected_target(ctx, candidates, "read_bytes")
|
||||
|
|
@ -222,9 +634,17 @@ def shell_block_reason(ctx: Any, raw_cmd: Any, *, cwd: str = "", default_cwd: pa
|
|||
return default_block
|
||||
tail_text = " ".join(str(part or "") for part in [*env_values, *argv[1:]])
|
||||
tail_text_posix = slash_normalize_path_text(tail_text)
|
||||
records = _artifact_records(ctx)
|
||||
for protected in protected_paths:
|
||||
protected = pathlib.Path(protected).resolve(strict=False)
|
||||
needles = {str(protected), protected.as_posix(), slash_normalize_path_text(protected)}
|
||||
for record in records:
|
||||
for raw_path in record.get("paths") or []:
|
||||
protected_path = _resolve_policy_path(ctx, str(raw_path))
|
||||
if protected_path is not None and _matches(protected, protected_path):
|
||||
backend_spellings = _policy_backend_spellings(ctx, str(raw_path), protected_path)
|
||||
needles.update(backend_spellings)
|
||||
needles.update(_backend_cwd_relative_spellings(ctx, pathlib.Path(work_dir), backend_spellings))
|
||||
try:
|
||||
rel = protected.relative_to(pathlib.Path(work_dir).resolve(strict=False))
|
||||
if str(rel) not in {"", "."}:
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@ _HEALTH_SKIP_DIR_PREFIXES = (
|
|||
".mypy_cache/",
|
||||
"node_modules/",
|
||||
".venv/",
|
||||
"devtools/",
|
||||
"tests/",
|
||||
)
|
||||
TARGET_MODULE_LINES = 1000
|
||||
|
|
@ -91,7 +92,14 @@ MAX_FUNCTION_LINES = 300
|
|||
# verified smoke result; pay it down after the recovery release lands.
|
||||
# Recovery completion restores strict scope-review contract parsing after the
|
||||
# atlas/context-limit incident; keep the cap at the verified smoke result.
|
||||
MAX_TOTAL_FUNCTIONS = 2768
|
||||
# v6.19.0-rc.1 adds the workspace executor bridge as a first-class runtime
|
||||
# boundary for container-backed benchmark workspaces, including service
|
||||
# process-group lifecycle helpers, shared devtools output-root guards, and
|
||||
# strict cleanup/protected-artifact review follow-ups. Review cycle 2 added
|
||||
# durable executor panic-cleanup/backend-path helpers. Keep deliberate headroom
|
||||
# for small safety/review helpers so minor fixes do not churn this gate; pay down
|
||||
# with a focused simplification pass after the prerelease lands.
|
||||
MAX_TOTAL_FUNCTIONS = 3000
|
||||
# Grandfathered modules are accepted debt until their surfaces stabilize/split.
|
||||
GRANDFATHERED_OVERSIZED_MODULES = {
|
||||
"llm.py",
|
||||
|
|
|
|||
|
|
@ -83,10 +83,17 @@ def execute_panic_stop(
|
|||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
from ouroboros.workspace_executor import kill_all_foreground
|
||||
|
||||
kill_all_foreground(data_dir, wait=False)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
from ouroboros.tools.services import kill_all_services
|
||||
|
||||
kill_all_services(wait=False)
|
||||
kill_all_services(data_dir, wait=False)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
|
|
|||
|
|
@ -236,6 +236,15 @@ def _schedule_task(
|
|||
or metadata.get("allowed_resources")
|
||||
or {}
|
||||
)
|
||||
executor_ref = {}
|
||||
executor_accessor = getattr(ctx, "workspace_executor_ref", None)
|
||||
if callable(executor_accessor):
|
||||
try:
|
||||
candidate = executor_accessor()
|
||||
if isinstance(candidate, dict) and candidate:
|
||||
executor_ref = dict(candidate)
|
||||
except Exception:
|
||||
executor_ref = {}
|
||||
lane_slots = expand_subagent_lane_slots(requested_model_lane, depth=new_depth)
|
||||
if not lane_slots:
|
||||
return "⚠️ SUBTASK_STATUS_ERROR: no subagent lane slots resolved; subagent was not scheduled."
|
||||
|
|
@ -350,6 +359,9 @@ def _schedule_task(
|
|||
evt["workspace_root"] = workspace_root
|
||||
if workspace_mode:
|
||||
evt["workspace_mode"] = workspace_mode
|
||||
if executor_ref:
|
||||
evt["executor_ref"] = executor_ref
|
||||
evt["metadata"] = {**(evt.get("metadata") if isinstance(evt.get("metadata"), dict) else {}), "executor_ref": executor_ref}
|
||||
if context:
|
||||
evt["context"] = context
|
||||
if parent_task_id:
|
||||
|
|
@ -372,6 +384,7 @@ def _schedule_task(
|
|||
context=context,
|
||||
workspace_root=workspace_root,
|
||||
workspace_mode=workspace_mode,
|
||||
executor_ref=executor_ref,
|
||||
allowed_resources=allowed_resources,
|
||||
task_contract=child_contract,
|
||||
chat_id=current_chat_id or None,
|
||||
|
|
|
|||
|
|
@ -136,6 +136,17 @@ def is_skill_control_plane_path(target: pathlib.Path, data_root: pathlib.Path) -
|
|||
return _policy_is_skill_control_plane_path(target, data_root)
|
||||
|
||||
|
||||
def _is_workspace_executor_control_state_path(target: pathlib.Path, data_root: pathlib.Path) -> bool:
|
||||
try:
|
||||
rel_parts = pathlib.Path(target).resolve(strict=False).relative_to(
|
||||
pathlib.Path(data_root).resolve(strict=False)
|
||||
).parts
|
||||
except (OSError, ValueError):
|
||||
return False
|
||||
lowered = [str(part).casefold() for part in rel_parts]
|
||||
return "state" in lowered and "workspace_executor_processes" in lowered
|
||||
|
||||
|
||||
def _list_dir(root: pathlib.Path, rel: str, max_entries: int = 500) -> List[str]:
|
||||
target = (root / safe_relpath(rel)).resolve()
|
||||
if not target.exists():
|
||||
|
|
@ -557,6 +568,7 @@ def _data_write(
|
|||
target_path = pathlib.Path(p)
|
||||
settings_path = pathlib.Path(_cfg.SETTINGS_PATH)
|
||||
data_root = pathlib.Path(_cfg.DATA_DIR).resolve(strict=False)
|
||||
ctx_data_root = pathlib.Path(ctx.drive_root).resolve(strict=False)
|
||||
if task_constraint and task_constraint.mode == "skill_repair" and task_constraint.payload_root:
|
||||
lexical_target = pathlib.Path(p).resolve(strict=False)
|
||||
else:
|
||||
|
|
@ -596,6 +608,17 @@ def _data_write(
|
|||
"SKILL.openclaw.md, .seed-origin) are owner/review controlled. "
|
||||
"Edit the payload's user-authored files instead and rerun skill_review."
|
||||
)
|
||||
if (
|
||||
_is_workspace_executor_control_state_path(lexical_target, ctx_data_root)
|
||||
or _is_workspace_executor_control_state_path(target_path, ctx_data_root)
|
||||
or _is_workspace_executor_control_state_path(lexical_target, data_root)
|
||||
or _is_workspace_executor_control_state_path(target_path, data_root)
|
||||
):
|
||||
return (
|
||||
"⚠️ DATA_WRITE_BLOCKED: workspace executor process records are "
|
||||
"owner/runtime control-plane state. Use process/service lifecycle "
|
||||
"tools instead of writing state/workspace_executor_processes directly."
|
||||
)
|
||||
matches = False
|
||||
try:
|
||||
if target_path.exists() and settings_path.exists():
|
||||
|
|
@ -721,6 +744,46 @@ def _join_write_results(results: List[str]) -> str:
|
|||
return rendered
|
||||
|
||||
|
||||
def _protected_artifact_write_block(
|
||||
ctx: ToolContext,
|
||||
root: str,
|
||||
paths: List[str],
|
||||
*,
|
||||
bucket: str = "",
|
||||
skill_name: str = "",
|
||||
prefix: str,
|
||||
) -> str:
|
||||
for rel_path in paths:
|
||||
if not str(rel_path or "").strip():
|
||||
continue
|
||||
try:
|
||||
target = resolve_resource_path(ctx, root=root, path=str(rel_path), bucket=bucket, skill_name=skill_name)
|
||||
except Exception:
|
||||
continue
|
||||
block_reason = block_reason_for_path(ctx, target, "write")
|
||||
if block_reason:
|
||||
return f"⚠️ {prefix}: protected artifact path blocked: {block_reason}"
|
||||
return ""
|
||||
|
||||
|
||||
def _protected_artifact_list_block(
|
||||
ctx: ToolContext,
|
||||
root: str,
|
||||
path: str,
|
||||
*,
|
||||
bucket: str = "",
|
||||
skill_name: str = "",
|
||||
) -> str:
|
||||
try:
|
||||
target = resolve_resource_path(ctx, root=root, path=path, bucket=bucket, skill_name=skill_name)
|
||||
except Exception:
|
||||
return ""
|
||||
direct_block = block_reason_for_path(ctx, target, "static_introspection")
|
||||
if direct_block:
|
||||
return direct_block
|
||||
return ""
|
||||
|
||||
|
||||
def _read_file(
|
||||
ctx: ToolContext,
|
||||
path: str,
|
||||
|
|
@ -798,6 +861,9 @@ def _list_files(
|
|||
normalized, block = _access_or_block(ctx, root, "list")
|
||||
if block:
|
||||
return block
|
||||
protected_list_block = _protected_artifact_list_block(ctx, normalized, path, bucket=bucket, skill_name=skill_name)
|
||||
if protected_list_block:
|
||||
return protected_list_block
|
||||
if normalized == "active_workspace":
|
||||
return _repo_list(ctx, dir=path, max_entries=max_entries)
|
||||
if normalized == "runtime_data":
|
||||
|
|
@ -846,6 +912,20 @@ def _write_file(
|
|||
return "⚠️ WRITE_FILE_BLOCKED: root=system_repo writes require the active workspace to be the system repo."
|
||||
except Exception as exc:
|
||||
return f"⚠️ WRITE_FILE_BLOCKED: could not validate system_repo root: {type(exc).__name__}: {exc}"
|
||||
write_paths = [path]
|
||||
for item in files or []:
|
||||
if isinstance(item, dict):
|
||||
write_paths.append(str(item.get("path") or ""))
|
||||
protected_block = _protected_artifact_write_block(
|
||||
ctx,
|
||||
normalized,
|
||||
write_paths,
|
||||
bucket=bucket,
|
||||
skill_name=skill_name,
|
||||
prefix="WRITE_FILE_BLOCKED",
|
||||
)
|
||||
if protected_block:
|
||||
return protected_block
|
||||
if normalized in {"active_workspace", "system_repo"}:
|
||||
from ouroboros.tools.git import _repo_write
|
||||
|
||||
|
|
@ -947,6 +1027,16 @@ def _edit_text(
|
|||
return "⚠️ EDIT_TEXT_BLOCKED: root=system_repo edits require the active workspace to be the system repo."
|
||||
except Exception as exc:
|
||||
return f"⚠️ EDIT_TEXT_BLOCKED: could not validate system_repo root: {type(exc).__name__}: {exc}"
|
||||
protected_block = _protected_artifact_write_block(
|
||||
ctx,
|
||||
normalized,
|
||||
[path],
|
||||
bucket=bucket,
|
||||
skill_name=skill_name,
|
||||
prefix="EDIT_TEXT_BLOCKED",
|
||||
)
|
||||
if protected_block:
|
||||
return protected_block
|
||||
if normalized in {"active_workspace", "system_repo"}:
|
||||
from ouroboros.tools.git import _str_replace_editor
|
||||
|
||||
|
|
@ -975,6 +1065,14 @@ def _edit_text(
|
|||
)
|
||||
try:
|
||||
target = resolve_resource_path(ctx, root=normalized, path=path, bucket=bucket, skill_name=skill_name)
|
||||
if normalized == "runtime_data":
|
||||
data_root = pathlib.Path(ctx.drive_root).resolve(strict=False)
|
||||
if _is_workspace_executor_control_state_path(target, data_root):
|
||||
return (
|
||||
"⚠️ EDIT_TEXT_BLOCKED: workspace executor process records are "
|
||||
"owner/runtime control-plane state. Use process/service lifecycle "
|
||||
"tools instead of editing state/workspace_executor_processes directly."
|
||||
)
|
||||
if normalized == "artifact_store":
|
||||
block_reason = artifact_store_path_block_reason(target)
|
||||
if block_reason:
|
||||
|
|
@ -1161,6 +1259,12 @@ def _code_search(ctx: ToolContext, query: str, path: str = ".",
|
|||
return f"⚠️ SEARCH_ERROR: {type(exc).__name__}: {exc}"
|
||||
if not search_root.exists():
|
||||
return f"⚠️ SEARCH_ERROR: path not found: {display_search_path}"
|
||||
protected_root_block = block_reason_for_path(ctx, search_root, "static_introspection")
|
||||
if protected_root_block:
|
||||
return protected_root_block
|
||||
protected_root_read_block = block_reason_for_path(ctx, search_root, "read_bytes")
|
||||
if protected_root_read_block and search_root.is_file():
|
||||
return protected_root_read_block
|
||||
subagent_readonly = _is_local_readonly_subagent(ctx)
|
||||
if subagent_readonly:
|
||||
block_msg = _local_readonly_resource_block(ctx, normalized, search_root, root_path, action="SEARCH")
|
||||
|
|
@ -1177,6 +1281,7 @@ def _code_search(ctx: ToolContext, query: str, path: str = ".",
|
|||
|
||||
matches: List[str] = []
|
||||
files_searched = 0
|
||||
protected_omitted = 0
|
||||
truncated = False
|
||||
|
||||
for dirpath, dirnames, filenames in os.walk(str(search_root)):
|
||||
|
|
@ -1203,6 +1308,9 @@ def _code_search(ctx: ToolContext, query: str, path: str = ".",
|
|||
continue
|
||||
if normalized == "user_files" and user_files_path_block_reason(ctx, fp):
|
||||
continue
|
||||
if block_reason_for_path(ctx, fp, "read_bytes"):
|
||||
protected_omitted += 1
|
||||
continue
|
||||
|
||||
if _is_search_skippable(fp):
|
||||
continue
|
||||
|
|
@ -1227,11 +1335,14 @@ def _code_search(ctx: ToolContext, query: str, path: str = ".",
|
|||
break
|
||||
|
||||
if not matches:
|
||||
return f"No matches found for {'regex' if regex else 'literal'} `{query}` in {display_search_path} ({files_searched} files searched)."
|
||||
suffix = f" {protected_omitted} protected artifact file(s) omitted." if protected_omitted else ""
|
||||
return f"No matches found for {'regex' if regex else 'literal'} `{query}` in {display_search_path} ({files_searched} files searched).{suffix}"
|
||||
|
||||
header = f"Found {len(matches)} match{'es' if len(matches) != 1 else ''} in {display_search_path} ({files_searched} files searched)"
|
||||
if truncated:
|
||||
header += f" — truncated at {max_results} results"
|
||||
if protected_omitted:
|
||||
header += f" — {protected_omitted} protected artifact file(s) omitted"
|
||||
return header + "\n\n" + "\n".join(matches)
|
||||
|
||||
_SKIP_DIRS = frozenset({
|
||||
|
|
@ -1261,14 +1372,29 @@ def _extract_python_symbols(file_path: pathlib.Path) -> Tuple[List[str], List[st
|
|||
def _codebase_digest(ctx: ToolContext) -> str:
|
||||
"""Generate a compact file/symbol digest for the codebase."""
|
||||
from ouroboros.code_intelligence import build_code_inventory, render_codebase_digest
|
||||
from ouroboros.protected_artifacts import block_reason_for_path, protected_artifact_paths
|
||||
|
||||
repo_root = active_repo_dir_for(ctx)
|
||||
protected_paths = protected_artifact_paths(ctx)
|
||||
inventory = build_code_inventory(
|
||||
active_repo_dir_for(ctx),
|
||||
repo_root,
|
||||
drive_root=pathlib.Path(ctx.drive_root),
|
||||
persist=not _is_local_readonly_subagent(ctx),
|
||||
persist=not _is_local_readonly_subagent(ctx) and not protected_paths,
|
||||
exclude_paths=protected_paths,
|
||||
)
|
||||
if protected_paths:
|
||||
inventory.files = [
|
||||
file for file in inventory.files
|
||||
if not (
|
||||
block_reason_for_path(ctx, repo_root / file.path, "hash")
|
||||
or block_reason_for_path(ctx, repo_root / file.path, "static_introspection")
|
||||
)
|
||||
]
|
||||
coverage: dict[str, int] = {}
|
||||
for file in inventory.files:
|
||||
coverage[file.disposition] = coverage.get(file.disposition, 0) + 1
|
||||
inventory.coverage = coverage
|
||||
if _is_local_readonly_subagent(ctx):
|
||||
repo_root = active_repo_dir_for(ctx)
|
||||
inventory.files = [
|
||||
file for file in inventory.files
|
||||
if not _is_subagent_secret_repo_target(repo_root / file.path, repo_root)
|
||||
|
|
|
|||
|
|
@ -1278,6 +1278,7 @@ def _git_diff(
|
|||
max_chars: int = 0,
|
||||
) -> str:
|
||||
try:
|
||||
repo_dir = active_repo_dir_for(ctx)
|
||||
cmd = ["git", "diff"]
|
||||
if staged:
|
||||
cmd.append("--staged")
|
||||
|
|
@ -1287,7 +1288,12 @@ def _git_diff(
|
|||
cmd.append("--stat")
|
||||
if str(path or "").strip():
|
||||
cmd.extend(["--", safe_relpath(str(path))])
|
||||
return _limit_git_output(run_cmd(cmd, cwd=active_repo_dir_for(ctx)), max_chars)
|
||||
from ouroboros.protected_artifacts import shell_block_reason as protected_artifact_shell_block_reason
|
||||
|
||||
protected_block = protected_artifact_shell_block_reason(ctx, cmd, cwd=str(repo_dir), default_cwd=repo_dir)
|
||||
if protected_block:
|
||||
return protected_block
|
||||
return _limit_git_output(run_cmd(cmd, cwd=repo_dir), max_chars)
|
||||
except Exception as e:
|
||||
return f"⚠️ GIT_ERROR: {_sanitize_git_error(str(e))}"
|
||||
|
||||
|
|
|
|||
|
|
@ -46,6 +46,7 @@ from ouroboros.tools.shell_guards import (
|
|||
shell_has_write_indicator,
|
||||
runtime_data_write_targets,
|
||||
shell_writer_targets_protected,
|
||||
workspace_executor_state_write_block,
|
||||
writer_target_tokens,
|
||||
)
|
||||
from ouroboros.artifacts import task_artifact_dir_path, task_id_for_artifacts
|
||||
|
|
@ -102,6 +103,21 @@ def system_repo_dir_for(ctx: Any) -> pathlib.Path:
|
|||
|
||||
return pathlib.Path(getattr(ctx, "system_repo_dir", None) or getattr(ctx, "repo_dir"))
|
||||
|
||||
|
||||
def _executor_backend_candidate_allowed(ctx: Any, candidate: str, allowed_roots: List[pathlib.Path]) -> bool:
|
||||
try:
|
||||
from ouroboros.workspace_executor import executor_ref_from_ctx as _executor_ref_from_ctx
|
||||
from ouroboros.workspace_executor import map_backend_path as _executor_map_backend_path
|
||||
|
||||
executor_ref = _executor_ref_from_ctx(ctx)
|
||||
if executor_ref is None:
|
||||
return False
|
||||
resolved = _executor_map_backend_path(executor_ref, candidate)
|
||||
return any(resolved.is_relative_to(root) for root in allowed_roots)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _detect_runtime_mode_elevation(text_lower: str) -> bool:
|
||||
"""Detect shell/script attempts to change ``OUROBOROS_RUNTIME_MODE``."""
|
||||
has_save = "save_settings" in text_lower
|
||||
|
|
@ -478,6 +494,7 @@ class ToolContext:
|
|||
memory_mode: str = ""
|
||||
budget_drive_root: str = ""
|
||||
task_metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
executor_ref: Dict[str, Any] = field(default_factory=dict)
|
||||
pending_events: List[Dict[str, Any]] = field(default_factory=list)
|
||||
current_chat_id: Optional[int] = None
|
||||
current_task_type: Optional[str] = None
|
||||
|
|
@ -553,6 +570,13 @@ class ToolContext:
|
|||
def task_drive_root(self) -> pathlib.Path:
|
||||
return (pathlib.Path(self.drive_root).resolve(strict=False) / "task_drives" / task_id_for_artifacts(self)).resolve(strict=False)
|
||||
|
||||
def workspace_executor_ref(self) -> Dict[str, Any]:
|
||||
if isinstance(self.executor_ref, dict) and self.executor_ref:
|
||||
return dict(self.executor_ref)
|
||||
if isinstance(self.task_metadata, dict) and isinstance(self.task_metadata.get("executor_ref"), dict):
|
||||
return dict(self.task_metadata["executor_ref"])
|
||||
return {}
|
||||
|
||||
|
||||
@dataclass
|
||||
class ToolEntry:
|
||||
|
|
@ -877,16 +901,19 @@ class ToolRegistry:
|
|||
argv_executable = pathlib.PurePath(argv_for_write[0]).name.lower().removesuffix(".exe") if argv_for_write else ""
|
||||
write_target_argvs = [argv_for_write] if argv_for_write else []
|
||||
if argv_executable in {"sh", "bash", "zsh"}:
|
||||
inline_cmd = shell_command_string(argv_for_write)
|
||||
inline_cmd = next((str(argv_for_write[idx + 1] or "") for idx, token in enumerate(argv_for_write[1:], start=1) if str(token or "") in {"-c", "--command"} and idx + 1 < len(argv_for_write)), "")
|
||||
if not inline_cmd:
|
||||
inline_cmd = shell_command_string(argv_for_write)
|
||||
inline_argv = strip_leading_env_assignments(unwrap_env_argv(shell_argv(inline_cmd)))
|
||||
if inline_argv:
|
||||
write_target_argvs.append(inline_argv)
|
||||
explicit_write_targets = list(dict.fromkeys(str(token) for target_argv in write_target_argvs for token in writer_target_tokens(target_argv) if str(token or "").strip()))
|
||||
executable_path_tokens = {str(target_argv[0]) for target_argv in write_target_argvs if target_argv}
|
||||
writeish = shell_has_write_indicator(raw_cmd) or (bool(argv_for_write) and argv_executable in LIGHT_SHELL_WRITER_COMMANDS) or bool(explicit_write_targets)
|
||||
protected_artifact_block = protected_artifact_shell_block_reason(self._ctx, raw_cmd, cwd=str(args.get("cwd") or ""), default_cwd=active_repo_dir_for(self._ctx))
|
||||
if protected_artifact_block:
|
||||
if protected_artifact_block := protected_artifact_shell_block_reason(self._ctx, raw_cmd, cwd=str(args.get("cwd") or ""), default_cwd=active_repo_dir_for(self._ctx)):
|
||||
return protected_artifact_block
|
||||
if writeish and (executor_state_block := workspace_executor_state_write_block(raw_cmd, drive_root=pathlib.Path(self._ctx.drive_root), cwd=str(args.get("cwd") or ""), default_cwd=active_repo_dir_for(self._ctx))):
|
||||
return executor_state_block
|
||||
if workspace_mode and writeish:
|
||||
active_root_declared = active_repo_dir_for(self._ctx)
|
||||
active_root = active_root_declared.resolve(strict=False)
|
||||
|
|
@ -959,6 +986,8 @@ class ToolRegistry:
|
|||
if candidate == "/dev/null":
|
||||
continue
|
||||
if is_absolute_path_text(candidate):
|
||||
if _executor_backend_candidate_allowed(self._ctx, candidate, [*allowed_relative_roots, *allowed_data_roots]):
|
||||
continue
|
||||
if not re.match(r"^[A-Za-z]:[\\/]", candidate) and not candidate.startswith("\\\\"):
|
||||
try:
|
||||
resolved = pathlib.Path(candidate).resolve(strict=False)
|
||||
|
|
@ -1017,15 +1046,9 @@ class ToolRegistry:
|
|||
|
||||
# Elevation pattern: blocked in all modes.
|
||||
if _detect_runtime_mode_elevation(cmd_lower):
|
||||
return (
|
||||
"⚠️ ELEVATION_BLOCKED: shell command pattern looks like an OUROBOROS_RUNTIME_MODE elevation attempt "
|
||||
"(mentions ``save_settings`` together with ``OUROBOROS_RUNTIME_MODE``, or invokes ``ouroboros.config.save_settings`` directly). Runtime mode is owner-controlled — change it by stopping the agent and editing settings.json directly, then restart."
|
||||
)
|
||||
return "⚠️ ELEVATION_BLOCKED: shell command pattern looks like an OUROBOROS_RUNTIME_MODE elevation attempt (mentions ``save_settings`` together with ``OUROBOROS_RUNTIME_MODE``, or invokes ``ouroboros.config.save_settings`` directly). Runtime mode is owner-controlled — change it by stopping the agent and editing settings.json directly, then restart."
|
||||
if _detect_context_mode_self_lowering(cmd_lower):
|
||||
return (
|
||||
"⚠️ CONTEXT_MODE_SELF_LOWERING_BLOCKED: shell command pattern looks like an attempt to lower OUROBOROS_CONTEXT_MODE to low "
|
||||
"through settings.json or /api/owner/context-mode. Context mode is owner-controlled — ask the owner to change the Low/Max toggle or edit settings while the agent is stopped."
|
||||
)
|
||||
return "⚠️ CONTEXT_MODE_SELF_LOWERING_BLOCKED: shell command pattern looks like an attempt to lower OUROBOROS_CONTEXT_MODE to low through settings.json or /api/owner/context-mode. Context mode is owner-controlled — ask the owner to change the Low/Max toggle or edit settings while the agent is stopped."
|
||||
if _mentions_skill_owner_state(cmd_lower):
|
||||
return (
|
||||
"⚠️ SKILL_STATE_WRITE_BLOCKED: skill review, enablement, "
|
||||
|
|
|
|||
|
|
@ -344,7 +344,7 @@ def _build_file_facts(
|
|||
facts.disposition = "excluded_test"
|
||||
facts.reason = "wider tests excluded by atlas policy"
|
||||
return facts
|
||||
if _skip_by_dir(rel) and not force_include and not (include_tests and rel.startswith("tests/")):
|
||||
if _skip_by_dir(rel) and rel not in anchors and not force_include and not (include_tests and rel.startswith("tests/")):
|
||||
facts.disposition = "excluded_dir"
|
||||
facts.reason = "excluded non-agent-logic directory"
|
||||
return facts
|
||||
|
|
|
|||
|
|
@ -198,6 +198,9 @@ _FULL_REPO_BINARY_EXTENSIONS = frozenset({
|
|||
})
|
||||
_FULL_REPO_SKIP_DIR_PREFIXES = (
|
||||
".cursor/", ".github/", ".vscode/", ".idea/", "assets/",
|
||||
# Operator/devtools sources are tracked and reviewed when touched, but are
|
||||
# not core runtime context for unrelated broad scope packs.
|
||||
"devtools/",
|
||||
# Full pack excludes tests; touched tests are still sent separately.
|
||||
"tests/",
|
||||
)
|
||||
|
|
|
|||
|
|
@ -24,6 +24,14 @@ from ouroboros.platform_layer import (
|
|||
from ouroboros.tools.registry import ToolContext, ToolEntry
|
||||
from ouroboros.tool_access import resolve_shell_cwd
|
||||
from ouroboros.utils import append_jsonl, utc_now_iso
|
||||
from ouroboros.workspace_executor import executor_ref_from_ctx
|
||||
from ouroboros.workspace_executor import kill_all_services as executor_kill_all_services
|
||||
from ouroboros.workspace_executor import map_host_path as executor_map_host_path
|
||||
from ouroboros.workspace_executor import service_logs as executor_service_logs
|
||||
from ouroboros.workspace_executor import service_status as executor_service_status
|
||||
from ouroboros.workspace_executor import start_service as executor_start_service
|
||||
from ouroboros.workspace_executor import stop_service as executor_stop_service
|
||||
from ouroboros.workspace_executor import stop_task_services as executor_stop_task_services
|
||||
|
||||
|
||||
@dataclass
|
||||
|
|
@ -56,6 +64,17 @@ def _service_key(ctx: ToolContext, name: str) -> str:
|
|||
return f"{task_id}:{name}"
|
||||
|
||||
|
||||
def _executor_can_run_cwd(ctx: ToolContext, workdir: pathlib.Path) -> bool:
|
||||
executor_ref = executor_ref_from_ctx(ctx)
|
||||
if executor_ref is None:
|
||||
return False
|
||||
try:
|
||||
executor_map_host_path(executor_ref, pathlib.Path(workdir).resolve(strict=False))
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _tail(path: pathlib.Path, chars: int) -> str:
|
||||
if not path.exists():
|
||||
return ""
|
||||
|
|
@ -328,6 +347,14 @@ def _start_service(
|
|||
workdir = pathlib.Path(workdir).resolve(strict=False)
|
||||
except Exception as exc:
|
||||
return f"⚠️ SERVICE_CWD_ERROR: {type(exc).__name__}: {exc}"
|
||||
try:
|
||||
from ouroboros.protected_artifacts import shell_block_reason
|
||||
|
||||
protected_block = shell_block_reason(ctx, cmd, cwd=str(workdir), default_cwd=workdir)
|
||||
if protected_block:
|
||||
return protected_block
|
||||
except Exception:
|
||||
pass
|
||||
declared_outputs = [str(item) for item in (outputs or []) if str(item or "").strip()]
|
||||
try:
|
||||
from ouroboros.tools.shell import _snapshot_declared_outputs
|
||||
|
|
@ -335,6 +362,21 @@ def _start_service(
|
|||
before_outputs = _snapshot_declared_outputs(ctx, declared_outputs, workdir, cwd_root=cwd_root)
|
||||
except Exception:
|
||||
before_outputs = {}
|
||||
if _executor_can_run_cwd(ctx, workdir):
|
||||
try:
|
||||
payload = executor_start_service(
|
||||
ctx,
|
||||
name=service_name,
|
||||
cmd=[str(part) for part in cmd],
|
||||
host_cwd=workdir,
|
||||
cwd_root=cwd_root,
|
||||
readiness=dict(readiness or {}),
|
||||
outputs=declared_outputs,
|
||||
before_outputs=before_outputs,
|
||||
)
|
||||
return json.dumps(payload, ensure_ascii=False, indent=2)
|
||||
except Exception as exc:
|
||||
return f"⚠️ SERVICE_START_ERROR: executor backend failed: {type(exc).__name__}: {exc}"
|
||||
task_id = str(getattr(ctx, "task_id", "") or "manual")
|
||||
log_dir = pathlib.Path(ctx.drive_root) / "services" / task_id
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
|
@ -423,9 +465,14 @@ def _service_status(ctx: ToolContext, name: str = "service") -> str:
|
|||
key = _service_key(ctx, service_name)
|
||||
with _LOCK:
|
||||
record = _SERVICES.get(key)
|
||||
if not record:
|
||||
return f"⚠️ SERVICE_NOT_FOUND: {name}"
|
||||
return json.dumps(_status_payload(record), ensure_ascii=False, indent=2)
|
||||
if record:
|
||||
return json.dumps(_status_payload(record), ensure_ascii=False, indent=2)
|
||||
if executor_ref_from_ctx(ctx) is not None:
|
||||
payload = executor_service_status(ctx, service_name)
|
||||
if payload is None:
|
||||
return f"⚠️ SERVICE_NOT_FOUND: {name}"
|
||||
return json.dumps(payload, ensure_ascii=False, indent=2)
|
||||
return f"⚠️ SERVICE_NOT_FOUND: {name}"
|
||||
|
||||
|
||||
def _service_logs(ctx: ToolContext, name: str = "service", tail: int = 8000) -> str:
|
||||
|
|
@ -435,32 +482,41 @@ def _service_logs(ctx: ToolContext, name: str = "service", tail: int = 8000) ->
|
|||
key = _service_key(ctx, service_name)
|
||||
with _LOCK:
|
||||
record = _SERVICES.get(key)
|
||||
if not record:
|
||||
return f"⚠️ SERVICE_NOT_FOUND: {name}"
|
||||
try:
|
||||
tail_chars = int(tail or 8000)
|
||||
except (TypeError, ValueError):
|
||||
return "⚠️ TOOL_ARG_ERROR (service_logs): tail must be an integer."
|
||||
tail_chars = min(max(1, tail_chars), _MAX_SERVICE_LOG_TAIL_CHARS)
|
||||
text = str(redact_projection(_tail(record.log_path, tail_chars)).value)
|
||||
ref = {}
|
||||
omitted_reason = ""
|
||||
try:
|
||||
size = record.log_path.stat().st_size if record.log_path.exists() else 0
|
||||
if size <= _MAX_SERVICE_LOG_BLOB_BYTES:
|
||||
full = record.log_path.read_text(encoding="utf-8", errors="replace") if record.log_path.exists() else ""
|
||||
ref = write_blob(pathlib.Path(ctx.drive_root), full, kind="txt")
|
||||
else:
|
||||
omitted_reason = f"log exceeds {_MAX_SERVICE_LOG_BLOB_BYTES} byte blob cap"
|
||||
except Exception:
|
||||
if record:
|
||||
try:
|
||||
tail_chars = int(tail or 8000)
|
||||
except (TypeError, ValueError):
|
||||
return "⚠️ TOOL_ARG_ERROR (service_logs): tail must be an integer."
|
||||
tail_chars = min(max(1, tail_chars), _MAX_SERVICE_LOG_TAIL_CHARS)
|
||||
text = str(redact_projection(_tail(record.log_path, tail_chars)).value)
|
||||
ref = {}
|
||||
return json.dumps({
|
||||
"service_id": record.service_id,
|
||||
"name": record.name,
|
||||
"tail": text,
|
||||
"full_log_ref": ref,
|
||||
"full_log_omitted": omitted_reason,
|
||||
}, ensure_ascii=False, indent=2)
|
||||
omitted_reason = ""
|
||||
try:
|
||||
size = record.log_path.stat().st_size if record.log_path.exists() else 0
|
||||
if size <= _MAX_SERVICE_LOG_BLOB_BYTES:
|
||||
full = record.log_path.read_text(encoding="utf-8", errors="replace") if record.log_path.exists() else ""
|
||||
ref = write_blob(pathlib.Path(ctx.drive_root), full, kind="txt")
|
||||
else:
|
||||
omitted_reason = f"log exceeds {_MAX_SERVICE_LOG_BLOB_BYTES} byte blob cap"
|
||||
except Exception:
|
||||
ref = {}
|
||||
return json.dumps({
|
||||
"service_id": record.service_id,
|
||||
"name": record.name,
|
||||
"tail": text,
|
||||
"full_log_ref": ref,
|
||||
"full_log_omitted": omitted_reason,
|
||||
}, ensure_ascii=False, indent=2)
|
||||
if executor_ref_from_ctx(ctx) is not None:
|
||||
try:
|
||||
tail_chars = int(tail or 8000)
|
||||
except (TypeError, ValueError):
|
||||
return "⚠️ TOOL_ARG_ERROR (service_logs): tail must be an integer."
|
||||
payload = executor_service_logs(ctx, service_name, tail_chars)
|
||||
if payload is None:
|
||||
return f"⚠️ SERVICE_NOT_FOUND: {name}"
|
||||
return json.dumps(payload, ensure_ascii=False, indent=2)
|
||||
return f"⚠️ SERVICE_NOT_FOUND: {name}"
|
||||
|
||||
|
||||
def _stop_service(ctx: ToolContext, name: str = "service") -> str:
|
||||
|
|
@ -470,40 +526,70 @@ def _stop_service(ctx: ToolContext, name: str = "service") -> str:
|
|||
key = _service_key(ctx, service_name)
|
||||
with _LOCK:
|
||||
record = _SERVICES.pop(key, None)
|
||||
if not record:
|
||||
return f"⚠️ SERVICE_NOT_FOUND: {name}"
|
||||
_stop_record(record)
|
||||
payload = _status_payload(record)
|
||||
payload["log_finalization"] = _finalize_service_log(ctx, record)
|
||||
artifact_note = ""
|
||||
artifact_failed = False
|
||||
if record.outputs:
|
||||
try:
|
||||
from ouroboros.tools.shell import _register_process_outputs
|
||||
if record:
|
||||
_stop_record(record)
|
||||
payload = _status_payload(record)
|
||||
payload["log_finalization"] = _finalize_service_log(ctx, record)
|
||||
artifact_note = ""
|
||||
artifact_failed = False
|
||||
if record.outputs:
|
||||
try:
|
||||
from ouroboros.tools.shell import _register_process_outputs
|
||||
|
||||
artifact_note, artifact_failed = _register_process_outputs(
|
||||
ctx,
|
||||
record.outputs,
|
||||
pathlib.Path(record.cwd),
|
||||
cwd_root=record.cwd_root,
|
||||
before_outputs=record.before_outputs,
|
||||
artifact_note, artifact_failed = _register_process_outputs(
|
||||
ctx,
|
||||
record.outputs,
|
||||
pathlib.Path(record.cwd),
|
||||
cwd_root=record.cwd_root,
|
||||
before_outputs=record.before_outputs,
|
||||
)
|
||||
except Exception as exc:
|
||||
artifact_note = f"\n\n⚠️ ARTIFACT_OUTPUT_ERROR:\n- service output finalization failed: {type(exc).__name__}: {exc}"
|
||||
artifact_failed = True
|
||||
elif record.cwd_root == "user_files":
|
||||
payload["artifact_audit_gap"] = (
|
||||
"⚠️ ARTIFACT_AUDIT_GAP: service ran in user_files cwd without outputs=[...]. "
|
||||
"If it created a deliverable, rerun/register the file with outputs or "
|
||||
"write_file(root=artifact_store) before claiming it."
|
||||
)
|
||||
except Exception as exc:
|
||||
artifact_note = f"\n\n⚠️ ARTIFACT_OUTPUT_ERROR:\n- service output finalization failed: {type(exc).__name__}: {exc}"
|
||||
artifact_failed = True
|
||||
elif record.cwd_root == "user_files":
|
||||
payload["artifact_audit_gap"] = (
|
||||
"⚠️ ARTIFACT_AUDIT_GAP: service ran in user_files cwd without outputs=[...]. "
|
||||
"If it created a deliverable, rerun/register the file with outputs or "
|
||||
"write_file(root=artifact_store) before claiming it."
|
||||
)
|
||||
if artifact_note:
|
||||
payload["artifact_outputs"] = artifact_note.strip()
|
||||
payload["artifact_output_failed"] = bool(artifact_failed)
|
||||
rendered = json.dumps(payload, ensure_ascii=False, indent=2)
|
||||
if artifact_failed:
|
||||
return "⚠️ ARTIFACT_OUTPUT_ERROR (stop_service): declared service outputs were not finalized.\n\n" + rendered
|
||||
return rendered
|
||||
if artifact_note:
|
||||
payload["artifact_outputs"] = artifact_note.strip()
|
||||
payload["artifact_output_failed"] = bool(artifact_failed)
|
||||
rendered = json.dumps(payload, ensure_ascii=False, indent=2)
|
||||
if artifact_failed:
|
||||
return "⚠️ ARTIFACT_OUTPUT_ERROR (stop_service): declared service outputs were not finalized.\n\n" + rendered
|
||||
return rendered
|
||||
if executor_ref_from_ctx(ctx) is not None:
|
||||
payload = executor_stop_service(ctx, service_name)
|
||||
if payload is None:
|
||||
return f"⚠️ SERVICE_NOT_FOUND: {name}"
|
||||
if payload.get("stop_failed"):
|
||||
return "⚠️ SERVICE_STOP_ERROR (stop_service): executor backend did not confirm service termination.\n\n" + json.dumps(payload, ensure_ascii=False, indent=2)
|
||||
artifact_note = ""
|
||||
artifact_failed = False
|
||||
before_outputs = payload.pop("_before_outputs", {})
|
||||
if payload.get("outputs"):
|
||||
try:
|
||||
from ouroboros.tools.shell import _register_process_outputs
|
||||
|
||||
artifact_note, artifact_failed = _register_process_outputs(
|
||||
ctx,
|
||||
[str(item) for item in (payload.get("outputs") or [])],
|
||||
pathlib.Path(str(payload.get("host_cwd") or ".")),
|
||||
cwd_root=str(payload.get("cwd_root") or ""),
|
||||
before_outputs=before_outputs if isinstance(before_outputs, dict) else None,
|
||||
)
|
||||
except Exception as exc:
|
||||
artifact_note = f"\n\n⚠️ ARTIFACT_OUTPUT_ERROR:\n- executor service output finalization failed: {type(exc).__name__}: {exc}"
|
||||
artifact_failed = True
|
||||
if artifact_note:
|
||||
payload["artifact_outputs"] = artifact_note.strip()
|
||||
payload["artifact_output_failed"] = bool(artifact_failed)
|
||||
rendered = json.dumps(payload, ensure_ascii=False, indent=2)
|
||||
if artifact_failed:
|
||||
return "⚠️ ARTIFACT_OUTPUT_ERROR (stop_service): declared executor service outputs were not finalized.\n\n" + rendered
|
||||
return rendered
|
||||
return f"⚠️ SERVICE_NOT_FOUND: {name}"
|
||||
|
||||
|
||||
def stop_task_services(ctx: ToolContext) -> List[Dict[str, Any]]:
|
||||
|
|
@ -524,6 +610,10 @@ def stop_task_services(ctx: ToolContext) -> List[Dict[str, Any]]:
|
|||
stopped.append(payload)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
stopped.extend(executor_stop_task_services(ctx))
|
||||
except Exception:
|
||||
pass
|
||||
return stopped
|
||||
|
||||
|
||||
|
|
@ -544,6 +634,10 @@ def kill_all_services(
|
|||
if wait and drive_root is not None:
|
||||
payload["log_finalization"] = _finalize_service_log_for_drive(pathlib.Path(drive_root), record)
|
||||
stopped.append(payload)
|
||||
try:
|
||||
stopped.extend(executor_kill_all_services(drive_root, wait=wait))
|
||||
except Exception:
|
||||
pass
|
||||
if wait and drive_root is not None and stopped:
|
||||
def _compact(payload: Dict[str, Any]) -> Dict[str, Any]:
|
||||
item = dict(payload)
|
||||
|
|
|
|||
|
|
@ -51,6 +51,10 @@ from ouroboros.contracts.skill_payload_policy import (
|
|||
decide_payload_short_form,
|
||||
resolve_skill_payload_target,
|
||||
)
|
||||
from ouroboros.workspace_executor import execute as executor_execute
|
||||
from ouroboros.workspace_executor import executor_ref_from_ctx
|
||||
from ouroboros.workspace_executor import map_backend_path as executor_map_backend_path
|
||||
from ouroboros.workspace_executor import map_host_path as executor_map_host_path
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
|
@ -167,6 +171,15 @@ def _allowed_output_roots(ctx: ToolContext, work_dir: pathlib.Path, cwd_root: st
|
|||
def _protected_output_source_reason(ctx: ToolContext, source: pathlib.Path, label: str, changed_paths: set[str]) -> str:
|
||||
"""Return a block reason for protected/control-plane output sources."""
|
||||
|
||||
try:
|
||||
from ouroboros.protected_artifacts import block_reason_for_path
|
||||
|
||||
protected_artifact_reason = block_reason_for_path(ctx, source, "copy")
|
||||
if protected_artifact_reason:
|
||||
return protected_artifact_reason
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
name_lower = source.name.lower()
|
||||
if (
|
||||
source.name.startswith(".")
|
||||
|
|
@ -214,7 +227,13 @@ def _resolve_declared_output(
|
|||
if not text:
|
||||
return None, "empty output path"
|
||||
raw = pathlib.Path(text).expanduser()
|
||||
if raw.is_absolute() or text.startswith("~"):
|
||||
executor_ref = executor_ref_from_ctx(ctx)
|
||||
if executor_ref is not None and raw.is_absolute() and not text.startswith("~"):
|
||||
try:
|
||||
source = executor_map_backend_path(executor_ref, text)
|
||||
except ValueError:
|
||||
source = raw.resolve(strict=False)
|
||||
elif raw.is_absolute() or text.startswith("~"):
|
||||
source = raw.resolve(strict=False)
|
||||
else:
|
||||
source = (pathlib.Path(work_dir) / safe_relpath(text)).resolve(strict=False)
|
||||
|
|
@ -318,6 +337,17 @@ def _register_process_outputs(
|
|||
return "\n\n" + prefix + ":\n" + "\n".join(f"- {note}" for note in notes), failed
|
||||
|
||||
|
||||
def _executor_can_run_cwd(ctx: ToolContext, work_dir: pathlib.Path) -> bool:
|
||||
executor_ref = executor_ref_from_ctx(ctx)
|
||||
if executor_ref is None:
|
||||
return False
|
||||
try:
|
||||
executor_map_host_path(executor_ref, pathlib.Path(work_dir).resolve(strict=False))
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _resolve_git_root(path: pathlib.Path) -> pathlib.Path | None:
|
||||
try:
|
||||
from ouroboros.review_state import discover_repo_root
|
||||
|
|
@ -783,18 +813,25 @@ def _run_shell(ctx: ToolContext, cmd, cwd: str = "", outputs: List[str] | None =
|
|||
timeout_sec = _resolve_effective_timeout(_RUN_SHELL_DEFAULT_TIMEOUT_SEC)
|
||||
bootstrap_process_path()
|
||||
try:
|
||||
res = _tracked_subprocess_run(
|
||||
cmd, cwd=str(work_dir),
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
text=True, timeout=timeout_sec,
|
||||
)
|
||||
if _executor_can_run_cwd(ctx, pathlib.Path(work_dir)):
|
||||
res = executor_execute(ctx, cmd, pathlib.Path(work_dir), timeout_sec)
|
||||
else:
|
||||
res = _tracked_subprocess_run(
|
||||
cmd, cwd=str(work_dir),
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
text=True, timeout=timeout_sec,
|
||||
)
|
||||
if res.returncode != 0:
|
||||
executor_note = ""
|
||||
if getattr(res, "backend_trace", None):
|
||||
executor_note = "\n\nEXECUTOR_TRACE:\n" + json.dumps(res.backend_trace, ensure_ascii=False, indent=2)
|
||||
if _is_search_no_match(res):
|
||||
return autocorrect_note + (
|
||||
f"{_describe_returncode(res.returncode, cwd=work_dir)} (no matches)\n"
|
||||
f"{_format_process_output(res.stdout or '', '')}"
|
||||
f"{executor_note}"
|
||||
)
|
||||
return autocorrect_note + f"⚠️ SHELL_EXIT_ERROR: command exited with {_describe_returncode(res.returncode, cwd=work_dir)}.\n\n{_format_process_output(res.stdout or '', res.stderr or '')}"
|
||||
return autocorrect_note + f"⚠️ SHELL_EXIT_ERROR: command exited with {_describe_returncode(res.returncode, cwd=work_dir)}.\n\n{_format_process_output(res.stdout or '', res.stderr or '')}{executor_note}"
|
||||
after_changed = _status_snapshot(repo_root)
|
||||
if after_changed != before_changed:
|
||||
_invalidate_advisory(
|
||||
|
|
@ -837,7 +874,10 @@ def _run_shell(ctx: ToolContext, cmd, cwd: str = "", outputs: List[str] | None =
|
|||
+ f"{_format_process_output(res.stdout or '', res.stderr or '')}"
|
||||
+ artifact_note
|
||||
)
|
||||
return autocorrect_note + f"{_describe_returncode(0, cwd=work_dir)}\n{_format_process_output(res.stdout or '', res.stderr or '')}{artifact_note}{audit_note}"
|
||||
executor_note = ""
|
||||
if getattr(res, "backend_trace", None):
|
||||
executor_note = "\n\nEXECUTOR_TRACE:\n" + json.dumps(res.backend_trace, ensure_ascii=False, indent=2)
|
||||
return autocorrect_note + f"{_describe_returncode(0, cwd=work_dir)}\n{_format_process_output(res.stdout or '', res.stderr or '')}{artifact_note}{audit_note}{executor_note}"
|
||||
except subprocess.TimeoutExpired:
|
||||
return (
|
||||
f"⚠️ TOOL_TIMEOUT (run_command): command exceeded {timeout_sec}s. "
|
||||
|
|
@ -924,6 +964,18 @@ def _control_restore_note(restored: list[str]) -> str:
|
|||
)
|
||||
|
||||
|
||||
def _claude_code_executor_block_reason(ctx: ToolContext, work_dir_path: pathlib.Path) -> str:
|
||||
executor_ref = executor_ref_from_ctx(ctx)
|
||||
if executor_ref is None or executor_ref.kind != "docker_exec" or not _executor_can_run_cwd(ctx, work_dir_path):
|
||||
return ""
|
||||
return (
|
||||
"⚠️ CLAUDE_CODE_EDIT_BLOCKED: docker executor-backed workspaces route "
|
||||
"process execution through a backend, but claude_code_edit edits host "
|
||||
"paths directly. Use read/write/edit tools or run_command inside the "
|
||||
"mapped workspace until a reviewed backend-safe Claude Code path exists."
|
||||
)
|
||||
|
||||
|
||||
def _claude_code_edit(ctx: ToolContext, prompt: str, cwd: str = "", budget: float = 5.0, validate: bool = False, bucket: str = "", skill_name: str = "", outputs: List[str] | None = None) -> str:
|
||||
"""Delegate SDK edits with cwd and protected-path safety hooks."""
|
||||
from ouroboros.tools.git import _acquire_git_lock, _release_git_lock
|
||||
|
|
@ -994,7 +1046,7 @@ def _claude_code_edit(ctx: ToolContext, prompt: str, cwd: str = "", budget: floa
|
|||
allowed_roots = []
|
||||
allowed_text = ", ".join(f"{name}={pathlib.Path(root).resolve(strict=False)}" for name, root in allowed_roots)
|
||||
return f"⚠️ CLAUDE_CODE_ERROR: cwd escapes allowed workspace edit roots. {e}. allowed_roots: {allowed_text}. Use the active workspace, task_drive, or artifact_store for workspace tasks."
|
||||
if cwd_root not in {"active_workspace", "task_drive", "artifact_store"}:
|
||||
if cwd_root not in {"active_workspace", "task_drive", "artifact_store", "user_files"}:
|
||||
return "⚠️ CLAUDE_CODE_ERROR: cwd root is unavailable for workspace task edits."
|
||||
work_dir_root = cwd_root
|
||||
else:
|
||||
|
|
@ -1021,6 +1073,9 @@ def _claude_code_edit(ctx: ToolContext, prompt: str, cwd: str = "", budget: floa
|
|||
return f"⚠️ CLAUDE_CODE_ERROR: cwd not found or not a directory: {cwd}"
|
||||
work_dir = str(candidate)
|
||||
work_dir_path = pathlib.Path(work_dir).resolve()
|
||||
executor_block = _claude_code_executor_block_reason(ctx, work_dir_path)
|
||||
if executor_block:
|
||||
return executor_block
|
||||
before_outputs = _snapshot_declared_outputs(ctx, outputs, work_dir_path, cwd_root=work_dir_root)
|
||||
skill_control_snapshots = {}
|
||||
sidecar_root = pathlib.Path(skill_payload_root).resolve() if skill_payload_root is not None else None
|
||||
|
|
@ -1242,9 +1297,20 @@ def _run_script(
|
|||
+ ". Re-run with outputs=[...] or write the canonical deliverable via root=artifact_store."
|
||||
)
|
||||
try:
|
||||
root = pathlib.Path(ctx.task_drive_root()) / "tmp_scripts"
|
||||
workdir, _cwd_root, _allowed = resolve_shell_cwd(ctx, cwd)
|
||||
resolved_workdir = pathlib.Path(workdir).resolve(strict=False)
|
||||
except Exception:
|
||||
root = pathlib.Path(ctx.drive_root) / "tmp_scripts"
|
||||
if executor_ref_from_ctx(ctx) is not None:
|
||||
return f"⚠️ RUN_SCRIPT_BLOCKED: executor-backed run_script could not resolve mapped cwd {cwd!r}."
|
||||
resolved_workdir = pathlib.Path("")
|
||||
executor_active = _executor_can_run_cwd(ctx, resolved_workdir) if str(resolved_workdir) else False
|
||||
if executor_active:
|
||||
root = resolved_workdir / ".ouroboros" / "tmp_scripts"
|
||||
else:
|
||||
try:
|
||||
root = pathlib.Path(ctx.task_drive_root()) / "tmp_scripts"
|
||||
except Exception:
|
||||
root = pathlib.Path(ctx.drive_root) / "tmp_scripts"
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
suffix = ".py" if "python" in pathlib.PurePath(interp).name else ".sh"
|
||||
script_path = root / f"script_{uuid.uuid4().hex}{suffix}"
|
||||
|
|
@ -1253,7 +1319,16 @@ def _run_script(
|
|||
os.chmod(script_path, 0o600)
|
||||
except OSError:
|
||||
pass
|
||||
argv = [interp, str(script_path), *[str(item) for item in (args or [])]]
|
||||
script_arg = str(script_path)
|
||||
if executor_active:
|
||||
executor = executor_ref_from_ctx(ctx)
|
||||
if executor is not None and executor.kind != "local":
|
||||
try:
|
||||
script_arg = executor_map_host_path(executor, script_path)
|
||||
except Exception as exc:
|
||||
script_path.unlink(missing_ok=True)
|
||||
return f"⚠️ RUN_SCRIPT_BLOCKED: executor-backed run_script could not map temp script path: {type(exc).__name__}: {exc}"
|
||||
argv = [interp, script_arg, *[str(item) for item in (args or [])]]
|
||||
effective_cwd = str(cwd or "")
|
||||
if (
|
||||
not effective_cwd.strip()
|
||||
|
|
@ -1261,7 +1336,16 @@ def _run_script(
|
|||
and not bool(getattr(ctx, "is_workspace_mode", lambda: False)())
|
||||
):
|
||||
effective_cwd = str(pathlib.Path(ctx.task_drive_root()).resolve(strict=False))
|
||||
result = _run_shell(ctx, argv, cwd=effective_cwd, outputs=outputs)
|
||||
try:
|
||||
result = _run_shell(ctx, argv, cwd=effective_cwd, outputs=outputs)
|
||||
finally:
|
||||
if executor_active:
|
||||
try:
|
||||
script_path.unlink(missing_ok=True)
|
||||
script_path.parent.rmdir()
|
||||
script_path.parent.parent.rmdir()
|
||||
except OSError:
|
||||
pass
|
||||
if str(result).lstrip().startswith("⚠️"):
|
||||
return f"{result}\n# script_path={script_path}"
|
||||
return f"# script_path={script_path}\n{result}"
|
||||
|
|
@ -1323,7 +1407,10 @@ def get_tools() -> List[ToolEntry]:
|
|||
"description": (
|
||||
"Working directory under the active repo/workspace, task_drive, artifact_store, "
|
||||
"an external absolute/~/ user_files path, or an explicit "
|
||||
"data/skills/<bucket>/<skill> payload path for skill repair."
|
||||
"data/skills/<bucket>/<skill> payload path for skill repair. "
|
||||
"For docker executor-backed external workspaces, mapped active_workspace cwd is "
|
||||
"blocked until a backend-safe Claude Code path exists; unmapped task_drive, "
|
||||
"artifact_store, and user_files cwd remain valid where runtime mode permits."
|
||||
),
|
||||
},
|
||||
"budget": {"type": "number", "default": 5.0},
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ SHELL_WRITE_INDICATORS = (
|
|||
"mv ", "cp ", "chmod ", "chown ", "unlink ", "delete", "trash",
|
||||
"rsync ", "write_text", "open(", ".write(", ".writelines(",
|
||||
"os.remove(", "os.unlink(", "os.mkdir(", "os.makedirs(", "sort -o",
|
||||
"writefilesync", "appendfilesync", "createwritestream",
|
||||
)
|
||||
_SAFE_STDIO_REDIRECT_TOKENS = frozenset({
|
||||
">/dev/null",
|
||||
|
|
@ -327,6 +328,46 @@ def shell_writer_targets_protected(raw_cmd: Any) -> bool:
|
|||
return bool(target_text and any(cf in target_text for cf in PROTECTED_RUNTIME_PATHS_LOWER))
|
||||
|
||||
|
||||
def _workspace_executor_state_target(path: pathlib.Path, drive_root: pathlib.Path) -> bool:
|
||||
try:
|
||||
rel_parts = pathlib.Path(path).resolve(strict=False).relative_to(
|
||||
pathlib.Path(drive_root).resolve(strict=False)
|
||||
).parts
|
||||
except (OSError, ValueError):
|
||||
return False
|
||||
lowered = [str(part).casefold() for part in rel_parts]
|
||||
return "state" in lowered and "workspace_executor_processes" in lowered
|
||||
|
||||
|
||||
def workspace_executor_state_write_block(
|
||||
raw_cmd: Any,
|
||||
*,
|
||||
drive_root: pathlib.Path,
|
||||
cwd: str = "",
|
||||
default_cwd: pathlib.Path | None = None,
|
||||
) -> str:
|
||||
try:
|
||||
drive = pathlib.Path(drive_root).resolve(strict=False)
|
||||
work_dir = pathlib.Path(cwd).expanduser() if str(cwd or "").strip() else pathlib.Path(default_cwd or ".")
|
||||
if not work_dir.is_absolute():
|
||||
work_dir = pathlib.Path(default_cwd or ".") / work_dir
|
||||
work_dir = work_dir.resolve(strict=False)
|
||||
except Exception:
|
||||
return ""
|
||||
targets = [
|
||||
target for target in runtime_data_write_targets(raw_cmd, drive_root=drive, work_dir=work_dir, allowed_roots=[])
|
||||
if _workspace_executor_state_target(pathlib.Path(target), drive)
|
||||
]
|
||||
if not targets:
|
||||
return ""
|
||||
return (
|
||||
"⚠️ WORKSPACE_EXECUTOR_STATE_WRITE_BLOCKED: workspace executor process records "
|
||||
"are owner/runtime control-plane state. Use process/service lifecycle tools "
|
||||
"instead of shell-writing state/workspace_executor_processes. Paths: "
|
||||
+ ", ".join(targets[:5])
|
||||
)
|
||||
|
||||
|
||||
def light_shell_repo_mutation(
|
||||
raw_cmd: Any,
|
||||
*,
|
||||
|
|
|
|||
1059
ouroboros/workspace_executor.py
Normal file
1059
ouroboros/workspace_executor.py
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -350,7 +350,7 @@ Use `web_search` when external API/library/model behavior may be stale or versio
|
|||
- New files or intentional full rewrites: `write_file` (shrink guard applies) → `commit_reviewed`.
|
||||
- Coordinated/multi-file/non-obvious edits: plan the data flow, apply focused `edit_text`/`write_file` calls, inspect diff → `commit_reviewed`.
|
||||
- For non-trivial, headless, workspace, or effectful work, state success criteria early and call `plan_task` before major design/build/edit work unless it is explicitly unnecessary; choose its `context_level` yourself (`minimal`, `localized`, `broad`, or `constitutional`) based on the actual risk and scope. If you skip `plan_task`, say why in the reasoning trace or final summary.
|
||||
- For substantial external code artifacts, `claude_code_edit` may work in an external `user_files`, `task_drive`, or `artifact_store` cwd in direct tasks; workspace tasks use the active workspace plus task/artifact roots. This is a first-class coding path, not a shell workaround. Pass `outputs=[...]` for generated deliverables so they are copied into the task artifact store. Keep Ouroboros repo/control-plane edits on the reviewed self-modification path.
|
||||
- For substantial external code artifacts, `claude_code_edit` may work in an external `user_files`, `task_drive`, or `artifact_store` cwd in direct tasks; workspace tasks use the active workspace plus task/artifact roots. In docker executor-backed external workspaces, mapped active workspace cwd is blocked until a reviewed backend-safe Claude Code path exists; unmapped `task_drive`, `artifact_store`, and `user_files` cwd remain valid where the active profile permits them. This is a first-class coding path, not a shell workaround. Pass `outputs=[...]` for generated deliverables so they are copied into the task artifact store. Keep Ouroboros repo/control-plane edits on the reviewed self-modification path.
|
||||
- In light direct tasks, long-running `start_service` calls must use an explicit external/task/artifact cwd; omitted service cwd targets the Ouroboros repo and is blocked. Pass service `outputs=[...]` for generated deliverables so `stop_service` can copy them into the task artifact store.
|
||||
- Before saying work is done, reopen or otherwise verify the changed deliverable/artifact through the most authoritative available surface; if verification is blocked or incomplete, say that explicitly.
|
||||
- For shared-state or multi-pass logic, write the data flow/invariants before editing.
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||
|
||||
[project]
|
||||
name = "ouroboros"
|
||||
version = "6.18.1"
|
||||
version = "6.19.0rc1"
|
||||
description = "Self-creating AI agent with constitution, background consciousness, and persistent identity"
|
||||
readme = "README.md"
|
||||
license = {text = "MIT"}
|
||||
|
|
@ -96,7 +96,6 @@ web = [
|
|||
"providers/*.png",
|
||||
"providers/*.ico",
|
||||
]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 120
|
||||
target-version = "py310"
|
||||
|
|
|
|||
|
|
@ -1,231 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Minimal SWE-bench prediction helper backed by ``ouroboros run``.
|
||||
|
||||
Input is a JSONL file whose rows include ``instance_id``, ``workspace_root``,
|
||||
and an instruction field (``problem_statement`` or ``prompt``). Output is a
|
||||
SWE-bench-compatible predictions JSONL.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from ouroboros.config import get_finalization_grace_sec
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--input", required=True, help="JSONL instances")
|
||||
parser.add_argument("--output", required=True, help="predictions JSONL")
|
||||
parser.add_argument("--model-name", default="ouroboros-cli")
|
||||
parser.add_argument("--cli", default="", help="optional Ouroboros CLI command prefix, e.g. 'ouroboros'")
|
||||
parser.add_argument("--timeout", type=int, default=7200, help="per-instance Ouroboros CLI timeout seconds")
|
||||
parser.add_argument("--continue-on-error", action="store_true", help="continue after failed instances and write errors JSONL")
|
||||
parser.add_argument("--errors-output", default="", help="errors JSONL path; defaults to <output>.errors.jsonl when continuing")
|
||||
parser.add_argument("--logs-dir", default="", help="optional directory for per-instance stdout/stderr logs")
|
||||
parser.add_argument(
|
||||
"--workspaces-root",
|
||||
default="",
|
||||
help="optional directory containing per-instance or repo-name local checkouts",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
rows = []
|
||||
errors = []
|
||||
def record_error(row: dict) -> bool:
|
||||
if not args.continue_on_error:
|
||||
raise RuntimeError(str(row.get("error") or row))
|
||||
errors.append(row)
|
||||
return True
|
||||
|
||||
for raw in Path(args.input).read_text(encoding="utf-8").splitlines():
|
||||
if not raw.strip():
|
||||
continue
|
||||
item: Any = json.loads(raw)
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
instance_id = str(item.get("instance_id") or "")
|
||||
workspace = str(item.get("workspace_root") or "").strip()
|
||||
if not workspace and args.workspaces_root:
|
||||
root = Path(args.workspaces_root).expanduser()
|
||||
repo = str(item.get("repo") or "").strip()
|
||||
candidates = [root / instance_id]
|
||||
if repo:
|
||||
candidates.extend([root / repo.replace("/", "__"), root / repo.split("/")[-1]])
|
||||
for candidate in candidates:
|
||||
if candidate.is_dir():
|
||||
workspace = str(candidate)
|
||||
break
|
||||
prompt = str(item.get("problem_statement") or item.get("prompt") or "")
|
||||
if not instance_id or not workspace or not prompt:
|
||||
record_error({
|
||||
"instance_id": instance_id,
|
||||
"error": "each row must include instance_id, workspace_root or --workspaces-root, and problem_statement/prompt",
|
||||
"reason_code": "invalid_instance",
|
||||
})
|
||||
continue
|
||||
workspace_path = Path(workspace).expanduser().resolve(strict=False)
|
||||
if not workspace_path.is_dir():
|
||||
record_error({
|
||||
"instance_id": instance_id,
|
||||
"error": f"workspace_root is not a directory for {instance_id}: {workspace}",
|
||||
"reason_code": "invalid_workspace",
|
||||
})
|
||||
continue
|
||||
head = subprocess.run(["git", "rev-parse", "HEAD"], cwd=workspace_path, capture_output=True, text=True, timeout=10)
|
||||
if head.returncode != 0:
|
||||
record_error({
|
||||
"instance_id": instance_id,
|
||||
"error": f"workspace_root is not a git checkout for {instance_id}: {workspace_path}",
|
||||
"reason_code": "not_git_checkout",
|
||||
})
|
||||
continue
|
||||
base_commit = str(item.get("base_commit") or "").strip()
|
||||
if base_commit and head.stdout.strip() != base_commit:
|
||||
record_error({
|
||||
"instance_id": instance_id,
|
||||
"error": f"workspace HEAD for {instance_id} is {head.stdout.strip()}, expected base_commit {base_commit}",
|
||||
"reason_code": "wrong_base_commit",
|
||||
})
|
||||
continue
|
||||
status = subprocess.run(
|
||||
["git", "status", "--porcelain=v1", "--untracked-files=all"],
|
||||
cwd=workspace_path,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
)
|
||||
if status.returncode != 0 or status.stdout.strip():
|
||||
record_error({
|
||||
"instance_id": instance_id,
|
||||
"error": f"workspace must be clean before SWE-bench run for {instance_id}",
|
||||
"reason_code": "dirty_workspace",
|
||||
})
|
||||
continue
|
||||
cli_prefix = shlex.split(args.cli) if args.cli else [sys.executable, "-m", "ouroboros.cli"]
|
||||
if args.logs_dir:
|
||||
result_json_path = Path(args.logs_dir).expanduser() / instance_id / "task_result.json"
|
||||
else:
|
||||
result_json_path = Path(tempfile.gettempdir()) / f"ouroboros_swebench_{instance_id}.task_result.json"
|
||||
result_json_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
cmd = [
|
||||
*cli_prefix,
|
||||
"run",
|
||||
"--workspace",
|
||||
str(workspace_path),
|
||||
"--memory-mode",
|
||||
"empty",
|
||||
"--timeout",
|
||||
str(int(args.timeout)),
|
||||
"--patch",
|
||||
"--result-json-out",
|
||||
str(result_json_path),
|
||||
prompt,
|
||||
]
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=int(args.timeout) + get_finalization_grace_sec() + 60,
|
||||
)
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
stdout = exc.stdout if isinstance(exc.stdout, str) else (exc.stdout or b"").decode("utf-8", errors="replace")
|
||||
stderr = exc.stderr if isinstance(exc.stderr, str) else (exc.stderr or b"").decode("utf-8", errors="replace")
|
||||
if args.logs_dir:
|
||||
log_dir = Path(args.logs_dir).expanduser() / instance_id
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
(log_dir / "ouroboros.stdout").write_text(stdout, encoding="utf-8")
|
||||
(log_dir / "ouroboros.stderr").write_text(stderr, encoding="utf-8")
|
||||
error_row = {
|
||||
"instance_id": instance_id,
|
||||
"returncode": 124,
|
||||
"error": f"ouroboros run timed out after {int(args.timeout)}s",
|
||||
"timeout": True,
|
||||
}
|
||||
if not args.continue_on_error:
|
||||
raise RuntimeError(error_row["error"]) from exc
|
||||
errors.append(error_row)
|
||||
continue
|
||||
task_result: dict[str, Any] = {}
|
||||
if result_json_path.exists():
|
||||
try:
|
||||
loaded = json.loads(result_json_path.read_text(encoding="utf-8"))
|
||||
if isinstance(loaded, dict):
|
||||
task_result = loaded
|
||||
except Exception:
|
||||
task_result = {}
|
||||
if args.logs_dir:
|
||||
log_dir = Path(args.logs_dir).expanduser() / instance_id
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
(log_dir / "ouroboros.stdout").write_text(result.stdout, encoding="utf-8")
|
||||
(log_dir / "ouroboros.stderr").write_text(result.stderr, encoding="utf-8")
|
||||
(log_dir / "ouroboros-agent-result.json").write_text(json.dumps({
|
||||
"instance_id": instance_id,
|
||||
"returncode": result.returncode,
|
||||
"stdout_chars": len(result.stdout or ""),
|
||||
"stderr_chars": len(result.stderr or ""),
|
||||
"patch_empty": not bool((result.stdout or "").strip()),
|
||||
"timeout_sec": int(args.timeout),
|
||||
"outcome_axes": task_result.get("outcome_axes"),
|
||||
"reason_code": task_result.get("reason_code"),
|
||||
"artifact_bundle": task_result.get("artifact_bundle"),
|
||||
}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
if result.returncode != 0:
|
||||
details = (result.stderr or result.stdout or "").strip()
|
||||
if len(details) > 4000:
|
||||
details = details[:4000] + "\n...[truncated]"
|
||||
error_row = {
|
||||
"instance_id": instance_id,
|
||||
"returncode": result.returncode,
|
||||
"error": details or f"ouroboros run exited {result.returncode}",
|
||||
"outcome_axes": task_result.get("outcome_axes"),
|
||||
"reason_code": task_result.get("reason_code"),
|
||||
"artifact_bundle": task_result.get("artifact_bundle"),
|
||||
"trace_refs": task_result.get("trace_refs"),
|
||||
}
|
||||
if not args.continue_on_error:
|
||||
raise RuntimeError(error_row["error"])
|
||||
errors.append(error_row)
|
||||
continue
|
||||
if not (result.stdout or "").strip():
|
||||
error_row = {
|
||||
"instance_id": instance_id,
|
||||
"returncode": 0,
|
||||
"error": "ouroboros run produced no patch",
|
||||
"outcome_axes": task_result.get("outcome_axes"),
|
||||
"reason_code": task_result.get("reason_code") or "no_patch",
|
||||
"artifact_bundle": task_result.get("artifact_bundle"),
|
||||
"trace_refs": task_result.get("trace_refs"),
|
||||
}
|
||||
if not args.continue_on_error:
|
||||
raise RuntimeError(error_row["error"])
|
||||
errors.append(error_row)
|
||||
continue
|
||||
rows.append({
|
||||
"instance_id": instance_id,
|
||||
"model_name_or_path": args.model_name,
|
||||
"model_patch": result.stdout,
|
||||
})
|
||||
Path(args.output).write_text(
|
||||
"\n".join(json.dumps(row, ensure_ascii=False) for row in rows) + ("\n" if rows else ""),
|
||||
encoding="utf-8",
|
||||
)
|
||||
if errors:
|
||||
error_path = Path(args.errors_output).expanduser() if args.errors_output else Path(str(args.output) + ".errors.jsonl")
|
||||
error_path.write_text(
|
||||
"\n".join(json.dumps(row, ensure_ascii=False) for row in errors) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -1,186 +0,0 @@
|
|||
"""Terminal-Bench custom agent bridge for Ouroboros CLI."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import json
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
try: # Terminal-Bench is an optional benchmark dependency.
|
||||
from terminal_bench.agents.base_agent import AgentResult, BaseAgent
|
||||
from terminal_bench.agents.failure_mode import FailureMode
|
||||
except Exception: # pragma: no cover - exercised only when tbench is installed.
|
||||
AgentResult = None # type: ignore[assignment]
|
||||
BaseAgent = object # type: ignore[assignment]
|
||||
FailureMode = None # type: ignore[assignment]
|
||||
|
||||
|
||||
def _git_dirty_reason(workspace: Path) -> str:
|
||||
"""Return a failure reason for dirty git workspaces; allow non-git mounts."""
|
||||
|
||||
probe = subprocess.run(
|
||||
["git", "rev-parse", "--is-inside-work-tree"],
|
||||
cwd=workspace,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if probe.returncode != 0 or probe.stdout.strip().lower() != "true":
|
||||
return ""
|
||||
status = subprocess.run(
|
||||
["git", "status", "--porcelain"],
|
||||
cwd=workspace,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if status.returncode != 0:
|
||||
return "git_status_failed"
|
||||
if status.stdout.strip():
|
||||
return "dirty_git_workspace"
|
||||
return ""
|
||||
|
||||
|
||||
class OuroborosTerminalBenchAgent(BaseAgent): # type: ignore[misc, valid-type]
|
||||
"""Bridge Terminal-Bench to a mounted workspace served by Ouroboros CLI."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
workspace_root: str = "",
|
||||
model_name: str = "ouroboros-cli",
|
||||
timeout_sec: int = 7200,
|
||||
cli: str = "",
|
||||
**kwargs: Any,
|
||||
) -> None:
|
||||
try:
|
||||
super().__init__(**kwargs)
|
||||
except TypeError:
|
||||
super().__init__()
|
||||
self.workspace_root = workspace_root or os.environ.get("OUROBOROS_TBENCH_WORKSPACE_ROOT", "")
|
||||
self.model_name = model_name
|
||||
self.timeout_sec = int(timeout_sec)
|
||||
self.cli = cli or os.environ.get("OUROBOROS_CLI", "")
|
||||
|
||||
@staticmethod
|
||||
def name() -> str:
|
||||
return "Ouroboros CLI"
|
||||
|
||||
def perform_task(self, task_description: str, session: Any, logging_dir: Path | None = None) -> Any:
|
||||
workspace = Path(self.workspace_root).expanduser().resolve(strict=False) if self.workspace_root else None
|
||||
if workspace is None or not workspace.is_dir():
|
||||
if AgentResult is None or FailureMode is None:
|
||||
return {"success": False, "output": "workspace_root must point to the mounted Terminal-Bench task workspace"}
|
||||
return AgentResult(failure_mode=FailureMode.UNKNOWN_AGENT_ERROR)
|
||||
dirty_reason = _git_dirty_reason(workspace)
|
||||
if dirty_reason:
|
||||
summary = {
|
||||
"cmd": [],
|
||||
"workspace_root": str(workspace),
|
||||
"returncode": None,
|
||||
"stdout_chars": 0,
|
||||
"stderr_chars": 0,
|
||||
"timeout_sec": self.timeout_sec,
|
||||
"failure_mode": dirty_reason,
|
||||
}
|
||||
if logging_dir is not None:
|
||||
Path(logging_dir).mkdir(parents=True, exist_ok=True)
|
||||
(Path(logging_dir) / "ouroboros-agent-result.json").write_text(
|
||||
json.dumps(summary, ensure_ascii=False, indent=2),
|
||||
encoding="utf-8",
|
||||
)
|
||||
if AgentResult is None or FailureMode is None:
|
||||
return {"success": False, "output": f"refusing dirty git workspace: {dirty_reason}"}
|
||||
return AgentResult(failure_mode=FailureMode.UNKNOWN_AGENT_ERROR)
|
||||
prompt = self._render_instruction(task_description) if hasattr(self, "_render_instruction") else task_description
|
||||
cli_prefix = shlex.split(self.cli) if self.cli else [sys.executable, "-m", "ouroboros.cli"]
|
||||
cmd = [
|
||||
*cli_prefix,
|
||||
"run",
|
||||
"--workspace",
|
||||
str(workspace),
|
||||
"--memory-mode",
|
||||
"empty",
|
||||
"--timeout",
|
||||
str(self.timeout_sec),
|
||||
prompt,
|
||||
]
|
||||
try:
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=self.timeout_sec + 60)
|
||||
final = result.stdout.strip()
|
||||
summary = {
|
||||
"cmd": cmd,
|
||||
"workspace_root": str(workspace),
|
||||
"returncode": result.returncode,
|
||||
"stdout_chars": len(result.stdout or ""),
|
||||
"stderr_chars": len(result.stderr or ""),
|
||||
"final_text_empty": not bool(final),
|
||||
"timeout_sec": self.timeout_sec,
|
||||
"failure_mode": "",
|
||||
}
|
||||
if logging_dir is not None:
|
||||
Path(logging_dir).mkdir(parents=True, exist_ok=True)
|
||||
(Path(logging_dir) / "ouroboros.stdout").write_text(result.stdout, encoding="utf-8")
|
||||
(Path(logging_dir) / "ouroboros.stderr").write_text(result.stderr, encoding="utf-8")
|
||||
if result.returncode != 0:
|
||||
summary["failure_mode"] = "non_zero_exit"
|
||||
if logging_dir is not None:
|
||||
(Path(logging_dir) / "ouroboros-agent-result.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
if AgentResult is None or FailureMode is None:
|
||||
return {"success": False, "output": result.stderr or result.stdout or f"exit {result.returncode}"}
|
||||
return AgentResult(failure_mode=FailureMode.UNKNOWN_AGENT_ERROR)
|
||||
if not final:
|
||||
summary["failure_mode"] = "empty_final_text"
|
||||
if logging_dir is not None:
|
||||
(Path(logging_dir) / "ouroboros-agent-result.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
if AgentResult is None or FailureMode is None:
|
||||
return {"success": False, "output": "ouroboros run produced empty final text"}
|
||||
return AgentResult(failure_mode=FailureMode.UNKNOWN_AGENT_ERROR)
|
||||
if final:
|
||||
try:
|
||||
submit_command = f"/submit {shlex.quote(final)}"
|
||||
if hasattr(session, "send_keys"):
|
||||
session.send_keys([submit_command, "Enter"], block=True)
|
||||
elif hasattr(session, "send_command"):
|
||||
session.send_command(submit_command)
|
||||
elif hasattr(session, "run"):
|
||||
session.run(submit_command)
|
||||
else:
|
||||
raise RuntimeError("Terminal-Bench session has no supported submit method")
|
||||
except Exception as exc:
|
||||
summary["failure_mode"] = "submit_failed"
|
||||
summary["submit_error"] = f"{type(exc).__name__}: {exc}"
|
||||
if logging_dir is not None:
|
||||
(Path(logging_dir) / "ouroboros-agent-result.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
if AgentResult is None or FailureMode is None:
|
||||
return {"success": False, "output": summary["submit_error"]}
|
||||
return AgentResult(failure_mode=FailureMode.UNKNOWN_AGENT_ERROR)
|
||||
summary["failure_mode"] = "none"
|
||||
if logging_dir is not None:
|
||||
(Path(logging_dir) / "ouroboros-agent-result.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
if AgentResult is None or FailureMode is None:
|
||||
return {"success": True, "output": final}
|
||||
return AgentResult(failure_mode=FailureMode.NONE)
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
stdout = (exc.stdout or "").decode("utf-8", errors="replace") if isinstance(exc.stdout, bytes) else str(exc.stdout or "")
|
||||
stderr = (exc.stderr or "").decode("utf-8", errors="replace") if isinstance(exc.stderr, bytes) else str(exc.stderr or "")
|
||||
if logging_dir is not None:
|
||||
Path(logging_dir).mkdir(parents=True, exist_ok=True)
|
||||
(Path(logging_dir) / "ouroboros.stdout").write_text(stdout, encoding="utf-8")
|
||||
(Path(logging_dir) / "ouroboros.stderr").write_text(stderr, encoding="utf-8")
|
||||
(Path(logging_dir) / "ouroboros-agent-result.json").write_text(json.dumps({
|
||||
"cmd": cmd,
|
||||
"workspace_root": str(workspace),
|
||||
"returncode": 124,
|
||||
"stdout_chars": len(stdout),
|
||||
"stderr_chars": len(stderr),
|
||||
"timeout_sec": self.timeout_sec,
|
||||
"failure_mode": "timeout",
|
||||
}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
if AgentResult is None or FailureMode is None:
|
||||
return {"success": False, "output": f"ouroboros cli timed out after {self.timeout_sec}s", "timeout": True}
|
||||
return AgentResult(failure_mode=FailureMode.AGENT_TIMEOUT)
|
||||
|
||||
|
||||
__all__ = ["OuroborosTerminalBenchAgent"]
|
||||
12
server.py
12
server.py
|
|
@ -992,6 +992,11 @@ async def lifespan(app):
|
|||
kill_all_tracked_subprocesses()
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
from ouroboros.workspace_executor import kill_all_foreground
|
||||
kill_all_foreground(lifespan_drive_root)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
from ouroboros.tools.services import kill_all_services
|
||||
kill_all_services(lifespan_drive_root)
|
||||
|
|
@ -1056,9 +1061,14 @@ def _emergency_process_cleanup(*, port_sweep: bool = True) -> None:
|
|||
kill_all_tracked_subprocesses()
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
from ouroboros.workspace_executor import kill_all_foreground
|
||||
kill_all_foreground(DATA_DIR, wait=False)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
from ouroboros.tools.services import kill_all_services
|
||||
kill_all_services(wait=False)
|
||||
kill_all_services(DATA_DIR, wait=False)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -889,6 +889,7 @@ def test_ci_branch_filters_include_packaging_assets():
|
|||
workflow = _ci_workflow()
|
||||
|
||||
assert "- 'packaging/**'" in workflow
|
||||
assert "- 'devtools/**'" in workflow
|
||||
|
||||
|
||||
def test_ci_release_prerelease_flag_uses_preflight_output():
|
||||
|
|
|
|||
|
|
@ -978,3 +978,56 @@ def test_state_response_declares_runtime_and_capability_keys():
|
|||
f"StateResponse lost the runtime/capability key {required!r}; "
|
||||
"ARCHITECTURE.md §11.3 contract is out of sync."
|
||||
)
|
||||
|
||||
|
||||
def test_task_create_request_declares_executor_ref_contract():
|
||||
"""TaskCreateRequest pins executor_ref as the gateway-owned backend contract."""
|
||||
from ouroboros.gateway.contracts import ExecutorRef, TaskCreateRequest
|
||||
from ouroboros.workspace_executor import normalize_executor_ref
|
||||
|
||||
request_keys = set(TaskCreateRequest.__annotations__.keys())
|
||||
for required in (
|
||||
"description",
|
||||
"task_id",
|
||||
"type",
|
||||
"chat_id",
|
||||
"depth",
|
||||
"session_id",
|
||||
"workspace_root",
|
||||
"workspace_mode",
|
||||
"memory_mode",
|
||||
"attachments",
|
||||
"allowed_resources",
|
||||
"resource_policy",
|
||||
"executor_ref",
|
||||
"deadline_at",
|
||||
"timeout_sec",
|
||||
"timeout",
|
||||
"context",
|
||||
"expected_output",
|
||||
"constraints",
|
||||
"context_requires_self_body_docs",
|
||||
"actor_id",
|
||||
"source",
|
||||
"metadata",
|
||||
):
|
||||
assert required in request_keys
|
||||
assert TaskCreateRequest.__required_keys__ == frozenset({"description"})
|
||||
|
||||
executor_keys = set(ExecutorRef.__annotations__.keys())
|
||||
for required in ("type", "workspace_host_path", "workspace_backend_path", "network", "container_name", "path_mappings"):
|
||||
assert required in executor_keys
|
||||
assert normalize_executor_ref(
|
||||
{
|
||||
"type": "local",
|
||||
"path_mappings": [{"host_path": tempfile.gettempdir(), "backend_path": "/workspace"}],
|
||||
}
|
||||
)
|
||||
with pytest.raises(ValueError, match="requires container_name"):
|
||||
normalize_executor_ref(
|
||||
{
|
||||
"type": "docker_exec",
|
||||
"workspace_host_path": tempfile.gettempdir(),
|
||||
"workspace_backend_path": "/workspace",
|
||||
}
|
||||
)
|
||||
|
|
|
|||
671
tests/test_devtools_benchmarks.py
Normal file
671
tests/test_devtools_benchmarks.py
Normal file
|
|
@ -0,0 +1,671 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import io
|
||||
import importlib.util
|
||||
import json
|
||||
import shlex
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from devtools.benchmarks.common.official_commands import programbench_eval_cmd, swebench_eval_cmd
|
||||
from devtools.benchmarks.osworld.normalize_logs import normalize_bundle
|
||||
from devtools.benchmarks.programbench.programbench_adapter import (
|
||||
build_ouroboros_task_body,
|
||||
create_submission_tarball,
|
||||
preflight_cleanroom_container,
|
||||
)
|
||||
from devtools.benchmarks.swe_bench.presets import resolve_preset
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
_BASH_CAPTURE_AVAILABLE = sys.platform != "win32" and shutil.which("bash") is not None
|
||||
|
||||
|
||||
def _git_repo(path: Path) -> str:
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
subprocess.run(["git", "init"], cwd=path, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
subprocess.run(["git", "config", "user.email", "test@example.invalid"], cwd=path, check=True)
|
||||
subprocess.run(["git", "config", "user.name", "Test"], cwd=path, check=True)
|
||||
(path / "app.py").write_text("print('base')\n", encoding="utf-8")
|
||||
subprocess.run(["git", "add", "app.py"], cwd=path, check=True)
|
||||
subprocess.run(["git", "commit", "-m", "base"], cwd=path, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
return subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=path, text=True).strip()
|
||||
|
||||
|
||||
def test_runtime_core_does_not_import_devtools():
|
||||
runtime_paths = [REPO_ROOT / "ouroboros", REPO_ROOT / "server.py"]
|
||||
offenders: list[str] = []
|
||||
for root in runtime_paths:
|
||||
files = [root] if root.is_file() else sorted(root.rglob("*.py"))
|
||||
for path in files:
|
||||
text = path.read_text(encoding="utf-8", errors="ignore")
|
||||
if "import devtools" in text or "from devtools" in text:
|
||||
offenders.append(str(path.relative_to(REPO_ROOT)))
|
||||
assert not offenders
|
||||
|
||||
|
||||
def test_official_command_builders_do_not_replace_scoring():
|
||||
assert programbench_eval_cmd(Path("/runs/pb")) == ["programbench", "eval", "/runs/pb"]
|
||||
assert swebench_eval_cmd("princeton-nlp/SWE-bench_Verified", Path("/runs/predictions.jsonl"), "ouroboros", 2) == [
|
||||
"python",
|
||||
"-m",
|
||||
"swebench.harness.run_evaluation",
|
||||
"--dataset_name",
|
||||
"princeton-nlp/SWE-bench_Verified",
|
||||
"--predictions_path",
|
||||
"/runs/predictions.jsonl",
|
||||
"--max_workers",
|
||||
"2",
|
||||
"--run_id",
|
||||
"ouroboros",
|
||||
]
|
||||
|
||||
|
||||
def test_pyproject_does_not_package_devtools_runtime_assets():
|
||||
pyproject = (REPO_ROOT / "pyproject.toml").read_text(encoding="utf-8")
|
||||
assert '"devtools*"' not in pyproject
|
||||
assert "devtools = [" not in pyproject
|
||||
assert '"benchmarks/**/*.sh"' not in pyproject
|
||||
assert '"benchmarks/**/*.md"' not in pyproject
|
||||
|
||||
|
||||
def test_executable_devtools_entrypoints_support_direct_help():
|
||||
scripts = [
|
||||
"devtools/benchmarks/programbench/run_programbench.py",
|
||||
"devtools/benchmarks/terminal_bench/run_harbor_smoke.py",
|
||||
"devtools/benchmarks/swe_bench/swebench_predictions.py",
|
||||
"devtools/benchmarks/swe_bench_pro/grade_pro.py",
|
||||
"devtools/benchmarks/swe_bench_pro/pro_predictions.py",
|
||||
"devtools/benchmarks/osworld/normalize_logs.py",
|
||||
"devtools/benchmarks/osworld/osworld_adapter_skeleton.py",
|
||||
]
|
||||
for rel in scripts:
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(REPO_ROOT / rel), "--help"],
|
||||
cwd=REPO_ROOT,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
timeout=20,
|
||||
)
|
||||
assert proc.returncode == 0, f"{rel} failed:\nSTDOUT:\n{proc.stdout}\nSTDERR:\n{proc.stderr}"
|
||||
assert "usage:" in proc.stdout.lower()
|
||||
|
||||
|
||||
def test_programbench_task_body_sets_executor_and_protected_policy(tmp_path):
|
||||
workspace = tmp_path / "workspace"
|
||||
_git_repo(workspace)
|
||||
|
||||
body = build_ouroboros_task_body(
|
||||
instruction="solve",
|
||||
workspace_host_path=workspace,
|
||||
container_name="pb-cleanroom",
|
||||
protected_backend_paths=["/workspace/executable"],
|
||||
)
|
||||
|
||||
assert body["allowed_resources"] == {"web": False, "network": False, "internet": False}
|
||||
assert body["actor_id"] == "programbench"
|
||||
assert body["source"] == "programbench"
|
||||
assert "actor_id" not in body["metadata"]
|
||||
assert body["executor_ref"]["type"] == "docker_exec"
|
||||
assert body["executor_ref"]["network"] == "none"
|
||||
protected = body["resource_policy"]["protected_artifacts"][0]
|
||||
assert protected["role"] == "black_box_reference"
|
||||
assert protected["allow"] == ["execute"]
|
||||
assert {"read_bytes", "hash", "static_introspection", "dynamic_trace", "debug"} <= set(protected["deny"])
|
||||
|
||||
|
||||
def test_programbench_git_workspace_does_not_commit_protected_reference(tmp_path):
|
||||
workspace = tmp_path / "workspace"
|
||||
workspace.mkdir()
|
||||
(workspace / "executable").write_text("protected-bytes\n", encoding="utf-8")
|
||||
|
||||
build_ouroboros_task_body(
|
||||
instruction="solve",
|
||||
workspace_host_path=workspace,
|
||||
container_name="pb-cleanroom",
|
||||
protected_backend_paths=["/workspace/executable"],
|
||||
)
|
||||
|
||||
head = subprocess.run(["git", "rev-parse", "--verify", "HEAD"], cwd=workspace, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
show = subprocess.run(["git", "show", "HEAD:executable"], cwd=workspace, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
assert head.returncode != 0
|
||||
assert show.returncode != 0
|
||||
|
||||
|
||||
def test_programbench_submission_tarball_excludes_repo_noise(tmp_path):
|
||||
workspace = tmp_path / "workspace"
|
||||
(workspace / ".git").mkdir(parents=True)
|
||||
(workspace / ".git" / "HEAD").write_text("ref\n", encoding="utf-8")
|
||||
(workspace / ".ouroboros").mkdir()
|
||||
(workspace / ".ouroboros" / "trace.json").write_text("{}\n", encoding="utf-8")
|
||||
(workspace / "node_modules" / "pkg").mkdir(parents=True)
|
||||
(workspace / "node_modules" / "pkg" / "index.js").write_text("junk\n", encoding="utf-8")
|
||||
(workspace / "build").mkdir()
|
||||
(workspace / "build" / "out.o").write_text("junk\n", encoding="utf-8")
|
||||
(workspace / "dist").mkdir()
|
||||
(workspace / "dist" / "bundle.js").write_text("junk\n", encoding="utf-8")
|
||||
(workspace / "executable").write_text("protected\n", encoding="utf-8")
|
||||
(workspace / "solution.py").write_text("print('ok')\n", encoding="utf-8")
|
||||
|
||||
tar_path = create_submission_tarball(
|
||||
workspace,
|
||||
tmp_path / "submission.tar.gz",
|
||||
protected_paths=["/workspace/executable", "executable"],
|
||||
)
|
||||
|
||||
with tarfile.open(tar_path, "r:gz") as tar:
|
||||
names = set(tar.getnames())
|
||||
assert "solution.py" in names
|
||||
assert ".git/HEAD" not in names
|
||||
assert ".ouroboros/trace.json" not in names
|
||||
assert "node_modules/pkg/index.js" not in names
|
||||
assert "build/out.o" not in names
|
||||
assert "dist/bundle.js" not in names
|
||||
assert "executable" not in names
|
||||
|
||||
|
||||
def test_programbench_instance_path_stays_under_run_root(tmp_path):
|
||||
from devtools.benchmarks.common.run_roots import safe_join_under
|
||||
|
||||
root = tmp_path / "programbench-run"
|
||||
assert safe_join_under(root, "cheat/cheat") == root.resolve(strict=False) / "cheat" / "cheat"
|
||||
with pytest.raises(ValueError, match="escapes run root"):
|
||||
safe_join_under(root, "../escape")
|
||||
with pytest.raises(ValueError, match="escapes run root"):
|
||||
safe_join_under(root, "/tmp/escape")
|
||||
|
||||
|
||||
def test_programbench_cleanroom_preflight_requires_task_cleanroom_and_no_network(monkeypatch):
|
||||
calls = []
|
||||
|
||||
def fake_run(cmd, **kwargs):
|
||||
calls.append(cmd)
|
||||
return subprocess.CompletedProcess(
|
||||
cmd,
|
||||
0,
|
||||
stdout=json.dumps([
|
||||
{
|
||||
"Config": {"Image": "ghcr.io/facebookresearch/programbench/foo:task_cleanroom"},
|
||||
"HostConfig": {"NetworkMode": "none"},
|
||||
}
|
||||
]),
|
||||
stderr="",
|
||||
)
|
||||
|
||||
import devtools.benchmarks.programbench.programbench_adapter as adapter
|
||||
|
||||
monkeypatch.setattr(adapter.subprocess, "run", fake_run)
|
||||
assert preflight_cleanroom_container("pb") == {
|
||||
"image": "ghcr.io/facebookresearch/programbench/foo:task_cleanroom",
|
||||
"network": "none",
|
||||
}
|
||||
assert calls[0][:2] == ["docker", "inspect"]
|
||||
|
||||
|
||||
def test_swe_verified_preset_uses_official_dataset_name():
|
||||
assert resolve_preset("verified") == "princeton-nlp/SWE-bench_Verified"
|
||||
assert resolve_preset("SWE-bench/SWE-bench_Verified") == "princeton-nlp/SWE-bench_Verified"
|
||||
|
||||
|
||||
def test_terminal_bench_harbor_adapter_is_optional_import():
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"tb_harbor_adapter",
|
||||
REPO_ROOT / "devtools" / "benchmarks" / "terminal_bench" / "harbor_installed_agent.py",
|
||||
)
|
||||
assert spec and spec.loader
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
assert module.OuroborosTerminalBenchAgent.name() == "Ouroboros Installed"
|
||||
|
||||
|
||||
def test_terminal_bench_adapter_does_not_commit_target_workspace():
|
||||
adapter = (REPO_ROOT / "devtools" / "benchmarks" / "terminal_bench" / "harbor_installed_agent.py").read_text(encoding="utf-8")
|
||||
assert "git add -A" not in adapter
|
||||
assert "git commit --allow-empty" not in adapter
|
||||
|
||||
|
||||
def test_terminal_bench_source_copy_excludes_secret_shaped_files(tmp_path):
|
||||
import devtools.benchmarks.terminal_bench.harbor_installed_agent as tb_agent
|
||||
|
||||
source = tmp_path / "source"
|
||||
target = tmp_path / "target"
|
||||
source.mkdir()
|
||||
(source / "module.py").write_text("print('ok')\n", encoding="utf-8")
|
||||
secret_names = (
|
||||
".env",
|
||||
".env.example",
|
||||
".git-credentials",
|
||||
".netrc",
|
||||
".npmrc",
|
||||
".pypirc",
|
||||
"aws-credentials.json",
|
||||
"credentials.json",
|
||||
"gcp-service-account.json",
|
||||
"id_rsa",
|
||||
"repo.bundle",
|
||||
"repo_bundle_manifest.json",
|
||||
"secrets.json",
|
||||
"service-account.json",
|
||||
)
|
||||
for name in secret_names:
|
||||
(source / name).write_text("secret\n", encoding="utf-8")
|
||||
(source / "cert.pem").write_text("secret\n", encoding="utf-8")
|
||||
(source / "python-standalone").mkdir()
|
||||
(source / "python-standalone" / "python").write_text("binary\n", encoding="utf-8")
|
||||
|
||||
tb_agent._copy_clean_source(source, target)
|
||||
|
||||
assert (target / "module.py").exists()
|
||||
for name in (*secret_names, "cert.pem", "python-standalone"):
|
||||
assert not (target / name).exists()
|
||||
|
||||
|
||||
def test_terminal_bench_network_preflight_uses_configured_provider(tmp_path, monkeypatch):
|
||||
import devtools.benchmarks.terminal_bench.harbor_installed_agent as tb_agent
|
||||
|
||||
def fake_urlopen(req, timeout=0):
|
||||
raise urllib.error.HTTPError(req.full_url, 401, "Unauthorized", hdrs=None, fp=None)
|
||||
|
||||
monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen)
|
||||
|
||||
class Env:
|
||||
def __init__(self) -> None:
|
||||
self.command = ""
|
||||
|
||||
async def exec(self, *, command, timeout_sec=None, env=None, cwd=None):
|
||||
self.command = command
|
||||
script = command.split("python3 - <<'PY'\n", 1)[1].rsplit("\nPY", 1)[0]
|
||||
stdout = io.StringIO()
|
||||
code = 0
|
||||
try:
|
||||
with contextlib.redirect_stdout(stdout):
|
||||
exec(script, {})
|
||||
except SystemExit as exc:
|
||||
code = int(exc.code or 0)
|
||||
return SimpleNamespace(return_code=code, stdout=stdout.getvalue(), stderr="")
|
||||
|
||||
from types import SimpleNamespace
|
||||
|
||||
env = Env()
|
||||
agent = tb_agent.OuroborosTerminalBenchAgent(logs_dir=tmp_path)
|
||||
|
||||
asyncio.run(agent._network_preflight(env, {"OPENAI_API_KEY": "sk-test"}))
|
||||
|
||||
assert "api.openai.com" in env.command
|
||||
assert "openrouter.ai" not in env.command
|
||||
assert "urllib.error.HTTPError" in env.command
|
||||
assert "openai_preflight_status 401" in (tmp_path / "network-preflight.txt").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_terminal_bench_adapter_forwards_gigachat_and_preflights_direct_provider(tmp_path, monkeypatch):
|
||||
import devtools.benchmarks.terminal_bench.harbor_installed_agent as tb_agent
|
||||
|
||||
monkeypatch.setenv("OUROBOROS_BENCH_ALLOW_CONTAINER_SECRETS", "1")
|
||||
monkeypatch.setenv("GIGACHAT_CREDENTIALS", "gigachat-test-credentials")
|
||||
monkeypatch.setenv("GIGACHAT_BASE_URL", "https://gigachat.example.invalid/api/v1")
|
||||
|
||||
class Env:
|
||||
def __init__(self) -> None:
|
||||
self.command = ""
|
||||
|
||||
async def exec(self, *, command, timeout_sec=None, env=None, cwd=None):
|
||||
self.command = command
|
||||
script = command.split("python3 - <<'PY'\n", 1)[1].rsplit("\nPY", 1)[0]
|
||||
stdout = io.StringIO()
|
||||
code = 0
|
||||
try:
|
||||
with contextlib.redirect_stdout(stdout):
|
||||
exec(script, {})
|
||||
except SystemExit as exc:
|
||||
code = int(exc.code or 0)
|
||||
return SimpleNamespace(return_code=code, stdout=stdout.getvalue(), stderr="")
|
||||
|
||||
def fake_urlopen(req, timeout=0):
|
||||
raise urllib.error.HTTPError(req.full_url, 401, "Unauthorized", hdrs=None, fp=None)
|
||||
|
||||
monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen)
|
||||
agent = tb_agent.OuroborosTerminalBenchAgent(logs_dir=tmp_path)
|
||||
injected = agent._container_env()
|
||||
env = Env()
|
||||
|
||||
asyncio.run(agent._network_preflight(env, injected))
|
||||
|
||||
assert injected["GIGACHAT_CREDENTIALS"] == "gigachat-test-credentials"
|
||||
assert "gigachat.example.invalid/api/v1/models" in env.command
|
||||
assert "gigachat_preflight_status 401" in (tmp_path / "network-preflight.txt").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_terminal_bench_adapter_refuses_container_secret_injection_by_default(tmp_path, monkeypatch):
|
||||
import devtools.benchmarks.terminal_bench.harbor_installed_agent as tb_agent
|
||||
|
||||
monkeypatch.delenv("OUROBOROS_BENCH_ALLOW_CONTAINER_SECRETS", raising=False)
|
||||
monkeypatch.setenv("OPENROUTER_API_KEY", "sk-test-container-secret")
|
||||
agent = tb_agent.OuroborosTerminalBenchAgent(logs_dir=tmp_path)
|
||||
injected = agent._container_env()
|
||||
|
||||
assert "OPENROUTER_API_KEY" not in injected
|
||||
with pytest.raises(RuntimeError, match="refuses to inject long-lived provider credentials"):
|
||||
agent._enforce_container_secret_policy(injected)
|
||||
|
||||
|
||||
def test_terminal_bench_task_body_uses_top_level_actor_id():
|
||||
adapter = (REPO_ROOT / "devtools" / "benchmarks" / "terminal_bench" / "harbor_installed_agent.py").read_text(encoding="utf-8")
|
||||
assert '"actor_id": "harbor-terminal-bench"' in adapter
|
||||
assert '"metadata": {{"source": "terminal-bench", "delegation_role": "root"}}' in adapter
|
||||
assert '"metadata": {{"actor_id": "harbor-terminal-bench"' not in adapter
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _BASH_CAPTURE_AVAILABLE, reason="capture_patch.sh is a POSIX shell helper; Python wrappers are covered separately")
|
||||
def test_swe_pro_capture_keeps_untracked_text_and_drops_binary(tmp_path):
|
||||
repo = tmp_path / "repo"
|
||||
base = _git_repo(repo)
|
||||
(repo / "new_file.py").write_text("print('new')\n", encoding="utf-8")
|
||||
(repo / "binary.bin").write_bytes(b"\x00\x01\x02\x03")
|
||||
(repo / "build").mkdir()
|
||||
(repo / "build" / "out.txt").write_text("junk\n", encoding="utf-8")
|
||||
(repo / "dist").mkdir()
|
||||
(repo / "dist" / "out.txt").write_text("junk\n", encoding="utf-8")
|
||||
(repo / "app.py").write_text("print('changed')\n", encoding="utf-8")
|
||||
capture = REPO_ROOT / "devtools" / "benchmarks" / "swe_bench_pro" / "capture_patch.sh"
|
||||
out = tmp_path / "patch.diff"
|
||||
|
||||
subprocess.run(["bash", str(capture), str(repo), base, str(out)], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
patch = out.read_text(encoding="utf-8")
|
||||
|
||||
assert "new_file.py" in patch
|
||||
assert "app.py" in patch
|
||||
assert "binary.bin" not in patch
|
||||
assert "build/out.txt" not in patch
|
||||
assert "dist/out.txt" not in patch
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _BASH_CAPTURE_AVAILABLE, reason="capture_patch.sh is a POSIX shell helper; Python wrappers are covered separately")
|
||||
def test_swe_pro_capture_requires_valid_base_and_external_output(tmp_path):
|
||||
repo = tmp_path / "repo"
|
||||
base = _git_repo(repo)
|
||||
(repo / "app.py").write_text("print('changed')\n", encoding="utf-8")
|
||||
capture = REPO_ROOT / "devtools" / "benchmarks" / "swe_bench_pro" / "capture_patch.sh"
|
||||
|
||||
missing_output = subprocess.run(["bash", str(capture), str(repo), base], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
bad_base = subprocess.run(
|
||||
["bash", str(capture), str(repo), "not-a-commit", str(tmp_path / "bad.diff")],
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
)
|
||||
internal_output = REPO_ROOT / "devtools" / "should-not-write.diff"
|
||||
internal_dir = REPO_ROOT / "_test_rejected_capture_output_dir"
|
||||
nested_internal_output = internal_dir / "out.diff"
|
||||
shutil.rmtree(internal_dir, ignore_errors=True)
|
||||
try:
|
||||
repo_internal = subprocess.run(
|
||||
["bash", str(capture), str(repo), base, str(internal_output)],
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
)
|
||||
nested_repo_internal = subprocess.run(
|
||||
["bash", str(capture), str(repo), base, str(nested_internal_output)],
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
)
|
||||
finally:
|
||||
internal_output.unlink(missing_ok=True)
|
||||
shutil.rmtree(internal_dir, ignore_errors=True)
|
||||
|
||||
assert missing_output.returncode != 0
|
||||
assert bad_base.returncode != 0
|
||||
assert repo_internal.returncode != 0
|
||||
assert "outside the Ouroboros repo" in repo_internal.stderr
|
||||
assert nested_repo_internal.returncode != 0
|
||||
assert "outside the Ouroboros repo" in nested_repo_internal.stderr
|
||||
assert not internal_dir.exists()
|
||||
|
||||
|
||||
def test_swe_pro_grade_runs_official_eval_with_raw_sample(tmp_path, monkeypatch):
|
||||
import devtools.benchmarks.swe_bench_pro.grade_pro as grade_pro
|
||||
|
||||
eval_repo = tmp_path / "SWE-bench_Pro-os"
|
||||
helper = eval_repo / "helper_code"
|
||||
helper.mkdir(parents=True)
|
||||
raw_sample = helper / "sweap_eval_full_v2.jsonl"
|
||||
raw_sample.write_text(json.dumps({"instance_id": "x", "FAIL_TO_PASS": [], "PASS_TO_PASS": []}) + "\n", encoding="utf-8")
|
||||
predictions = tmp_path / "predictions.jsonl"
|
||||
predictions.write_text(json.dumps({"instance_id": "x", "model_patch": "diff --git a/a b/a\n", "model_name_or_path": "m"}) + "\n", encoding="utf-8")
|
||||
captured: dict[str, object] = {}
|
||||
|
||||
def fake_run(cmd, **kwargs):
|
||||
captured["cmd"] = list(cmd)
|
||||
captured["cwd"] = kwargs.get("cwd")
|
||||
return subprocess.CompletedProcess(cmd, 0, stdout="", stderr="")
|
||||
|
||||
monkeypatch.setattr(grade_pro.subprocess, "run", fake_run)
|
||||
monkeypatch.setattr(
|
||||
sys,
|
||||
"argv",
|
||||
[
|
||||
"grade_pro.py",
|
||||
"--predictions",
|
||||
str(predictions),
|
||||
"--out-dir",
|
||||
str(tmp_path / "out"),
|
||||
"--eval-repo",
|
||||
str(eval_repo),
|
||||
],
|
||||
)
|
||||
|
||||
assert grade_pro.main() == 0
|
||||
assert "--raw_sample_path" in captured["cmd"]
|
||||
assert str(raw_sample) in captured["cmd"]
|
||||
assert captured["cwd"] == str(eval_repo)
|
||||
|
||||
|
||||
def test_swe_pro_grade_rejects_repo_internal_output(tmp_path, monkeypatch):
|
||||
import devtools.benchmarks.swe_bench_pro.grade_pro as grade_pro
|
||||
|
||||
eval_repo = tmp_path / "SWE-bench_Pro-os"
|
||||
helper = eval_repo / "helper_code"
|
||||
helper.mkdir(parents=True)
|
||||
raw_sample = helper / "sweap_eval_full_v2.jsonl"
|
||||
raw_sample.write_text(json.dumps({"instance_id": "x", "FAIL_TO_PASS": [], "PASS_TO_PASS": []}) + "\n", encoding="utf-8")
|
||||
predictions = tmp_path / "predictions.jsonl"
|
||||
predictions.write_text(json.dumps({"instance_id": "x", "model_patch": "diff --git a/a b/a\n", "model_name_or_path": "m"}) + "\n", encoding="utf-8")
|
||||
internal_out = REPO_ROOT / "_test_rejected_grade_output_dir"
|
||||
shutil.rmtree(internal_out, ignore_errors=True)
|
||||
monkeypatch.setattr(
|
||||
sys,
|
||||
"argv",
|
||||
[
|
||||
"grade_pro.py",
|
||||
"--predictions",
|
||||
str(predictions),
|
||||
"--out-dir",
|
||||
str(internal_out),
|
||||
"--eval-repo",
|
||||
str(eval_repo),
|
||||
"--skip-run",
|
||||
],
|
||||
)
|
||||
try:
|
||||
with pytest.raises(ValueError, match="under repo"):
|
||||
grade_pro.main()
|
||||
assert not internal_out.exists()
|
||||
finally:
|
||||
shutil.rmtree(internal_out, ignore_errors=True)
|
||||
|
||||
|
||||
def test_swe_pro_prediction_capture_rejects_empty_patch(tmp_path, monkeypatch):
|
||||
import devtools.benchmarks.swe_bench_pro.pro_predictions as pro_predictions
|
||||
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
out = tmp_path / "empty.diff"
|
||||
|
||||
def fake_run(cmd, **kwargs):
|
||||
out.write_text("", encoding="utf-8")
|
||||
return subprocess.CompletedProcess(cmd, 0, stdout="", stderr="")
|
||||
|
||||
monkeypatch.setattr(pro_predictions.subprocess, "run", fake_run)
|
||||
|
||||
with pytest.raises(RuntimeError, match="empty patch"):
|
||||
pro_predictions._capture_patch(repo, "HEAD", out)
|
||||
|
||||
|
||||
def test_swe_predictions_rejects_unsafe_instance_id_before_logs_escape(tmp_path, monkeypatch):
|
||||
import devtools.benchmarks.swe_bench.swebench_predictions as swe_predictions
|
||||
|
||||
input_jsonl = tmp_path / "instances.jsonl"
|
||||
output_jsonl = tmp_path / "predictions.jsonl"
|
||||
logs_dir = tmp_path / "logs"
|
||||
input_jsonl.write_text(
|
||||
json.dumps({"instance_id": "../escape", "workspace_root": "/missing", "problem_statement": "fix"}) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
sys,
|
||||
"argv",
|
||||
[
|
||||
"swebench_predictions.py",
|
||||
"--input",
|
||||
str(input_jsonl),
|
||||
"--output",
|
||||
str(output_jsonl),
|
||||
"--logs-dir",
|
||||
str(logs_dir),
|
||||
"--continue-on-error",
|
||||
],
|
||||
)
|
||||
|
||||
assert swe_predictions.main() == 0
|
||||
errors = json.loads((tmp_path / "predictions.jsonl.errors.jsonl").read_text(encoding="utf-8").splitlines()[0])
|
||||
assert errors["reason_code"] == "invalid_instance_id"
|
||||
assert not (tmp_path / "escape").exists()
|
||||
|
||||
|
||||
def test_swe_pro_predictions_rejects_unsafe_instance_id_before_patch_path(tmp_path, monkeypatch):
|
||||
import devtools.benchmarks.swe_bench_pro.pro_predictions as pro_predictions
|
||||
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
input_jsonl = tmp_path / "instances.jsonl"
|
||||
output_jsonl = tmp_path / "predictions.jsonl"
|
||||
patch_dir = tmp_path / "patches"
|
||||
input_jsonl.write_text(
|
||||
json.dumps({"instance_id": "../escape", "repo_dir": str(repo), "base_commit": "HEAD"}) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
monkeypatch.setattr(pro_predictions, "_capture_patch", lambda *a, **k: pytest.fail("unsafe id should fail before capture"))
|
||||
monkeypatch.setattr(
|
||||
sys,
|
||||
"argv",
|
||||
[
|
||||
"pro_predictions.py",
|
||||
"--input",
|
||||
str(input_jsonl),
|
||||
"--output",
|
||||
str(output_jsonl),
|
||||
"--patch-dir",
|
||||
str(patch_dir),
|
||||
],
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="single safe path component"):
|
||||
pro_predictions.main()
|
||||
assert not (tmp_path / "escape").exists()
|
||||
|
||||
|
||||
def test_benchmark_output_helpers_reject_repo_internal_outputs(tmp_path, monkeypatch):
|
||||
import devtools.benchmarks.swe_bench.swebench_predictions as swe_predictions
|
||||
import devtools.benchmarks.terminal_bench.run_harbor_smoke as harbor_smoke
|
||||
|
||||
input_jsonl = tmp_path / "instances.jsonl"
|
||||
input_jsonl.write_text("", encoding="utf-8")
|
||||
|
||||
monkeypatch.setattr(sys, "argv", ["swebench_predictions.py", "--input", str(input_jsonl), "--output", str(REPO_ROOT / "devtools" / "bad.jsonl")])
|
||||
with pytest.raises(ValueError, match="benchmark run output must not be under repo"):
|
||||
swe_predictions.main()
|
||||
|
||||
monkeypatch.setattr(sys, "argv", ["run_harbor_smoke.py", "--run-root", str(REPO_ROOT / "devtools" / "bad_run")])
|
||||
with pytest.raises(ValueError, match="benchmark run output must not be under repo"):
|
||||
harbor_smoke.main()
|
||||
|
||||
|
||||
def test_osworld_logs_only_normalizer(tmp_path):
|
||||
bundle = tmp_path / "osworld_logs"
|
||||
(bundle / "sample1").mkdir(parents=True)
|
||||
(bundle / "SUMMARY.json").write_text(json.dumps({"count": 1}), encoding="utf-8")
|
||||
(bundle / "sample_manifest.json").write_text(json.dumps({"samples": ["sample1"]}), encoding="utf-8")
|
||||
(bundle / "trace_manifest.json").write_text(json.dumps({"traces": ["sample1/traj.jsonl"]}), encoding="utf-8")
|
||||
(bundle / "sample1" / "traj.jsonl").write_text(
|
||||
json.dumps({"type": "start"}) + "\n" + json.dumps({"type": "end"}) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
normalized = normalize_bundle(bundle)
|
||||
|
||||
assert normalized["traj_count"] == 1
|
||||
assert normalized["traces"][0]["events"] == 2
|
||||
assert normalized["traces"][0]["last_type"] == "end"
|
||||
|
||||
|
||||
def test_osworld_logs_only_normalizer_accepts_nested_trace_manifests(tmp_path):
|
||||
bundle = tmp_path / "osworld_logs"
|
||||
sample = bundle / "chrome" / "sample1"
|
||||
(sample / "traces").mkdir(parents=True)
|
||||
(bundle / "SUMMARY.json").write_text(json.dumps({"count": 1}), encoding="utf-8")
|
||||
(bundle / "sample_manifest.json").write_text(json.dumps({"samples": ["sample1"]}), encoding="utf-8")
|
||||
(sample / "traces" / "trace_manifest.json").write_text(json.dumps({"trace": "sample1"}), encoding="utf-8")
|
||||
(sample / "traj.jsonl").write_text(json.dumps({"event": "done"}) + "\n", encoding="utf-8")
|
||||
|
||||
normalized = normalize_bundle(bundle)
|
||||
|
||||
assert normalized["trace_manifest"]["trace_manifest_paths"] == ["chrome/sample1/traces/trace_manifest.json"]
|
||||
assert normalized["traj_count"] == 1
|
||||
|
||||
|
||||
def test_terminal_bench_adapter_quotes_hostile_workspace_dir(tmp_path):
|
||||
from devtools.benchmarks.terminal_bench.harbor_installed_agent import OuroborosTerminalBenchAgent
|
||||
|
||||
class FakeResult:
|
||||
return_code = 0
|
||||
stdout = '{"return_code": 0}\n'
|
||||
stderr = ""
|
||||
|
||||
class FakeEnvironment:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
|
||||
async def exec(self, **kwargs):
|
||||
self.calls.append(kwargs)
|
||||
return FakeResult()
|
||||
|
||||
hostile = "/tmp/ws'; touch /tmp/pwn; echo '"
|
||||
agent = OuroborosTerminalBenchAgent(logs_dir=tmp_path, workspace_dir=hostile)
|
||||
environment = FakeEnvironment()
|
||||
|
||||
asyncio.run(agent._resolve_workspace_dir(environment))
|
||||
asyncio.run(agent._ensure_workspace_git_root(environment))
|
||||
summary = asyncio.run(agent._run_ouroboros_task(environment, {}))
|
||||
|
||||
assert summary["return_code"] == 0
|
||||
quoted = shlex.quote(hostile)
|
||||
assert environment.calls[0]["command"] == f"test -d {quoted}"
|
||||
git_command = environment.calls[1]["command"]
|
||||
assert f"workspace_dir={quoted}" in git_command
|
||||
assert "cd \"$workspace_dir\"" in git_command
|
||||
runner_command = environment.calls[-1]["command"]
|
||||
runner = runner_command.split("cat > /tmp/run_ouroboros_task.py <<'PY'\n", 1)[1].split("\nPY\n", 1)[0]
|
||||
assert f'"workspace_root": {json.dumps(hostile)}' in runner
|
||||
compile(runner, "run_ouroboros_task.py", "exec")
|
||||
|
|
@ -145,6 +145,20 @@ def test_task_api_enqueue_workspace_creates_child_drive(tmp_path, monkeypatch):
|
|||
assert "target workspace, not the Ouroboros system repo" in captured[0]["text"]
|
||||
|
||||
|
||||
def test_api_tasks_create_requires_description_not_legacy_aliases(monkeypatch):
|
||||
captured = []
|
||||
monkeypatch.setattr("supervisor.queue.enqueue_task", lambda task: captured.append(task) or task)
|
||||
app = Starlette(routes=[Route("/api/tasks", endpoint=api_tasks_create, methods=["POST"])])
|
||||
client = TestClient(app)
|
||||
|
||||
for payload in ({"text": "legacy task"}, {"prompt": "legacy task"}, {"description": ""}):
|
||||
response = client.post("/api/tasks", json=payload)
|
||||
assert response.status_code == 400, (payload, response.text)
|
||||
assert "description is required" in response.json().get("error", "")
|
||||
|
||||
assert captured == []
|
||||
|
||||
|
||||
def test_api_tasks_create_rejects_internal_task_types(tmp_path, monkeypatch):
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
|
|
@ -1820,6 +1834,9 @@ def test_cli_run_actor_id_is_sent_as_gateway_root_field(monkeypatch, capsys):
|
|||
assert cli.main(["run", "--detach", "--timeout", "7", "--actor-id", "operator-1", "hello"]) == 0
|
||||
assert captured["method"] == "POST"
|
||||
assert captured["path"] == "/api/tasks"
|
||||
assert captured["body"]["description"] == "hello"
|
||||
assert "text" not in captured["body"]
|
||||
assert "prompt" not in captured["body"]
|
||||
assert captured["body"]["actor_id"] == "operator-1"
|
||||
assert captured["body"]["timeout_sec"] == 7.0
|
||||
assert captured["body"]["source"] == "cli"
|
||||
|
|
@ -1879,8 +1896,14 @@ def test_cli_wait_task_caps_poll_request_by_timeout(monkeypatch):
|
|||
|
||||
|
||||
def test_swebench_helper_records_cli_timeout_with_continue(tmp_path, monkeypatch):
|
||||
script_path = pathlib.Path(__file__).resolve().parent.parent / "scripts" / "swebench_cli_agent.py"
|
||||
spec = importlib.util.spec_from_file_location("swebench_cli_agent_test", script_path)
|
||||
script_path = (
|
||||
pathlib.Path(__file__).resolve().parent.parent
|
||||
/ "devtools"
|
||||
/ "benchmarks"
|
||||
/ "swe_bench"
|
||||
/ "swebench_predictions.py"
|
||||
)
|
||||
spec = importlib.util.spec_from_file_location("swebench_predictions_test", script_path)
|
||||
assert spec and spec.loader
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
|
|
@ -1911,7 +1934,7 @@ def test_swebench_helper_records_cli_timeout_with_continue(tmp_path, monkeypatch
|
|||
sys,
|
||||
"argv",
|
||||
[
|
||||
"swebench_cli_agent.py",
|
||||
"swebench_predictions.py",
|
||||
"--input",
|
||||
str(rows_path),
|
||||
"--output",
|
||||
|
|
@ -1932,29 +1955,21 @@ def test_swebench_helper_records_cli_timeout_with_continue(tmp_path, monkeypatch
|
|||
assert (logs_dir / "inst1" / "ouroboros.stderr").read_text(encoding="utf-8") == "partial-err"
|
||||
|
||||
|
||||
def test_terminal_bench_helper_refuses_dirty_git_workspace(tmp_path):
|
||||
script_path = pathlib.Path(__file__).resolve().parent.parent / "scripts" / "terminal_bench_cli_agent.py"
|
||||
spec = importlib.util.spec_from_file_location("terminal_bench_cli_agent_test", script_path)
|
||||
def test_terminal_bench_harbor_adapter_imports_without_harbor():
|
||||
script_path = (
|
||||
pathlib.Path(__file__).resolve().parent.parent
|
||||
/ "devtools"
|
||||
/ "benchmarks"
|
||||
/ "terminal_bench"
|
||||
/ "harbor_installed_agent.py"
|
||||
)
|
||||
spec = importlib.util.spec_from_file_location("terminal_bench_harbor_adapter_test", script_path)
|
||||
assert spec and spec.loader
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
|
||||
workspace = tmp_path / "workspace"
|
||||
_init_repo_with_file(workspace)
|
||||
(workspace / "tracked.txt").write_text("dirty\n", encoding="utf-8")
|
||||
logs_dir = tmp_path / "logs"
|
||||
|
||||
agent = module.OuroborosTerminalBenchAgent(
|
||||
workspace_root=str(workspace),
|
||||
cli=f"{sys.executable} -c 'raise SystemExit(99)'",
|
||||
)
|
||||
result = agent.perform_task("fix", SimpleNamespace(), logging_dir=logs_dir)
|
||||
|
||||
if isinstance(result, dict):
|
||||
assert result["success"] is False
|
||||
assert "dirty_git_workspace" in result["output"]
|
||||
summary = json.loads((logs_dir / "ouroboros-agent-result.json").read_text(encoding="utf-8"))
|
||||
assert summary["failure_mode"] == "dirty_git_workspace"
|
||||
assert module.OuroborosTerminalBenchAgent.name() == "Ouroboros Installed"
|
||||
assert module._repo_root() == pathlib.Path(__file__).resolve().parent.parent
|
||||
|
||||
|
||||
def test_queue_restore_accepts_headless_chat_zero(tmp_path, monkeypatch):
|
||||
|
|
|
|||
|
|
@ -267,9 +267,9 @@ def test_advisory_pre_review_timeout_1200():
|
|||
|
||||
|
||||
def test_full_repo_pack_excludes_junk_dirs():
|
||||
"""build_full_repo_pack must skip non-agent-logic directories (assets/, tests/)."""
|
||||
"""build_full_repo_pack must skip broad non-core directories."""
|
||||
from ouroboros.tools.review_helpers import _FULL_REPO_SKIP_DIR_PREFIXES
|
||||
for prefix in ("assets/", "tests/"):
|
||||
for prefix in ("assets/", "tests/", "devtools/"):
|
||||
assert prefix in _FULL_REPO_SKIP_DIR_PREFIXES, f"{prefix} not in skip list"
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -139,6 +139,42 @@ def test_atlas_force_includes_protected_workflow_even_under_skipped_github_dir(t
|
|||
assert "asset text" not in pack.text
|
||||
|
||||
|
||||
def test_atlas_devtools_manifest_only_unless_touched(tmp_path):
|
||||
_write(tmp_path / "devtools" / "benchmarks" / "programbench" / "run.py", "VALUE = 'devtools full text'\n")
|
||||
_write(tmp_path / "ouroboros" / "core.py", "print('core')\n")
|
||||
|
||||
pack = compile_review_context_atlas(
|
||||
ReviewContextAtlasRequest(
|
||||
repo_dir=tmp_path,
|
||||
tracked_paths=("devtools/benchmarks/programbench/run.py", "ouroboros/core.py"),
|
||||
anchors=("ouroboros/core.py",),
|
||||
fixed_prompt_tokens=100,
|
||||
target_total_tokens=20_000,
|
||||
hard_total_tokens=25_000,
|
||||
)
|
||||
)
|
||||
|
||||
coverage = _coverage(pack)
|
||||
assert coverage["devtools/benchmarks/programbench/run.py"]["disposition"] == "excluded_dir"
|
||||
assert "devtools full text" not in pack.text
|
||||
assert coverage["ouroboros/core.py"]["disposition"] == "full"
|
||||
|
||||
touched = compile_review_context_atlas(
|
||||
ReviewContextAtlasRequest(
|
||||
repo_dir=tmp_path,
|
||||
tracked_paths=("devtools/benchmarks/programbench/run.py",),
|
||||
anchors=("devtools/benchmarks/programbench/run.py",),
|
||||
fixed_prompt_tokens=100,
|
||||
target_total_tokens=20_000,
|
||||
hard_total_tokens=25_000,
|
||||
)
|
||||
)
|
||||
|
||||
touched_coverage = _coverage(touched)
|
||||
assert touched_coverage["devtools/benchmarks/programbench/run.py"]["disposition"] == "full"
|
||||
assert "devtools full text" in touched.text
|
||||
|
||||
|
||||
def test_atlas_marks_sensitive_binary_oversized_and_vendored_files(tmp_path):
|
||||
_write(tmp_path / ".env.example", "TOKEN=secret\n")
|
||||
(tmp_path / "image.png").write_bytes(b"\x89PNG\r\n\x00")
|
||||
|
|
|
|||
|
|
@ -1,6 +1,18 @@
|
|||
from types import SimpleNamespace
|
||||
|
||||
|
||||
def test_lifespan_shutdown_kills_executor_foreground_before_services():
|
||||
import inspect
|
||||
import server
|
||||
|
||||
source = inspect.getsource(server.lifespan)
|
||||
|
||||
shell_idx = source.index("kill_all_tracked_subprocesses()")
|
||||
foreground_idx = source.index("kill_all_foreground(lifespan_drive_root)")
|
||||
service_idx = source.index("kill_all_services(lifespan_drive_root)")
|
||||
assert shell_idx < foreground_idx < service_idx
|
||||
|
||||
|
||||
def test_shutdown_task_cleanup_args_never_reports_crash_storm():
|
||||
"""Graceful shutdown (requested restart or external signal) must finalize a
|
||||
running task as cancelled/interrupted, never as a worker crash storm."""
|
||||
|
|
@ -87,10 +99,12 @@ def test_main_graceful_restart_cleanup_avoids_port_sweep(monkeypatch):
|
|||
def test_emergency_cleanup_kills_services_without_log_finalization(monkeypatch):
|
||||
import server
|
||||
|
||||
foreground_calls = []
|
||||
service_calls = []
|
||||
worker_calls = []
|
||||
|
||||
monkeypatch.setattr("ouroboros.tools.shell.kill_all_tracked_subprocesses", lambda: None)
|
||||
monkeypatch.setattr("ouroboros.workspace_executor.kill_all_foreground", lambda *a, **k: foreground_calls.append((a, k)))
|
||||
monkeypatch.setattr("ouroboros.tools.services.kill_all_services", lambda *a, **k: service_calls.append((a, k)))
|
||||
monkeypatch.setattr("supervisor.workers.kill_workers", lambda **kw: worker_calls.append(kw))
|
||||
monkeypatch.setattr("multiprocessing.active_children", lambda: [])
|
||||
|
|
@ -101,7 +115,8 @@ def test_emergency_cleanup_kills_services_without_log_finalization(monkeypatch):
|
|||
|
||||
server._emergency_process_cleanup(port_sweep=False)
|
||||
|
||||
assert service_calls == [((), {"wait": False})]
|
||||
assert foreground_calls == [((server.DATA_DIR,), {"wait": False})]
|
||||
assert service_calls == [((server.DATA_DIR,), {"wait": False})]
|
||||
assert worker_calls == [{"force": True, "archive_service_logs": False}]
|
||||
|
||||
|
||||
|
|
@ -113,6 +128,7 @@ def test_emergency_cleanup_during_restart_marks_tasks_cancelled(monkeypatch):
|
|||
worker_calls = []
|
||||
|
||||
monkeypatch.setattr("ouroboros.tools.shell.kill_all_tracked_subprocesses", lambda: None)
|
||||
monkeypatch.setattr("ouroboros.workspace_executor.kill_all_foreground", lambda *a, **k: None)
|
||||
monkeypatch.setattr("ouroboros.tools.services.kill_all_services", lambda *a, **k: None)
|
||||
monkeypatch.setattr("supervisor.workers.kill_workers", lambda **kw: worker_calls.append(kw))
|
||||
monkeypatch.setattr("multiprocessing.active_children", lambda: [])
|
||||
|
|
@ -137,6 +153,7 @@ def test_emergency_cleanup_during_restart_marks_tasks_cancelled(monkeypatch):
|
|||
def test_panic_stop_kills_services_without_log_finalization(monkeypatch, tmp_path):
|
||||
from ouroboros import server_control
|
||||
|
||||
foreground_calls = []
|
||||
service_calls = []
|
||||
worker_calls = []
|
||||
|
||||
|
|
@ -144,6 +161,7 @@ def test_panic_stop_kills_services_without_log_finalization(monkeypatch, tmp_pat
|
|||
pass
|
||||
|
||||
monkeypatch.setattr("ouroboros.tools.shell.kill_all_tracked_subprocesses", lambda: None)
|
||||
monkeypatch.setattr("ouroboros.workspace_executor.kill_all_foreground", lambda *a, **k: foreground_calls.append((a, k)))
|
||||
monkeypatch.setattr("ouroboros.tools.services.kill_all_services", lambda *a, **k: service_calls.append((a, k)))
|
||||
monkeypatch.setattr("ouroboros.local_model.get_manager", lambda: SimpleNamespace(stop_server=lambda: None))
|
||||
monkeypatch.setattr("supervisor.state.load_state", lambda: {})
|
||||
|
|
@ -165,5 +183,6 @@ def test_panic_stop_kills_services_without_log_finalization(monkeypatch, tmp_pat
|
|||
except ExitCalled:
|
||||
pass
|
||||
|
||||
assert service_calls == [((), {"wait": False})]
|
||||
assert foreground_calls == [((tmp_path,), {"wait": False})]
|
||||
assert service_calls == [((tmp_path,), {"wait": False})]
|
||||
assert worker_calls == [{"force": True, "archive_service_logs": False}]
|
||||
|
|
|
|||
|
|
@ -440,7 +440,7 @@ def test_no_bare_except_pass():
|
|||
# ── AST-based function size check ───────────────────────────────
|
||||
|
||||
_SKIP_DIRS = {'.git', '__pycache__', 'tests', 'python-standalone', 'build', 'dist',
|
||||
'venv', '.venv', 'node_modules', 'assets', '.pytest_cache'}
|
||||
'venv', '.venv', 'node_modules', 'assets', 'devtools', '.pytest_cache'}
|
||||
|
||||
|
||||
def _get_function_sizes():
|
||||
|
|
|
|||
|
|
@ -9,7 +9,9 @@ Verifies:
|
|||
- search_code tool works
|
||||
"""
|
||||
import inspect
|
||||
import json
|
||||
import os
|
||||
import base64
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
|
@ -569,6 +571,9 @@ def test_protected_black_box_artifact_policy_blocks_introspection(tmp_path, monk
|
|||
direct_cmd = [str(protected)]
|
||||
protected.write_text("#!/bin/sh\nprintf 'reference\\n'\n", encoding="utf-8")
|
||||
generated.write_text("#!/bin/sh\nprintf 'generated\\n'\n", encoding="utf-8")
|
||||
protected_dir = repo / "protected_dir"
|
||||
protected_dir.mkdir()
|
||||
(protected_dir / "secret.txt").write_text("secret\n", encoding="utf-8")
|
||||
protected.chmod(0o755)
|
||||
generated.chmod(0o755)
|
||||
task_contract = build_task_contract({
|
||||
|
|
@ -580,6 +585,12 @@ def test_protected_black_box_artifact_policy_blocks_introspection(tmp_path, monk
|
|||
"paths": [str(protected)],
|
||||
"allow": ["execute"],
|
||||
"deny": ["read_bytes", "copy", "hash", "static_introspection", "dynamic_trace", "debug"],
|
||||
},
|
||||
{
|
||||
"id": "reference-dir",
|
||||
"role": "black_box_reference",
|
||||
"paths": [str(protected_dir)],
|
||||
"allow": ["execute"],
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -597,6 +608,78 @@ def test_protected_black_box_artifact_policy_blocks_introspection(tmp_path, monk
|
|||
assert "RESOURCE_POLICY_BLOCKED" not in direct
|
||||
assert "reference" in direct
|
||||
assert "RESOURCE_POLICY_BLOCKED" in registry.execute("read_file", {"path": protected.name})
|
||||
protected_content = protected.read_text(encoding="utf-8")
|
||||
write_attempt = registry.execute("write_file", {"path": protected.name, "content": "tamper\n"})
|
||||
assert "RESOURCE_POLICY_BLOCKED" in write_attempt
|
||||
assert protected.read_text(encoding="utf-8") == protected_content
|
||||
edit_attempt = registry.execute("edit_text", {"path": protected.name, "old_str": "reference", "new_str": "tamper"})
|
||||
assert "RESOURCE_POLICY_BLOCKED" in edit_attempt
|
||||
assert protected.read_text(encoding="utf-8") == protected_content
|
||||
shell_write_attempt = registry.execute(
|
||||
"run_command",
|
||||
{"cmd": ["sh", "-c", f"printf tamper > {protected.name}"], "cwd": str(repo)},
|
||||
)
|
||||
assert "RESOURCE_POLICY_BLOCKED" in shell_write_attempt
|
||||
assert protected.read_text(encoding="utf-8") == protected_content
|
||||
shell_delete_attempt = registry.execute(
|
||||
"run_command",
|
||||
{"cmd": ["rm", protected.name], "cwd": str(repo)},
|
||||
)
|
||||
assert "RESOURCE_POLICY_BLOCKED" in shell_delete_attempt
|
||||
assert protected.exists()
|
||||
recursive_delete_attempt = registry.execute(
|
||||
"run_command",
|
||||
{"cmd": ["rm", "-rf", "."], "cwd": str(repo)},
|
||||
)
|
||||
assert "RESOURCE_POLICY_BLOCKED" in recursive_delete_attempt
|
||||
assert protected.exists()
|
||||
glob_delete_attempt = registry.execute(
|
||||
"run_command",
|
||||
{"cmd": ["sh", "-c", "rm -rf *"], "cwd": str(repo)},
|
||||
)
|
||||
assert "RESOURCE_POLICY_BLOCKED" in glob_delete_attempt
|
||||
assert protected.exists()
|
||||
glob_read_attempt = registry.execute(
|
||||
"run_command",
|
||||
{"cmd": ["sh", "-c", "cat *"], "cwd": str(repo)},
|
||||
)
|
||||
assert "RESOURCE_POLICY_BLOCKED" in glob_read_attempt
|
||||
find_exec_read = registry.execute(
|
||||
"run_command",
|
||||
{"cmd": ["find", ".", "-type", "f", "-exec", "cat", "{}", "+"], "cwd": str(repo)},
|
||||
)
|
||||
assert "RESOURCE_POLICY_BLOCKED" in find_exec_read
|
||||
find_delete = registry.execute(
|
||||
"run_command",
|
||||
{"cmd": ["find", ".", "-delete"], "cwd": str(repo)},
|
||||
)
|
||||
assert "RESOURCE_POLICY_BLOCKED" in find_delete
|
||||
assert protected.exists()
|
||||
pathless_find_exec_read = registry.execute(
|
||||
"run_command",
|
||||
{"cmd": ["find", "-type", "f", "-exec", "cat", "{}", "+"], "cwd": str(repo)},
|
||||
)
|
||||
assert "RESOURCE_POLICY_BLOCKED" in pathless_find_exec_read
|
||||
pathless_find_delete = registry.execute(
|
||||
"run_command",
|
||||
{"cmd": ["find", "-delete"], "cwd": str(repo)},
|
||||
)
|
||||
assert "RESOURCE_POLICY_BLOCKED" in pathless_find_delete
|
||||
assert protected.exists()
|
||||
safe_interpreter = registry.execute(
|
||||
"run_command",
|
||||
{
|
||||
"cmd": [
|
||||
sys.executable,
|
||||
"-c",
|
||||
"print(1)",
|
||||
],
|
||||
"cwd": str(repo),
|
||||
},
|
||||
)
|
||||
assert "RESOURCE_POLICY_BLOCKED" not in safe_interpreter
|
||||
assert "1" in safe_interpreter
|
||||
assert "RESOURCE_POLICY_BLOCKED" in registry.execute("list_files", {"path": protected_dir.name})
|
||||
interpreter_read = registry.execute(
|
||||
"run_command",
|
||||
{
|
||||
|
|
@ -620,6 +703,18 @@ def test_protected_black_box_artifact_policy_blocks_introspection(tmp_path, monk
|
|||
},
|
||||
)
|
||||
assert "RESOURCE_POLICY_BLOCKED" in relative_interpreter_read
|
||||
versioned_interpreter_read = registry.execute(
|
||||
"run_command",
|
||||
{
|
||||
"cmd": [
|
||||
"python3.12",
|
||||
"-c",
|
||||
f"from pathlib import Path; print(Path({protected.name!r}).read_bytes())",
|
||||
],
|
||||
"cwd": str(repo),
|
||||
},
|
||||
)
|
||||
assert "RESOURCE_POLICY_BLOCKED" in versioned_interpreter_read
|
||||
constructed_path_read = registry.execute(
|
||||
"run_command",
|
||||
{
|
||||
|
|
@ -663,6 +758,88 @@ def test_protected_black_box_artifact_policy_blocks_introspection(tmp_path, monk
|
|||
assert "RESOURCE_POLICY_BLOCKED" in env_assignment_read
|
||||
shell_script_read = registry.execute("run_command", {"cmd": ["sh", str(protected)]})
|
||||
assert "RESOURCE_POLICY_BLOCKED" in shell_script_read
|
||||
for cmd in (
|
||||
["cmd.exe", "/c", "type", protected.name],
|
||||
["cmd.exe", "/c", "copy", protected.name, str(repo / "copy.cmd")],
|
||||
["cmd.exe", "/c", "xcopy", protected.name, str(repo / "copy-dir")],
|
||||
["powershell.exe", "-Command", "Get-Content", protected.name],
|
||||
["powershell.exe", "-Command", "Select-String", "reference", protected.name],
|
||||
["powershell.exe", "-Command", "Copy-Item", protected.name, str(repo / "copy.ps1")],
|
||||
["pwsh", "-Command", "Get-FileHash", protected.name],
|
||||
["cmd.exe", "/c", "certutil", "-hashfile", protected.name],
|
||||
):
|
||||
result = registry.execute("run_command", {"cmd": cmd, "cwd": str(repo)})
|
||||
assert "RESOURCE_POLICY_BLOCKED" in result, cmd
|
||||
encoded_read = base64.b64encode(f"Get-Content {protected.name}".encode("utf-16le")).decode("ascii")
|
||||
for cmd in (
|
||||
["powershell.exe", "-EncodedCommand", encoded_read],
|
||||
["pwsh", "-enc", encoded_read],
|
||||
):
|
||||
result = registry.execute("run_command", {"cmd": cmd, "cwd": str(repo)})
|
||||
assert "RESOURCE_POLICY_BLOCKED" in result, cmd
|
||||
search_direct = registry.execute("search_code", {"query": "reference", "path": protected.name})
|
||||
assert "RESOURCE_POLICY_BLOCKED" in search_direct
|
||||
search_protected_dir = registry.execute("search_code", {"query": "secret", "path": protected_dir.name})
|
||||
assert "RESOURCE_POLICY_BLOCKED" in search_protected_dir
|
||||
grep_read = registry.execute("run_command", {"cmd": ["grep", "reference", str(protected)]})
|
||||
assert "RESOURCE_POLICY_BLOCKED" in grep_read
|
||||
grep_recursive = registry.execute("run_command", {"cmd": ["grep", "-R", "reference", "."], "cwd": str(repo)})
|
||||
assert "RESOURCE_POLICY_BLOCKED" in grep_recursive
|
||||
rg_read = registry.execute("run_command", {"cmd": ["rg", "reference", str(protected)]})
|
||||
assert "RESOURCE_POLICY_BLOCKED" in rg_read
|
||||
rg_recursive = registry.execute("run_command", {"cmd": ["rg", "reference", "."], "cwd": str(repo)})
|
||||
assert "RESOURCE_POLICY_BLOCKED" in rg_recursive
|
||||
copy_recursive = registry.execute("run_command", {"cmd": ["cp", "-R", ".", str(repo / "copy")], "cwd": str(repo)})
|
||||
assert "RESOURCE_POLICY_BLOCKED" in copy_recursive
|
||||
for cmd in (
|
||||
["git", "diff", "--", protected.name],
|
||||
["git", "diff"],
|
||||
["git", "show", f"HEAD:{protected.name}"],
|
||||
["git", "show", "HEAD"],
|
||||
["git", "grep", "reference", "--", protected.name],
|
||||
["git", "grep", "reference"],
|
||||
["git", "cat-file", "-p", f"HEAD:{protected.name}"],
|
||||
["git", "log", "-p", "--", protected.name],
|
||||
["git", "log", "-p"],
|
||||
):
|
||||
result = registry.execute("run_command", {"cmd": cmd, "cwd": str(repo)})
|
||||
assert "RESOURCE_POLICY_BLOCKED" in result, cmd
|
||||
assert "RESOURCE_POLICY_BLOCKED" in registry.execute("vcs_diff", {"path": protected.name})
|
||||
assert "RESOURCE_POLICY_BLOCKED" in registry.execute("vcs_diff", {})
|
||||
import ouroboros.code_intelligence as code_intelligence
|
||||
|
||||
original_file_fact = code_intelligence._file_fact
|
||||
|
||||
def guarded_file_fact(repo_root, path):
|
||||
assert pathlib.Path(path).resolve(strict=False) != protected.resolve(strict=False)
|
||||
return original_file_fact(repo_root, path)
|
||||
|
||||
monkeypatch.setattr(code_intelligence, "_file_fact", guarded_file_fact)
|
||||
digest = registry.execute("codebase_digest", {})
|
||||
assert protected.name not in digest
|
||||
assert generated.name in digest
|
||||
run_output_export = registry.execute("run_command", {"cmd": direct_cmd, "outputs": [protected.name], "cwd": str(repo)})
|
||||
assert "ARTIFACT_OUTPUT_ERROR" in run_output_export
|
||||
assert "RESOURCE_POLICY_BLOCKED" in run_output_export
|
||||
script_output_export = registry.execute(
|
||||
"run_script",
|
||||
{"interpreter": "python3", "script": "print('ok')", "outputs": [protected.name], "cwd": str(repo)},
|
||||
)
|
||||
assert "RESOURCE_POLICY_BLOCKED" in script_output_export
|
||||
service_cmd = ["cmd.exe", "/c", "ping", "127.0.0.1", "-n", "30"] if os.name == "nt" else ["sleep", "30"]
|
||||
service_start = registry.execute(
|
||||
"start_service",
|
||||
{
|
||||
"name": "protected-output",
|
||||
"cmd": service_cmd,
|
||||
"cwd": str(repo),
|
||||
"outputs": [protected.name],
|
||||
},
|
||||
)
|
||||
assert "protected-output" in service_start
|
||||
service_stop = registry.execute("stop_service", {"name": "protected-output"})
|
||||
assert "ARTIFACT_OUTPUT_ERROR" in service_stop
|
||||
assert "RESOURCE_POLICY_BLOCKED" in service_stop
|
||||
for cmd in (
|
||||
["strings", str(protected)],
|
||||
["objdump", "-d", str(protected)],
|
||||
|
|
@ -673,14 +850,145 @@ def test_protected_black_box_artifact_policy_blocks_introspection(tmp_path, monk
|
|||
["lldb", str(protected)],
|
||||
["cp", str(protected), str(repo / "copy.sh")],
|
||||
["dd", f"if={protected}", f"of={repo / 'copy2.sh'}"],
|
||||
["tar", "-czf", str(repo / "out.tgz"), protected.name],
|
||||
["tar", "-czf", str(repo / "tree.tgz"), "."],
|
||||
["zip", str(repo / "out.zip"), protected.name],
|
||||
["rsync", protected.name, str(repo / "copy.sh")],
|
||||
):
|
||||
result = registry.execute("run_command", {"cmd": cmd})
|
||||
result = registry.execute("run_command", {"cmd": cmd, "cwd": str(repo)})
|
||||
assert "RESOURCE_POLICY_BLOCKED" in result, cmd
|
||||
|
||||
generated_result = registry.execute("run_command", {"cmd": ["strings", str(generated)]})
|
||||
assert "RESOURCE_POLICY_BLOCKED" not in generated_result
|
||||
|
||||
|
||||
def test_protected_black_box_recursive_policy_maps_executor_backend_paths(tmp_path, monkeypatch):
|
||||
from ouroboros.contracts.task_contract import build_task_contract
|
||||
from ouroboros.tools.registry import ToolContext, ToolRegistry
|
||||
|
||||
system_repo = tmp_path / "system"
|
||||
workspace = tmp_path / "workspace"
|
||||
data = tmp_path / "data"
|
||||
for path in (system_repo, workspace, data):
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
protected = workspace / "executable"
|
||||
protected.write_text("reference bytes\n", encoding="utf-8")
|
||||
task_contract = build_task_contract({
|
||||
"resource_policy": {
|
||||
"protected_artifacts": [
|
||||
{
|
||||
"id": "reference",
|
||||
"role": "black_box_reference",
|
||||
"paths": ["/workspace/executable"],
|
||||
"allow": ["execute"],
|
||||
"deny": ["read_bytes", "copy", "hash", "static_introspection", "dynamic_trace", "debug"],
|
||||
}
|
||||
]
|
||||
}
|
||||
})
|
||||
registry = ToolRegistry(repo_dir=system_repo, drive_root=data)
|
||||
registry.set_context(
|
||||
ToolContext(
|
||||
repo_dir=system_repo,
|
||||
drive_root=data,
|
||||
workspace_root=workspace,
|
||||
workspace_mode="external",
|
||||
task_contract=task_contract,
|
||||
task_metadata={"task_contract": task_contract},
|
||||
executor_ref={
|
||||
"type": "docker_exec",
|
||||
"id": "pb-container",
|
||||
"container_name": "pb-container",
|
||||
"network": "none",
|
||||
"workspace_host_path": str(workspace),
|
||||
"workspace_backend_path": "/workspace",
|
||||
},
|
||||
)
|
||||
)
|
||||
monkeypatch.setattr("ouroboros.safety.check_safety", lambda *a, **k: (True, ""))
|
||||
|
||||
grep_recursive = registry.execute("run_command", {"cmd": ["grep", "-R", "reference", "."], "cwd": str(workspace)})
|
||||
copy_recursive = registry.execute("run_command", {"cmd": ["cp", "-R", ".", str(workspace / "copy")], "cwd": str(workspace)})
|
||||
import ouroboros.code_intelligence as code_intelligence
|
||||
|
||||
original_file_fact = code_intelligence._file_fact
|
||||
|
||||
def guarded_file_fact(repo_root, path):
|
||||
assert pathlib.Path(path).resolve(strict=False) != protected.resolve(strict=False)
|
||||
return original_file_fact(repo_root, path)
|
||||
|
||||
monkeypatch.setattr(code_intelligence, "_file_fact", guarded_file_fact)
|
||||
digest = registry.execute("codebase_digest", {})
|
||||
|
||||
assert "RESOURCE_POLICY_BLOCKED" in grep_recursive
|
||||
assert "RESOURCE_POLICY_BLOCKED" in copy_recursive
|
||||
assert "executable" not in digest
|
||||
|
||||
|
||||
def test_schedule_subagent_inherits_workspace_executor_ref(tmp_path, monkeypatch):
|
||||
from ouroboros.contracts.task_contract import build_task_contract
|
||||
from ouroboros.tools.registry import ToolContext, ToolRegistry
|
||||
|
||||
system_repo = tmp_path / "system"
|
||||
workspace = tmp_path / "workspace"
|
||||
data = tmp_path / "data"
|
||||
for path in (system_repo, workspace, data):
|
||||
path.mkdir(parents=True)
|
||||
task_contract = build_task_contract({
|
||||
"resource_policy": {
|
||||
"protected_artifacts": [
|
||||
{
|
||||
"id": "reference",
|
||||
"role": "black_box_reference",
|
||||
"paths": ["/workspace/executable"],
|
||||
"allow": ["execute"],
|
||||
}
|
||||
]
|
||||
}
|
||||
})
|
||||
executor_ref = {
|
||||
"type": "docker_exec",
|
||||
"id": "pb-container",
|
||||
"container_name": "pb-container",
|
||||
"network": "none",
|
||||
"workspace_host_path": str(workspace),
|
||||
"workspace_backend_path": "/workspace",
|
||||
}
|
||||
registry = ToolRegistry(repo_dir=system_repo, drive_root=data)
|
||||
ctx = ToolContext(
|
||||
repo_dir=system_repo,
|
||||
drive_root=data,
|
||||
workspace_root=workspace,
|
||||
workspace_mode="external",
|
||||
task_id="parent-task",
|
||||
task_contract=task_contract,
|
||||
task_metadata={"task_contract": task_contract},
|
||||
executor_ref=executor_ref,
|
||||
)
|
||||
registry.set_context(ctx)
|
||||
monkeypatch.setenv("OUROBOROS_MAX_SUBAGENT_DEPTH", "4")
|
||||
|
||||
result = registry.execute(
|
||||
"schedule_subagent",
|
||||
{
|
||||
"objective": "Inspect the workspace contract.",
|
||||
"expected_output": "A concise report.",
|
||||
"role": "auditor",
|
||||
"model_lane": "light",
|
||||
},
|
||||
)
|
||||
|
||||
assert "Subagent request queued" in result
|
||||
assert ctx.pending_events
|
||||
event = ctx.pending_events[0]
|
||||
assert event["executor_ref"] == executor_ref
|
||||
assert event["metadata"]["executor_ref"] == executor_ref
|
||||
child_id = event["task_id"]
|
||||
persisted = json.loads((data / "task_results" / f"{child_id}.json").read_text(encoding="utf-8"))
|
||||
assert persisted["executor_ref"] == executor_ref
|
||||
assert persisted["task_contract"]["resource_policy"]["protected_artifacts"][0]["paths"] == ["/workspace/executable"]
|
||||
|
||||
|
||||
def test_capability_omission_manifest_surfaces_extension_discovery_failure(tmp_path, monkeypatch):
|
||||
from ouroboros import extension_loader
|
||||
from ouroboros.tools import tool_discovery
|
||||
|
|
@ -772,6 +1080,84 @@ def test_local_readonly_subagent_data_read_denies_secret_files(tmp_path):
|
|||
assert "{}" in readable
|
||||
|
||||
|
||||
def test_runtime_data_write_blocks_workspace_executor_control_state(tmp_path, monkeypatch):
|
||||
from ouroboros.tools.registry import ToolContext, ToolRegistry
|
||||
|
||||
repo = tmp_path / "repo"
|
||||
data = tmp_path / "data"
|
||||
repo.mkdir()
|
||||
data.mkdir()
|
||||
state_dir = data / "state" / "workspace_executor_processes"
|
||||
state_dir.mkdir(parents=True)
|
||||
existing = state_dir / "foreground-forged.json"
|
||||
existing.write_text("original", encoding="utf-8")
|
||||
registry = ToolRegistry(repo_dir=repo, drive_root=data)
|
||||
registry.set_context(ToolContext(repo_dir=repo, drive_root=data))
|
||||
monkeypatch.setattr("ouroboros.safety.check_safety", lambda *a, **k: (True, ""))
|
||||
|
||||
direct_write = registry.execute(
|
||||
"write_file",
|
||||
{
|
||||
"root": "runtime_data",
|
||||
"path": "state/workspace_executor_processes/foreground-forged.json",
|
||||
"content": "{}",
|
||||
},
|
||||
)
|
||||
assert "DATA_WRITE_BLOCKED" in direct_write
|
||||
assert existing.read_text(encoding="utf-8") == "original"
|
||||
|
||||
nested_write = registry.execute(
|
||||
"write_file",
|
||||
{
|
||||
"root": "runtime_data",
|
||||
"path": "state/headless_tasks/child/data/state/workspace_executor_processes/foreground-forged.json",
|
||||
"content": "{}",
|
||||
},
|
||||
)
|
||||
assert "DATA_WRITE_BLOCKED" in nested_write
|
||||
|
||||
edit = registry.execute(
|
||||
"edit_text",
|
||||
{
|
||||
"root": "runtime_data",
|
||||
"path": "state/workspace_executor_processes/foreground-forged.json",
|
||||
"old_str": "original",
|
||||
"new_str": "tampered",
|
||||
},
|
||||
)
|
||||
assert "EDIT_TEXT_BLOCKED" in edit
|
||||
assert existing.read_text(encoding="utf-8") == "original"
|
||||
|
||||
shell_write = registry.execute(
|
||||
"run_command",
|
||||
{
|
||||
"cmd": [
|
||||
sys.executable,
|
||||
"-c",
|
||||
(
|
||||
"from pathlib import Path; "
|
||||
f"Path(r'{existing}').write_text('{{\"owner\":\"ouroboros_workspace_executor\"}}')"
|
||||
),
|
||||
],
|
||||
},
|
||||
)
|
||||
assert "WORKSPACE_EXECUTOR_STATE_WRITE_BLOCKED" in shell_write
|
||||
assert existing.read_text(encoding="utf-8") == "original"
|
||||
|
||||
node_eval_write = registry.execute(
|
||||
"run_command",
|
||||
{
|
||||
"cmd": [
|
||||
"node",
|
||||
"-e",
|
||||
f"require('fs').writeFileSync({str(existing)!r}, '{{}}')",
|
||||
],
|
||||
},
|
||||
)
|
||||
assert "WORKSPACE_EXECUTOR_STATE_WRITE_BLOCKED" in node_eval_write
|
||||
assert existing.read_text(encoding="utf-8") == "original"
|
||||
|
||||
|
||||
def test_local_readonly_subagent_repo_read_denies_secret_files(tmp_path):
|
||||
from ouroboros.contracts.task_constraint import TaskConstraint
|
||||
from ouroboros.tools.registry import ToolContext, ToolRegistry
|
||||
|
|
|
|||
1525
tests/test_workspace_executor.py
Normal file
1525
tests/test_workspace_executor.py
Normal file
File diff suppressed because it is too large
Load diff
214
tests/test_workspace_executor_cleanup.py
Normal file
214
tests/test_workspace_executor_cleanup.py
Normal file
|
|
@ -0,0 +1,214 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
|
||||
|
||||
def _write_docker_records(state_dir, *, foreground_pidfile="/tmp/ouroboros-exec-test.pid", service_pid="12345"):
|
||||
state_dir.mkdir(parents=True)
|
||||
(state_dir / "foreground-docker.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"id": "foreground-docker",
|
||||
"schema_version": 1,
|
||||
"owner": "ouroboros_workspace_executor",
|
||||
"record_type": "foreground",
|
||||
"executor_type": "docker_exec",
|
||||
"executor_id": "docker",
|
||||
"host_pid": 0,
|
||||
"container_name": "bench",
|
||||
"backend_pidfile": foreground_pidfile,
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(state_dir / "service-docker.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"id": "service-docker",
|
||||
"schema_version": 1,
|
||||
"owner": "ouroboros_workspace_executor",
|
||||
"record_type": "service",
|
||||
"service_id": "task:svc",
|
||||
"task_id": "task",
|
||||
"name": "svc",
|
||||
"executor_type": "docker_exec",
|
||||
"executor_id": "docker",
|
||||
"container_name": "bench",
|
||||
"backend_pid": service_pid,
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def _install_live_docker_service(workspace_executor, tmp_path):
|
||||
executor = workspace_executor.ExecutorRef(
|
||||
kind="docker_exec",
|
||||
executor_id="docker",
|
||||
network="none",
|
||||
mappings=(),
|
||||
container_name="bench",
|
||||
)
|
||||
with workspace_executor._STATE_LOCK:
|
||||
workspace_executor._SERVICES.clear()
|
||||
workspace_executor._SERVICES["task:live"] = workspace_executor._ExecutorService(
|
||||
service_id="task:live",
|
||||
task_id="task",
|
||||
name="live",
|
||||
executor=executor,
|
||||
cmd=["sleep", "30"],
|
||||
host_cwd=tmp_path,
|
||||
backend_cwd="/workspace",
|
||||
cwd_root="active_workspace",
|
||||
outputs=[],
|
||||
before_outputs={},
|
||||
backend_pid="67890",
|
||||
)
|
||||
|
||||
|
||||
def test_executor_panic_cleanup_wait_false_uses_bounded_docker_stop(tmp_path, monkeypatch):
|
||||
import ouroboros.workspace_executor as workspace_executor
|
||||
|
||||
data = tmp_path / "data"
|
||||
state_dir = data / "state" / "workspace_executor_processes"
|
||||
_write_docker_records(state_dir)
|
||||
_install_live_docker_service(workspace_executor, tmp_path)
|
||||
|
||||
docker_run_calls: list[list[str]] = []
|
||||
|
||||
def fake_docker_wait(cmd, **kwargs):
|
||||
docker_run_calls.append([str(part) for part in cmd])
|
||||
return subprocess.CompletedProcess(cmd, 0, stdout="", stderr="")
|
||||
|
||||
class FakePopen:
|
||||
def __init__(self, cmd, **kwargs):
|
||||
raise AssertionError("panic Docker cleanup must not spawn untracked helpers")
|
||||
|
||||
monkeypatch.setattr(workspace_executor.subprocess, "run", fake_docker_wait)
|
||||
monkeypatch.setattr(workspace_executor.subprocess, "Popen", FakePopen)
|
||||
|
||||
killed_foreground = workspace_executor.kill_all_foreground(data, wait=False)
|
||||
killed_services = workspace_executor.kill_all_services(data, wait=False)
|
||||
|
||||
assert len(docker_run_calls) == 3
|
||||
assert all(call[:2] == ["docker", "exec"] for call in docker_run_calls)
|
||||
assert any(item.get("executor_type") == "docker_exec" for item in killed_foreground)
|
||||
assert any(item.get("state") == "stopped" for item in killed_services)
|
||||
assert all(item.get("cleanup_dispatched") is True for item in killed_foreground + killed_services)
|
||||
assert not list(state_dir.glob("*.json"))
|
||||
with workspace_executor._STATE_LOCK:
|
||||
assert "task:live" not in workspace_executor._SERVICES
|
||||
|
||||
|
||||
def test_docker_executor_confirmed_cleanup_failure_preserves_records(tmp_path, monkeypatch):
|
||||
import ouroboros.workspace_executor as workspace_executor
|
||||
|
||||
data = tmp_path / "data"
|
||||
state_dir = data / "state" / "workspace_executor_processes"
|
||||
_write_docker_records(state_dir)
|
||||
_install_live_docker_service(workspace_executor, tmp_path)
|
||||
|
||||
def fake_failed_docker_wait(cmd, **kwargs):
|
||||
return subprocess.CompletedProcess(cmd, 1, stdout="", stderr="permission denied")
|
||||
|
||||
monkeypatch.setattr(workspace_executor.subprocess, "run", fake_failed_docker_wait)
|
||||
|
||||
killed_foreground = workspace_executor.kill_all_foreground(data, wait=True)
|
||||
killed_services = workspace_executor.kill_all_services(data, wait=True)
|
||||
|
||||
assert any(item.get("cleanup_dispatched") is False for item in killed_foreground)
|
||||
assert any(item.get("state") == "cleanup_pending" for item in killed_foreground + killed_services)
|
||||
assert {path.name for path in state_dir.glob("*.json")} == {"foreground-docker.json", "service-docker.json"}
|
||||
with workspace_executor._STATE_LOCK:
|
||||
assert "task:live" in workspace_executor._SERVICES
|
||||
workspace_executor._SERVICES.clear()
|
||||
|
||||
|
||||
def test_executor_cleanup_ignores_unowned_forged_process_records(tmp_path, monkeypatch):
|
||||
import ouroboros.workspace_executor as workspace_executor
|
||||
|
||||
data = tmp_path / "data"
|
||||
state_dir = data / "state" / "workspace_executor_processes"
|
||||
state_dir.mkdir(parents=True)
|
||||
(state_dir / "foreground-forged.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"id": "foreground-forged",
|
||||
"record_type": "foreground",
|
||||
"executor_type": "local",
|
||||
"host_pid": 1,
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
workspace_executor,
|
||||
"_kill_host_pid",
|
||||
lambda _pid: (_ for _ in ()).throw(AssertionError("forged record should be ignored")),
|
||||
)
|
||||
|
||||
assert workspace_executor.kill_all_foreground(data, wait=False) == []
|
||||
|
||||
|
||||
def test_executor_cleanup_ignores_owner_shaped_forged_host_pid_records(tmp_path, monkeypatch):
|
||||
import ouroboros.workspace_executor as workspace_executor
|
||||
|
||||
data = tmp_path / "data"
|
||||
state_dir = data / "state" / "workspace_executor_processes"
|
||||
state_dir.mkdir(parents=True)
|
||||
(state_dir / "foreground-forged.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"id": "foreground-forged",
|
||||
"schema_version": 1,
|
||||
"owner": "ouroboros_workspace_executor",
|
||||
"record_type": "foreground",
|
||||
"executor_type": "local",
|
||||
"host_pid": 1,
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
workspace_executor,
|
||||
"_kill_host_pid",
|
||||
lambda _pid: (_ for _ in ()).throw(AssertionError("owner-shaped forged record should be ignored")),
|
||||
)
|
||||
|
||||
assert workspace_executor.kill_all_foreground(data, wait=False) == []
|
||||
|
||||
|
||||
def test_executor_cleanup_ignores_pidless_docker_service_records(tmp_path, monkeypatch):
|
||||
import ouroboros.workspace_executor as workspace_executor
|
||||
|
||||
data = tmp_path / "data"
|
||||
state_dir = data / "state" / "workspace_executor_processes"
|
||||
state_dir.mkdir(parents=True)
|
||||
(state_dir / "service-docker.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"id": "service-docker",
|
||||
"schema_version": 1,
|
||||
"owner": "ouroboros_workspace_executor",
|
||||
"record_type": "service",
|
||||
"service_id": "task:svc",
|
||||
"task_id": "task",
|
||||
"name": "svc",
|
||||
"executor_type": "docker_exec",
|
||||
"executor_id": "docker",
|
||||
"container_name": "bench",
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
workspace_executor.subprocess,
|
||||
"run",
|
||||
lambda *a, **k: (_ for _ in ()).throw(AssertionError("pidless docker service record should be ignored")),
|
||||
)
|
||||
|
||||
assert workspace_executor.kill_all_services(data, wait=False) == []
|
||||
|
|
@ -196,6 +196,44 @@
|
|||
* @property {Object=} grants
|
||||
*/
|
||||
|
||||
/**
|
||||
* @typedef {Object} ExecutorRef
|
||||
* @property {"local"|"docker_exec"} type
|
||||
* @property {string=} id
|
||||
* @property {"host"|"none"=} network
|
||||
* @property {string=} workspace_host_path
|
||||
* @property {string=} workspace_backend_path
|
||||
* @property {string=} container_name Required when type is "docker_exec".
|
||||
* @property {Object[]=} path_mappings
|
||||
*/
|
||||
|
||||
/**
|
||||
* @typedef {Object} TaskCreateRequest
|
||||
* @property {string} description
|
||||
* @property {string=} task_id
|
||||
* @property {string=} type
|
||||
* @property {number=} chat_id
|
||||
* @property {number=} depth
|
||||
* @property {string=} session_id
|
||||
* @property {string=} workspace_root
|
||||
* @property {"external"|"self"=} workspace_mode
|
||||
* @property {"forked"|"empty"|"shared"=} memory_mode
|
||||
* @property {Object[]=} attachments
|
||||
* @property {Object=} allowed_resources
|
||||
* @property {Object=} resource_policy
|
||||
* @property {ExecutorRef=} executor_ref
|
||||
* @property {string=} deadline_at
|
||||
* @property {number=} timeout_sec
|
||||
* @property {number=} timeout
|
||||
* @property {string=} context
|
||||
* @property {string=} expected_output
|
||||
* @property {string=} constraints
|
||||
* @property {boolean=} context_requires_self_body_docs
|
||||
* @property {string=} actor_id Top-level task actor/provenance id; metadata.actor_id is reserved.
|
||||
* @property {string=} source Top-level task source/provenance label.
|
||||
* @property {Object=} metadata Arbitrary task metadata; executor_ref/workspace_executor keys are reserved.
|
||||
*/
|
||||
|
||||
/**
|
||||
* @typedef {Object} TaskCreateResponse
|
||||
* @property {boolean} ok
|
||||
|
|
@ -269,4 +307,4 @@
|
|||
* @property {boolean=} ok
|
||||
*/
|
||||
|
||||
export const GATEWAY_CONTRACT_VERSION = '6.18.1';
|
||||
export const GATEWAY_CONTRACT_VERSION = '6.19.0-rc.1';
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"name": "ouroboros-web",
|
||||
"version": "6.18.1",
|
||||
"version": "6.19.0-rc.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"description": "Ouroboros browser UI package boundary",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue