ouroboros/devtools/benchmarks/common
Andrei Kaznacheev b38e461e58 feat: add apply_patch and edit_batch editing tools with editbench evidence
Two new repo-lane editing primitives beyond exact-match edit_text and
full-file write_file, both validated on the included editbench benchmark
before adoption:

- edit_batch: atomic batch of COUNTED exact replacements. Each edit
  declares the occurrence count it expects and replaces all of them; any
  mismatch aborts the whole batch with per-edit diagnostics. The safe
  replace-all: a miscount is an atomic refusal, not a silent corruption.
- apply_patch: context-anchored multi-file patch (V4A-style, no line
  numbers; optional @@ anchors; trailing-whitespace fuzzy fallback).
  Atomic across all files/hunks with per-hunk diagnostics; parser
  tolerates decorative trailing *** on headers (observed model quirk).

write_file repo lane gains two verification rails inherited from an
edit_sketch fast-apply tool that was implemented, benchmarked, and
REJECTED (data in devtools/benchmarks/editbench/README.md): a pre-write
syntax guard for .py/.json (force bypasses with an explicit
SYNTAX_GUARD_BYPASSED disclosure — no silent bypass, P3) and the unified
diff vs the previous version appended to overwrite results (capped with
an explicit truncation count, P1).

Registration follows the New Tool rule across every canonical surface:
safety.py TOOL_POLICY, tool_capabilities (CORE/ACTING envelopes + result
limits; local-readonly and heal lanes deliberately excluded), registry
(_FROZEN_TOOL_MODULES/_WORKSPACE_ALLOWED_TOOLS/_REPO_MUTATION_TOOLS),
outcomes (_ROOT_WRITE_TOOLS so patch/batch-only turns stay acceptance-
review-eligible; _RECOVERY_TOOL_NAMES), smoke EXPECTED_TOOLS, and the
SYSTEM.md/ARCHITECTURE.md/DEVELOPMENT.md/CHECKLISTS.md documentation in
the same commit (P6). The edit_text/write_file descriptions steer tool
selection at the schema source (per DEVELOPMENT's anti-pattern guidance,
not SYSTEM.md prose accretion).

devtools/benchmarks/editbench/ is the evidence and the regression rig:
5 editing tasks (trap-laden rename, surgical edit in a 1691-line module,
whole-function replacement, cross-file move, near-full-file transform),
per-config tool restriction via disabled_tools, deterministic grading,
token/latency/error mining. Migrated under the benchmark admission
contract (admit_benchmark_run/finalize_run_manifest, launcher_audit
MIGRATED_LAUNCHERS); it measures the current working tree by design, so
real runs use the recorded --allow-dirty-seed escape.

Headline numbers (grok-4.5, 3 runs/config, every run solved every task;
full tables in the editbench README): edit_batch finishes the rename
task in ONE tool call vs ~6 for edit_text/write_file; on a weak model
(gemini-3.6-flash) its advantage grows to 1.5-2.5x cheaper/faster; a
free-choice agent picked the right tool per task shape in every run.

No version carriers touched (maintainer assigns the release version).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-04 19:55:26 +05:00
..
__init__.py feat(devtools-benchmarks): add official benchmark harnesses and workspace executor 2026-06-06 12:03:30 +03:00
launcher_audit.py feat: add apply_patch and edit_batch editing tools with editbench evidence 2026-08-04 19:55:26 +05:00
manifests.py v6.81.0: benchmark artefacts must not carry or claim what did not happen 2026-07-26 05:04:14 +00:00
model_slots.py devtools(swe-bench-pro): adapter hardening — crash-resilience + musl install-in-image + gold-history strip 2026-06-25 18:40:52 +03:00
official_commands.py feat: v6.55.0 bench devtools alignment — scaffold defaults, ProgramBench e2e, CLB launcher, OSWorld 2.0 2026-07-03 22:49:01 +03:00
result_index.py benchmarks: derive the truncation vocabulary from the runtime, not beside the check 2026-07-26 05:26:48 +00:00
run_roots.py feat(v6.81.0): benchmark provenance becomes a gate, and the artefacts stop lying 2026-07-26 03:40:44 +00:00
secrets.py v6.81.0: benchmark artefacts must not carry or claim what did not happen 2026-07-26 05:04:14 +00:00
server_runner.py v6.81.0: benchmark artefacts must not carry or claim what did not happen 2026-07-26 05:04:14 +00:00
subprocesses.py feat(devtools-benchmarks): add official benchmark harnesses and workspace executor 2026-06-06 12:03:30 +03:00