Commit graph

4 commits

Author SHA1 Message Date
Ouroboros
96dc4fb6bb Fix benchmark actor provenance
Co-authored-by: Ouroboros <311266734+ouroboros-agent@users.noreply.github.com>
2026-08-20 02:32:32 +03:00
Ouroboros
1947a945fc Migrate benchmark subagent profiles
Co-authored-by: Ouroboros <311266734+ouroboros-agent@users.noreply.github.com>
2026-08-20 02:32:32 +03:00
Anton Razzhigaev
4d75721c30 fix: give apply_patch/edit_batch the same repo-write fences as edit_text
Guard review of the editing tools found the new pair weaker than the tools
they sit beside — not in what they can do, but in what refuses them.

Root cause: a guard judged a different spelling of the path than the write
used. `ctx.repo_path` runs `normalize_root_relative`, so an absolute path
inside the root and a redundant root-basename prefix land on the same file a
bare relative path does; `_resolve_edit_target` checked the RAW spelling, for
which `repo/BIBLE.md` is not a member of the protected-path table while
`BIBLE.md` is. In runtime_mode=advanced, `apply_patch`/`edit_batch` therefore
rewrote BIBLE.md and ouroboros/safety.py where edit_text returns
CORE_PROTECTION_BLOCKED. Reproduced against the real dispatcher, both
spellings, on the default root.

`edit_text`/`write_file` are immune only because the dispatcher canonicalizes
their `path` ARG once (`_PATH_NORMALIZED_TOOLS`). The new tools carry paths
inside the payload, so the module comment claiming their handlers "normalize
each target themselves via the shared edit_text guard chain" described a
normalization the shared chain never had.

One canonicalization contract fixes the whole class:
- `tool_access.canonical_repo_relative_path` is the single normalization both
  ends call: the handler before its own protected checks, and the dispatch
  gates through `_payload_write_paths`, which reads apply_patch's targets back
  out of the REAL parser (`patch_target_paths`) so the gate cannot drift.
- `_resolve_edit_target` RETURNS that canonical rel, because it is the file's
  identity. Keying the plan by the raw spelling meant two spellings of one file
  in a single call produced two buffers and two writes, and the last silently
  discarded the first while the result reported both as applied — reproduced as
  real data loss on edit_batch and apply_patch.
- `_ROOT_ARG_REPO_WRITE_TOOLS` replaces the hardcoded write_file/edit_text pair
  at the three fences that missed the new tools: the acting-no-workspace fence
  (an acting child with no isolated workspace reached the LIVE repo), the
  protected-write gate (which also carries the pro + protected_paths_grant
  condition), and the acting root-enum narrowing.

Parity with the other repo writers, item by item:
- The assisted managed-update resolver keeps its protected-path exemption here
  too; without it these tools were the one lane that could not finish a
  conflict resolution.
- A pro-mode protected edit appends `core_patch_notice`, as git._repo_write and
  _str_replace_editor do. The mode ALLOWS the write; the notice is what keeps
  it visible.
- Validation refusals get the typed `edit_ops_blocked` status: a counted or
  context miss is the designed path and is user-correctable exactly like
  edit_text's "old_str not found", so it is a policy denial, not the false
  tool_failure headline v6.57.0 removed for the other write tools. A partial
  write is NOT that: it carries its own `EDIT_OPS_PARTIAL_WRITE_FAILED` prefix
  and stays a real execution failure.
- A write failure after some files were written invalidates the advisory
  snapshot for them and says PARTIALLY APPLIED, so commit_reviewed cannot accept
  them against a pre-review taken before they existed.

Honest contracts instead of implied ones:
- Atomicity is over VALIDATION. Every schema, prompt and doc that implied the
  writes roll back now says so, and the residual (a mid-write I/O fault can
  leave earlier files applied) is disclosed rather than papered over.
- The fuzzy trailing-whitespace fallback states what it actually did: the
  replaced lines, context included, now carry the patch's trailing whitespace.
- `_unified_diff` reports a final-newline change instead of "(no textual
  changes)" — the rail exists to let the agent verify an overwrite, so the one
  answer it must never give is "nothing changed" for changed bytes.
- `_syntax_check` names the format it actually checked (a NUL byte in a .py file
  reported "not valid JSON").
- `edit_batch` gets apply_patch's 80k result limit: an aborted batch reports
  every failed edit so one retry can fix them all.

Newline handling is deliberately NOT changed: the repo-write lane reads with
universal newlines and writes \n, so edit_text and write_file flatten CRLF the
same way today. Documented in the module rather than diverging one tool from
its lane.

editbench: generated `fixtures_v2/` is gitignored (running the benchmark dirtied
the seed gate), missing fixture trees fail fast before the isolated server and
the paid jobs rather than inside copytree, the run root is DERIVED instead of
mkdtemp'd so a seed-gate refusal leaves no footprint ahead of admission, and the
README and launcher docstring stop offering `edit_sketch_only` and a `default`
config they no longer have.

Tests exercise the REAL guard chain (a registry over a throwaway repo, no
patched resolver): protected paths refused in all three spellings for all three
tools, the acting-no-workspace fence for all three write tools, one file under
two spellings staying one target, the pro-mode notice, the resolver exemption,
parser-derived patch targets, partial-write disclosure and its status, and the
diff rail. DEVELOPMENT's New Tool rule now names the guard surfaces, since every
visibility list was already green while these fences were missing.

Co-Authored-By: Andrei Kaznacheev <a.kaznacheev@sdgroup.ai>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Ouroboros <311266734+ouroboros-agent@users.noreply.github.com>
2026-08-06 02:54:05 +03:00
Andrei Kaznacheev
6641081b10 feat: add apply_patch and edit_batch editing tools with editbench evidence
Two new repo-lane editing primitives beyond exact-match edit_text and
full-file write_file, both validated on the included editbench benchmark
before adoption:

- edit_batch: atomic batch of COUNTED exact replacements. Each edit
  declares the occurrence count it expects and replaces all of them; any
  mismatch aborts the whole batch with per-edit diagnostics. The safe
  replace-all: a miscount is an atomic refusal, not a silent corruption.
- apply_patch: context-anchored multi-file patch (V4A-style, no line
  numbers; optional @@ anchors; trailing-whitespace fuzzy fallback).
  Atomic across all files/hunks with per-hunk diagnostics; parser
  tolerates decorative trailing *** on headers (observed model quirk).

write_file repo lane gains two verification rails inherited from an
edit_sketch fast-apply tool that was implemented, benchmarked, and
REJECTED (data in devtools/benchmarks/editbench/README.md): a pre-write
syntax guard for .py/.json (force bypasses with an explicit
SYNTAX_GUARD_BYPASSED disclosure — no silent bypass, P3) and the unified
diff vs the previous version appended to overwrite results (capped with
an explicit truncation count, P1).

Registration follows the New Tool rule across every canonical surface:
safety.py TOOL_POLICY, tool_capabilities (CORE/ACTING envelopes + result
limits; local-readonly and heal lanes deliberately excluded), registry
(_FROZEN_TOOL_MODULES/_WORKSPACE_ALLOWED_TOOLS/_REPO_MUTATION_TOOLS),
outcomes (_ROOT_WRITE_TOOLS so patch/batch-only turns stay acceptance-
review-eligible; _RECOVERY_TOOL_NAMES), smoke EXPECTED_TOOLS, and the
SYSTEM.md/ARCHITECTURE.md/DEVELOPMENT.md/CHECKLISTS.md documentation in
the same commit (P6). The edit_text/write_file descriptions steer tool
selection at the schema source (per DEVELOPMENT's anti-pattern guidance,
not SYSTEM.md prose accretion).

devtools/benchmarks/editbench/ is the evidence and the regression rig:
5 editing tasks (trap-laden rename, surgical edit in a 1691-line module,
whole-function replacement, cross-file move, near-full-file transform),
per-config tool restriction via disabled_tools, deterministic grading,
token/latency/error mining. Migrated under the benchmark admission
contract (admit_benchmark_run/finalize_run_manifest, launcher_audit
MIGRATED_LAUNCHERS); it measures the current working tree by design, so
real runs use the recorded --allow-dirty-seed escape.

Headline numbers (grok-4.5, 3 runs/config, every run solved every task;
full tables in the editbench README): edit_batch finishes the rename
task in ONE tool call vs ~6 for edit_text/write_file; on a weak model
(gemini-3.6-flash) its advantage grows to 1.5-2.5x cheaper/faster; a
free-choice agent picked the right tool per task shape in every run.

No version carriers touched (maintainer assigns the release version).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-06 00:03:01 +03:00