docs(adr): record WP33 audit findings in ADR-337 and #920 field evidence in ADR-335

The WP33 audit (#919) returned one strong result and four corrections.
Two of the corrections were claims that would otherwise have travelled
downstream as facts, so they are recorded before the ADRs merge.

The enforcement guarantee held under brute force and is now evidenced
rather than asserted: 24,300 constructible configurations times 5
mandatory operations is 121,500 checks, zero bypasses. The forced
purchase, classification-before-detection ordering, union-not-first-match
classification, must_use outcome, the cost>0 and round-cap pair, the
oracle exit with its non-finite guard, and unit-range rescaling all
verified clean.

Four corrections, each of which narrows a claim rather than weakening the
mechanism:

The 0.06% overhead figure now never appears without its caveat. It is
honest for what it measures -- the ladder's own cost with an
instant-return fixture, and the accounting genuinely wraps classification,
detection, the VoI decision, investigation and observation -- but it is
not the number the <5% target is about. With investigators that sleep
their declared latency the measured overhead is 14.8%, roughly three times
target. Both numbers and what each measures are stated.

The <5% target is unenforced, and said so plainly. latency_price defaults
to 0.0, so a rung declaring ten seconds costs what a 0us rung costs, and
only max_rounds bounds wall-clock cost. Pricing latency honestly refuses
the shipped reference ladder at construction, so the honest configuration
is currently hard to reach.

"100% inspection of mandatory classes" is not yet a checkable metric.
mandatory_inspections has no denominator, so a run of 200 operations each
matching all five mandatory classes reported zero -- correct behaviour,
since they halted fail-closed before purchase, but indistinguishable from
a ladder that silently stopped classifying. Enforcement is airtight; the
measurement of it is not, and the ADR now separates the two.

The module is not wired into the gate's request path, so nothing in the
ADR describes the running system. Deployment status is explicit, and #921
(42 crates including bundled SQLite, ndarray and safetensors pulled into a
security gate's closure to reuse six pure-math items) should land first.

Also recorded: the recognition gap does not close by swapping the
detector. Recognition lives in the classes.rs marker list, not in
KeywordDetector, so a better detector changes nothing.

ADR-335 gains the #920 field evidence, which strengthens its own argument:
the flywheel's hard_regression gate is permanently stuck closed, rejecting
120 of 120 candidates across 30 unattended generations including 30 that
were strictly better on primary, recall, qps and cost, because it tests an
absolute threshold rather than comparing to baseline. That is the case for
per-repo declaration made empirically rather than rhetorically.

Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_016QSCkKnxDjqU49NVVpWMK5
This commit is contained in:
ruv 2026-08-23 10:24:11 -04:00
parent a51d84db20
commit 099340004c
2 changed files with 172 additions and 13 deletions

View file

@ -3,7 +3,7 @@
- **Status**: Proposed
- **Date**: 2026-08-23
- **Deciders**: RuV Perpetual Intelligence Runtime (PIR) Program
- **Related**: ADR-282 (research-gate), ADR-306 (dream-machine consolidation), ADR-313 (SHAPER loop), ADR-324 (external-grounding veto); `crates/ruvector-sota-bench/harness/`; `schemas/research-manifest-v1.json`; `scripts/research-gate/`; see `docs/research/perpetual-intelligence-runtime/12-wave5-evidence-review.md` and `13-wave5-program-plan.md`
- **Related**: ADR-282 (research-gate), ADR-306 (dream-machine consolidation), ADR-313 (SHAPER loop), ADR-324 (external-grounding veto); `crates/ruvector-sota-bench/harness/`; `schemas/research-manifest-v1.json`; `scripts/research-gate/`; **#920** (flywheel `hard_regression` gate stuck closed — field evidence for this ADR); see `docs/research/perpetual-intelligence-runtime/12-wave5-evidence-review.md` and `13-wave5-program-plan.md`
- **Tags**: pir, wave-5, optimization, manifest, benchmark, darwin, research-gate
## Context
@ -65,6 +65,23 @@ TypeScript or specified per-experiment in
`schemas/research-manifest-v1.json`. A second repository adopting this
loop today would have to fork the harness.
**Field evidence for why declaration matters — issue #920.** Running the
Wave-5 acceptance test surfaced that the flywheel's `hard_regression` gate
is **permanently stuck closed**: **120 of 120 candidates rejected across 30
unattended generations**, including **30 candidates that were strictly
better on primary, recall, qps *and* cost.** The cause is that the gate
tests an **absolute threshold** rather than comparing against the
baseline.
This is the argument for this ADR made empirically rather than
rhetorically. A hard-coded absolute is a value that was correct for one
repository at one moment, frozen into TypeScript, silently wrong
thereafter — and wrong in the direction that rejects genuine improvements
without saying so. It is precisely the class of value that belongs in a
per-repo declaration, where it is visible, versioned, and reviewable.
Tracked separately as **#920**; the fix is not in scope here, but the
manifest layer is what stops the next one recurring.
**Second gap: the Pareto frontier is not ours.** `darwin.ts` passes
`selection: "pareto"` to `evolve()` from the **external** npm package
`@metaharness/darwin@^0.9.1`. This repo consumes a frontier

View file

@ -3,7 +3,7 @@
- **Status**: Proposed
- **Date**: 2026-08-23
- **Deciders**: RuV Perpetual Intelligence Runtime (PIR) Program
- **Related**: ADR-331 (PIR — VoI cost-aware routing; **consumed as-is, not reimplemented**); ADR-317 (PIR — HarnessRisk lifecycle-security gate); `crates/ruvector-tiny-dancer-core/src/voi.rs`; `crates/mcp-gate/`; see `docs/research/perpetual-intelligence-runtime/12-wave5-evidence-review.md` and `13-wave5-program-plan.md`
- **Related**: ADR-331 (PIR — VoI cost-aware routing; **consumed as-is, not reimplemented**); ADR-317 (PIR — HarnessRisk lifecycle-security gate); `crates/ruvector-tiny-dancer-core/src/voi.rs`; `crates/mcp-gate/`; WP33 audit **#919** (the measured evidence and the four corrections recorded below); **#921** (dependency-closure extraction, gates deployment); see `docs/research/perpetual-intelligence-runtime/12-wave5-evidence-review.md` and `13-wave5-program-plan.md`
- **Tags**: pir, wave-5, monitoring, voi, escalation, mcp-gate, security
## Context
@ -94,11 +94,14 @@ Build an escalation ladder in `crates/mcp-gate/` that **consumes**
**discretionary** investigation only. This is the monitoring analog of
ADR-331's escalate-only invariant: the failure direction is
conservative by construction.
4. **Cost target, stated as a target and not as a claim**: under 5%
average monitoring overhead, measured against a real unmonitored
baseline on the identical path, with downgrade-only accounting — a
skipped inspection may never be scored as a cheap inspection. Until
measured, the figure is a design goal.
4. **Cost target, stated as a target and — as of the WP33 audit —
explicitly NOT a guarantee**: under 5% average monitoring overhead,
measured against a real unmonitored baseline on the identical path,
with downgrade-only accounting (a skipped inspection may never be
scored as a cheap inspection). **Nothing in the implementation enforces
this bound.** See "Measured overhead" below for the two numbers and
"The <5% target is unenforced" for why the honest configuration is
currently hard to reach.
5. **Four binding caveats inherited from the primitive**, each recorded in
`voi.rs` and each a real failure mode here:
- **Every rung must have `cost > 0`, or the caller must cap rounds.**
@ -127,6 +130,98 @@ Build an escalation ladder in `crates/mcp-gate/` that **consumes**
`ruvnet/midstream`. This ADR covers request- and tool-call-level
monitoring at the `mcp-gate` boundary.
## Implementation status and measured evidence (WP33 audit, issue #919)
This section records what the WP33 audit actually established, and — more
importantly — what it did **not**. Four of the items below are corrections
to claims that would otherwise have travelled downstream as facts.
### The enforcement guarantee held under brute force
The mandatory-inspection floor (Decision §3) was attacked exhaustively
rather than sampled: **24,300 constructible configurations × 5 mandatory
operations = 121,500 checks, with ZERO bypasses.** Verified clean
alongside it: the forced purchase, classification-before-detection
ordering, union-not-first-match classification, `#[must_use]` on the
outcome type, the `cost > 0` requirement paired with an independent round
cap, the oracle exit and its non-finite guard, and unit-range rescaling.
**This is the acceptance evidence for the enforcement claim**, and it is
strong. Every caveat below concerns measurement, calibration, deployment,
or recognition — none of them weakens it.
### Measured overhead — two numbers, and only one of them is the target's
| Measurement | Overhead | What it measures |
|---|---|---|
| Instant-return investigator fixture | **0.06%** | The ladder's *own* cost. Accounting genuinely wraps classification, detection, the VoI decision, investigation, and observation |
| Investigators that sleep their declared latency (5 ms cheap rung, 200 ms strong rung — modest for real verifiers) | **14.8%** | Roughly **3× the <5% target** |
**The 0.06% figure must never appear without this caveat.** It is honest
for what it measures and the accounting behind it is sound — but it is not
the number the <5% target is about. Quoting it alone would describe a
ladder whose investigators cost nothing, which is not a ladder anyone
would deploy.
### The <5% target is unenforced
`latency_price` defaults to `0.0`, so **a rung declaring ten seconds of
latency costs exactly what a 0 µs rung costs.** Only `max_rounds` bounds
wall-clock cost, and it bounds *count*, not *duration*.
Worse, the honest configuration is currently hard to reach: pricing
latency at a plausible $1/second causes **the shipped reference ladder to
be REFUSED at construction as miscalibrated.** So the two available states
today are "latency is free" or "the reference ladder will not build." That
is a real gap, not a tuning note, and the target must not be described as
a guarantee until it closes.
### "100% inspection of mandatory classes" is not currently a checkable metric
**The enforcement is airtight; the measurement of it is not.**
`mandatory_inspections` counts inspections **with no denominator** — there
is no counter for mandatory operations *seen*. A run of 200 operations
each matching all five mandatory classes reported
`mandatory_inspections = 0`. That behaviour is **correct** (they halted
fail-closed before reaching a purchase), but the resulting counter is
**indistinguishable from a ladder that silently stopped classifying**.
Until a seen-counter exists, "100% inspection" is a property of the code
established by the 121,500-check sweep, **not** a figure any deployment can
report from its own telemetry.
### The module is not wired into the gate's request path
`monitor` is not referenced from `server.rs`, `tools.rs`, `main.rs`, or
`types.rs`. **It is a library module with no call site.** Nobody should
read "mcp-gate inspects mandatory classes at 100%" as a statement about
the running gate — today it is a statement about a module the running gate
does not call.
### The recognition gap does not close by swapping the detector
This is the caveat most likely to be misread, so it is stated flatly: the
mandatory guarantee is airtight in **enforcement** and entirely dependent
on **recognition** — and recognition lives in **`classes.rs`'s own marker
list**, not in `KeywordDetector`. Replacing the detector with a better one
does not fix it.
Concretely, the plain-ASCII verbs `unlink`, `shred`, `rmdir`, `wipefs`,
`mkfs`, and `dd` currently match nothing (being fixed). Obfuscation
evasions remain an acknowledged, open gap. An operation the marker list
does not recognise is not a mandatory operation as far as the floor is
concerned — however good the detector in front of it.
### Dependency-closure gap — issue #921, and it gates deployment
`mcp-gate` currently pulls **42 crates** into a *security gate's*
dependency closure — including bundled SQLite, `ndarray`, and
`safetensors` — **solely to reuse six pure-math items from `voi.rs`.**
Extraction into a leaf crate is tracked as **#921** and **should land
before the ladder is wired into the real request path.** Consuming the
primitive rather than forking it (Decision §2) remains correct; the
current packaging of that consumption is not.
## Consequences
### Positive
@ -138,7 +233,8 @@ Build an escalation ladder in `crates/mcp-gate/` that **consumes**
escalation, so the two cannot drift into divergent cost heuristics.
- The unconditional floor means the economically-cheapest configuration is
still safe on the operations that matter, which is what makes an
adaptive budget defensible at all.
adaptive budget defensible at all. **This is now evidenced, not
asserted**: 121,500 exhaustive checks, zero bypasses.
### Negative
@ -146,13 +242,32 @@ Build an escalation ladder in `crates/mcp-gate/` that **consumes**
fail open.** Every one of the four caveats above is a way to
accidentally build a never-investigate switch. The unconditional floor
is the backstop, and it is why the floor is not negotiable.
- The "under 5%" target is unvalidated. It is materially below OpenAI's
~20%, but their denominator is a deliberately narrow high-risk slice
while ours would be broader — the two numbers are **not comparable**,
and this ADR does not claim to beat theirs.
- **The "under 5%" target is unvalidated *and* unenforced.** Measured at
0.06% with an instant-return fixture and **14.8% — roughly 3× target —
with investigators that sleep their declared latency**. `latency_price`
defaults to 0.0, so declared latency is free; pricing it honestly
refuses the reference ladder at construction. It is materially below
OpenAI's ~20%, but their denominator is a deliberately narrow high-risk
slice while ours would be broader — the two numbers are **not
comparable**, and this ADR does not claim to beat theirs.
- **The enforcement guarantee is not self-reportable.**
`mandatory_inspections` has no denominator, so a deployment cannot
distinguish "nothing mandatory occurred" from "classification silently
stopped". The property is established by the code sweep, not by
telemetry.
- **Nothing is deployed.** `monitor` has no call site in the gate's
request path, so none of the above currently describes the running
system.
- **Recognition, not detection, is the ceiling.** The marker list in
`classes.rs` decides what counts as mandatory; a better detector in
front of it changes nothing. Plain-ASCII destructive verbs currently
match nothing (being fixed), and obfuscation evasions are open.
- Risk-score calibration is the whole ballgame and is unsolved here. A
miscalibrated detector makes every downstream purchase decision wrong in
the same direction.
- **The dependency closure is wrong for a security gate** — 42 crates
including bundled SQLite, `ndarray`, and `safetensors`, to reuse six
pure-math items. Tracked as #921.
## Security / Validation Gates
@ -161,7 +276,31 @@ Build an escalation ladder in `crates/mcp-gate/` that **consumes**
destructive operations are inspected regardless of VoI output.
Regression-tested by configuring an economics that would suppress all
discretionary investigation and asserting the five classes are still
inspected.
inspected. **Status: verified clean** — 24,300 configurations × 5
mandatory operations = 121,500 checks, zero bypasses (WP33 audit, #919),
along with the forced purchase, classification-before-detection
ordering, union-not-first-match classification, `#[must_use]` outcome,
the `cost > 0` + round-cap pair, the oracle exit and its non-finite
guard, and unit-range rescaling.
- **Recognition coverage is a separate gate from enforcement** (open) —
the floor protects what `classes.rs`'s marker list recognises. Missing
plain-ASCII verbs (`unlink`, `shred`, `rmdir`, `wipefs`, `mkfs`, `dd`)
are being fixed; obfuscation evasion is open. **A better detector does
not close this** — the gap is in the marker list, not the detector.
- **A mandatory-operations-seen counter is required before "100%
inspection" may be reported from telemetry** (open) —
`mandatory_inspections` currently has no denominator, so zero is
ambiguous between "none occurred" and "classification stopped".
Enforcement is airtight; the metric is not yet checkable.
- **Latency must be priced before the <5% target may be described as
enforced** (open) — `latency_price` defaults to 0.0, and pricing it
honestly refuses the reference ladder at construction. Until both are
resolved, <5% is a design goal with a measured 14.8% counterexample
under realistic investigator latency.
- **#921 (dependency-closure extraction) should land before the ladder is
wired into the real request path** — a security gate should not carry
bundled SQLite, `ndarray`, and `safetensors` to reuse six pure-math
items.
- **Termination** (blocking, Decision §5) — every rung `cost > 0` or an
explicit round cap, validated at ladder construction, not at first use.
- **Oracle rungs exit the protocol** (blocking) — a `noise_std == 0`
@ -182,6 +321,9 @@ Build an escalation ladder in `crates/mcp-gate/` that **consumes**
- `ruvnet/ruvector``crates/mcp-gate/` (detector, ladder, accounting).
Consumes `crates/ruvector-tiny-dancer-core/src/voi.rs` unchanged.
**Deployment status: not wired.** `monitor` has no call site in
`server.rs`, `tools.rs`, `main.rs`, or `types.rs` — it is a library
module the running gate does not call. #921 should land first.
- `ruvnet/midstream` — token-level stream inspection, **out of scope this
wave**, recorded so the boundary is explicit.