3 commits

Author	SHA1	Message	Date
ruv	db047e0bf2	fix(sota-bench): validate every Pareto axis, and drop to two objectives ... Audit of the new Pareto gate found its headline safety property was implemented for only one of its three axes, and that when the guard did fire it took the run down with it. `dominates` returned out of its comparison loop on the first axis where the candidate was worse, before `readObjective` ran on the remaining ones, so the non-finite guard only ever inspected an axis the loop reached. Through the real promotion rule, a candidate carrying NaN in `costPerWin`, NaN in `p99Us`, -Infinity in `costPerWin`, or simply MISSING the `p99Us` field -- which is the shape an older replay bundle carries -- each produced `{promote: true, reasons: []}`. Nothing else caught it: the cost rule compares with `>`, and `NaN > x` is false. Both existing tests put the bad value in `primary`, the one axis where the guard worked, so they appeared to protect the property and did not. `assertComparable` now pre-reads every objective of every point before any comparison, in `dominates`, `frontier` and `admitToFrontier`, and the latter validates frontier members too. When the guard fired it threw out of the promotion rule. `@metaharness/ flywheel` calls the rule without a try/catch in both `run.js` and inside `verifyReplayBundle`, so a bad objective aborted the generation loop in-run and, in replay, prevented the structured `checks` output from ever being produced. The same class of data therefore yielded either a silent promotion or a dead run, and never "reject this candidate." The gate call site now catches and pushes `non_finite_objective`; the pure API still throws, which is right for a library and wrong for a gate. THE THIRD AXIS IS REMOVED, reversing this PR's own framing. Dominance requires "no worse on every axis", so each added axis is one more chance for that condition to fail: a Pareto gate weakens monotonically as axes grow, and this one was described as tightening. `p99Us` was also not independent evidence -- it is already a factor of `costPerWin` (`memoryMb * p99Us / qps`) and a term inside `primary`'s `darwinScore`, so it bought no information while costing real blocking power. Concretely, a candidate burning ten times the memory to halve tail latency is dominated and blocked under two axes and sailed through under three. A test pins both halves so the frontier cannot quietly become a second cost gate. The candidate can no longer choose its own judges: the `frontier` field is removed from the score outright rather than special-casing the empty array, because it rode on the object the evaluator produces and a deserialized bundle carrying `frontier: []` would have switched the gate off silently. The manifest's declared directions are now cross-checked against the gate's actual objectives, so a manifest can no longer declare an inverted direction that would invert promotion the day it is wired in, and declared bounds must be ones `normalizePolicy` can actually effect. The module now opens by stating plainly that it is inert: `declaredObjectives()` has no caller and schema registration happens by glob, so the schema constant was decorative -- a research-gate test now validates the fixture through it. All six negative checks bit before their fix. 157 of 158 harness tests pass (one pre-existing skip), 26 research-gate tests pass. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_016QSCkKnxDjqU49NVVpWMK5	2026-08-23 10:40:47 -04:00
ruv	0224fec1b4	feat(sota-bench): repo-native optimization manifest and in-repo Pareto frontier (PIR WP31, ADR-335) ... NVIDIA's Dynamo agent skillpack (ai-dynamo/dynamo PR #13557, 2026-08-21, Apache-2.0) makes an optimizing agent state its objective, build a benchmark, isolate one variable, and justify the spend before consuming GPU time. Most of that already existed here as code -- the closed lever set in benchmark.ts, the single-lever step in flywheel.ts, the paired bootstrap in statistics.ts, the conjunctive vetoes in vetoes.ts. What was missing is the ability for a repository to DECLARE those choices in one auditable place instead of having them implied by five modules. Adds schemas/optimization-manifest-v1.json: a per-repository standing policy declaring the objective and how it is computed, the benchmark identity, the tunable levers, the veto providers that must be composed, the cost budget, and the promotion rules. A per-experiment ADR-282 research manifest is derived from it; derivation emits only the fields the standing policy fixes and leaves genuinely per-run provenance (datasets, embedding space, environment) to the caller, because manufacturing those from a repository-level document would fabricate provenance rather than record it. The lever list now exists in two places -- the schema enum, for offline validation, and POLICY_LEVERS exported from benchmark.ts, which normalizePolicy actually enforces. The loader checks against POLICY_LEVERS rather than trusting the enum, and a drift test asserts the two lists stay identical, so a manifest can never declare a lever the native runner would reject at spawn time. Also adds an in-repo Pareto frontier. darwin.ts passes selection: "pareto" to the external @metaharness/darwin, so the frontier guiding Darwin's SEARCH is owned by a dependency; this one is what our own promotion rule consults, so a dependency bump cannot silently change what we are willing to promote. Both exist on purpose. ruvectorPromotionRule now refuses a candidate dominated on the objective vector (primary, costPerWin, p99Us) even when the paired test passes: a scalar gain bought with strictly worse cost and latency is a trade, not a win. A trade is explicitly not domination, so the frontier does not become a second hidden cost gate -- the existing cost rule still decides those. Non-finite objectives throw at a choke point before any comparison. NaN defeats both < and > in either direction, so a NaN candidate would otherwise read as non-dominated against everything. No existing gate is weakened: minimumEffect passes through unchanged, vetoes stay conjunctive, the closed lever set stays closed, and holdout/anchor disjointness is untouched. Scope, stated rather than implied: the manifest declares the frame and says nothing about strategy. NVIDIA's skillpack also carries domain-specific optimization knowledge and an adversarial-review step run by a separate agent; neither is encoded here. Their reported 15-77% throughput figure comes from two paired runs and is recorded nowhere in this change as a target. Verified: 150 harness tests (149 pass, 1 pre-existing skip), npm run check and doctor clean, cargo check -p ruvector-sota-bench --bin sota-all clean, 25 research-gate tests pass with the new schema compiled offline. Five negative checks were run -- removing the non-finite choke point, the POLICY_LEVERS enforcement, the schema/runner drift test, the Pareto gate, and the require_non_dominated check each fail the corresponding test. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_016QSCkKnxDjqU49NVVpWMK5	2026-08-23 10:00:21 -04:00
rUv	a2326c0449	feat: ADR-280/281/282 — durable RVF metadata, role-aware embeddings, nightly research quality gate (#774) ... Three ADRs implemented and hardened across five rounds of adversarial review, plus the fixes that review surfaced. **ADR-280 — durable RVF metadata.** Delta-encoded generations with a snapshot every 32. The first implementation wrote a full snapshot per commit and replayed every one at open: 600 commits produced a 725 MiB file that could no longer be opened, with no repair path. Now 241 KB of META payload for the same workload, opening in ~4 ms. Review also closed: derive-children that could not be reopened, an 80-byte file driving a 512 MiB allocation, delete() rollback leaving in-memory tombstones that bricked the artifact, ten BufWriter sites discarding flush errors before sync_all, corrupt mid-chain deltas made unopenable (now recovers the longest valid prefix), and an ordering bug where recovery pruning committed without its re-anchoring snapshot so `rvf ingest` printed a repair warning and then destroyed the file. **ADR-281 — role-aware embeddings.** Query/passage routing with an attested embedding-space identity. Review found the space id hashed CARGO_PKG_VERSION, so a routine version bump would have rejected every persisted corpus and invalidated every cache key — with the test suite structurally blind to it. Now keyed on a dedicated format revision with a golden-id test. Also: three constructors that failed unconditionally with ten unmigrated callers, prompt templates applied from the attested identity rather than hardcoded strings, and ApiEmbedding no longer bypassing templating. **ADR-282 — nightly research quality gate.** Review found the gate had never completed a single run: the candidate checkout was shallow so its git diff always failed, and a jq quoting bug made the override path dead code. Check-run queries were unpaginated — on a real main commit 8 of 22 failures were invisible, so a red base could be certified green. Schemas are now load-bearing with a hashed dependency closure. **CI note.** The two red checks are both pre-existing on main, not regressions from this branch: `Tests (core-and-rest)` routinely exceeds its 4-hour window, and `Hooks CI` has failed on main since 2026-08-02 (and in May) on `cp -r node_modules $GITHUB_WORKSPACE/npm/packages/cli/` in hooks-ci.yml — this branch's one-line version sync merely re-triggered its path filter. 72 checks pass. Follow-ups filed and not blocking: #770, #771, #772. 🤖 Generated with [claude-flow](https://github.com/ruvnet/claude-flow)	2026-08-03 14:13:37 -03:00