ruvector

vrr/ruvector

Fork 0

mirror of https://github.com/ruvnet/RuVector.git synced 2026-07-13 11:18:43 +00:00

Commit graph

Author SHA1 Message Date

Author	SHA1	Message	Date
Ofer Shaal	18dedfac7b	BET 5 (SepRAG #534 ): PQ/IVFADC within-list pruning vs tuned IVF nprobe — scale-gated WIN (ADR-206) (#542 ) * docs(bet4): pre-register LB-B&B IVF vs plain-IVF nprobe gate (FROZEN) Closes the BET 4 caveat left open by ADR-201: the region-pruning IVF kernel was only run against ACORN (BET 2), never against its natural incumbent, plain IVF nprobe, on unfiltered ANN. Frozen gate: WIN = >=2x member-scan reduction at matched recall@10 (R=0.95) AND wall-clock win across nclusters in {64,256,1024}; KILL = <1.5x or wall-clock reverses. Two controls: exact-vs-exact pruning-fraction probe + low-d (PCA-8) soundness control. Honest prior: NO-GO lean (128-d concentration makes the triangle-inequality bound loose) — the IVF-level companion to ADR-199. Branch off clean main; B&B kernel rebuilt self-contained (BET 2's lives only on #536). * feat(bet4): M0 — self-contained BnBIvf kernel + oracle gate (exactness certified) New crate ruvector-bet4-ivf-bench (deps: ruvector-rairs, rand). - data.rs: aligned arxiv 128-d feature CSV loader. - kernel.rs: BnBIvf — IVF probed in ascending lower-bound order with B&B early termination (break when LB >= kth-best); LB(q,c)=max(0,\|q-mu_c\|-r_c), r_c=max member radius. Full budget = exact; max_probe cap = nprobe analogue. Built on ruvector-rairs kmeans so it shares centroids with the IvfFlat incumbent (shared-index pre-reg requirement). - oracle.rs: brute-force exact kNN + recall@k + shared true-L2 helper. - M0 gate test PASSES on real arxiv slice: full-budget B&B == oracle (recall@10 >= 0.999) → B&B invariant certified. clippy clean. Frozen gate: docs/plans/bet4-ivf-pruning/PRE-REGISTRATION.md. Off clean main. * feat(bet4): M1 — instrumented plain-IVF incumbent on shared index + faithfulness gate BnBIvf::search_nprobe: the plain-IVF incumbent strategy (nprobe nearest centroids, scan all members, no B&B) on the SAME centroids/lists as the B&B contender, with member-eval counting. Refactored top-k accumulation into shared consider()/finalize() so both strategies accumulate identically and only the probe loop differs (shared-index pre-reg requirement). New gate instrumented_nprobe_matches_rairs PASSES: recall matches ruvector-rairs::IvfFlat within 0.01 at matched params → the cost-measured incumbent is algorithmically the real one. 3 tests green. * feat(bet4): M2/M3 — steelman B&B + PCA-8 control + matched-recall sweep - kernel: search_bnb_skip — the STEELMAN. Centroid-distance order (the effective nprobe ordering) + per-cluster LB-skip (correctness-safe in any order, unlike the LB-order global break). The strongest cluster-level B&B: if it can't beat tuned nprobe, the bound doesn't pay. - pca: minimal power-iteration top-m PCA (no linalg dep) for the low-dim control — projects real arxiv features to 8-d where the bound is tight. - examples/ivf_pruning_sweep: 3 contenders share one index per nclusters (plain nprobe / B&B LB-order / B&B steelman) x 2 regimes (128-d, PCA-8), exact-regime pruning probe, matched-recall@0.95, frozen-gate verdict. RESULT (n=20k & n=50k both): steelman = 1.00x evals vs nprobe in EVERY cell, BOTH regimes. NO-GO. Mechanism is structural, not dimensional: the LB bound only prunes FAR clusters that tuned nprobe already skips, so it's redundant with nprobe's centroid-distance cutoff. Exact-prune fraction scales correctly with dim (0-13% @128-d, 8-87% @PCA-8) => kernel sound; the redundancy is fundamental. LB-ORDER (faithful BET-2 kernel) is strictly WORSE (0.18-0.25x) — LB-ordering probes far large-radius clusters early. * docs(bet4): ADR-205 — cluster-pruning vs plain IVF nprobe = structural NO-GO Verdict: NO-GO (robust, structural). Steelman B&B (centroid order + LB-skip) ties tuned nprobe at exactly 1.00x member-evals in every cell, n=20k & n=50k, 128-d & PCA-8. Mechanism: the triangle-inequality bound only prunes FAR clusters that tuned nprobe already skips => redundant with nprobe's centroid-distance cutoff; win is structurally impossible, not just hard in high-d. LB-order (faithful BET-2 kernel) strictly worse (0.18-0.25x). Companion to ADR-199. Honest deviation recorded: the pre-registered PCA-8 control expected a B&B WIN (tight bound). It tied instead — the premise was false (tight bound beats full-scan, not tuned nprobe). Control still valid: exact-prune fraction scales correctly with dim (0-13% @128-d, 8-82% @PCA-8) => kernel sound; it revealed the structural redundancy. Scoreboard 2 WINS / 4 KILLS. * chore(bet4): lockfile for ruvector-bet4-ivf-bench workspace member * docs(bet5): FROZEN pre-registration — PQ/IVFADC within-list pruning vs tuned nprobe Opens the one lever ADR-205 left explicitly open (within-list PQ asymmetric distance, orthogonal to the killed cluster-level bound). Frozen gate: PQ must beat the cheaper of {plain full-L2, early-abandon exact-L2} nprobe by >=2x full-L2-equivalent member-evals at recall@10=0.95 AND wall-clock, across nclusters{64,256,1024} at >=1 scale N>=50k. Honest prior: ~55% win-at-scale, named kill-paths = amortization crossover + concentration re-rank ceiling. Stacked on feat/seprag-bet4-ivf-pruning to reuse ruvector-bet4-ivf-bench. Thread #534. * feat(bet5): M0 — PqIvf (IVFADC) kernel + early-abandon steelman + gate PqIvf trains m sub-quantizers on the shared ruvector-rairs k-means substrate (kmeans assignments ARE the PQ codes), encodes corpus to m-byte codes, and adds search_adc_rerank (cheap ADC scan of nprobe lists + exact L2 re-rank of top-R) plus search_adc_only (pure-ADC ceiling probe). AdcCost charges everything in one honest unit: 256 (LUT) + adc_membersm/D + rerank1 full-L2-equivalents. BnBIvf gains search_nprobe_abandon = the early-abandon exact-L2 steelman incumbent (user-confirmed verdict-setter), charged in dims_touched/D. Gates (real 2k arxiv slice): PqIvf shares centroids w/ BnBIvf; PQ@full-rerank exact (recall>=0.999); early-abandon exact vs full L2 (<0.001). 6 tests green, clippy clean. Thread #534, BET5 pre-reg frozen at `1d920b3a`. * feat(bet5): M1/M2/M3 — matched-recall PQ sweep harness examples/pq_pruning_sweep.rs: shared index per nclusters; tune incumbent nprobe to min reaching recall@10>=0.95; PQ scans the SAME nprobe lists (cannot rerank an unscanned neighbour) and we tune the smallest re-rank R recovering >=0.95. Charges all PQ ops in full-L2-equivalents (256 LUT + adcm/D + R rerank). Reports pure-ADC ceiling, R, early-abandon dim-prune fraction, wall-clock, crossover n, frozen gate. Thread #534. style(bet5): clippy-clean PQ kernel + sweep (iterator idioms, type alias) * perf(bet5): shared IvfParts — build k-means once per cell, not per contender Extract build_ivf -> IvfParts; BnBIvf::from_parts + PqIvf::from_parts reuse one seeded k-means for the incumbent and every PQ(m). Cuts the worst cell (nc=1024 @100k) from 3x k-means to 1x while guaranteeing the shared-index property by construction. Behavior-preserving (N=5000 numbers identical). 6 tests green. * fix(bet5): charge routing (nclusters centroid evals) to both contenders Pre-reg accounting + 'no free routing' adversarial check require the nclusters query-centroid routing evals charged equally to incumbent AND PQ. Harness omitted it, silently flattering PQ where routing dominates (high nclusters). Now prints member-only ratio (transparency) AND the gate-deciding TOTAL ratio with routing; verdict decided on total. Wall-clock already included routing (search computes centroid dists) so the wall guard was already honest. Re-run authoritative. * docs(bet5): ADR-206 — PQ/IVFADC within-list pruning = scale-gated WIN Opens ADR-205's one open lever (within-list PQ asymmetric distance, orthogonal to the killed cluster-level bound). PQ (cheap ADC scan + exact top-R rerank) beats tuned plain nprobe AND the early-abandon exact-L2 steelman by >=2x full-L2-equivalent member-evals at recall@10=0.95 AND wall-clock, across all three nclusters{64,256,1024} at N=100k. Win GROWS with N, crossover n* RISES with nclusters (routing amortization) -> >=2x at nclusters~sqrt(n) from n~20-50k. Honest caveats (none buried): win rides on the exact rerank not pure ADC (ceiling ~0.5) = IVFADC+refine validated, not a new method; scale-gated (full sweep only at 100k); nc=1024/100k knife-edge 2.03x; m=16 tuned; recall-floor tunability flatters PQ modestly; steelman halved the naive-L2 ratio. Routing charge bug in my own harness caught by the pre-registered 'no free routing' check (nc=1024/50k 2.24x member -> 1.65x total). Scoreboard 3 WINS / 4 KILLS. Thread #534, pre-reg frozen at `1d920b3a`. --------- Co-authored-by: ruv <ruvnet@users.noreply.github.com>	2026-06-17 22:48:32 -04:00
Ofer Shaal	dfe22d62a7	feat(bet1): productionize reuse-under-drift + validate on a real learned-GNN trajectory (ADR-202 WIN) (#537 ) * docs(bet1): pre-register reuse-under-drift gate on real GNN trajectory Productionize BET 1 (ADR-200 WIN under synthetic drift) by wiring re-weight + periodic-rebuild into the ruvector-diskann loop behind a feature flag, validated on a REAL contrastive-link-prediction embedding trajectory on ogbn-arxiv (ADR-200 next-step #4). Gate frozen before any contender run (prove-not-hype): WIN = ReweightOnly within 2% recall@10 of AlwaysRebuild + Periodic{k} within 1% at <=50% cumulative rebuild cost; KILL = no transfer from synthetic to real drift. Minimum-drift precondition (>=15% top-10 churn) guards against a vacuous pass. Self-contained off main; independent of PR #535. Outcome -> ADR-202. Linked: ruvnet/RuVector#534 * feat(diskann): M0 — reuse-under-drift policy module behind feature flag DriftingIndex wraps a VamanaGraph and owns only the rebuild decision (RebuildPolicy: AlwaysRebuild / ReweightOnly / Periodic{k}); the consumer owns the drifting vectors and passes snapshots to on_metric_update + search. Native reuse hook: greedy_search takes vectors externally, so adapt-to-drift recomputes only distances. Feature-gated (reuse-under-drift, default off) — default build byte-identical. 5 unit tests green (cadence + search). Refs ruvnet/RuVector#534 * feat(bet1): M1-M3 real-trajectory validation harness examples/diskann_real_trajectory.rs: generates a REAL learned-GNN metric trajectory via contrastive link-prediction (InfoNCE over ogbn-arxiv citations, ruvector-gnn Optimizer + info_nce_loss, embeddings on the unit sphere so cosine==dot and L2 ranking agrees), then drives the diskann reuse policy (DriftingIndex) through all four contenders step-by-step. Result (n=20k, gradual trajectory to 67% churn): - WIN. Reuse holds within 2% recall@10 of full rebuild up to 40% top-10 churn (>= ADR-200's synthetic ~36% regime) -- transfer confirmed on real learned drift. Stale control collapses 92%->33% (teeth). - Periodic recovers the high-churn tail: P k=4 = 98.7% (gap -0.01%) at 24% of rebuild cost, evals 1.00x B. ADR-200 hybrid reproduced on real drift. - Honest caveat: pure reuse past the ceiling decays (-4.73% over the whole overdriven trajectory, 1.05x evals); the shippable periodic policy does not. Refs ruvnet/RuVector#534 * style(bet1): rustfmt the reuse module + trajectory harness * docs(adr): ADR-202 — reuse-under-drift WIN on a real learned-GNN trajectory Outcome ADR for BET 1 productionization (closes ADR-200 next-step #4). Fixed-topology reuse + periodic rebuild, validated on a real contrastive- link-prediction trajectory over ogbn-arxiv (not synthetic A(t)). WIN at n=20k AND n=50k: pure reuse holds within 2% recall@10 of full rebuild up to a 40% top-10 churn ceiling (identical at both scales, >= ADR-200's synthetic ~36%); Periodic{k:4} recovers the high-churn tail to within 0.01% (20k) / above rebuild (50k) at 20-24% of rebuild cost, equal per-query work. Stale control collapses (teeth). Honest caveat: pure reuse past the ceiling decays -- the shippable policy is periodic, not never. Refs ruvnet/RuVector#534 * docs(bet1): record WIN outcome pointer to ADR-202 in pre-registration * docs(bet1): pre-register sampled-recall trigger gate + force_rebuild plumbing Pre-register (frozen before any run) the ADR-200 next-step #2 bet: does a sampled-recall rebuild trigger beat fixed Periodic{k} under VARIABLE-RATE drift, and beat the Frobenius monitor ADR-200 found wanting? Honest test = the (rebuilds, recall) Pareto frontier; WIN = trigger >=25% fewer rebuilds at matched recall with probe cost counted; KILL = no frontier dominance. Plumbing (allowed pre-freeze): DriftingIndex::force_rebuild + harness. Refs ruvnet/RuVector#534 * fix(bet1): trigger harness — Adam + enforced churn precondition (first run was VOID) The first variable-rate run was VOID (0% churn): plain SGD at lr 0.002-0.03 on unit-normalized embeddings doesn't move them. Switched to Adam (real motion in bursts), n=20k for edge density, and ENFORCED the >=15% churn precondition (abort before rendering a verdict) so a no-drift trajectory can't masquerade as a result. Gate criteria unchanged. Result (n=20k, bursty trajectory, per-step Δchurn ~45 burst / ~2 calm, 89% end churn): WIN. Recall{floor=0.95} = 97.2% @ 7 rebuilds beats Periodic{k=2} (96.8% @ 12) on BOTH axes; probe cost ~1s vs ~73s rebuild time saved (trap passed); beats best Frobenius (97.3% @ 9) on rebuilds. Refs ruvnet/RuVector#534 * feat(bet1): productionize RecallTrigger (WIN) + ADR-202 addendum The sampled-recall trigger WON (ADR-200 next-step #2): under bursty drift it uses ~42% fewer rebuilds than fixed Periodic{k} at matched recall, beats the Frobenius monitor ADR-200 found wanting, and passes the probe-cost trap (~1s probe vs ~73s rebuild saved). Productionized as RecallTrigger in ruvector_diskann::reuse (DriftingIndex in ReweightOnly mode + a probe-driven force_rebuild); its knob 'floor' IS the recall SLA, unlike k/tau. 8 reuse tests (incl. holds-under-no-drift + fires-then-recovers). ADR-202 addendum records the result; pre-registration carries the WIN outcome pointer. Refs ruvnet/RuVector#534 * docs(bet1): pre-register objective-dependence check + nodeclass trajectory Frozen-before-run generality check of ADR-202's 40% holding ceiling: does it generalize beyond contrastive link-prediction to a DIFFERENT learned objective? Adds a node-classification trajectory (real arxiv 40-class labels, CE on a linear head, embeddings as params) selectable via an 'objective=nodeclass' arg to the existing harness — same contenders + 2% gate, only the objective changes. CONFIRM = holding ceiling >=30% churn + periodic recovers; CAVEAT = <20% or materially different (reportable). Refs ruvnet/RuVector#534 * docs(bet1): objective-dependence CONFIRMED + class-collapse degeneracy caveat Node-classification trajectory (2nd objective) holds reuse within 2% of rebuild up to a 54% churn ceiling (>= link-pred's 40%) -> the ADR-202 holding-ceiling result GENERALIZES across two learned objectives; the objective-dependence caveat is resolved. Honest finding (reported, not buried): past ~60% churn node-class CE collapses embeddings into ~40 class blobs where recall@10 is ill-posed (intra-blob near-ties) and the FULL-REBUILD baseline itself destabilizes (B swings 55-96%). The trajectory-wide 'reuse > rebuild +4.3%' is a benchmark-degeneracy artifact (ADR-200's t=0.25 dip amplified), NOT a genuine superiority claim. Operational conclusion unaffected (reuse+periodic never worse). ADR-202 addendum + next-step #5 (collapse-aware metric). Refs ruvnet/RuVector#534	2026-06-17 20:18:50 -04:00
rUv	116fc9c7b4	chore(docs): Clean up and reorganize documentation structure Changes: - Remove outdated status/ directory (old build status from Dec 2) - Remove temporary fix docs (BENCHMARK_FIXES, quantization-fixes, SONA_NAPI_COMPLETE) - Move cognitive-frontier/ to research/cognitive-frontier/ - Move latent-space/ to research/latent-space/ - Move localkcut docs to research/mincut/ - Move PGLITE/WASM architecture docs to research/ - Move monitoring_example.md to examples/ - Move DEEP-OPTIMIZATION-ANALYSIS.md to optimization/ - Add subpolynomial-time-mincut plans to docs/plans/ - Update INDEX.md with new structure and version 0.1.29 Documentation structure now: - docs/research/ - All research docs (cognitive-frontier, latent-space, mincut, gnn-v2) - docs/examples/ - Example documentation - docs/optimization/ - Performance optimization - docs/plans/ - Implementation plans Reduced from 186 to 172 markdown files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	2025-12-25 19:39:44 +00:00

Ofer Shaal

18dedfac7b

BET 5 (SepRAG #534 ): PQ/IVFADC within-list pruning vs tuned IVF nprobe — scale-gated WIN (ADR-206) (#542 )

* docs(bet4): pre-register LB-B&B IVF vs plain-IVF nprobe gate (FROZEN)

Closes the BET 4 caveat left open by ADR-201: the region-pruning IVF
kernel was only run against ACORN (BET 2), never against its natural
incumbent, plain IVF nprobe, on unfiltered ANN. Frozen gate: WIN = >=2x
member-scan reduction at matched recall@10 (R=0.95) AND wall-clock win
across nclusters in {64,256,1024}; KILL = <1.5x or wall-clock reverses.
Two controls: exact-vs-exact pruning-fraction probe + low-d (PCA-8)
soundness control. Honest prior: NO-GO lean (128-d concentration makes
the triangle-inequality bound loose) — the IVF-level companion to
ADR-199. Branch off clean main; B&B kernel rebuilt self-contained
(BET 2's lives only on #536).

* feat(bet4): M0 — self-contained BnBIvf kernel + oracle gate (exactness certified)

New crate ruvector-bet4-ivf-bench (deps: ruvector-rairs, rand).
- data.rs: aligned arxiv 128-d feature CSV loader.
- kernel.rs: BnBIvf — IVF probed in ascending lower-bound order with B&B
  early termination (break when LB >= kth-best); LB(q,c)=max(0,|q-mu_c|-r_c),
  r_c=max member radius. Full budget = exact; max_probe cap = nprobe analogue.
  Built on ruvector-rairs kmeans so it shares centroids with the IvfFlat
  incumbent (shared-index pre-reg requirement).
- oracle.rs: brute-force exact kNN + recall@k + shared true-L2 helper.
- M0 gate test PASSES on real arxiv slice: full-budget B&B == oracle
  (recall@10 >= 0.999) → B&B invariant certified. clippy clean.

Frozen gate: docs/plans/bet4-ivf-pruning/PRE-REGISTRATION.md. Off clean main.

* feat(bet4): M1 — instrumented plain-IVF incumbent on shared index + faithfulness gate

BnBIvf::search_nprobe: the plain-IVF incumbent strategy (nprobe nearest
centroids, scan all members, no B&B) on the SAME centroids/lists as the
B&B contender, with member-eval counting. Refactored top-k accumulation
into shared consider()/finalize() so both strategies accumulate
identically and only the probe loop differs (shared-index pre-reg
requirement). New gate instrumented_nprobe_matches_rairs PASSES: recall
matches ruvector-rairs::IvfFlat within 0.01 at matched params → the
cost-measured incumbent is algorithmically the real one. 3 tests green.

* feat(bet4): M2/M3 — steelman B&B + PCA-8 control + matched-recall sweep

- kernel: search_bnb_skip — the STEELMAN. Centroid-distance order (the
  effective nprobe ordering) + per-cluster LB-skip (correctness-safe in
  any order, unlike the LB-order global break). The strongest cluster-level
  B&B: if it can't beat tuned nprobe, the bound doesn't pay.
- pca: minimal power-iteration top-m PCA (no linalg dep) for the low-dim
  control — projects real arxiv features to 8-d where the bound is tight.
- examples/ivf_pruning_sweep: 3 contenders share one index per nclusters
  (plain nprobe / B&B LB-order / B&B steelman) x 2 regimes (128-d, PCA-8),
  exact-regime pruning probe, matched-recall@0.95, frozen-gate verdict.

RESULT (n=20k & n=50k both): steelman = 1.00x evals vs nprobe in EVERY
cell, BOTH regimes. NO-GO. Mechanism is structural, not dimensional: the
LB bound only prunes FAR clusters that tuned nprobe already skips, so it's
redundant with nprobe's centroid-distance cutoff. Exact-prune fraction
scales correctly with dim (0-13% @128-d, 8-87% @PCA-8) => kernel sound;
the redundancy is fundamental. LB-ORDER (faithful BET-2 kernel) is strictly
WORSE (0.18-0.25x) — LB-ordering probes far large-radius clusters early.

* docs(bet4): ADR-205 — cluster-pruning vs plain IVF nprobe = structural NO-GO

Verdict: NO-GO (robust, structural). Steelman B&B (centroid order +
LB-skip) ties tuned nprobe at exactly 1.00x member-evals in every cell,
n=20k & n=50k, 128-d & PCA-8. Mechanism: the triangle-inequality bound
only prunes FAR clusters that tuned nprobe already skips => redundant with
nprobe's centroid-distance cutoff; win is structurally impossible, not
just hard in high-d. LB-order (faithful BET-2 kernel) strictly worse
(0.18-0.25x). Companion to ADR-199.

Honest deviation recorded: the pre-registered PCA-8 control expected a B&B
WIN (tight bound). It tied instead — the premise was false (tight bound
beats full-scan, not tuned nprobe). Control still valid: exact-prune
fraction scales correctly with dim (0-13% @128-d, 8-82% @PCA-8) => kernel
sound; it revealed the structural redundancy. Scoreboard 2 WINS / 4 KILLS.

* chore(bet4): lockfile for ruvector-bet4-ivf-bench workspace member

* docs(bet5): FROZEN pre-registration — PQ/IVFADC within-list pruning vs tuned nprobe

Opens the one lever ADR-205 left explicitly open (within-list PQ asymmetric
distance, orthogonal to the killed cluster-level bound). Frozen gate: PQ must
beat the cheaper of {plain full-L2, early-abandon exact-L2} nprobe by >=2x
full-L2-equivalent member-evals at recall@10=0.95 AND wall-clock, across
nclusters{64,256,1024} at >=1 scale N>=50k. Honest prior: ~55% win-at-scale,
named kill-paths = amortization crossover + concentration re-rank ceiling.
Stacked on feat/seprag-bet4-ivf-pruning to reuse ruvector-bet4-ivf-bench.
Thread #534.

* feat(bet5): M0 — PqIvf (IVFADC) kernel + early-abandon steelman + gate

PqIvf trains m sub-quantizers on the shared ruvector-rairs k-means substrate
(kmeans assignments ARE the PQ codes), encodes corpus to m-byte codes, and adds
search_adc_rerank (cheap ADC scan of nprobe lists + exact L2 re-rank of top-R)
plus search_adc_only (pure-ADC ceiling probe). AdcCost charges everything in one
honest unit: 256 (LUT) + adc_members*m/D + rerank*1 full-L2-equivalents.
BnBIvf gains search_nprobe_abandon = the early-abandon exact-L2 steelman
incumbent (user-confirmed verdict-setter), charged in dims_touched/D.

Gates (real 2k arxiv slice): PqIvf shares centroids w/ BnBIvf; PQ@full-rerank
exact (recall>=0.999); early-abandon exact vs full L2 (<0.001). 6 tests green,
clippy clean. Thread #534, BET5 pre-reg frozen at 1d920b3a.

* feat(bet5): M1/M2/M3 — matched-recall PQ sweep harness

examples/pq_pruning_sweep.rs: shared index per nclusters; tune incumbent nprobe
to min reaching recall@10>=0.95; PQ scans the SAME nprobe lists (cannot rerank an
unscanned neighbour) and we tune the smallest re-rank R recovering >=0.95. Charges
all PQ ops in full-L2-equivalents (256 LUT + adc*m/D + R rerank). Reports pure-ADC
ceiling, R*, early-abandon dim-prune fraction, wall-clock, crossover n*, frozen gate.
Thread #534.

* style(bet5): clippy-clean PQ kernel + sweep (iterator idioms, type alias)

* perf(bet5): shared IvfParts — build k-means once per cell, not per contender

Extract build_ivf -> IvfParts; BnBIvf::from_parts + PqIvf::from_parts reuse one
seeded k-means for the incumbent and every PQ(m). Cuts the worst cell (nc=1024
@100k) from 3x k-means to 1x while guaranteeing the shared-index property by
construction. Behavior-preserving (N=5000 numbers identical). 6 tests green.

* fix(bet5): charge routing (nclusters centroid evals) to both contenders

Pre-reg accounting + 'no free routing' adversarial check require the nclusters
query-centroid routing evals charged equally to incumbent AND PQ. Harness omitted
it, silently flattering PQ where routing dominates (high nclusters). Now prints
member-only ratio (transparency) AND the gate-deciding TOTAL ratio with routing;
verdict decided on total. Wall-clock already included routing (search computes
centroid dists) so the wall guard was already honest. Re-run authoritative.

* docs(bet5): ADR-206 — PQ/IVFADC within-list pruning = scale-gated WIN

Opens ADR-205's one open lever (within-list PQ asymmetric distance, orthogonal
to the killed cluster-level bound). PQ (cheap ADC scan + exact top-R rerank)
beats tuned plain nprobe AND the early-abandon exact-L2 steelman by >=2x
full-L2-equivalent member-evals at recall@10=0.95 AND wall-clock, across all
three nclusters{64,256,1024} at N=100k. Win GROWS with N, crossover n* RISES
with nclusters (routing amortization) -> >=2x at nclusters~sqrt(n) from n~20-50k.

Honest caveats (none buried): win rides on the exact rerank not pure ADC
(ceiling ~0.5) = IVFADC+refine validated, not a new method; scale-gated (full
sweep only at 100k); nc=1024/100k knife-edge 2.03x; m=16 tuned; recall-floor
tunability flatters PQ modestly; steelman halved the naive-L2 ratio. Routing
charge bug in my own harness caught by the pre-registered 'no free routing'
check (nc=1024/50k 2.24x member -> 1.65x total). Scoreboard 3 WINS / 4 KILLS.
Thread #534, pre-reg frozen at 1d920b3a.

---------

Co-authored-by: ruv <ruvnet@users.noreply.github.com>

2026-06-17 22:48:32 -04:00

Ofer Shaal

dfe22d62a7

feat(bet1): productionize reuse-under-drift + validate on a real learned-GNN trajectory (ADR-202 WIN) (#537 )

* docs(bet1): pre-register reuse-under-drift gate on real GNN trajectory

Productionize BET 1 (ADR-200 WIN under synthetic drift) by wiring
re-weight + periodic-rebuild into the ruvector-diskann loop behind a
feature flag, validated on a REAL contrastive-link-prediction embedding
trajectory on ogbn-arxiv (ADR-200 next-step #4).

Gate frozen before any contender run (prove-not-hype): WIN = ReweightOnly
within 2% recall@10 of AlwaysRebuild + Periodic{k} within 1% at <=50%
cumulative rebuild cost; KILL = no transfer from synthetic to real drift.
Minimum-drift precondition (>=15% top-10 churn) guards against a vacuous
pass. Self-contained off main; independent of PR #535. Outcome -> ADR-202.

Linked: ruvnet/RuVector#534

* feat(diskann): M0 — reuse-under-drift policy module behind feature flag

DriftingIndex wraps a VamanaGraph and owns only the rebuild decision
(RebuildPolicy: AlwaysRebuild / ReweightOnly / Periodic{k}); the consumer
owns the drifting vectors and passes snapshots to on_metric_update + search.
Native reuse hook: greedy_search takes vectors externally, so adapt-to-drift
recomputes only distances. Feature-gated (reuse-under-drift, default off) —
default build byte-identical. 5 unit tests green (cadence + search).

Refs ruvnet/RuVector#534

* feat(bet1): M1-M3 real-trajectory validation harness

examples/diskann_real_trajectory.rs: generates a REAL learned-GNN metric
trajectory via contrastive link-prediction (InfoNCE over ogbn-arxiv
citations, ruvector-gnn Optimizer + info_nce_loss, embeddings on the unit
sphere so cosine==dot and L2 ranking agrees), then drives the diskann
reuse policy (DriftingIndex) through all four contenders step-by-step.

Result (n=20k, gradual trajectory to 67% churn):
- WIN. Reuse holds within 2% recall@10 of full rebuild up to 40% top-10
  churn (>= ADR-200's synthetic ~36% regime) -- transfer confirmed on real
  learned drift. Stale control collapses 92%->33% (teeth).
- Periodic recovers the high-churn tail: P k=4 = 98.7% (gap -0.01%) at 24%
  of rebuild cost, evals 1.00x B. ADR-200 hybrid reproduced on real drift.
- Honest caveat: pure reuse past the ceiling decays (-4.73% over the whole
  overdriven trajectory, 1.05x evals); the shippable periodic policy does not.

Refs ruvnet/RuVector#534

* style(bet1): rustfmt the reuse module + trajectory harness

* docs(adr): ADR-202 — reuse-under-drift WIN on a real learned-GNN trajectory

Outcome ADR for BET 1 productionization (closes ADR-200 next-step #4).
Fixed-topology reuse + periodic rebuild, validated on a real contrastive-
link-prediction trajectory over ogbn-arxiv (not synthetic A(t)).

WIN at n=20k AND n=50k: pure reuse holds within 2% recall@10 of full
rebuild up to a 40% top-10 churn ceiling (identical at both scales, >=
ADR-200's synthetic ~36%); Periodic{k:4} recovers the high-churn tail to
within 0.01% (20k) / above rebuild (50k) at 20-24% of rebuild cost, equal
per-query work. Stale control collapses (teeth). Honest caveat: pure reuse
past the ceiling decays -- the shippable policy is periodic, not never.

Refs ruvnet/RuVector#534

* docs(bet1): record WIN outcome pointer to ADR-202 in pre-registration

* docs(bet1): pre-register sampled-recall trigger gate + force_rebuild plumbing

Pre-register (frozen before any run) the ADR-200 next-step #2 bet: does a
sampled-recall rebuild trigger beat fixed Periodic{k} under VARIABLE-RATE
drift, and beat the Frobenius monitor ADR-200 found wanting? Honest test =
the (rebuilds, recall) Pareto frontier; WIN = trigger >=25% fewer rebuilds
at matched recall with probe cost counted; KILL = no frontier dominance.

Plumbing (allowed pre-freeze): DriftingIndex::force_rebuild + harness.

Refs ruvnet/RuVector#534

* fix(bet1): trigger harness — Adam + enforced churn precondition (first run was VOID)

The first variable-rate run was VOID (0% churn): plain SGD at lr 0.002-0.03
on unit-normalized embeddings doesn't move them. Switched to Adam (real
motion in bursts), n=20k for edge density, and ENFORCED the >=15% churn
precondition (abort before rendering a verdict) so a no-drift trajectory
can't masquerade as a result. Gate criteria unchanged.

Result (n=20k, bursty trajectory, per-step Δchurn ~45 burst / ~2 calm,
89% end churn): WIN. Recall{floor=0.95} = 97.2% @ 7 rebuilds beats
Periodic{k=2} (96.8% @ 12) on BOTH axes; probe cost ~1s vs ~73s rebuild
time saved (trap passed); beats best Frobenius (97.3% @ 9) on rebuilds.

Refs ruvnet/RuVector#534

* feat(bet1): productionize RecallTrigger (WIN) + ADR-202 addendum

The sampled-recall trigger WON (ADR-200 next-step #2): under bursty drift it
uses ~42% fewer rebuilds than fixed Periodic{k} at matched recall, beats the
Frobenius monitor ADR-200 found wanting, and passes the probe-cost trap
(~1s probe vs ~73s rebuild saved). Productionized as RecallTrigger in
ruvector_diskann::reuse (DriftingIndex in ReweightOnly mode + a probe-driven
force_rebuild); its knob 'floor' IS the recall SLA, unlike k/tau. 8 reuse
tests (incl. holds-under-no-drift + fires-then-recovers). ADR-202 addendum
records the result; pre-registration carries the WIN outcome pointer.

Refs ruvnet/RuVector#534

* docs(bet1): pre-register objective-dependence check + nodeclass trajectory

Frozen-before-run generality check of ADR-202's 40% holding ceiling: does
it generalize beyond contrastive link-prediction to a DIFFERENT learned
objective? Adds a node-classification trajectory (real arxiv 40-class
labels, CE on a linear head, embeddings as params) selectable via an
'objective=nodeclass' arg to the existing harness — same contenders + 2%
gate, only the objective changes. CONFIRM = holding ceiling >=30% churn +
periodic recovers; CAVEAT = <20% or materially different (reportable).

Refs ruvnet/RuVector#534

* docs(bet1): objective-dependence CONFIRMED + class-collapse degeneracy caveat

Node-classification trajectory (2nd objective) holds reuse within 2% of
rebuild up to a 54% churn ceiling (>= link-pred's 40%) -> the ADR-202
holding-ceiling result GENERALIZES across two learned objectives; the
objective-dependence caveat is resolved.

Honest finding (reported, not buried): past ~60% churn node-class CE
collapses embeddings into ~40 class blobs where recall@10 is ill-posed
(intra-blob near-ties) and the FULL-REBUILD baseline itself destabilizes
(B swings 55-96%). The trajectory-wide 'reuse > rebuild +4.3%' is a
benchmark-degeneracy artifact (ADR-200's t=0.25 dip amplified), NOT a
genuine superiority claim. Operational conclusion unaffected (reuse+periodic
never worse). ADR-202 addendum + next-step #5 (collapse-aware metric).

Refs ruvnet/RuVector#534

2026-06-17 20:18:50 -04:00

rUv

116fc9c7b4

chore(docs): Clean up and reorganize documentation structure

Changes:
- Remove outdated status/ directory (old build status from Dec 2)
- Remove temporary fix docs (BENCHMARK_FIXES, quantization-fixes, SONA_NAPI_COMPLETE)
- Move cognitive-frontier/ to research/cognitive-frontier/
- Move latent-space/ to research/latent-space/
- Move localkcut docs to research/mincut/
- Move PGLITE/WASM architecture docs to research/
- Move monitoring_example.md to examples/
- Move DEEP-OPTIMIZATION-ANALYSIS.md to optimization/
- Add subpolynomial-time-mincut plans to docs/plans/
- Update INDEX.md with new structure and version 0.1.29

Documentation structure now:
- docs/research/ - All research docs (cognitive-frontier, latent-space, mincut, gnn-v2)
- docs/examples/ - Example documentation
- docs/optimization/ - Performance optimization
- docs/plans/ - Implementation plans

Reduced from 186 to 172 markdown files.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2025-12-25 19:39:44 +00:00

3 commits