Wires the previously-shipped rabitq::persist module into ruLake's
lake.rs as first-class cache-save/restore APIs. The architectural
blocker I've deferred across 3 waves is now closed.
=== Agent A: rabitq::RabitqPlusIndex::export_items() ===
crates/ruvector-rabitq/src/index.rs +1 method, +1 test.
Exposes `export_items() -> Vec<(usize, Vec<f32>)>` — each row as
(pos, original_vec) extracted from originals_flat with one clone per
row. Feeds directly into persist::save_index or
from_vectors_parallel_with_rotation. No new deps, no public API
breakage.
Regression test (`export_items_roundtrip_via_from_vectors_parallel`)
builds via serial add(), exports, rebuilds via the parallel path,
asserts byte-identical search results on 5 queries. Tests: 36 → 37.
=== Agent B: RuLake save_cache_to_dir + warm_from_dir ===
crates/ruvector-rulake/src/{cache.rs, lake.rs, tests/federation_smoke.rs}.
New API:
pub fn save_cache_to_dir(&self, key, dir) -> Result<PathBuf>
— writes dir/index.rbpx (atomic temp+rename+fsync) alongside
the table.rulake.json bundle sidecar. Uses export_items +
persist::save_index.
pub fn warm_from_dir(&self, key, dir) -> Result<usize>
— reads bundle, witness-verifies, loads index.rbpx via
persist::load_index, cross-checks dim+rerank_factor, installs
into cache via the new install_prebuilt path. Returns n vectors.
Does NOT require the backend to be registered — warm restart
without backend RTT is the point.
New on CacheStats: warm_installs counter (separate from primes so
warm-restart cost isn't confused with cold-prime cost).
New on VectorCache: install_prebuilt + install_prebuilt_interned —
insert a pre-built Arc<RabitqPlusIndex> at a known witness without
any prime-timer bookkeeping. Respects the LRU cap. Shared-entry
path reuses an existing witness entry if another pointer already
holds it (witness-addressed cache sharing remains the headline).
New test: `warm_from_dir_skips_backend_and_returns_bit_exact_results`
Prime a 50-vec D=8 collection, save, spin up a FRESH RuLake with
NO backend registered + Consistency::Frozen, warm_from_dir, run the
same query, assert byte-identical ids + f32 score bits,
warm_installs=1, primes=0. Closes the "restart without re-prime"
gap end-to-end.
Documented limitation: pos_to_id reconstructed as (0..n) identity
because RabitqPlusIndex doesn't expose outer ids() accessor, and
the rabitq agent's scope prohibited adding it. Every current prime
path uses positional ids so this is byte-equivalent to the real
ids; external non-dense u64 ids would collapse (a known M2+ issue
filed inline).
Tests: 37 rabitq + 21 rulake lib + 22 rulake federation = 80 total.
Clippy -D warnings clean across both crates.
Co-Authored-By: claude-flow <ruv@ruv.net>
Ships the runnable acceptance test ADR-156 spec'd. Drives a single
LocalBackend through the full substrate contract in one test:
1. Recall: search_one → results
2. Verify: publish_bundle → read_from_dir → verify_witness
→ cache pointer matches on-disk witness
3. Forget: invalidate_cache → pointer is None
4. Rehydrate: next search_one → primes+1, pointer reinstalled
5. Location- results before forget ≡ results after rehydrate
transparency (byte-exact ids + scores at the same seed); the
caller never touched data_ref or knew which tier
served the call
6. Compact: explicitly out of scope per ADR-156 — belongs to
RVM/Cognitum, not the substrate
If this test stays green on every commit, the agent-facing memory
substrate claim is mechanical, not aspirational.
Also closes ADR-156 open question #4 (substrate test needed) as
resolved.
21 federation + 9 bundle + 3 fs_backend = 33 tests passing. Clippy
-D warnings clean.
Co-Authored-By: claude-flow <ruv@ruv.net>
Lets operators see which backend is hot (high hit_rate) vs cold (high
miss+prime cost). The global CacheStats are unchanged; the per-backend
counters are populated lazily on first activity against a given
backend id.
Attribution touches four events: hits, misses, primes, and
invalidations. Shared-hits (witness-match cache shares) attribute to
the *receiving* backend — that's the one that saved the prime work.
Mark_hit/mark_miss take &CacheKey so attribution is explicit, not
threaded through shared state.
API:
- VectorCache::stats_by_backend() -> HashMap<BackendId, PerBackendStats>
- RuLake::cache_stats_by_backend() (thin delegate)
- PerBackendStats::hit_rate() mirrors CacheStats::hit_rate()
Test cache_stats_by_backend_attributes_hits_to_the_right_backend:
two backends, hammer one, cold one stays at 0 hits. Hot hit_rate ≥
0.95 after warmup; both primed once.
Motivation (ADR-157): kernel dispatch decisions need per-backend
signals — a cold backend with high miss rate should not trigger GPU
dispatch the same way a hot backend serving at high hit rate does.
Also generally useful for capacity planning.
20 federation + 9 bundle + 3 fs_backend = 32 tests passing. Clippy
green.
Co-Authored-By: claude-flow <ruv@ruv.net>
Batched single-collection search that amortizes per-query overhead
across the batch:
- ensure_fresh() runs once, not N times (big for Fresh consistency
where it's a backend RTT each time).
- VectorCache::search_cached_batch takes the cache mutex once and
runs all N scans under that single acquisition.
- pos_to_id clone is amortized across the whole batch.
Layered trait shape (ADR-157 preparation):
RuLake::search_batch
└─ VectorCache::search_cached_batch
└─ N × RabitqPlusIndex::search (today — CPU per-query in loop)
A future VectorKernel trait plugs in under the innermost step. The
signature up to search_cached_batch is already kernel-agnostic —
GPU / SIMD kernels cross over CPU only at batch sizes ≥ their
min_batch, so a per-query API would never let dispatch pick them.
Tests:
- search_batch_matches_per_query_results: byte-exact parity with
search_one called individually. No recall loss.
- search_batch_acquires_cache_lock_once: a batch of 32 registers
as 1 coherence-skip hit (Eventual), not 32 — proves the
amortization is real, not nominal.
19 federation + 9 bundle + 3 fs_backend = 31 tests passing. Clippy
-D warnings clean.
Co-Authored-By: claude-flow <ruv@ruv.net>
Two changes from the 2026-04-22 strategic review reframing ruLake as
the memory substrate for agent brain systems:
1. Consistency::Frozen variant — caller asserts bundle immutability;
never automatic backend recheck. Maps to "Frozen for audit" from
the reviewer's three-mode product knob. Automatic coherence is
suppressed; explicit refresh_from_bundle_dir still works (lets
operators invalidate frozen caches without needing Fresh mode).
can_skip_check short-circuits when the pointer is already
installed — first prime still runs, subsequent queries never
round-trip to the backend.
Test frozen_consistency_never_rechecks_after_prime: prime → bump
backend → 10 warm searches still hit on the old witness, primes
stay at 1. Explicit refresh on a re-published bundle correctly
reports Invalidated, proving operator control remains.
2. ADR-156 — positioning addendum, not replacement of ADR-155.
ruLake stays as substrate (memory hierarchy); brain system stays
above (memory type, recall policy, mutation policy). Decomposes
the reviewer's "recall / verify / forget / compact / rehydrate"
acceptance test into six guarantees, five of which are shipped.
Rejects:
- absorbing the brain into ruLake (violates substrate separation)
- a new rulake-memory crate (premature; M1 primitives suffice)
- forking into two products (identical properties; no win)
17 federation + 9 bundle + 3 fs_backend = 29 tests passing. Clippy
green.
Co-Authored-By: claude-flow <ruv@ruv.net>
Ships the cross-crate fix that iter 12's concurrent bench identified:
K-shard federation no longer pays K× the rerank cost.
Changes:
- rabitq: RabitqPlusIndex::search_with_rerank(query, k, rerank_factor)
— non-mutating per-call override, same body as search(). The stored
field stays the default used by plain search().
- rulake: VectorCache::search_cached_with_rerank(key, q, k, rf_opt)
forwards through. search_cached() remains the default path.
- rulake: RuLake::search_federated uses an adaptive default of
max(MIN_PER_SHARD_RERANK=5, global / K). search_federated_with_rerank
lets callers override explicitly (None = adaptive, Some(global) =
byte-exact parity with single-shard).
Bench (n=100k, 8 clients × 300 queries, same box):
shards before QPS after QPS per-shard rerank
1 2,963 2,854 20
2 2,500 2,959 (1.04×) 10
4 1,778 2,791 (0.98×) 5
4-shard federation went from 0.60× the single-shard baseline to
0.98×. At 2 shards, the mutex serialization overhead even nets us
slightly above 1-shard. Federation is genuinely free now.
Recall gate: adaptive_per_shard_rerank_preserves_recall asserts
recall@10 ≥ 0.85 at K=2 and K=4 on clustered D=128 n=5k.
This closes the M2 cross-crate task filed in ADR-155 (iter 13). The
strategic review's "immediate optimization, high impact" is shipped.
27 → 28 tests passing. Clippy -D warnings clean in both crates.
Co-Authored-By: claude-flow <ruv@ruv.net>
Cache-first reframe (ADR-155) makes hit_rate the primary KPI. Before
this, operators were flying blind — the raw hits/misses counters told
them nothing about the 95% gate the acceptance test targets.
- CacheStats gets total_prime_ms + last_prime_ms (prime timed inside
prime() with Instant::now() around the lock-free build).
- CacheStats::hit_rate() → Option<f64> (None when no searches yet).
- CacheStats::avg_prime_ms() → Option<f64> (None when no primes).
Test stats_expose_hit_rate_and_prime_duration: 1 prime + 99 warm
queries → hit_rate ≥ 0.95, last_prime_ms in the expected range.
This is step 1 of the strategic reframe the latest review surfaced:
cache-coherent execution layer, federation as refill. The KPI it
exposes is what the 95% acceptance gate will measure.
15 federation + 9 bundle + 3 fs_backend = 27 passing. Clippy green.
Co-Authored-By: claude-flow <ruv@ruv.net>
Completes the sidecar loop (publish → disk → refresh). Given a key
and a directory, read the on-disk table.rulake.json and:
- UpToDate: witness matches cache pointer, nothing to do
- Invalidated: witnesses differ, cache pointer for key is dropped
- BundleMissing: no sidecar present (caller decides)
A corrupt/tampered sidecar surfaces as InvalidParameter via
RuLakeBundle::read_from_dir's witness verification — a poisoned
publish cannot silently invalidate the cache.
This is the minimal primitive a cache sidecar daemon needs. The
daemon itself is a ~10-line loop in user code: for each watched
(key, dir), call refresh_from_bundle_dir periodically or in
response to inotify events; handle the three outcomes.
Closes the "cache sidecar daemon protocol" open question from
ADR-155. The protocol is: filesystem-based, witness-authenticated,
atomic-write on publish, three-state on refresh.
14 federation + 9 bundle + 3 fs_backend = 26 tests passing.
Co-Authored-By: claude-flow <ruv@ruv.net>
Pairs with iter 4's read_from_dir: given a registered (backend,
collection) key, emit the current table.rulake.json to a directory.
This is what a cache sidecar daemon calls when the warehouse triggers
a bundle refresh — the daemon publishes the new bundle, any serving
ruLake watching that directory swaps in the new witness on next
search.
Does NOT prime the cache — publish is a metadata emission, not a
data load. That keeps publish cheap and lets operators stage bundle
updates without moving any compressed data.
Test publish_bundle_roundtrips_through_disk: publish → read_from_dir
on a third party → witness matches what a cache prime would see.
13 federation + 9 bundle + 3 fs_backend = 25 passing. Clippy green.
Co-Authored-By: claude-flow <ruv@ruv.net>
8 threads × 50 queries against a shared RuLake, alternating single-shard
and federated calls. Validates:
- no deadlocks (bounded time to completion)
- no panics from the cache Mutex or backend RwLock under contention
- every returned hit is finite and the per-call result is sorted
- prime count stays at ≤ 2 (one per shard) — hits serve the rest
Closes the M3 "concurrent multi-client throughput" smoke item from
BENCHMARK.md. The Send + Sync bound on RuLake is now exercised, not
just declared.
12/12 federation + 9 bundle + 3 fs_backend tests passing (24 total).
Clippy -D warnings green.
Co-Authored-By: claude-flow <ruv@ruv.net>
First concrete adapter that reads real persistent data. Uses a simple
'ruvec1' binary format (8-byte magic + u64 count + u32 dim + records)
and takes the mtime as the generation token. This proves the full
bundle → witness → cache → search loop works against the filesystem
without pulling arrow/parquet deps — a real ParquetBackend reuses the
exact same shape, only the decoder and generation source change.
- current_bundle() reads only the 24-byte header to pick up dim —
real-backend hot-path ergonomics; a full pull per coherence check
would be catastrophic on a warehouse adapter.
- Atomic write via temp+rename so concurrent reads never observe a
torn record stream (matches the bundle sidecar write pattern).
- data_ref is 'file://<path>', anchoring the witness on the local
filesystem location — two FsBackends pointing at the same file
share the cache entry (content-addressed, per ADR-155).
Tests:
- fs_write_then_pull_roundtrip: write vectors, read them back bitwise.
- fs_bundle_has_file_uri_and_header_dim: verify witness + data_ref.
- fs_pull_rejects_bad_magic: magic-byte guard on pull.
- fs_backend_end_to_end_search_and_recache_on_mtime_bump (federation
smoke): full RuLake → FsBackend → mtime bump → re-prime cycle.
23/23 passing (9 bundle + 3 fs_backend + 11 federation). Clippy green.
Co-Authored-By: claude-flow <ruv@ruv.net>
Direct dependency of the BQ UDF + cache sidecar daemon: the daemon
needs to read `table.rulake.json` off GCS (or a local mount) and
verify its witness before swapping in a new compressed entry.
- Atomic write via temp+rename so concurrent readers never see a
truncated sidecar (matches the pattern a warehouse-push path needs).
- Read verifies witness on-disk → malformed or tampered bundles
surface as InvalidParameter with a "witness" message.
- Canonical filename is exposed as SIDECAR_FILENAME so callers
don't hardcode the string.
Tests:
- fs_roundtrip: write + read preserves witness + optional fields.
- fs_read_rejects_tampered_sidecar: edit dim on disk → read errors.
- fs_write_is_atomic_under_crash_simulation: leftover .tmp.* files
don't corrupt reads of the canonical sidecar.
19/19 passing (9 bundle + 10 federation). Clippy -D warnings green.
Co-Authored-By: claude-flow <ruv@ruv.net>
MVP shipped an unbounded cache. v1 must-have: a hard cap on the number
of distinct compressed entries, evicting the least-recently-used
*unpinned* (refcount=0) entry when the cap is exceeded.
Design note: entries pinned by a live `(backend, collection)` pointer
are never evicted — dropping them would orphan a caller. If every
entry is pinned, the cap is temporarily exceeded rather than return
an error. Correctness over strict bounds.
API:
- `VectorCache::with_max_entries(n)` — builder-mode cap.
- `RuLake::with_max_cache_entries(n)` — user-facing constructor flag.
- `RuLake::invalidate_cache(key)` — drop a pointer explicitly so its
entry becomes evictable.
- `CacheEntry.last_used` bumped on every search_cached; LRU picks the
oldest unpinned entry as victim.
Eviction runs opportunistically at the end of each prime when a cap
is set. Zero overhead when `max_entries == None` (default path).
Test: `lru_eviction_caps_entry_count_when_pointers_dropped` pins three
entries, invalidates one, asserts the cap=2 holds after the next
prime runs the sweep.
16/16 tests pass. Clippy clean under -D warnings.
Co-Authored-By: claude-flow <ruv@ruv.net>
Implements the reviewer's "use RVF witness chain hash as cache-key
anchor" design. Cache entries are now keyed by the RuLakeBundle
witness, not (backend_id, collection). Two backends advertising the
same logical dataset (same data_ref + seed + rerank + generation)
produce the same witness and share one compressed index.
## The change
### BackendAdapter::current_bundle() (new trait method)
Returns the backend's authoritative bundle for a collection. Default
impl synthesizes from `id() + generation()`; real backends override to
report a shared data_ref when they're replicas of the same source of
truth. LocalBackend overrides to avoid the default's pull-to-read-dim
round-trip.
### VectorCache: two-layer storage
- `entries: HashMap<WitnessKey, CacheEntry>` — content-addressed
- `pointers: HashMap<CacheKey, WitnessKey>` — (backend, collection) → witness
- `last_checked: HashMap<CacheKey, Instant>` — for Eventual-mode TTL
`CacheEntry` now carries a `refcount` so an entry is GC'd only when
its last pointer drops. New stat: `shared_hits` — incremented when a
pointer move finds the target witness already cached.
### RuLake::ensure_fresh flow
1. Eventual within TTL → skip check (fast).
2. Witness matches pointer → hit, no-op.
3. Witness mismatch, target witness already in pool (another pointer
has it) → just swap the pointer, zero prime work. This is the
cross-backend share.
4. Witness not in pool → pull + prime as before.
### Prime is now race-tolerant
A concurrent thread racing to prime the same witness doesn't rebuild —
whichever thread gets the lock second observes the entry and drops
its own build. Two builds for the same witness are byte-identical by
determinism, so no data is lost.
## Test added
`two_backends_share_cache_when_witness_matches` — uses a
`SharedLocalBackend` shim that overrides `current_bundle()` to advertise
a shared data_ref. Two distinct `LocalBackend`s behind shims report
identical witnesses; the second search finds `primes=1, shared_hits=1`
and only ONE compressed entry in the pool despite two pointers. Both
pointers' `refcount_of(witness) == 2`.
## Lint + test status
```
cargo test -p ruvector-rulake --release ✓ 15/0
cargo clippy -p ruvector-rulake --release --all-targets -- -D warnings ✓ clean
cargo fmt -p ruvector-rulake -- --check ✓ clean
```
## Closes open question from earlier ADR review
"Cache invalidation drift" — the witness is now the cache-key anchor.
Backend generation bumps become witness changes; witness changes are
content-addressable so old entries can drop but shared ones survive.
"Where does freshness truth live?" — answered: in the bundle.
Co-Authored-By: claude-flow <ruv@ruv.net>
Applies the reviewer's architectural feedback (docs/research/ruLake/
chat thread): ruLake is a cache-first vector execution fabric, not a
federation engine. Federation is the cache's refill mechanism.
## Perf fix — cache prime now runs lock-free
`VectorCache::prime()` previously built a fresh `RabitqPlusIndex`
(~400 ms at n=100k) while holding the cache mutex, serialising all
other queries. Now builds entirely before touching `inner`; the lock
is only taken to swap the finished entry in. No benchmark regression —
intermediary tax still 1.00× on LocalBackend at n=100k.
## New: bundle sidecar (`table.rulake.json`)
`ruvector_rulake::bundle` — the portable unit that defines ruLake's
reproducibility + governance scope. Flagged by the reviewer as more
important than the UDF because it's what travels between teams,
clouds, and backups.
Carries: `data_ref`, `dim`, `rotation_seed`, `rerank_factor`,
`generation`, `rvf_witness` (SHAKE-256 over the preceding fields),
`pii_policy`, `lineage_id`.
`Generation` is a serde-untagged union of `Num(u64)` (Parquet mtime,
Iceberg version, Snowflake offset) and `Opaque(String)` (UUIDs,
hashes, base64 blobs) — fixes the "u64 doesn't fit an Iceberg snapshot
id" open question from the M1 review.
Witness fn is domain-separated, length-prefixed, and verifiable via
`bundle.verify_witness()`. 6 new tests: determinism,
field-change-detection, length-prefix-anti-collision, serde roundtrip,
tamper-detection, format-version-downgrade-rejected.
## New: recall-vs-brute-force gate
`rulake_recall_at_10_above_90pct_vs_brute_force` — the missing
correctness test. Builds brute-force L2 truth over 5k clustered
Gaussian vectors, asserts ruLake's top-10 hits ≥ 90% at rerank×20.
Uses the same n + cluster-count + methodology as
`ruvector-rabitq::BENCHMARK.md` so a regression shows up as a
divergence from the known-good estimator baseline.
## ADR-155 v2 — cache-first decision explicit
- Decision opens with "cache-first vector execution fabric; federation
is the refill mechanism", lifts the reviewer's 5-axis decision
matrix (cache-first wins 4/5 axes).
- New Decision §6 declares the bundle sidecar as the portable unit
(not the UDF) and documents how the witness acts as the cache-key
anchor, closing the "cache invalidation drift" failure mode.
## Test + lint status
```
cargo test -p ruvector-rulake --release ✓ 14/0
cargo clippy -p ruvector-rulake --release --all-targets -- -D warnings ✓ clean
cargo fmt -p ruvector-rulake -- --check ✓ clean
cargo run -p ruvector-rulake --release --bin rulake-demo -- --fast ✓ no regression
```
Co-Authored-By: claude-flow <ruv@ruv.net>
