ruvnet
|
3a1afa2284
|
feat(rulake): vector-native federation intermediary — ADR-155 + MVP crate
Implements the M1 scope of docs/research/ruLake/ as an intermediary that
fans out vector queries across heterogeneous backends (Parquet, BigQuery,
Snowflake, Delta, Iceberg, local) behind a single RVF wire protocol, with
a RaBitQ-compressed cache in front.
## What ships
- **Research docs** under docs/research/ruLake/ (9 files, ~2.5k lines),
reframed from the earlier "plug RVF into BigQuery" shape to the
intermediary/federation shape. BigQuery-native compute becomes a Tier-2
push-down optimization inside the BigQueryBackend adapter, not a new
product shape.
- **ADR-155 v2** as "Proposed" — captures the seven alternatives
considered (plug-in-per-lake, standalone vector DB, Iceberg extension,
Trino connector, JVM intermediary, notebook-only, push-through-only),
consequences, and eight open questions.
- **crates/ruvector-rulake/** — new workspace member:
- `BackendAdapter` trait with minimum surface (id / list_collections /
pull_vectors / generation / supports_pushdown).
- `LocalBackend` in-memory reference implementation (thread-safe).
- `VectorCache` wrapping ruvector_rabitq::RabitqPlusIndex, with per-
collection generation tracking and `Consistency::{Fresh, Eventual}`
policies.
- `RuLake` entry point: register backends, search single or federated,
cache-stats introspection.
- 7 smoke tests (`tests/federation_smoke.rs`): byte-exact match vs
direct RaBitQ, cache-coherence after backend mutation, cross-backend
fan-out with correct score ordering, cache-hit-faster-than-miss,
three error-path tests.
- `rulake-demo` bin: unified benchmark producing the same-run table in
BENCHMARK.md.
## Measured numbers (LocalBackend, D=128, rerank×20, 300 queries)
| n | direct RaBitQ+ QPS | ruLake Fresh QPS | ruLake Eventual QPS | tax |
|--------:|-------------------:|-----------------:|--------------------:|------:|
| 5,000 | 17,311 | 17,874 | 17,858 | 0.97× |
| 50,000 | 5,162 | 5,123 | 5,050 | 1.01× |
| 100,000 | 3,122 | 3,117 | 3,114 | 1.00× |
**Intermediary tax is effectively zero on a local backend.** Federated
across 2 shards: 2,470 QPS @ n=100k (0.79× of single-shard); 4 shards:
1,781 QPS (0.57×) — sequential fan-out, parallel merge is the v2
optimisation per ADR-155 §Consequences.
## Build + test status (this crate only)
```
cargo build -p ruvector-rulake --release ✓
cargo test -p ruvector-rulake --release ✓ 7 passed
cargo clippy -p ruvector-rulake --release --all-targets -- -D warnings ✓ clean
cargo fmt -p ruvector-rulake -- --check ✓ clean
cargo run -p ruvector-rulake --release --bin rulake-demo ✓ reproduces BENCHMARK.md
```
## Scope this commit does NOT cover (M2-M5, see 07-implementation-plan.md)
- ParquetBackend, BigQueryBackend, SnowflakeBackend, IcebergBackend,
DeltaBackend (real-backend adapters).
- Push-down paths into backends with native vector ops.
- Governance / RBAC / PII / lineage / audit (M4).
- SIFT1M recall measurement on the real-backend path.
- Parallel fan-out via rayon.
- LRU cache eviction.
Co-Authored-By: claude-flow <ruv@ruv.net>
|
2026-04-23 18:38:49 -04:00 |
|