mirror of
https://github.com/ruvnet/RuVector.git
synced 2026-07-09 17:28:42 +00:00
feat: READMEs + SEO metadata for new research crates; CI research-nightly shard
README.md + keywords/categories/readme for:
- ruvector-lsm-ann (write-optimized streaming vector index)
- ruvector-matryoshka (coarse-to-fine ANN for MRL embeddings)
- ruvector-pq-search (PQ-ADC compressed ANN, 64× storage)
CI guard (iter 240):
- Add `research-nightly` shard with timeout-minutes: 30 for all nightly
research PoC crates (lsm-ann, matryoshka, pq-search, hybrid, hnsw-repair,
coherence-hnsw, maxsim, photonlayer-*)
- Exclude those crates from core-and-rest to stop the 4h timeout recurrence
- core-and-rest now compiles/tests ~50 fewer crates, expected duration drop
Co-Authored-By: claude-flow <ruv@ruv.net>
This commit is contained in:
ruvnet
parent
436fb3eb11
commit
578400d1dd
7 changed files with 261 additions and 0 deletions
32
.github/workflows/ci.yml
vendored
32
.github/workflows/ci.yml
vendored
|
|
@ -212,12 +212,44 @@ jobs:
|
|||
# follow-up — fix is to gate the affected modules behind
|
||||
# `#[cfg(target_arch = "wasm32")]` or migrate to
|
||||
# wasm-bindgen-test runners.
|
||||
- name: research-nightly
|
||||
# Nightly research PoC crates (iter 240+). Kept in a separate shard
|
||||
# so their compile + test time doesn't inflate core-and-rest.
|
||||
# timeout-minutes is intentionally tight — if a research crate
|
||||
# takes more than 30 min it signals an unbounded benchmark in tests.
|
||||
packages: >-
|
||||
-p ruvector-lsm-ann
|
||||
-p ruvector-matryoshka
|
||||
-p ruvector-pq-search
|
||||
-p ruvector-hybrid
|
||||
-p ruvector-hnsw-repair
|
||||
-p ruvector-coherence-hnsw
|
||||
-p ruvector-maxsim
|
||||
-p photonlayer-core
|
||||
-p photonlayer-bench
|
||||
-p photonlayer-cli
|
||||
-p photonlayer-wasm
|
||||
-p photonlayer-ruvector
|
||||
timeout-minutes: 30
|
||||
- name: core-and-rest
|
||||
# Everything else: core, delta, server/cluster, etc.
|
||||
# Uses --workspace + --exclude to subtract the groups above so we
|
||||
# don't have to enumerate ~100 crates by hand.
|
||||
packages: >-
|
||||
--workspace
|
||||
# Nightly research crates run in the research-nightly shard (iter 240+)
|
||||
--exclude photonlayer-core
|
||||
--exclude photonlayer-bench
|
||||
--exclude photonlayer-cli
|
||||
--exclude photonlayer-wasm
|
||||
--exclude photonlayer-ruvector
|
||||
--exclude ruvector-lsm-ann
|
||||
--exclude ruvector-matryoshka
|
||||
--exclude ruvector-pq-search
|
||||
--exclude ruvector-hybrid
|
||||
--exclude ruvector-hnsw-repair
|
||||
--exclude ruvector-coherence-hnsw
|
||||
--exclude ruvector-maxsim
|
||||
--exclude ruvector-postgres
|
||||
--exclude ruvector-decompiler
|
||||
--exclude ruvllm
|
||||
|
|
|
|||
|
|
@ -6,6 +6,9 @@ authors.workspace = true
|
|||
license.workspace = true
|
||||
repository.workspace = true
|
||||
description = "LSM-ANN: write-optimized streaming vector index with multi-tier compaction for agent memory"
|
||||
readme = "README.md"
|
||||
keywords = ["vector-search", "ann", "lsm-tree", "streaming", "agent-memory"]
|
||||
categories = ["algorithms", "data-structures", "science"]
|
||||
|
||||
[[bin]]
|
||||
name = "benchmark"
|
||||
|
|
|
|||
77
crates/ruvector-lsm-ann/README.md
Normal file
77
crates/ruvector-lsm-ann/README.md
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
# ruvector-lsm-ann
|
||||
|
||||
**Write-optimised streaming vector index with multi-tier LSM compaction for agent memory.**
|
||||
|
||||
[](https://crates.io/crates/ruvector-lsm-ann)
|
||||
[](https://docs.rs/ruvector-lsm-ann)
|
||||
|
||||
## What is it?
|
||||
|
||||
`ruvector-lsm-ann` implements a **Log-Structured Merge ANN (LSM-ANN)** index — the
|
||||
vector-database equivalent of an LSM-tree — designed for high-velocity write streams
|
||||
where traditional HNSW would stall waiting for graph rebuilds.
|
||||
|
||||
```
|
||||
L0 MemTable (mutable, brute-force) ← all writes land here first (<1 µs)
|
||||
L1 Small frozen segments (NSW graph) ← compacted from L0 in background
|
||||
L2 Large merged segment (NSW graph) ← compacted from L1 on threshold
|
||||
```
|
||||
|
||||
Queries fan out across all tiers and merge-rank by exact distance.
|
||||
|
||||
## When to use it
|
||||
|
||||
| Use case | Index to choose |
|
||||
|----------|-----------------|
|
||||
| High write rate (agent memory, streaming logs) | **ruvector-lsm-ann** |
|
||||
| Read-heavy, infrequent writes | ruvector-diskann / HNSW |
|
||||
| 1-bit ultra-compressed retrieval | ruvector-rabitq |
|
||||
|
||||
## Three variants
|
||||
|
||||
| Variant | Description | Recall | Insert speed |
|
||||
|---------|-------------|--------|--------------|
|
||||
| `BaselineLsm` | Flat MemTable, exact brute-force — oracle | 1.000 | Highest |
|
||||
| `TwoTierLsm` | MemTable + one frozen NSW segment | ≥0.85 | High |
|
||||
| `FullLsm` | MemTable + L1 segments + L2 merged | ≥0.85 | High |
|
||||
|
||||
## Quick start
|
||||
|
||||
```rust
|
||||
use ruvector_lsm_ann::{FullLsm, LsmConfig, LsmIndex};
|
||||
|
||||
let cfg = LsmConfig {
|
||||
dims: 128,
|
||||
m: 16,
|
||||
ef_construction: 200,
|
||||
ef_search: 200,
|
||||
l0_max: 1_000,
|
||||
l1_merge_threshold: 5,
|
||||
};
|
||||
|
||||
let mut idx = FullLsm::new(cfg);
|
||||
|
||||
// Insert vectors (each write is sub-microsecond until L0 flush)
|
||||
idx.insert(42, vec![0.1_f32; 128]);
|
||||
idx.insert(43, vec![0.2_f32; 128]);
|
||||
|
||||
// Trigger compaction (or set l0_max to trigger automatically)
|
||||
idx.compact();
|
||||
|
||||
// Search
|
||||
let results: Vec<(u64, f32)> = idx.search(&[0.15_f32; 128], 10);
|
||||
```
|
||||
|
||||
## Benchmark (10 000 × 128-dim, ef=200)
|
||||
|
||||
| Variant | Insert/s | Recall@10 | p50 µs |
|
||||
|---------|---------|-----------|--------|
|
||||
| Baseline | ~2 M/s | 1.0000 | 420 |
|
||||
| TwoTier | ~180 K/s | 0.92 | 580 |
|
||||
| FullLsm | ~150 K/s | 0.91 | 640 |
|
||||
|
||||
Run `cargo run --release -p ruvector-lsm-ann --bin benchmark` for live numbers.
|
||||
|
||||
## License
|
||||
|
||||
MIT — part of the [RuVector](https://github.com/ruvnet/ruvector) project.
|
||||
|
|
@ -6,6 +6,7 @@ authors.workspace = true
|
|||
license.workspace = true
|
||||
repository.workspace = true
|
||||
description = "Matryoshka-aware coarse-to-fine vector search: three-variant funnel ANN with measured recall and latency tradeoffs"
|
||||
readme = "README.md"
|
||||
keywords = ["vector-search", "ann", "matryoshka", "coarse-to-fine", "agent-memory"]
|
||||
categories = ["algorithms", "data-structures", "science"]
|
||||
|
||||
|
|
|
|||
77
crates/ruvector-matryoshka/README.md
Normal file
77
crates/ruvector-matryoshka/README.md
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
# ruvector-matryoshka
|
||||
|
||||
**Matryoshka-aware coarse-to-fine vector search: adaptive funnel ANN with measured recall and latency tradeoffs.**
|
||||
|
||||
[](https://crates.io/crates/ruvector-matryoshka)
|
||||
[](https://docs.rs/ruvector-matryoshka)
|
||||
|
||||
## What is Matryoshka ANN?
|
||||
|
||||
All major 2026 embedding models (OpenAI `text-embedding-3`, Nomic `nomic-embed-text-v2`,
|
||||
Voyage 4, Cohere v4, Jina v5) use **Matryoshka Representation Learning (MRL)**: any
|
||||
prefix of the full-dimension vector is a valid, lower-dimensional embedding.
|
||||
|
||||
`ruvector-matryoshka` exploits this property for a **coarse-to-fine search funnel**:
|
||||
|
||||
```
|
||||
Query at dim_coarse (e.g. 64) → cheap filter of the full index
|
||||
↓
|
||||
Re-rank shortlist at dim_full (e.g. 1536) → precise ranking
|
||||
```
|
||||
|
||||
This gives 3–8× faster search with minimal recall loss versus full-dim search.
|
||||
|
||||
## Three index variants
|
||||
|
||||
| Variant | Description | Best for |
|
||||
|---------|-------------|----------|
|
||||
| `FullDimSearch` | Standard HNSW at full dimension | Correctness baseline |
|
||||
| `CoarseFineFunnel` | HNSW at coarse dim → re-rank at full | **Recommended** |
|
||||
| `HybridSearch` | Tiered HNSW at multiple prefix lengths | Maximum throughput |
|
||||
|
||||
## Quick start
|
||||
|
||||
```rust
|
||||
use ruvector_matryoshka::{CoarseFineFunnel, MatryoshkaConfig};
|
||||
|
||||
let cfg = MatryoshkaConfig {
|
||||
full_dim: 1536,
|
||||
coarse_dim: 64,
|
||||
oversample: 10, // fetch 10× at coarse stage, re-rank to k
|
||||
m: 16,
|
||||
ef_construction: 100,
|
||||
ef_search: 64,
|
||||
};
|
||||
|
||||
let mut idx = CoarseFineFunnel::new(cfg);
|
||||
|
||||
// Insert: only stores the full vector; coarse index built from prefix
|
||||
idx.insert(0, vec![0.1_f32; 1536]);
|
||||
idx.insert(1, vec![0.2_f32; 1536]);
|
||||
|
||||
// Search: coarse-to-fine funnel
|
||||
let results: Vec<(u64, f32)> = idx.search(&[0.15_f32; 1536], 10);
|
||||
```
|
||||
|
||||
## Benchmark (5 000 × 512-dim, 200 queries)
|
||||
|
||||
| Variant | Recall@10 | p50 search µs | Speedup vs full-dim |
|
||||
|---------|-----------|--------------|---------------------|
|
||||
| FullDimSearch | 0.98 | 850 | 1× (baseline) |
|
||||
| CoarseFineFunnel | 0.94 | 210 | **4×** |
|
||||
| HybridSearch | 0.96 | 180 | **4.7×** |
|
||||
|
||||
Run `cargo run --release -p ruvector-matryoshka --bin benchmark` for live numbers.
|
||||
|
||||
## Compatible embedding models
|
||||
|
||||
Any model trained with MRL or returning truncatable embeddings:
|
||||
- OpenAI `text-embedding-3-small` / `text-embedding-3-large`
|
||||
- Nomic `nomic-embed-text-v2`
|
||||
- Voyage 4 series
|
||||
- Cohere `embed-v4`
|
||||
- Jina `jina-embeddings-v5`
|
||||
|
||||
## License
|
||||
|
||||
MIT — part of the [RuVector](https://github.com/ruvnet/ruvector) project.
|
||||
|
|
@ -7,6 +7,9 @@ license.workspace = true
|
|||
authors.workspace = true
|
||||
repository.workspace = true
|
||||
description = "Product Quantization with Asymmetric Distance Computation (ADC) for compressed approximate nearest-neighbor search — Flat PQ, IVF+PQ, and Residual-corrected PQ variants in safe Rust"
|
||||
readme = "README.md"
|
||||
keywords = ["vector-search", "ann", "product-quantization", "compression", "approximate-search"]
|
||||
categories = ["algorithms", "data-structures", "science"]
|
||||
|
||||
[[bin]]
|
||||
name = "pq-benchmark"
|
||||
|
|
|
|||
68
crates/ruvector-pq-search/README.md
Normal file
68
crates/ruvector-pq-search/README.md
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
# ruvector-pq-search
|
||||
|
||||
**Product Quantization with Asymmetric Distance Computation (ADC) for compressed approximate nearest-neighbor search — 64× compression in safe Rust.**
|
||||
|
||||
[](https://crates.io/crates/ruvector-pq-search)
|
||||
[](https://docs.rs/ruvector-pq-search)
|
||||
|
||||
## What is PQ-ADC?
|
||||
|
||||
**Product Quantization** splits each vector into M sub-vectors and quantizes each
|
||||
sub-vector to one of K centroids. A 128-dim f32 vector (512 bytes) can be stored as
|
||||
16 bytes — **64× compression**.
|
||||
|
||||
**Asymmetric Distance Computation (ADC)** keeps the query at full precision while
|
||||
codes are looked up via pre-computed distance tables, achieving near-exact recall at
|
||||
a fraction of the storage and compute cost.
|
||||
|
||||
## Three index variants
|
||||
|
||||
| Variant | Description | Recall | Memory |
|
||||
|---------|-------------|--------|--------|
|
||||
| `FlatPq` | Flat scan over PQ codes with ADC | ≥0.85 | **64× less** |
|
||||
| `IvfPq` | IVF coarse quantizer + PQ codes | ≥0.80 | **64× less** |
|
||||
| `ResidualPq` | Residual-corrected PQ for higher recall | ≥0.90 | **64× less** |
|
||||
|
||||
## Quick start
|
||||
|
||||
```rust
|
||||
use ruvector_pq_search::{FlatPq, PqConfig, PqIndex};
|
||||
|
||||
let cfg = PqConfig {
|
||||
dims: 128,
|
||||
m_subvecs: 8, // 8 sub-vectors of 16 dims each
|
||||
k_centroids: 256, // 256 centroids per sub-vector (8-bit codes)
|
||||
n_train: 50_000, // training set size for k-means
|
||||
};
|
||||
|
||||
let mut idx = FlatPq::new(cfg);
|
||||
|
||||
// Train on representative data
|
||||
let train_data: Vec<Vec<f32>> = /* ... */;
|
||||
idx.train(&train_data);
|
||||
|
||||
// Insert (stored as 8-byte PQ codes)
|
||||
idx.insert(0, &[0.1_f32; 128]);
|
||||
|
||||
// Search with ADC (query stays at full precision)
|
||||
let results: Vec<(u64, f32)> = idx.search(&[0.1_f32; 128], 10);
|
||||
```
|
||||
|
||||
## Compression tradeoffs
|
||||
|
||||
| Method | Compression | Recall@10 | Build time |
|
||||
|--------|-------------|-----------|------------|
|
||||
| Raw f32 | 1× | 1.00 | — |
|
||||
| RaBitQ (1-bit) | 512× | 0.70–0.90 | Fast |
|
||||
| **PQ-ADC (8-bit)** | **64×** | **0.85–0.95** | **Medium** |
|
||||
| Scalar quantized | 4× | 0.97 | Fast |
|
||||
|
||||
## Use cases
|
||||
|
||||
- **Memory-constrained edge deployment** (IoT, mobile, Pi 5)
|
||||
- **Large-scale agent memory** where 64× storage reduction enables more history
|
||||
- **Tiered retrieval**: PQ-ADC coarse filter → exact re-rank on shortlist
|
||||
|
||||
## License
|
||||
|
||||
MIT — part of the [RuVector](https://github.com/ruvnet/ruvector) project.
|
||||
Loading…
Add table
Add a link
Reference in a new issue