mirror of
https://github.com/ruvnet/RuVector.git
synced 2026-05-24 22:15:18 +00:00
* feat(rvf): add RuVector Format universal substrate specification Research and design for RVF — a streaming, progressive, adaptive, quantum-secure binary format for vector intelligence. Covers append-only segment model, two-level tail manifests, temperature tiering, progressive HNSW indexing, epoch-based overlay system, SIMD-optimized query paths, WASM microkernel for Cognitum tiles, domain profiles (RVDNA, RVText, RVGraph, RVVision), and post-quantum cryptography. https://claude.ai/code/session_01DDqjGE51JpsRE3DgUjFyjW * feat(rvf): add deletion, filtered search, concurrency, and operations specs Fill four specification gaps in the RVF format design: - spec/07: Vector deletion lifecycle, JOURNAL_SEG wire format, deletion bitmaps - spec/08: Filtered search with META_SEG, METAIDX_SEG, filter expression language - spec/09: Writer locking, reader-writer coordination, versioning, space reclamation - spec/10: Batch operations API, error codes, network streaming protocol Also fixes the segment header field conflict between spec/01 and wire/binary-layout.md (checksum_algo/compression now u8, adds uncompressed_len at 0x38). https://claude.ai/code/session_01DDqjGE51JpsRE3DgUjFyjW * feat(rvf): add RuVector Format SDK, 40 examples, MCP server, and documentation Complete RVF implementation including: - 12 Rust crates (rvf-types, rvf-wire, rvf-manifest, rvf-index, rvf-quant, rvf-crypto, rvf-runtime, rvf-import, rvf-wasm, rvf-node, rvf-server, plus integration tests) - 40 runnable examples covering core storage, agentic AI, production patterns, vertical domains, exotic capabilities, runtime targets, network/security, POSIX/systems, and network operations - TypeScript SDK (npm/packages/rvf) with RvfDatabase class - MCP server (npm/packages/rvf-mcp-server) with stdio and SSE transports - Node.js N-API bindings (npm/packages/rvf-node) - WASM package (npm/packages/rvf-wasm) - ADR-029 (canonical format), ADR-030 (computational container), ADR-031 (example repository) - DNA-style lineage provenance, computational containers (KERNEL_SEG, EBPF_SEG), witness chains, TEE attestation, domain profiles - Superseded ADR annotations for ADR-001, ADR-005, ADR-006, ADR-018-021 Co-Authored-By: claude-flow <ruv@ruv.net> * feat(rvf): add CLI, WASM store, generate_all, and 46 output .rvf files - Add rvf-cli crate (665 lines, 9 subcommands: create/ingest/query/delete/status/inspect/compact/derive/serve) - Add WASM control plane store (alloc_setup, segment, store modules) for ~46 KB binary - Add generate_all.rs example producing 46 persistent .rvf files in output/ - Add Node.js N-API bindings for lineage, kernel/eBPF, and inspection - Add npm TypeScript backend/database/types for RVF integration - Update READMEs with CLI sections, MCP server docs, and crate map (13 crates) - All 40 examples verified passing Co-Authored-By: claude-flow <ruv@ruv.net> * feat(rvf): add Claude Code appliance, improve Quick Start, fix API docs - Add claude_code_appliance.rs: self-booting RVF with SSH + Claude Code install (curl -fsSL https://claude.ai/install.sh | bash), 3 SSH users, eBPF filter, 20-package manifest, witness chain, lineage snapshot - Improve Quick Start: Install section (crate/CLI/npm/WASM/MCP), WASM browser example, generate_all reference, expanded Rust crate deps - Fix embed_kernel/embed_ebpf API docs to match actual signatures (u8 params with `as u8` cast, 6-param kernel, Option<&[u8]> btf) - Update generate_all.rs: add claude_code_appliance generator (47 files) - Regenerate all 47 output .rvf files Co-Authored-By: claude-flow <ruv@ruv.net> * feat(rvf): add RVCOW branching, real kernel/eBPF/launcher, 795 tests Vector-native copy-on-write branching (ADR-031) with four new segment types (COW_MAP 0x20, REFCOUNT 0x21, MEMBERSHIP 0x22, DELTA 0x23), real Linux microkernel builder, QEMU microVM launcher, real eBPF programs, and 128-byte KernelBinding for tamper-evident kernel-manifest linkage. New crates: - rvf-kernel: Docker-based kernel build, real cpio/newc initramfs builder, SHA3-256 verification, prebuilt kernel support (37 tests) - rvf-launch: QEMU microVM launcher with QMP shutdown, KVM/TCG detection, virtio-blk/net port forwarding, kernel extraction (8 tests) - rvf-ebpf: 3 real BPF C programs (xdp_distance, socket_filter, tc_query_route) with clang compilation support (17 tests) RVCOW runtime: - CowEngine with read/write paths, write coalescing, snapshot-freeze - CowMap (flat-array), MembershipFilter (bitmap), CowCompactor - 3x read performance via pread optimization (1.3us/vector) - Branch creation: 2.6ms for 10K vectors, child = 162 bytes Security: 20-finding audit, 7 fixes applied including division-by-zero guards, integer overflow checks, and KernelBinding::from_bytes_validated(). CLI: 8 new commands (launch, embed-kernel, embed-ebpf, filter, freeze, verify-witness, verify-attestation, rebuild-refcounts), serve wired to real rvf-server. Co-Authored-By: claude-flow <ruv@ruv.net> * feat(rvf): update README, add crate/npm READMEs, publish to crates.io and npm - Rewrite README with cognitive container terminology, grouped features, 4 comparison tables (vs Docker, Vector DBs, Git LFS, SQLite), updated benchmarks, architecture diagram, and 45 examples - Add READMEs for rvf-kernel, rvf-launch, rvf-ebpf, rvf-import crates - Add READMEs for @ruvector/rvf, rvf-node, rvf-wasm, rvf-mcp-server npm packages - Fix Cargo.toml metadata (homepage, readme, categories, keywords) and add version specs to all path dependencies for crates.io publishing - Fix clippy warnings in rvf-kernel/initramfs.rs and rvf-launch/lib.rs - Published to crates.io: rvf-types, rvf-wire, rvf-manifest, rvf-quant, rvf-index, rvf-crypto (remaining crates pending rate limit) - Published to npm: @ruvector/rvf, @ruvector/rvf-node, @ruvector/rvf-wasm, @ruvector/rvf-mcp-server Co-Authored-By: claude-flow <ruv@ruv.net> * chore: add rvf-kernel, rvf-ebpf, rvf-launch, rvf-server, rvf-import, rvf-cli to workspace Include all 15 RVF crates plus integration tests and benchmarks in the root workspace members list so cargo publish can resolve them by name. Co-Authored-By: claude-flow <ruv@ruv.net> * feat(rvf): add published packages, cognitive container branding, grouped capabilities - Add Published Packages section with 13 crates.io + 4 npm tables - Add Platform Support table (Linux, macOS, Windows, WASM, no_std) - Expand capability table from 9 to 15 rows in 4 groups - Rewrite all "How" descriptions in plain language - Update .rvf diagram to show all 20 segment types - Rename ADRs: computational container -> cognitive container - Add emojis to all section headers Co-Authored-By: claude-flow <ruv@ruv.net> * feat: update root README with RVF cognitive containers, expanded capabilities - Update intro: "gets smarter + ships as cognitive container" - Add self-booting microservice row to Pinecone comparison table - Expand capabilities from 34 to 42 features with dedicated RVF section - Update "Think of it as" to include Docker comparison and RVF explanation - Add RVF collapsed group to Ecosystem (13 crates, 4 npm, install commands) - Add RVF to Platform & Edge section with install commands - Add RVF npm packages (4) and Rust crates (13) to package reference - Add RVF rows to feature comparison table (6 new rows) - Add ADR-030/031 to ADR list - Add RVF to Installation table, Project Structure - Update attention mechanisms count from 39 to 40+ - Update npm count to 49+, Rust crates to 83 - Update footer with crates.io and RVF links Co-Authored-By: claude-flow <ruv@ruv.net> * feat: expand comparison table with emojis, cost, audit, branching, single-file Co-Authored-By: claude-flow <ruv@ruv.net> * docs: rewrite comparison table in plain language Co-Authored-By: claude-flow <ruv@ruv.net> * chore: clean up empty code change sections in the changes log --------- Co-authored-by: Claude <noreply@anthropic.com>
128 lines
4.4 KiB
Rust
128 lines
4.4 KiB
Rust
//! Basic RVF Store — Getting Started
|
|
//!
|
|
//! Demonstrates the simplest possible RVF workflow:
|
|
//! 1. Create a store with RvfOptions (384 dims, L2 metric)
|
|
//! 2. Insert 100 random vectors with IDs
|
|
//! 3. Query top-5 nearest neighbors
|
|
//! 4. Print results (id + distance)
|
|
//! 5. Close and reopen the store
|
|
//! 6. Query again to show persistence
|
|
//! 7. Clean up
|
|
|
|
use rvf_runtime::{QueryOptions, RvfOptions, RvfStore, SearchResult};
|
|
use rvf_runtime::options::DistanceMetric;
|
|
use tempfile::TempDir;
|
|
|
|
/// Simple pseudo-random number generator (LCG) for deterministic results.
|
|
fn random_vector(dim: usize, seed: u64) -> Vec<f32> {
|
|
let mut v = Vec::with_capacity(dim);
|
|
let mut x = seed.wrapping_add(1);
|
|
for _ in 0..dim {
|
|
x = x.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407);
|
|
v.push(((x >> 33) as f32) / (u32::MAX as f32) - 0.5);
|
|
}
|
|
v
|
|
}
|
|
|
|
fn main() {
|
|
println!("=== RVF Basic Store Example ===\n");
|
|
|
|
let dim = 384;
|
|
let num_vectors = 100;
|
|
|
|
// -- Step 1: Create a temporary directory and a new store --
|
|
let tmp_dir = TempDir::new().expect("failed to create temp dir");
|
|
let store_path = tmp_dir.path().join("basic.rvf");
|
|
|
|
// Note: We use L2 (squared Euclidean) metric here because the current
|
|
// manifest format does not persist the metric choice. After reopening,
|
|
// the store defaults to L2, so using L2 ensures consistency.
|
|
let options = RvfOptions {
|
|
dimension: dim as u16,
|
|
metric: DistanceMetric::L2,
|
|
..Default::default()
|
|
};
|
|
|
|
println!("Creating store at {:?}", store_path);
|
|
println!(" Dimensions: {}", dim);
|
|
println!(" Metric: L2 (squared Euclidean)");
|
|
let mut store = RvfStore::create(&store_path, options).expect("failed to create store");
|
|
|
|
// -- Step 2: Insert 100 random vectors --
|
|
let vectors: Vec<Vec<f32>> = (0..num_vectors)
|
|
.map(|i| random_vector(dim, i as u64))
|
|
.collect();
|
|
let vec_refs: Vec<&[f32]> = vectors.iter().map(|v| v.as_slice()).collect();
|
|
let ids: Vec<u64> = (0..num_vectors as u64).collect();
|
|
|
|
let ingest_result = store
|
|
.ingest_batch(&vec_refs, &ids, None)
|
|
.expect("failed to ingest batch");
|
|
println!(
|
|
"\nIngested {} vectors (rejected: {}, epoch: {})",
|
|
ingest_result.accepted, ingest_result.rejected, ingest_result.epoch
|
|
);
|
|
|
|
// -- Step 3: Query top-5 nearest neighbors --
|
|
let query_seed = 42;
|
|
let query_vec = random_vector(dim, query_seed);
|
|
let k = 5;
|
|
|
|
let results = store
|
|
.query(&query_vec, k, &QueryOptions::default())
|
|
.expect("failed to query");
|
|
|
|
println!("\nQuery (seed={}): top-{} nearest neighbors:", query_seed, k);
|
|
print_results(&results);
|
|
|
|
// -- Step 4: Show store status --
|
|
let status = store.status();
|
|
println!("\nStore status:");
|
|
println!(" Total vectors: {}", status.total_vectors);
|
|
println!(" File size: {} bytes", status.file_size);
|
|
println!(" Epoch: {}", status.current_epoch);
|
|
println!(" Segments: {}", status.total_segments);
|
|
|
|
// -- Step 5: Close the store --
|
|
println!("\nClosing store...");
|
|
store.close().expect("failed to close store");
|
|
|
|
// -- Step 6: Reopen and query again to verify persistence --
|
|
println!("Reopening store...");
|
|
let reopened = RvfStore::open(&store_path).expect("failed to reopen store");
|
|
|
|
let results_after = reopened
|
|
.query(&query_vec, k, &QueryOptions::default())
|
|
.expect("failed to query after reopen");
|
|
|
|
println!("\nQuery after reopen: top-{} nearest neighbors:", k);
|
|
print_results(&results_after);
|
|
|
|
// Verify results match
|
|
assert_eq!(
|
|
results.len(),
|
|
results_after.len(),
|
|
"result count mismatch after reopen"
|
|
);
|
|
for (a, b) in results.iter().zip(results_after.iter()) {
|
|
assert_eq!(a.id, b.id, "ID mismatch after reopen");
|
|
assert!(
|
|
(a.distance - b.distance).abs() < 1e-6,
|
|
"distance mismatch after reopen"
|
|
);
|
|
}
|
|
println!("\nPersistence verified: results match before and after reopen.");
|
|
|
|
reopened.close().expect("failed to close reopened store");
|
|
|
|
// -- Step 7: Clean up happens automatically (TempDir) --
|
|
println!("\nDone. Temp directory will be cleaned up automatically.");
|
|
}
|
|
|
|
fn print_results(results: &[SearchResult]) {
|
|
println!(" {:>6} {:>12}", "ID", "Distance");
|
|
println!(" {:->6} {:->12}", "", "");
|
|
for r in results {
|
|
println!(" {:>6} {:>12.6}", r.id, r.distance);
|
|
}
|
|
}
|