mirror of
https://github.com/ruvnet/RuVector.git
synced 2026-05-30 03:53:34 +00:00
f8870b3c71
* feat(rvf): add RuVector Format universal substrate specification Research and design for RVF — a streaming, progressive, adaptive, quantum-secure binary format for vector intelligence. Covers append-only segment model, two-level tail manifests, temperature tiering, progressive HNSW indexing, epoch-based overlay system, SIMD-optimized query paths, WASM microkernel for Cognitum tiles, domain profiles (RVDNA, RVText, RVGraph, RVVision), and post-quantum cryptography. https://claude.ai/code/session_01DDqjGE51JpsRE3DgUjFyjW * feat(rvf): add deletion, filtered search, concurrency, and operations specs Fill four specification gaps in the RVF format design: - spec/07: Vector deletion lifecycle, JOURNAL_SEG wire format, deletion bitmaps - spec/08: Filtered search with META_SEG, METAIDX_SEG, filter expression language - spec/09: Writer locking, reader-writer coordination, versioning, space reclamation - spec/10: Batch operations API, error codes, network streaming protocol Also fixes the segment header field conflict between spec/01 and wire/binary-layout.md (checksum_algo/compression now u8, adds uncompressed_len at 0x38). https://claude.ai/code/session_01DDqjGE51JpsRE3DgUjFyjW * feat(rvf): add RuVector Format SDK, 40 examples, MCP server, and documentation Complete RVF implementation including: - 12 Rust crates (rvf-types, rvf-wire, rvf-manifest, rvf-index, rvf-quant, rvf-crypto, rvf-runtime, rvf-import, rvf-wasm, rvf-node, rvf-server, plus integration tests) - 40 runnable examples covering core storage, agentic AI, production patterns, vertical domains, exotic capabilities, runtime targets, network/security, POSIX/systems, and network operations - TypeScript SDK (npm/packages/rvf) with RvfDatabase class - MCP server (npm/packages/rvf-mcp-server) with stdio and SSE transports - Node.js N-API bindings (npm/packages/rvf-node) - WASM package (npm/packages/rvf-wasm) - ADR-029 (canonical format), ADR-030 (computational container), ADR-031 (example repository) - DNA-style lineage provenance, computational containers (KERNEL_SEG, EBPF_SEG), witness chains, TEE attestation, domain profiles - Superseded ADR annotations for ADR-001, ADR-005, ADR-006, ADR-018-021 Co-Authored-By: claude-flow <ruv@ruv.net> * feat(rvf): add CLI, WASM store, generate_all, and 46 output .rvf files - Add rvf-cli crate (665 lines, 9 subcommands: create/ingest/query/delete/status/inspect/compact/derive/serve) - Add WASM control plane store (alloc_setup, segment, store modules) for ~46 KB binary - Add generate_all.rs example producing 46 persistent .rvf files in output/ - Add Node.js N-API bindings for lineage, kernel/eBPF, and inspection - Add npm TypeScript backend/database/types for RVF integration - Update READMEs with CLI sections, MCP server docs, and crate map (13 crates) - All 40 examples verified passing Co-Authored-By: claude-flow <ruv@ruv.net> * feat(rvf): add Claude Code appliance, improve Quick Start, fix API docs - Add claude_code_appliance.rs: self-booting RVF with SSH + Claude Code install (curl -fsSL https://claude.ai/install.sh | bash), 3 SSH users, eBPF filter, 20-package manifest, witness chain, lineage snapshot - Improve Quick Start: Install section (crate/CLI/npm/WASM/MCP), WASM browser example, generate_all reference, expanded Rust crate deps - Fix embed_kernel/embed_ebpf API docs to match actual signatures (u8 params with `as u8` cast, 6-param kernel, Option<&[u8]> btf) - Update generate_all.rs: add claude_code_appliance generator (47 files) - Regenerate all 47 output .rvf files Co-Authored-By: claude-flow <ruv@ruv.net> * feat(rvf): add RVCOW branching, real kernel/eBPF/launcher, 795 tests Vector-native copy-on-write branching (ADR-031) with four new segment types (COW_MAP 0x20, REFCOUNT 0x21, MEMBERSHIP 0x22, DELTA 0x23), real Linux microkernel builder, QEMU microVM launcher, real eBPF programs, and 128-byte KernelBinding for tamper-evident kernel-manifest linkage. New crates: - rvf-kernel: Docker-based kernel build, real cpio/newc initramfs builder, SHA3-256 verification, prebuilt kernel support (37 tests) - rvf-launch: QEMU microVM launcher with QMP shutdown, KVM/TCG detection, virtio-blk/net port forwarding, kernel extraction (8 tests) - rvf-ebpf: 3 real BPF C programs (xdp_distance, socket_filter, tc_query_route) with clang compilation support (17 tests) RVCOW runtime: - CowEngine with read/write paths, write coalescing, snapshot-freeze - CowMap (flat-array), MembershipFilter (bitmap), CowCompactor - 3x read performance via pread optimization (1.3us/vector) - Branch creation: 2.6ms for 10K vectors, child = 162 bytes Security: 20-finding audit, 7 fixes applied including division-by-zero guards, integer overflow checks, and KernelBinding::from_bytes_validated(). CLI: 8 new commands (launch, embed-kernel, embed-ebpf, filter, freeze, verify-witness, verify-attestation, rebuild-refcounts), serve wired to real rvf-server. Co-Authored-By: claude-flow <ruv@ruv.net> * feat(rvf): update README, add crate/npm READMEs, publish to crates.io and npm - Rewrite README with cognitive container terminology, grouped features, 4 comparison tables (vs Docker, Vector DBs, Git LFS, SQLite), updated benchmarks, architecture diagram, and 45 examples - Add READMEs for rvf-kernel, rvf-launch, rvf-ebpf, rvf-import crates - Add READMEs for @ruvector/rvf, rvf-node, rvf-wasm, rvf-mcp-server npm packages - Fix Cargo.toml metadata (homepage, readme, categories, keywords) and add version specs to all path dependencies for crates.io publishing - Fix clippy warnings in rvf-kernel/initramfs.rs and rvf-launch/lib.rs - Published to crates.io: rvf-types, rvf-wire, rvf-manifest, rvf-quant, rvf-index, rvf-crypto (remaining crates pending rate limit) - Published to npm: @ruvector/rvf, @ruvector/rvf-node, @ruvector/rvf-wasm, @ruvector/rvf-mcp-server Co-Authored-By: claude-flow <ruv@ruv.net> * chore: add rvf-kernel, rvf-ebpf, rvf-launch, rvf-server, rvf-import, rvf-cli to workspace Include all 15 RVF crates plus integration tests and benchmarks in the root workspace members list so cargo publish can resolve them by name. Co-Authored-By: claude-flow <ruv@ruv.net> * feat(rvf): add published packages, cognitive container branding, grouped capabilities - Add Published Packages section with 13 crates.io + 4 npm tables - Add Platform Support table (Linux, macOS, Windows, WASM, no_std) - Expand capability table from 9 to 15 rows in 4 groups - Rewrite all "How" descriptions in plain language - Update .rvf diagram to show all 20 segment types - Rename ADRs: computational container -> cognitive container - Add emojis to all section headers Co-Authored-By: claude-flow <ruv@ruv.net> * feat: update root README with RVF cognitive containers, expanded capabilities - Update intro: "gets smarter + ships as cognitive container" - Add self-booting microservice row to Pinecone comparison table - Expand capabilities from 34 to 42 features with dedicated RVF section - Update "Think of it as" to include Docker comparison and RVF explanation - Add RVF collapsed group to Ecosystem (13 crates, 4 npm, install commands) - Add RVF to Platform & Edge section with install commands - Add RVF npm packages (4) and Rust crates (13) to package reference - Add RVF rows to feature comparison table (6 new rows) - Add ADR-030/031 to ADR list - Add RVF to Installation table, Project Structure - Update attention mechanisms count from 39 to 40+ - Update npm count to 49+, Rust crates to 83 - Update footer with crates.io and RVF links Co-Authored-By: claude-flow <ruv@ruv.net> * feat: expand comparison table with emojis, cost, audit, branching, single-file Co-Authored-By: claude-flow <ruv@ruv.net> * docs: rewrite comparison table in plain language Co-Authored-By: claude-flow <ruv@ruv.net> * chore: clean up empty code change sections in the changes log --------- Co-authored-by: Claude <noreply@anthropic.com>
87 lines
3.4 KiB
Markdown
87 lines
3.4 KiB
Markdown
# RVF: RuVector Format
|
|
|
|
## A Living, Self-Reorganizing Runtime Substrate for Vector Intelligence
|
|
|
|
---
|
|
|
|
### Document Index
|
|
|
|
#### Core Specification (`spec/`)
|
|
|
|
| # | Document | Description |
|
|
|---|----------|-------------|
|
|
| 00 | [Overview](spec/00-overview.md) | The Four Laws, design coordinates, philosophy |
|
|
| 01 | [Segment Model](spec/01-segment-model.md) | Append-only segments, headers, lifecycle, multi-file |
|
|
| 02 | [Manifest System](spec/02-manifest-system.md) | Two-level manifests, hotset pointers, progressive boot |
|
|
| 03 | [Temperature Tiering](spec/03-temperature-tiering.md) | Adaptive layout, access sketches, promotion/demotion |
|
|
| 04 | [Progressive Indexing](spec/04-progressive-indexing.md) | Layer A/B/C availability, lazy build, partial search |
|
|
| 05 | [Overlay Epochs](spec/05-overlay-epochs.md) | Streaming min-cut, epoch boundaries, rollback |
|
|
| 06 | [Query Optimization](spec/06-query-optimization.md) | SIMD alignment, prefetch, varint IDs, cache analysis |
|
|
| 07 | [Deletion & Lifecycle](spec/07-deletion-lifecycle.md) | Vector deletion, JOURNAL_SEG wire format, deletion bitmaps, compaction |
|
|
| 08 | [Filtered Search](spec/08-filtered-search.md) | META_SEG wire format, filter expressions, metadata indexes |
|
|
| 09 | [Concurrency & Versioning](spec/09-concurrency-versioning.md) | Writer locking, reader-writer coordination, space reclamation |
|
|
| 10 | [Operations API](spec/10-operations-api.md) | Batch ops, error codes, network streaming, compaction scheduling |
|
|
|
|
#### Wire Format (`wire/`)
|
|
|
|
| Document | Description |
|
|
|----------|-------------|
|
|
| [Binary Layout](wire/binary-layout.md) | Byte-level format reference, all segment payloads |
|
|
|
|
#### WASM Microkernel (`microkernel/`)
|
|
|
|
| Document | Description |
|
|
|----------|-------------|
|
|
| [WASM Runtime](microkernel/wasm-runtime.md) | Cognitum tile mapping, 14 exports, hub-tile protocol |
|
|
|
|
#### Domain Profiles (`profiles/`)
|
|
|
|
| Document | Description |
|
|
|----------|-------------|
|
|
| [Domain Profiles](profiles/domain-profiles.md) | RVDNA, RVText, RVGraph, RVVision specifications |
|
|
|
|
#### Cryptography (`crypto/`)
|
|
|
|
| Document | Description |
|
|
|----------|-------------|
|
|
| [Quantum Signatures](crypto/quantum-signatures.md) | ML-DSA-65, SHAKE-256, hybrid encryption, witnesses |
|
|
|
|
#### Benchmarks (`benchmarks/`)
|
|
|
|
| Document | Description |
|
|
|----------|-------------|
|
|
| [Acceptance Tests](benchmarks/acceptance-tests.md) | Performance targets, crash safety, scalability |
|
|
|
|
---
|
|
|
|
### Quick Reference
|
|
|
|
**The Four Laws**
|
|
1. Truth lives at the tail
|
|
2. Every segment is independently valid
|
|
3. Data and state are separated
|
|
4. The format adapts to its workload
|
|
|
|
**Minimal Upgrade Path** (smallest changes that unlock everything)
|
|
1. Add tail manifest segments
|
|
2. Make every payload a segment with its own hash and length
|
|
3. Add hotset pointers in the manifest
|
|
4. Add an epoch overlay model
|
|
|
|
**Hardware Profiles**
|
|
- **Core**: 8 KB code + 8 KB data + 64 KB SIMD (Cognitum tile)
|
|
- **Hot**: Multi-tile chip with shared memory
|
|
- **Full**: Desktop/server with mmap and full feature set
|
|
|
|
**Key Numbers**
|
|
- Boot: 4 KB read, < 5 ms
|
|
- First query: <= 4 MB read, recall >= 0.70
|
|
- Full quality: recall >= 0.95
|
|
- Signing: ML-DSA-65, 3,309 B signatures, ~4,500 sign/s
|
|
- Distance: 384-dim fp16 L2 in ~12 AVX-512 cycles
|
|
- Hot entry: 960 bytes (vector + 16 neighbors, cache-line aligned)
|
|
|
|
**Design Choices**
|
|
- Append-only + compaction (not random writes)
|
|
- Both mmap desktop and microcontroller tiles
|
|
- Priority: streamable > progressive > adaptive > p95 speed
|