ruvector/crates/rvf/rvf-cli
rUv 3e8429739f
perf(rvf,rvm): HNSW query path, RaBitQ, contiguous slab, witness v2, mincut wiring + security hardening (#555)
* chore(rvf): sync Cargo.lock with rvf-wire deps (sha3, subtle)

https://claude.ai/code/session_01C83hbozEXPgoz9iJN5Smhp

* fix(rvf-runtime): deterministic tie-breaking in query result ordering

Equal-distance vectors were selected and ordered by HashMap iteration
order, which changes across process restarts and made query results
non-reproducible (flaky smoke_rvlite_adapter_persistence). Break ties
by vector id in both the top-k heap eviction and the final sort, in
query() and query_with_envelope().

https://claude.ai/code/session_01C83hbozEXPgoz9iJN5Smhp

* perf(rvf): optimize index/runtime hot paths, fix quant codec and manifest discovery

rvf-index:
- Cache SIMD distance-kernel dispatch in a OnceLock function-pointer
  table instead of re-running is_x86_feature_detected! on every call
- Rewrite HNSW search_layer with BinaryHeap min/max-heaps (was sorted
  Vec + O(n) mid-inserts) and a dense Vec<bool> visited bitmap (was a
  per-call SipHash HashSet); deterministic (distance, id) tie-breaking

rvf-runtime:
- Replace per-bit CRC32 loops with crc32fast (same IEEE polynomial,
  byte-identical hashes, ~100x faster) on segment write and verify
- Hoist cosine query-norm computation out of the per-vector scan loop
- Safety-net scan: single pass with HashSet membership (was
  O(k*N*neighbors) with Vec::contains)
- Bulk little-endian f32 serialization in write_vec_seg (one memcpy
  per vector instead of per-element appends)
- Progressively widen the manifest tail scan (64KB -> 1MB -> 16MB ->
  whole file): stores with large segment directories were becoming
  unreadable once the latest manifest fell outside the fixed 64KB
  window; with regression test

rvf-quant:
- encode_quant_seg now emits fully decodable payloads (delegates to
  the real scalar/product encoders; placeholders removed)
- decode_quant_seg returns Result instead of panicking on malformed
  or unknown-type payloads; round-trip and malformed-input tests

https://claude.ai/code/session_01C83hbozEXPgoz9iJN5Smhp

* fix(rvm): bind witness chain to record content; optimize coherence, cap, sched hot paths

rvm-witness (security-critical):
- The chain hash covered only (prev_hash, sequence) — record content
  (action, actor, target, payload, timestamp) was never hashed, so
  verify_chain accepted arbitrarily rewritten history. record_hash is
  now computed over the 44 content bytes (as its doc always claimed)
  and the chain binds it: H(prev || seq || record_hash). verify_chain
  recomputes content hashes; tamper-regression tests added.
- HMAC signer keys the Mac template once at construction instead of
  re-running the key schedule per record (fixed-vector test pins
  signature bytes)
- Witness ring overflow is now observable: total_overwritten counter
  and needs_drain() accessor

rvm-cap:
- Nonce replay window: colliding nonces (A + k*4096) could evict and
  re-admit nonce A. Replaced the two 32KB arrays with one 32KB
  open-addressed table (8-probe bounded); eviction raises the
  watermark so it fails closed. Regression test included.

rvm-coherence:
- internal_weight: O(MAX_EDGES) self-loop scan replaced with O(1)
  adj_matrix[i][i] read (invariant verified across all mutation paths)
- Skip ticks return a cached CoherenceDecision instead of re-running
  the O(n^2) merge-pair pass over stale data; zero-weight pairs skipped
- Mincut: scratch buffers moved into the long-lived bridge (~17KB less
  stack per call), in-place Stoer-Wagner (no working copy), bitmask
  membership, column-scan in-neighbors
- Compile-time guard: CoherenceGraph MAX_NODES > ADJ_DIM now fails to
  compile instead of panicking at the 33rd node; u64 weight deltas
  clamped at the engine boundary

rvm-coherence/rvm-partition:
- Single-slot hash indexes (id_to_node, edge_index) degraded to
  permanent O(N) scans after any collision; both now use bounded
  linear probing with tombstones and probe-proven absence

rvm-sched:
- enqueue() rejects the HYPERVISOR sentinel id, which previously
  wedged a run-queue slot permanently; defensive cleanup in
  switch_next

Tests: 733 workspace + 67 rvm-kernel lib pass (baseline 712); 23 new
tests including tamper-evidence and collision regressions.

https://claude.ai/code/session_01C83hbozEXPgoz9iJN5Smhp

* feat(rvf): wire HNSW index into the runtime query path (~14x speedup)

RvfStore::query was a brute-force O(N*dim) scan; the rvf-index crate was
unused by production queries and QualityEnvelope.evidence fabricated
layer_a=true. The index is now built lazily on first eligible query,
maintained incrementally on ingest, persisted on close() via the existing
INDEX_SEG codec (with a versioned, backward-readable trailer for the
sparse-id mapping), and validated-or-rebuilt on open. Exact scan remains
for small stores (<1024), filtered/COW/membership queries, >25% deleted,
and force_exact; deterministic (distance, id) tie-breaking preserved on
both paths. evidence.layer_a is now set only when the index served the
query.

Measured: 21.7ms -> 1.51ms per query at 100k x 64-dim (criterion,
release), recall@10 = 0.968 at the ef_search=256 floor (>=0.95 gated by
test). +15 tests (recall, index persistence round-trip, evidence honesty,
fallback routing, compaction/overwrite invalidation).

Co-Authored-By: claude-flow <ruv@ruv.net>

* feat(rvm): witness v2 — keyed-BLAKE3 chain with 128-bit links + Merkle sealing

v1 records folded chain links to 32 bits and left the head unanchorable.
The 96-byte v2 record embeds the predecessor MAC full-width and chains via
one keyed-BLAKE3 compression per append (~112ns measured, 9x under the 1us
target); keyed MACs detect last-record tampering and unkeyed forgery,
which v1 could not. Segment sealing accumulates record MACs into a
domain-separated Merkle tree (256/segment) sealed with one signature via
the existing signer infra (HMAC/dual-HMAC/Ed25519/TEE), with inclusion
proofs — expensive crypto moves off the per-record path and roots are
externally anchorable.

v1 logs still verify (version-byte dispatch; v1 only as prefix, head
anchored into the first v2 record); v1 writing is frozen. blake3 added as
pure-Rust no_std. +46 tests covering content/reorder/truncation/wrong-key
/forgery tamper modes, v1 compat, proofs, seals, and mixed logs.

Co-Authored-By: claude-flow <ruv@ruv.net>

* feat(rvm): wire mincut into split decisions; honest partition-switch claim

execute_split previously created an empty child and ignored the computed
cut. It now resolves the boundary from a cached epoch SplitPlan (or
computes on demand) and re-homes move-side neighbors to the child with
their edge weights. Two-tier decisions: exact Stoer-Wagner mincut runs as
a pressure-triggered epoch task; a new Fennel placer (O(degree),
fixed-point gamma=1.5, no_std) handles hot-path placement. Split policy
combines pressure and cut quality: mid-band (8000-9500bp) splits only on
a cut with conductance <= 5000bp; critical pressure stays an
unconditional safety valve.

The sub-10us partition-switch claim was a stub certified by a no-op bench
(~6ns) reported as 1600x faster than target. The real path needs EL2
assembly the crate forbids; instead the measurable register save/restore
lower bound is implemented and benchmarked, the bench is renamed
partition_switch_validation_stub with an honesty gate, a canary test
fails if HARDWARE_SWITCH_IMPLEMENTED flips without revisiting the claim,
and the README row now reads: not validated. +30 tests.

Co-Authored-By: claude-flow <ruv@ruv.net>

* feat(rvf): RaBitQ binary quantization + Vamana alpha-pruning (opt-in)

rvf-quant gains a RaBitQ-style codec: global-centroid centering, 3-round
seeded randomized-Hadamard rotation (orthonormal, reproducible from a
stored u64 seed), 1-bit sign codes with per-vector norm/dot-correction
scalars, and an asymmetric full-precision-query estimator. QUANT_SEG adds
versioned type tag 4 (legacy payloads byte-frozen and still decode;
unknown versions rejected; decode stays panic-free on untrusted bytes).

Query path: opt-in two-stage search (QueryOptions::rabitq, default off) —
estimator scan with oversampling (640-candidate floor) then exact f32
rescore; deterministic (distance, id) tie-breaking; falls back to default
routing for filtered/COW/IP/cosine queries. Measured recall@10 = 0.972 vs
exact on 10k x 128 (gate >= 0.95, test-enforced); code-only compression
exactly 32x.

rvf-index: Vamana-style robust prune (alpha = 1.2, occluded backfill) at
insert and prune time; recall@10 at ef=30 improved 0.986 -> 0.996;
construction determinism preserved. +42 tests (1254 passing, no new
failures).

Co-Authored-By: claude-flow <ruv@ruv.net>

* fix(rvf): harden untrusted decode paths against crafted-file DoS

An adversarial audit confirmed a crafted .rvf could panic or OOM the
process on RvfStore::open(): unvalidated length fields drove
Vec::with_capacity before any byte-availability check. decode_payload now
bounds id_count by available delta bytes (u64 compare before the usize
cast, so 32-bit truncation cannot bypass it); decode_index_seg bounds
restart_count/layer_count/neighbor_count by remaining bytes and rejects
truncated restart padding (was a reachable slice panic); decode_sketch_seg
converts from assert-and-panic to Result with width/depth validated via
checked_mul (closes the width=0 + depth=u32::MAX bypass); decode_product
size products use checked u64 arithmetic so 32-bit (wasm32) targets cannot
wrap usize and read out of bounds. +8 adversarial regression tests.

Co-Authored-By: claude-flow <ruv@ruv.net>

* perf(rvf): contiguous vector slab, non-blocking index rebuild, unified hashing

Vector storage moves from HashMap<u64, Vec<f32>> to a contiguous row-major
slab (id->ordinal map, tombstoned deletes, slot reuse only via compaction);
HNSW/RaBitQ paths read rows as zero-copy slices and iteration is ordinal-
ordered (deterministic across restarts). Brute-force query at 100k x 64:
24.5ms -> 3.8ms (~6.4x). boot() pre-sizes the slab and bulk-copies VEC_SEG
payloads (no per-vector allocs): cold open 257ms -> 202ms (-21.5%). mmap
deferred (CRC verify touches all bytes anyway; memmap2 not in this
workspace) and documented as follow-up.

Audit finding 5: index/RaBitQ lazy builds now run with no lock held behind
an AtomicBool gate (panic-safe clear-on-drop); concurrent queries fall
back to exact scan and keep serving through the entire O(N log N) build.
Overwrite still invalidates and unlinks the stale INDEX_SEG.

Hashing: the two identical bespoke CRC32-rotation implementations in
write_path/read_path now delegate to one source of truth
(hashing::legacy_content_hash); on-disk bytes unchanged. Full rvf-wire
checksum-registry conformance (XXH3-128 + format-version bump +
dual-accept reader) documented as the remaining delta. read_path.rs also
carries the audit''s checked vec-seg size arithmetic. +11 tests; suite
1271 passing, no new failures (one pre-existing wall-clock bench
assertion flakes under load, passes in isolation).

Co-Authored-By: claude-flow <ruv@ruv.net>

* chore(release): prepare rvf 0.2.1/0.2.0/0.3.0 crate bumps, npm 0.2.2/0.1.7, measured-benchmark READMEs

- rvf-types 0.2.0 -> 0.2.1 (QuantType::RaBitQ format extension)
- rvf-index 0.1.0 -> 0.2.0 (Vamana alpha-pruning, hardened INDEX_SEG codec)
- rvf-quant 0.1.0 -> 0.2.0 (RaBitQ codec; decode_sketch_seg now returns Result)
- rvf-runtime 0.2.0 -> 0.3.0 (HNSW query path, INDEX_SEG trailer, QueryOptions::rabitq, vector slab)
- dependent path-dep version reqs updated (cli, import, launch, node, server)
- @ruvector/rvf 0.2.0 -> 0.2.2, @ruvector/rvf-wasm 0.1.6 -> 0.1.7 (rebuilt wasm artifact, 1.89 toolchain + wasm-opt -Oz)
- READMEs: HNSW/RaBitQ/slab docs with measured numbers (Windows x64, criterion release, 100k x 64-dim); rvm witness v2 bench rows

Co-Authored-By: claude-flow <ruv@ruv.net>

* fix(robotics): bump rvf-runtime requirement to 0.3 after release bump

The rvf-runtime 0.2 -> 0.3.0 version bump updated dependents inside the
rvf workspace but missed the root-workspace consumer: ruvector-robotics
pins version 0.2 alongside its path dep, which fails cargo resolution
against the bumped crate (PR #555 CI: failed to select a version for the
requirement rvf-runtime ^0.2). Root Cargo.lock refreshed.

Co-Authored-By: claude-flow <ruv@ruv.net>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: ruv <ruvnet@users.noreply.github.com>
2026-06-12 13:09:30 -04:00
..
src fix: apply cargo fmt across workspace and fix CI issues 2026-02-21 20:56:38 +00:00
Cargo.toml perf(rvf,rvm): HNSW query path, RaBitQ, contiguous slab, witness v2, mincut wiring + security hardening (#555) 2026-06-12 13:09:30 -04:00
README.md feat(rvf-cli): add cross-platform release workflow and update README 2026-02-16 23:19:39 +00:00

rvf — RuVector Format CLI

Standalone command-line tool for creating, inspecting, querying, and managing RVF vector stores. Runs on Windows, macOS, and Linux with zero runtime dependencies.

Install

Download from GitHub Releases:

# macOS (Apple Silicon)
curl -L -o rvf https://github.com/ruvnet/ruvector/releases/latest/download/rvf-darwin-arm64
chmod +x rvf && sudo mv rvf /usr/local/bin/

# macOS (Intel)
curl -L -o rvf https://github.com/ruvnet/ruvector/releases/latest/download/rvf-darwin-x64
chmod +x rvf && sudo mv rvf /usr/local/bin/

# Linux x64
curl -L -o rvf https://github.com/ruvnet/ruvector/releases/latest/download/rvf-linux-x64
chmod +x rvf && sudo mv rvf /usr/local/bin/

# Linux ARM64
curl -L -o rvf https://github.com/ruvnet/ruvector/releases/latest/download/rvf-linux-arm64
chmod +x rvf && sudo mv rvf /usr/local/bin/

Windows (PowerShell):

Invoke-WebRequest -Uri https://github.com/ruvnet/ruvector/releases/latest/download/rvf-windows-x64.exe -OutFile rvf.exe

Build from source

Requires Rust:

cargo install --git https://github.com/ruvnet/ruvector.git rvf-cli

Or clone and build:

git clone https://github.com/ruvnet/ruvector.git
cd ruvector
cargo build -p rvf-cli --release
# Binary: target/release/rvf (or rvf.exe on Windows)

Quick start

# Create a 128-dimensional vector store with cosine distance
rvf create mydb.rvf --dimension 128 --metric cosine

# Ingest vectors from JSON
rvf ingest mydb.rvf --input vectors.json

# Search for nearest neighbors
rvf query mydb.rvf --vector "0.1,0.2,0.3,..." --k 10

# Check store status
rvf status mydb.rvf

Running the examples

The repo includes 48 pre-built .rvf example stores in examples/rvf/output/. Use the CLI to inspect, query, and manipulate them:

# List all example stores
ls examples/rvf/output/*.rvf

# Inspect a store
rvf status examples/rvf/output/basic_store.rvf
rvf inspect examples/rvf/output/basic_store.rvf

# Query the semantic search example (500 vectors, 384 dimensions)
rvf status examples/rvf/output/semantic_search.rvf
rvf inspect examples/rvf/output/semantic_search.rvf --json

# Inspect the RAG pipeline store
rvf status examples/rvf/output/rag_pipeline.rvf

# Look at COW lineage (parent -> child)
rvf inspect examples/rvf/output/lineage_parent.rvf
rvf inspect examples/rvf/output/lineage_child.rvf

# Check financial signals store
rvf status examples/rvf/output/financial_signals.rvf

# View the compacted store
rvf status examples/rvf/output/compacted.rvf

# Inspect agent memory store
rvf inspect examples/rvf/output/agent_memory.rvf

# View all stores at once (JSON)
for f in examples/rvf/output/*.rvf; do
  echo "--- $(basename $f) ---"
  rvf status "$f" 2>/dev/null
done

Available example stores

Store Vectors Dim Description
basic_store.rvf 100 384 Basic vector store
semantic_search.rvf 500 384 Semantic search embeddings
rag_pipeline.rvf 300 256 RAG pipeline embeddings
embedding_cache.rvf 500 384 Embedding cache
filtered_search.rvf 200 256 Filtered search with metadata
financial_signals.rvf 100 512 Financial signal vectors
recommendation.rvf 100 256 Recommendation engine
medical_imaging.rvf 100 768 Medical imaging features
multimodal_fusion.rvf 100 2048 Multimodal fusion vectors
legal_discovery.rvf 100 768 Legal document embeddings
progressive_index.rvf 1000 384 Progressive HNSW index
quantization.rvf 1000 384 Quantized vectors
swarm_knowledge.rvf 100 128 Swarm intelligence KB
agent_memory.rvf 50 128 Agent conversation memory
experience_replay.rvf 50 64 RL experience replay buffer
lineage_parent.rvf COW parent (lineage demo)
lineage_child.rvf COW child (lineage demo)
compacted.rvf Post-compaction store

Commands

create

Create a new empty RVF store.

rvf create store.rvf --dimension 128 --metric cosine
rvf create store.rvf -d 384 -m l2 --profile 1 --json

Options:

  • -d, --dimension — Vector dimensionality (required)
  • -m, --metric — Distance metric: l2, ip (inner product), cosine (default: l2)
  • -p, --profile — Hardware profile 0-3 (default: 0)
  • --json — Output as JSON

ingest

Import vectors from a JSON file.

rvf ingest store.rvf --input data.json
rvf ingest store.rvf -i data.json --batch-size 500 --json

Input JSON format:

[
  {"id": 1, "vector": [0.1, 0.2, 0.3, ...]},
  {"id": 2, "vector": [0.4, 0.5, 0.6, ...]}
]

query

Search for k nearest neighbors.

rvf query store.rvf --vector "1.0,0.0,0.5,0.3" --k 10
rvf query store.rvf -v "0.5,0.5,0.0,0.0" -k 5 --json

With filters:

rvf query store.rvf -v "1.0,0.0" -k 10 \
  --filter '{"eq":{"field":0,"value":{"string":"category_a"}}}'

delete

Delete vectors by ID or filter.

rvf delete store.rvf --ids 1,2,3
rvf delete store.rvf --filter '{"gt":{"field":0,"value":{"u64":100}}}'

status

Show store status.

rvf status store.rvf
rvf status store.rvf --json

inspect

Inspect store segments and lineage.

rvf inspect store.rvf
rvf inspect store.rvf --json

compact

Reclaim dead space from deleted vectors.

rvf compact store.rvf
rvf compact store.rvf --strip-unknown --json

derive

Create a derived child store (COW branching).

rvf derive parent.rvf child.rvf --derivation-type clone
rvf derive parent.rvf child.rvf -t snapshot --json

Derivation types: clone, filter, merge, quantize, reindex, transform, snapshot

freeze

Snapshot-freeze the current state.

rvf freeze store.rvf

verify-witness

Verify the tamper-evident witness chain.

rvf verify-witness store.rvf

verify-attestation

Verify kernel binding and attestation.

rvf verify-attestation store.rvf

serve

Start an HTTP server (requires serve feature).

cargo build -p rvf-cli --features serve
rvf serve store.rvf --port 8080

launch

Boot an RVF file in a QEMU microVM (requires launch feature).

cargo build -p rvf-cli --features launch
rvf launch store.rvf --port 8080 --memory-mb 256

JSON output

All commands support --json for machine-readable output:

rvf status store.rvf --json | jq '.total_vectors'
rvf query store.rvf -v "1,0,0,0" -k 5 --json | jq '.results[].id'

Platform scripts

Platform-specific quickstart scripts are in examples/rvf/scripts/:

# Linux / macOS
bash examples/rvf/scripts/rvf-quickstart.sh

# Windows PowerShell
.\examples\rvf\scripts\rvf-quickstart.ps1

License

MIT OR Apache-2.0