ruvector/docs/research/rvf/benchmarks/acceptance-tests.md

RVF Acceptance Tests and Performance Targets

1. Primary Acceptance Test

Cold start on a 10 million vector file: load and answer the first query with a useful result (recall@10 >= 0.70) without reading more than the last 4 MB, then converge to full quality (recall@10 >= 0.95) as it progressively maps more segments.

Test Parameters

Dataset:         10 million vectors
Dimensions:      384 (sentence embedding size)
Base dtype:      fp16 (768 bytes per vector)
Raw file size:   ~7.2 GB (vectors only)
With index:      ~10-12 GB total
Query set:       1000 queries from held-out test set
Ground truth:    Brute-force exact k-NN (k=10)
Metric:          L2 distance

Success Criteria

Phase	Time Budget	Data Read	Min Recall@10	Description
Boot	< 5 ms	4 KB (Level 0)	N/A	Parse root manifest
First query	< 50 ms	<= 4 MB	>= 0.70	Layer A + hot cache
Working quality	< 500 ms	<= 200 MB	>= 0.85	Layer A + B
Full quality	< 5 s	<= 4 GB	>= 0.95	Layers A + B + C
Optimized	< 30 s	Full file	>= 0.98	All layers + hot tier

Measurement Methodology

1. Create RVF file from 10M vector dataset
   - Build full HNSW index (M=16, ef_construction=200)
   - Compute temperature tiers (default: all warm initially)
   - Write with all segment types

2. Cold start measurement
   - Drop filesystem cache: echo 3 > /proc/sys/vm/drop_caches
   - Open file, start timer
   - Read Level 0 (4 KB), record time T_boot
   - Read hotset data, record time T_hotset
   - Execute first query, record time T_first_query and recall@10
   - Continue progressive loading
   - At each milestone: record time, data read, recall@10

3. Throughput measurement (warm)
   - After full load, execute 1000 queries
   - Measure queries per second (QPS)
   - Measure p50, p95, p99 latency
   - Measure recall@10 average

4. Streaming ingest measurement
   - Start with empty file
   - Ingest 10M vectors in streaming mode
   - Measure ingest rate (vectors/second)
   - Measure file size over time
   - Verify crash safety (kill -9 at random points, verify recovery)

2. Performance Targets

Query Latency (10M vectors, 384 dim, fp16)

Hardware	QPS (single thread)	p50 Latency	p95 Latency	p99 Latency
Desktop (AVX-512)	5,000-15,000	0.1 ms	0.3 ms	1.0 ms
Desktop (AVX2)	3,000-8,000	0.2 ms	0.5 ms	2.0 ms
Laptop (NEON)	2,000-5,000	0.3 ms	1.0 ms	3.0 ms
WASM (browser)	500-2,000	1.0 ms	3.0 ms	10.0 ms
Cognitum tile	100-500	2.0 ms	5.0 ms	15.0 ms

Streaming Ingest Rate

Hardware	Vectors/Second	Bytes/Second	Notes
NVMe SSD	200K-500K	150-380 MB/s	fsync every 1000 vectors
SATA SSD	50K-100K	38-76 MB/s	fsync every 1000 vectors
HDD	10K-30K	7-23 MB/s	Sequential append
Network (1 Gbps)	50K-100K	38-76 MB/s	Streaming over network

Progressive Load Times

Phase	NVMe SSD	SATA SSD	HDD	Network
Boot (4 KB)	< 0.1 ms	< 0.5 ms	< 10 ms	< 50 ms
First query (4 MB)	< 2 ms	< 10 ms	< 100 ms	< 500 ms
Working quality (200 MB)	< 100 ms	< 500 ms	< 5 s	< 20 s
Full quality (4 GB)	< 2 s	< 10 s	< 120 s	< 400 s

Space Efficiency

Configuration	Bytes/Vector	File Size (10M)	Ratio vs Raw
Raw fp32	1,536	14.3 GB	1.0x
RVF uniform fp16	768 + overhead	8.0 GB	0.56x
RVF adaptive (equilibrium)	~300 avg	3.2 GB	0.22x
RVF aggressive (binary cold)	~100 avg	1.1 GB	0.08x

3. Crash Safety Tests

Test 1: Kill During Vector Ingest

1. Start ingesting 1M vectors
2. After 500K vectors: kill -9 the writer
3. Verify: file is readable
4. Verify: latest valid manifest is found
5. Verify: all vectors referenced by latest manifest are intact
6. Verify: no data corruption (all segment hashes valid)

Pass criteria: Zero data loss for committed segments. At most the last incomplete segment is lost (bounded by fsync interval).

Test 2: Kill During Manifest Write

1. Create file with 1M vectors
2. Trigger manifest rewrite (add metadata, trigger compaction)
3. Kill -9 during manifest write
4. Verify: file falls back to previous valid manifest
5. Verify: all queries work correctly with previous manifest

Pass criteria: Automatic fallback to previous manifest. No manual recovery needed.

Test 3: Kill During Compaction

1. Create file with 1M vectors across 100 small VEC_SEGs
2. Trigger compaction
3. Kill -9 during compaction
4. Verify: file is readable (old segments still valid)
5. Verify: partial compaction output is safely ignored

Pass criteria: Old segments remain valid. Incomplete compaction output has no manifest reference and is safely orphaned.

Test 4: Bit Flip Detection

1. Create valid RVF file
2. Flip random bits in various locations
3. Verify: corruption detected by hash/CRC checks
4. Verify: specific corrupted segment identified
5. Verify: other segments still readable

Pass criteria: 100% detection of single-bit flips. Corruption isolated to affected segment.

4. Scalability Tests

Test: 1 Billion Vectors

Dataset:     1B vectors, 384 dimensions, fp16
File size:   ~700 GB (raw) -> ~200 GB (adaptive RVF)
Hardware:    Server with 256 GB RAM, NVMe array

Verify:
  - Boot time < 10 ms
  - First query < 100 ms
  - Full quality convergence < 60 s
  - Recall@10 >= 0.95 at full quality
  - Streaming ingest sustained at 100K+ vectors/second

Test: High Dimensionality

Dataset:     1M vectors, 4096 dimensions (LLM embeddings)
File size:   ~8 GB (fp16)

Verify:
  - PQ compression to 5-bit achieves >= 10x compression
  - Recall@10 >= 0.90 with PQ
  - Query latency < 5 ms (p95) with PQ + HNSW

Test: Multi-File Sharding

Dataset:     100M vectors across 10 shard files
Verify:
  - Transparent query across all shards
  - Shard addition without full rebuild
  - Individual shard compaction
  - Shard removal with manifest update only

5. WASM Performance Tests

Browser Environment

Runtime:     Chrome V8 / Firefox SpiderMonkey
SIMD:        WASM v128
Memory:      Limited to 4 GB WASM heap

Test: Load 1M vector RVF file via fetch()
  - Boot time < 50 ms
  - First query < 200 ms (after boot)
  - QPS >= 500 (single thread)
  - Memory usage < 500 MB

Cognitum Tile Simulation

Runtime:     wasmtime with memory limits
Code limit:  8 KB
Data limit:  8 KB
Scratch:     64 KB

Test: Process 1000 blocks via hub protocol
  - Distance computation matches reference implementation
  - Top-K results match brute-force within quantization tolerance
  - No memory access out of bounds
  - Tile recovers from simulated faults

6. Interoperability Tests

Round-Trip Test

1. Create RVF file from numpy arrays
2. Read back with independent implementation
3. Verify: all vectors bit-identical
4. Verify: all metadata preserved
5. Verify: index produces same results

Profile Compatibility Test

1. Create RVDNA file with genomic data
2. Create RVText file with text embeddings
3. Read both with generic RVF reader
4. Verify: generic reader can access vectors and metadata
5. Verify: profile-specific features degrade gracefully

Version Forward Compatibility Test

1. Create RVF file with version 1
2. Add segments with hypothetical version 2 features (unknown tags)
3. Read with version 1 reader
4. Verify: version 1 reader skips unknown segments/tags
5. Verify: version 1 data is fully accessible

7. Security Tests

Signature Verification

1. Create signed RVF file (ML-DSA-65)
2. Verify all segment signatures
3. Modify one byte in a signed segment
4. Verify: modification detected
5. Verify: other segments still valid

Encryption Round-Trip

1. Create encrypted RVF file (ML-KEM-768 + AES-256-GCM)
2. Decrypt with correct key
3. Verify: plaintext matches original
4. Attempt decrypt with wrong key
5. Verify: decryption fails (GCM auth tag mismatch)

Key Rotation

1. Create file signed with key A
2. Rotate to key B (write CRYPTO_SEG rotation record)
3. Write new segments signed with key B
4. Verify: old segments valid with key A
5. Verify: new segments valid with key B
6. Verify: cross-signature in rotation record is valid

8. Benchmark Harness

Recommended Tools

Purpose	Tool	Notes
Latency measurement	criterion (Rust) / benchmark.js	Statistical rigor
Recall measurement	Custom recall@K computation	Against brute-force ground truth
Memory profiling	valgrind massif / Chrome DevTools	Peak and sustained
I/O profiling	blktrace / iostat	Verify read patterns
SIMD verification	Intel SDE / ARM emulator	Correct SIMD codegen
Crash testing	Custom harness with kill -9	Random timing

Report Format

Each benchmark run produces a report:

{
  "test_name": "cold_start_10m",
  "dataset": {
    "vector_count": 10000000,
    "dimensions": 384,
    "dtype": "fp16",
    "file_size_bytes": 10737418240
  },
  "hardware": {
    "cpu": "Intel Xeon w5-3435X",
    "simd": "AVX-512",
    "ram_gb": 256,
    "storage": "NVMe Samsung 990 Pro"
  },
  "results": {
    "boot_ms": 0.08,
    "first_query_ms": 12.3,
    "first_query_recall_at_10": 0.73,
    "working_quality_ms": 340,
    "working_quality_recall_at_10": 0.87,
    "full_quality_ms": 3200,
    "full_quality_recall_at_10": 0.96,
    "steady_state_qps": 8500,
    "steady_state_p50_ms": 0.12,
    "steady_state_p95_ms": 0.28,
    "steady_state_p99_ms": 0.85,
    "data_read_first_query_mb": 3.2,
    "data_read_working_quality_mb": 180
  }
}