mirror of https://github.com/ruvnet/RuVector.git synced 2026-08-22 23:23:46 +00:00

History

rUv feb4ee2753 perf(hnsw): 4-acc AVX-512 + parallel-insert — +9.7% build throughput (query QPS unchanged: memory-bound at 1M scale) (#619 ) * perf(hnsw): 4-accumulator AVX-512 kernels + SIMD wiring into search hot path - Replace single-accumulator AVX-512 distance kernels with 4-accumulator versions in simd_intrinsics.rs (euclidean, cosine, dot, manhattan). On Zen 5 with 4-cycle FMA latency, single-accumulator was latency-bound (96 cycles for 384-dim); 4-accumulator hides this to ~24 cycles. - Wire HNSW search hot path in DistanceFn::eval to call simd_intrinsics directly (inline, no Result wrapping, no simsimd FFI overhead). - Enable parallel batch insert via hnsw_rs::parallel_insert_slice (rayon). Measured: 6-10% QPS improvement on 128-dim/1K-vector bench; larger gains expected on 1M-vector workloads where distance computation dominates. 228 unit tests pass. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019rVRYrRDKyxYK18kuVrDSf * perf(hnsw): gate parallel_insert_slice behind 10K-vector threshold Rayon-based parallel insert (hnsw_rs::parallel_insert_slice) degrades graph connectivity for small batches (<10K vectors) because worker threads can't see each other's in-flight insertions, reducing optimal neighbor links. Add PARALLEL_THRESHOLD=10_000: use parallel insert only when the batch is large enough that the graph quality converges. Below threshold: sequential insert_data (same as before this PR). Above threshold: parallel_insert_slice for build-time speedup. 228 unit tests pass. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019rVRYrRDKyxYK18kuVrDSf * bench(sift1m): add SIFT-1M fvecs benchmark + hnswlib comparison tooling Adds two benchmark binaries driven by the real TEXMEX SIFT-1M dataset: * crates/ruvector-sota-bench/src/bin/sift1m_bench.rs Reads sift_base.fvecs / sift_query.fvecs / sift_groundtruth.ivecs directly (no HDF5 required). Sweeps ef_search to produce a recall@10 vs QPS table used for before/after PR #619 comparison. * scripts/sift1m_hnswlib_bench.mjs Same sweep via hnswlib-node (C++ HNSW) to measure the competitive gap. Cargo.toml: add simd-avx512 feature to sota-bench dependency so the full optimised kernel path is exercised. Measured on AMD Ryzen 9 9950X (Zen 5, AVX-512), M=16, efC=200, 1M vecs: Source Build ef=100 recall ef=100 QPS ef=200 recall ef=200 QPS before PR 849 s 0.9585 1,849 0.9713 1,058 after PR (#619) 774 s 0.9592 1,768 0.9722 1,024 hnswlib-node 322 s 0.9828 5,339 0.9957 2,897 Build speedup: +9.7 %. Query QPS at 1M-scale: within noise (memory- bandwidth bound, not compute-bound). Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019rVRYrRDKyxYK18kuVrDSf * style: cargo fmt for sift1m benchmark binary Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019rVRYrRDKyxYK18kuVrDSf --------- Co-authored-by: ruvnet <ruvnet@gmail.com>		2026-06-28 20:45:35 -04:00
..
benchmark
build
ci
deploy
lib
patches/hnsw_rs
publish
sona-drift	ci(sona): cross-implementation drift gate — parity harness, RVF fingerprints, stub tripwire (#554 )	2026-06-11 22:40:27 -04:00
test
training
validate
analyze-evolution.js
analyze-ham10000.js
build-solver.sh
build-sonic-ct-wasm.sh	feat(sonic_ct): acoustic digital human workbench — Rust/WASM USCT + R3F UI (#595 )	2026-06-22 09:54:22 -04:00
check_brain_status.sh
claude-code-decompile.sh
claude-code-rvf-corpus.sh
create-brainpedia.py
deploy-crawl-phase1.sh
deploy-dragnes.sh	feat(sse): decouple SSE to mcp.pi.ruv.io proxy + Claude Code source research	2026-04-02 23:39:56 +00:00
deploy-gemini-agents.sh	feat: DrAgnes + Common Crawl WET + Gemini grounding agents (#282 )	2026-03-23 10:12:50 -04:00
deploy-wet-job.sh
deploy_brain_services.sh
deploy_trainer.sh
discover_and_train.sh
extract-ruqu-rvdna-submodules.sh	ADR-257: extract ruqu + rvdna into two standalone repos (git submodules) (#579 )	2026-06-17 17:33:40 -04:00
gemini-agents.js
generate-rvf-manifest.py
historical-crawl-import.sh
publish-rvf.sh
README.md
rebuild-all-versions.mjs
run_mincut_bench.sh	feat: Add min-cut gating experiment scaffolding (WIP)	2026-02-20 06:52:43 +00:00
seed-brain-all.py
seed-brain.rs	fix: ruvector-postgres v0.3.1 — audit bug fixes, 46 SQL functions, Docker publish (#227 )	2026-03-03 12:53:10 -05:00
seed-specialized.py
setup-gcs-examples.sh
sift1m_hnswlib_bench.mjs	perf(hnsw): 4-acc AVX-512 + parallel-insert — +9.7% build throughput (query QPS unchanged: memory-bound at 1M scale) (#619 )	2026-06-28 20:45:35 -04:00
sql-audit-v3.sql
swarm_train_15.sh
sync-lockfile.sh
train-lora.py
train_brain.sh
training_orchestrator.sh
upvote_memories.py
vote-boost.py
wet-filter-inject.js
wet-full-import.sh
wet-job.yaml
wet-orchestrate.sh
wet-processor.sh

README.md

RuVector Automation Scripts

This directory contains automation scripts organized by purpose.

📁 Directory Structure

scripts/
├── README.md           # This file
├── benchmark/          # Performance benchmarking
├── build/              # Build utilities
├── ci/                 # CI/CD automation
├── deploy/             # Deployment scripts
├── patches/            # Patch files
├── publish/            # Package publishing
├── test/               # Testing scripts
└── validate/           # Validation & verification

🚀 Deployment

Scripts for deploying to production.

Script	Description
`deploy/deploy.sh`	Comprehensive deployment (crates.io + npm)
`deploy/test-deploy.sh`	Test deployment without publishing
`deploy/DEPLOYMENT.md`	Full deployment documentation
`deploy/DEPLOYMENT-QUICKSTART.md`	Quick deployment guide

Usage:

# Full deployment
./scripts/deploy/deploy.sh

# Dry run
./scripts/deploy/deploy.sh --dry-run

# Test deployment
./scripts/deploy/test-deploy.sh

📦 Publishing

Scripts for publishing packages to registries.

Script	Description
`publish/publish-all.sh`	Publish all packages
`publish/publish-crates.sh`	Publish Rust crates to crates.io
`publish/publish-cli.sh`	Publish CLI package
`publish/publish-router-wasm.sh`	Publish router WASM package
`publish/check-and-publish-router-wasm.sh`	Check and publish router WASM

Usage:

# Set credentials first
export CRATES_API_KEY="your-crates-io-token"
export NPM_TOKEN="your-npm-token"

# Publish all
./scripts/publish/publish-all.sh

# Publish crates only
./scripts/publish/publish-crates.sh

📊 Benchmarking

Performance benchmarking scripts.

Script	Description
`benchmark/run_benchmarks.sh`	Run core benchmarks
`benchmark/run_llm_benchmarks.sh`	Run LLM inference benchmarks

Usage:

# Run core benchmarks
./scripts/benchmark/run_benchmarks.sh

# Run LLM benchmarks
./scripts/benchmark/run_llm_benchmarks.sh

🧪 Testing

Testing and validation scripts.

Script	Description
`test/test-wasm.mjs`	Test WASM bindings
`test/test-graph-cli.sh`	Test graph CLI commands
`test/test-all-graph-commands.sh`	Test all graph commands
`test/test-docker-package.sh`	Test Docker packaging

Usage:

# Test WASM
node ./scripts/test/test-wasm.mjs

# Test graph CLI
./scripts/test/test-graph-cli.sh

✅ Validation

Package and build verification scripts.

Script	Description
`validate/validate-packages.sh`	Validate package configs
`validate/validate-packages-simple.sh`	Simple package validation
`validate/verify-paper-impl.sh`	Verify paper implementation
`validate/verify_hnsw_build.sh`	Verify HNSW build

Usage:

# Validate packages
./scripts/validate/validate-packages.sh

# Verify HNSW
./scripts/validate/verify_hnsw_build.sh

🔄 CI/CD

Continuous integration scripts.

Script	Description
`ci/ci-sync-lockfile.sh`	Auto-fix lock files in CI
`ci/sync-lockfile.sh`	Sync package-lock.json
`ci/install-hooks.sh`	Install git hooks

Usage:

# Install git hooks (recommended)
./scripts/ci/install-hooks.sh

# Sync lockfile
./scripts/ci/sync-lockfile.sh

🛠️ Build

Build utility scripts located in build/.

🩹 Patches

Patch files for dependencies located in patches/.

🚀 Quick Start

For Development

Install git hooks (recommended):
```
./scripts/ci/install-hooks.sh
```
Run tests:
```
./scripts/test/test-wasm.mjs
```

For Deployment

Set credentials:

export CRATES_API_KEY="your-crates-io-token"
export NPM_TOKEN="your-npm-token"

Dry run first:
```
./scripts/deploy/deploy.sh --dry-run
```
Deploy:
```
./scripts/deploy/deploy.sh
```

🔐 Security

Never commit credentials! Always use environment variables or .env file.

See deploy/DEPLOYMENT.md for security best practices.