mirror of
https://github.com/ruvnet/RuVector.git
synced 2026-07-09 17:28:42 +00:00
feb4ee2753
* perf(hnsw): 4-accumulator AVX-512 kernels + SIMD wiring into search hot path - Replace single-accumulator AVX-512 distance kernels with 4-accumulator versions in simd_intrinsics.rs (euclidean, cosine, dot, manhattan). On Zen 5 with 4-cycle FMA latency, single-accumulator was latency-bound (96 cycles for 384-dim); 4-accumulator hides this to ~24 cycles. - Wire HNSW search hot path in DistanceFn::eval to call simd_intrinsics directly (inline, no Result wrapping, no simsimd FFI overhead). - Enable parallel batch insert via hnsw_rs::parallel_insert_slice (rayon). Measured: 6-10% QPS improvement on 128-dim/1K-vector bench; larger gains expected on 1M-vector workloads where distance computation dominates. 228 unit tests pass. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019rVRYrRDKyxYK18kuVrDSf * perf(hnsw): gate parallel_insert_slice behind 10K-vector threshold Rayon-based parallel insert (hnsw_rs::parallel_insert_slice) degrades graph connectivity for small batches (<10K vectors) because worker threads can't see each other's in-flight insertions, reducing optimal neighbor links. Add PARALLEL_THRESHOLD=10_000: use parallel insert only when the batch is large enough that the graph quality converges. Below threshold: sequential insert_data (same as before this PR). Above threshold: parallel_insert_slice for build-time speedup. 228 unit tests pass. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019rVRYrRDKyxYK18kuVrDSf * bench(sift1m): add SIFT-1M fvecs benchmark + hnswlib comparison tooling Adds two benchmark binaries driven by the real TEXMEX SIFT-1M dataset: * crates/ruvector-sota-bench/src/bin/sift1m_bench.rs Reads sift_base.fvecs / sift_query.fvecs / sift_groundtruth.ivecs directly (no HDF5 required). Sweeps ef_search to produce a recall@10 vs QPS table used for before/after PR #619 comparison. * scripts/sift1m_hnswlib_bench.mjs Same sweep via hnswlib-node (C++ HNSW) to measure the competitive gap. Cargo.toml: add simd-avx512 feature to sota-bench dependency so the full optimised kernel path is exercised. Measured on AMD Ryzen 9 9950X (Zen 5, AVX-512), M=16, efC=200, 1M vecs: Source Build ef=100 recall ef=100 QPS ef=200 recall ef=200 QPS before PR 849 s 0.9585 1,849 0.9713 1,058 after PR (#619) 774 s 0.9592 1,768 0.9722 1,024 hnswlib-node 322 s 0.9828 5,339 0.9957 2,897 Build speedup: +9.7 %. Query QPS at 1M-scale: within noise (memory- bandwidth bound, not compute-bound). Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019rVRYrRDKyxYK18kuVrDSf * style: cargo fmt for sift1m benchmark binary Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019rVRYrRDKyxYK18kuVrDSf --------- Co-authored-by: ruvnet <ruvnet@gmail.com> |
||
|---|---|---|
| .. | ||
| benchmark | ||
| build | ||
| ci | ||
| deploy | ||
| lib | ||
| patches/hnsw_rs | ||
| publish | ||
| sona-drift | ||
| test | ||
| training | ||
| validate | ||
| analyze-evolution.js | ||
| analyze-ham10000.js | ||
| build-solver.sh | ||
| build-sonic-ct-wasm.sh | ||
| check_brain_status.sh | ||
| claude-code-decompile.sh | ||
| claude-code-rvf-corpus.sh | ||
| create-brainpedia.py | ||
| deploy-crawl-phase1.sh | ||
| deploy-dragnes.sh | ||
| deploy-gemini-agents.sh | ||
| deploy-wet-job.sh | ||
| deploy_brain_services.sh | ||
| deploy_trainer.sh | ||
| discover_and_train.sh | ||
| extract-ruqu-rvdna-submodules.sh | ||
| gemini-agents.js | ||
| generate-rvf-manifest.py | ||
| historical-crawl-import.sh | ||
| publish-rvf.sh | ||
| README.md | ||
| rebuild-all-versions.mjs | ||
| run_mincut_bench.sh | ||
| seed-brain-all.py | ||
| seed-brain.rs | ||
| seed-specialized.py | ||
| setup-gcs-examples.sh | ||
| sift1m_hnswlib_bench.mjs | ||
| sql-audit-v3.sql | ||
| swarm_train_15.sh | ||
| sync-lockfile.sh | ||
| train-lora.py | ||
| train_brain.sh | ||
| training_orchestrator.sh | ||
| upvote_memories.py | ||
| vote-boost.py | ||
| wet-filter-inject.js | ||
| wet-full-import.sh | ||
| wet-job.yaml | ||
| wet-orchestrate.sh | ||
| wet-processor.sh | ||
RuVector Automation Scripts
This directory contains automation scripts organized by purpose.
📁 Directory Structure
scripts/
├── README.md # This file
├── benchmark/ # Performance benchmarking
├── build/ # Build utilities
├── ci/ # CI/CD automation
├── deploy/ # Deployment scripts
├── patches/ # Patch files
├── publish/ # Package publishing
├── test/ # Testing scripts
└── validate/ # Validation & verification
🚀 Deployment
Scripts for deploying to production.
| Script | Description |
|---|---|
deploy/deploy.sh |
Comprehensive deployment (crates.io + npm) |
deploy/test-deploy.sh |
Test deployment without publishing |
deploy/DEPLOYMENT.md |
Full deployment documentation |
deploy/DEPLOYMENT-QUICKSTART.md |
Quick deployment guide |
Usage:
# Full deployment
./scripts/deploy/deploy.sh
# Dry run
./scripts/deploy/deploy.sh --dry-run
# Test deployment
./scripts/deploy/test-deploy.sh
📦 Publishing
Scripts for publishing packages to registries.
| Script | Description |
|---|---|
publish/publish-all.sh |
Publish all packages |
publish/publish-crates.sh |
Publish Rust crates to crates.io |
publish/publish-cli.sh |
Publish CLI package |
publish/publish-router-wasm.sh |
Publish router WASM package |
publish/check-and-publish-router-wasm.sh |
Check and publish router WASM |
Usage:
# Set credentials first
export CRATES_API_KEY="your-crates-io-token"
export NPM_TOKEN="your-npm-token"
# Publish all
./scripts/publish/publish-all.sh
# Publish crates only
./scripts/publish/publish-crates.sh
📊 Benchmarking
Performance benchmarking scripts.
| Script | Description |
|---|---|
benchmark/run_benchmarks.sh |
Run core benchmarks |
benchmark/run_llm_benchmarks.sh |
Run LLM inference benchmarks |
Usage:
# Run core benchmarks
./scripts/benchmark/run_benchmarks.sh
# Run LLM benchmarks
./scripts/benchmark/run_llm_benchmarks.sh
🧪 Testing
Testing and validation scripts.
| Script | Description |
|---|---|
test/test-wasm.mjs |
Test WASM bindings |
test/test-graph-cli.sh |
Test graph CLI commands |
test/test-all-graph-commands.sh |
Test all graph commands |
test/test-docker-package.sh |
Test Docker packaging |
Usage:
# Test WASM
node ./scripts/test/test-wasm.mjs
# Test graph CLI
./scripts/test/test-graph-cli.sh
✅ Validation
Package and build verification scripts.
| Script | Description |
|---|---|
validate/validate-packages.sh |
Validate package configs |
validate/validate-packages-simple.sh |
Simple package validation |
validate/verify-paper-impl.sh |
Verify paper implementation |
validate/verify_hnsw_build.sh |
Verify HNSW build |
Usage:
# Validate packages
./scripts/validate/validate-packages.sh
# Verify HNSW
./scripts/validate/verify_hnsw_build.sh
🔄 CI/CD
Continuous integration scripts.
| Script | Description |
|---|---|
ci/ci-sync-lockfile.sh |
Auto-fix lock files in CI |
ci/sync-lockfile.sh |
Sync package-lock.json |
ci/install-hooks.sh |
Install git hooks |
Usage:
# Install git hooks (recommended)
./scripts/ci/install-hooks.sh
# Sync lockfile
./scripts/ci/sync-lockfile.sh
🛠️ Build
Build utility scripts located in build/.
🩹 Patches
Patch files for dependencies located in patches/.
🚀 Quick Start
For Development
-
Install git hooks (recommended):
./scripts/ci/install-hooks.sh -
Run tests:
./scripts/test/test-wasm.mjs
For Deployment
-
Set credentials:
export CRATES_API_KEY="your-crates-io-token" export NPM_TOKEN="your-npm-token" -
Dry run first:
./scripts/deploy/deploy.sh --dry-run -
Deploy:
./scripts/deploy/deploy.sh
🔐 Security
Never commit credentials! Always use environment variables or .env file.
See deploy/DEPLOYMENT.md for security best practices.