mirror of
https://github.com/ruvnet/RuVector.git
synced 2026-05-23 21:25:02 +00:00
bd1e253755
* fix(brain): SSE connection limiter, pipeline rate limit, Firestore pagination fallback (ADR-130) Three fixes for recurring pi.ruv.io outages: 1. SSE connection limiter (max 50) — prevents MCP reconnect storms from exhausting Cloud Run concurrency slots. Tracks active count with AtomicUsize, rejects excess with 429. 2. Pipeline optimize rate limiter — max 1 concurrent request with 30s cooldown. Prevents scheduler thundering herd from CPU-saturating the instance. 3. Firestore pagination offset fallback — when page tokens go stale after OOM restart (400 Bad Request), switches to offset-based pagination to load all documents instead of stopping at first batch. Also adds /v1/ready lightweight probe (zero-cost, no state access) for Cloud Run health checks. ADR-130 documents the full decoupling architecture (SSE service split). Co-Authored-By: claude-flow <ruv@ruv.net> * feat(brain): ADR-130 service split — SSE proxy, worker binary, internal queue Implements full MCP SSE decoupling to eliminate recurring outages: 1. ruvbrain-sse: Thin SSE proxy (308 lines) that manages MCP connections independently from the API. Max 200 concurrent SSE, forwards JSON-RPC to the API, polls /internal/queue/drain for responses. No business logic. 2. ruvbrain-worker: Batch worker binary (202 lines) for Cloud Run Jobs. Runs scheduler actions (train, drift, transfer, graph, cleanup, attractor) with direct Firestore access. Runs once and exits. 3. Internal queue endpoints on the API: - POST /internal/queue/push (forward JSON-RPC to session) - GET /internal/queue/drain (poll for responses) - POST /internal/session/create (register session) - DELETE /internal/session/:id (cleanup) 4. Deploy infrastructure: - Dockerfile.sse, Dockerfile.worker - cloudbuild-sse.yaml, cloudbuild-worker.yaml - scripts/deploy_brain_services.sh [api|sse|worker|all] Architecture: SSE (500 concurrency, 512MB) → API (80 concurrency, 4GB) ← Worker (Cloud Run Job, 4GB) Co-Authored-By: claude-flow <ruv@ruv.net> |
||
|---|---|---|
| .. | ||
| benchmark | ||
| build | ||
| ci | ||
| deploy | ||
| patches/hnsw_rs | ||
| publish | ||
| test | ||
| training | ||
| validate | ||
| analyze-evolution.js | ||
| analyze-ham10000.js | ||
| build-solver.sh | ||
| check_brain_status.sh | ||
| create-brainpedia.py | ||
| deploy-crawl-phase1.sh | ||
| deploy-dragnes.sh | ||
| deploy-gemini-agents.sh | ||
| deploy-wet-job.sh | ||
| deploy_brain_services.sh | ||
| deploy_trainer.sh | ||
| discover_and_train.sh | ||
| gemini-agents.js | ||
| generate-rvf-manifest.py | ||
| historical-crawl-import.sh | ||
| publish-rvf.sh | ||
| README.md | ||
| run_mincut_bench.sh | ||
| seed-brain-all.py | ||
| seed-brain.rs | ||
| seed-specialized.py | ||
| setup-gcs-examples.sh | ||
| sql-audit-v3.sql | ||
| swarm_train_15.sh | ||
| sync-lockfile.sh | ||
| train-lora.py | ||
| train_brain.sh | ||
| training_orchestrator.sh | ||
| upvote_memories.py | ||
| vote-boost.py | ||
| wet-filter-inject.js | ||
| wet-full-import.sh | ||
| wet-job.yaml | ||
| wet-orchestrate.sh | ||
| wet-processor.sh | ||
RuVector Automation Scripts
This directory contains automation scripts organized by purpose.
📁 Directory Structure
scripts/
├── README.md # This file
├── benchmark/ # Performance benchmarking
├── build/ # Build utilities
├── ci/ # CI/CD automation
├── deploy/ # Deployment scripts
├── patches/ # Patch files
├── publish/ # Package publishing
├── test/ # Testing scripts
└── validate/ # Validation & verification
🚀 Deployment
Scripts for deploying to production.
| Script | Description |
|---|---|
deploy/deploy.sh |
Comprehensive deployment (crates.io + npm) |
deploy/test-deploy.sh |
Test deployment without publishing |
deploy/DEPLOYMENT.md |
Full deployment documentation |
deploy/DEPLOYMENT-QUICKSTART.md |
Quick deployment guide |
Usage:
# Full deployment
./scripts/deploy/deploy.sh
# Dry run
./scripts/deploy/deploy.sh --dry-run
# Test deployment
./scripts/deploy/test-deploy.sh
📦 Publishing
Scripts for publishing packages to registries.
| Script | Description |
|---|---|
publish/publish-all.sh |
Publish all packages |
publish/publish-crates.sh |
Publish Rust crates to crates.io |
publish/publish-cli.sh |
Publish CLI package |
publish/publish-router-wasm.sh |
Publish router WASM package |
publish/check-and-publish-router-wasm.sh |
Check and publish router WASM |
Usage:
# Set credentials first
export CRATES_API_KEY="your-crates-io-token"
export NPM_TOKEN="your-npm-token"
# Publish all
./scripts/publish/publish-all.sh
# Publish crates only
./scripts/publish/publish-crates.sh
📊 Benchmarking
Performance benchmarking scripts.
| Script | Description |
|---|---|
benchmark/run_benchmarks.sh |
Run core benchmarks |
benchmark/run_llm_benchmarks.sh |
Run LLM inference benchmarks |
Usage:
# Run core benchmarks
./scripts/benchmark/run_benchmarks.sh
# Run LLM benchmarks
./scripts/benchmark/run_llm_benchmarks.sh
🧪 Testing
Testing and validation scripts.
| Script | Description |
|---|---|
test/test-wasm.mjs |
Test WASM bindings |
test/test-graph-cli.sh |
Test graph CLI commands |
test/test-all-graph-commands.sh |
Test all graph commands |
test/test-docker-package.sh |
Test Docker packaging |
Usage:
# Test WASM
node ./scripts/test/test-wasm.mjs
# Test graph CLI
./scripts/test/test-graph-cli.sh
✅ Validation
Package and build verification scripts.
| Script | Description |
|---|---|
validate/validate-packages.sh |
Validate package configs |
validate/validate-packages-simple.sh |
Simple package validation |
validate/verify-paper-impl.sh |
Verify paper implementation |
validate/verify_hnsw_build.sh |
Verify HNSW build |
Usage:
# Validate packages
./scripts/validate/validate-packages.sh
# Verify HNSW
./scripts/validate/verify_hnsw_build.sh
🔄 CI/CD
Continuous integration scripts.
| Script | Description |
|---|---|
ci/ci-sync-lockfile.sh |
Auto-fix lock files in CI |
ci/sync-lockfile.sh |
Sync package-lock.json |
ci/install-hooks.sh |
Install git hooks |
Usage:
# Install git hooks (recommended)
./scripts/ci/install-hooks.sh
# Sync lockfile
./scripts/ci/sync-lockfile.sh
🛠️ Build
Build utility scripts located in build/.
🩹 Patches
Patch files for dependencies located in patches/.
🚀 Quick Start
For Development
-
Install git hooks (recommended):
./scripts/ci/install-hooks.sh -
Run tests:
./scripts/test/test-wasm.mjs
For Deployment
-
Set credentials:
export CRATES_API_KEY="your-crates-io-token" export NPM_TOKEN="your-npm-token" -
Dry run first:
./scripts/deploy/deploy.sh --dry-run -
Deploy:
./scripts/deploy/deploy.sh
🔐 Security
Never commit credentials! Always use environment variables or .env file.
See deploy/DEPLOYMENT.md for security best practices.