ruvector/examples/ruvLLM/README.md

RuvLLM

Self-Optimizing Neural Architecture (SONA) with LFM2 Cortex, Ruvector Memory, and Intelligent Routing

"The intelligence is not in one model anymore. It is in the loop."

---

What is RuvLLM?

RuvLLM is a self-learning language model orchestration system that combines frozen foundation models with adaptive memory and intelligent routing. Unlike traditional LLMs that rely solely on static parameters, RuvLLM continuously improves from every interaction through three temporal learning loops.

Key Innovation: RuvLLM doesn't replace your LLM—it makes any LLM smarter over time by learning from experience, routing intelligently, and preventing catastrophic forgetting.

┌─────────────────────────────────────────────────────────────────────────┐
│                         RuvLLM Architecture                              │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│    Query ──► Embedding ──► Memory Search ──► Router Decision            │
│                               │                    │                     │
│                               ▼                    ▼                     │
│                         Graph Attention      Model Selection             │
│                               │                    │                     │
│                               └────────┬───────────┘                     │
│                                        ▼                                 │
│                              ┌─────────────────────┐                     │
│                              │   LLM Inference    │                     │
│                              │  (Any LLM Backend)  │                     │
│                              └─────────────────────┘                     │
│                                        │                                 │
│                                        ▼                                 │
│                    ┌───────────────────────────────────┐                │
│                    │  SONA Learning (3 Temporal Loops) │                │
│                    │  • Instant: Per-request MicroLoRA │                │
│                    │  • Background: Hourly patterns    │                │
│                    │  • Deep: Weekly EWC++ updates     │                │
│                    └───────────────────────────────────┘                │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

---

Features

Core Components

Component	Description	Implementation
LFM2 Cortex	Frozen reasoning engine (135M-2.6B params)	Mock, Candle, or external (llama.cpp/vLLM)
Ruvector Memory	Adaptive synaptic mesh with HNSW indexing	Full CPU implementation with graph expansion
FastGRNN Router	Intelligent model selection circuit	Sparse + low-rank matrices with EWC learning
Graph Attention	Multi-head attention with edge features	8-head attention, layer normalization
SONA Engine	Self-optimizing neural architecture	LoRA + EWC++ + ReasoningBank

SONA: Self-Optimizing Neural Architecture

RuvLLM introduces SONA, a three-tier temporal learning system:

┌──────────────────────────────────────────────────────────────────────────┐
│  Loop A: Instant (Per-Request)                           Latency: <100μs │
│  ──────────────────────────────────────                                  │
│  • Records query trajectories with activation patterns                   │
│  • MicroLoRA adaptation (rank 1-2) for immediate improvement             │
│  • SIMD-optimized: 2,236 ops/sec throughput                              │
├──────────────────────────────────────────────────────────────────────────┤
│  Loop B: Background (Hourly)                                             │
│  ─────────────────────────────                                           │
│  • K-means++ clustering extracts patterns (100 clusters = 1.3ms search)  │
│  • Base LoRA updates (rank 4-16) from successful patterns                │
│  • ReasoningBank stores learned strategies                               │
├──────────────────────────────────────────────────────────────────────────┤
│  Loop C: Deep (Weekly)                                                   │
│  ─────────────────────                                                   │
│  • Dream consolidation across all memory                                 │
│  • EWC++ prevents catastrophic forgetting (λ=2000 optimal)               │
│  • Concept hierarchies created, old nodes archived                       │
└──────────────────────────────────────────────────────────────────────────┘

Advanced Features

Feature	Description
SIMD Inference	Native AVX2/AVX512/SSE4.1 operations for CPU optimization
Q4 Quantization	4-bit weight quantization for memory efficiency
MicroLoRA	Per-request adaptation with rank 1-2 (benchmark: rank-2 is 5% faster)
EWC++	Enhanced elastic weight consolidation with online Fisher estimation
ReasoningBank	Pattern storage with K-means++ clustering
HuggingFace Export	Export LoRA weights, patterns, and preference pairs
Real Inference	Candle-based inference with HuggingFace model support
Multi-Model Routing	Automatic selection between SmolLM, Qwen2, TinyLlama
Federated Learning	Distributed learning across ephemeral agents with central coordinator
WASM Support	Run SONA in browsers and edge devices
Training Pipelines	Templated training for code, chat, reasoning, and custom agents
Agent Factory	Create and manage multiple specialized learning agents

Federated Learning Architecture

RuvLLM supports federated learning where ephemeral agents collect trajectories and export to a central coordinator:

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│  Agent A    │     │  Agent B    │     │  Agent C    │
│ (ephemeral) │     │ (ephemeral) │     │ (ephemeral) │
└──────┬──────┘     └──────┬──────┘     └──────┬──────┘
       │                   │                   │
       │    export()       │    export()       │    export()
       ▼                   ▼                   ▼
  ┌────────────────────────────────────────────────┐
  │            Federated Coordinator               │
  │         (persistent, large capacity)           │
  │  • Aggregates trajectories from all agents     │
  │  • Quality-filtered acceptance (threshold)     │
  │  • Auto-consolidation every N agents           │
  │  • Shares patterns with new agents             │
  └────────────────────────────────────────────────┘

Key Components:

• EphemeralAgent: Short-lived agents that process tasks and export learned state
• FederatedCoordinator: Central aggregator with 50K trajectory capacity
• AgentExport: Serializable state containing trajectories, stats, and patterns
• Quality Filtering: Only high-quality trajectories (>0.4 score) are aggregated

---

Performance Benchmarks

Orchestration Latency (CPU-Only)

Metric	Value	Notes
Initialization	3.71ms	Full system startup
Average Query	0.09ms	Single query latency
Session Query	0.04ms	With context reuse
Throughput	~38,000 q/s	8 concurrent queries
Memory Footprint	~50MB	Base system

Latency Breakdown

Embedding:    ~0.02ms  ████░░░░░░  (20%)
Retrieval:    ~0.01ms  ██░░░░░░░░  (10%)
Routing:      ~0.01ms  ██░░░░░░░░  (10%)
Attention:    ~0.02ms  ████░░░░░░  (20%)
Generation:   ~0.04ms  ████████░░  (40%)

SONA Learning Performance

Component	Metric	Value
MicroLoRA	Throughput	2,236 ops/sec
MicroLoRA	Batch-32 Latency	0.447ms
ReasoningBank	Pattern Search	1.3ms (100 clusters)
EWC++	Fisher Update	<1ms

Comparison with Traditional Systems

System	P50 (ms)	P95 (ms)	vs GPT-4o
GPT-4o (API)	450.00	585.00	1.0x (baseline)
Claude 3.5 Sonnet	380.00	456.00	1.2x
Gemini 2.0 Flash	180.00	234.00	2.5x
Llama 3.3 70B (vLLM)	120.00	168.00	3.8x
RuvLLM Orchestration	0.06	0.08	~7,500x

Note

: RuvLLM orchestration latency measures memory retrieval, routing, and context preparation—NOT LLM generation. Actual response quality depends on your LLM backend.

---

Feature Comparison

Feature	GPT-4o	Claude	RAG	vLLM	RuvLLM
On-device Inference	✗	✗	✗	✓	✓
Continuous Learning	✗	✗	✗	✗	✓
Graph-based Memory	✗	✗	△	✗	✓
Adaptive Model Routing	✗	✗	✗	✗	✓
EWC Anti-Forgetting	✗	✗	✗	✗	✓
LoRA Adaptation	✗	✗	✗	✗	✓
Pattern Extraction	✗	✗	✗	✗	✓
HuggingFace Export	✗	✗	✗	✗	✓
SIMD Optimization	✗	✗	✗	△	✓
Sub-ms Orchestration	✗	✗	✗	✗	✓
Federated Learning	✗	✗	✗	✗	✓
WASM/Browser Support	✗	✗	✗	✗	✓
Training Pipelines	✗	✗	✗	✗	✓
Works with ANY LLM	✗	✗	✓	✗	✓

Legend: ✓ = Full Support, △ = Partial, ✗ = Not Supported

---

Quick Start

Prerequisites

• Rust 1.77+
• Cargo

Installation

# Clone the repository
git clone https://github.com/ruvnet/ruvector.git
cd ruvector/examples/ruvLLM

# Build in release mode
cargo build --release

Run the Demo

# Interactive demo with mock inference
cargo run --bin ruvllm-demo --release

# SIMD capabilities demo
cargo run --bin ruvllm-simd-demo --release

# Quick benchmark
cargo run --bin ruvllm-bench --release

# Full benchmark suite
cargo run --bin ruvllm-benchmark-suite --release

# HTTP server (requires 'server' feature)
cargo run --bin ruvllm-server --release --features server

# Pretraining pipeline
cargo run --bin ruvllm-pretrain --release

# HuggingFace export (requires 'hf-export' feature)
cargo run --bin ruvllm-export --release --features hf-export -- help

Library Usage

use ruvllm::{Config, RuvLLM, Result};

#[tokio::main]
async fn main() -> Result<()> {
    // Configure the system
    let config = Config::builder()
        .embedding_dim(768)
        .router_hidden_dim(128)
        .hnsw_params(32, 200, 64)  // M, ef_construction, ef_search
        .learning_enabled(true)
        .build()?;

    // Initialize
    let llm = RuvLLM::new(config).await?;

    // Create a session for multi-turn conversation
    let session = llm.new_session();

    // Query with session context
    let response = llm.query_session(&session, "What is machine learning?").await?;

    println!("Response: {}", response.text);
    println!("Model: {:?}", response.routing_info.model);
    println!("Confidence: {:.2}%", response.confidence * 100.0);

    // Provide feedback for learning
    llm.feedback(Feedback {
        request_id: response.request_id,
        rating: Some(5),
        correction: None,
        task_success: Some(true),
    }).await?;

    Ok(())
}

SIMD Inference Engine

use ruvllm::{SimdInferenceEngine, SimdGenerationConfig, SimdOps};

// Create SIMD-optimized engine
let engine = SimdInferenceEngine::new(256, 128, 4, 4)?;

// Configure generation
let config = SimdGenerationConfig {
    max_tokens: 50,
    temperature: 0.7,
    top_p: 0.9,
    ..Default::default()
};

// Generate with SIMD acceleration
let result = engine.generate("Once upon a time", &config)?;

SONA Learning Loops

use ruvllm::sona::{LoopCoordinator, SonaConfig, InstantLoop, BackgroundLoop};

// Initialize SONA coordinator
let config = SonaConfig {
    hidden_dim: 256,
    embedding_dim: 256,
    pattern_clusters: 100,
    ..Default::default()
};

let coordinator = LoopCoordinator::new(config);

// Instant learning (per-request)
coordinator.instant_loop().record_trajectory(query, response, quality);

// Background learning (hourly)
coordinator.background_loop().extract_patterns().await;

// Deep learning (weekly) - automatically handles EWC++
coordinator.deep_consolidation().await;

Federated Learning

use ruvector_sona::training::{EphemeralAgent, FederatedCoordinator, SonaConfig};

// Create central coordinator (persistent, large capacity)
let mut coordinator = FederatedCoordinator::default_coordinator("main", 3072);
coordinator.set_quality_threshold(0.4);  // Only accept high-quality trajectories
coordinator.set_consolidation_interval(50);  // Auto-consolidate every 50 agents

// Create ephemeral agents for distributed learning
let mut agent = EphemeralAgent::default_federated("agent-1", 3072);

// Agent processes tasks and learns locally
agent.process_trajectory(
    embedding,      // Query embedding
    activations,    // Hidden state activations
    quality,        // Quality score [0.0, 1.0]
    Some("gpt-4".to_string()),  // Model route
    vec!["code".to_string()],   // Context tags
);

// Export state before agent termination
let export = agent.export_state();
println!("Agent exported {} trajectories", export.trajectories.len());

// Coordinator aggregates learning from all agents
let result = coordinator.aggregate(export);
println!("Accepted: {}, Rejected: {}",
    result.trajectories_accepted,
    result.trajectories_rejected
);

// Get patterns for warm-starting new agents
let patterns = coordinator.get_initial_patterns(10);

WASM Usage (Browser/Edge)

Build SONA for WebAssembly:

# Build WASM package
cd crates/sona
wasm-pack build --target web --features wasm

Use in JavaScript:

import init, { WasmSonaEngine } from './pkg/sona.js';

async function main() {
  await init();

  // Create SONA engine
  const engine = new WasmSonaEngine(256);  // hidden_dim = 256

  // Or with custom configuration
  const engineCustom = WasmSonaEngine.withConfig({
    hidden_dim: 256,
    embedding_dim: 256,
    micro_lora_rank: 2,
    base_lora_rank: 16,
    ewc_lambda: 1000.0,
    pattern_clusters: 128,
  });

  // Start trajectory
  const embedding = new Float32Array(256).fill(0.1);
  const trajectoryId = engine.startTrajectory(embedding);

  // Record steps
  engine.recordStep(trajectoryId, 42, 0.8, 1000);

  // End trajectory with quality score
  engine.endTrajectory(trajectoryId, 0.85);

  // Apply LoRA transformation
  const input = new Float32Array(256).fill(1.0);
  const output = engine.applyLora(input);

  // Run learning cycles
  engine.runInstantCycle();  // Flush micro-LoRA updates
  if (engine.tick()) {       // Background learning
    console.log('Background learning completed');
  }

  // Get statistics
  const stats = engine.stats();
  console.log('Patterns:', stats.patterns_stored);
}

---

HuggingFace Export

Export learned patterns, LoRA weights, and preference pairs to HuggingFace:

# Export LoRA weights in PEFT-compatible SafeTensors format
ruvllm-export safetensors ./exports/lora

# Export learned patterns as JSONL dataset
ruvllm-export patterns ./exports/patterns

# Export DPO/RLHF preference pairs
ruvllm-export preferences ./exports/preferences

# Export all artifacts
ruvllm-export all ./exports

# Push to HuggingFace Hub
HF_TOKEN=your_token ruvllm-export push username/my-sona-model

# Generate pretraining pipeline configuration
ruvllm-export pretrain ./exports

---

Architecture Deep Dive

HNSW Memory Index

The memory system uses Hierarchical Navigable Small World graphs:

Layer 2:  [3] ─────────────────── [7]
           │                       │
Layer 1:  [3] ─── [5] ─────────── [7] ─── [9]
           │      │                │       │
Layer 0:  [1]─[2]─[3]─[4]─[5]─[6]─[7]─[8]─[9]─[10]

• M = 32 connections per node
• ef_construction = 200 for build quality
• ef_search = 64 for query speed
• O(log N) search complexity

FastGRNN Router

Sparse + Low-rank matrices for efficient routing:

           Input (128-dim)
                │
        ┌───────┴───────┐
        │  LayerNorm    │
        └───────┬───────┘
                │
    ┌───────────┴───────────┐
    │   FastGRNN Cell       │
    │                       │
    │  W_sparse (90% zero)  │
    │  U = A @ B (rank-8)   │
    │                       │
    │  z = σ(Wx + Uh + b)   │
    │  h' = z⊙h + (1-z)⊙ν   │
    └───────────┬───────────┘
                │
        ┌───────┴───────┐
        │ Output Heads  │
        ├───────────────┤
        │ Model Select  │ → 4 classes
        │ Context Size  │ → 5 buckets
        │ Temperature   │ → continuous
        │ Top-p         │ → continuous
        │ Confidence    │ → continuous
        └───────────────┘

MicroLoRA Architecture

Two-tier LoRA system for adaptive learning:

┌─────────────────────────────────────────────────────────────┐
│                      MicroLoRA (Rank 1-2)                   │
│                   Per-Request Adaptation                    │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│   Input ──► Down Proj ──► Up Proj ──► Scale ──► Add        │
│   (dim)     (dim→rank)   (rank→dim)   (α/r)    to output   │
│                                                             │
│   Performance: <100μs latency, 2,236 ops/sec               │
│   Rank-2 is ~5% faster than Rank-1 (better SIMD)           │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                      BaseLoRA (Rank 4-16)                   │
│                   Background Adaptation                     │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│   Aggregated from successful MicroLoRA patterns             │
│   Merged hourly into base weights                           │
│   EWC++ regularization prevents forgetting                  │
│                                                             │
└─────────────────────────────────────────────────────────────┘

EWC++ (Enhanced Elastic Weight Consolidation)

Prevents catastrophic forgetting:

Loss = Task_Loss + λ * Σᵢ Fᵢ(θᵢ - θ*ᵢ)²

Where:
• Fᵢ = Online Fisher information (EMA decay 0.999)
• θ*ᵢ = Optimal weights for previous tasks
• λ = Adaptive (2000 default, range 100-15000)
• Multi-task memory with circular buffer (10 tasks)
• Automatic task boundary detection

SIMD Operations

Native CPU acceleration:

// AVX2 dot product (8 floats at a time)
#[target_feature(enable = "avx2")]
unsafe fn dot_product_avx2(a: &[f32], b: &[f32]) -> f32

// SSE4.1 fallback (4 floats at a time)
#[target_feature(enable = "sse4.1")]
unsafe fn dot_product_sse(a: &[f32], b: &[f32]) -> f32

// Automatic detection and dispatch
let result = SimdOps::dot_product(&a, &b);

---

Supported Models

Real Inference (CPU SIMD)

Model	Parameters	Context	Repo
SmolLM 135M	135M	2048	HuggingFaceTB/SmolLM-135M
SmolLM 360M	360M	2048	HuggingFaceTB/SmolLM-360M
Qwen2 0.5B	500M	4096	Qwen/Qwen2-0.5B
TinyLlama 1.1B	1.1B	2048	TinyLlama/TinyLlama-1.1B-Chat

All models support Q4_K_M quantization for efficient CPU inference.

---

HTTP Server API

When running with the server feature:

Endpoint	Method	Description
/health	GET	Health check
/query	POST	Submit query
/stats	GET	Get statistics
/feedback	POST	Submit feedback
/session	POST	Create new session

# Example query
curl -X POST http://localhost:3000/query \
  -H "Content-Type: application/json" \
  -d '{"query": "What is Rust?", "session_id": null}'

---

Testing

# Run all tests
cargo test -p ruvllm

# Unit tests only (47 tests)
cargo test -p ruvllm --lib

# Integration tests (15 tests)
cargo test -p ruvllm --test integration

# With output
cargo test -p ruvllm -- --nocapture

Test Coverage

Module	Tests	Coverage
Memory (HNSW)	12	Search, insertion, graph expansion
Router (FastGRNN)	8	Forward pass, training, EWC
Attention	6	Multi-head, edge features, cross-attention
Embedding	9	Tokenization, caching, pooling
SONA	10	LoRA, EWC++, ReasoningBank, loops
Orchestrator	2	End-to-end pipeline
Integration	15	Full system tests

---

Project Structure

examples/ruvLLM/
├── Cargo.toml              # Dependencies and features
├── README.md               # This file
├── src/
│   ├── lib.rs              # Library entry point
│   ├── config.rs           # Configuration system
│   ├── error.rs            # Error types
│   ├── types.rs            # Core domain types
│   ├── orchestrator.rs     # Main RuvLLM coordinator
│   ├── memory.rs           # HNSW memory service
│   ├── router.rs           # FastGRNN router
│   ├── attention.rs        # Graph attention engine
│   ├── embedding.rs        # Embedding service
│   ├── inference.rs        # Mock inference pool
│   ├── inference_real.rs   # Candle-based real inference
│   ├── simd_inference.rs   # SIMD-optimized transformer
│   ├── learning.rs         # Self-learning service
│   ├── compression.rs      # Memory compression
│   ├── training.rs         # Pretraining pipeline
│   ├── sona/               # SONA module
│   │   ├── mod.rs          # Module exports
│   │   ├── types.rs        # SONA types
│   │   ├── lora.rs         # MicroLoRA & BaseLoRA
│   │   ├── ewc.rs          # EWC++ implementation
│   │   ├── reasoning_bank.rs  # Pattern storage
│   │   ├── trajectory.rs   # Trajectory recording
│   │   ├── engine.rs       # SONA engine
│   │   └── loops/          # Temporal learning loops
│   │       ├── instant.rs  # Per-request loop
│   │       ├── background.rs  # Hourly loop
│   │       └── coordinator.rs # Loop coordinator
│   └── bin/
│       ├── demo.rs         # Interactive demo
│       ├── bench.rs        # Quick benchmarks
│       ├── benchmark_suite.rs  # Full benchmark suite
│       ├── simd_demo.rs    # SIMD capabilities demo
│       ├── pretrain.rs     # Pretraining pipeline
│       ├── export.rs       # HuggingFace export
│       └── server.rs       # HTTP server
├── tests/
│   └── integration.rs      # Integration tests
├── benches/
│   ├── pipeline.rs         # Full pipeline benchmarks
│   ├── router.rs           # Router benchmarks
│   ├── memory.rs           # Memory benchmarks
│   ├── attention.rs        # Attention benchmarks
│   └── sona_bench.rs       # SONA benchmarks
├── config/                 # Configuration files
└── docs/
    └── sparc/              # SPARC methodology docs

---

Feature Flags

RuvLLM Features

Feature	Default	Description
storage	✓	Persistent storage and HNSW indexing
metrics	✓	Prometheus metrics export
server	✗	HTTP server with Axum
real-inference	✗	Candle-based real LLM inference
hf-export	✗	HuggingFace export via ruvector-sona
full	✗	All features enabled

# Build with all features
cargo build --release --features full

ruvector-sona Features (Dependency)

Feature	Default	Description
serde-support	✓	Serialization for export, training, and federated learning
wasm	✗	WebAssembly bindings for browser/edge deployment
napi	✗	N-API bindings for Node.js integration

# Build SONA with WASM support
cd crates/sona
wasm-pack build --target web --features wasm

---

Configuration Options

Option	Default	Description
embedding.dimension	768	Embedding vector size
embedding.max_tokens	512	Max tokens per input
memory.hnsw_m	16	HNSW connections per node
memory.hnsw_ef_construction	100	Build quality parameter
memory.hnsw_ef_search	64	Search quality parameter
router.input_dim	128	Router input features
router.hidden_dim	64	FastGRNN hidden size
router.sparsity	0.9	Weight matrix sparsity
router.rank	8	Low-rank decomposition
learning.enabled	true	Enable self-learning
learning.quality_threshold	0.7	Min quality for writeback
learning.ewc_lambda	2000	EWC regularization strength
sona.pattern_clusters	100	K-means++ clusters
sona.micro_lora_rank	2	MicroLoRA rank

Federated Learning Configuration

Option	Default	Description
federated.quality_threshold	0.4	Min quality for trajectory acceptance
federated.consolidation_interval	50	Auto-consolidate every N agents
federated.coordinator_capacity	50000	Trajectory buffer size for coordinator
federated.agent_capacity	500	Trajectory buffer size per agent
federated.base_lora_rank	16	Coordinator LoRA rank (deeper for aggregation)

---

Self-Learning Improvement Over Time

Epoch	Queries	Quality	Routing	Cache Hit	Memory	Improvement
0	0	65.0%	50.0%	0.0%	0	0.0% (baseline)
1	50	67.2%	58.0%	10.0%	25	+3.4%
2	100	69.8%	66.0%	20.0%	50	+7.4%
3	150	71.5%	74.0%	30.0%	75	+10.0%
4	200	73.2%	82.0%	40.0%	100	+12.6%
5	250	74.8%	90.0%	50.0%	125	+15.1%

---

References

• LFM2: Liquid Foundation Models - Gated convolutions + grouped query attention
• FastGRNN - Fast, Accurate, Stable and Tiny GRU
• HNSW - Hierarchical Navigable Small World Graphs
• EWC - Elastic Weight Consolidation
• LoRA - Low-Rank Adaptation of Large Language Models

---

License

Licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

---

Built with Rust + Ruvector
Self-Learning AI that gets smarter with every interaction